anyone define port types?

anyone define port types?

[Andy Wingo]

Hi!

I am working on improving our port implementation to take advantage of
the opportunity to break ABI in 2.2.  I am wondering how much I can
break C API as well -- there are some changes that would allow better
user-space threading
(e.g. http://thread.gmane.org/gmane.lisp.guile.devel/14158/focus=15463
or Chris Webber's 8sync).  But those changes would require some
incompatible changes to the C API related to port internals.  This API
is pretty much only used when implementing port types.  So, I would like
to collect a list of people that implement their own ports :) I know
Guile-GNOME does it for GNOME-VFS, though GNOME-VFS is super-old at this
point...  Anyway.  Looking forward to your links :)

Andy

Re: anyone define port types?

[Matt Wette]

> On Mar 28, 2016, at 12:04 PM, Andy Wingo <wingo@pobox.com> wrote:
> I am working on improving our port implementation to take advantage of
> the opportunity to break ABI in 2.2.  I am wondering how much I can
> break C API as well -- there are some changes that would allow better
> user-space threading

I made an attempt to use soft ports, but backed out after I could not figure a way to extra data I wanted to associate with my port.  

Addition of an access function to user-defined data would enlarge the cases for which soft ports could be used.

Matt

Re: anyone define port types?

[Nala Ginrut]

On Mon, 2016-03-28 at 21:04 +0200, Andy Wingo wrote:
> Hi!
> 
> I am working on improving our port implementation to take advantage of
> the opportunity to break ABI in 2.2.  I am wondering how much I can
> break C API as well -- there are some changes that would allow better
> user-space threading
> (e.g. http://thread.gmane.org/gmane.lisp.guile.devel/14158/focus=15463
> or Chris Webber's 8sync).  But those changes would require some
> incompatible changes to the C API related to port internals.  This API
> is pretty much only used when implementing port types.  So, I would like
> to collect a list of people that implement their own ports :) I know
> Guile-GNOME does it for GNOME-VFS, though GNOME-VFS is super-old at this
> point...  Anyway.  Looking forward to your links :)


Fortunately, I haven't do the work for the port, but it's likely to
customize the soft port in the future. If so, I would choose to stick to
2.1+

Best regards.

Re: anyone define port types?

[Panicz Maciej Godek]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]

Hi Andy,
I have been using soft ports to implement a text widget in my GUI
framework. I also used GOOPS, which I regret to this day, and so the whole
framework needs a serious rewrite, but if you're collecting various species
to the museum of make-soft-port, you can have a look:

https://bitbucket.org/panicz/slayer/src/8b741c21b7372c24874d7cd1875e2aae3fc43abe/guile-modules/widgets/text-area.scm?at=default&fileviewer=file-view-default

Regards,
Panicz

[-- Attachment #2: Type: text/html, Size: 808 bytes --]

Re: anyone define port types?

[Jan Nieuwenhuizen]

Panicz Maciej Godek writes:

> I also used GOOPS, which I regret to this day, and so the
> whole framework needs a serious rewrite

What is it that you do not like about GOOPS?  I have a project that may
be in the same position.  I used GOOPS initially, then rewrote it to use
either GOOPS, plain lists or records backends.

When I kind of decided to go for plain lists, I discovered people (eg,
guix) are using records; and meanwhile Andy has made GOOPS perform.  Now
I'm thinking: if you only use GOOPS as a friendlier way to create
records, that may be nice after all.  Just wondering.

Greetings,
Jan

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: anyone define port types?

[Panicz Maciej Godek]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

2016-03-30 13:18 GMT+02:00 Jan Nieuwenhuizen <janneke@gnu.org>:

> Panicz Maciej Godek writes:
>
> > I also used GOOPS, which I regret to this day, and so the
> > whole framework needs a serious rewrite
>
> What is it that you do not like about GOOPS?


Most specifically, I dislike its middle three letters. The problem with OOP
is that it requires to know exactly what ones want -- it is difficult to
change the design of your program after it's been written (and it is also
difficult to come up with a good design from the beginning), and -- since
it is based on state mutation - it makes it difficult to reason about your
program.

On the practical side, it was a bit counterintuitive that <uvec> and
<vector> were unrelated, and I think that there were some issues with
<pair> and <list> types.

[-- Attachment #2: Type: text/html, Size: 1202 bytes --]

Re: anyone define port types?

[Marko Rauhamaa]

Panicz Maciej Godek <godek.maciek@gmail.com>:

> 2016-03-30 13:18 GMT+02:00 Jan Nieuwenhuizen <janneke@gnu.org>:
>
>> Panicz Maciej Godek writes:
>>
>> > I also used GOOPS, which I regret to this day, and so the
>> > whole framework needs a serious rewrite
>>
>> What is it that you do not like about GOOPS?
>
> Most specifically, I dislike its middle three letters. The problem
> with OOP is that it requires to know exactly what ones want -- it is
> difficult to change the design of your program after it's been written
> (and it is also difficult to come up with a good design from the
> beginning), and -- since it is based on state mutation - it makes it
> difficult to reason about your program.
>
> On the practical side, it was a bit counterintuitive that <uvec> and
> <vector> were unrelated, and I think that there were some issues with
> <pair> and <list> types.

I like OOP, only I don't like GOOPS. Its classes and generic functions
seem so idiomatically out of place, unschemish, if you will.

This is how OOP ought to be done:

  <URL: https://www.gnu.org/software/guile/manual/html_node/OO-Closure.htm
  l#OO-Closure>

I have created a tiny Guile module ("simpleton") that generalizes the
principle. In particular,

 * You don't need classes for OOP. You only need objects.

 * Do tie methods to objects. Don't pretend methods are external to
   objects.

 * Don't expose the internal state of objects. Only interact with the
   object through methods.


Marko

Re: anyone define port types?

[Panicz Maciej Godek]

[-- Attachment #1: Type: text/plain, Size: 1600 bytes --]

2016-03-30 19:53 GMT+02:00 Marko Rauhamaa <marko@pacujo.net>:


> I like OOP, only I don't like GOOPS. Its classes and generic functions
> seem so idiomatically out of place, unschemish, if you will.
>
> This is how OOP ought to be done:
>
>   <URL: https://www.gnu.org/software/guile/manual/html_node/OO-Closure.htm
>   l#OO-Closure>
>
>
The problem with closures is, among others, that they are non-serializable
(I think that Termite solves some of those issues gracefully)

I have created a tiny Guile module ("simpleton") that generalizes the
> principle. In particular,
>
>  * You don't need classes for OOP. You only need objects.
>
>
JavaScript made a similar assumption, which -- I believe -- turned out
cumbersome, because I sometimes have to pretend that it has classes despite
that it doesn't.


>  * Do tie methods to objects. Don't pretend methods are external to
>    objects.

  * Don't expose the internal state of objects. Only interact with the
>    object through methods.


I think it is a good rule, but it's better if you can do without state.

I believe that methods *are* external to objects. Linguistically, you don't
"button.press()"; you "press(button)".

I also think that tying methods to objects is one of the problems of OOP,
because the designer of an object has to know in advance which actions on
an object are conceivable.

Perhaps certain devices, like tape recorders, are modelled well within OOP
-- you have a well-defined interface (like the play, rewind, fast-forward
button) and a hidden internal state.

But I'm afraid that this metaphor doesn't scale well.

[-- Attachment #2: Type: text/html, Size: 2875 bytes --]

Re: anyone define port types?

[Jan Wedekind]

On Wed, 30 Mar 2016, Marko Rauhamaa wrote:

> Panicz Maciej Godek <godek.maciek@gmail.com>:
>
>> 2016-03-30 13:18 GMT+02:00 Jan Nieuwenhuizen <janneke@gnu.org>:
>>
>>> Panicz Maciej Godek writes:
>>>
>>>> I also used GOOPS, which I regret to this day, and so the
>>>> whole framework needs a serious rewrite
>>>
>>> What is it that you do not like about GOOPS?
>>
>> Most specifically, I dislike its middle three letters. The problem
>> with OOP is that it requires to know exactly what ones want -- it is
>> difficult to change the design of your program after it's been written
>> (and it is also difficult to come up with a good design from the
>> beginning), and -- since it is based on state mutation - it makes it
>> difficult to reason about your program.
>>
>> On the practical side, it was a bit counterintuitive that <uvec> and
>> <vector> were unrelated, and I think that there were some issues with
>> <pair> and <list> types.
>
> I like OOP, only I don't like GOOPS. Its classes and generic functions
> seem so idiomatically out of place, unschemish, if you will.
>
> This is how OOP ought to be done:
>
>  <URL: https://www.gnu.org/software/guile/manual/html_node/OO-Closure.htm
>  l#OO-Closure>
>
> I have created a tiny Guile module ("simpleton") that generalizes the
> principle. In particular,
>
> * You don't need classes for OOP. You only need objects.
>
> * Do tie methods to objects. Don't pretend methods are external to
>   objects.
>
> * Don't expose the internal state of objects. Only interact with the
>   object through methods.
>
>
> Marko

GOOPS supports "open" classes and multiple-dispatch. E.g. you can extend 
the "write" method to control how an object is displayed within the Guile 
REPL [1]. Another interesting approach are multi-methods in Clojure which 
don't even require explicit types for dispatching.

[1] http://wedesoft.de/oop-with-goops.html

Re: anyone define port types?

[Marko Rauhamaa]

Panicz Maciej Godek <godek.maciek@gmail.com>:

> 2016-03-30 19:53 GMT+02:00 Marko Rauhamaa <marko@pacujo.net>:
> The problem with closures is, among others, that they are
> non-serializable

What is there to serialize in objects? How do you serialize a car? How
do you serialize an ant? How do you serialize a person?

All you can serialize is information. Objects are living things we
experience through interactions alone.

And yet, UNIX knows how to suspend a process. Virtualization
technologies are very good at snapshotting live virtual machines. Tricky
business but the results are quite impressive albeit not fully flawless.

>>  * You don't need classes for OOP. You only need objects.
>
> JavaScript made a similar assumption, which -- I believe -- turned out
> cumbersome, because I sometimes have to pretend that it has classes
> despite that it doesn't.

I can't speak for JavaScript.

Point is, you should only care about interfacing the objects, not about
their pedigree.

>>  * Don't expose the internal state of objects. Only interact with the
>>    object through methods.
>
> I think it is a good rule, but it's better if you can do without
> state.

Stateless objects are dead as stone. However you present it,
statefulness is the core of objects.

> I believe that methods *are* external to objects. Linguistically, you
> don't "button.press()"; you "press(button)".

I won't debate linguistics. I can only say the methods-of-objects is a
very convenient paradigm that helps me organize practical programs.

> I also think that tying methods to objects is one of the problems of
> OOP, because the designer of an object has to know in advance which
> actions on an object are conceivable.

You mean you can't associate new methods to an object. That's true and
can be annoying sometimes. However, GOOPS' cure is worse than the
disease: it exposes the slots of the object.


Marko

Re: anyone define port types?

[Marko Rauhamaa]

Jan Wedekind <jan@wedesoft.de>:

> GOOPS supports "open" classes and multiple-dispatch. E.g. you can extend
> the "write" method to control how an object is displayed within the
> Guile REPL [1]. Another interesting approach are multi-methods in
> Clojure which don't even require explicit types for dispatching.
>
> [1] http://wedesoft.de/oop-with-goops.html

Your example demonstrates my problem with GOOPS: you are redefining
(display) by penetrating the black box with (slot-ref). What would
happen to your method if I changed the implementation of <a> so it
no longer has the slot x, even virtually.

GOOPS' has the worst possible object model: objects are seen as mere
data records. The concept of a "slot" is an anathema to OOP.

Think of a UNIX file. You access it through open(2), write(2), read(2),
close(2) and so on. You don't know and you shouldn't care how the file
contents are organized on the disk (if there is a disk!). GOOPS would
like to abandon the system calls and just expose the bytes (slots) on
the disk. Then, GOOPS lets you define any kind of system calls you see
fit.


Marko

Re: anyone define port types?

[Jan Wedekind]

On Wed, 30 Mar 2016, Marko Rauhamaa wrote:

> Jan Wedekind <jan@wedesoft.de>:
>
>> GOOPS supports "open" classes and multiple-dispatch. E.g. you can extend
>> the "write" method to control how an object is displayed within the
>> Guile REPL [1]. Another interesting approach are multi-methods in
>> Clojure which don't even require explicit types for dispatching.
>>
>> [1] http://wedesoft.de/oop-with-goops.html
>
> Your example demonstrates my problem with GOOPS: you are redefining
> (display) by penetrating the black box with (slot-ref). What would
> happen to your method if I changed the implementation of <a> so it
> no longer has the slot x, even virtually.
>
> GOOPS' has the worst possible object model: objects are seen as mere
> data records. The concept of a "slot" is an anathema to OOP.
>
> Think of a UNIX file. You access it through open(2), write(2), read(2),
> close(2) and so on. You don't know and you shouldn't care how the file
> contents are organized on the disk (if there is a disk!). GOOPS would
> like to abandon the system calls and just expose the bytes (slots) on
> the disk. Then, GOOPS lets you define any kind of system calls you see
> fit.
>
>
> Marko
Ok, I have updated the example to use accessor functions instead of 
"slot-ref". But yes, a GOOPS generic only dispatches on types where 
instances in turn need to define slots (attributes). That's why I pointed 
out Clojure multimethods which uses a more flexible dispatch mechanism 
(using an inheritance graph and dispatching on the return value of an 
arbitrary function).

GOOPS (as well as Clojure multimethods) provide "open" classes and 
multiple-dispatch. Two things which I can really recommend to play with.

Re: anyone define port types?

[Marko Rauhamaa]

Jan Wedekind <jan@wedesoft.de>:

> On Wed, 30 Mar 2016, Marko Rauhamaa wrote:
>> GOOPS' has the worst possible object model: objects are seen as mere
>> data records. The concept of a "slot" is an anathema to OOP.
>
> Ok, I have updated the example to use accessor functions instead of
> "slot-ref".

(get-x) is only a fig leaf for (slot-ref). In general, no user of an <a>
object should think the object holds a piece of information called x.
Instead, you should be interacting with the abstract object <a>.

Python people call it duck-typing.

Java, Go et al use interfaces.

Even C can do opaque structs.

C++ suffers from "private" data members, and GOOPS strips away even that
thin veil.

Remember:

   [...] during her journey south to be married, the young queen-to-be
   passed through a town that was famed for its silk stockings, then
   rare and expensive items. Wishing to show her due courtesy, the
   merchants of the town offered to present her with a pair. [...] The
   Queen’s courtiers were aghast at this embarrassing breach of decorum,
   regarded as both indecent and audacious, and one replied loftily that
   “The Queen of Spain has no legs”.
   <URL: http://www.worldwidewords.org/qa/qa-que2.htm>


Marko

Re: anyone define port types?

[Barry Fishman]

On 2016-03-30 22:57:25 +0300, Marko Rauhamaa wrote:
> Panicz Maciej Godek <godek.maciek@gmail.com>:
>
>> 2016-03-30 19:53 GMT+02:00 Marko Rauhamaa <marko@pacujo.net>:
>> The problem with closures is, among others, that they are
>> non-serializable
>
> What is there to serialize in objects? How do you serialize a car? How
> do you serialize an ant? How do you serialize a person?
>
> All you can serialize is information. Objects are living things we
> experience through interactions alone.

Do we really want our computers to behave like organic black boxes?

I think is is more limiting to look at Objects as organic entities
rather than encapsulation of state.  If objects are state they can be
recorded.  When your bank account balance looks wrong its good to be
able to replay transactions to find out what you are missing.  When
transactions are happening across a network its nice to be able to
restart them rather deal with lost transactions or single points of
failure.

Yes, Objects can produce their own logs, but this becomes more code to
debug rather than a better debugging tool.  And the act of building such
an interface, really establishes them as state.

It is nice to think about the fact that a single implementation of that
state is not required by a design, or that it can be changed with little
impact on the rest of the system.  But such implementation flexibility
is not dependent on a "methods within object" type of object construction.
It is more a matter of what is exported at the module level.

In Guile you can export, at the module level, what interface you like
and hide any raw "get-x" style assessors.  True you can get at raw state
through the slot interface, but Guile is built around observably, not
security, and you could get at the contents of closures if you really
wanted to.  The fact that objects represent state doe not force the
exact implementation of that state be considered outside of a single
module.

> And yet, UNIX knows how to suspend a process. Virtualization
> technologies are very good at snapshotting live virtual machines. Tricky
> business but the results are quite impressive albeit not fully
> flawless.

And a crude way to try to deal with this.  Serialization does not have
to be lost.

I used to do large scale development in C++.  Books like "Design
Patterns" are really just series of hacks to get somewhat around the
fact that objects built around a fixed, rigidly defined set of methods
becomes difficult to adapt and maintain as projects grow over time.  I
found that Common Lisp's CLOS style way of looking at objects would have
made thing far easier to develop and maintain and produce a more stable
code base.  I really got to dislike C++'s and Java's object model.

With distributed environments and the more functional way of looking at
things that are possible in Scheme, I think we would better off allowing
for extending objects by treating them as immutable state and using
monads to perform actions, rather than just allowing for mutable objects
and methods.  Monads don't constrain object state implementation either.

I don't mean this to be construed as trying to make Guile a purely
functional language.  I like choices.

>> I also think that tying methods to objects is one of the problems of
>> OOP, because the designer of an object has to know in advance which
>> actions on an object are conceivable.
>
> You mean you can't associate new methods to an object. That's true and
> can be annoying sometimes. However, GOOPS' cure is worse than the
> disease: it exposes the slots of the object.

What is exposed or not exposed seems to be something better handled by
what modules export, and not be intrinsic to objects themselves.  This
allows methods, defined outside the module, that uses just the module
interface to the object (or several objects) to be easily defined and
somewhat separately manged.  This becomes crucial as systems grow and
change.  Especially allowing for the idea of multi-methods.

One of the security issues with Apple's Objective C is that object
methods are always available, if you know about them.  Closure based
interfaces do make hiding of state easier, but I think localizing all
method code in one place makes complex large systems more difficult to
maintain.  Of course at the true security level, both can be hacked.  We
are dealing more with what developers might presume about an interface
rather a malicious act.

Although Scheme itself is not the usual place to think about complex
systems, I think Guile does, given what is currently being implemented.

--
Barry Fishman

Re: anyone define port types?

[Marko Rauhamaa]

Barry Fishman <barry@ecubist.org>:

> On 2016-03-30 22:57:25 +0300, Marko Rauhamaa wrote:
>> All you can serialize is information. Objects are living things we
>> experience through interactions alone.
>
> Do we really want our computers to behave like organic black boxes?

I'll say yes.

> In Guile you can export, at the module level, what interface you like
> and hide any raw "get-x" style assessors.

You mean GOOPS. Guile can do a lot better.

> True you can get at raw state through the slot interface, but Guile is
> built around observably, not security, and you could get at the
> contents of closures if you really wanted to.

I don't care about security. I care about the approach. To me, objects
are thingies that interact with their surroundings. GOOPS' model is the
opposite: objects are conceptually mere dead records, structs.

> The fact that objects represent state doe not force the
> exact implementation of that state be considered outside of a single
> module.

The accessor (get-x) evokes the idea that there's a piece of information
guarded by the object that the object is willing to divulge through the
accessor, however implicitly that piece of information is encoded
internally.

I don't usually think of objects as containing information. I don't like
Java's and Python's properties because they also make objects seem like
records.

> Serialization does not have to be lost.

You serialize data, you don't serialize objects. You can serialize
configuration information. You can serialize messages. You can serialize
collected data. But objects exist, live, breathe and interact.

> I used to do large scale development in C++. Books like "Design
> Patterns" are really just series of hacks to get somewhat around the
> fact that objects built around a fixed, rigidly defined set of methods
> becomes difficult to adapt and maintain as projects grow over time.

I have some development on my belt as well. I don't know what trauma
design patterns might have inflicted on you on your career.

> I found that Common Lisp's CLOS style way of looking at objects would
> have made thing far easier to develop and maintain and produce a more
> stable code base. I really got to dislike C++'s and Java's object
> model.

So you haven't personally experienced the blessings of CLOS?

> I don't mean this to be construed as trying to make Guile a purely
> functional language. I like choices.

GOOPS is the antithesis of functional programming, I think.

>> You mean you can't associate new methods to an object. That's true
>> and can be annoying sometimes. However, GOOPS' cure is worse than the
>> disease: it exposes the slots of the object.
>
> What is exposed or not exposed seems to be something better handled by
> what modules export, and not be intrinsic to objects themselves. This
> allows methods, defined outside the module, that uses just the module
> interface to the object (or several objects) to be easily defined and
> somewhat separately manged. This becomes crucial as systems grow and
> change. Especially allowing for the idea of multi-methods.

External, derived functionality is possible at that level with the
classic rigid methods as well. If the external methods depend on the
methods the module chooses to expose, you are back with the classic
model you are trying to avoid.


Marko

Re: anyone define port types?

[Jan Wedekind]

On Thu, 31 Mar 2016, Marko Rauhamaa wrote:

> Jan Wedekind <jan@wedesoft.de>:
>
>> On Wed, 30 Mar 2016, Marko Rauhamaa wrote:
>>> GOOPS' has the worst possible object model: objects are seen as mere
>>> data records. The concept of a "slot" is an anathema to OOP.
>>
>> Ok, I have updated the example to use accessor functions instead of
>> "slot-ref".
>
> (get-x) is only a fig leaf for (slot-ref). In general, no user of an <a>
> object should think the object holds a piece of information called x.
> Instead, you should be interacting with the abstract object <a>.
Well, actually (get-x) is a generic as well. I.e. it is polymorphic and 
does not have to be a simple accessor for a slot. I.e. the accessor syntax 
is just a shortcut for declaring a generic for accessing a slot 
separately:

(use-modules (oop goops))
(define-class <a> () (x #:init-keyword #:x))
(define-method (get-x (self <a>)) (slot-ref self 'x))

"get-x" can be defined differently for each class and its implementation 
can be doing something different than accessing a slot.

>
> Python people call it duck-typing.
>
> Java, Go et al use interfaces.
>
> Even C can do opaque structs.
>
> C++ suffers from "private" data members, and GOOPS strips away even that
> thin veil.

Here is an example using duck-typing with the generic "get-x":

(define-class <b> ())
(define-method (get-x (self <b>)) 123)
(define-method (use-duck-typing (x <object>)) (get-x x))
(use-duck-typing (make <a> #:x 3))
; => 3
(use-duck-typing (make <b>))
; => 123

Re: anyone define port types?

[Marko Rauhamaa]

Jan Wedekind <jan@wedesoft.de>:

> On Thu, 31 Mar 2016, Marko Rauhamaa wrote:
>> (get-x) is only a fig leaf for (slot-ref). In general, no user of an
>> <a> object should think the object holds a piece of information
>> called x. Instead, you should be interacting with the abstract object
>> <a>.
>
> Well, actually (get-x) is a generic as well. I.e. it is polymorphic
> and does not have to be a simple accessor for a slot.

Naturally, that's not my point.

In practice, GOOPS can be used just like any classic object system.
However,

 * The (make)/(initialize) mechanism is strongly tied to the slots.
   There is a strong temptation to define a nongeneric constructor
   function for each class that internally calls (make) with appropriate
   slot settings. This is suggested at the bottom of <URL: https://www.gn
   u.org/software/guile/manual/html_node/Slot-Description-Example.html#Sl
   ot-Description-Example>.

   Unfortunately, the separate constructor can't invoke a base class's
   constructor function but must contend with (make), which exposes the
   base class's slots to the derived class.

   Ideally, you should be able to interpret the keyword args to
   (make)/(initialize) independently of the slots. However, the
   (next-method) mechanism makes this tricky.

 * The (self <class>) notation makes code look very un-Schemey and
   noisy. For example, classic port dispatching is handled with the
   assumption that the port knows its methods.

 * The slot emphasis permeates the documentation and available
   facilities. The long <my-complex> is advertised as "a better
   solution" even though it is extremely noisy and complex numbers are
   extremely passive as objects (<URL: https://www.gnu.org/software/gui
   le/manual/html_node/Slot-Description-Example.html#Slot-Description-E
   xample>).

   What on earth are (shallow-clone) and (deep-clone) supposed to
   accomplish? Duplicate the database? Open another session to the
   server?


Marko

Re: anyone define port types?

[Ludovic Courtès]

Hello!

Here’s a port type!  :-)

  https://gitlab.com/gnutls/gnutls/blob/master/guile/src/core.c#L785

Ludo’.

Re: anyone define port types?

[Mark H Weaver]

Matt Wette <matthew.wette@verizon.net> writes:

>> On Mar 28, 2016, at 12:04 PM, Andy Wingo <wingo@pobox.com> wrote:
>> I am working on improving our port implementation to take advantage of
>> the opportunity to break ABI in 2.2.  I am wondering how much I can
>> break C API as well -- there are some changes that would allow better
>> user-space threading
>
> I made an attempt to use soft ports, but backed out after I could not
> figure a way to extra data I wanted to associate with my port.
>
> Addition of an access function to user-defined data would enlarge the
> cases for which soft ports could be used.

Object properties can be used to associate extra information with any
object that has identity, which includes ports.  See
'make-object-property' in the Guile manual.

      Mark

Re: anyone define port types?

[Matt Wette]

> On Apr 5, 2016, at 7:06 AM, Mark H Weaver <mhw@netris.org> wrote:
> 
> Matt Wette <matthew.wette@verizon.net> writes:
> 
>>> On Mar 28, 2016, at 12:04 PM, Andy Wingo <wingo@pobox.com> wrote:
>>> I am working on improving our port implementation to take advantage of
>>> the opportunity to break ABI in 2.2.  I am wondering how much I can
>>> break C API as well -- there are some changes that would allow better
>>> user-space threading
>> 
>> I made an attempt to use soft ports, but backed out after I could not
>> figure a way to extra data I wanted to associate with my port.
>> 
>> Addition of an access function to user-defined data would enlarge the
>> cases for which soft ports could be used.
> 
> Object properties can be used to associate extra information with any
> object that has identity, which includes ports.  See
> 'make-object-property' in the Guile manual.

Thanks.  I had forgotten about that. — Matt

Re: anyone define port types?

[Ludovic Courtès]

Hey!

Andy Wingo <wingo@pobox.com> skribis:

> I am working on improving our port implementation to take advantage of
> the opportunity to break ABI in 2.2.  I am wondering how much I can
> break C API as well -- there are some changes that would allow better
> user-space threading
> (e.g. http://thread.gmane.org/gmane.lisp.guile.devel/14158/focus=15463
> or Chris Webber's 8sync).  But those changes would require some
> incompatible changes to the C API related to port internals.  This API
> is pretty much only used when implementing port types.  So, I would like
> to collect a list of people that implement their own ports :) I know
> Guile-GNOME does it for GNOME-VFS, though GNOME-VFS is super-old at this
> point...  Anyway.  Looking forward to your links :)

What do you conclude from this poll?  :-)

From what you’ve seen, how do you think current uses would impact the
refactoring work (and vice versa)?

Thanks,
Ludo’.

Re: anyone define port types?

[Andy Wingo]

On Wed 06 Apr 2016 01:55, Matt Wette <matthew.wette@verizon.net> writes:

>> On Apr 5, 2016, at 7:06 AM, Mark H Weaver <mhw@netris.org> wrote:
>> 
>> Matt Wette <matthew.wette@verizon.net> writes:
>> 
>>>> On Mar 28, 2016, at 12:04 PM, Andy Wingo <wingo@pobox.com> wrote:
>>>> I am working on improving our port implementation to take advantage of
>>>> the opportunity to break ABI in 2.2.  I am wondering how much I can
>>>> break C API as well -- there are some changes that would allow better
>>>> user-space threading
>>> 
>>> I made an attempt to use soft ports, but backed out after I could not
>>> figure a way to extra data I wanted to associate with my port.
>>> 
>>> Addition of an access function to user-defined data would enlarge the
>>> cases for which soft ports could be used.
>> 
>> Object properties can be used to associate extra information with any
>> object that has identity, which includes ports.  See
>> 'make-object-property' in the Guile manual.
>
> Thanks.  I had forgotten about that. — Matt

I think for soft ports (and custom binary i/o ports) the idea is
actually that the "handler" procedures are actually closures.  I.e.

  (let ((my-obj ...))
    (make-soft-port (vector (lambda (c) (do-write my-obj c)) ...) modes))

No need for extra arguments if you have closures.

Andy

Re: anyone define port types?

[Andy Wingo]

On Wed 30 Mar 2016 08:29, Panicz Maciej Godek <godek.maciek@gmail.com> writes:

> Hi Andy,
> I have been using soft ports to implement a text widget in my GUI
> framework. I also used GOOPS, which I regret to this day, and so the
> whole framework needs a serious rewrite, but if you're collecting
> various species to the museum of make-soft-port, you can have a look:
>
> https://bitbucket.org/panicz/slayer/src/8b741c21b7372c24874d7cd1875e2aae3fc43abe/guile-modules/widgets/text-area.scm?at=default&fileviewer=file-view-default

Thanks.  The interface to soft ports hasn't changed FWIW; nothing has
changed from a Scheme programmer's perspective, except that all ports
can have buffering now.  See:

  http://www.gnu.org/software/guile/docs/master/guile.html/Buffering.html

Cheers,

Andy

Re: anyone define port types?

[Andy Wingo]

On Fri 01 Apr 2016 16:38, ludo@gnu.org (Ludovic Courtès) writes:

> Here’s a port type!  :-)
>
>   https://gitlab.com/gnutls/gnutls/blob/master/guile/src/core.c#L785

Thanks!

So port types written in C will have to change unfortunately :/  From
NEWS:

    ** Complete overhaul of port internals

    Guile's ports have been completely overhauled to allow Guile developers
    and eventually Guile users to write low-level input and output routines
    in Scheme.  The new internals will eventually allow for user-space
    tasklets or green threads that suspend to a scheduler when they would
    cause blocking I/O, allowing users to write straightforward network
    services that parse their input and send their output as if it were
    blocking, while under the hood Guile can multiplex many active
    connections at once.

    At the same time, this change makes Guile's ports implementation much
    more maintainable, rationalizing the many legacy port internals and
    making sure that the abstractions between the user, Guile's core ports
    facility, and the port implementations result in a system that is as
    performant and expressive as possible.

    The interface to the user has no significant change, neither on the C
    side nor on the Scheme side.  However this refactoring has changed the
    interface to the port implementor in an incompatible way.  See below for
    full details.

And later:

    ** API to define new port types from C has changed

    See the newly expanded "I/O Extensions" in the manual, for full details.
    Notably:

    *** Remove `scm_set_port_mark'

    Port mark functions have not been called since the switch to the BDW
    garbage collector.

    *** Remove `scm_set_port_equalp'

    Likewise port equal functions weren't being called.  Given that ports
    have their own internal buffers, it doesn't make sense to hook them into
    equal? anyway.

    *** Remove `scm_set_port_free'

    It used to be that if an open port became unreachable, a special "free"
    function would be called instead of the "close" function.  Now that the
    BDW-GC collector allows us to run arbitrary code in finalizers, we can
    simplify to just call "close" on the port and remove the separate free
    functions.  Note that hooking into the garbage collector has some
    overhead.  For that reason Guile exposes a new interface,
    `scm_set_port_needs_close_on_gc', allowing port implementations to
    indicate to Guile whether they need closing on GC or not.

    *** Remove `scm_set_port_end_input', `scm_set_port_flush'

    As buffering is handled by Guile itself, these functions which were to
    manage an implementation-side buffer are no longer needed.

    *** Change prototype of `scm_make_port_type'

    The `read' (renamed from `fill_input') and `write' functions now operate
    on bytevectors.  Also the `mode_bits' argument now inplicitly includes
    SCM_OPN, so you don't need to include these.

    *** Change prototype of port `close' function

    The port close function now returns void.

    *** Port and port type data structures are now opaque

    Port type implementations should now use API to access port state.
    However, since the change to handle port buffering centrally, port type
    implementations rarely need to access unrelated port state.

    *** Port types are now `scm_t_port_type*', not a tc16 value

    `scm_make_port_type' now returns an opaque pointer, not a tc16.
    Relatedly, the limitation that there only be 256 port types has been
    lifted.

Now.  How to adapt?  You can do the #if thing with versions.  Or, maybe
we should flesh out the custom binary input/output port support so that
you can implement a GnuTLS port in Scheme.  WDYT?

Andy

Re: anyone define port types?

[Andy Wingo]

On Thu 14 Apr 2016 16:08, ludo@gnu.org (Ludovic Courtès) writes:

> Andy Wingo <wingo@pobox.com> skribis:
>
>> I am working on improving our port implementation to take advantage of
>> the opportunity to break ABI in 2.2.  I am wondering how much I can
>> break C API as well -- there are some changes that would allow better
>> user-space threading
>> (e.g. http://thread.gmane.org/gmane.lisp.guile.devel/14158/focus=15463
>> or Chris Webber's 8sync).  But those changes would require some
>> incompatible changes to the C API related to port internals.  This API
>> is pretty much only used when implementing port types.  So, I would like
>> to collect a list of people that implement their own ports :) I know
>> Guile-GNOME does it for GNOME-VFS, though GNOME-VFS is super-old at this
>> point...  Anyway.  Looking forward to your links :)
>
> What do you conclude from this poll?  :-)
>
> From what you’ve seen, how do you think current uses would impact the
> refactoring work (and vice versa)?

Sorry for the late response :)

My conclusion is that we should not change anything about the Scheme
interface, but that with close communication with C port hackers, we can
feel OK about changing the C interface to make it both more simple and
more expressive.  Since libguile is parallel-installable and you have to
select the version of your Guile when you build your project, of course
people will be able to update / upgrade when they choose to.

I put in a lot of effort to the documentation; check it out:

  http://www.gnu.org/software/guile/docs/master/guile.html/Input-and-Output.html

Andy

Re: anyone define port types?

[Chris Vine]

On Sat, 11 Jun 2016 19:02:09 +0200
Andy Wingo <wingo@pobox.com> wrote:
> On Thu 14 Apr 2016 16:08, ludo@gnu.org (Ludovic Courtès) writes:
> 
> > Andy Wingo <wingo@pobox.com> skribis:
> >  
> >> I am working on improving our port implementation to take
> >> advantage of the opportunity to break ABI in 2.2.  I am wondering
> >> how much I can break C API as well -- there are some changes that
> >> would allow better user-space threading
> >> (e.g.
> >> http://thread.gmane.org/gmane.lisp.guile.devel/14158/focus=15463
> >> or Chris Webber's 8sync).  But those changes would require some
> >> incompatible changes to the C API related to port internals.  This
> >> API is pretty much only used when implementing port types.  So, I
> >> would like to collect a list of people that implement their own
> >> ports :) I know Guile-GNOME does it for GNOME-VFS, though
> >> GNOME-VFS is super-old at this point...  Anyway.  Looking forward
> >> to your links :)  
> >
> > What do you conclude from this poll?  :-)
> >
> > From what you’ve seen, how do you think current uses would impact
> > the refactoring work (and vice versa)?  
> 
> Sorry for the late response :)
> 
> My conclusion is that we should not change anything about the Scheme
> interface, but that with close communication with C port hackers, we
> can feel OK about changing the C interface to make it both more
> simple and more expressive.  Since libguile is parallel-installable
> and you have to select the version of your Guile when you build your
> project, of course people will be able to update / upgrade when they
> choose to.
> 
> I put in a lot of effort to the documentation; check it out:
> 
>   http://www.gnu.org/software/guile/docs/master/guile.html/Input-and-Output.html

The documentation indicates that with the C ports implementation in
guile-2.2, reads will block on non-blocking file descriptors.

This will stop the approach to asynchronicity used in 8sync and
guile-a-sync (the latter of which I have written) from working
correctly with sockets on linux operating systems, because at present
both of these use guile's wrapper for select.  This arises because with
linux you can get spurious wake-ups with select() or poll() on sockets,
whereby a read on a file descriptor for a socket can block even though
the descriptor is reported by select() or poll() as ready for
reading[1].

Presumably suspendable ports will use some method to work around
unwanted blocking arising from spurious select()/poll() wake-ups.
However, to cater for other asynchronous implementations of file
watches, would it be possible to provide a configurable option either
to retain the guile-2.0 behaviour in such cases (which is to throw a
system-error with errno set to EAGAIN or EWOULDBLOCK), or to provide a
non-blocking alternative whereby the read operation would, instead of
blocking, return some special value such as an EAGAIN symbol?  Either
would enable user code then to resume to its prompt and let other code
execute.

Chris

[1] The man page for select states "Under Linux, select() may report a
socket file descriptor as 'ready for reading', while nevertheless a
subsequent read blocks.  This could for example happen when data has
arrived but upon examination has wrong checksum and is discarded.  There
may be other circumstances in which a file descriptor is spuriously
reported as ready.  Thus it may be safer to use O_NONBLOCK on sockets
that should not block."

Re: anyone define port types?

[Andy Wingo]

Hi :)

On Sun 12 Jun 2016 10:25, Chris Vine <chris@cvine.freeserve.co.uk> writes:

>>   http://www.gnu.org/software/guile/docs/master/guile.html/Input-and-Output.html
>
> The documentation indicates that with the C ports implementation in
> guile-2.2, reads will block on non-blocking file descriptors.

Correct.

> This will stop the approach to asynchronicity used in 8sync and
> guile-a-sync (the latter of which I have written) from working
> correctly with sockets on linux operating systems, because at present
> both of these use guile's wrapper for select.

The trouble is that AFAIU there is no way to make non-blocking input
work reliably with O_NONBLOCK file descriptors in the approach that
Guile has always used.

As you know, the current behavior for Guile 2.0 is to throw an exception
when you get EAGAIN / EWOULDBLOCK.  If I am understanding you correctly,
your approach is to only read from a port if you have done a select() /
poll() / etc on it beforehand indicating that you can read at least one
byte.

The problem with this is not only spurious wakeups, as you note, but
also buffering.  Throwing an exception when reading in Guile 2.0 will
discard input buffers in many cases.  Likewise when writing, you won't
be able to know how much you've written.

This goes not only for the explicit bufffers attached to ports and which
you can control with `setvbuf', but also implicit buffers, and it's in
this case that it's particularly pernicious: if you `read-char' on a
UTF-8 port, you might end up using local variables in the stack as a
buffer for reconstructing that codepoint.  If you throw an exception in
the middle, you discard those bytes.  Likewise for writing.

For suspendable ports, you don't throw an exception: you just assume the
operation is going to work, but if you get EAGAIN / EWOULDBLOCK, you
call the current-read-waiter / current-write-waiter and when that
returns retry the operation.  Since it operates on the lowest level of
bytes, it's reliable.  Looping handles the spurious wakeup case.

> However, to cater for other asynchronous implementations of file
> watches, would it be possible to provide a configurable option either
> to retain the guile-2.0 behaviour in such cases (which is to throw a
> system-error with errno set to EAGAIN or EWOULDBLOCK), or to provide a
> non-blocking alternative whereby the read operation would, instead of
> blocking, return some special value such as an EAGAIN symbol?  Either
> would enable user code then to resume to its prompt and let other code
> execute.

Why not just (install-suspendable-ports!) and

  (parameterize ((current-read-waiter my-read-waiter)) ...)

etc?  It is entirely possible with Guile 2.1.3 to build an asynchronous
coroutine-style concurrent system in user-space using these primitives.
See the wip-ethread branch for an example implementation.

Andy

Re: anyone define port types?

[Marko Rauhamaa]

Andy Wingo <wingo@pobox.com>:

> The trouble is that AFAIU there is no way to make non-blocking input
> work reliably with O_NONBLOCK file descriptors in the approach that
> Guile has always used.
>
> [...]
>
> The problem with this is not only spurious wakeups, as you note, but
> also buffering. Throwing an exception when reading in Guile 2.0 will
> discard input buffers in many cases. Likewise when writing, you won't
> be able to know how much you've written.

Guile's POSIX support provides immediate access to most OS facilities.
However, support for read(2) and write(2) seem to be missing. (Note that
Python, for example, provides both buffered, native facilities and the
low-level os.read() and os.write().)

Sure, recv! and send are there, but asynchronous I/O is more general
than socket communication.

> For suspendable ports, you don't throw an exception: you just assume
> the operation is going to work, but if you get EAGAIN / EWOULDBLOCK,
> you call the current-read-waiter / current-write-waiter and when that
> returns retry the operation. Since it operates on the lowest level of
> bytes, it's reliable. Looping handles the spurious wakeup case.

The POSIX system call simply returns whatever bytes are available at the
moment. That paradigm works perfectly. IOW, if you have buffered 10
bytes and get an EGAIN, just return those 10 bytes.

> Why not just (install-suspendable-ports!) and
>
>   (parameterize ((current-read-waiter my-read-waiter)) ...)
>
> etc? It is entirely possible with Guile 2.1.3 to build an asynchronous
> coroutine-style concurrent system in user-space using these
> primitives. See the wip-ethread branch for an example implementation.

I haven't thought it through if that callback paradigm would offer the
same degrees of freedom as the return-whatever-you-have paradigm. It
seems unnecessarily complicated, though.

I would simply violate the letter of the R?RS port semantics in Guile
and go with the POSIX semantics. It seems clear the R?RS port
specification was written without considering asynchronous communication
at all.


Marko

Re: anyone define port types?

[Andy Wingo]

Hi,

You are arguing for new I/O primitives with different semantics, and
that's fine and good :)  My goal was to add the ability to sensibly work
with non-blocking ports using Guile's existing primitives, especially
the textual ones.

This effort was started because of the ethreads work I did about four
years ago.  That required a separate implementation of buffered ports,
which is fine for a library but not good for a system.  You want to be
able to set a non-blocking port as the current input port, for example,
and you want to be able to (display x p) where p is a non-blocking port,
and you want to be able to do this both in a blocking way, because
Guile doesn't currently include any kind of user-space scheduler, but
also you want to support experiments to make a user-space scheduler.

The ethreads work from four years ago also required me to patch various
parts of Guile, like the web server, so that they could use different
primitives.  That's what I want to avoid.  I want to be able to run an
SSAX parser or a Scheme reader or writer over a non-blocking port and
have it just work.  (For SSAX it's close to working; there are a couple
of primitives that need to be reimplemented in terms of the subset of
port interfaces that suspendable ports replaces.  For Scheme
`read'/`write' the situation is more complicated.)

On Sun 19 Jun 2016 11:55, Marko Rauhamaa <marko@pacujo.net> writes:

> Guile's POSIX support provides immediate access to most OS facilities.
> However, support for read(2) and write(2) seem to be missing. (Note that
> Python, for example, provides both buffered, native facilities and the
> low-level os.read() and os.write().)

A read operation is supported via `get-bytevector-some', but it's true
that `put-bytevector-some' is not implemented.

Incidentally, have you seen ljsyscall, for LuaJIT?  What a great
library.  I wish we had something like that for Guile.

> I haven't thought it through if that callback paradigm would offer the
> same degrees of freedom as the return-whatever-you-have paradigm. It
> seems unnecessarily complicated, though.

There is complication, but it is necessary complication.  If you want to
support blocking textual operations with all of the encodings that Guile
supports over non-blocking ports, you need to be able to poll() when you
get EAGAIN; and if you also want to be able to suspend in the middle of
writing a byte sequence, many levels removed from the original write,
then you need a parameter; and if you don't know yet the entire set of
things you might want to do, that parameter better be a procedure.

In practice I find it a delight to work with this interface.  See the
memcached server and client implementations in the recently updated
wip-ethreads branch; to my eye they are lovely.

Andy

Re: anyone define port types?

[Chris Vine]

On Sun, 19 Jun 2016 11:13:17 +0200
Andy Wingo <wingo@pobox.com> wrote:
> Hi :)
> 
> On Sun 12 Jun 2016 10:25, Chris Vine <chris@cvine.freeserve.co.uk>
> writes:
> 
> >>   http://www.gnu.org/software/guile/docs/master/guile.html/Input-and-Output.html  
> >
> > The documentation indicates that with the C ports implementation in
> > guile-2.2, reads will block on non-blocking file descriptors.  
> 
> Correct.
> 
> > This will stop the approach to asynchronicity used in 8sync and
> > guile-a-sync (the latter of which I have written) from working
> > correctly with sockets on linux operating systems, because at
> > present both of these use guile's wrapper for select.  
> 
> The trouble is that AFAIU there is no way to make non-blocking input
> work reliably with O_NONBLOCK file descriptors in the approach that
> Guile has always used.
> 
> As you know, the current behavior for Guile 2.0 is to throw an
> exception when you get EAGAIN / EWOULDBLOCK.  If I am understanding
> you correctly, your approach is to only read from a port if you have
> done a select() / poll() / etc on it beforehand indicating that you
> can read at least one byte.

My approach would be to to do that, if it worked.  And it does work with
pipes and unix domain sockets provided you only read a byte at a time
(see further below), but does not work on linux with TCP sockets because
linux's select() and poll() system calls are not POSIX compliant.

Therefore with TCP sockets on linux you have to use non-blocking reads
and cater for the possibility of an EAGAIN/EWOULDBLOCK exception.

> The problem with this is not only spurious wakeups, as you note, but
> also buffering.  Throwing an exception when reading in Guile 2.0 will
> discard input buffers in many cases.  Likewise when writing, you won't
> be able to know how much you've written.
> 
> This goes not only for the explicit bufffers attached to ports and
> which you can control with `setvbuf', but also implicit buffers, and
> it's in this case that it's particularly pernicious: if you
> `read-char' on a UTF-8 port, you might end up using local variables
> in the stack as a buffer for reconstructing that codepoint.  If you
> throw an exception in the middle, you discard those bytes.  Likewise
> for writing.

I recognise this problem.  The answer I have adopted when reading from
TCP sockets is to extract individual bytes only from the port into a
bytevector using R6RS's get-u8 procedure and (if the port is textual
rather than binary) to reconstruct characters from that using
bytevector->string at, say, a line end[1].  An EAGAIN/EWOULDBLOCK
exception is then just treated as an invitation to return to the
prompt, and read state is retained in the bytevector.

In effect this is doing by hand what a more complete non-blocking
EAGAIN-safe port implementation might otherwise do for you.

Writing is something else.  To do it effectively the writer to the port
must in any event cater for the fact that when the buffer is full but
the underlying file descriptor is ready for writing, the next write will
cause a buffer flush, and if the size of the buffer is greater than the
number of characters that the file can receive without blocking,
blocking might still occur.  You usually need to switch off buffering
for writes (but you quite often may want to do that anyway on output
ports for sockets).

> For suspendable ports, you don't throw an exception: you just assume
> the operation is going to work, but if you get EAGAIN / EWOULDBLOCK,
> you call the current-read-waiter / current-write-waiter and when that
> returns retry the operation.  Since it operates on the lowest level of
> bytes, it's reliable.  Looping handles the spurious wakeup case.
> 
> > However, to cater for other asynchronous implementations of file
> > watches, would it be possible to provide a configurable option
> > either to retain the guile-2.0 behaviour in such cases (which is to
> > throw a system-error with errno set to EAGAIN or EWOULDBLOCK), or
> > to provide a non-blocking alternative whereby the read operation
> > would, instead of blocking, return some special value such as an
> > EAGAIN symbol?  Either would enable user code then to resume to its
> > prompt and let other code execute.  
> 
> Why not just (install-suspendable-ports!) and
> 
>   (parameterize ((current-read-waiter my-read-waiter)) ...)
> 
> etc?  It is entirely possible with Guile 2.1.3 to build an
> asynchronous coroutine-style concurrent system in user-space using
> these primitives. See the wip-ethread branch for an example
> implementation.

I would want to continue using an external event loop implemented with
poll() or select() and delimited continuations.  This  makes it
relatively trivial to implement asynchronous timeouts and
single-threaded task multiplexing (albeit co-operative rather than
pre-emptive) as well as file operations, and would also enable the glib
event loop to be used for programs which happen to use guile-gnome
(although guile-gnome has other issues at present).

I don't think I have got to grips with how to do that with read-waiter,
because the read-waiter comprises in effect another loop (in which the
main event loop with its own prompts would have to run) until the read
request has been satisfied.  I would need to think about it.  Since
ethreads use a poll()/epoll() loop, presumably you think it is
straightforward enough to integrate the two, even if at present I don't.

Writing custom C ports which do what I want with non-blocking
descriptors is another option, but I would hope that could be avoided:
at that point one would probably use some other solution entirely.

On a side issue, I am still trying to understand the point of causing
guile-2.2's read of a non-blocking C port to block.  The whole point of
making a descriptor non-blocking is that that shouldn't happen, and
there are circumstances where pealing individual bytes off a
non-blocking port as they become available is what you want to do.  It
makes guile's select wrapper unusable with TCP sockets on linux.  I
understand that suspendable-ports work differently, but that is another
matter.

Chris

[1]   A sample implementation can be seen at
https://github.com/ChrisVine/guile-a-sync/blob/master/a-sync/event-loop.scm
from line 900 on, and elsewhere in that file.

Re: anyone define port types?

[Andy Wingo]

On Sun 19 Jun 2016 17:33, Chris Vine <chris@cvine.freeserve.co.uk> writes:

> The answer I have adopted when reading from TCP sockets is to extract
> individual bytes only from the port into a bytevector using R6RS's
> get-u8 procedure and (if the port is textual rather than binary) to
> reconstruct characters from that using bytevector->string at, say, a
> line end[1].  An EAGAIN/EWOULDBLOCK exception is then just treated as
> an invitation to return to the prompt, and read state is retained in
> the bytevector.

Yep, makes sense, though it will be really slow of course.  And, you'd
have to make people rewrite their code to use your I/O primitives
instead of using read-line from (ice-9 rdelim); a bit of a drag, but
OK.

>> Why not just (install-suspendable-ports!) and
>> 
>>   (parameterize ((current-read-waiter my-read-waiter)) ...)
>> 
>> etc?  It is entirely possible with Guile 2.1.3 to build an
>> asynchronous coroutine-style concurrent system in user-space using
>> these primitives. See the wip-ethread branch for an example
>> implementation.
>
> I would want to continue using an external event loop implemented with
> poll() or select() and delimited continuations.

Yes yes a million times yes!  That's what I want too.  To that end, the
current-read-waiter parameter's value is a function which takes the port
being waited on, and when it returns the operation will retry.  In
wip-ethreads I suspend to a prompt, add the port's fd to an epoll set,
and when the fd becomes readable again we resume the continuation,
resulting in a return from the current-read-waiter.

> I don't think I have got to grips with how to do that with read-waiter,
> because the read-waiter comprises in effect another loop (in which the
> main event loop with its own prompts would have to run) until the read
> request has been satisfied.  I would need to think about it.  Since
> ethreads use a poll()/epoll() loop, presumably you think it is
> straightforward enough to integrate the two, even if at present I don't.

Here is the code.  First, a helper:

  ;; The AFTER-SUSPEND thunk allows the user to suspend the current
  ;; thread, saving its state, and then perform some other nonlocal
  ;; control flow.
  ;;
  (define* (suspend #:optional (after-suspend (lambda (ctx thread) #f)))
    ((abort-to-prompt (econtext-prompt-tag (current-econtext))
                      after-suspend)))

Assume there is some current-econtext parameter with a "context" which
holds the prompt tag associated with the scheduler.  As you see when the
continuation resumes, it resumes by calling a thunk, allowing exceptions
to be thrown from the context of the suspend.

Here's the main loop function, which you could replace with the GLib
main loop or whatever:

  (define* (run #:optional (ctx (ensure-current-econtext))
                #:key (install-suspendable-ports? #t))
    (when install-suspendable-ports? (install-suspendable-ports!))
    (parameterize ((current-econtext ctx)
                   (current-read-waiter wait-for-readable)
                   (current-write-waiter wait-for-writable))
      (let lp ()
        (run-ethread ctx (next-thread ctx))
        (lp))))

Cool.  Now the wait functions:

  (define (wait-for-readable port)
    (wait-for-events port (port-read-wait-fd port) (logior EPOLLIN EPOLLRDHUP)))

  (define (wait-for-writable port)
    (wait-for-events port (port-write-wait-fd port) EPOLLOUT))

Now the wait-for-events function:

  (define (wait-for-events port fd events)
    (handle-events
     port
     events
     (suspend
      (lambda (ctx thread)
        ...))))

Well that's a mess, but the thing to know is that the `suspend' will
abort to the relevant prompt, and then invoke the thunk that's its
argument.  Here's `handle-events' and we are done:

  (define (handle-events port events revents)
    (unless (zero? (logand revents EPOLLERR))
      (error "error reading from port" port)))

But I guess that error could be "reading or writing"; oh well.

See
http://git.savannah.gnu.org/cgit/guile.git/tree/module/ice-9/ethreads.scm?h=wip-ethreads&id=253dc1a7114b89351a3aa330caf173b98c5a65dd,
it's not long but it takes some time to read.  I think I can fairly ask
that of you though, given your interest in this area :)

> On a side issue, I am still trying to understand the point of causing
> guile-2.2's read of a non-blocking C port to block.  The whole point of
> making a descriptor non-blocking is that that shouldn't happen, and
> there are circumstances where pealing individual bytes off a
> non-blocking port as they become available is what you want to do.  It
> makes guile's select wrapper unusable with TCP sockets on linux.  I
> understand that suspendable-ports work differently, but that is another
> matter.

I would start by saying that many things are possible including adding
the ability to enable the error-throwing behavior.  I think your use
case is a great one and we should support it well.  I think though that
perhaps you didn't see how the waiter parameters could apply to your use
case; are things clearer now?  I think that once you allow yourself to
use port buffers, things will be much easier and much more efficient as
well.  But, I could be misunderstanding too.

Guile's I/O in the past was very oriented towards ASCII.  2.0 went a
long way towards good textual I/O with other encodings, and with 2.2 we
will have good textual non-blocking I/O.  It could be that by focusing
on this, I have neglected some binary non-blocking I/O use cases, though
I think that get-bytevector-some, having enabled suspendable ports,
serves many of these purposes very well.  But if there is a need for an
additional binary I/O primitive, let's figure out what it is and
implement it.

Andy

Re: anyone define port types?

[Chris Vine]

On Sun, 19 Jun 2016 19:48:03 +0200
Andy Wingo <wingo@pobox.com> wrote:
> On Sun 19 Jun 2016 17:33, Chris Vine <chris@cvine.freeserve.co.uk>
> writes:
> 
> > The answer I have adopted when reading from TCP sockets is to
> > extract individual bytes only from the port into a bytevector using
> > R6RS's get-u8 procedure and (if the port is textual rather than
> > binary) to reconstruct characters from that using
> > bytevector->string at, say, a line end[1].  An EAGAIN/EWOULDBLOCK
> > exception is then just treated as an invitation to return to the
> > prompt, and read state is retained in the bytevector.  
> 
> Yep, makes sense, though it will be really slow of course.  And, you'd
> have to make people rewrite their code to use your I/O primitives
> instead of using read-line from (ice-9 rdelim); a bit of a drag, but
> OK.

It's not excessively slow provided that the port itself (as opposed
to the conversion to unicode characters, which you have to do yourself)
can be buffered. You can loop calls to get-u8 on char-ready? to vacate
the buffers so far as wanted, without returning to the prompt for every
byte.  From that point of view it shouldn't be that much slower than
repeatedly calling read-char on a port with latin-1 encoding on a
normal blocking port.
 
[snip] 
> > I don't think I have got to grips with how to do that with
> > read-waiter, because the read-waiter comprises in effect another
> > loop (in which the main event loop with its own prompts would have
> > to run) until the read request has been satisfied.  I would need to
> > think about it.  Since ethreads use a poll()/epoll() loop,
> > presumably you think it is straightforward enough to integrate the
> > two, even if at present I don't.  
> 
> Here is the code.  First, a helper:
> 
>   ;; The AFTER-SUSPEND thunk allows the user to suspend the current
>   ;; thread, saving its state, and then perform some other nonlocal
>   ;; control flow.
>   ;;
>   (define* (suspend #:optional (after-suspend (lambda (ctx thread)
> #f))) ((abort-to-prompt (econtext-prompt-tag (current-econtext))
>                       after-suspend)))
> 
> Assume there is some current-econtext parameter with a "context" which
> holds the prompt tag associated with the scheduler.  As you see when
> the continuation resumes, it resumes by calling a thunk, allowing
> exceptions to be thrown from the context of the suspend.
> 
> Here's the main loop function, which you could replace with the GLib
> main loop or whatever:
> 
>   (define* (run #:optional (ctx (ensure-current-econtext))
>                 #:key (install-suspendable-ports? #t))
>     (when install-suspendable-ports? (install-suspendable-ports!))
>     (parameterize ((current-econtext ctx)
>                    (current-read-waiter wait-for-readable)
>                    (current-write-waiter wait-for-writable))
>       (let lp ()
>         (run-ethread ctx (next-thread ctx))
>         (lp))))
> 
> Cool.  Now the wait functions:
> 
>   (define (wait-for-readable port)
>     (wait-for-events port (port-read-wait-fd port) (logior EPOLLIN
> EPOLLRDHUP)))
> 
>   (define (wait-for-writable port)
>     (wait-for-events port (port-write-wait-fd port) EPOLLOUT))
> 
> Now the wait-for-events function:
> 
>   (define (wait-for-events port fd events)
>     (handle-events
>      port
>      events
>      (suspend
>       (lambda (ctx thread)
>         ...))))
> 
> Well that's a mess, but the thing to know is that the `suspend' will
> abort to the relevant prompt, and then invoke the thunk that's its
> argument.  Here's `handle-events' and we are done:
> 
>   (define (handle-events port events revents)
>     (unless (zero? (logand revents EPOLLERR))
>       (error "error reading from port" port)))
> 
> But I guess that error could be "reading or writing"; oh well.
> 
> See
> http://git.savannah.gnu.org/cgit/guile.git/tree/module/ice-9/ethreads.scm?h=wip-ethreads&id=253dc1a7114b89351a3aa330caf173b98c5a65dd,
> it's not long but it takes some time to read.  I think I can fairly
> ask that of you though, given your interest in this area :)

OK I am grateful for your patience in explaining this.  I need to think
about it, but while this works where all events come from user-derived
events, I doubt that this would work with guile-gnome and the glib main
loop in the round, because for gtk+ to function the glib main loop must
block within glib's own internal call to the poll() and not within the
guile prompt, or gdk and other events will jam up.  You would probably
need to run the glib main loop in its own (pthread) thread, about which
you have previously given dire warnings so far as guile as it currently
stands is concerned.

As I say, I really need to think more about this and look further at
the code.  I may well be missing something.

> > On a side issue, I am still trying to understand the point of
> > causing guile-2.2's read of a non-blocking C port to block.  The
> > whole point of making a descriptor non-blocking is that that
> > shouldn't happen, and there are circumstances where pealing
> > individual bytes off a non-blocking port as they become available
> > is what you want to do.  It makes guile's select wrapper unusable
> > with TCP sockets on linux.  I understand that suspendable-ports
> > work differently, but that is another matter.  
> 
> I would start by saying that many things are possible including adding
> the ability to enable the error-throwing behavior.  I think your use
> case is a great one and we should support it well.  I think though
> that perhaps you didn't see how the waiter parameters could apply to
> your use case; are things clearer now?  I think that once you allow
> yourself to use port buffers, things will be much easier and much
> more efficient as well.  But, I could be misunderstanding too.
> 
> Guile's I/O in the past was very oriented towards ASCII.  2.0 went a
> long way towards good textual I/O with other encodings, and with 2.2
> we will have good textual non-blocking I/O.  It could be that by
> focusing on this, I have neglected some binary non-blocking I/O use
> cases, though I think that get-bytevector-some, having enabled
> suspendable ports, serves many of these purposes very well.  But if
> there is a need for an additional binary I/O primitive, let's figure
> out what it is and implement it.

This is an issue with R6RS.  In my view it ought to have a
get-bytevector-some! procedure which emulates unix read(), but once you
decide that C ports should not have non-blocking reads, that is an end
to it really.

In my view though, a non-blocking get-bytevector-some! would be really
nice.  It could return the number of bytes extracted, with a return
value of 0 for EAGAIN.

Chris

Re: anyone define port types?

[William ML Leslie]

On 20 June 2016 at 06:09, Chris Vine <chris@cvine.freeserve.co.uk> wrote:
> OK I am grateful for your patience in explaining this.  I need to think
> about it, but while this works where all events come from user-derived
> events, I doubt that this would work with guile-gnome and the glib main
> loop in the round, because for gtk+ to function the glib main loop must
> block within glib's own internal call to the poll() and not within the
> guile prompt, or gdk and other events will jam up.  You would probably
> need to run the glib main loop in its own (pthread) thread, about which
> you have previously given dire warnings so far as guile as it currently
> stands is concerned.
>
> As I say, I really need to think more about this and look further at
> the code.  I may well be missing something.

If the event loop is external as you suggested, you could use a
different implementation of the loop that used g_io_add_watch and
returned to the glib mainloop when you're in a glib or gtk
application.

-- 
William Leslie

Notice:
Likely much of this email is, by the nature of copyright, covered
under copyright law.  You absolutely MAY reproduce any part of it in
accordance with the copyright law of the nation you are reading this
in.  Any attempt to DENY YOU THOSE RIGHTS would be illegal without
prior contractual agreement.

Re: anyone define port types?

[Chris Vine]

On Mon, 20 Jun 2016 13:38:39 +1000
William ML Leslie <william.leslie.ttg@gmail.com> wrote:
> On 20 June 2016 at 06:09, Chris Vine <chris@cvine.freeserve.co.uk>
> wrote:
> > OK I am grateful for your patience in explaining this.  I need to
> > think about it, but while this works where all events come from
> > user-derived events, I doubt that this would work with guile-gnome
> > and the glib main loop in the round, because for gtk+ to function
> > the glib main loop must block within glib's own internal call to
> > the poll() and not within the guile prompt, or gdk and other events
> > will jam up.  You would probably need to run the glib main loop in
> > its own (pthread) thread, about which you have previously given
> > dire warnings so far as guile as it currently stands is concerned.
> >
> > As I say, I really need to think more about this and look further at
> > the code.  I may well be missing something.  
> 
> If the event loop is external as you suggested, you could use a
> different implementation of the loop that used g_io_add_watch and
> returned to the glib mainloop when you're in a glib or gtk
> application.

Sure, but that's not really the difficulty.

On further reflection on Andy's post, I think what I understand to be
his approach to suspendable-ports could be made to work with a glib
main loop without invoking additional pthread threads, but I think it
would be barely scalable or tractable.

For Andy's sake, can I give a concrete example of an entirely single
threaded program?  Let's say the program uses guile-gnome and is
running a glib main loop, on which the program is (of necessity if
using a glib main loop) blocking in g-main-loop-run, so all code is
running as events in the main loop. For simplicity, let's say you have
a file watch in the glib event loop which has made a non-blocking read
of the first byte of a multi-byte UTF-8 character, and the
suspendable-ports implementation is in use because it is a non-blocking
read of a TCP socket.  (It could be something different: there may be a
non-blocking read request for a complete line of text which has so far
only partially been satisfied, but the partially complete character is
the easiest example to deal with.)  The read request is therefore in
read-waiter waiting for a complete UTF-8 byte sequence to arrive.  On
the current hypothesis, read-waiter comprises a procedure which is
suspended on a prompt waiting for an event to occur in the glib main
loop which will cause it to resume, comprising the file descriptor
becoming ready which will satisfy the read request.  But while
suspended in read-waiter, this prompt would have to service any user
event sources which might become ready in the glib main loop, not just
the particular file descriptor in question becoming ready. These could
be nested with other non-blocking reads, also in read-waiter, on other
descriptors which also have a pending partial read, or with time-outs
or with tasks composed with idle handlers.  Any user event source which
has been attached to the glib main loop which fires would have to cause
all pending partial reads to go through the nested non-blocking read
cycles to see if anything is available, as well as servicing the
particular user event in question.  It seems hardly scalable, if it
would work at all.

My approach on the other hand does not nest events from the glib main
loop in this way.

It is still possible I have missed something.  The proof of all these
things is in the pudding.  I suppose what is needed is an attempt at a
practical implementation using read-waiter: but for the moment I don't
see it.

Chris

Re: anyone define port types?

[Andy Wingo]

On Mon 20 Jun 2016 08:45, Chris Vine <chris@cvine.freeserve.co.uk> writes:

> For simplicity, let's say you have a file watch in the glib event loop
> which has made a non-blocking read of the first byte of a multi-byte
> UTF-8 character, and the suspendable-ports implementation is in use
> because it is a non-blocking read of a TCP socket.  (It could be
> something different: there may be a non-blocking read request for a
> complete line of text which has so far only partially been satisfied,
> but the partially complete character is the easiest example to deal
> with.)  The read request is therefore in read-waiter waiting for a
> complete UTF-8 byte sequence to arrive.  On the current hypothesis,
> read-waiter comprises a procedure which is suspended on a prompt
> waiting for an event to occur in the glib main loop which will cause
> it to resume, comprising the file descriptor becoming ready which will
> satisfy the read request.  But while suspended in read-waiter, this
> prompt would have to service any user event sources which might become
> ready in the glib main loop, not just the particular file descriptor
> in question becoming ready.

I must not be communicating clearly because this is definitely not what
I am proposing.  The prompt doesn't service anything, and it's just the
one user-space thread which is suspended, and when it suspends, it
suspends back to the main loop which runs as usual, timers and all.

                        prompt
   /------\ /------------\|/-----------\ /------------\ /----------\
   | main --> run-thread -|>(user code)-->  read-char -->  waiter  |
   | loop | |            |||           | |            | |          |
   \------/ \------------/|\-----------/ \------------/ \--|-------/
                          ^                                |
                           \-------------------------------/
   stack grows this way ->

The current-read-waiter aborts to a prompt.  That prompt is instated
when the thread is run or resumed.  When you abort to that prompt, you
add the FD to the poll set / main loop / *, remember the delimited
continuation, and return to the main loop.  When the fd becomes readable
or the gsource fires or whatever, you reinstate the delimited
continuation via a new invocation of run-thread (prompt and all).

> My approach on the other hand does not nest events from the glib main
> loop in this way.

"Nesting events" should not be a thing you have to do.

Andy

Re: anyone define port types?

[Chris Vine]

On Mon, 20 Jun 2016 09:34:26 +0200
Andy Wingo <wingo@pobox.com> wrote:
[snip]
> I must not be communicating clearly because this is definitely not
> what I am proposing.  The prompt doesn't service anything, and it's
> just the one user-space thread which is suspended, and when it
> suspends, it suspends back to the main loop which runs as usual,
> timers and all.
> 
>                         prompt
>    /------\ /------------\|/-----------\ /------------\ /----------\
>    | main --> run-thread -|>(user code)-->  read-char -->  waiter  |
>    | loop | |            |||           | |            | |          |
>    \------/ \------------/|\-----------/ \------------/ \--|-------/
>                           ^                                |
>                            \-------------------------------/
>    stack grows this way ->
> 
> The current-read-waiter aborts to a prompt.  That prompt is instated
> when the thread is run or resumed.  When you abort to that prompt, you
> add the FD to the poll set / main loop / *, remember the delimited
> continuation, and return to the main loop.  When the fd becomes
> readable or the gsource fires or whatever, you reinstate the
> delimited continuation via a new invocation of run-thread (prompt and
> all).

Ah right, that is clearer, thank you.  There would indeed be a prompt
for each user glib event source comprised in the "thread" abstraction,
which the read-waiter (or whatever) aborts to.  It is that abstraction
that I was missing and will need to look at.

Chris

Re: anyone define port types?

[Chris Vine]

On Mon, 20 Jun 2016 10:01:57 +0100
Chris Vine <chris@cvine.freeserve.co.uk> wrote:
> On Mon, 20 Jun 2016 09:34:26 +0200
> Andy Wingo <wingo@pobox.com> wrote:
> [snip]
> > I must not be communicating clearly because this is definitely not
> > what I am proposing.  The prompt doesn't service anything, and it's
> > just the one user-space thread which is suspended, and when it
> > suspends, it suspends back to the main loop which runs as usual,
> > timers and all.
> > 
> >                         prompt
> >    /------\ /------------\|/-----------\ /------------\ /----------\
> >    | main --> run-thread -|>(user code)-->  read-char -->  waiter  |
> >    | loop | |            |||           | |            | |          |
> >    \------/ \------------/|\-----------/ \------------/ \--|-------/
> >                           ^                                |
> >                            \-------------------------------/
> >    stack grows this way ->
> > 
> > The current-read-waiter aborts to a prompt.  That prompt is instated
> > when the thread is run or resumed.  When you abort to that prompt,
> > you add the FD to the poll set / main loop / *, remember the
> > delimited continuation, and return to the main loop.  When the fd
> > becomes readable or the gsource fires or whatever, you reinstate the
> > delimited continuation via a new invocation of run-thread (prompt
> > and all).  
> 
> Ah right, that is clearer, thank you.  There would indeed be a prompt
> for each user glib event source comprised in the "thread" abstraction,
> which the read-waiter (or whatever) aborts to.  It is that abstraction
> that I was missing and will need to look at.

I have stirred myself and installed guile-2.1.3.  On looking more at
the suspendable ports code it became obvious and I haven't needed to
adopt anything like ethreads with its "thread" abstraction: instead I
have kept the approach already adopted in the guile-a-sync library.
However, the consequence of using suspendable ports instead of C ports
is that the await-getline! procedure (as an example) has been reduced
to a mere 16 lines of code, mainly because it is possible to use (ice-9
rdelim)'s read-line procedure with non-blocking ports.

I have made a new repository for guile-a-sync for use with guile-2.2 and
when I am happy with the new interfaces (and assuming nothing else goes
wrong) I will put it up on github.

This is very nice.  Thanks for taking the time to go through it with me.

Chris

Re: anyone define port types?

[Andy Wingo]

On Thu 23 Jun 2016 00:44, Chris Vine <chris@cvine.freeserve.co.uk> writes:

> I have stirred myself and installed guile-2.1.3.  On looking more at
> the suspendable ports code it became obvious and I haven't needed to
> adopt anything like ethreads with its "thread" abstraction: instead I
> have kept the approach already adopted in the guile-a-sync library.

Excellent.  Though I think that eventually we will want to bless one of
these concurrency patterns as the default one, we're a long way away
from that, and even if we do bless one I think we will always want to
allow people to experiment and deploy different ones.  So, great, glad
to hear you are doing work in this area :)

> However, the consequence of using suspendable ports instead of C ports
> is that the await-getline! procedure (as an example) has been reduced
> to a mere 16 lines of code, mainly because it is possible to use (ice-9
> rdelim)'s read-line procedure with non-blocking ports.

Excellent!  That's the result I was hoping for.

> I have made a new repository for guile-a-sync for use with guile-2.2 and
> when I am happy with the new interfaces (and assuming nothing else goes
> wrong) I will put it up on github.
>
> This is very nice.  Thanks for taking the time to go through it with me.

Thank you for your patience too!

The only drawback that I know of with the strategy of simply allowing
users to use Guile's I/O primitives (e.g., `read-line') and assuming
that they'll suspend when they block is that not all of the primitives
that you would like to you have been rebased on top of the ones
implemented by (ice-9 suspendable-ports).  So if you find such an I/O
procedure that needs to be rewritten to use put-string instead of
display, for example, then please do send a patch.

As for `display' and `write'... I guess we can at least make suspendable
ports handle these in a non-blocking way for some limited number of data
types like strings, characters, and symbols, and dispatch to the C
`display' implementation for more complicated data types like SMOBs or
so.  We'll see I guess.

Happy hacking,

Andy

Re: anyone define port types?

[Andy Wingo]

Little typo:

On Thu 23 Jun 2016 09:36, Andy Wingo <wingo@pobox.com> writes:

> The only drawback that I know of with the strategy of simply allowing
> users to use Guile's I/O primitives (e.g., `read-line') and assuming
> that they'll suspend when they block is that not all of the primitives
> that you would like to you have been rebased on top of the ones
                        s/you/use/
> implemented by (ice-9 suspendable-ports).  So if you find such an I/O
> procedure that needs to be rewritten to use put-string instead of
> display, for example, then please do send a patch.

Cheers,

Andy

Re: anyone define port types?

[Chris Vine]

On Thu, 23 Jun 2016 09:36:48 +0200
Andy Wingo <wingo@pobox.com> wrote:
[snip]
> Excellent.  Though I think that eventually we will want to bless one
> of these concurrency patterns as the default one, we're a long way
> away from that, and even if we do bless one I think we will always
> want to allow people to experiment and deploy different ones.  So,
> great, glad to hear you are doing work in this area :)

A few things on that.  First, there will always be a use for an event
loop to do event-loopy things, irrespective of whether and how a
coroutine interface is put around it.  Sometimes you want to abstract
things away, sometimes you don't.

Secondly, as I understand it in the end you want pre-emptive "green"
threads for guile, whereas my code equates to co-operative
multi-tasking, whilst also working with native threads.  I must come
clean and say that I don't like "green" threads.

Which leads on to the third point, which is that I would like to see
guile match its words (in its documentation) with respect to native
threads.  I have found they work fine, with caution about shared global
state.  You think they don't, except possibly on intel, because some of
its lock-free structures/variables -- and I think you are possibly
referring to the VM here -- lack appropriate fences or other atomics.
(The higher level C and scheme code has plainly had serious attempts at
thread-safety made for it using mutexes.)

Chris

Re: anyone define port types?

[Marko Rauhamaa]

Chris Vine <chris@cvine.freeserve.co.uk>:

> First, there will always be a use for an event loop to do event-loopy
> things, irrespective of whether and how a coroutine interface is put
> around it. Sometimes you want to abstract things away, sometimes you
> don't.

Callback hell is my preferred programming paradigm. Any attempts to
abstract it away are going to lead to a worse mess.


Marko

Re: anyone define port types?

[Andy Wingo]

On Thu 23 Jun 2016 11:24, Chris Vine <chris@cvine.freeserve.co.uk> writes:

> A few things on that.  First, there will always be a use for an event
> loop to do event-loopy things, irrespective of whether and how a
> coroutine interface is put around it.  Sometimes you want to abstract
> things away, sometimes you don't.

Yep!

> Secondly, as I understand it in the end you want pre-emptive "green"
> threads for guile, whereas my code equates to co-operative
> multi-tasking, whilst also working with native threads.  I must come
> clean and say that I don't like "green" threads.

I actually don't want *pre-emptive* threads -- I'd rather have threads
suspend at I/O points instead.  But yeah, my preferred abstraction is
threads; and those user-space threads could be bound to a core, or we
could schedule them on multiple cores.  But there's room for many things
:)

> Which leads on to the third point, which is that I would like to see
> guile match its words (in its documentation) with respect to native
> threads.  I have found they work fine, with caution about shared global
> state.  You think they don't, except possibly on intel, because some of
> its lock-free structures/variables -- and I think you are possibly
> referring to the VM here -- lack appropriate fences or other atomics.
> (The higher level C and scheme code has plainly had serious attempts at
> thread-safety made for it using mutexes.)

I see what you mean.  On the other hand I don't like documenting bugs :/
(I prefer to fix them, and find that often the effort is similar.)  Good
native thread support is within Guile's remit and everyone should have
the good experience that you have had.  But that's not something we can
guarantee right now unfortunately :/

Andy

Re: anyone define port types?

[William ML Leslie]

On 23 June 2016 at 20:43, Andy Wingo <wingo@pobox.com> wrote:
> On Thu 23 Jun 2016 11:24, Chris Vine <chris@cvine.freeserve.co.uk> writes:
>> Secondly, as I understand it in the end you want pre-emptive "green"
>> threads for guile, whereas my code equates to co-operative
>> multi-tasking, whilst also working with native threads.  I must come
>> clean and say that I don't like "green" threads.
>
> I actually don't want *pre-emptive* threads -- I'd rather have threads
> suspend at I/O points instead.  But yeah, my preferred abstraction is
> threads; and those user-space threads could be bound to a core, or we
> could schedule them on multiple cores.  But there's room for many things
> :)

It would be nice to be able to avoid pre-emption at I/O points also,
ensuring that turns on the event loop run atomically to completion,
and any IO wait happens on the next turn.

I guess I can discover any pre-emption at I/O points easily with a
dynamic-wind handler or custom loop and go and rewrite them.

-- 
William Leslie

Notice:
Likely much of this email is, by the nature of copyright, covered
under copyright law.  You absolutely MAY reproduce any part of it in
accordance with the copyright law of the nation you are reading this
in.  Any attempt to DENY YOU THOSE RIGHTS would be illegal without
prior contractual agreement.