on coding a portable R6RS package supporting Guile and its FFI

on coding a portable R6RS package supporting Guile and its FFI

[Marco Maggi]

Ciao,

  in my latest package[1] I am using Guile 2.0.7 and its FFI
along with  Mosh Scheme, Racket, Sagittarius  Scheme, Vicare
Scheme, Ypsilon Scheme.  I have the following observations:

* Guile  does  not  come  with  the  very  simple  SRFI  78:
  lightweight testing.  I had to include it[2][3].

* It appears  that there  is no  facility to  handle "output
  arguments" from  C functions; I  mean the cases where  a C
  function accepts  as argument  a pointer to  variable that
  will be  filled with  some computed value.   I am  using a
  WITH-LOCAL-STORAGE[4]  macro  which  is  maybe  ugly,  but
  works.

  Such  arguments are  common, and  represent a  nuisance to
  handle.  Racket has a sophisticated interface, complicated
  to use  when writing adapter code.   Something simpler but
  built in would be useful (this is the sort of thing a user
  does  not want  to think  about: it  should be  an already
  solved problem).

* Whenever  a callout  to C  accepts a  pointer argument:  a
  bytevector argument  is rejected.   Is this not  a useless
  complication?

  One   can    work   around   it   by    explicitly   using
  BYTEVECTOR->POINTER, so everything is ready in Guile.  The
  other  Scheme   implementations  using   a  non-compacting
  garbage collector already support  this feature and I find
  it truly convenient.

* There are no raw memory  getters and setters[5]?  The fact
  that it is  possible to create a wrapping  bytevector is a
  plus for  sure, but I  find it inconvenient to  allocate a
  bytevector  when I  do not  need it  (and raw  getters and
  setters are really small functions).

* The limit of  10 arguments for callouts to  C is annoying.
  It forced me  to exclude some SOFA  functions resulting in
  amputated Guile support.

TIA

[1] <http://github.com/marcomaggi/r6rs-sofa/>
[2] <http://github.com/marcomaggi/r6rs-sofa/tree/master/lib/guile/srfi>
[3] <http://github.com/marcomaggi/vicare/blob/devel/doc/srfi-lightweight-testing.texi>
[4] <http://github.com/marcomaggi/r6rs-sofa/blob/master/lib/sofa/compat.guile.sls>
[5] <http://marcomaggi.github.com/docs/vicare.html/iklib-pointers.html>
-- 
Marco Maggi

Re: on coding a portable R6RS package supporting Guile and its FFI

[Ludovic Courtès]

Hi Marco,

Sorry for the delay.

Marco Maggi <marco.maggi-ipsu@poste.it> skribis:

> * Guile  does  not  come  with  the  very  simple  SRFI  78:
>   lightweight testing.  I had to include it[2][3].

Patches welcome.  :-)

> * It appears  that there  is no  facility to  handle "output
>   arguments" from  C functions; I  mean the cases where  a C
>   function accepts  as argument  a pointer to  variable that
>   will be  filled with  some computed value.   I am  using a
>   WITH-LOCAL-STORAGE[4]  macro  which  is  maybe  ugly,  but
>   works.

I typically roll my own allocation and dereference routines as well,
such as ‘make-int-pointer’ and ‘dereference-int’ at:

  http://git.savannah.gnu.org/cgit/libchop.git/tree/guile2/chop/internal.scm#n255

Perhaps adding them to (system foreign) would help?

>   Such  arguments are  common, and  represent a  nuisance to
>   handle.  Racket has a sophisticated interface, complicated
>   to use  when writing adapter code.   Something simpler but
>   built in would be useful (this is the sort of thing a user
>   does  not want  to think  about: it  should be  an already
>   solved problem).

I realize it would be good to have facilities already available for
this.

However, I’m not familiar with Racket’s FFI, and it’s not clear to me
what a good generic API would be.

For instance, one could imagine a layer on top of ‘pointer->procedure’
that would allow to specify whether pointer arguments really are output
arguments.  But then, you’d also have to allow programmers to tell how
output arguments are to be allocated (“pointerless” or not, ‘malloc’,
etc.), who owns them, what their life cycle is, etc.

All in all, it’s always seemed easier to me to do that manually, with
helpers specifically adapted to the C library I write bindings for.

WDYT?

> * Whenever  a callout  to C  accepts a  pointer argument:  a
>   bytevector argument  is rejected.   Is this not  a useless
>   complication?
>
>   One   can    work   around   it   by    explicitly   using
>   BYTEVECTOR->POINTER, so everything is ready in Guile.  The
>   other  Scheme   implementations  using   a  non-compacting
>   garbage collector already support  this feature and I find
>   it truly convenient.

Well, the ‘pointer’ type is useful, because it’s inherently a more
low-level representation than bytevectors.

That said, the FFI call could implicitly convert bytevectors to
pointers.  However, I generally prefer avoiding implicit type
conversions like these, for clarify.

Thoughts?

> * There are no raw memory  getters and setters[5]?

There’s only ‘dereference-pointer’ now, but I agree we could add more of
these, as well as pointer arithmetic operators.

> * The limit of  10 arguments for callouts to  C is annoying.
>   It forced me  to exclude some SOFA  functions resulting in
>   amputated Guile support.

Agreed.

Would you like to propose a patch in some of these areas?

Thanks for your feedback,
Ludo’.

Re: on coding a portable R6RS package supporting Guile and its FFI

[Daniel Hartwig]

On 27 January 2013 23:08, Ludovic Courtès <ludo@gnu.org> wrote:
> Hi Marco,
>
> Sorry for the delay.
>
> Marco Maggi <marco.maggi-ipsu@poste.it> skribis:
>> * It appears  that there  is no  facility to  handle "output
>>   arguments" from  C functions; I  mean the cases where  a C
>>   function accepts  as argument  a pointer to  variable that
>>   will be  filled with  some computed value.   I am  using a
>>   WITH-LOCAL-STORAGE[4]  macro  which  is  maybe  ugly,  but
>>   works.
>
> I typically roll my own allocation and dereference routines as well,
> such as ‘make-int-pointer’ and ‘dereference-int’ at:

Likewise.

>
>   http://git.savannah.gnu.org/cgit/libchop.git/tree/guile2/chop/internal.scm#n255
>
> Perhaps adding them to (system foreign) would help?

Yes, in these generic cases.

>
>>   Such  arguments are  common, and  represent a  nuisance to
>>   handle.  Racket has a sophisticated interface, complicated
>>   to use  when writing adapter code.  Something simpler but
>>   built in would be useful

Pointer arguments are common, yes, but they have a variety of usage
patterns across different APIs.  This is evident in the complexity of
the Racket API for dealing with this.

In particular, the questions of who (caller or callee) allocates the
memory and who deallocates the memory complicate the handling.  In
some APIs, the party in control of deallocation can change.

>>  (this is the sort of thing a user
>>   does  not want  to think  about: it  should be  an already
>>   solved problem).

Mapping any pointer interface requires some user thought to consider
its unique behaviour.  It is risky to abstract some of that process
away from the programmer, even if only to accomodate a single, “most
common” usage.

IMO the best that could be done without complicating the FFI is to
bring in a small set of generic helpers such as Ludo's
make-int-pointer, etc..

> …
>
> All in all, it’s always seemed easier to me to do that manually, with
> helpers specifically adapted to the C library I write bindings for.
>
> WDYT?

Right.  After considering the requirements for any given interface it
is only a small amount of work to create the appropriate helpers.
Being specific to the API under consideration, such helpers can
present an optimal interface.

>
>> * Whenever  a callout  to C  accepts a  pointer argument:  a
>>   bytevector argument  is rejected.   Is this not  a useless
>>   complication?

No, it is sensible type checking: bytevector is not pointer.

>>
>>   One   can    work   around   it   by    explicitly   using
>>   BYTEVECTOR->POINTER, so everything is ready in Guile.  The
>>   other  Scheme   implementations  using   a  non-compacting
>>   garbage collector already support  this feature and I find
>>   it truly convenient.
>
> Well, the ‘pointer’ type is useful, because it’s inherently a more
> low-level representation than bytevectors.
>
> That said, the FFI call could implicitly convert bytevectors to
> pointers.  However, I generally prefer avoiding implicit type
> conversions like these, for clarify.
>
> Thoughts?

This implicit conversion must be avoided.  Instead, how about
supporting supporting SRFI-4 types (s16vector, etc.) or typed arrays
in the FFI.  This would only cover some use cases, where the caller
controls the memory and the foreign argument is a /typed/ pointer.
This much could be implemented without seriously complicating the FFI.

For “void *” it is still required to explicitly pass a pointer object,
though it is not difficult to create helpers appropriate to the
/particular/ interface.

For callee allocated or deallocated memory it is still required to
interface using pointer objects.

SRFI-4 has the advantage that slots are always stored contiguously in
memory.  Typed array slots can be stored non-contiguously, and any
such array would have to be rejected as “array-contents” does.

Regards

Re: on coding a portable R6RS package supporting Guile and its FFI

[Andy Wingo]

On Thu 17 Jan 2013 11:53, Marco Maggi <marco.maggi-ipsu@poste.it> writes:

> * Whenever  a callout  to C  accepts a  pointer argument:  a
>   bytevector argument  is rejected.   Is this not  a useless
>   complication?
>
>   One   can    work   around   it   by    explicitly   using
>   BYTEVECTOR->POINTER, so everything is ready in Guile.  The
>   other  Scheme   implementations  using   a  non-compacting
>   garbage collector already support  this feature and I find
>   it truly convenient.

I agree that we would do well to allow bytevectors wherever an argument
type is a pointer.  It avoids an allocation in a very common case
without changing the expressiveness of the interface or adding
significant additional overhead.

Ludovic sounded like he could go either way on this point.  Daniel
brought up the argument that "a bytevector is not a pointer"; but for
many purposes it's not a terrible thing to think of it that way.  IMHO.

Andy
-- 
http://wingolog.org/

Re: on coding a portable R6RS package supporting Guile and its FFI

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

Hello,

That does sound very useful. The only thing that makes me worried is


On Tue, Feb 5, 2013 at 12:38 PM, Andy Wingo <wingo@pobox.com> wrote:

> On Thu 17 Jan 2013 11:53, Marco Maggi <marco.maggi-ipsu@poste.it> writes:
> >   The  other  Scheme   implementations  using   a  non-compacting
> >   garbage collector
>

Do we want to guarantee that our garbage collector will always be
non-compacting, or force major interface changes if it is? (I realize we've
already taken out smob mark procedures, but this is in some ways even
harder to add back.)

I'm not sure this is a big deal, but maybe we should consider what would
happen in the future if we did want to change. Using bytevector->pointer is
an acceptable cost to pay for flexibility, IMHO.

Noah

[-- Attachment #2: Type: text/html, Size: 1340 bytes --]

Re: on coding a portable R6RS package supporting Guile and its FFI

[Andy Wingo]

On Tue 05 Feb 2013 20:29, Noah Lavine <noah.b.lavine@gmail.com> writes:

> On Tue, Feb 5, 2013 at 12:38 PM, Andy Wingo <wingo@pobox.com> wrote:
>
>     On Thu 17 Jan 2013 11:53, Marco Maggi <marco.maggi-ipsu@poste.it>
>     writes:
>     >   The  other  Scheme   implementations  using   a  non-compacting
>     >   garbage collector 
>
> Do we want to guarantee that our garbage collector will always be
> non-compacting, or force major interface changes if it is? (I realize
> we've already taken out smob mark procedures, but this is in some ways
> even harder to add back.)

Smob mark procedures are still supported.  Changing the GC would be
quite difficult -- we would need to progress to the point that we had
significantly less C.  That would be years away if it comes.  Anyway, we
could have a compacting collector and still allow fixed-position
objects:

  http://www.scheme.com/csug8/smgmt.html#./smgmt:h3

Andy
-- 
http://wingolog.org/

Re: on coding a portable R6RS package supporting Guile and its FFI

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 747 bytes --]

On Wed, Feb 6, 2013 at 3:36 AM, Andy Wingo <wingo@pobox.com> wrote:

> On Tue 05 Feb 2013 20:29, Noah Lavine <noah.b.lavine@gmail.com> writes:
>
> Smob mark procedures are still supported.  Changing the GC would be
> quite difficult -- we would need to progress to the point that we had
> significantly less C.  That would be years away if it comes.  Anyway, we
> could have a compacting collector and still allow fixed-position
> objects:
>

I see. I hadn't thought about what a big project it would be. I was
imagining bytevector->pointer as a procedure that just set some
"fixed-position bit" on its object, so there wasn't any allocation.
However, that would fit weirdly with the "->" convention. It's probably not
worth worrying about.

Noah

[-- Attachment #2: Type: text/html, Size: 1247 bytes --]

Re: on coding a portable R6RS package supporting Guile and its FFI

[Ludovic Courtès]

Andy Wingo <wingo@pobox.com> skribis:

> I agree that we would do well to allow bytevectors wherever an argument
> type is a pointer.  It avoids an allocation in a very common case
> without changing the expressiveness of the interface or adding
> significant additional overhead.

I find it somewhat inelegant, though.

This would be less of a problem if we had “immediate pointers” [0].
Would the retagging in 2.1 allow this?

(At FOSDEM, Luca Saiu rightfully noted that we could use 4-bit tags
instead of 3-bit tags on 64-bit arches, which would give us the needed
room here.)

Ludo’.

[0] http://lists.gnu.org/archive/html/guile-devel/2011-01/msg00159.html

Re: on coding a portable R6RS package supporting Guile and its FFI

[Andy Wingo]

On Tue 05 Feb 2013 23:31, ludo@gnu.org (Ludovic Courtès) writes:

> This would be less of a problem if we had “immediate pointers” [0].
> Would the retagging in 2.1 allow this?

The tc3 itag allocation in wip-retagging is:

  * tc3:
  *   000:  A heap object with a tag word on the heap
  *   001:  Some other immediate
  *   010:  A pair
  *   011:  Small integer (odd)
  *   100:  (Unallocated tc3.)
  *   101:  (Unallocated tc3.)
  *   110:  A struct
  *   111:  Small integer (even)

And SCM_IMP is:

  #define SCM_IMP(x) 		(1 & SCM_UNPACK (x))

So perhaps 101 could be an immediate pointer.  Dunno.  Pretty nasty
stuff!

> (At FOSDEM, Luca Saiu rightfully noted that we could use 4-bit tags
> instead of 3-bit tags on 64-bit arches, which would give us the needed
> room here.)

GC_malloc still returns eight-byte aligned memory hunks on 64-bit, no?

Andy
-- 
http://wingolog.org/

Re: on coding a portable R6RS package supporting Guile and its FFI

[Ludovic Courtès]

Andy Wingo <wingo@pobox.com> skribis:

> On Tue 05 Feb 2013 23:31, ludo@gnu.org (Ludovic Courtès) writes:
>
>> This would be less of a problem if we had “immediate pointers” [0].
>> Would the retagging in 2.1 allow this?
>
> The tc3 itag allocation in wip-retagging is:
>
>   * tc3:
>   *   000:  A heap object with a tag word on the heap
>   *   001:  Some other immediate
>   *   010:  A pair
>   *   011:  Small integer (odd)
>   *   100:  (Unallocated tc3.)
>   *   101:  (Unallocated tc3.)
>   *   110:  A struct
>   *   111:  Small integer (even)
>
> And SCM_IMP is:
>
>   #define SCM_IMP(x) 		(1 & SCM_UNPACK (x))
>
> So perhaps 101 could be an immediate pointer.  Dunno.  Pretty nasty
> stuff!

Would be cool, I think.  Maybe we should make an auction for unused tc3
tags?  :-)

>> (At FOSDEM, Luca Saiu rightfully noted that we could use 4-bit tags
>> instead of 3-bit tags on 64-bit arches, which would give us the needed
>> room here.)
>
> GC_malloc still returns eight-byte aligned memory hunks on 64-bit, no?

Oh right:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (use-modules (system foreign))
scheme@(guile-user)> (pointer->procedure '* (dynamic-func "GC_malloc" (dynamic-link)) (list size_t))
$2 = #<procedure 1b60ac0 (_)>
scheme@(guile-user)> ,use (srfi srfi-1)
scheme@(guile-user)> ,use (srfi srfi-26)
scheme@(guile-user)> (unfold (cut > <> 100) (lambda (i) (logand (pointer-address ($2 (* i 2)) ) #x15)) 1+ 0)
$7 = (0 0 16 0 16 0 16 0 16 0 0 0 0 0 0 0 0 16 0 0 16 0 0 16 0 0 0 0 0 0 0 0 0 16 0 16 0 16 0 16 0 0 0 0 0 0 0 0 0 0 16 0 16 0 16 0 16 0 0 0 0 0 0 0 0 16 0 16 0 16 0 16 0 0 0 0 0 0 0 0 0 16 0 16 0 16 0 16 0 0 0 0 0 0 0 0 0 16 0 16 0)
--8<---------------cut here---------------end--------------->8---

If needed, we could just round up all the addresses returned.

And with 16 cases, we could even put single-precision floats in a word,
chars, and who knows what!

The downside is that we’d have to maintain two tag systems.

Ludo’.

Re: on coding a portable R6RS package supporting Guile and its FFI

[Andy Wingo]

Hi :)

On Fri 08 Feb 2013 15:10, ludo@gnu.org (Ludovic Courtès) writes:

>> GC_malloc still returns eight-byte aligned memory hunks on 64-bit, no?
>
> Oh right:

Does it?  I couldn't tell from your test.  I'm not a native unfolder :)

> And with 16 cases, we could even put single-precision floats in a word,
> chars, and who knows what!

Chars are already immediate :)

Cheers,

Andy
-- 
http://wingolog.org/

Re: on coding a portable R6RS package supporting Guile and its FFI

[Ludovic Courtès]

Andy Wingo <wingo@pobox.com> skribis:

> On Fri 08 Feb 2013 15:10, ludo@gnu.org (Ludovic Courtès) writes:
>
>>> GC_malloc still returns eight-byte aligned memory hunks on 64-bit, no?
>>
>> Oh right:
>
> Does it?  I couldn't tell from your test.  I'm not a native unfolder :)

Yes, it’s always eight-byte aligned (even on 32-bit IIRC), and often
16-byte aligned.

>> And with 16 cases, we could even put single-precision floats in a word,
>> chars, and who knows what!
>
> Chars are already immediate :)

Right, well, strings then!  ;-)

Ludo’.