C calling Scheme and garbage collection

C calling Scheme and garbage collection

[Isaac Jurado]

[-- Attachment #1: Type: text/plain, Size: 1793 bytes --]

Hello,

I'm playing with event loop libraries implemented in C (libev, libevent,
etc... in my case libsystemd), but configuring them from Guile.

The qsort example in the documentation [1] seems safe because the qsort C
function directly calls back, so the callback Scheme bindings stay
referenced (by the Scheme code calling qsort) during all the C code
execution.

Now, in C event loops the situation is different.  There is one call to
configure the event callback, in which the function and data pointers are
lent to the loop; and then there is the main loop or the single iteration
call.

The way I see it, suppose I add a timer.  I call one C function passing a
(proceudre->pointer) and an (scm->pointer).  In a future time, those
pointers will be used by the C event loop.  If a garbage collection happens
in the middle, the results of (procedure->pointer) and (scm->pointer) may
have been reclaimed by the time the C event loop calls back.

However, I've tried forcing (gc) between the two steps mentioned and it
looks to be working fine.  I have also reviewed some of the code [2][3] and
some additional weak references seem to be created.

I'll try a couple of combinations to see if I can "break" it, in order to
learn the limitations of C calling Scheme within Guile.  But if someone in
the list has experienced with this already, I would very glad to read some
advice about it.

Best regards.

References:

[1]
https://www.gnu.org/software/guile/manual/html_node/Dynamic-FFI.html#index-procedure_002d_003epointer
[2]
http://git.savannah.gnu.org/cgit/guile.git/tree/libguile/foreign.c?h=stable-2.2#n185
[3]
http://git.savannah.gnu.org/cgit/guile.git/tree/libguile/foreign.c?h=stable-2.2#n1080

-- 
Isaac Jurado

"The noblest pleasure is the joy of understanding"
Leonardo da Vinci

[-- Attachment #2: Type: text/html, Size: 2523 bytes --]

Re: C calling Scheme and garbage collection

[Greg Troxel]

Isaac Jurado <diptongo@gmail.com> writes:

> Hello,
>
> I'm playing with event loop libraries implemented in C (libev, libevent,
> etc... in my case libsystemd), but configuring them from Guile.
>
> The qsort example in the documentation [1] seems safe because the qsort C
> function directly calls back, so the callback Scheme bindings stay
> referenced (by the Scheme code calling qsort) during all the C code
> execution.
>
> Now, in C event loops the situation is different.  There is one call to
> configure the event callback, in which the function and data pointers are
> lent to the loop; and then there is the main loop or the single iteration
> call.
>
> The way I see it, suppose I add a timer.  I call one C function passing a
> (proceudre->pointer) and an (scm->pointer).  In a future time, those
> pointers will be used by the C event loop.  If a garbage collection happens
> in the middle, the results of (procedure->pointer) and (scm->pointer) may
> have been reclaimed by the time the C event loop calls back.

I have been down this path before, with guile and with lua.  Basically,
if C (or non-scheme) has a pointer to a scheme object, then you need to
hold a logical reference for it and protect the scheme object, and when
the C pointer is dropped decrease the refcnt.

I am unclear on the details of how you have a ref that gc is made aware
of.  One way is to have a scheme array of the object and a count, and
have the code null out the object when the count goes to zero or
something like that.  But the point is that you need to have  a proxy in
the scheme world, visible to gc, when a pointer to a scheme object is
held outside of the scheme world.

Forcing gc is not going to be reliable.   If you have a reliable scheme,
gc can happen at any random time and things will be ok.

Re: C calling Scheme and garbage collection

[Isaac Jurado]

[-- Attachment #1.1: Type: text/plain, Size: 1822 bytes --]

On Thu, Jun 27, 2019 at 9:52 PM Greg Troxel <gdt@lexort.com> wrote:

>
> I have been down this path before, with guile and with lua.  Basically,
> if C (or non-scheme) has a pointer to a scheme object, then you need to
> hold a logical reference for it and protect the scheme object, and when
> the C pointer is dropped decrease the refcnt.
>
> I am unclear on the details of how you have a ref that gc is made aware
> of.  One way is to have a scheme array of the object and a count, and
> have the code null out the object when the count goes to zero or
> something like that.  But the point is that you need to have  a proxy in
> the scheme world, visible to gc, when a pointer to a scheme object is
> held outside of the scheme world.
>

That's more or less what I had in mind, although instead of an array I
would use a hash table indexed by a fundamental type (e.g. integer) which
can be converted painlessly between Scheme and C.


> Forcing gc is not going to be reliable.   If you have a reliable scheme,
> gc can happen at any random time and things will be ok.
>

I prepared a minimal case of the kind of C interactions that I'm trying.
I'm attaching the files, the C code has to be compiled with:

gcc -shared -fPIC -o mysalsa.so mysalsa.c

Running the Scheme script yields something like the following:

Captured: Closure without collection
Argument: noitcelloc tuohtiw erusolC
Captured: Closure with garbate collected
Argument:

So primitive values seem to be garbage collected, but closures are treated
slightly differently.  This is very interesting, since working with
closures eliminates the need of those common "void *userdata" extra
arguments.

Testing continuation is going to be interesting too.

Best regards.

-- 
Isaac Jurado

"The noblest pleasure is the joy of understanding"
Leonardo da Vinci

[-- Attachment #1.2: Type: text/html, Size: 2630 bytes --]

[-- Attachment #2: mysalsa.c --]
[-- Type: text/x-csrc, Size: 257 bytes --]

typedef void (*callback_t) (void *);

static callback_t saved_callback;
static void *saved_arg;

void give (callback_t callback, void *arg)
{
        saved_callback = callback;
        saved_arg = arg;
}

void call ()
{
        saved_callback(saved_arg);
}

[-- Attachment #3: mysalsa.scm --]
[-- Type: text/x-scheme, Size: 755 bytes --]

(use-modules (system foreign))

(define lib (dynamic-link (string-append (getcwd) "/mysalsa")))
(define give (pointer->procedure void (dynamic-func "give" lib) '(* *)))
(define call (pointer->procedure void (dynamic-func "call" lib) '()))

(define (simple-closure capture)
  (let ([proc (lambda (arg)
                (display (string-append "Captured: " capture))
                (newline)
                (display (string-append "Argument: " (pointer->string arg)))
                (newline))]
        [arg (string-reverse capture)])
    (give (procedure->pointer void proc '(*)) (string->pointer arg))))


(simple-closure "Closure without collection")
(call)
(gc)

(simple-closure "Closure with garbate collected")
(gc)
(call)


             
  




  

Re: C calling Scheme and garbage collection

[Greg Troxel]

Isaac Jurado <diptongo@gmail.com> writes:

> On Thu, Jun 27, 2019 at 9:52 PM Greg Troxel <gdt@lexort.com> wrote:
>
>> I have been down this path before, with guile and with lua.  Basically,
>> if C (or non-scheme) has a pointer to a scheme object, then you need to
>> hold a logical reference for it and protect the scheme object, and when
>> the C pointer is dropped decrease the refcnt.
>>
>> I am unclear on the details of how you have a ref that gc is made aware
>> of.  One way is to have a scheme array of the object and a count, and
>> have the code null out the object when the count goes to zero or
>> something like that.  But the point is that you need to have  a proxy in
>> the scheme world, visible to gc, when a pointer to a scheme object is
>> held outside of the scheme world.
>>
>
> That's more or less what I had in mind, although instead of an array I
> would use a hash table indexed by a fundamental type (e.g. integer) which
> can be converted painlessly between Scheme and C.

Sure - it just needs to be something the gc will find.

> I prepared a minimal case of the kind of C interactions that I'm trying.
> I'm attaching the files, the C code has to be compiled with:
>
> gcc -shared -fPIC -o mysalsa.so mysalsa.c
>
> Running the Scheme script yields something like the following:
>
> Captured: Closure without collection
> Argument: noitcelloc tuohtiw erusolC
> Captured: Closure with garbate collected
> Argument:
>
> So primitive values seem to be garbage collected, but closures are treated
> slightly differently.  This is very interesting, since working with
> closures eliminates the need of those common "void *userdata" extra
> arguments.
>
> Testing continuation is going to be interesting too.

Just because something wasn't collected doesn't mean it is safe.  You
don't actually know that the closure wasn't garbage collected, just that
when you used it the bits were still there.  You might try running under
valgrind.  Or modify guile to clear all objects that are garbage
collected, which I'm guessing it doesn't do.

In my experience, it's definitely not ok to capture a pointer to a
scheme object and store it for later use without protecting the scheme
object from gc by holding a reference.

Re: C calling Scheme and garbage collection

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 1423 bytes --]

Hello,

>...
> In my experience, it's definitely not ok to capture a pointer to a
> scheme object and store it for later use without protecting the scheme
> object from gc by holding a reference.

I recently had a similar conversation, here is one of the emails of the thread, the
one that lists a series of examples, and out of those, only the last example works:

	https://lists.gnu.org/archive/html/guile-devel/2019-03/msg00013.html

In that last example, I am holding a reference to both the data and their pointer,
but it seemed to me, after further local tests, that it is sufficient to hold on a
reference to the pointers, not the pointers and the data:

scheme@(guile-user)> ,use (system foreign)
scheme@(guile-user)> (define ptr-1 (string->pointer "Hello"))
scheme@(guile-user)> (define ptr-2 (string->pointer "there!"))
scheme@(guile-user)> (make-c-struct (list '* '*) (list ptr-1 ptr-2))
$2 = #<pointer 0x5623b599a350>
scheme@(guile-user)> (parse-c-struct $2 (list '* '*))
$3 = (#<pointer 0x5623b51b1110> #<pointer 0x5623b591e6f0>)
scheme@(guile-user)> (map pointer->string $3)
$4 = ("Hello" "there!")
scheme@(guile-user)> (gc)
scheme@(guile-user)> (map pointer->string $3)
$5 = ("Hello" "there!")

Is this correct, or am I just lucky here? What is certain is that holding a
reference to the data (only) does not work, as shown in the examples
in the 'pointed' email ...

David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: C calling Scheme and garbage collection

[Hans Åberg]

> On 27 Jun 2019, at 21:52, Greg Troxel <gdt@lexort.com> wrote:
> 
> I have been down this path before, with guile and with lua.  Basically,
> if C (or non-scheme) has a pointer to a scheme object, then you need to
> hold a logical reference for it and protect the scheme object, and when
> the C pointer is dropped decrease the refcnt.
> 
> I am unclear on the details of how you have a ref that gc is made aware
> of.

If one allocates with the underlying Boehm GC, is the SCM object kept alive?

Re: C calling Scheme and garbage collection

[Isaac Jurado]

[-- Attachment #1: Type: text/plain, Size: 2255 bytes --]

On Sat, Jun 29, 2019 at 7:44 PM Greg Troxel <gdt@lexort.com> wrote:

> Isaac Jurado <diptongo@gmail.com> writes:
>
> > On Thu, Jun 27, 2019 at 9:52 PM Greg Troxel <gdt@lexort.com> wrote:
> >
> >> I have been down this path before, with guile and with lua.  Basically,
> >> if C (or non-scheme) has a pointer to a scheme object, then you need to
> >> hold a logical reference for it and protect the scheme object, and when
> >> the C pointer is dropped decrease the refcnt.
> >>
> >> I am unclear on the details of how you have a ref that gc is made aware
> >> of.  One way is to have a scheme array of the object and a count, and
> >> have the code null out the object when the count goes to zero or
> >> something like that.  But the point is that you need to have  a proxy in
> >> the scheme world, visible to gc, when a pointer to a scheme object is
> >> held outside of the scheme world.
> >>
> >
> > That's more or less what I had in mind, although instead of an array I
> > would use a hash table indexed by a fundamental type (e.g. integer) which
> > can be converted painlessly between Scheme and C.
>
> Sure - it just needs to be something the gc will find.
>

My intention was quite the opposite, use something that the GC will NOT
find, like 32/64 bit integers.  I assumed that (pointer->procedure) and its
inverse would convert from/to uint32, uint64, etc. natively, without
creating GC objects.  This way I can safely give a way such a value to the
C code, while using it as a key to a Scheme hashmap.


> Just because something wasn't collected doesn't mean it is safe.  You
> don't actually know that the closure wasn't garbage collected, just that
> when you used it the bits were still there.  You might try running under
> valgrind.  Or modify guile to clear all objects that are garbage
> collected, which I'm guessing it doesn't do.
>

Agreed, just like the old use after free().


> In my experience, it's definitely not ok to capture a pointer to a
> scheme object and store it for later use without protecting the scheme
> object from gc by holding a reference.
>

It's the sort of advice I was seeking to confirm, so thanks :-)

Best regards.

-- 
Isaac Jurado

"The noblest pleasure is the joy of understanding"
Leonardo da Vinci

[-- Attachment #2: Type: text/html, Size: 3347 bytes --]

Re: C calling Scheme and garbage collection

[Mark H Weaver]

Hi Isaac,

Isaac Jurado <diptongo@gmail.com> writes:

> I'm playing with event loop libraries implemented in C (libev,
> libevent, etc... in my case libsystemd), but configuring them from
> Guile.
>
> The qsort example in the documentation [1] seems safe because the
> qsort C function directly calls back, so the callback Scheme bindings
> stay referenced (by the Scheme code calling qsort) during all the C
> code execution.
>
> Now, in C event loops the situation is different.  There is one call
> to configure the event callback, in which the function and data
> pointers are lent to the loop; and then there is the main loop or the
> single iteration call.
>
> The way I see it, suppose I add a timer.  I call one C function
> passing a (proceudre->pointer) and an (scm->pointer).  In a future
> time, those pointers will be used by the C event loop.  If a garbage
> collection happens in the middle, the results of (procedure->pointer)
> and (scm->pointer) may have been reclaimed by the time the C event
> loop calls back.

That's right.  When passing pointers to GC-allocated blocks to arbitrary
C code, you must take care to keep GC-visible references to those
pointers until they are no longer needed.  This will not happen
automatically, unless the C code was specifically written to ensure that
the references will be visible to the GC.

BDW-GC does *not* scan arbitrary C data structures.  In particular,
blocks allocated using C 'malloc' or C++ 'new' are not scanned.

> However, I've tried forcing (gc) between the two steps mentioned and
> it looks to be working fine.

As Greg mentioned, this isn't a sufficient test.

In general, I would avoid relying on experiments to determine what is
safe to do.  Experiments can only tell you how the tested version(s)
behave.  They cannot give you assurances about how future versions are
supposed to behave.

> I have also reviewed some of the code [2][3] and some additional weak
> references seem to be created.

Those weak references do not eliminate the need to keep GC-visible
references to the relevant objects.  They merely ensure that *if* you
keep a reference to the returned pointer, the other associated data
structures are also kept alive.

     Regards,
       Mark