Need for scm_remember_upto_here_* in guile-2.0

Need for scm_remember_upto_here_* in guile-2.0

[Chris Vine]

Hi,

The guile manual suggests (section 5.5.5) that  if a smob's internals
are referenced in a function after the last time that the SCM variable
representing the smob is accessed in the function,
scm_remember_upto_here_* should be used.  Otherwise the suggestion is
made that the optimizer may treat the smob as expended after the last
access to its variable is made in the function, with the result that
the collector may reclaim its memory, including any internals (in the
case of the example in section 5.5.5, the smob's 'image' struct).

scm_remember_upto_here_1() does this by the simple expedient of taking
the SCM object as an argument and doing nothing, so the garbage
collector still thinks the smob is in use until then.  (This is fine as
far as it goes although could perhaps be tripped up by
link-time-optimization).

However I am finding difficulty in understanding why that should be the
case, because in that example all the smob's internals are allocated
using scm_gc_malloc()/GC_MALLOC or scm_gc_malloc_pointerless()/
GC_MALLOC_ATOMIC.  The garbage collector should therefore be fully
aware of those internals and be able to mark them for survival if still
in use, without any help from the programmer.

Is the point here that the example code contains an unnecessary (and
ill-advised) free function for the smob, which calls scm_gc_free() on
its internals?  If so, is the point that the scm_remember_upto_here_*
functions are in practice only needed for smobs which provide their own
free function?

If so, that would further imply (in relation to the 'image' example)
that finding a live managed pointer allocated by scm_gc_malloc() in the
smob's data field (as passed when calling scm_new_smob()) does not save
the smob from collection by the garbage collector (and so its free
function being invoked by guile).  Is that really the case?

Regards,

Chris

Re: Need for scm_remember_upto_here_* in guile-2.0

[Ludovic Courtès]

Hello,

In practice scm_remember_upto_here_* is useless with libgc, since libgc
also tracks pointers in registers.

scm_gc_free should not be used, unless you really know what you are
doing (it must be passed a valid pointer to the beginning of a
GC-allocated area) and have a real need for it (like you know you’re
freeing a big memory region and waiting for it to be collected would be
impractical.)

Ludo’.

Re: Need for scm_remember_upto_here_* in guile-2.0

[David Kastrup]

ludo@gnu.org (Ludovic Courtès) writes:

> Hello,
>
> In practice scm_remember_upto_here_* is useless with libgc, since libgc
> also tracks pointers in registers.

Sounds like a big non-sequitur.  scm_remember_upto_here_* is for keeping
SCM values active (including in registers).  Any conversion of an SCM
value to some other data depending on it will not protect the SCM value
as such and may lead to collection/finalization of the SCM cell (calling
guardians, removing from weak hashtables and so on).

-- 
David Kastrup

Re: Need for scm_remember_upto_here_* in guile-2.0

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> In practice scm_remember_upto_here_* is useless with libgc, since libgc
>> also tracks pointers in registers.
>
> Sounds like a big non-sequitur.  scm_remember_upto_here_* is for keeping
> SCM values active (including in registers).  Any conversion of an SCM
> value to some other data depending on it will not protect the SCM value
> as such and may lead to collection/finalization of the SCM cell (calling
> guardians, removing from weak hashtables and so on).

I think David is right.  Consider the following case: a SMOB contains a
pointer to a block allocated by 'malloc' and freed by the SMOB freeing
procedure.  Some C code starts with a SCM value for that SMOB, uses it
to obtain a pointer to the malloc'd block, and then discards the SCM
value and proceeds to do a long-running job on the malloc'd block.  If
there are no other references to the SCM value, Boehm GC can free the
SMOB and call its finalizer, which frees the malloc'd block.  The
long-running job in C continues to access the freed block.

Am I missing something?

      Mark

Re: Need for scm_remember_upto_here_* in guile-2.0

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> In practice scm_remember_upto_here_* is useless with libgc, since libgc
>>> also tracks pointers in registers.
>>
>> Sounds like a big non-sequitur.  scm_remember_upto_here_* is for keeping
>> SCM values active (including in registers).  Any conversion of an SCM
>> value to some other data depending on it will not protect the SCM value
>> as such and may lead to collection/finalization of the SCM cell (calling
>> guardians, removing from weak hashtables and so on).
>
> I think David is right.  Consider the following case: a SMOB contains a
> pointer to a block allocated by 'malloc' and freed by the SMOB freeing
> procedure.  Some C code starts with a SCM value for that SMOB, uses it
> to obtain a pointer to the malloc'd block, and then discards the SCM
> value and proceeds to do a long-running job on the malloc'd block.  If
> there are no other references to the SCM value, Boehm GC can free the
> SMOB and call its finalizer, which frees the malloc'd block.  The
> long-running job in C continues to access the freed block.
>
> Am I missing something?

Well, the underlying idea was probably not to free any memory in the
finalizer and have the malloc'd block be independently protected/freed
by libgc.  If you do things like that, you don't have any structural
integrity associated with an SCM value.  Its various components will be
independently protected from low-level collection by libgc and the SCM
value might be gone/finalized considerably before processing on parts of
it ends.

At any rate, the statement at issue was

>>> In practice scm_remember_upto_here_* is useless with libgc, since
>>> libgc also tracks pointers in registers.

and registers containing SCM values were already tracked in version 1.8
(since registers are saved in the stack frame when functions possibly
invoking a garbage collection are called, tracking SCM values in
registers is not a challenge separate from marking the whole stack).
scm_remember_upto_here_* is not about SCM values being in registers but
rather about the compiler not bothering to keep them around at all.

-- 
David Kastrup

Re: Need for scm_remember_upto_here_* in guile-2.0

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> David Kastrup <dak@gnu.org> writes:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> In practice scm_remember_upto_here_* is useless with libgc, since libgc
>>> also tracks pointers in registers.
>>
>> Sounds like a big non-sequitur.  scm_remember_upto_here_* is for keeping
>> SCM values active (including in registers).  Any conversion of an SCM
>> value to some other data depending on it will not protect the SCM value
>> as such and may lead to collection/finalization of the SCM cell (calling
>> guardians, removing from weak hashtables and so on).
>
> I think David is right.  Consider the following case: a SMOB contains a
> pointer to a block allocated by 'malloc' and freed by the SMOB freeing
> procedure.  Some C code starts with a SCM value for that SMOB, uses it
> to obtain a pointer to the malloc'd block, and then discards the SCM
> value and proceeds to do a long-running job on the malloc'd block.  If
> there are no other references to the SCM value, Boehm GC can free the
> SMOB and call its finalizer, which frees the malloc'd block.  The
> long-running job in C continues to access the freed block.

Indeed, that’s a valid scenario.

My “in practice” assumed no finalizers and only GC-managed memory,
I guess.

Ludo’.

Re: Need for scm_remember_upto_here_* in guile-2.0

[Chris Vine]

On Tue, 22 Sep 2015 23:13:45 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> Mark H Weaver <mhw@netris.org> skribis:
> 
> > David Kastrup <dak@gnu.org> writes:
> >
> >> ludo@gnu.org (Ludovic Courtès) writes:
> >>
> >>> In practice scm_remember_upto_here_* is useless with libgc, since
> >>> libgc also tracks pointers in registers.
> >>
> >> Sounds like a big non-sequitur.  scm_remember_upto_here_* is for
> >> keeping SCM values active (including in registers).  Any
> >> conversion of an SCM value to some other data depending on it will
> >> not protect the SCM value as such and may lead to
> >> collection/finalization of the SCM cell (calling guardians,
> >> removing from weak hashtables and so on).
> >
> > I think David is right.  Consider the following case: a SMOB
> > contains a pointer to a block allocated by 'malloc' and freed by
> > the SMOB freeing procedure.  Some C code starts with a SCM value
> > for that SMOB, uses it to obtain a pointer to the malloc'd block,
> > and then discards the SCM value and proceeds to do a long-running
> > job on the malloc'd block.  If there are no other references to the
> > SCM value, Boehm GC can free the SMOB and call its finalizer, which
> > frees the malloc'd block.  The long-running job in C continues to
> > access the freed block.
> 
> Indeed, that’s a valid scenario.
> 
> My “in practice” assumed no finalizers and only GC-managed memory,
> I guess.

I think you were trying to answer my question, which was: "... is the
point that the scm_remember_upto_here_* functions are in practice only
needed for smobs which provide their own free function?" (that is, which
have no finalisers). I think you were giving the answer "Yes".

Chris

Re: Need for scm_remember_upto_here_* in guile-2.0

[Chris Vine]

On Wed, 23 Sep 2015 00:35:09 +0100
Chris Vine <chris@cvine.freeserve.co.uk> wrote:
> I think you were trying to answer my question, which was: "... is the
> point that the scm_remember_upto_here_* functions are in practice only
> needed for smobs which provide their own free function?" (that is,
> which have no finalisers). I think you were giving the answer "Yes".

Err, "... which have finalisers".

Assuming this is the case, the guile manual example is rather unhelpful.

Re: Need for scm_remember_upto_here_* in guile-2.0

[David Kastrup]

Chris Vine <chris@cvine.freeserve.co.uk> writes:

> On Wed, 23 Sep 2015 00:35:09 +0100
> Chris Vine <chris@cvine.freeserve.co.uk> wrote:
>> I think you were trying to answer my question, which was: "... is the
>> point that the scm_remember_upto_here_* functions are in practice only
>> needed for smobs which provide their own free function?" (that is,
>> which have no finalisers). I think you were giving the answer "Yes".
>
> Err, "... which have finalisers".
>
> Assuming this is the case, the guile manual example is rather unhelpful.

There are other forms of finalization than giving back memory, like
closing file descriptors, relinquishing a lock or semaphore and so on.
Also, a pointer to an array (rather than something more opaque like SCM)
is much more likely to be subject to strength reduction and address
arithmetic by the compiler, leading to a situation where looping through
some array does not leave a live pointer in registers and stack frame in
a form recognizable by libgc.

-- 
David Kastrup

Re: Need for scm_remember_upto_here_* in guile-2.0

[Chris Vine]

On Wed, 23 Sep 2015 09:26:27 +0200
David Kastrup <dak@gnu.org> wrote:
> Chris Vine <chris@cvine.freeserve.co.uk> writes:
> 
> > On Wed, 23 Sep 2015 00:35:09 +0100
> > Chris Vine <chris@cvine.freeserve.co.uk> wrote:
> >> I think you were trying to answer my question, which was: "... is
> >> the point that the scm_remember_upto_here_* functions are in
> >> practice only needed for smobs which provide their own free
> >> function?" (that is, which have no finalisers). I think you were
> >> giving the answer "Yes".
> >
> > Err, "... which have finalisers".
> >
> > Assuming this is the case, the guile manual example is rather
> > unhelpful.
> 
> There are other forms of finalization than giving back memory, like
> closing file descriptors, relinquishing a lock or semaphore and so on.
> Also, a pointer to an array (rather than something more opaque like
> SCM) is much more likely to be subject to strength reduction and
> address arithmetic by the compiler, leading to a situation where
> looping through some array does not leave a live pointer in
> registers and stack frame in a form recognizable by libgc.

I am not sure if we are on the same page.  I realise that finalisers are
not only concerned with memory.  In fact, because there is proper
garbage collection in guile-2.0, in a well designed smob memory is
likely to be the least common thing they manage.  It so happens that
you set the smob's finalisers (whatever they happen to do) with
scm_set_smob_free() (the "free function" that I referred to).

However, I haven't understood your point about arrays.  If they are
allocated with malloc() you need a finaliser and you may need to call
scm_remember_upto_here_1() to prevent the smob's finaliser from firing
while they are still in use.  If they are allocated with scm_gc_malloc()
you don't, as I understand it.  Were you making a point in addition to
that, and if so could you expand on it for me?

Chris

Re: Need for scm_remember_upto_here_* in guile-2.0

[David Kastrup]

Chris Vine <chris@cvine.freeserve.co.uk> writes:

> On Wed, 23 Sep 2015 09:26:27 +0200
> David Kastrup <dak@gnu.org> wrote:
>
>> Also, a pointer to an array (rather than something more opaque like
>> SCM) is much more likely to be subject to strength reduction and
>> address arithmetic by the compiler, leading to a situation where
>> looping through some array does not leave a live pointer in
>> registers and stack frame in a form recognizable by libgc.
>
> However, I haven't understood your point about arrays.  If they are
> allocated with malloc() you need a finaliser and you may need to call
> scm_remember_upto_here_1() to prevent the smob's finaliser from firing
> while they are still in use.  If they are allocated with
> scm_gc_malloc() you don't, as I understand it.  Were you making a
> point in addition to that, and if so could you expand on it for me?

The point is that libgc may be less reliable with recognizing a typical
C/C++ array still being in use than with SCM values because the compiler
is more likely to mangle array access beyond recognition by stack frame
inspection.

For example, if you index arrays starting from 1 (or have indexing
expressions with similar effects), the actual register from which the
compiler does its indexing may lie strictly outside of the allocated
area for an array.

A compiler may also choose to convert a loop running from indexes 0 to
n-1 to one running from -n to -1 in order to have a nicer terminating
condition.  Again, with strength reduction this may lead to the actual
libgc-recognizable pointer being clear outside of the allocated array.

SCM is defined in a manner where the compiler has far fewer useful
transformations at its disposal for the normal operations.  So the
likelihood of libgc making mistakes is just lower.

-- 
David Kastrup

Re: Need for scm_remember_upto_here_* in guile-2.0

[Chris Vine]

On Wed, 23 Sep 2015 10:54:50 +0200
David Kastrup <dak@gnu.org> wrote:
> Chris Vine <chris@cvine.freeserve.co.uk> writes:
> 
> > On Wed, 23 Sep 2015 09:26:27 +0200
> > David Kastrup <dak@gnu.org> wrote:
> >
> >> Also, a pointer to an array (rather than something more opaque like
> >> SCM) is much more likely to be subject to strength reduction and
> >> address arithmetic by the compiler, leading to a situation where
> >> looping through some array does not leave a live pointer in
> >> registers and stack frame in a form recognizable by libgc.
> >
> > However, I haven't understood your point about arrays.  If they are
> > allocated with malloc() you need a finaliser and you may need to
> > call scm_remember_upto_here_1() to prevent the smob's finaliser
> > from firing while they are still in use.  If they are allocated with
> > scm_gc_malloc() you don't, as I understand it.  Were you making a
> > point in addition to that, and if so could you expand on it for me?
> 
> The point is that libgc may be less reliable with recognizing a
> typical C/C++ array still being in use than with SCM values because
> the compiler is more likely to mangle array access beyond recognition
> by stack frame inspection.
> 
> For example, if you index arrays starting from 1 (or have indexing
> expressions with similar effects), the actual register from which the
> compiler does its indexing may lie strictly outside of the allocated
> area for an array.
> 
> A compiler may also choose to convert a loop running from indexes 0 to
> n-1 to one running from -n to -1 in order to have a nicer terminating
> condition.  Again, with strength reduction this may lead to the actual
> libgc-recognizable pointer being clear outside of the allocated array.
> 
> SCM is defined in a manner where the compiler has far fewer useful
> transformations at its disposal for the normal operations.  So the
> likelihood of libgc making mistakes is just lower.

Your trees look excellent, but your pedagogical approach to the wood is
still a mystery to me.  So are you saying:

(i) libgc's garbage collector is unreliable with arrays allocated with
scm_gc_malloc() (or scm_gc_malloc_pointerless()), so using those
functions for garbage collected arrays should be avoided, and hand
written memory management for arrays using smob finalisers should be
employed instead,

or

(ii) if you are doing hand written memory management for arrays using
smob finalisers instead of using garbage collected arrays, you should
make liberal use of scm_remember_upto_here_* because libgc is
particularly apt to finalise your smob while the arrays are still in
use?

Chris 

Re: Need for scm_remember_upto_here_* in guile-2.0

[David Kastrup]

Chris Vine <chris@cvine.freeserve.co.uk> writes:

> On Wed, 23 Sep 2015 10:54:50 +0200
> David Kastrup <dak@gnu.org> wrote:
>> Chris Vine <chris@cvine.freeserve.co.uk> writes:
>> 
>> > On Wed, 23 Sep 2015 09:26:27 +0200
>> > David Kastrup <dak@gnu.org> wrote:
>> >
>> >> Also, a pointer to an array (rather than something more opaque like
>> >> SCM) is much more likely to be subject to strength reduction and
>> >> address arithmetic by the compiler, leading to a situation where
>> >> looping through some array does not leave a live pointer in
>> >> registers and stack frame in a form recognizable by libgc.
>> >
>> > However, I haven't understood your point about arrays.  If they are
>> > allocated with malloc() you need a finaliser and you may need to
>> > call scm_remember_upto_here_1() to prevent the smob's finaliser
>> > from firing while they are still in use.  If they are allocated with
>> > scm_gc_malloc() you don't, as I understand it.  Were you making a
>> > point in addition to that, and if so could you expand on it for me?
>> 
>> The point is that libgc may be less reliable with recognizing a
>> typical C/C++ array still being in use than with SCM values because
>> the compiler is more likely to mangle array access beyond recognition
>> by stack frame inspection.
>> 
>> For example, if you index arrays starting from 1 (or have indexing
>> expressions with similar effects), the actual register from which the
>> compiler does its indexing may lie strictly outside of the allocated
>> area for an array.
>> 
>> A compiler may also choose to convert a loop running from indexes 0 to
>> n-1 to one running from -n to -1 in order to have a nicer terminating
>> condition.  Again, with strength reduction this may lead to the actual
>> libgc-recognizable pointer being clear outside of the allocated array.
>> 
>> SCM is defined in a manner where the compiler has far fewer useful
>> transformations at its disposal for the normal operations.  So the
>> likelihood of libgc making mistakes is just lower.
>
> Your trees look excellent, but your pedagogical approach to the wood
> is still a mystery to me.  So are you saying:
>
> (i) libgc's garbage collector is unreliable with arrays allocated with
> scm_gc_malloc() (or scm_gc_malloc_pointerless()), so using those
> functions for garbage collected arrays should be avoided, and hand
> written memory management for arrays using smob finalisers should be
> employed instead,

Unreliable with arrays?  That's like defining what ratio of oak trees as
compared to fir trees a wood should have in order to be considered safe
from muggings at times 6am to 20pm.

Because, you know, foliage and light and density of growth.  There are
just no hard numbers.  Yet you can still pick a place where to build a
house.  Or a town.  Is a naturally protected area safe?  That's all a
matter of degrees.

> or
>
> (ii) if you are doing hand written memory management for arrays using
> smob finalisers instead of using garbage collected arrays,

That's not really "hand-written" since it still relies on conservative
stack scanning.  It's just wiring down the libgc work to the closed
subset of SCM marking and collection rather than letting it deal with
everything.  So libgc is working under better-controlled and tested
circumstances.  That relies less on the skills of the libgc programmer
(which may find a hard limit in the realities of code generation) and
more on the skills of the GUILE application programmer.

Of course, given bug #19883
<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19883> it appears that
in this case it is more of a question of the libgc programmers getting
the hard part of collection right while messing up with finalizers.
Fallout from that problem is likely one of the driving factors of GUILE
developers actively recommending practices that amount to not relying on
finalizers for memory management.

GUILE would do better to mark SMOBs as dead, dead, dead (presumably by
setting the type to 0) once it calls the finalizer on it instead of
continuing to call the mark hooks (in effect, we do something comparable
in LilyPond now in order not to mark collected objects).  And/or report
this problem to the libgc authors.

Alas, "This bug report was last modified 205 days ago.".


> you should make liberal use of scm_remember_upto_here_* because libgc
> is particularly apt to finalise your smob while the arrays are still
> in use?

You should be aware what you are doing.  That's all.  One good rule of
thumb is to pass data around as SCM as long as possible and not
dereference the SCM before it is needed.  An SCM in the call frame will
usually be sufficient protection even without scm_remember_upto_here_*
as long as you don't overwrite it with other values.  Or it is passed by
register.  Which is an architecture-dependent thing so you cannot really
rely on code not needing scm_remember_upto_here_* on one platform to run
equally well on another.

No, I cannot give you hard and fast rules for woods.  I still have my
preferences, and there are reasons for them.  And there may be multiple
reasons that are mostly independent.

-- 
David Kastrup

Re: Need for scm_remember_upto_here_* in guile-2.0

[Chris Vine]

On Wed, 23 Sep 2015 14:27:15 +0200
David Kastrup <dak@gnu.org> wrote:
[snip]
> No, I cannot give you hard and fast rules for woods.  I still have my
> preferences, and there are reasons for them.  And there may be
> multiple reasons that are mostly independent.

I wasn't asking for rules.  I was trying to extract your meaning in an
intelligible form.

I think what you are approximately saying is that libgc as a
conservative garbage collector is not particularly good in all cases
(although I suspect that that is more likely to result in leaks rather
than unwanted finalisation), that notwithstanding this it is better at
detecting whether an allocated smob cell is still in use rather than
some other things (of which the allocated data struct kept by the smob
cell and any arrays are an example), and that therefore you think you
are generally better off in letting libgc handle the smob cell with
diligent use of remembering by the programmer, and handle other things
yourself, including the smob's data struct and its members, by
allocating them with malloc() and by explicit deallocation in the
smob's finalisers (which in turn means you need a mark function for any
SCM members of a malloc()'ed smob struct, which doesn't apparently work
correctly).

My modest requirements mean that I do not need to follow your advice,
but even so I am grateful for your views.

Chris