doco scm_remember_upto_1

doco scm_remember_upto_1

[Kevin Ryde]

In the "Type checking" node of the manual showing the clear_image
example smob code, should that function have an scm_remember_upto_1
just before returning, in order to keep image_smob visible to a
garbage collect that might run in another thread?

I was going to write some words about scm_remember_upto_1.  I was
going to have a new "map_image" to illustrate the point, then wondered
whether clear_image ought already show it.  Either that or it's an
example of when not needed.

I'm assuming since scm_remember_upto_1 is mentioned in the 1.6 NEWS
that it's a documented (or meant to be documented) feature.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: doco scm_remember_upto_1

[Rob Browning]

Kevin Ryde <user42@zip.com.au> writes:

> In the "Type checking" node of the manual showing the clear_image
> example smob code, should that function have an scm_remember_upto_1
> just before returning, in order to keep image_smob visible to a
> garbage collect that might run in another thread?

(I'm looking at the 1.6 docs)

Marius may need to comment here, but I was under the impression that
at the moment we haven't been trying to write all the code to
accomodate a C-side preemptive GC.  This means that you don't need a
"remember" if there are no intervening calls (usually just scheme
calls) that could cause a GC.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org; previously @cs.utexas.edu
GPG starting 2002-11-03 = 14DD 432F AE39 534D B592  F9A0 25C8 D377 8C7E 73A4


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: doco scm_remember_upto_1

[Marius Vollmer]

Rob Browning <rlb@defaultvalue.org> writes:

> Kevin Ryde <user42@zip.com.au> writes:
> 
> > In the "Type checking" node of the manual showing the clear_image
> > example smob code, should that function have an scm_remember_upto_1
> > just before returning, in order to keep image_smob visible to a
> > garbage collect that might run in another thread?

Yes, that would be an improvement.  As Rob say, you are safe right
now, but having one scm_remember_upto_here too many does not hurt.

> Marius may need to comment here, but I was under the impression that
> at the moment we haven't been trying to write all the code to
> accomodate a C-side preemptive GC.

Yes, but I'd say that for new code we could be more careful with the
scm_remembers.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: doco scm_remember_upto_1

[Rob Browning]

Marius Vollmer <marius.vollmer@uni-dortmund.de> writes:

> Yes, that would be an improvement.  As Rob say, you are safe right
> now, but having one scm_remember_upto_here too many does not hurt.

I presume they affect the available optimizations, but I suppose the
effect of "too many" would still be likely to be minimal...

>> Marius may need to comment here, but I was under the impression that
>> at the moment we haven't been trying to write all the code to
>> accomodate a C-side preemptive GC.
>
> Yes, but I'd say that for new code we could be more careful with the
> scm_remembers.

Hmm.  So what level of preemption are we planning to support, or
rather for what kind of preemption should we be coding defensively?  I
ask because it's been my experience that trying to write libraries to
support fine grained preemption can sometimes be expensive, both with
respect to coding and runtime.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org; previously @cs.utexas.edu
GPG starting 2002-11-03 = 14DD 432F AE39 534D B592  F9A0 25C8 D377 8C7E 73A4


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: doco scm_remember_upto_1

[Kevin Ryde]

Rob Browning <rlb@defaultvalue.org> writes:
>
> So what level of preemption are we planning to support, or
> rather for what kind of preemption should we be coding defensively?

Some guidance in the smob section of the manual will no doubt be
vital, one way or another, for authors of add-ons.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: doco scm_remember_upto_1

[Kevin Ryde]

I'd like to propose the words below, assuming they're true.  Perhaps
experts can review the last para in particular, and decide how much
can or should be said about gc and pre-emption.

I don't think the varargs scm_remember_upto_here should be documented,
because I don't think it can be implemented as an inline.  I guess it
appeared in the NEWS file though, so probably has to go in the
deprecated functions chapter.  (Likewise the old scm_remember I
suppose.)


Remembering During Operations
-----------------------------

It's important that a smob is visible to the garbage collector whenever
its contents are being accessed.  Otherwise it could be freed while
code is still using it.

 - C Macro: void scm_remember_upto_here_1 (SCM obj)
 - C Macro: void scm_remember_upto_here_2 (SCM obj1, SCM obj2)
     Create a reference to the given object or objects, so they're
     certain to be present on the stack or in a register up to the
     point of these macro calls, and hence won't be freed by the
     garbage collector before then.

For example consider a procedure to convert image data to a list of
pixel values.

     SCM
     image_to_list (SCM image_smob)
     {
       struct image *image;
       SCM lst;
       int i;
       SCM_ASSERT (SCM_SMOB_PREDICATE (image_tag, image_smob),
                   image_smob, SCM_ARG1, "image->list");
     
       image = (struct image *) SCM_SMOB_DATA (image_smob);
       lst = SCM_EOL;
       for (i = image->width * image->height - 1; i >= 0; i--)
         lst = scm_cons (SCM_MAKINUM (image->pixels[i]), lst);
     
       scm_remember_upto_here_1 (image_smob);
       return lst;
     }

   In the loop, only the `image' pointer is used and the C compiler has
no reason to keep the `image_smob' `SCM' anywhere.  The use of
`scm_remember_upto_here_1' however forces it to be kept until that
point, thereby ensuring there's no danger of the smob being freed and
the loop accessing freed data.

   It's not necessary to do the same for the `lst' `SCM', since that's
the return value and the compiler will therefore certainly keep it in a
register or somewhere throughout the routine.

   The previous `clear_image' example (*note Type checking::) did not
use `scm_remember_upto_here_1'.  This is because it didn't do anything
that allocated memory and hence won't trigger the garbage collector.
`image_to_list' on the other hand uses `scm_cons', which will garbage
collect when out of memory, hence requiring protection for `image_smob'.

   It's only in quite rare circumstances that a missing
`scm_remember_upto_here_1' will bite, but when it happens the
consequences are serious.  Fortunately the rule is simple: ensure the
`SCM' of a smob is referenced somewhere past all accesses to its
insides.  Do this by adding an `scm_remember_upto_here_1' if there's no
other references.

   In a multi-threaded program, it might be thought that a garbage
collect could occur in another thread at any time, and hence mean a
routine like `clear_image' would require protection too.  But currently
this is not the case, pre-emption is restricted.  Perhaps, however,
this will change in the future.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: doco scm_remember_upto_1

[Marius Vollmer]

Rob Browning <rlb@defaultvalue.org> writes:

> Hmm.  So what level of preemption are we planning to support, or
> rather for what kind of preemption should we be coding defensively?

Well, the situation in CVS HEAD right now is this

   Multiple threads can use Guile concurrently but must follow some
   rules.  A thread can either be in 'Guile mode' or not.

    - When it is not in Guile mode it can not make any calls to
      libguile functions (other than those that explicitely allow
      this)

    - When it is in Guile mode, it can use libguile functions but it
      must cooperate with all other threads that are in Guile mode.
      This means that it must call libguile functions (that do
      consing?)  'regularily' since the global GC can only run from
      within such a function.  When long computations are to be done
      that don't require calls into libguile, either drop out of Guile
      mode or call SCM_TICK at reasonable intervals.

    - Local variables that are created while not being in Guile mode
      are not scanned by the GC.  However, local variables that are
      created while being in Guile mode are continued to being scanned
      also when the thread leaves Guile mode.  Local SCM variables
      that have been created in Guile mode must not be modified while
      being in non-Guile mode.

      That is, you can imagine the stack of a thread to be partitioned
      into alternating Guile mode and non-Guile mode segements.  Only
      SCM references in the Guile segments are being found and they
      must remain constant.  (Requiring them to remain constant is no
      big constraint: threads shouldnot be manipulating SCM objects
      anyway when not being in Guile mode.)

    - When being in Guile mode, all the strange control flow things
      can happen, like exceptions, continuations, asyncs, etc.  Be
      prepared.  This can not happen in non-Guile mode.


I think this is reasonable and it is not necessary to make the GC more
preemptive and then remove the SCM_TICK requirement.  Such a SCM_TICK
is good for asyncs and amybe other things.  When a thread doesn't want
this, it should drop out of Guile mode.

(There are no functions for entering/leaving Guile mode yet...)

So the take-home point is probably that one should deal with whole SCM
objects whenever possible.  When peeking into smobs, care must be
taken, but not on a instruction-by-instruction level, but only from
libguile-function to libguile-function.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: doco scm_remember_upto_1

[Marius Vollmer]

Kevin Ryde <user42@zip.com.au> writes:

> I'd like to propose the words below, assuming they're true.  Perhaps
> experts can review the last para in particular, and decide how much
> can or should be said about gc and pre-emption.

We can make the guarantee that a GC wont run asynchronously.  What
abut this wording:

       In a multi-threaded program, it might be thought that a garbage
    collection could occur in another thread at any time, and hence
    mean a routine like `clear_image' would require protection too.
    But this is not the case, a GC will only run when all threads are
    in a safe place and have been stopped.  But when in doubt, be
    conservative: include the call to scm_remember_upto_here_1 when
    you are not sure that it is safe to leave it out.  A call to
    scm_remember_1 will cost at most as much as a call to an empty
    function.

> I don't think the varargs scm_remember_upto_here should be documented,
> because I don't think it can be implemented as an inline.

Is that important?  A note that says that the non-varargs functions
might be more efficient should suffice, no?


The rest is very nice!

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: doco scm_remember_upto_1

[Kevin Ryde]

Marius Vollmer <mvo@zagadka.de> writes:
>
>     But this is not the case, a GC will only run when all threads are
>     in a safe place and have been stopped.

Right, yep.  I reworded it in a more direct sense as follows, dropping
the "it might be thought" business, which was only really me getting
wrong ideas.

   In a multi-threaded program, the rule is the same.  As far as a
   given thread is concerned, a garbage collect still only occurs
   within a memory allocation function, not at an arbitrary time.
   (Guile waits for all threads to reach a memory function, and holds
   them there while the collector runs.)

>     But when in doubt, be
>     conservative: include the call to scm_remember_upto_here_1 when
>     you are not sure that it is safe to leave it out.

I think I'd rather see something unambiguous said about when a
remember must be used, instead of talking about being unsure.  I tried
to reword a little to emphasise it's memory activity which provokes a
gc.

>     A call to
>     scm_remember_1 will cost at most as much as a call to an empty
>     function.

Or less, if the asm block approach is right, or if storing to a global
"volatile" variable would similarly be right.  :-)

> Is that important?  A note that says that the non-varargs functions
> might be more efficient should suffice, no?

I'm only thinking to encourage the best way.  I suppose the cost is
always going to be small compared to real work done, it's just a case
of not wanting to go along an inferior path.

In any event, I checked in the new section.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: doco scm_remember_upto_1

[Marius Vollmer]

Kevin Ryde <user42@zip.com.au> writes:

>    In a multi-threaded program, the rule is the same.  As far as a
>    given thread is concerned, a garbage collect still only occurs
>    within a memory allocation function, not at an arbitrary time.
>    (Guile waits for all threads to reach a memory function, and holds
>    them there while the collector runs.)

Don't say "memory allocation function", that is too specific.  A GC
can occur in any libguile function, since all of them might eventually
allocate memory or run a async that does so, or simply invoke SCM_TICK
which is also a safe point for stopping a thread.

> >     But when in doubt, be
> >     conservative: include the call to scm_remember_upto_here_1 when
> >     you are not sure that it is safe to leave it out.
> 
> I think I'd rather see something unambiguous said about when a
> remember must be used, instead of talking about being unsure.  I tried
> to reword a little to emphasise it's memory activity which provokes a
> gc.

Hmm, the most concrete I can think of is:

  If there is a call to any libguile function after you have extracted
  the innards of some SCM object (from a smob, or with
  SCM_STRING_CHARS, etc.) you can't count on that innard still being
  there unless the associated SCM is used again later on (and that use
  is not being optimized away), if only by virtue of appearing in a
  call to scm_remember_upto_here.

The "doubt" would come when one isn't sure whether a libguile function
is being called because one doesn't know what kind of stuff third
party functions do.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: doco scm_remember_upto_1

[Kevin Ryde]

Marius Vollmer <mvo@zagadka.de> writes:
>
> Don't say "memory allocation function", that is too specific.

Ah, good.  I changed to "guile library function".

> The "doubt" would come when one isn't sure whether a libguile function
> is being called because one doesn't know what kind of stuff third
> party functions do.

Right.  I hope "calling a Guile library function or doing something
that might" can cover that.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: doco scm_remember_upto_1

[Marius Vollmer]

Kevin Ryde <user42@zip.com.au> writes:

> > The "doubt" would come when one isn't sure whether a libguile function
> > is being called because one doesn't know what kind of stuff third
> > party functions do.
> 
> Right.  I hope "calling a Guile library function or doing something
> that might" can cover that.

Yes, definitely.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel