bug#20907: [PATCH] Manual bug for scm_gc_protect_object

bug#20907: [PATCH] Manual bug for scm_gc_protect_object

[Mike Gran]

Manual claims C globals weren't scanned by GC in 1.8.  The opposite
is true.

* doc/ref/api-memory.texi [scm_gc_protect_object]: modified
---
doc/ref/api-memory.texi | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/ref/api-memory.texi b/doc/ref/api-memory.texi
index 0e37d16..3496cc5 100644
--- a/doc/ref/api-memory.texi
+++ b/doc/ref/api-memory.texi
@@ -42,7 +42,7 @@ as it was protected. It is an error to unprotect an object more times
than it has been protected. Returns the SCM object it was passed.

Note that storing @var{obj} in a C global variable has the same
-effect@footnote{In Guile up to version 1.8, C global variables were not
+effect@footnote{In Guile up to version 1.8, C global variables were
scanned by the garbage collector; hence, @code{scm_gc_protect_object}
was the only way in C to prevent a Scheme object from being freed.}.
@end deftypefn
--
2.1.0

bug#20907: [PATCH] Manual bug for scm_gc_protect_object

[Mark H Weaver]

Hi Mike,

Mike Gran <spk121@yahoo.com> writes:
> Manual claims C globals weren't scanned by GC in 1.8.  The opposite
> is true.

Ludovic wrote that text in 2009, commit
f07c349eb38d6c7b160b8980fc4007fb502e3433.

Ludovic, what do you make of this?

> * doc/ref/api-memory.texi [scm_gc_protect_object]: modified
> ---
> doc/ref/api-memory.texi | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/doc/ref/api-memory.texi b/doc/ref/api-memory.texi
> index 0e37d16..3496cc5 100644
> --- a/doc/ref/api-memory.texi
> +++ b/doc/ref/api-memory.texi
> @@ -42,7 +42,7 @@ as it was protected. It is an error to unprotect an object more times
> than it has been protected. Returns the SCM object it was passed.
>
> Note that storing @var{obj} in a C global variable has the same
> -effect@footnote{In Guile up to version 1.8, C global variables were not
> +effect@footnote{In Guile up to version 1.8, C global variables were
> scanned by the garbage collector; hence, @code{scm_gc_protect_object}
> was the only way in C to prevent a Scheme object from being freed.}.

If what you say is true, then this patch would not be sufficient,
because the footnote would not make sense.  If you're right, then the
entire paragraph above should be removed.

     Mark

bug#20907: [PATCH] Manual bug for scm_gc_protect_object

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> Mike Gran <spk121@yahoo.com> writes:
>> Manual claims C globals weren't scanned by GC in 1.8.  The opposite
>> is true.
>
> Ludovic wrote that text in 2009, commit
> f07c349eb38d6c7b160b8980fc4007fb502e3433.

I think the manual is correct: global C variables were *not* scanned by
the GC.  As an example, see ‘scm_sys_protects’ in 1.8: It’s a global
array that was explicitly scanned by the GC, because that’s basically
the only mechanism to add new GC root:

  j = SCM_NUM_PROTECTS;
  while (j--)
    scm_gc_mark (scm_sys_protects[j]);

The 1.8 manual reads:

     Other references to 'SCM' objects, such as global variables of type
  'SCM' or other random data structures in the heap that contain fields of
  type 'SCM', can be made visible to the garbage collector by calling the
  functions 'scm_gc_protect' or 'scm_permanent_object'.  You normally use
  these funtions for long lived objects such as a hash table that is
  stored in a global variable.  For temporary references in local
  variables or function arguments, using these functions would be too
  expensive.

http://www.gnu.org/software/guile/docs/docs-1.8/guile-ref/Garbage-Collection.html

So I think we can close as ‘notabug’?  :-)

Ludo’.

bug#20907: [PATCH] Manual bug for scm_gc_protect_object

[Mike Gran]

[-- Attachment #1: Type: text/plain, Size: 1527 bytes --]

> On Wednesday, September 2, 2015 5:09 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> I think the manual is correct: global C variables were *not* scanned by
> the GC.  As an example, see ‘scm_sys_protects’ in 1.8: It’s a global
> array that was explicitly scanned by the GC, because that’s basically
> the only mechanism to add new GC root:
> 
>   j = SCM_NUM_PROTECTS;
>   while (j--)
>     scm_gc_mark (scm_sys_protects[j]);
> 
> The 1.8 manual reads:
> 
>      Other references to 'SCM' objects, such as global variables of type
>   'SCM' or other random data structures in the heap that contain fields 
> of
>   type 'SCM', can be made visible to the garbage collector by calling 
> the
>   functions 'scm_gc_protect' or 'scm_permanent_object'.  You 
> normally use
>   these funtions for long lived objects such as a hash table that is
>   stored in a global variable.  For temporary references in local
>   variables or function arguments, using these functions would be too
>   expensive.

For what it is worth, the effect that I was seeing that made me
question the documentation can be demonstrated by the attached program,
where two 100MB Guile strings are stored in a C globals: one protected
and one not. 


In 1.8, a GC operation reduces memory from 200MB to 100MB, which I
assume is freeing the memory from the unprotected string.

In 2.0, the heap size always stays at 200MB.


The output of the program for Guile 1.8.8 and Guile 2.0.9 is attached.


Thanks,
Mike

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: main.c --]
[-- Type: text/x-csrc, Size: 729 bytes --]

#include <libguile.h>


#define ONE_HUNDRED_MILLION (100*1000*1000)

SCM protected_string;
SCM unprotected_string;

void inner_main (void *data, int argc, char **argv)
{
  protected_string = scm_c_make_string(ONE_HUNDRED_MILLION, SCM_MAKE_CHAR('1'));
  scm_gc_protect_object (protected_string);

  unprotected_string = scm_c_make_string(ONE_HUNDRED_MILLION, SCM_MAKE_CHAR('2'));

  scm_display(scm_c_eval_string("(gc-stats)"), scm_current_output_port());
  scm_newline(scm_current_output_port());

  scm_gc();

  scm_display(scm_c_eval_string("(gc-stats)"), scm_current_output_port());
  scm_newline(scm_current_output_port());
}

int main (int argc, char **argv)
{
  scm_boot_guile (argc, argv, inner_main, NULL);
  return 0;
}

[-- Attachment #3: out18.txt --]
[-- Type: text/plain, Size: 884 bytes --]

((gc-time-taken . 0) (cells-allocated . 61611) (total-cells-allocated . 61722) (cell-heap-size . 83383) (bytes-malloced . 200081608) (gc-malloc-threshold . 366816085) (gc-times . 4) (gc-mark-time-taken . 0) (cells-marked . 410017.0) (cells-swept . 458490.0) (malloc-yield . 0) (cell-yield . 0) (protected-objects . 1) (cell-heap-segments (162449408 . 162484224) (3076524032 . 3076694016) (3076700160 . 3076900864) (3076909056 . 3077173248)))
((gc-time-taken . 0) (cells-allocated . 61662) (total-cells-allocated . 61797) (cell-heap-size . 83383) (bytes-malloced . 100081878) (gc-malloc-threshold . 366816085) (gc-times . 5) (gc-mark-time-taken . 0) (cells-marked . 493378.0) (cells-swept . 541875.0) (malloc-yield . 0) (cell-yield . 0) (protected-objects . 1) (cell-heap-segments (162449408 . 162484224) (3076524032 . 3076694016) (3076700160 . 3076900864) (3076909056 . 3077173248)))

[-- Attachment #4: out20.txt --]
[-- Type: text/plain, Size: 370 bytes --]

((gc-time-taken . 967) (heap-size . 219127808) (heap-free-size . 16781312) (heap-total-allocated . 202352336) (heap-allocated-since-gc . 156376) (protected-objects . 40) (gc-times . 6))
((gc-time-taken . 971) (heap-size . 219127808) (heap-free-size . 16809984) (heap-total-allocated . 202354168) (heap-allocated-since-gc . 1384) (protected-objects . 40) (gc-times . 7))

bug#20907: [PATCH] Manual bug for scm_gc_protect_object

[Mark H Weaver]

Mike Gran <spk121@yahoo.com> writes:

>> On Wednesday, September 2, 2015 5:09 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> I think the manual is correct: global C variables were *not* scanned by
>> the GC.

> For what it is worth, the effect that I was seeing that made me
> question the documentation can be demonstrated by the attached program,
> where two 100MB Guile strings are stored in a C globals: one protected
> and one not. 
>
>
> In 1.8, a GC operation reduces memory from 200MB to 100MB, which I
> assume is freeing the memory from the unprotected string.

I'm not sure why this result would make you question the current
documentation.  To my mind, it clearly confirms what Ludovic wrote.

If global C variables were scanned by default in 1.8, as you asserted,
then why would the unprotected string have been freed?

Am I missing something?

    Thanks,
      Mark

bug#20907: [PATCH] Manual bug for scm_gc_protect_object

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> Mike Gran <spk121@yahoo.com> writes:
>
>>> On Wednesday, September 2, 2015 5:09 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>>
>>> I think the manual is correct: global C variables were *not* scanned by
>>> the GC.
>
>> For what it is worth, the effect that I was seeing that made me
>> question the documentation can be demonstrated by the attached program,
>> where two 100MB Guile strings are stored in a C globals: one protected
>> and one not. 
>>
>>
>> In 1.8, a GC operation reduces memory from 200MB to 100MB, which I
>> assume is freeing the memory from the unprotected string.
>
> I'm not sure why this result would make you question the current
> documentation.  To my mind, it clearly confirms what Ludovic wrote.
>
> If global C variables were scanned by default in 1.8, as you asserted,
> then why would the unprotected string have been freed?

It occurs to me that this confusion might have arisen from a lack of
knowledge about garbage collection and what it means to "scan"
something.  "Scanning" is more or less a synonym for "marking", where
the reachable objects are first marked starting from the roots, and
after that's done, anything that is not marked will be freed in the
sweep phase.

I wonder if Mike might have been thinking that something cannot be freed
unless it is scanned, so if it is freed that is evidence that it was
scanned.  In fact, scanning (marking) is necessary to *prevent* freeing,
not to enable it.

      Mark

bug#20907: [PATCH] Manual bug for scm_gc_protect_object

[Mike Gran]

> On Wednesday, September 2, 2015 9:16 AM, Mark H Weaver <mhw@netris.org> wrote:
 > If global C variables were scanned by default in 1.8, as you asserted,
> then why would the unprotected string have been freed?
> 
> Am I missing something?

I apologize in advance for the pedantry below.

Maybe I just don't understand GC nomenclature.
 
I thought that if a global variable were not "scanned" by a garbage
collector, it would be invisible to the garbage collector, and could
not be freed by the garbage collector. 
 
The footnote in question says that for 1.8, globals were not
"scanned", which I understood to mean that globals were invisible to
to the garbage collector and could not be freed by the GC.
 
The implication is that, for 2.0, globals were "scanned", which I
understood to mean that they were visible to the GC and could be freed.
 
My test program shows that globals are GC'd in 1.8, but, not 2.0.
 
That is why I believe the logic of the footnote to be wrong, and that the
"not" should be removed.
 
But I guess from context that you and Ludo have a different definition
for "to scan", and that scanning protects something from being GC'd?
 
In the "Garbage Collection" of the manual in both 1.8 and 2.0,
it says that in 1.8, that "global variables of type SCM ... can be made
visible to the garbage collector by calling the functions scm_gc_protect".
(That's a typo I guess.  It should say scm_gc_protect_object, I think.)
 
The implication is that if I do not call
scm_gc_protect_object, my global is still
"invisible" and thus can't be freed by the GC.  But my "invisible" global
in 1.8 is being freed and in 2.0 it is not freed.
 
The only mention of the difference between 1.8 and 2.0 is in the footnote
in question.
 
Look, it is fine, though.  Don't waste your time on this if I'm just
confused.
 
Thanks,
 
Mike 

bug#20907: [PATCH] Manual bug for scm_gc_protect_object

[Mark H Weaver]

Mike Gran <spk121@yahoo.com> writes:

> In the "Garbage Collection" of the manual in both 1.8 and 2.0,
> it says that in 1.8, that "global variables of type SCM ... can be made
> visible to the garbage collector by calling the functions scm_gc_protect".
> (That's a typo I guess.  It should say scm_gc_protect_object, I
> think.)

Indeed, good catch!  Fixed in 4c5788d1ab14550afd86117e96f91164fbe04a72.

> The implication is that if I do not call scm_gc_protect_object, my
> global is still "invisible" and thus can't be freed by the GC.  But my
> "invisible" global in 1.8 is being freed and in 2.0 it is not freed.

Here's the crux of the confusion: it's not the global variable that is
being freed here.  The variable only holds a *reference* to the
heap-allocated string.  That may seem pedantic, but it's a crucial
distinction here.  Anything in the heap that is not referenced from
somewhere visible to the GC is freed.

Would it help to replace all uses of the term "scan" with "mark", in
connection with garbage collection?  In the papers I've read on GC,
"mark" is the word I usually see, and it seems much clearer to me,
because anyone who knows the basics of GC knows that "marking" is needed
to prevent an object from being freed, whereas "scanning" could mean
anything.

If you have other ideas of how to make this more clear, I'm open to
suggestions.

     Thanks!
       Mark

bug#20907: [PATCH] Manual bug for scm_gc_protect_object

[Mike Gran]

On Wednesday, September 2, 2015 11:06 AM, Mark H Weaver <mhw@netris.org> wrote:
 
>Would it help to replace all uses of the term "scan" with "mark", in
>connection with garbage collection?  In the papers I've read on GC,
>"mark" is the word I usually see, and it seems much clearer to me,
>because anyone who knows the basics of GC knows that "marking" is needed
>to prevent an object from being freed, whereas "scanning" could mean
>anything.

That would help, I think.  I guess I was associating "scan" with
the "sweep" part of mark and sweep.
 
Thanks very much for the clarification.
 
Mike

bug#20907: [PATCH] Manual bug for scm_gc_protect_object

[Andy Wingo]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

On Wed 02 Sep 2015 20:34, Mike Gran <spk121@yahoo.com> writes:

> On Wednesday, September 2, 2015 11:06 AM, Mark H Weaver <mhw@netris.org> wrote:
>  
>>Would it help to replace all uses of the term "scan" with "mark", in
>>connection with garbage collection?  In the papers I've read on GC,
>>"mark" is the word I usually see, and it seems much clearer to me,
>>because anyone who knows the basics of GC knows that "marking" is needed
>>to prevent an object from being freed, whereas "scanning" could mean
>>anything.
>
> That would help, I think.  I guess I was associating "scan" with
> the "sweep" part of mark and sweep.
>  
> Thanks very much for the clarification.

I think "scanning" is a term that is used sometimes, like tracing or
visiting.  But it's true that in the case of the manual it was used
without much context.  I made some tweaks that hopefully clarify things,
enough to close this bug anyway :)

Andy


[-- Attachment #2: 0001-Clarify-use-of-the-term-scanning-in-the-manual.patch --]
[-- Type: text/plain, Size: 8391 bytes --]

From f84006c5644997ce9854df9070ec2f1fb8acd420 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@pobox.com>
Date: Fri, 24 Jun 2016 08:56:21 +0200
Subject: [PATCH] Clarify use of the term "scanning" in the manual

* doc/ref/api-memory.texi (Garbage Collection Functions):
* doc/ref/libguile-concepts.texi (Garbage Collection): Attempt to be
  clear that scanning is a thing that happens in the mark phase.  Fixes
  #20907 I think.
---
 doc/ref/api-memory.texi        | 42 ++++++++++++++++++++----------------
 doc/ref/libguile-concepts.texi | 48 ++++++++++++++++++++++++++++--------------
 2 files changed, 56 insertions(+), 34 deletions(-)

diff --git a/doc/ref/api-memory.texi b/doc/ref/api-memory.texi
index 142eb01..ce0187b 100644
--- a/doc/ref/api-memory.texi
+++ b/doc/ref/api-memory.texi
@@ -27,9 +27,10 @@ collection relates to using Guile from C.
 
 @deffn {Scheme Procedure} gc
 @deffnx {C Function} scm_gc ()
-Scans all of SCM objects and reclaims for further use those that are
-no longer accessible.  You normally don't need to call this function
-explicitly.  It is called automatically when appropriate.
+Finds all of the ``live'' @code{SCM} objects and reclaims for further
+use those that are no longer accessible.  You normally don't need to
+call this function explicitly.  Its functionality is invoked
+automatically as needed.
 @end deffn
 
 @deftypefn {C Function} SCM scm_gc_protect_object (SCM @var{obj})
@@ -43,8 +44,9 @@ than it has been protected. Returns the SCM object it was passed.
 
 Note that storing @var{obj} in a C global variable has the same
 effect@footnote{In Guile up to version 1.8, C global variables were not
-scanned by the garbage collector; hence, @code{scm_gc_protect_object}
-was the only way in C to prevent a Scheme object from being freed.}.
+visited by the garbage collector in the mark phase; hence,
+@code{scm_gc_protect_object} was the only way in C to prevent a Scheme
+object from being freed.}.
 @end deftypefn
 
 @deftypefn {C Function} SCM scm_gc_unprotect_object (SCM @var{obj})
@@ -123,16 +125,18 @@ live reference to it@footnote{In Guile up to version 1.8, memory
 allocated with @code{scm_gc_malloc} @emph{had} to be freed with
 @code{scm_gc_free}.}.
 
-Memory allocated with @code{scm_gc_malloc} is scanned for live pointers.
-This means that if @code{scm_gc_malloc}-allocated memory contains a
-pointer to some other part of the memory, the garbage collector notices
-it and prevents it from being reclaimed@footnote{In Guile up to 1.8,
-memory allocated with @code{scm_gc_malloc} was @emph{not} scanned.
-Consequently, the GC had to be told explicitly about pointers to live
-objects contained in the memory block, e.g., @i{via} SMOB mark functions
-(@pxref{Smobs, @code{scm_set_smob_mark}})}.  Conversely, memory
-allocated with @code{scm_gc_malloc_pointerless} is assumed to be
-``pointer-less'' and is not scanned.
+When garbage collection occurs, Guile will visit the words in memory
+allocated with @code{scm_gc_malloc}, looking for live pointers.  This
+means that if @code{scm_gc_malloc}-allocated memory contains a pointer
+to some other part of the memory, the garbage collector notices it and
+prevents it from being reclaimed@footnote{In Guile up to 1.8, memory
+allocated with @code{scm_gc_malloc} was @emph{not} visited by the
+collector in the mark phase.  Consequently, the GC had to be told
+explicitly about pointers to live objects contained in the memory block,
+e.g., @i{via} SMOB mark functions (@pxref{Smobs,
+@code{scm_set_smob_mark}})}.  Conversely, memory allocated with
+@code{scm_gc_malloc_pointerless} is assumed to be ``pointer-less'' and
+is not scanned for pointers.
 
 For memory that is not associated with a Scheme object, you can use
 @code{scm_malloc} instead of @code{malloc}.  Like
@@ -193,9 +197,11 @@ Allocate @var{size} bytes of automatically-managed memory.  The memory
 is automatically freed when no longer referenced from any live memory
 block.
 
-Memory allocated with @code{scm_gc_malloc} or @code{scm_gc_calloc} is
-scanned for pointers.  Memory allocated by
-@code{scm_gc_malloc_pointerless} is not scanned.
+When garbage collection occurs, Guile will visit the words in memory
+allocated with @code{scm_gc_malloc} or @code{scm_gc_calloc}, looking for
+pointers to other memory allocations that are managed by the GC.  In
+contrast, memory allocated by @code{scm_gc_malloc_pointerless} is not
+scanned for pointers.
 
 The @code{scm_gc_realloc} call preserves the ``pointerlessness'' of the
 memory area pointed to by @var{mem}.  Note that you need to pass the old
diff --git a/doc/ref/libguile-concepts.texi b/doc/ref/libguile-concepts.texi
index 9785f4d..e93d987 100644
--- a/doc/ref/libguile-concepts.texi
+++ b/doc/ref/libguile-concepts.texi
@@ -203,22 +203,38 @@ set'' of garbage collection; any value on the heap that is referenced
 directly or indirectly by a member of the root set is preserved, and all
 other objects are eligible for reclamation.
 
-The Scheme stack and heap are scanned precisely; that is to say, Guile
-knows about all inter-object pointers on the Scheme stack and heap.
-This is not the case, unfortunately, for pointers on the C stack and
-static data segment.  For this reason we have to scan the C stack and
-static data segment @dfn{conservatively}; any value that looks like a
-pointer to a GC-managed object is treated as such, whether it actually
-is a reference or not.  Thus, scanning the C stack and static data
-segment is guaranteed to find all actual references, but it might also
-find words that only accidentally look like references.  These ``false
-positives'' might keep @code{SCM} objects alive that would otherwise be
-considered dead.  While this might waste memory, keeping an object
-around longer than it strictly needs to is harmless.  This is why this
-technique is called ``conservative garbage collection''.  In practice,
-the wasted memory seems to be no problem, as the static C root set is
-almost always finite and small, given that the Scheme stack is separate
-from the C stack.
+In Guile, garbage collection has two logical phases: the @dfn{mark
+phase}, in which the collector discovers the set of all live objects,
+and the @dfn{sweep phase}, in which the collector reclaims the resources
+associated with dead objects.  The mark phase pauses the program and
+traces all @code{SCM} object references, starting with the root set.
+The sweep phase actually runs concurrently with the main program,
+incrementally reclaiming memory as needed by allocation.
+
+In the mark phase, the garbage collector traces the Scheme stack and
+heap @dfn{precisely}.  Because the Scheme stack and heap are managed by
+Guile, Guile can know precisely where in those data structures it might
+find references to other heap objects.  This is not the case,
+unfortunately, for pointers on the C stack and static data segment.
+Instead of requiring the user to inform Guile about all variables in C
+that might point to heap objects, Guile traces the C stack and static
+data segment @dfn{conservatively}.  That is to say, Guile just treats
+every word on the C stack and every C global variable as a potential
+reference in to the Scheme heap@footnote{Note that Guile does not scan
+the C heap for references, so a reference to a @code{SCM} object from a
+memory segment allocated with @code{malloc} will have to use some other
+means to keep the @code{SCM} object alive.  @xref{Garbage Collection
+Functions}.}.  Any value that looks like a pointer to a GC-managed
+object is treated as such, whether it actually is a reference or not.
+Thus, scanning the C stack and static data segment is guaranteed to find
+all actual references, but it might also find words that only
+accidentally look like references.  These ``false positives'' might keep
+@code{SCM} objects alive that would otherwise be considered dead.  While
+this might waste memory, keeping an object around longer than it
+strictly needs to is harmless.  This is why this technique is called
+``conservative garbage collection''.  In practice, the wasted memory
+seems to be no problem, as the static C root set is almost always finite
+and small, given that the Scheme stack is separate from the C stack.
 
 The stack of every thread is scanned in this way and the registers of
 the CPU and all other memory locations where local variables or function
-- 
2.8.3