Plan for 2.0

Plan for 2.0

[Neil Jerram]

We're clearly moving towards a 2.0 release.  Here is my attempt to
pull that together a bit and flesh out what needs to be done.

What will go into 2.0:

1. The git "master" branch.  In principle, everything here, but we
need to review and check for

  - anything that should be excluded

  - any applicable fixes that were made in 1.8.x and didn't get copied
to master.

I've started doing this review and will hopefully complete soon.  If
there is anything that shouldn't be in 2.0, I'll move it into a new
branch.  If there are missing fixes from 1.8.x, I'll cherry pick them
into master.

2. The "vm" branch.  Once the review of "master" is done, we'll merge
"vm" into "master".

3. The "ossau-gds-dev" branch.  This contains some minor improvements
to the Emacs interface.  After the review of "master" is done, we'll
merge "ossau-gds-dev" into "master".

4. Any other changes (including bug fixes) that we think are important
to get done before 2.0.  I propose to review the bugs in Savannah, and
also recent email discussions, to identify these.

Is there anything else?  In particular, am I right in thinking that
the BDW-GC work is not ready yet?

One specific query...  Although I advocated removing GH before, I
don't feel 100% confident that that's the right thing for 2.0.  I'm
wondering now if we should instead move the GH code into a separate
library, "libgh", but continue to provide this as part of the Guile
distribution.  Moving the code out of libguile will still achieve the
important objectives of (1) reducing the size of the libguile code
that developers need to look at and work with, and (2) ensuring that
GH is implementable on top of the advertised SCM API; but keeping
libgh in the distribution will be a significant help for users who are
still using GH (who will just need to add -lgh to their link line).

I still think we should remove all GH-related documentation, as we
don't want to do anything to encourage further GH usage.  The GH code
itself is sufficient IMO for showing how someone can migrate their
code from GH to SCM.

That's all for now.  Any concerns or comments?

Regards,
       Neil

Re: Plan for 2.0

[David Séverin]

Hi Guilers,

It might be a small thing [and of course not a priority at all], but I'd love to see
a small evolution of the manual index structure in order to separate scheme
procedures from others, scheme variables from others...:

    * Concept Index				
    * Scheme Prodedure Index	* C Procedure Index
    * Scheme Variable Index	    	* C Variable Index
    * Scheme Type Index			* C Type Index
    * R5RS Index 

Being a scheme 'only' programmer, I'd love not to have to scroll through
gh_* and scm_* ... when I am looking for something in an index.

David

;; --

Le Sat, 3 Jan 2009 18:38:13 +0000,
"Neil Jerram" <neiljerram@googlemail.com> a écrit :

> We're clearly moving towards a 2.0 release.  Here is my attempt to
> pull that together a bit and flesh out what needs to be done.
> 
> What will go into 2.0:
> 
> 1. The git "master" branch.  In principle, everything here, but we
> need to review and check for
> 
>   - anything that should be excluded
> 
>   - any applicable fixes that were made in 1.8.x and didn't get copied
> to master.
> 
> I've started doing this review and will hopefully complete soon.  If
> there is anything that shouldn't be in 2.0, I'll move it into a new
> branch.  If there are missing fixes from 1.8.x, I'll cherry pick them
> into master.
> 
> 2. The "vm" branch.  Once the review of "master" is done, we'll merge
> "vm" into "master".
> 
> 3. The "ossau-gds-dev" branch.  This contains some minor improvements
> to the Emacs interface.  After the review of "master" is done, we'll
> merge "ossau-gds-dev" into "master".
> 
> 4. Any other changes (including bug fixes) that we think are important
> to get done before 2.0.  I propose to review the bugs in Savannah, and
> also recent email discussions, to identify these.
> 
> Is there anything else?  In particular, am I right in thinking that
> the BDW-GC work is not ready yet?
> 
> One specific query...  Although I advocated removing GH before, I
> don't feel 100% confident that that's the right thing for 2.0.  I'm
> wondering now if we should instead move the GH code into a separate
> library, "libgh", but continue to provide this as part of the Guile
> distribution.  Moving the code out of libguile will still achieve the
> important objectives of (1) reducing the size of the libguile code
> that developers need to look at and work with, and (2) ensuring that
> GH is implementable on top of the advertised SCM API; but keeping
> libgh in the distribution will be a significant help for users who are
> still using GH (who will just need to add -lgh to their link line).
> 
> I still think we should remove all GH-related documentation, as we
> don't want to do anything to encourage further GH usage.  The GH code
> itself is sufficient IMO for showing how someone can migrate their
> code from GH to SCM.
> 
> That's all for now.  Any concerns or comments?
> 
> Regards,
>        Neil
> 
> 

Re: Plan for 2.0

[Neil Jerram]

2009/1/4 David Séverin <david@altosw.be>:
> Hi Guilers,
>
> It might be a small thing [and of course not a priority at all], but I'd love to see
> a small evolution of the manual index structure in order to separate scheme
> procedures from others, scheme variables from others...:
>
>    * Concept Index
>    * Scheme Prodedure Index    * C Procedure Index
>    * Scheme Variable Index             * C Variable Index
>    * Scheme Type Index                 * C Type Index
>    * R5RS Index
>
> Being a scheme 'only' programmer, I'd love not to have to scroll through
> gh_* and scm_* ... when I am looking for something in an index.

Thanks for this idea!

I'm not persuaded by the procedure/variable/type separation, but I
agree that the Scheme/C separation would be useful.  Do you know
Texinfo well enough to look at how we could achieve this?

Regards,
     Neil

Re: Plan for 2.0

[Andy Wingo]

Hi Neil,

On Sat 03 Jan 2009 19:38, "Neil Jerram" <neiljerram@googlemail.com> writes:

> We're clearly moving towards a 2.0 release.  Here is my attempt to
> pull that together a bit and flesh out what needs to be done.

I think the plan is sensible.

> 2. The "vm" branch.  Once the review of "master" is done, we'll merge
> "vm" into "master".

I still need to finish documenting things, and tying up loose ends. I'm
especially concerned about ABI and future compatibility. But this can be
done in vm or master.

> I'm wondering now if we should instead move the GH code into a
> separate library, "libgh", but continue to provide this as part of the
> Guile distribution.

If we do this, the resulting "product" should have "deprecated"
somewhere in its name. You wouldn't want people mistaking it for
"high-level".

My personal opinion is that we should move this stuff to C files that
the user can include in her own project, and distribute those C files
somewhere. But perhaps this is not feasible.

> That's all for now.  Any concerns or comments?

Yay! :-)

Cheers,

Andy
-- 
http://wingolog.org/

Re: Plan for 2.0

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]


  We're clearly moving towards a 2.0 release.  Here is my attempt to
  pull that together a bit and flesh out what needs to be done.

That seems like a good plan on all counts, plus perhaps

  period of feature freeze on master, with testing on many OS and
  architecture variants, with a special emphasis on things that aren't
  GNU/Linux i386.

  very clear understanding of what's withdrawn from 1.6 and 1.8.
  Currently pkgsrc has 1.6 and 1.8, but I think 1.6 is only needed for
  guile-pg.  It would be nice to update the main package from 1.8 to 2.0
  and not have to keep 1.8.


[-- Attachment #2: Type: application/pgp-signature, Size: 193 bytes --]

Re: Plan for 2.0

[Neil Jerram]

2009/1/4 Neil Jerram <neiljerram@googlemail.com>:
> 2009/1/4 David Séverin <david@altosw.be>:
>> Hi Guilers,
>>
>> It might be a small thing [and of course not a priority at all], but I'd love to see
>> a small evolution of the manual index structure in order to separate scheme
>> procedures from others, scheme variables from others...:
>>
>>    * Concept Index
>>    * Scheme Prodedure Index    * C Procedure Index
>>    * Scheme Variable Index             * C Variable Index
>>    * Scheme Type Index                 * C Type Index
>>    * R5RS Index
>>
>> Being a scheme 'only' programmer, I'd love not to have to scroll through
>> gh_* and scm_* ... when I am looking for something in an index.
>
> Thanks for this idea!
>
> I'm not persuaded by the procedure/variable/type separation

I'm sorry, that's complete nonsense, as we already have the
procedure/variable/type separation.

So in fact now I completely agree with what you've suggested.

Regards,
        Neil

Re: Plan for 2.0

[David Séverin]

Le Sun, 4 Jan 2009 16:25:47 +0000,
"Neil Jerram" <neiljerram@googlemail.com> a écrit :

> 2009/1/4 David Séverin <david@altosw.be>:
> > Hi Guilers,
> >
> > It might be a small thing [and of course not a priority at all], but I'd love to
> > see a small evolution of the manual index structure in order to separate scheme
> > procedures from others, scheme variables from others...:
> >
> >    * Concept Index
> >    * Scheme Prodedure Index    * C Procedure Index
> >    * Scheme Variable Index             * C Variable Index
> >    * Scheme Type Index                 * C Type Index
> >    * R5RS Index
> >
> > Being a scheme 'only' programmer, I'd love not to have to scroll through
> > gh_* and scm_* ... when I am looking for something in an index.
> 
> Thanks for this idea!
> 
> I'm not persuaded by the procedure/variable/type separation, but I
> agree that the Scheme/C separation would be useful.  Do you know
> Texinfo well enough to look at how we could achieve this?

Never used Texinfo :( I could learn, but right now I can not offer more than using
guile and giving the best feedback I can on matters I feel knowledgeable enough to
do so [I am alone managing a small company [+- a year late in my work]].

Cheers,
David

Re: Plan for 2.0

[Ludovic Courtès]

Hello!

"Neil Jerram" <neiljerram@googlemail.com> writes:

> We're clearly moving towards a 2.0 release.

Cool!

> 2. The "vm" branch.  Once the review of "master" is done, we'll merge
> "vm" into "master".
>
> 3. The "ossau-gds-dev" branch.  This contains some minor improvements
> to the Emacs interface.  After the review of "master" is done, we'll
> merge "ossau-gds-dev" into "master".

I'd do (3) before (2) because it's probably easier.

> 4. Any other changes (including bug fixes) that we think are important
> to get done before 2.0.  I propose to review the bugs in Savannah, and
> also recent email discussions, to identify these.

Yeah, let's fix bugs!  We still have several bugs reported against 1.8
that need care.

> Is there anything else?  In particular, am I right in thinking that
> the BDW-GC work is not ready yet?

The BDW-GC branch is fully functional, and the user-visible API changes
are frozen.  What may (or may not) be a stopper are:

  1. The lack of `gc-live-object-stats'.

  2. Different fields of `gc-stats'.

  3. Different behavior of weak hash tables, see
     http://lists.gnu.org/archive/html/guile-devel/2008-11/msg00015.html .
     This can be fixed, but I'm unclear whether it's worth it (comments
     welcome!).

  4. Possible guardian glitches (`guardians.test' seems to be too
     permissive to catch non-obvious problems).

I also need to post additional benchmark results.

> One specific query...  Although I advocated removing GH before, I
> don't feel 100% confident that that's the right thing for 2.0.  I'm
> wondering now if we should instead move the GH code into a separate
> library, "libgh", but continue to provide this as part of the Guile
> distribution.  Moving the code out of libguile will still achieve the
> important objectives of (1) reducing the size of the libguile code
> that developers need to look at and work with, and (2) ensuring that
> GH is implementable on top of the advertised SCM API; but keeping
> libgh in the distribution will be a significant help for users who are
> still using GH (who will just need to add -lgh to their link line).

I never considered it urgent, but I think it should be either completely
removed (as is currently the case) or left in `libguile'.  Moving it to
another library would make it essentially worthless since it would make
it incompatible anyway.

We could ship a C compatibility header as Andy suggested, but I'm not
sure it's 100% needed.

> I still think we should remove all GH-related documentation, as we
> don't want to do anything to encourage further GH usage.  The GH code
> itself is sufficient IMO for showing how someone can migrate their
> code from GH to SCM.

Agreed.

Thanks,
Ludo'.

Re: Plan for 2.0

[Neil Jerram]

2009/1/3 Neil Jerram <neiljerram@googlemail.com>:
>
> I've started doing this review and will hopefully complete soon.

Below is a raw summary of all diffs between current branch_release-1-8
and master.  Next step is to check that everything here is correct,
and properly+fully documented in the manual and in NEWS.  The
"Queries" at the end are bits that I'm not sure I understand yet.

       Neil

Mechanical/straightforward stuff
- version number changes, some will need advancing
  - new i18n lib
  - note also in guile-readline
- copyright years
- top level docs: NEWS, README, THANKS
- textual stuff / typos (doc strings, manual)

Use of Gnulib
- linker warning
- alloca - Have we inadvertently removed requirement for a real alloca?
- count one bits
- full read
- full write: complete write through async pipe; also in fport_flush
- safe read
- safe write
- stdbool
- strcase - instead of Guile's own defn
- strftime - use to fix portability problems
- strings
- time
- time_r
- unistd
- verify
- wchar
- write

Announce build tool versions in autogen.sh

I18N (doc in api-i18n.texi) (including SRFI-19 changes)

memoize-symbol trap and associated options
- Doc refers to "retval", which isn't in the signature

SRFI-18 threads, and core thread extensions for this

strftime %Z fix

serial number in guile.m4

Change to readline.scm's repl-reader

scm_options not taking a count; also scm_init_opts; private-options.h;
- dry run (avoiding exception in crit sec)
- private-options (making option macros private)

Module implementation changes
- duplicates-handlers, observers, weak-observers, import-obarray
- removed: duplicates-interface, observer-id
- module-variable and module-local-variable -> C; also
- module-import-interface, module-reverse-lookup, ...
- Catch a module trying to use itself
- Change to implementation of autoloads?

Autoload (ice-9 debugger) instead of loading it unconditionally

Remove environments.[ch] (to avoid confusion)

Han-Wen GC (including related struct changes)

eval.c/eval.i.c
- still need to compare old eval.c against new eval.i.c
- why does eval.i.c contain code that is common to both modes and that
  is not compiled twice?
- removal of deprecated functions: scm_ceval, scm_deval, scm_ceval_ptr

SCM_INTERNAL (grep diffs for SCM_INTERNAL to get list of affected functions)

Port stuff
- port eviction change
- scm_fdes_to_ports
- port table/weak hash

Src prop impl changes

Signal delivery thread
- allow it to exit, by closing signal delivery pipe
- add mutex to serialize creation and closing

Print debug info when throw from crit src

GH removal

Queries
=======
AC_SUBST(GCC_FLAGS)
lib-version.texi
ChangeLogs still in distribution?
libguile in subdirs list of pre-inst-guile.in

Re: Plan for 2.0

[Neil Jerram]

2009/1/5 David Séverin <david@altosw.be>:
>
> Never used Texinfo :( I could learn, but right now I can not offer more than using
> guile and giving the best feedback I can on matters I feel knowledgeable enough to
> do so [I am alone managing a small company [+- a year late in my work]].

No problem, I'll see what I can do, for implementing this.  Your
feedback is appreciated!

     Neil

Re: Plan for 2.0

[Neil Jerram]

2009/1/5 Ludovic Courtès <ludo@gnu.org>:
>
>> One specific query...  Although I advocated removing GH before, I
>> don't feel 100% confident that that's the right thing for 2.0.  I'm
>> wondering now if we should instead move the GH code into a separate
>> library, "libgh", but continue to provide this as part of the Guile
>> distribution.  Moving the code out of libguile will still achieve the
>> important objectives of (1) reducing the size of the libguile code
>> that developers need to look at and work with, and (2) ensuring that
>> GH is implementable on top of the advertised SCM API; but keeping
>> libgh in the distribution will be a significant help for users who are
>> still using GH (who will just need to add -lgh to their link line).
>
> I never considered it urgent, but I think it should be either completely
> removed (as is currently the case) or left in `libguile'.  Moving it to
> another library would make it essentially worthless since it would make
> it incompatible anyway.

Why would that make it worthless and incompatible?  Wouldn't it allow
existing source code to continue to compile and link?

Thanks,
       Neil

Re: Plan for 2.0

[Ludovic Courtès]

Hi!

"Neil Jerram" <neiljerram@googlemail.com> writes:

> 2009/1/5 Ludovic Courtès <ludo@gnu.org>:
>>
>>> One specific query...  Although I advocated removing GH before, I
>>> don't feel 100% confident that that's the right thing for 2.0.  I'm
>>> wondering now if we should instead move the GH code into a separate
>>> library, "libgh", but continue to provide this as part of the Guile
>>> distribution.  Moving the code out of libguile will still achieve the
>>> important objectives of (1) reducing the size of the libguile code
>>> that developers need to look at and work with, and (2) ensuring that
>>> GH is implementable on top of the advertised SCM API; but keeping
>>> libgh in the distribution will be a significant help for users who are
>>> still using GH (who will just need to add -lgh to their link line).
>>
>> I never considered it urgent, but I think it should be either completely
>> removed (as is currently the case) or left in `libguile'.  Moving it to
>> another library would make it essentially worthless since it would make
>> it incompatible anyway.
>
> Why would that make it worthless and incompatible?  Wouldn't it allow
> existing source code to continue to compile and link?

Yes, but only if the existing code is modified to use `-lgh' instead of
`-lguile'.  That'd be "half-compatible".

Thanks,
Ludo'.

Re: Plan for 2.0

[Ludovic Courtès]

Hi Neil,

"Neil Jerram" <neiljerram@googlemail.com> writes:

> Below is a raw summary of all diffs between current branch_release-1-8
> and master.  Next step is to check that everything here is correct,
> and properly+fully documented in the manual and in NEWS.  The
> "Queries" at the end are bits that I'm not sure I understand yet.

Thanks for going through this.

> Use of Gnulib
> - linker warning
> - alloca - Have we inadvertently removed requirement for a real alloca?

No.  Gnulib's `alloca' provides a substitute when that's needed and most
importantly provides the right #ifdefs to get a working `alloca ()' (see
`lib/alloca.in.h' and (info "(autoconf) Particular Functions")).

> serial number in guile.m4

Why is that?  Are there differences?

> eval.c/eval.i.c
> - still need to compare old eval.c against new eval.i.c
> - why does eval.i.c contain code that is common to both modes and that
>   is not compiled twice?

Like what?  The top of the file is in `#ifdef DEVAL'.

> SCM_INTERNAL (grep diffs for SCM_INTERNAL to get list of affected functions)

Speaking of which, some functions were left external (e.g.,
`scm_i_string_chars ()') under the assumption that if we changed that in
1.8 (which was my plan back then) it would break apps.  We may need to
revise that.

> Queries
> =======
> AC_SUBST(GCC_FLAGS)

This is so that we don't compile Gnulib code with `-Wall -Werror' since
Gnulib doesn't guarantee that this would work.

> lib-version.texi

This is for use in `api-i18n.texi', for instance.

> ChangeLogs still in distribution?

Yes, same as for `branch_release-1-8'.

> libguile in subdirs list of pre-inst-guile.in

Dunno why.  The Right Way would be to use `libtool --mode=execute
-dlopen foo-bar.la' anyway.

Thanks,
Ludo'.

Re: Plan for 2.0

[Neil Jerram]

Hi Ludo!  Thanks for your responses...

2009/1/8 Ludovic Courtès <ludo@gnu.org>:
> Hi Neil,
>
> "Neil Jerram" <neiljerram@googlemail.com> writes:
>
>> Use of Gnulib
>> - linker warning
>> - alloca - Have we inadvertently removed requirement for a real alloca?
>
> No.  Gnulib's `alloca' provides a substitute when that's needed and most
> importantly provides the right #ifdefs to get a working `alloca ()' (see
> `lib/alloca.in.h' and (info "(autoconf) Particular Functions")).

That's good, but I think I didn't explain the possible problem fully,
i.e. that the substitute won't work.  Because of how Guile saves and
restores continuations (by copying the stack), and how it uses alloca
to create space for debug information on the stack, we must have an
alloca() that really does use the stack, and not one that uses the
heap.  And when I last checked, the common substitute definition of
alloca() uses the heap.

So, (i) am I still missing something, or (ii) does the substitute
definition use the stack now, or (iii) is there a way to tell Gnulib
that we really need an on-the-stack alloca (and that the build should
fail if there isn't one), or (iv) are we already doing (iii) and I
just missed it :-) ?

(We originally addressed this, somewhere in 1.6.x or 1.8.x, here:
http://git.savannah.gnu.org/gitweb/?p=guile.git;a=commitdiff;h=31e4c69e95bc77b7f701348a32ffc8374ad8a0e5)

>> serial number in guile.m4
>
> Why is that?  Are there differences?

master has a line "serial 9", branch_release-1-8 doesn't.  I believe
it's a standard of some kind to have a "serial" line, so we added it
in master, but preferred to avoid perturbing 1.8.x.

>> eval.c/eval.i.c
>> - still need to compare old eval.c against new eval.i.c
>> - why does eval.i.c contain code that is common to both modes and that
>>   is not compiled twice?
>
> Like what?  The top of the file is in `#ifdef DEVAL'.

I would have thought that the point of eval.i.c is to hold the code
that gets compiled twice (debug and non-debug).  If there is #ifdef
DEVAL code in eval.i.c, it seems to be better to move it back to
eval.c.

>> SCM_INTERNAL (grep diffs for SCM_INTERNAL to get list of affected functions)
>
> Speaking of which, some functions were left external (e.g.,
> `scm_i_string_chars ()') under the assumption that if we changed that in
> 1.8 (which was my plan back then) it would break apps.  We may need to
> revise that.

I can't immediately remember what the latest consensus on that was...

>> Queries
>> =======
>> AC_SUBST(GCC_FLAGS)
>
> This is so that we don't compile Gnulib code with `-Wall -Werror' since
> Gnulib doesn't guarantee that this would work.
>
>> lib-version.texi
>
> This is for use in `api-i18n.texi', for instance.

Thanks for those.

>> ChangeLogs still in distribution?
>
> Yes, same as for `branch_release-1-8'.

OK, thanks.

>> libguile in subdirs list of pre-inst-guile.in
>
> Dunno why.  The Right Way would be to use `libtool --mode=execute
> -dlopen foo-bar.la' anyway.

Thanks, I'll look into that.

> Thanks,
> Ludo'.

Regards,
        Neil

Re: Plan for 2.0

[David Séverin]

Le Sat, 3 Jan 2009 18:38:13 +0000,
"Neil Jerram" <neiljerram@googlemail.com> a écrit :

> We're clearly moving towards a 2.0 release.  Here is my attempt to
> pull that together a bit and flesh out what needs to be done.
> 
> What will go into 2.0:

Hi Guilers,

Don't you think that the 'using thread' on debian [and other distro?] shouldn't be
solved before to call a release '2.xx'?

David

Re: Plan for 2.0

[Ludovic Courtès]

Hi,

David Séverin <lpce.rio@gmail.com> writes:

> Don't you think that the 'using thread' on debian [and other distro?] shouldn't be
> solved before to call a release '2.xx'?

That Debian builds Guile --without-threads is a Debian-specific problem.

The fact that Guile with and without threads aren't ABI-compatible is
our problem, but I don't think there's a plan to "fix" it.  Probably we
should document it.

Thanks,
Ludo'.

Re: Plan for 2.0

[Ludovic Courtès]

Hello,

"Neil Jerram" <neiljerram@googlemail.com> writes:

> That's good, but I think I didn't explain the possible problem fully,
> i.e. that the substitute won't work.  Because of how Guile saves and
> restores continuations (by copying the stack), and how it uses alloca
> to create space for debug information on the stack, we must have an
> alloca() that really does use the stack, and not one that uses the
> heap.  And when I last checked, the common substitute definition of
> alloca() uses the heap.

Aaah, you're right.  Then we should use the `alloca-opt' module instead,
which does what we want AFAIU.

(Note that systems without `alloca ()', are probably very rare.)

> master has a line "serial 9", branch_release-1-8 doesn't.  I believe
> it's a standard of some kind to have a "serial" line, so we added it
> in master, but preferred to avoid perturbing 1.8.x.

Right, it's "good practice" (info "(automake) Serials").

>>> eval.c/eval.i.c
>>> - still need to compare old eval.c against new eval.i.c
>>> - why does eval.i.c contain code that is common to both modes and that
>>>   is not compiled twice?
>>
>> Like what?  The top of the file is in `#ifdef DEVAL'.
>
> I would have thought that the point of eval.i.c is to hold the code
> that gets compiled twice (debug and non-debug).  If there is #ifdef
> DEVAL code in eval.i.c, it seems to be better to move it back to
> eval.c.

Agreed.

>>> SCM_INTERNAL (grep diffs for SCM_INTERNAL to get list of affected functions)
>>
>> Speaking of which, some functions were left external (e.g.,
>> `scm_i_string_chars ()') under the assumption that if we changed that in
>> 1.8 (which was my plan back then) it would break apps.  We may need to
>> revise that.
>
> I can't immediately remember what the latest consensus on that was...

IIRC, nobody cares.  :-)

Thanks,
Ludo'.

Re: Plan for 2.0

[Neil Jerram]

2009/1/12 Ludovic Courtès <ludo@gnu.org>:
> Hello,
>
> "Neil Jerram" <neiljerram@googlemail.com> writes:
>
>> That's good, but I think I didn't explain the possible problem fully,
>> i.e. that the substitute won't work.  Because of how Guile saves and
>> restores continuations (by copying the stack), and how it uses alloca
>> to create space for debug information on the stack, we must have an
>> alloca() that really does use the stack, and not one that uses the
>> heap.  And when I last checked, the common substitute definition of
>> alloca() uses the heap.
>
> Aaah, you're right.  Then we should use the `alloca-opt' module instead,
> which does what we want AFAIU.

Ah, great; just trying that out now...

    Neil

Re: Plan for 2.0

[Neil Jerram]

2009/1/12 Neil Jerram <neiljerram@googlemail.com>:
> 2009/1/12 Ludovic Courtès <ludo@gnu.org>:
>> Hello,
>>
>> "Neil Jerram" <neiljerram@googlemail.com> writes:
>>
>>> That's good, but I think I didn't explain the possible problem fully,
>>> i.e. that the substitute won't work.  Because of how Guile saves and
>>> restores continuations (by copying the stack), and how it uses alloca
>>> to create space for debug information on the stack, we must have an
>>> alloca() that really does use the stack, and not one that uses the
>>> heap.  And when I last checked, the common substitute definition of
>>> alloca() uses the heap.
>>
>> Aaah, you're right.  Then we should use the `alloca-opt' module instead,
>> which does what we want AFAIU.
>
> Ah, great; just trying that out now...

Seems fine, so I've committed the change.  That commit also includes a
general Gnulib update; I hope that is OK, and that I haven't committed
any files that shouldn't have been; please let me know if you see any
problems or mistakes.

     Neil

Re: Plan for 2.0

[Neil Jerram]

2009/1/5 Ludovic Courtès <ludo@gnu.org>:
> Hello!
>
> "Neil Jerram" <neiljerram@googlemail.com> writes:
>
>> 3. The "ossau-gds-dev" branch.  This contains some minor improvements
>> to the Emacs interface.  After the review of "master" is done, we'll
>> merge "ossau-gds-dev" into "master".
>
> I'd do (3) before (2) because it's probably easier.

Agreed.

>> Is there anything else?  In particular, am I right in thinking that
>> the BDW-GC work is not ready yet?
>
> The BDW-GC branch is fully functional, and the user-visible API changes
> are frozen.  What may (or may not) be a stopper are:
>
>  1. The lack of `gc-live-object-stats'.

I doubt it's a stopper, but current `gc-live-object-stats' is quite
nice.  Doesn't libgc have a lightweight object-collected hook that
would allow us to implement this?  Or is it the problem that would be
too bad for performance?

>  2. Different fields of `gc-stats'.

I would say that this is fine.

>  3. Different behavior of weak hash tables, see
>     http://lists.gnu.org/archive/html/guile-devel/2008-11/msg00015.html .
>     This can be fixed, but I'm unclear whether it's worth it (comments
>     welcome!).

IIUC, your example to the GC list can be fixed (in the application) by
using a (BGC) doubly weak hash table instead.  I wonder what is the
simplest example of a current Guile weak-key hash table that can't be
mapped to a BGC doubly weak one?

>> One specific query...  Although I advocated removing GH before, I
>> don't feel 100% confident that that's the right thing for 2.0.  I'm
>> wondering now if we should instead move the GH code into a separate
>> library, "libgh", but continue to provide this as part of the Guile
>> distribution.  Moving the code out of libguile will still achieve the
>> important objectives of (1) reducing the size of the libguile code
>> that developers need to look at and work with, and (2) ensuring that
>> GH is implementable on top of the advertised SCM API; but keeping
>> libgh in the distribution will be a significant help for users who are
>> still using GH (who will just need to add -lgh to their link line).
>
> I never considered it urgent, but I think it should be either completely
> removed (as is currently the case) or left in `libguile'.  Moving it to
> another library would make it essentially worthless since it would make
> it incompatible anyway.
>
> We could ship a C compatibility header as Andy suggested, but I'm not
> sure it's 100% needed.

Is your view on this a strong one?  I feel fairly sure that we ought
to continue to distribute this code - but in a deprecated and
undocumented separate library - because I think by doing so we can
help users with negligible ongoing maintenance cost.

Thanks!
      Neil

BDW-GC-Guile incompatibilities

[Ludovic Courtès]

Hi,

"Neil Jerram" <neiljerram@googlemail.com> writes:

> 2009/1/5 Ludovic Courtès <ludo@gnu.org>:

>>  1. The lack of `gc-live-object-stats'.
>
> I doubt it's a stopper, but current `gc-live-object-stats' is quite
> nice.  Doesn't libgc have a lightweight object-collected hook that
> would allow us to implement this?  Or is it the problem that would be
> too bad for performance?

We'd need to write something similar to `scm_i_card_statistics ()',
which basically traverses a card and accumulates live object info in a
hash table.  The difficulty is that, while Guile's GC is only able to
scan the cell heap (which is sufficient), BDW-GC doesn't know what a
cell is, so we'd have to find tricks to somehow not traverse the whole
heap but only cells.

I can look into it into more details to see whether/how that can be done
before we settle on a decision.

It'd be a potential source of incompatibility if `gc-live-object-stats'
behaved differently.  For instance, `gc.test' in `master' uses it to see
whether a given object was reclaimed, so the same test was rewritten in
the BDW-GC branch.

>>  3. Different behavior of weak hash tables, see
>>     http://lists.gnu.org/archive/html/guile-devel/2008-11/msg00015.html .
>>     This can be fixed, but I'm unclear whether it's worth it (comments
>>     welcome!).
>
> IIUC, your example to the GC list can be fixed (in the application) by
> using a (BGC) doubly weak hash table instead.

Yes, but that's a visible change for the application.

> I wonder what is the simplest example of a current Guile weak-key hash
> table that can't be mapped to a BGC doubly weak one?

The test that led me to find it is "Unused modules are removed" in
`gc.test'.  The culprit is `set-module-eval-closure!', which creates a
circular reference between CLOSURE and MODULE via the procedure property
weak-key hash table.

Here are the details (from an old message I never posted since I
understood what happens, in the meantime...):

Basically, `make-module' invokes a record constructor, after which it
does:

  (set-module-eval-closure! module (standard-eval-closure module))

Note that `standard-eval-closure' returns a fresh SMOB that points to
MODULE.

In turn, `set-module-eval-closure!' invokes the record setter and then
does this:

  (set-procedure-property! closure 'module module)

Here, CLOSURE is actually the SMOB returned by `standard-eval-closure'.
Procedure properties of "real" closures are stored in the closure itself
(see `SCM_SETPROCPROPS ()').  But for non-closure objects such as subrs
or SMOBs, an indirection is used: a phony closure is created and
associated with the subr/SMOB via a weakly-keyed hash table (see
`scm_stand_in_scm_proc ()').  Still here?

So, in the case of `make-module', the object graph is as follows:


    .---------------------------------.
    |                                 |
    V				      |
.--------.	 .--------------.  .----------------------------.
| module |<----->| eval-closure	|  | closure 			|
`--------'     	 `--------------'  | (from			|
       	       	   ^   	       	   |  `scm_stand_in_scm_proc ())|
		   :   	       	   `----------------------------'
		   :		      ^	       	       	       	 
		   :		      :	       	       	       	 
		   :..................'	       	       	       	
		   :		       	
		 .---------------------.
		 | scm_stand_in_procs  |
		 | (weak-key hash tab.)|
		 `---------------------'

(Dotted arrows denote weak references, while solid arrows denote
"regular" references.)

In the BDW-GC-based Guile, the reference from the phony closure to the
module prevents them from being collected.

Cheers,
Ludo'.

Re: Plan for 2.0

[Ludovic Courtès]

>> We could ship a C compatibility header as Andy suggested, but I'm not
>> sure it's 100% needed.
>
> Is your view on this a strong one?  I feel fairly sure that we ought
> to continue to distribute this code - but in a deprecated and
> undocumented separate library - because I think by doing so we can
> help users with negligible ongoing maintenance cost.

But if it's not documented, how are people going to know that they
should link against `libgh'?  :-)

This solution would be OK for me, but I find this "in-between" libgh
trick more complex than either removing the gh code or leaving it
entirely.

Thanks,
Ludo'.

Re: BDW-GC-Guile incompatibilities

[Neil Jerram]

Hi Ludo,

Sorry for my delay in following up here...

ludo@gnu.org (Ludovic Courtès) writes:

> Hi,
>
> "Neil Jerram" <neiljerram@googlemail.com> writes:
>
>> 2009/1/5 Ludovic Courtès <ludo@gnu.org>:
>
>>>  1. The lack of `gc-live-object-stats'.
>>
>> I doubt it's a stopper, but current `gc-live-object-stats' is quite
>> nice.  Doesn't libgc have a lightweight object-collected hook that
>> would allow us to implement this?  Or is it the problem that would be
>> too bad for performance?
>
> We'd need to write something similar to `scm_i_card_statistics ()',
> which basically traverses a card and accumulates live object info in a
> hash table.  The difficulty is that, while Guile's GC is only able to
> scan the cell heap (which is sufficient), BDW-GC doesn't know what a
> cell is, so we'd have to find tricks to somehow not traverse the whole
> heap but only cells.

I was actually thinking of decrementing an object type counter when an
object is collected - and having looked more at the docs I see that
that would mean using GC_register_finalizer ().  However I also see
from [1] that that (having most SCM objects being finalizable) would
be very bad for the effectiveness of the GC - because of the point
that in a chain of finalizable objects, only one object (the head) is
finalized in each collection.

[1] http://www.hpl.hp.com/personal/Hans_Boehm/gc/finalization.html

> I can look into it into more details to see whether/how that can be done
> before we settle on a decision.

OK.

> It'd be a potential source of incompatibility if `gc-live-object-stats'
> behaved differently.  For instance, `gc.test' in `master' uses it to see
> whether a given object was reclaimed, so the same test was rewritten in
> the BDW-GC branch.

Ultimately I think `gc-live-object-stats' can be sacrificed if need
be.  If there are specific application needs for part of the
information that it provides, I would guess that they are actually for
specific object types only and could be met by using a guardian.

>>>  3. Different behavior of weak hash tables, see
>>>     http://lists.gnu.org/archive/html/guile-devel/2008-11/msg00015.html .
>>>     This can be fixed, but I'm unclear whether it's worth it (comments
>>>     welcome!).
>>
>> IIUC, your example to the GC list can be fixed (in the application) by
>> using a (BGC) doubly weak hash table instead.
>
> Yes, but that's a visible change for the application.

Yes, but at least it is a change that we can document, and we can
precisely advise application authors what they need to do if they see
this problem.

>> I wonder what is the simplest example of a current Guile weak-key hash
>> table that can't be mapped to a BGC doubly weak one?
>
> The test that led me to find it is "Unused modules are removed" in
> `gc.test'.  The culprit is `set-module-eval-closure!', which creates a
> circular reference between CLOSURE and MODULE via the procedure property
> weak-key hash table.
>
> Here are the details (from an old message I never posted since I
> understood what happens, in the meantime...):
>
> Basically, `make-module' invokes a record constructor, after which it
> does:
>
>   (set-module-eval-closure! module (standard-eval-closure module))
>
> Note that `standard-eval-closure' returns a fresh SMOB that points to
> MODULE.
>
> In turn, `set-module-eval-closure!' invokes the record setter and then
> does this:
>
>   (set-procedure-property! closure 'module module)
>
> Here, CLOSURE is actually the SMOB returned by `standard-eval-closure'.
> Procedure properties of "real" closures are stored in the closure itself
> (see `SCM_SETPROCPROPS ()').  But for non-closure objects such as subrs
> or SMOBs, an indirection is used: a phony closure is created and
> associated with the subr/SMOB via a weakly-keyed hash table (see
> `scm_stand_in_scm_proc ()').  Still here?

Yes!

> So, in the case of `make-module', the object graph is as follows:
>
>
>     .---------------------------------.
>     |                                 |
>     V				        |
> .--------.       .--------------.  .----------------------------.
> | module |<----->| eval-closure |  | closure 			  |
> `--------'       `--------------'  | (from			  |
>        	       	   ^   	     |  `scm_stand_in_scm_proc ())|
> 		           :         `----------------------------'
> 		           :		      ^	       	       	       	 
> 		           :		      :	       	       	       	 
> 		           :..................'	       	       	       	
> 		           :		       	
> 		 .---------------------.
> 		 | scm_stand_in_procs  |
> 		 | (weak-key hash tab.)|
> 		 `---------------------'
>
> (Dotted arrows denote weak references, while solid arrows denote
> "regular" references.)
>
> In the BDW-GC-based Guile, the reference from the phony closure to the
> module prevents them from being collected.

Thanks for the detailed description.  But this case could also be solved
by making scm_stand_in_procs doubly weak, can't it?

(Also, I wonder if it could be solved by getting rid of the stand-in
thing.  Why does eval-closure need 2 references to module?  Sounds
like it might be a case of design rot.)

Finally, another concern from [1] is "Cycles involving one or more
finalizable objects are never finalized."  Given that, can our
guardians still work?  You mentioned in another email about "possible
guardian glitches", is this what you had in mind?

Regards,
        Neil

Re: BDW-GC-Guile incompatibilities

[Ludovic Courtès]

Hi Neil,

Neil Jerram <neil@ossau.uklinux.net> writes:

> Sorry for my delay in following up here...

Same for me!

>>>>  1. The lack of `gc-live-object-stats'.

> I was actually thinking of decrementing an object type counter when an
> object is collected - and having looked more at the docs I see that
> that would mean using GC_register_finalizer ().  However I also see
> from [1] that that (having most SCM objects being finalizable) would
> be very bad for the effectiveness of the GC - because of the point
> that in a chain of finalizable objects, only one object (the head) is
> finalized in each collection.

Yes, finalization should be used sparsely.

> Ultimately I think `gc-live-object-stats' can be sacrificed if need
> be.  If there are specific application needs for part of the
> information that it provides, I would guess that they are actually for
> specific object types only and could be met by using a guardian.

Yes, that's right.  It's just less convenient, especially when searching
for a leak of unknown origin.

>>>>  3. Different behavior of weak hash tables, see
>>>>     http://lists.gnu.org/archive/html/guile-devel/2008-11/msg00015.html .
>>>>     This can be fixed, but I'm unclear whether it's worth it (comments
>>>>     welcome!).

> Yes, but at least it is a change that we can document, and we can
> precisely advise application authors what they need to do if they see
> this problem.

Right, this may be acceptable.

> Thanks for the detailed description.  But this case could also be solved
> by making scm_stand_in_procs doubly weak, can't it?

Yes (there are doubly-weak tests in `weaks.test').

> (Also, I wonder if it could be solved by getting rid of the stand-in
> thing.  Why does eval-closure need 2 references to module?  Sounds
> like it might be a case of design rot.)

Indeed, there may be room for improvement here.  :-)

> Finally, another concern from [1] is "Cycles involving one or more
> finalizable objects are never finalized."  Given that, can our
> guardians still work?  You mentioned in another email about "possible
> guardian glitches", is this what you had in mind?

What I had in mind is the various tests in `guardians.test' that end up
being "unresolved" (see, e.g.,
http://autobuild.josefsson.org/guile/log-200902271203718687000.txt).  On
my machine I was actually seeing more of them.

The issue is that these tests never fail, they just throw `unresolved'
at worst.  This is due to the fact that we use a conservative GC so if
an object remains alive, it can be due to the fact that a stale
reference to the object is visible on the stack, for instance.

I investigated these `unresolved' test cases, which led to this patch:

  http://git.savannah.gnu.org/gitweb/?p=guile.git;a=commitdiff;h=6a7489ace3f07a8d190110cd1244963526c65729

In most cases, the tests can be made to succeed instead of being
unresolved by changing the code in a way that increases chances that
no references to the object we expect to die are left on the stack.

However, the "g2-garbage saved" is clearly a bug, as explained in the
commit above.  OTOH, this use case is probably not representative of any
real-world application, so I wouldn't worry about it.

Besides, it would be possible for an application to create a cycle of
guardians, which would prevent them from being GC'd, as you noted.
Similarly, an application could create mutually referenced SMOBs, both
of which have a `free ()' procedure; these could not be GC'd either.
But again, that sounds rather unrealistic IMO.  What do you think?

Thanks,
Ludo'.