BDW-GC branch updated

BDW-GC branch updated

[Ludovic Courtès]

Hello!

I merged `master' in the `boehm-demers-weiser-gc' branch yesterday,
which hadn't been done since before 1.9.0 (no, you won't see the 4 MiB+
commit message on `guile-commits'...).

So now is a good time to test it and report back!  It requires libgc 7.1
or later, which isn't packaged in Debian, although it was released in
May 2008.

I'll post updated comparative benchmarks hopefully in the next few days.

I hope we can merge it by the time 2.0 is released, which leaves us with
just a couple of months (in theory, at least).  Thus I encourage people
enthusiastic about it to try it and report back.  Otherwise we'll have
to wait for Guile 3.0, which is due in 2017 Q4.  You've been warned.
:-)

Thanks,
Ludo'.

Re: BDW-GC branch updated

[Andreas Rottmann]

ludo@gnu.org (Ludovic Courtès) writes:

> Hello!
>
> I merged `master' in the `boehm-demers-weiser-gc' branch yesterday,
> which hadn't been done since before 1.9.0 (no, you won't see the 4 MiB+
> commit message on `guile-commits'...).
>
> So now is a good time to test it and report back!  It requires libgc 7.1
> or later, which isn't packaged in Debian, although it was released in
> May 2008.
>
It's in experimental since recently; I assume its maintainer will upload
to unstable soonish.

>
> I'll post updated comparative benchmarks hopefully in the next few days.
>
Will going from a precise GC to BDW-GC not have drawbacks? IIRC, the PLT
people went in the opposite direction. A quick google turned up this:

http://www.cs.brown.edu/pipermail/plt-scheme/2006-June/013840.html

I wonder if the reasons for switching to a precise GC listed in there
will also apply to Guile.

Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>

Re: BDW-GC branch updated

[Ludovic Courtès]

Hi,

Andreas Rottmann <a.rottmann@gmx.at> writes:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> So now is a good time to test it and report back!  It requires libgc 7.1
>> or later, which isn't packaged in Debian, although it was released in
>> May 2008.
>>
> It's in experimental since recently; I assume its maintainer will upload
> to unstable soonish.

Good.

> Will going from a precise GC to BDW-GC not have drawbacks? IIRC, the PLT
> people went in the opposite direction. A quick google turned up this:
>
> http://www.cs.brown.edu/pipermail/plt-scheme/2006-June/013840.html
>
> I wonder if the reasons for switching to a precise GC listed in there
> will also apply to Guile.

Thanks for the link!

They write:

  There is one known problem, though, related to linked lists [Boehm,
  POPL'02].  Unfortunately, we seem to hit this problem often in
  practice, due to the way that threads and continuations are
  implemented, and there doesn't seem to be a reliable way around it.

The paper is "Bounding Space Usage of Conservative Garbage Collectors",
available from http://www.hpl.hp.com/personal/Hans_Boehm/gc/ .  It
depicts scenarios where "false references" lead to unbounded data
retention.  My interpretation of these scenarios and the "Summary"
section is that these cases are hopefully quite rare.

Now, I don't have enough experience of long-running BDW-GC applications
to know whether it's a problem in practice.  The PLT folks surely had
more experience (but with a different implementation IIUC).  There are
also other schemes that use BDW-GC, such as Bigloo.

However, it doesn't worry me as much as the current GC bugs (e.g., [0, 1]).

Also, there are definite benefits to using a conservative GC for
libguile, given how tightly it can be integrated with C (e.g., [2]).

What do you think?

Thanks,
Ludo'.

[0] http://permalink.gmane.org/gmane.lisp.guile.devel/6832
[1] http://lists.gnu.org/archive/html/guile-devel/2008-08/msg00120.html
[2] http://lists.gnu.org/archive/html/guile-devel/2008-11/msg00009.html

Re: BDW-GC branch updated

[Andreas Rottmann]

ludo@gnu.org (Ludovic Courtès) writes:

> Andreas Rottmann <a.rottmann@gmx.at> writes:
>
>> Will going from a precise GC to BDW-GC not have drawbacks? IIRC, the PLT
>> people went in the opposite direction. A quick google turned up this:
>>
>> http://www.cs.brown.edu/pipermail/plt-scheme/2006-June/013840.html
>>
>> I wonder if the reasons for switching to a precise GC listed in there
>> will also apply to Guile.
>
> Thanks for the link!
>
> They write:
>
>   There is one known problem, though, related to linked lists [Boehm,
>   POPL'02].  Unfortunately, we seem to hit this problem often in
>   practice, due to the way that threads and continuations are
>   implemented, and there doesn't seem to be a reliable way around it.
>
> The paper is "Bounding Space Usage of Conservative Garbage Collectors",
> available from http://www.hpl.hp.com/personal/Hans_Boehm/gc/ .  It
> depicts scenarios where "false references" lead to unbounded data
> retention.  My interpretation of these scenarios and the "Summary"
> section is that these cases are hopefully quite rare.
>
> Now, I don't have enough experience of long-running BDW-GC applications
> to know whether it's a problem in practice.  The PLT folks surely had
> more experience (but with a different implementation IIUC).  There are
> also other schemes that use BDW-GC, such as Bigloo.
>
> However, it doesn't worry me as much as the current GC bugs (e.g., [0, 1]).
>
> Also, there are definite benefits to using a conservative GC for
> libguile, given how tightly it can be integrated with C (e.g., [2]).
>
My main concern is/was that by moving to a conservatice GC, and
_consequently changing the API of libguile to assume a conservative GC_
(as outlined in [2]), you get third code relying on that as well. This
would make it effectively impossible to ever switch back to a precise GC
without potentially breaking all third-party code using the libguile
API.

However, take that just as my < 2€-cent ;-).

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>

Re: BDW-GC branch updated

[dsmich]

---- Andreas Rottmann <a.rottmann@gmx.at> wrote: 
> ludo@gnu.org (Ludovic Courtès) writes:
> > Also, there are definite benefits to using a conservative GC for
> > libguile, given how tightly it can be integrated with C (e.g., [2]).
> >
> My main concern is/was that by moving to a conservatice GC, and
> _consequently changing the API of libguile to assume a conservative GC_
> (as outlined in [2]), you get third code relying on that as well. This
> would make it effectively impossible to ever switch back to a precise GC
> without potentially breaking all third-party code using the libguile
> API.

Isn't the current gc in guile already conservative?

-Dale

Re: BDW-GC branch updated

[Ludovic Courtès]

Hi,

Andreas Rottmann <a.rottmann@gmx.at> writes:

> My main concern is/was that by moving to a conservatice GC, and
> _consequently changing the API of libguile to assume a conservative GC_
> (as outlined in [2]), you get third code relying on that as well. This
> would make it effectively impossible to ever switch back to a precise GC
> without potentially breaking all third-party code using the libguile
> API.

The current GC is "semi-conservative" in that it scans the C stack and
the "cell heap" for pointers, but nothing beyond that.  The current API
is built around this assumption.

BDW-GC is conservative: it scans everything, unless it's told otherwise
(e.g., `GC_malloc_atomic ()').  So you're right, the BDW-GC branch does
introduce API changes to take advantage of this, the most important of
which being that `scm_gc_malloc ()' memory is scanned and automatically
managed.

It's true that going back to the current "semi-precise" GC API would be
hard once we've made that change.

However, using a fully conservative GC makes it much easier to interact
with C, which is one of Guile's main goals as an extension language.

Furthermore, the concern about the risk of excess data retention caused
by conservative scanning holds as well for the current semi-conservative
GC.

Thus I believe the advantages of a conservative GC for Guile outweighs
its presumed disadvantages.

Thanks,
Ludo'.

Re: BDW-GC branch updated

[Neil Jerram]

ludo@gnu.org (Ludovic Courtès) writes:

>>> So now is a good time to test it and report back!  It requires libgc 7.1
>>> or later, which isn't packaged in Debian, although it was released in
>>> May 2008.
>>>
>> It's in experimental since recently; I assume its maintainer will upload
>> to unstable soonish.
>
> Good.

I just installed libgc1c2 and libgc-dev (both 1:7.1-3) on my Debian
stable/testing machine.  Apparently no problem there.

But there's still no pkgconfig for libgc, and so 

PKG_CHECK_MODULES([BDW_GC], [bdw-gc])

fails:

checking for BDW_GC... configure: error: Package requirements (bdw-gc) were not met:

No package 'bdw-gc' found

Am I missing some easy solution?

(I haven't tried the approach of setting BDW_GC_CFLAGS and BDW_GC_LIBS
yet.)

    Neil

Re: BDW-GC branch updated

[Ludovic Courtès]

Hi Neil,

Neil Jerram <neil@ossau.uklinux.net> writes:

> I just installed libgc1c2 and libgc-dev (both 1:7.1-3) on my Debian
> stable/testing machine.  Apparently no problem there.
>
> But there's still no pkgconfig for libgc, and so 
>
> PKG_CHECK_MODULES([BDW_GC], [bdw-gc])
>
> fails:

I checked the upstream tarballs and both 7.0 and 7.1 come with
‘bdw-gc.pc.in’.  Thus I suspect this is a packaging issue.  Can you
report it on the Debian side?

Thanks,
Ludo’.

Re: BDW-GC branch updated

[Andreas Rottmann]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Neil,
>
> Neil Jerram <neil@ossau.uklinux.net> writes:
>
>> I just installed libgc1c2 and libgc-dev (both 1:7.1-3) on my Debian
>> stable/testing machine.  Apparently no problem there.
>>
>> But there's still no pkgconfig for libgc, and so 
>>
>> PKG_CHECK_MODULES([BDW_GC], [bdw-gc])
>>
>> fails:
>
> I checked the upstream tarballs and both 7.0 and 7.1 come with
> ‘bdw-gc.pc.in’.  Thus I suspect this is a packaging issue.  Can you
> report it on the Debian side?
>
I've taken the liberty to do so:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=546833

Regards, Rotty
-- 
Andreas Rottmann -- <http://rotty.yi.org/>

Re: BDW-GC branch updated

[Neil Jerram]

Andreas Rottmann <a.rottmann@gmx.at> writes:

>> I checked the upstream tarballs and both 7.0 and 7.1 come with
>> ‘bdw-gc.pc.in’.  Thus I suspect this is a packaging issue.  Can you
>> report it on the Debian side?
>>
> I've taken the liberty to do so:
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=546833
>
> Regards, Rotty

Thanks!  FWIW, I'm now building successfully with

export BDW_GC_CFLAGS=-I/usr/include
export BDW_GC_LIBS=-lgc

     Neil

Re: BDW-GC branch updated

[Andy Wingo]

Hi Andreas,

On Tue 18 Aug 2009 14:19, Andreas Rottmann <a.rottmann@gmx.at> writes:

> Will going from a precise GC to BDW-GC not have drawbacks? IIRC, the PLT
> people went in the opposite direction. A quick google turned up this:
>
> http://www.cs.brown.edu/pipermail/plt-scheme/2006-June/013840.html
>
> I wonder if the reasons for switching to a precise GC listed in there
> will also apply to Guile.

I'm sure they do.

So, as you're probably aware, Guile's GC right now is something of a
hybrid. It conservatively scans the stack, but precisely marks the heap.
BDW scans the heap with less precision. Of course there are pointerless
blocks and custom mark functions in BDW as well.

I was concerned about this issue myself. Of course a moving GC is a
better option in the long run. But it seems an impossible change to
make, given Guile's long and public history as a C library.

BDW-GC is simply a better conservative GC than Guile's current
conservative GC.

But to take a specific example: I don't want Guile to be the project
that broke Emacs. Emacs has a precise GC, though not a moving GC iirc,
and if conservative collection broke long-running emacs sessions, that
would be Bad. But we shouldn't have any problem with Emacs. There will
be almost nothing on the C stack for any length of time. We control all
of the heap as well. We have pointerless mark functions in place where
necessary. Only tagged values make it onto the VM stack. So it is highly
unlikely to ever hit those corner cases PLT seemed to hit all the time.

Anyway, need to head out. Your concerns are important ones. My opinion
is that BDW-GC sounds like the right choice for the next 10 years of
Guile.

Andy
-- 
http://wingolog.org/