Killing off scm_init_guile for Guile 2.0 ?

Killing off scm_init_guile for Guile 2.0 ?

[Neil Jerram]

scm_init_guile has always been a bit problematic, as it requires lots
of heuristic and OS-dependent code to try to determine where the base
of the stack is.  It's never been formally deprecated, but we have
always advised people to use scm_boot_guile or scm_with_guile if they
can, and the manual says:

     The function `scm_init_guile' might not be available on all
     platforms since it requires some stack-bounds-finding magic that
     might not have been ported to all platforms that Guile runs on.
     Thus, if you can, it is better to use `scm_with_guile' or its
     variation `scm_boot_guile' instead of this function.

Now I've found another problem with it (while investigating Greg's
NetBSD "unlocking unlocked mutex" report), so I'm wondering if it's
time now (i.e. in Guile 2.0) to remove it.  Can you respond to this
thread if you think you would be inconvenienced by this, or are aware
of a library or application that currently uses scm_init_guile ?

The problem is that if a thread is initialized (for Guile) by calling
scm_init_guile, Guile doesn't push a cleanup function for it.  This
means that if that thread is later terminated while in Guile mode, it
will die holding its "heap" mutex, and sooner or later (at the next
GC) that will cause all other Guile processing to hang.  It isn't
possible for scm_init_guile to push a cleanup function, because the
pthreads API requires pthread_cleanup_push () and pthread_cleanup_pop
() to be paired in the same lexical scope.

Thoughts?  On the other hand, does moving to BDW-GC take a lot of this
out of our hands anyway?  In any case, it would be interesting to hear
whether people think that scm_init_guile is really needed.

Regards,
       Neil

Re: Killing off scm_init_guile for Guile 2.0 ?

[Neil Jerram]

2009/1/15 Neil Jerram <neiljerram@googlemail.com>:
>
> Thoughts?  On the other hand, does moving to BDW-GC take a lot of this
> out of our hands anyway?

From a quick look, it seems the answer to that is "yes", and hence
moving to BDW-GC will mean that we can keep scm_init_guile with no
cost.  So this is probably all moot... but it would still be
interesting to hear if anyone has related thoughts.

      Neil

Re: Killing off scm_init_guile for Guile 2.0 ?

[Linas Vepstas]

I feel obligated to respond, having made all sorts of noise.

2009/1/15 Neil Jerram <neiljerram@googlemail.com>:

> whether people think that scm_init_guile is really needed.

kill it. there seem to be perfectly adequate ways of
living without it.  Unfortunately, the current documentation
describing how to use guile with threads is confusing.
It is certainly the case that, for naive, new users,
scm_init_guile seems to be the easiest way to
get guile going in a thread. This makes it a popular
choice.  Its not until you dig in deeply, and discover
how guile actually works (and then think about it a bit),
that you discover that perhaps scm_init_guile wasn't
the right choice. And then you have to refactor your
code ... possibly in large ways ...

So, the real question is -- how many existing guile
apps call scm_init_guile()?

On the other hand, breaking them by removing
scm_init_guile is possibly a good thing ... it will
probably cause them to fix bugs that were lurking
and waiting to bite.

--linas

Re: Killing off scm_init_guile for Guile 2.0 ?

[Ludovic Courtès]

Hi!

"Neil Jerram" <neiljerram@googlemail.com> writes:

> 2009/1/15 Neil Jerram <neiljerram@googlemail.com>:
>>
>> Thoughts?  On the other hand, does moving to BDW-GC take a lot of this
>> out of our hands anyway?
>
> From a quick look, it seems the answer to that is "yes", and hence
> moving to BDW-GC will mean that we can keep scm_init_guile with no
> cost.  So this is probably all moot... but it would still be
> interesting to hear if anyone has related thoughts.

Indeed, `scm_i_thread' no longer has a `heap_mutex' field (how
relieving!  ;-)).

Ludo'.

Re: Killing off scm_init_guile for Guile 2.0 ?

[Ludovic Courtès]

Hi,

Linas Vepstas <linasvepstas@gmail.com> writes:

> So, the real question is -- how many existing guile
> apps call scm_init_guile()?

The Evil Empire's Codesearch reveals that yes, a number of applications
use it:

  http://google.com/codesearch?q=scm_init_guile&hl=en&btnG=Search+Code

This may be partly due to the fact that the name is more natural than,
say, `scm_with_guile'.

Thanks,
Ludo'.

Re: Killing off scm_init_guile for Guile 2.0 ?

[Bill Schottstaedt]

in that list, the Snd xen.c case is not a true use -- that macro
is never true (because I apparently forgot to include it in my
configure script), so the code is never used.  I probably had
gh_init 10 years ago, then changed it to scm_init_guile --
can't remember now.

Re: Killing off scm_init_guile for Guile 2.0 ?

[Kjetil S. Matheussen]

Linas Vepstas:
>
> I feel obligated to respond, having made all sorts of noise.
>
> 2009/1/15 Neil Jerram <neiljerram@googlemail.com>:
>
>> whether people think that scm_init_guile is really needed.
>
> kill it. there seem to be perfectly adequate ways of
> living without it.  Unfortunately, the current documentation
> describing how to use guile with threads is confusing.
> It is certainly the case that, for naive, new users,
> scm_init_guile seems to be the easiest way to
> get guile going in a thread. This makes it a popular
> choice.  Its not until you dig in deeply, and discover
> how guile actually works (and then think about it a bit),
> that you discover that perhaps scm_init_guile wasn't
> the right choice. And then you have to refactor your
> code ... possibly in large ways ...
>
> So, the real question is -- how many existing guile
> apps call scm_init_guile()?
>
> On the other hand, breaking them by removing
> scm_init_guile is possibly a good thing ... it will
> probably cause them to fix bugs that were lurking
> and waiting to bite.
>

Snd uses scm_init_guile when compiled as a Pd external.
Pd is a graphical programming language for audio processing,
and "externals" are linked into the program during runtime.

When Snd is a Pd external, it has to run Guile in a separate
thread, so that it won't interrupt sound processing.

Is there an alternative to using scm_init_guile() for this
kind of usage?

Re: Killing off scm_init_guile for Guile 2.0 ?

[dsmich]

---- Neil Jerram <neiljerram@googlemail.com> wrote: 
> scm_init_guile has always been a bit problematic, as it requires lots
> of heuristic and OS-dependent code to try to determine where the base
> of the stack is.  It's never been formally deprecated, but we have
> always advised people to use scm_boot_guile or scm_with_guile if they
> can, and the manual says:
> 
>      The function `scm_init_guile' might not be available on all
>      platforms since it requires some stack-bounds-finding magic that
>      might not have been ported to all platforms that Guile runs on.
>      Thus, if you can, it is better to use `scm_with_guile' or its
>      variation `scm_boot_guile' instead of this function.

[ ... ]

> Thoughts?  On the other hand, does moving to BDW-GC take a lot of this
> out of our hands anyway?  In any case, it would be interesting to hear
> whether people think that scm_init_guile is really needed.


Iirc, scm_init_guile is mainly used when you do not have access to main(), for example writing a module for apache.  Generally, when you want to add Guile to an already existing application that has plugins or extension modules via .so's.

-Dale

Re: Killing off scm_init_guile for Guile 2.0 ?

[Linas Vepstas]

2009/1/16  <dsmich@roadrunner.com>:
> ---- Neil Jerram <neiljerram@googlemail.com> wrote:
>> scm_init_guile has always been a bit problematic, as it requires lots
>> of heuristic and OS-dependent code to try to determine where the base
>> of the stack is.  It's never been formally deprecated, but we have
>> always advised people to use scm_boot_guile or scm_with_guile if they

> Iirc, scm_init_guile is mainly used when you do not have access to main(), for example writing a module for apache.  Generally, when you want to add Guile to an already existing application that has plugins or extension modules via .so's.

This is an excellent example of where scm_init_guile
is exactly the *wrong* function to use. The problem is,
of course, that if you scm_init_guile in some .so,
you will accidentally place the entire system into guile
mode, and not just the .so, as intended.

Instead, you want to call scm_with_guile on entry to
the so, it will unwind on exit.

This could be cleared up by having the docs make
the case more forcefully.

--linas

Re: Killing off scm_init_guile for Guile 2.0 ?

[Zeeshan Ali (Khattak)]

Hi!
   I am back (in guile world). :)

On Sat, Jan 17, 2009 at 3:36 AM, Linas Vepstas <linasvepstas@gmail.com> wrote:
> 2009/1/16  <dsmich@roadrunner.com>:
>> ---- Neil Jerram <neiljerram@googlemail.com> wrote:
>>> scm_init_guile has always been a bit problematic, as it requires lots
>>> of heuristic and OS-dependent code to try to determine where the base
>>> of the stack is.  It's never been formally deprecated, but we have
>>> always advised people to use scm_boot_guile or scm_with_guile if they
>
>> Iirc, scm_init_guile is mainly used when you do not have access to main(), for example writing a module for apache.  Generally, when you want to add Guile to an already existing application that has plugins or extension modules via .so's.
>
> This is an excellent example of where scm_init_guile
> is exactly the *wrong* function to use. The problem is,
> of course, that if you scm_init_guile in some .so,
> you will accidentally place the entire system into guile
> mode, and not just the .so, as intended.

  Ok, my xchat-guile is exactly that kind of thing so i'll just
replace the call to scm_init_guile with scm_with_guile before the next
release.

> This could be cleared up by having the docs make
> the case more forcefully.

  Yeah makes sese!  Talking of docs, seems while i had been away the
reference manual has been getting some love. IIRC i wrote xchat-guile
mainly by poking around headers file and asking questions on IRC. Now
it will be easier for me to re-learn guile/scheme. :)

-- 
Regards,

Zeeshan Ali (Khattak)
FSF member#5124

Re: Killing off scm_init_guile for Guile 2.0 ?

[Neil Jerram]

2009/1/16 Kjetil S. Matheussen <k.s.matheussen@notam02.no>:
>
> Snd uses scm_init_guile when compiled as a Pd external.
> Pd is a graphical programming language for audio processing,
> and "externals" are linked into the program during runtime.
>
> When Snd is a Pd external, it has to run Guile in a separate
> thread, so that it won't interrupt sound processing.
>
> Is there an alternative to using scm_init_guile() for this
> kind of usage?

Yes: just use scm_with_guile() at all the entry points into
libguile-land.  Would that be possible?

Regards,
       Neil

Re: Killing off scm_init_guile for Guile 2.0 ?

[Neil Jerram]

2009/1/16 Linas Vepstas <linasvepstas@gmail.com>:
> I feel obligated to respond, having made all sorts of noise.
>
> 2009/1/15 Neil Jerram <neiljerram@googlemail.com>:
>
>> whether people think that scm_init_guile is really needed.
>
> kill it. there seem to be perfectly adequate ways of
> living without it.  Unfortunately, the current documentation
> describing how to use guile with threads is confusing.
> It is certainly the case that, for naive, new users,
> scm_init_guile seems to be the easiest way to
> get guile going in a thread. This makes it a popular
> choice.  Its not until you dig in deeply, and discover
> how guile actually works (and then think about it a bit),
> that you discover that perhaps scm_init_guile wasn't
> the right choice. And then you have to refactor your
> code ... possibly in large ways ...
>
> So, the real question is -- how many existing guile
> apps call scm_init_guile()?

A good handful, apparently.

> On the other hand, breaking them by removing
> scm_init_guile is possibly a good thing ... it will
> probably cause them to fix bugs that were lurking
> and waiting to bite.

I'm not sure... if the bugs aren't apparent in any way, they're not a
problem.  If the bugs are apparent, then I agree that moving away from
scm_init_guile could be part of the solution.

Assuming that we move to using BDW-GC, we can easily keep
scm_init_guile, and so I think we should.  We should also look at the
manual, though, to try to promote other ways in new projects.

Regards,
        Neil

Re: Killing off scm_init_guile for Guile 2.0 ?

[Neil Jerram]

2009/1/17 Linas Vepstas <linasvepstas@gmail.com>:
> 2009/1/16  <dsmich@roadrunner.com>:
>> ---- Neil Jerram <neiljerram@googlemail.com> wrote:
>>> scm_init_guile has always been a bit problematic, as it requires lots
>>> of heuristic and OS-dependent code to try to determine where the base
>>> of the stack is.  It's never been formally deprecated, but we have
>>> always advised people to use scm_boot_guile or scm_with_guile if they
>
>> Iirc, scm_init_guile is mainly used when you do not have access to main(), for example writing a module for apache.  Generally, when you want to add Guile to an already existing application that has plugins or extension modules via .so's.
>
> This is an excellent example of where scm_init_guile
> is exactly the *wrong* function to use.

I would say that it's not definitely wrong, but it's certainly risky.
The point is that "guile mode"-ness applies on a per-thread basis, so
scm_init_guile (on its own) can only be correct if you are sure that
the application is always going to execute your libguile-using code on
the same thread (the one where you called scm_init_guile).

> The problem is,
> of course, that if you scm_init_guile in some .so,
> you will accidentally place the entire system into guile
> mode, and not just the .so, as intended.

Or, to put that another way, the "guile mode"-ness persists on the
thread that called your libguile-using code, even after the thread has
returned back into the depths of the application.  That might not be a
problem, but it could be.  For example, threads in guile mode are
(currently; I'm not sure what BDW-GC will do) paused when a garbage
collection is needed; so activity on another guile thread could
suddenly block the application thread; or more likely the other way
round: the GC would not be able to proceed, because the application
thread would not actually pause.

> Instead, you want to call scm_with_guile on entry to
> the so, it will unwind on exit.
>
> This could be cleared up by having the docs make
> the case more forcefully.

Yes x 2.

   Neil

Re: Killing off scm_init_guile for Guile 2.0 ?

[Clinton Ebadi]

Neil Jerram <neiljerram@googlemail.com> writes:

>> The problem is,
>> of course, that if you scm_init_guile in some .so,
>> you will accidentally place the entire system into guile
>> mode, and not just the .so, as intended.
>
> Or, to put that another way, the "guile mode"-ness persists on the
> thread that called your libguile-using code, even after the thread has
> returned back into the depths of the application.  That might not be a
> problem, but it could be.  For example, threads in guile mode are
> (currently; I'm not sure what BDW-GC will do) paused when a garbage
> collection is needed; so activity on another guile thread could
> suddenly block the application thread; or more likely the other way
> round: the GC would not be able to proceed, because the application
> thread would not actually pause.

Does Guile mode actually have any meaning when using the BDW GC? The BDW
GC algorithm does not need to put other threads to sleep when sweeping;
it does this implicitly by causing the functions that register
collectable memory to block during the sweep phase as far as I can
tell. The libc malloc/free can continue doing whatever they want in the
meantime. A thread blocking for any reason should not be a problem as a
result (does Guile mode have another other restrictions?).

http://www.hpl.hp.com/personal/Hans_Boehm/gc/gcdescr.html has a good
overview of the BDW GC algorithm, and a few links to related papers
describing the algorithms used in more detail.

-- 
                   It's no contest, but we still race there                   
                Like the saintly tortoise and the godless hare                

Re: Killing off scm_init_guile for Guile 2.0 ?

[Kjetil S. Matheussen]

On Fri, 23 Jan 2009, Neil Jerram wrote:

> 2009/1/16 Kjetil S. Matheussen <k.s.matheussen@notam02.no>:
>>
>> Snd uses scm_init_guile when compiled as a Pd external.
>> Pd is a graphical programming language for audio processing,
>> and "externals" are linked into the program during runtime.
>>
>> When Snd is a Pd external, it has to run Guile in a separate
>> thread, so that it won't interrupt sound processing.
>>
>> Is there an alternative to using scm_init_guile() for this
>> kind of usage?
>
> Yes: just use scm_with_guile() at all the entry points into
> libguile-land.  Would that be possible?
>

Yes, that seems to work just fine. Thanks!

Re: Killing off scm_init_guile for Guile 2.0 ?

[Ludovic Courtès]

Hello,

Neil Jerram <neiljerram@googlemail.com> writes:

> For example, threads in guile mode are
> (currently; I'm not sure what BDW-GC will do) paused when a garbage
> collection is needed; so activity on another guile thread could
> suddenly block the application thread; or more likely the other way
> round: the GC would not be able to proceed, because the application
> thread would not actually pause.

BDW-GC doesn't require cooperation from all threads to operate.  Thus, a
blocking system call in one thread doesn't prevent GC operation, for
instance [0].  This makes `scm_without_guile ()' optional.

Thanks,
Ludo'.

[0] http://thread.gmane.org/gmane.comp.programming.garbage-collection.boehmgc/2319

Re: Killing off scm_init_guile for Guile 2.0 ?

[Ludovic Courtès]

Hi,

Clinton Ebadi <clinton@unknownlamer.org> writes:

> Does Guile mode actually have any meaning when using the BDW GC? The BDW
> GC algorithm does not need to put other threads to sleep when sweeping;
> it does this implicitly by causing the functions that register
> collectable memory to block during the sweep phase as far as I can
> tell. The libc malloc/free can continue doing whatever they want in the
> meantime. A thread blocking for any reason should not be a problem as a
> result (does Guile mode have another other restrictions?).

Exactly (sorry I hit "reply" before reading your message).

Thanks,
Ludo'.