SCM_DEFER_INTS versus error

SCM_DEFER_INTS versus error

[Kevin Ryde]

I noticed in the little program below, the thread seems to get hung
after an error in the mainline has done an SCM_DEFER_INTS in strptime
then escaped with an error, apparently not doing an SCM_ALLOW_INTS.

Is there a theory on what should happen with this?  Is an error meant
to re-allow ints or should code be careful to do an ALLOW after any
DEFER?  The latter no doubt makes sense irrespective of what an error
throw does.

Incidentally, I don't think I understand why current-time has a
SCM_DEFER_INTS.  A simple call to time() ought to be safe shouldn't
it?




(use-modules (ice-9 threads))

(begin-thread
 (display "thread started\n")
 (sleep 2)
 (display "thread awake again\n")
 
 (display "thread trying current-time\n")
 (current-time)
 (display "thread done\n"))


(sleep 1)
(catch #t
       (lambda () (strptime "xxx" "%Y"))
       (lambda args
	 (display "main caught strptime error\n")))

(sleep 10)
(display "main exiting\n")


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: SCM_DEFER_INTS versus error

[Marius Vollmer]

Kevin Ryde <user42@zip.com.au> writes:

> I noticed in the little program below, the thread seems to get hung
> after an error in the mainline has done an SCM_DEFER_INTS in strptime
> then escaped with an error, apparently not doing an SCM_ALLOW_INTS.
>
> Is there a theory on what should happen with this?  Is an error meant
> to re-allow ints or should code be careful to do an ALLOW after any
> DEFER?  The latter no doubt makes sense irrespective of what an error
> throw does.

The whole DEFER/ALLOW business is anachronistic (in my view at least)
and should go away.  Originally, it was used to mark sections of code
that could not tolerate being interrupted, at a time when POSIX
signals could run Scheme code right from the signal handler and that
Scheme code could invoke continuations or throw to a catch.
SCM_DEFER_INTS would, well, defer the execution of signal handlers
until the next ALLOW_INTS.  Originally, it had nothing to do with
threads.

We have a different model for signal delivery now: Scheme signal
handlers are always deferred and are run by SCM_TICK, when it is safe
to do so.

So there no longer is the danger of code being interrupted in a
massive way (of course, C signal handlers still run asynchronously,
but they are careful not to mess things up).

With concurrent threads, we need to protect global data structures,
but I would say that the existing DEFER/ALLOW markup is not the right
tool for this.  Right now (if I'm still uptodate), only one thread can
execute 'in Guile', and thus we need no more explicit protection than
for the old coop threads.

Pragmatically, I don't think we can rely on DEFER/ALLOW to always come
in perfectly balanced pairs and I don't think it is worthwhile to make
sure that they do.  The way DEFER/ALLOW are defined now (as
locking/unlocking a mutex) is subtly different from what they did
previously (setting/clearing a flag in a non-nested way).

I don't think we should keep the current meaning of DEFER/ALLOW.
Instead, we should make them noops and deprecate their usage.  We
should be able to make them noops right now.  Mikael, do you agree?

> Incidentally, I don't think I understand why current-time has a
> SCM_DEFER_INTS.  A simple call to time() ought to be safe shouldn't
> it?

Yes.  Traditionally, all calls to functions outside of libguile were
wrapped in DEFER/ALLOW since we didn't want external functions to be
interrputed by Scheme signal handlers.  It was not to ensure proper
behavior with preemptive threads.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: SCM_DEFER_INTS versus error

[Tom Lord]

    > From: Marius Vollmer <mvo@zagadka.de>

Have things _really_ diverged so far that the following no longer
applies?   Just a word of caution:



    > The whole DEFER/ALLOW business is anachronistic (in my view at least)
    > and should go away.  Originally, it was used to mark sections of code
    > that could not tolerate being interrupted, at a time when POSIX
    > signals could run Scheme code right from the signal handler and that
    > Scheme code could invoke continuations or throw to a catch.

More specifically, they marked segments of code during which the heap
and flow-control could be in an inconsistent state as far as the usual
macros, gc, etc. were concerned.  That's an "extended" notion of
"inconsitent state" -- it included data structures and system state
that most of scheme didn't care about at all but that had to be
correlated with scheme heap state and flow of control.

    > SCM_DEFER_INTS would, well, defer the execution of signal handlers
    > until the next ALLOW_INTS.  Originally, it had nothing to do with
    > threads.

It did more than that, at least in the late days of when I was
maintainer.  For example, some C functions were safe to invoke within
the dynamic context of DEFER/ALLOW, others were not.  It was a handy
hack, that shook out many bugs, to:

      a) add a declaration in the body of each function that indicated
         when it could be safely run

      b) write an analyzer that roughly parsed the C code, did flow
         analysis, and labeled each call as "known ok", "known bogus", 
         or "too complex to analyze".


(Have you dropped the SCM_INTS_{ENABLED,DISABLED,INDIFFERENT} decls?!?)


    > I don't think we should keep the current meaning of DEFER/ALLOW.
    > Instead, we should make them noops and deprecate their usage.  We
    > should be able to make them noops right now.  Mikael, do you agree?

Traditionally, as an example, between DEFER/ALLOW, the
pointer-to-malloced-data in an object such as string was not required
to be valid.   Consequently, GC had to be excluded between
DEFER/ALLOW.


    > We have a different model for signal delivery now: Scheme signal
    > handlers are always deferred and are run by SCM_TICK, when it is safe
    > to do so.

    > So there no longer is the danger of code being interrupted in a
    > massive way (of course, C signal handlers still run asynchronously,
    > but they are careful not to mess things up).

Isn't there still a danger of bogusly calling a function that can, for
example, invoke GC at a point in the code at which the heap is in a
bogus state?   DEFER/ALLOW is useful, at least, for finding that
statically or at least noticing it dynamically.

In short, there's an extensible bunch of invariants that characterize
the heap and flow-control state.  Modules that add new tyeps and
functions can add new invariants.  The dynamic segments between
DEFER/ALLOW are where (and only where) those invariants can be
violated.

One way to look at it that might be helpful:  you have a kind of
virtual machine with the scheme heap as its store.   That VM has an
infinitely extensible set of macro-instructions as new C code is
added.   C itself is a kind of "micro-code" and the DEFER/ALLOW pairs 
mark the boundaries between macro-isntructions.

Say, do you still have REDEFER/REALLOW?   

-t





_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: SCM_DEFER_INTS versus error

[Kevin Ryde]

Marius Vollmer <mvo@zagadka.de> writes:
>
> Right now (if I'm still uptodate), only one thread can
> execute 'in Guile',

Oh, I thought you'd said previously there could be concurrent such
threads.  (I'd meant to try to work up a section for the manual on
such things.)

> Yes.

I realized since posting, that time() on a DOS system might be
affected by the TZ changes made by the other stime.c functions.

I guess all that stuff ought to change to use a little mutex
controlling access to TZ.  getenv/putenv would have to cooperate with
that too so a mere temporary change is not seen or overridden.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: SCM_DEFER_INTS versus error

[Marius Vollmer]

Tom Lord <lord@emf.net> writes:

>     > From: Marius Vollmer <mvo@zagadka.de>
>
> Have things _really_ diverged so far that the following no longer
> applies?

Yes, I would say so.

>     > The whole DEFER/ALLOW business is anachronistic (in my view at least)
>     > and should go away.  Originally, it was used to mark sections of code
>     > that could not tolerate being interrupted, at a time when POSIX
>     > signals could run Scheme code right from the signal handler and that
>     > Scheme code could invoke continuations or throw to a catch.
>
> More specifically, they marked segments of code during which the heap
> and flow-control could be in an inconsistent state as far as the usual
> macros, gc, etc. were concerned.  That's an "extended" notion of
> "inconsitent state" -- it included data structures and system state
> that most of scheme didn't care about at all but that had to be
> correlated with scheme heap state and flow of control.

Yes, your are right of course.  Not only interrupts could mess things
up, but they were the only asynchronous source of lossage.  The only
other source would be a programming error, right?

> (Have you dropped the SCM_INTS_{ENABLED,DISABLED,INDIFFERENT} decls?!?)

I didn't even know they were there, at some time!  So, yes, they are
gone, too.

> Traditionally, as an example, between DEFER/ALLOW, the
> pointer-to-malloced-data in an object such as string was not required
> to be valid.   Consequently, GC had to be excluded between
> DEFER/ALLOW.

We now do this with careful coding only.  Still having the DEFER/ALLOW
markups in the code is helpful for statically checking this, as you
say.  Hmm...

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: SCM_DEFER_INTS versus error

[Marius Vollmer]

Kevin Ryde <user42@zip.com.au> writes:

> Marius Vollmer <mvo@zagadka.de> writes:
>>
>> Right now (if I'm still uptodate), only one thread can
>> execute 'in Guile',
>
> Oh, I thought you'd said previously there could be concurrent such
> threads.  (I'd meant to try to work up a section for the manual on
> such things.)

What are you referring to precisely?  We do use concurrent threads,
but we (currently) restrict them to cooperate so that only one of them
has access to Guile data structures at any one time.  (We have the
equivalent of the Big Kernel Lock.)  When a thread might block or has
executed long enough, it leaves Guile-mode temporarily, allowing the
next thread to execute.

(I think. I have to admit, that I am probably not fully uptodate with
the details, as Mikael has done the most work on this.)

>> Yes.
>
> I realized since posting, that time() on a DOS system might be
> affected by the TZ changes made by the other stime.c functions.
>
> I guess all that stuff ought to change to use a little mutex
> controlling access to TZ.  getenv/putenv would have to cooperate with
> that too so a mere temporary change is not seen or overridden.

Blech.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: SCM_DEFER_INTS versus error

[Mikael Djurfeldt]

Marius Vollmer <mvo@zagadka.de> writes:

> Kevin Ryde <user42@zip.com.au> writes:
>
>> Marius Vollmer <mvo@zagadka.de> writes:
>>>
>>> Right now (if I'm still uptodate), only one thread can
>>> execute 'in Guile',
>>
>> Oh, I thought you'd said previously there could be concurrent such
>> threads.  (I'd meant to try to work up a section for the manual on
>> such things.)
>
> What are you referring to precisely?  We do use concurrent threads,
> but we (currently) restrict them to cooperate so that only one of them
> has access to Guile data structures at any one time.  (We have the
> equivalent of the Big Kernel Lock.)  When a thread might block or has
> executed long enough, it leaves Guile-mode temporarily, allowing the
> next thread to execute.

Well, that was the situation with your COPT threads.  The current
PTHREADS thread support of HEAD actually allow true concurrent access
to Guile data structures.  The "kernel lock" is only used to force
single-threaded GC.  The rest of the time, threads run in parallel.

It has been tested with promising results on an dual-CPU SMP machine.
I now have access to a four-CPU machine and hope to be running Guile
on it in the near future.

The concept of "Guile-mode" is still required, though: A thread must
be in Guile-mode in order to access Guile data structures.  This makes
it possible for the GC to guarantee that the HEAP is untouched during
GC and that all Guile data structures are in a well-defined state.

Mikael


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: SCM_DEFER_INTS versus error

[Marius Vollmer]

Mikael Djurfeldt <djurfeldt@nada.kth.se> writes:

> Well, that was the situation with your COPT threads.

Ooops, yes, I could have known this...

> The current PTHREADS thread support of HEAD actually allow true
> concurrent access to Guile data structures.  The "kernel lock" is
> only used to force single-threaded GC.  The rest of the time,
> threads run in parallel.

Impressive.  I _am_ a bit nervous about programming in such an
environmnt, I have to say, but I think I just have to get used to it.
For Scheme code, this is no change compared to the copt model since
Scheme code had to expect preemption anyway.

-- 
GPG: D5D4E405 - 2F9B BCCC 8527 692A 04E3  331E FAF8 226A D5D4 E405


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel

Re: SCM_DEFER_INTS versus error

[Kevin Ryde]

Marius Vollmer <mvo@zagadka.de> writes:
>
> Kevin Ryde <user42@zip.com.au> writes:
>>
>> I guess all that stuff ought to change to use a little mutex
>> controlling access to TZ.  getenv/putenv would have to cooperate with
>> that too so a mere temporary change is not seen or overridden.
>
> Blech.

I put an entry in the TODO file for this.


_______________________________________________
Guile-devel mailing list
Guile-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/guile-devel