System asyncs and mutexes: a combination prone to deadlocks

System asyncs and mutexes: a combination prone to deadlocks

[Mark H Weaver]

Hello all,

While working on making (ice-9 popen) thread-safe, I've discovered a
serious problem with system asyncs and mutexes.

System asyncs can run while mutexes are locked.  Asyncs can run
arbitrary scheme code, so of course mutexes will often be locked within
asyncs as well.  So what happens if an async tries to lock a mutex that
has already been locked by the same thread?  Deadlock, of course.

Recursive mutexes are not a solution.  They would avoid the deadlock,
but they would leave open the possibility of corrupted data structures,
because the async might be run while a data structure is in an
inconsistent state.  If the async tries to access that data structure,
things could get ugly.

In popen, there are data structures (the port table and the guardian)
that need to be locked both outside and within asyncs, so I addressed
the problem by blocking asyncs before grabbing the lock:

  (define-syntax-rule (with-popen-tables-locked e0 e ...)
    (call-with-blocked-asyncs
     (lambda ()
       (with-mutex popen-mutex e0 e ...))))

This prevents deadlock by this particular mutex, but what about all the
other mutexes used throughout Guile?

The deadlock I happen to be seeing during 'make check' is from the
'overrides_lock' in procprop.c, but there are scores of other mutexes
around the system that could cause the same problem.

It seems to me that system asyncs are a fundamentally flawed concept in
any system that uses mutexes.  They need to be run in a different thread
to prevent these deadlocks.

   Thoughts?
      Mark

Re: System asyncs and mutexes: a combination prone to deadlocks

[Chaos Eternal]

Mark,
 then what's the purpose that the asyncs supposed to be ?
 one thing i know which uses async is signal handler, something else?

BTW, i used to compare performance that using asyncs as an
inter-thread communication method. not good.

On Tue, Aug 20, 2013 at 11:00 AM, Mark H Weaver <mhw@netris.org> wrote:
> Hello all,
>
> While working on making (ice-9 popen) thread-safe, I've discovered a
> serious problem with system asyncs and mutexes.
>
> System asyncs can run while mutexes are locked.  Asyncs can run
> arbitrary scheme code, so of course mutexes will often be locked within
> asyncs as well.  So what happens if an async tries to lock a mutex that
> has already been locked by the same thread?  Deadlock, of course.
>
> Recursive mutexes are not a solution.  They would avoid the deadlock,
> but they would leave open the possibility of corrupted data structures,
> because the async might be run while a data structure is in an
> inconsistent state.  If the async tries to access that data structure,
> things could get ugly.
>
> In popen, there are data structures (the port table and the guardian)
> that need to be locked both outside and within asyncs, so I addressed
> the problem by blocking asyncs before grabbing the lock:
>
>   (define-syntax-rule (with-popen-tables-locked e0 e ...)
>     (call-with-blocked-asyncs
>      (lambda ()
>        (with-mutex popen-mutex e0 e ...))))
>
> This prevents deadlock by this particular mutex, but what about all the
> other mutexes used throughout Guile?
>
> The deadlock I happen to be seeing during 'make check' is from the
> 'overrides_lock' in procprop.c, but there are scores of other mutexes
> around the system that could cause the same problem.
>
> It seems to me that system asyncs are a fundamentally flawed concept in
> any system that uses mutexes.  They need to be run in a different thread
> to prevent these deadlocks.
>
>    Thoughts?
>       Mark
>

Re: System asyncs and mutexes: a combination prone to deadlocks

[Mark H Weaver]

Chaos Eternal <chaoseternal@shlug.org> writes:
>  then what's the purpose that the asyncs supposed to be ?

https://www.gnu.org/software/guile/manual/html_node/Asyncs.html explains
what their purpose is supposed to be.

>  one thing i know which uses async is signal handler, something else?

The 'after-gc-hook' is also run via system asyncs.  (ice-9 popen) uses
this hook to check 'pipe-guardian' and thus reap pipes that are no
longer accessible, which involves clearing entries in 'port/pid-table',
a global hash table.

      Mark

Re: System asyncs and mutexes: a combination prone to deadlocks

[Mark H Weaver]

I wrote:
> System asyncs can run while mutexes are locked.  Asyncs can run
> arbitrary scheme code, so of course mutexes will often be locked within
> asyncs as well.  So what happens if an async tries to lock a mutex that
> has already been locked by the same thread?  Deadlock, of course.

I've mulled this over, and I think the solution will involve finding all
of the mutexes that are likely to be locked within asyncs without the
user's knowledge (i.e. essentially all mutexes used in core Guile, both
in C and in Scheme), and ensuring that asyncs are disabled before any of
those mutexes are locked.

What do you think?

      Mark

Re: System asyncs and mutexes: a combination prone to deadlocks

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> I wrote:
>> System asyncs can run while mutexes are locked.  Asyncs can run
>> arbitrary scheme code, so of course mutexes will often be locked within
>> asyncs as well.  So what happens if an async tries to lock a mutex that
>> has already been locked by the same thread?  Deadlock, of course.
>
> I've mulled this over, and I think the solution will involve finding all
> of the mutexes that are likely to be locked within asyncs without the
> user's knowledge (i.e. essentially all mutexes used in core Guile, both
> in C and in Scheme), and ensuring that asyncs are disabled before any of
> those mutexes are locked.
>
> What do you think?

(At the risk of looking silly:)

Alternately, what about documenting system asyncs as having restrictions
on the type of code they can run?

For example, can you think of a way to achieve what you wanted in (ice-9
popen) without resorting to system asyncs?

My 2¢,
Ludo’.