[0/10] threads for emacs

[0/10] threads for emacs

[Tom Tromey]

Here is a somewhat different version of the threads patches for Emacs.

I finally admitted to myself that I will never find the time or mental
energy to update the complicated-but-efficient approach to let bindings
that I had originally planned.  Instead, this series implements
something much slower, but also simpler -- it conceptually unwinds the
bindings when a thread loses the global lock, and rewinds them when a
thread acquires the lock.  Hopefully some motivated person can improve
this situation.

I'm posting this now not because it is complete, but because it is
complete enough to be tried out and discussed.  I'm sure it has bugs.
It is also missing ChangeLog entries and documentation.  Plus I have a
to-do list...

I would appreciate your reasoned comments.

Tom

Re: [0/10] threads for emacs

[Tom Tromey]

Tom> Plus I have a to-do list...

I thought it would be worthwhile to post it.

First, I think the patches I've posted are all (except maybe #9 and #10)
necessary for a first cut at threads in Emacs.

There are a few other things I think are necessary before the feature
can be easily useful:

* Documentation

* Emacs Lisp condition variable code.  This is probably not too hard.

* Changes to the process code so that a process can be restricted to
  having its output accepted by a single thread.  (In fact I think the
  default should be to do this for the creating thread.)  This avoids
  introducing bugs into lisp code that may currently make assumptions
  about let bindings visible to process filters.

* Hook into the Emacs I/O code so that the global lock is released when
  a thread enters 'select' (and maybe other primitives).  I think this
  is a bit harder than it sounds, mostly because I think you don't want
  multiple threads to select on the same fds.  It probably requires the
  process changes.

* Figure out what, if anything, to do about the lisp debugger,
  redisplay, and the keyboard when running a "non-background" thread.

* More tests


Some nice-to-have things:

* Have mutex-lock automatically detect deadlocks.

* Add a mutex function to see what thread owns a lock.
  Less sure about this one, but it seems handy for debugging.

* See if we can make it so there is a thread per keyboard.

* (provide 'thread) at the right spot

* Some convenience macros, like 'with-mutex'

* Change gnus so 'g' runs in the background.  Isn't that what this is
  really all about?


The far-flung future, aka stuff I don't anticipate getting to:

* Redo the bindings code to be more efficient

* Change thread switching to happen more often, say every N calls to
  QUIT.

* Eventually, remove the interpreter lock.  I think the current design
  still allows this (though some details in the current code would have
  to be changed).  This is very tricky though.

Tom

Re: [0/10] threads for emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

On 8/9/2012 12:59 PM, Tom Tromey wrote:
> Tom> Plus I have a to-do list...

This patch set is powerful enough to support Python-style generators, yes?
Coroutines would be insanely useful for fontification. I have a half-working
parser combinators library right now, but it's very much complicated by having
to compile everything down to (essentially) continuation-passing style so that
we can execute the code in small chunks and not block the UI.

Being able to write fontification code as a coroutine would be very nice indeed.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [0/10] threads for emacs

[Dmitry Antipov]

On 08/09/2012 11:36 PM, Tom Tromey wrote:

> Here is a somewhat different version of the threads patches for Emacs.

Great job.

I'm trying to improve GC (by making it generational at first), and it's
almost obvious that full-featured multithreading requires substantial
support from GC; our final goal might be a fully parallel GC performed
simultaneously by the all threads, or special GC thread(s) to perform
an incremental collection, or whatever else. Do you have any thoughts
about how the GC should be improved for the sake of threading support?

Dmitry

Re: [0/10] threads for emacs

[Tom Tromey]

Daniel> This patch set is powerful enough to support Python-style
Daniel> generators, yes?

Yeah, I think so.

Tom

Re: [0/10] threads for emacs

[Tom Tromey]

>>>>> "Dmitry" == Dmitry Antipov <dmantipov@yandex.ru> writes:

Dmitry> I'm trying to improve GC (by making it generational at first), and it's
Dmitry> almost obvious that full-featured multithreading requires substantial
Dmitry> support from GC; our final goal might be a fully parallel GC performed
Dmitry> simultaneously by the all threads, or special GC thread(s) to perform
Dmitry> an incremental collection, or whatever else. Do you have any thoughts
Dmitry> about how the GC should be improved for the sake of threading support?

The current patch series isn't too interesting from the GC point of
view, I think, because only one thread runs Lisp at a time.

It would be interesting to run the GC concurrently with Lisp.  I think
the hard part here is ensuring that all GC-visible changes occur
atomically.  E.g., writing an untagged pointer to the heap would cause
problems.

This may already work "well enough".  It is hard to know for sure
though.  I think some static analysis would perhaps help.

Tom

Re: [0/10] threads for emacs

[Tom Tromey]

>>>>> "Tom" == Tom Tromey <tromey@redhat.com> writes:

Tom> * Changes to the process code so that a process can be restricted to
Tom>   having its output accepted by a single thread.  (In fact I think the
Tom>   default should be to do this for the creating thread.)  This avoids
Tom>   introducing bugs into lisp code that may currently make assumptions
Tom>   about let bindings visible to process filters.

Tom> * Hook into the Emacs I/O code so that the global lock is released when
Tom>   a thread enters 'select' (and maybe other primitives).  I think this
Tom>   is a bit harder than it sounds, mostly because I think you don't want
Tom>   multiple threads to select on the same fds.  It probably requires the
Tom>   process changes.

I implemented these two.  I'll post the patches a bit later.

Tom

Re: [0/10] threads for emacs

[Tom Tromey]

Dmitry> I'm trying to improve GC (by making it generational at first), and it's
Dmitry> almost obvious that full-featured multithreading requires substantial
Dmitry> support from GC; our final goal might be a fully parallel GC performed
Dmitry> simultaneously by the all threads, or special GC thread(s) to perform
Dmitry> an incremental collection, or whatever else. Do you have any thoughts
Dmitry> about how the GC should be improved for the sake of threading support?

Tom> The current patch series isn't too interesting from the GC point of
Tom> view, I think, because only one thread runs Lisp at a time.

I was thinking more about your question...

If the GC could provide a way to mark the stack without needing
flush_stack_call_func, that would be a big improvement.  Right now any
spot that can release the global lock has to call through
flush_stack_call_func and provide a helper function.  This is a bit of a
pain.

Tom

Re: [0/10] threads for emacs

[Wojciech Meyer]

Daniel Colascione <dancol@dancol.org> writes:

> Coroutines would be insanely useful for fontification. I have a half-working
> parser combinators library right now, but it's very much complicated by having
> to compile everything down to (essentially) continuation-passing style so that
> we can execute the code in small chunks and not block the UI.

You can implement co-operative threading without system threads,
assuming of course lexical binding. Your CPS is really a monadic
threading library.

--
Wojciech Meyer
http://danmey.org