* advice needed for multi-threading patch
@ 2009-08-26 4:38 Tom Tromey
2009-08-26 7:06 ` Ken Raeburn
` (2 more replies)
0 siblings, 3 replies; 54+ messages in thread
From: Tom Tromey @ 2009-08-26 4:38 UTC (permalink / raw
To: Emacs development discussions
I was inspired by Giuseppe Scrivano's work from last year:
http://lists.gnu.org/archive/html/emacs-devel/2008-11/msg01067.html
... so I started working on my own patch to implement multi-threading in
Emacs. I've tried to follow Stefan's advice as given in that thread.
In particular I've (initially) implemented cooperative multi-threading.
Some background for my particular question:
My implementation of dynamic binding uses a structure like so:
struct Lisp_ThreadLocal
{
enum Lisp_Misc_Type type : 16; /* = Lisp_Misc_ThreadLocal */
unsigned gcmarkbit : 1;
int spacer : 15;
Lisp_Object global;
Lisp_Object thread_alist;
};
... the idea being that a let-bound variable will be on 'thread_alist'
(keyed by the thread), and other threads will see the value in 'global'.
These objects are found in symbol value slots.
I managed to handle the V* global variables by writing some elisp that
changed every declaration of a variable mentioned in a DEFVAR_LISP from
Vfoo to impl_Vfoo and also emitted a new header with a lot of:
#define Vfoo *find_variable_location (&impl_Vfoo)
This was pretty simple and non-intrusive, in the sense that it is
totally automated, so I can easily reapply it as Emacs changes.
... which brings me to my problem. I'd also like to apply a similar
treatment to buffer-local variables. However, those do not have
convenient accessor macros, and before starting the laborious task of
wrapping all buffer field accesses, I thought I'd ask for advice. Is
this a sane way to proceed? Should I do something different? Could I
get such a patch in before the rest of this work, just to make my local
divergence smaller?
I'll also note that the above approach does not work for DEFVAR_INT. I
have a plan for those but I fear it is somewhat expensive. If you have
an idea...
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-26 4:38 advice needed for multi-threading patch Tom Tromey
@ 2009-08-26 7:06 ` Ken Raeburn
2009-08-26 14:52 ` Stefan Monnier
2009-08-26 16:08 ` Tom Tromey
2009-08-26 15:02 ` Stefan Monnier
2009-08-29 0:28 ` Giuseppe Scrivano
2 siblings, 2 replies; 54+ messages in thread
From: Ken Raeburn @ 2009-08-26 7:06 UTC (permalink / raw
To: Tom Tromey; +Cc: Emacs development discussions
On Aug 26, 2009, at 00:38, Tom Tromey wrote:
> In particular I've (initially) implemented cooperative multi-
> threading.
Interesting!
(BTW, my guile-emacs work is also coming along slowly [1], and Guile
supports multithreading, though my current variable-binding support
isn't thread-safe. OTOH, Daniel Kraft has been working on loading/
compiling basic elisp in Guile, with thread-safe dynamic bindings.
We've got some work to do to make these development projects converge
though. I hope you and Daniel and I aren't doing redundant work.)
> I managed to handle the V* global variables by writing some elisp that
> changed every declaration of a variable mentioned in a DEFVAR_LISP
> [....]
I've written similar elisp code for doing transformations of patterns
within the Emacs code -- mostly more localized things like "XCONS
(expr)->car" to "XCAR (expr)", but with support for somewhat complex
values of "expr" that have to have balanced parens, etc. I wonder if
anyone's looking at collecting elisp tools for C refactoring somewhere.
> This was pretty simple and non-intrusive, in the sense that it is
> totally automated, so I can easily reapply it as Emacs changes.
Same here. It can *look* very intrusive, even pervasive, when the
uses are scattered around the source tree, but simple transforms like
this aren't as scary as they might look to someone who doesn't
recognize what's happening.
> ... which brings me to my problem. I'd also like to apply a similar
> treatment to buffer-local variables. However, those do not have
> convenient accessor macros, and before starting the laborious task of
> wrapping all buffer field accesses, I thought I'd ask for advice. Is
> this a sane way to proceed? Should I do something different? Could I
> get such a patch in before the rest of this work, just to make my
> local
> divergence smaller?
Should I take this to mean you've already dealt with the buffer-local
variables created by make-variable-buffer-local, which don't live in
the buffer structure? If so, I wonder if you can make use of those
changes, by replacing XBUFFER(b)->thing with SYMBOL_VALUE(Qthing)
where Qthing holds a symbol that's marked as buffer-local; then all
the changes for the interaction between thread-local and buffer-local
bindings would be localized in indirect_variable and friends, which
would then have to deal with both Lisp_Misc_Buffer_Objfwd and
Lisp_Misc_Buffer_Local_Value. The Guile work I'm doing would also
probably benefit from such changes, down the road.
(One change I'm looking at in my Guile work is changing
indirect_variable to accept and return Lisp_Objects instead of struct
Lisp_Symbol pointers, for better GC safety. I doubt such a change
would significantly affect your work, though it may touch on
overlapping bits of code.)
Ken
[1] http://lists.gnu.org/archive/html/guile-devel/2009-08/
msg00163.html ; I also have a sort of "project page" I wrote up a
while back, but it's not quite current.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-26 7:06 ` Ken Raeburn
@ 2009-08-26 14:52 ` Stefan Monnier
2009-08-26 18:50 ` Ken Raeburn
2009-08-26 16:08 ` Tom Tromey
1 sibling, 1 reply; 54+ messages in thread
From: Stefan Monnier @ 2009-08-26 14:52 UTC (permalink / raw
To: Ken Raeburn; +Cc: Tom Tromey, Emacs development discussions
> (One change I'm looking at in my Guile work is changing indirect_variable to
> accept and return Lisp_Objects instead of struct Lisp_Symbol pointers, for
> better GC safety.
I've done the exact opposite change fairly recently, so it would be
going backwards.
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-26 4:38 advice needed for multi-threading patch Tom Tromey
2009-08-26 7:06 ` Ken Raeburn
@ 2009-08-26 15:02 ` Stefan Monnier
2009-08-26 15:31 ` Tom Tromey
2009-08-29 0:28 ` Giuseppe Scrivano
2 siblings, 1 reply; 54+ messages in thread
From: Stefan Monnier @ 2009-08-26 15:02 UTC (permalink / raw
To: Tom Tromey; +Cc: Emacs development discussions
> ... which brings me to my problem. I'd also like to apply a similar
> treatment to buffer-local variables. However, those do not have
> convenient accessor macros, and before starting the laborious task of
> wrapping all buffer field accesses, I thought I'd ask for advice. Is
> this a sane way to proceed? Should I do something different? Could I
> get such a patch in before the rest of this work, just to make my local
> divergence smaller?
I'm not sure I understand what you want to do and which problem you want
to solve. In my mind, in the first iteration of "adding multithreading
to Emacs", no buffer could have two active threads at the same time
(i.e. set-buffer would impose some kind of synchronization). In such
a context, I think that buffer-local variables (and fields in the buffer
structures) don't need any special treatment, right?
> I'll also note that the above approach does not work for DEFVAR_INT.
> I have a plan for those but I fear it is somewhat expensive. If you
> have an idea...
You can start with
#define Vfoo XINT(*find_variable_location (&impl_Vfoo))
and then fix up all the cases where the variable in accessed as an lvalue.
In general, I'm not opposed to introducing macros for such accesses in
the trunk, if it can make such work easier.
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-26 15:02 ` Stefan Monnier
@ 2009-08-26 15:31 ` Tom Tromey
2009-08-26 19:18 ` Stefan Monnier
0 siblings, 1 reply; 54+ messages in thread
From: Tom Tromey @ 2009-08-26 15:31 UTC (permalink / raw
To: Stefan Monnier; +Cc: Emacs development discussions
>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:
Tom> ... which brings me to my problem. I'd also like to apply a similar
Tom> treatment to buffer-local variables. However, those do not have
Tom> convenient accessor macros, and before starting the laborious task of
Tom> wrapping all buffer field accesses, I thought I'd ask for advice. Is
Tom> this a sane way to proceed? Should I do something different? Could I
Tom> get such a patch in before the rest of this work, just to make my local
Tom> divergence smaller?
Stefan> I'm not sure I understand what you want to do and which problem you want
Stefan> to solve.
I'm trying to properly implement per-thread let binding in the presence
of buffer-local variables.
I was planning to do this by inserting a thread-based lookup anywhere a
buffer-local was accessed. However, to do this, I need to insert a
function call anywhere a field corresponding to a DEFVAR_PER_BUFFER is
used.
So, e.g., I would like to change:
b->major_mode = Qfundamental_mode;
to:
BUFFER_MAJOR_MODE (b) = Qfundamental_mode;
Doing this transform for all 54 DEFVAR_PER_BUFFER fields is a big pain,
though, so I thought I'd ask about it first.
Stefan> In my mind, in the first iteration of "adding multithreading
Stefan> to Emacs", no buffer could have two active threads at the same time
Stefan> (i.e. set-buffer would impose some kind of synchronization). In such
Stefan> a context, I think that buffer-local variables (and fields in the buffer
Stefan> structures) don't need any special treatment, right?
I will try that.
Tom> I'll also note that the above approach does not work for DEFVAR_INT.
Tom> I have a plan for those but I fear it is somewhat expensive. If you
Tom> have an idea...
Stefan> You can start with
Stefan> #define Vfoo XINT(*find_variable_location (&impl_Vfoo))
Stefan> and then fix up all the cases where the variable in accessed as
Stefan> an lvalue. In general, I'm not opposed to introducing macros
Stefan> for such accesses in the trunk, if it can make such work easier.
Yeah, this was my plan :-)
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-26 7:06 ` Ken Raeburn
2009-08-26 14:52 ` Stefan Monnier
@ 2009-08-26 16:08 ` Tom Tromey
2009-08-26 18:50 ` Ken Raeburn
1 sibling, 1 reply; 54+ messages in thread
From: Tom Tromey @ 2009-08-26 16:08 UTC (permalink / raw
To: Ken Raeburn; +Cc: Emacs development discussions
>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:
Ken> I hope you and Daniel and I aren't doing redundant work.
Unless your changes apply to Emacs I don't think they are really
redundant. I think the key thing is agree how it ought to work, so we
end up with the same model, but I don't anticipate any problems there.
Ken> I've written similar elisp code for doing transformations of patterns
Ken> within the Emacs code -- mostly more localized things like "XCONS
Ken> (expr)->car" to "XCAR (expr)", but with support for somewhat complex
Ken> values of "expr" that have to have balanced parens, etc.
If you still have these, I would appreciate a copy, so that I don't have
to reinvent them. Are they in your git repository?
Ken> I wonder if anyone's looking at collecting elisp tools for C
Ken> refactoring somewhere.
There may be some on the CEDET site. I don't remember seeing any on the
wiki or elsewhere.
Ken> Should I take this to mean you've already dealt with the buffer-local
Ken> variables created by make-variable-buffer-local, which don't live in
Ken> the buffer structure?
Well... nominally. It is still a buggy mess.
Ken> If so, I wonder if you can make use of those
Ken> changes, by replacing XBUFFER(b)->thing with SYMBOL_VALUE(Qthing)
Ken> where Qthing holds a symbol that's marked as buffer-local; then all
Ken> the changes for the interaction between thread-local and buffer-local
Ken> bindings would be localized in indirect_variable and friends, which
Ken> would then have to deal with both Lisp_Misc_Buffer_Objfwd and
Ken> Lisp_Misc_Buffer_Local_Value.
I will have to think about this. This would imply removing all those
slots from struct buffer; I suppose my concern would be that the result
would be too inefficient.
I've also been considering a similar thing for the Vfoo globals: remove
them and have the C code always refer to them via SYMBOL_VALUE.
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-26 14:52 ` Stefan Monnier
@ 2009-08-26 18:50 ` Ken Raeburn
2009-08-27 3:12 ` Stefan Monnier
0 siblings, 1 reply; 54+ messages in thread
From: Ken Raeburn @ 2009-08-26 18:50 UTC (permalink / raw
To: Stefan Monnier; +Cc: Tom Tromey, Emacs development discussions
On Aug 26, 2009, at 10:52, Stefan Monnier wrote:
>> (One change I'm looking at in my Guile work is changing
>> indirect_variable to
>> accept and return Lisp_Objects instead of struct Lisp_Symbol
>> pointers, for
>> better GC safety.
>
> I've done the exact opposite change fairly recently, so it would be
> going backwards.
Oh, right... I meant to ask you at some point if there was a reason
for that other than streamlining performance a little.
In the guile-emacs code, lots of things can trigger GC that wouldn't
in Emacs, and it can happen in any thread; with some changes in the
works on the Guile side, it may not even require a thread to reach a
special synchronization point. The Guile model for random application-
defined object types involves a SCM value that points to an cell or
double-cell object, holding an integer indicating the type and 1-3
additional words that can be pointers or SCM values or whatever. So
if you've got a big structure like a buffer, one of those words points
to it. But if the SCM value isn't seen during GC, the cell doesn't
get marked and gets released, and the big structure gets freed up, and
much badness ensues.
Now, with the GC work in progress, it's possible that during
conservative stack scanning, the pointer to the big structure might be
enough to cause it to not get freed up; I'm not that familiar with the
new code. But the conservative approach seems to be to keep the SCM
(Lisp_Object) values live as long as the pointed-to data is needed.
It may also be that all of the callers will keep references to the
symbol object until the indirect_variable work is done, even with
uninterned symbols. If that's the case, maybe I don't need to be
concerned on this score.
However, there's also the little matter that eventually I want to make
Emacs use Guile symbols for its own symbols, for better interaction
between the languages. That means that "struct Lisp_Symbol" would go
away. If it's part of the internal function interfaces, that means
the Guile work has to have greater divergence from the upstream code.
Ken
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-26 16:08 ` Tom Tromey
@ 2009-08-26 18:50 ` Ken Raeburn
2009-08-27 5:07 ` Miles Bader
0 siblings, 1 reply; 54+ messages in thread
From: Ken Raeburn @ 2009-08-26 18:50 UTC (permalink / raw
To: Tom Tromey; +Cc: Emacs development discussions
On Aug 26, 2009, at 12:08, Tom Tromey wrote:
> Ken> I hope you and Daniel and I aren't doing redundant work.
>
> Unless your changes apply to Emacs I don't think they are really
> redundant. I think the key thing is agree how it ought to work, so we
> end up with the same model, but I don't anticipate any problems there.
Well, in their current form, my Emacs changes are kind of ugly, so
they won't be applied in their current form. Some of them are hacks
to get things working, so we can (eventually) evaluate the feasibility
of merging in Guile for real. It's not that I don't want to fix
things properly; but fixing some of them properly requires either
large-scale divergence from the Emacs repository -- in small but
pervasive and less well automated ways -- or changes to how things are
done in the Emacs repository that can't be enforced very cleanly, and
which my only argument for is, "well, my Guile work that may or may
not ever get folded in is going to need it."
The big issue -- the *huge* one -- is that all-bits-zero, the default
C initializer for a static Lisp_Object and the result of bzero, is an
integer-zero in Emacs currently and an invalid object representation
in Guile. There are lots of places where such values get used, or
Lisp_Object variables or fields are made visible to the GC system
before being properly "initialized" with valid representations.
I could go change all of the places like that to first initialize a
location with a valid object before using it, but I don't see how to
tweak the non-Guile version to help me catch such cases without, say,
changing Lisp_Int to use a nonzero tag, which would affect performance
(how much?), and would stop XINT/XUINT from being a shortcut for
"gimme all the bits" in the hash code and perhaps a few other places I
haven't tracked down yet.
Or, I could change things like staticpro to first store a valid object
value, which then requires that the assignment of the desired value in
the various init_foo functions *must* happen after the staticpro call,
whereas now the ordering is inconsistent. That might be more
enforceable, because I can make staticpro and the like complain if
they see anything other than all-bits-zero, which implies that the
code tried to store the desired value (other than integer-zero)
first. I'd still have to patch up anything using bzero to follow it
up with explicit initialization of some fields.
It's a bit annoying that the unexec setup basically prohibits the use
of static initializers for Lisp_Object variables because they might
become read-only; otherwise static initialization of V* and Q*
variables with a macro expanding to a valid object representation
would be an option.
I was figuring on pushing this work a bit further along, and having
something I could show off better, to make a stronger case for
integrating the guile-emacs work, before trying to argue for such
changes in the main Emacs code base. (Even if the decision eventually
went against integrating Guile specifically, I might be able to make a
case for going through a little pain to better isolate the Lisp engine
so it could perhaps be replaced someday by something else.) And in
the meantime, I'd prefer to keep it easier to resync with upstream
changes, by not rewriting all that code, just as long as I know
roughly what needs to be fixed later.
But, I digress. :-)
My own changes are, in part, to insulate the C Lisp implementation in
Emacs as much as possible from the rest of Emacs, and in part, to
replace the most fundamental layer -- representation, allocation and
GC -- with Guile. Whether this project eventually leads to replacing
the Lisp evaluator itself, I don't know; some people seem to think it
would be a great idea, and some seem to think it would be a bad idea.
I'm on the fence until I can see some performance numbers and serious
testing, and that requires a more or less complete implementation;
there's been some encouraging work happening on the Guile side, but
it's not done yet. But it's appealing to think we might someday be
able to have elisp available as an extension language for other
programs, via Guile; and if that happens, do we really need two
implementations? (And if we go with only one for both Emacs and
Guile, would it be the current Guile one, or taken from Emacs and
dropped into Guile? Dunno.)
I'm not worrying about things like multithreading, but my changes will
need to ensure that symbol value lookups and updates are sufficiently
isolated through macros or functions that it should be easy to slip in
thread-specific processing.
Daniel Kraft and others are working on the Guile elisp implementation,
which in part requires learning the specs of how elisp functions in
various corner cases as well as normal usage. There has been some
discussion of multithreading, but as far as I know there's currently
no implementation of buffer-local variables.
Daniel's work is using Guile's string and symbol representations. I
started with the "smob" application-defined-object handles so I could
store pointers to Lisp_String and Lisp_Symbol and so forth, and not
have to rewrite all that code right away; that's coming up. So
there's a bit of work needed before Daniel's work and mine can
possibly work together.
> Ken> I've written similar elisp code for doing transformations of
> patterns
> Ken> within the Emacs code -- mostly more localized things like "XCONS
> Ken> (expr)->car" to "XCAR (expr)", but with support for somewhat
> complex
> Ken> values of "expr" that have to have balanced parens, etc.
>
> If you still have these, I would appreciate a copy, so that I don't
> have
> to reinvent them. Are they in your git repository?
No, but I've got them floating around somewhere, and I'll send you a
copy.
They're not pretty, but they get the job done.
> I will have to think about this. This would imply removing all those
> slots from struct buffer; I suppose my concern would be that the
> result
> would be too inefficient.
Not necessarily. With Lisp_Misc_Buffer_Objfwd, one of the existing
fields is the offset into the buffer structure. Maybe that could be
where the thread-local object handle is stored.
> I've also been considering a similar thing for the Vfoo globals:
> remove
> them and have the C code always refer to them via SYMBOL_VALUE.
I think that might also simplify some things for my work.
Ken
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-26 15:31 ` Tom Tromey
@ 2009-08-26 19:18 ` Stefan Monnier
2009-09-18 22:59 ` Tom Tromey
0 siblings, 1 reply; 54+ messages in thread
From: Stefan Monnier @ 2009-08-26 19:18 UTC (permalink / raw
To: Tom Tromey; +Cc: Emacs development discussions
> I'm trying to properly implement per-thread let binding in the presence
> of buffer-local variables.
The problem with that is that we first need to figure out what its
semantics should be. And to me the semantics of a let-bound
buffer-local variable is very much unclear in the presence of threads
(it's already pretty messy even without threads).
So I'd rather disallow other threads from accessing this buffer's
variable while the let-binding is active.
> I was planning to do this by inserting a thread-based lookup anywhere a
> buffer-local was accessed. However, to do this, I need to insert a
> function call anywhere a field corresponding to a DEFVAR_PER_BUFFER is
> used.
> So, e.g., I would like to change:
b-> major_mode = Qfundamental_mode;
> to:
> BUFFER_MAJOR_MODE (b) = Qfundamental_mode;
> Doing this transform for all 54 DEFVAR_PER_BUFFER fields is a big pain,
> though, so I thought I'd ask about it first.
This may have a very significant performance impact. Now, maybe we'll
have to bite the bullet sooner or later. But I'd rather not yet.
This said, I'd accept a patch that turns all
.... b->major_mode ...
into
.... PER_BUFFER_VAR (b, major_mode) ...
or even with two different macros (one for lvalue, and one for rvalue).
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-26 18:50 ` Ken Raeburn
@ 2009-08-27 3:12 ` Stefan Monnier
2009-08-27 6:28 ` Ken Raeburn
0 siblings, 1 reply; 54+ messages in thread
From: Stefan Monnier @ 2009-08-27 3:12 UTC (permalink / raw
To: Ken Raeburn; +Cc: Tom Tromey, Emacs development discussions
> Oh, right... I meant to ask you at some point if there was a reason for that
> other than streamlining performance a little.
It has nothing to do with performance, and everything to do with code
cleanliness and robustness. Using Lisp_Symbol* rather than Lisp_Object
makes it clear (both to the reader and to the type-checker) that the
object received (or returned) can only be a symbol.
> Now, with the GC work in progress, it's possible that during conservative
> stack scanning, the pointer to the big structure might be enough to cause
> it to not get freed up; I'm not that familiar with the new code. But the
> conservative approach seems to be to keep the SCM (Lisp_Object) values live
> as long as the pointed-to data is needed.
Emacs's stack scanning already has to check both Lisp_Object values
(i.e. tagged pointers) and plain direct untagged pointers. This was
needed IIRC because even if the C code appears to hold on to
a Lisp_Object value, the optimizer may get rid of it and only keep the
untagged pointer.
So, I think Guile's stack scanning will have to do that as well, I think
(regardless of Emacs integration).
> However, there's also the little matter that eventually I want to make Emacs
> use Guile symbols for its own symbols, for better interaction between the
> languages. That means that "struct Lisp_Symbol" would go away. If it's
> part of the internal function interfaces, that means the Guile work has to
> have greater divergence from the upstream code.
We'll cross that bridge when we get there. I think we'll find that the
particular interface used by the function will be a very minor issue
(the whole function will probably need to be rewritten).
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-26 18:50 ` Ken Raeburn
@ 2009-08-27 5:07 ` Miles Bader
2009-08-27 6:39 ` Ken Raeburn
0 siblings, 1 reply; 54+ messages in thread
From: Miles Bader @ 2009-08-27 5:07 UTC (permalink / raw
To: Ken Raeburn; +Cc: Tom Tromey, Emacs development discussions
Ken Raeburn <raeburn@raeburn.org> writes:
> in part, to replace the most fundamental layer -- representation,
> allocation and GC -- with Guile.
Do we actually want to do this? How is guile regarded these days (back
in the day, it was a horrid bloated mess)?
-Miles
--
The trouble with most people is that they think with their hopes or
fears or wishes rather than with their minds. -- Will Durant
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-27 3:12 ` Stefan Monnier
@ 2009-08-27 6:28 ` Ken Raeburn
2009-08-27 17:02 ` Stefan Monnier
0 siblings, 1 reply; 54+ messages in thread
From: Ken Raeburn @ 2009-08-27 6:28 UTC (permalink / raw
To: Stefan Monnier; +Cc: Tom Tromey, Emacs development discussions
On Aug 26, 2009, at 23:12, Stefan Monnier wrote:
> Emacs's stack scanning already has to check both Lisp_Object values
> (i.e. tagged pointers) and plain direct untagged pointers. This was
> needed IIRC because even if the C code appears to hold on to
> a Lisp_Object value, the optimizer may get rid of it and only keep the
> untagged pointer.
Does that mean that GC_USE_GCPROS_AS_BEFORE doesn't work reliably? If
so, maybe we should get rid of it. (Though, it appears to be the
default.)
>> [...getting rid of struct Lisp_Symbol...]
>
> We'll cross that bridge when we get there. I think we'll find that
> the
> particular interface used by the function will be a very minor issue
> (the whole function will probably need to be rewritten).
Agreed. It wasn't particularly even important to bring up, except
that I think Tom may be looking at changing related code.
Ken
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-27 5:07 ` Miles Bader
@ 2009-08-27 6:39 ` Ken Raeburn
2009-08-27 6:50 ` Miles Bader
0 siblings, 1 reply; 54+ messages in thread
From: Ken Raeburn @ 2009-08-27 6:39 UTC (permalink / raw
To: Miles Bader; +Cc: Tom Tromey, Emacs development discussions
On Aug 27, 2009, at 01:07, Miles Bader wrote:
> Ken Raeburn <raeburn@raeburn.org> writes:
>> in part, to replace the most fundamental layer -- representation,
>> allocation and GC -- with Guile.
>
> Do we actually want to do this? How is guile regarded these days
> (back
> in the day, it was a horrid bloated mess)?
I still think of this project partly as being in the "let's see how it
works out and decide if we want to keep it" stage, or even "let's try
this big application and see what problems we find in Guile". So I'm
not quite ready to answer -- or ask -- if we want to do it. With a
bit more polish, and some more intense testing, and performance
analysis, maybe...
I can't really speak to how Guile is regarded; I do much more stuff in
the Emacs and Guile internals than with ordinary applications that
might use it, and I'm by no stretch of the imagination any kind of
Scheme expert. But I don't think how Guile is regarded should be as
important as how Guile *is*. If there are real problems with using it
in Emacs -- as opposed to everyone just remembering how it *used* to
be considered a "horrid bloated mess" -- then we can try to fix them
on the Guile side, or decide to drop the project.
It's had performance problems for some time, but recent work has made
some big improvements, putting it back in competition with some of the
other implementations, and there's a bit of talk about compiling not
just to VM byte codes but even to native code for some architectures
someday. It sounds a bit like pie-in-the-sky stuff, and some of these
sorts of things have moved very slowly in Guile development in the
past, but the guy doing the compiler and optimizer work right now is
making some good progress, and I wouldn't put it past him. There's
also some GC system work happening, but I haven't been following that
closely. And if we wind up in a situation where doing such work
benefits both Guile and Emacs because one is using the code from the
other, it *could* be good for both projects. (And, of course, if it
goes badly, it could make it harder to improve things all around.)
Thread support is another example -- if Emacs were already using Guile
underneath the Lisp engine, thread support would probably be quite a
bit easier.
My reasoning for trying this is in some ways political as well as
technical. Guile is touted as the GNU project's extension language.
Yet one of the highest-profile, most-extensible and most-extended
programs GNU ships doesn't use it, but instead uses a separate,
private implementation of a rather similar language. If "GNU's
extension language" isn't good enough for this application, and can't
be made good enough, then maybe we should declare it a failure as
"GNU's extension language" and either do something else or drop the
idea entirely. But I don't think we've gotten to that point.
And on the technical side, I'm trying to keep most of the work in the
Emacs "application-level" code non-specific to Guile, so if someone
decides to replace the Lisp engine with something other than Guile, my
changes to isolate the implementation details and assumptions made may
still not be wasted.
BTW, if anyone is interested in helping on the Guile-Emacs work, I'd
love to have some help. There's quite a number of pieces still to be
tackled, some purely on the Emacs side, others having to do with
making the interaction between the two work better....
Ken
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-27 6:39 ` Ken Raeburn
@ 2009-08-27 6:50 ` Miles Bader
2009-08-28 20:48 ` Juri Linkov
0 siblings, 1 reply; 54+ messages in thread
From: Miles Bader @ 2009-08-27 6:50 UTC (permalink / raw
To: Ken Raeburn; +Cc: Tom Tromey, Emacs development discussions
Ken Raeburn <raeburn@raeburn.org> writes:
> But I don't think how Guile is regarded should be as
> important as how Guile *is*.
How important is guile?
15 years ago there were grand plans, but... dunno about now.
It's a different world in many ways.
-miles
--
=====
(^o^;
(()))
*This is the cute octopus virus, please copy it into your sig so it can spread.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-27 6:28 ` Ken Raeburn
@ 2009-08-27 17:02 ` Stefan Monnier
0 siblings, 0 replies; 54+ messages in thread
From: Stefan Monnier @ 2009-08-27 17:02 UTC (permalink / raw
To: Ken Raeburn; +Cc: Tom Tromey, Emacs development discussions
>> Emacs's stack scanning already has to check both Lisp_Object values
>> (i.e. tagged pointers) and plain direct untagged pointers. This was
>> needed IIRC because even if the C code appears to hold on to
>> a Lisp_Object value, the optimizer may get rid of it and only keep the
>> untagged pointer.
> Does that mean that GC_USE_GCPROS_AS_BEFORE doesn't work reliably?
It's very little tested, but it should work reliably. The reason it
works reliably is that the GCPRO macros cause the Lisp_Object variables
to be accessed via "&" and their address is stored in a globally
accessible data structure, so the compiler cannot optimize those
variables away, or even keep them in registers.
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-27 6:50 ` Miles Bader
@ 2009-08-28 20:48 ` Juri Linkov
2009-08-28 22:15 ` Miles Bader
0 siblings, 1 reply; 54+ messages in thread
From: Juri Linkov @ 2009-08-28 20:48 UTC (permalink / raw
To: Miles Bader; +Cc: Tom Tromey, Ken Raeburn, Emacs development discussions
> How important is guile?
>
> 15 years ago there were grand plans, but... dunno about now.
> It's a different world in many ways.
AFAIK, Guile and Scheme as a whole still flourish these days,
and the only serious contender is Clojure that could integrate
with Emacs the same way as it does with Java.
--
Juri Linkov
http://www.jurta.org/emacs/
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-28 20:48 ` Juri Linkov
@ 2009-08-28 22:15 ` Miles Bader
2009-08-28 23:27 ` Juri Linkov
2009-08-29 20:21 ` Richard Stallman
0 siblings, 2 replies; 54+ messages in thread
From: Miles Bader @ 2009-08-28 22:15 UTC (permalink / raw
To: Juri Linkov; +Cc: Tom Tromey, Ken Raeburn, Emacs development discussions
Juri Linkov <juri@jurta.org> writes:
>> 15 years ago there were grand plans, but... dunno about now.
>> It's a different world in many ways.
>
> AFAIK, Guile and Scheme as a whole still flourish these days,
There are a huge number of scheme implementations now; even thinking
just about scheme, is guile still competitive?
> and the only serious contender is Clojure that could integrate
> with Emacs the same way as it does with Java.
Why? There are tons of random little languages around now, some very
popular (i.e., javascript), many not so much. Clojure is not scheme, so
why is it any better than those? [Compatibility with the java runtime
is important for some people, but AFAIK, not for Emacs!]
-Miles
--
.Numeric stability is probably not all that important when you're guessing.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-28 22:15 ` Miles Bader
@ 2009-08-28 23:27 ` Juri Linkov
2009-08-28 23:54 ` Miles Bader
2009-08-29 20:21 ` Richard Stallman
1 sibling, 1 reply; 54+ messages in thread
From: Juri Linkov @ 2009-08-28 23:27 UTC (permalink / raw
To: Miles Bader; +Cc: Tom Tromey, Ken Raeburn, Emacs development discussions
>>> 15 years ago there were grand plans, but... dunno about now.
>>> It's a different world in many ways.
>>
>> AFAIK, Guile and Scheme as a whole still flourish these days,
>
> There are a huge number of scheme implementations now; even thinking
> just about scheme, is guile still competitive?
Among all Scheme implementations Guile is in the same position
as bzr among all DVCSs - it is the "official" selection for the
GNU project.
>> and the only serious contender is Clojure that could integrate
>> with Emacs the same way as it does with Java.
>
> Why? There are tons of random little languages around now,
Clojure is a little language with a big impact.
> some very popular (i.e., javascript), many not so much.
JavaScript is not Lisp.
> Clojure is not scheme, so why is it any better than those?
Clojure is not Scheme, but Clojure is Lisp, like Emacs Lisp is.
> [Compatibility with the java runtime is important for some people, but
> AFAIK, not for Emacs!]
True, but I thought about replacing its Java VM part with Emacs.
Anyway, this point is moot, since Clojure concurrency relies on
Java threads, but Emacs has no threads yet. At least, Clojure
is a good example how concurrency with STM, agent, atoms and
dynamic var systems could be implemented in Emacs.
--
Juri Linkov
http://www.jurta.org/emacs/
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-28 23:27 ` Juri Linkov
@ 2009-08-28 23:54 ` Miles Bader
0 siblings, 0 replies; 54+ messages in thread
From: Miles Bader @ 2009-08-28 23:54 UTC (permalink / raw
To: Juri Linkov; +Cc: Tom Tromey, Ken Raeburn, Emacs development discussions
On Sat, Aug 29, 2009 at 8:27 AM, Juri Linkov<juri@jurta.org> wrote:
>> There are a huge number of scheme implementations now; even thinking
>> just about scheme, is guile still competitive?
>
> Among all Scheme implementations Guile is in the same position
> as bzr among all DVCSs - it is the "official" selection for the
> GNU project.
I'm not asking about labels, I'm asking about merit.
>>> and the only serious contender is Clojure that could integrate
>>> with Emacs the same way as it does with Java.
>>
>> Why? There are tons of random little languages around now,
>
> Clojure is a little language with a big impact.
Clojure has been very hyped, but it's still far from clear it will
have any lasting impact.
Anyway, clojure (or at least its engine) has merit, but it seems very
premature to call it "the only serious contender."
-Miles
--
Do not taunt Happy Fun Ball.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-26 4:38 advice needed for multi-threading patch Tom Tromey
2009-08-26 7:06 ` Ken Raeburn
2009-08-26 15:02 ` Stefan Monnier
@ 2009-08-29 0:28 ` Giuseppe Scrivano
2009-08-29 4:57 ` Tom Tromey
2 siblings, 1 reply; 54+ messages in thread
From: Giuseppe Scrivano @ 2009-08-29 0:28 UTC (permalink / raw
To: Tom Tromey; +Cc: Emacs development discussions
Hello Tom,
I only now saw this thread and multi-threaded Emacs is still in my
wish-list :-) Do you already have some working code? I was expecting
the final migration to BZR before start working again on it.
Last time that I rebased my patch, approximately two months ago, it was
still working and I think something can be re-used from it.
In my work, I didn't consider threads local storage for two reasons the
first and most important is that elisp packages (yet?) don't know about
it and wouldn't use; the second reason is that it adds a lot of
complexity while I don't see much benefits so I chose simplicity.
Probably threads local storage can be useful when the cooperative model
will be changed to something more serious, but that is another story :-)
Cheers,
Giuseppe
Tom Tromey <tromey@redhat.com> writes:
> I was inspired by Giuseppe Scrivano's work from last year:
>
> http://lists.gnu.org/archive/html/emacs-devel/2008-11/msg01067.html
>
> ... so I started working on my own patch to implement multi-threading in
> Emacs. I've tried to follow Stefan's advice as given in that thread.
> In particular I've (initially) implemented cooperative multi-threading.
>
> Some background for my particular question:
>
> My implementation of dynamic binding uses a structure like so:
>
> struct Lisp_ThreadLocal
> {
> enum Lisp_Misc_Type type : 16; /* = Lisp_Misc_ThreadLocal */
> unsigned gcmarkbit : 1;
> int spacer : 15;
> Lisp_Object global;
> Lisp_Object thread_alist;
> };
>
> ... the idea being that a let-bound variable will be on 'thread_alist'
> (keyed by the thread), and other threads will see the value in 'global'.
> These objects are found in symbol value slots.
>
> I managed to handle the V* global variables by writing some elisp that
> changed every declaration of a variable mentioned in a DEFVAR_LISP from
> Vfoo to impl_Vfoo and also emitted a new header with a lot of:
>
> #define Vfoo *find_variable_location (&impl_Vfoo)
>
> This was pretty simple and non-intrusive, in the sense that it is
> totally automated, so I can easily reapply it as Emacs changes.
>
> ... which brings me to my problem. I'd also like to apply a similar
> treatment to buffer-local variables. However, those do not have
> convenient accessor macros, and before starting the laborious task of
> wrapping all buffer field accesses, I thought I'd ask for advice. Is
> this a sane way to proceed? Should I do something different? Could I
> get such a patch in before the rest of this work, just to make my local
> divergence smaller?
>
> I'll also note that the above approach does not work for DEFVAR_INT. I
> have a plan for those but I fear it is somewhat expensive. If you have
> an idea...
>
> Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-29 0:28 ` Giuseppe Scrivano
@ 2009-08-29 4:57 ` Tom Tromey
0 siblings, 0 replies; 54+ messages in thread
From: Tom Tromey @ 2009-08-29 4:57 UTC (permalink / raw
To: Giuseppe Scrivano; +Cc: Emacs development discussions
>>>>> "Giuseppe" == Giuseppe Scrivano <gscrivano@gnu.org> writes:
Giuseppe> I only now saw this thread and multi-threaded Emacs is still
Giuseppe> in my wish-list :-) Do you already have some working code?
Yeah. Well, I did ... it is not working today because I am in the
middle of changing the variable binding implementation and implementing
buffer locking.
Giuseppe> In my work, I didn't consider threads local storage for two
Giuseppe> reasons the first and most important is that elisp packages
Giuseppe> (yet?) don't know about it and wouldn't use; the second reason
Giuseppe> is that it adds a lot of complexity while I don't see much
Giuseppe> benefits so I chose simplicity.
I suspect we may be talking about different things. When I say
thread-local storage, I just mean having 'let' bindings work properly
when two different threads use the same symbol name.
In case not: my goal was to write the smallest patch that would yield a
usefully multi-threaded Emacs. It seemed to me that the binding issue
had to be dealt with somehow, because ignoring it would mean that the
resulting Emacs could not run any real programs.
As I saw it, the choices were either to undo and redo all the bindings
at every context switch, or implement thread-local bindings. I looked
into both a bit, and thread-local bindings did not seem too hard, except
for defining the semantics of the interaction between multiple threads
and buffer-local variables.
Right now I have something similar to your patch (except using a single
thread-local current_thread pointer instead of arrays); plus changes to
the GC and to make DEFVAR_LISP work.
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-28 22:15 ` Miles Bader
2009-08-28 23:27 ` Juri Linkov
@ 2009-08-29 20:21 ` Richard Stallman
1 sibling, 0 replies; 54+ messages in thread
From: Richard Stallman @ 2009-08-29 20:21 UTC (permalink / raw
To: Miles Bader; +Cc: juri, tromey, raeburn, emacs-devel
There are a huge number of scheme implementations now; even thinking
just about scheme, is guile still competitive?
Guile is the GNU Scheme implementation; unless it is so
inadequate that we should drop it entirely, it should be the
one we use in Emacs (if we use one).
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-08-26 19:18 ` Stefan Monnier
@ 2009-09-18 22:59 ` Tom Tromey
2009-09-19 0:09 ` Stephen J. Turnbull
` (2 more replies)
0 siblings, 3 replies; 54+ messages in thread
From: Tom Tromey @ 2009-09-18 22:59 UTC (permalink / raw
To: Stefan Monnier; +Cc: Emacs development discussions
Tom> I'm trying to properly implement per-thread let binding in the presence
Tom> of buffer-local variables.
Stefan> The problem with that is that we first need to figure out what its
Stefan> semantics should be. And to me the semantics of a let-bound
Stefan> buffer-local variable is very much unclear in the presence of threads
Stefan> (it's already pretty messy even without threads).
Stefan> So I'd rather disallow other threads from accessing this buffer's
Stefan> variable while the let-binding is active.
I thought I'd post a status report on the multi-threading patch.
First, thanks to Giuseppe, it is all hosted on Gitorious:
http://gitorious.org/emacs-mt
'mt' is the branch I've been working on, he's done some additional fixes
and changes on g-exp. (I am behind on pulling from him... and we don't
agree about everything, hence two branches :-)
What works:
* You can start a new thread with run-in-thread. The exact use differs
between the branches. Threads are cooperative, you can yield with an
explicit call to yield.
* The GC handles multiple threads.
* Many, but not all, of the 9 kinds of variables that Emacs implements
work properly with threads. In particular, working ones are
defvaralias, objfwd, and ordinary. These all should do the right
thing in all cases. Buffer-local and buffer objfwd mostly work,
because we do buffer locking. (However, I think make-local-variable
doesn't work properly with let-binding and multiple threads.) Not
working on intfwd, boolfwd, keyboard-local and frame-local.
* Buffer locking.
There are also a number of problem areas.
* Buffer locking is actually quite a pain.
If you have a background thread locking some buffer, you can't switch
to that buffer. Emacs will just pause. I don't think we allow
interrupting a lock acquisition (clearly a bug).
* Giuseppe implemented minibuffer exclusion. But really the keyboard
should only be available to one thread at a time.
* I think there are some V* variables that probably should always be
per-thread. My notes say Vsignaling_function, but I know I've run
into other ones. (What is noteworthy about this is that it implies
that a big audit of the C code is necessary..)
* I/O is also an unsolved problem. Right now it is easy to make emacs
appear to hang by running a process filter in one thread while a
different thread owns the buffer lock. (Probably, if a thread owns a
buffer lock, then only that thread should be able to run associated
process filters.)
I think Giuseppe probably knows of more problems.
Also, there are tons more problems if you want preemptive
multithreading... redisplay probably needs work, regex is not reentrant,
the GC needs a bit more work, you need lisp-level locks, etc.
Sometimes I wonder whether it would be better to just fork a second
emacs and communicate with it using pipes + princ + read. This uses
more memory but the implementation would be a lot simpler and (I think)
it would suffice for the one case that seems most important: Gnus.
Anyway, give it a try. We welcome your advice and input.
Finally, we've been doing all our discussions privately on irc. I'd
prefer we discuss everything in the open. Is it ok to use emacs-devel
for this? If not we'll set up a new mailing list somewhere.
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-18 22:59 ` Tom Tromey
@ 2009-09-19 0:09 ` Stephen J. Turnbull
2009-09-19 0:32 ` Chong Yidong
2009-09-21 21:19 ` Stefan Monnier
2 siblings, 0 replies; 54+ messages in thread
From: Stephen J. Turnbull @ 2009-09-19 0:09 UTC (permalink / raw
To: Tom Tromey; +Cc: Stefan Monnier, Emacs development discussions
Tom Tromey writes:
> Sometimes I wonder whether it would be better to just fork a second
> emacs and communicate with it using pipes + princ + read.
Everybody at XEmacs who has thought about the problem has come to that
conclusion, although the specifics of how the communication would take
place vary. (Except for those people who think that changing the Lisp
engine to something with sane semantics for a threading environment
should come first; they usually have a multithreaded implementation of
Scheme in mind anyway.)
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-18 22:59 ` Tom Tromey
2009-09-19 0:09 ` Stephen J. Turnbull
@ 2009-09-19 0:32 ` Chong Yidong
2009-09-21 21:19 ` Stefan Monnier
2 siblings, 0 replies; 54+ messages in thread
From: Chong Yidong @ 2009-09-19 0:32 UTC (permalink / raw
To: Tom Tromey; +Cc: Stefan Monnier, Emacs development discussions
Tom Tromey <tromey@redhat.com> writes:
> Finally, we've been doing all our discussions privately on irc. I'd
> prefer we discuss everything in the open. Is it ok to use emacs-devel
> for this? If not we'll set up a new mailing list somewhere.
Yes, discussing this on emacs-devel is fine.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-18 22:59 ` Tom Tromey
2009-09-19 0:09 ` Stephen J. Turnbull
2009-09-19 0:32 ` Chong Yidong
@ 2009-09-21 21:19 ` Stefan Monnier
2009-09-21 21:50 ` Tom Tromey
2 siblings, 1 reply; 54+ messages in thread
From: Stefan Monnier @ 2009-09-21 21:19 UTC (permalink / raw
To: Tom Tromey; +Cc: Emacs development discussions
> * Many, but not all, of the 9 kinds of variables that Emacs implements
> work properly with threads. In particular, working ones are
> defvaralias, objfwd, and ordinary. These all should do the right
> thing in all cases. Buffer-local and buffer objfwd mostly work,
> because we do buffer locking. (However, I think make-local-variable
> doesn't work properly with let-binding and multiple threads.) Not
> working on intfwd, boolfwd, keyboard-local and frame-local.
How do you handle objfwd objects? Do you swap them in&out during
context switches?
> * Buffer locking is actually quite a pain.
> If you have a background thread locking some buffer, you can't switch
> to that buffer. Emacs will just pause. I don't think we allow
> interrupting a lock acquisition (clearly a bug).
Yes, we may have to relax the locking somehow. E.g. we should clearly
allow to display a buffer in any window, if the buffer is "locked" by
a thread. I.e. things like switch-to-buffer should not take the
buffer's lock.
Of course, if the buffer is locked by a thread, we also need to be
careful what we do if the user then types a command in it (should we
wait for the thread to yield or should we signal an erreur?)
> * Giuseppe implemented minibuffer exclusion. But really the keyboard
> should only be available to one thread at a time.
You mean "each keyboard", right?
> Also, there are tons more problems if you want preemptive
> multithreading... redisplay probably needs work, regex is not
> reentrant, the GC needs a bit more work, you need lisp-level
> locks, etc.
Yes, lots of fun stuff, indeed.
> Sometimes I wonder whether it would be better to just fork a second
> emacs and communicate with it using pipes + princ + read. This uses
> more memory but the implementation would be a lot simpler and (I think)
> it would suffice for the one case that seems most important: Gnus.
But you can already do that right now, so it's a different "solution".
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-21 21:19 ` Stefan Monnier
@ 2009-09-21 21:50 ` Tom Tromey
2009-09-22 14:24 ` Stefan Monnier
0 siblings, 1 reply; 54+ messages in thread
From: Tom Tromey @ 2009-09-21 21:50 UTC (permalink / raw
To: Stefan Monnier; +Cc: Emacs development discussions
>>>>> "Stefan" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
Tom> * Many, but not all, of the 9 kinds of variables that Emacs implements
Tom> work properly with threads. In particular, working ones are
Tom> defvaralias, objfwd, and ordinary. These all should do the right
Tom> thing in all cases. Buffer-local and buffer objfwd mostly work,
Tom> because we do buffer locking. (However, I think make-local-variable
Tom> doesn't work properly with let-binding and multiple threads.) Not
Tom> working on intfwd, boolfwd, keyboard-local and frame-local.
Stefan> How do you handle objfwd objects? Do you swap them in&out during
Stefan> context switches?
We have a script to edit the Emacs source. In the end all DEFUN_LISP
variables are defined like:
#define Vafter_change_functions *find_variable_location (&impl_Vafter_change_functions)
Then, find_variable_location understands Lisp_ThreadLocal binding.
A Lisp_ThreadLocal is put into an impl_V* by specbind.
This works great as long as the C code follows lispy rules. However, I
don't think that is always the case -- there are places doing direct
assignment to some of these variables where, really, a per-thread
assignment is meant.
We don't swap anything in and out during a context switch. I've tried
to make all the decisions with an eye on the long term: preemptive
threading. Ultimately I'd like to get rid of the global lock and have
all threads run free.
Tom> * Buffer locking is actually quite a pain.
Tom> If you have a background thread locking some buffer, you can't switch
Tom> to that buffer. Emacs will just pause. I don't think we allow
Tom> interrupting a lock acquisition (clearly a bug).
Stefan> Yes, we may have to relax the locking somehow. E.g. we should clearly
Stefan> allow to display a buffer in any window, if the buffer is "locked" by
Stefan> a thread. I.e. things like switch-to-buffer should not take the
Stefan> buffer's lock.
Stefan> Of course, if the buffer is locked by a thread, we also need to be
Stefan> careful what we do if the user then types a command in it (should we
Stefan> wait for the thread to yield or should we signal an erreur?)
Giuseppe has some code to let a thread acquire a buffer without locking.
But, this is done in an unsafe way; I think it is just a hack to let him
experiment with Gnus a bit.
Right now when a thread yields it does not release its buffer lock.
The only way to do that is to set-buffer to another buffer.
One way forward would be to work out the semantics properly, and
eliminate buffer locking. Some of the semantics seems tricky. And, the
work itself is hard, because it means adding tons of accessor macros and
then updating a lot of source code to use them. (I did this, once, for
a lot of Emacs, thinking to implement a software write barrier. It is
really quite painful.)
Tom> * Giuseppe implemented minibuffer exclusion. But really the keyboard
Tom> should only be available to one thread at a time.
Stefan> You mean "each keyboard", right?
Yes.
Locking a keyboard will probably have strange effects. There are some
things here that I have not thought through; like, what is the right
behavior of debug-on-error? Under what conditions would the main thread
ever release the keyboard?
Tom> Sometimes I wonder whether it would be better to just fork a second
Tom> emacs and communicate with it using pipes + princ + read. This uses
Tom> more memory but the implementation would be a lot simpler and (I think)
Tom> it would suffice for the one case that seems most important: Gnus.
Stefan> But you can already do that right now, so it's a different "solution".
Yeah. I just wonder why nobody has done it and whether it would not be
a better approach.
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-21 21:50 ` Tom Tromey
@ 2009-09-22 14:24 ` Stefan Monnier
2009-09-22 23:59 ` Ken Raeburn
` (3 more replies)
0 siblings, 4 replies; 54+ messages in thread
From: Stefan Monnier @ 2009-09-22 14:24 UTC (permalink / raw
To: Tom Tromey; +Cc: Emacs development discussions
Stefan> How do you handle objfwd objects? Do you swap them in&out during
Stefan> context switches?
> We have a script to edit the Emacs source. In the end all DEFUN_LISP
> variables are defined like:
> #define Vafter_change_functions *find_variable_location (&impl_Vafter_change_functions)
Oh, right, I remember now, sorry.
> This works great as long as the C code follows lispy rules. However, I
> don't think that is always the case -- there are places doing direct
> assignment to some of these variables where, really, a per-thread
> assignment is meant.
I'm not sure what you mean. Are you still talking about objfwd
variables?
> We don't swap anything in and out during a context switch. I've tried
> to make all the decisions with an eye on the long term: preemptive
> threading. Ultimately I'd like to get rid of the global lock and have
> all threads run free.
That was the motivation for my question, indeed.
> Right now when a thread yields it does not release its buffer lock.
I think it should, unless some of the buffer-local variables are
let-bound by the thread.
> One way forward would be to work out the semantics properly, and
> eliminate buffer locking.
Yes, although the result of this may end up requiring some form of
buffer-locking, still.
> Some of the semantics seems tricky.
Oh, yes.
Stefan> You mean "each keyboard", right?
> Yes.
> Locking a keyboard will probably have strange effects. There are some
> things here that I have not thought through; like, what is the right
> behavior of debug-on-error? Under what conditions would the main thread
> ever release the keyboard?
Maybe another way to look at all these problems is to take an "agent"
point of view: rather than threads moving around, we could consider each
keyboard and each buffer as an active object (i.e. with its own thread),
which communicate among each other. I.e. a buffer-thread never leaves
its buffer, instead it does an RPC to another buffer-thread, or to
a keyboard-thread, ...
> Yeah. I just wonder why nobody has done it and whether it would not be
> a better approach.
IIUC people do it all the time, tho not with another Emacs process: with
an hexl process, a shell process, an openssl process, ...
Emacs currently is not very good at using stdin/stdout so running an
inferior Emacs process is poorly supported.
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-22 14:24 ` Stefan Monnier
@ 2009-09-22 23:59 ` Ken Raeburn
2009-09-23 3:11 ` Stefan Monnier
2009-09-23 15:53 ` Chong Yidong
2009-09-23 3:16 ` Tom Tromey
` (2 subsequent siblings)
3 siblings, 2 replies; 54+ messages in thread
From: Ken Raeburn @ 2009-09-22 23:59 UTC (permalink / raw
To: Stefan Monnier; +Cc: Tom Tromey, Emacs development discussions
On Sep 22, 2009, at 10:24, Stefan Monnier wrote:
>> Yeah. I just wonder why nobody has done it and whether it would
>> not be
>> a better approach.
>
> IIUC people do it all the time, tho not with another Emacs process:
> with
> an hexl process, a shell process, an openssl process, ...
> Emacs currently is not very good at using stdin/stdout so running an
> inferior Emacs process is poorly supported.
I've sometimes thought a command-line option to provide a REPL (read-
eval-print loop) interface would be handy. Either directly in Emacs,
or via the client/server interface... much like interactive emacs lisp
mode, but purely on the command line.
Ken
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-22 23:59 ` Ken Raeburn
@ 2009-09-23 3:11 ` Stefan Monnier
2009-09-23 15:53 ` Chong Yidong
1 sibling, 0 replies; 54+ messages in thread
From: Stefan Monnier @ 2009-09-23 3:11 UTC (permalink / raw
To: Ken Raeburn; +Cc: Tom Tromey, Emacs development discussions
> I've sometimes thought a command-line option to provide a REPL (read-
> eval-print loop) interface would be handy. Either directly in Emacs, or via
> the client/server interface... much like interactive emacs lisp mode, but
> purely on the command line.
Actually --batch isn't far from that.
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-22 14:24 ` Stefan Monnier
2009-09-22 23:59 ` Ken Raeburn
@ 2009-09-23 3:16 ` Tom Tromey
2009-09-24 17:25 ` Stefan Monnier
2009-09-23 18:43 ` Giuseppe Scrivano
2009-09-28 7:44 ` Lynbech Christian
3 siblings, 1 reply; 54+ messages in thread
From: Tom Tromey @ 2009-09-23 3:16 UTC (permalink / raw
To: Stefan Monnier; +Cc: Emacs development discussions
Tom> This works great as long as the C code follows lispy rules. However, I
Tom> don't think that is always the case -- there are places doing direct
Tom> assignment to some of these variables where, really, a per-thread
Tom> assignment is meant.
Stefan> I'm not sure what you mean. Are you still talking about objfwd
Stefan> variables?
Yeah. What I meant here is that specbind has a little extra code in it
to install a Lisp_ThreadLocal object when needed. So, if the C code
uses specbind to do let-like binding, everything will work fine. If the
C code does not do this, then things will break.
For example, I am not certain that Vquit_flag is handled in a "safe"
way. It is directly assigned to in a number of places, but those
assignments should probably be thread-local. (This isn't the greatest
example, maybe, because Vquit_flag is also bound up in keyboard locking
and maybe other issues.)
Tom> Right now when a thread yields it does not release its buffer lock.
Stefan> I think it should, unless some of the buffer-local variables are
Stefan> let-bound by the thread.
I've been thinking about this a bit. It is trickier than it seems,
because a thread can actually have let-bindings for buffer-locals from
multiple buffers at the same time. There's an example of this in the
elisp manual. (This is also an area the threading code does not handle
well yet.)
I think this means it does not suffice to keep a simple per-thread count
of buffer-local bindings.
While thinking about this I realized that (I think) the current code
does the wrong thing if one thread let-binds a variable and then another
thread calls make-variable-buffer-local on it.
Stefan> Maybe another way to look at all these problems is to take an
Stefan> "agent" point of view: rather than threads moving around, we
Stefan> could consider each keyboard and each buffer as an active object
Stefan> (i.e. with its own thread), which communicate among each other.
Stefan> I.e. a buffer-thread never leaves its buffer, instead it does an
Stefan> RPC to another buffer-thread, or to a keyboard-thread, ...
I had considered the agent model -- that, plus a discussion on irc, is
actually what lead me to ask about running a second Emacs as a
subprocess.
It is hard for me to see how this could be done in a compatible way.
Right now elisp operates with few constraints; an elisp program can call
select-frame, which seems to imply that per-frame or per-keyboard
threads can't be done.
Tom> Yeah. I just wonder why nobody has done it and whether it would not be
Tom> a better approach.
Stefan> IIUC people do it all the time, tho not with another Emacs
Stefan> process: with an hexl process, a shell process, an openssl
Stefan> process, ... Emacs currently is not very good at using
Stefan> stdin/stdout so running an inferior Emacs process is poorly
Stefan> supported.
If I were doing it "for real" I might consider coming up with a
higher-bandwidth way to transfer an entire buffer between processes.
But, a princ/read approach could be done without modifying Emacs, by
having the parent Emacs make a server socket, and then passing the port
number to the subprocess as an argument. This avoids the stdin/stdout
difficulties.
One thing that would help me is having a concrete idea of what subset of
features would make this work be actually useful. I mostly implemented
it because it was cool, and because Giuseppe's initial patch convinced
me that it was possible (before that I'd written it off as totally
impractical). Now that a proof of concept works it would be nice to
have a concrete goal.
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-22 23:59 ` Ken Raeburn
2009-09-23 3:11 ` Stefan Monnier
@ 2009-09-23 15:53 ` Chong Yidong
1 sibling, 0 replies; 54+ messages in thread
From: Chong Yidong @ 2009-09-23 15:53 UTC (permalink / raw
To: Ken Raeburn; +Cc: Tom Tromey, Stefan Monnier, Emacs development discussions
Ken Raeburn <raeburn@raeburn.org> writes:
> I've sometimes thought a command-line option to provide a REPL (read-
> eval-print loop) interface would be handy. Either directly in Emacs,
> or via the client/server interface... much like interactive emacs lisp
> mode, but purely on the command line.
emacs --eval "(while t (print (eval (read))))" --batch
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-22 14:24 ` Stefan Monnier
2009-09-22 23:59 ` Ken Raeburn
2009-09-23 3:16 ` Tom Tromey
@ 2009-09-23 18:43 ` Giuseppe Scrivano
2009-09-24 17:29 ` Stefan Monnier
2009-09-28 7:44 ` Lynbech Christian
3 siblings, 1 reply; 54+ messages in thread
From: Giuseppe Scrivano @ 2009-09-23 18:43 UTC (permalink / raw
To: Stefan Monnier; +Cc: Tom Tromey, Emacs development discussions
Hello,
Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>> Right now when a thread yields it does not release its buffer lock.
>
> I think it should, unless some of the buffer-local variables are
> let-bound by the thread.
I think actually this is one of the most important decision to take, it
will have a direct influence on buffer-local variables semantic. In my
opinion, it is better to have few constraints and allow switch as often
as possible.
As Tom said, I was experimenting with gnus and it made me quite
optimistic; in fact these commands are completed successfully:
(run-in-thread '(gnus) t) ;; with t the thread doesn't attempt to get
;; a lock on the buffer.
(while (yield)) ;; yield returns nil when there are not other
;; threads -- simulate join.
The nolock argument in run-in-thread is just a hack to get things
working and see quickly some results. Acquiring always the lock when
accessing the buffer is too restrictive IMHO as too often the buffer is
already locked.
Following your advice to don't allow a switch only when buffer-local
variables are let-bound by the thread, how many times it happens than a
thread let-bound these variables? Let's say on 100 times, how many
times approximately do you think a switch will not be allowed and the
calling thread blocked?
Cheers,
Giuseppe
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-23 3:16 ` Tom Tromey
@ 2009-09-24 17:25 ` Stefan Monnier
2009-09-24 17:57 ` Tom Tromey
2009-09-27 20:59 ` Tom Tromey
0 siblings, 2 replies; 54+ messages in thread
From: Stefan Monnier @ 2009-09-24 17:25 UTC (permalink / raw
To: Tom Tromey; +Cc: Emacs development discussions
> Yeah. What I meant here is that specbind has a little extra code in it
> to install a Lisp_ThreadLocal object when needed. So, if the C code
> uses specbind to do let-like binding, everything will work fine. If the
> C code does not do this, then things will break.
I see, yes of course it's a problem. And it's worse than that: the same
holds for *all* global variables, whether they are exported to Lisp
or not.
Stefan> I think it should, unless some of the buffer-local variables are
Stefan> let-bound by the thread.
> I've been thinking about this a bit. It is trickier than it seems,
> because a thread can actually have let-bindings for buffer-locals from
> multiple buffers at the same time. There's an example of this in the
> elisp manual. (This is also an area the threading code does not handle
> well yet.)
> I think this means it does not suffice to keep a simple per-thread count
> of buffer-local bindings.
Indeed. The lock should probably be on the buffer-local binding itself
(i.e. the combination of the buffer and the variable).
> While thinking about this I realized that (I think) the current code
> does the wrong thing if one thread let-binds a variable and then another
> thread calls make-variable-buffer-local on it.
Calling make-variable-buffer-local or make-local-variable when the
variable is let-bound is a serious problem, yes. IIRC we already have
bugs in such cases in the current code.
AFAICT, the only good answer is to disallow such things (e.g. signal an
error in make-variable-buffer-local and make-local-variable when the var
is let-bound).
I already took a tiny step in that direction when I added a warning to
defvar when called while the variable is let-bound.
> It is hard for me to see how this could be done in a compatible way.
Agreed.
> Right now elisp operates with few constraints; an elisp program can call
> select-frame, which seems to imply that per-frame or per-keyboard
> threads can't be done.
The way I think of it is that select-frame would send a message to the
relevant frame, along with a function to run there, and the destination
to which to send the result when the function is done.
But how to link it with the dynamic and foo-local scoping of
let-bindings is still far from clear.
> One thing that would help me is having a concrete idea of what subset of
> features would make this work be actually useful. I mostly implemented
> it because it was cool, and because Giuseppe's initial patch convinced
> me that it was possible (before that I'd written it off as totally
> impractical). Now that a proof of concept works it would be nice to
> have a concrete goal.
Some goals could be:
- run Gnus in its own thread. This should be easy-ish since Gnus
shouldn't interact with pretty much anything else.
- split Gnus into several threads (e.g. in my case, the most important
one would be a thread to do the "g", i.e. refresh the *Group* buffer,
which with IMAP boxes can takes ages).
This is a lot more difficult since this thread would need to interact
with the main Gnus thread.
- move the jit-lock-context to a thread. Might be non-trivial, but at
least the code is small.
- move some VC operations to threads (e.g. vc-checkin).
- run some of the buffer-parsing code in threads (e.g. the one in
CEDET's semantic, or the one in js2-mode).
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-23 18:43 ` Giuseppe Scrivano
@ 2009-09-24 17:29 ` Stefan Monnier
2009-09-24 18:53 ` Giuseppe Scrivano
0 siblings, 1 reply; 54+ messages in thread
From: Stefan Monnier @ 2009-09-24 17:29 UTC (permalink / raw
To: Giuseppe Scrivano; +Cc: Tom Tromey, Emacs development discussions
> As Tom said, I was experimenting with gnus and it made me quite
> optimistic; in fact these commands are completed successfully:
> (run-in-thread '(gnus) t) ;; with t the thread doesn't attempt to get
> ;; a lock on the buffer.
> (while (yield)) ;; yield returns nil when there are not other
> ;; threads -- simulate join.
That's wonderful. Does that mean that Gnus really runs in its own
thread while you can do normal editing in other buffers?
> Following your advice to don't allow a switch only when buffer-local
> variables are let-bound by the thread, how many times it happens than a
> thread let-bound these variables? Let's say on 100 times, how many
> times approximately do you think a switch will not be allowed and the
> calling thread blocked?
I have no idea. I don't think the number of such situations (counted
dynamically) is very important. What matters is the number of places in
the source code where this is problematic.
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-24 17:25 ` Stefan Monnier
@ 2009-09-24 17:57 ` Tom Tromey
2009-09-27 20:59 ` Tom Tromey
1 sibling, 0 replies; 54+ messages in thread
From: Tom Tromey @ 2009-09-24 17:57 UTC (permalink / raw
To: Stefan Monnier; +Cc: Emacs development discussions
>>>>> "Stefan" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
Tom> Yeah. What I meant here is that specbind has a little extra code in it
Tom> to install a Lisp_ThreadLocal object when needed. So, if the C code
Tom> uses specbind to do let-like binding, everything will work fine. If the
Tom> C code does not do this, then things will break.
Stefan> I see, yes of course it's a problem. And it's worse than that: the same
Stefan> holds for *all* global variables, whether they are exported to Lisp
Stefan> or not.
Yes. We move a bunch of globals into a "struct thread_state" -- specpdl
stuff, gcprolist, catchlist, etc. I haven't yet done an audit of
globals to see whether we caught them all. These are also redefined
using the preprocessor, e.g.:
/* Recording what needs to be marked for gc. */
struct gcpro *m_gcprolist;
#define gcprolist (current_thread->m_gcprolist)
Stefan> Some goals could be:
[...]
Thanks. That is very helpful.
I will think about your other points and reply later.
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-24 17:29 ` Stefan Monnier
@ 2009-09-24 18:53 ` Giuseppe Scrivano
2009-09-24 20:04 ` Tom Tromey
0 siblings, 1 reply; 54+ messages in thread
From: Giuseppe Scrivano @ 2009-09-24 18:53 UTC (permalink / raw
To: Stefan Monnier; +Cc: Tom Tromey, Emacs development discussions
Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>> (run-in-thread '(gnus) t) ;; with t the thread doesn't attempt to get
>> ;; a lock on the buffer.
>> (while (yield)) ;; yield returns nil when there are not other
>> ;; threads -- simulate join.
>
> That's wonderful. Does that mean that Gnus really runs in its own
> thread while you can do normal editing in other buffers?
Yes, it does (except some errors). I am concentrating all my efforts on
gnus, I think that if threads work well for gnus they will work quite
well with any other package.
At the moment it takes a lot to load folder, because there are not
enough yield on the other thread and probably gnus switch buffers too
often (causing a yield). Still there is much to do but some results
are already visible.
Cheers,
Giuseppe
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-24 18:53 ` Giuseppe Scrivano
@ 2009-09-24 20:04 ` Tom Tromey
2009-09-24 21:59 ` Stefan Monnier
` (4 more replies)
0 siblings, 5 replies; 54+ messages in thread
From: Tom Tromey @ 2009-09-24 20:04 UTC (permalink / raw
To: Giuseppe Scrivano; +Cc: Stefan Monnier, Emacs development discussions
>>>>> "Giuseppe" == Giuseppe Scrivano <gscrivano@gnu.org> writes:
>> That's wonderful. Does that mean that Gnus really runs in its own
>> thread while you can do normal editing in other buffers?
Giuseppe> Yes, it does (except some errors). I am concentrating all my
Giuseppe> efforts on gnus, I think that if threads work well for gnus
Giuseppe> they will work quite well with any other package.
This is awesome.
Giuseppe> At the moment it takes a lot to load folder, because there are
Giuseppe> not enough yield on the other thread and probably gnus switch
Giuseppe> buffers too often (causing a yield).
I think we need to yield when a thread is waiting for I/O. And, we
probably need a way to associate a process filter with a thread, so that
whatever input Gnus gets is processed in the right thread.
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-24 20:04 ` Tom Tromey
@ 2009-09-24 21:59 ` Stefan Monnier
2009-09-24 22:23 ` Giuseppe Scrivano
` (3 subsequent siblings)
4 siblings, 0 replies; 54+ messages in thread
From: Stefan Monnier @ 2009-09-24 21:59 UTC (permalink / raw
To: Tom Tromey; +Cc: Giuseppe Scrivano, Emacs development discussions
>>> That's wonderful. Does that mean that Gnus really runs in its own
>>> thread while you can do normal editing in other buffers?
Giuseppe> Yes, it does (except some errors). I am concentrating all my
Giuseppe> efforts on gnus, I think that if threads work well for gnus
Giuseppe> they will work quite well with any other package.
> This is awesome.
Indeed.
Giuseppe> At the moment it takes a lot to load folder, because there are
Giuseppe> not enough yield on the other thread and probably gnus switch
Giuseppe> buffers too often (causing a yield).
> I think we need to yield when a thread is waiting for I/O. And, we
> probably need a way to associate a process filter with a thread, so that
> whatever input Gnus gets is processed in the right thread.
Since process filters are currently processed "asynchronously", it would
seem like a natural idea would be to create a thread for each process,
and let that thread run all the process's filters and sentinels.
While I'm here, there's of course another relevant test case: open Emacs
on several ttys, and run some long-running commands like indent-region
in each (in different buffers at first, and then in the same one, but
on different parts of the text, ...).
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-24 20:04 ` Tom Tromey
2009-09-24 21:59 ` Stefan Monnier
@ 2009-09-24 22:23 ` Giuseppe Scrivano
2009-09-24 22:47 ` Ken Raeburn
` (2 subsequent siblings)
4 siblings, 0 replies; 54+ messages in thread
From: Giuseppe Scrivano @ 2009-09-24 22:23 UTC (permalink / raw
To: Tom Tromey; +Cc: Stefan Monnier, Emacs development discussions
Tom Tromey <tromey@redhat.com> writes:
> I think we need to yield when a thread is waiting for I/O. And, we
> probably need a way to associate a process filter with a thread, so that
> whatever input Gnus gets is processed in the right thread.
I did some changes today. Now a thread exits from the global lock
before calls `select' and it enters again immediately after `select'
returns, leaving other threads free to do their job.
Blocking operations affect only the caller thread. As direct effect,
now this is enough:
(run-in-thread '(gnus))
and gnus will be loaded in the background, without explicitly call
`yield'.
Cheers,
Giuseppe
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-24 20:04 ` Tom Tromey
2009-09-24 21:59 ` Stefan Monnier
2009-09-24 22:23 ` Giuseppe Scrivano
@ 2009-09-24 22:47 ` Ken Raeburn
2009-09-28 14:52 ` Ted Zlatanov
2009-10-05 6:02 ` joakim
4 siblings, 0 replies; 54+ messages in thread
From: Ken Raeburn @ 2009-09-24 22:47 UTC (permalink / raw
To: Tom Tromey
Cc: Emacs development discussions, Giuseppe Scrivano, Stefan Monnier
On Sep 24, 2009, at 16:04, Tom Tromey wrote:
>>>>>> "Giuseppe" == Giuseppe Scrivano <gscrivano@gnu.org> writes:
>
>>> That's wonderful. Does that mean that Gnus really runs in its own
>>> thread while you can do normal editing in other buffers?
>
> Giuseppe> Yes, it does (except some errors). I am concentrating all
> my
> Giuseppe> efforts on gnus, I think that if threads work well for gnus
> Giuseppe> they will work quite well with any other package.
>
> This is awesome.
Yes!
It was this very problem that initially got me thinking about threads
(and from that, it was just a hop, skip and a jump to Guile) years ago.
> Giuseppe> At the moment it takes a lot to load folder, because there
> are
> Giuseppe> not enough yield on the other thread and probably gnus
> switch
> Giuseppe> buffers too often (causing a yield).
>
> I think we need to yield when a thread is waiting for I/O.
Yes, I think that needs to happen pretty much automatically. Perhaps
it would be sufficient to make the Lisp functions that can cause
process filters to be run also yield to other threads.
> And, we
> probably need a way to associate a process filter with a thread, so
> that
> whatever input Gnus gets is processed in the right thread.
Yes, and this is an area where Gnus could be changed to take advantage
of it once the facility is a bit more stabilized. The output from one
NNTP server can be processed into a list of newsgroups in one thread,
while another thread converts IMAP output into a list of mailboxes, a
third thread talks to yet another IMAP server, etc.
It might be interesting to look at Apple's Grand Central model, and
see if something like that is practical to use in elisp with dynamic
bindings...
Ken
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-24 17:25 ` Stefan Monnier
2009-09-24 17:57 ` Tom Tromey
@ 2009-09-27 20:59 ` Tom Tromey
2009-09-27 23:05 ` Stefan Monnier
1 sibling, 1 reply; 54+ messages in thread
From: Tom Tromey @ 2009-09-27 20:59 UTC (permalink / raw
To: Stefan Monnier; +Cc: Emacs development discussions
>>>>> "Stefan" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
[ releasing the buffer lock when yielding ]
Tom> I think this means it does not suffice to keep a simple per-thread count
Tom> of buffer-local bindings.
Stefan> Indeed. The lock should probably be on the buffer-local binding itself
Stefan> (i.e. the combination of the buffer and the variable).
I looked into this a bit; I think it can lead to deadlock too easily:
Thread A acquires V1 and requests V2
Thread B acquires V2 and requests V1
Instead, I think we could remove the buffer lock. Because we currently
use cooperative threading, most buffer operations are atomic with
respect to thread switches. So, it seems that if we fix the bindings
problem, we can probably just remove the buffer lock.
What do you think of that? Are there problems I've missed?
I wrote some elisp to modify the Emacs sources to use buffer accessor
macros everywhere (it handles like 90% of the references, a few tweaks
by hand are also needed). The only thing remaining is to fix up the
swap_in_symval_forwarding logic, which we have to do anyway.
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-27 20:59 ` Tom Tromey
@ 2009-09-27 23:05 ` Stefan Monnier
2009-09-28 4:27 ` Tom Tromey
0 siblings, 1 reply; 54+ messages in thread
From: Stefan Monnier @ 2009-09-27 23:05 UTC (permalink / raw
To: Tom Tromey; +Cc: Emacs development discussions
Stefan> Indeed. The lock should probably be on the buffer-local binding itself
Stefan> (i.e. the combination of the buffer and the variable).
> I looked into this a bit; I think it can lead to deadlock too easily:
> Thread A acquires V1 and requests V2
> Thread B acquires V2 and requests V1
> Instead, I think we could remove the buffer lock. Because we currently
> use cooperative threading, most buffer operations are atomic with
> respect to thread switches. So, it seems that if we fix the bindings
> problem, we can probably just remove the buffer lock.
Removing locks sounds like a wonderful solution, but it's dangerous.
I think you understand the danger and refer to it as "the bindings
problem". But since I'm not sure I'll state what I think is the
problem:
thread T1 goes into buffer B and does a dynamic let-binding of
buffer-local variable X with value V1, then thread T2 does the same
thing with value V2.
What should that mean?
What should T1 see when it looks at X, V1 or V2?
What should T2 see when it looks at X, V1 or V2?
What did T2 see when it looked at X before let-binding it, V1 or V0?
What should T1 see when it looks at X after T2 ends its let-binding, V1 or V0?
What should T2 see when it looks at X after T1 ends its let-binding, V2 or V0?
You have to consider those cases and then look at existing code and see
what that code would expect. A good variable to start with is probably
default-directory (one of the most common buffer-local-and-let-bound
variables I know of), but others need to be considered as well (I'm not
sure they'll all agree on the behavior they expect).
Stefan
PS: Of course, similar questions arise with let-binding frame-local and
terminal-local variables, but I think these are much less common, so
it's probably OK to be less careful with them and just fix up the places
that break using explicit locks or things like that.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-27 23:05 ` Stefan Monnier
@ 2009-09-28 4:27 ` Tom Tromey
2009-09-29 0:27 ` Stefan Monnier
0 siblings, 1 reply; 54+ messages in thread
From: Tom Tromey @ 2009-09-28 4:27 UTC (permalink / raw
To: Stefan Monnier; +Cc: Emacs development discussions
Stefan> Removing locks sounds like a wonderful solution, but it's dangerous.
Stefan> I think you understand the danger and refer to it as "the bindings
Stefan> problem". But since I'm not sure I'll state what I think is the
Stefan> problem:
Yes. I forgot to write something important in that message, which is
what I think should happen.
Stefan> thread T1 goes into buffer B and does a dynamic let-binding of
Stefan> buffer-local variable X with value V1, then thread T2 does the same
Stefan> thing with value V2.
Stefan> What should that mean?
In my view, a let-binding should always be thread-local, and I think
buffer-localness shouldn't affect this. This is enough of a rule to
answer your questions -- but it is just one choice, we could make
others, I suppose.
The primary reason I like this particular choice is that it seems least
susceptible to races when running ordinary lisp code in parallel. Also,
it is simple to understand and consistent with the behavior of
non-buffer-local let-bindings.
Stefan> You have to consider those cases and then look at existing code and see
Stefan> what that code would expect. A good variable to start with is probably
Stefan> default-directory (one of the most common buffer-local-and-let-bound
Stefan> variables I know of), but others need to be considered as well (I'm not
Stefan> sure they'll all agree on the behavior they expect).
I don't think existing code could have an expectation, simply because
existing code can't run multi-threaded. The closest thing are process
filters, but I am not so sure that they are a good analogy in this case.
There are some oddities implied by making buffer-local let-bindings also
be thread-specific. For example, some buffer-locals affect redisplay,
so what the user sees will depend on the thread in which redisplay is
run.
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-22 14:24 ` Stefan Monnier
` (2 preceding siblings ...)
2009-09-23 18:43 ` Giuseppe Scrivano
@ 2009-09-28 7:44 ` Lynbech Christian
3 siblings, 0 replies; 54+ messages in thread
From: Lynbech Christian @ 2009-09-28 7:44 UTC (permalink / raw
To: Stefan Monnier; +Cc: Tom Tromey, Emacs development discussions
>>>>> "Stefan" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
Stefan> Maybe another way to look at all these problems is to take an "agent"
Stefan> point of view: rather than threads moving around, we could consider each
Stefan> keyboard and each buffer as an active object (i.e. with its own thread),
Stefan> which communicate among each other. I.e. a buffer-thread never leaves
Stefan> its buffer, instead it does an RPC to another buffer-thread, or to
Stefan> a keyboard-thread, ...
This seems like a really clever idea, but I didn't see any response to
it. Has anybody thought more about this?
Everything in Emacs is centered around buffers anyway and I guess that a
lot of the RPC (at least wrt. keyboards) would look a lot like the
events comming in from X anyway.
The biggest problem I see here is that you can have a large number of
buffers. I do not know what the practical limit on the number of threads
are on modern systems, but if that is a concern one could instead tie
the threads to processes/filters/sentinels (which aren't really
interactive anyway) and windows/frames which all user interaction goes
through.
------------------------+-----------------------------------------------------
Christian Lynbech | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
- petonic@hal.com (Michael A. Petonic)
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-24 20:04 ` Tom Tromey
` (2 preceding siblings ...)
2009-09-24 22:47 ` Ken Raeburn
@ 2009-09-28 14:52 ` Ted Zlatanov
2009-10-05 6:02 ` joakim
4 siblings, 0 replies; 54+ messages in thread
From: Ted Zlatanov @ 2009-09-28 14:52 UTC (permalink / raw
To: emacs-devel
On Thu, 24 Sep 2009 14:04:22 -0600 Tom Tromey <tromey@redhat.com> wrote:
Tom> I think we need to yield when a thread is waiting for I/O. And, we
Tom> probably need a way to associate a process filter with a thread, so that
Tom> whatever input Gnus gets is processed in the right thread.
On Thu, 24 Sep 2009 18:47:16 -0400 Ken Raeburn <raeburn@raeburn.org> wrote:
KR> Yes, and this is an area where Gnus could be changed to take advantage
KR> of it once the facility is a bit more stabilized. The output from one
KR> NNTP server can be processed into a list of newsgroups in one thread,
KR> while another thread converts IMAP output into a list of mailboxes, a
KR> third thread talks to yet another IMAP server, etc.
The complications from gnus-agent, the gnus-registry, and other Gnus
elements that assume single-threading can be quite nasty. It would
probably be productive to involve the Gnus developers (Ding Mailing List
<ding@gnus.org>).
Ted
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-28 4:27 ` Tom Tromey
@ 2009-09-29 0:27 ` Stefan Monnier
2009-09-29 2:26 ` Ken Raeburn
2009-09-29 2:30 ` Tom Tromey
0 siblings, 2 replies; 54+ messages in thread
From: Stefan Monnier @ 2009-09-29 0:27 UTC (permalink / raw
To: Tom Tromey; +Cc: Emacs development discussions
> In my view, a let-binding should always be thread-local, and I think
> buffer-localness shouldn't affect this. This is enough of a rule to
> answer your questions -- but it is just one choice, we could make
> others, I suppose.
So what should happen in the following case:
Thread A, running in buffer B1, runs the following code:
(let ((default-directory "/foo"))
(with-current-buffer B2 default-directory))
You seem to say that it should return "/foo", but currently it returns
something else. Maybe we should try to come up with a run-time test to
catch such cases in current Lisp code. A related case is when a process
filter or a sentinel is run via accept-process-output: we'd need to be
careful to make sure the code is run in the same thread as the code that
called accept-process-output.
> There are some oddities implied by making buffer-local let-bindings
> also be thread-specific. For example, some buffer-locals affect
> redisplay, so what the user sees will depend on the thread in which
> redisplay is run.
Redisplay should be run in a completely separate thread (at least
conceptually).
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-29 0:27 ` Stefan Monnier
@ 2009-09-29 2:26 ` Ken Raeburn
2009-09-29 3:20 ` Stefan Monnier
2009-09-29 3:33 ` Tom Tromey
2009-09-29 2:30 ` Tom Tromey
1 sibling, 2 replies; 54+ messages in thread
From: Ken Raeburn @ 2009-09-29 2:26 UTC (permalink / raw
To: Stefan Monnier; +Cc: Tom Tromey, Emacs development discussions
On Sep 28, 2009, at 20:27, Stefan Monnier wrote:
>> In my view, a let-binding should always be thread-local, and I think
>> buffer-localness shouldn't affect this. This is enough of a rule to
>> answer your questions -- but it is just one choice, we could make
>> others, I suppose.
>
> So what should happen in the following case:
> Thread A, running in buffer B1, runs the following code:
>
> (let ((default-directory "/foo"))
> (with-current-buffer B2 default-directory))
>
> You seem to say that it should return "/foo", but currently it returns
> something else. Maybe we should try to come up with a run-time test
> to
> catch such cases in current Lisp code.
I suspect that any sensible approach we come up with is going to
involve a change in behavior that may affect some existing code. So
coming up with ways to detect code that changes meaning is probably
going to be a good idea.
> A related case is when a process
> filter or a sentinel is run via accept-process-output: we'd need to be
> careful to make sure the code is run in the same thread as the code
> that
> called accept-process-output.
Not necessarily; two threads could each be running helper programs in
subprocesses (or using network connections), setting up let-bound
variables, and calling accept-process-output, expecting the process
filters to have access to the let-bound variables. We might want to,
by default, bind the filters to run only in the thread that originally
created the process, so long as it exists. It could be less
efficient, but I think it might easily preserve more of the current
behavior of packages written for single-threaded Emacs, even if they
get run concurrently (but independently) in multiple threads. And we
could make it easy for code to allow other threads to process the
output too.
>> There are some oddities implied by making buffer-local let-bindings
>> also be thread-specific. For example, some buffer-locals affect
>> redisplay, so what the user sees will depend on the thread in which
>> redisplay is run.
>
> Redisplay should be run in a completely separate thread (at least
> conceptually).
I think it's less likely, but a package might also let-bind some
variables that affect the display (e.g., truncate-lines, selective-
display, show-trailing-whitespace, all of which are buffer-local, or
anything referenced via mode-line) and then call y-or-n-p or read-file-
name or some other routine that will trigger redisplay and prompt the
user for some info that may depend on what's displayed.... Be that as
it may, I think making it conceptually a separate thread is probably
the right approach. And this minor incompatibility is something we
can test for now, for the most part -- if redisplay examines a Lisp
variable, it should check for the presence of that variable in specpdl
and cause a warning to be issued later if it's found. (Uses of
":eval" in mode-line-format are probably too expensive to process.)
The other approach that came to my mind was, use the context of the
thread that's currently taking the keyboard input if there is one, to
try to approximate the "right thing" for any code already doing what I
described above. (If one thread is prompting the user for input, and
then another thread triggers redisplay but doesn't prompt for input, I
suspect seeing what's intended by the prompting thread is probably
more important.) Of course that would mean that when switching which
thread "owns" the current keyboard input, you might need to redisplay
everything because of the change of context, or at least redisplay the
affected buffers, which could be many of them, depending on the
behavior of such code as:
(let ((default-directory foo))
(with-current buffer buf1
(let ((default-directory bar))
(with-current-buffer buf2
...repeat ad nauseam...
So... yeah, I think I like the conceptually-separate-thread approach
better, even if it could break existing code. I'm not certain we
couldn't do better in terms of backwards compatibility, but it's
simple and clean.
Ken
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-29 0:27 ` Stefan Monnier
2009-09-29 2:26 ` Ken Raeburn
@ 2009-09-29 2:30 ` Tom Tromey
1 sibling, 0 replies; 54+ messages in thread
From: Tom Tromey @ 2009-09-29 2:30 UTC (permalink / raw
To: Stefan Monnier; +Cc: Emacs development discussions
Stefan> So what should happen in the following case:
Stefan> Thread A, running in buffer B1, runs the following code:
Stefan> (let ((default-directory "/foo"))
Stefan> (with-current-buffer B2 default-directory))
Stefan> You seem to say that it should return "/foo", but currently it returns
Stefan> something else.
I didn't mean to imply that. I've been too terse, I'll try to expand a
bit more.
First, I think it is clear that the case where there is just a single
thread running should work exactly as elisp does today. So, the above
will continue to work -- it has to for compatibility.
To this end, I've written a bindings test suite that is on the branch.
It is incomplete but it does test things like the above. I'm happy to
add any tests you think appropriate to catch more possible problems.
(This is also the one piece that could probably go into trunk
immediately.)
The way ordinary variables work on the branch is, in the base case, just
like current elisp: the value is stored in Lisp_Symbol::value. Then, if
a thread let-binds the variable, we create a struct Lisp_ThreadLocal.
This has a slot for the global (un-shadowed) value, and also an alist of
thread-local values, keyed by thread.
There are 8 other kinds of variable bindings in Emacs:
* defvaralias. These work exactly like ordinary bindings, they
require no change.
* objfwd. These work exactly as above, because we wrote that "semantic
patch" to add #defines which indirect through the Lisp_ThreadLocal,
when needed.
* intfwd
* boolfwd. These are not done yet. I think I will write another
semantic patch to handle these, like we discussed earlier. Exactly
what approach I'll use, I don't know... maybe make them exactly like
objfwd and put a flag bit into Lisp_Objfwd, or keep the same structs
and change the value's field type.
* buffer-local
* buffer-objfwd.
Right now each of these has a single conceptual "value" field.
(Where this is stored and how it is accessed is different in the two
cases, but that doesn't matter.) What I propose to do is allow the
same sort of thread-local indirection where this value is stored, so
that a buffer-local will have a global value and then a separate
per-thread value in each thread that has let-bound the variable.
I wrote the big part of this patch already (adding accessors and
updating all the C code), all that remains is fixing up symval
forwarding and whatnot.
* keyboard-local
* frame-local
I haven't looked into these yet, but I assume we'll end up wanting to
do the same as buffer-locals.
Stefan> A related case is when a process filter or a sentinel is run via
Stefan> accept-process-output: we'd need to be careful to make sure the
Stefan> code is run in the same thread as the code that called
Stefan> accept-process-output.
Yes, good point.
Tom> There are some oddities implied by making buffer-local let-bindings
Tom> also be thread-specific. For example, some buffer-locals affect
Tom> redisplay, so what the user sees will depend on the thread in which
Tom> redisplay is run.
Stefan> Redisplay should be run in a completely separate thread (at least
Stefan> conceptually).
Yeah, that would be nice. I don't know much about redisplay, so I don't
have much idea of how hard this would be, or what problems might arise.
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-29 2:26 ` Ken Raeburn
@ 2009-09-29 3:20 ` Stefan Monnier
2009-09-29 3:57 ` Ken Raeburn
2009-09-29 3:33 ` Tom Tromey
1 sibling, 1 reply; 54+ messages in thread
From: Stefan Monnier @ 2009-09-29 3:20 UTC (permalink / raw
To: Ken Raeburn; +Cc: Tom Tromey, Emacs development discussions
>> You seem to say that it should return "/foo", but currently it returns
>> something else. Maybe we should try to come up with a run-time test to
>> catch such cases in current Lisp code.
> I suspect that any sensible approach we come up with is going to
> involve a change in behavior that may affect some existing code. So coming
> up with ways to detect code that changes meaning is probably going to
> be a good idea.
Yes, that's what I meant.
>> A related case is when a process filter or a sentinel is run via
>> accept-process-output: we'd need to be careful to make sure the code
>> is run in the same thread as the code that called
>> accept-process-output.
> Not necessarily; two threads could each be running helper programs in
> subprocesses (or using network connections), setting up let-bound
> variables, and calling accept-process-output, expecting the process filters
> to have access to the let-bound variables.
I think you're thinking of the case where accept-process-output is
called without specifying any particular process to wait for, whereas
I was thinking of the case where accept-process-output is called with
a process argument.
>> Redisplay should be run in a completely separate thread (at least
>> conceptually).
> I think it's less likely, but a package might also let-bind some variables
> that affect the display (e.g., truncate-lines, selective-
> display, show-trailing-whitespace, all of which are buffer-local, or
> anything referenced via mode-line) and then call y-or-n-p or read-file-
> name or some other routine that will trigger redisplay and prompt the user
> for some info that may depend on what's displayed....
Yes, I know this happens with things like cursor-in-echo-area, tho this
is a global variable. Not sure how often (or even if) this happens with
buffer-local vars.
> So... yeah, I think I like the conceptually-separate-thread approach better,
> even if it could break existing code. I'm not certain we couldn't do
> better in terms of backwards compatibility, but it's simple and clean.
It's improtant to maximize backward compatibility, but it's also to try
and avoid getting locked in overly complex semantics, especially in the
long run.
Stefan
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-29 2:26 ` Ken Raeburn
2009-09-29 3:20 ` Stefan Monnier
@ 2009-09-29 3:33 ` Tom Tromey
2009-09-29 4:07 ` Ken Raeburn
1 sibling, 1 reply; 54+ messages in thread
From: Tom Tromey @ 2009-09-29 3:33 UTC (permalink / raw
To: Ken Raeburn; +Cc: Stefan Monnier, Emacs development discussions
>>>>> "Ken" == Ken Raeburn <raeburn@raeburn.org> writes:
Ken> I suspect that any sensible approach we come up with is going to
Ken> involve a change in behavior that may affect some existing code. So
Ken> coming up with ways to detect code that changes meaning is probably
Ken> going to be a good idea.
It is worth noting that we don't have to solve all the problems up
front. Since threading is a new feature, I think it is ok to say that
code running in a "secondary" thread might have to follow some special
rules, or that some things (e.g., let-binding something then redisplay)
may only work in the "main" thread. These restrictions could then maybe
be lifted later.
Ken> We might want to, by default, bind the filters to run only in the
Ken> thread that originally created the process, so long as it exists.
We can also add a new process attribute controlling this.
Tom
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-29 3:20 ` Stefan Monnier
@ 2009-09-29 3:57 ` Ken Raeburn
0 siblings, 0 replies; 54+ messages in thread
From: Ken Raeburn @ 2009-09-29 3:57 UTC (permalink / raw
To: Stefan Monnier; +Cc: Tom Tromey, Emacs development discussions
On Sep 28, 2009, at 23:20, Stefan Monnier wrote:
> I think you're thinking of the case where accept-process-output is
> called without specifying any particular process to wait for, whereas
> I was thinking of the case where accept-process-output is called with
> a process argument.
True. Well, more specifically, if the just-this-one flag is set, too,
then my concern goes away. Just specifying the process doesn't mean
that output from another process won't get, uh, processed, during a
call. Given the single-threaded nature of elisp now, I doubt most
uses are likely to set the flag. In fact a quick grep over the top-
level lisp sources shows most calls there just specify the process and
timeout.
(Oh... hm, timers. Same issues, also probably unlikely to be a
problem in most cases.)
Ken
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-29 3:33 ` Tom Tromey
@ 2009-09-29 4:07 ` Ken Raeburn
0 siblings, 0 replies; 54+ messages in thread
From: Ken Raeburn @ 2009-09-29 4:07 UTC (permalink / raw
To: Tom Tromey; +Cc: Stefan Monnier, Emacs development discussions
On Sep 28, 2009, at 23:33, Tom Tromey wrote:
> It is worth noting that we don't have to solve all the problems up
> front. Since threading is a new feature, I think it is ok to say that
> code running in a "secondary" thread might have to follow some special
> rules, or that some things (e.g., let-binding something then
> redisplay)
> may only work in the "main" thread. These restrictions could then
> maybe
> be lifted later.
True. And as long as it's not yet folded into an Emacs release, we
can also alter behavior, impose new restrictions, etc., as it gets
exercised and we learn better about compatibility issues we might've
overlooked.
Ken
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: advice needed for multi-threading patch
2009-09-24 20:04 ` Tom Tromey
` (3 preceding siblings ...)
2009-09-28 14:52 ` Ted Zlatanov
@ 2009-10-05 6:02 ` joakim
4 siblings, 0 replies; 54+ messages in thread
From: joakim @ 2009-10-05 6:02 UTC (permalink / raw
To: Tom Tromey
Cc: Emacs development discussions, Giuseppe Scrivano, Stefan Monnier
Tom Tromey <tromey@redhat.com> writes:
>>>>>> "Giuseppe" == Giuseppe Scrivano <gscrivano@gnu.org> writes:
>
>>> That's wonderful. Does that mean that Gnus really runs in its own
>>> thread while you can do normal editing in other buffers?
>
> Giuseppe> Yes, it does (except some errors). I am concentrating all my
> Giuseppe> efforts on gnus, I think that if threads work well for gnus
> Giuseppe> they will work quite well with any other package.
>
> This is awesome.
>
> Giuseppe> At the moment it takes a lot to load folder, because there are
> Giuseppe> not enough yield on the other thread and probably gnus switch
> Giuseppe> buffers too often (causing a yield).
>
> I think we need to yield when a thread is waiting for I/O. And, we
> probably need a way to associate a process filter with a thread, so that
> whatever input Gnus gets is processed in the right thread.
How will the threading patch work for something like Eshell and Dired?
Non blocking file operations would be a major improvement IMHO.
> Tom
>
--
Joakim Verona
^ permalink raw reply [flat|nested] 54+ messages in thread
end of thread, other threads:[~2009-10-05 6:02 UTC | newest]
Thread overview: 54+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-08-26 4:38 advice needed for multi-threading patch Tom Tromey
2009-08-26 7:06 ` Ken Raeburn
2009-08-26 14:52 ` Stefan Monnier
2009-08-26 18:50 ` Ken Raeburn
2009-08-27 3:12 ` Stefan Monnier
2009-08-27 6:28 ` Ken Raeburn
2009-08-27 17:02 ` Stefan Monnier
2009-08-26 16:08 ` Tom Tromey
2009-08-26 18:50 ` Ken Raeburn
2009-08-27 5:07 ` Miles Bader
2009-08-27 6:39 ` Ken Raeburn
2009-08-27 6:50 ` Miles Bader
2009-08-28 20:48 ` Juri Linkov
2009-08-28 22:15 ` Miles Bader
2009-08-28 23:27 ` Juri Linkov
2009-08-28 23:54 ` Miles Bader
2009-08-29 20:21 ` Richard Stallman
2009-08-26 15:02 ` Stefan Monnier
2009-08-26 15:31 ` Tom Tromey
2009-08-26 19:18 ` Stefan Monnier
2009-09-18 22:59 ` Tom Tromey
2009-09-19 0:09 ` Stephen J. Turnbull
2009-09-19 0:32 ` Chong Yidong
2009-09-21 21:19 ` Stefan Monnier
2009-09-21 21:50 ` Tom Tromey
2009-09-22 14:24 ` Stefan Monnier
2009-09-22 23:59 ` Ken Raeburn
2009-09-23 3:11 ` Stefan Monnier
2009-09-23 15:53 ` Chong Yidong
2009-09-23 3:16 ` Tom Tromey
2009-09-24 17:25 ` Stefan Monnier
2009-09-24 17:57 ` Tom Tromey
2009-09-27 20:59 ` Tom Tromey
2009-09-27 23:05 ` Stefan Monnier
2009-09-28 4:27 ` Tom Tromey
2009-09-29 0:27 ` Stefan Monnier
2009-09-29 2:26 ` Ken Raeburn
2009-09-29 3:20 ` Stefan Monnier
2009-09-29 3:57 ` Ken Raeburn
2009-09-29 3:33 ` Tom Tromey
2009-09-29 4:07 ` Ken Raeburn
2009-09-29 2:30 ` Tom Tromey
2009-09-23 18:43 ` Giuseppe Scrivano
2009-09-24 17:29 ` Stefan Monnier
2009-09-24 18:53 ` Giuseppe Scrivano
2009-09-24 20:04 ` Tom Tromey
2009-09-24 21:59 ` Stefan Monnier
2009-09-24 22:23 ` Giuseppe Scrivano
2009-09-24 22:47 ` Ken Raeburn
2009-09-28 14:52 ` Ted Zlatanov
2009-10-05 6:02 ` joakim
2009-09-28 7:44 ` Lynbech Christian
2009-08-29 0:28 ` Giuseppe Scrivano
2009-08-29 4:57 ` Tom Tromey
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.