Very interesting analysis of "the state of Emacs"

Very interesting analysis of "the state of Emacs"

[Thomas Lord]

Emacs developers might find value in reading this analysis by Steve Yegge:

http://steve-yegge.blogspot.com/2008/04/xemacs-is-dead-long-live-xemacs.html


It is not only about XEmacs but considers some interesting comparisons
between Emacs and the IDEs that Emacs is currently thinking of
"catching up" to.  It calls into question, hopefully in a useful way,
what "catching up" might mean.

-t

Re: Very interesting analysis of "the state of Emacs"

[Paul Michael Reilly]

Thomas Lord wrote:
 > Emacs developers might find value in reading this analysis by Steve 
Yegge:
 >
 > 
http://steve-yegge.blogspot.com/2008/04/xemacs-is-dead-long-live-xemacs.html 


This is a terrific read.  It captures the top two items on my
Gnu/Emacs wish list: first class browsing and multi-threading.

If ever I prayed for the success of an Emacs library coding effort, it
would definitely have been for w3 mode by William Perry.  But it
clearly was a very hard problem.  I've seen signs of improvement in
graphics rendering over the years that makes me believe a decent
browsing experience can be had.  But it's not there yet.

The other hard problem is multi-tasking in the Emacs Lisp engine.
RMS once left me with the impression that this was virtually
intractable, especially if one wanted to have existing Elisp code base
compatibility, a reasonable thing to want.

So I see the best bet for achieving the holy grail of fast browsing,
emacs editing and multi-threading will be via embedding Emacs into the
browser such that every tab/screen IS an Emacs buffer/frame, rendered
in one of some small number (two?) of easily selectable alternatives.
Admittedly, how multi-threading fits into that model is blurry at
best.

But then Steve's point about this solution being fraught with ugliness
if left in the hands of the Mozilla foundation is extremely well
taken.

So all in all, suggesting that the XEmacs folks be encouraged to
pursue these kinds of hard problems under the Gnu/Emacs umbrella is a 
great suggestion, IMHO.

-pmr

Re: Very interesting analysis of "the state of Emacs"

[Richard M Stallman]

    The other hard problem is multi-tasking in the Emacs Lisp engine.
    RMS once left me with the impression that this was virtually
    intractable, especially if one wanted to have existing Elisp code base
    compatibility, a reasonable thing to want.

I think "intractable" might be too pessimistic.  It is certainly
a lot of work, but someone could give it a try.

Re: Very interesting analysis of "the state of Emacs"

[Thomas Lord]

Richard M Stallman wrote:
>     The other hard problem is multi-tasking in the Emacs Lisp engine.
>     RMS once left me with the impression that this was virtually
>     intractable, especially if one wanted to have existing Elisp code base
>     compatibility, a reasonable thing to want.
>
> I think "intractable" might be too pessimistic.  It is certainly
> a lot of work, but someone could give it a try.
>
>   

A study group to map out the design possibilities while giving 
consideration to what is practical with the existing code base might 
make a nice side project.   Y'know, the ol' "measure twice, cut once" 
thing.   Might not be "affordable" in practical terms, I acknowledge.

-t

Re: Very interesting analysis of "the state of Emacs"

[Stephen Eilert]

Richard M Stallman wrote:
>     The other hard problem is multi-tasking in the Emacs Lisp engine.
>     RMS once left me with the impression that this was virtually
>     intractable, especially if one wanted to have existing Elisp code base
>     compatibility, a reasonable thing to want.
>
> I think "intractable" might be too pessimistic.  It is certainly
> a lot of work, but someone could give it a try.
>
>
>   
Before diving in the merits of whether or not it is possible to add 
multi-tasking to Emacs (by that I assume full-blown threads), what are 
the problems this is trying to solve?

Is it to add background processing (as in, file indexing, background 
compilation, downloads and the like)? If then, a notion of task 
priorities could be discussed. For instance, Eclipse knows that, in 
order to deploy an application, it must be compiled first. It 
understands that those tasks cannot happen in parallel and should be 
queued. On the other hand, you can start downloads (usually, plugins and 
updates for said plugins) right away.

It could be a way to use the increasing amount of available processing 
cores in personal computers. Then again, Emacs doesn't seem like a 
particularly CPU-bound application.

So I am at a loss why this is so important. Could someone clarify?


Stephen

Re: Very interesting analysis of "the state of Emacs"

[dhruva]

If we are looking at concurrency, there is another paradigm based on
maintaining multiple internal function call stacks which a scheduler
can schedule in some fair fashion. I am talking of stackless Python
implementation. You really do not have multiple threads but get
simulated concurrency through stack switching. For IO intensive usage
using async IO with stackless might make a good candidate.

-dhruva

On 4/30/08, Stephen Eilert <spedrosa@gmail.com> wrote:
> Richard M Stallman wrote:
> >     The other hard problem is multi-tasking in the Emacs Lisp engine.
> >     RMS once left me with the impression that this was virtually
> >     intractable, especially if one wanted to have existing Elisp code base
> >     compatibility, a reasonable thing to want.
> >
> > I think "intractable" might be too pessimistic.  It is certainly
> > a lot of work, but someone could give it a try.
> >
> >
> >
> Before diving in the merits of whether or not it is possible to add
> multi-tasking to Emacs (by that I assume full-blown threads), what are
> the problems this is trying to solve?
>
> Is it to add background processing (as in, file indexing, background
> compilation, downloads and the like)? If then, a notion of task
> priorities could be discussed. For instance, Eclipse knows that, in
> order to deploy an application, it must be compiled first. It
> understands that those tasks cannot happen in parallel and should be
> queued. On the other hand, you can start downloads (usually, plugins and
> updates for said plugins) right away.
>
> It could be a way to use the increasing amount of available processing
> cores in personal computers. Then again, Emacs doesn't seem like a
> particularly CPU-bound application.
>
> So I am at a loss why this is so important. Could someone clarify?
>
>
> Stephen
>
>
>

-- 
Sent from Gmail for mobile | mobile.google.com

Contents reflect my personal views only!

Re: Very interesting analysis of "the state of Emacs"

[Miles Bader]

Stephen Eilert <spedrosa@gmail.com> writes:
> Before diving in the merits of whether or not it is possible to add
> multi-tasking to Emacs (by that I assume full-blown threads), what are
> the problems this is trying to solve?

One thing I've heard a lot is that it's annoying to wait 5 minutes for
gnus to finish reading in a huge group or something.

Of course Gnus could be rewritten to do potentially lengthy stuff in the
background (and that's what usually happens with such things ... e.g.,
"M-x man"), but it's probably not easy, and in many cases not even
desirable -- when the wait is short, I think it's less confusing for the
user for such actions to be synchronous.

-Miles

-- 
((lambda (x) (list x x)) (lambda (x) (list x x)))

Re: Very interesting analysis of "the state of Emacs"

[Paul Michael Reilly]

Stephen Eilert wrote:
> Richard M Stallman wrote:
>>     The other hard problem is multi-tasking in the Emacs Lisp engine.
>>     RMS once left me with the impression that this was virtually
>>     intractable, especially if one wanted to have existing Elisp code 
>> base
>>     compatibility, a reasonable thing to want.
>>
>> I think "intractable" might be too pessimistic.  It is certainly
>> a lot of work, but someone could give it a try.
>>
>>
>>   
> Before diving in the merits of whether or not it is possible to add 
> multi-tasking to Emacs (by that I assume full-blown threads), what are 
> the problems this is trying to solve?
> 
> Is it to add background processing (as in, file indexing, background 
> compilation, downloads and the like)? If then, a notion of task 
> priorities could be discussed. For instance, Eclipse knows that, in 
> order to deploy an application, it must be compiled first. It 
> understands that those tasks cannot happen in parallel and should be 
> queued. On the other hand, you can start downloads (usually, plugins and 
> updates for said plugins) right away.
> 
> It could be a way to use the increasing amount of available processing 
> cores in personal computers. Then again, Emacs doesn't seem like a 
> particularly CPU-bound application.
> 
> So I am at a loss why this is so important. Could someone clarify?

Over the years I've frequently encountered situations where I started 
executing some Elisp code that took longer than I expected.  Things like 
rgrepping and accessing the network where the network issues have Emacs' 
dedicated attention (and probably some more examples if I paid more 
close attention).  My first inclination was to start up another instance 
of Emacs and that was cool --- I now typically use half a dozen 
instances at any one time --- but it did occur to me that I was 
essentially running another Elisp engine, i.e. another thread.  So that 
was basically the reason why I had the conversation with Richard in the 
first place.  Much as I'd love to have multi-threading in Emacs it is 
way over my head, at least in any kind of reasonable timeframe.  Now 
fast forward to the era of multiple tabbed browsing, mostly firefox. 
Each tab is essentially a separate thread so if the holy grail of 
marrying multi-site browsing and Emacs is to be obtained some form of 
multi-threading seems intuitively required.  But if some wizard figures 
out how to get the grail with only a half a thread, that works for me. :-)

I think the main point is that those of us who feel annoyed every time 
we have to leave an Emacs frame to do some work can only get 
satisfaction from Emacs when it can do multiple of those things that now 
cause it prevent the User from doing something else inside of Emacs 
while it is cranking where it really doesn't matter whether the cranking 
is on the CPU(s) or in the I/O.

-pmr

Re: Very interesting analysis of "the state of Emacs"

[David Kastrup]

Miles Bader <miles.bader@necel.com> writes:

> Stephen Eilert <spedrosa@gmail.com> writes:
>> Before diving in the merits of whether or not it is possible to add
>> multi-tasking to Emacs (by that I assume full-blown threads), what are
>> the problems this is trying to solve?
>
> One thing I've heard a lot is that it's annoying to wait 5 minutes for
> gnus to finish reading in a huge group or something.
>
> Of course Gnus could be rewritten to do potentially lengthy stuff in the
> background (and that's what usually happens with such things ... e.g.,
> "M-x man"), but it's probably not easy,

M-x customize-variable RET gnus-asynchronous RET

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Very interesting analysis of "the state of Emacs"

[Mathias Dahl]

>  Before diving in the merits of whether or not it is possible to add
> multi-tasking to Emacs (by that I assume full-blown threads), what are the
> problems this is trying to solve?
> ...
>  So I am at a loss why this is so important. Could someone clarify?

From time to time I find myself "stuck" in Dired, doing some operation
on many files or on slow network drives. Some things I do:

 - Browsing the thumbnails of some image directory where no images
have thumbnails generated yet. Yes, this might be possible to fix by
changing `image-dired' in clever ways.
 - `dired-do-shell-command'. Most of the time I could have added "&"
at the end to make the call asynchronous.
 - Marking files containing regexp R (the `% g' command). This might
be possible to solve by rewriting that command, I don't know.

It is quite seldom that I find a need for multi threading (or
whatever) when I just execute a one time single command, but I can see
that those with a heavy Gnus config might suffer, as people have
mentioned.

Re: Very interesting analysis of "the state of Emacs"

[Tom Tromey]

>>>>> "Miles" == Miles Bader <miles.bader@necel.com> writes:

Miles> One thing I've heard a lot is that it's annoying to wait 5
Miles> minutes for gnus to finish reading in a huge group or
Miles> something.

I haven't seen 5 minute pauses since the bad old days -- faster
machines have made elisp look faster.

However, I do experience the occasional annoying pause when an nntp
server is down.  In cases like this it would be nice to have an
asynchronous gnus that connects in the background.

Also, blocking operations sometimes mess with other applications
running in Emacs.  E.g., if blocked too long ERC seems to time out its
connection to irc servers.

Miles> Of course Gnus could be rewritten to do potentially lengthy stuff in the
Miles> background (and that's what usually happens with such things ... e.g.,
Miles> "M-x man"), but it's probably not easy, and in many cases not even
Miles> desirable -- when the wait is short, I think it's less confusing for the
Miles> user for such actions to be synchronous.

My ideal is that no Emacs operation be allowed to block Emacs for
"very long" (like, less than 1 second would be nice).

It is easy to make an async operation look like it is blocking, if
that is desirable.  However, in general I think it probably is not.  I
rather like how M-x man works, or async vc-diff or vc-annotate, and I
don't think they are super confusing to the new user.

So, instead of having "g" in gnus block Emacs entirely, it could
simply block operations in the *Group* buffer and present a message
somewhere saying what is happening.

Tom

Re: Very interesting analysis of "the state of Emacs"

[Thomas Lord]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

Miles Bader wrote:
> Stephen Eilert <spedrosa@gmail.com> writes:
>   
>> Before diving in the merits of whether or not it is possible to add
>> multi-tasking to Emacs (by that I assume full-blown threads), what are
>> the problems this is trying to solve?
>>     
>
> One thing I've heard a lot is that it's annoying to wait 5 minutes for
> gnus to finish reading in a huge group or something.
>
> Of course Gnus could be rewritten to do potentially lengthy stuff in the
> background (and that's what usually happens with such things ... e.g.,
> "M-x man"), but it's probably not easy, and in many cases not even
> desirable -- when the wait is short, I think it's less confusing for the
> user for such actions to be synchronous.
>   

There is that.

Another place where threads could help is with complex commands.

At present, concurrent complex command invocations must strictly "stack":
the order of invocation is always the reverse of the order of execution of
commands.   This can be especially annoying and confusing when you are
invoking complex commands in two completely unrelated windows or frames.

With threads, you could (conceivably) have multiple, independent recursive
edits.

-t




> -Miles
>
>   


[-- Attachment #2: Type: text/html, Size: 1892 bytes --]

Re: Very interesting analysis of "the state of Emacs"

[Richard M Stallman]

    If we are looking at concurrency, there is another paradigm based on
    maintaining multiple internal function call stacks which a scheduler
    can schedule in some fair fashion. I am talking of stackless Python
    implementation. You really do not have multiple threads but get
    simulated concurrency through stack switching.

The idea is to get effective parallel execution for Lisp programs and
redisplay.  This doesn't mean using multiple threads at the C level.
It means using whatever method is convenient.

Re: Very interesting analysis of "the state of Emacs"

[Richard M Stallman]

    Before diving in the merits of whether or not it is possible to add 
    multi-tasking to Emacs (by that I assume full-blown threads), what are 
    the problems this is trying to solve?

Multi-terminal use by more than one user
needs parallelism in order to fully work right.

Re: Very interesting analysis of "the state of Emacs"

[David Hansen]

On Wed, 30 Apr 2008 18:00:49 -0400 Richard M. Stallman wrote:

>     If we are looking at concurrency, there is another paradigm based on
>     maintaining multiple internal function call stacks which a scheduler
>     can schedule in some fair fashion. I am talking of stackless Python
>     implementation. You really do not have multiple threads but get
>     simulated concurrency through stack switching.
>
> The idea is to get effective parallel execution for Lisp programs and
> redisplay.  This doesn't mean using multiple threads at the C level.
> It means using whatever method is convenient.

I don't think we would even need a scheduler, co-routines and something
like a `yield' function seem to be perfectly fine.  This would also
minimize the risk it breaks existing code (though existing code would
still block emacs, but that's a transition that can be made over time
when needed).

I would strongly oppose OS level threads.  It's just to easy to fuck
things up and I think elisp should keep things simple.

There are some small languages that implement this (e.g. Lua or Scheme
(not in the standard but it's a natural use for continuations)).  The
main problem I would see is emacs dynamic scope, e.g. what's the value
of a variable in one thread if another one let binds it.

David

Re: Very interesting analysis of "the state of Emacs"

[Thomas Lord]

Richard M Stallman wrote:
>     Before diving in the merits of whether or not it is possible to add 
>     multi-tasking to Emacs (by that I assume full-blown threads), what are 
>     the problems this is trying to solve?
>
> Multi-terminal use by more than one user
> needs parallelism in order to fully work right.
>
>
>   


That's brilliant.

As GNU Emacs turns attention to competing more strongly as an IDE, one 
application for that kind of multi-threadedness is (of course) "pair 
programming".   There are many other, perhaps more compelling, examples 
of the value of multi-user sessions -- but that is one that will be 
familiar to many, I think.

-t

Re: Very interesting analysis of "the state of Emacs"

[Thomas Lord]

David Hansen wrote:
> There are some small languages that implement this (e.g. Lua or Scheme
> (not in the standard but it's a natural use for continuations)).  The
> main problem I would see is emacs dynamic scope, e.g. what's the value
> of a variable in one thread if another one let binds it.
>   

As usual, Henry Baker wrote some stuff years ago that might be helpful 
there.
Poking around with the obvious web searches should turn it up.

Did anyone ever get any further on the "Emacs lisp on GNU Guile" idea?
I'm not too familiar with the current state of Guile these days but, at 
one time,
while it was never clear we could do it perfectly, there was non-trivial 
effort and
decision making in that direction.


-t

Re: Very interesting analysis of "the state of Emacs"

[Jonathan Rockway]

* On Wed, Apr 30 2008, David Hansen wrote:
> There are some small languages that implement this (e.g. Lua or Scheme
> (not in the standard but it's a natural use for continuations)).  The
> main problem I would see is emacs dynamic scope, e.g. what's the value
> of a variable in one thread if another one let binds it.

I don't think let bindings are the problem, it's easy for two
invocations of the same function to have separate dynamic scopes.
Consider:

  (defun foo nil (let (a b c) (bar)))
  (defun bar nil (message "(a %s) (b %s) (c %s)" a b c))
 
If we start two foos in parallel, everything works:

  1:                          2:
  foo                         foo
  let #<scope (a, b, c) 0x1>  let #<scope (a, b, c) 0x2>
  bar (uses scope 0x1)        bar (uses scope 0x2)

Basically, each invocation is logically a separate process.  I don't see
any problems with something like this.  The dynamic scope is not a
show-stopper, anyway.

The issue is when we start putting global variables into the mix:

  (defvar shared 42)
  (defun foo nil (incf shared) (incf shared))
  (defun bar nil (decf shared))

If foo and bar start running at the same time, what does shared end up
being?  Without parallelism, you can guarantee that the output of bar
will always be < the initial value of shared.  But not anymore; if "foo"
happens to run while "bar" is running, that's no longer true.

This is a bad example, but this sort of thing is a big problem.  Perhaps
it can be handled by giving each "thread" a copy of the global VM state.

But of course... what do you do for things that can't be copied, like
buffers or window layouts?  What happens if something like this
executes in parallel:

   A (some-slow-function) (goto-char 42)
   B (save-excursion (goto-char 0) (do-something))

And the execution order happens to be:

  (goto-char 1234)
  A (some-slow-function)
  B (save-excursion (goto-char 0)
  A (goto-char 42)
  B (do-something)
  B )

Where is the point now?  Logically it would be at 42, since the
save-excursion shouldn't have moved it... but the save-excursion sets it
back to the initial value of 1234 when it unwinds.  This makes sense
when you look at the execution order, but that's non-deterministic so it
will never be consistent.

Anyway, hopefully someone has some ideas on what to do here.  I admit I
haven't looked at how sxemacs handles this yet. Maybe we can just deal
with locks?  At least in that case my IMAP mail could download while I
am typing in another buffer :)

Regards,
Jonathan Rockway

-- 
print just => another => perl => hacker => if $,=$"

Re: Very interesting analysis of "the state of Emacs"

[David Hansen]

On Wed, 30 Apr 2008 23:23:03 -0500 Jonathan Rockway wrote:

> But of course... what do you do for things that can't be copied, like
> buffers or window layouts?  What happens if something like this
> executes in parallel:
>
>    A (some-slow-function) (goto-char 42)
>    B (save-excursion (goto-char 0) (do-something))
>
> And the execution order happens to be:
>
>   (goto-char 1234)
>   A (some-slow-function)
>   B (save-excursion (goto-char 0)
>   A (goto-char 42)
>   B (do-something)
>   B )
>
> Where is the point now?  Logically it would be at 42, since the
> save-excursion shouldn't have moved it... but the save-excursion sets it
> back to the initial value of 1234 when it unwinds.  This makes sense
> when you look at the execution order, but that's non-deterministic so it
> will never be consistent.
>

That's exactly why I think co-routines are the way to go.  You just
don't `yield' within a `save-excursion' (the macro could even flet it to
some NOP).  And if you do `yield' you know exactly that you can't rely
on some state before it.

I don't think globals are a huge problem.  It's either some internal
state of a package.  Then it's just the job of the programmer of the
package to be more careful.  Or it's some user setting, then you have to
let bind it at the beginning of your lengthy calculation.

(let ((option *user-setting*)) (do-lots-of-stuff))

This would be a rare exception anyways, most commands are short and fast
and won't `yield' at all.

David

Re: Very interesting analysis of "the state of Emacs"

[Miles Bader]

Jonathan Rockway <jon@jrock.us> writes:
> The issue is when we start putting global variables into the mix:
...
> Anyway, hopefully someone has some ideas on what to do here.  I admit I
> haven't looked at how sxemacs handles this yet. Maybe we can just deal
> with locks?  At least in that case my IMAP mail could download while I
> am typing in another buffer :)

If the multi-threading were cooperative (as rms suggested), then such
problems would obviously be a bit easier to manage -- you can basically
just say "no context switches except at well defined points", and define
these "points" to be (1) user interaction/recursive edits [where the
user can do something to "screw up the state" even today], or (2)
explicit calls to yield.

Even with such a restrictive system, I think it's _much_ easier to make
a potentially slow bit of elisp code (gnus or whatever) multi-threadable
by sticking in (yield) calls at strategic places, and of course you'd
make sure that these places were chosen so that any global state
depended on after the yield returns is restricted to places that are
very unlikely to be modified (e.g., if gnus is reading from nntp, for
instance, it could keep all state in its own scratch buffers).

However I think there are potentially additional problems with dynamic
scope:  remember, elisp uses shallow scoping, where binding a variable
is basically "save old value, and set global".  For normal variables,
this could be replaced by deep-binding, which is more multi-threading
friendly (my "lexbind" branch already uses deep-binding in the
interpreter), but afaik, the use of shallow-binding in elisp is kind of
intertwined with the implementation of buffer-local variables and the
like, and I'm not so sure how easy it would be to handle such things
with a new deep-binding implementation.

-Miles

-- 
Admiration, n. Our polite recognition of another's resemblance to ourselves.

Re: Very interesting analysis of "the state of Emacs"

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Hansen wrote:

> >There are some small languages that implement this (e.g. Lua or Scheme
> >(not in the standard but it's a natural use for continuations)) [...]

Yes, co-routines crossed my mind too as a "classical" and natural
soultion.

Thomas Lord wrote:

> As usual, Henry Baker wrote some stuff years ago that might be helpful 
> there.
> Poking around with the obvious web searches should turn it up.

Tom,

thanks for the pointer. My customers would curse you if I told them ;-)

- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFIGXGFBcgs9XrR2kYRAssKAJ9cVZvZDk3JKUketlQqRFLCXowahACfaZXJ
kcaxKQ/4rGhFYbnyhkoqlfE=
=Yd7b
-----END PGP SIGNATURE-----

Re: Very interesting analysis of "the state of Emacs"

[Jonathan Rockway]

* On Thu, May 01 2008, Miles Bader wrote:
> Jonathan Rockway <jon@jrock.us> writes:
>> The issue is when we start putting global variables into the mix:
> ...
>> Anyway, hopefully someone has some ideas on what to do here.  I admit I
>> haven't looked at how sxemacs handles this yet. Maybe we can just deal
>> with locks?  At least in that case my IMAP mail could download while I
>> am typing in another buffer :)
>
> If the multi-threading were cooperative (as rms suggested), then such
> problems would obviously be a bit easier to manage -- you can basically
> just say "no context switches except at well defined points", and define
> these "points" to be (1) user interaction/recursive edits [where the
> user can do something to "screw up the state" even today], or (2)
> explicit calls to yield.

Yeah, thinking about this some more, I definitely agree with you.  99.9%
of emacs works fine the way it is, so this seems like the cleanest way
to make long operations not block.

I don't know the emacs core well enough to start implementing this, but
I would be glad to help out if someone else takes the lead. :)

Regards,
Jonathan Rockway

-- 
print just => another => perl => hacker => if $,=$"

Re: Very interesting analysis of "the state of Emacs"

[Stefan Monnier]

>> The issue is when we start putting global variables into the mix:
> ...
>> Anyway, hopefully someone has some ideas on what to do here.  I admit I
>> haven't looked at how sxemacs handles this yet. Maybe we can just deal
>> with locks?  At least in that case my IMAP mail could download while I
>> am typing in another buffer :)

> If the multi-threading were cooperative (as rms suggested), then such
> problems would obviously be a bit easier to manage -- you can basically
> just say "no context switches except at well defined points", and define
> these "points" to be (1) user interaction/recursive edits [where the
> user can do something to "screw up the state" even today], or (2)
> explicit calls to yield.

W.r.t to concurrency, I think we need to think about how to introduce it
into Emacs, indeed.  The reason why I think so is because in order to
evolve Emacs needs to eat up more CPU.  You may think Emacs is not CPU
bound, but it's only the case because it only uses what is available.
Try to run Emacs-22 on a 486 to get an idea of what I mean.
Single-thread CPU power is very unlikely to increase significantly in
the future, so if we want to get access to the additional CPU power
we'll need concurrency.

Now we already have concurrency, in the form of start-process.
`flyparse' and `flymake' do just that.  And maybe that's good enough.
But I expect that closer integration of additional threads will be
needed/useful at some point.

Other than separate processes, we can add threads without affecting
Elisp, by using separate threads for the GUI and for the redisplay.
Even the redisplay could be parallel by using separate threads per
frame/window.

As for adding concurrency to Elisp, the core difficulty will be the heap
of code that assumes concurrency doesn't exist, as well as the
interaction between buffer-local and let-bound variables.

But indeed, I think a good first step is to add coroutines.  This should
not be terribly difficult to do (especially since it's OK for it to
break code as long as it's only broken when you use `yield').
Patches welcome.

> However I think there are potentially additional problems with dynamic
> scope:  remember, elisp uses shallow scoping, where binding a variable
> is basically "save old value, and set global".  For normal variables,
> this could be replaced by deep-binding, which is more multi-threading
> friendly (my "lexbind" branch already uses deep-binding in the
> interpreter), but afaik, the use of shallow-binding in elisp is kind of
> intertwined with the implementation of buffer-local variables and the
> like, and I'm not so sure how easy it would be to handle such things
> with a new deep-binding implementation.

The way I see it, multithreading within a buffer is a problem we can
ignore for the next 20 years.  So let-binding a buffer-local var can
be handled via shallow-binding (whereas let-binding of global variables
needs to use deep binding).  So the only difficulty is to deal with
variables which are not make-variable-buffer-local but only
make-local-variable where we need to make sure we can always
unequivocally know whether to use deep or shallow binding.


        Stefan

CEDET and threads (was Re: Very interesting analysis of "the state of Emacs")

[Eric M. Ludlam]

Hi,

  The stuff CEDET and ECB does has often caused us to wish for any
kind of concurrency.  Right now CEDET/Semantic parses buffers which,
right now, can be quick for reasonably sized buffers, but can be quite
slow for something like emacs/src/lisp.h.  In addition, providing
smart completion requires building large tables to make lookup go
faster.

  I use an idle timer, and do this work at idle time.  The parsing
code and other routines called from the idle timer will `throw' on
user input at key locations, and the next idle time will resume
processing.

  For each task that needs background processing it is required that
it be updated to be interruptible, which makes it that much harder to
move processing to idle time.  A side effect is that when a user
request comes in for some feature, often times there can be a long
wait, depending on the size of the code base.

  I've tuned CEDET to be close to instantaneous for smaller code
bases, but on larger code bases, I sometimes can have a 1 minute wait
building up databases to provide the right answer.  Once I have the
tables on hand, things go quickly again.

  I've toyed with the idea of having a second Emacs as a subprocess to
do some of this work concurrently.  This would free up CEDET to begin
performing speculative parsing to keep the user interaction snappy
when switching around between different sections of large project.
I'm not sure how to transfer the data around though.

  For me, the ideal concurrent task setup would share buffers and
share large lisp data structures that a user might be interacting with
in order to keep parsing data and textual decorations up to date.  It
would also be great if whenever a user typed ".", a concurrent task
could run to get completions, and allow the user to keep typing after
the period.  Once the answer is ready, an inline completion mode could
startup without having to wait for the user to explicitly stop typing.
This is very hard to do using idle timers and cooperative
multitasking.

  Well, this message is longer than I expected.  Suffice it to say
that CEDET could use concurrency at the full shared environment level,
but any kind of concurrency would be helpful.

Eric

-- 
          Eric Ludlam:                       eric@siege-engine.com
   Siege: www.siege-engine.com          Emacs: http://cedet.sourceforge.net

Re: Very interesting analysis of "the state of Emacs"

[Richard M Stallman]

    The way I see it, multithreading within a buffer is a problem we can
    ignore for the next 20 years.

I am not sure of that.  In multi-terminal use it is very likely that
more than one terminal will be operating in the same buffer.

With cooperative multithreading, it will be easy to avoid
changing Lisp threads in the middle of any operation that changes
the buffer contents,  So I don't think operating in the same buffer
will be any problem at all.

Re: Very interesting analysis of "the state of Emacs"

[Stefan Monnier]

>     The way I see it, multithreading within a buffer is a problem we can
>     ignore for the next 20 years.

> I am not sure of that.  In multi-terminal use it is very likely that
> more than one terminal will be operating in the same buffer.

Good point.  But I don't think it's a problem either: what I meant by
"multithreading within a single buffer" is that we'd have a lock per
buffer.  Whenever lisp code enters a buffer, we'd acquire the lock.
I.e. just because the user is looking at a buffer doesn't mean some
other user (with its own thread) elsewhere can't look at that
buffer either.  Just that their respective actions will be serialized by
the buffer's lock.


        Stefan

Re: Very interesting analysis of "the state of Emacs"

[Richard M Stallman]

    Good point.  But I don't think it's a problem either: what I meant by
    "multithreading within a single buffer" is that we'd have a lock per
    buffer.  Whenever lisp code enters a buffer, we'd acquire the lock.

What does it mean to "enter a buffer"?  Does calling `set-buffer' do
that?

If it means entering the code of a primitive that directly examines or
alters the buffer contents, I would suggest not allowing a thread
switch inside of them (or most primitives).  If the threads are
implemented explicitly in our C code, switching can happen only where
we want it to happen.  That would avoid lots of problems.  We would
only allow thread switches at places where Lisp code can be run.

buffer transactions (was Re: Very interesting analysis of "the state of Emacs")

[Nic]

Richard M Stallman <rms@gnu.org> writes:

>     Good point.  But I don't think it's a problem either: what I meant by
>     "multithreading within a single buffer" is that we'd have a lock per
>     buffer.  Whenever lisp code enters a buffer, we'd acquire the lock.
>
> What does it mean to "enter a buffer"?  Does calling `set-buffer' do
> that?
>
> If it means entering the code of a primitive that directly examines or
> alters the buffer contents, I would suggest not allowing a thread
> switch inside of them (or most primitives).  If the threads are
> implemented explicitly in our C code, switching can happen only where
> we want it to happen.  That would avoid lots of problems.  We would
> only allow thread switches at places where Lisp code can be run.

Just an idle thought... has anyone considered transactional support
for buffer modification in emacs? This would enable simultaneous
access to buffers by different processes/threads and (it seems) could
be implemented by changing the insert primitives in scope for a
background process to record a transaction log of all the changes a it
wants to make to a buffer (or variable for that matter) and merging it
with the state of the actual buffer on completion of the process.

-- 
Nic Ferrier
http://www.woome.com - Enjoy the minute!

Re: buffer transactions (was Re: Very interesting analysis of "the state of Emacs")

[Richard M Stallman]

    Just an idle thought... has anyone considered transactional support
    for buffer modification in emacs? This would enable simultaneous
    access to buffers by different processes/threads and (it seems) could
    be implemented by changing the insert primitives in scope for a
    background process to record a transaction log of all the changes a it
    wants to make to a buffer (or variable for that matter) and merging it
    with the state of the actual buffer on completion of the process.

We should just make `atomic-change-group' bind a variable to
temporarily prevent thread switches.  That would do the job
and would be much simpler.

ERC disconnects when blocked too long (was: Very interesting analysis of "the state of Emacs")

[Michael Olson]

[-- Attachment #1: Type: text/plain, Size: 884 bytes --]

Tom Tromey <tromey@redhat.com> writes:

> Also, blocking operations sometimes mess with other applications
> running in Emacs.  E.g., if blocked too long ERC seems to time out its
> connection to irc servers.

Yeah, that is happening because ERC periodically pings the IRC server by
default to make sure that the connection is alive.  When Emacs is
blocked, ERC can't process the server's responses, and it thinks the
connection is dead.  I've been trying to think of a way for ERC to
detect whether it is blocked for more than a certain amount of time, and
take some sort of remedial action like resetting the response timeout.

-- 
|       Michael Olson  |  FSF Associate Member #652     |
| http://mwolson.org/  |  Hobbies: Lisp, HCoop          |
| Projects: Emacs, Muse, ERC, EMMS, ErBot, DVC, Planner |
`-------------------------------------------------------'

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]