Re: What's the problem?

[Simon Josefsson <jas@extundo.com>]

Juri Linkov <juri@jurta.org> writes:

> Simon Josefsson <jas@extundo.com> writes:
>> Luke Gorrie <luke@bluetail.com> writes:
>>> People are writing concurrent programs in Elisp today. Most programs
>>> interacting with external processes and sockets do so without blocking
>>> Emacs. Can't we all just do the same?
>>
>> No.
>>
>> I believe several examples have been given already, but the most
>> prominent I notice frequently is generating summary buffers in Gnus.
>> This is CPU bound.  Most of the time is NOT spent waiting from network
>> IO, even for 64kbps users, if you profile large groups.  It is spent
>> traversing internal data structures, sorting and filtering the
>> headers.  If this is done asynchronously, it would still lock up
>> Emacs.
>
> Many CPU bound tasks can be improved by memoizing computation results.
> Since most cited here example is generating summary buffers in Gnus,
> I want to note that this particular problem can be solved by storing
> computed thread trees in files similar to `.marks' and `.overview'.
> To keep track of computed results input parameters should be stored
> along with computed data.

Yes, it can be improved, but you are talking about caching.  It does
not remove the need to at some point compute the data stored in the
cache.  Then Gnus would look up, which I find unacceptable.

Furthermore, because a lot of what is done during preparing summary
buffers is dynamic, or almost so because it is influenced by hundreds
of user variables that may change over time, I'm not convinced caching
summary buffer data can be done reliable without great pain.  For me,
the summary buffer look different most time I enter it, because each
added or removed article in the group may change large parts of the
summary buffer to be recomputed (which I don't consider a bug).  It is
certainly easier to compute the summary buffer every time it is
needed, and use the user variables during that computation, than
reading from a cache and figure out what has to be re-computed given
the then-current user variables, and then re-compute the data and
store it in the cache.

>> Another example is getting new mail in Gnus.  This is probably more
>> network bound than CPU bound, but doing it asynchronously would still
>> lock up Emacs completely for many seconds, which is unacceptable.  If
>> you use Gnus as a offline reader, getting new mail may take minutes.
>
> Many tasks can be performed by external processes.  Getting new mail
> shouldn't lock up Emacs when done by an external program like movemail.

I meant parsing the new incoming articles and splitting them into the
proper group.  This is done in elisp.

If you profile Gnus with ELP, you see that the amount of time spent
waiting for external processes like movemail is not the biggest cause
of the lockups.  Most of it is spent evaluating elisp.  Even when
getting new mail.  So optimizing the movemail case to be asynchronous
would not improve the situation much, but will make it more difficult
to debug and troubleshoot Gnus.