Re: Concurrency via isolated process/thread

[Ihor Radchenko <yantar92@posteo.net>]

Hugo Thunnissen <devel@hugot.nl> writes:

>> Yes, most of Elisp is about text processing. But when we really need to
>> utilize asynchronous code, it is usually not about reading/writing text
>> - it is about CPU-heavy analysis of text. This analysis is what truly
>> needs async threads.
>
> I am skeptical that there are all that many use cases for the 
> asynchronous analysis of text in buffers that are currently being 
> edited.

Org does it via idle timers.
Also, consider searching across many buffers - that's a legitimate use
case. Think about AG/RG and other multi-threaded text searchers.
If the buffers are also structured and need to be parsed, we arrive at
the situation when parallelism can be quite useful.

> ... Correct me if I'm wrong, AFAIU programs that analyze text as it 
> is being edited will usually need to re-parse after every edit anyways, 
> making a parse during the edit itself not all that useful: After all, 
> the result is no longer accurate for the buffer contents by the time it 
> becomes available. A synchronous parser can probably do just as good of 
> a job in such a scenario, as long as it is interruptible by user input.

No. It still makes sense to re-parse incrementally before the point
where the edits are being made. For example, Org parser has to trash
everything that may theoretically be affected by current edits and
re-parse later, synchronously, on demand. This trashing sometimes
involve rather big chunks of text, and it would significantly speed
things up if we could do part of the incremental parsing in parallel.

> I'm not familiar with the C core at all, do you think it might be more 
> realistic to add a little more facilities for asynchronous data streams 
> than to rework the execution model of Emacs to add multi threading? I'm 
> mainly thinking of something like "channels" or "queues" for lisp 
> objects, with an easy to use scheduling mechanism that makes it 
> straightforward for people to not stall the main thread for too long.

AFAIK, sentinels do this already. Of course, they are quite basic.
As for queues, see https://emacsconf.org/2022/talks/async/

> And another (very) nice to have feature: asynchronous filesystem IO. 
> Consider the code below which I use in a package I'm working on. My 
> package reads and parses a large amount of files in the background 
> within a short time frame. Parsing/processing the files in a timely 
> manner is never really an issue, but the blocking IO of 
> `insert-file-contents' does often take so long that it is  impossible to 
> not have the user notice, even if polling `input-pending-p' and yielding 
> in between operations.

Well. I have an opposite problem when `read' takes second to read few Mb
of Elisp data, while inserting it into a temporary buffer is instant.

Also, did you actually profile your use case? I actually doubt that
reading file takes a long time on modern systems. In my previous tests,
I tried to open multiple tens of thousands files using

(let ((buffer (get-buffer-create " *Processing*")))
  (with-current-buffer buffer
    (let (buffer-undo-list t)
      (dolist (f files)
	(insert-file-contents f nil nil nil 'replace)
        (do-staff)))))

And it was very, very fast.
What was not fast (orders of magnitude) is opening files directly that
triggers all kinds of user hooks.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>