From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Ihor Radchenko Newsgroups: gmane.emacs.devel Subject: Re: Concurrency via isolated process/thread Date: Mon, 10 Jul 2023 11:30:10 +0000 Message-ID: <87edlg6m2l.fsf@localhost> References: <871qhnr4ty.fsf@localhost> <87r0plxbep.fsf@localhost> <83ilawhpi6.fsf@gnu.org> <87zg48apwr.fsf@localhost> <83edljg8ub.fsf@gnu.org> <87o7knbxr7.fsf@localhost> <838rbrg4mg.fsf@gnu.org> <87ilavbvdr.fsf@localhost> <834jmffvhy.fsf@gnu.org> <878rbrbmwr.fsf@localhost> <83fs5zecpo.fsf@gnu.org> <87351zbi72.fsf@localhost> <83351yevde.fsf@gnu.org> <87cz12ad2w.fsf@localhost> <83a5w6cwdr.fsf@gnu.org> <87pm518m0g.fsf@localhost> <83o7kl9tyj.fsf@gnu.org> <874jmd89us.fsf@localhost> <83cz119lxu.fsf@gnu.org> <87v8et6q5m.fsf@localhost> <838rbp9h6c.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="7883"; mail-complaints-to="usenet@ciao.gmane.io" Cc: monnier@iro.umontreal.ca, luangruo@yahoo.com, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Mon Jul 10 13:30:54 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qIp6T-0001sr-NU for ged-emacs-devel@m.gmane-mx.org; Mon, 10 Jul 2023 13:30:53 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qIp5p-0003O0-6j; Mon, 10 Jul 2023 07:30:13 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qIp5m-0003HR-Om for emacs-devel@gnu.org; Mon, 10 Jul 2023 07:30:10 -0400 Original-Received: from mout01.posteo.de ([185.67.36.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qIp5j-00015z-6t for emacs-devel@gnu.org; Mon, 10 Jul 2023 07:30:10 -0400 Original-Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id 32284240027 for ; Mon, 10 Jul 2023 13:30:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1688988603; bh=viSgVd6heXuZRWgR2h6cs6Rn6Izc5Y+g+d6FebRLOp0=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From; b=NqCVwGtz1s4BfdvqA6KkGzfQYQwyM+xs3GP523xGnkdYRtsrL5zqYFa6aIjss8lUu 0uR1soU43Cs1kypCd+ZbHamHPEuWEK03tPjQ9VuRBbca/+r42jM+rFvVXaY4iJ0DXy 9dabV0914tB02EGn1NNZ2BMGRwUPiwXVnv2CVIvWxlzZBomMix9BFvHJ39lfUvntiP Gtk+paYTftpil/wvDSrcLxh4CZIWASIQ55v3rgRfDcPrOMfLRnGTMTj344rm1Osa5i HADaqBKdIS86/ztTgQ4jNIPDRKBubaX1AaEw7FbSENJcN1dx87+KpQ/1WxeR4pqSws 77M+VrVYi74Aw== Original-Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4R01y20mKPz6tyc; Mon, 10 Jul 2023 13:30:01 +0200 (CEST) In-Reply-To: <838rbp9h6c.fsf@gnu.org> Received-SPF: pass client-ip=185.67.36.65; envelope-from=yantar92@posteo.net; helo=mout01.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:307711 Archived-At: Eli Zaretskii writes: >> And how frequently are case-table and display-table changed? AFAIK, not >> frequently at all. > > Why does it matter? Would you disallow programs doing that just > because you think it's infrequent? Not disallow. But the programs setting case-table and display-table will block. That will be only a fraction of programs. > Processing of various protocols frequently requires to do stuff like > > (with-case-table ascii-case-table > DO SOMETHING...) > > because otherwise some locales, such as the Turkish one, cause trouble > with case conversion of the upper-case 'I'. So changing the > case-table of a buffer is not such an outlandish operation to do. Yet, a lot of Elisp programs do not need to deal with case tables. And if we talk about this particular use case, may we simply allow let-binding case tables? >> May you please explain a bit more about the situation you are referring >> to? My above statement was about consing, not GC. > > Consing can trigger GC if we detect the memory-full situation, see > alloc.c:memory_full. I see. AFAIU, we can then raise a flag that GC is necessary, so that other threads will stop next time they reach maybe_gc, until GC is complete. >> For GC, as I mentioned earlier, we can pause each thread once maybe_gc() >> determines that GC is necessary, until all the threads are paused. Then, >> GC is executed and the threads continue. > > If all the threads are paused, which thread will run GC? I imagine that a dedicated thread will be used for GC. It will do nothing until a signal, acquire a global lock, perform GC, release the lock, and continue waiting. >> If an async thread want to request redisplay, it should be possible. But >> the redisplay itself must not be done by this same thread. Instead, the >> thread will send a request that Emacs needs redisplay and optionally >> block until that redisplay finishes (optionally, because something like >> displaying notification may not require waiting). The redisplay requests >> will be processed separately. > > Ouch! this again kills opportunities for gains from concurrent > processing. It means, for example, that we will be unable to run in a > thread some processing that affects a window and reflect that > processing on display when it's ready. We may or may not be able to get async redisplay, depending on Emacs display implementation: 1. A thread that requests redisplay will send a signal request to perform redisplay. 2. Another thread, responsible for redisplay, will retrieve the signal and process it. That thread may or may not do it asynchronously. AFAIU, it is currently not possible to redisplay asynchronously. But if we can somehow make redisplay (or parts of it) asynchronous, we will just need to make adjustments to redisplay thread implementation. Other async threads will not need to be changed. I just do not know enough about redisplay to dive into possibility of async in it. I am afraid that this discussion is already complex enough, and discussing redisplay will add yet another layer of complexity on top. >> We can make reading input using similar idea to the above, but it will >> always block until the response. > > This will have to be designed and implemented first, since we > currently have no provision for multiple prompt sources. >> For non-blocking input, you said that it has been discussed. >> I do vaguely recall such discussion in the past and I even recall some >> ideas about it, but it would be better if you can link to that >> discussion, so that the participants of this thread can review the >> previously proposed ideas. > > https://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00456.html Thanks! After re-reading that discussion, it looks like the most discussed idea was about maintaining a queue of input requests and prompting users at appropriate times. This is similar to what I proposed - async threads should query to process input and then the input order and time will be decided elsewhere. The rest of the discussion revolved around the fact that 1. Input may involve non-trivial Elisp that requires thread context - so the input processing must know the context of the original thread. Ergo, thread itself may need to be involved to process input, not some other thread. For async threads, it means that input may need to pause the whole async thread until scheduler (or whatever code decides when and where to read/display input or query) decides that it is appropriate time to query the user. We may also need to allow some API to make other thread read input independently while the current thread continues on. But that's not a requirement - I imagine that it can be done with `make-thread' from inside the async thread. 2. There should be some kind of indication for the user which thread is requesting input: - Emacs may display that intput is pending in modeline indicator (aka "unread" notifications); or Emacs may display upcoming query in the minibuffer until the user explicitly switches there and inputs the answer. - When input query is actually displayed, there should be an indication which thread the query belongs to - either a thread name or an appropriately designed prompt. Possibly, prompt history for the relevant thread may be displayed. 3. Threads may want to group their prompts into chunks that should be read without interrupt, so that we do not mix two threads like "Login for host1: "->"Password for host1: " and "Login for host2: "->"Password for host2: ". >> > What about changes to frame-parameters? Those don't necessarily >> > affect display. >> >> But doesn't it depend on graphic toolkit? > > Not necessarily: frame parameters are also used for "frame-local" > variables, and those have nothing to do with GUI toolkits. I see. Then, when an async threads modifies frame parameters, it should be allowed to do so. However, during redisplay, redisplay code should block frame parameters - it should not be allowed to change during redisplay. >> >> This means that any code that is using things like >> >> `save-window-excursion', `display-buffer', and other display-related >> >> staff cannot run asynchronously. >> > >> > What about with-selected-window? also forbidden? >> >> Yes. > > Too bad. Well. In theory, this is a problem similar to set-buffer - we need to deal with the fact that Emacs assumes that there is always a single "current" window and frame. I assume that this problem is not more difficult than with set-buffer. If we can solve the problem with global state related to buffer, it should be doable to solve async window/frame setting. (I am being optimistic here) >> >> Async threads will make an assumption that >> >> (set-buffer "1") (goto-char 100) (set-buffer "2") (set-buffer "1") >> >> (= (point) 100) invalid. >> > > ... >> Hmm. I realized that it is already invalid. At least, if `thread-yield' >> is triggered somewhere between `set-buffer' calls and other thread >> happens to move point in buffer "1". > > But the programmer is in control! If no such API is called, point > stays put. And if such APIs are called, the program can save and > restore point around the calls. By contrast, you want to be able to > pause and resume threads at will, which means a thread can be > suspended at any time. So in this case, the programmer will be unable > to do anything against such calamities. Right. So, we may need to store per-thread history of PT, BEGV, and ZV for each buffer that was current during thread execution. >> >> > What if the main thread modifies buffer text, while one of the other >> >> > threads wants to read from it? >> >> >> >> Reading and writing should be blocked while buffer is being modified. >> > >> > This will basically mean many/most threads will be blocked most of the >> > time. Lisp programs in Emacs read and write buffers a lot, and the >> > notion of forcing a thread to work only on its own single set of >> > buffers is quite a restriction, IMO. >> >> But not the same buffers! > > I don't see why not. Of course, one may want to run two async threads that modify the same buffer simultaneously. Not all the Elisp code will need this, but some may. And it is indeed a restriction. But I do not see that it should be the restriction that stops us from implementing async support, even if we cannot solve it. > But that's the case with most Emacs Lisp programs: we rarely try too > hard to make functions pure even if they can be pure. What's more > important, most useful programs cannot be pure at all, because Emacs > is about text processing, which means modifying text, and good Emacs > Lisp programs process text in buffers, instead of returning strings, > which makes them not pure. I am not sure if it is a good aim to design async threads in such a way that _any_ (rather than explicitly designed) Elisp can work asynchronously. It would be cool if we could achieve this, but I do not feel like most of Elisp actually require async. Yes, most of Elisp is about text processing. But when we really need to utilize asynchronous code, it is usually not about reading/writing text - it is about CPU-heavy analysis of text. This analysis is what truly needs async threads. Writing back, if it is necessary, may be separated from the analysis code or even done in separate buffer followed by `buffer-swap-text' or `replace-buffer-contents'. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at . Support Org development at , or support my work at