From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: Concurrency via isolated process/thread Date: Sun, 09 Jul 2023 17:52:29 +0300 Message-ID: <83cz119lxu.fsf@gnu.org> References: <871qhnr4ty.fsf@localhost> <874jmh6v4s.fsf@localhost> <83y1jtgmbw.fsf@gnu.org> <87zg49xfke.fsf@localhost> <83sfa1gjns.fsf@gnu.org> <87r0plxbep.fsf@localhost> <83ilawhpi6.fsf@gnu.org> <87zg48apwr.fsf@localhost> <83edljg8ub.fsf@gnu.org> <87o7knbxr7.fsf@localhost> <838rbrg4mg.fsf@gnu.org> <87ilavbvdr.fsf@localhost> <834jmffvhy.fsf@gnu.org> <878rbrbmwr.fsf@localhost> <83fs5zecpo.fsf@gnu.org> <87351zbi72.fsf@localhost> <83351yevde.fsf@gnu.org> <87cz12ad2w.fsf@localhost> <83a5w6cwdr.fsf@gnu.org> <87pm518m0g.fsf@localhost> <83o7kl9tyj.fsf@gnu.org> <874jmd89us.fsf@localhost> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="32576"; mail-complaints-to="usenet@ciao.gmane.io" Cc: luangruo@yahoo.com, emacs-devel@gnu.org To: Ihor Radchenko , Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Jul 09 16:53:11 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qIVmf-0008Bt-F8 for ged-emacs-devel@m.gmane-mx.org; Sun, 09 Jul 2023 16:53:09 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qIVm3-0001CM-3G; Sun, 09 Jul 2023 10:52:31 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qIVlv-0001CD-VD for emacs-devel@gnu.org; Sun, 09 Jul 2023 10:52:25 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qIVlu-0006Ds-SI; Sun, 09 Jul 2023 10:52:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=PMQWkQfKuPrP4MrqEqNzOKqm0wshOxPNzZoAGNRM4FU=; b=azhILaWAB68b 0fu3cgAUoAXYw6AP73RBpjb9lNT4YfsUyI+vlyH59fiZFBlVGtN72v/eAFzTvM4p+KGGnIiSWo2lD 5pFfgL8OBIAV4MRAFCnQF/J15/363fQG7kIGOzNuZppfuPWI8W4YedbSpouIOBqdwuAd9F5oYTv9z +wb0kjZBvZEjvqfNpCY0T7COpO6jN8LWppOP34n9Fu9/M+1qnRv6O7J8yPVWLWqv11OB7YsX4dThE YtPpDmqTnWmOpjjApn6IcFtCZcPqUqoISA6c2YD+gjBFKhm0QJ0GeP9zgWZ2YvpDXbv0Dn8ActToZ gn87lPQEpXRVx+dfP3M31Q==; Original-Received: from [87.69.77.57] (helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qIVlu-0000Pa-CF; Sun, 09 Jul 2023 10:52:22 -0400 In-Reply-To: <874jmd89us.fsf@localhost> (message from Ihor Radchenko on Sun, 09 Jul 2023 13:58:51 +0000) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:307684 Archived-At: > From: Ihor Radchenko > Cc: luangruo@yahoo.com, emacs-devel@gnu.org > Date: Sun, 09 Jul 2023 13:58:51 +0000 > > Eli Zaretskii writes: > > >> >> > Which variables can safely and usefully be made thread-local? > > ... > > In this case, the assumption is that the gap is always accurate except > > for short periods of time, during which buffer text cannot be accessed > > via buffer positions. > > Thanks for the clarification. > But note that I was suggesting which global state variables may be > converted to thread-local. > > I now understand that the gap can be moved by the code that is not > actually writing text in buffer. However, I do not see how this is a > problem we need to care about more than about generic problem with > simultaneous write. Imagine a situation where we need to process XML or HTML, and since that's quite expensive, we want to do that in a thread. What you are saying is that this will either be impossible/impractical to do from a thread, or will require to lock the entire buffer from access, because the above processing moves the gap. If that is not a problem, I don't know what is, because there could be a lot of such scenarios, and they all will be either forbidden or very hard to implement. > If a variable or object value is being written, we need to block it. > If a buffer object is being written (like when moving the gap or writing > text), we need to block it. And this blocking will generally pose a > problem only when multiple threads try to access the same object, which > is generally unlikely. My impression is that this is very likely, because of the many global objects in Emacs. Moreover, if you intend to allow several threads using the same buffer (and I'm not yet sure whether you want that or not), then the buffer-local variables of that buffer present the same problem as global variables. Take the case-table or display-table, for example: those are buffer-local in many cases, but their changes will affect all the threads that work on the buffer. > 1. There is consing, that is principally not asynchronous. > > It is not guaranteed, but I _hope_ that lock during consing > can be manageable. > > We need to ensure that simultaneous consing will never happen. AFAIU, > it should be ok if something that does not involve consing is running > at the same time with cons (correct me if I am wrong here). What do you do if some thread hits the memory-full condition? The current handling includes GC. > 2. Redisplay cannot be asynchronous in a sense that it does not make > sense that multiple threads, possibly working with different buffers > and different points in those buffers, request redisplay > simultaneously. Of course, it is impossible to display several places > in a buffer at once. But what about different threads redisplaying different windows? is that allowed? If not, here goes one more benefit of concurrent threads. Also, that issue with prompting the user also needs some solution, otherwise the class of jobs that non-main threads can do will be even smaller. > Only a single `main-thread' should be allowed to modify frames, > window configurations, and generally trigger redisplay. And thread > that attempts to do such modifications must wait to become > `main-thread' first. What about changes to frame-parameters? Those don't necessarily affect display. > This means that any code that is using things like > `save-window-excursion', `display-buffer', and other display-related > staff cannot run asynchronously. What about with-selected-window? also forbidden? > Async threads will make an assumption that > (set-buffer "1") (goto-char 100) (set-buffer "2") (set-buffer "1") > (= (point) 100) invalid. If this is invalid, I don't see how one can write useful Lisp programs, except of we request Lisp to explicitly define critical sections. > > What if the main thread modifies buffer text, while one of the other > > threads wants to read from it? > > Reading and writing should be blocked while buffer is being modified. This will basically mean many/most threads will be blocked most of the time. Lisp programs in Emacs read and write buffers a lot, and the notion of forcing a thread to work only on its own single set of buffers is quite a restriction, IMO. > >> >> For example, `org-element-interpret-data' converts Org mode AST to > >> >> string. Just now, I tried it using AST of one of my large Org buffers. > >> >> It took 150seconds to complete, while blocking Emacs. > >> > > >> > It isn't side-effect-free, though. > >> > >> It is, just not declared so. > > > > No, it isn't. For starters, it changes obarray. > > Do you mean `intern'? `intern-soft' would be equivalent there. "Equivalent" in what way? AFAIU, the function does want to create a symbol when it doesn't already exist. > > Those assertions will fire in any useful program with 100% certainty. > > Imagine the plight of an Emacs Lisp programmer who has to write and > > debug such programs. > > > > We have in Emacs gazillion lines of Lisp code, written, debugged, and > > tested during 4 decades. We use those, almost without thinking, every > > day for writing Lisp programs. What you suggest means throwing away > > most of that and starting from scratch. > > There will indeed be a lot of work to make the range of Lisp functions > available for async code large enough. But it does not have to be done > all at once. No, it doesn't. But until we have enough of those functions available, one will be unable to write applications without implementing and debugging a lot of those new functions as part of the job. It will make simple programming jobs much larger and more complicated, especially since it will require the programmers to understand very well the limitations and requirements of concurrent code programming, something Lisp programmers don't know very well, and rightfully so. > > Of course, we first need to make sure that there are no hard blockers, > like global state. I do not think that Elisp code will be the blocker if > we find out how to deal with Emacs global state on C level. > > > I mean, take the simplest thing, like save-buffer-excursion or > > with-selected-window, something that we use all the time, and look how > > much of the global state they access and change. Then imagine that > > you don't have these and need to write programs that switch buffers > > and windows temporarily in thread-safe way. Then reflect on what this > > means for all the other useful APIs and subroutines we have. > > These examples are touching very basics aspects that we need to take > care of for async: (1) point/buffer; (2) unwind; (3) redisplay. > I think that (3) is not something that should be allowed as async. (1) > and (2) are to be discussed. > > P.S. I am struggling to understand swap_in_symval_forwarding: > > /* Unload the previously loaded binding. */ > tem1 = blv->valcell; > > Is the above assignment redundant? I'll let Stefan answer that, since he made the change in commit ce5b453a449 that resulted in this code.