From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Ihor Radchenko Newsgroups: gmane.emacs.devel Subject: Re: Concurrency via isolated process/thread Date: Tue, 18 Jul 2023 05:25:55 +0000 Message-ID: <87r0p53i58.fsf@localhost> References: <871qhnr4ty.fsf@localhost> <87pm57p57m.fsf@yahoo.com> <83mt0bjdh5.fsf@gnu.org> <03724893-5b8e-2df5-6001-f2616c8f5c1c@hugot.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="23657"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Eli Zaretskii , emacs-devel@gnu.org To: Hugo Thunnissen Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Tue Jul 18 07:26:51 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qLdEY-0005w2-Ui for ged-emacs-devel@m.gmane-mx.org; Tue, 18 Jul 2023 07:26:51 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qLdDl-0003gK-Mp; Tue, 18 Jul 2023 01:26:01 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qLdDh-0003gC-Go for emacs-devel@gnu.org; Tue, 18 Jul 2023 01:25:57 -0400 Original-Received: from mout02.posteo.de ([185.67.36.66]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qLdDe-0004Y4-69 for emacs-devel@gnu.org; Tue, 18 Jul 2023 01:25:57 -0400 Original-Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 8F8DF240103 for ; Tue, 18 Jul 2023 07:25:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1689657951; bh=KcoBX3YcFqNJ0x4hESq4WqiNiTRPKA6qfstgEwusnOg=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version: Content-Transfer-Encoding:From; b=cSQLrlplgPLgwbC052OBR4N1Pi9/O+PYUG9PmL4yvI/u0JQKJUYuVyXV+DCH3MLw5 iWNl6V5P82lQOOH+NclkOwNuAN7S4TKwKxQyj7WMCGI4GSf8JEv4ecdG05TL9IjKZ3 49nnV0O1OANz1ThfnC80WVaWnp2Vbn9iphHrumNd1AxSH85+/5DYKQkwRfu2sa6eWG QC42RL0cVBvZ8ghJJDMEGz5ewMOyAx1UvqjUg8WvLu83vmbioYqxPUIo7Z7Gn4kG1D hk00JQBAtcRGjQaJz8ve2lnHNZdMQEi40rSOz0jnGeCY9wNB6PJdcqrNG40Lh+uqUZ PQnLVTOKJuV+Q== Original-Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4R4nV670nHz9s3G; Tue, 18 Jul 2023 07:25:50 +0200 (CEST) In-Reply-To: <03724893-5b8e-2df5-6001-f2616c8f5c1c@hugot.nl> Received-SPF: pass client-ip=185.67.36.66; envelope-from=yantar92@posteo.net; helo=mout02.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:307944 Archived-At: Hugo Thunnissen writes: >> Yes, most of Elisp is about text processing. But when we really need to >> utilize asynchronous code, it is usually not about reading/writing text >> - it is about CPU-heavy analysis of text. This analysis is what truly >> needs async threads. > > I am skeptical that there are all that many use cases for the=20 > asynchronous analysis of text in buffers that are currently being=20 > edited. Org does it via idle timers. Also, consider searching across many buffers - that's a legitimate use case. Think about AG/RG and other multi-threaded text searchers. If the buffers are also structured and need to be parsed, we arrive at the situation when parallelism can be quite useful. > ... Correct me if I'm wrong, AFAIU programs that analyze text as it=20 > is being edited will usually need to re-parse after every edit anyways,=20 > making a parse during the edit itself not all that useful: After all,=20 > the result is no longer accurate for the buffer contents by the time it=20 > becomes available. A synchronous parser can probably do just as good of=20 > a job in such a scenario, as long as it is interruptible by user input. No. It still makes sense to re-parse incrementally before the point where the edits are being made. For example, Org parser has to trash everything that may theoretically be affected by current edits and re-parse later, synchronously, on demand. This trashing sometimes involve rather big chunks of text, and it would significantly speed things up if we could do part of the incremental parsing in parallel. > I'm not familiar with the C core at all, do you think it might be more=20 > realistic to add a little more facilities for asynchronous data streams=20 > than to rework the execution model of Emacs to add multi threading? I'm=20 > mainly thinking of something like "channels" or "queues" for lisp=20 > objects, with an easy to use scheduling mechanism that makes it=20 > straightforward for people to not stall the main thread for too long. AFAIK, sentinels do this already. Of course, they are quite basic. As for queues, see https://emacsconf.org/2022/talks/async/ > And another (very) nice to have feature: asynchronous filesystem IO.=20 > Consider the code below which I use in a package I'm working on. My=20 > package reads and parses a large amount of files in the background=20 > within a short time frame. Parsing/processing the files in a timely=20 > manner is never really an issue, but the blocking IO of=20 > `insert-file-contents' does often take so long that it is=C2=A0 impossibl= e to=20 > not have the user notice, even if polling `input-pending-p' and yielding= =20 > in between operations. Well. I have an opposite problem when `read' takes second to read few Mb of Elisp data, while inserting it into a temporary buffer is instant. Also, did you actually profile your use case? I actually doubt that reading file takes a long time on modern systems. In my previous tests, I tried to open multiple tens of thousands files using (let ((buffer (get-buffer-create " *Processing*"))) (with-current-buffer buffer (let (buffer-undo-list t) (dolist (f files) (insert-file-contents f nil nil nil 'replace) (do-staff))))) And it was very, very fast. What was not fast (orders of magnitude) is opening files directly that triggers all kinds of user hooks. --=20 Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at . Support Org development at , or support my work at