From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Ihor Radchenko Newsgroups: gmane.emacs.devel Subject: Async IO and queing process sentinels (was: Concurrency via isolated process/thread) Date: Tue, 18 Jul 2023 12:39:08 +0000 Message-ID: <87fs5l2y37.fsf@localhost> References: <871qhnr4ty.fsf@localhost> <87pm57p57m.fsf@yahoo.com> <83mt0bjdh5.fsf@gnu.org> <03724893-5b8e-2df5-6001-f2616c8f5c1c@hugot.nl> <87r0p53i58.fsf@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="22769"; mail-complaints-to="usenet@ciao.gmane.io" Cc: emacs-devel@gnu.org, Michael Albinus To: Hugo Thunnissen Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Tue Jul 18 14:39:55 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qLjze-0005fX-FA for ged-emacs-devel@m.gmane-mx.org; Tue, 18 Jul 2023 14:39:54 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qLjys-0003RY-Jk; Tue, 18 Jul 2023 08:39:06 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qLjyq-0003Qc-KZ for emacs-devel@gnu.org; Tue, 18 Jul 2023 08:39:04 -0400 Original-Received: from mout02.posteo.de ([185.67.36.66]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qLjyn-0003Q6-90 for emacs-devel@gnu.org; Tue, 18 Jul 2023 08:39:04 -0400 Original-Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 3544C240104 for ; Tue, 18 Jul 2023 14:38:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1689683939; bh=NruuCNku+IcDzlX76MRC7J298mbLmLvjBheKS65JxyM=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version: Content-Transfer-Encoding:From; b=cZe2wdi+RhBljh4fbjQYFdfHm8cQUNzIlr5WG0F24Cuy2hLi187ya+K6/DiDIpAw/ syI/ZAfmEEXNv5VR2lYmqx2v8N3n1RNbDWNrmXjT/RK+l4XO2wZHb9JKTUGA6YfVRn 1LlfH1d+fjt9xPt4DcUoGSB2qRYU5AnCrepiy6IY2tFvQFEd5jdgs7kjJebx19ejOB FiOutbo5ABSXpRR0k8l0i1s73UX6sbnIm9tYeGNj81ha+nBOvZSGQFNrL8vpX6TLeQ eGCmAMpd+8gctuTEAEkXVa8DB3DBRpevC4vve/QeKZETpKXs8hx/3s0W4Fuy8FTSqJ WrwvVDVtNH+wQ== Original-Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4R4z5t2tJQz6tvb; Tue, 18 Jul 2023 14:38:58 +0200 (CEST) In-Reply-To: Received-SPF: pass client-ip=185.67.36.66; envelope-from=yantar92@posteo.net; helo=mout02.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:307953 Archived-At: Hugo Thunnissen writes: >>> I'm not familiar with the C core at all, do you think it might be more >>> realistic to add a little more facilities for asynchronous data streams >>> than to rework the execution model of Emacs to add multi threading? I'm >>> mainly thinking of something like "channels" or "queues" for lisp >>> objects, with an easy to use scheduling mechanism that makes it >>> straightforward for people to not stall the main thread for too long. >> AFAIK, sentinels do this already. Of course, they are quite basic. >> As for queues, see https://emacsconf.org/2022/talks/async/ > > Sentinels are for external processes. I was thinking more in the vain of= =20 > a queue that is owned by two lisp threads, where one thread can write=20 > and prepare input for the other, and the other thread yields as long as=20 > there is no new input to process. When a thread receives an input=20 > message from the other, it will come alive at the next idling moment and= =20 > process the queue while yielding in between messages. This would make it= =20 > easier to process data in chunks without halting the main thread long=20 > enough to bother the user, as long as the main thread is prioritized=20 > over others. What you describe sounds very similar to `condition-wait'/`condition-notify= '. That said, I have seen scenarios when I have several dozens of network connections open, receive data from server, and fire all their sentinels near the same time. This effectively blocks Emacs because all the sentinels are suddenly scheduled to run together. It would be nice to have some better balancing in such case. > I still suspect that async filesystem IO could make a considerable=20 > difference in some scenarios. My package is for PHP projects and it is=20 > not unheard of for PHP files to be edited over network mounts where IO=20 > could have a lot more latency. For example, some people have asked me=20 > whether my package works over tramp. I haven't profiled this scenario=20 > yet as I have to make some tweaks first to make it work at all, but I=20 > fear that blocking on `insert-file-contents' may not be ideal in that=20 > scenario. AFAIR, Michael Albinus (the maintainer of TRAMP) is working on better support for async processes in TRAMP. And he had troubles with managing async process queues as well. CCing him, as this discussion might be relevant. > And then there also is `directory-files-recursively': > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 457=C2=A0 77% - phpinsp= ect-index-current-project > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 120=C2=A0 20% -direc= tory-files-recursively > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 79=C2=A0 13% = -directory-files-recursively > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 21=C2=A0=C2=A0 3%= -directory-files-recursively > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 11 1% = directory-files-recursively Note how functions involving actual IO do not show up in the backtrace: `file-name-all-completions' and `file-symlink-p' are not there. Most of the 120ms spend inside `directory-files-recursively' is Elisp recursive calls. I also did testing locally, on my $HOME dir, and got similar results with recursive calls taking most of the CPU time: 463 29% - directory-files-recursively 434 27% - directory-files-recursively 305 19% - directory-files-recursively 111 7% - directory-files-recursively 85 5% - directory-files-recursively 62 3% - directory-files-recursively 24 1% - directory-files-recursively 10 0% - directory-files-recursively 10 0% - directory-files-recursively 7 0% - directory-files-recursively 7 0% directory-files-recursively 3 0% sort 3 0% sort 3 0% sort If you want to improve performance here, you likely need to rewrite `directory-files-recursively' without recursion. Doable, and nothing to do with IO slowness.=20 --=20 Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at . Support Org development at , or support my work at