From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: "concurrency" branch updated Date: Wed, 04 Nov 2015 17:40:41 +0200 Message-ID: <83vb9hu612.fsf@gnu.org> References: <1B30AC54-4A83-4437-8BA8-B80F4ED6AF1A@raeburn.org> <831tc7vyex.fsf@gnu.org> Reply-To: Eli Zaretskii NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE X-Trace: ger.gmane.org 1446651688 30141 80.91.229.3 (4 Nov 2015 15:41:28 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Wed, 4 Nov 2015 15:41:28 +0000 (UTC) Cc: johnw@newartisans.com, emacs-devel@gnu.org To: Ken Raeburn Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Wed Nov 04 16:41:22 2015 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1Zu0Bd-0004vH-3c for ged-emacs-devel@m.gmane.org; Wed, 04 Nov 2015 16:41:21 +0100 Original-Received: from localhost ([::1]:55441 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zu0Bc-0007l4-DH for ged-emacs-devel@m.gmane.org; Wed, 04 Nov 2015 10:41:20 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:57169) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zu0BE-0007kk-Kj for emacs-devel@gnu.org; Wed, 04 Nov 2015 10:41:02 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zu0BB-00005E-BI for emacs-devel@gnu.org; Wed, 04 Nov 2015 10:40:56 -0500 Original-Received: from mtaout27.012.net.il ([80.179.55.183]:48649) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zu0BA-000058-Tq for emacs-devel@gnu.org; Wed, 04 Nov 2015 10:40:53 -0500 Original-Received: from conversion-daemon.mtaout27.012.net.il by mtaout27.012.net.il (HyperSendmail v2007.08) id <0NXA00800R8Y3G00@mtaout27.012.net.il> for emacs-devel@gnu.org; Wed, 04 Nov 2015 17:36:02 +0200 (IST) Original-Received: from HOME-C4E4A596F7 ([84.94.185.246]) by mtaout27.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0NXA004C4RC25B20@mtaout27.012.net.il>; Wed, 04 Nov 2015 17:36:02 +0200 (IST) In-reply-to: X-012-Sender: halo1@inter.net.il X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 80.179.55.183 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:193205 Archived-At: > From: Ken Raeburn > Date: Wed, 4 Nov 2015 04:20:37 -0500 > Cc: johnw@newartisans.com, > emacs-devel@gnu.org >=20 > > On Nov 3, 2015, at 11:29, Eli Zaretskii wrote: > >=20 > >> From: Ken Raeburn > >> Date: Tue, 3 Nov 2015 04:40:25 -0500 > >> Cc: "address@hidden discussions" > >>=20 > >> At some point, we=E2=80=99ll want to demonstrate practical utili= ty; not a trivial=20 > >> demo program that displays a few messages, and nothing on the sc= ale of=20 > >> rewriting all of Gnus to be multithreaded, but somewhere in betw= een. I=E2=80=99m=20 > >> not sure what would be a good example. A version of generator.e= l that uses=20 > >> threads instead of the CPS transformation of everything is a pos= sibility,=20 > >> and it would probably simplify the writing and compiling of the = generators,=20 > >> but it=E2=80=99d probably be more heavy-weight at run time. Pre= fetching files=E2=80=99=20 > >> contents, or searching already-loaded files, while tags-query-re= place waits=20 > >> for the user to respond to a prompt? Improving fontification so= mehow? > >=20 > > Given that only one thread can run Lisp, is the above even possib= le? >=20 > Implementing a generator with a thread seems somewhat straightforwa= rd, needing=20 > some sort of simple communication channel between the main thread a= nd the=20 > generator thread to pass =E2=80=9Cneed next value=E2=80=9D and = =E2=80=9Chere=E2=80=99s the next value=E2=80=9D messages=20 > back and forth; some extra work would be needed so that dropping al= l references=20 > to a generator makes everything, including the thread, go away. Ra= ising an=20 > error in the thread=E2=80=99s =E2=80=9Cyield=E2=80=9D calls may be = a way to tackle that, though it=20 > changes the semantics within the generator a bit. Both the generator and its consumer run Lisp, so they can only run in sequence. How is this different from running them both in a single thread? > For prefetching file contents or searching existing buffers, the = =E2=80=9Cmain=E2=80=9D thread=20 > can release the global lock when it prompts for the user=E2=80=99s = input, and a=20 > background thread can create buffers and load files, or search buff= ers for=20 > patterns, tossing results onto some sort of queue or other data str= ucture for=20 > consumption by the main thread when it finishes with the file it= =E2=80=99s on. But then you are not talking about "normal" visiting of files or searching of buffers. You are talking about specialized features tha= t visit large number of files or are capable of somehow marking lots of search hits for future presentation to users. That is a far cry from how we do this stuff currently -- you ask the user first, _then_ you search or visit the file she asked for. > refactoring insert-file-contents into a minimal file-reading routin= e > that does no Lisp callbacks and another to deal with file name > handlers and hooks and such could let us do the former on a helper > thread and the latter (which could prompt the user) in the main > thread at the expected time. You are talking about some significant refactoring here, we currently o all of this on the fly. In any case, I can understand how this would be a win with remote files, but with local files I'm quite sure most of the time for inserting a file is taken by stuff like decoding its contents, which we also do on the fly and which can call Lisp. The I/O itself is quite fast nowadays, I think. Just compare insert-file-contents with insert-file-contents-literally for the same large file, and see the big difference, especially if it includes som= e non-ASCII text. > Both of those examples are mainly about running some extra work in = the moments=20 > while we=E2=80=99re waiting for the user to respond to a prompt. W= e may be able to do=20 > the same with idle timers or other such mechanisms. In cases like = that, I=20 > think it may come down to whether it=E2=80=99s easier and/or more m= aintainable to write=20 > code that cranks through the next step of an explicitly managed sta= te machine,=20 > or structured code that maintains its state in program counters and= variables=20 > local to each stack frame=E2=80=A6 sometimes it=E2=80=99s one, some= times it=E2=80=99s the other. >=20 > As to fontification=E2=80=A6 I expect the code is pretty tight now,= but maybe someone=20 > who knows that code has some insight into whether we could do it be= tter with=20 > more CPU cores available. >=20 > So=E2=80=A6 yeah, I think some of them are possible, but I=E2=80= =99m not sure any of them would=20 > be a particularly good way to show off. Got any suggestions? I think features that use timers, and idle timers in particular, are natural candidates for using threads. Stealth font-lock comes to mind, for example. > >> Understood. I think there may also be places where we could use= threads=20 > >> less visible to the Lisp world; TLS and redisplay come to mind. > >=20 > > Given the general model-view-controller design of Emacs and the > > structure of its main loop, is making redisplay run in a separate > > thread really viable? >=20 > I=E2=80=99m not sure. I=E2=80=99m not picturing redisplay running = concurrently with Lisp so=20 > much as redisplay on display 1 running concurrently with redisplay = on display=20 > 2, all happening at the same point in the code where we now run red= isplay. =20 Why is this use case important? Do we really believe someone might look at 2 different X displays at the same time? Perhaps you meant frames, not displays. This could make a lot of sense, except that: > (Ignoring for the moment the bits where redisplay can trigger Lisp = evaluation.) We cannot really ignore that, because this feature is used a lot. However, the global lock will probably solve this. A more problematic issue is that the display engine currently assumes that (almost) nothing happens with buffers and strings while it does its job. If redisplay is multithreaded, we need to make sure no othe= r thread that can touch Lisp data could run. > I am making some assumptions that redisplay isn=E2=80=99t doing man= y costly > calculations compared to the cost of pushing the bits to the glass. That's not really true, although the display engine tries very hard t= o be very fast. But I've seen cycles taking 10 msec and even 50 msec (which already borders on crossing the annoyance threshold). So ther= e are some pretty costly calculations during redisplay, which is why th= e display engine is heavily optimized to avoid them as much as possible= . > I suspect TLS is probably the more interesting case. What do we have in TLS that we don't have in any network connection?