From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Ken Raeburn Newsgroups: gmane.emacs.devel Subject: Re: "concurrency" branch updated Date: Wed, 4 Nov 2015 14:48:12 -0500 Message-ID: <11F7A3CD-5203-42C9-93EF-842A1D4F9EEB@raeburn.org> References: <1B30AC54-4A83-4437-8BA8-B80F4ED6AF1A@raeburn.org> <831tc7vyex.fsf@gnu.org> <83vb9hu612.fsf@gnu.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Trace: ger.gmane.org 1446666512 2982 80.91.229.3 (4 Nov 2015 19:48:32 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Wed, 4 Nov 2015 19:48:32 +0000 (UTC) Cc: johnw@newartisans.com, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Wed Nov 04 20:48:24 2015 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1Zu42i-0005x1-Ax for ged-emacs-devel@m.gmane.org; Wed, 04 Nov 2015 20:48:24 +0100 Original-Received: from localhost ([::1]:56868 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zu42h-0000qW-GI for ged-emacs-devel@m.gmane.org; Wed, 04 Nov 2015 14:48:23 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:52634) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zu42d-0000qM-JJ for emacs-devel@gnu.org; Wed, 04 Nov 2015 14:48:21 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zu42a-0005AY-B0 for emacs-devel@gnu.org; Wed, 04 Nov 2015 14:48:19 -0500 Original-Received: from mail-yk0-x234.google.com ([2607:f8b0:4002:c07::234]:34400) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zu42a-00059l-3c for emacs-devel@gnu.org; Wed, 04 Nov 2015 14:48:16 -0500 Original-Received: by ykdr3 with SMTP id r3so91650338ykd.1 for ; Wed, 04 Nov 2015 11:48:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raeburn_org.20150623.gappssmtp.com; s=20150623; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=m7dQDpezqcYwhIcPf94QSaLoFc9KakqeCYOLY/hl7yA=; b=NM8psjavBNkBLfFE6cN3DS0cSw++N4/0ySwb80K+96rOxu8q6FJw6jr3xmYBN8EzDH lIACWL+6thfwSEG0fCXMp8CgHX8hHUitlHwuiR1HPuQ4qCsHympviKSW9dPF98qcCesA iVdrMsF0nFAjY57wfU4SCCpvdwRNH3Zm1UMQJBedzOKTmN8qdSEGBfY2hL9ZRWfGGwRS 8H0pKPFW++8BzWVKb99lYsYTlT9nT2OxQ+vNFv0FHAUzy4vC1xbvtSqsygHojNR7BOsO NlsG8iwsbYwn6MDYfT8OPYztR+HMkG7lM9wo/dKjO+/ZdD4VU4hw6s/pKFmtHzPSOVTW 8Ieg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=m7dQDpezqcYwhIcPf94QSaLoFc9KakqeCYOLY/hl7yA=; b=QLUeEEf4Hq3CQtHGMLw/8yakVwwvaXlLQasdCparV18KfZXMyL7nFBihFriX1+YqtK vWN7wlpeGhfj6TTiQ0lpAl1CFH6aYsmqAID7mqbhiHZDM90PXnhCYCqS2tyRiUYWxeL4 1TcOZXTfwHWqdiY7SO6evOWrpYq4oXyJMp/ypQva2PUIbKF8fZ7MlADhbbS29ylcTuBB PjbhkM+Q66OB7JFRm9aQTmZZ2VRHZDdiflFn2Ou7Gawi4tzejXfW6nLzB4r/u2FQhWNA vjv2sderSXibmLLkHjb0at/qnsP16e1u7wAdW0f7yq8YAw5wsMXq2tIIEQp67UyaKehv i+Bw== X-Gm-Message-State: ALoCoQln/CnuzGnzJtoPHnEOiZoAnNJIEJm9R2tMAZW70FA2YlHXCeTR+n5VDaH311D5zsc8Hlbq X-Received: by 10.31.48.73 with SMTP id w70mr3663817vkw.138.1446666495206; Wed, 04 Nov 2015 11:48:15 -0800 (PST) Original-Received: from [192.168.17.111] (c-66-31-203-101.hsd1.ma.comcast.net. [66.31.203.101]) by smtp.gmail.com with ESMTPSA id g16sm1926885vke.11.2015.11.04.11.48.14 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 04 Nov 2015 11:48:14 -0800 (PST) In-Reply-To: <83vb9hu612.fsf@gnu.org> X-Mailer: Apple Mail (2.2104) X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2607:f8b0:4002:c07::234 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:193233 Archived-At: >>=20 >> Implementing a generator with a thread seems somewhat = straightforward, needing=20 >> some sort of simple communication channel between the main thread and = the=20 >> generator thread to pass =E2=80=9Cneed next value=E2=80=9D and = =E2=80=9Chere=E2=80=99s the next value=E2=80=9D messages=20 >> back and forth; some extra work would be needed so that dropping all = references=20 >> to a generator makes everything, including the thread, go away. = Raising an=20 >> error in the thread=E2=80=99s =E2=80=9Cyield=E2=80=9D calls may be a = way to tackle that, though it=20 >> changes the semantics within the generator a bit. >=20 > Both the generator and its consumer run Lisp, so they can only run in > sequence. How is this different from running them both in a single > thread? In this case, it=E2=80=99s about how you'd write the generator code. = While the multithreaded version would have other issues (like having to = properly quit the new thread when we=E2=80=99re done with the = generator), it wouldn=E2=80=99t require writing everything using special = macros to do CPS transformations. If I want to yield values from within = a function invoked via mapcar, I don=E2=80=99t have to write an = iter-mapcar macro to turn everything inside-out under the covers. >=20 >> For prefetching file contents or searching existing buffers, the = =E2=80=9Cmain=E2=80=9D thread=20 >> can release the global lock when it prompts for the user=E2=80=99s = input, and a=20 >> background thread can create buffers and load files, or search = buffers for=20 >> patterns, tossing results onto some sort of queue or other data = structure for=20 >> consumption by the main thread when it finishes with the file it=E2=80=99= s on. >=20 > But then you are not talking about "normal" visiting of files or > searching of buffers. You are talking about specialized features that > visit large number of files or are capable of somehow marking lots of > search hits for future presentation to users. That is a far cry from > how we do this stuff currently -- you ask the user first, _then_ you > search or visit the file she asked for. I haven=E2=80=99t used tags-query-replace in a while, but I don=E2=80=99t = recall it asking me if I wanted to visit each file. But yes, I=E2=80=99m = thinking of larger operations where the next stage is fairly = predictable, and probably does no harm if we optimistically start it = early. Smaller stuff may be good too (I hope), but I=E2=80=99d guess = there=E2=80=99s a greater chance the thread-switching overhead could = become an issue; I could well be overestimating it. And some of the = simpler ones, like highlighting all regexp matches in the visible part = of the current buffer while doing a search, are already done, though we = could look at how the code would compare if rewritten to use threads. > You are talking about some significant refactoring here, we currently > o all of this on the fly. In any case, I can understand how this > would be a win with remote files, but with local files I'm quite sure > most of the time for inserting a file is taken by stuff like decoding > its contents, which we also do on the fly and which can call Lisp. > The I/O itself is quite fast nowadays, I think. Just compare > insert-file-contents with insert-file-contents-literally for the same > large file, and see the big difference, especially if it includes some > non-ASCII text. I haven=E2=80=99t done that test, but I have used an NFS server that got = slow at times. And NFS from Amazon virtual machines back to my office, = which is always a bit slow. And sshfs, which can be slow too. None of = which Emacs can do anything about directly. >> So=E2=80=A6 yeah, I think some of them are possible, but I=E2=80=99m = not sure any of them would=20 >> be a particularly good way to show off. Got any suggestions? >=20 > I think features that use timers, and idle timers in particular, are > natural candidates for using threads. Stealth font-lock comes to > mind, for example. That=E2=80=99s what I was thinking of when I mentioned fontification. I = hope thread switches are fast enough. >> I=E2=80=99m not sure. I=E2=80=99m not picturing redisplay running = concurrently with Lisp so=20 >> much as redisplay on display 1 running concurrently with redisplay on = display=20 >> 2, all happening at the same point in the code where we now run = redisplay. =20 >=20 > Why is this use case important? Do we really believe someone might > look at 2 different X displays at the same time? No, but occasionally redisplay still needs to talk to multiple displays = and get responses back, even with the work you and Stefan have done. = Fortunately, it=E2=80=99s much more rare now, and the color-handling = work I did may help further. And people using multiple displays *and* = slow display connections like me are probably not very common among the = user base. So it=E2=80=99s an area where threads might help, but maybe = not a terribly important one for the project. >> I am making some assumptions that redisplay isn=E2=80=99t doing many = costly >> calculations compared to the cost of pushing the bits to the glass. >=20 > That's not really true, although the display engine tries very hard to > be very fast. But I've seen cycles taking 10 msec and even 50 msec > (which already borders on crossing the annoyance threshold). So there > are some pretty costly calculations during redisplay, which is why the > display engine is heavily optimized to avoid them as much as possible. In that case, maybe it=E2=80=99s still worth considering after all. >=20 >> I suspect TLS is probably the more interesting case. >=20 > What do we have in TLS that we don't have in any network connection? Encryption, optional compression, possibly key renegotiation, possible = receipt of incomplete messages that can=E2=80=99t yet be decrypted and = thus can=E2=80=99t give us any new data bytes. The thread(s) running = user Lisp code needn=E2=80=99t spend any cycles on these things. Ken=