From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ihor Radchenko Subject: Re: Asynchronous org-agenda-redo Date: Fri, 13 Dec 2019 17:39:10 +0800 Message-ID: <87immk8szl.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> References: <87k172ot2m.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> <87pngtsppt.fsf@alphapapa.net> <87o8wda6nv.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> <87h824sos5.fsf@alphapapa.net> <87wob0fwsg.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:54688) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ifhRm-0006cJ-Qm for emacs-orgmode@gnu.org; Fri, 13 Dec 2019 04:41:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ifhRl-0002GV-03 for emacs-orgmode@gnu.org; Fri, 13 Dec 2019 04:41:18 -0500 Received: from mail-wr1-x435.google.com ([2a00:1450:4864:20::435]:39930) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ifhRk-0002FH-OL for emacs-orgmode@gnu.org; Fri, 13 Dec 2019 04:41:16 -0500 Received: by mail-wr1-x435.google.com with SMTP id y11so5898680wrt.6 for ; Fri, 13 Dec 2019 01:41:16 -0800 (PST) In-Reply-To: <87wob0fwsg.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: "Emacs-orgmode" To: Adam Porter , emacs-orgmode@gnu.org >> Asynchronous code is not faster; it's generally slower because of >> yielding and synchronization. > Anyway, I will try to throw yields into agenda code just to check how > bad the performance can degrade. With the following code, org-agenda-redo runs for 21 second on my system, while without threads it is 16 seconds. However, emacs remains responsive during rebuilding agenda! (define-advice org-agenda-redo (:around (oldfun &optional all) make-async) (make-thread (lambda () (funcall oldfun all)) "org-agenda-redo")) (define-advice org-agenda-skip-eval (:around (oldfun form) make-async) (thread-join (make-thread (lambda () (funcall oldfun form)) "org-agenda-s= kip-eval"))) The problem, of course, is that touching agenda buffer and org buffers may be risky while org-agenda-redo is running. Wondering if it is possible to block user commands during that time.=20 Best, Ihor Ihor Radchenko writes: >> Org Agenda code does not wait for keyboard input; it's busy building the >> agenda. This is the case with most code in Emacs: it's not written to >> be asynchronous, and it doesn't return to the main thread until done. >> So you can sprinkle yields here and there and maybe be able to move >> point around while some code is running, but that will decrease >> performance, as well as introducing another level of complexity and >> another class of bugs (e.g. what if the user modifies a buffer while the >> agenda code is scanning it?). > > Thanks for the explanation. > >> AFAIK there exists no way to do such a thing. Buffers are not designed >> to be serialized/deserialized like that. You could try writing some >> Elisp code to do it, but the end result would probably be much slower >> than existing agenda code, as well as more difficult to debug. > > Yeah. Even re-initialisation of, for example, overlays in org buffer is=20 > likely to take too much time. > >> As you can see in org-agenda.el, it's complicated. Remember that an >> Emacs process is like a Lisp image, full of state. The more symbols and >> other structures you copy to the async Emacs process (by printing and >> reading them as text, remember), the slower it's going to be--and it >> will always be slower than not using async. > >> Asynchronous code is not faster; it's generally slower because of >> yielding and synchronization. > > I see now that generating agenda in separate process will cause too much > overheads. > Anyway, I will try to throw yields into agenda code just to check how > bad the performance can degrade. > >> org-ql doesn't use skip functions, just queries. > > Skip functions are essentially used-defined queries as soon as the > queries are tested against every headline. > I can rewrite my skip functions into queries, but I don't expect much > improvement since org-ql seems to use org-entry-get, which is the main > performance bottleneck for my agenda generation. > > Best, > Ihor > > adam Porter writes: > >> Ihor Radchenko writes: >> >>>> Be sure to read the Emacs Lisp manual regarding threads. They are >>>> cooperative, so functions called as threads must yield back to the main >>>> thread for Emacs to do anything else before the function returns. >>> >>> I tried to read the manual, but I clearly misunderstand something. >>> The manual says: >>> >>>> Currently, thread switching will occur upon explicit request via >>>> =E2=80=98thread-yield=E2=80=99, when waiting for keyboard input...=20 >>> >>> So, except directly calling thread-yield, it should be possible to >>> trigger switching the current thread when keyboard input is expected. >>> I tried the following demo code: >>> >>> (defun test () >>> (let ((a 0)) >>> (dotimes (_ 5) >>> (setq a (1+ a)) >>> (sleep-for 2) >>> (message "%s" a)))) >>> >>> (progn ;This should return to command loop quickly >>> (make-thread #'test) >>> (message "Executed...")); `eval-last-sexp' here >>> >>> I can move around the buffer while the progn is running. >>> However, it is not the case with `org-agenda-redo' for a reason I do not >>> fully understand. >> >> Org Agenda code does not wait for keyboard input; it's busy building the >> agenda. This is the case with most code in Emacs: it's not written to >> be asynchronous, and it doesn't return to the main thread until done. >> So you can sprinkle yields here and there and maybe be able to move >> point around while some code is running, but that will decrease >> performance, as well as introducing another level of complexity and >> another class of bugs (e.g. what if the user modifies a buffer while the >> agenda code is scanning it?). >> >>>> 1. The process would have to load the same Org buffers, which takes >>>> time, especially in large buffers. Depending on configuration, it >>>> can take some time, indeed. >>> >>>> 3. Ensuring that configuration and state between the main Emacs proce= ss >>>> and the separate, agenda-generating process is not necessarily >>>> simple. Consider as well that if a buffer had unsaved changes, >>>> those would not be readable by the other process, which would lead >>>> to invalid results. One could force the buffers to be saved first, >>>> but that may not always be desirable, as saving buffers can have >>>> side effects. >>> >>> Why cannot org-buffer simply be copied into the subordinate process? If >>> all be buffer-locals, text properties, and overlays are copied directly >>> from the main emacs process, there may be no need to even initialise >>> org-mode (the idea is to do something similar to clone-buffer). >> >> AFAIK there exists no way to do such a thing. Buffers are not designed >> to be serialized/deserialized like that. You could try writing some >> Elisp code to do it, but the end result would probably be much slower >> than existing agenda code, as well as more difficult to debug. >> >>> The question though is whether buffer-locals + overlays + propertized >>> .org files text + org-agenda-buffer copy can be sufficient to make the >>> org-agenda-redo run properly. Are there any other buffers, variables, >>> or other environment settings used by org-agenda-redo? >> >> As you can see in org-agenda.el, it's complicated. Remember that an >> Emacs process is like a Lisp image, full of state. The more symbols and >> other structures you copy to the async Emacs process (by printing and >> reading them as text, remember), the slower it's going to be--and it >> will always be slower than not using async. >> >>>> If your agenda buffers are taking too long to refresh, you might >>>> consider org-ql's views/saved-searches as an alternative. ... >>> >>> I know org-ql and I am pretty sure that it will improve performance. >>> Actually, if one can make built-in org-agenda asynchronous, org-ql can >>> probably use similar approach and become even faster :) >> >> Asynchronous code is not faster; it's generally slower because of >> yielding and synchronization. >> >>> I am trying on default org-agenda now mostly because my current config >>> is heavily geared towards default agenda and I am not sure if >>> refactoring everything to use org-ql will worth it at the end in terms >>> of performance. I use too many slow custom skip-functions. >> >> org-ql doesn't use skip functions, just queries. >> >> > --=20 Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong U= niversity, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg