Asynchronous org-agenda-redo

Asynchronous org-agenda-redo

[Ihor Radchenko]

I am thinking if it is possible to implement org-agenda-redo
asynchronously.

Rebuilding agenda should normally not affect any buffer except agenda
buffer. So, it should be sufficient to block any agenda modifying
commands in the agenda buffer, redo the agenda buffer in separate
thread, and replace the old agenda with the calculated one.
Then, emacs should remain responsive while updating agenda (except for
modifying the agenda buffer).

For example, this naive code kind of works (forgetting that buffer-local
variables will not be passed to the thread):

(define-advice org-agenda-redo (:around (oldfun &optional all) make-async)
  (make-thread oldfun "org-agenda-redo"))

The problem is that emacs does not become responsive...

Another approach would be using async.el package, which allows calling
arbitrary function in subordinate emacs process. Then, the main emacs
instance should not be "frozen" (I use same approach for tangling and it
works fine).  

However, the question is how to pass the .org and agenda buffers to this
subordinate process. Opening the .org files there is not a good option
since it would give too much overhead to this asynchronous agenda.

Any suggestions? Alternative ideas?

Best,
Ihor

Re: Asynchronous org-agenda-redo

[Adam Porter]

Be sure to read the Emacs Lisp manual regarding threads.  They are
cooperative, so functions called as threads must yield back to the main
thread for Emacs to do anything else before the function returns.

If you're feeling adventurous, you could experiment with adding yields
in relevant agenda functions.  But that wouldn't be suitable for merging
into Org, because that yielding also decreases performance generally.

As long as Elisp threads are cooperative, they are of very limited use.

Generating agendas with async.el in a separate Emacs process is an
interesting idea, but probably generally impractical for a few reasons:

1.  The process would have to load the same Org buffers, which takes
    time, especially in large buffers.  Depending on configuration, it
    can take some time, indeed.
2.  The process would also have to load the same packages (or, at least,
    all the necessary ones, which depends on configuration), which takes
    time.
3.  Ensuring that configuration and state between the main Emacs process
    and the separate, agenda-generating process is not necessarily
    simple.  Consider as well that if a buffer had unsaved changes,
    those would not be readable by the other process, which would lead
    to invalid results.  One could force the buffers to be saved first,
    but that may not always be desirable, as saving buffers can have
    side effects.

If your agenda buffers are taking too long to refresh, you might
consider org-ql's views/saved-searches as an alternative.  The built-in
caching in org-ql significantly improves performance, especially when
refreshing views.

Re: Asynchronous org-agenda-redo

[Diego Zamboni]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

Hi Ihor,

I cannot answer your question, but I am curious about using async together
with tangling, since for some of my buffers, tangling takes some time and
freezes Emacs in the process. Do you have some examples of this that you
could share?

Thanks,
--Diego


On Thu, Dec 12, 2019 at 9:21 AM Ihor Radchenko <yantar92@gmail.com> wrote:

> I am thinking if it is possible to implement org-agenda-redo
> asynchronously.
>
> Rebuilding agenda should normally not affect any buffer except agenda
> buffer. So, it should be sufficient to block any agenda modifying
> commands in the agenda buffer, redo the agenda buffer in separate
> thread, and replace the old agenda with the calculated one.
> Then, emacs should remain responsive while updating agenda (except for
> modifying the agenda buffer).
>
> For example, this naive code kind of works (forgetting that buffer-local
> variables will not be passed to the thread):
>
> (define-advice org-agenda-redo (:around (oldfun &optional all) make-async)
>   (make-thread oldfun "org-agenda-redo"))
>
> The problem is that emacs does not become responsive...
>
> Another approach would be using async.el package, which allows calling
> arbitrary function in subordinate emacs process. Then, the main emacs
> instance should not be "frozen" (I use same approach for tangling and it
> works fine).
>
> However, the question is how to pass the .org and agenda buffers to this
> subordinate process. Opening the .org files there is not a good option
> since it would give too much overhead to this asynchronous agenda.
>
> Any suggestions? Alternative ideas?
>
> Best,
> Ihor
>
>
>

[-- Attachment #2: Type: text/html, Size: 2109 bytes --]

Re: Asynchronous org-agenda-redo

[Ihor Radchenko]

Hi Diego,
> I cannot answer your question, but I am curious about using async together
> with tangling, since for some of my buffers, tangling takes some time and
> freezes Emacs in the process. Do you have some examples of this that you
> could share?

See the relevant code from my config below. Let me know if you have any
questions or suggestions.

Also, my config is written in the way that everything related to user
interaction can be disabled if I load emacs for tangling
(with org-tangle-flag). This is to speed up the async process.  

#+begin_src emacs-lisp
(defvar yant/org-babel-tangle-async-process-list nil
  "Plist of (file . process) for all the currently running async tangle processes.")


(defun yant/org-babel-tangle-async (file &optional target-file lang)
  "Invoke `org-babel-tangle-file' asynchronously."
  (require 'async)
  (let ((oldproc (plist-get yant/org-babel-tangle-async-process-list file)))
    (when (or (not oldproc)
	      (async-wait oldproc))
      (message "Tangling %s..." (buffer-file-name))
      (plist-put yant/org-babel-tangle-async-process-list
		 file
		 (async-start
		  (let ((args (list file target-file lang)))
		    `(lambda ()
		       (require 'org)
		       (setq org-tangle-flag t)
		       (load "~/.emacs.d/config.el")
		       (apply #'org-babel-tangle-file ',args)))
		  (let ((message-string (format "Tangling (%S %S %S) completed." file target-file lang)))
		    `(lambda (result) (message ,message-string))))))))

(defvar yant/auto-tangle-list nil
  "List of files, which can be safely tangled on save.
The list is saved between Emacs sessions.")

(when init-flag
  (use-package savehist
    :config
    (add-to-list 'savehist-additional-variables 'yant/auto-tangle-list))
  (savehist-mode +1)
  (defun yant/toggle-buffer-auto-tangle (arg)
    "Toggle auto tangling of a buffer."
    (interactive "P")
    (if (not (eq major-mode 'org-mode))
	(message "Org-mode is not active in buffer \"%s\"" (buffer-name))
      (cond ((not arg)
	     (if (member (buffer-file-name) yant/auto-tangle-list)
		 (progn (setq yant/auto-tangle-list (delete (buffer-file-name) yant/auto-tangle-list))
			(message "Auto tangling disabled for %s" (buffer-file-name)))
	       (add-to-list 'yant/auto-tangle-list (buffer-file-name))
               (message "Auto tangling enabled for %s" (buffer-file-name))))
            ((or (and (not (listp arg)) (> arg 0))
		 (equal arg '(4)))
             (add-to-list 'yant/auto-tangle-list (buffer-file-name))
             (message "Auto tangling enabled for %s" (buffer-file-name)))
            (t
             (setq yant/auto-tangle-list (delete (buffer-file-name) yant/auto-tangle-list))
	     (message "Auto tangling disabled for %s" (buffer-file-name))))))

  (bind-key "C-c C-*" #'yant/toggle-buffer-auto-tangle org-mode-map))

(defun yant/org-babel-tangle-current-buffer-async ()
  "Tangle current buffer asynchronously."
  (when (and (eq major-mode 'org-mode)
	     (member (buffer-file-name) yant/auto-tangle-list))
    (yant/org-babel-tangle-async (buffer-file-name))))

(add-hook 'after-save-hook #'yant/org-babel-tangle-current-buffer-async)
#+end_src


Diego Zamboni <diego@zzamboni.org> writes:

> Hi Ihor,
>
> I cannot answer your question, but I am curious about using async together
> with tangling, since for some of my buffers, tangling takes some time and
> freezes Emacs in the process. Do you have some examples of this that you
> could share?
>
> Thanks,
> --Diego
>
>
> On Thu, Dec 12, 2019 at 9:21 AM Ihor Radchenko <yantar92@gmail.com> wrote:
>
>> I am thinking if it is possible to implement org-agenda-redo
>> asynchronously.
>>
>> Rebuilding agenda should normally not affect any buffer except agenda
>> buffer. So, it should be sufficient to block any agenda modifying
>> commands in the agenda buffer, redo the agenda buffer in separate
>> thread, and replace the old agenda with the calculated one.
>> Then, emacs should remain responsive while updating agenda (except for
>> modifying the agenda buffer).
>>
>> For example, this naive code kind of works (forgetting that buffer-local
>> variables will not be passed to the thread):
>>
>> (define-advice org-agenda-redo (:around (oldfun &optional all) make-async)
>>   (make-thread oldfun "org-agenda-redo"))
>>
>> The problem is that emacs does not become responsive...
>>
>> Another approach would be using async.el package, which allows calling
>> arbitrary function in subordinate emacs process. Then, the main emacs
>> instance should not be "frozen" (I use same approach for tangling and it
>> works fine).
>>
>> However, the question is how to pass the .org and agenda buffers to this
>> subordinate process. Opening the .org files there is not a good option
>> since it would give too much overhead to this asynchronous agenda.
>>
>> Any suggestions? Alternative ideas?
>>
>> Best,
>> Ihor
>>
>>
>>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: Asynchronous org-agenda-redo

[Ihor Radchenko]

> Be sure to read the Emacs Lisp manual regarding threads.  They are
> cooperative, so functions called as threads must yield back to the main
> thread for Emacs to do anything else before the function returns.

I tried to read the manual, but I clearly misunderstand something.
The manual says:

>   Currently, thread switching will occur upon explicit request via
> ‘thread-yield’, when waiting for keyboard input... 

So, except directly calling thread-yield, it should be possible to
trigger switching the current thread when keyboard input is expected.
I tried the following demo code:

(defun test ()
  (let ((a 0))
    (dotimes (_ 5)
      (setq a (1+ a))
      (sleep-for 2)
      (message "%s" a))))

(progn ;This should return to command loop quickly
  (make-thread #'test)
  (message "Executed...")); `eval-last-sexp' here

I can move around the buffer while the progn is running.
However, it is not the case with `org-agenda-redo' for a reason I do not
fully understand. 

For the async.el, I agree that loading packages may take time, but I
believe that the configuration might be transferred more easily (though
it may depend on the org-agenda and user-defined
org-agenda-skip-functions implementation).

> 1.  The process would have to load the same Org buffers, which takes
>     time, especially in large buffers.  Depending on configuration, it
>     can take some time, indeed.

> 3.  Ensuring that configuration and state between the main Emacs process
>     and the separate, agenda-generating process is not necessarily
>     simple.  Consider as well that if a buffer had unsaved changes,
>     those would not be readable by the other process, which would lead
>     to invalid results.  One could force the buffers to be saved first,
>     but that may not always be desirable, as saving buffers can have
>     side effects.

Why cannot org-buffer simply be copied into the subordinate process? If
all be buffer-locals, text properties, and overlays are copied directly
from the main emacs process, there may be no need to even initialise
org-mode (the idea is to do something similar to clone-buffer). The
question though is whether buffer-locals + overlays + propertized .org
files text + org-agenda-buffer copy can be sufficient to make the
org-agenda-redo run properly. Are there any other buffers, variables, or
other environment settings used by org-agenda-redo?

> If your agenda buffers are taking too long to refresh, you might
> consider org-ql's views/saved-searches as an alternative. ...

I know org-ql and I am pretty sure that it will improve performance.
Actually, if one can make built-in org-agenda asynchronous, org-ql can
probably use similar approach and become even faster :)
I am trying on default org-agenda now mostly because my current config
is heavily geared towards default agenda and I am not sure if
refactoring everything to use org-ql will worth it at the end in terms
of performance. I use too many slow custom skip-functions.

> ... The built-in
> caching in org-ql significantly improves performance, especially when
> refreshing views.

Yeah. I wish org-entry-get and other org-get* functions support caching
as well... Or, at least, org-agenda functions might also support simple
caching based on file modifications.

Best,
Ihor

Adam Porter <adam@alphapapa.net> writes:

> Be sure to read the Emacs Lisp manual regarding threads.  They are
> cooperative, so functions called as threads must yield back to the main
> thread for Emacs to do anything else before the function returns.
>
> If you're feeling adventurous, you could experiment with adding yields
> in relevant agenda functions.  But that wouldn't be suitable for merging
> into Org, because that yielding also decreases performance generally.
>
> As long as Elisp threads are cooperative, they are of very limited use.
>
> Generating agendas with async.el in a separate Emacs process is an
> interesting idea, but probably generally impractical for a few reasons:
>
> 1.  The process would have to load the same Org buffers, which takes
>     time, especially in large buffers.  Depending on configuration, it
>     can take some time, indeed.
> 2.  The process would also have to load the same packages (or, at least,
>     all the necessary ones, which depends on configuration), which takes
>     time.
> 3.  Ensuring that configuration and state between the main Emacs process
>     and the separate, agenda-generating process is not necessarily
>     simple.  Consider as well that if a buffer had unsaved changes,
>     those would not be readable by the other process, which would lead
>     to invalid results.  One could force the buffers to be saved first,
>     but that may not always be desirable, as saving buffers can have
>     side effects.
>
> If your agenda buffers are taking too long to refresh, you might
> consider org-ql's views/saved-searches as an alternative.  The built-in
> caching in org-ql significantly improves performance, especially when
> refreshing views.
>
>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: Asynchronous org-agenda-redo

[Adam Porter]

Ihor Radchenko <yantar92@gmail.com> writes:

>> Be sure to read the Emacs Lisp manual regarding threads.  They are
>> cooperative, so functions called as threads must yield back to the main
>> thread for Emacs to do anything else before the function returns.
>
> I tried to read the manual, but I clearly misunderstand something.
> The manual says:
>
>>   Currently, thread switching will occur upon explicit request via
>> ‘thread-yield’, when waiting for keyboard input... 
>
> So, except directly calling thread-yield, it should be possible to
> trigger switching the current thread when keyboard input is expected.
> I tried the following demo code:
>
> (defun test ()
>   (let ((a 0))
>     (dotimes (_ 5)
>       (setq a (1+ a))
>       (sleep-for 2)
>       (message "%s" a))))
>
> (progn ;This should return to command loop quickly
>   (make-thread #'test)
>   (message "Executed...")); `eval-last-sexp' here
>
> I can move around the buffer while the progn is running.
> However, it is not the case with `org-agenda-redo' for a reason I do not
> fully understand.

Org Agenda code does not wait for keyboard input; it's busy building the
agenda.  This is the case with most code in Emacs: it's not written to
be asynchronous, and it doesn't return to the main thread until done.
So you can sprinkle yields here and there and maybe be able to move
point around while some code is running, but that will decrease
performance, as well as introducing another level of complexity and
another class of bugs (e.g. what if the user modifies a buffer while the
agenda code is scanning it?).

>> 1.  The process would have to load the same Org buffers, which takes
>>     time, especially in large buffers.  Depending on configuration, it
>>     can take some time, indeed.
>
>> 3.  Ensuring that configuration and state between the main Emacs process
>>     and the separate, agenda-generating process is not necessarily
>>     simple.  Consider as well that if a buffer had unsaved changes,
>>     those would not be readable by the other process, which would lead
>>     to invalid results.  One could force the buffers to be saved first,
>>     but that may not always be desirable, as saving buffers can have
>>     side effects.
>
> Why cannot org-buffer simply be copied into the subordinate process? If
> all be buffer-locals, text properties, and overlays are copied directly
> from the main emacs process, there may be no need to even initialise
> org-mode (the idea is to do something similar to clone-buffer).

AFAIK there exists no way to do such a thing.  Buffers are not designed
to be serialized/deserialized like that.  You could try writing some
Elisp code to do it, but the end result would probably be much slower
than existing agenda code, as well as more difficult to debug.

> The question though is whether buffer-locals + overlays + propertized
> .org files text + org-agenda-buffer copy can be sufficient to make the
> org-agenda-redo run properly. Are there any other buffers, variables,
> or other environment settings used by org-agenda-redo?

As you can see in org-agenda.el, it's complicated.  Remember that an
Emacs process is like a Lisp image, full of state.  The more symbols and
other structures you copy to the async Emacs process (by printing and
reading them as text, remember), the slower it's going to be--and it
will always be slower than not using async.

>> If your agenda buffers are taking too long to refresh, you might
>> consider org-ql's views/saved-searches as an alternative. ...
>
> I know org-ql and I am pretty sure that it will improve performance.
> Actually, if one can make built-in org-agenda asynchronous, org-ql can
> probably use similar approach and become even faster :)

Asynchronous code is not faster; it's generally slower because of
yielding and synchronization.

> I am trying on default org-agenda now mostly because my current config
> is heavily geared towards default agenda and I am not sure if
> refactoring everything to use org-ql will worth it at the end in terms
> of performance. I use too many slow custom skip-functions.

org-ql doesn't use skip functions, just queries.

Re: Asynchronous org-agenda-redo

[Ihor Radchenko]

> Org Agenda code does not wait for keyboard input; it's busy building the
> agenda.  This is the case with most code in Emacs: it's not written to
> be asynchronous, and it doesn't return to the main thread until done.
> So you can sprinkle yields here and there and maybe be able to move
> point around while some code is running, but that will decrease
> performance, as well as introducing another level of complexity and
> another class of bugs (e.g. what if the user modifies a buffer while the
> agenda code is scanning it?).

Thanks for the explanation.

> AFAIK there exists no way to do such a thing.  Buffers are not designed
> to be serialized/deserialized like that.  You could try writing some
> Elisp code to do it, but the end result would probably be much slower
> than existing agenda code, as well as more difficult to debug.

Yeah. Even re-initialisation of, for example, overlays in org buffer is 
likely to take too much time.

> As you can see in org-agenda.el, it's complicated.  Remember that an
> Emacs process is like a Lisp image, full of state.  The more symbols and
> other structures you copy to the async Emacs process (by printing and
> reading them as text, remember), the slower it's going to be--and it
> will always be slower than not using async.

> Asynchronous code is not faster; it's generally slower because of
> yielding and synchronization.

I see now that generating agenda in separate process will cause too much
overheads.
Anyway, I will try to throw yields into agenda code just to check how
bad the performance can degrade.

> org-ql doesn't use skip functions, just queries.

Skip functions are essentially used-defined queries as soon as the
queries are tested against every headline.
I can rewrite my skip functions into queries, but I don't expect much
improvement since org-ql seems to use org-entry-get, which is the main
performance bottleneck for my agenda generation.

Best,
Ihor

adam Porter <adam@alphapapa.net> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>>> Be sure to read the Emacs Lisp manual regarding threads.  They are
>>> cooperative, so functions called as threads must yield back to the main
>>> thread for Emacs to do anything else before the function returns.
>>
>> I tried to read the manual, but I clearly misunderstand something.
>> The manual says:
>>
>>>   Currently, thread switching will occur upon explicit request via
>>> ‘thread-yield’, when waiting for keyboard input... 
>>
>> So, except directly calling thread-yield, it should be possible to
>> trigger switching the current thread when keyboard input is expected.
>> I tried the following demo code:
>>
>> (defun test ()
>>   (let ((a 0))
>>     (dotimes (_ 5)
>>       (setq a (1+ a))
>>       (sleep-for 2)
>>       (message "%s" a))))
>>
>> (progn ;This should return to command loop quickly
>>   (make-thread #'test)
>>   (message "Executed...")); `eval-last-sexp' here
>>
>> I can move around the buffer while the progn is running.
>> However, it is not the case with `org-agenda-redo' for a reason I do not
>> fully understand.
>
> Org Agenda code does not wait for keyboard input; it's busy building the
> agenda.  This is the case with most code in Emacs: it's not written to
> be asynchronous, and it doesn't return to the main thread until done.
> So you can sprinkle yields here and there and maybe be able to move
> point around while some code is running, but that will decrease
> performance, as well as introducing another level of complexity and
> another class of bugs (e.g. what if the user modifies a buffer while the
> agenda code is scanning it?).
>
>>> 1.  The process would have to load the same Org buffers, which takes
>>>     time, especially in large buffers.  Depending on configuration, it
>>>     can take some time, indeed.
>>
>>> 3.  Ensuring that configuration and state between the main Emacs process
>>>     and the separate, agenda-generating process is not necessarily
>>>     simple.  Consider as well that if a buffer had unsaved changes,
>>>     those would not be readable by the other process, which would lead
>>>     to invalid results.  One could force the buffers to be saved first,
>>>     but that may not always be desirable, as saving buffers can have
>>>     side effects.
>>
>> Why cannot org-buffer simply be copied into the subordinate process? If
>> all be buffer-locals, text properties, and overlays are copied directly
>> from the main emacs process, there may be no need to even initialise
>> org-mode (the idea is to do something similar to clone-buffer).
>
> AFAIK there exists no way to do such a thing.  Buffers are not designed
> to be serialized/deserialized like that.  You could try writing some
> Elisp code to do it, but the end result would probably be much slower
> than existing agenda code, as well as more difficult to debug.
>
>> The question though is whether buffer-locals + overlays + propertized
>> .org files text + org-agenda-buffer copy can be sufficient to make the
>> org-agenda-redo run properly. Are there any other buffers, variables,
>> or other environment settings used by org-agenda-redo?
>
> As you can see in org-agenda.el, it's complicated.  Remember that an
> Emacs process is like a Lisp image, full of state.  The more symbols and
> other structures you copy to the async Emacs process (by printing and
> reading them as text, remember), the slower it's going to be--and it
> will always be slower than not using async.
>
>>> If your agenda buffers are taking too long to refresh, you might
>>> consider org-ql's views/saved-searches as an alternative. ...
>>
>> I know org-ql and I am pretty sure that it will improve performance.
>> Actually, if one can make built-in org-agenda asynchronous, org-ql can
>> probably use similar approach and become even faster :)
>
> Asynchronous code is not faster; it's generally slower because of
> yielding and synchronization.
>
>> I am trying on default org-agenda now mostly because my current config
>> is heavily geared towards default agenda and I am not sure if
>> refactoring everything to use org-ql will worth it at the end in terms
>> of performance. I use too many slow custom skip-functions.
>
> org-ql doesn't use skip functions, just queries.
>
>

Re: Asynchronous org-agenda-redo

[Ihor Radchenko]

>> Asynchronous code is not faster; it's generally slower because of
>> yielding and synchronization.

> Anyway, I will try to throw yields into agenda code just to check how
> bad the performance can degrade.

With the following code, org-agenda-redo runs for 21 second on my
system, while without threads it is 16 seconds. However, emacs remains
responsive during rebuilding agenda!

(define-advice org-agenda-redo (:around (oldfun &optional all) make-async)
  (make-thread (lambda () (funcall oldfun all)) "org-agenda-redo"))
(define-advice org-agenda-skip-eval (:around (oldfun form) make-async)
  (thread-join (make-thread (lambda () (funcall oldfun form)) "org-agenda-skip-eval")))

The problem, of course, is that touching agenda buffer and org buffers
may be risky while org-agenda-redo is running.
Wondering if it is possible to block user commands during that time. 

Best,
Ihor


Ihor Radchenko <yantar92@gmail.com> writes:

>> Org Agenda code does not wait for keyboard input; it's busy building the
>> agenda.  This is the case with most code in Emacs: it's not written to
>> be asynchronous, and it doesn't return to the main thread until done.
>> So you can sprinkle yields here and there and maybe be able to move
>> point around while some code is running, but that will decrease
>> performance, as well as introducing another level of complexity and
>> another class of bugs (e.g. what if the user modifies a buffer while the
>> agenda code is scanning it?).
>
> Thanks for the explanation.
>
>> AFAIK there exists no way to do such a thing.  Buffers are not designed
>> to be serialized/deserialized like that.  You could try writing some
>> Elisp code to do it, but the end result would probably be much slower
>> than existing agenda code, as well as more difficult to debug.
>
> Yeah. Even re-initialisation of, for example, overlays in org buffer is 
> likely to take too much time.
>
>> As you can see in org-agenda.el, it's complicated.  Remember that an
>> Emacs process is like a Lisp image, full of state.  The more symbols and
>> other structures you copy to the async Emacs process (by printing and
>> reading them as text, remember), the slower it's going to be--and it
>> will always be slower than not using async.
>
>> Asynchronous code is not faster; it's generally slower because of
>> yielding and synchronization.
>
> I see now that generating agenda in separate process will cause too much
> overheads.
> Anyway, I will try to throw yields into agenda code just to check how
> bad the performance can degrade.
>
>> org-ql doesn't use skip functions, just queries.
>
> Skip functions are essentially used-defined queries as soon as the
> queries are tested against every headline.
> I can rewrite my skip functions into queries, but I don't expect much
> improvement since org-ql seems to use org-entry-get, which is the main
> performance bottleneck for my agenda generation.
>
> Best,
> Ihor
>
> adam Porter <adam@alphapapa.net> writes:
>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>>> Be sure to read the Emacs Lisp manual regarding threads.  They are
>>>> cooperative, so functions called as threads must yield back to the main
>>>> thread for Emacs to do anything else before the function returns.
>>>
>>> I tried to read the manual, but I clearly misunderstand something.
>>> The manual says:
>>>
>>>>   Currently, thread switching will occur upon explicit request via
>>>> ‘thread-yield’, when waiting for keyboard input... 
>>>
>>> So, except directly calling thread-yield, it should be possible to
>>> trigger switching the current thread when keyboard input is expected.
>>> I tried the following demo code:
>>>
>>> (defun test ()
>>>   (let ((a 0))
>>>     (dotimes (_ 5)
>>>       (setq a (1+ a))
>>>       (sleep-for 2)
>>>       (message "%s" a))))
>>>
>>> (progn ;This should return to command loop quickly
>>>   (make-thread #'test)
>>>   (message "Executed...")); `eval-last-sexp' here
>>>
>>> I can move around the buffer while the progn is running.
>>> However, it is not the case with `org-agenda-redo' for a reason I do not
>>> fully understand.
>>
>> Org Agenda code does not wait for keyboard input; it's busy building the
>> agenda.  This is the case with most code in Emacs: it's not written to
>> be asynchronous, and it doesn't return to the main thread until done.
>> So you can sprinkle yields here and there and maybe be able to move
>> point around while some code is running, but that will decrease
>> performance, as well as introducing another level of complexity and
>> another class of bugs (e.g. what if the user modifies a buffer while the
>> agenda code is scanning it?).
>>
>>>> 1.  The process would have to load the same Org buffers, which takes
>>>>     time, especially in large buffers.  Depending on configuration, it
>>>>     can take some time, indeed.
>>>
>>>> 3.  Ensuring that configuration and state between the main Emacs process
>>>>     and the separate, agenda-generating process is not necessarily
>>>>     simple.  Consider as well that if a buffer had unsaved changes,
>>>>     those would not be readable by the other process, which would lead
>>>>     to invalid results.  One could force the buffers to be saved first,
>>>>     but that may not always be desirable, as saving buffers can have
>>>>     side effects.
>>>
>>> Why cannot org-buffer simply be copied into the subordinate process? If
>>> all be buffer-locals, text properties, and overlays are copied directly
>>> from the main emacs process, there may be no need to even initialise
>>> org-mode (the idea is to do something similar to clone-buffer).
>>
>> AFAIK there exists no way to do such a thing.  Buffers are not designed
>> to be serialized/deserialized like that.  You could try writing some
>> Elisp code to do it, but the end result would probably be much slower
>> than existing agenda code, as well as more difficult to debug.
>>
>>> The question though is whether buffer-locals + overlays + propertized
>>> .org files text + org-agenda-buffer copy can be sufficient to make the
>>> org-agenda-redo run properly. Are there any other buffers, variables,
>>> or other environment settings used by org-agenda-redo?
>>
>> As you can see in org-agenda.el, it's complicated.  Remember that an
>> Emacs process is like a Lisp image, full of state.  The more symbols and
>> other structures you copy to the async Emacs process (by printing and
>> reading them as text, remember), the slower it's going to be--and it
>> will always be slower than not using async.
>>
>>>> If your agenda buffers are taking too long to refresh, you might
>>>> consider org-ql's views/saved-searches as an alternative. ...
>>>
>>> I know org-ql and I am pretty sure that it will improve performance.
>>> Actually, if one can make built-in org-agenda asynchronous, org-ql can
>>> probably use similar approach and become even faster :)
>>
>> Asynchronous code is not faster; it's generally slower because of
>> yielding and synchronization.
>>
>>> I am trying on default org-agenda now mostly because my current config
>>> is heavily geared towards default agenda and I am not sure if
>>> refactoring everything to use org-ql will worth it at the end in terms
>>> of performance. I use too many slow custom skip-functions.
>>
>> org-ql doesn't use skip functions, just queries.
>>
>>
>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: Asynchronous org-agenda-redo

[Adam Porter]

Ihor Radchenko <yantar92@gmail.com> writes:

>> org-ql doesn't use skip functions, just queries.
>
> Skip functions are essentially used-defined queries as soon as the
> queries are tested against every headline.

Skip functions aren't necessary with org-ql, because the query itself
can have arbitrary Lisp code.  So, of course, you can call custom
functions in queries, even your own skip functions (with `not', of
course), but in most cases, they can be covered with built-in
predicates.

> I can rewrite my skip functions into queries, but I don't expect much
> improvement since org-ql seems to use org-entry-get, which is the main
> performance bottleneck for my agenda generation.

org-entry-get is only called for the (property) predicate.  It's the
correct way to get Org properties, because it handles special
properties, inheritance, etc.  However, when possible, queries are
optimized to a whole-buffer regexp search that finds possible matches.
So, for example, a query like '(property "owner" "yantar") would be
optimized to a whole-buffer regexp search that would be very fast.  See
function org-ql--query-preamble.

Re: Asynchronous org-agenda-redo

[Adam Porter]

Ihor Radchenko <yantar92@gmail.com> writes:

>>> Asynchronous code is not faster; it's generally slower because of
>>> yielding and synchronization.
>
>> Anyway, I will try to throw yields into agenda code just to check how
>> bad the performance can degrade.
>
> With the following code, org-agenda-redo runs for 21 second on my
> system, while without threads it is 16 seconds. However, emacs remains
> responsive during rebuilding agenda!
>
> (define-advice org-agenda-redo (:around (oldfun &optional all) make-async)
>   (make-thread (lambda () (funcall oldfun all)) "org-agenda-redo"))
> (define-advice org-agenda-skip-eval (:around (oldfun form) make-async)
>   (thread-join (make-thread (lambda () (funcall oldfun form)) "org-agenda-skip-eval")))

That's a very elegant way to add the threading!  The performance penalty
is noticeable, but the tradeoff could be worth it in some cases, like a
background agenda refresh on a timer, or after a "remote" edit.  I can
imagine an org-agenda-refresh-async command that would add that advice
and remove them in an unwind-protect.

> The problem, of course, is that touching agenda buffer and org buffers
> may be risky while org-agenda-redo is running.
> Wondering if it is possible to block user commands during that time.

The first thing that comes to mind is to set buffer-read-only on each
buffer before it is scanned, and unset it when done with a buffer.  That
might not be doable with advice.

Asynchronous org-babel-tangle (was Re: Asynchronous org-agenda-redo)

[Diego Zamboni]

[-- Attachment #1: Type: text/plain, Size: 7593 bytes --]

Hi Ihor,

On Thu, Dec 12, 2019 at 4:00 PM Ihor Radchenko <yantar92@gmail.com> wrote:

> See the relevant code from my config below. Let me know if you have any
> questions or suggestions.
>

Thank you, this was extremely helpful. I am now using a modified version of
your code (see below) with the following comments/questions:

- For me, the tangle works if I only load the org library in the
sub-process, so I disabled the loading of my config. I guess this depends
heavily and what other configuration one has loaded, but I think for
tangling, not much else is needed :)

- In you code, the process-list variable was not actually being modified
since plist-put returns the new value but does not modify the variable. I
tried a few variations wrapping the plist-put in (setq ...), and adding
corresponding code to clear the element after the tangle was finished, but
could not get it to work. As this is not really critical for me, I just
removed the whole process-checking code, significantly simplifying the rest.

- I wanted my code to report the time the tangle took. This works well by
passing the time as the result of the initial lambda. Interestingly, it
works if I return the time already formatted as a string (as in my code)
but not if I return the result of (float-time ..) directly.

Here's my code:

  #+begin_src emacs-lisp
    (defun zz/org-babel-tangle-async (file)
      "Invoke `org-babel-tangle-file' asynchronously."
      (message "Tangling %s..." (buffer-file-name))
      (async-start
       (let ((args (list file)))
         `(lambda ()
            (require 'org)
            ;;(load "~/.emacs.d/init.el")
            (let ((start-time (current-time)))
              (apply #'org-babel-tangle-file ',args)
              (format "%.2f" (float-time (time-since start-time))))))
       (let ((message-string (format "Tangling %S completed after " file)))
         `(lambda (tangle-time)
            (message (concat ,message-string
                             (format "%s seconds" tangle-time)))))))

    (defun zz/org-babel-tangle-current-buffer-async ()
      "Tangle current buffer asynchronously."
      (zz/org-babel-tangle-async (buffer-file-name)))
    #+end_src

Thanks again for your help, and for the inspiration to finally get this
done, it had been bugging me for a while :)

--Diego



> Also, my config is written in the way that everything related to user
> interaction can be disabled if I load emacs for tangling
> (with org-tangle-flag). This is to speed up the async process.
>
> #+begin_src emacs-lisp
> (defvar yant/org-babel-tangle-async-process-list nil
>   "Plist of (file . process) for all the currently running async tangle
> processes.")
>
>
> (defun yant/org-babel-tangle-async (file &optional target-file lang)
>   "Invoke `org-babel-tangle-file' asynchronously."
>   (require 'async)
>   (let ((oldproc (plist-get yant/org-babel-tangle-async-process-list
> file)))
>     (when (or (not oldproc)
>               (async-wait oldproc))
>       (message "Tangling %s..." (buffer-file-name))
>       (plist-put yant/org-babel-tangle-async-process-list
>                  file
>                  (async-start
>                   (let ((args (list file target-file lang)))
>                     `(lambda ()
>                        (require 'org)
>                        (setq org-tangle-flag t)
>                        (load "~/.emacs.d/config.el")
>                        (apply #'org-babel-tangle-file ',args)))
>                   (let ((message-string (format "Tangling (%S %S %S)
> completed." file target-file lang)))
>                     `(lambda (result) (message ,message-string))))))))
>
> (defvar yant/auto-tangle-list nil
>   "List of files, which can be safely tangled on save.
> The list is saved between Emacs sessions.")
>
> (when init-flag
>   (use-package savehist
>     :config
>     (add-to-list 'savehist-additional-variables 'yant/auto-tangle-list))
>   (savehist-mode +1)
>   (defun yant/toggle-buffer-auto-tangle (arg)
>     "Toggle auto tangling of a buffer."
>     (interactive "P")
>     (if (not (eq major-mode 'org-mode))
>         (message "Org-mode is not active in buffer \"%s\"" (buffer-name))
>       (cond ((not arg)
>              (if (member (buffer-file-name) yant/auto-tangle-list)
>                  (progn (setq yant/auto-tangle-list (delete
> (buffer-file-name) yant/auto-tangle-list))
>                         (message "Auto tangling disabled for %s"
> (buffer-file-name)))
>                (add-to-list 'yant/auto-tangle-list (buffer-file-name))
>                (message "Auto tangling enabled for %s"
> (buffer-file-name))))
>             ((or (and (not (listp arg)) (> arg 0))
>                  (equal arg '(4)))
>              (add-to-list 'yant/auto-tangle-list (buffer-file-name))
>              (message "Auto tangling enabled for %s" (buffer-file-name)))
>             (t
>              (setq yant/auto-tangle-list (delete (buffer-file-name)
> yant/auto-tangle-list))
>              (message "Auto tangling disabled for %s"
> (buffer-file-name))))))
>
>   (bind-key "C-c C-*" #'yant/toggle-buffer-auto-tangle org-mode-map))
>
> (defun yant/org-babel-tangle-current-buffer-async ()
>   "Tangle current buffer asynchronously."
>   (when (and (eq major-mode 'org-mode)
>              (member (buffer-file-name) yant/auto-tangle-list))
>     (yant/org-babel-tangle-async (buffer-file-name))))
>
> (add-hook 'after-save-hook #'yant/org-babel-tangle-current-buffer-async)
> #+end_src
>
>
> Diego Zamboni <diego@zzamboni.org> writes:
>
> > Hi Ihor,
> >
> > I cannot answer your question, but I am curious about using async
> together
> > with tangling, since for some of my buffers, tangling takes some time and
> > freezes Emacs in the process. Do you have some examples of this that you
> > could share?
> >
> > Thanks,
> > --Diego
> >
> >
> > On Thu, Dec 12, 2019 at 9:21 AM Ihor Radchenko <yantar92@gmail.com>
> wrote:
> >
> >> I am thinking if it is possible to implement org-agenda-redo
> >> asynchronously.
> >>
> >> Rebuilding agenda should normally not affect any buffer except agenda
> >> buffer. So, it should be sufficient to block any agenda modifying
> >> commands in the agenda buffer, redo the agenda buffer in separate
> >> thread, and replace the old agenda with the calculated one.
> >> Then, emacs should remain responsive while updating agenda (except for
> >> modifying the agenda buffer).
> >>
> >> For example, this naive code kind of works (forgetting that buffer-local
> >> variables will not be passed to the thread):
> >>
> >> (define-advice org-agenda-redo (:around (oldfun &optional all)
> make-async)
> >>   (make-thread oldfun "org-agenda-redo"))
> >>
> >> The problem is that emacs does not become responsive...
> >>
> >> Another approach would be using async.el package, which allows calling
> >> arbitrary function in subordinate emacs process. Then, the main emacs
> >> instance should not be "frozen" (I use same approach for tangling and it
> >> works fine).
> >>
> >> However, the question is how to pass the .org and agenda buffers to this
> >> subordinate process. Opening the .org files there is not a good option
> >> since it would give too much overhead to this asynchronous agenda.
> >>
> >> Any suggestions? Alternative ideas?
> >>
> >> Best,
> >> Ihor
> >>
> >>
> >>
>
> --
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong
> University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg
>

[-- Attachment #2: Type: text/html, Size: 9846 bytes --]

Re: Asynchronous org-babel-tangle (was Re: Asynchronous org-agenda-redo)

[Ihor Radchenko]

Thanks for the feedback.

> - For me, the tangle works if I only load the org library in the
> sub-process, so I disabled the loading of my config. I guess this depends
> heavily and what other configuration one has loaded, but I think for
> tangling, not much else is needed :)

Note that your version tangling may not work as expected if your
init file changes any default headers args affecting the tangling.
For example, I set (:comments . "link") in my init file. Not loading
init file would disallow detangling in my case. 

> - In you code, the process-list variable was not actually being modified
> since plist-put returns the new value but does not modify the
> variable.

(facepalm)

> I
> tried a few variations wrapping the plist-put in (setq ...), and adding
> corresponding code to clear the element after the tangle was finished, but
> could not get it to work. 

Well. Things did not work because plist-put compares keys by #'eq, which
returns nil on equal strings.

> - I wanted my code to report the time the tangle took. This works well by
> passing the time as the result of the initial lambda. Interestingly, it
> works if I return the time already formatted as a string (as in my code)
> but not if I return the result of (float-time ..) directly.

Thanks for this neat idea. Will also grab it to my config.

float-time did not work because of bug in async.el. It uses
backward-sexp to find the returned value of the lambda (in the process
buffer). But backwards-sexp fails on float (it moves point just after
the point).  

Best,
Ihor


Diego Zamboni <diego@zzamboni.org> writes:

> Hi Ihor,
>
> On Thu, Dec 12, 2019 at 4:00 PM Ihor Radchenko <yantar92@gmail.com> wrote:
>
>> See the relevant code from my config below. Let me know if you have any
>> questions or suggestions.
>>
>
> Thank you, this was extremely helpful. I am now using a modified version of
> your code (see below) with the following comments/questions:
>
> - For me, the tangle works if I only load the org library in the
> sub-process, so I disabled the loading of my config. I guess this depends
> heavily and what other configuration one has loaded, but I think for
> tangling, not much else is needed :)
>
> - In you code, the process-list variable was not actually being modified
> since plist-put returns the new value but does not modify the variable. I
> tried a few variations wrapping the plist-put in (setq ...), and adding
> corresponding code to clear the element after the tangle was finished, but
> could not get it to work. As this is not really critical for me, I just
> removed the whole process-checking code, significantly simplifying the rest.
>
> - I wanted my code to report the time the tangle took. This works well by
> passing the time as the result of the initial lambda. Interestingly, it
> works if I return the time already formatted as a string (as in my code)
> but not if I return the result of (float-time ..) directly.
>
> Here's my code:
>
>   #+begin_src emacs-lisp
>     (defun zz/org-babel-tangle-async (file)
>       "Invoke `org-babel-tangle-file' asynchronously."
>       (message "Tangling %s..." (buffer-file-name))
>       (async-start
>        (let ((args (list file)))
>          `(lambda ()
>             (require 'org)
>             ;;(load "~/.emacs.d/init.el")
>             (let ((start-time (current-time)))
>               (apply #'org-babel-tangle-file ',args)
>               (format "%.2f" (float-time (time-since start-time))))))
>        (let ((message-string (format "Tangling %S completed after " file)))
>          `(lambda (tangle-time)
>             (message (concat ,message-string
>                              (format "%s seconds" tangle-time)))))))
>
>     (defun zz/org-babel-tangle-current-buffer-async ()
>       "Tangle current buffer asynchronously."
>       (zz/org-babel-tangle-async (buffer-file-name)))
>     #+end_src
>
> Thanks again for your help, and for the inspiration to finally get this
> done, it had been bugging me for a while :)
>
> --Diego
>
>
>
>> Also, my config is written in the way that everything related to user
>> interaction can be disabled if I load emacs for tangling
>> (with org-tangle-flag). This is to speed up the async process.
>>
>> #+begin_src emacs-lisp
>> (defvar yant/org-babel-tangle-async-process-list nil
>>   "Plist of (file . process) for all the currently running async tangle
>> processes.")
>>
>>
>> (defun yant/org-babel-tangle-async (file &optional target-file lang)
>>   "Invoke `org-babel-tangle-file' asynchronously."
>>   (require 'async)
>>   (let ((oldproc (plist-get yant/org-babel-tangle-async-process-list
>> file)))
>>     (when (or (not oldproc)
>>               (async-wait oldproc))
>>       (message "Tangling %s..." (buffer-file-name))
>>       (plist-put yant/org-babel-tangle-async-process-list
>>                  file
>>                  (async-start
>>                   (let ((args (list file target-file lang)))
>>                     `(lambda ()
>>                        (require 'org)
>>                        (setq org-tangle-flag t)
>>                        (load "~/.emacs.d/config.el")
>>                        (apply #'org-babel-tangle-file ',args)))
>>                   (let ((message-string (format "Tangling (%S %S %S)
>> completed." file target-file lang)))
>>                     `(lambda (result) (message ,message-string))))))))
>>
>> (defvar yant/auto-tangle-list nil
>>   "List of files, which can be safely tangled on save.
>> The list is saved between Emacs sessions.")
>>
>> (when init-flag
>>   (use-package savehist
>>     :config
>>     (add-to-list 'savehist-additional-variables 'yant/auto-tangle-list))
>>   (savehist-mode +1)
>>   (defun yant/toggle-buffer-auto-tangle (arg)
>>     "Toggle auto tangling of a buffer."
>>     (interactive "P")
>>     (if (not (eq major-mode 'org-mode))
>>         (message "Org-mode is not active in buffer \"%s\"" (buffer-name))
>>       (cond ((not arg)
>>              (if (member (buffer-file-name) yant/auto-tangle-list)
>>                  (progn (setq yant/auto-tangle-list (delete
>> (buffer-file-name) yant/auto-tangle-list))
>>                         (message "Auto tangling disabled for %s"
>> (buffer-file-name)))
>>                (add-to-list 'yant/auto-tangle-list (buffer-file-name))
>>                (message "Auto tangling enabled for %s"
>> (buffer-file-name))))
>>             ((or (and (not (listp arg)) (> arg 0))
>>                  (equal arg '(4)))
>>              (add-to-list 'yant/auto-tangle-list (buffer-file-name))
>>              (message "Auto tangling enabled for %s" (buffer-file-name)))
>>             (t
>>              (setq yant/auto-tangle-list (delete (buffer-file-name)
>> yant/auto-tangle-list))
>>              (message "Auto tangling disabled for %s"
>> (buffer-file-name))))))
>>
>>   (bind-key "C-c C-*" #'yant/toggle-buffer-auto-tangle org-mode-map))
>>
>> (defun yant/org-babel-tangle-current-buffer-async ()
>>   "Tangle current buffer asynchronously."
>>   (when (and (eq major-mode 'org-mode)
>>              (member (buffer-file-name) yant/auto-tangle-list))
>>     (yant/org-babel-tangle-async (buffer-file-name))))
>>
>> (add-hook 'after-save-hook #'yant/org-babel-tangle-current-buffer-async)
>> #+end_src
>>
>>
>> Diego Zamboni <diego@zzamboni.org> writes:
>>
>> > Hi Ihor,
>> >
>> > I cannot answer your question, but I am curious about using async
>> together
>> > with tangling, since for some of my buffers, tangling takes some time and
>> > freezes Emacs in the process. Do you have some examples of this that you
>> > could share?
>> >
>> > Thanks,
>> > --Diego
>> >
>> >
>> > On Thu, Dec 12, 2019 at 9:21 AM Ihor Radchenko <yantar92@gmail.com>
>> wrote:
>> >
>> >> I am thinking if it is possible to implement org-agenda-redo
>> >> asynchronously.
>> >>
>> >> Rebuilding agenda should normally not affect any buffer except agenda
>> >> buffer. So, it should be sufficient to block any agenda modifying
>> >> commands in the agenda buffer, redo the agenda buffer in separate
>> >> thread, and replace the old agenda with the calculated one.
>> >> Then, emacs should remain responsive while updating agenda (except for
>> >> modifying the agenda buffer).
>> >>
>> >> For example, this naive code kind of works (forgetting that buffer-local
>> >> variables will not be passed to the thread):
>> >>
>> >> (define-advice org-agenda-redo (:around (oldfun &optional all)
>> make-async)
>> >>   (make-thread oldfun "org-agenda-redo"))
>> >>
>> >> The problem is that emacs does not become responsive...
>> >>
>> >> Another approach would be using async.el package, which allows calling
>> >> arbitrary function in subordinate emacs process. Then, the main emacs
>> >> instance should not be "frozen" (I use same approach for tangling and it
>> >> works fine).
>> >>
>> >> However, the question is how to pass the .org and agenda buffers to this
>> >> subordinate process. Opening the .org files there is not a good option
>> >> since it would give too much overhead to this asynchronous agenda.
>> >>
>> >> Any suggestions? Alternative ideas?
>> >>
>> >> Best,
>> >> Ihor
>> >>
>> >>
>> >>
>>
>> --
>> Ihor Radchenko,
>> PhD,
>> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
>> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong
>> University, Xi'an, China
>> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg
>>

Re: Asynchronous org-babel-tangle (was Re: Asynchronous org-agenda-redo)

[Ihor Radchenko]

Thanks for the feedback.

> - For me, the tangle works if I only load the org library in the
> sub-process, so I disabled the loading of my config. I guess this depends
> heavily and what other configuration one has loaded, but I think for
> tangling, not much else is needed :)

Note that your version tangling may not work as expected if your
init file changes any default headers args affecting the tangling.
For example, I set (:comments . "link") in my init file. Not loading
init file would disallow detangling in my case. 

> - In you code, the process-list variable was not actually being modified
> since plist-put returns the new value but does not modify the
> variable.

(facepalm)

> I
> tried a few variations wrapping the plist-put in (setq ...), and adding
> corresponding code to clear the element after the tangle was finished, but
> could not get it to work. 

Well. Things did not work because plist-put compares keys by #'eq, which
returns nil on equal strings.

> - I wanted my code to report the time the tangle took. This works well by
> passing the time as the result of the initial lambda. Interestingly, it
> works if I return the time already formatted as a string (as in my code)
> but not if I return the result of (float-time ..) directly.

Thanks for this neat idea. Will also grab it to my config.

float-time did not work because of bug in async.el. It uses
backward-sexp to find the returned value of the lambda (in the process
buffer). But backwards-sexp fails on float (it moves point just after
the point).  

P.S. I just noticed that async tangle fails when I try to tangle into
"/sudo::filename". Not even sure how to fix it. 

Best,
Ihor


Diego Zamboni <diego@zzamboni.org> writes:

> Hi Ihor,
>
> On Thu, Dec 12, 2019 at 4:00 PM Ihor Radchenko <yantar92@gmail.com> wrote:
>
>> See the relevant code from my config below. Let me know if you have any
>> questions or suggestions.
>>
>
> Thank you, this was extremely helpful. I am now using a modified version of
> your code (see below) with the following comments/questions:
>
> - For me, the tangle works if I only load the org library in the
> sub-process, so I disabled the loading of my config. I guess this depends
> heavily and what other configuration one has loaded, but I think for
> tangling, not much else is needed :)
>
> - In you code, the process-list variable was not actually being modified
> since plist-put returns the new value but does not modify the variable. I
> tried a few variations wrapping the plist-put in (setq ...), and adding
> corresponding code to clear the element after the tangle was finished, but
> could not get it to work. As this is not really critical for me, I just
> removed the whole process-checking code, significantly simplifying the rest.
>
> - I wanted my code to report the time the tangle took. This works well by
> passing the time as the result of the initial lambda. Interestingly, it
> works if I return the time already formatted as a string (as in my code)
> but not if I return the result of (float-time ..) directly.
>
> Here's my code:
>
>   #+begin_src emacs-lisp
>     (defun zz/org-babel-tangle-async (file)
>       "Invoke `org-babel-tangle-file' asynchronously."
>       (message "Tangling %s..." (buffer-file-name))
>       (async-start
>        (let ((args (list file)))
>          `(lambda ()
>             (require 'org)
>             ;;(load "~/.emacs.d/init.el")
>             (let ((start-time (current-time)))
>               (apply #'org-babel-tangle-file ',args)
>               (format "%.2f" (float-time (time-since start-time))))))
>        (let ((message-string (format "Tangling %S completed after " file)))
>          `(lambda (tangle-time)
>             (message (concat ,message-string
>                              (format "%s seconds" tangle-time)))))))
>
>     (defun zz/org-babel-tangle-current-buffer-async ()
>       "Tangle current buffer asynchronously."
>       (zz/org-babel-tangle-async (buffer-file-name)))
>     #+end_src
>
> Thanks again for your help, and for the inspiration to finally get this
> done, it had been bugging me for a while :)
>
> --Diego
>
>
>
>> Also, my config is written in the way that everything related to user
>> interaction can be disabled if I load emacs for tangling
>> (with org-tangle-flag). This is to speed up the async process.
>>
>> #+begin_src emacs-lisp
>> (defvar yant/org-babel-tangle-async-process-list nil
>>   "Plist of (file . process) for all the currently running async tangle
>> processes.")
>>
>>
>> (defun yant/org-babel-tangle-async (file &optional target-file lang)
>>   "Invoke `org-babel-tangle-file' asynchronously."
>>   (require 'async)
>>   (let ((oldproc (plist-get yant/org-babel-tangle-async-process-list
>> file)))
>>     (when (or (not oldproc)
>>               (async-wait oldproc))
>>       (message "Tangling %s..." (buffer-file-name))
>>       (plist-put yant/org-babel-tangle-async-process-list
>>                  file
>>                  (async-start
>>                   (let ((args (list file target-file lang)))
>>                     `(lambda ()
>>                        (require 'org)
>>                        (setq org-tangle-flag t)
>>                        (load "~/.emacs.d/config.el")
>>                        (apply #'org-babel-tangle-file ',args)))
>>                   (let ((message-string (format "Tangling (%S %S %S)
>> completed." file target-file lang)))
>>                     `(lambda (result) (message ,message-string))))))))
>>
>> (defvar yant/auto-tangle-list nil
>>   "List of files, which can be safely tangled on save.
>> The list is saved between Emacs sessions.")
>>
>> (when init-flag
>>   (use-package savehist
>>     :config
>>     (add-to-list 'savehist-additional-variables 'yant/auto-tangle-list))
>>   (savehist-mode +1)
>>   (defun yant/toggle-buffer-auto-tangle (arg)
>>     "Toggle auto tangling of a buffer."
>>     (interactive "P")
>>     (if (not (eq major-mode 'org-mode))
>>         (message "Org-mode is not active in buffer \"%s\"" (buffer-name))
>>       (cond ((not arg)
>>              (if (member (buffer-file-name) yant/auto-tangle-list)
>>                  (progn (setq yant/auto-tangle-list (delete
>> (buffer-file-name) yant/auto-tangle-list))
>>                         (message "Auto tangling disabled for %s"
>> (buffer-file-name)))
>>                (add-to-list 'yant/auto-tangle-list (buffer-file-name))
>>                (message "Auto tangling enabled for %s"
>> (buffer-file-name))))
>>             ((or (and (not (listp arg)) (> arg 0))
>>                  (equal arg '(4)))
>>              (add-to-list 'yant/auto-tangle-list (buffer-file-name))
>>              (message "Auto tangling enabled for %s" (buffer-file-name)))
>>             (t
>>              (setq yant/auto-tangle-list (delete (buffer-file-name)
>> yant/auto-tangle-list))
>>              (message "Auto tangling disabled for %s"
>> (buffer-file-name))))))
>>
>>   (bind-key "C-c C-*" #'yant/toggle-buffer-auto-tangle org-mode-map))
>>
>> (defun yant/org-babel-tangle-current-buffer-async ()
>>   "Tangle current buffer asynchronously."
>>   (when (and (eq major-mode 'org-mode)
>>              (member (buffer-file-name) yant/auto-tangle-list))
>>     (yant/org-babel-tangle-async (buffer-file-name))))
>>
>> (add-hook 'after-save-hook #'yant/org-babel-tangle-current-buffer-async)
>> #+end_src
>>
>>
>> Diego Zamboni <diego@zzamboni.org> writes:
>>
>> > Hi Ihor,
>> >
>> > I cannot answer your question, but I am curious about using async
>> together
>> > with tangling, since for some of my buffers, tangling takes some time and
>> > freezes Emacs in the process. Do you have some examples of this that you
>> > could share?
>> >
>> > Thanks,
>> > --Diego
>> >
>> >
>> > On Thu, Dec 12, 2019 at 9:21 AM Ihor Radchenko <yantar92@gmail.com>
>> wrote:
>> >
>> >> I am thinking if it is possible to implement org-agenda-redo
>> >> asynchronously.
>> >>
>> >> Rebuilding agenda should normally not affect any buffer except agenda
>> >> buffer. So, it should be sufficient to block any agenda modifying
>> >> commands in the agenda buffer, redo the agenda buffer in separate
>> >> thread, and replace the old agenda with the calculated one.
>> >> Then, emacs should remain responsive while updating agenda (except for
>> >> modifying the agenda buffer).
>> >>
>> >> For example, this naive code kind of works (forgetting that buffer-local
>> >> variables will not be passed to the thread):
>> >>
>> >> (define-advice org-agenda-redo (:around (oldfun &optional all)
>> make-async)
>> >>   (make-thread oldfun "org-agenda-redo"))
>> >>
>> >> The problem is that emacs does not become responsive...
>> >>
>> >> Another approach would be using async.el package, which allows calling
>> >> arbitrary function in subordinate emacs process. Then, the main emacs
>> >> instance should not be "frozen" (I use same approach for tangling and it
>> >> works fine).
>> >>
>> >> However, the question is how to pass the .org and agenda buffers to this
>> >> subordinate process. Opening the .org files there is not a good option
>> >> since it would give too much overhead to this asynchronous agenda.
>> >>
>> >> Any suggestions? Alternative ideas?
>> >>
>> >> Best,
>> >> Ihor
>> >>
>> >>
>> >>
>>
>> --
>> Ihor Radchenko,
>> PhD,
>> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
>> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong
>> University, Xi'an, China
>> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg
>>

Re: Asynchronous org-agenda-redo

[Ihor Radchenko]

> So, of course, you can call custom
> functions in queries, even your own skip functions (with `not', of
> course), but in most cases, they can be covered with built-in
> predicates.

Unfortunately, it does not seem to be the case for me.
My main agenda view needs to  take into account multiple properties,
which cannot be handled within framework of built-in org-ql predicates
(AFAIK):

- =:SHOWFROMTIME:= (always inheriting) :: The purpose of this is to be
     able to assign specific projects for different days of week or,
     say, show the home items only in the evening of weekdays and not
     annoy me at work when I cannot do it any way. Hence, I can focus on
     the items I really need to do now in this agenda. Additionally, the
     time of the day after midnight is treated specially here. If
     =org-extend-today-until= is not 0 and the current time is before
     its value, the current time is still considered to be yesterday.
- =:SHOWFROMDATE:= :: The purpose of this is to be able to postpone the
     scheduled tasks for future if I cannot do it. The property is
     formatted as an org date. This property is especially useful if
     there is something more pressing, so that there is a temptation to
     reschedule less pressing event to another day. If the more pressing
     task is done earlier than expected, the postponed tasks can be
     still find in normal agenda view (not in the
     [[id:ff70b03f-3876-4b2b-9aab-c3209bd31cb8][focused]] one).
- =:SHOWDATES:= (always inheriting) :: It contains dairy =sexps= to set
     when the project should be shown. For example, I may want to work on
     Saturday once or twice, but the working items should not be shown on
     weekend normally. Hence, I can define it. Or some things can only be
     done on specific dates (say, going to some shop, which is open few
     days a week only)

> org-entry-get is only called for the (property) predicate.  It's the
> correct way to get Org properties, because it handles special
> properties, inheritance, etc.  However, when possible, queries are
> optimized to a whole-buffer regexp search that finds possible matches.
> So, for example, a query like '(property "owner" "yantar") would be
> optimized to a whole-buffer regexp search that would be very fast.  See
> function org-ql--query-preamble.

Thanks for the info! I did not know about this optimisation in org-ql.
org-entry-get consumes most of cpu time while building my agenda views.
Though I don't think that there will be much difference for me since
most of my property conditions in agenda involve inherited properties.

Regards,
Ihor

Adam Porter <adam@alphapapa.net> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>>> org-ql doesn't use skip functions, just queries.
>>
>> Skip functions are essentially used-defined queries as soon as the
>> queries are tested against every headline.
>
> Skip functions aren't necessary with org-ql, because the query itself
> can have arbitrary Lisp code.  So, of course, you can call custom
> functions in queries, even your own skip functions (with `not', of
> course), but in most cases, they can be covered with built-in
> predicates.
>
>> I can rewrite my skip functions into queries, but I don't expect much
>> improvement since org-ql seems to use org-entry-get, which is the main
>> performance bottleneck for my agenda generation.
>
> org-entry-get is only called for the (property) predicate.  It's the
> correct way to get Org properties, because it handles special
> properties, inheritance, etc.  However, when possible, queries are
> optimized to a whole-buffer regexp search that finds possible matches.
> So, for example, a query like '(property "owner" "yantar") would be
> optimized to a whole-buffer regexp search that would be very fast.  See
> function org-ql--query-preamble.
>
>
>

Re: Asynchronous org-agenda-redo

[Adam Porter]

Ihor Radchenko <yantar92@gmail.com> writes:

>> So, of course, you can call custom functions in queries, even your
>> own skip functions (with `not', of course), but in most cases, they
>> can be covered with built-in predicates.
>
> Unfortunately, it does not seem to be the case for me.  My main agenda
> view needs to take into account multiple properties, which cannot be
> handled within framework of built-in org-ql predicates (AFAIK):
>
> - =:SHOWFROMTIME:= (always inheriting) :: The purpose of this is to be
>      able to assign specific projects for different days of week or,
>      say, show the home items only in the evening of weekdays and not
>      annoy me at work when I cannot do it any way. Hence, I can focus on
>      the items I really need to do now in this agenda. Additionally, the
>      time of the day after midnight is treated specially here. If
>      =org-extend-today-until= is not 0 and the current time is before
>      its value, the current time is still considered to be yesterday.
> - =:SHOWFROMDATE:= :: The purpose of this is to be able to postpone the
>      scheduled tasks for future if I cannot do it. The property is
>      formatted as an org date. This property is especially useful if
>      there is something more pressing, so that there is a temptation to
>      reschedule less pressing event to another day. If the more pressing
>      task is done earlier than expected, the postponed tasks can be
>      still find in normal agenda view (not in the
>      [[id:ff70b03f-3876-4b2b-9aab-c3209bd31cb8][focused]] one).
> - =:SHOWDATES:= (always inheriting) :: It contains dairy =sexps= to set
>      when the project should be shown. For example, I may want to work on
>      Saturday once or twice, but the working items should not be shown on
>      weekend normally. Hence, I can define it. Or some things can only be
>      done on specific dates (say, going to some shop, which is open few
>      days a week only)

There may still be some performance improvement available for these kind
of queries.  For example, assuming that a function `yantar/showfromdate'
is a predicate which accepts the value of property SHOWFROMDATE as its
argument and returns non-nil when an entry matches a query, an org-ql
query like this could work:

    (and (property "SHOWFROMDATE")
         (yantar/showfromdate (property "SHOWFROMDATE")))

The first (property "SHOWFROMDATE") clause allows org-ql to optimize the
query using a regexp preamble that jumps directly to entries having that
property, while the second clause passes the value of the property to
the custom predicate using the built-in predicate (property
"SHOWFROMDATE") to retrieve the property from the org-ql node-value
cache, which reduces the number of calls to org-entry-get.  (That the
`property' predicate returns the value of the property when called with
only a property argument is a helpful feature that I should document...)

> Thanks for the info! I did not know about this optimisation in org-ql.
> org-entry-get consumes most of cpu time while building my agenda views.
> Though I don't think that there will be much difference for me since
> most of my property conditions in agenda involve inherited properties.

For properties using inheritance, the preamble regexp can't be used to
optimize the query across the buffer (I have some ideas about how to do
that, maybe someday...), but the node-value cache can improve
performance of retrieving the values of inherited properties within a
buffer, because it avoids repeated calls to org-entry-get going up an
outline tree.

For example, see the benchmark here:

https://github.com/alphapapa/org-ql/blob/master/notes.org#caching-of-inherited-tags

The node-value cache, which is used for properties, uses essentially the
same code as the tag cache benchmarked here (I haven't yet converted the
tag cache to use the generic node-value cache), and it provides about a
6x speedup, so property-inheritance-based queries should benefit
similarly.

So it sounds like your custom agendas would benefit from being written
as org-ql queries.  Please let me know if you try it, as your examples
would give the code a good workout and might reveal some improvements to
be made.

Re: Asynchronous org-agenda-redo

[Ihor Radchenko]

> That's a very elegant way to add the threading!

Thanks for kind words.

> The performance penalty
> is noticeable, but the tradeoff could be worth it in some cases, like a
> background agenda refresh on a timer, or after a "remote" edit.  I can
> imagine an org-agenda-refresh-async command that would add that advice
> and remove them in an unwind-protect.

That may be a good way to go. Probably, even refreshing agenda in a
separate copy of agenda buffer, so that the current version remains
usable. 

> The first thing that comes to mind is to set buffer-read-only on each
> buffer before it is scanned, and unset it when done with a buffer.  That
> might not be doable with advice.

This is not enough.
If you accidentally move the point in the buffer being processed by
agenda, the results may be unpredictable (org-agenda-get-* functions
move across the buffer with re-search-forward).

Any ideas how to deal with this?

Best,
Ihor


Adam Porter <adam@alphapapa.net> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>>>> Asynchronous code is not faster; it's generally slower because of
>>>> yielding and synchronization.
>>
>>> Anyway, I will try to throw yields into agenda code just to check how
>>> bad the performance can degrade.
>>
>> With the following code, org-agenda-redo runs for 21 second on my
>> system, while without threads it is 16 seconds. However, emacs remains
>> responsive during rebuilding agenda!
>>
>> (define-advice org-agenda-redo (:around (oldfun &optional all) make-async)
>>   (make-thread (lambda () (funcall oldfun all)) "org-agenda-redo"))
>> (define-advice org-agenda-skip-eval (:around (oldfun form) make-async)
>>   (thread-join (make-thread (lambda () (funcall oldfun form)) "org-agenda-skip-eval")))
>
> That's a very elegant way to add the threading!  The performance penalty
> is noticeable, but the tradeoff could be worth it in some cases, like a
> background agenda refresh on a timer, or after a "remote" edit.  I can
> imagine an org-agenda-refresh-async command that would add that advice
> and remove them in an unwind-protect.
>
>> The problem, of course, is that touching agenda buffer and org buffers
>> may be risky while org-agenda-redo is running.
>> Wondering if it is possible to block user commands during that time.
>
> The first thing that comes to mind is to set buffer-read-only on each
> buffer before it is scanned, and unset it when done with a buffer.  That
> might not be doable with advice.
>
>

Re: Asynchronous org-agenda-redo

[Adam Porter]

Ihor Radchenko <yantar92@gmail.com> writes:

> If you accidentally move the point in the buffer being processed by
> agenda, the results may be unpredictable (org-agenda-get-* functions
> move across the buffer with re-search-forward).

I'm afraid that's the basic problem with threading in Emacs: shared
state.  I don't know of a way to work around that.  I suppose you could
modify a lot of the agenda code to, e.g. store the position of the
current search in a variable and go to it before each search.  Of
course, that would also reduce performance to some extent.  It's a hard
problem.