Re: Asynchronous org-babel-tangle (was Re: Asynchronous org-agenda-redo)

[Ihor Radchenko <yantar92@gmail.com>]

Thanks for the feedback.

> - For me, the tangle works if I only load the org library in the
> sub-process, so I disabled the loading of my config. I guess this depends
> heavily and what other configuration one has loaded, but I think for
> tangling, not much else is needed :)

Note that your version tangling may not work as expected if your
init file changes any default headers args affecting the tangling.
For example, I set (:comments . "link") in my init file. Not loading
init file would disallow detangling in my case. 

> - In you code, the process-list variable was not actually being modified
> since plist-put returns the new value but does not modify the
> variable.

(facepalm)

> I
> tried a few variations wrapping the plist-put in (setq ...), and adding
> corresponding code to clear the element after the tangle was finished, but
> could not get it to work. 

Well. Things did not work because plist-put compares keys by #'eq, which
returns nil on equal strings.

> - I wanted my code to report the time the tangle took. This works well by
> passing the time as the result of the initial lambda. Interestingly, it
> works if I return the time already formatted as a string (as in my code)
> but not if I return the result of (float-time ..) directly.

Thanks for this neat idea. Will also grab it to my config.

float-time did not work because of bug in async.el. It uses
backward-sexp to find the returned value of the lambda (in the process
buffer). But backwards-sexp fails on float (it moves point just after
the point).  

Best,
Ihor


Diego Zamboni <diego@zzamboni.org> writes:

> Hi Ihor,
>
> On Thu, Dec 12, 2019 at 4:00 PM Ihor Radchenko <yantar92@gmail.com> wrote:
>
>> See the relevant code from my config below. Let me know if you have any
>> questions or suggestions.
>>
>
> Thank you, this was extremely helpful. I am now using a modified version of
> your code (see below) with the following comments/questions:
>
> - For me, the tangle works if I only load the org library in the
> sub-process, so I disabled the loading of my config. I guess this depends
> heavily and what other configuration one has loaded, but I think for
> tangling, not much else is needed :)
>
> - In you code, the process-list variable was not actually being modified
> since plist-put returns the new value but does not modify the variable. I
> tried a few variations wrapping the plist-put in (setq ...), and adding
> corresponding code to clear the element after the tangle was finished, but
> could not get it to work. As this is not really critical for me, I just
> removed the whole process-checking code, significantly simplifying the rest.
>
> - I wanted my code to report the time the tangle took. This works well by
> passing the time as the result of the initial lambda. Interestingly, it
> works if I return the time already formatted as a string (as in my code)
> but not if I return the result of (float-time ..) directly.
>
> Here's my code:
>
>   #+begin_src emacs-lisp
>     (defun zz/org-babel-tangle-async (file)
>       "Invoke `org-babel-tangle-file' asynchronously."
>       (message "Tangling %s..." (buffer-file-name))
>       (async-start
>        (let ((args (list file)))
>          `(lambda ()
>             (require 'org)
>             ;;(load "~/.emacs.d/init.el")
>             (let ((start-time (current-time)))
>               (apply #'org-babel-tangle-file ',args)
>               (format "%.2f" (float-time (time-since start-time))))))
>        (let ((message-string (format "Tangling %S completed after " file)))
>          `(lambda (tangle-time)
>             (message (concat ,message-string
>                              (format "%s seconds" tangle-time)))))))
>
>     (defun zz/org-babel-tangle-current-buffer-async ()
>       "Tangle current buffer asynchronously."
>       (zz/org-babel-tangle-async (buffer-file-name)))
>     #+end_src
>
> Thanks again for your help, and for the inspiration to finally get this
> done, it had been bugging me for a while :)
>
> --Diego
>
>
>
>> Also, my config is written in the way that everything related to user
>> interaction can be disabled if I load emacs for tangling
>> (with org-tangle-flag). This is to speed up the async process.
>>
>> #+begin_src emacs-lisp
>> (defvar yant/org-babel-tangle-async-process-list nil
>>   "Plist of (file . process) for all the currently running async tangle
>> processes.")
>>
>>
>> (defun yant/org-babel-tangle-async (file &optional target-file lang)
>>   "Invoke `org-babel-tangle-file' asynchronously."
>>   (require 'async)
>>   (let ((oldproc (plist-get yant/org-babel-tangle-async-process-list
>> file)))
>>     (when (or (not oldproc)
>>               (async-wait oldproc))
>>       (message "Tangling %s..." (buffer-file-name))
>>       (plist-put yant/org-babel-tangle-async-process-list
>>                  file
>>                  (async-start
>>                   (let ((args (list file target-file lang)))
>>                     `(lambda ()
>>                        (require 'org)
>>                        (setq org-tangle-flag t)
>>                        (load "~/.emacs.d/config.el")
>>                        (apply #'org-babel-tangle-file ',args)))
>>                   (let ((message-string (format "Tangling (%S %S %S)
>> completed." file target-file lang)))
>>                     `(lambda (result) (message ,message-string))))))))
>>
>> (defvar yant/auto-tangle-list nil
>>   "List of files, which can be safely tangled on save.
>> The list is saved between Emacs sessions.")
>>
>> (when init-flag
>>   (use-package savehist
>>     :config
>>     (add-to-list 'savehist-additional-variables 'yant/auto-tangle-list))
>>   (savehist-mode +1)
>>   (defun yant/toggle-buffer-auto-tangle (arg)
>>     "Toggle auto tangling of a buffer."
>>     (interactive "P")
>>     (if (not (eq major-mode 'org-mode))
>>         (message "Org-mode is not active in buffer \"%s\"" (buffer-name))
>>       (cond ((not arg)
>>              (if (member (buffer-file-name) yant/auto-tangle-list)
>>                  (progn (setq yant/auto-tangle-list (delete
>> (buffer-file-name) yant/auto-tangle-list))
>>                         (message "Auto tangling disabled for %s"
>> (buffer-file-name)))
>>                (add-to-list 'yant/auto-tangle-list (buffer-file-name))
>>                (message "Auto tangling enabled for %s"
>> (buffer-file-name))))
>>             ((or (and (not (listp arg)) (> arg 0))
>>                  (equal arg '(4)))
>>              (add-to-list 'yant/auto-tangle-list (buffer-file-name))
>>              (message "Auto tangling enabled for %s" (buffer-file-name)))
>>             (t
>>              (setq yant/auto-tangle-list (delete (buffer-file-name)
>> yant/auto-tangle-list))
>>              (message "Auto tangling disabled for %s"
>> (buffer-file-name))))))
>>
>>   (bind-key "C-c C-*" #'yant/toggle-buffer-auto-tangle org-mode-map))
>>
>> (defun yant/org-babel-tangle-current-buffer-async ()
>>   "Tangle current buffer asynchronously."
>>   (when (and (eq major-mode 'org-mode)
>>              (member (buffer-file-name) yant/auto-tangle-list))
>>     (yant/org-babel-tangle-async (buffer-file-name))))
>>
>> (add-hook 'after-save-hook #'yant/org-babel-tangle-current-buffer-async)
>> #+end_src
>>
>>
>> Diego Zamboni <diego@zzamboni.org> writes:
>>
>> > Hi Ihor,
>> >
>> > I cannot answer your question, but I am curious about using async
>> together
>> > with tangling, since for some of my buffers, tangling takes some time and
>> > freezes Emacs in the process. Do you have some examples of this that you
>> > could share?
>> >
>> > Thanks,
>> > --Diego
>> >
>> >
>> > On Thu, Dec 12, 2019 at 9:21 AM Ihor Radchenko <yantar92@gmail.com>
>> wrote:
>> >
>> >> I am thinking if it is possible to implement org-agenda-redo
>> >> asynchronously.
>> >>
>> >> Rebuilding agenda should normally not affect any buffer except agenda
>> >> buffer. So, it should be sufficient to block any agenda modifying
>> >> commands in the agenda buffer, redo the agenda buffer in separate
>> >> thread, and replace the old agenda with the calculated one.
>> >> Then, emacs should remain responsive while updating agenda (except for
>> >> modifying the agenda buffer).
>> >>
>> >> For example, this naive code kind of works (forgetting that buffer-local
>> >> variables will not be passed to the thread):
>> >>
>> >> (define-advice org-agenda-redo (:around (oldfun &optional all)
>> make-async)
>> >>   (make-thread oldfun "org-agenda-redo"))
>> >>
>> >> The problem is that emacs does not become responsive...
>> >>
>> >> Another approach would be using async.el package, which allows calling
>> >> arbitrary function in subordinate emacs process. Then, the main emacs
>> >> instance should not be "frozen" (I use same approach for tangling and it
>> >> works fine).
>> >>
>> >> However, the question is how to pass the .org and agenda buffers to this
>> >> subordinate process. Opening the .org files there is not a good option
>> >> since it would give too much overhead to this asynchronous agenda.
>> >>
>> >> Any suggestions? Alternative ideas?
>> >>
>> >> Best,
>> >> Ihor
>> >>
>> >>
>> >>
>>
>> --
>> Ihor Radchenko,
>> PhD,
>> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
>> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong
>> University, Xi'an, China
>> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg
>>