From mboxrd@z Thu Jan 1 00:00:00 1970 From: Diego Zamboni Subject: Asynchronous org-babel-tangle (was Re: Asynchronous org-agenda-redo) Date: Sun, 15 Dec 2019 12:56:08 +0100 Message-ID: References: <87k172ot2m.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> <87r219a8uu.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="0000000000006beab10599bcc80c" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:55870) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1igSVg-0005S0-NN for emacs-orgmode@gnu.org; Sun, 15 Dec 2019 06:56:31 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1igSVe-000646-BT for emacs-orgmode@gnu.org; Sun, 15 Dec 2019 06:56:28 -0500 Received: from mail-wr1-x42c.google.com ([2a00:1450:4864:20::42c]:35078) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1igSVc-0005tS-S2 for emacs-orgmode@gnu.org; Sun, 15 Dec 2019 06:56:25 -0500 Received: by mail-wr1-x42c.google.com with SMTP id g17so3806952wro.2 for ; Sun, 15 Dec 2019 03:56:22 -0800 (PST) In-Reply-To: <87r219a8uu.fsf@yantar92-laptop.i-did-not-set--mail-host-address--so-tickle-me> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: "Emacs-orgmode" To: Ihor Radchenko Cc: emacs-orgmode Mailinglist --0000000000006beab10599bcc80c Content-Type: text/plain; charset="UTF-8" Hi Ihor, On Thu, Dec 12, 2019 at 4:00 PM Ihor Radchenko wrote: > See the relevant code from my config below. Let me know if you have any > questions or suggestions. > Thank you, this was extremely helpful. I am now using a modified version of your code (see below) with the following comments/questions: - For me, the tangle works if I only load the org library in the sub-process, so I disabled the loading of my config. I guess this depends heavily and what other configuration one has loaded, but I think for tangling, not much else is needed :) - In you code, the process-list variable was not actually being modified since plist-put returns the new value but does not modify the variable. I tried a few variations wrapping the plist-put in (setq ...), and adding corresponding code to clear the element after the tangle was finished, but could not get it to work. As this is not really critical for me, I just removed the whole process-checking code, significantly simplifying the rest. - I wanted my code to report the time the tangle took. This works well by passing the time as the result of the initial lambda. Interestingly, it works if I return the time already formatted as a string (as in my code) but not if I return the result of (float-time ..) directly. Here's my code: #+begin_src emacs-lisp (defun zz/org-babel-tangle-async (file) "Invoke `org-babel-tangle-file' asynchronously." (message "Tangling %s..." (buffer-file-name)) (async-start (let ((args (list file))) `(lambda () (require 'org) ;;(load "~/.emacs.d/init.el") (let ((start-time (current-time))) (apply #'org-babel-tangle-file ',args) (format "%.2f" (float-time (time-since start-time)))))) (let ((message-string (format "Tangling %S completed after " file))) `(lambda (tangle-time) (message (concat ,message-string (format "%s seconds" tangle-time))))))) (defun zz/org-babel-tangle-current-buffer-async () "Tangle current buffer asynchronously." (zz/org-babel-tangle-async (buffer-file-name))) #+end_src Thanks again for your help, and for the inspiration to finally get this done, it had been bugging me for a while :) --Diego > Also, my config is written in the way that everything related to user > interaction can be disabled if I load emacs for tangling > (with org-tangle-flag). This is to speed up the async process. > > #+begin_src emacs-lisp > (defvar yant/org-babel-tangle-async-process-list nil > "Plist of (file . process) for all the currently running async tangle > processes.") > > > (defun yant/org-babel-tangle-async (file &optional target-file lang) > "Invoke `org-babel-tangle-file' asynchronously." > (require 'async) > (let ((oldproc (plist-get yant/org-babel-tangle-async-process-list > file))) > (when (or (not oldproc) > (async-wait oldproc)) > (message "Tangling %s..." (buffer-file-name)) > (plist-put yant/org-babel-tangle-async-process-list > file > (async-start > (let ((args (list file target-file lang))) > `(lambda () > (require 'org) > (setq org-tangle-flag t) > (load "~/.emacs.d/config.el") > (apply #'org-babel-tangle-file ',args))) > (let ((message-string (format "Tangling (%S %S %S) > completed." file target-file lang))) > `(lambda (result) (message ,message-string)))))))) > > (defvar yant/auto-tangle-list nil > "List of files, which can be safely tangled on save. > The list is saved between Emacs sessions.") > > (when init-flag > (use-package savehist > :config > (add-to-list 'savehist-additional-variables 'yant/auto-tangle-list)) > (savehist-mode +1) > (defun yant/toggle-buffer-auto-tangle (arg) > "Toggle auto tangling of a buffer." > (interactive "P") > (if (not (eq major-mode 'org-mode)) > (message "Org-mode is not active in buffer \"%s\"" (buffer-name)) > (cond ((not arg) > (if (member (buffer-file-name) yant/auto-tangle-list) > (progn (setq yant/auto-tangle-list (delete > (buffer-file-name) yant/auto-tangle-list)) > (message "Auto tangling disabled for %s" > (buffer-file-name))) > (add-to-list 'yant/auto-tangle-list (buffer-file-name)) > (message "Auto tangling enabled for %s" > (buffer-file-name)))) > ((or (and (not (listp arg)) (> arg 0)) > (equal arg '(4))) > (add-to-list 'yant/auto-tangle-list (buffer-file-name)) > (message "Auto tangling enabled for %s" (buffer-file-name))) > (t > (setq yant/auto-tangle-list (delete (buffer-file-name) > yant/auto-tangle-list)) > (message "Auto tangling disabled for %s" > (buffer-file-name)))))) > > (bind-key "C-c C-*" #'yant/toggle-buffer-auto-tangle org-mode-map)) > > (defun yant/org-babel-tangle-current-buffer-async () > "Tangle current buffer asynchronously." > (when (and (eq major-mode 'org-mode) > (member (buffer-file-name) yant/auto-tangle-list)) > (yant/org-babel-tangle-async (buffer-file-name)))) > > (add-hook 'after-save-hook #'yant/org-babel-tangle-current-buffer-async) > #+end_src > > > Diego Zamboni writes: > > > Hi Ihor, > > > > I cannot answer your question, but I am curious about using async > together > > with tangling, since for some of my buffers, tangling takes some time and > > freezes Emacs in the process. Do you have some examples of this that you > > could share? > > > > Thanks, > > --Diego > > > > > > On Thu, Dec 12, 2019 at 9:21 AM Ihor Radchenko > wrote: > > > >> I am thinking if it is possible to implement org-agenda-redo > >> asynchronously. > >> > >> Rebuilding agenda should normally not affect any buffer except agenda > >> buffer. So, it should be sufficient to block any agenda modifying > >> commands in the agenda buffer, redo the agenda buffer in separate > >> thread, and replace the old agenda with the calculated one. > >> Then, emacs should remain responsive while updating agenda (except for > >> modifying the agenda buffer). > >> > >> For example, this naive code kind of works (forgetting that buffer-local > >> variables will not be passed to the thread): > >> > >> (define-advice org-agenda-redo (:around (oldfun &optional all) > make-async) > >> (make-thread oldfun "org-agenda-redo")) > >> > >> The problem is that emacs does not become responsive... > >> > >> Another approach would be using async.el package, which allows calling > >> arbitrary function in subordinate emacs process. Then, the main emacs > >> instance should not be "frozen" (I use same approach for tangling and it > >> works fine). > >> > >> However, the question is how to pass the .org and agenda buffers to this > >> subordinate process. Opening the .org files there is not a good option > >> since it would give too much overhead to this asynchronous agenda. > >> > >> Any suggestions? Alternative ideas? > >> > >> Best, > >> Ihor > >> > >> > >> > > -- > Ihor Radchenko, > PhD, > Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) > State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong > University, Xi'an, China > Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg > --0000000000006beab10599bcc80c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi=C2=A0Ihor,

On Thu, Dec 12, 2019 at 4:00 PM Ihor Rad= chenko <yantar92@gmail.com>= wrote:
See the relevant code from my config below. Let me know if you have any
questions or suggestions.

Thank you, th= is was extremely helpful. I am now using a modified version of your code (s= ee below) with the following comments/questions:

-= For me, the tangle works if I only load the org library in the sub-process= , so I disabled the loading of my config. I guess this depends heavily and = what other configuration one has loaded, but I think for tangling, not much= else is needed :)

- In you code, the process-list= variable was not actually being modified since plist-put returns the new v= alue but does not modify the variable. I tried a few variations wrapping th= e plist-put in (setq ...), and adding corresponding code to clear the eleme= nt after the tangle was finished, but could not get it to work. As this is = not really critical for me, I just removed the whole process-checking code,= significantly simplifying the rest.

- I wanted my= code to report the time the tangle took. This works well by passing the ti= me as the result of the initial lambda. Interestingly, it works if I return= the time already formatted as a string (as in my code) but not if I return= the result of (float-time ..) directly.

Here'= s my code:

=C2=A0 #+begin_src emacs-lisp
=C2=A0= =C2=A0 (defun zz/org-babel-tangle-async (file)
=C2=A0 =C2=A0 =C2=A0 &qu= ot;Invoke `org-babel-tangle-file' asynchronously."
=C2=A0 =C2= =A0 =C2=A0 (message "Tangling %s..." (buffer-file-name))
=C2= =A0 =C2=A0 =C2=A0 (async-start
=C2=A0 =C2=A0 =C2=A0 =C2=A0(let ((args (l= ist file)))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0`(lambda ()
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (require 'org)
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 ;;(load "~/.emacs.d/init.el")
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (let ((start-time (current-time)))
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (apply #'org-babel-tangle= -file ',args)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (form= at "%.2f" (float-time (time-since start-time))))))
=C2=A0 =C2= =A0 =C2=A0 =C2=A0(let ((message-string (format "Tangling %S completed = after " file)))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0`(lambda (tangle-= time)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (message (concat ,messag= e-string
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(format "%s seconds" tan= gle-time)))))))

=C2=A0 =C2=A0 (defun zz/org-babel-tangle-current-buf= fer-async ()
=C2=A0 =C2=A0 =C2=A0 "Tangle current buffer asynchrono= usly."
=C2=A0 =C2=A0 =C2=A0 (zz/org-babel-tangle-async (buffer-file= -name)))
=C2=A0 =C2=A0 #+end_src

Thanks again for your= help, and for the inspiration to finally get this done, it had been buggin= g me for a while :)

--Diego



Also, my config is written in the way that everything related to user
interaction can be disabled if I load emacs for tangling
(with org-tangle-flag). This is to speed up the async process.=C2=A0

#+begin_src emacs-lisp
(defvar yant/org-babel-tangle-async-process-list nil
=C2=A0 "Plist of (file . process) for all the currently running async = tangle processes.")


(defun yant/org-babel-tangle-async (file &optional target-file lang) =C2=A0 "Invoke `org-babel-tangle-file' asynchronously."
=C2=A0 (require 'async)
=C2=A0 (let ((oldproc (plist-get yant/org-babel-tangle-async-process-list f= ile)))
=C2=A0 =C2=A0 (when (or (not oldproc)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (async-wait oldproc))
=C2=A0 =C2=A0 =C2=A0 (message "Tangling %s..." (buffer-file-name)= )
=C2=A0 =C2=A0 =C2=A0 (plist-put yant/org-babel-tangle-async-process-list =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0file
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(async-start<= br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (let ((args = (list file target-file lang)))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 `(lam= bda ()
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0(require 'org)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0(setq org-tangle-flag t)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0(load "~/.emacs.d/config.el")
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0(apply #'org-babel-tangle-file ',args)))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (let ((messa= ge-string (format "Tangling (%S %S %S) completed." file target-fi= le lang)))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 `(lam= bda (result) (message ,message-string))))))))

(defvar yant/auto-tangle-list nil
=C2=A0 "List of files, which can be safely tangled on save.
The list is saved between Emacs sessions.")

(when init-flag
=C2=A0 (use-package savehist
=C2=A0 =C2=A0 :config
=C2=A0 =C2=A0 (add-to-list 'savehist-additional-variables 'yant/aut= o-tangle-list))
=C2=A0 (savehist-mode +1)
=C2=A0 (defun yant/toggle-buffer-auto-tangle (arg)
=C2=A0 =C2=A0 "Toggle auto tangling of a buffer."
=C2=A0 =C2=A0 (interactive "P")
=C2=A0 =C2=A0 (if (not (eq major-mode 'org-mode))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (message "Org-mode is not active in buffer= \"%s\"" (buffer-name))
=C2=A0 =C2=A0 =C2=A0 (cond ((not arg)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(if (member (buffer-file-na= me) yant/auto-tangle-list)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(progn (setq = yant/auto-tangle-list (delete (buffer-file-name) yant/auto-tangle-list)) =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 (message "Auto tangling disabled for %s" (buffer-file-= name)))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(add-to-list 'ya= nt/auto-tangle-list (buffer-file-name))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(message "Auto = tangling enabled for %s" (buffer-file-name))))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ((or (and (not (listp arg)) (>= arg 0))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(equal arg &#= 39;(4)))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(add-to-list 'yant/auto= -tangle-list (buffer-file-name))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(message "Auto tanglin= g enabled for %s" (buffer-file-name)))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (t
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setq yant/auto-tangle-list= (delete (buffer-file-name) yant/auto-tangle-list))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(message "Auto tanglin= g disabled for %s" (buffer-file-name))))))

=C2=A0 (bind-key "C-c C-*" #'yant/toggle-buffer-auto-tangle o= rg-mode-map))

(defun yant/org-babel-tangle-current-buffer-async ()
=C2=A0 "Tangle current buffer asynchronously."
=C2=A0 (when (and (eq major-mode 'org-mode)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(member (buffer-file-name) = yant/auto-tangle-list))
=C2=A0 =C2=A0 (yant/org-babel-tangle-async (buffer-file-name))))

(add-hook 'after-save-hook #'yant/org-babel-tangle-current-buffer-a= sync)
#+end_src


Diego Zamboni <d= iego@zzamboni.org> writes:

> Hi Ihor,
>
> I cannot answer your question, but I am curious about using async toge= ther
> with tangling, since for some of my buffers, tangling takes some time = and
> freezes Emacs in the process. Do you have some examples of this that y= ou
> could share?
>
> Thanks,
> --Diego
>
>
> On Thu, Dec 12, 2019 at 9:21 AM Ihor Radchenko <yantar92@gmail.com> wrote:
>
>> I am thinking if it is possible to implement org-agenda-redo
>> asynchronously.
>>
>> Rebuilding agenda should normally not affect any buffer except age= nda
>> buffer. So, it should be sufficient to block any agenda modifying<= br> >> commands in the agenda buffer, redo the agenda buffer in separate<= br> >> thread, and replace the old agenda with the calculated one.
>> Then, emacs should remain responsive while updating agenda (except= for
>> modifying the agenda buffer).
>>
>> For example, this naive code kind of works (forgetting that buffer= -local
>> variables will not be passed to the thread):
>>
>> (define-advice org-agenda-redo (:around (oldfun &optional all)= make-async)
>>=C2=A0 =C2=A0(make-thread oldfun "org-agenda-redo"))
>>
>> The problem is that emacs does not become responsive...
>>
>> Another approach would be using async.el package, which allows cal= ling
>> arbitrary function in subordinate emacs process. Then, the main em= acs
>> instance should not be "frozen" (I use same approach for= tangling and it
>> works fine).
>>
>> However, the question is how to pass the .org and agenda buffers t= o this
>> subordinate process. Opening the .org files there is not a good op= tion
>> since it would give too much overhead to this asynchronous agenda.=
>>
>> Any suggestions? Alternative ideas?
>>
>> Best,
>> Ihor
>>
>>
>>

--
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaoto= ng University, Xi'an, China
Email: yantar92@gma= il.com, ihor_radchenko@alumni.sutd.edu.sg
--0000000000006beab10599bcc80c--