Fun with async processes

Fun with async processes

[Manuel Giraud]

Hi,

Imagine, you have some async emacs processes with a sentinel each. What
is the common emacs lisp pattern to have some kind of sentinel for all
those processes (ie. do something when they all have finished)? I'd like
to avoid a busy waiting like in the following example:
--8<---------------cut here---------------start------------->8---
(defun chatty-sentinel (process event)
  (when (string-match "finished" event)
    (message "Chatty has finished talking.")
    (kill-buffer (process-buffer process))))

(defun chatty ()
  (let* ((buffer (generate-new-buffer "chatty"))
	 (process (start-process-shell-command
		   (buffer-name buffer) buffer
		   "find ~/.emacs.d -type f")))
    (when process
      (set-process-sentinel process #'chatty-sentinel)
      process)))

(defun thinker-sentinel (process event)
  (when (string-match "finished" event)
    (message "Thinker has finished thinking.")
    (kill-buffer (process-buffer process))))

(defun thinker ()
  (let* ((buffer (generate-new-buffer "thinker"))
	 (process (start-process-shell-command
		   (buffer-name buffer) buffer
		   "for i in $(jot 10); do (echo $i && sleep $i) done")))
    (when process
      (set-process-sentinel process #'thinker-sentinel)
      process)))

(defun myrun ()
  (interactive)
  (let ((processes (list (chatty) (thinker))))
    (while (cl-loop for process in processes
		    thereis (eq (process-status process) 'run))
      (sit-for 0.1))
    (message "Both have finished.")))
--8<---------------cut here---------------end--------------->8---

Cheers,
-- 
Manuel Giraud

Re: Fun with async processes

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1229 bytes --]

On Mon, Jan 31, 2022 at 04:44:36PM +0100, Manuel Giraud wrote:
> Hi,
> 
> Imagine, you have some async emacs processes with a sentinel each. What
> is the common emacs lisp pattern to have some kind of sentinel for all
> those processes (ie. do something when they all have finished)? I'd like
> to avoid a busy waiting like in the following example:
> --8<---------------cut here---------------start------------->8---
> (defun chatty-sentinel (process event)
>   (when (string-match "finished" event)
>     (message "Chatty has finished talking.")
>     (kill-buffer (process-buffer process))))

An obvious approach is to have each sentinel call a "global" sentinel
which checks whether there are any processes still running (akin to your
myrun, only that it just gets called after each process's termination).

Alternatively (this is typically the approach I take) is to have one
sentinel (taking an arg) for all processes of interest. Then I call a
closure with that one parameter set to distinguish among the "clients".

In general this is a pretty natural approach, since there's usually
quite a bit of "common code" for all the sentinels which gets repeated
senselessly otherwise.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Fun with async processes

[Emanuel Berg via Users list for the GNU Emacs text editor]

tomas wrote:

> An obvious approach is to have each sentinel call a "global"
> sentinel which checks whether there are any processes still
> running (akin to your myrun, only that it just gets called
> after each process's termination).

Won't that move the "busy wait" to the global sentinel?

Send signals? Yeah, heavy-handed ...

;;; -*- lexical-binding: t -*-
;;;
;;; this file:
;;;   http://user.it.uu.se/~embe8573/emacs-init/signal.el
;;;   https://dataswamp.org/~incal/emacs-init/signal.el
;;;
;;; test from Emacs:
;;;   (signal-process (emacs-pid) 'sigusr1)
;;;
;;; test from zsh:
;;;   $ kill -s usr1 $(pidof emacs)

(defun signal-usr1-f ()
  (interactive)
  (message "USR1 signal") )

(define-key special-event-map [sigusr1] #'signal-usr1-f)

-- 
underground experts united
https://dataswamp.org/~incal

Re: Fun with async processes

[tomas]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

On Mon, Jan 31, 2022 at 06:11:44PM +0100, Emanuel Berg via Users list for the GNU Emacs text editor wrote:
> tomas wrote:
> 
> > An obvious approach is to have each sentinel call a "global"
> > sentinel which checks whether there are any processes still
> > running (akin to your myrun, only that it just gets called
> > after each process's termination).
> 
> Won't that move the "busy wait" to the global sentinel?

No. Assume you are interested in 12 processes: that "global sentinel"
would be called 12 times. The busy wait, as envisioned, gets called
10 times /per second/.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Fun with async processes

[Emanuel Berg via Users list for the GNU Emacs text editor]

tomas wrote:

>>> An obvious approach is to have each sentinel call a "global"
>>> sentinel which checks whether there are any processes still
>>> running (akin to your myrun, only that it just gets called
>>> after each process's termination).
>> 
>> Won't that move the "busy wait" to the global sentinel?
>
> No. Assume you are interested in 12 processes: that "global
> sentinel" would be called 12 times. The busy wait, as
> envisioned, gets called 10 times /per second/.

A dedicated resource server/scheduler ... it _is_ busy, doing
it's intended thing :)

But this type of overhead is a very common solution so by all
means ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Fun with async processes

[Manuel Giraud]

<tomas@tuxteam.de> writes:

> An obvious approach is to have each sentinel call a "global" sentinel
> which checks whether there are any processes still running (akin to your
> myrun, only that it just gets called after each process's
> termination).

So I have to have an external variable that lists all the processes?

> Alternatively (this is typically the approach I take) is to have one
> sentinel (taking an arg) for all processes of interest. Then I call a
> closure with that one parameter set to distinguish among the
> "clients".

Do you have an example or somewhere to look at in emacs for this
approach? Thanks.
-- 
Manuel Giraud

Re: Fun with async processes

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2563 bytes --]

On Tue, Feb 01, 2022 at 09:54:24AM +0100, Manuel Giraud wrote:
> <tomas@tuxteam.de> writes:
> 
> > An obvious approach is to have each sentinel call a "global" sentinel
> > which checks whether there are any processes still running (akin to your
> > myrun, only that it just gets called after each process's
> > termination).
> 
> So I have to have an external variable that lists all the processes?

You have it already...

> (defun myrun ()
>  (interactive)
>   (let ((processes (list (chatty) (thinker))))

... here

> > Alternatively (this is typically the approach I take) is to have one
> > sentinel [...]

> Do you have an example or somewhere to look at in emacs for this
> approach? Thanks.

I haven't one handy, sorry. But to illustrate my proposal (caveat:
untested!), I'll munge your code in that direction (sorry for that):

(defun my-sentinel (tag process event)
  (when (string-match "finished" event)
    (cond
      ((eq tag 'chatty)
       (message "Chatty has finished talking."))
      ((eq tag 'thinker)
       (message "Thinker has finished thinking."))
      (t
       (message "Process %S has finished. I don't know that guy" tag)))
    (kill-buffer (process-buffer process))))

(defun chatty ()
  (let* ((buffer (generate-new-buffer "chatty"))
         (process (start-process-shell-command
                   (buffer-name buffer) buffer
                   "find ~/.emacs.d -type f")))
    (when process
      (set-process-sentinel process
        (lambda (process event) (my-sentinel 'chatty process event))
        process))))

(defun thinker ()
  (let* ((buffer (generate-new-buffer "thinker"))
         (process (start-process-shell-command
                   (buffer-name buffer) buffer
                   "for i in $(jot 10); do (echo $i && sleep $i) done")))
    (when process
      (set-process-sentinel process
        (lambda (process event) (my-sentinel 'thinker process event))
        process))))

Embellishments always possible, of course. For example, instead of
putting everywhere a (lambda (...) ...), you could build a function
`make-sentinel' taking a tag and returning a sentinel; Also, instead of
that `cond' which gets longer and longer, you could have a hash table
dispatching to the tag-specific code; Your `make-sentinel' could
populate that hash table.

If you (or your application) prefer static code to this (admittedly
highly) dynamic approach, you might consider translating all that to
the macro realm.

Lots of fun :)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Fun with async processes

[Manuel Giraud]

tomas@tuxteam.de writes:

> I haven't one handy, sorry. But to illustrate my proposal (caveat:
> untested!), I'll munge your code in that direction (sorry for that):

[...]

Thanks for this!… but your code is cleaner yet it is still missing the
"global" sentinel that says that both (chatty) and (thinker) have
finished. I've done this:
--8<---------------cut here---------------start------------->8---
(defun chatty-sentinel (process event)
  (when (string-match "finished" event)
    (message "Chatty has finished talking.")
    (kill-buffer (process-buffer process))
    (global-sentinel)))

(defun chatty ()
  (let* ((buffer (generate-new-buffer "chatty"))
	 (process (start-process-shell-command
		   (buffer-name buffer) buffer
		   "find ~/.emacs.d -type f")))
    (when process
      (set-process-sentinel process #'chatty-sentinel)
      process)))

(defun thinker-sentinel (process event)
  (when (string-match "finished" event)
    (message "Thinker has finished thinking.")
    (kill-buffer (process-buffer process))
    (global-sentinel)))

(defun thinker ()
  (let* ((buffer (generate-new-buffer "thinker"))
	 (process (start-process-shell-command
		   (buffer-name buffer) buffer
		   "for i in $(jot 10); do (echo $i && sleep $i) done")))
    (when process
      (set-process-sentinel process #'thinker-sentinel)
      process)))

(defvar count-processes 0)
(defvar mtx (make-mutex))

(defun global-sentinel ()
  (with-mutex mtx
    (decf count-processes))
  (when (zerop count-processes)
    (message "Both have finished.")))

(defun myrun ()
  (interactive)
  (setq count-processes (length (list (chatty) (thinker)))))
--8<---------------cut here---------------end--------------->8---

But I'd like to avoid the global defvar if possible.

> Lots of fun :)

Indeed :)
-- 
Manuel Giraud

Re: Fun with async processes

[tomas]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

On Tue, Feb 01, 2022 at 02:30:11PM +0100, Manuel Giraud wrote:
> tomas@tuxteam.de writes:
> 
> > I haven't one handy, sorry. But to illustrate my proposal (caveat:
> > untested!), I'll munge your code in that direction (sorry for that):
> 
> [...]
> 
> Thanks for this!… but your code is cleaner yet it is still missing the
> "global" sentinel that says that both (chatty) and (thinker) have
> finished.

Right. I forgot that one :)

>  I've done this:

[...]

Nice. Many ways to go about that.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Fun with async processes

[Manuel Giraud]

tomas@tuxteam.de writes:

[...]

> Nice. Many ways to go about that.

And do you think I could avoid having those defvar?

Cheers,
-- 
Manuel Giraud

Re: Fun with async processes

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]

On Tue, Feb 01, 2022 at 02:54:51PM +0100, Manuel Giraud wrote:
> tomas@tuxteam.de writes:
> 
> [...]
> 
> > Nice. Many ways to go about that.
> 
> And do you think I could avoid having those defvar?

How about hiding stuff in a closure? You'll need to have lexical
environments for that:

  (defun make-sentinel-factory ()
    (let (sentinels (list))
      (lambda (tag)
        (setq sentinels (push tag sentinels)) ; FIXME: fail if already there!
        (lambda (process event)
          (when (equal "finished" event)
            (setq sentinels (delq tag sentinels))
            (message "%S is done; still left: %S" tag sentinels)
            (when (null sentinels)
              (message "All sentinels done")))))))
  
  (setq make-sentinel (make-sentinel-factory))
  
  (setq s1 (funcall make-sentinel 's1))
  (setq s2 (funcall make-sentinel 's2))
  (setq s3 (funcall make-sentinel 's3))
  
  (funcall s2 "process" "finished")
  (funcall s3 "process" "finished")
  (funcall s1 "process" "finished")

Catching errors is left as an exercise to ...

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Fun with async processes

[Emanuel Berg via Users list for the GNU Emacs text editor]

Manuel Giraud wrote:

> (defvar count-processes 0)
> (defvar mtx (make-mutex))

Better, `defvar' then `setq' so you can reset them.

> (defun global-sentinel ()
>   (with-mutex mtx
>     (decf count-processes))
>   (when (zerop count-processes)
>     (message "Both have finished.")))
>
> (defun myrun ()
>   (interactive)
>   (setq count-processes (length (list (chatty) (thinker)))))
>
>
> But I'd like to avoid the global defvar if possible.

You can with lexical/static binding and a closure, see

  https://dataswamp.org/~incal/emacs-init/w3m/w3m-survivor.el

-- 
underground experts united
https://dataswamp.org/~incal