bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel jobs finish

bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel jobs finish

[Tino Calancha]

[-- Attachment #1: Type: text/plain, Size: 652 bytes --]


Good (or morning afternoon evening night)

When `dired-do-shell-command' runs COMMAND over FILES in
parallel, the `shell-command' process may finish before all the
files have being processed.

emacs -Q ~

* / & du SPC -s RET
; If the number of lines in #<buffer *Async Shell Command*> is less
; than (1+ (length (dired-get-marked-files))) then NOT all files
; has being processed.

Attached patch fix this issue adding 'wait' at the end of COMMAND
when FILES are procesed in parallel.

In GNU Emacs 25.0.92.1 (x86_64-pc-linux-gnu, GTK+ Version 2.24.30)
  of 2016-04-01 built on calancha-pc
Repository revision: a3daa34336da158555d96d670077eedb9eaeeb9c

[-- Attachment #2: Type: text/plain, Size: 3537 bytes --]

a3daa34336da158555d96d670077eedb9eaeeb9cdiff --git a/lisp/dired-aux.el b/lisp/dired-aux.el
index d29abf3..f351d79 100644
--- a/lisp/dired-aux.el
+++ b/lisp/dired-aux.el
@@ -722,33 +722,43 @@ dired-shell-stuff-it
 ;; (coming from interactive P and currently ignored) to decide what to do.
 ;; Smart would be a way to access basename or extension of file names.
   (let* ((in-background (string-match "[ \t]*&[ \t]*\\'" command))
-	 (command (if in-background
-		      (substring command 0 (match-beginning 0))
-		    command))
-	 (sequentially (string-match "[ \t]*;[ \t]*\\'" command))
-	 (command (if sequentially
-		      (substring command 0 (match-beginning 0))
-		    command))
-	 (stuff-it
-	  (if (or (string-match-p dired-star-subst-regexp command)
-		  (string-match-p dired-quark-subst-regexp command))
-	      (lambda (x)
-		(let ((retval command))
-		  (while (string-match
-			  "\\(^\\|[ \t]\\)\\([*?]\\)\\([ \t]\\|$\\)" retval)
-		    (setq retval (replace-match x t t retval 2)))
-		  retval))
-	    (lambda (x) (concat command dired-mark-separator x)))))
+         (command (if in-background
+                      (let ((cmd command))
+                        (while (string-match "[ \t]*[&]+[ \t]*\\'" cmd)
+                          (setq cmd (substring cmd 0 (match-beginning 0)))) cmd)
+                    command))
+         (sequentially (string-match "[ \t]*;[ \t]*\\'" command))
+         (command (if sequentially
+                      (let ((cmd command))
+                        (while (string-match "[ \t]*[;]+[ \t]*\\'" cmd)
+                          (setq cmd (substring cmd 0 (match-beginning 0)))) cmd)
+                    command))
+         (in-back-no-seq (and in-background (not sequentially)))
+         (stuff-it
+          (if (or (string-match-p dired-star-subst-regexp command)
+                  (string-match-p dired-quark-subst-regexp command))
+              (lambda (x)
+                (let ((retval command))
+                  (while (string-match
+                          "\\(^\\|[ \t]\\)\\([*?]\\)\\([ \t]\\|$\\)" retval)
+                    (setq retval (replace-match x t t retval 2)))
+                  retval))
+            (lambda (x) (concat command dired-mark-separator x)))))
     (concat
-     (if on-each
-	 (mapconcat stuff-it (mapcar 'shell-quote-argument file-list)
-		    (if (and in-background (not sequentially)) "&" ";"))
-       (let ((files (mapconcat 'shell-quote-argument
-			       file-list dired-mark-separator)))
-	 (if (> (length file-list) 1)
-	     (setq files (concat dired-mark-prefix files dired-mark-postfix)))
-	 (funcall stuff-it files)))
-     (if in-background "&" ""))))
+     (cond (on-each
+            (format "%s%s"
+                    (mapconcat stuff-it (mapcar 'shell-quote-argument file-list)
+                               (or (and in-back-no-seq "&")
+                                   ";"))
+                    (or (and in-back-no-seq "&\nwait"); Wait for background jobs finish.
+                        "")))
+           (t
+            (let ((files (mapconcat 'shell-quote-argument
+                                    file-list dired-mark-separator)))
+              (when (cdr file-list)
+                (setq files (concat dired-mark-prefix files dired-mark-postfix)))
+              (funcall stuff-it files))))
+     (or (and in-background "&") ""))))
 
 ;; This is an extra function so that it can be redefined by ange-ftp.
 ;;;###autoload

bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel jobs finish

[Eli Zaretskii]

> Date: Sun, 3 Apr 2016 22:02:17 +0900 (JST)
> From: Tino Calancha <f92capac@gmail.com>
> 
> When `dired-do-shell-command' runs COMMAND over FILES in
> parallel, the `shell-command' process may finish before all the
> files have being processed.
> 
> emacs -Q ~
> 
> * / & du SPC -s RET
> ; If the number of lines in #<buffer *Async Shell Command*> is less
> ; than (1+ (length (dired-get-marked-files))) then NOT all files
> ; has being processed.
> 
> Attached patch fix this issue adding 'wait' at the end of COMMAND
> when FILES are procesed in parallel.

Thanks.  Can you elaborate on why such a wait is needed?

bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel jobs finish

[Tino Calancha]

> Thanks.  Can you elaborate on why such a wait is needed?

Let's assume COMMAND is "du -s"
and LIST contains the dirs foo bar baz:
foo is ~ Gb  with depth > 1
and bar and baz are few kB dirs with depth 1.

The effective shell command to run is:
du -s foo& du -s bar& du -s baz&

This creates a new process PROC in the system.
PROC returns with the return code of the last command in the list
(du -s baz):  this is a shell feature.
The output from the remaining jobs is still comming to stdout,
but is not associated to PROC, which already succeded.

With `dired-do-shell-command', we get all output only if foo
is the last dir in LIST.

POSIX shells provide the builtin 'wait' to change this behaviour:
it forces PROC to wait until all background jobs end.

You can run from terminal following scripts:
I)
#!/bin/sh
echo "id $$"
du -s foo& du -s bar& du -s baz&

II)
#!/bin/sh
echo "id $$"
du -s foo& du -s bar& du -s baz&
wait

Then list the process id of the scripts while you still have
output comming:
ps -f -p ID

I) No process in the system with ID.
II) There is a process with ID.


Well, maybe it can be solve in many other ways.  I just found
very simple: just adding 4 letters fix it; also very natural,
because we are building a shell command, so its logical to use the
shell sintaxis.

bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel jobs finish

[Tino Calancha]

I must clarify the tittle of this thread (the word wait).
Of course, the process is still asynchronous! :-)

The command wait just avoid `shell-command-sentinel' be triggered
too early.  Once all jobs in the background finish, the shell
returns with the return code of the last command in the list; then
`shell-command-sentinel' do its stuff.

The command 'wait' is specified by posix.
I have verified that it is present in following shells:

bash: 4.3, 3.2
dash: 0.5.8-2.2
posh: 0.12.6
bsd-csh: 1.65 (20110502-2.1)
tcsh: 6.18, 6.14
ksh: 93u+ 2012-08-01,  20100202-1
zsh: 5.2, 4.2

bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel jobs finish

[Eli Zaretskii]

> Date: Mon, 4 Apr 2016 01:38:52 +0900 (JST)
> From: Tino Calancha <f92capac@gmail.com>
> cc: Tino Calancha <f92capac@gmail.com>, 23206@debbugs.gnu.org
> 
> Let's assume COMMAND is "du -s"
> and LIST contains the dirs foo bar baz:
> foo is ~ Gb  with depth > 1
> and bar and baz are few kB dirs with depth 1.
> 
> The effective shell command to run is:
> du -s foo& du -s bar& du -s baz&
> 
> This creates a new process PROC in the system.
> PROC returns with the return code of the last command in the list
> (du -s baz):  this is a shell feature.
> The output from the remaining jobs is still comming to stdout,
> but is not associated to PROC, which already succeded.
> 
> With `dired-do-shell-command', we get all output only if foo
> is the last dir in LIST.
> 
> POSIX shells provide the builtin 'wait' to change this behaviour:
> it forces PROC to wait until all background jobs end.

Thanks.  This means the fix should only affect systems with a Posix
shell.  The stock MS-Windows shells don't have a 'wait' command (and
don't need this fix in the first place, as the problem doesn't exist
on MS-Windows, AFAICS).

Can you modify the patch to do that?

Thanks.

bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel jobs finish

[Tino Calancha]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]

>This means the fix should only affect systems with a Posix
>shell.
>The stock MS-Windows shells don't have a 'wait' command (and
>don't need this fix in the first place, as the problem doesn't exist
>on MS-Windows, AFAICS).
I see.  Thank you.
I rewrote the patch so that it adds 'wait' when 'system-type' is
not in '(ms-dos windows-nt).

[-- Attachment #2: Type: text/plain, Size: 4627 bytes --]

From 5d4b99d84b1b329d7a0047b9ff00be94dd58e18c Mon Sep 17 00:00:00 2001
From: Tino Calancha <f92capac@gmail.com>
Date: Tue, 5 Apr 2016 14:08:57 +0900
Subject: [PATCH] Wait until all parallel shell commands finish

* lisp/dired-aux.el (dired-shell-stuff-it):
Drop all trailing ';' and '&' in command;
change indentation with \t to ?\s;
Force POSIX shells to wait until all background jobs finish.
(Bug#23206).
---
 lisp/dired-aux.el | 69 ++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 43 insertions(+), 26 deletions(-)

diff --git a/lisp/dired-aux.el b/lisp/dired-aux.el
index d29abf3..f5106bb 100644
--- a/lisp/dired-aux.el
+++ b/lisp/dired-aux.el
@@ -722,33 +722,50 @@ dired-shell-stuff-it
 ;; (coming from interactive P and currently ignored) to decide what to do.
 ;; Smart would be a way to access basename or extension of file names.
   (let* ((in-background (string-match "[ \t]*&[ \t]*\\'" command))
-	 (command (if in-background
-		      (substring command 0 (match-beginning 0))
-		    command))
-	 (sequentially (string-match "[ \t]*;[ \t]*\\'" command))
-	 (command (if sequentially
-		      (substring command 0 (match-beginning 0))
-		    command))
-	 (stuff-it
-	  (if (or (string-match-p dired-star-subst-regexp command)
-		  (string-match-p dired-quark-subst-regexp command))
-	      (lambda (x)
-		(let ((retval command))
-		  (while (string-match
-			  "\\(^\\|[ \t]\\)\\([*?]\\)\\([ \t]\\|$\\)" retval)
-		    (setq retval (replace-match x t t retval 2)))
-		  retval))
-	    (lambda (x) (concat command dired-mark-separator x)))))
+         (command (if in-background
+                      (let ((cmd command))
+                        (while (string-match "[ \t]*[&]+[ \t]*\\'" cmd)
+                          (setq cmd (substring cmd 0 (match-beginning 0)))) cmd)
+                    command))
+         (sequentially (string-match "[ \t]*;[ \t]*\\'" command))
+         (command (if sequentially
+                      (let ((cmd command))
+                        (while (string-match "[ \t]*[;]+[ \t]*\\'" cmd)
+                          (setq cmd (substring cmd 0 (match-beginning 0)))) cmd)
+                    command))
+         (in-back-no-seq (and in-background (not sequentially)))
+         (stuff-it
+          (if (or (string-match-p dired-star-subst-regexp command)
+                  (string-match-p dired-quark-subst-regexp command))
+              (lambda (x)
+                (let ((retval command))
+                  (while (string-match
+                          "\\(^\\|[ \t]\\)\\([*?]\\)\\([ \t]\\|$\\)" retval)
+                    (setq retval (replace-match x t t retval 2)))
+                  retval))
+            (lambda (x) (concat command dired-mark-separator x)))))
     (concat
-     (if on-each
-	 (mapconcat stuff-it (mapcar 'shell-quote-argument file-list)
-		    (if (and in-background (not sequentially)) "&" ";"))
-       (let ((files (mapconcat 'shell-quote-argument
-			       file-list dired-mark-separator)))
-	 (if (> (length file-list) 1)
-	     (setq files (concat dired-mark-prefix files dired-mark-postfix)))
-	 (funcall stuff-it files)))
-     (if in-background "&" ""))))
+     (cond (on-each
+            (format "%s%s"
+                    (mapconcat stuff-it (mapcar 'shell-quote-argument file-list)
+                               (or (and in-back-no-seq "&") ";"))
+                    ;; POSIX shells running a list of commands in the background
+                    ;; (LIST = cmd_1 & [cmd_2 & ... cmd_i & ... cmd_N &])
+                    ;; return once cmd_N ends, i.e., the shell does not
+                    ;; wait for cmd_i to finish before executing cmd_i+1.
+                    ;; That means, running (shell-command LIST) may not show
+                    ;; the output of all the commands (Bug#23206).
+                    ;; Add 'wait' to force those POSIX shells to wait until
+                    ;; all commands finish.
+                    (or (and in-back-no-seq (not (memq system-type '(ms-dos windows-nt))) "&wait")
+                        "")))
+           (t
+            (let ((files (mapconcat 'shell-quote-argument
+                                    file-list dired-mark-separator)))
+              (when (cdr file-list)
+                (setq files (concat dired-mark-prefix files dired-mark-postfix)))
+              (funcall stuff-it files))))
+     (or (and in-background "&") ""))))
 
 ;; This is an extra function so that it can be redefined by ange-ftp.
 ;;;###autoload
-- 
2.8.0.rc3

bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel jobs finish

[Eli Zaretskii]

> Date: Tue, 5 Apr 2016 14:24:41 +0900 (JST)
> From: Tino Calancha <f92capac@gmail.com>
> cc: Tino Calancha <f92capac@gmail.com>, 23206@debbugs.gnu.org
> 
> >This means the fix should only affect systems with a Posix
> >shell.
> >The stock MS-Windows shells don't have a 'wait' command (and
> >don't need this fix in the first place, as the problem doesn't exist
> >on MS-Windows, AFAICS).
> I see.  Thank you.
> I rewrote the patch so that it adds 'wait' when 'system-type' is
> not in '(ms-dos windows-nt).

Thanks.

I have one more nit:

> * lisp/dired-aux.el (dired-shell-stuff-it):
> Drop all trailing ';' and '&' in command;
> change indentation with \t to ?\s;

These are unrelated changes, so please don't mix them with the changes
that really fix the problem at hand.  In particular, removing all the
trailing ';' and '&' characters might cause bugs, because the code you
are modifying doesn't really understand what the shell command is
about, so it might inadvertently delete characters that are there for
a reason.  It is up to the user to type a correct command, and Emacs
shouldn't try second-guessing those commands.  So let's only remove a
single instance of these characters at the end, as the original code
did.

Thanks.

bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel jobs finish

[Tino Calancha]

[-- Attachment #1: Type: text/plain, Size: 546 bytes --]


>These are unrelated changes, so please don't mix them with the changes
>that really fix the problem at hand.
I am sorry tha's my fault.  I should always do atomic changes.

>It is up to the user to type a correct command, and Emacs
>shouldn't try second-guessing those commands.  So let's only remove a
>single instance of these characters at the end, as the original code
>did.
I was 'baby sitting' the user (and my self). Yeah, too much care, 
Emacs users can protect themselves very well ;-)

Attached atomic commits without the extra care.

[-- Attachment #2: Type: text/plain, Size: 2895 bytes --]

From bd476a1f43be47fe79db76e7fea36c750549f72b Mon Sep 17 00:00:00 2001
From: Tino Calancha <f92capac@gmail.com>
Date: Wed, 6 Apr 2016 00:36:17 +0900
Subject: [PATCH 1/2] Wait until all parallel shell commands finish

* lisp/dired-aux.el (dired-shell-stuff-it):
Force POSIX shells to wait until all background jobs finish.
(Bug#23206).
---
 lisp/dired-aux.el | 31 ++++++++++++++++++++++---------
 1 file changed, 22 insertions(+), 9 deletions(-)

diff --git a/lisp/dired-aux.el b/lisp/dired-aux.el
index d29abf3..cc6f168 100644
--- a/lisp/dired-aux.el
+++ b/lisp/dired-aux.el
@@ -729,6 +729,7 @@ dired-shell-stuff-it
 	 (command (if sequentially
 		      (substring command 0 (match-beginning 0))
 		    command))
+         (parallel-in-background (and in-background (not sequentially)))
 	 (stuff-it
 	  (if (or (string-match-p dired-star-subst-regexp command)
 		  (string-match-p dired-quark-subst-regexp command))
@@ -740,15 +741,27 @@ dired-shell-stuff-it
 		  retval))
 	    (lambda (x) (concat command dired-mark-separator x)))))
     (concat
-     (if on-each
-	 (mapconcat stuff-it (mapcar 'shell-quote-argument file-list)
-		    (if (and in-background (not sequentially)) "&" ";"))
-       (let ((files (mapconcat 'shell-quote-argument
-			       file-list dired-mark-separator)))
-	 (if (> (length file-list) 1)
-	     (setq files (concat dired-mark-prefix files dired-mark-postfix)))
-	 (funcall stuff-it files)))
-     (if in-background "&" ""))))
+     (cond (on-each
+            (format "%s%s"
+                    (mapconcat stuff-it (mapcar 'shell-quote-argument file-list)
+                               (or (and parallel-in-background "&") ";"))
+                    ;; POSIX shells running a list of commands in the background
+                    ;; (LIST = cmd_1 & [cmd_2 & ... cmd_i & ... cmd_N &])
+                    ;; return once cmd_N ends, i.e., the shell does not
+                    ;; wait for cmd_i to finish before executing cmd_i+1.
+                    ;; That means, running (shell-command LIST) may not show
+                    ;; the output of all the commands (Bug#23206).
+                    ;; Add 'wait' to force those POSIX shells to wait until
+                    ;; all commands finish.
+                    (or (and parallel-in-background (not (memq system-type '(ms-dos windows-nt))) "&wait")
+                        "")))
+           (t
+            (let ((files (mapconcat 'shell-quote-argument
+                                    file-list dired-mark-separator)))
+              (when (cdr file-list)
+                (setq files (concat dired-mark-prefix files dired-mark-postfix)))
+              (funcall stuff-it files))))
+     (or (and in-background "&") ""))))
 
 ;; This is an extra function so that it can be redefined by ange-ftp.
 ;;;###autoload
-- 
2.8.0.rc3


[-- Attachment #3: Type: text/plain, Size: 2495 bytes --]

From f0ed483aa3e8bf8345c807a6c0d944f177347c53 Mon Sep 17 00:00:00 2001
From: Tino Calancha <f92capac@gmail.com>
Date: Wed, 6 Apr 2016 00:46:02 +0900
Subject: [PATCH 2/2] dired-shell-stuff-it: Consistent indentation

* lisp/dired-aux.el (dired-shell-stuff-it):
Use spaces for indent this function.
---
 lisp/dired-aux.el | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)

diff --git a/lisp/dired-aux.el b/lisp/dired-aux.el
index cc6f168..993b453 100644
--- a/lisp/dired-aux.el
+++ b/lisp/dired-aux.el
@@ -722,24 +722,24 @@ dired-shell-stuff-it
 ;; (coming from interactive P and currently ignored) to decide what to do.
 ;; Smart would be a way to access basename or extension of file names.
   (let* ((in-background (string-match "[ \t]*&[ \t]*\\'" command))
-	 (command (if in-background
-		      (substring command 0 (match-beginning 0))
-		    command))
-	 (sequentially (string-match "[ \t]*;[ \t]*\\'" command))
-	 (command (if sequentially
-		      (substring command 0 (match-beginning 0))
-		    command))
+         (command (if in-background
+                      (substring command 0 (match-beginning 0))
+                    command))
+         (sequentially (string-match "[ \t]*;[ \t]*\\'" command))
+         (command (if sequentially
+                      (substring command 0 (match-beginning 0))
+                    command))
          (parallel-in-background (and in-background (not sequentially)))
-	 (stuff-it
-	  (if (or (string-match-p dired-star-subst-regexp command)
-		  (string-match-p dired-quark-subst-regexp command))
-	      (lambda (x)
-		(let ((retval command))
-		  (while (string-match
-			  "\\(^\\|[ \t]\\)\\([*?]\\)\\([ \t]\\|$\\)" retval)
-		    (setq retval (replace-match x t t retval 2)))
-		  retval))
-	    (lambda (x) (concat command dired-mark-separator x)))))
+         (stuff-it
+          (if (or (string-match-p dired-star-subst-regexp command)
+                  (string-match-p dired-quark-subst-regexp command))
+              (lambda (x)
+                (let ((retval command))
+                  (while (string-match
+                          "\\(^\\|[ \t]\\)\\([*?]\\)\\([ \t]\\|$\\)" retval)
+                    (setq retval (replace-match x t t retval 2)))
+                  retval))
+            (lambda (x) (concat command dired-mark-separator x)))))
     (concat
      (cond (on-each
             (format "%s%s"
-- 
2.8.0.rc3

bug#23206: 25.0.92; dired-shell-stuff-it: wait until all parallel jobs finish

[Eli Zaretskii]

> Date: Wed, 6 Apr 2016 00:58:30 +0900 (JST)
> From: Tino Calancha <f92capac@gmail.com>
> cc: Tino Calancha <f92capac@gmail.com>, 23206@debbugs.gnu.org
> 
> Attached atomic commits without the extra care.

Thanks, pushed to the master branch.