bug#12689: 24.2; Eshell ${cmd} substitution

bug#12689: 24.2; Eshell ${cmd} substitution

[Aidan Gauland]

In Eshell, ${cmd} expands to nil.
e.g.
$ echo ${/bin/echo -e "foo\nbar"}
$ 
Note the absence of any output from `echo'.

$ echo ${/bin/echo -e "foo\nbar"}-foo
nil-foo
n$ 

`*echo' prints expected output, but only for the first case.
$ echo ${/bin/echo -e "foo\nbar"}
foo
bar
$ 

$ echo ${/bin/echo -e "foo\nbar"}-foo
("foo" "bar")-foo
$ 


In GNU Emacs 24.2.1 (x86_64-pc-linux-gnu, GTK+ Version 2.24.10)
 of 2012-09-10 on trouble, modified by Debian
Windowing system distributor `The X.Org Foundation', version 11.0.11203902
Configured using:
 `configure '--build' 'x86_64-linux-gnu' '--build' 'x86_64-linux-gnu'
 '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib'
 '--localstatedir=/var/lib' '--infodir=/usr/share/info'
 '--mandir=/usr/share/man' '--with-pop=yes'
 '--enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/24.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.2/site-lisp:/usr/share/emacs/site-lisp'
 '--with-crt-dir=/usr/lib/x86_64-linux-gnu' '--with-x=yes'
 '--with-x-toolkit=gtk' '--with-toolkit-scroll-bars'
 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fstack-protector
 --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall'
 'CPPFLAGS=-D_FORTIFY_SOURCE=2''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_NZ.utf8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: EShell

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
M-x e s h <tab> <return> e c h o SPC $ { / b i n / 
e c h o M-b C-e SPC } <backspace> - e SPC " f o o <return> 
<backspace> \ n b a r " } <return> M-p - f o o <return> 
M-p C-a * <return> M-p <M-backspace> <backspace> <return> 
M-x r e p o r t <tab> <return>

Recent messages:
Loading em-dirs...done
Loading em-glob...done
Loading em-hist...done
Loading em-ls...done
Loading em-prompt...done
Loading em-script...done
Loading em-term...done
Loading em-unix...done
Expecting completion of delimiter { ...
History item: 128 [3 times]

Load-path shadows:
/usr/share/emacs/24.2/site-lisp/debian-startup hides /usr/share/emacs/site-lisp/debian-startup
/usr/share/emacs/24.2/site-lisp/dictionaries-common/flyspell hides /usr/share/emacs/24.2/lisp/textmodes/flyspell
/usr/share/emacs/24.2/site-lisp/dictionaries-common/ispell hides /usr/share/emacs/24.2/lisp/textmodes/ispell
/usr/local/share/emacs/site-lisp/org-freemind hides /usr/share/emacs/24.2/lisp/org/org-freemind
/usr/local/share/emacs/site-lisp/ob-octave hides /usr/share/emacs/24.2/lisp/org/ob-octave
/usr/local/share/emacs/site-lisp/ob-comint hides /usr/share/emacs/24.2/lisp/org/ob-comint
/usr/local/share/emacs/site-lisp/org-id hides /usr/share/emacs/24.2/lisp/org/org-id
/usr/local/share/emacs/site-lisp/org-timer hides /usr/share/emacs/24.2/lisp/org/org-timer
/usr/local/share/emacs/site-lisp/org-agenda hides /usr/share/emacs/24.2/lisp/org/org-agenda
/usr/local/share/emacs/site-lisp/org-vm hides /usr/share/emacs/24.2/lisp/org/org-vm
/usr/local/share/emacs/site-lisp/org-list hides /usr/share/emacs/24.2/lisp/org/org-list
/usr/local/share/emacs/site-lisp/ob-haskell hides /usr/share/emacs/24.2/lisp/org/ob-haskell
/usr/local/share/emacs/site-lisp/ob-lisp hides /usr/share/emacs/24.2/lisp/org/ob-lisp
/usr/local/share/emacs/site-lisp/org-mhe hides /usr/share/emacs/24.2/lisp/org/org-mhe
/usr/local/share/emacs/site-lisp/ob-ocaml hides /usr/share/emacs/24.2/lisp/org/ob-ocaml
/usr/local/share/emacs/site-lisp/org-xoxo hides /usr/share/emacs/24.2/lisp/org/org-xoxo
/usr/local/share/emacs/site-lisp/ob-calc hides /usr/share/emacs/24.2/lisp/org/ob-calc
/usr/local/share/emacs/site-lisp/org-rmail hides /usr/share/emacs/24.2/lisp/org/org-rmail
/usr/local/share/emacs/site-lisp/org-taskjuggler hides /usr/share/emacs/24.2/lisp/org/org-taskjuggler
/usr/local/share/emacs/site-lisp/org-mks hides /usr/share/emacs/24.2/lisp/org/org-mks
/usr/local/share/emacs/site-lisp/org-archive hides /usr/share/emacs/24.2/lisp/org/org-archive
/usr/local/share/emacs/site-lisp/org-bbdb hides /usr/share/emacs/24.2/lisp/org/org-bbdb
/usr/local/share/emacs/site-lisp/ob-eval hides /usr/share/emacs/24.2/lisp/org/ob-eval
/usr/local/share/emacs/site-lisp/org hides /usr/share/emacs/24.2/lisp/org/org
/usr/local/share/emacs/site-lisp/org-table hides /usr/share/emacs/24.2/lisp/org/org-table
/usr/local/share/emacs/site-lisp/org-docbook hides /usr/share/emacs/24.2/lisp/org/org-docbook
/usr/local/share/emacs/site-lisp/ob-exp hides /usr/share/emacs/24.2/lisp/org/ob-exp
/usr/local/share/emacs/site-lisp/ob-C hides /usr/share/emacs/24.2/lisp/org/ob-C
/usr/local/share/emacs/site-lisp/org-irc hides /usr/share/emacs/24.2/lisp/org/org-irc
/usr/local/share/emacs/site-lisp/org-footnote hides /usr/share/emacs/24.2/lisp/org/org-footnote
/usr/local/share/emacs/site-lisp/org-jsinfo hides /usr/share/emacs/24.2/lisp/org/org-jsinfo
/usr/local/share/emacs/site-lisp/org-bibtex hides /usr/share/emacs/24.2/lisp/org/org-bibtex
/usr/local/share/emacs/site-lisp/ob-perl hides /usr/share/emacs/24.2/lisp/org/ob-perl
/usr/local/share/emacs/site-lisp/org-publish hides /usr/share/emacs/24.2/lisp/org/org-publish
/usr/local/share/emacs/site-lisp/org-exp-blocks hides /usr/share/emacs/24.2/lisp/org/org-exp-blocks
/usr/local/share/emacs/site-lisp/ob-java hides /usr/share/emacs/24.2/lisp/org/ob-java
/usr/local/share/emacs/site-lisp/ob-table hides /usr/share/emacs/24.2/lisp/org/ob-table
/usr/local/share/emacs/site-lisp/ob-scheme hides /usr/share/emacs/24.2/lisp/org/ob-scheme
/usr/local/share/emacs/site-lisp/org-install hides /usr/share/emacs/24.2/lisp/org/org-install
/usr/local/share/emacs/site-lisp/ob-latex hides /usr/share/emacs/24.2/lisp/org/ob-latex
/usr/local/share/emacs/site-lisp/org-special-blocks hides /usr/share/emacs/24.2/lisp/org/org-special-blocks
/usr/local/share/emacs/site-lisp/ob-mscgen hides /usr/share/emacs/24.2/lisp/org/ob-mscgen
/usr/local/share/emacs/site-lisp/org-gnus hides /usr/share/emacs/24.2/lisp/org/org-gnus
/usr/local/share/emacs/site-lisp/org-beamer hides /usr/share/emacs/24.2/lisp/org/org-beamer
/usr/local/share/emacs/site-lisp/ob-sass hides /usr/share/emacs/24.2/lisp/org/ob-sass
/usr/local/share/emacs/site-lisp/org-datetree hides /usr/share/emacs/24.2/lisp/org/org-datetree
/usr/local/share/emacs/site-lisp/org-src hides /usr/share/emacs/24.2/lisp/org/org-src
/usr/local/share/emacs/site-lisp/org-mouse hides /usr/share/emacs/24.2/lisp/org/org-mouse
/usr/local/share/emacs/site-lisp/org-entities hides /usr/share/emacs/24.2/lisp/org/org-entities
/usr/local/share/emacs/site-lisp/org-odt hides /usr/share/emacs/24.2/lisp/org/org-odt
/usr/local/share/emacs/site-lisp/org-plot hides /usr/share/emacs/24.2/lisp/org/org-plot
/usr/local/share/emacs/site-lisp/org-eshell hides /usr/share/emacs/24.2/lisp/org/org-eshell
/usr/local/share/emacs/site-lisp/org-latex hides /usr/share/emacs/24.2/lisp/org/org-latex
/usr/local/share/emacs/site-lisp/ob-sql hides /usr/share/emacs/24.2/lisp/org/ob-sql
/usr/local/share/emacs/site-lisp/ob-ref hides /usr/share/emacs/24.2/lisp/org/ob-ref
/usr/local/share/emacs/site-lisp/ob-js hides /usr/share/emacs/24.2/lisp/org/ob-js
/usr/local/share/emacs/site-lisp/ob-gnuplot hides /usr/share/emacs/24.2/lisp/org/ob-gnuplot
/usr/local/share/emacs/site-lisp/org-remember hides /usr/share/emacs/24.2/lisp/org/org-remember
/usr/local/share/emacs/site-lisp/org-protocol hides /usr/share/emacs/24.2/lisp/org/org-protocol
/usr/local/share/emacs/site-lisp/ob-shen hides /usr/share/emacs/24.2/lisp/org/ob-shen
/usr/local/share/emacs/site-lisp/org-macs hides /usr/share/emacs/24.2/lisp/org/org-macs
/usr/local/share/emacs/site-lisp/ob-plantuml hides /usr/share/emacs/24.2/lisp/org/ob-plantuml
/usr/local/share/emacs/site-lisp/org-exp hides /usr/share/emacs/24.2/lisp/org/org-exp
/usr/local/share/emacs/site-lisp/ob-R hides /usr/share/emacs/24.2/lisp/org/ob-R
/usr/local/share/emacs/site-lisp/ob-sh hides /usr/share/emacs/24.2/lisp/org/ob-sh
/usr/local/share/emacs/site-lisp/org-capture hides /usr/share/emacs/24.2/lisp/org/org-capture
/usr/local/share/emacs/site-lisp/org-colview hides /usr/share/emacs/24.2/lisp/org/org-colview
/usr/local/share/emacs/site-lisp/org-clock hides /usr/share/emacs/24.2/lisp/org/org-clock
/usr/local/share/emacs/site-lisp/org-info hides /usr/share/emacs/24.2/lisp/org/org-info
/usr/local/share/emacs/site-lisp/org-ascii hides /usr/share/emacs/24.2/lisp/org/org-ascii
/usr/local/share/emacs/site-lisp/org-ctags hides /usr/share/emacs/24.2/lisp/org/org-ctags
/usr/local/share/emacs/site-lisp/ob-maxima hides /usr/share/emacs/24.2/lisp/org/ob-maxima
/usr/local/share/emacs/site-lisp/org-inlinetask hides /usr/share/emacs/24.2/lisp/org/org-inlinetask
/usr/local/share/emacs/site-lisp/ob-emacs-lisp hides /usr/share/emacs/24.2/lisp/org/ob-emacs-lisp
/usr/local/share/emacs/site-lisp/ob-python hides /usr/share/emacs/24.2/lisp/org/ob-python
/usr/local/share/emacs/site-lisp/org-lparse hides /usr/share/emacs/24.2/lisp/org/org-lparse
/usr/local/share/emacs/site-lisp/org-compat hides /usr/share/emacs/24.2/lisp/org/org-compat
/usr/local/share/emacs/site-lisp/org-docview hides /usr/share/emacs/24.2/lisp/org/org-docview
/usr/local/share/emacs/site-lisp/ob-dot hides /usr/share/emacs/24.2/lisp/org/ob-dot
/usr/local/share/emacs/site-lisp/org-feed hides /usr/share/emacs/24.2/lisp/org/org-feed
/usr/local/share/emacs/site-lisp/ob-keys hides /usr/share/emacs/24.2/lisp/org/ob-keys
/usr/local/share/emacs/site-lisp/org-mac-message hides /usr/share/emacs/24.2/lisp/org/org-mac-message
/usr/local/share/emacs/site-lisp/org-attach hides /usr/share/emacs/24.2/lisp/org/org-attach
/usr/local/share/emacs/site-lisp/org-mew hides /usr/share/emacs/24.2/lisp/org/org-mew
/usr/local/share/emacs/site-lisp/ob-ditaa hides /usr/share/emacs/24.2/lisp/org/ob-ditaa
/usr/local/share/emacs/site-lisp/ob-asymptote hides /usr/share/emacs/24.2/lisp/org/ob-asymptote
/usr/local/share/emacs/site-lisp/org-crypt hides /usr/share/emacs/24.2/lisp/org/org-crypt
/usr/local/share/emacs/site-lisp/ob-sqlite hides /usr/share/emacs/24.2/lisp/org/ob-sqlite
/usr/local/share/emacs/site-lisp/org-mobile hides /usr/share/emacs/24.2/lisp/org/org-mobile
/usr/local/share/emacs/site-lisp/ob-matlab hides /usr/share/emacs/24.2/lisp/org/ob-matlab
/usr/local/share/emacs/site-lisp/org-faces hides /usr/share/emacs/24.2/lisp/org/org-faces
/usr/local/share/emacs/site-lisp/ob-fortran hides /usr/share/emacs/24.2/lisp/org/ob-fortran
/usr/local/share/emacs/site-lisp/ob-ruby hides /usr/share/emacs/24.2/lisp/org/ob-ruby
/usr/local/share/emacs/site-lisp/ob-clojure hides /usr/share/emacs/24.2/lisp/org/ob-clojure
/usr/local/share/emacs/site-lisp/org-w3m hides /usr/share/emacs/24.2/lisp/org/org-w3m
/usr/local/share/emacs/site-lisp/ob-tangle hides /usr/share/emacs/24.2/lisp/org/ob-tangle
/usr/local/share/emacs/site-lisp/ob-screen hides /usr/share/emacs/24.2/lisp/org/ob-screen
/usr/local/share/emacs/site-lisp/ob-picolisp hides /usr/share/emacs/24.2/lisp/org/ob-picolisp
/usr/local/share/emacs/site-lisp/org-pcomplete hides /usr/share/emacs/24.2/lisp/org/org-pcomplete
/usr/local/share/emacs/site-lisp/ob-css hides /usr/share/emacs/24.2/lisp/org/ob-css
/usr/local/share/emacs/site-lisp/org-icalendar hides /usr/share/emacs/24.2/lisp/org/org-icalendar
/usr/local/share/emacs/site-lisp/org-indent hides /usr/share/emacs/24.2/lisp/org/org-indent
/usr/local/share/emacs/site-lisp/org-habit hides /usr/share/emacs/24.2/lisp/org/org-habit
/usr/local/share/emacs/site-lisp/ob hides /usr/share/emacs/24.2/lisp/org/ob
/usr/local/share/emacs/site-lisp/ob-ledger hides /usr/share/emacs/24.2/lisp/org/ob-ledger
/usr/local/share/emacs/site-lisp/org-html hides /usr/share/emacs/24.2/lisp/org/org-html
/usr/local/share/emacs/site-lisp/ob-org hides /usr/share/emacs/24.2/lisp/org/ob-org
/usr/local/share/emacs/site-lisp/ob-awk hides /usr/share/emacs/24.2/lisp/org/ob-awk
/usr/local/share/emacs/site-lisp/ob-lilypond hides /usr/share/emacs/24.2/lisp/org/ob-lilypond
/usr/local/share/emacs/site-lisp/org-wl hides /usr/share/emacs/24.2/lisp/org/org-wl
/usr/local/share/emacs/site-lisp/ob-lob hides /usr/share/emacs/24.2/lisp/org/ob-lob

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils pp em-unix pcomplete comint regexp-opt
ansi-color em-term term disp-table ehelp electric em-script em-prompt
em-ls em-hist em-pred em-glob em-dirs em-cmpl em-basic esh-opt em-banner
em-alias ring esh-var esh-io esh-cmd esh-ext esh-proc esh-arg eldoc
help-fns esh-groups eshell esh-module esh-mode esh-util time-date
tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar
dnd fontset image fringe lisp-mode register page menu-bar rfn-eshadow
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core frame cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer loaddefs button faces
cus-face files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote
make-network-process dbusbind dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)

bug#12689: 24.2; Eshell ${cmd} substitution

[samer]

There are two issues contained in this bug: eshell-plain-command doesn't
wait for the process to finish before returning, and echo parses output
from subcommands as lisp objects instead of as args.

The following patch makes eshell wait for external commands to finish
before getting their output.

The second bug is more complicated. Arguments are parsed with
eshell-parse-arguments, which takes into account eshell's rules and
syntax. When things are run in subshells, they're converted to lisp
objects with eshell-convert, with no further processing. For example,
for the command "echo ${/bin/echo hello there}" the subshell will return
"hello there", which is seen as a single argument by the outer "echo".
This isn't a big deal, but it means some constructions won't make sense.
For instance:

$ echo hello there
("hello" "there")
$ echo ${/bin/echo hello there}
hello there

I tried hacking with eshell-parse-variable-ref to parse the arguments
correctly, but it's trickier than it seems at first glance, so I'll work
on it later. This patch gets us 90% of the way there, though.

Best,
Samer

---- Patch below ----

Changes from HEAD to working tree
2 files changed, 6 insertions(+), 1 deletion(-)
  lisp/ChangeLog         |    5 +++++
  lisp/eshell/esh-cmd.el |    2 +-

	Modified   lisp/ChangeLog
diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 45ba279..aa00661 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,8 @@
+2014-12-14  Samer Masterson  <samer@samertm.com>
+
+	* eshell/esh-cmd.el (eshell-plain-command): Wait for subprocess to
+	finish before evaluating command (bug#12680).
+
  2014-11-28  Martin Rudalics  <rudalics@gmx.at>

  	Fix two issues around help-window-select.  (Bug#11039) (Bug#19012)
	Modified   lisp/eshell/esh-cmd.el
diff --git a/lisp/eshell/esh-cmd.el b/lisp/eshell/esh-cmd.el
index d20b66a..f17aa41 100644
--- a/lisp/eshell/esh-cmd.el
+++ b/lisp/eshell/esh-cmd.el
@@ -1241,7 +1241,7 @@ or an external command."
  	     (or esym eshell-prefer-lisp-functions
  		 (not (eshell-search-path command))))
  	(eshell-lisp-command sym args)
-      (eshell-external-command command args))))
+      (eshell-wait-for-process (eshell-external-command command 
args)))))

  (defun eshell-exec-lisp (printer errprint func-or-form args form-p)
    "Execute a lisp FUNC-OR-FORM, maybe passing ARGS.

bug#12689: 24.2; Eshell ${cmd} substitution

[Samer Masterson]

[-- Attachment #1: Type: text/plain, Size: 1299 bytes --]

Hi,

My copyright papers have been processed. Can we apply this to master?

Best,
Samer

Changes from HEAD to working tree
2 files changed, 6 insertions(+), 1 deletion(-)
 lisp/ChangeLog         |    5 +++++
 lisp/eshell/esh-cmd.el |    2 +-

	Modified   lisp/ChangeLog
diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 45ba279..aa00661 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,8 @@
+2014-12-14  Samer Masterson  <samer <at> samertm.com>
+
+	* eshell/esh-cmd.el (eshell-plain-command): Wait for subprocess to
+	finish before evaluating command (bug#12680).
+
 2014-11-28  Martin Rudalics  <rudalics <at> gmx.at>

 	Fix two issues around help-window-select.  (Bug#11039) (Bug#19012)
	Modified   lisp/eshell/esh-cmd.el
diff --git a/lisp/eshell/esh-cmd.el b/lisp/eshell/esh-cmd.el
index d20b66a..f17aa41 100644
--- a/lisp/eshell/esh-cmd.el
+++ b/lisp/eshell/esh-cmd.el
@@ -1241,7 +1241,7 @@ or an external command."
 	     (or esym eshell-prefer-lisp-functions
 		 (not (eshell-search-path command))))
 	(eshell-lisp-command sym args)
-      (eshell-external-command command args))))
+      (eshell-wait-for-process (eshell-external-command command
args)))))

 (defun eshell-exec-lisp (printer errprint func-or-form args form-p)
   "Execute a lisp FUNC-OR-FORM, maybe passing ARGS.

[-- Attachment #2: Type: text/html, Size: 2408 bytes --]

bug#12689: 24.2; Eshell ${cmd} substitution

[Eli Zaretskii]

> Date: Tue, 24 Feb 2015 02:56:12 -0800
> From: Samer Masterson <samer@samertm.com>
> 
> Changes from HEAD to working tree
> 2 files changed, 6 insertions(+), 1 deletion(-)
> lisp/ChangeLog | 5 +++++
> lisp/eshell/esh-cmd.el | 2 +-
> 
> Modified lisp/ChangeLog
> diff --git a/lisp/ChangeLog b/lisp/ChangeLog
> index 45ba279..aa00661 100644
> --- a/lisp/ChangeLog
> +++ b/lisp/ChangeLog
> @@ -1,3 +1,8 @@
> +2014-12-14 Samer Masterson <samer <at> samertm.com>
> +
> + * eshell/esh-cmd.el (eshell-plain-command): Wait for subprocess to
> + finish before evaluating command (bug#12680).

Does this have any effect on the feature where Eshell can run several
commands simultaneously via a shell-like pipe?

Thanks.

bug#12689: 24.2; Eshell ${cmd} substitution

[Samer Masterson]

[-- Attachment #1: Type: text/plain, Size: 983 bytes --]

On Tue, Mar 3, 2015 at 7:54 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> Does this have any effect on the feature where Eshell can run several
> commands simultaneously via a shell-like pipe?
> 
> Thanks.

I don't know what feature you're describing. Here is the output from 
running regular pipes:

~ $ echo hello | /bin/echo there | /bin/echo hey | echo hi # Without 
patch
hi
hey
~ $ echo hello | /bin/echo there | /bin/echo hey | echo hi # With patch
hi
hey
there
hello

First, we should ignore the fact that echo does nothing with stdin, and 
the above commands print "hi" in bash. I do not understand why the 
command without the patch only prints out "hey\nhi", but it printed 
that consistently on my computer. The second output is incorrect (or at 
the very least unexpected?), but less incorrect than the first. If this 
is what you were talking about, then the commands may have appeared to 
be simultaneous because they were racing, but I'm not sure if that's 
intentional.

[-- Attachment #2: Type: text/html, Size: 1215 bytes --]

bug#12689: 24.2; Eshell ${cmd} substitution

[Eli Zaretskii]

> Date: Wed, 04 Mar 2015 04:14:07 -0800
> From: Samer Masterson <samer@samertm.com>
> Cc: 12689@debbugs.gnu.org
> 
> On Tue, Mar 3, 2015 at 7:54 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>     Does this have any effect on the feature where Eshell can run several
>     commands simultaneously via a shell-like pipe? Thanks.
> 
> I don't know what feature you're describing.

eshell-execute-pipeline was what I had in mind.

It's been a while since I last hacked Eshell, so perhaps that was a
stupid question, but what prompted it was the fact that your change
would wait until a process exited under some conditions, so I wondered
whether it will prevent Eshell from firing up several asynchronous
processes connected through a pipe.

> Here is the output from running
> regular pipes:
> 
> ~ $ echo hello | /bin/echo there | /bin/echo hey | echo hi # Without patch
> hi
> hey
> ~ $ echo hello | /bin/echo there | /bin/echo hey | echo hi # With patch
> hi
> hey
> there
> hello

Not sure why you intersperse internal and external echo, or why did
you use echo to begin with.

In any case, what you see are bugs, at least in the case of a pipe
that cannot connect because the receiving end doesn't read stdin.

> First, we should ignore the fact that echo does nothing with stdin, and the
> above commands print "hi" in bash. I do not understand why the command without
> the patch only prints out "hey\nhi", but it printed that consistently on my
> computer. The second output is incorrect (or at the very least unexpected?),
> but less incorrect than the first. If this is what you were talking about, then
> the commands may have appeared to be simultaneous because they were racing, but
> I'm not sure if that's intentional.

I'd rather see that a pipe that does make sense, something like

  cat some-large-file | wc

with external commands works as expected, and both programs run
simultaneously in parallel (e.g., according to 'top' or a similar
tool), not sequentially.

Thanks.

bug#12689: 24.2; Eshell ${cmd} substitution

[Samer Masterson]

[-- Attachment #1: Type: text/plain, Size: 1349 bytes --]


On Wed, Mar 4, 2015 at 9:38 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> eshell-execute-pipeline was what I had in mind.
> 
> It's been a while since I last hacked Eshell, so perhaps that was a
> stupid question, but what prompted it was the fact that your change
> would wait until a process exited under some conditions, so I wondered
> whether it will prevent Eshell from firing up several asynchronous
> processes connected through a pipe.

Oh, I see what you're saying. My confusion came from not knowing that 
piped programs executed in parallel in traditional shells.
> 
> Not sure why you intersperse internal and external echo, or why did
> you use echo to begin with.
I thought the external echos would race and output out-of-order, but 
they just cut off after the first external echo.
> I'd rather see that a pipe that does make sense, something like
> 
>   cat some-large-file | wc
> 
> with external commands works as expected, and both programs run
> simultaneously in parallel (e.g., according to 'top' or a similar
> tool), not sequentially.
The following command

~$ sleep 1 | sleep 1 | sleep 1

runs in 1 second without my patch and 3 with it (which doesn't conform 
with Bash). Also

~$ cat any-file | wc

never returns with my patch. Thanks for catching these errors, and I'll 
ping this bug report when my new patch is ready.
-s

[-- Attachment #2: Type: text/html, Size: 1741 bytes --]

bug#12689: 24.2; Eshell ${cmd} substitution

[Samer Masterson]

Hi,

I haven't finished, but I'm putting this work on hold for now as I've 
spent ages on it, and I'll return to this when I've learned more about 
eshell's architecture. The state of my work is below.

The patch below addresses the points you brought up, but it is 
incomplete for two reasons:
 - my code uses eshell-wait-for-process, which doesn't seem to be the 
correct wait to actually wait for a process to end because it locks up 
Emacs (because the core of eshell-wait-for-process is a while loop).
 - "prompt$ ${ext_command_one} | ext_command_two" will not reset the 
eshell prompt to normal if ext_command_one takes longer than 
ext_command_two. It works normally when ext_command_two takes longer 
than ext_command_one.

So I need to figure out a couple things in order to unblock myself:
 - how eshell knows to wait for other external processes (excluding 
eshell-wait-for-process), and whether that can be used at command-level 
granularity. E.g. for "cmd ${ext_cmd0} | ext_cmd1", the entire process 
needs to wait for both sides of the pipe to finish, and "cmd" needs to 
wait for "ext_cmd0" to finish before executing). "cmd ${ext_cmd0}" also 
shouldn't lock Emacs.
 - where the entry point is for the command prompt, and how the command 
prompt is reset after a set of commands finishes.

I've cc'd johnw because he might know the answer to the above 
questions, and what the correct approach to fixing this should be. I'll 
return to this when I've learned more about how eshell works.

-samer

work-in-progress patch:

--- a/lisp/eshell/esh-cmd.el
+++ b/lisp/eshell/esh-cmd.el
@@ -851,8 +851,9 @@ This is used on systems where `start-process' is 
not supported."
 (defmacro eshell-as-subcommand (command)
   "Execute COMMAND using a temp buffer.
 This is used so that certain Lisp commands, such as `cd', when
-executed in a subshell, do not disturb the environment of the main
-Eshell buffer."
+executed in a subshell, do not disturb the environment of the
+main Eshell buffer.  If COMMAND is an external command, it is
+called synchronously."
   `(let ,eshell-subcommand-bindings
      ,command))

@@ -1234,14 +1235,21 @@ COMMAND may result in an alias being executed, 
or a plain command."
 (defun eshell-plain-command (command args)
   "Insert output from a plain COMMAND, using ARGS.
 COMMAND may result in either a Lisp function being executed by name,
-or an external command."
+or an external command.
+
+If COMMAND is in an external command and `eshell-in-subcommand-p'
+is true, `eshell-plain-command' is synchronous and waits for
+COMMAND to finish."
   (let* ((esym (eshell-find-alias-function command))
 	 (sym (or esym (intern-soft command))))
     (if (and sym (fboundp sym)
 	     (or esym eshell-prefer-lisp-functions
 		 (not (eshell-search-path command))))
 	(eshell-lisp-command sym args)
-      (eshell-external-command command args))))
+      (let ((ext (eshell-external-command command args)))
+        (if eshell-in-subcommand-p
+            (eshell-wait-for-process ext)
+          ext)))))

bug#12689: 24.2; Eshell ${cmd} substitution

[John Wiegley]

>>>>> Samer Masterson <samer@samertm.com> writes:

> So I need to figure out a couple things in order to unblock myself: - how
> eshell knows to wait for other external processes (excluding
> eshell-wait-for-process), and whether that can be used at command-level
> granularity. E.g. for "cmd ${ext_cmd0} | ext_cmd1", the entire process needs
> to wait for both sides of the pipe to finish, and "cmd" needs to wait for
> "ext_cmd0" to finish before executing). "cmd ${ext_cmd0}" also shouldn't
> lock Emacs. - where the entry point is for the command prompt, and how the
> command prompt is reset after a set of commands finishes.

> I've cc'd johnw because he might know the answer to the above questions, and
> what the correct approach to fixing this should be. I'll return to this when
> I've learned more about how eshell works.

I don't really know the answer anymore (Eshell is 16 years old now!), but I'm
wondering what the rationale is for making this change?  Anything that messes
with the way sub-processes are handled should be treated with extreme care.

John

bug#12689: 24.2; Eshell ${cmd} substitution

[Samer Masterson]

On Sun, Apr 5, 2015 at 9:19 PM, John Wiegley <johnw@newartisans.com> wrote
> I'm wondering what the rationale is for making this change?

The issue is that subprocesses that call external commands don't
wait for the command to complete before returning. For instance,
the following command:

  $ echo ${/bin/echo "hi"}

returns nothing w/o the patch applied, where:

  $ echo ${echo "hi"}

returns "hi".

-samer

bug#12689: 24.2; Eshell ${cmd} substitution

[Lars Ingebrigtsen]

Aidan Gauland <aidalgol@no8wireless.co.nz> writes:

> In Eshell, ${cmd} expands to nil.
> e.g.
> $ echo ${/bin/echo -e "foo\nbar"}
> $ 
> Note the absence of any output from `echo'.
>
> $ echo ${/bin/echo -e "foo\nbar"}-foo
> nil-foo
> n$ 

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

Looks like this has been fixed in the decade since it was reported:

/ $ echo ${/bin/echo -e "foo\nbar"}
("foo" "bar")
/ $ echo ${/bin/echo -e "foo\nbar"}-foo
("foo" "bar")-foo

So I'm therefore closing this bug report.  If there's more to be done
here, please respond to the debbugs address and we'll reopen.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#12689: 24.2; Eshell ${cmd} substitution

[Jim Porter]

On 2/9/2022 1:12 AM, Lars Ingebrigtsen wrote:
> Aidan Gauland <aidalgol@no8wireless.co.nz> writes:
> 
>> In Eshell, ${cmd} expands to nil.
>> e.g.
>> $ echo ${/bin/echo -e "foo\nbar"}
>> $
>> Note the absence of any output from `echo'.
>>
>> $ echo ${/bin/echo -e "foo\nbar"}-foo
>> nil-foo
>> n$
> 
> (I'm going through old bug reports that unfortunately weren't resolved
> at the time.)
> 
> Looks like this has been fixed in the decade since it was reported:
> 
> / $ echo ${/bin/echo -e "foo\nbar"}
> ("foo" "bar")
> / $ echo ${/bin/echo -e "foo\nbar"}-foo
> ("foo" "bar")-foo
> 
> So I'm therefore closing this bug report.  If there's more to be done
> here, please respond to the debbugs address and we'll reopen.

I think the fix is from bug#30725. I'd meant to post an update to this 
bug as well, but I haven't had a chance to investigate the other half of 
this bug yet. Specifically, I think there's at least an argument that 
the result *should* look like this:

   / $ echo ${/bin/echo -e "foo\nbar"}
   foo
   bar
   / $ echo ${/bin/echo -e "foo\nbar"}-foo
   foo
   bar-foo

I'm still not 100% sure what's going on here, but I think the ${} 
subcommand evaluation splits the output on "\n" to make a list (which 
seems reasonable to me). However, when it gets converted back to a 
string (either to print directly, as in the first case, or to 
concatenate, as in the second case), it ends up looking like '("foo" 
"bar")'. You can see a little more clearly how these cases are different 
by calling `message' instead of `echo':

   ~ $ message %S ${*echo -e "foo\nbar"}
   ("foo" "bar")
   ~ $ message %S ${*echo -e "foo\nbar"}-foo
   "(\"foo\" \"bar\")-foo"

So the result of "${...}" is a list, and the result of "${...}-foo" is a 
string.

I think it would make sense to do something to improve how "foo\nbar" 
gets round-tripped here so that the result is more consistent with 
regular shells. I have a couple of ideas, but I'll need a bit of time to 
tinker with things to see how they work, and to find a solution that 
keeps incompatible changes to a minimum.

bug#12689: 24.2; Eshell ${cmd} substitution

[Lars Ingebrigtsen]

Jim Porter <jporterbugs@gmail.com> writes:

> I think it would make sense to do something to improve how "foo\nbar"
> gets round-tripped here so that the result is more consistent with
> regular shells. I have a couple of ideas, but I'll need a bit of time
> to tinker with things to see how they work, and to find a solution
> that keeps incompatible changes to a minimum.

OK; I'm reopening this bug report, then.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#12689: 24.2; Eshell ${cmd} substitution

[Jim Porter]

On 2/9/2022 12:12 PM, Lars Ingebrigtsen wrote:
> OK; I'm reopening this bug report, then.

I filed bug#55236 to fix some wider issues I encountered with Eshell 
expansions, which may also fix this bug. I think it depends on what 
exactly we want for the behavior here. With my patches in that bug, this 
is the behavior I get:

Default:

   ~ $ echo ${/bin/echo -e "foo\nbar"}
   ("foo" "bar")
   ~ $ echo ${/bin/echo -e "foo\nbar"}-baz
   ("foo" "bar-baz")
   ~ $ /bin/echo ${/bin/echo -e "foo\nbar"}
   foo bar
   ~ $ /bin/echo ${/bin/echo -e "foo\nbar"}-baz
   foo bar-baz

With `eshell-plain-echo-behavior' == t:

   ~ $ echo ${*echo -e "foo\nbar"}
   foo bar
   ~ $ echo ${*echo -e "foo\nbar"}-baz
   foo bar-baz
   ~ $ /bin/echo ${/bin/echo -e "foo\nbar"}
   foo bar
   ~ $ /bin/echo ${/bin/echo -e "foo\nbar"}-baz
   foo bar-baz

I think the "plain" echo behavior is what the original report expected, 
so maybe we can consider this fixed then. However, maybe we should 
consider whether the default, non-"plain" behavior should be changed. I 
don't have a strong opinion here, although I'm hesitant to make any big 
changes to the default behavior of Eshell's echo builtin, since I feel 
like it could break a lot of things really easily...

bug#12689: 24.2; Eshell ${cmd} substitution

[Lars Ingebrigtsen]

Jim Porter <jporterbugs@gmail.com> writes:

> I think the "plain" echo behavior is what the original report
> expected, so maybe we can consider this fixed then. However, maybe we
> should consider whether the default, non-"plain" behavior should be
> changed. I don't have a strong opinion here, although I'm hesitant to
> make any big changes to the default behavior of Eshell's echo builtin,
> since I feel like it could break a lot of things really easily...

I think the behaviour after your patch seems reasonable (but I don't use
eshell regularly).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no