bug#46351: 28.0.50; Add convenient way to bypass Eshell's own pipelining

[Sean Whitton <spwhitton@spwhitton.name>]

[-- Attachment #1: Type: text/plain, Size: 1134 bytes --]

Hello,

On Mon 08 Feb 2021 at 11:28AM +01, Michael Albinus wrote:

> Maybe we could add it to the defaults. But at least I would
> like to see it documented in the eshell manual, otherwise nobody will
> know the effect of "!!".

I'm attaching new patches which

- switch !! to ||, because !! in shells typically has something to do
  with rerunning the last command

- activate the syntax by default -- I do use Eshell regularly myself,
  and I don't believe that activating it by default is likely to disrupt
  anyone's usage

- adds a section to the Eshell manual introducing the new syntax.

> And I have the impression, that the eshell history is not preserved any
> longer, when I have applied these settings.

I have not been able to reproduce this -- can you confirm?

> It doesn't work for me. The reason is a little bit subtle: you use
> `shell-file-name' in order to call a remote shell. In my case, it is
> "/usr/local/bin/tcsh", which doesn't exist on the remote machine I've
> used for testing.

I was able to fix this by using `with-connection-local-variables'.
Thank you for pointing out the problem.

-- 
Sean Whitton

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v2-0001-Add-eshell-restore-unexpanded-input.patch --]
[-- Type: text/x-patch, Size: 2720 bytes --]

From d47446cbe7db5c7fb4eba71ab022cfae3de07799 Mon Sep 17 00:00:00 2001
From: Sean Whitton <spwhitton@spwhitton.name>
Date: Sat, 6 Feb 2021 00:01:50 -0700
Subject: [PATCH v2 1/2] Add eshell-restore-unexpanded-input

* lisp/eshell/esh-mode.el (eshell-send-input): Store the original
input before running eshell-expand-input-functions.
* lisp/eshell/esh-mode.el (eshell-restore-unexpanded-input): Define
new function and register as a customization option for
eshell-input-filter-functions.
---
 lisp/eshell/esh-mode.el | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/lisp/eshell/esh-mode.el b/lisp/eshell/esh-mode.el
index cae5236d89..87166ef3bf 100644
--- a/lisp/eshell/esh-mode.el
+++ b/lisp/eshell/esh-mode.el
@@ -62,6 +62,7 @@
 (require 'esh-module)
 (require 'esh-cmd)
 (require 'esh-arg)                      ;For eshell-parse-arguments
+(require 'cl-lib)
 
 (defgroup eshell-mode nil
   "This module contains code for handling input from the user."
@@ -197,6 +198,7 @@ eshell-first-time-p
 ;; byte-compiler, when compiling other files which `require' this one
 (defvar eshell-mode nil)
 (defvar eshell-command-running-string "--")
+(defvar eshell-last-unexpanded-input nil)
 (defvar eshell-last-input-start nil)
 (defvar eshell-last-input-end nil)
 (defvar eshell-last-output-start nil)
@@ -641,6 +643,7 @@ eshell-send-input
 		(progn
 		  (setq input (buffer-substring-no-properties
 			       eshell-last-output-end (1- (point))))
+                  (setq eshell-last-unexpanded-input input)
 		  (run-hook-with-args 'eshell-expand-input-functions
 				      eshell-last-output-end (1- (point)))
 		  (let ((cmd (eshell-parse-command-input
@@ -674,6 +677,23 @@ eshell-kill-new
 
 (custom-add-option 'eshell-input-filter-functions 'eshell-kill-new)
 
+(defun eshell-restore-unexpanded-input ()
+  "Restore the input text before `eshell-expand-input-functions' ran.
+Useful when you want to see the unexpanded input rather than the
+expanded input in the Eshell buffer, and when you want later
+entries in `eshell-input-filter-functions' to use the unexpanded
+input."
+  (setf (buffer-substring eshell-last-input-start
+                          (1- eshell-last-input-end))
+        eshell-last-unexpanded-input)
+  ;; We have to move point for compatibility with smart display.
+  (save-excursion
+    (goto-char eshell-last-input-end)
+    (eshell-update-markers eshell-last-output-end)))
+
+(custom-add-option 'eshell-input-filter-functions
+                   'eshell-restore-unexpanded-input)
+
 (defun eshell-output-filter (process string)
   "Send the output from PROCESS (STRING) to the interactive display.
 This is done after all necessary filtering has been done."
-- 
2.30.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: v2-0002-Add-eshell-shell-command-and-eshell-expand-to-esh.patch --]
[-- Type: text/x-patch, Size: 7533 bytes --]

From 98df5163018b9c99ad8e65f03422f908c6068c78 Mon Sep 17 00:00:00 2001
From: Sean Whitton <spwhitton@spwhitton.name>
Date: Sat, 6 Feb 2021 00:48:32 -0700
Subject: [PATCH v2 2/2] Add eshell-shell-command and
 eshell-expand-to-eshell-shell-command

* lisp/eshell/esh-mode.el (eshell-shell-command,
eshell-expand-to-eshell-shell-command): Define new functions.
Register eshell-expand-to-eshell-shell-command as a customization
option for eshell-expand-input-functions, and add it by default.
* etc/NEWS:
* doc/misc/eshell.texi: Document the new syntax.
---
 doc/misc/eshell.texi    | 35 ++++++++++++++++++
 etc/NEWS                |  9 +++++
 lisp/eshell/esh-mode.el | 79 ++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 122 insertions(+), 1 deletion(-)

diff --git a/doc/misc/eshell.texi b/doc/misc/eshell.texi
index a87dd4308c..7e9233c09e 100644
--- a/doc/misc/eshell.texi
+++ b/doc/misc/eshell.texi
@@ -875,6 +875,7 @@ Expansion
 @menu
 * Dollars Expansion::
 * Globbing::
+* Running Shell Pipelines Natively::
 @end menu
 
 @node Dollars Expansion
@@ -945,6 +946,40 @@ Globbing
 The GNU Emacs Manual}.}
 groups ``eshell-glob'' and ``eshell-pred''.
 
+@node Running Shell Pipelines Natively
+@section Running Shell Pipelines Natively
+When constructing shell pipelines that will move a lot of data, it is
+a good idea to bypass Eshell's own pipelining support and use the
+operating system shell's instead.  This is especially relevant when
+executing commands on a remote machine using Eshell's Tramp
+integration: using the remote shell's pipelining avoids copying the
+data which will flow through the pipeline to local Emacs buffers and
+then right back again.
+
+To quickly construct a command for the native shell, prefix your
+Eshell input with the string @code{||}.  For example,
+
+@example
+|| cat *.ogg | my-cool-decoder >file
+@end example
+
+Executing this command will not copy all the data in the *.ogg files
+into Emacs buffers, as would normally happen with Eshell's own
+pipelining.
+
+As this feature is implemented as an expansion, your original input
+will be replaced with something like this:
+
+@example
+eshell-shell-command \"cat *.ogg | my-cool-decoder >file\"
+@end example
+
+This makes it clear what Eshell is doing, but has the disadvantage of
+being harder to edit, especially for complex pipelines with a lot of
+quoting.  If you would prefer that the @code{||}-prefixed input be
+restored, customize the variable @code{eshell-input-filter-functions}
+to include the function @code{eshell-restore-unexpanded-input}.
+
 @node Input/Output
 @chapter Input/Output
 Since Eshell does not communicate with a terminal like most command
diff --git a/etc/NEWS b/etc/NEWS
index 57fe40c488..767763098b 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -749,6 +749,15 @@ the Netscape web browser was released in February, 2008.
 This support has been obsolete since Emacs 25.1.  The final version of
 the Galeon web browser was released in September, 2008.
 
+** Eshell
+
++++
+*** New feature to easily bypass Eshell's own pipelining.
+Prefixing commands with '||' causes them to be executed using the
+operating system shell, which is particularly useful to bypass
+Eshell's own pipelining for pipelines which will move a lot of data.
+See "Running Shell Pipelines Natively" in the Eshell manual.
+
 \f
 * New Modes and Packages in Emacs 29.1
 
diff --git a/lisp/eshell/esh-mode.el b/lisp/eshell/esh-mode.el
index 87166ef3bf..04680e4305 100644
--- a/lisp/eshell/esh-mode.el
+++ b/lisp/eshell/esh-mode.el
@@ -63,6 +63,7 @@
 (require 'esh-cmd)
 (require 'esh-arg)                      ;For eshell-parse-arguments
 (require 'cl-lib)
+(require 'subr-x)
 
 (defgroup eshell-mode nil
   "This module contains code for handling input from the user."
@@ -105,7 +106,8 @@ eshell-send-direct-to-subprocesses
   "If t, send any input immediately to a subprocess."
   :type 'boolean)
 
-(defcustom eshell-expand-input-functions nil
+(defcustom eshell-expand-input-functions
+  '(eshell-expand-to-eshell-shell-command)
   "Functions to call before input is parsed.
 Each function is passed two arguments, which bounds the region of the
 current input text."
@@ -671,6 +673,81 @@ eshell-send-input
 	       (run-hooks 'eshell-post-command-hook)
 	       (insert-and-inherit input)))))))))
 
+(defun eshell-shell-command (command)
+  "Execute COMMAND using the operating system shell."
+  (throw 'eshell-replace-command
+         (with-connection-local-variables
+	  (eshell-parse-command shell-file-name
+			        (list shell-command-switch command)))))
+
+(defun eshell-expand-to-eshell-shell-command (beg end)
+  "Expand Eshell input starting with '||' to use `eshell-shell-command'.
+
+This is intended to provide a convenient way to bypass Eshell's
+own pipelining which can be slow when executing shell pipelines
+which move a lot of data.  It is also useful to avoid
+roundtripping data when running pipelines on remote hosts.
+
+For example, this function expands the input
+
+    || cat *.ogg | my-cool-decoder >file
+
+to
+
+    eshell-shell-command \"cat *.ogg | my-cool-decoder >file\"
+
+Executing the latter command will not copy all the data in the
+*.ogg files into Emacs buffers, as would normally happen with
+Eshell's own pipelining.
+
+This function tries to extract Eshell-specific redirects and
+avoids passing these to the operating system shell.  For example,
+
+    || foo | bar >>#<buffer scratch>
+
+will be expanded to
+
+    eshell-shell-command \"foo | bar\" >>#<buffer scratch>
+
+This function works well in combination with adding
+`eshell-restore-unexpanded-input' to `eshell-input-filter-functions'."
+  (save-excursion
+    (goto-char beg)
+    (when (looking-at "||\\s-*")
+      (let ((end (copy-marker end)) ; needs to be a marker
+            redirects)
+        ;; drop the ||
+        (delete-region beg (match-end 0))
+        ;; extract Eshell-specific redirects
+        (while (search-forward-regexp "[0-9]?>+&?[0-9]?\\s-*\\S-" nil t)
+          (let ((beg (match-beginning 0)))
+            (forward-char -1) ; start from the redirect target
+            (when-let ((end (cond
+                             ;; this is a redirect to a process or a
+                             ;; buffer
+                             ((looking-at "#<")
+                              (forward-char 1)
+                              (1+ (eshell-find-delimiter ?\< ?\>)))
+                             ;; this is a redirect to a virtual target
+                             ((and (looking-at "/\\S-+")
+                                   (assoc (match-string 0)
+                                          eshell-virtual-targets))
+                              (match-end 0)))))
+              (push (buffer-substring-no-properties beg end) redirects)
+              (delete-region beg end)
+              (just-one-space))))
+        ;; wrap the remaining text and reinstate Eshell redirects
+        (let ((pipeline (string-trim
+                         (buffer-substring-no-properties beg end))))
+          (delete-region beg end)
+          (insert "eshell-shell-command ")
+          (prin1 pipeline (current-buffer))
+          (when redirects
+            (insert " " (string-join (nreverse redirects) " "))))))))
+
+(custom-add-option 'eshell-expand-input-functions
+                   'eshell-expand-to-eshell-shell-command)
+
 (defsubst eshell-kill-new ()
   "Add the last input text to the kill ring."
   (kill-ring-save eshell-last-input-start eshell-last-input-end))
-- 
2.30.2