From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Sean Whitton Newsgroups: gmane.emacs.bugs Subject: bug#46351: 28.0.50; Add convenient way to bypass Eshell's own pipelining Date: Fri, 24 Dec 2021 14:20:27 -0700 Message-ID: <87tuex1yzo.fsf@melete.silentflame.com> References: <878s812c6a.fsf@melete.silentflame.com> <87eehsz170.fsf@gmx.de> <874kin1z2x.fsf@melete.silentflame.com> <87ft26etuh.fsf@gmx.de> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="39930"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Notmuch/0.31.4 (https://notmuchmail.org) Emacs/29.0.50 (x86_64-pc-linux-gnu) To: 46351@debbugs.gnu.org, Michael Albinus Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Fri Dec 24 22:21:29 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1n0s0F-000A82-Vz for geb-bug-gnu-emacs@m.gmane-mx.org; Fri, 24 Dec 2021 22:21:29 +0100 Original-Received: from localhost ([::1]:39480 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n0s0E-0003WQ-W5 for geb-bug-gnu-emacs@m.gmane-mx.org; Fri, 24 Dec 2021 16:21:27 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:60002) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n0rzq-0003W9-3g for bug-gnu-emacs@gnu.org; Fri, 24 Dec 2021 16:21:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:54853) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n0rzp-00041A-QW for bug-gnu-emacs@gnu.org; Fri, 24 Dec 2021 16:21:01 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n0rzp-0002Qq-Mo for bug-gnu-emacs@gnu.org; Fri, 24 Dec 2021 16:21:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Sean Whitton Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Fri, 24 Dec 2021 21:21:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46351 X-GNU-PR-Package: emacs Original-Received: via spool by 46351-submit@debbugs.gnu.org id=B46351.16403808409307 (code B ref 46351); Fri, 24 Dec 2021 21:21:01 +0000 Original-Received: (at 46351) by debbugs.gnu.org; 24 Dec 2021 21:20:40 +0000 Original-Received: from localhost ([127.0.0.1]:38166 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n0rzT-0002Q2-9i for submit@debbugs.gnu.org; Fri, 24 Dec 2021 16:20:39 -0500 Original-Received: from out4-smtp.messagingengine.com ([66.111.4.28]:40037) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n0rzO-0002Pj-20 for 46351@debbugs.gnu.org; Fri, 24 Dec 2021 16:20:37 -0500 Original-Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id B31925C003B; Fri, 24 Dec 2021 16:20:28 -0500 (EST) Original-Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Fri, 24 Dec 2021 16:20:28 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=spwhitton.name; h=from:to:subject:in-reply-to:references:date:message-id :mime-version:content-type; s=fm1; bh=0PT3cYT6OFZIVtOBz/d1QmmLaF 8nr7QJvE4qpPXmZzk=; b=ETAa71HWPQqJa+ROhAYlX/5rD+MOYElDPkwGDrL9pz 6k2kdr1TSgjGqZeFyaQErnPpiu4yhwtuFoULySMt3nCY6aBP7qZEVIzUVsCMIA/y aPVoEByN0tmrNUewpsyQZu6eE7V/voBzJ24VjuGsEkHsepJN60/L5CYytiKQ4yzy kVQXddqxL1OxMUMuPOgO+de3vaVhKw52/52A748u1QUbk+ApkbFYGwyjWEA2Y65F Sa390B0nN16Hdf08j0WvOusgCx5pUU9Jzs1YGysejnxGmSlYfof3capIYSZcN819 1ZlkW2SHg8eLXjLZRQzTm4Tm1dPW0FJvxPYTAMxXi8vA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=0PT3cY T6OFZIVtOBz/d1QmmLaF8nr7QJvE4qpPXmZzk=; b=KDyEz2HaCWsmbdxZyIIoBA xqnElPObJCU/SQHO3TNt2vNVfSsKcte1zmkFJeTpItja48rNx2HNcbq9WsX2jcRN wegvUqDdsiP8EP6UWjY4Wg9orUtLYpxwdHRMz/rfHWT6K4FE+ECorbB4Vv3rd1ZL hJQ62psl0YVFS0d9iyl4zOgml9Tl3ovT/gUljJE2yzUWP5ZY/oyy0xRcT85rHHdN 3FcByvQJJd4EW14oT+hemGmZxcR+q7/TXzJBsRareS5EJ9y1ck4vgEdYJTQdZNA4 pcCZAUIkKk6tz18MbcPYV1WSkYsGZmw72W1bqCYYYHNjBeT7fTST7vb137cTIp5A == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddruddutddgudeghecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffujghffgffkfggtgesmhdttdertdertdenucfhrhhomhepufgvrghn ucghhhhithhtohhnuceoshhpfihhihhtthhonhesshhpfihhihhtthhonhdrnhgrmhgvqe enucggtffrrghtthgvrhhnpefgueeutedvkeffgedvveetheejieetvdduvdffheevgfet vdfghfetteffvdevjeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehsphifhhhithhtohhnsehsphifhhhithhtohhnrdhnrghmvg X-ME-Proxy: Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 24 Dec 2021 16:20:28 -0500 (EST) Original-Received: by melete.silentflame.com (Postfix, from userid 1000) id 5EAF77F3128; Fri, 24 Dec 2021 14:20:27 -0700 (MST) In-Reply-To: <87ft26etuh.fsf@gmx.de> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:223071 Archived-At: --=-=-= Content-Type: text/plain Hello, On Mon 08 Feb 2021 at 11:28AM +01, Michael Albinus wrote: > Maybe we could add it to the defaults. But at least I would > like to see it documented in the eshell manual, otherwise nobody will > know the effect of "!!". I'm attaching new patches which - switch !! to ||, because !! in shells typically has something to do with rerunning the last command - activate the syntax by default -- I do use Eshell regularly myself, and I don't believe that activating it by default is likely to disrupt anyone's usage - adds a section to the Eshell manual introducing the new syntax. > And I have the impression, that the eshell history is not preserved any > longer, when I have applied these settings. I have not been able to reproduce this -- can you confirm? > It doesn't work for me. The reason is a little bit subtle: you use > `shell-file-name' in order to call a remote shell. In my case, it is > "/usr/local/bin/tcsh", which doesn't exist on the remote machine I've > used for testing. I was able to fix this by using `with-connection-local-variables'. Thank you for pointing out the problem. -- Sean Whitton --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=v2-0001-Add-eshell-restore-unexpanded-input.patch >From d47446cbe7db5c7fb4eba71ab022cfae3de07799 Mon Sep 17 00:00:00 2001 From: Sean Whitton Date: Sat, 6 Feb 2021 00:01:50 -0700 Subject: [PATCH v2 1/2] Add eshell-restore-unexpanded-input * lisp/eshell/esh-mode.el (eshell-send-input): Store the original input before running eshell-expand-input-functions. * lisp/eshell/esh-mode.el (eshell-restore-unexpanded-input): Define new function and register as a customization option for eshell-input-filter-functions. --- lisp/eshell/esh-mode.el | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/lisp/eshell/esh-mode.el b/lisp/eshell/esh-mode.el index cae5236d89..87166ef3bf 100644 --- a/lisp/eshell/esh-mode.el +++ b/lisp/eshell/esh-mode.el @@ -62,6 +62,7 @@ (require 'esh-module) (require 'esh-cmd) (require 'esh-arg) ;For eshell-parse-arguments +(require 'cl-lib) (defgroup eshell-mode nil "This module contains code for handling input from the user." @@ -197,6 +198,7 @@ eshell-first-time-p ;; byte-compiler, when compiling other files which `require' this one (defvar eshell-mode nil) (defvar eshell-command-running-string "--") +(defvar eshell-last-unexpanded-input nil) (defvar eshell-last-input-start nil) (defvar eshell-last-input-end nil) (defvar eshell-last-output-start nil) @@ -641,6 +643,7 @@ eshell-send-input (progn (setq input (buffer-substring-no-properties eshell-last-output-end (1- (point)))) + (setq eshell-last-unexpanded-input input) (run-hook-with-args 'eshell-expand-input-functions eshell-last-output-end (1- (point))) (let ((cmd (eshell-parse-command-input @@ -674,6 +677,23 @@ eshell-kill-new (custom-add-option 'eshell-input-filter-functions 'eshell-kill-new) +(defun eshell-restore-unexpanded-input () + "Restore the input text before `eshell-expand-input-functions' ran. +Useful when you want to see the unexpanded input rather than the +expanded input in the Eshell buffer, and when you want later +entries in `eshell-input-filter-functions' to use the unexpanded +input." + (setf (buffer-substring eshell-last-input-start + (1- eshell-last-input-end)) + eshell-last-unexpanded-input) + ;; We have to move point for compatibility with smart display. + (save-excursion + (goto-char eshell-last-input-end) + (eshell-update-markers eshell-last-output-end))) + +(custom-add-option 'eshell-input-filter-functions + 'eshell-restore-unexpanded-input) + (defun eshell-output-filter (process string) "Send the output from PROCESS (STRING) to the interactive display. This is done after all necessary filtering has been done." -- 2.30.2 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=v2-0002-Add-eshell-shell-command-and-eshell-expand-to-esh.patch >From 98df5163018b9c99ad8e65f03422f908c6068c78 Mon Sep 17 00:00:00 2001 From: Sean Whitton Date: Sat, 6 Feb 2021 00:48:32 -0700 Subject: [PATCH v2 2/2] Add eshell-shell-command and eshell-expand-to-eshell-shell-command * lisp/eshell/esh-mode.el (eshell-shell-command, eshell-expand-to-eshell-shell-command): Define new functions. Register eshell-expand-to-eshell-shell-command as a customization option for eshell-expand-input-functions, and add it by default. * etc/NEWS: * doc/misc/eshell.texi: Document the new syntax. --- doc/misc/eshell.texi | 35 ++++++++++++++++++ etc/NEWS | 9 +++++ lisp/eshell/esh-mode.el | 79 ++++++++++++++++++++++++++++++++++++++++- 3 files changed, 122 insertions(+), 1 deletion(-) diff --git a/doc/misc/eshell.texi b/doc/misc/eshell.texi index a87dd4308c..7e9233c09e 100644 --- a/doc/misc/eshell.texi +++ b/doc/misc/eshell.texi @@ -875,6 +875,7 @@ Expansion @menu * Dollars Expansion:: * Globbing:: +* Running Shell Pipelines Natively:: @end menu @node Dollars Expansion @@ -945,6 +946,40 @@ Globbing The GNU Emacs Manual}.} groups ``eshell-glob'' and ``eshell-pred''. +@node Running Shell Pipelines Natively +@section Running Shell Pipelines Natively +When constructing shell pipelines that will move a lot of data, it is +a good idea to bypass Eshell's own pipelining support and use the +operating system shell's instead. This is especially relevant when +executing commands on a remote machine using Eshell's Tramp +integration: using the remote shell's pipelining avoids copying the +data which will flow through the pipeline to local Emacs buffers and +then right back again. + +To quickly construct a command for the native shell, prefix your +Eshell input with the string @code{||}. For example, + +@example +|| cat *.ogg | my-cool-decoder >file +@end example + +Executing this command will not copy all the data in the *.ogg files +into Emacs buffers, as would normally happen with Eshell's own +pipelining. + +As this feature is implemented as an expansion, your original input +will be replaced with something like this: + +@example +eshell-shell-command \"cat *.ogg | my-cool-decoder >file\" +@end example + +This makes it clear what Eshell is doing, but has the disadvantage of +being harder to edit, especially for complex pipelines with a lot of +quoting. If you would prefer that the @code{||}-prefixed input be +restored, customize the variable @code{eshell-input-filter-functions} +to include the function @code{eshell-restore-unexpanded-input}. + @node Input/Output @chapter Input/Output Since Eshell does not communicate with a terminal like most command diff --git a/etc/NEWS b/etc/NEWS index 57fe40c488..767763098b 100644 --- a/etc/NEWS +++ b/etc/NEWS @@ -749,6 +749,15 @@ the Netscape web browser was released in February, 2008. This support has been obsolete since Emacs 25.1. The final version of the Galeon web browser was released in September, 2008. +** Eshell + ++++ +*** New feature to easily bypass Eshell's own pipelining. +Prefixing commands with '||' causes them to be executed using the +operating system shell, which is particularly useful to bypass +Eshell's own pipelining for pipelines which will move a lot of data. +See "Running Shell Pipelines Natively" in the Eshell manual. + * New Modes and Packages in Emacs 29.1 diff --git a/lisp/eshell/esh-mode.el b/lisp/eshell/esh-mode.el index 87166ef3bf..04680e4305 100644 --- a/lisp/eshell/esh-mode.el +++ b/lisp/eshell/esh-mode.el @@ -63,6 +63,7 @@ (require 'esh-cmd) (require 'esh-arg) ;For eshell-parse-arguments (require 'cl-lib) +(require 'subr-x) (defgroup eshell-mode nil "This module contains code for handling input from the user." @@ -105,7 +106,8 @@ eshell-send-direct-to-subprocesses "If t, send any input immediately to a subprocess." :type 'boolean) -(defcustom eshell-expand-input-functions nil +(defcustom eshell-expand-input-functions + '(eshell-expand-to-eshell-shell-command) "Functions to call before input is parsed. Each function is passed two arguments, which bounds the region of the current input text." @@ -671,6 +673,81 @@ eshell-send-input (run-hooks 'eshell-post-command-hook) (insert-and-inherit input))))))))) +(defun eshell-shell-command (command) + "Execute COMMAND using the operating system shell." + (throw 'eshell-replace-command + (with-connection-local-variables + (eshell-parse-command shell-file-name + (list shell-command-switch command))))) + +(defun eshell-expand-to-eshell-shell-command (beg end) + "Expand Eshell input starting with '||' to use `eshell-shell-command'. + +This is intended to provide a convenient way to bypass Eshell's +own pipelining which can be slow when executing shell pipelines +which move a lot of data. It is also useful to avoid +roundtripping data when running pipelines on remote hosts. + +For example, this function expands the input + + || cat *.ogg | my-cool-decoder >file + +to + + eshell-shell-command \"cat *.ogg | my-cool-decoder >file\" + +Executing the latter command will not copy all the data in the +*.ogg files into Emacs buffers, as would normally happen with +Eshell's own pipelining. + +This function tries to extract Eshell-specific redirects and +avoids passing these to the operating system shell. For example, + + || foo | bar >># + +will be expanded to + + eshell-shell-command \"foo | bar\" >># + +This function works well in combination with adding +`eshell-restore-unexpanded-input' to `eshell-input-filter-functions'." + (save-excursion + (goto-char beg) + (when (looking-at "||\\s-*") + (let ((end (copy-marker end)) ; needs to be a marker + redirects) + ;; drop the || + (delete-region beg (match-end 0)) + ;; extract Eshell-specific redirects + (while (search-forward-regexp "[0-9]?>+&?[0-9]?\\s-*\\S-" nil t) + (let ((beg (match-beginning 0))) + (forward-char -1) ; start from the redirect target + (when-let ((end (cond + ;; this is a redirect to a process or a + ;; buffer + ((looking-at "#<") + (forward-char 1) + (1+ (eshell-find-delimiter ?\< ?\>))) + ;; this is a redirect to a virtual target + ((and (looking-at "/\\S-+") + (assoc (match-string 0) + eshell-virtual-targets)) + (match-end 0))))) + (push (buffer-substring-no-properties beg end) redirects) + (delete-region beg end) + (just-one-space)))) + ;; wrap the remaining text and reinstate Eshell redirects + (let ((pipeline (string-trim + (buffer-substring-no-properties beg end)))) + (delete-region beg end) + (insert "eshell-shell-command ") + (prin1 pipeline (current-buffer)) + (when redirects + (insert " " (string-join (nreverse redirects) " ")))))))) + +(custom-add-option 'eshell-expand-input-functions + 'eshell-expand-to-eshell-shell-command) + (defsubst eshell-kill-new () "Add the last input text to the kill ring." (kill-ring-save eshell-last-input-start eshell-last-input-end)) -- 2.30.2 --=-=-=--