From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Sean Whitton Newsgroups: gmane.emacs.bugs Subject: bug#46351: 28.0.50; Add convenient way to bypass Eshell's own pipelining Date: Sat, 06 Feb 2021 13:06:53 -0700 Message-ID: <878s812c6a.fsf@melete.silentflame.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="40632"; mail-complaints-to="usenet@ciao.gmane.io" To: 46351@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sat Feb 06 21:07:43 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1l8Trr-000ASB-Q0 for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 06 Feb 2021 21:07:43 +0100 Original-Received: from localhost ([::1]:49216 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l8Trq-00024c-69 for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 06 Feb 2021 15:07:42 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:37162) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l8TrC-00021H-Cm for bug-gnu-emacs@gnu.org; Sat, 06 Feb 2021 15:07:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:34961) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l8TrC-00042w-3R for bug-gnu-emacs@gnu.org; Sat, 06 Feb 2021 15:07:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l8TrB-0006G2-UT for bug-gnu-emacs@gnu.org; Sat, 06 Feb 2021 15:07:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Sean Whitton Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 06 Feb 2021 20:07:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 46351 X-GNU-PR-Package: emacs X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.161264197423995 (code B ref -1); Sat, 06 Feb 2021 20:07:01 +0000 Original-Received: (at submit) by debbugs.gnu.org; 6 Feb 2021 20:06:14 +0000 Original-Received: from localhost ([127.0.0.1]:46507 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l8TqQ-0006Ew-3Q for submit@debbugs.gnu.org; Sat, 06 Feb 2021 15:06:14 -0500 Original-Received: from lists.gnu.org ([209.51.188.17]:41388) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l8TqK-0006Eg-7c for submit@debbugs.gnu.org; Sat, 06 Feb 2021 15:06:12 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:37002) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l8TqJ-0001CF-Ug for bug-gnu-emacs@gnu.org; Sat, 06 Feb 2021 15:06:08 -0500 Original-Received: from wout1-smtp.messagingengine.com ([64.147.123.24]:39175) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l8TqE-0003WB-Kn for bug-gnu-emacs@gnu.org; Sat, 06 Feb 2021 15:06:07 -0500 Original-Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id F182FD43 for ; Sat, 6 Feb 2021 15:06:00 -0500 (EST) Original-Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Sat, 06 Feb 2021 15:06:01 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=spwhitton.name; h=from:to:subject:date:message-id:mime-version:content-type; s= fm2; bh=xFVppfuaA/xSv23xhAoBA9UZEx2Wo1lsi1ufwdwgpMM=; b=QGSrkhHY Yv10jcwaOND6I5REHP2wQGwsv19RbMGPE03KhZKXYENFgZUZCui4rqW0Tkha8TXK OkItXe9Fk3tXD5Bt6zkZ2HA3IexRLHb31M1F6QVaUET+oJmoJGoEZ7pRjmJrq2hv XxLKn7AwGtB7yb84bmyPqdDHaQIuAwEkpygVsOBw04zdsNsNHCfald2Q1au2uGhJ Mk18KuW/7M9lkwK9JgeUmwFD4nMqJKE+/b7nx9Mk/RaqLH+4uWOi/Ntz9A/nNKx/ zfAyHn+pyzORBLNpNIrCjy+WtqmQSgv1tV2rmzrifrWK4xo8KnaxnwWISTpyYOM1 6T+lFnVfQgBl7A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm2; bh=xFVppfuaA/xSv23xhAoBA9UZEx2Wo 1lsi1ufwdwgpMM=; b=AEOEE9LgoSEYjTxB6ZtSbJJplGhJaspOoSUPiyWZetDwL MsQ0+bwtlARVBi9ezNrmMz0ICPkhdhlrDEFIo+mIb+/ivDCWzy03Y5zFM463K18g 2lM7h7b/Lyyep6GhtRfBXFp6R9e1Q5v4s+PPfh9QnvqsEcReDqJg8MkjpCWtidXZ 0zn8LkcA39bghTSE8AkAmfZ3BQdMM6Z4EkPQVE+9VJ1B0hVt/gN8tKJoopcgIDKe fAwOb2Rb2NWykHli0k/lr2JTWBAj+CJ6CEz3+cKk9eLw7K2lVjSzUJJ0v59aVBcX pd8MoyyUlBOfOdAMAXBbcWZokWPPXRJKg3frvzL8A== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrgeekgddufeefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkgggtsehmtderredttd dtnecuhfhrohhmpefuvggrnhcuhghhihhtthhonhcuoehsphifhhhithhtohhnsehsphif hhhithhtohhnrdhnrghmvgeqnecuggftrfgrthhtvghrnhepteegtdejheejvdehgeettd dvjeffudetuedutdegkefhtdetteeuueetkeefkeegnecuvehluhhsthgvrhfuihiivgep tdenucfrrghrrghmpehmrghilhhfrhhomhepshhpfihhihhtthhonhesshhpfihhihhtth honhdrnhgrmhgv X-ME-Proxy: Received-SPF: pass client-ip=64.147.123.24; envelope-from=spwhitton@spwhitton.name; helo=wout1-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:199459 Archived-At: --=-=-= Content-Type: text/plain Most people who use Eshell a lot need to be aware of the limitations of its support for shell pipelines: primarily, that all data in pipelines has to go through Emacs buffers, and that can be slow. If you're running pipelines which will move a lot of data, you should use a system shell which will be more performant. Currently, if Eshell is your primary shell, it's not particularly convenient to switch to using an operating system shell when you're preparing a pipeline that you know is going to move a lot of data. You would need either to switch to a shell-mode buffer, or quote and escape your Eshell input and put something like `bash -c' in front of it. I think that it would be good to have a way to easily toggle Eshell's own pipelining support on and off for particular commands. I have come up with the attached patches. When the new functions I've defined are enabled, you can type !! foo | bar 'arg' >baz >># and it will be executed by Eshell as if you had typed bash -c 'foo | bar '"'"'arg'"'"' >baz' >># The idea is that you can easily toggle Eshell's pipelining on and off as appropriate to your needs just by adding and removing the "!!" prefix. I think that the patches I've prepared are a clean implementation of this feature that would be good to include in Emacs. -- Sean Whitton --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0001-Add-eshell-restore-unexpanded-input.patch >From ccc4427ff06bf6dc63840517aefbb7fb899d4b8c Mon Sep 17 00:00:00 2001 From: Sean Whitton Date: Sat, 6 Feb 2021 00:01:50 -0700 Subject: [PATCH 1/2] Add eshell-restore-unexpanded-input * lisp/eshell/esh-mode.el (eshell-send-input): Store the original input before running eshell-expand-input-functions. * lisp/eshell/esh-mode.el (eshell-restore-unexpanded-input): Define new function and register as a customization option for eshell-input-filter-functions. --- lisp/eshell/esh-mode.el | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/lisp/eshell/esh-mode.el b/lisp/eshell/esh-mode.el index d29b010ea0..6036829941 100644 --- a/lisp/eshell/esh-mode.el +++ b/lisp/eshell/esh-mode.el @@ -197,6 +197,7 @@ This is used by `eshell-watch-for-password-prompt'." ;; byte-compiler, when compiling other files which `require' this one (defvar eshell-mode nil) (defvar eshell-command-running-string "--") +(defvar eshell-last-unexpanded-input nil) (defvar eshell-last-input-start nil) (defvar eshell-last-input-end nil) (defvar eshell-last-output-start nil) @@ -631,6 +632,7 @@ newline." (progn (setq input (buffer-substring-no-properties eshell-last-output-end (1- (point)))) + (setq eshell-last-unexpanded-input input) (run-hook-with-args 'eshell-expand-input-functions eshell-last-output-end (1- (point))) (let ((cmd (eshell-parse-command-input @@ -664,6 +666,21 @@ newline." (custom-add-option 'eshell-input-filter-functions 'eshell-kill-new) +(defun eshell-restore-unexpanded-input () + "Restore the input text before `eshell-expand-input-functions' ran. +Useful when you want to see the unexpanded input rather than the +expanded input in the Eshell buffer, and when you want later +entries in `eshell-input-filter-functions' to use the unexpanded +input. For example, you might want the unexpanded input to be +what gets stored in the history list." + (setf (buffer-substring eshell-last-input-start + (1- eshell-last-input-end)) + eshell-last-unexpanded-input) + (eshell-update-markers eshell-last-output-end)) + +(custom-add-option 'eshell-input-filter-functions + 'eshell-restore-unexpanded-input) + (defun eshell-output-filter (process string) "Send the output from PROCESS (STRING) to the interactive display. This is done after all necessary filtering has been done." -- 2.29.2 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0002-Add-eshell-shell-command-and-eshell-expand-to-eshell.patch >From d5d48427c75555cd5cf2f6eb5b89e0c832cf7edf Mon Sep 17 00:00:00 2001 From: Sean Whitton Date: Sat, 6 Feb 2021 00:48:32 -0700 Subject: [PATCH 2/2] Add eshell-shell-command and eshell-expand-to-eshell-shell-command * lisp/eshell/esh-mode.el (eshell-shell-command, eshell-expand-to-eshell-shell-command): Define new functions. Register eshell-expand-to-eshell-shell-command as a customization option for eshell-expand-input-functions. --- lisp/eshell/esh-mode.el | 77 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/lisp/eshell/esh-mode.el b/lisp/eshell/esh-mode.el index 6036829941..b3132393a5 100644 --- a/lisp/eshell/esh-mode.el +++ b/lisp/eshell/esh-mode.el @@ -62,6 +62,7 @@ (require 'esh-module) (require 'esh-cmd) (require 'esh-arg) ;For eshell-parse-arguments +(require 'subr-x) (defgroup eshell-mode nil "This module contains code for handling input from the user." @@ -660,6 +661,82 @@ newline." (run-hooks 'eshell-post-command-hook) (insert-and-inherit input))))))))) +(defun eshell-shell-command (command) + "Execute COMMAND using the operating system shell." + (throw 'eshell-replace-command + (eshell-parse-command shell-file-name + (list shell-command-switch command)))) + +(defun eshell-expand-to-eshell-shell-command (beg end) + "Expand Eshell input starting with '!!' to use `eshell-shell-command'. + +This is intended to provide a convenient way to bypass Eshell's +own pipelining which can be slow when executing shell pipelines +which move a lot of data. It is also useful to avoid +roundtripping data when running pipelines on remote hosts. + +To enable, add this function to `eshell-expand-input-functions'. + +For example, this function expands the input + + !! cat *.ogg | my-cool-decoder >file + +to + + eshell-shell-command \"cat *.ogg | my-cool-decoder >file\" + +Executing the latter command will not copy all the data in the +*.ogg files into Emacs buffers, as would normally happen with +Eshell's own pipelining. + +This function tries to extract Eshell-specific redirects and +avoids passing these to the operating system shell. For example, + + !! foo | bar >># + +will be expanded to + + eshell-shell-command \"foo | bar\" >># + +This function works well in combination with adding +`eshell-restore-unexpanded-input' to `eshell-input-filter-functions'." + (save-excursion + (goto-char beg) + (when (looking-at "!!\\s-*") + (let ((end (copy-marker end)) ; needs to be a marker + redirects) + ;; drop the !! + (delete-region beg (match-end 0)) + ;; extract Eshell-specific redirects + (while (search-forward-regexp "[0-9]?>+&?[0-9]?\\s-*\\S-" nil t) + (let ((beg (match-beginning 0))) + (forward-char -1) ; start from the redirect target + (when-let ((end (cond + ;; this is a redirect to a process or a + ;; buffer + ((looking-at "#<") + (forward-char 1) + (1+ (eshell-find-delimiter ?\< ?\>))) + ;; this is a redirect to a virtual target + ((and (looking-at "/\\S-+") + (assoc (match-string 0) + eshell-virtual-targets)) + (match-end 0))))) + (push (buffer-substring-no-properties beg end) redirects) + (delete-region beg end) + (just-one-space)))) + ;; wrap the remaining text and reinstate Eshell redirects + (let ((pipeline (string-trim + (buffer-substring-no-properties beg end)))) + (delete-region beg end) + (insert "eshell-shell-command ") + (prin1 pipeline (current-buffer)) + (when redirects + (insert " " (string-join (nreverse redirects) " ")))))))) + +(custom-add-option 'eshell-expand-input-functions + 'eshell-expand-to-eshell-shell-command) + (defsubst eshell-kill-new () "Add the last input text to the kill ring." (kill-ring-save eshell-last-input-start eshell-last-input-end)) -- 2.29.2 --=-=-=--