From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Jim Porter Newsgroups: gmane.emacs.bugs Subject: bug#54603: 29.0.50; [PATCH] Eshell's external pipe module interferes with other argument parsing hooks Date: Thu, 31 Mar 2022 13:58:11 -0700 Message-ID: References: <70677cd2-f741-16d1-b38f-c39b507cc95e@gmail.com> <871qyij7vx.fsf@melete.silentflame.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="15391"; mail-complaints-to="usenet@ciao.gmane.io" To: Sean Whitton , 54603@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Mar 31 22:59:09 2022 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1na1sr-0003tM-ES for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 31 Mar 2022 22:59:09 +0200 Original-Received: from localhost ([::1]:34330 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1na1sq-0007Vr-DR for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 31 Mar 2022 16:59:08 -0400 Original-Received: from eggs.gnu.org ([209.51.188.92]:51176) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1na1sk-0007Vd-UE for bug-gnu-emacs@gnu.org; Thu, 31 Mar 2022 16:59:02 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:46080) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1na1sk-000260-Lw for bug-gnu-emacs@gnu.org; Thu, 31 Mar 2022 16:59:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1na1sk-0001NL-Hh for bug-gnu-emacs@gnu.org; Thu, 31 Mar 2022 16:59:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Jim Porter Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 31 Mar 2022 20:59:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54603 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 54603-submit@debbugs.gnu.org id=B54603.16487603025236 (code B ref 54603); Thu, 31 Mar 2022 20:59:02 +0000 Original-Received: (at 54603) by debbugs.gnu.org; 31 Mar 2022 20:58:22 +0000 Original-Received: from localhost ([127.0.0.1]:39977 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1na1s5-0001MO-Sq for submit@debbugs.gnu.org; Thu, 31 Mar 2022 16:58:22 -0400 Original-Received: from mail-pg1-f170.google.com ([209.85.215.170]:39850) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1na1s3-0001M8-CO for 54603@debbugs.gnu.org; Thu, 31 Mar 2022 16:58:19 -0400 Original-Received: by mail-pg1-f170.google.com with SMTP id q19so730485pgm.6 for <54603@debbugs.gnu.org>; Thu, 31 Mar 2022 13:58:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=subject:to:references:from:message-id:date:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=y1aGsrLtJJr3MZwlGTLh3G2uelnICkSiqnx0Zb9wcwQ=; b=BI6n1DLYDx9hyWBN9J8E7kMk9U+l5LHl5iNeaXLXinkRAJ35qYw3A4Aj5WNPYvLlZR dhlNBvRCpuX+UEWaLno3ESR23qdordUwKnb5wCL/4rrsOANP6e5noDeTiWozdSGo5jjO 6kvjaEgh/rArS8O/rYx/KQIoaISBYY0VEaCUahWE7AYHN+HpKsmUaQ2da0QNrIG3UmKm 6s5iPcWLFclDMyHJtso3yCY1wnaSPBJIy7TIAQItaWlghZA9WItjDWgufI2jx6o/PF5K kfuWcG6zf7vtyUYTDYZM/ItPXltLr0nV4hPLq146VruwGgTEfxMPSXIbCDbmPV/gkD3Q RpZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:subject:to:references:from:message-id:date :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=y1aGsrLtJJr3MZwlGTLh3G2uelnICkSiqnx0Zb9wcwQ=; b=307Dllnc4wgYy99eQF0d0lOKqxIijsTnRjtvPo0bBkVGd55k7S07bwJG8TH4Cc9m97 8i7vpo8jn+0fWAV4dSj0OSIzgre+pXYRthtmllIlLm8yBhPSH19tVmvWtvPx1KQiPmXk nShAq9en3r0LCCIharmEM84xg6SjW7Z1d00OkeSx50xkWafbiOeW1MfS9dazrYzC3tlW BsEjfUXeNaRR3sOfl0NZaTZ0wEAHDCs64gc7o9w30OPEw6fAAXFO9SZtJeBe6PG7w+SU wK5G2UaZJl6sg9pHiEMnr9ygpO+9LKLq6n28/8HQsAh4H+aOwEp33JhTK5mIduqSSBuA YpLA== X-Gm-Message-State: AOAM531DWhkv2tTq26VI4GNbDJ+hpVWqqy0yTpSuPLPl3b8/ylYHJHnm o3NPc9oiRH7Vrqq4tyzzIzSgN3uwRQ0= X-Google-Smtp-Source: ABdhPJwusfulxEqwDiJ3GB+0Vi0IgLo3phwEt5/f4Tc5xWpgRDlT8t/iwW/cbl3TCPYMjH950GUPoQ== X-Received: by 2002:a63:e70f:0:b0:380:d919:beb with SMTP id b15-20020a63e70f000000b00380d9190bebmr12166864pgi.58.1648760293273; Thu, 31 Mar 2022 13:58:13 -0700 (PDT) Original-Received: from [192.168.1.2] (cpe-76-168-148-233.socal.res.rr.com. [76.168.148.233]) by smtp.googlemail.com with ESMTPSA id i6-20020a633c46000000b003817d623f72sm239267pgn.24.2022.03.31.13.58.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 31 Mar 2022 13:58:12 -0700 (PDT) In-Reply-To: <871qyij7vx.fsf@melete.silentflame.com> Content-Language: en-US X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:229183 Archived-At: On 3/31/2022 11:26 AM, Sean Whitton wrote: > The call to `eshell-parse-lisp-argument' is meant to handle precisely > your case. It isn't doing atm, and I think it might actually be a bug > over in that function. To see this, with point on the first nonblank > char of each of these lines, do 'M-: (eshell-parse-lisp-argument) RET': > > #'foo > > 'foo > > In the first case it successfully parses it and skips point forward, but > in the latter case it does not. But I think it should, right? I'm not so sure. That would mean "'foo" in "echo 'foo" is treated as a Lisp form, but Eshell expects you to use "#'" (or "(" or "`") to introduce a Lisp form. As I understand it, that's so that typos don't do such surprising things. There's a good chance that a user typing "echo 'foo" actually meant "echo 'foo'". Because of that, in general, once you've moved point to partway inside a Lisp form, you can't use `eshell-parse-lisp-argument'. Maybe that should be changed? It's a bit surprising that you have to sharp-quote symbols in Eshell if you're not already inside a Lisp form (especially when those symbols aren't necessarily functions). Still, the typo-protection of the current implementation seems like a good feature... That said, this isn't the only situation where "unbalanced" single quotes can occur in an Eshell command. For example, see this command: echo $(list "one" "two")(:s'o'x') This creates a list of two strings, and then performs a regexp substitution from "o" to "x", so the output is: ("xne" "twx") Under Emacs 27/28, this works correctly, but it fails in 29 for the same reason as the original issue. The extpipe module could account for this and try to parse argument predicates/modifiers so that it knows when to stay out of things, but then what about configurations where that module is disabled? (And for that matter, a third-party Eshell module could cause conflicts in a similar manner.) >> If possible, I think it would be better for >> `eshell-parse-external-pipeline' to solely focus on finding the >> external pipe operators ("*|", "*<", and "*>")[1] and then for >> `eshell-rewrite-external-pipeline' to prepare the command string to >> pass to sh. This would also have the advantage[2] of making it >> possible to support a richer set of Eshell features with external >> pipes, such as the following: >> >> ~ $ echo $(message "[%s]" "hi") *| cat >> zsh:1: command not found: message >> >> (If you remove the "*", this outputs "[hi]", and it should be >> technically possible to make this work with external pipes too, provided >> it executes the Lisp code before generating the command string for sh.) > > In this case I would want the whole '$(message ..' construction to go > to the external shell. Although extpipe supports some combinations of > piping Eshell in and out of the external shell, fundamentally it's more > about making it easier to bypass Eshell features than to make complex > usage of them. It's also simpler to understand as a user. If you do > want more involved combinations of Eshell and external shell commands, > you can always do the 'sh -c' wrapping yourself. From my point of view, since the only difference between using an "Eshell pipe" and an extpipe is that the pipe operator has a "*", I'd expect the commands to work largely the same, except that the extpipe is faster. When I see a command like 'echo $(message "[%s]" "hi") *| cat', I usually think of it as a two-phase operation: expansions/subcommands are expanded first, and then the final command is executed. In that model, the expansions would still be "in Eshell". On the other hand, maybe there's enough practical use for passing the raw command string to `sh' that there should be a very simple way of invoking it (and without having to mess around with escaping interior quotes, as you would if you used `sh -c' manually). Maybe the parsing would be more robust if it used special sigils for the start/end of the external command? I suppose that's similar to your original proposal of using !! or || to introduce an external pipeline, so maybe it's not feasible to go this route. Another possibility would be to keep the current behavior (or close to it), but to reconstruct the command to pass to `sh' during Eshell's rewrite phase. I'm not quite sure if that would actually work, but if it did, it would allow other argument parsers to run normally without extpipe needing to know what parsers to try. Perhaps if we kept around the substring that each argument parser consumed, it would be possible to reconstruct the relevant bits for extpipe's purposes? More generally though, maybe there are really two different use cases? 1) Eshell's built-in pipelines are slow because they go through Emacs buffers. 2) It would be convenient to invoke a whole command (or some large part of a command) using `sh' syntax. For (1), Eshell could opportunistically use external pipelines without any special syntax. It should be possible to tell just by looking at the parsed command form if "foo | bar" connects two external processes on the same host, and then perform the appropriate rewrite to connect the processes efficiently (e.g. using `sh -c'). This would happen after expansion of variables/subcommands, so to the user it would work just like any other Eshell command, but it would be faster. For (2), we'd need a convenient syntax for forwarding some command string to `sh'. Something like your proposed !! or || syntax, or maybe something to wrap around part of a command? (Or maybe an even something like an interactive `eshell-externalize' function that replaces the selected region with the correct `sh' invocation?) And finally, sorry for bringing up these issues months after bug#46351. At the time, I didn't really understand the internals of Eshell, so I didn't have anything of substance to say then. Since then I've delved a bit *too* deep into Eshell's internals while trying to prove to myself that my implementation of Lisp function pipelines is sufficiently-flexible. :)