bug#54603: 29.0.50; [PATCH] Eshell's external pipe module interferes with other argument parsing hooks

[Jim Porter <jporterbugs@gmail.com>]

On 3/31/2022 11:26 AM, Sean Whitton wrote:
> The call to `eshell-parse-lisp-argument' is meant to handle precisely
> your case.  It isn't doing atm, and I think it might actually be a bug
> over in that function.  To see this, with point on the first nonblank
> char of each of these lines, do 'M-: (eshell-parse-lisp-argument) RET':
> 
>      #'foo
> 
>      'foo
> 
> In the first case it successfully parses it and skips point forward, but
> in the latter case it does not.  But I think it should, right?

I'm not so sure. That would mean "'foo" in "echo 'foo" is treated as a 
Lisp form, but Eshell expects you to use "#'" (or "(" or "`") to 
introduce a Lisp form. As I understand it, that's so that typos don't do 
such surprising things. There's a good chance that a user typing "echo 
'foo" actually meant "echo 'foo'".

Because of that, in general, once you've moved point to partway inside a 
Lisp form, you can't use `eshell-parse-lisp-argument'. Maybe that should 
be changed? It's a bit surprising that you have to sharp-quote symbols 
in Eshell if you're not already inside a Lisp form (especially when 
those symbols aren't necessarily functions). Still, the typo-protection 
of the current implementation seems like a good feature...

That said, this isn't the only situation where "unbalanced" single 
quotes can occur in an Eshell command. For example, see this command:

   echo $(list "one" "two")(:s'o'x')

This creates a list of two strings, and then performs a regexp 
substitution from "o" to "x", so the output is:

   ("xne" "twx")

Under Emacs 27/28, this works correctly, but it fails in 29 for the same 
reason as the original issue. The extpipe module could account for this 
and try to parse argument predicates/modifiers so that it knows when to 
stay out of things, but then what about configurations where that module 
is disabled? (And for that matter, a third-party Eshell module could 
cause conflicts in a similar manner.)

>> If possible, I think it would be better for
>> `eshell-parse-external-pipeline' to solely focus on finding the
>> external pipe operators ("*|", "*<", and "*>")[1] and then for
>> `eshell-rewrite-external-pipeline' to prepare the command string to
>> pass to sh. This would also have the advantage[2] of making it
>> possible to support a richer set of Eshell features with external
>> pipes, such as the following:
>>
>>     ~ $ echo $(message "[%s]" "hi") *| cat
>>     zsh:1: command not found: message
>>
>> (If you remove the "*", this outputs "[hi]", and it should be
>> technically possible to make this work with external pipes too, provided
>> it executes the Lisp code before generating the command string for sh.)
> 
> In this case I would want the whole '$(message ..'  construction to go
> to the external shell.  Although extpipe supports some combinations of
> piping Eshell in and out of the external shell, fundamentally it's more
> about making it easier to bypass Eshell features than to make complex
> usage of them.  It's also simpler to understand as a user.  If you do
> want more involved combinations of Eshell and external shell commands,
> you can always do the 'sh -c' wrapping yourself.

 From my point of view, since the only difference between using an 
"Eshell pipe" and an extpipe is that the pipe operator has a "*", I'd 
expect the commands to work largely the same, except that the extpipe is 
faster. When I see a command like 'echo $(message "[%s]" "hi") *| cat', 
I usually think of it as a two-phase operation: expansions/subcommands 
are expanded first, and then the final command is executed. In that 
model, the expansions would still be "in Eshell".

On the other hand, maybe there's enough practical use for passing the 
raw command string to `sh' that there should be a very simple way of 
invoking it (and without having to mess around with escaping interior 
quotes, as you would if you used `sh -c' manually). Maybe the parsing 
would be more robust if it used special sigils for the start/end of the 
external command? I suppose that's similar to your original proposal of 
using !! or || to introduce an external pipeline, so maybe it's not 
feasible to go this route.

Another possibility would be to keep the current behavior (or close to 
it), but to reconstruct the command to pass to `sh' during Eshell's 
rewrite phase. I'm not quite sure if that would actually work, but if it 
did, it would allow other argument parsers to run normally without 
extpipe needing to know what parsers to try. Perhaps if we kept around 
the substring that each argument parser consumed, it would be possible 
to reconstruct the relevant bits for extpipe's purposes?

More generally though, maybe there are really two different use cases?

1) Eshell's built-in pipelines are slow because they go through Emacs 
buffers.

2) It would be convenient to invoke a whole command (or some large part 
of a command) using `sh' syntax.

For (1), Eshell could opportunistically use external pipelines without 
any special syntax. It should be possible to tell just by looking at the 
parsed command form if "foo | bar" connects two external processes on 
the same host, and then perform the appropriate rewrite to connect the 
processes efficiently (e.g. using `sh -c'). This would happen after 
expansion of variables/subcommands, so to the user it would work just 
like any other Eshell command, but it would be faster.

For (2), we'd need a convenient syntax for forwarding some command 
string to `sh'. Something like your proposed !! or || syntax, or maybe 
something to wrap around part of a command? (Or maybe an even something 
like an interactive `eshell-externalize' function that replaces the 
selected region with the correct `sh' invocation?)

And finally, sorry for bringing up these issues months after bug#46351. 
At the time, I didn't really understand the internals of Eshell, so I 
didn't have anything of substance to say then. Since then I've delved a 
bit *too* deep into Eshell's internals while trying to prove to myself 
that my implementation of Lisp function pipelines is 
sufficiently-flexible. :)