Re: Eshell's external pipe module interferes with other argument parsing hooks

Re: Eshell's external pipe module interferes with other argument parsing hooks

[Sean Whitton]

[switching debbugs -> emacs-devel for this subthread]

Hello,

On Thu 31 Mar 2022 at 01:58PM -07, Jim Porter wrote:

> Another possibility would be to keep the current behavior (or close to
> it), but to reconstruct the command to pass to `sh' during Eshell's
> rewrite phase. I'm not quite sure if that would actually work, but if it
> did, it would allow other argument parsers to run normally without
> extpipe needing to know what parsers to try. Perhaps if we kept around
> the substring that each argument parser consumed, it would be possible
> to reconstruct the relevant bits for extpipe's purposes?

Well, in your case (2), you don't want the other parsers to get a chance
to run -- that's the whole point.

> More generally though, maybe there are really two different use cases?
>
> 1) Eshell's built-in pipelines are slow because they go through Emacs
> buffers.
>
> 2) It would be convenient to invoke a whole command (or some large part
> of a command) using `sh' syntax.

These are both things that extpipe is meant to make easy, though I'm not
sure how separate they are -- often I want both.

> For (1), Eshell could opportunistically use external pipelines without
> any special syntax. It should be possible to tell just by looking at the
> parsed command form if "foo | bar" connects two external processes on
> the same host, and then perform the appropriate rewrite to connect the
> processes efficiently (e.g. using `sh -c'). This would happen after
> expansion of variables/subcommands, so to the user it would work just
> like any other Eshell command, but it would be faster.

This could just be added to Eshell right now, right?  Definitely useful.

> For (2), we'd need a convenient syntax for forwarding some command
> string to `sh'. Something like your proposed !! or || syntax, or maybe
> something to wrap around part of a command?

Yeah, extpipe's syntax covers most such cases but not quite all of them.

> (Or maybe an even something like an interactive `eshell-externalize'
> function that replaces the selected region with the correct `sh'
> invocation?)

I had something like this previously.  The problem is it makes editing
and re-running the command a fair bit more fiddly.

> And finally, sorry for bringing up these issues months after
> bug#46351.  At the time, I didn't really understand the internals of
> Eshell, so I didn't have anything of substance to say then. Since then
> I've delved a bit *too* deep into Eshell's internals while trying to
> prove to myself that my implementation of Lisp function pipelines is
> sufficiently-flexible. :)

No problem, but could I request that you spend a little more time
editing your messages for length?  And perhaps consider separating out
discussion of significant future possible enhancements from fixing bugs
with the existing code into separate bugs or ML threads, as I've done
with this message.  Thanks in advance :)

-- 
Sean Whitton

Re: Eshell's external pipe module interferes with other argument parsing hooks

[Jim Porter]

On 3/31/2022 2:56 PM, Sean Whitton wrote:
> On Thu 31 Mar 2022 at 01:58PM -07, Jim Porter wrote:
> 
>> Another possibility would be to keep the current behavior (or close to
>> it), but to reconstruct the command to pass to `sh' during Eshell's
>> rewrite phase. I'm not quite sure if that would actually work, but if it
>> did, it would allow other argument parsers to run normally without
>> extpipe needing to know what parsers to try. Perhaps if we kept around
>> the substring that each argument parser consumed, it would be possible
>> to reconstruct the relevant bits for extpipe's purposes?
> 
> Well, in your case (2), you don't want the other parsers to get a chance
> to run -- that's the whole point.

In practice, yes. However, the implementation could allow the other 
parsers to run, but then discard their parsed results during the rewrite 
phase if applicable. This is probably subject to a different set of 
unusual corner cases, but would allow other parsers to consume the parts 
of the command that they care about so that extpipe can just look for 
`*|' and friends. (A `*|' sequence inside quotes or something similar 
would already have been consumed by the time extpipe sees it.)

>> More generally though, maybe there are really two different use cases?
>>
>> 1) Eshell's built-in pipelines are slow because they go through Emacs
>> buffers.
>>
>> 2) It would be convenient to invoke a whole command (or some large part
>> of a command) using `sh' syntax.
> 
> These are both things that extpipe is meant to make easy, though I'm not
> sure how separate they are -- often I want both.
> 
>> For (1), Eshell could opportunistically use external pipelines without
>> any special syntax.
[snip]
> 
> This could just be added to Eshell right now, right?  Definitely useful.

Unless there's a reason for Eshell's current behavior that I'm not aware 
of, I can't think of any problems with doing this, so long as everything 
is escaped properly.

>> For (2), we'd need a convenient syntax for forwarding some command
>> string to `sh'. Something like your proposed !! or || syntax, or maybe
>> something to wrap around part of a command?
> 
> Yeah, extpipe's syntax covers most such cases but not quite all of them.

For the purposes of parsing, having the token that activates the extpipe 
module be at the beginning of the relevant portion would make things a 
lot easier. Then `eshell-parse-external-pipeline' can just check if 
that's the next token and if so, read until the end of the extpipe 
portion. That would eliminate all the complexity of trying to identify 
unquoted/literal `*|' operators.

In practice though, I'm happy with any syntax so long as the 
implementation is robust. If the current implementation using `*|' 
operators is significantly nicer to use (I don't have an opinion either 
way since I haven't used it enough), then we should stick with it, even 
if it makes it harder to implement.

> No problem, but could I request that you spend a little more time
> editing your messages for length?  And perhaps consider separating out
> discussion of significant future possible enhancements from fixing bugs
> with the existing code into separate bugs or ML threads, as I've done
> with this message.  Thanks in advance :)

I'll try. :) I usually prefer shorter messages myself, but I've had a 
hard time finding the right balance on the Emacs lists. Sometimes I 
think I'm keeping things focused, only to find that I haven't relayed 
enough information, and instead have just confused matters. Splitting 
these threads off is probably the right call though. Thanks.

Re: Eshell's external pipe module interferes with other argument parsing hooks

[Sean Whitton]

Hello,

On Thu 31 Mar 2022 at 04:11PM -07, Jim Porter wrote:

> In practice, yes. However, the implementation could allow the other
> parsers to run, but then discard their parsed results during the rewrite
> phase if applicable. This is probably subject to a different set of
> unusual corner cases, but would allow other parsers to consume the parts
> of the command that they care about so that extpipe can just look for
> `*|' and friends. (A `*|' sequence inside quotes or something similar
> would already have been consumed by the time extpipe sees it.)

Okay, I think I see what you mean.  Worth keeping in mind.

> For the purposes of parsing, having the token that activates the extpipe
> module be at the beginning of the relevant portion would make things a
> lot easier. Then `eshell-parse-external-pipeline' can just check if
> that's the next token and if so, read until the end of the extpipe
> portion. That would eliminate all the complexity of trying to identify
> unquoted/literal `*|' operators.
>
> In practice though, I'm happy with any syntax so long as the
> implementation is robust. If the current implementation using `*|'
> operators is significantly nicer to use (I don't have an opinion either
> way since I haven't used it enough), then we should stick with it, even
> if it makes it harder to implement.

I think it's pretty nice to use, indeed.  But one thing that isn't so
easy that I would additionally like to have is just converting a single
command, with no pipes or redirection, to use the external shell.  You
can do something like append "*| cat" to your command but that's not
nice.  One possibility discussed previously was to support putting || at
the very beginning of the command to request that the whole thing be
sent to the external shell.  I'm still thinking I might add that.

-- 
Sean Whitton

Re: Eshell's external pipe module interferes with other argument parsing hooks

[Jim Porter]

On 3/31/2022 4:11 PM, Jim Porter wrote:
> On 3/31/2022 2:56 PM, Sean Whitton wrote:
>> On Thu 31 Mar 2022 at 01:58PM -07, Jim Porter wrote:
>>> 1) Eshell's built-in pipelines are slow because they go through Emacs
>>> buffers.
[snip]
>>> For (1), Eshell could opportunistically use external pipelines without
>>> any special syntax.
> [snip]
>>
>> This could just be added to Eshell right now, right?  Definitely useful.
> 
> Unless there's a reason for Eshell's current behavior that I'm not aware 
> of, I can't think of any problems with doing this, so long as everything 
> is escaped properly.

On the subject of Eshell's built-in pipelines being slow, this might be 
of interest: 
<https://tdodge.consulting/blog/eshell/background-output-thread>. The 
author seems to have some interest in upstreaming the patch too[1].

I haven't had a chance to look at the patch in very much detail yet, but 
if this worked for all cases (apparently it's limited to PTY 
subprocesses for now, and I'm sure there are other cases where it would 
need to stay out of the way), it would be a huge improvement to 
"regular" Eshell pipes.

- Jim

[1] 
https://www.reddit.com/r/emacs/comments/usghki/comment/i93hkmv/?utm_source=reddit&utm_medium=web2x&context=3