bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Ihor Radchenko]

Hello,

On Linux, if I put

echo foo

into scratch, select it, and run M-x shell-command-on-region sh,
I get "foo" in echo area.

However, on Windows, the result of M-x shell-command-on-region cmdproxy is

Microsoft Windows [Version 10.0.19045.2251]
(c) Microsoft Corporation. All rights reserved.

c:\Users\johndoe\Org>echo foo
foo

I expected just "foo".

Original bug report:
https://list.orgmode.org/orgmode/CACjq+cy8G+EkMgSwr5KWNbN7jjLE3tzqKwbQiTy84ZvJsumMwQ@mail.gmail.com/

AFAIU, the culprit is how cmdproxy handles piped input:

cmdproxy -c 'echo foo' works fine, outputting "foo".
However (in powershell) echo 'echo foo' | cmdproxy yields the above
verbose text - all in stdout.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Eli Zaretskii]

tags 71081 notabug
thanks

> From: Ihor Radchenko <yantar92@posteo.net>
> Date: Mon, 20 May 2024 18:34:52 +0000
> 
> Hello,
> 
> On Linux, if I put
> 
> echo foo
> 
> into scratch, select it, and run M-x shell-command-on-region sh,
> I get "foo" in echo area.
> 
> However, on Windows, the result of M-x shell-command-on-region cmdproxy is
> 
> Microsoft Windows [Version 10.0.19045.2251]
> (c) Microsoft Corporation. All rights reserved.
> 
> c:\Users\johndoe\Org>echo foo
> foo
> 
> I expected just "foo".
> 
> Original bug report:
> https://list.orgmode.org/orgmode/CACjq+cy8G+EkMgSwr5KWNbN7jjLE3tzqKwbQiTy84ZvJsumMwQ@mail.gmail.com/
> 
> AFAIU, the culprit is how cmdproxy handles piped input:
> 
> cmdproxy -c 'echo foo' works fine, outputting "foo".
> However (in powershell) echo 'echo foo' | cmdproxy yields the above
> verbose text - all in stdout.

It is not cmdproxy that does this, it's cmd.exe.  cmdproxy is not a
shell, it's a simply proxy of the shell that is there to fix some
obscure problems (such as some quoting issues).  To run shell
commands, it eventually invokes cmd.exe.  Try

   echo echo foo | cmd

and you will see the same behavior.

Conclusion: Windows is not a Posix system.  But you already knew
that...

As for the original bug report: AFAIR, when cmd.exe says "More?", it
needs a final newline.  Other than that, I don't really understand
what is the problem that bug report is trying to solve (I don't use
org-babel).  If it expects the same behavior from cmd.exe as a Posix
shell would produce, the expectation is simply incorrect and
unjustified.

I see no Emacs bug here.

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Dmitry Gutov]

On 20/05/2024 21:59, Eli Zaretskii wrote:
> If it expects the same behavior from cmd.exe as a Posix
> shell would produce, the expectation is simply incorrect and
> unjustified.

Perhaps we should run something different, or with different arguments, 
when on Windows?

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Eli Zaretskii]

> Date: Tue, 21 May 2024 23:18:39 +0300
> Cc: 71081@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 20/05/2024 21:59, Eli Zaretskii wrote:
> > If it expects the same behavior from cmd.exe as a Posix
> > shell would produce, the expectation is simply incorrect and
> > unjustified.
> 
> Perhaps we should run something different, or with different arguments, 
> when on Windows?

Maybe.  But as I said, I don't have a clear understanding of the
issue.  If I did, perhaps I could give some advice or even suggest a
code change if required.

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:
>> On 20/05/2024 21:59, Eli Zaretskii wrote:
>> > If it expects the same behavior from cmd.exe as a Posix
>> > shell would produce, the expectation is simply incorrect and
>> > unjustified.
>> 
>> Perhaps we should run something different, or with different arguments, 
>> when on Windows?
>
> Maybe.  But as I said, I don't have a clear understanding of the
> issue.  If I did, perhaps I could give some advice or even suggest a
> code change if required.

The goal is to evaluate a shell script - multi-line string.

For example, given

"cd foo
pwd"

we need to get the output/value separated from the stderr and stderr
output.

Org mode uses the approach borrowed from `shell-command-on-region' -
passing the script as input to shell process (we use
`process-file'). However, as demonstrated by my reproducer, this
approach generates more than just script output.

In theory, we might pass the script as -c '<shell-escaped script body>',
but that's not all the story. We also sometimes want to pass the script
in chunks (via comint interaction).

When using comint, piping commands to shell process input is the only
way to keep the shell running as we send it more commands. However, the
same problem persists - when sending the very first command, cmd.exe
generates its version data, which we cannot easily distinguish from the
rest of the output. We cannot easily see stderr either.

P.S. CCing Matt, ob-shell maintainer.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Dmitry Gutov <dmitry@gutov.dev>, 71081@debbugs.gnu.org, matt@excalamus.com
> Date: Wed, 22 May 2024 11:45:23 +0000
> 
> The goal is to evaluate a shell script - multi-line string.
> 
> For example, given
> 
> "cd foo
> pwd"
> 
> we need to get the output/value separated from the stderr and stderr
> output.

OK.

> Org mode uses the approach borrowed from `shell-command-on-region' -
> passing the script as input to shell process (we use
> `process-file'). However, as demonstrated by my reproducer, this
> approach generates more than just script output.

Right.

> In theory, we might pass the script as -c '<shell-escaped script body>',

Won't work on Windows, either, because of the newlines.

> When using comint, piping commands to shell process input is the only
> way to keep the shell running as we send it more commands. However, the
> same problem persists - when sending the very first command, cmd.exe
> generates its version data, which we cannot easily distinguish from the
> rest of the output. We cannot easily see stderr either.

The easiest way of solving this conundrum is to create a temporary
batch file, insert all the commands into it, with the first line being

  @echo off

and then submit the name of that batch file to the Windows shell as
the sole command-line argument after shell-command-switch.  So my
recommendation is to do just that, on Windows.

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> When using comint, piping commands to shell process input is the only
>> way to keep the shell running as we send it more commands. However, the
>> same problem persists - when sending the very first command, cmd.exe
>> generates its version data, which we cannot easily distinguish from the
>> rest of the output. We cannot easily see stderr either.
>
> The easiest way of solving this conundrum is to create a temporary
> batch file, insert all the commands into it, with the first line being
>
>   @echo off
>
> and then submit the name of that batch file to the Windows shell as
> the sole command-line argument after shell-command-switch.  So my
> recommendation is to do just that, on Windows.

May it be something you also want to do for `shell-command-on-region'?

At the end, the current Emacs behaviour is unexpected from the user
perspective, even though it is a result of the cmd.exe behaviour. If
there is a workable workaround, `shell-command-on-region' and Org mode
may as well use it; preferably implemented on Emacs side to not
duplicate the code.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, 71081@debbugs.gnu.org, matt@excalamus.com
> Date: Wed, 22 May 2024 14:26:47 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The easiest way of solving this conundrum is to create a temporary
> > batch file, insert all the commands into it, with the first line being
> >
> >   @echo off
> >
> > and then submit the name of that batch file to the Windows shell as
> > the sole command-line argument after shell-command-switch.  So my
> > recommendation is to do just that, on Windows.
> 
> May it be something you also want to do for `shell-command-on-region'?

No, because in general the region is some text we want to give a
program as input.  The usual "program" in these cases is not the
shell, but some program that gets the text and processes it in some
way.

So in most "usual" uses of shell-command-on-region, going through a
batch file is an unnecessary overhead.

> At the end, the current Emacs behaviour is unexpected from the user
> perspective

Only if the user is not aware of cmd.exe's different (some say
idiosyncratic) behavior wrt commands you pipe into its stdin, and wrt
command lines that include newlines.  If the user _is_ aware of that,
the behavior is very expected.

> even though it is a result of the cmd.exe behaviour. If there is a
> workable workaround, `shell-command-on-region' and Org mode may as
> well use it; preferably implemented on Emacs side to not duplicate
> the code.

I use Emacs for 30 years, and it's the first time I hear about such a
strange use of shell-command-on-region.  When I will have finished
wondering whatever gave Org developers this crazy idea, my next
question would be whether indeed this strange way of using
shell-command-on-region is a reason good enough for coding something
in core.  I tend to think no.  I think if Org needs that, Org should
solve it for itself.  It isn't very hard, is it?  When we hear about
enough clients for this strange feature, we might consider providing a
general solution.

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > The easiest way of solving this conundrum is to create a temporary
>> > batch file, insert all the commands into it, with the first line being
>> >
>> >   @echo off
>> >
>> > and then submit the name of that batch file to the Windows shell as
>> > the sole command-line argument after shell-command-switch.  So my
>> > recommendation is to do just that, on Windows.
>> 
>> May it be something you also want to do for `shell-command-on-region'?
>
> No, because in general the region is some text we want to give a
> program as input.  The usual "program" in these cases is not the
> shell, but some program that gets the text and processes it in some
> way.
>
> So in most "usual" uses of shell-command-on-region, going through a
> batch file is an unnecessary overhead.

Clear. I am not a big user of shell-command-on-region.

Then, what about something like

(shell-command-to-string "echo foo\necho bar")

On Linux, it yields
"foo
bar"

On Windows, it yields

"foo"

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, 71081@debbugs.gnu.org, matt@excalamus.com
> Date: Thu, 23 May 2024 12:06:25 +0000
> 
> Then, what about something like
> 
> (shell-command-to-string "echo foo\necho bar")
> 
> On Linux, it yields
> "foo
> bar"
> 
> On Windows, it yields
> 
> "foo"

Yes, because the Windows shell doesn't support command lines with
embedded newlines, so it ignores everything past the first newline.
(Actually, there are complicated tricks to do that, but I'll spare
you.)

Instead, you should do this on Windows:

  (shell-command-to-string "echo foo && echo bar")

This is the documented method of invoking cmd.exe with multiple
commands.

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> On Windows, it yields
>> 
>> "foo"
>
> Yes, because the Windows shell doesn't support command lines with
> embedded newlines, so it ignores everything past the first newline.
> (Actually, there are complicated tricks to do that, but I'll spare
> you.)

Should it be documented in the docstring of `shell-command-to-string'?
Currently, it only mentions `shell-quote-argument' and the current
behaviour is not expected - there is nothing indicating that one should
be careful about newlines (or any other characters passed to system
shell).

> Instead, you should do this on Windows:
>
>   (shell-command-to-string "echo foo && echo bar")
>
> This is the documented method of invoking cmd.exe with multiple
> commands.

Maybe. But `shell-command-on-region' does work with multi-line commands,
so it all boils down to implementation details, which are not really
described... Why would anyone assume that `shell-command-to-string' is
passed as an argument to cmd.exe and not piped as input? Or wise
versa. Or maybe by some other means - all the cases have subtle
differences. (not just on Windows; on Linux as well; more over, on
Linux, different shells have different behaviors wrt input/command
argument/script file).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, 71081@debbugs.gnu.org, matt@excalamus.com
> Date: Thu, 23 May 2024 14:00:44 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> On Windows, it yields
> >> 
> >> "foo"
> >
> > Yes, because the Windows shell doesn't support command lines with
> > embedded newlines, so it ignores everything past the first newline.
> > (Actually, there are complicated tricks to do that, but I'll spare
> > you.)
> 
> Should it be documented in the docstring of `shell-command-to-string'?

We could, but IMO such tangential information belongs to the manual,
not to the doc string.  We should also somehow avoid the slippery
slope of describing all the differences between the Posix shells and
the Windows shell.  E.g., newlines are also not allowed in Windows
file names, but we don't mention that in doc strings of every function
that deals with file names.

> Currently, it only mentions `shell-quote-argument' and the current
> behaviour is not expected - there is nothing indicating that one should
> be careful about newlines (or any other characters passed to system
> shell).

Characters special to a shell always need special attention.  And
which characters are special to a given shell depends on the shell.
Emacs supports special characters for a couple of widely-used shells,
but not for all of them.

> > Instead, you should do this on Windows:
> >
> >   (shell-command-to-string "echo foo && echo bar")
> >
> > This is the documented method of invoking cmd.exe with multiple
> > commands.
> 
> Maybe. But `shell-command-on-region' does work with multi-line commands,

No, it works with multi-line _regions_, since the region is sent to
the shell command as input to process.

> Why would anyone assume that `shell-command-to-string' is
> passed as an argument to cmd.exe and not piped as input?

Because the doc string says so:

  Execute shell command COMMAND and return its output as a string.

"Execute COMMAND" means run it as "SHELL -c COMMAND", in any
reasonable interpretation.  Anything else is not the usual way of
running commands.

> more over, on Linux, different shells have different behaviors wrt
> input/command argument/script file).

Same on Windows.  Try PowerShell some day.  Does it mean we need to
document all of that in our documentation?

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Should it be documented in the docstring of `shell-command-to-string'?
>
> We could, but IMO such tangential information belongs to the manual,
> not to the doc string.  We should also somehow avoid the slippery
> slope of describing all the differences between the Posix shells and
> the Windows shell.  E.g., newlines are also not allowed in Windows
> file names, but we don't mention that in doc strings of every function
> that deals with file names.

Understood.

>> Why would anyone assume that `shell-command-to-string' is
>> passed as an argument to cmd.exe and not piped as input?
>
> Because the doc string says so:
>
>   Execute shell command COMMAND and return its output as a string.
>
> "Execute COMMAND" means run it as "SHELL -c COMMAND", in any
> reasonable interpretation.  Anything else is not the usual way of
> running commands.

Another reasonable interpretation is: "do the same thing as if I type
COMMAND into shell prompt". But that's me. If you think that my
interpretation is uncommon, so be it.

>> more over, on Linux, different shells have different behaviors wrt
>> input/command argument/script file).
>
> Same on Windows.  Try PowerShell some day.  Does it mean we need to
> document all of that in our documentation?

It would be nice, yes. Without details - just point that there is a
difference. Maybe in the "41.3 Creating a Synchronous Process" section
of the manual.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, 71081@debbugs.gnu.org, matt@excalamus.com
> Date: Thu, 23 May 2024 14:36:22 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> more over, on Linux, different shells have different behaviors wrt
> >> input/command argument/script file).
> >
> > Same on Windows.  Try PowerShell some day.  Does it mean we need to
> > document all of that in our documentation?
> 
> It would be nice, yes. Without details - just point that there is a
> difference. Maybe in the "41.3 Creating a Synchronous Process" section
> of the manual.

Done, and closing the bug.

bug#71081: 30.0.50; shell-command-on-region outputs boilerplate text on Windows

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, 71081@debbugs.gnu.org, matt@excalamus.com
> Date: Thu, 23 May 2024 14:36:22 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> more over, on Linux, different shells have different behaviors wrt
> >> input/command argument/script file).
> >
> > Same on Windows.  Try PowerShell some day.  Does it mean we need to
> > document all of that in our documentation?
> 
> It would be nice, yes. Without details - just point that there is a
> difference. Maybe in the "41.3 Creating a Synchronous Process" section
> of the manual.

Done, and closing the bug.