Problems with call-process (= identifying run-python issues)

Problems with call-process (= identifying run-python issues)

[Juan José García-Ripoll]

Hi,

I am running into a weird issue in Windows 10, both with Emacs 27 and
Emacs 26.3. It all boils down to whether Emacs is invoked from the
command line or not. In one case, subprocesses get their input properly
redirected, in other cases not. All tests are done with plain vanilla
Emacs, using core libraries (no programming mode).

My test files are below. Essentially, the file test.el is trying to
imitate what python.el uses to call a python process, but instead of
that, it uses a simple copycat script that copies the input stream to
the output stream.

- If I launch the program from the command line, as in
      C:\Users\juanj\scoop\apps\emacs-27.1\bin\runemacs.exe -Q --load c:\Users\juanj\Downloads\test.el
  everything goes fine. In the *Messages* buffer I see this output
"Output:
\"import sys
ps = [getattr(sys, 'ps%s' % i, '') for i in range(1,4)]
ps_json = '\\\\\\\\n[\\\\\\\"%s\\\\\\\", \\\\\\\"%s\\\\\\\", \\\\\\\"%s\\\\\\\"]\\\\\\\\n' % tuple(ps)
print (ps_json)
sys.exit(0)
\""

- If I create a shortcut for runemacs, adding "-Q --load test.el" as
  arguments, then process-file outputs nothing. The *Messages* buffer
  shows  
"Output:
\"\""

- If I type #+R (Windows key + R) and enter directly the command line
"C:\Users\juanj\scoop\apps\emacs-27.1\bin\runemacs -Q --load
c:\Users\juanj\Downloads\test.el" the script also fails and produces the
wrong output.

There are no error messages. The script gets invoked (you may add a line
"echo foo" to chain.cmd and see that it is output), but it does not
get the redirected input.

---foo---(test file 1, saved as c:\Users\juanj\Downloads\foo)
import sys
ps = [getattr(sys, 'ps%s' % i, '') for i in range(1,4)]
ps_json = '\\n[\"%s\", \"%s\", \"%s\"]\\n' % tuple(ps)
print (ps_json)
sys.exit(0)

--test.el---(test file 2, saved as c:\Users\juanj\Downloads\test.el)
(with-current-buffer "*Messages*"
  (with-temp-buffer
    (call-process "c:/Users/juanj/Downloads/chain.cmd" ;conda_python.cmd"
                  "c:/Users/juanj/Downloads/foo"
                  '(t nil) nil "-i")
    (print (format "Output:\n%S" (buffer-string)))
    ))

---chain.cmd---(test file 2, saved as c:\Users\juanj\Downloads\chain.cmd)
@echo off
rem copies lines from the input until it is empty
:begin
set /p line= || goto :end
echo %line%
goto :begin
:end


-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Problems with call-process (= identifying run-python issues)

[Eli Zaretskii]

> From: Juan José García-Ripoll <juanjose.garciaripoll@gmail.com>
> Date: Tue, 18 Aug 2020 15:36:23 +0200
> 
> I am running into a weird issue in Windows 10, both with Emacs 27 and
> Emacs 26.3. It all boils down to whether Emacs is invoked from the
> command line or not. In one case, subprocesses get their input properly
> redirected, in other cases not. All tests are done with plain vanilla
> Emacs, using core libraries (no programming mode).
> 
> My test files are below. Essentially, the file test.el is trying to
> imitate what python.el uses to call a python process, but instead of
> that, it uses a simple copycat script that copies the input stream to
> the output stream.

Could you please present a complete recipe for reproducing the
problems, starting from "runemacs -Q"?  I got confused by the test
files.  Can this be done without involving Python, for example?

Thanks.

Re: Problems with call-process (= identifying run-python issues)

[Eli Zaretskii]

[Please use Reply All to keep the list on the CC.]

> From: Juan Jose Garcia-Ripoll <juanjose.garciaripoll@gmail.com>
> Date: Tue, 18 Aug 2020 17:54:35 +0200
> 
> The recipe was in my previous email. I never mention Python other than in the subject (it is not used by the
> tests). It only uses pure Emacs, emacs-lisp (one script) and one command line script (chain.cmd). The text
> in "foo" is arbitrary. It is a Python script, but it can be anything.

Is it all needed?  If so, can you explain its role here?

> Option A:
> 1. Save files with the specified names and locations
> 2. Open a command line processor (cmd.exe or PowerShell)
> 3. Invoke "runemacs -Q --load c:\Users\juanj\Downloads\test.el"
> 
> Option B:
> 1. Save files with the specified names
> 2. Press Windows Key + R (to enter commands)
> 2. Enter "runemacs -Q --load c:\Users\juanj\Downloads\test.el"

Then I'm afraid I don't understand what the script chain.cmd does and
what it expects from Emacs or the user.  Can you please take me though
this recipe one step at a time?

Thanks.

Re: Problems with call-process (= identifying run-python issues)

[Juan José García-Ripoll]

Eli Zaretskii <eliz@gnu.org> writes:
> Is it all needed?  If so, can you explain its role here?
>> Option A:
>> 1. Save files with the specified names and locations
>> 2. Open a command line processor (cmd.exe or PowerShell)
>> 3. Invoke "runemacs -Q --load c:\Users\juanj\Downloads\test.el"
>> 
>> Option B:
>> 1. Save files with the specified names
>> 2. Press Windows Key + R (to enter commands)
>> 2. Enter "runemacs -Q --load c:\Users\juanj\Downloads\test.el"
>
> Then I'm afraid I don't understand what the script chain.cmd does and
> what it expects from Emacs or the user.  Can you please take me though
> this recipe one step at a time?

Ok, let me please explain the problem: I want to run a process that gets
its input from a file, while Emacs collects its output. This is standard
working in many programming modes. For instance, it is the way
run-python calls python to find out the prompt.

I am finding that when I launch Emacs from the windowing environment
(e.g. shortcuts, Start menu, Windows key + R, explorer), Emacs does not
redirect the input of the subprocess properly. If I launch the Emacs
from the command line, the subprocess works as expected. 

To reproduce this I have created a minimal example:

- A bogus input file (foo), that is going to be the redirected input of our
program

- A script, chain.cmd that copies all input to its output, line by
line. This is going to be the subprocess. Unfortunately, Windows does
not have a 'cat' command, which is why I did it this way.

- An elisp file (test.el) that launches the process with the redirected
input and collects the output

This example can be ran in three different ways:

- Invoking runemacs.exe -Q --load test.el from the command line
- Running the same command with Windows key + R
- Creating a shortcut that does the same.

All ingredients are needed, because we need a reproducible program, an
input file and some sophisticated commands that use Emacs'
call-process. However, it is *not* complicated at all and everything is
pretty standard.

One consequence of this problem is that run-python does not work as
expected (Emacs fails to find the prompt), but as I try to show, this is
not due to Python.

Note that this problem is related to the ispell problems I mentioned
before, but it is not the same. I already knew about the issues I was
experiencing with call-process when using python or other programs, and
those problems made me assume the aspell.exe was experiencing them as
well. *That* analysis was incorrect, but this case is way simpler and it
is reproducible using only Windows tools.

Cheers,

Juanjo

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Problems with call-process (= identifying run-python issues)

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Wed, 19 Aug 2020 10:59:39 +0200
> 
> Ok, let me please explain the problem: I want to run a process that gets
> its input from a file, while Emacs collects its output. This is standard
> working in many programming modes. For instance, it is the way
> run-python calls python to find out the prompt.
> 
> I am finding that when I launch Emacs from the windowing environment
> (e.g. shortcuts, Start menu, Windows key + R, explorer), Emacs does not
> redirect the input of the subprocess properly. If I launch the Emacs
> from the command line, the subprocess works as expected. 
> 
> To reproduce this I have created a minimal example:
> 
> - A bogus input file (foo), that is going to be the redirected input of our
> program
> 
> - A script, chain.cmd that copies all input to its output, line by
> line. This is going to be the subprocess. Unfortunately, Windows does
> not have a 'cat' command, which is why I did it this way.
> 
> - An elisp file (test.el) that launches the process with the redirected
> input and collects the output
> 
> This example can be ran in three different ways:
> 
> - Invoking runemacs.exe -Q --load test.el from the command line
> - Running the same command with Windows key + R
> - Creating a shortcut that does the same.
> 
> All ingredients are needed, because we need a reproducible program, an
> input file and some sophisticated commands that use Emacs'
> call-process. However, it is *not* complicated at all and everything is
> pretty standard.

OK, thanks.

I have now tried this on 2 different Windows systems, one of them
Windows 10, and I cannot reproduce what you describe.   No matter how
I invoke Emacs, I get the expected output in the buffer: the contents
of the file passed as standard input to the subprocess.

So I think either this is due to some local configuration issue on
your system, or there's some part of the reproduction recipe that you
omitted.

> One consequence of this problem is that run-python does not work as
> expected (Emacs fails to find the prompt), but as I try to show, this is
> not due to Python.

Working with Python on Windows has its quirks, because you need to
make sure Python uses unbuffered writes to its standard output.  That
is why I wanted to be sure Python is not involved here.

Re: Problems with call-process (= identifying run-python issues)

[Juan José García-Ripoll]

Eli Zaretskii <eliz@gnu.org> writes:
> I have now tried this on 2 different Windows systems, one of them
> Windows 10, and I cannot reproduce what you describe [...]
> So I think either this is due to some local configuration issue on
> your system, or there's some part of the reproduction recipe that you
> omitted.

Those are the only the steps needed. I am using runemacs.exe -Q in the
tests. I do not recall any "weird" configuration and I reproduce this
problem with Emacs 26.3 and 27, both on my laptop. I am a bit at a loss
at how to debug this further. Maybe I will have to attach a gdb while
Emacs is running if that is even possible on Windows.

> Working with Python on Windows has its quirks, because you need to
> make sure Python uses unbuffered writes to its standard output.  That
> is why I wanted to be sure Python is not involved here.

Precisely because it has its quirks, I had to create a batch file as
wrapper for Python (invoking Anaconda environment, setting right input
mode, etc). That is how I realized what was going on -- apparently only
on my system :-/

-- 
Juan José García Ripoll

Quantum Information and Foundations Group
Institute of Fundamental Physics IFF-CSIC
Calle Serrano 113b, Madrid 28006 Spain
http://quinfog.hbar.es - http://juanjose.garcia.ripoll

Re: Problems with call-process (= identifying run-python issues)

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Cc: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>,
>   emacs-devel@gnu.org
> Date: Wed, 19 Aug 2020 18:23:05 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > I have now tried this on 2 different Windows systems, one of them
> > Windows 10, and I cannot reproduce what you describe [...]
> > So I think either this is due to some local configuration issue on
> > your system, or there's some part of the reproduction recipe that you
> > omitted.
> 
> Those are the only the steps needed. I am using runemacs.exe -Q in the
> tests. I do not recall any "weird" configuration and I reproduce this
> problem with Emacs 26.3 and 27, both on my laptop.

It doesn't have to be "weird", it can be as simple as some antivirus
software or some other such stuff, they could be blocking or tweaking
cmd.exe invocations in some situations.  (Yes, in some quarters
cmd.exe invocations are considered potential security risk.)

> I am a bit at a loss at how to debug this further. Maybe I will have
> to attach a gdb while Emacs is running if that is even possible on
> Windows.

It's possible to attach GDB to Emacs on Windows, yes.

I actually looked at the source code involved in redirecting the
standard handles of subprocesses on Windows, and saw that if
redirection fails, Emacs should signal an error.  So something morfe
subtle happens in your case.  Maybe the batch file doesn't even run?

Re: Problems with call-process (= identifying run-python issues)

[Juan José García-Ripoll]

Eli Zaretskii <eliz@gnu.org> writes:
> It doesn't have to be "weird", it can be as simple as some antivirus
> software or some other such stuff, they could be blocking or tweaking
> cmd.exe invocations in some situations.  (Yes, in some quarters
> cmd.exe invocations are considered potential security risk.)

The script gets invoked. If I add an "echo I have been executed" line,
it shows in the output. That's the difference with the aspell.exe
problem -- here the executable from msys2/mingw64 was broken.

> It's possible to attach GDB to Emacs on Windows, yes.

It may not be that complicated. I just experienced the same problem with
a freshly new compiled Emacs, ran from a mingw prompt.

> I actually looked at the source code involved in redirecting the
> standard handles of subprocesses on Windows, and saw that if
> redirection fails, Emacs should signal an error.  So something morfe
> subtle happens in your case.  Maybe the batch file doesn't even run?

As I said above, the script does get executed. I hope it's not something
more complicated in the exec() code. Let's see if I can find anything --
hopefully it's not something as complicated as the Nextcloud
problems... And apologies for the noise this brings to the list.

Cheers,

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es