bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[macross84]

We all know this is a stupid behaviour

Unix programs ported to windows are going to return exit statuses that go
in 8 bits, but emacs truncating the exit status to 8 bits by force is
making to lose 24 bits of the exit status of native windows programs.


--------------------------------------------------------------------------------------------
Busca, compara... pero busca en Qué!
Descúbrelo en http://buscar.que.es/
--------------------------------------------------------------------------------------------
Correo enviado desde http://www.ozu.es

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Noam Postavsky]

tag 6430 + wontfix
quit

Problem is 32bit Emacs has 28 bit integers, so there's no way to fit
the whole exit code anyway. At least truncating to 8 bits is portable.
Plus, isn't it against Emacs policy to provide features on non-Free
platforms before Free ones?

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Óscar Fuentes]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

> Problem is 32bit Emacs has 28 bit integers, so there's no way to fit
> the whole exit code anyway.

There is -with-wide-int for 32bit Emacs and it can be compiled as a
64bit executable.

> At least truncating to 8 bits is portable.

Imposing the limits of one platform on another is the contrary of being
portable.

> Plus, isn't it against Emacs policy to provide features on non-Free
> platforms before Free ones?

Nobody is asking for more than 8 bits of exit status on *nix. There is
no competition here about who has the wider exit status.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Mon, 13 Jun 2016 22:48:25 -0400
> Cc: macross84@ozu.es
> 
> Problem is 32bit Emacs has 28 bit integers, so there's no way to fit
> the whole exit code anyway. At least truncating to 8 bits is portable.
> Plus, isn't it against Emacs policy to provide features on non-Free
> platforms before Free ones?

All the above is true, but I don't understand the original complaint.
Emacs doesn't truncate exit status of subprocesses in any way, it
retrieves the full value of the status, and then reformats it
according to what the Posix-style WIF* macros (which Emacs uses
elsewhere) expect.  No information is lost during this reformatting,
see the implementation of waitpid in w32proc.c.  Currently, we only
report SIGINT in the WSTOPSIG macro, but that's only because no one
ever needed anything else; if needed, we could also report SIGSEGV
there.

IOW, I see no problem here to begin with.

Thanks.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Noam Postavsky]

On Tue, Jun 14, 2016 at 10:35 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> All the above is true, but I don't understand the original complaint.

My understanding of the original complaint is that if you compile the
program below:

#include <stdlib.h>

int main(int argc, char *argv[])
{
    return atoi(argv[1]);
}

/*
(call-process (expand-file-name "exit-arg1.exe") nil nil nil "300") ;=> 44
 */

To lisp programs the exit status gets truncated to 8 bits, whereas
running from cmd.exe:

C:\Users\npostavs\src\emacs\bug-6430-w32-exit-rc>exit-arg1.exe 300

C:\Users\npostavs\src\emacs\bug-6430-w32-exit-rc>echo %errorlevel%
300

> Emacs doesn't truncate exit status of subprocesses in any way, it
> retrieves the full value of the status, and then reformats it
> according to what the Posix-style WIF* macros (which Emacs uses
> elsewhere) expect.  No information is lost during this reformatting,
> see the implementation of waitpid in w32proc.c.

Not really sure what this reformatting is about, but I think the point
is that the original value does not return to lisp.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Tue, 14 Jun 2016 17:48:22 -0400
> Cc: 6430@debbugs.gnu.org, macross84@ozu.es
> 
> > Emacs doesn't truncate exit status of subprocesses in any way, it
> > retrieves the full value of the status, and then reformats it
> > according to what the Posix-style WIF* macros (which Emacs uses
> > elsewhere) expect.  No information is lost during this reformatting,
> > see the implementation of waitpid in w32proc.c.
> 
> Not really sure what this reformatting is about, but I think the point
> is that the original value does not return to lisp.

Why is that a problem?

The important information that I thought this was about is in the
upper 4 bits of the status, and it doesn't get lost -- it's passed
back to Emacs as the signal (if any) that caused the subprocess to
exit.

If there are any important use cases with programs that return status
above 255, we can easily change the definition of WEXITSTATUS for
Windows.  But I have yet to see a real-life example of such a program,
or any complaint about the current WEXITSTATUS definition in Emacs.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Not really sure what this reformatting is about, but I think the point
>> is that the original value does not return to lisp.
>
> Why is that a problem?

Because Emacs is reporting something false?

> The important information that I thought this was about is in the
> upper 4 bits of the status, and it doesn't get lost -- it's passed
> back to Emacs as the signal (if any) that caused the subprocess to
> exit.
>
> If there are any important use cases with programs that return status
> above 255, we can easily change the definition of WEXITSTATUS for
> Windows.  But I have yet to see a real-life example of such a program,
> or any complaint about the current WEXITSTATUS definition in Emacs.

You are fortunate enough to live in a world of applications ported from
*nix :-) (and so do I, for the most part.)

When a Windows application calls an API that fails and the application
has no method for recovering from the error, it is customary to exit
with the error code of the API (usually obtained with GetLastError).

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Cc: Noam Postavsky <npostavs@users.sourceforge.net>,  macross84@ozu.es,  6430@debbugs.gnu.org
> Date: Wed, 15 Jun 2016 14:30:20 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Not really sure what this reformatting is about, but I think the point
> >> is that the original value does not return to lisp.
> >
> > Why is that a problem?
> 
> Because Emacs is reporting something false?

Only if the value is greater than 255.

> > The important information that I thought this was about is in the
> > upper 4 bits of the status, and it doesn't get lost -- it's passed
> > back to Emacs as the signal (if any) that caused the subprocess to
> > exit.
> >
> > If there are any important use cases with programs that return status
> > above 255, we can easily change the definition of WEXITSTATUS for
> > Windows.  But I have yet to see a real-life example of such a program,
> > or any complaint about the current WEXITSTATUS definition in Emacs.
> 
> You are fortunate enough to live in a world of applications ported from
> *nix :-) (and so do I, for the most part.)

Emacs does more than that.  To see that, interrupt a console program
with Ctrl-C, and then display its exit status using %ERRORLEVEL%.
Then do the same inside Emacs and compare the results.  That's what I
thought the OP was alluding to.

> When a Windows application calls an API that fails and the application
> has no method for recovering from the error, it is customary to exit
> with the error code of the API (usually obtained with GetLastError).

GetLastError has nothing to do with this, because it's not an API that
fails, it's a program that exits with some arbitrary exit code.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> When a Windows application calls an API that fails and the application
>> has no method for recovering from the error, it is customary to exit
>> with the error code of the API (usually obtained with GetLastError).
>
> GetLastError has nothing to do with this, because it's not an API that
> fails, it's a program that exits with some arbitrary exit code.

if( ! CreateProcess(....) ) {
  DWORD ec = GetLastError();
  // do some cleanup
  ExitProcess(ec);
}
  
The program exits with the error code obtained with GetLastError, which
is a 32 bit number.

The program may have its own exit codes, which can be wider than 8 bits.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Cc: npostavs@users.sourceforge.net,  macross84@ozu.es,  6430@debbugs.gnu.org
> Date: Wed, 15 Jun 2016 17:15:54 +0200
> 
> > GetLastError has nothing to do with this, because it's not an API that
> > fails, it's a program that exits with some arbitrary exit code.
> 
> if( ! CreateProcess(....) ) {
>   DWORD ec = GetLastError();
>   // do some cleanup
>   ExitProcess(ec);
> }

That's not how Emacs does that, see waitpid and its callers.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> > GetLastError has nothing to do with this, because it's not an API that
>> > fails, it's a program that exits with some arbitrary exit code.
>> 
>> if( ! CreateProcess(....) ) {
>>   DWORD ec = GetLastError();
>>   // do some cleanup
>>   ExitProcess(ec);
>> }
>
> That's not how Emacs does that, see waitpid and its callers.

Once again, we are miscommunicating. AFAIU this is about Emacs calling a
process foo which returns an exit code larger than 255, but Emacs only
reports the lower 8 bits to the Elisp function that started the process.

In w32proc.c waitpid:

  if (!GetExitCodeProcess (wait_hnd[active], &retval))
  ...
  retval <<= 8;

Windows applications make use of the full 32 bits of the return code.
With Emacs it is impossible to know those larger-than-255 exit codes.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Cc: npostavs@users.sourceforge.net,  macross84@ozu.es,  6430@debbugs.gnu.org
> Date: Wed, 15 Jun 2016 19:14:25 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > GetLastError has nothing to do with this, because it's not an API that
> >> > fails, it's a program that exits with some arbitrary exit code.
> >> 
> >> if( ! CreateProcess(....) ) {
> >>   DWORD ec = GetLastError();
> >>   // do some cleanup
> >>   ExitProcess(ec);
> >> }
> >
> > That's not how Emacs does that, see waitpid and its callers.
> 
> Once again, we are miscommunicating. AFAIU this is about Emacs calling a
> process foo which returns an exit code larger than 255, but Emacs only
> reports the lower 8 bits to the Elisp function that started the process.
> 
> In w32proc.c waitpid:
> 
>   if (!GetExitCodeProcess (wait_hnd[active], &retval))
>   ...
>   retval <<= 8;
> 
> Windows applications make use of the full 32 bits of the return code.
> With Emacs it is impossible to know those larger-than-255 exit codes.

But not because of the code you cite: the 8-bit left shift just means
we can report 24 bits out of 32.  And all of that is completely
unrelated to GetLastError, because errors in starting the subprocess
are not reported via waitpid.

Once again, if this is important (I don't think so), we can redefine
WEXITSTATUS etc. macros to keep all the 32 bits of the exit status
(actually, the top 2 bits are reserved, so only 30 bits can be used).
Feel free to submit patches to that effect if you like.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Cc: npostavs@users.sourceforge.net,  macross84@ozu.es,  6430@debbugs.gnu.org
>> Date: Wed, 15 Jun 2016 19:14:25 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> > GetLastError has nothing to do with this, because it's not an API that
>> >> > fails, it's a program that exits with some arbitrary exit code.
>> >> 
>> >> if( ! CreateProcess(....) ) {
>> >>   DWORD ec = GetLastError();
>> >>   // do some cleanup
>> >>   ExitProcess(ec);
>> >> }
>> >
>> > That's not how Emacs does that, see waitpid and its callers.
>> 
>> Once again, we are miscommunicating. AFAIU this is about Emacs calling a
>> process foo which returns an exit code larger than 255, but Emacs only
>> reports the lower 8 bits to the Elisp function that started the process.
>> 
>> In w32proc.c waitpid:
>> 
>>   if (!GetExitCodeProcess (wait_hnd[active], &retval))
>>   ...
>>   retval <<= 8;
>> 
>> Windows applications make use of the full 32 bits of the return code.
>> With Emacs it is impossible to know those larger-than-255 exit codes.
>
> But not because of the code you cite: the 8-bit left shift just means
> we can report 24 bits out of 32.  And all of that is completely
> unrelated to GetLastError, because errors in starting the subprocess
> are not reported via waitpid.

GetLastError on my example was on the *called* process, not on Emacs.

> Once again, if this is important (I don't think so), we can redefine
> WEXITSTATUS etc. macros to keep all the 32 bits of the exit status
> (actually, the top 2 bits are reserved, so only 30 bits can be used).

When using --with-wide-int (or when running on Windows 64bits) we have
the full 32 bits, right?

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Cc: npostavs@users.sourceforge.net,  macross84@ozu.es,  6430@debbugs.gnu.org
> Date: Wed, 15 Jun 2016 19:36:14 +0200
> 
> > Once again, if this is important (I don't think so), we can redefine
> > WEXITSTATUS etc. macros to keep all the 32 bits of the exit status
> > (actually, the top 2 bits are reserved, so only 30 bits can be used).
> 
> When using --with-wide-int (or when running on Windows 64bits) we have
> the full 32 bits, right?

No.  The 2 bits are reserved by Windows, not by Emacs integer's
representation.  To see that in action, interrupt a program with
Ctrl-C and look at the exit status: you will see 0xC000013A; the
0xC0000000 bits are the reserved ones.  (There are more values for
other fatal exit reasons, but all of them set those 2 bits.)

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> When using --with-wide-int (or when running on Windows 64bits) we have
>> the full 32 bits, right?
>
> No.  The 2 bits are reserved by Windows, not by Emacs integer's
> representation.  To see that in action, interrupt a program with
> Ctrl-C and look at the exit status: you will see 0xC000013A; the
> 0xC0000000 bits are the reserved ones.  (There are more values for
> other fatal exit reasons, but all of them set those 2 bits.)

I thought that the limitation you referred to was about how many bits
Emacs has for representing integers on 32 bit platforms. The upper 2
bits reserved by Windows is an integral part of the exit code. A process
can return any value on the 32 bit range.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Cc: npostavs@users.sourceforge.net,  macross84@ozu.es,  6430@debbugs.gnu.org
> Date: Wed, 15 Jun 2016 20:22:25 +0200
> 
> A process can return any value on the 32 bit range.

It can, but using those 2 bits for that would be silly, because doing
so will cause the exit code be interpreted as exit due to a fatal
error.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> A process can return any value on the 32 bit range.
>
> It can, but using those 2 bits for that would be silly, because doing
> so will cause the exit code be interpreted as exit due to a fatal
> error.

AFAIU Emacs currently picks apart the higher bits that (by convention)
signals the fatal/special condition, keeps the lower 8 bits as the
reported exit code and throws away the rest. The point is that the Elisp
code that invoked the external program is not receiving the exit code of
the program, but some part of it. If the external program uses exit
codes above 255 (apart from the "fatal" ones that Emacs detects), the
exit code reported to Elisp is useless, at best.

Personally I don't care, because I live on a world of cross-platform
software, but it was surprising to learn about this limitation of
process handling on Windows, considering how much work was invested on
making Emacs a true Windows citizen.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Cc: npostavs@users.sourceforge.net, 6430@debbugs.gnu.org
> Date: Wed, 15 Jun 2016 21:32:04 +0200
> 
> Personally I don't care, because I live on a world of cross-platform
> software, but it was surprising to learn about this limitation of
> process handling on Windows, considering how much work was invested on
> making Emacs a true Windows citizen.

We only solve real-life problems in Emacs, not academical ones.  Life
is too hard for anything else.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> We only solve real-life problems in Emacs, not academical ones.  Life
> is too hard for anything else.

The MS toolchain (to name just a case that we could expect to be used
from Emacs) outputs quite a few exit codes larger than 255. If the user
is interested on detecting one of those codes, he will be out of luck.

The fact that there is just one bug report about this limitation of
Emacs possibly shows that users are only interested on differentiating
success from failure, zero from non-zero exit codes.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Cc: npostavs@users.sourceforge.net,  6430@debbugs.gnu.org
> Date: Thu, 16 Jun 2016 06:13:46 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We only solve real-life problems in Emacs, not academical ones.  Life
> > is too hard for anything else.
> 
> The MS toolchain (to name just a case that we could expect to be used
> from Emacs) outputs quite a few exit codes larger than 255.

Any pointers to where these values are documented?

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> The MS toolchain (to name just a case that we could expect to be used
>> from Emacs) outputs quite a few exit codes larger than 255.
>
> Any pointers to where these values are documented?

cl.exe, for instance, uses the codes in winerror.h for reporting
failures:

https://msdn.microsoft.com/en-us/library/ebh0y918.aspx

AFAIR it is a common practice among MS applications, and widely used by
Windows-only software vendors.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Cc: npostavs@users.sourceforge.net,  6430@debbugs.gnu.org
> Date: Thu, 16 Jun 2016 19:56:48 +0200
> 
> cl.exe, for instance, uses the codes in winerror.h for reporting
> failures:
> 
> https://msdn.microsoft.com/en-us/library/ebh0y918.aspx

Those are just the normal error codes in disguise, and the ones
relevant to cl.exe are all below 255 (when the top 2 bits are
stripped).  So I see no problem here with the 8-bit limitation of the
status we report.

bug#6430: Emacs WINDOWS truncates exit status of processes to 8 bits

[Stefan Kangas]

tags 6430 + wontfix
close 6430
thanks

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Cc: npostavs@users.sourceforge.net,  6430@debbugs.gnu.org
>> Date: Thu, 16 Jun 2016 19:56:48 +0200
>>
>> cl.exe, for instance, uses the codes in winerror.h for reporting
>> failures:
>>
>> https://msdn.microsoft.com/en-us/library/ebh0y918.aspx
>
> Those are just the normal error codes in disguise, and the ones
> relevant to cl.exe are all below 255 (when the top 2 bits are
> stripped).  So I see no problem here with the 8-bit limitation of the
> status we report.

There was a discussion here where both Noam and Eli seemed to think
there is no bug.  And I only see one differing opinion from that; in any
case, it seems unlikely we will do anything about this.

I'm therefore closing this bug report.

If anyone really feels strongly about this, I suppose they should
reopen the bug or propose a patch for discussion.