bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 5900 bytes --]

Hi all,

On Windows, sending more that 4096 bytes at once to a subprocess will cause Emacs to hang (freeze) indefinitely. Killing the subprocess by external means restores functionality. This happens in Emacs 25, 24.5.1, and 24.4 on Windows 8 and 10; I did not try in earlier versions.

Here is how to reproduce the issue:

1. Create a loop.cmd file in C:\. Put the following in it:

:loop
goto loop

2. Run the following snippet in emacs -Q:

(process-send-string
 (start-process "bug" nil "c:/loop.cmd")
 (make-string 4097 ?a))

3. Emacs hangs. Kill cmd.exe from the task manager to make it responsive again. At this point Emacs will display the following:

   Debugger entered--Lisp error: (file-error "Writing to process" "Invalid argument" #<process bug>)
     process-send-string(#<process bug> "(...)")
     eval-region(192 277 t #[257 "\300\242b\210\301\207" [(277) (process-send-string (start-process "bug" nil "c:/loop.cmd") (make-string 4097 97))] 2 "\n\n(fn IGNORE)"])  ; Reading at buffer position 277
     elisp--eval-defun()
     eval-defun(nil)
     funcall-interactively(eval-defun nil)
     call-interactively(eval-defun nil nil)
     command-execute(eval-defun)

Note that the following snippet, on the other hand, runs without issues:

(process-send-string
 (start-process "bug" nil "c:/loop.cmd")
 (make-string 4096 ?a))

I have read sections 7.1 and 7.2 of https://www.gnu.org/software/emacs/manual/html_mono/efaq-w32.html , but I do not believe that the bug is covered by these sections. Indeed, sending the same input in smaller batches works fine. That is, sending 4097 bytes in batches of 50 bytes works fine: Emacs does not hang, and the subprocess properly receives the input.

Here is some context: we've had many users of Flycheck (an alternative to Flymake) on Windows complaining that Flycheck caused Emacs to hang; this happened since we transitioned from passing file names to linters to feeding them source code directly on their stdin. It turns out that this happens when the files in question are more than 4096 characters long. The problem was confirmed with jshint, eshint, and phpcs. I do not know if that means that other linters work fine, or if we didn't get reports from other users. Investigating the bug led me to the minified test case above.

As I mentionned before, sending input in smaller batches works fine. In the case of Flycheck, this means changing this:

  (process-send-region (point-min) (point-max))

to this:

  (goto-char (point-mint))
  (while (not (eobp))
    (let ((from (point)))
      (forward-char (min 50 (- (point-max) (point))))
      (process-send-region process from (point))))

fixes the issue entirely.

I do not use Windows often, but I'll be happy to provide more information, if I can help. I did not compile my own Emacs; I used the one provided at https://sourceforge.net/projects/emacsbinw64/ . The original bug report about Flycheck, which did not identify process-send-string or process-send-region as the culprit, is here: https://github.com/flycheck/flycheck/issues/794 . I have included debug output below.

Thanks,
Clément.

In GNU Emacs 25.0.50.1 (x86_64-w64-mingw32)
 of 2015-11-10
Repository revision: 9145e79dc2042fb477959ddda59c3e2ff5fa3914
Windowing system distributor 'Microsoft Corp.', version 6.3.9600
Configured using:
 'configure --prefix=/z/emacs --host=x86_64-w64-mingw32
 --target=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --with-wide-int
 --with-jpeg --with-xpm --with-png --with-tiff --with-rsvg --with-xml2
 --with-gnutls --with-sound=yes --with-file-notification=yes
 --without-dbus --without-imagemagick 'CFLAGS=-O3 -fomit-frame-pointer
 -g0 -pipe' 'LDFLAGS=-static-libgcc -static-libstdc++ -static -s
 -Wl,-s''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message dired format-spec
rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util help-fns mail-prsvr mail-utils help-mode easymenu cl-loaddefs
pcase cl-lib debug time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp disp-table
w32-win w32-vars term/common-win tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote w32notify w32 multi-tty
make-network-process emacs)

Memory information:
((conses 16 84225 4487)
 (symbols 56 19038 0)
 (miscs 48 64 168)
 (strings 32 14204 4737)
 (string-bytes 1 410641)
 (vectors 16 10727)
 (vector-slots 8 405995 4577)
 (floats 8 130 34)
 (intervals 56 274 32)
 (buffers 976 12))


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Eli Zaretskii]

> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Mon, 11 Jan 2016 00:42:01 -0500
> 
> On Windows, sending more that 4096 bytes at once to a subprocess will cause 
> Emacs to hang (freeze) indefinitely.

That's not true, at least not when phrased in such a general form.
See below.

> 1. Create a loop.cmd file in C:\. Put the following in it:
> 
> :loop
> goto loop
> 
> 2. Run the following snippet in emacs -Q:
> 
> (process-send-string
>  (start-process "bug" nil "c:/loop.cmd")
>  (make-string 4097 ?a))
> 
> 3. Emacs hangs.

That's expected, when your subprocess doesn't read its stdin.  Emacs
on MS-Windows uses pipes to communicate with subprocesses; a pipe on
Windows has a 4K buffer.  So as soon as Emacs writes more than 4096
bytes to the pipe, the pipe becomes full, and Emacs then waits for the
subprocess to read its end of the pipe, at which time Emacs will write
the rest of the stuff.  Since your batch file doesn't read its stdin,
Emacs waits forever.

If I modify your loop.cmd to have the following trivial command:

  @cat > foobar

then Emacs doesn't hang, it happily writes those 4097 bytes, and after
that, I see all of them in the 'foobar' file.  Moreover, I tried the
same test with 100 times as many bytes (i.e. 409700), and it still
worked as expected, with all of the bytes ending up in the file.

I also tried a more elaborate batch file that reads stdin in small
chunks, sleeping 1 sec between chunks.  That, too, didn't cause Emacs
to hang, although it took some time to run to completion, due to the
1-sec waits.

So the issue is more complicated than what you describe above, since I
believe the real-life use cases involved programs that do read their
stdin.

> I do not use Windows often, but I'll be happy to provide more information, if I 
> can help. I did not compile my own Emacs; I used the one provided at 
> https://sourceforge.net/projects/emacsbinw64/ . The original bug report about 
> Flycheck, which did not identify process-send-string or process-send-region as 
> the culprit, is here: https://github.com/flycheck/flycheck/issues/794 . I have 
> included debug output below.

There could be some flaw in the logic that handles the overflowing
pipe, which your use cases somehow trigger.  But it's not a silly bug
like what your description suggests: large writes to subprocesses do
work, this was tested extensively back in Sep 2014 when bug#18420 was
fixed by making the pipes to subprocesses non-blocking.

Or maybe some other factor is at work in the real-life use cases.

So someone with access to the programs in question will have to step
with a debugger through the code involved in this (I can point to the
relevant parts of the Emacs C code), and find out what goes wrong
there, and why.  Alternatively, if a reproducible test case can be
posted with a simple program which reads its stdin and causes the
hang, I will try debugging it myself.  (By "simple program" I meant
something that won't involve installing lots of other packages,
ideally something as simple as a batch file calling 'cat' or a similar
program.)

Thanks.

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

On 01/11/2016 10:37 AM, Eli Zaretskii wrote:
> So someone with access to the programs in question will have to step
> with a debugger through the code involved in this (I can point to the
> relevant parts of the Emacs C code), and find out what goes wrong
> there, and why.  Alternatively, if a reproducible test case can be
> posted with a simple program which reads its stdin and causes the
> hang, I will try debugging it myself.  (By "simple program" I meant
> something that won't involve installing lots of other packages,
> ideally something as simple as a batch file calling 'cat' or a similar
> program.)

Hi Eli,

Thanks for looking into this. Would a recipe requiring node.js and jshint be too much? Node.js has a windows installer and jshint can be installed with npm install --global jshint.

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Eli Zaretskii]

> Cc: 22344@debbugs.gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Mon, 11 Jan 2016 11:11:42 -0500
> 
> Thanks for looking into this. Would a recipe requiring node.js and jshint be too much? Node.js has a windows installer and jshint can be installed with npm install --global jshint.

I'd prefer a program that doesn't need to be installed, and ideally
in a language I could hack, if the problem is on the other side.

What's so special about these programs that they behave differently
from 'cat' etc.?  What language are they written in?

Thanks.

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 2114 bytes --]

On 01/11/2016 10:37 AM, Eli Zaretskii wrote:
> If I modify your loop.cmd to have the following trivial command:
> 
>   @cat > foobar
>
> then Emacs doesn't hang, it happily writes those 4097 bytes, and after
> that, I see all of them in the 'foobar' file.  Moreover, I tried the
> same test with 100 times as many bytes (i.e. 409700), and it still
> worked as expected, with all of the bytes ending up in the file.

My Windows doesn't have cat; maybe you installed it through cygwin?
However I tried with Python, and indeed it didn't hang. It seems I oversimplified the example.

> So someone with access to the programs in question will have to step
> with a debugger through the code involved in this (I can point to the
> relevant parts of the Emacs C code), and find out what goes wrong
> there, and why. 

I don't think my Emacs experience on Windows is quite sufficient for that, but I would be happy ot help if you can guide me. Bear in mind that at the moment I don't even have a build chain set up for Emacs on Windows.

> Alternatively, if a reproducible test case can be
> posted with a simple program which reads its stdin and causes the
> hang, I will try debugging it myself.  (By "simple program" I meant
> something that won't involve installing lots of other packages,
> ideally something as simple as a batch file calling 'cat' or a similar
> program.)

Thanks for offering your help Eli. This is the simplest that I can get for now:

Assuming node.exe (installed using https://nodejs.org/dist/v4.2.4/node-v4.2.4-x64.msi) and jshint (installed using npm install --global jshint), here is a new recipe:

* This hangs:

(let ((process  (start-process "bug" nil "node" "c:/Users/cpitclaudel/AppData/Roaming/npm/node_modules/jshint/bin/jshint" "-")))
  (process-send-string process (make-string 4097 ?a)))

* This does not hang:

(let ((process  (start-process "bug" nil "node" "c:/Users/cpitclaudel/AppData/Roaming/npm/node_modules/jshint/bin/jshint" "-")))
  (dotimes (_ 25)
    (process-send-string process (make-string 4096 ?a))))

Cheers,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

On 01/11/2016 11:31 AM, Eli Zaretskii wrote:
>> Cc: 22344@debbugs.gnu.org
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Mon, 11 Jan 2016 11:11:42 -0500
>>
>> Thanks for looking into this. Would a recipe requiring node.js and jshint be too much? Node.js has a windows installer and jshint can be installed with npm install --global jshint.
> 
> I'd prefer a program that doesn't need to be installed, and ideally
> in a language I could hack, if the problem is on the other side.
> 
> What's so special about these programs that they behave differently
> from 'cat' etc.?  What language are they written in?

I have no idea: I don't write Javascript, I don't use node, and I don't run Windows :/ node.js seems to be written in C.
I will try to remove the jshint dependency, at least.

Cheers,
Clément.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 1649 bytes --]

On 01/11/2016 11:52 AM, Clément Pit--Claudel wrote:
> On 01/11/2016 11:31 AM, Eli Zaretskii wrote:
>>> Cc: 22344@debbugs.gnu.org
>>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>>> Date: Mon, 11 Jan 2016 11:11:42 -0500
>>>
>>> Thanks for looking into this. Would a recipe requiring node.js and jshint be too much? Node.js has a windows installer and jshint can be installed with npm install --global jshint.
>>
>> I'd prefer a program that doesn't need to be installed, and ideally
>> in a language I could hack, if the problem is on the other side.
>>
>> What's so special about these programs that they behave differently
>> from 'cat' etc.?  What language are they written in?
> 
> I have no idea: I don't write Javascript, I don't use node, and I don't run Windows :/ node.js seems to be written in C.
> I will try to remove the jshint dependency, at least.

Hi Eli,

Here is a simplified test case, which only assumes node.js:

* In a file C:\blackhole.js

function blackhole() {
    var stream = process.openStdin();
    stream.setEncoding("utf-8");
    stream.on('data', function (chunk) { process.stdout.write(chunk); });
}
blackhole();

* Then from Emacs:

;; This hangs on Windows (works fine on Linux)

(let ((process  (start-process "bug" nil "node" "C:/blackhole.js")))
  (process-send-string process (make-string 4097 ?a)))

;; This works fine on both platforms:

(let ((process  (start-process "bug" nil "node" "C:/blackhole.js")))
  (dotimes (_ 25)
    (process-send-string process (make-string 4096 ?a))))

Do you think you could run this example?
Thanks again for your help.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Eli Zaretskii]

> Cc: 22344@debbugs.gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Mon, 11 Jan 2016 11:40:11 -0500
> 
> > If I modify your loop.cmd to have the following trivial command:
> > 
> >   @cat > foobar
> >
> > then Emacs doesn't hang, it happily writes those 4097 bytes, and after
> > that, I see all of them in the 'foobar' file.  Moreover, I tried the
> > same test with 100 times as many bytes (i.e. 409700), and it still
> > worked as expected, with all of the bytes ending up in the file.
> 
> My Windows doesn't have cat; maybe you installed it through cygwin?

No, it's a native Windows port of 'cat' from GNU Coreutils.

> However I tried with Python, and indeed it didn't hang. It seems I oversimplified the example.

So what do the offending programs do differently?  Can you describe
what "jshint", whatever it is, does, it how it reads its stdin?  Does
it also write its stdout, btw?

> > So someone with access to the programs in question will have to step
> > with a debugger through the code involved in this (I can point to the
> > relevant parts of the Emacs C code), and find out what goes wrong
> > there, and why. 
> 
> I don't think my Emacs experience on Windows is quite sufficient for that, but I would be happy ot help if you can guide me. Bear in mind that at the moment I don't even have a build chain set up for Emacs on Windows.

Do you have experience with GDB?  Also, was the Emacs you have
compiled with debugging symbols?  if so, I can tell you where to look.

> * This hangs:
> 
> (let ((process  (start-process "bug" nil "node" "c:/Users/cpitclaudel/AppData/Roaming/npm/node_modules/jshint/bin/jshint" "-")))
>   (process-send-string process (make-string 4097 ?a)))

What does jshint do with the stuff it reads from stdin?  Does it read
single bytes or larger chunks?

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Eli Zaretskii]

> Cc: 22344@debbugs.gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Mon, 11 Jan 2016 12:18:27 -0500
> 
> Here is a simplified test case, which only assumes node.js:
> 
> * In a file C:\blackhole.js
> 
> function blackhole() {
>     var stream = process.openStdin();
>     stream.setEncoding("utf-8");
>     stream.on('data', function (chunk) { process.stdout.write(chunk); });
> }
> blackhole();

Thanks.  Are all the affected programs written in JavaScript?  Do they
all set stdin encoding to UTF-8?  If so, does the problem go away if
you remove the line that sets the encoding?

> Do you think you could run this example?

It doesn't run on this machine, I guess I never bothered to set up
JavaScript on it.

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 2118 bytes --]

On 01/11/2016 12:39 PM, Eli Zaretskii wrote:
>> Cc: 22344@debbugs.gnu.org
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Mon, 11 Jan 2016 11:40:11 -0500
>>
>>> If I modify your loop.cmd to have the following trivial command:
>>>
>>>   @cat > foobar
>>>
>>> then Emacs doesn't hang, it happily writes those 4097 bytes, and after
>>> that, I see all of them in the 'foobar' file.  Moreover, I tried the
>>> same test with 100 times as many bytes (i.e. 409700), and it still
>>> worked as expected, with all of the bytes ending up in the file.
>>
>> My Windows doesn't have cat; maybe you installed it through cygwin?
> 
> No, it's a native Windows port of 'cat' from GNU Coreutils.
> 
>> However I tried with Python, and indeed it didn't hang. It seems I oversimplified the example.
> 
> So what do the offending programs do differently?  Can you describe
> what "jshint", whatever it is, does, it how it reads its stdin?  Does
> it also write its stdout, btw?
> 
>>> So someone with access to the programs in question will have to step
>>> with a debugger through the code involved in this (I can point to the
>>> relevant parts of the Emacs C code), and find out what goes wrong
>>> there, and why. 
>>
>> I don't think my Emacs experience on Windows is quite sufficient for that, but I would be happy ot help if you can guide me. Bear in mind that at the moment I don't even have a build chain set up for Emacs on Windows.
> 
> Do you have experience with GDB?

No; but I could learn.

> Also, was the Emacs you have
> compiled with debugging symbols?  if so, I can tell you where to look.

I would guess not.


> 
>> * This hangs:
>>
>> (let ((process  (start-process "bug" nil "node" "c:/Users/cpitclaudel/AppData/Roaming/npm/node_modules/jshint/bin/jshint" "-")))
>>   (process-send-string process (make-string 4097 ?a)))
> 
> What does jshint do with the stuff it reads from stdin?  Does it read
> single bytes or larger chunks?

I've created a simple example without JsHint (in my other response). It seems to read from stdin asynchronously.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

On 01/11/2016 12:51 PM, Eli Zaretskii wrote:
>> Cc: 22344@debbugs.gnu.org
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Mon, 11 Jan 2016 12:18:27 -0500
>>
>> Here is a simplified test case, which only assumes node.js:
>>
>> * In a file C:\blackhole.js
>>
>> function blackhole() {
>>     var stream = process.openStdin();
>>     stream.setEncoding("utf-8");
>>     stream.on('data', function (chunk) { process.stdout.write(chunk); });
>> }
>> blackhole();
> 
> Thanks.  Are all the affected programs written in JavaScript? 

Quite possibly.

> Do they
> all set stdin encoding to UTF-8?

I don't know.

> If so, does the problem go away if
> you remove the line that sets the encoding?

I'll check this immediately (I need to reboot).

>> Do you think you could run this example?
> 
> It doesn't run on this machine, I guess I never bothered to set up
> JavaScript on it.

Normally, installing node.js from https://nodejs.org/dist/v4.2.4/node-v4.2.4-x64.msi should be enough.

Thanks again!
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

On 01/11/2016 12:51 PM, Eli Zaretskii wrote:
>> Cc: 22344@debbugs.gnu.org
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Mon, 11 Jan 2016 12:18:27 -0500
>>
>> Here is a simplified test case, which only assumes node.js:
>>
>> * In a file C:\blackhole.js
>>
>> function blackhole() {
>>     var stream = process.openStdin();
>>     stream.setEncoding("utf-8");
>>     stream.on('data', function (chunk) { process.stdout.write(chunk); });
>> }
>> blackhole();
> 
> Thanks.  Are all the affected programs written in JavaScript?  Do they
> all set stdin encoding to UTF-8?  If so, does the problem go away if
> you remove the line that sets the encoding?

No, the following javascript implementation of cat also reproduces the issue:

function blackhole() {
    process.stdin.resume();
    process.stdin.setEncoding("ascii");
    process.stdin.on('data', function (chunk) { process.stdout.write(chunk); });
}
blackhole();


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Eli Zaretskii]

> Cc: 22344@debbugs.gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Mon, 11 Jan 2016 13:02:47 -0500
> 
> > Thanks.  Are all the affected programs written in JavaScript?  Do they
> > all set stdin encoding to UTF-8?  If so, does the problem go away if
> > you remove the line that sets the encoding?
> 
> No, the following javascript implementation of cat also reproduces the issue:
> 
> function blackhole() {
>     process.stdin.resume();
>     process.stdin.setEncoding("ascii");
>     process.stdin.on('data', function (chunk) { process.stdout.write(chunk); });
> }
> blackhole();

Could it be that the script exits as soon as it processed the first 4K
chunk, without waiting for the next one, or without waiting long
enough?

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 1263 bytes --]

On 01/11/2016 02:33 PM, Eli Zaretskii wrote:
>> Cc: 22344@debbugs.gnu.org
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Mon, 11 Jan 2016 13:02:47 -0500
>>
>>> Thanks.  Are all the affected programs written in JavaScript?  Do they
>>> all set stdin encoding to UTF-8?  If so, does the problem go away if
>>> you remove the line that sets the encoding?
>>
>> No, the following javascript implementation of cat also reproduces the issue:
>>
>> function blackhole() {
>>     process.stdin.resume();
>>     process.stdin.setEncoding("ascii");
>>     process.stdin.on('data', function (chunk) { process.stdout.write(chunk); });
>> }
>> blackhole();
> 
> Could it be that the script exits as soon as it processed the first 4K
> chunk, without waiting for the next one, or without waiting long
> enough?

I'm not sure if I understand correctly. In the working case (sending small batches) the underlying script does receive everything Emacs sends. In fact, the blackhole example shown above never exits; it keeps reading on stdin and echoing on stdout.

With a more complex setup, Emacs hangs when we send the whole buffer, but when sending it in small chunks the underlying process returns the expected results.

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Eli Zaretskii]

> Cc: 22344@debbugs.gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Mon, 11 Jan 2016 14:41:20 -0500
> 
> > Could it be that the script exits as soon as it processed the first 4K
> > chunk, without waiting for the next one, or without waiting long
> > enough?
> 
> I'm not sure if I understand correctly. In the working case (sending small batches) the underlying script does receive everything Emacs sends. In fact, the blackhole example shown above never exits; it keeps reading on stdin and echoing on stdout.
> 
> With a more complex setup, Emacs hangs when we send the whole buffer, but when sending it in small chunks the underlying process returns the expected results.

I tried to solve this in commit 58a622d on the emacs-25 branch.
Please try the latest and see if your problem is solved by those
changes.

Thanks.

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Benjamin Schwehn]

Hi,

I reproduced the hang using the 64bit build at
http://emacsbinw64.sourceforge.net/

I used Clément's first example using jshint to repro.

I then built a 64bit version myself, using the instructions from

http://sourceforge.net/p/emacsbinw64/wiki/Build%20guideline%20for%20MSYS2-MinGW-w64%20system/

(on a Windows 7 x64 system).

Using the latest emacs-25 sources, I can now no longer reproduce the hang.

So it looks to me like the issue if fixed. But if there is interest, I
could rebuild using the last version without 58a622d to double-check
tomorrow that there is no other difference between my build and the
build from http://emacsbinw64.sourceforge.net/

Thanks a lot for the quick fix!
Ben

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Eli Zaretskii]

> Date: Wed, 13 Jan 2016 21:20:09 +0100
> From: Benjamin Schwehn <bschwehn@gmail.com>
> 
> I reproduced the hang using the 64bit build at
> http://emacsbinw64.sourceforge.net/
> 
> I used Clément's first example using jshint to repro.
> 
> I then built a 64bit version myself, using the instructions from
> 
> http://sourceforge.net/p/emacsbinw64/wiki/Build%20guideline%20for%20MSYS2-MinGW-w64%20system/

The repository now includes a file nt/INSTALL.W64 which should cover
the same turf, and be at least as accurate as the MSYS2 instructions,
since INSTALL.W64 is maintained as part of Emacs.

> Using the latest emacs-25 sources, I can now no longer reproduce the hang.

Thanks for testing.  It is important that the original use case that
prompted this bug report will also be checked, as I understand it is
more complicated.  Can you ask the relevant people to try that?

> So it looks to me like the issue if fixed. But if there is interest, I
> could rebuild using the last version without 58a622d to double-check
> tomorrow that there is no other difference between my build and the
> build from http://emacsbinw64.sourceforge.net/

If you can afford that, yes, it would help to see that this commit
fixed the issue.

Thanks.

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 1501 bytes --]

On 01/12/2016 11:44 AM, Eli Zaretskii wrote:
>> Cc: 22344@debbugs.gnu.org
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Mon, 11 Jan 2016 14:41:20 -0500
>>
>>> Could it be that the script exits as soon as it processed the first 4K
>>> chunk, without waiting for the next one, or without waiting long
>>> enough?
>>
>> I'm not sure if I understand correctly. In the working case (sending small batches) the underlying script does receive everything Emacs sends. In fact, the blackhole example shown above never exits; it keeps reading on stdin and echoing on stdout.
>>
>> With a more complex setup, Emacs hangs when we send the whole buffer, but when sending it in small chunks the underlying process returns the expected results.
> 
> I tried to solve this in commit 58a622d on the emacs-25 branch.
> Please try the latest and see if your problem is solved by those
> changes.

Eli, there's something magical about your debugging abilities.
I downloaded and compiled the latest master on Windows 8; I could reproduce the bug there. Then I checked out emacs-25 and recompiled, and the bug seems to be gone! Amazing.

This is only for my personal curiosity, but would you mind expanding a bit on what the issue was (and how you found out)? I read the commit message and looked at the diff, but I don't think I understood everything from there. Did you have to look at node.js' source code? Would it be helpful for me to file a bug with node.js?

Clément.




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Benjamin Schwehn]

On Wed, Jan 13, 2016 at 9:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 13 Jan 2016 21:20:09 +0100
>> From: Benjamin Schwehn <bschwehn@gmail.com>
>> http://sourceforge.net/p/emacsbinw64/wiki/Build%20guideline%20for%20MSYS2-MinGW-w64%20system/
>
> The repository now includes a file nt/INSTALL.W64 which should cover
> the same turf, and be at least as accurate as the MSYS2 instructions,
> since INSTALL.W64 is maintained as part of Emacs.

Yes, i referred to those instructions and found them to be pretty
good. I mentioned the link above only because it shows the exact
params I used for ./configure.

>> Using the latest emacs-25 sources, I can now no longer reproduce the hang.
>
> Thanks for testing.  It is important that the original use case that
> prompted this bug report will also be checked, as I understand it is
> more complicated.  Can you ask the relevant people to try that?

I now also reproduced the original scenario (flycheck with checkers).
It hangs when building with the last check-in in branch emacs-25
before your fix and no longer hangs with the current version. Also
various other people where happy to found the issue fixed in the
thread over at https://github.com/flycheck/flycheck/issues/794

Thanks again,
Ben

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Eli Zaretskii]

> Cc: 22344@debbugs.gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Wed, 13 Jan 2016 19:48:38 -0500
> 
> Eli, there's something magical about your debugging abilities.
> I downloaded and compiled the latest master on Windows 8; I could reproduce the bug there. Then I checked out emacs-25 and recompiled, and the bug seems to be gone! Amazing.

I wish I had such magic, but I don't.  It's just a lot of hard work,
that's all.  But thank you for your kind words.

> This is only for my personal curiosity, but would you mind expanding a bit on what the issue was (and how you found out)? I read the commit message and looked at the diff, but I don't think I understood everything from there. Did you have to look at node.js' source code? Would it be helpful for me to file a bug with node.js?

I'm not entirely sure what caused the issue.  From the Emacs side, it
looked like the other end of the pipe wasn't read at all: each time
Emacs called _write, it got the return value of -1 with errno set to
ENOSPC, which means the pipe's buffer is full.  No other program with
which I tried the same test case did this.  With those other programs,
after a few first attempts which returned -1, _write started to return
positive values, meaning that it succeeded to write some part of the
stuff.

I tried to google for similar problems related to node.js, but didn't
find anything that looked useful.

So I just went with the only workaround you reported: never write to
the pipe more than its buffer size.  I.e., subdivide the write into
chinks the size of the pipe's buffer.  This required a system call, to
find out what that size was, and it also required a small change in
the logic of the loop that writes the stuff in chunks, because it
wasn't ready for the situation that a chunk succeeds, but the very
next one returns -1 and ENOSPC -- again something that only node.js
causes somehow.

No, I didn't look into node.js's sources.  It's a large program with
whose design I'm not familiar, so I expected it won't be easy to find
the code that is responsible for this.  I took a brief look now, and I
see that my expectations were correct: I see some very elaborate "pipe
server" there, which I'm not sure is relevant to the original problem.

If we want to try to understand this deeper, we should be talking to
some expert on how node.js works on Windows wrt I/O on its standard
input and standard output.  Filing a bug report with node.js might be
useful, but again, only if someone there will want to talk to us so
that we could understand the problem.

If you want to file a node.js bug report, you can start by saying that
it reads its stdin very slowly: once it reads a 4K chunk from the
pipe, it takes about 40 to 60 msec before it is ready to read the next
chunk.  (I can tell because when Emacs receives ENOSPC from the pipe,
it loops retrying, waiting 20 msec between retries, and I see 2 to 3
retries, sometimes even 4, before the next successful write.)  I don't
know why it takes so long -- this was on a Core i7 system which was
otherwise very quiet.  Maybe if we understand what takes so long, we
will also understand what did Emacs do wrongly, and how to avoid that.

For now, I guess we can be happy with the workaround I committed.

Thanks.

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Eli Zaretskii]

> Date: Thu, 14 Jan 2016 17:36:28 +0100
> From: Benjamin Schwehn <bschwehn@gmail.com>
> Cc: 22344@debbugs.gnu.org
> 
> > Thanks for testing.  It is important that the original use case that
> > prompted this bug report will also be checked, as I understand it is
> > more complicated.  Can you ask the relevant people to try that?
> 
> I now also reproduced the original scenario (flycheck with checkers).
> It hangs when building with the last check-in in branch emacs-25
> before your fix and no longer hangs with the current version. Also
> various other people where happy to found the issue fixed in the
> thread over at https://github.com/flycheck/flycheck/issues/794

Thanks, that's good enough for me.  So I'm marking this bug done.

bug#22344: 25.0.50 (and 24.5.1, 24.4); Sending 4097 bytes to a subprocess hangs Emacs on Windows 8 and 10

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 1619 bytes --]

On 01/14/2016 01:01 PM, Eli Zaretskii wrote:
>> Cc: 22344@debbugs.gnu.org
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Wed, 13 Jan 2016 19:48:38 -0500
>>
>> Eli, there's something magical about your debugging abilities.
>> I downloaded and compiled the latest master on Windows 8; I could reproduce the bug there. Then I checked out emacs-25 and recompiled, and the bug seems to be gone! Amazing.
> 
> I wish I had such magic, but I don't.  It's just a lot of hard work,
> that's all.  But thank you for your kind words.
> 
>> This is only for my personal curiosity, but would you mind expanding a bit on what the issue was (and how you found out)? I read the commit message and looked at the diff, but I don't think I understood everything from there. Did you have to look at node.js' source code? Would it be helpful for me to file a bug with node.js?
> 
> I'm not entirely sure what caused the issue.  From the Emacs side, it
> looked like the other end of the pipe wasn't read at all: each time
> Emacs called _write, it got the return value of -1 with errno set to
> ENOSPC, which means the pipe's buffer is full.  No other program with
> which I tried the same test case did this.  With those other programs,
> after a few first attempts which returned -1, _write started to return
> positive values, meaning that it succeeded to write some part of the
> stuff.
> (...)

Thanks a lot for the explanations! I've raised an issue with the Node.js people, inviting them to join this discussion if there was interest about fixing this on their side.

Cheers,
Clément. 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]