master reporting "creating pipe: too many open files"

master reporting "creating pipe: too many open files"

[Stephen Leake]

[-- Attachment #1: Type: text/plain, Size: 972 bytes --]

I have a custom git front-end that works fine in emacs 28, but in
current emacs master it always reports "creating pipe: too many open
files" after a commands. Then any subsequent command in emacs that
requires a file (such as saving a buffer) also fails.

This is on Windows 8.1, emacs built with mingw64.

I'm guessing there is some file/process handle that is opened and not
closed.

I update my local copy of emacs master rarely, so this change happened
sometime since Dec 2021; I have not tried to bisect.

Attached is some simplified code that shows the problem; it runs a
subprocess the same way the git front-end does. To run, load the file,
then run M-x dd-test. On my system, master is ok with 10 processes,
fails with 100 (I did not bisect further). Emacs 28 is ok forever.

Before I submit a bug report, I want to check that there is nothing
simple wrong with the code; is it relying on undocumented behaviour in
emacs 28, or something similar?

-- 
-- Stephe

[-- Attachment #2: debug-process.el --]
[-- Type: application/emacs-lisp, Size: 1368 bytes --]

Re: master reporting "creating pipe: too many open files"

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 16 Jul 2022 09:58:18 -0700
> 
> I have a custom git front-end that works fine in emacs 28, but in
> current emacs master it always reports "creating pipe: too many open
> files" after a commands. Then any subsequent command in emacs that
> requires a file (such as saving a buffer) also fails.
> 
> This is on Windows 8.1, emacs built with mingw64.
> 
> I'm guessing there is some file/process handle that is opened and not
> closed.
> 
> I update my local copy of emacs master rarely, so this change happened
> sometime since Dec 2021; I have not tried to bisect.
> 
> Attached is some simplified code that shows the problem; it runs a
> subprocess the same way the git front-end does. To run, load the file,
> then run M-x dd-test. On my system, master is ok with 10 processes,
> fails with 100 (I did not bisect further). Emacs 28 is ok forever.
> 
> Before I submit a bug report, I want to check that there is nothing
> simple wrong with the code; is it relying on undocumented behaviour in
> emacs 28, or something similar?

You code starts subprocesses in a loop.

Emacs on MS-Windows cannot support more than 30 subprocesses (assuming
no other process objects, like network connection, exist).  That's
because the pselect emulation is limited to 64 handles, and we use 2
handles per subprocess.

Is what you see consistent with this limitation?

Re: master reporting "creating pipe: too many open files"

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 16 Jul 2022 09:58:18 -0700
> 
> I have a custom git front-end that works fine in emacs 28, but in
> current emacs master it always reports "creating pipe: too many open
> files" after a commands. Then any subsequent command in emacs that
> requires a file (such as saving a buffer) also fails.
> 
> This is on Windows 8.1, emacs built with mingw64.
> 
> I'm guessing there is some file/process handle that is opened and not
> closed.
> 
> I update my local copy of emacs master rarely, so this change happened
> sometime since Dec 2021; I have not tried to bisect.
> 
> Attached is some simplified code that shows the problem; it runs a
> subprocess the same way the git front-end does. To run, load the file,
> then run M-x dd-test. On my system, master is ok with 10 processes,
> fails with 100 (I did not bisect further). Emacs 28 is ok forever.

AFAICT, this loop in deactivate_process:

  /* Beware SIGCHLD hereabouts.  */

  for (i = 0; i < PROCESS_OPEN_FDS; i++)
    close_process_fd (&p->open_fd[i]);

doesn't close the last of the 4 descriptors opened by the 2 emacs_pipe
calls in make-pipe-process.  It calls 'close' with the right value,
and close returns zero, but the pipe stays open.  In Emacs 28, this
same loop successfully closes the descriptor.  I don't know why.

Perhaps bisecting could help.

Re: master reporting "creating pipe: too many open files"

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Sat, 16 Jul 2022 09:58:18 -0700
>> 
>> I have a custom git front-end that works fine in emacs 28, but in
>> current emacs master it always reports "creating pipe: too many open
>> files" after a commands. Then any subsequent command in emacs that
>> requires a file (such as saving a buffer) also fails.
>> 
>> This is on Windows 8.1, emacs built with mingw64.
>> 
>> I'm guessing there is some file/process handle that is opened and not
>> closed.
>> 
>> I update my local copy of emacs master rarely, so this change happened
>> sometime since Dec 2021; I have not tried to bisect.
>> 
>> Attached is some simplified code that shows the problem; it runs a
>> subprocess the same way the git front-end does. To run, load the file,
>> then run M-x dd-test. On my system, master is ok with 10 processes,
>> fails with 100 (I did not bisect further). Emacs 28 is ok forever.
>> 
>> Before I submit a bug report, I want to check that there is nothing
>> simple wrong with the code; is it relying on undocumented behaviour in
>> emacs 28, or something similar?
>
> You code starts subprocesses in a loop.

Yes, but it waits for each to complete before starting the next one.
That's why it works fine in Emacs 28.

> Emacs on MS-Windows cannot support more than 30 subprocesses (assuming
> no other process objects, like network connection, exist).  That's
> because the pselect emulation is limited to 64 handles, and we use 2
> handles per subprocess.
>
> Is what you see consistent with this limitation?

No. That has not changed between 28 and master.

I did test on Debian; no problems there.

I'll create a bug report.

-- 
-- Stephe

Re: master reporting "creating pipe: too many open files"

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Sat, 16 Jul 2022 09:58:18 -0700
>> 
>> I have a custom git front-end that works fine in emacs 28, but in
>> current emacs master it always reports "creating pipe: too many open
>> files" after a commands. Then any subsequent command in emacs that
>> requires a file (such as saving a buffer) also fails.
>> 
>> This is on Windows 8.1, emacs built with mingw64.
>> 
>> I'm guessing there is some file/process handle that is opened and not
>> closed.
>> 
>> I update my local copy of emacs master rarely, so this change happened
>> sometime since Dec 2021; I have not tried to bisect.
>> 
>> Attached is some simplified code that shows the problem; it runs a
>> subprocess the same way the git front-end does. To run, load the file,
>> then run M-x dd-test. On my system, master is ok with 10 processes,
>> fails with 100 (I did not bisect further). Emacs 28 is ok forever.
>
> AFAICT, this loop in deactivate_process:
>
>   /* Beware SIGCHLD hereabouts.  */
>
>   for (i = 0; i < PROCESS_OPEN_FDS; i++)
>     close_process_fd (&p->open_fd[i]);
>
> doesn't close the last of the 4 descriptors opened by the 2 emacs_pipe
> calls in make-pipe-process.  It calls 'close' with the right value,
> and close returns zero, but the pipe stays open.  In Emacs 28, this
> same loop successfully closes the descriptor.  I don't know why.
>
> Perhaps bisecting could help.

I'll do that.

I reported this as bug#56606

-- 
-- Stephe