bug#51110: Native-comp leaks fds open on /dev/ptmx

bug#51110: Native-comp leaks fds open on /dev/ptmx

[Stephen Gildea]

Version: 28.0.60
Severity: important

Native-comp leaks fds open on /dev/ptmx if you change 'invocation-name'
and then run a function that loads libraries and causes some compilation.
Emacs seems to compile each file multiple times (16 or more), and it soon
runs out of file descriptors.

To reproduce:

Use a newly built (configure --with-native-compilation) Emacs that has
never (successfully) run MH-E before; if your eln-cache has already been
populated, you will need to delete the cache to repeat the failure:

rm -r ~/.emacs.d/eln-cache

The Emacs must be interactive, and you can reproduce quickly as follows:

src/emacs -Q --eval '(setq invocation-name "emacs-leaks")' --eval '(mh-version)'

In a few seconds the echo area will say: "Creating pipe: Too many open files"

What should appear in the echo area: "nmh 1.7.1 installed as MH variant"
or maybe: "No MH variant found on the system"

bug#51110: Native-comp leaks fds open on /dev/ptmx

[Eli Zaretskii]

> From: Stephen Gildea <stepheng+emacs@gildea.com>
> Date: Sat, 09 Oct 2021 08:45:25 -0700
> 
> src/emacs -Q --eval '(setq invocation-name "emacs-leaks")' --eval '(mh-version)'

What real-life use case requires you to modify invocation-name?  That
variable is supposed to be read-only: it tells the program name used
to invoke Emacs of the running session.  If you change it, other
features will become confused.

IOW, I don't think we want to support this kind of usage.

bug#51110: Native-comp leaks fds open on /dev/ptmx

[Stephen Gildea]

The recipe I gave provides a simple way to show the unbounded resource
use of Emacs while compiling; I don't propose it as an actual use case.

Apparently what's happening is that Emacs forks off a child Emacs to do
the compile and allocates a pty to the child.  These ptys are not
closed until the compile completes.  It seems Emacs needs to better
manage the number of simultaneous open ptys.  Or at least be able to
wait and retry if an open fails.

bug#51110: Native-comp leaks fds open on /dev/ptmx

[Eli Zaretskii]

> From: Stephen Gildea <stepheng+emacs@gildea.com>
> cc: Andrea Corallo <akrl@sdf.org>, 51110@debbugs.gnu.org
> Date: Sat, 09 Oct 2021 10:01:43 -0700
> 
> Apparently what's happening is that Emacs forks off a child Emacs to do
> the compile and allocates a pty to the child.  These ptys are not
> closed until the compile completes.  It seems Emacs needs to better
> manage the number of simultaneous open ptys.  Or at least be able to
> wait and retry if an open fails.

Emacs limits the number of these background processes, see
native-comp-async-jobs-number.  (The value zero means use half of the
number of execution units available in the system's processor.)  So we
generally shouldn't have more than that number of open pipes at any
given time.  For the obvious reasons they cannot be closed until the
sub-process exits.

bug#51110: Native-comp leaks fds open on /dev/ptmx

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Gildea <stepheng+emacs@gildea.com>
>> cc: Andrea Corallo <akrl@sdf.org>, 51110@debbugs.gnu.org
>> Date: Sat, 09 Oct 2021 10:01:43 -0700
>> 
>> Apparently what's happening is that Emacs forks off a child Emacs to do
>> the compile and allocates a pty to the child.  These ptys are not
>> closed until the compile completes.  It seems Emacs needs to better
>> manage the number of simultaneous open ptys.  Or at least be able to
>> wait and retry if an open fails.
>
> Emacs limits the number of these background processes, see
> native-comp-async-jobs-number.  (The value zero means use half of the
> number of execution units available in the system's processor.)  So we
> generally shouldn't have more than that number of open pipes at any
> given time.  For the obvious reasons they cannot be closed until the
> sub-process exits.

Is there any action we need/can take on this bug?

Thanks

  Andrea

bug#51110: Native-comp leaks fds open on /dev/ptmx

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Stephen Gildea <stepheng+emacs@gildea.com>, 51110@debbugs.gnu.org
> Date: Tue, 30 Nov 2021 20:13:07 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Stephen Gildea <stepheng+emacs@gildea.com>
> >> cc: Andrea Corallo <akrl@sdf.org>, 51110@debbugs.gnu.org
> >> Date: Sat, 09 Oct 2021 10:01:43 -0700
> >> 
> >> Apparently what's happening is that Emacs forks off a child Emacs to do
> >> the compile and allocates a pty to the child.  These ptys are not
> >> closed until the compile completes.  It seems Emacs needs to better
> >> manage the number of simultaneous open ptys.  Or at least be able to
> >> wait and retry if an open fails.
> >
> > Emacs limits the number of these background processes, see
> > native-comp-async-jobs-number.  (The value zero means use half of the
> > number of execution units available in the system's processor.)  So we
> > generally shouldn't have more than that number of open pipes at any
> > given time.  For the obvious reasons they cannot be closed until the
> > sub-process exits.
> 
> Is there any action we need/can take on this bug?

I don't think there's anything we _can_ do here, and I tried to
explain why.

bug#51110: Native-comp leaks fds open on /dev/ptmx

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Stephen Gildea <stepheng+emacs@gildea.com>, 51110@debbugs.gnu.org
>> Date: Tue, 30 Nov 2021 20:13:07 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Stephen Gildea <stepheng+emacs@gildea.com>
>> >> cc: Andrea Corallo <akrl@sdf.org>, 51110@debbugs.gnu.org
>> >> Date: Sat, 09 Oct 2021 10:01:43 -0700
>> >> 
>> >> Apparently what's happening is that Emacs forks off a child Emacs to do
>> >> the compile and allocates a pty to the child.  These ptys are not
>> >> closed until the compile completes.  It seems Emacs needs to better
>> >> manage the number of simultaneous open ptys.  Or at least be able to
>> >> wait and retry if an open fails.
>> >
>> > Emacs limits the number of these background processes, see
>> > native-comp-async-jobs-number.  (The value zero means use half of the
>> > number of execution units available in the system's processor.)  So we
>> > generally shouldn't have more than that number of open pipes at any
>> > given time.  For the obvious reasons they cannot be closed until the
>> > sub-process exits.
>> 
>> Is there any action we need/can take on this bug?
>
> I don't think there's anything we _can_ do here, and I tried to
> explain why.

I agree, I don't think we really want support `invocation-name'
modifications at runtime.

Shall we close this bug?

  Andrea

bug#51110: Native-comp leaks fds open on /dev/ptmx

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: stepheng+emacs@gildea.com, 51110@debbugs.gnu.org
> Date: Tue, 30 Nov 2021 20:24:59 +0000
> 
> > I don't think there's anything we _can_ do here, and I tried to
> > explain why.
> 
> I agree, I don't think we really want support `invocation-name'
> modifications at runtime.
> 
> Shall we close this bug?

I think so.  But before we do, I'd like to let Stephen a chance to
convince us otherwise.

bug#51110: Native-comp leaks fds open on /dev/ptmx

[Stephen Gildea]

Eli Zaretskii <eliz@gnu.org> wrote:

>   > From: Andrea Corallo <akrl@sdf.org>
>   > Cc: stepheng+emacs@gildea.com, 51110@debbugs.gnu.org
>   > Date: Tue, 30 Nov 2021 20:24:59 +0000
>   > 
>   > > I don't think there's anything we _can_ do here, and I tried to
>   > > explain why.
>   > 
>   > I agree, I don't think we really want support `invocation-name'
>   > modifications at runtime.
>   > 
>   > Shall we close this bug?
>   
>   I think so.  But before we do, I'd like to let Stephen a chance to
>   convince us otherwise.

I'm happy to see this bug closed, and I hereby do so.
Thank you for the discussion.