bug#46494: 28.0.50; [native-comp] Problems with async background compile

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andy Moreton]

Problems notes with async compile in native-comp branch on Windows:

a) Bug #46256 describes problems with AOT compiled native-comp emacs not
    finding prebuilt .eln files when built for mingw64 64bit on Windows.

    As a result, emacs complains with an echo area warning for every .eln
    file that it cannot find in the expected location.

    The stream of frequent warnings that causes make emacs mostly
    unresponsive to user input.

b) The "background" async compilation of .eln files is CPU intensive and
    somewhat slow. The default settings run a compile on every available
    core, which is unfriendly for other workloads running on the same
    machine.

    It would be helpful to users to have a command to show the state of
    the async background compilation, including the running compile
    processes and the queue of pending compilation requests.

c) Quitting emacs when async compilation processes are running sometimes
    causes crashes in the compile processes, which show the emacs abort
    dialog (once for each async process). The dialogs disappear after a
    short delay (presumably due to the parent emacs having exited).

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 13 Feb 2021 16:58:11 +0000
> 
> c) Quitting emacs when async compilation processes are running sometimes
>     causes crashes in the compile processes, which show the emacs abort
>     dialog (once for each async process). The dialogs disappear after a
>     short delay (presumably due to the parent emacs having exited).

Do the crashing processes run Emacs or do they run the compiler?  If
the former, can you attach GDB to them and show backtraces?

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andy Moreton]

On Sat 13 Feb 2021, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Sat, 13 Feb 2021 16:58:11 +0000
>> 
>> c) Quitting emacs when async compilation processes are running sometimes
>>     causes crashes in the compile processes, which show the emacs abort
>>     dialog (once for each async process). The dialogs disappear after a
>>     short delay (presumably due to the parent emacs having exited).
>
> Do the crashing processes run Emacs or do they run the compiler?  If
> the former, can you attach GDB to them and show backtraces?

I think it is the compiler processes: I get multiple emacs abort dialogs
rather than just one. I've tried attaching gdb, which does not manage to
produce a backtrace.

    AndyM

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 13 Feb 2021 20:20:46 +0000
> 
> >> c) Quitting emacs when async compilation processes are running sometimes
> >>     causes crashes in the compile processes, which show the emacs abort
> >>     dialog (once for each async process). The dialogs disappear after a
> >>     short delay (presumably due to the parent emacs having exited).
> >
> > Do the crashing processes run Emacs or do they run the compiler?  If
> > the former, can you attach GDB to them and show backtraces?
> 
> I think it is the compiler processes: I get multiple emacs abort dialogs
> rather than just one. I've tried attaching gdb, which does not manage to
> produce a backtrace.

I think it's Emacs that crashes, because otherwise why is the Emacs
abort di1alog shown?

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> Problems notes with async compile in native-comp branch on Windows:
>
> a) Bug #46256 describes problems with AOT compiled native-comp emacs not
>    finding prebuilt .eln files when built for mingw64 64bit on Windows.
>
>    As a result, emacs complains with an echo area warning for every .eln
>    file that it cannot find in the expected location.

Please lets discuss each bug in each own thread, there's no need to use
more then one place.

>    The stream of frequent warnings that causes make emacs mostly
>    unresponsive to user input.

You can set `comp-async-report-warnings-errors' to nil if the packages
you use have to many compilation warnings.

For the motivation on reporting these warnings and more please see
bug#44746.

> b) The "background" async compilation of .eln files is CPU intensive and
>    somewhat slow. The default settings run a compile on every available
>    core, which is unfriendly for other workloads running on the same
>    machine.

The default setting ATM should be to run on half of the cores (see
`comp-effective-async-max-jobs'), if that's not the case on Windows
should be fixed.

>    It would be helpful to users to have a command to show the state of
>    the async background compilation, including the running compile
>    processes and the queue of pending compilation requests.

M-x list-processes
M: comp-files-queue

Ideally would be nicer to have a dedicated way to present the async
compilation status, but this does the job (for me at least).

> c) Quitting emacs when async compilation processes are running sometimes
>    causes crashes in the compile processes, which show the emacs abort
>    dialog (once for each async process). The dialogs disappear after a
>    short delay (presumably due to the parent emacs having exited).

Mmmh, I guess this is a Windows specific behavior.  Is there a specific
way to shut-down child processes we would use on Windows not to get
this error?

Thanks

  Andrea

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Eli Zaretskii]

> Cc: 46494@debbugs.gnu.org
> Date: Sun, 14 Feb 2021 08:01:59 +0000
> From:  Andrea Corallo via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> > c) Quitting emacs when async compilation processes are running sometimes
> >    causes crashes in the compile processes, which show the emacs abort
> >    dialog (once for each async process). The dialogs disappear after a
> >    short delay (presumably due to the parent emacs having exited).
> 
> Mmmh, I guess this is a Windows specific behavior.  Is there a specific
> way to shut-down child processes we would use on Windows not to get
> this error?

Andrea, can you point me to the place where we interrupt async
compilations when Emacs exits?  Is that just a normal delete-process,
or do we do something else, like sending a signal?  Also, does the
Emacs subprocess invoked to perform async compilation spawn further
child processes, or is everything happening inside a single Emacs
process?

Armed with that knowledge, I will try to look at the code and figure
out why the compiling process crashes.

Thanks.

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: 46494@debbugs.gnu.org
>> Date: Sun, 14 Feb 2021 08:01:59 +0000
>> From:  Andrea Corallo via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> > c) Quitting emacs when async compilation processes are running sometimes
>> >    causes crashes in the compile processes, which show the emacs abort
>> >    dialog (once for each async process). The dialogs disappear after a
>> >    short delay (presumably due to the parent emacs having exited).
>> 
>> Mmmh, I guess this is a Windows specific behavior.  Is there a specific
>> way to shut-down child processes we would use on Windows not to get
>> this error?
>
> Andrea, can you point me to the place where we interrupt async
> compilations when Emacs exits?  Is that just a normal delete-process,
> or do we do something else, like sending a signal?  Also, does the
> Emacs subprocess invoked to perform async compilation spawn further
> child processes, or is everything happening inside a single Emacs
> process?

Hi Eli,

we have no special handling for closing async compilation processes,
they should be closed as all child processes started by Emacs are.

In the child process we do not spawn directly any other process, but
libgccjit might do it (ex to call gas).

Thanks

  Andrea

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: andrewjmoreton@gmail.com, 46494@debbugs.gnu.org
> Date: Sun, 14 Feb 2021 18:28:02 +0000
> 
> we have no special handling for closing async compilation processes,
> they should be closed as all child processes started by Emacs are.
> 
> In the child process we do not spawn directly any other process, but
> libgccjit might do it (ex to call gas).

OK, I will take a look.  Thanks.

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: andrewjmoreton@gmail.com, 46494@debbugs.gnu.org
> Date: Sun, 14 Feb 2021 18:28:02 +0000
> 
> >> > c) Quitting emacs when async compilation processes are running sometimes
> >> >    causes crashes in the compile processes, which show the emacs abort
> >> >    dialog (once for each async process). The dialogs disappear after a
> >> >    short delay (presumably due to the parent emacs having exited).
> >> 
> >> Mmmh, I guess this is a Windows specific behavior.  Is there a specific
> >> way to shut-down child processes we would use on Windows not to get
> >> this error?
> >
> > Andrea, can you point me to the place where we interrupt async
> > compilations when Emacs exits?  Is that just a normal delete-process,
> > or do we do something else, like sending a signal?  Also, does the
> > Emacs subprocess invoked to perform async compilation spawn further
> > child processes, or is everything happening inside a single Emacs
> > process?
> 
> Hi Eli,
> 
> we have no special handling for closing async compilation processes,
> they should be closed as all child processes started by Emacs are.
> 
> In the child process we do not spawn directly any other process, but
> libgccjit might do it (ex to call gas).

OK, I've reviewed the code which kills subprocesses when Emacs is shut
down, and I have some questions:

  . How does libgccjit handle the case that its process is exiting?
    Does it have any atexit handlers or static destructors?  IOW, how
    does it ensure its own subprocesses, like gas etc. are terminated?

  . When we invoke Emacs in a subprocess to do the async compilation,
    do we specify that it should be killed without query?  I don't see
    this in the code (did I miss it?), but if we don't, then exiting
    Emacs will ask the user whether to kill the subprocesses -- does
    it?

Andy, if instead if exiting Emacs, you use signal-process, like this:

  M-: (signal-process PROC-ID 'SIGHUP) RET

(where PROC-ID is the process ID of the Emacs subprocess running the
native compilation), do you see the same crash, or does the subprocess
exit cleanly?  To see the PROC-ID, you can use the Task manager or the
'pslist' command from the PsTools suite.

Thanks.

P.S. Andrea, I see you use "path" in comp.el (and perhaps elsewhere)
to mean "file name", but the GNU Coding Standards frown on using this
for anything other than PATH-style directory lists.  So this should at
some point be replaced with "file name".

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andy Moreton]

On Sat 20 Feb 2021, Eli Zaretskii wrote:

> OK, I've reviewed the code which kills subprocesses when Emacs is shut
> down, and I have some questions:
>
>   . How does libgccjit handle the case that its process is exiting?
>     Does it have any atexit handlers or static destructors?  IOW, how
>     does it ensure its own subprocesses, like gas etc. are terminated?
>
>   . When we invoke Emacs in a subprocess to do the async compilation,
>     do we specify that it should be killed without query?  I don't see
>     this in the code (did I miss it?), but if we don't, then exiting
>     Emacs will ask the user whether to kill the subprocesses -- does
>     it?
>
> Andy, if instead if exiting Emacs, you use signal-process, like this:
>
>   M-: (signal-process PROC-ID 'SIGHUP) RET
>
> (where PROC-ID is the process ID of the Emacs subprocess running the
> native compilation), do you see the same crash, or does the subprocess
> exit cleanly?  To see the PROC-ID, you can use the Task manager or the
> 'pslist' command from the PsTools suite.

I tried that by adding binding this to a key:
  (defun signal-hup (proc)
    (interactive "nProcess: ")
    (signal-process proc 'SIGHUP))

On a x86_64-w64-mingw32 build, sending SIGHUP to a compilation
subprocess results in the emacs abort dialog being shown briefly and
then disappearing (without user interaction). That dialog should require
pressing a button to dismiss it.

    AndyM

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 20 Feb 2021 12:09:06 +0000
> 
> >   M-: (signal-process PROC-ID 'SIGHUP) RET
> >
> > (where PROC-ID is the process ID of the Emacs subprocess running the
> > native compilation), do you see the same crash, or does the subprocess
> > exit cleanly?  To see the PROC-ID, you can use the Task manager or the
> > 'pslist' command from the PsTools suite.
> 
> I tried that by adding binding this to a key:
>   (defun signal-hup (proc)
>     (interactive "nProcess: ")
>     (signal-process proc 'SIGHUP))
> 
> On a x86_64-w64-mingw32 build, sending SIGHUP to a compilation
> subprocess results in the emacs abort dialog being shown briefly and
> then disappearing (without user interaction). That dialog should require
> pressing a button to dismiss it.

I think if the process dies or exits, the dialog is closed.

So, since you seem to be able to reproduce this with a simpler setup,
please try these:

  . repeat the experiment using 'SIGINT and 'SIGBREAK instead of
    'SIGHUP
  . repeat the experiment with w32-start-process-share-console set to
    a non-nil value (both with SIGHUP and the other 2 SIG* signals)

I'd be interested to know whether the results are different.

(If you have no time for these experiments, just leave this until I
get to testing the branch.  TIA.)

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andy Moreton]

On Sat 20 Feb 2021, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Sat, 20 Feb 2021 12:09:06 +0000
>> 
>> >   M-: (signal-process PROC-ID 'SIGHUP) RET
>> >
>> > (where PROC-ID is the process ID of the Emacs subprocess running the
>> > native compilation), do you see the same crash, or does the subprocess
>> > exit cleanly?  To see the PROC-ID, you can use the Task manager or the
>> > 'pslist' command from the PsTools suite.
>> 
>> I tried that by adding binding this to a key:
>>   (defun signal-hup (proc)
>>     (interactive "nProcess: ")
>>     (signal-process proc 'SIGHUP))
>> 
>> On a x86_64-w64-mingw32 build, sending SIGHUP to a compilation
>> subprocess results in the emacs abort dialog being shown briefly and
>> then disappearing (without user interaction). That dialog should require
>> pressing a button to dismiss it.
>
> I think if the process dies or exits, the dialog is closed.
>
> So, since you seem to be able to reproduce this with a simpler setup,
> please try these:
>
>   . repeat the experiment using 'SIGINT and 'SIGBREAK instead of
>     'SIGHUP
>   . repeat the experiment with w32-start-process-share-console set to
>     a non-nil value (both with SIGHUP and the other 2 SIG* signals)
>
> I'd be interested to know whether the results are different.

Using 'SIGINT or 'SIGBREAK did not seem affect the subprocess at all.

Using "(w32-start-process-share-console t)" with 'SIGHUP seemed to work
without triggering the abort dialog, and killing emacs with async
processes running also did, not trigger the aborts.

    AndyM

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 20 Feb 2021 16:40:38 +0000
> 
> Using "(w32-start-process-share-console t)" with 'SIGHUP seemed to work
> without triggering the abort dialog, and killing emacs with async
> processes running also did, not trigger the aborts.

Great, thanks for trying this.  So at the very least we have a
workaround, if not the solution.

FTR, setting w32-start-process-share-console non-nil causes the
subprocess to start a new process group, and sending SIGHUP to the
inferior Emacs then kills it via the TerminateProcess API, whereas
when w32-start-process-share-console is nil (the default), we try
being nice, and send it the WM_CLOSE message.

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andy Moreton]

On Sat 20 Feb 2021, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Sat, 20 Feb 2021 16:40:38 +0000
>> 
>> Using "(w32-start-process-share-console t)" with 'SIGHUP seemed to work
>> without triggering the abort dialog, and killing emacs with async
>> processes running also did, not trigger the aborts.
>
> Great, thanks for trying this.  So at the very least we have a
> workaround, if not the solution.

Yes. It also seemed faster, in that I had to be quicker at entering
process IDs to send a single before the subprocess exited normally.

> FTR, setting w32-start-process-share-console non-nil causes the
> subprocess to start a new process group, and sending SIGHUP to the
> inferior Emacs then kills it via the TerminateProcess API, whereas
> when w32-start-process-share-console is nil (the default), we try
> being nice, and send it the WM_CLOSE message.

If being polite does not work, then we will need the more forceful
approach.

    AndyM

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 20 Feb 2021 17:30:40 +0000
> 
> > Great, thanks for trying this.  So at the very least we have a
> > workaround, if not the solution.
> 
> Yes. It also seemed faster, in that I had to be quicker at entering
> process IDs to send a single before the subprocess exited normally.

Maybe because when w32-start-process-share-console is nil, we tell
CreateProcess to create a new console for the sub-process, and that
takes more time for the OS.

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 20 Feb 2021 16:40:38 +0000
> 
> >> On a x86_64-w64-mingw32 build, sending SIGHUP to a compilation
> >> subprocess results in the emacs abort dialog being shown briefly and
> >> then disappearing (without user interaction). That dialog should require
> >> pressing a button to dismiss it.
> >
> > I think if the process dies or exits, the dialog is closed.
> >
> > So, since you seem to be able to reproduce this with a simpler setup,
> > please try these:
> >
> >   . repeat the experiment using 'SIGINT and 'SIGBREAK instead of
> >     'SIGHUP
> >   . repeat the experiment with w32-start-process-share-console set to
> >     a non-nil value (both with SIGHUP and the other 2 SIG* signals)
> >
> > I'd be interested to know whether the results are different.
> 
> Using 'SIGINT or 'SIGBREAK did not seem affect the subprocess at all.
> 
> Using "(w32-start-process-share-console t)" with 'SIGHUP seemed to work
> without triggering the abort dialog, and killing emacs with async
> processes running also did, not trigger the aborts.

I think I found and fixed the problem which caused these abort dialogs
to pop up when exiting Emacs while native-compile subprocesses are
running.  Please try the latest native-comp branch.

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andy Moreton]

On Sun 07 Mar 2021, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Sat, 20 Feb 2021 16:40:38 +0000
>> 
>> >> On a x86_64-w64-mingw32 build, sending SIGHUP to a compilation
>> >> subprocess results in the emacs abort dialog being shown briefly and
>> >> then disappearing (without user interaction). That dialog should require
>> >> pressing a button to dismiss it.
>> >
>> > I think if the process dies or exits, the dialog is closed.
>> >
>> > So, since you seem to be able to reproduce this with a simpler setup,
>> > please try these:
>> >
>> >   . repeat the experiment using 'SIGINT and 'SIGBREAK instead of
>> >     'SIGHUP
>> >   . repeat the experiment with w32-start-process-share-console set to
>> >     a non-nil value (both with SIGHUP and the other 2 SIG* signals)
>> >
>> > I'd be interested to know whether the results are different.
>> 
>> Using 'SIGINT or 'SIGBREAK did not seem affect the subprocess at all.
>> 
>> Using "(w32-start-process-share-console t)" with 'SIGHUP seemed to work
>> without triggering the abort dialog, and killing emacs with async
>> processes running also did, not trigger the aborts.
>
> I think I found and fixed the problem which caused these abort dialogs
> to pop up when exiting Emacs while native-compile subprocesses are
> running.  Please try the latest native-comp branch.

From commit 15aa239ba058ef02544e5dfaf066bd985d9b2f4f I tried building
for for i686-w64-mingw32 using "--with-wide-int" and no longer see the
abort dialogs (after a few tests with 4 async compiles running), so I
think this is fixed.

Thanks,

    AndyM

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Mon, 08 Mar 2021 00:19:27 +0000
> 
> > I think I found and fixed the problem which caused these abort dialogs
> > to pop up when exiting Emacs while native-compile subprocesses are
> > running.  Please try the latest native-comp branch.
> 
> >From commit 15aa239ba058ef02544e5dfaf066bd985d9b2f4f I tried building
> for for i686-w64-mingw32 using "--with-wide-int" and no longer see the
> abort dialogs (after a few tests with 4 async compiles running), so I
> think this is fixed.

Great, thanks for testing.

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: andrewjmoreton@gmail.com, 46494@debbugs.gnu.org
>> Date: Sun, 14 Feb 2021 18:28:02 +0000
>> 
>> >> > c) Quitting emacs when async compilation processes are running sometimes
>> >> >    causes crashes in the compile processes, which show the emacs abort
>> >> >    dialog (once for each async process). The dialogs disappear after a
>> >> >    short delay (presumably due to the parent emacs having exited).
>> >> 
>> >> Mmmh, I guess this is a Windows specific behavior.  Is there a specific
>> >> way to shut-down child processes we would use on Windows not to get
>> >> this error?
>> >
>> > Andrea, can you point me to the place where we interrupt async
>> > compilations when Emacs exits?  Is that just a normal delete-process,
>> > or do we do something else, like sending a signal?  Also, does the
>> > Emacs subprocess invoked to perform async compilation spawn further
>> > child processes, or is everything happening inside a single Emacs
>> > process?
>> 
>> Hi Eli,
>> 
>> we have no special handling for closing async compilation processes,
>> they should be closed as all child processes started by Emacs are.
>> 
>> In the child process we do not spawn directly any other process, but
>> libgccjit might do it (ex to call gas).
>
> OK, I've reviewed the code which kills subprocesses when Emacs is shut
> down, and I have some questions:
>
>   . How does libgccjit handle the case that its process is exiting?
>     Does it have any atexit handlers or static destructors?  IOW, how
>     does it ensure its own subprocesses, like gas etc. are terminated?

No precise idea about sorry.  Perhaps the best place to ask and discuss
that would be jit@gcc.gnu.org?

>   . When we invoke Emacs in a subprocess to do the async compilation,
>     do we specify that it should be killed without query?  I don't see
>     this in the code (did I miss it?), but if we don't, then exiting
>     Emacs will ask the user whether to kill the subprocesses -- does
>     it?

Yes it does, should we change this?

> Andy, if instead if exiting Emacs, you use signal-process, like this:
>
>   M-: (signal-process PROC-ID 'SIGHUP) RET
>
> (where PROC-ID is the process ID of the Emacs subprocess running the
> native compilation), do you see the same crash, or does the subprocess
> exit cleanly?  To see the PROC-ID, you can use the Task manager or the
> 'pslist' command from the PsTools suite.
>
> Thanks.
>
> P.S. Andrea, I see you use "path" in comp.el (and perhaps elsewhere)
> to mean "file name", but the GNU Coding Standards frown on using this
> for anything other than PATH-style directory lists.  So this should at
> some point be replaced with "file name".

da4da88c76 fix one case of this.

We use `paths' in `native-compile-async' and `native--compile-async' as
arg name.  This can be either a file, a list of file or a list of
directories.  What would be the suggested name for something like that?

Thanks

  Andrea

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: andrewjmoreton@gmail.com, 46494@debbugs.gnu.org
> Date: Sun, 21 Feb 2021 21:22:16 +0000
> 
> >   . How does libgccjit handle the case that its process is exiting?
> >     Does it have any atexit handlers or static destructors?  IOW, how
> >     does it ensure its own subprocesses, like gas etc. are terminated?
> 
> No precise idea about sorry.  Perhaps the best place to ask and discuss
> that would be jit@gcc.gnu.org?

If we need to, perhaps we should.

> >   . When we invoke Emacs in a subprocess to do the async compilation,
> >     do we specify that it should be killed without query?  I don't see
> >     this in the code (did I miss it?), but if we don't, then exiting
> >     Emacs will ask the user whether to kill the subprocesses -- does
> >     it?
> 
> Yes it does, should we change this?

no, I don't think so.  It's just that no one here mentioned that
question, so I assumed it isn't being asked.

> > P.S. Andrea, I see you use "path" in comp.el (and perhaps elsewhere)
> > to mean "file name", but the GNU Coding Standards frown on using this
> > for anything other than PATH-style directory lists.  So this should at
> > some point be replaced with "file name".
> 
> da4da88c76 fix one case of this.
> 
> We use `paths' in `native-compile-async' and `native--compile-async' as
> arg name.  This can be either a file, a list of file or a list of
> directories.  What would be the suggested name for something like that?

"files"?

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> We use `paths' in `native-compile-async' and `native--compile-async' as
>> arg name.  This can be either a file, a list of file or a list of
>> directories.  What would be the suggested name for something like that?
>
> "files"?

Well.. fair :D 81b1013555 does that.

Thanks

  Andrea

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Andy,

is there anything left to be done for this bug?

Thanks

  Andrea

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Eli Zaretskii]

> Cc: 46494@debbugs.gnu.org
> Date: Tue, 16 Mar 2021 16:53:21 +0000
> From:  Andrea Corallo via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Hi Andy,
> 
> is there anything left to be done for this bug?

I'm not Andy (so let's wait for him to speak up), but here's my take.

This bug had 3 parts:

  a) Bug #46256 describes problems with AOT compiled native-comp emacs not
     finding prebuilt .eln files when built for mingw64 64bit on Windows.

     As a result, emacs complains with an echo area warning for every .eln
     file that it cannot find in the expected location.

     The stream of frequent warnings that causes make emacs mostly
     unresponsive to user input.

  b) The "background" async compilation of .eln files is CPU intensive and
     somewhat slow. The default settings run a compile on every available
     core, which is unfriendly for other workloads running on the same
     machine.

     It would be helpful to users to have a command to show the state of
     the async background compilation, including the running compile
     processes and the queue of pending compilation requests.

  c) Quitting emacs when async compilation processes are running sometimes
     causes crashes in the compile processes, which show the emacs abort
     dialog (once for each async process). The dialogs disappear after a
     short delay (presumably due to the parent emacs having exited).

c) Has been solved.
b) doesn't seem to be a problem IME, we use half the cores, and
   there's a way to customize that number
a) I didn't see at all, so I think it's also solved.

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andy Moreton]

On Tue 16 Mar 2021, Eli Zaretskii wrote:

>> Cc: 46494@debbugs.gnu.org
>> Date: Tue, 16 Mar 2021 16:53:21 +0000
>> From:  Andrea Corallo via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> Hi Andy,
>> 
>> is there anything left to be done for this bug?
>
> I'm not Andy (so let's wait for him to speak up), but here's my take.
>
> This bug had 3 parts:
>
>   a) Bug #46256 describes problems with AOT compiled native-comp emacs not
>      finding prebuilt .eln files when built for mingw64 64bit on Windows.
>
>      As a result, emacs complains with an echo area warning for every .eln
>      file that it cannot find in the expected location.
>
>      The stream of frequent warnings that causes make emacs mostly
>      unresponsive to user input.
>
>   b) The "background" async compilation of .eln files is CPU intensive and
>      somewhat slow. The default settings run a compile on every available
>      core, which is unfriendly for other workloads running on the same
>      machine.
>
>      It would be helpful to users to have a command to show the state of
>      the async background compilation, including the running compile
>      processes and the queue of pending compilation requests.
>
>   c) Quitting emacs when async compilation processes are running sometimes
>      causes crashes in the compile processes, which show the emacs abort
>      dialog (once for each async process). The dialogs disappear after a
>      short delay (presumably due to the parent emacs having exited).
>
> c) Has been solved.
> b) doesn't seem to be a problem IME, we use half the cores, and
>    there's a way to customize that number
> a) I didn't see at all, so I think it's also solved.


Eli's take is a good summary of the issues.
(a) is fixed.
(b) is a wish-list item that will help ordinary users understand what
the compiler is doing.
(c) is fixed.

Given the epic length of the discussion in this bug, I think we should
probably close this bug report, and open fresh ones for any other
issues.

The native branch has received attention from a wider group of testers
over the last few weeks, and as a result is much more solid. Thanks to
all concerned for much hard work.

    AndyM

bug#46494: 28.0.50; [native-comp] Problems with async background compile

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> On Tue 16 Mar 2021, Eli Zaretskii wrote:
>
>>> Cc: 46494@debbugs.gnu.org
>>> Date: Tue, 16 Mar 2021 16:53:21 +0000
>>> From:  Andrea Corallo via "Bug reports for GNU Emacs,
>>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>>> 
>>> Hi Andy,
>>> 
>>> is there anything left to be done for this bug?
>>
>> I'm not Andy (so let's wait for him to speak up), but here's my take.
>>
>> This bug had 3 parts:
>>
>>   a) Bug #46256 describes problems with AOT compiled native-comp emacs not
>>      finding prebuilt .eln files when built for mingw64 64bit on Windows.
>>
>>      As a result, emacs complains with an echo area warning for every .eln
>>      file that it cannot find in the expected location.
>>
>>      The stream of frequent warnings that causes make emacs mostly
>>      unresponsive to user input.
>>
>>   b) The "background" async compilation of .eln files is CPU intensive and
>>      somewhat slow. The default settings run a compile on every available
>>      core, which is unfriendly for other workloads running on the same
>>      machine.
>>
>>      It would be helpful to users to have a command to show the state of
>>      the async background compilation, including the running compile
>>      processes and the queue of pending compilation requests.
>>
>>   c) Quitting emacs when async compilation processes are running sometimes
>>      causes crashes in the compile processes, which show the emacs abort
>>      dialog (once for each async process). The dialogs disappear after a
>>      short delay (presumably due to the parent emacs having exited).
>>
>> c) Has been solved.
>> b) doesn't seem to be a problem IME, we use half the cores, and
>>    there's a way to customize that number
>> a) I didn't see at all, so I think it's also solved.
>
>
> Eli's take is a good summary of the issues.
> (a) is fixed.
> (b) is a wish-list item that will help ordinary users understand what
> the compiler is doing.
> (c) is fixed.
>
> Given the epic length of the discussion in this bug, I think we should
> probably close this bug report, and open fresh ones for any other
> issues.
>
> The native branch has received attention from a wider group of testers
> over the last few weeks, and as a result is much more solid. Thanks to
> all concerned for much hard work.

Nice, I'm closing this then.

Thanks!

  Andrea