Suppressing native compilation (short and long term)

Suppressing native compilation (short and long term)

[Rob Browning]

Perhaps relevant in other contexts, but at least in the context of
Debian work, we'd like to be able to suppress all native compilation in
some contexts, or more specifically all writes to HOME (or really, to
avoid any implicit file manipulations[1]).

One key case is package builds and package installs (whether the emacs
packages themselves, or any of the various emacs add-on packages
(e.g. elpa-*)).

For package builds, HOME is typically set to /nonexistent (or similar),
and for package installs we don't want the install commands (preinst,
postinst, etc.), which are running as root, to write anything to /root/.

It's also been suggested by some users that they might like to have a
supported way to disable native compilation (even when it's the system
default), in order to avoid the unpredictable cpu and/or battery cost
sometimes.  Of course, assuming we eventually augment debian's current
emacs infrastructure to generate system-level .eln files during (add-on)
package installs, in addition to the .elc files it currently produces,
much of that concern for debian and its derivatives might recede.

In any case, I noticed that there is a no-native-compile variable, and
wondered if (in the short term), I might be able to craft a (possibly
debian-specific for now) way to disable native compilation across the
board.

Just to see, I changed no-native-compile to default to

  (getenv "DEBIAN_EMACS_DISABLE_NATIVE_COMPILATION")

instead of nil, and then set that in the environment, but the ebib
package tests (which set HOME=/nonexistent) still crashed (as had been
originally reported here https://bugs.debian.org/1020191).

The crash was in comp-trampoline-compile, and after a bit of
investigation I wondered if it might be possible to fix the problem (at
least approximately) by adding a no-native-compile clause to the
comp-subr-trampoline-install guard like this:

  (defun comp-subr-trampoline-install (subr-name)
    "Make SUBR-NAME effectively advice-able when called from native code."
    (unless (or no-native-compilation     ; this is new
                (null comp-enable-subr-trampolines)
                (memq subr-name native-comp-never-optimize-functions)
                (gethash subr-name comp-installed-trampolines-h))
      ...))

That did allow the package tests to run successfully.

So two questions:

  - As possibly just a short term, debian-specific fix, do those changes
    sound plausible, or are there other things we're missing (I might
    guess yes)?


  - Longer term, might it make sense to have some emacs-proper way to
    completely disable native compilation from the outside, either via
    environment variable, argument, or something else, for situations
    like this?

[1] For add-on packages (e.g. elpa-*, etc.), we want to explicitly build
    the local .elc files (at least) during the install, but we don't
    want to write to any other unexpected locations.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

> It's also been suggested by some users that they might like to have a
> supported way to disable native compilation (even when it's the system
> default), in order to avoid the unpredictable cpu and/or battery cost
> sometimes.

I think you can set native-comp-deferred-compilation-deny-list to
(".*") to switch this stuff off.  That's not the most logical interface,
so perhaps there should also be a `inhibit-native-compilation' variable
or the like?

Andrea probably has some comments; added to the CCs.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Date: Tue, 27 Sep 2022 20:04:24 -0500
> 
> 
> Perhaps relevant in other contexts, but at least in the context of
> Debian work, we'd like to be able to suppress all native compilation in
> some contexts, or more specifically all writes to HOME (or really, to
> avoid any implicit file manipulations[1]).
> 
> One key case is package builds and package installs (whether the emacs
> packages themselves, or any of the various emacs add-on packages
> (e.g. elpa-*)).
> 
> For package builds, HOME is typically set to /nonexistent (or similar),
> and for package installs we don't want the install commands (preinst,
> postinst, etc.), which are running as root, to write anything to /root/.
> 
> It's also been suggested by some users that they might like to have a
> supported way to disable native compilation (even when it's the system
> default), in order to avoid the unpredictable cpu and/or battery cost
> sometimes.  Of course, assuming we eventually augment debian's current
> emacs infrastructure to generate system-level .eln files during (add-on)
> package installs, in addition to the .elc files it currently produces,
> much of that concern for debian and its derivatives might recede.
> 
> In any case, I noticed that there is a no-native-compile variable, and
> wondered if (in the short term), I might be able to craft a (possibly
> debian-specific for now) way to disable native compilation across the
> board.
> 
> Just to see, I changed no-native-compile to default to
> 
>   (getenv "DEBIAN_EMACS_DISABLE_NATIVE_COMPILATION")
> 
> instead of nil, and then set that in the environment, but the ebib
> package tests (which set HOME=/nonexistent) still crashed (as had been
> originally reported here https://bugs.debian.org/1020191).
> 
> The crash was in comp-trampoline-compile, and after a bit of
> investigation I wondered if it might be possible to fix the problem (at
> least approximately) by adding a no-native-compile clause to the
> comp-subr-trampoline-install guard like this:
> 
>   (defun comp-subr-trampoline-install (subr-name)
>     "Make SUBR-NAME effectively advice-able when called from native code."
>     (unless (or no-native-compilation     ; this is new
>                 (null comp-enable-subr-trampolines)
>                 (memq subr-name native-comp-never-optimize-functions)
>                 (gethash subr-name comp-installed-trampolines-h))
>       ...))
> 
> That did allow the package tests to run successfully.
> 
> So two questions:
> 
>   - As possibly just a short term, debian-specific fix, do those changes
>     sound plausible, or are there other things we're missing (I might
>     guess yes)?
> 
> 
>   - Longer term, might it make sense to have some emacs-proper way to
>     completely disable native compilation from the outside, either via
>     environment variable, argument, or something else, for situations
>     like this?

I admit that I lost you somewhere near the beginning of your post.  I
think what's missing here is some more detailed explanation of the
problems you are trying to solve.  Without them, it is hard to give
you intelligent answers.  Here are some inline questions I'd like to
be answered:

> Perhaps relevant in other contexts, but at least in the context of
> Debian work, we'd like to be able to suppress all native compilation in
> some contexts, or more specifically all writes to HOME (or really, to
> avoid any implicit file manipulations[1]).

What do you mean by "writes to HOME"?  Native-compilation doesn't
write to HOME, it writes to directories in native-comp-eln-load-path.
So basically, you already have a way to redirect stuff to wherever you
want it.

> For package builds, HOME is typically set to /nonexistent (or similar),
> and for package installs we don't want the install commands (preinst,
> postinst, etc.), which are running as root, to write anything to /root/.

By "package installs" do you mean when end-users install a prebuilt
Emacs?  If so, how come that should ever need to write something
outside of the installation tree?  The Emacs "make install" procedure
doesn't, so why do your "package installs" somehow do?

Also, AFAIR installing an ELPA package doesn't produce any *.eln files
until the first time Emacs 'load's their *.elc files.  So why is this
a problem?

OTOH, installing a package from ELPA does write stuff to the user's
directories, so how is the behavior of native-compilation different
here?

> It's also been suggested by some users that they might like to have a
> supported way to disable native compilation (even when it's the system
> default), in order to avoid the unpredictable cpu and/or battery cost
> sometimes.

What do you mean by "disable native compilation" here, and what
exactly are the use cases for this?  Are we talking about users who
installed Emacs with some *.eln files, and from time to time load
*.elc files that have no corresponding *.eln files?  Or are we talking
about something else?  If the former, why do these users install Emacs
with native-compilation to begin with?

More generally, we never expected people who have Emacs with native
compilation available to want to disable it.  It made no sense to us
during development of Emacs 28, and frankly, it still doesn't, at
least to me.  It is so easy to install Emacs without these
capabilities and never look back that we thought providing a
build-time option should be enough.  (Well, except for MS-Windows
users, where more is needed, but that's not your worry.)

Therefore, if we need to discuss introduction of run-time features for
disabling native-compilation, we need to have a broader view of the
relevant use cases and user needs.  This includes at least the
following aspects that need to be discussed and clarified:

  . What does "disable native-compilation" mean, exactly?  There are
    several possible interpretations of that.  Do people want to
    prevent Emacs from looking for and loading the *.eln files, and
    use the *.elc files instead?  Or do people only want to prevent
    Emacs from compiling new *.eln files?  Or do people want that
    *.eln files be produced only by an explicit user command, not
    transparently and asynchronously?  Or something else?

  . When and why would people want any of the above to be possible?

  . How and why would downstream distros want to support such
    capabilities?

Armed with this information, we could then see which use cases are
something we'd like to support in the core and how.

Thanks.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Hi Rob,

Rob Browning <rlb@defaultvalue.org> writes:

> Perhaps relevant in other contexts, but at least in the context of
> Debian work, we'd like to be able to suppress all native compilation in
> some contexts, or more specifically all writes to HOME (or really, to
> avoid any implicit file manipulations[1]).
>
> One key case is package builds and package installs (whether the emacs
> packages themselves, or any of the various emacs add-on packages
> (e.g. elpa-*)).
>
> For package builds, HOME is typically set to /nonexistent (or similar),
> and for package installs we don't want the install commands (preinst,
> postinst, etc.), which are running as root, to write anything to /root/.
>
> It's also been suggested by some users that they might like to have a
> supported way to disable native compilation (even when it's the system
> default), in order to avoid the unpredictable cpu and/or battery cost
> sometimes.  Of course, assuming we eventually augment debian's current
> emacs infrastructure to generate system-level .eln files during (add-on)
> package installs, in addition to the .elc files it currently produces,
> much of that concern for debian and its derivatives might recede.

This is what `native-comp-deferred-compilation' does.  Well except for
trampolines but this is extremly light as cpu/energy cost.

> In any case, I noticed that there is a no-native-compile variable, and
> wondered if (in the short term), I might be able to craft a (possibly
> debian-specific for now) way to disable native compilation across the
> board.
>
> Just to see, I changed no-native-compile to default to
>
>   (getenv "DEBIAN_EMACS_DISABLE_NATIVE_COMPILATION")
>
> instead of nil, and then set that in the environment, but the ebib
> package tests (which set HOME=/nonexistent) still crashed (as had been
> originally reported here https://bugs.debian.org/1020191).
>
> The crash was in comp-trampoline-compile, and after a bit of
> investigation I wondered if it might be possible to fix the problem (at
> least approximately) by adding a no-native-compile clause to the
> comp-subr-trampoline-install guard like this:
>
>   (defun comp-subr-trampoline-install (subr-name)
>     "Make SUBR-NAME effectively advice-able when called from native code."
>     (unless (or no-native-compilation     ; this is new
>                 (null comp-enable-subr-trampolines)
>                 (memq subr-name native-comp-never-optimize-functions)
>                 (gethash subr-name comp-installed-trampolines-h))
>       ...))

A native compiled Emacs needs to compile a trampoline each time a
primitive is redefined or advised.  If we disable this mechanism ATM we
cannot guarantee Emacs to function correctly.

> That did allow the package tests to run successfully.
>
> So two questions:
>
>   - As possibly just a short term, debian-specific fix, do those changes
>     sound plausible, or are there other things we're missing (I might
>     guess yes)?

IMO the suggested change is not okay for the reason mentioned above
sorry :/

>
>   - Longer term, might it make sense to have some emacs-proper way to
>     completely disable native compilation from the outside, either via
>     environment variable, argument, or something else, for situations
>     like this?
>
> [1] For add-on packages (e.g. elpa-*, etc.), we want to explicitly build
>     the local .elc files (at least) during the install, but we don't
>     want to write to any other unexpected locations.
>
> Thanks

IIUC the issue is when Emacs is run as root (I didn't know Debian does
that for installing packages!).  The workaround I'd suggest is to target
a temporary HOME and clean it afterwards with its eln-cache (if this is
viable).

Best Regards

  Andrea

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 28 Sep 2022 12:52:59 +0000
> 
> This is what `native-comp-deferred-compilation' does.  Well except for
> trampolines but this is extremly light as cpu/energy cost.

I think if we want to allow people to turn off async compilation, we
should provide a way to force native-comp-available-p to return nil.
When that function returns nil, Emacs already does react correctly (or
at least it's supposed to).

But I still would like to understand what kind of use cases are we
talking about here, because without that it's impossible to decide
whether a given measure will provide a reasonable solution in the long
run.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: emacs-devel@gnu.org
>> Date: Wed, 28 Sep 2022 12:52:59 +0000
>> 
>> This is what `native-comp-deferred-compilation' does.  Well except for
>> trampolines but this is extremly light as cpu/energy cost.
>
> I think if we want to allow people to turn off async compilation, we
> should provide a way to force native-comp-available-p to return nil.
> When that function returns nil, Emacs already does react correctly (or
> at least it's supposed to).

Yes `native-comp-deferred-compilation' AFAIK does exactly this already,
`native-comp-available-p' is to check if the native compiler is
available (not necessarily the deferred/async mechanism).

Best Regards

  Andera

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: rlb@defaultvalue.org, emacs-devel@gnu.org
> Date: Wed, 28 Sep 2022 18:49:43 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andrea Corallo <akrl@sdf.org>
> >> Cc: emacs-devel@gnu.org
> >> Date: Wed, 28 Sep 2022 12:52:59 +0000
> >> 
> >> This is what `native-comp-deferred-compilation' does.  Well except for
> >> trampolines but this is extremly light as cpu/energy cost.
> >
> > I think if we want to allow people to turn off async compilation, we
> > should provide a way to force native-comp-available-p to return nil.
> > When that function returns nil, Emacs already does react correctly (or
> > at least it's supposed to).
> 
> Yes `native-comp-deferred-compilation' AFAIK does exactly this already,
> `native-comp-available-p' is to check if the native compiler is
> available (not necessarily the deferred/async mechanism).

But then it should disable the trampolines as well, see startup.el.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: rlb@defaultvalue.org, emacs-devel@gnu.org
>> Date: Wed, 28 Sep 2022 18:49:43 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Andrea Corallo <akrl@sdf.org>
>> >> Cc: emacs-devel@gnu.org
>> >> Date: Wed, 28 Sep 2022 12:52:59 +0000
>> >> 
>> >> This is what `native-comp-deferred-compilation' does.  Well except for
>> >> trampolines but this is extremly light as cpu/energy cost.
>> >
>> > I think if we want to allow people to turn off async compilation, we
>> > should provide a way to force native-comp-available-p to return nil.
>> > When that function returns nil, Emacs already does react correctly (or
>> > at least it's supposed to).
>> 
>> Yes `native-comp-deferred-compilation' AFAIK does exactly this already,
>> `native-comp-available-p' is to check if the native compiler is
>> available (not necessarily the deferred/async mechanism).
>
> But then it should disable the trampolines as well, see startup.el.

Not in my opinion, trampolines are not deferred async compilation.

Also as mentioned ATM is not possible to disable trampolines and have a
fully working native comp Emacs (if we assume primitives can be
redefined).

Best Regards

  Andrea

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: rlb@defaultvalue.org, emacs-devel@gnu.org
> Date: Wed, 28 Sep 2022 21:32:13 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Yes `native-comp-deferred-compilation' AFAIK does exactly this already,
> >> `native-comp-available-p' is to check if the native compiler is
> >> available (not necessarily the deferred/async mechanism).
> >
> > But then it should disable the trampolines as well, see startup.el.
> 
> Not in my opinion, trampolines are not deferred async compilation.
> 
> Also as mentioned ATM is not possible to disable trampolines and have a
> fully working native comp Emacs (if we assume primitives can be
> redefined).

I'm confused.  I alluded to this part of startup.el:

    (when (featurep 'native-compile)
      (unless (native-comp-available-p)
        ;; Disable deferred async compilation and trampoline synthesis
        ;; in this session.  This is necessary if libgccjit is not
        ;; available on MS-Windows, but Emacs was built with
        ;; native-compilation support.
        (setq native-comp-deferred-compilation nil
              comp-enable-subr-trampolines nil))

The last part disables trampolines, AFAIU.  So what am I missing here?

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: rlb@defaultvalue.org, emacs-devel@gnu.org
>> Date: Wed, 28 Sep 2022 21:32:13 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> Yes `native-comp-deferred-compilation' AFAIK does exactly this already,
>> >> `native-comp-available-p' is to check if the native compiler is
>> >> available (not necessarily the deferred/async mechanism).
>> >
>> > But then it should disable the trampolines as well, see startup.el.
>> 
>> Not in my opinion, trampolines are not deferred async compilation.
>> 
>> Also as mentioned ATM is not possible to disable trampolines and have a
>> fully working native comp Emacs (if we assume primitives can be
>> redefined).
>
> I'm confused.  I alluded to this part of startup.el:
>
>     (when (featurep 'native-compile)
>       (unless (native-comp-available-p)
>         ;; Disable deferred async compilation and trampoline synthesis
>         ;; in this session.  This is necessary if libgccjit is not
>         ;; available on MS-Windows, but Emacs was built with
>         ;; native-compilation support.
>         (setq native-comp-deferred-compilation nil
>               comp-enable-subr-trampolines nil))
>
> The last part disables trampolines, AFAIU.  So what am I missing here?

Hi Eli,

yes it does, what do you find confusing about this?

This is how I see things, we have two compilation mechanisms:

1- Syncronous native compilation.  This can be triggered:
   1.1- By the user (calling `native-compile').
   1.2- By Emacs in the need of generating a trampoline.

2- Async/deferred/jit (or how we wanna call it), this is triggered
   automatically when loafing a .elc with no corresponding .eln.

We have two customize to control the two automatic mechanisms:
`comp-enable-subr-trampolines' gates 1.2,
`native-comp-deferred-compilation' gates 2.

Indeed `native-comp-available-p' gates all native compilations.

Best Regards

  Andrea

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: rlb@defaultvalue.org, emacs-devel@gnu.org
> Date: Thu, 29 Sep 2022 08:18:08 +0000
> 
> >> >> Yes `native-comp-deferred-compilation' AFAIK does exactly this already,
> >> >> `native-comp-available-p' is to check if the native compiler is
> >> >> available (not necessarily the deferred/async mechanism).
> >> >
> >> > But then it should disable the trampolines as well, see startup.el.
> >> 
> >> Not in my opinion, trampolines are not deferred async compilation.
> >> 
> >> Also as mentioned ATM is not possible to disable trampolines and have a
> >> fully working native comp Emacs (if we assume primitives can be
> >> redefined).
> >
> > I'm confused.  I alluded to this part of startup.el:
> >
> >     (when (featurep 'native-compile)
> >       (unless (native-comp-available-p)
> >         ;; Disable deferred async compilation and trampoline synthesis
> >         ;; in this session.  This is necessary if libgccjit is not
> >         ;; available on MS-Windows, but Emacs was built with
> >         ;; native-compilation support.
> >         (setq native-comp-deferred-compilation nil
> >               comp-enable-subr-trampolines nil))
> >
> > The last part disables trampolines, AFAIU.  So what am I missing here?
> 
> Hi Eli,
> 
> yes it does, what do you find confusing about this?
> 
> This is how I see things, we have two compilation mechanisms:
> 
> 1- Syncronous native compilation.  This can be triggered:
>    1.1- By the user (calling `native-compile').
>    1.2- By Emacs in the need of generating a trampoline.
> 
> 2- Async/deferred/jit (or how we wanna call it), this is triggered
>    automatically when loafing a .elc with no corresponding .eln.
> 
> We have two customize to control the two automatic mechanisms:
> `comp-enable-subr-trampolines' gates 1.2,
> `native-comp-deferred-compilation' gates 2.
> 
> Indeed `native-comp-available-p' gates all native compilations.

So I was asking whether when the user wants to disable native
compilations, we should disable both 2 and 1.2.  We already do that
when Emacs detects at startup that it doesn't have access to
libgccjit, so why not do the same when the user wants to disable
native compilations for whatever reasons?  You seem to be saying that
1.2 cannot be disabled in that case?  But if so, how come we do
disable it in startup.el?  That is the root of my confusion.

Thanks.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: rlb@defaultvalue.org, emacs-devel@gnu.org
>> Date: Thu, 29 Sep 2022 08:18:08 +0000
>> 
>> >> >> Yes `native-comp-deferred-compilation' AFAIK does exactly this already,
>> >> >> `native-comp-available-p' is to check if the native compiler is
>> >> >> available (not necessarily the deferred/async mechanism).
>> >> >
>> >> > But then it should disable the trampolines as well, see startup.el.
>> >> 
>> >> Not in my opinion, trampolines are not deferred async compilation.
>> >> 
>> >> Also as mentioned ATM is not possible to disable trampolines and have a
>> >> fully working native comp Emacs (if we assume primitives can be
>> >> redefined).
>> >
>> > I'm confused.  I alluded to this part of startup.el:
>> >
>> >     (when (featurep 'native-compile)
>> >       (unless (native-comp-available-p)
>> >         ;; Disable deferred async compilation and trampoline synthesis
>> >         ;; in this session.  This is necessary if libgccjit is not
>> >         ;; available on MS-Windows, but Emacs was built with
>> >         ;; native-compilation support.
>> >         (setq native-comp-deferred-compilation nil
>> >               comp-enable-subr-trampolines nil))
>> >
>> > The last part disables trampolines, AFAIU.  So what am I missing here?
>> 
>> Hi Eli,
>> 
>> yes it does, what do you find confusing about this?
>> 
>> This is how I see things, we have two compilation mechanisms:
>> 
>> 1- Syncronous native compilation.  This can be triggered:
>>    1.1- By the user (calling `native-compile').
>>    1.2- By Emacs in the need of generating a trampoline.
>> 
>> 2- Async/deferred/jit (or how we wanna call it), this is triggered
>>    automatically when loafing a .elc with no corresponding .eln.
>> 
>> We have two customize to control the two automatic mechanisms:
>> `comp-enable-subr-trampolines' gates 1.2,
>> `native-comp-deferred-compilation' gates 2.
>> 
>> Indeed `native-comp-available-p' gates all native compilations.
>
> So I was asking whether when the user wants to disable native
> compilations, we should disable both 2 and 1.2.  We already do that
> when Emacs detects at startup that it doesn't have access to
> libgccjit, so why not do the same when the user wants to disable
> native compilations for whatever reasons?

Ah I see.  We could allow for disabling completely native compilation,
but that's something different from what those two customizes are for.

That said I think that given the very low cost of building trampolines,
and the deep impact that their absence would have in the Emacs
machinery, we should acctually discourage the user to do so unless some
very specific reason is present (I can't think of any ATM).

  Andrea

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: rlb@defaultvalue.org, emacs-devel@gnu.org
> Date: Thu, 29 Sep 2022 14:32:07 +0000
> 
> That said I think that given the very low cost of building trampolines,
> and the deep impact that their absence would have in the Emacs
> machinery, we should acctually discourage the user to do so unless some
> very specific reason is present (I can't think of any ATM).

The reason stated by Rob was that they want to prevent "writing to
user's HOME directory".  I don't think I understand the rationale well
enough, and I asked some questions about it.  I hope Rob will answer,
and then we could continue discussing what is reasonable for that use
case.

Re: Suppressing native compilation (short and long term)

[David Bremner]

> The reason stated by Rob was that they want to prevent "writing to
> user's HOME directory".  I don't think I understand the rationale well
> enough, and I asked some questions about it.  I hope Rob will answer,
> and then we could continue discussing what is reasonable for that use
> case.

For package builds, Debian has the following policy 

    https://www.debian.org/doc/debian-policy/ch-source.html#main-building-script-debian-rules

in particular

    Required targets must not attempt to write outside of the unpacked
    source package tree. There are two exceptions. Firstly, the binary
    targets may write the binary packages to the parent directory of the
    unpacked source package tree. Secondly, required targets may write
    to /tmp, /var/tmp and to the directory specified by the TMPDIR
    environment variable, but must not depend on the contents of any of
    these.

    This restriction is intended to prevent source package builds
    creating and depending on state outside of themselves, thus
    affecting multiple independent rebuilds. In particular, the required
    targets must not attempt to write into HOME.

Some additional byte compilation happens at package install time. In my
view the same restrictions should apply there. In addition to the
unintentional sharing of state mentioned in policy, there is also the
issue of cleaning up after a package on uninstall. This is manageable
now because any generated .elc files go in a package specific directory,
which can just be removed on purging the package.

I think just turning off native compilation when HOME is not writeable
would be sensible from a Debian point of view. Emacs did not previously
require a writable HOME (although maybe most users never tested
this). It would be unfortunate if this changed for Emacs with native
compilation support.

David

PS. Please CC me on any replies (that you want me to read). I'm not
subscribed to the list.

PPS. This reply is synthesized from "reply via email" on the archive
     page. Apologies in advance for any list etiquette failures.
     

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: David Bremner <david@tethera.net>
> Cc: emacs-devel@gnu.org, akrl@sdf.org, rlb@defaultvalue.org
> Date: Fri, 30 Sep 2022 10:13:56 -0300
> 
> 
> > The reason stated by Rob was that they want to prevent "writing to
> > user's HOME directory".  I don't think I understand the rationale well
> > enough, and I asked some questions about it.  I hope Rob will answer,
> > and then we could continue discussing what is reasonable for that use
> > case.
> 
> For package builds, Debian has the following policy 
> 
>     https://www.debian.org/doc/debian-policy/ch-source.html#main-building-script-debian-rules
> 
> in particular
> 
>     Required targets must not attempt to write outside of the unpacked
>     source package tree. There are two exceptions. Firstly, the binary
>     targets may write the binary packages to the parent directory of the
>     unpacked source package tree. Secondly, required targets may write
>     to /tmp, /var/tmp and to the directory specified by the TMPDIR
>     environment variable, but must not depend on the contents of any of
>     these.
> 
>     This restriction is intended to prevent source package builds
>     creating and depending on state outside of themselves, thus
>     affecting multiple independent rebuilds. In particular, the required
>     targets must not attempt to write into HOME.
> 
> Some additional byte compilation happens at package install time.

This is what I'm asking about: what exactly triggers the compilation?
Just installing a package shouldn't do that, only loading it into
Emacs should.

The other part of what I asked is that if for some reason installation
does need to compile files (something that I still need to understand
why it happens), there's nothing that hard-codes HOME in the directory
list used for that.  You can set native-comp-eln-load-path to anything
you want, and only the directories in that list will be used.

> In my
> view the same restrictions should apply there. In addition to the
> unintentional sharing of state mentioned in policy, there is also the
> issue of cleaning up after a package on uninstall. This is manageable
> now because any generated .elc files go in a package specific directory,
> which can just be removed on purging the package.

See above: you can do the same with *.eln files, if, for some reason,
they are produced by the installation.

> I think just turning off native compilation when HOME is not writeable
> would be sensible from a Debian point of view. Emacs did not previously
> require a writable HOME (although maybe most users never tested
> this). It would be unfortunate if this changed for Emacs with native
> compilation support.

Emacs doesn't require a writable HOME, it requires a writable
directory to store *.eln files.  It doesn't have to be HOME.  And once
again, I still don't understand why *.eln files are produced at
installation time in the first place.

Re: Suppressing native compilation (short and long term)

[David Bremner]

Eli Zaretskii <eliz@gnu.org> writes:

>
> This is what I'm asking about: what exactly triggers the compilation?
> Just installing a package shouldn't do that, only loading it into
> Emacs should.

When I talk about "package installation" I mean installing a Debian
binary package. This is a more general notion than a package as defined
by package.el

We have one copy of the .elc files for all users. Because of this, and
the cleanup issue I mentioned above, the elc files need to generated
either at (Debian) package build time, or at (Debian) package
installation time.

> The other part of what I asked is that if for some reason installation
> does need to compile files (something that I still need to understand
> why it happens), there's nothing that hard-codes HOME in the directory
> list used for that.  You can set native-comp-eln-load-path to anything
> you want, and only the directories in that list will be used.

Does that restrict where eln files are written? Or just where emacs
tries to read?

> Emacs doesn't require a writable HOME, it requires a writable
> directory to store *.eln files.  It doesn't have to be HOME.

Fair enough. I tried setting native-compile-target-directory, but that
seemed to be ignored by the trampoline stuff. Maybe both variables need
to be set.

> And once again, I still don't understand why *.eln files are produced
> at installation time in the first place.

The short version is that we need to run emacs at Debian package build
and install time. Byte-compilation is one reason. Running tests at build
time is another.

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Fri 30 Sep 2022 at 04:56PM +03, Eli Zaretskii wrote:

> This is what I'm asking about: what exactly triggers the compilation?
> Just installing a package shouldn't do that, only loading it into
> Emacs should.

We currently bytecompile during package installation, and would like
also to do the native compilation at that time.

The main reason for bytecompiling at package installation is that we
have logic to purge and recompile anything whose dependency versions
have changed, which means that all bytecode installed on Debian systems
was generated using known-correct macro definitions.  Historically
ordinary Emacs processes running in non-batch mode have not guaranteed
this; I recall that the Org-mode developers discouraged use of ELPA to
install newer versions of Org because of these sorts of issues.

A secondary reason is that it makes sense to compile once, system-wide,
rather than repeatedly in each user's home directory.  It is also nice
that you can know everything is already in place once your package is
installed, so that I can unplug my laptop once the package manager has
exited, and I know that it isn't going to do any CPU-intensive
compilation and drain my battery.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

>> Some additional byte compilation happens at package install time.

[ BTW, In Rob's case, we're talking about installation via Debian's
  `dpkg`, i.e. system-wide installation of ELisp packages, which also
  causes the .el files to be byte-compiled for&by the currently installed
  Emacs binary.  ]

> This is what I'm asking about: what exactly triggers the compilation?
> Just installing a package shouldn't do that, only loading it into
> Emacs should.

Byte-compilation does load files (not the one being compiled, but many
others, starting with `bytecomp` of course), so it can trigger
native-compilation of some files (including some of the files being
byte-compiled, if they `require` each other, which is very common).


        Stefan

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

David Bremner [2022-09-30 10:13:56] wrote:
> I think just turning off native compilation when HOME is not writeable
> would be sensible from a Debian point of view. Emacs did not previously

That's not the problem: the problem is when $HOME is writable but doesn't
belong to the user (e.g. because the user is root): then Emacs ends up
writing files (and creating directories) which then end up being
"unwritable" by the owner of $HOME.

IIUC this happens very "naturally" with `su` followed by running Emacs,
because many version of `su` (contrary to `su -`) preserve the $LOGNAME
and Emacs uses ~$LOGNAME as "the HOME" (instead of $HOME) when run
as root.


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: David Bremner <david@tethera.net>
> Cc: emacs-devel@gnu.org, akrl@sdf.org, rlb@defaultvalue.org
> Date: Fri, 30 Sep 2022 11:33:29 -0300
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >
> > This is what I'm asking about: what exactly triggers the compilation?
> > Just installing a package shouldn't do that, only loading it into
> > Emacs should.
> 
> When I talk about "package installation" I mean installing a Debian
> binary package. This is a more general notion than a package as defined
> by package.el
> 
> We have one copy of the .elc files for all users. Because of this, and
> the cleanup issue I mentioned above, the elc files need to generated
> either at (Debian) package build time, or at (Debian) package
> installation time.

So I understand that this issue is caused by the Debian installation
process?  If so, the installation process should make sure the *.eln
files are written only where you want them to be written.

> > The other part of what I asked is that if for some reason installation
> > does need to compile files (something that I still need to understand
> > why it happens), there's nothing that hard-codes HOME in the directory
> > list used for that.  You can set native-comp-eln-load-path to anything
> > you want, and only the directories in that list will be used.
> 
> Does that restrict where eln files are written? Or just where emacs
> tries to read?

Both.

> > Emacs doesn't require a writable HOME, it requires a writable
> > directory to store *.eln files.  It doesn't have to be HOME.
> 
> Fair enough. I tried setting native-compile-target-directory, but that
> seemed to be ignored by the trampoline stuff. Maybe both variables need
> to be set.

That's the wrong variable.  Please use native-comp-eln-load-path.

> > And once again, I still don't understand why *.eln files are produced
> > at installation time in the first place.
> 
> The short version is that we need to run emacs at Debian package build
> and install time. Byte-compilation is one reason. Running tests at build
> time is another.

OK, in that case you get to arrange for the environment which will not
produce files where you don't want them.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: David Bremner <david@tethera.net>,  emacs-devel@gnu.org,  akrl@sdf.org,
>   rlb@defaultvalue.org
> Date: Fri, 30 Sep 2022 07:55:08 -0700
> 
> We currently bytecompile during package installation, and would like
> also to do the native compilation at that time.
> 
> The main reason for bytecompiling at package installation is that we
> have logic to purge and recompile anything whose dependency versions
> have changed, which means that all bytecode installed on Debian systems
> was generated using known-correct macro definitions.  Historically
> ordinary Emacs processes running in non-batch mode have not guaranteed
> this; I recall that the Org-mode developers discouraged use of ELPA to
> install newer versions of Org because of these sorts of issues.

This problem doesn't exist with *.eln files, because Emacs will
regenerate new .eln files with different hashes in its name when the
source file or the Emacs binary changes.  So you do not have to
trigger native compilation for these reasons.

> A secondary reason is that it makes sense to compile once, system-wide,
> rather than repeatedly in each user's home directory.

I think this is misguided.  If the user never loads a package, there's
no reason to produce the *.eln files for it.  At best it litters the
disk with useless files.  At worst, compilation could cause warnings
or even errors, which are just an annoyance when the package is not
going to be used for a long time, or not at all.

These considerations are the reason why *.eln files are compiled JIT
and on-demand.  It is not a coincidence or lack of thought on our
part, it is the result of considering many factors.

I think you are trying to deal with *.eln files and with
native-compilation in general as if it were a slightly different kind
of byte-compilation.  That is a mistake: it's a qualitatively
different feature with different traits and different user-facing
and administrator-facing aspects.

> It is also nice that you can know everything is already in place
> once your package is installed, so that I can unplug my laptop once
> the package manager has exited, and I know that it isn't going to do
> any CPU-intensive compilation and drain my battery.

The *.eln files should not be part of "everything is in place",
because they will be produced when needed.  No user involvement or
interaction is necessary.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: David Bremner <david@tethera.net>,  emacs-devel@gnu.org,  akrl@sdf.org,
>   rlb@defaultvalue.org
> Date: Fri, 30 Sep 2022 11:32:57 -0400
> 
> Byte-compilation does load files (not the one being compiled, but many
> others, starting with `bytecomp` of course), so it can trigger
> native-compilation of some files (including some of the files being
> byte-compiled, if they `require` each other, which is very common).

The solution for this that I'd suggest is to precompile them, so that
the package comes with those files already as *.eln.  That's what we
did in Emacs 28.1 for -nw sessions, which always load the terminal
support file from lisp/term/.

Re: Suppressing native compilation (short and long term)

[David Bremner]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> David Bremner [2022-09-30 10:13:56] wrote:
>> I think just turning off native compilation when HOME is not writeable
>> would be sensible from a Debian point of view. Emacs did not previously
>
> That's not the problem: the problem is when $HOME is writable but doesn't
> belong to the user (e.g. because the user is root): then Emacs ends up
> writing files (and creating directories) which then end up being
> "unwritable" by the owner of $HOME.

That is _also_ a problem, but not the one causing many Debian packages
to fail to build from source.

Re: Suppressing native compilation (short and long term)

[David Bremner]

David Bremner <david@tethera.net> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> David Bremner [2022-09-30 10:13:56] wrote:
>>> I think just turning off native compilation when HOME is not writeable
>>> would be sensible from a Debian point of view. Emacs did not previously
>>
>> That's not the problem: the problem is when $HOME is writable but doesn't
>> belong to the user (e.g. because the user is root): then Emacs ends up
>> writing files (and creating directories) which then end up being
>> "unwritable" by the owner of $HOME.
>
> That is _also_ a problem, but not the one causing many Debian packages
> to fail to build from source.

Sorry, that came off snappier than intended. You are of course correct
that the issue you report is a real problem for users, I've just been up
to my elbows for the last week or so dealing with the fallout of the
problem I described.

d

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> The solution for this that I'd suggest is to precompile them, so that
> the package comes with those files already as *.eln.  That's what we
> did in Emacs 28.1 for -nw sessions, which always load the terminal
> support file from lisp/term/.

I've finally had a chance to catch up, and this thread to date appears
to cover a good bit of the relevant ground.  As requested, I'll try to
add some additional context that hopefully better explains how and why
we arrived at the current situation, asking the questions we're asking.

Long ago, we (speaking from the Debian side) came up with the Debian
Emacs policy.  This was both a policy, and the infrastructure to
implement some of it (in the form of the emacsen-common package, and
now, additionally, the dh-elpa infrastructure).

At the time, XEmacs was still active, and Emacs itself was about to
transition from 19 to 20, and my recollection is that there was a
nontrivial amount of backward incompatibility with the latter, or at
least I recall hearing from people who really didn't want to be forced
to upgrade immediately.

I'm not completely familiar with those concerns because this was just
about the time I started working on Debian's Emacs support.  In any
case, one of the key goals of that support was to allow Debian to
package, and people to install, one or more major versions of Emacs,
and/or XEmacs simultaneously.  As a result, we added support for
"emacsen flavors": emacs20, emacs21, emacs22, xemacs19, etc.  (Worth
noting that a few years back we did "unversion" the Emacs packages, so
that we no longer have emacs27, emacs28, etc., just emacs.)

We also wanted to support the creation of "add-on" packages like org,
newer versions of gnus, etc., and we wanted each of those packages to be
able to work with as many flavors (Emacs or XEmacs) as the add-on
package maintainer decided was feasible.

We wanted to have only one system-level .elc file for each flavor for
each .el file.  We did not want to have to build and maintain separate
packages for each flavor in the archive, i.e. org-emacs21, org-emacs22,
etc., each with a full set of the relevant .elc files.  (I'm also not
sure that would have worked (easily), given the versioning and
dependency concerns described next.)

We didn't want to risk breakage from backward-incompatible changes to
macros in dependencies, i.e. if the magit package depended on the foo
(library) package, we assumed we'd need to rebuild (during package
upgrade) all the majit .elc files whenever the foo package changed
because foo might have changed a macro backward incompatibly.

Since we didn't have any (reasonable?) way to detect the inter-package
dependency graph, we just insisted that those be correctly described by
the Debian package dependencies.  Then at install time, we use that
graph to determine what packages to "rebuild" (mostly just the .elc
files), and the order in which to rebuild them.

We also decided to rebuild *all* the add-on packages (.elc files) for a
given flavor whenever the version of the flavor's package changes
(e.g. when there's a new Emacs or XEmacs version) because we didn't know
whether or not any given release might make that necessary/desirable.

Uninstalling an add-on, of course, cleans up all the relevant bits
(usually, mostly just the automatically generated .elc files).

So when you "apt install elpa-magit", Debian builds all the .elc files
and puts them in the right place in /usr for every flavor you have
installed.  When you "apt install emacs" and it's a new install, Debian
builds all of the .elc files for every installed add-on package in
dependency order for that flavor.  The same thing happens during an
emacs package upgrade.

Note that we were also thinking of the case where you have a large
machine with hundreds of users (as was more often the case when we
started).  There we didn't want to have N copies of the same file for N
users (whether .elc or now .eln).

Regarding the writes to HOME, etc.  I think David covered that fairly
well -- Debian runs emacs to "do things" (build .elcs, run tests, etc.)
at package build time, package install time, and during package testing.
In all of these cases, we're likely to want to avoid side-effects outside
the build/test dir like writing .elc or .eln files to the current user's
HOME, whether that's /root/ or /home/*.  It sounds like we may be able
to accomplish that by redirecting everything to a temp dir, which is
likely fine.

Hope this helps.
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: david@tethera.net, emacs-devel@gnu.org, akrl@sdf.org
> Date: Sat, 01 Oct 2022 11:38:54 -0500
> 
> I've finally had a chance to catch up, and this thread to date appears
> to cover a good bit of the relevant ground.  As requested, I'll try to
> add some additional context that hopefully better explains how and why
> we arrived at the current situation, asking the questions we're asking.

Thanks.

> Long ago, we (speaking from the Debian side) came up with the Debian
> Emacs policy.  This was both a policy, and the infrastructure to
> implement some of it (in the form of the emacsen-common package, and
> now, additionally, the dh-elpa infrastructure).
> 
> At the time, XEmacs was still active, and Emacs itself was about to
> transition from 19 to 20, and my recollection is that there was a
> nontrivial amount of backward incompatibility with the latter, or at
> least I recall hearing from people who really didn't want to be forced
> to upgrade immediately.
> 
> I'm not completely familiar with those concerns because this was just
> about the time I started working on Debian's Emacs support.  In any
> case, one of the key goals of that support was to allow Debian to
> package, and people to install, one or more major versions of Emacs,
> and/or XEmacs simultaneously.  As a result, we added support for
> "emacsen flavors": emacs20, emacs21, emacs22, xemacs19, etc.  (Worth
> noting that a few years back we did "unversion" the Emacs packages, so
> that we no longer have emacs27, emacs28, etc., just emacs.)
> 
> We also wanted to support the creation of "add-on" packages like org,
> newer versions of gnus, etc., and we wanted each of those packages to be
> able to work with as many flavors (Emacs or XEmacs) as the add-on
> package maintainer decided was feasible.
> 
> We wanted to have only one system-level .elc file for each flavor for
> each .el file.  We did not want to have to build and maintain separate
> packages for each flavor in the archive, i.e. org-emacs21, org-emacs22,
> etc., each with a full set of the relevant .elc files.  (I'm also not
> sure that would have worked (easily), given the versioning and
> dependency concerns described next.)
> 
> We didn't want to risk breakage from backward-incompatible changes to
> macros in dependencies, i.e. if the magit package depended on the foo
> (library) package, we assumed we'd need to rebuild (during package
> upgrade) all the majit .elc files whenever the foo package changed
> because foo might have changed a macro backward incompatibly.
> 
> Since we didn't have any (reasonable?) way to detect the inter-package
> dependency graph, we just insisted that those be correctly described by
> the Debian package dependencies.  Then at install time, we use that
> graph to determine what packages to "rebuild" (mostly just the .elc
> files), and the order in which to rebuild them.
> 
> We also decided to rebuild *all* the add-on packages (.elc files) for a
> given flavor whenever the version of the flavor's package changes
> (e.g. when there's a new Emacs or XEmacs version) because we didn't know
> whether or not any given release might make that necessary/desirable.
> 
> Uninstalling an add-on, of course, cleans up all the relevant bits
> (usually, mostly just the automatically generated .elc files).
> 
> So when you "apt install elpa-magit", Debian builds all the .elc files
> and puts them in the right place in /usr for every flavor you have
> installed.  When you "apt install emacs" and it's a new install, Debian
> builds all of the .elc files for every installed add-on package in
> dependency order for that flavor.  The same thing happens during an
> emacs package upgrade.
> 
> Note that we were also thinking of the case where you have a large
> machine with hundreds of users (as was more often the case when we
> started).  There we didn't want to have N copies of the same file for N
> users (whether .elc or now .eln).

Thanks for the background.

You should know that the problems you needed to address with the *.elc
files don't exist with the *.eln files.  The *.eln files have version
information of the Emacs to which they "belong" hard-coded into their
names.  That's why window.el gets compiled into something like
window-0d1b8b93-7ef4271a.eln, and lives under a directory named
something like 28.2-4fafb146 -- these 3 hashes encode both the Emacs
binary and its version, and the original .el file's absolute file name
and contents.  So there's no way users will have any trouble when
multiple Emacs versions are installed or when different users use
different versions: the *.eln files are effectively automatically
"versioned", and Emacs will only load a .eln file that was compiled
for it.  Moreover, as soon as some user decides to modify a .el file,
the .eln file produced from it will have a different name, and if the
Emacs binary changes as result of rebuilding, the new .eln file will
be written to a different directory.

> Regarding the writes to HOME, etc.  I think David covered that fairly
> well -- Debian runs emacs to "do things" (build .elcs, run tests, etc.)
> at package build time, package install time, and during package testing.
> In all of these cases, we're likely to want to avoid side-effects outside
> the build/test dir like writing .elc or .eln files to the current user's
> HOME, whether that's /root/ or /home/*.  It sounds like we may be able
> to accomplish that by redirecting everything to a temp dir, which is
> likely fine.

Yes, by changing native-comp-eln-load-path you should be able to avoid
writing to the home directory of the user under whose credentials the
installation runs.  If you need that.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> You should know that the problems you needed to address with the *.elc
> files don't exist with the *.eln files.  The *.eln files have version
> information of the Emacs to which they "belong" hard-coded into their
> names.

Right, I'd noticed that, and while I haven't fully understood the
expectations/semantics yet, I'd started to assume much of what you've
described.  I also assumed we'd need to understand things even better
before the Debian support is completely "ready" (right now we're working
all this out in unstable).

My current expectation is that if it ends up being feasible, we'll
probably want to treat the package-related .eln files like the .elc
files, and build them during the package install.  For example,
elpa-magit (the current magit add-on package) will want to generate both
the .elc and .eln files for all of its .el files when its turn comes
around during installs/upgrades.

I say that because if it's a single-user machine and the user invokes
"apt install elpa-magit", I'd imagine they're doing that because they
want to use magit, in which case it doesn't hurt to get the work out of
the way up-front, and it might help in that all the work will be
finished at once, so they won't incur costs (battery-consumption, etc.)
later, when they might not expect or want it.  (Not a critical issue,
possibly, but perhaps friendlier.)

And if it's a multi-user machine, with a lot of emacs users, at the
moment I don't see any reason to want to compile the same file 50 times
for 50 users (or even more than just once), incurring the attendant
power and storage costs.

> Yes, by changing native-comp-eln-load-path you should be able to avoid
> writing to the home directory of the user under whose credentials the
> installation runs.  If you need that.

OK, thanks much.
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org,
>  akrl@sdf.org
> Date: Sat, 01 Oct 2022 15:42:06 -0500
> 
> My current expectation is that if it ends up being feasible, we'll
> probably want to treat the package-related .eln files like the .elc
> files, and build them during the package install.  For example,
> elpa-magit (the current magit add-on package) will want to generate both
> the .elc and .eln files for all of its .el files when its turn comes
> around during installs/upgrades.
> 
> I say that because if it's a single-user machine and the user invokes
> "apt install elpa-magit", I'd imagine they're doing that because they
> want to use magit, in which case it doesn't hurt to get the work out of
> the way up-front, and it might help in that all the work will be
> finished at once, so they won't incur costs (battery-consumption, etc.)
> later, when they might not expect or want it.  (Not a critical issue,
> possibly, but perhaps friendlier.)
> 
> And if it's a multi-user machine, with a lot of emacs users, at the
> moment I don't see any reason to want to compile the same file 50 times
> for 50 users (or even more than just once), incurring the attendant
> power and storage costs.

I don't think you should try to second-guess the user who installs a
package.  They could just want to study the sources, for example.

The native compilation is implemented as JIT capability for a reason.
We dicussed the various aspects of that for a long time before
deciding on what you see today.  My advice is not to reject that
without very good reasons (and those reasons should probably be
communicated to us when found), just because of some analogy with byte
compilation, as that analogy breaks on several levels.  By using
native compilation in its default JIT manner, you are basically using
Emacs as the upstream project meant it to be used, which means, among
others, that you don't need to fight an uphill battle against various
defaults and discover situations that no one expected to happen.

The reasons which you mention in favor of AOT native compilation don't
sound serious enough: I see no problem in having the compilation
happen only when it's needed in those cases.  Battery consumption
doesn't seem very relevant, because JIT compilation will happen when
the system is used, not when it sleeps.  And it is entirely not
guaranteed that by compiling everything you will save power in the
long run, because there are good chances you will be compiling stuff
that won't be used for a long time.  Without quantitative data of
long-term power usage on which to base the conclusions, I don't see
why you should a-priori assume that compiling everything from the
get-go should use less power.  Same goes for disk space by multiple
users.

All in all, I think JIT compilation strikes a good balance between
resources and their actual usage.  This also matches my personal
experience of using Emacs 28 with native compilation since day one: I
never had to look back on the decision not to use AOT compilation of
everything.

Of course, this is eventually your call.  But my recommendation is to
try to stick to the default behavior unless it causes actual (as
opposed to imaginary or presumed) problems, based on actual usage.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3188 bytes --]

On Fri, Sep 30, 2022 at 04:56:56PM +0300, Eli Zaretskii wrote:
> > From: David Bremner <david@tethera.net>

[...]

> > Some additional byte compilation happens at package install time.
> 
> This is what I'm asking about: what exactly triggers the compilation?
> Just installing a package shouldn't do that, only loading it into
> Emacs should.

I see there two views: the distributor's (represented here by David's)
and the jit's (Eli). For Eli, the file system objects are just a long
term cache of the jit's work (longer than one Emacs session), for
David, since the package's .el (and possibly .elc) are immutable (and
of course, the whole Emacs runtime), the corresponding .eln can well
be built at package install time. That would avoid each user to have
an identical copy of them (and of course, the initial wait).

That does make a lot of sense in a multi-user system (see below for
more on that).

> The other part of what I asked is that if for some reason installation
> does need to compile files (something that I still need to understand
> why it happens), there's nothing that hard-codes HOME in the directory
> list used for that.  You can set native-comp-eln-load-path to anything
> you want, and only the directories in that list will be used.

Yes, but. It's not about HOME, but about a user-writable space. A shared
space for .eln would have to be writable by all users if the compilation
is to succeed when invoked on behalf of each one. This is a no-no under
a multiuser system.

Install is invoked as root, so it can put the results of its compilation
to a place readable by /all/ users (typically some /usr/share or /usr/lib,
depending on whether the results are architecture-independent).

> > In my
> > view the same restrictions should apply there. In addition to the
> > unintentional sharing of state mentioned in policy, there is also the
> > issue of cleaning up after a package on uninstall. This is manageable
> > now because any generated .elc files go in a package specific directory,
> > which can just be removed on purging the package.
> 
> See above: you can do the same with *.eln files, if, for some reason,
> they are produced by the installation.
> 
> > I think just turning off native compilation when HOME is not writeable
> > would be sensible from a Debian point of view. Emacs did not previously
> > require a writable HOME (although maybe most users never tested
> > this). It would be unfortunate if this changed for Emacs with native
> > compilation support.
> 
> Emacs doesn't require a writable HOME, it requires a writable
> directory to store *.eln files.  It doesn't have to be HOME.  And once
> again, I still don't understand why *.eln files are produced at
> installation time in the first place.

That's all fine, but then users wouldn't profit from the pre-compiled
.eln. In a Debian-distributed Emacs (caveat: I'm compiling my own
one!) there are .elc in /usr/share for all to use; due to the search
path, a user still can install her own in a directory writable by
that user and it will take precedence.

Can you do the same for .elc?

Cheers
-- 
t


> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

On Sun, Oct 02, 2022 at 08:57:13AM +0300, Eli Zaretskii wrote:
> > From: Rob Browning <rlb@defaultvalue.org>
> > Cc: monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org,
> >  akrl@sdf.org
> > Date: Sat, 01 Oct 2022 15:42:06 -0500

[...]

> > And if it's a multi-user machine, with a lot of emacs users, at the
> > moment I don't see any reason to want to compile the same file 50 times
> > for 50 users (or even more than just once), incurring the attendant
> > power and storage costs.
> 
> I don't think you should try to second-guess the user who installs a
> package.  They could just want to study the sources, for example.

That's what apt-get source and friends are. The user can download,
build, modify, etc. the sources corresponding to packages. Those
are purely user operations, no admin powers needed.

But installing a binary pacage on a system does modify the system
for all users, so admin powers do make sense there. Nobody is being
second-guessed, just the roles separated (again, on a single user
system, this might feel artificial).

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Sun, 2 Oct 2022 07:58:33 +0200
> From: <tomas@tuxteam.de>
> 
> I see there two views: the distributor's (represented here by David's)
> and the jit's (Eli). For Eli, the file system objects are just a long
> term cache of the jit's work (longer than one Emacs session), for
> David, since the package's .el (and possibly .elc) are immutable (and
> of course, the whole Emacs runtime), the corresponding .eln can well
> be built at package install time. That would avoid each user to have
> an identical copy of them

As explained in my other message, I see no reason to make such
assumptions.  There's no reason to believe that a user who installs a
package will immediately start using it.  And even it they do, why
should we care about some extra disk space or identical files being
present? disk space is cheap, while the flexibility this allows (like
each user can have a slightly differently-configured Emacs) is
non-negligible.

> (and of course, the initial wait).

There's no wait: a package can be used immediately after loading or
requiring it.  The replacement of a byte-compiled by natively-compiled
code happens transparently and doesn't affect usage nor causes any
waits.

The advantage of using JIT compilation is that this is how the
upstream project meant it to be used, and this is how the defaults are
set.  Any deviation from the defaults should have a good reason, and I
submit that such good reasons can only be based on actual usage, not
on theoretical presumptions.  My recommendation is to use the default
JIT manner until and unless actual problems are reported by users.

> That does make a lot of sense in a multi-user system

Not to me, it doesn't.  It makes _some_ sense, but there are other
considerations.

> > The other part of what I asked is that if for some reason installation
> > does need to compile files (something that I still need to understand
> > why it happens), there's nothing that hard-codes HOME in the directory
> > list used for that.  You can set native-comp-eln-load-path to anything
> > you want, and only the directories in that list will be used.
> 
> Yes, but. It's not about HOME, but about a user-writable space. A shared
> space for .eln would have to be writable by all users if the compilation
> is to succeed when invoked on behalf of each one. This is a no-no under
> a multiuser system.

??? What "user-writable space"?  If we are talking about install-time
compilation, then the *.eln files should be deposited in some
centralized place to which the installation process can write.  And if
we are talking about JIT compilation when the user runs Emacs, then
the home directory is perfectly appropriate, and is writable.  No one
suggested that JIT compilations from "normal" user session should
write to shared directories (and if someone did suggest that, IMO this
suggestion makes no sense).

So I think you confused these two possibilities, because the problem
you raise doesn't exist.

> Install is invoked as root, so it can put the results of its compilation
> to a place readable by /all/ users (typically some /usr/share or /usr/lib,
> depending on whether the results are architecture-independent).

Exactly.  So what is the problem with directories writable by all
users?

> > Emacs doesn't require a writable HOME, it requires a writable
> > directory to store *.eln files.  It doesn't have to be HOME.  And once
> > again, I still don't understand why *.eln files are produced at
> > installation time in the first place.
> 
> That's all fine, but then users wouldn't profit from the pre-compiled
> .eln.

There's no profit, IME.  There are only disadvantages: you are in
effect fighting against the Emacs defaults, for reasons that are
purely theoretical.

> In a Debian-distributed Emacs (caveat: I'm compiling my own one!)
> there are .elc in /usr/share for all to use; due to the search path,
> a user still can install her own in a directory writable by that
> user and it will take precedence.
> 
> Can you do the same for .elc?

I guess you meant "with .eln files"?  Yes, see
native-comp-eln-load-path, which was already mentioned here several
times.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Sun, 2 Oct 2022 08:10:15 +0200
> From: <tomas@tuxteam.de>
> 
> > I don't think you should try to second-guess the user who installs a
> > package.  They could just want to study the sources, for example.
> 
> That's what apt-get source and friends are. The user can download,
> build, modify, etc. the sources corresponding to packages. Those
> are purely user operations, no admin powers needed.
> 
> But installing a binary pacage on a system does modify the system
> for all users, so admin powers do make sense there.

What do you mean by "binary package" in this context?  I believe
"package" in this discussion generally alludes to Lisp packages, from
ELPA and elsewhere.  They aren't "binary" in my book.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3484 bytes --]

On Sun, Oct 02, 2022 at 09:42:04AM +0300, Eli Zaretskii wrote:
> > Date: Sun, 2 Oct 2022 07:58:33 +0200
> > From: <tomas@tuxteam.de>
> > 
> > I see there two views: the distributor's (represented here by David's)
> > and the jit's (Eli) [...]

> As explained in my other message, I see no reason to make such
> assumptions.  There's no reason to believe that a user who installs a
> package will immediately start using it [...]

(NOTE Eli, I admire your patience :-)

As I see from your other post, there is a confusion potential with
the word "package". I was talking about Debian binary packages, which
are thought for those who want to use them. For those wanting to
change or build there are the source packages (which come with a
convenient build "kit" enabling you to build the binary package as
it comes with Debian, which is a convenient starting point for
tinkering).

> And even it they do, why
> should we care about some extra disk space or identical files being
> present? disk space is cheap, while the flexibility this allows (like
> each user can have a slightly differently-configured Emacs) is
> non-negligible.

Definitely. I am not that much concerned about disk usage, more about
a uniform "baseline" installation all users can rely on.

[...]

> The advantage of using JIT compilation is that this is how the
> upstream project meant it to be used, and this is how the defaults are
> set.  Any deviation from the defaults should have a good reason, and I
> submit that such good reasons can only be based on actual usage, not
> on theoretical presumptions.  My recommendation is to use the default
> JIT manner until and unless actual problems are reported by users.

I see that, and this is the one view I mention above: the .eln are but
a JIT cache, and each user (or instance, if there are more than one
per user) has its own.

> > That does make a lot of sense in a multi-user system
> 
> Not to me, it doesn't.  It makes _some_ sense, but there are other
> considerations.

Two views, as I said :)

[HOME, user-writable space]

> Exactly.  So what is the problem with directories writable by all
> users?

User separation goes out of the window, and this is one important
service the OS provides. To illustrate, one user could put malicious
.eln files all other users would execute.

> > > Emacs doesn't require a writable HOME, it requires a writable
> > > directory to store *.eln files.  It doesn't have to be HOME.  And once
> > > again, I still don't understand why *.eln files are produced at
> > > installation time in the first place.
> > 
> > That's all fine, but then users wouldn't profit from the pre-compiled
> > .eln.
> 
> There's no profit, IME.  There are only disadvantages: you are in
> effect fighting against the Emacs defaults, for reasons that are
> purely theoretical.

I have the impression that some of that reasons are quite practical
for Debian packagers.

> > In a Debian-distributed Emacs (caveat: I'm compiling my own one!)
> > there are .elc in /usr/share for all to use; due to the search path,
> > a user still can install her own in a directory writable by that
> > user and it will take precedence.
> > 
> > Can you do the same for .elc?
> 
> I guess you meant "with .eln files"?  Yes, see
> native-comp-eln-load-path, which was already mentioned here several
> times.

So that might be one part of the way out.

Cheers
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 597 bytes --]

On Sun, Oct 02, 2022 at 09:44:16AM +0300, Eli Zaretskii wrote:
> > Date: Sun, 2 Oct 2022 08:10:15 +0200
> > From: <tomas@tuxteam.de>

[...]

> > But installing a binary pacage on a system does modify the system
> > for all users, so admin powers do make sense there.
> 
> What do you mean by "binary package" in this context?  I believe
> "package" in this discussion generally alludes to Lisp packages, from
> ELPA and elsewhere.  They aren't "binary" in my book.

Ah, I see. I was too sloppy. I was talking about Debian packages.

Sorry for the confusion.

Cheers
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Sun, 2 Oct 2022 17:54:44 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > > But installing a binary pacage on a system does modify the system
> > > for all users, so admin powers do make sense there.
> > 
> > What do you mean by "binary package" in this context?  I believe
> > "package" in this discussion generally alludes to Lisp packages, from
> > ELPA and elsewhere.  They aren't "binary" in my book.
> 
> Ah, I see. I was too sloppy. I was talking about Debian packages.

That still doesn't explain it to me (I don't use Debian).  Can you
elaborate what kind of packages are we talking about, in the context
of this discussion?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Sun, 2 Oct 2022 17:53:46 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > The advantage of using JIT compilation is that this is how the
> > upstream project meant it to be used, and this is how the defaults are
> > set.  Any deviation from the defaults should have a good reason, and I
> > submit that such good reasons can only be based on actual usage, not
> > on theoretical presumptions.  My recommendation is to use the default
> > JIT manner until and unless actual problems are reported by users.
> 
> I see that, and this is the one view I mention above: the .eln are but
> a JIT cache, and each user (or instance, if there are more than one
> per user) has its own.

Let me be blunt: this is currently _the_only_ view of the Emacs
project.  After a lot of deliberations, we didn't find any reasons for
alternative views.  My suggestion is to try our view before concluding
that it doesn't fit some situations.

> > Exactly.  So what is the problem with directories writable by all
> > users?
> 
> User separation goes out of the window, and this is one important
> service the OS provides. To illustrate, one user could put malicious
> .eln files all other users would execute.

This is about installation writing files into a shared space on disk,
right?  If so, this is something for the Debian packagers to figure
out, because doing that is their request.  And anyway, I don't
understand how do *.eln files are different from *.elc files, which
are already written to shared disk space upon installation.  What am I
missing?

> > > That's all fine, but then users wouldn't profit from the pre-compiled
> > > .eln.
> > 
> > There's no profit, IME.  There are only disadvantages: you are in
> > effect fighting against the Emacs defaults, for reasons that are
> > purely theoretical.
> 
> I have the impression that some of that reasons are quite practical
> for Debian packagers.

I submit that those reasons were most probably derived from a broken
analogy with the *.elc files and with byte-compilation in general.
Not from actual usage experience.  Native compilation looks
deceptively similar to byte compilation, but it isn't.  So if
producing the *.eln files seems to contradict some Debian rules and
procedures, my suggestion is to talk to the upstream project, before
inventing solutions, because of 2 considerations:

  . the problems may not be real ones, only perceived ones
  . the upstream codebase might already provide solutions

> > > In a Debian-distributed Emacs (caveat: I'm compiling my own one!)
> > > there are .elc in /usr/share for all to use; due to the search path,
> > > a user still can install her own in a directory writable by that
> > > user and it will take precedence.
> > > 
> > > Can you do the same for .elc?
> > 
> > I guess you meant "with .eln files"?  Yes, see
> > native-comp-eln-load-path, which was already mentioned here several
> > times.
> 
> So that might be one part of the way out.

If one needs it.  I don't think they do, and I don't recommend going
there.

Re: Suppressing native compilation (short and long term)

[chad]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]

On Sun, Oct 2, 2022 at 12:03 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > Ah, I see. I was too sloppy. I was talking about Debian packages.
>
> That still doesn't explain it to me (I don't use Debian).  Can you
> elaborate what kind of packages are we talking about, in the context
> of this discussion?
>

Debian supports a user installing, for example, emacs+magit, via Debian
packages. This gets the user a stable, known-tested version of emacs plus
the package usable on machines that, for another example, cannot or should
not connect to the internet. This is less important to developers, but is
an important part of the Debian support "contract".

For the record: I personally know of situations where such a setup would
like to use native-comp and would very much prefer NOT to duplicate either
the eln files per-user nor the effort of creating such. Whether or not that
situation is important enough to the combination of emacs-devel and
debian-maintainers to justify effort on either side is, of course,
debatable, but I am highly confident that they exist (at least, did before
the pandemic).

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 1557 bytes --]

Re: Suppressing native compilation (short and long term)

[chad]

[-- Attachment #1: Type: text/plain, Size: 1761 bytes --]

On Sun, Oct 2, 2022 at 12:13 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > User separation goes out of the window, and this is one important
> > service the OS provides. To illustrate, one user could put malicious
> > .eln files all other users would execute.
>
> This is about installation writing files into a shared space on disk,
> right?  If so, this is something for the Debian packagers to figure
> out, because doing that is their request.  And anyway, I don't
> understand how do *.eln files are different from *.elc files, which
> are already written to shared disk space upon installation.  What am I
> missing?
>

At the risk of over-explaining due to message crossing mid-flight: the
thing you might be missing is that Debian provides a mechanism for people
to install *.elc files in a space shared by all users that is not
writable by those users, and there are people that use this provision.
Since these are used for *.elc files, it seems highly likely that they will
be desired for *.eln files.

Even in the face of a theoretical issue like "potential package
combinations make it unworkable to pre-build a single set of
emacs+emacs-packages", the practical situation is that such combinations
exist and are used by Debian users to build a stable base of
emacs+emacs-packages that is shared across users who cannot change that
shared base (but can, of course, supplement it with their own packages, via
site-lisp, user packages, etc.) As a practical goal, there is at least
*some* impetus for Debian to provide such a stable base, and to make it as
wide as reasonable. The basic mechanism for determining the size of that
base is "which emacs-packages are made into debian-packages", (iff I
understand correctly; I'm not a Debian expert).

~Chad

[-- Attachment #2: Type: text/html, Size: 2215 bytes --]

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3544 bytes --]

On Sun, Oct 02, 2022 at 07:11:52PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 2 Oct 2022 17:53:46 +0200
> > From: tomas@tuxteam.de
> > Cc: emacs-devel@gnu.org

[two views: JIT cache vs. pre-compiled .eln]

> Let me be blunt: this is currently _the_only_ view of the Emacs
> project.  After a lot of deliberations, we didn't find any reasons for
> alternative views.  My suggestion is to try our view before concluding
> that it doesn't fit some situations.

Thanks for being blunt :-)

My whole intention was to make this difference clear, because
I had the impression that there were unspoken assumptions
making the discussion unnecessarily difficult.

> > > Exactly.  So what is the problem with directories writable by all
> > > users?
> > 
> > User separation goes out of the window, and this is one important
> > service the OS provides. To illustrate, one user could put malicious
> > .eln files all other users would execute.
> 
> This is about installation writing files into a shared space on disk,
> right?

No. I was picking up on your "directories writable by all users".
Perhaps I misunderstood and you didn't mean common directories:
if so, please ignore.

>  If so, this is something for the Debian packagers to figure
> out, because doing that is their request.  And anyway, I don't
> understand how do *.eln files are different from *.elc files, which
> are already written to shared disk space upon installation.  What am I
> missing?

Perhaps nothing. With the native-comp-eln-load-path, there seems
to be a way for Debian to "do it its way"; you don't recommend
it (I still don't quite understand), and there are very strong
reasons to take your recommendations seriously.

> > > > That's all fine, but then users wouldn't profit from the pre-compiled
> > > > .eln.
> > > 
> > > There's no profit, IME.  There are only disadvantages: you are in
> > > effect fighting against the Emacs defaults, for reasons that are
> > > purely theoretical.
> > 
> > I have the impression that some of that reasons are quite practical
> > for Debian packagers.
> 
> I submit that those reasons were most probably derived from a broken
> analogy with the *.elc files and with byte-compilation in general.
> Not from actual usage experience.  Native compilation looks
> deceptively similar to byte compilation, but it isn't.  So if
> producing the *.eln files seems to contradict some Debian rules and
> procedures, my suggestion is to talk to the upstream project, before
> inventing solutions, because of 2 considerations:

I understand that there is a difference between .elc and .eln (the
set of dependencies is significantly bigger in the second case, for
one).

> 
>   . the problems may not be real ones, only perceived ones
>   . the upstream codebase might already provide solutions

I can understand Debian's position here (yours too).

> > > > In a Debian-distributed Emacs [...]
> > > > there are .elc in /usr/share for all to use; due to the search path,

[...]

> > > > Can you do the same for .elc?
> > > 
> > > I guess you meant "with .eln files"?

Uh -- yes, sorry. Well spotted.

> > > Yes, see native-comp-eln-load-path, which was already mentioned
> > > here several times.
> > 
> > So that might be one part of the way out.
> 
> If one needs it.  I don't think they do, and I don't recommend going
> there.

Hm. I don't want to steal your time more, but... if you could illustrate
why, I'd eager to learn.

Cheers
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> I don't think you should try to second-guess the user who installs a
> package.  They could just want to study the sources, for example.

I'm quite happy with the use of JIT as the default in Emacs.
But I think I agree with Rob that it makes a lot of sense in the context
of Debian to eagerly native-compile the packages when they're installed
via APT.

In APT there's a clearly distinction between installing the package
(which requires admin rights and has non-trivial effects on the whole
system) and looking at the source code of the package (this distinction
is natural in the context of Debian where many/most packages are written
in languages like C, but it naturally carries over to ELisp packages).

So if user "just want to study the sources" they wouldn't *install* the
package at all.

> All in all, I think JIT compilation strikes a good balance between
> resources and their actual usage.

Yes.  But the balance is different in different contexts.

FWIW, I'm not convinced it's really useful in Debian's `emacs` package
to eagerly native compile all the bundled .elc files, but I think it
does make a lot of sense for ELisp packages installed separately.

In any case, I'd let Debian's maintainers make their own choices for
their own specific needs which are slightly different from ours (where
our release tarballs and default config are designed in large part for
users who'll compile Emacs themselves and who install third party
ELisp packages into their $HOME).


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Sun, 2 Oct 2022 12:13:44 -0400
> Cc: tomas@tuxteam.de, emacs-devel@gnu.org
> 
>  > Ah, I see. I was too sloppy. I was talking about Debian packages.
> 
>  That still doesn't explain it to me (I don't use Debian).  Can you
>  elaborate what kind of packages are we talking about, in the context
>  of this discussion?
> 
> Debian supports a user installing, for example, emacs+magit, via Debian packages. This gets the user a
> stable, known-tested version of emacs plus the package usable on machines that, for another example,
> cannot or should not connect to the internet. This is less important to developers, but is an important part of
> the Debian support "contract".

Which part of the "emacs+magit" package is the "binary package"?  The
only part of that which could produce *.eln files at installation time
is Magit, right?  Because Emacs itself already comes with all the
preloaded *.eln files, and those aren't what we are talking about,
right?

And if we are talking about Magit in this example, then how come its
being bundled to Emacs change anything of what I said?

> For the record: I personally know of situations where such a setup would like to use native-comp and would
> very much prefer NOT to duplicate either the eln files per-user nor the effort of creating such. Whether or
> not that situation is important enough to the combination of emacs-devel and debian-maintainers to justify
> effort on either side is, of course, debatable, but I am highly confident that they exist (at least, did before the
> pandemic).

Please forgive me, but you cannot expect us to regard such situations
seriously as long as you don't describe them.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Sun, 2 Oct 2022 12:23:43 -0400
> Cc: tomas@tuxteam.de, emacs-devel@gnu.org
> 
>  > User separation goes out of the window, and this is one important
>  > service the OS provides. To illustrate, one user could put malicious
>  > .eln files all other users would execute.
> 
>  This is about installation writing files into a shared space on disk,
>  right?  If so, this is something for the Debian packagers to figure
>  out, because doing that is their request.  And anyway, I don't
>  understand how do *.eln files are different from *.elc files, which
>  are already written to shared disk space upon installation.  What am I
>  missing?
> 
> At the risk of over-explaining due to message crossing mid-flight: the thing you might be missing is that
> Debian provides a mechanism for people to install *.elc files in a space shared by all users that is not
> writable by those users, and there are people that use this provision. Since these are used for *.elc files, it
> seems highly likely that they will be desired for *.eln files.

No, I don't think the similar handling makes sense here.  The *.elc
files are architecture- and configuration-independent, whereas the
*.eln files are not.  E.g., the same foo.elc could be used by user A
who runs Emacs 28 and by user B who runs Emacs 29.  But the
corresponding *.eln files will be different, even though they were
produced from the same foo.el.

> Even in the face of a theoretical issue like "potential package combinations make it unworkable to pre-build a
> single set of emacs+emacs-packages", the practical situation is that such combinations exist and are used
> by Debian users to build a stable base of  emacs+emacs-packages that is shared across users who
> cannot change that shared base (but can, of course, supplement it with their own packages, via site-lisp,
> user packages, etc.) As a practical goal, there is at least *some* impetus for Debian to provide such a stable
> base, and to make it as wide as reasonable. The basic mechanism for determining the size of that base is
> "which emacs-packages are made into debian-packages", (iff I understand correctly; I'm not a Debian
> expert).

I don't think this is relevant to the issue at hand.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Sun, 2 Oct 2022 18:27:05 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > > > Yes, see native-comp-eln-load-path, which was already mentioned
> > > > here several times.
> > > 
> > > So that might be one part of the way out.
> > 
> > If one needs it.  I don't think they do, and I don't recommend going
> > there.
> 
> Hm. I don't want to steal your time more, but... if you could illustrate
> why, I'd eager to learn.

I already did: the probability of different users to produce different
*.eln files from the same *.el sources is high, and disk space is
cheap.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> Which part of the "emacs+magit" package is the "binary package"?

The "binary" Debian package for `elpa-magit` mostly contains Magit's
`.el` files plus the doc and a few other things.  IOW it's fairly
similar to our ELPA tarballs.

So it's very close to the source code itself, but it's still separate
from "the source" (which you can get via things like `apt source elpa-magit`
and will fetch some tarball rather than a `.deb` file) from which the
`.deb` is built.

> And if we are talking about Magit in this example, then how come its
> being bundled to Emacs change anything of what I said?

I don't think there's a `emacs+magit` package in Debian.  There's an
`elpa-magit` package and an `emacs` package.

>> For the record: I personally know of situations where such a setup
>> would like to use native-comp and would very much prefer NOT to
>> duplicate either the eln files per-user nor the effort of creating
>> such. Whether or not that situation is important enough to the
>> combination of emacs-devel and debian-maintainers to justify effort
>> on either side is, of course, debatable, but I am highly confident
>> that they exist (at least, did before the pandemic).
>
> Please forgive me, but you cannot expect us to regard such situations
> seriously as long as you don't describe them.

We don't have to take their opinion into account when choosing
Emacs's defaults.  We just need to provide the tools that let them get
the behavior they want.


        Stefan

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Battery consumption doesn't seem very relevant, because JIT
> compilation will happen when the system is used, not when it sleeps.
> And it is entirely not guaranteed that by compiling everything you
> will save power in the long run, because there are good chances you
> will be compiling stuff that won't be used for a long time.  Without
> quantitative data of long-term power usage on which to base the
> conclusions, I don't see why you should a-priori assume that compiling
> everything from the get-go should use less power.  Same goes for disk
> space by multiple users.

On this particular topic, I was actually just communicating a concern
that was communicated to me -- that a user wasn't happy about the
unpredictable, compilation spike long after the install on their laptop,
(in part due to concerns about power consumption).

(Personally, as a user, I'd also prefer to pay the cost up front, most
 of the time, during package install, and to avoid per-user duplications
 where possible, but that said, I generally do try to avoid basing
 Debian packaging decisions solely on my preferences, and I do want to
 try to favor upstream preferences in general.)

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Rob Browning <rlb@defaultvalue.org>,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Sun, 02 Oct 2022 12:53:56 -0400
> 
> > All in all, I think JIT compilation strikes a good balance between
> > resources and their actual usage.
> 
> Yes.  But the balance is different in different contexts.

Theoretically, yes.  But I don't yet see why the particular context
described by Rob is different to the degree that it would need special
procedures.

> FWIW, I'm not convinced it's really useful in Debian's `emacs` package
> to eagerly native compile all the bundled .elc files, but I think it
> does make a lot of sense for ELisp packages installed separately.

Maybe so, but even if that is the decision about the *.elc files, it
doesn't automatically follow that the *.eln files should be handled
the same.  I explained in previous messages why I think so, with
enough important differences between them to facilitate rethinking, I
hope.

> In any case, I'd let Debian's maintainers make their own choices for
> their own specific needs which are slightly different from ours (where
> our release tarballs and default config are designed in large part for
> users who'll compile Emacs themselves and who install third party
> ELisp packages into their $HOME).

We can only advise them, yes.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> But I think I agree with Rob that it makes a lot of sense in the context
> of Debian to eagerly native-compile the packages when they're installed
> via APT.

I've always assumed that that's what distributions like Debian would
do.  Which means that there should be an easy way to switch JIT off, but
I've just forgotten to add it.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: chad <yandros@gmail.com>,  tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 13:13:59 -0400
> 
> > Which part of the "emacs+magit" package is the "binary package"?
> 
> The "binary" Debian package for `elpa-magit` mostly contains Magit's
> `.el` files plus the doc and a few other things.  IOW it's fairly
> similar to our ELPA tarballs.

That's what I thought.  But then I don't see how this is special in
the context of this discussion, so that what Emacs users do when they
download from ELPA is different from them installing a Debian "binary"
package.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> That still doesn't explain it to me (I don't use Debian).  Can you
> elaborate what kind of packages are we talking about, in the context
> of this discussion?

One possible way to look at it -- on a Debian system I think you'd
generally

  apt install emacs  # (or gcc-12 or ...)

if you just wanted to use emacs or gcc, or...  And that would install
the binaries, more or less, i.e. roughly like "make install", but with
automatic dependency resolution, etc.

And note that with many tools that would get you *just* the binaries,
i.e. no source at all for say gcc, libc, grep, git, etc.

If you wanted to investigate, fix, enhance the packaging itself, then
you might

  apt source emacs

which would download the packaging and unpack it in the current
directory, as say emacs-28.1+1-3/, and where all the packaging
related-changes, including any patches applied to the upstream code,
would be clearly visible in a emacs-28.1+1-3/debian/ subdirectory in the
emacs source.

For many packages these days you might also be able to just clone the
Debian packaging archive, e.g.

  git clone https://salsa.debian.org/rlb/deb-emacs.git

which might be preferable if you're more comfortable with git.

And finally, if your primary interest is the upstream source, and/or
running and/or working on anything other than the versions debian
currently provides, I suspect you'd likely want to

  git clone git://git.savannah.gnu.org/emacs.git

i.e. you wouldn't need and might well not want the debian packag(ing|es)
at all.

Hope this helps.
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> On this particular topic, I was actually just communicating a concern
> that was communicated to me -- that a user wasn't happy about the
> unpredictable, compilation spike long after the install on their laptop,
> (in part due to concerns about power consumption).

FWIW, regardless of what Debian decides to do, I think it would make
sense to offer some way to configure Emacs so that it native compiles
packages a bit more eagerly.

E.g. we could offer a configuration option that causes `package-install`
to eagerly native compile the installed files.  And we could offer
a "recompile" command that users can use after upgrading Emacs to
eagerly native-(re)compile all the ELPA-installed packages in the
user's $HOME.

The situation is very different for the bundled packages, since most of
them will never be used by most users, so JIT compilation is much
more beneficial.


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org,
>  akrl@sdf.org
> Date: Sun, 02 Oct 2022 12:15:27 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Battery consumption doesn't seem very relevant, because JIT
> > compilation will happen when the system is used, not when it sleeps.
> > And it is entirely not guaranteed that by compiling everything you
> > will save power in the long run, because there are good chances you
> > will be compiling stuff that won't be used for a long time.  Without
> > quantitative data of long-term power usage on which to base the
> > conclusions, I don't see why you should a-priori assume that compiling
> > everything from the get-go should use less power.  Same goes for disk
> > space by multiple users.
> 
> On this particular topic, I was actually just communicating a concern
> that was communicated to me -- that a user wasn't happy about the
> unpredictable, compilation spike long after the install on their laptop,
> (in part due to concerns about power consumption).

That's just something that takes getting used to.

> (Personally, as a user, I'd also prefer to pay the cost up front, most
>  of the time, during package install, and to avoid per-user duplications
>  where possible, but that said, I generally do try to avoid basing
>  Debian packaging decisions solely on my preferences, and I do want to
>  try to favor upstream preferences in general.)

Those seem like old habit that die hard, IMO.  For users of some
programming languages, JIT compilation is routine, and I bet they
don't share these habits.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Lars Ingebrigtsen [2022-10-02 19:17:27] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> But I think I agree with Rob that it makes a lot of sense in the context
>> of Debian to eagerly native-compile the packages when they're installed
>> via APT.
> I've always assumed that that's what distributions like Debian would
> do.  Which means that there should be an easy way to switch JIT off, but
> I've just forgotten to add it.

Switching it off means that the native compilation is never
auto-triggered, so it's probably not quite what you meant :-)


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Rob Browning <rlb@defaultvalue.org>,
>   david@tethera.net,  emacs-devel@gnu.org,  akrl@sdf.org
> Date: Sun, 02 Oct 2022 19:17:27 +0200
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > But I think I agree with Rob that it makes a lot of sense in the context
> > of Debian to eagerly native-compile the packages when they're installed
> > via APT.
> 
> I've always assumed that that's what distributions like Debian would
> do.  Which means that there should be an easy way to switch JIT off, but
> I've just forgotten to add it.

If everything is already natively-compiled, there's no reason to
switch it off: it will simply not happen.

OTOH, if for some reason an existing .eln file is incompatible with
the user's Emacs (something that can happen easily on a multi-user
system), having JIT compilation active is a clear advantage.

So there's a net win in having it enabled at all times.

In any case, we already have a way of switching off JIT compilation,
at least for some situations which we considered.  If there are
situations which we didn't consider, we should consider them as well,
but for that we need to understand the situation in more detail,
something that no one presented yet in this discussion.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Switching it off means that the native compilation is never
> auto-triggered, so it's probably not quite what you meant :-)

I'm not quite sure what you mean -- I'm saying that there should be a
way for users to switch off native compilation (and for distributions to
have native compilation for users switched off).

Which was how this thread started: With a request for such a thing.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> In any case, I'd let Debian's maintainers make their own choices for
> their own specific needs which are slightly different from ours (where
> our release tarballs and default config are designed in large part for
> users who'll compile Emacs themselves and who install third party
> ELisp packages into their $HOME).

For what it's worth, we've also had repeated requests for additional
variants of emacs.  Right now we have emacs-nox (No X), emacs-lucid, and
emacs-gtk.

The first is often useful for server installs, but we've had requests
for something even smaller, say an emacs-min that configures even less,
and is even smaller (including the dependency tree), for some
situations.  Presumably people who would rather be able to easily use
emacs in some constrained environments instead of nano, zile, vi,
... etc.

We've also for a long time made the emacs-el package (containing all the
.el files) optional for similar reasons (constrained environments), but
we recently had to start requiring it because (not sure why yet) emacs
just crashes in some situations when the .el files aren't available and
when native compilation is enabled.

If it hasn't already been fixed in 28.2 (haven't tested yet), it'd be
nice if we could eventually track that down and then switch emacs-el
back to optional (back to "suggested" in Debian dependency parlance).

We'll likely try to help with that later, after we get the other native
compilation-related issues we've been discussing settled down on the
Debian side.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> So there's a net win in having it enabled at all times.

You may think so, and I may think so, but it should be up to the user
(and the distribution).

> In any case, we already have a way of switching off JIT compilation,
> at least for some situations which we considered.

I thought there wasn't a user option to switch it off?  I must admit I
haven't been paying that much attention to the thread, but if somebody
has identified such a user option, then everything's OK.  What's it
called?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Rob Browning <rlb@defaultvalue.org>,
>  david@tethera.net,  emacs-devel@gnu.org,  akrl@sdf.org
> Date: Sun, 02 Oct 2022 19:36:01 +0200
> 
> I'm not quite sure what you mean -- I'm saying that there should be a
> way for users to switch off native compilation (and for distributions to
> have native compilation for users switched off).
> 
> Which was how this thread started: With a request for such a thing.

Which turned out to be a request for something else: for controlling
where the *.eln files are written.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: david@tethera.net, emacs-devel@gnu.org, akrl@sdf.org
> Date: Sun, 02 Oct 2022 12:37:33 -0500
> 
> We've also for a long time made the emacs-el package (containing all the
> .el files) optional for similar reasons (constrained environments), but
> we recently had to start requiring it because (not sure why yet) emacs
> just crashes in some situations when the .el files aren't available and
> when native compilation is enabled.

Loading of natively-compiled files cannot work if the source files
aren't accessible, because loading a .eln file involves verifying that
it was produced from that particular .el file.  (The .el file can be
compressed if Emacs was compiled with zlib support.)

But Emacs should not "crash" if the *.el files aren't available, it
should simply refuse to load any *.eln files and load the *.elc files
instead.  That produces many warnings, of course, but I hope your
users don't consider that "crashing".

> If it hasn't already been fixed in 28.2 (haven't tested yet), it'd be
> nice if we could eventually track that down and then switch emacs-el
> back to optional (back to "suggested" in Debian dependency parlance).

It isn't a bug, but intended behavior.  If we want to remove this
dependency, some non-trivial ideas about reworking the current load
procedure should emerge.  I don't thin we have any such ideas at this
time.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: monnier@iro.umontreal.ca,  rlb@defaultvalue.org,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Sun, 02 Oct 2022 19:38:01 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So there's a net win in having it enabled at all times.
> 
> You may think so, and I may think so, but it should be up to the user
> (and the distribution).

If they present their case clearly, we might change our minds.  If
they don't present the case clearly, I don't see how we can provide a
solution, because disabling JIT compilation is already possible.  So
someone should explain why what is there is not good enough, and that
requires details.

> > In any case, we already have a way of switching off JIT compilation,
> > at least for some situations which we considered.
> 
> I thought there wasn't a user option to switch it off?

There are several, actually, each one doing something slightly
different and disabling native compilation on a different level.
That's why I'm asking for details: we need to understand whether
something is missing and why.  For example, "disable native
compilation" may or may not be the same as "disable writing *.eln
files".

Re: Suppressing native compilation (short and long term)

[Rob Browning]

chad <yandros@gmail.com> writes:

> At the risk of over-explaining due to message crossing mid-flight: the
> thing you might be missing is that Debian provides a mechanism for people
> to install *.elc files in a space shared by all users that is not
> writable by those users, and there are people that use this provision.
> Since these are used for *.elc files, it seems highly likely that they will
> be desired for *.eln files.

To be clear, I think there are (at least) two concerns here.

First, from the Debian side, we may need some way to avoid writing files
(i.e. the .eln files) to certain locations during testing, installs,
etc.  That problem may be solved -- via the variable that's already been
mentioned.

Though note that for reasons we can elaborate on if desired, it might be
easier for us if the default for that variable could also be set via a
corresponding environment variable, but that's a separate question.

(For example, we have relevant emacs-related packages where the upstream
 build process runs emacs a level or two "down" (subprocess-wise), and
 so it'd be a lot less invasive if we could just set an environment
 variable to change the .eln destination, instead of having to figure
 out how to change each invocation of emacs in that package (and
 maintain those changes across future upstream versions).


A second, and a separable question, is whether Debian should try to
maintain system-level (root owned) .eln files the same way it does for
.elc files.

I consider that an open question, and it could well be that the answer
ends up being "no".  That's what we're trying to find out, and of course
we have to begin by trying to communicate why that might be desirable.

So here we are :)

It's certainly the case that if the consensus here (among the upstream
developers) is that we shouldn't do that, and that future
choices/changes aren't likely to take that use case into consideration,
then we have our answer.

And that's fine.  In the end, we just wanted to explain the potential
use case(s), and see how the developers felt about it.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

> A second, and a separable question, is whether Debian should try to
> maintain system-level (root owned) .eln files the same way it does for
> .elc files.

I think that makes sense.

Re: Suppressing native compilation (short and long term)

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> E.g. we could offer a configuration option that causes `package-install`
> to eagerly native compile the installed files.

Isn't that just `package-native-compile'?

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Theoretically, yes.  But I don't yet see why the particular context
> described by Rob is different to the degree that it would need special
> procedures.

While I'm not sure where we'll end up, I do think others may put more
weight on some of the concerns.  For example, across the broader
world/users-of-debian-and-derivative packages, I suspect there are some
who care more about storage space.

As mentioned elswhere, we get regular requests to make emacs-nox even
smaller.  And my eln-cache is currently 40MB, which is storage space
we'd thought we might not have to duplicate across the emacs users on a
system in the common case (Of course I know the duplication rate would
vary depending on which users use which things, but unless the sets were
disjoint, there'd be duplication that didn't seem necessary.)

But again, in the end, we just wanted to present the broader case for
consideration.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I'm not quite sure what you mean -- I'm saying that there should be a
> way for users to switch off native compilation (and for distributions to
> have native compilation for users switched off).
>
> Which was how this thread started: With a request for such a thing.

At the top level, we wanted a way to avoid writing to HOME during
packaging, testing, installs (in this case, it's the .eln files, now
that we've enabled native compilation).

That could be handled by some way to turn off native compilation, or by
some way to comprehensively divert those writes to another location
(e.g. temp dir).  Either is fine, though we'd originally thought the
former might make things a bit easier.

Whether or not we can (or should) try to build system-level (root owned)
.eln files along with the .elc files during package installs (as we
already do for .elc files) is a separate question, and I think the more
controversial one?

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> But Emacs should not "crash" if the *.el files aren't available, it
> should simply refuse to load any *.eln files and load the *.elc files
> instead.  That produces many warnings, of course, but I hope your
> users don't consider that "crashing".

I believe it was *crashing*.  I can't recall if that one was a segfault,
or something a bit less drastic, but I'll try to remember to track it
down later.

> It isn't a bug, but intended behavior.  If we want to remove this
> dependency, some non-trivial ideas about reworking the current load
> procedure should emerge.  I don't thin we have any such ideas at this
> time.

OK, so we should consider that a hard dependency now, i.e. the emacs-el
package can't be optional anymore, at least not on architectures where
we can enable native compilation, and so probably just "everywhere" for
simplicity, if nothing else.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> Though note that for reasons we can elaborate on if desired, it might be
> easier for us if the default for that variable could also be set via a
> corresponding environment variable, but that's a separate question.

Usually Emacs should obey $HOME so if you can set that to a tmp dir that
should let you avoid touching the user's real $HOME.

This said, there is some code in Emacs that prefers to use ~<user> over
$HOME in some cases (like `su` without the `-`).  I'm not sure exactly
where this happens, but I suspect that the following lines in
`startup.el` are part of the answer:

    ;; Figure out which user's init file to load,
    ;; either from the environment or from the options.
    (setq init-file-user (if noninteractive nil (user-login-name)))
    ;; If user has not done su, use current $HOME to find .emacs.
    (and init-file-user
         (equal init-file-user (user-real-login-name))
	 (setq init-file-user ""))


-- Stefan

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Stefan Kangas [2022-10-02 18:11:39] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> E.g. we could offer a configuration option that causes `package-install`
>> to eagerly native compile the installed files.
> Isn't that just `package-native-compile'?

Oh, it's there!
Wee!!


        Stefan

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> My recommendation is to use the default JIT manner until and unless
> actual problems are reported by users.

[...]

> There's no profit, IME.  There are only disadvantages: you are in
> effect fighting against the Emacs defaults, for reasons that are
> purely theoretical.

If I understand your meaning in both of these cases, I'll just note that
for the things we've been discussing here, I believe we've already had
complaints/requests from Debian users.  Whether that's significant
enough to warrant accommodation is another question, but that may not
leave the concerns theoretical, strictly speaking.

And for what it's worth, I can see both sides of the argument(s), i.e. I
can understand why upstream, it could be that on balance, those concerns
won't (and maybe shouldn't) be considered sufficient when balanced
against other considerations.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 12:57:54 -0500
> 
> Though note that for reasons we can elaborate on if desired, it might be
> easier for us if the default for that variable could also be set via a
> corresponding environment variable, but that's a separate question.
> 
> (For example, we have relevant emacs-related packages where the upstream
>  build process runs emacs a level or two "down" (subprocess-wise), and
>  so it'd be a lot less invasive if we could just set an environment
>  variable to change the .eln destination, instead of having to figure
>  out how to change each invocation of emacs in that package (and
>  maintain those changes across future upstream versions).

We could introduce such a variable, similarly to EMACSLOADPATH.  But
note several important considerations:

  . unlike with EMACSLOADPATH, the actual place where *.eln files will
    live is in a subdirectory of any directory in the list, due to the
    need of having them separately for different Emacs binaries and
    versions
  . EMACSLOADPATH is a mixed blessing: if you set it incorrectly, or
    forget that its value is inherited by subprocesses, you can
    completely hose an Emacs session, which is why we generally
    recommend against its use

> A second, and a separable question, is whether Debian should try to
> maintain system-level (root owned) .eln files the same way it does for
> .elc files.
> 
> I consider that an open question, and it could well be that the answer
> ends up being "no".  That's what we're trying to find out, and of course
> we have to begin by trying to communicate why that might be desirable.

Given the much more strict requirements for *.eln files wrt the target
architecture, certain crucial aspects of the Emacs binary, and the
contents and file name of the corresponding source file, my suggestion
is to start from "no" and only consider "yes" if you discover good
reasons through experience.  E.g., my eln-cache directory has no less
than 20 subdirectories, each one for a slightly different Emacs
version and configuration.  That might be somewhat extreme, given that
I work on development of and use several versions in parallel, but I
wouldn't be surprised to see several subdirectories for each of your
users, and even less surprised to see different subdirectories for
different users.  So I think the case for common compiled Lisp files
is much weaker for *.eln files than it is for *.elc files/

> It's certainly the case that if the consensus here (among the upstream
> developers) is that we shouldn't do that, and that future
> choices/changes aren't likely to take that use case into consideration,
> then we have our answer.

I only know that I didn't yet hear any good reason for rushing to
natively-compile optional Lisp packages.  That doesn't mean I'm dead
certain there could be no such good reasons, of course, just that I'd
like to see them described in enough detail to consider something
different from what was envisioned during Emacs 28 development.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: chad <yandros@gmail.com>,  Eli Zaretskii <eliz@gnu.org>,
>   tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 20:06:11 +0200
> 
> Rob Browning <rlb@defaultvalue.org> writes:
> 
> > A second, and a separable question, is whether Debian should try to
> > maintain system-level (root owned) .eln files the same way it does for
> > .elc files.
> 
> I think that makes sense.

Why do you think so?  A user who installs emacs-gtk and a user who
installs emacs-nox will need different *.eln files.  So Debian will
end up with a separate subdirectory for each configuration they
produce and for each Emacs version.  That's a lot of files and
directories, and high risk that many of them will never be used.
Also, a lot of problematic management of those directories. like the
decision when to delete them etc.  It is much easier to leave that to
users.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> No, I don't think the similar handling makes sense here.  The *.elc
> files are architecture- and configuration-independent, whereas the
> *.eln files are not.  E.g., the same foo.elc could be used by user A
> who runs Emacs 28 and by user B who runs Emacs 29.  But the
> corresponding *.eln files will be different, even though they were
> produced from the same foo.el.

Right, but for what it's worth, the Debian infrastructure is already set
up to, and would, maintain separate .eln files/trees for those two
cases, *if* we ever re-versioned the emacs packages.

Right now, there's only one GNU Emacs flavor in debian, we don't provide
versioned packages/flavors like emacs27 and emacs28 anymore.  Though we
could return to that again if it were ever deemed sufficiently valuable
to people.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> E.g., my eln-cache directory has no less than 20 subdirectories, each
> one for a slightly different Emacs version and configuration.

You're an Emacs developer, so that's to be expected.

But for normal users, the .eln files are neither more nor less specific
to an Emacs version than, say, the .pdmp file.  If Debian distributes a
specific Emacs version, it will be accompanied with the matching .pdmp
file -- and the matching .eln files, if that is what Debian decides to
do.

This is not complicated.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> I already did: the probability of different users to produce different
> *.eln files from the same *.el sources is high

As mentioned in another message, not in Debian's infrastructure.  It'd
be impossible (wrt the shared, system-level .elc and/or .eln files),
*if* we ended up deciding to pursue system-level .eln files.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: david@tethera.net, emacs-devel@gnu.org, akrl@sdf.org
> Date: Sun, 02 Oct 2022 13:12:01 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Theoretically, yes.  But I don't yet see why the particular context
> > described by Rob is different to the degree that it would need special
> > procedures.
> 
> While I'm not sure where we'll end up, I do think others may put more
> weight on some of the concerns.  For example, across the broader
> world/users-of-debian-and-derivative packages, I suspect there are some
> who care more about storage space.
> 
> As mentioned elswhere, we get regular requests to make emacs-nox even
> smaller.  And my eln-cache is currently 40MB, which is storage space
> we'd thought we might not have to duplicate across the emacs users on a
> system in the common case (Of course I know the duplication rate would
> vary depending on which users use which things, but unless the sets were
> disjoint, there'd be duplication that didn't seem necessary.)

IMO, this is actually an argument _against_ compiling everything AOT.

Whether the duplication is significant can only be decided based on
actual usage figures.  It is incorrect to assess this based on the
*.elc files, since those are independent of almost everything.
There's high probability of wrong decisions based on that analogy.
There are many factors that affect compatibility of *.eln files to
Emacs binaries; for example, it's enough to add or remove a primitive,
and you will need a whol;e new set of *.eln files.  Thus, it is quite
possible that duplication will be smaller and OTOH waste of disk space
due to unnecessarily compiled *.eln files will be higher than you
envision.  Only practice will show the real situation.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Why do you think so?  A user who installs emacs-gtk and a user who
> installs emacs-nox will need different *.eln files.

You can't install more than one of those at a time in Debian.  They're
mutually exclusive, at least right now, and have been since the variants
were added.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org,
>  akrl@sdf.org
> Date: Sun, 02 Oct 2022 13:21:54 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > But Emacs should not "crash" if the *.el files aren't available, it
> > should simply refuse to load any *.eln files and load the *.elc files
> > instead.  That produces many warnings, of course, but I hope your
> > users don't consider that "crashing".
> 
> I believe it was *crashing*.  I can't recall if that one was a segfault,
> or something a bit less drastic, but I'll try to remember to track it
> down later.

If it's a real crash, I'd appreciate bug reports, TIA.

> > It isn't a bug, but intended behavior.  If we want to remove this
> > dependency, some non-trivial ideas about reworking the current load
> > procedure should emerge.  I don't thin we have any such ideas at this
> > time.
> 
> OK, so we should consider that a hard dependency now, i.e. the emacs-el
> package can't be optional anymore, at least not on architectures where
> we can enable native compilation, and so probably just "everywhere" for
> simplicity, if nothing else.

More accurately, emacs-el cannot be optional when the installed Emacs
supports native compilation.  Emacs can still be built without native
compilation, and I presume those of your users who want the minimal
possible package will want that, since it removes several significant
dependencies, like libgccjit itself.  When Emacs is built without
native compilation, it can work without the *.el files.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> E.g., my eln-cache directory has no less than 20 subdirectories, each
>> one for a slightly different Emacs version and configuration.
>
> You're an Emacs developer, so that's to be expected.
>
> But for normal users, the .eln files are neither more nor less specific
> to an Emacs version than, say, the .pdmp file.  If Debian distributes a
> specific Emacs version, it will be accompanied with the matching .pdmp
> file -- and the matching .eln files, if that is what Debian decides to
> do.

With the current Debian arrangment, there would only ever be *one*
system-level .eln tree for the one installed Debian variant (emacs-nox,
emacs-lucid, or emacs-gtk).  Though of course the dir name hash might/would
change during each upgrade, but we already assume we have to rebuild all
the .elc files on each upgrade, and do (in dependency order), so that
shouldn't be a problem.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Whether the duplication is significant can only be decided based on
> actual usage figures.  It is incorrect to assess this based on the
> *.elc files, since those are independent of almost everything.
> There's high probability of wrong decisions based on that analogy.
> There are many factors that affect compatibility of *.eln files to
> Emacs binaries; for example, it's enough to add or remove a primitive,
> and you will need a whol;e new set of *.eln files.  Thus, it is quite
> possible that duplication will be smaller and OTOH waste of disk space
> due to unnecessarily compiled *.eln files will be higher than you
> envision.  Only practice will show the real situation.

I think I've probably made this clear elsewhere, but in case not, that
wouldn't be the case for Debian specifically.  There could/would only be
*one* eln tree for the whole system with the current packaging.

And that tree would be rebuilt when appropriate, in dependency order
(*if* we end up deciding to pursuse this).

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 13:32:02 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > My recommendation is to use the default JIT manner until and unless
> > actual problems are reported by users.
> 
> [...]
> 
> > There's no profit, IME.  There are only disadvantages: you are in
> > effect fighting against the Emacs defaults, for reasons that are
> > purely theoretical.
> 
> If I understand your meaning in both of these cases, I'll just note that
> for the things we've been discussing here, I believe we've already had
> complaints/requests from Debian users.  Whether that's significant
> enough to warrant accommodation is another question, but that may not
> leave the concerns theoretical, strictly speaking.

Please try to understand the nature of the complaints.  It is quite
possible that users simply use the (broken) analogy with the *.elc
files, because they misunderstand the quantitatively new aspects of
native-compilation.  There's more here than meets the eye, as has been
demonstrated even in this discussion.

Please don't hesitate to involve us in these discussions with your
complaining users, if you think we could help.

> And for what it's worth, I can see both sides of the argument(s), i.e. I
> can understand why upstream, it could be that on balance, those concerns
> won't (and maybe shouldn't) be considered sufficient when balanced
> against other considerations.

One other aspect that should be kept in mind is complexity.  The
introduction of the pdumper file in Emacs 27 and of native compilation
in Emacs 28 made Emacs deployment and startup code significantly more
complex, and as a side effect invalidated some of the deployment
methods people used to use, like some "clever" symlinking of the
binaries or the directories.  We are still learning these consequences
(although some of them already caused fixes in the code).  In this
situation, adding even more possible deployment method that upstream
should support risks making the startup code a maintenance burden.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 13:35:33 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > No, I don't think the similar handling makes sense here.  The *.elc
> > files are architecture- and configuration-independent, whereas the
> > *.eln files are not.  E.g., the same foo.elc could be used by user A
> > who runs Emacs 28 and by user B who runs Emacs 29.  But the
> > corresponding *.eln files will be different, even though they were
> > produced from the same foo.el.
> 
> Right, but for what it's worth, the Debian infrastructure is already set
> up to, and would, maintain separate .eln files/trees for those two
> cases, *if* we ever re-versioned the emacs packages.

Alas, there are many more than just two cases!

> Right now, there's only one GNU Emacs flavor in debian, we don't provide
> versioned packages/flavors like emacs27 and emacs28 anymore.

But a user who installs Emacs 28 doesn't need to nuke the Emacs 27
installation, does he?

And the same with a user who wants both emacs-nox, emacs-pgtk, and
emacs-lucid (say) installed on the same system.  Right?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Rob Browning <rlb@defaultvalue.org>,  yandros@gmail.com,
>   tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 20:37:20 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > E.g., my eln-cache directory has no less than 20 subdirectories, each
> > one for a slightly different Emacs version and configuration.
> 
> You're an Emacs developer, so that's to be expected.

As I wrote (and you elided), I don't expect to see 20 subdirectories
on other systems, but I wouldn't be surprised to see 5 or 10.

> But for normal users, the .eln files are neither more nor less specific
> to an Emacs version than, say, the .pdmp file.  If Debian distributes a
> specific Emacs version, it will be accompanied with the matching .pdmp
> file -- and the matching .eln files, if that is what Debian decides to
> do.

We are talking about optional packages, not about preloaded Lisp
files.  The preloaded ones indeed go together with the pdumper file,
but the optional ones don't.

> This is not complicated.

Famous last words.

Re: Suppressing native compilation (short and long term)

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Isn't that just `package-native-compile'?
>
> Oh, it's there!
> Wee!!

BTW, I think nil is the best default for that variable if and only if
`native-comp-async-report-warnings-errors' also defaults to nil.
Otherwise, users will have a rather scary warning buffer pop up in
the middle of trying to do work (or playing `M-x pong', more likely).

Maybe we could revisit the latter in time for Emacs 29, or at the latest
before switching native compilation to on by default (Emacs 30?).

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 13:37:49 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I already did: the probability of different users to produce different
> > *.eln files from the same *.el sources is high
> 
> As mentioned in another message, not in Debian's infrastructure.  It'd
> be impossible (wrt the shared, system-level .elc and/or .eln files),
> *if* we ended up deciding to pursue system-level .eln files.

Which IMO a significant limitation that is the direct result of such a
decision.  Right?  Whereas leaving it up to the users to produce *.el
in the JIT manner lifts this limitation.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: yandros@gmail.com, tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 13:41:32 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Why do you think so?  A user who installs emacs-gtk and a user who
> > installs emacs-nox will need different *.eln files.
> 
> You can't install more than one of those at a time in Debian.  They're
> mutually exclusive, at least right now, and have been since the variants
> were added.

Even if we are talking about two different users on the same system?
IOW, this is a system-wide restriction?  Isn't that too harsh?

And what about users who make changes to Emacs -- is that a legitimate
use case supported by Debian installations?

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> We are talking about optional packages, not about preloaded Lisp
> files.  The preloaded ones indeed go together with the pdumper file,
> but the optional ones don't.

I'm not sure what you mean by "optional packages" -- we're talking about
.eln files for the .elc files in the Emacs distribution.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: rlb@defaultvalue.org,  yandros@gmail.com,  tomas@tuxteam.de,
>   emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 21:01:53 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We are talking about optional packages, not about preloaded Lisp
> > files.  The preloaded ones indeed go together with the pdumper file,
> > but the optional ones don't.
> 
> I'm not sure what you mean by "optional packages" -- we're talking about
> .eln files for the .elc files in the Emacs distribution.

No, we are not.  We are talking about unbundled packages, like Magit.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

> At the top level, we wanted a way to avoid writing to HOME during
> packaging, testing, installs (in this case, it's the .eln files, now
> that we've enabled native compilation).
>
> That could be handled by some way to turn off native compilation, or by
> some way to comprehensively divert those writes to another location
> (e.g. temp dir).  Either is fine, though we'd originally thought the
> former might make things a bit easier.

Yeah, I think the former is both easier to implement and easier for
users.

So I'm thinking of introducing a user option like
`native-compile-inhibit', which will make Emacs skip the native-comp
machinery when loading .elc files.  It will default to nil, of course,
but perhaps it would be convenient to make it depend on an environment
variable like "NATIVE_COMPILE_INHIBIT"?  Then users (and the Debian
build system) could say "NATIVE_COMPILE_INHIBIT=true emacs ..." when
doing testing etc?  Would that fit your use case?

(This will be for Emacs 29, but you can cherry-pick this for Debian, if
that's something you want to do, and it will probably not affect
trampolines, since those are necessary for redefining functions.)

> Whether or not we can (or should) try to build system-level (root owned)
> .eln files along with the .elc files during package installs (as we
> already do for .elc files) is a separate question, and I think the more
> controversial one?

Not controversial from where I'm, er, sitting -- sounds both useful and
convenient to me.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  Eli Zaretskii
>  <eliz@gnu.org>,  david@tethera.net,  emacs-devel@gnu.org,  akrl@sdf.org
> Date: Sun, 02 Oct 2022 21:08:15 +0200
> 
> Rob Browning <rlb@defaultvalue.org> writes:
> 
> > At the top level, we wanted a way to avoid writing to HOME during
> > packaging, testing, installs (in this case, it's the .eln files, now
> > that we've enabled native compilation).
> >
> > That could be handled by some way to turn off native compilation, or by
> > some way to comprehensively divert those writes to another location
> > (e.g. temp dir).  Either is fine, though we'd originally thought the
> > former might make things a bit easier.
> 
> Yeah, I think the former is both easier to implement and easier for
> users.

The request that started this discussion was not from users, it was
from distributors.

If we want to consider providing (yet another) user option for
disabling native compilation, then we should:

  . understand why and in which situations they may need it
  . what exactly needs to be disabled (e.g.: do you want to disable
    loading the existing *.eln files?)
  . understand why the existing options don't already provide
    solutions

I object to introducing new options before we do the above and
understand well what we are talking about.

> So I'm thinking of introducing a user option like
> `native-compile-inhibit', which will make Emacs skip the native-comp
> machinery when loading .elc files.  It will default to nil, of course,
> but perhaps it would be convenient to make it depend on an environment
> variable like "NATIVE_COMPILE_INHIBIT"?  Then users (and the Debian
> build system) could say "NATIVE_COMPILE_INHIBIT=true emacs ..." when
> doing testing etc?  Would that fit your use case?

Their use case is not the use case of Emacs users.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> But a user who installs Emacs 28 doesn't need to nuke the Emacs 27
> installation, does he?

Yes, they do.  When the Debian emacs pacakges move from a 27.x version
to a 28.x version, upgrading will just move you to 28.  You can't have
more than one version of emacs installed via the packages right now.

(You could in the past (we had emacs25-nox, etc.), but we "unversioned"
 them for now because, in part, upstream backward compatbility with
 respect to anything "substantial" has been excellent for a long time.)

> And the same with a user who wants both emacs-nox, emacs-pgtk, and
> emacs-lucid (say) installed on the same system.  Right?

That's not allowed (and that in particular never has been in Debian and
(I assume) derivatives).  e.g. installing emacs-nox will automatically
force out/replace emacs-lucid if the former was installed, and then all
of the .elcs (and .elns if we go that direction) will be automatically
regenerated during the switch.

-- 
Rob Browning rlb @defaultvalue.org and @debian.org GPG as of 2011-07-10
E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A GPG as of 2002-11-03
14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Even if we are talking about two different users on the same system?
> IOW, this is a system-wide restriction?  Isn't that too harsh?

The available Debian packages are a balance, intended to cover a broad
set of common cases, i.e. Emacs without X, Emacs with the Lucid toolkit
(because of, if nothing else, gtk issues), and Emacs with the GTK
toolkit.

You can only have one of them installed at a time, and you can
(currently) only have one major version installed at a time.

  https://packages.debian.org/search?keywords=emacs

We could of course try to accommodate multiple major versions (we did
for a good while), and/or multiple simultaneous variants (nox, lucid,
gtk), but we'd need to feel like the additional complexity and archive
space (multiplied across the architecture-dependent packages
(emacs-bin-common, etc.)[1]) was worth it for a large enough audience.

  [1] https://buildd.debian.org/status/package.php?p=emacs

> And what about users who make changes to Emacs -- is that a legitimate
> use case supported by Debian installations?

I'd say that up to a point you can, and I have symlinked the relevant
.el files into a ~/ directory, made sure that it's in my load-path, and
then made adjustments, but past a certain point, I'd say that you'd want
to switch to building Emacs yourself.

Because at that point, you're perhaps no longer in the target audience
for the Debian packages, or at least non on that particular machine
(i.e. you might be fine with the Debian packages on most of your
machines, servers, etc. but want to make much more extensive changes to
Emacs on some others).

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Please try to understand the nature of the complaints.

For what it's worth, one of the complaints was:

  I do not want emacs suddenly swamping the CPU on my laptop
  unpredictably.  I want all this work (in the common case) done while I
  know I'm connected to power, before I head out.

...and we have for a very long time, perhaps as long as I've been
working with the packaging, had users who are concerned with disk space
usage, and ask for smaller package footprints.

Those concerns may not carry the day, but they do exist.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> We are talking about optional packages, not about preloaded Lisp
> files.  The preloaded ones indeed go together with the pdumper file,
> but the optional ones don't.

Debian always recompiles separately all the .el files in third party
packages (installed vi Debian) for all the installed Emacsen.
IOW if you have `emacs24`, `xemacs`, and `emacs25` installed, you'll
have 3 sets of .elc files for every ELisp package installed via Debian's
package manager.

So the `.eln` files aren't really more specific than the `.elc` in
that setting (except for the situation where you have `emacs-lucid` and
`emacs-gtk` and `emacs-nox` installed where these (unnecessarily)
require different `.eln` files but will share the same `elc` files).


        Stefan

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Which IMO a significant limitation that is the direct result of such a
> decision.  Right?  Whereas leaving it up to the users to produce *.el
> in the JIT manner lifts this limitation.

Maybe my other mail about the packaging tradeoffs helps here, but if
not, I can try to elaborate further.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So I'm thinking of introducing a user option like
> `native-compile-inhibit', which will make Emacs skip the native-comp
> machinery when loading .elc files.  It will default to nil, of course,
> but perhaps it would be convenient to make it depend on an environment
> variable like "NATIVE_COMPILE_INHIBIT"?  Then users (and the Debian
> build system) could say "NATIVE_COMPILE_INHIBIT=true emacs ..." when
> doing testing etc?  Would that fit your use case?

It might?  I'd also suggest adding an EMACS_... prefix
to the variable name (of course I'd suggest that for all of any
program's variables).

Though I'm not positive because if we still really need compilation for
the trampolines(?), then we may need to just follow the "redirect to a
tempdir" approach, which is also fine, and then the ability to disable
native compilation might not be relevant to the packaging.

In either case, we might ideally want an EMACS_... var to make
propagation a bit easier if that sounds reasonable.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> So the `.eln` files aren't really more specific than the `.elc` in
> that setting (except for the situation where you have `emacs-lucid` and
> `emacs-gtk` and `emacs-nox` installed where these (unnecessarily)
> require different `.eln` files but will share the same `elc` files).

Apparently that exception doesn't apply either, since Debian doesn't
allow simultaneously installing those different flavors :-)
[ Funny, I never noticed it in all those years using Debian.  ]


        Stefan

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

> Though I'm not positive because if we still really need compilation for
> the trampolines(?), then we may need to just follow the "redirect to a
> tempdir" approach, which is also fine, and then the ability to disable
> native compilation might not be relevant to the packaging.

I'm not sure we need to save the trampolines at all (in this
don't-do-native-compilation configuration) -- Andrea probably knows.
Andrea, would it be possible to just create and use the trampolines
without writing them to a file?

> In either case, we might ideally want an EMACS_... var to make
> propagation a bit easier if that sounds reasonable.

Sure.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

On Sun, Oct 02, 2022 at 08:01:58PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 2 Oct 2022 18:27:05 +0200
> > From: tomas@tuxteam.de
> > Cc: emacs-devel@gnu.org

[...]

> > Hm. I don't want to steal your time more, but... if you could illustrate
> > why, I'd eager to learn.
> 
> I already did: the probability of different users to produce different
> *.eln files from the same *.el sources is high, and disk space is
> cheap.

Thanks

Cheers
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Sun 02 Oct 2022 at 08:57AM +03, Eli Zaretskii wrote:

> The reasons which you mention in favor of AOT native compilation don't
> sound serious enough: I see no problem in having the compilation
> happen only when it's needed in those cases.  Battery consumption
> doesn't seem very relevant, because JIT compilation will happen when
> the system is used, not when it sleeps.  And it is entirely not
> guaranteed that by compiling everything you will save power in the
> long run, because there are good chances you will be compiling stuff
> that won't be used for a long time.  Without quantitative data of
> long-term power usage on which to base the conclusions, I don't see
> why you should a-priori assume that compiling everything from the
> get-go should use less power.  Same goes for disk space by multiple
> users.

The point with battery consumption is not about running vs. standby.
The issue is that while users expect that running apt-get will drain the
battery, they expect that once apt-get is done, the only battery-hungry
processes are ones they start themselves.  Laptop users typically avoid
running apt-get without mains power plugged in.  If I do an 'apt-get
upgrade' then afterwards my CPU will be churning away compiling addon
packages, so I can't just unplug.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Sun 02 Oct 2022 at 12:53PM -04, Stefan Monnier wrote:

> FWIW, I'm not convinced it's really useful in Debian's `emacs` package
> to eagerly native compile all the bundled .elc files, but I think it
> does make a lot of sense for ELisp packages installed separately.

Could you elaborate on this point?

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Sun 02 Oct 2022 at 07:11PM +03, Eli Zaretskii wrote:

> I submit that those reasons were most probably derived from a broken
> analogy with the *.elc files and with byte-compilation in general.
> Not from actual usage experience.  Native compilation looks
> deceptively similar to byte compilation, but it isn't.  So if
> producing the *.eln files seems to contradict some Debian rules and
> procedures, my suggestion is to talk to the upstream project, before
> inventing solutions, because of 2 considerations:
>
>   . the problems may not be real ones, only perceived ones
>   . the upstream codebase might already provide solutions

The battery usage thing is real, from my own user experience.

I have to remember not to unplug my laptop until the load indicator in
my status bar has dropped back down, else the battery won't last as long
as I expect it to.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> If we want to consider providing (yet another) user option for
> disabling native compilation, then we should:

I asked what the user option to disable native compilation was, but
didn't get an answer, and here you say "(yet another)", so...  what is
the current user option to disable native compilation?

>   . understand why and in which situations they may need it

Doing repeatable testing is one obvious situation.

>   . what exactly needs to be disabled (e.g.: do you want to disable
>     loading the existing *.eln files?)

I don't think anybody has suggested the "e.g." portion, so I'm wondering
why you ask?

>   . understand why the existing options don't already provide
>     solutions

See first paragraph.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I asked what the user option to disable native compilation was, but
> didn't get an answer, and here you say "(yet another)", so...  what is
> the current user option to disable native compilation?

Looking at the code, I can't see any user options to disable automatic
compilation of .elc files.  However, there's the confusingly named
`native-comp-deferred-compilation' variable which does the trick.

And looking at this a bit more, having a user option for this won't
quite work, because user options are set in .emacs, and .emacs may
trigger loading stuff before handling user options, so it'll have to
remain a variable.

But should perhaps be renamed to something more understandable, or be
deprecated in favour of a `inhibit-native-compilation' variable?

In any case, initialising this variable from an environment variable
like EMACS_INHIBIT_NATIVE_COMPILATION seems totally trivial.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

AFAICT so far I've heard of two reasons to "disable automatic native
compilation":

A) Avoid the "uncontrolled" (mostly in terms of *when* it happens)
   resource usage associated with it (e.g. battery).

B) Avoid writing to $HOME.

For (A), I think that (setq native-comp-deferred-compilation nil) seems
perfectly sufficient (you can set it early in your (early) init file and if
every blue moon some native compilation still happens because of a file
loaded before that `setq` it shouldn't be a big deal).

For (B) I seem to remember that it's a problem that goes a bit further
than just native compilation (we've had bug reports about other files
being silently written to ~/.emacs.d in the past and I'd be surprised if
there aren't still many such cases, especially if we consider third
party packages).  So if avoiding all files under $HOME is important,
I suspect the only reliable answer is to set $HOME elsewhere (and if
you run Emacs under `su`, make sure you use `su -`).

This said, we generally do make efforts to try and avoid writing
silently to ~/.emacs.d, so maybe we should rethink our choice of
~/.emacs.d/eln-cache as the default location.
E.g. maybe when run as root, we should write .eln files somewhere under
something like /var/cache?  And maybe when not run as root we should
favor something underneath ~/.cache?


        Stefan "who finds the idea of running Emacs as root a bit scary"

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> AFAICT so far I've heard of two reasons to "disable automatic native
> compilation":
>
> A) Avoid the "uncontrolled" (mostly in terms of *when* it happens)
>    resource usage associated with it (e.g. battery).
>
> B) Avoid writing to $HOME.
>
> For (A), I think that (setq native-comp-deferred-compilation nil) seems
> perfectly sufficient (you can set it early in your (early) init file and if
> every blue moon some native compilation still happens because of a file
> loaded before that `setq` it shouldn't be a big deal).

I've now introduced the new inhibit-native-compilation variable and
deprecated the native-comp-deferred-compilation variable, and also
initialised stuff from EMACS_INHIBIT_NATIVE_COMPILATION.

(And made trampolines not write to the cache in this case.)

> This said, we generally do make efforts to try and avoid writing
> silently to ~/.emacs.d, so maybe we should rethink our choice of
> ~/.emacs.d/eln-cache as the default location.

I think that's a fine location in general, and now there's an
understandable and documented way to make that not happen, I think it's
even better.

> E.g. maybe when run as root, we should write .eln files somewhere under
> something like /var/cache?  And maybe when not run as root we should
> favor something underneath ~/.cache?
>
>         Stefan "who finds the idea of running Emacs as root a bit scary"

I run Emacs as root all the time.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Lars Ingebrigtsen [2022-10-03 15:29:40] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> AFAICT so far I've heard of two reasons to "disable automatic native
>> compilation":
>>
>> A) Avoid the "uncontrolled" (mostly in terms of *when* it happens)
>>    resource usage associated with it (e.g. battery).
>>
>> B) Avoid writing to $HOME.
>>
>> For (A), I think that (setq native-comp-deferred-compilation nil) seems
>> perfectly sufficient (you can set it early in your (early) init file and if
>> every blue moon some native compilation still happens because of a file
>> loaded before that `setq` it shouldn't be a big deal).
>
> I've now introduced the new inhibit-native-compilation variable and
> deprecated the native-comp-deferred-compilation variable, and also
> initialised stuff from EMACS_INHIBIT_NATIVE_COMPILATION.
> (And made trampolines not write to the cache in this case.)

AFAICT for the case (A), we *do* want to save trampolines for the next
time around, and those users probably do want to use native compilation
(e.g. they would likely set `package-native-compile` to non-nil), just
not the deferred kind, so the var name is a bit odd for them.


        Stefan

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> AFAICT for the case (A), we *do* want to save trampolines for the next
> time around, and those users probably do want to use native compilation
> (e.g. they would likely set `package-native-compile` to non-nil), just
> not the deferred kind, so the var name is a bit odd for them.

The trampolines are very fast to make, so creating them once per Emacs
session doesn't seem like a problem.  If that turns out to be an issue,
we can tweak the variable.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Lars Ingebrigtsen [2022-10-03 16:09:35] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> AFAICT for the case (A), we *do* want to save trampolines for the next
>> time around,
>
> The trampolines are very fast to make, so creating them once per Emacs
> session doesn't seem like a problem.  If that turns out to be an issue,
> we can tweak the variable.

Yes, it's probably not a big deal indeed.

But this one is more annoying I think:

>> and those users probably do want to use native compilation
>> (e.g. they would likely set `package-native-compile` to non-nil), just
>> not the deferred kind, so the var name is a bit odd for them.

`inhibit-native-compilation` sounds like it really prevents all forms of
native compilation, whether "deferred" or not.

E.g. going by its name, I'd argue that it's a bug if package.el does
native-compile the files when both `package-native-compile` when
`inhibit-native-compilation` are non-nil.


        Stefan

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> and those users probably do want to use native compilation
>>> (e.g. they would likely set `package-native-compile` to non-nil), just
>>> not the deferred kind, so the var name is a bit odd for them.
>
> `inhibit-native-compilation` sounds like it really prevents all forms of
> native compilation, whether "deferred" or not.

We can expand the variable name, but I couldn't think of a good name.
And "deferred" doesn't convey anything.

> E.g. going by its name, I'd argue that it's a bug if package.el does
> native-compile the files when both `package-native-compile` when
> `inhibit-native-compilation` are non-nil.

But not by going by the documentation of the variable.  😀

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Lars Ingebrigtsen [2022-10-03 16:45:22] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> and those users probably do want to use native compilation
>>>> (e.g. they would likely set `package-native-compile` to non-nil), just
>>>> not the deferred kind, so the var name is a bit odd for them.
>> `inhibit-native-compilation` sounds like it really prevents all forms of
>> native compilation, whether "deferred" or not.
> We can expand the variable name, but I couldn't think of a good name.
> And "deferred" doesn't convey anything.

Lazy?  Auto(matic)?  Ondemand?

>> E.g. going by its name, I'd argue that it's a bug if package.el does
>> native-compile the files when both `package-native-compile` when
>> `inhibit-native-compilation` are non-nil.
> But not by going by the documentation of the variable.  😀

Sorry, my dictionary doesn't know that word.  What's "documentation"?
Does it have to do with some Norse mythology, maybe?


        Stefan

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> We can expand the variable name, but I couldn't think of a good name.
>> And "deferred" doesn't convey anything.
>
> Lazy?  Auto(matic)?  Ondemand?

`inhibit-lazy-native-compilation' sounds like it makes the compilation
less lazy.  `inhibit-automatic-native-compilation' might work.

> Sorry, my dictionary doesn't know that word.  What's "documentation"?
> Does it have to do with some Norse mythology, maybe?

I think the concept was invented by Loki, the god of mischief and lies.

RE: [External] : Re: Suppressing native compilation (short and long term)

[Drew Adams]

> Sorry, my dictionary doesn't know that word.  What's "documentation"?
> Does it have to do with some Norse mythology, maybe?

Doctor U. Mentation
___

Noun: mentation

The process of using your mind to consider something carefully.

 - WordWeb

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 14:57:49 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Please try to understand the nature of the complaints.
> 
> For what it's worth, one of the complaints was:
> 
>   I do not want emacs suddenly swamping the CPU on my laptop
>   unpredictably.  I want all this work (in the common case) done while I
>   know I'm connected to power, before I head out.
> 
> ...and we have for a very long time, perhaps as long as I've been
> working with the packaging, had users who are concerned with disk space
> usage, and ask for smaller package footprints.

But Emacs does that all the time: there are many features that invoke
sub-processes and many more features that write to the disk.  I never
heard anyone complaining seriously about that, and I'm quite sure many
users don't even know which Emacs commands invoke subprocesses under
the hood.

So I'm not sure these complaints are based on real problems.  Did
anyone compare the "sudden swamp of the CPU" caused by JIT native
compilation with what happens with other commands that invoke
subprocesses?  If so, did they present some quantitative data?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: Rob Browning <rlb@defaultvalue.org>,  monnier@iro.umontreal.ca,
>   david@tethera.net,  emacs-devel@gnu.org,  akrl@sdf.org
> Date: Sun, 02 Oct 2022 16:51:12 -0700
> 
> The point with battery consumption is not about running vs. standby.
> The issue is that while users expect that running apt-get will drain the
> battery, they expect that once apt-get is done, the only battery-hungry
> processes are ones they start themselves.

How does this work in practice?  I mean, what exactly do you mean by
"processes they start themselves"?  You mean, you know by heart each
Emacs command that invokes a subprocess, and avoid to invoke them all
unless the laptop is plugged in?  I don't believe this is practical,
because even "C-x C-f" invokes a a subprocess when you visit a file
under VCS (which I guess happens a lot to the likes of you and me).
And how can a person avoid all of those commands?  What am I missing?

> Laptop users typically avoid running apt-get without mains power
> plugged in.  If I do an 'apt-get upgrade' then afterwards my CPU
> will be churning away compiling addon packages, so I can't just
> unplug.

Did you actually try to see that "churning away" in practice, did you
look at the time this typically takes, the resources (battery and
otherwise) it consumes, etc.?  If so, can you share the data: how many
*.eln files were produced in how many minutes, how much battery did
that consume?

You see, I have my data, and it seems to indicate a very short period
of initial compilations for a modest consumption of resources.  (This
isn't a laptop, but I'm acutely aware of my system's load at all
times, and have it on the mode line, so I know what kind of
computation is going on during that time.)  The data I see here
doesn't look like it should be a problem for anyone, as long as files
are compiled only as they are used. In my case, for example, I see
maybe half a dozen *.eln files compiled after the initial startup.  I
expect to see that on any machine where a user has steady usage
patterns -- the compilation flats out very quickly.  I cannot imagine
how will your CPU churn out for any significant time after an upgrade.
A typical compilation of a single .el file is usually very fast,
exceptions are extremely rare.  And I barely notice the effect of that
on system responsiveness.

So I could imagine that many people perhaps base these judgments on
something they didn't actually try, but just imagined to themselves,
and imagined incorrectly, or based their opinions on some other
compilations of different languages in different environments.

I urge people to actually try this and see what it does, before making
up your minds.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 16:58:34 -0700
> 
> The battery usage thing is real, from my own user experience.
> 
> I have to remember not to unplug my laptop until the load indicator in
> my status bar has dropped back down, else the battery won't last as long
> as I expect it to.

When I start an Emacs session for a new version (which needs to
recompile all the packages I use, because my sessions are restored by
desktop.el), I see about 100 Lisp files compiled into *.eln inside of
1 minute, and then another bunch of 100 inside another minute (with 10
minutes between these two groups).  Does this strike you as a long
time to avoid unplugging?  I hope your battery can hold more than a
few minutes.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> But Emacs does that all the time: there are many features that invoke
> sub-processes and many more features that write to the disk.  I never
> heard anyone complaining seriously about that, and I'm quite sure many
> users don't even know which Emacs commands invoke subprocesses under
> the hood.

FWIW, during the stealth jit-lock discussion, several people mentioned the
battery impact.

And the issue is not subprocesses per se, but it's extra processing that
happens outside of the control of the user.

> So I'm not sure these complaints are based on real problems.  Did
> anyone compare the "sudden swamp of the CPU" caused by JIT native
> compilation with what happens with other commands that invoke
> subprocesses?  If so, did they present some quantitative data?

Probably not, no.  It's likely mostly a question of perception, so if we
could make it completely invisible the "problem" would disappear :-)
But the fact that lazy native compilation tends to pop up warnings (and
to make matters worse, it does so ... without warning) makes it very
much visible instead.


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Mon, 03 Oct 2022 15:29:40 +0200
> 
> I've now introduced the new inhibit-native-compilation variable and
> deprecated the native-comp-deferred-compilation variable, and also
> initialised stuff from EMACS_INHIBIT_NATIVE_COMPILATION.
> 
> (And made trampolines not write to the cache in this case.)

Thanks a lot for ignoring everything I wrote during the last two days!

Makes one wonder what kind of development community this is.  And what
am I doing here, anyway.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Mon, 03 Oct 2022 16:09:35 +0200
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > AFAICT for the case (A), we *do* want to save trampolines for the next
> > time around, and those users probably do want to use native compilation
> > (e.g. they would likely set `package-native-compile` to non-nil), just
> > not the deferred kind, so the var name is a bit odd for them.
> 
> The trampolines are very fast to make, so creating them once per Emacs
> session doesn't seem like a problem.  If that turns out to be an issue,
> we can tweak the variable.

I think that entire changeset should be reverted.  It is not well
thought out, and ion some aspects downright dangerous.  The
environment variable is especially egregious: it will affect all the
sub-process Emacsen as well, something that will bite users, a lesson
we've learned long ago with the likes of EMACSLOADPATH.

Such changes should not be installed without a great deal of careful
thought and consideration of the different use cases.  Which I tried
to explain all the way, but apparently to deaf ears.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Rob Browning <rlb@defaultvalue.org>,  tomas@tuxteam.de,
>   emacs-devel@gnu.org
> Date: Mon, 03 Oct 2022 12:39:44 -0400
> 
> > But Emacs does that all the time: there are many features that invoke
> > sub-processes and many more features that write to the disk.  I never
> > heard anyone complaining seriously about that, and I'm quite sure many
> > users don't even know which Emacs commands invoke subprocesses under
> > the hood.
> 
> FWIW, during the stealth jit-lock discussion, several people mentioned the
> battery impact.

Yes.  And jit-stealth is different, in that it takes a much longer
time for it to become quiet, because it works in small chinks, and
only when Emacs is actually idle.  JIT native-compilation is much
faster, and also uses several execution units of the CPU in parallel.

> And the issue is not subprocesses per se, but it's extra processing that
> happens outside of the control of the user.

How do you mean "outside of the control of the user"?  The user causes
it by loading a feature.  If no new features are loaded, no
native-compilation will happen.  How is that different from any other
command that uses a subprocess under the hood?

> > So I'm not sure these complaints are based on real problems.  Did
> > anyone compare the "sudden swamp of the CPU" caused by JIT native
> > compilation with what happens with other commands that invoke
> > subprocesses?  If so, did they present some quantitative data?
> 
> Probably not, no.  It's likely mostly a question of perception, so if we
> could make it completely invisible the "problem" would disappear :-)
> But the fact that lazy native compilation tends to pop up warnings (and
> to make matters worse, it does so ... without warning) makes it very
> much visible instead.

On my system, I don't see any warnings.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I think that entire changeset should be reverted.  It is not well
> thought out, and ion some aspects downright dangerous.  The
> environment variable is especially egregious: it will affect all the
> sub-process Emacsen as well, something that will bite users, a lesson
> we've learned long ago with the likes of EMACSLOADPATH.

How is setting EMACS_INHIBIT_NATIVE_COMPILE dangerous?

> Such changes should not be installed without a great deal of careful
> thought and consideration of the different use cases.  Which I tried
> to explain all the way, but apparently to deaf ears.

The discussion has been going for apparently weeks no with no progress.
It's easier to discuss code when there's code to discuss.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

[Full disclosure: evidently, none of what I write below matters, so
feel free to ignore.]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: larsi@gnus.org, yandros@gmail.com, tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 14:50:58 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Even if we are talking about two different users on the same system?
> > IOW, this is a system-wide restriction?  Isn't that too harsh?
> 
> The available Debian packages are a balance, intended to cover a broad
> set of common cases, i.e. Emacs without X, Emacs with the Lucid toolkit
> (because of, if nothing else, gtk issues), and Emacs with the GTK
> toolkit.
> 
> You can only have one of them installed at a time, and you can
> (currently) only have one major version installed at a time.
> 
>   https://packages.debian.org/search?keywords=emacs

That's strange to hear.  Even MS-Windows allows per-user variations of
PATH and per-use environment variables.  It is strange to learn that
Debian doesn't.

> > And what about users who make changes to Emacs -- is that a legitimate
> > use case supported by Debian installations?
> 
> I'd say that up to a point you can, and I have symlinked the relevant
> .el files into a ~/ directory, made sure that it's in my load-path, and
> then made adjustments, but past a certain point, I'd say that you'd want
> to switch to building Emacs yourself.

If you want to support even just installing a different minor version,
it will already require to recompile all of the *.eln files.

But okay, yes, if Debian users live under such severe restrictions,
then the case for having user-specific *.eln directories becomes
weaker.  But I still don't see that it is weaker than disallowing
native compilation at run time.

And I guess now I'm confused what is it exactly that you'd want to
achieve.  Do you want to disable native compilation, or do you want to
have all *.eln files in a shared location?  Because it seems to me you
said you wanted both, and I don't see how both could be reconciled.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: rlb@defaultvalue.org,  monnier@iro.umontreal.ca,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Mon, 03 Oct 2022 11:12:22 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If we want to consider providing (yet another) user option for
> > disabling native compilation, then we should:
> 
> I asked what the user option to disable native compilation was, but
> didn't get an answer, and here you say "(yet another)", so...  what is
> the current user option to disable native compilation?

You haven't bothered waiting for my answer, so I won't.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: monnier@iro.umontreal.ca,  rlb@defaultvalue.org,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Mon, 03 Oct 2022 19:39:04 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think that entire changeset should be reverted.  It is not well
> > thought out, and ion some aspects downright dangerous.  The
> > environment variable is especially egregious: it will affect all the
> > sub-process Emacsen as well, something that will bite users, a lesson
> > we've learned long ago with the likes of EMACSLOADPATH.
> 
> How is setting EMACS_INHIBIT_NATIVE_COMPILE dangerous?

I just explained it, above.

> > Such changes should not be installed without a great deal of careful
> > thought and consideration of the different use cases.  Which I tried
> > to explain all the way, but apparently to deaf ears.
> 
> The discussion has been going for apparently weeks no with no progress.

"Weeks"?  Boy, I guess time really flies when one's enjoying: the
first message of this thread was posted less than a week ago:

  https://lists.gnu.org/archive/html/emacs-devel/2022-09/msg01911.html

And it really started just 2 days ago:

  https://lists.gnu.org/archive/html/emacs-devel/2022-10/msg00015.html

But whatever.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1440 bytes --]

On Mon, Oct 03, 2022 at 08:41:22PM +0300, Eli Zaretskii wrote:
> [Full disclosure: evidently, none of what I write below matters, so
> feel free to ignore.]

Now I think this is a bit unfair. You have a string opinion, and
I think it shhould be respected, but the others have, too.


> > You can only have one of them installed at a time, and you can
> > (currently) only have one major version installed at a time.
> > 
> >   https://packages.debian.org/search?keywords=emacs
> 
> That's strange to hear.  Even MS-Windows allows per-user variations of
> PATH and per-use environment variables.  It is strange to learn that
> Debian doesn't.

Note that this is only as far as the Debian packaging system is
concerned. Per-user you can have as many Emacsen installed as you
want -- the Debian packaging system doesn't know about them.

Alternatively, you can have a system-local install (going to
/usr/local) besides the Debian-installed one without them interfering
(that's the setup I use).

This is actually what I love in Debian: you have single applications
you care about which you install from souces and take over the hand
feeding, and for the rest you get hight quality packaging with pretty
low noise.

Emacs, for me, happens to be one of those care-and-feeding applications,
but for others, the choice might be different. It's them whom the
Debian-packaged Emacsen are for.

Cheers
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Mon 03 Oct 2022 at 09:07AM -04, Stefan Monnier wrote:

> AFAICT so far I've heard of two reasons to "disable automatic native
> compilation":
>
> A) Avoid the "uncontrolled" (mostly in terms of *when* it happens)
>    resource usage associated with it (e.g. battery).
>
> B) Avoid writing to $HOME.
>
> For (A), I think that (setq native-comp-deferred-compilation nil) seems
> perfectly sufficient (you can set it early in your (early) init file and if
> every blue moon some native compilation still happens because of a file
> loaded before that `setq` it shouldn't be a big deal).

We'd also like to be able to compile and install system-wide -- it would
be good to be able to benefit from native compilation without it
happening by surrpise.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Mon 03 Oct 2022 at 07:19PM +03, Eli Zaretskii wrote:

>> From: Sean Whitton <spwhitton@spwhitton.name>
>> Cc: Rob Browning <rlb@defaultvalue.org>,  monnier@iro.umontreal.ca,
>>   david@tethera.net,  emacs-devel@gnu.org,  akrl@sdf.org
>> Date: Sun, 02 Oct 2022 16:51:12 -0700
>>
>> The point with battery consumption is not about running vs. standby.
>> The issue is that while users expect that running apt-get will drain the
>> battery, they expect that once apt-get is done, the only battery-hungry
>> processes are ones they start themselves.
>
> How does this work in practice?  I mean, what exactly do you mean by
> "processes they start themselves"?  You mean, you know by heart each
> Emacs command that invokes a subprocess, and avoid to invoke them all
> unless the laptop is plugged in?  I don't believe this is practical,
> because even "C-x C-f" invokes a a subprocess when you visit a file
> under VCS (which I guess happens a lot to the likes of you and me).
> And how can a person avoid all of those commands?  What am I missing?

Native compilation is significantly more CPU intensive than any of these
other things, though.

> Did you actually try to see that "churning away" in practice, did you
> look at the time this typically takes, the resources (battery and
> otherwise) it consumes, etc.?  If so, can you share the data: how many
> *.eln files were produced in how many minutes, how much battery did
> that consume?
>
> You see, I have my data, and it seems to indicate a very short period
> of initial compilations for a modest consumption of resources.  (This
> isn't a laptop, but I'm acutely aware of my system's load at all
> times, and have it on the mode line, so I know what kind of
> computation is going on during that time.)  The data I see here
> doesn't look like it should be a problem for anyone, as long as files
> are compiled only as they are used. In my case, for example, I see
> maybe half a dozen *.eln files compiled after the initial startup.  I
> expect to see that on any machine where a user has steady usage
> patterns -- the compilation flats out very quickly.  I cannot imagine
> how will your CPU churn out for any significant time after an upgrade.
> A typical compilation of a single .el file is usually very fast,
> exceptions are extremely rare.  And I barely notice the effect of that
> on system responsiveness.
>
> So I could imagine that many people perhaps base these judgments on
> something they didn't actually try, but just imagined to themselves,
> and imagined incorrectly, or based their opinions on some other
> compilations of different languages in different environments.
>
> I urge people to actually try this and see what it does, before making
> up your minds.

I have actually tried it, for more than a year now, but admittedly I do
not have hard data.  Thank you for sharing yours.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Mon 03 Oct 2022 at 07:24PM +03, Eli Zaretskii wrote:

>> From: Sean Whitton <spwhitton@spwhitton.name>
>> Cc: tomas@tuxteam.de,  emacs-devel@gnu.org
>> Date: Sun, 02 Oct 2022 16:58:34 -0700
>>
>> The battery usage thing is real, from my own user experience.
>>
>> I have to remember not to unplug my laptop until the load indicator in
>> my status bar has dropped back down, else the battery won't last as long
>> as I expect it to.
>
> When I start an Emacs session for a new version (which needs to
> recompile all the packages I use, because my sessions are restored by
> desktop.el), I see about 100 Lisp files compiled into *.eln inside of
> 1 minute, and then another bunch of 100 inside another minute (with 10
> minutes between these two groups).  Does this strike you as a long
> time to avoid unplugging?  I hope your battery can hold more than a
> few minutes.

Sounds like your machine is just a lot better than mine.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

>> FWIW, during the stealth jit-lock discussion, several people mentioned the
>> battery impact.
> Yes.  And jit-stealth is different, in that it takes a much longer
> time for it to become quiet, because it works in small chinks, and
> only when Emacs is actually idle.  JIT native-compilation is much
> faster, and also uses several execution units of the CPU in parallel.

Indeed, it's quite different.  I think even more important than the
above is the fact that it's a one-time thing only (for typical use cases
at least), whereas the jit-lock-stealth cost was paid over and over
again after every buffer modification.

>> And the issue is not subprocesses per se, but it's extra processing that
>> happens outside of the control of the user.
> How do you mean "outside of the control of the user"?  The user causes
> it by loading a feature.

When a user loads a feature, this request doesn't say explicitly "and
native compile the file".  And there is no direct way to "just load this
package without compiling it".  So the compilation is not really under
the control of the user.

I don't think users know, when they load a feature, if it's the first
time they loaded this feature in this version of Emacs, so by and large
they can't predict when the compilation does happen.

To be clear: I don't think it's a problem.  I just think it's part of
the reason for the reaction.

> How is that different from any other command that uses a subprocess
> under the hood?

It's different in that for those other commands, the users cannot get
the result they want without running that subprocess, so even if they're
not fully aware of it, they do have some vague idea it's inevitably
going to happen.

In contrast, the compilation is not indispensable, (so much so that the
past it never happened).

>> Probably not, no.  It's likely mostly a question of perception, so if we
>> could make it completely invisible the "problem" would disappear :-)
>> But the fact that lazy native compilation tends to pop up warnings (and
>> to make matters worse, it does so ... without warning) makes it very
>> much visible instead.
>
> On my system, I don't see any warnings.

That's presumably because you stick to the bundled packages (where we've
managed to keep a "no compilation warnings" state for the last few
years) or the few third party packages which are careful enough to
eliminate all warnings.

The rise of ELPA has removed most of the cases where native compilation
will miscompile files (because ELPA forces the .el files to be compiled
in a "standard" way rather than using ad-hoc Makefile rules which the
native compiler cannot and does not try to reproduce), but compilation
warnings are still very common (the situation is improving, tho, and
arguably the warnings popping up out of the blue during lazy native
compilation may prove to be a good way to convince more package authors
to clean up their act).


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Mon, 3 Oct 2022 20:17:53 +0200
> From: tomas@tuxteam.de
> Cc: Rob Browning <rlb@defaultvalue.org>, larsi@gnus.org, yandros@gmail.com,
> 	emacs-devel@gnu.org
> 
> On Mon, Oct 03, 2022 at 08:41:22PM +0300, Eli Zaretskii wrote:
> > [Full disclosure: evidently, none of what I write below matters, so
> > feel free to ignore.]
> 
> Now I think this is a bit unfair. You have a string opinion, and
> I think it shhould be respected, but the others have, too.

You misunderstood the intent.

> > That's strange to hear.  Even MS-Windows allows per-user variations of
> > PATH and per-use environment variables.  It is strange to learn that
> > Debian doesn't.
> 
> Note that this is only as far as the Debian packaging system is
> concerned. Per-user you can have as many Emacsen installed as you
> want -- the Debian packaging system doesn't know about them.

Windows installers allow you to install into a directory of your
choice, and some will even set PATH to point to there; if not, you can
do it manually.

> Alternatively, you can have a system-local install (going to
> /usr/local) besides the Debian-installed one without them interfering
> (that's the setup I use).

I don't understand what that means.  Do you mean that you can subvert
the general non-support by Debian packages of several Emacs versions
installed on the same system?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: rlb@defaultvalue.org,  monnier@iro.umontreal.ca,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Mon, 03 Oct 2022 11:23:15 -0700
> 
> > How does this work in practice?  I mean, what exactly do you mean by
> > "processes they start themselves"?  You mean, you know by heart each
> > Emacs command that invokes a subprocess, and avoid to invoke them all
> > unless the laptop is plugged in?  I don't believe this is practical,
> > because even "C-x C-f" invokes a a subprocess when you visit a file
> > under VCS (which I guess happens a lot to the likes of you and me).
> > And how can a person avoid all of those commands?  What am I missing?
> 
> Native compilation is significantly more CPU intensive than any of these
> other things, though.

Do you have data to support that?  I just cited mine, and it doesn't
sound intensive to me.  E.g., when some command invokes Git, Git could
decide that it's time to do a background GC, and that will be quite
CPU intensive.  And yet we all use VC without fear.

But I must stop raising the noise level here, because that's what all
I write seems to be good for -- making noise that no one takes
seriously, starting from Lars.  Sorry.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

>> You can only have one of them installed at a time, and you can
>> (currently) only have one major version installed at a time.
>> 
>>   https://packages.debian.org/search?keywords=emacs
>
> That's strange to hear.  Even MS-Windows allows per-user variations of
> PATH and per-use environment variables.  It is strange to learn that
> Debian doesn't.

Debian does as well.  What Rob is saying is that *if* you want to use
the Emacs provided by Debian, then the choice of which flavor (and
version) you get will apply system-wide.  If that's not good enough for
you, then you can install your own build of Emacs in the usual way, and
it's very easy to do so.

All my Debian boxes come with "Debian's Emacs" installed, and then most
of them have a few other versions built locally for my own personal use.

> And I guess now I'm confused what is it exactly that you'd want to
> achieve.  Do you want to disable native compilation, or do you want to
> have all *.eln files in a shared location?

AFAIK he wants:

- To eagerly compile the `.eln` files for the third party ELisp packages
  installed system-wide using Debian's package manager.

- That every time Emacs is run as part of
  (un)installing/building/testing a Debian package, that Emacs should
  not write any file under $HOME/ (or ~$REALUSER/ either).


-- Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Mon, 03 Oct 2022 11:23:46 -0700
> 
> > When I start an Emacs session for a new version (which needs to
> > recompile all the packages I use, because my sessions are restored by
> > desktop.el), I see about 100 Lisp files compiled into *.eln inside of
> > 1 minute, and then another bunch of 100 inside another minute (with 10
> > minutes between these two groups).  Does this strike you as a long
> > time to avoid unplugging?  I hope your battery can hold more than a
> > few minutes.
> 
> Sounds like your machine is just a lot better than mine.

It's a 2012 vintage Core i7.  And it runs Windows XP, so both process
startup and disk I/O are slower than GNU/Linux.  Though the main disk
is SSD.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: rlb@defaultvalue.org,  tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Mon, 03 Oct 2022 14:33:09 -0400
> 
> >> And the issue is not subprocesses per se, but it's extra processing that
> >> happens outside of the control of the user.
> > How do you mean "outside of the control of the user"?  The user causes
> > it by loading a feature.
> 
> When a user loads a feature, this request doesn't say explicitly "and
> native compile the file".  And there is no direct way to "just load this
> package without compiling it".  So the compilation is not really under
> the control of the user.
> 
> I don't think users know, when they load a feature, if it's the first
> time they loaded this feature in this version of Emacs, so by and large
> they can't predict when the compilation does happen.

It just takes getting used to, that's all.

> > How is that different from any other command that uses a subprocess
> > under the hood?
> 
> It's different in that for those other commands, the users cannot get
> the result they want without running that subprocess, so even if they're
> not fully aware of it, they do have some vague idea it's inevitably
> going to happen.
> 
> In contrast, the compilation is not indispensable, (so much so that the
> past it never happened).

So native-compilation is "guilty" because it can be disabled? ;-)

> > On my system, I don't see any warnings.
> 
> That's presumably because you stick to the bundled packages (where we've
> managed to keep a "no compilation warnings" state for the last few
> years) or the few third party packages which are careful enough to
> eliminate all warnings.
> 
> The rise of ELPA has removed most of the cases where native compilation
> will miscompile files (because ELPA forces the .el files to be compiled
> in a "standard" way rather than using ad-hoc Makefile rules which the
> native compiler cannot and does not try to reproduce), but compilation
> warnings are still very common (the situation is improving, tho, and
> arguably the warnings popping up out of the blue during lazy native
> compilation may prove to be a good way to convince more package authors
> to clean up their act).

So this, too, is extremely temporary, and will disappear soon enough,
at least in popular packages.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Rob Browning <rlb@defaultvalue.org>,  larsi@gnus.org,
>   yandros@gmail.com,  tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Mon, 03 Oct 2022 14:45:32 -0400
> 
> > And I guess now I'm confused what is it exactly that you'd want to
> > achieve.  Do you want to disable native compilation, or do you want to
> > have all *.eln files in a shared location?
> 
> AFAIK he wants:
> 
> - To eagerly compile the `.eln` files for the third party ELisp packages
>   installed system-wide using Debian's package manager.
> 
> - That every time Emacs is run as part of
>   (un)installing/building/testing a Debian package, that Emacs should
>   not write any file under $HOME/ (or ~$REALUSER/ either).

That's where I'm confused: if *.eln files are installed system-wide,
then why would there be any compilation that writes to the home
directory, which then causes Rob to wish to disable it?  All *.eln
files are already available, no?

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]

On Mon, Oct 03, 2022 at 09:41:16PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 3 Oct 2022 20:17:53 +0200
> > From: tomas@tuxteam.de
> > Cc: Rob Browning <rlb@defaultvalue.org>, larsi@gnus.org, yandros@gmail.com,
> > 	emacs-devel@gnu.org
> > 
> > On Mon, Oct 03, 2022 at 08:41:22PM +0300, Eli Zaretskii wrote:
> > > [Full disclosure: evidently, none of what I write below matters, so
> > > feel free to ignore.]
> > 
> > Now I think this is a bit unfair. You have a string opinion, and
> > I think it shhould be respected, but the others have, too.
> 
> You misunderstood the intent.

Quite possibly.

> Windows installers allow you to install into a directory of your
> choice, and some will even set PATH to point to there; if not, you can
> do it manually.

I have no clue about Windows (at least since 1990). I try to keep
it that way for reasons.

> > Alternatively, you can have a system-local install (going to
> > /usr/local) besides the Debian-installed one without them interfering
> > (that's the setup I use).
> 
> I don't understand what that means.  Do you mean that you can subvert
> the general non-support by Debian packages of several Emacs versions
> installed on the same system?

I don't understand what you mean by "subverting the general non-support",
(I somehow feel that it is a loaded term, but I can't be sure).

Cheers
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Mon, 3 Oct 2022 21:02:11 +0200
> From: tomas@tuxteam.de
> Cc: rlb@defaultvalue.org, larsi@gnus.org, yandros@gmail.com,
> 	emacs-devel@gnu.org
> 
> > > Alternatively, you can have a system-local install (going to
> > > /usr/local) besides the Debian-installed one without them interfering
> > > (that's the setup I use).
> > 
> > I don't understand what that means.  Do you mean that you can subvert
> > the general non-support by Debian packages of several Emacs versions
> > installed on the same system?
> 
> I don't understand what you mean by "subverting the general non-support",

Rob said they don't support that with Debian packages.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> We'd also like to be able to compile and install system-wide -- it would
> be good to be able to benefit from native compilation without it
> happening by surrpise.

I haven't heard any specific request in this regard: is there something
preventing you from doing that (for your definition of "that")?


        Stefan

Re: Suppressing native compilation (short and long term)

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> compilation warnings are still very common (the situation is
>> improving, tho, and arguably the warnings popping up out of the blue
>> during lazy native compilation may prove to be a good way to convince
>> more package authors to clean up their act).
>
> So this, too, is extremely temporary, and will disappear soon enough,
> at least in popular packages.

I don't see that the situation has substantially improved, though I'm of
course happy to hear that Stefan M remains optimistic.

In far too many cases, it is very hard to get fixes for warnings merged
in a timely manner, and it's even harder to get those fixes actually
released.

Here are some recent examples, to give an idea:

    https://github.com/deb0ch/emacs-winum/pull/31
    https://github.com/lewang/ws-butler/pull/40
    https://github.com/Malabarba/aggressive-indent-mode/pull/150

These examples can be multiplied at will.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> [Full disclosure: evidently, none of what I write below matters, so
> feel free to ignore.]

Well, I can't speak for others, but it matters to me (your opinion and
the other upstream developers'), or I wouldn't be here, and I'm very
glad there's a "here for me to be" because I'm quite fond of Emacs :)

> But okay, yes, if Debian users live under such severe restrictions,
> then the case for having user-specific *.eln directories becomes
> weaker.  But I still don't see that it is weaker than disallowing
> native compilation at run time.

To be clear, I've never been talking about disallowing it for normal
user use, and apologies if I made it sound that way.

For normal users, if we pursued this, the idea would be that after "apt
install emacs" finishes, they'd have a full set of .elc and .eln files
corresponding to their version of Emacs, and the .el files they have.

Then if anything changes in a way that warrants it, Emacs would
(re)compile to their ~/.emacs.d/eln-cache/ as "normal".  But for those
who don't ever change their .el files (or shadow them), that would never
happen, or would only happen for the files that they change/shadow.

> And I guess now I'm confused what is it exactly that you'd want to
> achieve.  Do you want to disable native compilation, or do you want to
> have all *.eln files in a shared location?  Because it seems to me you
> said you wanted both, and I don't see how both could be reconciled.

The ability to disable .eln compilation entirely is only for some
Debian-specific (though maybe useful for others) special cases, not
anything we're proposing for "normal use".

And it doesn't have to be "disabling" -- being able to redirect the .eln
files to another (temp) dir would be fine too, as long as it wasn't too
awkward to arrange for the entire (sub)process tree.

Those special cases are (at least):

  - When building the relevant Debian packages -- for Emacs itself, for
    add-ons like elpa-magit, etc.  Debian forbids package builds from
    writing outside of the package build directory /tmp, and another
    tree or two, e.g. HOME is not allowed.  Note that this is normally
    handled by the Debian autobuilders, it's not something users
    typically do.

  - During package installs, i.e. whenever Emacs is run during an "apt
    install emacs" or "apt upgrade emacs", etc., and it may be run many
    times there, both directly, and indirectly as it rebuilds whichever
    packages need rebuilding (e.g. add-on packages).

  - During some autmated Debian tests.

Hope this helps
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Po Lu]

Sean Whitton <spwhitton@spwhitton.name> writes:

> The point with battery consumption is not about running vs. standby.
> The issue is that while users expect that running apt-get will drain the
> battery, they expect that once apt-get is done, the only battery-hungry
> processes are ones they start themselves.  Laptop users typically avoid
> running apt-get without mains power plugged in.  If I do an 'apt-get
> upgrade' then afterwards my CPU will be churning away compiling addon
> packages, so I can't just unplug.

I don't know if that's true with Debian users, but nobody I know of
objects to running "dnf install" on battery.  After all, installing new
packages only takes a short amount of time, certainly not enough to
significantly affect battery usage.

OTOH, async native compilation is something I would definitely plug in
for.  That applies to other programs too, notably web browsers.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Mon, 3 Oct 2022 23:58:28 +0200
> Cc: rlb@defaultvalue.org, tomas@tuxteam.de, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> compilation warnings are still very common (the situation is
> >> improving, tho, and arguably the warnings popping up out of the blue
> >> during lazy native compilation may prove to be a good way to convince
> >> more package authors to clean up their act).
> >
> > So this, too, is extremely temporary, and will disappear soon enough,
> > at least in popular packages.
> 
> I don't see that the situation has substantially improved

You are too impatient, IMO.  This stuff takes time to propagate; Emacs
28 is just beginning to be widely used (otherwise we'd have this
discussion many moons ago).  We had these problems in Emacs as well,
originally.

> Here are some recent examples, to give an idea:
> 
>     https://github.com/deb0ch/emacs-winum/pull/31
>     https://github.com/lewang/ws-butler/pull/40
>     https://github.com/Malabarba/aggressive-indent-mode/pull/150
> 
> These examples can be multiplied at will.

Yes, it is easy to think the problem is larger and more significant
than it really is.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Rob Browning <rlb@defaultvalue.org>,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org,
>   akrl@sdf.org
> Date: Tue, 04 Oct 2022 08:31:01 +0800
> 
> Sean Whitton <spwhitton@spwhitton.name> writes:
> 
> > The point with battery consumption is not about running vs. standby.
> > The issue is that while users expect that running apt-get will drain the
> > battery, they expect that once apt-get is done, the only battery-hungry
> > processes are ones they start themselves.  Laptop users typically avoid
> > running apt-get without mains power plugged in.  If I do an 'apt-get
> > upgrade' then afterwards my CPU will be churning away compiling addon
> > packages, so I can't just unplug.
> 
> I don't know if that's true with Debian users, but nobody I know of
> objects to running "dnf install" on battery.  After all, installing new
> packages only takes a short amount of time, certainly not enough to
> significantly affect battery usage.
> 
> OTOH, async native compilation is something I would definitely plug in
> for.

Why the difference, I wonder?  Async native compilation also takes a
short amount of time, as the data I posted indicates.

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Why the difference, I wonder?  Async native compilation also takes a
> short amount of time, as the data I posted indicates.

I don't know, but the fans spin up during async native compilation and
not package installation.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: spwhitton@spwhitton.name,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org,
>   akrl@sdf.org
> Date: Tue, 04 Oct 2022 16:37:10 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Why the difference, I wonder?  Async native compilation also takes a
> > short amount of time, as the data I posted indicates.
> 
> I don't know, but the fans spin up during async native compilation and
> not package installation.

Maybe you should customize native-comp-async-jobs-number to a small
enough value, like 1?  The default zero means half of your CPU's
execution units.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: larsi@gnus.org, yandros@gmail.com, tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Mon, 03 Oct 2022 19:16:29 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > [Full disclosure: evidently, none of what I write below matters, so
> > feel free to ignore.]
> 
> Well, I can't speak for others, but it matters to me (your opinion and
> the other upstream developers'), or I wouldn't be here, and I'm very
> glad there's a "here for me to be" because I'm quite fond of Emacs :)

I didn't mean you, FWIW.

> > And I guess now I'm confused what is it exactly that you'd want to
> > achieve.  Do you want to disable native compilation, or do you want to
> > have all *.eln files in a shared location?  Because it seems to me you
> > said you wanted both, and I don't see how both could be reconciled.
> 
> The ability to disable .eln compilation entirely is only for some
> Debian-specific (though maybe useful for others) special cases, not
> anything we're proposing for "normal use".
> 
> And it doesn't have to be "disabling" -- being able to redirect the .eln
> files to another (temp) dir would be fine too, as long as it wasn't too
> awkward to arrange for the entire (sub)process tree.
> 
> Those special cases are (at least):
> 
>   - When building the relevant Debian packages -- for Emacs itself, for
>     add-ons like elpa-magit, etc.  Debian forbids package builds from
>     writing outside of the package build directory /tmp, and another
>     tree or two, e.g. HOME is not allowed.  Note that this is normally
>     handled by the Debian autobuilders, it's not something users
>     typically do.
> 
>   - During package installs, i.e. whenever Emacs is run during an "apt
>     install emacs" or "apt upgrade emacs", etc., and it may be run many
>     times there, both directly, and indirectly as it rebuilds whichever
>     packages need rebuilding (e.g. add-on packages).
> 
>   - During some autmated Debian tests.

The above means (AFAIU) that disabling is not the right solution,
because you want the *.eln files to go to the shared location where
users could have them loaded when needed.  Is that correct, or am I
again missing something?

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Maybe you should customize native-comp-async-jobs-number to a small
> enough value, like 1?  The default zero means half of your CPU's
> execution units.

I will try that and see how it goes, thanks.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> I don't see that the situation has substantially improved, though I'm of
> course happy to hear that Stefan M remains optimistic.

Seen from the K-side, the glass is half full, but I'm on the M-side
where I see it as half-empty.


        Stefan

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Tue 04 Oct 2022 at 08:31AM +08, Po Lu wrote:

> Sean Whitton <spwhitton@spwhitton.name> writes:
>
>> The point with battery consumption is not about running vs. standby.
>> The issue is that while users expect that running apt-get will drain the
>> battery, they expect that once apt-get is done, the only battery-hungry
>> processes are ones they start themselves.  Laptop users typically avoid
>> running apt-get without mains power plugged in.  If I do an 'apt-get
>> upgrade' then afterwards my CPU will be churning away compiling addon
>> packages, so I can't just unplug.
>
> I don't know if that's true with Debian users, but nobody I know of
> objects to running "dnf install" on battery.  After all, installing new
> packages only takes a short amount of time, certainly not enough to
> significantly affect battery usage.

The issue is Debian Emacs addon packages, in particular.  Upgrading
those kicks off a whole lot of bytecompilation.  It's much
heavier-weight than usual package upgrades.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> The above means (AFAIU) that disabling is not the right solution,
> because you want the *.eln files to go to the shared location where
> users could have them loaded when needed.  Is that correct, or am I
> again missing something?

If I understand you, I think the answer may be no?

The test and package build cases are (often) isolated, automated,
debien-specific processes that happen on debian build machines (as root
and chrooted), and the entire workspace is then thrown away.  It's never
relevant to a "normal" user.

The package install/upgrade process is system-wide, runs as root, and is
specifically and quite intentionally not per-user.  It's intended to
establish the static, common baseline for *all* users on the machine
until the next package upgrade.

Hope this helps
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: larsi@gnus.org, yandros@gmail.com, tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Tue, 04 Oct 2022 19:48:01 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The above means (AFAIU) that disabling is not the right solution,
> > because you want the *.eln files to go to the shared location where
> > users could have them loaded when needed.  Is that correct, or am I
> > again missing something?
> 
> If I understand you, I think the answer may be no?
> 
> The test and package build cases are (often) isolated, automated,
> debien-specific processes that happen on debian build machines (as root
> and chrooted), and the entire workspace is then thrown away.  It's never
> relevant to a "normal" user.

If the workspace is thrown away, you shouldn't care about the *.eln
files it produces, right?  They will be thrown away together with the
workspace.  So why bother disabling their production in this case?

> The package install/upgrade process is system-wide, runs as root, and is
> specifically and quite intentionally not per-user.  It's intended to
> establish the static, common baseline for *all* users on the machine
> until the next package upgrade.

When this package install/upgrade process runs, doesn't it include
installing the *.eln files that come with the package, and need to be
used when the users use the package after the installation?  AFAIU,
you want to install those *.eln files in a shared location, which is
fine and supported: just put them in the same place under
/usr/lib/emacs/VERSION/native-lisp/ directory where Emacs installs its
precompiled *.eln files.  Or, if you want to have the *.eln files from
add-on packages to live in a separate place, define where that
directory will be, install the package's *.eln files there, and tweak
native-comp-eln-load-path to include that additional directory.

Again, in this case as well, I see no problem with generation of *.eln
files that you'd like to prevent.  And yet you are still saying that
disabling *.eln generation might be needed.  Could you please describe
in enough detail when are those unwanted *.eln produced in the two
situations you described (test and package build case, and package
install/upgrade case), and why are those *.eln files unwanted in that
case?

Thanks.

P.S. I'm sorry to keep this thread alive for so long, but I really
don't have a clear idea why you'd want to disable generation of *.eln
files, and I think it's important for us to understand your reasons.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Rob Browning <rlb@defaultvalue.org> writes:
>
>> At the top level, we wanted a way to avoid writing to HOME during
>> packaging, testing, installs (in this case, it's the .eln files, now
>> that we've enabled native compilation).
>>
>> That could be handled by some way to turn off native compilation, or by
>> some way to comprehensively divert those writes to another location
>> (e.g. temp dir).  Either is fine, though we'd originally thought the
>> former might make things a bit easier.
>
> Yeah, I think the former is both easier to implement and easier for
> users.
>
> So I'm thinking of introducing a user option like
> `native-compile-inhibit', which will make Emacs skip the native-comp
> machinery when loading .elc files.

IIUC this would just control `load-no-native' and
`native-comp-deferred-compilation'?  I think these two are doing what
you suggest here no?

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Lars Ingebrigtsen [2022-10-03 15:29:40] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> AFAICT so far I've heard of two reasons to "disable automatic native
>>> compilation":
>>>
>>> A) Avoid the "uncontrolled" (mostly in terms of *when* it happens)
>>>    resource usage associated with it (e.g. battery).
>>>
>>> B) Avoid writing to $HOME.
>>>
>>> For (A), I think that (setq native-comp-deferred-compilation nil) seems
>>> perfectly sufficient (you can set it early in your (early) init file and if
>>> every blue moon some native compilation still happens because of a file
>>> loaded before that `setq` it shouldn't be a big deal).
>>
>> I've now introduced the new inhibit-native-compilation variable and
>> deprecated the native-comp-deferred-compilation variable, and also
>> initialised stuff from EMACS_INHIBIT_NATIVE_COMPILATION.
>> (And made trampolines not write to the cache in this case.)
>
> AFAICT for the case (A), we *do* want to save trampolines for the next
> time around, and those users probably do want to use native compilation
> (e.g. they would likely set `package-native-compile` to non-nil), just
> not the deferred kind, so the var name is a bit odd for them.

I find this name odd as well. `inhibit-native-compilation' is *not*
disabling native compilation for two reasons:

1- One can still inoke it manually
2- Trampoline automatic ocmpilation is still active

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> We can expand the variable name, but I couldn't think of a good name.
>>> And "deferred" doesn't convey anything.
>>
>> Lazy?  Auto(matic)?  Ondemand?
>
> `inhibit-lazy-native-compilation' sounds like it makes the compilation
> less lazy.  `inhibit-automatic-native-compilation' might work.

Again `inhibit-automatic-native-compilation' does not disable automatic
trampoline native compilation :/

BTW I think most of people refers to what was controlled by
`native-comp-deferred-compilation' as a jitter mechanism, so I think a
better name would have been `inhibit-native-jit-compilation'.  I wish
this change was not rushed.

Best Regards

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> I think that entire changeset should be reverted.  It is not well
>> thought out, and ion some aspects downright dangerous.  The
>> environment variable is especially egregious: it will affect all the
>> sub-process Emacsen as well, something that will bite users, a lesson
>> we've learned long ago with the likes of EMACSLOADPATH.
>
> How is setting EMACS_INHIBIT_NATIVE_COMPILE dangerous?
>
>> Such changes should not be installed without a great deal of careful
>> thought and consideration of the different use cases.  Which I tried
>> to explain all the way, but apparently to deaf ears.
>
> The discussion has been going for apparently weeks no with no progress.
> It's easier to discuss code when there's code to discuss.

We can discuss code also on branches, there's no need to install changes
on master to discuss them.

I'm back now after a long weekend and your proposal of introduciung
`inhibit-native-compilation' is form 2 days and 1 hour ago.  Sorry but I
don't want to feel that changes can be rushed into Emacs code and in
order to partecipate to the discussion people can't have some time off.

Best Regards

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,  david@tethera.net,
>>   emacs-devel@gnu.org,  akrl@sdf.org
>> Date: Mon, 03 Oct 2022 16:09:35 +0200
>> 
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> 
>> > AFAICT for the case (A), we *do* want to save trampolines for the next
>> > time around, and those users probably do want to use native compilation
>> > (e.g. they would likely set `package-native-compile` to non-nil), just
>> > not the deferred kind, so the var name is a bit odd for them.
>> 
>> The trampolines are very fast to make, so creating them once per Emacs
>> session doesn't seem like a problem.  If that turns out to be an issue,
>> we can tweak the variable.
>
> I think that entire changeset should be reverted.

Definitely agree here.

Other than the (IMO) bad naming, is still not clear to me what's exactly
the problem is trying to solve.

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Again `inhibit-automatic-native-compilation' does not disable automatic
> trampoline native compilation :/
>
> BTW I think most of people refers to what was controlled by
> `native-comp-deferred-compilation' as a jitter mechanism, so I think a
> better name would have been `inhibit-native-jit-compilation'.  I wish
> this change was not rushed.

I don't think `inhibit-native-jit-compilation' conveys anything more to
the user than `inhibit-automatic-native-compilation'?

Andrea Corallo <akrl@sdf.org> writes:

> We can discuss code also on branches, there's no need to install changes
> on master to discuss them.
>
> I'm back now after a long weekend and your proposal of introduciung
> `inhibit-native-compilation' is form 2 days and 1 hour ago.  Sorry but I
> don't want to feel that changes can be rushed into Emacs code and in
> order to partecipate to the discussion people can't have some time off.

Development takes place on the "master" branch -- code appearing there
does not curtail further discussion.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> IIUC this would just control `load-no-native' and
> `native-comp-deferred-compilation'?  I think these two are doing what
> you suggest here no?

The latter yes -- nobody has discussed doing anything with
`load-no-native' (which is another variable that's difficult to discover
due to how it's named, and isn't documented anywhere, either).

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> If we want to consider providing (yet another) user option for
>> disabling native compilation, then we should:
>
> I asked what the user option to disable native compilation was, but
> didn't get an answer, and here you say "(yet another)", so...  what is
> the current user option to disable native compilation?

Sorry for being late: `native-comp-deferred-compilation' and
`load-no-native'.

>>   . understand why and in which situations they may need it
>
> Doing repeatable testing is one obvious situation.

I think the two previously mentioned knobs should be sufficient for
repeatable testing no?

Also repeatable testing are most likely executed in batch mode and...

...surprise surprise deferred compilation is *already* *disabled* in
this mode!!

I've the impression no one mentioned this small detail in this huge
thread, but still we have installed changes to disable a feature for
debian pkg installation that is in fact already disabled :( :(

Best Regards

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Rob Browning <rlb@defaultvalue.org> writes:
>
>> Though I'm not positive because if we still really need compilation for
>> the trampolines(?), then we may need to just follow the "redirect to a
>> tempdir" approach, which is also fine, and then the ability to disable
>> native compilation might not be relevant to the packaging.
>
> I'm not sure we need to save the trampolines at all (in this
> don't-do-native-compilation configuration) -- Andrea probably knows.
> Andrea, would it be possible to just create and use the trampolines
> without writing them to a file?

No

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> IIUC this would just control `load-no-native' and
>> `native-comp-deferred-compilation'?  I think these two are doing what
>> you suggest here no?
>
> The latter yes -- nobody has discussed doing anything with
> `load-no-native'

I wasn't clear to me what you meant with "native-comp machinery when
loading .elc files".

> (which is another variable that's difficult to discover due to how
> it's named, and isn't documented anywhere, either).

It is named like that since before the native comp branch was reviewed
and merged, I think you were in that thread.  Sorry we all try our best.

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Again `inhibit-automatic-native-compilation' does not disable automatic
>> trampoline native compilation :/
>>
>> BTW I think most of people refers to what was controlled by
>> `native-comp-deferred-compilation' as a jitter mechanism, so I think a
>> better name would have been `inhibit-native-jit-compilation'.  I wish
>> this change was not rushed.
>
> I don't think `inhibit-native-jit-compilation' conveys anything more to
> the user than `inhibit-automatic-native-compilation'?

That's your opinion and I respect it.  Still
`inhibit-automatic-native-compilation' does *not* disable automatic
native compilation but only a mechanism contributing to it, so it's IMO
a bad naming decision.

> Andrea Corallo <akrl@sdf.org> writes:
>
>> We can discuss code also on branches, there's no need to install changes
>> on master to discuss them.
>>
>> I'm back now after a long weekend and your proposal of introduciung
>> `inhibit-native-compilation' is form 2 days and 1 hour ago.  Sorry but I
>> don't want to feel that changes can be rushed into Emacs code and in
>> order to partecipate to the discussion people can't have some time off.
>
> Development takes place on the "master" branch -- code appearing there
> does not curtail further discussion.

I have not said that once code is in master discussion is forbidden.  I
said that to discuss a change there's *no* requirement to install it in
master, especially before sufficient discussion is done on the list for
these tricky interfaces.  My 2cts are that these mechanisms and changes
should be very well thought and participated before being modified.

As maintainer of comp.c and related I ask to have this changeset
reverted and then we restart thinking again what's the best change (if
any) needed here.

Thanks

  Andrea

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii
>  <eliz@gnu.org>,
>         rlb@defaultvalue.org, david@tethera.net, emacs-devel@gnu.org
> Date: Wed, 05 Oct 2022 12:55:40 +0000
> 
> I wish this change was not rushed.

Me too.  Especially since I'm still struggling to understand the
rationale of the Debian packagers to request this.

Re: Suppressing native compilation (short and long term)

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Maybe you should customize native-comp-async-jobs-number to a small
>> enough value, like 1?  The default zero means half of your CPU's
>> execution units.
>
> I will try that and see how it goes, thanks.

That makes the fans less loud (they are still noticable), but it also
takes twice as long for the fans to subside, as expected.

Hope this data point ends up meaningful.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Also repeatable testing are most likely executed in batch mode and...
>
> ...surprise surprise deferred compilation is *already* *disabled* in
> this mode!!

Not all testing happens in batch mode.

Andrea Corallo <akrl@sdf.org> writes:

>> I'm not sure we need to save the trampolines at all (in this
>> don't-do-native-compilation configuration) -- Andrea probably knows.
>> Andrea, would it be possible to just create and use the trampolines
>> without writing them to a file?
>
> No

Yes, it is.  (That is, the trampolines get written to /tmp, but can be
deleted after loading.)

Andrea Corallo <akrl@sdf.org> writes:

> That's your opinion and I respect it.  Still
> `inhibit-automatic-native-compilation' does *not* disable automatic
> native compilation but only a mechanism contributing to it, so it's IMO
> a bad naming decision.

Like I said earlier, if anybody has a better name for the variable,
renaming it is fine, but it has to be an improvement.

> I have not said that once code is in master discussion is forbidden.  I
> said that to discuss a change there's *no* requirement to install it in
> master, especially before sufficient discussion is done on the list for
> these tricky interfaces.  My 2cts are that these mechanisms and changes
> should be very well thought and participated before being modified.
>
> As maintainer of comp.c and related I ask to have this changeset
> reverted and then we restart thinking again what's the best change (if
> any) needed here.

I don't see any advantages to doing that -- the changes that are in now
seem to work fine for the stated use cases (which are both "don't write
to $HOME while testing" and "I want to use a AOT-compiled Emacs, but
don't do any further JIT compilation while running Emacs").

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Rob Browning <rlb@defaultvalue.org>,  Stefan Monnier
>  <monnier@iro.umontreal.ca>,  Eli Zaretskii <eliz@gnu.org>,
>   david@tethera.net,  emacs-devel@gnu.org
> Date: Wed, 05 Oct 2022 15:16:31 +0200
> 
> Andrea Corallo <akrl@sdf.org> writes:
> 
> > IIUC this would just control `load-no-native' and
> > `native-comp-deferred-compilation'?  I think these two are doing what
> > you suggest here no?
> 
> The latter yes -- nobody has discussed doing anything with
> `load-no-native' (which is another variable that's difficult to discover
> due to how it's named, and isn't documented anywhere, either).

Variables that are "hard to discover" are intentionally named like
that: we didn't intend them to be discoverable, because we didn't
think they should be used except internally.

During the last stages of native-comp development, we went through the
variables in comp.c and comp.el with Andrea, and renamed those we
thought would be useful so that they are easily discoverable.  Please
don't assume that the names you see are due to omission or negligence
or ineptitude.  Maybe additional ones should be renamed/aliased to be
more discoverable, but before we do that, we need to understand the
problems we are solving.  That requires careful consideration and
discussion, whereas rushing with installing changes in the area where
you don't have enough prior knowledge and history ends up wreaking
havoc, to say nothing of personal aggravation.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii
>>  <eliz@gnu.org>,
>>         rlb@defaultvalue.org, david@tethera.net, emacs-devel@gnu.org
>> Date: Wed, 05 Oct 2022 12:55:40 +0000
>> 
>> I wish this change was not rushed.
>
> Me too.  Especially since I'm still struggling to understand the
> rationale of the Debian packagers to request this.

Agree and that's not all!  As we know native compilations happens
automatically for:

1- Trampolines, this we *cannot* disable in any case.

2- Deferred/jit compilation, this is *already* disabled in non
interactive sessions! (the way I guess Debian Emacs process is used for
package instgallation).

So I think this changeset does not help Debian packagers at all but ATM
is just more surface for future issues!

  Andrea

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org
> Date: Wed, 05 Oct 2022 16:01:35 +0200
> 
> > As maintainer of comp.c and related I ask to have this changeset
> > reverted and then we restart thinking again what's the best change (if
> > any) needed here.
> 
> I don't see any advantages to doing that -- the changes that are in now
> seem to work fine for the stated use cases (which are both "don't write
> to $HOME while testing"

That use case is not yet clear, not at all.

> and "I want to use a AOT-compiled Emacs, but don't do any further
> JIT compilation while running Emacs").

This use case makes no sense at all.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Also repeatable testing are most likely executed in batch mode and...
>>
>> ...surprise surprise deferred compilation is *already* *disabled* in
>> this mode!!
>
> Not all testing happens in batch mode.
>
> Andrea Corallo <akrl@sdf.org> writes:
>
>>> I'm not sure we need to save the trampolines at all (in this
>>> don't-do-native-compilation configuration) -- Andrea probably knows.
>>> Andrea, would it be possible to just create and use the trampolines
>>> without writing them to a file?
>>
>> No
>
> Yes, it is.  (That is, the trampolines get written to /tmp, but can be
> deleted after loading.)

You asked: "would it be possible to just create and use the trampolines
without writing them to a file?"

I aswered "no".  Of course writing a file into /tmp is still writing to
a file.

> Andrea Corallo <akrl@sdf.org> writes:
>
>> That's your opinion and I respect it.  Still
>> `inhibit-automatic-native-compilation' does *not* disable automatic
>> native compilation but only a mechanism contributing to it, so it's IMO
>> a bad naming decision.
>
> Like I said earlier, if anybody has a better name for the variable,
> renaming it is fine, but it has to be an improvement.

I think `inhibit-jit-native-compilation' is better as I believe users
perceive the JIT word related to user code and not internal mechanisms
as trampolines.

I've the perception that this change was done without the full picture
in mind of how the native compiler and his mechanisms works.  As a
result the current naming is IMO just wrong, and as such is a step
backward the original state.

As maintainer of the native compiler is my duty to point this out and
ask for reversion.

>> I have not said that once code is in master discussion is forbidden.  I
>> said that to discuss a change there's *no* requirement to install it in
>> master, especially before sufficient discussion is done on the list for
>> these tricky interfaces.  My 2cts are that these mechanisms and changes
>> should be very well thought and participated before being modified.
>>
>> As maintainer of comp.c and related I ask to have this changeset
>> reverted and then we restart thinking again what's the best change (if
>> any) needed here.
>
> I don't see any advantages to doing that -- the changes that are in now
> seem to work fine for the stated use cases (which are both "don't write
> to $HOME while testing" and "I want to use a AOT-compiled Emacs, but
> don't do any further JIT compilation while running Emacs").

I explained in two other mails that these changesets are orthogonal to
what the user asked and don't help them.  Also is still not clear why
the user asked for this knob.

It is *extremely* important that we analyze the usecase *before*
introducing new interfaces.  The user might ask for a new interface
without knowing we can provide or suggest a better solution. It might
even already exists!!

Developers can't just add mechanisms without a full understanding of
the current ones just because there was a user request, even worst
without an in deep discussion, sorry this is IMO just recipe for
disaster.

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> and "I want to use a AOT-compiled Emacs, but don't do any further
>> JIT compilation while running Emacs").
>
> This use case makes no sense at all.

It's been requested several times, whether it makes sense to you or not.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> I think `inhibit-jit-native-compilation' is better as I believe users
> perceive the JIT word related to user code and not internal mechanisms
> as trampolines.

It's possible -- I'm not married to the current name.  Perhaps we should
take a poll?

> I've the perception that this change was done without the full picture
> in mind of how the native compiler and his mechanisms works.  As a
> result the current naming is IMO just wrong, and as such is a step
> backward the original state.

I don't know where you got that perception from.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> I think `inhibit-jit-native-compilation' is better as I believe users
>> perceive the JIT word related to user code and not internal mechanisms
>> as trampolines.
>
> It's possible -- I'm not married to the current name.  Perhaps we should
> take a poll?
>
>> I've the perception that this change was done without the full picture
>> in mind of how the native compiler and his mechanisms works.  As a
>> result the current naming is IMO just wrong, and as such is a step
>> backward the original state.
>
> I don't know where you got that perception from.

Well to give few examples you were not aware of: the `load-no-native'
mechanism, the fact that deferred compilation is not the only mechanism
concurring to automatic native compilation (and that's why it was named
as such and not just automatic native compilation), the fact that native
compilation does not happen in non interactive sessions.

There is nothing wrong with that, the native compiler is a complex
machine and its interface as well, but still: there's no single aspect
of this changeset that I see as an improvement, so as maintainer of the
native compiler I ask to have it reverted.

Thanks

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Well to give few examples you were not aware of: the `load-no-native'
> mechanism, the fact that deferred compilation is not the only mechanism
> concurring to automatic native compilation (and that's why it was named
> as such and not just automatic native compilation), the fact that native
> compilation does not happen in non interactive sessions.

I didn't know about the first (because it's badly named and
undocumented, as well as totally irrelevant to the discussion in this
thread), but I was aware of all the other things here, and I'm not sure
why you'd think otherwise.

My perception here is that you're mostly angry that somebody else is
working on your code -- but that's pretty common.  Many people feel
proprietary towards code they've written.

> There is nothing wrong with that, the native compiler is a complex
> machine and its interface as well, but still: there's no single aspect
> of this changeset that I see as an improvement, so as maintainer of the
> native compiler I ask to have it reverted.

Like I said before, the code solves (at least) two real problems, so I'm
not in favour of doing so.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> That makes the fans less loud (they are still noticable), but it also
> takes twice as long for the fans to subside, as expected.
>
> Hope this data point ends up meaningful.

Yes.

Software is distributed pre-compiled for a reason -- because people run
the software on hardware where compiling the software takes a long time.
It's entirely reasonable for people to want to have a fully-built
native-compiled Emacs installation on their laptops, without making that
Emacs do any further JIT compilation.  (Except where necessary for
trampolines, of course.)

This has been requested a number of times over several years now, but
these requests have been ignored because apparently "people shouldn't
want that".

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Well to give few examples you were not aware of: the `load-no-native'
>> mechanism, the fact that deferred compilation is not the only mechanism
>> concurring to automatic native compilation (and that's why it was named
>> as such and not just automatic native compilation), the fact that native
>> compilation does not happen in non interactive sessions.
>
> I didn't know about the first (because it's badly named and
> undocumented, as well as totally irrelevant to the discussion in this
> thread), but I was aware of all the other things here, and I'm not sure
> why you'd think otherwise.

Well because I cannot understand why one would do any of these changes
if these mechanisms are known.

> My perception here is that you're mostly angry that somebody else is
> working on your code -- but that's pretty common.  Many people feel
> proprietary towards code they've written.

This is not the case at all, please trust me, your changeset does two
things:

1- change the name a knob, but it goes from a maybe un-intuitive one to
   just (as explained) a plain wrong one.

2- add a mechanism that (as explained) cannot help with the user request
   in this discussion at all.

These are both not improvements, this is the reason I'm asking for it to
be reverted now.

Please don't accuse me to feel the ownership on this code, I've never
objected to other changes on this done on master as I thought were
correct.

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> This is not the case at all, please trust me, your changeset does two
> things:
>
> 1- change the name a knob, but it goes from a maybe un-intuitive one to
>    just (as explained) a plain wrong one.

It goes from an un-intuitive (and undocumented) one to an
intuitively-named (and documented) one.  In my opinion, the old variable
is just as wrong as the current one (because it seemed to imply that
compilation would be immediate instead of deferred).

> 2- add a mechanism that (as explained) cannot help with the user request
>    in this discussion at all.

And I've explained twice now that 2 is wrong -- these changes do exactly
what was requested:

  1) It allows testing without writing to $HOME.  (This has nothing to
  do with --batch -- testing happens in interactive Emacsen, too.)

  2) It allows people to run an AOT Emacs without triggering further
  compilation.

If you have a suggestion for an alternative change that achieves these
two things, I'm all ears.  (But if your objection is "people shouldn't
want those two things", I'm down to just two ears again.)

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>   1) It allows testing without writing to $HOME.  (This has nothing to
>   do with --batch -- testing happens in interactive Emacsen, too.)

(Which reminds me -- one thing I've been pondering for a while is
whether "emacs -Q" should imply having the JIT off or not.  We use "-Q"
to get a repeatable Emacs for users, so it would make some sense to have
-Q now imply `inhibit-automatic-native-compilation'.  But I'm not sure.
If it does, should it also imply `load-no-native', which should then be
renamed to something less awkward and documented?  Probably not -- but
should there then be yet another twiddle to inhibit from
~/.emacs/eln-cache/?  That seems like overkill, probably, but it would
be in the spirit of "-Q".)

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> This is not the case at all, please trust me, your changeset does two
>> things:
>>
>> 1- change the name a knob, but it goes from a maybe un-intuitive one to
>>    just (as explained) a plain wrong one.
>
> It goes from an un-intuitive (and undocumented) one to an
> intuitively-named (and documented) one.

The documentation topic is orthogonal, you could have improved
`native-comp-deferred-compilation' documentation if you wanted.

The naming went first to `inhibit-native-compilation', this was really
just sooo wrong in so many levels :/ So many as many knobs we have to
control and disable different parts of the native compiler.  I can't
really think of one understanding all this machinery and then picking up
this name sorry, I can't.

Then the name became `inhibit-automatic-native-compilation' that is
fortunately just wrong on one level :/

The original name was the name of one of the two mechanisms concurring
to native compilation, that is the reason why it was just not called
"automatic".

> In my opinion, the old variable
> is just as wrong as the current one (because it seemed to imply that
> compilation would be immediate instead of deferred).

`native-comp-deferred-compilation' implies that native compilation can
happen deferred.  But anyway this name was reviewed and the name used
here countless times for long time.  Changing it with zero discussion is
just not good cooperation spirit, sorry.  You can think of it
differently but you just risk to remain alone doing development.

>> 2- add a mechanism that (as explained) cannot help with the user request
>>    in this discussion at all.
>
> And I've explained twice now that 2 is wrong -- these changes do exactly
> what was requested:
>
>   1) It allows testing without writing to $HOME.  (This has nothing to
>   do with --batch -- testing happens in interactive Emacsen, too.)

The user request is for non interactive sessions AFAIU.  And still I've
to understand exactly what the user wants to solve.  Most importantly I
feel I'm not alone here.

>   2) It allows people to run an AOT Emacs without triggering further
>   compilation.

Sorry as changeset I meant 5fec9182db + f97993ee66.  I'm not against
e245c4f226.

> If you have a suggestion for an alternative change that achieves these
> two things, I'm all ears.  (But if your objection is "people shouldn't
> want those two things", I'm down to just two ears again.)

Again 5fec9182db + f97993ee66 are IMO not useful / wrong, they were not
discussed and are just a step backward.  The best change I can suggest
is to revert them now.

Thanks

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>>   1) It allows testing without writing to $HOME.  (This has nothing to
>>   do with --batch -- testing happens in interactive Emacsen, too.)
>
> (Which reminds me -- one thing I've been pondering for a while is
> whether "emacs -Q" should imply having the JIT off or not.  We use "-Q"
> to get a repeatable Emacs for users, so it would make some sense to have
> -Q now imply `inhibit-automatic-native-compilation'.

Noooo please!! -Q has just one single clear meaning, let please not add
other implications to it!!  The situation would become very quickly
unmanageable.

And BTW -Q ATM is usefull for testing the native compiler as well :/ :/

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> The naming went first to `inhibit-native-compilation', this was really
> just sooo wrong in so many levels :/ So many as many knobs we have to
> control and disable different parts of the native compiler.  I can't
> really think of one understanding all this machinery and then picking up
> this name sorry, I can't.

Naming is the most difficult thing in computing, after all.

The doc string to the variable explained what it does just fine, so I
think your understanding needs a check-up.  (And I don't appreciated
being talked to in this tone, in case you wondered.)

>>   1) It allows testing without writing to $HOME.  (This has nothing to
>>   do with --batch -- testing happens in interactive Emacsen, too.)
>
> The user request is for non interactive sessions AFAIU.  And still I've
> to understand exactly what the user wants to solve.  Most importantly I
> feel I'm not alone here.

And I'm telling you that's not all that was requested for...  the third
time?  I'm not sure I'm getting any further here by repeating myself, so
I think I'll stop.

>>   2) It allows people to run an AOT Emacs without triggering further
>>   compilation.
>
> Sorry as changeset I meant 5fec9182db + f97993ee66.  I'm not against
> e245c4f226.
>
>> If you have a suggestion for an alternative change that achieves these
>> two things, I'm all ears.  (But if your objection is "people shouldn't
>> want those two things", I'm down to just two ears again.)
>
> Again 5fec9182db + f97993ee66 are IMO not useful / wrong, they were not
> discussed and are just a step backward.  The best change I can suggest
> is to revert them now.

You're just repeating that you're against it without proposing an
alternative, which is not helpful, so I think we've come to the of the
discussion on this point, too.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>> (Which reminds me -- one thing I've been pondering for a while is
>> whether "emacs -Q" should imply having the JIT off or not.  We use "-Q"
>> to get a repeatable Emacs for users, so it would make some sense to have
>> -Q now imply `inhibit-automatic-native-compilation'.
>
> Noooo please!! -Q has just one single clear meaning, let please not add
> other implications to it!!  The situation would become very quickly
> unmanageable.

Yes, that one clear meaning is:

              -Q, --quick
                      Similar to "-q --no-site-file --no-splash".  Also, avoid
                      processing X resources.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> The naming went first to `inhibit-native-compilation', this was really
>> just sooo wrong in so many levels :/ So many as many knobs we have to
>> control and disable different parts of the native compiler.  I can't
>> really think of one understanding all this machinery and then picking up
>> this name sorry, I can't.
>
> Naming is the most difficult thing in computing, after all.
>
> The doc string to the variable explained what it does just fine, so I
> think your understanding needs a check-up.  (And I don't appreciated
> being talked to in this tone, in case you wondered.)
>
>>>   1) It allows testing without writing to $HOME.  (This has nothing to
>>>   do with --batch -- testing happens in interactive Emacsen, too.)
>>
>> The user request is for non interactive sessions AFAIU.  And still I've
>> to understand exactly what the user wants to solve.  Most importantly I
>> feel I'm not alone here.
>
> And I'm telling you that's not all that was requested for...  the third
> time?  I'm not sure I'm getting any further here by repeating myself, so
> I think I'll stop.
>
>>>   2) It allows people to run an AOT Emacs without triggering further
>>>   compilation.
>>
>> Sorry as changeset I meant 5fec9182db + f97993ee66.  I'm not against
>> e245c4f226.
>>
>>> If you have a suggestion for an alternative change that achieves these
>>> two things, I'm all ears.  (But if your objection is "people shouldn't
>>> want those two things", I'm down to just two ears again.)
>>
>> Again 5fec9182db + f97993ee66 are IMO not useful / wrong, they were not
>> discussed and are just a step backward.  The best change I can suggest
>> is to revert them now.
>
> You're just repeating that you're against it without proposing an
> alternative, which is not helpful, so I think we've come to the of the
> discussion on this point, too.

You asked for an ameliorative change, I gave you one, that is just to
revert it.  If you don't like it I don't know what to do, I'll not
speculate on your psychology as you did with mine ;)

Best Regards

  Andrea

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: akrl@sdf.org,  rlb@defaultvalue.org,  monnier@iro.umontreal.ca,
>   david@tethera.net,  emacs-devel@gnu.org
> Date: Wed, 05 Oct 2022 16:27:38 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> and "I want to use a AOT-compiled Emacs, but don't do any further
> >> JIT compilation while running Emacs").
> >
> > This use case makes no sense at all.
> 
> It's been requested several times, whether it makes sense to you or not.

Where it was requested?  Please point me to those requests.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org
> Date: Wed, 05 Oct 2022 16:29:29 +0200
> 
> > I've the perception that this change was done without the full picture
> > in mind of how the native compiler and his mechanisms works.  As a
> > result the current naming is IMO just wrong, and as such is a step
> > backward the original state.
> 
> I don't know where you got that perception from.

It's clear as daylight that this is what happened.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: spwhitton@spwhitton.name,  rlb@defaultvalue.org,
>  monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org,
>  akrl@sdf.org
> Date: Wed, 05 Oct 2022 21:58:02 +0800
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> Maybe you should customize native-comp-async-jobs-number to a small
> >> enough value, like 1?  The default zero means half of your CPU's
> >> execution units.
> >
> > I will try that and see how it goes, thanks.
> 
> That makes the fans less loud (they are still noticable), but it also
> takes twice as long for the fans to subside, as expected.

What happens with those fans when you build Emacs with "make -jN"? do
they sound the same or louder?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  spwhitton@spwhitton.name,
>   rlb@defaultvalue.org,  monnier@iro.umontreal.ca,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Wed, 05 Oct 2022 17:02:45 +0200
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > That makes the fans less loud (they are still noticable), but it also
> > takes twice as long for the fans to subside, as expected.
> >
> > Hope this data point ends up meaningful.
> 
> Yes.
> 
> Software is distributed pre-compiled for a reason -- because people run
> the software on hardware where compiling the software takes a long time.
> It's entirely reasonable for people to want to have a fully-built
> native-compiled Emacs installation on their laptops, without making that
> Emacs do any further JIT compilation.  (Except where necessary for
> trampolines, of course.)

Why would people want to have N files compiled, but not the N+1st
file?  How are the first N files different from the N+1st?

> This has been requested a number of times over several years now, but
> these requests have been ignored because apparently "people shouldn't
> want that".

That's an incorrect and unfair accusation, so please stop.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Where it was requested?  Please point me to those requests.

Sorry, I don't have a list -- people have been mentioning this for
years.

Po Lu seemed to request it just now, though.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: akrl@sdf.org,  rlb@defaultvalue.org,  monnier@iro.umontreal.ca,
>   david@tethera.net,  emacs-devel@gnu.org
> Date: Wed, 05 Oct 2022 19:08:00 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Where it was requested?  Please point me to those requests.
> 
> Sorry, I don't have a list -- people have been mentioning this for
> years.

I don't remember even a single request like that.  I suspect there's a
confusion of sorts.

> Po Lu seemed to request it just now, though.

He did?

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>> (Which reminds me -- one thing I've been pondering for a while is
>>> whether "emacs -Q" should imply having the JIT off or not.  We use "-Q"
>>> to get a repeatable Emacs for users, so it would make some sense to have
>>> -Q now imply `inhibit-automatic-native-compilation'.
>>
>> Noooo please!! -Q has just one single clear meaning, let please not add
>> other implications to it!!  The situation would become very quickly
>> unmanageable.
>
> Yes, that one clear meaning is:
>
>               -Q, --quick
>                       Similar to "-q --no-site-file --no-splash".  Also, avoid
>                       processing X resources.

Exactly, adding complexity involving the execution engine here is IMO
just searching for troubles (and having -Q not usable for debugging the
execution engine itself).  Let's please do not write patches solving
issues we never encountered or implementing unrequested features.

  Andrea

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2557 bytes --]

On Wed, Oct 05, 2022 at 07:47:58PM +0300, Eli Zaretskii wrote:
> > From: Lars Ingebrigtsen <larsi@gnus.org>
> > Cc: Eli Zaretskii <eliz@gnu.org>,  spwhitton@spwhitton.name,
> >   rlb@defaultvalue.org,  monnier@iro.umontreal.ca,  david@tethera.net,
> >   emacs-devel@gnu.org,  akrl@sdf.org
> > Date: Wed, 05 Oct 2022 17:02:45 +0200
> > 
> > Po Lu <luangruo@yahoo.com> writes:
> > 
> > > That makes the fans less loud (they are still noticable), but it also
> > > takes twice as long for the fans to subside, as expected.
> > >
> > > Hope this data point ends up meaningful.
> > 
> > Yes.
> > 
> > Software is distributed pre-compiled for a reason -- because people run
> > the software on hardware where compiling the software takes a long time.
> > It's entirely reasonable for people to want to have a fully-built
> > native-compiled Emacs installation on their laptops, without making that
> > Emacs do any further JIT compilation.  (Except where necessary for
> > trampolines, of course.)
> 
> Why would people want to have N files compiled, but not the N+1st
> file?  How are the first N files different from the N+1st?

Perhaps because 1500 <= N <= 2000 (or so) and the N+1st ist just one?
Perhaps because in the "normal case", the N+1st won't even happen?

The idea of a Debian package is to provide a baseline for those
just interested in using that software (with an easy ramp to
upgrade to actually hack at the sources and build a modified
version).

I actually install a few packages from source, those I "personally"
care about (Emacs is among them), But I couldn't possibly do it for
the > 2000 currently installed on my system.

The (Debian) packager's job is to make sure all that stuff works
nicely (and as repeatably as possible) together, and still receive
the necessary security fixes.

Perhaps it would make sense for the Debian Emacs packagers (Rob?)
to state their requirements in some more abstract way and work
from there.

As far as I understand, the wishes are:

 (a) deliver a package with (all? as many as possible? most? .eln
  pre-compiled
 (b) build Emacs in a way that is idempotent (and doesn't change
  overall system state
 (c) perhaps run tests (possibly, ideally part of b)

Did I forget anything?

Cheers
-- 
tomás
> 
> > This has been requested a number of times over several years now, but
> > these requests have been ignored because apparently "people shouldn't
> > want that".
> 
> That's an incorrect and unfair accusation, so please stop.
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Exactly, adding complexity involving the execution engine here is IMO
> just searching for troubles (and having -Q not usable for debugging the
> execution engine itself). 

You missed my point.  "-Q" doesn't have "one clear meaning" -- it's a
mish-mash of stuff, affecting how many things in Emacs works (like
Customize, X resources, packages, and so on).

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Wed, 5 Oct 2022 19:59:38 +0200
> From: <tomas@tuxteam.de>
> 
> > Why would people want to have N files compiled, but not the N+1st
> > file?  How are the first N files different from the N+1st?
> 
> Perhaps because 1500 <= N <= 2000 (or so) and the N+1st ist just one?

Then it isn't the same N, is it?

> Perhaps because in the "normal case", the N+1st won't even happen?

Why disable something that will never happen?

> The idea of a Debian package is to provide

I think the Debian case is not relevant, because they provide all the
*.eln files with the package you install (or so I understand).  So
either the user will only use those packages where *.eln are already
provided (in which case there's no reason to disable anything), or
they will want to use packages outside of the Debian distribution, in
which case I'm asking why they would like to give up on compiling
those additional ones.

> I actually install a few packages from source, those I "personally"
> care about (Emacs is among them), But I couldn't possibly do it for
> the > 2000 currently installed on my system.

What is "it" in "do it" above?  And what does the 2000 number counts
here? are you talking about 2000 Emacs Lisp packages that are not
bundled with the Emacs distribution and need to be installed
separately?  Or are you counting some other kind of "packages"?

> As far as I understand, the wishes are:
> 
>  (a) deliver a package with (all? as many as possible? most? .eln
>   pre-compiled
>  (b) build Emacs in a way that is idempotent (and doesn't change
>   overall system state
>  (c) perhaps run tests (possibly, ideally part of b)

I don't see how disabling JIT compilation is needed for these 3
purposes.  In particular, if all the *.eln files are already present,
there will be no need to recompile them.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3893 bytes --]

On Wed, Oct 05, 2022 at 09:28:55PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 5 Oct 2022 19:59:38 +0200
> > From: <tomas@tuxteam.de>
> > 
> > > Why would people want to have N files compiled, but not the N+1st
> > > file?  How are the first N files different from the N+1st?
> > 
> > Perhaps because 1500 <= N <= 2000 (or so) and the N+1st ist just one?
> 
> Then it isn't the same N, is it?

- Base .el(c)s pre-compiled. N.
- Additional .el (perhaps written by the user, perhaps
 downloaded from Emacswiki or what not) (jit) compiled,
 go to somewhere writable by the user (perhaps somewhere
 under ~/.emacs.d). 1. Or 2.

> > Perhaps because in the "normal case", the N+1st won't even happen?
> 
> Why disable something that will never happen?

...in the normal case. The user should be still capable of tackling
the not-that-normal case.

> > The idea of a Debian package is to provide
> 
> I think the Debian case is not relevant, because they provide all the
> *.eln files with the package you install (or so I understand).  So
> either the user will only use those packages where *.eln are already
> provided (in which case there's no reason to disable anything), or
> they will want to use packages outside of the Debian distribution, in
> which case I'm asking why they would like to give up on compiling
> those additional ones.

It seems we are in violent agreement, then. Whatever the user installs
"outside" the Debian packaging system is the user's business. It is
desirable that it works well together with the Debian-provided baseline
(and Debian tries to make that possible).

> > I actually install a few packages from source, those I "personally"
> > care about (Emacs is among them), But I couldn't possibly do it for
> > the > 2000 currently installed on my system.
> 
> What is "it" in "do it" above?

Installing software from source.

>   And what does the 2000 number counts
> here?

Debian packages: The Gnu compiler toolchain. The mail reader (mutt in my
case). A mail transfer agent (Exim). A Web browser. A web server. Lots
of different scripting languages. Networking stuff. SSH, rsync, git.
Qemu. Cross build tools. X and all those little helpers. R, Octave, I
don't know, all that stuff one needs to be happy :-)

>  are you talking about 2000 Emacs Lisp packages that are not
> bundled with the Emacs distribution and need to be installed
> separately?  Or are you counting some other kind of "packages"?

No, Debian packages.

> > As far as I understand, the wishes are:
> > 
> >  (a) deliver a package with (all? as many as possible? most? .eln
> >   pre-compiled
> >  (b) build Emacs in a way that is idempotent (and doesn't change
> >   overall system state
> >  (c) perhaps run tests (possibly, ideally part of b)
> 
> I don't see how disabling JIT compilation is needed for these 3
> purposes.  In particular, if all the *.eln files are already present,
> there will be no need to recompile them.

I don't know exactly how the Debian packaging for Emacs proceeds, but
I think we are talking about two distinct situations here:

(a) the binary install for the end user (for which the target is to
   end up with (some, most) .eln files pre-compiled and typically
   in /usr/lib or /usr/share, depending on whether those files are
   architecture-dependent (I think they are) or not

(b) the package build, which happens at the Debian developer's
   box to build the Debian package to be installed to achieve
   (a).

The question is whether the pre-compiling of the .eln happens at
(a) (i.e. package install time) or at (b). It seems Rob has opted
for the first (perhaps because there are dependencies which have
to be resolved "late"?).

Anyway, this "disabling of JIT" would be relevant for (b), if
at all (assuming I've been paying attention).

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Wed, 5 Oct 2022 20:56:00 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > > > Why would people want to have N files compiled, but not the N+1st
> > > > file?  How are the first N files different from the N+1st?
> > > 
> > > Perhaps because 1500 <= N <= 2000 (or so) and the N+1st ist just one?
> > 
> > Then it isn't the same N, is it?
> 
> - Base .el(c)s pre-compiled. N.
> - Additional .el (perhaps written by the user, perhaps
>  downloaded from Emacswiki or what not) (jit) compiled,
>  go to somewhere writable by the user (perhaps somewhere
>  under ~/.emacs.d). 1. Or 2.

Then I ask again: why wouldn't the user want to have the addition .el
compiled to .eln?

> > > Perhaps because in the "normal case", the N+1st won't even happen?
> > 
> > Why disable something that will never happen?
> 
> ...in the normal case. The user should be still capable of tackling
> the not-that-normal case.

The not-that-normal case being user's own *.el files?  Why wouldn't
the user want to compile them into *.eln in that user's own eln-cache?
Why would that user want to disable native-compilation instead?

> > > I actually install a few packages from source, those I "personally"
> > > care about (Emacs is among them), But I couldn't possibly do it for
> > > the > 2000 currently installed on my system.
> > 
> > What is "it" in "do it" above?
> 
> Installing software from source.
> 
> >   And what does the 2000 number counts
> > here?
> 
> Debian packages: The Gnu compiler toolchain. The mail reader (mutt in my
> case). A mail transfer agent (Exim). A Web browser. A web server. Lots
> of different scripting languages. Networking stuff. SSH, rsync, git.
> Qemu. Cross build tools. X and all those little helpers. R, Octave, I
> don't know, all that stuff one needs to be happy :-)

How is this relevant to the issue at hand?  Only Emacs comes with
*.eln files and can use them.  Why are we talking about all the other
packages?

> > I don't see how disabling JIT compilation is needed for these 3
> > purposes.  In particular, if all the *.eln files are already present,
> > there will be no need to recompile them.
> 
> I don't know exactly how the Debian packaging for Emacs proceeds, but
> I think we are talking about two distinct situations here:
> 
> (a) the binary install for the end user (for which the target is to
>    end up with (some, most) .eln files pre-compiled and typically
>    in /usr/lib or /usr/share, depending on whether those files are
>    architecture-dependent (I think they are) or not
> 
> (b) the package build, which happens at the Debian developer's
>    box to build the Debian package to be installed to achieve
>    (a).
> 
> The question is whether the pre-compiling of the .eln happens at
> (a) (i.e. package install time) or at (b). It seems Rob has opted
> for the first (perhaps because there are dependencies which have
> to be resolved "late"?).
> 
> Anyway, this "disabling of JIT" would be relevant for (b), if
> at all (assuming I've been paying attention).

For case (b) Rob said that the workspace is thrown away after the
build, so whether the *.eln files are or aren't built there makes no
difference.  (And if the package build is done in batch mode, the
async compilation is disabled there by default.)

So, once again, I see no reason to disable JIT compilation for these
use cases.

Re: Suppressing native compilation (short and long term)

[Gregory Heytings]

>
> I think the Debian case is not relevant, because they provide all the 
> *.eln files with the package you install (or so I understand).
>

I'm not 100% sure what they will do (the version of Emacs distributed by 
Emacs is at the moment still 27.1), but my guess is that this is not what 
they will do.  There are, in fact, two cases:

1. When a user does "apt install emacs", this actually installs (by 
default) the "emacs-gtk" package, which contains only the "emacs" binary, 
and triggers the installation of two other packages: "emacs-common", which 
contains the precompiled elc files (and the files in etc), and 
"emacs-bin-common", which contains the emacsclient, etags, ctags and 
ebrowse binaries.  I would guess that in this case, when the user chooses 
to install an emacs with native compilation enabled (say 
"emacs-gtk-native"), a third package will be installed, say 
"emacs-common-native", containing the precompiled eln files.

2. When a user does "apt install elpa-magit" (for example), the package 
only contains el files.  These files are compiled to elc files during 
installation (and stored in a shared directory, namely 
/usr/share/emacs/site-lisp).  I would guess that, when the installed emacs 
binary is one with native compilation enabled, these el files will be 
compiled to eln files during installation, and stored in a shared 
directory, too.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Wed, 05 Oct 2022 19:29:45 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: tomas@tuxteam.de, emacs-devel@gnu.org
> 
> > I think the Debian case is not relevant, because they provide all the 
> > *.eln files with the package you install (or so I understand).
> 
> I'm not 100% sure what they will do (the version of Emacs distributed by 
> Emacs is at the moment still 27.1), but my guess is that this is not what 
> they will do.  There are, in fact, two cases:
> 
> 1. When a user does "apt install emacs", this actually installs (by 
> default) the "emacs-gtk" package, which contains only the "emacs" binary, 
> and triggers the installation of two other packages: "emacs-common", which 
> contains the precompiled elc files (and the files in etc), and 
> "emacs-bin-common", which contains the emacsclient, etags, ctags and 
> ebrowse binaries.  I would guess that in this case, when the user chooses 
> to install an emacs with native compilation enabled (say 
> "emacs-gtk-native"), a third package will be installed, say 
> "emacs-common-native", containing the precompiled eln files.
> 
> 2. When a user does "apt install elpa-magit" (for example), the package 
> only contains el files.  These files are compiled to elc files during 
> installation (and stored in a shared directory, namely 
> /usr/share/emacs/site-lisp).  I would guess that, when the installed emacs 
> binary is one with native compilation enabled, these el files will be 
> compiled to eln files during installation, and stored in a shared 
> directory, too.

I agree with the above.  But then why did you say that my description
was inaccurate?  What you described matches what I wrote perfectly:
the end result, after installing Emacs and elpa-magit, is that the
*.eln files are available for all the *.el files and stored in shared
directories.  Whether those *.eln files are produced on the system
where the package is packaged or as part of the installation is not
important; the important part is that all the *.eln files are there
after the installation, and therefore there's no need to disable JIT
compilation.  And yet Rob says that he thinks there _is_ a need for
disabling it.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2824 bytes --]

On Wed, Oct 05, 2022 at 10:04:17PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 5 Oct 2022 20:56:00 +0200
> > From: tomas@tuxteam.de
> > Cc: emacs-devel@gnu.org
> > 
> > > > > Why would people want to have N files compiled, but not the N+1st
> > > > > file?  How are the first N files different from the N+1st?
> > > > 
> > > > Perhaps because 1500 <= N <= 2000 (or so) and the N+1st ist just one?
> > > 
> > > Then it isn't the same N, is it?
> > 
> > - Base .el(c)s pre-compiled. N.
> > - Additional .el (perhaps written by the user, perhaps
> >  downloaded from Emacswiki or what not) (jit) compiled,
> >  go to somewhere writable by the user (perhaps somewhere
> >  under ~/.emacs.d). 1. Or 2.
> 
> Then I ask again: why wouldn't the user want to have the addition .el
> compiled to .eln?

I think nobody is proposing to prevent the user from doing that.

> > > > Perhaps because in the "normal case", the N+1st won't even happen?
> > > 
> > > Why disable something that will never happen?
> > 
> > ...in the normal case. The user should be still capable of tackling
> > the not-that-normal case.
> 
> The not-that-normal case being user's own *.el files?  Why wouldn't
> the user want to compile them into *.eln in that user's own eln-cache?
> Why would that user want to disable native-compilation instead?

See above. This must be (part of) the misunderstanding. See above
my text, perhaps it was unclear:

  "Additional .el [...] (jit) compiled go to somewhere writable
   by the user"

We are in violent agreement here, I think.

> > > > I actually install a few packages from source, those I "personally"
> > > > care about [...]

> How is this relevant to the issue at hand?  Only Emacs comes with
> *.eln files and can use them.  Why are we talking about all the other
> packages?

Just trying to explain how the Debian packaging works and what it is
good for. Your questions seem to suggest that this is somewhat alien
to you (but my impression might be wrong, too).

> > > I don't see how disabling JIT compilation is needed for these 3
> > > purposes.  In particular, if all the *.eln files are already present,
> > > there will be no need to recompile them.

[...]

> For case (b) Rob said that the workspace is thrown away after the
> build, so whether the *.eln files are or aren't built there makes no
> difference.  (And if the package build is done in batch mode, the
> async compilation is disabled there by default.)

As I understant, this might work well if it's possible to redirect/
restrict those target directories to specific places, yes. But Rob
will know much better than I could.

> So, once again, I see no reason to disable JIT compilation for these
> use cases.

Perhaps we can get away with it, yes.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Gregory Heytings]

>
> I agree with the above.  But then why did you say that my description 
> was inaccurate?
>

I did not say that, or at least I did not mean to say that.  I only tried 
to clarify something that seemed unclear, namely that eln files are not 
present in the package, but generated at installation time.  It's a subtle 
difference, but it seems important in the current discussion.

>
> What you described matches what I wrote perfectly: the end result, after 
> installing Emacs and elpa-magit, is that the *.eln files are available 
> for all the *.el files and stored in shared directories.  Whether those 
> *.eln files are produced on the system where the package is packaged or 
> as part of the installation is not important; the important part is that 
> all the *.eln files are there after the installation, and therefore 
> there's no need to disable JIT compilation.  And yet Rob says that he 
> thinks there _is_ a need for disabling it.
>

If I understand Rob's initial message correctly, this subtle difference is 
relevant:

>
> Perhaps relevant in other contexts, but at least in the context of 
> Debian work, we'd like to be able to suppress all native compilation in 
> some contexts, or more specifically all writes to HOME (or really, to 
> avoid any implicit file manipulations[1]).
>
> One key case is package builds and package installs (whether the emacs 
> packages themselves, or any of the various emacs add-on packages (e.g. 
> elpa-*)).
>
> For package builds, HOME is typically set to /nonexistent (or similar), 
> and for package installs we don't want the install commands (preinst, 
> postinst, etc.), which are running as root, to write anything to /root/.
>

IIUC, what Rob wants is that

(1) on Debian systems on which Emacs packages are built, the build process 
is running as root, and nothing should be written in /root, and

(2) on user systems on which elpa-* packages are installed, the 
installation process, during which el files are compiled to elc and eln 
files, is running as root, and it should not write anything in /root.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Also repeatable testing are most likely executed in batch mode and...
>
> ...surprise surprise deferred compilation is *already* *disabled* in
> this mode!!

I forgot to mention that while this is (strictly speaking) true, if the
--batch Emacs redefines any built-in functions, the trampoline files
will get written to ~/.emacs.d/eln-cache/.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Exactly, adding complexity involving the execution engine here is IMO
>> just searching for troubles (and having -Q not usable for debugging the
>> execution engine itself). 
>
> You missed my point.  "-Q" doesn't have "one clear meaning" -- it's a
> mish-mash of stuff, affecting how many things in Emacs works (like
> Customize, X resources, packages, and so on).

Still I think none of these as anything to do with the execution engine
(and should not).

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>> You missed my point.  "-Q" doesn't have "one clear meaning" -- it's a
>> mish-mash of stuff, affecting how many things in Emacs works (like
>> Customize, X resources, packages, and so on).
>
> Still I think none of these as anything to do with the execution engine
> (and should not).

Did I say they did?  You claimed that "-Q" has "one clear meaning", and
that's false.  "-Q" means, sort of, "without any local changes", which
is really wishy-washy semantics.

Which reminds me of another thing -- was there a reason that --batch
implies "avoid most JIT compilation", like it does now, by the way?
It's always seemed pretty odd to me, because JITting could well be quite
useful when doing batch stuff, and there's no handy way to switch it on
when doing --batch.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Also repeatable testing are most likely executed in batch mode and...
>>
>> ...surprise surprise deferred compilation is *already* *disabled* in
>> this mode!!
>
> I forgot to mention that while this is (strictly speaking) true,

Not sure what you mean by strictly speaking, I think is just true.

> if the
> --batch Emacs redefines any built-in functions, the trampoline files
> will get written to ~/.emacs.d/eln-cache/.

Indeed, and I don't find anything surprising with that.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>>> Also repeatable testing are most likely executed in batch mode and...
>>>
>>> ...surprise surprise deferred compilation is *already* *disabled* in
>>> this mode!!
>>
>> I forgot to mention that while this is (strictly speaking) true,
>
> Not sure what you mean by strictly speaking, I think is just true.
>
>> if the
>> --batch Emacs redefines any built-in functions, the trampoline files
>> will get written to ~/.emacs.d/eln-cache/.
>
> Indeed, and I don't find anything surprising with that.

I'm sure you don't.  But you claimed that --batch was the panacea for
repeatable testing -- and it's not, since the second time you run the
same test, the trampoline files are created.  I.e., not a repeat of the
same test.

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> What happens with those fans when you build Emacs with "make -jN"? do
> they sound the same or louder?

Louder, but building Emacs doesn't happen in the background
automatically.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>>> Also repeatable testing are most likely executed in batch mode and...
>>>>
>>>> ...surprise surprise deferred compilation is *already* *disabled* in
>>>> this mode!!
>>>
>>> I forgot to mention that while this is (strictly speaking) true,
>>
>> Not sure what you mean by strictly speaking, I think is just true.
>>
>>> if the
>>> --batch Emacs redefines any built-in functions, the trampoline files
>>> will get written to ~/.emacs.d/eln-cache/.
>>
>> Indeed, and I don't find anything surprising with that.
>
> I'm sure you don't.  But you claimed that --batch was the panacea for
> repeatable testing

Did I??

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>>>>> Also repeatable testing are most likely executed in batch mode and...
>>>>>
>>>>> ...surprise surprise deferred compilation is *already* *disabled* in
>>>>> this mode!!

[...]

>> I'm sure you don't.  But you claimed that --batch was the panacea for
>> repeatable testing
>
> Did I??

Yes.  *points*

Re: Suppressing native compilation (short and long term)

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Po Lu seemed to request it just now, though.

Nope, my solution is to wait the 30 minutes for the fans to subside
after startup, every time I update Emacs.

I think the end result and time spent is more-or-less the same as a
NATIVE_FULL_AOT build of Emacs.

Thanks.

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> He did?

Not really, but I build my own Emacs.  The difference may be more
meaningful for packagers.

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,
>>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org
>> Date: Wed, 05 Oct 2022 16:29:29 +0200
>> 
>> > I've the perception that this change was done without the full picture
>> > in mind of how the native compiler and his mechanisms works.  As a
>> > result the current naming is IMO just wrong, and as such is a step
>> > backward the original state.
>> 
>> I don't know where you got that perception from.
>
> It's clear as daylight that this is what happened.

Since the argument seems to be going nowhere, could we first revert the
change in question, and then ask Rob Browning exactly what the problem
is with `native-comp-deferred-compilation'?

I did not figure that out reading this thread.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>> You missed my point.  "-Q" doesn't have "one clear meaning" -- it's a
>>> mish-mash of stuff, affecting how many things in Emacs works (like
>>> Customize, X resources, packages, and so on).
>>
>> Still I think none of these as anything to do with the execution engine
>> (and should not).
>
> Did I say they did?

No you didn't nor I claimed you did.  I'm saying -Q does control a
totally different area of Emacs that is not the execution engine.

For instance it does not affect bytecode execution, consequentially I
don't see why it should affect native code execution.

> You claimed that "-Q" has "one clear meaning", and
> that's false.  "-Q" means, sort of, "without any local changes", which
> is really wishy-washy semantics.

Sorry for me this meaning is clear.

> Which reminds me of another thing -- was there a reason that --batch
> implies "avoid most JIT compilation", like it does now, by the way?
> It's always seemed pretty odd to me, because JITting could well be quite
> useful when doing batch stuff, and there's no handy way to switch it on
> when doing --batch.

The rational is that "tipically" batch executions are short in time, so
spawning there native async compilation would be a waste of cycles if
they do not complete in time.  I think no one has statistical prove of
this, but the rational was at least discussed and I believe agreed here.

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> The rational is that "tipically" batch executions are short in time, so
> spawning there native async compilation would be a waste of cycles if
> they do not complete in time.  I think no one has statistical prove of
> this, but the rational was at least discussed and I believe agreed here.

Sure, sounds reasonable as a default, at least.  But perhaps there
should be a way to allow JIT in --batch?  It looks like currently the
only way is to set `noninteractive', which has wider side effects.

Yet Another Variable here would be possible, but perhaps some other
mechanism would be nicer.  Hm...  Perhaps
inhibit-automatic-native-compilation should be refashioned into
something else, like `native-compilation-types' or something, and would
default to 'all in non-batch, but could be 'force to force it even in
--batch?  (And possibly also 'trampoline to save trampolines without
doing any more JIT-it, like Stefan M sort of suggested.)

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> Since the argument seems to be going nowhere, could we first revert the
> change in question, and then ask Rob Browning exactly what the problem
> is with `native-comp-deferred-compilation'?
>
> I did not figure that out reading this thread.

I don't know whether this was Rob's problem, but I found it problematic
that there was no way to not write to ~/.emacs.d/eln-cache.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>>>>> Also repeatable testing are most likely executed in batch mode and...
>>>>>>
>>>>>> ...surprise surprise deferred compilation is *already* *disabled* in
>>>>>> this mode!!
>
> [...]
>
>>> I'm sure you don't.  But you claimed that --batch was the panacea for
>>> repeatable testing
>>
>> Did I??
>
> Yes.  *points*

Where?

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>>>>>>> Also repeatable testing are most likely executed in batch mode and...
>>>>>>>
>>>>>>> ...surprise surprise deferred compilation is *already* *disabled* in
>>>>>>> this mode!!

[...]

> Where?

There.

But if you were just doing another non-sequitur -- that you didn't mean
that "deferred compilation" and --batch is related to repeatable
testing, that's fine.  You were just making conversation and not
claiming anything?

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>>>>>>> Also repeatable testing are most likely executed in batch mode and...
>>>>>>>>
>>>>>>>> ...surprise surprise deferred compilation is *already* *disabled* in
>>>>>>>> this mode!!
>
> [...]
>
>> Where?
>
> There.

Oh my!  FYI this is what we wrote:

> >>   . understand why and in which situations they may need it
> >
> > Doing repeatable testing is one obvious situation.
> 
> I think the two previously mentioned knobs should be sufficient for
> repeatable testing no?

As you can see I said that the *two* mentioned knobs should be
sufficient for test repeatability, not that "--batch is the panacea for
repeatable testing".

> But if you were just doing another non-sequitur -- that you didn't mean
> that "deferred compilation" and --batch is related to repeatable
> testing, that's fine.  You were just making conversation and not
> claiming anything?

Sorry are you serious?

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> As you can see I said that the *two* mentioned knobs should be
> sufficient for test repeatability, not that "--batch is the panacea for
> repeatable testing".

The two mentioned knobs are `native-comp-deferred-compilation' and
`load-no-native'?  And as I've explained, those are not sufficient,
because trampolines are written to ~/.emacs.d/eln-cache even those two
switched off.

And you're being revisionist, because this is what you said:

> I think the two previously mentioned knobs should be sufficient for
> repeatable testing no?
> 
> Also repeatable testing are most likely executed in batch mode and...
> 
> ...surprise surprise deferred compilation is *already* *disabled* in
> this mode!!
> 
> I've the impression no one mentioned this small detail in this huge
> thread, but still we have installed changes to disable a feature for
> debian pkg installation that is in fact already disabled :( :(

The latter is simply not true.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> As you can see I said that the *two* mentioned knobs should be
>> sufficient for test repeatability, not that "--batch is the panacea for
>> repeatable testing".
>
> The two mentioned knobs are `native-comp-deferred-compilation' and
> `load-no-native'?

yes

> And as I've explained, those are not sufficient,
> because trampolines are written to ~/.emacs.d/eln-cache even those two
> switched off.

Sorry in my limited experience in debugging Emacs native code execution
those knobs are the one to be used for this case.  Sure trampolines add
a state (there are probably other states Emacs can generate I'm not
aware) but that said I've never had test repeatability issues with
trampolines.

Anyway I thought we were discussing / trying to solve mainly the user
request.  Is this about test repeatability?  Did we had repeatability
issues with native code I'm not aware?

Repeatable testing [was: Suppressing native compilation...]

[tomas]

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

On Thu, Oct 06, 2022 at 12:34:50AM +0000, Andrea Corallo wrote:
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Andrea Corallo <akrl@sdf.org> writes:

[...]

> > I'm sure you don't.  But you claimed that --batch was the panacea for
> > repeatable testing
> 
> Did I??

Now who did or did not isn't that important. More important is:
how would one get about repeatable testing?

Throw away $HOME?

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Repeatable testing [was: Suppressing native compilation...]

[Emanuel Berg]

tomas wrote:

> how would one get about repeatable testing?

One would define what that is in unambiguous terms ...

Then one would work to realize that bottom-up ...

With no shortcuts to the top!

-- 
underground experts united
https://dataswamp.org/~incal

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Wed, 05 Oct 2022 20:04:34 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: tomas@tuxteam.de, emacs-devel@gnu.org
> 
> IIUC, what Rob wants is that
> 
> (1) on Debian systems on which Emacs packages are built, the build process 
> is running as root, and nothing should be written in /root, and

For this case, Rob said the workspace is thrown away after the build,
so whether the *.eln files are or aren't produced is not relevant.
But if, for some reason, the *.eln files could survive the deletion of
the workspace, then tweaking native-comp-eln-load-path to have /tmp at
the beginning should handle that as well.

> (2) on user systems on which elpa-* packages are installed, the 
> installation process, during which el files are compiled to elc and eln 
> files, is running as root, and it should not write anything in /root.

Since the elpa-* package installation process is supposed to leave the
users of the system with ready *.eln files from the packages being
installed, the installation process should NOT disable
native-compilation.  Instead, it should tweak
native-comp-eln-load-path so that it includes the shared directory
where Debian wants to have the *.eln files of ELPA packages instead or
ahead of the user's eln-cache directory.  This will produce the *.eln
files in the place where Debian wants them, and allow users to use
those packages without worrying about compilation.

IOW, an option to disable native-compilation is not TRT in this case.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: spwhitton@spwhitton.name,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org,
>   akrl@sdf.org
> Date: Thu, 06 Oct 2022 08:34:03 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What happens with those fans when you build Emacs with "make -jN"? do
> > they sound the same or louder?
> 
> Louder, but building Emacs doesn't happen in the background
> automatically.

It does happen in the background, in the sense that Make launches
several processes in parallel, each one of which running in the
background.

As for "automatically", the JIT compilation is not automatic, either.
If Emacs is idle, it will not suddenly decide to compile.

Anyway, what would you suggest as a solution for the problem you
perceive with JIT native-compilation, which would refrain from being
"in the background" and "automatic"?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  akrl@sdf.org,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org
> Date: Thu, 06 Oct 2022 08:40:36 +0800
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Po Lu seemed to request it just now, though.
> 
> Nope, my solution is to wait the 30 minutes for the fans to subside
> after startup, every time I update Emacs.

I'd like to point out that the situation with frequent updates of
Emacs that require recompilation of *.eln files is peculiar to Emacs
development on the master branch, and will generally rare if ever
happen for "normal" users, and even for us developers on the release
branch.

Like I said before, the pattern of JIT compilation in "normal" usage
is that it happens for a few minutes after installing a new version of
Emacs, and thereafter happens only rarely.

> I think the end result and time spent is more-or-less the same as a
> NATIVE_FULL_AOT build of Emacs.

I think NATIVE_FULL_AOT takes more time, because it compiles much more
than an average user will ever use.  (But I don't object to supporting
such a build for people who want it.)

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  akrl@sdf.org,
>   rlb@defaultvalue.org,  monnier@iro.umontreal.ca,  david@tethera.net,
>   emacs-devel@gnu.org
> Date: Thu, 06 Oct 2022 08:44:21 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> Since the argument seems to be going nowhere, could we first revert the
> change in question, and then ask Rob Browning exactly what the problem
> is with `native-comp-deferred-compilation'?

I'm still discussing with Rob his needs.  I hope to eventually
understand their needs.  For now my only conclusion is that they need
to tweak native-comp-eln-load-path in some situations, to control
where the *.eln files are deposited.  Which is easy and supported
already, AFAIU.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, rlb@defaultvalue.org,
>         monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org
> Date: Thu, 06 Oct 2022 00:45:49 +0000
> 
> > Which reminds me of another thing -- was there a reason that --batch
> > implies "avoid most JIT compilation", like it does now, by the way?
> > It's always seemed pretty odd to me, because JITting could well be quite
> > useful when doing batch stuff, and there's no handy way to switch it on
> > when doing --batch.
> 
> The rational is that "tipically" batch executions are short in time, so
> spawning there native async compilation would be a waste of cycles if
> they do not complete in time.  I think no one has statistical prove of
> this, but the rational was at least discussed and I believe agreed here.

Yes, we discussed that, and yes, we agreed.  And I don't think the
decision was wrong.  For starters, it would be strange to delay
exiting Emacs in batch so it could wait for the async compilation to
complete.  Moreover, in most cases waiting will not help, since the
job of the batch invocation would have been long done anyway, so the
produced *.eln files will not be loaded in time to make any
difference.

We do have batch commands to compile *.eln files explicitly (this is
used when a release tarball is built).  This was deemed to be enough,
at least at the time we discussed these issues.  I don't yet see why
we'd need to revisit that decision.  When does it cause problems?

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 185 bytes --]

On Thu, Oct 06, 2022 at 08:28:04AM +0300, Eli Zaretskii wrote:

[...]

> IOW, an option to disable native-compilation is not TRT in this case.

s/TRT/necessary/

;-)

-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  akrl@sdf.org,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org
> Date: Thu, 06 Oct 2022 02:56:54 +0200
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > Since the argument seems to be going nowhere, could we first revert the
> > change in question, and then ask Rob Browning exactly what the problem
> > is with `native-comp-deferred-compilation'?
> >
> > I did not figure that out reading this thread.
> 
> I don't know whether this was Rob's problem, but I found it problematic
> that there was no way to not write to ~/.emacs.d/eln-cache.

But there was such a way: modify native-comp-eln-load-path to have
another directory at the front.  Which, it seems, is what Rob actually
wants, at least in some cases, because they _want_ to keep the *.eln
files, just not in the user's home directory.

Re: Repeatable testing

[Eli Zaretskii]

> Date: Thu, 6 Oct 2022 06:32:08 +0200
> From: <tomas@tuxteam.de>
> 
> Now who did or did not isn't that important. More important is:
> how would one get about repeatable testing?
> 
> Throw away $HOME?

See what we do in the Emacs test suite: we point HOME to a
non-existent directory.  This is useful for other reasons besides
native-compilation, btw.

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Anyway, what would you suggest as a solution for the problem you
> perceive with JIT native-compilation, which would refrain from being
> "in the background" and "automatic"?

The solution I would propose would be to defer JIT native-compilation
until the computer is on AC power, as determined by battery.el.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: spwhitton@spwhitton.name,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org,
>   akrl@sdf.org
> Date: Thu, 06 Oct 2022 15:11:19 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Anyway, what would you suggest as a solution for the problem you
> > perceive with JIT native-compilation, which would refrain from being
> > "in the background" and "automatic"?
> 
> The solution I would propose would be to defer JIT native-compilation
> until the computer is on AC power, as determined by battery.el.

We could have such a feature, but how to implement it?  If we use a
timer for that, the timer itself will drain the battery.  And if defer
it to the next invocation of the bytecode, we might never compile,
because who can guarantee that the laptop is on AC when some arbitrary
bytecode is executed?  Any other ideas?

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

On Thu, Oct 06, 2022 at 11:03:47AM +0300, Eli Zaretskii wrote:
> > From: Po Lu <luangruo@yahoo.com>
> > Cc: spwhitton@spwhitton.name,  rlb@defaultvalue.org,
> >   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org,
> >   akrl@sdf.org
> > Date: Thu, 06 Oct 2022 15:11:19 +0800
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > > Anyway, what would you suggest as a solution for the problem you
> > > perceive with JIT native-compilation, which would refrain from being
> > > "in the background" and "automatic"?
> > 
> > The solution I would propose would be to defer JIT native-compilation
> > until the computer is on AC power, as determined by battery.el.
> 
> We could have such a feature, but how to implement it?  If we use a
> timer for that, the timer itself will drain the battery.

Guessing from a previous mail by you, the correct way to achieve this
is to compiler results to a non-existent directory, right?

To me that feels strange, but I could live with that. Can it be documented?

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> We could have such a feature, but how to implement it?  If we use a
> timer for that, the timer itself will drain the battery.

I think display-battery-mode users (I am one such user) will not agree
with that assessment.

> And if defer it to the next invocation of the bytecode, we might never
> compile, because who can guarantee that the laptop is on AC when some
> arbitrary bytecode is executed?

We could push it onto a list of files to native compile, the files in
which are then compiled once we detect the laptop starts to run on AC
power.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Thu, 6 Oct 2022 11:02:44 +0200
> From: <tomas@tuxteam.de>
> 
> > > The solution I would propose would be to defer JIT native-compilation
> > > until the computer is on AC power, as determined by battery.el.
> > 
> > We could have such a feature, but how to implement it?  If we use a
> > timer for that, the timer itself will drain the battery.
> 
> Guessing from a previous mail by you, the correct way to achieve this
> is to compiler results to a non-existent directory, right?

No, this is a completely different use case.

We _can_ disable native-compilation (and had this ability for a while,
since Emacs 28 development); the issue here is how to re-enable it
again "when computer is on AC power".

> To me that feels strange, but I could live with that. Can it be documented?

What would you like to be documented, exactly?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org,  akrl@sdf.org
> Date: Thu, 06 Oct 2022 17:52:35 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We could have such a feature, but how to implement it?  If we use a
> > timer for that, the timer itself will drain the battery.
> 
> I think display-battery-mode users (I am one such user) will not agree
> with that assessment.

I didn't invent that, I've heard laptop users complain about timers.

But okay, if it's acceptable to run a timer in order to re-enable
compilation when AC power is plugged in, I'm okay with such an
optional feature.  Patches are welcome.

> > And if defer it to the next invocation of the bytecode, we might never
> > compile, because who can guarantee that the laptop is on AC when some
> > arbitrary bytecode is executed?
> 
> We could push it onto a list of files to native compile, the files in
> which are then compiled once we detect the laptop starts to run on AC
> power.

You will see that comp.el already has a queue of files that await
compilation.  The new feature will newed to plug itself into that
mechanism, I think.  Unless Andrea has a better idea.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Po Lu <luangruo@yahoo.com>
>> Cc: emacs-devel@gnu.org,  akrl@sdf.org
>> Date: Thu, 06 Oct 2022 17:52:35 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > We could have such a feature, but how to implement it?  If we use a
>> > timer for that, the timer itself will drain the battery.
>> 
>> I think display-battery-mode users (I am one such user) will not agree
>> with that assessment.
>
> I didn't invent that, I've heard laptop users complain about timers.

For my education: why timer are computational expensive?

> But okay, if it's acceptable to run a timer in order to re-enable
> compilation when AC power is plugged in, I'm okay with such an
> optional feature.  Patches are welcome.
>
>> > And if defer it to the next invocation of the bytecode, we might never
>> > compile, because who can guarantee that the laptop is on AC when some
>> > arbitrary bytecode is executed?
>> 
>> We could push it onto a list of files to native compile, the files in
>> which are then compiled once we detect the laptop starts to run on AC
>> power.
>
> You will see that comp.el already has a queue of files that await
> compilation.  The new feature will newed to plug itself into that
> mechanism, I think.  Unless Andrea has a better idea.

I'd suggest this way as well.

BR

  Andrea

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Po Lu <luangruo@yahoo.com>, emacs-devel@gnu.org
> Date: Thu, 06 Oct 2022 10:41:14 +0000
> 
> >> I think display-battery-mode users (I am one such user) will not agree
> >> with that assessment.
> >
> > I didn't invent that, I've heard laptop users complain about timers.
> 
> For my education: why timer are computational expensive?

They are not, at least not normally.  But they interfere with the OS's
attempts to conserve power, because Emacs is not idle, as far as the
OS is concerned -- it runs some stuff once every so-and-so seconds.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]

On Thu, Oct 06, 2022 at 01:13:40PM +0300, Eli Zaretskii wrote:
> > Date: Thu, 6 Oct 2022 11:02:44 +0200
> > From: <tomas@tuxteam.de>
> > 
> > > > The solution I would propose would be to defer JIT native-compilation
> > > > until the computer is on AC power, as determined by battery.el.
> > > 
> > > We could have such a feature, but how to implement it?  If we use a
> > > timer for that, the timer itself will drain the battery.
> > 
> > Guessing from a previous mail by you, the correct way to achieve this
> > is to compiler results to a non-existent directory, right?
> 
> No, this is a completely different use case.

Now I'm confused. Does the non-existence of a target directory
disable compilation or just the writing of the compilation's
results?

> We _can_ disable native-compilation (and had this ability for a while,
> since Emacs 28 development); the issue here is how to re-enable it
> again "when computer is on AC power".
> 
> > To me that feels strange, but I could live with that. Can it be documented?
> 
> What would you like to be documented, exactly?

that pointing HOME a non-existing directory is the way to either
inhibit JIT compilation or writing of the compilation results to
disk (depending on the answer above).

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Anyway I thought we were discussing / trying to solve mainly the user
> request.  Is this about test repeatability?  Did we had repeatability
> issues with native code I'm not aware?

There were two things requested, and one of them was related to test
repeatability (which then also requires not writing anything to
eln-cache).

It's not a matter of specific issues with native code, but a general
principle -- you should be able to run the same test twice and have it
trigger all the same code paths.  That's just a general maintenance
principle.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Thu, 6 Oct 2022 13:49:44 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> On Thu, Oct 06, 2022 at 01:13:40PM +0300, Eli Zaretskii wrote:
> > > Date: Thu, 6 Oct 2022 11:02:44 +0200
> > > From: <tomas@tuxteam.de>
> > > 
> > > > > The solution I would propose would be to defer JIT native-compilation
> > > > > until the computer is on AC power, as determined by battery.el.
> > > > 
> > > > We could have such a feature, but how to implement it?  If we use a
> > > > timer for that, the timer itself will drain the battery.
> > > 
> > > Guessing from a previous mail by you, the correct way to achieve this
> > > is to compiler results to a non-existent directory, right?
> > 
> > No, this is a completely different use case.
> 
> Now I'm confused. Does the non-existence of a target directory
> disable compilation or just the writing of the compilation's
> results?

??? What does the existence of the target directory have to do with
the use case of laptop running on batteries?

> > What would you like to be documented, exactly?
> 
> that pointing HOME a non-existing directory is the way to either
> inhibit JIT compilation or writing of the compilation results to
> disk (depending on the answer above).

That is not something we should advertise, I think, because it
shouldn't be needed.  We do that in the test suite, but not in order
to disable native-compilation.

Whether there are valid use cases where users would need to disable
native-compilation is still an open question.  The use case presented
by Po Lu does not require disabling native-compilation, it requires
_delaying_ it until some opportune moment.

Re: Suppressing native compilation (short and long term)

[Michael Welsh Duggan]

In an effort to try and bridge the understanding gap, I'm going to
contribute my current understanding of Debian's requirements, in the
hopes that we might be able to identify any misunderstandings.
Apologies in advance if this just retreads old ground and doesn't add
anything useful to the conversation.

From Debian's point of view, there are a few scenarios under which emacs
is run:

1) When building emacs (used as part of emacs bootstrap) 
2) When installing emacs via dpkg (maybe?  Not certain)
3) When building an emacs package (maybe?  Not certain)
4) When installing an emacs package via dpkg
5) When a user runs emacs

Here are the constraints that I have intuited from the conversation:

a) Case 1 builds .eln files, which will be packaged and installed in
   case 2.

b) Cases 1 and 3 normally happen on a Debian build machine.  These do
   *not* want emacs to write anything to $HOME, though writing to a
   temporary location that will subsequently be thrown away is okay.

c) Case 3 might not require running emacs at all, but I can imagine that
   emacs might be run as part of the build process to auto-generate some
   files.

d) Cases 2, 4, and 5 occur on a Debian user's machine.

e) Cases 2 and 4 are run under root (or similar) and should not write to
   $HOME.

f) Case 2 will install the .eln files packaged in case 1 into a
   world-readable, read-only location.  An Emacs run in case 5 will
   include that location in its native file search path.

g) Case 4 might run emacs to build .elc and .eln files for the package's
   .el files and place them in world-readable, read-only locations.  An
   Emacs run in case 5 will include that location in its native file
   search path.

h) Case 5 should read .eln files from the world-readable, read-only
   locations mentioned above, when possible, but otherwise should do
   native compilation and store the generated .eln files in the standard
   user locations based on $HOME.

Open questions:

i  ) In case 2, are the emacs binaries and the elisp files in the same
     package, or are they split into different packages?  If the latter,
     which package should contain the .eln files?

ii ) Do we want (g) to actually happen?  If so, do we want it to happen
     always, or should this be configurable in the emacs package
     (dpkg-configure)?

iii) Does case 2 also delete files created in (g) and re-generate them
     using the newly installed Emacs?

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Suppressing native compilation (short and long term)

[Liliana Marie Prikler]

Hi Eli,

Am Mittwoch, dem 28.09.2022 um 14:35 +0300 schrieb Eli Zaretskii:
> 
> What do you mean by "disable native compilation" here, and what
> exactly are the use cases for this?  Are we talking about users who
> installed Emacs with some *.eln files, and from time to time load
> *.elc files that have no corresponding *.eln files?  Or are we
> talking about something else?  If the former, why do these users
> install Emacs with native-compilation to begin with?
> 
> More generally, we never expected people who have Emacs with native
> compilation available to want to disable it.  It made no sense to us
> during development of Emacs 28, and frankly, it still doesn't, at
> least to me.  It is so easy to install Emacs without these
> capabilities and never look back that we thought providing a
> build-time option should be enough.  (Well, except for MS-Windows
> users, where more is needed, but that's not your worry.)
> 
> Therefore, if we need to discuss introduction of run-time features
> for disabling native-compilation, we need to have a broader view of
> the relevant use cases and user needs.  This includes at least the
> following aspects that need to be discussed and clarified:
> 
>   . What does "disable native-compilation" mean, exactly?  There are
>     several possible interpretations of that.  Do people want to
>     prevent Emacs from looking for and loading the *.eln files, and
>     use the *.elc files instead?  Or do people only want to prevent
>     Emacs from compiling new *.eln files?  Or do people want that
>     *.eln files be produced only by an explicit user command, not
>     transparently and asynchronously?  Or something else?
> 
>   . When and why would people want any of the above to be possible?
> 
>   . How and why would downstream distros want to support such
>     capabilities?
In GNU Guix, we observe certain packages breaking when trying to
compile them natively.  While some of that is on us for messing up the
package descriptions, one could also encounter genuine bugs in native
compilation that one wants to work around by disabling it.  (For added
trouble, whether or not your package breaks could depend on the CPU due
to being *native* compilation; thus the "no-native-compile" local
variable is insufficient to address this generally.)

In GNU Guix, we default to not compiling packages ahead-of-time,
instead using a minimal emacs that can only do byte compilation for
most packages.  Users can however pretty easily switch to ahead-of-time
compilation by swapping out the emacs package (using what Guix calls
package transformations).  This is also important because apart from
the current Emacs we typically provide an emacs-next package for
upcoming versions, as well as some other variants that the user might
want to use instead.  Again, we assume that users who want to opt in to
native compilation do so via a transformation.

In this context, the default of compiling everything that has hitherto
only been byte-compiled is an ill-advised choice.  Now, there is a
chance that the user meant to do this anyway, but I think they rather
a) use whatever the distro default is without caring either way or b)
actually didn't want this package natively compiled.  Due to some
packages breaking, we had a lot of b) in our mailing lists in the past
few days.

As for loading, I think there could be a case made to restrict loading
to certain directories managed by "trusted entities", but I think
native-comp-eln-load-path already accomplishes this.  If one wanted to
disable loading altogether, I'm pretty sure one could set it to nil and
be done (with perhaps the exception that deferred compilation breaks if
there's no path to store binaries in).

Thus, I think the issue is really only one of inadvertently launching
deferred compilation when all the packages the user *expects* to be
natively compiled are already ensured to be compiled ahead-of-time.  I
admit that this is taking a rather conservative stance, in which the
users don't want to have anything natively compiled that they don't
know of (and as a side result are happy to run that as bytecode or
interpreted code).

I hope this answers some of the questions raised.  In short, we believe
that users might want to control not only from where native shared
libraries will be loaded, but also whether to generate them at runtime,
and that (distro) package managers should enable them to compile all
such shared libraries ahead of time – and better yet, in a reproducible
manner, but this hits other bugs I do not want to discuss in detail at
the moment.

Cheers 

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Liliana Marie Prikler <liliana.prikler@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 12 Oct 2022 22:22:46 +0200
> 
> In GNU Guix, we observe certain packages breaking when trying to
> compile them natively.  While some of that is on us for messing up the
> package descriptions, one could also encounter genuine bugs in native
> compilation that one wants to work around by disabling it.

When you encounter bugs in native compilation, please report them to
us, so we could fix them.  As of now, we are not aware of any such
bugs that were reported and haven't been fixed.  So if you still have
such problem, please report them ASAP.

> (For added
> trouble, whether or not your package breaks could depend on the CPU due
> to being *native* compilation; thus the "no-native-compile" local
> variable is insufficient to address this generally.)

Why isn't it sufficient to use no-native-compile?  It just means that
on some architectures the corresponding file will be loaded as
byte-compiled, and thus will be slightly slower (how much slower
depends on the code, so if you are worried, my recommendation is first
to measure the difference -- you might be surprised).

In any case, if no-native-compile doesn't for some reason solve the
problem, I don't understand how disabling native compilation will: the
latter is a more blunt instrument than the former, e.g., it cannot be
applied on the per-file resolution.

> In GNU Guix, we default to not compiling packages ahead-of-time,
> instead using a minimal emacs that can only do byte compilation for
> most packages.  Users can however pretty easily switch to ahead-of-time
> compilation by swapping out the emacs package (using what Guix calls
> package transformations).  This is also important because apart from
> the current Emacs we typically provide an emacs-next package for
> upcoming versions, as well as some other variants that the user might
> want to use instead.  Again, we assume that users who want to opt in to
> native compilation do so via a transformation.

Sorry, I'm unfamiliar with this terminology.  When you say
"ahead-of-time compilation", do you mean native compilation of all the
Lisp files before they are loaded for the first time?  Or do you mean
something else?

And what is "swapping out the emacs package", and what is "package
transformation"?

> In this context, the default of compiling everything that has hitherto
> only been byte-compiled is an ill-advised choice.  Now, there is a
> chance that the user meant to do this anyway, but I think they rather
> a) use whatever the distro default is without caring either way or b)
> actually didn't want this package natively compiled.  Due to some
> packages breaking, we had a lot of b) in our mailing lists in the past
> few days.

If a package is a single file or a small number of files, those users
can add the no-native-compile cookies in those files.  And again,
disabling native compilation is a method that doesn't allow them
fine-tuning anyway, so I fail to see how it could help them here.  If
the problem is real (and I don't yet see it is), we should perhaps
discuss its details and invent some new method of disabling
compilation with finer granularity.

> As for loading, I think there could be a case made to restrict loading
> to certain directories managed by "trusted entities", but I think
> native-comp-eln-load-path already accomplishes this.

Yes, but see below.

> If one wanted to disable loading altogether, I'm pretty sure one
> could set it to nil and be done (with perhaps the exception that
> deferred compilation breaks if there's no path to store binaries
> in).

I don't think you can set native-comp-eln-load-path to nil.  The last
entry, the one which points to where the preloaded *.eln files live,
must be there, or else Emacs will refuse to start.  And at least one
other directory should be there as well, because if and when some
package advises some primitive, Emacs will need to generate and
compile a trampoline .eln file.  But yes, if users want to prevent
loading from a certain directory, they can remove it from
native-comp-eln-load-path, provided that the two necessary entries are
still left in the list.

> Thus, I think the issue is really only one of inadvertently launching
> deferred compilation when all the packages the user *expects* to be
> natively compiled are already ensured to be compiled ahead-of-time.

This will not happen, so no need to prevent compilation in that case.
If a .eln file already exists and is up-to-date, Emacs will not try to
compile it again as part of loading the package.  If some of the
packages aren't natively compiled, against user's expectations, I
still see no reason for users to want to disallow JIT compilation,
except as a kind of surprise effect due to something they see for the
first time.  How is JIT compilation different from ahead-of-time
compilation?  There's nothing really dangerous or harmful in JIT
compilation, so all it takes is some getting used to it.  Educating
your users to get used to it will go a long way towards eliminating
the fears, because the dangers aren't real, they are largely
imaginary.

> I hope this answers some of the questions raised.  In short, we believe
> that users might want to control not only from where native shared
> libraries will be loaded, but also whether to generate them at runtime,
> and that (distro) package managers should enable them to compile all
> such shared libraries ahead of time – and better yet, in a reproducible
> manner, but this hits other bugs I do not want to discuss in detail at
> the moment.

Thanks for the explanations.  I still think the reasons for disabling
native compilation are rather weak at best, and the users' requests to
allow that based on either bugs that need to be solved or surprise and
fears that have no real basis.  Moreover, disabling native compilation
is a very blunt instrument that cannot be applied better than the
existing ones, like no-native-compile (and a few others that we didn't
mention; see the defcustom's in comp.el).

Re: Suppressing native compilation (short and long term)

[Liliana Marie Prikler]

Am Donnerstag, dem 13.10.2022 um 08:46 +0300 schrieb Eli Zaretskii:
> > From: Liliana Marie Prikler <liliana.prikler@gmail.com>
> > Cc: emacs-devel@gnu.org
> > Date: Wed, 12 Oct 2022 22:22:46 +0200
> > 
> > In GNU Guix, we observe certain packages breaking when trying to
> > compile them natively.  While some of that is on us for messing up
> > the package descriptions, one could also encounter genuine bugs in
> > native compilation that one wants to work around by disabling it.
> 
> When you encounter bugs in native compilation, please report them to
> us, so we could fix them.  As of now, we are not aware of any such
> bugs that were reported and haven't been fixed.  So if you still have
> such problem, please report them ASAP.
Of course, that's the intention, but this fix will only make it into
the next Emacs release.  Thus, if you're between releases, you still
need a workaround.

A particular candidate known to cause issues with the currently
packaged 28.1 is [1].

> > (For added trouble, whether or not your package breaks could depend
> > on the CPU due to being *native* compilation; thus the "no-native-
> > compile" local variable is insufficient to address this generally.)
> 
> Why isn't it sufficient to use no-native-compile?  It just means that
> on some architectures the corresponding file will be loaded as
> byte-compiled, and thus will be slightly slower (how much slower
> depends on the code, so if you are worried, my recommendation is
> first to measure the difference -- you might be surprised).
Because it'd require a distro-wide fix to address something that e.g.
only happens on some AMD CPUs.  Why AMD?  Because we already had bugs
in other languages, where AMD and Intel disagreed about floating points
and that caused tests to fail (though admittedly that's not relevant.

> In any case, if no-native-compile doesn't for some reason solve the
> problem, I don't understand how disabling native compilation will:
> the latter is a more blunt instrument than the former, e.g., it
> cannot be applied on the per-file resolution.
> 
> > In GNU Guix, we default to not compiling packages ahead-of-time,
> > instead using a minimal emacs that can only do byte compilation for
> > most packages.  Users can however pretty easily switch to ahead-of-
> > time compilation by swapping out the emacs package (using what Guix
> > calls package transformations).  This is also important because
> > apart from the current Emacs we typically provide an emacs-next
> > package for upcoming versions, as well as some other variants that
> > the user might want to use instead.  Again, we assume that users
> > who want to opt in to native compilation do so via a
> > transformation.
> 
> Sorry, I'm unfamiliar with this terminology.  When you say
> "ahead-of-time compilation", do you mean native compilation of all
> the Lisp files before they are loaded for the first time?  Or do you
> mean something else?
Exactly, it's more or less the same as ahead-of-time compilation via
package.el, which Emacs supports out of the box.

> And what is "swapping out the emacs package", and what is "package
> transformation"?
Guix users can decide between an Emacs package that only does byte-
compilation (emacs-minimal, the default) or native compilation (emacs).
They can easily write this by using the command-line switch --with-
input=emacs-minimal=emacs while building their Emacs package (e.g.
emacs-dash for dash.el).

For the record, we could probably also offer transformations, that
annotate files with no-native-compile, but this would probably be a
little harder to access as a user.  What I'm instead hoping for would
be a variable that can be set in early-init.el or similar.

> > In this context, the default of compiling everything that has
> > hitherto only been byte-compiled is an ill-advised choice.  Now,
> > there is a chance that the user meant to do this anyway, but I
> > think they rather a) use whatever the distro default is without
> > caring either way or b) actually didn't want this package natively
> > compiled.  Due to some packages breaking, we had a lot of b) in our
> > mailing lists in the past few days.
> 
> If a package is a single file or a small number of files, those users
> can add the no-native-compile cookies in those files.
This is not trivial in the case where the Elisp code is placed in
system-managed storage and thus requires elevated privileges to modify
(as is the default in most package managers, I assume).  Of course, you
can copy the file to your $HOME, but editing it with a broken Emacs is
rather painful.  We wouldn't want to resort to users using nano ;)

> And again, disabling native compilation is a method that doesn't
> allow them fine-tuning anyway, so I fail to see how it could help
> them here.  If the problem is real (and I don't yet see it is), we
> should perhaps discuss its details and invent some new method of
> disabling compilation with finer granularity.
The granularity here is disabling compilation of anything that isn't
already compiled – this makes it so that people can still use their
Emacs for byte compilation by invoking it with "emacs -Q", they just
won't compile anything that their package manager hasn't compiled for
them.

> > As for loading, I think there could be a case made to restrict
> > loading to certain directories managed by "trusted entities", but I
> > think native-comp-eln-load-path already accomplishes this.
> 
> Yes, but see below.
> 
> > If one wanted to disable loading altogether, I'm pretty sure one
> > could set it to nil and be done (with perhaps the exception that
> > deferred compilation breaks if there's no path to store binaries
> > in).
> 
> I don't think you can set native-comp-eln-load-path to nil.  The last
> entry, the one which points to where the preloaded *.eln files live,
> must be there, or else Emacs will refuse to start.  And at least one
> other directory should be there as well, because if and when some
> package advises some primitive, Emacs will need to generate and
> compile a trampoline .eln file.  But yes, if users want to prevent
> loading from a certain directory, they can remove it from
> native-comp-eln-load-path, provided that the two necessary entries
> are still left in the list.
I already found this annoying while implementing native compilation as
part of our emacs-build-system (i.e. the build system used to compile
Emacs packages), and I find it extra questionable that users on
traditional distros, where they don't usually get to choose their
Emacs, have no means of disabling this loading.  Calling back to the
earlier point on measuring performance, an easy way to measure
performance between bytecode and native code (in a benchmark) would be
to simply disable native code loading in one process.  But here, it
requires two separate builds of Emacs.

> > Thus, I think the issue is really only one of inadvertently
> > launching deferred compilation when all the packages the user
> > *expects* to be natively compiled are already ensured to be
> > compiled ahead-of-time.
> 
> This will not happen, so no need to prevent compilation in that case.
> If a .eln file already exists and is up-to-date, Emacs will not try
> to compile it again as part of loading the package.  If some of the
> packages aren't natively compiled, against user's expectations, I
> still see no reason for users to want to disallow JIT compilation,
> except as a kind of surprise effect due to something they see for the
> first time.  How is JIT compilation different from ahead-of-time
> compilation?  There's nothing really dangerous or harmful in JIT
> compilation, so all it takes is some getting used to it.  Educating
> your users to get used to it will go a long way towards eliminating
> the fears, because the dangers aren't real, they are largely
> imaginary.
I agree for the most part that JIT compilation is harmless.  However,
there are some cases in which it isn't.  For one, on slower hardware,
this can take unreasonable times at startup.  While bytecode
performance on such machines might too be slow (but perhaps tolerable
for the task), ahead-of-time compilation, perhaps with offloading, is
preferable.  For another, it can cause bugs like [2].

> > I hope this answers some of the questions raised.  In short, we
> > believe that users might want to control not only from where native
> > shared libraries will be loaded, but also whether to generate them
> > at runtime and that (distro) package managers should enable them to
> > compile all such shared libraries ahead of time – and better yet,
> > in a reproducible manner, but this hits other bugs I do not want to
> > discuss in detail at the moment.
> 
> Thanks for the explanations.  I still think the reasons for disabling
> native compilation are rather weak at best, and the users' requests
> to allow that based on either bugs that need to be solved or surprise
> and fears that have no real basis.  Moreover, disabling native
> compilation is a very blunt instrument that cannot be applied better
> than the existing ones, like no-native-compile (and a few others that
> we didn't mention; see the defcustom's in comp.el).
Which defcustom?  I fear that for all of its customizability, Emacs is
currently lacking a good way of disabling native compilation outside of
package management.  I also tried setting no-native-compile globally,
but it seems to only have an effect as a file-local variable.

Cheers

[1] https://github.com/DarwinAwardWinner/ido-ubiquitous
[2] https://issues.guix.gnu.org/issue/57878

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> [2] https://issues.guix.gnu.org/issue/57878

Do we have a bug open on our (Emacs) side for this issue?
Could it simply be that the async processes we launch to native-compile
some file(s), themselves decide to launch more processes to native
compile more files (or maybe even the same files)?


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Liliana Marie Prikler <liliana.prikler@gmail.com>
> Cc: rlb@defaultvalue.org, emacs-devel@gnu.org
> Date: Thu, 13 Oct 2022 21:20:13 +0200
> 
> > When you encounter bugs in native compilation, please report them to
> > us, so we could fix them.  As of now, we are not aware of any such
> > bugs that were reported and haven't been fixed.  So if you still have
> > such problem, please report them ASAP.
> Of course, that's the intention, but this fix will only make it into
> the next Emacs release.  Thus, if you're between releases, you still
> need a workaround.

If the fix is urgent, why can't you patch the sources when you prepare
your distribution?

> A particular candidate known to cause issues with the currently
> packaged 28.1 is [1].

Where's the description of the actual problem with natively compiling
that package?  And would you please submit a bug report with the
details, if you know them?

> > Why isn't it sufficient to use no-native-compile?  It just means that
> > on some architectures the corresponding file will be loaded as
> > byte-compiled, and thus will be slightly slower (how much slower
> > depends on the code, so if you are worried, my recommendation is
> > first to measure the difference -- you might be surprised).
> Because it'd require a distro-wide fix to address something that e.g.
> only happens on some AMD CPUs.

I'm asking why doing so is a problem?  Did you measure the effect on
performance and found it to be unacceptable in some cases?

> > > In GNU Guix, we default to not compiling packages ahead-of-time,
> > > instead using a minimal emacs that can only do byte compilation for
> > > most packages.  Users can however pretty easily switch to ahead-of-
> > > time compilation by swapping out the emacs package (using what Guix
> > > calls package transformations).  This is also important because
> > > apart from the current Emacs we typically provide an emacs-next
> > > package for upcoming versions, as well as some other variants that
> > > the user might want to use instead.  Again, we assume that users
> > > who want to opt in to native compilation do so via a
> > > transformation.
> > 
> > Sorry, I'm unfamiliar with this terminology.  When you say
> > "ahead-of-time compilation", do you mean native compilation of all
> > the Lisp files before they are loaded for the first time?  Or do you
> > mean something else?
> Exactly, it's more or less the same as ahead-of-time compilation via
> package.el, which Emacs supports out of the box.
> 
> > And what is "swapping out the emacs package", and what is "package
> > transformation"?
> Guix users can decide between an Emacs package that only does byte-
> compilation (emacs-minimal, the default) or native compilation (emacs).
> They can easily write this by using the command-line switch --with-
> input=emacs-minimal=emacs while building their Emacs package (e.g.
> emacs-dash for dash.el).

OK, so why is this relevant to the issue of disabling?  Those who
choose ahead-of-time compilation will never see async JIT compilation,
and those who selected not to do ahead-of-time will naturally see JIT
compilation, as they've chosen.  What is the problem here?

> > > In this context, the default of compiling everything that has
> > > hitherto only been byte-compiled is an ill-advised choice.  Now,
> > > there is a chance that the user meant to do this anyway, but I
> > > think they rather a) use whatever the distro default is without
> > > caring either way or b) actually didn't want this package natively
> > > compiled.  Due to some packages breaking, we had a lot of b) in our
> > > mailing lists in the past few days.
> > 
> > If a package is a single file or a small number of files, those users
> > can add the no-native-compile cookies in those files.
> This is not trivial in the case where the Elisp code is placed in
> system-managed storage and thus requires elevated privileges to modify
> (as is the default in most package managers, I assume).  Of course, you
> can copy the file to your $HOME, but editing it with a broken Emacs is
> rather painful.

Using broken packages is always painful, and native compilation
doesn't change that.  Packages provided by a distribution and
installed into directories where users cannot easily write should be
well tested by the distributor.  IOW, I don't understand why the
upstream project should be held responsible for packages distributed
by downstream distributions without testing them well enough to let
users use them without pain, and why should the upstream project
introduce options to support such "broken" distros.  It isn't fair to
expect us to solve such problems, because they should be solved
elsewhere.

> > And again, disabling native compilation is a method that doesn't
> > allow them fine-tuning anyway, so I fail to see how it could help
> > them here.  If the problem is real (and I don't yet see it is), we
> > should perhaps discuss its details and invent some new method of
> > disabling compilation with finer granularity.
> The granularity here is disabling compilation of anything that isn't
> already compiled – this makes it so that people can still use their
> Emacs for byte compilation by invoking it with "emacs -Q", they just
> won't compile anything that their package manager hasn't compiled for
> them.

That's quite a blunt weapon.  Why not tell them to stay with Emacs 27,
until the problems are solved, or install Emacs 28 without native
compilation?  They are similarly drastic solutions, but they are
already available and will definitely work.

> > I don't think you can set native-comp-eln-load-path to nil.  The last
> > entry, the one which points to where the preloaded *.eln files live,
> > must be there, or else Emacs will refuse to start.  And at least one
> > other directory should be there as well, because if and when some
> > package advises some primitive, Emacs will need to generate and
> > compile a trampoline .eln file.  But yes, if users want to prevent
> > loading from a certain directory, they can remove it from
> > native-comp-eln-load-path, provided that the two necessary entries
> > are still left in the list.
> I already found this annoying while implementing native compilation as
> part of our emacs-build-system (i.e. the build system used to compile
> Emacs packages), and I find it extra questionable that users on
> traditional distros, where they don't usually get to choose their
> Emacs, have no means of disabling this loading.

You mean, you find the loading of preloaded *.eln files at startup
annoying?  Then you should know that this is the best solution we
found for dumping Emacs with natively-compiled preloaded code.  If you
know of a better solution that doesn't suffer from any fatal issues we
found with the alternatives, please suggest such solutions, and we
will definitely consider them.

And again, if Emacs with natively-compilation is annoying, by all
means offer your users an Emacs built without natively-compilation.
This is supported and will continue to be supported for the observable
future.

> Calling back to the earlier point on measuring performance, an easy
> way to measure performance between bytecode and native code (in a
> benchmark) would be to simply disable native code loading in one
> process.  But here, it requires two separate builds of Emacs.

As I told earlier, disabling loading of native code made no sense to
us while Emacs 28 was in development; it still doesn't.  Either one
wants native-compilation, or one doesn't.  Making Emacs code more
complicated and harder to maintain due to features that make no sense
to us is a non-starter.  I see no problem with having to use a
separate build, since building a release tarball takes a minute or so
on a modern system.  And distros should definitely have a build
without native-compilation on offer, for a variety of valid reasons.

> I agree for the most part that JIT compilation is harmless.  However,
> there are some cases in which it isn't.  For one, on slower hardware,
> this can take unreasonable times at startup.

System that are that slow should not install Emacs with
native-compilation, plain and simple.  My suggestion is to clearly
explain this in your README files, so that users could decide which
build is for them, exactly like they decide whether they want a GTK
build or a Lucid build (or any other of the configurations you offer).

> While bytecode performance on such machines might too be slow (but
> perhaps tolerable for the task), ahead-of-time compilation, perhaps
> with offloading, is preferable.

I recommend against this, because it is impossible to rely on AOT
installations to never compile at run time.  Users cannot rely on
that, and should be advised accordingly.

> For another, it can cause bugs like [2].

That bug by itself (the cause of massive launching of async
subprocesses) was never explored or described in that thread?  It
seems like the discussion switched to looking for ways of disabling
native-compilation right away, without a good understand of what was
happening.  Or did I mis something?  Async compilation by default
never launches more subprocesses than half the execution units of the
CPU, so what is described there should be carefully investigated and
the findings described.

The other problem in that discussion, with warnings during async JIT
compilation is well-known, was reported several times, and the culprit
is always in the 3rd-part packages being compiled, which should be
fixed.  In any case, those are just warnings in almost all cases, so
their only adverse effect is annoyance (that can be suppressed by
clicking the button in the message).

Again, I see no reason to blame the upstream project for these issues.
They should be solved by the offending 3rd-party packages, and the
distro should ideally uncover and fix them before they get to users (I
presume that you build and compile the add-on packages you offer?).

> > Thanks for the explanations.  I still think the reasons for disabling
> > native compilation are rather weak at best, and the users' requests
> > to allow that based on either bugs that need to be solved or surprise
> > and fears that have no real basis.  Moreover, disabling native
> > compilation is a very blunt instrument that cannot be applied better
> > than the existing ones, like no-native-compile (and a few others that
> > we didn't mention; see the defcustom's in comp.el).
> Which defcustom?

Begin with those described in the ELisp manual, in the
"Native-Compilation Variables" node.  And my recommendation is to
review _all_ of the defcustoms in comp.el

> I fear that for all of its customizability, Emacs is
> currently lacking a good way of disabling native compilation outside of
> package management.

Yes, because, as mentioned, this makes no sense.  And disabling
native-compilation completely is currently technically impossible (for
te same reason: Emacs wasn't designed to support that because it
didn't and still doesn't seem needed).

> I also tried setting no-native-compile globally, but it seems to
> only have an effect as a file-local variable.

Yes, as designed.  This variable is the equivalent of no-byte-compile,
and works very similarly.

To summarize: native compilation in a build which supports it is
ubiquitous, and is not designed to be disabled except by
no-native-compile on a file by file level.  If a more general
disabling is needed for some reason, users should simply use a build
without native-compilation.  It's the same as various toolkit builds:
if the toolkit is broken or doesn't fit the user's needs, those users
should install a build with a different toolkit.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,  emacs-devel@gnu.org
> Date: Thu, 13 Oct 2022 16:10:19 -0400
> 
> Could it simply be that the async processes we launch to native-compile
> some file(s), themselves decide to launch more processes to native
> compile more files (or maybe even the same files)?

No, it should not happen, because async JIT compilation processes run
Emacs in batch mode, and async compilation is disabled in batch mode
(for this very reason).

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> No, it should not happen, because async JIT compilation processes run
> Emacs in batch mode, and async compilation is disabled in batch mode
> (for this very reason).

As we've covered many times recently -- yes, but no.

.elc -> .eln generation is off, but trampolines are on, and comp.el
forks an Emacs to generate those, too, I think?  So if the forked Emacs
also needs to generate the same trampoline, you have an infinite
recursion fork bomb.

You'd need a pretty tortured setup to get there, but the build process
here seems really tweaked anyway.

Let's see if I can reproduce the phenomenon...

Ooops!  Just crashed my laptop!  That's fun.  😀

So don't follow this recipe:

# cat /usr/local/share/emacs/29.0.50/site-lisp/site-start.el 
(fset 'yes-or-no-p 'y-or-n-p)

rm -r  ~/.emacs.d/eln-cache/; ./src/emacs -Q

M-: (fset 'yes-or-no-p 'y-or-n-p) RET

Dead laptop (except for the fan that spins up), so handle with caution.

So we need a recursion blocker in the trampoline generation function.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So we need a recursion blocker in the trampoline generation function.

Or -- since trampolines are really fast to generate and we need to do
that synchronously anyway -- we could just avoid forking an Emacs do
create the trampoline?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  liliana.prikler@gmail.com,
>   rlb@defaultvalue.org,  emacs-devel@gnu.org, Andrea Corallo <akrl@sdf.org>
> Date: Thu, 13 Oct 2022 22:57:51 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > No, it should not happen, because async JIT compilation processes run
> > Emacs in batch mode, and async compilation is disabled in batch mode
> > (for this very reason).
> 
> As we've covered many times recently -- yes, but no.
> 
> .elc -> .eln generation is off, but trampolines are on, and comp.el
> forks an Emacs to generate those, too, I think?  So if the forked Emacs
> also needs to generate the same trampoline, you have an infinite
> recursion fork bomb.

Andrea, how can we prevent that?

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> I'm still discussing with Rob his needs.  I hope to eventually
> understand their needs.  For now my only conclusion is that they need
> to tweak native-comp-eln-load-path in some situations, to control
> where the *.eln files are deposited.  Which is easy and supported
> already, AFAIU.

Eli, Lars, Andrea, and others, I think I'm mostly caught back up with
this thread, and hope to summarize a few things here:

  - I made a mistake in emphasizing (via the thread subject) a
    particular "solution" (suppressing native compilation).  In the end,
    from the Debian perspective, for the most immedate issue, we don't
    need that particular solution (and I wasn't actually set on it at
    the time).  I'd mostly started there because it's *a* possible
    solution, and one I'd imagined might be easyish to implement
    (clearly that was naive).  But other solutions would be fine too.

  - For example, redirecting all incidental writes away from HOME would
    be fine too.

  - We'd prefer to still be able to set HOME=/does-not-exist, which I
    assume would mean that we'd need some other way to redirect *all* of
    the eln file writes (including trampolines).

    Our practice of setting HOME to a nonexistent directory during some
    packaging operations allows us to detect unexpected, and erroneous
    attempts to write to HOME during those processes.

  - If we are going to have "some other way" to redirect the eln files,
    then for us an environment variable might be easier, so that we can
    just export it before we start the relevant build/test/etc. and then
    it'll affect all invocations of emacs in that environment.  I
    suspect an environment variable might also make it easier to
    establish the setting "early enough" in the emacs startup process,
    but don't know that.

  - If just setting HOME will do the job, and there's no interest here
    in anything further, then we can probably just do that, or if we
    ended up needing to, we could likely add our own
    DEBIAN_... environment variable, and carry a patch, but of course
    that's less desirable.

  - As an aside, I suspect Emacs may eventually want to have some way to
    restore the ability to tolerate an unwritable filesystem.  I have
    more than once in the past launched emacs from an emergency shell to
    fix a broken host where the filesystem was read-only and /home might
    not be mounted (and if it were on NFS and there's no network yet,
    could not be).

    I'd much rather use emacs there, than /usr/bin/sensible-editor.  Of
    course now I know that I could just set HOME=/tmp and proceed, but
    will others?  Might at least be worth making sure any error messages
    would lead people to that option.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

>   - As an aside, I suspect Emacs may eventually want to have some way to
>     restore the ability to tolerate an unwritable filesystem.

If it doesn't work, it's a bug.  Please `M-x report-emacs-bug` when you
see such problems.


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: larsi@gnus.org, akrl@sdf.org, monnier@iro.umontreal.ca,
>  david@tethera.net, emacs-devel@gnu.org
> Date: Fri, 14 Oct 2022 12:53:48 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'm still discussing with Rob his needs.  I hope to eventually
> > understand their needs.  For now my only conclusion is that they need
> > to tweak native-comp-eln-load-path in some situations, to control
> > where the *.eln files are deposited.  Which is easy and supported
> > already, AFAIU.
> 
> Eli, Lars, Andrea, and others, I think I'm mostly caught back up with
> this thread, and hope to summarize a few things here:

Thank you for your summary.

>   - We'd prefer to still be able to set HOME=/does-not-exist, which I
>     assume would mean that we'd need some other way to redirect *all* of
>     the eln file writes (including trampolines).
> 
>     Our practice of setting HOME to a nonexistent directory during some
>     packaging operations allows us to detect unexpected, and erroneous
>     attempts to write to HOME during those processes.

This should work for preventing the native-compilation from storing
the *.eln files.  If it doesn't work for you, please tell the details,
and we will investigate.

>   - If we are going to have "some other way" to redirect the eln files,
>     then for us an environment variable might be easier, so that we can
>     just export it before we start the relevant build/test/etc. and then
>     it'll affect all invocations of emacs in that environment.  I
>     suspect an environment variable might also make it easier to
>     establish the setting "early enough" in the emacs startup process,
>     but don't know that.

One other way is to change the value of native-comp-eln-load-path.  I
hesitate to add an environment variable for that, because environment
variables are inherited to subprocesses, and so setting a variable for
some Emacs process will affect all of its Emacs subprocesses.  This
was found to be a reason of tricky and hard-to-debug problems when
users set EMACSLOADPATH like that, and I presume the same can easily
happen with the equivalent variable for *.eln files.  So I'd rather
not add such a variable unless we find a very good reason.

>   - As an aside, I suspect Emacs may eventually want to have some way to
>     restore the ability to tolerate an unwritable filesystem.

With respect to writing the *.eln files, Emacs will look along
native-comp-eln-load-path for the first directory that is writable,
and use that.  So you could at least partially handle this case by
making sure native-comp-eln-load-path includes at least one writable
directory.

>     I'd much rather use emacs there, than /usr/bin/sensible-editor.  Of
>     course now I know that I could just set HOME=/tmp and proceed, but
>     will others?

I've now documented this method in the ELisp reference manual, in the
hope that it will make this more discoverable.

Thanks again for your comments and perseverance.

Re: Suppressing native compilation (short and long term)

[Liliana Marie Prikler]

Am Donnerstag, dem 13.10.2022 um 23:22 +0300 schrieb Eli Zaretskii:
> > From: Liliana Marie Prikler <liliana.prikler@gmail.com>
> > Cc: rlb@defaultvalue.org, emacs-devel@gnu.org
> > Date: Thu, 13 Oct 2022 21:20:13 +0200
> > 
> > > When you encounter bugs in native compilation, please report them
> > > to us, so we could fix them.  As of now, we are not aware of any
> > > such bugs that were reported and haven't been fixed.  So if you
> > > still have such problem, please report them ASAP.
> > Of course, that's the intention, but this fix will only make it
> > into the next Emacs release.  Thus, if you're between releases, you
> > still need a workaround.
> 
> If the fix is urgent, why can't you patch the sources when you
> prepare your distribution?
Guix prides itself in being a package manager that can work around many
failures (even as the proper workaround to bugs is discussed in mailing
lists).  The fact, that the solutions to this issue is "compile 28.1
without native-comp" or "use Emacs 27" does not reflect that
particularly well.

> > A particular candidate known to cause issues with the currently
> > packaged 28.1 is [1].
> 
> Where's the description of the actual problem with natively compiling
> that package?  And would you please submit a bug report with the
> details, if you know them?
I am not personally affected, so I can't.  I could direct people to the
Emacs mailing lists, but it seems people in other threads have already
started debugging.  Do you still wish me to do so? 

> > > Why isn't it sufficient to use no-native-compile?  It just means
> > > that on some architectures the corresponding file will be loaded
> > > as byte-compiled, and thus will be slightly slower (how much
> > > slower depends on the code, so if you are worried, my
> > > recommendation is first to measure the difference -- you might be
> > > surprised).
> > Because it'd require a distro-wide fix to address something that
> > e.g. only happens on some AMD CPUs.
> 
> I'm asking why doing so is a problem?  Did you measure the effect on
> performance and found it to be unacceptable in some cases?
Isn't performance one of the main reasons to use native compilation? 
Note that I am talking in hypotheticals here when mentioning the AMD
thing, i.e. we could very well imagine a performance-critical Emacs
package having a native-compilation bug (I imagine those to be
particularly likely for those trailing unreleased Emacs versions,
though thankfully I don't think we've encountered one so far.)

> > > > In GNU Guix, we default to not compiling packages ahead-of-
> > > > time, instead using a minimal emacs that can only do byte
> > > > compilation for most packages.  Users can however pretty easily
> > > > switch to ahead-of-time compilation by swapping out the emacs
> > > > package (using what Guix calls package transformations).  This
> > > > is also important because apart from the current Emacs we
> > > > typically provide an emacs-next package for upcoming versions,
> > > > as well as some other variants that the user might want to use
> > > > instead.  Again, we assume that users who want to opt in to
> > > > native compilation do so via a transformation.
> > > 
> > > Sorry, I'm unfamiliar with this terminology.  When you say
> > > "ahead-of-time compilation", do you mean native compilation of
> > > all the Lisp files before they are loaded for the first time?  Or
> > > do you mean something else?
> > Exactly, it's more or less the same as ahead-of-time compilation
> > via package.el, which Emacs supports out of the box.
> > 
> > > And what is "swapping out the emacs package", and what is
> > > "package transformation"?
> > Guix users can decide between an Emacs package that only does byte-
> > compilation (emacs-minimal, the default) or native compilation
> > (emacs).
> > They can easily write this by using the command-line switch --with-
> > input=emacs-minimal=emacs while building their Emacs package (e.g.
> > emacs-dash for dash.el).
> 
> OK, so why is this relevant to the issue of disabling?  Those who
> choose ahead-of-time compilation will never see async JIT
> compilation, and those who selected not to do ahead-of-time will
> naturally see JIT compilation, as they've chosen.  What is the
> problem here?
The problem is that I can't meaningfully choose the "I don't want JIT
for stuff I haven't AOT'd" option, especially not combined with "but I
do want to load what I have AOT'd".

> > > > In this context, the default of compiling everything that has
> > > > hitherto only been byte-compiled is an ill-advised choice. 
> > > > Now, there is a chance that the user meant to do this anyway,
> > > > but I think they rather a) use whatever the distro default is
> > > > without caring either way or b) actually didn't want this
> > > > package natively compiled.  Due to some packages breaking, we
> > > > had a lot of b) in our mailing lists in the past few days.
> > > 
> > > If a package is a single file or a small number of files, those
> > > users can add the no-native-compile cookies in those files.
> > This is not trivial in the case where the Elisp code is placed in
> > system-managed storage and thus requires elevated privileges to
> > modify (as is the default in most package managers, I assume).  Of
> > course, you can copy the file to your $HOME, but editing it with a
> > broken Emacs is rather painful.
> 
> Using broken packages is always painful, and native compilation
> doesn't change that.
Using broken packages normally doesn't result in the OOM killer firing
off.

> Packages provided by a distribution and installed into directories
> where users cannot easily write should be well tested by the
> distributor.  
I think you're underestimating the number of breakages that can happen
in a rolling release model.  Not every distro is as stable as Debian,
but the joke's still on you because despite Debian's hard requirements,
they still ended up encountering this bug.

> IOW, I don't understand why the upstream project should be held
> responsible for packages distributed by downstream distributions
> without testing them well enough to let users use them without pain,
> and why should the upstream project introduce options to support such
> "broken" distros.  It isn't fair to expect us to solve such problems,
> because they should be solved elsewhere.
There are limits to testing.  When I added native comp to Guix, I gave
folks in the mailing lists a heads up to try their own configuration
and report bugs.  But people don't always read the mailing lists and
therefore aren't aware of upcoming changes that may break their system,
and I personally can't test every Emacs configuration in existence.

> > > And again, disabling native compilation is a method that doesn't
> > > allow them fine-tuning anyway, so I fail to see how it could help
> > > them here.  If the problem is real (and I don't yet see it is),
> > > we should perhaps discuss its details and invent some new method
> > > of disabling compilation with finer granularity.
> > The granularity here is disabling compilation of anything that
> > isn't already compiled – this makes it so that people can still use
> > their Emacs for byte compilation by invoking it with "emacs -Q",
> > they just won't compile anything that their package manager hasn't
> > compiled for them.
> 
> That's quite a blunt weapon.  Why not tell them to stay with Emacs
> 27, until the problems are solved, or install Emacs 28 without native
> compilation?
That's what they're currently resorting to.  Guix already supports this
use case.

> They are similarly drastic solutions, but they are already available
> and will definitely work.
Yet they are suboptimal, particularly on traditional distributions,
that don't support this use-case well.

> > > I don't think you can set native-comp-eln-load-path to nil.  The
> > > last entry, the one which points to where the preloaded *.eln
> > > files live, must be there, or else Emacs will refuse to start. 
> > > And at least one other directory should be there as well, because
> > > if and when some package advises some primitive, Emacs will need
> > > to generate and compile a trampoline .eln file.  But yes, if
> > > users want to prevent loading from a certain directory, they can
> > > remove it from native-comp-eln-load-path, provided that the two
> > > necessary entries are still left in the list.
> > I already found this annoying while implementing native compilation
> > as part of our emacs-build-system (i.e. the build system used to
> > compile Emacs packages), and I find it extra questionable that
> > users on traditional distros, where they don't usually get to
> > choose their Emacs, have no means of disabling this loading.
> 
> You mean, you find the loading of preloaded *.eln files at startup
> annoying?  Then you should know that this is the best solution we
> found for dumping Emacs with natively-compiled preloaded code.
No, I find it annoying that Emacs supposes it has a writable eln-cache
always.  This is not the case in typical package manager scenarios and
it also isn't the case when users choose to make (parts of) their $HOME
read-only, which is a supported configuration in Nix and Guix.  I can't
think of a good reason why one would want to assume this invariant.

> If you know of a better solution that doesn't suffer from any fatal
> issues we found with the alternatives, please suggest such solutions,
> and we will definitely consider them.
I haven't read the discussions around the alternatives, but couldn't
you just generate one trampoline per function which you use as soon as
it's advised?

Also, how come advice isn't breaking byte-compilation in exactly the
same manner?

> And again, if Emacs with natively-compilation is annoying, by all
> means offer your users an Emacs built without natively-compilation.
> This is supported and will continue to be supported for the
> observable future.
> 
> > Calling back to the earlier point on measuring performance, an easy
> > way to measure performance between bytecode and native code (in a
> > benchmark) would be to simply disable native code loading in one
> > process.  But here, it requires two separate builds of Emacs.
> 
> As I told earlier, disabling loading of native code made no sense to
> us while Emacs 28 was in development; it still doesn't.  Either one
> wants native-compilation, or one doesn't.  Making Emacs code more
> complicated and harder to maintain due to features that make no sense
> to us is a non-starter.  I see no problem with having to use a
> separate build, since building a release tarball takes a minute or so
> on a modern system.  And distros should definitely have a build
> without native-compilation on offer, for a variety of valid reasons.
I don't think that asking distros to package every Emacs variant twice
is a great idea.  At Guix, we prefer to offer the most complete version
of a package, so we ship with native compilation enabled.  If a user
wants a UI, but no native compilation (i.e. neither emacs nor emacs-
minimal), they have to roll their own package description.
> 

> > While bytecode performance on such machines might too be slow (but
> > perhaps tolerable for the task), ahead-of-time compilation, perhaps
> > with offloading, is preferable.
> 
> I recommend against this, because it is impossible to rely on AOT
> installations to never compile at run time.  Users cannot rely on
> that, and should be advised accordingly.
But why can't they?

> > For another, it can cause bugs like [2].
> 
> That bug by itself (the cause of massive launching of async
> subprocesses) was never explored or described in that thread?  It
> seems like the discussion switched to looking for ways of disabling
> native-compilation right away, without a good understand of what was
> happening.  Or did I mis something?  Async compilation by default
> never launches more subprocesses than half the execution units of the
> CPU, so what is described there should be carefully investigated and
> the findings described.
It'd be weird if someone found a counterexample to the above statement.

> The other problem in that discussion, with warnings during async JIT
> compilation is well-known, was reported several times, and the
> culprit is always in the 3rd-part packages being compiled, which
> should be fixed.  In any case, those are just warnings in almost all
> cases, so their only adverse effect is annoyance (that can be
> suppressed by clicking the button in the message).
I read no such problem in that discussion.  Do we read the same thread?

> Again, I see no reason to blame the upstream project for these
> issues.  They should be solved by the offending 3rd-party packages,
> and the distro should ideally uncover and fix them before they get to
> users (I presume that you build and compile the add-on packages you
> offer?).
I'd like to tap at the "rolling release distro is not Debian" sign, but
again, stable distros like Debian are experiencing issues with native
compilation.

> > > Thanks for the explanations.  I still think the reasons for
> > > disabling native compilation are rather weak at best, and the
> > > users' requests to allow that based on either bugs that need to
> > > be solved or surprise and fears that have no real basis. 
> > > Moreover, disabling native compilation is a very blunt instrument
> > > that cannot be applied better than the existing ones, like no-
> > > native-compile (and a few others that we didn't mention; see the
> > > defcustom's in comp.el).
> > Which defcustom?
> 
> Begin with those described in the ELisp manual, in the
> "Native-Compilation Variables" node.  And my recommendation is to
> review _all_ of the defcustoms in comp.el
The only one I found is setting native-comp-speed to -1.  Is that the
solution?  It doesn't appear to be.

> > I fear that for all of its customizability, Emacs is
> > currently lacking a good way of disabling native compilation
> > outside of package management.
> 
> Yes, because, as mentioned, this makes no sense.
I think you'll find it does.

> And disabling native-compilation completely is currently technically
> impossible (for the same reason: Emacs wasn't designed to support
> that because it didn't and still doesn't seem needed).
> 
> > I also tried setting no-native-compile globally, but it seems to
> > only have an effect as a file-local variable.
> 
> Yes, as designed.  This variable is the equivalent of no-byte-
> compile, and works very similarly.
> 
> To summarize: native compilation in a build which supports it is
> ubiquitous, and is not designed to be disabled except by
> no-native-compile on a file by file level.  If a more general
> disabling is needed for some reason, users should simply use a build
> without native-compilation.  It's the same as various toolkit builds:
> if the toolkit is broken or doesn't fit the user's needs, those users
> should install a build with a different toolkit.
Pardon my French, but that thinking in and of itself is broken.  Native
compilation is not a choice in which you pick the one that most suits
your fancy from a range of options – it could be that if you allowed
the user to choose between libgccjit, clang and some other compilers
that shall not be named, not that I recommend you implement this.  As
such, I think users who do want to use native compilation should get
some more say in when, where, and what to compile.

Cheers

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Rob Browning <rlb@defaultvalue.org>

>>   - We'd prefer to still be able to set HOME=/does-not-exist, which I
>>     assume would mean that we'd need some other way to redirect *all* of
>>     the eln file writes (including trampolines).
>> 
>>     Our practice of setting HOME to a nonexistent directory during some
>>     packaging operations allows us to detect unexpected, and erroneous
>>     attempts to write to HOME during those processes.
>
> This should work for preventing the native-compilation from storing
> the *.eln files.  If it doesn't work for you, please tell the details,
> and we will investigate.

Hmm, which "this" did you mean?  If you meant HOME=/does-not-exist, I
thought people had already said that wouldn't work because the attempt
to build trampolines would still cause a crash.  But I may well have
misunderstood.

> One other way is to change the value of native-comp-eln-load-path.  I
> hesitate to add an environment variable for that, because environment
> variables are inherited to subprocesses, and so setting a variable for
> some Emacs process will affect all of its Emacs subprocesses.  This
> was found to be a reason of tricky and hard-to-debug problems when
> users set EMACSLOADPATH like that, and I presume the same can easily
> happen with the equivalent variable for *.eln files.  So I'd rather
> not add such a variable unless we find a very good reason.

For what it's worth, the reason we'd been at least a bit inclined that
direction is because if you have hundreds of emacs add-on packages
(which I believe we do in Debian), and they all have build and or test
suites that vary in any way they like, it could be much more difficult
to track down all of the emacs invocations in each project's source to
add some argument than it would be to just export an environment
variable before running the suite.

I suppose we could just shadow emacs in the path with a wrapper that
includes the argument(s), assuming few of the projects have their own
interfering options, and that they always use the same (or a handful of
predictable) relative invocation of emacs, e.g. "emacs" not
"/usr/bin/emacs" or...

> With respect to writing the *.eln files, Emacs will look along
> native-comp-eln-load-path for the first directory that is writable,
> and use that.  So you could at least partially handle this case by
> making sure native-comp-eln-load-path includes at least one writable
> directory.

Right, though we also have to make sure we can do that sufficiently
early in *every* case.

> Thanks again for your comments and perseverance.

Certainly, appreciate the help.
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> So we need a recursion blocker in the trampoline generation function.
>
> Or -- since trampolines are really fast to generate and we need to do
> that synchronously anyway -- we could just avoid forking an Emacs do
> create the trampoline?

We don't do that because libgccjit (well GCC really) leaks memory.

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  liliana.prikler@gmail.com,
>>   rlb@defaultvalue.org,  emacs-devel@gnu.org, Andrea Corallo <akrl@sdf.org>
>> Date: Thu, 13 Oct 2022 22:57:51 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > No, it should not happen, because async JIT compilation processes run
>> > Emacs in batch mode, and async compilation is disabled in batch mode
>> > (for this very reason).
>> 
>> As we've covered many times recently -- yes, but no.
>> 
>> .elc -> .eln generation is off, but trampolines are on, and comp.el
>> forks an Emacs to generate those, too, I think?  So if the forked Emacs
>> also needs to generate the same trampoline, you have an infinite
>> recursion fork bomb.
>
> Andrea, how can we prevent that?

Dumb question: can't we just run the spawned compilation processes with
--no-site-file?

Other option is to break circularity with an ad-hoc global variable set
in the spawned process.  I've a cooked patch for that but no energy left
to test it tonight.  If that's the preferred way I can test and push it
tomorrow.

Best Regards

  Andrea

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> I suppose we could just shadow emacs in the path with a wrapper that
> includes the argument(s), assuming few of the projects have their own
> interfering options, and that they always use the same (or a handful of
> predictable) relative invocation of emacs, e.g. "emacs" not
> "/usr/bin/emacs" or...

Or put a small chunk of ELisp in Debian's site-start file which sets the
eln load path according to some env-var :-)


        Stefan

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Andrea Corallo [2022-10-14 22:37:51] wrote:
> Lars Ingebrigtsen <larsi@gnus.org> writes:
>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>> So we need a recursion blocker in the trampoline generation function.
>> Or -- since trampolines are really fast to generate and we need to do
>> that synchronously anyway -- we could just avoid forking an Emacs do
>> create the trampoline?
> We don't do that because libgccjit (well GCC really) leaks memory.

But we can do that *in the forked Emacs* (this one will exit after
compilation, so leakage is not an issue), so we do fork once but we avoid
forking infinitely.


        Stefan

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> Dumb question: can't we just run the spawned compilation processes with
> --no-site-file?

For trampolines, I guess that should work since they shouldn't depend on
local customizations.  Of course, a tempting alternative is to resort to
"binary hacking", i.e. compile *one* template-trampoline and then
generate all every other trampoline by copying that template and
patching the right "stuff" into it.  That would save us from running the
compiler to generate the trampolines (i.e. it would let us behave
correctly on Windows even when GCC/libgccjit is not found at run time),
but it would force us to write architecture-dependent code to patch the
binary template.


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: luangruo@yahoo.com, larsi@gnus.org, akrl@sdf.org,
>  monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org
> Date: Fri, 14 Oct 2022 16:17:56 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Rob Browning <rlb@defaultvalue.org>
> 
> >>   - We'd prefer to still be able to set HOME=/does-not-exist, which I
> >>     assume would mean that we'd need some other way to redirect *all* of
> >>     the eln file writes (including trampolines).
> >> 
> >>     Our practice of setting HOME to a nonexistent directory during some
> >>     packaging operations allows us to detect unexpected, and erroneous
> >>     attempts to write to HOME during those processes.
> >
> > This should work for preventing the native-compilation from storing
> > the *.eln files.  If it doesn't work for you, please tell the details,
> > and we will investigate.
> 
> Hmm, which "this" did you mean?  If you meant HOME=/does-not-exist, I
> thought people had already said that wouldn't work because the attempt
> to build trampolines would still cause a crash.  But I may well have
> misunderstood.

I meant HOME=/does-not-exist, yes.  If that doesn't work (but please
test), we should fix that, I think.  Please report your findings if
that doesn't work.  We use that in our own test suite.

> > One other way is to change the value of native-comp-eln-load-path.  I
> > hesitate to add an environment variable for that, because environment
> > variables are inherited to subprocesses, and so setting a variable for
> > some Emacs process will affect all of its Emacs subprocesses.  This
> > was found to be a reason of tricky and hard-to-debug problems when
> > users set EMACSLOADPATH like that, and I presume the same can easily
> > happen with the equivalent variable for *.eln files.  So I'd rather
> > not add such a variable unless we find a very good reason.
> 
> For what it's worth, the reason we'd been at least a bit inclined that
> direction is because if you have hundreds of emacs add-on packages
> (which I believe we do in Debian), and they all have build and or test
> suites that vary in any way they like, it could be much more difficult
> to track down all of the emacs invocations in each project's source to
> add some argument than it would be to just export an environment
> variable before running the suite.

I understand, but don't you have some kind of centralized script or
group of scripts that runs the testing?  In that case, you could make
the Emacs command, or its template, be part of those few scripts, and
then the change will be less painful.

Please understand my POV: adding an environment variable affects all
Emacs users, not just distros that have testing suites.  There's much
more at stake than your particular testing needs, and many Emacs users
know much less about the potential effects of such environment
variables than you and me.

> I suppose we could just shadow emacs in the path with a wrapper that
> includes the argument(s), assuming few of the projects have their own
> interfering options, and that they always use the same (or a handful of
> predictable) relative invocation of emacs, e.g. "emacs" not
> "/usr/bin/emacs" or...

Yes, something like that.  At least that's what we do in the Emacs's
own test suite (which, of course, is much smaller than yours).

> > With respect to writing the *.eln files, Emacs will look along
> > native-comp-eln-load-path for the first directory that is writable,
> > and use that.  So you could at least partially handle this case by
> > making sure native-comp-eln-load-path includes at least one writable
> > directory.
> 
> Right, though we also have to make sure we can do that sufficiently
> early in *every* case.

Yes, ideally from the same/similar wrapper script that actually
invokes Emacs, via the --eval option.

Re: Suppressing native compilation (short and long term)

[Liliana Marie Prikler]

Am Freitag, dem 14.10.2022 um 23:14 -0400 schrieb Stefan Monnier:
> 
> Of course, a tempting alternative is to resort to "binary hacking",
> i.e. compile *one* template-trampoline and then generate all every
> other trampoline by copying that template and patching the right
> "stuff" into it.  That would save us from running the
> compiler to generate the trampolines (i.e. it would let us behave
> correctly on Windows even when GCC/libgccjit is not found at run
> time), but it would force us to write architecture-dependent code to
> patch the binary template.
Could table jumps work as an architecture-independent solution or is
there something else I'm not considering atm?

Cheers

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, monnier@iro.umontreal.ca,
>         liliana.prikler@gmail.com, rlb@defaultvalue.org, emacs-devel@gnu.org
> Date: Fri, 14 Oct 2022 23:20:43 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> .elc -> .eln generation is off, but trampolines are on, and comp.el
> >> forks an Emacs to generate those, too, I think?  So if the forked Emacs
> >> also needs to generate the same trampoline, you have an infinite
> >> recursion fork bomb.
> >
> > Andrea, how can we prevent that?
> 
> Dumb question: can't we just run the spawned compilation processes with
> --no-site-file?

I don't think I understand how --no-site-file could help.  Are you
assuming that such advises could only come from sit-init files?  If
so, I'm sure they could come from other sources, or what am I missing?

> Other option is to break circularity with an ad-hoc global variable set
> in the spawned process.  I've a cooked patch for that but no energy left
> to test it tonight.  If that's the preferred way I can test and push it
> tomorrow.

Maybe it's preferable, but I'm not sure the idea of the change I get
from your short description is what you really meant.  Can you tell
more about that?  What ad-hoc variables did you have in mind, and how
would we use them in this case to prevent infinite forking of
sub-processes?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Lars Ingebrigtsen <larsi@gnus.org>,
>   liliana.prikler@gmail.com,  rlb@defaultvalue.org,  emacs-devel@gnu.org
> Date: Fri, 14 Oct 2022 23:14:41 -0400
> 
> > Dumb question: can't we just run the spawned compilation processes with
> > --no-site-file?
> 
> For trampolines, I guess that should work since they shouldn't depend on
> local customizations.

But it will potentially cause other issues, because the environment in
which the compiling Emacs sub-process run is different from that of
the Emacs session which triggered the compilation?  So we could have
warnings and errors in the compilation process due to stuff being
undefined etc.?  Isn't this why we do read the init files in the async
sub-process which performs the compilation?

> Of course, a tempting alternative is to resort to
> "binary hacking", i.e. compile *one* template-trampoline and then
> generate all every other trampoline by copying that template and
> patching the right "stuff" into it.  That would save us from running the
> compiler to generate the trampolines (i.e. it would let us behave
> correctly on Windows even when GCC/libgccjit is not found at run time),
> but it would force us to write architecture-dependent code to patch the
> binary template.

I don't think we should depend on architecture-dependent code.  It
would mean we don't support new platforms until someone writes such
code for them, for starters.  It's against our development tendencies
for the past 10 years at least.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Liliana Marie Prikler <liliana.prikler@gmail.com>
> Cc: rlb@defaultvalue.org, emacs-devel@gnu.org
> Date: Fri, 14 Oct 2022 21:46:09 +0200
> 
> > > > When you encounter bugs in native compilation, please report them
> > > > to us, so we could fix them.  As of now, we are not aware of any
> > > > such bugs that were reported and haven't been fixed.  So if you
> > > > still have such problem, please report them ASAP.
> > > Of course, that's the intention, but this fix will only make it
> > > into the next Emacs release.  Thus, if you're between releases, you
> > > still need a workaround.
> > 
> > If the fix is urgent, why can't you patch the sources when you
> > prepare your distribution?
> Guix prides itself in being a package manager that can work around many
> failures (even as the proper workaround to bugs is discussed in mailing
> lists).  The fact, that the solutions to this issue is "compile 28.1
> without native-comp" or "use Emacs 27" does not reflect that
> particularly well.

I think this answers a different question.  I asked why you cannot
patch the Emacs you distribute when you consider a fix to be important
enough to not wait until the next Emacs release.  My point is that
reporting bugs in a timely fashion will help us fix them early on, and
you will then have a possibility of backporting the fixes to a
released Emacs and distributing an updated package with the fix, if
you think that's important enough.

> > > A particular candidate known to cause issues with the currently
> > > packaged 28.1 is [1].
> > 
> > Where's the description of the actual problem with natively compiling
> > that package?  And would you please submit a bug report with the
> > details, if you know them?
> I am not personally affected, so I can't.  I could direct people to the
> Emacs mailing lists, but it seems people in other threads have already
> started debugging.  Do you still wish me to do so? 

Which threads are you alluding to here?  Your [1] is just a reference
to ido-completing-read-plus package, and I don't see the description
of the problems with native-compilation on that site.  So yes, I'd
like to hear a description of the problem in that case.

> > > > Why isn't it sufficient to use no-native-compile?  It just means
> > > > that on some architectures the corresponding file will be loaded
> > > > as byte-compiled, and thus will be slightly slower (how much
> > > > slower depends on the code, so if you are worried, my
> > > > recommendation is first to measure the difference -- you might be
> > > > surprised).
> > > Because it'd require a distro-wide fix to address something that
> > > e.g. only happens on some AMD CPUs.
> > 
> > I'm asking why doing so is a problem?  Did you measure the effect on
> > performance and found it to be unacceptable in some cases?
> Isn't performance one of the main reasons to use native compilation? 

On average, yes.  But it depends on what the original Lisp code does.
We've found that in some cases the performance gains are minimal, and
in at least one very special case we found that native-compilation
produces a slightly slower code.  Which is why I asked the question
above: it is quite possible that the (hopefully, few) packages where
you need to avoid native-compilation for now don't gain performance
from using native-compilation enough for justify any more elaborate
measures.  And this is a temporary measure anyway, because those
problems will eventually be fixed, whether in the packages themselves
or in Emacs core.

> Note that I am talking in hypotheticals here when mentioning the AMD
> thing, i.e. we could very well imagine a performance-critical Emacs
> package having a native-compilation bug (I imagine those to be
> particularly likely for those trailing unreleased Emacs versions,
> though thankfully I don't think we've encountered one so far.)

Let's not be bothered by hypothetical cases until they actually
emerge.  When there are specific situations where this happens and
performance gains from native-compilation are critical, we can always
look for specific solutions for those cases, something that is
impossible without concrete cases.

> > OK, so why is this relevant to the issue of disabling?  Those who
> > choose ahead-of-time compilation will never see async JIT
> > compilation, and those who selected not to do ahead-of-time will
> > naturally see JIT compilation, as they've chosen.  What is the
> > problem here?
> The problem is that I can't meaningfully choose the "I don't want JIT
> for stuff I haven't AOT'd" option, especially not combined with "but I
> do want to load what I have AOT'd".

As I already explained, this mode of operation doesn't make sense to
me, and is currently not supported for that reason.  I fail to see why
people would want native-compilation for some parts of Emacs, but not
for others.  I haven't yet seen a valid use case where that would make
sense as the desired, clean, and non-kludgey solution.

Only one valid use case was brought p to this date, where it would be
desirable to delay JIT native-compilation temporarily: when the user
runs a laptop on batteries.  We will probably provide a solution for
that, which will automatically re-enable JIT compilation when AC power
is connected.  This would be a clean, non-kludgey solution for that
case.

None of the problems you describe are of that nature.  They all sound
like someone wants to arbitrarily disable native-compilation in some
cases, but not others, where reasonable solutions already exist.

And if you still disagree, then let's agree to disagree, because we
are just repeating the same arguments over and over again.

> > > > If a package is a single file or a small number of files, those
> > > > users can add the no-native-compile cookies in those files.
> > > This is not trivial in the case where the Elisp code is placed in
> > > system-managed storage and thus requires elevated privileges to
> > > modify (as is the default in most package managers, I assume).  Of
> > > course, you can copy the file to your $HOME, but editing it with a
> > > broken Emacs is rather painful.
> > 
> > Using broken packages is always painful, and native compilation
> > doesn't change that.
> Using broken packages normally doesn't result in the OOM killer firing
> off.

It could, rarely.

And which problem of native-compilation caused the OOM killer?  Where
is that problem described in enough detail for us to investigate it?
Was it reported to the Emacs bug-tracker, and if so, what is the bug
number, please?

IOW, we'd definitely want to avoid such catastrophic failures, but we
need the details to investigate and fix them.  I can tell you that I'm
using Emacs 28 with JIT native-compilation enabled for the best part
of this year, and have yet to see any problems even approaching the
one mentioned above.  So such problems are quite exceptional, and need
to be reported with every possible detail for us to be able to fix
them quickly.  They are definitely not a reason to disable
native-compilation.  We generally try to provide at least a workaround
for critical problems, once we have enough detail to understand what's
going on, so reporting a problem quickly will in many cases yield a
quick solution that doesn't hamper unrelated parts of the user's usage
patterns.

> > Packages provided by a distribution and installed into directories
> > where users cannot easily write should be well tested by the
> > distributor.  
> I think you're underestimating the number of breakages that can happen
> in a rolling release model.  Not every distro is as stable as Debian,
> but the joke's still on you because despite Debian's hard requirements,
> they still ended up encountering this bug.

Sure, that's understandable.  But each new problem that is found and
reported should cause the corresponding package to be updated with a
fix.  I don't see why such problems are deemed as reasons to disable
native-compilation for the entire Emacs session, or for requirements
that they be "fixed" in core.  Bugs should be fixed where their root
cause is.

> > You mean, you find the loading of preloaded *.eln files at startup
> > annoying?  Then you should know that this is the best solution we
> > found for dumping Emacs with natively-compiled preloaded code.
> No, I find it annoying that Emacs supposes it has a writable eln-cache
> always.

The user's home directory should always be writable.  This is required
by many Emacs features regardless of native-compilation.  For example,
saving customizations writes to a subdirectory of the user's home
directory, as does desktop.el or save-place etc.

If this is a problem during installation of packages, which run at
root level, the installation procedure can tweak
native-comp-eln-load-path to make sure there's a writable directory
there, or point HOME to a non-existent directory.

> This is not the case in typical package manager scenarios and
> it also isn't the case when users choose to make (parts of) their $HOME
> read-only, which is a supported configuration in Nix and Guix.

Users make ~/.emacs.d/ read-only?  Then how do they use all the
features, some of which mentioned above, that write to that directory?

> I can't think of a good reason why one would want to assume this
> invariant.

If this use case is supported by pointing the relevant variables, like
save-place-file, eshell-directory-name, desktop-dirname, etc., to
non-default places, then they can do the same with
native-comp-eln-load-path.  If this is not what you mean, please
describe how Nix and Guix support this use case where parts of $HOME
are read-only, and let's see how native-compilation should support it.

> > If you know of a better solution that doesn't suffer from any fatal
> > issues we found with the alternatives, please suggest such solutions,
> > and we will definitely consider them.
> I haven't read the discussions around the alternatives, but couldn't
> you just generate one trampoline per function which you use as soon as
> it's advised?

And then re-generate it again each time the advised function is called
again?

> Also, how come advice isn't breaking byte-compilation in exactly the
> same manner?

Andrea, can you please answer that?  I have only a very general
understanding of why trampolines are needed for native-compilation.

> > As I told earlier, disabling loading of native code made no sense to
> > us while Emacs 28 was in development; it still doesn't.  Either one
> > wants native-compilation, or one doesn't.  Making Emacs code more
> > complicated and harder to maintain due to features that make no sense
> > to us is a non-starter.  I see no problem with having to use a
> > separate build, since building a release tarball takes a minute or so
> > on a modern system.  And distros should definitely have a build
> > without native-compilation on offer, for a variety of valid reasons.
> I don't think that asking distros to package every Emacs variant twice
> is a great idea.  At Guix, we prefer to offer the most complete version
> of a package, so we ship with native compilation enabled.

I think this is a mistake.  Native-compilation is not for everyone.
It requires GCC and Binutils to be installed, and who says every Emacs
user wants that?

More generally, when we add optional features, we don't consider
whether having them all in the same build will make sense.  For
example, ImageMagick support has some advantages and some (quite
serious, IMO) disadvantages, so always providing it because it's "the
most complete version" doesn't necessarily make sense for the users.

> > > While bytecode performance on such machines might too be slow (but
> > > perhaps tolerable for the task), ahead-of-time compilation, perhaps
> > > with offloading, is preferable.
> > 
> > I recommend against this, because it is impossible to rely on AOT
> > installations to never compile at run time.  Users cannot rely on
> > that, and should be advised accordingly.
> But why can't they?

Trampolines is one reason.  I'm sure there are others.  Again, we
didn't design native-compilation support in Emacs to be switchable on
and off at run time, so it's small wonder that it doesn't work
reliably.  It would be a surprise if it did.

> > > For another, it can cause bugs like [2].
> > 
> > That bug by itself (the cause of massive launching of async
> > subprocesses) was never explored or described in that thread?  It
> > seems like the discussion switched to looking for ways of disabling
> > native-compilation right away, without a good understand of what was
> > happening.  Or did I mis something?  Async compilation by default
> > never launches more subprocesses than half the execution units of the
> > CPU, so what is described there should be carefully investigated and
> > the findings described.
> It'd be weird if someone found a counterexample to the above statement.

I don't understand this comment, sorry.

> > The other problem in that discussion, with warnings during async JIT
> > compilation is well-known, was reported several times, and the
> > culprit is always in the 3rd-part packages being compiled, which
> > should be fixed.  In any case, those are just warnings in almost all
> > cases, so their only adverse effect is annoyance (that can be
> > suppressed by clicking the button in the message).
> I read no such problem in that discussion.  Do we read the same thread?

I hope so.  I referred to this:

  https://issues.guix.gnu.org/issue/57878#13

> > Again, I see no reason to blame the upstream project for these
> > issues.  They should be solved by the offending 3rd-party packages,
> > and the distro should ideally uncover and fix them before they get to
> > users (I presume that you build and compile the add-on packages you
> > offer?).
> I'd like to tap at the "rolling release distro is not Debian" sign, but
> again, stable distros like Debian are experiencing issues with native
> compilation.

Once again: no one expects all the issues to be found in advance, but
when a new issue in a package is found, I do expect the distro to fix
it and publish an updated package.  I do not expect the distro to come
back to the upstream project and ask for knobs to deal with bugs in
3rd-party packages uncovered by latest Emacs features.

> > > Which defcustom?
> > 
> > Begin with those described in the ELisp manual, in the
> > "Native-Compilation Variables" node.  And my recommendation is to
> > review _all_ of the defcustoms in comp.el
> The only one I found is setting native-comp-speed to -1.  Is that the
> solution?  It doesn't appear to be.

It _is_ a solution, for one class of problems with native-compilation.
Another solution is to tweak native-comp-eln-load-path.
Yet another one is to temporarily point HOME to another, perhaps
non-existent, directory.

> > To summarize: native compilation in a build which supports it is
> > ubiquitous, and is not designed to be disabled except by
> > no-native-compile on a file by file level.  If a more general
> > disabling is needed for some reason, users should simply use a build
> > without native-compilation.  It's the same as various toolkit builds:
> > if the toolkit is broken or doesn't fit the user's needs, those users
> > should install a build with a different toolkit.
> Pardon my French, but that thinking in and of itself is broken.  Native
> compilation is not a choice in which you pick the one that most suits
> your fancy from a range of options – it could be that if you allowed
> the user to choose between libgccjit, clang and some other compilers
> that shall not be named, not that I recommend you implement this.  As
> such, I think users who do want to use native compilation should get
> some more say in when, where, and what to compile.

We hear the users and their complaints, and fix stuff that we think
belongs to Emacs.  This was and will always be the case.  In this
case, I'm not convinced that the issues you describe justify a new
knob in Emacs.  You've described several issues which either already
have solutions (which you reject), or should be solved elsewhere, not
in Emacs.

And if that still doesn't convince you, let's agree to disagree.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>>> So we need a recursion blocker in the trampoline generation function.
>>
>> Or -- since trampolines are really fast to generate and we need to do
>> that synchronously anyway -- we could just avoid forking an Emacs do
>> create the trampoline?
>
> We don't do that because libgccjit (well GCC really) leaks memory.

Ah, darn.  Then I guess we do have to fork, and then come up with some
kind of way to tell that forked Emacs to not recurse any further?

(Do you know whether libgccjit might stop leaking memory at some point?)

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Dumb question: can't we just run the spawned compilation processes with
> --no-site-file?

Having the `fset' in the site-file was just an example -- there must be
fifty ways to put an `fset' into the Emacs startup sequence.

> Other option is to break circularity with an ad-hoc global variable set
> in the spawned process.  I've a cooked patch for that but no energy left
> to test it tonight.  If that's the preferred way I can test and push it
> tomorrow.

Would this be an environment variable or an elisp variable?

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

>   - We'd prefer to still be able to set HOME=/does-not-exist, which I
>     assume would mean that we'd need some other way to redirect *all* of
>     the eln file writes (including trampolines).

I think this should work, but there may be regressions, of course.
Let's see...  I tried it now, and I got the warning below, but things
otherwise seem to work fine:

Warning (initialization): Unable to create `user-emacs-directory' (~/.emacs.d/).
Any data that would normally be written there may be lost!
If you never want to see this message again,
customize the variable `user-emacs-directory-warning'.

>   - If we are going to have "some other way" to redirect the eln files,
>     then for us an environment variable might be easier, so that we can
>     just export it before we start the relevant build/test/etc. and then
>     it'll affect all invocations of emacs in that environment.  I
>     suspect an environment variable might also make it easier to
>     establish the setting "early enough" in the emacs startup process,
>     but don't know that.

Emacs 29 now has the `inhibit-automatic-native-compilation' variable and
and `EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment variables
to inhibit writing to ~/.emacs.d/eln-cache...

>   - As an aside, I suspect Emacs may eventually want to have some way to
>     restore the ability to tolerate an unwritable filesystem.  I have
>     more than once in the past launched emacs from an emergency shell to
>     fix a broken host where the filesystem was read-only and /home might
>     not be mounted (and if it were on NFS and there's no network yet,
>     could not be).

Emacs should work with an unwritable file system, but there's probably
code that bugs out in that situation -- but those things should be
fixed.

If you have a test case that demonstrates the problem, please open a bug
report for that, so that we can get fixin'.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

> The problem is that I can't meaningfully choose the "I don't want JIT
> for stuff I haven't AOT'd" option, especially not combined with "but I
> do want to load what I have AOT'd".

I haven't followed the Guix part of this thread in detail, but I thought
I'd just note that Emacs 29 now has the
`inhibit-automatic-native-compilation' variable and and
`EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment variables, so
this is implemented.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

>> > Dumb question: can't we just run the spawned compilation processes with
>> > --no-site-file?
>> For trampolines, I guess that should work since they shouldn't depend on
>> local customizations.
> But it will potentially cause other issues, because the environment in
> which the compiling Emacs sub-process run is different from that of
> the Emacs session which triggered the compilation?  So we could have
> warnings and errors in the compilation process due to stuff being
> undefined etc.?  Isn't this why we do read the init files in the async
> sub-process which performs the compilation?

Yes, when compiling actual ELisp files.  I'm talking about compiling the
trampolines: that's different because it's just compiling a fixed chunk
of code *we* write.

>> Of course, a tempting alternative is to resort to
>> "binary hacking", i.e. compile *one* template-trampoline and then
>> generate all every other trampoline by copying that template and
>> patching the right "stuff" into it.  That would save us from running the
>> compiler to generate the trampolines (i.e. it would let us behave
>> correctly on Windows even when GCC/libgccjit is not found at run time),
>> but it would force us to write architecture-dependent code to patch the
>> binary template.
>
> I don't think we should depend on architecture-dependent code.  It
> would mean we don't support new platforms until someone writes such
> code for them, for starters.  It's against our development tendencies
> for the past 10 years at least.

I know, I know.  From a technical point of view, it's The Right Thing to
do, I think, tho: it will eliminate all the other issues surrounding
trampolines (e.g. disk space to store trampolines, dependence on
libgccjit for the creation of those trampolines, fork bombs) and will be
much more efficient both in CPU time and memory use (we don't care too
much, because trampolines are hopefully not too frequents, so we live
with them being quite inefficient in the time to create them (and set
them up, via dyn-linking) and in their memory use).

But yes, the architecture-dependence is a serious problem, which is why
we've so far avoided that solution.


        Stefan

Re: Suppressing native compilation (short and long term)

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]

On Sat, Oct 15, 2022, 10:02 AM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

>
> > I don't think we should depend on architecture-dependent code.  It
> > would mean we don't support new platforms until someone writes such
> > code for them, for starters.  It's against our development tendencies
> > for the past 10 years at least.
>
> I know, I know.  From a technical point of view, it's The Right Thing to
> do, I think, tho: it will eliminate all the other issues surrounding
> trampolines (e.g. disk space to store trampolines, dependence on
> libgccjit for the creation of those trampolines, fork bombs) and will be
> much more efficient both in CPU time and memory use (we don't care too
> much, because trampolines are hopefully not too frequents, so we live
> with them being quite inefficient in the time to create them (and set
> them up, via dyn-linking) and in their memory use).
>
> But yes, the architecture-dependence is a serious problem, which is why
> we've so far avoided that solution.


Didn't you, a couple of years ago, suggest generating generic trampolines
(per function signature), compile those with gccjit, then just use them as
needed rather than compiling separate trampolines for every subr?
Presumably the system would still occasionally need to compile a trampoline
for a new signature, but the generic trampoline would not be named so it
couldn't cause an infinite recursion by being advised before being compiled.

Lynn

[-- Attachment #2: Type: text/html, Size: 1976 bytes --]

Re: Suppressing native compilation (short and long term)

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1592 bytes --]

On Fri, Oct 14, 2022, 7:22 PM Andrea Corallo <akrl@sdf.org> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> From: Lars Ingebrigtsen <larsi@gnus.org>
> >> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
> liliana.prikler@gmail.com,
> >>   rlb@defaultvalue.org,  emacs-devel@gnu.org, Andrea Corallo <
> akrl@sdf.org>
> >> Date: Thu, 13 Oct 2022 22:57:51 +0200
> >>
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >>
> >> > No, it should not happen, because async JIT compilation processes run
> >> > Emacs in batch mode, and async compilation is disabled in batch mode
> >> > (for this very reason).
> >>
> >> As we've covered many times recently -- yes, but no.
> >>
> >> .elc -> .eln generation is off, but trampolines are on, and comp.el
> >> forks an Emacs to generate those, too, I think?  So if the forked Emacs
> >> also needs to generate the same trampoline, you have an infinite
> >> recursion fork bomb.
> >
> > Andrea, how can we prevent that?
>
> Dumb question: can't we just run the spawned compilation processes with
> --no-site-file?
>
> Other option is to break circularity with an ad-hoc global variable set
> in the spawned process.  I've a cooked patch for that but no energy left
> to test it tonight.  If that's the preferred way I can test and push it
> tomorrow.


The cleanest way would be to prepare a dump file specifically for compiling
that is known to work, then use that for the async compiler job with any
required settings passed in the command line.  This would make the system
robust when the system dump file is not limited to just the standard build.

Lynn

[-- Attachment #2: Type: text/html, Size: 2808 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Sat, 15 Oct 2022 10:10:23 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, Andrea Corallo <akrl@sdf.org>, Lars Ingebrigtsen <larsi@gnus.org>, 
> 	liliana.prikler@gmail.com, rlb@defaultvalue.org, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
> Didn't you, a couple of years ago, suggest generating generic trampolines (per function signature), compile
> those with gccjit, then just use them as needed rather than compiling separate trampolines for every subr?
> Presumably the system would still occasionally need to compile a trampoline for a new signature, but the
> generic trampoline would not be named so it couldn't cause an infinite recursion by being advised before
> being compiled.

If we want to compile all the trampolines AOT, that is already
possible in Emacs 29, one just needs to actually do it when building
Emacs.

Re: Suppressing native compilation (short and long term)

[Liliana Marie Prikler]

Am Samstag, dem 15.10.2022 um 11:33 +0200 schrieb Lars Ingebrigtsen:
> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> 
> > The problem is that I can't meaningfully choose the "I don't want
> > JIT
> > for stuff I haven't AOT'd" option, especially not combined with
> > "but I
> > do want to load what I have AOT'd".
> 
> I haven't followed the Guix part of this thread in detail, but I
> thought I'd just note that Emacs 29 now has the
> `inhibit-automatic-native-compilation' variable and and
> `EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment variables,
> so this is implemented.
Thanks.  I've bumped emacs-next in Guix and tried it; seems to work as
advertised.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>>> So we need a recursion blocker in the trampoline generation function.
>>>
>>> Or -- since trampolines are really fast to generate and we need to do
>>> that synchronously anyway -- we could just avoid forking an Emacs do
>>> create the trampoline?
>>
>> We don't do that because libgccjit (well GCC really) leaks memory.
>
> Ah, darn.  Then I guess we do have to fork, and then come up with some
> kind of way to tell that forked Emacs to not recurse any further?

Yes

> (Do you know whether libgccjit might stop leaking memory at some point?)

Yes, there must be a bug on bugzilla ATM I've no trace of.  A colleague
of mine did some investigation in the past and discovered IIRC that the
GCC driver was the main source of the leakage. I'm Ccing him now
hopefully he recalls/knows more than me.

Also I think there was some fixing in this area but even in case is
fixed completly now we'll have to cope with older version for a while.

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Andrea Corallo [2022-10-14 22:37:51] wrote:
>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>>> So we need a recursion blocker in the trampoline generation function.
>>> Or -- since trampolines are really fast to generate and we need to do
>>> that synchronously anyway -- we could just avoid forking an Emacs do
>>> create the trampoline?
>> We don't do that because libgccjit (well GCC really) leaks memory.
>
> But we can do that *in the forked Emacs* (this one will exit after
> compilation, so leakage is not an issue), so we do fork once but we avoid
> forking infinitely.

Hi Stefan,

by *in the forked Emacs* you mean Emacs just after fork is called or
after calling fork+exec to start a fresh session?

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Dumb question: can't we just run the spawned compilation processes with
>> --no-site-file?
>
> For trampolines, I guess that should work since they shouldn't depend on
> local customizations.  Of course, a tempting alternative is to resort to
> "binary hacking", i.e. compile *one* template-trampoline and then
> generate all every other trampoline by copying that template and
> patching the right "stuff" into it.  That would save us from running the
> compiler to generate the trampolines (i.e. it would let us behave
> correctly on Windows even when GCC/libgccjit is not found at run time),
> but it would force us to write architecture-dependent code to patch the
> binary template.

I think writing and maintaining arch dependent code to fix and
manipulate binaries is really a road we don't want to go down!

 (a terrified) Andrea :)

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Lars Ingebrigtsen <larsi@gnus.org>, monnier@iro.umontreal.ca,
>>         liliana.prikler@gmail.com, rlb@defaultvalue.org, emacs-devel@gnu.org
>> Date: Fri, 14 Oct 2022 23:20:43 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> .elc -> .eln generation is off, but trampolines are on, and comp.el
>> >> forks an Emacs to generate those, too, I think?  So if the forked Emacs
>> >> also needs to generate the same trampoline, you have an infinite
>> >> recursion fork bomb.
>> >
>> > Andrea, how can we prevent that?
>> 
>> Dumb question: can't we just run the spawned compilation processes with
>> --no-site-file?
>
> I don't think I understand how --no-site-file could help.  Are you
> assuming that such advises could only come from sit-init files?

Yes this was the assumption.

> If
> so, I'm sure they could come from other sources, or what am I missing?

I think you are not missing much... as not said.

>> Other option is to break circularity with an ad-hoc global variable set
>> in the spawned process.  I've a cooked patch for that but no energy left
>> to test it tonight.  If that's the preferred way I can test and push it
>> tomorrow.
>
> Maybe it's preferable, but I'm not sure the idea of the change I get
> from your short description is what you really meant.  Can you tell
> more about that?  What ad-hoc variables did you have in mind, and how
> would we use them in this case to prevent infinite forking of
> sub-processes?

We define a new variable say `comp-spawned-compilation' and we set it to
t in all spawed compilation sub processes.  Then when we need a
trampoline:

- If it's available we load it.
- If it's not and `comp-spawned-compilation' is nil we compile it and
  load it, otherwise if `comp-spawned-compilation' is t we just do
  nothing (so we break circularity).

WDYT?

Bests

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Dumb question: can't we just run the spawned compilation processes with
>> --no-site-file?
>
> Having the `fset' in the site-file was just an example -- there must be
> fifty ways to put an `fset' into the Emacs startup sequence.

I see

>> Other option is to break circularity with an ad-hoc global variable set
>> in the spawned process.  I've a cooked patch for that but no energy left
>> to test it tonight.  If that's the preferred way I can test and push it
>> tomorrow.
>
> Would this be an environment variable or an elisp variable?

Elisp, I've just posted a reply to Eli with some derscription.

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Liliana Marie Prikler <liliana.prikler@gmail.com>
>> Cc: rlb@defaultvalue.org, emacs-devel@gnu.org
>> Date: Fri, 14 Oct 2022 21:46:09 +0200
>> 
>> > > > When you encounter bugs in native compilation, please report them
>> > > > to us, so we could fix them.  As of now, we are not aware of any
>> > > > such bugs that were reported and haven't been fixed.  So if you
>> > > > still have such problem, please report them ASAP.
>> > > Of course, that's the intention, but this fix will only make it
>> > > into the next Emacs release.  Thus, if you're between releases, you
>> > > still need a workaround.
>> > 
>> > If the fix is urgent, why can't you patch the sources when you
>> > prepare your distribution?
>> Guix prides itself in being a package manager that can work around many
>> failures (even as the proper workaround to bugs is discussed in mailing
>> lists).  The fact, that the solutions to this issue is "compile 28.1
>> without native-comp" or "use Emacs 27" does not reflect that
>> particularly well.
>
> I think this answers a different question.  I asked why you cannot
> patch the Emacs you distribute when you consider a fix to be important
> enough to not wait until the next Emacs release.  My point is that
> reporting bugs in a timely fashion will help us fix them early on, and
> you will then have a possibility of backporting the fixes to a
> released Emacs and distributing an updated package with the fix, if
> you think that's important enough.
>
>> > > A particular candidate known to cause issues with the currently
>> > > packaged 28.1 is [1].
>> > 
>> > Where's the description of the actual problem with natively compiling
>> > that package?  And would you please submit a bug report with the
>> > details, if you know them?
>> I am not personally affected, so I can't.  I could direct people to the
>> Emacs mailing lists, but it seems people in other threads have already
>> started debugging.  Do you still wish me to do so? 
>
> Which threads are you alluding to here?  Your [1] is just a reference
> to ido-completing-read-plus package, and I don't see the description
> of the problems with native-compilation on that site.  So yes, I'd
> like to hear a description of the problem in that case.
>
>> > > > Why isn't it sufficient to use no-native-compile?  It just means
>> > > > that on some architectures the corresponding file will be loaded
>> > > > as byte-compiled, and thus will be slightly slower (how much
>> > > > slower depends on the code, so if you are worried, my
>> > > > recommendation is first to measure the difference -- you might be
>> > > > surprised).
>> > > Because it'd require a distro-wide fix to address something that
>> > > e.g. only happens on some AMD CPUs.
>> > 
>> > I'm asking why doing so is a problem?  Did you measure the effect on
>> > performance and found it to be unacceptable in some cases?
>> Isn't performance one of the main reasons to use native compilation? 
>
> On average, yes.  But it depends on what the original Lisp code does.
> We've found that in some cases the performance gains are minimal, and
> in at least one very special case we found that native-compilation
> produces a slightly slower code.  Which is why I asked the question
> above: it is quite possible that the (hopefully, few) packages where
> you need to avoid native-compilation for now don't gain performance
> from using native-compilation enough for justify any more elaborate
> measures.  And this is a temporary measure anyway, because those
> problems will eventually be fixed, whether in the packages themselves
> or in Emacs core.
>
>> Note that I am talking in hypotheticals here when mentioning the AMD
>> thing, i.e. we could very well imagine a performance-critical Emacs
>> package having a native-compilation bug (I imagine those to be
>> particularly likely for those trailing unreleased Emacs versions,
>> though thankfully I don't think we've encountered one so far.)
>
> Let's not be bothered by hypothetical cases until they actually
> emerge.  When there are specific situations where this happens and
> performance gains from native-compilation are critical, we can always
> look for specific solutions for those cases, something that is
> impossible without concrete cases.
>
>> > OK, so why is this relevant to the issue of disabling?  Those who
>> > choose ahead-of-time compilation will never see async JIT
>> > compilation, and those who selected not to do ahead-of-time will
>> > naturally see JIT compilation, as they've chosen.  What is the
>> > problem here?
>> The problem is that I can't meaningfully choose the "I don't want JIT
>> for stuff I haven't AOT'd" option, especially not combined with "but I
>> do want to load what I have AOT'd".
>
> As I already explained, this mode of operation doesn't make sense to
> me, and is currently not supported for that reason.  I fail to see why
> people would want native-compilation for some parts of Emacs, but not
> for others.  I haven't yet seen a valid use case where that would make
> sense as the desired, clean, and non-kludgey solution.
>
> Only one valid use case was brought p to this date, where it would be
> desirable to delay JIT native-compilation temporarily: when the user
> runs a laptop on batteries.  We will probably provide a solution for
> that, which will automatically re-enable JIT compilation when AC power
> is connected.  This would be a clean, non-kludgey solution for that
> case.
>
> None of the problems you describe are of that nature.  They all sound
> like someone wants to arbitrarily disable native-compilation in some
> cases, but not others, where reasonable solutions already exist.
>
> And if you still disagree, then let's agree to disagree, because we
> are just repeating the same arguments over and over again.
>
>> > > > If a package is a single file or a small number of files, those
>> > > > users can add the no-native-compile cookies in those files.
>> > > This is not trivial in the case where the Elisp code is placed in
>> > > system-managed storage and thus requires elevated privileges to
>> > > modify (as is the default in most package managers, I assume).  Of
>> > > course, you can copy the file to your $HOME, but editing it with a
>> > > broken Emacs is rather painful.
>> > 
>> > Using broken packages is always painful, and native compilation
>> > doesn't change that.
>> Using broken packages normally doesn't result in the OOM killer firing
>> off.
>
> It could, rarely.
>
> And which problem of native-compilation caused the OOM killer?  Where
> is that problem described in enough detail for us to investigate it?
> Was it reported to the Emacs bug-tracker, and if so, what is the bug
> number, please?
>
> IOW, we'd definitely want to avoid such catastrophic failures, but we
> need the details to investigate and fix them.  I can tell you that I'm
> using Emacs 28 with JIT native-compilation enabled for the best part
> of this year, and have yet to see any problems even approaching the
> one mentioned above.  So such problems are quite exceptional, and need
> to be reported with every possible detail for us to be able to fix
> them quickly.  They are definitely not a reason to disable
> native-compilation.  We generally try to provide at least a workaround
> for critical problems, once we have enough detail to understand what's
> going on, so reporting a problem quickly will in many cases yield a
> quick solution that doesn't hamper unrelated parts of the user's usage
> patterns.
>
>> > Packages provided by a distribution and installed into directories
>> > where users cannot easily write should be well tested by the
>> > distributor.  
>> I think you're underestimating the number of breakages that can happen
>> in a rolling release model.  Not every distro is as stable as Debian,
>> but the joke's still on you because despite Debian's hard requirements,
>> they still ended up encountering this bug.
>
> Sure, that's understandable.  But each new problem that is found and
> reported should cause the corresponding package to be updated with a
> fix.  I don't see why such problems are deemed as reasons to disable
> native-compilation for the entire Emacs session, or for requirements
> that they be "fixed" in core.  Bugs should be fixed where their root
> cause is.
>
>> > You mean, you find the loading of preloaded *.eln files at startup
>> > annoying?  Then you should know that this is the best solution we
>> > found for dumping Emacs with natively-compiled preloaded code.
>> No, I find it annoying that Emacs supposes it has a writable eln-cache
>> always.
>
> The user's home directory should always be writable.  This is required
> by many Emacs features regardless of native-compilation.  For example,
> saving customizations writes to a subdirectory of the user's home
> directory, as does desktop.el or save-place etc.
>
> If this is a problem during installation of packages, which run at
> root level, the installation procedure can tweak
> native-comp-eln-load-path to make sure there's a writable directory
> there, or point HOME to a non-existent directory.
>
>> This is not the case in typical package manager scenarios and
>> it also isn't the case when users choose to make (parts of) their $HOME
>> read-only, which is a supported configuration in Nix and Guix.
>
> Users make ~/.emacs.d/ read-only?  Then how do they use all the
> features, some of which mentioned above, that write to that directory?
>
>> I can't think of a good reason why one would want to assume this
>> invariant.
>
> If this use case is supported by pointing the relevant variables, like
> save-place-file, eshell-directory-name, desktop-dirname, etc., to
> non-default places, then they can do the same with
> native-comp-eln-load-path.  If this is not what you mean, please
> describe how Nix and Guix support this use case where parts of $HOME
> are read-only, and let's see how native-compilation should support it.
>
>> > If you know of a better solution that doesn't suffer from any fatal
>> > issues we found with the alternatives, please suggest such solutions,
>> > and we will definitely consider them.
>> I haven't read the discussions around the alternatives, but couldn't
>> you just generate one trampoline per function which you use as soon as
>> it's advised?
>
> And then re-generate it again each time the advised function is called
> again?
>
>> Also, how come advice isn't breaking byte-compilation in exactly the
>> same manner?
>
> Andrea, can you please answer that?  I have only a very general
> understanding of why trampolines are needed for native-compilation.

The quick answer is that advising or redefining primitive functions
breaks byte-code as well.  Liliana can try redefining `+' and let us
know if it works as expected :/

To be more precise byte-code breaks for redefining each primitive that
got a dedicated byte-opcode.

Now, the situation is even worst!  Redefining primitives fails to take
effect for *all* calls to primitives happening from C code.

Actually the Elisp manual suggests not to redefine primitives *but*...
users kept on doing it since forever and just got used to this bizarre
scenario where it does "often" works.

When I introduced native compilation being native code the newest into
the game it was deemed not acceptable to increase the level of existing
incompatibility.  Unfortunately Emacs is defined also by what's on the
field and not only by what's written the manual, not every behavior or
corner case is defined by a spec we comply to.  Not supporting this use
case would have translated in tons of bug reports from users probably
angrier than Liliana.

So I added this trampoline mechanism.  And yeah, funny enough, as a
consequence at present native Lisp (and interpreted Lisp) are the only
execution modes supporting for real primitive redefinition!

End of the tale :)

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
>
>> The problem is that I can't meaningfully choose the "I don't want JIT
>> for stuff I haven't AOT'd" option, especially not combined with "but I
>> do want to load what I have AOT'd".
>
> I haven't followed the Guix part of this thread in detail, but I thought
> I'd just note that Emacs 29 now has the
> `inhibit-automatic-native-compilation' variable and and
> `EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment variables, so
> this is implemented.

I thought this change was just into master "for discussion", are we
already suggesting users to integrate it in their infrastructures?  BTW
AFAIU this usecase was already supported in 28 using
`native-comp-deferred-compilation'.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> by *in the forked Emacs* you mean Emacs just after fork is called or
> after calling fork+exec to start a fresh session?

I wrote "fork" sloppily to mean "spawn a new process".
AFAIK we can't rely on `fork` literally because it's not
really/efficiently/reliably available under Windows.


        Stefan

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Andrea Corallo [2022-10-15 15:10:06] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> Dumb question: can't we just run the spawned compilation processes with
>>> --no-site-file?
>>
>> For trampolines, I guess that should work since they shouldn't depend on
>> local customizations.  Of course, a tempting alternative is to resort to
>> "binary hacking", i.e. compile *one* template-trampoline and then
>> generate all every other trampoline by copying that template and
>> patching the right "stuff" into it.  That would save us from running the
>> compiler to generate the trampolines (i.e. it would let us behave
>> correctly on Windows even when GCC/libgccjit is not found at run time),
>> but it would force us to write architecture-dependent code to patch the
>> binary template.
>
> I think writing and maintaining arch dependent code to fix and
> manipulate binaries is really a road we don't want to go down!
>
>  (a terrified) Andrea :)

I tend to agree.  I just wish we could rely on some other tool to do
that for us.


        Stefan

Re: Suppressing native compilation (short and long term)

[Liliana Marie Prikler]

Am Samstag, dem 15.10.2022 um 15:56 +0000 schrieb Andrea Corallo:
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> > 
> > > The problem is that I can't meaningfully choose the "I don't want
> > > JIT
> > > for stuff I haven't AOT'd" option, especially not combined with
> > > "but I
> > > do want to load what I have AOT'd".
> > 
> > I haven't followed the Guix part of this thread in detail, but I
> > thought
> > I'd just note that Emacs 29 now has the
> > `inhibit-automatic-native-compilation' variable and and
> > `EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment variables,
> > so
> > this is implemented.
> 
> I thought this change was just into master "for discussion", are we
> already suggesting users to integrate it in their infrastructures? 
> BTW AFAIU this usecase was already supported in 28 using
> `native-comp-deferred-compilation'.
The deferred-compilation switch still ends up writing to $HOME, which
the new inhibit one doesn't.  Should the former be deprecated in favor
of the latter or should it simply take up its meaning?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: larsi@gnus.org, monnier@iro.umontreal.ca, liliana.prikler@gmail.com,
>         rlb@defaultvalue.org, emacs-devel@gnu.org
> Date: Sat, 15 Oct 2022 15:19:40 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Other option is to break circularity with an ad-hoc global variable set
> >> in the spawned process.  I've a cooked patch for that but no energy left
> >> to test it tonight.  If that's the preferred way I can test and push it
> >> tomorrow.
> >
> > Maybe it's preferable, but I'm not sure the idea of the change I get
> > from your short description is what you really meant.  Can you tell
> > more about that?  What ad-hoc variables did you have in mind, and how
> > would we use them in this case to prevent infinite forking of
> > sub-processes?
> 
> We define a new variable say `comp-spawned-compilation' and we set it to
> t in all spawed compilation sub processes.  Then when we need a
> trampoline:
> 
> - If it's available we load it.
> - If it's not and `comp-spawned-compilation' is nil we compile it and
>   load it, otherwise if `comp-spawned-compilation' is t we just do
>   nothing (so we break circularity).
> 
> WDYT?

Yes, this is more-or-less what I thought you had in mind.  I guess
setting of this new variable for the forked subprocesses will be via
the command line?

I think it's a fine solution, but either comp-spawned-compilation
should be renamed to comp-no-spawn, or its value in the forked Emacs
processes should be nil...

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I suppose we could just shadow emacs in the path with a wrapper that
>> includes the argument(s), assuming few of the projects have their own
>> interfering options, and that they always use the same (or a handful of
>> predictable) relative invocation of emacs, e.g. "emacs" not
>> "/usr/bin/emacs" or...
>
> Or put a small chunk of ELisp in Debian's site-start file which sets the
> eln load path according to some env-var :-)

I wondered if that would be early enough.  I'll have to double-check, but I
think people may have found that some of the attempts to change it via
the command line may not have been.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Sat 15 Oct 2022 at 11:32AM +02, Lars Ingebrigtsen wrote:

> Emacs 29 now has the `inhibit-automatic-native-compilation' variable and
> and `EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment variables
> to inhibit writing to ~/.emacs.d/eln-cache...

Is it likely to be okay for us to backport those commits to our 28.2?

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> I meant HOME=/does-not-exist, yes.  If that doesn't work (but please
> test), we should fix that, I think.  Please report your findings if
> that doesn't work.  We use that in our own test suite.

I've been told that it doesn't because (the person thought) of
trampolines.  i.e. Emacs can be reliably crashed that way.  Though if
that's changed since 28.[12], they wouldn't have seen it yet, and there
may be patches we should cherry-pick.

> I understand, but don't you have some kind of centralized script or
> group of scripts that runs the testing?  In that case, you could make
> the Emacs command, or its template, be part of those few scripts, and
> then the change will be less painful.

For us, this is about hundreds of upstream trees we don't directly
control.  How and where does magit invoke emacs during its build or
during its tests?  Or buttercup, or org, or bbdb, or emacspeak, or ivy,
or lsp, or...

How do we make sure we affect *all* of the invocations of emacs in all
of those?  Note here is just a subset things that Debian has packaged
that might be affected (i.e. only the ones whose maintainers have
decided to keep the Debian tree on salsa in the emacsen-team project):

  https://salsa.debian.org/emacsen-team

> Yes, something like that.  At least that's what we do in the Emacs's
> own test suite (which, of course, is much smaller than yours).

...as long as "most" invocations are either "emacs" or respect say
EMACS, this could work.  I have no idea if that's true, but we do
already know there are likely to be any number of places we'll have to
fix.  i.e. someone has told me that this approach will result in "a lot
of bugs" (in the debian packages).  They already know of places where
"some of them try to choose which emacs to use, rather than just
'emacs'".

We can handle that, but if we have to patch a lot of the upstream trees,
that could be a good bit of work.  That's certainly possible, but
hopefully that helps give more context about why we were interested in
some solution, among the possibilities, that's less potentially
invasive.

For what it's worth, we're also under a (not too close yet) deadline on
the Debian side.  We'd really like to have at least emacs 28 in the next
stable release, and the first stage of the freeze is currently scheduled
for I think January (though Emacs would I think be affected by a
slightly later stage).

For that to be possible, whatever we end up choosing here will need to
be something we can finish accommodating across both the emacs package
and all the add-ons by then.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Thu 13 Oct 2022 at 04:10PM -04, Stefan Monnier wrote:

>> [2] https://issues.guix.gnu.org/issue/57878
>
> Do we have a bug open on our (Emacs) side for this issue?
> Could it simply be that the async processes we launch to native-compile
> some file(s), themselves decide to launch more processes to native
> compile more files (or maybe even the same files)?

Just for reference, Debian's filings for this issue:

https://bugs.debian.org/1017817
https://bugs.debian.org/1017845

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: Liliana Marie Prikler <liliana.prikler@gmail.com>,  Eli Zaretskii
>  <eliz@gnu.org>,  rlb@defaultvalue.org,  emacs-devel@gnu.org
> Date: Sat, 15 Oct 2022 10:06:03 -0700
> 
> >> [2] https://issues.guix.gnu.org/issue/57878
> >
> > Do we have a bug open on our (Emacs) side for this issue?
> > Could it simply be that the async processes we launch to native-compile
> > some file(s), themselves decide to launch more processes to native
> > compile more files (or maybe even the same files)?
> 
> Just for reference, Debian's filings for this issue:

Thanks.

> https://bugs.debian.org/1017817

This seems to be about byte-compilation?

> https://bugs.debian.org/1017845

This is about trampolines indeed, and I believe we will have a
solution VSN.

P.S. I wish such problems didn't have to wait since August before they
are communicated to us.  If we were told back then about the problem,
we could have it solved in Emacs 28.2...

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Sat, 15 Oct 2022 19:26:53 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, monnier@iro.umontreal.ca,
>  liliana.prikler@gmail.com, rlb@defaultvalue.org, emacs-devel@gnu.org
> 
> I think it's a fine solution, but either comp-spawned-compilation
> should be renamed to comp-no-spawn, or its value in the forked Emacs
> processes should be nil...

Btw, instead of inventing yet another variable, we could make
native-comp-async-jobs-number accept one more value: -1, to mean
"don't run async jobs at all".

Your call.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think this should work, but there may be regressions, of course.
> Let's see...  I tried it now, and I got the warning below, but things
> otherwise seem to work fine:
>
> Warning (initialization): Unable to create `user-emacs-directory' (~/.emacs.d/).
> Any data that would normally be written there may be lost!
> If you never want to see this message again,
> customize the variable `user-emacs-directory-warning'.

We just got 28.2 into Debian, and we should likely re-evaluate that
version, as compared to 28.1.  See if the behaviors have changed.

> Emacs should work with an unwritable file system, but there's probably
> code that bugs out in that situation -- but those things should be
> fixed.
>
> If you have a test case that demonstrates the problem, please open a bug
> report for that, so that we can get fixin'.

OK, so if there is (or we come to) an upstream consensus that
HOME=/does-not-exist should work (would that be considered a subset of
the unwritable filesystem case?), then I think that's likely sufficient
for Debian packaging.  (That's what we were initially attempting.)

And if it doesn't yet work, we're very likely to be able to help track
down and fix those cases as we work through all the add-on packages.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, Po Lu <luangruo@yahoo.com>, akrl@sdf.org,
>  monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org
> Date: Sat, 15 Oct 2022 12:21:29 -0500
> 
> > Emacs should work with an unwritable file system, but there's probably
> > code that bugs out in that situation -- but those things should be
> > fixed.
> >
> > If you have a test case that demonstrates the problem, please open a bug
> > report for that, so that we can get fixin'.
> 
> OK, so if there is (or we come to) an upstream consensus that
> HOME=/does-not-exist should work (would that be considered a subset of
> the unwritable filesystem case?), then I think that's likely sufficient
> for Debian packaging.  (That's what we were initially attempting.)

It should work, yes.  And note that the message cited here, i.e.

> Warning (initialization): Unable to create `user-emacs-directory' (~/.emacs.d/).
> Any data that would normally be written there may be lost!
> If you never want to see this message again,
> customize the variable `user-emacs-directory-warning'.

is just a warning, it shouldn't prevent Emacs from running.  And yes,
features that want to save files under ~/.emacs.d will be unable to do
so, and could signal errors, unless you redefine the variables which
hold the respective file names to point to some other place.  This is
the intended behavior, I believe.

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Sat 15 Oct 2022 at 08:12PM +03, Eli Zaretskii wrote:

> This seems to be about byte-compilation?

I think it's mistitled, but nvm.

>> https://bugs.debian.org/1017845
>
> This is about trampolines indeed, and I believe we will have a
> solution VSN.

Nice.

> P.S. I wish such problems didn't have to wait since August before they
> are communicated to us.  If we were told back then about the problem,
> we could have it solved in Emacs 28.2...

Yes, sorry about that.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Michael Welsh Duggan <mwd@md5i.com> writes:

> 1) When building emacs (used as part of emacs bootstrap) 
> 2) When installing emacs via dpkg (maybe?  Not certain)
> 3) When building an emacs package (maybe?  Not certain)
> 4) When installing an emacs package via dpkg
> 5) When a user runs emacs

> 1) When building emacs (used as part of emacs bootstrap) 

"building the emacs package", which we do from an upstream tree usually
pretty close to the upstream tag, plus a varying number of patches.
Then it's mostly configure, make check, make DESTDIR=... install to
build the tree(s) that are packaged.

> 2) When installing emacs via dpkg (maybe?  Not certain)

Yes, "apt install emacs".  But that also includes upgrades.  The upgrade
and install scripts run as root, and should not affect HOME in most, if
not all, cases.

And note that both install and upgrade will run emacs a bazillion times
to rebuild all of the system-level .elc files for all of the installed
emacs "add-on" packages in dependency order.

> 3) When building an emacs package (maybe?  Not certain)

Yes, and that will likely run the installed /usr/bin/emacs any number of
times, in unpredictable ways, including via any test harnesses/processes
the package might have.

> 4) When installing an emacs package via dpkg

Yes, "apt install elpa-magit", which also includes upgrades, and will
invoke emacs to rebuild all of the elc files for the add-on.

> 5) When a user runs emacs

Yes, but in this case, 

> g) Case 4 might run emacs to build .elc and .eln files for the package's
>    .el files and place them in world-readable, read-only locations.  An
>    Emacs run in case 5 will include that location in its native file
>    search path.

Yes, if we proceed with ELN compilation (and don't also decide to start
packaging the .elc and .eln files too, rather than building them
on-demand -- which may be up for discussion, but would only be viable if
the emacs fingerprint only changed very deliberatlely).

> Open questions:
>
> i  ) In case 2, are the emacs binaries and the elisp files in the same
>      package, or are they split into different packages?  If the latter,
>      which package should contain the .eln files?

Currently, the elisp files are in a separate emacs-el package because
they used to be optional.  It sounds like they can't be anymore, so
emacs-common now depends on emacs-el, and everything else depends on
emacs-common.

I'd assume that the .eln files would go in the arch-dependent
emacs-bin-common package where other things like ctags, browse, etc. go.

Summary seems pretty accurate, overall.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Rob Browning [2022-10-15 11:34:14] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> I suppose we could just shadow emacs in the path with a wrapper that
>>> includes the argument(s), assuming few of the projects have their own
>>> interfering options, and that they always use the same (or a handful of
>>> predictable) relative invocation of emacs, e.g. "emacs" not
>>> "/usr/bin/emacs" or...
>> Or put a small chunk of ELisp in Debian's site-start file which sets the
>> eln load path according to some env-var :-)
> I wondered if that would be early enough.

The site-start file is basically the first chunk of non-builtin ELisp
code that is executed except for the `early-init.el` which is loaded
even before.

Trampolines should not be needed before we load such non-builtin code,
so I think the answer is that it's early enough unless the running
user doesn't has `early-init.el` which itself needs a trampoline.

> I'll have to double-check, but I think people may have found that some
> of the attempts to change it via the command line may not have been.

I think `--eval` is run too late (after the site-start), indeed.


        Stefan

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Also I think there was some fixing in this area but even in case is
> fixed completly now we'll have to cope with older version for a while.

Indeed.

But hopefully at some point in the future, we'll be able to use
libgccjit in-process, which should simplify things a bit, at least.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> I thought this change was just into master "for discussion", are we
> already suggesting users to integrate it in their infrastructures?

Yes, I should have mentioned that, as with everything on "master",
things may change.  But a mechanism more or less like this will exist.

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

>> BTW AFAIU this usecase was already supported in 28 using
>> `native-comp-deferred-compilation'.
> The deferred-compilation switch still ends up writing to $HOME, which
> the new inhibit one doesn't.  Should the former be deprecated in favor
> of the latter or should it simply take up its meaning?

It has been deprecated on "master".

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Sean Whitton <spwhitton@spwhitton.name> writes:

>> Emacs 29 now has the `inhibit-automatic-native-compilation' variable and
>> and `EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment variables
>> to inhibit writing to ~/.emacs.d/eln-cache...
>
> Is it likely to be okay for us to backport those commits to our 28.2?

Yes.  But probably wait a bit -- the semantics aren't set in stone yet.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

> OK, so if there is (or we come to) an upstream consensus that
> HOME=/does-not-exist should work (would that be considered a subset of
> the unwritable filesystem case?),

The cases are somewhat different, and may tickle different code paths.
For instance, there may be guards in the code where we want to write to,
for instance the autosave file, that checks whether the directory
exists, and if not, handles that gracefully.  But perhaps there's no
error handling around actually writing the file, leading Emacs to bug
out in bad ways.  (Just a random theoretical example.)

My guess is that we don't do much testing for the unwritable filesystems
case, so there may be many bugs in this area.

Re: Suppressing native compilation (short and long term)

[Liliana Marie Prikler]

Am Sonntag, dem 16.10.2022 um 10:55 +0200 schrieb Lars Ingebrigtsen:
> Andrea Corallo <akrl@sdf.org> writes:
> 
> > I thought this change was just into master "for discussion", are we
> > already suggesting users to integrate it in their infrastructures?
> 
> Yes, I should have mentioned that, as with everything on "master",
> things may change.  But a mechanism more or less like this will
> exist.
> 
> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> 
> > > BTW AFAIU this usecase was already supported in 28 using
> > > `native-comp-deferred-compilation'.
> > The deferred-compilation switch still ends up writing to $HOME,
> > which the new inhibit one doesn't.  Should the former be deprecated
> > in favor of the latter or should it simply take up its meaning?
> 
> It has been deprecated on "master".
But this deprecation may be reverted; I'm simply asking that if it is,
then native-comp-deferred-compilation should take up some of the
meaning of the switch you've implemented.  As a user, I find it
somewhat surprising that whatever means I have to inhibit deferred
compilation actually inhibit neither deferred compilation nor its side-
effects.

Cheers

Re: Suppressing native compilation (short and long term)

[Liliana Marie Prikler]

Am Sonntag, dem 16.10.2022 um 10:52 +0200 schrieb Lars Ingebrigtsen:
> Andrea Corallo <akrl@sdf.org> writes:
> 
> > Also I think there was some fixing in this area but even in case is
> > fixed completly now we'll have to cope with older version for a
> > while.
> 
> Indeed.
> 
> But hopefully at some point in the future, we'll be able to use
> libgccjit in-process, which should simplify things a bit, at least.
Also, you could detect whether an in-process libgccjit is available at
configure time.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

> But this deprecation may be reverted; I'm simply asking that if it is,
> then native-comp-deferred-compilation should take up some of the
> meaning of the switch you've implemented.

I don't think it's likely that it will be reverted.

Re: Suppressing native compilation (short and long term)

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1727 bytes --]

On Sat, Oct 15, 2022, 10:29 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Sat, 15 Oct 2022 10:10:23 -0400
> > Cc: Eli Zaretskii <eliz@gnu.org>, Andrea Corallo <akrl@sdf.org>, Lars
> Ingebrigtsen <larsi@gnus.org>,
> >       liliana.prikler@gmail.com, rlb@defaultvalue.org,
> >       emacs-devel <emacs-devel@gnu.org>
> >
> > Didn't you, a couple of years ago, suggest generating generic
> trampolines (per function signature), compile
> > those with gccjit, then just use them as needed rather than compiling
> separate trampolines for every subr?
> > Presumably the system would still occasionally need to compile a
> trampoline for a new signature, but the
> > generic trampoline would not be named so it couldn't cause an infinite
> recursion by being advised before
> > being compiled.
>
> If we want to compile all the trampolines AOT, that is already
> possible in Emacs 29, one just needs to actually do it when building
> Emacs.
>

The question was about whether trampolines could be designed in such a way
as to avoid invoking a separate compiler process without requiring
architecture-dependent binary hacking on templates.
I believe I was recalling one of Stefan M's suggestions for just this
problem from a couple of years ago.
Also, my understanding from this conversation is that trampolines may be
required at run-time that aren't known at build-time.  The issue is when
the advised function is somehow called by the compiler while compiling it's
own trampoline.  Since the call may come from the advised version of some
function used by the compiler, it's difficult to limit the risk without
controlling both the dump and the startup/configuration files.

Lynn

[-- Attachment #2: Type: text/html, Size: 2961 bytes --]

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> But hopefully at some point in the future, we'll be able to use
> libgccjit in-process, which should simplify things a bit, at least.

While an in-process calls to libgccjit make a lot of sense to build
trampolines (since we have to wait until the code is generated anyway),
but for all deferred compilations proceeding in a forked process seems
like a much better option.


        Stefan

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> While an in-process calls to libgccjit make a lot of sense to build
> trampolines (since we have to wait until the code is generated anyway),
> but for all deferred compilations proceeding in a forked process seems
> like a much better option.

Indeed.  But we don't always want the compilation to be deferred.  See
bug#58509.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> My guess is that we don't do much testing for the unwritable filesystems
> case, so there may be many bugs in this area.

Oh, bugs are "fine".

Right now, on that front, we just wanted to explain the situation and
determine what the expectations are.  i.e. if we hit trouble there, from
the upstream perspective are we in "don't do that then" territory, or
have we found something that might warrant attention?

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Andrea Corallo [2022-10-15 15:10:06] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> Dumb question: can't we just run the spawned compilation processes with
>>>> --no-site-file?
>>>
>>> For trampolines, I guess that should work since they shouldn't depend on
>>> local customizations.  Of course, a tempting alternative is to resort to
>>> "binary hacking", i.e. compile *one* template-trampoline and then
>>> generate all every other trampoline by copying that template and
>>> patching the right "stuff" into it.  That would save us from running the
>>> compiler to generate the trampolines (i.e. it would let us behave
>>> correctly on Windows even when GCC/libgccjit is not found at run time),
>>> but it would force us to write architecture-dependent code to patch the
>>> binary template.
>>
>> I think writing and maintaining arch dependent code to fix and
>> manipulate binaries is really a road we don't want to go down!
>>
>>  (a terrified) Andrea :)
>
> I tend to agree.  I just wish we could rely on some other tool to do
> that for us.

Yeah, I'm not aware of such a tool.  I guess a similar path lead GCL to
maintaining a (now very stale) in tree custom version of binutils.

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 15 Oct 2022 19:26:53 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: larsi@gnus.org, monnier@iro.umontreal.ca,
>>  liliana.prikler@gmail.com, rlb@defaultvalue.org, emacs-devel@gnu.org
>> 
>> I think it's a fine solution, but either comp-spawned-compilation
>> should be renamed to comp-no-spawn, or its value in the forked Emacs
>> processes should be nil...
>
> Btw, instead of inventing yet another variable, we could make
> native-comp-async-jobs-number accept one more value: -1, to mean
> "don't run async jobs at all".
>
> Your call.

SGTM thanks.

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

> Am Samstag, dem 15.10.2022 um 15:56 +0000 schrieb Andrea Corallo:
>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>> 
>> > Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
>> > 
>> > > The problem is that I can't meaningfully choose the "I don't want
>> > > JIT
>> > > for stuff I haven't AOT'd" option, especially not combined with
>> > > "but I
>> > > do want to load what I have AOT'd".
>> > 
>> > I haven't followed the Guix part of this thread in detail, but I
>> > thought
>> > I'd just note that Emacs 29 now has the
>> > `inhibit-automatic-native-compilation' variable and and
>> > `EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment variables,
>> > so
>> > this is implemented.
>> 
>> I thought this change was just into master "for discussion", are we
>> already suggesting users to integrate it in their infrastructures? 
>> BTW AFAIU this usecase was already supported in 28 using
>> `native-comp-deferred-compilation'.
> The deferred-compilation switch still ends up writing to $HOME, which
> the new inhibit one doesn't.

I think Eli explained more than once how to avoid that also without
using the new switch.

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
>
>> But this deprecation may be reverted; I'm simply asking that if it is,
>> then native-comp-deferred-compilation should take up some of the
>> meaning of the switch you've implemented.
>
> I don't think it's likely that it will be reverted.

You might be a little self-referential here.  This change was asked to
be reverted by the other head maintainer and by the maintainer of the
subsystem in object.

BTW how long is the "discussion period" supposed to last?

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lynn Winebarger <owinebar@gmail.com> writes:

> On Sat, Oct 15, 2022, 10:29 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
>  > From: Lynn Winebarger <owinebar@gmail.com>
>  > Date: Sat, 15 Oct 2022 10:10:23 -0400
>  > Cc: Eli Zaretskii <eliz@gnu.org>, Andrea Corallo <akrl@sdf.org>, Lars Ingebrigtsen <larsi@gnus.org>, 
>  >       liliana.prikler@gmail.com, rlb@defaultvalue.org, 
>  >       emacs-devel <emacs-devel@gnu.org>
>  > 
>  > Didn't you, a couple of years ago, suggest generating generic trampolines (per function signature), compile
>  > those with gccjit, then just use them as needed rather than compiling separate trampolines for every subr?
>  > Presumably the system would still occasionally need to compile a trampoline for a new signature, but the
>  > generic trampoline would not be named so it couldn't cause an infinite recursion by being advised before
>  > being compiled.
>
>  If we want to compile all the trampolines AOT, that is already
>  possible in Emacs 29, one just needs to actually do it when building
>  Emacs.
>
> The question was about whether trampolines could be designed in such a way as to avoid invoking a separate compiler
> process without requiring architecture-dependent binary hacking on templates.
> I believe I was recalling one of Stefan M's suggestions for just this problem from a couple of years ago.  
> Also, my understanding from this conversation is that trampolines may be required at run-time that aren't known at
> build-time.

The list of primitives is known at build time, so is the list of
possibly needed trampolines.

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

> Right now, on that front, we just wanted to explain the situation and
> determine what the expectations are.  i.e. if we hit trouble there, from
> the upstream perspective are we in "don't do that then" territory, or
> have we found something that might warrant attention?

The latter, definitely.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> You might be a little self-referential here.  This change was asked to
> be reverted by the other head maintainer and by the maintainer of the
> subsystem in object.

Well, I disagree, so here we are.

Re: Suppressing native compilation (short and long term)

[Liliana Marie Prikler]

Am Montag, dem 17.10.2022 um 07:23 +0000 schrieb Andrea Corallo:
> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> 
> > Am Samstag, dem 15.10.2022 um 15:56 +0000 schrieb Andrea Corallo:
> > > Lars Ingebrigtsen <larsi@gnus.org> writes:
> > > 
> > > > Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> > > > 
> > > > > The problem is that I can't meaningfully choose the "I don't
> > > > > want JIT for stuff I haven't AOT'd" option, especially not
> > > > > combined with "but I do want to load what I have AOT'd".
> > > > 
> > > > I haven't followed the Guix part of this thread in detail, but
> > > > I thought I'd just note that Emacs 29 now has the
> > > > `inhibit-automatic-native-compilation' variable and and
> > > > `EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment
> > > > variables, so this is implemented.
> > > 
> > > I thought this change was just into master "for discussion", are
> > > we already suggesting users to integrate it in their
> > > infrastructures? 
> > > BTW AFAIU this usecase was already supported in 28 using
> > > `native-comp-deferred-compilation'.
> > The deferred-compilation switch still ends up writing to $HOME,
> > which the new inhibit one doesn't.
> 
> I think Eli explained more than once how to avoid that also without
> using the new switch.
Now, I'm not following all branches of this discussion, so the chances
that I missed something are admittedly high, but the last time I
conversed with Eli about this, they said my use case was neither
supported nor something they'd consider supporting.  Thus, I'd be
positively surprised to see it in fact supported.

In any case, please do inform me about the solution once everything has
settled; I'll need to bump the emacs-next package in Guix, which
currently uses Lars' commit.

Cheers

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Liliana Marie Prikler <liliana.prikler@gmail.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, Eli Zaretskii <eliz@gnu.org>, 
> 	rlb@defaultvalue.org, emacs-devel@gnu.org
> Date: Mon, 17 Oct 2022 18:59:11 +0200
> 
> > > > > I haven't followed the Guix part of this thread in detail, but
> > > > > I thought I'd just note that Emacs 29 now has the
> > > > > `inhibit-automatic-native-compilation' variable and and
> > > > > `EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment
> > > > > variables, so this is implemented.
> > > > 
> > > > I thought this change was just into master "for discussion", are
> > > > we already suggesting users to integrate it in their
> > > > infrastructures? 
> > > > BTW AFAIU this usecase was already supported in 28 using
> > > > `native-comp-deferred-compilation'.
> > > The deferred-compilation switch still ends up writing to $HOME,
> > > which the new inhibit one doesn't.
> > 
> > I think Eli explained more than once how to avoid that also without
> > using the new switch.
> Now, I'm not following all branches of this discussion, so the chances
> that I missed something are admittedly high, but the last time I
> conversed with Eli about this, they said my use case was neither
> supported nor something they'd consider supporting.  Thus, I'd be
> positively surprised to see it in fact supported.

Which use case are you alluding to?

Andrea refers to preventing writes to user's HOME directory.  That
case is supported (in more than one way), and doesn't need the new
variable.  AFAIU, the use case you were presenting was not about
preventing writes to $HOME.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Date: Sat, 15 Oct 2022 19:26:53 +0300
>>> From: Eli Zaretskii <eliz@gnu.org>
>>> Cc: larsi@gnus.org, monnier@iro.umontreal.ca,
>>>  liliana.prikler@gmail.com, rlb@defaultvalue.org, emacs-devel@gnu.org
>>> 
>>> I think it's a fine solution, but either comp-spawned-compilation
>>> should be renamed to comp-no-spawn, or its value in the forked Emacs
>>> processes should be nil...
>>
>> Btw, instead of inventing yet another variable, we could make
>> native-comp-async-jobs-number accept one more value: -1, to mean
>> "don't run async jobs at all".
>>
>> Your call.
>
> SGTM thanks.
>
>   Andrea

Okay 1a8015b837 should do the job.  I eventually opted for adding
`comp-no-spawn' for now, seen the current "debated" situation on what we
should offer as interface to the user, I thought was better for now to
add a variable supposed to be used for internal use only.  We can remove
it and link the mechanism to `native-comp-async-jobs-number' afterwards
if we decide for it.

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>>>> I think it's a fine solution, but either comp-spawned-compilation
>>>> should be renamed to comp-no-spawn, or its value in the forked Emacs
>>>> processes should be nil...

[...]

> Okay 1a8015b837 should do the job.  I eventually opted for adding
> `comp-no-spawn' for now, seen the current "debated" situation on what we
> should offer as interface to the user, I thought was better for now to
> add a variable supposed to be used for internal use only.  We can remove
> it and link the mechanism to `native-comp-async-jobs-number' afterwards
> if we decide for it.

Is this supposed to fix the trampoline fork bomb?  (There's a lot of
stuff discussed in this thread...)  If so, it doesn't, because the
variable is set too late.  (As Eli suggested in another thread, we
should add a command line argument for this, and as I noted, that kind
of fits with the stuff discussed in bug#58509.)

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> Okay 1a8015b837 should do the job.  I eventually opted for adding
> `comp-no-spawn' for now, seen the current "debated" situation on what we
> should offer as interface to the user, I thought was better for now to
> add a variable supposed to be used for internal use only.  We can remove

Please use "--" in identifiers when they are "for internal use only".


        Stefan

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>>>> I think it's a fine solution, but either comp-spawned-compilation
>>>>> should be renamed to comp-no-spawn, or its value in the forked Emacs
>>>>> processes should be nil...
>
> [...]
>
>> Okay 1a8015b837 should do the job.  I eventually opted for adding
>> `comp-no-spawn' for now, seen the current "debated" situation on what we
>> should offer as interface to the user, I thought was better for now to
>> add a variable supposed to be used for internal use only.  We can remove
>> it and link the mechanism to `native-comp-async-jobs-number' afterwards
>> if we decide for it.
>
> Is this supposed to fix the trampoline fork bomb?  (There's a lot of
> stuff discussed in this thread...)  If so, it doesn't, because the
> variable is set too late.  (As Eli suggested in another thread, we
> should add a command line argument for this, and as I noted, that kind
> of fits with the stuff discussed in bug#58509.)

Right, I replied in bug#58509 with a patch that should fix the issue.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>>> Okay 1a8015b837 should do the job.  I eventually opted for adding

[...]

>> Is this supposed to fix the trampoline fork bomb?  (There's a lot of
>> stuff discussed in this thread...)  If so, it doesn't, because the
>> variable is set too late.  (As Eli suggested in another thread, we
>> should add a command line argument for this, and as I noted, that kind
>> of fits with the stuff discussed in bug#58509.)
>
> Right, I replied in bug#58509 with a patch that should fix the issue.

Does this mean that 1a8015b837 was not intended to address this issue?

Re: Suppressing native compilation (short and long term)

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 629 bytes --]

On Mon, Oct 17, 2022, 3:34 AM Andrea Corallo <akrl@sdf.org> wrote:

> Lynn Winebarger <owinebar@gmail.com> writes:
> > Also, my understanding from this conversation is that trampolines may be
> required at run-time that aren't known at
> > build-time.
>
> The list of primitives is known at build time, so is the list of
> possibly needed trampolines.


Dumb question: if trampolines are only used for primitives from the C
source code, could they be generated as C code at build time?  That would
eliminate the issue entirely and make the semantics of advising primitives
not depend on whether native compilation enabled.

Lynn

[-- Attachment #2: Type: text/html, Size: 1192 bytes --]

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lynn Winebarger <owinebar@gmail.com> writes:

> On Mon, Oct 17, 2022, 3:34 AM Andrea Corallo <akrl@sdf.org> wrote:
>
>  Lynn Winebarger <owinebar@gmail.com> writes:
>  > Also, my understanding from this conversation is that trampolines may be required at run-time that aren't known at
>  > build-time.
>
>  The list of primitives is known at build time, so is the list of
>  possibly needed trampolines.
>
> Dumb question: if trampolines are only used for primitives from the C
> source code,

Trampolines are used for call sites to primitives in native code.  They
are the reason why native code is fully resilient to primitive
redefinition (in contrast to C or bytecode).

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Andrea Corallo <akrl@sdf.org> writes:
>>
>>>>> So we need a recursion blocker in the trampoline generation function.
>>>>
>>>> Or -- since trampolines are really fast to generate and we need to do
>>>> that synchronously anyway -- we could just avoid forking an Emacs do
>>>> create the trampoline?
>>>
>>> We don't do that because libgccjit (well GCC really) leaks memory.
>>
>> Ah, darn.  Then I guess we do have to fork, and then come up with some
>> kind of way to tell that forked Emacs to not recurse any further?
>
> Yes
>
>> (Do you know whether libgccjit might stop leaking memory at some point?)
>
> Yes, there must be a bug on bugzilla ATM I've no trace of.  A colleague
> of mine did some investigation in the past and discovered IIRC that the
> GCC driver was the main source of the leakage. I'm Ccing him now
> hopefully he recalls/knows more than me.
>
> Also I think there was some fixing in this area but even in case is
> fixed completly now we'll have to cope with older version for a while.

Alex reached me out and pointed me to PR63854 [1].

Some fixing happened, but apparently ATM libgccjit is still, unless an
external driver is used, leaking quite badly :/

  Andrea

[1] <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63854>

Re: Suppressing native compilation (short and long term)

[Gregor Zattler]

Hi Eli,
* Eli Zaretskii <eliz@gnu.org> [2022-10-03; 19:19 +03]:
[... native compilation and laptop battery ..]
> You see, I have my data, and it seems to indicate a very short period
> of initial compilations for a modest consumption of resources.  (This
> isn't a laptop, but I'm acutely aware of my system's load at all
> times, and have it on the mode line, so I know what kind of
> computation is going on during that time.)  The data I see here
> doesn't look like it should be a problem for anyone, as long as files
> are compiled only as they are used. In my case, for example, I see
> maybe half a dozen *.eln files compiled after the initial startup.  I
> expect to see that on any machine where a user has steady usage
> patterns -- the compilation flats out very quickly.

Strangely, that is not what I see.  Instead every time I
start Emacs (since a week or so) like this

cd ~/src/emacs-master-next/src; gdb emacs -ex 'set logging file /tmp/gdb.txt' -ex 'set logging on' -ex 'set logging file /tmp/gdb.txt' -ex 'run --fg-daemon'

after roughly 30 seconds I see asynchronous compilation
processes which last at least 120 to most 150 seconds.
While this is the case in htop these compilation processes
are most of the time the top most cpu consumers, in powertop
I see several emacs processes are consuming power right
after the devices of my laptop.

What this means in actual power consumption I cannot say.

What is strange is, that these compilation processes start
over and over again, every time I start Emacs even if I do
not install/upgrade Emacs, nor packages.

I just did this three times in a row and it's a consistent
pattern.

This is GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu,
cairo version 1.16.0) of 2022-10-19

on

Debian bullseye right now with a backported kernel: Linux no
5.18.0-0.deb11.4-amd64 #1 SMP PREEMPT_DYNAMIC Debian
5.18.16-1~bpo11+1 (2022-08-12) x86_64 GNU/Linux



Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Gregor Zattler <telegraph@gmx.net>
> Cc: rlb@defaultvalue.org, monnier@iro.umontreal.ca, david@tethera.net,
>  emacs-devel@gnu.org, akrl@sdf.org
> Date: Sun, 23 Oct 2022 17:22:06 +0200
> 
> Hi Eli,
> * Eli Zaretskii <eliz@gnu.org> [2022-10-03; 19:19 +03]:
> [... native compilation and laptop battery ..]
> > You see, I have my data, and it seems to indicate a very short period
> > of initial compilations for a modest consumption of resources.  (This
> > isn't a laptop, but I'm acutely aware of my system's load at all
> > times, and have it on the mode line, so I know what kind of
> > computation is going on during that time.)  The data I see here
> > doesn't look like it should be a problem for anyone, as long as files
> > are compiled only as they are used. In my case, for example, I see
> > maybe half a dozen *.eln files compiled after the initial startup.  I
> > expect to see that on any machine where a user has steady usage
> > patterns -- the compilation flats out very quickly.
> 
> Strangely, that is not what I see.  Instead every time I
> start Emacs (since a week or so) like this
> 
> cd ~/src/emacs-master-next/src; gdb emacs -ex 'set logging file /tmp/gdb.txt' -ex 'set logging on' -ex 'set logging file /tmp/gdb.txt' -ex 'run --fg-daemon'
> 
> after roughly 30 seconds I see asynchronous compilation
> processes which last at least 120 to most 150 seconds.
> While this is the case in htop these compilation processes
> are most of the time the top most cpu consumers, in powertop
> I see several emacs processes are consuming power right
> after the devices of my laptop.

Please tell the details: which files are being compiled, and whether
do you see the corresponding *.eln files in the eln-cache afterwards.
Once a .eln file is in the eln-cache, the next invocation should not
re-compile the same file, it should load the compiled .eln file from
the eln-cache.

Could it be that your eln-cache is deleted between sessions?

Re: Suppressing native compilation (short and long term)

[Gregor Zattler]

Hi Eli,
* Eli Zaretskii <eliz@gnu.org> [2022-10-23; 19:07 +03]:
>> From: Gregor Zattler <telegraph@gmx.net>
>> Cc: rlb@defaultvalue.org, monnier@iro.umontreal.ca, david@tethera.net,
>>  emacs-devel@gnu.org, akrl@sdf.org
>> Date: Sun, 23 Oct 2022 17:22:06 +0200

>> Strangely, that is not what I see.  Instead every time I
>> start Emacs (since a week or so) like this
>>
>> cd ~/src/emacs-master-next/src; gdb emacs -ex 'set logging file /tmp/gdb.txt' -ex 'set logging on' -ex 'set logging file /tmp/gdb.txt' -ex 'run --fg-daemon'
>>
>> after roughly 30 seconds I see asynchronous compilation
>> processes which last at least 120 to most 150 seconds.
>> While this is the case in htop these compilation processes
>> are most of the time the top most cpu consumers, in powertop
>> I see several emacs processes are consuming power right
>> after the devices of my laptop.
>
> Please tell the details: which files are being compiled, and whether
> do you see the corresponding *.eln files in the eln-cache afterwards.
> Once a .eln file is in the eln-cache, the next invocation should not
> re-compile the same file, it should load the compiled .eln file from
> the eln-cache.

perhaps I misinterpreted and what I saw was the testing of
the .eln cache?:

I did

find "/home/grfz/.config/emacs/eln-cache/" "/home/grfz/src/emacs-master-next/native-lisp/" -type f | xargs -r ls -Altr > /tmp/before

<start Emacs again>

while sleep 1 ; do ps -eo pid,tty,stat,user,group,etime,time,cgroup,args  fax >> faxme; done  # while emacs started

find "/home/grfz/.config/emacs/eln-cache/" "/home/grfz/src/emacs-master-next/native-lisp/" -type f | xargs -r ls -Altr > /tmp/before

now

$ diff -aNur before after
$

--> No new .eln files were written.


But file faxme contains 280 lines like this:

/home/grfz/src/emacs-master-next/src/emacs --batch -l /tmp/emacs-async-comp-cl-lib-7Go9da.el



0 grfz@no:/tmp$ grep -o 'emacs-async-comp-.*' faxme | cut -d - -f 4- | sed -e "s/-[^-]*\.el$//" |sort -u | while read ; do grep -qs $REPLY before || echo $REPLY; done
0 grfz@no:/tmp$


So all these "emacs-async-comp-cl-lib-7Go9da.el" like files
have corresponding files in the .eln cache.

Is it possible that it takes 150 secs to test the .eln cache?


> Could it be that your eln-cache is deleted between sessions?

no.

Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Gregor Zattler <telegraph@gmx.net>
> Cc: spwhitton@spwhitton.name, rlb@defaultvalue.org,
>  monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org,
>  akrl@sdf.org
> Date: Sun, 23 Oct 2022 19:08:23 +0200
> 
> > Please tell the details: which files are being compiled, and whether
> > do you see the corresponding *.eln files in the eln-cache afterwards.
> > Once a .eln file is in the eln-cache, the next invocation should not
> > re-compile the same file, it should load the compiled .eln file from
> > the eln-cache.
> 
> perhaps I misinterpreted and what I saw was the testing of
> the .eln cache?:
> 
> I did
> 
> find "/home/grfz/.config/emacs/eln-cache/" "/home/grfz/src/emacs-master-next/native-lisp/" -type f | xargs -r ls -Altr > /tmp/before
> 
> <start Emacs again>
> 
> while sleep 1 ; do ps -eo pid,tty,stat,user,group,etime,time,cgroup,args  fax >> faxme; done  # while emacs started
> 
> find "/home/grfz/.config/emacs/eln-cache/" "/home/grfz/src/emacs-master-next/native-lisp/" -type f | xargs -r ls -Altr > /tmp/before
                                                ^^^^^^
I guess you mean "after", not "before"?

> But file faxme contains 280 lines like this:
> 
> /home/grfz/src/emacs-master-next/src/emacs --batch -l /tmp/emacs-async-comp-cl-lib-7Go9da.el

And there's already a cl-lib-XXXXX.eln file under eln-cache?

> 0 grfz@no:/tmp$ grep -o 'emacs-async-comp-.*' faxme | cut -d - -f 4- | sed -e "s/-[^-]*\.el$//" |sort -u | while read ; do grep -qs $REPLY before || echo $REPLY; done
> 0 grfz@no:/tmp$
> 
> 
> So all these "emacs-async-comp-cl-lib-7Go9da.el" like files
> have corresponding files in the .eln cache.
> 
> Is it possible that it takes 150 secs to test the .eln cache?

"Test" in what sense?  Who do you think needs to "test" the cache?

Anyway, the above is not what I meant.  You present some commands and
scripts, and expect me to guess what happens by showing only their
results.  It's very hard to analyze a problem that way.

Instead, please do this:

  . find the latest subdirectory of the eln-cache whose name is
    29.0.5-XXXX (where XXXX is some hash) and list is contents
  . start Emacs
  . use 'top' or similar command to see if async compilations are
    running which were started by Emacs you run above
  . compare the files these async compilations are compiling with what
    you see in that latest subdirectory of eln-cache
  . show the list of files that were compiled although they already
    had corresponding *.eln files in that subdirectory
  . let the compilations run to completion (use 'top' to see when
    there are no more compilations running)
  . look in the *Async-native-compile-log* buffer to see if there are
    any warning or error messages there; if there are, post them
  . kill Emacs
  . look into that subdirectory of the eln-cache and see whether there
    are any new *.eln files in it as result of the compilation

One reason why Emacs could keep compiling files that already have
*.eln files in the cache is because the existing *.eln files are
outdated or incompatible with the Emacs binary.  This happens a lot if
you are using the master branch, because we make significant changes
there frequently, and they require recompilation.  You will in those
cases see *.eln files whose base names are the same, but the hashes
are different.

Re: Suppressing native compilation (short and long term)

[Gregor Zattler]

Hi Eli,
* Gregor Zattler <telegraph@gmx.net> [2022-10-23; 19:08 +02]:
[...]
> I did
>
> find "/home/grfz/.config/emacs/eln-cache/" "/home/grfz/src/emacs-master-next/native-lisp/" -type f | xargs -r ls -Altr > /tmp/before
>
> <start Emacs again>
>
> while sleep 1 ; do ps -eo pid,tty,stat,user,group,etime,time,cgroup,args  fax >> faxme; done  # while emacs started
>
> find "/home/grfz/.config/emacs/eln-cache/" "/home/grfz/src/emacs-master-next/native-lisp/" -type f | xargs -r ls -Altr > /tmp/before

arhgs, that should have been:

find "/home/grfz/.config/emacs/eln-cache/" "/home/grfz/src/emacs-master-next/native-lisp/" -type f | xargs -r ls -Altr > /tmp/after

> now
>
> $ diff -aNur before after
> $
>
> --> No new .eln files were written.
>
>
> But file faxme contains 280 lines like this:
>
> /home/grfz/src/emacs-master-next/src/emacs --batch -l /tmp/emacs-async-comp-cl-lib-7Go9da.el
>
>
>
> 0 grfz@no:/tmp$ grep -o 'emacs-async-comp-.*' faxme | cut -d - -f 4- | sed -e "s/-[^-]*\.el$//" |sort -u | while read ; do grep -qs $REPLY before || echo $REPLY; done
> 0 grfz@no:/tmp$
>
>
> So all these "emacs-async-comp-cl-lib-7Go9da.el" like files
> have corresponding files in the .eln cache.
>
> Is it possible that it takes 150 secs to test the .eln cache?

I assumed this is only a test of mtime of files?


Ciao; Gregor

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Gregor Zattler <telegraph@gmx.net>
> Cc: spwhitton@spwhitton.name, rlb@defaultvalue.org,
>  monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org,
>  akrl@sdf.org
> Date: Sun, 23 Oct 2022 19:33:05 +0200
> 
> > Is it possible that it takes 150 secs to test the .eln cache?
> 
> I assumed this is only a test of mtime of files?

No, it's much more.  But it doesn't take so many seconds, of course.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Gregor Zattler <telegraph@gmx.net>
> Cc: rlb@defaultvalue.org, monnier@iro.umontreal.ca, david@tethera.net,
>  emacs-devel@gnu.org, akrl@sdf.org
> Date: Sun, 23 Oct 2022 17:22:06 +0200
> 
> This is GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu,
> cairo version 1.16.0) of 2022-10-19

Actually, there was a bug in native compilation that was fixed on that
very day.  So perhaps update from Git ans see if the problem of
repeated compilation goes away.

Re: Suppressing native compilation (short and long term)

[Gregor Zattler]

Hi Eli,
* Eli Zaretskii <eliz@gnu.org> [2022-10-23; 20:30 +03]:
>> From: Gregor Zattler <telegraph@gmx.net>
>> Cc: spwhitton@spwhitton.name, rlb@defaultvalue.org,
>>  monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org,
>>  akrl@sdf.org
>> Date: Sun, 23 Oct 2022 19:08:23 +0200

>> I did


A)

>> find "/home/grfz/.config/emacs/eln-cache/" "/home/grfz/src/emacs-master-next/native-lisp/" -type f | xargs -r ls -Altr > /tmp/before


B)

>> <start Emacs again>


C)

>> while sleep 1 ; do ps -eo pid,tty,stat,user,group,etime,time,cgroup,args  fax >> faxme; done  # while emacs started

this writes a snapshot of the currently running processes
every second and accumulates them.


D)

>> find "/home/grfz/.config/emacs/eln-cache/" "/home/grfz/src/emacs-master-next/native-lisp/" -type f | xargs -r ls -Altr > /tmp/before
                                                 ^^^^^^
> I guess you mean "after", not "before"?

You are right.


E)

$ diff -aNur before after
$

--> No new .eln files were written.


>> But file faxme contains 280 lines like this:
>>
>> /home/grfz/src/emacs-master-next/src/emacs --batch -l /tmp/emacs-async-comp-cl-lib-7Go9da.el
>
> And there's already a cl-lib-XXXXX.eln file under eln-cache?

Yes (see below)


G)

>> 0 grfz@no:/tmp$ grep -o 'emacs-async-comp-.*' faxme | cut -d - -f 4- | sed -e "s/-[^-]*\.el$//" |sort -u | while read ; do grep -qs $REPLY before || echo $REPLY; done
>> 0 grfz@no:/tmp$

This command shows that all the

/tmp/emacs-async-comp-cl-lib-7Go9da.el

like filenames in the processes reduced to (in this example)
"cl-lib-" have a corresponding "cl-lib-XXXXX.eln" file in
the .eln cache.


H)

Actually the command did not ensure that
the hit was in the most recent .eln cache directories, the
most recent (as an example) cl-lib-XXXXX.eln is

-rwxr-xr-x 1 grfz grfz 65832 Okt 19 12:56 /home/grfz/src/emacs-master-next/native-lisp/29.0.50-5a57c71f/cl-lib-8b938900-c76f14d9.eln


>> So all these "emacs-async-comp-cl-lib-7Go9da.el" like files
>> have corresponding files in the .eln cache.
>>
>> Is it possible that it takes 150 secs to test the .eln cache?
>
> "Test" in what sense?  Who do you think needs to "test" the cache?

Test if an existing .el file needs to be compiled to an .eln
file or if a corresponding and up-to-date .eln file for this
very .el file already exists.



> Anyway, the above is not what I meant.  You present some commands and
> scripts, and expect me to guess what happens by showing only their
> results.  It's very hard to analyze a problem that way.
>
> Instead, please do this:

>   . find the latest subdirectory of the eln-cache whose name is
>     29.0.5-XXXX (where XXXX is some hash) and list is contents


I)

$ ls -Altrd /home/grfz/.config/emacs/eln-cache/* /home/grfz/src/emacs-master-next/native-lisp/*
-rw------- 2 grfz grfz   188 Mai 19  2010 /home/grfz/.config/emacs/eln-cache/CACHEDIR.TAG
drwxr-xr-x 2 grfz grfz 36864 Jul 21 22:26 /home/grfz/.config/emacs/eln-cache/29.0.50-50117688/
drwxr-xr-x 2 grfz grfz 36864 Jul 31 11:31 /home/grfz/.config/emacs/eln-cache/29.0.50-9e3d75e3/
drwxr-xr-x 2 grfz grfz  4096 Aug  1 10:47 /home/grfz/.config/emacs/eln-cache/29.0.50-03ee865e/
drwxr-xr-x 2 grfz grfz 20480 Aug  2 21:00 /home/grfz/.config/emacs/eln-cache/29.0.50-f8d87b98/
drwxr-xr-x 2 grfz grfz  4096 Aug 11 13:26 /home/grfz/.config/emacs/eln-cache/28.1.91-6470e6f9/
drwxr-xr-x 2 grfz grfz 12288 Aug 20 17:01 /home/grfz/.config/emacs/eln-cache/29.0.50-548f8cdf/
drwxr-xr-x 2 grfz grfz 20480 Aug 20 19:11 /home/grfz/.config/emacs/eln-cache/29.0.50-b189cc80/
drwxr-xr-x 2 grfz grfz  4096 Aug 24 21:18 /home/grfz/.config/emacs/eln-cache/29.0.50-9d768eff/
drwxr-xr-x 2 grfz grfz 20480 Aug 26 10:43 /home/grfz/.config/emacs/eln-cache/29.0.50-121fd72d/
drwxr-xr-x 2 grfz grfz  4096 Aug 26 20:14 /home/grfz/.config/emacs/eln-cache/29.0.50-11460309/
drwxr-xr-x 2 grfz grfz 20480 Aug 27 11:27 /home/grfz/.config/emacs/eln-cache/29.0.50-00ab4019/
drwxr-xr-x 2 grfz grfz 20480 Sep  3 14:41 /home/grfz/.config/emacs/eln-cache/29.0.50-4cc289c2/
drwxr-xr-x 2 grfz grfz  4096 Sep 13 17:13 /home/grfz/.config/emacs/eln-cache/29.0.50-db6a9ec4/
drwxr-xr-x 2 grfz grfz 36864 Sep 23 19:02 /home/grfz/.config/emacs/eln-cache/29.0.50-54242f81/
drwxr-xr-x 2 grfz grfz 20480 Okt  8 21:09 /home/grfz/.config/emacs/eln-cache/29.0.50-89a6c1cd/
drwxr-xr-x 2 grfz grfz  4096 Okt  8 22:10 /home/grfz/.config/emacs/eln-cache/29.0.50-d0addadc/
drwxr-xr-x 2 grfz grfz 40960 Okt 19 12:01 /home/grfz/.config/emacs/eln-cache/29.0.50-4c1d7d1b/
drwxr-xr-x 2 grfz grfz 20480 Okt 19 13:04 /home/grfz/.config/emacs/eln-cache/29.0.50-1baaebd1/
drwxr-xr-x 3 grfz grfz 28672 Okt 19 14:03 /home/grfz/src/emacs-master-next/native-lisp/29.0.50-5a57c71f/
drwxr-xr-x 2 grfz grfz  4096 Okt 19 14:41 /home/grfz/src/emacs-master-next/native-lisp/29.0.50-9ff4cf87/
drwxr-xr-x 2 grfz grfz  4096 Okt 19 20:06 /home/grfz/.config/emacs/eln-cache/29.0.50-9ff4cf87/

The latest on only shows trampoline files:

$ ls -Altr /home/grfz/.config/emacs/eln-cache/29.0.50-9ff4cf87/
total 180
-rwxr-xr-x 1 grfz grfz 16872 Okt 19 15:22 subr--trampoline-73746172742d6b62642d6d6163726f_start_kbd_macro_0.eln*
-rwxr-xr-x 1 grfz grfz 16872 Okt 19 15:23 subr--trampoline-656e642d6b62642d6d6163726f_end_kbd_macro_0.eln*
-rwxr-xr-x 1 grfz grfz 16872 Okt 19 15:23 subr--trampoline-7965732d6f722d6e6f2d70_yes_or_no_p_0.eln*
-rwxr-xr-x 1 grfz grfz 16872 Okt 19 15:23 subr--trampoline-6b696c6c2d627566666572_kill_buffer_0.eln*
-rwxr-xr-x 1 grfz grfz 16872 Okt 19 15:33 subr--trampoline-782d666f6375732d6672616d65_x_focus_frame_0.eln*
-rwxr-xr-x 1 grfz grfz 16872 Okt 19 19:52 subr--trampoline-61626f72742d7265637572736976652d65646974_abort_recursive_edit_0.eln*
-rwxr-xr-x 1 grfz grfz 16872 Okt 19 19:52 subr--trampoline-746f702d6c6576656c_top_level_0.eln*
-rwxr-xr-x 1 grfz grfz 16872 Okt 19 19:52 subr--trampoline-6d616b652d70726f63657373_make_process_0.eln*
-rwxr-xr-x 1 grfz grfz 16872 Okt 19 20:06 subr--trampoline-6d657373616765_message_0.eln*


Surprise, the second to last eln cache directoriy only has
one file in it:

ls -Altr /home/grfz/src/emacs-master-next/native-lisp/29.0.50-9ff4cf87/
total 40
-rwxr-xr-x 1 grfz grfz 39536 Okt 19 14:41 debug-ce0d397b-01a0d81b.eln*


J)

But the third to last eln cache directory has more files:

$ ls -Altr /home/grfz/src/emacs-master-next/native-lisp/29.0.50-5a57c71f/
total 41744
-rwxr-xr-x 1 grfz grfz   86624 Okt 19 12:21 cconv-3b1f1f98-aafc4e46.eln*
-rwxr-xr-x 1 grfz grfz  221592 Okt 19 12:23 byte-opt-9c5f25f5-ca1fc8cd.eln*
-rwxr-xr-x 1 grfz grfz  452592 Okt 19 12:23 bytecomp-12882072-a85f7aaf.eln*
-rwxr-xr-x 1 grfz grfz   77528 Okt 19 12:24 loaddefs-gen-e8a3ad9c-8046a1f5.eln*
-rwxr-xr-x 1 grfz grfz   35288 Okt 19 12:24 radix-tree-669a468d-fa18562f.eln*
-rwxr-xr-x 1 grfz grfz  213952 Okt 19 12:24 comp-cstr-ef162ef7-0f1f9b64.eln*
-rwxr-xr-x 1 grfz grfz  950544 Okt 19 12:27 comp-7672a6ed-0c77f4fc.eln*
-rwxr-xr-x 1 grfz grfz   72832 Okt 19 12:28 titdic-cnv-765ac3be-7e3dd66e.eln*
-rwxr-xr-x 1 grfz grfz   37272 Okt 19 12:38 emoji-zwj-4f682c68-6523b72a.eln*
-rwxr-xr-x 1 grfz grfz   29080 Okt 19 12:38 charscript-600dca1a-50cd3215.eln*
drwxr-xr-x 2 grfz grfz   12288 Okt 19 12:52 preloaded/
-rwxr-xr-x 1 grfz grfz   79000 Okt 19 12:53 eieio-base-3621993d-e4d40210.eln*
-rwxr-xr-x 1 grfz grfz  103736 Okt 19 12:53 eieio-0db8d1d4-9439d1d2.eln*
-rwxr-xr-x 1 grfz grfz  115320 Okt 19 12:54 db-8199ba49-de500a31.eln*
-rwxr-xr-x 1 grfz grfz   47408 Okt 19 12:54 ja-dic-cnv-2eacdaa8-b3b103fd.eln*
-rwxr-xr-x 1 grfz grfz   48280 Okt 19 12:54 org-macro-6e7b0632-2b02ba5c.eln*
-rwxr-xr-x 1 grfz grfz  163496 Okt 19 12:54 ox-texinfo-e762f923-42b317b1.eln*
-rwxr-xr-x 1 grfz grfz  163536 Okt 19 12:55 oc-aec59d52-e1541149.eln*
-rwxr-xr-x 1 grfz grfz  143912 Okt 19 12:56 ol-1680a4ec-493f104f.eln*
-rwxr-xr-x 1 grfz grfz   65832 Okt 19 12:56 cl-lib-8b938900-c76f14d9.eln*
-rwxr-xr-x 1 grfz grfz  554632 Okt 19 12:56 ox-9aa46d10-36fd1809.eln*
-rwxr-xr-x 1 grfz grfz  450840 Okt 19 12:57 org-element-763f8d74-c2ed5726.eln*
-rwxr-xr-x 1 grfz grfz  103448 Okt 19 12:58 align-da164663-36320772.eln*
-rwxr-xr-x 1 grfz grfz  136608 Okt 19 12:59 allout-widgets-0bef4edc-759a8d7d.eln*
-rwxr-xr-x 1 grfz grfz 4661816 Okt 19 12:59 ja-dic-283bfd77-a5e4a1e7.eln*
-rwxr-xr-x 1 grfz grfz   30200 Okt 19 12:59 ansi-osc-b447f6a8-1801ba7c.eln*
-rwxr-xr-x 1 grfz grfz   73632 Okt 19 13:00 ansi-color-75eac800-c8aa485f.eln*
-rwxr-xr-x 1 grfz grfz  371920 Okt 19 13:01 allout-54433bcc-e0e759ed.eln*
-rwxr-xr-x 1 grfz grfz  109488 Okt 19 13:01 apropos-7c1ecbdf-9650b3d6.eln*
-rwxr-xr-x 1 grfz grfz   70240 Okt 19 13:02 array-b56b5eac-ec506bd8.eln*
-rwxr-xr-x 1 grfz grfz   58064 Okt 19 13:02 auth-source-pass-c498fbad-499c397b.eln*
-rwxr-xr-x 1 grfz grfz   42640 Okt 19 13:02 autoinsert-09f12447-b6d4206c.eln*
-rwxr-xr-x 1 grfz grfz   79232 Okt 19 13:04 autorevert-841d6890-8b9bcbc0.eln*
-rwxr-xr-x 1 grfz grfz  200848 Okt 19 13:04 arc-mode-bac5621c-4ac9d455.eln*
-rwxr-xr-x 1 grfz grfz   52040 Okt 19 13:05 avoid-3a50b57d-a616e3f0.eln*
-rwxr-xr-x 1 grfz grfz  180984 Okt 19 13:05 auth-source-49df7eef-1d56c61f.eln*
-rwxr-xr-x 1 grfz grfz   99000 Okt 19 13:05 battery-ff1f50e8-f1bc4ce1.eln*
-rwxr-xr-x 1 grfz grfz  187672 Okt 19 13:05 bookmark-8667481e-cc4ccd04.eln*
-rwxr-xr-x 1 grfz grfz   17088 Okt 19 13:05 cdl-b6e4b7af-883d0dac.eln*
-rwxr-xr-x 1 grfz grfz  125120 Okt 19 13:05 bs-87de9342-7fb047b4.eln*
-rwxr-xr-x 1 grfz grfz   30224 Okt 19 13:05 chistory-7b662674-200867ad.eln*
-rwxr-xr-x 1 grfz grfz   52544 Okt 19 13:06 cmuscheme-673be037-c6eb383e.eln*
-rwxr-xr-x 1 grfz grfz  372072 Okt 19 13:06 char-fold-b79b1b8c-7ee3c37f.eln*
-rwxr-xr-x 1 grfz grfz  136352 Okt 19 13:06 calculator-1eb85010-340dc1ca.eln*
-rwxr-xr-x 1 grfz grfz  116184 Okt 19 13:06 completion-a15d2372-8acb394d.eln*
-rwxr-xr-x 1 grfz grfz   56264 Okt 19 13:06 color-9d7980a5-8de89906.eln*
-rwxr-xr-x 1 grfz grfz   38832 Okt 19 13:06 cus-dep-6d2e8064-6e7c7f8e.eln*
-rwxr-xr-x 1 grfz grfz   69456 Okt 19 13:07 cus-theme-6f9d22e7-94be69bb.eln*
-rwxr-xr-x 1 grfz grfz  271296 Okt 19 13:07 comint-faef15ad-db425943.eln*
-rwxr-xr-x 1 grfz grfz   34944 Okt 19 13:07 delim-col-6e65f4b3-5c6927c7.eln*
-rwxr-xr-x 1 grfz grfz   65896 Okt 19 13:07 dabbrev-ed4e5147-48ba83e7.eln*
-rwxr-xr-x 1 grfz grfz   34496 Okt 19 13:07 delsel-1c603c42-f9174b7b.eln*
-rwxr-xr-x 1 grfz grfz  127184 Okt 19 13:07 desktop-428f9887-5221ebda.eln*
-rwxr-xr-x 1 grfz grfz   61928 Okt 19 13:08 dframe-2a07085b-9ec7b9c5.eln*
-rwxr-xr-x 1 grfz grfz   89816 Okt 19 13:08 descr-text-4ed9ee33-6e51a3e5.eln*
-rwxr-xr-x 1 grfz grfz  348400 Okt 19 13:08 cus-edit-3cd01345-0ff67bb8.eln*
-rwxr-xr-x 1 grfz grfz   78192 Okt 19 13:08 dired-x-a2e5c184-b25c9c36.eln*
-rwxr-xr-x 1 grfz grfz   30264 Okt 19 13:08 dirtrack-49e0129c-0551af6d.eln*
-rwxr-xr-x 1 grfz grfz   30632 Okt 19 13:08 display-fill-column-indicator-82af8525-87ab4f49.eln*
-rwxr-xr-x 1 grfz grfz   43720 Okt 19 13:08 display-line-numbers-1d060f2e-ac11bdb2.eln*
-rwxr-xr-x 1 grfz grfz  251832 Okt 19 13:09 dired-aux-1ff8c91a-d5f7e763.eln*
-rwxr-xr-x 1 grfz grfz  315872 Okt 19 13:09 dired-6a3ae2bc-07da73b7.eln*
-rwxr-xr-x 1 grfz grfz   25680 Okt 19 13:09 double-1384b58f-7d071d40.eln*
-rwxr-xr-x 1 grfz grfz   47824 Okt 19 13:09 dom-a7939831-14efc4a3.eln*
-rwxr-xr-x 1 grfz grfz   21776 Okt 19 13:09 echistory-286d3fb8-4d42b515.eln*
-rwxr-xr-x 1 grfz grfz   30576 Okt 19 13:09 ebuff-menu-ecbd6e97-bff4bb07.eln*
-rwxr-xr-x 1 grfz grfz  200600 Okt 19 13:09 doc-view-164df236-dea301f5.eln*
-rwxr-xr-x 1 grfz grfz   44400 Okt 19 13:09 ecomplete-d9b6d92f-48f548b1.eln*
-rwxr-xr-x 1 grfz grfz   44048 Okt 19 13:09 ehelp-a1fe76fc-c81b17aa.eln*
-rwxr-xr-x 1 grfz grfz   25760 Okt 19 13:09 elide-head-f64ec400-b3dc6449.eln*
-rwxr-xr-x 1 grfz grfz   30560 Okt 19 13:09 emacs-lock-9dfb7362-56bba4df.eln*
-rwxr-xr-x 1 grfz grfz   70720 Okt 19 13:09 elec-pair-9d724d9a-cf155de9.eln*
-rwxr-xr-x 1 grfz grfz   17328 Okt 19 13:09 epa-dired-774fd12a-c82884e5.eln*
-rwxr-xr-x 1 grfz grfz   46904 Okt 19 13:10 edmacro-74bf45a3-b6df9481.eln*
-rwxr-xr-x 1 grfz grfz   43568 Okt 19 13:10 epa-file-d15dd0b4-eb0a4471.eln*
-rwxr-xr-x 1 grfz grfz   38736 Okt 19 13:10 epa-mail-bf61c0db-10c78409.eln*
-rwxr-xr-x 1 grfz grfz  111520 Okt 19 13:10 epa-bdd8ea1c-902eeb9e.eln*
-rwxr-xr-x 1 grfz grfz   35072 Okt 19 13:10 epg-config-78240760-6b96d0a3.eln*
-rwxr-xr-x 1 grfz grfz   75080 Okt 19 13:10 epa-ks-e5664732-fb52afb2.eln*
-rwxr-xr-x 1 grfz grfz   35080 Okt 19 13:10 expand-9d6a83fd-bc169727.eln*
-rwxr-xr-x 1 grfz grfz   30280 Okt 19 13:10 ezimage-53d8406d-3f590e41.eln*
-rwxr-xr-x 1 grfz grfz   60872 Okt 19 13:10 face-remap-9d6c47ed-ef6ab2ca.eln*
-rwxr-xr-x 1 grfz grfz   74272 Okt 19 13:10 facemenu-cceb809d-28feab5d.eln*
-rwxr-xr-x 1 grfz grfz   58232 Okt 19 13:11 filecache-c607f1cc-b015ebc1.eln*
-rwxr-xr-x 1 grfz grfz   39304 Okt 19 13:11 fileloop-f8cb1238-1dba446f.eln*
-rwxr-xr-x 1 grfz grfz  147456 Okt 19 13:11 ffap-4b3c5789-a74be4a4.eln*
-rwxr-xr-x 1 grfz grfz   96744 Okt 19 13:11 filenotify-65939d6e-9aac6817.eln*
-rwxr-xr-x 1 grfz grfz   83128 Okt 19 13:11 files-x-59c65c89-ab7c86d0.eln*
-rwxr-xr-x 1 grfz grfz  404816 Okt 19 13:11 epg-de089247-9e84ee77.eln*
-rwxr-xr-x 1 grfz grfz  188904 Okt 19 13:11 filesets-6b603b3e-1859fd55.eln*
-rwxr-xr-x 1 grfz grfz   25760 Okt 19 13:11 find-cmd-c1ed7ad8-0eb680ba.eln*
-rwxr-xr-x 1 grfz grfz   38920 Okt 19 13:12 find-dired-b47bcf4a-89f704c0.eln*
-rwxr-xr-x 1 grfz grfz   56376 Okt 19 13:12 finder-cfd3373c-4f1b537b.eln*
-rwxr-xr-x 1 grfz grfz   39544 Okt 19 13:12 find-lisp-7a0199b9-bf8a387b.eln*
-rwxr-xr-x 1 grfz grfz   56784 Okt 19 13:12 find-file-29570019-e91c163b.eln*
-rwxr-xr-x 1 grfz grfz   17200 Okt 19 13:12 flow-ctrl-a0b1783a-1e34cedb.eln*
-rwxr-xr-x 1 grfz grfz   34288 Okt 19 13:12 foldout-a29a272c-443fd9d0.eln*
-rwxr-xr-x 1 grfz grfz   25744 Okt 19 13:12 format-spec-644c0068-b9fc233f.eln*
-rwxr-xr-x 1 grfz grfz  114856 Okt 19 13:12 follow-6a4faeee-cd78e901.eln*
-rwxr-xr-x 1 grfz grfz   87472 Okt 19 13:12 forms-21950d0a-c751d5ec.eln*
-rwxr-xr-x 1 grfz grfz   34640 Okt 19 13:12 help-at-pt-24a2e4cf-eb94cf74.eln*
-rwxr-xr-x 1 grfz grfz  112216 Okt 19 13:12 generic-x-3d4b3f79-be7d840a.eln*
-rwxr-xr-x 1 grfz grfz   25296 Okt 19 13:12 help-macro-b11d1ebc-ce6478ee.eln*
-rwxr-xr-x 1 grfz grfz  131032 Okt 19 13:12 frameset-94504960-582f574c.eln*
-rwxr-xr-x 1 grfz grfz   74464 Okt 19 13:13 help-mode-d4dbae3d-942d04d2.eln*
-rwxr-xr-x 1 grfz grfz   17336 Okt 19 13:13 hex-util-0952f412-6bf54c1d.eln*
-rwxr-xr-x 1 grfz grfz   42040 Okt 19 13:13 hfy-cmap-a102c87f-d37fadac.eln*
-rwxr-xr-x 1 grfz grfz  101136 Okt 19 13:13 hexl-eddf9831-49318734.eln*
-rwxr-xr-x 1 grfz grfz   78080 Okt 19 13:13 hi-lock-42477945-48eeb072.eln*
-rwxr-xr-x 1 grfz grfz   74952 Okt 19 13:13 hilit-chg-61a068fb-c53042c7.eln*
-rwxr-xr-x 1 grfz grfz  184240 Okt 19 13:13 help-fns-d233c6e8-63a53820.eln*
-rwxr-xr-x 1 grfz grfz   43568 Okt 19 13:13 hl-line-8fa29c14-822de8e8.eln*
-rwxr-xr-x 1 grfz grfz   70736 Okt 19 13:13 hippie-exp-9919e80b-c63cbd06.eln*
-rwxr-xr-x 1 grfz grfz  230144 Okt 19 13:14 ibuf-ext-5624402a-6d6997f3.eln*
-rwxr-xr-x 1 grfz grfz  170632 Okt 19 13:14 htmlfontify-accab028-69f1eafb.eln*
-rwxr-xr-x 1 grfz grfz  212960 Okt 19 13:14 ibuffer-c2dc0626-63015081.eln*
-rwxr-xr-x 1 grfz grfz   50328 Okt 19 13:14 ibuf-macs-eb4b06c3-7628d5cc.eln*
-rwxr-xr-x 1 grfz grfz   60864 Okt 19 13:14 ielm-2a8237b7-74589b82.eln*
-rwxr-xr-x 1 grfz grfz   25856 Okt 19 13:14 iimage-13f380a2-2ebdaf0d.eln*
-rwxr-xr-x 1 grfz grfz   30264 Okt 19 13:14 image-file-9034ac7e-db87497b.eln*
-rwxr-xr-x 1 grfz grfz   91728 Okt 19 13:15 icomplete-92f7cbf4-68920226.eln*
-rwxr-xr-x 1 grfz grfz   74640 Okt 19 13:15 imenu-a6693d03-0364eafc.eln*
-rwxr-xr-x 1 grfz grfz  131976 Okt 19 13:15 image-mode-f854d2db-fcb7dfeb.eln*
-rwxr-xr-x 1 grfz grfz   95608 Okt 19 13:15 info-look-27e24920-f99c60b4.eln*
-rwxr-xr-x 1 grfz grfz   34088 Okt 19 13:15 informat-c758f251-bf09288c.eln*
-rwxr-xr-x 1 grfz grfz  322568 Okt 19 13:16 ido-ccb260dc-a497d40d.eln*
-rwxr-xr-x 1 grfz grfz   52040 Okt 19 13:16 info-xref-df600c94-e0fb13e9.eln*
-rwxr-xr-x 1 grfz grfz   25848 Okt 19 13:16 isearchb-14a844f5-9cffa663.eln*
-rwxr-xr-x 1 grfz grfz   47904 Okt 19 13:16 jka-compr-40ebe92b-873e7826.eln*
-rwxr-xr-x 1 grfz grfz   88568 Okt 19 13:16 json-a90a1eab-350c449d.eln*
-rwxr-xr-x 1 grfz grfz   21776 Okt 19 13:16 kermit-847c9fbe-db99bcfa.eln*
-rwxr-xr-x 1 grfz grfz  101400 Okt 19 13:16 kmacro-048feaec-e2ae5d33.eln*
-rwxr-xr-x 1 grfz grfz  118800 Okt 19 13:16 jsonrpc-e62a9c36-4ac0ad99.eln*
-rwxr-xr-x 1 grfz grfz   39072 Okt 19 13:16 loadhist-e9175ed1-a9ab5961.eln*
-rwxr-xr-x 1 grfz grfz  331600 Okt 19 13:17 info-ce12c0ca-4d447d17.eln*
-rwxr-xr-x 1 grfz grfz   34904 Okt 19 13:17 lpr-06c2a39c-b453b930.eln*
-rwxr-xr-x 1 grfz grfz   56936 Okt 19 13:17 locate-bbb85898-15b7a742.eln*
-rwxr-xr-x 1 grfz grfz   25976 Okt 19 13:17 master-d6d42554-90ec46ea.eln*
-rwxr-xr-x 1 grfz grfz   25808 Okt 19 13:17 macros-51ef7b0a-141955fb.eln*
-rwxr-xr-x 1 grfz grfz   21416 Okt 19 13:17 mb-depth-a4691d0b-2aa9169a.eln*
-rwxr-xr-x 1 grfz grfz   34616 Okt 19 13:17 midnight-0df4540a-a34daf80.eln*
-rwxr-xr-x 1 grfz grfz   30048 Okt 19 13:17 minibuf-eldef-8b1407ef-3edcda03.eln*
-rwxr-xr-x 1 grfz grfz   30488 Okt 19 13:17 misc-9bdb6d96-71987c28.eln*
-rwxr-xr-x 1 grfz grfz   46776 Okt 19 13:17 md4-234ce043-38a388a1.eln*
-rwxr-xr-x 1 grfz grfz   43920 Okt 19 13:17 misearch-3d1286b0-5dfbae2a.eln*
-rwxr-xr-x 1 grfz grfz   21512 Okt 19 13:17 mouse-copy-747f314d-a0716908.eln*
-rwxr-xr-x 1 grfz grfz  134360 Okt 19 13:17 man-9b8001be-e10db098.eln*
-rwxr-xr-x 1 grfz grfz   30272 Okt 19 13:17 mouse-drag-b34287d2-5cd220d2.eln*
-rwxr-xr-x 1 grfz grfz   42408 Okt 19 13:17 notifications-bd0b4b34-4d586541.eln*
-rwxr-xr-x 1 grfz grfz   83968 Okt 19 13:18 msb-e477e33b-5de02d4c.eln*
-rwxr-xr-x 1 grfz grfz   25648 Okt 19 13:18 novice-cc2ef463-3555311a.eln*
-rwxr-xr-x 1 grfz grfz   21696 Okt 19 13:18 password-cache-187e4eec-58743954.eln*
-rwxr-xr-x 1 grfz grfz   35432 Okt 19 13:18 pcmpl-cvs-743b5ebc-1b2226ef.eln*
-rwxr-xr-x 1 grfz grfz   26824 Okt 19 13:18 pcmpl-git-794ef7f3-f17db3ff.eln*
-rwxr-xr-x 1 grfz grfz  132416 Okt 19 13:18 outline-afc41f82-860c9481.eln*
-rwxr-xr-x 1 grfz grfz   63816 Okt 19 13:18 pcmpl-gnu-5ece415b-b97b3f4c.eln*
-rwxr-xr-x 1 grfz grfz   39752 Okt 19 13:18 pcmpl-linux-060b988a-b0dca077.eln*
-rwxr-xr-x 1 grfz grfz  220336 Okt 19 13:18 mpc-ebb105de-612804e4.eln*
-rwxr-xr-x 1 grfz grfz   63840 Okt 19 13:18 pcmpl-rpm-d791f76b-92b51ed3.eln*
-rwxr-xr-x 1 grfz grfz   89448 Okt 19 13:18 pcmpl-unix-e208196d-5c6b0b5f.eln*
-rwxr-xr-x 1 grfz grfz   49544 Okt 19 13:18 pcmpl-x-c16436ad-e19e302a.eln*
-rwxr-xr-x 1 grfz grfz   53456 Okt 19 13:19 plstore-f04d6393-38883ed2.eln*
-rwxr-xr-x 1 grfz grfz   79416 Okt 19 13:19 pixel-scroll-2f9465ae-242e22e4.eln*
-rwxr-xr-x 1 grfz grfz  123792 Okt 19 13:19 pcomplete-81dbd8b0-8d6a7537.eln*
-rwxr-xr-x 1 grfz grfz  152680 Okt 19 13:19 proced-8fac17dd-0ddbe361.eln*
-rwxr-xr-x 1 grfz grfz  130952 Okt 19 13:19 profiler-345cb85e-b5357a8a.eln*
-rwxr-xr-x 1 grfz grfz  322056 Okt 19 13:19 printing-8558eaeb-ac24b34d.eln*
-rwxr-xr-x 1 grfz grfz   48480 Okt 19 13:20 ps-bdf-e53407b2-3787d715.eln*
-rwxr-xr-x 1 grfz grfz   26760 Okt 19 13:20 ps-samp-f57360e6-bbf3fa10.eln*
-rwxr-xr-x 1 grfz grfz   94872 Okt 19 13:20 ps-mule-2e9a6c45-d7e24a5c.eln*
-rwxr-xr-x 1 grfz grfz  133832 Okt 19 13:20 recentf-3c64dc62-3d5e44b7.eln*
-rwxr-xr-x 1 grfz grfz   60328 Okt 19 13:20 registry-bca51796-80e1f6d0.eln*
-rwxr-xr-x 1 grfz grfz   94080 Okt 19 13:20 rect-cd288962-0c863cb3.eln*
-rwxr-xr-x 1 grfz grfz   52096 Okt 19 13:20 repeat-443831f9-f727a7bb.eln*
-rwxr-xr-x 1 grfz grfz   25536 Okt 19 13:20 reposition-5c59bc1d-ea0cf33c.eln*
-rwxr-xr-x 1 grfz grfz   34288 Okt 19 13:20 reveal-65b32910-8009efbd.eln*
-rwxr-xr-x 1 grfz grfz   21456 Okt 19 13:20 rot13-82cc11d3-341fbb00.eln*
-rwxr-xr-x 1 grfz grfz  320752 Okt 19 13:20 ps-print-74a80610-186b597f.eln*
-rwxr-xr-x 1 grfz grfz   34848 Okt 19 13:20 rtree-8e3a7d8e-bea31fc8.eln*
-rwxr-xr-x 1 grfz grfz   39056 Okt 19 13:21 savehist-b722b772-8de510f4.eln*
-rwxr-xr-x 1 grfz grfz   43528 Okt 19 13:21 saveplace-a17120d6-0b02506a.eln*
-rwxr-xr-x 1 grfz grfz   26280 Okt 19 13:21 scroll-all-b76288a5-c97066ae.eln*
-rwxr-xr-x 1 grfz grfz   26064 Okt 19 13:21 scroll-lock-7af2dd31-80b5e08c.eln*
-rwxr-xr-x 1 grfz grfz   65328 Okt 19 13:21 ruler-mode-3c3fd53f-df5fcb4c.eln*
-rwxr-xr-x 1 grfz grfz   97672 Okt 19 13:21 shadowfile-a07f492d-dcf2798b.eln*
-rwxr-xr-x 1 grfz grfz  131296 Okt 19 13:21 server-0cc44189-7d051c39.eln*
-rwxr-xr-x 1 grfz grfz   46928 Okt 19 13:21 skeleton-435acb71-fdc776de.eln*
-rwxr-xr-x 1 grfz grfz  109504 Okt 19 13:21 so-long-a68117bf-f2a3256b.eln*
-rwxr-xr-x 1 grfz grfz  141176 Okt 19 13:22 shell-57930f41-a0615147.eln*
-rwxr-xr-x 1 grfz grfz   56360 Okt 19 13:22 sort-14dd51e7-e6a8ba97.eln*
-rwxr-xr-x 1 grfz grfz   17032 Okt 19 13:22 soundex-1b2d2ac2-ba002f59.eln*
-rwxr-xr-x 1 grfz grfz   17024 Okt 19 13:22 sqlite-351fdac0-27a5a873.eln*
-rwxr-xr-x 1 grfz grfz   34784 Okt 19 13:22 sqlite-mode-83559d5a-767317c6.eln*
-rwxr-xr-x 1 grfz grfz  117424 Okt 19 13:23 strokes-19dc35b2-fd9cd395.eln*
-rwxr-xr-x 1 grfz grfz   53976 Okt 19 13:23 svg-4c391752-1deded8e.eln*
-rwxr-xr-x 1 grfz grfz  263440 Okt 19 13:23 speedbar-2a9b6d1b-a500b18c.eln*
-rwxr-xr-x 1 grfz grfz   17120 Okt 19 13:23 tabify-b74f3a50-9f5191d5.eln*
-rwxr-xr-x 1 grfz grfz   25864 Okt 19 13:23 talk-dfb156fa-5efb1587.eln*
-rwxr-xr-x 1 grfz grfz   98064 Okt 19 13:23 tab-line-e55f541b-044b2bf8.eln*
-rwxr-xr-x 1 grfz grfz   61128 Okt 19 13:23 tempo-5756037f-5c44c637.eln*
-rwxr-xr-x 1 grfz grfz  136552 Okt 19 13:24 tar-mode-8829ee02-03a59aac.eln*
-rwxr-xr-x 1 grfz grfz  368256 Okt 19 13:24 ses-d640128c-4ea056d7.eln*
-rwxr-xr-x 1 grfz grfz   30824 Okt 19 13:24 thread-1b59497f-5823bb9c.eln*
-rwxr-xr-x 1 grfz grfz   62344 Okt 19 13:24 thingatpt-6fc8a4ab-5c620eb5.eln*
-rwxr-xr-x 1 grfz grfz   61416 Okt 19 13:24 time-bbe7023e-418f30bd.eln*
-rwxr-xr-x 1 grfz grfz   34968 Okt 19 13:24 timezone-f5339c31-838485ba.eln*
-rwxr-xr-x 1 grfz grfz   51744 Okt 19 13:24 tmm-12a7d710-bd9f1f90.eln*
-rwxr-xr-x 1 grfz grfz   55280 Okt 19 13:24 time-stamp-20b00c46-359942ab.eln*
-rwxr-xr-x 1 grfz grfz   21624 Okt 19 13:24 t-mouse-b934bcc3-6ea766b8.eln*
-rwxr-xr-x 1 grfz grfz   61016 Okt 19 13:24 tree-widget-8dccf6ba-aa31b64f.eln*
-rwxr-xr-x 1 grfz grfz  262400 Okt 19 13:25 term-fb034019-5137e438.eln*
-rwxr-xr-x 1 grfz grfz   60104 Okt 19 13:25 tutorial-2418da58-0d5c21f4.eln*
-rwxr-xr-x 1 grfz grfz   26464 Okt 19 13:25 userlock-60e3749c-fd2130e2.eln*
-rwxr-xr-x 1 grfz grfz   96448 Okt 19 13:25 type-break-1b71d0e6-73d5d80f.eln*
-rwxr-xr-x 1 grfz grfz   70336 Okt 19 13:25 vcursor-f24573d4-c99a7c93.eln*
-rwxr-xr-x 1 grfz grfz   74856 Okt 19 13:25 view-faefc6b2-9336d0e6.eln*
-rwxr-xr-x 1 grfz grfz   79464 Okt 19 13:25 wdired-01202d01-ece5bc7d.eln*
-rwxr-xr-x 1 grfz grfz   34728 Okt 19 13:25 wid-browse-e182a231-e688a13d.eln*
-rwxr-xr-x 1 grfz grfz  147024 Okt 19 13:26 whitespace-48717ff3-11a121a8.eln*
-rwxr-xr-x 1 grfz grfz  423304 Okt 19 13:26 transient-376febf1-21ae079d.eln*
-rwxr-xr-x 1 grfz grfz   71272 Okt 19 13:26 windmove-15827e24-5e608432.eln*
-rwxr-xr-x 1 grfz grfz   48536 Okt 19 13:26 winner-c3a8b092-b4847128.eln*
-rwxr-xr-x 1 grfz grfz   48072 Okt 19 13:26 xdg-9947111f-7737cb26.eln*
-rwxr-xr-x 1 grfz grfz   64824 Okt 19 13:26 xml-78b9f1ba-ca346c9b.eln*
-rwxr-xr-x 1 grfz grfz  283392 Okt 19 13:27 wid-edit-5b92861a-20de1749.eln*
-rwxr-xr-x 1 grfz grfz  246696 Okt 19 13:27 woman-468654b8-ce2d4a59.eln*
-rwxr-xr-x 1 grfz grfz   48080 Okt 19 13:27 xt-mouse-ad97d877-6cd66896.eln*
-rwxr-xr-x 1 grfz grfz   26080 Okt 19 13:27 yank-media-62540c94-43b16516.eln*
-rwxr-xr-x 1 grfz grfz  125792 Okt 19 13:27 xwidget-9ccb93b3-c50a90c3.eln*
-rwxr-xr-x 1 grfz grfz   98920 Okt 19 13:27 calc-aent-1719b1cd-cb2f991d.eln*
-rwxr-xr-x 1 grfz grfz  174488 Okt 19 13:29 calcalg3-cdd43fbd-eb7bd16b.eln*
-rwxr-xr-x 1 grfz grfz  233368 Okt 19 13:29 calc-alg-5fa19fcf-e239539a.eln*
-rwxr-xr-x 1 grfz grfz   83360 Okt 19 13:29 calc-bin-61359665-90237c69.eln*
-rwxr-xr-x 1 grfz grfz  338824 Okt 19 13:29 calcalg2-3641d50d-b28bb50c.eln*
-rwxr-xr-x 1 grfz grfz  117920 Okt 19 13:30 calc-comb-4d223239-ac552c55.eln*
-rwxr-xr-x 1 grfz grfz   48312 Okt 19 13:30 calc-cplx-49f3d288-e0e82bea.eln*
-rwxr-xr-x 1 grfz grfz  128984 Okt 19 13:31 calccomp-91a23141-f1acbb70.eln*
-rwxr-xr-x 1 grfz grfz  257304 Okt 19 13:31 calc-222b057e-99d16d60.eln*
-rwxr-xr-x 1 grfz grfz  104080 Okt 19 13:32 calc-embed-12840cac-ab2cd86f.eln*
-rwxr-xr-x 1 grfz grfz  296280 Okt 19 13:32 calc-arith-97da4592-ca2c9343.eln*
-rwxr-xr-x 1 grfz grfz   57840 Okt 19 13:33 calc-fin-5313d12c-5ee336ed.eln*
-rwxr-xr-x 1 grfz grfz   47816 Okt 19 13:33 calc-frac-13a225bf-a45c3aeb.eln*
-rwxr-xr-x 1 grfz grfz  268584 Okt 19 13:34 calc-ext-169a1473-fd44d6be.eln*
-rwxr-xr-x 1 grfz grfz  108632 Okt 19 13:34 calc-funcs-41ddedba-66c010cd.eln*
-rwxr-xr-x 1 grfz grfz   57576 Okt 19 13:35 calc-help-f82e4a83-c16aaedb.eln*
-rwxr-xr-x 1 grfz grfz   34936 Okt 19 13:35 calc-incom-bb13bc57-abb51ab2.eln*
-rwxr-xr-x 1 grfz grfz  130000 Okt 19 13:35 calc-graph-f953862d-c98a959c.eln*
-rwxr-xr-x 1 grfz grfz   47248 Okt 19 13:35 calc-keypd-5235a410-ba679b1c.eln*
-rwxr-xr-x 1 grfz grfz  236264 Okt 19 13:35 calc-forms-0364e2fe-ae86f85d.eln*
-rwxr-xr-x 1 grfz grfz   43776 Okt 19 13:36 calc-macs-86f6acaa-ee00de71.eln*
-rwxr-xr-x 1 grfz grfz  164760 Okt 19 13:36 calc-lang-77b67574-226c90a4.eln*
-rwxr-xr-x 1 grfz grfz   57912 Okt 19 13:36 calc-menu-43d1e6da-d07b5d5b.eln*
-rwxr-xr-x 1 grfz grfz  103912 Okt 19 13:36 calc-map-cec1a5a6-9f53b206.eln*
-rwxr-xr-x 1 grfz grfz   79000 Okt 19 13:37 calc-misc-0f75b984-58211cbd.eln*
-rwxr-xr-x 1 grfz grfz   99392 Okt 19 13:37 calc-mode-3448000a-f206306e.eln*
-rwxr-xr-x 1 grfz grfz  215728 Okt 19 13:37 calc-math-5c62dc12-adaa1128.eln*
-rwxr-xr-x 1 grfz grfz   51672 Okt 19 13:37 calc-mtx-709af3c9-4c36bd74.eln*
-rwxr-xr-x 1 grfz grfz   74928 Okt 19 13:37 calc-nlfit-a260e11b-76d3530d.eln*
-rwxr-xr-x 1 grfz grfz  120592 Okt 19 13:38 calc-poly-ff0ca9b3-b3108f6a.eln*
-rwxr-xr-x 1 grfz grfz   34120 Okt 19 13:38 calc-rules-162a6cbc-b28539a3.eln*
-rwxr-xr-x 1 grfz grfz   42872 Okt 19 13:39 calcsel2-45ef433d-de378d55.eln*
-rwxr-xr-x 1 grfz grfz   96872 Okt 19 13:40 calc-sel-92dc8d3c-64be2a50.eln*
-rwxr-xr-x 1 grfz grfz  198312 Okt 19 13:40 calc-prog-78e4aeb2-f714efaf.eln*
-rwxr-xr-x 1 grfz grfz   80528 Okt 19 13:40 calc-store-a2093015-9b25d5d8.eln*
-rwxr-xr-x 1 grfz grfz   66408 Okt 19 13:40 calc-stat-bb7f9833-6d8af6dd.eln*
-rwxr-xr-x 1 grfz grfz   26824 Okt 19 13:40 calc-trail-e8bfa5bd-9a0f4a7d.eln*
-rwxr-xr-x 1 grfz grfz   30184 Okt 19 13:40 calc-undo-21bf6c23-e7fcd3b8.eln*
-rwxr-xr-x 1 grfz grfz   43384 Okt 19 13:40 calc-stuff-781739a1-23a9165d.eln*
-rwxr-xr-x 1 grfz grfz  163016 Okt 19 13:40 calc-rewr-0c4bb3e0-58e58a9f.eln*
-rwxr-xr-x 1 grfz grfz   74448 Okt 19 13:40 calc-yank-a4aa7301-975f7821.eln*
-rwxr-xr-x 1 grfz grfz   64624 Okt 19 13:41 appt-6933a307-3f500bce.eln*
-rwxr-xr-x 1 grfz grfz  177520 Okt 19 13:41 calc-vec-8d9dd57c-3c676778.eln*
-rwxr-xr-x 1 grfz grfz  189112 Okt 19 13:41 calc-units-12fe779d-de3ff8ab.eln*
-rwxr-xr-x 1 grfz grfz   56288 Okt 19 13:41 cal-bahai-caf89889-51cab3a9.eln*
-rwxr-xr-x 1 grfz grfz   47984 Okt 19 13:42 cal-coptic-d7417646-2da80eb6.eln*
-rwxr-xr-x 1 grfz grfz   68608 Okt 19 13:42 cal-dst-f85a5a15-9c95e4aa.eln*
-rwxr-xr-x 1 grfz grfz  108440 Okt 19 13:42 cal-china-28218d30-7928f9a2.eln*
-rwxr-xr-x 1 grfz grfz   51904 Okt 19 13:42 cal-french-443613b5-72ea1e9a.eln*
-rwxr-xr-x 1 grfz grfz   69624 Okt 19 13:43 cal-html-84e76775-ddb1ca5a.eln*
-rwxr-xr-x 1 grfz grfz  256024 Okt 19 13:43 calendar-d19e5c14-866d5903.eln*
-rwxr-xr-x 1 grfz grfz   34512 Okt 19 13:43 cal-iso-8ac36734-fc91459c.eln*
-rwxr-xr-x 1 grfz grfz   60712 Okt 19 13:43 cal-islam-74de7ee1-4a919494.eln*
-rwxr-xr-x 1 grfz grfz   39216 Okt 19 13:43 cal-julian-969ba5f7-1c57247a.eln*
-rwxr-xr-x 1 grfz grfz   30088 Okt 19 13:43 cal-menu-9380b697-a0f24163.eln*
-rwxr-xr-x 1 grfz grfz   64816 Okt 19 13:44 cal-move-aed18331-70622161.eln*
-rwxr-xr-x 1 grfz grfz  153336 Okt 19 13:44 cal-hebrew-32b77d47-480945fd.eln*
-rwxr-xr-x 1 grfz grfz   69664 Okt 19 13:44 cal-mayan-43c35bcf-42168f0b.eln*
-rwxr-xr-x 1 grfz grfz   21696 Okt 19 13:44 cal-x-8d62ebf5-7d7cc605.eln*
-rwxr-xr-x 1 grfz grfz   34736 Okt 19 13:44 cal-persia-39d88853-f564a483.eln*
-rwxr-xr-x 1 grfz grfz  101760 Okt 19 13:46 holidays-39d09682-859a6888.eln*
-rwxr-xr-x 1 grfz grfz  212920 Okt 19 13:46 cal-tex-9f624404-454ba4e9.eln*
-rwxr-xr-x 1 grfz grfz   38912 Okt 19 13:47 iso8601-3903451a-7b67c90d.eln*
-rwxr-xr-x 1 grfz grfz  166456 Okt 19 13:47 icalendar-3066204f-9fec696b.eln*
-rwxr-xr-x 1 grfz grfz   34864 Okt 19 13:47 parse-time-ca7017be-39be3991.eln*
-rwxr-xr-x 1 grfz grfz   58944 Okt 19 13:47 lunar-791aa94f-a3620369.eln*
-rwxr-xr-x 1 grfz grfz  264280 Okt 19 13:47 diary-lib-57afc63e-19a7cbdf.eln*
-rwxr-xr-x 1 grfz grfz   60192 Okt 19 13:48 time-date-40951a48-f2fbd30f.eln*
-rwxr-xr-x 1 grfz grfz  131584 Okt 19 13:48 timeclock-a2d57d06-bbbfe488.eln*
-rwxr-xr-x 1 grfz grfz   26128 Okt 19 13:48 cedet-cscope-87f0c476-c02142a9.eln*
-rwxr-xr-x 1 grfz grfz   21152 Okt 19 13:48 cedet-e5d89324-af54da04.eln*
-rwxr-xr-x 1 grfz grfz   17352 Okt 19 13:48 cedet-files-ec97a64b-70698494.eln*
-rwxr-xr-x 1 grfz grfz   26424 Okt 19 13:49 cedet-global-4ae0bafb-c5909a7c.eln*
-rwxr-xr-x 1 grfz grfz   26344 Okt 19 13:49 cedet-idutils-e5eca828-03569772.eln*
-rwxr-xr-x 1 grfz grfz  111560 Okt 19 13:49 solar-41ca7795-7b23192e.eln*
-rwxr-xr-x 1 grfz grfz   77136 Okt 19 13:49 data-debug-01abd8ab-23637260.eln*
-rwxr-xr-x 1 grfz grfz  142040 Okt 19 13:49 ede-5a8a99f9-3c1866b0.eln*
-rwxr-xr-x 1 grfz grfz   26232 Okt 19 13:49 pulse-35e729a5-4385997e.eln*
-rwxr-xr-x 1 grfz grfz   96112 Okt 19 13:49 mode-local-52e88da7-e3e6b204.eln*
-rwxr-xr-x 1 grfz grfz   21504 Okt 19 13:49 srecode-f9a0bb35-8d1e660d.eln*
-rwxr-xr-x 1 grfz grfz  100024 Okt 19 13:49 semantic-0e6d83f5-373d236b.eln*
-rwxr-xr-x 1 grfz grfz   43736 Okt 19 13:50 autoconf-edit-9fac18a2-11513ca0.eln*
-rwxr-xr-x 1 grfz grfz   43472 Okt 19 13:50 auto-aad53e20-8f2ff3bd.eln*
-rwxr-xr-x 1 grfz grfz   75416 Okt 19 13:50 base-064e31be-14a3d5c4.eln*
-rwxr-xr-x 1 grfz grfz   80216 Okt 19 13:50 config-5e70a6cb-c9e05f95.eln*
-rwxr-xr-x 1 grfz grfz   57064 Okt 19 13:50 cpp-root-435af696-63bfb39a.eln*
-rwxr-xr-x 1 grfz grfz   43816 Okt 19 13:50 custom-d88895eb-e6050aad.eln*
-rwxr-xr-x 1 grfz grfz   30952 Okt 19 13:50 detect-c9d7c2b3-95c0f88a.eln*
-rwxr-xr-x 1 grfz grfz   30232 Okt 19 13:50 dired-e18206ac-bec97729.eln*
-rwxr-xr-x 1 grfz grfz   53192 Okt 19 13:50 emacs-e09048e1-793049b3.eln*
-rwxr-xr-x 1 grfz grfz   61720 Okt 19 13:50 files-9f30c50f-3fd74907.eln*
-rwxr-xr-x 1 grfz grfz   66408 Okt 19 13:50 linux-eedacb0e-0742f0d4.eln*
-rwxr-xr-x 1 grfz grfz   67008 Okt 19 13:51 generic-c019b5ab-d72bf563.eln*
-rwxr-xr-x 1 grfz grfz  442640 Okt 19 13:51 todo-mode-595bef2f-75c9cbb3.eln*
-rwxr-xr-x 1 grfz grfz   21296 Okt 19 13:51 make-a4d1a972-dbe8735f.eln*
-rwxr-xr-x 1 grfz grfz   21936 Okt 19 13:51 makefile-edit-92f94c53-b029ddab.eln*
-rwxr-xr-x 1 grfz grfz   66856 Okt 19 13:51 locate-63ce93dd-49c06f90.eln*
-rwxr-xr-x 1 grfz grfz   39136 Okt 19 13:51 pconf-e1039d7f-2f89b351.eln*
-rwxr-xr-x 1 grfz grfz   34784 Okt 19 13:51 proj-archive-931a920a-7ded753a.eln*
-rwxr-xr-x 1 grfz grfz   30408 Okt 19 13:51 proj-aux-6dbd327a-d98fcf8c.eln*
-rwxr-xr-x 1 grfz grfz   61856 Okt 19 13:51 proj-comp-2f18b2f6-4d3fc736.eln*
-rwxr-xr-x 1 grfz grfz   91488 Okt 19 13:51 pmake-41620d85-b1c90f4f.eln*
-rwxr-xr-x 1 grfz grfz   74856 Okt 19 13:51 proj-elisp-9926ccad-66a68f91.eln*
-rwxr-xr-x 1 grfz grfz   91888 Okt 19 13:51 proj-3b75ff59-dbc45196.eln*
-rwxr-xr-x 1 grfz grfz   52096 Okt 19 13:51 proj-info-e54aa80a-96a42632.eln*
-rwxr-xr-x 1 grfz grfz  147144 Okt 19 13:51 project-am-8c1ba5b1-579cb35a.eln*
-rwxr-xr-x 1 grfz grfz   30600 Okt 19 13:51 proj-misc-cd3625c9-e7df458d.eln*
-rwxr-xr-x 1 grfz grfz   26216 Okt 19 13:51 proj-scheme-e19dcdf2-503c4a73.eln*
-rwxr-xr-x 1 grfz grfz   48352 Okt 19 13:52 proj-obj-160a1567-cfdde3d4.eln*
-rwxr-xr-x 1 grfz grfz   52352 Okt 19 13:52 proj-prog-104d733c-dba5db4c.eln*
-rwxr-xr-x 1 grfz grfz   43248 Okt 19 13:52 proj-shared-e6b4fd1d-1d145605.eln*
-rwxr-xr-x 1 grfz grfz   26200 Okt 19 13:52 shell-2dd33f2c-bff6236d.eln*
-rwxr-xr-x 1 grfz grfz   31048 Okt 19 13:52 simple-b70c3268-12362a7a.eln*
-rwxr-xr-x 1 grfz grfz   30536 Okt 19 13:52 source-a030dba0-865d485b.eln*
-rwxr-xr-x 1 grfz grfz   53072 Okt 19 13:52 speedbar-be64e296-c298692a.eln*
-rwxr-xr-x 1 grfz grfz   25880 Okt 19 13:52 srecode-5803a755-b642c232.eln*
-rwxr-xr-x 1 grfz grfz   30560 Okt 19 13:52 system-7bc2aed2-791db77e.eln*
-rwxr-xr-x 1 grfz grfz   30616 Okt 19 13:52 util-fac6ec1c-20e119f0.eln*
-rwxr-xr-x 1 grfz grfz   43168 Okt 19 13:52 bovine-2f253aba-d7a490d5.eln*
-rwxr-xr-x 1 grfz grfz   39632 Okt 19 13:52 chart-1df0f00d-4cbc69db.eln*
-rwxr-xr-x 1 grfz grfz  114520 Okt 19 13:52 analyze-0736102d-0e404dae.eln*
-rwxr-xr-x 1 grfz grfz   26768 Okt 19 13:52 db-debug-1b7ffd95-14eb2e22.eln*
-rwxr-xr-x 1 grfz grfz  221576 Okt 19 13:53 complete-2b67d14d-529c6816.eln*
-rwxr-xr-x 1 grfz grfz   75904 Okt 19 13:53 db-el-4fc170b9-afb4defb.eln*
-rwxr-xr-x 1 grfz grfz  101848 Okt 19 13:53 db-ebrowse-54480d61-c988f53d.eln*
-rwxr-xr-x 1 grfz grfz   97920 Okt 19 13:53 ctxt-dc25695a-389f480a.eln*
-rwxr-xr-x 1 grfz grfz   65416 Okt 19 13:53 db-file-3009064d-efaeeedc.eln*
-rwxr-xr-x 1 grfz grfz   62392 Okt 19 13:53 db-javascript-88df227b-61de1a13.eln*
-rwxr-xr-x 1 grfz grfz   57832 Okt 19 13:53 db-global-c899a3cf-bb0095c0.eln*
-rwxr-xr-x 1 grfz grfz   39320 Okt 19 13:53 db-mode-28bbbf2a-b043681a.eln*
-rwxr-xr-x 1 grfz grfz   43264 Okt 19 13:53 db-ref-1cbbe2c8-10294282.eln*
-rwxr-xr-x 1 grfz grfz   71016 Okt 19 13:53 debug-1b989c3f-790d7b9d.eln*
-rwxr-xr-x 1 grfz grfz  138160 Okt 19 13:53 db-find-f00ee9ff-4fd341c2.eln*
-rwxr-xr-x 1 grfz grfz   44968 Okt 19 13:53 decorate-059cc323-065b8f90.eln*
-rwxr-xr-x 1 grfz grfz   79440 Okt 19 13:54 db-typecache-81d46fa2-c186a322.eln*
-rwxr-xr-x 1 grfz grfz   30552 Okt 19 13:54 dep-77b6feec-52556097.eln*
-rwxr-xr-x 1 grfz grfz   34824 Okt 19 13:54 doc-4efe007d-9aad7d0e.eln*
-rwxr-xr-x 1 grfz grfz   56728 Okt 19 13:54 ede-grammar-8cc19dfb-913e4127.eln*
-rwxr-xr-x 1 grfz grfz   97816 Okt 19 13:54 find-03e42e4a-bd0cc84a.eln*
-rwxr-xr-x 1 grfz grfz   73752 Okt 19 13:54 edit-359026ad-0390b97c.eln*
-rwxr-xr-x 1 grfz grfz   48088 Okt 19 13:54 fw-fda2d4f2-e46232f6.eln*
-rwxr-xr-x 1 grfz grfz  101864 Okt 19 13:55 format-ebbc6838-9a6b7ccf.eln*
-rwxr-xr-x 1 grfz grfz   44200 Okt 19 13:55 html-0dfe9b31-0cd90921.eln*
-rwxr-xr-x 1 grfz grfz   56712 Okt 19 13:55 ia-6745d2f4-caa6a23a.eln*
-rwxr-xr-x 1 grfz grfz  233848 Okt 19 13:55 grammar-7a8b1aed-883a55b3.eln*
-rwxr-xr-x 1 grfz grfz   61480 Okt 19 13:55 ia-sb-f610bf29-bf8b45ec.eln*
-rwxr-xr-x 1 grfz grfz  274488 Okt 19 13:55 grm-wy-boot-72e0b49d-872491e8.eln*
-rwxr-xr-x 1 grfz grfz   53344 Okt 19 13:56 imenu-a1821d64-6331a67c.eln*
-rwxr-xr-x 1 grfz grfz   80072 Okt 19 13:56 java-888c7d7a-3b86345f.eln*
-rwxr-xr-x 1 grfz grfz  158872 Okt 19 13:56 idle-ddea1a59-ac68dd67.eln*
-rwxr-xr-x 1 grfz grfz   65544 Okt 19 13:56 mru-bookmark-392dffc3-349f8408.eln*
-rwxr-xr-x 1 grfz grfz   61512 Okt 19 13:56 sb-0ac49e47-5600e7b5.eln*
-rwxr-xr-x 1 grfz grfz  126808 Okt 19 13:56 lex-spp-7e2ae6b1-99c6bfed.eln*
-rwxr-xr-x 1 grfz grfz  193640 Okt 19 13:56 lex-ce219d01-54a1050c.eln*
-rwxr-xr-x 1 grfz grfz  102920 Okt 19 13:58 senator-8eaf8c42-875ae1b7.eln*
-rwxr-xr-x 1 grfz grfz   91808 Okt 19 13:58 scope-3d34b0cd-7a18b846.eln*
-rwxr-xr-x 1 grfz grfz   85136 Okt 19 13:58 sort-cdd6576f-c7778ba3.eln*
-rwxr-xr-x 1 grfz grfz   66064 Okt 19 13:58 symref-6c63732c-db778e72.eln*
-rwxr-xr-x 1 grfz grfz   43344 Okt 19 13:58 tag-file-17a3ab0a-bab06b4e.eln*
-rwxr-xr-x 1 grfz grfz   43632 Okt 19 13:59 tag-write-5b235879-8b8b5c70.eln*
-rwxr-xr-x 1 grfz grfz  167176 Okt 19 14:00 tag-baa603ad-e2302d3a.eln*
-rwxr-xr-x 1 grfz grfz   71176 Okt 19 14:00 texi-b4fe6646-53621ac0.eln*
-rwxr-xr-x 1 grfz grfz   52280 Okt 19 14:01 util-f6c26979-d4c0f66c.eln*
-rwxr-xr-x 1 grfz grfz   74800 Okt 19 14:01 tag-ls-7f8a4456-0f2fc2ff.eln*
-rwxr-xr-x 1 grfz grfz   42856 Okt 19 14:02 wisent-786cee7a-9eb33ebf.eln*
-rwxr-xr-x 1 grfz grfz  108912 Okt 19 14:02 util-modes-835f6a2c-502a1dc6.eln*
-rwxr-xr-x 1 grfz grfz   43128 Okt 19 14:02 complete-8049fcd7-27a541b2.eln*
-rwxr-xr-x 1 grfz grfz   47800 Okt 19 14:03 fcn-9a27a4a6-e051d56e.eln*
-rwxr-xr-x 1 grfz grfz   64848 Okt 19 14:03 debug-e2a68966-df0ca671.eln*
-rwxr-xr-x 1 grfz grfz   56936 Okt 19 14:03 refs-bbf70991-389b9d4b.eln*

>   . start Emacs
>   . use 'top' or similar command to see if async compilations are
>     running which were started by Emacs you run above
>   . compare the files these async compilations are compiling with what
>     you see in that latest subdirectory of eln-cache


K)

This is a list of probably all the files from processes like
this one:

/home/grfz/src/emacs-master-next/src/emacs --batch -l /tmp/emacs-async-comp-.......el

/tmp/emacs-async-comp-ansi-color-pUV33H.el
/tmp/emacs-async-comp-ansi-osc-9uy5Z7.el
/tmp/emacs-async-comp-async-bytecomp-19BQgz.el
/tmp/emacs-async-comp-async-CG3661.el
/tmp/emacs-async-comp-auth-source-nLKdu9.el
/tmp/emacs-async-comp-avoid-qTHnAl.el
/tmp/emacs-async-comp-battery-nR8yrq.el
/tmp/emacs-async-comp-bbdb-wNjNnU.el
/tmp/emacs-async-comp-bind-key-nT82iR.el
/tmp/emacs-async-comp-bookmark-NaKH2G.el
/tmp/emacs-async-comp-bytecomp-cOfm1Y.el
/tmp/emacs-async-comp-byte-opt-D13On8.el
/tmp/emacs-async-comp-calendar-nLweLU.el
/tmp/emacs-async-comp-calfw-cal-112gxS.el
/tmp/emacs-async-comp-calfw-HgNycP.el
/tmp/emacs-async-comp-calfw-ical-1eRC8T.el
/tmp/emacs-async-comp-calfw-org-xpzBat.el
/tmp/emacs-async-comp-cal-menu-uzBSOE.el
/tmp/emacs-async-comp-cconv-XgTAiT.el
/tmp/emacs-async-comp-cedet-wW0nUH.el
/tmp/emacs-async-comp-cl-lib-NwHLAC.el
/tmp/emacs-async-comp-comint-6TAyYC.el
/tmp/emacs-async-comp-comp-cstr-yIgiNj.el
/tmp/emacs-async-comp-comp-qSvZwH.el
/tmp/emacs-async-comp-coolj-PLHerE.el
/tmp/emacs-async-comp-cus-edit-lLi8uj.el
/tmp/emacs-async-comp-cus-start-THOIFJ.el
/tmp/emacs-async-comp-delsel-Qt9mj1.el
/tmp/emacs-async-comp-dframe-3hG8KX.el
/tmp/emacs-async-comp-diary-lib-a5R3pl.el
/tmp/emacs-async-comp-dired-91E78r.el
/tmp/emacs-async-comp-dired-async-bbDW10.el
/tmp/emacs-async-comp-dired-aux-QNUzoL.el
/tmp/emacs-async-comp-doc-view-9gaxyb.el
/tmp/emacs-async-comp-dom-zYf21G.el
/tmp/emacs-async-comp-easy-repeat-M5fqHV.el
/tmp/emacs-async-comp-edmacro-M69ewc.el
/tmp/emacs-async-comp-eieio-xPFMh0.el
/tmp/emacs-async-comp-epa-jlpMNe.el
/tmp/emacs-async-comp-epg-cav6Av.el
/tmp/emacs-async-comp-epg-config-6MLDpu.el
/tmp/emacs-async-comp-ezimage-cePW4l.el
/tmp/emacs-async-comp-fileloop-w6xkto.el
/tmp/emacs-async-comp-filenotify-qrZxPs.el
/tmp/emacs-async-comp-files-x-wlf2T4.el
/tmp/emacs-async-comp-fix-word-k0pNwM.el
/tmp/emacs-async-comp-format-spec-IIwMhG.el
/tmp/emacs-async-comp-fw-SVCTHV.el
/tmp/emacs-async-comp-gcmh-ICqagW.el
/tmp/emacs-async-comp-gnus-alias-DSIgPD.el
/tmp/emacs-async-comp-helm-adaptive-bc0euJ.el
/tmp/emacs-async-comp-helm-buffers-c1OBZP.el
/tmp/emacs-async-comp-helm-core-1L6l6w.el
/tmp/emacs-async-comp-helm-descbinds-nBVJcV.el
/tmp/emacs-async-comp-helm-elisp-9O0NSM.el
/tmp/emacs-async-comp-helm-eshell-jMCK3z.el
/tmp/emacs-async-comp-helm-eval-5xmr5q.el
/tmp/emacs-async-comp-helm-files-ivUhiA.el
/tmp/emacs-async-comp-helm-grep-cKmbH7.el
/tmp/emacs-async-comp-helm-help-vUbDHI.el
/tmp/emacs-async-comp-helm-info-J7puEc.el
/tmp/emacs-async-comp-helm-lib-Z1mots.el
/tmp/emacs-async-comp-helm-locate-fTFYrR.el
/tmp/emacs-async-comp-helm-misc-BIuWXI.el
/tmp/emacs-async-comp-helm-mode-dVvqxf.el
/tmp/emacs-async-comp-helm-multi-match-iQguey.el
/tmp/emacs-async-comp-helm-occur-H54Zxt.el
/tmp/emacs-async-comp-helm-regexp-FpbDC4.el
/tmp/emacs-async-comp-helm-source-qY20or.el
/tmp/emacs-async-comp-helm-tags-vWu94v.el
/tmp/emacs-async-comp-helm-types-2bHjYR.el
/tmp/emacs-async-comp-helm-utils-GHnDEU.el
/tmp/emacs-async-comp-help-mode-yErcaX.el
/tmp/emacs-async-comp-hl-line-gBxrcO.el
/tmp/emacs-async-comp-holidays-zSdSGL.el
/tmp/emacs-async-comp-ibuf-ext-c7AUbB.el
/tmp/emacs-async-comp-ibuffer-PFWaNZ.el
/tmp/emacs-async-comp-icalendar-TTHuqR.el
/tmp/emacs-async-comp-image-mode-laEAEj.el
/tmp/emacs-async-comp-imenu-oKJvjT.el
/tmp/emacs-async-comp-info-nUAJJc.el
/tmp/emacs-async-comp-iso8601-EmlH4S.el
/tmp/emacs-async-comp-jka-compr-Ncejiv.el
/tmp/emacs-async-comp-json-Qm0QzY.el
/tmp/emacs-async-comp-keychain-environment-9bE1aC.el
/tmp/emacs-async-comp-key-chord-NfeT60.el
/tmp/emacs-async-comp-kmacro-74hYo4.el
/tmp/emacs-async-comp-lex-qSD2QJ.el
/tmp/emacs-async-comp-ls-lisp-n35tqw.el
/tmp/emacs-async-comp-minibuffer-line-k4Hcwv.el
/tmp/emacs-async-comp-mode-local-EXHmzg.el
/tmp/emacs-async-comp-notmuch-address-yI72IX.el
/tmp/emacs-async-comp-notmuch-company-wIfqpI.el
/tmp/emacs-async-comp-notmuch-compat-Yv03rR.el
/tmp/emacs-async-comp-notmuch-crypto-j48lnj.el
/tmp/emacs-async-comp-notmuch-draft-qoOWho.el
/tmp/emacs-async-comp-notmuch-hello-q93ftT.el
/tmp/emacs-async-comp-notmuch-jump-FrkZJd.el
/tmp/emacs-async-comp-notmuch-lib-iAd5zR.el
/tmp/emacs-async-comp-notmuch-maildir-fcc-3XMcqn.el
/tmp/emacs-async-comp-notmuch-message-CGYlRv.el
/tmp/emacs-async-comp-notmuch-Mk1MXH.el
/tmp/emacs-async-comp-notmuch-mua-OfK8KY.el
/tmp/emacs-async-comp-notmuch-parser-W3d0cV.el
/tmp/emacs-async-comp-notmuch-print-AKP3L3.el
/tmp/emacs-async-comp-notmuch-show-xhy5jM.el
/tmp/emacs-async-comp-notmuch-tag-ty2SaC.el
/tmp/emacs-async-comp-notmuch-tree-GIkIm4.el
/tmp/emacs-async-comp-notmuch-wash-3bNXz4.el
/tmp/emacs-async-comp-ob-comint-leu52R.el
/tmp/emacs-async-comp-ob-core-9IF6w2.el
/tmp/emacs-async-comp-ob-emacs-lisp-NHr2JM.el
/tmp/emacs-async-comp-ob-eval-kdn2CF.el
/tmp/emacs-async-comp-ob-exp-uRjsnv.el
/tmp/emacs-async-comp-ob-lob-JlPplD.el
/tmp/emacs-async-comp-ob-plantuml-iaKkPY.el
/tmp/emacs-async-comp-ob-ref-IDHirv.el
/tmp/emacs-async-comp-ob-table-NcWkqD.el
/tmp/emacs-async-comp-ob-tangle-xvWX3j.el
/tmp/emacs-async-comp-oc-gL0Q7V.el
/tmp/emacs-async-comp-ol-bbdb-tTsEgm.el
/tmp/emacs-async-comp-ol-docview-RNEmM8.el
/tmp/emacs-async-comp-ol-eshell-GB2uD4.el
/tmp/emacs-async-comp-ol-eww-u1naaW.el
/tmp/emacs-async-comp-ol-info-LlKsUV.el
/tmp/emacs-async-comp-ol-man-OyKHv2.el
/tmp/emacs-async-comp-ol-xHgouj.el
/tmp/emacs-async-comp-org-agenda-RnA3rh.el
/tmp/emacs-async-comp-org-capture-wQjIom.el
/tmp/emacs-async-comp-org-clock-kof9Tb.el
/tmp/emacs-async-comp-org-compat-A1486e.el
/tmp/emacs-async-comp-org-crypt-NnnZHp.el
/tmp/emacs-async-comp-org-ctags-KCGK3l.el
/tmp/emacs-async-comp-org-cycle-P3ekty.el
/tmp/emacs-async-comp-org-element-nxRVDJ.el
/tmp/emacs-async-comp-org-entities-5MCpLH.el
/tmp/emacs-async-comp-org-faces-3V9ztT.el
/tmp/emacs-async-comp-org-fold-core-kk4sHQ.el
/tmp/emacs-async-comp-org-fold-LL85YZ.el
/tmp/emacs-async-comp-org-footnote-5bxu5j.el
/tmp/emacs-async-comp-org-habit-bsn3xf.el
/tmp/emacs-async-comp-org-id-LCwa1b.el
/tmp/emacs-async-comp-org-inlinetask-BAJGxw.el
/tmp/emacs-async-comp-org-keys-vyNR55.el
/tmp/emacs-async-comp-org-list-EMyWLz.el
/tmp/emacs-async-comp-org-macro-yNuUMT.el
/tmp/emacs-async-comp-org-macs-9LcYMN.el
/tmp/emacs-async-comp-org-mouse-PVr4YJ.el
/tmp/emacs-async-comp-org-pcomplete-8LTxe1.el
/tmp/emacs-async-comp-org-persist-wTY1ec.el
/tmp/emacs-async-comp-org-protocol-CstUr4.el
/tmp/emacs-async-comp-org-refile-kn1Mqg.el
/tmp/emacs-async-comp-org-src-a1yMg3.el
/tmp/emacs-async-comp-org-table-ctto0o.el
/tmp/emacs-async-comp-org-tempo-RQxJZZ.el
/tmp/emacs-async-comp-org-vDvm4j.el
/tmp/emacs-async-comp-outline-XIF8wK.el
/tmp/emacs-async-comp-parse-time-o7UU9I.el
/tmp/emacs-async-comp-password-cache-hZWu1i.el
/tmp/emacs-async-comp-pcomplete-3HhI5i.el
/tmp/emacs-async-comp-pdf-cache-5aOEpV.el
/tmp/emacs-async-comp-pdf-info-DnL1so.el
/tmp/emacs-async-comp-pdf-isearch-ZLzJ0i.el
/tmp/emacs-async-comp-pdf-misc-RROccj.el
/tmp/emacs-async-comp-pdf-occur-FDc2Zt.el
/tmp/emacs-async-comp-pdf-tools-iz18hS.el
/tmp/emacs-async-comp-pdf-util-XdVNac.el
/tmp/emacs-async-comp-pdf-view-LLPvNl.el
/tmp/emacs-async-comp-rainbow-delimiters-q5cqNt.el
/tmp/emacs-async-comp-savehist-3lwnL5.el
/tmp/emacs-async-comp-saveplace-WAXkTV.el
/tmp/emacs-async-comp-semantic-t4FM26.el
/tmp/emacs-async-comp-server-JA6jnJ.el
/tmp/emacs-async-comp-shell-R4LOWu.el
/tmp/emacs-async-comp-speedbar-6xZINu.el
/tmp/emacs-async-comp-ssh-deploy-oPA4CT.el
/tmp/emacs-async-comp-svg-Ud8wup.el
/tmp/emacs-async-comp-tablist-filter-kVfrYJ.el
/tmp/emacs-async-comp-tablist-Wcu4Bo.el
/tmp/emacs-async-comp-tag-cdvAIV.el
/tmp/emacs-async-comp-tempo-d1jp5D.el
/tmp/emacs-async-comp-thingatpt-75vfGI.el
/tmp/emacs-async-comp-time-date-jmTl2i.el
/tmp/emacs-async-comp-time-stamp-HHkvFL.el
/tmp/emacs-async-comp-timezone-8snfx1.el
/tmp/emacs-async-comp-use-package-bind-key-IEkGne.el
/tmp/emacs-async-comp-use-package-core-NTc8be.el
/tmp/emacs-async-comp-use-package-delight-VkxJon.el
/tmp/emacs-async-comp-use-package-diminish-171jJ0.el
/tmp/emacs-async-comp-use-package-ensure-wghU4i.el
/tmp/emacs-async-comp-util-a74ohp.el
/tmp/emacs-async-comp-util-modes-dJtQaT.el
/tmp/emacs-async-comp-wcheck-mode-Xg92bL.el
/tmp/emacs-async-comp-which-key-MS9s2R.el
/tmp/emacs-async-comp-wid-edit-RDFM4w.el
/tmp/emacs-async-comp-windmove-S0EC8E.el
/tmp/emacs-async-comp-winner-ntjK5a.el
/tmp/emacs-async-comp-wisent-wY2J2F.el
/tmp/emacs-async-comp-ws-butler-Jie5Lx.el
/tmp/emacs-async-comp-xdg-qvyYQs.el
/tmp/emacs-async-comp-xml-lU4Geh.el
/tmp/emacs-async-comp-xt-mouse-GiY2sT.el
/tmp/emacs-async-comp-yank-media-QYt4Fg.el

>   . show the list of files that were compiled although they already
>     had corresponding *.eln files in that subdirectory

All the files from the list to your last question already
have a corresponding file in the third to last eln cache directory
(see J)).  E.g.

/tmp/emacs-async-comp-yank-media-QYt4Fg.el corresponds to
yank-media-62540c94-43b16516.eln

>   . show the list of files that were compiled although they already
>     had corresponding *.eln files in that subdirectory

see K) above

>   . let the compilations run to completion (use 'top' to see when
>     there are no more compilations running)
>   . look in the *Async-native-compile-log* buffer to see if there are
>     any warning or error messages there; if there are, post them


L)

There are no warnings or error messages, only this:

Compiling /home/grfz/src/emacs-master-next/lisp/wid-edit.el...
Compiling /home/grfz/src/emacs-master-next/lisp/emacs-lisp/cl-lib.el...
Compiling /home/grfz/src/emacs-master-next/lisp/cus-start.el...
Compiling /home/grfz/src/emacs-master-next/lisp/cus-edit.el...
Compiling /home/grfz/src/emacs-master-next/lisp/epg-config.el...
Compiling /home/grfz/src/emacs-master-next/lisp/epg.el...
Compiling /home/grfz/src/emacs-master-next/lisp/emacs-lisp/cconv.el...
Compiling /home/grfz/src/emacs-master-next/lisp/emacs-lisp/bytecomp.el...
Compiling /home/grfz/src/emacs-master-next/lisp/emacs-lisp/byte-opt.el...
Compiling /home/grfz/src/emacs-master-next/lisp/json.el...
Compiling /home/grfz/src/emacs-master-next/lisp/password-cache.el...
Compiling /home/grfz/src/emacs-master-next/lisp/emacs-lisp/eieio.el...
Compiling /home/grfz/src/emacs-master-next/lisp/auth-source.el...
Compiling /home/grfz/src/emacs-master-next/lisp/calendar/time-date.el...
Compiling /home/grfz/src/emacs-master-next/lisp/epa.el...
Compiling /home/grfz/src/emacs-master-next/lisp/help-mode.el...
Compiling /home/grfz/src/emacs-master-next/lisp/info.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/fix-word-20210319.1414/fix-word.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/use-package-20221012.1743/use-package-core.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/bind-key-20220910.2157/bind-key.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/use-package-20221012.1743/use-package-bind-key.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/use-package-20221012.1743/use-package-diminish.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/use-package-20221012.1743/use-package-delight.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/use-package-20221012.1743/use-package-ensure.el...
Compiling /home/grfz/src/emacs-master-next/lisp/delsel.el...
Compiling /home/grfz/src/emacs-master-next/lisp/dired.el...
Compiling /home/grfz/src/emacs-master-next/lisp/dired-aux.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/async-20221014.2225/async.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/async-20221014.2225/dired-async.el...
Compiling /home/grfz/src/emacs-master-next/lisp/xml.el...
Compiling /home/grfz/src/emacs-master-next/lisp/battery.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/minibuffer-line-0.1/minibuffer-line.el...
Compiling /home/grfz/src/emacs-master-next/lisp/avoid.el...
Compiling /home/grfz/src/emacs-master-next/lisp/savehist.el...
Compiling /home/grfz/src/emacs-master-next/lisp/format-spec.el...
Compiling /home/grfz/src/org-mode/lisp/org-macs.el...
Compiling /home/grfz/src/org-mode/lisp/ob-eval.el...
Compiling /home/grfz/src/org-mode/lisp/org-compat.el...
Compiling /home/grfz/src/org-mode/lisp/org-fold-core.el...
Compiling /home/grfz/src/org-mode/lisp/org-fold.el...
Compiling /home/grfz/src/org-mode/lisp/org-cycle.el...
Compiling /home/grfz/src/org-mode/lisp/ob-core.el...
Compiling /home/grfz/src/emacs-master-next/lisp/ansi-color.el...
Compiling /home/grfz/src/emacs-master-next/lisp/ansi-osc.el...
Compiling /home/grfz/src/emacs-master-next/lisp/comint.el...
Compiling /home/grfz/src/org-mode/lisp/ob-comint.el...
Compiling /home/grfz/src/org-mode/lisp/oc.el...
Compiling /home/grfz/src/org-mode/lisp/org-keys.el...
Compiling /home/grfz/src/org-mode/lisp/org-src.el...
Compiling /home/grfz/src/org-mode/lisp/ol.el...
Compiling /home/grfz/src/org-mode/lisp/ob-tangle.el...
Compiling /home/grfz/src/emacs-master-next/lisp/calendar/cal-menu.el...
Compiling /home/grfz/src/emacs-master-next/lisp/calendar/calendar.el...
Compiling /home/grfz/src/org-mode/lisp/org-table.el...
Compiling /home/grfz/src/org-mode/lisp/org.el...
Compiling /home/grfz/src/org-mode/lisp/ob-emacs-lisp.el...
Compiling /home/grfz/src/org-mode/lisp/ob-exp.el...
Compiling /home/grfz/src/org-mode/lisp/ob-table.el...
Compiling /home/grfz/src/org-mode/lisp/ob-lob.el...
Compiling /home/grfz/src/org-mode/lisp/ob-ref.el...
Compiling /home/grfz/src/org-mode/lisp/ob-plantuml.el...
Compiling /home/grfz/src/emacs-master-next/lisp/outline.el...
Compiling /home/grfz/src/org-mode/lisp/org-entities.el...
Compiling /home/grfz/src/org-mode/lisp/org-faces.el...
Compiling /home/grfz/src/org-mode/lisp/org-list.el...
Compiling /home/grfz/src/emacs-master-next/lisp/pcomplete.el...
Compiling /home/grfz/src/org-mode/lisp/org-pcomplete.el...
Compiling /home/grfz/src/org-mode/lisp/org-footnote.el...
Compiling /home/grfz/src/org-mode/lisp/org-macro.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/key-chord-20201222.2030/key-chord.el...
Compiling /home/grfz/src/emacs-master-next/lisp/emacs-lisp/comp-cstr.el...
Compiling /home/grfz/src/emacs-master-next/lisp/emacs-lisp/comp.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/gcmh-20201116.2251/gcmh.el...
Compiling /home/grfz/src/org-mode/lisp/ol-bbdb.el...
Compiling /home/grfz/src/org-mode/lisp/org-crypt.el...
Compiling /home/grfz/src/org-mode/lisp/org-ctags.el...
Compiling /home/grfz/src/emacs-master-next/lisp/image-mode.el...
Compiling /home/grfz/src/emacs-master-next/lisp/jka-compr.el...
Compiling /home/grfz/src/emacs-master-next/lisp/filenotify.el...
Compiling /home/grfz/src/emacs-master-next/lisp/doc-view.el...
Compiling /home/grfz/src/org-mode/lisp/ol-docview.el...
Compiling /home/grfz/src/emacs-master-next/lisp/yank-media.el...
Compiling /home/grfz/src/emacs-master-next/lisp/dom.el...
Compiling /home/grfz/src/emacs-master-next/lisp/svg.el...
Compiling /home/grfz/src/emacs-master-next/lisp/thingatpt.el...
Compiling /home/grfz/src/emacs-master-next/lisp/xdg.el...
Compiling /home/grfz/src/org-mode/lisp/ol-eww.el...
Compiling /home/grfz/src/org-mode/lisp/org-refile.el...
Compiling /home/grfz/src/org-mode/lisp/org-agenda.el...
Compiling /home/grfz/src/org-mode/lisp/org-habit.el...
Compiling /home/grfz/src/org-mode/lisp/org-id.el...
Compiling /home/grfz/src/org-mode/lisp/ol-info.el...
Compiling /home/grfz/src/org-mode/lisp/org-inlinetask.el...
Compiling /home/grfz/src/org-mode/lisp/org-mouse.el...
Compiling /home/grfz/src/org-mode/lisp/org-protocol.el...
Compiling /home/grfz/src/emacs-master-next/lisp/files-x.el...
Compiling /home/grfz/src/org-mode/lisp/ol-eshell.el...
Compiling /home/grfz/src/org-mode/lisp/ol-man.el...
Compiling /home/grfz/src/emacs-master-next/lisp/hl-line.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-compat.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-lib.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-tag.el...
Compiling /home/grfz/src/emacs-master-next/lisp/calendar/diary-lib.el...
Compiling /home/grfz/src/emacs-master-next/lisp/calendar/icalendar.el...
Compiling /home/grfz/src/notmuch/emacs/coolj.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-wash.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-company.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-parser.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-address.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-maildir-fcc.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-draft.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-message.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-mua.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-crypto.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-show.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-print.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-jump.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-hello.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch-tree.el...
Compiling /home/grfz/src/notmuch/emacs/notmuch.el...
Compiling /home/grfz/src/emacs-master-next/lisp/tempo.el...
Compiling /home/grfz/src/org-mode/lisp/org-tempo.el...
Compiling /home/grfz/src/org-mode/lisp/org-persist.el...
Compiling /home/grfz/src/org-mode/lisp/org-element.el...
Compiling /home/grfz/src/emacs-master-next/lisp/edmacro.el...
Compiling /home/grfz/src/emacs-master-next/lisp/kmacro.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/bbdb-20220706.433/bbdb.el...
Compiling /home/grfz/src/emacs-master-next/lisp/timezone.el...
Compiling /home/grfz/src/emacs-master-next/lisp/fileloop.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/gnus-alias-20150316.42/gnus-alias.el...
Compiling /home/grfz/src/emacs-master-next/lisp/imenu.el...
Compiling /home/grfz/src/emacs-master-next/lisp/windmove.el...
Compiling /home/grfz/src/emacs-master-next/lisp/xt-mouse.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/pdf-tools-20221007.1404/pdf-util.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/pdf-tools-20221007.1404/pdf-info.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/pdf-tools-20221007.1404/pdf-cache.el...
Compiling /home/grfz/src/emacs-master-next/lisp/bookmark.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/pdf-tools-20221007.1404/pdf-view.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/pdf-tools-20221007.1404/pdf-tools.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/pdf-tools-20221007.1404/pdf-misc.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/pdf-tools-20221007.1404/pdf-isearch.el...
Compiling /home/grfz/src/emacs-master-next/lisp/cedet/cedet.el...
Compiling /home/grfz/src/emacs-master-next/lisp/cedet/mode-local.el...
Compiling /home/grfz/src/emacs-master-next/lisp/cedet/semantic/fw.el...
Compiling /home/grfz/src/emacs-master-next/lisp/cedet/semantic/lex.el...
Compiling /home/grfz/src/emacs-master-next/lisp/cedet/semantic/tag.el...
Compiling /home/grfz/src/emacs-master-next/lisp/cedet/semantic.el...
Compiling /home/grfz/src/emacs-master-next/lisp/cedet/semantic/util.el...
Compiling /home/grfz/src/emacs-master-next/lisp/cedet/semantic/util-modes.el...
Compiling /home/grfz/src/emacs-master-next/lisp/cedet/semantic/wisent.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/tablist-20200427.2205/tablist-filter.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/tablist-20200427.2205/tablist.el...
Compiling /home/grfz/src/emacs-master-next/lisp/ibuffer.el...
Compiling /home/grfz/src/emacs-master-next/lisp/ibuf-ext.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/pdf-tools-20221007.1404/pdf-occur.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/keychain-environment-20180318.2223/keychain-environment.el...
Compiling /home/grfz/src/emacs-master-next/lisp/saveplace.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/wcheck-mode-2021/wcheck-mode.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/ws-butler-20201117.1528/ws-butler.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/ssh-deploy-20220126.658/ssh-deploy.el...
Compiling /home/grfz/src/org-mode/lisp/org-clock.el...
Compiling /home/grfz/src/emacs-master-next/lisp/dframe.el...
Compiling /home/grfz/src/emacs-master-next/lisp/ezimage.el...
Compiling /home/grfz/src/emacs-master-next/lisp/speedbar.el...
Compiling /home/grfz/src/emacs-master-next/lisp/calendar/holidays.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/calfw-20180118.45/calfw.el...
Compiling /home/grfz/src/org-mode/lisp/org-capture.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/calfw-org-20160303.258/calfw-org.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/calfw-cal-20170320.1206/calfw-cal.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/calfw-ical-20150703.819/calfw-ical.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/which-key-20220811.1616/which-key.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-core-20221020.1530/helm-lib.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-core-20221020.1530/helm-multi-match.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-core-20221020.1530/helm-source.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/async-20221014.2225/async-bytecomp.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-core-20221020.1530/helm-core.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-types.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-help.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-utils.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-regexp.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-grep.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-locate.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-tags.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-occur.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-buffers.el...
Compiling /home/grfz/src/emacs-master-next/lisp/ls-lisp.el...
Compiling /home/grfz/src/emacs-master-next/lisp/calendar/iso8601.el...
Compiling /home/grfz/src/emacs-master-next/lisp/calendar/parse-time.el...
Compiling /home/grfz/src/emacs-master-next/lisp/shell.el...
Compiling /home/grfz/src/emacs-master-next/lisp/time-stamp.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-files.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-misc.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-mode.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-adaptive.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-info.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-eval.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-elisp.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-eshell.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-descbinds-20190501.935/helm-descbinds.el...
Compiling /home/grfz/src/emacs-master-next/lisp/winner.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/rainbow-delimiters-20210515.1254/rainbow-delimiters.el...
Compiling /home/grfz/src/emacs-master-next/lisp/server.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/easy-repeat-20150516.848/easy-repeat.el...
Compilation finished.
Compiling /home/grfz/src/emacs-master-next/lisp/international/mule-util.el...
Compiling /home/grfz/src/emacs-master-next/lisp/ecomplete.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/orgalist-1.13/orgalist.el...
Compilation finished.
Compiling /home/grfz/src/emacs-master-next/lisp/misearch.el...
Compilation finished.
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-net.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-external.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-bookmark.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-x-files.el...
Compiling /home/grfz/.config/emacs/elpa-29.0/helm-20221020.1512/helm-for-files.el...
Compiling /home/grfz/src/emacs-master-next/lisp/tree-widget.el...
Compiling /home/grfz/src/emacs-master-next/lisp/recentf.el...
Compiling /home/grfz/src/emacs-master-next/lisp/ffap.el...
Compilation finished.


>   . kill Emacs
>   . look into that subdirectory of the eln-cache and see whether there
>     are any new *.eln files in it as result of the compilation

There is no file from today.


> One reason why Emacs could keep compiling files that already have
> *.eln files in the cache is because the existing *.eln files are
> outdated or incompatible with the Emacs binary.  This happens a lot if
> you are using the master branch, because we make significant changes
> there frequently, and they require recompilation.  You will in those
> cases see *.eln files whose base names are the same, but the hashes
> are different.

This is not the case here since I did not fetch from master
since days.




Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: Suppressing native compilation (short and long term)

[Gregor Zattler]

Hi Eli,
* Eli Zaretskii <eliz@gnu.org> [2022-10-23; 20:39 +03]:
>> From: Gregor Zattler <telegraph@gmx.net>
>> Cc: rlb@defaultvalue.org, monnier@iro.umontreal.ca, david@tethera.net,
>>  emacs-devel@gnu.org, akrl@sdf.org
>> Date: Sun, 23 Oct 2022 17:22:06 +0200
>>
>> This is GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu,
>> cairo version 1.16.0) of 2022-10-19
>
> Actually, there was a bug in native compilation that was fixed on that
> very day.  So perhaps update from Git ans see if the problem of
> repeated compilation goes away.


OK, I'll do so right now.  This will take some time,
though...

Thanks, gregor

Re: Suppressing native compilation (short and long term)

[Gregor Zattler]

Hi Eli,
* Gregor Zattler <telegraph@gmx.net> [2022-10-23; 20:55 +02]:
> * Eli Zaretskii <eliz@gnu.org> [2022-10-23; 20:39 +03]:
>>> From: Gregor Zattler <telegraph@gmx.net>
>>> Cc: rlb@defaultvalue.org, monnier@iro.umontreal.ca, david@tethera.net,
>>>  emacs-devel@gnu.org, akrl@sdf.org
>>> Date: Sun, 23 Oct 2022 17:22:06 +0200
>>>
>>> This is GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu,
>>> cairo version 1.16.0) of 2022-10-19
>>
>> Actually, there was a bug in native compilation that was fixed on that
>> very day.  So perhaps update from Git ans see if the problem of
>> repeated compilation goes away.
>
>
> OK, I'll do so right now.  This will take some time,
> though...

I did as you said, that did the trick, no spurious
compilations any more.  Thanks.

For the record the new version without annoying compilations is:

In GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu, cairo
 version 1.16.0) of 2022-10-24 built on no
Repository revision: f7816c94b61f87919afccbedbea5270ca5db4e15
Repository branch: master



Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Gregor Zattler <telegraph@gmx.net>
> Cc: emacs-devel@gnu.org
> Date: Mon, 24 Oct 2022 20:35:18 +0200
> 
> Hi Eli,
> * Gregor Zattler <telegraph@gmx.net> [2022-10-23; 20:55 +02]:
> > * Eli Zaretskii <eliz@gnu.org> [2022-10-23; 20:39 +03]:
> >>> From: Gregor Zattler <telegraph@gmx.net>
> >>> Cc: rlb@defaultvalue.org, monnier@iro.umontreal.ca, david@tethera.net,
> >>>  emacs-devel@gnu.org, akrl@sdf.org
> >>> Date: Sun, 23 Oct 2022 17:22:06 +0200
> >>>
> >>> This is GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu,
> >>> cairo version 1.16.0) of 2022-10-19
> >>
> >> Actually, there was a bug in native compilation that was fixed on that
> >> very day.  So perhaps update from Git ans see if the problem of
> >> repeated compilation goes away.
> >
> >
> > OK, I'll do so right now.  This will take some time,
> > though...
> 
> I did as you said, that did the trick, no spurious
> compilations any more.  Thanks.

OK, thanks for testing.

Re: Suppressing native compilation (short and long term)

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

Your idea about -Q inhibiting native comp was obviously not implemented.

In my setup, I call `startup-redirect-eln-cache' in init.el because I don't
want the cache in my .emacs.d/.

However, when I'm using -Q to test something, I do
M-x whatever-which-loads-a-module and end up getting an unwanted
.emacs.d/eln-cache/

So either a --no-native flag (setting
`inhibit-automatic-native-compilation',
and perhaps `load-no-native', to t) or having such behavior in -Q would be
quite helpful.

Yes, I can define an alias to use

--eval "(setq inhibit-automatic-native-compilation t)"

I already have. Still, I regularly forget it and default to emacs -Q.

So, FWIW, I agree with your sentiment that -Q has no single defining goal
other than "do not do unnecessary things, please", and inhibiting native
comp would fit quite well.

[-- Attachment #2: Type: text/html, Size: 1920 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Sat, 5 Nov 2022 02:09:55 +0100
> Cc: Andrea Corallo <akrl@sdf.org>, Eli Zaretskii <eliz@gnu.org>, rlb@defaultvalue.org, 
> 	monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org
> 
> In my setup, I call `startup-redirect-eln-cache' in init.el because I don't
> want the cache in my .emacs.d/.

Why do you need to do that?  What's the problem with having the
eln-cache under your ~/.emacs.d/?

> However, when I'm using -Q to test something, I do
> M-x whatever-which-loads-a-module and end up getting an unwanted
> .emacs.d/eln-cache/

If this testing is for the purposes of Emacs development, then my
suggestion is to have a separate build configured without native
compilation support.  That's what I do.

> So either a --no-native flag (setting `inhibit-automatic-native-compilation',
> and perhaps `load-no-native', to t) or having such behavior in -Q would be
> quite helpful.
> 
> Yes, I can define an alias to use
> 
> --eval "(setq inhibit-automatic-native-compilation t)"
> 
> I already have. Still, I regularly forget it and default to emacs -Q.

So it's just a matter of getting used to using the alias, and will be
solved with time.

> So, FWIW, I agree with your sentiment that -Q has no single defining goal
> other than "do not do unnecessary things, please", and inhibiting native
> comp would fit quite well.

I disagree.  We have too many knobs already, so adding a new one
without a good reason would be a mistake in the long run.

Re: Suppressing native compilation (short and long term)

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]


Eli Zaretskii <eliz@gnu.org> writes:

>> In my setup, I call `startup-redirect-eln-cache' in init.el because I don't
>> want the cache in my .emacs.d/.
>
> Why do you need to do that?  What's the problem with having the
> eln-cache under your ~/.emacs.d/?

My ~/.emacs.d/ is actually versiontracked as part of my repository,
because emacs is my build tool to compile org-mode to PDF and I need a
reproducible environment so others can use it, too.

During building, ~/.emacs.d/ is in the non-writable source directory
used by make distcheck.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
> Cc: Juanma Barranquero <lekktu@gmail.com>, larsi@gnus.org, akrl@sdf.org,
>  rlb@defaultvalue.org, monnier@iro.umontreal.ca, david@tethera.net,
>  emacs-devel@gnu.org
> Date: Sat, 05 Nov 2022 11:12:10 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> In my setup, I call `startup-redirect-eln-cache' in init.el because I don't
> >> want the cache in my .emacs.d/.
> >
> > Why do you need to do that?  What's the problem with having the
> > eln-cache under your ~/.emacs.d/?
> 
> My ~/.emacs.d/ is actually versiontracked as part of my repository,
> because emacs is my build tool to compile org-mode to PDF and I need a
> reproducible environment so others can use it, too.
> 
> During building, ~/.emacs.d/ is in the non-writable source directory
> used by make distcheck.

So what do you do with the *.elc files for the *.el files under
~/.emacs.d/?

And anyway, I think your use case is very different from Juanma's, and
is easily handled by customizing native-comp-eln-load-path.

Re: Suppressing native compilation (short and long term)

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

On Sat, Nov 5, 2022 at 7:45 AM Eli Zaretskii <eliz@gnu.org> wrote:

> Why do you need to do that?  What's the problem with having the
> eln-cache under your ~/.emacs.d/?

There's no problem. It's a matter of taste.

> If this testing is for the purposes of Emacs development, then my
> suggestion is to have a separate build configured without native
> compilation support.  That's what I do.

I prefer to use just one build.

> So it's just a matter of getting used to using the alias, and will be
> solved with time.

Yep.

> I disagree.  We have too many knobs already, so adding a new one
> without a good reason would be a mistake in the long run.

In fact, my thinking yesterday was "-Q should stop native
compilation...Wait, I bet
this was already discussed and rejected", and so I stumbled upon this
thread and
read or perused its several hundred messages. Believe me, I'm not
*proposing* any
change. I'm just telling Lars that I agree with him that this fits under -Q.

[-- Attachment #2: Type: text/html, Size: 1653 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Sat, 5 Nov 2022 12:53:42 +0100
> Cc: larsi@gnus.org, akrl@sdf.org, rlb@defaultvalue.org, 
> 	monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org
> 
> In fact, my thinking yesterday was "-Q should stop native compilation...Wait, I bet
> this was already discussed and rejected", and so I stumbled upon this thread and
> read or perused its several hundred messages. Believe me, I'm not *proposing* any
> change. I'm just telling Lars that I agree with him that this fits under -Q.

Well, we'll have to disagree.  The -Q switch is documented as
disabling various things that happen at startup, specifically loading
stuff that changes the defaults.  Native compilation is not in that
class, exactly like support for image files or GnuTLS aren't.  It is
part of the built Emacs, and is thus part of its default operation.  I
see no reason to change what -Q means, even though some people, for
reasons I cannot grasp, consider JIT native compilation to be
"unusual".

Suppose you start "emacs -Q" where some of the *.el files were already
compiled into the corresponding *.eln files, would you then expect
"emacs -Q" not to use those *.eln files, and instead to load the *.elc
files?  If yes, why?  If not, how does this differ from when you
invoke "emacs -Q" and the *.eln files do not yet exist, but are
produced when Emacs loads the corresponding package?

Re: Suppressing native compilation (short and long term)

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

On Sat, Nov 5, 2022 at 1:44 PM Eli Zaretskii <eliz@gnu.org> wrote:

> Well, we'll have to disagree.  The -Q switch is documented as
> disabling various things that happen at startup, specifically loading
> stuff that changes the defaults.

Yeah, well, there's --no-splash ;-)

> I see no reason to change what -Q means, even though some people, for
> reasons I cannot grasp, consider JIT native compilation to be
> "unusual".

I don't consider it unusual, except that in my build, if I enter Emacs,
load something
that triggers native compilation, and exit quickly, the subprocesses crash
(I get
invitations to "connect with gdb and debug them", which disappear after a
few seconds).
That had me puzzled for a while.

> Suppose you start "emacs -Q" where some of the *.el files were already
> compiled into the corresponding *.eln files, would you then expect
> "emacs -Q" not to use those *.eln files, and instead to load the *.elc
> files?  If yes, why?  If not, how does this differ from when you
> invoke "emacs -Q" and the *.eln files do not yet exist, but are
> produced when Emacs loads the corresponding package?

Loading them and using them wouldn't be a problem, because it does *not*
write anywhere,
while producing them does. In my case, they do just where I don't want them
to be.

As you say, we'll have to agree to disagree. I admit the issue is nuanced
and there's no
single solution that will please everyone.

[-- Attachment #2: Type: text/html, Size: 2363 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Sat, 5 Nov 2022 14:12:03 +0100
> Cc: larsi@gnus.org, akrl@sdf.org, rlb@defaultvalue.org, 
> 	monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org
> 
> > I see no reason to change what -Q means, even though some people, for
> > reasons I cannot grasp, consider JIT native compilation to be
> > "unusual".
> 
> I don't consider it unusual, except that in my build, if I enter Emacs, load something
> that triggers native compilation, and exit quickly, the subprocesses crash (I get
> invitations to "connect with gdb and debug them", which disappear after a few seconds).

They aren't supposed to crash, and they didn't in Emacs 28.  So this
should be reported with enough detail for us to be able to fix the
crashes.  This could be related to bug#58956, btw.

> > Suppose you start "emacs -Q" where some of the *.el files were already
> > compiled into the corresponding *.eln files, would you then expect
> > "emacs -Q" not to use those *.eln files, and instead to load the *.elc
> > files?  If yes, why?  If not, how does this differ from when you
> > invoke "emacs -Q" and the *.eln files do not yet exist, but are
> > produced when Emacs loads the corresponding package?
> 
> Loading them and using them wouldn't be a problem, because it does *not* write anywhere,
> while producing them does. In my case, they do just where I don't want them to be.

But we have features who write even under "emacs -Q", don't we?

Re: Suppressing native compilation (short and long term)

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]

On Sat, Nov 5, 2022 at 2:35 PM Eli Zaretskii <eliz@gnu.org> wrote:

> They aren't supposed to crash, and they didn't in Emacs 28.  So this
> should be reported with enough detail for us to be able to fix the
> crashes.  This could be related to bug#58956, btw.

Yes. I'm waiting to have a more easily reproducible case or a better
understanding
of how to trigger it. I'll look into #58956.

> But we have features who write even under "emacs -Q", don't we?

Yes, sure. But if I do "emacs -Q" and then I save a bookmark, I know
bookmark.el
is going to save a file and I will make sure it is going to .emacs.d/ or
wherever
else I want it to go. That's not surprising. But I don't usually expect to
do
M-x bs-show and get something written to disk, in a place I wasn't
expecting.

[-- Attachment #2: Type: text/html, Size: 1313 bytes --]

Re: Suppressing native compilation (short and long term)

[Max Brieiev]

"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

> My ~/.emacs.d/ is actually versiontracked as part of my repository,
> because emacs is my build tool to compile org-mode to PDF and I need a
> reproducible environment so others can use it, too.

If Emacs is your build tool, then you probably run it in a batch mode.

I think in one of his messages Andrea said that native compiler is
disabled when Emacs is run in batch mode.

Re: Suppressing native compilation (short and long term)

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]


Max Brieiev <max.brieiev@gmail.com> writes:

> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>
>> My ~/.emacs.d/ is actually versiontracked as part of my repository,
>> because emacs is my build tool to compile org-mode to PDF and I need a
>> reproducible environment so others can use it, too.
>
> If Emacs is your build tool, then you probably run it in a batch mode.

No, I cannot run it in batch mode, because batch mode has different
font-locking than visual mode, so html-export is worse.

I need a full X11 Emacs (kept working via Xvfb, if there is no X11).

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> No, I cannot run it in batch mode, because batch mode has different
> font-locking than visual mode, so html-export is worse.

Please report this as a bug.


        Stefan

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Max Brieiev <max.brieiev@gmail.com> writes:

> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>
>> My ~/.emacs.d/ is actually versiontracked as part of my repository,
>> because emacs is my build tool to compile org-mode to PDF and I need a
>> reproducible environment so others can use it, too.
>
> If Emacs is your build tool, then you probably run it in a batch mode.
>
> I think in one of his messages Andrea said that native compiler is
> disabled when Emacs is run in batch mode.

Yep that's correct.

BR

  Andrea

Re: Suppressing native compilation (short and long term)

[Björn Bidar]

Andrea Corallo <akrl@sdf.org> writes:

> Max Brieiev <max.brieiev@gmail.com> writes:
>
>> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>>
>>> My ~/.emacs.d/ is actually versiontracked as part of my repository,
>>> because emacs is my build tool to compile org-mode to PDF and I need a
>>> reproducible environment so others can use it, too.
>>
>> If Emacs is your build tool, then you probably run it in a batch mode.
>>
>> I think in one of his messages Andrea said that native compiler is
>> disabled when Emacs is run in batch mode.
>
> Yep that's correct.

But I think it possible to call `native-compile` in batch mode I think.
I do this with borg to precompile all packages when I update them.

I think doing so is quite useful e.g. for device that not always run on
cable but on battery or are otherwise limited in power.

Br,

Björn

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Max Brieiev <max.brieiev@gmail.com> writes:
>>
>>> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>>>
>>>> My ~/.emacs.d/ is actually versiontracked as part of my repository,
>>>> because emacs is my build tool to compile org-mode to PDF and I need a
>>>> reproducible environment so others can use it, too.
>>>
>>> If Emacs is your build tool, then you probably run it in a batch mode.
>>>
>>> I think in one of his messages Andrea said that native compiler is
>>> disabled when Emacs is run in batch mode.
>>
>> Yep that's correct.
>
> But I think it possible to call `native-compile` in batch mode I think.
> I do this with borg to precompile all packages when I update them.

Yes that's correct, only the machanism that automatically triggers async
native compilations is disabled but you can still invoke compilations
manually.

  Andrea

Parallel native compilation

[Björn Bidar]

Andrea Corallo <akrl@sdf.org> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Andrea Corallo <akrl@sdf.org> writes:
>>
>>> Max Brieiev <max.brieiev@gmail.com> writes:
>>>
>>>> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>>>>
>>>>> My ~/.emacs.d/ is actually versiontracked as part of my repository,
>>>>> because emacs is my build tool to compile org-mode to PDF and I need a
>>>>> reproducible environment so others can use it, too.
>>>>
>>>> If Emacs is your build tool, then you probably run it in a batch mode.
>>>>
>>>> I think in one of his messages Andrea said that native compiler is
>>>> disabled when Emacs is run in batch mode.
>>>
>>> Yep that's correct.
>>
>> But I think it possible to call `native-compile` in batch mode I think.
>> I do this with borg to precompile all packages when I update them.
>
> Yes that's correct, only the machanism that automatically triggers async
> native compilations is disabled but you can still invoke compilations
> manually.

What is the suggested what of doing so in parallel?
While using borg I've noticed that doing so would be quite handy.
It would also help to control the precompile process better when
building emacs so that the build process can be amplified by increasing
the parallel make jobs.

Br,

Björn

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>,  Eli Zaretskii
>  <eliz@gnu.org>,  Juanma Barranquero <lekktu@gmail.com>,  larsi@gnus.org,
>   rlb@defaultvalue.org,  monnier@iro.umontreal.ca,  david@tethera.net,
>   emacs-devel@gnu.org
> Date: Tue, 08 Nov 2022 07:51:52 +0200
> 
> Andrea Corallo <akrl@sdf.org> writes:
> 
> >> I think in one of his messages Andrea said that native compiler is
> >> disabled when Emacs is run in batch mode.
> >
> > Yep that's correct.
> 
> But I think it possible to call `native-compile` in batch mode I think.

Yes, it is.  We actually do that when building a release tarball on
the end-user machine.

Re: Parallel native compilation

[Andrea Corallo]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>
>>> Andrea Corallo <akrl@sdf.org> writes:
>>>
>>>> Max Brieiev <max.brieiev@gmail.com> writes:
>>>>
>>>>> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>>>>>
>>>>>> My ~/.emacs.d/ is actually versiontracked as part of my repository,
>>>>>> because emacs is my build tool to compile org-mode to PDF and I need a
>>>>>> reproducible environment so others can use it, too.
>>>>>
>>>>> If Emacs is your build tool, then you probably run it in a batch mode.
>>>>>
>>>>> I think in one of his messages Andrea said that native compiler is
>>>>> disabled when Emacs is run in batch mode.
>>>>
>>>> Yep that's correct.
>>>
>>> But I think it possible to call `native-compile` in batch mode I think.
>>> I do this with borg to precompile all packages when I update them.
>>
>> Yes that's correct, only the machanism that automatically triggers async
>> native compilations is disabled but you can still invoke compilations
>> manually.
>
> What is the suggested what of doing so in parallel?

You can do it from outside if you use N instanceses of Emacs leveraging
`batch-native-compile'.

Another option would be to run one Emacs leveraging
`native-compile-async', but you'll have to add some simple custom code
to have Emacs wainting for all compilation to be finished before
exiting.  `native-comp-async-all-done-hook' might be of hand there.

  Andrea