Re: Suppressing native compilation (short and long term)

Re: Suppressing native compilation (short and long term)

[David Bremner]

> The reason stated by Rob was that they want to prevent "writing to
> user's HOME directory".  I don't think I understand the rationale well
> enough, and I asked some questions about it.  I hope Rob will answer,
> and then we could continue discussing what is reasonable for that use
> case.

For package builds, Debian has the following policy 

    https://www.debian.org/doc/debian-policy/ch-source.html#main-building-script-debian-rules

in particular

    Required targets must not attempt to write outside of the unpacked
    source package tree. There are two exceptions. Firstly, the binary
    targets may write the binary packages to the parent directory of the
    unpacked source package tree. Secondly, required targets may write
    to /tmp, /var/tmp and to the directory specified by the TMPDIR
    environment variable, but must not depend on the contents of any of
    these.

    This restriction is intended to prevent source package builds
    creating and depending on state outside of themselves, thus
    affecting multiple independent rebuilds. In particular, the required
    targets must not attempt to write into HOME.

Some additional byte compilation happens at package install time. In my
view the same restrictions should apply there. In addition to the
unintentional sharing of state mentioned in policy, there is also the
issue of cleaning up after a package on uninstall. This is manageable
now because any generated .elc files go in a package specific directory,
which can just be removed on purging the package.

I think just turning off native compilation when HOME is not writeable
would be sensible from a Debian point of view. Emacs did not previously
require a writable HOME (although maybe most users never tested
this). It would be unfortunate if this changed for Emacs with native
compilation support.

David

PS. Please CC me on any replies (that you want me to read). I'm not
subscribed to the list.

PPS. This reply is synthesized from "reply via email" on the archive
     page. Apologies in advance for any list etiquette failures.
     

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: David Bremner <david@tethera.net>
> Cc: emacs-devel@gnu.org, akrl@sdf.org, rlb@defaultvalue.org
> Date: Fri, 30 Sep 2022 10:13:56 -0300
> 
> 
> > The reason stated by Rob was that they want to prevent "writing to
> > user's HOME directory".  I don't think I understand the rationale well
> > enough, and I asked some questions about it.  I hope Rob will answer,
> > and then we could continue discussing what is reasonable for that use
> > case.
> 
> For package builds, Debian has the following policy 
> 
>     https://www.debian.org/doc/debian-policy/ch-source.html#main-building-script-debian-rules
> 
> in particular
> 
>     Required targets must not attempt to write outside of the unpacked
>     source package tree. There are two exceptions. Firstly, the binary
>     targets may write the binary packages to the parent directory of the
>     unpacked source package tree. Secondly, required targets may write
>     to /tmp, /var/tmp and to the directory specified by the TMPDIR
>     environment variable, but must not depend on the contents of any of
>     these.
> 
>     This restriction is intended to prevent source package builds
>     creating and depending on state outside of themselves, thus
>     affecting multiple independent rebuilds. In particular, the required
>     targets must not attempt to write into HOME.
> 
> Some additional byte compilation happens at package install time.

This is what I'm asking about: what exactly triggers the compilation?
Just installing a package shouldn't do that, only loading it into
Emacs should.

The other part of what I asked is that if for some reason installation
does need to compile files (something that I still need to understand
why it happens), there's nothing that hard-codes HOME in the directory
list used for that.  You can set native-comp-eln-load-path to anything
you want, and only the directories in that list will be used.

> In my
> view the same restrictions should apply there. In addition to the
> unintentional sharing of state mentioned in policy, there is also the
> issue of cleaning up after a package on uninstall. This is manageable
> now because any generated .elc files go in a package specific directory,
> which can just be removed on purging the package.

See above: you can do the same with *.eln files, if, for some reason,
they are produced by the installation.

> I think just turning off native compilation when HOME is not writeable
> would be sensible from a Debian point of view. Emacs did not previously
> require a writable HOME (although maybe most users never tested
> this). It would be unfortunate if this changed for Emacs with native
> compilation support.

Emacs doesn't require a writable HOME, it requires a writable
directory to store *.eln files.  It doesn't have to be HOME.  And once
again, I still don't understand why *.eln files are produced at
installation time in the first place.

Re: Suppressing native compilation (short and long term)

[David Bremner]

Eli Zaretskii <eliz@gnu.org> writes:

>
> This is what I'm asking about: what exactly triggers the compilation?
> Just installing a package shouldn't do that, only loading it into
> Emacs should.

When I talk about "package installation" I mean installing a Debian
binary package. This is a more general notion than a package as defined
by package.el

We have one copy of the .elc files for all users. Because of this, and
the cleanup issue I mentioned above, the elc files need to generated
either at (Debian) package build time, or at (Debian) package
installation time.

> The other part of what I asked is that if for some reason installation
> does need to compile files (something that I still need to understand
> why it happens), there's nothing that hard-codes HOME in the directory
> list used for that.  You can set native-comp-eln-load-path to anything
> you want, and only the directories in that list will be used.

Does that restrict where eln files are written? Or just where emacs
tries to read?

> Emacs doesn't require a writable HOME, it requires a writable
> directory to store *.eln files.  It doesn't have to be HOME.

Fair enough. I tried setting native-compile-target-directory, but that
seemed to be ignored by the trampoline stuff. Maybe both variables need
to be set.

> And once again, I still don't understand why *.eln files are produced
> at installation time in the first place.

The short version is that we need to run emacs at Debian package build
and install time. Byte-compilation is one reason. Running tests at build
time is another.

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Fri 30 Sep 2022 at 04:56PM +03, Eli Zaretskii wrote:

> This is what I'm asking about: what exactly triggers the compilation?
> Just installing a package shouldn't do that, only loading it into
> Emacs should.

We currently bytecompile during package installation, and would like
also to do the native compilation at that time.

The main reason for bytecompiling at package installation is that we
have logic to purge and recompile anything whose dependency versions
have changed, which means that all bytecode installed on Debian systems
was generated using known-correct macro definitions.  Historically
ordinary Emacs processes running in non-batch mode have not guaranteed
this; I recall that the Org-mode developers discouraged use of ELPA to
install newer versions of Org because of these sorts of issues.

A secondary reason is that it makes sense to compile once, system-wide,
rather than repeatedly in each user's home directory.  It is also nice
that you can know everything is already in place once your package is
installed, so that I can unplug my laptop once the package manager has
exited, and I know that it isn't going to do any CPU-intensive
compilation and drain my battery.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

>> Some additional byte compilation happens at package install time.

[ BTW, In Rob's case, we're talking about installation via Debian's
  `dpkg`, i.e. system-wide installation of ELisp packages, which also
  causes the .el files to be byte-compiled for&by the currently installed
  Emacs binary.  ]

> This is what I'm asking about: what exactly triggers the compilation?
> Just installing a package shouldn't do that, only loading it into
> Emacs should.

Byte-compilation does load files (not the one being compiled, but many
others, starting with `bytecomp` of course), so it can trigger
native-compilation of some files (including some of the files being
byte-compiled, if they `require` each other, which is very common).


        Stefan

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

David Bremner [2022-09-30 10:13:56] wrote:
> I think just turning off native compilation when HOME is not writeable
> would be sensible from a Debian point of view. Emacs did not previously

That's not the problem: the problem is when $HOME is writable but doesn't
belong to the user (e.g. because the user is root): then Emacs ends up
writing files (and creating directories) which then end up being
"unwritable" by the owner of $HOME.

IIUC this happens very "naturally" with `su` followed by running Emacs,
because many version of `su` (contrary to `su -`) preserve the $LOGNAME
and Emacs uses ~$LOGNAME as "the HOME" (instead of $HOME) when run
as root.


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: David Bremner <david@tethera.net>
> Cc: emacs-devel@gnu.org, akrl@sdf.org, rlb@defaultvalue.org
> Date: Fri, 30 Sep 2022 11:33:29 -0300
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >
> > This is what I'm asking about: what exactly triggers the compilation?
> > Just installing a package shouldn't do that, only loading it into
> > Emacs should.
> 
> When I talk about "package installation" I mean installing a Debian
> binary package. This is a more general notion than a package as defined
> by package.el
> 
> We have one copy of the .elc files for all users. Because of this, and
> the cleanup issue I mentioned above, the elc files need to generated
> either at (Debian) package build time, or at (Debian) package
> installation time.

So I understand that this issue is caused by the Debian installation
process?  If so, the installation process should make sure the *.eln
files are written only where you want them to be written.

> > The other part of what I asked is that if for some reason installation
> > does need to compile files (something that I still need to understand
> > why it happens), there's nothing that hard-codes HOME in the directory
> > list used for that.  You can set native-comp-eln-load-path to anything
> > you want, and only the directories in that list will be used.
> 
> Does that restrict where eln files are written? Or just where emacs
> tries to read?

Both.

> > Emacs doesn't require a writable HOME, it requires a writable
> > directory to store *.eln files.  It doesn't have to be HOME.
> 
> Fair enough. I tried setting native-compile-target-directory, but that
> seemed to be ignored by the trampoline stuff. Maybe both variables need
> to be set.

That's the wrong variable.  Please use native-comp-eln-load-path.

> > And once again, I still don't understand why *.eln files are produced
> > at installation time in the first place.
> 
> The short version is that we need to run emacs at Debian package build
> and install time. Byte-compilation is one reason. Running tests at build
> time is another.

OK, in that case you get to arrange for the environment which will not
produce files where you don't want them.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: David Bremner <david@tethera.net>,  emacs-devel@gnu.org,  akrl@sdf.org,
>   rlb@defaultvalue.org
> Date: Fri, 30 Sep 2022 07:55:08 -0700
> 
> We currently bytecompile during package installation, and would like
> also to do the native compilation at that time.
> 
> The main reason for bytecompiling at package installation is that we
> have logic to purge and recompile anything whose dependency versions
> have changed, which means that all bytecode installed on Debian systems
> was generated using known-correct macro definitions.  Historically
> ordinary Emacs processes running in non-batch mode have not guaranteed
> this; I recall that the Org-mode developers discouraged use of ELPA to
> install newer versions of Org because of these sorts of issues.

This problem doesn't exist with *.eln files, because Emacs will
regenerate new .eln files with different hashes in its name when the
source file or the Emacs binary changes.  So you do not have to
trigger native compilation for these reasons.

> A secondary reason is that it makes sense to compile once, system-wide,
> rather than repeatedly in each user's home directory.

I think this is misguided.  If the user never loads a package, there's
no reason to produce the *.eln files for it.  At best it litters the
disk with useless files.  At worst, compilation could cause warnings
or even errors, which are just an annoyance when the package is not
going to be used for a long time, or not at all.

These considerations are the reason why *.eln files are compiled JIT
and on-demand.  It is not a coincidence or lack of thought on our
part, it is the result of considering many factors.

I think you are trying to deal with *.eln files and with
native-compilation in general as if it were a slightly different kind
of byte-compilation.  That is a mistake: it's a qualitatively
different feature with different traits and different user-facing
and administrator-facing aspects.

> It is also nice that you can know everything is already in place
> once your package is installed, so that I can unplug my laptop once
> the package manager has exited, and I know that it isn't going to do
> any CPU-intensive compilation and drain my battery.

The *.eln files should not be part of "everything is in place",
because they will be produced when needed.  No user involvement or
interaction is necessary.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: David Bremner <david@tethera.net>,  emacs-devel@gnu.org,  akrl@sdf.org,
>   rlb@defaultvalue.org
> Date: Fri, 30 Sep 2022 11:32:57 -0400
> 
> Byte-compilation does load files (not the one being compiled, but many
> others, starting with `bytecomp` of course), so it can trigger
> native-compilation of some files (including some of the files being
> byte-compiled, if they `require` each other, which is very common).

The solution for this that I'd suggest is to precompile them, so that
the package comes with those files already as *.eln.  That's what we
did in Emacs 28.1 for -nw sessions, which always load the terminal
support file from lisp/term/.

Re: Suppressing native compilation (short and long term)

[David Bremner]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> David Bremner [2022-09-30 10:13:56] wrote:
>> I think just turning off native compilation when HOME is not writeable
>> would be sensible from a Debian point of view. Emacs did not previously
>
> That's not the problem: the problem is when $HOME is writable but doesn't
> belong to the user (e.g. because the user is root): then Emacs ends up
> writing files (and creating directories) which then end up being
> "unwritable" by the owner of $HOME.

That is _also_ a problem, but not the one causing many Debian packages
to fail to build from source.

Re: Suppressing native compilation (short and long term)

[David Bremner]

David Bremner <david@tethera.net> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> David Bremner [2022-09-30 10:13:56] wrote:
>>> I think just turning off native compilation when HOME is not writeable
>>> would be sensible from a Debian point of view. Emacs did not previously
>>
>> That's not the problem: the problem is when $HOME is writable but doesn't
>> belong to the user (e.g. because the user is root): then Emacs ends up
>> writing files (and creating directories) which then end up being
>> "unwritable" by the owner of $HOME.
>
> That is _also_ a problem, but not the one causing many Debian packages
> to fail to build from source.

Sorry, that came off snappier than intended. You are of course correct
that the issue you report is a real problem for users, I've just been up
to my elbows for the last week or so dealing with the fallout of the
problem I described.

d

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> The solution for this that I'd suggest is to precompile them, so that
> the package comes with those files already as *.eln.  That's what we
> did in Emacs 28.1 for -nw sessions, which always load the terminal
> support file from lisp/term/.

I've finally had a chance to catch up, and this thread to date appears
to cover a good bit of the relevant ground.  As requested, I'll try to
add some additional context that hopefully better explains how and why
we arrived at the current situation, asking the questions we're asking.

Long ago, we (speaking from the Debian side) came up with the Debian
Emacs policy.  This was both a policy, and the infrastructure to
implement some of it (in the form of the emacsen-common package, and
now, additionally, the dh-elpa infrastructure).

At the time, XEmacs was still active, and Emacs itself was about to
transition from 19 to 20, and my recollection is that there was a
nontrivial amount of backward incompatibility with the latter, or at
least I recall hearing from people who really didn't want to be forced
to upgrade immediately.

I'm not completely familiar with those concerns because this was just
about the time I started working on Debian's Emacs support.  In any
case, one of the key goals of that support was to allow Debian to
package, and people to install, one or more major versions of Emacs,
and/or XEmacs simultaneously.  As a result, we added support for
"emacsen flavors": emacs20, emacs21, emacs22, xemacs19, etc.  (Worth
noting that a few years back we did "unversion" the Emacs packages, so
that we no longer have emacs27, emacs28, etc., just emacs.)

We also wanted to support the creation of "add-on" packages like org,
newer versions of gnus, etc., and we wanted each of those packages to be
able to work with as many flavors (Emacs or XEmacs) as the add-on
package maintainer decided was feasible.

We wanted to have only one system-level .elc file for each flavor for
each .el file.  We did not want to have to build and maintain separate
packages for each flavor in the archive, i.e. org-emacs21, org-emacs22,
etc., each with a full set of the relevant .elc files.  (I'm also not
sure that would have worked (easily), given the versioning and
dependency concerns described next.)

We didn't want to risk breakage from backward-incompatible changes to
macros in dependencies, i.e. if the magit package depended on the foo
(library) package, we assumed we'd need to rebuild (during package
upgrade) all the majit .elc files whenever the foo package changed
because foo might have changed a macro backward incompatibly.

Since we didn't have any (reasonable?) way to detect the inter-package
dependency graph, we just insisted that those be correctly described by
the Debian package dependencies.  Then at install time, we use that
graph to determine what packages to "rebuild" (mostly just the .elc
files), and the order in which to rebuild them.

We also decided to rebuild *all* the add-on packages (.elc files) for a
given flavor whenever the version of the flavor's package changes
(e.g. when there's a new Emacs or XEmacs version) because we didn't know
whether or not any given release might make that necessary/desirable.

Uninstalling an add-on, of course, cleans up all the relevant bits
(usually, mostly just the automatically generated .elc files).

So when you "apt install elpa-magit", Debian builds all the .elc files
and puts them in the right place in /usr for every flavor you have
installed.  When you "apt install emacs" and it's a new install, Debian
builds all of the .elc files for every installed add-on package in
dependency order for that flavor.  The same thing happens during an
emacs package upgrade.

Note that we were also thinking of the case where you have a large
machine with hundreds of users (as was more often the case when we
started).  There we didn't want to have N copies of the same file for N
users (whether .elc or now .eln).

Regarding the writes to HOME, etc.  I think David covered that fairly
well -- Debian runs emacs to "do things" (build .elcs, run tests, etc.)
at package build time, package install time, and during package testing.
In all of these cases, we're likely to want to avoid side-effects outside
the build/test dir like writing .elc or .eln files to the current user's
HOME, whether that's /root/ or /home/*.  It sounds like we may be able
to accomplish that by redirecting everything to a temp dir, which is
likely fine.

Hope this helps.
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: david@tethera.net, emacs-devel@gnu.org, akrl@sdf.org
> Date: Sat, 01 Oct 2022 11:38:54 -0500
> 
> I've finally had a chance to catch up, and this thread to date appears
> to cover a good bit of the relevant ground.  As requested, I'll try to
> add some additional context that hopefully better explains how and why
> we arrived at the current situation, asking the questions we're asking.

Thanks.

> Long ago, we (speaking from the Debian side) came up with the Debian
> Emacs policy.  This was both a policy, and the infrastructure to
> implement some of it (in the form of the emacsen-common package, and
> now, additionally, the dh-elpa infrastructure).
> 
> At the time, XEmacs was still active, and Emacs itself was about to
> transition from 19 to 20, and my recollection is that there was a
> nontrivial amount of backward incompatibility with the latter, or at
> least I recall hearing from people who really didn't want to be forced
> to upgrade immediately.
> 
> I'm not completely familiar with those concerns because this was just
> about the time I started working on Debian's Emacs support.  In any
> case, one of the key goals of that support was to allow Debian to
> package, and people to install, one or more major versions of Emacs,
> and/or XEmacs simultaneously.  As a result, we added support for
> "emacsen flavors": emacs20, emacs21, emacs22, xemacs19, etc.  (Worth
> noting that a few years back we did "unversion" the Emacs packages, so
> that we no longer have emacs27, emacs28, etc., just emacs.)
> 
> We also wanted to support the creation of "add-on" packages like org,
> newer versions of gnus, etc., and we wanted each of those packages to be
> able to work with as many flavors (Emacs or XEmacs) as the add-on
> package maintainer decided was feasible.
> 
> We wanted to have only one system-level .elc file for each flavor for
> each .el file.  We did not want to have to build and maintain separate
> packages for each flavor in the archive, i.e. org-emacs21, org-emacs22,
> etc., each with a full set of the relevant .elc files.  (I'm also not
> sure that would have worked (easily), given the versioning and
> dependency concerns described next.)
> 
> We didn't want to risk breakage from backward-incompatible changes to
> macros in dependencies, i.e. if the magit package depended on the foo
> (library) package, we assumed we'd need to rebuild (during package
> upgrade) all the majit .elc files whenever the foo package changed
> because foo might have changed a macro backward incompatibly.
> 
> Since we didn't have any (reasonable?) way to detect the inter-package
> dependency graph, we just insisted that those be correctly described by
> the Debian package dependencies.  Then at install time, we use that
> graph to determine what packages to "rebuild" (mostly just the .elc
> files), and the order in which to rebuild them.
> 
> We also decided to rebuild *all* the add-on packages (.elc files) for a
> given flavor whenever the version of the flavor's package changes
> (e.g. when there's a new Emacs or XEmacs version) because we didn't know
> whether or not any given release might make that necessary/desirable.
> 
> Uninstalling an add-on, of course, cleans up all the relevant bits
> (usually, mostly just the automatically generated .elc files).
> 
> So when you "apt install elpa-magit", Debian builds all the .elc files
> and puts them in the right place in /usr for every flavor you have
> installed.  When you "apt install emacs" and it's a new install, Debian
> builds all of the .elc files for every installed add-on package in
> dependency order for that flavor.  The same thing happens during an
> emacs package upgrade.
> 
> Note that we were also thinking of the case where you have a large
> machine with hundreds of users (as was more often the case when we
> started).  There we didn't want to have N copies of the same file for N
> users (whether .elc or now .eln).

Thanks for the background.

You should know that the problems you needed to address with the *.elc
files don't exist with the *.eln files.  The *.eln files have version
information of the Emacs to which they "belong" hard-coded into their
names.  That's why window.el gets compiled into something like
window-0d1b8b93-7ef4271a.eln, and lives under a directory named
something like 28.2-4fafb146 -- these 3 hashes encode both the Emacs
binary and its version, and the original .el file's absolute file name
and contents.  So there's no way users will have any trouble when
multiple Emacs versions are installed or when different users use
different versions: the *.eln files are effectively automatically
"versioned", and Emacs will only load a .eln file that was compiled
for it.  Moreover, as soon as some user decides to modify a .el file,
the .eln file produced from it will have a different name, and if the
Emacs binary changes as result of rebuilding, the new .eln file will
be written to a different directory.

> Regarding the writes to HOME, etc.  I think David covered that fairly
> well -- Debian runs emacs to "do things" (build .elcs, run tests, etc.)
> at package build time, package install time, and during package testing.
> In all of these cases, we're likely to want to avoid side-effects outside
> the build/test dir like writing .elc or .eln files to the current user's
> HOME, whether that's /root/ or /home/*.  It sounds like we may be able
> to accomplish that by redirecting everything to a temp dir, which is
> likely fine.

Yes, by changing native-comp-eln-load-path you should be able to avoid
writing to the home directory of the user under whose credentials the
installation runs.  If you need that.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> You should know that the problems you needed to address with the *.elc
> files don't exist with the *.eln files.  The *.eln files have version
> information of the Emacs to which they "belong" hard-coded into their
> names.

Right, I'd noticed that, and while I haven't fully understood the
expectations/semantics yet, I'd started to assume much of what you've
described.  I also assumed we'd need to understand things even better
before the Debian support is completely "ready" (right now we're working
all this out in unstable).

My current expectation is that if it ends up being feasible, we'll
probably want to treat the package-related .eln files like the .elc
files, and build them during the package install.  For example,
elpa-magit (the current magit add-on package) will want to generate both
the .elc and .eln files for all of its .el files when its turn comes
around during installs/upgrades.

I say that because if it's a single-user machine and the user invokes
"apt install elpa-magit", I'd imagine they're doing that because they
want to use magit, in which case it doesn't hurt to get the work out of
the way up-front, and it might help in that all the work will be
finished at once, so they won't incur costs (battery-consumption, etc.)
later, when they might not expect or want it.  (Not a critical issue,
possibly, but perhaps friendlier.)

And if it's a multi-user machine, with a lot of emacs users, at the
moment I don't see any reason to want to compile the same file 50 times
for 50 users (or even more than just once), incurring the attendant
power and storage costs.

> Yes, by changing native-comp-eln-load-path you should be able to avoid
> writing to the home directory of the user under whose credentials the
> installation runs.  If you need that.

OK, thanks much.
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org,
>  akrl@sdf.org
> Date: Sat, 01 Oct 2022 15:42:06 -0500
> 
> My current expectation is that if it ends up being feasible, we'll
> probably want to treat the package-related .eln files like the .elc
> files, and build them during the package install.  For example,
> elpa-magit (the current magit add-on package) will want to generate both
> the .elc and .eln files for all of its .el files when its turn comes
> around during installs/upgrades.
> 
> I say that because if it's a single-user machine and the user invokes
> "apt install elpa-magit", I'd imagine they're doing that because they
> want to use magit, in which case it doesn't hurt to get the work out of
> the way up-front, and it might help in that all the work will be
> finished at once, so they won't incur costs (battery-consumption, etc.)
> later, when they might not expect or want it.  (Not a critical issue,
> possibly, but perhaps friendlier.)
> 
> And if it's a multi-user machine, with a lot of emacs users, at the
> moment I don't see any reason to want to compile the same file 50 times
> for 50 users (or even more than just once), incurring the attendant
> power and storage costs.

I don't think you should try to second-guess the user who installs a
package.  They could just want to study the sources, for example.

The native compilation is implemented as JIT capability for a reason.
We dicussed the various aspects of that for a long time before
deciding on what you see today.  My advice is not to reject that
without very good reasons (and those reasons should probably be
communicated to us when found), just because of some analogy with byte
compilation, as that analogy breaks on several levels.  By using
native compilation in its default JIT manner, you are basically using
Emacs as the upstream project meant it to be used, which means, among
others, that you don't need to fight an uphill battle against various
defaults and discover situations that no one expected to happen.

The reasons which you mention in favor of AOT native compilation don't
sound serious enough: I see no problem in having the compilation
happen only when it's needed in those cases.  Battery consumption
doesn't seem very relevant, because JIT compilation will happen when
the system is used, not when it sleeps.  And it is entirely not
guaranteed that by compiling everything you will save power in the
long run, because there are good chances you will be compiling stuff
that won't be used for a long time.  Without quantitative data of
long-term power usage on which to base the conclusions, I don't see
why you should a-priori assume that compiling everything from the
get-go should use less power.  Same goes for disk space by multiple
users.

All in all, I think JIT compilation strikes a good balance between
resources and their actual usage.  This also matches my personal
experience of using Emacs 28 with native compilation since day one: I
never had to look back on the decision not to use AOT compilation of
everything.

Of course, this is eventually your call.  But my recommendation is to
try to stick to the default behavior unless it causes actual (as
opposed to imaginary or presumed) problems, based on actual usage.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3188 bytes --]

On Fri, Sep 30, 2022 at 04:56:56PM +0300, Eli Zaretskii wrote:
> > From: David Bremner <david@tethera.net>

[...]

> > Some additional byte compilation happens at package install time.
> 
> This is what I'm asking about: what exactly triggers the compilation?
> Just installing a package shouldn't do that, only loading it into
> Emacs should.

I see there two views: the distributor's (represented here by David's)
and the jit's (Eli). For Eli, the file system objects are just a long
term cache of the jit's work (longer than one Emacs session), for
David, since the package's .el (and possibly .elc) are immutable (and
of course, the whole Emacs runtime), the corresponding .eln can well
be built at package install time. That would avoid each user to have
an identical copy of them (and of course, the initial wait).

That does make a lot of sense in a multi-user system (see below for
more on that).

> The other part of what I asked is that if for some reason installation
> does need to compile files (something that I still need to understand
> why it happens), there's nothing that hard-codes HOME in the directory
> list used for that.  You can set native-comp-eln-load-path to anything
> you want, and only the directories in that list will be used.

Yes, but. It's not about HOME, but about a user-writable space. A shared
space for .eln would have to be writable by all users if the compilation
is to succeed when invoked on behalf of each one. This is a no-no under
a multiuser system.

Install is invoked as root, so it can put the results of its compilation
to a place readable by /all/ users (typically some /usr/share or /usr/lib,
depending on whether the results are architecture-independent).

> > In my
> > view the same restrictions should apply there. In addition to the
> > unintentional sharing of state mentioned in policy, there is also the
> > issue of cleaning up after a package on uninstall. This is manageable
> > now because any generated .elc files go in a package specific directory,
> > which can just be removed on purging the package.
> 
> See above: you can do the same with *.eln files, if, for some reason,
> they are produced by the installation.
> 
> > I think just turning off native compilation when HOME is not writeable
> > would be sensible from a Debian point of view. Emacs did not previously
> > require a writable HOME (although maybe most users never tested
> > this). It would be unfortunate if this changed for Emacs with native
> > compilation support.
> 
> Emacs doesn't require a writable HOME, it requires a writable
> directory to store *.eln files.  It doesn't have to be HOME.  And once
> again, I still don't understand why *.eln files are produced at
> installation time in the first place.

That's all fine, but then users wouldn't profit from the pre-compiled
.eln. In a Debian-distributed Emacs (caveat: I'm compiling my own
one!) there are .elc in /usr/share for all to use; due to the search
path, a user still can install her own in a directory writable by
that user and it will take precedence.

Can you do the same for .elc?

Cheers
-- 
t


> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

On Sun, Oct 02, 2022 at 08:57:13AM +0300, Eli Zaretskii wrote:
> > From: Rob Browning <rlb@defaultvalue.org>
> > Cc: monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org,
> >  akrl@sdf.org
> > Date: Sat, 01 Oct 2022 15:42:06 -0500

[...]

> > And if it's a multi-user machine, with a lot of emacs users, at the
> > moment I don't see any reason to want to compile the same file 50 times
> > for 50 users (or even more than just once), incurring the attendant
> > power and storage costs.
> 
> I don't think you should try to second-guess the user who installs a
> package.  They could just want to study the sources, for example.

That's what apt-get source and friends are. The user can download,
build, modify, etc. the sources corresponding to packages. Those
are purely user operations, no admin powers needed.

But installing a binary pacage on a system does modify the system
for all users, so admin powers do make sense there. Nobody is being
second-guessed, just the roles separated (again, on a single user
system, this might feel artificial).

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Sun, 2 Oct 2022 07:58:33 +0200
> From: <tomas@tuxteam.de>
> 
> I see there two views: the distributor's (represented here by David's)
> and the jit's (Eli). For Eli, the file system objects are just a long
> term cache of the jit's work (longer than one Emacs session), for
> David, since the package's .el (and possibly .elc) are immutable (and
> of course, the whole Emacs runtime), the corresponding .eln can well
> be built at package install time. That would avoid each user to have
> an identical copy of them

As explained in my other message, I see no reason to make such
assumptions.  There's no reason to believe that a user who installs a
package will immediately start using it.  And even it they do, why
should we care about some extra disk space or identical files being
present? disk space is cheap, while the flexibility this allows (like
each user can have a slightly differently-configured Emacs) is
non-negligible.

> (and of course, the initial wait).

There's no wait: a package can be used immediately after loading or
requiring it.  The replacement of a byte-compiled by natively-compiled
code happens transparently and doesn't affect usage nor causes any
waits.

The advantage of using JIT compilation is that this is how the
upstream project meant it to be used, and this is how the defaults are
set.  Any deviation from the defaults should have a good reason, and I
submit that such good reasons can only be based on actual usage, not
on theoretical presumptions.  My recommendation is to use the default
JIT manner until and unless actual problems are reported by users.

> That does make a lot of sense in a multi-user system

Not to me, it doesn't.  It makes _some_ sense, but there are other
considerations.

> > The other part of what I asked is that if for some reason installation
> > does need to compile files (something that I still need to understand
> > why it happens), there's nothing that hard-codes HOME in the directory
> > list used for that.  You can set native-comp-eln-load-path to anything
> > you want, and only the directories in that list will be used.
> 
> Yes, but. It's not about HOME, but about a user-writable space. A shared
> space for .eln would have to be writable by all users if the compilation
> is to succeed when invoked on behalf of each one. This is a no-no under
> a multiuser system.

??? What "user-writable space"?  If we are talking about install-time
compilation, then the *.eln files should be deposited in some
centralized place to which the installation process can write.  And if
we are talking about JIT compilation when the user runs Emacs, then
the home directory is perfectly appropriate, and is writable.  No one
suggested that JIT compilations from "normal" user session should
write to shared directories (and if someone did suggest that, IMO this
suggestion makes no sense).

So I think you confused these two possibilities, because the problem
you raise doesn't exist.

> Install is invoked as root, so it can put the results of its compilation
> to a place readable by /all/ users (typically some /usr/share or /usr/lib,
> depending on whether the results are architecture-independent).

Exactly.  So what is the problem with directories writable by all
users?

> > Emacs doesn't require a writable HOME, it requires a writable
> > directory to store *.eln files.  It doesn't have to be HOME.  And once
> > again, I still don't understand why *.eln files are produced at
> > installation time in the first place.
> 
> That's all fine, but then users wouldn't profit from the pre-compiled
> .eln.

There's no profit, IME.  There are only disadvantages: you are in
effect fighting against the Emacs defaults, for reasons that are
purely theoretical.

> In a Debian-distributed Emacs (caveat: I'm compiling my own one!)
> there are .elc in /usr/share for all to use; due to the search path,
> a user still can install her own in a directory writable by that
> user and it will take precedence.
> 
> Can you do the same for .elc?

I guess you meant "with .eln files"?  Yes, see
native-comp-eln-load-path, which was already mentioned here several
times.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Sun, 2 Oct 2022 08:10:15 +0200
> From: <tomas@tuxteam.de>
> 
> > I don't think you should try to second-guess the user who installs a
> > package.  They could just want to study the sources, for example.
> 
> That's what apt-get source and friends are. The user can download,
> build, modify, etc. the sources corresponding to packages. Those
> are purely user operations, no admin powers needed.
> 
> But installing a binary pacage on a system does modify the system
> for all users, so admin powers do make sense there.

What do you mean by "binary package" in this context?  I believe
"package" in this discussion generally alludes to Lisp packages, from
ELPA and elsewhere.  They aren't "binary" in my book.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3484 bytes --]

On Sun, Oct 02, 2022 at 09:42:04AM +0300, Eli Zaretskii wrote:
> > Date: Sun, 2 Oct 2022 07:58:33 +0200
> > From: <tomas@tuxteam.de>
> > 
> > I see there two views: the distributor's (represented here by David's)
> > and the jit's (Eli) [...]

> As explained in my other message, I see no reason to make such
> assumptions.  There's no reason to believe that a user who installs a
> package will immediately start using it [...]

(NOTE Eli, I admire your patience :-)

As I see from your other post, there is a confusion potential with
the word "package". I was talking about Debian binary packages, which
are thought for those who want to use them. For those wanting to
change or build there are the source packages (which come with a
convenient build "kit" enabling you to build the binary package as
it comes with Debian, which is a convenient starting point for
tinkering).

> And even it they do, why
> should we care about some extra disk space or identical files being
> present? disk space is cheap, while the flexibility this allows (like
> each user can have a slightly differently-configured Emacs) is
> non-negligible.

Definitely. I am not that much concerned about disk usage, more about
a uniform "baseline" installation all users can rely on.

[...]

> The advantage of using JIT compilation is that this is how the
> upstream project meant it to be used, and this is how the defaults are
> set.  Any deviation from the defaults should have a good reason, and I
> submit that such good reasons can only be based on actual usage, not
> on theoretical presumptions.  My recommendation is to use the default
> JIT manner until and unless actual problems are reported by users.

I see that, and this is the one view I mention above: the .eln are but
a JIT cache, and each user (or instance, if there are more than one
per user) has its own.

> > That does make a lot of sense in a multi-user system
> 
> Not to me, it doesn't.  It makes _some_ sense, but there are other
> considerations.

Two views, as I said :)

[HOME, user-writable space]

> Exactly.  So what is the problem with directories writable by all
> users?

User separation goes out of the window, and this is one important
service the OS provides. To illustrate, one user could put malicious
.eln files all other users would execute.

> > > Emacs doesn't require a writable HOME, it requires a writable
> > > directory to store *.eln files.  It doesn't have to be HOME.  And once
> > > again, I still don't understand why *.eln files are produced at
> > > installation time in the first place.
> > 
> > That's all fine, but then users wouldn't profit from the pre-compiled
> > .eln.
> 
> There's no profit, IME.  There are only disadvantages: you are in
> effect fighting against the Emacs defaults, for reasons that are
> purely theoretical.

I have the impression that some of that reasons are quite practical
for Debian packagers.

> > In a Debian-distributed Emacs (caveat: I'm compiling my own one!)
> > there are .elc in /usr/share for all to use; due to the search path,
> > a user still can install her own in a directory writable by that
> > user and it will take precedence.
> > 
> > Can you do the same for .elc?
> 
> I guess you meant "with .eln files"?  Yes, see
> native-comp-eln-load-path, which was already mentioned here several
> times.

So that might be one part of the way out.

Cheers
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 597 bytes --]

On Sun, Oct 02, 2022 at 09:44:16AM +0300, Eli Zaretskii wrote:
> > Date: Sun, 2 Oct 2022 08:10:15 +0200
> > From: <tomas@tuxteam.de>

[...]

> > But installing a binary pacage on a system does modify the system
> > for all users, so admin powers do make sense there.
> 
> What do you mean by "binary package" in this context?  I believe
> "package" in this discussion generally alludes to Lisp packages, from
> ELPA and elsewhere.  They aren't "binary" in my book.

Ah, I see. I was too sloppy. I was talking about Debian packages.

Sorry for the confusion.

Cheers
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Sun, 2 Oct 2022 17:54:44 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > > But installing a binary pacage on a system does modify the system
> > > for all users, so admin powers do make sense there.
> > 
> > What do you mean by "binary package" in this context?  I believe
> > "package" in this discussion generally alludes to Lisp packages, from
> > ELPA and elsewhere.  They aren't "binary" in my book.
> 
> Ah, I see. I was too sloppy. I was talking about Debian packages.

That still doesn't explain it to me (I don't use Debian).  Can you
elaborate what kind of packages are we talking about, in the context
of this discussion?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Sun, 2 Oct 2022 17:53:46 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > The advantage of using JIT compilation is that this is how the
> > upstream project meant it to be used, and this is how the defaults are
> > set.  Any deviation from the defaults should have a good reason, and I
> > submit that such good reasons can only be based on actual usage, not
> > on theoretical presumptions.  My recommendation is to use the default
> > JIT manner until and unless actual problems are reported by users.
> 
> I see that, and this is the one view I mention above: the .eln are but
> a JIT cache, and each user (or instance, if there are more than one
> per user) has its own.

Let me be blunt: this is currently _the_only_ view of the Emacs
project.  After a lot of deliberations, we didn't find any reasons for
alternative views.  My suggestion is to try our view before concluding
that it doesn't fit some situations.

> > Exactly.  So what is the problem with directories writable by all
> > users?
> 
> User separation goes out of the window, and this is one important
> service the OS provides. To illustrate, one user could put malicious
> .eln files all other users would execute.

This is about installation writing files into a shared space on disk,
right?  If so, this is something for the Debian packagers to figure
out, because doing that is their request.  And anyway, I don't
understand how do *.eln files are different from *.elc files, which
are already written to shared disk space upon installation.  What am I
missing?

> > > That's all fine, but then users wouldn't profit from the pre-compiled
> > > .eln.
> > 
> > There's no profit, IME.  There are only disadvantages: you are in
> > effect fighting against the Emacs defaults, for reasons that are
> > purely theoretical.
> 
> I have the impression that some of that reasons are quite practical
> for Debian packagers.

I submit that those reasons were most probably derived from a broken
analogy with the *.elc files and with byte-compilation in general.
Not from actual usage experience.  Native compilation looks
deceptively similar to byte compilation, but it isn't.  So if
producing the *.eln files seems to contradict some Debian rules and
procedures, my suggestion is to talk to the upstream project, before
inventing solutions, because of 2 considerations:

  . the problems may not be real ones, only perceived ones
  . the upstream codebase might already provide solutions

> > > In a Debian-distributed Emacs (caveat: I'm compiling my own one!)
> > > there are .elc in /usr/share for all to use; due to the search path,
> > > a user still can install her own in a directory writable by that
> > > user and it will take precedence.
> > > 
> > > Can you do the same for .elc?
> > 
> > I guess you meant "with .eln files"?  Yes, see
> > native-comp-eln-load-path, which was already mentioned here several
> > times.
> 
> So that might be one part of the way out.

If one needs it.  I don't think they do, and I don't recommend going
there.

Re: Suppressing native compilation (short and long term)

[chad]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]

On Sun, Oct 2, 2022 at 12:03 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > Ah, I see. I was too sloppy. I was talking about Debian packages.
>
> That still doesn't explain it to me (I don't use Debian).  Can you
> elaborate what kind of packages are we talking about, in the context
> of this discussion?
>

Debian supports a user installing, for example, emacs+magit, via Debian
packages. This gets the user a stable, known-tested version of emacs plus
the package usable on machines that, for another example, cannot or should
not connect to the internet. This is less important to developers, but is
an important part of the Debian support "contract".

For the record: I personally know of situations where such a setup would
like to use native-comp and would very much prefer NOT to duplicate either
the eln files per-user nor the effort of creating such. Whether or not that
situation is important enough to the combination of emacs-devel and
debian-maintainers to justify effort on either side is, of course,
debatable, but I am highly confident that they exist (at least, did before
the pandemic).

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 1557 bytes --]

Re: Suppressing native compilation (short and long term)

[chad]

[-- Attachment #1: Type: text/plain, Size: 1761 bytes --]

On Sun, Oct 2, 2022 at 12:13 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > User separation goes out of the window, and this is one important
> > service the OS provides. To illustrate, one user could put malicious
> > .eln files all other users would execute.
>
> This is about installation writing files into a shared space on disk,
> right?  If so, this is something for the Debian packagers to figure
> out, because doing that is their request.  And anyway, I don't
> understand how do *.eln files are different from *.elc files, which
> are already written to shared disk space upon installation.  What am I
> missing?
>

At the risk of over-explaining due to message crossing mid-flight: the
thing you might be missing is that Debian provides a mechanism for people
to install *.elc files in a space shared by all users that is not
writable by those users, and there are people that use this provision.
Since these are used for *.elc files, it seems highly likely that they will
be desired for *.eln files.

Even in the face of a theoretical issue like "potential package
combinations make it unworkable to pre-build a single set of
emacs+emacs-packages", the practical situation is that such combinations
exist and are used by Debian users to build a stable base of
emacs+emacs-packages that is shared across users who cannot change that
shared base (but can, of course, supplement it with their own packages, via
site-lisp, user packages, etc.) As a practical goal, there is at least
*some* impetus for Debian to provide such a stable base, and to make it as
wide as reasonable. The basic mechanism for determining the size of that
base is "which emacs-packages are made into debian-packages", (iff I
understand correctly; I'm not a Debian expert).

~Chad

[-- Attachment #2: Type: text/html, Size: 2215 bytes --]

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3544 bytes --]

On Sun, Oct 02, 2022 at 07:11:52PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 2 Oct 2022 17:53:46 +0200
> > From: tomas@tuxteam.de
> > Cc: emacs-devel@gnu.org

[two views: JIT cache vs. pre-compiled .eln]

> Let me be blunt: this is currently _the_only_ view of the Emacs
> project.  After a lot of deliberations, we didn't find any reasons for
> alternative views.  My suggestion is to try our view before concluding
> that it doesn't fit some situations.

Thanks for being blunt :-)

My whole intention was to make this difference clear, because
I had the impression that there were unspoken assumptions
making the discussion unnecessarily difficult.

> > > Exactly.  So what is the problem with directories writable by all
> > > users?
> > 
> > User separation goes out of the window, and this is one important
> > service the OS provides. To illustrate, one user could put malicious
> > .eln files all other users would execute.
> 
> This is about installation writing files into a shared space on disk,
> right?

No. I was picking up on your "directories writable by all users".
Perhaps I misunderstood and you didn't mean common directories:
if so, please ignore.

>  If so, this is something for the Debian packagers to figure
> out, because doing that is their request.  And anyway, I don't
> understand how do *.eln files are different from *.elc files, which
> are already written to shared disk space upon installation.  What am I
> missing?

Perhaps nothing. With the native-comp-eln-load-path, there seems
to be a way for Debian to "do it its way"; you don't recommend
it (I still don't quite understand), and there are very strong
reasons to take your recommendations seriously.

> > > > That's all fine, but then users wouldn't profit from the pre-compiled
> > > > .eln.
> > > 
> > > There's no profit, IME.  There are only disadvantages: you are in
> > > effect fighting against the Emacs defaults, for reasons that are
> > > purely theoretical.
> > 
> > I have the impression that some of that reasons are quite practical
> > for Debian packagers.
> 
> I submit that those reasons were most probably derived from a broken
> analogy with the *.elc files and with byte-compilation in general.
> Not from actual usage experience.  Native compilation looks
> deceptively similar to byte compilation, but it isn't.  So if
> producing the *.eln files seems to contradict some Debian rules and
> procedures, my suggestion is to talk to the upstream project, before
> inventing solutions, because of 2 considerations:

I understand that there is a difference between .elc and .eln (the
set of dependencies is significantly bigger in the second case, for
one).

> 
>   . the problems may not be real ones, only perceived ones
>   . the upstream codebase might already provide solutions

I can understand Debian's position here (yours too).

> > > > In a Debian-distributed Emacs [...]
> > > > there are .elc in /usr/share for all to use; due to the search path,

[...]

> > > > Can you do the same for .elc?
> > > 
> > > I guess you meant "with .eln files"?

Uh -- yes, sorry. Well spotted.

> > > Yes, see native-comp-eln-load-path, which was already mentioned
> > > here several times.
> > 
> > So that might be one part of the way out.
> 
> If one needs it.  I don't think they do, and I don't recommend going
> there.

Hm. I don't want to steal your time more, but... if you could illustrate
why, I'd eager to learn.

Cheers
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> I don't think you should try to second-guess the user who installs a
> package.  They could just want to study the sources, for example.

I'm quite happy with the use of JIT as the default in Emacs.
But I think I agree with Rob that it makes a lot of sense in the context
of Debian to eagerly native-compile the packages when they're installed
via APT.

In APT there's a clearly distinction between installing the package
(which requires admin rights and has non-trivial effects on the whole
system) and looking at the source code of the package (this distinction
is natural in the context of Debian where many/most packages are written
in languages like C, but it naturally carries over to ELisp packages).

So if user "just want to study the sources" they wouldn't *install* the
package at all.

> All in all, I think JIT compilation strikes a good balance between
> resources and their actual usage.

Yes.  But the balance is different in different contexts.

FWIW, I'm not convinced it's really useful in Debian's `emacs` package
to eagerly native compile all the bundled .elc files, but I think it
does make a lot of sense for ELisp packages installed separately.

In any case, I'd let Debian's maintainers make their own choices for
their own specific needs which are slightly different from ours (where
our release tarballs and default config are designed in large part for
users who'll compile Emacs themselves and who install third party
ELisp packages into their $HOME).


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Sun, 2 Oct 2022 12:13:44 -0400
> Cc: tomas@tuxteam.de, emacs-devel@gnu.org
> 
>  > Ah, I see. I was too sloppy. I was talking about Debian packages.
> 
>  That still doesn't explain it to me (I don't use Debian).  Can you
>  elaborate what kind of packages are we talking about, in the context
>  of this discussion?
> 
> Debian supports a user installing, for example, emacs+magit, via Debian packages. This gets the user a
> stable, known-tested version of emacs plus the package usable on machines that, for another example,
> cannot or should not connect to the internet. This is less important to developers, but is an important part of
> the Debian support "contract".

Which part of the "emacs+magit" package is the "binary package"?  The
only part of that which could produce *.eln files at installation time
is Magit, right?  Because Emacs itself already comes with all the
preloaded *.eln files, and those aren't what we are talking about,
right?

And if we are talking about Magit in this example, then how come its
being bundled to Emacs change anything of what I said?

> For the record: I personally know of situations where such a setup would like to use native-comp and would
> very much prefer NOT to duplicate either the eln files per-user nor the effort of creating such. Whether or
> not that situation is important enough to the combination of emacs-devel and debian-maintainers to justify
> effort on either side is, of course, debatable, but I am highly confident that they exist (at least, did before the
> pandemic).

Please forgive me, but you cannot expect us to regard such situations
seriously as long as you don't describe them.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Sun, 2 Oct 2022 12:23:43 -0400
> Cc: tomas@tuxteam.de, emacs-devel@gnu.org
> 
>  > User separation goes out of the window, and this is one important
>  > service the OS provides. To illustrate, one user could put malicious
>  > .eln files all other users would execute.
> 
>  This is about installation writing files into a shared space on disk,
>  right?  If so, this is something for the Debian packagers to figure
>  out, because doing that is their request.  And anyway, I don't
>  understand how do *.eln files are different from *.elc files, which
>  are already written to shared disk space upon installation.  What am I
>  missing?
> 
> At the risk of over-explaining due to message crossing mid-flight: the thing you might be missing is that
> Debian provides a mechanism for people to install *.elc files in a space shared by all users that is not
> writable by those users, and there are people that use this provision. Since these are used for *.elc files, it
> seems highly likely that they will be desired for *.eln files.

No, I don't think the similar handling makes sense here.  The *.elc
files are architecture- and configuration-independent, whereas the
*.eln files are not.  E.g., the same foo.elc could be used by user A
who runs Emacs 28 and by user B who runs Emacs 29.  But the
corresponding *.eln files will be different, even though they were
produced from the same foo.el.

> Even in the face of a theoretical issue like "potential package combinations make it unworkable to pre-build a
> single set of emacs+emacs-packages", the practical situation is that such combinations exist and are used
> by Debian users to build a stable base of  emacs+emacs-packages that is shared across users who
> cannot change that shared base (but can, of course, supplement it with their own packages, via site-lisp,
> user packages, etc.) As a practical goal, there is at least *some* impetus for Debian to provide such a stable
> base, and to make it as wide as reasonable. The basic mechanism for determining the size of that base is
> "which emacs-packages are made into debian-packages", (iff I understand correctly; I'm not a Debian
> expert).

I don't think this is relevant to the issue at hand.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Sun, 2 Oct 2022 18:27:05 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > > > Yes, see native-comp-eln-load-path, which was already mentioned
> > > > here several times.
> > > 
> > > So that might be one part of the way out.
> > 
> > If one needs it.  I don't think they do, and I don't recommend going
> > there.
> 
> Hm. I don't want to steal your time more, but... if you could illustrate
> why, I'd eager to learn.

I already did: the probability of different users to produce different
*.eln files from the same *.el sources is high, and disk space is
cheap.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> Which part of the "emacs+magit" package is the "binary package"?

The "binary" Debian package for `elpa-magit` mostly contains Magit's
`.el` files plus the doc and a few other things.  IOW it's fairly
similar to our ELPA tarballs.

So it's very close to the source code itself, but it's still separate
from "the source" (which you can get via things like `apt source elpa-magit`
and will fetch some tarball rather than a `.deb` file) from which the
`.deb` is built.

> And if we are talking about Magit in this example, then how come its
> being bundled to Emacs change anything of what I said?

I don't think there's a `emacs+magit` package in Debian.  There's an
`elpa-magit` package and an `emacs` package.

>> For the record: I personally know of situations where such a setup
>> would like to use native-comp and would very much prefer NOT to
>> duplicate either the eln files per-user nor the effort of creating
>> such. Whether or not that situation is important enough to the
>> combination of emacs-devel and debian-maintainers to justify effort
>> on either side is, of course, debatable, but I am highly confident
>> that they exist (at least, did before the pandemic).
>
> Please forgive me, but you cannot expect us to regard such situations
> seriously as long as you don't describe them.

We don't have to take their opinion into account when choosing
Emacs's defaults.  We just need to provide the tools that let them get
the behavior they want.


        Stefan

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Battery consumption doesn't seem very relevant, because JIT
> compilation will happen when the system is used, not when it sleeps.
> And it is entirely not guaranteed that by compiling everything you
> will save power in the long run, because there are good chances you
> will be compiling stuff that won't be used for a long time.  Without
> quantitative data of long-term power usage on which to base the
> conclusions, I don't see why you should a-priori assume that compiling
> everything from the get-go should use less power.  Same goes for disk
> space by multiple users.

On this particular topic, I was actually just communicating a concern
that was communicated to me -- that a user wasn't happy about the
unpredictable, compilation spike long after the install on their laptop,
(in part due to concerns about power consumption).

(Personally, as a user, I'd also prefer to pay the cost up front, most
 of the time, during package install, and to avoid per-user duplications
 where possible, but that said, I generally do try to avoid basing
 Debian packaging decisions solely on my preferences, and I do want to
 try to favor upstream preferences in general.)

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Rob Browning <rlb@defaultvalue.org>,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Sun, 02 Oct 2022 12:53:56 -0400
> 
> > All in all, I think JIT compilation strikes a good balance between
> > resources and their actual usage.
> 
> Yes.  But the balance is different in different contexts.

Theoretically, yes.  But I don't yet see why the particular context
described by Rob is different to the degree that it would need special
procedures.

> FWIW, I'm not convinced it's really useful in Debian's `emacs` package
> to eagerly native compile all the bundled .elc files, but I think it
> does make a lot of sense for ELisp packages installed separately.

Maybe so, but even if that is the decision about the *.elc files, it
doesn't automatically follow that the *.eln files should be handled
the same.  I explained in previous messages why I think so, with
enough important differences between them to facilitate rethinking, I
hope.

> In any case, I'd let Debian's maintainers make their own choices for
> their own specific needs which are slightly different from ours (where
> our release tarballs and default config are designed in large part for
> users who'll compile Emacs themselves and who install third party
> ELisp packages into their $HOME).

We can only advise them, yes.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> But I think I agree with Rob that it makes a lot of sense in the context
> of Debian to eagerly native-compile the packages when they're installed
> via APT.

I've always assumed that that's what distributions like Debian would
do.  Which means that there should be an easy way to switch JIT off, but
I've just forgotten to add it.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: chad <yandros@gmail.com>,  tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 13:13:59 -0400
> 
> > Which part of the "emacs+magit" package is the "binary package"?
> 
> The "binary" Debian package for `elpa-magit` mostly contains Magit's
> `.el` files plus the doc and a few other things.  IOW it's fairly
> similar to our ELPA tarballs.

That's what I thought.  But then I don't see how this is special in
the context of this discussion, so that what Emacs users do when they
download from ELPA is different from them installing a Debian "binary"
package.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> That still doesn't explain it to me (I don't use Debian).  Can you
> elaborate what kind of packages are we talking about, in the context
> of this discussion?

One possible way to look at it -- on a Debian system I think you'd
generally

  apt install emacs  # (or gcc-12 or ...)

if you just wanted to use emacs or gcc, or...  And that would install
the binaries, more or less, i.e. roughly like "make install", but with
automatic dependency resolution, etc.

And note that with many tools that would get you *just* the binaries,
i.e. no source at all for say gcc, libc, grep, git, etc.

If you wanted to investigate, fix, enhance the packaging itself, then
you might

  apt source emacs

which would download the packaging and unpack it in the current
directory, as say emacs-28.1+1-3/, and where all the packaging
related-changes, including any patches applied to the upstream code,
would be clearly visible in a emacs-28.1+1-3/debian/ subdirectory in the
emacs source.

For many packages these days you might also be able to just clone the
Debian packaging archive, e.g.

  git clone https://salsa.debian.org/rlb/deb-emacs.git

which might be preferable if you're more comfortable with git.

And finally, if your primary interest is the upstream source, and/or
running and/or working on anything other than the versions debian
currently provides, I suspect you'd likely want to

  git clone git://git.savannah.gnu.org/emacs.git

i.e. you wouldn't need and might well not want the debian packag(ing|es)
at all.

Hope this helps.
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> On this particular topic, I was actually just communicating a concern
> that was communicated to me -- that a user wasn't happy about the
> unpredictable, compilation spike long after the install on their laptop,
> (in part due to concerns about power consumption).

FWIW, regardless of what Debian decides to do, I think it would make
sense to offer some way to configure Emacs so that it native compiles
packages a bit more eagerly.

E.g. we could offer a configuration option that causes `package-install`
to eagerly native compile the installed files.  And we could offer
a "recompile" command that users can use after upgrading Emacs to
eagerly native-(re)compile all the ELPA-installed packages in the
user's $HOME.

The situation is very different for the bundled packages, since most of
them will never be used by most users, so JIT compilation is much
more beneficial.


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org,
>  akrl@sdf.org
> Date: Sun, 02 Oct 2022 12:15:27 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Battery consumption doesn't seem very relevant, because JIT
> > compilation will happen when the system is used, not when it sleeps.
> > And it is entirely not guaranteed that by compiling everything you
> > will save power in the long run, because there are good chances you
> > will be compiling stuff that won't be used for a long time.  Without
> > quantitative data of long-term power usage on which to base the
> > conclusions, I don't see why you should a-priori assume that compiling
> > everything from the get-go should use less power.  Same goes for disk
> > space by multiple users.
> 
> On this particular topic, I was actually just communicating a concern
> that was communicated to me -- that a user wasn't happy about the
> unpredictable, compilation spike long after the install on their laptop,
> (in part due to concerns about power consumption).

That's just something that takes getting used to.

> (Personally, as a user, I'd also prefer to pay the cost up front, most
>  of the time, during package install, and to avoid per-user duplications
>  where possible, but that said, I generally do try to avoid basing
>  Debian packaging decisions solely on my preferences, and I do want to
>  try to favor upstream preferences in general.)

Those seem like old habit that die hard, IMO.  For users of some
programming languages, JIT compilation is routine, and I bet they
don't share these habits.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Lars Ingebrigtsen [2022-10-02 19:17:27] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> But I think I agree with Rob that it makes a lot of sense in the context
>> of Debian to eagerly native-compile the packages when they're installed
>> via APT.
> I've always assumed that that's what distributions like Debian would
> do.  Which means that there should be an easy way to switch JIT off, but
> I've just forgotten to add it.

Switching it off means that the native compilation is never
auto-triggered, so it's probably not quite what you meant :-)


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Rob Browning <rlb@defaultvalue.org>,
>   david@tethera.net,  emacs-devel@gnu.org,  akrl@sdf.org
> Date: Sun, 02 Oct 2022 19:17:27 +0200
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > But I think I agree with Rob that it makes a lot of sense in the context
> > of Debian to eagerly native-compile the packages when they're installed
> > via APT.
> 
> I've always assumed that that's what distributions like Debian would
> do.  Which means that there should be an easy way to switch JIT off, but
> I've just forgotten to add it.

If everything is already natively-compiled, there's no reason to
switch it off: it will simply not happen.

OTOH, if for some reason an existing .eln file is incompatible with
the user's Emacs (something that can happen easily on a multi-user
system), having JIT compilation active is a clear advantage.

So there's a net win in having it enabled at all times.

In any case, we already have a way of switching off JIT compilation,
at least for some situations which we considered.  If there are
situations which we didn't consider, we should consider them as well,
but for that we need to understand the situation in more detail,
something that no one presented yet in this discussion.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Switching it off means that the native compilation is never
> auto-triggered, so it's probably not quite what you meant :-)

I'm not quite sure what you mean -- I'm saying that there should be a
way for users to switch off native compilation (and for distributions to
have native compilation for users switched off).

Which was how this thread started: With a request for such a thing.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> In any case, I'd let Debian's maintainers make their own choices for
> their own specific needs which are slightly different from ours (where
> our release tarballs and default config are designed in large part for
> users who'll compile Emacs themselves and who install third party
> ELisp packages into their $HOME).

For what it's worth, we've also had repeated requests for additional
variants of emacs.  Right now we have emacs-nox (No X), emacs-lucid, and
emacs-gtk.

The first is often useful for server installs, but we've had requests
for something even smaller, say an emacs-min that configures even less,
and is even smaller (including the dependency tree), for some
situations.  Presumably people who would rather be able to easily use
emacs in some constrained environments instead of nano, zile, vi,
... etc.

We've also for a long time made the emacs-el package (containing all the
.el files) optional for similar reasons (constrained environments), but
we recently had to start requiring it because (not sure why yet) emacs
just crashes in some situations when the .el files aren't available and
when native compilation is enabled.

If it hasn't already been fixed in 28.2 (haven't tested yet), it'd be
nice if we could eventually track that down and then switch emacs-el
back to optional (back to "suggested" in Debian dependency parlance).

We'll likely try to help with that later, after we get the other native
compilation-related issues we've been discussing settled down on the
Debian side.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> So there's a net win in having it enabled at all times.

You may think so, and I may think so, but it should be up to the user
(and the distribution).

> In any case, we already have a way of switching off JIT compilation,
> at least for some situations which we considered.

I thought there wasn't a user option to switch it off?  I must admit I
haven't been paying that much attention to the thread, but if somebody
has identified such a user option, then everything's OK.  What's it
called?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Rob Browning <rlb@defaultvalue.org>,
>  david@tethera.net,  emacs-devel@gnu.org,  akrl@sdf.org
> Date: Sun, 02 Oct 2022 19:36:01 +0200
> 
> I'm not quite sure what you mean -- I'm saying that there should be a
> way for users to switch off native compilation (and for distributions to
> have native compilation for users switched off).
> 
> Which was how this thread started: With a request for such a thing.

Which turned out to be a request for something else: for controlling
where the *.eln files are written.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: david@tethera.net, emacs-devel@gnu.org, akrl@sdf.org
> Date: Sun, 02 Oct 2022 12:37:33 -0500
> 
> We've also for a long time made the emacs-el package (containing all the
> .el files) optional for similar reasons (constrained environments), but
> we recently had to start requiring it because (not sure why yet) emacs
> just crashes in some situations when the .el files aren't available and
> when native compilation is enabled.

Loading of natively-compiled files cannot work if the source files
aren't accessible, because loading a .eln file involves verifying that
it was produced from that particular .el file.  (The .el file can be
compressed if Emacs was compiled with zlib support.)

But Emacs should not "crash" if the *.el files aren't available, it
should simply refuse to load any *.eln files and load the *.elc files
instead.  That produces many warnings, of course, but I hope your
users don't consider that "crashing".

> If it hasn't already been fixed in 28.2 (haven't tested yet), it'd be
> nice if we could eventually track that down and then switch emacs-el
> back to optional (back to "suggested" in Debian dependency parlance).

It isn't a bug, but intended behavior.  If we want to remove this
dependency, some non-trivial ideas about reworking the current load
procedure should emerge.  I don't thin we have any such ideas at this
time.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: monnier@iro.umontreal.ca,  rlb@defaultvalue.org,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Sun, 02 Oct 2022 19:38:01 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So there's a net win in having it enabled at all times.
> 
> You may think so, and I may think so, but it should be up to the user
> (and the distribution).

If they present their case clearly, we might change our minds.  If
they don't present the case clearly, I don't see how we can provide a
solution, because disabling JIT compilation is already possible.  So
someone should explain why what is there is not good enough, and that
requires details.

> > In any case, we already have a way of switching off JIT compilation,
> > at least for some situations which we considered.
> 
> I thought there wasn't a user option to switch it off?

There are several, actually, each one doing something slightly
different and disabling native compilation on a different level.
That's why I'm asking for details: we need to understand whether
something is missing and why.  For example, "disable native
compilation" may or may not be the same as "disable writing *.eln
files".

Re: Suppressing native compilation (short and long term)

[Rob Browning]

chad <yandros@gmail.com> writes:

> At the risk of over-explaining due to message crossing mid-flight: the
> thing you might be missing is that Debian provides a mechanism for people
> to install *.elc files in a space shared by all users that is not
> writable by those users, and there are people that use this provision.
> Since these are used for *.elc files, it seems highly likely that they will
> be desired for *.eln files.

To be clear, I think there are (at least) two concerns here.

First, from the Debian side, we may need some way to avoid writing files
(i.e. the .eln files) to certain locations during testing, installs,
etc.  That problem may be solved -- via the variable that's already been
mentioned.

Though note that for reasons we can elaborate on if desired, it might be
easier for us if the default for that variable could also be set via a
corresponding environment variable, but that's a separate question.

(For example, we have relevant emacs-related packages where the upstream
 build process runs emacs a level or two "down" (subprocess-wise), and
 so it'd be a lot less invasive if we could just set an environment
 variable to change the .eln destination, instead of having to figure
 out how to change each invocation of emacs in that package (and
 maintain those changes across future upstream versions).


A second, and a separable question, is whether Debian should try to
maintain system-level (root owned) .eln files the same way it does for
.elc files.

I consider that an open question, and it could well be that the answer
ends up being "no".  That's what we're trying to find out, and of course
we have to begin by trying to communicate why that might be desirable.

So here we are :)

It's certainly the case that if the consensus here (among the upstream
developers) is that we shouldn't do that, and that future
choices/changes aren't likely to take that use case into consideration,
then we have our answer.

And that's fine.  In the end, we just wanted to explain the potential
use case(s), and see how the developers felt about it.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

> A second, and a separable question, is whether Debian should try to
> maintain system-level (root owned) .eln files the same way it does for
> .elc files.

I think that makes sense.

Re: Suppressing native compilation (short and long term)

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> E.g. we could offer a configuration option that causes `package-install`
> to eagerly native compile the installed files.

Isn't that just `package-native-compile'?

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Theoretically, yes.  But I don't yet see why the particular context
> described by Rob is different to the degree that it would need special
> procedures.

While I'm not sure where we'll end up, I do think others may put more
weight on some of the concerns.  For example, across the broader
world/users-of-debian-and-derivative packages, I suspect there are some
who care more about storage space.

As mentioned elswhere, we get regular requests to make emacs-nox even
smaller.  And my eln-cache is currently 40MB, which is storage space
we'd thought we might not have to duplicate across the emacs users on a
system in the common case (Of course I know the duplication rate would
vary depending on which users use which things, but unless the sets were
disjoint, there'd be duplication that didn't seem necessary.)

But again, in the end, we just wanted to present the broader case for
consideration.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I'm not quite sure what you mean -- I'm saying that there should be a
> way for users to switch off native compilation (and for distributions to
> have native compilation for users switched off).
>
> Which was how this thread started: With a request for such a thing.

At the top level, we wanted a way to avoid writing to HOME during
packaging, testing, installs (in this case, it's the .eln files, now
that we've enabled native compilation).

That could be handled by some way to turn off native compilation, or by
some way to comprehensively divert those writes to another location
(e.g. temp dir).  Either is fine, though we'd originally thought the
former might make things a bit easier.

Whether or not we can (or should) try to build system-level (root owned)
.eln files along with the .elc files during package installs (as we
already do for .elc files) is a separate question, and I think the more
controversial one?

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> But Emacs should not "crash" if the *.el files aren't available, it
> should simply refuse to load any *.eln files and load the *.elc files
> instead.  That produces many warnings, of course, but I hope your
> users don't consider that "crashing".

I believe it was *crashing*.  I can't recall if that one was a segfault,
or something a bit less drastic, but I'll try to remember to track it
down later.

> It isn't a bug, but intended behavior.  If we want to remove this
> dependency, some non-trivial ideas about reworking the current load
> procedure should emerge.  I don't thin we have any such ideas at this
> time.

OK, so we should consider that a hard dependency now, i.e. the emacs-el
package can't be optional anymore, at least not on architectures where
we can enable native compilation, and so probably just "everywhere" for
simplicity, if nothing else.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> Though note that for reasons we can elaborate on if desired, it might be
> easier for us if the default for that variable could also be set via a
> corresponding environment variable, but that's a separate question.

Usually Emacs should obey $HOME so if you can set that to a tmp dir that
should let you avoid touching the user's real $HOME.

This said, there is some code in Emacs that prefers to use ~<user> over
$HOME in some cases (like `su` without the `-`).  I'm not sure exactly
where this happens, but I suspect that the following lines in
`startup.el` are part of the answer:

    ;; Figure out which user's init file to load,
    ;; either from the environment or from the options.
    (setq init-file-user (if noninteractive nil (user-login-name)))
    ;; If user has not done su, use current $HOME to find .emacs.
    (and init-file-user
         (equal init-file-user (user-real-login-name))
	 (setq init-file-user ""))


-- Stefan

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Stefan Kangas [2022-10-02 18:11:39] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> E.g. we could offer a configuration option that causes `package-install`
>> to eagerly native compile the installed files.
> Isn't that just `package-native-compile'?

Oh, it's there!
Wee!!


        Stefan

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> My recommendation is to use the default JIT manner until and unless
> actual problems are reported by users.

[...]

> There's no profit, IME.  There are only disadvantages: you are in
> effect fighting against the Emacs defaults, for reasons that are
> purely theoretical.

If I understand your meaning in both of these cases, I'll just note that
for the things we've been discussing here, I believe we've already had
complaints/requests from Debian users.  Whether that's significant
enough to warrant accommodation is another question, but that may not
leave the concerns theoretical, strictly speaking.

And for what it's worth, I can see both sides of the argument(s), i.e. I
can understand why upstream, it could be that on balance, those concerns
won't (and maybe shouldn't) be considered sufficient when balanced
against other considerations.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 12:57:54 -0500
> 
> Though note that for reasons we can elaborate on if desired, it might be
> easier for us if the default for that variable could also be set via a
> corresponding environment variable, but that's a separate question.
> 
> (For example, we have relevant emacs-related packages where the upstream
>  build process runs emacs a level or two "down" (subprocess-wise), and
>  so it'd be a lot less invasive if we could just set an environment
>  variable to change the .eln destination, instead of having to figure
>  out how to change each invocation of emacs in that package (and
>  maintain those changes across future upstream versions).

We could introduce such a variable, similarly to EMACSLOADPATH.  But
note several important considerations:

  . unlike with EMACSLOADPATH, the actual place where *.eln files will
    live is in a subdirectory of any directory in the list, due to the
    need of having them separately for different Emacs binaries and
    versions
  . EMACSLOADPATH is a mixed blessing: if you set it incorrectly, or
    forget that its value is inherited by subprocesses, you can
    completely hose an Emacs session, which is why we generally
    recommend against its use

> A second, and a separable question, is whether Debian should try to
> maintain system-level (root owned) .eln files the same way it does for
> .elc files.
> 
> I consider that an open question, and it could well be that the answer
> ends up being "no".  That's what we're trying to find out, and of course
> we have to begin by trying to communicate why that might be desirable.

Given the much more strict requirements for *.eln files wrt the target
architecture, certain crucial aspects of the Emacs binary, and the
contents and file name of the corresponding source file, my suggestion
is to start from "no" and only consider "yes" if you discover good
reasons through experience.  E.g., my eln-cache directory has no less
than 20 subdirectories, each one for a slightly different Emacs
version and configuration.  That might be somewhat extreme, given that
I work on development of and use several versions in parallel, but I
wouldn't be surprised to see several subdirectories for each of your
users, and even less surprised to see different subdirectories for
different users.  So I think the case for common compiled Lisp files
is much weaker for *.eln files than it is for *.elc files/

> It's certainly the case that if the consensus here (among the upstream
> developers) is that we shouldn't do that, and that future
> choices/changes aren't likely to take that use case into consideration,
> then we have our answer.

I only know that I didn't yet hear any good reason for rushing to
natively-compile optional Lisp packages.  That doesn't mean I'm dead
certain there could be no such good reasons, of course, just that I'd
like to see them described in enough detail to consider something
different from what was envisioned during Emacs 28 development.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: chad <yandros@gmail.com>,  Eli Zaretskii <eliz@gnu.org>,
>   tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 20:06:11 +0200
> 
> Rob Browning <rlb@defaultvalue.org> writes:
> 
> > A second, and a separable question, is whether Debian should try to
> > maintain system-level (root owned) .eln files the same way it does for
> > .elc files.
> 
> I think that makes sense.

Why do you think so?  A user who installs emacs-gtk and a user who
installs emacs-nox will need different *.eln files.  So Debian will
end up with a separate subdirectory for each configuration they
produce and for each Emacs version.  That's a lot of files and
directories, and high risk that many of them will never be used.
Also, a lot of problematic management of those directories. like the
decision when to delete them etc.  It is much easier to leave that to
users.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> No, I don't think the similar handling makes sense here.  The *.elc
> files are architecture- and configuration-independent, whereas the
> *.eln files are not.  E.g., the same foo.elc could be used by user A
> who runs Emacs 28 and by user B who runs Emacs 29.  But the
> corresponding *.eln files will be different, even though they were
> produced from the same foo.el.

Right, but for what it's worth, the Debian infrastructure is already set
up to, and would, maintain separate .eln files/trees for those two
cases, *if* we ever re-versioned the emacs packages.

Right now, there's only one GNU Emacs flavor in debian, we don't provide
versioned packages/flavors like emacs27 and emacs28 anymore.  Though we
could return to that again if it were ever deemed sufficiently valuable
to people.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> E.g., my eln-cache directory has no less than 20 subdirectories, each
> one for a slightly different Emacs version and configuration.

You're an Emacs developer, so that's to be expected.

But for normal users, the .eln files are neither more nor less specific
to an Emacs version than, say, the .pdmp file.  If Debian distributes a
specific Emacs version, it will be accompanied with the matching .pdmp
file -- and the matching .eln files, if that is what Debian decides to
do.

This is not complicated.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> I already did: the probability of different users to produce different
> *.eln files from the same *.el sources is high

As mentioned in another message, not in Debian's infrastructure.  It'd
be impossible (wrt the shared, system-level .elc and/or .eln files),
*if* we ended up deciding to pursue system-level .eln files.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: david@tethera.net, emacs-devel@gnu.org, akrl@sdf.org
> Date: Sun, 02 Oct 2022 13:12:01 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Theoretically, yes.  But I don't yet see why the particular context
> > described by Rob is different to the degree that it would need special
> > procedures.
> 
> While I'm not sure where we'll end up, I do think others may put more
> weight on some of the concerns.  For example, across the broader
> world/users-of-debian-and-derivative packages, I suspect there are some
> who care more about storage space.
> 
> As mentioned elswhere, we get regular requests to make emacs-nox even
> smaller.  And my eln-cache is currently 40MB, which is storage space
> we'd thought we might not have to duplicate across the emacs users on a
> system in the common case (Of course I know the duplication rate would
> vary depending on which users use which things, but unless the sets were
> disjoint, there'd be duplication that didn't seem necessary.)

IMO, this is actually an argument _against_ compiling everything AOT.

Whether the duplication is significant can only be decided based on
actual usage figures.  It is incorrect to assess this based on the
*.elc files, since those are independent of almost everything.
There's high probability of wrong decisions based on that analogy.
There are many factors that affect compatibility of *.eln files to
Emacs binaries; for example, it's enough to add or remove a primitive,
and you will need a whol;e new set of *.eln files.  Thus, it is quite
possible that duplication will be smaller and OTOH waste of disk space
due to unnecessarily compiled *.eln files will be higher than you
envision.  Only practice will show the real situation.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Why do you think so?  A user who installs emacs-gtk and a user who
> installs emacs-nox will need different *.eln files.

You can't install more than one of those at a time in Debian.  They're
mutually exclusive, at least right now, and have been since the variants
were added.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org,
>  akrl@sdf.org
> Date: Sun, 02 Oct 2022 13:21:54 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > But Emacs should not "crash" if the *.el files aren't available, it
> > should simply refuse to load any *.eln files and load the *.elc files
> > instead.  That produces many warnings, of course, but I hope your
> > users don't consider that "crashing".
> 
> I believe it was *crashing*.  I can't recall if that one was a segfault,
> or something a bit less drastic, but I'll try to remember to track it
> down later.

If it's a real crash, I'd appreciate bug reports, TIA.

> > It isn't a bug, but intended behavior.  If we want to remove this
> > dependency, some non-trivial ideas about reworking the current load
> > procedure should emerge.  I don't thin we have any such ideas at this
> > time.
> 
> OK, so we should consider that a hard dependency now, i.e. the emacs-el
> package can't be optional anymore, at least not on architectures where
> we can enable native compilation, and so probably just "everywhere" for
> simplicity, if nothing else.

More accurately, emacs-el cannot be optional when the installed Emacs
supports native compilation.  Emacs can still be built without native
compilation, and I presume those of your users who want the minimal
possible package will want that, since it removes several significant
dependencies, like libgccjit itself.  When Emacs is built without
native compilation, it can work without the *.el files.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> E.g., my eln-cache directory has no less than 20 subdirectories, each
>> one for a slightly different Emacs version and configuration.
>
> You're an Emacs developer, so that's to be expected.
>
> But for normal users, the .eln files are neither more nor less specific
> to an Emacs version than, say, the .pdmp file.  If Debian distributes a
> specific Emacs version, it will be accompanied with the matching .pdmp
> file -- and the matching .eln files, if that is what Debian decides to
> do.

With the current Debian arrangment, there would only ever be *one*
system-level .eln tree for the one installed Debian variant (emacs-nox,
emacs-lucid, or emacs-gtk).  Though of course the dir name hash might/would
change during each upgrade, but we already assume we have to rebuild all
the .elc files on each upgrade, and do (in dependency order), so that
shouldn't be a problem.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Whether the duplication is significant can only be decided based on
> actual usage figures.  It is incorrect to assess this based on the
> *.elc files, since those are independent of almost everything.
> There's high probability of wrong decisions based on that analogy.
> There are many factors that affect compatibility of *.eln files to
> Emacs binaries; for example, it's enough to add or remove a primitive,
> and you will need a whol;e new set of *.eln files.  Thus, it is quite
> possible that duplication will be smaller and OTOH waste of disk space
> due to unnecessarily compiled *.eln files will be higher than you
> envision.  Only practice will show the real situation.

I think I've probably made this clear elsewhere, but in case not, that
wouldn't be the case for Debian specifically.  There could/would only be
*one* eln tree for the whole system with the current packaging.

And that tree would be rebuilt when appropriate, in dependency order
(*if* we end up deciding to pursuse this).

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 13:32:02 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > My recommendation is to use the default JIT manner until and unless
> > actual problems are reported by users.
> 
> [...]
> 
> > There's no profit, IME.  There are only disadvantages: you are in
> > effect fighting against the Emacs defaults, for reasons that are
> > purely theoretical.
> 
> If I understand your meaning in both of these cases, I'll just note that
> for the things we've been discussing here, I believe we've already had
> complaints/requests from Debian users.  Whether that's significant
> enough to warrant accommodation is another question, but that may not
> leave the concerns theoretical, strictly speaking.

Please try to understand the nature of the complaints.  It is quite
possible that users simply use the (broken) analogy with the *.elc
files, because they misunderstand the quantitatively new aspects of
native-compilation.  There's more here than meets the eye, as has been
demonstrated even in this discussion.

Please don't hesitate to involve us in these discussions with your
complaining users, if you think we could help.

> And for what it's worth, I can see both sides of the argument(s), i.e. I
> can understand why upstream, it could be that on balance, those concerns
> won't (and maybe shouldn't) be considered sufficient when balanced
> against other considerations.

One other aspect that should be kept in mind is complexity.  The
introduction of the pdumper file in Emacs 27 and of native compilation
in Emacs 28 made Emacs deployment and startup code significantly more
complex, and as a side effect invalidated some of the deployment
methods people used to use, like some "clever" symlinking of the
binaries or the directories.  We are still learning these consequences
(although some of them already caused fixes in the code).  In this
situation, adding even more possible deployment method that upstream
should support risks making the startup code a maintenance burden.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 13:35:33 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > No, I don't think the similar handling makes sense here.  The *.elc
> > files are architecture- and configuration-independent, whereas the
> > *.eln files are not.  E.g., the same foo.elc could be used by user A
> > who runs Emacs 28 and by user B who runs Emacs 29.  But the
> > corresponding *.eln files will be different, even though they were
> > produced from the same foo.el.
> 
> Right, but for what it's worth, the Debian infrastructure is already set
> up to, and would, maintain separate .eln files/trees for those two
> cases, *if* we ever re-versioned the emacs packages.

Alas, there are many more than just two cases!

> Right now, there's only one GNU Emacs flavor in debian, we don't provide
> versioned packages/flavors like emacs27 and emacs28 anymore.

But a user who installs Emacs 28 doesn't need to nuke the Emacs 27
installation, does he?

And the same with a user who wants both emacs-nox, emacs-pgtk, and
emacs-lucid (say) installed on the same system.  Right?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Rob Browning <rlb@defaultvalue.org>,  yandros@gmail.com,
>   tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 20:37:20 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > E.g., my eln-cache directory has no less than 20 subdirectories, each
> > one for a slightly different Emacs version and configuration.
> 
> You're an Emacs developer, so that's to be expected.

As I wrote (and you elided), I don't expect to see 20 subdirectories
on other systems, but I wouldn't be surprised to see 5 or 10.

> But for normal users, the .eln files are neither more nor less specific
> to an Emacs version than, say, the .pdmp file.  If Debian distributes a
> specific Emacs version, it will be accompanied with the matching .pdmp
> file -- and the matching .eln files, if that is what Debian decides to
> do.

We are talking about optional packages, not about preloaded Lisp
files.  The preloaded ones indeed go together with the pdumper file,
but the optional ones don't.

> This is not complicated.

Famous last words.

Re: Suppressing native compilation (short and long term)

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Isn't that just `package-native-compile'?
>
> Oh, it's there!
> Wee!!

BTW, I think nil is the best default for that variable if and only if
`native-comp-async-report-warnings-errors' also defaults to nil.
Otherwise, users will have a rather scary warning buffer pop up in
the middle of trying to do work (or playing `M-x pong', more likely).

Maybe we could revisit the latter in time for Emacs 29, or at the latest
before switching native compilation to on by default (Emacs 30?).

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 13:37:49 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I already did: the probability of different users to produce different
> > *.eln files from the same *.el sources is high
> 
> As mentioned in another message, not in Debian's infrastructure.  It'd
> be impossible (wrt the shared, system-level .elc and/or .eln files),
> *if* we ended up deciding to pursue system-level .eln files.

Which IMO a significant limitation that is the direct result of such a
decision.  Right?  Whereas leaving it up to the users to produce *.el
in the JIT manner lifts this limitation.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: yandros@gmail.com, tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 13:41:32 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Why do you think so?  A user who installs emacs-gtk and a user who
> > installs emacs-nox will need different *.eln files.
> 
> You can't install more than one of those at a time in Debian.  They're
> mutually exclusive, at least right now, and have been since the variants
> were added.

Even if we are talking about two different users on the same system?
IOW, this is a system-wide restriction?  Isn't that too harsh?

And what about users who make changes to Emacs -- is that a legitimate
use case supported by Debian installations?

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> We are talking about optional packages, not about preloaded Lisp
> files.  The preloaded ones indeed go together with the pdumper file,
> but the optional ones don't.

I'm not sure what you mean by "optional packages" -- we're talking about
.eln files for the .elc files in the Emacs distribution.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: rlb@defaultvalue.org,  yandros@gmail.com,  tomas@tuxteam.de,
>   emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 21:01:53 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We are talking about optional packages, not about preloaded Lisp
> > files.  The preloaded ones indeed go together with the pdumper file,
> > but the optional ones don't.
> 
> I'm not sure what you mean by "optional packages" -- we're talking about
> .eln files for the .elc files in the Emacs distribution.

No, we are not.  We are talking about unbundled packages, like Magit.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

> At the top level, we wanted a way to avoid writing to HOME during
> packaging, testing, installs (in this case, it's the .eln files, now
> that we've enabled native compilation).
>
> That could be handled by some way to turn off native compilation, or by
> some way to comprehensively divert those writes to another location
> (e.g. temp dir).  Either is fine, though we'd originally thought the
> former might make things a bit easier.

Yeah, I think the former is both easier to implement and easier for
users.

So I'm thinking of introducing a user option like
`native-compile-inhibit', which will make Emacs skip the native-comp
machinery when loading .elc files.  It will default to nil, of course,
but perhaps it would be convenient to make it depend on an environment
variable like "NATIVE_COMPILE_INHIBIT"?  Then users (and the Debian
build system) could say "NATIVE_COMPILE_INHIBIT=true emacs ..." when
doing testing etc?  Would that fit your use case?

(This will be for Emacs 29, but you can cherry-pick this for Debian, if
that's something you want to do, and it will probably not affect
trampolines, since those are necessary for redefining functions.)

> Whether or not we can (or should) try to build system-level (root owned)
> .eln files along with the .elc files during package installs (as we
> already do for .elc files) is a separate question, and I think the more
> controversial one?

Not controversial from where I'm, er, sitting -- sounds both useful and
convenient to me.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  Eli Zaretskii
>  <eliz@gnu.org>,  david@tethera.net,  emacs-devel@gnu.org,  akrl@sdf.org
> Date: Sun, 02 Oct 2022 21:08:15 +0200
> 
> Rob Browning <rlb@defaultvalue.org> writes:
> 
> > At the top level, we wanted a way to avoid writing to HOME during
> > packaging, testing, installs (in this case, it's the .eln files, now
> > that we've enabled native compilation).
> >
> > That could be handled by some way to turn off native compilation, or by
> > some way to comprehensively divert those writes to another location
> > (e.g. temp dir).  Either is fine, though we'd originally thought the
> > former might make things a bit easier.
> 
> Yeah, I think the former is both easier to implement and easier for
> users.

The request that started this discussion was not from users, it was
from distributors.

If we want to consider providing (yet another) user option for
disabling native compilation, then we should:

  . understand why and in which situations they may need it
  . what exactly needs to be disabled (e.g.: do you want to disable
    loading the existing *.eln files?)
  . understand why the existing options don't already provide
    solutions

I object to introducing new options before we do the above and
understand well what we are talking about.

> So I'm thinking of introducing a user option like
> `native-compile-inhibit', which will make Emacs skip the native-comp
> machinery when loading .elc files.  It will default to nil, of course,
> but perhaps it would be convenient to make it depend on an environment
> variable like "NATIVE_COMPILE_INHIBIT"?  Then users (and the Debian
> build system) could say "NATIVE_COMPILE_INHIBIT=true emacs ..." when
> doing testing etc?  Would that fit your use case?

Their use case is not the use case of Emacs users.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> But a user who installs Emacs 28 doesn't need to nuke the Emacs 27
> installation, does he?

Yes, they do.  When the Debian emacs pacakges move from a 27.x version
to a 28.x version, upgrading will just move you to 28.  You can't have
more than one version of emacs installed via the packages right now.

(You could in the past (we had emacs25-nox, etc.), but we "unversioned"
 them for now because, in part, upstream backward compatbility with
 respect to anything "substantial" has been excellent for a long time.)

> And the same with a user who wants both emacs-nox, emacs-pgtk, and
> emacs-lucid (say) installed on the same system.  Right?

That's not allowed (and that in particular never has been in Debian and
(I assume) derivatives).  e.g. installing emacs-nox will automatically
force out/replace emacs-lucid if the former was installed, and then all
of the .elcs (and .elns if we go that direction) will be automatically
regenerated during the switch.

-- 
Rob Browning rlb @defaultvalue.org and @debian.org GPG as of 2011-07-10
E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A GPG as of 2002-11-03
14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Even if we are talking about two different users on the same system?
> IOW, this is a system-wide restriction?  Isn't that too harsh?

The available Debian packages are a balance, intended to cover a broad
set of common cases, i.e. Emacs without X, Emacs with the Lucid toolkit
(because of, if nothing else, gtk issues), and Emacs with the GTK
toolkit.

You can only have one of them installed at a time, and you can
(currently) only have one major version installed at a time.

  https://packages.debian.org/search?keywords=emacs

We could of course try to accommodate multiple major versions (we did
for a good while), and/or multiple simultaneous variants (nox, lucid,
gtk), but we'd need to feel like the additional complexity and archive
space (multiplied across the architecture-dependent packages
(emacs-bin-common, etc.)[1]) was worth it for a large enough audience.

  [1] https://buildd.debian.org/status/package.php?p=emacs

> And what about users who make changes to Emacs -- is that a legitimate
> use case supported by Debian installations?

I'd say that up to a point you can, and I have symlinked the relevant
.el files into a ~/ directory, made sure that it's in my load-path, and
then made adjustments, but past a certain point, I'd say that you'd want
to switch to building Emacs yourself.

Because at that point, you're perhaps no longer in the target audience
for the Debian packages, or at least non on that particular machine
(i.e. you might be fine with the Debian packages on most of your
machines, servers, etc. but want to make much more extensive changes to
Emacs on some others).

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Please try to understand the nature of the complaints.

For what it's worth, one of the complaints was:

  I do not want emacs suddenly swamping the CPU on my laptop
  unpredictably.  I want all this work (in the common case) done while I
  know I'm connected to power, before I head out.

...and we have for a very long time, perhaps as long as I've been
working with the packaging, had users who are concerned with disk space
usage, and ask for smaller package footprints.

Those concerns may not carry the day, but they do exist.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> We are talking about optional packages, not about preloaded Lisp
> files.  The preloaded ones indeed go together with the pdumper file,
> but the optional ones don't.

Debian always recompiles separately all the .el files in third party
packages (installed vi Debian) for all the installed Emacsen.
IOW if you have `emacs24`, `xemacs`, and `emacs25` installed, you'll
have 3 sets of .elc files for every ELisp package installed via Debian's
package manager.

So the `.eln` files aren't really more specific than the `.elc` in
that setting (except for the situation where you have `emacs-lucid` and
`emacs-gtk` and `emacs-nox` installed where these (unnecessarily)
require different `.eln` files but will share the same `elc` files).


        Stefan

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> Which IMO a significant limitation that is the direct result of such a
> decision.  Right?  Whereas leaving it up to the users to produce *.el
> in the JIT manner lifts this limitation.

Maybe my other mail about the packaging tradeoffs helps here, but if
not, I can try to elaborate further.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So I'm thinking of introducing a user option like
> `native-compile-inhibit', which will make Emacs skip the native-comp
> machinery when loading .elc files.  It will default to nil, of course,
> but perhaps it would be convenient to make it depend on an environment
> variable like "NATIVE_COMPILE_INHIBIT"?  Then users (and the Debian
> build system) could say "NATIVE_COMPILE_INHIBIT=true emacs ..." when
> doing testing etc?  Would that fit your use case?

It might?  I'd also suggest adding an EMACS_... prefix
to the variable name (of course I'd suggest that for all of any
program's variables).

Though I'm not positive because if we still really need compilation for
the trampolines(?), then we may need to just follow the "redirect to a
tempdir" approach, which is also fine, and then the ability to disable
native compilation might not be relevant to the packaging.

In either case, we might ideally want an EMACS_... var to make
propagation a bit easier if that sounds reasonable.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> So the `.eln` files aren't really more specific than the `.elc` in
> that setting (except for the situation where you have `emacs-lucid` and
> `emacs-gtk` and `emacs-nox` installed where these (unnecessarily)
> require different `.eln` files but will share the same `elc` files).

Apparently that exception doesn't apply either, since Debian doesn't
allow simultaneously installing those different flavors :-)
[ Funny, I never noticed it in all those years using Debian.  ]


        Stefan

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

> Though I'm not positive because if we still really need compilation for
> the trampolines(?), then we may need to just follow the "redirect to a
> tempdir" approach, which is also fine, and then the ability to disable
> native compilation might not be relevant to the packaging.

I'm not sure we need to save the trampolines at all (in this
don't-do-native-compilation configuration) -- Andrea probably knows.
Andrea, would it be possible to just create and use the trampolines
without writing them to a file?

> In either case, we might ideally want an EMACS_... var to make
> propagation a bit easier if that sounds reasonable.

Sure.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

On Sun, Oct 02, 2022 at 08:01:58PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 2 Oct 2022 18:27:05 +0200
> > From: tomas@tuxteam.de
> > Cc: emacs-devel@gnu.org

[...]

> > Hm. I don't want to steal your time more, but... if you could illustrate
> > why, I'd eager to learn.
> 
> I already did: the probability of different users to produce different
> *.eln files from the same *.el sources is high, and disk space is
> cheap.

Thanks

Cheers
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Sun 02 Oct 2022 at 08:57AM +03, Eli Zaretskii wrote:

> The reasons which you mention in favor of AOT native compilation don't
> sound serious enough: I see no problem in having the compilation
> happen only when it's needed in those cases.  Battery consumption
> doesn't seem very relevant, because JIT compilation will happen when
> the system is used, not when it sleeps.  And it is entirely not
> guaranteed that by compiling everything you will save power in the
> long run, because there are good chances you will be compiling stuff
> that won't be used for a long time.  Without quantitative data of
> long-term power usage on which to base the conclusions, I don't see
> why you should a-priori assume that compiling everything from the
> get-go should use less power.  Same goes for disk space by multiple
> users.

The point with battery consumption is not about running vs. standby.
The issue is that while users expect that running apt-get will drain the
battery, they expect that once apt-get is done, the only battery-hungry
processes are ones they start themselves.  Laptop users typically avoid
running apt-get without mains power plugged in.  If I do an 'apt-get
upgrade' then afterwards my CPU will be churning away compiling addon
packages, so I can't just unplug.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Sun 02 Oct 2022 at 12:53PM -04, Stefan Monnier wrote:

> FWIW, I'm not convinced it's really useful in Debian's `emacs` package
> to eagerly native compile all the bundled .elc files, but I think it
> does make a lot of sense for ELisp packages installed separately.

Could you elaborate on this point?

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Sun 02 Oct 2022 at 07:11PM +03, Eli Zaretskii wrote:

> I submit that those reasons were most probably derived from a broken
> analogy with the *.elc files and with byte-compilation in general.
> Not from actual usage experience.  Native compilation looks
> deceptively similar to byte compilation, but it isn't.  So if
> producing the *.eln files seems to contradict some Debian rules and
> procedures, my suggestion is to talk to the upstream project, before
> inventing solutions, because of 2 considerations:
>
>   . the problems may not be real ones, only perceived ones
>   . the upstream codebase might already provide solutions

The battery usage thing is real, from my own user experience.

I have to remember not to unplug my laptop until the load indicator in
my status bar has dropped back down, else the battery won't last as long
as I expect it to.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> If we want to consider providing (yet another) user option for
> disabling native compilation, then we should:

I asked what the user option to disable native compilation was, but
didn't get an answer, and here you say "(yet another)", so...  what is
the current user option to disable native compilation?

>   . understand why and in which situations they may need it

Doing repeatable testing is one obvious situation.

>   . what exactly needs to be disabled (e.g.: do you want to disable
>     loading the existing *.eln files?)

I don't think anybody has suggested the "e.g." portion, so I'm wondering
why you ask?

>   . understand why the existing options don't already provide
>     solutions

See first paragraph.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I asked what the user option to disable native compilation was, but
> didn't get an answer, and here you say "(yet another)", so...  what is
> the current user option to disable native compilation?

Looking at the code, I can't see any user options to disable automatic
compilation of .elc files.  However, there's the confusingly named
`native-comp-deferred-compilation' variable which does the trick.

And looking at this a bit more, having a user option for this won't
quite work, because user options are set in .emacs, and .emacs may
trigger loading stuff before handling user options, so it'll have to
remain a variable.

But should perhaps be renamed to something more understandable, or be
deprecated in favour of a `inhibit-native-compilation' variable?

In any case, initialising this variable from an environment variable
like EMACS_INHIBIT_NATIVE_COMPILATION seems totally trivial.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

AFAICT so far I've heard of two reasons to "disable automatic native
compilation":

A) Avoid the "uncontrolled" (mostly in terms of *when* it happens)
   resource usage associated with it (e.g. battery).

B) Avoid writing to $HOME.

For (A), I think that (setq native-comp-deferred-compilation nil) seems
perfectly sufficient (you can set it early in your (early) init file and if
every blue moon some native compilation still happens because of a file
loaded before that `setq` it shouldn't be a big deal).

For (B) I seem to remember that it's a problem that goes a bit further
than just native compilation (we've had bug reports about other files
being silently written to ~/.emacs.d in the past and I'd be surprised if
there aren't still many such cases, especially if we consider third
party packages).  So if avoiding all files under $HOME is important,
I suspect the only reliable answer is to set $HOME elsewhere (and if
you run Emacs under `su`, make sure you use `su -`).

This said, we generally do make efforts to try and avoid writing
silently to ~/.emacs.d, so maybe we should rethink our choice of
~/.emacs.d/eln-cache as the default location.
E.g. maybe when run as root, we should write .eln files somewhere under
something like /var/cache?  And maybe when not run as root we should
favor something underneath ~/.cache?


        Stefan "who finds the idea of running Emacs as root a bit scary"

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> AFAICT so far I've heard of two reasons to "disable automatic native
> compilation":
>
> A) Avoid the "uncontrolled" (mostly in terms of *when* it happens)
>    resource usage associated with it (e.g. battery).
>
> B) Avoid writing to $HOME.
>
> For (A), I think that (setq native-comp-deferred-compilation nil) seems
> perfectly sufficient (you can set it early in your (early) init file and if
> every blue moon some native compilation still happens because of a file
> loaded before that `setq` it shouldn't be a big deal).

I've now introduced the new inhibit-native-compilation variable and
deprecated the native-comp-deferred-compilation variable, and also
initialised stuff from EMACS_INHIBIT_NATIVE_COMPILATION.

(And made trampolines not write to the cache in this case.)

> This said, we generally do make efforts to try and avoid writing
> silently to ~/.emacs.d, so maybe we should rethink our choice of
> ~/.emacs.d/eln-cache as the default location.

I think that's a fine location in general, and now there's an
understandable and documented way to make that not happen, I think it's
even better.

> E.g. maybe when run as root, we should write .eln files somewhere under
> something like /var/cache?  And maybe when not run as root we should
> favor something underneath ~/.cache?
>
>         Stefan "who finds the idea of running Emacs as root a bit scary"

I run Emacs as root all the time.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Lars Ingebrigtsen [2022-10-03 15:29:40] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> AFAICT so far I've heard of two reasons to "disable automatic native
>> compilation":
>>
>> A) Avoid the "uncontrolled" (mostly in terms of *when* it happens)
>>    resource usage associated with it (e.g. battery).
>>
>> B) Avoid writing to $HOME.
>>
>> For (A), I think that (setq native-comp-deferred-compilation nil) seems
>> perfectly sufficient (you can set it early in your (early) init file and if
>> every blue moon some native compilation still happens because of a file
>> loaded before that `setq` it shouldn't be a big deal).
>
> I've now introduced the new inhibit-native-compilation variable and
> deprecated the native-comp-deferred-compilation variable, and also
> initialised stuff from EMACS_INHIBIT_NATIVE_COMPILATION.
> (And made trampolines not write to the cache in this case.)

AFAICT for the case (A), we *do* want to save trampolines for the next
time around, and those users probably do want to use native compilation
(e.g. they would likely set `package-native-compile` to non-nil), just
not the deferred kind, so the var name is a bit odd for them.


        Stefan

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> AFAICT for the case (A), we *do* want to save trampolines for the next
> time around, and those users probably do want to use native compilation
> (e.g. they would likely set `package-native-compile` to non-nil), just
> not the deferred kind, so the var name is a bit odd for them.

The trampolines are very fast to make, so creating them once per Emacs
session doesn't seem like a problem.  If that turns out to be an issue,
we can tweak the variable.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Lars Ingebrigtsen [2022-10-03 16:09:35] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> AFAICT for the case (A), we *do* want to save trampolines for the next
>> time around,
>
> The trampolines are very fast to make, so creating them once per Emacs
> session doesn't seem like a problem.  If that turns out to be an issue,
> we can tweak the variable.

Yes, it's probably not a big deal indeed.

But this one is more annoying I think:

>> and those users probably do want to use native compilation
>> (e.g. they would likely set `package-native-compile` to non-nil), just
>> not the deferred kind, so the var name is a bit odd for them.

`inhibit-native-compilation` sounds like it really prevents all forms of
native compilation, whether "deferred" or not.

E.g. going by its name, I'd argue that it's a bug if package.el does
native-compile the files when both `package-native-compile` when
`inhibit-native-compilation` are non-nil.


        Stefan

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> and those users probably do want to use native compilation
>>> (e.g. they would likely set `package-native-compile` to non-nil), just
>>> not the deferred kind, so the var name is a bit odd for them.
>
> `inhibit-native-compilation` sounds like it really prevents all forms of
> native compilation, whether "deferred" or not.

We can expand the variable name, but I couldn't think of a good name.
And "deferred" doesn't convey anything.

> E.g. going by its name, I'd argue that it's a bug if package.el does
> native-compile the files when both `package-native-compile` when
> `inhibit-native-compilation` are non-nil.

But not by going by the documentation of the variable.  😀

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Lars Ingebrigtsen [2022-10-03 16:45:22] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>> and those users probably do want to use native compilation
>>>> (e.g. they would likely set `package-native-compile` to non-nil), just
>>>> not the deferred kind, so the var name is a bit odd for them.
>> `inhibit-native-compilation` sounds like it really prevents all forms of
>> native compilation, whether "deferred" or not.
> We can expand the variable name, but I couldn't think of a good name.
> And "deferred" doesn't convey anything.

Lazy?  Auto(matic)?  Ondemand?

>> E.g. going by its name, I'd argue that it's a bug if package.el does
>> native-compile the files when both `package-native-compile` when
>> `inhibit-native-compilation` are non-nil.
> But not by going by the documentation of the variable.  😀

Sorry, my dictionary doesn't know that word.  What's "documentation"?
Does it have to do with some Norse mythology, maybe?


        Stefan

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> We can expand the variable name, but I couldn't think of a good name.
>> And "deferred" doesn't convey anything.
>
> Lazy?  Auto(matic)?  Ondemand?

`inhibit-lazy-native-compilation' sounds like it makes the compilation
less lazy.  `inhibit-automatic-native-compilation' might work.

> Sorry, my dictionary doesn't know that word.  What's "documentation"?
> Does it have to do with some Norse mythology, maybe?

I think the concept was invented by Loki, the god of mischief and lies.

RE: [External] : Re: Suppressing native compilation (short and long term)

[Drew Adams]

> Sorry, my dictionary doesn't know that word.  What's "documentation"?
> Does it have to do with some Norse mythology, maybe?

Doctor U. Mentation
___

Noun: mentation

The process of using your mind to consider something carefully.

 - WordWeb

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 14:57:49 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Please try to understand the nature of the complaints.
> 
> For what it's worth, one of the complaints was:
> 
>   I do not want emacs suddenly swamping the CPU on my laptop
>   unpredictably.  I want all this work (in the common case) done while I
>   know I'm connected to power, before I head out.
> 
> ...and we have for a very long time, perhaps as long as I've been
> working with the packaging, had users who are concerned with disk space
> usage, and ask for smaller package footprints.

But Emacs does that all the time: there are many features that invoke
sub-processes and many more features that write to the disk.  I never
heard anyone complaining seriously about that, and I'm quite sure many
users don't even know which Emacs commands invoke subprocesses under
the hood.

So I'm not sure these complaints are based on real problems.  Did
anyone compare the "sudden swamp of the CPU" caused by JIT native
compilation with what happens with other commands that invoke
subprocesses?  If so, did they present some quantitative data?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: Rob Browning <rlb@defaultvalue.org>,  monnier@iro.umontreal.ca,
>   david@tethera.net,  emacs-devel@gnu.org,  akrl@sdf.org
> Date: Sun, 02 Oct 2022 16:51:12 -0700
> 
> The point with battery consumption is not about running vs. standby.
> The issue is that while users expect that running apt-get will drain the
> battery, they expect that once apt-get is done, the only battery-hungry
> processes are ones they start themselves.

How does this work in practice?  I mean, what exactly do you mean by
"processes they start themselves"?  You mean, you know by heart each
Emacs command that invokes a subprocess, and avoid to invoke them all
unless the laptop is plugged in?  I don't believe this is practical,
because even "C-x C-f" invokes a a subprocess when you visit a file
under VCS (which I guess happens a lot to the likes of you and me).
And how can a person avoid all of those commands?  What am I missing?

> Laptop users typically avoid running apt-get without mains power
> plugged in.  If I do an 'apt-get upgrade' then afterwards my CPU
> will be churning away compiling addon packages, so I can't just
> unplug.

Did you actually try to see that "churning away" in practice, did you
look at the time this typically takes, the resources (battery and
otherwise) it consumes, etc.?  If so, can you share the data: how many
*.eln files were produced in how many minutes, how much battery did
that consume?

You see, I have my data, and it seems to indicate a very short period
of initial compilations for a modest consumption of resources.  (This
isn't a laptop, but I'm acutely aware of my system's load at all
times, and have it on the mode line, so I know what kind of
computation is going on during that time.)  The data I see here
doesn't look like it should be a problem for anyone, as long as files
are compiled only as they are used. In my case, for example, I see
maybe half a dozen *.eln files compiled after the initial startup.  I
expect to see that on any machine where a user has steady usage
patterns -- the compilation flats out very quickly.  I cannot imagine
how will your CPU churn out for any significant time after an upgrade.
A typical compilation of a single .el file is usually very fast,
exceptions are extremely rare.  And I barely notice the effect of that
on system responsiveness.

So I could imagine that many people perhaps base these judgments on
something they didn't actually try, but just imagined to themselves,
and imagined incorrectly, or based their opinions on some other
compilations of different languages in different environments.

I urge people to actually try this and see what it does, before making
up your minds.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 16:58:34 -0700
> 
> The battery usage thing is real, from my own user experience.
> 
> I have to remember not to unplug my laptop until the load indicator in
> my status bar has dropped back down, else the battery won't last as long
> as I expect it to.

When I start an Emacs session for a new version (which needs to
recompile all the packages I use, because my sessions are restored by
desktop.el), I see about 100 Lisp files compiled into *.eln inside of
1 minute, and then another bunch of 100 inside another minute (with 10
minutes between these two groups).  Does this strike you as a long
time to avoid unplugging?  I hope your battery can hold more than a
few minutes.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> But Emacs does that all the time: there are many features that invoke
> sub-processes and many more features that write to the disk.  I never
> heard anyone complaining seriously about that, and I'm quite sure many
> users don't even know which Emacs commands invoke subprocesses under
> the hood.

FWIW, during the stealth jit-lock discussion, several people mentioned the
battery impact.

And the issue is not subprocesses per se, but it's extra processing that
happens outside of the control of the user.

> So I'm not sure these complaints are based on real problems.  Did
> anyone compare the "sudden swamp of the CPU" caused by JIT native
> compilation with what happens with other commands that invoke
> subprocesses?  If so, did they present some quantitative data?

Probably not, no.  It's likely mostly a question of perception, so if we
could make it completely invisible the "problem" would disappear :-)
But the fact that lazy native compilation tends to pop up warnings (and
to make matters worse, it does so ... without warning) makes it very
much visible instead.


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Mon, 03 Oct 2022 15:29:40 +0200
> 
> I've now introduced the new inhibit-native-compilation variable and
> deprecated the native-comp-deferred-compilation variable, and also
> initialised stuff from EMACS_INHIBIT_NATIVE_COMPILATION.
> 
> (And made trampolines not write to the cache in this case.)

Thanks a lot for ignoring everything I wrote during the last two days!

Makes one wonder what kind of development community this is.  And what
am I doing here, anyway.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Mon, 03 Oct 2022 16:09:35 +0200
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > AFAICT for the case (A), we *do* want to save trampolines for the next
> > time around, and those users probably do want to use native compilation
> > (e.g. they would likely set `package-native-compile` to non-nil), just
> > not the deferred kind, so the var name is a bit odd for them.
> 
> The trampolines are very fast to make, so creating them once per Emacs
> session doesn't seem like a problem.  If that turns out to be an issue,
> we can tweak the variable.

I think that entire changeset should be reverted.  It is not well
thought out, and ion some aspects downright dangerous.  The
environment variable is especially egregious: it will affect all the
sub-process Emacsen as well, something that will bite users, a lesson
we've learned long ago with the likes of EMACSLOADPATH.

Such changes should not be installed without a great deal of careful
thought and consideration of the different use cases.  Which I tried
to explain all the way, but apparently to deaf ears.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Rob Browning <rlb@defaultvalue.org>,  tomas@tuxteam.de,
>   emacs-devel@gnu.org
> Date: Mon, 03 Oct 2022 12:39:44 -0400
> 
> > But Emacs does that all the time: there are many features that invoke
> > sub-processes and many more features that write to the disk.  I never
> > heard anyone complaining seriously about that, and I'm quite sure many
> > users don't even know which Emacs commands invoke subprocesses under
> > the hood.
> 
> FWIW, during the stealth jit-lock discussion, several people mentioned the
> battery impact.

Yes.  And jit-stealth is different, in that it takes a much longer
time for it to become quiet, because it works in small chinks, and
only when Emacs is actually idle.  JIT native-compilation is much
faster, and also uses several execution units of the CPU in parallel.

> And the issue is not subprocesses per se, but it's extra processing that
> happens outside of the control of the user.

How do you mean "outside of the control of the user"?  The user causes
it by loading a feature.  If no new features are loaded, no
native-compilation will happen.  How is that different from any other
command that uses a subprocess under the hood?

> > So I'm not sure these complaints are based on real problems.  Did
> > anyone compare the "sudden swamp of the CPU" caused by JIT native
> > compilation with what happens with other commands that invoke
> > subprocesses?  If so, did they present some quantitative data?
> 
> Probably not, no.  It's likely mostly a question of perception, so if we
> could make it completely invisible the "problem" would disappear :-)
> But the fact that lazy native compilation tends to pop up warnings (and
> to make matters worse, it does so ... without warning) makes it very
> much visible instead.

On my system, I don't see any warnings.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I think that entire changeset should be reverted.  It is not well
> thought out, and ion some aspects downright dangerous.  The
> environment variable is especially egregious: it will affect all the
> sub-process Emacsen as well, something that will bite users, a lesson
> we've learned long ago with the likes of EMACSLOADPATH.

How is setting EMACS_INHIBIT_NATIVE_COMPILE dangerous?

> Such changes should not be installed without a great deal of careful
> thought and consideration of the different use cases.  Which I tried
> to explain all the way, but apparently to deaf ears.

The discussion has been going for apparently weeks no with no progress.
It's easier to discuss code when there's code to discuss.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

[Full disclosure: evidently, none of what I write below matters, so
feel free to ignore.]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: larsi@gnus.org, yandros@gmail.com, tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Sun, 02 Oct 2022 14:50:58 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Even if we are talking about two different users on the same system?
> > IOW, this is a system-wide restriction?  Isn't that too harsh?
> 
> The available Debian packages are a balance, intended to cover a broad
> set of common cases, i.e. Emacs without X, Emacs with the Lucid toolkit
> (because of, if nothing else, gtk issues), and Emacs with the GTK
> toolkit.
> 
> You can only have one of them installed at a time, and you can
> (currently) only have one major version installed at a time.
> 
>   https://packages.debian.org/search?keywords=emacs

That's strange to hear.  Even MS-Windows allows per-user variations of
PATH and per-use environment variables.  It is strange to learn that
Debian doesn't.

> > And what about users who make changes to Emacs -- is that a legitimate
> > use case supported by Debian installations?
> 
> I'd say that up to a point you can, and I have symlinked the relevant
> .el files into a ~/ directory, made sure that it's in my load-path, and
> then made adjustments, but past a certain point, I'd say that you'd want
> to switch to building Emacs yourself.

If you want to support even just installing a different minor version,
it will already require to recompile all of the *.eln files.

But okay, yes, if Debian users live under such severe restrictions,
then the case for having user-specific *.eln directories becomes
weaker.  But I still don't see that it is weaker than disallowing
native compilation at run time.

And I guess now I'm confused what is it exactly that you'd want to
achieve.  Do you want to disable native compilation, or do you want to
have all *.eln files in a shared location?  Because it seems to me you
said you wanted both, and I don't see how both could be reconciled.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: rlb@defaultvalue.org,  monnier@iro.umontreal.ca,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Mon, 03 Oct 2022 11:12:22 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If we want to consider providing (yet another) user option for
> > disabling native compilation, then we should:
> 
> I asked what the user option to disable native compilation was, but
> didn't get an answer, and here you say "(yet another)", so...  what is
> the current user option to disable native compilation?

You haven't bothered waiting for my answer, so I won't.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: monnier@iro.umontreal.ca,  rlb@defaultvalue.org,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Mon, 03 Oct 2022 19:39:04 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think that entire changeset should be reverted.  It is not well
> > thought out, and ion some aspects downright dangerous.  The
> > environment variable is especially egregious: it will affect all the
> > sub-process Emacsen as well, something that will bite users, a lesson
> > we've learned long ago with the likes of EMACSLOADPATH.
> 
> How is setting EMACS_INHIBIT_NATIVE_COMPILE dangerous?

I just explained it, above.

> > Such changes should not be installed without a great deal of careful
> > thought and consideration of the different use cases.  Which I tried
> > to explain all the way, but apparently to deaf ears.
> 
> The discussion has been going for apparently weeks no with no progress.

"Weeks"?  Boy, I guess time really flies when one's enjoying: the
first message of this thread was posted less than a week ago:

  https://lists.gnu.org/archive/html/emacs-devel/2022-09/msg01911.html

And it really started just 2 days ago:

  https://lists.gnu.org/archive/html/emacs-devel/2022-10/msg00015.html

But whatever.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1440 bytes --]

On Mon, Oct 03, 2022 at 08:41:22PM +0300, Eli Zaretskii wrote:
> [Full disclosure: evidently, none of what I write below matters, so
> feel free to ignore.]

Now I think this is a bit unfair. You have a string opinion, and
I think it shhould be respected, but the others have, too.


> > You can only have one of them installed at a time, and you can
> > (currently) only have one major version installed at a time.
> > 
> >   https://packages.debian.org/search?keywords=emacs
> 
> That's strange to hear.  Even MS-Windows allows per-user variations of
> PATH and per-use environment variables.  It is strange to learn that
> Debian doesn't.

Note that this is only as far as the Debian packaging system is
concerned. Per-user you can have as many Emacsen installed as you
want -- the Debian packaging system doesn't know about them.

Alternatively, you can have a system-local install (going to
/usr/local) besides the Debian-installed one without them interfering
(that's the setup I use).

This is actually what I love in Debian: you have single applications
you care about which you install from souces and take over the hand
feeding, and for the rest you get hight quality packaging with pretty
low noise.

Emacs, for me, happens to be one of those care-and-feeding applications,
but for others, the choice might be different. It's them whom the
Debian-packaged Emacsen are for.

Cheers
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Mon 03 Oct 2022 at 09:07AM -04, Stefan Monnier wrote:

> AFAICT so far I've heard of two reasons to "disable automatic native
> compilation":
>
> A) Avoid the "uncontrolled" (mostly in terms of *when* it happens)
>    resource usage associated with it (e.g. battery).
>
> B) Avoid writing to $HOME.
>
> For (A), I think that (setq native-comp-deferred-compilation nil) seems
> perfectly sufficient (you can set it early in your (early) init file and if
> every blue moon some native compilation still happens because of a file
> loaded before that `setq` it shouldn't be a big deal).

We'd also like to be able to compile and install system-wide -- it would
be good to be able to benefit from native compilation without it
happening by surrpise.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Mon 03 Oct 2022 at 07:19PM +03, Eli Zaretskii wrote:

>> From: Sean Whitton <spwhitton@spwhitton.name>
>> Cc: Rob Browning <rlb@defaultvalue.org>,  monnier@iro.umontreal.ca,
>>   david@tethera.net,  emacs-devel@gnu.org,  akrl@sdf.org
>> Date: Sun, 02 Oct 2022 16:51:12 -0700
>>
>> The point with battery consumption is not about running vs. standby.
>> The issue is that while users expect that running apt-get will drain the
>> battery, they expect that once apt-get is done, the only battery-hungry
>> processes are ones they start themselves.
>
> How does this work in practice?  I mean, what exactly do you mean by
> "processes they start themselves"?  You mean, you know by heart each
> Emacs command that invokes a subprocess, and avoid to invoke them all
> unless the laptop is plugged in?  I don't believe this is practical,
> because even "C-x C-f" invokes a a subprocess when you visit a file
> under VCS (which I guess happens a lot to the likes of you and me).
> And how can a person avoid all of those commands?  What am I missing?

Native compilation is significantly more CPU intensive than any of these
other things, though.

> Did you actually try to see that "churning away" in practice, did you
> look at the time this typically takes, the resources (battery and
> otherwise) it consumes, etc.?  If so, can you share the data: how many
> *.eln files were produced in how many minutes, how much battery did
> that consume?
>
> You see, I have my data, and it seems to indicate a very short period
> of initial compilations for a modest consumption of resources.  (This
> isn't a laptop, but I'm acutely aware of my system's load at all
> times, and have it on the mode line, so I know what kind of
> computation is going on during that time.)  The data I see here
> doesn't look like it should be a problem for anyone, as long as files
> are compiled only as they are used. In my case, for example, I see
> maybe half a dozen *.eln files compiled after the initial startup.  I
> expect to see that on any machine where a user has steady usage
> patterns -- the compilation flats out very quickly.  I cannot imagine
> how will your CPU churn out for any significant time after an upgrade.
> A typical compilation of a single .el file is usually very fast,
> exceptions are extremely rare.  And I barely notice the effect of that
> on system responsiveness.
>
> So I could imagine that many people perhaps base these judgments on
> something they didn't actually try, but just imagined to themselves,
> and imagined incorrectly, or based their opinions on some other
> compilations of different languages in different environments.
>
> I urge people to actually try this and see what it does, before making
> up your minds.

I have actually tried it, for more than a year now, but admittedly I do
not have hard data.  Thank you for sharing yours.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Mon 03 Oct 2022 at 07:24PM +03, Eli Zaretskii wrote:

>> From: Sean Whitton <spwhitton@spwhitton.name>
>> Cc: tomas@tuxteam.de,  emacs-devel@gnu.org
>> Date: Sun, 02 Oct 2022 16:58:34 -0700
>>
>> The battery usage thing is real, from my own user experience.
>>
>> I have to remember not to unplug my laptop until the load indicator in
>> my status bar has dropped back down, else the battery won't last as long
>> as I expect it to.
>
> When I start an Emacs session for a new version (which needs to
> recompile all the packages I use, because my sessions are restored by
> desktop.el), I see about 100 Lisp files compiled into *.eln inside of
> 1 minute, and then another bunch of 100 inside another minute (with 10
> minutes between these two groups).  Does this strike you as a long
> time to avoid unplugging?  I hope your battery can hold more than a
> few minutes.

Sounds like your machine is just a lot better than mine.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

>> FWIW, during the stealth jit-lock discussion, several people mentioned the
>> battery impact.
> Yes.  And jit-stealth is different, in that it takes a much longer
> time for it to become quiet, because it works in small chinks, and
> only when Emacs is actually idle.  JIT native-compilation is much
> faster, and also uses several execution units of the CPU in parallel.

Indeed, it's quite different.  I think even more important than the
above is the fact that it's a one-time thing only (for typical use cases
at least), whereas the jit-lock-stealth cost was paid over and over
again after every buffer modification.

>> And the issue is not subprocesses per se, but it's extra processing that
>> happens outside of the control of the user.
> How do you mean "outside of the control of the user"?  The user causes
> it by loading a feature.

When a user loads a feature, this request doesn't say explicitly "and
native compile the file".  And there is no direct way to "just load this
package without compiling it".  So the compilation is not really under
the control of the user.

I don't think users know, when they load a feature, if it's the first
time they loaded this feature in this version of Emacs, so by and large
they can't predict when the compilation does happen.

To be clear: I don't think it's a problem.  I just think it's part of
the reason for the reaction.

> How is that different from any other command that uses a subprocess
> under the hood?

It's different in that for those other commands, the users cannot get
the result they want without running that subprocess, so even if they're
not fully aware of it, they do have some vague idea it's inevitably
going to happen.

In contrast, the compilation is not indispensable, (so much so that the
past it never happened).

>> Probably not, no.  It's likely mostly a question of perception, so if we
>> could make it completely invisible the "problem" would disappear :-)
>> But the fact that lazy native compilation tends to pop up warnings (and
>> to make matters worse, it does so ... without warning) makes it very
>> much visible instead.
>
> On my system, I don't see any warnings.

That's presumably because you stick to the bundled packages (where we've
managed to keep a "no compilation warnings" state for the last few
years) or the few third party packages which are careful enough to
eliminate all warnings.

The rise of ELPA has removed most of the cases where native compilation
will miscompile files (because ELPA forces the .el files to be compiled
in a "standard" way rather than using ad-hoc Makefile rules which the
native compiler cannot and does not try to reproduce), but compilation
warnings are still very common (the situation is improving, tho, and
arguably the warnings popping up out of the blue during lazy native
compilation may prove to be a good way to convince more package authors
to clean up their act).


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Mon, 3 Oct 2022 20:17:53 +0200
> From: tomas@tuxteam.de
> Cc: Rob Browning <rlb@defaultvalue.org>, larsi@gnus.org, yandros@gmail.com,
> 	emacs-devel@gnu.org
> 
> On Mon, Oct 03, 2022 at 08:41:22PM +0300, Eli Zaretskii wrote:
> > [Full disclosure: evidently, none of what I write below matters, so
> > feel free to ignore.]
> 
> Now I think this is a bit unfair. You have a string opinion, and
> I think it shhould be respected, but the others have, too.

You misunderstood the intent.

> > That's strange to hear.  Even MS-Windows allows per-user variations of
> > PATH and per-use environment variables.  It is strange to learn that
> > Debian doesn't.
> 
> Note that this is only as far as the Debian packaging system is
> concerned. Per-user you can have as many Emacsen installed as you
> want -- the Debian packaging system doesn't know about them.

Windows installers allow you to install into a directory of your
choice, and some will even set PATH to point to there; if not, you can
do it manually.

> Alternatively, you can have a system-local install (going to
> /usr/local) besides the Debian-installed one without them interfering
> (that's the setup I use).

I don't understand what that means.  Do you mean that you can subvert
the general non-support by Debian packages of several Emacs versions
installed on the same system?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: rlb@defaultvalue.org,  monnier@iro.umontreal.ca,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Mon, 03 Oct 2022 11:23:15 -0700
> 
> > How does this work in practice?  I mean, what exactly do you mean by
> > "processes they start themselves"?  You mean, you know by heart each
> > Emacs command that invokes a subprocess, and avoid to invoke them all
> > unless the laptop is plugged in?  I don't believe this is practical,
> > because even "C-x C-f" invokes a a subprocess when you visit a file
> > under VCS (which I guess happens a lot to the likes of you and me).
> > And how can a person avoid all of those commands?  What am I missing?
> 
> Native compilation is significantly more CPU intensive than any of these
> other things, though.

Do you have data to support that?  I just cited mine, and it doesn't
sound intensive to me.  E.g., when some command invokes Git, Git could
decide that it's time to do a background GC, and that will be quite
CPU intensive.  And yet we all use VC without fear.

But I must stop raising the noise level here, because that's what all
I write seems to be good for -- making noise that no one takes
seriously, starting from Lars.  Sorry.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

>> You can only have one of them installed at a time, and you can
>> (currently) only have one major version installed at a time.
>> 
>>   https://packages.debian.org/search?keywords=emacs
>
> That's strange to hear.  Even MS-Windows allows per-user variations of
> PATH and per-use environment variables.  It is strange to learn that
> Debian doesn't.

Debian does as well.  What Rob is saying is that *if* you want to use
the Emacs provided by Debian, then the choice of which flavor (and
version) you get will apply system-wide.  If that's not good enough for
you, then you can install your own build of Emacs in the usual way, and
it's very easy to do so.

All my Debian boxes come with "Debian's Emacs" installed, and then most
of them have a few other versions built locally for my own personal use.

> And I guess now I'm confused what is it exactly that you'd want to
> achieve.  Do you want to disable native compilation, or do you want to
> have all *.eln files in a shared location?

AFAIK he wants:

- To eagerly compile the `.eln` files for the third party ELisp packages
  installed system-wide using Debian's package manager.

- That every time Emacs is run as part of
  (un)installing/building/testing a Debian package, that Emacs should
  not write any file under $HOME/ (or ~$REALUSER/ either).


-- Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Mon, 03 Oct 2022 11:23:46 -0700
> 
> > When I start an Emacs session for a new version (which needs to
> > recompile all the packages I use, because my sessions are restored by
> > desktop.el), I see about 100 Lisp files compiled into *.eln inside of
> > 1 minute, and then another bunch of 100 inside another minute (with 10
> > minutes between these two groups).  Does this strike you as a long
> > time to avoid unplugging?  I hope your battery can hold more than a
> > few minutes.
> 
> Sounds like your machine is just a lot better than mine.

It's a 2012 vintage Core i7.  And it runs Windows XP, so both process
startup and disk I/O are slower than GNU/Linux.  Though the main disk
is SSD.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: rlb@defaultvalue.org,  tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Mon, 03 Oct 2022 14:33:09 -0400
> 
> >> And the issue is not subprocesses per se, but it's extra processing that
> >> happens outside of the control of the user.
> > How do you mean "outside of the control of the user"?  The user causes
> > it by loading a feature.
> 
> When a user loads a feature, this request doesn't say explicitly "and
> native compile the file".  And there is no direct way to "just load this
> package without compiling it".  So the compilation is not really under
> the control of the user.
> 
> I don't think users know, when they load a feature, if it's the first
> time they loaded this feature in this version of Emacs, so by and large
> they can't predict when the compilation does happen.

It just takes getting used to, that's all.

> > How is that different from any other command that uses a subprocess
> > under the hood?
> 
> It's different in that for those other commands, the users cannot get
> the result they want without running that subprocess, so even if they're
> not fully aware of it, they do have some vague idea it's inevitably
> going to happen.
> 
> In contrast, the compilation is not indispensable, (so much so that the
> past it never happened).

So native-compilation is "guilty" because it can be disabled? ;-)

> > On my system, I don't see any warnings.
> 
> That's presumably because you stick to the bundled packages (where we've
> managed to keep a "no compilation warnings" state for the last few
> years) or the few third party packages which are careful enough to
> eliminate all warnings.
> 
> The rise of ELPA has removed most of the cases where native compilation
> will miscompile files (because ELPA forces the .el files to be compiled
> in a "standard" way rather than using ad-hoc Makefile rules which the
> native compiler cannot and does not try to reproduce), but compilation
> warnings are still very common (the situation is improving, tho, and
> arguably the warnings popping up out of the blue during lazy native
> compilation may prove to be a good way to convince more package authors
> to clean up their act).

So this, too, is extremely temporary, and will disappear soon enough,
at least in popular packages.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Rob Browning <rlb@defaultvalue.org>,  larsi@gnus.org,
>   yandros@gmail.com,  tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Mon, 03 Oct 2022 14:45:32 -0400
> 
> > And I guess now I'm confused what is it exactly that you'd want to
> > achieve.  Do you want to disable native compilation, or do you want to
> > have all *.eln files in a shared location?
> 
> AFAIK he wants:
> 
> - To eagerly compile the `.eln` files for the third party ELisp packages
>   installed system-wide using Debian's package manager.
> 
> - That every time Emacs is run as part of
>   (un)installing/building/testing a Debian package, that Emacs should
>   not write any file under $HOME/ (or ~$REALUSER/ either).

That's where I'm confused: if *.eln files are installed system-wide,
then why would there be any compilation that writes to the home
directory, which then causes Rob to wish to disable it?  All *.eln
files are already available, no?

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]

On Mon, Oct 03, 2022 at 09:41:16PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 3 Oct 2022 20:17:53 +0200
> > From: tomas@tuxteam.de
> > Cc: Rob Browning <rlb@defaultvalue.org>, larsi@gnus.org, yandros@gmail.com,
> > 	emacs-devel@gnu.org
> > 
> > On Mon, Oct 03, 2022 at 08:41:22PM +0300, Eli Zaretskii wrote:
> > > [Full disclosure: evidently, none of what I write below matters, so
> > > feel free to ignore.]
> > 
> > Now I think this is a bit unfair. You have a string opinion, and
> > I think it shhould be respected, but the others have, too.
> 
> You misunderstood the intent.

Quite possibly.

> Windows installers allow you to install into a directory of your
> choice, and some will even set PATH to point to there; if not, you can
> do it manually.

I have no clue about Windows (at least since 1990). I try to keep
it that way for reasons.

> > Alternatively, you can have a system-local install (going to
> > /usr/local) besides the Debian-installed one without them interfering
> > (that's the setup I use).
> 
> I don't understand what that means.  Do you mean that you can subvert
> the general non-support by Debian packages of several Emacs versions
> installed on the same system?

I don't understand what you mean by "subverting the general non-support",
(I somehow feel that it is a loaded term, but I can't be sure).

Cheers
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Mon, 3 Oct 2022 21:02:11 +0200
> From: tomas@tuxteam.de
> Cc: rlb@defaultvalue.org, larsi@gnus.org, yandros@gmail.com,
> 	emacs-devel@gnu.org
> 
> > > Alternatively, you can have a system-local install (going to
> > > /usr/local) besides the Debian-installed one without them interfering
> > > (that's the setup I use).
> > 
> > I don't understand what that means.  Do you mean that you can subvert
> > the general non-support by Debian packages of several Emacs versions
> > installed on the same system?
> 
> I don't understand what you mean by "subverting the general non-support",

Rob said they don't support that with Debian packages.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> We'd also like to be able to compile and install system-wide -- it would
> be good to be able to benefit from native compilation without it
> happening by surrpise.

I haven't heard any specific request in this regard: is there something
preventing you from doing that (for your definition of "that")?


        Stefan

Re: Suppressing native compilation (short and long term)

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> compilation warnings are still very common (the situation is
>> improving, tho, and arguably the warnings popping up out of the blue
>> during lazy native compilation may prove to be a good way to convince
>> more package authors to clean up their act).
>
> So this, too, is extremely temporary, and will disappear soon enough,
> at least in popular packages.

I don't see that the situation has substantially improved, though I'm of
course happy to hear that Stefan M remains optimistic.

In far too many cases, it is very hard to get fixes for warnings merged
in a timely manner, and it's even harder to get those fixes actually
released.

Here are some recent examples, to give an idea:

    https://github.com/deb0ch/emacs-winum/pull/31
    https://github.com/lewang/ws-butler/pull/40
    https://github.com/Malabarba/aggressive-indent-mode/pull/150

These examples can be multiplied at will.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> [Full disclosure: evidently, none of what I write below matters, so
> feel free to ignore.]

Well, I can't speak for others, but it matters to me (your opinion and
the other upstream developers'), or I wouldn't be here, and I'm very
glad there's a "here for me to be" because I'm quite fond of Emacs :)

> But okay, yes, if Debian users live under such severe restrictions,
> then the case for having user-specific *.eln directories becomes
> weaker.  But I still don't see that it is weaker than disallowing
> native compilation at run time.

To be clear, I've never been talking about disallowing it for normal
user use, and apologies if I made it sound that way.

For normal users, if we pursued this, the idea would be that after "apt
install emacs" finishes, they'd have a full set of .elc and .eln files
corresponding to their version of Emacs, and the .el files they have.

Then if anything changes in a way that warrants it, Emacs would
(re)compile to their ~/.emacs.d/eln-cache/ as "normal".  But for those
who don't ever change their .el files (or shadow them), that would never
happen, or would only happen for the files that they change/shadow.

> And I guess now I'm confused what is it exactly that you'd want to
> achieve.  Do you want to disable native compilation, or do you want to
> have all *.eln files in a shared location?  Because it seems to me you
> said you wanted both, and I don't see how both could be reconciled.

The ability to disable .eln compilation entirely is only for some
Debian-specific (though maybe useful for others) special cases, not
anything we're proposing for "normal use".

And it doesn't have to be "disabling" -- being able to redirect the .eln
files to another (temp) dir would be fine too, as long as it wasn't too
awkward to arrange for the entire (sub)process tree.

Those special cases are (at least):

  - When building the relevant Debian packages -- for Emacs itself, for
    add-ons like elpa-magit, etc.  Debian forbids package builds from
    writing outside of the package build directory /tmp, and another
    tree or two, e.g. HOME is not allowed.  Note that this is normally
    handled by the Debian autobuilders, it's not something users
    typically do.

  - During package installs, i.e. whenever Emacs is run during an "apt
    install emacs" or "apt upgrade emacs", etc., and it may be run many
    times there, both directly, and indirectly as it rebuilds whichever
    packages need rebuilding (e.g. add-on packages).

  - During some autmated Debian tests.

Hope this helps
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Po Lu]

Sean Whitton <spwhitton@spwhitton.name> writes:

> The point with battery consumption is not about running vs. standby.
> The issue is that while users expect that running apt-get will drain the
> battery, they expect that once apt-get is done, the only battery-hungry
> processes are ones they start themselves.  Laptop users typically avoid
> running apt-get without mains power plugged in.  If I do an 'apt-get
> upgrade' then afterwards my CPU will be churning away compiling addon
> packages, so I can't just unplug.

I don't know if that's true with Debian users, but nobody I know of
objects to running "dnf install" on battery.  After all, installing new
packages only takes a short amount of time, certainly not enough to
significantly affect battery usage.

OTOH, async native compilation is something I would definitely plug in
for.  That applies to other programs too, notably web browsers.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Mon, 3 Oct 2022 23:58:28 +0200
> Cc: rlb@defaultvalue.org, tomas@tuxteam.de, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> compilation warnings are still very common (the situation is
> >> improving, tho, and arguably the warnings popping up out of the blue
> >> during lazy native compilation may prove to be a good way to convince
> >> more package authors to clean up their act).
> >
> > So this, too, is extremely temporary, and will disappear soon enough,
> > at least in popular packages.
> 
> I don't see that the situation has substantially improved

You are too impatient, IMO.  This stuff takes time to propagate; Emacs
28 is just beginning to be widely used (otherwise we'd have this
discussion many moons ago).  We had these problems in Emacs as well,
originally.

> Here are some recent examples, to give an idea:
> 
>     https://github.com/deb0ch/emacs-winum/pull/31
>     https://github.com/lewang/ws-butler/pull/40
>     https://github.com/Malabarba/aggressive-indent-mode/pull/150
> 
> These examples can be multiplied at will.

Yes, it is easy to think the problem is larger and more significant
than it really is.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Rob Browning <rlb@defaultvalue.org>,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org,
>   akrl@sdf.org
> Date: Tue, 04 Oct 2022 08:31:01 +0800
> 
> Sean Whitton <spwhitton@spwhitton.name> writes:
> 
> > The point with battery consumption is not about running vs. standby.
> > The issue is that while users expect that running apt-get will drain the
> > battery, they expect that once apt-get is done, the only battery-hungry
> > processes are ones they start themselves.  Laptop users typically avoid
> > running apt-get without mains power plugged in.  If I do an 'apt-get
> > upgrade' then afterwards my CPU will be churning away compiling addon
> > packages, so I can't just unplug.
> 
> I don't know if that's true with Debian users, but nobody I know of
> objects to running "dnf install" on battery.  After all, installing new
> packages only takes a short amount of time, certainly not enough to
> significantly affect battery usage.
> 
> OTOH, async native compilation is something I would definitely plug in
> for.

Why the difference, I wonder?  Async native compilation also takes a
short amount of time, as the data I posted indicates.

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Why the difference, I wonder?  Async native compilation also takes a
> short amount of time, as the data I posted indicates.

I don't know, but the fans spin up during async native compilation and
not package installation.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: spwhitton@spwhitton.name,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org,
>   akrl@sdf.org
> Date: Tue, 04 Oct 2022 16:37:10 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Why the difference, I wonder?  Async native compilation also takes a
> > short amount of time, as the data I posted indicates.
> 
> I don't know, but the fans spin up during async native compilation and
> not package installation.

Maybe you should customize native-comp-async-jobs-number to a small
enough value, like 1?  The default zero means half of your CPU's
execution units.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: larsi@gnus.org, yandros@gmail.com, tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Mon, 03 Oct 2022 19:16:29 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > [Full disclosure: evidently, none of what I write below matters, so
> > feel free to ignore.]
> 
> Well, I can't speak for others, but it matters to me (your opinion and
> the other upstream developers'), or I wouldn't be here, and I'm very
> glad there's a "here for me to be" because I'm quite fond of Emacs :)

I didn't mean you, FWIW.

> > And I guess now I'm confused what is it exactly that you'd want to
> > achieve.  Do you want to disable native compilation, or do you want to
> > have all *.eln files in a shared location?  Because it seems to me you
> > said you wanted both, and I don't see how both could be reconciled.
> 
> The ability to disable .eln compilation entirely is only for some
> Debian-specific (though maybe useful for others) special cases, not
> anything we're proposing for "normal use".
> 
> And it doesn't have to be "disabling" -- being able to redirect the .eln
> files to another (temp) dir would be fine too, as long as it wasn't too
> awkward to arrange for the entire (sub)process tree.
> 
> Those special cases are (at least):
> 
>   - When building the relevant Debian packages -- for Emacs itself, for
>     add-ons like elpa-magit, etc.  Debian forbids package builds from
>     writing outside of the package build directory /tmp, and another
>     tree or two, e.g. HOME is not allowed.  Note that this is normally
>     handled by the Debian autobuilders, it's not something users
>     typically do.
> 
>   - During package installs, i.e. whenever Emacs is run during an "apt
>     install emacs" or "apt upgrade emacs", etc., and it may be run many
>     times there, both directly, and indirectly as it rebuilds whichever
>     packages need rebuilding (e.g. add-on packages).
> 
>   - During some autmated Debian tests.

The above means (AFAIU) that disabling is not the right solution,
because you want the *.eln files to go to the shared location where
users could have them loaded when needed.  Is that correct, or am I
again missing something?

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Maybe you should customize native-comp-async-jobs-number to a small
> enough value, like 1?  The default zero means half of your CPU's
> execution units.

I will try that and see how it goes, thanks.

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> I don't see that the situation has substantially improved, though I'm of
> course happy to hear that Stefan M remains optimistic.

Seen from the K-side, the glass is half full, but I'm on the M-side
where I see it as half-empty.


        Stefan

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Tue 04 Oct 2022 at 08:31AM +08, Po Lu wrote:

> Sean Whitton <spwhitton@spwhitton.name> writes:
>
>> The point with battery consumption is not about running vs. standby.
>> The issue is that while users expect that running apt-get will drain the
>> battery, they expect that once apt-get is done, the only battery-hungry
>> processes are ones they start themselves.  Laptop users typically avoid
>> running apt-get without mains power plugged in.  If I do an 'apt-get
>> upgrade' then afterwards my CPU will be churning away compiling addon
>> packages, so I can't just unplug.
>
> I don't know if that's true with Debian users, but nobody I know of
> objects to running "dnf install" on battery.  After all, installing new
> packages only takes a short amount of time, certainly not enough to
> significantly affect battery usage.

The issue is Debian Emacs addon packages, in particular.  Upgrading
those kicks off a whole lot of bytecompilation.  It's much
heavier-weight than usual package upgrades.

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> The above means (AFAIU) that disabling is not the right solution,
> because you want the *.eln files to go to the shared location where
> users could have them loaded when needed.  Is that correct, or am I
> again missing something?

If I understand you, I think the answer may be no?

The test and package build cases are (often) isolated, automated,
debien-specific processes that happen on debian build machines (as root
and chrooted), and the entire workspace is then thrown away.  It's never
relevant to a "normal" user.

The package install/upgrade process is system-wide, runs as root, and is
specifically and quite intentionally not per-user.  It's intended to
establish the static, common baseline for *all* users on the machine
until the next package upgrade.

Hope this helps
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: larsi@gnus.org, yandros@gmail.com, tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Tue, 04 Oct 2022 19:48:01 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The above means (AFAIU) that disabling is not the right solution,
> > because you want the *.eln files to go to the shared location where
> > users could have them loaded when needed.  Is that correct, or am I
> > again missing something?
> 
> If I understand you, I think the answer may be no?
> 
> The test and package build cases are (often) isolated, automated,
> debien-specific processes that happen on debian build machines (as root
> and chrooted), and the entire workspace is then thrown away.  It's never
> relevant to a "normal" user.

If the workspace is thrown away, you shouldn't care about the *.eln
files it produces, right?  They will be thrown away together with the
workspace.  So why bother disabling their production in this case?

> The package install/upgrade process is system-wide, runs as root, and is
> specifically and quite intentionally not per-user.  It's intended to
> establish the static, common baseline for *all* users on the machine
> until the next package upgrade.

When this package install/upgrade process runs, doesn't it include
installing the *.eln files that come with the package, and need to be
used when the users use the package after the installation?  AFAIU,
you want to install those *.eln files in a shared location, which is
fine and supported: just put them in the same place under
/usr/lib/emacs/VERSION/native-lisp/ directory where Emacs installs its
precompiled *.eln files.  Or, if you want to have the *.eln files from
add-on packages to live in a separate place, define where that
directory will be, install the package's *.eln files there, and tweak
native-comp-eln-load-path to include that additional directory.

Again, in this case as well, I see no problem with generation of *.eln
files that you'd like to prevent.  And yet you are still saying that
disabling *.eln generation might be needed.  Could you please describe
in enough detail when are those unwanted *.eln produced in the two
situations you described (test and package build case, and package
install/upgrade case), and why are those *.eln files unwanted in that
case?

Thanks.

P.S. I'm sorry to keep this thread alive for so long, but I really
don't have a clear idea why you'd want to disable generation of *.eln
files, and I think it's important for us to understand your reasons.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Rob Browning <rlb@defaultvalue.org> writes:
>
>> At the top level, we wanted a way to avoid writing to HOME during
>> packaging, testing, installs (in this case, it's the .eln files, now
>> that we've enabled native compilation).
>>
>> That could be handled by some way to turn off native compilation, or by
>> some way to comprehensively divert those writes to another location
>> (e.g. temp dir).  Either is fine, though we'd originally thought the
>> former might make things a bit easier.
>
> Yeah, I think the former is both easier to implement and easier for
> users.
>
> So I'm thinking of introducing a user option like
> `native-compile-inhibit', which will make Emacs skip the native-comp
> machinery when loading .elc files.

IIUC this would just control `load-no-native' and
`native-comp-deferred-compilation'?  I think these two are doing what
you suggest here no?

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Lars Ingebrigtsen [2022-10-03 15:29:40] wrote:
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> AFAICT so far I've heard of two reasons to "disable automatic native
>>> compilation":
>>>
>>> A) Avoid the "uncontrolled" (mostly in terms of *when* it happens)
>>>    resource usage associated with it (e.g. battery).
>>>
>>> B) Avoid writing to $HOME.
>>>
>>> For (A), I think that (setq native-comp-deferred-compilation nil) seems
>>> perfectly sufficient (you can set it early in your (early) init file and if
>>> every blue moon some native compilation still happens because of a file
>>> loaded before that `setq` it shouldn't be a big deal).
>>
>> I've now introduced the new inhibit-native-compilation variable and
>> deprecated the native-comp-deferred-compilation variable, and also
>> initialised stuff from EMACS_INHIBIT_NATIVE_COMPILATION.
>> (And made trampolines not write to the cache in this case.)
>
> AFAICT for the case (A), we *do* want to save trampolines for the next
> time around, and those users probably do want to use native compilation
> (e.g. they would likely set `package-native-compile` to non-nil), just
> not the deferred kind, so the var name is a bit odd for them.

I find this name odd as well. `inhibit-native-compilation' is *not*
disabling native compilation for two reasons:

1- One can still inoke it manually
2- Trampoline automatic ocmpilation is still active

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> We can expand the variable name, but I couldn't think of a good name.
>>> And "deferred" doesn't convey anything.
>>
>> Lazy?  Auto(matic)?  Ondemand?
>
> `inhibit-lazy-native-compilation' sounds like it makes the compilation
> less lazy.  `inhibit-automatic-native-compilation' might work.

Again `inhibit-automatic-native-compilation' does not disable automatic
trampoline native compilation :/

BTW I think most of people refers to what was controlled by
`native-comp-deferred-compilation' as a jitter mechanism, so I think a
better name would have been `inhibit-native-jit-compilation'.  I wish
this change was not rushed.

Best Regards

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> I think that entire changeset should be reverted.  It is not well
>> thought out, and ion some aspects downright dangerous.  The
>> environment variable is especially egregious: it will affect all the
>> sub-process Emacsen as well, something that will bite users, a lesson
>> we've learned long ago with the likes of EMACSLOADPATH.
>
> How is setting EMACS_INHIBIT_NATIVE_COMPILE dangerous?
>
>> Such changes should not be installed without a great deal of careful
>> thought and consideration of the different use cases.  Which I tried
>> to explain all the way, but apparently to deaf ears.
>
> The discussion has been going for apparently weeks no with no progress.
> It's easier to discuss code when there's code to discuss.

We can discuss code also on branches, there's no need to install changes
on master to discuss them.

I'm back now after a long weekend and your proposal of introduciung
`inhibit-native-compilation' is form 2 days and 1 hour ago.  Sorry but I
don't want to feel that changes can be rushed into Emacs code and in
order to partecipate to the discussion people can't have some time off.

Best Regards

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,  david@tethera.net,
>>   emacs-devel@gnu.org,  akrl@sdf.org
>> Date: Mon, 03 Oct 2022 16:09:35 +0200
>> 
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> 
>> > AFAICT for the case (A), we *do* want to save trampolines for the next
>> > time around, and those users probably do want to use native compilation
>> > (e.g. they would likely set `package-native-compile` to non-nil), just
>> > not the deferred kind, so the var name is a bit odd for them.
>> 
>> The trampolines are very fast to make, so creating them once per Emacs
>> session doesn't seem like a problem.  If that turns out to be an issue,
>> we can tweak the variable.
>
> I think that entire changeset should be reverted.

Definitely agree here.

Other than the (IMO) bad naming, is still not clear to me what's exactly
the problem is trying to solve.

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Again `inhibit-automatic-native-compilation' does not disable automatic
> trampoline native compilation :/
>
> BTW I think most of people refers to what was controlled by
> `native-comp-deferred-compilation' as a jitter mechanism, so I think a
> better name would have been `inhibit-native-jit-compilation'.  I wish
> this change was not rushed.

I don't think `inhibit-native-jit-compilation' conveys anything more to
the user than `inhibit-automatic-native-compilation'?

Andrea Corallo <akrl@sdf.org> writes:

> We can discuss code also on branches, there's no need to install changes
> on master to discuss them.
>
> I'm back now after a long weekend and your proposal of introduciung
> `inhibit-native-compilation' is form 2 days and 1 hour ago.  Sorry but I
> don't want to feel that changes can be rushed into Emacs code and in
> order to partecipate to the discussion people can't have some time off.

Development takes place on the "master" branch -- code appearing there
does not curtail further discussion.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> IIUC this would just control `load-no-native' and
> `native-comp-deferred-compilation'?  I think these two are doing what
> you suggest here no?

The latter yes -- nobody has discussed doing anything with
`load-no-native' (which is another variable that's difficult to discover
due to how it's named, and isn't documented anywhere, either).

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> If we want to consider providing (yet another) user option for
>> disabling native compilation, then we should:
>
> I asked what the user option to disable native compilation was, but
> didn't get an answer, and here you say "(yet another)", so...  what is
> the current user option to disable native compilation?

Sorry for being late: `native-comp-deferred-compilation' and
`load-no-native'.

>>   . understand why and in which situations they may need it
>
> Doing repeatable testing is one obvious situation.

I think the two previously mentioned knobs should be sufficient for
repeatable testing no?

Also repeatable testing are most likely executed in batch mode and...

...surprise surprise deferred compilation is *already* *disabled* in
this mode!!

I've the impression no one mentioned this small detail in this huge
thread, but still we have installed changes to disable a feature for
debian pkg installation that is in fact already disabled :( :(

Best Regards

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Rob Browning <rlb@defaultvalue.org> writes:
>
>> Though I'm not positive because if we still really need compilation for
>> the trampolines(?), then we may need to just follow the "redirect to a
>> tempdir" approach, which is also fine, and then the ability to disable
>> native compilation might not be relevant to the packaging.
>
> I'm not sure we need to save the trampolines at all (in this
> don't-do-native-compilation configuration) -- Andrea probably knows.
> Andrea, would it be possible to just create and use the trampolines
> without writing them to a file?

No

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> IIUC this would just control `load-no-native' and
>> `native-comp-deferred-compilation'?  I think these two are doing what
>> you suggest here no?
>
> The latter yes -- nobody has discussed doing anything with
> `load-no-native'

I wasn't clear to me what you meant with "native-comp machinery when
loading .elc files".

> (which is another variable that's difficult to discover due to how
> it's named, and isn't documented anywhere, either).

It is named like that since before the native comp branch was reviewed
and merged, I think you were in that thread.  Sorry we all try our best.

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Again `inhibit-automatic-native-compilation' does not disable automatic
>> trampoline native compilation :/
>>
>> BTW I think most of people refers to what was controlled by
>> `native-comp-deferred-compilation' as a jitter mechanism, so I think a
>> better name would have been `inhibit-native-jit-compilation'.  I wish
>> this change was not rushed.
>
> I don't think `inhibit-native-jit-compilation' conveys anything more to
> the user than `inhibit-automatic-native-compilation'?

That's your opinion and I respect it.  Still
`inhibit-automatic-native-compilation' does *not* disable automatic
native compilation but only a mechanism contributing to it, so it's IMO
a bad naming decision.

> Andrea Corallo <akrl@sdf.org> writes:
>
>> We can discuss code also on branches, there's no need to install changes
>> on master to discuss them.
>>
>> I'm back now after a long weekend and your proposal of introduciung
>> `inhibit-native-compilation' is form 2 days and 1 hour ago.  Sorry but I
>> don't want to feel that changes can be rushed into Emacs code and in
>> order to partecipate to the discussion people can't have some time off.
>
> Development takes place on the "master" branch -- code appearing there
> does not curtail further discussion.

I have not said that once code is in master discussion is forbidden.  I
said that to discuss a change there's *no* requirement to install it in
master, especially before sufficient discussion is done on the list for
these tricky interfaces.  My 2cts are that these mechanisms and changes
should be very well thought and participated before being modified.

As maintainer of comp.c and related I ask to have this changeset
reverted and then we restart thinking again what's the best change (if
any) needed here.

Thanks

  Andrea

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii
>  <eliz@gnu.org>,
>         rlb@defaultvalue.org, david@tethera.net, emacs-devel@gnu.org
> Date: Wed, 05 Oct 2022 12:55:40 +0000
> 
> I wish this change was not rushed.

Me too.  Especially since I'm still struggling to understand the
rationale of the Debian packagers to request this.

Re: Suppressing native compilation (short and long term)

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Maybe you should customize native-comp-async-jobs-number to a small
>> enough value, like 1?  The default zero means half of your CPU's
>> execution units.
>
> I will try that and see how it goes, thanks.

That makes the fans less loud (they are still noticable), but it also
takes twice as long for the fans to subside, as expected.

Hope this data point ends up meaningful.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Also repeatable testing are most likely executed in batch mode and...
>
> ...surprise surprise deferred compilation is *already* *disabled* in
> this mode!!

Not all testing happens in batch mode.

Andrea Corallo <akrl@sdf.org> writes:

>> I'm not sure we need to save the trampolines at all (in this
>> don't-do-native-compilation configuration) -- Andrea probably knows.
>> Andrea, would it be possible to just create and use the trampolines
>> without writing them to a file?
>
> No

Yes, it is.  (That is, the trampolines get written to /tmp, but can be
deleted after loading.)

Andrea Corallo <akrl@sdf.org> writes:

> That's your opinion and I respect it.  Still
> `inhibit-automatic-native-compilation' does *not* disable automatic
> native compilation but only a mechanism contributing to it, so it's IMO
> a bad naming decision.

Like I said earlier, if anybody has a better name for the variable,
renaming it is fine, but it has to be an improvement.

> I have not said that once code is in master discussion is forbidden.  I
> said that to discuss a change there's *no* requirement to install it in
> master, especially before sufficient discussion is done on the list for
> these tricky interfaces.  My 2cts are that these mechanisms and changes
> should be very well thought and participated before being modified.
>
> As maintainer of comp.c and related I ask to have this changeset
> reverted and then we restart thinking again what's the best change (if
> any) needed here.

I don't see any advantages to doing that -- the changes that are in now
seem to work fine for the stated use cases (which are both "don't write
to $HOME while testing" and "I want to use a AOT-compiled Emacs, but
don't do any further JIT compilation while running Emacs").

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Rob Browning <rlb@defaultvalue.org>,  Stefan Monnier
>  <monnier@iro.umontreal.ca>,  Eli Zaretskii <eliz@gnu.org>,
>   david@tethera.net,  emacs-devel@gnu.org
> Date: Wed, 05 Oct 2022 15:16:31 +0200
> 
> Andrea Corallo <akrl@sdf.org> writes:
> 
> > IIUC this would just control `load-no-native' and
> > `native-comp-deferred-compilation'?  I think these two are doing what
> > you suggest here no?
> 
> The latter yes -- nobody has discussed doing anything with
> `load-no-native' (which is another variable that's difficult to discover
> due to how it's named, and isn't documented anywhere, either).

Variables that are "hard to discover" are intentionally named like
that: we didn't intend them to be discoverable, because we didn't
think they should be used except internally.

During the last stages of native-comp development, we went through the
variables in comp.c and comp.el with Andrea, and renamed those we
thought would be useful so that they are easily discoverable.  Please
don't assume that the names you see are due to omission or negligence
or ineptitude.  Maybe additional ones should be renamed/aliased to be
more discoverable, but before we do that, we need to understand the
problems we are solving.  That requires careful consideration and
discussion, whereas rushing with installing changes in the area where
you don't have enough prior knowledge and history ends up wreaking
havoc, to say nothing of personal aggravation.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii
>>  <eliz@gnu.org>,
>>         rlb@defaultvalue.org, david@tethera.net, emacs-devel@gnu.org
>> Date: Wed, 05 Oct 2022 12:55:40 +0000
>> 
>> I wish this change was not rushed.
>
> Me too.  Especially since I'm still struggling to understand the
> rationale of the Debian packagers to request this.

Agree and that's not all!  As we know native compilations happens
automatically for:

1- Trampolines, this we *cannot* disable in any case.

2- Deferred/jit compilation, this is *already* disabled in non
interactive sessions! (the way I guess Debian Emacs process is used for
package instgallation).

So I think this changeset does not help Debian packagers at all but ATM
is just more surface for future issues!

  Andrea

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org
> Date: Wed, 05 Oct 2022 16:01:35 +0200
> 
> > As maintainer of comp.c and related I ask to have this changeset
> > reverted and then we restart thinking again what's the best change (if
> > any) needed here.
> 
> I don't see any advantages to doing that -- the changes that are in now
> seem to work fine for the stated use cases (which are both "don't write
> to $HOME while testing"

That use case is not yet clear, not at all.

> and "I want to use a AOT-compiled Emacs, but don't do any further
> JIT compilation while running Emacs").

This use case makes no sense at all.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Also repeatable testing are most likely executed in batch mode and...
>>
>> ...surprise surprise deferred compilation is *already* *disabled* in
>> this mode!!
>
> Not all testing happens in batch mode.
>
> Andrea Corallo <akrl@sdf.org> writes:
>
>>> I'm not sure we need to save the trampolines at all (in this
>>> don't-do-native-compilation configuration) -- Andrea probably knows.
>>> Andrea, would it be possible to just create and use the trampolines
>>> without writing them to a file?
>>
>> No
>
> Yes, it is.  (That is, the trampolines get written to /tmp, but can be
> deleted after loading.)

You asked: "would it be possible to just create and use the trampolines
without writing them to a file?"

I aswered "no".  Of course writing a file into /tmp is still writing to
a file.

> Andrea Corallo <akrl@sdf.org> writes:
>
>> That's your opinion and I respect it.  Still
>> `inhibit-automatic-native-compilation' does *not* disable automatic
>> native compilation but only a mechanism contributing to it, so it's IMO
>> a bad naming decision.
>
> Like I said earlier, if anybody has a better name for the variable,
> renaming it is fine, but it has to be an improvement.

I think `inhibit-jit-native-compilation' is better as I believe users
perceive the JIT word related to user code and not internal mechanisms
as trampolines.

I've the perception that this change was done without the full picture
in mind of how the native compiler and his mechanisms works.  As a
result the current naming is IMO just wrong, and as such is a step
backward the original state.

As maintainer of the native compiler is my duty to point this out and
ask for reversion.

>> I have not said that once code is in master discussion is forbidden.  I
>> said that to discuss a change there's *no* requirement to install it in
>> master, especially before sufficient discussion is done on the list for
>> these tricky interfaces.  My 2cts are that these mechanisms and changes
>> should be very well thought and participated before being modified.
>>
>> As maintainer of comp.c and related I ask to have this changeset
>> reverted and then we restart thinking again what's the best change (if
>> any) needed here.
>
> I don't see any advantages to doing that -- the changes that are in now
> seem to work fine for the stated use cases (which are both "don't write
> to $HOME while testing" and "I want to use a AOT-compiled Emacs, but
> don't do any further JIT compilation while running Emacs").

I explained in two other mails that these changesets are orthogonal to
what the user asked and don't help them.  Also is still not clear why
the user asked for this knob.

It is *extremely* important that we analyze the usecase *before*
introducing new interfaces.  The user might ask for a new interface
without knowing we can provide or suggest a better solution. It might
even already exists!!

Developers can't just add mechanisms without a full understanding of
the current ones just because there was a user request, even worst
without an in deep discussion, sorry this is IMO just recipe for
disaster.

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> and "I want to use a AOT-compiled Emacs, but don't do any further
>> JIT compilation while running Emacs").
>
> This use case makes no sense at all.

It's been requested several times, whether it makes sense to you or not.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> I think `inhibit-jit-native-compilation' is better as I believe users
> perceive the JIT word related to user code and not internal mechanisms
> as trampolines.

It's possible -- I'm not married to the current name.  Perhaps we should
take a poll?

> I've the perception that this change was done without the full picture
> in mind of how the native compiler and his mechanisms works.  As a
> result the current naming is IMO just wrong, and as such is a step
> backward the original state.

I don't know where you got that perception from.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> I think `inhibit-jit-native-compilation' is better as I believe users
>> perceive the JIT word related to user code and not internal mechanisms
>> as trampolines.
>
> It's possible -- I'm not married to the current name.  Perhaps we should
> take a poll?
>
>> I've the perception that this change was done without the full picture
>> in mind of how the native compiler and his mechanisms works.  As a
>> result the current naming is IMO just wrong, and as such is a step
>> backward the original state.
>
> I don't know where you got that perception from.

Well to give few examples you were not aware of: the `load-no-native'
mechanism, the fact that deferred compilation is not the only mechanism
concurring to automatic native compilation (and that's why it was named
as such and not just automatic native compilation), the fact that native
compilation does not happen in non interactive sessions.

There is nothing wrong with that, the native compiler is a complex
machine and its interface as well, but still: there's no single aspect
of this changeset that I see as an improvement, so as maintainer of the
native compiler I ask to have it reverted.

Thanks

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Well to give few examples you were not aware of: the `load-no-native'
> mechanism, the fact that deferred compilation is not the only mechanism
> concurring to automatic native compilation (and that's why it was named
> as such and not just automatic native compilation), the fact that native
> compilation does not happen in non interactive sessions.

I didn't know about the first (because it's badly named and
undocumented, as well as totally irrelevant to the discussion in this
thread), but I was aware of all the other things here, and I'm not sure
why you'd think otherwise.

My perception here is that you're mostly angry that somebody else is
working on your code -- but that's pretty common.  Many people feel
proprietary towards code they've written.

> There is nothing wrong with that, the native compiler is a complex
> machine and its interface as well, but still: there's no single aspect
> of this changeset that I see as an improvement, so as maintainer of the
> native compiler I ask to have it reverted.

Like I said before, the code solves (at least) two real problems, so I'm
not in favour of doing so.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> That makes the fans less loud (they are still noticable), but it also
> takes twice as long for the fans to subside, as expected.
>
> Hope this data point ends up meaningful.

Yes.

Software is distributed pre-compiled for a reason -- because people run
the software on hardware where compiling the software takes a long time.
It's entirely reasonable for people to want to have a fully-built
native-compiled Emacs installation on their laptops, without making that
Emacs do any further JIT compilation.  (Except where necessary for
trampolines, of course.)

This has been requested a number of times over several years now, but
these requests have been ignored because apparently "people shouldn't
want that".

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Well to give few examples you were not aware of: the `load-no-native'
>> mechanism, the fact that deferred compilation is not the only mechanism
>> concurring to automatic native compilation (and that's why it was named
>> as such and not just automatic native compilation), the fact that native
>> compilation does not happen in non interactive sessions.
>
> I didn't know about the first (because it's badly named and
> undocumented, as well as totally irrelevant to the discussion in this
> thread), but I was aware of all the other things here, and I'm not sure
> why you'd think otherwise.

Well because I cannot understand why one would do any of these changes
if these mechanisms are known.

> My perception here is that you're mostly angry that somebody else is
> working on your code -- but that's pretty common.  Many people feel
> proprietary towards code they've written.

This is not the case at all, please trust me, your changeset does two
things:

1- change the name a knob, but it goes from a maybe un-intuitive one to
   just (as explained) a plain wrong one.

2- add a mechanism that (as explained) cannot help with the user request
   in this discussion at all.

These are both not improvements, this is the reason I'm asking for it to
be reverted now.

Please don't accuse me to feel the ownership on this code, I've never
objected to other changes on this done on master as I thought were
correct.

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> This is not the case at all, please trust me, your changeset does two
> things:
>
> 1- change the name a knob, but it goes from a maybe un-intuitive one to
>    just (as explained) a plain wrong one.

It goes from an un-intuitive (and undocumented) one to an
intuitively-named (and documented) one.  In my opinion, the old variable
is just as wrong as the current one (because it seemed to imply that
compilation would be immediate instead of deferred).

> 2- add a mechanism that (as explained) cannot help with the user request
>    in this discussion at all.

And I've explained twice now that 2 is wrong -- these changes do exactly
what was requested:

  1) It allows testing without writing to $HOME.  (This has nothing to
  do with --batch -- testing happens in interactive Emacsen, too.)

  2) It allows people to run an AOT Emacs without triggering further
  compilation.

If you have a suggestion for an alternative change that achieves these
two things, I'm all ears.  (But if your objection is "people shouldn't
want those two things", I'm down to just two ears again.)

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>   1) It allows testing without writing to $HOME.  (This has nothing to
>   do with --batch -- testing happens in interactive Emacsen, too.)

(Which reminds me -- one thing I've been pondering for a while is
whether "emacs -Q" should imply having the JIT off or not.  We use "-Q"
to get a repeatable Emacs for users, so it would make some sense to have
-Q now imply `inhibit-automatic-native-compilation'.  But I'm not sure.
If it does, should it also imply `load-no-native', which should then be
renamed to something less awkward and documented?  Probably not -- but
should there then be yet another twiddle to inhibit from
~/.emacs/eln-cache/?  That seems like overkill, probably, but it would
be in the spirit of "-Q".)

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> This is not the case at all, please trust me, your changeset does two
>> things:
>>
>> 1- change the name a knob, but it goes from a maybe un-intuitive one to
>>    just (as explained) a plain wrong one.
>
> It goes from an un-intuitive (and undocumented) one to an
> intuitively-named (and documented) one.

The documentation topic is orthogonal, you could have improved
`native-comp-deferred-compilation' documentation if you wanted.

The naming went first to `inhibit-native-compilation', this was really
just sooo wrong in so many levels :/ So many as many knobs we have to
control and disable different parts of the native compiler.  I can't
really think of one understanding all this machinery and then picking up
this name sorry, I can't.

Then the name became `inhibit-automatic-native-compilation' that is
fortunately just wrong on one level :/

The original name was the name of one of the two mechanisms concurring
to native compilation, that is the reason why it was just not called
"automatic".

> In my opinion, the old variable
> is just as wrong as the current one (because it seemed to imply that
> compilation would be immediate instead of deferred).

`native-comp-deferred-compilation' implies that native compilation can
happen deferred.  But anyway this name was reviewed and the name used
here countless times for long time.  Changing it with zero discussion is
just not good cooperation spirit, sorry.  You can think of it
differently but you just risk to remain alone doing development.

>> 2- add a mechanism that (as explained) cannot help with the user request
>>    in this discussion at all.
>
> And I've explained twice now that 2 is wrong -- these changes do exactly
> what was requested:
>
>   1) It allows testing without writing to $HOME.  (This has nothing to
>   do with --batch -- testing happens in interactive Emacsen, too.)

The user request is for non interactive sessions AFAIU.  And still I've
to understand exactly what the user wants to solve.  Most importantly I
feel I'm not alone here.

>   2) It allows people to run an AOT Emacs without triggering further
>   compilation.

Sorry as changeset I meant 5fec9182db + f97993ee66.  I'm not against
e245c4f226.

> If you have a suggestion for an alternative change that achieves these
> two things, I'm all ears.  (But if your objection is "people shouldn't
> want those two things", I'm down to just two ears again.)

Again 5fec9182db + f97993ee66 are IMO not useful / wrong, they were not
discussed and are just a step backward.  The best change I can suggest
is to revert them now.

Thanks

  Andrea

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>>   1) It allows testing without writing to $HOME.  (This has nothing to
>>   do with --batch -- testing happens in interactive Emacsen, too.)
>
> (Which reminds me -- one thing I've been pondering for a while is
> whether "emacs -Q" should imply having the JIT off or not.  We use "-Q"
> to get a repeatable Emacs for users, so it would make some sense to have
> -Q now imply `inhibit-automatic-native-compilation'.

Noooo please!! -Q has just one single clear meaning, let please not add
other implications to it!!  The situation would become very quickly
unmanageable.

And BTW -Q ATM is usefull for testing the native compiler as well :/ :/

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> The naming went first to `inhibit-native-compilation', this was really
> just sooo wrong in so many levels :/ So many as many knobs we have to
> control and disable different parts of the native compiler.  I can't
> really think of one understanding all this machinery and then picking up
> this name sorry, I can't.

Naming is the most difficult thing in computing, after all.

The doc string to the variable explained what it does just fine, so I
think your understanding needs a check-up.  (And I don't appreciated
being talked to in this tone, in case you wondered.)

>>   1) It allows testing without writing to $HOME.  (This has nothing to
>>   do with --batch -- testing happens in interactive Emacsen, too.)
>
> The user request is for non interactive sessions AFAIU.  And still I've
> to understand exactly what the user wants to solve.  Most importantly I
> feel I'm not alone here.

And I'm telling you that's not all that was requested for...  the third
time?  I'm not sure I'm getting any further here by repeating myself, so
I think I'll stop.

>>   2) It allows people to run an AOT Emacs without triggering further
>>   compilation.
>
> Sorry as changeset I meant 5fec9182db + f97993ee66.  I'm not against
> e245c4f226.
>
>> If you have a suggestion for an alternative change that achieves these
>> two things, I'm all ears.  (But if your objection is "people shouldn't
>> want those two things", I'm down to just two ears again.)
>
> Again 5fec9182db + f97993ee66 are IMO not useful / wrong, they were not
> discussed and are just a step backward.  The best change I can suggest
> is to revert them now.

You're just repeating that you're against it without proposing an
alternative, which is not helpful, so I think we've come to the of the
discussion on this point, too.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>> (Which reminds me -- one thing I've been pondering for a while is
>> whether "emacs -Q" should imply having the JIT off or not.  We use "-Q"
>> to get a repeatable Emacs for users, so it would make some sense to have
>> -Q now imply `inhibit-automatic-native-compilation'.
>
> Noooo please!! -Q has just one single clear meaning, let please not add
> other implications to it!!  The situation would become very quickly
> unmanageable.

Yes, that one clear meaning is:

              -Q, --quick
                      Similar to "-q --no-site-file --no-splash".  Also, avoid
                      processing X resources.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> The naming went first to `inhibit-native-compilation', this was really
>> just sooo wrong in so many levels :/ So many as many knobs we have to
>> control and disable different parts of the native compiler.  I can't
>> really think of one understanding all this machinery and then picking up
>> this name sorry, I can't.
>
> Naming is the most difficult thing in computing, after all.
>
> The doc string to the variable explained what it does just fine, so I
> think your understanding needs a check-up.  (And I don't appreciated
> being talked to in this tone, in case you wondered.)
>
>>>   1) It allows testing without writing to $HOME.  (This has nothing to
>>>   do with --batch -- testing happens in interactive Emacsen, too.)
>>
>> The user request is for non interactive sessions AFAIU.  And still I've
>> to understand exactly what the user wants to solve.  Most importantly I
>> feel I'm not alone here.
>
> And I'm telling you that's not all that was requested for...  the third
> time?  I'm not sure I'm getting any further here by repeating myself, so
> I think I'll stop.
>
>>>   2) It allows people to run an AOT Emacs without triggering further
>>>   compilation.
>>
>> Sorry as changeset I meant 5fec9182db + f97993ee66.  I'm not against
>> e245c4f226.
>>
>>> If you have a suggestion for an alternative change that achieves these
>>> two things, I'm all ears.  (But if your objection is "people shouldn't
>>> want those two things", I'm down to just two ears again.)
>>
>> Again 5fec9182db + f97993ee66 are IMO not useful / wrong, they were not
>> discussed and are just a step backward.  The best change I can suggest
>> is to revert them now.
>
> You're just repeating that you're against it without proposing an
> alternative, which is not helpful, so I think we've come to the of the
> discussion on this point, too.

You asked for an ameliorative change, I gave you one, that is just to
revert it.  If you don't like it I don't know what to do, I'll not
speculate on your psychology as you did with mine ;)

Best Regards

  Andrea

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: akrl@sdf.org,  rlb@defaultvalue.org,  monnier@iro.umontreal.ca,
>   david@tethera.net,  emacs-devel@gnu.org
> Date: Wed, 05 Oct 2022 16:27:38 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> and "I want to use a AOT-compiled Emacs, but don't do any further
> >> JIT compilation while running Emacs").
> >
> > This use case makes no sense at all.
> 
> It's been requested several times, whether it makes sense to you or not.

Where it was requested?  Please point me to those requests.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org
> Date: Wed, 05 Oct 2022 16:29:29 +0200
> 
> > I've the perception that this change was done without the full picture
> > in mind of how the native compiler and his mechanisms works.  As a
> > result the current naming is IMO just wrong, and as such is a step
> > backward the original state.
> 
> I don't know where you got that perception from.

It's clear as daylight that this is what happened.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: spwhitton@spwhitton.name,  rlb@defaultvalue.org,
>  monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org,
>  akrl@sdf.org
> Date: Wed, 05 Oct 2022 21:58:02 +0800
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> Maybe you should customize native-comp-async-jobs-number to a small
> >> enough value, like 1?  The default zero means half of your CPU's
> >> execution units.
> >
> > I will try that and see how it goes, thanks.
> 
> That makes the fans less loud (they are still noticable), but it also
> takes twice as long for the fans to subside, as expected.

What happens with those fans when you build Emacs with "make -jN"? do
they sound the same or louder?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  spwhitton@spwhitton.name,
>   rlb@defaultvalue.org,  monnier@iro.umontreal.ca,  david@tethera.net,
>   emacs-devel@gnu.org,  akrl@sdf.org
> Date: Wed, 05 Oct 2022 17:02:45 +0200
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > That makes the fans less loud (they are still noticable), but it also
> > takes twice as long for the fans to subside, as expected.
> >
> > Hope this data point ends up meaningful.
> 
> Yes.
> 
> Software is distributed pre-compiled for a reason -- because people run
> the software on hardware where compiling the software takes a long time.
> It's entirely reasonable for people to want to have a fully-built
> native-compiled Emacs installation on their laptops, without making that
> Emacs do any further JIT compilation.  (Except where necessary for
> trampolines, of course.)

Why would people want to have N files compiled, but not the N+1st
file?  How are the first N files different from the N+1st?

> This has been requested a number of times over several years now, but
> these requests have been ignored because apparently "people shouldn't
> want that".

That's an incorrect and unfair accusation, so please stop.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Where it was requested?  Please point me to those requests.

Sorry, I don't have a list -- people have been mentioning this for
years.

Po Lu seemed to request it just now, though.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: akrl@sdf.org,  rlb@defaultvalue.org,  monnier@iro.umontreal.ca,
>   david@tethera.net,  emacs-devel@gnu.org
> Date: Wed, 05 Oct 2022 19:08:00 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Where it was requested?  Please point me to those requests.
> 
> Sorry, I don't have a list -- people have been mentioning this for
> years.

I don't remember even a single request like that.  I suspect there's a
confusion of sorts.

> Po Lu seemed to request it just now, though.

He did?

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>> (Which reminds me -- one thing I've been pondering for a while is
>>> whether "emacs -Q" should imply having the JIT off or not.  We use "-Q"
>>> to get a repeatable Emacs for users, so it would make some sense to have
>>> -Q now imply `inhibit-automatic-native-compilation'.
>>
>> Noooo please!! -Q has just one single clear meaning, let please not add
>> other implications to it!!  The situation would become very quickly
>> unmanageable.
>
> Yes, that one clear meaning is:
>
>               -Q, --quick
>                       Similar to "-q --no-site-file --no-splash".  Also, avoid
>                       processing X resources.

Exactly, adding complexity involving the execution engine here is IMO
just searching for troubles (and having -Q not usable for debugging the
execution engine itself).  Let's please do not write patches solving
issues we never encountered or implementing unrequested features.

  Andrea

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2557 bytes --]

On Wed, Oct 05, 2022 at 07:47:58PM +0300, Eli Zaretskii wrote:
> > From: Lars Ingebrigtsen <larsi@gnus.org>
> > Cc: Eli Zaretskii <eliz@gnu.org>,  spwhitton@spwhitton.name,
> >   rlb@defaultvalue.org,  monnier@iro.umontreal.ca,  david@tethera.net,
> >   emacs-devel@gnu.org,  akrl@sdf.org
> > Date: Wed, 05 Oct 2022 17:02:45 +0200
> > 
> > Po Lu <luangruo@yahoo.com> writes:
> > 
> > > That makes the fans less loud (they are still noticable), but it also
> > > takes twice as long for the fans to subside, as expected.
> > >
> > > Hope this data point ends up meaningful.
> > 
> > Yes.
> > 
> > Software is distributed pre-compiled for a reason -- because people run
> > the software on hardware where compiling the software takes a long time.
> > It's entirely reasonable for people to want to have a fully-built
> > native-compiled Emacs installation on their laptops, without making that
> > Emacs do any further JIT compilation.  (Except where necessary for
> > trampolines, of course.)
> 
> Why would people want to have N files compiled, but not the N+1st
> file?  How are the first N files different from the N+1st?

Perhaps because 1500 <= N <= 2000 (or so) and the N+1st ist just one?
Perhaps because in the "normal case", the N+1st won't even happen?

The idea of a Debian package is to provide a baseline for those
just interested in using that software (with an easy ramp to
upgrade to actually hack at the sources and build a modified
version).

I actually install a few packages from source, those I "personally"
care about (Emacs is among them), But I couldn't possibly do it for
the > 2000 currently installed on my system.

The (Debian) packager's job is to make sure all that stuff works
nicely (and as repeatably as possible) together, and still receive
the necessary security fixes.

Perhaps it would make sense for the Debian Emacs packagers (Rob?)
to state their requirements in some more abstract way and work
from there.

As far as I understand, the wishes are:

 (a) deliver a package with (all? as many as possible? most? .eln
  pre-compiled
 (b) build Emacs in a way that is idempotent (and doesn't change
  overall system state
 (c) perhaps run tests (possibly, ideally part of b)

Did I forget anything?

Cheers
-- 
tomás
> 
> > This has been requested a number of times over several years now, but
> > these requests have been ignored because apparently "people shouldn't
> > want that".
> 
> That's an incorrect and unfair accusation, so please stop.
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Exactly, adding complexity involving the execution engine here is IMO
> just searching for troubles (and having -Q not usable for debugging the
> execution engine itself). 

You missed my point.  "-Q" doesn't have "one clear meaning" -- it's a
mish-mash of stuff, affecting how many things in Emacs works (like
Customize, X resources, packages, and so on).

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Wed, 5 Oct 2022 19:59:38 +0200
> From: <tomas@tuxteam.de>
> 
> > Why would people want to have N files compiled, but not the N+1st
> > file?  How are the first N files different from the N+1st?
> 
> Perhaps because 1500 <= N <= 2000 (or so) and the N+1st ist just one?

Then it isn't the same N, is it?

> Perhaps because in the "normal case", the N+1st won't even happen?

Why disable something that will never happen?

> The idea of a Debian package is to provide

I think the Debian case is not relevant, because they provide all the
*.eln files with the package you install (or so I understand).  So
either the user will only use those packages where *.eln are already
provided (in which case there's no reason to disable anything), or
they will want to use packages outside of the Debian distribution, in
which case I'm asking why they would like to give up on compiling
those additional ones.

> I actually install a few packages from source, those I "personally"
> care about (Emacs is among them), But I couldn't possibly do it for
> the > 2000 currently installed on my system.

What is "it" in "do it" above?  And what does the 2000 number counts
here? are you talking about 2000 Emacs Lisp packages that are not
bundled with the Emacs distribution and need to be installed
separately?  Or are you counting some other kind of "packages"?

> As far as I understand, the wishes are:
> 
>  (a) deliver a package with (all? as many as possible? most? .eln
>   pre-compiled
>  (b) build Emacs in a way that is idempotent (and doesn't change
>   overall system state
>  (c) perhaps run tests (possibly, ideally part of b)

I don't see how disabling JIT compilation is needed for these 3
purposes.  In particular, if all the *.eln files are already present,
there will be no need to recompile them.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3893 bytes --]

On Wed, Oct 05, 2022 at 09:28:55PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 5 Oct 2022 19:59:38 +0200
> > From: <tomas@tuxteam.de>
> > 
> > > Why would people want to have N files compiled, but not the N+1st
> > > file?  How are the first N files different from the N+1st?
> > 
> > Perhaps because 1500 <= N <= 2000 (or so) and the N+1st ist just one?
> 
> Then it isn't the same N, is it?

- Base .el(c)s pre-compiled. N.
- Additional .el (perhaps written by the user, perhaps
 downloaded from Emacswiki or what not) (jit) compiled,
 go to somewhere writable by the user (perhaps somewhere
 under ~/.emacs.d). 1. Or 2.

> > Perhaps because in the "normal case", the N+1st won't even happen?
> 
> Why disable something that will never happen?

...in the normal case. The user should be still capable of tackling
the not-that-normal case.

> > The idea of a Debian package is to provide
> 
> I think the Debian case is not relevant, because they provide all the
> *.eln files with the package you install (or so I understand).  So
> either the user will only use those packages where *.eln are already
> provided (in which case there's no reason to disable anything), or
> they will want to use packages outside of the Debian distribution, in
> which case I'm asking why they would like to give up on compiling
> those additional ones.

It seems we are in violent agreement, then. Whatever the user installs
"outside" the Debian packaging system is the user's business. It is
desirable that it works well together with the Debian-provided baseline
(and Debian tries to make that possible).

> > I actually install a few packages from source, those I "personally"
> > care about (Emacs is among them), But I couldn't possibly do it for
> > the > 2000 currently installed on my system.
> 
> What is "it" in "do it" above?

Installing software from source.

>   And what does the 2000 number counts
> here?

Debian packages: The Gnu compiler toolchain. The mail reader (mutt in my
case). A mail transfer agent (Exim). A Web browser. A web server. Lots
of different scripting languages. Networking stuff. SSH, rsync, git.
Qemu. Cross build tools. X and all those little helpers. R, Octave, I
don't know, all that stuff one needs to be happy :-)

>  are you talking about 2000 Emacs Lisp packages that are not
> bundled with the Emacs distribution and need to be installed
> separately?  Or are you counting some other kind of "packages"?

No, Debian packages.

> > As far as I understand, the wishes are:
> > 
> >  (a) deliver a package with (all? as many as possible? most? .eln
> >   pre-compiled
> >  (b) build Emacs in a way that is idempotent (and doesn't change
> >   overall system state
> >  (c) perhaps run tests (possibly, ideally part of b)
> 
> I don't see how disabling JIT compilation is needed for these 3
> purposes.  In particular, if all the *.eln files are already present,
> there will be no need to recompile them.

I don't know exactly how the Debian packaging for Emacs proceeds, but
I think we are talking about two distinct situations here:

(a) the binary install for the end user (for which the target is to
   end up with (some, most) .eln files pre-compiled and typically
   in /usr/lib or /usr/share, depending on whether those files are
   architecture-dependent (I think they are) or not

(b) the package build, which happens at the Debian developer's
   box to build the Debian package to be installed to achieve
   (a).

The question is whether the pre-compiling of the .eln happens at
(a) (i.e. package install time) or at (b). It seems Rob has opted
for the first (perhaps because there are dependencies which have
to be resolved "late"?).

Anyway, this "disabling of JIT" would be relevant for (b), if
at all (assuming I've been paying attention).

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Wed, 5 Oct 2022 20:56:00 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > > > Why would people want to have N files compiled, but not the N+1st
> > > > file?  How are the first N files different from the N+1st?
> > > 
> > > Perhaps because 1500 <= N <= 2000 (or so) and the N+1st ist just one?
> > 
> > Then it isn't the same N, is it?
> 
> - Base .el(c)s pre-compiled. N.
> - Additional .el (perhaps written by the user, perhaps
>  downloaded from Emacswiki or what not) (jit) compiled,
>  go to somewhere writable by the user (perhaps somewhere
>  under ~/.emacs.d). 1. Or 2.

Then I ask again: why wouldn't the user want to have the addition .el
compiled to .eln?

> > > Perhaps because in the "normal case", the N+1st won't even happen?
> > 
> > Why disable something that will never happen?
> 
> ...in the normal case. The user should be still capable of tackling
> the not-that-normal case.

The not-that-normal case being user's own *.el files?  Why wouldn't
the user want to compile them into *.eln in that user's own eln-cache?
Why would that user want to disable native-compilation instead?

> > > I actually install a few packages from source, those I "personally"
> > > care about (Emacs is among them), But I couldn't possibly do it for
> > > the > 2000 currently installed on my system.
> > 
> > What is "it" in "do it" above?
> 
> Installing software from source.
> 
> >   And what does the 2000 number counts
> > here?
> 
> Debian packages: The Gnu compiler toolchain. The mail reader (mutt in my
> case). A mail transfer agent (Exim). A Web browser. A web server. Lots
> of different scripting languages. Networking stuff. SSH, rsync, git.
> Qemu. Cross build tools. X and all those little helpers. R, Octave, I
> don't know, all that stuff one needs to be happy :-)

How is this relevant to the issue at hand?  Only Emacs comes with
*.eln files and can use them.  Why are we talking about all the other
packages?

> > I don't see how disabling JIT compilation is needed for these 3
> > purposes.  In particular, if all the *.eln files are already present,
> > there will be no need to recompile them.
> 
> I don't know exactly how the Debian packaging for Emacs proceeds, but
> I think we are talking about two distinct situations here:
> 
> (a) the binary install for the end user (for which the target is to
>    end up with (some, most) .eln files pre-compiled and typically
>    in /usr/lib or /usr/share, depending on whether those files are
>    architecture-dependent (I think they are) or not
> 
> (b) the package build, which happens at the Debian developer's
>    box to build the Debian package to be installed to achieve
>    (a).
> 
> The question is whether the pre-compiling of the .eln happens at
> (a) (i.e. package install time) or at (b). It seems Rob has opted
> for the first (perhaps because there are dependencies which have
> to be resolved "late"?).
> 
> Anyway, this "disabling of JIT" would be relevant for (b), if
> at all (assuming I've been paying attention).

For case (b) Rob said that the workspace is thrown away after the
build, so whether the *.eln files are or aren't built there makes no
difference.  (And if the package build is done in batch mode, the
async compilation is disabled there by default.)

So, once again, I see no reason to disable JIT compilation for these
use cases.

Re: Suppressing native compilation (short and long term)

[Gregory Heytings]

>
> I think the Debian case is not relevant, because they provide all the 
> *.eln files with the package you install (or so I understand).
>

I'm not 100% sure what they will do (the version of Emacs distributed by 
Emacs is at the moment still 27.1), but my guess is that this is not what 
they will do.  There are, in fact, two cases:

1. When a user does "apt install emacs", this actually installs (by 
default) the "emacs-gtk" package, which contains only the "emacs" binary, 
and triggers the installation of two other packages: "emacs-common", which 
contains the precompiled elc files (and the files in etc), and 
"emacs-bin-common", which contains the emacsclient, etags, ctags and 
ebrowse binaries.  I would guess that in this case, when the user chooses 
to install an emacs with native compilation enabled (say 
"emacs-gtk-native"), a third package will be installed, say 
"emacs-common-native", containing the precompiled eln files.

2. When a user does "apt install elpa-magit" (for example), the package 
only contains el files.  These files are compiled to elc files during 
installation (and stored in a shared directory, namely 
/usr/share/emacs/site-lisp).  I would guess that, when the installed emacs 
binary is one with native compilation enabled, these el files will be 
compiled to eln files during installation, and stored in a shared 
directory, too.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Wed, 05 Oct 2022 19:29:45 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: tomas@tuxteam.de, emacs-devel@gnu.org
> 
> > I think the Debian case is not relevant, because they provide all the 
> > *.eln files with the package you install (or so I understand).
> 
> I'm not 100% sure what they will do (the version of Emacs distributed by 
> Emacs is at the moment still 27.1), but my guess is that this is not what 
> they will do.  There are, in fact, two cases:
> 
> 1. When a user does "apt install emacs", this actually installs (by 
> default) the "emacs-gtk" package, which contains only the "emacs" binary, 
> and triggers the installation of two other packages: "emacs-common", which 
> contains the precompiled elc files (and the files in etc), and 
> "emacs-bin-common", which contains the emacsclient, etags, ctags and 
> ebrowse binaries.  I would guess that in this case, when the user chooses 
> to install an emacs with native compilation enabled (say 
> "emacs-gtk-native"), a third package will be installed, say 
> "emacs-common-native", containing the precompiled eln files.
> 
> 2. When a user does "apt install elpa-magit" (for example), the package 
> only contains el files.  These files are compiled to elc files during 
> installation (and stored in a shared directory, namely 
> /usr/share/emacs/site-lisp).  I would guess that, when the installed emacs 
> binary is one with native compilation enabled, these el files will be 
> compiled to eln files during installation, and stored in a shared 
> directory, too.

I agree with the above.  But then why did you say that my description
was inaccurate?  What you described matches what I wrote perfectly:
the end result, after installing Emacs and elpa-magit, is that the
*.eln files are available for all the *.el files and stored in shared
directories.  Whether those *.eln files are produced on the system
where the package is packaged or as part of the installation is not
important; the important part is that all the *.eln files are there
after the installation, and therefore there's no need to disable JIT
compilation.  And yet Rob says that he thinks there _is_ a need for
disabling it.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2824 bytes --]

On Wed, Oct 05, 2022 at 10:04:17PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 5 Oct 2022 20:56:00 +0200
> > From: tomas@tuxteam.de
> > Cc: emacs-devel@gnu.org
> > 
> > > > > Why would people want to have N files compiled, but not the N+1st
> > > > > file?  How are the first N files different from the N+1st?
> > > > 
> > > > Perhaps because 1500 <= N <= 2000 (or so) and the N+1st ist just one?
> > > 
> > > Then it isn't the same N, is it?
> > 
> > - Base .el(c)s pre-compiled. N.
> > - Additional .el (perhaps written by the user, perhaps
> >  downloaded from Emacswiki or what not) (jit) compiled,
> >  go to somewhere writable by the user (perhaps somewhere
> >  under ~/.emacs.d). 1. Or 2.
> 
> Then I ask again: why wouldn't the user want to have the addition .el
> compiled to .eln?

I think nobody is proposing to prevent the user from doing that.

> > > > Perhaps because in the "normal case", the N+1st won't even happen?
> > > 
> > > Why disable something that will never happen?
> > 
> > ...in the normal case. The user should be still capable of tackling
> > the not-that-normal case.
> 
> The not-that-normal case being user's own *.el files?  Why wouldn't
> the user want to compile them into *.eln in that user's own eln-cache?
> Why would that user want to disable native-compilation instead?

See above. This must be (part of) the misunderstanding. See above
my text, perhaps it was unclear:

  "Additional .el [...] (jit) compiled go to somewhere writable
   by the user"

We are in violent agreement here, I think.

> > > > I actually install a few packages from source, those I "personally"
> > > > care about [...]

> How is this relevant to the issue at hand?  Only Emacs comes with
> *.eln files and can use them.  Why are we talking about all the other
> packages?

Just trying to explain how the Debian packaging works and what it is
good for. Your questions seem to suggest that this is somewhat alien
to you (but my impression might be wrong, too).

> > > I don't see how disabling JIT compilation is needed for these 3
> > > purposes.  In particular, if all the *.eln files are already present,
> > > there will be no need to recompile them.

[...]

> For case (b) Rob said that the workspace is thrown away after the
> build, so whether the *.eln files are or aren't built there makes no
> difference.  (And if the package build is done in batch mode, the
> async compilation is disabled there by default.)

As I understant, this might work well if it's possible to redirect/
restrict those target directories to specific places, yes. But Rob
will know much better than I could.

> So, once again, I see no reason to disable JIT compilation for these
> use cases.

Perhaps we can get away with it, yes.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Gregory Heytings]

>
> I agree with the above.  But then why did you say that my description 
> was inaccurate?
>

I did not say that, or at least I did not mean to say that.  I only tried 
to clarify something that seemed unclear, namely that eln files are not 
present in the package, but generated at installation time.  It's a subtle 
difference, but it seems important in the current discussion.

>
> What you described matches what I wrote perfectly: the end result, after 
> installing Emacs and elpa-magit, is that the *.eln files are available 
> for all the *.el files and stored in shared directories.  Whether those 
> *.eln files are produced on the system where the package is packaged or 
> as part of the installation is not important; the important part is that 
> all the *.eln files are there after the installation, and therefore 
> there's no need to disable JIT compilation.  And yet Rob says that he 
> thinks there _is_ a need for disabling it.
>

If I understand Rob's initial message correctly, this subtle difference is 
relevant:

>
> Perhaps relevant in other contexts, but at least in the context of 
> Debian work, we'd like to be able to suppress all native compilation in 
> some contexts, or more specifically all writes to HOME (or really, to 
> avoid any implicit file manipulations[1]).
>
> One key case is package builds and package installs (whether the emacs 
> packages themselves, or any of the various emacs add-on packages (e.g. 
> elpa-*)).
>
> For package builds, HOME is typically set to /nonexistent (or similar), 
> and for package installs we don't want the install commands (preinst, 
> postinst, etc.), which are running as root, to write anything to /root/.
>

IIUC, what Rob wants is that

(1) on Debian systems on which Emacs packages are built, the build process 
is running as root, and nothing should be written in /root, and

(2) on user systems on which elpa-* packages are installed, the 
installation process, during which el files are compiled to elc and eln 
files, is running as root, and it should not write anything in /root.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Also repeatable testing are most likely executed in batch mode and...
>
> ...surprise surprise deferred compilation is *already* *disabled* in
> this mode!!

I forgot to mention that while this is (strictly speaking) true, if the
--batch Emacs redefines any built-in functions, the trampoline files
will get written to ~/.emacs.d/eln-cache/.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Exactly, adding complexity involving the execution engine here is IMO
>> just searching for troubles (and having -Q not usable for debugging the
>> execution engine itself). 
>
> You missed my point.  "-Q" doesn't have "one clear meaning" -- it's a
> mish-mash of stuff, affecting how many things in Emacs works (like
> Customize, X resources, packages, and so on).

Still I think none of these as anything to do with the execution engine
(and should not).

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>> You missed my point.  "-Q" doesn't have "one clear meaning" -- it's a
>> mish-mash of stuff, affecting how many things in Emacs works (like
>> Customize, X resources, packages, and so on).
>
> Still I think none of these as anything to do with the execution engine
> (and should not).

Did I say they did?  You claimed that "-Q" has "one clear meaning", and
that's false.  "-Q" means, sort of, "without any local changes", which
is really wishy-washy semantics.

Which reminds me of another thing -- was there a reason that --batch
implies "avoid most JIT compilation", like it does now, by the way?
It's always seemed pretty odd to me, because JITting could well be quite
useful when doing batch stuff, and there's no handy way to switch it on
when doing --batch.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Also repeatable testing are most likely executed in batch mode and...
>>
>> ...surprise surprise deferred compilation is *already* *disabled* in
>> this mode!!
>
> I forgot to mention that while this is (strictly speaking) true,

Not sure what you mean by strictly speaking, I think is just true.

> if the
> --batch Emacs redefines any built-in functions, the trampoline files
> will get written to ~/.emacs.d/eln-cache/.

Indeed, and I don't find anything surprising with that.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>>> Also repeatable testing are most likely executed in batch mode and...
>>>
>>> ...surprise surprise deferred compilation is *already* *disabled* in
>>> this mode!!
>>
>> I forgot to mention that while this is (strictly speaking) true,
>
> Not sure what you mean by strictly speaking, I think is just true.
>
>> if the
>> --batch Emacs redefines any built-in functions, the trampoline files
>> will get written to ~/.emacs.d/eln-cache/.
>
> Indeed, and I don't find anything surprising with that.

I'm sure you don't.  But you claimed that --batch was the panacea for
repeatable testing -- and it's not, since the second time you run the
same test, the trampoline files are created.  I.e., not a repeat of the
same test.

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> What happens with those fans when you build Emacs with "make -jN"? do
> they sound the same or louder?

Louder, but building Emacs doesn't happen in the background
automatically.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>>> Also repeatable testing are most likely executed in batch mode and...
>>>>
>>>> ...surprise surprise deferred compilation is *already* *disabled* in
>>>> this mode!!
>>>
>>> I forgot to mention that while this is (strictly speaking) true,
>>
>> Not sure what you mean by strictly speaking, I think is just true.
>>
>>> if the
>>> --batch Emacs redefines any built-in functions, the trampoline files
>>> will get written to ~/.emacs.d/eln-cache/.
>>
>> Indeed, and I don't find anything surprising with that.
>
> I'm sure you don't.  But you claimed that --batch was the panacea for
> repeatable testing

Did I??

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>>>>> Also repeatable testing are most likely executed in batch mode and...
>>>>>
>>>>> ...surprise surprise deferred compilation is *already* *disabled* in
>>>>> this mode!!

[...]

>> I'm sure you don't.  But you claimed that --batch was the panacea for
>> repeatable testing
>
> Did I??

Yes.  *points*

Re: Suppressing native compilation (short and long term)

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Po Lu seemed to request it just now, though.

Nope, my solution is to wait the 30 minutes for the fans to subside
after startup, every time I update Emacs.

I think the end result and time spent is more-or-less the same as a
NATIVE_FULL_AOT build of Emacs.

Thanks.

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> He did?

Not really, but I build my own Emacs.  The difference may be more
meaningful for packagers.

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  rlb@defaultvalue.org,
>>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org
>> Date: Wed, 05 Oct 2022 16:29:29 +0200
>> 
>> > I've the perception that this change was done without the full picture
>> > in mind of how the native compiler and his mechanisms works.  As a
>> > result the current naming is IMO just wrong, and as such is a step
>> > backward the original state.
>> 
>> I don't know where you got that perception from.
>
> It's clear as daylight that this is what happened.

Since the argument seems to be going nowhere, could we first revert the
change in question, and then ask Rob Browning exactly what the problem
is with `native-comp-deferred-compilation'?

I did not figure that out reading this thread.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>> You missed my point.  "-Q" doesn't have "one clear meaning" -- it's a
>>> mish-mash of stuff, affecting how many things in Emacs works (like
>>> Customize, X resources, packages, and so on).
>>
>> Still I think none of these as anything to do with the execution engine
>> (and should not).
>
> Did I say they did?

No you didn't nor I claimed you did.  I'm saying -Q does control a
totally different area of Emacs that is not the execution engine.

For instance it does not affect bytecode execution, consequentially I
don't see why it should affect native code execution.

> You claimed that "-Q" has "one clear meaning", and
> that's false.  "-Q" means, sort of, "without any local changes", which
> is really wishy-washy semantics.

Sorry for me this meaning is clear.

> Which reminds me of another thing -- was there a reason that --batch
> implies "avoid most JIT compilation", like it does now, by the way?
> It's always seemed pretty odd to me, because JITting could well be quite
> useful when doing batch stuff, and there's no handy way to switch it on
> when doing --batch.

The rational is that "tipically" batch executions are short in time, so
spawning there native async compilation would be a waste of cycles if
they do not complete in time.  I think no one has statistical prove of
this, but the rational was at least discussed and I believe agreed here.

  Andrea

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> The rational is that "tipically" batch executions are short in time, so
> spawning there native async compilation would be a waste of cycles if
> they do not complete in time.  I think no one has statistical prove of
> this, but the rational was at least discussed and I believe agreed here.

Sure, sounds reasonable as a default, at least.  But perhaps there
should be a way to allow JIT in --batch?  It looks like currently the
only way is to set `noninteractive', which has wider side effects.

Yet Another Variable here would be possible, but perhaps some other
mechanism would be nicer.  Hm...  Perhaps
inhibit-automatic-native-compilation should be refashioned into
something else, like `native-compilation-types' or something, and would
default to 'all in non-batch, but could be 'force to force it even in
--batch?  (And possibly also 'trampoline to save trampolines without
doing any more JIT-it, like Stefan M sort of suggested.)

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> Since the argument seems to be going nowhere, could we first revert the
> change in question, and then ask Rob Browning exactly what the problem
> is with `native-comp-deferred-compilation'?
>
> I did not figure that out reading this thread.

I don't know whether this was Rob's problem, but I found it problematic
that there was no way to not write to ~/.emacs.d/eln-cache.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>>>>> Also repeatable testing are most likely executed in batch mode and...
>>>>>>
>>>>>> ...surprise surprise deferred compilation is *already* *disabled* in
>>>>>> this mode!!
>
> [...]
>
>>> I'm sure you don't.  But you claimed that --batch was the panacea for
>>> repeatable testing
>>
>> Did I??
>
> Yes.  *points*

Where?

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>>>>>>> Also repeatable testing are most likely executed in batch mode and...
>>>>>>>
>>>>>>> ...surprise surprise deferred compilation is *already* *disabled* in
>>>>>>> this mode!!

[...]

> Where?

There.

But if you were just doing another non-sequitur -- that you didn't mean
that "deferred compilation" and --batch is related to repeatable
testing, that's fine.  You were just making conversation and not
claiming anything?

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>>>>>>> Also repeatable testing are most likely executed in batch mode and...
>>>>>>>>
>>>>>>>> ...surprise surprise deferred compilation is *already* *disabled* in
>>>>>>>> this mode!!
>
> [...]
>
>> Where?
>
> There.

Oh my!  FYI this is what we wrote:

> >>   . understand why and in which situations they may need it
> >
> > Doing repeatable testing is one obvious situation.
> 
> I think the two previously mentioned knobs should be sufficient for
> repeatable testing no?

As you can see I said that the *two* mentioned knobs should be
sufficient for test repeatability, not that "--batch is the panacea for
repeatable testing".

> But if you were just doing another non-sequitur -- that you didn't mean
> that "deferred compilation" and --batch is related to repeatable
> testing, that's fine.  You were just making conversation and not
> claiming anything?

Sorry are you serious?

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> As you can see I said that the *two* mentioned knobs should be
> sufficient for test repeatability, not that "--batch is the panacea for
> repeatable testing".

The two mentioned knobs are `native-comp-deferred-compilation' and
`load-no-native'?  And as I've explained, those are not sufficient,
because trampolines are written to ~/.emacs.d/eln-cache even those two
switched off.

And you're being revisionist, because this is what you said:

> I think the two previously mentioned knobs should be sufficient for
> repeatable testing no?
> 
> Also repeatable testing are most likely executed in batch mode and...
> 
> ...surprise surprise deferred compilation is *already* *disabled* in
> this mode!!
> 
> I've the impression no one mentioned this small detail in this huge
> thread, but still we have installed changes to disable a feature for
> debian pkg installation that is in fact already disabled :( :(

The latter is simply not true.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> As you can see I said that the *two* mentioned knobs should be
>> sufficient for test repeatability, not that "--batch is the panacea for
>> repeatable testing".
>
> The two mentioned knobs are `native-comp-deferred-compilation' and
> `load-no-native'?

yes

> And as I've explained, those are not sufficient,
> because trampolines are written to ~/.emacs.d/eln-cache even those two
> switched off.

Sorry in my limited experience in debugging Emacs native code execution
those knobs are the one to be used for this case.  Sure trampolines add
a state (there are probably other states Emacs can generate I'm not
aware) but that said I've never had test repeatability issues with
trampolines.

Anyway I thought we were discussing / trying to solve mainly the user
request.  Is this about test repeatability?  Did we had repeatability
issues with native code I'm not aware?

Repeatable testing [was: Suppressing native compilation...]

[tomas]

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

On Thu, Oct 06, 2022 at 12:34:50AM +0000, Andrea Corallo wrote:
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Andrea Corallo <akrl@sdf.org> writes:

[...]

> > I'm sure you don't.  But you claimed that --batch was the panacea for
> > repeatable testing
> 
> Did I??

Now who did or did not isn't that important. More important is:
how would one get about repeatable testing?

Throw away $HOME?

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Repeatable testing [was: Suppressing native compilation...]

[Emanuel Berg]

tomas wrote:

> how would one get about repeatable testing?

One would define what that is in unambiguous terms ...

Then one would work to realize that bottom-up ...

With no shortcuts to the top!

-- 
underground experts united
https://dataswamp.org/~incal

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Wed, 05 Oct 2022 20:04:34 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: tomas@tuxteam.de, emacs-devel@gnu.org
> 
> IIUC, what Rob wants is that
> 
> (1) on Debian systems on which Emacs packages are built, the build process 
> is running as root, and nothing should be written in /root, and

For this case, Rob said the workspace is thrown away after the build,
so whether the *.eln files are or aren't produced is not relevant.
But if, for some reason, the *.eln files could survive the deletion of
the workspace, then tweaking native-comp-eln-load-path to have /tmp at
the beginning should handle that as well.

> (2) on user systems on which elpa-* packages are installed, the 
> installation process, during which el files are compiled to elc and eln 
> files, is running as root, and it should not write anything in /root.

Since the elpa-* package installation process is supposed to leave the
users of the system with ready *.eln files from the packages being
installed, the installation process should NOT disable
native-compilation.  Instead, it should tweak
native-comp-eln-load-path so that it includes the shared directory
where Debian wants to have the *.eln files of ELPA packages instead or
ahead of the user's eln-cache directory.  This will produce the *.eln
files in the place where Debian wants them, and allow users to use
those packages without worrying about compilation.

IOW, an option to disable native-compilation is not TRT in this case.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: spwhitton@spwhitton.name,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org,
>   akrl@sdf.org
> Date: Thu, 06 Oct 2022 08:34:03 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What happens with those fans when you build Emacs with "make -jN"? do
> > they sound the same or louder?
> 
> Louder, but building Emacs doesn't happen in the background
> automatically.

It does happen in the background, in the sense that Make launches
several processes in parallel, each one of which running in the
background.

As for "automatically", the JIT compilation is not automatic, either.
If Emacs is idle, it will not suddenly decide to compile.

Anyway, what would you suggest as a solution for the problem you
perceive with JIT native-compilation, which would refrain from being
"in the background" and "automatic"?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  akrl@sdf.org,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org
> Date: Thu, 06 Oct 2022 08:40:36 +0800
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Po Lu seemed to request it just now, though.
> 
> Nope, my solution is to wait the 30 minutes for the fans to subside
> after startup, every time I update Emacs.

I'd like to point out that the situation with frequent updates of
Emacs that require recompilation of *.eln files is peculiar to Emacs
development on the master branch, and will generally rare if ever
happen for "normal" users, and even for us developers on the release
branch.

Like I said before, the pattern of JIT compilation in "normal" usage
is that it happens for a few minutes after installing a new version of
Emacs, and thereafter happens only rarely.

> I think the end result and time spent is more-or-less the same as a
> NATIVE_FULL_AOT build of Emacs.

I think NATIVE_FULL_AOT takes more time, because it compiles much more
than an average user will ever use.  (But I don't object to supporting
such a build for people who want it.)

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  akrl@sdf.org,
>   rlb@defaultvalue.org,  monnier@iro.umontreal.ca,  david@tethera.net,
>   emacs-devel@gnu.org
> Date: Thu, 06 Oct 2022 08:44:21 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> Since the argument seems to be going nowhere, could we first revert the
> change in question, and then ask Rob Browning exactly what the problem
> is with `native-comp-deferred-compilation'?

I'm still discussing with Rob his needs.  I hope to eventually
understand their needs.  For now my only conclusion is that they need
to tweak native-comp-eln-load-path in some situations, to control
where the *.eln files are deposited.  Which is easy and supported
already, AFAIU.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, rlb@defaultvalue.org,
>         monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org
> Date: Thu, 06 Oct 2022 00:45:49 +0000
> 
> > Which reminds me of another thing -- was there a reason that --batch
> > implies "avoid most JIT compilation", like it does now, by the way?
> > It's always seemed pretty odd to me, because JITting could well be quite
> > useful when doing batch stuff, and there's no handy way to switch it on
> > when doing --batch.
> 
> The rational is that "tipically" batch executions are short in time, so
> spawning there native async compilation would be a waste of cycles if
> they do not complete in time.  I think no one has statistical prove of
> this, but the rational was at least discussed and I believe agreed here.

Yes, we discussed that, and yes, we agreed.  And I don't think the
decision was wrong.  For starters, it would be strange to delay
exiting Emacs in batch so it could wait for the async compilation to
complete.  Moreover, in most cases waiting will not help, since the
job of the batch invocation would have been long done anyway, so the
produced *.eln files will not be loaded in time to make any
difference.

We do have batch commands to compile *.eln files explicitly (this is
used when a release tarball is built).  This was deemed to be enough,
at least at the time we discussed these issues.  I don't yet see why
we'd need to revisit that decision.  When does it cause problems?

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 185 bytes --]

On Thu, Oct 06, 2022 at 08:28:04AM +0300, Eli Zaretskii wrote:

[...]

> IOW, an option to disable native-compilation is not TRT in this case.

s/TRT/necessary/

;-)

-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  akrl@sdf.org,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org
> Date: Thu, 06 Oct 2022 02:56:54 +0200
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > Since the argument seems to be going nowhere, could we first revert the
> > change in question, and then ask Rob Browning exactly what the problem
> > is with `native-comp-deferred-compilation'?
> >
> > I did not figure that out reading this thread.
> 
> I don't know whether this was Rob's problem, but I found it problematic
> that there was no way to not write to ~/.emacs.d/eln-cache.

But there was such a way: modify native-comp-eln-load-path to have
another directory at the front.  Which, it seems, is what Rob actually
wants, at least in some cases, because they _want_ to keep the *.eln
files, just not in the user's home directory.

Re: Repeatable testing

[Eli Zaretskii]

> Date: Thu, 6 Oct 2022 06:32:08 +0200
> From: <tomas@tuxteam.de>
> 
> Now who did or did not isn't that important. More important is:
> how would one get about repeatable testing?
> 
> Throw away $HOME?

See what we do in the Emacs test suite: we point HOME to a
non-existent directory.  This is useful for other reasons besides
native-compilation, btw.

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Anyway, what would you suggest as a solution for the problem you
> perceive with JIT native-compilation, which would refrain from being
> "in the background" and "automatic"?

The solution I would propose would be to defer JIT native-compilation
until the computer is on AC power, as determined by battery.el.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: spwhitton@spwhitton.name,  rlb@defaultvalue.org,
>   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org,
>   akrl@sdf.org
> Date: Thu, 06 Oct 2022 15:11:19 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Anyway, what would you suggest as a solution for the problem you
> > perceive with JIT native-compilation, which would refrain from being
> > "in the background" and "automatic"?
> 
> The solution I would propose would be to defer JIT native-compilation
> until the computer is on AC power, as determined by battery.el.

We could have such a feature, but how to implement it?  If we use a
timer for that, the timer itself will drain the battery.  And if defer
it to the next invocation of the bytecode, we might never compile,
because who can guarantee that the laptop is on AC when some arbitrary
bytecode is executed?  Any other ideas?

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

On Thu, Oct 06, 2022 at 11:03:47AM +0300, Eli Zaretskii wrote:
> > From: Po Lu <luangruo@yahoo.com>
> > Cc: spwhitton@spwhitton.name,  rlb@defaultvalue.org,
> >   monnier@iro.umontreal.ca,  david@tethera.net,  emacs-devel@gnu.org,
> >   akrl@sdf.org
> > Date: Thu, 06 Oct 2022 15:11:19 +0800
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > > Anyway, what would you suggest as a solution for the problem you
> > > perceive with JIT native-compilation, which would refrain from being
> > > "in the background" and "automatic"?
> > 
> > The solution I would propose would be to defer JIT native-compilation
> > until the computer is on AC power, as determined by battery.el.
> 
> We could have such a feature, but how to implement it?  If we use a
> timer for that, the timer itself will drain the battery.

Guessing from a previous mail by you, the correct way to achieve this
is to compiler results to a non-existent directory, right?

To me that feels strange, but I could live with that. Can it be documented?

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> We could have such a feature, but how to implement it?  If we use a
> timer for that, the timer itself will drain the battery.

I think display-battery-mode users (I am one such user) will not agree
with that assessment.

> And if defer it to the next invocation of the bytecode, we might never
> compile, because who can guarantee that the laptop is on AC when some
> arbitrary bytecode is executed?

We could push it onto a list of files to native compile, the files in
which are then compiled once we detect the laptop starts to run on AC
power.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Thu, 6 Oct 2022 11:02:44 +0200
> From: <tomas@tuxteam.de>
> 
> > > The solution I would propose would be to defer JIT native-compilation
> > > until the computer is on AC power, as determined by battery.el.
> > 
> > We could have such a feature, but how to implement it?  If we use a
> > timer for that, the timer itself will drain the battery.
> 
> Guessing from a previous mail by you, the correct way to achieve this
> is to compiler results to a non-existent directory, right?

No, this is a completely different use case.

We _can_ disable native-compilation (and had this ability for a while,
since Emacs 28 development); the issue here is how to re-enable it
again "when computer is on AC power".

> To me that feels strange, but I could live with that. Can it be documented?

What would you like to be documented, exactly?

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org,  akrl@sdf.org
> Date: Thu, 06 Oct 2022 17:52:35 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We could have such a feature, but how to implement it?  If we use a
> > timer for that, the timer itself will drain the battery.
> 
> I think display-battery-mode users (I am one such user) will not agree
> with that assessment.

I didn't invent that, I've heard laptop users complain about timers.

But okay, if it's acceptable to run a timer in order to re-enable
compilation when AC power is plugged in, I'm okay with such an
optional feature.  Patches are welcome.

> > And if defer it to the next invocation of the bytecode, we might never
> > compile, because who can guarantee that the laptop is on AC when some
> > arbitrary bytecode is executed?
> 
> We could push it onto a list of files to native compile, the files in
> which are then compiled once we detect the laptop starts to run on AC
> power.

You will see that comp.el already has a queue of files that await
compilation.  The new feature will newed to plug itself into that
mechanism, I think.  Unless Andrea has a better idea.

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Po Lu <luangruo@yahoo.com>
>> Cc: emacs-devel@gnu.org,  akrl@sdf.org
>> Date: Thu, 06 Oct 2022 17:52:35 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > We could have such a feature, but how to implement it?  If we use a
>> > timer for that, the timer itself will drain the battery.
>> 
>> I think display-battery-mode users (I am one such user) will not agree
>> with that assessment.
>
> I didn't invent that, I've heard laptop users complain about timers.

For my education: why timer are computational expensive?

> But okay, if it's acceptable to run a timer in order to re-enable
> compilation when AC power is plugged in, I'm okay with such an
> optional feature.  Patches are welcome.
>
>> > And if defer it to the next invocation of the bytecode, we might never
>> > compile, because who can guarantee that the laptop is on AC when some
>> > arbitrary bytecode is executed?
>> 
>> We could push it onto a list of files to native compile, the files in
>> which are then compiled once we detect the laptop starts to run on AC
>> power.
>
> You will see that comp.el already has a queue of files that await
> compilation.  The new feature will newed to plug itself into that
> mechanism, I think.  Unless Andrea has a better idea.

I'd suggest this way as well.

BR

  Andrea

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Po Lu <luangruo@yahoo.com>, emacs-devel@gnu.org
> Date: Thu, 06 Oct 2022 10:41:14 +0000
> 
> >> I think display-battery-mode users (I am one such user) will not agree
> >> with that assessment.
> >
> > I didn't invent that, I've heard laptop users complain about timers.
> 
> For my education: why timer are computational expensive?

They are not, at least not normally.  But they interfere with the OS's
attempts to conserve power, because Emacs is not idle, as far as the
OS is concerned -- it runs some stuff once every so-and-so seconds.

Re: Suppressing native compilation (short and long term)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]

On Thu, Oct 06, 2022 at 01:13:40PM +0300, Eli Zaretskii wrote:
> > Date: Thu, 6 Oct 2022 11:02:44 +0200
> > From: <tomas@tuxteam.de>
> > 
> > > > The solution I would propose would be to defer JIT native-compilation
> > > > until the computer is on AC power, as determined by battery.el.
> > > 
> > > We could have such a feature, but how to implement it?  If we use a
> > > timer for that, the timer itself will drain the battery.
> > 
> > Guessing from a previous mail by you, the correct way to achieve this
> > is to compiler results to a non-existent directory, right?
> 
> No, this is a completely different use case.

Now I'm confused. Does the non-existence of a target directory
disable compilation or just the writing of the compilation's
results?

> We _can_ disable native-compilation (and had this ability for a while,
> since Emacs 28 development); the issue here is how to re-enable it
> again "when computer is on AC power".
> 
> > To me that feels strange, but I could live with that. Can it be documented?
> 
> What would you like to be documented, exactly?

that pointing HOME a non-existing directory is the way to either
inhibit JIT compilation or writing of the compilation results to
disk (depending on the answer above).

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Anyway I thought we were discussing / trying to solve mainly the user
> request.  Is this about test repeatability?  Did we had repeatability
> issues with native code I'm not aware?

There were two things requested, and one of them was related to test
repeatability (which then also requires not writing anything to
eln-cache).

It's not a matter of specific issues with native code, but a general
principle -- you should be able to run the same test twice and have it
trigger all the same code paths.  That's just a general maintenance
principle.

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> Date: Thu, 6 Oct 2022 13:49:44 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> On Thu, Oct 06, 2022 at 01:13:40PM +0300, Eli Zaretskii wrote:
> > > Date: Thu, 6 Oct 2022 11:02:44 +0200
> > > From: <tomas@tuxteam.de>
> > > 
> > > > > The solution I would propose would be to defer JIT native-compilation
> > > > > until the computer is on AC power, as determined by battery.el.
> > > > 
> > > > We could have such a feature, but how to implement it?  If we use a
> > > > timer for that, the timer itself will drain the battery.
> > > 
> > > Guessing from a previous mail by you, the correct way to achieve this
> > > is to compiler results to a non-existent directory, right?
> > 
> > No, this is a completely different use case.
> 
> Now I'm confused. Does the non-existence of a target directory
> disable compilation or just the writing of the compilation's
> results?

??? What does the existence of the target directory have to do with
the use case of laptop running on batteries?

> > What would you like to be documented, exactly?
> 
> that pointing HOME a non-existing directory is the way to either
> inhibit JIT compilation or writing of the compilation results to
> disk (depending on the answer above).

That is not something we should advertise, I think, because it
shouldn't be needed.  We do that in the test suite, but not in order
to disable native-compilation.

Whether there are valid use cases where users would need to disable
native-compilation is still an open question.  The use case presented
by Po Lu does not require disabling native-compilation, it requires
_delaying_ it until some opportune moment.

Re: Suppressing native compilation (short and long term)

[Michael Welsh Duggan]

In an effort to try and bridge the understanding gap, I'm going to
contribute my current understanding of Debian's requirements, in the
hopes that we might be able to identify any misunderstandings.
Apologies in advance if this just retreads old ground and doesn't add
anything useful to the conversation.

From Debian's point of view, there are a few scenarios under which emacs
is run:

1) When building emacs (used as part of emacs bootstrap) 
2) When installing emacs via dpkg (maybe?  Not certain)
3) When building an emacs package (maybe?  Not certain)
4) When installing an emacs package via dpkg
5) When a user runs emacs

Here are the constraints that I have intuited from the conversation:

a) Case 1 builds .eln files, which will be packaged and installed in
   case 2.

b) Cases 1 and 3 normally happen on a Debian build machine.  These do
   *not* want emacs to write anything to $HOME, though writing to a
   temporary location that will subsequently be thrown away is okay.

c) Case 3 might not require running emacs at all, but I can imagine that
   emacs might be run as part of the build process to auto-generate some
   files.

d) Cases 2, 4, and 5 occur on a Debian user's machine.

e) Cases 2 and 4 are run under root (or similar) and should not write to
   $HOME.

f) Case 2 will install the .eln files packaged in case 1 into a
   world-readable, read-only location.  An Emacs run in case 5 will
   include that location in its native file search path.

g) Case 4 might run emacs to build .elc and .eln files for the package's
   .el files and place them in world-readable, read-only locations.  An
   Emacs run in case 5 will include that location in its native file
   search path.

h) Case 5 should read .eln files from the world-readable, read-only
   locations mentioned above, when possible, but otherwise should do
   native compilation and store the generated .eln files in the standard
   user locations based on $HOME.

Open questions:

i  ) In case 2, are the emacs binaries and the elisp files in the same
     package, or are they split into different packages?  If the latter,
     which package should contain the .eln files?

ii ) Do we want (g) to actually happen?  If so, do we want it to happen
     always, or should this be configurable in the emacs package
     (dpkg-configure)?

iii) Does case 2 also delete files created in (g) and re-generate them
     using the newly installed Emacs?

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> I'm still discussing with Rob his needs.  I hope to eventually
> understand their needs.  For now my only conclusion is that they need
> to tweak native-comp-eln-load-path in some situations, to control
> where the *.eln files are deposited.  Which is easy and supported
> already, AFAIU.

Eli, Lars, Andrea, and others, I think I'm mostly caught back up with
this thread, and hope to summarize a few things here:

  - I made a mistake in emphasizing (via the thread subject) a
    particular "solution" (suppressing native compilation).  In the end,
    from the Debian perspective, for the most immedate issue, we don't
    need that particular solution (and I wasn't actually set on it at
    the time).  I'd mostly started there because it's *a* possible
    solution, and one I'd imagined might be easyish to implement
    (clearly that was naive).  But other solutions would be fine too.

  - For example, redirecting all incidental writes away from HOME would
    be fine too.

  - We'd prefer to still be able to set HOME=/does-not-exist, which I
    assume would mean that we'd need some other way to redirect *all* of
    the eln file writes (including trampolines).

    Our practice of setting HOME to a nonexistent directory during some
    packaging operations allows us to detect unexpected, and erroneous
    attempts to write to HOME during those processes.

  - If we are going to have "some other way" to redirect the eln files,
    then for us an environment variable might be easier, so that we can
    just export it before we start the relevant build/test/etc. and then
    it'll affect all invocations of emacs in that environment.  I
    suspect an environment variable might also make it easier to
    establish the setting "early enough" in the emacs startup process,
    but don't know that.

  - If just setting HOME will do the job, and there's no interest here
    in anything further, then we can probably just do that, or if we
    ended up needing to, we could likely add our own
    DEBIAN_... environment variable, and carry a patch, but of course
    that's less desirable.

  - As an aside, I suspect Emacs may eventually want to have some way to
    restore the ability to tolerate an unwritable filesystem.  I have
    more than once in the past launched emacs from an emergency shell to
    fix a broken host where the filesystem was read-only and /home might
    not be mounted (and if it were on NFS and there's no network yet,
    could not be).

    I'd much rather use emacs there, than /usr/bin/sensible-editor.  Of
    course now I know that I could just set HOME=/tmp and proceed, but
    will others?  Might at least be worth making sure any error messages
    would lead people to that option.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

>   - As an aside, I suspect Emacs may eventually want to have some way to
>     restore the ability to tolerate an unwritable filesystem.

If it doesn't work, it's a bug.  Please `M-x report-emacs-bug` when you
see such problems.


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: larsi@gnus.org, akrl@sdf.org, monnier@iro.umontreal.ca,
>  david@tethera.net, emacs-devel@gnu.org
> Date: Fri, 14 Oct 2022 12:53:48 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'm still discussing with Rob his needs.  I hope to eventually
> > understand their needs.  For now my only conclusion is that they need
> > to tweak native-comp-eln-load-path in some situations, to control
> > where the *.eln files are deposited.  Which is easy and supported
> > already, AFAIU.
> 
> Eli, Lars, Andrea, and others, I think I'm mostly caught back up with
> this thread, and hope to summarize a few things here:

Thank you for your summary.

>   - We'd prefer to still be able to set HOME=/does-not-exist, which I
>     assume would mean that we'd need some other way to redirect *all* of
>     the eln file writes (including trampolines).
> 
>     Our practice of setting HOME to a nonexistent directory during some
>     packaging operations allows us to detect unexpected, and erroneous
>     attempts to write to HOME during those processes.

This should work for preventing the native-compilation from storing
the *.eln files.  If it doesn't work for you, please tell the details,
and we will investigate.

>   - If we are going to have "some other way" to redirect the eln files,
>     then for us an environment variable might be easier, so that we can
>     just export it before we start the relevant build/test/etc. and then
>     it'll affect all invocations of emacs in that environment.  I
>     suspect an environment variable might also make it easier to
>     establish the setting "early enough" in the emacs startup process,
>     but don't know that.

One other way is to change the value of native-comp-eln-load-path.  I
hesitate to add an environment variable for that, because environment
variables are inherited to subprocesses, and so setting a variable for
some Emacs process will affect all of its Emacs subprocesses.  This
was found to be a reason of tricky and hard-to-debug problems when
users set EMACSLOADPATH like that, and I presume the same can easily
happen with the equivalent variable for *.eln files.  So I'd rather
not add such a variable unless we find a very good reason.

>   - As an aside, I suspect Emacs may eventually want to have some way to
>     restore the ability to tolerate an unwritable filesystem.

With respect to writing the *.eln files, Emacs will look along
native-comp-eln-load-path for the first directory that is writable,
and use that.  So you could at least partially handle this case by
making sure native-comp-eln-load-path includes at least one writable
directory.

>     I'd much rather use emacs there, than /usr/bin/sensible-editor.  Of
>     course now I know that I could just set HOME=/tmp and proceed, but
>     will others?

I've now documented this method in the ELisp reference manual, in the
hope that it will make this more discoverable.

Thanks again for your comments and perseverance.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Rob Browning <rlb@defaultvalue.org>

>>   - We'd prefer to still be able to set HOME=/does-not-exist, which I
>>     assume would mean that we'd need some other way to redirect *all* of
>>     the eln file writes (including trampolines).
>> 
>>     Our practice of setting HOME to a nonexistent directory during some
>>     packaging operations allows us to detect unexpected, and erroneous
>>     attempts to write to HOME during those processes.
>
> This should work for preventing the native-compilation from storing
> the *.eln files.  If it doesn't work for you, please tell the details,
> and we will investigate.

Hmm, which "this" did you mean?  If you meant HOME=/does-not-exist, I
thought people had already said that wouldn't work because the attempt
to build trampolines would still cause a crash.  But I may well have
misunderstood.

> One other way is to change the value of native-comp-eln-load-path.  I
> hesitate to add an environment variable for that, because environment
> variables are inherited to subprocesses, and so setting a variable for
> some Emacs process will affect all of its Emacs subprocesses.  This
> was found to be a reason of tricky and hard-to-debug problems when
> users set EMACSLOADPATH like that, and I presume the same can easily
> happen with the equivalent variable for *.eln files.  So I'd rather
> not add such a variable unless we find a very good reason.

For what it's worth, the reason we'd been at least a bit inclined that
direction is because if you have hundreds of emacs add-on packages
(which I believe we do in Debian), and they all have build and or test
suites that vary in any way they like, it could be much more difficult
to track down all of the emacs invocations in each project's source to
add some argument than it would be to just export an environment
variable before running the suite.

I suppose we could just shadow emacs in the path with a wrapper that
includes the argument(s), assuming few of the projects have their own
interfering options, and that they always use the same (or a handful of
predictable) relative invocation of emacs, e.g. "emacs" not
"/usr/bin/emacs" or...

> With respect to writing the *.eln files, Emacs will look along
> native-comp-eln-load-path for the first directory that is writable,
> and use that.  So you could at least partially handle this case by
> making sure native-comp-eln-load-path includes at least one writable
> directory.

Right, though we also have to make sure we can do that sufficiently
early in *every* case.

> Thanks again for your comments and perseverance.

Certainly, appreciate the help.
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> I suppose we could just shadow emacs in the path with a wrapper that
> includes the argument(s), assuming few of the projects have their own
> interfering options, and that they always use the same (or a handful of
> predictable) relative invocation of emacs, e.g. "emacs" not
> "/usr/bin/emacs" or...

Or put a small chunk of ELisp in Debian's site-start file which sets the
eln load path according to some env-var :-)


        Stefan

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: luangruo@yahoo.com, larsi@gnus.org, akrl@sdf.org,
>  monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org
> Date: Fri, 14 Oct 2022 16:17:56 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Rob Browning <rlb@defaultvalue.org>
> 
> >>   - We'd prefer to still be able to set HOME=/does-not-exist, which I
> >>     assume would mean that we'd need some other way to redirect *all* of
> >>     the eln file writes (including trampolines).
> >> 
> >>     Our practice of setting HOME to a nonexistent directory during some
> >>     packaging operations allows us to detect unexpected, and erroneous
> >>     attempts to write to HOME during those processes.
> >
> > This should work for preventing the native-compilation from storing
> > the *.eln files.  If it doesn't work for you, please tell the details,
> > and we will investigate.
> 
> Hmm, which "this" did you mean?  If you meant HOME=/does-not-exist, I
> thought people had already said that wouldn't work because the attempt
> to build trampolines would still cause a crash.  But I may well have
> misunderstood.

I meant HOME=/does-not-exist, yes.  If that doesn't work (but please
test), we should fix that, I think.  Please report your findings if
that doesn't work.  We use that in our own test suite.

> > One other way is to change the value of native-comp-eln-load-path.  I
> > hesitate to add an environment variable for that, because environment
> > variables are inherited to subprocesses, and so setting a variable for
> > some Emacs process will affect all of its Emacs subprocesses.  This
> > was found to be a reason of tricky and hard-to-debug problems when
> > users set EMACSLOADPATH like that, and I presume the same can easily
> > happen with the equivalent variable for *.eln files.  So I'd rather
> > not add such a variable unless we find a very good reason.
> 
> For what it's worth, the reason we'd been at least a bit inclined that
> direction is because if you have hundreds of emacs add-on packages
> (which I believe we do in Debian), and they all have build and or test
> suites that vary in any way they like, it could be much more difficult
> to track down all of the emacs invocations in each project's source to
> add some argument than it would be to just export an environment
> variable before running the suite.

I understand, but don't you have some kind of centralized script or
group of scripts that runs the testing?  In that case, you could make
the Emacs command, or its template, be part of those few scripts, and
then the change will be less painful.

Please understand my POV: adding an environment variable affects all
Emacs users, not just distros that have testing suites.  There's much
more at stake than your particular testing needs, and many Emacs users
know much less about the potential effects of such environment
variables than you and me.

> I suppose we could just shadow emacs in the path with a wrapper that
> includes the argument(s), assuming few of the projects have their own
> interfering options, and that they always use the same (or a handful of
> predictable) relative invocation of emacs, e.g. "emacs" not
> "/usr/bin/emacs" or...

Yes, something like that.  At least that's what we do in the Emacs's
own test suite (which, of course, is much smaller than yours).

> > With respect to writing the *.eln files, Emacs will look along
> > native-comp-eln-load-path for the first directory that is writable,
> > and use that.  So you could at least partially handle this case by
> > making sure native-comp-eln-load-path includes at least one writable
> > directory.
> 
> Right, though we also have to make sure we can do that sufficiently
> early in *every* case.

Yes, ideally from the same/similar wrapper script that actually
invokes Emacs, via the --eval option.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

>   - We'd prefer to still be able to set HOME=/does-not-exist, which I
>     assume would mean that we'd need some other way to redirect *all* of
>     the eln file writes (including trampolines).

I think this should work, but there may be regressions, of course.
Let's see...  I tried it now, and I got the warning below, but things
otherwise seem to work fine:

Warning (initialization): Unable to create `user-emacs-directory' (~/.emacs.d/).
Any data that would normally be written there may be lost!
If you never want to see this message again,
customize the variable `user-emacs-directory-warning'.

>   - If we are going to have "some other way" to redirect the eln files,
>     then for us an environment variable might be easier, so that we can
>     just export it before we start the relevant build/test/etc. and then
>     it'll affect all invocations of emacs in that environment.  I
>     suspect an environment variable might also make it easier to
>     establish the setting "early enough" in the emacs startup process,
>     but don't know that.

Emacs 29 now has the `inhibit-automatic-native-compilation' variable and
and `EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment variables
to inhibit writing to ~/.emacs.d/eln-cache...

>   - As an aside, I suspect Emacs may eventually want to have some way to
>     restore the ability to tolerate an unwritable filesystem.  I have
>     more than once in the past launched emacs from an emergency shell to
>     fix a broken host where the filesystem was read-only and /home might
>     not be mounted (and if it were on NFS and there's no network yet,
>     could not be).

Emacs should work with an unwritable file system, but there's probably
code that bugs out in that situation -- but those things should be
fixed.

If you have a test case that demonstrates the problem, please open a bug
report for that, so that we can get fixin'.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I suppose we could just shadow emacs in the path with a wrapper that
>> includes the argument(s), assuming few of the projects have their own
>> interfering options, and that they always use the same (or a handful of
>> predictable) relative invocation of emacs, e.g. "emacs" not
>> "/usr/bin/emacs" or...
>
> Or put a small chunk of ELisp in Debian's site-start file which sets the
> eln load path according to some env-var :-)

I wondered if that would be early enough.  I'll have to double-check, but I
think people may have found that some of the attempts to change it via
the command line may not have been.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Sean Whitton]

Hello,

On Sat 15 Oct 2022 at 11:32AM +02, Lars Ingebrigtsen wrote:

> Emacs 29 now has the `inhibit-automatic-native-compilation' variable and
> and `EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment variables
> to inhibit writing to ~/.emacs.d/eln-cache...

Is it likely to be okay for us to backport those commits to our 28.2?

-- 
Sean Whitton

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> I meant HOME=/does-not-exist, yes.  If that doesn't work (but please
> test), we should fix that, I think.  Please report your findings if
> that doesn't work.  We use that in our own test suite.

I've been told that it doesn't because (the person thought) of
trampolines.  i.e. Emacs can be reliably crashed that way.  Though if
that's changed since 28.[12], they wouldn't have seen it yet, and there
may be patches we should cherry-pick.

> I understand, but don't you have some kind of centralized script or
> group of scripts that runs the testing?  In that case, you could make
> the Emacs command, or its template, be part of those few scripts, and
> then the change will be less painful.

For us, this is about hundreds of upstream trees we don't directly
control.  How and where does magit invoke emacs during its build or
during its tests?  Or buttercup, or org, or bbdb, or emacspeak, or ivy,
or lsp, or...

How do we make sure we affect *all* of the invocations of emacs in all
of those?  Note here is just a subset things that Debian has packaged
that might be affected (i.e. only the ones whose maintainers have
decided to keep the Debian tree on salsa in the emacsen-team project):

  https://salsa.debian.org/emacsen-team

> Yes, something like that.  At least that's what we do in the Emacs's
> own test suite (which, of course, is much smaller than yours).

...as long as "most" invocations are either "emacs" or respect say
EMACS, this could work.  I have no idea if that's true, but we do
already know there are likely to be any number of places we'll have to
fix.  i.e. someone has told me that this approach will result in "a lot
of bugs" (in the debian packages).  They already know of places where
"some of them try to choose which emacs to use, rather than just
'emacs'".

We can handle that, but if we have to patch a lot of the upstream trees,
that could be a good bit of work.  That's certainly possible, but
hopefully that helps give more context about why we were interested in
some solution, among the possibilities, that's less potentially
invasive.

For what it's worth, we're also under a (not too close yet) deadline on
the Debian side.  We'd really like to have at least emacs 28 in the next
stable release, and the first stage of the freeze is currently scheduled
for I think January (though Emacs would I think be affected by a
slightly later stage).

For that to be possible, whatever we end up choosing here will need to
be something we can finish accommodating across both the emacs package
and all the add-ons by then.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think this should work, but there may be regressions, of course.
> Let's see...  I tried it now, and I got the warning below, but things
> otherwise seem to work fine:
>
> Warning (initialization): Unable to create `user-emacs-directory' (~/.emacs.d/).
> Any data that would normally be written there may be lost!
> If you never want to see this message again,
> customize the variable `user-emacs-directory-warning'.

We just got 28.2 into Debian, and we should likely re-evaluate that
version, as compared to 28.1.  See if the behaviors have changed.

> Emacs should work with an unwritable file system, but there's probably
> code that bugs out in that situation -- but those things should be
> fixed.
>
> If you have a test case that demonstrates the problem, please open a bug
> report for that, so that we can get fixin'.

OK, so if there is (or we come to) an upstream consensus that
HOME=/does-not-exist should work (would that be considered a subset of
the unwritable filesystem case?), then I think that's likely sufficient
for Debian packaging.  (That's what we were initially attempting.)

And if it doesn't yet work, we're very likely to be able to help track
down and fix those cases as we work through all the add-on packages.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, Po Lu <luangruo@yahoo.com>, akrl@sdf.org,
>  monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org
> Date: Sat, 15 Oct 2022 12:21:29 -0500
> 
> > Emacs should work with an unwritable file system, but there's probably
> > code that bugs out in that situation -- but those things should be
> > fixed.
> >
> > If you have a test case that demonstrates the problem, please open a bug
> > report for that, so that we can get fixin'.
> 
> OK, so if there is (or we come to) an upstream consensus that
> HOME=/does-not-exist should work (would that be considered a subset of
> the unwritable filesystem case?), then I think that's likely sufficient
> for Debian packaging.  (That's what we were initially attempting.)

It should work, yes.  And note that the message cited here, i.e.

> Warning (initialization): Unable to create `user-emacs-directory' (~/.emacs.d/).
> Any data that would normally be written there may be lost!
> If you never want to see this message again,
> customize the variable `user-emacs-directory-warning'.

is just a warning, it shouldn't prevent Emacs from running.  And yes,
features that want to save files under ~/.emacs.d will be unable to do
so, and could signal errors, unless you redefine the variables which
hold the respective file names to point to some other place.  This is
the intended behavior, I believe.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Michael Welsh Duggan <mwd@md5i.com> writes:

> 1) When building emacs (used as part of emacs bootstrap) 
> 2) When installing emacs via dpkg (maybe?  Not certain)
> 3) When building an emacs package (maybe?  Not certain)
> 4) When installing an emacs package via dpkg
> 5) When a user runs emacs

> 1) When building emacs (used as part of emacs bootstrap) 

"building the emacs package", which we do from an upstream tree usually
pretty close to the upstream tag, plus a varying number of patches.
Then it's mostly configure, make check, make DESTDIR=... install to
build the tree(s) that are packaged.

> 2) When installing emacs via dpkg (maybe?  Not certain)

Yes, "apt install emacs".  But that also includes upgrades.  The upgrade
and install scripts run as root, and should not affect HOME in most, if
not all, cases.

And note that both install and upgrade will run emacs a bazillion times
to rebuild all of the system-level .elc files for all of the installed
emacs "add-on" packages in dependency order.

> 3) When building an emacs package (maybe?  Not certain)

Yes, and that will likely run the installed /usr/bin/emacs any number of
times, in unpredictable ways, including via any test harnesses/processes
the package might have.

> 4) When installing an emacs package via dpkg

Yes, "apt install elpa-magit", which also includes upgrades, and will
invoke emacs to rebuild all of the elc files for the add-on.

> 5) When a user runs emacs

Yes, but in this case, 

> g) Case 4 might run emacs to build .elc and .eln files for the package's
>    .el files and place them in world-readable, read-only locations.  An
>    Emacs run in case 5 will include that location in its native file
>    search path.

Yes, if we proceed with ELN compilation (and don't also decide to start
packaging the .elc and .eln files too, rather than building them
on-demand -- which may be up for discussion, but would only be viable if
the emacs fingerprint only changed very deliberatlely).

> Open questions:
>
> i  ) In case 2, are the emacs binaries and the elisp files in the same
>      package, or are they split into different packages?  If the latter,
>      which package should contain the .eln files?

Currently, the elisp files are in a separate emacs-el package because
they used to be optional.  It sounds like they can't be anymore, so
emacs-common now depends on emacs-el, and everything else depends on
emacs-common.

I'd assume that the .eln files would go in the arch-dependent
emacs-bin-common package where other things like ctags, browse, etc. go.

Summary seems pretty accurate, overall.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

Rob Browning [2022-10-15 11:34:14] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> I suppose we could just shadow emacs in the path with a wrapper that
>>> includes the argument(s), assuming few of the projects have their own
>>> interfering options, and that they always use the same (or a handful of
>>> predictable) relative invocation of emacs, e.g. "emacs" not
>>> "/usr/bin/emacs" or...
>> Or put a small chunk of ELisp in Debian's site-start file which sets the
>> eln load path according to some env-var :-)
> I wondered if that would be early enough.

The site-start file is basically the first chunk of non-builtin ELisp
code that is executed except for the `early-init.el` which is loaded
even before.

Trampolines should not be needed before we load such non-builtin code,
so I think the answer is that it's early enough unless the running
user doesn't has `early-init.el` which itself needs a trampoline.

> I'll have to double-check, but I think people may have found that some
> of the attempts to change it via the command line may not have been.

I think `--eval` is run too late (after the site-start), indeed.


        Stefan

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Sean Whitton <spwhitton@spwhitton.name> writes:

>> Emacs 29 now has the `inhibit-automatic-native-compilation' variable and
>> and `EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION' environment variables
>> to inhibit writing to ~/.emacs.d/eln-cache...
>
> Is it likely to be okay for us to backport those commits to our 28.2?

Yes.  But probably wait a bit -- the semantics aren't set in stone yet.

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

> OK, so if there is (or we come to) an upstream consensus that
> HOME=/does-not-exist should work (would that be considered a subset of
> the unwritable filesystem case?),

The cases are somewhat different, and may tickle different code paths.
For instance, there may be guards in the code where we want to write to,
for instance the autosave file, that checks whether the directory
exists, and if not, handles that gracefully.  But perhaps there's no
error handling around actually writing the file, leading Emacs to bug
out in bad ways.  (Just a random theoretical example.)

My guess is that we don't do much testing for the unwritable filesystems
case, so there may be many bugs in this area.

Re: Suppressing native compilation (short and long term)

[Rob Browning]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> My guess is that we don't do much testing for the unwritable filesystems
> case, so there may be many bugs in this area.

Oh, bugs are "fine".

Right now, on that front, we just wanted to explain the situation and
determine what the expectations are.  i.e. if we hit trouble there, from
the upstream perspective are we in "don't do that then" territory, or
have we found something that might warrant attention?

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Suppressing native compilation (short and long term)

[Lars Ingebrigtsen]

Rob Browning <rlb@defaultvalue.org> writes:

> Right now, on that front, we just wanted to explain the situation and
> determine what the expectations are.  i.e. if we hit trouble there, from
> the upstream perspective are we in "don't do that then" territory, or
> have we found something that might warrant attention?

The latter, definitely.

Re: Suppressing native compilation (short and long term)

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

Your idea about -Q inhibiting native comp was obviously not implemented.

In my setup, I call `startup-redirect-eln-cache' in init.el because I don't
want the cache in my .emacs.d/.

However, when I'm using -Q to test something, I do
M-x whatever-which-loads-a-module and end up getting an unwanted
.emacs.d/eln-cache/

So either a --no-native flag (setting
`inhibit-automatic-native-compilation',
and perhaps `load-no-native', to t) or having such behavior in -Q would be
quite helpful.

Yes, I can define an alias to use

--eval "(setq inhibit-automatic-native-compilation t)"

I already have. Still, I regularly forget it and default to emacs -Q.

So, FWIW, I agree with your sentiment that -Q has no single defining goal
other than "do not do unnecessary things, please", and inhibiting native
comp would fit quite well.

[-- Attachment #2: Type: text/html, Size: 1920 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Sat, 5 Nov 2022 02:09:55 +0100
> Cc: Andrea Corallo <akrl@sdf.org>, Eli Zaretskii <eliz@gnu.org>, rlb@defaultvalue.org, 
> 	monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org
> 
> In my setup, I call `startup-redirect-eln-cache' in init.el because I don't
> want the cache in my .emacs.d/.

Why do you need to do that?  What's the problem with having the
eln-cache under your ~/.emacs.d/?

> However, when I'm using -Q to test something, I do
> M-x whatever-which-loads-a-module and end up getting an unwanted
> .emacs.d/eln-cache/

If this testing is for the purposes of Emacs development, then my
suggestion is to have a separate build configured without native
compilation support.  That's what I do.

> So either a --no-native flag (setting `inhibit-automatic-native-compilation',
> and perhaps `load-no-native', to t) or having such behavior in -Q would be
> quite helpful.
> 
> Yes, I can define an alias to use
> 
> --eval "(setq inhibit-automatic-native-compilation t)"
> 
> I already have. Still, I regularly forget it and default to emacs -Q.

So it's just a matter of getting used to using the alias, and will be
solved with time.

> So, FWIW, I agree with your sentiment that -Q has no single defining goal
> other than "do not do unnecessary things, please", and inhibiting native
> comp would fit quite well.

I disagree.  We have too many knobs already, so adding a new one
without a good reason would be a mistake in the long run.

Re: Suppressing native compilation (short and long term)

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]


Eli Zaretskii <eliz@gnu.org> writes:

>> In my setup, I call `startup-redirect-eln-cache' in init.el because I don't
>> want the cache in my .emacs.d/.
>
> Why do you need to do that?  What's the problem with having the
> eln-cache under your ~/.emacs.d/?

My ~/.emacs.d/ is actually versiontracked as part of my repository,
because emacs is my build tool to compile org-mode to PDF and I need a
reproducible environment so others can use it, too.

During building, ~/.emacs.d/ is in the non-writable source directory
used by make distcheck.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
> Cc: Juanma Barranquero <lekktu@gmail.com>, larsi@gnus.org, akrl@sdf.org,
>  rlb@defaultvalue.org, monnier@iro.umontreal.ca, david@tethera.net,
>  emacs-devel@gnu.org
> Date: Sat, 05 Nov 2022 11:12:10 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> In my setup, I call `startup-redirect-eln-cache' in init.el because I don't
> >> want the cache in my .emacs.d/.
> >
> > Why do you need to do that?  What's the problem with having the
> > eln-cache under your ~/.emacs.d/?
> 
> My ~/.emacs.d/ is actually versiontracked as part of my repository,
> because emacs is my build tool to compile org-mode to PDF and I need a
> reproducible environment so others can use it, too.
> 
> During building, ~/.emacs.d/ is in the non-writable source directory
> used by make distcheck.

So what do you do with the *.elc files for the *.el files under
~/.emacs.d/?

And anyway, I think your use case is very different from Juanma's, and
is easily handled by customizing native-comp-eln-load-path.

Re: Suppressing native compilation (short and long term)

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

On Sat, Nov 5, 2022 at 7:45 AM Eli Zaretskii <eliz@gnu.org> wrote:

> Why do you need to do that?  What's the problem with having the
> eln-cache under your ~/.emacs.d/?

There's no problem. It's a matter of taste.

> If this testing is for the purposes of Emacs development, then my
> suggestion is to have a separate build configured without native
> compilation support.  That's what I do.

I prefer to use just one build.

> So it's just a matter of getting used to using the alias, and will be
> solved with time.

Yep.

> I disagree.  We have too many knobs already, so adding a new one
> without a good reason would be a mistake in the long run.

In fact, my thinking yesterday was "-Q should stop native
compilation...Wait, I bet
this was already discussed and rejected", and so I stumbled upon this
thread and
read or perused its several hundred messages. Believe me, I'm not
*proposing* any
change. I'm just telling Lars that I agree with him that this fits under -Q.

[-- Attachment #2: Type: text/html, Size: 1653 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Sat, 5 Nov 2022 12:53:42 +0100
> Cc: larsi@gnus.org, akrl@sdf.org, rlb@defaultvalue.org, 
> 	monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org
> 
> In fact, my thinking yesterday was "-Q should stop native compilation...Wait, I bet
> this was already discussed and rejected", and so I stumbled upon this thread and
> read or perused its several hundred messages. Believe me, I'm not *proposing* any
> change. I'm just telling Lars that I agree with him that this fits under -Q.

Well, we'll have to disagree.  The -Q switch is documented as
disabling various things that happen at startup, specifically loading
stuff that changes the defaults.  Native compilation is not in that
class, exactly like support for image files or GnuTLS aren't.  It is
part of the built Emacs, and is thus part of its default operation.  I
see no reason to change what -Q means, even though some people, for
reasons I cannot grasp, consider JIT native compilation to be
"unusual".

Suppose you start "emacs -Q" where some of the *.el files were already
compiled into the corresponding *.eln files, would you then expect
"emacs -Q" not to use those *.eln files, and instead to load the *.elc
files?  If yes, why?  If not, how does this differ from when you
invoke "emacs -Q" and the *.eln files do not yet exist, but are
produced when Emacs loads the corresponding package?

Re: Suppressing native compilation (short and long term)

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

On Sat, Nov 5, 2022 at 1:44 PM Eli Zaretskii <eliz@gnu.org> wrote:

> Well, we'll have to disagree.  The -Q switch is documented as
> disabling various things that happen at startup, specifically loading
> stuff that changes the defaults.

Yeah, well, there's --no-splash ;-)

> I see no reason to change what -Q means, even though some people, for
> reasons I cannot grasp, consider JIT native compilation to be
> "unusual".

I don't consider it unusual, except that in my build, if I enter Emacs,
load something
that triggers native compilation, and exit quickly, the subprocesses crash
(I get
invitations to "connect with gdb and debug them", which disappear after a
few seconds).
That had me puzzled for a while.

> Suppose you start "emacs -Q" where some of the *.el files were already
> compiled into the corresponding *.eln files, would you then expect
> "emacs -Q" not to use those *.eln files, and instead to load the *.elc
> files?  If yes, why?  If not, how does this differ from when you
> invoke "emacs -Q" and the *.eln files do not yet exist, but are
> produced when Emacs loads the corresponding package?

Loading them and using them wouldn't be a problem, because it does *not*
write anywhere,
while producing them does. In my case, they do just where I don't want them
to be.

As you say, we'll have to agree to disagree. I admit the issue is nuanced
and there's no
single solution that will please everyone.

[-- Attachment #2: Type: text/html, Size: 2363 bytes --]

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Sat, 5 Nov 2022 14:12:03 +0100
> Cc: larsi@gnus.org, akrl@sdf.org, rlb@defaultvalue.org, 
> 	monnier@iro.umontreal.ca, david@tethera.net, emacs-devel@gnu.org
> 
> > I see no reason to change what -Q means, even though some people, for
> > reasons I cannot grasp, consider JIT native compilation to be
> > "unusual".
> 
> I don't consider it unusual, except that in my build, if I enter Emacs, load something
> that triggers native compilation, and exit quickly, the subprocesses crash (I get
> invitations to "connect with gdb and debug them", which disappear after a few seconds).

They aren't supposed to crash, and they didn't in Emacs 28.  So this
should be reported with enough detail for us to be able to fix the
crashes.  This could be related to bug#58956, btw.

> > Suppose you start "emacs -Q" where some of the *.el files were already
> > compiled into the corresponding *.eln files, would you then expect
> > "emacs -Q" not to use those *.eln files, and instead to load the *.elc
> > files?  If yes, why?  If not, how does this differ from when you
> > invoke "emacs -Q" and the *.eln files do not yet exist, but are
> > produced when Emacs loads the corresponding package?
> 
> Loading them and using them wouldn't be a problem, because it does *not* write anywhere,
> while producing them does. In my case, they do just where I don't want them to be.

But we have features who write even under "emacs -Q", don't we?

Re: Suppressing native compilation (short and long term)

[Juanma Barranquero]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]

On Sat, Nov 5, 2022 at 2:35 PM Eli Zaretskii <eliz@gnu.org> wrote:

> They aren't supposed to crash, and they didn't in Emacs 28.  So this
> should be reported with enough detail for us to be able to fix the
> crashes.  This could be related to bug#58956, btw.

Yes. I'm waiting to have a more easily reproducible case or a better
understanding
of how to trigger it. I'll look into #58956.

> But we have features who write even under "emacs -Q", don't we?

Yes, sure. But if I do "emacs -Q" and then I save a bookmark, I know
bookmark.el
is going to save a file and I will make sure it is going to .emacs.d/ or
wherever
else I want it to go. That's not surprising. But I don't usually expect to
do
M-x bs-show and get something written to disk, in a place I wasn't
expecting.

[-- Attachment #2: Type: text/html, Size: 1313 bytes --]

Re: Suppressing native compilation (short and long term)

[Max Brieiev]

"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

> My ~/.emacs.d/ is actually versiontracked as part of my repository,
> because emacs is my build tool to compile org-mode to PDF and I need a
> reproducible environment so others can use it, too.

If Emacs is your build tool, then you probably run it in a batch mode.

I think in one of his messages Andrea said that native compiler is
disabled when Emacs is run in batch mode.

Re: Suppressing native compilation (short and long term)

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]


Max Brieiev <max.brieiev@gmail.com> writes:

> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>
>> My ~/.emacs.d/ is actually versiontracked as part of my repository,
>> because emacs is my build tool to compile org-mode to PDF and I need a
>> reproducible environment so others can use it, too.
>
> If Emacs is your build tool, then you probably run it in a batch mode.

No, I cannot run it in batch mode, because batch mode has different
font-locking than visual mode, so html-export is worse.

I need a full X11 Emacs (kept working via Xvfb, if there is no X11).

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Suppressing native compilation (short and long term)

[Stefan Monnier]

> No, I cannot run it in batch mode, because batch mode has different
> font-locking than visual mode, so html-export is worse.

Please report this as a bug.


        Stefan

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Max Brieiev <max.brieiev@gmail.com> writes:

> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>
>> My ~/.emacs.d/ is actually versiontracked as part of my repository,
>> because emacs is my build tool to compile org-mode to PDF and I need a
>> reproducible environment so others can use it, too.
>
> If Emacs is your build tool, then you probably run it in a batch mode.
>
> I think in one of his messages Andrea said that native compiler is
> disabled when Emacs is run in batch mode.

Yep that's correct.

BR

  Andrea

Re: Suppressing native compilation (short and long term)

[Björn Bidar]

Andrea Corallo <akrl@sdf.org> writes:

> Max Brieiev <max.brieiev@gmail.com> writes:
>
>> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>>
>>> My ~/.emacs.d/ is actually versiontracked as part of my repository,
>>> because emacs is my build tool to compile org-mode to PDF and I need a
>>> reproducible environment so others can use it, too.
>>
>> If Emacs is your build tool, then you probably run it in a batch mode.
>>
>> I think in one of his messages Andrea said that native compiler is
>> disabled when Emacs is run in batch mode.
>
> Yep that's correct.

But I think it possible to call `native-compile` in batch mode I think.
I do this with borg to precompile all packages when I update them.

I think doing so is quite useful e.g. for device that not always run on
cable but on battery or are otherwise limited in power.

Br,

Björn

Re: Suppressing native compilation (short and long term)

[Andrea Corallo]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Max Brieiev <max.brieiev@gmail.com> writes:
>>
>>> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>>>
>>>> My ~/.emacs.d/ is actually versiontracked as part of my repository,
>>>> because emacs is my build tool to compile org-mode to PDF and I need a
>>>> reproducible environment so others can use it, too.
>>>
>>> If Emacs is your build tool, then you probably run it in a batch mode.
>>>
>>> I think in one of his messages Andrea said that native compiler is
>>> disabled when Emacs is run in batch mode.
>>
>> Yep that's correct.
>
> But I think it possible to call `native-compile` in batch mode I think.
> I do this with borg to precompile all packages when I update them.

Yes that's correct, only the machanism that automatically triggers async
native compilations is disabled but you can still invoke compilations
manually.

  Andrea

Parallel native compilation

[Björn Bidar]

Andrea Corallo <akrl@sdf.org> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Andrea Corallo <akrl@sdf.org> writes:
>>
>>> Max Brieiev <max.brieiev@gmail.com> writes:
>>>
>>>> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>>>>
>>>>> My ~/.emacs.d/ is actually versiontracked as part of my repository,
>>>>> because emacs is my build tool to compile org-mode to PDF and I need a
>>>>> reproducible environment so others can use it, too.
>>>>
>>>> If Emacs is your build tool, then you probably run it in a batch mode.
>>>>
>>>> I think in one of his messages Andrea said that native compiler is
>>>> disabled when Emacs is run in batch mode.
>>>
>>> Yep that's correct.
>>
>> But I think it possible to call `native-compile` in batch mode I think.
>> I do this with borg to precompile all packages when I update them.
>
> Yes that's correct, only the machanism that automatically triggers async
> native compilations is disabled but you can still invoke compilations
> manually.

What is the suggested what of doing so in parallel?
While using borg I've noticed that doing so would be quite handy.
It would also help to control the precompile process better when
building emacs so that the build process can be amplified by increasing
the parallel make jobs.

Br,

Björn

Re: Suppressing native compilation (short and long term)

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>,  Eli Zaretskii
>  <eliz@gnu.org>,  Juanma Barranquero <lekktu@gmail.com>,  larsi@gnus.org,
>   rlb@defaultvalue.org,  monnier@iro.umontreal.ca,  david@tethera.net,
>   emacs-devel@gnu.org
> Date: Tue, 08 Nov 2022 07:51:52 +0200
> 
> Andrea Corallo <akrl@sdf.org> writes:
> 
> >> I think in one of his messages Andrea said that native compiler is
> >> disabled when Emacs is run in batch mode.
> >
> > Yep that's correct.
> 
> But I think it possible to call `native-compile` in batch mode I think.

Yes, it is.  We actually do that when building a release tarball on
the end-user machine.

Re: Parallel native compilation

[Andrea Corallo]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>>
>>> Andrea Corallo <akrl@sdf.org> writes:
>>>
>>>> Max Brieiev <max.brieiev@gmail.com> writes:
>>>>
>>>>> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>>>>>
>>>>>> My ~/.emacs.d/ is actually versiontracked as part of my repository,
>>>>>> because emacs is my build tool to compile org-mode to PDF and I need a
>>>>>> reproducible environment so others can use it, too.
>>>>>
>>>>> If Emacs is your build tool, then you probably run it in a batch mode.
>>>>>
>>>>> I think in one of his messages Andrea said that native compiler is
>>>>> disabled when Emacs is run in batch mode.
>>>>
>>>> Yep that's correct.
>>>
>>> But I think it possible to call `native-compile` in batch mode I think.
>>> I do this with borg to precompile all packages when I update them.
>>
>> Yes that's correct, only the machanism that automatically triggers async
>> native compilations is disabled but you can still invoke compilations
>> manually.
>
> What is the suggested what of doing so in parallel?

You can do it from outside if you use N instanceses of Emacs leveraging
`batch-native-compile'.

Another option would be to run one Emacs leveraging
`native-compile-async', but you'll have to add some simple custom code
to have Emacs wainting for all compilation to be finished before
exiting.  `native-comp-async-all-done-hook' might be of hand there.

  Andrea