Add a configure option for NATIVE_FULL_AOT?

Add a configure option for NATIVE_FULL_AOT?

[Ulrich Mueller]

lisp/Makefile.in has a conditional for NATIVE_FULL_AOT which enables
native compilation of all elisp files. Presumably distros would want to
define this when packaging Emacs.

AFAICS this variable is not documented. I wonder if it could be made a
configure option, so a) it would be discoverable and b) its value would
be recorded in system-configuration-options.

Not sure if --with-native-full-aot would be the best possible name.

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Ulrich Mueller <ulm@gentoo.org>
> Date: Tue, 17 Aug 2021 10:53:01 +0200
> 
> lisp/Makefile.in has a conditional for NATIVE_FULL_AOT which enables
> native compilation of all elisp files. Presumably distros would want to
> define this when packaging Emacs.

Why would you want to do that?  It's in a sense against the original
intent: Lisp files should only be natively-compiled when they are
first needed, and stored under the user's home directory.  Otherwise,
you will litter the system-wide directories with a lot of quite large
binary files, which doesn't seem to be a good idea.

> AFAICS this variable is not documented. I wonder if it could be made a
> configure option, so a) it would be discoverable and b) its value would
> be recorded in system-configuration-options.

It's not documented precisely because I'm not sure it should be.

I'd be interested to hear your motivation for using this possibility.

Re: Add a configure option for NATIVE_FULL_AOT?

[Ulrich Mueller]

>>>>> On Tue, 17 Aug 2021, Eli Zaretskii wrote:

>> From: Ulrich Mueller <ulm@gentoo.org>
>> Date: Tue, 17 Aug 2021 10:53:01 +0200
>> 
>> lisp/Makefile.in has a conditional for NATIVE_FULL_AOT which enables
>> native compilation of all elisp files. Presumably distros would want to
>> define this when packaging Emacs.

> Why would you want to do that?  It's in a sense against the original
> intent: Lisp files should only be natively-compiled when they are
> first needed, and stored under the user's home directory.  Otherwise,
> you will litter the system-wide directories with a lot of quite large
> binary files, which doesn't seem to be a good idea.

Storing them in users' home directories also means that every user has
to recompile them, and they will be present multiple times on the
system.

>> AFAICS this variable is not documented. I wonder if it could be made a
>> configure option, so a) it would be discoverable and b) its value would
>> be recorded in system-configuration-options.

> It's not documented precisely because I'm not sure it should be.

> I'd be interested to hear your motivation for using this possibility.

The motivation is to compile and install these files once system-wide,
instead of duplicating them in all users' directories. Conceptually they
aren't user specific but belong to the Emacs package.

Re: Add a configure option for NATIVE_FULL_AOT?

[Lars Ingebrigtsen]

Ulrich Mueller <ulm@gentoo.org> writes:

> The motivation is to compile and install these files once system-wide,
> instead of duplicating them in all users' directories. Conceptually they
> aren't user specific but belong to the Emacs package.

Yup.  I think there should be a configure option for NATIVE_FULL_AOT,
and it would be good if distributions distributed the compiled .eln files.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Ulrich Mueller <ulm@gentoo.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 17 Aug 2021 13:56:28 +0200
> 
> > Why would you want to do that?  It's in a sense against the original
> > intent: Lisp files should only be natively-compiled when they are
> > first needed, and stored under the user's home directory.  Otherwise,
> > you will litter the system-wide directories with a lot of quite large
> > binary files, which doesn't seem to be a good idea.
> 
> Storing them in users' home directories also means that every user has
> to recompile them, and they will be present multiple times on the
> system.

No, only the users who need a file FOO.eln will compile it and have it
in their eln-cache.

> > I'd be interested to hear your motivation for using this possibility.
> 
> The motivation is to compile and install these files once system-wide,
> instead of duplicating them in all users' directories. Conceptually they
> aren't user specific but belong to the Emacs package.

That's... not how this was intended to work.

I wonder if anyone else has an opinion here.

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 17 Aug 2021 14:09:40 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> Ulrich Mueller <ulm@gentoo.org> writes:
> 
> > The motivation is to compile and install these files once system-wide,
> > instead of duplicating them in all users' directories. Conceptually they
> > aren't user specific but belong to the Emacs package.
> 
> Yup.  I think there should be a configure option for NATIVE_FULL_AOT,
> and it would be good if distributions distributed the compiled .eln files.

My fear is that this change of concept will delay the release of Emacs
28, because we are moving the carpet under our feet too close to
cutting the release branch, and will most probably bump into problems
we didn't see until now.

Also, please note that the *.eln files are stored without keeping the
subdirectory structure below lisp/, they are all lumped in the same
directory, unlike the *.elc files.  I guess next we will be asked to
preserve the tree structure...

Re: Add a configure option for NATIVE_FULL_AOT?

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ulrich Mueller <ulm@gentoo.org>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 17 Aug 2021 13:56:28 +0200
>> 
>> > Why would you want to do that?  It's in a sense against the original
>> > intent: Lisp files should only be natively-compiled when they are
>> > first needed, and stored under the user's home directory.  Otherwise,
>> > you will litter the system-wide directories with a lot of quite large
>> > binary files, which doesn't seem to be a good idea.
>> 
>> Storing them in users' home directories also means that every user has
>> to recompile them, and they will be present multiple times on the
>> system.
>
> No, only the users who need a file FOO.eln will compile it and have it
> in their eln-cache.
>
>> > I'd be interested to hear your motivation for using this possibility.
>> 
>> The motivation is to compile and install these files once system-wide,
>> instead of duplicating them in all users' directories. Conceptually they
>> aren't user specific but belong to the Emacs package.
>
> That's... not how this was intended to work.
>
> I wonder if anyone else has an opinion here.

We can set optimization level in init file, or early init, and that will
affect how lisp files are compiled, so for some advanced users
precompiled binaries are of not so much use.

For some less advanced users, maybe with slower computers, precompiled
binaries could save some time and computer resource.

I have personally no inclination for either yes or no, just wanted to
add some arguments to the discussion.

Re: Add a configure option for NATIVE_FULL_AOT?

[Yuri D'Elia]

On Tue, Aug 17 2021, Eli Zaretskii wrote:
>> Storing them in users' home directories also means that every user has
>> to recompile them, and they will be present multiple times on the
>> system.
>
> No, only the users who need a file FOO.eln will compile it and have it
> in their eln-cache.

I run emacs in a server environment under different users regularly, and
I consider this a drawback.

>> The motivation is to compile and install these files once system-wide,
>> instead of duplicating them in all users' directories. Conceptually they
>> aren't user specific but belong to the Emacs package.
>
> That's... not how this was intended to work.
>
> I wonder if anyone else has an opinion here.

As soon as I discovered NATIVE_FULL_AOT, I've been using it. JIT is
nice, but it's relatively easy to trigger. Even if that's async, it's
quite noticeable.

I personally prefer to pay the AOT price in disk space just from a
latency perspective. Which is actually a space saving anyway, in a
multiuser system.

And speaking of opinions, I'd also wish the eln-cache would be stored in
an opportune xdg path when an xdg config is being used ;)

Re: Add a configure option for NATIVE_FULL_AOT?

[Tom Gillespie]

> My fear is that this change of concept will delay the release of Emacs
> 28, because we are moving the carpet under our feet too close to
> cutting the release branch, and will most probably bump into problems
> we didn't see until now.

I don't think that there is any change of concept here. This was one of the
original ways to build the eln files and I have been using it for over a year.

I understand the concern, but wouldn't it be better to get the functionality
out so that we can get more eyeballs/systems using it to see if there are
any issues?

> Also, please note that the *.eln files are stored without keeping the
> subdirectory structure below lisp/, they are all lumped in the same
> directory, unlike the *.elc files.  I guess next we will be asked to
> preserve the tree structure...

We had this conversation almost exactly a year ago as well.
https://lists.gnu.org/archive/html/emacs-devel/2020-08/msg01036.html
The eln files are architecture specific and should not be stored along
with the elc files (amusingly I would love to be able to store all my elc
files in a single location ...). Quite a bit of though and testing have gone
in to make sure that system wide full AOT compilation works and is
distro and packaging friendly.

More on this later. Best!
Tom

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Yuri D'Elia <wavexx@thregr.org>
> Date: Tue, 17 Aug 2021 17:32:29 +0200
> 
> > No, only the users who need a file FOO.eln will compile it and have it
> > in their eln-cache.
> 
> I run emacs in a server environment under different users regularly, and
> I consider this a drawback.
> 
> >> The motivation is to compile and install these files once system-wide,
> >> instead of duplicating them in all users' directories. Conceptually they
> >> aren't user specific but belong to the Emacs package.
> >
> > That's... not how this was intended to work.
> >
> > I wonder if anyone else has an opinion here.
> 
> As soon as I discovered NATIVE_FULL_AOT, I've been using it. JIT is
> nice, but it's relatively easy to trigger. Even if that's async, it's
> quite noticeable.
> 
> I personally prefer to pay the AOT price in disk space just from a
> latency perspective. Which is actually a space saving anyway, in a
> multiuser system.
> 
> And speaking of opinions, I'd also wish the eln-cache would be stored in
> an opportune xdg path when an xdg config is being used ;)

Well, then it's a pity you are speaking up just now, instead of 8
months ago, or whenever you discovered NATIVE_FULL_AOT.  We've spent
these months discovering non-trivial problems with the way Emacs is
built by default with natively-compiled Lisp, fixing those problems,
and stabilizing the solutions so that Emacs could go into pretest
soon.  If we now change things under the assumption that
NATIVE_FULL_AOT will be used frequently, let alone by default, we'll
have to redo some of this work, or risk releasing Emacs with potential
build, installation, and usage problems.

I'm not even sure we have infrastructure and user commands to support
this paradigm.  For example, what do you do if all the Lisp files in
Emacs are natively-compiled AOT, and you then modify one of the *.el
files?  With JIT compilation, the new .el file will be automatically
compiled behind the scenes when first loaded.  With all of them
compiled AOT, you will now have an outdated .eln file in the standard
place and a new one in your eln-cache, which will definitely confuse
or bite you some day.

The XDG thing was discussed and we decided not to do that.  Again,
perhaps if you and others spoke back then, maybe we would make a
different decision.

That is why people are generally encouraged to bring up issues as soon
as they discover them and not much later -- so that we could fix them
in time for the next release, and be sure we solve any fallout.

Re: Add a configure option for NATIVE_FULL_AOT?

[Yuri D'Elia]

On Tue, Aug 17 2021, Eli Zaretskii wrote:
> Well, then it's a pity you are speaking up just now, instead of 8
> months ago, or whenever you discovered NATIVE_FULL_AOT.  We've spent
> these months discovering non-trivial problems with the way Emacs is
> built by default with natively-compiled Lisp, fixing those problems,
> and stabilizing the solutions so that Emacs could go into pretest
> soon.  If we now change things under the assumption that
> NATIVE_FULL_AOT will be used frequently, let alone by default, we'll
> have to redo some of this work, or risk releasing Emacs with potential
> build, installation, and usage problems.

Eeh, I know. I try to keep up with emacs development but even though I
like emacs and use it daily, I'm not involved enough to keep up at the
same same rate I rebuild it ;).

I have a few bug reports I submitted myself I'd wish to work on (I still
think clicking in the scroll-margin zone is completely broken), but
coulnd't put down any time on it.

I browse through the list infrequently (sometimes ignoring it for months
at time), so even such important discussions can escape my radar. My bad
here.

> compiled behind the scenes when first loaded.  With all of them
> compiled AOT, you will now have an outdated .eln file in the standard
> place and a new one in your eln-cache, which will definitely confuse
> or bite you some day.

The way it works now seems logical to me, and I find the current
behavior totally expected.

A real question I had for a while now is whether emacs attempts to
solve the cleanup problem in the user's eln-cache itself.

> The XDG thing was discussed and we decided not to do that.  Again,
> perhaps if you and others spoke back then, maybe we would make a
> different decision.

I've currently symlinked the directory to the proper root in ~/.cache,
mostly to avoid backing it up.

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Tom Gillespie <tgbugs@gmail.com>
> Date: Tue, 17 Aug 2021 09:03:45 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>
> 
> > My fear is that this change of concept will delay the release of Emacs
> > 28, because we are moving the carpet under our feet too close to
> > cutting the release branch, and will most probably bump into problems
> > we didn't see until now.
> 
> I don't think that there is any change of concept here.

Maybe for you and some others, but it doesn't mean there isn't a
conceptual shift.  I've been following the development of the
native-compilation feature for many months, during which we bumped
into and resolved quite a few problems with it, and I can tell you
that some solutions explicitly assumed JIT compilation as the main
paradigm, otherwise they would have made little sense.  I just
mentioned some of them up-thread.

> This was one of the original ways to build the eln files and I have
> been using it for over a year.

I'm not talking about you or anyone else personally.  The issues we
found depend on many different installation/usage habits and patterns,
and typically rear their heads only on some system configurations.  It
is not a coincidence it took us several months to have all the issues
reported, analyzed, and resolved.  Some of the solutions required
several iterations of non-trivial changes to code parts that are quite
sensitive, because Emacs runs them early during startup, where not
everything is yet set up.

> I understand the concern, but wouldn't it be better to get the functionality
> out so that we can get more eyeballs/systems using it to see if there are
> any issues?

I'm not against letting this out, I'm just saying that it will
probably delay the release of Emacs 28, which is a pity.

> > Also, please note that the *.eln files are stored without keeping the
> > subdirectory structure below lisp/, they are all lumped in the same
> > directory, unlike the *.elc files.  I guess next we will be asked to
> > preserve the tree structure...
> 
> We had this conversation almost exactly a year ago as well.
> https://lists.gnu.org/archive/html/emacs-devel/2020-08/msg01036.html
> The eln files are architecture specific and should not be stored along
> with the elc files

I didn't say *.eln should be near the *.elc, I said that they maybe
should have the same directory hierarchy as the *.el files, just under
a different root.

> Quite a bit of though and testing have gone in to make sure that
> system wide full AOT compilation works and is distro and packaging
> friendly.

I didn't invest any thought in it, FWIW.  For me, it was always a "you
are on your own" option.  I didn't even try it, as all of the efforts
went into making the JIT compilation working well and reliably.

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Yuri D'Elia <wavexx@thregr.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 17 Aug 2021 19:12:28 +0200
> 
> > compiled behind the scenes when first loaded.  With all of them
> > compiled AOT, you will now have an outdated .eln file in the standard
> > place and a new one in your eln-cache, which will definitely confuse
> > or bite you some day.
> 
> The way it works now seems logical to me, and I find the current
> behavior totally expected.

It doesn't seem logical to me, FWIW.  It definitely differs from the
behavior with *.elc files: we _replace_ the old ones with the
newly-compiled ones.

> A real question I had for a while now is whether emacs attempts to
> solve the cleanup problem in the user's eln-cache itself.

No.  The user's eln-cache is considered to be the user's
responsibility.

> > The XDG thing was discussed and we decided not to do that.  Again,
> > perhaps if you and others spoke back then, maybe we would make a
> > different decision.
> 
> I've currently symlinked the directory to the proper root in ~/.cache,
> mostly to avoid backing it up.

You assume symlinks will transparently work, but they not always do,
as we learned while working on the natively-compilation and pdumper
stuff.

Re: Add a configure option for NATIVE_FULL_AOT?

[Andreas Schwab]

On Aug 17 2021, Eli Zaretskii wrote:

> For example, what do you do if all the Lisp files in Emacs are
> natively-compiled AOT, and you then modify one of the *.el files?

The installed el files are not modifiable, so you won't do that.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Yuri D'Elia <wavexx@thregr.org>,  emacs-devel@gnu.org
> Date: Tue, 17 Aug 2021 20:33:32 +0200
> 
> On Aug 17 2021, Eli Zaretskii wrote:
> 
> > For example, what do you do if all the Lisp files in Emacs are
> > natively-compiled AOT, and you then modify one of the *.el files?
> 
> The installed el files are not modifiable, so you won't do that.

They will be modifiable if I install them to be modifiable, like under
my home directory or with appropriate access rights.

Re: Add a configure option for NATIVE_FULL_AOT?

[Andreas Schwab]

On Aug 17 2021, Eli Zaretskii wrote:

> They will be modifiable if I install them to be modifiable, like under
> my home directory or with appropriate access rights.

You are not using NATIVE_FULL_AOT.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Tue, 17 Aug 2021 20:46:03 +0200
> Cc: wavexx@thregr.org, emacs-devel@gnu.org
> 
> On Aug 17 2021, Eli Zaretskii wrote:
> 
> > They will be modifiable if I install them to be modifiable, like under
> > my home directory or with appropriate access rights.
> 
> You are not using NATIVE_FULL_AOT.

How does NATIVE_FULL_AOT affect the way the *.el files are installed?

Re: Add a configure option for NATIVE_FULL_AOT?

[Andreas Schwab]

On Aug 17 2021, Eli Zaretskii wrote:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Date: Tue, 17 Aug 2021 20:46:03 +0200
>> Cc: wavexx@thregr.org, emacs-devel@gnu.org
>> 
>> On Aug 17 2021, Eli Zaretskii wrote:
>> 
>> > They will be modifiable if I install them to be modifiable, like under
>> > my home directory or with appropriate access rights.
>> 
>> You are not using NATIVE_FULL_AOT.
>
> How does NATIVE_FULL_AOT affect the way the *.el files are installed?

It's the other way round.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: wavexx@thregr.org,  emacs-devel@gnu.org
> Date: Tue, 17 Aug 2021 21:05:10 +0200
> 
> >> > They will be modifiable if I install them to be modifiable, like under
> >> > my home directory or with appropriate access rights.
> >> 
> >> You are not using NATIVE_FULL_AOT.
> >
> > How does NATIVE_FULL_AOT affect the way the *.el files are installed?
> 
> It's the other way round.

OK, how do the installed *.el files affect NATIVE_FULL_AOT?

Re: Add a configure option for NATIVE_FULL_AOT?

[Ulrich Mueller]

>>>>> On Tue, 17 Aug 2021, Eli Zaretskii wrote:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: Yuri D'Elia <wavexx@thregr.org>,  emacs-devel@gnu.org
>> Date: Tue, 17 Aug 2021 20:33:32 +0200
>> 
>> On Aug 17 2021, Eli Zaretskii wrote:
>> 
>> > For example, what do you do if all the Lisp files in Emacs are
>> > natively-compiled AOT, and you then modify one of the *.el files?
>> 
>> The installed el files are not modifiable, so you won't do that.

> They will be modifiable if I install them to be modifiable, like under
> my home directory or with appropriate access rights.

In the scenario I had in mind the *.el files (as well as the *.eln
files) would be installed as part of a distro's Emacs package.
They would live somewhere under /usr where the user has no business.

Re: Add a configure option for NATIVE_FULL_AOT?

[Tom Gillespie]

I remember reading the threads where you were working out the
issues using the async approach, and the difficulty of getting
the dumping step to work without issues, so much appreciated.

Short version: I think concerns about NATIVE_FULL_AOT are
orthogonal from issues with having native comp enabled at all.

Long version:

> Maybe for you and some others, but it doesn't mean there isn't a
> conceptual shift.  I've been following the development of the
> native-compilation feature for many months, during which we bumped
> into and resolved quite a few problems with it, and I can tell you
> that some solutions explicitly assumed JIT compilation as the main
> paradigm, otherwise they would have made little sense.

Indeed, however, the presence of the native-lisp/*/preloaded folder
means that the
concerns you raised in the other thread are present regardless of whether
NATIVE_FULL_AOT is set. For example, trying to modify files.el and get those
changes to propagate for an invocation of emacs -q is currently not possible due
to fact that the /preloaded/ els will always be present and will take
priority when
native compilation is enabled (removing files-*.eln from preloaded will also
leave Emacs in a completely broken state as I had the misfortune to learn).

Thus I think those issues are orthogonal to the issue at hand with
NATIVE_FULL_AOT. As Ulrich notes, in the context of a release

> They would live somewhere under /usr where the user has no business.

The issue I ran into was me being an idiot/developer and not something
that users would encounter. That said, you mention the cases where
someone compiles Emacs and installs it locally as their own user. In
those cases they could also shoot themselves in the foot as I did,
but I don't think NATIVE_FULL_AOT has any impact on making it
better or worse. One outstanding question is whether eln-cache
always has priority over native-lisp, but again, that seems like an
edge case.

> several iterations of non-trivial changes to code parts that are quite
> sensitive, because Emacs runs them early during startup, where not
> everything is yet set up.

Looking at the way NATIVE_FULL_AOT is implemented in the make file,
NATIVE_DISABLED is set by default and NATIVE_FULL_AOT disables
that option. Everything after that point works in the same way as the normal
async compilation except that the files are placed in /usr/lib64 (or similar)
instead of in the user eln-cache. The value of native-comp-eln-load-path
includes the /usr/lib64 path by default, so if NATIVE_FULL_AOT is empty
then nothing happens. the preloaded/ directory lives inside of that cach
as well, and that is present regardless of whether NATIVE_FULL_AOT
is set or not.

All of this to say that one of the reasons this never came up is because
there shouldn't be anything special about this (not that there isn't), it shifts
the compilation in time and puts the files further down the search path.

> I didn't invest any thought in it, FWIW.  For me, it was always a "you
> are on your own" option.  I didn't even try it, as all of the efforts
> went into making the JIT compilation working well and reliably.

I think that that will pay off in either case because the underlying
compilation process is identical (if confusingly named). JIT -> async
AOT, and FULL AOT just skipping the async bit for the core and doing
it all at the same time.

Best!
Tom

PS As a point of curiosity, here are the current installed sizes that
I am seeing.

app-editors/emacs-18.59-r14: 413 files, 21 non-files, 6.6M
app-editors/emacs-23.4-r21: 3883 files, 91 non-files, 131.2M
app-editors/emacs-24.5-r11: 4100 files, 97 non-files, 146.9M
app-editors/emacs-25.3-r10: 4234 files, 97 non-files, 150.9M
app-editors/emacs-26.3-r5: 4186 files, 101 non-files, 177.2M
app-editors/emacs-27.2-r3: 4222 files, 102 non-files, 157.3M
app-editors/emacs-28.0.9999-r1: 7292 files, 122 non-files, 304.1M

Re: Add a configure option for NATIVE_FULL_AOT?

[Arthur Miller]

Tom Gillespie <tgbugs@gmail.com> writes:

>                    (amusingly I would love to be able to store all my elc
> files in a single location ...).

Amusingly I would like that too, and was my thought when I saw Elis post
about dir tree and answer with xdg path :).

Re: Add a configure option for NATIVE_FULL_AOT?

[Arthur Miller]

Ulrich Mueller <ulm@gentoo.org> writes:

>> They will be modifiable if I install them to be modifiable, like under
>> my home directory or with appropriate access rights.
>
> In the scenario I had in mind the *.el files (as well as the *.eln
> files) would be installed as part of a distro's Emacs package.
> They would live somewhere under /usr where the user has no business.

Sorry, I picking on it, I know that most of distributions do so, but
that is unfortunate practice against the nature of Emacs as application,
since Emacs comes with sources as fully modifiable and extendable
editor. 

I understand it is ortogonal to any multiuser server sysadmin who would
prefer to save drive and have one copy of sources, but I think sources
should be installed somewhere where users can modiry their own copy.

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Ulrich Mueller <ulm@gentoo.org>
> Date: Tue, 17 Aug 2021 21:36:28 +0200
> Cc: wavexx@thregr.org, Andreas Schwab <schwab@linux-m68k.org>,
>  emacs-devel@gnu.org
> 
> >>>>> On Tue, 17 Aug 2021, Eli Zaretskii wrote:
> 
> >> > For example, what do you do if all the Lisp files in Emacs are
> >> > natively-compiled AOT, and you then modify one of the *.el files?
> >> 
> >> The installed el files are not modifiable, so you won't do that.
> 
> > They will be modifiable if I install them to be modifiable, like under
> > my home directory or with appropriate access rights.
> 
> In the scenario I had in mind the *.el files (as well as the *.eln
> files) would be installed as part of a distro's Emacs package.
> They would live somewhere under /usr where the user has no business.

It is strange for a Free Software project to assume the user will
never want to modify the sources.

Re: Add a configure option for NATIVE_FULL_AOT?

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> In the scenario I had in mind the *.el files (as well as the *.eln
>> files) would be installed as part of a distro's Emacs package.  They
>> would live somewhere under /usr where the user has no business.
>
> It is strange for a Free Software project to assume the user will
> never want to modify the sources.

I don't think that assumption is implied by making a system-wide install
of a package in a location not writable by users.  By that logic, any
Free Software package should only be installed by having a VC checkout
in each user's HOME because any modification is mostly useless from a
practical perspective without either being able submitting a patch to
upstream or at least have a private branch one can use to incorporate
future upstream changes back into the modified version.

Well, for emacs, you can also just copy and modify individual lisp files
or functions, of course.

Bye,
Tassilo

Re: Add a configure option for NATIVE_FULL_AOT?

[Ulrich Mueller]

>>>>> On Wed, 18 Aug 2021, Eli Zaretskii wrote:

>> From: Ulrich Mueller <ulm@gentoo.org>
>> Date: Tue, 17 Aug 2021 21:36:28 +0200
>> Cc: wavexx@thregr.org, Andreas Schwab <schwab@linux-m68k.org>,
>> emacs-devel@gnu.org
>> 
>> In the scenario I had in mind the *.el files (as well as the *.eln
>> files) would be installed as part of a distro's Emacs package.
>> They would live somewhere under /usr where the user has no business.

> It is strange for a Free Software project to assume the user will
> never want to modify the sources.

This is simply how distributions work. You don't modify files installed
by the package manager, because your modifications would be overwritten
with the next update of the package. This applies even to a single-user
system; on multi-user systems it would also have security implications.

So, of course users can modify the sources, but they would typically
copy the *.el file to a private location (in their load-path) before
doing so.

Re: Add a configure option for NATIVE_FULL_AOT?

[Andreas Schwab]

On Aug 18 2021, Arthur Miller wrote:

> Sorry, I picking on it, I know that most of distributions do so, but
> that is unfortunate practice against the nature of Emacs as application,
> since Emacs comes with sources as fully modifiable and extendable
> editor. 

Nothing prevents you from reading and modifying the lisp files.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2950 bytes --]

On Wed, Aug 18, 2021 at 05:23:40AM +0300, Eli Zaretskii wrote:
> > From: Ulrich Mueller <ulm@gentoo.org>

[...]

> > In the scenario I had in mind the *.el files (as well as the *.eln
> > files) would be installed as part of a distro's Emacs package.
> > They would live somewhere under /usr where the user has no business.
> 
> It is strange for a Free Software project to assume the user will
> never want to modify the sources.

I think that's the wrong conclusion there hidden between your two
lines.

A distribution has to strike a balance between a novice user not
messing up the whole system for herself and the other users, those
individual users being able to override selected parts of the
distribution-provided stuff and system administrators (which most
of the times are the users themselves) changing the system in ways
that an operating system upgrade doesn't cause havoc.

That's why there are, in general ascending order of precedence,
several layers of "places" to consult when looking for an executable,
a library, a package, a doc, or whatever tidbit of infrastructure
you OS offers.

 - system "directories" [1]
   (think /usr/lib, /usr/bin and so on)
   Those are typically provided by the system installation. E.g.
   the distro. The distro is free to change that stuff on upgrades
   and is free to assume the sysadmin doesn't change these. If
   and when she does, she gets to hold both pieces, especially
   after a system update.

 - system-local "directories"
   (think /usr/local/lib and so on [2])
   Locally (in "this system") installed stuff, accessible by all
   users ot this system). Typically the stuff my sysadmin (hey,
   that's me) and myself decide to compile and install. Distro
   has no business there. My Emacs, since it is one very important
   app for me, is in /usr/local/bin & friends.

 - user-local "directories"
   what each user does for herself. For example I have a ~/.bin.
   For Emacs, I have some ~/.emacs.d/lisp

One popular mechanism to cope with that is that apps have one or
more "paths", fashioned after the shell's $PATH, which list, in
descending order of precedence, where to look for some service
or other bit of infrastructure.

Cf. Emacs's variable `load-path'.

Now I assume I'm boring all of you with well-known things, but I
really don't understand why .eln files would be so different that
they can't follow (some variation of) the `load-path' pattern?

Cheers

[1] I'm putting directory in quotes, because not all is in the
   file system. It's more the pattern than the actual implementation
   (although the implementation /is/ often at this level, but there
   are counter-examples).

[2] Some vendors have a crush on /opt. I don't like it that much,
   because I don't want a third-party vendor giving itself so
   much importance. My sysadmin and me decide on what is important
   and what not).

 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Tom Gillespie <tgbugs@gmail.com>
> Date: Tue, 17 Aug 2021 14:52:10 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>
> 
> Short version: I think concerns about NATIVE_FULL_AOT are
> orthogonal from issues with having native comp enabled at all.

I agree (and never said anything to the contrary).  I was only talking
about the former, if it is treated as the main or even very important
paradigm.

> Indeed, however, the presence of the native-lisp/*/preloaded folder
> means that the
> concerns you raised in the other thread are present regardless of whether
> NATIVE_FULL_AOT is set. For example, trying to modify files.el and get those
> changes to propagate for an invocation of emacs -q is currently not possible due
> to fact that the /preloaded/ els will always be present and will take
> priority when
> native compilation is enabled (removing files-*.eln from preloaded will also
> leave Emacs in a completely broken state as I had the misfortune to learn).

Modification of preloaded Lisp files requires re-dumping Emacs,
always did.  In general, you cannot easily do that without having the
source tree and rebuilding there, not unless you know very well what
to do.  If you do know what to do, the presence of the *.eln files in
preloaded/ doesn't prevent you from rebuilding Emacs: you just need to
replace files*.eln with the newly-compiled one, and re-dump, which is
basically what happens during a re-build.

> One outstanding question is whether eln-cache always has priority
> over native-lisp

It does, except when Emacs knows the particular .eln file should be
put in preloaded/.

> > I didn't invest any thought in it, FWIW.  For me, it was always a "you
> > are on your own" option.  I didn't even try it, as all of the efforts
> > went into making the JIT compilation working well and reliably.
> 
> I think that that will pay off in either case because the underlying
> compilation process is identical (if confusingly named). JIT -> async
> AOT, and FULL AOT just skipping the async bit for the core and doing
> it all at the same time.

I wish it were that simple.  It isn't.  Adding natively-compiled files
raises quite a few issues that didn't exist before, and up to now,
when I thought about these issues, I always considered the JIT modus
operandi, and it alone -- if things sounded right with JIT, I
considered the issue closed.

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Date: Wed, 18 Aug 2021 06:53:45 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> In the scenario I had in mind the *.el files (as well as the *.eln
> >> files) would be installed as part of a distro's Emacs package.  They
> >> would live somewhere under /usr where the user has no business.
> >
> > It is strange for a Free Software project to assume the user will
> > never want to modify the sources.
> 
> I don't think that assumption is implied by making a system-wide install
> of a package in a location not writable by users.

That's not what I said or meant.  What I meant was that considering
the problem non-existent because this is how distros install the Lisp
files sounds strange to me, because it assumes no user will ever want
to modify these files enough to make them writable.  IOW, the
assumption that bothered me was that no one will ever want to modify
those files, e.g., to fix some bug or add a feature.

> Well, for emacs, you can also just copy and modify individual lisp files
> or functions, of course.

Exactly.  And then the problems I had in mind will rear their heads.

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Ulrich Mueller <ulm@gentoo.org>
> Cc: Ulrich Mueller <ulm@gentoo.org>,  wavexx@thregr.org,
>   schwab@linux-m68k.org,  emacs-devel@gnu.org
> Date: Wed, 18 Aug 2021 09:04:34 +0200
> 
> So, of course users can modify the sources, but they would typically
> copy the *.el file to a private location (in their load-path) before
> doing so.

My problem is what happens after they do whatever it takes to make
those files modifiable, and actually modify them.  With JIT
native-compilation, everything still works the same, and the updated
*.eln files get written to the same eln-cache and used thereafter (and
even automatically reloaded inside the same session).  With AOT
compilation, things are not so simple, and I'm actually unsure what
would happen because we didn't actively investigate this method.  It
will probably work in many/most cases (otherwise we'd have bug
reports, I think), but we didn't scrutinize it nearly as well as the
JIT method.

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> Date: Wed, 18 Aug 2021 09:33:49 +0200
> From: <tomas@tuxteam.de>
> 
> Now I assume I'm boring all of you with well-known things, but I
> really don't understand why .eln files would be so different that
> they can't follow (some variation of) the `load-path' pattern?

They can, and they do.  See native-comp-eln-load-path.  Assuming I
understood correctly what bothers you.

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

On Wed, Aug 18, 2021 at 03:14:32PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 18 Aug 2021 09:33:49 +0200
> > From: <tomas@tuxteam.de>
> > 
> > Now I assume I'm boring all of you with well-known things, but I
> > really don't understand why .eln files would be so different that
> > they can't follow (some variation of) the `load-path' pattern?
> 
> They can, and they do.  See native-comp-eln-load-path.  Assuming I
> understood correctly what bothers you.

Thanks.

I think you do. Well, I'm quite certain :)

Then what would prevent a distribution from distributing their .eln
files to go somewhere in /usr/lib/emacs.../<version>/..., to be
overriden by the user when needed?

I've been following the native compilation threads with quite some
excitement, but I'm sure I haven't digested many "interesting"
details :)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> Date: Wed, 18 Aug 2021 15:32:33 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > They can, and they do.  See native-comp-eln-load-path.  Assuming I
> > understood correctly what bothers you.
> 
> Thanks.
> 
> I think you do. Well, I'm quite certain :)
> 
> Then what would prevent a distribution from distributing their .eln
> files to go somewhere in /usr/lib/emacs.../<version>/..., to be
> overriden by the user when needed?

I don't think I understand.  load-path is used for only one purpose:
finding Lisp files you want to load.  It is not used to decide where
to _write_ *.elc files.  The same with native-comp-eln-load-path (but
see below).  So how can this variable solve the problem at hand, which
is where to store *.eln files produced by compilation?

We actually do use native-comp-eln-load-path to also decide where to
store the *.eln files, but that uses a simplistic heuristic that
treats the last element of the list specially.  The heuristic is
hard-coded in several places, so not easily overridden.

> I've been following the native compilation threads with quite some
> excitement, but I'm sure I haven't digested many "interesting"
> details :)

"Interesting" as in the infamous Chinese curse?

It's those "interesting" details that bother me: I needed to think
about them and discuss them with Andrea many times over the last
months, and in many cases the issue was put to rest by considering the
JIT scenario.  I don't know what would have happened if we considered
the AOT scenario as seriously as people want us to do now.

Re: Add a configure option for NATIVE_FULL_AOT?

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> > As soon as I discovered NATIVE_FULL_AOT, I've been using it. JIT is
> > nice, but it's relatively easy to trigger. Even if that's async, it's
> > quite noticeable.
> >
> > I personally prefer to pay the AOT price in disk space just from a
> > latency perspective. Which is actually a space saving anyway, in a
> > multiuser system.
[...]
>
> Well, then it's a pity you are speaking up just now, instead of 8
> months ago, or whenever you discovered NATIVE_FULL_AOT.

AFAIR, this was discussed already several months ago.  For example in
this thread:

    https://lists.gnu.org/r/emacs-devel/2021-02/msg00950.html

My conclusion from those discussions was that we do want to support
AOT compilation so that it can be used by GNU/Linux distributions.  I
haven't been able to follow development closely for a couple of
months, but I am surprised to hear that this is now not even
considered something we want to document.

Re: Add a configure option for NATIVE_FULL_AOT?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> My fear is that this change of concept will delay the release of Emacs
> 28, because we are moving the carpet under our feet too close to
> cutting the release branch, and will most probably bump into problems
> we didn't see until now.

Distributions are going to do NATIVE_FULL_AOT builds, and we'll get bug
reports for any problems in this area in any case.

So trying to "hide" the functionality just means that we get less testing.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Add a configure option for NATIVE_FULL_AOT?

[Yuri D'Elia]

On Wed, Aug 18 2021, Andreas Schwab wrote:
> On Aug 18 2021, Arthur Miller wrote:
>
>> Sorry, I picking on it, I know that most of distributions do so, but
>> that is unfortunate practice against the nature of Emacs as application,
>> since Emacs comes with sources as fully modifiable and extendable
>> editor.
>
> Nothing prevents you from reading and modifying the lisp files.

I don't want to add anything which hasn't been said by others already,
but just point out that the way that emacs is packaged in debian is
actually pretty nice and convenient for many users, especially in a
multi-tenant setup.

I'm actually aware of several friends that have emacs installed without
the emacs-el package at all (which, to clarify, contains the actual
elisp source files - not just the bytecompiled ones) and are perfectly
happy about that. They are not fluent in elisp, so they treat those (as
well as most of the configuration) as a black-box. I even did so myself
for many years early on.

I definitely see the same concept being extended to AOT and being a net
advantage in such cases.

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Wed, 18 Aug 2021 16:11:10 +0200
> Cc: "Yuri D'Elia" <wavexx@thregr.org>, Emacs developers <emacs-devel@gnu.org>
> 
> > Well, then it's a pity you are speaking up just now, instead of 8
> > months ago, or whenever you discovered NATIVE_FULL_AOT.
> 
> AFAIR, this was discussed already several months ago.  For example in
> this thread:
> 
>     https://lists.gnu.org/r/emacs-devel/2021-02/msg00950.html

I don't think I see the issue being _discussed_ there.  It's a single
message.

We aren't talking about the AOT build in general being supported, we
are talking about making it first-class citizen.  Can you point to
discussions of the advantages and disadvantages of that?  Perhaps I
missed that, or forgot.

> My conclusion from those discussions was that we do want to support
> AOT compilation so that it can be used by GNU/Linux distributions.

From that single message, or from others?

> I haven't been able to follow development closely for a couple of
> months, but I am surprised to hear that this is now not even
> considered something we want to document.

That's not what I said, please don't misrepresent.  I said that if we
want to make this more mainstream than before, we may need to delay
the release of Emacs 28.  That's a far cry from not considering it
documented.

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2991 bytes --]

On Wed, Aug 18, 2021 at 04:45:42PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 18 Aug 2021 15:32:33 +0200
> > From: tomas@tuxteam.de

[...]

> > Then what would prevent a distribution from distributing their .eln
> > files to go somewhere in /usr/lib/emacs.../<version>/..., to be
> > overriden by the user when needed?
> 
> I don't think I understand.  load-path is used for only one purpose:
> finding Lisp files you want to load.  It is not used to decide where
> to _write_ *.elc files.  The same with native-comp-eln-load-path (but
> see below).  So how can this variable solve the problem at hand, which
> is where to store *.eln files produced by compilation?

Then all would be well: the idea is not for the user to replace the
distribution's .eln files, just to override them in case she wants
to change the corresponding .el files.

(at least not to change them unwittingly).

The distribution provides an Emacs (say somewhere in /usr/bin), the
.el and .elc files (somewhere in /usr/share/lib), and the .eln files,
possibly in an architecture-specific subdir in /usr/lib. 

When the user wants to change a distribution-specific .el, the default
way to do it would be to make a user-writable copy somewhere in a
user-controlled directory (ISTR there was a mechanism doing this
semi-transparently, but I may be confused). If `load-path' is set up
correctly, the user's variant takes precedence. Same would go for
.eln files resp. native-comp-eln-load-path, right?

> We actually do use native-comp-eln-load-path to also decide where to
> store the *.eln files, but that uses a simplistic heuristic that
> treats the last element of the list specially.  The heuristic is
> hard-coded in several places, so not easily overridden.

Oh, the /last/ element, i.e. the one with the least precedence is the
one where .eln files are stored? There goes my house of cards.
Interesting ;-)

For my (most probably incomplete) distro scenario, the first would be
the more natural candidate (assuming the first takes precedence in
the search, as with `load-path'). Or a separate knob, but then you'd
have another "interesting" problem: keeping those two consistent, for
some value of "consistent". Interesting, again.

> > I've been following the native compilation threads with quite some
> > excitement, but I'm sure I haven't digested many "interesting"
> > details :)
> 
> "Interesting" as in the infamous Chinese curse?

See above ;-)

> It's those "interesting" details that bother me: I needed to think
> about them and discuss them with Andrea many times over the last
> months, and in many cases the issue was put to rest by considering the
> JIT scenario.  I don't know what would have happened if we considered
> the AOT scenario as seriously as people want us to do now.

Don't get me wrong. I don't "want" anything. Besides saying "thank
you for your outstanding work". I'll be happy however it comes out.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> Date: Wed, 18 Aug 2021 18:22:16 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> When the user wants to change a distribution-specific .el, the default
> way to do it would be to make a user-writable copy somewhere in a
> user-controlled directory (ISTR there was a mechanism doing this
> semi-transparently, but I may be confused). If `load-path' is set up
> correctly, the user's variant takes precedence. Same would go for
> .eln files resp. native-comp-eln-load-path, right?

Nominally, yes.  But then you have 2 copies of the same .eln file, in
2 different places.  That's got to confuse people and perhaps create
hard-to-debug situations where Emacs behaves erratically.

> > We actually do use native-comp-eln-load-path to also decide where to
> > store the *.eln files, but that uses a simplistic heuristic that
> > treats the last element of the list specially.  The heuristic is
> > hard-coded in several places, so not easily overridden.
> 
> Oh, the /last/ element, i.e. the one with the least precedence is the
> one where .eln files are stored? There goes my house of cards.
> Interesting ;-)

The preloaded *.eln files are stored there, yes.  How else would you
arrange for them to be of the least precedence, when lists are
generally searched head to tail?

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2154 bytes --]

On Wed, Aug 18, 2021 at 07:26:57PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 18 Aug 2021 18:22:16 +0200
> > From: tomas@tuxteam.de
> > Cc: emacs-devel@gnu.org
> > 
> > When the user wants to change a distribution-specific .el, the default
> > way to do it would be to make a user-writable copy somewhere in a
> > user-controlled directory (ISTR there was a mechanism doing this
> > semi-transparently, but I may be confused). If `load-path' is set up
> > correctly, the user's variant takes precedence. Same would go for
> > .eln files resp. native-comp-eln-load-path, right?
> 
> Nominally, yes.  But then you have 2 copies of the same .eln file, in
> 2 different places.

They would be different, one of them from an .el with user changes.

>                     That's got to confuse people and perhaps create
> hard-to-debug situations where Emacs behaves erratically.
> 
> > > We actually do use native-comp-eln-load-path to also decide where to
> > > store the *.eln files, but that uses a simplistic heuristic that
> > > treats the last element of the list specially.  The heuristic is
> > > hard-coded in several places, so not easily overridden.
> > 
> > Oh, the /last/ element, i.e. the one with the least precedence is the
> > one where .eln files are stored? There goes my house of cards.
> > Interesting ;-)
> 
> The preloaded *.eln files are stored there, yes.

But why on the "least precedence" slot, i.e. in the place at the
list's tail? I'd have expected them to be stored in the place
at the list's head.

>                                                  How else would you
> arrange for them to be of the least precedence, when lists are
> generally searched head to tail?

Why would I want to arrange "them" [1] to be of the least precedence?
A freshly user-compiled file should have the highest precedence, IMO.

You see: those are the interesting details I've not yet digested, it
seems :-)

Cheers
[1] provided our "them" refers to the same thing: "the .eln files
   freshly compiled in the user's behalf, because Emacs has seen
   an .el(c) file it had not seen before"

 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> Date: Wed, 18 Aug 2021 18:34:55 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > > When the user wants to change a distribution-specific .el, the default
> > > way to do it would be to make a user-writable copy somewhere in a
> > > user-controlled directory (ISTR there was a mechanism doing this
> > > semi-transparently, but I may be confused). If `load-path' is set up
> > > correctly, the user's variant takes precedence. Same would go for
> > > .eln files resp. native-comp-eln-load-path, right?
> > 
> > Nominally, yes.  But then you have 2 copies of the same .eln file, in
> > 2 different places.
> 
> They would be different, one of them from an .el with user changes.

Sure, that's my point.

> > The preloaded *.eln files are stored there, yes.
> 
> But why on the "least precedence" slot, i.e. in the place at the
> list's tail? I'd have expected them to be stored in the place
> at the list's head.
> 
> >                                                  How else would you
> > arrange for them to be of the least precedence, when lists are
> > generally searched head to tail?
> 
> Why would I want to arrange "them" [1] to be of the least precedence?
> A freshly user-compiled file should have the highest precedence, IMO.

How would that precedence work, except by relying on the order of the
directories in the list?

> [1] provided our "them" refers to the same thing: "the .eln files
>    freshly compiled in the user's behalf, because Emacs has seen
>    an .el(c) file it had not seen before"

I don't understand this part.  The *.elc files add yet another
"interesting" aspect to the issue, but that's a story for another day.

Re: Add a configure option for NATIVE_FULL_AOT?

[Stefan Monnier]

> Then all would be well: the idea is not for the user to replace the
> distribution's .eln files, just to override them in case she wants
> to change the corresponding .el files.

The way the .eln file naming has been designed is for the .eln files to
be "transparent", i.e. Emacs first looks for a `.elc` or `.el` file and once
it finds it, it looks for the corresponding `.eln` file.

If the `.eln` file it uses does not correspond to the content of the
`.el` or `.elc` file it found, it means we have a *bug*.


        Stefan

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1922 bytes --]

I feel sorry for wasting your time in this way.

On Wed, Aug 18, 2021 at 07:43:23PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 18 Aug 2021 18:34:55 +0200
> > From: tomas@tuxteam.de
> > Cc: emacs-devel@gnu.org
> > 
> > > > When the user wants to change a distribution-specific .el, the default
> > > > way to do it would be to make a user-writable copy somewhere in a
> > > > user-controlled directory (ISTR there was a mechanism doing this
> > > > semi-transparently, but I may be confused). If `load-path' is set up
> > > > correctly, the user's variant takes precedence. Same would go for
> > > > .eln files resp. native-comp-eln-load-path, right?
> > > 
> > > Nominally, yes.  But then you have 2 copies of the same .eln file, in
> > > 2 different places.
> > 
> > They would be different, one of them from an .el with user changes.
> 
> Sure, that's my point.

And we want the user's .el and the corresponding .eln to override
the system-provided ones. So those (more precisely: their directories
should go first in their corresponding paths, right?

> How would that precedence work, except by relying on the order of the
> directories in the list?

I must have expressed my thoughts too clumsily. I do agree with you
that the several paths are in order of descending preferences.

What I'm implying from that is that the place for the user's .eln
files should be more at the front of this path, so the .jit mechanism
should put them there, if they have to override the system-provided
ones.

> > [1] provided our "them" refers to the same thing: "the .eln files
> >    freshly compiled in the user's behalf, because Emacs has seen
> >    an .el(c) file it had not seen before"
> 
> I don't understand this part.  The *.elc files add yet another
> "interesting" aspect to the issue, but that's a story for another day.

Let's forget about those for now :-)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

On Wed, Aug 18, 2021 at 12:48:04PM -0400, Stefan Monnier wrote:
> > Then all would be well: the idea is not for the user to replace the
> > distribution's .eln files, just to override them in case she wants
> > to change the corresponding .el files.
> 
> The way the .eln file naming has been designed is for the .eln files to
> be "transparent", i.e. Emacs first looks for a `.elc` or `.el` file and once
> it finds it, it looks for the corresponding `.eln` file.

Oh, so the .elc file is not searched directly. First, an .el file is
searched, and then, based on its path a corresponding .elc file is
found (or jit-made)? Then, `load-path' is the relevant search order,
right?

> If the `.eln` file it uses does not correspond to the content of the
> `.el` or `.elc` file it found, it means we have a *bug*.

In that case, the "distribution model" would "Just Work"?

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Wed, 18 Aug 2021 12:48:04 -0400
> 
> > Then all would be well: the idea is not for the user to replace the
> > distribution's .eln files, just to override them in case she wants
> > to change the corresponding .el files.
> 
> The way the .eln file naming has been designed is for the .eln files to
> be "transparent", i.e. Emacs first looks for a `.elc` or `.el` file and once
> it finds it, it looks for the corresponding `.eln` file.

I think it actually requires to see the .el file, otherwise it won't
load the .eln file.  The .elc file is not enough.  Andrea, am I right?

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> Date: Wed, 18 Aug 2021 18:56:53 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > > > Nominally, yes.  But then you have 2 copies of the same .eln file, in
> > > > 2 different places.
> > > 
> > > They would be different, one of them from an .el with user changes.
> > 
> > Sure, that's my point.
> 
> And we want the user's .el and the corresponding .eln to override
> the system-provided ones. So those (more precisely: their directories
> should go first in their corresponding paths, right?

They should go first in the list that is the value of
native-comp-eln-load-path, yes.

> > How would that precedence work, except by relying on the order of the
> > directories in the list?
> 
> I must have expressed my thoughts too clumsily. I do agree with you
> that the several paths are in order of descending preferences.
> 
> What I'm implying from that is that the place for the user's .eln
> files should be more at the front of this path, so the .jit mechanism
> should put them there, if they have to override the system-provided
> ones.

That's where they are, and that's where JIT compilation puts them.

But that is TRT for *.el files _other_ than the ones inside
/usr/lib/.../native-lisp/, since those other files are _always_
compiled into the user's eln-cache directory.  Thus we avoid the
danger of having several variants of the same .eln file with the same
name.  That doesn't work if the original .eln file is under /usr/lib.
And therein lies the problem I was trying to describe.

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> Date: Wed, 18 Aug 2021 19:00:20 +0200
> From: tomas@tuxteam.de
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> Oh, so the .elc file is not searched directly. First, an .el file is
> searched, and then, based on its path a corresponding .elc file is
> found (or jit-made)?

No.  If there's no .eln for the file you load/require, Emacs will try
to load the .elc, and if that fails, will load the .el.  In any case,
if there's no .eln, Emacs will start a native-compilation in the
background, and will load the .eln file when it is ready, replacing
the .el/.elc.

All this is by default, of course; you can disable the JIT
compilation, and you can disable loading the .eln file after the JIT
compilation.  And you can do other things ;-)

> In that case, the "distribution model" would "Just Work"?

Famous last words.

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

On Wed, Aug 18, 2021 at 08:17:17PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 18 Aug 2021 19:00:20 +0200
> > From: tomas@tuxteam.de
> > Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> > 
> > Oh, so the .elc file is not searched directly. First, an .el file is
> > searched, and then, based on its path a corresponding .elc file is
> > found (or jit-made)?
> 
> No.  If there's no .eln for the file you load/require, Emacs will try
> to load the .elc, and if that fails, will load the .el.  In any case,
> if there's no .eln, Emacs will start a native-compilation in the
> background, and will load the .eln file when it is ready, replacing
> the .el/.elc.
> 
> All this is by default, of course; you can disable the JIT
> compilation, and you can disable loading the .eln file after the JIT
> compilation.  And you can do other things ;-)
> 
> > In that case, the "distribution model" would "Just Work"?
> 
> Famous last words.

Rather a last question :)

OK. It seems the proof will be in the pudding. Anyway, thanks you both
for your patience. I've gained some more insight.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Gunnar Horrigmo]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Yuri D'Elia <wavexx@thregr.org>

>> And speaking of opinions, I'd also wish the eln-cache would be stored in
>> an opportune xdg path when an xdg config is being used ;)
>
> The XDG thing was discussed and we decided not to do that.  Again,
> perhaps if you and others spoke back then, maybe we would make a
> different decision.

Well, that's depressing, but not paying attention is on me.

I won't attempt to revive that discussion, but is there some guidance on
how to configure this in .emacs (probably not, but by any means), so we
can have a more... compliant environment (if you'll excuse the term)?

-- 
Gunnar

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Gunnar Horrigmo <horrigmo@runbox.no>
> Date: Wed, 18 Aug 2021 21:13:46 +0200
> 
> > The XDG thing was discussed and we decided not to do that.  Again,
> > perhaps if you and others spoke back then, maybe we would make a
> > different decision.
> 
> Well, that's depressing, but not paying attention is on me.

AFAIR, the main argument against using the XDG's cache was that the
definition of its purpose didn't fit, and that XDG directories are
ephemeral.

> I won't attempt to revive that discussion, but is there some guidance on
> how to configure this in .emacs (probably not, but by any means), so we
> can have a more... compliant environment (if you'll excuse the term)?

I don't think I have a clear understanding of what you want to
achieve, so please tell more.

Re: Add a configure option for NATIVE_FULL_AOT?

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 18 Aug 2021 19:00:20 +0200
>> From: tomas@tuxteam.de
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>> 
>> Oh, so the .elc file is not searched directly. First, an .el file is
>> searched, and then, based on its path a corresponding .elc file is
>> found (or jit-made)?
>
> No.  If there's no .eln for the file you load/require, Emacs will try
> to load the .elc, and if that fails, will load the .el.  In any case,
> if there's no .eln, Emacs will start a native-compilation in the
> background, and will load the .eln file when it is ready, replacing
> the .el/.elc.

Just to touch on this.  ATM we do start a native-compilation only when
we load a .elc for which no .eln is found.  The same for the .el files
is not enabled.  I guess the main reason is that we would loose the
ability to step but I suspect wouldn't be the only drawback.

  Andrea

Re: Add a configure option for NATIVE_FULL_AOT?

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Wed, 18 Aug 2021 12:48:04 -0400
>> 
>> > Then all would be well: the idea is not for the user to replace the
>> > distribution's .eln files, just to override them in case she wants
>> > to change the corresponding .el files.
>> 
>> The way the .eln file naming has been designed is for the .eln files to
>> be "transparent", i.e. Emacs first looks for a `.elc` or `.el` file and once
>> it finds it, it looks for the corresponding `.eln` file.
>
> I think it actually requires to see the .el file, otherwise it won't
> load the .eln file.  The .elc file is not enough.  Andrea, am I right?

That is correct.

Re: Add a configure option for NATIVE_FULL_AOT?

[Arthur Miller]

Yuri D'Elia <wavexx@thregr.org> writes:

> On Wed, Aug 18 2021, Andreas Schwab wrote:
>> On Aug 18 2021, Arthur Miller wrote:
>>
>>> Sorry, I picking on it, I know that most of distributions do so, but
>>> that is unfortunate practice against the nature of Emacs as application,
>>> since Emacs comes with sources as fully modifiable and extendable
>>> editor.
>>
>> Nothing prevents you from reading and modifying the lisp files.
Y
> I don't want to add anything which hasn't been said by others already,
> but just point out that the way that emacs is packaged in debian is
> actually pretty nice and convenient for many users, especially in a
> multi-tenant setup.
I haven't seen a Debian since somewhere around 2001 or something, so I
really don't know how they do. But I think that many distros put elisp
in /usr/share which is not user modifiable location by default.

I am trying to see what Emacs uses by default choice in configure script
for lisp sources, but I am not sure I understand in which category are
those:

Installation directories:
  --prefix=PREFIX         install architecture-independent files in PREFIX
                          [/usr/local]
  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]

By default, `make install' will install all the files in
`/usr/local/bin', `/usr/local/lib' etc.  You can specify
an installation prefix other than `/usr/local' using `--prefix',
for instance `--prefix=$HOME'.

For better control, use the options below.

Fine tuning of the installation directories:
  --bindir=DIR            user executables [EPREFIX/bin]
  --sbindir=DIR           system admin executables [EPREFIX/sbin]
  --libexecdir=DIR        program executables [EPREFIX/libexec]
  --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
  --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
  --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
  --runstatedir=DIR       modifiable per-process data [LOCALSTATEDIR/run]
  --libdir=DIR            object code libraries [EPREFIX/lib]
  --includedir=DIR        C header files [PREFIX/include]
  --oldincludedir=DIR     C header files for non-gcc [/usr/include]
  --datarootdir=DIR       read-only arch.-independent data root [PREFIX/share]
  --datadir=DIR           read-only architecture-independent data [DATAROOTDIR]
  --infodir=DIR           info documentation [DATAROOTDIR/info]
  --localedir=DIR         locale-dependent data [DATAROOTDIR/locale]
  --mandir=DIR            man documentation [DATAROOTDIR/man]
  --docdir=DIR            documentation root [DATAROOTDIR/doc/emacs]
  --htmldir=DIR           html documentation [DOCDIR]
  --dvidir=DIR            dvi documentation [DOCDIR]
  --pdfdir=DIR            pdf documentation [DOCDIR]
  --psdir=DIR             ps documentation [DOCDIR]




> I'm actually aware of several friends that have emacs installed without
> the emacs-el package at all (which, to clarify, contains the actual
> elisp source files - not just the bytecompiled ones) and are perfectly
> happy about that. They are not fluent in elisp, so they treat those (as
> well as most of the configuration) as a black-box. I even did so myself
> for many years early on.
Me too, but after I learned some Elisp, I don't do that any more. How do
you do now when I are more fluent in Elisp?

> I definitely see the same concept being extended to AOT and being a net
> advantage in such cases.

A problem with Emacs is that, there are different cases for different
users, which sometimes even get orthogonal to each other so it can be
hard to make everyone happy att same time.

Re: Add a configure option for NATIVE_FULL_AOT?

[Stefan Monnier]

> Just to touch on this.  ATM we do start a native-compilation only when
> we load a .elc for which no .eln is found.  The same for the .el files
> is not enabled.  I guess the main reason is that we would loose the
> ability to step but I suspect wouldn't be the only drawback.

Also it would probably break on some corner case files that can't
be byte-compiled correctly (typically because they need other files to
be loaded before byte-compilation can be performed).

Basically, byte-compilation is *not* transparent (and it's currently
important because byte-compilation is the step where we give feedback
to the coder, so we don't want to postpone that to the moment the coder
is gone and has been replaced by the end-user who doesn't know what to
do with byte-compilation warnings/errors).

Fundamentally, .eln files are generated from .elc files (tho
currently, the code works a bit differently, which is arguably a bug,
which may(will?) bump into the above problem with .el files).


        Stefan

Re: Add a configure option for NATIVE_FULL_AOT?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > > It is strange for a Free Software project to assume the user will
    > > never want to modify the sources.
    > 
    > I don't think that assumption is implied by making a system-wide install
    > of a package in a location not writable by users.

    That's not what I said or meant.  What I meant was that considering
    the problem non-existent because this is how distros install the Lisp
    files sounds strange to me, because it assumes no user will ever want
    to modify these files enough to make them writable.  IOW, the
    assumption that bothered me was that no one will ever want to modify
    those files, e.g., to fix some bug or add a feature.

Distros must provide a way to download the sources.  You could
download the sources into your home directory, make your modified
versions, and put them in a directory in load-path.

Does this solve the problem, in practical terms?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1791 bytes --]

On Wed, Aug 18, 2021 at 10:34:10PM -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>     > > It is strange for a Free Software project to assume the user will
>     > > never want to modify the sources.
>     > 
>     > I don't think that assumption is implied by making a system-wide install
>     > of a package in a location not writable by users.
> 
>     That's not what I said or meant.  What I meant was that considering
>     the problem non-existent because this is how distros install the Lisp
>     files sounds strange to me, because it assumes no user will ever want
>     to modify these files enough to make them writable.  IOW, the
>     assumption that bothered me was that no one will ever want to modify
>     those files, e.g., to fix some bug or add a feature.
> 
> Distros must provide a way to download the sources.

They actually do. At least those worth their salt :-)

>                                                   You could
> download the sources into your home directory, make your modified
> versions, and put them in a directory in load-path.

The topic being discussed is more of a convenience/organisational
thing: keeping those spaces where a distro update can overwrite
things separate from those where the user can count on things not
being overwritten.

On multi-user systems, there is the additional dimension of keeping
changes local to the system (made available by the sysadmin to all
users) separate from the distro.

> Does this solve the problem, in practical terms?

So we hope :-)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3166 bytes --]

On Thu, Aug 19, 2021 at 02:57:55AM +0200, Arthur Miller wrote:
> Yuri D'Elia <wavexx@thregr.org> writes:
> 
> > On Wed, Aug 18 2021, Andreas Schwab wrote:
> >> On Aug 18 2021, Arthur Miller wrote:
> >>
> >>> Sorry, I picking on it, I know that most of distributions do so, but
> >>> that is unfortunate practice against the nature of Emacs as application,
> >>> since Emacs comes with sources as fully modifiable and extendable
> >>> editor.
> >>
> >> Nothing prevents you from reading and modifying the lisp files.
> Y
> > I don't want to add anything which hasn't been said by others already,
> > but just point out that the way that emacs is packaged in debian is
> > actually pretty nice and convenient for many users, especially in a
> > multi-tenant setup.
> I haven't seen a Debian since somewhere around 2001 or something, so I
> really don't know how they do. But I think that many distros put elisp
> in /usr/share which is not user modifiable location by default.

Basically, this is the FHS. /usr/share is for architecture-independent,
mostly immutable [1] stuff. Scripts written in some scripting language.
Timezone data. Bytecodes. That kind of stuff.

The idea of separating arch-independent and arch-dependent stuff stems
from old times where disk space was at a premium and you wanted to share
one /usr via NFS in your heterogeneous network. A kind of deduplication,
if you like.

But in these days of emulators, cross-compiles and cross-builds it does
reveal a big potential. With qemu and some luck I can run things meant
for a Raspberry Pi on my AMD64 laptop [2] and share... my /usr/share,
which is kind of nifty :-)

> I am trying to see what Emacs uses by default choice [...]

[...]

Actually those are the things the FHS developed from. And yes, by
default /usr/local makes sense: that's where you want to have the
stuff installed when you compile things yourself: the distro won't
touch them.

But it provides infrastructure for you to use them. Have a look at
your default $PATH: you'll typically see "/usr/local/bin" before
"/usr/bin". There are many, many places where this precedence of
/usr/local before /usr is encoded. /etc/ld.so.conf*, as another
example.

> > I definitely see the same concept being extended to AOT and being a net
> > advantage in such cases.
> 
> A problem with Emacs is that, there are different cases for different
> users, which sometimes even get orthogonal to each other so it can be
> hard to make everyone happy att same time.

Definitely. Not only different users, but also different usages. My
system has about 2k packages installed. Some of them I barely know
by name. I'm infinitely thankful to the Debian Developer who keeps
them happy and humming for me. Others are my pets, I download their
source, learn their build quirks (Emacs is one of them, you guessed).
Those go to /usr/local :-)

Cheers

[1] During installation & updates all bets are off :-)
[2] A bit more of effort is needed, of course, like Debian's Multi-Arch,
   which splits /usr/lib into /usr/lib/<arch-triple>, but I think it's
   totally worth it :-)

 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: tsdh@gnu.org, emacs-devel@gnu.org
> Date: Wed, 18 Aug 2021 22:34:10 -0400
> 
>     That's not what I said or meant.  What I meant was that considering
>     the problem non-existent because this is how distros install the Lisp
>     files sounds strange to me, because it assumes no user will ever want
>     to modify these files enough to make them writable.  IOW, the
>     assumption that bothered me was that no one will ever want to modify
>     those files, e.g., to fix some bug or add a feature.
> 
> Distros must provide a way to download the sources.  You could
> download the sources into your home directory, make your modified
> versions, and put them in a directory in load-path.
> 
> Does this solve the problem, in practical terms?

Not the problem I was describing, it doesn't.

The problem I was describing was with the user modifying the *.el
files installed by a distro.  (The fact that distros by default
install *.el files in a place that is generally not writable by users
is irrelevant, because they can be made writable, or the *.el files
can be copied to a writable location and modified there.)  When users
do modify the *.el files, the corresponding distributed *.eln files
will no longer be loaded by Emacs, and there will be another version
of those *.eln files in a different location.  That is a gate to the
"DLL hell" in its Emacs incarnation: several different shared-library
objects of the same name in different locations.  At the very least,
users will be confused, and several obscure problems could happen that
will be hard to debug.  For example, the user's eln-cache directory,
where the updated *.eln files will be stored, could be cleaned up by
the user, not knowing that by doing that he/she effectively reverts to
using the old code, triggers JIT compilation when he/she loads the
same file the next time, etc.  Not a catastrophe, but certainly
confusing and not expected.

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  tomas@tuxteam.de,  emacs-devel@gnu.org
> Date: Wed, 18 Aug 2021 21:19:13 -0400
> 
> Fundamentally, .eln files are generated from .elc files (tho
> currently, the code works a bit differently, which is arguably a bug,
> which may(will?) bump into the above problem with .el files).

Currently, AFAIU native compilation is a side effect of byte
compilation, in the sense that byte compilation _drives_ the
native-compiler passes which produce the intermediate-language
representation of the Lisp being compiled, to be submitted to the
libgccjit library for producing the native code.  So I'm not sure
describing this as "from .elc fils" is close enough to the reality to
be helpful for understanding the process of native-compilation.
Andrea will correct me, but I don't see how one can switch to using
the .elc files as input for the native-compiler, as some information
about the program structure is probably missing from there.

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Date: Thu, 19 Aug 2021 02:57:55 +0200
> Cc: Ulrich Mueller <ulm@gentoo.org>, Andreas Schwab <schwab@linux-m68k.org>,
>  Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> I am trying to see what Emacs uses by default choice in configure script
> for lisp sources, but I am not sure I understand in which category are
> those:

It's in the place specified by --datadir (which defaults to --datadirroot).

Re: Add a configure option for NATIVE_FULL_AOT?

[Andreas Schwab]

On Aug 19 2021, Eli Zaretskii wrote:

> When users do modify the *.el files, the corresponding distributed
> *.eln files will no longer be loaded by Emacs, and there will be
> another version of those *.eln files in a different location.  That is
> a gate to the "DLL hell" in its Emacs incarnation: several different
> shared-library objects of the same name in different locations.

This problem exists independent of how emacs has been built and
installed.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, tomas@tuxteam.de,
>         emacs-devel@gnu.org
> Date: Wed, 18 Aug 2021 19:44:20 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> The way the .eln file naming has been designed is for the .eln files to
> >> be "transparent", i.e. Emacs first looks for a `.elc` or `.el` file and once
> >> it finds it, it looks for the corresponding `.eln` file.
> >
> > I think it actually requires to see the .el file, otherwise it won't
> > load the .eln file.  The .elc file is not enough.  Andrea, am I right?
> 
> That is correct.

So, something to consider for the distros which want to distribute the
*.eln files: users who want to use those *.eln files will need to also
install the *.el files.

And if the *.el files aren't installed, I think comp.el will complain
each time a .elc file is loaded, saying it couldn't use the .eln, is
that right?

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2535 bytes --]

On Thu, Aug 19, 2021 at 10:07:02AM +0300, Eli Zaretskii wrote:
> > From: Richard Stallman <rms@gnu.org>
> > Cc: tsdh@gnu.org, emacs-devel@gnu.org
> > Date: Wed, 18 Aug 2021 22:34:10 -0400
> > 
> >     That's not what I said or meant.  What I meant was that considering
> >     the problem non-existent because this is how distros install the Lisp
> >     files sounds strange to me, because it assumes no user will ever want
> >     to modify these files enough to make them writable.  IOW, the
> >     assumption that bothered me was that no one will ever want to modify
> >     those files, e.g., to fix some bug or add a feature.
> > 
> > Distros must provide a way to download the sources.  You could
> > download the sources into your home directory, make your modified
> > versions, and put them in a directory in load-path.
> > 
> > Does this solve the problem, in practical terms?
> 
> Not the problem I was describing, it doesn't.
> 
> The problem I was describing was with the user modifying the *.el
> files installed by a distro.  (The fact that distros by default
> install *.el files in a place that is generally not writable by users
> is irrelevant, because they can be made writable, or the *.el files
> can be copied to a writable location and modified there.)  When users
> do modify the *.el files, the corresponding distributed *.eln files
> will no longer be loaded by Emacs, and there will be another version
> of those *.eln files in a different location.  That is a gate to the
> "DLL hell" in its Emacs incarnation: several different shared-library
> objects of the same name in different locations.  At the very least,
> users will be confused, and several obscure problems could happen that
> will be hard to debug.  For example, the user's eln-cache directory,
> where the updated *.eln files will be stored, could be cleaned up by
> the user, not knowing that by doing that he/she effectively reverts to
> using the old code, triggers JIT compilation when he/she loads the
> same file the next time, etc.  Not a catastrophe, but certainly
> confusing and not expected.

Sometimes, questions just have to be put by somewhere else. Now
I understand your position better :-)

I agree that this problem hasn't a trivial (or unique) solution.
But this is something for distros to solve. Emacs has just to
provide the mechanisms to make that possible (and, as far as I
can see, it does).

Note that this problem already exists with .el/.elc files.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: rms@gnu.org,  emacs-devel@gnu.org,  tsdh@gnu.org
> Date: Thu, 19 Aug 2021 09:17:19 +0200
> 
> On Aug 19 2021, Eli Zaretskii wrote:
> 
> > When users do modify the *.el files, the corresponding distributed
> > *.eln files will no longer be loaded by Emacs, and there will be
> > another version of those *.eln files in a different location.  That is
> > a gate to the "DLL hell" in its Emacs incarnation: several different
> > shared-library objects of the same name in different locations.
> 
> This problem exists independent of how emacs has been built and
> installed.

No, it doesn't.

Re: Add a configure option for NATIVE_FULL_AOT?

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, tomas@tuxteam.de,
>>         emacs-devel@gnu.org
>> Date: Wed, 18 Aug 2021 19:44:20 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> The way the .eln file naming has been designed is for the .eln files to
>> >> be "transparent", i.e. Emacs first looks for a `.elc` or `.el` file and once
>> >> it finds it, it looks for the corresponding `.eln` file.
>> >
>> > I think it actually requires to see the .el file, otherwise it won't
>> > load the .eln file.  The .elc file is not enough.  Andrea, am I right?
>> 
>> That is correct.
>
> So, something to consider for the distros which want to distribute the
> *.eln files: users who want to use those *.eln files will need to also
> install the *.el files.
>
> And if the *.el files aren't installed, I think comp.el will complain
> each time a .elc file is loaded, saying it couldn't use the .eln, is
> that right?

Correct, we have a warning for that
(`native-comp-warning-on-missing-source').

  Andrea

Re: Add a configure option for NATIVE_FULL_AOT?

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  tomas@tuxteam.de,  emacs-devel@gnu.org
>> Date: Wed, 18 Aug 2021 21:19:13 -0400
>> 
>> Fundamentally, .eln files are generated from .elc files (tho
>> currently, the code works a bit differently, which is arguably a bug,
>> which may(will?) bump into the above problem with .el files).
>
> Currently, AFAIU native compilation is a side effect of byte
> compilation, in the sense that byte compilation _drives_ the
> native-compiler passes which produce the intermediate-language
> representation of the Lisp being compiled, to be submitted to the
> libgccjit library for producing the native code.  So I'm not sure
> describing this as "from .elc fils" is close enough to the reality to
> be helpful for understanding the process of native-compilation.
> Andrea will correct me, but I don't see how one can switch to using
> the .elc files as input for the native-compiler, as some information
> about the program structure is probably missing from there.

I think we could try to disassemble the bytecode and use it as input,
but as you suggests some information about the program structure might
be lost.  My mind now goes to declarations but there might be others.

In general it's a complex problem and I think one could prove it can
work or not only after having attempted an implementation.

BR

  Andrea

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> Date: Thu, 19 Aug 2021 09:27:06 +0200
> From: <tomas@tuxteam.de>
> 
> I agree that this problem hasn't a trivial (or unique) solution.
> But this is something for distros to solve. Emacs has just to
> provide the mechanisms to make that possible (and, as far as I
> can see, it does).

This discussion started when a distro came here asking for a change
where Emacs already provides the mechanism to make that possible.  So
evidently, distros don't expect to have to solve all such problems by
themselves, they want some of them solved upstream.  I have no doubt
that at least some of the problems caused by AOT native-compilation
will be brought back to us for a solution.  Which is as it should be,
of course; all I'm saying since the beginning of this thread is that
we are opening a can of worms we didn't consider important enough till
now, and I fear it will affect the schedule for Emacs 28 release.  I'm
not against the change; if it's what many people want, we should
provide it.  But during the last 8 months this paradigm wasn't
considered as important, and therefore it could be that some parts of
the implementation will produce unpleasant surprises when AOT
compilation is used.

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1580 bytes --]

On Thu, Aug 19, 2021 at 11:09:23AM +0300, Eli Zaretskii wrote:
> > Date: Thu, 19 Aug 2021 09:27:06 +0200
> > From: <tomas@tuxteam.de>
> > 
> > I agree that this problem hasn't a trivial (or unique) solution.
> > But this is something for distros to solve. Emacs has just to
> > provide the mechanisms to make that possible (and, as far as I
> > can see, it does).
> 
> This discussion started when a distro came here asking for a change
> where Emacs already provides the mechanism to make that possible.

But perhaps, it's possible as it is. What distors want, one way or
another is to provide pre-compiled .eln files for the .el libraries
they provide. At the same time giving users a way to override some
of them locally.

If Emacs is capable to locate the correct .eln (as far as I have
understood now, it searches for the .el to resolve that), the
above should be possible, no?

For example, if the distro provides the .el (most probably those
will be compressed, as .el.gz, but I think that's a transparent
detail), the corresponding .elc and .eln) all would be well.

If a user wants to hack on a single .el, she would typically
copy that to a user controlled place [1] and the corresponding
.elc and .eln would also go to corresponding user-controlled
spaces and would, hopefully, override the system-provided ones,
if all `path' variables are set up correctly.

It could well be that I'm missing something important, though.

[1] She could (most of the time) also hack the files in-place,
   but updates would obliterate those changes.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> Date: Thu, 19 Aug 2021 12:05:00 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > This discussion started when a distro came here asking for a change
> > where Emacs already provides the mechanism to make that possible.
> 
> But perhaps, it's possible as it is. What distors want, one way or
> another is to provide pre-compiled .eln files for the .el libraries
> they provide. At the same time giving users a way to override some
> of them locally.
> 
> If Emacs is capable to locate the correct .eln (as far as I have
> understood now, it searches for the .el to resolve that), the
> above should be possible, no?

It is possible.  The question is how widely it will (or should be)
used.

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 501 bytes --]

On Thu, Aug 19, 2021 at 01:51:00PM +0300, Eli Zaretskii wrote:
> > Date: Thu, 19 Aug 2021 12:05:00 +0200
> > From: tomas@tuxteam.de
> > Cc: emacs-devel@gnu.org
> > 
> > > This discussion started when a distro came here asking for a change
> > > where Emacs already provides the mechanism to make that possible.
> > 
> > But perhaps, it's possible as it is [...]

> It is possible.  The question is how widely it will (or should be)
> used.

I see, you remain sceptic :-)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> Date: Thu, 19 Aug 2021 14:49:35 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > It is possible.  The question is how widely it will (or should be)
> > used.
> 
> I see, you remain sceptic :-)

I just remember how many times during the last 8 months I said to
myself: oh, but with JIT compilation this (whatever "this" was) is
okay, I guess".  I don't even remember all the issues I considered,
just that this was what solved them for me.

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]

On Thu, Aug 19, 2021 at 03:52:31PM +0300, Eli Zaretskii wrote:

[...]

> I just remember how many times during the last 8 months I said to
> myself: oh, but with JIT compilation this (whatever "this" was) is
> okay, I guess".  I don't even remember all the issues I considered,
> just that this was what solved them for me.

Thanks for your huge patience :)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Date: Thu, 19 Aug 2021 02:57:55 +0200
>> Cc: Ulrich Mueller <ulm@gentoo.org>, Andreas Schwab <schwab@linux-m68k.org>,
>>  Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>> 
>> I am trying to see what Emacs uses by default choice in configure script
>> for lisp sources, but I am not sure I understand in which category are
>> those:
>
> It's in the place specified by --datadir (which defaults to --datadirroot).

Thank you.

Re: Add a configure option for NATIVE_FULL_AOT?

[Arthur Miller]

<tomas@tuxteam.de> writes:

> On Thu, Aug 19, 2021 at 02:57:55AM +0200, Arthur Miller wrote:
>> Yuri D'Elia <wavexx@thregr.org> writes:
>> 
>> > On Wed, Aug 18 2021, Andreas Schwab wrote:
>> >> On Aug 18 2021, Arthur Miller wrote:
>> >>
>> >>> Sorry, I picking on it, I know that most of distributions do so, but
>> >>> that is unfortunate practice against the nature of Emacs as application,
>> >>> since Emacs comes with sources as fully modifiable and extendable
>> >>> editor.
>> >>
>> >> Nothing prevents you from reading and modifying the lisp files.
>> Y
>> > I don't want to add anything which hasn't been said by others already,
>> > but just point out that the way that emacs is packaged in debian is
>> > actually pretty nice and convenient for many users, especially in a
>> > multi-tenant setup.
>> I haven't seen a Debian since somewhere around 2001 or something, so I
>> really don't know how they do. But I think that many distros put elisp
>> in /usr/share which is not user modifiable location by default.
>
> Basically, this is the FHS. /usr/share is for architecture-independent,
> mostly immutable [1] stuff. Scripts written in some scripting language.
> Timezone data. Bytecodes. That kind of stuff.

So emacs sources should not be in there. But than as other stuff also
will also not end there, like .elc files if .el files are not there it
almost implied that nothing of Emacs should be in /usr/share :-). So I
guess, as suggested, someone who wishes to modify Emacs sources should
download sources to their home directories and load after so all
headaches avoided :).

> The idea of separating arch-independent and arch-dependent stuff stems
> from old times where disk space was at a premium and you wanted to share
> one /usr via NFS in your heterogeneous network. A kind of deduplication,
> if you like.
>
> But in these days of emulators, cross-compiles and cross-builds it does
> reveal a big potential. With qemu and some luck I can run things meant
> for a Raspberry Pi on my AMD64 laptop [2] and share... my /usr/share,
> which is kind of nifty :-)

Indeed, that is nifty.
 
Thanks for deatiled mail, that was indeed an interesting read!

Cheers!

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 4731 bytes --]

On Thu, Aug 19, 2021 at 11:17:57PM +0200, Arthur Miller wrote:
> <tomas@tuxteam.de> writes:
> 
> > On Thu, Aug 19, 2021 at 02:57:55AM +0200, Arthur Miller wrote:
> >> Yuri D'Elia <wavexx@thregr.org> writes:
> >> 
> >> > On Wed, Aug 18 2021, Andreas Schwab wrote:
> >> >> On Aug 18 2021, Arthur Miller wrote:
> >> >>
> >> >>> Sorry, I picking on it, I know that most of distributions do so, but
> >> >>> that is unfortunate practice against the nature of Emacs as application,
> >> >>> since Emacs comes with sources as fully modifiable and extendable
> >> >>> editor.
> >> >>
> >> >> Nothing prevents you from reading and modifying the lisp files.
> >> Y
> >> > I don't want to add anything which hasn't been said by others already,
> >> > but just point out that the way that emacs is packaged in debian is
> >> > actually pretty nice and convenient for many users, especially in a
> >> > multi-tenant setup.
> >> I haven't seen a Debian since somewhere around 2001 or something, so I
> >> really don't know how they do. But I think that many distros put elisp
> >> in /usr/share which is not user modifiable location by default.
> >
> > Basically, this is the FHS. /usr/share is for architecture-independent,
> > mostly immutable [1] stuff. Scripts written in some scripting language.
> > Timezone data. Bytecodes. That kind of stuff.
> 
> So emacs sources should not be in there.

That depends on what you mean by "sources" :-)

C sources "typically" not. But .el sources can be thought of as "scripts",
so possibly yes. Operatively, they can be replaced by the .elc, so a
distro which wants to give users the choice of "not installing sources"
(e.g. you want to *use* Emacs on your constrained Raspi, but to hack
on it you've got your shining new $10K Linux laptop), .el wouldn't go
there.

Debian, for example has one source package for each (set of) binary
package(s), including all the necessary buildery to reproduce those.
If you want sources, you download those. I think RedHat-ish distros
work more or less the same.

>                                          But than as other stuff also
> will also not end there, like .elc files if .el files are not there it
> almost implied that nothing of Emacs should be in /usr/share :-).

.elc are arch-independent files. Those are the typical examples of
things which go there. Also possibly .el, especially in the cases
you don't want them compiled. Icons, .xml, that kind of stuff. Docs
(/usr/share/doc).

To get an idea on what Debian (courtesy of the Emacsen maintainer,
Rob Browning) puts there for Emacs, have a look at [1].

>                                                                  So I
> guess, as suggested, someone who wishes to modify Emacs sources should
> download sources to their home directories and load after so all
> headaches avoided :).

That's the way I do it, since I compile from sources anyway. But if
someone wants to just override one .el in the whole kaboodle, and
this perhaps only temporarily, I don't see why the distro would have
to force her to do that, while there are easier ways.

> > But in these days of emulators [...] share... my /usr/share,
> > which is kind of nifty :-)
> 
> Indeed, that is nifty.

Actually this is just the repetition of one pattern, that of the
"nested environments", so dear to us in programming languages (think
scopes). The innermost environment takes precedence over the next
outer layers of the onion [2]. The world is a tree, where "your"
private world only depends on your path to the root. Yadda, yadda.

In the current OS context, there are three main layers -- operating
system (/bin, /lib, /usr/bin, etc.), machine instance (/usr/local
and so on, but also /etc), user whithin machine (/home/foo, mostly).

Somewhere between OS and machine instance there is another layer,
machine architecture.

That's why I was so surprised to see the tendency here "every user
has all of their .eln files in a local place". I'm not convinced
at all that this is a good idea [4]. Eli seems to be convinced of
this, and he's a much smarter person than me. Interesting :-)

Cheers
[1] https://packages.debian.org/buster/all/emacs-common/filelist
[2] I should rather say "shallot" [3], since it has several cores,
   resembling more the tree structure. Besides... yummy ;-)
[3] https://packages.debian.org/buster/all/emacs-common/filelist
[4] I'd prefer those .eln derived from (unchanged) system files
   to live in system places (outer layers of the shallot), and
   be provided, pre-compiled, by the system. Possibly in the
   architecture-dependent layer (they're architecture-dependent,
   I guess).

 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Gunnar Horrigmo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gunnar Horrigmo <horrigmo@runbox.no>
>> 
>> Well, that's depressing, but not paying attention is on me.
>
> AFAIR, the main argument against using the XDG's cache was that the
> definition of its purpose didn't fit, and that XDG directories are
> ephemeral.

I don't know why that is a problem, but if it is, then the right place
would be $XDG_STATE_HOME or $XDG_DATA_HOME? I think? 

>> I won't attempt to revive that discussion, but is there some guidance on
>> how to configure this in .emacs (probably not, but by any means), so we
>> can have a more... compliant environment (if you'll excuse the term)?
>
> I don't think I have a clear understanding of what you want to
> achieve, so please tell more.

I simply want to store the .eln files in an XDG defined directory, even
if you guys think I shouldn't. :)

-- 
Gunnar

Re: Add a configure option for NATIVE_FULL_AOT?

[Eli Zaretskii]

> From: Gunnar Horrigmo <horrigmo@runbox.no>
> Date: Fri, 20 Aug 2021 10:22:59 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Gunnar Horrigmo <horrigmo@runbox.no>
> >> 
> >> Well, that's depressing, but not paying attention is on me.
> >
> > AFAIR, the main argument against using the XDG's cache was that the
> > definition of its purpose didn't fit, and that XDG directories are
> > ephemeral.
> 
> I don't know why that is a problem, but if it is, then the right place
> would be $XDG_STATE_HOME or $XDG_DATA_HOME? I think? 

What is the definition of "state files" and "data files" for this
purpose?  Just be considering the usual meaning of these, *.eln files
are neither, but maybe the Freedesktop definitions are not "usual".

> >> I won't attempt to revive that discussion, but is there some guidance on
> >> how to configure this in .emacs (probably not, but by any means), so we
> >> can have a more... compliant environment (if you'll excuse the term)?
> >
> > I don't think I have a clear understanding of what you want to
> > achieve, so please tell more.
> 
> I simply want to store the .eln files in an XDG defined directory, even
> if you guys think I shouldn't. :)

You can always customize native-comp-eln-load-path, can't you?

Re: Add a configure option for NATIVE_FULL_AOT?

[Arthur Miller]

tomas@tuxteam.de writes:

> On Thu, Aug 19, 2021 at 11:17:57PM +0200, Arthur Miller wrote:
>> <tomas@tuxteam.de> writes:
>> 
>> > On Thu, Aug 19, 2021 at 02:57:55AM +0200, Arthur Miller wrote:
>> >> Yuri D'Elia <wavexx@thregr.org> writes:
>> >> 
>> >> > On Wed, Aug 18 2021, Andreas Schwab wrote:
>> >> >> On Aug 18 2021, Arthur Miller wrote:
>> >> >>
>> >> >>> Sorry, I picking on it, I know that most of distributions do so, but
>> >> >>> that is unfortunate practice against the nature of Emacs as application,
>> >> >>> since Emacs comes with sources as fully modifiable and extendable
>> >> >>> editor.
>> >> >>
>> >> >> Nothing prevents you from reading and modifying the lisp files.
>> >> Y
>> >> > I don't want to add anything which hasn't been said by others already,
>> >> > but just point out that the way that emacs is packaged in debian is
>> >> > actually pretty nice and convenient for many users, especially in a
>> >> > multi-tenant setup.
>> >> I haven't seen a Debian since somewhere around 2001 or something, so I
>> >> really don't know how they do. But I think that many distros put elisp
>> >> in /usr/share which is not user modifiable location by default.
>> >
>> > Basically, this is the FHS. /usr/share is for architecture-independent,
>> > mostly immutable [1] stuff. Scripts written in some scripting language.
>> > Timezone data. Bytecodes. That kind of stuff.
>> 
>> So emacs sources should not be in there.
>
> That depends on what you mean by "sources" :-)
>
> C sources "typically" not. But .el sources can be thought of as "scripts",
> so possibly yes. Operatively, they can be replaced by the .elc, so a
> distro which wants to give users the choice of "not installing sources"
> (e.g. you want to *use* Emacs on your constrained Raspi, but to hack
> on it you've got your shining new $10K Linux laptop), .el wouldn't go
> there.
Is there a laptop that costs $10k? What is deal with that? Cover made of
gold? :-)

> Debian, for example has one source package for each (set of) binary
> package(s), including all the necessary buildery to reproduce those.
> If you want sources, you download those. I think RedHat-ish distros
> work more or less the same.
>
>>                                          But than as other stuff also
>> will also not end there, like .elc files if .el files are not there it
>> almost implied that nothing of Emacs should be in /usr/share :-).
>
> .elc are arch-independent files. Those are the typical examples of
> things which go there. Also possibly .el, especially in the cases
> you don't want them compiled. Icons, .xml, that kind of stuff. Docs
> (/usr/share/doc).
But with Emacs one might wish to modify sources (at least in theory)
even if when not compiling them to elc or eln. I don't know why would
someone wish to run only interpretted code, but a theoretical
possibility. Even docs are subject of use modification. I mean at least
on ideological level. Emacs is very atypical application in this regard.

But as suggested by several here, I can also agree that is more
pragmatic to just download/copy whichever file needs to be modified and
place it elsewhere, so I don't think we have to be super lazer precise
of what goes where.

> To get an idea on what Debian (courtesy of the Emacsen maintainer,
> Rob Browning) puts there for Emacs, have a look at [1].
>
>>                                                                  So I
>> guess, as suggested, someone who wishes to modify Emacs sources should
>> download sources to their home directories and load after so all
>> headaches avoided :).
>
> That's the way I do it, since I compile from sources anyway. But if
> someone wants to just override one .el in the whole kaboodle, and
> this perhaps only temporarily, I don't see why the distro would have
> to force her to do that, while there are easier ways.
I actually run emacs from source dir. I compile it every few days, so it
does not really make sense to me to run make-install and copy over
everything to some other directory every now and then. Also when running
from source directory, without compressing sources, looking up stuff
with built in help is more efficient, and even on my i7 6700k desktop
with 32 gig ram and m.2 970 Pro drive feels faster. I still though copy
lisp file I wish to modify elsewhere, and if wish changes to persist I
copy entire function somewhere to my config and re-eval it after Emacs
starts. That way I don't have to switch branches when I wish to pull
latest and rebuild, or have two source trees etc. I don't know. Many
ways to play with a dear child.

>> > But in these days of emulators [...] share... my /usr/share,
>> > which is kind of nifty :-)
>> 
>> Indeed, that is nifty.
>
> Actually this is just the repetition of one pattern, that of the
> "nested environments", so dear to us in programming languages (think
> scopes). The innermost environment takes precedence over the next
> outer layers of the onion [2]. The world is a tree, where "your"
> private world only depends on your path to the root. Yadda, yadda.
>
> In the current OS context, there are three main layers -- operating
> system (/bin, /lib, /usr/bin, etc.), machine instance (/usr/local
> and so on, but also /etc), user whithin machine (/home/foo, mostly).
I agree, and I am aware of that; I use myself /usr/local/bin to place
few scripts where I pre-configure environments for few apps, Emacs
amongst them, where I add some special paths or envs which I don't use
system-wide. Isn't it also the same idea that "containers"
(docker/kubernettes) exploit so successfuly? But I think we are
regressing quite a bit now :).

> Somewhere between OS and machine instance there is another layer,
> machine architecture.
>
> That's why I was so surprised to see the tendency here "every user
> has all of their .eln files in a local place". I'm not convinced
> at all that this is a good idea [4]. Eli seems to be convinced of
> this, and he's a much smarter person than me. Interesting :-)
I have learned is that what Eli says is usually the right thing to do,
and he got a migrain after this comming up, so I don't think I have so
much more to add here, since my expertise with native comp does not go
further from enabliing it in the configure script, unfortunately.

> Cheers
> [1] https://packages.debian.org/buster/all/emacs-common/filelist
> [2] I should rather say "shallot" [3], since it has several cores,
>    resembling more the tree structure. Besides... yummy ;-)
> [3] https://packages.debian.org/buster/all/emacs-common/filelist
> [4] I'd prefer those .eln derived from (unchanged) system files
>    to live in system places (outer layers of the shallot), and
>    be provided, pre-compiled, by the system. Possibly in the
>    architecture-dependent layer (they're architecture-dependent,
>    I guess).
>
>  - t

Re: Add a configure option for NATIVE_FULL_AOT?

[Gunnar Horrigmo]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Gunnar Horrigmo <horrigmo@runbox.no>
>> Eli Zaretskii <eliz@gnu.org> writes:
>> >
>> > AFAIR, the main argument against using the XDG's cache was that the
>> > definition of its purpose didn't fit, and that XDG directories are
>> > ephemeral.
>> 
>> I don't know why that is a problem, but if it is, then the right place
>> would be $XDG_STATE_HOME or $XDG_DATA_HOME? I think? 
>
> What is the definition of "state files" and "data files" for this
> purpose?  Just be considering the usual meaning of these, *.eln files
> are neither, but maybe the Freedesktop definitions are not "usual".

| $XDG_DATA_HOME defines the base directory relative to which
| user-specific data files should be stored. If $XDG_DATA_HOME is either
| not set or empty, a default equal to $HOME/.local/share should be
| used.
| 
| $XDG_STATE_HOME defines the base directory relative to which
| user-specific state files should be stored. If $XDG_STATE_HOME is
| either not set or empty, a default equal to $HOME/.local/state should
| be used.
| 
| The $XDG_STATE_HOME contains state data that should persist between
| (application) restarts, but that is not important or portable enough
| to the user that it should be stored in $XDG_DATA_HOME. It may
| contain:
| 
|     actions history (logs, history, recently used files, …)
| 
|     current state of the application that can be reused on a restart
|     (view, layout, open files, undo history, …)

So I guess you're right about XDG_STATE_HOME. But I think we should be 
free to define what data is ourselves?

There is also this, just for completeness sake:

| $XDG_RUNTIME_DIR defines the base directory relative to which
| user-specific non-essential runtime files and other file objects
| (such as sockets, named pipes, ...) should be stored. The directory
| MUST be owned by the user, and he MUST be the only one having read
| and write access to it. Its Unix access mode MUST be 0700.
|
| The lifetime of the directory MUST be bound to the user being logged
| in. It MUST be created when the user first logs in and if the user
| fully logs out the directory MUST be removed. If the user logs in more
| than once he should get pointed to the same directory, and it is
| mandatory that the directory continues to exist from his first login
| to his last logout on the system, and not removed in between. Files in
| the directory MUST not survive reboot or a full logout/login cycle.
|
| The directory MUST be on a local file system and not shared with any
| other system. The directory MUST by fully-featured by the standards of
| the operating system. More specifically, on Unix-like operating
| systems AF_UNIX sockets, symbolic links, hard links, proper
| permissions, file locking, sparse files, memory mapping, file change
| notifications, a reliable hard link count must be supported, and no
| restrictions on the file name character set should be imposed. Files
| in this directory MAY be subjected to periodic clean-up. To ensure
| that your files are not removed, they should have their access time
| timestamp modified at least once every 6 hours of monotonic time or
| the 'sticky' bit should be set on the file.

That said, I really didn't mean to challenge already made decisions.
Especially in the runup to 28. I'm perfectly happy to live with the
current situation.


>> >> I won't attempt to revive that discussion, but is there some guidance on
>> >> how to configure this in .emacs (probably not, but by any means), so we
>> >> can have a more... compliant environment (if you'll excuse the term)?
>> >
>> > I don't think I have a clear understanding of what you want to
>> > achieve, so please tell more.
>> 
>> I simply want to store the .eln files in an XDG defined directory, even
>> if you guys think I shouldn't. :)
>
> You can always customize native-comp-eln-load-path, can't you?

I can indeed. I guess this is what I originally meant to ask. Thank you
for the pointer, and sorry for the disruption. I should have
investigated instead of letting my OCD panic. :)

-- 
Gunnar

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3687 bytes --]

On Fri, Aug 20, 2021 at 02:06:39PM +0200, Arthur Miller wrote:
> tomas@tuxteam.de writes:
> 
> > On Thu, Aug 19, 2021 at 11:17:57PM +0200, Arthur Miller wrote:
> >> <tomas@tuxteam.de> writes:

[...]

> > on it you've got your shining new $10K Linux laptop), .el wouldn't go
> > there.
> Is there a laptop that costs $10k? What is deal with that? Cover made of
> gold? :-)

There used to be [1], but alas, the times, the times ;-)

[...]

> But with Emacs one might wish to modify sources (at least in theory)

And in practice. Your "But..." somehow suggests someone is trying
to prevent the user from doing that. I wouldn't touch such a
distribution :-)


> even if when not compiling them to elc or eln. I don't know why would
> someone wish to run only interpretted code, but a theoretical
> possibility. Even docs are subject of use modification. I mean at least
> on ideological level. Emacs is very atypical application in this regard.

Suppose I hear about this nice function `dired-jump' I do C-x f and
find out it's in `dired.el.gz', which is in a system location, and
thus read-only. I write it to my own location, which for me is in
~/.emacs.d/lisp, change that to my heart's content.

Due to how `load-path' is set up, this modified version takes precedence.
I take over responsibility for this one file and let the package manager
(or, in my case, Emacs's "make install") to take care of the other
7878 files. The package manager (or "make install") won't touch "my"
file.

That would be the "low profile" approach.

Of course, if I want to keep that up for a longer time, I'd clone
the Emacs repo and open a branch there. Or download the Emacs Debian
source package, if my aim is to package my personal changes to Emacs.

> But as suggested by several here, I can also agree that is more
> pragmatic to just download/copy whichever file needs to be modified and
> place it elsewhere, so I don't think we have to be super lazer precise
> of what goes where.

Hacking "in place" things which are installed by a package manager (or
by an install script) is a recipe for pain: it's you against the computer.
The computer always wins :-)

> > To get an idea on what Debian (courtesy of the Emacsen maintainer,
> > Rob Browning) puts there for Emacs, have a look at [1].

> I actually run emacs from source dir. I compile it every few days, so it
> does not really make sense to me to run make-install and copy over
> everything to some other directory every now and then.

Hm. I do, too, but I always do "make install".

>                                                  Also when running
> from source directory, without compressing sources, looking up stuff
> with built in help is more efficient, and even on my i7 6700k desktop
> with 32 gig ram and m.2 970 Pro drive feels faster. I still though copy
> lisp file I wish to modify elsewhere, and if wish changes to persist I
> copy entire function somewhere to my config and re-eval it after Emacs
> starts. That way I don't have to switch branches when I wish to pull
> latest and rebuild, or have two source trees etc. I don't know. Many
> ways to play with a dear child.

Sure :)


[...]

> > That's why I was so surprised to see the tendency here "every user
> > has all of their .eln files in a local place" [...] Eli seems to
> > be convinced of this, and he's a much smarter person than me.
> > Interesting :-)
>
> I have learned is that what Eli says is usually the right thing to do,

Same with me, that's why this situation is "interesting" :-)

Cheers

[1] https://en.wikipedia.org/wiki/IBM_ThinkPad_Power_Series
 
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Arthur Miller]

tomas@tuxteam.de writes:

> On Fri, Aug 20, 2021 at 02:06:39PM +0200, Arthur Miller wrote:
>> tomas@tuxteam.de writes:
>> 
>> > On Thu, Aug 19, 2021 at 11:17:57PM +0200, Arthur Miller wrote:
>> >> <tomas@tuxteam.de> writes:
>
> [...]
>
>> > on it you've got your shining new $10K Linux laptop), .el wouldn't go
>> > there.
>> Is there a laptop that costs $10k? What is deal with that? Cover made of
>> gold? :-)
>
> There used to be [1], but alas, the times, the times ;-)
"These units could also record PAL and NTSC signals with onboard
composite connectors, and the batteries contained internal processors to
regulate power usage for optimized battery longevity. "

Hey! You could watch TV on them, and that in both Europe and US, back in
1994! Of course if you were a dev travelign the seas you should be able
to watch that footbal game whereever you are; crucial back in time where
no company was streaming matches on the net! If there was an
application that could show that signal of course. And you were the only
one to carry around a webcam, back in time when nobody had one. Maybe not so
many to talk to, but one could make selfies while working.

"cursor driven GUI-based BIOS"

Whauh, 40 years before its time. Now when mouse in bios become
mainstream (in UEFI bios), I have yet to see the real benefit.

https://en.wikipedia.org/wiki/IBM_ThinkPad_Power_Series#/media/File:IBM_RS6000_860_laptop.jpg

Hah! CDE on Solaris, I still remember that background paper as if it was
yesterday! :).


>> I actually run emacs from source dir. I compile it every few days, so it
>> does not really make sense to me to run make-install and copy over
>> everything to some other directory every now and then.
>
> Hm. I do, too, but I always do "make install".

I don't understand, why make install if you run from the source dir?
What is the benefit? 

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 759 bytes --]

On Fri, Aug 20, 2021 at 09:51:42PM +0200, Arthur Miller wrote:
> tomas@tuxteam.de writes:
> 
> > On Fri, Aug 20, 2021 at 02:06:39PM +0200, Arthur Miller wrote:
> >> tomas@tuxteam.de writes:

[...]

> >> I actually run emacs from source dir. I compile it every few days, so it
> >> does not really make sense to me to run make-install and copy over
> >> everything to some other directory every now and then.
> >
> > Hm. I do, too, but I always do "make install".
> 
> I don't understand, why make install if you run from the source dir?
> What is the benefit? 

Ouch, no. My message got corrupted on its way to the keyboard.
I wanted to say that I self-compile too, but then do "make install".

Sorry for that sloppiness.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Arthur Miller]

tomas@tuxteam.de writes:

> On Fri, Aug 20, 2021 at 09:51:42PM +0200, Arthur Miller wrote:
>> tomas@tuxteam.de writes:
>> 
>> > On Fri, Aug 20, 2021 at 02:06:39PM +0200, Arthur Miller wrote:
>> >> tomas@tuxteam.de writes:
>
> [...]
>
>> >> I actually run emacs from source dir. I compile it every few days, so it
>> >> does not really make sense to me to run make-install and copy over
>> >> everything to some other directory every now and then.
>> >
>> > Hm. I do, too, but I always do "make install".
>> 
>> I don't understand, why make install if you run from the source dir?
>> What is the benefit? 
>
> Ouch, no. My message got corrupted on its way to the keyboard.
> I wanted to say that I self-compile too, but then do "make install".
>
> Sorry for that sloppiness.

Ahh, those electrons sometimes have strange ways. Don't worry, I should
have thought more of what was the intention, so it is a bit my rushy
personality who should have thought more.

> Cheers
>  - t

Cheers.

Re: Add a configure option for NATIVE_FULL_AOT?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 295 bytes --]

On Fri, Aug 20, 2021 at 11:25:41PM +0200, Arthur Miller wrote:

[...]

> Ahh, those electrons sometimes have strange ways. Don't worry, I should
> have thought more of what was the intention, so it is a bit my rushy
> personality who should have thought more.

Always so polite :-)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Add a configure option for NATIVE_FULL_AOT?

[Arthur Miller]

tomas@tuxteam.de writes:

> On Fri, Aug 20, 2021 at 11:25:41PM +0200, Arthur Miller wrote:
>
> [...]
>
>> Ahh, those electrons sometimes have strange ways. Don't worry, I should
>> have thought more of what was the intention, so it is a bit my rushy
>> personality who should have thought more.
>
> Always so polite :-)

The least I can be to a gentlman as yourself sir! :-)

> Cheers
>  - t