Native compilation: the bird-eye view

Native compilation: the bird-eye view

[Eli Zaretskii]

Maybe this was already discussed (in which case please point me to
that discussion), but if not: how will this feature be integrated into
the Emacs distribution and usage patterns?

First, we cannot bundle the *.eln files with the source tarball of the
Emacs release, because these files are too specific to the
architecture (and perhaps also to the versions of the OS and the libc)
of the system where they were produced.  Right?  If so, they will have
to be generated on the end-user machines, or perhaps by whoever
prepares the Emacs binary distributions -- assuming that the binary
distributions are sufficiently compatible to allow that,
notwithstanding the differences between the system where the *.eln
files were generated and the system where they will be installed and
used.  (I don't know what is the granularity of the binary distros wrt
machine architecture and OS versions -- will that allow to be sure the
*.eln files are compatible with the target system?)

The next question is whether we want the *.eln files to exist up front
for all the Lisp files on the end-user system, or we want them to be
generated in JIT-like manner, whenever the corresponding Lisp library
is loaded on demand?  The answer to this question might then influence
the place where the *.eln files are kept -- the JIT alternative would
suggest to have some kind of cache directory where we put the compiled
files, similar to what Guile does.

And finally, what about the *.elc files and their relation to the
corresponding *.eln files?  Do we always load a .eln file if
available, do we let the user specify their preferences, something
else?

Comments are welcome.

Re: Native compilation: the bird-eye view

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

> Maybe this was already discussed (in which case please point me to
> that discussion), but if not: how will this feature be integrated into
> the Emacs distribution and usage patterns?

I don't think this was discussed already.

> First, we cannot bundle the *.eln files with the source tarball of the
> Emacs release, because these files are too specific to the
> architecture (and perhaps also to the versions of the OS and the libc)
> of the system where they were produced.  Right?

Correct.

> If so, they will have
> to be generated on the end-user machines, or perhaps by whoever
> prepares the Emacs binary distributions -- assuming that the binary
> distributions are sufficiently compatible to allow that,
> notwithstanding the differences between the system where the *.eln
> files were generated and the system where they will be installed and
> used.  (I don't know what is the granularity of the binary distros wrt
> machine architecture and OS versions -- will that allow to be sure the
> *.eln files are compatible with the target system?)

Given .eln are just shareds with (almost) no dependecies [1] their
degree of compatibility should be higher then the Emacs binary itself.
Who is preparing the Emacs binary should be able to compile and
distribute the eln files too.

> The next question is whether we want the *.eln files to exist up front
> for all the Lisp files on the end-user system, or we want them to be
> generated in JIT-like manner, whenever the corresponding Lisp library
> is loaded on demand?  The answer to this question might then influence
> the place where the *.eln files are kept -- the JIT alternative would
> suggest to have some kind of cache directory where we put the compiled
> files, similar to what Guile does.

I suspect that for the average user the best is to have the distribution
do all the compilation upfront for him and have deferred compilation
handling only additional libraries and packages.

> And finally, what about the *.elc files and their relation to the
> corresponding *.eln files?  Do we always load a .eln file if
> available, do we let the user specify their preferences, something
> else?

I'd say: if the .eln is present and the suffix is not forced, this
should be preferred to the .elc in the same way now we prefer the .elc
to the .el.


  Andrea


[1] This is what an .eln dynamically link against on GNU/Linux:

====
$ ldd test.eln
        linux-vdso.so.1 (0x00007fff6d215000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3f9bb7a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3f9c16f000)
====
--
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Paul Eggert]

On 5/16/20 5:58 AM, Andrea Corallo wrote:
> I'd say: if the .eln is present and the suffix is not forced, this
> should be preferred to the .elc in the same way now we prefer the .elc
> to the .el.

Surely the .eln files won't be in the same directory as the .el and .elc files,
at least not on GNU/Linux platforms. On a Debian-style system, .el and .elc will
be in /usr/share/emacs/28.1/lisp/foo.elc (or foo.el.gz) whereas .eln will be in
/usr/lib/emacs/28.1/x86_64-linux-gnu/lisp/foo.eln because it's machine-dependent
and can't be shared across architectures. This will require more load on the
filesystem due to the extra directory lookups.

Also, how should site-lisp work? Currently on Debian it's under /usr/share/emacs
because it's architecture independent. Will we need two site-lisp directories too?

Similarly for users' .emacs.d directories -- will they need a more complicated
structure, for users who run different versions of Emacs or Emacs on different
architectures with shared home directories?

Re: Native compilation: the bird-eye view

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Date: Sat, 16 May 2020 12:58:11 +0000
> Cc: emacs-devel@gnu.org
> 
> > If so, they will have
> > to be generated on the end-user machines, or perhaps by whoever
> > prepares the Emacs binary distributions -- assuming that the binary
> > distributions are sufficiently compatible to allow that,
> > notwithstanding the differences between the system where the *.eln
> > files were generated and the system where they will be installed and
> > used.  (I don't know what is the granularity of the binary distros wrt
> > machine architecture and OS versions -- will that allow to be sure the
> > *.eln files are compatible with the target system?)
> 
> Given .eln are just shareds with (almost) no dependecies [1] their
> degree of compatibility should be higher then the Emacs binary itself.

Well, the "almost" part means we aren't actually sure about this.
E.g., what about the dependencies related to the GCC version used to
build libgccjit and compile the .eln files?

> > The next question is whether we want the *.eln files to exist up front
> > for all the Lisp files on the end-user system, or we want them to be
> > generated in JIT-like manner, whenever the corresponding Lisp library
> > is loaded on demand?  The answer to this question might then influence
> > the place where the *.eln files are kept -- the JIT alternative would
> > suggest to have some kind of cache directory where we put the compiled
> > files, similar to what Guile does.
> 
> I suspect that for the average user the best is to have the distribution
> do all the compilation upfront for him and have deferred compilation
> handling only additional libraries and packages.

And for the "non-average" user, who builds Emacs from sources?  AFAIU,
native compilation of all Lisp files takes many hours even on fast
machines -- won't this be an annoyance if we don't come up with a JIT
mechanism?

Re: Native compilation: the bird-eye view

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Date: Sat, 16 May 2020 12:58:11 +0000
>> Cc: emacs-devel@gnu.org
>> 
>> > If so, they will have
>> > to be generated on the end-user machines, or perhaps by whoever
>> > prepares the Emacs binary distributions -- assuming that the binary
>> > distributions are sufficiently compatible to allow that,
>> > notwithstanding the differences between the system where the *.eln
>> > files were generated and the system where they will be installed and
>> > used.  (I don't know what is the granularity of the binary distros wrt
>> > machine architecture and OS versions -- will that allow to be sure the
>> > *.eln files are compatible with the target system?)
>> 
>> Given .eln are just shareds with (almost) no dependecies [1] their
>> degree of compatibility should be higher then the Emacs binary itself.
>
> Well, the "almost" part means we aren't actually sure about this.
> E.g., what about the dependencies related to the GCC version used to
> build libgccjit and compile the .eln files?

For 'almost' I meant what I listed in [1].  AFAIK The GCC version used
to build libgccjit and compile the eln as no dependency implication once
the eln has been produced.

>> > The next question is whether we want the *.eln files to exist up front
>> > for all the Lisp files on the end-user system, or we want them to be
>> > generated in JIT-like manner, whenever the corresponding Lisp library
>> > is loaded on demand?  The answer to this question might then influence
>> > the place where the *.eln files are kept -- the JIT alternative would
>> > suggest to have some kind of cache directory where we put the compiled
>> > files, similar to what Guile does.
>> 
>> I suspect that for the average user the best is to have the distribution
>> do all the compilation upfront for him and have deferred compilation
>> handling only additional libraries and packages.
>
> And for the "non-average" user, who builds Emacs from sources?  AFAIU,
> native compilation of all Lisp files takes many hours even on fast
> machines -- won't this be an annoyance if we don't come up with a JIT
> mechanism?

I'm not sure but I guess than the 'JIT mechanism' is already in place
and called 'deferred-compilation'.  Isn't it?

  Andrea

-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 16 May 2020 17:03:13 +0000
> 
> > And for the "non-average" user, who builds Emacs from sources?  AFAIU,
> > native compilation of all Lisp files takes many hours even on fast
> > machines -- won't this be an annoyance if we don't come up with a JIT
> > mechanism?
> 
> I'm not sure but I guess than the 'JIT mechanism' is already in place
> and called 'deferred-compilation'.  Isn't it?

You mean, you implemented it as part of the branch?  Then I apologize
for not knowing about its existence.

Re: Native compilation: the bird-eye view

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 16 May 2020 17:03:13 +0000
>> 
>> > And for the "non-average" user, who builds Emacs from sources?  AFAIU,
>> > native compilation of all Lisp files takes many hours even on fast
>> > machines -- won't this be an annoyance if we don't come up with a JIT
>> > mechanism?
>> 
>> I'm not sure but I guess than the 'JIT mechanism' is already in place
>> and called 'deferred-compilation'.  Isn't it?
>
> You mean, you implemented it as part of the branch?

Yes

> Then I apologize
> for not knowing about its existence.

No issue, just to make sure we are on the same page.  At the time I
wrote some description for that here:

https://akrl.sdf.org/gccemacs.html#org3422e30

I discuss that also around 22.30 at the presentation I've recorded for
ELS:

https://akrl.sdf.org/gccemacs.html#orgf6587ae

Regards

  Andrea

-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 16 May 2020 17:44:28 +0000
> 
> >> I'm not sure but I guess than the 'JIT mechanism' is already in place
> >> and called 'deferred-compilation'.  Isn't it?
> >
> > You mean, you implemented it as part of the branch?
> 
> Yes

Then why not make this the default?

Re: Native compilation: the bird-eye view

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 16 May 2020 17:44:28 +0000
>> 
>> >> I'm not sure but I guess than the 'JIT mechanism' is already in place
>> >> and called 'deferred-compilation'.  Isn't it?
>> >
>> > You mean, you implemented it as part of the branch?
>> 
>> Yes
>
> Then why not make this the default?

I think it works well but:

1- I need inputs on decisions, and this is one of those :)

2- If Emacs is installed, often is not able to compile because the user
   has possibly no write privileges on the folder where the compiler is
   trying to output the eln.  Either we switch this feature off in this
   case or we go for a different directory model.

Thanks

  Andrea

-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 16 May 2020 18:26:53 +0000
> 
> > Then why not make this the default?
> 
> I think it works well but:
> 
> 1- I need inputs on decisions, and this is one of those :)
> 
> 2- If Emacs is installed, often is not able to compile because the user
>    has possibly no write privileges on the folder where the compiler is
>    trying to output the eln.  Either we switch this feature off in this
>    case or we go for a different directory model.

This goes back to the proposal to have a cache directory for
JIT-compiled Lisp files.  That cache directory can be under the user's
home directory.

Re: Native compilation: the bird-eye view

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 16 May 2020 18:26:53 +0000
>> 
>> > Then why not make this the default?
>> 
>> I think it works well but:
>> 
>> 1- I need inputs on decisions, and this is one of those :)
>> 
>> 2- If Emacs is installed, often is not able to compile because the user
>>    has possibly no write privileges on the folder where the compiler is
>>    trying to output the eln.  Either we switch this feature off in this
>>    case or we go for a different directory model.
>
> This goes back to the proposal to have a cache directory for
> JIT-compiled Lisp files.  That cache directory can be under the user's
> home directory.

Yes, that's the option.

Implementation wise could just translate into prefixing the output path
with this cache directory I think.

  Andrea

-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Yuan Fu]

> 
> 2- If Emacs is installed, often is not able to compile because the user
>   has possibly no write privileges on the folder where the compiler is
>   trying to output the eln.  Either we switch this feature off in this
>   case or we go for a different directory model.

It’s also possible that the user doesn’t the proper version of gcc & libgccjit.

Yuan

Re: Native compilation: the bird-eye view

[John Wiegley]

>>>>> "AC" == Andrea Corallo <akrl@sdf.org> writes:

>> Then why not make this the default?
AC> I think it works well but:

AC> 1- I need inputs on decisions, and this is one of those :)

I would suggest that we make it "opt-in" for at least one major release cycle,
to explore the issues among early adopters, before making it the default.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Native compilation: the bird-eye view

[Ihor Radchenko]

> It’s also possible that the user doesn’t the proper version of gcc & libgccjit.

Then it should be another dependency for Emacs package (or packaged
together with binary?).
Though libgccjit is not readily available in some systems. I am using
Gentoo and libgccjit is blocked by  default
(https://bugs.gentoo.org/569608).

Best,
Ihor

Yuan Fu <casouri@gmail.com> writes:

>> 
>> 2- If Emacs is installed, often is not able to compile because the user
>>   has possibly no write privileges on the folder where the compiler is
>>   trying to output the eln.  Either we switch this feature off in this
>>   case or we go for a different directory model.
>
> It’s also possible that the user doesn’t the proper version of gcc & libgccjit.
>
> Yuan

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: Native compilation: the bird-eye view

[Ihor Radchenko]

> And for the "non-average" user, who builds Emacs from sources?  AFAIU,
> native compilation of all Lisp files takes many hours even on fast
> machines -- won't this be an annoyance if we don't come up with a JIT
> mechanism?

I am using Gentoo and thus compiling pretty much everything from source.
According to my experience so far, compilation takes even more time than
compiling something like chromium. Moreover, I got memory overflow (for
8Gb RAM) when compiling ja-dic.elc. In order to compile that file, I had
to manually fallback to comp-speed 0. 

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Date: Sat, 16 May 2020 12:58:11 +0000
>> Cc: emacs-devel@gnu.org
>> 
>> > If so, they will have
>> > to be generated on the end-user machines, or perhaps by whoever
>> > prepares the Emacs binary distributions -- assuming that the binary
>> > distributions are sufficiently compatible to allow that,
>> > notwithstanding the differences between the system where the *.eln
>> > files were generated and the system where they will be installed and
>> > used.  (I don't know what is the granularity of the binary distros wrt
>> > machine architecture and OS versions -- will that allow to be sure the
>> > *.eln files are compatible with the target system?)
>> 
>> Given .eln are just shareds with (almost) no dependecies [1] their
>> degree of compatibility should be higher then the Emacs binary itself.
>
> Well, the "almost" part means we aren't actually sure about this.
> E.g., what about the dependencies related to the GCC version used to
> build libgccjit and compile the .eln files?
>
>> > The next question is whether we want the *.eln files to exist up front
>> > for all the Lisp files on the end-user system, or we want them to be
>> > generated in JIT-like manner, whenever the corresponding Lisp library
>> > is loaded on demand?  The answer to this question might then influence
>> > the place where the *.eln files are kept -- the JIT alternative would
>> > suggest to have some kind of cache directory where we put the compiled
>> > files, similar to what Guile does.
>> 
>> I suspect that for the average user the best is to have the distribution
>> do all the compilation upfront for him and have deferred compilation
>> handling only additional libraries and packages.
>
> And for the "non-average" user, who builds Emacs from sources?  AFAIU,
> native compilation of all Lisp files takes many hours even on fast
> machines -- won't this be an annoyance if we don't come up with a JIT
> mechanism?
>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: Native compilation: the bird-eye view

[Andrea Corallo]

Ihor Radchenko <yantar92@gmail.com> writes:

>> And for the "non-average" user, who builds Emacs from sources?  AFAIU,
>> native compilation of all Lisp files takes many hours even on fast
>> machines -- won't this be an annoyance if we don't come up with a JIT
>> mechanism?
>
> I am using Gentoo and thus compiling pretty much everything from source.
> According to my experience so far, compilation takes even more time than
> compiling something like chromium. Moreover, I got memory overflow (for
> 8Gb RAM) when compiling ja-dic.elc. In order to compile that file, I had
> to manually fallback to comp-speed 0. 

FWIW ja-dic.elc is not anymore compiled and the overall compile time
should be reduced by a factor two since you have tried it.  Also the
memory necessary should be now around 3GB.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41077#28

I hope we can optimize this number further.

Andrea

-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Arthur Miller]

Yuan Fu <casouri@gmail.com> writes:

>> 
>> 2- If Emacs is installed, often is not able to compile because the user
>>   has possibly no write privileges on the folder where the compiler is
>>   trying to output the eln.  Either we switch this feature off in this
>>   case or we go for a different directory model.
>
> It’s also possible that the user doesn’t the proper version of gcc & libgccjit.
>
> Yuan
Yeah that happened to me when I was compiling it once on update 7, after my
system updated GCC, but libgccjit is not updated automatically, I have
to fetch it and compile it manually (Arch Linux).

Re: Native compilation: the bird-eye view

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> This goes back to the proposal to have a cache directory for
>> JIT-compiled Lisp files.  That cache directory can be under the user's
>> home directory.
>
> Yes, that's the option.
>
> Implementation wise could just translate into prefixing the output path
> with this cache directory I think.
>
>   Andrea

I think we should reason on the point if moving the .elns into some
cache folder would prevent distributing precompiled eln binaries or not.
I think is mandatory to retain this capability.

  Andrea

-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[ASSI]

Andrea Corallo writes:
> I think we should reason on the point if moving the .elns into some
> cache folder would prevent distributing precompiled eln binaries or not.
> I think is mandatory to retain this capability.

You would ultimately need to support at least three locations for these:
as distributed, local system level and user level.  The user level at
least would have the additional complication that it needs to support
different Emacs versions the user might use simultaneously.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Native compilation: the bird-eye view

[Andrea Corallo]

Currently a .el file is compiled as:

/bar/foo.el => /bar/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/foo.eln

where the directory name eln-... disamiguates architecture triplet and
Emacs version/configuration.

One option would be in case to fall back in deposing the eln in something
like:

~/.emacs.d/eln-cache/bar/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/foo.eln

Both the directories would be added into the effective load path during
load if existing.

The logic to fallback from the first option to the second could be as
simple as "if I have no write access on /bar just fallback" to something
more complex.

This way we could distribute binaries but also have deferred compilation
working in all situations.

Bests

  Andrea

Re: Native compilation: the bird-eye view

[Stefan Kangas]

Andrea Corallo <akrl@sdf.org> writes:

> Currently a .el file is compiled as:
>
> /bar/foo.el => /bar/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/foo.eln

Would it make sense to include `emacs-version' in the name, as in:

~/.emacs.d/eln-cache/bar/eln-x86_64-pc-linux-gnu-27.1-d241bf45dde51f21/foo.eln

I think that could make it easier to see if the directory is still
current or not.  I imagine users would eventually want to delete these
directories for old versions of Emacs.

(Of course, one could also look at the creation date.)

Best regards,
Stefan Kangas

Re: Native compilation: the bird-eye view

[Andrea Corallo]

Stefan Kangas <stefankangas@gmail.com> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Currently a .el file is compiled as:
>>
>> /bar/foo.el => /bar/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/foo.eln
>
> Would it make sense to include `emacs-version' in the name, as in:
>
> ~/.emacs.d/eln-cache/bar/eln-x86_64-pc-linux-gnu-27.1-d241bf45dde51f21/foo.eln

Given the only downside I see is the directory name length increase I
think is a good idea.

  Andrea
  
-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

> Currently a .el file is compiled as:
>
> /bar/foo.el => /bar/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/foo.eln
>
> where the directory name eln-... disamiguates architecture triplet and
> Emacs version/configuration.
>
> One option would be in case to fall back in deposing the eln in something
> like:
>
> ~/.emacs.d/eln-cache/bar/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/foo.eln
>
> Both the directories would be added into the effective load path during
> load if existing.
>
> The logic to fallback from the first option to the second could be as
> simple as "if I have no write access on /bar just fallback" to something
> more complex.
>
> This way we could distribute binaries but also have deferred compilation
> working in all situations.
>
> Bests
>
>   Andrea

Hi all,

I'd like to get some feedback on the above proposal.

Thanks

  Andrea

-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Stefan Monnier]

I see you haven't gotten much feedback about this yet, so I'll chime in.

> Currently a .el file is compiled as:
> /bar/foo.el => /bar/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/foo.eln
> where the directory name eln-... disamiguates architecture triplet and
> Emacs version/configuration.
>
> One option would be in case to fall back in deposing the eln in something
> like:
>
> ~/.emacs.d/eln-cache/bar/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/foo.eln
>
> Both the directories would be added into the effective load path during
> load if existing.

This is OK but doubles the length of the effective `load-path`.
Another option would be to use

    ~/.emacs.d/eln-cache/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/<HASH>.eln

where <HASH> is a hash of the corresponding .el(c) file name.  Similarly, instead of

    /bar/foo.el => /bar/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/foo.eln

we would store those compiled files to a "global eln-cache"

    /bar/foo.el => /usr/lib/eln-cache/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/<HASH>.eln

So we'd have a `eln-cache-path` where to search for those files and when
loading a file we'd first look for the .el file in `load-path` and once
found, we'd look inside `eln-cache-path` to see if that file has
a compiled version available.  `eln-cache-path` would be expected to be
short (2 entries in the typical case).


        Stefan

Re: Native compilation: the bird-eye view

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> This is OK but doubles the length of the effective `load-path`.
> Another option would be to use
>
>     ~/.emacs.d/eln-cache/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/<HASH>.eln
>
> where <HASH> is a hash of the corresponding .el(c) file name.  Similarly, instead of
>
>     /bar/foo.el => /bar/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/foo.eln
>
> we would store those compiled files to a "global eln-cache"
>
>     /bar/foo.el => /usr/lib/eln-cache/eln-x86_64-pc-linux-gnu-d241bf45dde51f21/<HASH>.eln
>
> So we'd have a `eln-cache-path` where to search for those files and when
> loading a file we'd first look for the .el file in `load-path` and once
> found, we'd look inside `eln-cache-path` to see if that file has
> a compiled version available.  `eln-cache-path` would be expected to be
> short (2 entries in the typical case).

This is a good idea.  My concerns are on how the user would interact
with this cache ex: searching if a given eln exists, what's its date,
removing one eln etc.

Should the user goes always through an API we provide?  Or do we want have
everything 100% transparent?

  Andrea

-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Stefan Monnier]

> This is a good idea.  My concerns are on how the user would interact
> with this cache ex: searching if a given eln exists, what's its date,
> removing one eln etc.
> Should the user goes always through an API we provide?  Or do we want have
> everything 100% transparent?

I think the idea would be to aim to make it "100% transparent" for
normal users.


        Stefan

Re: Native compilation: the bird-eye view

[Andrea Corallo via Emacs development discussions.]

Hi all,

so I did some experimentation on moving the eln into a cache directory.

ATM I've something that works this way:

Compiling Emacs elns are produced into a dedicate folder in the source
tree as emacs/eln-cache

Inside there is another directory level to disambiguate triplet and
Emacs configuration:

~/emacs$ ls eln-cache/
x86_64-pc-linux-gnu-f618cbdb0cd39c5f

Inside we have the eln files:

~/emacs$ ls eln-cache/x86_64-pc-linux-gnu-f618cbdb0cd39c5f/
4Corner-6622083dd2e93eda9a23ab8fb261bfd716e557cd3f955484db09a43143948f77.eln
5x5-cf035056577934b61cfe135fec6b6c67af96dd8378fa341ecd6a1b68ee789f48.eln
abbrev-e1c1055cee82bacc9771ef6694dd80578cd97e869a489b42ff5b74ca8e00cbb6.eln
abbrevlist-9cb12f8701f1f34beea0fbf333c35a3643bdec0691490b8fd70579ec26904723.eln
add-log-4a4d094c86ae3143226b9e64aab356b7ced423a8c9b72b2156f75117b4ae1a66.eln
...

I left the original name for comodity at the beginning and the hash
afterwards.  We could enconde in this hash also triplet and
configuration to remove one directory layer but I thought is more handy
for the user to have the eln divided this way.

Anyway, these elns are the one produced during the build, so the one
that will be installed in a sys directory.  As a consequence I've added
a second eln-cache directory for the compilations produced during normal
use placed like ~/.emacs.d/eln-cache (well the exact value is computed
using `user-emacs-directory').

This implies that during a load we check if the file exists first in the
eln user directory and then if necessary in the system one.

I'd like to get some feedback if you guys think this schema makes sense
or you see some issues.

Another question I have: do you think would be accettable at (first)
startup to create the eln user directory and populate it with sym links
for each eln file pointing to the eln in the system eln-cache directory?
This way we could save one file look-up and simplify the code given we
would point always and only to the user directory.

Thanks!

  Andrea

--
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Stefan Monnier]

> Inside we have the eln files:
>
> ~/emacs$ ls eln-cache/x86_64-pc-linux-gnu-f618cbdb0cd39c5f/
> 4Corner-6622083dd2e93eda9a23ab8fb261bfd716e557cd3f955484db09a43143948f77.eln
> 5x5-cf035056577934b61cfe135fec6b6c67af96dd8378fa341ecd6a1b68ee789f48.eln
> abbrev-e1c1055cee82bacc9771ef6694dd80578cd97e869a489b42ff5b74ca8e00cbb6.eln
> abbrevlist-9cb12f8701f1f34beea0fbf333c35a3643bdec0691490b8fd70579ec26904723.eln
> add-log-4a4d094c86ae3143226b9e64aab356b7ced423a8c9b72b2156f75117b4ae1a66.eln
> ...

Looks good.

> I left the original name for comodity at the beginning and the hash
> afterwards.  We could enconde in this hash also triplet and
> configuration to remove one directory layer but I thought is more handy
> for the user to have the eln divided this way.

Long file names are annoying, so removing that level of directory would
make it worse in my book.

> Anyway, these elns are the one produced during the build, so the one
> that will be installed in a sys directory.  As a consequence I've added
> a second eln-cache directory for the compilations produced during normal
> use placed like ~/.emacs.d/eln-cache (well the exact value is computed
> using `user-emacs-directory').

Good.

> This implies that during a load we check if the file exists first in the
> eln user directory and then if necessary in the system one.

Fine.  We might want to introduce an `eln-load-path` for that.

> Another question I have: do you think would be accettable at (first)
> startup to create the eln user directory and populate it with sym links
> for each eln file pointing to the eln in the system eln-cache directory?

I don't like this idea (and symlinks are problematic under w32).

> This way we could save one file look-up and simplify the code given we
> would point always and only to the user directory.

A loop around `eln-load-path` seems like a fairly small price to pay and
the extra lookup is likely to be negligible compared to all the lookups we
do to find the `.elc` file anyway.


        Stefan

Re: Native compilation: the bird-eye view

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> A loop around `eln-load-path` seems like a fairly small price to pay and
> the extra lookup is likely to be negligible compared to all the lookups we
> do to find the `.elc` file anyway.

Sounds good, I'll follow up.

Thanks
  Andrea
  
-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Andrea Corallo via Emacs development discussions.]

All right,

I've pushed what I'm working on to scratch/native-comp-eln-caches

This is removing .eln from the load suffixes and now the load is done
automatically only if a .elc is being loaded and the correpsonding .eln
is found.  (We may revisit this later if we want to remove the elc files
from the source tree)

Loading manually an eln is with this setup only possible using a lower
level primitive as `native-elisp-load'.

The search of the eln is done looping on `comp-eln-load-path' as
suggested.  This by default has two entries (user and system one).  The
system one is assumed to be always the last.

This branch enable by default deferred-compilation too.  At this stage
I'm wondering if we should rename it as jit-something.

make install is ATM not functional.  On this subject wanted to ask, in
which position do we want to install the eln-cache system directory?

Thanks

  Andrea

-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii <eliz@gnu.org>,
>         Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 15 Aug 2020 10:51:35 +0000
> 
> make install is ATM not functional.  On this subject wanted to ask, in
> which position do we want to install the eln-cache system directory?

Since the eln files are architecture-dependent, it should be somewhere
under exec-directory, I think.

Re: Native compilation: the bird-eye view

[Andrea Corallo via Emacs development discussions.]

Hi all,

debugging I discovered that on my system 'dlopen' can return the same
handle if two different shared with the same filename are loaded in two
different times (even if the first was deleted).  To prevent this
condition I added to the hashing algorithm that create the eln filename
the last modification time of the source being compiled.

This works but implies that once a source file is modified we have no
more possibility to remove the old eln in the eln-cache dir because we
loose its exact name.

I'm not sure this is a requirement as the eln-cache folders (except the
sys one) can even be deleted in every moment as the regenerate as
necessary, but in case is a requirement we'll have to come-up with a
different solution.

Thanks

  Andrea

--
akrl@sdf.org

Re:Re: Native compilation: the bird-eye view

[tumashu]

At 2020-08-19 23:01:22, "Andrea Corallo via \"Emacs development discussions.\"" <emacs-devel@gnu.org> wrote:
>Hi all,
>
>debugging I discovered that on my system 'dlopen' can return the same
>handle if two different shared with the same filename are loaded in two
>different times (even if the first was deleted).  To prevent this
>condition I added to the hashing algorithm that create the eln filename
>the last modification time of the source being compiled.
>
>This works but implies that once a source file is modified we have no
>more possibility to remove the old eln in the eln-cache dir because we
>loose its exact name.
>
>I'm not sure this is a requirement as the eln-cache folders (except the
>sys one) can even be deleted in every moment as the regenerate as
>necessary, but in case is a requirement we'll have to come-up with a
>different solution.
>
>Thanks
>
>  Andrea
>
>--
>akrl@sdf.org

Hello, after I git push and compile,  my emacs broken:

when I run emacs, it only show:

    feng@debian:~/emacs/emacs28-gtk/bin $ ./emacs-28.0.50 
    /home/feng/emacs/emacs28-gtk/share/emacs/28.0.50/lisp/language/china-util.el
    feng@debian:~/emacs/emacs28-gtk/bin $ 

Re: Native compilation: the bird-eye view

[Andrea Corallo via Emacs development discussions.]

tumashu  <tumashu@163.com> writes:

> Hello, after I git push and compile,  my emacs broken:
>
> when I run emacs, it only show:
>
>     feng@debian:~/emacs/emacs28-gtk/bin $ ./emacs-28.0.50 
>     /home/feng/emacs/emacs28-gtk/share/emacs/28.0.50/lisp/language/china-util.el
>     feng@debian:~/emacs/emacs28-gtk/bin $ 

Hi tumashu,

please open a bug in the Emacs bug tracker[1] describing the sequence of
commands that lead to this condition so we can reproduce it and work it
out.

Thanks

  Andrea

[1] https://www.gnu.org/software/emacs/manual/html_node/efaq/Reporting-bugs.html

-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Andrea Corallo via Emacs development discussions.]

Hi all,

I've updated how we generate the eln filenames and now they are in the
form: filename-path_hash-content_hash.eln

where path_hash is computed using the absoulte path of the source file
and content_hash using its content.

This should enable us for keepeng the eln-cache folders clean and solve
the issue mentioned in my previous message.

So far I've no idea if hashing the source file content has a measurable
impact on the startup time tho, I guess we'll see and in case search for
a different solution.

  Andrea

-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Stefan Monnier]

> I've updated how we generate the eln filenames and now they are in the
> form: filename-path_hash-content_hash.eln
>
> where path_hash is computed using the absoulte path of the source file
> and content_hash using its content.

[ Note: it's not a "path" but a "file name", according the GNU's
  conventions.  ]

Hmm... this results in too-long file names IMO, and the absolute file
name can change "gratuitously", making those eln files suddenly unusable.

> This should enable us for keepeng the eln-cache folders clean and solve
> the issue mentioned in my previous message.

I don't have a clear understanding of the problem this is trying to solve.
Could you summarize it?

If you're worried about how to delete old .eln files, I think there are
two problems: identifying which .elc files still exist and identifying
which Emacs binaries still exist.

But the first problem can arguably be solved by scanning the `load-path`,
collecting the .elc files we find there (and kind of GC algorithm
where `load-path` acts as the roots).


        Stefan

Re: Native compilation: the bird-eye view

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I've updated how we generate the eln filenames and now they are in the
>> form: filename-path_hash-content_hash.eln
>>
>> where path_hash is computed using the absoulte path of the source file
>> and content_hash using its content.
>
> [ Note: it's not a "path" but a "file name", according the GNU's
>   conventions.  ]
>
> Hmm... this results in too-long file names IMO, and the absolute file
> name can change "gratuitously", making those eln files suddenly unusable.

I shorted also the hash length for this reason, I think should be very
unlikely to get collisions anyway.  This is a file I've here as an example:

advice-505b05f39e8e7db0ef9610bf48b6298e-7c08dcc023e8926affda38e4071cba52.eln

>> This should enable us for keepeng the eln-cache folders clean and solve
>> the issue mentioned in my previous message.
>
> I don't have a clear understanding of the problem this is trying to solve.
> Could you summarize it?

Sure,

we need a rule to go from the source file to the eln filename.

/xxx/foo.el -> foo-something1.eln
/yyy/foo.el -> foo-something2.eln

Given we store all eln "flatted" in a single directory indeed we need to
have "something1" different from "something2".

As a first approach I went for using just the abs filename as input to
the hashing algo (with some tweaking for the eln being installed by make
install).

Unfortunatelly dlopen may return the same handle on two subsequent loads
if the filename is the same, even if the first file has been renamed or
removed.  This make recompiling and reloading a file uneffective.  Here
comes in the requirement of generating a different filename when
recompiling.

Facing that I came up with the idea of using the last modificaiton date
as additional input to the hash, modifying the source then generates a
different eln filename and everything works.  Unfortunatelly make
install is using tar to zip the source file and the last modificaiton
date is not preserved precisely.  Aside from that I perceived the last
modification date is a bit weak for this scope.

I then came-up with the current idea of using the source content as
input of the hash, this should be robust.

At the moment we are using 2 separate hashes to make the cleaning (easy)
possible, this is a separate problem that's correct.  The idea is that
with the two hashes when recompiling is easy to just remove the old file
as you can identify using the first hash the old one.  Additionally to
that, at any point in time, if you have two file sharing the first hash
you know that all but the most recent can be removed.

> If you're worried about how to delete old .eln files, I think there are
> two problems: identifying which .elc files still exist and identifying
> which Emacs binaries still exist.
>
> But the first problem can arguably be solved by scanning the `load-path`,
> collecting the .elc files we find there (and kind of GC algorithm
> where `load-path` acts as the roots).

I think this may be problematic as we could have two different Emacs
with two different load-path that are sharing the eln-cache folders.
This may be a bit of patological case tho.

If we want to solve the cleaning in some other way we could certanly
remove the hash computed from the filename.

Thanks

  Andrea

--
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Stefan Monnier]

>> I don't have a clear understanding of the problem this is trying to solve.
>> Could you summarize it?
>
> Sure,
>
> we need a rule to go from the source file to the eln filename.
>
> /xxx/foo.el -> foo-something1.eln
> /yyy/foo.el -> foo-something2.eln
>
> Given we store all eln "flatted" in a single directory indeed we need to
> have "something1" different from "something2".
[...]
> I then came-up with the current idea of using the source content as
> input of the hash, this should be robust.

Yes, the use of the content-hash should solve this problem "once and
forall".  Maybe the performance cost will force us to reevaluate this
decision, but for now I'm happy with it, so consider it a "solved problem".

> At the moment we are using 2 separate hashes to make the cleaning (easy)
> possible, this is a separate problem that's correct.  The idea is that
> with the two hashes when recompiling is easy to just remove the old file
> as you can identify using the first hash the old one.

I must be missing something: in which way does the filename-hash help here?
Oh, you mean we could look for <same-name-hash>-*.eln and remove them?

What I was thinking of doing was: when we generate a new .elc file, just
before saving the file over the old .elc file, we compute the hash of the
old .elc file and erase any matching .eln file.

> Additionally to that, at any point in time, if you have two file
> sharing the first hash you know that all but the most recent can
> be removed.

Then we could move the content-hash from the .eln filename to within the
.eln file itself.

Note that for the "system-wide" .eln files it can be trickier because we
may want to generate them once and then use them with Emacs installed at
difference locations, so we'd then want to try and avoid using the
absolute file name (and instead use file names that are relative to the
root of the Emacs install or something).

> I think this may be problematic as we could have two different Emacsen
> with two different load-path that are sharing the eln-cache folders.
> This may be a bit of patological case tho.

No, it's definitely not pathological.  It's pretty common for packages to
modify the `load-path` upon first use, so your `load-path` can change
long after startup.

Not to mention situations where you have several different configs
(e.g. I always have 2 Emacs sessions, one dedicated to Gnus and
a "general" one and they share *some* of the config but not all, so
they don't have the same `load-path`).


        Stefan

Re: Native compilation: the bird-eye view

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> I don't have a clear understanding of the problem this is trying to solve.
>>> Could you summarize it?
>>
>> Sure,
>>
>> we need a rule to go from the source file to the eln filename.
>>
>> /xxx/foo.el -> foo-something1.eln
>> /yyy/foo.el -> foo-something2.eln
>>
>> Given we store all eln "flatted" in a single directory indeed we need to
>> have "something1" different from "something2".
> [...]
>> I then came-up with the current idea of using the source content as
>> input of the hash, this should be robust.
>
> Yes, the use of the content-hash should solve this problem "once and
> forall".  Maybe the performance cost will force us to reevaluate this
> decision, but for now I'm happy with it, so consider it a "solved problem".

Agree, I think as a last caveat we should add to this hash also
parameters like comp-speed to have them effective when recompiling.

>> At the moment we are using 2 separate hashes to make the cleaning (easy)
>> possible, this is a separate problem that's correct.  The idea is that
>> with the two hashes when recompiling is easy to just remove the old file
>> as you can identify using the first hash the old one.
>
> I must be missing something: in which way does the filename-hash help here?
> Oh, you mean we could look for <same-name-hash>-*.eln and remove them?

Yep

> What I was thinking of doing was: when we generate a new .elc file, just
> before saving the file over the old .elc file, we compute the hash of the
> old .elc file and erase any matching .eln file.

Remeber ATM the input to the ashing is the .el content and not the .elc,
this motivated by:

  - For the future we may not have to necessarily produce the .elc in
    certain condition (AoT compilation during the build).

  - The .elc may not capture all the input to the compilation that have
    an effect on the native code generation (I don't have a working
    example for this is just a suspect).

>> Additionally to that, at any point in time, if you have two file
>> sharing the first hash you know that all but the most recent can
>> be removed.
>
> Then we could move the content-hash from the .eln filename to within the
> .eln file itself.

Yes we can do that, the API would be just more complex than the ls
<same-name-hash>-*.eln one.

> Note that for the "system-wide" .eln files it can be trickier because we
> may want to generate them once and then use them with Emacs installed at
> difference locations, so we'd then want to try and avoid using the
> absolute file name (and instead use file names that are relative to the
> root of the Emacs install or something).

I think in this case we could generalize what we have in place now to
allow reusing the same eln for local build and installed.

We are currently replacing in the filename if matches
PATH_DUMPLOADSEARCH or PATH_LOADSEARCH with '//' before feeding the hash
algorithm.

>> I think this may be problematic as we could have two different Emacsen
>> with two different load-path that are sharing the eln-cache folders.
>> This may be a bit of patological case tho.
>
> No, it's definitely not pathological.  It's pretty common for packages to
> modify the `load-path` upon first use, so your `load-path` can change
> long after startup.
>
> Not to mention situations where you have several different configs
> (e.g. I always have 2 Emacs sessions, one dedicated to Gnus and
> a "general" one and they share *some* of the config but not all, so
> they don't have the same `load-path`).

Thanks

  Andrea

-- 
akrl@sdf.org

Re: Native compilation: the bird-eye view

[Stefan Monnier]

>> What I was thinking of doing was: when we generate a new .elc file, just
>> before saving the file over the old .elc file, we compute the hash of the
>> old .elc file and erase any matching .eln file.
>
> Remeber ATM the input to the ashing is the .el content and not the .elc,
> this motivated by:

Hmm...

>   - The .elc may not capture all the input to the compilation that have
>     an effect on the native code generation (I don't have a working
>     example for this is just a suspect).

The problem is the opposite: it's the .el file which doesn't
account for the complete information (e.g. you may have to recompile
a .el file even if it hasn't changed, because some of the macros it
uses have been modified).

The .elc files are less affected by such problems.

>> Note that for the "system-wide" .eln files it can be trickier because we
>> may want to generate them once and then use them with Emacs installed at
>> difference locations, so we'd then want to try and avoid using the
>> absolute file name (and instead use file names that are relative to the
>> root of the Emacs install or something).
>
> I think in this case we could generalize what we have in place now to
> allow reusing the same eln for local build and installed.
>
> We are currently replacing in the filename if matches
> PATH_DUMPLOADSEARCH or PATH_LOADSEARCH with '//' before feeding the hash
> algorithm.

Sounds good,


        Stefan

Re: Native compilation: the bird-eye view

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> What I was thinking of doing was: when we generate a new .elc file, just
>>> before saving the file over the old .elc file, we compute the hash of the
>>> old .elc file and erase any matching .eln file.
>>
>> Remeber ATM the input to the ashing is the .el content and not the .elc,
>> this motivated by:
>
> Hmm...
>
>>   - The .elc may not capture all the input to the compilation that have
>>     an effect on the native code generation (I don't have a working
>>     example for this is just a suspect).
>
> The problem is the opposite: it's the .el file which doesn't
> account for the complete information (e.g. you may have to recompile
> a .el file even if it hasn't changed, because some of the macros it
> uses have been modified).
>
> The .elc files are less affected by such problems.

Very good point.

Considering the two problems:

1 having a way to identify the eln filename from source or bytecode file

2 prevent dlopen to be called two times with the same filename (before
  the initial returned handler is closed).

I believe we could work around ourself problem 2 identifying the
specific condition that is: we are loading an eln filename that is
currently dlopend.  In this case we could temporary rename the eln just
for having it loaded correctly.

If we do this we solve 2 and reloading always works, 1 should be we
already solved.

How does it sounds?

  Andrea

-- 
akrl@sdf.org

Path for system-wide .eln files (was: Native compilation: the bird-eye view)

[Ulrich Mueller]

Heading over from https://github.com/gentoo/gentoo/pull/16962 to here,
as suggested by Andrea:

System-wide .eln files are currently installed in:
/usr/libexec/emacs/28.0.50/x86_64-pc-linux-gnu/eln-cache/

(I am using /usr here and in the following for better readability,
replace by ${exec_prefix} as appropriate.)

I see two issues with this:

- According to [1] the directory contains ".eln files compiled during
  the build process." Therefore it is not a cache but a normal install
  directory, and "eln-cache" is a misleading name.
  (Or, from a wider perspective, a system-wide cache can never be in
  /usr but should be in /var/cache. And it wouldn't be populated at
  install time but at run time.)

- The FHS [2] says that /usr/libexec is intended for "binaries run
  by other programs". Do .eln files qualify as executable binaries?
  IIUC they are loaded by dlopen, so they're pretty much libraries or
  plugins which should go under /usr/lib{,64} instead.

[1] https://akrl.sdf.org/gccemacs.html#org01229a9
[2] https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s07.html

Re: Path for system-wide .eln files

[Andrea Corallo via Emacs development discussions.]

Ulrich Mueller <ulm@gentoo.org> writes:

> Heading over from https://github.com/gentoo/gentoo/pull/16962 to here,
> as suggested by Andrea:
>
> System-wide .eln files are currently installed in:
> /usr/libexec/emacs/28.0.50/x86_64-pc-linux-gnu/eln-cache/
>
> (I am using /usr here and in the following for better readability,
> replace by ${exec_prefix} as appropriate.)
>
> I see two issues with this:
>
> - According to [1] the directory contains ".eln files compiled during
>   the build process." Therefore it is not a cache but a normal install
>   directory, and "eln-cache" is a misleading name.
>   (Or, from a wider perspective, a system-wide cache can never be in
>   /usr but should be in /var/cache. And it wouldn't be populated at
>   install time but at run time.)
>
> - The FHS [2] says that /usr/libexec is intended for "binaries run
>   by other programs". Do .eln files qualify as executable binaries?
>   IIUC they are loaded by dlopen, so they're pretty much libraries or
>   plugins which should go under /usr/lib{,64} instead.
>
> [1] https://akrl.sdf.org/gccemacs.html#org01229a9
> [2] https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s07.html

Hi Ulrich,

thanks for the interesting input.

As for point 1 if this folder does not classify as cache because read
only, should be easy to go for something more appropriate.  Something as
simple as "sys-eln" would work? (I'm not good with names sorry)

For point 2: eln are opened with dlopen so they are technically shared
libraries.  They need to be loaded with some special treatment and
cannot be used as simple shareds by C programs tho.

To me this fits decently in the definition for /usr/libexec from your
link:

"includes internal binaries that are not intended to be executed
directly by users or shell scripts. Applications may use a single
subdirectory under /usr/libexec."

But I understand they could fit in /usr/lib as well.  Honestly I'm fine
with one or the other :)

  Andrea

Re: Path for system-wide .eln files

[Stefan Monnier]

> As for point 1 if this folder does not classify as cache because read
> only, should be easy to go for something more appropriate.  Something as
> simple as "sys-eln" would work? (I'm not good with names sorry)

No need for `sys` in that name, AFAICT: the rest of the name makes it
clear that it belongs to the Emacs installation or to the system but not
to the user.

> For point 2: eln are opened with dlopen so they are technically shared
> libraries.  They need to be loaded with some special treatment and
> cannot be used as simple shareds by C programs tho.
>
> To me this fits decently in the definition for /usr/libexec from your
> link:

AFAICT, `/usr/libexec` contains executables, not shared libraries.
They're just executables that are not supposed to be in $PATH.
So I think `/usr/lib` is more appropriate.


        Stefan

Re: Path for system-wide .eln files

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> As for point 1 if this folder does not classify as cache because read
>> only, should be easy to go for something more appropriate.  Something as
>> simple as "sys-eln" would work? (I'm not good with names sorry)
>
> No need for `sys` in that name, AFAICT: the rest of the name makes it
> clear that it belongs to the Emacs installation or to the system but not
> to the user.

Right, so just "eln"? "native-lisp"?  We gotta choose a name here :)

>> For point 2: eln are opened with dlopen so they are technically shared
>> libraries.  They need to be loaded with some special treatment and
>> cannot be used as simple shareds by C programs tho.
>>
>> To me this fits decently in the definition for /usr/libexec from your
>> link:
>
> AFAICT, `/usr/libexec` contains executables, not shared libraries.
> They're just executables that are not supposed to be in $PATH.
> So I think `/usr/lib` is more appropriate.

AFAIU `/usr/libexec` would be for binaries in general, infact we store
there the pdumper img.  Okay for `/usr/lib`.

Thanks

  Andrea

Re: Path for system-wide .eln files

[Ulrich Mueller]

>>>>> On Mon, 31 Aug 2020, Andrea Corallo wrote:

> AFAIU `/usr/libexec` would be for binaries in general, infact we store
> there the pdumper img.  Okay for `/usr/lib`.

It should be ${libdir} I think? That is, "lib64" on 64-bit systems.

Re: Path for system-wide .eln files

[Yuri Khan]

On Tue, 1 Sep 2020 at 04:27, Ulrich Mueller <ulm@gentoo.org> wrote:
>
> >>>>> On Mon, 31 Aug 2020, Andrea Corallo wrote:
>
> > AFAIU `/usr/libexec` would be for binaries in general, infact we store
> > there the pdumper img.  Okay for `/usr/lib`.
>
> It should be ${libdir} I think? That is, "lib64" on 64-bit systems.

I am on a 64-bit system and my /usr/lib64 is empty except for a single
ld-linux-x86-64.so.2 symlink. On the other hand, /usr/lib32 contains a
number of shared objects belonging to the libc6-i386 package. This
tells me maybe native bitness belongs in /usr/lib.

Re: Path for system-wide .eln files

[Ulrich Mueller]

>>>>> On Tue, 01 Sep 2020, Yuri Khan wrote:

>> > AFAIU `/usr/libexec` would be for binaries in general, infact we store
>> > there the pdumper img.  Okay for `/usr/lib`.
>> 
>> It should be ${libdir} I think? That is, "lib64" on 64-bit systems.

> I am on a 64-bit system and my /usr/lib64 is empty except for a single
> ld-linux-x86-64.so.2 symlink. On the other hand, /usr/lib32 contains a
> number of shared objects belonging to the libc6-i386 package. This
> tells me maybe native bitness belongs in /usr/lib.

I should have been more precise. It is in fact distro specific, so there
may be 64-bit systems where the native libs are installed in /usr/lib
(and 32-bit libs would go to /usr/lib32), while others install the
native libs in /usr/lib64.

So the right thing to do is to use configure's ${libdir}.

Re: Path for system-wide .eln files

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1765 bytes --]

On Tue, Sep 01, 2020 at 09:16:48AM +0200, Ulrich Mueller wrote:
> >>>>> On Tue, 01 Sep 2020, Yuri Khan wrote:
> 
> >> > AFAIU `/usr/libexec` would be for binaries in general, infact we store
> >> > there the pdumper img.  Okay for `/usr/lib`.
> >> 
> >> It should be ${libdir} I think? That is, "lib64" on 64-bit systems.
> 
> > I am on a 64-bit system and my /usr/lib64 is empty except for a single
> > ld-linux-x86-64.so.2 symlink. On the other hand, /usr/lib32 contains a
> > number of shared objects belonging to the libc6-i386 package. This
> > tells me maybe native bitness belongs in /usr/lib.
> 
> I should have been more precise. It is in fact distro specific, so there
> may be 64-bit systems where the native libs are installed in /usr/lib
> (and 32-bit libs would go to /usr/lib32), while others install the
> native libs in /usr/lib64.

Yes. Those are the ad-hoc "solutions" you find "in the wild". And
then there is Debian multiarch [1], which is the "right" solution
(roughly: /lib and /usr/lib contain the "native" stuff, for
backward-compatibility; "foreign" stuff lives in
{/lib,/usr/lib}/<arch-triplet>, e.g. /usr/lib/i386-linux-gnu. [2]

After all, there's no reason not to want to have Arm binaries in
an x86_64 machine (there's qemu, after all).

> So the right thing to do is to use configure's ${libdir}.

Exactly. Looking at one's distro and generalizing is going to bite
someone. And then they'll be angry at us :)

So use the configury, and fix that should it be broken.

Cheers

[1] https://wiki.debian.org/Multiarch/TheCaseForMultiarch
[2] I said roughly. It's not (alas!) the well-known GNU triplet,
   but something similar, for Reasons [3]
[3] https://wiki.debian.org/Multiarch/Tuples

 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Path for system-wide .eln files

[Andrea Corallo via Emacs development discussions.]

Ulrich Mueller <ulm@gentoo.org> writes:

>>>>>> On Tue, 01 Sep 2020, Yuri Khan wrote:
>
>>> > AFAIU `/usr/libexec` would be for binaries in general, infact we store
>>> > there the pdumper img.  Okay for `/usr/lib`.
>>> 
>>> It should be ${libdir} I think? That is, "lib64" on 64-bit systems.
>
>> I am on a 64-bit system and my /usr/lib64 is empty except for a single
>> ld-linux-x86-64.so.2 symlink. On the other hand, /usr/lib32 contains a
>> number of shared objects belonging to the libc6-i386 package. This
>> tells me maybe native bitness belongs in /usr/lib.
>
> I should have been more precise. It is in fact distro specific, so there
> may be 64-bit systems where the native libs are installed in /usr/lib
> (and 32-bit libs would go to /usr/lib32), while others install the
> native libs in /usr/lib64.
>
> So the right thing to do is to use configure's ${libdir}.

What I've pushed now (67c5369156) should be doing what we want in this
respect.

On my system the eln are now installed in:

/usr/local/lib/emacs/native-lisp/28.0.50-x86_64-pc-linux-gnu-679caf986.../

ELN_DESTDIR is computed as "$(DESTDIR)${libdir}/emacs/" in the Makefile.

Please give it a try to see if we are happy with that.

Thanks

  Andrea

Re: Path for system-wide .eln files

[Ulrich Mueller]

>>>>> On Fri, 04 Sep 2020, Andrea Corallo wrote:

> What I've pushed now (67c5369156) should be doing what we want in this
> respect.

> On my system the eln are now installed in:

> /usr/local/lib/emacs/native-lisp/28.0.50-x86_64-pc-linux-gnu-679caf986.../

> ELN_DESTDIR is computed as "$(DESTDIR)${libdir}/emacs/" in the Makefile.

> Please give it a try to see if we are happy with that.

Why would the architecture be included in that path? That's not what
packages normally do (except for compilers and other programs of the
binary toolchain).

IMHO it should have been /usr{,/local}/lib/emacs/28.0.50/native-lisp/...,
i.e., a structure parallel to the one in
/usr{,/local}/share/emacs/28.0.50/lisp/.

Re: Path for system-wide .eln files

[Andrea Corallo via Emacs development discussions.]

Ulrich Mueller <ulm@gentoo.org> writes:

>>>>>> On Fri, 04 Sep 2020, Andrea Corallo wrote:
>
>> What I've pushed now (67c5369156) should be doing what we want in this
>> respect.
>
>> On my system the eln are now installed in:
>
>> /usr/local/lib/emacs/native-lisp/28.0.50-x86_64-pc-linux-gnu-679caf986.../
>
>> ELN_DESTDIR is computed as "$(DESTDIR)${libdir}/emacs/" in the Makefile.
>
>> Please give it a try to see if we are happy with that.
>
> Why would the architecture be included in that path? 

The reason is that this folder name is the same we use to disambiguate
the Emacs version for every folder in `comp-eln-load-path'.  I was asked
to add the version there so is more user friendly and I think is a good
suggestion.  So now became like:

~/.emacs.d/eln-cache/28.0.50-x86_64-pc-linux-gnu-679caf986.../

At this point the system cache was

/usr/local/lib/emacs/28.0.50/native-lisp/28.0.50-x86_64-pc-linux-gnu-679caf986.../

I thought this is reduntant and removed the version there.

> That's not what packages normally do (except for compilers and other
> programs of the binary toolchain).

Well doesn't sound that different :)

I'll admit I don't have any strong opinion/interest on this directory
topic, as long as we find a satisfying solution for everyone is no big
deal to re-add the version there if we like.

Regards

  Andrea

Re: Path for system-wide .eln files

[Ulrich Mueller]

>>>>> On Fri, 04 Sep 2020, Emacs development discussions wrote:

> Ulrich Mueller <ulm@gentoo.org> writes:
>> Why would the architecture be included in that path? 

> The reason is that this folder name is the same we use to disambiguate
> the Emacs version for every folder in `comp-eln-load-path'.  I was asked
> to add the version there so is more user friendly and I think is a good
> suggestion.  So now became like:

> ~/.emacs.d/eln-cache/28.0.50-x86_64-pc-linux-gnu-679caf986.../

> At this point the system cache was

> /usr/local/lib/emacs/28.0.50/native-lisp/28.0.50-x86_64-pc-linux-gnu-679caf986.../

> I thought this is reduntant and removed the version there.

The difference between the two directories is that the home directory
may be shared between different architectures, while /usr/lib (or
/usr/local/lib) is _not_ shared by its definition. Therefore the
architecture makes sense in ~/.emacs.d/ but is redundant in /usr/lib/.

>> That's not what packages normally do (except for compilers and other
>> programs of the binary toolchain).

> Well doesn't sound that different :)

> I'll admit I don't have any strong opinion/interest on this directory
> topic, as long as we find a satisfying solution for everyone is no big
> deal to re-add the version there if we like.

I'd very much prefer (also from a distro point of view) if the directory
trees in /usr/share and /usr/lib would have a similar structure. That
is, /usr/lib/emacs/<version>/..., and no arch triplet.

Re: Path for system-wide .eln files

[Stefan Monnier]

> The difference between the two directories is that the home directory
> may be shared between different architectures, while /usr/lib (or
> /usr/local/lib) is _not_ shared by its definition. Therefore the
> architecture makes sense in ~/.emacs.d/ but is redundant in /usr/lib/.

The extra info may be redundant but it does no harm.

The benefit is that /$DESTDIR/$libdir/emacs/native-lisp can be a normal
member of the "eln files search path", whereas otherwise it would have
to be treated specially.

> I'd very much prefer (also from a distro point of view) if the directory
> trees in /usr/share and /usr/lib would have a similar structure. That
> is, /usr/lib/emacs/<version>/..., and no arch triplet.

Other than aesthetic concerns, what is at stake?


        Stefan

Re: Path for system-wide .eln files

[Ulrich Mueller]

>>>>> On Fri, 04 Sep 2020, Stefan Monnier wrote:

>> The difference between the two directories is that the home directory
>> may be shared between different architectures, while /usr/lib (or
>> /usr/local/lib) is _not_ shared by its definition. Therefore the
>> architecture makes sense in ~/.emacs.d/ but is redundant in /usr/lib/.

> The extra info may be redundant but it does no harm.

> The benefit is that /$DESTDIR/$libdir/emacs/native-lisp can be a normal
> member of the "eln files search path", whereas otherwise it would have
> to be treated specially.

Would that really be a problem? The tradeoff is an ugly directory
structure, in order to save a few lines of code.

>> I'd very much prefer (also from a distro point of view) if the directory
>> trees in /usr/share and /usr/lib would have a similar structure. That
>> is, /usr/lib/emacs/<version>/..., and no arch triplet.

> Other than aesthetic concerns, what is at stake?

Is it completely excluded that additional types of files will be
installed in /usr/lib/emacs/ in future? I believe it would make sense
to establish a directory structure that can be generalised, instead of
something that looks like an ad-hoc solution for one specific use case.

Re: Path for system-wide .eln files

[Stefan Monnier]

>> The benefit is that /$DESTDIR/$libdir/emacs/native-lisp can be a normal
>> member of the "eln files search path", whereas otherwise it would have
>> to be treated specially.
>
> Would that really be a problem?

Beside the unneeded extra complexity, it would also make it either
impossible to change the order between the files in the
one-and-only-system-dir and the-other-dirs, or would require yet more
complexity to provide extra flexibility.

So, yes, I think it would really be a problem.

>> Other than aesthetic concerns, what is at stake?
> Is it completely excluded that additional types of files will be
> installed in /usr/lib/emacs/ in future?

No, but I can't think of any reason why that would be more likely to
collide with `native-files` than with `26.3`.
If you're still worried, we can go back to
$DESTDIR/$libdir/emacs/$VERSION/native-lisp where the $VERSION
is redundant.  I wouldn't be bothered by this redundancy.


        Stefan

Re: Path for system-wide .eln files

[Ulrich Mueller]

>>>>> On Sat, 05 Sep 2020, Stefan Monnier wrote:

> If you're still worried, we can go back to
> $DESTDIR/$libdir/emacs/$VERSION/native-lisp where the $VERSION
> is redundant.  I wouldn't be bothered by this redundancy.

Yes, including the version would be more consistent with the way
the other install directories (/usr/share/emacs/<version>/ and
/usr/libexec/emacs/<version>/) are organized. I also think it would
scale better if any other subdirectories were to be added in future.

Re: Path for system-wide .eln files

[Alfred M. Szmidt]

   > The difference between the two directories is that the home directory
   > may be shared between different architectures, while /usr/lib (or
   > /usr/local/lib) is _not_ shared by its definition. Therefore the
   > architecture makes sense in ~/.emacs.d/ but is redundant in /usr/lib/.

   The extra info may be redundant but it does no harm.

If it is redundant, it also means that it doesn't have to be there.

But since there is no need to keep such information in libdir, it is
by definition for one specific architecture, it is better to remove
it.

There is also precedent in the GNU coding standards, e.g. most
anything under datarootdir, so it makes more sense to make libdir here
/usr/lib/emacs/VERSION as is the case with other things.

Re: Path for system-wide .eln files

[Andrea Corallo via Emacs development discussions.]

Ulrich Mueller <ulm@gentoo.org> writes:

>>>>>> On Sat, 05 Sep 2020, Stefan Monnier wrote:
>
>> If you're still worried, we can go back to
>> $DESTDIR/$libdir/emacs/$VERSION/native-lisp where the $VERSION
>> is redundant.  I wouldn't be bothered by this redundancy.
>
> Yes, including the version would be more consistent with the way
> the other install directories (/usr/share/emacs/<version>/ and
> /usr/libexec/emacs/<version>/) are organized. I also think it would
> scale better if any other subdirectories were to be added in future.

All right, with eb87425988 I've added back $version so now we have:

$DESTDIR/$libdir/emacs/$version/native-lisp

I think should be fine now.

Thanks

  Andrea