Building a tarball with native-compilation support

Building a tarball with native-compilation support

[Eli Zaretskii]

I think we have an unsolved issue for when we will release an Emacs
tarball that supports native-compilation: how to compile the preloaded
*.el files natively at build time.  This has to be done on the machine
where the tarball is built, because the *.eln files are
architecture-dependent, and thus cannot be in the source tarball.

Currently, compiling the preloaded *.el into *.eln is a side effect of
byte-compilation, so a new .eln file is produced when the *.el file is
newer than the corresponding .elc file, which causes Make to invoke
the byte+native-compilation.  But in a release tarball, all *.elc
files are newer than the corresponding *.el files, so this Make rule
will not be triggered, and the *.eln files will not be produced.  So
Emacs will be dumped with byte-compiled *.elc files, which is not what
we want, I think.

So unless I'm missing something, we need some Make wizardry to trigger
native-compilation at build time, even if the corresponding *.elc
files don't need to be re-generated.

Comments?

Re: Building a tarball with native-compilation support

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

> I think we have an unsolved issue for when we will release an Emacs
> tarball that supports native-compilation: how to compile the preloaded
> *.el files natively at build time.  This has to be done on the machine
> where the tarball is built, because the *.eln files are
> architecture-dependent, and thus cannot be in the source tarball.
>
> Currently, compiling the preloaded *.el into *.eln is a side effect of
> byte-compilation, so a new .eln file is produced when the *.el file is
> newer than the corresponding .elc file, which causes Make to invoke
> the byte+native-compilation.  But in a release tarball, all *.elc
> files are newer than the corresponding *.el files, so this Make rule
> will not be triggered, and the *.eln files will not be produced.  So
> Emacs will be dumped with byte-compiled *.elc files, which is not what
> we want, I think.
>
> So unless I'm missing something, we need some Make wizardry to trigger
> native-compilation at build time, even if the corresponding *.elc
> files don't need to be re-generated.
>
> Comments?

Hi Eli,

IIUC in the release tarballs we precompile .el files but not the .c one
as indeed this is arch specific.

Now I'm asking my self:

- given that as you mentioned the .eln will have to be produced in the
  target machine
- given that producing .eln can produce at the same
  time also the corresponding .elc

Does it make sense to still precompile and distribute the .elc files to
the users?  The reason for that is to keep one single release tarball
that can serve native and non native builds or there are other reasons?

Thanks

  Andrea

Re: Building a tarball with native-compilation support

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, emacs-devel@gnu.org
> Date: Thu, 05 Aug 2021 21:55:30 +0000
> 
> IIUC in the release tarballs we precompile .el files but not the .c one
> as indeed this is arch specific.

That's correct.

> Now I'm asking my self:
> 
> - given that as you mentioned the .eln will have to be produced in the
>   target machine
> - given that producing .eln can produce at the same
>   time also the corresponding .elc
> 
> Does it make sense to still precompile and distribute the .elc files to
> the users?  The reason for that is to keep one single release tarball
> that can serve native and non native builds or there are other reasons?

Yes, there are valid reasons to have the *.elc files in the tarball:

  . They make the build faster, because the step of using only *.el
    for COMPILE_FIRST is avoided;
  . They are a must if the user wants Emacs without native-compilation
    support, and in that case make the build extremely fast;
  . They provide canonical *.elc files produce on a system where most
    of the *.el files can be meaningfully byte-compiled because the
    related functionality is supported.

I think if we come up with a way to perform only the
native-compilation of a .el file when the corresponding .elc already
exists, the build will also faster, because we will use the
byte-compiled comp.elc and comp-cstr.elc, not the *.el files.

So I think we should work in this direction, and not decide to work
around the issue by omitting *.elc from the tarball.

How about if we start by providing a separate Makefile target to
native-compile a single .el file when its .elc exists and is
up-to-date?

Re: Building a tarball with native-compilation support

[Stefan Monnier]

> Yes, there are valid reasons to have the *.elc files in the tarball:
>
>   . They make the build faster, because the step of using only *.el
>     for COMPILE_FIRST is avoided;
>   . They are a must if the user wants Emacs without native-compilation
>     support, and in that case make the build extremely fast;
>   . They provide canonical *.elc files produce on a system where most
>     of the *.el files can be meaningfully byte-compiled because the
>     related functionality is supported.

Interesting.  The second point is the one I remembered (it does make
a very substantial difference, indeed).  I'm not sure what the first
point means, really (it seems to be just a subset of the second point).

The third point is new to me: which .el files can't be meaningfully
bytecompiled on all systems?


        Stefan

Re: Building a tarball with native-compilation support

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Andrea Corallo <akrl@sdf.org>,  larsi@gnus.org,  emacs-devel@gnu.org
> Date: Fri, 06 Aug 2021 10:37:10 -0400
> 
> The third point is new to me: which .el files can't be meaningfully
> bytecompiled on all systems?

The ones that require primitives that are not compiled into the Emacs
binary.

Re: Building a tarball with native-compilation support

[Stefan Monnier]

Eli Zaretskii [2021-08-06 20:41:11] wrote:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Andrea Corallo <akrl@sdf.org>,  larsi@gnus.org,  emacs-devel@gnu.org
>> Date: Fri, 06 Aug 2021 10:37:10 -0400
>> 
>> The third point is new to me: which .el files can't be meaningfully
>> bytecompiled on all systems?
>
> The ones that require primitives that are not compiled into the Emacs
> binary.

That much I can guess, but I don't know of such file.
Do you have concrete examples?


        Stefan

Re: Building a tarball with native-compilation support

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: akrl@sdf.org,  larsi@gnus.org,  emacs-devel@gnu.org
> Date: Sat, 07 Aug 2021 09:28:44 -0400
> 
> Eli Zaretskii [2021-08-06 20:41:11] wrote:
> >> From: Stefan Monnier <monnier@iro.umontreal.ca>
> >> Cc: Andrea Corallo <akrl@sdf.org>,  larsi@gnus.org,  emacs-devel@gnu.org
> >> Date: Fri, 06 Aug 2021 10:37:10 -0400
> >> 
> >> The third point is new to me: which .el files can't be meaningfully
> >> bytecompiled on all systems?
> >
> > The ones that require primitives that are not compiled into the Emacs
> > binary.
> 
> That much I can guess, but I don't know of such file.
> Do you have concrete examples?

I'm not sure I understand the purpose of these questions.  I'm sure
you know the answers, so what am I missing?

Re: Building a tarball with native-compilation support

[Eli Zaretskii]

> Date: Fri, 06 Aug 2021 09:05:02 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> How about if we start by providing a separate Makefile target to
> native-compile a single .el file when its .elc exists and is
> up-to-date?

There are some complications with doing this.  For a Makefile target
we need to know the file name of the .eln file to look for.  That file
name includes 2 parts that only Emacs knows: the ABI hash (which
determines the subdirectory of native-lisp/), and the 2 hashes
included in the basename of the .eln file itself.  For example, the
.eln file corresponding to window.el on my machine is this:

   native-lisp/28.0.50-2e9d394d/window-0d1b8b93-dcb2375f.eln

This should be the target name of the Makefile rule.  We can ask Emacs
to provide the hashes (the value of comp-abi-hash and what
comp-el-to-eln-rel-filename returns for "window.el"), but the problem
is that it must be the up-to-date rebuilt Emacs, otherwise the values
could be incorrect, or the Emacs binary could not yet exist at all.
This complicates things because Make expands and calculates some of
the stuff in the Makefile when it reads the Makefile, before it starts
executing the rules, and we cannot guarantee we can know those hashes
at that time.

So I think a better way would be to write a function in Emacs Lisp
that is passed a list of preloaded Lisp files, and natively-compiles
those of them whose *.eln files are either outdated or do not exist.
Then we could have a single Makefile target which invokes this
function once Emacs is fully rebuilt.

Comments?

Re: Building a tarball with native-compilation support

[Eli Zaretskii]

> Date: Sat, 07 Aug 2021 09:23:42 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> So I think a better way would be to write a function in Emacs Lisp
> that is passed a list of preloaded Lisp files, and natively-compiles
> those of them whose *.eln files are either outdated or do not exist.

Actually, such a function already exists: native-compile-async.  We
just need to write a suitable function to act as the SELECTOR for it.

Re: Building a tarball with native-compilation support

[Eli Zaretskii]

> Date: Sat, 07 Aug 2021 09:48:13 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> > Date: Sat, 07 Aug 2021 09:23:42 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: larsi@gnus.org, emacs-devel@gnu.org
> > 
> > So I think a better way would be to write a function in Emacs Lisp
> > that is passed a list of preloaded Lisp files, and natively-compiles
> > those of them whose *.eln files are either outdated or do not exist.
> 
> Actually, such a function already exists: native-compile-async.  We
> just need to write a suitable function to act as the SELECTOR for it.

Hmm... I seem to be unable to force native-compile-async to produce
the .eln file under native-lisp/preloaded/.  Andrea, how can I do
that?  I tried to set native-compile-target-directory, but it didn't
seem to help (??), the resulting .eln file is always under
~/.emacs.d/eln-cache/ for some reason.

What am I missing?

Re: Building a tarball with native-compilation support

[Eli Zaretskii]

> Date: Sat, 07 Aug 2021 13:32:07 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> > Date: Sat, 07 Aug 2021 09:48:13 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: larsi@gnus.org, emacs-devel@gnu.org
> > 
> > > Date: Sat, 07 Aug 2021 09:23:42 +0300
> > > From: Eli Zaretskii <eliz@gnu.org>
> > > Cc: larsi@gnus.org, emacs-devel@gnu.org
> > > 
> > > So I think a better way would be to write a function in Emacs Lisp
> > > that is passed a list of preloaded Lisp files, and natively-compiles
> > > those of them whose *.eln files are either outdated or do not exist.
> > 
> > Actually, such a function already exists: native-compile-async.  We
> > just need to write a suitable function to act as the SELECTOR for it.
> 
> Hmm... I seem to be unable to force native-compile-async to produce
> the .eln file under native-lisp/preloaded/.  Andrea, how can I do
> that?  I tried to set native-compile-target-directory, but it didn't
> seem to help (??), the resulting .eln file is always under
> ~/.emacs.d/eln-cache/ for some reason.
> 
> What am I missing?

Andrea, can you help me out here, please?  How do I tell
native-compile-async to produce the .eln file in native-lisp/preloaded/?

TIA

Re: Building a tarball with native-compilation support

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 07 Aug 2021 13:32:07 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> 
>> > Date: Sat, 07 Aug 2021 09:48:13 +0300
>> > From: Eli Zaretskii <eliz@gnu.org>
>> > Cc: larsi@gnus.org, emacs-devel@gnu.org
>> > 
>> > > Date: Sat, 07 Aug 2021 09:23:42 +0300
>> > > From: Eli Zaretskii <eliz@gnu.org>
>> > > Cc: larsi@gnus.org, emacs-devel@gnu.org
>> > > 
>> > > So I think a better way would be to write a function in Emacs Lisp
>> > > that is passed a list of preloaded Lisp files, and natively-compiles
>> > > those of them whose *.eln files are either outdated or do not exist.
>> > 
>> > Actually, such a function already exists: native-compile-async.  We
>> > just need to write a suitable function to act as the SELECTOR for it.
>> 
>> Hmm... I seem to be unable to force native-compile-async to produce
>> the .eln file under native-lisp/preloaded/.  Andrea, how can I do
>> that?  I tried to set native-compile-target-directory, but it didn't
>> seem to help (??), the resulting .eln file is always under
>> ~/.emacs.d/eln-cache/ for some reason.
>> 
>> What am I missing?
>
> Andrea, can you help me out here, please?  How do I tell
> native-compile-async to produce the .eln file in native-lisp/preloaded/?
>
> TIA

Sorry I'm just back online after one week.  I'll have a look this
evening and come back here.

  Andrea

Re: Building a tarball with native-compilation support

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Tue, 17 Aug 2021 10:16:44 +0000
> 
> > Andrea, can you help me out here, please?  How do I tell
> > native-compile-async to produce the .eln file in native-lisp/preloaded/?
> >
> > TIA
> 
> Sorry I'm just back online after one week.  I'll have a look this
> evening and come back here.

Great, thanks in advance.

Re: Building a tarball with native-compilation support

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> Date: Tue, 17 Aug 2021 10:16:44 +0000
>> 
>> > Andrea, can you help me out here, please?  How do I tell
>> > native-compile-async to produce the .eln file in native-lisp/preloaded/?
>> >
>> > TIA
>> 
>> Sorry I'm just back online after one week.  I'll have a look this
>> evening and come back here.
>
> Great, thanks in advance.

Hi Eli,

7fa5dec86e works for me, please have a look if it satisfies your usage
as well.

Thanks

  Andrea

Re: Building a tarball with native-compilation support

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Tue, 17 Aug 2021 21:49:37 +0000
> 
> >> > Andrea, can you help me out here, please?  How do I tell
> >> > native-compile-async to produce the .eln file in native-lisp/preloaded/?
> >> >
> >> > TIA
> >> 
> >> Sorry I'm just back online after one week.  I'll have a look this
> >> evening and come back here.
> >
> > Great, thanks in advance.
> 
> Hi Eli,
> 
> 7fa5dec86e works for me, please have a look if it satisfies your usage
> as well.

Oh, it was a bug?  I thought it was some kind of cockpit error on my
part.

Thanks, will look into this soon.

Re: Building a tarball with native-compilation support

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> Date: Tue, 17 Aug 2021 21:49:37 +0000
>> 
>> >> > Andrea, can you help me out here, please?  How do I tell
>> >> > native-compile-async to produce the .eln file in native-lisp/preloaded/?
>> >> >
>> >> > TIA
>> >> 
>> >> Sorry I'm just back online after one week.  I'll have a look this
>> >> evening and come back here.
>> >
>> > Great, thanks in advance.
>> 
>> Hi Eli,
>> 
>> 7fa5dec86e works for me, please have a look if it satisfies your usage
>> as well.
>
> Oh, it was a bug?  I thought it was some kind of cockpit error on my
> part.

If it's a bug or not I think depends on the definition we give to
`native-compile-target-directory'.  IMO it is nice to have it functional
also on async function, so yeah I think was a bug.

> Thanks, will look into this soon.

Thanks

  Andrea

Re: Building a tarball with native-compilation support

[Eli Zaretskii]

> Date: Wed, 18 Aug 2021 16:17:12 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> 
> > 7fa5dec86e works for me, please have a look if it satisfies your usage
> > as well.
> 
> Oh, it was a bug?  I thought it was some kind of cockpit error on my
> part.
> 
> Thanks, will look into this soon.

Finally got some time to work on this.

It works fine now, thanks.  But I think we should also pass the value
of comp-file-preloaded-p to the async compilation subprocesses,
because otherwise one needs to set LISP_PRELOADED in the environment,
which is somewhat less clean, IMO.  This is not important for what I
want to do, because src/Makefile takes care to set up LISP_PRELOADED.
But I think it is good to have that for other use cases.

Re: Building a tarball with native-compilation support

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 18 Aug 2021 16:17:12 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> 
>> > 7fa5dec86e works for me, please have a look if it satisfies your usage
>> > as well.
>> 
>> Oh, it was a bug?  I thought it was some kind of cockpit error on my
>> part.
>> 
>> Thanks, will look into this soon.
>
> Finally got some time to work on this.
>
> It works fine now, thanks.

Wonderful

> But I think we should also pass the value
> of comp-file-preloaded-p to the async compilation subprocesses,
> because otherwise one needs to set LISP_PRELOADED in the environment,
> which is somewhat less clean, IMO.  This is not important for what I
> want to do, because src/Makefile takes care to set up LISP_PRELOADED.
> But I think it is good to have that for other use cases.

Right good point, f7da671493 should do it.

Thanks

  Andrea