Sharing native-lisp system load-path between builds

Sharing native-lisp system load-path between builds

[Björn Bidar]

Hey,

I'm building Emacs master for OpenSUSE to provide testing.
OpenSUSE packaging packages Emacs so that they have three builds with
different feature sets: X11 without GTK, GTK and one build just without
X11 or GTK.

This helps different users to chose their preferred builds and keeps
build workers small for Emacs packages build.

The packages share the elisp, native-lisp ahead of time compilations and
docs.

Recently the native-lisp load path changed to change per Emacs
executable. I don't know the exact commit when this changed but it was
between ref 3c8167ec0f9647e6fc33e65b0a0324f96cb795ee and
d46a2fa319808963bbe8d3a90e7dbb13fcd844f5.

Now my questions: Is this a bug or can a different hash be used to
determine the system load-path? It is very useful to precompile the
lisp code once per emacs builds that share the same code.

I tried to bisect when exactly this changed but didn't found the exact
commit.

My package can be found here:
https://build.opensuse.org/package/show/home:Thaodan:emacs/emacs

Thanks, 

Björn Bidar

Re: Sharing native-lisp system load-path between builds

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Date: Mon, 26 Jun 2023 03:14:17 +0300
> 
> I'm building Emacs master for OpenSUSE to provide testing.
> OpenSUSE packaging packages Emacs so that they have three builds with
> different feature sets: X11 without GTK, GTK and one build just without
> X11 or GTK.
> 
> This helps different users to chose their preferred builds and keeps
> build workers small for Emacs packages build.
> 
> The packages share the elisp, native-lisp ahead of time compilations and
> docs.
> 
> Recently the native-lisp load path changed to change per Emacs
> executable.

It isn't a recent change, at least not in principle.

> I don't know the exact commit when this changed but it was
> between ref 3c8167ec0f9647e6fc33e65b0a0324f96cb795ee and
> d46a2fa319808963bbe8d3a90e7dbb13fcd844f5.
> 
> Now my questions: Is this a bug or can a different hash be used to
> determine the system load-path? It is very useful to precompile the
> lisp code once per emacs builds that share the same code.

It's impossible to share native-compiled *.eln files between different
builds, because they depend on the primitives compiled into the build
and on the architecture of the host CPU.

So you must have a separate directory for each build configuration,
and any significant change in the Emacs C code will require
recompilation of all the *.eln files.

Re: Sharing native-lisp system load-path between builds

[Björn Bidar]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Date: Mon, 26 Jun 2023 03:14:17 +0300
>> 
>> I'm building Emacs master for OpenSUSE to provide testing.
>> OpenSUSE packaging packages Emacs so that they have three builds with
>> different feature sets: X11 without GTK, GTK and one build just without
>> X11 or GTK.
>> 
>> This helps different users to chose their preferred builds and keeps
>> build workers small for Emacs packages build.
>> 
>> The packages share the elisp, native-lisp ahead of time compilations and
>> docs.
>> 
>> Recently the native-lisp load path changed to change per Emacs
>> executable.
>
> It isn't a recent change, at least not in principle.

What could have change this?

>> I don't know the exact commit when this changed but it was
>> between ref 3c8167ec0f9647e6fc33e65b0a0324f96cb795ee and
>> d46a2fa319808963bbe8d3a90e7dbb13fcd844f5.
>> 
>> Now my questions: Is this a bug or can a different hash be used to
>> determine the system load-path? It is very useful to precompile the
>> lisp code once per emacs builds that share the same code.
>
> It's impossible to share native-compiled *.eln files between different
> builds, because they depend on the primitives compiled into the build
> and on the architecture of the host CPU.

> So you must have a separate directory for each build configuration,
> and any significant change in the Emacs C code will require
> recompilation of all the *.eln files.

Each build configuration is 100% the same except the optional
dependencies they are build with.

There's no change to any source they are build one after the other.

See the spec file here:
https://build.opensuse.org/package/view_file/home:Thaodan:emacs/emacs/emacs.spec?expand=1

Re: Sharing native-lisp system load-path between builds

[Andrea Corallo]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>>> Date: Mon, 26 Jun 2023 03:14:17 +0300
>>> 
>>> I'm building Emacs master for OpenSUSE to provide testing.
>>> OpenSUSE packaging packages Emacs so that they have three builds with
>>> different feature sets: X11 without GTK, GTK and one build just without
>>> X11 or GTK.
>>> 
>>> This helps different users to chose their preferred builds and keeps
>>> build workers small for Emacs packages build.
>>> 
>>> The packages share the elisp, native-lisp ahead of time compilations and
>>> docs.
>>> 
>>> Recently the native-lisp load path changed to change per Emacs
>>> executable.
>>
>> It isn't a recent change, at least not in principle.
>
> What could have change this?
>
>>> I don't know the exact commit when this changed but it was
>>> between ref 3c8167ec0f9647e6fc33e65b0a0324f96cb795ee and
>>> d46a2fa319808963bbe8d3a90e7dbb13fcd844f5.
>>> 
>>> Now my questions: Is this a bug or can a different hash be used to
>>> determine the system load-path? It is very useful to precompile the
>>> lisp code once per emacs builds that share the same code.
>>
>> It's impossible to share native-compiled *.eln files between different
>> builds, because they depend on the primitives compiled into the build
>> and on the architecture of the host CPU.
>
>> So you must have a separate directory for each build configuration,
>> and any significant change in the Emacs C code will require
>> recompilation of all the *.eln files.
>
> Each build configuration is 100% the same except the optional
> dependencies they are build with.
>
> There's no change to any source they are build one after the other.
>
> See the spec file here:
> https://build.opensuse.org/package/view_file/home:Thaodan:emacs/emacs/emacs.spec?expand=1

The hash depends on the set of C primitives present in Emacs.  If the
hash differs it means your configurations are producing Emacsen with
different sets of primitives, and this makes eln files incompatible.

Best Regards

  Andrea

Re: Sharing native-lisp system load-path between builds

[Björn Bidar]

Andrea Corallo <acorallo@gnu.org> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>>>> Date: Mon, 26 Jun 2023 03:14:17 +0300
>>>> 
>>>> I'm building Emacs master for OpenSUSE to provide testing.
>>>> OpenSUSE packaging packages Emacs so that they have three builds with
>>>> different feature sets: X11 without GTK, GTK and one build just without
>>>> X11 or GTK.
>>>> 
>>>> This helps different users to chose their preferred builds and keeps
>>>> build workers small for Emacs packages build.
>>>> 
>>>> The packages share the elisp, native-lisp ahead of time compilations and
>>>> docs.
>>>> 
>>>> Recently the native-lisp load path changed to change per Emacs
>>>> executable.
>>>
>>> It isn't a recent change, at least not in principle.
>>
>> What could have change this?
>>
>>>> I don't know the exact commit when this changed but it was
>>>> between ref 3c8167ec0f9647e6fc33e65b0a0324f96cb795ee and
>>>> d46a2fa319808963bbe8d3a90e7dbb13fcd844f5.
>>>> 
>>>> Now my questions: Is this a bug or can a different hash be used to
>>>> determine the system load-path? It is very useful to precompile the
>>>> lisp code once per emacs builds that share the same code.
>>>
>>> It's impossible to share native-compiled *.eln files between different
>>> builds, because they depend on the primitives compiled into the build
>>> and on the architecture of the host CPU.
>>
>>> So you must have a separate directory for each build configuration,
>>> and any significant change in the Emacs C code will require
>>> recompilation of all the *.eln files.
>>
>> Each build configuration is 100% the same except the optional
>> dependencies they are build with.
>>
>> There's no change to any source they are build one after the other.
>>
>> See the spec file here:
>> https://build.opensuse.org/package/view_file/home:Thaodan:emacs/emacs/emacs.spec?expand=1
>
> The hash depends on the set of C primitives present in Emacs.  If the
> hash differs it means your configurations are producing Emacsen with
> different sets of primitives, and this makes eln files incompatible.

OK but so why did work with the commit
3c8167ec0f9647e6fc33e65b0a0324f96cb795ee and before?

Nothing has changed in the build environment except the commits between
good commit and d46a2fa319808963bbe8d3a90e7dbb13fcd844f5 or later.

Re: Sharing native-lisp system load-path between builds

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Mon, 14 Aug 2023 02:47:23 +0300
> 
> Andrea Corallo <acorallo@gnu.org> writes:
> 
> > The hash depends on the set of C primitives present in Emacs.  If the
> > hash differs it means your configurations are producing Emacsen with
> > different sets of primitives, and this makes eln files incompatible.
> 
> OK but so why did work with the commit
> 3c8167ec0f9647e6fc33e65b0a0324f96cb795ee and before?
> 
> Nothing has changed in the build environment except the commits between
> good commit and d46a2fa319808963bbe8d3a90e7dbb13fcd844f5 or later.

What exactly did work back then?  Please describe in detail what you
did that worked before 3c8167ec0f9647e6fc33e65b0a0324f96cb795ee and
doesn't work now.

Re: Sharing native-lisp system load-path between builds

[Björn Bidar]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Mon, 14 Aug 2023 02:47:23 +0300
>> 
>> Andrea Corallo <acorallo@gnu.org> writes:
>> 
>> > The hash depends on the set of C primitives present in Emacs.  If the
>> > hash differs it means your configurations are producing Emacsen with
>> > different sets of primitives, and this makes eln files incompatible.
>> 
>> OK but so why did work with the commit
>> 3c8167ec0f9647e6fc33e65b0a0324f96cb795ee and before?
>> 
>> Nothing has changed in the build environment except the commits between
>> good commit and d46a2fa319808963bbe8d3a90e7dbb13fcd844f5 or later.
>
> What exactly did work back then?  Please describe in detail what you
> did that worked before 3c8167ec0f9647e6fc33e65b0a0324f96cb795ee and
> doesn't work now.

Building Emacs once without X11, then X11 Without GTK and PGTK all with
native-compilation=aot.

When packaging Emacs using the precompiled eln from the last build Emacs
for all the build variants.

The spec file can be found here:
https://build.opensuse.org/package/view_file/home:Thaodan:emacs/emacs/emacs.spec?expand=1

The build process starts at line 319.

Re: Sharing native-lisp system load-path between builds

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: acorallo@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 14 Aug 2023 13:04:06 +0300
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Björn Bidar <bjorn.bidar@thaodan.de>
> >> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> >> Date: Mon, 14 Aug 2023 02:47:23 +0300
> >> 
> >> Andrea Corallo <acorallo@gnu.org> writes:
> >> 
> >> > The hash depends on the set of C primitives present in Emacs.  If the
> >> > hash differs it means your configurations are producing Emacsen with
> >> > different sets of primitives, and this makes eln files incompatible.
> >> 
> >> OK but so why did work with the commit
> >> 3c8167ec0f9647e6fc33e65b0a0324f96cb795ee and before?
> >> 
> >> Nothing has changed in the build environment except the commits between
> >> good commit and d46a2fa319808963bbe8d3a90e7dbb13fcd844f5 or later.
> >
> > What exactly did work back then?  Please describe in detail what you
> > did that worked before 3c8167ec0f9647e6fc33e65b0a0324f96cb795ee and
> > doesn't work now.
> 
> Building Emacs once without X11, then X11 Without GTK and PGTK all with
> native-compilation=aot.
> 
> When packaging Emacs using the precompiled eln from the last build Emacs
> for all the build variants.
> 
> The spec file can be found here:
> https://build.opensuse.org/package/view_file/home:Thaodan:emacs/emacs/emacs.spec?expand=1
> 
> The build process starts at line 319.

Thanks.  So what does "work" and "does not work" mean in this context?

Re: Sharing native-lisp system load-path between builds

[Eli Zaretskii]

[Why private email?]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Date: Mon, 14 Aug 2023 21:29:34 +0300
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks.  So what does "work" and "does not work" mean in this context?
> 
> Work means the eln files could be shared, doesn't mean the files can't
> be shared the hash in the native lisp path is different and the files
> are no longer compatible.

How did you share them? did you manually copy them into the same
directory or forced Emacs to write them there?  Or did Emacs create
*.eln files with the same hashes and in the same 29.0.50-NNNNN
subdirectory?  Also, which *.el files were compiled into *.eln files
you could share -- were those the preloaded *.el files, non-preloaded
*.el files from the Emacs tree, or third-party *.el files from
packages that are not bundled with Emacs?

In general, such different configurations could only be able to share
*.eln files by sheer luck.  It is enough to have one more or one less
primitive in one of the builds to require a separate set of *.eln
files, because the hash of the versioned subdirectory of native-lisp/
and of eln-cache changes when primitives are added or deleted.  That
has been so for a very long time, definitely long before commit
3c8167ec0f964.

Re: Sharing native-lisp system load-path between builds

[Björn Bidar]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Date: Mon, 14 Aug 2023 21:29:34 +0300
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Thanks.  So what does "work" and "does not work" mean in this context?
>> 
>> Work means the eln files could be shared, doesn't mean the files can't
>> be shared the hash in the native lisp path is different and the files
>> are no longer compatible.
>
> How did you share them? did you manually copy them into the same
> directory or forced Emacs to write them there?  Or did Emacs create
> *.eln files with the same hashes and in the same 29.0.50-NNNNN
> subdirectory?
Yes exactly. The <version>-<hash> directory was the same.

> Also, which *.el files were compiled into *.eln files
> you could share -- were those the preloaded *.el files, non-preloaded
> *.el files from the Emacs tree, or third-party *.el files from
> packages that are not bundled with Emacs?
Only the builtin el files as can be seen in the spec file.

>In general, such different configurations could only be able to share
>*.eln files by sheer luck.  It is enough to have one more or one less
>primitive in one of the builds to require a separate set of *.eln
>files, because the hash of the versioned subdirectory of native-lisp/
>and of eln-cache changes when primitives are added or deleted.  That
>has been so for a very long time, definitely long before commit
>3c8167ec0f964.

Does enabling or disabling lets say png change any of those primitives?
The package has different subpackages build from 100% the same source.
The configuration is only different when it comes to ui support.

Re: Sharing native-lisp system load-path between builds

[Andrea Corallo]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>>> Date: Mon, 14 Aug 2023 21:29:34 +0300
>>> 
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>> 
>>> > Thanks.  So what does "work" and "does not work" mean in this context?
>>> 
>>> Work means the eln files could be shared, doesn't mean the files can't
>>> be shared the hash in the native lisp path is different and the files
>>> are no longer compatible.
>>
>> How did you share them? did you manually copy them into the same
>> directory or forced Emacs to write them there?  Or did Emacs create
>> *.eln files with the same hashes and in the same 29.0.50-NNNNN
>> subdirectory?
> Yes exactly. The <version>-<hash> directory was the same.
>
>> Also, which *.el files were compiled into *.eln files
>> you could share -- were those the preloaded *.el files, non-preloaded
>> *.el files from the Emacs tree, or third-party *.el files from
>> packages that are not bundled with Emacs?
> Only the builtin el files as can be seen in the spec file.
>
>>In general, such different configurations could only be able to share
>>*.eln files by sheer luck.  It is enough to have one more or one less
>>primitive in one of the builds to require a separate set of *.eln
>>files, because the hash of the versioned subdirectory of native-lisp/
>>and of eln-cache changes when primitives are added or deleted.  That
>>has been so for a very long time, definitely long before commit
>>3c8167ec0f964.
>
> Does enabling or disabling lets say png change any of those
> primitives?

Yes

  Andrea

Re: Sharing native-lisp system load-path between builds

[Björn Bidar]

Andrea Corallo <acorallo@gnu.org> writes:

> Björn Bidar <bjorn.bidar@thaodan.de> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>>>> Date: Mon, 14 Aug 2023 21:29:34 +0300
>>>> 
>>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>> 
>>>> > Thanks.  So what does "work" and "does not work" mean in this context?
>>>> 
>>>> Work means the eln files could be shared, doesn't mean the files can't
>>>> be shared the hash in the native lisp path is different and the files
>>>> are no longer compatible.
>>>
>>> How did you share them? did you manually copy them into the same
>>> directory or forced Emacs to write them there?  Or did Emacs create
>>> *.eln files with the same hashes and in the same 29.0.50-NNNNN
>>> subdirectory?
>> Yes exactly. The <version>-<hash> directory was the same.
>>
>>> Also, which *.el files were compiled into *.eln files
>>> you could share -- were those the preloaded *.el files, non-preloaded
>>> *.el files from the Emacs tree, or third-party *.el files from
>>> packages that are not bundled with Emacs?
>> Only the builtin el files as can be seen in the spec file.
>>
>>>In general, such different configurations could only be able to share
>>>*.eln files by sheer luck.  It is enough to have one more or one less
>>>primitive in one of the builds to require a separate set of *.eln
>>>files, because the hash of the versioned subdirectory of native-lisp/
>>>and of eln-cache changes when primitives are added or deleted.  That
>>>has been so for a very long time, definitely long before commit
>>>3c8167ec0f964.
>>
>> Does enabling or disabling lets say png change any of those
>> primitives?
>
> Yes

What changed that now these primitives change this way or is this simply
a side effect of another change?

Re: Sharing native-lisp system load-path between builds

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sat, 19 Aug 2023 01:23:00 +0300
> 
> Andrea Corallo <acorallo@gnu.org> writes:
> 
> >> Does enabling or disabling lets say png change any of those
> >> primitives?
> >
> > Yes
> 
> What changed that now these primitives change this way or is this simply
> a side effect of another change?

Nothing fundamental changed recently, this was always the case, since
Emacs 28 was released.  So if it worked for you until now, it was only
by sheer luck: somehow the configurations you have been building had
the same primitives in them, and now they don't.

The assumption that you can share the *.eln files between different
configurations was never valid.

Re: Sharing native-lisp system load-path between builds, my own error

[Björn Bidar]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Björn Bidar <bjorn.bidar@thaodan.de>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Sat, 19 Aug 2023 01:23:00 +0300
>> 
>> Andrea Corallo <acorallo@gnu.org> writes:
>> 
>> >> Does enabling or disabling lets say png change any of those
>> >> primitives?
>> >
>> > Yes
>> 
>> What changed that now these primitives change this way or is this simply
>> a side effect of another change?
>
> Nothing fundamental changed recently, this was always the case, since
> Emacs 28 was released.  So if it worked for you until now, it was only
> by sheer luck: somehow the configurations you have been building had
> the same primitives in them, and now they don't.
>
> The assumption that you can share the *.eln files between different
> configurations was never valid.

The error was on my side. I assumed that my package only tracked on set
of eln files since without further thinking it does sound like that
compiled lisp code should be the same between 3 variants of the same
build besides different libraries linked to Emacs.

But the package contained compiled elisp for all three which worked
before commit 0cf6e0998ba. I ran distclean after each build, which
before said commit didn't delete the native-lisp directory but after it
deleted them to keeping only the last build.

After keeping each native-lisp directory again for each variant everything
is fine.

Re: Sharing native-lisp system load-path between builds

[Eli Zaretskii]

1> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: emacs-devel@gnu.org
> Date: Mon, 26 Jun 2023 07:26:03 +0300
> 
> > It isn't a recent change, at least not in principle.
> 
> What could have change this?

Something either in your build environments or in Emacs itself that
made the different builds incompatible, as far as the *.eln files are
concerned.

> Each build configuration is 100% the same except the optional
> dependencies they are build with.
> 
> There's no change to any source they are build one after the other.

If Emacs decided that they are not compatible, then they aren't.
Using the same source is not enough, since the same sources are used
to produce very different builds: with and without X, with and without
Lucid/GTK, to say nothing of builds for Windows and macOS.  The cause
of these differences in the binaries are the preprocessor directives
and conditions that change the outcome of the build under the control
of the configure script which decides what parts of the sources to
compile into the binary and what parts not to compile.