bug#66864: emacs-build-system builds .eln-files with mismatching path-hashes

bug#66864: emacs-build-system builds .eln-files with mismatching path-hashes

[Mekeor Melire]

BUG EXPLANATION

Emacs's natively-compiled .eln-files have a basename following the pattern "{feature-name}-{path-hash}-{content-hash}.eln". [0]

Guix' emacs-build-system is used to build Emacs-related packages. By 
 default, it uses the "emacs-minimal" package during build, which 
 does not support native-compilation. But if you replace the 
 "emacs-minimal" input with "emacs-no-x", e.g. by using 
 --with-input=emacs-minimal=emacs-no-x, then emacs-build-system 
 will make use of emacs-no-x' support of native-compilation [1]: 
 The build will contain .eln-files.

Hereby I'd like to report the bug that consists of mismatched path-hashes in the .eln-files that builds of Emacs-related packages contain when build with emacs-no-x (or any other Emacs that supports native compilation).

BUG REPRODUCTION

To reproduce this bug follow the following steps. Please note that guix-shell seems to leak .eln-files. (This should be reported as 
another bug.) That why the reproduction steps avoid guix-shell. 
Instead, we'll work with the current user profile.

Delete Emacs' eln-cache (so that we can later see if new 
.eln-files have been generated):

	rm -rf ~/.emacs.d/eln-cache

Remove all Emacs- and Emacs-related packages from Guix profile:

	guix package -I | cut -f 4 | grep emacs | xargs guix remove

Install Emacs and emacs-unfill, as exemplary package, while 
replacing input "emacs-minimal" with "emacs", so that .eln-files 
are generated during the build:

	guix install emacs emacs-unfill 
	--with-input=emacs-minimal=emacs

Launch the freshly installed Emacs and load the "unfill" package. 
If the .eln-files that the emacs-unfill package provides match 
Emacs' expectations (path- and content-hash), it'll use it; 
otherwise, Emacs will compile a new .eln-file and save it into 
~/.emacs.d/eln-cache/*/unfill-{path-hash}-{content-hash}.eln.

	emacs -q --eval "(require 'unfill)"

Close Emacs after some seconds. Now determine the path-hash from 
Guix' build:

	basename 
	~/.guix-profile/lib/emacs/native-site-lisp/*/unfill-*.eln \
	  | cut -d - -f 2

Determine the path-hash from Emacs' native-compilation, which 
apparently has happened:

	basename ~/.emacs.d/eln-cache/*/unfill*.eln \
	  | cut -d - -f 2

The path-hashes from the last two steps are not equal.

BUG SOLUTION HINTS

In the #guix:libera.chat IRC channel, jpoiret pointed out: "the .eln file hash problem is due to grafts, grafts change the 
final output name, but they can't also update the file hashes... 
we'd need to modify emacs' behavior for this to work".

CITATIONS

[0]: Emacs' source code documents the meaning of the two hashes here: https://git.sv.gnu.org/cgit/emacs.git/tree/src/comp.c?h=194a8f5c1406dd7e762376bdfde78d1b7d01b6b1#n4405

[1]: Here you can see that emacs-no-x supports native-compilation unlike emacs-minimal: https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/emacs.scm?h=92913703448c8e1a488ab066f60741262cdbf923#n294

bug#66864: emacs-build-system builds .eln-files with mismatching path-hashes

[Liliana Marie Prikler]

Am Dienstag, dem 31.10.2023 um 23:49 +0000 schrieb Mekeor Melire:
> BUG EXPLANATION
> 
> Emacs's natively-compiled .eln-files have a basename following the
> pattern "{feature-name}-{path-hash}-{content-hash}.eln". [0]
> 
> Guix' emacs-build-system is used to build Emacs-related packages. By 
>  default, it uses the "emacs-minimal" package during build, which 
>  does not support native-compilation. But if you replace the 
>  "emacs-minimal" input with "emacs-no-x", e.g. by using 
>  --with-input=emacs-minimal=emacs-no-x, then emacs-build-system 
>  will make use of emacs-no-x' support of native-compilation [1]: 
>  The build will contain .eln-files.
> 
> Hereby I'd like to report the bug that consists of mismatched path-
> hashes in the .eln-files that builds of Emacs-related packages
> contain when build with emacs-no-x (or any other Emacs that supports
> native compilation).
> 
> BUG REPRODUCTION
> 
> To reproduce this bug follow the following steps. Please note that
> guix-shell seems to leak .eln-files. (This should be reported as 
> another bug.)
What do you mean by "leaks .eln-files"?

> That why the reproduction steps avoid guix-shell.  Instead, we'll
> work with the current user profile.
> 
> Delete Emacs' eln-cache (so that we can later see if new 
> .eln-files have been generated):
> 
>         rm -rf ~/.emacs.d/eln-cache
> 
> Remove all Emacs- and Emacs-related packages from Guix profile:
> 
>         guix package -I | cut -f 4 | grep emacs | xargs guix remove
> 
> Install Emacs and emacs-unfill, as exemplary package, while 
> replacing input "emacs-minimal" with "emacs", so that .eln-files 
> are generated during the build:
> 
>         guix install emacs emacs-unfill 
>         --with-input=emacs-minimal=emacs
Just deleting the eln-cache should be enough for a MWE.  When doing an
MWE, make sure that its actually minimal :)

> Launch the freshly installed Emacs and load the "unfill" package. 
> If the .eln-files that the emacs-unfill package provides match 
> Emacs' expectations (path- and content-hash), it'll use it; 
> otherwise, Emacs will compile a new .eln-file and save it into 
> ~/.emacs.d/eln-cache/*/unfill-{path-hash}-{content-hash}.eln.
> 
>         emacs -q --eval "(require 'unfill)"
> 
> Close Emacs after some seconds. Now determine the path-hash from 
> Guix' build:
> 
>         basename 
>         ~/.guix-profile/lib/emacs/native-site-lisp/*/unfill-*.eln \
>           | cut -d - -f 2
> 
> Determine the path-hash from Emacs' native-compilation, which 
> apparently has happened:
> 
>         basename ~/.emacs.d/eln-cache/*/unfill*.eln \
>           | cut -d - -f 2
This is already the bug.  There should not be a file written to the
eln-cache (save for the trampolines that we still write there, which is
also a known bug among those who care).

> The path-hashes from the last two steps are not equal.
> 
> BUG SOLUTION HINTS
> 
> In the #guix:libera.chat IRC channel, jpoiret pointed out: "the .eln
> file hash problem is due to grafts, grafts change the 
> final output name, but they can't also update the file hashes... 
> we'd need to modify emacs' behavior for this to work".
As jpoiret points out, this has to do with the file naming choices of
Emacs, not with emacs-build-system per se.  We would need to get rid of
a lot of hashes if we wanted interoperable native-compiled Emacs
libraries.  I wonder what upstream has to say about this.

Cheers

bug#66864: emacs-build-system builds .eln-files with mismatching path-hashes

[Mekeor Melire]

2023-11-01 12:52 liliana.prikler@gmail.com:

> Am Dienstag, dem 31.10.2023 um 23:49 +0000 schrieb Mekeor 
> Melire:

> > To reproduce this bug follow the following steps. Please note 
> > that
> > guix-shell seems to leak .eln-files. (This should be reported 
> > as
> > another bug.)

> What do you mean by "leaks .eln-files"?

To be honest, I can't reproduce the leakage right now. I'll create another bug report if I can.

> > That why the reproduction steps avoid guix-shell.  Instead, 
> > we'll
> > work with the current user profile.
> >
> > Delete Emacs' eln-cache (so that we can later see if new
> > .eln-files have been generated):
> >
> >         rm -rf ~/.emacs.d/eln-cache
> >
> > Remove all Emacs- and Emacs-related packages from Guix 
> > profile:
> >
> >         guix package -I | cut -f 4 | grep emacs | xargs guix 
> > remove
> >
> > Install Emacs and emacs-unfill, as exemplary package, while
> > replacing input "emacs-minimal" with "emacs", so that 
> > .eln-files
> > are generated during the build:
> >
> >         guix install emacs emacs-unfill
> >         --with-input=emacs-minimal=emacs

> Just deleting the eln-cache should be enough for a MWE.  When 
> doing an
> MWE, make sure that its actually minimal :)

I wanted to make sure that an Emacs-related package is installed, and specifically with the --with-input=emacs-minimal=emacs transformation because otherwise .eln-files won't be built. The MRE is minimal in that sense that it ensures what's needed; only one Emacs-related package is installed; and commands are kept simple.

> > Launch the freshly installed Emacs and load the "unfill" 
> > package.
> > If the .eln-files that the emacs-unfill package provides match
> > Emacs' expectations (path- and content-hash), it'll use it;
> > otherwise, Emacs will compile a new .eln-file and save it into
> > ~/.emacs.d/eln-cache/*/unfill-{path-hash}-{content-hash}.eln.
> >
> >         emacs -q --eval "(require 'unfill)"
> >
> > Close Emacs after some seconds. Now determine the path-hash 
> > from
> > Guix' build:
> >
> >         basename
> >         ~/.guix-profile/lib/emacs/native-site-lisp/*/unfill-*.eln 
> > \
> >           | cut -d - -f 2
> >
> > Determine the path-hash from Emacs' native-compilation, which
> > apparently has happened:
> >
> >         basename ~/.emacs.d/eln-cache/*/unfill*.eln \
> >           | cut -d - -f 2

> This is already the bug.  There should not be a file written to 
> the
> eln-cache (save for the trampolines that we still write there, 
> which is
> also a known bug among those who care).

Yes, this is already the bug. The reason for the eln-cache to be created is that the two path-hashes do not equal.

> > The path-hashes from the last two steps are not equal.
> >
> > BUG SOLUTION HINTS
> >
> > In the #guix:libera.chat IRC channel, jpoiret pointed out: 
> > "the .eln
> > file hash problem is due to grafts, grafts change the
> > final output name, but they can't also update the file 
> > hashes...
> > we'd need to modify emacs' behavior for this to work".

> As jpoiret points out, this has to do with the file naming 
> choices of
> Emacs, not with emacs-build-system per se.  We would need to get 
> rid of
> a lot of hashes if we wanted interoperable native-compiled Emacs
> libraries.  I wonder what upstream has to say about this.

The problem is that the .el-file-path that is passed to the Emacs function comp-el-to-eln-filename during build [1] does not equal to the 
.el-file-path when Emacs is invoked. Personally, I do not 
understand how grafting causes this. But I can confirm that when 
--no-grafts is passed to "guix install emacs emacs-unfill 
--with-input=emacs-minimal=emacs", then no eln-cache is created.

[1]: See these lines of code:
https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/emacs-build-system.scm?h=92913703448c8e1a488ab066f60741262cdbf923#n133
https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/emacs-utils.scm?h=92913703448c8e1a488ab066f60741262cdbf923#n149

bug#66864: emacs-build-system builds .eln-files with mismatching path-hashes

[Liliana Marie Prikler]

Am Mittwoch, dem 01.11.2023 um 13:03 +0000 schrieb Mekeor Melire:
> The problem is that the .el-file-path that is passed to the Emacs
> function comp-el-to-eln-filename during build [1] does not equal to
> the .el-file-path when Emacs is invoked. Personally, I do not 
> understand how grafting causes this. But I can confirm that when 
> --no-grafts is passed to "guix install emacs emacs-unfill 
> --with-input=emacs-minimal=emacs", then no eln-cache is created.
I think Emacs might be calculating its own hash at runtime rather than
baking in the value at build time.  I would need to investigate this,
however.

Cheers

bug#66864: emacs-build-system builds .eln-files with mismatching path-hashes

[Mekeor Melire]

2023-11-01 15:16 liliana.prikler@gmail.com:

> I think Emacs might be calculating its own hash at runtime 
> rather than
> baking in the value at build time.

Exactly. That's what I was trying to express.

bug#66864: emacs-build-system builds .eln-files with mismatching path-hashes

[Liliana Marie Prikler]

Am Donnerstag, dem 02.11.2023 um 08:13 +0000 schrieb Mekeor Melire:
> 
> 2023-11-01 15:16 liliana.prikler@gmail.com:
> 
> > I think Emacs might be calculating its own hash at runtime 
> > rather than baking in the value at build time.
> 
> Exactly. That's what I was trying to express.
I'm not sure whether this is reproducible.  On my system
  $ guix build emacs-dash --with-input=emacs-minimal=emacs
  /gnu/store/zr16hd25338imljqxxfsf07smbfv3wxd-emacs-dash-2.19.1
  $ ls /gnu/store/zr16hd25338imljqxxfsf07smbfv3wxd-emacs-dash-2.19.1/lib/emacs/native-site-lisp
  29.1-e9e5c1ce
  $ emacs --batch --eval='(message "%s" comp-abi-hash)'
  e9e5c1ce
Looks like everything's alright?

Cheers

bug#66864: emacs-build-system builds .eln-files with mismatching path-hashes

[Josselin Poiret via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 944 bytes --]

Hi,

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

> Am Donnerstag, dem 02.11.2023 um 08:13 +0000 schrieb Mekeor Melire:
>> 
>> 2023-11-01 15:16 liliana.prikler@gmail.com:
>> 
>> > I think Emacs might be calculating its own hash at runtime 
>> > rather than baking in the value at build time.
>> 
>> Exactly. That's what I was trying to express.
> I'm not sure whether this is reproducible.  On my system
>   $ guix build emacs-dash --with-input=emacs-minimal=emacs
>   /gnu/store/zr16hd25338imljqxxfsf07smbfv3wxd-emacs-dash-2.19.1
>   $ ls /gnu/store/zr16hd25338imljqxxfsf07smbfv3wxd-emacs-dash-2.19.1/lib/emacs/native-site-lisp
>   29.1-e9e5c1ce
>   $ emacs --batch --eval='(message "%s" comp-abi-hash)'
>   e9e5c1ce
> Looks like everything's alright?

It's the .eln file itself that has the hash of the .el's path in its
name.  That's computed by `comp-el-to-eln-filename`.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

bug#66864: emacs-build-system builds .eln-files with mismatching path-hashes

[Liliana Marie Prikler]

Am Donnerstag, dem 09.11.2023 um 12:21 +0100 schrieb Josselin Poiret:
> Hi,
> 
> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> 
> > Am Donnerstag, dem 02.11.2023 um 08:13 +0000 schrieb Mekeor Melire:
> > > 
> > > 2023-11-01 15:16 liliana.prikler@gmail.com:
> > > 
> > > > I think Emacs might be calculating its own hash at runtime 
> > > > rather than baking in the value at build time.
> > > 
> > > Exactly. That's what I was trying to express.
> > I'm not sure whether this is reproducible.  On my system
> >   $ guix build emacs-dash --with-input=emacs-minimal=emacs
> >   /gnu/store/zr16hd25338imljqxxfsf07smbfv3wxd-emacs-dash-2.19.1
> >   $ ls /gnu/store/zr16hd25338imljqxxfsf07smbfv3wxd-emacs-dash-
> > 2.19.1/lib/emacs/native-site-lisp
> >   29.1-e9e5c1ce
> >   $ emacs --batch --eval='(message "%s" comp-abi-hash)'
> >   e9e5c1ce
> > Looks like everything's alright?
> 
> It's the .eln file itself that has the hash of the .el's path in its
> name.  That's computed by `comp-el-to-eln-filename`.
Does this still occur on the emacs-team branch, where we compile
everything from the build directory?

Cheers