bug#49270: 28.0.50: native-comp: macOS self-contained .app does not use bundled *.eln files

bug#49270: 28.0.50: native-comp: macOS self-contained .app does not use bundled *.eln files

[Jim Myhrberg]

When performing a self-contained Emacs.app build, *.eln files for
Emacs' built-in lisp files stored within the Emacs.app bundle are not
used.

I believe commit 5dd2d50 which moved *.eln files and various paths
around a bit for macOS builds, has re-introduced an old bug from last
year. Basically the checksums that makes up part of the *.eln file
names uses the absolute path of the .el file in question, but for self
contained .app builds it needs to just use the relative path to the
app itself.

In short, any *.el files which contain ".app/Contents/" within their
absolute file path, need to have anything before ".app/Contents/"
removed before the *.eln filename checksum is calculated. Potentially
some extra logic might be needed for build-time native-comp, as the
checksums in the *.eln files bundled into the app right now are not
based on the final location of lisp files within the .app bundle.

In my testing, of the two checksums which are part of all *.eln
filenames, the first checksum is different in the newly async
generated *.eln files compared to those within the application bundle.
The second checksum is identical. Also moving Emacs.app to different
locations on the file system will cause it to re-compile *.eln files
again, with the first checksum changing again.

For the sake of any doubt, this issue occurs both with and without
NATIVE_FULL_AOT=1.

I've seen this behavior with all builds from the master branch that
I've done since the 27th of June.

Hopefully I've provided enough information here, please don't hesitate
to ask if anything more is needed.

P.S. I'm pretty new to emacs bug reporting, so apologies if I'm doing
something wrong, and please let me know if so, and what I should be
doing instead :)

bug#49270: 28.0.50: native-comp: macOS self-contained .app does not use bundled *.eln files

[Eli Zaretskii]

> From: Jim Myhrberg <contact@jimeh.me>
> Date: Tue, 29 Jun 2021 12:51:48 +0100
> 
> When performing a self-contained Emacs.app build, *.eln files for
> Emacs' built-in lisp files stored within the Emacs.app bundle are not
> used.

Thanks, but how do you see that?

bug#49270: 28.0.50: native-comp: macOS self-contained .app does not use bundled *.eln files

[Jim Myhrberg]

On Tue, Jun 29, 2021 at 1:25 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Jim Myhrberg <contact@jimeh.me>
> > Date: Tue, 29 Jun 2021 12:51:48 +0100
> >
> > When performing a self-contained Emacs.app build, *.eln files for
> > Emacs' built-in lisp files stored within the Emacs.app bundle are not
> > used.
>
> Thanks, but how do you see that?

For better or worse, in a rather unsophisticated way; by launching Emacs.app and
seeing the async native-comp kick off for all of Emacs' built-in lisp files. And
by doing some additional experiments with *.eln files just now.

The Emacs.app bundle I built earlier today includes
"bytecomp-12882072-3f2b96ca.eln" in
"Emacs.app/Contents/MacOS/lib/emacs/28.0.50/native-lisp", but when launching
Emacs.app it performs async native-comp on bytecomp.el, and produced
"bytecomp-a3d51747-3f2b96ca.eln" in my user eln cache folder. If I just relaunch
Emacs again it does not re-compile bytecomp.el.

If I move the newly created "bytecomp-a3d51747-3f2b96ca.eln" from my user eln
cache folder into the Emacs.app bundle next to the original bytecomp-*.eln, and
launch Emacs.app, it also does not re-compile bytecomp.el.

However, if I then move Emacs.app itself (from /Users/jimeh/Downloads to
/Users/jimeh/Desktop in this case), it will re-compile bytecomp.el again on
launch, producing "bytecomp-30860bee-3f2b96ca.eln" in my user eln cache folder.

At the end I have three separate bytecomp-*.eln files:

- "bytecomp-12882072-3f2b96ca.eln" - produced at Emacs build time and bundled
  into the app
- "bytecomp-a3d51747-3f2b96ca.eln" - produced via async native-comp at launch of
  Emacs.app while it was located in "/Users/jimeh/Downloads/", with absolute
  path to bytecomp.el.gz being
  "/Users/jimeh/Downloads/Emacs.app/Contents/Resources/lisp/emacs-lisp/bytecomp.el.gz"
- "bytecomp-30860bee-3f2b96ca.eln" - produced via async native-comp at launch of
  Emacs.app while it was located in "/Users/jimeh/Desktop/", with absolute path
  to bytecomp.el.gz being
  "/Users/jimeh/Desktop/Emacs.app/Contents/Resources/lisp/emacs-lisp/bytecomp.el.gz"

And this is why I believe the first checksum within *.eln file names is based on
the absolute path even when the source .el/.el.gz files are located within a
.app bundle.

Also, I forgot mention earlier, but the previous bug for this issue from last
year is bug#43532, potentially that might give some additional context.

bug#49270: 28.0.50: native-comp: macOS self-contained .app does not use bundled *.eln files

[Eli Zaretskii]

> From: Jim Myhrberg <contact@jimeh.me>
> Date: Tue, 29 Jun 2021 12:51:48 +0100
> 
> When performing a self-contained Emacs.app build, *.eln files for
> Emacs' built-in lisp files stored within the Emacs.app bundle are not
> used.
> 
> I believe commit 5dd2d50 which moved *.eln files and various paths
> around a bit for macOS builds, has re-introduced an old bug from last
> year. Basically the checksums that makes up part of the *.eln file
> names uses the absolute path of the .el file in question, but for self
> contained .app builds it needs to just use the relative path to the
> app itself.

That's not (exactly) how this stuff is supposed to work, AFAIR, not
after the recent changes.  Please look at the code and the comments in
comp.c:comp-el-to-eln-rel-filename, and tell if you understand why
this isn't working for you.

bug#49270: 28.0.50: native-comp: macOS self-contained .app does not use bundled *.eln files

[Alan Third]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

On Tue, Jun 29, 2021 at 12:51:48PM +0100, Jim Myhrberg wrote:
> When performing a self-contained Emacs.app build, *.eln files for
> Emacs' built-in lisp files stored within the Emacs.app bundle are not
> used.
> 
> I believe commit 5dd2d50 which moved *.eln files and various paths
> around a bit for macOS builds, has re-introduced an old bug from last
> year. Basically the checksums that makes up part of the *.eln file
> names uses the absolute path of the .el file in question, but for self
> contained .app builds it needs to just use the relative path to the
> app itself.

I don't think this is the same bug (although I guess it has the same
symptoms).

Please try the attached patch, it fixes it here.
-- 
Alan Third

[-- Attachment #2: 0001-Fix-NS-native-comp-search-path-bug-49270.patch --]
[-- Type: text/plain, Size: 850 bytes --]

From 0517874eb4bce2cbbcdf58516c43d50b0174c0ec Mon Sep 17 00:00:00 2001
From: Alan Third <alan@idiocy.org>
Date: Tue, 29 Jun 2021 22:02:43 +0100
Subject: [PATCH] Fix NS native comp search path (bug#49270)

* configure.ac (NS_SELF_CONTAINED): We need to make lispdirrel the
same as lispdir when building a self contained app bundle as they're
both relative paths.
---
 configure.ac | 1 +
 1 file changed, 1 insertion(+)

diff --git a/configure.ac b/configure.ac
index c8920d877e..6e2cda947a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2025,6 +2025,7 @@ AC_DEFUN
      infodir="\${ns_appresdir}/info"
      mandir="\${ns_appresdir}/man"
      lispdir="\${ns_appresdir}/lisp"
+     lispdirrel="\${ns_appresdir}/lisp"
      test "$locallisppathset" = no && locallisppath="\${ns_appresdir}/site-lisp"
      INSTALL_ARCH_INDEP_EXTRA=
   fi
-- 
2.29.2

bug#49270: 28.0.50: native-comp: macOS self-contained .app does not use bundled *.eln files

[Jim Myhrberg]

On Tue, Jun 29, 2021 at 10:06 PM Alan Third <alan@idiocy.org> wrote:
> Please try the attached patch, it fixes it here.

Awesome, thanks. I've just tested the patch and can confirm it fixes
the issue on my end too. All *.eln files in the .app bundle are now
located correctly, even when moving the app around the filesystem :)

bug#49270: 28.0.50: native-comp: macOS self-contained .app does not use bundled *.eln files

[Alan Third]

On Wed, Jun 30, 2021 at 10:32:09AM +0100, Jim Myhrberg wrote:
> On Tue, Jun 29, 2021 at 10:06 PM Alan Third <alan@idiocy.org> wrote:
> > Please try the attached patch, it fixes it here.
> 
> Awesome, thanks. I've just tested the patch and can confirm it fixes
> the issue on my end too. All *.eln files in the .app bundle are now
> located correctly, even when moving the app around the filesystem :)

Pushed to master.
-- 
Alan Third