bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Jim Myhrberg]

Hi,

TL;DR:

If I've understood correctly, the filename of the cached *.eln files
includes a hash based on the absolute file path and content of the
source *.el/*.el.gz file. On macOS when producing a self-contained
Emacs.app bundle this means that the cached *.eln files bundled into
the app cannot be used, as their absolute path won't match what they
were during build time. And app bundles on macOS can be placed and
launched from anywhere on disk.

I'm not sure what the best solution here might be. But if the actual
content of the file is used to produce the hash, maybe there's no need
to use the full absolute file path, and instead just the filename
itself, along with the content?

The Long Version:

I'm not sure how familiar people here might be with macOS application
bundles, so here's a brief summary just in case; Application bundles
are self-contained macOS GUI applications which can be launched by for
example double clicking on them, among other ways. On a technical
level, they're simply a folder called "<AppName>.app" which contains
all assets needed to run the application. The main executable is
"<AppName>.app/Contents/MacOS/<AppName>".

In the case of building a self-contained Emacs.app on macOS, it looks
the *.eln caches are generated for the files located in
"<workdir>/nextstep/Emacs.app/Contents/Resources/lisp", and the *.eln
cache folder is
"<workdir>/nextstep/Emacs.app/Contents/MacOS/lib/emacs/28.0.50/native-lisp".
Which means the full set of *.eln caches can only be used when
launched from within the Emacs source directory where you built the
application.

Typical usage on macOS would have the Emacs.app movied/copied to the
"/Applications" folder, but there's no guarantee about that, as
applications are supposed work fine regardless of where it's located
on disk they're located.

I've done some tests while using NATIVE_FULL_AOT=1, which produces
1,572 *.eln files within Emacs.app. But when starting emacs with my
config, it starts by compiling a whole bunch of files from
"Emacs.app/Contents/Resources/lisp/", like "emacs-lisp/cconv.el.gz",
"emacs-lisp/bytecomp.el.gz", "help-mode.el.gz", and others. If I grab
the new *.eln files it produced in my user cache, and move them to
within Emacs.app, it doesn't compile them on launch again. If I then
move Emacs.app to another location on disk, it does compile them
again, as the absolute path has changed.

However there's 83 *.eln files which are loaded from Emacs.app,
because if they're removed, it fails to start at all with a dlopen
error instead. I don't know enough about the internals of Emacs, but
I'd assume they are referred to from the dump file, while the others
are loaded up due to my config requiring them, at which point a cache
miss happens due to the absolute path yielding a different change
filename.


Thanks :)

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Jim Myhrberg <contact@jimeh.me> writes:

> Hi,
>
> TL;DR:
>
> If I've understood correctly, the filename of the cached *.eln files
> includes a hash based on the absolute file path and content of the
> source *.el/*.el.gz file. On macOS when producing a self-contained
> Emacs.app bundle this means that the cached *.eln files bundled into
> the app cannot be used, as their absolute path won't match what they
> were during build time. And app bundles on macOS can be placed and
> launched from anywhere on disk.

Hi Jim,

I think we should be able to improve the filename hashing mechanism on
MacOS to deal with that.  Perhaps something like removing
*/Emacs.app/Contents/MacOS/ from the input of it.

Could you post the value of PATH_DUMPLOADSEARCH and PATH_LOADSEARCH
macros?  You should find them defined in epaths.h in your build
directory.

Thanks

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Jim Myhrberg]

Here they are:

#define PATH_LOADSEARCH
"/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-5b41545/nextstep/Emacs.app/Contents/Resources/lisp"
#define PATH_DUMPLOADSEARCH
"/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-5b41545/lisp"

"/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-5b41545"
is obviously my local source/build directory that my build script
extracts the source tarball to and builds from.

On Mon, Sep 21, 2020 at 8:53 AM Andrea Corallo <akrl@sdf.org> wrote:
>
> Jim Myhrberg <contact@jimeh.me> writes:
>
> > Hi,
> >
> > TL;DR:
> >
> > If I've understood correctly, the filename of the cached *.eln files
> > includes a hash based on the absolute file path and content of the
> > source *.el/*.el.gz file. On macOS when producing a self-contained
> > Emacs.app bundle this means that the cached *.eln files bundled into
> > the app cannot be used, as their absolute path won't match what they
> > were during build time. And app bundles on macOS can be placed and
> > launched from anywhere on disk.
>
> Hi Jim,
>
> I think we should be able to improve the filename hashing mechanism on
> MacOS to deal with that.  Perhaps something like removing
> */Emacs.app/Contents/MacOS/ from the input of it.
>
> Could you post the value of PATH_DUMPLOADSEARCH and PATH_LOADSEARCH
> macros?  You should find them defined in epaths.h in your build
> directory.
>
> Thanks
>
>   Andrea

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 607 bytes --]

Jim Myhrberg <contact@jimeh.me> writes:

> Here they are:
>
> #define PATH_LOADSEARCH
> "/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-5b41545/nextstep/Emacs.app/Contents/Resources/lisp"
> #define PATH_DUMPLOADSEARCH
> "/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-5b41545/lisp"
>
> "/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-5b41545"
> is obviously my local source/build directory that my build script
> extracts the source tarball to and builds from.

Could you have a run with the attached blind patch?

Thanks

  Andrea
  

[-- Attachment #2: macos-relpath.path --]
[-- Type: text/plain, Size: 552 bytes --]

diff --git a/src/comp.c b/src/comp.c
index 15d85d30fc..c51d13ade1 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -4060,6 +4060,10 @@ DEFUN ("comp-el-to-eln-filename", Fcomp_el_to_eln_filename,
       FOR_EACH_TAIL (loadsearch_list)
 	loadsearch_re_list =
 	  Fcons (Fregexp_quote (XCAR (loadsearch_list)), loadsearch_re_list);
+      loadsearch_re_list =
+	Fcons (
+	  build_string ("^[[:ascii:]]+/Emacs\\.app/Contents/Resources/lisp"),
+	  loadsearch_re_list);
     }
   Lisp_Object loadsearch_res = loadsearch_re_list;
   FOR_EACH_TAIL (loadsearch_res)

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Jim Myhrberg]

That works perfectly. It's picking up all *.eln files from within
Emacs.app, regardless of where I place it on disk, and only generating
new  *.eln files into my user cache for external packages I've
installed myself.

Thanks yet again for your help :)

On Mon, Sep 21, 2020 at 11:26 AM Andrea Corallo <akrl@sdf.org> wrote:
>
> Jim Myhrberg <contact@jimeh.me> writes:
>
> > Here they are:
> >
> > #define PATH_LOADSEARCH
> > "/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-5b41545/nextstep/Emacs.app/Contents/Resources/lisp"
> > #define PATH_DUMPLOADSEARCH
> > "/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-5b41545/lisp"
> >
> > "/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-5b41545"
> > is obviously my local source/build directory that my build script
> > extracts the source tarball to and builds from.
>
> Could you have a run with the attached blind patch?
>
> Thanks
>
>   Andrea
>

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Jim Myhrberg <contact@jimeh.me> writes:

> That works perfectly. It's picking up all *.eln files from within
> Emacs.app, regardless of where I place it on disk, and only generating
> new  *.eln files into my user cache for external packages I've
> installed myself.
>
> Thanks yet again for your help :)

Welcome :) I'll rewrite it a little more decently and push it soon.  We
can keep the bug open since it's there.

Thanks

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Jim Myhrberg]

I was just reminded by someone on GitHub that the "Emacs.app"
application can be renamed to anything by users. So I think we
probably need to effectively match against and remove
"^.*\.app\/Contents\/Resources\/lisp", basically wildcare anything
before the ".app" part.

Sorry I didn't think of this earlier myself >_<

On Mon, Sep 21, 2020 at 2:41 PM Andrea Corallo <akrl@sdf.org> wrote:
>
> Jim Myhrberg <contact@jimeh.me> writes:
>
> > That works perfectly. It's picking up all *.eln files from within
> > Emacs.app, regardless of where I place it on disk, and only generating
> > new  *.eln files into my user cache for external packages I've
> > installed myself.
> >
> > Thanks yet again for your help :)
>
> Welcome :) I'll rewrite it a little more decently and push it soon.  We
> can keep the bug open since it's there.
>
> Thanks
>
>   Andrea

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Jim,

could you check what I've pushed to scratch/native-comp-macos-43532
works for you?

I've done what I could to test it but is not much as I'm not on MacOS.

Thanks

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Jim Myhrberg]

Hi Andrea,

I've just tested that branch, and I'm afraid it doesn't work. It
starts native compiling cconv.el and others like before. Looking at
the changes, I believe it's looking for "Emacs.app" within
PATH_DUMPLOADSEARCH, which didn't actually contain that in my build
directory, it was PATH_LOADSEARCH that did. So currently that branch
doesn't find the bundled *.eln files even when the app is named
"Emacs.app".

And annoyingly we also need to support it if the user renames the app
to something else, like "Emacs 28.app", or "Emacs Native.app", or even
something crazy like "fooBar! LaLa.app".

My knowledge of C is mostly non-existent, but in an effort to reduce
the feedback loop and not take up too much of your time, I'll try some
guess-based tweaks and see if I have any success :)

Thanks again :)

On Mon, Sep 21, 2020 at 8:38 PM Andrea Corallo <akrl@sdf.org> wrote:
>
> Hi Jim,
>
> could you check what I've pushed to scratch/native-comp-macos-43532
> works for you?
>
> I've done what I could to test it but is not much as I'm not on MacOS.
>
> Thanks
>
>   Andrea

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Jim Myhrberg]

I'm afraid I've tried all the guess-based tweaks I can think of
without any luck, mostly just changing the two PATH_* variables
around.

On Mon, Sep 21, 2020 at 10:17 PM Jim Myhrberg <contact@jimeh.me> wrote:
>
> Hi Andrea,
>
> I've just tested that branch, and I'm afraid it doesn't work. It
> starts native compiling cconv.el and others like before. Looking at
> the changes, I believe it's looking for "Emacs.app" within
> PATH_DUMPLOADSEARCH, which didn't actually contain that in my build
> directory, it was PATH_LOADSEARCH that did. So currently that branch
> doesn't find the bundled *.eln files even when the app is named
> "Emacs.app".
>
> And annoyingly we also need to support it if the user renames the app
> to something else, like "Emacs 28.app", or "Emacs Native.app", or even
> something crazy like "fooBar! LaLa.app".
>
> My knowledge of C is mostly non-existent, but in an effort to reduce
> the feedback loop and not take up too much of your time, I'll try some
> guess-based tweaks and see if I have any success :)
>
> Thanks again :)
>
> On Mon, Sep 21, 2020 at 8:38 PM Andrea Corallo <akrl@sdf.org> wrote:
> >
> > Hi Jim,
> >
> > could you check what I've pushed to scratch/native-comp-macos-43532
> > works for you?
> >
> > I've done what I could to test it but is not much as I'm not on MacOS.
> >
> > Thanks
> >
> >   Andrea

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Jim Myhrberg <contact@jimeh.me> writes:

> Hi Andrea,
>
> I've just tested that branch, and I'm afraid it doesn't work. It
> starts native compiling cconv.el and others like before. Looking at
> the changes, I believe it's looking for "Emacs.app" within
> PATH_DUMPLOADSEARCH, which didn't actually contain that in my build
> directory, it was PATH_LOADSEARCH that did. So currently that branch
> doesn't find the bundled *.eln files even when the app is named
> "Emacs.app".
>
> And annoyingly we also need to support it if the user renames the app
> to something else, like "Emacs 28.app", or "Emacs Native.app", or even
> something crazy like "fooBar! LaLa.app".
>
> My knowledge of C is mostly non-existent, but in an effort to reduce
> the feedback loop and not take up too much of your time, I'll try some
> guess-based tweaks and see if I have any success :)
>
> Thanks again :)

Hi Jim,

apologies to a quick look I've inverted PATH_LOADSEARCH and
PATH_DUMPLOADSEARCH in the code :/

I've also addressed the *.app suggestion (yesterday I've missed your
mail).

Lets see if I got it right now.

Thanks

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Jim Myhrberg]

HI Andrea,

I'm afraid it's still not picking up the *.eln files from within the
app bundle. I tried inverting those two variables myself last night
without any luck.

Anything I can check on my end to yield some more light on what's going on?

On Tue, Sep 22, 2020 at 8:59 AM Andrea Corallo <akrl@sdf.org> wrote:
>
> Jim Myhrberg <contact@jimeh.me> writes:
>
> > Hi Andrea,
> >
> > I've just tested that branch, and I'm afraid it doesn't work. It
> > starts native compiling cconv.el and others like before. Looking at
> > the changes, I believe it's looking for "Emacs.app" within
> > PATH_DUMPLOADSEARCH, which didn't actually contain that in my build
> > directory, it was PATH_LOADSEARCH that did. So currently that branch
> > doesn't find the bundled *.eln files even when the app is named
> > "Emacs.app".
> >
> > And annoyingly we also need to support it if the user renames the app
> > to something else, like "Emacs 28.app", or "Emacs Native.app", or even
> > something crazy like "fooBar! LaLa.app".
> >
> > My knowledge of C is mostly non-existent, but in an effort to reduce
> > the feedback loop and not take up too much of your time, I'll try some
> > guess-based tweaks and see if I have any success :)
> >
> > Thanks again :)
>
> Hi Jim,
>
> apologies to a quick look I've inverted PATH_LOADSEARCH and
> PATH_DUMPLOADSEARCH in the code :/
>
> I've also addressed the *.app suggestion (yesterday I've missed your
> mail).
>
> Lets see if I got it right now.
>
> Thanks
>
>   Andrea

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Jim Myhrberg <contact@jimeh.me> writes:

> HI Andrea,
>
> I'm afraid it's still not picking up the *.eln files from within the
> app bundle. I tried inverting those two variables myself last night
> without any luck.
>
> Anything I can check on my end to yield some more light on what's going on?

Glab, it may work now please have a run when you've time, apologies for
the back and forward.

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Jim Myhrberg]

We have success :D

It now works and picks up the bundled *.eln files regardless of where
I place the app bundle or what I rename it to.

Thanks again, and, no, I'm sorry for the back and forth. You're the
one fixing stuff for me :)

On Tue, Sep 22, 2020 at 11:21 AM Andrea Corallo <akrl@sdf.org> wrote:
>
> Jim Myhrberg <contact@jimeh.me> writes:
>
> > HI Andrea,
> >
> > I'm afraid it's still not picking up the *.eln files from within the
> > app bundle. I tried inverting those two variables myself last night
> > without any luck.
> >
> > Anything I can check on my end to yield some more light on what's going on?
>
> Glab, it may work now please have a run when you've time, apologies for
> the back and forward.
>
>   Andrea

bug#43532: [feature/native-comp] *.eln file name hashing algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Jim Myhrberg <contact@jimeh.me> writes:

> We have success :D

Amazing, see was easy :D

I've pushed it as 4a50f54144 so I'm closing this.

Thanks for reporting and your patience in testing the fix.

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Aloxaf]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

Hello,

Sorry for disturbing you. In fact, this bug not only affects macOS. I am 
an Arch Linux user and install native-comp branch by AUR package 
emacs-native-comp-git 
(https://aur.archlinux.org/packages/emacs-native-comp-git/). It will 
build and install the whole package in a temporary directory and then 
pack the directory to a standalone package. So I also suffer from this bug.

For example, in my computer, the package will be built in $srcdir ( 
/tmp/makepkg/emacs-native-comp-git/src ), and will be installed to 
$pkgdir ( /tmp/makepkg/emacs-native-comp-git/pkg/emacs-native-comp-git 
). Then the whole $pkgdir will be packed into a package.

You can find more information here: 
https://wiki.archlinux.org/index.php/Creating_packages#package()

Thanks,

Aloxaf


[-- Attachment #2: Type: text/html, Size: 1356 bytes --]

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Aloxaf <aloxafx@gmail.com> writes:

> Hello,
>
> Sorry for disturbing you. In fact, this bug not only affects macOS. I am an Arch Linux user and install native-comp
> branch by AUR package emacs-native-comp-git (https://aur.archlinux.org/packages/emacs-native-comp-git/). It will build
> and install the whole package in a temporary directory and then pack the directory to a standalone package. So I also
> suffer from this bug.
>
> For example, in my computer, the package will be built in $srcdir ( /tmp/makepkg/emacs-native-comp-git/src ), and will be
> installed to $pkgdir ( /tmp/makepkg/emacs-native-comp-git/pkg/emacs-native-comp-git ). Then the whole $pkgdir will be
> packed into a package.
>
> You can find more information here: https://wiki.archlinux.org/index.php/Creating_packages#package()
>
> Thanks,

Hi Aloxaf,

no disturb I see.

I think I should be able to rework it for a sulution that is not MacOS
specific but generic.  I'll come up with a patch.

Thanks

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Could you share the values of your PATH_DUMPLOADSEARCH and
PATH_LOADSEARCH macros?  You'll find them defined in epaths.h

Thanks

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Aloxaf Yin <aloxafx@gmail.com> writes:

> The value of PATH_DUMPLOADSEARCH is
> /tmp/makepkg/emacs-native-comp-git/src/emacs-git/lisp
> and PATH_LOADSEARCH is /usr/share/emacs/28.0.50/lisp
>
> The build dir will not always be /tmp/makepkg. It depends on user's settings.

Once the package is produced it can be in any directory with no
restrinction on the name or like in MacOS the pkg directory will be
called like name.app?

Thanks

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Aloxaf <aloxafx@gmail.com> writes:

> 在 2020/9/30 下午3:13, Andrea Corallo 写道:
>
>> Once the package is produced it can be in any directory with no
>> restrinction on the name or like in MacOS the pkg directory will be
>> called like name.app?
>>
>> Thanks
>>
>>    Andrea
>
> If you mean where it will be installed. It will always be installed to
> / by default, but there is not some special restriction.
> The $pkgdir just like a fake root and install it with AUR should have
> the same directory structure as `make install DSTDIR='/'`
>
> If you mean where the $pkgdir will be. Its value is
> $BUILDDIR/$PACKAGE_NAME/pkg/$PACKAGE_NAME, $BUILDDIR can be configured
> by user.

Yes that's what I meant.  Okay then if is correct to assume
$PACKAGE_NAME is know at compile time should be possible on our side to
offer a build option to specify that while compiling.

> BTW, is there any special reason to consider file path when hashing?
> IMHO, it may be better not to consider it?

It makes simple and very cheap to keep the cache directory clean, the
fact that's cheap allow us to the this clean-up often.

Thanks

  Andrea

PS you should do a replay-all when replaying to keep the bug posted

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Aloxaf]

[-- Attachment #1: Type: text/plain, Size: 1430 bytes --]



在 2020/10/1 下午9:19, Andrea Corallo 写道:
> Aloxaf <aloxafx@gmail.com> writes:
>
>> 在 2020/9/30 下午3:13, Andrea Corallo 写道:
>>
>>> Once the package is produced it can be in any directory with no
>>> restrinction on the name or like in MacOS the pkg directory will be
>>> called like name.app?
>>>
>>> Thanks
>>>
>>>     Andrea
>> If you mean where it will be installed. It will always be installed to
>> / by default, but there is not some special restriction.
>> The $pkgdir just like a fake root and install it with AUR should have
>> the same directory structure as `make install DSTDIR='/'`
>>
>> If you mean where the $pkgdir will be. Its value is
>> $BUILDDIR/$PACKAGE_NAME/pkg/$PACKAGE_NAME, $BUILDDIR can be configured
>> by user.
> Yes that's what I meant.  Okay then if is correct to assume
> $PACKAGE_NAME is know at compile time should be possible on our side to
> offer a build option to specify that while compiling.

You mean something like `./configure --pkgdir=xxx`? That sounds good and 
flexible.

>
>> BTW, is there any special reason to consider file path when hashing?
>> IMHO, it may be better not to consider it?
> It makes simple and very cheap to keep the cache directory clean, the
> fact that's cheap allow us to the this clean-up often.
>
> Thanks
>
>    Andrea
>
> PS you should do a replay-all when replaying to keep the bug posted

Oops, sorry for that.


Thanks

     Aloxaf


[-- Attachment #2: Type: text/html, Size: 2445 bytes --]

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Aloxaf <aloxafx@gmail.com> writes:

> 在 2020/10/1 下午9:19, Andrea Corallo 写道:
>
>  Aloxaf <aloxafx@gmail.com> writes:
>
>  在 2020/9/30 下午3:13, Andrea Corallo 写道:
>
>  Once the package is produced it can be in any directory with no
> restrinction on the name or like in MacOS the pkg directory will be
> called like name.app?
>
> Thanks
>
>    Andrea
>
>
> If you mean where it will be installed. It will always be installed to
> / by default, but there is not some special restriction.
> The $pkgdir just like a fake root and install it with AUR should have
> the same directory structure as `make install DSTDIR='/'`
>
> If you mean where the $pkgdir will be. Its value is
> $BUILDDIR/$PACKAGE_NAME/pkg/$PACKAGE_NAME, $BUILDDIR can be configured
> by user.
>
>
> Yes that's what I meant.  Okay then if is correct to assume
> $PACKAGE_NAME is know at compile time should be possible on our side to
> offer a build option to specify that while compiling.
>
> You mean something like `./configure --pkgdir=xxx`? That sounds good and flexible.
>

Yes something like, perhaps to begin with something to pass to make as
`make PKGDIR=xxx`.

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Aloxaf,

I believe the following should fix the issue:

afb765ab3c * Make filename hashing compatible with self contained builds (bug#43532)

No additional parameter is needed.

Could you check and confirm works for you?

Cc-ing Jim to kindly check does not break the same functionality on
MacOS.

Thanks!

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Jim Myhrberg]

Thanks Andrea,

I just created a build from commit 44ef243, and it did not work out
quite as expected :)

Is there anything extra I should do/pass in to make it work? Cause
right now it seems to have taken the string "{lispdirrel}" literally
without any replacement, as there's a folder called exactly that
directly within the application bundle with most things I'd normally
expect to find in "Contents/Resources".

To illustrate with a screenshot to avoid any possible confusion:
https://share.jimeh.me/Screen-Shot-2020-10-04-20-47-12.95-goSgK6urIRqD.png

And this is what it's supposed to look like:
https://share.jimeh.me/Screen-Shot-2020-10-04-20-48-18.29-gd8o91tZcuLj.png

And for good measure, here's what I believe are the relevant things
from src/epaths.h:

#define PATH_LOADSEARCH
"/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-44ef243/nextstep/Emacs.app/{lispdirrel}/lisp"
#define PATH_REL_LOADSEARCH "Contents/Resources"
#define PATH_DUMPLOADSEARCH
"/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-44ef243/lisp"

On Sun, Oct 4, 2020 at 7:04 PM Andrea Corallo <akrl@sdf.org> wrote:
>
> Hi Aloxaf,
>
> I believe the following should fix the issue:
>
> afb765ab3c * Make filename hashing compatible with self contained builds (bug#43532)
>
> No additional parameter is needed.
>
> Could you check and confirm works for you?
>
> Cc-ing Jim to kindly check does not break the same functionality on
> MacOS.
>
> Thanks!
>
>   Andrea
>

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Jim Myhrberg <contact@jimeh.me> writes:

> Thanks Andrea,
>
> I just created a build from commit 44ef243, and it did not work out
> quite as expected :)

This 43532 is becoming my nightmare :)

915214ac9a should fix.

Thanks for testing.

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Jim Myhrberg]

Haha, apologies for the nightmare. There are few things worse than
writing code you can't actually test and run yourself
<insert-some-form-of-nightmare-emoji>

With 915214a the application is now structured correctly, but I'm
afraid it doesn't look to be truncating the paths correctly during the
initial build process. As the *.eln files bundled into the app bundle
during the build process with NATIVE_FULL_AOT=1 are ignored. But *.eln
files dynamically compiled into the user eln-cache directory are
re-used regardless of where you place the application on disk.

And relevant stuff from epaths.h just to be safe:

#define PATH_LOADSEARCH
"/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-915214a/nextstep/Emacs.app/Contents/Resources/lisp"
#define PATH_REL_LOADSEARCH "Contents/Resources"
#define PATH_DUMPLOADSEARCH
"/Users/jimeh/Projects/build-emacs-for-macos/sources/emacs-mirror-emacs-915214a/lisp"

And thanks again for all your work :)

On Sun, Oct 4, 2020 at 9:54 PM Andrea Corallo <akrl@sdf.org> wrote:
>
> Jim Myhrberg <contact@jimeh.me> writes:
>
> > Thanks Andrea,
> >
> > I just created a build from commit 44ef243, and it did not work out
> > quite as expected :)
>
> This 43532 is becoming my nightmare :)
>
> 915214ac9a should fix.
>
> Thanks for testing.
>
>   Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Jim,

323200044f should fix.

Thanks again

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Jim Myhrberg]

That did the trick, bundled *.eln files are correctly picked up again.

Thanks yet again for all your work :)

On Mon, Oct 5, 2020 at 7:57 AM Andrea Corallo <akrl@sdf.org> wrote:
>
> Hi Jim,
>
> 323200044f should fix.
>
> Thanks again
>
>   Andrea
>

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Jim Myhrberg <contact@jimeh.me> writes:

> That did the trick, bundled *.eln files are correctly picked up again.

Great, lets wait for Aloxaf feedback then.

Thanks

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Aloxaf Yin]

Oh, I forgot to send the draft! Sorry for the late reply.

Thanks for your work!

I tried 915214ac but it doesn't seem to work.

Here are some relevant definitions from src/epath.h:
#define PATH_LOADSEARCH "/usr/share/emacs/28.0.50/lisp"
#define PATH_REL_LOADSEARCH "28.0.50/lisp"
#define PATH_DUMPLOADSEARCH
"/tmp/makepkg/emacs-native-comp-git/src/emacs-git/lisp"

That's strange. Can I do anything to help analyze it?

Andrea Corallo <akrl@sdf.org> 于2020年10月5日周一 上午2:04写道：

>
> Hi Aloxaf,
>
> I believe the following should fix the issue:
>
> afb765ab3c * Make filename hashing compatible with self contained builds (bug#43532)
>
> No additional parameter is needed.
>
> Could you check and confirm works for you?
>
> Cc-ing Jim to kindly check does not break the same functionality on
> MacOS.
>
> Thanks!
>
>   Andrea
>

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Aloxaf Yin <aloxafx@gmail.com> writes:

> Oh, I forgot to send the draft! Sorry for the late reply.
>
> Thanks for your work!
>
> I tried 915214ac but it doesn't seem to work.
>
> Here are some relevant definitions from src/epath.h:
> #define PATH_LOADSEARCH "/usr/share/emacs/28.0.50/lisp"
> #define PATH_REL_LOADSEARCH "28.0.50/lisp"
> #define PATH_DUMPLOADSEARCH
> "/tmp/makepkg/emacs-native-comp-git/src/emacs-git/lisp"
>
> That's strange. Can I do anything to help analyze it?

Hi Aloxaf,

Yes that's curious the above looks correct.  I just ask you to post the
full path of an .el file (say comp.el) while being compiled and after
the packaging.

Thanks

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Aloxaf Yin]

Hi Andrea,

Take comp.el as an example, its path is
`/tmp/makepkg/emacs-native-comp-git/src/emacs-git/lisp/emacs-lisp/comp.el`
while being compiled and
`/usr/share/emacs/28.0.50/lisp/emacs-lisp/comp.el` after the
packaging.

Thanks

Andrea Corallo <akrl@sdf.org> 于2020年10月7日周三 下午11:00写道：
>
> Aloxaf Yin <aloxafx@gmail.com> writes:
>
> > Oh, I forgot to send the draft! Sorry for the late reply.
> >
> > Thanks for your work!
> >
> > I tried 915214ac but it doesn't seem to work.
> >
> > Here are some relevant definitions from src/epath.h:
> > #define PATH_LOADSEARCH "/usr/share/emacs/28.0.50/lisp"
> > #define PATH_REL_LOADSEARCH "28.0.50/lisp"
> > #define PATH_DUMPLOADSEARCH
> > "/tmp/makepkg/emacs-native-comp-git/src/emacs-git/lisp"
> >
> > That's strange. Can I do anything to help analyze it?
>
> Hi Aloxaf,
>
> Yes that's curious the above looks correct.  I just ask you to post the
> full path of an .el file (say comp.el) while being compiled and after
> the packaging.
>
> Thanks
>
>   Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrew Whatson]

Hi Andrea,

I see similar problems with my Guix builds of emacs-native-comp.  I
run with deferred compilation disabled, so the native versions of core
elisp packages are simply not loaded.

As a workaround, I've had success copying
`/path/to/emacs/lib/emacs/28.0.50/native-lisp/.../*.eln` into
`~/.emacs.d/eln-cache/.../`.  The native-compiled code is then loaded
correctly as expected.

Perhaps this helps to narrow down the problem; seems it's not a
hashing problem, but a load-path one?

Cheers,
Andrew

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andrew Whatson <whatson@gmail.com> writes:

> Hi Andrea,
>
> I see similar problems with my Guix builds of emacs-native-comp.  I
> run with deferred compilation disabled, so the native versions of core
> elisp packages are simply not loaded.
>
> As a workaround, I've had success copying
> `/path/to/emacs/lib/emacs/28.0.50/native-lisp/.../*.eln` into
> `~/.emacs.d/eln-cache/.../`.  The native-compiled code is then loaded
> correctly as expected.
>
> Perhaps this helps to narrow down the problem; seems it's not a
> hashing problem, but a load-path one?

Hi Andrew,

from where I stand I don't see why it should not work so I'll have to
recreate the problem here and have a look.  I'll try to do it this week.

Thanks!

  Andrea

bug#43532: [PATCH] Fix timestamp check when loading from eln (bug #43532)

[Andrew Whatson]

	* src/lread.c (maybe_swap_for_eln): Allow eln files to have the
	same timestamp as their source file.
---
 src/lread.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/lread.c b/src/lread.c
index ea31131b75..ab16fdadb4 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -1622,7 +1622,7 @@ maybe_swap_for_eln (Lisp_Object *filename, int *fd, struct timespec mtime)
 	  else
 	    {
 	      struct timespec eln_mtime = get_stat_mtime (&eln_st);
-	      if (timespec_cmp (eln_mtime, mtime) > 0)
+	      if (timespec_cmp (eln_mtime, mtime) >= 0)
 		{
 		  *filename = eln_name;
 		  emacs_close (*fd);
-- 
2.28.0

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrew Whatson]

Hi Andrea,

I added some debug messages to the loading procedure and determined
that the files were ignored because the timestamp check required the
eln to be newer than the source file.  This makes sense when running
from the build directory or when the eln's are built with deferred
compilation, but doesn't work for packaging where they're usually
installed with the same timestamp.

Though the fix is trivial, I wonder whether a timestamp check is
needed at all.  If we're already hashing the contents of the source
file, perhaps that's sufficient to detect when things are outdated?

Cheers,
Andrew

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andrew Whatson <whatson@gmail.com> writes:

> Hi Andrea,
>
> I added some debug messages to the loading procedure and determined
> that the files were ignored because the timestamp check required the
> eln to be newer than the source file.  This makes sense when running
> from the build directory or when the eln's are built with deferred
> compilation, but doesn't work for packaging where they're usually
> installed with the same timestamp.
>
> Though the fix is trivial, I wonder whether a timestamp check is
> needed at all.  If we're already hashing the contents of the source
> file, perhaps that's sufficient to detect when things are outdated?

Hi Andrew,

thanks for investigating!

yes you are correct, the system is over-constrained and I believe too
the timestamp is not anymore necessary now that we hash the file content
to verify the .el -> .eln relation.

I've pushed the following.

03dfa83dc3 * Do not check eln timestamp as superseded by source hashing (bug#43532)

Let us know if the issue is solved.

Thanks!

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrew Whatson]

On Thu, 15 Oct 2020 at 20:54, Andrea Corallo <akrl@sdf.org> wrote:

> I've pushed the following.
>
> 03dfa83dc3 * Do not check eln timestamp as superseded by source hashing (bug#43532)
>
> Let us know if the issue is solved.

Yes, this is now working properly for me, thanks!

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andrew Whatson <whatson@gmail.com> writes:

> On Thu, 15 Oct 2020 at 20:54, Andrea Corallo <akrl@sdf.org> wrote:
>
>> I've pushed the following.
>>
>> 03dfa83dc3 * Do not check eln timestamp as superseded by source hashing (bug#43532)
>>
>> Let us know if the issue is solved.
>
> Yes, this is now working properly for me, thanks!

Great, lets see Aloxaf feedback.

  Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Jim Myhrberg]

For what it's worth, things still work just fine on macOS with commit 03dfa83 :)

On Thu, Oct 15, 2020 at 1:56 PM Andrea Corallo <akrl@sdf.org> wrote:
>
> Andrew Whatson <whatson@gmail.com> writes:
>
> > On Thu, 15 Oct 2020 at 20:54, Andrea Corallo <akrl@sdf.org> wrote:
> >
> >> I've pushed the following.
> >>
> >> 03dfa83dc3 * Do not check eln timestamp as superseded by source hashing (bug#43532)
> >>
> >> Let us know if the issue is solved.
> >
> > Yes, this is now working properly for me, thanks!
>
> Great, lets see Aloxaf feedback.
>
>   Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Aloxaf Yin]

It works well. Thank you all!

Andrea Corallo <akrl@sdf.org> 于2020年10月15日周四 下午8:56写道：
>
> Andrew Whatson <whatson@gmail.com> writes:
>
> > On Thu, 15 Oct 2020 at 20:54, Andrea Corallo <akrl@sdf.org> wrote:
> >
> >> I've pushed the following.
> >>
> >> 03dfa83dc3 * Do not check eln timestamp as superseded by source hashing (bug#43532)
> >>
> >> Let us know if the issue is solved.
> >
> > Yes, this is now working properly for me, thanks!
>
> Great, lets see Aloxaf feedback.
>
>   Andrea

bug#43532: [feature/native-comp] *.eln file name hashing, algorithm doesn't seem to play nice with NATIVE_FULL_AOT and self-contained, Emacs.app builds for macOS

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Aloxaf Yin <aloxafx@gmail.com> writes:

> It works well. Thank you all!

Excellent! I'm closing then :)

  Andrea