bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Spencer Baugh via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: Ship Mints <shipmints@gmail.com>,  larsi@gnus.org,  acorallo@gnu.org,
>>    73318@debbugs.gnu.org
>> Date: Tue, 17 Sep 2024 18:31:05 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > I don't remember the details, sorry.  You are welcome to look up the
>> > past discussions in the archives.  I think they were triggered by look
>> > up of the pdumper file, but the results of that are also used by the
>> > code which decides where to look for the *.eln files.
>> 
>> I looked up /proc/self/exe in the archives and the only mention is
>> https://lists.gnu.org/archive/html/emacs-devel/2019-05/msg00951.html
>
> A more interesting discussion starts here:
>
>   https://lists.gnu.org/archive/html/emacs-devel/2019-01/msg00635.html
>
> That discussion is about finding the pdumper file, but the side effect
> of looking for pdumper file is the directory where we think the Emacs
> executable file is.  That discussion mentions several issues related
> to finding the leading directories of the Emacs executable.

Ah, this is indeed an interesting discussion.  I have to say, I agree
with Andreas Schwab when they say that argv[0] is not reliable :)

> Another useful read is here:
>
>   https://stackoverflow.com/questions/1023306/finding-current-executables-path-without-proc-self-exe
>
>> With all due humility, I think I personally am enough of an expert on
>> Linux minutiae to say that /proc/self/exe will be substantially more
>> reliable than using argv[0].
>
> And I will see your humility and raise ya.  Please describe your ideas
> for the patch before actually writing the code.  Because there's more
> here than meets the eye.  Some issues the related code needs to
> handle:
>
>   . what if /proc/self/exe is unreadable? AFAIK, on some systems you
>     need special privileges to follow its symlink
>   . what if /proc/self/exe points to a file name that is a symlink, or
>     some of its leading directories are symlinks?
>   . what if Emacs is invoked via a script which is in the correct
>     installation directory, but the actual binary the script invokes
>     is not in the expected location relative to the native-lisp/
>     directory where we have the preloaded *.eln files?
>
> The existing code handles all these cases, and some others.  We could
> perhaps _add_ the use of /proc/self/exe to what we have, but we'd need
> to be sure that it doesn't break for the above situations.
>
> I also don't understand why your script insists on removing the
> leading directories from argv[0] of Emacs.  Is there any problem for
> you to modify your script such that the leading directories would
> still be present in argv[0]?

We have a generic script, part of our packaging system, which wraps most
executables, and does "exec -a executable-name /path/to/executable".
The motivation, quoting one of the developers of said script, is:

  For context, I believe the reason why we pass `-a` is to make the prog
  more identifiable when users try to find it in the output of
  `ps`. That still sounds like the right thing to do in the majority of
  the cases.

Since this is a generic script used for everything, it's difficult to
modify it just for Emacs.

> And finally, your description of the original issue seems to omit some
> crucial details (or maybe I'm missing something):
>
>> 1. Compile and install Emacs with --with-native-compilation=aot, e.g.:
>> prefix=~/prefix
>> mkdir $prefix
>> ./configure --with-native-compilation=aot --prefix=$prefix
>> make -j64 && make install
>> 2. Run emacs with "exec -a" to change its argv[0]:
>> sh -c "exec -a emacs $prefix/bin/emacs -Q --batch"
>> 3. Observe an error like:
>> Error using execdir /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/:
>> emacs: /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/../native-lisp/31.0.50-a88a37f5/preloaded/minibuffer-b2d9c221-284ab177.eln: cannot open shared object file: No such file or directory
>
> Where did the /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/
> part come from?  I'm guessing that /usr/local/home/sbaugh/ is the
> expansion of "~" in your case, but where did the rest come from if
> your $prefix is just "~/prefix"?

Sorry, that first bit was just noise from my development environment.
Here's a re-run without the dev env vars to produce a less noisy error:

$ sh -c "exec -a emacs /home/sbaugh/prefix/bin/emacs -Q --batch"
Error using execdir /home/sbaugh/.dispatch/bin/:
emacs: /home/sbaugh/.dispatch/bin/../native-lisp/31.0.50-ef69cec6/preloaded/simple-e50f0a67-a4bb4e10.eln: cannot open shared object file: No such file or directory

> When Emacs does not find its executable file using argv[0], it assumes
> that the executable is in PATH_EXEC/../../../../bin/.  Since you are
> running an installed Emacs, that should have worked, unless you also
> somehow changed the relative path from $prefix/bin to the directory
> where the native-lisp/ directory is installed.  Why didn't it work?

Since argv[0] is "emacs", Emacs searched PATH for "emacs" and found it:

$ type emacs
emacs is /home/sbaugh/.dispatch/bin/emacs

Unfortunately, that file is not the actual Emacs executable, it's the
aforementioned generic wrapper script, which in this case ends with
this:

exec -a emacs /j/office/app/emacs/dev/bin/emacs "$@"

So Emacs looks for native-lisp in /home/sbaugh/.dispatch/bin/, but
that's wrong.  It should be looking in /j/office/app/emacs/dev/bin/.

> Bottom line: I think there are still unclear aspects of what happened
> in your case, and using /proc/self/exe to fix that is not as simple as
> it might seem, especially since we don't yet understand fully what
> failed and why.