bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Spencer Baugh via Bug reports for GNU Emacs, the Swiss army knife of text editors]

1. Compile and install Emacs with --with-native-compilation=aot, e.g.:
prefix=~/prefix
mkdir $prefix
./configure --with-native-compilation=aot --prefix=$prefix
make -j64 && make install
2. Run emacs with "exec -a" to change its argv[0]:
sh -c "exec -a emacs $prefix/bin/emacs -Q --batch"
3. Observe an error like:
Error using execdir /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/:
emacs: /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/../native-lisp/31.0.50-a88a37f5/preloaded/minibuffer-b2d9c221-284ab177.eln: cannot open shared object file: No such file or directory

"exec -a emacs" works fine for with-native-compilation=yes or
with-native-compilation=no.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Spencer Baugh via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Spencer Baugh <sbaugh@janestreet.com> writes:

> 1. Compile and install Emacs with --with-native-compilation=aot, e.g.:
> prefix=~/prefix
> mkdir $prefix
> ./configure --with-native-compilation=aot --prefix=$prefix
> make -j64 && make install
> 2. Run emacs with "exec -a" to change its argv[0]:
> sh -c "exec -a emacs $prefix/bin/emacs -Q --batch"
> 3. Observe an error like:
> Error using execdir /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/:
> emacs: /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/../native-lisp/31.0.50-a88a37f5/preloaded/minibuffer-b2d9c221-284ab177.eln: cannot open shared object file: No such file or directory
>
> "exec -a emacs" works fine for with-native-compilation=yes or
> with-native-compilation=no.

Actually this also breaks with with-native-compilation=yes

It seems sensitive to cached compilation or configure results; I can
only get a reliable failure if I make clean in between attempts.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> Cc: Lars Ingebrigtsen <larsi@gnus.org>, Andrea Corallo <acorallo@gnu.org>
> Date: Tue, 17 Sep 2024 11:18:41 -0400
> From:  Spencer Baugh via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> 
> 1. Compile and install Emacs with --with-native-compilation=aot, e.g.:
> prefix=~/prefix
> mkdir $prefix
> ./configure --with-native-compilation=aot --prefix=$prefix
> make -j64 && make install
> 2. Run emacs with "exec -a" to change its argv[0]:
> sh -c "exec -a emacs $prefix/bin/emacs -Q --batch"
> 3. Observe an error like:
> Error using execdir /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/:
> emacs: /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/../native-lisp/31.0.50-a88a37f5/preloaded/minibuffer-b2d9c221-284ab177.eln: cannot open shared object file: No such file or directory
> 
> "exec -a emacs" works fine for with-native-compilation=yes or
> with-native-compilation=no.

Invocation via "exec -a" is not supported, if it messes with the
leading directories of the argv[0] value passed to Emacs.  That's
because the search for the preloaded *.eln files is based on the
directory in which the Emacs executable is installed, as passed via
argv[0], and breaks if "exec -a" messes with that.

IOW, "don't do that, it will hurt".

P.S. If someone knows how to teach Emacs how to find the absolute file
name of its executable without depending on argv[0], speak up.  We do
that on Windows, but not on Posix platforms, since (I'm being told)
there's no reliable way of having that on GNU/Linux and other Posix
platforms.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 2097 bytes --]

Google's build system, gn, used for Chromium, et.al., has support for this
which can provide some inspiration for most of the major platforms on which
Emacs runs; qv
https://gn.googlesource.com/gn/+/refs/heads/main/src/util/exe_path.cc

On Tue, Sep 17, 2024 at 1:51 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > Cc: Lars Ingebrigtsen <larsi@gnus.org>, Andrea Corallo <acorallo@gnu.org
> >
> > Date: Tue, 17 Sep 2024 11:18:41 -0400
> > From:  Spencer Baugh via "Bug reports for GNU Emacs,
> >  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> >
> >
> > 1. Compile and install Emacs with --with-native-compilation=aot, e.g.:
> > prefix=~/prefix
> > mkdir $prefix
> > ./configure --with-native-compilation=aot --prefix=$prefix
> > make -j64 && make install
> > 2. Run emacs with "exec -a" to change its argv[0]:
> > sh -c "exec -a emacs $prefix/bin/emacs -Q --batch"
> > 3. Observe an error like:
> > Error using execdir
> /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/:
> > emacs:
> /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/../native-lisp/31.0.50-a88a37f5/preloaded/minibuffer-b2d9c221-284ab177.eln:
> cannot open shared object file: No such file or directory
> >
> > "exec -a emacs" works fine for with-native-compilation=yes or
> > with-native-compilation=no.
>
> Invocation via "exec -a" is not supported, if it messes with the
> leading directories of the argv[0] value passed to Emacs.  That's
> because the search for the preloaded *.eln files is based on the
> directory in which the Emacs executable is installed, as passed via
> argv[0], and breaks if "exec -a" messes with that.
>
> IOW, "don't do that, it will hurt".
>
> P.S. If someone knows how to teach Emacs how to find the absolute file
> name of its executable without depending on argv[0], speak up.  We do
> that on Windows, but not on Posix platforms, since (I'm being told)
> there's no reliable way of having that on GNU/Linux and other Posix
> platforms.
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 2944 bytes --]

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Ship Mints <shipmints@gmail.com>
> Date: Tue, 17 Sep 2024 14:14:54 -0400
> Cc: Spencer Baugh <sbaugh@janestreet.com>, 73318@debbugs.gnu.org, larsi@gnus.org, 
> 	acorallo@gnu.org
> 
> Google's build system, gn, used for Chromium, et.al., has support for this which can provide some inspiration
> for most of the major platforms on which Emacs runs; qv
> https://gn.googlesource.com/gn/+/refs/heads/main/src/util/exe_path.cc

Thanks.  That's exactly the method that I was told was unreliable on
GNU/Linux.  I don't know enough to argue with people who said that
about this particular feature.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 1020 bytes --]

I think the kind of "unreliability" in question is, for example, when a
process starts and unlinks itself. I doubt Emacs will ever do this. Using
the proc file system is "technically" unreliable, unable to cover 100% of
all potential cases, but is practically reliable, especially in this case.

On Tue, Sep 17, 2024 at 3:07 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Ship Mints <shipmints@gmail.com>
> > Date: Tue, 17 Sep 2024 14:14:54 -0400
> > Cc: Spencer Baugh <sbaugh@janestreet.com>, 73318@debbugs.gnu.org,
> larsi@gnus.org,
> >       acorallo@gnu.org
> >
> > Google's build system, gn, used for Chromium, et.al., has support for
> this which can provide some inspiration
> > for most of the major platforms on which Emacs runs; qv
> > https://gn.googlesource.com/gn/+/refs/heads/main/src/util/exe_path.cc
>
> Thanks.  That's exactly the method that I was told was unreliable on
> GNU/Linux.  I don't know enough to argue with people who said that
> about this particular feature.
>

[-- Attachment #2: Type: text/html, Size: 1899 bytes --]

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Ship Mints <shipmints@gmail.com>
> Date: Tue, 17 Sep 2024 15:22:18 -0400
> Cc: sbaugh@janestreet.com, 73318@debbugs.gnu.org, larsi@gnus.org, 
> 	acorallo@gnu.org
> I think the kind of "unreliability" in question is, for example, when a process starts and unlinks itself. I doubt
> Emacs will ever do this. Using the proc file system is "technically" unreliable, unable to cover 100% of all
> potential cases, but is practically reliable, especially in this case.

I don't remember the details, sorry.  You are welcome to look up the
past discussions in the archives.  I think they were triggered by look
up of the pdumper file, but the results of that are also used by the
code which decides where to look for the *.eln files.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Spencer Baugh via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ship Mints <shipmints@gmail.com>
>> Date: Tue, 17 Sep 2024 15:22:18 -0400
>> Cc: sbaugh@janestreet.com, 73318@debbugs.gnu.org, larsi@gnus.org, 
>> 	acorallo@gnu.org
>> I think the kind of "unreliability" in question is, for example, when a process starts and unlinks itself. I doubt
>> Emacs will ever do this. Using the proc file system is "technically" unreliable, unable to cover 100% of all
>> potential cases, but is practically reliable, especially in this case.
>
> I don't remember the details, sorry.  You are welcome to look up the
> past discussions in the archives.  I think they were triggered by look
> up of the pdumper file, but the results of that are also used by the
> code which decides where to look for the *.eln files.

I looked up /proc/self/exe in the archives and the only mention is
https://lists.gnu.org/archive/html/emacs-devel/2019-05/msg00951.html

Gnulib uses /proc/self/exe to provide support for relocatability (in
progreloc.c).  If it's reliable enough for Gnulib, it should be reliable
enough for Emacs.

With all due humility, I think I personally am enough of an expert on
Linux minutiae to say that /proc/self/exe will be substantially more
reliable than using argv[0].

I can provide a patch to make invocation-directory use /proc/self/exe,
why don't we just try installing it on master?  If it's worse, we should
learn soon enough.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 1801 bytes --]

I agree in the main. This thread looks relevant to pdump concerns. I have
no experience making my own pdumps as some seem to do.

https://lists.gnu.org/archive/html/emacs-devel/2019-01/msg00558.html

On Tue, Sep 17, 2024 at 6:31 PM Spencer Baugh <sbaugh@janestreet.com> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> From: Ship Mints <shipmints@gmail.com>
> >> Date: Tue, 17 Sep 2024 15:22:18 -0400
> >> Cc: sbaugh@janestreet.com, 73318@debbugs.gnu.org, larsi@gnus.org,
> >>      acorallo@gnu.org
> >> I think the kind of "unreliability" in question is, for example, when a
> process starts and unlinks itself. I doubt
> >> Emacs will ever do this. Using the proc file system is "technically"
> unreliable, unable to cover 100% of all
> >> potential cases, but is practically reliable, especially in this case.
> >
> > I don't remember the details, sorry.  You are welcome to look up the
> > past discussions in the archives.  I think they were triggered by look
> > up of the pdumper file, but the results of that are also used by the
> > code which decides where to look for the *.eln files.
>
> I looked up /proc/self/exe in the archives and the only mention is
> https://lists.gnu.org/archive/html/emacs-devel/2019-05/msg00951.html
>
> Gnulib uses /proc/self/exe to provide support for relocatability (in
> progreloc.c).  If it's reliable enough for Gnulib, it should be reliable
> enough for Emacs.
>
> With all due humility, I think I personally am enough of an expert on
> Linux minutiae to say that /proc/self/exe will be substantially more
> reliable than using argv[0].
>
> I can provide a patch to make invocation-directory use /proc/self/exe,
> why don't we just try installing it on master?  If it's worse, we should
> learn soon enough.
>

[-- Attachment #2: Type: text/html, Size: 2964 bytes --]

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: Ship Mints <shipmints@gmail.com>,  larsi@gnus.org,  acorallo@gnu.org,
>    73318@debbugs.gnu.org
> Date: Tue, 17 Sep 2024 18:31:05 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I don't remember the details, sorry.  You are welcome to look up the
> > past discussions in the archives.  I think they were triggered by look
> > up of the pdumper file, but the results of that are also used by the
> > code which decides where to look for the *.eln files.
> 
> I looked up /proc/self/exe in the archives and the only mention is
> https://lists.gnu.org/archive/html/emacs-devel/2019-05/msg00951.html

A more interesting discussion starts here:

  https://lists.gnu.org/archive/html/emacs-devel/2019-01/msg00635.html

That discussion is about finding the pdumper file, but the side effect
of looking for pdumper file is the directory where we think the Emacs
executable file is.  That discussion mentions several issues related
to finding the leading directories of the Emacs executable.

Another useful read is here:

  https://stackoverflow.com/questions/1023306/finding-current-executables-path-without-proc-self-exe

> With all due humility, I think I personally am enough of an expert on
> Linux minutiae to say that /proc/self/exe will be substantially more
> reliable than using argv[0].

And I will see your humility and raise ya.  Please describe your ideas
for the patch before actually writing the code.  Because there's more
here than meets the eye.  Some issues the related code needs to
handle:

  . what if /proc/self/exe is unreadable? AFAIK, on some systems you
    need special privileges to follow its symlink
  . what if /proc/self/exe points to a file name that is a symlink, or
    some of its leading directories are symlinks?
  . what if Emacs is invoked via a script which is in the correct
    installation directory, but the actual binary the script invokes
    is not in the expected location relative to the native-lisp/
    directory where we have the preloaded *.eln files?

The existing code handles all these cases, and some others.  We could
perhaps _add_ the use of /proc/self/exe to what we have, but we'd need
to be sure that it doesn't break for the above situations.

I also don't understand why your script insists on removing the
leading directories from argv[0] of Emacs.  Is there any problem for
you to modify your script such that the leading directories would
still be present in argv[0]?

And finally, your description of the original issue seems to omit some
crucial details (or maybe I'm missing something):

> 1. Compile and install Emacs with --with-native-compilation=aot, e.g.:
> prefix=~/prefix
> mkdir $prefix
> ./configure --with-native-compilation=aot --prefix=$prefix
> make -j64 && make install
> 2. Run emacs with "exec -a" to change its argv[0]:
> sh -c "exec -a emacs $prefix/bin/emacs -Q --batch"
> 3. Observe an error like:
> Error using execdir /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/:
> emacs: /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/../native-lisp/31.0.50-a88a37f5/preloaded/minibuffer-b2d9c221-284ab177.eln: cannot open shared object file: No such file or directory

Where did the /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/
part come from?  I'm guessing that /usr/local/home/sbaugh/ is the
expansion of "~" in your case, but where did the rest come from if
your $prefix is just "~/prefix"?

When Emacs does not find its executable file using argv[0], it assumes
that the executable is in PATH_EXEC/../../../../bin/.  Since you are
running an installed Emacs, that should have worked, unless you also
somehow changed the relative path from $prefix/bin to the directory
where the native-lisp/ directory is installed.  Why didn't it work?

Bottom line: I think there are still unclear aspects of what happened
in your case, and using /proc/self/exe to fix that is not as simple as
it might seem, especially since we don't yet understand fully what
failed and why.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> And I will see your humility and raise ya.  Please describe your ideas
> for the patch before actually writing the code.  Because there's more
> here than meets the eye.  Some issues the related code needs to
> handle:
>
>   . what if /proc/self/exe is unreadable? AFAIK, on some systems you
>     need special privileges to follow its symlink

Above all, /proc is liable simply to be unmounted.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

Even if you umount /proc, it will remain until active process references to
/proc nodes are closed. If one tries umount /proc outside of a shutdown
sequence, Emacs is the least of her worries. This is not a practical
deterrent.

On Wed, Sep 18, 2024 at 11:09 PM Po Lu <luangruo@yahoo.com> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> > And I will see your humility and raise ya.  Please describe your ideas
> > for the patch before actually writing the code.  Because there's more
> > here than meets the eye.  Some issues the related code needs to
> > handle:
> >
> >   . what if /proc/self/exe is unreadable? AFAIK, on some systems you
> >     need special privileges to follow its symlink
>
> Above all, /proc is liable simply to be unmounted.
>

[-- Attachment #2: Type: text/html, Size: 1222 bytes --]

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Ship Mints <shipmints@gmail.com> writes:

> Even if you umount /proc, it will remain until active process
> references to /proc nodes are closed. If one tries umount /proc
> outside of a shutdown sequence, Emacs is the least of her
> worries. This is not a practical deterrent.

It is possible to run systems with Linux (the kernel) without mounting
/proc at all, and Emacs is very much interested in functioning correctly
there.  Moreover, the value of /proc/self/exe is sometimes completely
meaningless, as on Android, where Emacs is loaded into
/system/bin/app_process64 as a shared library.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

I see. It pays, then, to disambiguate between a Linux "system" (which is
the typical Emacs case) and Linux O/S-based, essentially embedded,
environments. In the "system" case, /proc/self/exe should be the
recommended method, I'd think. In embedded cases, whatever is appropriate
in those environments is what should be used.

On Thu, Sep 19, 2024 at 9:44 AM Po Lu <luangruo@yahoo.com> wrote:

> Ship Mints <shipmints@gmail.com> writes:
>
> > Even if you umount /proc, it will remain until active process
> > references to /proc nodes are closed. If one tries umount /proc
> > outside of a shutdown sequence, Emacs is the least of her
> > worries. This is not a practical deterrent.
>
> It is possible to run systems with Linux (the kernel) without mounting
> /proc at all, and Emacs is very much interested in functioning correctly
> there.  Moreover, the value of /proc/self/exe is sometimes completely
> meaningless, as on Android, where Emacs is loaded into
> /system/bin/app_process64 as a shared library.
>
>

[-- Attachment #2: Type: text/html, Size: 1497 bytes --]

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Ship Mints <shipmints@gmail.com>
> Date: Thu, 19 Sep 2024 09:51:44 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, Spencer Baugh <sbaugh@janestreet.com>, 73318@debbugs.gnu.org, 
> 	larsi@gnus.org, acorallo@gnu.org
> 
> I see. It pays, then, to disambiguate between a Linux "system" (which is the typical Emacs case) and Linux
> O/S-based, essentially embedded, environments. In the "system" case, /proc/self/exe should be the
> recommended method, I'd think. In embedded cases, whatever is appropriate in those environments is what
> should be used.

Please re-read what I wrote about the use cases we have to support and
the discussion to which I pointed.  No single method can support all
the situations which Emacs needs to support.  So there's no "the
recommended method": even if we add /proc/self/exe, it can only be one
of the methods we use to find where the *.eln files are, and not
necessarily the most important one.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Spencer Baugh via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: Ship Mints <shipmints@gmail.com>,  larsi@gnus.org,  acorallo@gnu.org,
>>    73318@debbugs.gnu.org
>> Date: Tue, 17 Sep 2024 18:31:05 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > I don't remember the details, sorry.  You are welcome to look up the
>> > past discussions in the archives.  I think they were triggered by look
>> > up of the pdumper file, but the results of that are also used by the
>> > code which decides where to look for the *.eln files.
>> 
>> I looked up /proc/self/exe in the archives and the only mention is
>> https://lists.gnu.org/archive/html/emacs-devel/2019-05/msg00951.html
>
> A more interesting discussion starts here:
>
>   https://lists.gnu.org/archive/html/emacs-devel/2019-01/msg00635.html
>
> That discussion is about finding the pdumper file, but the side effect
> of looking for pdumper file is the directory where we think the Emacs
> executable file is.  That discussion mentions several issues related
> to finding the leading directories of the Emacs executable.

Ah, this is indeed an interesting discussion.  I have to say, I agree
with Andreas Schwab when they say that argv[0] is not reliable :)

> Another useful read is here:
>
>   https://stackoverflow.com/questions/1023306/finding-current-executables-path-without-proc-self-exe
>
>> With all due humility, I think I personally am enough of an expert on
>> Linux minutiae to say that /proc/self/exe will be substantially more
>> reliable than using argv[0].
>
> And I will see your humility and raise ya.  Please describe your ideas
> for the patch before actually writing the code.  Because there's more
> here than meets the eye.  Some issues the related code needs to
> handle:
>
>   . what if /proc/self/exe is unreadable? AFAIK, on some systems you
>     need special privileges to follow its symlink
>   . what if /proc/self/exe points to a file name that is a symlink, or
>     some of its leading directories are symlinks?
>   . what if Emacs is invoked via a script which is in the correct
>     installation directory, but the actual binary the script invokes
>     is not in the expected location relative to the native-lisp/
>     directory where we have the preloaded *.eln files?
>
> The existing code handles all these cases, and some others.  We could
> perhaps _add_ the use of /proc/self/exe to what we have, but we'd need
> to be sure that it doesn't break for the above situations.
>
> I also don't understand why your script insists on removing the
> leading directories from argv[0] of Emacs.  Is there any problem for
> you to modify your script such that the leading directories would
> still be present in argv[0]?

We have a generic script, part of our packaging system, which wraps most
executables, and does "exec -a executable-name /path/to/executable".
The motivation, quoting one of the developers of said script, is:

  For context, I believe the reason why we pass `-a` is to make the prog
  more identifiable when users try to find it in the output of
  `ps`. That still sounds like the right thing to do in the majority of
  the cases.

Since this is a generic script used for everything, it's difficult to
modify it just for Emacs.

> And finally, your description of the original issue seems to omit some
> crucial details (or maybe I'm missing something):
>
>> 1. Compile and install Emacs with --with-native-compilation=aot, e.g.:
>> prefix=~/prefix
>> mkdir $prefix
>> ./configure --with-native-compilation=aot --prefix=$prefix
>> make -j64 && make install
>> 2. Run emacs with "exec -a" to change its argv[0]:
>> sh -c "exec -a emacs $prefix/bin/emacs -Q --batch"
>> 3. Observe an error like:
>> Error using execdir /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/:
>> emacs: /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/../native-lisp/31.0.50-a88a37f5/preloaded/minibuffer-b2d9c221-284ab177.eln: cannot open shared object file: No such file or directory
>
> Where did the /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/
> part come from?  I'm guessing that /usr/local/home/sbaugh/ is the
> expansion of "~" in your case, but where did the rest come from if
> your $prefix is just "~/prefix"?

Sorry, that first bit was just noise from my development environment.
Here's a re-run without the dev env vars to produce a less noisy error:

$ sh -c "exec -a emacs /home/sbaugh/prefix/bin/emacs -Q --batch"
Error using execdir /home/sbaugh/.dispatch/bin/:
emacs: /home/sbaugh/.dispatch/bin/../native-lisp/31.0.50-ef69cec6/preloaded/simple-e50f0a67-a4bb4e10.eln: cannot open shared object file: No such file or directory

> When Emacs does not find its executable file using argv[0], it assumes
> that the executable is in PATH_EXEC/../../../../bin/.  Since you are
> running an installed Emacs, that should have worked, unless you also
> somehow changed the relative path from $prefix/bin to the directory
> where the native-lisp/ directory is installed.  Why didn't it work?

Since argv[0] is "emacs", Emacs searched PATH for "emacs" and found it:

$ type emacs
emacs is /home/sbaugh/.dispatch/bin/emacs

Unfortunately, that file is not the actual Emacs executable, it's the
aforementioned generic wrapper script, which in this case ends with
this:

exec -a emacs /j/office/app/emacs/dev/bin/emacs "$@"

So Emacs looks for native-lisp in /home/sbaugh/.dispatch/bin/, but
that's wrong.  It should be looking in /j/office/app/emacs/dev/bin/.

> Bottom line: I think there are still unclear aspects of what happened
> in your case, and using /proc/self/exe to fix that is not as simple as
> it might seem, especially since we don't yet understand fully what
> failed and why.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 73318@debbugs.gnu.org,  larsi@gnus.org,  acorallo@gnu.org,
>    shipmints@gmail.com, schwab@linux-m68k.org
> Date: Fri, 04 Oct 2024 08:09:59 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > A more interesting discussion starts here:
> >
> >   https://lists.gnu.org/archive/html/emacs-devel/2019-01/msg00635.html
> >
> > That discussion is about finding the pdumper file, but the side effect
> > of looking for pdumper file is the directory where we think the Emacs
> > executable file is.  That discussion mentions several issues related
> > to finding the leading directories of the Emacs executable.
> 
> Ah, this is indeed an interesting discussion.  I have to say, I agree
> with Andreas Schwab when they say that argv[0] is not reliable :)

For some reason that evades me, you take only part of the discussion
and ignore the rest.

The specific issue with finding the pdumper file more reliably is
solved in Emacs 30 in the way Andreas suggested, but it has no effect
on the problem which you describe, with finding the preloaded *.eln
files.

It is true that argv[0] alone is not reliable enough, which is why we
use other techniques when argv[0] cannot help.  But /proc/self/exe is
not reliable enough, either.

> >   . what if /proc/self/exe is unreadable? AFAIK, on some systems you
> >     need special privileges to follow its symlink
> >   . what if /proc/self/exe points to a file name that is a symlink, or
> >     some of its leading directories are symlinks?
> >   . what if Emacs is invoked via a script which is in the correct
> >     installation directory, but the actual binary the script invokes
> >     is not in the expected location relative to the native-lisp/
> >     directory where we have the preloaded *.eln files?
> >
> > The existing code handles all these cases, and some others.  We could
> > perhaps _add_ the use of /proc/self/exe to what we have, but we'd need
> > to be sure that it doesn't break for the above situations.

I'm still waiting for some answers to these.

> > I also don't understand why your script insists on removing the
> > leading directories from argv[0] of Emacs.  Is there any problem for
> > you to modify your script such that the leading directories would
> > still be present in argv[0]?
> 
> We have a generic script, part of our packaging system, which wraps most
> executables, and does "exec -a executable-name /path/to/executable".
> The motivation, quoting one of the developers of said script, is:
> 
>   For context, I believe the reason why we pass `-a` is to make the prog
>   more identifiable when users try to find it in the output of
>   `ps`. That still sounds like the right thing to do in the majority of
>   the cases.
> 
> Since this is a generic script used for everything, it's difficult to
> modify it just for Emacs.

Why cannot you modify the script for all the commands to include the
leading directories in executable-name?  That is all that is needed
for Emacs to find its *.eln files.

> $ sh -c "exec -a emacs /home/sbaugh/prefix/bin/emacs -Q --batch"
> Error using execdir /home/sbaugh/.dispatch/bin/:
> emacs: /home/sbaugh/.dispatch/bin/../native-lisp/31.0.50-ef69cec6/preloaded/simple-e50f0a67-a4bb4e10.eln: cannot open shared object file: No such file or directory
> 
> > When Emacs does not find its executable file using argv[0], it assumes
> > that the executable is in PATH_EXEC/../../../../bin/.  Since you are
> > running an installed Emacs, that should have worked, unless you also
> > somehow changed the relative path from $prefix/bin to the directory
> > where the native-lisp/ directory is installed.  Why didn't it work?
> 
> Since argv[0] is "emacs", Emacs searched PATH for "emacs" and found it:
> 
> $ type emacs
> emacs is /home/sbaugh/.dispatch/bin/emacs
> 
> Unfortunately, that file is not the actual Emacs executable, it's the
> aforementioned generic wrapper script, which in this case ends with
> this:
> 
> exec -a emacs /j/office/app/emacs/dev/bin/emacs "$@"
> 
> So Emacs looks for native-lisp in /home/sbaugh/.dispatch/bin/, but
> that's wrong.  It should be looking in /j/office/app/emacs/dev/bin/.

If you include the leading directories in the "-a NAME" switch to
'exec', the problem will be solved.  Since the script evidently knows
the exact absolute file name of the program it invokes, there should
be no problem including that absolute file name in the NAME argument
of the -a option.

> > Bottom line: I think there are still unclear aspects of what happened
> > in your case, and using /proc/self/exe to fix that is not as simple as
> > it might seem, especially since we don't yet understand fully what
> > failed and why.

Using /proc/self/exe to fix this kind of situations is not trivial,
due to the issues with /proc/self/exe I mentioned in my previous
email.  So my suggestion would be first to try to adapt your generic
script to the Emacs expectations, namely, to make sure argv[0] of
Emacs includes the leading directories of the real installation tree.
This is by far the simplest solution, and will not require any changes
in Emacs.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Spencer Baugh via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: 73318@debbugs.gnu.org,  larsi@gnus.org,  acorallo@gnu.org,
>>    shipmints@gmail.com, schwab@linux-m68k.org
>> Date: Fri, 04 Oct 2024 08:09:59 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > A more interesting discussion starts here:
>> >
>> >   https://lists.gnu.org/archive/html/emacs-devel/2019-01/msg00635.html
>> >
>> > That discussion is about finding the pdumper file, but the side effect
>> > of looking for pdumper file is the directory where we think the Emacs
>> > executable file is.  That discussion mentions several issues related
>> > to finding the leading directories of the Emacs executable.
>> 
>> Ah, this is indeed an interesting discussion.  I have to say, I agree
>> with Andreas Schwab when they say that argv[0] is not reliable :)
>
> For some reason that evades me, you take only part of the discussion
> and ignore the rest.
>
> The specific issue with finding the pdumper file more reliably is
> solved in Emacs 30 in the way Andreas suggested, but it has no effect
> on the problem which you describe, with finding the preloaded *.eln
> files.

Ah, interesting.  So Emacs 30 falls back on looking up the pdmp in
PATH_EXEC, a path compiled into the Emacs binary.

Should we perhaps do the same for the native-lisp directory?  If we
can't find it in other ways, look it up relative to a path compiled into
the Emacs binary?  I don't know if that should be PATH_EXEC or some
other path.

That would work on my system.  Then we wouldn't need to use
/proc/self/exe at all.

Might that be the best solution?

> It is true that argv[0] alone is not reliable enough, which is why we
> use other techniques when argv[0] cannot help.  But /proc/self/exe is
> not reliable enough, either.

Yes, I'm persuaded now that if we use /proc/self/exe, we should also use
argv[0].  As Po Lu said, /proc can be unmounted, so we must have some
other mechanism besides just /proc/self/exe.

>> >   . what if /proc/self/exe is unreadable? AFAIK, on some systems you
>> >     need special privileges to follow its symlink
>> >   . what if /proc/self/exe points to a file name that is a symlink, or
>> >     some of its leading directories are symlinks?
>> >   . what if Emacs is invoked via a script which is in the correct
>> >     installation directory, but the actual binary the script invokes
>> >     is not in the expected location relative to the native-lisp/
>> >     directory where we have the preloaded *.eln files?
>> >
>> > The existing code handles all these cases, and some others.  We could
>> > perhaps _add_ the use of /proc/self/exe to what we have, but we'd need
>> > to be sure that it doesn't break for the above situations.
>
> I'm still waiting for some answers to these.

If we use /proc/self/exe, I'm fine with it being a fallback if all other
mechanisms fail.  That should make these cases still work fine, right?

>> > I also don't understand why your script insists on removing the
>> > leading directories from argv[0] of Emacs.  Is there any problem for
>> > you to modify your script such that the leading directories would
>> > still be present in argv[0]?
>> 
>> We have a generic script, part of our packaging system, which wraps most
>> executables, and does "exec -a executable-name /path/to/executable".
>> The motivation, quoting one of the developers of said script, is:
>> 
>>   For context, I believe the reason why we pass `-a` is to make the prog
>>   more identifiable when users try to find it in the output of
>>   `ps`. That still sounds like the right thing to do in the majority of
>>   the cases.
>> 
>> Since this is a generic script used for everything, it's difficult to
>> modify it just for Emacs.
>
> Why cannot you modify the script for all the commands to include the
> leading directories in executable-name?  That is all that is needed
> for Emacs to find its *.eln files.

See the motivation that I quoted above:

 For context, I believe the reason why we pass `-a` is to make the prog
 more identifiable when users try to find it in the output of
 `ps`. That still sounds like the right thing to do in the majority of
 the cases.

Including the leading directories would make them show up in the output
of "ps", which is uglier.

I realize this might not seem like an important justification, but it
works for every other program we run, and has worked for decades.  And
other distributors might be doing this too, so I think it's reasonable
to make Emacs robust to this by having it fall back to looking up
native-lisp in something like PATH_EXEC.

>> $ sh -c "exec -a emacs /home/sbaugh/prefix/bin/emacs -Q --batch"
>> Error using execdir /home/sbaugh/.dispatch/bin/:
>> emacs: /home/sbaugh/.dispatch/bin/../native-lisp/31.0.50-ef69cec6/preloaded/simple-e50f0a67-a4bb4e10.eln: cannot open shared object file: No such file or directory
>> 
>> > When Emacs does not find its executable file using argv[0], it assumes
>> > that the executable is in PATH_EXEC/../../../../bin/.  Since you are
>> > running an installed Emacs, that should have worked, unless you also
>> > somehow changed the relative path from $prefix/bin to the directory
>> > where the native-lisp/ directory is installed.  Why didn't it work?
>> 
>> Since argv[0] is "emacs", Emacs searched PATH for "emacs" and found it:
>> 
>> $ type emacs
>> emacs is /home/sbaugh/.dispatch/bin/emacs
>> 
>> Unfortunately, that file is not the actual Emacs executable, it's the
>> aforementioned generic wrapper script, which in this case ends with
>> this:
>> 
>> exec -a emacs /j/office/app/emacs/dev/bin/emacs "$@"
>> 
>> So Emacs looks for native-lisp in /home/sbaugh/.dispatch/bin/, but
>> that's wrong.  It should be looking in /j/office/app/emacs/dev/bin/.
>
> If you include the leading directories in the "-a NAME" switch to
> 'exec', the problem will be solved.  Since the script evidently knows
> the exact absolute file name of the program it invokes, there should
> be no problem including that absolute file name in the NAME argument
> of the -a option.
>
>> > Bottom line: I think there are still unclear aspects of what happened
>> > in your case, and using /proc/self/exe to fix that is not as simple as
>> > it might seem, especially since we don't yet understand fully what
>> > failed and why.
>
> Using /proc/self/exe to fix this kind of situations is not trivial,
> due to the issues with /proc/self/exe I mentioned in my previous
> email.  So my suggestion would be first to try to adapt your generic
> script to the Emacs expectations, namely, to make sure argv[0] of
> Emacs includes the leading directories of the real installation tree.
> This is by far the simplest solution, and will not require any changes
> in Emacs.

Sure, see the motivation I quoted above.

But now I think we maybe don't need to use /proc/self/exe at all, and
can just have Emacs fall back on something like PATH_EXEC when it fails
to find the native lisp files.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 73318@debbugs.gnu.org,  larsi@gnus.org,  acorallo@gnu.org,
>    schwab@linux-m68k.org,  shipmints@gmail.com
> Date: Fri, 04 Oct 2024 09:22:08 -0400
> 
> > The specific issue with finding the pdumper file more reliably is
> > solved in Emacs 30 in the way Andreas suggested, but it has no effect
> > on the problem which you describe, with finding the preloaded *.eln
> > files.
> 
> Ah, interesting.  So Emacs 30 falls back on looking up the pdmp in
> PATH_EXEC, a path compiled into the Emacs binary.
> 
> Should we perhaps do the same for the native-lisp directory?

We already do, sort of.  We just rely on the search for the pdumper
file to supply us with the answer.  See below.

> If we can't find it in other ways, look it up relative to a path
> compiled into the Emacs binary?  I don't know if that should be
> PATH_EXEC or some other path.
> 
> That would work on my system.  Then we wouldn't need to use
> /proc/self/exe at all.
> 
> Might that be the best solution?

It isn't that easy.  We need to support 2 possible locations for the
preloaded *.eln files: in the source tree and in the installation
tree.  You now want us to look in yet a third place.  Take a look at
dump_do_dump_relocation where we look for the *.eln files.  What you
suggest is to add a third place to that code.  Currently, we infer
where the *.eln files are by looking at the file name of the Emacs
executable file.  And your script violates this protocol by removing
the leading directories from argv[0] and placing Emacs in a directory
not on PATH.

> >> >   . what if /proc/self/exe is unreadable? AFAIK, on some systems you
> >> >     need special privileges to follow its symlink
> >> >   . what if /proc/self/exe points to a file name that is a symlink, or
> >> >     some of its leading directories are symlinks?
> >> >   . what if Emacs is invoked via a script which is in the correct
> >> >     installation directory, but the actual binary the script invokes
> >> >     is not in the expected location relative to the native-lisp/
> >> >     directory where we have the preloaded *.eln files?
> >> >
> >> > The existing code handles all these cases, and some others.  We could
> >> > perhaps _add_ the use of /proc/self/exe to what we have, but we'd need
> >> > to be sure that it doesn't break for the above situations.
> >
> > I'm still waiting for some answers to these.
> 
> If we use /proc/self/exe, I'm fine with it being a fallback if all other
> mechanisms fail.  That should make these cases still work fine, right?

To use /proc/self/exe as fallback, we need to somehow reject the Emacs
"executable" found in your case.  Because otherwise fallbacks are only
used if the previous attempts fail, and the attempt to find atgv[0]
along PATH does not fail in your case, it just finds the "wrong" file.

> > Why cannot you modify the script for all the commands to include the
> > leading directories in executable-name?  That is all that is needed
> > for Emacs to find its *.eln files.
> 
> See the motivation that I quoted above:
> 
>  For context, I believe the reason why we pass `-a` is to make the prog
>  more identifiable when users try to find it in the output of
>  `ps`. That still sounds like the right thing to do in the majority of
>  the cases.
> 
> Including the leading directories would make them show up in the output
> of "ps", which is uglier.

"Uglier" is in the eyes of the beholder.  The important thing is that
it solves the problem without any other changes, and will then use
code which was tested and validated by many users.

> I realize this might not seem like an important justification, but it
> works for every other program we run, and has worked for decades.  And
> other distributors might be doing this too, so I think it's reasonable
> to make Emacs robust to this by having it fall back to looking up
> native-lisp in something like PATH_EXEC.

I disagree that this is reasonable, and what you suggest is not
possible/practical anyway.

> But now I think we maybe don't need to use /proc/self/exe at all, and
> can just have Emacs fall back on something like PATH_EXEC when it fails
> to find the native lisp files.

See above.  If you insist to go this much more complicated way, we'll
need to modify the code in dump_do_dump_relocation to use PATH_EXEC,
and deal with possible false positives.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[shipmints]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

On Fri, Oct 4, 2024 at 9:58 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Spencer Baugh <sbaugh@janestreet.com>
> > Cc: 73318@debbugs.gnu.org,  larsi@gnus.org,  acorallo@gnu.org,
> >    schwab@linux-m68k.org,  shipmints@gmail.com
> > Date: Fri, 04 Oct 2024 09:22:08 -0400
> >
> > Including the leading directories would make them show up in the output
> > of "ps", which is uglier.
>
> "Uglier" is in the eyes of the beholder.  The important thing is that
> it solves the problem without any other changes, and will then use
> code which was tested and validated by many users.
>

I prefer the "determinism" (even if pseudo determinism, as binaries can
"lie") of fully-qualified path names in ps output. I'll take ugly over
mystery.

[-- Attachment #2: Type: text/html, Size: 1631 bytes --]

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Spencer Baugh via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: 73318@debbugs.gnu.org,  larsi@gnus.org,  acorallo@gnu.org,
>>    schwab@linux-m68k.org,  shipmints@gmail.com
>> Date: Fri, 04 Oct 2024 09:22:08 -0400
>> 
>> > The specific issue with finding the pdumper file more reliably is
>> > solved in Emacs 30 in the way Andreas suggested, but it has no effect
>> > on the problem which you describe, with finding the preloaded *.eln
>> > files.
>> 
>> Ah, interesting.  So Emacs 30 falls back on looking up the pdmp in
>> PATH_EXEC, a path compiled into the Emacs binary.
>> 
>> Should we perhaps do the same for the native-lisp directory?
>
> We already do, sort of.  We just rely on the search for the pdumper
> file to supply us with the answer.  See below.
>
>> If we can't find it in other ways, look it up relative to a path
>> compiled into the Emacs binary?  I don't know if that should be
>> PATH_EXEC or some other path.
>> 
>> That would work on my system.  Then we wouldn't need to use
>> /proc/self/exe at all.
>> 
>> Might that be the best solution?
>
> It isn't that easy.  We need to support 2 possible locations for the
> preloaded *.eln files: in the source tree and in the installation
> tree.  You now want us to look in yet a third place.  Take a look at
> dump_do_dump_relocation where we look for the *.eln files.  What you
> suggest is to add a third place to that code.

Yes.  I suggest that if installation_state == INSTALLED, and we fail to
find an eln file in emacs_execdir, we fall back to look in PATH_EXEC (or
some other path compiled into the Emacs binary).

But it occurs to me that there's another possible solution.  We already
have a robust way to find the pdump.  My Emacs finds the pdump and loads
it just fine.  Maybe we could look for the eln files relative to the
pdump?

So the two solutions that seem possible right now are:

A. If we fail to find an eln file relative to the Emacs executable, fall
   back to PATH_EXEC (or something similar)

B. Always look for eln files at a path relative to the pdmp, instead of
   relative to the Emacs executable.

>> >> >   . what if /proc/self/exe is unreadable? AFAIK, on some systems you
>> >> >     need special privileges to follow its symlink
>> >> >   . what if /proc/self/exe points to a file name that is a symlink, or
>> >> >     some of its leading directories are symlinks?
>> >> >   . what if Emacs is invoked via a script which is in the correct
>> >> >     installation directory, but the actual binary the script invokes
>> >> >     is not in the expected location relative to the native-lisp/
>> >> >     directory where we have the preloaded *.eln files?
>> >> >
>> >> > The existing code handles all these cases, and some others.  We could
>> >> > perhaps _add_ the use of /proc/self/exe to what we have, but we'd need
>> >> > to be sure that it doesn't break for the above situations.
>> >
>> > I'm still waiting for some answers to these.
>> 
>> If we use /proc/self/exe, I'm fine with it being a fallback if all other
>> mechanisms fail.  That should make these cases still work fine, right?
>
> To use /proc/self/exe as fallback, we need to somehow reject the Emacs
> "executable" found in your case.  Because otherwise fallbacks are only
> used if the previous attempts fail, and the attempt to find atgv[0]
> along PATH does not fail in your case, it just finds the "wrong" file.

True, so maybe /proc/self/exe is not a good solution.

>> > Why cannot you modify the script for all the commands to include the
>> > leading directories in executable-name?  That is all that is needed
>> > for Emacs to find its *.eln files.
>> 
>> See the motivation that I quoted above:
>> 
>>  For context, I believe the reason why we pass `-a` is to make the prog
>>  more identifiable when users try to find it in the output of
>>  `ps`. That still sounds like the right thing to do in the majority of
>>  the cases.
>> 
>> Including the leading directories would make them show up in the output
>> of "ps", which is uglier.
>
> "Uglier" is in the eyes of the beholder.  The important thing is that
> it solves the problem without any other changes, and will then use
> code which was tested and validated by many users.
>
>> I realize this might not seem like an important justification, but it
>> works for every other program we run, and has worked for decades.  And
>> other distributors might be doing this too, so I think it's reasonable
>> to make Emacs robust to this by having it fall back to looking up
>> native-lisp in something like PATH_EXEC.
>
> I disagree that this is reasonable, and what you suggest is not
> possible/practical anyway.
>
>> But now I think we maybe don't need to use /proc/self/exe at all, and
>> can just have Emacs fall back on something like PATH_EXEC when it fails
>> to find the native lisp files.
>
> See above.  If you insist to go this much more complicated way, we'll
> need to modify the code in dump_do_dump_relocation to use PATH_EXEC,
> and deal with possible false positives.

Yes, I am willing to put in the effort to do that.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 73318@debbugs.gnu.org,  larsi@gnus.org,  acorallo@gnu.org,
>    schwab@linux-m68k.org,  shipmints@gmail.com
> Date: Fri, 04 Oct 2024 12:51:59 -0400
> 
> > It isn't that easy.  We need to support 2 possible locations for the
> > preloaded *.eln files: in the source tree and in the installation
> > tree.  You now want us to look in yet a third place.  Take a look at
> > dump_do_dump_relocation where we look for the *.eln files.  What you
> > suggest is to add a third place to that code.
> 
> Yes.  I suggest that if installation_state == INSTALLED, and we fail to
> find an eln file in emacs_execdir, we fall back to look in PATH_EXEC (or
> some other path compiled into the Emacs binary).
> 
> But it occurs to me that there's another possible solution.  We already
> have a robust way to find the pdump.  My Emacs finds the pdump and loads
> it just fine.  Maybe we could look for the eln files relative to the
> pdump?

That means to throw away a lot of code in emacs.c which we use now and
was tested for several Emacs releases.  No, thanks, not for this
obscure scenario.

I still don't understand why you insist on not changing your script to
include the leading directories in argv[0].  It makes absolutely no
sense to me to invent new code in such a place, when the alternative
is so easy and reliable.

> > See above.  If you insist to go this much more complicated way, we'll
> > need to modify the code in dump_do_dump_relocation to use PATH_EXEC,
> > and deal with possible false positives.
> 
> Yes, I am willing to put in the effort to do that.

Makes no sense to me, so I urge you to reconsider.  You are
destabilizing Emacs for the benefit of an obscure use pattern that is
easy to fix without any changes in Emacs.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 73318@debbugs.gnu.org,  larsi@gnus.org,  acorallo@gnu.org,
>    schwab@linux-m68k.org,  shipmints@gmail.com
> Date: Fri, 04 Oct 2024 12:51:59 -0400
> 
> Yes.  I suggest that if installation_state == INSTALLED, and we fail to
> find an eln file in emacs_execdir, we fall back to look in PATH_EXEC (or
> some other path compiled into the Emacs binary).

Btw, this is a misunderstanding of the code's logic: you must do this
when installation_state == UNKNOWN.  When it's INSTALLED, it means we
already found the first .eln file we needed to load in the
installation tree.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Spencer Baugh via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: 73318@debbugs.gnu.org,  larsi@gnus.org,  acorallo@gnu.org,
>>    schwab@linux-m68k.org,  shipmints@gmail.com
>> Date: Fri, 04 Oct 2024 12:51:59 -0400
>> 
>> > It isn't that easy.  We need to support 2 possible locations for the
>> > preloaded *.eln files: in the source tree and in the installation
>> > tree.  You now want us to look in yet a third place.  Take a look at
>> > dump_do_dump_relocation where we look for the *.eln files.  What you
>> > suggest is to add a third place to that code.
>> 
>> Yes.  I suggest that if installation_state == INSTALLED, and we fail to
>> find an eln file in emacs_execdir, we fall back to look in PATH_EXEC (or
>> some other path compiled into the Emacs binary).
>> 
>> But it occurs to me that there's another possible solution.  We already
>> have a robust way to find the pdump.  My Emacs finds the pdump and loads
>> it just fine.  Maybe we could look for the eln files relative to the
>> pdump?
>
> That means to throw away a lot of code in emacs.c which we use now and
> was tested for several Emacs releases.  No, thanks, not for this
> obscure scenario.

OK, then perhaps just falling back on finding it relative to the pdump.

> I still don't understand why you insist on not changing your script to
> include the leading directories in argv[0].  It makes absolutely no
> sense to me to invent new code in such a place, when the alternative
> is so easy and reliable.

I would change the script if I could, but I can't: It's already been
copied into thousands of users' home directories.  It's such a tiny
wrapper (it just does "exec -a program /path/to/program") that it has
never needed to be modified for any program before this.

>> > See above.  If you insist to go this much more complicated way, we'll
>> > need to modify the code in dump_do_dump_relocation to use PATH_EXEC,
>> > and deal with possible false positives.
>> 
>> Yes, I am willing to put in the effort to do that.
>
> Makes no sense to me, so I urge you to reconsider.  You are
> destabilizing Emacs for the benefit of an obscure use pattern that is
> easy to fix without any changes in Emacs.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 73318@debbugs.gnu.org,  larsi@gnus.org,  acorallo@gnu.org,
>    schwab@linux-m68k.org,  shipmints@gmail.com
> Date: Mon, 07 Oct 2024 10:18:37 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That means to throw away a lot of code in emacs.c which we use now and
> > was tested for several Emacs releases.  No, thanks, not for this
> > obscure scenario.
> 
> OK, then perhaps just falling back on finding it relative to the pdump.
> 
> > I still don't understand why you insist on not changing your script to
> > include the leading directories in argv[0].  It makes absolutely no
> > sense to me to invent new code in such a place, when the alternative
> > is so easy and reliable.
> 
> I would change the script if I could, but I can't: It's already been
> copied into thousands of users' home directories.  It's such a tiny
> wrapper (it just does "exec -a program /path/to/program") that it has
> never needed to be modified for any program before this.

Suit yourself, but I don't promise I will agree to accept the code
changes you want to make.  This makes less and less sense to me as we
continue talking about it.

Emacs built with native compilation has additional requirements, and
those requirements invalidate the (fragile) assumptions on which your
wrapper script is based.  So my suggestion is to adapt, like we adapt
to any changes to software we use that are outside of our control.
That's life.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Spencer Baugh via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: 73318@debbugs.gnu.org,  larsi@gnus.org,  acorallo@gnu.org,
>>    schwab@linux-m68k.org,  shipmints@gmail.com
>> Date: Mon, 07 Oct 2024 10:18:37 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > That means to throw away a lot of code in emacs.c which we use now and
>> > was tested for several Emacs releases.  No, thanks, not for this
>> > obscure scenario.
>> 
>> OK, then perhaps just falling back on finding it relative to the pdump.
>> 
>> > I still don't understand why you insist on not changing your script to
>> > include the leading directories in argv[0].  It makes absolutely no
>> > sense to me to invent new code in such a place, when the alternative
>> > is so easy and reliable.
>> 
>> I would change the script if I could, but I can't: It's already been
>> copied into thousands of users' home directories.  It's such a tiny
>> wrapper (it just does "exec -a program /path/to/program") that it has
>> never needed to be modified for any program before this.
>
> Suit yourself, but I don't promise I will agree to accept the code
> changes you want to make.  This makes less and less sense to me as we
> continue talking about it.

That's fair enough.

Of the ways to make "exec -a emacs" work that I've proposed, which one
do you dislike the least?

So far I think the choice is between:

- Fallback to finding elns relative to PATH_EXEC (or some similar hardcoded path)
- Fallback to finding elns relative to the pdump location

Or maybe there's some other way to make "exec -a emacs" work that you
like better?

If you dislike them all equally, I'll probably implement the "fallback
to finding elns relative to pdump" approach.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: 73318@debbugs.gnu.org,  larsi@gnus.org,  acorallo@gnu.org,
>    schwab@linux-m68k.org,  shipmints@gmail.com
> Date: Mon, 07 Oct 2024 16:16:06 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Suit yourself, but I don't promise I will agree to accept the code
> > changes you want to make.  This makes less and less sense to me as we
> > continue talking about it.
> 
> That's fair enough.
> 
> Of the ways to make "exec -a emacs" work that I've proposed, which one
> do you dislike the least?
> 
> So far I think the choice is between:
> 
> - Fallback to finding elns relative to PATH_EXEC (or some similar hardcoded path)
> - Fallback to finding elns relative to the pdump location
> 
> Or maybe there's some other way to make "exec -a emacs" work that you
> like better?
> 
> If you dislike them all equally, I'll probably implement the "fallback
> to finding elns relative to pdump" approach.

How will any of this DTRT in both Emacs that is run uninstalled
(i.e. from its build tree) and Emacs that is run installed?

We have this figured out now, but we rely on the directory of the
Emacs binary.  If you insist to hide that information from Emacs,
please explain how you intend to solve the above conundrum.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> Cc: 73318@debbugs.gnu.org, larsi@gnus.org, acorallo@gnu.org,
>  schwab@linux-m68k.org, shipmints@gmail.com
> Date: Tue, 08 Oct 2024 14:47:42 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Spencer Baugh <sbaugh@janestreet.com>
> > If you dislike them all equally, I'll probably implement the "fallback
> > to finding elns relative to pdump" approach.
> 
> How will any of this DTRT in both Emacs that is run uninstalled
> (i.e. from its build tree) and Emacs that is run installed?
> 
> We have this figured out now, but we rely on the directory of the
> Emacs binary.  If you insist to hide that information from Emacs,
> please explain how you intend to solve the above conundrum.

Btw, the "relative to pdump" approach might have a problem if Emacs
was invoked with --dump-file=FILE command-line option, because such a
file might have no relation to where the *.eln files live.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Stefan Kangas]

Spencer Baugh via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> Yes, I'm persuaded now that if we use /proc/self/exe, we should also use
> argv[0].  As Po Lu said, /proc can be unmounted, so we must have some
> other mechanism besides just /proc/self/exe.
[...]
> If we use /proc/self/exe, I'm fine with it being a fallback if all other
> mechanisms fail.  That should make these cases still work fine, right?

Did we discard the idea to check for /proc/self/exe first (on systems
that support it, of course), and falling back to argv[0] if that fails?

Could we use getauxval(AT_EXECFN) when we have glibc?

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Tue, 8 Oct 2024 23:35:11 +0000
> Cc: 73318@debbugs.gnu.org, larsi@gnus.org, acorallo@gnu.org, 
> 	schwab@linux-m68k.org, shipmints@gmail.com
> 
> Spencer Baugh via "Bug reports for GNU Emacs, the Swiss army knife of
> text editors" <bug-gnu-emacs@gnu.org> writes:
> 
> > Yes, I'm persuaded now that if we use /proc/self/exe, we should also use
> > argv[0].  As Po Lu said, /proc can be unmounted, so we must have some
> > other mechanism besides just /proc/self/exe.
> [...]
> > If we use /proc/self/exe, I'm fine with it being a fallback if all other
> > mechanisms fail.  That should make these cases still work fine, right?
> 
> Did we discard the idea to check for /proc/self/exe first (on systems
> that support it, of course), and falling back to argv[0] if that fails?

This will have to somehow resolve the potential problems I listed in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=73318#32 (and perhaps
some others I didn't mention there).  That is not trivial, and will
probably need to do something we already do, just without using
/proc/self/exe.

So I very much dislike this alternative.  At the time Daniel coded the
initial version of this code, he decided (and others generally agreed)
that use of /proc/self/exe does not bring us any advantages.  The code
which was written then and improved and enhanced since then is solid,
supports quite a few use patterns (some of them quite surprising), and
was tested by 3 Emacs releases.  Significant modifications of the
code's logic (which will be necessary of /proc/self/exe is the first
method to be used) will basically set us back to where we were in
Emacs 28, something I'd like to avoid.

> Could we use getauxval(AT_EXECFN) when we have glibc?

Isn't that the same method as /proc/self/exe, just thinly disguised?