bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Spencer Baugh via Bug reports for GNU Emacs, the Swiss army knife of text editors]

1. Compile and install Emacs with --with-native-compilation=aot, e.g.:
prefix=~/prefix
mkdir $prefix
./configure --with-native-compilation=aot --prefix=$prefix
make -j64 && make install
2. Run emacs with "exec -a" to change its argv[0]:
sh -c "exec -a emacs $prefix/bin/emacs -Q --batch"
3. Observe an error like:
Error using execdir /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/:
emacs: /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/../native-lisp/31.0.50-a88a37f5/preloaded/minibuffer-b2d9c221-284ab177.eln: cannot open shared object file: No such file or directory

"exec -a emacs" works fine for with-native-compilation=yes or
with-native-compilation=no.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Spencer Baugh via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Spencer Baugh <sbaugh@janestreet.com> writes:

> 1. Compile and install Emacs with --with-native-compilation=aot, e.g.:
> prefix=~/prefix
> mkdir $prefix
> ./configure --with-native-compilation=aot --prefix=$prefix
> make -j64 && make install
> 2. Run emacs with "exec -a" to change its argv[0]:
> sh -c "exec -a emacs $prefix/bin/emacs -Q --batch"
> 3. Observe an error like:
> Error using execdir /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/:
> emacs: /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/../native-lisp/31.0.50-a88a37f5/preloaded/minibuffer-b2d9c221-284ab177.eln: cannot open shared object file: No such file or directory
>
> "exec -a emacs" works fine for with-native-compilation=yes or
> with-native-compilation=no.

Actually this also breaks with with-native-compilation=yes

It seems sensitive to cached compilation or configure results; I can
only get a reliable failure if I make clean in between attempts.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> Cc: Lars Ingebrigtsen <larsi@gnus.org>, Andrea Corallo <acorallo@gnu.org>
> Date: Tue, 17 Sep 2024 11:18:41 -0400
> From:  Spencer Baugh via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> 
> 1. Compile and install Emacs with --with-native-compilation=aot, e.g.:
> prefix=~/prefix
> mkdir $prefix
> ./configure --with-native-compilation=aot --prefix=$prefix
> make -j64 && make install
> 2. Run emacs with "exec -a" to change its argv[0]:
> sh -c "exec -a emacs $prefix/bin/emacs -Q --batch"
> 3. Observe an error like:
> Error using execdir /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/:
> emacs: /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/../native-lisp/31.0.50-a88a37f5/preloaded/minibuffer-b2d9c221-284ab177.eln: cannot open shared object file: No such file or directory
> 
> "exec -a emacs" works fine for with-native-compilation=yes or
> with-native-compilation=no.

Invocation via "exec -a" is not supported, if it messes with the
leading directories of the argv[0] value passed to Emacs.  That's
because the search for the preloaded *.eln files is based on the
directory in which the Emacs executable is installed, as passed via
argv[0], and breaks if "exec -a" messes with that.

IOW, "don't do that, it will hurt".

P.S. If someone knows how to teach Emacs how to find the absolute file
name of its executable without depending on argv[0], speak up.  We do
that on Windows, but not on Posix platforms, since (I'm being told)
there's no reliable way of having that on GNU/Linux and other Posix
platforms.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 2097 bytes --]

Google's build system, gn, used for Chromium, et.al., has support for this
which can provide some inspiration for most of the major platforms on which
Emacs runs; qv
https://gn.googlesource.com/gn/+/refs/heads/main/src/util/exe_path.cc

On Tue, Sep 17, 2024 at 1:51 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > Cc: Lars Ingebrigtsen <larsi@gnus.org>, Andrea Corallo <acorallo@gnu.org
> >
> > Date: Tue, 17 Sep 2024 11:18:41 -0400
> > From:  Spencer Baugh via "Bug reports for GNU Emacs,
> >  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> >
> >
> > 1. Compile and install Emacs with --with-native-compilation=aot, e.g.:
> > prefix=~/prefix
> > mkdir $prefix
> > ./configure --with-native-compilation=aot --prefix=$prefix
> > make -j64 && make install
> > 2. Run emacs with "exec -a" to change its argv[0]:
> > sh -c "exec -a emacs $prefix/bin/emacs -Q --batch"
> > 3. Observe an error like:
> > Error using execdir
> /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/:
> > emacs:
> /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/../native-lisp/31.0.50-a88a37f5/preloaded/minibuffer-b2d9c221-284ab177.eln:
> cannot open shared object file: No such file or directory
> >
> > "exec -a emacs" works fine for with-native-compilation=yes or
> > with-native-compilation=no.
>
> Invocation via "exec -a" is not supported, if it messes with the
> leading directories of the argv[0] value passed to Emacs.  That's
> because the search for the preloaded *.eln files is based on the
> directory in which the Emacs executable is installed, as passed via
> argv[0], and breaks if "exec -a" messes with that.
>
> IOW, "don't do that, it will hurt".
>
> P.S. If someone knows how to teach Emacs how to find the absolute file
> name of its executable without depending on argv[0], speak up.  We do
> that on Windows, but not on Posix platforms, since (I'm being told)
> there's no reliable way of having that on GNU/Linux and other Posix
> platforms.
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 2944 bytes --]

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Ship Mints <shipmints@gmail.com>
> Date: Tue, 17 Sep 2024 14:14:54 -0400
> Cc: Spencer Baugh <sbaugh@janestreet.com>, 73318@debbugs.gnu.org, larsi@gnus.org, 
> 	acorallo@gnu.org
> 
> Google's build system, gn, used for Chromium, et.al., has support for this which can provide some inspiration
> for most of the major platforms on which Emacs runs; qv
> https://gn.googlesource.com/gn/+/refs/heads/main/src/util/exe_path.cc

Thanks.  That's exactly the method that I was told was unreliable on
GNU/Linux.  I don't know enough to argue with people who said that
about this particular feature.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 1020 bytes --]

I think the kind of "unreliability" in question is, for example, when a
process starts and unlinks itself. I doubt Emacs will ever do this. Using
the proc file system is "technically" unreliable, unable to cover 100% of
all potential cases, but is practically reliable, especially in this case.

On Tue, Sep 17, 2024 at 3:07 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Ship Mints <shipmints@gmail.com>
> > Date: Tue, 17 Sep 2024 14:14:54 -0400
> > Cc: Spencer Baugh <sbaugh@janestreet.com>, 73318@debbugs.gnu.org,
> larsi@gnus.org,
> >       acorallo@gnu.org
> >
> > Google's build system, gn, used for Chromium, et.al., has support for
> this which can provide some inspiration
> > for most of the major platforms on which Emacs runs; qv
> > https://gn.googlesource.com/gn/+/refs/heads/main/src/util/exe_path.cc
>
> Thanks.  That's exactly the method that I was told was unreliable on
> GNU/Linux.  I don't know enough to argue with people who said that
> about this particular feature.
>

[-- Attachment #2: Type: text/html, Size: 1899 bytes --]

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Ship Mints <shipmints@gmail.com>
> Date: Tue, 17 Sep 2024 15:22:18 -0400
> Cc: sbaugh@janestreet.com, 73318@debbugs.gnu.org, larsi@gnus.org, 
> 	acorallo@gnu.org
> I think the kind of "unreliability" in question is, for example, when a process starts and unlinks itself. I doubt
> Emacs will ever do this. Using the proc file system is "technically" unreliable, unable to cover 100% of all
> potential cases, but is practically reliable, especially in this case.

I don't remember the details, sorry.  You are welcome to look up the
past discussions in the archives.  I think they were triggered by look
up of the pdumper file, but the results of that are also used by the
code which decides where to look for the *.eln files.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Spencer Baugh via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ship Mints <shipmints@gmail.com>
>> Date: Tue, 17 Sep 2024 15:22:18 -0400
>> Cc: sbaugh@janestreet.com, 73318@debbugs.gnu.org, larsi@gnus.org, 
>> 	acorallo@gnu.org
>> I think the kind of "unreliability" in question is, for example, when a process starts and unlinks itself. I doubt
>> Emacs will ever do this. Using the proc file system is "technically" unreliable, unable to cover 100% of all
>> potential cases, but is practically reliable, especially in this case.
>
> I don't remember the details, sorry.  You are welcome to look up the
> past discussions in the archives.  I think they were triggered by look
> up of the pdumper file, but the results of that are also used by the
> code which decides where to look for the *.eln files.

I looked up /proc/self/exe in the archives and the only mention is
https://lists.gnu.org/archive/html/emacs-devel/2019-05/msg00951.html

Gnulib uses /proc/self/exe to provide support for relocatability (in
progreloc.c).  If it's reliable enough for Gnulib, it should be reliable
enough for Emacs.

With all due humility, I think I personally am enough of an expert on
Linux minutiae to say that /proc/self/exe will be substantially more
reliable than using argv[0].

I can provide a patch to make invocation-directory use /proc/self/exe,
why don't we just try installing it on master?  If it's worse, we should
learn soon enough.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 1801 bytes --]

I agree in the main. This thread looks relevant to pdump concerns. I have
no experience making my own pdumps as some seem to do.

https://lists.gnu.org/archive/html/emacs-devel/2019-01/msg00558.html

On Tue, Sep 17, 2024 at 6:31 PM Spencer Baugh <sbaugh@janestreet.com> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> From: Ship Mints <shipmints@gmail.com>
> >> Date: Tue, 17 Sep 2024 15:22:18 -0400
> >> Cc: sbaugh@janestreet.com, 73318@debbugs.gnu.org, larsi@gnus.org,
> >>      acorallo@gnu.org
> >> I think the kind of "unreliability" in question is, for example, when a
> process starts and unlinks itself. I doubt
> >> Emacs will ever do this. Using the proc file system is "technically"
> unreliable, unable to cover 100% of all
> >> potential cases, but is practically reliable, especially in this case.
> >
> > I don't remember the details, sorry.  You are welcome to look up the
> > past discussions in the archives.  I think they were triggered by look
> > up of the pdumper file, but the results of that are also used by the
> > code which decides where to look for the *.eln files.
>
> I looked up /proc/self/exe in the archives and the only mention is
> https://lists.gnu.org/archive/html/emacs-devel/2019-05/msg00951.html
>
> Gnulib uses /proc/self/exe to provide support for relocatability (in
> progreloc.c).  If it's reliable enough for Gnulib, it should be reliable
> enough for Emacs.
>
> With all due humility, I think I personally am enough of an expert on
> Linux minutiae to say that /proc/self/exe will be substantially more
> reliable than using argv[0].
>
> I can provide a patch to make invocation-directory use /proc/self/exe,
> why don't we just try installing it on master?  If it's worse, we should
> learn soon enough.
>

[-- Attachment #2: Type: text/html, Size: 2964 bytes --]

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: Ship Mints <shipmints@gmail.com>,  larsi@gnus.org,  acorallo@gnu.org,
>    73318@debbugs.gnu.org
> Date: Tue, 17 Sep 2024 18:31:05 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I don't remember the details, sorry.  You are welcome to look up the
> > past discussions in the archives.  I think they were triggered by look
> > up of the pdumper file, but the results of that are also used by the
> > code which decides where to look for the *.eln files.
> 
> I looked up /proc/self/exe in the archives and the only mention is
> https://lists.gnu.org/archive/html/emacs-devel/2019-05/msg00951.html

A more interesting discussion starts here:

  https://lists.gnu.org/archive/html/emacs-devel/2019-01/msg00635.html

That discussion is about finding the pdumper file, but the side effect
of looking for pdumper file is the directory where we think the Emacs
executable file is.  That discussion mentions several issues related
to finding the leading directories of the Emacs executable.

Another useful read is here:

  https://stackoverflow.com/questions/1023306/finding-current-executables-path-without-proc-self-exe

> With all due humility, I think I personally am enough of an expert on
> Linux minutiae to say that /proc/self/exe will be substantially more
> reliable than using argv[0].

And I will see your humility and raise ya.  Please describe your ideas
for the patch before actually writing the code.  Because there's more
here than meets the eye.  Some issues the related code needs to
handle:

  . what if /proc/self/exe is unreadable? AFAIK, on some systems you
    need special privileges to follow its symlink
  . what if /proc/self/exe points to a file name that is a symlink, or
    some of its leading directories are symlinks?
  . what if Emacs is invoked via a script which is in the correct
    installation directory, but the actual binary the script invokes
    is not in the expected location relative to the native-lisp/
    directory where we have the preloaded *.eln files?

The existing code handles all these cases, and some others.  We could
perhaps _add_ the use of /proc/self/exe to what we have, but we'd need
to be sure that it doesn't break for the above situations.

I also don't understand why your script insists on removing the
leading directories from argv[0] of Emacs.  Is there any problem for
you to modify your script such that the leading directories would
still be present in argv[0]?

And finally, your description of the original issue seems to omit some
crucial details (or maybe I'm missing something):

> 1. Compile and install Emacs with --with-native-compilation=aot, e.g.:
> prefix=~/prefix
> mkdir $prefix
> ./configure --with-native-compilation=aot --prefix=$prefix
> make -j64 && make install
> 2. Run emacs with "exec -a" to change its argv[0]:
> sh -c "exec -a emacs $prefix/bin/emacs -Q --batch"
> 3. Observe an error like:
> Error using execdir /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/:
> emacs: /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/emacs/bin/../native-lisp/31.0.50-a88a37f5/preloaded/minibuffer-b2d9c221-284ab177.eln: cannot open shared object file: No such file or directory

Where did the /usr/local/home/sbaugh/workspaces/24833141-bffb-3c99-a9d6-c366d37c4f5e/+share+/app/
part come from?  I'm guessing that /usr/local/home/sbaugh/ is the
expansion of "~" in your case, but where did the rest come from if
your $prefix is just "~/prefix"?

When Emacs does not find its executable file using argv[0], it assumes
that the executable is in PATH_EXEC/../../../../bin/.  Since you are
running an installed Emacs, that should have worked, unless you also
somehow changed the relative path from $prefix/bin to the directory
where the native-lisp/ directory is installed.  Why didn't it work?

Bottom line: I think there are still unclear aspects of what happened
in your case, and using /proc/self/exe to fix that is not as simple as
it might seem, especially since we don't yet understand fully what
failed and why.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> And I will see your humility and raise ya.  Please describe your ideas
> for the patch before actually writing the code.  Because there's more
> here than meets the eye.  Some issues the related code needs to
> handle:
>
>   . what if /proc/self/exe is unreadable? AFAIK, on some systems you
>     need special privileges to follow its symlink

Above all, /proc is liable simply to be unmounted.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

Even if you umount /proc, it will remain until active process references to
/proc nodes are closed. If one tries umount /proc outside of a shutdown
sequence, Emacs is the least of her worries. This is not a practical
deterrent.

On Wed, Sep 18, 2024 at 11:09 PM Po Lu <luangruo@yahoo.com> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> > And I will see your humility and raise ya.  Please describe your ideas
> > for the patch before actually writing the code.  Because there's more
> > here than meets the eye.  Some issues the related code needs to
> > handle:
> >
> >   . what if /proc/self/exe is unreadable? AFAIK, on some systems you
> >     need special privileges to follow its symlink
>
> Above all, /proc is liable simply to be unmounted.
>

[-- Attachment #2: Type: text/html, Size: 1222 bytes --]

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Ship Mints <shipmints@gmail.com> writes:

> Even if you umount /proc, it will remain until active process
> references to /proc nodes are closed. If one tries umount /proc
> outside of a shutdown sequence, Emacs is the least of her
> worries. This is not a practical deterrent.

It is possible to run systems with Linux (the kernel) without mounting
/proc at all, and Emacs is very much interested in functioning correctly
there.  Moreover, the value of /proc/self/exe is sometimes completely
meaningless, as on Android, where Emacs is loaded into
/system/bin/app_process64 as a shared library.

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Ship Mints]

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

I see. It pays, then, to disambiguate between a Linux "system" (which is
the typical Emacs case) and Linux O/S-based, essentially embedded,
environments. In the "system" case, /proc/self/exe should be the
recommended method, I'd think. In embedded cases, whatever is appropriate
in those environments is what should be used.

On Thu, Sep 19, 2024 at 9:44 AM Po Lu <luangruo@yahoo.com> wrote:

> Ship Mints <shipmints@gmail.com> writes:
>
> > Even if you umount /proc, it will remain until active process
> > references to /proc nodes are closed. If one tries umount /proc
> > outside of a shutdown sequence, Emacs is the least of her
> > worries. This is not a practical deterrent.
>
> It is possible to run systems with Linux (the kernel) without mounting
> /proc at all, and Emacs is very much interested in functioning correctly
> there.  Moreover, the value of /proc/self/exe is sometimes completely
> meaningless, as on Android, where Emacs is loaded into
> /system/bin/app_process64 as a shared library.
>
>

[-- Attachment #2: Type: text/html, Size: 1497 bytes --]

bug#73318: 31.0.50; with-native-compilation=aot breaks exec -a emacs

[Eli Zaretskii]

> From: Ship Mints <shipmints@gmail.com>
> Date: Thu, 19 Sep 2024 09:51:44 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, Spencer Baugh <sbaugh@janestreet.com>, 73318@debbugs.gnu.org, 
> 	larsi@gnus.org, acorallo@gnu.org
> 
> I see. It pays, then, to disambiguate between a Linux "system" (which is the typical Emacs case) and Linux
> O/S-based, essentially embedded, environments. In the "system" case, /proc/self/exe should be the
> recommended method, I'd think. In embedded cases, whatever is appropriate in those environments is what
> should be used.

Please re-read what I wrote about the use cases we have to support and
the discussion to which I pointed.  No single method can support all
the situations which Emacs needs to support.  So there's no "the
recommended method": even if we add /proc/self/exe, it can only be one
of the methods we use to find where the *.eln files are, and not
necessarily the most important one.