bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

[Andreas Fuchs]

[-- Attachment #1: Type: text/plain, Size: 2864 bytes --]

On macos (if you use --with-ns), there are two ways the emacs binary
gets installed:

* as <prefix>/bin/emacs, as normal
* as <prefix>/Application/Emacs.app/Contents/MacOS/Emacs, in a macOS app
  bundle.

Both of these cause problems, unfortunately:

The "normal" emacs binary you can't invoke from $PATH:

$ emacs -q --batch --eval '(message "hi")'
emacs:
dlopen(../eln-cache/x86_64-apple-darwin19.6.0-9cab85d51a8656a0/lisp-mode-0189ba85598c041b7504f0a916c04219-5bf5ce8701bca2b361b97967a6f95776.eln,
1): image not found

But it does work when run with an absolute path:

$
/nix/store/a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0/bin/emacs
-q --batch --eval '(message "hi")'
hi


The app bundle Emacs, on the other hand, doesn't like the relative path
at all either:

$
/nix/store/a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0/Applications/Emacs.app/Contents/MacOS/Emacs
emacs:
dlopen(/nix/store/a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0/Applications/Emacs.app/Contents/MacOS/../eln-cache/x86_64-apple-darwin19.6.0-9cab85d51a8656a0/lisp-mode-0189ba85598c041b7504f0a916c04219-5bf5ce8701bca2b361b97967a6f95776.eln,
1): image not found

I have traced both of these down to the relocation logic in pdumper.c:
https://github.com/emacs-mirror/emacs/blob/feature/native-comp/src/pdumper.c#L5255-L5277

This code (and the corresponding bit in
https://github.com/emacs-mirror/emacs/blob/feature/native-comp/lisp/loadup.el#L471-L477
both seem to take the absolute installation directory and turn them into
relative directories that then fail to work. That the plain "emacs"
binary fails to run might be due to the fact that `invocation-directory`
is not set in that case, and so it has nothing to prepend to the
relative name.

In nix, an installation never moves - it's a content-addressed store, so
there's a very fixed pathname that everything lives at forever; is
it possible to turn off that pathname relativization?

Alternatively, I guess for darwin I'd also be good if we could get the
correct relative names in the application bundle; There is just one
`../Resources` too few in there. I think to handle this right, we'd have
to add another case to the installation_state enum in pdumper.c, is
that right?

Cheers,
Andreas.



In GNU Emacs 28.0.50 (build 1, x86_64-apple-darwin19.6.0, NS appkit-1894.60
Version 10.15.6 (Build 19G2021))
Repository revision: aa526c9470d679e9144af55d9e56928a111d2ceb
Repository branch: master
Windowing system distributor 'Apple', version 10.3.1894
System Description:  Mac OS X 10.15.6

Configured using:
 'configure
--prefix=/nix/store/a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0
--disable-build-details --with-modules --with-ns
--disable-ns-self-contained --with-nativecomp
CFLAGS=-DMAC_OS_X_VERSION_MAX_ALLOWED=101200'


--
Andreas Fuchs, (http://|im:asf@|mailto:asf@)boinkor.net, antifuchs

[-- Attachment #2: Type: text/html, Size: 3460 bytes --]

bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andreas Fuchs <asf@boinkor.net> writes:

> On macos (if you use --with-ns), there are two ways the emacs binary
> gets installed:
>
> * as <prefix>/bin/emacs, as normal
> * as <prefix>/Application/Emacs.app/Contents/MacOS/Emacs, in a macOS
> app
>   bundle.
>
> Both of these cause problems, unfortunately:
>
> The "normal" emacs binary you can't invoke from $PATH:
>
> $ emacs -q --batch --eval '(message "hi")'
> emacs: dlopen(../eln-cache/x86_64-apple-darwin19.6.0-9cab85d51a8656a0
> /
> lisp-mode-0189ba85598c041b7504f0a916c04219-5bf5ce8701bca2b361b97967a6f95776.eln,
> 1): image not found
>
> But it does work when run with an absolute path:
>
> $ /nix/store/a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0/
> bin/emacs -q --batch --eval '(message "hi")'
> hi
>
>
> The app bundle Emacs, on the other hand, doesn't like the relative
> path
> at all either:
>
> $ /nix/store/a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0/
> Applications/Emacs.app/Contents/MacOS/Emacs
> emacs: dlopen(/nix/store/
> a55i9aws0j96z0zfzxfffmfb6k2lw53v-emacs-gcc-20200814.0/Applications/
> Emacs.app/Contents/MacOS/../eln-cache/
> x86_64-apple-darwin19.6.0-9cab85d51a8656a0/
> lisp-mode-0189ba85598c041b7504f0a916c04219-5bf5ce8701bca2b361b97967a6f95776.eln,
> 1): image not found
>  
> I have traced both of these down to the relocation logic in
> pdumper.c:
> https://github.com/emacs-mirror/emacs/blob/feature/native-comp/src/
> pdumper.c#L5255-L5277
>
> This code (and the corresponding bit in
> https://github.com/emacs-mirror/emacs/blob/feature/native-comp/lisp/
> loadup.el#L471-L477
> both seem to take the absolute installation directory and turn them
> into
> relative directories that then fail to work. That the plain "emacs"
> binary fails to run might be due to the fact that
> `invocation-directory`
> is not set in that case, and so it has nothing to prepend to the
> relative name.
>
> In nix, an installation never moves - it's a content-addressed store,
> so
> there's a very fixed pathname that everything lives at forever; is
> it possible to turn off that pathname relativization?
>
> Alternatively, I guess for darwin I'd also be good if we could get
> the
> correct relative names in the application bundle; There is just one
> `../Resources` too few in there. I think to handle this right, we'd
> have
> to add another case to the installation_state enum in pdumper.c, is
> that right?
>
> Cheers,
> Andreas.

Hi Andreas,

I think we might have two issues here.

The first is to provide the correct eln destination directory during the
build so it can be used correctly by the logic that starts at
loadup.el:452.  This should fixup the filenames so afterwards when
resurrecting the logic in pdumper.c can work correctly.

About this there's a branch in feature/native-comp-macos-fixes (by
Vibhav Pant Cc'ed) with a fix.  See also [1].  Maybe you like to give it
a go.

The other possible source of problems may be Vinvocation_directory still
not set when we pass into pdumper.c:5270, I can't verify that as I don't
have use macos but should be realitively easy to verify.

Ciao

  Andrea

[1] <https://lists.gnu.org/archive/html/emacs-devel/2020-08/msg00700.html>

bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

[Andreas Fuchs]

[-- Attachment #1: Type: text/plain, Size: 1258 bytes --]

Andrea Corallo <akrl@sdf.org> writes:
> The first is to provide the correct eln destination directory during the
> build so it can be used correctly by the logic that starts at
> loadup.el:452.  This should fixup the filenames so afterwards when
> resurrecting the logic in pdumper.c can work correctly.
>
> About this there's a branch in feature/native-comp-macos-fixes (by
> Vibhav Pant Cc'ed) with a fix.  See also [1].  Maybe you like to give it
> a go.

Oooh, that's a good catch. If I read the diff correctly, with a
not-only-self-contained build, it will set things up such that the
Application bundle works, but the (also installed) emacs binary won't.

A solution here might be to build two pdmp files - one for the
relocatable app bundle and one for commandline usage. I'll try to build
that.

> The other possible source of problems may be Vinvocation_directory still
> not set when we pass into pdumper.c:5270, I can't verify that as I don't
> have use macos but should be realitively easy to verify.

This makes sense too. I'll investigate when `Vinvocation_directory` gets
set. Does sound like that doesn't happen early enough, maybe we can pull
it forward a bit.

Thanks,
-- 
Andreas Fuchs, (http://|im:asf@|mailto:asf@)boinkor.net, antifuchs

[-- Attachment #2: Type: text/html, Size: 1560 bytes --]

bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

[Andreas Fuchs]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

I <asf@boinkor.net> wrote:
> This makes sense too. I'll investigate when `Vinvocation_directory` gets
> set. Does sound like that doesn't happen early enough, maybe we can pull
> it forward a bit.

I dug into this, and the way main() progresses is that it first tries to
load the pdump (via load_pdump in line 1076,
https://github.com/emacs-mirror/emacs/blob/feature/native-comp/src/emacs.c#L1076
),
and only much later, sets invocation-directory and other cmdargs via
set_invocation_vars called from init_cmdargs (called in main on line 1794,
https://github.com/emacs-mirror/emacs/blob/feature/native-comp/src/emacs.c#L1794
).

I think this should be failing cross-platform; is it working for you?

Thanks,
--
Andreas Fuchs, (http://|im:asf@|mailto:asf@)boinkor.net, antifuchs

[-- Attachment #2: Type: text/html, Size: 1155 bytes --]

bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andreas Fuchs <asf@boinkor.net> writes:

> I <asf@boinkor.net> wrote:
>> This makes sense too. I'll investigate when `Vinvocation_directory`
> gets
>> set. Does sound like that doesn't happen early enough, maybe we can
> pull
>> it forward a bit.
>
> I dug into this, and the way main() progresses is that it first tries
> to
> load the pdump (via load_pdump in line 1076,
> https://github.com/emacs-mirror/emacs/blob/feature/native-comp/src/
> emacs.c#L1076),
> and only much later, sets invocation-directory and other cmdargs via
> set_invocation_vars called from init_cmdargs (called in main on line
> 1794,
> https://github.com/emacs-mirror/emacs/blob/feature/native-comp/src/
> emacs.c#L1794).
>
> I think this should be failing cross-platform; is it working for you?
>
> Thanks,

Hi Andreas,

yes for me is set before.  I've made set_invocation_vars to be called
before doing what I've called LATE_RELOCS in the pdumper.

Here is my backtrace of where gdb stops just after Vinvocation_directory
is set.

 (gdb) bt
 #0  set_invocation_vars (argv0=0x7fffffffe693 "/home/andcor03/emacs2/src/emacs",
     original_pwd=0xacb6f0 "/home/andcor03/emacs2/src") at emacs.c:455
 #1  0x000000000057ade2 in pdumper_load (
     dump_filename=0xacb750 "/home/andcor03/emacs2/src/emacs.pdmp",
     argv0=0x7fffffffe693 "/home/andcor03/emacs2/src/emacs",
     original_pwd=0xacb6f0 "/home/andcor03/emacs2/src") at pdumper.c:5593
 #2  0x00000000004e3a2f in load_pdump (argc=2, argv=0x7fffffffe428,
     original_pwd=0xacb6f0 "/home/andcor03/emacs2/src") at emacs.c:868
 #3  0x00000000004e3f4d in main (argc=2, argv=0x7fffffffe428) at emacs.c:1076

Is it possible that set_invocation_vars on macos fails to set the
correct value?  I'd look into it as I guess something may go wrong
there.

  Andrea

bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

[Andreas Fuchs]

[-- Attachment #1: Type: text/plain, Size: 666 bytes --]

I think I found the solution: set_invocation_vars requires Vexec_path to be
initialized correctly, via init_callproc_1. Somehow, mine ends up being
something other than $PATH. If I set init_callproc_1 up with a
double_run_guard the same way set_invocation_var is and call it from
pdumper_load, it results in a correctly-set Vexec_path variable and a
correctly-set Vinvocation_directory, as well.

This is really curious, but I guess it would explain why the invocation
directory variable is empty! What it doesn't explain is, why does this work
for you? Maybe your $PATH variable is set up differently? (Though I don't
see where exec-path would be set otherwise...)

[-- Attachment #2: Type: text/html, Size: 708 bytes --]

bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

[Andreas Fuchs]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

For the exec-path issue, I have a patch here:
https://github.com/emacs-mirror/emacs/commit/2c91d2c0fef162ecce87915863931bea47305eac;
this ensures running `emacs` from PATH on my system finds the right binary,
and lets emacs start up (there is a weird delay though, which may well be
my system being creaky).

The same branch that this change is on, I also have a change to build a
separate .pdmp file for the macOS app bundle - but that set of changes
isn't yet correct. It still picks up the wrong file name, still.

However, good news: One part of the bug is dead, at least (:

[-- Attachment #2: Type: text/html, Size: 766 bytes --]

bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andreas Fuchs <asf@boinkor.net> writes:

> For the exec-path issue, I have a patch here: https://github.com/
> emacs-mirror/emacs/commit/2c91d2c0fef162ecce87915863931bea47305eac;

I guess the best at this point would be to move out the invocation of
'set_invocation_vars' from 'init_cmdargs' and have 'set_invocation_vars'
and 'init_callproc_1' under a same wrapper with only one double
invocation guard.

Have you already done the copyright paperwork?  If not would you like to
do so?

  Andrea

bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

[Andreas Fuchs]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

On Wed, Sep 2, 2020 at 3:40 AM Andrea Corallo <akrl@sdf.org> wrote:

> Andreas Fuchs <asf@boinkor.net> writes
> > For the exec-path issue, I have a patch here: https://github.com/
> > emacs-mirror/emacs/commit/2c91d2c0fef162ecce87915863931bea47305eac;
>
> I guess the best at this point would be to move out the invocation of
> 'set_invocation_vars' from 'init_cmdargs' and have 'set_invocation_vars'
> and 'init_callproc_1' under a same wrapper with only one double
> invocation guard.
>

That makes sense - they're a bit far removed from each other in `main' but
are somewhat deeply intertwined, so I punted on this for now... your
suggestion is the right thing to do, of course (:


> Have you already done the copyright paperwork?  If not would you like to
> do so?


I've done my assignment for past & future contributions to GNU Emacs, a few
weeks ago! (:

Cheers,
-- 
Andreas Fuchs, (http://|im:asf@|mailto:asf@)boinkor.net, antifuchs

[-- Attachment #2: Type: text/html, Size: 1745 bytes --]

bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Andreas,

I think
"dc0cf16c7a Always set 'Vexec_path' before 'Vinvocation_directory' (bug#43137)"
should fix the issue.  Could you have a look if it works for you?

Thanks!

  Andrea

bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> Hi Andreas,
>
> I think
> "dc0cf16c7a Always set 'Vexec_path' before 'Vinvocation_directory' (bug#43137)"
> should fix the issue.  Could you have a look if it works for you?
>

I'm closing this as I believe this is solved.  Please follow-up if some
problem shows-up so we can repoen it.

Thanks

  Andrea

bug#43137: 28.0.50; [feature/native-comp] .eln path fixup confused using relative paths

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andreas Fuchs <asf@boinkor.net> writes:

> I think I found the solution: set_invocation_vars requires Vexec_path
> to be initialized correctly, via init_callproc_1. Somehow, mine ends
> up being something other than $PATH. If I set init_callproc_1 up with
> a double_run_guard the same way set_invocation_var is and call it
> from pdumper_load, it results in a correctly-set Vexec_path variable
> and a correctly-set Vinvocation_directory, as well.
> This is really curious, but I guess it would explain why
> the invocation directory variable is empty! What it doesn't explain
> is, why does this work for you? Maybe your $PATH variable is set up
> differently? (Though I don't see where exec-path would be set
> otherwise...)

I guess is beacuse argv0 is not set on macos so raw_name is empty.
Vexec_path comes into play only in this case.

  Andrea