Finding the dump

Finding the dump

[Stefan Monnier]

The new pdumper code is very welcome, but of course it introduces some
regressions and new problems:

- When I start my Emacs, it now says

    Loading loadup.el (source)...
    dump mode: nil
    [...]

  and goes on to (re)load the loadup.el instead of using the .pdmp file

  This is because I run my Emacs via a symlink, and load_pdump is not
  careful to try and follow symlinks while looking for the .pdmp next to
  the executable.  I think we should try and handle the use case rather
  than requiring the user to make a second matching symlink to the
  pdmp file.

- I wonder what distributions like Debian will say about having a .pdmp
  file in /usr/bin (AFAICT they normally only have executable files in
  there).  While we can let them hack their solution if they want to
  keep the dumps elsewhere, maybe we should directly add support for
  having dumps elsewhere since that might be useful in general.

  E.g., I think we should also search for the pdmp files in exec-directory.


        Stefan


PS: While "portable dumper" is a perfectly natural name for the code,
    I think the things we save aren't "portable dumps" (the code is
    portable but not the dump) but "heap images", so maybe the file name
    we use should reflect that.

Re: Finding the dump

[Robert Pluim]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> The new pdumper code is very welcome, but of course it introduces some
> regressions and new problems:
>
> - When I start my Emacs, it now says
>
>     Loading loadup.el (source)...
>     dump mode: nil
>     [...]
>
>   and goes on to (re)load the loadup.el instead of using the .pdmp file
>
>   This is because I run my Emacs via a symlink, and load_pdump is not
>   careful to try and follow symlinks while looking for the .pdmp next to
>   the executable.  I think we should try and handle the use case rather
>   than requiring the user to make a second matching symlink to the
>   pdmp file.
>
> - I wonder what distributions like Debian will say about having a .pdmp
>   file in /usr/bin (AFAICT they normally only have executable files in
>   there).  While we can let them hack their solution if they want to
>   keep the dumps elsewhere, maybe we should directly add support for
>   having dumps elsewhere since that might be useful in general.
>
>   E.g., I think we should also search for the pdmp files in exec-directory.
>

Doesnʼt the current code look in PATH_EXEC, which in my build is
/usr/local/libexec/emacs?

Robert

Re: Finding the dump

[Stefan Monnier]

> Doesnʼt the current code look in PATH_EXEC, which in my build is
> /usr/local/libexec/emacs?

Could be.  I tested in a "in-place" build, and it doesn't find the dump
in .../emacs/lib-src


        Stefan

Re: Finding the dump

[Andreas Schwab]

On Jan 23 2019, Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:

> - I wonder what distributions like Debian will say about having a .pdmp
>   file in /usr/bin (AFAICT they normally only have executable files in
>   there).  While we can let them hack their solution if they want to
>   keep the dumps elsewhere, maybe we should directly add support for
>   having dumps elsewhere since that might be useful in general.

IMHO there should be an option to change the default dump name
(argv0_base in load_pdump) in the exec-directory, so that you can have
multiple configurations of emacs sharing the same installation prefix.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Finding the dump

[Stefan Monnier]

>> - I wonder what distributions like Debian will say about having a .pdmp
>>   file in /usr/bin (AFAICT they normally only have executable files in
>>   there).  While we can let them hack their solution if they want to
>>   keep the dumps elsewhere, maybe we should directly add support for
>>   having dumps elsewhere since that might be useful in general.
>
> IMHO there should be an option to change the default dump name
> (argv0_base in load_pdump) in the exec-directory, so that you can have
> multiple configurations of emacs sharing the same installation prefix.

Not completely sure what you mean by that: if you rename `emacs` to
`emacs-foo` then you need to rename `emacs.pdmp` to
`emacs-foo.pdmp` already.

But that reminds me of another direction: could we make the .pdmp images
executable (via a shebang)?


        Stefan

Re: Finding the dump

[Andreas Schwab]

On Jan 23 2019, Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:

> Not completely sure what you mean by that: if you rename `emacs` to
> `emacs-foo` then you need to rename `emacs.pdmp` to
> `emacs-foo.pdmp` already.

No, that doesn't change the name load_pdump uses (it's compiled-in,
after all).

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Finding the dump

[Andreas Schwab]

On Jan 23 2019, Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:

>   This is because I run my Emacs via a symlink, and load_pdump is not
>   careful to try and follow symlinks while looking for the .pdmp next to
>   the executable.

Since argv[0] is almost never absolute, that route doesn't work anyway.
It's even a security bug, since it happily loads a (possibly
manipulated) dump file from the current directory if it has the right
hash.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Finding the dump

[Stefan Monnier]

>>   This is because I run my Emacs via a symlink, and load_pdump is not
>>   careful to try and follow symlinks while looking for the .pdmp next to
>>   the executable.
> Since argv[0] is almost never absolute, that route doesn't work anyway.
> It's even a security bug, since it happily loads a (possibly
> manipulated) dump file from the current directory if it has the right
> hash.

Oh, indeed, it looks for it in the current directory!
I opened a bug report.


        Stefan

Re: Finding the dump

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Wed, 23 Jan 2019 08:15:57 -0500
> 
> The new pdumper code is very welcome, but of course it introduces some
> regressions and new problems:
> 
> - When I start my Emacs, it now says
> 
>     Loading loadup.el (source)...
>     dump mode: nil
>     [...]
> 
>   and goes on to (re)load the loadup.el instead of using the .pdmp file
> 
>   This is because I run my Emacs via a symlink, and load_pdump is not
>   careful to try and follow symlinks while looking for the .pdmp next to
>   the executable.  I think we should try and handle the use case rather
>   than requiring the user to make a second matching symlink to the
>   pdmp file.

This was already mentioned in passing in one of the bug reports.
Please report a bug about this specifically.

> - I wonder what distributions like Debian will say about having a .pdmp
>   file in /usr/bin (AFAICT they normally only have executable files in
>   there).

emacs.pdmp should be installed in data-directory, so it should not be
in /usr/bin, AFAIU.  Daniel, please correct me if I'm wrong.

Re: Finding the dump

[Stefan Monnier]

>> - I wonder what distributions like Debian will say about having a .pdmp
>>   file in /usr/bin (AFAICT they normally only have executable files in
>>   there).
> emacs.pdmp should be installed in data-directory, so it should not be
> in /usr/bin, AFAIU.  Daniel, please correct me if I'm wrong.

Indeed, I now see that this is the main/official place
after installation.  Good.


        Stefan

Re: Finding the dump

[Paul Eggert]

Eli Zaretskii wrote:
> emacs.pdmp should be installed in data-directory, so it should not be
> in /usr/bin, AFAIU.

Yes, /usr/bin should contain only programs intended to be run by users.

Re: Finding the dump

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> - I wonder what distributions like Debian will say about having a .pdmp
>>   file in /usr/bin (AFAICT they normally only have executable files in
>>   there).
>
> emacs.pdmp should be installed in data-directory, so it should not be
> in /usr/bin, AFAIU.  Daniel, please correct me if I'm wrong.

I thought data-directory was for architecture independent files. Do
you mean exec-directory?

Robert

Re: Finding the dump

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,  emacs-devel@gnu.org
> Date: Wed, 23 Jan 2019 17:38:38 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> - I wonder what distributions like Debian will say about having a .pdmp
> >>   file in /usr/bin (AFAICT they normally only have executable files in
> >>   there).
> >
> > emacs.pdmp should be installed in data-directory, so it should not be
> > in /usr/bin, AFAIU.  Daniel, please correct me if I'm wrong.
> 
> I thought data-directory was for architecture independent files. Do
> you mean exec-directory?

Yes, sorry.  And there's a FIXME there to add support for versioned
emacs-X.Y.Z.pdmp files in that case.  IOW, the hardcoded emacs.pdmp
should be only the last resort.

Re: Finding the dump

[Eli Zaretskii]

> Date: Wed, 23 Jan 2019 19:24:30 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > > emacs.pdmp should be installed in data-directory, so it should not be
> > > in /usr/bin, AFAIU.  Daniel, please correct me if I'm wrong.
> > 
> > I thought data-directory was for architecture independent files. Do
> > you mean exec-directory?
> 
> Yes, sorry.  And there's a FIXME there to add support for versioned
> emacs-X.Y.Z.pdmp files in that case.  IOW, the hardcoded emacs.pdmp
> should be only the last resort.

Actually, since exec-directory is versioned, there's no problem with
supporting emacs-X.Y.Z versions.  But I added a feature whereby
renaming the Emacs executable and the pdump file to have the same
basename will also work.

Re: Finding the dump

[Andreas Schwab]

On Jan 26 2019, Eli Zaretskii <eliz@gnu.org> wrote:

> Actually, since exec-directory is versioned, there's no problem with
> supporting emacs-X.Y.Z versions.  But I added a feature whereby
> renaming the Emacs executable and the pdump file to have the same
> basename will also work.

This depends on argv[0] being accurate, though.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Finding the dump

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: rpluim@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 26 Jan 2019 16:03:33 +0100
> 
> On Jan 26 2019, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Actually, since exec-directory is versioned, there's no problem with
> > supporting emacs-X.Y.Z versions.  But I added a feature whereby
> > renaming the Emacs executable and the pdump file to have the same
> > basename will also work.
> 
> This depends on argv[0] being accurate, though.

I don't think I follow.  Can you give an example when it won't work as
intended?

Re: Finding the dump

[Andreas Schwab]

On Jan 26 2019, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: rpluim@gmail.com,  emacs-devel@gnu.org
>> Date: Sat, 26 Jan 2019 16:03:33 +0100
>> 
>> On Jan 26 2019, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> > Actually, since exec-directory is versioned, there's no problem with
>> > supporting emacs-X.Y.Z versions.  But I added a feature whereby
>> > renaming the Emacs executable and the pdump file to have the same
>> > basename will also work.
>> 
>> This depends on argv[0] being accurate, though.
>
> I don't think I follow.  Can you give an example when it won't work as
> intended?

You can set argv[0] to whatever you want.  It doesn't have to match the
executable name in any way.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Finding the dump

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: rpluim@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 26 Jan 2019 16:44:46 +0100
> 
> >> This depends on argv[0] being accurate, though.
> >
> > I don't think I follow.  Can you give an example when it won't work as
> > intended?
> 
> You can set argv[0] to whatever you want.  It doesn't have to match the
> executable name in any way.

OK, but are there practical use cases that you know of where this
could happen as part of a user or a "sane" program invoking Emacs?

Re: Finding the dump

[Andreas Schwab]

On Jan 26 2019, Eli Zaretskii <eliz@gnu.org> wrote:

> OK, but are there practical use cases that you know of where this
> could happen as part of a user or a "sane" program invoking Emacs?

The wrapper script on openSUSE does that, for example.

https://build.opensuse.org/package/view_file/openSUSE:Factory/emacs/emacs.sh?expand=1

Another way is by calling it via a symlink, for example by using
update-alternative.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Finding the dump

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: rpluim@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 26 Jan 2019 21:44:48 +0100
> 
> On Jan 26 2019, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > OK, but are there practical use cases that you know of where this
> > could happen as part of a user or a "sane" program invoking Emacs?
> 
> The wrapper script on openSUSE does that, for example.
> 
> https://build.opensuse.org/package/view_file/openSUSE:Factory/emacs/emacs.sh?expand=1

I guess scripts like that will need to use the --dump-file
command-line argument.

> Another way is by calling it via a symlink

Right, this is the subject of Stefan's bug report, AFAIU.

Re: Finding the dump

[Andreas Schwab]

On Jan 27 2019, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: rpluim@gmail.com,  emacs-devel@gnu.org
>> Date: Sat, 26 Jan 2019 21:44:48 +0100
>> 
>> On Jan 26 2019, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> > OK, but are there practical use cases that you know of where this
>> > could happen as part of a user or a "sane" program invoking Emacs?
>> 
>> The wrapper script on openSUSE does that, for example.
>> 
>> https://build.opensuse.org/package/view_file/openSUSE:Factory/emacs/emacs.sh?expand=1
>
> I guess scripts like that will need to use the --dump-file
> command-line argument.

Having to use --dump-file is awkward.  It would be better if emacs used
a unique name for the dump file.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Finding the dump

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: rpluim@gmail.com,  emacs-devel@gnu.org
> Date: Sun, 27 Jan 2019 16:46:48 +0100
> 
> >> https://build.opensuse.org/package/view_file/openSUSE:Factory/emacs/emacs.sh?expand=1
> >
> > I guess scripts like that will need to use the --dump-file
> > command-line argument.
> 
> Having to use --dump-file is awkward.

When you type it on the command line, yes.  But we are talking about a
shell script, where you add the --dump-file option once and for all.

Re: Finding the dump

[Andreas Schwab]

On Jan 27 2019, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: rpluim@gmail.com,  emacs-devel@gnu.org
>> Date: Sun, 27 Jan 2019 16:46:48 +0100
>> 
>> >> https://build.opensuse.org/package/view_file/openSUSE:Factory/emacs/emacs.sh?expand=1
>> >
>> > I guess scripts like that will need to use the --dump-file
>> > command-line argument.
>> 
>> Having to use --dump-file is awkward.
>
> When you type it on the command line, yes.  But we are talking about a
> shell script, where you add the --dump-file option once and for all.

It's still awkward, because you have to know the full name, which
depends on the version and architecture.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Finding the dump

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Sun, 27 Jan 2019 17:37:39 +0100
> Cc: rpluim@gmail.com, emacs-devel@gnu.org
> 
> >> Having to use --dump-file is awkward.
> >
> > When you type it on the command line, yes.  But we are talking about a
> > shell script, where you add the --dump-file option once and for all.
> 
> It's still awkward, because you have to know the full name, which
> depends on the version and architecture.

So your proposal is to have all the *.pdmp files in the same
directory, regardless of the version and architecture?  That's quite a
deviation from the current arrangement and policies.

Re: Finding the dump

[Andreas Schwab]

On Jan 27 2019, Eli Zaretskii <eliz@gnu.org> wrote:

> So your proposal is to have all the *.pdmp files in the same
> directory, regardless of the version and architecture? 

No, my propsal makes it unnecessary to use --dump-file, because the dump
file has a unique name that emacs is always able to find itself, no
matter how the executable is named or executed.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Finding the dump

[Daniel Colascione]

> On Jan 27 2019, Eli Zaretskii <eliz@gnu.org> wrote:
>
>>> From: Andreas Schwab <schwab@linux-m68k.org>
>>> Cc: rpluim@gmail.com,  emacs-devel@gnu.org
>>> Date: Sun, 27 Jan 2019 16:46:48 +0100
>>>
>>> >> https://build.opensuse.org/package/view_file/openSUSE:Factory/emacs/emacs.sh?expand=1
>>> >
>>> > I guess scripts like that will need to use the --dump-file
>>> > command-line argument.
>>>
>>> Having to use --dump-file is awkward.
>>
>> When you type it on the command line, yes.  But we are talking about a
>> shell script, where you add the --dump-file option once and for all.
>
> It's still awkward, because you have to know the full name, which
> depends on the version and architecture.
>

Launcher scripts are annoying: they break tools that want to see Emacs as
an executable, e.g., ldd, or gdb. I'd much rather Emacs Just Work.

People should have to use --dump-file only if they're using a non-default
Emacs dump, e.g., if they're dumping after loading their customizations.

There's some confusion on this thread. argv[0] *is* reliable, at least on
every system I've seen. Here's the algorithm: look at argv[0]: if it's not
an absolute path, make it absolute by prepending the startup CWD. The
difficulty is that the file to which this now-absolute name points might
be  symlink, so finding emacs.pdmp relative to it won't work. Most
programs in this situation just use realpath(3) or equivalent on that
executable name and find data files relative to the result, and this
approach should work fine for us too.

Re: Finding the dump

[Andreas Schwab]

On Jan 27 2019, "Daniel Colascione" <dancol@dancol.org> wrote:

> There's some confusion on this thread. argv[0] *is* reliable

Nope.  The caller can set argv[0] to any string.  It is in not required
to be related to the name of the executable in any way.

> every system I've seen. Here's the algorithm: look at argv[0]: if it's not
> an absolute path, make it absolute by prepending the startup CWD.

If the executable is found on $PATH then argv[0] is *not* relative to CWD.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Finding the dump

[Stefan Monnier]

>> every system I've seen. Here's the algorithm: look at argv[0]: if it's not
>> an absolute path, make it absolute by prepending the startup CWD.
> If the executable is found on $PATH then argv[0] is *not* relative to CWD.

More to the point, except for the case where `.` is included in $PATH,
I can't see how argv[0] could ever be relative and found in $PWD.


        Stefan

Re: Finding the dump

[Daniel Colascione]

> On Jan 27 2019, "Daniel Colascione" <dancol@dancol.org> wrote:
>
>> There's some confusion on this thread. argv[0] *is* reliable
>
> Nope.  The caller can set argv[0] to any string.  It is in not required
> to be related to the name of the executable in any way.

Sure, but such callers are holding it wrong. If you set argv[0] to some
random string unrelated to Emacs, you break startup. That's fine.

>> every system I've seen. Here's the algorithm: look at argv[0]: if it's
>> not
>> an absolute path, make it absolute by prepending the startup CWD.
>
> If the executable is found on $PATH then argv[0] is *not* relative to CWD.

In that case, argv[0] will be absolute.

Re: Finding the dump

[Stefan Monnier]

> Sure, but such callers are holding it wrong. If you set argv[0] to some
> random string unrelated to Emacs, you break startup. That's fine.

Indeed, this is sufficiently rare in my experience that I wouldn't worry
about that case.

>>> every system I've seen. Here's the algorithm: look at argv[0]: if it's
>>> not an absolute path, make it absolute by prepending the startup CWD.
>> If the executable is found on $PATH then argv[0] is *not* relative to CWD.
> In that case, argv[0] will be absolute.

That's not what I'm seeing on my system.
When I type `emacs` into my shell (zsh under Debian), it is started with
`emacs` as argv[0] and not with an absolute file name.


        Stefan

Re: Finding the dump

[Daniel Colascione]

>> Sure, but such callers are holding it wrong. If you set argv[0] to some
>> random string unrelated to Emacs, you break startup. That's fine.
>
> Indeed, this is sufficiently rare in my experience that I wouldn't worry
> about that case.
>
>>>> every system I've seen. Here's the algorithm: look at argv[0]: if it's
>>>> not an absolute path, make it absolute by prepending the startup CWD.
>>> If the executable is found on $PATH then argv[0] is *not* relative to
>>> CWD.
>> In that case, argv[0] will be absolute.
>
> That's not what I'm seeing on my system.
> When I type `emacs` into my shell (zsh under Debian), it is started with
> `emacs` as argv[0] and not with an absolute file name.

Ah, in my tests, the path to the program run becomes absolute during
shebang execution. For a raw executable, it's indeed pathless, but a
search should PATH should solve that problem. That's what Emacs does today
in init_cmdargs.

Re: Finding the dump

[Andreas Schwab]

On Jan 27 2019, "Daniel Colascione" <dancol@dancol.org> wrote:

>> On Jan 27 2019, "Daniel Colascione" <dancol@dancol.org> wrote:
>>
>>> There's some confusion on this thread. argv[0] *is* reliable
>>
>> Nope.  The caller can set argv[0] to any string.  It is in not required
>> to be related to the name of the executable in any way.
>
> Sure, but such callers are holding it wrong. If you set argv[0] to some
> random string unrelated to Emacs, you break startup. That's fine.

No, emacs should not depend on the contents on argv[0].  There is no
need to do that.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Finding the dump

[Daniel Colascione]

On 1/27/19 12:12 PM, Andreas Schwab wrote:
> On Jan 27 2019, "Daniel Colascione" <dancol@dancol.org> wrote:
> 
>>> On Jan 27 2019, "Daniel Colascione" <dancol@dancol.org> wrote:
>>>
>>>> There's some confusion on this thread. argv[0] *is* reliable
>>>
>>> Nope.  The caller can set argv[0] to any string.  It is in not required
>>> to be related to the name of the executable in any way.
>>
>> Sure, but such callers are holding it wrong. If you set argv[0] to some
>> random string unrelated to Emacs, you break startup. That's fine.
> 
> No, emacs should not depend on the contents on argv[0].  There is no
> need to do that.

Why not? Plenty of other programs do. Looking at realpath(argv[0]) is 
what _everyone_ does.

Besides, don't we already depend on it for executing out of the build 
directory? _After installation_ we can just look in our data directory, 
but if I've just built Emacs and run it our of the source directory 
through some random symlink, how else are we supposed to find it?

Re: Finding the dump

[Andreas Schwab]

On Jan 27 2019, Daniel Colascione <dancol@dancol.org> wrote:

> Why not? Plenty of other programs do. Looking at realpath(argv[0]) is what
> _everyone_ does.

Which one?  They are all broken.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Finding the dump

[Paul Eggert]

Andreas Schwab wrote:
> On Jan 27 2019, Daniel Colascione<dancol@dancol.org>  wrote:
> 
>> Why not? Plenty of other programs do. Looking at realpath(argv[0]) is what
>> _everyone_  does.
> Which one?  They are all broken.

Andreas is quite right here. One cannot trust realpath (argv[0]).

This discussion is reminding me what a hassle that dump file is. I plan to look 
into the possibility of putting the dump file inside the executable (portably, 
of course) so that Emacs startup needn't worry about finding the dump file. This 
would be significantly better for installers and users.

Re: Finding the dump

[Daniel Colascione]

> Andreas Schwab wrote:
>> On Jan 27 2019, Daniel Colascione<dancol@dancol.org>  wrote:
>>
>>> Why not? Plenty of other programs do. Looking at realpath(argv[0]) is
>>> what
>>> _everyone_  does.
>> Which one?  They are all broken.
>
> Andreas is quite right here. One cannot trust realpath (argv[0]).
>
> This discussion is reminding me what a hassle that dump file is. I plan to
> look
> into the possibility of putting the dump file inside the executable
> (portably,
> of course) so that Emacs startup needn't worry about finding the dump
> file. This
> would be significantly better for installers and users.

No, he's wrong. Looking at argv[0] is normal practice. GCC does it. GDB
does it. Python does it. The Perl cookbook tells people to do it. Running
with the wrong argv[0] is an imaginary requirement that achieves nothing
and is required for no real-world use case. When is argv[0] ever actually
wrong?

Besides, we _already_ do the moral equivalent of realpath(argv[0]), in
init_cmdargs, which sets invocation-directory and from there various
internal paths, including installation-directory. If you run Emacs (26,
pre-pdumper) with the wrong argv0, and Emacs isn't installed, then Emacs
falls over and dies, unable to find simple.el. What we should do is use
the argv[0] directory more consistently, not make Emacs brittle and
non-relocatable for no reason.

It's unreal to see something we've done since at least 1994 and that GCC
and GDB *also* do, presumably for just as long, suddenly called some
intolerable bug. No, it's not.

We do need to make the case of argv[0] being a symlink work correctly. I
imagine gnulib has a realpath: we should use it. There's no other bug
here.

Re: Finding the dump

[Paul Eggert]

Daniel Colascione wrote:
> we_already_  do the moral equivalent of realpath(argv[0]), in
> init_cmdargs, which sets invocation-directory and from there various
> internal paths, including installation-directory.

Emacs can run without any of that stuff, so people can run Emacs 26 with argv[0] 
being any string they like. That will stop working if Emacs can't run when 
argv[0] is "wrong".

The GNU Coding Standards have long said that a program's behavior shouldn't 
depend its name (i.e., on the contents of argv[0]), and there are advantages to 
sticking to standard practices even if some programs depart from them.

Re: Finding the dump

[Daniel Colascione]

> Daniel Colascione wrote:
>> we_already_  do the moral equivalent of realpath(argv[0]), in
>> init_cmdargs, which sets invocation-directory and from there various
>> internal paths, including installation-directory.
>
> Emacs can run without any of that stuff, so people can run Emacs 26 with
> argv[0]
> being any string they like. That will stop working if Emacs can't run when
> argv[0] is "wrong".

Emacs can run with a wrong argv[0] if it's installed in its final
location, since we use the installation prefix as a fallback path. Nobody
is arguing that we should remove this feature. We're talking about what we
should do in the case that somebody is running an uninstalled Emacs, and
in this case, an executable-relative search *is* standard practice.

> The GNU Coding Standards have long said that a program's behavior
> shouldn't
> depend its name (i.e., on the contents of argv[0]),

Yet the original GNU program, Emacs, until just now, behaved differently
if it was called temacs. :-)

Anyway, I generally agree that program behavior shouldn't change depending
on its name --- that's why I changed how temacs works --- but finding the
location of the program doesn't count as "behavior". Emacs doesn't care
whether you call it "emacs", "trogdor", or "vim" so long as it can find
"lib-src", "info", and "etc" relative to wherever the binary happens to
be.

> and there are
> advantages to
> sticking to standard practices even if some programs depart from them.

I agree that we should stick to standard practice. That's why we should
continue looking at argv[0] to find Emacs-internal files in uninstalled
Emacs. We have some bugs that make this not quit work today: in order to
successfully run an uninstalled Emacs, you need to 1) have the correct
argv[0], _and_ 2) keep Emacs in its build location. It's #2 that's the
bug, not #1.

Re: Finding the dump

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The GNU Coding Standards have long said that a program's behavior shouldn't 
  > depend its name (i.e., on the contents of argv[0]), 

What this means is that the behavior of the program should
not have a test based on its name.  If you rename /bin/ls to /bin/foo,
it should still function the same.

It is ok to use the directory part of argv[0] to find associated files
for a non-installed program.

I will clarify the text about that.


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Finding the dump

[Paul Eggert]

On 1/28/19 11:03 AM, Richard Stallman wrote:
> It is ok to use the directory part of argv[0] to find associated files
> for a non-installed program.
>
> I will clarify the text about that.

When doing that, please consider the scenario where argv[0] does not 
have a directory part (i.e., does not have a '/' in POSIX). In this case 
in the past it has been reasonably common for argv[0] to not identify 
the executable that would be found by searching PATH for a file with 
that name, either because PATH itself has changed since the parent did 
the search, or because the program is being executed in a chrooted jail, 
or for some other reason. If this usage need not be supported by GNU 
programs it would be helpful to have that clearly stated.

Re: Finding the dump

[Stefan Monnier]

> a directory part (i.e., does not have a '/' in POSIX). In this case in the
> past it has been reasonably common for argv[0] to not identify the
> executable that would be found by searching PATH for a file with that name,

In my experience it is much more in the "corner case" camp than "reasonably
common".  I understand your mileage will vary, but I think the above
wording makes it sound like it's much more frequent than I believe it is.


        Stefan

Re: Finding the dump

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > When doing that, please consider the scenario where argv[0] does not 
  > have a directory part (i.e., does not have a '/' in POSIX). In this case 
  > in the past it has been reasonably common for argv[0] to not identify 
  > the executable that would be found by searching PATH for a file with 
  > that name, either because PATH itself has changed since the parent did 
  > the search, or because the program is being executed in a chrooted jail, 
  > or for some other reason. If this usage need not be supported by GNU 
  > programs it would be helpful to have that clearly stated.

The case exists, but is there anything to be said about it there?
I don't see anything.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Finding the dump

[Paul Eggert]

On 1/29/19 12:50 AM, Richard Stallman wrote:
>    > When doing that, please consider the scenario where argv[0] does not
>    > have a directory part (i.e., does not have a '/' in POSIX). In this case
>    > in the past it has been reasonably common for argv[0] to not identify
>    > the executable that would be found by searching PATH for a file with
>    > that name, either because PATH itself has changed since the parent did
>    > the search, or because the program is being executed in a chrooted jail,
>    > or for some other reason. If this usage need not be supported by GNU
>    > programs it would be helpful to have that clearly stated.
>
> The case exists, but is there anything to be said about it there?
> I don't see anything.

Perhaps you could write something like the following?

Sometimes it is convenient for a program to determine the name of the 
file used to execute the program, in order to re-execute the program or 
to find other files near the executable file. In general argv[0] is not 
a reliable way to do this, as a program's invoker can set argv[0] to any 
string. In practice, if argv[0] contains a slash it normally names the 
executable; otherwise, for various reasons it might not name the 
executable even after searching through the directories named in the 
PATH environment variable. In any case if a program needs its 
installation location it is good practice to let the user specify it in 
some other way, such as via a command-line argument.

Re: Finding the dump

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

What do people think of this?

@node Finding the Executable
@section Finding the Executable

If the program wants to relaunch itself, or find other files that correspond
to its executable file, it should check @code{argv[0]}.  If that string
contains a slash, it is the file name of the executable and its directory
part says where to find other related files.

If there is no slash, you should search for the executable in the
directories in @envvar{PATH}, and other related files should be in
the installation directory for the program's version.
@c ??? Someone please add a cross reference to info about that.

Any program that needs to know the location of its executable or that
of other associated files should offer the user an option to specify
another location.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Finding the dump

[Paul Eggert]

On 1/31/19 4:23 PM, Richard Stallman wrote:
> If the program wants to relaunch itself, or find other files that correspond
> to its executable file, it should check @code{argv[0]}.  If that string
> contains a slash, it is the file name of the executable and its directory
> part says where to find other related files.
>
> If there is no slash, you should search for the executable in the
> directories in @envvar{PATH}, and other related files should be in
> the installation directory for the program's version.
> @c ??? Someone please add a cross reference to info about that.
>
> Any program that needs to know the location of its executable or that
> of other associated files should offer the user an option to specify
> another location.
>
This looks good, except that it's not clear that the guideline is a 
convention that is not enforced by the operating system. Perhaps append 
something like the following?

"Such an option can be useful when @code{argv[0]} does not follow the 
usual convention for naming the executable file."

Re: Finding the dump

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Thanks.  Here is a new version.  See any problems?

Can you tell me where the reference should point to?


If the program wants to relaunch itself, or find other files that correspond
to its executable file, it should check @code{argv[0]}.  If that string
contains a slash, it is the file name of the executable and its directory
part says where to find other related files.

If there is no slash, the program should search for the executable in
the directories in @envvar{PATH}, and other related files should be in
the installation directory for the program's version.
@c ??? Someone please add a cross reference to info about that.

Providing this information in @code{argv[0]} is a convention, not a
guarantee.  Well-behaved programs that launch other programs, such as
shells follow the convention, and your code should follow it too when
launching other programs.  However, any program that needs to know the
location of its executable, or of other associated files, should offer
the user an option to specify the location.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Finding the dump

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, dancol@dancol.org, schwab@linux-m68k.org,
> 	rpluim@gmail.com, emacs-devel@gnu.org
> Date: Fri, 01 Feb 2019 22:25:57 -0500
> 
> If there is no slash, the program should search for the executable in
> the directories in @envvar{PATH}, and other related files should be in

I'm not familiar with @envvar.  Did you mean @env?

Re: Finding the dump

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I'm not familiar with @envvar.  Did you mean @env?

I guess so.  It is a long time since I wrote Texinfo.
-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Finding the dump

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: dancol@dancol.org, eliz@gnu.org, schwab@linux-m68k.org,
> 	rpluim@gmail.com, emacs-devel@gnu.org
> Date: Thu, 31 Jan 2019 19:23:58 -0500
> 
> If the program wants to relaunch itself, or find other files that correspond
> to its executable file, it should check @code{argv[0]}.

I would suggest to use another word instead of "should".  Using
argv[0] has its drawbacks, e.g., if the string there neither has a
slash nor is a file found along PATH -- this could happen when a
program is invoked via a symlink or some other method, or because the
calling program puts there something unrelated to where the executable
lives.  There's a reliable way to find where the executable file of a
running process lives -- on GNU/Linux, this is via /proc/self.  But
saying "should" here could be interpreted to mean that the GNU Coding
Standards frown on any other method of doing this job.

Perhaps "could" or "might".  Or "it is customary to ...".

Re: Finding the dump

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I would suggest to use another word instead of "should".  Using
  > argv[0] has its drawbacks, e.g., if the string there neither has a
  > slash nor is a file found along PATH -- this could happen when a
  > program is invoked via a symlink

How so?  I don't see how this could happen.  If the symlink was found
in PATH to run the program, it should be there when the program looks
for it, except in the case where it was deleted or renamed in the mean
time.

				     or some other method, or because the
  > calling program puts there something unrelated to where the executable
  > lives.

Yes, that can happen, but I think that is an error on the caller's
part.  Anyway, there is no other portable method besides argv[0].  I
don't know whether we want to assume that all kernels for GNU will
support /proc/self.


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Finding the dump

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: schwab@linux-m68k.org, rpluim@gmail.com, eggert@cs.ucla.edu,
> 	dancol@dancol.org, emacs-devel@gnu.org
> Date: Fri, 01 Feb 2019 22:25:50 -0500
> 
>   > I would suggest to use another word instead of "should".  Using
>   > argv[0] has its drawbacks, e.g., if the string there neither has a
>   > slash nor is a file found along PATH -- this could happen when a
>   > program is invoked via a symlink
> 
> How so?  I don't see how this could happen.  If the symlink was found
> in PATH to run the program, it should be there when the program looks
> for it, except in the case where it was deleted or renamed in the mean
> time.

I mean the case where a symlink is to an explicit absolute file name,
and the related files can be found only if you know the target of the
symlink.  But AFAIK argv[0] will not give you the resolved target file
name, it will give you the symlink name.

> 				     or some other method, or because the
>   > calling program puts there something unrelated to where the executable
>   > lives.
> 
> Yes, that can happen, but I think that is an error on the caller's
> part.  Anyway, there is no other portable method besides argv[0].  I
> don't know whether we want to assume that all kernels for GNU will
> support /proc/self.

Maybe so, but still, saying "should" can be interpreted to mean any
other method is discouraged, which I don't think is the case.  I think
this text wants to make a point that in this case it is OK to use
argv[0], unlike when it is used to change the behavior of the program
in some fundamental way.

Re: Finding the dump

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I mean the case where a symlink is to an explicit absolute file name,
  > and the related files can be found only if you know the target of the
  > symlink.  But AFAIK argv[0] will not give you the resolved target file
  > name, it will give you the symlink name.

If you look for the file in PATH you will find the symlink,
and can then find the right directory.

But maybe it is not worth worrying about or documenting specially.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Finding the dump

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, rpluim@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 27 Jan 2019 15:47:06 -0800
> 
> This discussion is reminding me what a hassle that dump file is. I plan to look 
> into the possibility of putting the dump file inside the executable (portably, 
> of course) so that Emacs startup needn't worry about finding the dump file. This 
> would be significantly better for installers and users.

I very much doubt that you could do that portably, let alone allow
repeated dumping after the initial one.

I really don't see why we should waste our energy on making the Emacs
package one file less (out of 3000 it already has).  There's nothing
wrong with having a file we load at startup before we get a fully
functional Emacs.  It certainly isn't a "hassle".

Re: Finding the dump

[Paul Eggert]

Eli Zaretskii wrote:
>> I plan to look
>> into the possibility of putting the dump file inside the executable (portably,
>> of course) so that Emacs startup needn't worry about finding the dump file. This
>> would be significantly better for installers and users.
> I very much doubt that you could do that portably, let alone allow
> repeated dumping after the initial one.

For repeated dumping without a C compiler present, we'd need to stick to 
something like the current approach. But hardly anybody needs repeated dumping, 
and those who do typically have a C compiler available, so the approach that I'm 
thinking of should work for the vast majority of real-world cases.

> I really don't see why we should waste our energy on making the Emacs
> package one file less (out of 3000 it already has).

This particular file is a bigger deal than the rest because Emacs cannot run 
without it. Emacs can run without all those other 3000 files.

> It certainly isn't a "hassle".

It's already been a hassle for us, as seen in this thread. It will be a hassle 
for installers and maintainers too. The separate file also slows down startup 
compared to the approach I have in mind.

But you're the maintainer, so if you are opposed to the idea I'll drop it.

Re: Finding the dump

[Daniel Colascione]

> Eli Zaretskii wrote:
>>> I plan to look
>>> into the possibility of putting the dump file inside the executable
>>> (portably,
>>> of course) so that Emacs startup needn't worry about finding the dump
>>> file. This
>>> would be significantly better for installers and users.
>> I very much doubt that you could do that portably, let alone allow
>> repeated dumping after the initial one.
>
> For repeated dumping without a C compiler present, we'd need to stick to
> something like the current approach. But hardly anybody needs repeated
> dumping,

A lot more people will want repeated dumping once it's known to be viable.
Baseline Emacs starts pretty fast, but starting Emacs for users with tons
of packages and starter kits and things is still painful. With a custom
dump file, all of that can go away. I've gotten a lot of interest in the
approach already.

> and those who do typically have a C compiler available,

Requiring a C compiler at runtime is a non-starter.

>> I really don't see why we should waste our energy on making the Emacs
>> package one file less (out of 3000 it already has).
>
> This particular file is a bigger deal than the rest because Emacs cannot
> run
> without it. Emacs can run without all those other 3000 files.

Not really it can't. There is no single-file distribution of Emacs.

>> It certainly isn't a "hassle".
>
> It's already been a hassle for us, as seen in this thread. It will be a
> hassle
> for installers and maintainers too. The separate file also slows down
> startup
> compared to the approach I have in mind.

I want to get out of the business of specially crafting executable files.
Finding the dump file isn't any additional hassle for maintainers than
finding any of the other numerous files we require today.

Re: Finding the dump

[Eli Zaretskii]

> Cc: schwab@linux-m68k.org, dancol@dancol.org, rpluim@gmail.com,
>  emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 27 Jan 2019 20:13:11 -0800
> 
> > I very much doubt that you could do that portably, let alone allow
> > repeated dumping after the initial one.
> 
> For repeated dumping without a C compiler present, we'd need to stick to 
> something like the current approach. But hardly anybody needs repeated dumping, 
> and those who do typically have a C compiler available, so the approach that I'm 
> thinking of should work for the vast majority of real-world cases.

Like Daniel, I very much disagree that repeated dumping is
unimportant.  I think we want that capability restored very much, and
I think that it will be quite popular.  Having a working compiler as a
prerequisite for that would be a severe limitation.

> > I really don't see why we should waste our energy on making the Emacs
> > package one file less (out of 3000 it already has).
> 
> This particular file is a bigger deal than the rest because Emacs cannot run 
> without it. Emacs can run without all those other 3000 files.

Emacs can run without them, but it is quite limited, so I don't think
we should consider that a significant difference.

> > It certainly isn't a "hassle".
> 
> It's already been a hassle for us, as seen in this thread.

Well, by that measure we have a lot of "hassle" in Emacs already ;-)

> It will be a hassle for installers and maintainers too.

I don't see why.  It's just one more file in libexec, that's all.

> The separate file also slows down startup compared to the approach I
> have in mind.
> 
> But you're the maintainer, so if you are opposed to the idea I'll drop it.

I wouldn't tell you to drop it, certainly not without knowing more
details, but in general I think we should not make any significant
design changes in how the pdumper works, until we (a) stabilize what
we have now (for now, a bug related to pdumper is reported roughly
every other day, so we still have some fallout), and (b) collect some
significant experience with it.  Based on that experience, we might
find good reasons to make some radical changes, but we are not there
yet, and I certainly don't see any serious problems with the current
design at this time, nothing that would justify yet another
significant change in that area.

Re: Finding the dump

[Paul Eggert]

On 1/28/19 7:35 AM, Eli Zaretskii wrote:
> Based on that experience, we might
> find good reasons to make some radical changes

I wasn't thinking of making any significant change to how pdumper works. 
I was thinking only of another mechanism, in *addition* to pdumper. This 
mechanism would make startup faster and installation/operation a bit 
less of a hassle. People could still use pdumper even with an Emacs 
built with the new mechanism.

Re: Finding the dump

[Daniel Colascione]

On 1/28/19 7:57 AM, Paul Eggert wrote:
> On 1/28/19 7:35 AM, Eli Zaretskii wrote:
>> Based on that experience, we might
>> find good reasons to make some radical changes
> 
> I wasn't thinking of making any significant change to how pdumper works. 
> I was thinking only of another mechanism, in *addition* to pdumper. This 
> mechanism would make startup faster and installation/operation a bit 
> less of a hassle. People could still use pdumper even with an Emacs 
> built with the new mechanism.

Where would this performance benefit come from? The overhead of opening 
a file in libexec is negligible.

Re: Finding the dump

[Paul Eggert]

On 1/28/19 8:02 AM, Daniel Colascione wrote:
> The overhead of opening a file in libexec is negligible.

It's not just opening the file; it's reading the file and eagerly 
converting its data to internal format.

Not everybody cares as much about startup latency as I do, and I can 
understand it if the issue does not seem that important to others. But 
on my platform, Emacs master's startup time is more than twice as long 
than (say) Python 3's, and I'd like to see Emacs catching up rather than 
falling further behind.

Re: Finding the dump

[Daniel Colascione]

> On 1/28/19 8:02 AM, Daniel Colascione wrote:
>> The overhead of opening a file in libexec is negligible.
>
> It's not just opening the file; it's reading the file and eagerly
> converting its data to internal format.

Data conversion is slow, which is why pdumper doesn't do it. The dump is a
memory image, not an elc file. We do need to relocate the dump in order to
adjust it for ASLR, but any dumper approach needs to do that, or disable
ASLR.

I experimented with a non-relocatable, relocation-less mode for pdumper
--- just mark Emacs as non-PIC and embed the dump in the emacs data
segment, so it's located at a constant address in memory on each run ---
but it turns out that relocation is so fast, only 1-2ms, that it's really
not worth the trouble. And from a security POV, we shouldn't be
encouraging people to disable ASLR.

I'm not sure what your proposal is, exactly, or how it would differ from
this non-PIC pdumper mode.

> Not everybody cares as much about startup latency as I do, and I can
> understand it if the issue does not seem that important to others. But
> on my platform, Emacs master's startup time is more than twice as long
> than (say) Python 3's, and I'd like to see Emacs catching up rather than
> falling further behind.

One option is to do pdumper relocations on-demand, in a SIGSEGV handler. I
actually implemented this approach, and it works fine, but because we have
a dumb stop-the-world non-generational GC, we ended up paging in most of
the dump on the first GC anyway, so doing relocations on demand doesn't
actually help. If we had a generational GC, we could avoid paging in most
of the dump, speeding both startup and steady-state performance.

But look at it from a different POV: we can make emacs -Q faster, but the
performance of emacs -Q doesn't matter for most users. Most user-visible
startup latency comes from user customization. The single biggest
improvement we can make to observed startup time for typical users is to
use pdumper to automatically checkpoint Emacs after user customization
and, on subsequent startups, skip all the ~/.emacs work. I explicitly
designed pdumper to make this use case possible. This change, along with
automatic .elc maintenance, will do more for real-world Emacs performance
than GC tweaks or incremental improvements to pdumper load performance.

Re: Finding the dump

[Stefan Monnier]

> I plan to look into the possibility of putting the dump file inside
> the executable (portably, of course) so that Emacs startup needn't
> worry about finding the dump file. This would be significantly better
> for installers and users.

Allowing the dump file to be a shebang-executable is easy, doesn't
require a C compiler at dump-time, and works well under GNU/Linux.
It can't be "the only way" because it works less well under MacOS and
not at all under Windows (IIUC), but I see no reason not to support it.


        Stefan

Re: Finding the dump

[Fu Yuan]

FYI shebang file can be easily packaged into a .app on Mac. 

Just create a folder xxxx.app/Contents and rename the executable shebang to the same name as the app. Then put it under Contents . For example Dump.app/Contents/Dump. Now Dump.app is double-clickable.

Yuan

在 2019年1月28日，上午3:53，Stefan Monnier <monnier@iro.umontreal.ca> 写道：

>> I plan to look into the possibility of putting the dump file inside
>> the executable (portably, of course) so that Emacs startup needn't
>> worry about finding the dump file. This would be significantly better
>> for installers and users.
> 
> Allowing the dump file to be a shebang-executable is easy, doesn't
> require a C compiler at dump-time, and works well under GNU/Linux.
> It can't be "the only way" because it works less well under MacOS and
> not at all under Windows (IIUC), but I see no reason not to support it.
> 
> 
>        Stefan
> 
> 

Re: Finding the dump

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 28 Jan 2019 03:53:33 -0500
> 
> > I plan to look into the possibility of putting the dump file inside
> > the executable (portably, of course) so that Emacs startup needn't
> > worry about finding the dump file. This would be significantly better
> > for installers and users.
> 
> Allowing the dump file to be a shebang-executable is easy, doesn't
> require a C compiler at dump-time, and works well under GNU/Linux.
> It can't be "the only way" because it works less well under MacOS and
> not at all under Windows (IIUC), but I see no reason not to support it.

Its relative complexity is such a reason, IMO.

By contrast, opening and reading a file is simple, is portable, and
works well.  The issue with finding it via argv[0] is relatively
simple to solve.  And if we for some reason decide that argv[0] is not
reliable enough, I already mentioned up-thread an alternative of using
platform-specific techniques to obtain the full absolute file name of
the executable without relying on argv[0].

I see no reason to increase complexity and introduce methods that
expose system dependencies on the user level due to the issue that
started this thread.  IME, having OS-dependent differences in Emacs
invocation and installation is a disadvantage, it complicates the user
manual and bug-reporting instructions, if nothing else.

Re: Finding the dump

[Stefan Monnier]

> By contrast, opening and reading a file is simple, is portable, and
> works well.  The issue with finding it via argv[0] is relatively
> simple to solve.  And if we for some reason decide that argv[0] is not
> reliable enough, I already mentioned up-thread an alternative of using
> platform-specific techniques to obtain the full absolute file name of
> the executable without relying on argv[0].

The issue is the following:

User FOO makes his own dump.  Maybe even several different dumps
(e.g. one for his Gnus session, one for his Org session, and one for
his programming sessions).

How is he going to start each one of those?

He can go and create shell scripts that do `emacs --dump-file <foo>`, or
he can manually start his Emacs with the corresponding options, but it's
much simpler for him to make the dumps executable (and have the dumps
look for the only matching `emacs` rather than have `emacs` guess which
dump was meant).

> I see no reason to increase complexity and introduce methods that
> expose system dependencies on the user level due to the issue that
> started this thread.

AFAIK, the only complexity is to (optionally) skip the first page when
loading a dump file.


        Stefan

Re: Finding the dump

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 28 Jan 2019 13:20:24 -0500
> 
> User FOO makes his own dump.  Maybe even several different dumps
> (e.g. one for his Gnus session, one for his Org session, and one for
> his programming sessions).
> 
> How is he going to start each one of those?

Why do we need to support such a strange use case, except by the
existing --dump-file option?  Why wouldn't that user make only one
dump file for all the 3 kinds of session?

In any case, even if this use case is something we should support,
it's purely theoretical at this time.  Let's wait until enough people
want to use Emacs in this strange way, before we invent a solution for
it.

> > I see no reason to increase complexity and introduce methods that
> > expose system dependencies on the user level due to the issue that
> > started this thread.
> 
> AFAIK, the only complexity is to (optionally) skip the first page when
> loading a dump file.

That, and creating such a file in the first place.

Re: Finding the dump

[Stefan Monnier]

>> AFAIK, the only complexity is to (optionally) skip the first page when
>> loading a dump file.
> That, and creating such a file in the first place.

We can let motivated users do that on their own.


        Stefan

Re: Finding the dump

[Stefan Monnier]

>> Actually, since exec-directory is versioned, there's no problem with
>> supporting emacs-X.Y.Z versions.  But I added a feature whereby
>> renaming the Emacs executable and the pdump file to have the same
>> basename will also work.
> This depends on argv[0] being accurate, though.

Given that there can be several dumps for a given `emacs` executable,
I think it would make a lot of sense to try and support the creation of
executable dumps (e.g. with a shebang which only needs to look for the
one-and-only corresponding `emacs` executable regardless of argv[0]).


        Stefan

Re: Finding the dump

[Andreas Schwab]

On Jan 27 2019, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Given that there can be several dumps for a given `emacs` executable,
> I think it would make a lot of sense to try and support the creation of
> executable dumps (e.g. with a shebang which only needs to look for the
> one-and-only corresponding `emacs` executable regardless of argv[0]).

I think the best option would be to add the fingerprint to name of the
dump file.  That way the dump file would have a fixed name, independent
of how the corresponding emacs binaries is named, or how it is called.
It would just try to load emacs-${fingerprint}.pdmp, and always find the
right one.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Finding the dump

[Stefan Monnier]

>> Given that there can be several dumps for a given `emacs` executable,
>> I think it would make a lot of sense to try and support the creation of
>> executable dumps (e.g. with a shebang which only needs to look for the
>> one-and-only corresponding `emacs` executable regardless of argv[0]).
>
> I think the best option would be to add the fingerprint to name of the
> dump file.  That way the dump file would have a fixed name, independent
> of how the corresponding emacs binaries is named, or how it is called.
> It would just try to load emacs-${fingerprint}.pdmp, and always find the
> right one.

That prevents having several dumps for the same executable and that
doesn't help finding the directory in which the dump is stored.

I agree we should add the fingerprint, but I think it should be added to
the `emacs` executable (so the executable-dump can easily find it).


        Stefan

Re: Finding the dump

[Andreas Schwab]

On Jan 27 2019, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> That prevents having several dumps for the same executable and that
> doesn't help finding the directory in which the dump is stored.

That problem already exists today, adding the fingerprint doesn't make
it worse.  IMHO the dump file should really only be searched in
exec-directory.  If you want to find it somewhere else, just pass
--dump-file (or a new option --dump-directory).

> I agree we should add the fingerprint, but I think it should be added to
> the `emacs` executable (so the executable-dump can easily find it).

And installing the emacs executable in exec-directory?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Finding the dump

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Sun, 27 Jan 2019 11:54:01 +0100
> Cc: emacs-devel@gnu.org
> 
> On Jan 27 2019, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
> > That prevents having several dumps for the same executable and that
> > doesn't help finding the directory in which the dump is stored.
> 
> That problem already exists today, adding the fingerprint doesn't make
> it worse.  IMHO the dump file should really only be searched in
> exec-directory.  If you want to find it somewhere else, just pass
> --dump-file (or a new option --dump-directory).

Indeed.  In fact, Stefan's suggestion is just a slightly fancy (and
less portable) way of having an executable script which does

  /foo/bar/emacs --dump-file=/wherever/something.pdmp

This is something users with certain needs and workflows may wish
doing, but I see no reason why we as a project would need to force
everyone on using such a setup.  I see no advantages in it that would
justify it.

Re: Finding the dump

[Stefan Monnier]

> Indeed.  In fact, Stefan's suggestion is just a slightly fancy (and
> less portable) way of having an executable script which does
>   /foo/bar/emacs --dump-file=/wherever/something.pdmp
> This is something users with certain needs and workflows may wish
> doing, but I see no reason why we as a project would need to force
> everyone on using such a setup.

If the format of the dump file were changed to allow the first page to
be skipped, it'd be for end-users to make it work without needing an
extra file.

> I see no advantages in it that would justify it.

The only reason I can see not to want to support such a use case is the
need to keep using the current setup (presumably because it's
hard/inconvenient to support the other setup everywhere, i.e. because
shebangs don't work in Windows and because they don't work quite right
in MacOS (IIUC they'd break string using "#!emacs --script")).

Other than that, making the dump be the file that is put in $PATH is the
better option.


        Stefan

Re: Finding the dump

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Andreas Schwab <schwab@linux-m68k.org>,  emacs-devel@gnu.org
> Date: Sun, 27 Jan 2019 13:52:03 -0500
> 
> > Indeed.  In fact, Stefan's suggestion is just a slightly fancy (and
> > less portable) way of having an executable script which does
> >   /foo/bar/emacs --dump-file=/wherever/something.pdmp
> > This is something users with certain needs and workflows may wish
> > doing, but I see no reason why we as a project would need to force
> > everyone on using such a setup.
> 
> If the format of the dump file were changed to allow the first page to
> be skipped, it'd be for end-users to make it work without needing an
> extra file.

The Emacs installation comes with some 3000 files; why should we care
about one more or less?  The energy we will have spent on finding a
way to do what you want, and do that in a way that won't get us back
into the same kind of trouble as unexec, is better invested in other
areas.

> > I see no advantages in it that would justify it.
> 
> The only reason I can see not to want to support such a use case is the
> need to keep using the current setup (presumably because it's
> hard/inconvenient to support the other setup everywhere, i.e. because
> shebangs don't work in Windows and because they don't work quite right
> in MacOS (IIUC they'd break string using "#!emacs --script")).
> 
> Other than that, making the dump be the file that is put in $PATH is the
> better option.

It's just not economical, IMO.  The disadvantages you enumerate are
already enough to not like this.

Re: Finding the dump

[Stefan Monnier]

> it worse.  IMHO the dump file should really only be searched in
> exec-directory.

Fine by me, assuming this also includes `lib-src` for the case where
Emacs is run in-place.

>> I agree we should add the fingerprint, but I think it should be added to
>> the `emacs` executable (so the executable-dump can easily find it).
> And installing the emacs executable in exec-directory?

Right.


        Stefan

Re: Finding the dump

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> Actually, since exec-directory is versioned, there's no problem with
> supporting emacs-X.Y.Z versions.  But I added a feature whereby
> renaming the Emacs executable and the pdump file to have the same
> basename will also work.

Seems your last commit broke Emacs for me: "emacs" doesn't start, I only
get an error saying the dump can't be found.  Returning to ce085f1d "*
lisp/loadup.el (load-file-name): Set back to nil" from Fri Jan 25 16:15
makes Emacs start again.

Michael.

Re: Finding the dump

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: emacs-devel@gnu.org
> Date: Sat, 26 Jan 2019 18:27:35 +0100
> 
> Seems your last commit broke Emacs for me: "emacs" doesn't start, I only
> get an error saying the dump can't be found.  Returning to ce085f1d "*
> lisp/loadup.el (load-file-name): Set back to nil" from Fri Jan 25 16:15
> makes Emacs start again.

How do you run Emacs -- installed or from the src directory where it
was built?  Or in some other way?

Re: Finding the dump

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> How do you run Emacs -- installed or from the src directory where it
> was built?  Or in some other way?

Installed.

Michael.

Re: Finding the dump

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: emacs-devel@gnu.org
> Date: Sat, 26 Jan 2019 19:05:25 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How do you run Emacs -- installed or from the src directory where it
> > was built?  Or in some other way?
> 
> Installed.

Right, then please try the latest master.

Re: Finding the dump

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> Right, then please try the latest master.

Fixed for me.


Thanks,

Michael.

Re: Finding the dump

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: emacs-devel@gnu.org
> Date: Sat, 26 Jan 2019 21:05:54 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Right, then please try the latest master.
> 
> Fixed for me.

Thanks, and sorry for the mess I made.

Re: Finding the dump

[Van L]

> >> > supporting emacs-X.Y.Z versions.  But I added a feature whereby
> >> > renaming the Emacs executable and the pdump file to have the same
> >> > basename will also work.
> >>
> >> This depends on argv[0] being accurate, though.
> >
> > I don't think I follow.  Can you give an example when it won't work as
> > intended?
>
> You can set argv[0] to whatever you want.  It doesn't have to match the
> executable name in any way.

As an example use-case, I have links in ~/bin for emacs like

  emacs-22 -> /usr/bin/emacs
  emacs-26-mac -> /Applications/MacPorts/EmacsMac.app/Contents/MacOS/bin/emacs
  emacs-27 -> /opt/local/bin/emacs-27.0.50

and, there's

  /opt/local/bin/emacs -> /opt/local/bin/emacs-27.0.50

if that will be the cause of trouble ahead.