bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Stefan Monnier]

Package: Emacs
Version: 27.0.50


Currently, the first .pdmp file that we try to load is found by adding
".pdmp" to argv[0].
This has 2 problems:

1- It fails miserably if argv[0] is a name relative to $PATH since it
   performs the lookup relative to $PWD instead, which is additionally
   a security issue.

2- If the executable named by argv[0] is a symlink, it does not try to
   follow the symlink in case the .pdmp is stored next to the
   destination rather than next to the source.
   

-- Stefan



In GNU Emacs 27.0.50 (build 1, x86_64-unknown-linux-gnu, GTK+ Version 3.24.3)
 of 2019-01-22 built on alfajor
Repository revision: 4e56ca18c9760d9a9429d71e36bedfe4da879a9c
Repository branch: work
Windowing system distributor 'The X.Org Foundation', version 11.0.12003000
System Description: Debian GNU/Linux buster/sid

Recent messages:
Mark set
Auto-saving...done
Saving file /home/monnier/src/emacs/trunk/src/emacs.c...
Wrote /home/monnier/src/emacs/trunk/src/emacs.c
Saving file /home/monnier/src/emacs/trunk/ChangeLog...
Wrote /home/monnier/src/emacs/trunk/ChangeLog
Mark set
Press C-c C-c when you are done editing.
Enter a change comment.  Type C-c C-c when done
Checking in /home/monnier/src/emacs/trunk/src/emacs.c...done

Configured using:
 'configure -C --enable-checking --with-modules --enable-check-lisp-object-type
 'CFLAGS=-Wall -g3 -Og -Wno-pointer-sign'
 PKG_CONFIG_PATH=/home/monnier/lib/pkgconfig'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB
NOTIFY INOTIFY GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS CANNOT_DUMP LCMS2
GMP

Important settings:
  value of $LANG: fr_CH.UTF-8
  locale-coding-system: utf-8-unix

Major mode: InactiveMinibuffer

Minor modes in effect:
  c-electric-flag: t
  shell-dirtrack-mode: t
  diff-auto-refine-mode: t
  electric-pair-mode: t
  global-reveal-mode: t
  reveal-mode: t
  auto-insert-mode: t
  savehist-mode: t
  minibuffer-electric-default-mode: t
  global-compact-docstrings-mode: t
  url-handler-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  global-prettify-symbols-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/monnier/src/emacs/elpa/packages/svg/svg hides /home/monnier/src/emacs/work/lisp/svg
/home/monnier/src/emacs/elpa/packages/ada-mode/ada-mode hides /home/monnier/src/emacs/work/lisp/progmodes/ada-mode
/home/monnier/src/emacs/elpa/packages/ada-mode/ada-stmt hides /home/monnier/src/emacs/work/lisp/progmodes/ada-stmt
/home/monnier/src/emacs/elpa/packages/ada-mode/ada-prj hides /home/monnier/src/emacs/work/lisp/progmodes/ada-prj
/home/monnier/src/emacs/elpa/packages/ada-mode/ada-xref hides /home/monnier/src/emacs/work/lisp/progmodes/ada-xref
/home/monnier/src/emacs/elpa/packages/nadvice/nadvice hides /home/monnier/src/emacs/work/lisp/emacs-lisp/nadvice
/home/monnier/src/emacs/elpa/packages/hyperbole/set hides /home/monnier/src/emacs/work/lisp/emacs-lisp/set
/home/monnier/src/emacs/elpa/packages/landmark/landmark hides /home/monnier/src/emacs/work/lisp/obsolete/landmark
/home/monnier/src/emacs/elpa/packages/crisp/crisp hides /home/monnier/src/emacs/work/lisp/obsolete/crisp

Features:
(sort mail-extr emacsbug log-edit message sendmail rmc puny dired
dired-loaddefs format-spec rfc822 mml mml-sec epa derived epg gnus-util
rmail rmail-loaddefs time-date mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev
mail-utils mailheader pcvs-util bug-reference add-log smerge-mode
whitespace vc vc-dispatcher make-mode pulse cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-langs cc-vars cc-defs
etags multifile generator xref project shell pcomplete grep cl-print
cl-extra help-fns radix-tree sm-c-mode smie misearch multi-isearch
lisp-mnt xscheme byte-opt unsafep trace testcover shadow scheme
re-builder profiler inf-lisp ielm gmm-utils ert pp ewoc debug elp edebug
backtrace find-func cl-indent advice cus-edit cus-start cus-load
wid-edit executable copyright view cal-china lunar solar cal-dst
cal-bahai cal-islam cal-hebrew holidays hol-loaddefs cal-french vc-git
diff-mode filecache diary-lib diary-loaddefs cal-move cal-menu calendar
cal-loaddefs server flymake-proc flymake compile comint ansi-color ring
warnings noutline outline easy-mmode flyspell ispell checkdoc thingatpt
help-mode load-dir elec-pair reveal autoinsert savehist minibuf-eldef
disp-table compact-docstrings cl-seq inline kotl-autoloads proof-site
proof-autoloads realgud-recursive-autoloads finder-inf url-auth info
package easymenu epg-config url-handlers url-parse auth-source eieio
eieio-core cl-macs gv eieio-loaddefs password-cache json map url-vars
seq bytecomp byte-compile cconv cl-loaddefs cl-lib mule-util tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame simple minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help abbrev obarray cl-preloaded
nadvice loaddefs button faces cus-face macroexp files text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 8 239985 30547)
 (symbols 24 18919 0) (strings 16 72751 4613) (string-bytes 1 2323909)
 (vectors 8 43743)
 (vector-slots 4 1324674 45672) (floats 8 584 263) (intervals 28 6233 0)
 (buffers 528 39))

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Daniel Colascione]

On 1/23/19 8:07 AM, Stefan Monnier wrote:
> Package: Emacs
> Version: 27.0.50
> 
> 
> Currently, the first .pdmp file that we try to load is found by adding
> ".pdmp" to argv[0].
> This has 2 problems:
> 
> 1- It fails miserably if argv[0] is a name relative to $PATH since it
>     performs the lookup relative to $PWD instead, which is additionally
>     a security issue.
> 
> 2- If the executable named by argv[0] is a symlink, it does not try to
>     follow the symlink in case the .pdmp is stored next to the
>     destination rather than next to the source.

Yep. We should definitely fix that. realpath on argv[0] seems like the 
right thing.

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Sat, 26 Jan 2019 19:54:29 -0800
> 
> > 1- It fails miserably if argv[0] is a name relative to $PATH since it
> >     performs the lookup relative to $PWD instead, which is additionally
> >     a security issue.
> > 
> > 2- If the executable named by argv[0] is a symlink, it does not try to
> >     follow the symlink in case the .pdmp is stored next to the
> >     destination rather than next to the source.
> 
> Yep. We should definitely fix that. realpath on argv[0] seems like the 
> right thing.

Wouldn't it be better to have a method of finding the absolute file
name of the executable from which the process was started, regardless
of what argv[0] says?  The way to do that is system-dependent (on
GNU/Linux, I believe you need to readlink ("/proc/self/exe") or
somesuch, for example), but AFAIK this can be done on all platforms we
support.

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Wouldn't it be better to have a method of finding the absolute file
> name of the executable from which the process was started, regardless
> of what argv[0] says?  The way to do that is system-dependent (on
> GNU/Linux, I believe you need to readlink ("/proc/self/exe") or
> somesuch, for example), but AFAIK this can be done on all platforms we
> support.

It looks like find_executable from progreloc in gnulib provides a
portable interface for this?  I've added Paul to the CCs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Wouldn't it be better to have a method of finding the absolute file
>> name of the executable from which the process was started, regardless
>> of what argv[0] says?  The way to do that is system-dependent (on
>> GNU/Linux, I believe you need to readlink ("/proc/self/exe") or
>> somesuch, for example), but AFAIK this can be done on all platforms we
>> support.
>
> It looks like find_executable from progreloc in gnulib provides a
> portable interface for this?  I've added Paul to the CCs.

(Wrong address for Paul; resending to fix that.)

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Paul Eggert]

On 10/11/21 7:02 AM, Lars Ingebrigtsen wrote:
> It looks like find_executable from progreloc in gnulib provides a
> portable interface for this?

It does, although it drags in a bunch of other Gnulib modules, as this 
stuff is wildly system-dependent.

For ordinary Emacs installation, I've long thought that a better 
approach is to store the default .pdmp file as a readonly char array 
within the Emacs executable itself. This would be easier for installers, 
sysadmins and users, as it would entail no funny rules about installing 
two files, keeping them in sync, symlinks, PATH, argv[0], relative 
names, security, etc.

Perhaps native compilation effectively does this for us already? If so, 
then the fix for this bug report would be "use native compilation".

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Eli Zaretskii]

> Cc: Daniel Colascione <dancol@dancol.org>, 34180@debbugs.gnu.org,
>  monnier@IRO.UMontreal.CA
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 11 Oct 2021 08:10:56 -0700
> 
> For ordinary Emacs installation, I've long thought that a better 
> approach is to store the default .pdmp file as a readonly char array 
> within the Emacs executable itself. This would be easier for installers, 
> sysadmins and users, as it would entail no funny rules about installing 
> two files, keeping them in sync, symlinks, PATH, argv[0], relative 
> names, security, etc.
> 
> Perhaps native compilation effectively does this for us already?

No, it doesn't, not if I understand what you mean.

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Daniel Colascione]

On 10/11/21 8:10 AM, Paul Eggert wrote:
> On 10/11/21 7:02 AM, Lars Ingebrigtsen wrote:
>> It looks like find_executable from progreloc in gnulib provides a
>> portable interface for this?
> 
> It does, although it drags in a bunch of other Gnulib modules, as this 
> stuff is wildly system-dependent.
> 
> For ordinary Emacs installation, I've long thought that a better 
> approach is to store the default .pdmp file as a readonly char array 
> within the Emacs executable itself. This would be easier for installers, 
> sysadmins and users, as it would entail no funny rules about installing 
> two files, keeping them in sync, symlinks, PATH, argv[0], relative 
> names, security, etc.

It's not quite that simple though. The pdmp file includes offsets of 
data structures within the Emacs executable. Rebuilding the executable 
with a big char array will change these offsets and invalidate the pdmp 
blob you're trying to embed. Now, you could try to guess the size of the 
blob ahead of time, include a dummy embedded array of that size in 
Emacs, dump, and then overwrite the embedded array post-build, but 
there's no guarantee that doing that would actually work on all systems.

I'd rather get out of the business of mucking with executable files even 
if it means we have a bit of extra complexity arising from having to 
deal with out-of-band pdmp files.

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> It's not quite that simple though. The pdmp file includes offsets of data
> structures within the Emacs executable. Rebuilding the executable with a big
> char array will change these offsets and invalidate the pdmp blob you're
> trying to embed. Now, you could try to guess the size of the blob ahead of
> time, include a dummy embedded array of that size in Emacs, dump, and then
> overwrite the embedded array post-build, but there's no guarantee that doing
> that would actually work on all systems.

Maybe we could avoid this problem by moving most of the Emacs executable
to a shared library, so the Emacs executable would be just a `dump`
variable and ` main` function which passes it to an entry point provided
by libemacs<fingerpring>.so`.

I'm not sure I like the idea of building a shared lib and the extra
complexity of making sure the Emacs executable finds it.


        Stefan

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Lars Ingebrigtsen]

Daniel Colascione <dancol@dancol.org> writes:

> I'd rather get out of the business of mucking with executable files
> even if it means we have a bit of extra complexity arising from having
> to deal with out-of-band pdmp files.

Not mucking with the executable files is a definite plus, but on the
other hand -- having a self-contained emacs executable again would also
be really attractive.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Philipp Stephani]

Am Di., 12. Okt. 2021 um 13:27 Uhr schrieb Lars Ingebrigtsen <larsi@gnus.org>:
>
> Daniel Colascione <dancol@dancol.org> writes:
>
> > I'd rather get out of the business of mucking with executable files
> > even if it means we have a bit of extra complexity arising from having
> > to deal with out-of-band pdmp files.
>
> Not mucking with the executable files is a definite plus, but on the
> other hand -- having a self-contained emacs executable again would also
> be really attractive.

Is it really self-contained though? It would still need *.elc files, right?

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Lars Ingebrigtsen]

Philipp Stephani <p.stephani2@gmail.com> writes:

> Is it really self-contained though? It would still need *.elc files, right?

For most things, yes -- but not for basic stuff.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  Eli Zaretskii <eliz@gnu.org>,
>   34180@debbugs.gnu.org,  monnier@IRO.UMontreal.CA
> Date: Tue, 12 Oct 2021 12:51:00 +0200
> 
> Daniel Colascione <dancol@dancol.org> writes:
> 
> > I'd rather get out of the business of mucking with executable files
> > even if it means we have a bit of extra complexity arising from having
> > to deal with out-of-band pdmp files.
> 
> Not mucking with the executable files is a definite plus, but on the
> other hand -- having a self-contained emacs executable again would also
> be really attractive.

But I agree with Daniel that the former is more important than the
latter.  If we do find a way of achieving the latter without risking
the former, then sure.

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Daniel Colascione <dancol@dancol.org>,  34180@debbugs.gnu.org,
>   monnier@IRO.UMontreal.CA, Paul Eggert <paul.eggert@verizon.net>
> Date: Mon, 11 Oct 2021 16:02:44 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Wouldn't it be better to have a method of finding the absolute file
> > name of the executable from which the process was started, regardless
> > of what argv[0] says?  The way to do that is system-dependent (on
> > GNU/Linux, I believe you need to readlink ("/proc/self/exe") or
> > somesuch, for example), but AFAIK this can be done on all platforms we
> > support.
> 
> It looks like find_executable from progreloc in gnulib provides a
> portable interface for this?

We already solved that in load_pdump_find_executable, didn't we?

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> We already solved that in load_pdump_find_executable, didn't we?

Oh yeah, so we did -- it was added half a year after this bug report was
opened, so I guess there's nothing here to fix, and I'm closing this bug
report.

(I guess it's an open question whether the gnulib-equivalent function
has better cross-platform support, but I can't recall seeing any
complaints about not finding the .pdmp file lately...  If it turns out
to be a problem, we can revisit this then.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

For Emacs, finding the executable requires explicitly following
symlinks one by one and testing other related file names.
If you are thinking of using a gnulib function for this,
we need to check very carefully to make sure it does the right thing
in each and every case.

I think we made this work correctly, some months after pdumping was
added.  Did it break since them?
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Mattias Engdegård]

I don't think Paul meant that we necessarily have to use the embedded dump in-place. It could just as well be the source of a memory copy to its runtime location; everything would then work just like today except that the dump file is embedded into the executable.

Basically, regenerating the dump file means relinking the executable and that's it, no mucking about with executables required. It is easy to write a portable solution that works everywhere.

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Daniel Colascione]

On 10/12/21 5:27 AM, Mattias Engdegård wrote:
> I don't think Paul meant that we necessarily have to use the embedded dump in-place. It could just as well be the source of a memory copy to its runtime location; everything would then work just like today except that the dump file is embedded into the executable.

Copying the dump on startup will hurt performance --- the dump is meant 
to be used directly from a disk-backed file.

I'm also not entirely clear on how you're planning on avoiding the usual 
problems with executable modification --- relinking the executable can 
change all the locations of the symbols in the binary, and if symbol 
locations change, any previously-generated dump becomes invalid.

Even if on *some* platforms *today* we can replace an embedded dump 
image in an already-built executable without re-linking the thing, 
there's no guarantee that we can continue doing that. (For example --- 
imagine a future platform that signs all binaries during the build 
process.) Modifying binaries is always a platform-specific thing and 
doing it for Emacs risks forward compatibility. Usin a separate dump 
file relies only on public APIs guaranteed to work forever.

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Eli Zaretskii]

> Cc: Paul Eggert <eggert@cs.ucla.edu>, Lars Ingebrigtsen <larsi@gnus.org>,
>  Eli Zaretskii <eliz@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>,
>  34180@debbugs.gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 12 Oct 2021 09:24:48 -0700
> 
> (For example --- imagine a future platform that signs all binaries
> during the build process.)

That future is already here, on macOS, no?

bug#34180: 27.0.50; argv[0] used incorrectly to find the .pdmp

[Mattias Engdegård]

> Copying the dump on startup will hurt performance --- the dump is meant to be used directly from a disk-backed file.

Is the cost of the all-sequential) paging-in noticeable? Surely a fair part of it will be required more or less immediately anyway.

The worst-case cost of an empty `-batch -Q` run actually decreased somewhat when forcing the dump to be read up front, although admittedly that was from a hot cache.

> relinking the executable can change all the locations of the symbols in the binary, and if symbol locations change, any previously-generated dump becomes invalid.

Partial linking would help, unless we use a diabolic linker that rearranges the symbols based on the contents of a data array.

> Even if on *some* platforms *today* we can replace an embedded dump image in an already-built executable without re-linking the thing, there's no guarantee that we can continue doing that.

No, I don't think that's a viable way. As you say, in-place modification wrecks havoc with any sort of binary checksumming.