* pdumping "into" the executable @ 2018-02-26 13:38 Stefan Monnier 2018-02-26 17:00 ` Daniel Colascione 0 siblings, 1 reply; 17+ messages in thread From: Stefan Monnier @ 2018-02-26 13:38 UTC (permalink / raw) To: emacs-devel; +Cc: Daniel Colascione Another way to avoid having a "separate dump" file is to turn things around: make the dump start with "#!/..../emacs --dump-file". Of course, it's got its own set of problems. Stefan ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-26 13:38 pdumping "into" the executable Stefan Monnier @ 2018-02-26 17:00 ` Daniel Colascione 2018-02-26 21:10 ` Tom Tromey 0 siblings, 1 reply; 17+ messages in thread From: Daniel Colascione @ 2018-02-26 17:00 UTC (permalink / raw) To: Stefan Monnier, emacs-devel On 02/26/2018 05:38 AM, Stefan Monnier wrote: > Another way to avoid having a "separate dump" file is to turn things > around: make the dump start with "#!/..../emacs --dump-file". > Of course, it's got its own set of problems. Such as working at all only on unix-ish systems. (Windows batch files aren't quite good enough.) If we're going to go this route, the append-to-the-binary trick is probably the best approach. Its only real disadvantages are that strip(1) removes the dump and that changing the executable would disturb the signature, if we signed the binary. I don't think strip-robustness is all that important, and I'd hope we could re-sign binaries as needed. I have a PoC of the append thing, but for now, I think the separate file approach is working well enough. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-26 17:00 ` Daniel Colascione @ 2018-02-26 21:10 ` Tom Tromey 2018-02-26 21:18 ` dancol 2018-02-26 22:34 ` Clément Pit-Claudel 0 siblings, 2 replies; 17+ messages in thread From: Tom Tromey @ 2018-02-26 21:10 UTC (permalink / raw) To: Daniel Colascione; +Cc: Stefan Monnier, emacs-devel >>>>> "Daniel" == Daniel Colascione <dancol@dancol.org> writes: Daniel> I don't think strip-robustness is all that important, and I'd Daniel> hope we could re-sign binaries as needed. Distros usually build everything with debuginfo and then strip off the debuginfo. So, building this way would require special build hacks for the distros. Maybe instead you could use objcopy to stuff the data into some section in the executable. Or, just turn the dump to a C file, then compile it and do a second link. Aside from (maybe hypothetical) C compiler limits, that would be very portable. Tom ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-26 21:10 ` Tom Tromey @ 2018-02-26 21:18 ` dancol 2018-02-26 22:07 ` Paul Eggert 2018-02-26 22:34 ` Clément Pit-Claudel 1 sibling, 1 reply; 17+ messages in thread From: dancol @ 2018-02-26 21:18 UTC (permalink / raw) To: Tom Tromey; +Cc: Daniel Colascione, Stefan Monnier, emacs-devel >>>>>> "Daniel" == Daniel Colascione <dancol@dancol.org> writes: > > Daniel> I don't think strip-robustness is all that important, and I'd > Daniel> hope we could re-sign binaries as needed. > > Distros usually build everything with debuginfo and then strip off the > debuginfo. So, building this way would require special build hacks for > the distros. > > Maybe instead you could use objcopy to stuff the data into some section > in the executable. Or, just turn the dump to a C file, then compile it > and do a second link. Aside from (maybe hypothetical) C compiler > limits, that would be very portable. But not redumpable. Besides, the problem with a second link is that it might change the relative positions of symbols within Emacs, leading to havoc when Emacs then tries to load the dump. It doesn't feel like a good idea to depend on object layout being stable across different links. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-26 21:18 ` dancol @ 2018-02-26 22:07 ` Paul Eggert 2018-02-26 22:22 ` Daniel Colascione 0 siblings, 1 reply; 17+ messages in thread From: Paul Eggert @ 2018-02-26 22:07 UTC (permalink / raw) To: Daniel Colascione; +Cc: Tom Tromey, Stefan Monnier, emacs-devel On 02/26/2018 01:18 PM, dancol@dancol.org wrote: >> Or, just turn the dump to a C file, then compile it >> and do a second link. Aside from (maybe hypothetical) C compiler >> limits, that would be very portable. > But not redumpable. Why couldn't it be redumpable? All that the user should need is a C compiler. That's not unreasonable. > Besides, the problem with a second link is that it might change the > relative positions of symbols within Emac How could that be a problem? Symbols known to C code are known by their fixed offsets and the second link wouldn't change these offsets, and symbols not known to C code need to be reallocated anyway during loading. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-26 22:07 ` Paul Eggert @ 2018-02-26 22:22 ` Daniel Colascione 2018-02-26 23:17 ` Paul Eggert 0 siblings, 1 reply; 17+ messages in thread From: Daniel Colascione @ 2018-02-26 22:22 UTC (permalink / raw) To: Paul Eggert; +Cc: Tom Tromey, Stefan Monnier, emacs-devel On 12/31/1969 04:00 PM, Paul Eggert wrote: > On 02/26/2018 01:18 PM, dancol@dancol.org wrote: >>> Or, just turn the dump to a C file, then compile it >>> and do a second link. Aside from (maybe hypothetical) C compiler >>> limits, that would be very portable. >> But not redumpable. > > Why couldn't it be redumpable? All that the user should need is a C > compiler. That's not unreasonable. I thought we had this argument back when we were talking about modules. I think it's unacceptable to require a C compiler and the presence of Emacs unlinked objects for the proper operation of an end-user feature. >> Besides, the problem with a second link is that it might change the >> relative positions of symbols within Emac > > How could that be a problem? Symbols known to C code are known by their > fixed offsets and the second link wouldn't change these offsets, and > symbols not known to C code need to be reallocated anyway during loading. The linker is what determines into-translation-unit offsets, not the compiler. The offsets are patched in at link time. There's no particular requirement to perform the link the same way every time. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-26 22:22 ` Daniel Colascione @ 2018-02-26 23:17 ` Paul Eggert 2018-02-26 23:34 ` dancol 2018-02-27 0:09 ` Drew Adams 0 siblings, 2 replies; 17+ messages in thread From: Paul Eggert @ 2018-02-26 23:17 UTC (permalink / raw) To: Daniel Colascione; +Cc: Tom Tromey, Stefan Monnier, emacs-devel On 02/26/2018 02:22 PM, Daniel Colascione wrote: > On 12/31/1969 04:00 PM, Paul Eggert wrote: >> On 02/26/2018 01:18 PM, dancol@dancol.org wrote: >>>> Or, just turn the dump to a C file, then compile it >>>> and do a second link. Aside from (maybe hypothetical) C compiler >>>> limits, that would be very portable. >>> But not redumpable. >> >> Why couldn't it be redumpable? All that the user should need is a C >> compiler. That's not unreasonable. > > I thought we had this argument back when we were talking about > modules. I think it's unacceptable to require a C compiler and the > presence of Emacs unlinked objects for the proper operation of an > end-user feature. Although using a dumped Emacs is an end-user feature, dumping and redumping are not. Dumping and/or redumping are techniques used to build Emacs, and in practice they should be part of a software build system; for such a thing it's reasonable to assume a C compiler. We already assume a C compiler (or equivalent) when building modules; we don't assume a C compiler only when people are using the modules. Dumping is similar. > >>> Besides, the problem with a second link is that it might change the >>> relative positions of symbols within Emac >> >> How could that be a problem? Symbols known to C code are known by >> their fixed offsets and the second link wouldn't change these >> offsets, and symbols not known to C code need to be reallocated >> anyway during loading. > > The linker is what determines into-translation-unit offsets, not the > compiler. The offsets are patched in at link time. There's no > particular requirement to perform the link the same way every time. For Lisp symbols visible to C, Emacs master currently determines symbol offsets at C-compile-time, not at link time. Does pdumper change this? If so, how and why does it change this? And if not, then I don't see the problem. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-26 23:17 ` Paul Eggert @ 2018-02-26 23:34 ` dancol 2018-02-26 23:41 ` dancol 2018-02-27 0:09 ` Drew Adams 1 sibling, 1 reply; 17+ messages in thread From: dancol @ 2018-02-26 23:34 UTC (permalink / raw) To: Paul Eggert; +Cc: Daniel Colascione, Tom Tromey, Stefan Monnier, emacs-devel > On 02/26/2018 02:22 PM, Daniel Colascione wrote: >> On 12/31/1969 04:00 PM, Paul Eggert wrote: >>> On 02/26/2018 01:18 PM, dancol@dancol.org wrote: >>>>> Or, just turn the dump to a C file, then compile it >>>>> and do a second link. Aside from (maybe hypothetical) C compiler >>>>> limits, that would be very portable. >>>> But not redumpable. >>> >>> Why couldn't it be redumpable? All that the user should need is a C >>> compiler. That's not unreasonable. >> >> I thought we had this argument back when we were talking about >> modules. I think it's unacceptable to require a C compiler and the >> presence of Emacs unlinked objects for the proper operation of an >> end-user feature. > > Although using a dumped Emacs is an end-user feature, dumping and > redumping are not. Dumping and/or redumping are techniques used to build > Emacs, and in practice they should be part of a software build system; > for such a thing it's reasonable to assume a C compiler. > > We already assume a C compiler (or equivalent) when building modules; we > don't assume a C compiler only when people are using the modules. > Dumping is similar. > >> >>>> Besides, the problem with a second link is that it might change the >>>> relative positions of symbols within Emac >>> >>> How could that be a problem? Symbols known to C code are known by >>> their fixed offsets and the second link wouldn't change these >>> offsets, and symbols not known to C code need to be reallocated >>> anyway during loading. >> >> The linker is what determines into-translation-unit offsets, not the >> compiler. The offsets are patched in at link time. There's no >> particular requirement to perform the link the same way every time. > > For Lisp symbols visible to C, Emacs master currently determines symbol > offsets at C-compile-time, not at link time. Does pdumper change this? > If so, how and why does it change this? And if not, then I don't see the > problem. The built-in symbols are in one big contiguous array. That's why the scheme works. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-26 23:34 ` dancol @ 2018-02-26 23:41 ` dancol 0 siblings, 0 replies; 17+ messages in thread From: dancol @ 2018-02-26 23:41 UTC (permalink / raw) To: dancol Cc: Tom Tromey, Paul Eggert, Daniel Colascione, Stefan Monnier, emacs-devel >> On 02/26/2018 02:22 PM, Daniel Colascione wrote: >>> On 12/31/1969 04:00 PM, Paul Eggert wrote: >>>> On 02/26/2018 01:18 PM, dancol@dancol.org wrote: >>>>>> Or, just turn the dump to a C file, then compile it >>>>>> and do a second link. Aside from (maybe hypothetical) C compiler >>>>>> limits, that would be very portable. >>>>> But not redumpable. >>>> >>>> Why couldn't it be redumpable? All that the user should need is a C >>>> compiler. That's not unreasonable. >>> >>> I thought we had this argument back when we were talking about >>> modules. I think it's unacceptable to require a C compiler and the >>> presence of Emacs unlinked objects for the proper operation of an >>> end-user feature. >> >> Although using a dumped Emacs is an end-user feature, dumping and >> redumping are not. Dumping and/or redumping are techniques used to build >> Emacs, and in practice they should be part of a software build system; >> for such a thing it's reasonable to assume a C compiler. >> >> We already assume a C compiler (or equivalent) when building modules; we >> don't assume a C compiler only when people are using the modules. >> Dumping is similar. We're talking about dumping an end-user Emacs for random end-user purposes, not the dump that occurs while building Emacs. ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: pdumping "into" the executable 2018-02-26 23:17 ` Paul Eggert 2018-02-26 23:34 ` dancol @ 2018-02-27 0:09 ` Drew Adams 2018-02-27 0:16 ` dancol 1 sibling, 1 reply; 17+ messages in thread From: Drew Adams @ 2018-02-27 0:09 UTC (permalink / raw) To: Paul Eggert, Daniel Colascione; +Cc: Tom Tromey, Stefan Monnier, emacs-devel > > I thought we had this argument back when we were talking about > > modules. I think it's unacceptable to require a C compiler and the > > presence of Emacs unlinked objects for the proper operation of an > > end-user feature. > > Although using a dumped Emacs is an end-user feature, dumping and > redumping are not. Dumping and/or redumping are techniques used to build > Emacs, and in practice they should be part of a software build system; > for such a thing it's reasonable to assume a C compiler. > > We already assume a C compiler (or equivalent) when building modules; we > don't assume a C compiler only when people are using the modules. > Dumping is similar. This probably doesn't help, but I remember dumping Emacs with my Lisp customizations back in the 80s, as an end user, to get quick start-up. I didn't build Emacs myself, and I had no C compiler and didn't need one. No, I don't recall more than that, except that this was with GNU Emacs (probably version 18?) on Unix. I don't recall how I (and others) did it. So if possible, I think it would be good for end users (with no C compiler) to be able to take advantage of this new, portable dumper. ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: pdumping "into" the executable 2018-02-27 0:09 ` Drew Adams @ 2018-02-27 0:16 ` dancol 2018-02-27 7:55 ` John Wiegley 0 siblings, 1 reply; 17+ messages in thread From: dancol @ 2018-02-27 0:16 UTC (permalink / raw) To: Drew Adams Cc: emacs-devel, Paul Eggert, Daniel Colascione, Stefan Monnier, Tom Tromey >> > I thought we had this argument back when we were talking about >> > modules. I think it's unacceptable to require a C compiler and the >> > presence of Emacs unlinked objects for the proper operation of an >> > end-user feature. >> >> Although using a dumped Emacs is an end-user feature, dumping and >> redumping are not. Dumping and/or redumping are techniques used to build >> Emacs, and in practice they should be part of a software build system; >> for such a thing it's reasonable to assume a C compiler. >> >> We already assume a C compiler (or equivalent) when building modules; we >> don't assume a C compiler only when people are using the modules. >> Dumping is similar. > > This probably doesn't help, but I remember dumping Emacs with my > Lisp customizations back in the 80s, as an end user, to get quick > start-up. I didn't build Emacs myself, and I had no C compiler > and didn't need one. No, I don't recall more than that, except > that this was with GNU Emacs (probably version 18?) on Unix. > I don't recall how I (and others) did it. > > So if possible, I think it would be good for end users (with no C > compiler) to be able to take advantage of this new, portable dumper. That is exactly the use case that we'll be able to support again. IMHO, it should be automatic, although I don't know exactly what the startup-file dependency tracking arrangement is going to look like. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-27 0:16 ` dancol @ 2018-02-27 7:55 ` John Wiegley 2018-02-27 13:47 ` Dmitry Gutov 2018-02-27 14:47 ` Stefan Monnier 0 siblings, 2 replies; 17+ messages in thread From: John Wiegley @ 2018-02-27 7:55 UTC (permalink / raw) To: dancol; +Cc: Paul Eggert, Tom Tromey, Stefan Monnier, Drew Adams, emacs-devel >>>>> "d" == dancol <dancol@dancol.org> writes: d> That is exactly the use case that we'll be able to support again. IMHO, it d> should be automatic, although I don't know exactly what the startup-file d> dependency tracking arrangement is going to look like. Will we gain an SBCL-like `save-lisp-and-die' function? -- John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2 ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-27 7:55 ` John Wiegley @ 2018-02-27 13:47 ` Dmitry Gutov 2018-02-27 14:47 ` Stefan Monnier 1 sibling, 0 replies; 17+ messages in thread From: Dmitry Gutov @ 2018-02-27 13:47 UTC (permalink / raw) To: dancol, Drew Adams, emacs-devel, Paul Eggert, Stefan Monnier, Tom Tromey On 2/27/18 9:55 AM, John Wiegley wrote: >>>>>> "d" == dancol <dancol@dancol.org> writes: > > d> That is exactly the use case that we'll be able to support again. IMHO, it > d> should be automatic, although I don't know exactly what the startup-file > d> dependency tracking arrangement is going to look like. > > Will we gain an SBCL-like `save-lisp-and-die' function? As a (more reliable) alternative to desktop-save? Sounds intriguing. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-27 7:55 ` John Wiegley 2018-02-27 13:47 ` Dmitry Gutov @ 2018-02-27 14:47 ` Stefan Monnier 2018-02-27 18:32 ` John Wiegley 1 sibling, 1 reply; 17+ messages in thread From: Stefan Monnier @ 2018-02-27 14:47 UTC (permalink / raw) To: emacs-devel > d> That is exactly the use case that we'll be able to support again. IMHO, it > d> should be automatic, although I don't know exactly what the startup-file > d> dependency tracking arrangement is going to look like. > Will we gain an SBCL-like `save-lisp-and-die' function? Why `-and-die`? Stefan ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-27 14:47 ` Stefan Monnier @ 2018-02-27 18:32 ` John Wiegley 0 siblings, 0 replies; 17+ messages in thread From: John Wiegley @ 2018-02-27 18:32 UTC (permalink / raw) To: Stefan Monnier; +Cc: emacs-devel >>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes: SM> Why `-and-die`? It saves core and exits the Lisp interpreter. I never did use a version that only "took a snapshot", though I can't see why it wouldn't be feasible. -- John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2 ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-26 21:10 ` Tom Tromey 2018-02-26 21:18 ` dancol @ 2018-02-26 22:34 ` Clément Pit-Claudel 2018-02-26 22:47 ` Daniel Colascione 1 sibling, 1 reply; 17+ messages in thread From: Clément Pit-Claudel @ 2018-02-26 22:34 UTC (permalink / raw) To: Tom Tromey, Daniel Colascione; +Cc: Stefan Monnier, emacs-devel On 2018-02-26 16:10, Tom Tromey wrote: > Maybe instead you could use objcopy to stuff the data into some section > in the executable. Or, just turn the dump to a C file, then compile it > and do a second link. Aside from (maybe hypothetical) C compiler > limits, that would be very portable. Note that having (the option to keep) the dump and the main binary separate can be a feature. For example, Emacs is often installed in a location that's writable by root only, and if the dump file can be separate then users can just redump in a directory that they can write to, without having to copy the entire Emacs binary. Separate dump files also makes it easier to experiment with multiple dumps. Clément. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pdumping "into" the executable 2018-02-26 22:34 ` Clément Pit-Claudel @ 2018-02-26 22:47 ` Daniel Colascione 0 siblings, 0 replies; 17+ messages in thread From: Daniel Colascione @ 2018-02-26 22:47 UTC (permalink / raw) To: Clément Pit-Claudel, Tom Tromey; +Cc: Stefan Monnier, emacs-devel On 02/26/2018 02:34 PM, Clément Pit-Claudel wrote: > On 2018-02-26 16:10, Tom Tromey wrote: >> Maybe instead you could use objcopy to stuff the data into some section >> in the executable. Or, just turn the dump to a C file, then compile it >> and do a second link. Aside from (maybe hypothetical) C compiler >> limits, that would be very portable. > > Note that having (the option to keep) the dump and the main binary separate can be a feature. For example, Emacs is often installed in a location that's writable by root only, and if the dump file can be separate then users can just redump in a directory that they can write to, without having to copy the entire Emacs binary. > > Separate dump files also makes it easier to experiment with multiple dumps. I don't think anyone is talking about removing the ability to use discrete dump files. ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2018-02-27 18:32 UTC | newest] Thread overview: 17+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-02-26 13:38 pdumping "into" the executable Stefan Monnier 2018-02-26 17:00 ` Daniel Colascione 2018-02-26 21:10 ` Tom Tromey 2018-02-26 21:18 ` dancol 2018-02-26 22:07 ` Paul Eggert 2018-02-26 22:22 ` Daniel Colascione 2018-02-26 23:17 ` Paul Eggert 2018-02-26 23:34 ` dancol 2018-02-26 23:41 ` dancol 2018-02-27 0:09 ` Drew Adams 2018-02-27 0:16 ` dancol 2018-02-27 7:55 ` John Wiegley 2018-02-27 13:47 ` Dmitry Gutov 2018-02-27 14:47 ` Stefan Monnier 2018-02-27 18:32 ` John Wiegley 2018-02-26 22:34 ` Clément Pit-Claudel 2018-02-26 22:47 ` Daniel Colascione
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/emacs.git https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.