pdumping "into" the executable

pdumping "into" the executable

[Stefan Monnier]

Another way to avoid having a "separate dump" file is to turn things
around: make the dump start with "#!/..../emacs --dump-file".
Of course, it's got its own set of problems.


        Stefan

Re: pdumping "into" the executable

[Daniel Colascione]

On 02/26/2018 05:38 AM, Stefan Monnier wrote:
> Another way to avoid having a "separate dump" file is to turn things
> around: make the dump start with "#!/..../emacs --dump-file".
> Of course, it's got its own set of problems.

Such as working at all only on unix-ish systems. (Windows batch files 
aren't quite good enough.) If we're going to go this route, the 
append-to-the-binary trick is probably the best approach. Its only real 
disadvantages are that strip(1) removes the dump and that changing the 
executable would disturb the signature, if we signed the binary. I don't 
think strip-robustness is all that important, and I'd hope we could 
re-sign binaries as needed.

I have a PoC of the append thing, but for now, I think the separate file 
approach is working well enough.

Re: pdumping "into" the executable

[Tom Tromey]

>>>>> "Daniel" == Daniel Colascione <dancol@dancol.org> writes:

Daniel> I don't think strip-robustness is all that important, and I'd
Daniel> hope we could re-sign binaries as needed.

Distros usually build everything with debuginfo and then strip off the
debuginfo.  So, building this way would require special build hacks for
the distros.

Maybe instead you could use objcopy to stuff the data into some section
in the executable.  Or, just turn the dump to a C file, then compile it
and do a second link.  Aside from (maybe hypothetical) C compiler
limits, that would be very portable.

Tom

Re: pdumping "into" the executable

[dancol]

>>>>>> "Daniel" == Daniel Colascione <dancol@dancol.org> writes:
>
> Daniel> I don't think strip-robustness is all that important, and I'd
> Daniel> hope we could re-sign binaries as needed.
>
> Distros usually build everything with debuginfo and then strip off the
> debuginfo.  So, building this way would require special build hacks for
> the distros.
>
> Maybe instead you could use objcopy to stuff the data into some section
> in the executable.  Or, just turn the dump to a C file, then compile it
> and do a second link.  Aside from (maybe hypothetical) C compiler
> limits, that would be very portable.

But not redumpable.

Besides, the problem with a second link is that it might change the
relative positions of symbols within Emacs, leading to havoc when Emacs
then tries to load the dump. It doesn't feel like a good idea to depend on
object layout being stable across different links.

Re: pdumping "into" the executable

[Paul Eggert]

On 02/26/2018 01:18 PM, dancol@dancol.org wrote:
>> Or, just turn the dump to a C file, then compile it
>> and do a second link.  Aside from (maybe hypothetical) C compiler
>> limits, that would be very portable.
> But not redumpable.

Why couldn't it be redumpable? All that the user should need is a C 
compiler. That's not unreasonable.

> Besides, the problem with a second link is that it might change the
> relative positions of symbols within Emac

How could that be a problem? Symbols known to C code are known by their 
fixed offsets and the second link wouldn't change these offsets, and 
symbols not known to C code need to be reallocated anyway during loading.

Re: pdumping "into" the executable

[Daniel Colascione]

On 12/31/1969 04:00 PM, Paul Eggert wrote:
> On 02/26/2018 01:18 PM, dancol@dancol.org wrote:
>>> Or, just turn the dump to a C file, then compile it
>>> and do a second link.  Aside from (maybe hypothetical) C compiler
>>> limits, that would be very portable.
>> But not redumpable.
> 
> Why couldn't it be redumpable? All that the user should need is a C 
> compiler. That's not unreasonable.

I thought we had this argument back when we were talking about modules. 
I think it's unacceptable to require a C compiler and the presence of 
Emacs unlinked objects for the proper operation of an end-user feature.

>> Besides, the problem with a second link is that it might change the
>> relative positions of symbols within Emac
> 
> How could that be a problem? Symbols known to C code are known by their 
> fixed offsets and the second link wouldn't change these offsets, and 
> symbols not known to C code need to be reallocated anyway during loading.

The linker is what determines into-translation-unit offsets, not the 
compiler. The offsets are patched in at link time. There's no particular 
requirement to perform the link the same way every time.

Re: pdumping "into" the executable

[Paul Eggert]

On 02/26/2018 02:22 PM, Daniel Colascione wrote:
> On 12/31/1969 04:00 PM, Paul Eggert wrote:
>> On 02/26/2018 01:18 PM, dancol@dancol.org wrote:
>>>> Or, just turn the dump to a C file, then compile it
>>>> and do a second link.  Aside from (maybe hypothetical) C compiler
>>>> limits, that would be very portable.
>>> But not redumpable.
>>
>> Why couldn't it be redumpable? All that the user should need is a C 
>> compiler. That's not unreasonable.
>
> I thought we had this argument back when we were talking about 
> modules. I think it's unacceptable to require a C compiler and the 
> presence of Emacs unlinked objects for the proper operation of an 
> end-user feature.

Although using a dumped Emacs is an end-user feature, dumping and 
redumping are not. Dumping and/or redumping are techniques used to build 
Emacs, and in practice they should be part of a software build system; 
for such a thing it's reasonable to assume a C compiler.

We already assume a C compiler (or equivalent) when building modules; we 
don't assume a C compiler only when people are using the modules. 
Dumping is similar.

>
>>> Besides, the problem with a second link is that it might change the
>>> relative positions of symbols within Emac
>>
>> How could that be a problem? Symbols known to C code are known by 
>> their fixed offsets and the second link wouldn't change these 
>> offsets, and symbols not known to C code need to be reallocated 
>> anyway during loading.
>
> The linker is what determines into-translation-unit offsets, not the 
> compiler. The offsets are patched in at link time. There's no 
> particular requirement to perform the link the same way every time.

For Lisp symbols visible to C, Emacs master currently determines symbol 
offsets at C-compile-time, not at link time. Does pdumper change this? 
If so, how and why does it change this? And if not, then I don't see the 
problem.

Re: pdumping "into" the executable

[dancol]

> On 02/26/2018 02:22 PM, Daniel Colascione wrote:
>> On 12/31/1969 04:00 PM, Paul Eggert wrote:
>>> On 02/26/2018 01:18 PM, dancol@dancol.org wrote:
>>>>> Or, just turn the dump to a C file, then compile it
>>>>> and do a second link.  Aside from (maybe hypothetical) C compiler
>>>>> limits, that would be very portable.
>>>> But not redumpable.
>>>
>>> Why couldn't it be redumpable? All that the user should need is a C
>>> compiler. That's not unreasonable.
>>
>> I thought we had this argument back when we were talking about
>> modules. I think it's unacceptable to require a C compiler and the
>> presence of Emacs unlinked objects for the proper operation of an
>> end-user feature.
>
> Although using a dumped Emacs is an end-user feature, dumping and
> redumping are not. Dumping and/or redumping are techniques used to build
> Emacs, and in practice they should be part of a software build system;
> for such a thing it's reasonable to assume a C compiler.
>
> We already assume a C compiler (or equivalent) when building modules; we
> don't assume a C compiler only when people are using the modules.
> Dumping is similar.
>
>>
>>>> Besides, the problem with a second link is that it might change the
>>>> relative positions of symbols within Emac
>>>
>>> How could that be a problem? Symbols known to C code are known by
>>> their fixed offsets and the second link wouldn't change these
>>> offsets, and symbols not known to C code need to be reallocated
>>> anyway during loading.
>>
>> The linker is what determines into-translation-unit offsets, not the
>> compiler. The offsets are patched in at link time. There's no
>> particular requirement to perform the link the same way every time.
>
> For Lisp symbols visible to C, Emacs master currently determines symbol
> offsets at C-compile-time, not at link time. Does pdumper change this?
> If so, how and why does it change this? And if not, then I don't see the
> problem.

The built-in symbols are in one big contiguous array. That's why the
scheme works.

Re: pdumping "into" the executable

[dancol]

>> On 02/26/2018 02:22 PM, Daniel Colascione wrote:
>>> On 12/31/1969 04:00 PM, Paul Eggert wrote:
>>>> On 02/26/2018 01:18 PM, dancol@dancol.org wrote:
>>>>>> Or, just turn the dump to a C file, then compile it
>>>>>> and do a second link.  Aside from (maybe hypothetical) C compiler
>>>>>> limits, that would be very portable.
>>>>> But not redumpable.
>>>>
>>>> Why couldn't it be redumpable? All that the user should need is a C
>>>> compiler. That's not unreasonable.
>>>
>>> I thought we had this argument back when we were talking about
>>> modules. I think it's unacceptable to require a C compiler and the
>>> presence of Emacs unlinked objects for the proper operation of an
>>> end-user feature.
>>
>> Although using a dumped Emacs is an end-user feature, dumping and
>> redumping are not. Dumping and/or redumping are techniques used to build
>> Emacs, and in practice they should be part of a software build system;
>> for such a thing it's reasonable to assume a C compiler.
>>
>> We already assume a C compiler (or equivalent) when building modules; we
>> don't assume a C compiler only when people are using the modules.
>> Dumping is similar.

We're talking about dumping an end-user Emacs for random end-user
purposes, not the dump that occurs while building Emacs.

RE: pdumping "into" the executable

[Drew Adams]

> > I thought we had this argument back when we were talking about
> > modules. I think it's unacceptable to require a C compiler and the
> > presence of Emacs unlinked objects for the proper operation of an
> > end-user feature.
> 
> Although using a dumped Emacs is an end-user feature, dumping and
> redumping are not. Dumping and/or redumping are techniques used to build
> Emacs, and in practice they should be part of a software build system;
> for such a thing it's reasonable to assume a C compiler.
> 
> We already assume a C compiler (or equivalent) when building modules; we
> don't assume a C compiler only when people are using the modules.
> Dumping is similar.

This probably doesn't help, but I remember dumping Emacs with my
Lisp customizations back in the 80s, as an end user, to get quick
start-up.  I didn't build Emacs myself, and I had no C compiler
and didn't need one.  No, I don't recall more than that, except
that this was with GNU Emacs (probably version 18?) on Unix.
I don't recall how I (and others) did it.

So if possible, I think it would be good for end users (with no C
compiler) to be able to take advantage of this new, portable dumper.

RE: pdumping "into" the executable

[dancol]

>> > I thought we had this argument back when we were talking about
>> > modules. I think it's unacceptable to require a C compiler and the
>> > presence of Emacs unlinked objects for the proper operation of an
>> > end-user feature.
>>
>> Although using a dumped Emacs is an end-user feature, dumping and
>> redumping are not. Dumping and/or redumping are techniques used to build
>> Emacs, and in practice they should be part of a software build system;
>> for such a thing it's reasonable to assume a C compiler.
>>
>> We already assume a C compiler (or equivalent) when building modules; we
>> don't assume a C compiler only when people are using the modules.
>> Dumping is similar.
>
> This probably doesn't help, but I remember dumping Emacs with my
> Lisp customizations back in the 80s, as an end user, to get quick
> start-up.  I didn't build Emacs myself, and I had no C compiler
> and didn't need one.  No, I don't recall more than that, except
> that this was with GNU Emacs (probably version 18?) on Unix.
> I don't recall how I (and others) did it.
>
> So if possible, I think it would be good for end users (with no C
> compiler) to be able to take advantage of this new, portable dumper.

That is exactly the use case that we'll be able to support again. IMHO, it
should be automatic, although I don't know exactly what the startup-file
dependency tracking arrangement is going to look like.

Re: pdumping "into" the executable

[John Wiegley]

>>>>> "d" == dancol  <dancol@dancol.org> writes:

d> That is exactly the use case that we'll be able to support again. IMHO, it
d> should be automatic, although I don't know exactly what the startup-file
d> dependency tracking arrangement is going to look like.

Will we gain an SBCL-like `save-lisp-and-die' function?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: pdumping "into" the executable

[Dmitry Gutov]

On 2/27/18 9:55 AM, John Wiegley wrote:
>>>>>> "d" == dancol  <dancol@dancol.org> writes:
> 
> d> That is exactly the use case that we'll be able to support again. IMHO, it
> d> should be automatic, although I don't know exactly what the startup-file
> d> dependency tracking arrangement is going to look like.
> 
> Will we gain an SBCL-like `save-lisp-and-die' function?

As a (more reliable) alternative to desktop-save? Sounds intriguing.

Re: pdumping "into" the executable

[Stefan Monnier]

> d> That is exactly the use case that we'll be able to support again. IMHO, it
> d> should be automatic, although I don't know exactly what the startup-file
> d> dependency tracking arrangement is going to look like.
> Will we gain an SBCL-like `save-lisp-and-die' function?

Why `-and-die`?


        Stefan

Re: pdumping "into" the executable

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> Why `-and-die`?

It saves core and exits the Lisp interpreter. I never did use a version that
only "took a snapshot", though I can't see why it wouldn't be feasible.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: pdumping "into" the executable

[Clément Pit-Claudel]

On 2018-02-26 16:10, Tom Tromey wrote:
> Maybe instead you could use objcopy to stuff the data into some section
> in the executable.  Or, just turn the dump to a C file, then compile it
> and do a second link.  Aside from (maybe hypothetical) C compiler
> limits, that would be very portable.

Note that having (the option to keep) the dump and the main binary separate can be a feature.  For example, Emacs is often installed in a location that's writable by root only, and if the dump file can be separate then users can just redump in a directory that they can write to, without having to copy the entire Emacs binary.

Separate dump files also makes it easier to experiment with multiple dumps.

Clément.

Re: pdumping "into" the executable

[Daniel Colascione]

On 02/26/2018 02:34 PM, Clément Pit-Claudel wrote:
> On 2018-02-26 16:10, Tom Tromey wrote:
>> Maybe instead you could use objcopy to stuff the data into some section
>> in the executable.  Or, just turn the dump to a C file, then compile it
>> and do a second link.  Aside from (maybe hypothetical) C compiler
>> limits, that would be very portable.
> 
> Note that having (the option to keep) the dump and the main binary separate can be a feature.  For example, Emacs is often installed in a location that's writable by root only, and if the dump file can be separate then users can just redump in a directory that they can write to, without having to copy the entire Emacs binary.
> 
> Separate dump files also makes it easier to experiment with multiple dumps.

I don't think anyone is talking about removing the ability to use 
discrete dump files.