Re: Preview: portable dumper

Re: Preview: portable dumper

[David Requena Zabala]

Paul Eggert wrote:

> I sympathize with Daniel about the politics of this. This is not the sort of change that we'd ordinarily create a new branch for, and the way this is being handled will likely discourage further contributions.

While I cannot really make an informed judgment merits of the portable dumper versus the sometime-to-come fast one-big-elc-file, and won't touch the branch vs master issue,  I can tell you this:

The only argument against portable dumper adoption I've seen amounts to "it's coded in C", and the perceived (bad) consequences of the fact.

Of course surely I'm not representative of the general programmer population. Stil, as a once would be emacs C contributor, I sense a definite avoid-anything-C-just-for-being-C atmosphere in the project.

You might remember me from a few months ago, asking about non-toolkit scrollbars on win32 emacs.
At that point:

- I was willing to learn about learning enough about emacs core and win32 to get those scrollbars.
- I was willing to incorporate any requested changes to better integrate into emacs code base, coding style guidelines, whatever...
- I was willing to go through the burden of the copyright attribution process.
- I wasn't willing to argue ad nausea about the merits of yet another feature implemented in C.

So these scrollbars never left my own private emacs build. In the process I never took the chance of leveraging my newly acquired knowledge and become a more regular emacs contributor.

One might wonder to which degree the current C hacker scarcity in the project could actually be caused by the very attitude the Project management holds against C features.

Re: Preview: portable dumper

[Filipe Silva]

[-- Attachment #1: Type: text/plain, Size: 2071 bytes --]

I agree with Daniel here (both Daniels). There is no shortage of linux
kernel hackers and​ it is written in C. Plus I think that is a consensus
that C is not dying anytime soon.

Can't say the same for elisp.

On Thu, Dec 1, 2016 at 4:50 PM, David Requena Zabala <
DRequena@gamalquiler.com> wrote:

> Paul Eggert wrote:
>
> > I sympathize with Daniel about the politics of this. This is not the
> sort of change that we'd ordinarily create a new branch for, and the way
> this is being handled will likely discourage further contributions.
>
> While I cannot really make an informed judgment merits of the portable
> dumper versus the sometime-to-come fast one-big-elc-file, and won't touch
> the branch vs master issue,  I can tell you this:
>
> The only argument against portable dumper adoption I've seen amounts to
> "it's coded in C", and the perceived (bad) consequences of the fact.
>
> Of course surely I'm not representative of the general programmer
> population. Stil, as a once would be emacs C contributor, I sense a
> definite avoid-anything-C-just-for-being-C atmosphere in the project.
>
> You might remember me from a few months ago, asking about non-toolkit
> scrollbars on win32 emacs.
> At that point:
>
> - I was willing to learn about learning enough about emacs core and win32
> to get those scrollbars.
> - I was willing to incorporate any requested changes to better integrate
> into emacs code base, coding style guidelines, whatever...
> - I was willing to go through the burden of the copyright attribution
> process.
> - I wasn't willing to argue ad nausea about the merits of yet another
> feature implemented in C.
>
> So these scrollbars never left my own private emacs build. In the process
> I never took the chance of leveraging my newly acquired knowledge and
> become a more regular emacs contributor.
>
> One might wonder to which degree the current C hacker scarcity in the
> project could actually be caused by the very attitude the Project
> management holds against C features.
>

[-- Attachment #2: Type: text/html, Size: 2886 bytes --]

Re: Preview: portable dumper

[John Wiegley]

>>>>> "FS" == Filipe Silva <filipe.silva@gmail.com> writes:

FS> I agree with Daniel here (both Daniels). There is no shortage of linux
FS> kernel hackers and​ it is written in C. Plus I think that is a consensus
FS> that C is not dying anytime soon.

I'd like to see less C, if only because I've never hacked on the C part of
Emacs, and find almost anything Emacs Lisp to be much simpler and more fun to
work on.

However, C is far from being the most difficult language out there. People are
quite excited about Rust these days, and it appears much harder to fully grok
than C.

If smart people want Emacs to do something, they'll code a machine emulator in
Minecraft to output the correct code using musical frequencies and LED lights.
You'd be shocked at what 15 year olds can do these days when they're bored.
I'm not worried about the intelligence or future willingness. What WE can do,
as Emacs developers, is to be as welcoming and supportive as possible, while
helping those upcoming geniuses to understand that unless they're willing to
stick around and help maintain their new additions, we might be inclined to
say "better in Emacs Lisp, please".

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Eli Zaretskii]

> From: David Requena Zabala <DRequena@gamalquiler.com>
> Date: Thu, 1 Dec 2016 19:50:28 +0100
> 
> 
> You might remember me from a few months ago, asking about non-toolkit scrollbars on win32 emacs.
> At that point:
> 
> - I was willing to learn about learning enough about emacs core and win32 to get those scrollbars.
> - I was willing to incorporate any requested changes to better integrate into emacs code base, coding style guidelines, whatever...
> - I was willing to go through the burden of the copyright attribution process.
> - I wasn't willing to argue ad nausea about the merits of yet another feature implemented in C.
> 
> So these scrollbars never left my own private emacs build. In the process I never took the chance of leveraging my newly acquired knowledge and become a more regular emacs contributor.
> 
> One might wonder to which degree the current C hacker scarcity in the project could actually be caused by the very attitude the Project management holds against C features.

I'm not sure I'm following your logic.  The only discussion with you
about scroll bars I found in the archives is the one that started
here:

  http://lists.gnu.org/archive/html/emacs-devel/2015-01/msg00525.html
  http://lists.gnu.org/archive/html/emacs-devel/2015-01/msg00549.html

In that discussion, you asked technical questions about the
implementation of scroll-bar support in Emacs, on X and on MS-Windows,
and I answered your questions by pointing to the relevant functions
which implement that support.  At no time in that discussion you
proposed something, let alone some code, that was rejected.  What am I
missing?

Re: Preview: portable dumper

[David Requena Zabala]

>In that discussion, you asked technical questions about the implementation of scroll-bar support in Emacs, on X and on MS-Windows,
and I answered your questions by pointing to the relevant functions which implement that support.

Thanks for that.‎ I also seem to remember you bluntly denied a related issue with the configure script which you latter corrected :-)

> At no time in that discussion you proposed something, let alone some code, that was rejected.

This! no code, good or bad, came ever out of this to be proposed for inclusion into emacs.
And that's for a very definite reason: the novice who wrote it wasn't willing to put up a fight just because no new C code is welcomed any longer into emacs. 

As a consequence I pretty much doubt I'll ever become seasoned enough as to be of any help in proper emacs maintenance. IOW, I won't become one of those much needed C coneisseur maintainers. 

> What am I missing?

If I weren't any wiser I'd say you missed the hole point of my message ;-)

Look, this was intended as an outsider take on the broader issue of avoiding C code at all costs. At some point decisions have to be made and the more points of view considered, the better those will be. 

At the moment we have:  x >= 1. Where x is the number of once would be C maintainers put aback by the issue at hand.

I don't know the actual magnitude of x; you don't know either. But it's you who seems most worried about the alleged C programmers drought. 

Re: Preview: portable dumper

[Óscar Fuentes]

David Requena Zabala <DRequena@gamalquiler.com> writes:

>>In that discussion, you asked technical questions about
>>the implementation of scroll-bar support in Emacs, on X and on
>>MS-Windows,
>
> and I answered your questions by pointing to the relevant
> functions which implement that support.
>
> Thanks for that.‎ I also seem to remember you bluntly denied a related
> issue with the configure script which you latter corrected :-)

I don't see it that way. You described a problem, Eli mentioned that he
was unable to reproduce it and asked for details. If there is any
bluntness on his message, blame it onto the limitations of written
communication combined with a very busy maintainer.

>> At no time in that discussion you proposed something, let alone some
>> code, that was rejected.
>
> This! no code, good or bad, came ever out of this to be proposed for
> inclusion into emacs. And that's for a very definite reason: the
> novice who wrote it wasn't willing to put up a fight just because no
> new C code is welcomed any longer into emacs. 

Where on that exchange was mentioned that a C-based contribution would
be problematic? Quite the contrary, when you asked:

   At any rate, if someone was to implement an hypothetical
   '--without-w32-toolkit-scroll-bars', would it take entering much into
   the elisp realm?

Eli replied:

   No, it shouldn't touch the Lisp level at all. The implementation of
   the scroll bars is entirely on the C level.

followed by a 8-line paragraph describing where to begin if you are
interested on implementing the feature.

So, by taking the time to guide you, Eli in fact was explicitly saying
that your C-based feature would be welcomed.

Please, it is very important to be rigorous with those matters. You are
accusing a maintainer of doing something toxic for the project when in
fact he did the opposite. By spreading such misinformation on a public
forum it is you who are discouraging would-be contributors.

[snip]

Re: Preview: portable dumper

[Eli Zaretskii]

> From: David Requena Zabala <DRequena@gamalquiler.com>
> Date: Thu, 1 Dec 2016 23:13:27 +0100
> 
> >In that discussion, you asked technical questions about the implementation of scroll-bar support in Emacs, on X and on MS-Windows,
> and I answered your questions by pointing to the relevant functions which implement that support.
> 
> Thanks for that.‎ I also seem to remember you bluntly denied a related issue with the configure script which you latter corrected :-)

AFAIR, you used a checkout of the development sources that was behind
the repository tip.  But maybe I'm misremembering things, this was
almost 2 years ago.

> > At no time in that discussion you proposed something, let alone some code, that was rejected.
> 
> This! no code, good or bad, came ever out of this to be proposed for inclusion into emacs.

AFAIK, you never proposed any code.

> And that's for a very definite reason: the novice who wrote it wasn't willing to put up a fight just because no new C code is welcomed any longer into emacs. 

I don't know where you got the idea that no new C code is welcome, or
from whom.  We are adding C code to Emacs all the time.  Just look at
the Git log.  I've re-read that past discussion, and didn't see any
sign of such a message in anything I wrote there.  This must be some
gross misunderstanding.  If something I wrote is the culprit, I
apologize.

RE: Preview: portable dumper

[David Requena Zabala]

I feel I'm not helping very much so I intend to refrain from further commenting on these side remarks, not really related to the main point of my 2-cents-style note.

> De: Eli Zaretskii [mailto:eliz@gnu.org] 
> Enviado el: viernes, 2 de diciembre de 2016 8:29

> AFAIR, you used a checkout of the development sources that was behind the repository tip.  But maybe I'm misremembering things, this was almost 2 years ago.

Yeah, memory is like that. That did actually happen, just later.
Previously I had made an observation saying that the configure script shouldn't accept target platform incompatible options, leading to a crash dumping emacs build. After some agreement by other seasoned list members you proceeded with the fix, which in turn I managed to miss!

Anyway no sense pursuing this subject any further. It's not as if I held any grudge about that particular exchange.

> > > At no time in that discussion you proposed something, let alone some code, that was rejected.
> > This! no code, good or bad, came ever out of this to be proposed for inclusion into emacs.
>AFAIK, you never proposed any code.

Again: that's precisely the case :-)

> > And that's for a very definite reason: the novice who wrote it wasn't 
> > willing to put up a fight just because no new C code is welcomed any longer into emacs.
> I don't know where you got the idea that no new C code is welcome, or from whom.  We are adding C code to Emacs all the time.  Just look at the Git log.  I've re-read that past discussion, and didn't see any sign of such a message in anything I wrote there.

I've been reading the list ever since. As noted in my first comment, it's a distinct impression which bulds up over time from regularly reading this list.
By its very nature this is admitedly subjective. Maybe it's generaly shared by newcomers as me or maybe not.
As such the point is not really debatable.
 
>  This must be some gross misunderstanding.

Of course it is. I failed miserabily to get my point through.

>  If something I wrote is the culprit, I apologize.

My fault, no worries. No need to apologize, lets just leave it at that.


Date: Fri, 02 Dec 2016 01:30:53 +0100
From: Óscar Fuentes <ofv@wanadoo.es>

> I don't see it that way. You described a problem, Eli mentioned that he was unable to reproduce it and asked for details. If there is any bluntness on his message, blame it onto the limitations of written communication combined with a very busy maintainer.

I don't have anyone to blame.
If I had, there'd be much better candidates as my own poor English, or issues of (yet to be built) trust, as exposed in Mark's brilliant post.

> followed by a 8-line paragraph describing where to begin if you are interested on implementing the feature.

But, but, but... I did thank him!

> Please, it is very important to be rigorous with those matters. You are accusing a maintainer of doing something toxic for the project when in fact he did the opposite

As in all human communication there's always how a given statement is meant, and how it is received. Both sides unknowingly apply their own filters.
I never intended to imply anything in that vein. 

> By spreading such misinformation on a public forum it is you who are discouraging would-be contributors.

That being the case I gather I must apologize here, which I wholeheartedly do.
I don't want any involvement in any further discouragement, lead maintainership stepping down, or emacs forking whatsoever.

TBH Óscar, it's a bit odd being corresponding with you about this very subject. You were truly encouraging and supportive to me at the time.

So that's it. Lets have the next generation emacs hackers keep coming to the list and drawing their own conclusions.

Best regards

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Support for Windows, a proprietary (thus, unjust) operating system, is
not high on our priorities, since our overall goal is to put an end
to Windows and other nonfree software.

Are you interested in writing something that would make GNU Emacs do a
better job as a part of the free world?  If so, we would be
enthusiastic about giving you whatever help and encouragement you need
in order to master the code.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Angelo Graziosi]

Eli Zaretskii wrote:
>
> I'd urge people to try this branch and report any issues they see.

What should we look for?

  Angelo

Re: Preview: portable dumper

[Angelo Graziosi]

> Il 13 febbraio 2018 alle 23.06 Angelo Graziosi <angelo.g0@libero.it> ha scritto:
> 
> 
> Eli Zaretskii wrote:
> >
> > I'd urge people to try this branch and report any issues they see.
> 
> What should we look for?
> 

Meanwhile... here the build fails (MSYS2/MinGW64):

[...]
pdumper.c: In function 'dump_read_all':
pdumper.c:4723:45: error: conversion to 'unsigned int' from 'size_t {aka long long unsigned int}' may alter its value [-Werror=conversion]
         read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
                                             ^~~~~~~~~~~~~
  CC       data.o
cc1.exe: some warnings being treated as errors
...

>   Angelo

Re: Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/html, Size: 107 bytes --]

Re: Preview: portable dumper

[Eli Zaretskii]

> Date: Tue, 13 Feb 2018 23:28:03 +0100 (CET)
> From: Angelo Graziosi <angelo.g0@libero.it>
> 
> pdumper.c: In function 'dump_read_all':
> pdumper.c:4723:45: error: conversion to 'unsigned int' from 'size_t {aka long long unsigned int}' may alter its value [-Werror=conversion]
>          read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
>                                              ^~~~~~~~~~~~~

This is because the MS-Windows version of 'read' has 'unsigned int' as
its last argument.  But now I wonder why you don't see the same
problem in sysdep.c, where emacs_read does something very similar.
Not sure what am I missing here.

Re: Preview: portable dumper

[Angelo Graziosi]

Just for the record,

> Meanwhile... here the build fails (MSYS2/MinGW64):
> 
> [...]
> pdumper.c: In function 'dump_read_all':
> pdumper.c:4723:45: error: conversion to 'unsigned int' from 'size_t {aka long long unsigned int}' may alter its value [-Werror=conversion]
>          read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
>                                              ^~~~~~~~~~~~~
>   CC       data.o
> cc1.exe: some warnings being treated as errors

I have seen you have commit many fixes but here the build still fails in the same manner.. Maybe you have not found the right solution yet.. 

Thanks,
  Angelo.

Re: Preview: portable dumper

[Daniel Colascione]

On 02/16/2018 01:14 PM, Angelo Graziosi wrote:
> Just for the record,
> 
>> Meanwhile... here the build fails (MSYS2/MinGW64):
>>
>> [...]
>> pdumper.c: In function 'dump_read_all':
>> pdumper.c:4723:45: error: conversion to 'unsigned int' from 'size_t {aka long long unsigned int}' may alter its value [-Werror=conversion]
>>           read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
>>                                               ^~~~~~~~~~~~~
>>    CC       data.o
>> cc1.exe: some warnings being treated as errors
> 
> I have seen you have commit many fixes but here the build still fails in the same manner.. Maybe you have not found the right solution yet..

On this one, I was waiting to see how the broader discussion went.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Fri, 16 Feb 2018 13:25:00 -0800
> 
> >> pdumper.c: In function 'dump_read_all':
> >> pdumper.c:4723:45: error: conversion to 'unsigned int' from 'size_t {aka long long unsigned int}' may alter its value [-Werror=conversion]
> >>           read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
> >>                                               ^~~~~~~~~~~~~
> >>    CC       data.o
> >> cc1.exe: some warnings being treated as errors
> > 
> > I have seen you have commit many fixes but here the build still fails in the same manner.. Maybe you have not found the right solution yet..
> 
> On this one, I was waiting to see how the broader discussion went.

OK, I've looked at the code, both in sysdep.c and in pdumper.c.  My
conclusion is that there's no real problem in sysdep.c, due to these
comments:

  /* Maximum number of bytes to read or write in a single system call.
     This works around a serious bug in Linux kernels before 2.6.16; see
     <https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=612839>.
     It's likely to work around similar bugs in other operating systems, so do it
     on all platforms.  Round INT_MAX down to a page size, with the conservative
     assumption that page sizes are at most 2**18 bytes (any kernel with a
     page size larger than that shouldn't have the bug).  */
  #ifndef MAX_RW_COUNT
  #define MAX_RW_COUNT (INT_MAX >> 18 << 18)
  #endif

  /* Read from FD to a buffer BUF with size NBYTE.
     If interrupted, process any quits and pending signals immediately
     if INTERRUPTIBLE, and then retry the read unless quitting.
     Return the number of bytes read, which might be less than NBYTE.
     On error, set errno to a value other than EINTR, and return -1.  */
  static ptrdiff_t
  emacs_intr_read (int fd, void *buf, ptrdiff_t nbyte, bool interruptible)
  {
    ssize_t result;

    /* There is no need to check against MAX_RW_COUNT, since no caller ever
       passes a size that large to emacs_read.  */
    do
      {
	if (interruptible)
	  maybe_quit ();
	result = read (fd, buf, nbyte);
      }
    while (result < 0 && errno == EINTR);

Since MAX_RW_COUNT is less than INT_MAX, we have no real problem here
for MS-Windows, as 'nbyte' will never overflow an unsigned int.  We
could add an eassert there, for Windows only, though, to make this
assumption explicit.

As for pdumper, we could do one of 2 things:

 . make a reasonable assumption that no .pdmp file will ever be larger
   than 2GB, change the assertion there which checks against SSIZE_MAX
   to check against UINT_MAX instead, and work internally with
   unsigned int instead of size_t counts; or

 . write a separate Windows-specific version of dump_read_all, which
   in the 64-bit build would limit to UINT_MAX the number of bytes we
   read on each iteration through the loop.

Daniel, which of these 2 alternatives do you prefer?  Or does anyone
have a better proposal?  (I already considered rewriting the code in
w32.c:sys_read so that it accepts a size_t last argument, and decided
it was unjustified for this single caller, as all the other direct
calls to 'read' in Emacs either use fixed small byte counts, or are
not compiled in the Windows build.)

Thanks.

Re: Preview: portable dumper

[Andy Moreton]

On Sat 17 Feb 2018, Eli Zaretskii wrote:
> Since MAX_RW_COUNT is less than INT_MAX, we have no real problem here
> for MS-Windows, as 'nbyte' will never overflow an unsigned int.  We
> could add an eassert there, for Windows only, though, to make this
> assumption explicit.

Comments mentioning MAX_RW_COUNT show that somebody checked an invariant
when writing the comment, but they do not guarantee that the invariant
will continue to be maintained in future versions of emacs. A check that
can be enforced by the compiler is preferable.

> As for pdumper, we could do one of 2 things:
>
>  . make a reasonable assumption that no .pdmp file will ever be larger
>    than 2GB, change the assertion there which checks against SSIZE_MAX
>    to check against UINT_MAX instead, and work internally with
>    unsigned int instead of size_t counts; or

This seems perfectly reasonable, and is the current implementation,
albeit checking for INT_MAX.

>  . write a separate Windows-specific version of dump_read_all, which
>    in the 64-bit build would limit to UINT_MAX the number of bytes we
>    read on each iteration through the loop.

This offers more opportunity for divergent implementations, so the other
option is preferable.

> Daniel, which of these 2 alternatives do you prefer?  Or does anyone
> have a better proposal?  (I already considered rewriting the code in
> w32.c:sys_read so that it accepts a size_t last argument, and decided
> it was unjustified for this single caller, as all the other direct
> calls to 'read' in Emacs either use fixed small byte counts, or are
> not compiled in the Windows build.)

Is there a gnulib replacement for read() that works for emacs, so that
we could use the proper prototype even on Windows ? If so we could get
rid of sys_read etc.

    AndyM

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Mon, 19 Feb 2018 22:23:17 +0000
> 
> Comments mentioning MAX_RW_COUNT show that somebody checked an invariant
> when writing the comment, but they do not guarantee that the invariant
> will continue to be maintained in future versions of emacs. A check that
> can be enforced by the compiler is preferable.

I added an assertion there (on master) that should make the code more
future-proof.

> > As for pdumper, we could do one of 2 things:
> >
> >  . make a reasonable assumption that no .pdmp file will ever be larger
> >    than 2GB, change the assertion there which checks against SSIZE_MAX
> >    to check against UINT_MAX instead, and work internally with
> >    unsigned int instead of size_t counts; or
> 
> This seems perfectly reasonable, and is the current implementation,
> albeit checking for INT_MAX.

No, the current code actually breaks the reads into chunks no larger
than INT_MAX.  So it doesn't assume a .pdmp file cannot be at most
2GB.

> >  . write a separate Windows-specific version of dump_read_all, which
> >    in the 64-bit build would limit to UINT_MAX the number of bytes we
> >    read on each iteration through the loop.
> 
> This offers more opportunity for divergent implementations, so the other
> option is preferable.

I presumed that we will not want to punish other platforms by making
them read piecemeal.  But if that's not an issue, I agree that the
second alternative is less elegant.

> Is there a gnulib replacement for read() that works for emacs, so that
> we could use the proper prototype even on Windows ?

No, because sys_read does much more than just fix the Windows _read.

Re: Preview: portable dumper

[Paul Eggert]

On 02/19/2018 02:23 PM, Andy Moreton wrote:
> Comments mentioning MAX_RW_COUNT show that somebody checked an invariant
> when writing the comment, but they do not guarantee that the invariant
> will continue to be maintained in future versions of emacs. A check that
> can be enforced by the compiler is preferable.

That "somebody" was me. That is, I looked at all the callers of these 
functions and checked that none of them go above MAX_RW_COUNT. That 
being said, it doesn't hurt to put in an eassert to check this as well 
(thanks, Eli). It would be a pain to check all the callers at 
compile-time, alas.

> Is there a gnulib replacement for read() that works for emacs, so that
> we could use the proper prototype even on Windows ?

There is. However, the Gnulib 'read' does some stuff that the Emacs 
'read' doesn't (and vice versa). Nobody has taken the time to try to 
unify them, and it's possible that unifying them would not be a good 
idea overall.

Re: Preview: portable dumper

[Eli Zaretskii]

> Date: Fri, 16 Feb 2018 22:14:18 +0100 (CET)
> From: Angelo Graziosi <angelo.g0@libero.it>
> 
> >          read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
> >                                              ^~~~~~~~~~~~~
> >   CC       data.o
> > cc1.exe: some warnings being treated as errors
> 
> I have seen you have commit many fixes but here the build still fails in the same manner.. Maybe you have not found the right solution yet.. 

We haven't.  Please be patient.

Re: Preview: portable dumper

[Daniel Colascione]

On 02/13/2018 02:28 PM, Angelo Graziosi wrote:
> 
>> Il 13 febbraio 2018 alle 23.06 Angelo Graziosi <angelo.g0@libero.it> ha scritto:
>>
>>
>> Eli Zaretskii wrote:
>>>
>>> I'd urge people to try this branch and report any issues they see.
>>
>> What should we look for?
>>
> 
> Meanwhile... here the build fails (MSYS2/MinGW64):
> 
> [...]
> pdumper.c: In function 'dump_read_all':
> pdumper.c:4723:45: error: conversion to 'unsigned int' from 'size_t {aka long long unsigned int}' may alter its value [-Werror=conversion]
>           read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
>                                               ^~~~~~~~~~~~~
>    CC       data.o
> cc1.exe: some warnings being treated as errors
> ...

Should be fixed.

Re: Preview: portable dumper

[Andy Moreton]

On Mon 19 Feb 2018, Daniel Colascione wrote:

> On 02/13/2018 02:28 PM, Angelo Graziosi wrote:
>>
>>> Il 13 febbraio 2018 alle 23.06 Angelo Graziosi <angelo.g0@libero.it> ha scritto:
>>>
>>>
>
>>> Eli Zaretskii wrote:
>>>>
>>>> I'd urge people to try this branch and report any issues they see.
>>>
>>> What should we look for?
>>>
>>
>> Meanwhile... here the build fails (MSYS2/MinGW64):
>>
>> [...]
>> pdumper.c: In function 'dump_read_all':
>> pdumper.c:4723:45: error: conversion to 'unsigned int' from 'size_t {aka long long unsigned int}' may alter its value [-Werror=conversion]
>>           read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
>>                                               ^~~~~~~~~~~~~
>>    CC       data.o
>> cc1.exe: some warnings being treated as errors
>> ...
>
> Should be fixed.

The patch for this in commit b7da1a5af8f1 is not quite right:

+      /* Some platforms accept only int-sized values to read.  */
+      unsigned chunk_to_read = UINT_MAX;
+      if (bytes_to_read - bytes_read < chunk_to_read)
+        chunk_to_read = (unsigned)(bytes_to_read - bytes_read);

Windows platforms use sys_read from w32.c:
    int sys_read(int fd, char *buffer, unsigned int count);

The limit for a chunk should be INT_MAX (not UINT_MAX) so that it
is representable in the (signed) return value.

    AndyM

Re: Preview: portable dumper

[Daniel Colascione]

On 02/19/2018 12:03 PM, Andy Moreton wrote:
> On Mon 19 Feb 2018, Daniel Colascione wrote:
> 
>> On 02/13/2018 02:28 PM, Angelo Graziosi wrote:
>>>
>>>> Il 13 febbraio 2018 alle 23.06 Angelo Graziosi <angelo.g0@libero.it> ha scritto:
>>>>
>>>>
>>
>>>> Eli Zaretskii wrote:
>>>>>
>>>>> I'd urge people to try this branch and report any issues they see.
>>>>
>>>> What should we look for?
>>>>
>>>
>>> Meanwhile... here the build fails (MSYS2/MinGW64):
>>>
>>> [...]
>>> pdumper.c: In function 'dump_read_all':
>>> pdumper.c:4723:45: error: conversion to 'unsigned int' from 'size_t {aka long long unsigned int}' may alter its value [-Werror=conversion]
>>>            read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
>>>                                                ^~~~~~~~~~~~~
>>>     CC       data.o
>>> cc1.exe: some warnings being treated as errors
>>> ...
>>
>> Should be fixed.
> 
> The patch for this in commit b7da1a5af8f1 is not quite right:
> 
> +      /* Some platforms accept only int-sized values to read.  */
> +      unsigned chunk_to_read = UINT_MAX;
> +      if (bytes_to_read - bytes_read < chunk_to_read)
> +        chunk_to_read = (unsigned)(bytes_to_read - bytes_read);
> 
> Windows platforms use sys_read from w32.c:
>      int sys_read(int fd, char *buffer, unsigned int count);
> 
> The limit for a chunk should be INT_MAX (not UINT_MAX) so that it
> is representable in the (signed) return value.

Is the return value int too? Or is it a ssize_t that can hold UINT_MAX? 
If the former, agreed.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Mon, 19 Feb 2018 20:03:09 +0000
> 
>     int sys_read(int fd, char *buffer, unsigned int count);
> 
> The limit for a chunk should be INT_MAX (not UINT_MAX) so that it
> is representable in the (signed) return value.

We should be able to change sys_read to return ssize_t instead.  Can
you try that?

Re: Preview: portable dumper

[Daniel Colascione]

On 02/19/2018 12:18 PM, Eli Zaretskii wrote:
>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Mon, 19 Feb 2018 20:03:09 +0000
>>
>>      int sys_read(int fd, char *buffer, unsigned int count);
>>
>> The limit for a chunk should be INT_MAX (not UINT_MAX) so that it
>> is representable in the (signed) return value.
> 
> We should be able to change sys_read to return ssize_t instead.  Can
> you try that?

I looked at the w32 code --- we give the size to _read directly in the 
non-pipe, non-socket, non-serial case, and MSDN doesn't say what happens 
when size > INT_MAX, so in the interest of being rather safe than sorry, 
I just changed the pdumper code to use INT_MAX. It shouldn't matter in 
practice.

Re: Preview: portable dumper

[Angelo Graziosi]

> Il 19 febbraio 2018 alle 18.04 Daniel Colascione ha scritto:
> 
> 
> On 02/13/2018 02:28 PM, Angelo Graziosi wrote:
> > 
> > [...]
> > pdumper.c: In function 'dump_read_all':
> > pdumper.c:4723:45: error: conversion to 'unsigned int' from 'size_t {aka long long unsigned int}' may alter its value [-Werror=conversion]
> >           read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
> >                                               ^~~~~~~~~~~~~
> >    CC       data.o
> > cc1.exe: some warnings being treated as errors
> > ...
> 
> Should be fixed.

Now it builds! 

Thanks,
  Angelo.

Re: Preview: portable dumper

[Angelo Graziosi]

> Il 19 febbraio 2018 alle 18.04 Daniel Colascione ha scritto:
> 
> Should be fixed.

...and now commit aacc1390f21ddb003c9d6a786597e89284bafec3 is broken..

[...]
  CC       keyboard.o
emacs.c: In function 'main':
emacs.c:1849:3: error: 'else' without a previous 'if'
   else
   ^~~~
make[1]: *** [Makefile:387: emacs.o] Error 1
make[1]: *** Attesa per i processi non terminati....
make[1]: uscita dalla directory "/tmp/emacs-pdumper/src"
make: *** [Makefile:420: src] Error 2

Re: Preview: portable dumper

[Daniel Colascione]

On 02/26/2018 04:03 AM, Angelo Graziosi wrote:
> 
>> Il 19 febbraio 2018 alle 18.04 Daniel Colascione ha scritto:
>>
>> Should be fixed.
> 
> ...and now commit aacc1390f21ddb003c9d6a786597e89284bafec3 is broken..
> 
> [...]
>    CC       keyboard.o
> emacs.c: In function 'main':
> emacs.c:1849:3: error: 'else' without a previous 'if'
>     else
>     ^~~~
> make[1]: *** [Makefile:387: emacs.o] Error 1
> make[1]: *** Attesa per i processi non terminati....
> make[1]: uscita dalla directory "/tmp/emacs-pdumper/src"
> make: *** [Makefile:420: src] Error 2

Now that's fixed too. I wish we had better CI.

Re: Preview: portable dumper

[Angelo Graziosi]

They are 4 weeks that pdumper branch is not synchronized, should we continue to test it? 

During this time, builds of that branch have been tested on Windows, GNU/Linux and macOS without showing any specific issue...

Angelo

Re: Preview: portable dumper

[Daniel Colascione]

On 03/29/2018 12:12 AM, Angelo Graziosi wrote:
> 
> They are 4 weeks that pdumper branch is not synchronized, should we continue to test it?
> 
> During this time, builds of that branch have been tested on Windows, GNU/Linux and macOS without showing any specific issue...

I'm not sure what's left to test. It works. What's stopping a merge?

Re: Preview: portable dumper

[Robert Pluim]

Daniel Colascione <dancol@dancol.org> writes:

> On 03/29/2018 12:12 AM, Angelo Graziosi wrote:
>>
>> They are 4 weeks that pdumper branch is not synchronized, should we continue to test it?
>>
>> During this time, builds of that branch have been tested on Windows, GNU/Linux and macOS without showing any specific issue...
>
> I'm not sure what's left to test. It works. What's stopping a merge?

I just test-merged it to master, with only two issues:

1. a merge conflict in src/nsfont.m as a result of 7ff62ed221c
2. The Lisp_Buffer_Local_Value hash has changed, although itʼs not
clear to me why, since I only see comment changes in its definition. I
updated dmpstruct.h, and the build works fine (it runs gnus, as proven
by this message).

Robert

Re: Preview: portable dumper

[Pip Cet]

I've just skimmed the diff, and I think the change is a significant
improvement, and will help with my GC experiments even though I'm
probably never going to be able to use the portable dumper. Some minor
comments:

it is not clear to me why some staticpro's are moved to happen after
the initialisation of the variable and some aren't. Maybe we should
change staticpro to have a
void staticpro(Lisp_Object *ptr, Lisp_Object initial_value);
signature and save some lines of code?

I don't understand this hunk in sysdep.c:

@@ -2138,7 +2149,7 @@ init_signals (bool dumping)
 #ifdef SIGUSR2
   add_user_signal (SIGUSR2, "sigusr2");
 #endif
-  sigaction (SIGABRT, &thread_fatal_action, 0);
+  //sigaction (SIGABRT, &thread_fatal_action, 0);
 #ifdef SIGPRE
   sigaction (SIGPRE, &thread_fatal_action, 0);
 #endif

Similarly, this hunk in pcase.el:

@@ -63,6 +63,7 @@
 ;; FIXME: Now that macroexpansion is also performed when loading an interpreted
 ;; file, this is not a real problem any more.
 (defconst pcase--memoize (make-hash-table :weakness 'key :test 'eq))
+;; (defconst pcase--memoize (make-hash-table :test 'eq))
 ;; (defconst pcase--memoize-1 (make-hash-table :test 'eq))
 ;; (defconst pcase--memoize-2 (make-hash-table :weakness 'key :test 'equal))

Are those two intentional?

I would vaguely prefer if "pdumper" names were limited to the actual
pdumper implementation, and "dumper" names were used instead in code
that does not depend on the implementation of the dumper. So we'd have
dumper_object_p(), defined to false if there's no dumper, and
pdumper_object_p otherwise. This would make things slightly easier for
(a) competing implementations (b) abusing the DUMPER_* machinery for
GC experiments (c) possibly extending pdumper to combine several
images rather than use a single one. It would also help avoid _impl
names.

I think it's worth mentioning that staticpro is now required for
variables that used to be safe: this change comes at the price of a
tiny performance hit because of the additional GC roots, but it's
worth it, IMHO.

Apart from that, there appear to be no disadvantages for people who
prefer to continue using unexec. Is that correct?

On Thu, Mar 29, 2018 at 9:39 AM, Robert Pluim <rpluim@gmail.com> wrote:
> Daniel Colascione <dancol@dancol.org> writes:
>
>> On 03/29/2018 12:12 AM, Angelo Graziosi wrote:
>>>
>>> They are 4 weeks that pdumper branch is not synchronized, should we continue to test it?
>>>
>>> During this time, builds of that branch have been tested on Windows, GNU/Linux and macOS without showing any specific issue...
>>
>> I'm not sure what's left to test. It works. What's stopping a merge?
>
> I just test-merged it to master, with only two issues:
>
> 1. a merge conflict in src/nsfont.m as a result of 7ff62ed221c
> 2. The Lisp_Buffer_Local_Value hash has changed, although itʼs not
> clear to me why, since I only see comment changes in its definition. I
> updated dmpstruct.h, and the build works fine (it runs gnus, as proven
> by this message).
>
> Robert
>

Re: Preview: portable dumper

[Daniel Colascione]

On 03/29/2018 06:35 AM, Pip Cet wrote:
> I've just skimmed the diff, and I think the change is a significant
> improvement, and will help with my GC experiments even though I'm
> probably never going to be able to use the portable dumper.

Why not? I eventually want to remove unexec entirely.

> Some minor
> comments:
> 
> it is not clear to me why some staticpro's are moved to happen after
> the initialisation of the variable and some aren't.

The code has always been a bit unclear about that; we don't GC that 
early, so it doesn't really matter- -- but if we did, we'd be saved by 
nil being all zero these days.

> Maybe we should
> change staticpro to have a
> void staticpro(Lisp_Object *ptr, Lisp_Object initial_value);
> signature and save some lines of code?

Sometimes the value from .data is perfectly good though, and at that 
point, the initial_value setting would be a waste.

> I don't understand this hunk in sysdep.c:
> 
> @@ -2138,7 +2149,7 @@ init_signals (bool dumping)
>   #ifdef SIGUSR2
>     add_user_signal (SIGUSR2, "sigusr2");
>   #endif
> -  sigaction (SIGABRT, &thread_fatal_action, 0);
> +  //sigaction (SIGABRT, &thread_fatal_action, 0);
>   #ifdef SIGPRE
>     sigaction (SIGPRE, &thread_fatal_action, 0);
>   #endif

Thanks; I'll clean that up. That's left over from the demand paging 
experiment.

> 
> Similarly, this hunk in pcase.el:
> 
> @@ -63,6 +63,7 @@
>   ;; FIXME: Now that macroexpansion is also performed when loading an interpreted
>   ;; file, this is not a real problem any more.
>   (defconst pcase--memoize (make-hash-table :weakness 'key :test 'eq))
> +;; (defconst pcase--memoize (make-hash-table :test 'eq))
>   ;; (defconst pcase--memoize-1 (make-hash-table :test 'eq))
>   ;; (defconst pcase--memoize-2 (make-hash-table :weakness 'key :test 'equal))
> 

Thanks for catching that.

> Are those two intentional?
> 
> I would vaguely prefer if "pdumper" names were limited to the actual
> pdumper implementation, and "dumper" names were used instead in code
> that does not depend on the implementation of the dumper. So we'd have
> dumper_object_p(), defined to false if there's no dumper, and
> pdumper_object_p otherwise.

I'd rather not do a big renaming at this point. Nothing stops someone 
doing cleanup work post-merge if sufficiently motivated.

> Apart from that, there appear to be no disadvantages for people who
> prefer to continue using unexec. Is that correct?

That's correct.

Re: Preview: portable dumper

[Pip Cet]

On Thu, Mar 29, 2018 at 3:31 PM, Daniel Colascione <dancol@dancol.org> wrote:
> On 03/29/2018 06:35 AM, Pip Cet wrote:
>> I've just skimmed the diff, and I think the change is a significant
>> improvement, and will help with my GC experiments even though I'm
>> probably never going to be able to use the portable dumper.
>
>
> Why not? I eventually want to remove unexec entirely.

No objections to removing unexec, as long as temacs remains an option
:-) I haven't thought about dumping/restoring SpiderMonkey state, but
as it's hairy enough to get things working with setjmp(), I'd rather
not worry about it now.

I do have a half-finished (working, but not thoroughly tested) unexec
for asm.js, too, which sped things up somewhat, but I don't see why
the portable dumper wouldn't work there, too.

>
>> Some minor
>> comments:
>>
>> it is not clear to me why some staticpro's are moved to happen after
>> the initialisation of the variable and some aren't.
>
>
> The code has always been a bit unclear about that; we don't GC that early,
> so it doesn't really matter- -- but if we did, we'd be saved by nil being
> all zero these days.

Well, Qnil isn't all zero (or even constant) here, and I do GC
unpredictably, though hopefully never that early; I'm still safe if
the staticpro goes first: the all-zero value, representing 0.0, is
fine for GC.

>> Maybe we should
>> change staticpro to have a
>> void staticpro(Lisp_Object *ptr, Lisp_Object initial_value);
>> signature and save some lines of code?
>
>
> Sometimes the value from .data is perfectly good though, and at that point,
> the initial_value setting would be a waste.

... as are explicit Qnil initializations, in general. So

inline void staticpro(Lisp_Object *ptr, Lisp_Object initial_value)
{
  assume(*ptr == Qnil);
  real_staticpro(ptr);
  if (initial_value != Qnil)
    *ptr = initial_value;
}

would actually reduce code size. But, again, it'd be better to merge
first, then optimize such minor things.

>> I would vaguely prefer if "pdumper" names were limited to the actual
>> pdumper implementation, and "dumper" names were used instead in code
>> that does not depend on the implementation of the dumper. So we'd have
>> dumper_object_p(), defined to false if there's no dumper, and
>> pdumper_object_p otherwise.
>
>
> I'd rather not do a big renaming at this point. Nothing stops someone doing
> cleanup work post-merge if sufficiently motivated.

Agreed.

>> Apart from that, there appear to be no disadvantages for people who
>> prefer to continue using unexec. Is that correct?
> That's correct.

Thanks!

Re: Preview: portable dumper

[Pip Cet]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

On Thu, Mar 29, 2018 at 4:15 PM, Pip Cet <pipcet@gmail.com> wrote:
> inline void staticpro(Lisp_Object *ptr, Lisp_Object initial_value)
> {
>   assume(*ptr == Qnil);
>   real_staticpro(ptr);
>   if (initial_value != Qnil)
>     *ptr = initial_value;
> }

Okay, GCC doesn't appear to optimize that expression by default, but
I've attached a patch which does keep stripped binary size constant.

It reduces C code size a little, avoids the ambiguity of whether
staticpro or the initialization goes first, removes the Qnil == 0
assumption from a few places and generally seems like a slight
improvement to me. Totally independent from the pdumper patch, of
course. (And I'd prefer the pdumper patch to go in first.)

[-- Attachment #2: emacs-staticpro-001.diff --]
[-- Type: text/x-patch, Size: 46971 bytes --]

diff --git a/src/alloc.c b/src/alloc.c
index 369592d70ee..ff32d1639ee 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -5696,7 +5696,7 @@ purecopy (Lisp_Object obj)
    VARADDRESS.  */
 
 void
-staticpro (Lisp_Object *varaddress)
+staticpro_1 (Lisp_Object *varaddress)
 {
   if (staticidx >= NSTATICS)
     fatal ("NSTATICS too small; try increasing and recompiling Emacs.");
diff --git a/src/bidi.c b/src/bidi.c
index 9bc8dbe8603..24e435bbc1d 100644
--- a/src/bidi.c
+++ b/src/bidi.c
@@ -1090,25 +1090,20 @@ bidi_unshelve_cache (void *databuf, bool just_free)
 static void
 bidi_initialize (void)
 {
-  bidi_type_table = uniprop_table (intern ("bidi-class"));
+  staticpro (&bidi_type_table, uniprop_table (intern ("bidi-class")));
   if (NILP (bidi_type_table))
     emacs_abort ();
-  staticpro (&bidi_type_table);
 
-  bidi_mirror_table = uniprop_table (intern ("mirroring"));
+  staticpro (&bidi_mirror_table, uniprop_table (intern ("mirroring")));
   if (NILP (bidi_mirror_table))
     emacs_abort ();
-  staticpro (&bidi_mirror_table);
 
-  bidi_brackets_table = uniprop_table (intern ("bracket-type"));
+  staticpro (&bidi_brackets_table, uniprop_table (intern ("bracket-type")));
   if (NILP (bidi_brackets_table))
     emacs_abort ();
-  staticpro (&bidi_brackets_table);
 
-  paragraph_start_re = build_string ("^\\(\f\\|[ \t]*\\)$");
-  staticpro (&paragraph_start_re);
-  paragraph_separate_re = build_string ("^[ \t\f]*$");
-  staticpro (&paragraph_separate_re);
+  staticpro (&paragraph_start_re, build_string ("^\\(\f\\|[ \t]*\\)$"));
+  staticpro (&paragraph_separate_re, build_string ("^[ \t\f]*$"));
 
   bidi_cache_sp = 0;
   bidi_cache_total_alloc = 0;
diff --git a/src/buffer.c b/src/buffer.c
index 14837372d34..e90bf6ece20 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -5441,12 +5441,11 @@ defvar_per_buffer (struct Lisp_Buffer_Objfwd *bo_fwd, const char *namestring,
 void
 syms_of_buffer (void)
 {
-  staticpro (&last_overlay_modification_hooks);
-  last_overlay_modification_hooks
-    = Fmake_vector (make_number (10), Qnil);
+  staticpro (&last_overlay_modification_hooks,
+             Fmake_vector (make_number (10), Qnil));
 
-  staticpro (&QSFundamental);
-  staticpro (&Vbuffer_alist);
+  staticpro (&QSFundamental, Qnil);
+  staticpro (&Vbuffer_alist, Qnil);
 
   DEFSYM (Qchoice, "choice");
   DEFSYM (Qleft, "left");
diff --git a/src/callint.c b/src/callint.c
index 08a8bba4646..8cac305e0b4 100644
--- a/src/callint.c
+++ b/src/callint.c
@@ -826,11 +826,9 @@ Its numeric meaning is what you would get from `(interactive "p")'.  */)
 void
 syms_of_callint (void)
 {
-  point_marker = Fmake_marker ();
-  staticpro (&point_marker);
+  staticpro (&point_marker, Fmake_marker ());
 
-  callint_message = Qnil;
-  staticpro (&callint_message);
+  staticpro (&callint_message, Qnil);
 
   preserved_fns = listn (CONSTYPE_PURE, 4,
 			 intern_c_string ("region-beginning"),
diff --git a/src/callproc.c b/src/callproc.c
index 973f324139c..5b3867288e1 100644
--- a/src/callproc.c
+++ b/src/callproc.c
@@ -1640,15 +1640,13 @@ void
 syms_of_callproc (void)
 {
 #ifndef DOS_NT
-  Vtemp_file_name_pattern = build_string ("emacsXXXXXX");
+  staticpro (&Vtemp_file_name_pattern, build_string ("emacsXXXXXX"));
 #else  /* DOS_NT */
-  Vtemp_file_name_pattern = build_string ("emXXXXXX");
+  staticpro (&Vtemp_file_name_pattern, build_string ("emXXXXXX"));
 #endif
-  staticpro (&Vtemp_file_name_pattern);
 
 #ifdef MSDOS
-  synch_process_tempfile = make_number (0);
-  staticpro (&synch_process_tempfile);
+  staticpro (&synch_process_tempfile, make_number (0));
 #endif
 
   DEFVAR_LISP ("shell-file-name", Vshell_file_name,
diff --git a/src/casetab.c b/src/casetab.c
index 8f806a0647c..d2388d5fc58 100644
--- a/src/casetab.c
+++ b/src/casetab.c
@@ -247,6 +247,11 @@ init_casetab_once (void)
   DEFSYM (Qcase_table, "case-table");
   Fput (Qcase_table, Qchar_table_extra_slots, make_number (3));
 
+  staticpro (&Vascii_canon_table, Qnil);
+  staticpro (&Vascii_downcase_table, Qnil);
+  staticpro (&Vascii_eqv_table, Qnil);
+  staticpro (&Vascii_upcase_table, Qnil);
+
   down = Fmake_char_table (Qcase_table, Qnil);
   Vascii_downcase_table = down;
   set_char_table_purpose (down, Qcase_table);
@@ -289,11 +294,6 @@ syms_of_casetab (void)
 {
   DEFSYM (Qcase_table_p, "case-table-p");
 
-  staticpro (&Vascii_canon_table);
-  staticpro (&Vascii_downcase_table);
-  staticpro (&Vascii_eqv_table);
-  staticpro (&Vascii_upcase_table);
-
   defsubr (&Scase_table_p);
   defsubr (&Scurrent_case_table);
   defsubr (&Sstandard_case_table);
diff --git a/src/ccl.c b/src/ccl.c
index ed8588d7f8a..b2a45c87419 100644
--- a/src/ccl.c
+++ b/src/ccl.c
@@ -2274,8 +2274,7 @@ Return index number of the registered map.  */)
 void
 syms_of_ccl (void)
 {
-  staticpro (&Vccl_program_table);
-  Vccl_program_table = Fmake_vector (make_number (32), Qnil);
+  staticpro (&Vccl_program_table, Fmake_vector (make_number (32), Qnil));
 
   DEFSYM (Qccl, "ccl");
   DEFSYM (Qcclp, "cclp");
diff --git a/src/character.c b/src/character.c
index 6a689808043..cbde3f414e9 100644
--- a/src/character.c
+++ b/src/character.c
@@ -1099,8 +1099,7 @@ syms_of_character (void)
   DEFSYM (Qcharacterp, "characterp");
   DEFSYM (Qauto_fill_chars, "auto-fill-chars");
 
-  staticpro (&Vchar_unify_table);
-  Vchar_unify_table = Qnil;
+  staticpro (&Vchar_unify_table, Qnil);
 
   defsubr (&Smax_char);
   defsubr (&Scharacterp);
diff --git a/src/charset.c b/src/charset.c
index 05290e86b4e..60b46137d19 100644
--- a/src/charset.c
+++ b/src/charset.c
@@ -2348,17 +2348,13 @@ syms_of_charset (void)
   DEFSYM (Qeight_bit, "eight-bit");
   DEFSYM (Qiso_8859_1, "iso-8859-1");
 
-  staticpro (&Vcharset_ordered_list);
-  Vcharset_ordered_list = Qnil;
+  staticpro (&Vcharset_ordered_list, Qnil);
 
-  staticpro (&Viso_2022_charset_list);
-  Viso_2022_charset_list = Qnil;
+  staticpro (&Viso_2022_charset_list, Qnil);
 
-  staticpro (&Vemacs_mule_charset_list);
-  Vemacs_mule_charset_list = Qnil;
+  staticpro (&Vemacs_mule_charset_list, Qnil);
 
-  staticpro (&Vcharset_hash_table);
-  Vcharset_hash_table = CALLN (Fmake_hash_table, QCtest, Qeq);
+  staticpro (&Vcharset_hash_table, CALLN (Fmake_hash_table, QCtest, Qeq));
 
   charset_table = charset_table_init;
   charset_table_size = ARRAYELTS (charset_table_init);
diff --git a/src/coding.c b/src/coding.c
index a16142a9b41..44c68e8b880 100644
--- a/src/coding.c
+++ b/src/coding.c
@@ -10818,20 +10818,15 @@ init_coding_once (void)
 void
 syms_of_coding (void)
 {
-  staticpro (&Vcoding_system_hash_table);
-  Vcoding_system_hash_table = CALLN (Fmake_hash_table, QCtest, Qeq);
+  staticpro (&Vcoding_system_hash_table, CALLN (Fmake_hash_table, QCtest, Qeq));
 
-  staticpro (&Vsjis_coding_system);
-  Vsjis_coding_system = Qnil;
+  staticpro (&Vsjis_coding_system, Qnil);
 
-  staticpro (&Vbig5_coding_system);
-  Vbig5_coding_system = Qnil;
+  staticpro (&Vbig5_coding_system, Qnil);
 
-  staticpro (&Vcode_conversion_reused_workbuf);
-  Vcode_conversion_reused_workbuf = Qnil;
+  staticpro (&Vcode_conversion_reused_workbuf, Qnil);
 
-  staticpro (&Vcode_conversion_workbuf_name);
-  Vcode_conversion_workbuf_name = build_pure_c_string (" *code-conversion-work*");
+  staticpro (&Vcode_conversion_workbuf_name, build_pure_c_string (" *code-conversion-work*"));
 
   reused_workbuf_in_use = 0;
 
@@ -10914,9 +10909,7 @@ syms_of_coding (void)
   DEFSYM (QCpre_write_conversion, ":pre-write-conversion");
   DEFSYM (QCascii_compatible_p, ":ascii-compatible-p");
 
-  Vcoding_category_table
-    = Fmake_vector (make_number (coding_category_max), Qnil);
-  staticpro (&Vcoding_category_table);
+  staticpro (&Vcoding_category_table, Fmake_vector (make_number (coding_category_max), Qnil));
   /* Followings are target of code detection.  */
   ASET (Vcoding_category_table, coding_category_iso_7,
 	intern_c_string ("coding-category-iso-7"));
@@ -11352,10 +11345,9 @@ internal character representation.  */);
     Fset (AREF (Vcoding_category_table, i), Qno_conversion);
 
 #if defined (DOS_NT)
-  system_eol_type = Qdos;
+  staticpro (&system_eol_type, Qdos);
 #else
-  system_eol_type = Qunix;
+  staticpro (&system_eol_type, Qunix);
 #endif
-  staticpro (&system_eol_type);
 }
 #endif /* emacs */
diff --git a/src/composite.c b/src/composite.c
index 746c2959f84..f7e95a0255b 100644
--- a/src/composite.c
+++ b/src/composite.c
@@ -1907,19 +1907,15 @@ syms_of_composite (void)
      and thus it's not worth to save memory in such a way.  So, we
      make the table not weak.  */
   Lisp_Object args[] = {QCtest, Qequal, QCsize, make_number (311)};
-  composition_hash_table = CALLMANY (Fmake_hash_table, args);
-  staticpro (&composition_hash_table);
+  staticpro (&composition_hash_table, CALLMANY (Fmake_hash_table, args));
 
   /* Make a hash table for glyph-string.  */
-  gstring_hash_table = CALLMANY (Fmake_hash_table, args);
-  staticpro (&gstring_hash_table);
+  staticpro (&gstring_hash_table, CALLMANY (Fmake_hash_table, args));
 
-  staticpro (&gstring_work_headers);
-  gstring_work_headers = make_uninit_vector (8);
+  staticpro (&gstring_work_headers, make_uninit_vector (8));
   for (i = 0; i < 8; i++)
     ASET (gstring_work_headers, i, Fmake_vector (make_number (i + 2), Qnil));
-  staticpro (&gstring_work);
-  gstring_work = Fmake_vector (make_number (10), Qnil);
+  staticpro (&gstring_work, Fmake_vector (make_number (10), Qnil));
 
   /* Text property `composition' should be nonsticky by default.  */
   Vtext_property_default_nonsticky
diff --git a/src/dbusbind.c b/src/dbusbind.c
index ec3707d18f3..3af20551993 100644
--- a/src/dbusbind.c
+++ b/src/dbusbind.c
@@ -1824,8 +1824,7 @@ be called when the D-Bus reply message arrives.  */);
 #endif
 
   /* Initialize internal objects.  */
-  xd_registered_buses = Qnil;
-  staticpro (&xd_registered_buses);
+  staticpro (&xd_registered_buses, Qnil);
 
   Fprovide (intern_c_string ("dbusbind"), Qnil);
 
diff --git a/src/dispnew.c b/src/dispnew.c
index 56f125218dc..608e1123459 100644
--- a/src/dispnew.c
+++ b/src/dispnew.c
@@ -6227,8 +6227,7 @@ syms_of_display (void)
   defsubr (&Sdump_redisplay_history);
 #endif
 
-  frame_and_buffer_state = Fmake_vector (make_number (20), Qlambda);
-  staticpro (&frame_and_buffer_state);
+  staticpro (&frame_and_buffer_state, Fmake_vector (make_number (20), Qlambda));
 
   /* This is the "purpose" slot of a display table.  */
   DEFSYM (Qdisplay_table, "display-table");
diff --git a/src/emacs.c b/src/emacs.c
index 8ea61b71fb7..3229c4d4648 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -2680,7 +2680,6 @@ libraries; only those already known by Emacs will be loaded.  */);
   Fput (intern_c_string ("dynamic-library-alist"), Qrisky_local_variable, Qt);
 
 #ifdef WINDOWSNT
-  Vlibrary_cache = Qnil;
-  staticpro (&Vlibrary_cache);
+  staticpro (&Vlibrary_cache, Qnil);
 #endif
 }
diff --git a/src/eval.c b/src/eval.c
index a6e1d86c4ab..ba304ee8456 100644
--- a/src/eval.c
+++ b/src/eval.c
@@ -4107,13 +4107,10 @@ alist of active lexical bindings.  */);
      (Just imagine if someone makes it buffer-local).  */
   Funintern (Qinternal_interpreter_environment, Qnil);
 
-  Vrun_hooks = intern_c_string ("run-hooks");
-  staticpro (&Vrun_hooks);
+  staticpro (&Vrun_hooks, intern_c_string ("run-hooks"));
 
-  staticpro (&Vautoload_queue);
-  Vautoload_queue = Qnil;
-  staticpro (&Vsignaling_function);
-  Vsignaling_function = Qnil;
+  staticpro (&Vautoload_queue, Qnil);
+  staticpro (&Vsignaling_function, Qnil);
 
   inhibit_lisp_code = Qnil;
 
diff --git a/src/fileio.c b/src/fileio.c
index 52ca8b6297e..17230a82fb0 100644
--- a/src/fileio.c
+++ b/src/fileio.c
@@ -6091,8 +6091,7 @@ annotation functions in `write-region-annotate-functions' changed the
 current buffer, the function stored in this variable is called for
 each of those additional buffers as well, in addition to the original
 buffer.  The relevant buffer is current during each function call.  */);
-  Vwrite_region_post_annotation_function = Qnil;
-  staticpro (&Vwrite_region_annotation_buffers);
+  staticpro (&Vwrite_region_annotation_buffers, Qnil);
 
   DEFVAR_LISP ("write-region-annotations-so-far",
 	       Vwrite_region_annotations_so_far,
diff --git a/src/fns.c b/src/fns.c
index 94b9d984f0d..16c842b7a12 100644
--- a/src/fns.c
+++ b/src/fns.c
@@ -5178,11 +5178,9 @@ compilation.  */);
   Voverriding_plist_environment = Qnil;
   DEFSYM (Qoverriding_plist_environment, "overriding-plist-environment");
 
-  staticpro (&string_char_byte_cache_string);
-  string_char_byte_cache_string = Qnil;
+  staticpro (&string_char_byte_cache_string, Qnil);
 
-  require_nesting_list = Qnil;
-  staticpro (&require_nesting_list);
+  staticpro (&require_nesting_list, Qnil);
 
   Fset (Qyes_or_no_p_history, Qnil);
 
diff --git a/src/font.c b/src/font.c
index a6d3f5d4798..34ff23b40d0 100644
--- a/src/font.c
+++ b/src/font.c
@@ -5307,8 +5307,7 @@ syms_of_font (void)
   sort_shift_bits[FONT_WIDTH_INDEX] = 23;
   /* Note that the other elements in sort_shift_bits are not used.  */
 
-  staticpro (&font_charset_alist);
-  font_charset_alist = Qnil;
+  staticpro (&font_charset_alist, Qnil);
 
   DEFSYM (Qopentype, "opentype");
 
@@ -5346,13 +5345,10 @@ syms_of_font (void)
 
   DEFSYM (QCuser_spec, ":user-spec");
 
-  staticpro (&scratch_font_spec);
-  scratch_font_spec = Ffont_spec (0, NULL);
-  staticpro (&scratch_font_prefer);
-  scratch_font_prefer = Ffont_spec (0, NULL);
+  staticpro (&scratch_font_spec, Ffont_spec (0, NULL));
+  staticpro (&scratch_font_prefer, Ffont_spec (0, NULL));
 
-  staticpro (&Vfont_log_deferred);
-  Vfont_log_deferred = Fmake_vector (make_number (3), Qnil);
+  staticpro (&Vfont_log_deferred, Fmake_vector (make_number (3), Qnil));
 
 #if 0
 #ifdef HAVE_LIBOTF
@@ -5447,8 +5443,7 @@ This variable cannot be set; trying to do so will signal an error.  */);
   Vfont_width_table = BUILD_STYLE_TABLE (width_table);
   make_symbol_constant (intern_c_string ("font-width-table"));
 
-  staticpro (&font_style_table);
-  font_style_table = make_uninit_vector (3);
+  staticpro (&font_style_table, make_uninit_vector (3));
   ASET (font_style_table, 0, Vfont_weight_table);
   ASET (font_style_table, 1, Vfont_slant_table);
   ASET (font_style_table, 2, Vfont_width_table);
diff --git a/src/fontset.c b/src/fontset.c
index 6ca64068717..71dd45c3237 100644
--- a/src/fontset.c
+++ b/src/fontset.c
@@ -2116,14 +2116,11 @@ syms_of_fontset (void)
   DEFSYM (Qappend, "append");
   DEFSYM (Qlatin, "latin");
 
-  Vcached_fontset_data = Qnil;
-  staticpro (&Vcached_fontset_data);
+  staticpro (&Vcached_fontset_data, Qnil);
 
-  Vfontset_table = Fmake_vector (make_number (32), Qnil);
-  staticpro (&Vfontset_table);
+  staticpro (&Vfontset_table, Fmake_vector (make_number (32), Qnil));
 
-  Vdefault_fontset = Fmake_char_table (Qfontset, Qnil);
-  staticpro (&Vdefault_fontset);
+  staticpro (&Vdefault_fontset, Fmake_char_table (Qfontset, Qnil));
   set_fontset_id (Vdefault_fontset, make_number (0));
   set_fontset_name
     (Vdefault_fontset,
@@ -2131,8 +2128,7 @@ syms_of_fontset (void)
   ASET (Vfontset_table, 0, Vdefault_fontset);
   next_fontset_id = 1;
 
-  auto_fontset_alist = Qnil;
-  staticpro (&auto_fontset_alist);
+  staticpro (&auto_fontset_alist, Qnil);
 
   DEFVAR_LISP ("font-encoding-charset-alist", Vfont_encoding_charset_alist,
 	       doc: /*
diff --git a/src/frame.c b/src/frame.c
index 86caa32615d..40abef3a66b 100644
--- a/src/frame.c
+++ b/src/frame.c
@@ -6092,7 +6092,7 @@ making the child frame unresponsive to user actions, the default is to
 iconify the top level frame instead.  */);
   iconify_child_frame = Qiconify_top_level;
 
-  staticpro (&Vframe_list);
+  staticpro (&Vframe_list, Qnil);
 
   defsubr (&Sframep);
   defsubr (&Sframe_live_p);
diff --git a/src/ftfont.c b/src/ftfont.c
index c2e093e633d..eaef4574b4e 100644
--- a/src/ftfont.c
+++ b/src/ftfont.c
@@ -2735,16 +2735,13 @@ syms_of_ftfont (void)
   DEFSYM (Qsans, "sans");
   DEFSYM (Qsans__serif, "sans serif");
 
-  staticpro (&freetype_font_cache);
-  freetype_font_cache = list1 (Qt);
+  staticpro (&freetype_font_cache, list1 (Qt));
 
-  staticpro (&ftfont_generic_family_list);
-  ftfont_generic_family_list = list3 (Fcons (Qmonospace, Qt),
-				      Fcons (Qsans_serif, Qt),
-				      Fcons (Qsans, Qt));
+  staticpro (&ftfont_generic_family_list, list3 (Fcons (Qmonospace, Qt),
+                                                 Fcons (Qsans_serif, Qt),
+                                                 Fcons (Qsans, Qt)));
 
-  staticpro (&ft_face_cache);
-  ft_face_cache = Qnil;
+  staticpro (&ft_face_cache, Qnil);
 
   register_font_driver (&ftfont_driver, NULL);
 }
diff --git a/src/gfilenotify.c b/src/gfilenotify.c
index 650df0fcbb5..63bfdb49460 100644
--- a/src/gfilenotify.c
+++ b/src/gfilenotify.c
@@ -331,7 +331,7 @@ syms_of_gfilenotify (void)
   DEFSYM (Qunmounted, "unmounted");	/* G_FILE_MONITOR_EVENT_UNMOUNTED  */
   DEFSYM (Qmoved, "moved");	/* G_FILE_MONITOR_EVENT_MOVED  */
 
-  staticpro (&watch_list);
+  staticpro (&watch_list, Qnil);
 
   Fprovide (intern_c_string ("gfilenotify"), Qnil);
 
diff --git a/src/inotify.c b/src/inotify.c
index e06cc97c6a7..b10d218b4db 100644
--- a/src/inotify.c
+++ b/src/inotify.c
@@ -546,7 +546,7 @@ syms_of_inotify (void)
   defsubr (&Sinotify_watch_list);
   defsubr (&Sinotify_allocated_p);
 #endif
-  staticpro (&watch_list);
+  staticpro (&watch_list, Qnil);
 
   Fprovide (intern_c_string ("inotify"), Qnil);
 }
diff --git a/src/insdel.c b/src/insdel.c
index 02e3f41bc9f..e9279f0a166 100644
--- a/src/insdel.c
+++ b/src/insdel.c
@@ -2328,10 +2328,8 @@ DEFUN ("combine-after-change-execute", Fcombine_after_change_execute,
 void
 syms_of_insdel (void)
 {
-  staticpro (&combine_after_change_list);
-  staticpro (&combine_after_change_buffer);
-  combine_after_change_list = Qnil;
-  combine_after_change_buffer = Qnil;
+  staticpro (&combine_after_change_list, Qnil);
+  staticpro (&combine_after_change_buffer, Qnil);
 
   DEFSYM (Qundo_auto__undoable_change, "undo-auto--undoable-change");
 
diff --git a/src/keyboard.c b/src/keyboard.c
index c0c863f0d3e..a3a61ed0c63 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -11051,11 +11051,9 @@ static const struct event_head head_table[] = {
 void
 syms_of_keyboard (void)
 {
-  pending_funcalls = Qnil;
-  staticpro (&pending_funcalls);
+  staticpro (&pending_funcalls, Qnil);
 
-  Vlispy_mouse_stem = build_pure_c_string ("mouse");
-  staticpro (&Vlispy_mouse_stem);
+  staticpro (&Vlispy_mouse_stem, build_pure_c_string ("mouse"));
 
   regular_top_level_message = build_pure_c_string ("Back to top level");
 #ifdef HAVE_STACK_OVERFLOW_HANDLING
@@ -11071,13 +11069,10 @@ syms_of_keyboard (void)
   DEFSYM (Qhelp_echo, "help-echo");
   DEFSYM (QCrtl, ":rtl");
 
-  staticpro (&item_properties);
-  item_properties = Qnil;
+  staticpro (&item_properties, Qnil);
 
-  staticpro (&tool_bar_item_properties);
-  tool_bar_item_properties = Qnil;
-  staticpro (&tool_bar_items_vector);
-  tool_bar_items_vector = Qnil;
+  staticpro (&tool_bar_item_properties, Qnil);
+  staticpro (&tool_bar_items_vector, Qnil);
 
   DEFSYM (Qtimer_event_handler, "timer-event-handler");
 
@@ -11242,65 +11237,48 @@ syms_of_keyboard (void)
       }
   }
 
-  button_down_location = Fmake_vector (make_number (5), Qnil);
-  staticpro (&button_down_location);
-  mouse_syms = Fmake_vector (make_number (5), Qnil);
-  staticpro (&mouse_syms);
-  wheel_syms = Fmake_vector (make_number (ARRAYELTS (lispy_wheel_names)),
-			     Qnil);
-  staticpro (&wheel_syms);
+  staticpro (&button_down_location, Fmake_vector (make_number (5), Qnil));
+  staticpro (&mouse_syms, Fmake_vector (make_number (5), Qnil));
+  staticpro (&wheel_syms, Fmake_vector (make_number (ARRAYELTS (lispy_wheel_names)),
+                                        Qnil));
 
   {
     int i;
     int len = ARRAYELTS (modifier_names);
 
-    modifier_symbols = Fmake_vector (make_number (len), Qnil);
+    staticpro (&modifier_symbols, Fmake_vector (make_number (len), Qnil));
     for (i = 0; i < len; i++)
       if (modifier_names[i])
 	ASET (modifier_symbols, i, intern_c_string (modifier_names[i]));
-    staticpro (&modifier_symbols);
   }
 
-  recent_keys = Fmake_vector (make_number (NUM_RECENT_KEYS), Qnil);
-  staticpro (&recent_keys);
+  staticpro (&recent_keys, Fmake_vector (make_number (NUM_RECENT_KEYS), Qnil));
 
-  this_command_keys = Fmake_vector (make_number (40), Qnil);
-  staticpro (&this_command_keys);
+  staticpro (&this_command_keys, Fmake_vector (make_number (40), Qnil));
 
-  raw_keybuf = Fmake_vector (make_number (30), Qnil);
-  staticpro (&raw_keybuf);
+  staticpro (&raw_keybuf, Fmake_vector (make_number (30), Qnil));
 
   DEFSYM (Qcommand_execute, "command-execute");
   DEFSYM (Qinternal_echo_keystrokes_prefix, "internal-echo-keystrokes-prefix");
 
-  accent_key_syms = Qnil;
-  staticpro (&accent_key_syms);
+  staticpro (&accent_key_syms, Qnil);
 
-  func_key_syms = Qnil;
-  staticpro (&func_key_syms);
+  staticpro (&func_key_syms, Qnil);
 
-  drag_n_drop_syms = Qnil;
-  staticpro (&drag_n_drop_syms);
+  staticpro (&drag_n_drop_syms, Qnil);
 
-  unread_switch_frame = Qnil;
-  staticpro (&unread_switch_frame);
+  staticpro (&unread_switch_frame, Qnil);
 
-  internal_last_event_frame = Qnil;
-  staticpro (&internal_last_event_frame);
+  staticpro (&internal_last_event_frame, Qnil);
 
-  read_key_sequence_cmd = Qnil;
-  staticpro (&read_key_sequence_cmd);
-  read_key_sequence_remapped = Qnil;
-  staticpro (&read_key_sequence_remapped);
+  staticpro (&read_key_sequence_cmd, Qnil);
+  staticpro (&read_key_sequence_remapped, Qnil);
 
-  menu_bar_one_keymap_changed_items = Qnil;
-  staticpro (&menu_bar_one_keymap_changed_items);
+  staticpro (&menu_bar_one_keymap_changed_items, Qnil);
 
-  menu_bar_items_vector = Qnil;
-  staticpro (&menu_bar_items_vector);
+  staticpro (&menu_bar_items_vector, Qnil);
 
-  help_form_saved_window_configs = Qnil;
-  staticpro (&help_form_saved_window_configs);
+  staticpro (&help_form_saved_window_configs, Qnil);
 
   defsubr (&Scurrent_idle_time);
   defsubr (&Sevent_symbol_parse_modifiers);
diff --git a/src/keymap.c b/src/keymap.c
index c8cc933e782..7e8b548317e 100644
--- a/src/keymap.c
+++ b/src/keymap.c
@@ -3599,10 +3599,8 @@ void
 syms_of_keymap (void)
 {
   DEFSYM (Qkeymap, "keymap");
-  staticpro (&apropos_predicate);
-  staticpro (&apropos_accumulate);
-  apropos_predicate = Qnil;
-  apropos_accumulate = Qnil;
+  staticpro (&apropos_predicate, Qnil);
+  staticpro (&apropos_accumulate, Qnil);
 
   DEFSYM (Qkeymap_canonicalize, "keymap-canonicalize");
 
@@ -3614,13 +3612,10 @@ syms_of_keymap (void)
      Each one is the value of a Lisp variable, and is also
      pointed to by a C variable */
 
-  global_map = Fmake_keymap (Qnil);
+  staticpro (&global_map, Fmake_keymap (Qnil));
+  staticpro (&current_global_map, global_map);
   Fset (intern_c_string ("global-map"), global_map);
 
-  current_global_map = global_map;
-  staticpro (&global_map);
-  staticpro (&current_global_map);
-
   meta_map = Fmake_keymap (Qnil);
   Fset (intern_c_string ("esc-map"), meta_map);
   Ffset (intern_c_string ("ESC-prefix"), meta_map);
@@ -3629,13 +3624,13 @@ syms_of_keymap (void)
   Fset (intern_c_string ("ctl-x-map"), control_x_map);
   Ffset (intern_c_string ("Control-X-prefix"), control_x_map);
 
-  exclude_keys = listn (CONSTYPE_PURE, 5,
-			pure_cons (build_pure_c_string ("DEL"), build_pure_c_string ("\\d")),
-			pure_cons (build_pure_c_string ("TAB"), build_pure_c_string ("\\t")),
-			pure_cons (build_pure_c_string ("RET"), build_pure_c_string ("\\r")),
-			pure_cons (build_pure_c_string ("ESC"), build_pure_c_string ("\\e")),
-			pure_cons (build_pure_c_string ("SPC"), build_pure_c_string (" ")));
-  staticpro (&exclude_keys);
+  staticpro (&exclude_keys,
+             listn (CONSTYPE_PURE, 5,
+                    pure_cons (build_pure_c_string ("DEL"), build_pure_c_string ("\\d")),
+                    pure_cons (build_pure_c_string ("TAB"), build_pure_c_string ("\\t")),
+                    pure_cons (build_pure_c_string ("RET"), build_pure_c_string ("\\r")),
+                    pure_cons (build_pure_c_string ("ESC"), build_pure_c_string ("\\e")),
+                    pure_cons (build_pure_c_string ("SPC"), build_pure_c_string (" "))));
 
   DEFVAR_LISP ("define-key-rebound-commands", Vdefine_key_rebound_commands,
 	       doc: /* List of commands given new key bindings recently.
@@ -3689,17 +3684,16 @@ be preferred.  */);
   DEFSYM (Qmenu_bar, "menu-bar");
   DEFSYM (Qmode_line, "mode-line");
 
-  staticpro (&Vmouse_events);
-  Vmouse_events = listn (CONSTYPE_PURE, 9,
-			 Qmenu_bar,
-			 Qtool_bar,
-			 Qheader_line,
-			 Qmode_line,
-			 intern_c_string ("mouse-1"),
-			 intern_c_string ("mouse-2"),
-			 intern_c_string ("mouse-3"),
-			 intern_c_string ("mouse-4"),
-			 intern_c_string ("mouse-5"));
+  staticpro (&Vmouse_events, listn (CONSTYPE_PURE, 9,
+                                    Qmenu_bar,
+                                    Qtool_bar,
+                                    Qheader_line,
+                                    Qmode_line,
+                                    intern_c_string ("mouse-1"),
+                                    intern_c_string ("mouse-2"),
+                                    intern_c_string ("mouse-3"),
+                                    intern_c_string ("mouse-4"),
+                                    intern_c_string ("mouse-5")));
 
   /* Keymap used for minibuffers when doing completion.  */
   /* Keymap used for minibuffers when doing completion and require a match.  */
@@ -3709,13 +3703,11 @@ be preferred.  */);
   DEFSYM (Qremap, "remap");
   DEFSYM (QCadvertised_binding, ":advertised-binding");
 
-  command_remapping_vector = Fmake_vector (make_number (2), Qremap);
-  staticpro (&command_remapping_vector);
+  staticpro (&command_remapping_vector,
+             Fmake_vector (make_number (2), Qremap));
 
-  where_is_cache_keymaps = Qt;
-  where_is_cache = Qnil;
-  staticpro (&where_is_cache);
-  staticpro (&where_is_cache_keymaps);
+  staticpro (&where_is_cache, Qnil);
+  staticpro (&where_is_cache_keymaps, Qt);
 
   defsubr (&Skeymapp);
   defsubr (&Skeymap_parent);
diff --git a/src/kqueue.c b/src/kqueue.c
index 7a4f6a471c4..185e0e35820 100644
--- a/src/kqueue.c
+++ b/src/kqueue.c
@@ -527,7 +527,7 @@ syms_of_kqueue (void)
   DEFSYM (Qlink, "link");	/* NOTE_LINK  */
   DEFSYM (Qrename, "rename");	/* NOTE_RENAME  */
 
-  staticpro (&watch_list);
+  staticpro (&watch_list, Qnil);
 
   Fprovide (intern_c_string ("kqueue"), Qnil);
 }
diff --git a/src/lisp.h b/src/lisp.h
index b931d23bf38..19a8a8027e3 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -3281,7 +3281,23 @@ extern Lisp_Object Vascii_canon_table;
 \f
 /* Call staticpro (&var) to protect static variable `var'.  */
 
-void staticpro (Lisp_Object *);
+void staticpro_1 (Lisp_Object *);
+
+INLINE void
+staticpro (Lisp_Object *ptr, Lisp_Object initial_value)
+{
+  eassume (*ptr == Qnil);
+#if 4 < __GNUC__
+  if (__builtin_constant_p(initial_value == Qnil) &&
+      initial_value == Qnil)
+    ;
+  else
+#endif
+    *ptr = initial_value;
+
+  staticpro_1 (ptr);
+}
+
 \f
 /* Forward declarations for prototypes.  */
 struct window;
diff --git a/src/lread.c b/src/lread.c
index 8fb61f56338..e63d7431e46 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -4344,8 +4344,7 @@ void
 init_obarray (void)
 {
   Vobarray = Fmake_vector (make_number (OBARRAY_SIZE), make_number (0));
-  initial_obarray = Vobarray;
-  staticpro (&initial_obarray);
+  staticpro (&initial_obarray, Vobarray);
 
   for (int i = 0; i < ARRAYELTS (lispsym); i++)
     define_symbol (builtin_lisp_symbol (i), defsym_name[i]);
@@ -4443,7 +4442,7 @@ defvar_lisp (struct Lisp_Objfwd *o_fwd,
 	     const char *namestring, Lisp_Object *address)
 {
   defvar_lisp_nopro (o_fwd, namestring, address);
-  staticpro (address);
+  staticpro (address, Qnil);
 }
 
 /* Similar but define a variable whose value is the Lisp Object stored
@@ -5089,13 +5088,10 @@ this variable will become obsolete.  */);
   DEFSYM (Qdir_ok, "dir-ok");
   DEFSYM (Qdo_after_load_evaluation, "do-after-load-evaluation");
 
-  staticpro (&read_objects_map);
-  read_objects_map = Qnil;
-  staticpro (&read_objects_completed);
-  read_objects_completed = Qnil;
+  staticpro (&read_objects_map, Qnil);
+  staticpro (&read_objects_completed, Qnil);
 
-  Vloads_in_progress = Qnil;
-  staticpro (&Vloads_in_progress);
+  staticpro (&Vloads_in_progress, Qnil);
 
   DEFSYM (Qhash_table, "hash-table");
   DEFSYM (Qdata, "data");
diff --git a/src/macfont.m b/src/macfont.m
index 817071fa44f..5953ce7fd23 100644
--- a/src/macfont.m
+++ b/src/macfont.m
@@ -4050,6 +4050,5 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
   /* The boolean-valued font property key specifying the use of leading.  */
   DEFSYM (QCminspace, ":minspace");
 
-  macfont_family_cache = Qnil;
-  staticpro (&macfont_family_cache);
+  staticpro (&macfont_family_cache, Qnil);
 }
diff --git a/src/menu.c b/src/menu.c
index 93e793a5d91..3be8ecd5fab 100644
--- a/src/menu.c
+++ b/src/menu.c
@@ -1580,8 +1580,7 @@ for instance using the window manager, then this produces a quit and
 void
 syms_of_menu (void)
 {
-  staticpro (&menu_items);
-  menu_items = Qnil;
+  staticpro (&menu_items, Qnil);
   menu_items_inuse = Qnil;
 
   defsubr (&Sx_popup_menu);
diff --git a/src/minibuf.c b/src/minibuf.c
index 45cf15224a5..11c6b94ae5d 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -1908,19 +1908,16 @@ If no minibuffer is active, return nil.  */)
 void
 init_minibuf_once (void)
 {
-  Vminibuffer_list = Qnil;
-  staticpro (&Vminibuffer_list);
+  staticpro (&Vminibuffer_list, Qnil);
 }
 
 void
 syms_of_minibuf (void)
 {
   minibuf_level = 0;
-  minibuf_prompt = Qnil;
-  staticpro (&minibuf_prompt);
+  staticpro (&minibuf_prompt, Qnil);
 
-  minibuf_save_list = Qnil;
-  staticpro (&minibuf_save_list);
+  staticpro (&minibuf_save_list, Qnil);
 
   DEFSYM (Qcompletion_ignore_case, "completion-ignore-case");
   DEFSYM (Qminibuffer_default, "minibuffer-default");
@@ -1928,8 +1925,7 @@ syms_of_minibuf (void)
 
   DEFSYM (Qminibuffer_completion_table, "minibuffer-completion-table");
 
-  staticpro (&last_minibuf_string);
-  last_minibuf_string = Qnil;
+  staticpro (&last_minibuf_string, Qnil);
 
   DEFSYM (Qminibuffer_history, "minibuffer-history");
   DEFSYM (Qbuffer_name_history, "buffer-name-history");
diff --git a/src/msdos.c b/src/msdos.c
index eedbf7b1a6c..7ca6d03917a 100644
--- a/src/msdos.c
+++ b/src/msdos.c
@@ -4196,8 +4196,7 @@ msdos_fatal_signal (int sig)
 void
 syms_of_msdos (void)
 {
-  recent_doskeys = Fmake_vector (make_number (NUM_RECENT_DOSKEYS), Qnil);
-  staticpro (&recent_doskeys);
+  staticpro (&recent_doskeys, Fmake_vector (make_number (NUM_RECENT_DOSKEYS), Qnil));
 
 #ifndef HAVE_X_WINDOWS
 
diff --git a/src/nsselect.m b/src/nsselect.m
index c72f179ab38..6d0b6342dc2 100644
--- a/src/nsselect.m
+++ b/src/nsselect.m
@@ -492,8 +492,7 @@ Updated by Christian Limpach (chris@nice.ch)
   defsubr (&Sns_selection_exists_p);
   defsubr (&Sns_selection_owner_p);
 
-  Vselection_alist = Qnil;
-  staticpro (&Vselection_alist);
+  staticpro (&Vselection_alist, Qnil);
 
   DEFVAR_LISP ("ns-sent-selection-hooks", Vns_sent_selection_hooks,
                "A list of functions to be called when Emacs answers a selection request.\n\
diff --git a/src/print.c b/src/print.c
index a8bbb9d37a1..6867981102b 100644
--- a/src/print.c
+++ b/src/print.c
@@ -2435,7 +2435,7 @@ priorities.  */);
   Vprint_charset_text_property = Qdefault;
 
   /* prin1_to_string_buffer initialized in init_buffer_once in buffer.c */
-  staticpro (&Vprin1_to_string_buffer);
+  staticpro (&Vprin1_to_string_buffer, Qnil);
 
   defsubr (&Sprin1);
   defsubr (&Sprin1_to_string);
@@ -2452,6 +2452,5 @@ priorities.  */);
   DEFSYM (Qprint_escape_nonascii, "print-escape-nonascii");
   DEFSYM (Qprint_escape_control_characters, "print-escape-control-characters");
 
-  print_prune_charset_plist = Qnil;
-  staticpro (&print_prune_charset_plist);
+  staticpro (&print_prune_charset_plist, Qnil);
 }
diff --git a/src/process.c b/src/process.c
index ed2cab7b51f..25634f753e3 100644
--- a/src/process.c
+++ b/src/process.c
@@ -8149,8 +8149,8 @@ syms_of_process (void)
 
   DEFSYM (Qlast_nonmenu_event, "last-nonmenu-event");
 
-  staticpro (&Vprocess_alist);
-  staticpro (&deleted_pid_list);
+  staticpro (&Vprocess_alist, Qnil);
+  staticpro (&deleted_pid_list, Qnil);
 
 #endif	/* subprocesses */
 
diff --git a/src/profiler.c b/src/profiler.c
index 312574d7526..b1e54b6a748 100644
--- a/src/profiler.c
+++ b/src/profiler.c
@@ -595,16 +595,14 @@ to make room for new entries.  */);
 
 #ifdef PROFILER_CPU_SUPPORT
   profiler_cpu_running = NOT_RUNNING;
-  cpu_log = Qnil;
-  staticpro (&cpu_log);
+  staticpro (&cpu_log, Qnil);
   defsubr (&Sprofiler_cpu_start);
   defsubr (&Sprofiler_cpu_stop);
   defsubr (&Sprofiler_cpu_running_p);
   defsubr (&Sprofiler_cpu_log);
 #endif
   profiler_memory_running = false;
-  memory_log = Qnil;
-  staticpro (&memory_log);
+  staticpro (&memory_log, Qnil);
   defsubr (&Sprofiler_memory_start);
   defsubr (&Sprofiler_memory_stop);
   defsubr (&Sprofiler_memory_running_p);
diff --git a/src/search.c b/src/search.c
index 842e9309a2e..aa4e39bdbbc 100644
--- a/src/search.c
+++ b/src/search.c
@@ -3367,12 +3367,9 @@ syms_of_search (void)
       searchbufs[i].buf.allocated = 100;
       searchbufs[i].buf.buffer = xmalloc (100);
       searchbufs[i].buf.fastmap = searchbufs[i].fastmap;
-      searchbufs[i].regexp = Qnil;
-      searchbufs[i].f_whitespace_regexp = Qnil;
-      searchbufs[i].syntax_table = Qnil;
-      staticpro (&searchbufs[i].regexp);
-      staticpro (&searchbufs[i].f_whitespace_regexp);
-      staticpro (&searchbufs[i].syntax_table);
+      staticpro (&searchbufs[i].regexp, Qnil);
+      staticpro (&searchbufs[i].f_whitespace_regexp, Qnil);
+      staticpro (&searchbufs[i].syntax_table, Qnil);
       searchbufs[i].next = (i == REGEXP_CACHE_SIZE-1 ? 0 : &searchbufs[i+1]);
     }
   searchbuf_head = &searchbufs[0];
@@ -3403,11 +3400,9 @@ syms_of_search (void)
   Fput (Qinvalid_regexp, Qerror_message,
 	build_pure_c_string ("Invalid regexp"));
 
-  last_thing_searched = Qnil;
-  staticpro (&last_thing_searched);
+  staticpro (&last_thing_searched, Qnil);
 
-  saved_last_thing_searched = Qnil;
-  staticpro (&saved_last_thing_searched);
+  staticpro (&saved_last_thing_searched, Qnil);
 
   DEFVAR_LISP ("search-spaces-regexp", Vsearch_spaces_regexp,
       doc: /* Regexp to substitute for bunches of spaces in regexp search.
diff --git a/src/syntax.c b/src/syntax.c
index c5a4b03955b..3ab82ad5d64 100644
--- a/src/syntax.c
+++ b/src/syntax.c
@@ -3722,15 +3722,15 @@ syms_of_syntax (void)
 	       doc: /* Non-nil means `forward-comment' can use `syntax-ppss' internally.  */);
   Vcomment_use_syntax_ppss = Qt;
 
-  staticpro (&Vsyntax_code_object);
+  staticpro (&Vsyntax_code_object, Qnil);
 
-  staticpro (&gl_state.object);
-  staticpro (&gl_state.global_code);
-  staticpro (&gl_state.current_syntax_table);
-  staticpro (&gl_state.old_prop);
+  staticpro (&gl_state.object, Qnil);
+  staticpro (&gl_state.global_code, Qnil);
+  staticpro (&gl_state.current_syntax_table, Qnil);
+  staticpro (&gl_state.old_prop, Qnil);
 
   /* Defined in regex.c.  */
-  staticpro (&re_match_object);
+  staticpro (&re_match_object, Qnil);
 
   DEFSYM (Qscan_error, "scan-error");
   Fput (Qscan_error, Qerror_conditions,
diff --git a/src/textprop.c b/src/textprop.c
index 984f2e66406..7f7d7359ffc 100644
--- a/src/textprop.c
+++ b/src/textprop.c
@@ -2340,10 +2340,8 @@ inherits it if NONSTICKINESS is nil.  The `front-sticky' and
   Vtext_property_default_nonsticky
     = list2 (Fcons (Qsyntax_table, Qt), Fcons (Qdisplay, Qt));
 
-  staticpro (&interval_insert_behind_hooks);
-  staticpro (&interval_insert_in_front_hooks);
-  interval_insert_behind_hooks = Qnil;
-  interval_insert_in_front_hooks = Qnil;
+  staticpro (&interval_insert_behind_hooks, Qnil);
+  staticpro (&interval_insert_in_front_hooks, Qnil);
 
 
   /* Common attributes one might give text.  */
diff --git a/src/thread.c b/src/thread.c
index f11e3e5addb..fe0c8688415 100644
--- a/src/thread.c
+++ b/src/thread.c
@@ -1074,8 +1074,7 @@ syms_of_threads (void)
       defsubr (&Scondition_name);
       defsubr (&Sthread_last_error);
 
-      staticpro (&last_thread_error);
-      last_thread_error = Qnil;
+      staticpro (&last_thread_error, Qnil);
     }
 
   DEFSYM (Qthreadp, "threadp");
diff --git a/src/undo.c b/src/undo.c
index c34faa42720..a34dd486094 100644
--- a/src/undo.c
+++ b/src/undo.c
@@ -430,8 +430,7 @@ syms_of_undo (void)
   /* Marker for function call undo list elements.  */
   DEFSYM (Qapply, "apply");
 
-  pending_boundary = Qnil;
-  staticpro (&pending_boundary);
+  staticpro (&pending_boundary, Qnil);
 
   defsubr (&Sundo_boundary);
 
diff --git a/src/w32fns.c b/src/w32fns.c
index 2b920f29c65..5903b2a1557 100644
--- a/src/w32fns.c
+++ b/src/w32fns.c
@@ -10164,8 +10164,7 @@ syms_of_w32fns (void)
   Fput (Qundefined_color, Qerror_message,
 	build_pure_c_string ("Undefined color"));
 
-  staticpro (&w32_grabbed_keys);
-  w32_grabbed_keys = Qnil;
+  staticpro (&w32_grabbed_keys, Qnil);
 
   DEFVAR_LISP ("w32-color-map", Vw32_color_map,
 	       doc: /* An array of color name mappings for Windows.  */);
@@ -10515,16 +10514,11 @@ tip frame.  */);
   defsubr (&Sset_message_beep);
   defsubr (&Sx_show_tip);
   defsubr (&Sx_hide_tip);
-  tip_timer = Qnil;
-  staticpro (&tip_timer);
-  tip_frame = Qnil;
-  staticpro (&tip_frame);
-  tip_last_frame = Qnil;
-  staticpro (&tip_last_frame);
-  tip_last_string = Qnil;
-  staticpro (&tip_last_string);
-  tip_last_parms = Qnil;
-  staticpro (&tip_last_parms);
+  staticpro (&tip_timer, Qnil);
+  staticpro (&tip_frame, Qnil);
+  staticpro (&tip_last_frame, Qnil);
+  staticpro (&tip_last_string, Qnil);
+  staticpro (&tip_last_parms, Qnil);
 
   defsubr (&Sx_file_dialog);
 #ifdef WINDOWSNT
diff --git a/src/w32proc.c b/src/w32proc.c
index 28d7b6611f6..83b8ef7f56d 100644
--- a/src/w32proc.c
+++ b/src/w32proc.c
@@ -3826,7 +3826,7 @@ in the strings.  So do NOT use this option when comparing file names
 for equality, only when you need to sort them.  */);
   Vw32_collate_ignore_punctuation = Qnil;
 
-  staticpro (&Vw32_valid_locale_ids);
-  staticpro (&Vw32_valid_codepages);
+  staticpro (&Vw32_valid_locale_ids, Qnil);
+  staticpro (&Vw32_valid_codepages, Qnil);
 }
 /* end of w32proc.c */
diff --git a/src/w32select.c b/src/w32select.c
index a9df3f770b7..cd1bf6a60cb 100644
--- a/src/w32select.c
+++ b/src/w32select.c
@@ -1181,13 +1181,13 @@ syms_of_w32select (void)
 
   DEFSYM (QCLIPBOARD, "CLIPBOARD");
 
-  cfg_coding_system = Qnil;     staticpro (&cfg_coding_system);
-  current_text = Qnil;		staticpro (&current_text);
-  current_coding_system = Qnil; staticpro (&current_coding_system);
+  staticpro (&cfg_coding_system, Qnil);
+  staticpro (&current_text, Qnil);
+  staticpro (&current_coding_system, Qnil);
 
   DEFSYM (Qutf_16le_dos, "utf-16le-dos");
-  QANSICP = Qnil; staticpro (&QANSICP);
-  QOEMCP = Qnil;  staticpro (&QOEMCP);
+  staticpro (&QANSICP, Qnil);
+  staticpro (&QOEMCP, Qnil);
 }
 
 /* One-time init.  Called in the dumped Emacs, but not in the
diff --git a/src/window.c b/src/window.c
index e6d0280d9b0..581d529bf8c 100644
--- a/src/window.c
+++ b/src/window.c
@@ -7566,10 +7566,8 @@ syms_of_window (void)
   DEFSYM (Qmode_line_format, "mode-line-format");
   DEFSYM (Qheader_line_format, "header-line-format");
 
-  staticpro (&Vwindow_list);
-
-  minibuf_selected_window = Qnil;
-  staticpro (&minibuf_selected_window);
+  staticpro (&Vwindow_list, Qnil);
+  staticpro (&minibuf_selected_window, Qnil);
 
   window_scroll_pixel_based_preserve_x = -1;
   window_scroll_pixel_based_preserve_y = -1;
diff --git a/src/xdisp.c b/src/xdisp.c
index df5335e4acc..e194f767dae 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -32418,11 +32418,9 @@ x_intersect_rectangles (XRectangle *r1, XRectangle *r2, XRectangle *result)
 void
 syms_of_xdisp (void)
 {
-  Vwith_echo_area_save_vector = Qnil;
-  staticpro (&Vwith_echo_area_save_vector);
+  staticpro (&Vwith_echo_area_save_vector, Qnil);
 
-  Vmessage_stack = Qnil;
-  staticpro (&Vmessage_stack);
+  staticpro (&Vmessage_stack, Qnil);
 
   /* Non-nil means don't actually do any redisplay.  */
   DEFSYM (Qinhibit_redisplay, "inhibit-redisplay");
@@ -32434,12 +32432,9 @@ syms_of_xdisp (void)
 They are still logged to the *Messages* buffer.  */);
   inhibit_message = 0;
 
-  message_dolog_marker1 = Fmake_marker ();
-  staticpro (&message_dolog_marker1);
-  message_dolog_marker2 = Fmake_marker ();
-  staticpro (&message_dolog_marker2);
-  message_dolog_marker3 = Fmake_marker ();
-  staticpro (&message_dolog_marker3);
+  staticpro (&message_dolog_marker1, Fmake_marker ());
+  staticpro (&message_dolog_marker2, Fmake_marker ());
+  staticpro (&message_dolog_marker3, Fmake_marker ());
 
   defsubr (&Sset_buffer_redisplay);
 #ifdef GLYPH_DEBUG
@@ -32553,8 +32548,7 @@ They are still logged to the *Messages* buffer.  */);
 
   DEFSYM (Qinhibit_free_realized_faces, "inhibit-free-realized-faces");
 
-  list_of_error = list1 (list2 (Qerror, Qvoid_variable));
-  staticpro (&list_of_error);
+  staticpro (&list_of_error, list1 (list2 (Qerror, Qvoid_variable)));
 
   /* Values of those variables at last redisplay are stored as
      properties on 'overlay-arrow-position' symbol.  However, if
@@ -32568,38 +32562,26 @@ They are still logged to the *Messages* buffer.  */);
   DEFSYM (Qoverlay_arrow_string, "overlay-arrow-string");
   DEFSYM (Qoverlay_arrow_bitmap, "overlay-arrow-bitmap");
 
-  echo_buffer[0] = echo_buffer[1] = Qnil;
-  staticpro (&echo_buffer[0]);
-  staticpro (&echo_buffer[1]);
+  staticpro (&echo_buffer[0], Qnil);
+  staticpro (&echo_buffer[1], Qnil);
 
-  echo_area_buffer[0] = echo_area_buffer[1] = Qnil;
-  staticpro (&echo_area_buffer[0]);
-  staticpro (&echo_area_buffer[1]);
+  staticpro (&echo_area_buffer[0], Qnil);
+  staticpro (&echo_area_buffer[1], Qnil);
 
-  Vmessages_buffer_name = build_pure_c_string ("*Messages*");
-  staticpro (&Vmessages_buffer_name);
+  staticpro (&Vmessages_buffer_name, build_pure_c_string ("*Messages*"));
 
-  mode_line_proptrans_alist = Qnil;
-  staticpro (&mode_line_proptrans_alist);
-  mode_line_string_list = Qnil;
-  staticpro (&mode_line_string_list);
-  mode_line_string_face = Qnil;
-  staticpro (&mode_line_string_face);
-  mode_line_string_face_prop = Qnil;
-  staticpro (&mode_line_string_face_prop);
-  Vmode_line_unwind_vector = Qnil;
-  staticpro (&Vmode_line_unwind_vector);
+  staticpro (&mode_line_proptrans_alist, Qnil);
+  staticpro (&mode_line_string_list, Qnil);
+  staticpro (&mode_line_string_face, Qnil);
+  staticpro (&mode_line_string_face_prop, Qnil);
+  staticpro (&Vmode_line_unwind_vector, Qnil);
 
   DEFSYM (Qmode_line_default_help_echo, "mode-line-default-help-echo");
 
-  help_echo_string = Qnil;
-  staticpro (&help_echo_string);
-  help_echo_object = Qnil;
-  staticpro (&help_echo_object);
-  help_echo_window = Qnil;
-  staticpro (&help_echo_window);
-  previous_help_echo_string = Qnil;
-  staticpro (&previous_help_echo_string);
+  staticpro (&help_echo_string, Qnil);
+  staticpro (&help_echo_object, Qnil);
+  staticpro (&help_echo_window, Qnil);
+  staticpro (&previous_help_echo_string, Qnil);
   help_echo_pos = -1;
 
   DEFSYM (Qright_to_left, "right-to-left");
diff --git a/src/xfaces.c b/src/xfaces.c
index a9c2f37e9f2..89c1065750e 100644
--- a/src/xfaces.c
+++ b/src/xfaces.c
@@ -6451,12 +6451,9 @@ syms_of_xfaces (void)
   /* The name of the function used to compute colors on TTYs.  */
   DEFSYM (Qtty_color_alist, "tty-color-alist");
 
-  Vparam_value_alist = list1 (Fcons (Qnil, Qnil));
-  staticpro (&Vparam_value_alist);
-  Vface_alternative_font_family_alist = Qnil;
-  staticpro (&Vface_alternative_font_family_alist);
-  Vface_alternative_font_registry_alist = Qnil;
-  staticpro (&Vface_alternative_font_registry_alist);
+  staticpro (&Vparam_value_alist, list1 (Fcons (Qnil, Qnil)));
+  staticpro (&Vface_alternative_font_family_alist, Qnil);
+  staticpro (&Vface_alternative_font_registry_alist, Qnil);
 
   defsubr (&Sinternal_make_lisp_face);
   defsubr (&Sinternal_lisp_face_p);
diff --git a/src/xfns.c b/src/xfns.c
index 78151c81380..38d9f91dc89 100644
--- a/src/xfns.c
+++ b/src/xfns.c
@@ -7926,16 +7926,11 @@ When using Gtk+ tooltips, the tooltip face is not used.  */);
   defsubr (&Sx_show_tip);
   defsubr (&Sx_hide_tip);
   defsubr (&Sx_double_buffered_p);
-  tip_timer = Qnil;
-  staticpro (&tip_timer);
-  tip_frame = Qnil;
-  staticpro (&tip_frame);
-  tip_last_frame = Qnil;
-  staticpro (&tip_last_frame);
-  tip_last_string = Qnil;
-  staticpro (&tip_last_string);
-  tip_last_parms = Qnil;
-  staticpro (&tip_last_parms);
+  staticpro (&tip_timer, Qnil);
+  staticpro (&tip_frame, Qnil);
+  staticpro (&tip_last_frame, Qnil);
+  staticpro (&tip_last_string, Qnil);
+  staticpro (&tip_last_parms, Qnil);
 
   defsubr (&Sx_uses_old_gtk_dialog);
 #if defined (USE_MOTIF) || defined (USE_GTK)
diff --git a/src/xfont.c b/src/xfont.c
index c2e416bc058..a67237aa2aa 100644
--- a/src/xfont.c
+++ b/src/xfont.c
@@ -1098,9 +1098,7 @@ struct font_driver const xfont_driver =
 void
 syms_of_xfont (void)
 {
-  staticpro (&xfont_scripts_cache);
-  xfont_scripts_cache = CALLN (Fmake_hash_table, QCtest, Qequal);
-  staticpro (&xfont_scratch_props);
-  xfont_scratch_props = Fmake_vector (make_number (8), Qnil);
+  staticpro (&xfont_scripts_cache, CALLN (Fmake_hash_table, QCtest, Qequal));
+  staticpro (&xfont_scratch_props, Fmake_vector (make_number (8), Qnil));
   register_font_driver (&xfont_driver, NULL);
 }
diff --git a/src/xselect.c b/src/xselect.c
index ecf59df2943..538b555f643 100644
--- a/src/xselect.c
+++ b/src/xselect.c
@@ -2633,15 +2633,13 @@ syms_of_xselect (void)
   defsubr (&Sx_send_client_message);
   defsubr (&Sx_register_dnd_atom);
 
-  reading_selection_reply = Fcons (Qnil, Qnil);
-  staticpro (&reading_selection_reply);
+  staticpro (&reading_selection_reply, Fcons (Qnil, Qnil));
   reading_selection_window = 0;
   reading_which_selection = 0;
 
   property_change_wait_list = 0;
   prop_location_identifier = 0;
-  property_change_reply = Fcons (Qnil, Qnil);
-  staticpro (&property_change_reply);
+  staticpro (&property_change_reply, Fcons (Qnil, Qnil));
 
   converted_selections = NULL;
   conversion_fail_tag = None;
diff --git a/src/xterm.c b/src/xterm.c
index 6ab4a03002d..d1c3797cf06 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -13257,8 +13257,7 @@ syms_of_xterm (void)
   DEFSYM (Qlatin_1, "latin-1");
 
 #ifdef USE_GTK
-  xg_default_icon_file = build_pure_c_string ("icons/hicolor/scalable/apps/emacs.svg");
-  staticpro (&xg_default_icon_file);
+  staticpro (&xg_default_icon_file, build_pure_c_string ("icons/hicolor/scalable/apps/emacs.svg"));
 
   DEFSYM (Qx_gtk_map_stock, "x-gtk-map-stock");
 #endif

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Thu, 29 Mar 2018 00:34:16 -0700
> 
> I'm not sure what's left to test. It works. What's stopping a merge?

I think what is left is minor.  Some things that come to mind:

  . Some minimal documentation (manual, NEWS).

  . I don't think versioned installs are supported, as the code will
    always try to load emacs.pdmp, AFAIR, even when the binary is
    called emacs-XX.YY.ZZ.

  . There's also the issue of the "legacy" dumping.  I'm guessing we
    will want to leave that in the sources for the time being?  If so,
    did anyone try to build the branch using that, and did it work as
    expected?

  . One other thing to try is to build on one system, then unpack the
    binary distro on another binary-compatible system, perhaps with a
    different directory tree structure, and see if everything works.

Re: Preview: portable dumper

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Thu, 29 Mar 2018 00:34:16 -0700
>> 
>> I'm not sure what's left to test. It works. What's stopping a merge?
>
> I think what is left is minor.  Some things that come to mind:
>
>   . Some minimal documentation (manual, NEWS).

And in configure. Itʼs not clear to me what the various combinations
of --with-dumping and --with-pdumper are supposed to see (see below).

>   . There's also the issue of the "legacy" dumping.  I'm guessing we
>     will want to leave that in the sources for the time being?  If so,
>     did anyone try to build the branch using that, and did it work as
>     expected?

./configure --with-dumping=unexec --with-pdumper=no && src/emacs -Q
# uses unexec, doesnʼt support pdump. OK

./configure --with-dumping=unexec && src/emacs -Q
# uses unexec, doesnʼt support pdump. OK

./configure --with-dumping=unexec --with-pdumper=yes && src/emacs -Q
# uses unexec, supports pdump. Runs OK. What does 'supports pdump'
mean in this context? That we'll try to load a dump file even though
we've been built with unexec?

In any case, the old unexec code looks like it still works on my
system.

Robert

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Daniel Colascione <dancol@dancol.org>,  angelo.g0@libero.it,  emacs-devel@gnu.org
> Date: Thu, 29 Mar 2018 15:03:05 +0200
> 
> In any case, the old unexec code looks like it still works on my
> system.

Thanks for testing it.

Re: Preview: portable dumper

[Angelo Graziosi]

> Il 29 marzo 2018 alle 13.53 Eli Zaretskii ha scritto:
> 
> 
> > From: Daniel Colascione
> > Date: Thu, 29 Mar 2018 00:34:16 -0700
> > 
> > I'm not sure what's left to test. It works. What's stopping a merge?
> 
> I think what is left is minor.  Some things that come to mind:
> 
>  [...]
>   . One other thing to try is to build on one system, then unpack the
>     binary distro on another binary-compatible system, perhaps with a
>     different directory tree structure, and see if everything works.

I build Emacs also for Windows Subsystem for Linux (WSL) but I do the build on GNU/Linux Mint (*) because it is a bit faster. The build creates a tar ball which I then install on WSL (both in Win 10 Pro 64 and in Win 10 Home 64). This build does not use X because WSL doesn't too..

Generally, the build I do on Win 10 64 (W64 native, i.e. MSYS2/MinGW64) is also installed in Win 10 Home.


---
(*) Both Mint and WSL uses Ubuntu 16.04

Re: Preview: portable dumper

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> There's also the issue of the "legacy" dumping. I'm guessing we will want
EZ> to leave that in the sources for the time being?

I think we should, but not for much longer than two major release cycles.

John

Re: Preview: portable dumper

[Stefan Monnier]

> EZ> There's also the issue of the "legacy" dumping. I'm guessing we will want
> EZ> to leave that in the sources for the time being?
> I think we should, but not for much longer than two major release cycles.
                                                  ^^^
                                                  one
-- Stefan

Re: Preview: portable dumper

[Alan Third]

On Thu, Mar 29, 2018 at 12:34:16AM -0700, Daniel Colascione wrote:
>
> I'm not sure what's left to test. It works. What's stopping a merge?
> 

I just tried to build on macOS for the first time in a while and I get
this in ‘make install’

    Unknown option ‘--temacs=pbootstrap’
    make[1]: *** [bootstrap-emacs.pdmp] Error 255
    make: *** [src] Error 2
    make: *** Waiting for unfinished jobs....

I also think we still need to get the pdmp file copied into the
Emacs.app file structure.
-- 
Alan Third

Re: Preview: portable dumper

[dancol]

> On Thu, Mar 29, 2018 at 12:34:16AM -0700, Daniel Colascione wrote:
>>
>> I'm not sure what's left to test. It works. What's stopping a merge?
>>
>
> I just tried to build on macOS for the first time in a while and I get
> this in ‘make install’
>
>     Unknown option ‘--temacs=pbootstrap’

Is pdumper enabled in your config.h?

>     make[1]: *** [bootstrap-emacs.pdmp] Error 255
>     make: *** [src] Error 2
>     make: *** Waiting for unfinished jobs....
>
> I also think we still need to get the pdmp file copied into the
> Emacs.app file structure.

Probably.

Re: Preview: portable dumper

[Alan Third]

On Thu, Mar 29, 2018 at 12:32:29PM -0700, dancol@dancol.org wrote:
> > On Thu, Mar 29, 2018 at 12:34:16AM -0700, Daniel Colascione wrote:
> >>
> >> I'm not sure what's left to test. It works. What's stopping a merge?
> >>
> >
> > I just tried to build on macOS for the first time in a while and I get
> > this in ‘make install’
> >
> >     Unknown option ‘--temacs=pbootstrap’
> 
> Is pdumper enabled in your config.h?

Sorry, never mind. I rebuilt from scratch and it’s fine. Must have
been something left over from a previous build.

-- 
Alan Third

Re: Preview: portable dumper

[Alan Third]

[-- Attachment #1: Type: text/plain, Size: 297 bytes --]

On Thu, Mar 29, 2018 at 08:17:57PM +0100, Alan Third wrote:
> I also think we still need to get the pdmp file copied into the
> Emacs.app file structure.

Patch attached. It looks like it works but I don’t know much about
makefiles so I don’t know if it’s the right approach.
-- 
Alan Third

[-- Attachment #2: 0001-Copy-pdmp-file-to-Emacs.app.patch --]
[-- Type: text/plain, Size: 1043 bytes --]

From faaaeceec4a9f199942bd35b1b83e0345c828d82 Mon Sep 17 00:00:00 2001
From: Alan Third <alan@idiocy.org>
Date: Sat, 31 Mar 2018 10:49:59 +0100
Subject: [PATCH] Copy pdmp file to Emacs.app

* nextstep/Makefile.in: Handle emacs.pdmp file.
---
 nextstep/Makefile.in | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/nextstep/Makefile.in b/nextstep/Makefile.in
index cb69898737..696a18b27d 100644
--- a/nextstep/Makefile.in
+++ b/nextstep/Makefile.in
@@ -44,7 +44,7 @@ ns_check_file =
 
 .PHONY: all
 
-all: ${ns_appdir} ${ns_appbindir}/Emacs
+all: ${ns_appdir} ${ns_appbindir}/Emacs ${ns_appbindir}/emacs.pdmp
 
 ${ns_check_file} ${ns_appdir}: ${srcdir}/${ns_appsrc} ${ns_appsrc}
 	rm -rf ${ns_appdir}
@@ -63,6 +63,10 @@ ${ns_appbindir}/Emacs:
 	${MKDIR_P} ${ns_appbindir}
 	cp -f ../src/emacs${EXEEXT} $@
 
+${ns_appbindir}/emacs.pdmp: ${ns_appdir} ${ns_check_file} ../src/emacs${EXEEXT}.pdmp
+	${MKDIR_P} ${ns_appbindir}
+	cp -f ../src/emacs${EXEEXT}.pdmp $@
+
 .PHONY: FORCE
 
 ../src/emacs${EXEEXT}: FORCE
-- 
2.16.1

Re: Preview: portable dumper

[Robert Pluim]

Alan Third <alan@idiocy.org> writes:

> On Thu, Mar 29, 2018 at 08:17:57PM +0100, Alan Third wrote:
>> I also think we still need to get the pdmp file copied into the
>> Emacs.app file structure.
>
> Patch attached. It looks like it works but I don’t know much about
> makefiles so I don’t know if it’s the right approach.

The pdumper branch has not yet been merged to master.

Is there anything holding up the merge? It worked fine last time I
tested it.

Regards

Robert

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Fri, 08 Jun 2018 11:29:29 +0200
> Cc: Alan Third <alan@idiocy.org>, Daniel Colascione <dancol@dancol.org>,
> 	Angelo Graziosi <angelo.g0@libero.it>
> 
> The pdumper branch has not yet been merged to master.
> 
> Is there anything holding up the merge?

Not much:

  http://lists.gnu.org/archive/html/emacs-devel/2018-03/msg00924.html
  http://lists.gnu.org/archive/html/emacs-devel/2018-03/msg00929.html
  http://lists.gnu.org/archive/html/emacs-devel/2018-03/msg00974.html

Re: Preview: portable dumper

[Alan Third]

On Fri, Jun 08, 2018 at 01:05:36PM +0300, Eli Zaretskii wrote:
> > From: Robert Pluim <rpluim@gmail.com>
> > Date: Fri, 08 Jun 2018 11:29:29 +0200
> > Cc: Alan Third <alan@idiocy.org>, Daniel Colascione <dancol@dancol.org>,
> > 	Angelo Graziosi <angelo.g0@libero.it>
> > 
> > The pdumper branch has not yet been merged to master.
> > 
> > Is there anything holding up the merge?
> 
> Not much:
> 
>   http://lists.gnu.org/archive/html/emacs-devel/2018-03/msg00924.html
>   http://lists.gnu.org/archive/html/emacs-devel/2018-03/msg00929.html
>   http://lists.gnu.org/archive/html/emacs-devel/2018-03/msg00974.html

I pushed the patch in the third link to the pdumper branch a while
back since nobody commented.

-- 
Alan Third

Re: Preview: portable dumper

[Eli Zaretskii]

> Date: Tue, 13 Feb 2018 23:06:15 +0100 (CET)
> From: Angelo Graziosi <angelo.g0@libero.it>
> 
> Eli Zaretskii wrote:
> >
> > I'd urge people to try this branch and report any issues they see.
> 
> What should we look for?

The usual stuff, I think: does it build, does it work as expected,
etc.

Re: Preview: portable dumper

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Tue, 13 Feb 2018 23:06:15 +0100 (CET)
>> From: Angelo Graziosi <angelo.g0@libero.it>
>> 
>> Eli Zaretskii wrote:
>> >
>> > I'd urge people to try this branch and report any issues they see.
>> 
>> What should we look for?
>
> The usual stuff, I think: does it build, does it work as expected,
> etc.

You mean like:

make[3]: Entering directory '/home/rpluim/repos/emacs-pdumper/admin/grammars'
  GEN      ../../lisp/cedet/semantic/bovine/c-by.el
load_dump completed in 63.0991 milliseconds
../../lisp/emacs-lisp/eieio.el: ‘object-print’ is an obsolete generic function (as of 26.1); use ‘cl-print-object’ instead.
../../lisp/emacs-lisp/eieio-base.el: ‘eieio-object-set-name-string’ is an obsolete generic function (as of 25.1); inherit from ‘eieio-named’ and use (setf (slot-value OBJ ’object-name) NAME) instead
../../lisp/cedet/semantic/db.el: ‘object-print’ is an obsolete generic function (as of 26.1); use ‘cl-print-object’ instead.
../../lisp/cedet/semantic/db.el: ‘object-print’ is an obsolete generic function (as of 26.1); use ‘cl-print-object’ instead.
../../lisp/cedet/semantic/db.el: ‘object-print’ is an obsolete generic function (as of 26.1); use ‘cl-print-object’ instead.
../../lisp/cedet/semantic/db-ref.el: Obsolete name arg "DEBUG" to constructor semanticdb-ref-adebug
Fatal error 11: Segmentation fault

This is with gcc 5.4.0 on Ubuntu 16.04. I rebuilt with '-O0 -ggdb' and
trapped it in gdb.

(gdb) l
4534	  int bits_per_word = xword_size * CHAR_BIT;
4535	  dump_bitset_word * slot = dump_bitset__bit_slot (bitset, bit_number);
4536	  dump_bitset_word bit = 1;
4537	  bit <<= bit_number % bits_per_word;
4538	  if (bit_is_set)
4539	    *slot = *slot | bit;
4540	  else
4541	    *slot = *slot & ~bit;
4542	}

(gdb) p slot
$1 = (dump_bitset_word *) 0x7ffff7ffb3a8
(gdb) p *slot
$2 = 12
(gdb) p bit
$3 = 134217728
(gdb) p bits_per_word 
$4 = 64
(gdb) p bit_number 
$5 = 1711323


(gdb) backtrace 
#0  0x000000000060015e in dump_bitset__set_bit_value (bitset=0xce6690 <dump_private+80>, bit_number=1711323, 
    bit_is_set=true) at pdumper.c:4539
#1  0x00000000006001a6 in dump_bitset_set_bit (bitset=0xce6690 <dump_private+80>, bit_number=1711323)
    at pdumper.c:4547
#2  0x00000000006004e7 in pdumper_set_marked_impl (obj=0x7fffe539b6d8) at pdumper.c:4704
#3  0x0000000000568f28 in pdumper_set_marked (obj=0x7fffe539b6d8) at pdumper.h:222
#4  0x00000000005f1941 in set_symbol_marked (s=0x7fffe539b6d8) at alloc.c:4205
#5  0x00000000005f600a in mark_object (arg=XIL(0x7fffe46b41f8)) at alloc.c:6997
#6  0x00000000005f2d52 in mark_maybe_object (obj=XIL(0x7fffe46b41f8)) at alloc.c:5018
#7  0x00000000005f30ef in mark_memory (start=0x7ffffffdf580, end=0x7fffffffd870) at alloc.c:5210
#8  0x00000000005f3124 in mark_stack (bottom=0x7fffffffd870 "", end=0x7ffffffdf580 "\220\365\375\377\377\177")
    at alloc.c:5417
#9  0x00000000006aa9d7 in mark_one_thread (thread=0xd0b400 <main_thread>) at thread.c:617
#10 0x00000000006aaad8 in mark_threads_callback (ignore=0x0) at thread.c:650
#11 0x00000000005f317a in flush_stack_call_func (func=0x6aaa92 <mark_threads_callback>, arg=0x0) at alloc.c:5444
#12 0x00000000006aab04 in mark_threads () at thread.c:657
#13 0x00000000005f4dd4 in garbage_collect_1 (end=0x7ffffffdf780) at alloc.c:6322
#14 0x00000000005f54a6 in Fgarbage_collect () at alloc.c:6493
#15 0x0000000000566abd in maybe_gc () at lisp.h:5002
#16 0x0000000000623523 in eval_sub (form=XIL(0x7fffe47709e3)) at eval.c:2195
#17 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#18 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe47709a3), nargs=2, arg_vector=0x7ffffffdfa10) at eval.c:3098
#19 0x0000000000625389 in Ffuncall (nargs=3, args=0x7ffffffdfa08) at eval.c:2836
#20 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d5cc), vector=XIL(0x7fffe506d525), 
    maxdepth=make_number(8), args_template=make_number(514), nargs=2, args=0x7ffffffdff50) at bytecode.c:632
#21 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe506d4d5), nargs=2, arg_vector=0x7ffffffdff40) at eval.c:3023
#22 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffdff38) at eval.c:2824
#23 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7ffffffe0bc0) at bytecode.c:632
#24 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7ffffffe0bb8) at eval.c:3023
#25 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffe0bb0) at eval.c:2824
#26 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=2, args=0x7ffffffe10a8) at bytecode.c:632
#27 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=2, arg_vector=0x7ffffffe1098) at eval.c:3023
#28 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffe1090) at eval.c:2824
#29 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d5cc), vector=XIL(0x7fffe506d525), 
    maxdepth=make_number(8), args_template=make_number(514), nargs=2, args=0x7ffffffe15d0) at bytecode.c:632
#30 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe506d4d5), nargs=2, arg_vector=0x7ffffffe15c0) at eval.c:3023
#31 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffe15b8) at eval.c:2824
#32 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7ffffffe2240) at bytecode.c:632
#33 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7ffffffe2238) at eval.c:3023
#34 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffe2230) at eval.c:2824
#35 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=2, args=0x7ffffffe2728) at bytecode.c:632
#36 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=2, arg_vector=0x7ffffffe2718) at eval.c:3023
#37 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffe2710) at eval.c:2824
#38 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d5cc), vector=XIL(0x7fffe506d525), 
    maxdepth=make_number(8), args_template=make_number(514), nargs=2, args=0x7ffffffe2c50) at bytecode.c:632
#39 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe506d4d5), nargs=2, arg_vector=0x7ffffffe2c40) at eval.c:3023
#40 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffe2c38) at eval.c:2824
#41 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7ffffffe38c0) at bytecode.c:632
#42 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7ffffffe38b8) at eval.c:3023
#43 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffe38b0) at eval.c:2824
#44 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=2, args=0x7ffffffe3dc8) at bytecode.c:632
#45 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=2, arg_vector=0x7ffffffe3db8) at eval.c:3023
#46 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffe3db0) at eval.c:2824
#47 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4bbadd4), vector=XIL(0x7fffe4bbadbd), 
    maxdepth=make_number(10), args_template=make_number(513), nargs=1, args=0x7ffffffe42e8) at bytecode.c:632
#48 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4bbad7d), nargs=1, arg_vector=0x7ffffffe42e0) at eval.c:3023
#49 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffe42d8) at eval.c:2824
#50 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7ffffffe4f40) at bytecode.c:632
#51 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7ffffffe4f38) at eval.c:3023
#52 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffe4f30) at eval.c:2824
#53 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=1, args=0x7ffffffe5448) at bytecode.c:632
#54 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=1, arg_vector=0x7ffffffe5440) at eval.c:3023
#55 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffe5438) at eval.c:2824
#56 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d4b4), vector=XIL(0x7fffe506d48d), 
    maxdepth=make_number(10), args_template=make_number(1028), nargs=4, args=0x7ffffffe5938) at bytecode.c:632
#57 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe506d43d), nargs=4, arg_vector=0x7ffffffe5918) at eval.c:3023
#58 0x00000000006252a9 in Ffuncall (nargs=5, args=0x7ffffffe5910) at eval.c:2824
#59 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7ffffffe6580) at bytecode.c:632
#60 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7ffffffe6578) at eval.c:3023
#61 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffe6570) at eval.c:2824
#62 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=2, args=0x7ffffffe6a68) at bytecode.c:632
#63 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=2, arg_vector=0x7ffffffe6a58) at eval.c:3023
#64 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffe6a50) at eval.c:2824
#65 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d5cc), vector=XIL(0x7fffe506d525), 
    maxdepth=make_number(8), args_template=make_number(514), nargs=2, args=0x7ffffffe6f90) at bytecode.c:632
#66 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe506d4d5), nargs=2, arg_vector=0x7ffffffe6f80) at eval.c:3023
#67 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffe6f78) at eval.c:2824
#68 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7ffffffe7c00) at bytecode.c:632
#69 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7ffffffe7bf8) at eval.c:3023
#70 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffe7bf0) at eval.c:2824
#71 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=1, args=0x7ffffffe8108) at bytecode.c:632
#72 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=1, arg_vector=0x7ffffffe8100) at eval.c:3023
#73 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffe80f8) at eval.c:2824
#74 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d4b4), vector=XIL(0x7fffe506d48d), 
    maxdepth=make_number(10), args_template=make_number(1028), nargs=4, args=0x7ffffffe85f8) at bytecode.c:632
#75 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe506d43d), nargs=4, arg_vector=0x7ffffffe85d8) at eval.c:3023
#76 0x00000000006252a9 in Ffuncall (nargs=5, args=0x7ffffffe85d0) at eval.c:2824
#77 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7ffffffe9240) at bytecode.c:632
#78 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7ffffffe9238) at eval.c:3023
#79 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffe9230) at eval.c:2824
#80 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=2, args=0x7ffffffe9748) at bytecode.c:632
#81 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=2, arg_vector=0x7ffffffe9738) at eval.c:3023
#82 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffe9730) at eval.c:2824
#83 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4bbadd4), vector=XIL(0x7fffe4bbadbd), 
    maxdepth=make_number(10), args_template=make_number(513), nargs=1, args=0x7ffffffe9c68) at bytecode.c:632
#84 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4bbad7d), nargs=1, arg_vector=0x7ffffffe9c60) at eval.c:3023
#85 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffe9c58) at eval.c:2824
#86 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7ffffffea8c0) at bytecode.c:632
#87 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7ffffffea8b8) at eval.c:3023
#88 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffea8b0) at eval.c:2824
#89 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=1, args=0x7ffffffeadc8) at bytecode.c:632
#90 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=1, arg_vector=0x7ffffffeadc0) at eval.c:3023
#91 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffeadb8) at eval.c:2824
#92 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d4b4), vector=XIL(0x7fffe506d48d), 
    maxdepth=make_number(10), args_template=make_number(1028), nargs=4, args=0x7ffffffeb2b8) at bytecode.c:632
#93 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe506d43d), nargs=4, arg_vector=0x7ffffffeb298) at eval.c:3023
#94 0x00000000006252a9 in Ffuncall (nargs=5, args=0x7ffffffeb290) at eval.c:2824
#95 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7ffffffebf00) at bytecode.c:632
#96 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7ffffffebef8) at eval.c:3023
#97 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffebef0) at eval.c:2824
#98 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=2, args=0x7ffffffec3e8) at bytecode.c:632
#99 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=2, arg_vector=0x7ffffffec3d8) at eval.c:3023
#100 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffec3d0) at eval.c:2824
#101 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d5cc), vector=XIL(0x7fffe506d525), 
    maxdepth=make_number(8), args_template=make_number(514), nargs=2, args=0x7ffffffec910) at bytecode.c:632
#102 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe506d4d5), nargs=2, arg_vector=0x7ffffffec900)
    at eval.c:3023
#103 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffec8f8) at eval.c:2824
#104 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7ffffffed580) at bytecode.c:632
#105 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7ffffffed578)
    at eval.c:3023
#106 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffed570) at eval.c:2824
#107 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=2, args=0x7ffffffeda68) at bytecode.c:632
#108 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=2, arg_vector=0x7ffffffeda58)
    at eval.c:3023
#109 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffeda50) at eval.c:2824
#110 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d5cc), vector=XIL(0x7fffe506d525), 
    maxdepth=make_number(8), args_template=make_number(514), nargs=2, args=0x7ffffffedf90) at bytecode.c:632
#111 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe506d4d5), nargs=2, arg_vector=0x7ffffffedf80)
    at eval.c:3023
#112 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffedf78) at eval.c:2824
#113 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7ffffffeec00) at bytecode.c:632
#114 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7ffffffeebf8)
    at eval.c:3023
#115 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7ffffffeebf0) at eval.c:2824
#116 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=2, args=0x7ffffffef0e8) at bytecode.c:632
#117 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=2, arg_vector=0x7ffffffef0d8)
    at eval.c:3023
#118 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffef0d0) at eval.c:2824
#119 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d5cc), vector=XIL(0x7fffe506d525), 
    maxdepth=make_number(8), args_template=make_number(514), nargs=2, args=0x7ffffffef610) at bytecode.c:632
#120 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe506d4d5), nargs=2, arg_vector=0x7ffffffef600)
    at eval.c:3023
#121 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7ffffffef5f8) at eval.c:2824
#122 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7fffffff0280) at bytecode.c:632
#123 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7fffffff0278)
    at eval.c:3023
#124 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7fffffff0270) at eval.c:2824
#125 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=1, args=0x7fffffff0788) at bytecode.c:632
#126 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=1, arg_vector=0x7fffffff0780)
    at eval.c:3023
#127 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7fffffff0778) at eval.c:2824
#128 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d4b4), vector=XIL(0x7fffe506d48d), 
    maxdepth=make_number(10), args_template=make_number(1028), nargs=4, args=0x7fffffff0c78) at bytecode.c:632
#129 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe506d43d), nargs=4, arg_vector=0x7fffffff0c58)
    at eval.c:3023
#130 0x00000000006252a9 in Ffuncall (nargs=5, args=0x7fffffff0c50) at eval.c:2824
#131 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7fffffff18c0) at bytecode.c:632
#132 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7fffffff18b8)
    at eval.c:3023
#133 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7fffffff18b0) at eval.c:2824
#134 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=2, args=0x7fffffff1e10) at bytecode.c:632
#135 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=2, arg_vector=0x7fffffff1e00)
    at eval.c:3023
#136 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7fffffff1df8) at eval.c:2824
#137 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7fffffff2a40) at bytecode.c:632
#138 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7fffffff2a38)
    at eval.c:3023
#139 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7fffffff2a30) at eval.c:2824
#140 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4a8b2b4), vector=XIL(0x7fffe4a8b295), 
    maxdepth=make_number(9), args_template=make_number(513), nargs=2, args=0x7fffffff2f28) at bytecode.c:632
#141 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4a8b255), nargs=2, arg_vector=0x7fffffff2f18)
    at eval.c:3023
#142 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7fffffff2f10) at eval.c:2824
#143 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d5cc), vector=XIL(0x7fffe506d525), 
    maxdepth=make_number(8), args_template=make_number(514), nargs=2, args=0x7fffffff3450) at bytecode.c:632
#144 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe506d4d5), nargs=2, arg_vector=0x7fffffff3440)
    at eval.c:3023
#145 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7fffffff3438) at eval.c:2824
#146 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d734), vector=XIL(0x7fffe470a44d), 
    maxdepth=make_number(28), args_template=make_number(257), nargs=1, args=0x7fffffff4098) at bytecode.c:632
#147 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a40d), nargs=1, arg_vector=0x7fffffff4090)
    at eval.c:3023
#148 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7fffffff4088) at eval.c:2824
#149 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe506d774), vector=XIL(0x7fffe470a375), 
    maxdepth=make_number(4), args_template=make_number(513), nargs=1, args=0x7fffffff4518) at bytecode.c:632
#150 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe470a335), nargs=1, arg_vector=0x7fffffff4510)
    at eval.c:3023
#151 0x00000000006252a9 in Ffuncall (nargs=2, args=0x7fffffff4508) at eval.c:2824
#152 0x000000000066fd76 in exec_byte_code (bytestr=XIL(0x7fffe4d23a34), vector=XIL(0x7fffe4d22d85), 
    maxdepth=make_number(11), args_template=make_number(514), nargs=2, args=0x7fffffff4a58) at bytecode.c:632
#153 0x0000000000625d1f in funcall_lambda (fun=XIL(0x7fffe4d22d45), nargs=2, arg_vector=0x7fffffff4a48)
    at eval.c:3023
#154 0x00000000006252a9 in Ffuncall (nargs=3, args=0x7fffffff4a40) at eval.c:2824
#155 0x0000000000624b6e in call2 (fn=XIL(0x7fffe403b830), arg1=XIL(0x1231393), arg2=XIL(0xc480)) at eval.c:2681
#156 0x000000000065548a in readevalloop_eager_expand_eval (val=XIL(0x1231393), macroexpand=XIL(0x7fffe403b830))
    at lread.c:1885
#157 0x0000000000655e24 in readevalloop (readcharfun=XIL(0x12b63b5), infile0=0x0, sourcename=XIL(0x12b7304), 
    printflag=false, unibyte=XIL(0), readfun=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2071
#158 0x000000000065624e in Feval_buffer (buffer=XIL(0x12b63b5), printflag=XIL(0), filename=XIL(0x12b5b94), 
    unibyte=XIL(0), do_allow_print=XIL(0xc480)) at lread.c:2138
#159 0x0000000000623c97 in eval_sub (form=XIL(0x7fffe4746a73)) at eval.c:2301
#160 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#161 0x0000000000620eba in Flet (args=XIL(0x7fffe4746a43)) at eval.c:973
#162 0x00000000006237be in eval_sub (form=XIL(0x7fffe4746a33)) at eval.c:2239
#163 0x00000000006213b2 in Funwind_protect (args=XIL(0x7fffe47469a3)) at eval.c:1195
#164 0x00000000006237be in eval_sub (form=XIL(0x7fffe4746993)) at eval.c:2239
#165 0x000000000061fc47 in Fprogn (body=XIL(0x7fffe4746703)) at eval.c:459
#166 0x0000000000620abf in FletX (args=XIL(0x7fffe47466c3)) at eval.c:904
#167 0x00000000006237be in eval_sub (form=XIL(0x7fffe47466b3)) at eval.c:2239
#168 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#169 0x000000000061fb41 in Fif (args=XIL(0x7fffe4746683)) at eval.c:415
#170 0x00000000006237be in eval_sub (form=XIL(0x7fffe4746673)) at eval.c:2239
#171 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#172 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe4746633), nargs=4, arg_vector=0x7fffffff5508)
    at eval.c:3098
#173 0x0000000000625389 in Ffuncall (nargs=5, args=0x7fffffff5500) at eval.c:2836
#174 0x0000000000624c56 in call4 (fn=XIL(0x7fffe3a5f120), arg1=XIL(0x12b5b94), arg2=XIL(0x12b5b94), arg3=XIL(0), 
    arg4=XIL(0xc480)) at eval.c:2698
#175 0x0000000000654040 in Fload (file=XIL(0x7fffe480b71c), noerror=XIL(0), nomessage=XIL(0xc480), nosuffix=XIL(0), 
    must_suffix=XIL(0xc480)) at lread.c:1367
#176 0x0000000000631ed7 in Frequire (feature=XIL(0x7fffe3b24178), filename=XIL(0), noerror=XIL(0)) at fns.c:2807
#177 0x0000000000623c31 in eval_sub (form=XIL(0x122e753)) at eval.c:2293
#178 0x0000000000655492 in readevalloop_eager_expand_eval (val=XIL(0x122e753), macroexpand=XIL(0x7fffe403b830))
    at lread.c:1885
#179 0x0000000000655e24 in readevalloop (readcharfun=XIL(0x115d015), infile0=0x0, sourcename=XIL(0x1116c34), 
    printflag=false, unibyte=XIL(0), readfun=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2071
#180 0x000000000065624e in Feval_buffer (buffer=XIL(0x115d015), printflag=XIL(0), filename=XIL(0x112fd64), 
    unibyte=XIL(0), do_allow_print=XIL(0xc480)) at lread.c:2138
#181 0x0000000000623c97 in eval_sub (form=XIL(0x7fffe4746a73)) at eval.c:2301
#182 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#183 0x0000000000620eba in Flet (args=XIL(0x7fffe4746a43)) at eval.c:973
#184 0x00000000006237be in eval_sub (form=XIL(0x7fffe4746a33)) at eval.c:2239
#185 0x00000000006213b2 in Funwind_protect (args=XIL(0x7fffe47469a3)) at eval.c:1195
#186 0x00000000006237be in eval_sub (form=XIL(0x7fffe4746993)) at eval.c:2239
#187 0x000000000061fc47 in Fprogn (body=XIL(0x7fffe4746703)) at eval.c:459
#188 0x0000000000620abf in FletX (args=XIL(0x7fffe47466c3)) at eval.c:904
#189 0x00000000006237be in eval_sub (form=XIL(0x7fffe47466b3)) at eval.c:2239
#190 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#191 0x000000000061fb41 in Fif (args=XIL(0x7fffe4746683)) at eval.c:415
#192 0x00000000006237be in eval_sub (form=XIL(0x7fffe4746673)) at eval.c:2239
#193 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#194 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe4746633), nargs=4, arg_vector=0x7fffffff6398)
    at eval.c:3098
#195 0x0000000000625389 in Ffuncall (nargs=5, args=0x7fffffff6390) at eval.c:2836
#196 0x0000000000624c56 in call4 (fn=XIL(0x7fffe3a5f120), arg1=XIL(0x112fd64), arg2=XIL(0x112fd64), arg3=XIL(0), 
    arg4=XIL(0xc480)) at eval.c:2698
#197 0x0000000000654040 in Fload (file=XIL(0x7fffe48e1324), noerror=XIL(0), nomessage=XIL(0xc480), nosuffix=XIL(0), 
    must_suffix=XIL(0)) at lread.c:1367
#198 0x0000000000623c97 in eval_sub (form=XIL(0x7fffe48e12e3)) at eval.c:2301
#199 0x000000000061fc47 in Fprogn (body=XIL(0x7fffe48e12b3)) at eval.c:459
#200 0x00000000006237be in eval_sub (form=XIL(0x7fffe48e1293)) at eval.c:2239
#201 0x000000000061fb23 in Fif (args=XIL(0x7fffe48e1273)) at eval.c:414
#202 0x00000000006237be in eval_sub (form=XIL(0x7fffe48e1263)) at eval.c:2239
#203 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#204 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe48e1223), nargs=1, arg_vector=0x7fffffff6b68)
    at eval.c:3098
#205 0x0000000000625389 in Ffuncall (nargs=2, args=0x7fffffff6b60) at eval.c:2836
#206 0x0000000000624173 in Fapply (nargs=2, args=0x7fffffff6b60) at eval.c:2399
#207 0x00000000006239f5 in eval_sub (form=XIL(0x7fffe48817b3)) at eval.c:2258
#208 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#209 0x000000000061fbcc in Fcond (args=XIL(0x7fffe4881783)) at eval.c:439
#210 0x00000000006237be in eval_sub (form=XIL(0x7fffe4881753)) at eval.c:2239
#211 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#212 0x0000000000620eba in Flet (args=XIL(0x7fffe4881723)) at eval.c:973
#213 0x00000000006237be in eval_sub (form=XIL(0x7fffe4881713)) at eval.c:2239
#214 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#215 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe48816d3), nargs=3, arg_vector=0x7fffffff7110)
    at eval.c:3098
#216 0x0000000000625a79 in apply_lambda (fun=XIL(0x7fffe48816d3), args=XIL(0x7fffe4e776bb), count=89) at eval.c:2959
#217 0x0000000000623fd5 in eval_sub (form=XIL(0x7fffe4e776ab)) at eval.c:2362
#218 0x000000000061fa9f in Fand (args=XIL(0x7fffe4e775db)) at eval.c:393
#219 0x00000000006237be in eval_sub (form=XIL(0x7fffe4e775bb)) at eval.c:2239
#220 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#221 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe4e7758b), nargs=1, arg_vector=0x7fffffff7578)
    at eval.c:3098
#222 0x0000000000625389 in Ffuncall (nargs=2, args=0x7fffffff7570) at eval.c:2836
#223 0x0000000000624b0c in call1 (fn=XIL(0x7fffe4e7758b), arg1=XIL(0x7fffe3c13f68)) at eval.c:2673
#224 0x0000000000630e6c in mapcar1 (leni=9, vals=0x0, fn=XIL(0x7fffe4e7758b), seq=XIL(0x7fffe48faebb)) at fns.c:2486
#225 0x0000000000631359 in Fmapc (function=XIL(0x7fffe4e7758b), sequence=XIL(0x7fffe48faebb)) at fns.c:2555
#226 0x0000000000623c01 in eval_sub (form=XIL(0x7fffe4e7742b)) at eval.c:2290
#227 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#228 0x0000000000620eba in Flet (args=XIL(0x7fffe4e7740b)) at eval.c:973
#229 0x00000000006237be in eval_sub (form=XIL(0x7fffe4e773fb)) at eval.c:2239
#230 0x000000000061fc47 in Fprogn (body=XIL(0x7fffe4e7735b)) at eval.c:459
#231 0x000000000062117e in internal_catch (tag=XIL(0x7fffe3a39ed8), func=0x61fc00 <Fprogn>, arg=XIL(0x7fffe4e7734b))
    at eval.c:1101
#232 0x0000000000621132 in Fcatch (args=XIL(0x7fffe4e7733b)) at eval.c:1078
#233 0x00000000006237be in eval_sub (form=XIL(0x7fffe4e7732b)) at eval.c:2239
#234 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#235 0x000000000061fbcc in Fcond (args=XIL(0x7fffe4e772fb)) at eval.c:439
#236 0x00000000006237be in eval_sub (form=XIL(0x7fffe4e772cb)) at eval.c:2239
#237 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#238 0x0000000000620eba in Flet (args=XIL(0x7fffe4e772ab)) at eval.c:973
#239 0x00000000006237be in eval_sub (form=XIL(0x7fffe4e7729b)) at eval.c:2239
#240 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#241 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe4e7725b), nargs=1, arg_vector=0x7fffffff7fc0)
    at eval.c:3098
#242 0x0000000000625a79 in apply_lambda (fun=XIL(0x7fffe4e7725b), args=XIL(0x7fffe4fe5c53), count=77) at eval.c:2959
#243 0x0000000000623fd5 in eval_sub (form=XIL(0x7fffe4fe5c43)) at eval.c:2362
#244 0x000000000061faed in Fif (args=XIL(0x7fffe4fe5bc3)) at eval.c:411
#245 0x00000000006237be in eval_sub (form=XIL(0x7fffe4fe5bb3)) at eval.c:2239
#246 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#247 0x000000000061fbcc in Fcond (args=XIL(0x7fffe4fe5b83)) at eval.c:439
#248 0x00000000006237be in eval_sub (form=XIL(0x7fffe4fe5b53)) at eval.c:2239
#249 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#250 0x0000000000620eba in Flet (args=XIL(0x7fffe4fe5b33)) at eval.c:973
#251 0x00000000006237be in eval_sub (form=XIL(0x7fffe4fe5b23)) at eval.c:2239
#252 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#253 0x000000000061fbcc in Fcond (args=XIL(0x7fffe4882ec3)) at eval.c:439
#254 0x00000000006237be in eval_sub (form=XIL(0x7fffe4882eb3)) at eval.c:2239
#255 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#256 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe4882e73), nargs=1, arg_vector=0x7fffffff88e0)
    at eval.c:3098
#257 0x0000000000625a79 in apply_lambda (fun=XIL(0x7fffe4882e73), args=XIL(0x7fffe5039a0b), count=70) at eval.c:2959
#258 0x0000000000623fd5 in eval_sub (form=XIL(0x7fffe50399fb)) at eval.c:2362
#259 0x000000000062183c in internal_lisp_condition_case (var=XIL(0x7fffe43524c8), bodyform=XIL(0x7fffe50399fb), 
    handlers=XIL(0x7fffe503993b)) at eval.c:1307
#260 0x000000000062142c in Fcondition_case (args=XIL(0x7fffe503991b)) at eval.c:1231
#261 0x00000000006237be in eval_sub (form=XIL(0x7fffe503990b)) at eval.c:2239
#262 0x000000000061fda4 in Fsetq (args=XIL(0x7fffe50398eb)) at eval.c:517
#263 0x00000000006237be in eval_sub (form=XIL(0x7fffe50398db)) at eval.c:2239
#264 0x000000000061fb80 in Fcond (args=XIL(0x7fffe477493b)) at eval.c:435
#265 0x00000000006237be in eval_sub (form=XIL(0x7fffe477492b)) at eval.c:2239
#266 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#267 0x0000000000620eba in Flet (args=XIL(0x7fffe477490b)) at eval.c:973
#268 0x00000000006237be in eval_sub (form=XIL(0x7fffe47748fb)) at eval.c:2239
#269 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#270 0x00000000006237be in eval_sub (form=XIL(0x7fffe47748bb)) at eval.c:2239
#271 0x000000000061fb23 in Fif (args=XIL(0x7fffe477489b)) at eval.c:414
#272 0x00000000006237be in eval_sub (form=XIL(0x7fffe477488b)) at eval.c:2239
#273 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#274 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe477482b), nargs=0, arg_vector=0x7fffffff9670)
    at eval.c:3098
#275 0x0000000000625389 in Ffuncall (nargs=1, args=0x7fffffff9668) at eval.c:2836
#276 0x0000000000624541 in funcall_nil (nargs=1, args=0x7fffffff9668) at eval.c:2453
#277 0x0000000000624963 in run_hook_with_args (nargs=1, args=0x7fffffff9668, funcall=0x62451e <funcall_nil>)
    at eval.c:2630
#278 0x00000000006245c8 in Frun_hook_with_args (nargs=1, args=0x7fffffff9668) at eval.c:2495
#279 0x00000000006249c8 in run_hook (hook=XIL(0x7fffe3a8d318)) at eval.c:2643
#280 0x0000000000624585 in Frun_hooks (nargs=1, args=0x7fffffff96b0) at eval.c:2477
#281 0x00000000006239f5 in eval_sub (form=XIL(0x7fffe48519cb)) at eval.c:2258
#282 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#283 0x000000000061fb41 in Fif (args=XIL(0x10f1803)) at eval.c:415
#284 0x00000000006237be in eval_sub (form=XIL(0x10f13d3)) at eval.c:2239
#285 0x0000000000623f7a in eval_sub (form=XIL(0x7fffe485195b)) at eval.c:2358
#286 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#287 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe48518db), nargs=2, arg_vector=0x7fffffff9b50)
    at eval.c:3098
#288 0x0000000000625a79 in apply_lambda (fun=XIL(0x7fffe48518cb), args=XIL(0x7fffe485185b), count=56) at eval.c:2959
#289 0x0000000000623fd5 in eval_sub (form=XIL(0x7fffe485184b)) at eval.c:2362
#290 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#291 0x000000000061fb41 in Fif (args=XIL(0x7fffe485181b)) at eval.c:415
#292 0x00000000006237be in eval_sub (form=XIL(0x7fffe485180b)) at eval.c:2239
#293 0x000000000061fc47 in Fprogn (body=XIL(0x104ed43)) at eval.c:459
#294 0x000000000060ee42 in Fsave_current_buffer (args=XIL(0x10700a3)) at editfns.c:1065
#295 0x00000000006237be in eval_sub (form=XIL(0x1070093)) at eval.c:2239
#296 0x0000000000623f7a in eval_sub (form=XIL(0x7fffe485170b)) at eval.c:2358
#297 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#298 0x0000000000620eba in Flet (args=XIL(0x7fffe48516eb)) at eval.c:973
#299 0x00000000006237be in eval_sub (form=XIL(0x7fffe48516db)) at eval.c:2239
#300 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#301 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe48516ab), nargs=6, arg_vector=0x7fffffffa420)
    at eval.c:3098
#302 0x0000000000625a79 in apply_lambda (fun=XIL(0x7fffe485169b), args=XIL(0x7fffe485156b), count=48) at eval.c:2959
#303 0x0000000000623fd5 in eval_sub (form=XIL(0x7fffe485155b)) at eval.c:2362
#304 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#305 0x000000000061fb41 in Fif (args=XIL(0x7fffe485151b)) at eval.c:415
#306 0x00000000006237be in eval_sub (form=XIL(0x7fffe485150b)) at eval.c:2239
#307 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#308 0x0000000000620abf in FletX (args=XIL(0x7fffe48514cb)) at eval.c:904
#309 0x00000000006237be in eval_sub (form=XIL(0x7fffe48514bb)) at eval.c:2239
#310 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#311 0x000000000061fb41 in Fif (args=XIL(0x7fffe485148b)) at eval.c:415
#312 0x00000000006237be in eval_sub (form=XIL(0x7fffe485147b)) at eval.c:2239
#313 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#314 0x000000000061fb41 in Fif (args=XIL(0x7fffe485144b)) at eval.c:415
#315 0x00000000006237be in eval_sub (form=XIL(0x7fffe485143b)) at eval.c:2239
#316 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#317 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe48513eb), nargs=1, arg_vector=0x7fffffffad10)
    at eval.c:3098
#318 0x0000000000625a79 in apply_lambda (fun=XIL(0x7fffe48513db), args=XIL(0x10650e3), count=41) at eval.c:2959
#319 0x0000000000623fd5 in eval_sub (form=XIL(0x1065403)) at eval.c:2362
#320 0x0000000000623afa in eval_sub (form=XIL(0x1069183)) at eval.c:2275
#321 0x000000000061fc47 in Fprogn (body=XIL(0x1069b23)) at eval.c:459
#322 0x000000000060ee42 in Fsave_current_buffer (args=XIL(0x1069173)) at editfns.c:1065
#323 0x00000000006237be in eval_sub (form=XIL(0x1069163)) at eval.c:2239
#324 0x000000000062183c in internal_lisp_condition_case (var=XIL(0x7fffe39d9288), bodyform=XIL(0x1069163), 
    handlers=XIL(0x1064683)) at eval.c:1307
#325 0x000000000062142c in Fcondition_case (args=XIL(0x1068a83)) at eval.c:1231
#326 0x00000000006237be in eval_sub (form=XIL(0x1068a73)) at eval.c:2239
#327 0x0000000000620d08 in Flet (args=XIL(0x1058b13)) at eval.c:943
#328 0x00000000006237be in eval_sub (form=XIL(0x1058b03)) at eval.c:2239
#329 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#330 0x0000000000626016 in funcall_lambda (fun=XIL(0x1058af3), nargs=2, arg_vector=0x7fffffffb690) at eval.c:3098
#331 0x0000000000625a79 in apply_lambda (fun=XIL(0x1058af3), args=XIL(0x1065b33), count=33) at eval.c:2959
#332 0x0000000000623fd5 in eval_sub (form=XIL(0x1065b43)) at eval.c:2362
#333 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#334 0x0000000000626016 in funcall_lambda (fun=XIL(0x104a2c3), nargs=0, arg_vector=0x7fffffffb988) at eval.c:3098
#335 0x0000000000625389 in Ffuncall (nargs=1, args=0x7fffffffb980) at eval.c:2836
#336 0x00000000006239f5 in eval_sub (form=XIL(0x7fffe4be0e9b)) at eval.c:2258
#337 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#338 0x000000000061fb41 in Fif (args=XIL(0x7fffe4be0e6b)) at eval.c:415
#339 0x00000000006237be in eval_sub (form=XIL(0x7fffe4be0e5b)) at eval.c:2239
#340 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#341 0x000000000061fbcc in Fcond (args=XIL(0x7fffe4bde66b)) at eval.c:439
#342 0x00000000006237be in eval_sub (form=XIL(0x7fffe4bde63b)) at eval.c:2239
#343 0x000000000061fc47 in Fprogn (body=XIL(0x7fffe4bde59b)) at eval.c:459
#344 0x0000000000620abf in FletX (args=XIL(0x7fffe4bde55b)) at eval.c:904
#345 0x00000000006237be in eval_sub (form=XIL(0x7fffe4bde54b)) at eval.c:2239
#346 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#347 0x000000000061fc75 in prog_ignore (body=XIL(0x7fffe4bde53b)) at eval.c:470
#348 0x0000000000620f56 in Fwhile (args=XIL(0x7fffe4bde52b)) at eval.c:992
#349 0x00000000006237be in eval_sub (form=XIL(0x7fffe4bde51b)) at eval.c:2239
#350 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#351 0x0000000000620abf in FletX (args=XIL(0x7fffe4bde4db)) at eval.c:904
#352 0x00000000006237be in eval_sub (form=XIL(0x7fffe4bde4cb)) at eval.c:2239
#353 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#354 0x00000000006237be in eval_sub (form=XIL(0x7fffe4bde4ab)) at eval.c:2239
#355 0x000000000061fb23 in Fif (args=XIL(0x7fffe4bde48b)) at eval.c:414
#356 0x00000000006237be in eval_sub (form=XIL(0x7fffe4bde47b)) at eval.c:2239
#357 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#358 0x0000000000620eba in Flet (args=XIL(0x7fffe4bde45b)) at eval.c:973
#359 0x00000000006237be in eval_sub (form=XIL(0x7fffe4bde44b)) at eval.c:2239
#360 0x000000000061fc47 in Fprogn (body=XIL(0x7fffe4bc59f3)) at eval.c:459
#361 0x0000000000620eba in Flet (args=XIL(0x7fffe4bc59d3)) at eval.c:973
#362 0x00000000006237be in eval_sub (form=XIL(0x7fffe4bc59c3)) at eval.c:2239
#363 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#364 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe4bc5963), nargs=1, arg_vector=0x7fffffffcaa0)
    at eval.c:3098
#365 0x0000000000625a79 in apply_lambda (fun=XIL(0x7fffe4bc5953), args=XIL(0x7fffe4bc58f3), count=13) at eval.c:2959
#366 0x0000000000623fd5 in eval_sub (form=XIL(0x7fffe4bc58e3)) at eval.c:2362
#367 0x000000000061fc47 in Fprogn (body=XIL(0x7fffe4bc4223)) at eval.c:459
#368 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe4bc4083), nargs=0, arg_vector=0x7fffffffcd30)
    at eval.c:3098
#369 0x0000000000625a79 in apply_lambda (fun=XIL(0x7fffe4bc4073), args=XIL(0), count=11) at eval.c:2959
#370 0x0000000000623fd5 in eval_sub (form=XIL(0x7fffe4bc4033)) at eval.c:2362
#371 0x00000000006213b2 in Funwind_protect (args=XIL(0x7fffe4bc1073)) at eval.c:1195
#372 0x00000000006237be in eval_sub (form=XIL(0x7fffe4bc1063)) at eval.c:2239
#373 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#374 0x0000000000620eba in Flet (args=XIL(0x7fffe4bc1043)) at eval.c:973
#375 0x00000000006237be in eval_sub (form=XIL(0x7fffe4bc1033)) at eval.c:2239
#376 0x000000000061fc47 in Fprogn (body=XIL(0x7fffe4bc0a23)) at eval.c:459
#377 0x000000000061fb41 in Fif (args=XIL(0x7fffe4bc0953)) at eval.c:415
#378 0x00000000006237be in eval_sub (form=XIL(0x7fffe4bc0943)) at eval.c:2239
#379 0x000000000061fc47 in Fprogn (body=XIL(0)) at eval.c:459
#380 0x0000000000626016 in funcall_lambda (fun=XIL(0x7fffe4bc0903), nargs=0, arg_vector=0x7fffffffd4a0)
    at eval.c:3098
#381 0x0000000000625a79 in apply_lambda (fun=XIL(0x7fffe4bc08f3), args=XIL(0), count=4) at eval.c:2959
#382 0x0000000000623fd5 in eval_sub (form=XIL(0x7fffe526d9c3)) at eval.c:2362
#383 0x00000000006231f9 in Feval (form=XIL(0x7fffe526d9c3), lexical=XIL(0)) at eval.c:2107
#384 0x000000000056de9f in top_level_2 () at keyboard.c:1122
#385 0x00000000006218ec in internal_condition_case (bfun=0x56de7c <top_level_2>, handlers=XIL(0x53d0), 
    hfun=0x56d90d <cmd_error>) at eval.c:1336
#386 0x000000000056dee3 in top_level_1 (ignore=XIL(0)) at keyboard.c:1130
#387 0x000000000062117e in internal_catch (tag=XIL(0xcae0), func=0x56dea1 <top_level_1>, arg=XIL(0)) at eval.c:1101
#388 0x000000000056ddce in command_loop () at keyboard.c:1091
#389 0x000000000056d4e2 in recursive_edit_1 () at keyboard.c:698
#390 0x000000000056d664 in Frecursive_edit () at keyboard.c:769
#391 0x000000000056b500 in main (argc=11, argv=0x7fffffffda28) at emacs.c:1907

Lisp Backtrace:
"Automatic GC" (0x0)
"function-get" (0xfffdfa10)
0xe506d4d0 PVEC_COMPILED
"macroexp--expand-all" (0xfffe0bb8)
"macroexp--all-forms" (0xfffe1098)
0xe506d4d0 PVEC_COMPILED
"macroexp--expand-all" (0xfffe2238)
"macroexp--all-forms" (0xfffe2718)
0xe506d4d0 PVEC_COMPILED
"macroexp--expand-all" (0xfffe38b8)
"macroexp--all-forms" (0xfffe3db8)
"macroexp--all-clauses" (0xfffe42e0)
"macroexp--expand-all" (0xfffe4f38)
"macroexp--all-forms" (0xfffe5440)
0xe506d438 PVEC_COMPILED
"macroexp--expand-all" (0xfffe6578)
"macroexp--all-forms" (0xfffe6a58)
0xe506d4d0 PVEC_COMPILED
"macroexp--expand-all" (0xfffe7bf8)
"macroexp--all-forms" (0xfffe8100)
0xe506d438 PVEC_COMPILED
"macroexp--expand-all" (0xfffe9238)
"macroexp--all-forms" (0xfffe9738)
"macroexp--all-clauses" (0xfffe9c60)
"macroexp--expand-all" (0xfffea8b8)
"macroexp--all-forms" (0xfffeadc0)
0xe506d438 PVEC_COMPILED
"macroexp--expand-all" (0xfffebef8)
"macroexp--all-forms" (0xfffec3d8)
0xe506d4d0 PVEC_COMPILED
"macroexp--expand-all" (0xfffed578)
"macroexp--all-forms" (0xfffeda58)
0xe506d4d0 PVEC_COMPILED
"macroexp--expand-all" (0xfffeebf8)
"macroexp--all-forms" (0xfffef0d8)
0xe506d4d0 PVEC_COMPILED
"macroexp--expand-all" (0xffff0278)
"macroexp--all-forms" (0xffff0780)
0xe506d438 PVEC_COMPILED
"macroexp--expand-all" (0xffff18b8)
"macroexp--all-forms" (0xffff1e00)
"macroexp--expand-all" (0xffff2a38)
"macroexp--all-forms" (0xffff2f18)
0xe506d4d0 PVEC_COMPILED
"macroexp--expand-all" (0xffff4090)
"macroexpand-all" (0xffff4510)
"internal-macroexpand-for-load" (0xffff4a48)
"eval-buffer" (0xffff4c80)
"let" (0xffff4e20)
"unwind-protect" (0xffff4f70)
"let*" (0xffff5130)
"if" (0xffff52b0)
"load-with-code-conversion" (0xffff5508)
"require" (0xffff5890)
"eval-buffer" (0xffff5b10)
"let" (0xffff5cb0)
"unwind-protect" (0xffff5e00)
"let*" (0xffff5fc0)
"if" (0xffff6140)
"load-with-code-conversion" (0xffff6398)
"load" (0xffff66a0)
"progn" (0xffff6760)
"if" (0xffff68b0)
"vc-git-registered" (0xffff6b68)
"apply" (0xffff6b60)
"cond" (0xffff6d30)
"let" (0xffff6f50)
"vc-call-backend" (0xffff7110)
"and" (0xffff7340)
0xe4e77588 Lisp type 3
"mapc" (0xffff76f0)
"let" (0xffff7880)
"catch" (0xffff7a50)
"cond" (0xffff7be0)
"let" (0xffff7e00)
"vc-registered" (0xffff7fc0)
"if" (0xffff81e0)
"cond" (0xffff8370)
"let" (0xffff8590)
"cond" (0xffff8720)
"vc-backend" (0xffff88e0)
"condition-case" (0xffff8c00)
"setq" (0xffff8d80)
"cond" (0xffff8ee0)
"let" (0xffff9100)
"progn" (0xffff9250)
"if" (0xffff93a0)
"vc-refresh-state" (0xffff9670)
"run-hooks" (0xffff96b0)
"if" (0xffff9870)
"unless" (0xffff9990)
"after-find-file" (0xffff9b50)
"if" (0xffff9da0)
"save-current-buffer" (0xffff9f20)
"with-current-buffer" (0xffffa040)
"let" (0xffffa260)
"find-file-noselect-1" (0xffffa420)
"if" (0xffffa690)
"let*" (0xffffa850)
"if" (0xffffa9d0)
"if" (0xffffab50)
"find-file-noselect" (0xffffad10)
"set-buffer" (0xffffaf00)
"save-current-buffer" (0xffffb080)
"condition-case" (0xffffb2d0)
"let" (0xffffb4d0)
"bovine--make-parser-1" (0xffffb690)
"bovine-batch-make-parser" (0xffffb988)
"funcall" (0xffffb980)
"if" (0xffffbb40)
"cond" (0xffffbcd0)
"let*" (0xffffbe90)
"while" (0xffffc040)
"let*" (0xffffc200)
"progn" (0xffffc350)
"if" (0xffffc4a0)
"let" (0xffffc6c0)
"let" (0xffffc8e0)
"command-line-1" (0xffffcaa0)
"command-line" (0xffffcd30)
"unwind-protect" (0xffffcf40)
"let" (0xffffd160)
"if" (0xffffd2e0)
"normal-top-level" (0xffffd4a0)

Re: Preview: portable dumper

[Daniel Colascione]

On 02/14/2018 02:30 AM, Robert Pluim wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> Date: Tue, 13 Feb 2018 23:06:15 +0100 (CET)
>>> From: Angelo Graziosi <angelo.g0@libero.it>
>>>
>>> Eli Zaretskii wrote:
>>>>
>>>> I'd urge people to try this branch and report any issues they see.
>>>
>>> What should we look for?
>>
>> The usual stuff, I think: does it build, does it work as expected,
>> etc.
> 
> You mean like:
> 
> make[3]: Entering directory '/home/rpluim/repos/emacs-pdumper/admin/grammars'
>    GEN      ../../lisp/cedet/semantic/bovine/c-by.el
> load_dump completed in 63.0991 milliseconds
> ../../lisp/emacs-lisp/eieio.el: ‘object-print’ is an obsolete generic function (as of 26.1); use ‘cl-print-object’ instead.
> ../../lisp/emacs-lisp/eieio-base.el: ‘eieio-object-set-name-string’ is an obsolete generic function (as of 25.1); inherit from ‘eieio-named’ and use (setf (slot-value OBJ ’object-name) NAME) instead
> ../../lisp/cedet/semantic/db.el: ‘object-print’ is an obsolete generic function (as of 26.1); use ‘cl-print-object’ instead.
> ../../lisp/cedet/semantic/db.el: ‘object-print’ is an obsolete generic function (as of 26.1); use ‘cl-print-object’ instead.
> ../../lisp/cedet/semantic/db.el: ‘object-print’ is an obsolete generic function (as of 26.1); use ‘cl-print-object’ instead.
> ../../lisp/cedet/semantic/db-ref.el: Obsolete name arg "DEBUG" to constructor semanticdb-ref-adebug
> Fatal error 11: Segmentation fault

Thanks. Did you try it with --enable-checking?

Re: Preview: portable dumper

[Robert Pluim]

Daniel Colascione <dancol@dancol.org> writes:

> On 02/14/2018 02:30 AM, Robert Pluim wrote:
>>
>> make[3]: Entering directory '/home/rpluim/repos/emacs-pdumper/admin/grammars'
>>    GEN      ../../lisp/cedet/semantic/bovine/c-by.el
>> load_dump completed in 63.0991 milliseconds
>> ../../lisp/emacs-lisp/eieio.el: ‘object-print’ is an obsolete generic function (as of 26.1); use ‘cl-print-object’ instead.
>> ../../lisp/emacs-lisp/eieio-base.el: ‘eieio-object-set-name-string’
>> is an obsolete generic function (as of 25.1); inherit from
>> ‘eieio-named’ and use (setf (slot-value OBJ ’object-name) NAME)
>> instead
>> ../../lisp/cedet/semantic/db.el: ‘object-print’ is an obsolete generic function (as of 26.1); use ‘cl-print-object’ instead.
>> ../../lisp/cedet/semantic/db.el: ‘object-print’ is an obsolete generic function (as of 26.1); use ‘cl-print-object’ instead.
>> ../../lisp/cedet/semantic/db.el: ‘object-print’ is an obsolete generic function (as of 26.1); use ‘cl-print-object’ instead.
>> ../../lisp/cedet/semantic/db-ref.el: Obsolete name arg "DEBUG" to constructor semanticdb-ref-adebug
>> Fatal error 11: Segmentation fault
>
> Thanks. Did you try it with --enable-checking?

That gives me a different error:

Converting /home/rpluim/repos/emacs-pdumper/leim/CXTERM-DIC/ZOZY.tit to quail-package...
Decoding with coding system cn-big5...
Processing header part...
Formatting translation rules...
  GEN      ../lisp/leim/quail/tsang-b5.el
load_dump completed in 39.7511 milliseconds
Converting cangjie-table.b5 to tsang-b5.el...

pdumper.c:4691: Emacs fatal error: assertion failed: offset < dump_private.header.discardable_start
Fatal error 6: Aborted

Robert

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Wed, 14 Feb 2018 11:30:07 +0100
> 
> >> What should we look for?
> >
> > The usual stuff, I think: does it build, does it work as expected,
> > etc.
> 
> You mean like:

Yes, like that.

I see something slightly different on MS-Windows (in a build with
"--enable-checking"), but maybe similar enough to be explained by the
same problem:

       ELC      ../lisp/composite.elc
     load_dump completed in 46.005 milliseconds

     insdel.c:1937: Emacs fatal error: assertion failed: !pdumper_object_p (BEG_ADDR)

Backtrace:

  #0  0x762c3227 in KERNELBASE!DebugBreak ()
     from C:\Windows\syswow64\KernelBase.dll
  #1  0x0131940d in emacs_abort () at w32fns.c:10874
  #2  0x01152dea in terminate_due_to_signal (sig=22, backtrace_limit=2147483647)
      at emacs.c:388
  #3  0x01206274 in die (
      msg=0x16ce7d2 <DEFAULT_REHASH_SIZE+354> "!pdumper_object_p (BEG_ADDR)",
      file=0x16ce674 <DEFAULT_REHASH_SIZE+4> "insdel.c", line=1937)
      at alloc.c:7789
  #4  0x011acb8b in prepare_to_modify_buffer_1 (start=1, end=1,
      preserve_ptr=0x0) at insdel.c:1937
  #5  0x011acfa0 in prepare_to_modify_buffer (start=1, end=1, preserve_ptr=0x0)
      at insdel.c:1995
  #6  0x011c4b26 in Finsert_file_contents (filename=XIL(0x8000000008f797d0),
      visit=XIL(0), beg=XIL(0), end=XIL(0), replace=XIL(0)) at fileio.c:4142
  #7  0x0124343e in eval_sub (form=XIL(0xc000000007e47628)) at eval.c:2301
  #8  0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #9  0x0123e1cd in Flet (args=XIL(0xc000000007e47608)) at eval.c:973
  #10 0x01242bf4 in eval_sub (form=XIL(0xc000000007e475f8)) at eval.c:2239
  #11 0x0123b034 in Fprogn (body=XIL(0xc000000008f57860)) at eval.c:459
  #12 0x01226b99 in Fsave_current_buffer (args=XIL(0xc000000008f57010))
      at editfns.c:1065
  #13 0x01242bf4 in eval_sub (form=XIL(0xc000000008f57020)) at eval.c:2239
  #14 0x01243ad7 in eval_sub (form=XIL(0xc000000007e47468)) at eval.c:2358
  #15 0x0123b034 in Fprogn (body=XIL(0xc000000007e47368)) at eval.c:459
  #16 0x0123e1cd in Flet (args=XIL(0xc000000007e47348)) at eval.c:973
  #17 0x01242bf4 in eval_sub (form=XIL(0xc000000007e47338)) at eval.c:2239
  #18 0x0123eca4 in Funwind_protect (args=XIL(0xc000000007e472a8))
      at eval.c:1195
  #19 0x01242bf4 in eval_sub (form=XIL(0xc000000007e47298)) at eval.c:2239
  #20 0x0123b034 in Fprogn (body=XIL(0xc000000007e47090)) at eval.c:459
  #21 0x0123d8e6 in FletX (args=XIL(0xc000000007e47050)) at eval.c:904
  #22 0x01242bf4 in eval_sub (form=XIL(0xc000000007e47040)) at eval.c:2239
  #23 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #24 0x0123accd in Fif (args=XIL(0xc000000007e47010)) at eval.c:415
  #25 0x01242bf4 in eval_sub (form=XIL(0xc000000007e47000)) at eval.c:2239
  #26 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #27 0x01246f3a in funcall_lambda (fun=XIL(0xc000000007e46fc0), nargs=4,
      arg_vector=0x886be0) at eval.c:3098
  #28 0x01245840 in Ffuncall (nargs=5, args=0x886bd8) at eval.c:2836
  #29 0x01244ca9 in call4 (fn=XIL(0x66b30c8), arg1=XIL(0x8000000008f797d0),
      arg2=XIL(0x8000000008f797d0), arg3=XIL(0), arg4=XIL(0xed20))
      at eval.c:2698
  #30 0x01289c64 in Fload (file=XIL(0x8000000007be85b0), noerror=XIL(0),
      nomessage=XIL(0xed20), nosuffix=XIL(0), must_suffix=XIL(0xed20))
      at lread.c:1367
  #31 0x01258f3b in Frequire (feature=XIL(0x64546b8), filename=XIL(0),
      noerror=XIL(0)) at fns.c:2807
  #32 0x012432e8 in eval_sub (form=XIL(0xc000000008f4ef30)) at eval.c:2293
  #33 0x0128bd2f in readevalloop_eager_expand_eval (
      val=XIL(0xc000000008f4ef30), macroexpand=XIL(0x6a197d8)) at lread.c:1885
  #34 0x0128cbdc in readevalloop (readcharfun=XIL(0xa000000008f3c528),
      infile0=0x0, sourcename=XIL(0x8000000008f78e70), printflag=false,
      unibyte=XIL(0), readfun=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2071
  #35 0x0128d21d in Feval_buffer (buffer=XIL(0xa000000008f3c528),
      printflag=XIL(0), filename=XIL(0x8000000008f6d470), unibyte=XIL(0),
      do_allow_print=XIL(0xed20)) at lread.c:2138
  #36 0x0124343e in eval_sub (form=XIL(0xc000000007e47378)) at eval.c:2301
  #37 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #38 0x0123e1cd in Flet (args=XIL(0xc000000007e47348)) at eval.c:973
  #39 0x01242bf4 in eval_sub (form=XIL(0xc000000007e47338)) at eval.c:2239
  #40 0x0123eca4 in Funwind_protect (args=XIL(0xc000000007e472a8))
      at eval.c:1195
  #41 0x01242bf4 in eval_sub (form=XIL(0xc000000007e47298)) at eval.c:2239
  #42 0x0123b034 in Fprogn (body=XIL(0xc000000007e47090)) at eval.c:459
  #43 0x0123d8e6 in FletX (args=XIL(0xc000000007e47050)) at eval.c:904
  #44 0x01242bf4 in eval_sub (form=XIL(0xc000000007e47040)) at eval.c:2239
  #45 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #46 0x0123accd in Fif (args=XIL(0xc000000007e47010)) at eval.c:415
  #47 0x01242bf4 in eval_sub (form=XIL(0xc000000007e47000)) at eval.c:2239
  #48 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #49 0x01246f3a in funcall_lambda (fun=XIL(0xc000000007e46fc0), nargs=4,
      arg_vector=0x888400) at eval.c:3098
  #50 0x01245840 in Ffuncall (nargs=5, args=0x8883f8) at eval.c:2836
  #51 0x01244ca9 in call4 (fn=XIL(0x66b30c8), arg1=XIL(0x8000000008f6d470),
      arg2=XIL(0x8000000008f6d470), arg3=XIL(0), arg4=XIL(0xed20))
      at eval.c:2698
  #52 0x01289c64 in Fload (file=XIL(0x8000000007bd9fa0), noerror=XIL(0),
      nomessage=XIL(0xed20), nosuffix=XIL(0), must_suffix=XIL(0xed20))
      at lread.c:1367
  #53 0x01258f3b in Frequire (feature=XIL(0x64460a8), filename=XIL(0),
      noerror=XIL(0)) at fns.c:2807
  #54 0x012432e8 in eval_sub (form=XIL(0xc000000007d997b0)) at eval.c:2293
  #55 0x0123b034 in Fprogn (body=XIL(0xc000000007d91060)) at eval.c:459
  #56 0x01246f3a in funcall_lambda (fun=XIL(0xc000000007d91020), nargs=1,
      arg_vector=0x888e10) at eval.c:3098
  #57 0x012463d3 in apply_lambda (fun=XIL(0xc000000007d91010),
      args=XIL(0xc0000000084a0fa0), count=72) at eval.c:2959
  #58 0x01243b7b in eval_sub (form=XIL(0xc0000000084a0f90)) at eval.c:2362
  #59 0x0123b4b4 in Fsetq (args=XIL(0xc0000000084a0f50)) at eval.c:517
  #60 0x01242bf4 in eval_sub (form=XIL(0xc0000000084a0f40)) at eval.c:2239
  #61 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #62 0x0123ae98 in Fcond (args=XIL(0xc0000000084a0df0)) at eval.c:439
  #63 0x01242bf4 in eval_sub (form=XIL(0xc0000000084a0de0)) at eval.c:2239
  #64 0x0123b034 in Fprogn (body=XIL(0xc000000007bfe478)) at eval.c:459
  #65 0x0123e1cd in Flet (args=XIL(0xc000000007bfe458)) at eval.c:973
  #66 0x01242bf4 in eval_sub (form=XIL(0xc000000007bfe448)) at eval.c:2239
  #67 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #68 0x01242bf4 in eval_sub (form=XIL(0xc000000008f21b20)) at eval.c:2239
  #69 0x0123ac55 in Fif (args=XIL(0xc000000008f21b40)) at eval.c:414
  #70 0x01242bf4 in eval_sub (form=XIL(0xc000000008f21b50)) at eval.c:2239
  #71 0x01243ad7 in eval_sub (form=XIL(0xc000000007bfe418)) at eval.c:2358
  #72 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #73 0x01246f3a in funcall_lambda (fun=XIL(0xc000000007bfe3d8), nargs=0,
      arg_vector=0x889d40) at eval.c:3098
  #74 0x012463d3 in apply_lambda (fun=XIL(0xc000000007bfe3c8), args=XIL(0),
      count=63) at eval.c:2959
  #75 0x01243b7b in eval_sub (form=XIL(0xc0000000084853e0)) at eval.c:2362
  #76 0x0123f734 in internal_lisp_condition_case (var=XIL(0x77d5e28),
      bodyform=XIL(0xc0000000084853e0), handlers=XIL(0xc000000008f21a90))
      at eval.c:1307
  #77 0x0123ef97 in Fcondition_case (args=XIL(0xc000000008f21ab0))
      at eval.c:1231
  #78 0x01242bf4 in eval_sub (form=XIL(0xc000000008f21ac0)) at eval.c:2239
  #79 0x01243ad7 in eval_sub (form=XIL(0xc000000008f22690)) at eval.c:2358
  #80 0x01243ad7 in eval_sub (form=XIL(0xc0000000084853b0)) at eval.c:2358
  #81 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #82 0x0123accd in Fif (args=XIL(0xc000000008ff95a0)) at eval.c:415
  #83 0x01242bf4 in eval_sub (form=XIL(0xc000000008ff94e0)) at eval.c:2239
  #84 0x01243ad7 in eval_sub (form=XIL(0xc000000008485370)) at eval.c:2358
  #85 0x0123b034 in Fprogn (body=XIL(0xc000000007c2ff60)) at eval.c:459
  #86 0x0123e1cd in Flet (args=XIL(0xc000000007c2ff40)) at eval.c:973
  #87 0x01242bf4 in eval_sub (form=XIL(0xc000000007c2ff30)) at eval.c:2239
  #88 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #89 0x01246f3a in funcall_lambda (fun=XIL(0xc000000007c2fef0), nargs=1,
      arg_vector=0x88aca0) at eval.c:3098
  #90 0x012463d3 in apply_lambda (fun=XIL(0xc000000007c2fee0),
      args=XIL(0xc000000007d6cbb0), count=53) at eval.c:2959
  #91 0x01243b7b in eval_sub (form=XIL(0xc000000007d6cba0)) at eval.c:2362
  #92 0x0123f734 in internal_lisp_condition_case (var=XIL(0x77d5df0),
      bodyform=XIL(0xc000000007d6cba0), handlers=XIL(0xc000000008ff9710))
      at eval.c:1307
  #93 0x0123ef97 in Fcondition_case (args=XIL(0xc000000008ff96f0))
      at eval.c:1231
  #94 0x01242bf4 in eval_sub (form=XIL(0xc000000008ff96c0)) at eval.c:2239
  #95 0x01243ad7 in eval_sub (form=XIL(0xc000000008ffab30)) at eval.c:2358
  #96 0x01243ad7 in eval_sub (form=XIL(0xc000000007d6cb70)) at eval.c:2358
  #97 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #98 0x01242bf4 in eval_sub (form=XIL(0xc000000008ff3870)) at eval.c:2239
  #99 0x0123ac55 in Fif (args=XIL(0xc000000008ff3850)) at eval.c:414
  #100 0x01242bf4 in eval_sub (form=XIL(0xc000000008ff3840)) at eval.c:2239
  #101 0x01243ad7 in eval_sub (form=XIL(0xc000000007d6cb40)) at eval.c:2358
  #102 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #103 0x0123e1cd in Flet (args=XIL(0xc000000007d6cb10)) at eval.c:973
  #104 0x01242bf4 in eval_sub (form=XIL(0xc000000007d6cb00)) at eval.c:2239
  #105 0x0123b034 in Fprogn (body=XIL(0xc000000007d6c970)) at eval.c:459
  #106 0x01246f3a in funcall_lambda (fun=XIL(0xc000000007d6c900), nargs=1,
      arg_vector=0x88be90) at eval.c:3098
  #107 0x01245840 in Ffuncall (nargs=2, args=0x88be88) at eval.c:2836
  #108 0x012ae87f in exec_byte_code (bytestr=XIL(0x8000000008f6c650),
      vector=XIL(0xa000000008ffd950), maxdepth=make_number(18),
      args_template=make_number(513), nargs=1, args=0x88c620) at bytecode.c:632
  #109 0x01246906 in funcall_lambda (fun=XIL(0xa000000008ffdc90), nargs=1,
      arg_vector=0x88c618) at eval.c:3023
  #110 0x01245686 in Ffuncall (nargs=2, args=0x88c610) at eval.c:2824
  #111 0x012ae87f in exec_byte_code (bytestr=XIL(0x8000000008f30208),
      vector=XIL(0xa000000008fff278), maxdepth=make_number(8),
      args_template=make_number(257), nargs=1, args=0x88cc18) at bytecode.c:632
  #112 0x01246906 in funcall_lambda (fun=XIL(0xa000000008fc1e90), nargs=1,
      arg_vector=0x88cc10) at eval.c:3023
  #113 0x01245686 in Ffuncall (nargs=2, args=0x88cc08) at eval.c:2824
  #114 0x012ae87f in exec_byte_code (bytestr=XIL(0x8000000008f30228),
      vector=XIL(0xa000000008fb80f8), maxdepth=make_number(10),
      args_template=make_number(256), nargs=0, args=0x88d258) at bytecode.c:632
  #115 0x01246906 in funcall_lambda (fun=XIL(0xa000000008fc1e60), nargs=0,
      arg_vector=0x88d258) at eval.c:3023
  #116 0x01245686 in Ffuncall (nargs=1, args=0x88d250) at eval.c:2824
  #117 0x0124304b in eval_sub (form=XIL(0xc0000000080c8e08)) at eval.c:2258
  #118 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #119 0x0123accd in Fif (args=XIL(0xc0000000080c8dd8)) at eval.c:415
  #120 0x01242bf4 in eval_sub (form=XIL(0xc0000000080c8dc8)) at eval.c:2239
  #121 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #122 0x0123ae98 in Fcond (args=XIL(0xc0000000080c6930)) at eval.c:439
  #123 0x01242bf4 in eval_sub (form=XIL(0xc0000000080c6900)) at eval.c:2239
  #124 0x0123b034 in Fprogn (body=XIL(0xc0000000080c6860)) at eval.c:459
  #125 0x0123d8e6 in FletX (args=XIL(0xc0000000080c6820)) at eval.c:904
  #126 0x01242bf4 in eval_sub (form=XIL(0xc0000000080c6810)) at eval.c:2239
  #127 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #128 0x0123b081 in prog_ignore (body=XIL(0xc0000000080c6800)) at eval.c:470
  #129 0x0123e311 in Fwhile (args=XIL(0xc0000000080c67f0)) at eval.c:992
  #130 0x01242bf4 in eval_sub (form=XIL(0xc0000000080c67e0)) at eval.c:2239
  #131 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #132 0x0123d8e6 in FletX (args=XIL(0xc0000000080c67a0)) at eval.c:904
  #133 0x01242bf4 in eval_sub (form=XIL(0xc0000000080c6790)) at eval.c:2239
  #134 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #135 0x01242bf4 in eval_sub (form=XIL(0xc0000000080c6770)) at eval.c:2239
  #136 0x0123ac55 in Fif (args=XIL(0xc0000000080c6750)) at eval.c:414
  #137 0x01242bf4 in eval_sub (form=XIL(0xc0000000080c6740)) at eval.c:2239
  #138 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #139 0x0123e1cd in Flet (args=XIL(0xc0000000080c6720)) at eval.c:973
  #140 0x01242bf4 in eval_sub (form=XIL(0xc0000000080c6710)) at eval.c:2239
  #141 0x0123b034 in Fprogn (body=XIL(0xc0000000080ba3e0)) at eval.c:459
  #142 0x0123e1cd in Flet (args=XIL(0xc0000000080ba3c0)) at eval.c:973
  #143 0x01242bf4 in eval_sub (form=XIL(0xc0000000080ba3b0)) at eval.c:2239
  #144 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #145 0x01246f3a in funcall_lambda (fun=XIL(0xc0000000080ba350), nargs=1,
      arg_vector=0x88e8c0) at eval.c:3098
  #146 0x012463d3 in apply_lambda (fun=XIL(0xc0000000080ba340),
      args=XIL(0xc0000000080ba2d8), count=13) at eval.c:2959
  #147 0x01243b7b in eval_sub (form=XIL(0xc0000000080ba2c8)) at eval.c:2362
  #148 0x0123b034 in Fprogn (body=XIL(0xc0000000080b97a0)) at eval.c:459
  #149 0x01246f3a in funcall_lambda (fun=XIL(0xc0000000080b9600), nargs=0,
      arg_vector=0x88ec10) at eval.c:3098
  #150 0x012463d3 in apply_lambda (fun=XIL(0xc0000000080b95f0), args=XIL(0),
      count=11) at eval.c:2959
  #151 0x01243b7b in eval_sub (form=XIL(0xc0000000080b95a8)) at eval.c:2362
  #152 0x0123eca4 in Funwind_protect (args=XIL(0xc0000000080b8900))
      at eval.c:1195
  #153 0x01242bf4 in eval_sub (form=XIL(0xc0000000080b88f0)) at eval.c:2239
  #154 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #155 0x0123e1cd in Flet (args=XIL(0xc0000000080b88d0)) at eval.c:973
  #156 0x01242bf4 in eval_sub (form=XIL(0xc0000000080b88c0)) at eval.c:2239
  #157 0x0123b034 in Fprogn (body=XIL(0xc0000000080b8320)) at eval.c:459
  #158 0x0123accd in Fif (args=XIL(0xc0000000080b8250)) at eval.c:415
  #159 0x01242bf4 in eval_sub (form=XIL(0xc0000000080b8240)) at eval.c:2239
  #160 0x0123b034 in Fprogn (body=XIL(0)) at eval.c:459
  #161 0x01246f3a in funcall_lambda (fun=XIL(0xc0000000080b8200), nargs=0,
      arg_vector=0x88f5e0) at eval.c:3098
  #162 0x012463d3 in apply_lambda (fun=XIL(0xc0000000080b81f0), args=XIL(0),
      count=4) at eval.c:2959
  #163 0x01243b7b in eval_sub (form=XIL(0xc000000008697260)) at eval.c:2362
  #164 0x01242437 in Feval (form=XIL(0xc000000008697260), lexical=XIL(0))
      at eval.c:2107
  #165 0x011587fa in top_level_2 () at keyboard.c:1122
  #166 0x0123f7d4 in internal_condition_case (bfun=0x11587c4 <top_level_2>,
      handlers=XIL(0x5c50), hfun=0x1157f43 <cmd_error>) at eval.c:1336
  #167 0x0115886f in top_level_1 (ignore=XIL(0)) at keyboard.c:1130
  #168 0x0123e8e5 in internal_catch (tag=XIL(0xf6c0),
      func=0x1158800 <top_level_1>, arg=XIL(0)) at eval.c:1101
  #169 0x011586d0 in command_loop () at keyboard.c:1091
  #170 0x0115797b in recursive_edit_1 () at keyboard.c:698
  #171 0x01157c19 in Frecursive_edit () at keyboard.c:769
  #172 0x011554c0 in main (argc=13, argv=0xcf3610) at emacs.c:1916

  Lisp Backtrace:
  "insert-file-contents" (0x8859c8)
  "let" (0x885cd8)
  "save-current-buffer" (0x885ed8)
  "with-current-buffer" (0x886058)
  "let" (0x886338)
  "unwind-protect" (0x8864f8)
  "let*" (0x886748)
  "if" (0x886948)
  "load-with-code-conversion" (0x886be0)
  "require" (0x8873e8)
  "eval-buffer" (0x887838)
  "let" (0x887b58)
  "unwind-protect" (0x887d18)
  "let*" (0x887f68)
  "if" (0x888168)
  "load-with-code-conversion" (0x888400)
  "require" (0x888c08)
  "dir-locals-read-from-dir" (0x888e10)
  "setq" (0x889198)
  "cond" (0x889398)
  "let" (0x889678)
  "progn" (0x889838)
  "if" (0x8899f8)
  "when" (0x889b78)
  "hack-dir-local-variables" (0x889d40)
  "condition-case" (0x88a188)
  "condition-case-unless-debug" (0x88a308)
  "with-demoted-errors" (0x88a488)
  "if" (0x88a688)
  "unless" (0x88a808)
  "let" (0x88aad8)
  "hack-local-variables" (0x88aca0)
  "condition-case" (0x88b0f8)
  "condition-case-unless-debug" (0x88b278)
  "with-demoted-errors" (0x88b3f8)
  "progn" (0x88b5b8)
  "if" (0x88b778)
  "when" (0x88b8f8)
  "let" (0x88bbc8)
  "normal-mode" (0x88be90)
  "byte-compile-file" (0x88c618)
  "batch-byte-compile-file" (0x88cc10)
  "batch-byte-compile" (0x88d258)
  "funcall" (0x88d250)
  "if" (0x88d508)
  "cond" (0x88d708)
  "let*" (0x88d958)
  "while" (0x88db88)
  "let*" (0x88ddd8)
  "progn" (0x88df98)
  "if" (0x88e158)
  "let" (0x88e428)
  "let" (0x88e6f8)
  "command-line-1" (0x88e8c0)
  "command-line" (0x88ec10)
  "unwind-protect" (0x88ef48)
  "let" (0x88f218)
  "if" (0x88f418)
  "normal-top-level" (0x88f5e0)
  (gdb) fr 4
  #4  0x011acb8b in prepare_to_modify_buffer_1 (start=1, end=1,
      preserve_ptr=0x0) at insdel.c:1937
  1937      eassert (!pdumper_object_p (BEG_ADDR));
  (gdb) p BEG_ADDR
  $4 = (unsigned char *) 0x86d0000 ""
  (gdb) p/x dump_public
  $5 = {
    start = 0x7b80000,
    end = 0x8f1e938
  }

Re: Preview: portable dumper

[Daniel Colascione]

On 02/14/2018 08:24 AM, Eli Zaretskii wrote:
>> From: Robert Pluim <rpluim@gmail.com>
>> Date: Wed, 14 Feb 2018 11:30:07 +0100
>>
>>>> What should we look for?
>>>
>>> The usual stuff, I think: does it build, does it work as expected,
>>> etc.
>>
>> You mean like:
> 
> Yes, like that.
> 
> I see something slightly different on MS-Windows (in a build with
> "--enable-checking"), but maybe similar enough to be explained by the
> same problem:
> 
>         ELC      ../lisp/composite.elc
>       load_dump completed in 46.005 milliseconds
> 
>       insdel.c:1937: Emacs fatal error: assertion failed: !pdumper_object_p (BEG_ADDR)
> 
> Backtrace:
> 
>    #0  0x762c3227 in KERNELBASE!DebugBreak ()
>       from C:\Windows\syswow64\KernelBase.dll
>    #1  0x0131940d in emacs_abort () at w32fns.c:10874
>    #2  0x01152dea in terminate_due_to_signal (sig=22, backtrace_limit=2147483647)
>        at emacs.c:388
>    #3  0x01206274 in die (
>        msg=0x16ce7d2 <DEFAULT_REHASH_SIZE+354> "!pdumper_object_p (BEG_ADDR)",
>        file=0x16ce674 <DEFAULT_REHASH_SIZE+4> "insdel.c", line=1937)
>        at alloc.c:7789
>    #4  0x011acb8b in prepare_to_modify_buffer_1 (start=1, end=1,
>        preserve_ptr=0x0) at insdel.c:1937

It's weird that we're failing there. If we're looking at a buffer with 
dumped contents, we set b->text->beg to NULL, then use the normal 
buffer-allocation procedure (whichever we're compiled to use) to 
allocate memory for the contents. How can the resulting address ever be 
equal to what we started with? Neither mmap_realloc nor r_re_alloc nor 
xrealloc should ever reuse the address.

Re: Preview: portable dumper

[Daniel Colascione]

On 02/14/2018 09:49 AM, Daniel Colascione wrote:
> On 02/14/2018 08:24 AM, Eli Zaretskii wrote:
>>> From: Robert Pluim <rpluim@gmail.com>
>>> Date: Wed, 14 Feb 2018 11:30:07 +0100
>>>
>>>>> What should we look for?
>>>>
>>>> The usual stuff, I think: does it build, does it work as expected,
>>>> etc.
>>>
>>> You mean like:
>>
>> Yes, like that.
>>
>> I see something slightly different on MS-Windows (in a build with
>> "--enable-checking"), but maybe similar enough to be explained by the
>> same problem:
>>
>>         ELC      ../lisp/composite.elc
>>       load_dump completed in 46.005 milliseconds
>>
>>       insdel.c:1937: Emacs fatal error: assertion failed: 
>> !pdumper_object_p (BEG_ADDR)
>>
>> Backtrace:
>>
>>    #0  0x762c3227 in KERNELBASE!DebugBreak ()
>>       from C:\Windows\syswow64\KernelBase.dll
>>    #1  0x0131940d in emacs_abort () at w32fns.c:10874
>>    #2  0x01152dea in terminate_due_to_signal (sig=22, 
>> backtrace_limit=2147483647)
>>        at emacs.c:388
>>    #3  0x01206274 in die (
>>        msg=0x16ce7d2 <DEFAULT_REHASH_SIZE+354> "!pdumper_object_p 
>> (BEG_ADDR)",
>>        file=0x16ce674 <DEFAULT_REHASH_SIZE+4> "insdel.c", line=1937)
>>        at alloc.c:7789
>>    #4  0x011acb8b in prepare_to_modify_buffer_1 (start=1, end=1,
>>        preserve_ptr=0x0) at insdel.c:1937
> 
> It's weird that we're failing there. If we're looking at a buffer with 
> dumped contents, we set b->text->beg to NULL, then use the normal 
> buffer-allocation procedure (whichever we're compiled to use) to 
> allocate memory for the contents. How can the resulting address ever be 
> equal to what we started with? Neither mmap_realloc nor r_re_alloc nor 
> xrealloc should ever reuse the address.

Oh, I think I know what's going on. We must be reallocating from the 
discardable region, which we unmapped on startup, but pdumper_object_p 
still thinks that this memory (which the allocator can legitimately use 
for some other purpose) is part of the dump (since pdumper_object_p uses 
a simple range check) and malfunctions.  I'll add a patch to keep this 
address space reserved.

I just added the actually-discard-the-discardable region code, so I'm 
not surprised it broke. On my machine, we happen to never reuse that 
address space.

Re: Preview: portable dumper

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Wed, 14 Feb 2018 09:49:49 -0800
> 
> >    #3  0x01206274 in die (
> >        msg=0x16ce7d2 <DEFAULT_REHASH_SIZE+354> "!pdumper_object_p (BEG_ADDR)",
> >        file=0x16ce674 <DEFAULT_REHASH_SIZE+4> "insdel.c", line=1937)
> >        at alloc.c:7789
> >    #4  0x011acb8b in prepare_to_modify_buffer_1 (start=1, end=1,
> >        preserve_ptr=0x0) at insdel.c:1937
> 
> It's weird that we're failing there. If we're looking at a buffer with 
> dumped contents, we set b->text->beg to NULL, then use the normal 
> buffer-allocation procedure (whichever we're compiled to use) to 
> allocate memory for the contents. How can the resulting address ever be 
> equal to what we started with? Neither mmap_realloc nor r_re_alloc nor 
> xrealloc should ever reuse the address.

You are talking about what enlarge_buffer_text does?  IOW, this:

    if (pdumper_object_p (old_beg))
      b->text->beg = NULL;
    else
      old_beg = NULL;

  #if defined USE_MMAP_FOR_BUFFERS
    p = mmap_realloc ((void **) &b->text->beg, new_nbytes);
  #elif defined REL_ALLOC
    p = r_re_alloc ((void **) &b->text->beg, new_nbytes);
  #else
    p = xrealloc (b->text->beg, new_nbytes);
  #endif

I will try to look around this when I have time, and see what I find.

Re: Preview: portable dumper

[Daniel Colascione]

On 02/14/2018 11:07 AM, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Wed, 14 Feb 2018 09:49:49 -0800
>>
>>>     #3  0x01206274 in die (
>>>         msg=0x16ce7d2 <DEFAULT_REHASH_SIZE+354> "!pdumper_object_p (BEG_ADDR)",
>>>         file=0x16ce674 <DEFAULT_REHASH_SIZE+4> "insdel.c", line=1937)
>>>         at alloc.c:7789
>>>     #4  0x011acb8b in prepare_to_modify_buffer_1 (start=1, end=1,
>>>         preserve_ptr=0x0) at insdel.c:1937
>>
>> It's weird that we're failing there. If we're looking at a buffer with
>> dumped contents, we set b->text->beg to NULL, then use the normal
>> buffer-allocation procedure (whichever we're compiled to use) to
>> allocate memory for the contents. How can the resulting address ever be
>> equal to what we started with? Neither mmap_realloc nor r_re_alloc nor
>> xrealloc should ever reuse the address.
> 
> You are talking about what enlarge_buffer_text does?  IOW, this:

It should be fixed now. Give it a shot.

Re: Preview: portable dumper

[Eli Zaretskii]

> Cc: rpluim@gmail.com, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Wed, 14 Feb 2018 11:26:37 -0800
> 
> >> It's weird that we're failing there. If we're looking at a buffer with
> >> dumped contents, we set b->text->beg to NULL, then use the normal
> >> buffer-allocation procedure (whichever we're compiled to use) to
> >> allocate memory for the contents. How can the resulting address ever be
> >> equal to what we started with? Neither mmap_realloc nor r_re_alloc nor
> >> xrealloc should ever reuse the address.
> > 
> > You are talking about what enlarge_buffer_text does?  IOW, this:
> 
> It should be fixed now. Give it a shot.

Thanks, the problem is indeed fixed, and with that I was able to
successfully complete the bootstrap.

I also tested an optimized build, and a parallel "make -j8" build, and
they all worked as expected.

The times to load the .pdmp file are around 20 msec for an unoptimized
build and 7 msec for an optimized build, which I think is very
impressive.

For the record, this is a 32-bit MinGW build configured with
"--with-large-int --with-modules --enable-checking", on a 3-year old
Core i7 box running Windows 7.

Re: Preview: portable dumper

[Andy Moreton]

On Thu 15 Feb 2018, Eli Zaretskii wrote:

>> Cc: rpluim@gmail.com, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Wed, 14 Feb 2018 11:26:37 -0800
>> 
>> >> It's weird that we're failing there. If we're looking at a buffer with
>> >> dumped contents, we set b->text->beg to NULL, then use the normal
>> >> buffer-allocation procedure (whichever we're compiled to use) to
>> >> allocate memory for the contents. How can the resulting address ever be
>> >> equal to what we started with? Neither mmap_realloc nor r_re_alloc nor
>> >> xrealloc should ever reuse the address.
>> > 
>> > You are talking about what enlarge_buffer_text does?  IOW, this:
>> 
>> It should be fixed now. Give it a shot.
>
> Thanks, the problem is indeed fixed, and with that I was able to
> successfully complete the bootstrap.
>
> I also tested an optimized build, and a parallel "make -j8" build, and
> they all worked as expected.
>
> The times to load the .pdmp file are around 20 msec for an unoptimized
> build and 7 msec for an optimized build, which I think is very
> impressive.
>
> For the record, this is a 32-bit MinGW build configured with
> "--with-large-int --with-modules --enable-checking", on a 3-year old
> Core i7 box running Windows 7.

It fails to build for 64bit Windows with Mingw64 on msys2:

C:/emacs/git/emacs/pdumper/src/pdumper.c: In function 'dump_read_all':
C:/emacs/git/emacs/pdumper/src/pdumper.c:4784:45: error: conversion to 'unsigned int' from 'size_t {aka long long unsigned int}' may alter its value [-Werror=conversion]
         read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
                                             ^~~~~~~~~~~~~
In file included from C:/emacs/git/emacs/pdumper/src/character.h:27:0,
                 from C:/emacs/git/emacs/pdumper/src/buffer.h:27,
                 from C:/emacs/git/emacs/pdumper/src/pdumper.c:18:
C:/emacs/git/emacs/pdumper/src/pdumper.c: In function 'dump_object_1':
C:/emacs/git/emacs/pdumper/src/lisp.h:206:5: warning: this statement may fall through [-Wimplicit-fallthrough=]
    (suppress_checking || (cond)     \
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     ? (void) 0       \
     ~~~~~~~~~~~~~~~~~~
     : die (# cond, __FILE__, __LINE__))
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:/emacs/git/emacs/pdumper/src/pdumper.c:2915:11: note: in expansion of macro 'eassert'
           eassert ("should not be dumping int: is self-representing" && 0);
           ^~~~~~~
C:/emacs/git/emacs/pdumper/src/pdumper.c:2916:9: note: here
         default:
         ^~~~~~~
cc1.exe: some warnings being treated as errors


This appears to be because sys_read() and _read() take an unsigned int
for count rather than size_t. Changing the code to:

read (fd, (char*) buf + bytes_read, (int)(bytes_to_read - bytes_read));

That builds and appears to run, but it would be nicer to get sys_read
etc. to use the proper types.

    AndyM

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Fri, 16 Feb 2018 11:33:35 +0000
> 
> On Thu 15 Feb 2018, Eli Zaretskii wrote:
> 
> C:/emacs/git/emacs/pdumper/src/pdumper.c: In function 'dump_read_all':
> C:/emacs/git/emacs/pdumper/src/pdumper.c:4784:45: error: conversion to 'unsigned int' from 'size_t {aka long long unsigned int}' may alter its value [-Werror=conversion]
>          read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
>                                              ^~~~~~~~~~~~~
> In file included from C:/emacs/git/emacs/pdumper/src/character.h:27:0,
>                  from C:/emacs/git/emacs/pdumper/src/buffer.h:27,
>                  from C:/emacs/git/emacs/pdumper/src/pdumper.c:18:
> C:/emacs/git/emacs/pdumper/src/pdumper.c: In function 'dump_object_1':
> C:/emacs/git/emacs/pdumper/src/lisp.h:206:5: warning: this statement may fall through [-Wimplicit-fallthrough=]
>     (suppress_checking || (cond)     \
>     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>      ? (void) 0       \
>      ~~~~~~~~~~~~~~~~~~
>      : die (# cond, __FILE__, __LINE__))
>      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> C:/emacs/git/emacs/pdumper/src/pdumper.c:2915:11: note: in expansion of macro 'eassert'
>            eassert ("should not be dumping int: is self-representing" && 0);
>            ^~~~~~~
> C:/emacs/git/emacs/pdumper/src/pdumper.c:2916:9: note: here
>          default:
>          ^~~~~~~
> cc1.exe: some warnings being treated as errors
> 
> 
> This appears to be because sys_read() and _read() take an unsigned int
> for count rather than size_t.

Yes.  Angelo already reported this a few days ago.

> Changing the code to:
> 
> read (fd, (char*) buf + bytes_read, (int)(bytes_to_read - bytes_read));

This cannot be the right solution, because you will lose bits in the
cast, when bytes_to_read is greater than INT_MAX.

When Angelo reported this, I said I didn't understand how a similar
code in sysdep.c:emacs_intr_read does compile without a problem,
although it seems to feed a ptrdiff_t value (which should be 64-bit
wide in the 64-bit Windows build, just like size_t, except for the
signedness), similarly to the above.  Can you spot why that works?
Maybe look at the preprocessed source?  Maybe then we will know how to
fix that properly (or maybe we will discover one more subtle bug ;-).

In general, breaking the read into chunks of INT_MAX should work as
well, but I'd like to know first what's going on in emacs_intr_read,
because perhaps there's a more elegant solution we have available.

Re: Preview: portable dumper

[Andy Moreton]

On Fri 16 Feb 2018, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> This appears to be because sys_read() and _read() take an unsigned int
>> for count rather than size_t.
>
> Yes.  Angelo already reported this a few days ago.
>
>> Changing the code to:
>> 
>> read (fd, (char*) buf + bytes_read, (int)(bytes_to_read - bytes_read));
>
> This cannot be the right solution, because you will lose bits in the
> cast, when bytes_to_read is greater than INT_MAX.

Agreed - it was a quick and dirty hack to get the build to complete.

> When Angelo reported this, I said I didn't understand how a similar
> code in sysdep.c:emacs_intr_read does compile without a problem,
> although it seems to feed a ptrdiff_t value (which should be 64-bit
> wide in the 64-bit Windows build, just like size_t, except for the
> signedness), similarly to the above.  Can you spot why that works?
> Maybe look at the preprocessed source?  Maybe then we will know how to
> fix that properly (or maybe we will discover one more subtle bug ;-).

No idea - I'll take a look over the weekend.

> In general, breaking the read into chunks of INT_MAX should work as
> well, but I'd like to know first what's going on in emacs_intr_read,
> because perhaps there's a more elegant solution we have available.

Given the mismatch between sys_read/_read and read, doesn't the smae
problem exist elsewhere in emacs also ?

    AndyM

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Fri, 16 Feb 2018 16:50:55 +0000
> 
> > In general, breaking the read into chunks of INT_MAX should work as
> > well, but I'd like to know first what's going on in emacs_intr_read,
> > because perhaps there's a more elegant solution we have available.
> 
> Given the mismatch between sys_read/_read and read, doesn't the smae
> problem exist elsewhere in emacs also ?

That other place is emacs_intr_read.

Re: Preview: portable dumper

[Andy Moreton]

On Fri 16 Feb 2018, Eli Zaretskii wrote:
> When Angelo reported this, I said I didn't understand how a similar
> code in sysdep.c:emacs_intr_read does compile without a problem,
> although it seems to feed a ptrdiff_t value (which should be 64-bit
> wide in the 64-bit Windows build, just like size_t, except for the
> signedness), similarly to the above.  Can you spot why that works?

The difference is at the top of pdumper.c:

#ifdef __GNUC__
# pragma GCC diagnostic error "-Wconversion"
# pragma GCC diagnostic error "-Wshadow"

Everything else appears to be built without that warning as error. If I
add similar pragmas to sysdep.c then it also emits a dozen or so
conversion warnings as errors.

    AndyM

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Fri, 16 Feb 2018 17:48:05 +0000
> 
> On Fri 16 Feb 2018, Eli Zaretskii wrote:
> > When Angelo reported this, I said I didn't understand how a similar
> > code in sysdep.c:emacs_intr_read does compile without a problem,
> > although it seems to feed a ptrdiff_t value (which should be 64-bit
> > wide in the 64-bit Windows build, just like size_t, except for the
> > signedness), similarly to the above.  Can you spot why that works?
> 
> The difference is at the top of pdumper.c:
> 
> #ifdef __GNUC__
> # pragma GCC diagnostic error "-Wconversion"
> # pragma GCC diagnostic error "-Wshadow"
> 
> Everything else appears to be built without that warning as error. If I
> add similar pragmas to sysdep.c then it also emits a dozen or so
> conversion warnings as errors.

Ah, okay.  So I guess we've just uncovered a subtle bug, which affects
emacs_intr_read as well, and perhaps other places.  Thanks.

Re: Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/html, Size: 1783 bytes --]

Re: Preview: portable dumper

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

On 02/16/2018 12:43 PM, Daniel Colascione wrote:
> IMHO, we should enable -Wconversion more broadly.

My experience is just the opposite: i.e., that -Wconversion causes more 
trouble than it cures. The pdumper change to INTEGER_TO_CONS is an 
example of trouble. The only reason for that change is to work around a 
compiler bug in GCC that is caused by -Wconversion, a bug that leads to 
a false alarm. I suggest at least the attached patch, which limits the 
damage to pdumper.c instead of letting it spread to other Emacs modules. 
But better yet, I suggest dropping the idea of using -Wconversion even 
on pdumper.c, as it's counterproductive and in high-quality code almost 
inevitably leads to further obfuscation like ALLOW_IMPLICIT_CONVERSION 
and DISALLOW_IMPLICIT_CONVERSION.

PS. Double-parens like that shouldn't be needed in macro bodies, as each 
macro must parenthesize its argument properly anyway.


[-- Attachment #2: 0001-Better-workaround-for-GCC-bug-with-Wconversion.patch --]
[-- Type: text/x-patch, Size: 1851 bytes --]

From 725d6c22f209c96d4ca263ccc173ae77ae7942cf Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@Penguin.CS.UCLA.EDU>
Date: Fri, 16 Feb 2018 13:06:49 -0800
Subject: [PATCH] Better workaround for GCC bug with -Wconversion

* src/lisp.h (INTEGER_TO_CONS): Undo previous change.
* src/pdumper.c (DEFINE_TOLISP_FUNC):
Disable -Wconversion when INTEGER_TO_CONS is used, too.
---
 src/lisp.h    | 6 +++---
 src/pdumper.c | 5 ++++-
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/src/lisp.h b/src/lisp.h
index 45e8a0d791..6d4635f2a7 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -3568,9 +3568,9 @@ extern Lisp_Object arithcompare (Lisp_Object num1, Lisp_Object num2,
    itself, or a cons of two or three integers, or if all else fails a float.
    I should not have side effects.  */
 #define INTEGER_TO_CONS(i)					    \
-  (! FIXNUM_OVERFLOW_P ((i))					    \
-   ? make_number ((EMACS_INT) (i))                                      \
-   : EXPR_SIGNED ((i)) ? intbig_to_lisp ((i)) : uintbig_to_lisp ((i)))
+  (! FIXNUM_OVERFLOW_P (i)					    \
+   ? make_number (i)						    \
+   : EXPR_SIGNED (i) ? intbig_to_lisp (i) : uintbig_to_lisp (i))
 extern Lisp_Object intbig_to_lisp (intmax_t);
 extern Lisp_Object uintbig_to_lisp (uintmax_t);
 
diff --git a/src/pdumper.c b/src/pdumper.c
index a05b2c47ce..cc1847356d 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -691,7 +691,10 @@ dump_object_self_representing_p (Lisp_Object object)
   static Lisp_Object                 \
   fn (type value)                    \
   {                                  \
-    return INTEGER_TO_CONS (value);  \
+    ALLOW_IMPLICIT_CONVERSION;       \
+    Lisp_Object result = INTEGER_TO_CONS (value); \
+    DISALLOW_IMPLICIT_CONVERSION;    \
+    return result;                   \
   }
 
 DEFINE_FROMLISP_FUNC (intmax_t_from_lisp, intmax_t);
-- 
2.14.3

Re: Preview: portable dumper

[Daniel Colascione]

On 02/16/2018 01:09 PM, Paul Eggert wrote:
> On 02/16/2018 12:43 PM, Daniel Colascione wrote:
>> IMHO, we should enable -Wconversion more broadly.
> 
> My experience is just the opposite: i.e., that -Wconversion causes more 
> trouble than it cures. The pdumper change to INTEGER_TO_CONS is an 
> example of trouble. The only reason for that change is to work around a 
> compiler bug in GCC that is caused by -Wconversion, a bug that leads to 
> a false alarm. I suggest at least the attached patch, which limits the 
> damage to pdumper.c instead of letting it spread to other Emacs modules. 
> But better yet, I suggest dropping the idea of using -Wconversion even 
> on pdumper.c, as it's counterproductive and in high-quality code almost 
> inevitably leads to further obfuscation like ALLOW_IMPLICIT_CONVERSION 
> and DISALLOW_IMPLICIT_CONVERSION.

-Wconversion helped me find real bugs and otherwise mysterious bugs in 
pdumper. It also, apparently, found a longstanding subtle bug in a 
completely different part of Emacs. Outside of certain specialized 
casting facilities, -Wconversion ought to warn only on things that are 
actually problems. Have a counter-example?

The ALLOW_IMPLICIT_CONVERSION and DISALLOW_IMPLICIT_CONVERSION macros 
are there in pdumper to account for the rest of Emacs not being 
conversion-clean; if everything worked with -Wconversion, we wouldn't 
need the bracketing.

> PS. Double-parens like that shouldn't be needed in macro bodies, as each 
> macro must parenthesize its argument properly anyway.

Thanks.

Re: Preview: portable dumper

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 2585 bytes --]

On 02/16/2018 01:23 PM, Daniel Colascione wrote:
> Outside of certain specialized casting facilities
DEFINE_TOLISP_FUNC is exactly the sort of specialized casting facility 
where -Wconversion is more likely to mess up than usual, which is why I 
suggested disabling -Wconversion there in pdumper.c. -Wconversion is not 
as useful elsewhere, which is why I suggested leaving INTEGER_TO_CONS alone.

> , -Wconversion ought to warn only on things that are actually 
> problems. Have a counter-example

Sure, revert the pdumper change to INTEGER_TO_CONS but leave pdumper.c 
alone, by applying the attached patch to pdumper. The compile on a 
platform where EMACS_INT is 32 bits, using GCC 7.3.1 20180130 (Red Hat 
7.3.1-2). The resulting diagnostic is a false alarm:

pdumper.c: In function ‘intmax_t_to_lisp’:
pdumper.c:694:29: error: conversion to ‘EMACS_INT {aka int}’ from 
‘intmax_t {aka long long int}’ may alter its value [-Werror=conversion]
      return INTEGER_TO_CONS (value);  \
                              ^
lisp.h:3572:19: note: in definition of macro ‘INTEGER_TO_CONS’
     ? make_number (i)          \
                    ^
pdumper.c:698:1: note: in expansion of macro ‘DEFINE_TOLISP_FUNC’
  DEFINE_TOLISP_FUNC (intmax_t_to_lisp, intmax_t);
  ^~~~~~~~~~~~~~~~~~

This false alarm is due to a GCC bug. GCC is supposed to take 
expressions like this one (adapted from INTEGER_TO_CONS):

   (MOST_NEGATIVE_FIXNUM <= i && i <= MOST_POSITIVE_FIXNUM
    ? make_number (i)
    : something_else (i))

and evaluate make_number (i) in the context of i being in range for 
fixnums (which means i is in range for EMACS_INT). In this particular 
case, GCC has a bug where it forgets i's range, and thus the false alarm.

I'd file a bug report with the GCC folks, but in the past my experience 
is that they don't take -Wconversion bug reports all that seriously. In 
practice, -Wconversion isn't good enough for high-quality code like what 
Emacs should be.


> -Wconversion helped me find real bugs and otherwise mysterious bugs in 
> pdumper.
Yes, -Wconversion can find bugs, particularly the first time one writes 
a program when the program has lots of bugs that need finding. But its 
false-alarm rate is too high to be useful in high-quality code. Although 
it's OK to use -Wconversion temporarily in new or otherwise-buggy areas, 
we shouldn't encourage its use in the part of Emacs that is intended to 
be stable and high-quality.


[-- Attachment #2: pdumper.diff --]
[-- Type: text/x-patch, Size: 776 bytes --]

diff --git a/src/lisp.h b/src/lisp.h
index 45e8a0d791..6d4635f2a7 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -3568,9 +3568,9 @@ extern Lisp_Object arithcompare (Lisp_Object num1, Lisp_Object num2,
    itself, or a cons of two or three integers, or if all else fails a float.
    I should not have side effects.  */
 #define INTEGER_TO_CONS(i)					    \
-  (! FIXNUM_OVERFLOW_P ((i))					    \
-   ? make_number ((EMACS_INT) (i))                                      \
-   : EXPR_SIGNED ((i)) ? intbig_to_lisp ((i)) : uintbig_to_lisp ((i)))
+  (! FIXNUM_OVERFLOW_P (i)					    \
+   ? make_number (i)						    \
+   : EXPR_SIGNED (i) ? intbig_to_lisp (i) : uintbig_to_lisp (i))
 extern Lisp_Object intbig_to_lisp (intmax_t);
 extern Lisp_Object uintbig_to_lisp (uintmax_t);
 

Re: Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/html, Size: 4440 bytes --]

Re: Preview: portable dumper

[Paul Eggert]

On 02/16/2018 02:02 PM, Daniel Colascione wrote:
> I'm not sure what you mean by high quality in this context.

I mean code that is written with reasonable care to check for ranges, 
much as Emacs normally does. For example, INTEGER_TO_CONS checks for 
ranges before converting its argument to EMACS_INT. When such code runs 
afoul of -Wconversion, my experience is that the false alarm rate is so 
high that -Wconversion causes more trouble than it's worth. I suspect 
that pdumper.c also had a bunch of false alarms, which is why it 
contains all that gorp with ALLOW_IMPLICIT_CONVERSION and 
DISALLOW_IMPLICIT_CONVERSION. I'll try to pry free some time to look in 
more detail at pdumper.c and see whether there's a way to fix it without 
worrying so much about -Wconversion.

>     if programmers were angels, no warnings would be necessary. The
>     point is to *keep* the code high quality,
>
Which is why I advocate not using -Wconversion: as a rule, -Wconversion 
takes good code and makes it worse. Of course there are always 
exceptions, and perhaps pdumper.c is one of those exceptions.

Re: Preview: portable dumper

[Robert Pluim]

Daniel Colascione <dancol@dancol.org> writes:

> On 02/14/2018 11:07 AM, Eli Zaretskii wrote:
>>> Cc: emacs-devel@gnu.org
>>> From: Daniel Colascione <dancol@dancol.org>
>>> Date: Wed, 14 Feb 2018 09:49:49 -0800
>>>
>>>>     #3  0x01206274 in die (
>>>>         msg=0x16ce7d2 <DEFAULT_REHASH_SIZE+354> "!pdumper_object_p (BEG_ADDR)",
>>>>         file=0x16ce674 <DEFAULT_REHASH_SIZE+4> "insdel.c", line=1937)
>>>>         at alloc.c:7789
>>>>     #4  0x011acb8b in prepare_to_modify_buffer_1 (start=1, end=1,
>>>>         preserve_ptr=0x0) at insdel.c:1937
>>>
>>> It's weird that we're failing there. If we're looking at a buffer with
>>> dumped contents, we set b->text->beg to NULL, then use the normal
>>> buffer-allocation procedure (whichever we're compiled to use) to
>>> allocate memory for the contents. How can the resulting address ever be
>>> equal to what we started with? Neither mmap_realloc nor r_re_alloc nor
>>> xrealloc should ever reuse the address.
>>
>> You are talking about what enlarge_buffer_text does?  IOW, this:
>
> It should be fixed now. Give it a shot.

I'm not sure if this was meant for me as well. In any case, I just
pulled and rebuilt, and everything seems to be working well as of
commit 9484bb3ab8e39add474400e5982802b61c56eb3a (I'm using it to write
this message).

Robert

Re: Preview: portable dumper

[Robert Pluim]

Robert Pluim <rpluim@gmail.com> writes:
> I'm not sure if this was meant for me as well. In any case, I just
> pulled and rebuilt, and everything seems to be working well as of
> commit 9484bb3ab8e39add474400e5982802b61c56eb3a (I'm using it to write
> this message).

I spoke too soon. Commit fca8372020248318f1d0007378607a912b19026e is causing

./temacs --batch  --load loadup pbootstrap

to segfault. As far as I can tell, all that commit does is replace
eassert with eassume, but reverting it causes the crash to go away.

$ gcc --version
gcc-5.real (Ubuntu 5.4.0-6ubuntu1~16.04.6) 5.4.0 20160609

Thread 1 "temacs" received signal SIGSEGV, Segmentation fault.
dump_fwd (ctx=0x7fffffffd020, fwd=0x0) at pdumper.c:2277
2277	  switch (XFWDTYPE (fwd))
(gdb) bt
#0  dump_fwd (ctx=0x7fffffffd020, fwd=0x0) at pdumper.c:2277
#1  0x0000000000559be5 in dump_pre_dump_symbol (symbol=0xbf9040 <lispsym>, ctx=0x7fffffffd020) at pdumper.c:2361
#2  dump_hot_parts_of_discardable_objects (ctx=0x7fffffffd020) at pdumper.c:3214
#3  Fdump_emacs_portable (filename=<optimized out>, track_referrers=<optimized out>) at pdumper.c:3909
#4  0x000000000057184c in eval_sub (form=<optimized out>) at eval.c:2293
#5  0x00000000005718a3 in eval_sub (form=<optimized out>) at eval.c:2242
#6  0x0000000000575228 in Fprogn (body=<optimized out>) at eval.c:459
#7  Flet (args=XIL(0xcbb513)) at eval.c:973
#8  0x00000000005718a3 in eval_sub (form=<optimized out>) at eval.c:2242
#9  0x00000000005718a3 in eval_sub (form=form@entry=XIL(0xcbc6e3)) at eval.c:2242
#10 0x0000000000597953 in readevalloop (readcharfun=readcharfun@entry=XIL(0x6a50), 
    infile0=infile0@entry=0x7fffffffd6e0, sourcename=sourcename@entry=XIL(0xcac744), printflag=printflag@entry=false, 
    unibyte=unibyte@entry=XIL(0), readfun=readfun@entry=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2073
#11 0x00000000005980fc in Fload (file=XIL(0xcac624), noerror=<optimized out>, nomessage=XIL(0), 
    nosuffix=<optimized out>, must_suffix=<optimized out>) at lread.c:1457
#12 0x0000000000571810 in eval_sub (form=form@entry=XIL(0xca8e73)) at eval.c:2304
#13 0x00000000005757f8 in Feval (form=XIL(0xca8e73), lexical=<optimized out>) at eval.c:2110
#14 0x000000000057088e in internal_condition_case (bfun=bfun@entry=0x4f53a0 <top_level_2>, 
    handlers=handlers@entry=XIL(0x53d0), hfun=hfun@entry=0x4fabd0 <cmd_error>) at eval.c:1336
#15 0x00000000004f7bfc in top_level_1 (ignore=ignore@entry=XIL(0)) at keyboard.c:1130
#16 0x000000000057082c in internal_catch (tag=tag@entry=XIL(0xcae0), func=func@entry=0x4f7ba0 <top_level_1>, 
    arg=arg@entry=XIL(0)) at eval.c:1101
#17 0x00000000004f5338 in command_loop () at keyboard.c:1091
#18 0x00000000004fa7b6 in recursive_edit_1 () at keyboard.c:698
#19 0x00000000004fab00 in Frecursive_edit () at keyboard.c:769
#20 0x000000000041b015 in main (argc=5, argv=0x7fffffffdaf8) at emacs.c:1929

Lisp Backtrace:
"dump-emacs-portable" (0xffffd200)
"if" (0xffffd2b8)
"let" (0xffffd418)
"if" (0xffffd4d8)
"load" (0xffffd7f0)

Re: Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/html, Size: 1145 bytes --]

Re: Preview: portable dumper

[Paul Eggert]

On 02/20/2018 09:19 AM, Daniel Colascione wrote:
> Do you get an assertion failure if you enable assertion checking? If 
> we're seeing this kind of problem from the compiler expectation alone, 
> I'll definitely revert that change.

In my experience, eassume (and 'assume') shouldn't be used for 
complicated expressions, as GCC is too-easily confused by them. eassume 
should be used only for expressions where the assumption really does 
help the compiler, either by generating significantly-better code or by 
pacifying a false alarm.

Re: Preview: portable dumper

[Daniel Colascione]

On 02/20/2018 09:28 AM, Paul Eggert wrote:
> On 02/20/2018 09:19 AM, Daniel Colascione wrote:
>> Do you get an assertion failure if you enable assertion checking? If 
>> we're seeing this kind of problem from the compiler expectation alone, 
>> I'll definitely revert that change.
> 
> In my experience, eassume (and 'assume') shouldn't be used for 
> complicated expressions, as GCC is too-easily confused by them. eassume 
> should be used only for expressions where the assumption really does 
> help the compiler, either by generating significantly-better code or by 
> pacifying a false alarm.

I'd hoped things had gotten better, and some local testing seemed to 
suggest it was safe to use these days. Apparently it's not, which is a 
shame. It'd be nice, but not urgent, to track down which specific 
assumptions caused what I think must be miscompilations so we could 
report them upstream.

Re: Preview: portable dumper

[Robert Pluim]

Daniel Colascione <dancol@dancol.org> writes:

> On 02/20/2018 09:28 AM, Paul Eggert wrote:
>> On 02/20/2018 09:19 AM, Daniel Colascione wrote:
>>> Do you get an assertion failure if you enable assertion checking?
>>> If we're seeing this kind of problem from the compiler expectation
>>> alone, I'll definitely revert that change.
>>
>> In my experience, eassume (and 'assume') shouldn't be used for
>> complicated expressions, as GCC is too-easily confused by
>> them. eassume should be used only for expressions where the
>> assumption really does help the compiler, either by generating
>> significantly-better code or by pacifying a false alarm.
>
> I'd hoped things had gotten better, and some local testing seemed to
> suggest it was safe to use these days. Apparently it's not, which is a
> shame. It'd be nice, but not urgent, to track down which specific
> assumptions caused what I think must be miscompilations so we could
> report them upstream.

The guilty change is:

--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -607,7 +607,7 @@ dump_set_have_current_referrer (struct dump_context *ctx, bool have)
 static void
 DUMP_CLEAR_REFERRER (struct dump_context *ctx)
 {
-  eassert (ctx->have_current_referrer);
+  eassume (ctx->have_current_referrer);
   dump_set_have_current_referrer (ctx, false);
   if (dump_tracking_referrers_p (ctx))
     ctx->current_referrer = Qnil;

If I revert just that hunk, the build succeeds.

Robert

Re: Preview: portable dumper

[Daniel Colascione]

On 02/20/2018 10:09 AM, Robert Pluim wrote:
> Daniel Colascione <dancol@dancol.org> writes:
> 
>> On 02/20/2018 09:28 AM, Paul Eggert wrote:
>>> On 02/20/2018 09:19 AM, Daniel Colascione wrote:
>>>> Do you get an assertion failure if you enable assertion checking?
>>>> If we're seeing this kind of problem from the compiler expectation
>>>> alone, I'll definitely revert that change.
>>>
>>> In my experience, eassume (and 'assume') shouldn't be used for
>>> complicated expressions, as GCC is too-easily confused by
>>> them. eassume should be used only for expressions where the
>>> assumption really does help the compiler, either by generating
>>> significantly-better code or by pacifying a false alarm.
>>
>> I'd hoped things had gotten better, and some local testing seemed to
>> suggest it was safe to use these days. Apparently it's not, which is a
>> shame. It'd be nice, but not urgent, to track down which specific
>> assumptions caused what I think must be miscompilations so we could
>> report them upstream.
> 
> The guilty change is:
> 
> --- a/src/pdumper.c
> +++ b/src/pdumper.c
> @@ -607,7 +607,7 @@ dump_set_have_current_referrer (struct dump_context *ctx, bool have)
>   static void
>   DUMP_CLEAR_REFERRER (struct dump_context *ctx)
>   {
> -  eassert (ctx->have_current_referrer);
> +  eassume (ctx->have_current_referrer);
>     dump_set_have_current_referrer (ctx, false);
>     if (dump_tracking_referrers_p (ctx))
>       ctx->current_referrer = Qnil;
> 
> If I revert just that hunk, the build succeeds.

Wow: thanks for tracking that down.

What on earth? I mean, we're instructing the compiler to assume 
something that's both trivial and true. What GCC version is this? If 
it's recent, I think it's probably worth reporting.

Re: Preview: portable dumper

[Robert Pluim]

Daniel Colascione <dancol@dancol.org> writes:

> On 02/20/2018 10:09 AM, Robert Pluim wrote:
>> Daniel Colascione <dancol@dancol.org> writes:
>>
>>> On 02/20/2018 09:28 AM, Paul Eggert wrote:
>>>> On 02/20/2018 09:19 AM, Daniel Colascione wrote:
>>>>> Do you get an assertion failure if you enable assertion checking?
>>>>> If we're seeing this kind of problem from the compiler expectation
>>>>> alone, I'll definitely revert that change.
>>>>
>>>> In my experience, eassume (and 'assume') shouldn't be used for
>>>> complicated expressions, as GCC is too-easily confused by
>>>> them. eassume should be used only for expressions where the
>>>> assumption really does help the compiler, either by generating
>>>> significantly-better code or by pacifying a false alarm.
>>>
>>> I'd hoped things had gotten better, and some local testing seemed to
>>> suggest it was safe to use these days. Apparently it's not, which is a
>>> shame. It'd be nice, but not urgent, to track down which specific
>>> assumptions caused what I think must be miscompilations so we could
>>> report them upstream.
>>
>> The guilty change is:
>>
>> --- a/src/pdumper.c
>> +++ b/src/pdumper.c
>> @@ -607,7 +607,7 @@ dump_set_have_current_referrer (struct dump_context *ctx, bool have)
>>   static void
>>   DUMP_CLEAR_REFERRER (struct dump_context *ctx)
>>   {
>> -  eassert (ctx->have_current_referrer);
>> +  eassume (ctx->have_current_referrer);
>>     dump_set_have_current_referrer (ctx, false);
>>     if (dump_tracking_referrers_p (ctx))
>>       ctx->current_referrer = Qnil;
>>
>> If I revert just that hunk, the build succeeds.
>
> Wow: thanks for tracking that down.
>

splitpatch + git bisect run can do wonders :-)

> What on earth? I mean, we're instructing the compiler to assume
> something that's both trivial and true. What GCC version is this? If
> it's recent, I think it's probably worth reporting.

$ gcc --version
gcc-5.real (Ubuntu 5.4.0-6ubuntu1~16.04.6) 5.4.0 20160609

I have a gcc-8-ish somewhere that I can rev up to see if it has the
same problem, but probably not soon.

Robert

Re: Preview: portable dumper

[Robert Pluim]

Robert Pluim <rpluim@gmail.com> writes:

> Daniel Colascione <dancol@dancol.org> writes:
>
>> On 02/20/2018 10:09 AM, Robert Pluim wrote:
>> What on earth? I mean, we're instructing the compiler to assume
>> something that's both trivial and true. What GCC version is this? If
>> it's recent, I think it's probably worth reporting.
>
> $ gcc --version
> gcc-5.real (Ubuntu 5.4.0-6ubuntu1~16.04.6) 5.4.0 20160609
>
> I have a gcc-8-ish somewhere that I can rev up to see if it has the
> same problem, but probably not soon.

That gcc-8 turned out to be easy to get ready, and it shows the same
problem (and solution). So now I have an alpha branch of emacs being
allegedly miscompiled by a non-released tip-of-tree version of gcc, in
a feature that messes around with the guts of Emacs' object
representations.

I find it hard to assign blame here. Should I start using unreleased
binutils next? :-)

Robert

Re: Preview: portable dumper

[Stefan Monnier]

> I find it hard to assign blame here.

Really ?!?
Isn't it just always the other guy?


        Stefan

Re: Preview: portable dumper

[Robert Pluim]

Daniel Colascione <dancol@dancol.org> writes:

> On Feb 20, 2018 8:37 AM, Robert Pluim <rpluim@gmail.com> wrote:
>
>  Robert Pluim <rpluim@gmail.com> writes: 
>  > I'm not sure if this was meant for me as well. In any case, I just 
>  > pulled and rebuilt, and everything seems to be working well as of 
>  > commit 9484bb3ab8e39add474400e5982802b61c56eb3a (I'm using it to write 
>  > this message). 
>
>  I spoke too soon. Commit fca8372020248318f1d0007378607a912b19026e is causing 
>
>  ./temacs --batch  --load loadup pbootstrap 
>
>  to segfault. As far as I can tell, all that commit does is replace 
>  eassert with eassume, but reverting it causes the crash to go away. 
>
> Do you get an assertion failure if you enable assertion checking? If we're seeing this kind of problem from the compiler expectation alone, I'll definitely revert that change.

No, --enable-checking=all gets me a different segfault:

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00000000005d4556 in unbind_to (count=<optimized out>, value=value@entry=XIL(0)) at eval.c:3576
#2  0x000000000061bb01 in exec_byte_code (bytestr=<optimized out>, vector=<optimized out>, maxdepth=<optimized out>, 
    args_template=args_template@entry=XIL(0), nargs=nargs@entry=0, args=<optimized out>, args@entry=0x0)
    at bytecode.c:652
#3  0x00000000005d5538 in funcall_lambda (fun=XIL(0xe005c5), nargs=nargs@entry=3, 
    arg_vector=arg_vector@entry=0x7fffffffccb0) at eval.c:3108
#4  0x00000000005d5983 in Ffuncall (nargs=4, args=args@entry=0x7fffffffcca8) at eval.c:2839
#5  0x000000000061bb88 in exec_byte_code (bytestr=<optimized out>, vector=<optimized out>, maxdepth=<optimized out>, 
    args_template=args_template@entry=XIL(0), nargs=nargs@entry=0, args=<optimized out>, args@entry=0x0)
    at bytecode.c:632
#6  0x00000000005d5538 in funcall_lambda (fun=XIL(0xe004a5), nargs=nargs@entry=4, 
    arg_vector=arg_vector@entry=0x7fffffffcec0) at eval.c:3108
#7  0x00000000005d5983 in Ffuncall (nargs=5, args=args@entry=0x7fffffffceb8) at eval.c:2839
#8  0x000000000061bb88 in exec_byte_code (bytestr=<optimized out>, vector=<optimized out>, maxdepth=<optimized out>, 
    args_template=args_template@entry=XIL(0), nargs=nargs@entry=0, args=<optimized out>, args@entry=0x0)
    at bytecode.c:632
#9  0x00000000005d5538 in funcall_lambda (fun=fun@entry=XIL(0xdfc155), nargs=nargs@entry=9, 
    arg_vector=arg_vector@entry=0x7fffffffd160) at eval.c:3108
#10 0x00000000005d4765 in apply_lambda (fun=XIL(0xdfc155), args=<optimized out>, count=count@entry=30) at eval.c:2962
#11 0x00000000005d4b42 in eval_sub (form=form@entry=XIL(0xdf94a3)) at eval.c:2365
#12 0x0000000000607604 in readevalloop (readcharfun=readcharfun@entry=XIL(0x6a50), 
    infile0=infile0@entry=0x7fffffffd480, sourcename=sourcename@entry=XIL(0xe37814), printflag=printflag@entry=false, 
    unibyte=unibyte@entry=XIL(0), readfun=readfun@entry=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2073
#13 0x0000000000607d9b in Fload (file=XIL(0xe37894), noerror=<optimized out>, nomessage=XIL(0), 
    nosuffix=<optimized out>, must_suffix=<optimized out>) at lread.c:1457
#14 0x00000000005d4d64 in eval_sub (form=form@entry=XIL(0xd99ea3)) at eval.c:2304
#15 0x0000000000607604 in readevalloop (readcharfun=readcharfun@entry=XIL(0x6a50), 
    infile0=infile0@entry=0x7fffffffd7a0, sourcename=sourcename@entry=XIL(0xde5394), printflag=printflag@entry=false, 
    unibyte=unibyte@entry=XIL(0), readfun=readfun@entry=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2073
#16 0x0000000000607d9b in Fload (file=XIL(0xde5274), noerror=<optimized out>, nomessage=XIL(0), 
    nosuffix=<optimized out>, must_suffix=<optimized out>) at lread.c:1457
#17 0x00000000005d4d64 in eval_sub (form=form@entry=XIL(0xde2aa3)) at eval.c:2304
#18 0x00000000005d9db8 in Feval (form=XIL(0xde2aa3), lexical=<optimized out>) at eval.c:2110
#19 0x00000000005d3be7 in internal_condition_case (bfun=bfun@entry=0x53b320 <top_level_2>, 
    handlers=handlers@entry=XIL(0x53d0), hfun=hfun@entry=0x541d80 <cmd_error>) at eval.c:1336
#20 0x000000000053ee4c in top_level_1 (ignore=ignore@entry=XIL(0)) at keyboard.c:1130
#21 0x00000000005d3b2c in internal_catch (tag=tag@entry=XIL(0xcae0), func=func@entry=0x53edf0 <top_level_1>, 
    arg=arg@entry=XIL(0)) at eval.c:1101
#22 0x000000000053b2b8 in command_loop () at keyboard.c:1091
#23 0x00000000005418b6 in recursive_edit_1 () at keyboard.c:698
#24 0x0000000000541c38 in Frecursive_edit () at keyboard.c:769
#25 0x000000000041c197 in main (argc=5, argv=0x7fffffffdbb8) at emacs.c:1929
(gdb) 

Re: Preview: portable dumper

[Robert Pluim]

Robert Pluim <rpluim@gmail.com> writes:

>> Do you get an assertion failure if you enable assertion checking? If we're seeing this kind of problem from the compiler expectation alone, I'll definitely revert that change.
>
> No, --enable-checking=all gets me a different segfault:

And in fact I get that segfault even with
fca8372020248318f1d0007378607a912b19026e reverted, but only with
--enable-checking=all

Robert

Re: Preview: portable dumper

[Daniel Colascione]

On 02/20/2018 09:45 AM, Robert Pluim wrote:
> Robert Pluim <rpluim@gmail.com> writes:
> 
>>> Do you get an assertion failure if you enable assertion checking? If we're seeing this kind of problem from the compiler expectation alone, I'll definitely revert that change.
>>
>> No, --enable-checking=all gets me a different segfault:
> 
> And in fact I get that segfault even with
> fca8372020248318f1d0007378607a912b19026e reverted, but only with
> --enable-checking=all

Can you repro with an -O0 build?

Re: Preview: portable dumper

[Robert Pluim]

Daniel Colascione <dancol@dancol.org> writes:

> On 02/20/2018 09:45 AM, Robert Pluim wrote:
>> Robert Pluim <rpluim@gmail.com> writes:
>>
>>>> Do you get an assertion failure if you enable assertion checking? If we're seeing this kind of problem from the compiler expectation alone, I'll definitely revert that change.
>>>
>>> No, --enable-checking=all gets me a different segfault:
>>
>> And in fact I get that segfault even with
>> fca8372020248318f1d0007378607a912b19026e reverted, but only with
>> --enable-checking=all
>
> Can you repro with an -O0 build?

Yes. Backtrace:

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x000000000066dd30 in do_one_unbind (this_binding=0x7fffffffbf40, unwinding=true, bindflag=SET_INTERNAL_UNBIND)
    at eval.c:3457
#2  0x000000000066e17e in unbind_to (count=48, value=XIL(0)) at eval.c:3576
#3  0x00000000006be902 in exec_byte_code (bytestr=XIL(0xf1dbf4), vector=XIL(0xef3585), maxdepth=make_number(5), 
    args_template=XIL(0), nargs=0, args=0x0) at bytecode.c:652
#4  0x000000000066cce0 in funcall_lambda (fun=XIL(0xf015c5), nargs=3, arg_vector=0xef3585) at eval.c:3108
#5  0x000000000066bbf8 in Ffuncall (nargs=4, args=0x7fffffffc438) at eval.c:2827
#6  0x00000000006be824 in exec_byte_code (bytestr=XIL(0xf178e4), vector=XIL(0xf2fca5), maxdepth=make_number(4), 
    args_template=XIL(0), nargs=0, args=0x0) at bytecode.c:632
#7  0x000000000066cce0 in funcall_lambda (fun=XIL(0xf014a5), nargs=4, arg_vector=0xf2fca5) at eval.c:3108
#8  0x000000000066bbf8 in Ffuncall (nargs=5, args=0x7fffffffc8e8) at eval.c:2827
#9  0x00000000006be824 in exec_byte_code (bytestr=XIL(0xef7444), vector=XIL(0xf0d6c5), maxdepth=make_number(6), 
    args_template=XIL(0), nargs=0, args=0x0) at bytecode.c:632
#10 0x000000000066cce0 in funcall_lambda (fun=XIL(0xefd155), nargs=9, arg_vector=0xf0d6c5) at eval.c:3108
#11 0x000000000066c4d1 in apply_lambda (fun=XIL(0xefd155), args=XIL(0xefa303), count=30) at eval.c:2962
#12 0x000000000066a40d in eval_sub (form=XIL(0xefa483)) at eval.c:2335
#13 0x00000000006a3627 in readevalloop (readcharfun=XIL(0x6a50), infile0=0x7fffffffd210, sourcename=XIL(0xf38814), 
    printflag=false, unibyte=XIL(0), readfun=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2073
#14 0x00000000006a17ce in Fload (file=XIL(0xf38894), noerror=XIL(0), nomessage=XIL(0), nosuffix=XIL(0), 
    must_suffix=XIL(0)) at lread.c:1457
#15 0x000000000066a2e4 in eval_sub (form=XIL(0xe9ae83)) at eval.c:2304
#16 0x00000000006a3627 in readevalloop (readcharfun=XIL(0x6a50), infile0=0x7fffffffd5f0, sourcename=XIL(0xee6394), 
    printflag=false, unibyte=XIL(0), readfun=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2073
#17 0x00000000006a17ce in Fload (file=XIL(0xee6274), noerror=XIL(0), nomessage=XIL(0), nosuffix=XIL(0), 
    must_suffix=XIL(0)) at lread.c:1457
#18 0x000000000066a2e4 in eval_sub (form=XIL(0xee3aa3)) at eval.c:2304
#19 0x00000000006696e0 in Feval (form=XIL(0xee3aa3), lexical=XIL(0)) at eval.c:2110
#20 0x000000000059ed23 in top_level_2 () at keyboard.c:1122
#21 0x0000000000667743 in internal_condition_case (bfun=0x59ed00 <top_level_2>, handlers=XIL(0x53d0), 
    hfun=0x59e6ef <cmd_error>) at eval.c:1336
#22 0x000000000059ed67 in top_level_1 (ignore=XIL(0)) at keyboard.c:1130
#23 0x0000000000666c04 in internal_catch (tag=XIL(0xcae0), func=0x59ed25 <top_level_1>, arg=XIL(0)) at eval.c:1101
#24 0x000000000059ec52 in command_loop () at keyboard.c:1091
#25 0x000000000059e1d4 in recursive_edit_1 () at keyboard.c:698
#26 0x000000000059e3ce in Frecursive_edit () at keyboard.c:769
#27 0x000000000059c223 in main (argc=5, argv=0x7fffffffdbb8) at emacs.c:1929
(gdb) up
#1  0x000000000066dd30 in do_one_unbind (this_binding=0x7fffffffbf40, unwinding=true, bindflag=SET_INTERNAL_UNBIND)
    at eval.c:3457
3457	      this_binding->unwind.func (this_binding->unwind.arg);
(gdb) p this_binding->unwind.arg
$1 = XIL(0)

Re: Preview: portable dumper

[Daniel Colascione]

On 02/20/2018 09:45 AM, Robert Pluim wrote:
> Robert Pluim <rpluim@gmail.com> writes:
> 
>>> Do you get an assertion failure if you enable assertion checking? If we're seeing this kind of problem from the compiler expectation alone, I'll definitely revert that change.
>>
>> No, --enable-checking=all gets me a different segfault:
> 
> And in fact I get that segfault even with
> fca8372020248318f1d0007378607a912b19026e reverted, but only with
> --enable-checking=all

Thanks. Can repro locally now.

Re: Preview: portable dumper

[Daniel Colascione]

On 02/20/2018 10:26 AM, Daniel Colascione wrote:
> On 02/20/2018 09:45 AM, Robert Pluim wrote:
>> Robert Pluim <rpluim@gmail.com> writes:
>>
>>>> Do you get an assertion failure if you enable assertion checking? If 
>>>> we're seeing this kind of problem from the compiler expectation 
>>>> alone, I'll definitely revert that change.
>>>
>>> No, --enable-checking=all gets me a different segfault:
>>
>> And in fact I get that segfault even with
>> fca8372020248318f1d0007378607a912b19026e reverted, but only with
>> --enable-checking=all
> 
> Thanks. Can repro locally now.

I wonder whether the --enable-checking=all code is just busted somehow. 
Emacs doesn't finish dumping itself in master either: loadup seems to 
spin after trying to load files.el, even with -O3, and on the pdumper 
branch, we crash in temacs during early loadup well before we do 
anything with pdumper. I'll keep looking when I can.

Re: Preview: portable dumper

[Alan Third]

On Wed, Feb 14, 2018 at 06:29:54AM +0200, Eli Zaretskii wrote:
> > Date: Tue, 13 Feb 2018 23:06:15 +0100 (CET)
> > From: Angelo Graziosi <angelo.g0@libero.it>
> > 
> > Eli Zaretskii wrote:
> > >
> > > I'd urge people to try this branch and report any issues they see.
> > 
> > What should we look for?
> 
> The usual stuff, I think: does it build, does it work as expected,
> etc.

Lots of errors on macOS 10.13 (clang). Looks like they’re mostly the
same thing repeated.

dispnew.c:3019:1: error: argument to 'section' attribute is not valid for this
      target: mach-o section specifier requires a segment and section separated
      by a comma
DEFUN ("redraw-frame", Fredraw_frame, Sredraw_frame, 0, 1, 0,
^
./lisp.h:3183:4: note: expanded from macro 'DEFUN'
   SUBR_SECTION_ATTRIBUTE                                               \
   ^
./lisp.h:3150:52: note: expanded from macro 'SUBR_SECTION_ATTRIBUTE'
# define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
                                                   ^
dispnew.c:3028:1: error: argument to 'section' attribute is not valid for this
      target: mach-o section specifier requires a segment and section separated
      by a comma
DEFUN ("redraw-display", Fredraw_display, Sredraw_display, 0, 0, "",
^
./lisp.h:3183:4: note: expanded from macro 'DEFUN'
   SUBR_SECTION_ATTRIBUTE                                               \
   ^
./lisp.h:3150:52: note: expanded from macro 'SUBR_SECTION_ATTRIBUTE'
# define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
                                                   ^
dispnew.c:5596:1: error: argument to 'section' attribute is not valid for this
      target: mach-o section specifier requires a segment and section separated
      by a comma
DEFUN ("open-termscript", Fopen_termscript, Sopen_termscript,
^
./lisp.h:3183:4: note: expanded from macro 'DEFUN'
   SUBR_SECTION_ATTRIBUTE                                               \
   ^
./lisp.h:3150:52: note: expanded from macro 'SUBR_SECTION_ATTRIBUTE'
# define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
                                                   ^
dispnew.c:5629:1: error: argument to 'section' attribute is not valid for this
      target: mach-o section specifier requires a segment and section separated
      by a comma
DEFUN ("send-string-to-terminal", Fsend_string_to_terminal,
^
./lisp.h:3183:4: note: expanded from macro 'DEFUN'
   SUBR_SECTION_ATTRIBUTE                                               \
   ^
./lisp.h:3150:52: note: expanded from macro 'SUBR_SECTION_ATTRIBUTE'
# define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
                                                   ^
dispnew.c:5672:1: error: argument to 'section' attribute is not valid for this
      target: mach-o section specifier requires a segment and section separated
      by a comma
DEFUN ("ding", Fding, Sding, 0, 1, 0,
^
./lisp.h:3183:4: note: expanded from macro 'DEFUN'
   SUBR_SECTION_ATTRIBUTE                                               \
   ^
./lisp.h:3150:52: note: expanded from macro 'SUBR_SECTION_ATTRIBUTE'
# define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
                                                   ^
dispnew.c:5712:1: error: argument to 'section' attribute is not valid for this
      target: mach-o section specifier requires a segment and section separated
      by a comma
DEFUN ("sleep-for", Fsleep_for, Ssleep_for, 1, 2, 0,
^
./lisp.h:3183:4: note: expanded from macro 'DEFUN'
   SUBR_SECTION_ATTRIBUTE                                               \
   ^
./lisp.h:3150:52: note: expanded from macro 'SUBR_SECTION_ATTRIBUTE'
# define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
                                                   ^
dispnew.c:5812:1: error: argument to 'section' attribute is not valid for this
      target: mach-o section specifier requires a segment and section separated
      by a comma
DEFUN ("redisplay", Fredisplay, Sredisplay, 0, 1, 0,
^
./lisp.h:3183:4: note: expanded from macro 'DEFUN'
   SUBR_SECTION_ATTRIBUTE                                               \
   ^
./lisp.h:3150:52: note: expanded from macro 'SUBR_SECTION_ATTRIBUTE'
# define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
                                                   ^
dispnew.c:5852:1: error: argument to 'section' attribute is not valid for this
      target: mach-o section specifier requires a segment and section separated
      by a comma
DEFUN ("frame-or-buffer-changed-p", Fframe_or_buffer_changed_p,
^
./lisp.h:3183:4: note: expanded from macro 'DEFUN'
   SUBR_SECTION_ATTRIBUTE                                               \
   ^
./lisp.h:3150:52: note: expanded from macro 'SUBR_SECTION_ATTRIBUTE'
# define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
                                                   ^
dispnew.c:6069:34: error: use of undeclared identifier 'will_dump'; did you mean
      'will_dump_p'?
  if (!inhibit_window_system && !will_dump)
                                 ^~~~~~~~~
                                 will_dump_p
./lisp.h:654:1: note: 'will_dump_p' declared here
will_dump_p (void)
^
dispnew.c:6069:34: warning: address of function 'will_dump_p' will always
      evaluate to 'true' [-Wpointer-bool-conversion]
  if (!inhibit_window_system && !will_dump)
                                ~^~~~~~~~~
dispnew.c:6069:34: note: prefix with the address-of operator to silence this
      warning
  if (!inhibit_window_system && !will_dump)
                                 ^
                                 &
dispnew.c:6069:34: note: suffix with parentheses to turn this into a function
      call
  if (!inhibit_window_system && !will_dump)
                                 ^
                                          ()
dispnew.c:6176:1: error: argument to 'section' attribute is not valid for this
      target: mach-o section specifier requires a segment and section separated
      by a comma
DEFUN ("internal-show-cursor", Finternal_show_cursor,
^
./lisp.h:3183:4: note: expanded from macro 'DEFUN'
   SUBR_SECTION_ATTRIBUTE                                               \
   ^
./lisp.h:3150:52: note: expanded from macro 'SUBR_SECTION_ATTRIBUTE'
# define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
                                                   ^
dispnew.c:6192:1: error: argument to 'section' attribute is not valid for this
      target: mach-o section specifier requires a segment and section separated
      by a comma
DEFUN ("internal-show-cursor-p", Finternal_show_cursor_p,
^
./lisp.h:3183:4: note: expanded from macro 'DEFUN'
   SUBR_SECTION_ATTRIBUTE                                               \
   ^
./lisp.h:3150:52: note: expanded from macro 'SUBR_SECTION_ATTRIBUTE'
# define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
                                                   ^
1 warning and 11 errors generated.
-- 
Alan Third

Re: Preview: portable dumper

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1609 bytes --]

Alan Third <alan@idiocy.org> schrieb am Mi., 14. Feb. 2018 um 21:35 Uhr:

> On Wed, Feb 14, 2018 at 06:29:54AM +0200, Eli Zaretskii wrote:
> > > Date: Tue, 13 Feb 2018 23:06:15 +0100 (CET)
> > > From: Angelo Graziosi <angelo.g0@libero.it>
> > >
> > > Eli Zaretskii wrote:
> > > >
> > > > I'd urge people to try this branch and report any issues they see.
> > >
> > > What should we look for?
> >
> > The usual stuff, I think: does it build, does it work as expected,
> > etc.
>
> Lots of errors on macOS 10.13 (clang). Looks like they’re mostly the
> same thing repeated.
>
> dispnew.c:3019:1: error: argument to 'section' attribute is not valid for
> this
>       target: mach-o section specifier requires a segment and section
> separated
>       by a comma
> DEFUN ("redraw-frame", Fredraw_frame, Sredraw_frame, 0, 1, 0,
> ^
> ./lisp.h:3183:4: note: expanded from macro 'DEFUN'
>    SUBR_SECTION_ATTRIBUTE                                               \
>    ^
> ./lisp.h:3150:52: note: expanded from macro 'SUBR_SECTION_ATTRIBUTE'
> # define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
>                                                    ^
> dispnew.c:3028:1: error: argument to 'section' attribute is not valid for
> this
>       target: mach-o section specifier requires a segment and section
> separated
>       by a comma
>

The segment should probably be __DATA, cf.
https://developer.apple.com/library/content/documentation/DeveloperTools/Reference/Assembler/040-Assembler_Directives/asm_directives.html#//apple_ref/doc/uid/TP30000823-TPXREF136
.

[-- Attachment #2: Type: text/html, Size: 2357 bytes --]

Re: Preview: portable dumper

[Daniel Colascione]

On 02/14/2018 12:34 PM, Alan Third wrote:
> On Wed, Feb 14, 2018 at 06:29:54AM +0200, Eli Zaretskii wrote:
>>> Date: Tue, 13 Feb 2018 23:06:15 +0100 (CET)
>>> From: Angelo Graziosi <angelo.g0@libero.it>
>>>
>>> Eli Zaretskii wrote:
>>>>
>>>> I'd urge people to try this branch and report any issues they see.
>>>
>>> What should we look for?
>>
>> The usual stuff, I think: does it build, does it work as expected,
>> etc.
> 
> Lots of errors on macOS 10.13 (clang). Looks like they’re mostly the
> same thing repeated.
> 
> dispnew.c:3019:1: error: argument to 'section' attribute is not valid for this
>        target: mach-o section specifier requires a segment and section separated
>        by a comma
> DEFUN ("redraw-frame", Fredraw_frame, Sredraw_frame, 0, 1, 0,
> ^
> ./lisp.h:3183:4: note: expanded from macro 'DEFUN'
>     SUBR_SECTION_ATTRIBUTE                                               \
>     ^
> ./lisp.h:3150:52: note: expanded from macro 'SUBR_SECTION_ATTRIBUTE'
> # define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
>                                                     ^
> dispnew.c:3028:1: error: argument to 'section' attribute is not valid for this
>        target: mach-o section specifier requires a segment and section separated
>        by a comma
> DEFUN ("redraw-display", Fredraw_display, Sredraw_display, 0, 0, "",
> ^

Thanks. I've pushed a fix, I think.

Re: Preview: portable dumper

[Alan Third]

On Wed, Feb 14, 2018 at 04:49:43PM -0800, Daniel Colascione wrote:
> Thanks. I've pushed a fix, I think.

Got some more. I had to make this change:

modified   src/dispnew.c
@@ -6066,7 +6066,7 @@ init_display (void)
 #endif /* HAVE_NTGUI */
 
 #ifdef HAVE_NS
-  if (!inhibit_window_system && !will_dump)
+  if (!inhibit_window_system && !will_dump_p ())
     {
       Vinitial_window_system = Qns;
       Vwindow_system_version = make_number (10);

and now I get these:

pdumper.c:2904:21: error: implicit conversion turns string literal into bool:
      'char [48]' to 'bool' [-Werror,-Wstring-conversion]
          eassert (!"should not be dumping int: is self-representing");
                   ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./lisp.h:195:43: note: expanded from macro 'eassert'
# define eassert(cond) ((void) (false && (cond))) /* Check COND compiles.  */
                                          ^~~~
pdumper.c:3217:25: error: implicit conversion loses integer precision:
      'EMACS_INT' (aka 'long') to 'enum cold_op' [-Werror,-Wshorten-64-to-32]
      enum cold_op op = XFASTINT (XCAR (item));
                   ~~   ^~~~~~~~~~~~~~~~~~~~~~
pdumper.c:3390:31: error: implicit conversion loses integer precision:
      'EMACS_INT' (aka 'long') to 'enum dump_fixup_type'
      [-Werror,-Wshorten-64-to-32]
  enum dump_fixup_type type = XFASTINT (XCAR (fixup));
                       ~~~~   ^~~~~~~~~~~~~~~~~~~~~~~
pdumper.c:3478:32: error: implicit conversion loses integer precision:
      'EMACS_INT' (aka 'long') to 'enum dump_reloc_type'
      [-Werror,-Wshorten-64-to-32]
  dump_reloc_set_type (&reloc, XFASTINT (dump_pop (&lreloc)));
  ~~~~~~~~~~~~~~~~~~~          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pdumper.c:4726:14: error: implicit conversion from enumeration type 'const enum
      dump_reloc_type' to different enumeration type 'enum Lisp_Type'
      [-Werror,-Wenum-conversion]
    ? reloc->type
      ~~~~~~~^~~~
pdumper.c:5007:62: warning: size argument in 'memcmp' call is a comparison
      [-Wmemsize-comparison]
  if (memcmp (header->magic, dump_magic, sizeof (dump_magic) != 0))
                                         ~~~~~~~~~~~~~~~~~~~~^~~~
pdumper.c:5007:7: note: did you mean to compare the result of 'memcmp' instead?
  if (memcmp (header->magic, dump_magic, sizeof (dump_magic) != 0))
      ^                                                          ~
                                                            )
pdumper.c:5007:42: note: explicitly cast the argument to size_t to silence this
      warning
  if (memcmp (header->magic, dump_magic, sizeof (dump_magic) != 0))
                                         ^
                                         (size_t)(               )
1 warning and 5 errors generated.

I suspect some of these are because of clang.
-- 
Alan Third

Re: Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/html, Size: 815 bytes --]

Re: Preview: portable dumper

[Alan Third]

On Thu, Feb 15, 2018 at 11:49:30AM -0800, Daniel Colascione wrote:
> Thanks for checking. If you disable Wconversion at the top of the
> file, you should be able to unblock yourself. The warning helps make
> the code robust, but it's annoying to account for compiler
> differences.

Thanks, I’ve got it to build and it seems to work fine.

Here are the changes I had to make:

3 files changed, 15 insertions(+), 15 deletions(-)
src/dispnew.c |  2 +-
src/macfont.m | 26 +++++++++++++-------------
src/pdumper.c |  2 +-

modified   src/dispnew.c
@@ -6066,7 +6066,7 @@ init_display (void)
 #endif /* HAVE_NTGUI */
 
 #ifdef HAVE_NS
-  if (!inhibit_window_system && !will_dump)
+  if (!inhibit_window_system && !will_dump_p ())
     {
       Vinitial_window_system = Qns;
       Vwindow_system_version = make_number (10);
modified   src/macfont.m
@@ -1646,7 +1646,7 @@ static int macfont_variation_glyphs (struct font *, int c,
                                      unsigned variations[256]);
 static void macfont_filter_properties (Lisp_Object, Lisp_Object);
 
-static struct font_driver const macfont_driver =
+static struct font_driver macfont_driver =
   {
   .type = LISPSYM_INITIALLY (Qmac_ct),
   .get_cache = macfont_get_cache,
@@ -4034,6 +4034,18 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
 
 \f
 
+static void
+syms_of_macfont_for_pdumper (void)
+{
+  // if (RESTORING_FROM_DUMP)
+  //   macfont_family_cache = Qnil;
+  // else
+  //   eassert (NILP (macfont_family_cache));
+
+  macfont_driver.type = Qmac_ct;
+  register_font_driver (&macfont_driver, NULL);
+}
+
 void
 syms_of_macfont (void)
 {
@@ -4056,15 +4068,3 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
 
   pdumper_do_now_and_after_load (syms_of_macfont_for_pdumper);
 }
-
-static void
-syms_of_macfont_for_pdumper (void)
-{
-  if (RESTORING_FROM_DUMP)
-    macfont_family_cache = Qnil;
-  else
-    eassert (NILP (macfont_family_cache));
-
-  macfont_driver.type = Qmac_ct;
-  register_font_driver (&macfont_driver, NULL);
-}
modified   src/pdumper.c
@@ -55,7 +55,7 @@
 #ifdef HAVE_PDUMPER
 
 #ifdef __GNUC__
-# pragma GCC diagnostic error "-Wconversion"
+/* # pragma GCC diagnostic error "-Wconversion"  */
 # pragma GCC diagnostic error "-Wshadow"
 # define ALLOW_IMPLICIT_CONVERSION                       \
   _Pragma ("GCC diagnostic push")                        \


Presumably RESTORING_FROM_DUMP is a left‐over from previous code? I
couldn’t see at a glance what to replace it with.

Oddly whenever I had a problem with macfont.m I had to do ‘make clean’
before make would work again. I’m not sure what was going on or if
it’s specific to this branch.
-- 
Alan Third

Re: Preview: portable dumper

[Daniel Colascione]

On 02/15/2018 12:35 PM, Alan Third wrote:
> On Thu, Feb 15, 2018 at 11:49:30AM -0800, Daniel Colascione wrote:
>> Thanks for checking. If you disable Wconversion at the top of the
>> file, you should be able to unblock yourself. The warning helps make
>> the code robust, but it's annoying to account for compiler
>> differences.
> 
> Thanks, I’ve got it to build and it seems to work fine.
> 
> Here are the changes I had to make:

All fixed, I think.
> @@ -1646,7 +1646,7 @@ static int macfont_variation_glyphs (struct font *, int c,
>                                        unsigned variations[256]);
>   static void macfont_filter_properties (Lisp_Object, Lisp_Object);
>   
> -static struct font_driver const macfont_driver =
> +static struct font_driver macfont_driver =
>     {
>     .type = LISPSYM_INITIALLY (Qmac_ct),
>     .get_cache = macfont_get_cache,
> @@ -4034,6 +4034,18 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
>   
>   

Why did you have to remove the const?

Re: Preview: portable dumper

[Alan Third]

On Thu, Feb 15, 2018 at 02:02:03PM -0800, Daniel Colascione wrote:
> All fixed, I think.

Thanks. I’m still seeing this:

macfont.m:4057:34: error: use of undeclared identifier
      'syms_of_macfont_for_pdumper'
  pdumper_do_now_and_after_load (syms_of_macfont_for_pdumper);

I moved syms_of_macfont_for_pdumper to before syms_of_macfont to fix
it.

> > @@ -1646,7 +1646,7 @@ static int macfont_variation_glyphs (struct font *, int c,
> >                                        unsigned variations[256]);
> >   static void macfont_filter_properties (Lisp_Object, Lisp_Object);
> > -static struct font_driver const macfont_driver =
> > +static struct font_driver macfont_driver =
> >     {
> >     .type = LISPSYM_INITIALLY (Qmac_ct),
> >     .get_cache = macfont_get_cache,
> > @@ -4034,6 +4034,18 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
> 
> Why did you have to remove the const?

macfont.m:4045:23: error: cannot assign to variable 'macfont_driver' with
      const-qualified type 'const struct font_driver'
  macfont_driver.type = Qmac_ct;
  ~~~~~~~~~~~~~~~~~~~ ^
macfont.m:1649:33: note: variable 'macfont_driver' declared const here
static struct font_driver const macfont_driver =
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~

I think we’ll also have to copy the pdmp file into Emacs.app on
install, but I don’t know enough about how that works to do it just
now.
-- 
Alan Third

Re: Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/html, Size: 3438 bytes --]

Re: Preview: portable dumper

[Paul Eggert]

On 02/15/2018 03:34 PM, Daniel Colascione wrote:
> I do wonder whether it makes sense to try to copy the dump into the 
> Emacs executable itself instead of leaving it as a separate file.

I like this idea.

Re: Preview: portable dumper

[Daniel Colascione]

On 02/15/2018 04:47 PM, Paul Eggert wrote:
> On 02/15/2018 03:34 PM, Daniel Colascione wrote:
>> I do wonder whether it makes sense to try to copy the dump into the 
>> Emacs executable itself instead of leaving it as a separate file.
> 
> I like this idea.

A simpler approach would be to just concatenate the dump to temacs, then 
add a special header with a signature and a dump length. On startup, 
Emacs could open itself, seek to the end of the file, look for the 
signature, and if it's present, back up to the start of the dump and 
proceed exactly as pdumper does now. This way, there's no overwriting 
stuff in the middle of the file and we don't waste disk space on a big 
data section we'll only partially use.

Re: Preview: portable dumper

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, Angelo Graziosi <angelo.g0@libero.it>,
>  emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Thu, 15 Feb 2018 17:07:29 -0800
> 
> A simpler approach would be to just concatenate the dump to temacs, then 
> add a special header with a signature and a dump length.

But it will have to be a special section, right?  Because otherwise we
again will prevent Binutils from doing useful stuff with such an
executable, right?

Re: Preview: portable dumper

[Daniel Colascione]

On 02/16/2018 12:30 AM, Eli Zaretskii wrote:
>> Cc: Eli Zaretskii <eliz@gnu.org>, Angelo Graziosi <angelo.g0@libero.it>,
>>   emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Thu, 15 Feb 2018 17:07:29 -0800
>>
>> A simpler approach would be to just concatenate the dump to temacs, then
>> add a special header with a signature and a dump length.
> 
> But it will have to be a special section, right?  Because otherwise we
> again will prevent Binutils from doing useful stuff with such an
> executable, right?

Just concatenating bytes to the end of an executable doesn't break that 
executable, at least not on any platform I know about. binutils keeps 
working. No special section required. Strip does remove the extra data 
though, unfortunately, and this procedure does break any cryptographic 
signatures (although we're not using any right now).

~/edev/trunk/src
$ readelf -h ./emacs
ELF Header:
   Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
   Class:                             ELF64
   Data:                              2's complement, little endian
   Version:                           1 (current)
   OS/ABI:                            UNIX - System V
   ABI Version:                       0
   Type:                              EXEC (Executable file)
   Machine:                           Advanced Micro Devices X86-64
   Version:                           0x1
   Entry point address:               0x41e9f0
   Start of program headers:          64 (bytes into file)
   Start of section headers:          51310168 (bytes into file)
   Flags:                             0x0
   Size of this header:               64 (bytes)
   Size of program headers:           56 (bytes)
   Number of program headers:         9
   Size of section headers:           64 (bytes)
   Number of section headers:         41
   Section header string table index: 40

~/edev/trunk/src
$ ./emacs -batch -Q --eval '(kill-emacs)'

~/edev/trunk/src
$ unzip -t kittens.zip
Archive:  kittens.zip
     testing: kittens.jpg              OK
No errors detected in compressed data of kittens.zip.

~/edev/trunk/src
$ cat emacs kittens.zip > emacs-with-kittens

~/edev/trunk/src
$ readelf -h ./emacs-with-kittens
ELF Header:
   Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
   Class:                             ELF64
   Data:                              2's complement, little endian
   Version:                           1 (current)
   OS/ABI:                            UNIX - System V
   ABI Version:                       0
   Type:                              EXEC (Executable file)
   Machine:                           Advanced Micro Devices X86-64
   Version:                           0x1
   Entry point address:               0x41e9f0
   Start of program headers:          64 (bytes into file)
   Start of section headers:          51310168 (bytes into file)
   Flags:                             0x0
   Size of this header:               64 (bytes)
   Size of program headers:           56 (bytes)
   Number of program headers:         9
   Size of section headers:           64 (bytes)
   Number of section headers:         41
   Section header string table index: 40

~/edev/trunk/src
$ chmod +x emacs-with-kittens

~/edev/trunk/src
$ ./emacs-with-kittens -batch -Q --eval '(kill-emacs)'

~/edev/trunk/src
$ unzip -t emacs-with-kittens
Archive:  emacs-with-kittens
warning [emacs-with-kittens]:  51312792 extra bytes at beginning or 
within zipfile
   (attempting to process anyway)
     testing: kittens.jpg              OK
No errors detected in compressed data of emacs-with-kittens.

Re: Preview: portable dumper

[Eli Zaretskii]

> Cc: eggert@cs.ucla.edu, alan@idiocy.org, angelo.g0@libero.it,
>  emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Fri, 16 Feb 2018 07:02:33 -0800
> 
> Just concatenating bytes to the end of an executable doesn't break that 
> executable, at least not on any platform I know about. binutils keeps 
> working. No special section required. Strip does remove the extra data 
> though, unfortunately

I'm guessing similar problems will happen with objcopy and objdump.

Re: Preview: portable dumper

[Andy Moreton]

On Fri 16 Feb 2018, Daniel Colascione wrote:
> Just concatenating bytes to the end of an executable doesn't break that
> executable, at least not on any platform I know about. binutils keeps working.
> No special section required. Strip does remove the extra data though,
> unfortunately, and this procedure does break any cryptographic signatures

...and that is why it should not be done. Most platforms are moving in
the direction of requiring cryptographically signed binaries.

> (although we're not using any right now).

However it will be necessary to support this in the near future.

Thanks for your work on the portable dumper though - this already looks
solid enough that the old dump code does not have long to live...

   AndyM

Re: Preview: portable dumper

[Stefan Monnier]

> I do wonder whether it makes sense to try to copy the dump into the
> Emacs executable itself instead of leaving it as a separate file.

We could try, but:
- it adds complexity and maybe system-dependent hacks.
- it closes the opportunity to have several dump files for a single
  executable (I was thinking we could try and let end-users build their
  own dump file).


        Stefan

Re: Preview: portable dumper

[Daniel Colascione]

On 02/15/2018 05:54 PM, Stefan Monnier wrote:
>> I do wonder whether it makes sense to try to copy the dump into the
>> Emacs executable itself instead of leaving it as a separate file.
> 
> We could try, but:
> - it adds complexity and maybe system-dependent hacks.

Not if we just tack onto the end.

> - it closes the opportunity to have several dump files for a single
>    executable (I was thinking we could try and let end-users build their
>    own dump file).
Not necessarily. Suppose we go with the append-to-the-executable option. 
Then, to "re-dump" emacs, we'd open the current executable, copy it to a 
temporary file, back up to the start of the dump, ftruncate. Now we've 
recovered temacs and we can go through normal loadup and dump.

Maybe we can even automatically detect changes to any file in 
load-history and perform this procedure automatically on startup.

Re: Preview: portable dumper

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I do wonder whether it makes sense to try to copy the dump into the
>> Emacs executable itself instead of leaving it as a separate file.
>
> We could try, but:
> - it adds complexity and maybe system-dependent hacks.
> - it closes the opportunity to have several dump files for a single
>   executable (I was thinking we could try and let end-users build their
>   own dump file).

While I've got little to add the discussion of the innards, this
occurred to me also. In fact, I've been trying it -- I figured that
a "redump Emacs after load .emacs" would be a fairly attractive option,
especially if it was automated.

Anyway, at the moment, I can't get it to work. Calling this:

(dump-emacs-portable "~/scratch/test.pdmp")

after loading my .emacs works (early versions crashed, so something's
been fixed there). However, trying to use this:

./src/emacs -q --dump-file ~/scratch/test.pdmp 
emacs: could not load dump file "/home/phillord/scratch/test.pdmp": not a dump file

Phil

Re: Preview: portable dumper

[Daniel Colascione]

On 02/21/2018 02:04 PM, Phillip Lord wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
>>> I do wonder whether it makes sense to try to copy the dump into the
>>> Emacs executable itself instead of leaving it as a separate file.
>>
>> We could try, but:
>> - it adds complexity and maybe system-dependent hacks.
>> - it closes the opportunity to have several dump files for a single
>>    executable (I was thinking we could try and let end-users build their
>>    own dump file).
> 
> While I've got little to add the discussion of the innards, this
> occurred to me also. In fact, I've been trying it -- I figured that
> a "redump Emacs after load .emacs" would be a fairly attractive option,
> especially if it was automated.
> 
> Anyway, at the moment, I can't get it to work. Calling this:
> 
> (dump-emacs-portable "~/scratch/test.pdmp")
> 
> after loading my .emacs works (early versions crashed, so something's
> been fixed there). However, trying to use this:
> 
> ./src/emacs -q --dump-file ~/scratch/test.pdmp
> emacs: could not load dump file "/home/phillord/scratch/test.pdmp": not a dump file

I'm able to re-dump now. Can you try it again?

Re: Preview: portable dumper

[Phillip Lord]

Daniel Colascione <dancol@dancol.org> writes:

> On 02/21/2018 02:04 PM, Phillip Lord wrote:
>> Anyway, at the moment, I can't get it to work. Calling this:
>>
>> (dump-emacs-portable "~/scratch/test.pdmp")
>>
>> after loading my .emacs works (early versions crashed, so something's
>> been fixed there). However, trying to use this:
>>
>> ./src/emacs -q --dump-file ~/scratch/test.pdmp
>> emacs: could not load dump file "/home/phillord/scratch/test.pdmp": not a dump file
>
> I'm able to re-dump now. Can you try it again?

Dumping from ./src/emacs -q

gives me this in *Messages* after the dump (although the call to
dump-emacs-portable returns without error).

eval: unsupported object type in dump: SAVE_POINTER

and this on trying to reload with

./src/emacs -q --dump-file ~/scratch/test.pdmp

Gives:

dump file is result of failed dump attempt

on the command line. On an emacs started with

./src/emacs

and running the various stuff in my .emacs, I get this:

eval: unsupported object type in dump: window configuration

Likewise, cannot restart with --dump-file. This is running on Ubuntu
16.04.3, 64 bit.

Phil

Re: Preview: portable dumper

[Clément Pit-Claudel]

On 2018-02-26 10:10, Phillip Lord wrote:
> eval: unsupported object type in dump: SAVE_POINTER

You can pass t to dump-emacs-portable to see where the unsupported values come from.  For me SAVE_POINTERs came from loading dbus, and window configurations from loading winner-mode.

Clément.

Re: Preview: portable dumper

[Daniel Colascione]

On 02/26/2018 07:23 AM, Clément Pit-Claudel wrote:
> On 2018-02-26 10:10, Phillip Lord wrote:
>> eval: unsupported object type in dump: SAVE_POINTER
> 
> You can pass t to dump-emacs-portable to see where the unsupported values come from.  For me SAVE_POINTERs came from loading dbus, and window configurations from loading winner-mode.

I love when people use features that aren't even properly documented 
yet. :-)

I'm imagining using 'with-wrapper-hook' (or, since it's deprecated, 
'add-function') to give modes a chance to tweak the global lisp 
environment that pdumper sees, avoiding some of these problems. Another 
option, which isn't necessarily mutually exclusive, is to provide a list 
of symbols to dump as nil.

For now, you're going to get the most reliable results dumping a -batch 
Emacs.

Re: Preview: portable dumper

[Andy Moreton]

On Mon 26 Feb 2018, Daniel Colascione wrote:

> On 02/26/2018 07:23 AM, Clément Pit-Claudel wrote:
>> On 2018-02-26 10:10, Phillip Lord wrote:
>>> eval: unsupported object type in dump: SAVE_POINTER
>>
>> You can pass t to dump-emacs-portable to see where the unsupported values
>> come from. For me SAVE_POINTERs came from loading dbus, and window
>> configurations from loading winner-mode.
>
> I love when people use features that aren't even properly documented yet. :-)

For the benefit of readers of this list who have been asked to test the
pdumper branch and report their findings, can you please describe what is
expected to work correctly, and which areas still need additional work.

Please add the "--dump-file" command line argument to the command line
help in usage_message[] in emacs.c.

    AndyM

Re: Preview: portable dumper

[Phillip Lord]

Daniel Colascione <dancol@dancol.org> writes:

> On 02/26/2018 07:23 AM, Clément Pit-Claudel wrote:
>> On 2018-02-26 10:10, Phillip Lord wrote:
>>> eval: unsupported object type in dump: SAVE_POINTER
>>
>> You can pass t to dump-emacs-portable to see where the unsupported
>> values come from.  For me SAVE_POINTERs came from loading dbus, and
>> window configurations from loading winner-mode.
>
> I love when people use features that aren't even properly documented
> yet. :-)
>
> I'm imagining using 'with-wrapper-hook' (or, since it's deprecated,
> 'add-function') to give modes a chance to tweak the global lisp
> environment that pdumper sees, avoiding some of these
> problems. Another option, which isn't necessarily mutually exclusive,
> is to provide a list of symbols to dump as nil.
>
> For now, you're going to get the most reliable results dumping a
> -batch Emacs.

I assumed that this was the case. Do you want bug reports about dumping
a non-batch? I think the use case of speeding up start up is a good one
to address, but I am aware that it's an add-on that Emacs doesn't
support at the moment. Getting the dumper stable for normal development
use is the main thing.

Phil

Re: Preview: portable dumper

[Eli Zaretskii]

> Date: Thu, 15 Feb 2018 15:34:13 -0800
> From: Daniel Colascione <dancol@dancol.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, Angelo Graziosi <angelo.g0@libero.it>,
>  emacs-devel@gnu.org
> 
> I do wonder whether it makes sense to try to copy the dump into the Emacs executable itself instead of
> leaving it as a separate file. We could do it independently of executable format by defining a data array in static
> storage that's initially full of, say, 15MB of zeroes prefixed by a long random header (like a MIME boundary),
> then, after we generate emacs.pdmp, copying the dump file into the executable at the place where we see
> that random header. If the dump turns out to be bigger than that 15MB, we can fail the build and ask the user
> to enlarge the array.
> 
> I don't know of any executable format for which this scheme would fail.

Wouldn't that make the dumper stuff less portable, in the sense that
it would need to be compatible with low-level details of executable
file formats on various systems?

At least on non-ELF systems, AFAIK the flexibility of putting
arbitrary sections into an executable is lower than desired.  For
example, before Emacs 25 the MS-Windows build would create a special
section for the initialized Emacs data, which had the annoying effect
of running afoul of 'strip', because Binutils don't know about this
section, and therefore stripping would produce a dysfunctional
executable.  It also prevented re-dumping Emacs, something we had in
the past and I'd like us to have again in the future.

Wouldn't copying the dump into the executable hit the same problems,
at least in principle?

> This way, we could get rid of the current complicated dump-file location logic early in Emacs init.

Can you elaborate why is that logic so complicated?

Re: Preview: portable dumper

[Andy Moreton]

On Fri 16 Feb 2018, Eli Zaretskii wrote:

>> Date: Thu, 15 Feb 2018 15:34:13 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>, Angelo Graziosi <angelo.g0@libero.it>,
>>  emacs-devel@gnu.org
>> 
>> I do wonder whether it makes sense to try to copy the dump into the Emacs executable itself instead of
>> leaving it as a separate file. We could do it independently of executable format by defining a data array in static
>> storage that's initially full of, say, 15MB of zeroes prefixed by a long random header (like a MIME boundary),
>> then, after we generate emacs.pdmp, copying the dump file into the executable at the place where we see
>> that random header. If the dump turns out to be bigger than that 15MB, we can fail the build and ask the user
>> to enlarge the array.
>> 
>> I don't know of any executable format for which this scheme would fail.
>
> Wouldn't that make the dumper stuff less portable, in the sense that
> it would need to be compatible with low-level details of executable
> file formats on various systems?
>
> At least on non-ELF systems, AFAIK the flexibility of putting
> arbitrary sections into an executable is lower than desired.  For
> example, before Emacs 25 the MS-Windows build would create a special
> section for the initialized Emacs data, which had the annoying effect
> of running afoul of 'strip', because Binutils don't know about this
> section, and therefore stripping would produce a dysfunctional
> executable.  It also prevented re-dumping Emacs, something we had in
> the past and I'd like us to have again in the future.
>
> Wouldn't copying the dump into the executable hit the same problems,
> at least in principle?

I don't understand the desire to put the dump within the eamcs
executable, as I thought the whole point of this exercise was to avoid
dodgy manipulation of executable file formats.

Anything that fiddles with the executable prevents signing the
executable, which in many environments is desireable.

    AndyM

Re: Preview: portable dumper

[Daniel Colascione]

On 02/16/2018 03:30 AM, Andy Moreton wrote:
> On Fri 16 Feb 2018, Eli Zaretskii wrote:
> 
>>> Date: Thu, 15 Feb 2018 15:34:13 -0800
>>> From: Daniel Colascione <dancol@dancol.org>
>>> Cc: Eli Zaretskii <eliz@gnu.org>, Angelo Graziosi <angelo.g0@libero.it>,
>>>   emacs-devel@gnu.org
>>>
>>> I do wonder whether it makes sense to try to copy the dump into the Emacs executable itself instead of
>>> leaving it as a separate file. We could do it independently of executable format by defining a data array in static
>>> storage that's initially full of, say, 15MB of zeroes prefixed by a long random header (like a MIME boundary),
>>> then, after we generate emacs.pdmp, copying the dump file into the executable at the place where we see
>>> that random header. If the dump turns out to be bigger than that 15MB, we can fail the build and ask the user
>>> to enlarge the array.
>>>
>>> I don't know of any executable format for which this scheme would fail.
>>
>> Wouldn't that make the dumper stuff less portable, in the sense that
>> it would need to be compatible with low-level details of executable
>> file formats on various systems?
>>
>> At least on non-ELF systems, AFAIK the flexibility of putting
>> arbitrary sections into an executable is lower than desired.  For
>> example, before Emacs 25 the MS-Windows build would create a special
>> section for the initialized Emacs data, which had the annoying effect
>> of running afoul of 'strip', because Binutils don't know about this
>> section, and therefore stripping would produce a dysfunctional
>> executable.  It also prevented re-dumping Emacs, something we had in
>> the past and I'd like us to have again in the future.
>>
>> Wouldn't copying the dump into the executable hit the same problems,
>> at least in principle?
> 
> I don't understand the desire to put the dump within the eamcs
> executable, as I thought the whole point of this exercise was to avoid
> dodgy manipulation of executable file formats.

It's just a matter of packaging. As I mentioned in the other message I 
sent, keeping the dump separate from the executable breaks some 
perfectly-natural things you might want to do with the Emacs executable, 
like copying and renaming it. If possible, I'd like to keep these things 
working. If we can't, that's a shame, but it's not the end of the world.

We do need to avoid platform-specific executable file manipulation and 
weird runtime hacks. There are three options on the table:

1) Keeping the dump as a separate file,
2) Appending the dump to the end of the executable, and
3) Overwriting a data-section array with the dump contents.

All of these options avoid dodgy platform-specific executable 
modifications of the sort you see in, say, unexw32.c, and they're all 
compatible with emacs as a position-independent executable.

> Anything that fiddles with the executable prevents signing the
> executable, which in many environments is desireable.

Ideally, we'd be able to re-sign after modification.

Re: Preview: portable dumper

[Robert Pluim]

Daniel Colascione <dancol@dancol.org> writes:

>
> We do need to avoid platform-specific executable file manipulation and
> weird runtime hacks. There are three options on the table:
>
> 1) Keeping the dump as a separate file,
> 2) Appending the dump to the end of the executable, and
> 3) Overwriting a data-section array with the dump contents.

If I remember correctly, XEmacs chose option (1), but that was a long
time ago. If we implement that, nothing prevents us from adding option
(2) later. Later builds that implement (2) would then simply not
provide the dump file separately, and once it's shown to work reliably
the code for (1) could be removed (or (1) and (2) could be swapped).

Option 3 seems less flexible, plus as someone mentioned it would be
nice to able to redump your own emacs, which could run into a
limitation of the size of the data section.

Robert

Re: Preview: portable dumper

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2241 bytes --]

Andy Moreton <andrewjmoreton@gmail.com> schrieb am Fr., 16. Feb. 2018 um
12:31 Uhr:

> On Fri 16 Feb 2018, Eli Zaretskii wrote:
>
> >> Date: Thu, 15 Feb 2018 15:34:13 -0800
> >> From: Daniel Colascione <dancol@dancol.org>
> >> Cc: Eli Zaretskii <eliz@gnu.org>, Angelo Graziosi <angelo.g0@libero.it
> >,
> >>  emacs-devel@gnu.org
> >>
> >> I do wonder whether it makes sense to try to copy the dump into the
> Emacs executable itself instead of
> >> leaving it as a separate file. We could do it independently of
> executable format by defining a data array in static
> >> storage that's initially full of, say, 15MB of zeroes prefixed by a
> long random header (like a MIME boundary),
> >> then, after we generate emacs.pdmp, copying the dump file into the
> executable at the place where we see
> >> that random header. If the dump turns out to be bigger than that 15MB,
> we can fail the build and ask the user
> >> to enlarge the array.
> >>
> >> I don't know of any executable format for which this scheme would fail.
> >
> > Wouldn't that make the dumper stuff less portable, in the sense that
> > it would need to be compatible with low-level details of executable
> > file formats on various systems?
> >
> > At least on non-ELF systems, AFAIK the flexibility of putting
> > arbitrary sections into an executable is lower than desired.  For
> > example, before Emacs 25 the MS-Windows build would create a special
> > section for the initialized Emacs data, which had the annoying effect
> > of running afoul of 'strip', because Binutils don't know about this
> > section, and therefore stripping would produce a dysfunctional
> > executable.  It also prevented re-dumping Emacs, something we had in
> > the past and I'd like us to have again in the future.
> >
> > Wouldn't copying the dump into the executable hit the same problems,
> > at least in principle?
>
> I don't understand the desire to put the dump within the eamcs
> executable, as I thought the whole point of this exercise was to avoid
> dodgy manipulation of executable file formats.
>

I agree. We already ship a lot of files that are expected at certain
locations, like the Emacs Lisp files or the files in `data-directory', why
should the pdump file be different?

[-- Attachment #2: Type: text/html, Size: 3018 bytes --]

Re: Preview: portable dumper

[Daniel Colascione]

On 02/16/2018 09:00 AM, Philipp Stephani wrote:
> Andy Moreton <andrewjmoreton@gmail.com 
> <mailto:andrewjmoreton@gmail.com>> schrieb am Fr., 16. Feb. 2018 um 
> 12:31 Uhr:
> 
>     On Fri 16 Feb 2018, Eli Zaretskii wrote:
> 
>      >> Date: Thu, 15 Feb 2018 15:34:13 -0800
>      >> From: Daniel Colascione <dancol@dancol.org
>     <mailto:dancol@dancol.org>>
>      >> Cc: Eli Zaretskii <eliz@gnu.org <mailto:eliz@gnu.org>>, Angelo
>     Graziosi <angelo.g0@libero.it <mailto:angelo.g0@libero.it>>,
>      >> emacs-devel@gnu.org <mailto:emacs-devel@gnu.org>
>      >>
>      >> I do wonder whether it makes sense to try to copy the dump into
>     the Emacs executable itself instead of
>      >> leaving it as a separate file. We could do it independently of
>     executable format by defining a data array in static
>      >> storage that's initially full of, say, 15MB of zeroes prefixed
>     by a long random header (like a MIME boundary),
>      >> then, after we generate emacs.pdmp, copying the dump file into
>     the executable at the place where we see
>      >> that random header. If the dump turns out to be bigger than that
>     15MB, we can fail the build and ask the user
>      >> to enlarge the array.
>      >>
>      >> I don't know of any executable format for which this scheme
>     would fail.
>      >
>      > Wouldn't that make the dumper stuff less portable, in the sense that
>      > it would need to be compatible with low-level details of executable
>      > file formats on various systems?
>      >
>      > At least on non-ELF systems, AFAIK the flexibility of putting
>      > arbitrary sections into an executable is lower than desired.  For
>      > example, before Emacs 25 the MS-Windows build would create a special
>      > section for the initialized Emacs data, which had the annoying effect
>      > of running afoul of 'strip', because Binutils don't know about this
>      > section, and therefore stripping would produce a dysfunctional
>      > executable.  It also prevented re-dumping Emacs, something we had in
>      > the past and I'd like us to have again in the future.
>      >
>      > Wouldn't copying the dump into the executable hit the same problems,
>      > at least in principle?
> 
>     I don't understand the desire to put the dump within the eamcs
>     executable, as I thought the whole point of this exercise was to avoid
>     dodgy manipulation of executable file formats.
> 
> 
> I agree. We already ship a lot of files that are expected at certain 
> locations, like the Emacs Lisp files or the files in `data-directory', 
> why should the pdump file be different?

The files you've mentioned aren't as intimately tied to the Emacs binary 
as the pdmp file is. You can load these elc and data files into any old 
Emacs, but the dump has to be an _exact_ match.

Re: Preview: portable dumper

[Daniel Colascione]

On 02/16/2018 12:24 AM, Eli Zaretskii wrote:
>> Date: Thu, 15 Feb 2018 15:34:13 -0800
>> From: Daniel Colascione <dancol@dancol.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>, Angelo Graziosi <angelo.g0@libero.it>,
>>   emacs-devel@gnu.org
>>
>> I do wonder whether it makes sense to try to copy the dump into the Emacs executable itself instead of
>> leaving it as a separate file. We could do it independently of executable format by defining a data array in static
>> storage that's initially full of, say, 15MB of zeroes prefixed by a long random header (like a MIME boundary),
>> then, after we generate emacs.pdmp, copying the dump file into the executable at the place where we see
>> that random header. If the dump turns out to be bigger than that 15MB, we can fail the build and ask the user
>> to enlarge the array.
>>
>> I don't know of any executable format for which this scheme would fail.
> 
> Wouldn't that make the dumper stuff less portable, in the sense that
> it would need to be compatible with low-level details of executable
> file formats on various systems?

No: I'm thinking about only options that work independently of 
particular executable formats.

> At least on non-ELF systems, AFAIK the flexibility of putting
> arbitrary sections into an executable is lower than desired.  For
> example, before Emacs 25 the MS-Windows build would create a special
> section for the initialized Emacs data, which had the annoying effect
> of running afoul of 'strip', because Binutils don't know about this
> section, and therefore stripping would produce a dysfunctional
> executable.  

That does sound annoying. The original proposal I had in mind was to 
make a normal data array:

uint8_t dump[15*1024*1024] = { embedded_dump_signature };

And then refer to this array on code. No part of the toolchain is 
allowed to remove this array. The "dump insertion" procedure we're 
considering would just amount to changing the contents of this array in 
an already-linked executable using dumb find-and-replace anchored on 
embedded_dump_signature.

> It also prevented re-dumping Emacs, something we had in
> the past and I'd like us to have again in the future.

I'd like to have it too. The data-section-array approach above wouldn't 
permit redumping unless the new dump fit into the old array. The 
concatenation approach would allow arbitrary redumping. And of course 
keeping the dump as a separate file would too.

> Wouldn't copying the dump into the executable hit the same problems,
> at least in principle?
> 
>> This way, we could get rid of the current complicated dump-file location logic early in Emacs init.
> 
> Can you elaborate why is that logic so complicated?

Two reasons:

1) The code to generate the dump file is complicated, especially because 
we want to support windows, and
2) with a separate dump file, it's easier to accidentally break the 
correspondence between the dump and the emacs binary itself. If we 
somehow stapled the dump to the binary, it'd be a lot harder to break 
this way.

I like the idea of being able to mv emacs emacs.old, make emacs, and 
then have emacs.old and emacs still both run and work normally. I don't 
think the numbered build mechanism you fixed completely solves the 
problem, since it only works in one specific context.

Re: Preview: portable dumper

[Eli Zaretskii]

> Cc: alan@idiocy.org, angelo.g0@libero.it, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Fri, 16 Feb 2018 07:10:42 -0800
> 
> The original proposal I had in mind was to make a normal data array:
> 
> uint8_t dump[15*1024*1024] = { embedded_dump_signature };
> 
> And then refer to this array on code. No part of the toolchain is 
> allowed to remove this array. The "dump insertion" procedure we're 
> considering would just amount to changing the contents of this array in 
> an already-linked executable using dumb find-and-replace anchored on 
> embedded_dump_signature.

I can understand how rewriting the array could work (but we'd need to
know its size in advance, otherwise we'd waste memory, like Emacs on
w32 does today with its dumped_data array, especially since the size
during bootstrap needs to be so much larger than with byte-compiled
Lisp files).  But I'm not sure I understand how references to that
array in the code would work.  Do you mean there would be a need to
"fix up" address references in the code as part of "dump insertion"?
If so, that's my original suggestion from 2 years ago, which was voted
down due to complexities.

> I like the idea of being able to mv emacs emacs.old, make emacs, and 
> then have emacs.old and emacs still both run and work normally. I don't 
> think the numbered build mechanism you fixed completely solves the 
> problem, since it only works in one specific context.

I imagined the fingerprint we compute is already used (or can be used)
to reject dump files from a different build?  If so, this could
provide at least some protection form calamities.

Re: Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/html, Size: 3765 bytes --]

Re: Preview: portable dumper

[Eli Zaretskii]

> Date: Fri, 16 Feb 2018 07:44:50 -0800
> From: Daniel Colascione <dancol@dancol.org>
> Cc: alan@idiocy.org, angelo.g0@libero.it, emacs-devel@gnu.org
> 
> That the dump-array proposal caps
> the size of the dump (which will probably break user redumping) and that strip(1) breaks the concatenation
> approach suggests that maybe the existing separate file option is right after all.

I do indeed think that we should make the separate file method stable
and collect some experience with it, before consider more advanced
and/or clever approaches.

Re: Preview: portable dumper

[Stefan Monnier]

> That the dump-array proposal caps the size of the dump (which will
> probably break user redumping) and that strip(1) breaks the
> concatenation approach suggests that maybe the existing separate file
> option is right after all.

IIUC these options actually aren't mutually exclusive.
We can easily provide a single `(t)emacs` executable which at startup
will:
- look for a valid dump in the big static `dump` array.
- if not found look for a valid dump in a particular ELF section.
- if not found, look for a valid dump at the end of the `emacs` binary.
- if not found, look for a dump as a separate file at some standard
  location (starting with ~/.emacs.d/<magicid>.dump, say).
- as support a "--preload <foo>.dump" command-line option.


        Stefan

Re: Preview: portable dumper

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

Alan Third <alan@idiocy.org> schrieb am Do., 15. Feb. 2018 um 21:36 Uhr:

> On Thu, Feb 15, 2018 at 11:49:30AM -0800, Daniel Colascione wrote:
> > Thanks for checking. If you disable Wconversion at the top of the
> > file, you should be able to unblock yourself. The warning helps make
> > the code robust, but it's annoying to account for compiler
> > differences.
>
> Thanks, I’ve got it to build and it seems to work fine.
>
> Here are the changes I had to make:
>
> 3 files changed, 15 insertions(+), 15 deletions(-)
> src/dispnew.c |  2 +-
> src/macfont.m | 26 +++++++++++++-------------
> src/pdumper.c |  2 +-
>

Another minor thing (that breaks the build with -Werror) is that the
functions set_{marker,vectorlike}_marked are unused. Same with
syms_of_nsfns_1. I also had to change the eassert(("...", 0)) to
eassert("..." && 0) to stop the compiler from complaining about an unused
expression.

[-- Attachment #2: Type: text/html, Size: 1264 bytes --]

Re: Preview: portable dumper

[Philipp Stephani]

[-- Attachment #1.1: Type: text/plain, Size: 1089 bytes --]

Philipp Stephani <p.stephani2@gmail.com> schrieb am Do., 15. Feb. 2018 um
23:38 Uhr:

> Alan Third <alan@idiocy.org> schrieb am Do., 15. Feb. 2018 um 21:36 Uhr:
>
>> On Thu, Feb 15, 2018 at 11:49:30AM -0800, Daniel Colascione wrote:
>> > Thanks for checking. If you disable Wconversion at the top of the
>> > file, you should be able to unblock yourself. The warning helps make
>> > the code robust, but it's annoying to account for compiler
>> > differences.
>>
>> Thanks, I’ve got it to build and it seems to work fine.
>>
>> Here are the changes I had to make:
>>
>> 3 files changed, 15 insertions(+), 15 deletions(-)
>> src/dispnew.c |  2 +-
>> src/macfont.m | 26 +++++++++++++-------------
>> src/pdumper.c |  2 +-
>>
>
> Another minor thing (that breaks the build with -Werror) is that the
> functions set_{marker,vectorlike}_marked are unused. Same with
> syms_of_nsfns_1. I also had to change the eassert(("...", 0)) to
> eassert("..." && 0) to stop the compiler from complaining about an unused
> expression.
>

This is the patch that I've applied:

[-- Attachment #1.2: Type: text/html, Size: 1726 bytes --]

[-- Attachment #2: pdump-patch.txt --]
[-- Type: text/plain, Size: 3182 bytes --]

diff --git a/src/alloc.c b/src/alloc.c
index 90f913da36..6f81195cc3 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -378,7 +378,6 @@ inline static bool vector_marked_p (const struct Lisp_Vector *v);
 inline static void set_vector_marked (struct Lisp_Vector *v);
 
 inline static bool vectorlike_marked_p (const union vectorlike_header *v);
-inline static void set_vectorlike_marked (union vectorlike_header *v);
 
 inline static bool cons_marked_p (const struct Lisp_Cons *c);
 inline static void set_cons_marked (struct Lisp_Cons *c);
@@ -393,7 +392,6 @@ inline static bool misc_any_marked_p (const struct Lisp_Misc_Any *m);
 inline static void set_misc_any_marked (struct Lisp_Misc_Any *m);
 
 inline static bool marker_marked_p (const struct Lisp_Marker *m);
-inline static void set_marker_marked (struct Lisp_Marker *m);
 
 inline static bool overlay_marked_p (const struct Lisp_Overlay *m);
 inline static void set_overlay_marked (struct Lisp_Overlay *m);
@@ -4150,12 +4148,6 @@ vectorlike_marked_p (const union vectorlike_header *header)
   return vector_marked_p ((const struct Lisp_Vector *) header);
 }
 
-static void
-set_vectorlike_marked (union vectorlike_header *header)
-{
-  set_vector_marked ((struct Lisp_Vector *) header);
-}
-
 static bool
 cons_marked_p (const struct Lisp_Cons *c)
 {
@@ -4230,12 +4222,6 @@ marker_marked_p (const struct Lisp_Marker *m)
   return misc_any_marked_p ((struct Lisp_Misc_Any *) m);
 }
 
-static void
-set_marker_marked (struct Lisp_Marker *m)
-{
-  set_misc_any_marked ((struct Lisp_Misc_Any *) m);
-}
-
 static bool
 overlay_marked_p (const struct Lisp_Overlay *m)
 {
diff --git a/src/macfont.m b/src/macfont.m
index 377fc3fa48..2cbe07d751 100644
--- a/src/macfont.m
+++ b/src/macfont.m
@@ -1646,7 +1646,7 @@ static int macfont_variation_glyphs (struct font *, int c,
                                      unsigned variations[256]);
 static void macfont_filter_properties (Lisp_Object, Lisp_Object);
 
-static struct font_driver const macfont_driver =
+static struct font_driver macfont_driver =
   {
   .type = LISPSYM_INITIALLY (Qmac_ct),
   .get_cache = macfont_get_cache,
@@ -4034,6 +4034,8 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
 
 \f
 
+static void syms_of_macfont_for_pdumper (void);
+
 void
 syms_of_macfont (void)
 {
diff --git a/src/nsfns.m b/src/nsfns.m
index 8911ad27fe..c2ee4df33f 100644
--- a/src/nsfns.m
+++ b/src/nsfns.m
@@ -3196,12 +3196,6 @@ - (NSString *)panel: (id)sender userEnteredFilename: (NSString *)filename
 
    ========================================================================== */
 
-static void
-syms_of_nsfns_1 (void)
-{
-
-}
-
 void
 syms_of_nsfns (void)
 {
diff --git a/src/pdumper.c b/src/pdumper.c
index 499c8dc27c..7053e52ad6 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -2917,7 +2917,7 @@ dump_object_1 (struct dump_context *ctx, Lisp_Object object)
           offset = dump_float (ctx, XFLOAT (object));
           break;
         case_Lisp_Int:
-          eassert (("should not be dumping int: is self-representing", 0));
+          eassert ("should not be dumping int: is self-representing" && false);
         default:
           emacs_abort ();
         }

Re: Preview: portable dumper

[Yoshiaki Kasahara]

Hello,

I cannot build pdumper branch on FreeBSD 11.1-STABLE with gcc 6.4.0.
I guess INLINE macro already contains 'static' on FreeBSD.

  CC       alloc.o
alloc.c:377:1: error: duplicate 'static'
 INLINE static bool vector_marked_p (const struct Lisp_Vector *v);
 ^~~~~~
alloc.c:378:1: error: duplicate 'static'
 INLINE static void set_vector_marked (struct Lisp_Vector *v);
 ^~~~~~
alloc.c:380:1: error: duplicate 'static'
 INLINE static bool vectorlike_marked_p (const union vectorlike_header *v);
 ^~~~~~
alloc.c:381:1: error: duplicate 'static'
 INLINE static void set_vectorlike_marked (union vectorlike_header *v);
 ^~~~~~
alloc.c:383:1: error: duplicate 'static'
 INLINE static bool cons_marked_p (const struct Lisp_Cons *c);
 ^~~~~~
alloc.c:384:1: error: duplicate 'static'
 INLINE static void set_cons_marked (struct Lisp_Cons *c);
 ^~~~~~
alloc.c:386:1: error: duplicate 'static'
 INLINE static bool string_marked_p (const struct Lisp_String *s);
 ^~~~~~
alloc.c:387:1: error: duplicate 'static'
 INLINE static void set_string_marked (struct Lisp_String *s);
 ^~~~~~
alloc.c:389:1: error: duplicate 'static'
 INLINE static bool symbol_marked_p (const struct Lisp_Symbol *s);
 ^~~~~~
alloc.c:390:1: error: duplicate 'static'
 INLINE static void set_symbol_marked (struct Lisp_Symbol *s);
 ^~~~~~
alloc.c:392:1: error: duplicate 'static'
 INLINE static bool misc_any_marked_p (const struct Lisp_Misc_Any *m);
 ^~~~~~
alloc.c:393:1: error: duplicate 'static'
 INLINE static void set_misc_any_marked (struct Lisp_Misc_Any *m);
 ^~~~~~
alloc.c:395:1: error: duplicate 'static'
 INLINE static bool marker_marked_p (const struct Lisp_Marker *m);
 ^~~~~~
alloc.c:396:1: error: duplicate 'static'
 INLINE static void set_marker_marked (struct Lisp_Marker *m);
 ^~~~~~
alloc.c:398:1: error: duplicate 'static'
 INLINE static bool overlay_marked_p (const struct Lisp_Overlay *m);
 ^~~~~~
alloc.c:399:1: error: duplicate 'static'
 INLINE static void set_overlay_marked (struct Lisp_Overlay *m);
 ^~~~~~
alloc.c:401:1: error: duplicate 'static'
 INLINE static bool interval_marked_p (INTERVAL i);
 ^~~~~~
alloc.c:402:1: error: duplicate 'static'
 INLINE static void set_interval_marked (INTERVAL i);
 ^~~~~~
gmake[2]: *** [Makefile:386: alloc.o] Error 1

Regards,
-- 
Yoshiaki Kasahara
kasahara@nc.kyushu-u.ac.jp

Re: Preview: portable dumper

[Daniel Colascione]

On 02/14/2018 11:44 PM, Yoshiaki Kasahara wrote:
> Hello,
> 
> I cannot build pdumper branch on FreeBSD 11.1-STABLE with gcc 6.4.0.
> I guess INLINE macro already contains 'static' on FreeBSD.

Should be fixed.

Re: Preview: portable dumper

[Yoshiaki Kasahara]

On Thu, 15 Feb 2018 14:17:55 -0800,
	Daniel Colascione <dancol@dancol.org> said:

> On 02/14/2018 11:44 PM, Yoshiaki Kasahara wrote:
>> Hello,
>> I cannot build pdumper branch on FreeBSD 11.1-STABLE with gcc 6.4.0.
>> I guess INLINE macro already contains 'static' on FreeBSD.
> 
> Should be fixed.

Thank you.  I could build it and it seems working as expected.  I'm
writing this message using pdumper emacs.

Regards,
-- 
Yoshiaki Kasahara
kasahara@nc.kyushu-u.ac.jp

Re: Preview: portable dumper

[Andreas Schwab]

Doesn't work on powerpc:

../../emacs/src/pdumper.c:2479: Emacs fatal error: assertion failed: vector_nbytes ((struct Lisp_Vector *) in) == out_size

(gdb) p out_size
$1 = 536
(gdb) p vector_nbytes ((struct Lisp_Vector *) in)
$2 = 544

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Preview: portable dumper

[Daniel Colascione]

On 02/17/2018 02:31 AM, Andreas Schwab wrote:
> Doesn't work on powerpc:
> 
> ../../emacs/src/pdumper.c:2479: Emacs fatal error: assertion failed: vector_nbytes ((struct Lisp_Vector *) in) == out_size
> 
> (gdb) p out_size
> $1 = 536
> (gdb) p vector_nbytes ((struct Lisp_Vector *) in)
> $2 = 544
> 
> Andreas.
> 

Thanks for testing. I don't have a PowerPC handy. I guess I can go find 
an emulator, but in the meantime: can you think of anything specific to 
PowerPC that might break the code? Word size? pdumper works for me in 
both 32- and 64-bit forms on x86. Endian? AIUI, PowerPC has selectable 
endianness. Are you big- or little-endian? Besides, pdumper *should* be 
endian-neutral. And if we were looking at an endianness mismatch, I'd 
expect a much bigger mismatch than 544 vs. 536. What stack do you have 
there? Does your system use natural alignment?

Re: Preview: portable dumper

[Andreas Schwab]

On Feb 19 2018, Daniel Colascione <dancol@dancol.org> wrote:

> pdumper works for me in both 32- and 64-bit forms on x86.

That's pure luck.  There is nothing that forces the size of struct
buffer to be EMACS_INT aligned.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/html, Size: 708 bytes --]

Re: Preview: portable dumper

[Andreas Schwab]

On Feb 19 2018, Daniel Colascione <dancol@dancol.org> wrote:

> <div dir='auto'><div><div class="gmail_extra"><div class="gmail_quote">On Feb 19, 2018 12:39 PM, Andreas Schwab &lt;schwab@linux-m68k.org&gt; wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">On Feb 19 2018, Daniel Colascione &lt;dancol@dancol.org&gt; wrote:
> <br>
>
> <br>
> &gt; pdumper works for me in both 32- and 64-bit forms on x86.
> <br>
>
> <br>
> That's pure luck.&nbsp; There is nothing that forces the size of struct
> <br>
> buffer to be EMACS_INT aligned.
> <br></p></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">No, there isn't. That's why the code generally doesn't require that it be.</div></div>

Then you need to fix that.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Preview: portable dumper

[Daniel Colascione]

On 02/19/2018 01:41 PM, Andreas Schwab wrote:
> On Feb 19 2018, Daniel Colascione <dancol@dancol.org> wrote:
> 
>> <div dir='auto'><div><div class="gmail_extra"><div class="gmail_quote">On Feb 19, 2018 12:39 PM, Andreas Schwab &lt;schwab@linux-m68k.org&gt; wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">On Feb 19 2018, Daniel Colascione &lt;dancol@dancol.org&gt; wrote:
>> <br>
>>
>> <br>
>> &gt; pdumper works for me in both 32- and 64-bit forms on x86.
>> <br>
>>
>> <br>
>> That's pure luck.&nbsp; There is nothing that forces the size of struct
>> <br>
>> buffer to be EMACS_INT aligned.
>> <br></p></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">No, there isn't. That's why the code generally doesn't require that it be.</div></div>
> 
> Then you need to fix that.

I will, but I don't understand why you've been so unhelpful.

Re: Preview: portable dumper

[Jacob Bachmeyer]

>
> (Or we could just randomize per-user and dump Emacs the first time it
> runs for a particular user?  If we do that after loading ~/.emacs, we
> also improve people's startup time.  Invalidating and regenerating the
> dump when configuration changes would be a challenge though.)

That should not be too difficult, if you can track which files were read 
when creating the dump and store some fields from the stat(2) 
information on those files in the dump.  I am using this approach in a 
packaging system that I am developing to close a race between attaching 
a file to an archive handle and actually writing the archive, at which 
time the digest of the file is computed.  (I wanted to avoid reading 
input files twice.)

I take a conservative approach and verify that the 
st_{ino,dev,size,blocks,{m,c}tim{e,.tv_nsec}} fields are all unchanged.  
For my use, writing the archive produces a hard failure if this check 
fails; for Emacs, failing that check would indicate "time to rebuild the 
fast-load cache".


On the other hand, I think that per-user dumps are a bad idea--the Emacs 
dump is an inscrutable binary blob and therefore a good place for an 
intruder to hide persistent nastiness.  This could allow an intruder to 
add a back door to a user's Emacs in a difficult-to-detect manner while 
needing only temporary access to that user's account, say, from 
exploiting any program that user runs.


-- Jacob

Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Dec 06 2016, Jacob Bachmeyer wrote:
>>
>> (Or we could just randomize per-user and dump Emacs the first time it
>> runs for a particular user?  If we do that after loading ~/.emacs, we
>> also improve people's startup time.  Invalidating and regenerating the
>> dump when configuration changes would be a challenge though.)
>
> That should not be too difficult, if you can track which files were
> read when creating the dump and store some fields from the stat(2)
> information on those files in the dump.  I am using this approach in a
> packaging system that I am developing to close a race between
> attaching a file to an archive handle and actually writing the
> archive, at which time the digest of the file is computed.  (I wanted
> to avoid reading input files twice.)
>
> I take a conservative approach and verify that the
> st_{ino,dev,size,blocks,{m,c}tim{e,.tv_nsec}} fields are all
> unchanged.  For my use, writing the archive produces a hard failure if
> this check fails; for Emacs, failing that check would indicate "time
> to rebuild the fast-load cache".
>
>
> On the other hand, I think that per-user dumps are a bad idea--the
> Emacs dump is an inscrutable binary blob

Users run lots of inscrutable binary blobs.  At least this one is made
from free software.  ("Sure", you might think, "we can just have the
system Emacs *sign* the blob." But an attacker could just read the
private key right out of the Emacs binary.  You really can't win.)

> and therefore a good place
> for an intruder to hide persistent nastiness.  This could allow an
> intruder to add a back door to a user's Emacs in a difficult-to-detect
> manner while needing only temporary access to that user's account,
> say, from exploiting any program that user runs.

I don't think attempting to defend against this sort of attack, at least
the way you suggest, is desirable.  An attacker who can modify user
files like that has already won --- there are all sorts of user-mode
"rootkits" that hide themselves very effectively.

https://blogs.msdn.microsoft.com/oldnewthing/20060508-22/?p=31283

Re: Preview: portable dumper

[Jacob Bachmeyer]

Daniel Colascione wrote:
> On Tue, Dec 06 2016, Jacob Bachmeyer wrote:
>   
>>> (Or we could just randomize per-user and dump Emacs the first time it
>>> runs for a particular user?  If we do that after loading ~/.emacs, we
>>> also improve people's startup time.  Invalidating and regenerating the
>>> dump when configuration changes would be a challenge though.)
>>>       
>> That should not be too difficult, if you can track which files were
>> read when creating the dump and store some fields from the stat(2)
>> information on those files in the dump.  I am using this approach in a
>> packaging system that I am developing to close a race between
>> attaching a file to an archive handle and actually writing the
>> archive, at which time the digest of the file is computed.  (I wanted
>> to avoid reading input files twice.)
>>
>> I take a conservative approach and verify that the
>> st_{ino,dev,size,blocks,{m,c}tim{e,.tv_nsec}} fields are all
>> unchanged.  For my use, writing the archive produces a hard failure if
>> this check fails; for Emacs, failing that check would indicate "time
>> to rebuild the fast-load cache".
>>
>>
>> On the other hand, I think that per-user dumps are a bad idea--the
>> Emacs dump is an inscrutable binary blob
>>     
>
> Users run lots of inscrutable binary blobs.  At least this one is made
> from free software.  ("Sure", you might think, "we can just have the
> system Emacs *sign* the blob." But an attacker could just read the
> private key right out of the Emacs binary.  You really can't win.)
>   

"A strange game.  The only winning move is not to play."

The system Emacs dump cannot be modified by a user's account.  A 
per-user dump could be.  Currently, this particular persistence option 
for a (user-level) intruder does not exist; I argue that we should think 
very carefully before creating it.  (No objection to using this for the 
system Emacs dump, though.)

A counter-proposal to signing the dump:  upon loading the dump, display 
a digest or randomart-like image that summarizes the dump.  Users who 
care can then notice if the dump changes when it should not.

>> and therefore a good place
>> for an intruder to hide persistent nastiness.  This could allow an
>> intruder to add a back door to a user's Emacs in a difficult-to-detect
>> manner while needing only temporary access to that user's account,
>> say, from exploiting any program that user runs.
>>     
>
> I don't think attempting to defend against this sort of attack, at least
> the way you suggest, is desirable.  An attacker who can modify user
> files like that has already won --- there are all sorts of user-mode
> "rootkits" that hide themselves very effectively.
>
> https://blogs.msdn.microsoft.com/oldnewthing/20060508-22/?p=31283
>   

Some of the comments on that blog are appropriate here.  This kind of 
"not a security problem because <insert excuse here>" is a position for 
which Microsoft is infamous.  The issue that I see is that a per-user 
Emacs dump (which of course must be writable by the user) is a place 
(platform-independent, even!) where an attacker can hide a bit of code 
to make future access easier, even after the exploit that initially 
allowed the attacker to get in is fixed.  I consider that an attacker 
who gets in, but leaves traces and is detected, has "won the battle and 
lost the war".


-- Jacob

Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Dec 06 2016, Jacob Bachmeyer wrote:
> Daniel Colascione wrote:
>> On Tue, Dec 06 2016, Jacob Bachmeyer wrote:
>>   
>>>> (Or we could just randomize per-user and dump Emacs the first time it
>>>> runs for a particular user?  If we do that after loading ~/.emacs, we
>>>> also improve people's startup time.  Invalidating and regenerating the
>>>> dump when configuration changes would be a challenge though.)
>>>>       
>>> That should not be too difficult, if you can track which files were
>>> read when creating the dump and store some fields from the stat(2)
>>> information on those files in the dump.  I am using this approach in a
>>> packaging system that I am developing to close a race between
>>> attaching a file to an archive handle and actually writing the
>>> archive, at which time the digest of the file is computed.  (I wanted
>>> to avoid reading input files twice.)
>>>
>>> I take a conservative approach and verify that the
>>> st_{ino,dev,size,blocks,{m,c}tim{e,.tv_nsec}} fields are all
>>> unchanged.  For my use, writing the archive produces a hard failure if
>>> this check fails; for Emacs, failing that check would indicate "time
>>> to rebuild the fast-load cache".
>>>
>>>
>>> On the other hand, I think that per-user dumps are a bad idea--the
>>> Emacs dump is an inscrutable binary blob
>>>     
>>
>> Users run lots of inscrutable binary blobs.  At least this one is made
>> from free software.  ("Sure", you might think, "we can just have the
>> system Emacs *sign* the blob." But an attacker could just read the
>> private key right out of the Emacs binary.  You really can't win.)
>>   
>
> "A strange game.  The only winning move is not to play."
>
> The system Emacs dump cannot be modified by a user's account.
> A per-user dump could be.  Currently, this particular persistence
> option for a (user-level) intruder does not exist; I argue that we
> should think very carefully before creating it.  (No objection to
> using this for the system Emacs dump, though.)

And you don't have any binaries under $HOME? No elc files? No
.pyc files? Really? Are you sure that you can read a pyc file and
detect exploit code?

> A counter-proposal to signing the dump:  upon loading the dump,
> display a digest or randomart-like image that summarizes the dump.
> Users who care can then notice if the dump changes when it should not.

That'll annoy users, and I've never seen a real human being pay
attention to those SSH key fingerprint ASCII art things, much less some
weird art thing that Emacs might display on startup.

>>> and therefore a good place
>>> for an intruder to hide persistent nastiness.  This could allow an
>>> intruder to add a back door to a user's Emacs in a difficult-to-detect
>>> manner while needing only temporary access to that user's account,
>>> say, from exploiting any program that user runs.
>>>     
>>
>> I don't think attempting to defend against this sort of attack, at least
>> the way you suggest, is desirable.  An attacker who can modify user
>> files like that has already won --- there are all sorts of user-mode
>> "rootkits" that hide themselves very effectively.
>>
>> https://blogs.msdn.microsoft.com/oldnewthing/20060508-22/?p=31283
>>   
>
> Some of the comments on that blog are appropriate here.  This kind of
> "not a security problem because <insert excuse here>" is a position
> for which Microsoft is infamous.

That the author of that post works for Microsoft is irrelevant.

"Security" is a meaningless word.  You need to be secure *against*
something.  You need a threat model.  Your threat model is an attacker
who already has the ability to overwrite files under $HOME.  Such an
attacker can do anything anyway (e.g., LD_PRELOAD something in
~/.profile that presents whatever secure fantasy the attacker wants to
present), so your proposed security measure doesn't actually help ---
and it has costs.

Sorry: I don't think that forgoing user dumps on security grounds makes
any sense.  There might be plenty of other reasons not to implement the
feature, but security isn't one of them.

That's not "an excuse" --- as if implementing any random feature
somebody proposes in the name of "security" is some moral imperative, we
need to be excused from our duty to perform it --- it's reason.

> The issue that I see is that a
> per-user Emacs dump (which of course must be writable by the user) is
> a place (platform-independent, even!) where an attacker can hide a bit
> of code to make future access easier, even after the exploit that
> initially allowed the attacker to get in is fixed.

System compromises are never "fixed": there are too many places for
attackers to hide code.  If your account has been compromised, you lose.
Restore from backup.

Re: Preview: portable dumper

[Jacob Bachmeyer]

Daniel Colascione wrote:
> On Tue, Dec 06 2016, Jacob Bachmeyer wrote:
>   
>> Daniel Colascione wrote:
>>     
>>> On Tue, Dec 06 2016, Jacob Bachmeyer wrote:
>>>   
>>>       
>>>>> (Or we could just randomize per-user and dump Emacs the first time it
>>>>> runs for a particular user?  If we do that after loading ~/.emacs, we
>>>>> also improve people's startup time.  Invalidating and regenerating the
>>>>> dump when configuration changes would be a challenge though.)
>>>>>       
>>>>>           
>>>> That should not be too difficult, if you can track which files were
>>>> read when creating the dump and store some fields from the stat(2)
>>>> information on those files in the dump.  I am using this approach in a
>>>> packaging system that I am developing to close a race between
>>>> attaching a file to an archive handle and actually writing the
>>>> archive, at which time the digest of the file is computed.  (I wanted
>>>> to avoid reading input files twice.)
>>>>
>>>> I take a conservative approach and verify that the
>>>> st_{ino,dev,size,blocks,{m,c}tim{e,.tv_nsec}} fields are all
>>>> unchanged.  For my use, writing the archive produces a hard failure if
>>>> this check fails; for Emacs, failing that check would indicate "time
>>>> to rebuild the fast-load cache".
>>>>
>>>>
>>>> On the other hand, I think that per-user dumps are a bad idea--the
>>>> Emacs dump is an inscrutable binary blob
>>>>     
>>>>         
>>> Users run lots of inscrutable binary blobs.  At least this one is made
>>> from free software.  ("Sure", you might think, "we can just have the
>>> system Emacs *sign* the blob." But an attacker could just read the
>>> private key right out of the Emacs binary.  You really can't win.)
>>>   
>>>       
>> "A strange game.  The only winning move is not to play."
>>
>> The system Emacs dump cannot be modified by a user's account.
>> A per-user dump could be.  Currently, this particular persistence
>> option for a (user-level) intruder does not exist; I argue that we
>> should think very carefully before creating it.  (No objection to
>> using this for the system Emacs dump, though.)
>>     
>
> And you don't have any binaries under $HOME? No elc files? No
> .pyc files? Really? Are you sure that you can read a pyc file and
> detect exploit code?
>   

A multi-user secure environment may forbid binaries in $HOME entirely 
and mount /home noexec.

>> A counter-proposal to signing the dump:  upon loading the dump,
>> display a digest or randomart-like image that summarizes the dump.
>> Users who care can then notice if the dump changes when it should not.
>>     
>
> That'll annoy users, and I've never seen a real human being pay
> attention to those SSH key fingerprint ASCII art things, much less some
> weird art thing that Emacs might display on startup.
>   

It is sad how many people just do not care.  Then they complain when 
told that their email account is sending spam.

Back to the topic, how would this annoy users?  Put it in the splash 
screen with other details like the Emacs version that the willfully 
ignorant will ignore.
"This is GNU Emacs <version> (<platform>) of <date> on <system> using 
image <digest of loaded dump>"

>>>> and therefore a good place
>>>> for an intruder to hide persistent nastiness.  This could allow an
>>>> intruder to add a back door to a user's Emacs in a difficult-to-detect
>>>> manner while needing only temporary access to that user's account,
>>>> say, from exploiting any program that user runs.
>>>>     
>>>>         
>>> I don't think attempting to defend against this sort of attack, at least
>>> the way you suggest, is desirable.  An attacker who can modify user
>>> files like that has already won --- there are all sorts of user-mode
>>> "rootkits" that hide themselves very effectively.
>>>
>>> https://blogs.msdn.microsoft.com/oldnewthing/20060508-22/?p=31283
>>>   
>>>       
>> Some of the comments on that blog are appropriate here.  This kind of
>> "not a security problem because <insert excuse here>" is a position
>> for which Microsoft is infamous.
>>     
>
> That the author of that post works for Microsoft is irrelevant.
>
> "Security" is a meaningless word.  You need to be secure *against*
> something.  You need a threat model.  Your threat model is an attacker
> who already has the ability to overwrite files under $HOME.  Such an
> attacker can do anything anyway (e.g., LD_PRELOAD something in
> ~/.profile that presents whatever secure fantasy the attacker wants to
> present), so your proposed security measure doesn't actually help ---
> and it has costs.
>   

Modifying .profile is riskier for the attacker than modifying a binary 
dump.  A system backup facility, for example, will notice that the 
user's .profile has changed and will probably store the diff.  A 
security policy may have a system daemon scanning user .profiles for 
references to LD_PRELOAD.

> Sorry: I don't think that forgoing user dumps on security grounds makes
> any sense.  There might be plenty of other reasons not to implement the
> feature, but security isn't one of them.
>
> That's not "an excuse" --- as if implementing any random feature
> somebody proposes in the name of "security" is some moral imperative, we
> need to be excused from our duty to perform it --- it's reason.
>   

I did not say that we absolutely should not have per-user dumps, only 
that we should consider the risks that they introduce.  At the least, 
there should be an option to not use per-user dumps.  This would be very 
little of a problem, since the portable dump mechanism will allow for 
dumps to be incrementally built.  This will enable a user dump to be 
built/updated by loading the system dump, processing .emacs (or some 
special user loadup file if the user does not want .emacs in the dump) 
and saving a new dump file.  Or for user dumps to be unused, and we stay 
right where we are now on those systems, but without depending on the 
internals of libc.

>> The issue that I see is that a
>> per-user Emacs dump (which of course must be writable by the user) is
>> a place (platform-independent, even!) where an attacker can hide a bit
>> of code to make future access easier, even after the exploit that
>> initially allowed the attacker to get in is fixed.
>>     
>
> System compromises are never "fixed": there are too many places for
> attackers to hide code.  If your account has been compromised, you lose.
> Restore from backup.
>   

This comes back to an attacker "winning the battle and losing the 
war"--if you do not know your account has been compromised, then you do 
not know that you should be restoring from backup--and the attacker 
"wins the war".  Non-persistent "smash-and-grab" attacks have an 
important advantage for the attacker in this sense.  That integrity in 
Free environments is currently a mess is no reason to charge ahead 
without considering what effects our efforts will have on the situation.



I had meant to suggest a solution to the problem of invalidating a user 
dump/fast-load cache; I have been surprised that the discussion has been 
more about an offhand remark expressing uncertainty about the wisdom of 
per-user dump files.

To be clear:  I am in favor of supporting per-user dumps, but vehemently 
opposed to forcing their use.  If nothing else { alias emacs='emacs 
--load-dump=~/.emacs.pdmp'; } will provide support for per-user dumps 
whether or not Emacs itself checks for one.


-- Jacob

Re: Preview: portable dumper

[Reini Urban]

> > C programmers are a dying breed
> I don't agree, as C is an essential component of GNU/Linux and is 
> robustly represented in the practical and research programming 
> communities. (For what it's worth, C's popularity is ranked #7 in PyPL 
> and #2 in TIOBE.)

>When did you last try to hire a C or C++ programmer?

>As another data point, just look in the logs at the names of people
>who make C-level changes, and count them.

>I didn't say I like this, but leading a project to be viable in the
>future needs to look at the reality, not at our wishful thinking.

Maybe I should tell you, that interestingly Emacs is not the only project affected by this glibc/dumper dispute.

I added the stone-old unexec patches for perl to my cperl fork, to be able to dump/compile perl scripts to native binaries the fast way.

Improved dumpers are here: https://github.com/perl11/cperl/commits/feature/gh176-unexec 
I guess I’m the only one who touched this code in the last decades, nobody in emacs and nobody in perl.

Mostly unified error handling and a few darwin segment instabilities. It is very fragile to use with a static library, but ok as dynamic library. Emacs uses the dumper in the main exe, not in a library. Solaris is the easiest to use.

So I know a little bit of the troubles they are talking about here. Dan's portable dumper would be nice to have, XEmacs had this decades ago, but it never made it over here. Wonder why :)

So looking at the new pdump, it really is horrible. I don't think I want to do that. I'll rather add a proper static malloc to cperl, such ptmalloc3, which is better than glibc malloc, i.e. ptmalloc2, anyway. They never switched to the better version, because it had more memory overhead. And I really can make use of the arena support there. Emacs should try the same. Much easier and much faster. Good bye glibc.

Reini Urban
rurban@cpan.org

Preview: portable dumper

[Tobias Gerdin]

Since some years there has been the intention of switching to Guile for Emacs' Lisp needs. As I understand it one of the major issues to be resolved is the lack of Guile unexec support, so would it not be prudent to consider the implications for a (possible) future Guile Emacs? Something along the lines of what Eli proposed (the ”one big .elc”approach) is a requested feature by one of the Guile maintainers[1].

During the last few years there has been a fair amount of work put in to modernise Emacs Lisp (lexical binding probably being the most notable accomplishment). Still, Emacs Lisp has a long way to go to match a high-quality Scheme implementation.
So I am wondering if the time used to play catch-up with modern Lisps would not be better spent by integrating Guile? There seems to have been a lot of focus on performance in Guile the last few years and it also comes with good concurrency support (which has been another area of focus in Emacs). Recently another concurrency mechanism which may be a better fit than threads has also appeared in Guile [2]. There has also been discussion of making it easier for recruit new Emacs developers. I believe that adopting Guile and Scheme would be the single most important step towards this aim. 

Given the number of talented hackers in the Emacs and Guile community I would say that making Guile-Emacs a reality is a definitely possible. However, it would probably require to work together actively towards this goal, as opposed to fragmented hacking by single developers.

-Tobias

[1]: https://wingolog.org/archives/2016/02/04/guile-compiler-tasks
[2]: https://github.com/wingo/fibers

Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2898 bytes --]

I've been working on a portable dumper for GNU Emacs. The attached patch 
is an early version of this work. I'll write up all the usual NEWS 
entries, changelogs, and (in this case necessary) dedicated 
documentation before I land it. I want to start getting review comments 
now that the code has roughly its final shape.

The point of this gargantuan patch is that we can rip out our unexec 
implementations and replace them with loading a data file that contains 
an Emacs heap image. There are no dependencies on executable rewriting, 
disabling ASLR, or saving and restoring internal malloc state. This 
system works with fully position-independent executables and with any 
malloc implementation.

Basically, there's a new dump-emacs-portable function that walks the 
Emacs heap and writes the data, along with necessary relocation, to a 
file called emacs.pdmp. On startup, early in main, we find emacs.pdmp 
and load it. Once we've loaded the file, we walk the list of relocations 
contained in the dump and adjust it to account for the runtime locations 
of Emacs and the dump data (neither of which we know in advance in a PIE 
world.)

There are a few subtleties: I've carefully designed the file format to 
be mmap-able and to minimize the number of on-demand copies the system 
makes while accessing this file. For example, we stick bool-vectors and 
string data at the end of the dump in a contiguous block. We follow this 
block with the relocations, which we can throw away as soon as we've 
used them.

An additional optimization follows, although this part isn't implemented 
yet: we can define a "preferred load address" for the dump and write 
relocation information such that if the dump and Emacs end up being 
where we expect them to be, we don't have to perform any relocations at all.

The system gracefully degrades though. If we can't use mmap or whatever 
on a given platform, it's possible to just slurp the whole file into a 
malloced region of memory and access it from there. This approach would 
benefit from compression, which will reduce IO loads: LZ4 reduces the 
dump size for me from ~12MB to ~4MB. As in the mmap case, we can throw away

The code isn't even close to optimized yet --- I've only tested it at 
-O0, defined GC_CHECK_MARKED_OBJECTS, and not yet inlined 
frequently-called functions pdumper.h --- but even so, it's within 100ms 
or so of an unexeced Emacs.

It's also possible to dump an already-dumped Emacs, so it should be 
possible for users to have their own dump files.

If we want to preserve the current model of a single "emacs" executable 
that contains itself, we can embed emacs.pdmp inside the emacs 
executable data section pretty easily. There's no behavior change involved.

If you want to try this code, build CANNOT_DUMP=1, run ./temacs -l 
loadup pdump, then ./emacs (or if that doesn't work, ./emacs 
--dump-file=emacs.pdmp).

[-- Attachment #2: pdumper.diff --]
[-- Type: text/x-patch, Size: 278645 bytes --]

diff --git a/.gitignore b/.gitignore
index 15f9c56..3b7975f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -182,6 +182,7 @@ lib-src/emacsclient
 lib-src/etags
 lib-src/hexl
 lib-src/make-docfile
+lib-src/make-fingerprint
 lib-src/movemail
 lib-src/profile
 lib-src/test-distrib
@@ -195,6 +196,9 @@ src/bootstrap-emacs
 src/emacs
 src/emacs-[0-9]*
 src/temacs
+src/temacs.in
+src/fingerprint.c
+src/*.pdmp
 
 # Character-set info.
 admin/charsets/jisx2131-filter
diff --git a/configure.ac b/configure.ac
index c9759e1..3e44716 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2167,6 +2167,10 @@ AC_DEFUN
   hybrid_malloc=yes
 fi
 
+## XXX
+hybrid_malloc=no
+system_malloc=yes
+
 GMALLOC_OBJ=
 HYBRID_MALLOC=
 if test "${system_malloc}" = "yes"; then
diff --git a/doc/lispref/loading.texi b/doc/lispref/loading.texi
index 44ce719..901a009 100644
--- a/doc/lispref/loading.texi
+++ b/doc/lispref/loading.texi
@@ -303,10 +303,7 @@ Library Search
 initializes @code{load-path} using the @file{lisp}
 directory in the directory containing the sources from which it
 was built.
-@c Though there should be no *.el files in builddir/lisp, so it's pointless.
-If you built Emacs in a separate directory from the
-sources, it also adds the lisp directories from the build directory.
-(In all cases, elements are represented as absolute file names.)
+Elements are represented as absolute file names.
 
 @cindex site-lisp directories
 Unless you start Emacs with the @option{--no-site-lisp} option,
@@ -329,9 +326,9 @@ Library Search
 The first one is for locally installed files for a specific Emacs
 version; the second is for locally installed files meant for use
 with all installed Emacs versions.  (If Emacs is running uninstalled,
-it also adds @file{site-lisp} directories from the source and build
-directories, if they exist.  Normally these directories do not contain
-@file{site-lisp} directories.)
+it also adds @file{site-lisp} directories from the source
+directory.  Normally this directory does not contain a
+@file{site-lisp} directory.)
 
 @cindex @env{EMACSLOADPATH} environment variable
 If the environment variable @env{EMACSLOADPATH} is set, it modifies
diff --git a/lib-src/Makefile.in b/lib-src/Makefile.in
index c3693ab..db768eb 100644
--- a/lib-src/Makefile.in
+++ b/lib-src/Makefile.in
@@ -158,7 +158,7 @@ INSTALLABLES =
 UTILITIES = profile${EXEEXT} movemail${EXEEXT} hexl${EXEEXT} \
             update-game-score${EXEEXT}
 
-DONT_INSTALL= make-docfile${EXEEXT}
+DONT_INSTALL= make-docfile${EXEEXT} make-fingerprint${EXEEXT}
 
 # Like UTILITIES, but they're not system-dependent, and should not be
 #  deleted by the distclean target.
@@ -385,6 +385,9 @@ profile${EXEEXT}:
 make-docfile${EXEEXT}: ${srcdir}/make-docfile.c $(NTLIB) $(config_h)
 	$(AM_V_CCLD)$(CC) ${ALL_CFLAGS} $< $(NTLIB) $(LOADLIBES) -o $@
 
+make-fingerprint${EXEEXT}: ${srcdir}/make-fingerprint.c $(NTLIB) $(config_h)
+	$(AM_V_CCLD)$(CC) ${ALL_CFLAGS} $< $(NTLIB) $(LOADLIBES) -o $@
+
 movemail${EXEEXT}: ${srcdir}/movemail.c pop.o $(NTLIB) $(config_h)
 	$(AM_V_CCLD)$(CC) ${ALL_CFLAGS} ${MOVE_FLAGS} $< pop.o \
 	  $(NTLIB) $(LOADLIBES) $(LIBS_MOVE) -o $@
diff --git a/lib-src/make-fingerprint.c b/lib-src/make-fingerprint.c
new file mode 100644
index 0000000..dd2f093
--- /dev/null
+++ b/lib-src/make-fingerprint.c
@@ -0,0 +1,86 @@
+/* Hash inputs and generate C file with the digest.
+
+Copyright (C) 1985-1986, 1992-1994, 1997, 1999-2016 Free Software
+Foundation, Inc.
+
+This file is part of GNU Emacs.
+
+GNU Emacs is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or (at
+your option) any later version.
+
+GNU Emacs is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
+
+
+/* The arguments given to this program are all the object files that
+ go into building GNU Emacs.  There is no special search logic to find
+ the files.  */
+
+#include <config.h>
+
+#include <stdarg.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sysstdio.h>
+#include <sha256.h>
+
+#ifdef WINDOWSNT
+/* Defined to be sys_fopen in ms-w32.h, but only #ifdef emacs, so this
+   is really just insurance.  */
+#undef fopen
+#include <direct.h>
+#endif /* WINDOWSNT */
+
+int
+main (int argc, char **argv)
+{
+  struct sha256_ctx ctx;
+  sha256_init_ctx (&ctx);
+
+  for (int i = 1; i < argc; ++i)
+    {
+      FILE *f = fopen (argv[i], "r" FOPEN_BINARY);
+      if (!f)
+        {
+          fprintf (stderr, "%s: Error: could not open %s\n",
+                   argv[0], argv[i]);
+          return 1;
+        }
+
+      char buf[128*1024];
+      do
+        {
+          size_t chunksz = fread (buf, 1, sizeof (buf), f);
+          if (ferror (f))
+            {
+              fprintf (stderr, "%s: Error: could not read %s\n",
+                       argv[0], argv[i]);
+              return 1;
+            }
+          sha256_process_bytes (buf, chunksz, &ctx);
+        } while (!feof (f));
+      fclose (f);
+    }
+
+  uint8_t digest[32];
+  sha256_finish_ctx (&ctx, digest);
+  printf ("#include \"fingerprint.h\"\n");
+  printf ("\n");
+  printf ("const uint8_t fingerprint[32] = { ");
+  for (int i = 0; i < 32; ++i)
+    printf ("%s0x%02X", i ? ", " : "", digest[i]);
+  printf (" };\n");
+
+  return EXIT_SUCCESS;
+}
+
+/* make-fingerprint.c ends here */
diff --git a/lisp/loadup.el b/lisp/loadup.el
index 5c16464..58a1ba9 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -26,6 +26,9 @@
 
 ;; This is loaded into a bare Emacs to make a dumpable one.
 
+;; Emacs injects the variable `dump-mode' to tell us how to dump.  We unintern
+;; it before allowing user code to run.
+
 ;; If you add a file to be loaded here, keep the following points in mind:
 
 ;; i) If the file is no-byte-compile, explicitly load the .el version.
@@ -55,28 +58,52 @@
 (setq redisplay--inhibit-bidi t)
 
 ;; Add subdirectories to the load-path for files that might get
-;; autoloaded when bootstrapping.
+;; autoloaded when bootstrapping or running Emacs normally.
 ;; This is because PATH_DUMPLOADSEARCH is just "../lisp".
-(if (or (equal (member "bootstrap" command-line-args) '("bootstrap"))
+(if (or (member dump-mode '("bootstrap" "pbootstrap"))
 	;; FIXME this is irritatingly fragile.
 	(equal (nth 4 command-line-args) "unidata-gen.el")
-	(equal (nth 7 command-line-args) "unidata-gen-files")
+        (equal (nth 7 command-line-args) "unidata-gen-files")
+        ;; XXX figure out why we need this here
 	(if (fboundp 'dump-emacs)
 	    (string-match "src/bootstrap-emacs" (nth 0 command-line-args))
 	  t))
-    (let ((dir (car load-path)))
-      ;; We'll probably overflow the pure space.
-      (setq purify-flag nil)
-      ;; Value of max-lisp-eval-depth when compiling initially.
-      ;; During bootstrapping the byte-compiler is run interpreted when
-      ;; compiling itself, which uses a lot more stack than usual.
-      (setq max-lisp-eval-depth 2200)
-      (setq load-path (list (expand-file-name "." dir)
-			    (expand-file-name "emacs-lisp" dir)
-			    (expand-file-name "language" dir)
-			    (expand-file-name "international" dir)
-			    (expand-file-name "textmodes" dir)
-			    (expand-file-name "vc" dir)))))
+    ;; Find the entry in load-path that contains Emacs elisp and
+    ;; splice some additional directories in there for the benefit
+    ;; of autoload and regular Emacs use.
+    (let ((subdirs '("emacs-lisp"
+                     "language"
+                     "international"
+                     "textmodes"
+                     "vc"))
+          (iter load-path))
+      (while iter
+        (let ((dir (car iter))
+              (subdirs subdirs)
+              esubdirs esubdir)
+          (while subdirs
+            (setq esubdir (expand-file-name (car subdirs) dir))
+            (setq subdirs (cdr subdirs))
+            (if (file-directory-p esubdir)
+                (setq esubdirs (cons esubdir esubdirs))
+              (setq subdirs nil esubdirs nil)))
+          (if esubdirs
+              (progn
+                (setcdr iter (nconc (nreverse esubdirs) (cdr iter)))
+                (setq iter nil))
+            (setq iter (cdr iter))
+            (if (null iter)
+                (signal
+                 'error (list
+                         (format-message
+                          "Could not find elisp load-path: searched %S"
+                          load-path))))))))
+  ;; We'll probably overflow the pure space.
+  (setq purify-flag nil)
+  ;; Value of max-lisp-eval-depth when compiling initially.
+  ;; During bootstrapping the byte-compiler is run interpreted when
+  ;; compiling itself, which uses a lot more stack than usual.
+  (setq max-lisp-eval-depth 2200))
 
 (if (eq t purify-flag)
     ;; Hash consing saved around 11% of pure space in my tests.
@@ -84,9 +111,7 @@
 
 (message "Using load-path %s" load-path)
 
-;; This is a poor man's `last', since we haven't loaded subr.el yet.
-(if (or (equal (member "bootstrap" command-line-args) '("bootstrap"))
-	(equal (member "dump" command-line-args) '("dump")))
+(if dump-mode
     (progn
       ;; To reduce the size of dumped Emacs, we avoid making huge char-tables.
       (setq inhibit-load-charset-map t)
@@ -343,8 +368,7 @@
 ;; file primitive.  So the only workable solution to support building
 ;; in non-ASCII directories is to manipulate unibyte strings in the
 ;; current locale's encoding.
-(if (and (member (car (last command-line-args)) '("dump" "bootstrap"))
-	 (multibyte-string-p default-directory))
+(if (and dump-mode (multibyte-string-p default-directory))
     (error "default-directory must be unibyte when dumping Emacs!"))
 
 ;; Determine which last version number to use
@@ -443,37 +467,39 @@
 ;; Make sure we will attempt bidi reordering henceforth.
 (setq redisplay--inhibit-bidi nil)
 
-(if (member (car (last command-line-args)) '("dump" "bootstrap"))
-    (progn
-      (message "Dumping under the name emacs")
+(if dump-mode
+    (let ((output (cond ((equal dump-mode "pdump") "emacs.pdmp")
+                        ((equal dump-mode "dump") "emacs")
+                        ((equal dump-mode "bootstrap") "emacs")
+                        ((equal dump-mode "pbootstrap") (error "XXX"))
+                        (t (error "unrecognized dump mode %s" dump-mode)))))
+      (message "Dumping under the name %s" output)
       (condition-case ()
-	  (delete-file "emacs")
-	(file-error nil))
-      ;; We used to dump under the name xemacs, but that occasionally
-      ;; confused people installing Emacs (they'd install the file
-      ;; under the name `xemacs'), and it's inconsistent with every
-      ;; other GNU program's build process.
-      (dump-emacs "emacs" "temacs")
-      (message "%d pure bytes used" pure-bytes-used)
-      ;; Recompute NAME now, so that it isn't set when we dump.
-      (if (not (or (eq system-type 'ms-dos)
-                   ;; Don't bother adding another name if we're just
-                   ;; building bootstrap-emacs.
-                   (equal (last command-line-args) '("bootstrap"))))
-	  (let ((name (concat "emacs-" emacs-version))
-		(exe (if (eq system-type 'windows-nt) ".exe" "")))
-	    (while (string-match "[^-+_.a-zA-Z0-9]+" name)
-	      (setq name (concat (downcase (substring name 0 (match-beginning 0)))
-				 "-"
-				 (substring name (match-end 0)))))
-	    (setq name (concat name exe))
-            (message "Adding name %s" name)
-	    ;; When this runs on Windows, invocation-directory is not
-	    ;; necessarily the current directory.
-	    (add-name-to-file (expand-file-name (concat "emacs" exe)
-						invocation-directory)
-			      (expand-file-name name invocation-directory)
-			      t)))
+          (delete-file output)
+        (file-error nil))
+      (if (equal dump-mode "pdump")
+          (dump-emacs-portable output)
+        (dump-emacs output "temacs")
+        (message "%d pure bytes used" pure-bytes-used)
+        ;; Recompute NAME now, so that it isn't set when we dump.
+        (if (not (or (eq system-type 'ms-dos)
+                     ;; Don't bother adding another name if we're just
+                     ;; building bootstrap-emacs.
+                     (equal dump-mode "bootstrap")))
+            (let ((name (concat "emacs-" emacs-version))
+                  (exe (if (eq system-type 'windows-nt) ".exe" "")))
+              (while (string-match "[^-+_.a-zA-Z0-9]+" name)
+                (setq name (concat (downcase (substring name 0 (match-beginning 0)))
+                                   "-"
+                                   (substring name (match-end 0)))))
+              (setq name (concat name exe))
+              (message "Adding name %s" name)
+              ;; When this runs on Windows, invocation-directory is not
+              ;; necessarily the current directory.
+              (add-name-to-file (expand-file-name (concat "emacs" exe)
+                                                  invocation-directory)
+                                (expand-file-name name invocation-directory)
+                                t))))
       (kill-emacs)))
 
 ;; For machines with CANNOT_DUMP defined in config.h,
@@ -484,6 +510,7 @@
 	 (equal (nth 2 command-line-args) "loadup"))
     (setcdr command-line-args (nthcdr 3 command-line-args)))
 
+(unintern 'dump-mode obarray)
 (eval top-level)
 
 \f
diff --git a/src/.gdbinit b/src/.gdbinit
index b0c0dfd..d9820e5 100644
--- a/src/.gdbinit
+++ b/src/.gdbinit
@@ -1215,7 +1215,7 @@ document xwhichsymbols
   maximum number of symbols referencing it to produce.
 end
 
-# Show Lisp backtrace after normal backtrace.
+# # Show Lisp backtrace after normal backtrace.
 define hookpost-backtrace
   set $bt = backtrace_top ()
   if backtrace_p ($bt)
diff --git a/src/Makefile.in b/src/Makefile.in
index dc0bfff..d1397b78 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -401,7 +401,7 @@ base_obj =
 	buffer.o filelock.o insdel.o marker.o \
 	minibuf.o fileio.o dired.o \
 	cmds.o casetab.o casefiddle.o indent.o search.o regex.o undo.o \
-	alloc.o data.o doc.o editfns.o callint.o \
+	alloc.o pdumper.o data.o doc.o editfns.o callint.o \
 	eval.o floatfns.o fns.o font.o print.o lread.o $(MODULES_OBJ) \
 	syntax.o $(UNEXEC_OBJ) bytecode.o \
 	process.o gnutls.o callproc.o \
@@ -602,14 +602,28 @@ LIBEGNU_ARCHIVE =
 $(LIBEGNU_ARCHIVE): $(config_h)
 	$(MAKE) -C $(lib) all
 
+EMACS_DEPS_PRE=$(LIBXMENU) $(ALLOBJS)
+EMACS_DEPS_POST=$(LIBEGNU_ARCHIVE) $(EMACSRES) ${charsets} ${charscript}
+BUILD_EMACS_PRE=$(AM_V_CCLD)$(CC) $(ALL_CFLAGS) $(TEMACS_LDFLAGS) $(LDFLAGS) \
+	  -o $@ $(ALLOBJS)
+BUILD_EMACS_POST=$(LIBEGNU_ARCHIVE) $(W32_RES_LINK) $(LIBES)
+
+## We hash this file to generate the build fingerprint
+temacs.in$(EXEEXT): $(EMACS_DEPS_PRE) fingerprint-dummy.o $(EMACS_DEPS_POST)
+	$(BUILD_EMACS_PRE) fingerprint-dummy.o $(BUILD_EMACS_POST)
+
+$(libsrc)/make-fingerprint$(EXEEXT): $(libsrc)/make-fingerprint.c $(lib)/libgnu.a
+	$(MAKE) -C $(libsrc) make-fingerprint$(EXEEXT)
+
+fingerprint.c: temacs.in$(EXEEXT) $(libsrc)/make-fingerprint$(EXEEXT)
+	$(libsrc)/make-fingerprint$(EXEEXT) temacs.in$(EEXIST) > fingerprint.c
+
 ## We have to create $(etc) here because init_cmdargs tests its
 ## existence when setting Vinstallation_directory (FIXME?).
 ## This goes on to affect various things, and the emacs binary fails
 ## to start if Vinstallation_directory has the wrong value.
-temacs$(EXEEXT): $(LIBXMENU) $(ALLOBJS) \
-	         $(LIBEGNU_ARCHIVE) $(EMACSRES) ${charsets} ${charscript}
-	$(AM_V_CCLD)$(CC) $(ALL_CFLAGS) $(TEMACS_LDFLAGS) $(LDFLAGS) \
-	  -o temacs $(ALLOBJS) $(LIBEGNU_ARCHIVE) $(W32_RES_LINK) $(LIBES)
+temacs$(EXEEXT): $(EMACS_DEPS_PRE) fingerprint.o $(EMACS_DEPS_POST)
+	$(BUILD_EMACS_PRE) fingerprint.o $(BUILD_EMACS_POST)
 	$(MKDIR_P) $(etc)
 ifneq ($(CANNOT_DUMP),yes)
   ifneq ($(PAXCTL_notdumped),)
@@ -651,6 +665,7 @@ .PHONY:
 
 mostlyclean:
 	rm -f temacs$(EXEEXT) core *.core \#* *.o
+	rm -f temacs.in$(EXEEXT) fingerprint.c
 	rm -f ../etc/DOC
 	rm -f bootstrap-emacs$(EXEEXT) emacs-$(version)$(EXEEXT)
 	rm -f buildobj.h
diff --git a/src/alloc.c b/src/alloc.c
index 90c6f94..e31f71d 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -42,6 +42,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "keyboard.h"
 #include "frame.h"
 #include "blockinput.h"
+#include "pdumper.h"
 #include "termhooks.h"		/* For struct terminal.  */
 #ifdef HAVE_WINDOW_SYSTEM
 #include TERM_HEADER
@@ -77,6 +78,8 @@ static bool valgrind_p;
 
 /* GC_CHECK_MARKED_OBJECTS means do sanity checks on allocated objects.  */
 
+#define GC_CHECK_MARKED_OBJECTS 1 // XXX
+
 /* GC_MALLOC_CHECK defined means perform validity checks of malloc'd
    memory.  Can do this only if using gmalloc.c and if not checking
    marked objects.  */
@@ -202,13 +205,13 @@ alloc_unexec_post (void)
 /* Mark, unmark, query mark bit of a Lisp string.  S must be a pointer
    to a struct Lisp_String.  */
 
-#define MARK_STRING(S)		((S)->size |= ARRAY_MARK_FLAG)
-#define UNMARK_STRING(S)	((S)->size &= ~ARRAY_MARK_FLAG)
-#define STRING_MARKED_P(S)	(((S)->size & ARRAY_MARK_FLAG) != 0)
+#define XMARK_STRING(S)		((S)->size |= ARRAY_MARK_FLAG)
+#define XUNMARK_STRING(S)	((S)->size &= ~ARRAY_MARK_FLAG)
+#define XSTRING_MARKED_P(S)	(((S)->size & ARRAY_MARK_FLAG) != 0)
 
-#define VECTOR_MARK(V)		((V)->header.size |= ARRAY_MARK_FLAG)
-#define VECTOR_UNMARK(V)	((V)->header.size &= ~ARRAY_MARK_FLAG)
-#define VECTOR_MARKED_P(V)	(((V)->header.size & ARRAY_MARK_FLAG) != 0)
+#define XMARK_VECTOR(V)		((V)->header.size |= ARRAY_MARK_FLAG)
+#define XUNMARK_VECTOR(V)	((V)->header.size &= ~ARRAY_MARK_FLAG)
+#define XVECTOR_MARKED_P(V)	(((V)->header.size & ARRAY_MARK_FLAG) != 0)
 
 /* Default value of gc_cons_threshold (see below).  */
 
@@ -357,6 +360,36 @@ static void compact_small_strings (void);
 static void free_large_strings (void);
 extern Lisp_Object which_symbols (Lisp_Object, EMACS_INT) EXTERNALLY_VISIBLE;
 
+/* Forward declare mark accessor functions: they're used all over the
+   place.  */
+
+INLINE static bool vector_marked_p (const struct Lisp_Vector *v);
+INLINE static void set_vector_marked (struct Lisp_Vector *v);
+
+INLINE static bool vectorlike_marked_p (const struct vectorlike_header *v);
+INLINE static void set_vectorlike_marked (struct vectorlike_header *v);
+
+INLINE static bool cons_marked_p (const struct Lisp_Cons *c);
+INLINE static void set_cons_marked (struct Lisp_Cons *c);
+
+INLINE static bool string_marked_p (const struct Lisp_String *s);
+INLINE static void set_string_marked (struct Lisp_String *s);
+
+INLINE static bool symbol_marked_p (const struct Lisp_Symbol *s);
+INLINE static void set_symbol_marked (struct Lisp_Symbol *s);
+
+INLINE static bool misc_any_marked_p (const struct Lisp_Misc_Any *m);
+INLINE static void set_misc_any_marked (struct Lisp_Misc_Any *m);
+
+INLINE static bool marker_marked_p (const struct Lisp_Marker *m);
+INLINE static void set_marker_marked (struct Lisp_Marker *m);
+
+INLINE static bool overlay_marked_p (const struct Lisp_Overlay *m);
+INLINE static void set_overlay_marked (struct Lisp_Overlay *m);
+
+INLINE static bool interval_marked_p (INTERVAL i);
+INLINE static void set_interval_marked (INTERVAL i);
+
 /* When scanning the C stack for live Lisp objects, Emacs keeps track of
    what memory allocated via lisp_malloc and lisp_align_malloc is intended
    for what purpose.  This enumeration specifies the type of memory.  */
@@ -383,7 +416,10 @@ enum mem_type
 /* A unique object in pure space used to make some Lisp objects
    on free lists recognizable in O(1).  */
 
-static Lisp_Object Vdead;
+#ifndef ENABLE_CHECKING
+static
+#endif
+Lisp_Object Vdead;
 #define DEADP(x) EQ (x, Vdead)
 
 #ifdef GC_MALLOC_CHECK
@@ -465,30 +501,21 @@ static struct mem_node *mem_find (void *);
 #endif
 
 /* Addresses of staticpro'd variables.  Initialize it to a nonzero
-   value; otherwise some compilers put it into BSS.  */
+   value if we might dump; otherwise some compilers put it into
+   BSS.  */
 
-enum { NSTATICS = 2048 };
-static Lisp_Object *staticvec[NSTATICS] = {&Vpurify_flag};
+Lisp_Object *staticvec[NSTATICS]
+#ifndef CANNOT_DUMP
+= {&Vpurify_flag}
+#endif
+  ;
 
 /* Index of next unused slot in staticvec.  */
 
-static int staticidx;
+int staticidx;
 
 static void *pure_alloc (size_t, int);
 
-/* True if N is a power of 2.  N should be positive.  */
-
-#define POWER_OF_2(n) (((n) & ((n) - 1)) == 0)
-
-/* Return X rounded to the next multiple of Y.  Y should be positive,
-   and Y - 1 + X should not overflow.  Arguments should not have side
-   effects, as they are evaluated more than once.  Tune for Y being a
-   power of 2.  */
-
-#define ROUNDUP(x, y) (POWER_OF_2 (y)					\
-		       ? ((y) - 1 + (x)) & ~ ((y) - 1)			\
-		       : ((y) - 1 + (x)) - ((y) - 1 + (x)) % (y))
-
 /* Return PTR rounded up to the next multiple of ALIGNMENT.  */
 
 static void *
@@ -553,18 +580,18 @@ mmap_lisp_allowed_p (void)
      over our address space.  We also can't use mmap for lisp objects
      if we might dump: unexec doesn't preserve the contents of mmapped
      regions.  */
-  return pointers_fit_in_lispobj_p () && !might_dump;
+  return pointers_fit_in_lispobj_p () && !might_dump_with_unexec;
 }
 #endif
 
 /* Head of a circularly-linked list of extant finalizers. */
-static struct Lisp_Finalizer finalizers;
+struct Lisp_Finalizer finalizers;
 
 /* Head of a circularly-linked list of finalizers that must be invoked
    because we deemed them unreachable.  This list must be global, and
    not a local inside garbage_collect_1, in case we GC again while
    running finalizers.  */
-static struct Lisp_Finalizer doomed_finalizers;
+struct Lisp_Finalizer doomed_finalizers;
 
 \f
 /************************************************************************
@@ -895,6 +922,8 @@ xfree (void *block)
 {
   if (!block)
     return;
+  if (pdumper_object_p (block))
+    return;
   MALLOC_BLOCK_INPUT;
   free (block);
   MALLOC_UNBLOCK_INPUT;
@@ -1117,6 +1146,9 @@ lisp_malloc (size_t nbytes, enum mem_type type)
 static void
 lisp_free (void *block)
 {
+  if (pdumper_object_p (block))
+    return;
+
   MALLOC_BLOCK_INPUT;
   free (block);
 #ifndef GC_MALLOC_CHECK
@@ -1548,22 +1580,23 @@ make_interval (void)
 /* Mark Lisp objects in interval I.  */
 
 static void
-mark_interval (register INTERVAL i, Lisp_Object dummy)
+mark_interval_tree_1 (register INTERVAL i, Lisp_Object dummy)
 {
   /* Intervals should never be shared.  So, if extra internal checking is
      enabled, GC aborts if it seems to have visited an interval twice.  */
-  eassert (!i->gcmarkbit);
-  i->gcmarkbit = 1;
+  eassert (!interval_marked_p (i));
+  set_interval_marked (i);
   mark_object (i->plist);
 }
 
 /* Mark the interval tree rooted in I.  */
 
-#define MARK_INTERVAL_TREE(i)					\
-  do {								\
-    if (i && !i->gcmarkbit)					\
-      traverse_intervals_noorder (i, mark_interval, Qnil);	\
-  } while (0)
+static void
+mark_interval_tree (INTERVAL i)
+{
+  if (i && !interval_marked_p (i))
+    traverse_intervals_noorder (i, mark_interval_tree_1, Qnil);
+}
 
 /***********************************************************************
 			  String Allocation
@@ -1798,7 +1831,9 @@ static void
 init_strings (void)
 {
   empty_unibyte_string = make_pure_string ("", 0, 0, 0);
+  staticpro (&empty_unibyte_string);
   empty_multibyte_string = make_pure_string ("", 0, 0, 1);
+  staticpro (&empty_multibyte_string);
 }
 
 
@@ -2091,10 +2126,10 @@ sweep_strings (void)
 	  if (s->data)
 	    {
 	      /* String was not on free-list before.  */
-	      if (STRING_MARKED_P (s))
+	      if (XSTRING_MARKED_P (s))
 		{
 		  /* String is live; unmark it and its intervals.  */
-		  UNMARK_STRING (s);
+		  XUNMARK_STRING (s);
 
 		  /* Do not use string_(set|get)_intervals here.  */
 		  s->intervals = balance_intervals (s->intervals);
@@ -2591,7 +2626,8 @@ make_formatted_string (char *buf, const char *format, ...)
    &= ~((bits_word) 1 << ((n) % BITS_PER_BITS_WORD)))
 
 #define FLOAT_BLOCK(fptr) \
-  ((struct float_block *) (((uintptr_t) (fptr)) & ~(BLOCK_ALIGN - 1)))
+  (eassert (!pdumper_object_p (fptr)),                                  \
+   ((struct float_block *) (((uintptr_t) (fptr)) & ~(BLOCK_ALIGN - 1))))
 
 #define FLOAT_INDEX(fptr) \
   ((((uintptr_t) (fptr)) & (BLOCK_ALIGN - 1)) / sizeof (struct Lisp_Float))
@@ -2604,13 +2640,13 @@ struct float_block
   struct float_block *next;
 };
 
-#define FLOAT_MARKED_P(fptr) \
+#define XFLOAT_MARKED_P(fptr) \
   GETMARKBIT (FLOAT_BLOCK (fptr), FLOAT_INDEX ((fptr)))
 
-#define FLOAT_MARK(fptr) \
+#define XFLOAT_MARK(fptr) \
   SETMARKBIT (FLOAT_BLOCK (fptr), FLOAT_INDEX ((fptr)))
 
-#define FLOAT_UNMARK(fptr) \
+#define XFLOAT_UNMARK(fptr) \
   UNSETMARKBIT (FLOAT_BLOCK (fptr), FLOAT_INDEX ((fptr)))
 
 /* Current float_block.  */
@@ -2660,7 +2696,7 @@ make_float (double float_value)
   MALLOC_UNBLOCK_INPUT;
 
   XFLOAT_INIT (val, float_value);
-  eassert (!FLOAT_MARKED_P (XFLOAT (val)));
+  eassert (!XFLOAT_MARKED_P (XFLOAT (val)));
   consing_since_gc += sizeof (struct Lisp_Float);
   floats_consed++;
   total_free_floats--;
@@ -2685,7 +2721,8 @@ make_float (double float_value)
    / (sizeof (struct Lisp_Cons) * CHAR_BIT + 1))
 
 #define CONS_BLOCK(fptr) \
-  ((struct cons_block *) ((uintptr_t) (fptr) & ~(BLOCK_ALIGN - 1)))
+  (eassert (!pdumper_object_p (fptr)),                                  \
+   ((struct cons_block *) ((uintptr_t) (fptr) & ~(BLOCK_ALIGN - 1))))
 
 #define CONS_INDEX(fptr) \
   (((uintptr_t) (fptr) & (BLOCK_ALIGN - 1)) / sizeof (struct Lisp_Cons))
@@ -2698,13 +2735,13 @@ struct cons_block
   struct cons_block *next;
 };
 
-#define CONS_MARKED_P(fptr) \
+#define XCONS_MARKED_P(fptr) \
   GETMARKBIT (CONS_BLOCK (fptr), CONS_INDEX ((fptr)))
 
-#define CONS_MARK(fptr) \
+#define XMARK_CONS(fptr) \
   SETMARKBIT (CONS_BLOCK (fptr), CONS_INDEX ((fptr)))
 
-#define CONS_UNMARK(fptr) \
+#define XUNMARK_CONS(fptr) \
   UNSETMARKBIT (CONS_BLOCK (fptr), CONS_INDEX ((fptr)))
 
 /* Current cons_block.  */
@@ -2766,7 +2803,7 @@ DEFUN ("cons", Fcons, Scons, 2, 2, 0,
 
   XSETCAR (val, car);
   XSETCDR (val, cdr);
-  eassert (!CONS_MARKED_P (XCONS (val)));
+  eassert (!XCONS_MARKED_P (XCONS (val)));
   consing_since_gc += sizeof (struct Lisp_Cons);
   total_free_conses--;
   cons_cells_consed++;
@@ -3089,6 +3126,7 @@ static void
 init_vectors (void)
 {
   zero_vector = make_pure_vector (0);
+  staticpro (&zero_vector);
 }
 
 /* Allocate vector from a vector block.  */
@@ -3159,7 +3197,7 @@ allocate_vector_from_block (size_t nbytes)
 
 /* Return the memory footprint of V in bytes.  */
 
-static ptrdiff_t
+ptrdiff_t
 vector_nbytes (struct Lisp_Vector *v)
 {
   ptrdiff_t size = v->header.size & ~ARRAY_MARK_FLAG;
@@ -3234,9 +3272,9 @@ sweep_vectors (void)
       for (vector = (struct Lisp_Vector *) block->data;
 	   VECTOR_IN_BLOCK (vector, block); vector = next)
 	{
-	  if (VECTOR_MARKED_P (vector))
+	  if (XVECTOR_MARKED_P (vector))
 	    {
-	      VECTOR_UNMARK (vector);
+	      XUNMARK_VECTOR (vector);
 	      total_vectors++;
 	      nbytes = vector_nbytes (vector);
 	      total_vector_slots += nbytes / word_size;
@@ -3256,7 +3294,7 @@ sweep_vectors (void)
 
 	      while (VECTOR_IN_BLOCK (next, block))
 		{
-		  if (VECTOR_MARKED_P (next))
+		  if (XVECTOR_MARKED_P (next))
 		    break;
 		  cleanup_vector (next);
 		  nbytes = vector_nbytes (next);
@@ -3296,9 +3334,9 @@ sweep_vectors (void)
   for (lv = large_vectors; lv; lv = *lvprev)
     {
       vector = large_vector_vec (lv);
-      if (VECTOR_MARKED_P (vector))
+      if (XVECTOR_MARKED_P (vector))
 	{
-	  VECTOR_UNMARK (vector);
+	  XUNMARK_VECTOR (vector);
 	  total_vectors++;
 	  if (vector->header.size & PSEUDOVECTOR_FLAG)
 	    {
@@ -3961,7 +3999,7 @@ mark_finalizer_list (struct Lisp_Finalizer *head)
        finalizer != head;
        finalizer = finalizer->next)
     {
-      finalizer->base.gcmarkbit = true;
+      set_misc_any_marked (&finalizer->base);
       mark_object (finalizer->function);
     }
 }
@@ -3978,7 +4016,8 @@ queue_doomed_finalizers (struct Lisp_Finalizer *dest,
   while (finalizer != src)
     {
       struct Lisp_Finalizer *next = finalizer->next;
-      if (!finalizer->base.gcmarkbit && !NILP (finalizer->function))
+      if (!misc_any_marked_p (&finalizer->base)
+          && !NILP (finalizer->function))
         {
           unchain_finalizer (finalizer);
           finalizer_insert (dest, finalizer);
@@ -4044,6 +4083,169 @@ FUNCTION.  FUNCTION will be run once per finalizer object.  */)
 
 \f
 /************************************************************************
+			 Mark bit access functions
+ ************************************************************************/
+
+/* With the rare excpetion of functions implementing block-based
+   allocation of various types, you should not directly test or set GC
+   mark bits on objects.  Some objects might live in special memory
+   regions (e.g., a dump image) and might store their mark bits
+   elsewhere.  */
+
+static enum pvec_type find_pvec_type (const struct Lisp_Vector *ptr);
+
+static bool
+vector_marked_p (const struct Lisp_Vector *v)
+{
+  if (pdumper_object_p (v))
+    {
+      /* TODO: look into using a range test against
+         hot_discardable_start instead of loading the pvec header.
+         We'll have already loaded the dump header cache line, after
+         all.  */
+      enum pvec_type pvectype = find_pvec_type (v);
+      if (pvectype == PVEC_BOOL_VECTOR)
+        return true;
+      return pdumper_marked_p (v);
+    }
+  return XVECTOR_MARKED_P (v);
+}
+
+static void
+set_vector_marked (struct Lisp_Vector *v)
+{
+  if (pdumper_object_p (v))
+    {
+      eassert (find_pvec_type (v) != PVEC_BOOL_VECTOR);
+      pdumper_set_marked (v);
+    }
+  else
+    XMARK_VECTOR (v);
+}
+
+static bool
+vectorlike_marked_p (const struct vectorlike_header *header)
+{
+  return vector_marked_p ((const struct Lisp_Vector *) header);
+}
+
+static void
+set_vectorlike_marked (struct vectorlike_header *header)
+{
+  set_vector_marked ((struct Lisp_Vector *) header);
+}
+
+static bool
+cons_marked_p (const struct Lisp_Cons *c)
+{
+  return pdumper_object_p (c)
+    ? pdumper_marked_p (c)
+    : XCONS_MARKED_P (c);
+}
+
+static void
+set_cons_marked (struct Lisp_Cons *c)
+{
+  if (pdumper_object_p (c))
+    pdumper_set_marked (c);
+  else
+    XMARK_CONS (c);
+}
+
+static bool
+string_marked_p (const struct Lisp_String *s)
+{
+  return pdumper_object_p (s)
+    ? pdumper_marked_p (s)
+    : XSTRING_MARKED_P (s);
+}
+
+static void
+set_string_marked (struct Lisp_String *s)
+{
+  if (pdumper_object_p (s))
+    pdumper_set_marked (s);
+  else
+    XMARK_STRING (s);
+}
+
+static bool
+symbol_marked_p (const struct Lisp_Symbol *s)
+{
+  return pdumper_object_p (s)
+    ? pdumper_marked_p (s)
+    : s->gcmarkbit;
+}
+
+static void
+set_symbol_marked (struct Lisp_Symbol *s)
+{
+  if (pdumper_object_p (s))
+    pdumper_set_marked (s);
+  else
+    s->gcmarkbit = true;
+}
+
+static bool
+misc_any_marked_p (const struct Lisp_Misc_Any *m)
+{
+  return pdumper_object_p (m)
+    ? pdumper_marked_p (m)
+    : m->gcmarkbit;
+}
+
+static void
+set_misc_any_marked (struct Lisp_Misc_Any *m)
+{
+  if (pdumper_object_p (m))
+    pdumper_set_marked (m);
+  else
+    m->gcmarkbit = true;
+}
+
+static bool
+marker_marked_p (const struct Lisp_Marker *m)
+{
+  return misc_any_marked_p ((struct Lisp_Misc_Any *) m);
+}
+
+static void
+set_marker_marked (struct Lisp_Marker *m)
+{
+  set_misc_any_marked ((struct Lisp_Misc_Any *) m);
+}
+
+static bool
+overlay_marked_p (const struct Lisp_Overlay *m)
+{
+  return misc_any_marked_p ((struct Lisp_Misc_Any *) m);
+}
+
+static void
+set_overlay_marked (struct Lisp_Overlay *m)
+{
+  set_misc_any_marked ((struct Lisp_Misc_Any *) m);
+}
+
+static bool
+interval_marked_p (INTERVAL i)
+{
+  return pdumper_object_p (i)
+    ? pdumper_marked_p (i)
+    : i->gcmarkbit;
+}
+
+static void
+set_interval_marked (INTERVAL i)
+{
+  if (pdumper_object_p (i))
+    pdumper_set_marked (i);
+  else
+    i->gcmarkbit = true;
+}
+
+\f
+/************************************************************************
 			   Memory Full Handling
  ************************************************************************/
 
@@ -4712,6 +4914,19 @@ mark_maybe_object (Lisp_Object obj)
     return;
 
   void *po = XPNTR (obj);
+
+  /* If the pointer is in the dumped image and the dump has a record
+     of the object starting at the place where the pointerp points, we
+     definitely have an object.  If the pointer is in the dumped image
+     and the dump has no idea what the pointer is pointing at, we
+     definitely _don't_ have an object.  */
+  if (pdumper_object_p (po))
+    {
+      if (pdumper_object_p_precise (po))
+        mark_object (obj);
+      return;
+    }
+
   struct mem_node *m = mem_find (po);
 
   if (m != MEM_NIL)
@@ -4722,11 +4937,11 @@ mark_maybe_object (Lisp_Object obj)
 	{
 	case Lisp_String:
 	  mark_p = (live_string_p (m, po)
-		    && !STRING_MARKED_P ((struct Lisp_String *) po));
+		    && !XSTRING_MARKED_P ((struct Lisp_String *) po));
 	  break;
 
 	case Lisp_Cons:
-	  mark_p = (live_cons_p (m, po) && !CONS_MARKED_P (XCONS (obj)));
+	  mark_p = (live_cons_p (m, po) && !XCONS_MARKED_P (XCONS (obj)));
 	  break;
 
 	case Lisp_Symbol:
@@ -4734,7 +4949,7 @@ mark_maybe_object (Lisp_Object obj)
 	  break;
 
 	case Lisp_Float:
-	  mark_p = (live_float_p (m, po) && !FLOAT_MARKED_P (XFLOAT (obj)));
+	  mark_p = (live_float_p (m, po) && !XFLOAT_MARKED_P (XFLOAT (obj)));
 	  break;
 
 	case Lisp_Vectorlike:
@@ -4742,9 +4957,9 @@ mark_maybe_object (Lisp_Object obj)
 	     buffer because checking that dereferences the pointer
 	     PO which might point anywhere.  */
 	  if (live_vector_p (m, po))
-	    mark_p = !SUBRP (obj) && !VECTOR_MARKED_P (XVECTOR (obj));
+	    mark_p = !SUBRP (obj) && !XVECTOR_MARKED_P (XVECTOR (obj));
 	  else if (live_buffer_p (m, po))
-	    mark_p = BUFFERP (obj) && !VECTOR_MARKED_P (XBUFFER (obj));
+	    mark_p = BUFFERP (obj) && !XVECTOR_MARKED_P (XBUFFER (obj));
 	  break;
 
 	case Lisp_Misc:
@@ -4799,6 +5014,17 @@ mark_maybe_pointer (void *p)
       p = (void *) ((uintptr_t) p & ~(GCALIGNMENT - 1));
     }
 
+  if (pdumper_object_p (p))
+    {
+      enum Lisp_Type type = pdumper_find_object_type (p);
+      if (type != PDUMPER_NO_OBJECT)
+        mark_object ((type == Lisp_Symbol)
+                     ? make_lisp_symbol(p)
+                     : make_lisp_ptr(p, type));
+      /* See mark_maybe_object for why we can confidently return.  */
+      return;
+    }
+
   m = mem_find (p);
   if (m != MEM_NIL)
     {
@@ -4812,18 +5038,18 @@ mark_maybe_pointer (void *p)
 	  break;
 
 	case MEM_TYPE_BUFFER:
-	  if (live_buffer_p (m, p) && !VECTOR_MARKED_P ((struct buffer *)p))
+	  if (live_buffer_p (m, p) && !XVECTOR_MARKED_P ((struct buffer *)p))
 	    XSETVECTOR (obj, p);
 	  break;
 
 	case MEM_TYPE_CONS:
-	  if (live_cons_p (m, p) && !CONS_MARKED_P ((struct Lisp_Cons *) p))
+	  if (live_cons_p (m, p) && !XCONS_MARKED_P ((struct Lisp_Cons *) p))
 	    XSETCONS (obj, p);
 	  break;
 
 	case MEM_TYPE_STRING:
 	  if (live_string_p (m, p)
-	      && !STRING_MARKED_P ((struct Lisp_String *) p))
+	      && !XSTRING_MARKED_P ((struct Lisp_String *) p))
 	    XSETSTRING (obj, p);
 	  break;
 
@@ -4838,7 +5064,7 @@ mark_maybe_pointer (void *p)
 	  break;
 
 	case MEM_TYPE_FLOAT:
-	  if (live_float_p (m, p) && !FLOAT_MARKED_P (p))
+	  if (live_float_p (m, p) && !XFLOAT_MARKED_P (p))
 	    XSETFLOAT (obj, p);
 	  break;
 
@@ -4848,7 +5074,7 @@ mark_maybe_pointer (void *p)
 	    {
 	      Lisp_Object tem;
 	      XSETVECTOR (tem, p);
-	      if (!SUBRP (tem) && !VECTOR_MARKED_P (XVECTOR (tem)))
+	      if (!SUBRP (tem) && !XVECTOR_MARKED_P (XVECTOR (tem)))
 		obj = tem;
 	    }
 	  break;
@@ -5122,6 +5348,9 @@ valid_lisp_object_p (Lisp_Object obj)
   if (p == &buffer_defaults || p == &buffer_local_symbols)
     return 2;
 
+  if (pdumper_object_p (p))
+    return pdumper_object_p_precise (p) ? 1 : 0;
+
   struct mem_node *m = mem_find (p);
 
   if (m == MEM_NIL)
@@ -5538,7 +5767,7 @@ compact_font_cache_entry (Lisp_Object entry)
 
       /* Consider OBJ if it is (font-spec . [font-entity font-entity ...]).  */
       if (CONSP (obj) && GC_FONT_SPEC_P (XCAR (obj))
-	  && !VECTOR_MARKED_P (GC_XFONT_SPEC (XCAR (obj)))
+	  && !vectorlike_marked_p (&GC_XFONT_SPEC (XCAR (obj))->header)
 	  /* Don't use VECTORP here, as that calls ASIZE, which could
 	     hit assertion violation during GC.  */
 	  && (VECTORLIKEP (XCDR (obj))
@@ -5554,7 +5783,8 @@ compact_font_cache_entry (Lisp_Object entry)
             {
               Lisp_Object objlist;
 
-              if (VECTOR_MARKED_P (GC_XFONT_ENTITY (AREF (obj_cdr, i))))
+              if (vectorlike_marked_p (
+                    &GC_XFONT_ENTITY (AREF (obj_cdr, i))->header))
                 break;
 
               objlist = AREF (AREF (obj_cdr, i), FONT_OBJLIST_INDEX);
@@ -5564,7 +5794,7 @@ compact_font_cache_entry (Lisp_Object entry)
                   struct font *font = GC_XFONT_OBJECT (val);
 
                   if (!NILP (AREF (val, FONT_TYPE_INDEX))
-                      && VECTOR_MARKED_P(font))
+                      && vectorlike_marked_p(&font->header))
                     break;
                 }
               if (CONSP (objlist))
@@ -5633,7 +5863,7 @@ compact_undo_list (Lisp_Object list)
     {
       if (CONSP (XCAR (tail))
 	  && MARKERP (XCAR (XCAR (tail)))
-	  && !XMARKER (XCAR (XCAR (tail)))->gcmarkbit)
+	  && !marker_marked_p (XMARKER (XCAR (XCAR (tail)))))
 	*prev = XCDR (tail);
       else
 	prev = xcdr_addr (tail);
@@ -5659,6 +5889,69 @@ mark_pinned_symbols (void)
     }
 }
 
+static void
+visit_vectorlike_root (struct gc_root_visitor visitor,
+                       struct Lisp_Vector *ptr,
+                       enum gc_root_type type)
+{
+  ptrdiff_t size = ptr->header.size;
+  ptrdiff_t i;
+
+  if (size & PSEUDOVECTOR_FLAG)
+    size &= PSEUDOVECTOR_SIZE_MASK;
+  for (i = 0; i < size; i++)
+    visitor.visit (&ptr->contents[i], type, visitor.data);
+}
+
+static void
+visit_buffer_root (struct gc_root_visitor visitor,
+                   struct buffer *buffer,
+                   enum gc_root_type type)
+{
+  /* Buffers that are roots don't have intervals, an undo list, or
+     other constructs that real buffers have.  */
+  eassert (buffer->base_buffer == NULL);
+  eassert (buffer->overlays_before == NULL);
+  eassert (buffer->overlays_after == NULL);
+
+  /* Visit the buffer-locals.  */
+  visit_vectorlike_root (visitor, (struct Lisp_Vector *) buffer, type);
+}
+
+/* Visit GC roots stored in the Emacs data section.  Used by both core
+   GC and by the portable dumping code.
+
+   There are other GC roots of course, but these roots are dynamic
+   runtime data structures that pdump doesn't care about and so we can
+   continue to mark those directly in garbage_collect_1.  */
+void
+visit_static_gc_roots (struct gc_root_visitor visitor)
+{
+  visit_buffer_root (visitor,
+                     &buffer_defaults,
+                     GC_ROOT_BUFFER_LOCAL_DEFAULT);
+  visit_buffer_root (visitor,
+                     &buffer_local_symbols,
+                     GC_ROOT_BUFFER_LOCAL_NAME);
+
+  for (int i = 0; i < ARRAYELTS (lispsym); i++)
+    {
+      Lisp_Object sptr = builtin_lisp_symbol (i);
+      visitor.visit (&sptr, GC_ROOT_C_SYMBOL, visitor.data);
+    }
+
+  for (int i = 0; i < staticidx; i++)
+    visitor.visit (staticvec[i], GC_ROOT_STATICPRO, visitor.data);
+}
+
+static void
+mark_object_root_visitor (Lisp_Object *root_ptr,
+                          enum gc_root_type type,
+                          void *data)
+{
+  mark_object (*root_ptr);
+}
+
 /* Subroutine of Fgarbage_collect that does most of the work.  It is a
    separate function so that we could limit mark_stack in searching
    the stack frames below this function, thus avoiding the rare cases
@@ -5671,7 +5964,6 @@ garbage_collect_1 (void *end)
 {
   struct buffer *nextb;
   char stack_top_variable;
-  ptrdiff_t i;
   bool message_p;
   ptrdiff_t count = SPECPDL_INDEX ();
   struct timespec start;
@@ -5752,14 +6044,10 @@ garbage_collect_1 (void *end)
 
   /* Mark all the special slots that serve as the roots of accessibility.  */
 
-  mark_buffer (&buffer_defaults);
-  mark_buffer (&buffer_local_symbols);
-
-  for (i = 0; i < ARRAYELTS (lispsym); i++)
-    mark_object (builtin_lisp_symbol (i));
-
-  for (i = 0; i < staticidx; i++)
-    mark_object (*staticvec[i]);
+  struct gc_root_visitor visitor;
+  memset (&visitor, 0, sizeof (visitor));
+  visitor.visit = mark_object_root_visitor;
+  visit_static_gc_roots (visitor);
 
   mark_pinned_symbols ();
   mark_specpdl ();
@@ -5811,10 +6099,6 @@ garbage_collect_1 (void *end)
 
   gc_sweep ();
 
-  /* Clear the mark bits that we set in certain root slots.  */
-  VECTOR_UNMARK (&buffer_defaults);
-  VECTOR_UNMARK (&buffer_local_symbols);
-
   check_cons_list ();
 
   gc_in_progress = 0;
@@ -6014,7 +6298,7 @@ mark_glyph_matrix (struct glyph_matrix *matrix)
 
 	    for (; glyph < end_glyph; ++glyph)
 	      if (STRINGP (glyph->object)
-		  && !STRING_MARKED_P (XSTRING (glyph->object)))
+		  && !string_marked_p (XSTRING (glyph->object)))
 		mark_object (glyph->object);
 	  }
       }
@@ -6035,13 +6319,18 @@ static int last_marked_index;
 ptrdiff_t mark_object_loop_halt EXTERNALLY_VISIBLE;
 
 static void
-mark_vectorlike (struct Lisp_Vector *ptr)
+mark_vectorlike (struct vectorlike_header *header)
 {
+  struct Lisp_Vector *ptr = (struct Lisp_Vector *) header;
   ptrdiff_t size = ptr->header.size;
   ptrdiff_t i;
 
-  eassert (!VECTOR_MARKED_P (ptr));
-  VECTOR_MARK (ptr);		/* Else mark it.  */
+  eassert (!vector_marked_p (ptr));
+
+  /* Bool vectors have a different case in mark_object.  */
+  eassert (PSEUDOVECTOR_TYPE (&ptr->header) != PVEC_BOOL_VECTOR);
+
+  set_vector_marked (ptr); /* Else mark it.  */
   if (size & PSEUDOVECTOR_FLAG)
     size &= PSEUDOVECTOR_SIZE_MASK;
 
@@ -6064,17 +6353,18 @@ mark_char_table (struct Lisp_Vector *ptr, enum pvec_type pvectype)
   /* Consult the Lisp_Sub_Char_Table layout before changing this.  */
   int i, idx = (pvectype == PVEC_SUB_CHAR_TABLE ? SUB_CHAR_TABLE_OFFSET : 0);
 
-  eassert (!VECTOR_MARKED_P (ptr));
-  VECTOR_MARK (ptr);
+  eassert (!vector_marked_p (ptr));
+  set_vector_marked (ptr);
   for (i = idx; i < size; i++)
     {
       Lisp_Object val = ptr->contents[i];
 
-      if (INTEGERP (val) || (SYMBOLP (val) && XSYMBOL (val)->gcmarkbit))
+      if (INTEGERP (val) ||
+          (SYMBOLP (val) && symbol_marked_p (XSYMBOL (val))))
 	continue;
       if (SUB_CHAR_TABLE_P (val))
 	{
-	  if (! VECTOR_MARKED_P (XVECTOR (val)))
+	  if (! vector_marked_p (XVECTOR (val)))
 	    mark_char_table (XVECTOR (val), PVEC_SUB_CHAR_TABLE);
 	}
       else
@@ -6088,7 +6378,7 @@ mark_compiled (struct Lisp_Vector *ptr)
 {
   int i, size = ptr->header.size & PSEUDOVECTOR_SIZE_MASK;
 
-  VECTOR_MARK (ptr);
+  set_vector_marked (ptr);
   for (i = 0; i < size; i++)
     if (i != COMPILED_CONSTANTS)
       mark_object (ptr->contents[i]);
@@ -6100,12 +6390,11 @@ mark_compiled (struct Lisp_Vector *ptr)
 static void
 mark_overlay (struct Lisp_Overlay *ptr)
 {
-  for (; ptr && !ptr->gcmarkbit; ptr = ptr->next)
+  for (; ptr && !overlay_marked_p (ptr); ptr = ptr->next)
     {
-      ptr->gcmarkbit = 1;
-      /* These two are always markers and can be marked fast.  */
-      XMARKER (ptr->start)->gcmarkbit = 1;
-      XMARKER (ptr->end)->gcmarkbit = 1;
+      set_overlay_marked (ptr);
+      mark_object (ptr->start);
+      mark_object (ptr->end);
       mark_object (ptr->plist);
     }
 }
@@ -6116,11 +6405,11 @@ static void
 mark_buffer (struct buffer *buffer)
 {
   /* This is handled much like other pseudovectors...  */
-  mark_vectorlike ((struct Lisp_Vector *) buffer);
+  mark_vectorlike (&buffer->header);
 
   /* ...but there are some buffer-specific things.  */
 
-  MARK_INTERVAL_TREE (buffer_intervals (buffer));
+  mark_interval_tree (buffer_intervals (buffer));
 
   /* For now, we just don't mark the undo_list.  It's done later in
      a special way just before the sweep phase, and after stripping
@@ -6130,7 +6419,8 @@ mark_buffer (struct buffer *buffer)
   mark_overlay (buffer->overlays_after);
 
   /* If this is an indirect buffer, mark its base buffer.  */
-  if (buffer->base_buffer && !VECTOR_MARKED_P (buffer->base_buffer))
+  if (buffer->base_buffer &&
+      !vectorlike_marked_p (&buffer->base_buffer->header))
     mark_buffer (buffer->base_buffer);
 }
 
@@ -6149,8 +6439,8 @@ mark_face_cache (struct face_cache *c)
 
 	  if (face)
 	    {
-	      if (face->font && !VECTOR_MARKED_P (face->font))
-		mark_vectorlike ((struct Lisp_Vector *) face->font);
+	      if (face->font && !vectorlike_marked_p (&face->font->header))
+		mark_vectorlike (&face->font->header);
 
 	      for (j = 0; j < LFACE_VECTOR_SIZE; ++j)
 		mark_object (face->lface[j]);
@@ -6209,7 +6499,7 @@ mark_discard_killed_buffers (Lisp_Object list)
 {
   Lisp_Object tail, *prev = &list;
 
-  for (tail = list; CONSP (tail) && !CONS_MARKED_P (XCONS (tail));
+  for (tail = list; CONSP (tail) && !cons_marked_p (XCONS (tail));
        tail = XCDR (tail))
     {
       Lisp_Object tem = XCAR (tail);
@@ -6219,7 +6509,7 @@ mark_discard_killed_buffers (Lisp_Object list)
 	*prev = XCDR (tail);
       else
 	{
-	  CONS_MARK (XCONS (tail));
+	  set_cons_marked (XCONS (tail));
 	  mark_object (XCAR (tail));
 	  prev = xcdr_addr (tail);
 	}
@@ -6228,6 +6518,79 @@ mark_discard_killed_buffers (Lisp_Object list)
   return list;
 }
 
+static void
+mark_frame (struct Lisp_Vector *ptr)
+{
+  struct frame *f = (struct frame *) ptr;
+  mark_vectorlike (&ptr->header);
+  mark_face_cache (f->face_cache);
+#ifdef HAVE_WINDOW_SYSTEM
+  if (FRAME_WINDOW_P (f) && FRAME_X_OUTPUT (f))
+    {
+      struct font *font = FRAME_FONT (f);
+
+      if (font && !vectorlike_marked_p (&font->header))
+        mark_vectorlike (&font->header);
+    }
+#endif
+}
+
+static void
+mark_window (struct Lisp_Vector *ptr)
+{
+  struct window *w = (struct window *) ptr;
+
+  mark_vectorlike (&ptr->header);
+
+  /* Mark glyph matrices, if any.  Marking window
+     matrices is sufficient because frame matrices
+     use the same glyph memory.  */
+  if (w->current_matrix)
+    {
+      mark_glyph_matrix (w->current_matrix);
+      mark_glyph_matrix (w->desired_matrix);
+    }
+
+  /* Filter out killed buffers from both buffer lists
+     in attempt to help GC to reclaim killed buffers faster.
+     We can do it elsewhere for live windows, but this is the
+     best place to do it for dead windows.  */
+  wset_prev_buffers
+    (w, mark_discard_killed_buffers (w->prev_buffers));
+  wset_next_buffers
+    (w, mark_discard_killed_buffers (w->next_buffers));
+}
+
+static void
+mark_hash_table (struct Lisp_Vector *ptr)
+{
+  struct Lisp_Hash_Table *h = (struct Lisp_Hash_Table *) ptr;
+
+  mark_vectorlike (&h->header);
+  mark_object (h->test.name);
+  mark_object (h->test.user_hash_function);
+  mark_object (h->test.user_cmp_function);
+  /* If hash table is not weak, mark all keys and values.  For weak
+     tables, mark only the vector and not its contents --- that's what
+     makes it weak.  */
+  if (NILP (h->weak))
+    mark_object (h->key_and_value);
+  else
+    set_vector_marked (XVECTOR (h->key_and_value));
+}
+
+static enum pvec_type
+find_pvec_type (const struct Lisp_Vector *ptr)
+{
+  enum pvec_type pvectype;
+  if (ptr->header.size & PSEUDOVECTOR_FLAG)
+    pvectype = ((ptr->header.size & PVEC_TYPE_MASK)
+                >> PSEUDOVECTOR_AREA_BITS);
+  else
+    pvectype = PVEC_NORMAL_VECTOR;
+  return pvectype;
+}
+
 /* Determine type of generic Lisp_Object and mark it accordingly.
 
    This function implements a straightforward depth-first marking
@@ -6266,6 +6629,12 @@ mark_object (Lisp_Object arg)
      structure allocated from the heap.  */
 #define CHECK_ALLOCATED()			\
   do {						\
+    if (pdumper_object_p(po))                   \
+      {                                         \
+        if (!pdumper_object_p_precise (po))     \
+          emacs_abort ();                       \
+        break;                                  \
+      }                                         \
     m = mem_find (po);				\
     if (m == MEM_NIL)				\
       emacs_abort ();				\
@@ -6275,6 +6644,8 @@ mark_object (Lisp_Object arg)
      function LIVEP.  */
 #define CHECK_LIVE(LIVEP)			\
   do {						\
+    if (pdumper_object_p(po))                   \
+      break;                                    \
     if (!LIVEP (m, po))				\
       emacs_abort ();				\
   } while (0)
@@ -6309,11 +6680,11 @@ mark_object (Lisp_Object arg)
     case Lisp_String:
       {
 	register struct Lisp_String *ptr = XSTRING (obj);
-	if (STRING_MARKED_P (ptr))
-	  break;
+        if (string_marked_p (ptr))
+          break;
 	CHECK_ALLOCATED_AND_LIVE (live_string_p);
-	MARK_STRING (ptr);
-	MARK_INTERVAL_TREE (ptr->intervals);
+        set_string_marked (ptr);
+        mark_interval_tree (ptr->intervals);
 #ifdef GC_CHECK_STRING_BYTES
 	/* Check that the string size recorded in the string is the
 	   same as the one recorded in the sdata structure.  */
@@ -6327,21 +6698,19 @@ mark_object (Lisp_Object arg)
 	register struct Lisp_Vector *ptr = XVECTOR (obj);
 	register ptrdiff_t pvectype;
 
-	if (VECTOR_MARKED_P (ptr))
+	if (vector_marked_p (ptr))
 	  break;
 
 #ifdef GC_CHECK_MARKED_OBJECTS
-	m = mem_find (po);
-	if (m == MEM_NIL && !SUBRP (obj))
-	  emacs_abort ();
+        if (!pdumper_object_p(po))
+          {
+	    m = mem_find (po);
+	    if (m == MEM_NIL && !SUBRP (obj))
+	      emacs_abort ();
+          }
 #endif /* GC_CHECK_MARKED_OBJECTS */
 
-	if (ptr->header.size & PSEUDOVECTOR_FLAG)
-	  pvectype = ((ptr->header.size & PVEC_TYPE_MASK)
-		      >> PSEUDOVECTOR_AREA_BITS);
-	else
-	  pvectype = PVEC_NORMAL_VECTOR;
-
+        pvectype = find_pvec_type (ptr);
 	if (pvectype != PVEC_SUBR && pvectype != PVEC_BUFFER)
 	  CHECK_LIVE (live_vector_p);
 
@@ -6359,77 +6728,28 @@ mark_object (Lisp_Object arg)
 	    }
 #endif /* GC_CHECK_MARKED_OBJECTS */
 	    mark_buffer ((struct buffer *) ptr);
-	    break;
-
-	  case PVEC_COMPILED:
-	    /* Although we could treat this just like a vector, mark_compiled
-	       returns the COMPILED_CONSTANTS element, which is marked at the
-	       next iteration of goto-loop here.  This is done to avoid a few
-	       recursive calls to mark_object.  */
-	    obj = mark_compiled (ptr);
-	    if (!NILP (obj))
-	      goto loop;
-	    break;
-
-	  case PVEC_FRAME:
-	    {
-	      struct frame *f = (struct frame *) ptr;
-
-	      mark_vectorlike (ptr);
-	      mark_face_cache (f->face_cache);
-#ifdef HAVE_WINDOW_SYSTEM
-	      if (FRAME_WINDOW_P (f) && FRAME_X_OUTPUT (f))
-		{
-		  struct font *font = FRAME_FONT (f);
-
-		  if (font && !VECTOR_MARKED_P (font))
-		    mark_vectorlike ((struct Lisp_Vector *) font);
-		}
-#endif
-	    }
-	    break;
-
-	  case PVEC_WINDOW:
-	    {
-	      struct window *w = (struct window *) ptr;
-
-	      mark_vectorlike (ptr);
-
-	      /* Mark glyph matrices, if any.  Marking window
-		 matrices is sufficient because frame matrices
-		 use the same glyph memory.  */
-	      if (w->current_matrix)
-		{
-		  mark_glyph_matrix (w->current_matrix);
-		  mark_glyph_matrix (w->desired_matrix);
-		}
-
-	      /* Filter out killed buffers from both buffer lists
-		 in attempt to help GC to reclaim killed buffers faster.
-		 We can do it elsewhere for live windows, but this is the
-		 best place to do it for dead windows.  */
-	      wset_prev_buffers
-		(w, mark_discard_killed_buffers (w->prev_buffers));
-	      wset_next_buffers
-		(w, mark_discard_killed_buffers (w->next_buffers));
-	    }
-	    break;
+            break;
+
+          case PVEC_COMPILED:
+            /* Although we could treat this just like a vector, mark_compiled
+               returns the COMPILED_CONSTANTS element, which is marked at the
+               next iteration of goto-loop here.  This is done to avoid a few
+               recursive calls to mark_object.  */
+            obj = mark_compiled (ptr);
+            if (!NILP (obj))
+              goto loop;
+            break;
+
+          case PVEC_FRAME:
+            mark_frame (ptr);
+            break;
+
+          case PVEC_WINDOW:
+            mark_window (ptr);
+            break;
 
 	  case PVEC_HASH_TABLE:
-	    {
-	      struct Lisp_Hash_Table *h = (struct Lisp_Hash_Table *) ptr;
-
-	      mark_vectorlike (ptr);
-	      mark_object (h->test.name);
-	      mark_object (h->test.user_hash_function);
-	      mark_object (h->test.user_cmp_function);
-	      /* If hash table is not weak, mark all keys and values.
-		 For weak tables, mark only the vector.  */
-	      if (NILP (h->weak))
-		mark_object (h->key_and_value);
-	      else
-		VECTOR_MARK (XVECTOR (h->key_and_value));
-	    }
+            mark_hash_table (ptr);
 	    break;
 
 	  case PVEC_CHAR_TABLE:
@@ -6438,8 +6758,10 @@ mark_object (Lisp_Object arg)
 	    break;
 
 	  case PVEC_BOOL_VECTOR:
-	    /* No Lisp_Objects to mark in a bool vector.  */
-	    VECTOR_MARK (ptr);
+            /* Do not mark bool vectors in a dump image: these objects
+               are "cold" and don't have mark bits.  */
+            if (!pdumper_object_p (ptr))
+              set_vector_marked (ptr);
 	    break;
 
 	  case PVEC_SUBR:
@@ -6449,7 +6771,7 @@ mark_object (Lisp_Object arg)
 	    emacs_abort ();
 
 	  default:
-	    mark_vectorlike (ptr);
+	    mark_vectorlike (&ptr->header);
 	  }
       }
       break;
@@ -6458,10 +6780,10 @@ mark_object (Lisp_Object arg)
       {
 	register struct Lisp_Symbol *ptr = XSYMBOL (obj);
       nextsym:
-	if (ptr->gcmarkbit)
-	  break;
-	CHECK_ALLOCATED_AND_LIVE_SYMBOL ();
-	ptr->gcmarkbit = 1;
+        if (symbol_marked_p (ptr))
+          break;
+        CHECK_ALLOCATED_AND_LIVE_SYMBOL ();
+        set_symbol_marked(ptr);
 	/* Attempt to catch bogus objects.  */
         eassert (valid_lisp_object_p (ptr->function));
 	mark_object (ptr->function);
@@ -6488,8 +6810,8 @@ mark_object (Lisp_Object arg)
 	  default: emacs_abort ();
 	  }
 	if (!PURE_P (XSTRING (ptr->name)))
-	  MARK_STRING (XSTRING (ptr->name));
-	MARK_INTERVAL_TREE (string_intervals (ptr->name));
+          set_string_marked (XSTRING (ptr->name));
+        mark_interval_tree (string_intervals (ptr->name));
 	/* Inner loop to mark next symbol in this bucket, if any.  */
 	po = ptr = ptr->next;
 	if (ptr)
@@ -6500,8 +6822,8 @@ mark_object (Lisp_Object arg)
     case Lisp_Misc:
       CHECK_ALLOCATED_AND_LIVE (live_misc_p);
 
-      if (XMISCANY (obj)->gcmarkbit)
-	break;
+      if (misc_any_marked_p (XMISCANY (obj)))
+        break;
 
       switch (XMISCTYPE (obj))
 	{
@@ -6509,11 +6831,11 @@ mark_object (Lisp_Object arg)
 	  /* DO NOT mark thru the marker's chain.
 	     The buffer's markers chain does not preserve markers from gc;
 	     instead, markers are removed from the chain when freed by gc.  */
-	  XMISCANY (obj)->gcmarkbit = 1;
+          set_misc_any_marked (XMISCANY (obj));
 	  break;
 
 	case Lisp_Misc_Save_Value:
-	  XMISCANY (obj)->gcmarkbit = 1;
+          set_misc_any_marked (XMISCANY (obj));
 	  mark_save_value (XSAVE_VALUE (obj));
 	  break;
 
@@ -6522,46 +6844,51 @@ mark_object (Lisp_Object arg)
           break;
 
         case Lisp_Misc_Finalizer:
-          XMISCANY (obj)->gcmarkbit = true;
+          set_misc_any_marked (XMISCANY (obj));
           mark_object (XFINALIZER (obj)->function);
           break;
 
 #ifdef HAVE_MODULES
 	case Lisp_Misc_User_Ptr:
-	  XMISCANY (obj)->gcmarkbit = true;
+          set_misc_any_marked (XMISCANY (obj));
 	  break;
 #endif
 
 	default:
 	  emacs_abort ();
 	}
+
       break;
 
     case Lisp_Cons:
       {
 	register struct Lisp_Cons *ptr = XCONS (obj);
-	if (CONS_MARKED_P (ptr))
-	  break;
-	CHECK_ALLOCATED_AND_LIVE (live_cons_p);
-	CONS_MARK (ptr);
-	/* If the cdr is nil, avoid recursion for the car.  */
-	if (EQ (ptr->u.cdr, Qnil))
-	  {
-	    obj = ptr->car;
-	    cdr_count = 0;
-	    goto loop;
-	  }
-	mark_object (ptr->car);
-	obj = ptr->u.cdr;
-	cdr_count++;
-	if (cdr_count == mark_object_loop_halt)
-	  emacs_abort ();
-	goto loop;
+        if (cons_marked_p (ptr))
+          break;
+        CHECK_ALLOCATED_AND_LIVE (live_cons_p);
+        set_cons_marked (ptr);
+        /* If the cdr is nil, avoid recursion for the car.  */
+        if (EQ (ptr->u.cdr, Qnil))
+          {
+            obj = ptr->car;
+            cdr_count = 0;
+            goto loop;
+          }
+        mark_object (ptr->car);
+        obj = ptr->u.cdr;
+        cdr_count++;
+        if (cdr_count == mark_object_loop_halt)
+          emacs_abort ();
+        goto loop;
       }
 
     case Lisp_Float:
       CHECK_ALLOCATED_AND_LIVE (live_float_p);
-      FLOAT_MARK (XFLOAT (obj));
+      /* Do not mark floats stored in a dump image: these floats are
+         "cold" and do not have mark bits.  */
+      if (!pdumper_object_p (XFLOAT (obj)) &&
+          !XFLOAT_MARKED_P (XFLOAT (obj)))
+        XFLOAT_MARK (XFLOAT (obj));
       break;
 
     case_Lisp_Int:
@@ -6575,6 +6902,7 @@ mark_object (Lisp_Object arg)
 #undef CHECK_ALLOCATED
 #undef CHECK_ALLOCATED_AND_LIVE
 }
+
 /* Mark the Lisp pointers in the terminal objects.
    Called by Fgarbage_collect.  */
 
@@ -6591,13 +6919,11 @@ mark_terminals (void)
 	 gets marked.  */
       mark_image_cache (t->image_cache);
 #endif /* HAVE_WINDOW_SYSTEM */
-      if (!VECTOR_MARKED_P (t))
-	mark_vectorlike ((struct Lisp_Vector *)t);
+      if (!vectorlike_marked_p (&t->header))
+	mark_vectorlike (&t->header);
     }
 }
 
-
-
 /* Value is non-zero if OBJ will survive the current GC because it's
    either marked or does not need to be marked to survive.  */
 
@@ -6613,27 +6939,29 @@ survives_gc_p (Lisp_Object obj)
       break;
 
     case Lisp_Symbol:
-      survives_p = XSYMBOL (obj)->gcmarkbit;
+      survives_p = symbol_marked_p (XSYMBOL (obj));
       break;
 
     case Lisp_Misc:
-      survives_p = XMISCANY (obj)->gcmarkbit;
+      survives_p = misc_any_marked_p (XMISCANY (obj));
       break;
 
     case Lisp_String:
-      survives_p = STRING_MARKED_P (XSTRING (obj));
+      survives_p = string_marked_p (XSTRING (obj));
       break;
 
     case Lisp_Vectorlike:
-      survives_p = SUBRP (obj) || VECTOR_MARKED_P (XVECTOR (obj));
+      survives_p = SUBRP (obj) || vector_marked_p (XVECTOR (obj));
       break;
 
     case Lisp_Cons:
-      survives_p = CONS_MARKED_P (XCONS (obj));
+      survives_p = cons_marked_p (XCONS (obj));
       break;
 
     case Lisp_Float:
-      survives_p = FLOAT_MARKED_P (XFLOAT (obj));
+      survives_p =
+        XFLOAT_MARKED_P (XFLOAT (obj)) ||
+        pdumper_object_p (XFLOAT (obj));
       break;
 
     default:
@@ -6686,7 +7014,7 @@ sweep_conses (void)
 
               for (pos = start; pos < stop; pos++)
                 {
-                  if (!CONS_MARKED_P (&cblk->conses[pos]))
+                  if (!XCONS_MARKED_P (&cblk->conses[pos]))
                     {
                       this_free++;
                       cblk->conses[pos].u.chain = cons_free_list;
@@ -6696,7 +7024,7 @@ sweep_conses (void)
                   else
                     {
                       num_used++;
-                      CONS_UNMARK (&cblk->conses[pos]);
+                      XUNMARK_CONS (&cblk->conses[pos]);
                     }
                 }
             }
@@ -6739,7 +7067,7 @@ sweep_floats (void)
       register int i;
       int this_free = 0;
       for (i = 0; i < lim; i++)
-        if (!FLOAT_MARKED_P (&fblk->floats[i]))
+        if (!XFLOAT_MARKED_P (&fblk->floats[i]))
           {
             this_free++;
             fblk->floats[i].u.chain = float_free_list;
@@ -6748,7 +7076,7 @@ sweep_floats (void)
         else
           {
             num_used++;
-            FLOAT_UNMARK (&fblk->floats[i]);
+            XFLOAT_UNMARK (&fblk->floats[i]);
           }
       lim = FLOAT_BLOCK_SIZE;
       /* If this block contains only free floats and we have already
@@ -6962,14 +7290,15 @@ sweep_buffers (void)
 
   total_buffers = 0;
   for (buffer = all_buffers; buffer; buffer = *bprev)
-    if (!VECTOR_MARKED_P (buffer))
+    if (!vectorlike_marked_p (&buffer->header))
       {
         *bprev = buffer->next;
         lisp_free (buffer);
       }
     else
       {
-        VECTOR_UNMARK (buffer);
+        if (!pdumper_object_p (buffer))
+          XUNMARK_VECTOR (buffer);
         /* Do not use buffer_(set|get)_intervals here.  */
         buffer->text->intervals = balance_intervals (buffer->text->intervals);
         total_buffers++;
@@ -6983,7 +7312,7 @@ gc_sweep (void)
 {
   /* Remove or mark entries in weak hash tables.
      This must be done before any object is unmarked.  */
-  sweep_weak_hash_tables ();
+  mark_and_sweep_weak_hash_tables ();
 
   sweep_strings ();
   check_string_bytes (!noninteractive);
@@ -6994,6 +7323,7 @@ gc_sweep (void)
   sweep_misc ();
   sweep_buffers ();
   sweep_vectors ();
+  pdumper_clear_marks ();
   check_string_bytes (!noninteractive);
 }
 
@@ -7267,19 +7597,31 @@ verify_alloca (void)
 
 /* Initialization.  */
 
+static void init_alloc_once_for_pdumper (void);
+
 void
 init_alloc_once (void)
 {
+  gc_cons_threshold = GC_DEFAULT_THRESHOLD;
   /* Even though Qt's contents are not set up, its address is known.  */
   Vpurify_flag = Qt;
 
-  purebeg = PUREBEG;
-  pure_size = PURESIZE;
+  /* Call init_alloc_once_for_pdumper now so we run mem_init early.
+     Keep in mind that when we reload from a dump, we'll run _only_
+     init_alloc_once_for_pdumper and not init_alloc_once at all.  */
+  pdumper_do_now_and_after_load (init_alloc_once_for_pdumper);
 
   verify_alloca ();
-  init_finalizer_list (&finalizers);
-  init_finalizer_list (&doomed_finalizers);
 
+  init_strings ();
+  init_vectors ();
+}
+
+static void
+init_alloc_once_for_pdumper (void)
+{
+  purebeg = PUREBEG;
+  pure_size = PURESIZE;
   mem_init ();
   Vdead = make_pure_string ("DEAD", 4, 4, 0);
 
@@ -7288,11 +7630,11 @@ init_alloc_once (void)
   mallopt (M_MMAP_THRESHOLD, 64 * 1024);  /* Mmap threshold.  */
   mallopt (M_MMAP_MAX, MMAP_MAX_AREAS);   /* Max. number of mmap'ed areas.  */
 #endif
-  init_strings ();
-  init_vectors ();
 
+
+  init_finalizer_list (&finalizers);
+  init_finalizer_list (&doomed_finalizers);
   refill_memory_reserve ();
-  gc_cons_threshold = GC_DEFAULT_THRESHOLD;
 }
 
 void
diff --git a/src/atimer.c b/src/atimer.c
index 64c9e8a..258c61f 100644
--- a/src/atimer.c
+++ b/src/atimer.c
@@ -584,6 +584,7 @@ init_atimer (void)
   sigaction (SIGALRM, &action, 0);
 
 #ifdef ENABLE_CHECKING
-  defsubr (&Sdebug_timer_check);
+  if (!initialized)
+    defsubr (&Sdebug_timer_check);
 #endif
 }
diff --git a/src/buffer.c b/src/buffer.c
index aa556b7..58fd64f 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -43,6 +43,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "keymap.h"
 #include "frame.h"
 #include "xwidget.h"
+#include "pdumper.h"
 
 #ifdef WINDOWSNT
 #include "w32heap.h"		/* for mmap_* */
@@ -524,6 +525,8 @@ even if it is dead.  The return value is never nil.  */)
   /* No one shows us now.  */
   b->window_count = 0;
 
+  memset (&b->local_flags, 0, sizeof (b->local_flags));
+
   BUF_GAP_SIZE (b) = 20;
   block_input ();
   /* We allocate extra 1-byte at the tail and keep it always '\0' for
@@ -776,6 +779,8 @@ CLONE nil means the indirect buffer's state is reset to default values.  */)
   /* Always -1 for an indirect buffer.  */
   b->window_count = -1;
 
+  memset (&b->local_flags, 0, sizeof (b->local_flags));
+
   b->pt = b->base_buffer->pt;
   b->begv = b->base_buffer->begv;
   b->zv = b->base_buffer->zv;
@@ -4959,24 +4964,37 @@ alloc_buffer_text (struct buffer *b, ptrdiff_t nbytes)
 void
 enlarge_buffer_text (struct buffer *b, ptrdiff_t delta)
 {
-  void *p;
-  ptrdiff_t nbytes = (BUF_Z_BYTE (b) - BUF_BEG_BYTE (b) + BUF_GAP_SIZE (b) + 1
-		      + delta);
   block_input ();
+  void *p;
+  unsigned char *old_beg = b->text->beg;
+  ptrdiff_t old_nbytes =
+    BUF_Z_BYTE (b) - BUF_BEG_BYTE (b) + BUF_GAP_SIZE (b) + 1;
+  ptrdiff_t new_nbytes = old_nbytes + delta;
+
+  if (pdumper_object_p (old_beg))
+    b->text->beg = NULL;
+  else
+    old_beg = NULL;
+
 #if defined USE_MMAP_FOR_BUFFERS
-  p = mmap_realloc ((void **) &b->text->beg, nbytes);
+  p = mmap_realloc ((void **) &b->text->beg, new_nbytes);
 #elif defined REL_ALLOC
-  p = r_re_alloc ((void **) &b->text->beg, nbytes);
+  p = r_re_alloc ((void **) &b->text->beg, new_nbytes);
 #else
-  p = xrealloc (b->text->beg, nbytes);
+  p = xrealloc (b->text->beg, new_nbytes);
 #endif
 
   if (p == NULL)
     {
+      if (old_beg)
+        b->text->beg = old_beg;
       unblock_input ();
-      memory_full (nbytes);
+      memory_full (new_nbytes);
     }
 
+  if (old_beg)
+    memcpy (p, old_beg, old_nbytes);
+
   BUF_BEG_ADDR (b) = p;
   unblock_input ();
 }
@@ -4989,13 +5007,16 @@ free_buffer_text (struct buffer *b)
 {
   block_input ();
 
+  if (!pdumper_object_p (b->text->beg))
+    {
 #if defined USE_MMAP_FOR_BUFFERS
-  mmap_free ((void **) &b->text->beg);
+      mmap_free ((void **) &b->text->beg);
 #elif defined REL_ALLOC
-  r_alloc_free ((void **) &b->text->beg);
+      r_alloc_free ((void **) &b->text->beg);
 #else
-  xfree (b->text->beg);
+      xfree (b->text->beg);
 #endif
+    }
 
   BUF_BEG_ADDR (b) = NULL;
   unblock_input ();
@@ -5006,12 +5027,23 @@ free_buffer_text (struct buffer *b)
 /***********************************************************************
 			    Initialization
  ***********************************************************************/
-
 void
 init_buffer_once (void)
 {
+  /* TODO: clean up the buffer-local machinery.  Right now,
+     we have:
+
+     buffer_defaults: default values of buffer-locals
+     buffer_local_flags: metadata
+     buffer_permanent_local_flags: metadata
+     buffer_local_symbols: metadata
+
+     There must be a simpler way to store the metadata.
+  */
+
   int idx;
 
+  PDUMPER_REMEMBER_SCALAR (buffer_permanent_local_flags);
   memset (buffer_permanent_local_flags, 0, sizeof buffer_permanent_local_flags);
 
   /* 0 means not a lisp var, -1 means always local, else mask.  */
@@ -5096,13 +5128,20 @@ init_buffer_once (void)
   XSETFASTINT (BVAR (&buffer_local_flags, extra_line_spacing), idx); ++idx;
   XSETFASTINT (BVAR (&buffer_local_flags, cursor_in_non_selected_windows), idx); ++idx;
 
+  /* buffer_local_flags contains no pointers, so it's safe to treat it
+     as a blob for pdumper.  */
+  PDUMPER_REMEMBER_SCALAR (buffer_local_flags);
+
   /* Need more room? */
   if (idx >= MAX_PER_BUFFER_VARS)
     emacs_abort ();
   last_per_buffer_idx = idx;
+  PDUMPER_REMEMBER_SCALAR (last_per_buffer_idx);
 
   /* Make sure all markable slots in buffer_defaults
      are initialized reasonably, so mark_buffer won't choke.  */
+  /* TODO: XXX: we no longer call mark_buffer on buffer_defaults, so
+     this work should be unnecessary.  */
   reset_buffer (&buffer_defaults);
   eassert (NILP (BVAR (&buffer_defaults, name)));
   reset_buffer_local_variables (&buffer_defaults, 1);
@@ -5192,7 +5231,9 @@ init_buffer_once (void)
 
   Vbuffer_alist = Qnil;
   current_buffer = 0;
+  pdumper_remember_lv_raw_ptr (&current_buffer, Lisp_Vectorlike);
   all_buffers = 0;
+  pdumper_remember_lv_raw_ptr (&all_buffers, Lisp_Vectorlike);
 
   QSFundamental = build_pure_c_string ("Fundamental");
 
@@ -5216,14 +5257,14 @@ init_buffer_once (void)
 }
 
 void
-init_buffer (int initialized)
+init_buffer (void)
 {
   char *pwd;
   Lisp_Object temp;
   ptrdiff_t len;
 
 #ifdef USE_MMAP_FOR_BUFFERS
-  if (initialized)
+  if (dumped_with_unexec)
     {
       struct buffer *b;
 
@@ -5264,9 +5305,6 @@ init_buffer (int initialized)
 	  eassert (b->text->beg != NULL);
 	}
     }
-#else  /* not USE_MMAP_FOR_BUFFERS */
-  /* Avoid compiler warnings.  */
-  (void) initialized;
 #endif /* USE_MMAP_FOR_BUFFERS */
 
   AUTO_STRING (scratch, "*scratch*");
diff --git a/src/callint.c b/src/callint.c
index c0afc7b..6baead9 100644
--- a/src/callint.c
+++ b/src/callint.c
@@ -883,7 +883,8 @@ syms_of_callint (void)
 			 intern_c_string ("region-beginning"),
 			 intern_c_string ("region-end"),
 			 intern_c_string ("point"),
-			 intern_c_string ("mark"));
+                         intern_c_string ("mark"));
+  staticpro (&preserved_fns);
 
   DEFSYM (Qlist, "list");
   DEFSYM (Qlet, "let");
diff --git a/src/callproc.c b/src/callproc.c
index dc3ca4a..0db4dc7 100644
--- a/src/callproc.c
+++ b/src/callproc.c
@@ -1577,9 +1577,7 @@ init_callproc (void)
 	}
     }
 
-#ifndef CANNOT_DUMP
-  if (initialized)
-#endif
+  if (!will_dump)
     {
       tempdir = Fdirectory_file_name (Vexec_directory);
       if (! file_accessible_directory_p (tempdir))
diff --git a/src/category.c b/src/category.c
index 8315797..3522ce4 100644
--- a/src/category.c
+++ b/src/category.c
@@ -42,15 +42,6 @@ bset_category_table (struct buffer *b, Lisp_Object val)
   b->category_table_ = val;
 }
 
-/* The version number of the latest category table.  Each category
-   table has a unique version number.  It is assigned a new number
-   also when it is modified.  When a regular expression is compiled
-   into the struct re_pattern_buffer, the version number of the
-   category table (of the current buffer) at that moment is also
-   embedded in the structure.
-
-   For the moment, we are not using this feature.  */
-static int category_table_version;
 \f
 /* Category set staff.  */
 
@@ -514,6 +505,4 @@ See the documentation of the variable `word-combining-categories'.  */);
   defsubr (&Schar_category_set);
   defsubr (&Scategory_set_mnemonics);
   defsubr (&Smodify_category_entry);
-
-  category_table_version = 0;
 }
diff --git a/src/charset.c b/src/charset.c
index ff937bc..8a339c2 100644
--- a/src/charset.c
+++ b/src/charset.c
@@ -40,6 +40,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "charset.h"
 #include "coding.h"
 #include "buffer.h"
+#include "pdumper.h"
 
 /*** GENERAL NOTES on CODED CHARACTER SETS (CHARSETS) ***
 
@@ -62,9 +63,8 @@ Lisp_Object Vcharset_hash_table;
 
 /* Table of struct charset.  */
 struct charset *charset_table;
-
-static ptrdiff_t charset_table_size;
-static int charset_table_used;
+ptrdiff_t charset_table_size;
+int charset_table_used;
 
 /* Special charsets corresponding to symbols.  */
 int charset_ascii;
@@ -843,6 +843,8 @@ usage: (define-charset-internal ...)  */)
   bool new_definition_p;
   int nchars;
 
+  memset (&charset, 0, sizeof (charset));
+
   if (nargs != charset_arg_max)
     Fsignal (Qwrong_number_of_arguments,
 	     Fcons (intern ("define-charset-internal"),
@@ -1134,9 +1136,9 @@ usage: (define-charset-internal ...)  */)
 	  struct charset *new_table =
 	    xpalloc (0, &new_size, 1,
 		     min (INT_MAX, MOST_POSITIVE_FIXNUM),
-		     sizeof *charset_table);
-	  memcpy (new_table, charset_table, old_size * sizeof *new_table);
-	  charset_table = new_table;
+                     sizeof *charset_table);
+          memcpy (new_table, charset_table, old_size * sizeof *new_table);
+          charset_table = new_table;
 	  charset_table_size = new_size;
 	  /* FIXME: This leaks memory, as the old charset_table becomes
 	     unreachable.  If the old charset table is charset_table_init
@@ -2303,15 +2305,26 @@ init_charset_once (void)
   for (i = 0; i < ISO_MAX_DIMENSION; i++)
     for (j = 0; j < ISO_MAX_CHARS; j++)
       for (k = 0; k < ISO_MAX_FINAL; k++)
-	iso_charset_table[i][j][k] = -1;
+        iso_charset_table[i][j][k] = -1;
+
+  PDUMPER_REMEMBER_SCALAR (iso_charset_table);
 
   for (i = 0; i < 256; i++)
     emacs_mule_charset[i] = -1;
 
+  PDUMPER_REMEMBER_SCALAR (emacs_mule_charset);
+
   charset_jisx0201_roman = -1;
+  PDUMPER_REMEMBER_SCALAR (charset_jisx0201_roman);
+
   charset_jisx0208_1978 = -1;
+  PDUMPER_REMEMBER_SCALAR (charset_jisx0208_1978);
+
   charset_jisx0208 = -1;
+  PDUMPER_REMEMBER_SCALAR (charset_jisx0208);
+
   charset_ksc5601 = -1;
+  PDUMPER_REMEMBER_SCALAR (charset_ksc5601);
 }
 
 #ifdef emacs
@@ -2400,21 +2413,32 @@ the value may be a list of mnemonics.  */);
 
   charset_ascii
     = define_charset_internal (Qascii, 1, "\x00\x7F\0\0\0\0\0",
-			       0, 127, 'B', -1, 0, 1, 0, 0);
+                               0, 127, 'B', -1, 0, 1, 0, 0);
+  PDUMPER_REMEMBER_SCALAR (charset_ascii);
+
   charset_iso_8859_1
     = define_charset_internal (Qiso_8859_1, 1, "\x00\xFF\0\0\0\0\0",
-			       0, 255, -1, -1, -1, 1, 0, 0);
+                               0, 255, -1, -1, -1, 1, 0, 0);
+  PDUMPER_REMEMBER_SCALAR (charset_iso_8859_1);
+
   charset_unicode
     = define_charset_internal (Qunicode, 3, "\x00\xFF\x00\xFF\x00\x10\0",
-			       0, MAX_UNICODE_CHAR, -1, 0, -1, 1, 0, 0);
+                               0, MAX_UNICODE_CHAR, -1, 0, -1, 1, 0, 0);
+  PDUMPER_REMEMBER_SCALAR (charset_unicode);
+
   charset_emacs
     = define_charset_internal (Qemacs, 3, "\x00\xFF\x00\xFF\x00\x3F\0",
-			       0, MAX_5_BYTE_CHAR, -1, 0, -1, 1, 1, 0);
+                               0, MAX_5_BYTE_CHAR, -1, 0, -1, 1, 1, 0);
+  PDUMPER_REMEMBER_SCALAR (charset_emacs);
+
   charset_eight_bit
     = define_charset_internal (Qeight_bit, 1, "\x80\xFF\0\0\0\0\0",
 			       128, 255, -1, 0, -1, 0, 1,
-			       MAX_5_BYTE_CHAR + 1);
+                               MAX_5_BYTE_CHAR + 1);
+  PDUMPER_REMEMBER_SCALAR (charset_eight_bit);
+
   charset_unibyte = charset_iso_8859_1;
+  PDUMPER_REMEMBER_SCALAR (charset_unibyte);
 }
 
 #endif /* emacs */
diff --git a/src/charset.h b/src/charset.h
index 8e77567..00c9b7e 100644
--- a/src/charset.h
+++ b/src/charset.h
@@ -248,6 +248,8 @@ extern Lisp_Object Vcharset_hash_table;
 
 /* Table of struct charset.  */
 extern struct charset *charset_table;
+extern ptrdiff_t charset_table_size;
+extern int charset_table_used;
 
 #define CHARSET_FROM_ID(id) (charset_table + (id))
 
diff --git a/src/coding.c b/src/coding.c
index feed9c8..4024bee 100644
--- a/src/coding.c
+++ b/src/coding.c
@@ -298,6 +298,7 @@ encode_coding_XXX (struct coding_system *coding)
 #include "composite.h"
 #include "coding.h"
 #include "termhooks.h"
+#include "pdumper.h"
 
 Lisp_Object Vcoding_system_hash_table;
 
@@ -10755,6 +10756,9 @@ init_coding_once (void)
       coding_priorities[i] = i;
     }
 
+  PDUMPER_REMEMBER_SCALAR (coding_categories);
+  PDUMPER_REMEMBER_SCALAR (coding_priorities);
+
   /* ISO2022 specific initialize routine.  */
   for (i = 0; i < 0x20; i++)
     iso_code_class[i] = ISO_control_0;
@@ -10774,6 +10778,8 @@ init_coding_once (void)
   iso_code_class[ISO_CODE_SS3] = ISO_single_shift_3;
   iso_code_class[ISO_CODE_CSI] = ISO_control_sequence_introducer;
 
+  PDUMPER_REMEMBER_SCALAR (iso_code_class);
+
   for (i = 0; i < 256; i++)
     {
       emacs_mule_bytes[i] = 1;
@@ -10782,6 +10788,8 @@ init_coding_once (void)
   emacs_mule_bytes[EMACS_MULE_LEADING_CODE_PRIVATE_12] = 3;
   emacs_mule_bytes[EMACS_MULE_LEADING_CODE_PRIVATE_21] = 4;
   emacs_mule_bytes[EMACS_MULE_LEADING_CODE_PRIVATE_22] = 4;
+
+  PDUMPER_REMEMBER_SCALAR (emacs_mule_bytes);
 }
 
 #ifdef emacs
@@ -10805,6 +10813,7 @@ syms_of_coding (void)
   Vcode_conversion_workbuf_name = build_pure_c_string (" *code-conversion-work*");
 
   reused_workbuf_in_use = 0;
+  PDUMPER_REMEMBER_SCALAR (reused_workbuf_in_use);
 
   DEFSYM (Qcharset, "charset");
   DEFSYM (Qtarget_idx, "target-idx");
diff --git a/src/conf_post.h b/src/conf_post.h
index 060b912..833beb0 100644
--- a/src/conf_post.h
+++ b/src/conf_post.h
@@ -265,8 +265,10 @@ extern int emacs_setenv_TZ (char const *);
 
 #if 3 <= __GNUC__
 # define ATTRIBUTE_MALLOC __attribute__ ((__malloc__))
+# define ATTRIBUTE_SECTION(name) __attribute__((section (name)))
 #else
 # define ATTRIBUTE_MALLOC
+#define ATTRIBUTE_SECTION(name)
 #endif
 
 #if __has_attribute (alloc_size)
diff --git a/src/data.c b/src/data.c
index d221db4..1de42b7 100644
--- a/src/data.c
+++ b/src/data.c
@@ -740,7 +740,7 @@ The return value is undefined.  */)
 
   {
     bool autoload = AUTOLOADP (definition);
-    if (NILP (Vpurify_flag) || !autoload)
+    if (!will_dump || !autoload)
       { /* Only add autoload entries after dumping, because the ones before are
 	   not useful and else we get loads of them from the loaddefs.el.  */
 
diff --git a/src/dbusbind.c b/src/dbusbind.c
index a0146a3..e7aabbf 100644
--- a/src/dbusbind.c
+++ b/src/dbusbind.c
@@ -1827,6 +1827,8 @@ be called when the D-Bus reply message arrives.  */);
   xd_registered_buses = Qnil;
   staticpro (&xd_registered_buses);
 
+  // XXX: reset buses on dump load
+
   Fprovide (intern_c_string ("dbusbind"), Qnil);
 
 }
diff --git a/src/dispnew.c b/src/dispnew.c
index 70d4de0..b266b6e 100644
--- a/src/dispnew.c
+++ b/src/dispnew.c
@@ -41,6 +41,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "systime.h"
 #include "tparam.h"
 #include "xwidget.h"
+#include "pdumper.h"
 
 #ifdef HAVE_WINDOW_SYSTEM
 #include TERM_HEADER
@@ -5960,9 +5961,7 @@ init_display (void)
      with.  Otherwise newly opened tty frames will not resize
      automatically. */
 #ifdef SIGWINCH
-#ifndef CANNOT_DUMP
-  if (initialized)
-#endif /* CANNOT_DUMP */
+  if (!will_dump)
     {
       struct sigaction action;
       emacs_sigaction_init (&action, deliver_window_change_signal);
@@ -6026,11 +6025,7 @@ init_display (void)
 #endif /* HAVE_NTGUI */
 
 #ifdef HAVE_NS
-  if (!inhibit_window_system
-#ifndef CANNOT_DUMP
-     && initialized
-#endif
-      )
+  if (!inhibit_window_system && !will_dump)
     {
       Vinitial_window_system = Qns;
       Vwindow_system_version = make_number (10);
@@ -6118,10 +6113,8 @@ init_display (void)
 
   calculate_costs (XFRAME (selected_frame));
 
-  /* Set up faces of the initial terminal frame of a dumped Emacs.  */
-  if (initialized
-      && !noninteractive
-      && NILP (Vinitial_window_system))
+  /* Set up faces of the initial terminal frame.  */
+  if (!noninteractive && NILP (Vinitial_window_system))
     {
       /* For the initial frame, we don't have any way of knowing what
 	 are the foreground and background colors of the terminal.  */
@@ -6168,6 +6161,8 @@ WINDOW nil or omitted means report on the selected window.  */)
 			    Initialization
  ***********************************************************************/
 
+static void syms_of_display_for_pdumper (void);
+
 void
 syms_of_display (void)
 {
@@ -6275,11 +6270,12 @@ See `buffer-display-table' for more information.  */);
      beginning of the next redisplay).  */
   redisplay_dont_pause = true;
 
-#ifdef CANNOT_DUMP
-  if (noninteractive)
-#endif
-    {
-      Vinitial_window_system = Qnil;
-      Vwindow_system_version = Qnil;
-    }
+  pdumper_do_now_and_after_load (syms_of_display_for_pdumper);
+}
+
+static void
+syms_of_display_for_pdumper (void)
+{
+  Vinitial_window_system = Qnil;
+  Vwindow_system_version = Qnil;
 }
diff --git a/src/doc.c b/src/doc.c
index ce4f89b..c71e393 100644
--- a/src/doc.c
+++ b/src/doc.c
@@ -118,17 +118,15 @@ get_doc_string (Lisp_Object filepos, bool unibyte, bool definition)
   Lisp_Object docdir
     = NILP (tem) ? ENCODE_FILE (Vdoc_directory) : empty_unibyte_string;
   ptrdiff_t docdir_sizemax = SBYTES (docdir) + 1;
-#ifndef CANNOT_DUMP
-  docdir_sizemax = max (docdir_sizemax, sizeof sibling_etc);
-#endif
+  if (will_dump)
+    docdir_sizemax = max (docdir_sizemax, sizeof sibling_etc);
   name = SAFE_ALLOCA (docdir_sizemax + SBYTES (file));
   lispstpcpy (lispstpcpy (name, docdir), file);
 
   fd = emacs_open (name, O_RDONLY, 0);
   if (fd < 0)
     {
-#ifndef CANNOT_DUMP
-      if (!NILP (Vpurify_flag))
+      if (will_dump)
 	{
 	  /* Preparing to dump; DOC file is probably not installed.
 	     So check in ../etc.  */
@@ -136,7 +134,6 @@ get_doc_string (Lisp_Object filepos, bool unibyte, bool definition)
 
 	  fd = emacs_open (name, O_RDONLY, 0);
 	}
-#endif
       if (fd < 0)
 	{
 	  if (errno == EMFILE || errno == ENFILE)
@@ -544,12 +541,7 @@ the same file name is found in the `doc-directory'.  */)
 
   CHECK_STRING (filename);
 
-  if
-#ifndef CANNOT_DUMP
-    (!NILP (Vpurify_flag))
-#else /* CANNOT_DUMP */
-      (0)
-#endif /* CANNOT_DUMP */
+  if (will_dump)
     {
       dirname = sibling_etc;
       dirlen = sizeof sibling_etc - 1;
diff --git a/src/editfns.c b/src/editfns.c
index 72c7a9c..3e64473 100644
--- a/src/editfns.c
+++ b/src/editfns.c
@@ -228,7 +228,7 @@ tzlookup (Lisp_Object zone, bool settz)
 }
 
 void
-init_editfns (bool dumping)
+init_editfns (void)
 {
   const char *user_name;
   register char *p;
@@ -238,10 +238,9 @@ init_editfns (bool dumping)
   /* Set up system_name even when dumping.  */
   init_and_cache_system_name ();
 
-#ifndef CANNOT_DUMP
   /* When just dumping out, set the time zone to a known unlikely value
      and skip the rest of this function.  */
-  if (dumping)
+  if (will_dump_with_unexec)
     {
 # ifdef HAVE_TZSET
       xputenv (dump_tz_string);
@@ -249,17 +248,17 @@ init_editfns (bool dumping)
 # endif
       return;
     }
-#endif
 
   char *tz = getenv ("TZ");
 
-#if !defined CANNOT_DUMP && defined HAVE_TZSET
+#if defined HAVE_TZSET
   /* If the execution TZ happens to be the same as the dump TZ,
      change it to some other value and then change it back,
      to force the underlying implementation to reload the TZ info.
      This is needed on implementations that load TZ info from files,
      since the TZ file contents may differ between dump and execution.  */
-  if (tz && strcmp (tz, &dump_tz_string[tzeqlen]) == 0)
+  if (dumped_with_unexec &&
+      tz && strcmp (tz, &dump_tz_string[tzeqlen]) == 0)
     {
       ++*tz;
       tzset ();
@@ -1341,7 +1340,7 @@ of the user with that uid, or nil if there is no such user.  */)
      (That can happen if Emacs is dumpable
      but you decide to run `temacs -l loadup' and not dump.  */
   if (NILP (Vuser_login_name))
-    init_editfns (false);
+    init_editfns ();
 
   if (NILP (uid))
     return Vuser_login_name;
@@ -1364,7 +1363,7 @@ This ignores the environment variables LOGNAME and USER, so it differs from
      (That can happen if Emacs is dumpable
      but you decide to run `temacs -l loadup' and not dump.  */
   if (NILP (Vuser_login_name))
-    init_editfns (false);
+    init_editfns ();
   return Vuser_real_login_name;
 }
 
diff --git a/src/emacs-module.c b/src/emacs-module.c
index 68aeb0c..ac84473 100644
--- a/src/emacs-module.c
+++ b/src/emacs-module.c
@@ -1010,7 +1010,10 @@ void
 syms_of_module (void)
 {
   if (!plain_values)
-    ltv_mark = Fcons (Qnil, Qnil);
+    {
+      ltv_mark = Fcons (Qnil, Qnil);
+      staticpro (&ltv_mark);
+    }
   eassert (NILP (value_to_lisp (module_nil)));
 
   DEFSYM (Qmodule_refs_hash, "module-refs-hash");
diff --git a/src/emacs.c b/src/emacs.c
index efd4fa3..0380b5f 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -113,6 +113,9 @@ extern void moncontrol (int mode);
 #include <sys/resource.h>
 #endif
 
+#include "pdumper.h"
+#include "epaths.h"
+
 static const char emacs_version[] = PACKAGE_VERSION;
 static const char emacs_copyright[] = COPYRIGHT;
 static const char emacs_bugreport[] = PACKAGE_BUGREPORT;
@@ -125,16 +128,7 @@ Lisp_Object empty_unibyte_string, empty_multibyte_string;
 Lisp_Object Vlibrary_cache;
 #endif
 
-/* Set after Emacs has started up the first time.
-   Prevents reinitialization of the Lisp world and keymaps
-   on subsequent starts.  */
-bool initialized;
-
-/* Set to true if this instance of Emacs might dump.  */
-#ifndef DOUG_LEA_MALLOC
-static
-#endif
-bool might_dump;
+struct gflags gflags;
 
 #ifdef DARWIN_OS
 extern void unexec_init_emacs_zone (void);
@@ -509,8 +503,7 @@ init_cmdargs (int argc, char **argv, int skip_args, char *original_pwd)
 	      etc_exists = Ffile_exists_p (tem);
 	      if (!NILP (etc_exists))
 		{
-		  Vinstallation_directory
-		    = Ffile_name_as_directory (dir);
+                  Vinstallation_directory = Ffile_name_as_directory (dir);
 		  break;
 		}
 	    }
@@ -535,8 +528,7 @@ init_cmdargs (int argc, char **argv, int skip_args, char *original_pwd)
 	      if (!NILP (etc_exists))
 		{
 		  tem = Fexpand_file_name (build_string (".."), dir);
-		  Vinstallation_directory
-		    = Ffile_name_as_directory (tem);
+                  Vinstallation_directory = Ffile_name_as_directory (tem);
 		  break;
 		}
 	    }
@@ -664,6 +656,136 @@ close_output_streams (void)
      _exit (EXIT_FAILURE);
 }
 
+static bool
+string_starts_with_p(const char* string, const char* prefix)
+{
+    return strncmp(string, prefix, strlen(prefix)) == 0;
+}
+
+#define DUMP_FILE_ARGUMENT "--dump-file"
+
+static char *
+find_and_remove_dump_file_argument (int *inout_argc, char ***inout_argv)
+{
+  int argc = *inout_argc;
+  char **argv = *inout_argv;
+  char *found = NULL;
+  int i;
+
+  for (i = 1; i < argc; ++i)
+    if (string_starts_with_p (argv[i], DUMP_FILE_ARGUMENT) &&
+        ((argv[i] + strlen (DUMP_FILE_ARGUMENT))[0] == '=' ||
+         (argv[i] + strlen (DUMP_FILE_ARGUMENT))[0] == '\0'))
+      {
+        int j = i;
+        found = argv[j++] + strlen (DUMP_FILE_ARGUMENT);
+        if (*found == '=')
+          ++found;
+        else if (i < argc)
+          found = argv[j++];
+        else
+          {
+            fprintf (stderr, "%s: no argument given for %s\n",
+                       argv[0], DUMP_FILE_ARGUMENT);
+            exit (1);
+          }
+
+        memmove (&argv[i], &argv[j], sizeof (*argv) * (argc - j));
+        argc -= (j - i);
+        argv[argc] = NULL;
+        break;
+      }
+    else if (strcmp (argv[i], "--") == 0)
+      break;
+
+  *inout_argc = argc;
+  *inout_argv = argv;
+  return found;
+}
+
+static const char *
+dump_error_to_string (enum pdumper_load_result result)
+{
+  switch (result)
+    {
+    case PDUMPER_LOAD_SUCCESS:
+      return "success";
+    case PDUMPER_LOAD_OOM:
+      return "out of memory";
+    case PDUMPER_NOT_LOADED:
+      return "not loaded";
+    case PDUMPER_LOAD_FILE_NOT_FOUND:
+      return "could not open file";
+    case PDUMPER_LOAD_BAD_FILE_TYPE:
+      return "not a dump file";
+    case PDUMPER_LOAD_VERSION_MISMATCH:
+      return "not built for this Emacs executable";
+    default:
+      return "generic error";
+    }
+}
+
+static const char *
+load_dump (int *inout_argc, char ***inout_argv, char *argv0_base)
+{
+  int argc = *inout_argc;
+  char **argv = *inout_argv;
+  const char *suffix = ".pdmp";
+  enum pdumper_load_result result;
+
+  char *dump_file = find_and_remove_dump_file_argument (&argc, &argv);
+  if (initialized && dump_file)
+    /* TODO: maybe more thoroughly scrub process environment in order
+       to make this use case possible?  Right now, we assume that
+       things we don't touch are zero-initialized, and in an unexeced
+       Emacs, this assumption doesn't hold.  */
+    fatal ("cannot load dump file into unexeced Emacs");
+
+  if (initialized)
+    return NULL;
+
+  result = PDUMPER_NOT_LOADED;
+  if (dump_file)
+    result = pdumper_load (dump_file);
+
+  if (dump_file && result != PDUMPER_LOAD_SUCCESS)
+    fatal ("could not load dump file \"%s\": %s",
+           dump_file, dump_error_to_string (result));
+
+  if (result == PDUMPER_LOAD_SUCCESS)
+    goto out;
+
+  dump_file = alloca (strlen (argv[0]) + strlen (suffix) + 1);
+  sprintf (dump_file, "%s%s", argv[0], suffix);
+
+  result = pdumper_load (dump_file);
+  if (result == PDUMPER_LOAD_SUCCESS)
+    goto out;
+
+  if (result != PDUMPER_LOAD_FILE_NOT_FOUND)
+    fatal ("could not load dump file \"%s\": %s",
+           dump_file, dump_error_to_string (result));
+
+ /* For searching PATH_EXEC, just use "emacs".  Emacs shouldn't break
+    if its binary is renamed.  */
+  argv0_base = "emacs";
+  dump_file = alloca (strlen (PATH_EXEC)
+                      + 1
+                      + strlen (argv0_base)
+                      + strlen (suffix)
+                      + 1);
+  sprintf (dump_file, "%s%c%s%s",
+           PATH_EXEC, DIRECTORY_SEP, argv0_base, suffix);
+  result = pdumper_load (dump_file);
+  if (result != PDUMPER_LOAD_SUCCESS)
+    dump_file = NULL;
+
+ out:
+  *inout_argc = argc;
+  *inout_argv = argv;
+           return dump_file ? strdup (dump_file) : NULL;
+}
+
 /* ARGSUSED */
 int
 main (int argc, char **argv)
@@ -671,7 +793,6 @@ main (int argc, char **argv)
   Lisp_Object dummy;
   char stack_bottom_variable;
   bool do_initial_setlocale;
-  bool dumping;
   int skip_args = 0;
   bool no_loadup = false;
   char *junk = 0;
@@ -686,14 +807,36 @@ main (int argc, char **argv)
 
   stack_base = &dummy;
 
-  dumping = !initialized && (strcmp (argv[argc - 1], "dump") == 0
-			     || strcmp (argv[argc - 1], "bootstrap") == 0);
+  /* Figure out where we are.  Fancy filesystem functions aren't
+     available at this point, so use pure text manipulation.  */
+  char *argv0_base = strrchr (argv[0], DIRECTORY_SEP);
+  argv0_base = argv0_base ? argv0_base + 1 : argv[0];
+  bool is_temacs = strcmp ("temacs", argv0_base) == 0;
+  const char *loaded_dump = NULL;
+
+  const char *dump_mode = NULL;
+  if (!initialized && is_temacs)
+    {
+#ifndef CANNOT_DUMP
+      if (strcmp (argv[argc - 1], "dump") == 0 ||
+          strcmp (argv[argc - 1], "bootstrap"))
+        gflags.will_dump_with_unexec_ = true;
+#endif
+      if (strcmp (argv[argc - 1], "pdump") == 0 ||
+          strcmp (argv[argc - 1], "pbootstrap"))
+        gflags.will_dump_with_pdumper_ = true;
+      gflags.will_dump_ = will_dump_with_pdumper || will_dump_with_unexec;
+      if (will_dump)
+        dump_mode = argv[argc - 1];
+    }
+  else if (!is_temacs)
+    loaded_dump = load_dump (&argc, &argv, argv0_base);
 
   /* True if address randomization interferes with memory allocation.  */
 # ifdef __PPC64__
   bool disable_aslr = true;
 # else
-  bool disable_aslr = dumping;
+  bool disable_aslr = will_dump_with_unexec;
 # endif
 
   if (disable_aslr && disable_address_randomization ())
@@ -710,10 +853,6 @@ main (int argc, char **argv)
       perror (argv[0]);
     }
 
-#ifndef CANNOT_DUMP
-  might_dump = !initialized;
-#endif
-
 #ifdef GNU_LINUX
   if (!initialized)
     {
@@ -841,10 +980,7 @@ main (int argc, char **argv)
       /* Extra space to cover what we're likely to use for other reasons.  */
       int extra = 200000;
 
-      bool try_to_grow_stack = true;
-#ifndef CANNOT_DUMP
-      try_to_grow_stack = !noninteractive || initialized;
-#endif
+      bool try_to_grow_stack = !noninteractive || initialized;
 
       if (try_to_grow_stack)
 	{
@@ -1173,17 +1309,15 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
 
 #if defined HAVE_PTHREAD && !defined SYSTEM_MALLOC \
   && !defined DOUG_LEA_MALLOC && !defined HYBRID_MALLOC
-# ifndef CANNOT_DUMP
   /* Do not make gmalloc thread-safe when creating bootstrap-emacs, as
      that causes an infinite recursive loop with FreeBSD.  See
      Bug#14569.  The part of this bug involving Cygwin is no longer
      relevant, now that Cygwin defines HYBRID_MALLOC.  */
-  if (!noninteractive || initialized)
-# endif
+  if (!noninteractive || !will_dump)
     malloc_enable_thread ();
 #endif
 
-  init_signals (dumping);
+  init_signals ();
 
   noninteractive1 = noninteractive;
 
@@ -1192,7 +1326,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
   if (!initialized)
     {
       init_alloc_once ();
-      init_obarray ();
+      init_obarray_once ();
       init_eval_once ();
       init_charset_once ();
       init_coding_once ();
@@ -1230,7 +1364,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
       /* Before init_window_once, because it sets up the
 	 Vcoding_system_hash_table.  */
       syms_of_coding ();	/* This should be after syms_of_fileio.  */
-
+      init_frame_once ();       /* Before init_window_once.  */
       init_window_once ();	/* Init the window system.  */
 #ifdef HAVE_WINDOW_SYSTEM
       init_fringe_once ();	/* Swap bitmaps if necessary.  */
@@ -1389,7 +1523,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
   /* egetenv is a pretty low-level facility, which may get called in
      many circumstances; it seems flimsy to put off initializing it
      until calling init_callproc.  Do not do it when dumping.  */
-  if (! dumping)
+  if (!will_dump)
     set_initial_environment ();
 
 #ifdef WINDOWSNT
@@ -1403,7 +1537,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
      variables from the parent process without modifications from
      Emacs.  */
   init_environment (argv);
-  init_ntproc (dumping); /* must precede init_editfns.  */
+  init_ntproc (will_dump); /* must precede init_editfns.  */
 #endif
 
   /* AIX crashes are reported in system versions 3.2.3 and 3.2.4
@@ -1415,7 +1549,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
 #endif
 
   /* Init buffer storage and default directory of main buffer.  */
-  init_buffer (initialized);
+  init_buffer ();
 
   init_callproc_1 ();	/* Must precede init_cmdargs and init_sys_modes.  */
 
@@ -1581,6 +1715,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
 #endif /* WINDOWSNT */
 
       syms_of_profiler ();
+      syms_of_pdumper ();
 
       keys_of_casefiddle ();
       keys_of_cmds ();
@@ -1608,7 +1743,7 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
 
   /* This calls putenv and so must precede init_process_emacs.  Also,
      it sets Voperating_system_release, which init_process_emacs uses.  */
-  init_editfns (dumping);
+  init_editfns ();
 
   /* These two call putenv.  */
 #ifdef HAVE_DBUS
@@ -1679,7 +1814,11 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
 #endif
 #endif
 
-  initialized = 1;
+  gflags.initialized_ = true;
+
+  Fset (intern ("dump-mode"), dump_mode ? build_string (dump_mode) : Qnil);
+  if (loaded_dump)
+    Vdump_file_name = build_string (loaded_dump); // XXX: filesystem decode
 
   /* Enter editor command loop.  This never returns.  */
   Frecursive_edit ();
@@ -2184,12 +2323,19 @@ You must run Emacs in batch mode in order to dump it.  */)
 #endif /* not WINDOWSNT */
 #endif /* not SYSTEM_MALLOC and not HYBRID_MALLOC */
 
+  struct gflags old_gflags = gflags;
+  gflags.will_dump_ = false;
+  gflags.will_dump_with_unexec = false;
+  gflags.dumped_with_unexec_ = true;
+
   alloc_unexec_pre ();
 
   unexec (SSDATA (filename), !NILP (symfile) ? SSDATA (symfile) : 0);
 
   alloc_unexec_post ();
 
+  gflags = old_gflags;
+
 #ifdef WINDOWSNT
   Vlibrary_cache = Qnil;
 #endif
@@ -2203,6 +2349,7 @@ You must run Emacs in batch mode in order to dump it.  */)
 }
 
 #endif /* not CANNOT_DUMP */
+
 \f
 #if HAVE_SETLOCALE
 /* Recover from setlocale (LC_ALL, "").  */
@@ -2531,7 +2678,11 @@ Don't rely on it for testing whether a feature you want to use is available.  */
   Vsystem_configuration_features = build_string (EMACS_CONFIG_FEATURES);
 
   DEFVAR_BOOL ("noninteractive", noninteractive1,
-	       doc: /* Non-nil means Emacs is running without interactive terminal.  */);
+               doc: /* Non-nil means Emacs is running without interactive terminal.  */);
+
+
+  DEFVAR_LISP ("dump-file-name", Vdump_file_name,
+               doc: /* Name of the dump file used to start this Emacs process.  */);
 
   DEFVAR_LISP ("kill-emacs-hook", Vkill_emacs_hook,
 	       doc: /* Hook run when `kill-emacs' is called.
diff --git a/src/eval.c b/src/eval.c
index 884e1eb..d3e9435 100644
--- a/src/eval.c
+++ b/src/eval.c
@@ -29,6 +29,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "keyboard.h"
 #include "dispextern.h"
 #include "buffer.h"
+#include "pdumper.h"
 
 /* Chain of condition and catch handlers currently in effect.  */
 
@@ -78,10 +79,6 @@ static EMACS_INT when_entered_debugger;
 /* FIXME: We should probably get rid of this!  */
 Lisp_Object Vsignaling_function;
 
-/* If non-nil, Lisp code must not be run since some part of Emacs is in
-   an inconsistent state.  Currently unused.  */
-Lisp_Object inhibit_lisp_code;
-
 /* These would ordinarily be static, but they need to be visible to GDB.  */
 bool backtrace_p (union specbinding *) EXTERNALLY_VISIBLE;
 Lisp_Object *backtrace_args (union specbinding *) EXTERNALLY_VISIBLE;
@@ -204,18 +201,25 @@ near_C_stack_top (void)
   return backtrace_args (backtrace_top ());
 }
 
+static void init_eval_once_for_pdumper (void);
+
 void
 init_eval_once (void)
 {
-  enum { size = 50 };
-  union specbinding *pdlvec = xmalloc ((size + 1) * sizeof *specpdl);
-  specpdl_size = size;
-  specpdl = specpdl_ptr = pdlvec + 1;
   /* Don't forget to update docs (lispref node "Local Variables").  */
   max_specpdl_size = 1300; /* 1000 is not enough for CEDET's c-by.el.  */
   max_lisp_eval_depth = 800;
-
   Vrun_hooks = Qnil;
+  pdumper_do_now_and_after_load (init_eval_once_for_pdumper);
+}
+
+static void
+init_eval_once_for_pdumper (void)
+{
+  enum { size = 50 };
+  union specbinding *pdlvec = malloc ((size + 1) * sizeof *specpdl);
+  specpdl_size = size;
+  specpdl = specpdl_ptr = pdlvec + 1;
 }
 
 static struct handler handlerlist_sentinel;
@@ -1954,7 +1958,7 @@ it defines a macro.  */)
 
   /* This is to make sure that loadup.el gives a clear picture
      of what files are preloaded and when.  */
-  if (! NILP (Vpurify_flag))
+  if (will_dump)
     error ("Attempt to autoload %s while preparing to dump",
 	   SDATA (SYMBOL_NAME (funname)));
 
@@ -3711,33 +3715,33 @@ mark_specpdl (void)
   for (pdl = specpdl; pdl != specpdl_ptr; pdl++)
     {
       switch (pdl->kind)
-	{
-	case SPECPDL_UNWIND:
-	  mark_object (specpdl_arg (pdl));
-	  break;
-
-	case SPECPDL_BACKTRACE:
-	  {
-	    ptrdiff_t nargs = backtrace_nargs (pdl);
-	    mark_object (backtrace_function (pdl));
-	    if (nargs == UNEVALLED)
-	      nargs = 1;
-	    while (nargs--)
-	      mark_object (backtrace_args (pdl)[nargs]);
-	  }
-	  break;
-
-	case SPECPDL_LET_DEFAULT:
-	case SPECPDL_LET_LOCAL:
-	  mark_object (specpdl_where (pdl));
-	  /* Fall through.  */
-	case SPECPDL_LET:
-	  mark_object (specpdl_symbol (pdl));
-	  mark_object (specpdl_old_value (pdl));
-	  break;
-
-	case SPECPDL_UNWIND_PTR:
-	case SPECPDL_UNWIND_INT:
+        {
+        case SPECPDL_UNWIND:
+          mark_object (specpdl_arg (pdl));
+          break;
+
+        case SPECPDL_BACKTRACE:
+          {
+            ptrdiff_t nargs = backtrace_nargs (pdl);
+            mark_object (backtrace_function (pdl));
+            if (nargs == UNEVALLED)
+              nargs = 1;
+            while (nargs--)
+              mark_object (backtrace_args (pdl)[nargs]);
+          }
+          break;
+
+        case SPECPDL_LET_DEFAULT:
+        case SPECPDL_LET_LOCAL:
+          mark_object (specpdl_where (pdl));
+          /* Fall through.  */
+        case SPECPDL_LET:
+          mark_object (specpdl_symbol (pdl));
+          mark_object (specpdl_old_value (pdl));
+          break;
+
+        case SPECPDL_UNWIND_PTR:
+        case SPECPDL_UNWIND_INT:
 	case SPECPDL_UNWIND_VOID:
 	  break;
 
@@ -3924,8 +3928,6 @@ alist of active lexical bindings.  */);
   staticpro (&Vsignaling_function);
   Vsignaling_function = Qnil;
 
-  inhibit_lisp_code = Qnil;
-
   defsubr (&Sor);
   defsubr (&Sand);
   defsubr (&Sif);
diff --git a/src/filelock.c b/src/filelock.c
index a4b742a..5ffdf72 100644
--- a/src/filelock.c
+++ b/src/filelock.c
@@ -171,13 +171,10 @@ get_boot_time (void)
     }
 
 #if defined (BOOT_TIME)
-#ifndef CANNOT_DUMP
-  /* The utmp routines maintain static state.
-     Don't touch that state unless we are initialized,
-     since it might not survive dumping.  */
-  if (! initialized)
+  /* The utmp routines maintain static state.  Don't touch that state
+     if we are going to dump, since it might not survive dumping.  */
+  if (will_dump)
     return boot_time;
-#endif /* not CANNOT_DUMP */
 
   /* Try to get boot time from utmp before wtmp,
      since utmp is typically much smaller than wtmp.
@@ -671,7 +668,7 @@ lock_file (Lisp_Object fn)
   /* Don't do locking while dumping Emacs.
      Uncompressing wtmp files uses call-process, which does not work
      in an uninitialized Emacs.  */
-  if (! NILP (Vpurify_flag))
+  if (will_dump)
     return;
 
   orig_fn = fn;
diff --git a/src/fingerprint-dummy.c b/src/fingerprint-dummy.c
new file mode 100644
index 0000000..295654a
--- /dev/null
+++ b/src/fingerprint-dummy.c
@@ -0,0 +1,24 @@
+/* Dummy fingerprint
+
+Copyright (C) 2016 Free Software Foundation,
+Inc.
+
+This file is part of GNU Emacs.
+
+GNU Emacs is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or (at
+your option) any later version.
+
+GNU Emacs is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
+
+#include "fingerprint.h"
+
+/* Dummy fingerprint to use as hash input.  */
+const uint8_t fingerprint[32] = { 0 };
diff --git a/src/fingerprint.h b/src/fingerprint.h
new file mode 100644
index 0000000..b48d40f
--- /dev/null
+++ b/src/fingerprint.h
@@ -0,0 +1,32 @@
+/* Header file for the Emacs build fingerprint.
+
+Copyright (C) 2016 Free Software Foundation,
+Inc.
+
+This file is part of GNU Emacs.
+
+GNU Emacs is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or (at
+your option) any later version.
+
+GNU Emacs is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
+
+#ifndef EMACS_FINGERPRINT_H
+#define EMACS_FINGERPRINT_H
+
+#include <stdint.h>
+
+/* We generate fingerprint.c and fingerprint.o from all the sources in
+   Emacs.  This way, we have a unique value that we can use to pair
+   data files (like a portable dump image) with a specific build of
+   Emacs.  */
+extern const uint8_t fingerprint[32];
+
+#endif
diff --git a/src/fns.c b/src/fns.c
index dfc7842..645c0bf 100644
--- a/src/fns.c
+++ b/src/fns.c
@@ -34,6 +34,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "buffer.h"
 #include "intervals.h"
 #include "window.h"
+#include "pdumper.h"
 
 static void sort_vector_copy (Lisp_Object, ptrdiff_t,
 			      Lisp_Object *restrict, Lisp_Object *restrict);
@@ -2864,7 +2865,7 @@ suppressed.  */)
 
       /* This is to make sure that loadup.el gives a clear picture
 	 of what files are preloaded and when.  */
-      if (! NILP (Vpurify_flag))
+      if (will_dump)
 	error ("(require %s) while preparing to dump",
 	       SDATA (SYMBOL_NAME (feature)));
 
@@ -3735,7 +3736,7 @@ hashfn_eq (struct hash_table_test *ht, Lisp_Object key)
    `equal' to compare keys.  The hash code returned is guaranteed to fit
    in a Lisp integer.  */
 
-static EMACS_UINT
+EMACS_UINT
 hashfn_equal (struct hash_table_test *ht, Lisp_Object key)
 {
   return sxhash (key, 0);
@@ -3745,7 +3746,7 @@ hashfn_equal (struct hash_table_test *ht, Lisp_Object key)
    `eql' to compare keys.  The hash code returned is guaranteed to fit
    in a Lisp integer.  */
 
-static EMACS_UINT
+EMACS_UINT
 hashfn_eql (struct hash_table_test *ht, Lisp_Object key)
 {
   return FLOATP (key) ? hashfn_equal (ht, key) : hashfn_eq (ht, key);
@@ -3980,6 +3981,51 @@ maybe_resize_hash_table (struct Lisp_Hash_Table *h)
     }
 }
 
+static bool
+hash_rehash_needed_p (const struct Lisp_Hash_Table *h)
+{
+  return h->count < 0;
+}
+
+static void
+hash_table_rehash (struct Lisp_Hash_Table *h)
+{
+  ptrdiff_t size = ASIZE (h->next);
+  Lisp_Object hash = h->hash;
+  Lisp_Object kv = h->key_and_value;
+  for (ptrdiff_t i = 0; i < size; ++i)
+    if (!NILP (AREF (hash, i)))
+      {
+        Lisp_Object key = AREF (kv, 2*i);
+        EMACS_UINT hash_code = h->test.hashfn (&h->test, key);
+        ASET (hash, i, make_number (hash_code));
+      }
+  Lisp_Object index = h->index;
+  Ffillarray (index, Qnil);
+  Lisp_Object next = h->next;
+  for (ptrdiff_t i = 0; i < size; ++i)
+    if (!NILP (AREF (hash, i)))
+      {
+        EMACS_UINT hash_code = XUINT (AREF (hash, i));
+        ptrdiff_t start_of_bucket = hash_code % ASIZE (index);
+        ASET (next, i, AREF (index, start_of_bucket));
+        ASET (index, start_of_bucket, make_number (i));
+        eassert (!EQ (AREF (next, i), make_number (i))); /* Stop loops.  */
+      }
+}
+
+static void
+hash_rehash_if_needed (struct Lisp_Hash_Table *h)
+{
+  if (hash_rehash_needed_p (h))
+    {
+      hash_table_rehash (h);
+      /* Do last so that if we're interrupted, we retry on next access. */
+      eassert (h->count < 0);
+      h->count = -h->count;
+      eassert (!hash_rehash_needed_p (h));
+    }
+}
 
 /* Lookup KEY in hash table H.  If HASH is non-null, return in *HASH
    the hash code of KEY.  Value is the index of the entry in H
@@ -3992,6 +4038,8 @@ hash_lookup (struct Lisp_Hash_Table *h, Lisp_Object key, EMACS_UINT *hash)
   ptrdiff_t start_of_bucket;
   Lisp_Object idx;
 
+  hash_rehash_if_needed (h);
+
   hash_code = h->test.hashfn (&h->test, key);
   eassert ((hash_code & ~INTMASK) == 0);
   if (hash)
@@ -4025,6 +4073,8 @@ hash_put (struct Lisp_Hash_Table *h, Lisp_Object key, Lisp_Object value,
 {
   ptrdiff_t start_of_bucket, i;
 
+  hash_rehash_if_needed (h);
+
   eassert ((hash & ~INTMASK) == 0);
 
   /* Increment count after resizing because resizing may fail.  */
@@ -4057,6 +4107,8 @@ hash_remove_from_table (struct Lisp_Hash_Table *h, Lisp_Object key)
   ptrdiff_t start_of_bucket;
   Lisp_Object idx, prev;
 
+  hash_rehash_if_needed (h);
+
   hash_code = h->test.hashfn (&h->test, key);
   eassert ((hash_code & ~INTMASK) == 0);
   start_of_bucket = hash_code % ASIZE (h->index);
@@ -4103,6 +4155,9 @@ hash_remove_from_table (struct Lisp_Hash_Table *h, Lisp_Object key)
 static void
 hash_clear (struct Lisp_Hash_Table *h)
 {
+  if (CONSP (h->index))
+    h->index = CDR (h->index);
+
   if (h->count > 0)
     {
       ptrdiff_t i, size = HASH_TABLE_SIZE (h);
@@ -4137,6 +4192,8 @@ hash_clear (struct Lisp_Hash_Table *h)
 static bool
 sweep_weak_table (struct Lisp_Hash_Table *h, bool remove_entries_p)
 {
+  eassert (!hash_rehash_needed_p (h));
+
   ptrdiff_t n = gc_asize (h->index);
   bool marked = false;
 
@@ -4221,9 +4278,17 @@ sweep_weak_table (struct Lisp_Hash_Table *h, bool remove_entries_p)
    current garbage collection.  Remove weak tables that don't survive
    from Vweak_hash_tables.  Called from gc_sweep.  */
 
+static bool
+ht_marked_p (struct Lisp_Hash_Table *h)
+{
+  return pdumper_object_p (h)
+    ? pdumper_marked_p (h)
+    : !!(h->header.size & ARRAY_MARK_FLAG);
+}
+
 NO_INLINE /* For better stack traces */
 void
-sweep_weak_hash_tables (void)
+mark_and_sweep_weak_hash_tables (void)
 {
   struct Lisp_Hash_Table *h, *used, *next;
   bool marked;
@@ -4239,7 +4304,7 @@ sweep_weak_hash_tables (void)
       marked = 0;
       for (h = weak_hash_tables; h; h = h->next_weak)
 	{
-	  if (h->header.size & ARRAY_MARK_FLAG)
+          if (ht_marked_p (h))
 	    marked |= sweep_weak_table (h, 0);
 	}
     }
@@ -4250,7 +4315,7 @@ sweep_weak_hash_tables (void)
     {
       next = h->next_weak;
 
-      if (h->header.size & ARRAY_MARK_FLAG)
+      if (ht_marked_p (h))
 	{
 	  /* TABLE is marked as used.  Sweep its contents.  */
 	  if (h->count > 0)
@@ -5066,9 +5131,14 @@ If nil, use the current buffer." */ )
 }
 
 \f
+static void syms_of_fns_for_pdumper (void);
+
 void
 syms_of_fns (void)
 {
+  /* Not staticpro!  */
+  pdumper_remember_lv_raw_ptr (&weak_hash_tables, Lisp_Vectorlike);
+
   DEFSYM (Qmd5,    "md5");
   DEFSYM (Qsha1,   "sha1");
   DEFSYM (Qsha224, "sha224");
@@ -5232,4 +5302,20 @@ this variable.  */);
   defsubr (&Ssecure_hash);
   defsubr (&Sbuffer_hash);
   defsubr (&Slocale_info);
+
+  pdumper_do_now_and_after_load (syms_of_fns_for_pdumper);
+}
+
+static void
+syms_of_fns_for_pdumper (void)
+{
+  /* Rehash weak tables eagerly on dump load so that we don't have to
+     support running hash_rehash_if_needed during GC.  (Rehashing can
+     invoke user-defined code, and we can't support running arbitrary
+     user code during GC.  */
+  if (dumped_with_pdumper)
+    for (struct Lisp_Hash_Table *ht = weak_hash_tables;
+         ht != NULL;
+         ht = ht->next_weak)
+      hash_rehash_if_needed (ht);
 }
diff --git a/src/font.c b/src/font.c
index ce63233..7612f48 100644
--- a/src/font.c
+++ b/src/font.c
@@ -38,6 +38,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "fontset.h"
 #include "font.h"
 #include "termhooks.h"
+#include "pdumper.h"
 
 #ifdef HAVE_WINDOW_SYSTEM
 #include TERM_HEADER
@@ -5293,9 +5294,10 @@ syms_of_font (void)
   sort_shift_bits[FONT_SIZE_INDEX] = 16;
   sort_shift_bits[FONT_WIDTH_INDEX] = 23;
   /* Note that the other elements in sort_shift_bits are not used.  */
+  PDUMPER_REMEMBER_SCALAR (sort_shift_bits);
 
-  staticpro (&font_charset_alist);
   font_charset_alist = Qnil;
+  staticpro (&font_charset_alist);
 
   DEFSYM (Qopentype, "opentype");
 
@@ -5333,13 +5335,13 @@ syms_of_font (void)
 
   DEFSYM (QCuser_spec, ":user-spec");
 
-  staticpro (&scratch_font_spec);
   scratch_font_spec = Ffont_spec (0, NULL);
-  staticpro (&scratch_font_prefer);
+  staticpro (&scratch_font_spec);
   scratch_font_prefer = Ffont_spec (0, NULL);
+  staticpro (&scratch_font_prefer);
 
-  staticpro (&Vfont_log_deferred);
   Vfont_log_deferred = Fmake_vector (make_number (3), Qnil);
+  staticpro (&Vfont_log_deferred);
 
 #if 0
 #ifdef HAVE_LIBOTF
diff --git a/src/fontset.c b/src/fontset.c
index 38ff780..130293d 100644
--- a/src/fontset.c
+++ b/src/fontset.c
@@ -39,6 +39,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include TERM_HEADER
 #endif /* HAVE_WINDOW_SYSTEM */
 #include "font.h"
+#include "pdumper.h"
 
 /* FONTSET
 
@@ -2126,6 +2127,7 @@ syms_of_fontset (void)
      build_pure_c_string ("-*-*-*-*-*-*-*-*-*-*-*-*-fontset-default"));
   ASET (Vfontset_table, 0, Vdefault_fontset);
   next_fontset_id = 1;
+  PDUMPER_REMEMBER_SCALAR (next_fontset_id);
 
   auto_fontset_alist = Qnil;
   staticpro (&auto_fontset_alist);
diff --git a/src/frame.c b/src/frame.c
index b1d89f3..6e75dc3 100644
--- a/src/frame.c
+++ b/src/frame.c
@@ -52,6 +52,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #ifdef USE_X_TOOLKIT
 #include "widget.h"
 #endif
+#include "pdumper.h"
 
 /* The currently selected frame.  */
 
@@ -840,10 +841,7 @@ make_initial_frame (void)
   Lisp_Object frame;
 
   eassert (initial_kboard);
-
-  /* The first call must initialize Vframe_list.  */
-  if (! (NILP (Vframe_list) || CONSP (Vframe_list)))
-    Vframe_list = Qnil;
+  eassert (NILP (Vframe_list) || CONSP (Vframe_list));
 
   terminal = init_initial_terminal ();
 
@@ -4904,6 +4902,26 @@ make_monitor_attribute_list (struct MonitorInfo *monitors,
 				Initialization
  ***********************************************************************/
 
+static void init_frame_once_for_pdumper (void);
+
+void
+init_frame_once (void)
+{
+  staticpro (&Vframe_list);
+  staticpro (&selected_frame);
+  PDUMPER_IGNORE (last_nonminibuf_frame);
+  Vframe_list = Qnil;
+  selected_frame = Qnil;
+  pdumper_do_now_and_after_load (init_frame_once_for_pdumper);
+}
+
+static void
+init_frame_once_for_pdumper (void)
+{
+  PDUMPER_RESET_LV (Vframe_list, Qnil);
+  PDUMPER_RESET_LV (selected_frame, Qnil);
+}
+
 void
 syms_of_frame (void)
 {
@@ -5304,8 +5322,6 @@ This variable is effective only with the X toolkit (and there only when
 Gtk+ tooltips are not used) and on Windows.  */);
   tooltip_reuse_hidden_frame = false;
 
-  staticpro (&Vframe_list);
-
   defsubr (&Sframep);
   defsubr (&Sframe_live_p);
   defsubr (&Swindow_system);
diff --git a/src/fringe.c b/src/fringe.c
index 986bde1..4515186 100644
--- a/src/fringe.c
+++ b/src/fringe.c
@@ -29,6 +29,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "buffer.h"
 #include "blockinput.h"
 #include "termhooks.h"
+#include "pdumper.h"
 
 /* Fringe bitmaps are represented in three different ways:
 
@@ -1725,12 +1726,18 @@ mark_fringe_data (void)
 
 /* Initialize this module when Emacs starts.  */
 
+static void init_fringe_once_for_pdumper (void);
+
 void
 init_fringe_once (void)
 {
-  int bt;
+  pdumper_do_now_and_after_load (init_fringe_once_for_pdumper);
+}
 
-  for (bt = NO_FRINGE_BITMAP + 1; bt < MAX_STANDARD_FRINGE_BITMAPS; bt++)
+static void
+init_fringe_once_for_pdumper (void)
+{
+  for (int bt = NO_FRINGE_BITMAP + 1; bt < MAX_STANDARD_FRINGE_BITMAPS; bt++)
     init_fringe_bitmap (bt, &standard_bitmaps[bt], 1);
 }
 
diff --git a/src/ftcrfont.c b/src/ftcrfont.c
index 2676502..4532152 100644
--- a/src/ftcrfont.c
+++ b/src/ftcrfont.c
@@ -26,6 +26,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "blockinput.h"
 #include "font.h"
 #include "ftfont.h"
+#include "pdumper.h"
 
 /* FTCR font driver.  */
 
@@ -282,6 +283,8 @@ ftcrfont_draw (struct glyph_string *s,
 
 \f
 
+static void syms_of_ftcrfont_for_pdumper (void);
+
 void
 syms_of_ftcrfont (void)
 {
@@ -289,7 +292,11 @@ syms_of_ftcrfont (void)
     abort ();
 
   DEFSYM (Qftcr, "ftcr");
+}
 
+static void
+syms_of_ftcrfont_for_pdumper (void)
+{
   ftcrfont_driver = ftfont_driver;
   ftcrfont_driver.type = Qftcr;
   ftcrfont_driver.list = ftcrfont_list;
diff --git a/src/ftfont.c b/src/ftfont.c
index 1ae3f88..d1e234f 100644
--- a/src/ftfont.c
+++ b/src/ftfont.c
@@ -34,6 +34,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "composite.h"
 #include "font.h"
 #include "ftfont.h"
+#include "pdumper.h"
 
 /* Flag to tell if FcInit is already called or not.  */
 static bool fc_initialized;
@@ -2776,6 +2777,8 @@ ftfont_combining_capability (struct font *font)
 #endif
 }
 
+static void syms_of_ftfont_for_pdumper (void);
+
 void
 syms_of_ftfont (void)
 {
@@ -2799,6 +2802,13 @@ syms_of_ftfont (void)
   staticpro (&ft_face_cache);
   ft_face_cache = Qnil;
 
+  pdumper_do_now_and_after_load (syms_of_ftfont_for_pdumper);
+}
+
+static void
+syms_of_ftfont_for_pdumper (void)
+{
+  PDUMPER_RESET_LV (ft_face_cache, Qnil);
   ftfont_driver.type = Qfreetype;
   register_font_driver (&ftfont_driver, NULL);
 }
diff --git a/src/ftxfont.c b/src/ftxfont.c
index bfdeb40..ea7f476 100644
--- a/src/ftxfont.c
+++ b/src/ftxfont.c
@@ -28,6 +28,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "frame.h"
 #include "blockinput.h"
 #include "font.h"
+#include "pdumper.h"
 
 /* FTX font driver.  */
 
@@ -345,10 +346,18 @@ ftxfont_end_for_frame (struct frame *f)
 
 \f
 
+static void syms_of_ftxfont_for_pdumper (void);
+
 void
 syms_of_ftxfont (void)
 {
   DEFSYM (Qftx, "ftx");
+  pdumper_do_now_and_after_load (syms_of_ftxfont_for_pdumper);
+}
+
+static void
+syms_of_ftxfont_for_pdumper (void)
+{
 
   ftxfont_driver = ftfont_driver;
   ftxfont_driver.type = Qftx;
diff --git a/src/gnutls.c b/src/gnutls.c
index af2ba52..c7c708b 100644
--- a/src/gnutls.c
+++ b/src/gnutls.c
@@ -24,6 +24,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "process.h"
 #include "gnutls.h"
 #include "coding.h"
+#include "pdumper.h"
 
 #ifdef HAVE_GNUTLS
 
@@ -1726,6 +1727,7 @@ syms_of_gnutls (void)
         );
 #ifdef HAVE_GNUTLS
   gnutls_global_initialized = 0;
+  PDUMPER_IGNORE (gnutls_global_initialized);
 
   DEFSYM (Qgnutls_code, "gnutls-code");
   DEFSYM (Qgnutls_anon, "gnutls-anon");
diff --git a/src/image.c b/src/image.c
index d82fedb..3c818a2 100644
--- a/src/image.c
+++ b/src/image.c
@@ -45,6 +45,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "coding.h"
 #include "termhooks.h"
 #include "font.h"
+#include "pdumper.h"
 
 #ifdef HAVE_SYS_STAT_H
 #include <sys/stat.h>
@@ -9794,7 +9795,9 @@ void
 syms_of_image (void)
 {
   /* Initialize this only once; it will be reset before dumping.  */
+  /* The portable dumper will just leave it NULL, so no need to reset.  */
   image_types = NULL;
+  PDUMPER_IGNORE (image_types);
 
   /* Must be defined now because we're going to update it below, while
      defining the supported image types.  */
diff --git a/src/insdel.c b/src/insdel.c
index ed914ec..3b4594e 100644
--- a/src/insdel.c
+++ b/src/insdel.c
@@ -29,6 +29,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "buffer.h"
 #include "window.h"
 #include "region-cache.h"
+#include "pdumper.h"
 
 static void insert_from_string_1 (Lisp_Object, ptrdiff_t, ptrdiff_t, ptrdiff_t,
 				  ptrdiff_t, bool, bool);
@@ -1912,6 +1913,14 @@ prepare_to_modify_buffer_1 (ptrdiff_t start, ptrdiff_t end,
   if (!NILP (BVAR (current_buffer, read_only)))
     Fbarf_if_buffer_read_only (temp);
 
+  /* If we're about to modify a buffer the contents of which come from
+     a dump file, copy the contents to private storage first so we
+     don't take a COW fault on the buffer text and keep it around
+     forever.  */
+  if (pdumper_object_p (BEG_ADDR))
+    enlarge_buffer_text (current_buffer, 0);
+  eassert (!pdumper_object_p (BEG_ADDR));
+
   run_undoable_change();
 
   bset_redisplay (current_buffer);
diff --git a/src/intervals.h b/src/intervals.h
index 9a38d84..8ea36d9 100644
--- a/src/intervals.h
+++ b/src/intervals.h
@@ -26,7 +26,6 @@ INLINE_HEADER_BEGIN
 struct interval
 {
   /* The first group of entries deal with the tree structure.  */
-
   ptrdiff_t total_length;       /* Length of myself and both children.  */
   ptrdiff_t position;	        /* Cache of interval's character position.  */
 				/* This field is usually updated
diff --git a/src/keyboard.c b/src/keyboard.c
index 65938a5..c5a7f4d 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -66,6 +66,8 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 
 #include <ignore-value.h>
 
+#include "pdumper.h"
+
 #ifdef HAVE_WINDOW_SYSTEM
 #include TERM_HEADER
 #endif /* HAVE_WINDOW_SYSTEM */
@@ -10996,6 +10998,8 @@ static const struct event_head head_table[] = {
   {SYMBOL_INDEX (Qselect_window),       SYMBOL_INDEX (Qswitch_frame)}
 };
 
+static void syms_of_keyboard_for_pdumper (void);
+
 void
 syms_of_keyboard (void)
 {
@@ -11006,9 +11010,11 @@ syms_of_keyboard (void)
   staticpro (&Vlispy_mouse_stem);
 
   regular_top_level_message = build_pure_c_string ("Back to top level");
+  staticpro (&regular_top_level_message);
 #ifdef HAVE_STACK_OVERFLOW_HANDLING
   recover_top_level_message
     = build_pure_c_string ("Re-entering top level after C stack overflow");
+  staticpro (&recover_top_level_message);
 #endif
   DEFVAR_LISP ("internal--top-level-message", Vinternal__top_level_message,
 	       doc: /* Message displayed by `normal-top-level'.  */);
@@ -11820,7 +11826,38 @@ otherwise be lost.  If nil, crash immediately in response to fatal
 signals.  */);
   attempt_orderly_shutdown_on_fatal_signal = true;
 
+  pdumper_do_now_and_after_load (syms_of_keyboard_for_pdumper);
+}
+
+static void
+syms_of_keyboard_for_pdumper (void)
+{
+  /* Make sure input state is pristine when restoring from a dump.
+     init_keyboard() also resets some of these, but the duplication
+     doesn't hurt and makes sure that allocate_kboard and subsequent
+     early init functions see the environment they expect.  */
+
+  PDUMPER_RESET_LV (pending_funcalls, Qnil);
+  PDUMPER_RESET_LV (unread_switch_frame, Qnil);
+  PDUMPER_RESET_LV (internal_last_event_frame, Qnil);
+  PDUMPER_RESET_LV (last_command_event, Qnil);
+  PDUMPER_RESET_LV (last_nonmenu_event, Qnil);
+  PDUMPER_RESET_LV (last_input_event, Qnil);
+  PDUMPER_RESET_LV (Vunread_command_events, Qnil);
+  PDUMPER_RESET_LV (Vunread_post_input_method_events, Qnil);
+  PDUMPER_RESET_LV (Vunread_input_method_events, Qnil);
+  PDUMPER_RESET_LV (Vthis_command, Qnil);
+  PDUMPER_RESET_LV (Vreal_this_command, Qnil);
+  PDUMPER_RESET_LV (Vthis_command_keys_shift_translated, Qnil);
+  PDUMPER_RESET_LV (Vthis_original_command, Qnil);
+  PDUMPER_RESET (num_input_keys, 0);
+  PDUMPER_RESET (num_nonmacro_input_events, 0);
+  PDUMPER_RESET_LV (Vlast_event_frame, Qnil);
+  PDUMPER_RESET_LV (Vdeferred_action_list, Qnil);
+  PDUMPER_RESET_LV (Vdelayed_warnings_list, Qnil);
+
   /* Create the initial keyboard.  Qt means 'unset'.  */
+  eassert (initial_kboard == NULL);
   initial_kboard = allocate_kboard (Qt);
 }
 
@@ -11911,8 +11948,8 @@ mark_kboards (void)
   for (kb = all_kboards; kb; kb = kb->next_kboard)
     {
       if (kb->kbd_macro_buffer)
-	for (p = kb->kbd_macro_buffer; p < kb->kbd_macro_ptr; p++)
-	  mark_object (*p);
+        for (p = kb->kbd_macro_buffer; p < kb->kbd_macro_ptr; p++)
+          mark_object (*p);
       mark_object (KVAR (kb, Voverriding_terminal_local_map));
       mark_object (KVAR (kb, Vlast_command));
       mark_object (KVAR (kb, Vreal_last_command));
@@ -11936,17 +11973,17 @@ mark_kboards (void)
     union buffered_input_event *event;
     for (event = kbd_fetch_ptr; event != kbd_store_ptr; event++)
       {
-	if (event == kbd_buffer + KBD_BUFFER_SIZE)
-	  event = kbd_buffer;
-	/* These two special event types has no Lisp_Objects to mark.  */
-	if (event->kind != SELECTION_REQUEST_EVENT
-	    && event->kind != SELECTION_CLEAR_EVENT)
-	  {
-	    mark_object (event->ie.x);
-	    mark_object (event->ie.y);
-	    mark_object (event->ie.frame_or_window);
-	    mark_object (event->ie.arg);
-	  }
+        if (event == kbd_buffer + KBD_BUFFER_SIZE)
+          event = kbd_buffer;
+        /* These two special event types has no Lisp_Objects to mark.  */
+        if (event->kind != SELECTION_REQUEST_EVENT
+            && event->kind != SELECTION_CLEAR_EVENT)
+          {
+            mark_object (event->ie.x);
+            mark_object (event->ie.y);
+            mark_object (event->ie.frame_or_window);
+            mark_object (event->ie.arg);
+          }
       }
   }
 }
diff --git a/src/lisp.h b/src/lisp.h
index e087828..73ccdee 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -343,9 +343,13 @@ error !;
      (struct Lisp_Symbol *) ((intptr_t) XLI (a) - Lisp_Symbol \
 			     + (char *) lispsym))
 # define lisp_h_XTYPE(a) ((enum Lisp_Type) (XLI (a) & ~VALMASK))
+#if 0 /* XXX */
 # define lisp_h_XUNTAG(a, type) \
     __builtin_assume_aligned ((void *) (intptr_t) (XLI (a) - (type)), \
-			      GCALIGNMENT)
+                              GCALIGNMENT)
+#else
+# define lisp_h_XUNTAG(a, type)  ((void *) (intptr_t) (XLI (a) - (type)))
+#endif
 #endif
 
 /* When compiling via gcc -O0, define the key operations as macros, as
@@ -604,12 +608,42 @@ extern _Noreturn Lisp_Object wrong_type_argument (Lisp_Object, Lisp_Object);
 extern _Noreturn void wrong_choice (Lisp_Object, Lisp_Object);
 
 /* Defined in emacs.c.  */
-#ifdef DOUG_LEA_MALLOC
-extern bool might_dump;
+
+extern struct gflags {
+  /* Set after Emacs has started up the first time.
+     Prevents reinitialization of the Lisp world and keymaps
+     on subsequent starts.  */
+  bool initialized_ : 1;
+  /* True means this Emacs instance was born to dump.  */
+  bool will_dump_ : 1;
+  bool will_dump_with_pdumper_ : 1;
+#ifndef CANNOT_DUMP
+  bool will_dump_with_unexec_ : 1;
+#endif
+#ifndef CANNOT_DUMP
+  /* Set in an Emacs process that has been restored from an unexec
+     dump.  */
+  bool dumped_with_unexec_ : 1;
 #endif
-/* True means Emacs has already been initialized.
-   Used during startup to detect startup of dumped Emacs.  */
-extern bool initialized;
+  /* Set in an Emacs process that has been restored from a portable
+     dump.  */
+  bool dumped_with_pdumper_ : 1;
+} gflags;
+
+#define CONST_GFLAGS ((const struct gflags *)(&gflags))
+
+#define initialized CONST_GFLAGS->initialized_
+#define will_dump CONST_GFLAGS->will_dump_
+#define will_dump_with_pdumper CONST_GFLAGS->will_dump_with_pdumper_
+#ifdef CANNOT_DUMP
+# define will_dump_with_unexec false
+# define dumped_with_unexec false
+#else
+# define will_dump_with_unexec CONST_GFLAGS->will_dump_with_unexec_
+# define dumped_with_unexec CONST_GFLAGS->dumped_with_unexec_
+#endif
+
+#define dumped_with_pdumper CONST_GFLAGS->dumped_with_pdumper_
 
 /* Defined in floatfns.c.  */
 extern double extract_float (Lisp_Object);
@@ -744,6 +778,19 @@ struct Lisp_Symbol
 # define DEFINE_NON_NIL_Q_SYMBOL_MACROS true
 #endif
 
+/* True if N is a power of 2.  N should be positive.  */
+
+#define POWER_OF_2(n) (((n) & ((n) - 1)) == 0)
+
+/* Return X rounded to the next multiple of Y.  Y should be positive,
+   and Y - 1 + X should not overflow.  Arguments should not have side
+   effects, as they are evaluated more than once.  Tune for Y being a
+   power of 2.  */
+
+#define ROUNDUP(x, y) (POWER_OF_2 (y)					\
+                       ? ((y) - 1 + (x)) & ~ ((y) - 1)			\
+                       : ((y) - 1 + (x)) - ((y) - 1 + (x)) % (y))
+
 #include "globals.h"
 
 /* Convert a Lisp_Object to the corresponding EMACS_INT and vice versa.
@@ -2279,6 +2326,9 @@ struct Lisp_Finalizer
     Lisp_Object function;
   };
 
+extern struct Lisp_Finalizer finalizers;
+extern struct Lisp_Finalizer doomed_finalizers;
+
 /* A miscellaneous object, when it's on the free list.  */
 struct Lisp_Free
   {
@@ -2664,8 +2714,16 @@ BUFFER_OBJFWDP (union Lisp_Fwd *a)
   return XFWDTYPE (a) == Lisp_Fwd_Buffer_Obj;
 }
 
+INLINE enum pvec_type
+PSEUDOVECTOR_TYPE (const struct vectorlike_header *a)
+{
+  return ((a->size & PSEUDOVECTOR_FLAG)
+          ? (a->size & PVEC_TYPE_MASK) >> PSEUDOVECTOR_AREA_BITS
+          : PVEC_NORMAL_VECTOR);
+}
+
 INLINE bool
-PSEUDOVECTOR_TYPEP (struct vectorlike_header *a, int code)
+PSEUDOVECTOR_TYPEP (const struct vectorlike_header *a, int code)
 {
   return ((a->size & (PSEUDOVECTOR_FLAG | PVEC_TYPE_MASK))
 	  == (PSEUDOVECTOR_FLAG | (code << PSEUDOVECTOR_AREA_BITS)));
@@ -2920,6 +2978,16 @@ CHECK_NUMBER_CDR (Lisp_Object x)
   XSETCDR (x, tmp);
 }
 \f
+
+/* If we're not dumping and we might be using the portable dumper, try
+   to bunch all the subr structures together for more efficient dump
+   loading.  */
+#ifdef CANNOT_DUMP
+# define SUBR_SECTION_ATTRIBUTE ATTRIBUTE_SECTION (".subrs")
+#else
+# define SUBR_SECTION_ATTRIBUTE
+#endif
+
 /* Define a built-in function for calling from Lisp.
  `lname' should be the name to give the function in Lisp,
     as a null-terminated C string.
@@ -2950,6 +3018,7 @@ CHECK_NUMBER_CDR (Lisp_Object x)
 #ifdef _MSC_VER
 #define DEFUN(lname, fnname, sname, minargs, maxargs, intspec, doc)	\
    Lisp_Object fnname DEFUN_ARGS_ ## maxargs ;				\
+   SUBR_SECTION_ATTRIBUTE                                               \
    static struct Lisp_Subr alignas (GCALIGNMENT) sname =		\
    { { (PVEC_SUBR << PSEUDOVECTOR_AREA_BITS)				\
        | (sizeof (struct Lisp_Subr) / sizeof (EMACS_INT)) },		\
@@ -2958,6 +3027,7 @@ CHECK_NUMBER_CDR (Lisp_Object x)
    Lisp_Object fnname
 #else  /* not _MSC_VER */
 #define DEFUN(lname, fnname, sname, minargs, maxargs, intspec, doc)	\
+   SUBR_SECTION_ATTRIBUTE                                               \
    static struct Lisp_Subr alignas (GCALIGNMENT) sname =		\
      { { PVEC_SUBR << PSEUDOVECTOR_AREA_BITS },				\
        { .a ## maxargs = fnname },					\
@@ -3234,6 +3304,11 @@ extern Lisp_Object Vascii_canon_table;
 /* Call staticpro (&var) to protect static variable `var'.  */
 
 void staticpro (Lisp_Object *);
+
+enum { NSTATICS = 2048 };
+extern Lisp_Object *staticvec[NSTATICS];
+extern int staticidx;
+
 \f
 /* Forward declarations for prototypes.  */
 struct window;
@@ -3426,9 +3501,11 @@ extern void syms_of_syntax (void);
 enum { NEXT_ALMOST_PRIME_LIMIT = 11 };
 extern EMACS_INT next_almost_prime (EMACS_INT) ATTRIBUTE_CONST;
 extern Lisp_Object larger_vector (Lisp_Object, ptrdiff_t, ptrdiff_t);
-extern void sweep_weak_hash_tables (void);
+extern void mark_and_sweep_weak_hash_tables (void);
 EMACS_UINT hash_string (char const *, ptrdiff_t);
 EMACS_UINT sxhash (Lisp_Object, int);
+EMACS_UINT hashfn_eql (struct hash_table_test *ht, Lisp_Object key);
+EMACS_UINT hashfn_equal (struct hash_table_test *ht, Lisp_Object key);
 Lisp_Object make_hash_table (struct hash_table_test, Lisp_Object, Lisp_Object,
                              Lisp_Object, Lisp_Object);
 ptrdiff_t hash_lookup (struct Lisp_Hash_Table *, Lisp_Object, EMACS_UINT *);
@@ -3590,6 +3667,9 @@ extern Lisp_Object *stack_base;
 extern EMACS_INT consing_since_gc;
 extern EMACS_INT gc_relative_threshold;
 extern EMACS_INT memory_full_cons_threshold;
+#ifdef ENABLE_CHECKING
+extern Lisp_Object Vdead;
+#endif
 extern Lisp_Object list1 (Lisp_Object);
 extern Lisp_Object list2 (Lisp_Object, Lisp_Object);
 extern Lisp_Object list3 (Lisp_Object, Lisp_Object, Lisp_Object);
@@ -3599,6 +3679,21 @@ extern Lisp_Object list5 (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object,
 enum constype {CONSTYPE_HEAP, CONSTYPE_PURE};
 extern Lisp_Object listn (enum constype, ptrdiff_t, Lisp_Object, ...);
 
+enum gc_root_type {
+  GC_ROOT_STATICPRO,
+  GC_ROOT_BUFFER_LOCAL_DEFAULT,
+  GC_ROOT_BUFFER_LOCAL_NAME,
+  GC_ROOT_C_SYMBOL
+};
+
+struct gc_root_visitor {
+  void (*visit)(Lisp_Object *root_ptr,
+                enum gc_root_type type,
+                void *data);
+  void *data;
+};
+extern void visit_static_gc_roots (struct gc_root_visitor visitor);
+
 /* Build a frequently used 2/3/4-integer lists.  */
 
 INLINE Lisp_Object
@@ -3627,6 +3722,7 @@ extern Lisp_Object make_string (const char *, ptrdiff_t);
 extern Lisp_Object make_formatted_string (char *, const char *, ...)
   ATTRIBUTE_FORMAT_PRINTF (2, 3);
 extern Lisp_Object make_unibyte_string (const char *, ptrdiff_t);
+extern ptrdiff_t vector_nbytes (struct Lisp_Vector *v);
 
 /* Make unibyte string from C string when the length isn't known.  */
 
@@ -3827,7 +3923,7 @@ extern Lisp_Object string_to_number (char const *, int, bool);
 extern void map_obarray (Lisp_Object, void (*) (Lisp_Object, Lisp_Object),
                          Lisp_Object);
 extern void dir_warning (const char *, Lisp_Object);
-extern void init_obarray (void);
+extern void init_obarray_once (void);
 extern void init_lread (void);
 extern void syms_of_lread (void);
 
@@ -3847,7 +3943,6 @@ intern_c_string (const char *str)
 extern Lisp_Object Vautoload_queue;
 extern Lisp_Object Vrun_hooks;
 extern Lisp_Object Vsignaling_function;
-extern Lisp_Object inhibit_lisp_code;
 extern struct handler *handlerlist;
 
 /* To run a normal hook, use the appropriate function from the list below.
@@ -3945,7 +4040,7 @@ extern _Noreturn void time_overflow (void);
 extern Lisp_Object make_buffer_string (ptrdiff_t, ptrdiff_t, bool);
 extern Lisp_Object make_buffer_string_both (ptrdiff_t, ptrdiff_t, ptrdiff_t,
 					    ptrdiff_t, bool);
-extern void init_editfns (bool);
+extern void init_editfns (void);
 extern void syms_of_editfns (void);
 
 /* Defined in buffer.c.  */
@@ -3960,7 +4055,7 @@ extern bool overlay_touches_p (ptrdiff_t);
 extern Lisp_Object other_buffer_safely (Lisp_Object);
 extern Lisp_Object get_truename_buffer (Lisp_Object);
 extern void init_buffer_once (void);
-extern void init_buffer (int);
+extern void init_buffer (void);
 extern void syms_of_buffer (void);
 extern void keys_of_buffer (void);
 
@@ -4104,6 +4199,7 @@ extern void store_in_alist (Lisp_Object *, Lisp_Object, Lisp_Object);
 extern Lisp_Object do_switch_frame (Lisp_Object, int, int, Lisp_Object);
 extern Lisp_Object get_frame_param (struct frame *, Lisp_Object);
 extern void frames_discard_buffer (Lisp_Object);
+extern void init_frame_once (void);
 extern void syms_of_frame (void);
 
 /* Defined in emacs.c.  */
diff --git a/src/lread.c b/src/lread.c
index eab9b8b..9466827 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -42,6 +42,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "systime.h"
 #include "termhooks.h"
 #include "blockinput.h"
+#include "pdumper.h"
 #include <c-ctype.h>
 
 #ifdef MSDOS
@@ -1826,7 +1827,7 @@ readevalloop (Lisp_Object readcharfun,
 	     ? Qnil : list1 (Qt)));
 
   /* Try to ensure sourcename is a truename, except whilst preloading.  */
-  if (NILP (Vpurify_flag)
+  if (!will_dump
       && !NILP (sourcename) && !NILP (Ffile_name_absolute_p (sourcename))
       && !NILP (Ffboundp (Qfile_truename)))
     sourcename = call1 (Qfile_truename, sourcename) ;
@@ -3980,13 +3981,6 @@ usage: (unintern NAME OBARRAY)  */)
   if (SYMBOLP (name) && !EQ (name, tem))
     return Qnil;
 
-  /* There are plenty of other symbols which will screw up the Emacs
-     session if we unintern them, as well as even more ways to use
-     `setq' or `fset' or whatnot to make the Emacs session
-     unusable.  Let's not go down this silly road.  --Stef  */
-  /* if (EQ (tem, Qnil) || EQ (tem, Qt))
-       error ("Attempt to unintern t or nil"); */
-
   XSYMBOL (tem)->interned = SYMBOL_UNINTERNED;
 
   hash = oblookup_last_bucket_number;
@@ -4101,11 +4095,12 @@ OBARRAY defaults to the value of `obarray'.  */)
 
 #define OBARRAY_SIZE 1511
 
+static void init_obarray_once_for_pdumper (void);
+
 void
-init_obarray (void)
+init_obarray_once (void)
 {
   Lisp_Object oblength;
-  ptrdiff_t size = 100 + MAX_MULTIBYTE_LENGTH;
 
   XSETFASTINT (oblength, OBARRAY_SIZE);
 
@@ -4128,15 +4123,25 @@ init_obarray (void)
   XSYMBOL (Qt)->constant = 1;
   XSYMBOL (Qt)->declared_special = true;
 
-  /* Qt is correct even if CANNOT_DUMP.  loadup.el will set to nil at end.  */
+  /* Qt is correct even if not dumping.  loadup.el will set to nil at end.  */
   Vpurify_flag = Qt;
 
   DEFSYM (Qvariable_documentation, "variable-documentation");
 
+  pdumper_do_now_and_after_load (init_obarray_once_for_pdumper);
+}
+
+static void
+init_obarray_once_for_pdumper (void)
+{
+  ptrdiff_t size = 100 + MAX_MULTIBYTE_LENGTH;
   read_buffer = xmalloc (size);
   read_buffer_size = size;
 }
+
 \f
+int ndefsubr;
+
 void
 defsubr (struct Lisp_Subr *sname)
 {
@@ -4145,6 +4150,7 @@ defsubr (struct Lisp_Subr *sname)
   XSETPVECTYPE (sname, PVEC_SUBR);
   XSETSUBR (tem, sname);
   set_symbol_function (sym, tem);
+  ++ndefsubr;
 }
 
 #ifdef NOTDEF /* Use fset in subr.el now!  */
@@ -4262,9 +4268,9 @@ load_path_check (Lisp_Object lpath)
    are running uninstalled.
 
    Uses the following logic:
-   If CANNOT_DUMP: Use PATH_LOADSEARCH.
-   The remainder is what happens when dumping works:
-   If purify-flag (ie dumping) just use PATH_DUMPLOADSEARCH.
+   If !will_dump: Use PATH_LOADSEARCH.
+   The remainder is what happens when dumping is about to happen:
+   If dumping, just use PATH_DUMPLOADSEARCH.
    Otherwise use PATH_LOADSEARCH.
 
    If !initialized, then just return PATH_DUMPLOADSEARCH.
@@ -4287,129 +4293,109 @@ load_path_check (Lisp_Object lpath)
 static Lisp_Object
 load_path_default (void)
 {
+  if (will_dump)
+    /* PATH_DUMPLOADSEARCH is the lisp dir in the source directory.
+       We used to add ../lisp (ie the lisp dir in the build
+       directory) at the front here, but that should not be
+       necessary, since in out of tree builds lisp/ is empty, save
+       for Makefile.  */
+    return decode_env_path (0, PATH_DUMPLOADSEARCH, 0);
+
   Lisp_Object lpath = Qnil;
-  const char *normal;
+  const char *normal = PATH_LOADSEARCH;
+  const char *loadpath = NULL;
 
-#ifdef CANNOT_DUMP
 #ifdef HAVE_NS
-  const char *loadpath = ns_load_path ();
+  loadpath = ns_load_path ();
 #endif
 
-  normal = PATH_LOADSEARCH;
-#ifdef HAVE_NS
   lpath = decode_env_path (0, loadpath ? loadpath : normal, 0);
-#else
-  lpath = decode_env_path (0, normal, 0);
-#endif
 
-#else  /* !CANNOT_DUMP */
-
-  normal = NILP (Vpurify_flag) ? PATH_LOADSEARCH : PATH_DUMPLOADSEARCH;
-
-  if (initialized)
+  if (!NILP (Vinstallation_directory))
     {
-#ifdef HAVE_NS
-      const char *loadpath = ns_load_path ();
-      lpath = decode_env_path (0, loadpath ? loadpath : normal, 0);
-#else
-      lpath = decode_env_path (0, normal, 0);
-#endif
-      if (!NILP (Vinstallation_directory))
+      Lisp_Object tem, tem1;
+
+      /* Add to the path the lisp subdir of the installation
+         dir, if it is accessible.  Note: in out-of-tree builds,
+         this directory is empty save for Makefile.  */
+      tem = Fexpand_file_name (build_string ("lisp"),
+                               Vinstallation_directory);
+      tem1 = Ffile_accessible_directory_p (tem);
+      if (!NILP (tem1))
         {
-          Lisp_Object tem, tem1;
+          if (NILP (Fmember (tem, lpath)))
+            {
+              /* We are running uninstalled.  The default load-path
+                 points to the eventual installed lisp directories.
+                 We should not use those now, even if they exist,
+                 so start over from a clean slate.  */
+              lpath = list1 (tem);
+            }
+        }
+      else
+        /* That dir doesn't exist, so add the build-time
+           Lisp dirs instead.  */
+        {
+          Lisp_Object dump_path =
+            decode_env_path (0, PATH_DUMPLOADSEARCH, 0);
+          lpath = nconc2 (lpath, dump_path);
+        }
 
-          /* Add to the path the lisp subdir of the installation
-             dir, if it is accessible.  Note: in out-of-tree builds,
-             this directory is empty save for Makefile.  */
-          tem = Fexpand_file_name (build_string ("lisp"),
+      /* Add site-lisp under the installation dir, if it exists.  */
+      if (!no_site_lisp)
+        {
+          tem = Fexpand_file_name (build_string ("site-lisp"),
                                    Vinstallation_directory);
           tem1 = Ffile_accessible_directory_p (tem);
           if (!NILP (tem1))
             {
               if (NILP (Fmember (tem, lpath)))
-                {
-                  /* We are running uninstalled.  The default load-path
-                     points to the eventual installed lisp directories.
-                     We should not use those now, even if they exist,
-                     so start over from a clean slate.  */
-                  lpath = list1 (tem);
-                }
-            }
-          else
-            /* That dir doesn't exist, so add the build-time
-               Lisp dirs instead.  */
-            {
-              Lisp_Object dump_path =
-                decode_env_path (0, PATH_DUMPLOADSEARCH, 0);
-              lpath = nconc2 (lpath, dump_path);
+                lpath = Fcons (tem, lpath);
             }
+        }
 
-          /* Add site-lisp under the installation dir, if it exists.  */
-          if (!no_site_lisp)
-            {
-              tem = Fexpand_file_name (build_string ("site-lisp"),
-                                       Vinstallation_directory);
-              tem1 = Ffile_accessible_directory_p (tem);
-              if (!NILP (tem1))
-                {
-                  if (NILP (Fmember (tem, lpath)))
-                    lpath = Fcons (tem, lpath);
-                }
-            }
+      /* If Emacs was not built in the source directory,
+         and it is run from where it was built, add to load-path
+         the lisp and site-lisp dirs under that directory.  */
+
+      if (NILP (Fequal (Vinstallation_directory, Vsource_directory)))
+        {
+          Lisp_Object tem2;
 
-          /* If Emacs was not built in the source directory,
-             and it is run from where it was built, add to load-path
-             the lisp and site-lisp dirs under that directory.  */
+          tem = Fexpand_file_name (build_string ("src/Makefile"),
+                                   Vinstallation_directory);
+          tem1 = Ffile_exists_p (tem);
 
-          if (NILP (Fequal (Vinstallation_directory, Vsource_directory)))
+          /* Don't be fooled if they moved the entire source tree
+             AFTER dumping Emacs.  If the build directory is indeed
+             different from the source dir, src/Makefile.in and
+             src/Makefile will not be found together.  */
+          tem = Fexpand_file_name (build_string ("src/Makefile.in"),
+                                   Vinstallation_directory);
+          tem2 = Ffile_exists_p (tem);
+          if (!NILP (tem1) && NILP (tem2))
             {
-              Lisp_Object tem2;
-
-              tem = Fexpand_file_name (build_string ("src/Makefile"),
-                                       Vinstallation_directory);
-              tem1 = Ffile_exists_p (tem);
-
-              /* Don't be fooled if they moved the entire source tree
-                 AFTER dumping Emacs.  If the build directory is indeed
-                 different from the source dir, src/Makefile.in and
-                 src/Makefile will not be found together.  */
-              tem = Fexpand_file_name (build_string ("src/Makefile.in"),
-                                       Vinstallation_directory);
-              tem2 = Ffile_exists_p (tem);
-              if (!NILP (tem1) && NILP (tem2))
-                {
-                  tem = Fexpand_file_name (build_string ("lisp"),
-                                           Vsource_directory);
+              tem = Fexpand_file_name (build_string ("lisp"),
+                                       Vsource_directory);
 
-                  if (NILP (Fmember (tem, lpath)))
-                    lpath = Fcons (tem, lpath);
+              if (NILP (Fmember (tem, lpath)))
+                lpath = Fcons (tem, lpath);
 
-                  if (!no_site_lisp)
+              if (!no_site_lisp)
+                {
+                  tem = Fexpand_file_name (build_string ("site-lisp"),
+                                           Vsource_directory);
+                  tem1 = Ffile_accessible_directory_p (tem);
+                  if (!NILP (tem1))
                     {
-                      tem = Fexpand_file_name (build_string ("site-lisp"),
-                                               Vsource_directory);
-                      tem1 = Ffile_accessible_directory_p (tem);
-                      if (!NILP (tem1))
-                        {
-                          if (NILP (Fmember (tem, lpath)))
-                            lpath = Fcons (tem, lpath);
-                        }
+                      if (NILP (Fmember (tem, lpath)))
+                        lpath = Fcons (tem, lpath);
                     }
                 }
-            } /* Vinstallation_directory != Vsource_directory */
+            }
+        } /* Vinstallation_directory != Vsource_directory */
 
-        } /* if Vinstallation_directory */
-    }
-  else                          /* !initialized */
-    {
-      /* NORMAL refers to PATH_DUMPLOADSEARCH, ie the lisp dir in the
-         source directory.  We used to add ../lisp (ie the lisp dir in
-         the build directory) at the front here, but that should not
-         be necessary, since in out of tree builds lisp/ is empty, save
-         for Makefile.  */
-      lpath = decode_env_path (0, normal, 0);
-    }
-#endif /* !CANNOT_DUMP */
+    } /* if Vinstallation_directory */
 
   return lpath;
 }
@@ -4420,11 +4406,7 @@ init_lread (void)
   /* First, set Vload_path.  */
 
   /* Ignore EMACSLOADPATH when dumping.  */
-#ifdef CANNOT_DUMP
-  bool use_loadpath = true;
-#else
-  bool use_loadpath = NILP (Vpurify_flag);
-#endif
+  bool use_loadpath = !will_dump;
 
   if (use_loadpath && egetenv ("EMACSLOADPATH"))
     {
@@ -4475,7 +4457,7 @@ init_lread (void)
       load_path_check (Vload_path);
 
       /* Add the site-lisp directories at the front.  */
-      if (initialized && !no_site_lisp && PATH_SITELOADSEARCH[0] != '\0')
+      if (!will_dump && !no_site_lisp && PATH_SITELOADSEARCH[0] != '\0')
         {
           Lisp_Object sitelisp;
           sitelisp = decode_env_path (0, PATH_SITELOADSEARCH, 0);
diff --git a/src/macfont.m b/src/macfont.m
index 3af9edc..233fa47 100644
--- a/src/macfont.m
+++ b/src/macfont.m
@@ -35,6 +35,7 @@
 #include "nsterm.h"
 #include "macfont.h"
 #include "macuvs.h"
+#include "pdumper.h"
 
 #include <libkern/OSByteOrder.h>
 
@@ -1032,12 +1033,12 @@ sorted in the canonical order (CTFontManagerCompareFontFamilyNames on
 static void
 macfont_init_font_change_handler (void)
 {
-  static bool initialized = false;
+  static bool xinitialized = false;
 
-  if (initialized)
+  if (xinitialized)
     return;
 
-  initialized = true;
+  xinitialized = true;
   CFNotificationCenterAddObserver
     (CFNotificationCenterGetLocalCenter (), NULL,
      macfont_handle_font_change_notification,
@@ -4054,13 +4055,12 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
 }
 
 \f
+
 void
 syms_of_macfont (void)
 {
   /* Core Text, for macOS.  */
   DEFSYM (Qmac_ct, "mac-ct");
-  macfont_driver.type = Qmac_ct;
-  register_font_driver (&macfont_driver, NULL);
 
   /* The font property key specifying the font design destination.  The
      value is an unsigned integer code: 0 for WYSIWYG, and 1 for Video
@@ -4075,4 +4075,18 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
 
   macfont_family_cache = Qnil;
   staticpro (&macfont_family_cache);
+
+  pdumper_do_now_and_after_load (syms_of_macfont_for_pdumper);
+}
+
+static void
+syms_of_macfont_for_pdumper (void)
+{
+  if (RESTORING_FROM_DUMP)
+    macfont_family_cache = Qnil;
+  else
+    eassert (NILP (macfont_family_cache));
+
+  macfont_driver.type = Qmac_ct;
+  register_font_driver (&macfont_driver, NULL);
 }
diff --git a/src/menu.c b/src/menu.c
index 638810b..d530689 100644
--- a/src/menu.c
+++ b/src/menu.c
@@ -1574,9 +1574,10 @@ for instance using the window manager, then this produces a quit and
 void
 syms_of_menu (void)
 {
-  staticpro (&menu_items);
   menu_items = Qnil;
+  staticpro (&menu_items);
   menu_items_inuse = Qnil;
+  staticpro (&menu_items_inuse);
 
   defsubr (&Sx_popup_menu);
   defsubr (&Sx_popup_dialog);
diff --git a/src/minibuf.c b/src/minibuf.c
index 57eea05..ffd1d9c 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -32,6 +32,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "window.h"
 #include "keymap.h"
 #include "systty.h"
+#include "pdumper.h"
 
 /* List of buffers for use as minibuffers.
    The first element of the list is used for the outermost minibuffer
@@ -1888,21 +1889,36 @@ If no minibuffer is active, return nil.  */)
 }
 
 \f
+
+static void init_minibuf_once_for_pdumper (void);
+
 void
 init_minibuf_once (void)
 {
-  Vminibuffer_list = Qnil;
   staticpro (&Vminibuffer_list);
+  pdumper_do_now_and_after_load (init_minibuf_once_for_pdumper);
 }
 
-void
-syms_of_minibuf (void)
+static void
+init_minibuf_once_for_pdumper (void)
 {
+  PDUMPER_IGNORE (minibuf_level);
+  PDUMPER_IGNORE (minibuf_prompt_width);
+
+  /* We run this function on first initialization and whenever we
+     restore from a pdumper image.  pdumper doesn't try to preserve
+     frames, windows, and so on, so reset everything related here.  */
+  Vminibuffer_list = Qnil;
   minibuf_level = 0;
   minibuf_prompt = Qnil;
-  staticpro (&minibuf_prompt);
-
   minibuf_save_list = Qnil;
+  last_minibuf_string = Qnil;
+}
+
+void
+syms_of_minibuf (void)
+{
+  staticpro (&minibuf_prompt);
   staticpro (&minibuf_save_list);
 
   DEFSYM (Qcompletion_ignore_case, "completion-ignore-case");
@@ -1912,7 +1928,6 @@ syms_of_minibuf (void)
   DEFSYM (Qminibuffer_completion_table, "minibuffer-completion-table");
 
   staticpro (&last_minibuf_string);
-  last_minibuf_string = Qnil;
 
   DEFSYM (Qminibuffer_history, "minibuffer-history");
   DEFSYM (Qbuffer_name_history, "buffer-name-history");
diff --git a/src/nsfns.m b/src/nsfns.m
index cfaaf53..340fc9c 100644
--- a/src/nsfns.m
+++ b/src/nsfns.m
@@ -49,7 +49,6 @@ Updated by Christian Limpach (chris@nice.ch)
 #include "macfont.h"
 #endif
 
-
 #ifdef HAVE_NS
 
 static EmacsTooltip *ns_tooltip = nil;
@@ -3068,6 +3067,11 @@ - (NSString *)panel: (id)sender userEnteredFilename: (NSString *)filename
 
    ========================================================================== */
 
+static void
+syms_of_nsfns_1 (void)
+{
+
+}
 
 void
 syms_of_nsfns (void)
@@ -3147,5 +3151,6 @@ - (NSString *)panel: (id)sender userEnteredFilename: (NSString *)filename
 
   as_status = 0;
   as_script = Qnil;
+  staticpro (&as_script);
   as_result = 0;
 }
diff --git a/src/nsfont.m b/src/nsfont.m
index 389d0ed..236bda2 100644
--- a/src/nsfont.m
+++ b/src/nsfont.m
@@ -36,6 +36,7 @@
 #include "character.h"
 #include "font.h"
 #include "termchar.h"
+#include "pdumper.h"
 
 /* TODO: Drop once we can assume gnustep-gui 0.17.1. */
 #ifdef NS_IMPL_GNUSTEP
@@ -1520,12 +1521,11 @@ - (void)setIntAttribute: (NSInteger)attributeTag value: (NSInteger)val
   fprintf (stderr, "\n");
 }
 
+static void syms_of_nsfont_for_pdumper (void);
 
 void
 syms_of_nsfont (void)
 {
-  nsfont_driver.type = Qns;
-  register_font_driver (&nsfont_driver, NULL);
   DEFSYM (Qcondensed, "condensed");
   DEFSYM (Qexpanded, "expanded");
   DEFSYM (Qapple, "apple");
@@ -1533,5 +1533,13 @@ - (void)setIntAttribute: (NSInteger)attributeTag value: (NSInteger)val
   DEFVAR_LISP ("ns-reg-to-script", Vns_reg_to_script,
                doc: /* Internal use: maps font registry to Unicode script. */);
 
+  pdumper_do_now_and_after_load (syms_of_nsfont_for_pdumper);
+}
+
+static void
+syms_of_nsfont_for_pdumper (void)
+{
+  nsfont_driver.type = Qns;
+  register_font_driver (&nsfont_driver, NULL);
   ascii_printable = NULL;
 }
diff --git a/src/nsmenu.m b/src/nsmenu.m
index 3e9887a..ed2e91e 100644
--- a/src/nsmenu.m
+++ b/src/nsmenu.m
@@ -37,6 +37,7 @@
 #include "termhooks.h"
 #include "keyboard.h"
 #include "menu.h"
+#include "pdumper.h"
 
 #define NSMENUPROFILE 0
 
@@ -1883,6 +1884,7 @@ - (Lisp_Object)runDialogAt: (NSPoint)p
   /* Don't know how to keep track of this in Next/Open/GNUstep.  Always
      update menus there.  */
   trackingMenu = 1;
+  PDUMPER_REMEMBER_SCALAR (trackingMenu);
 #endif
   defsubr (&Sns_reset_menu);
   defsubr (&Smenu_or_popup_active_p);
diff --git a/src/nsterm.m b/src/nsterm.m
index 7e6ec85..a7a62cb 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -59,6 +59,7 @@ Updated by Christian Limpach (chris@nice.ch)
 #include "keyboard.h"
 #include "buffer.h"
 #include "font.h"
+#include "pdumper.h"
 
 #ifdef NS_IMPL_GNUSTEP
 #include "process.h"
@@ -8620,6 +8621,7 @@ Convert an X font name (XLFD) to an NS font name.
   NSTRACE ("syms_of_nsterm");
 
   ns_antialias_threshold = 10.0;
+  PDUMPER_REMEMBER_SCALAR (ns_antialias_threshold);
 
   /* from 23+ we need to tell emacs what modifiers there are.. */
   DEFSYM (Qmodifier_value, "modifier-value");
diff --git a/src/pdumper.c b/src/pdumper.c
new file mode 100644
index 0000000..2efbad9
--- /dev/null
+++ b/src/pdumper.c
@@ -0,0 +1,3197 @@
+#include <config.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/param.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "blockinput.h"
+#include "buffer.h"
+#include "charset.h"
+#include "coding.h"
+#include "frame.h"
+#include "getpagesize.h"
+#include "intervals.h"
+#include "lisp.h"
+#include "pdumper.h"
+#include "window.h"
+#include "fingerprint.h"
+
+/* We require an architecture in which all pointers are the same size
+   and have the same layout.  */
+verify (sizeof (ptrdiff_t) == sizeof (void*));
+verify (sizeof (void (*)(void)) == sizeof (void*));
+verify (sizeof (ptrdiff_t) <= sizeof (Lisp_Object));
+verify (sizeof (ptrdiff_t) <= sizeof (EMACS_INT));
+
+bool pdumper_loading_dump;
+
+static const char dump_magic[16] = {
+  'D', 'U', 'M', 'P', 'E', 'D',
+  'G', 'N', 'U',
+  'E', 'M', 'A', 'C', 'S'
+};
+
+static pdumper_hook dump_hooks[24];
+static int nr_dump_hooks = 0;
+
+static struct
+{
+  void *mem;
+  int sz;
+} remembered_data[32];
+static int nr_remembered_data = 0;
+
+/* Maximum number of cons cells in a list to print in a contiguous
+   chunk before going back to the normal dumping strategy.  */
+static const ptrdiff_t max_cons_chain_depth = 256;
+
+typedef int32_t dump_off_t;
+
+enum dump_reloc_type
+  {
+    /* dump_ptr = dump_ptr + emacs_basis() */
+    RELOC_DUMP_TO_EMACS_RAW_PTR,
+    /* dump_ptr = dump_ptr + dump_base */
+    RELOC_DUMP_TO_DUMP_RAW_PTR,
+    /* dump_lv = make_lisp_ptr (
+         dump_lv + dump_base,
+         type - RELOC_DUMP_TO_DUMP_LV)
+       (Special case for symbols: make_lisp_symbol)
+       Must be second-last.  */
+    RELOC_DUMP_TO_DUMP_LV,
+    /* dump_lv = make_lisp_ptr (
+         dump_lv + emacs_basis(),
+         type - RELOC_DUMP_TO_DUMP_LV)
+       (Special case for symbols: make_lisp_symbol.)
+       Must be last.  */
+    RELOC_DUMP_TO_EMACS_LV = RELOC_DUMP_TO_DUMP_LV + 8,
+  };
+
+enum emacs_reloc_type
+  {
+    /* Copy raw bytes from the dump into Emacs.  */
+    RELOC_EMACS_COPY_FROM_DUMP,
+    /* Set a piece of memory in Emacs to a value we store directly in
+       this relocation.  The length field contains the number of bytes
+       we actually copy into Emacs.  */
+    RELOC_EMACS_IMMEDIATE,
+    /* Set an aligned pointer-sized object in Emacs to a dump offset.  */
+    RELOC_EMACS_DUMP_PTR_RAW,
+    /* Set an aligned pointer-sized object in Emacs to point to
+       something also in Emacs.  */
+    RELOC_EMACS_EMACS_PTR_RAW,
+    /* Set an aligned Lisp_Object in Emacs to point to a value in the
+       dump.  Must be last.  */
+    RELOC_EMACS_DUMP_LV,
+  };
+
+#define EMACS_RELOC_TYPE_BITS 3
+#define EMACS_RELOC_LENGTH_BITS                         \
+  (sizeof (dump_off_t) * 8 - EMACS_RELOC_TYPE_BITS)
+
+struct emacs_reloc
+{
+  ENUM_BF (emacs_reloc_type) type : EMACS_RELOC_TYPE_BITS;
+  dump_off_t length : EMACS_RELOC_LENGTH_BITS;
+  dump_off_t emacs_offset;
+  union
+  {
+    dump_off_t dump_offset;
+    dump_off_t emacs_offset2;
+    intmax_t immediate;
+    int8_t immediate_i8;
+    int16_t immediate_i16;
+    int32_t immediate_i32;
+  } u;
+};
+
+struct dump_table_locator
+{
+  dump_off_t offset;
+  dump_off_t nr_entries;
+};
+
+struct dump_reloc
+{
+  // XXX: We have a ton of these.  Combine type and offset into one
+  // 32-bit word.  Force alignment.
+  enum dump_reloc_type type;
+  dump_off_t offset;
+};
+
+/* Format of an Emacs portable dump file.  All offsets are relative to
+   the beginning of the file.  An Emacs portable dump file is coupled
+   to exactly the Emacs binary that produced it, so details of
+   alignment and endianness are unimportant.
+
+   An Emacs dump file contains the contents of the Lisp heap.
+   On startup, Emacs can start faster by mapping a dump file into
+   memory and using the objects contained inside it instead of
+   performing initialization from scratch.
+
+   The dump file can be loaded at arbitrary locations in memory, so it
+   includes a table of relocations that let Emacs adjust the pointers
+   embedded in the dump file to account for the location where it was
+   actually loaded.
+
+   Dump files can contain pointers to other objects in the dump file
+   or to parts of the Emacs binary.  */
+struct dump_header
+{
+  /* File type magic.  */
+  char magic[sizeof (dump_magic)];
+
+  /* Associated Emacs binary.  */
+  uint8_t fingerprint[32];
+
+  /* Relocation table for the dump file; each entry is a
+     struct dump_reloc.  */
+  struct dump_table_locator dump_relocs;
+
+  /* "Relocation" table we abuse to hold information about the
+     location and type of each lisp object in the dump.  We need for
+     pdumper_object_type and ultimately for conservative GC.  */
+  struct dump_table_locator object_starts;
+
+  /* Relocation table for Emacs; each entry is a struct
+     emacs_reloc.  */
+  struct dump_table_locator emacs_relocs;
+
+  /* Start of sub-region of hot region that we can discard after load
+     completes.  The discardable region ends at hot_end.  */
+  dump_off_t hot_discardable_start;
+
+  /* End of the region that we expect to have many relocations.  */
+  dump_off_t hot_end;
+
+};
+
+struct dump_tailq
+{
+  Lisp_Object head;
+  Lisp_Object tail;
+};
+
+enum cold_op
+  {
+    COLD_OP_OBJECT,
+    COLD_OP_STRING,
+    COLD_OP_CHARSET,
+    COLD_OP_BUFFER,
+  };
+
+/* Information we use while we dump.  Note that we're not the garbage
+   collector and can operate under looser constraints: specifically,
+   we allocate memory during the dumping process.  */
+struct dump_context
+{
+  /* Header we'll write to the dump file when done.  */
+  struct dump_header header;
+
+  Lisp_Object old_purify_flag;
+  Lisp_Object old_post_gc_hook;
+
+  /* File descriptor for dumpfile; < 0 if closed.  */
+  int fd;
+  /* Name of dump file --- used for error reporting.  */
+  Lisp_Object dump_filename;
+  /* Current offset in dump file.  */
+  ptrdiff_t offset;
+
+  /* Starting offset of current object.  */
+  ptrdiff_t obj_offset;
+  /* Flags for writing the current object.  */
+  int flags;
+  /* Depth of cons-chain dumping.  */
+  ptrdiff_t cons_chain_depth;
+
+  ptrdiff_t end_heap;
+
+  /* Hash mapping objects we've already dumped to their offsets.  */
+  Lisp_Object objects_dumped;
+
+  /* Hash mapping objects to where we got them.  Used for debugging.  */
+  Lisp_Object referrers;
+  Lisp_Object current_referrer;
+
+#ifdef ENABLE_CHECKING
+  bool have_current_referrer;
+#endif
+
+  /* Queue of objects to dump.  */
+  struct dump_tailq dump_queue;
+  /* Fixups in the dump file.  */
+  Lisp_Object fixups;
+  /* Queue of copied objects for special treatment.  */
+  Lisp_Object copied_queue;
+  /* Queue of cold objects to dump.  */
+  Lisp_Object cold_queue;
+
+  /* Relocations in the dump.  */
+  Lisp_Object dump_relocs;
+  /* Object starts.  */
+  Lisp_Object object_starts;
+  /* Relocations in Emacs.  */
+  Lisp_Object emacs_relocs;
+};
+
+\f
+
+#define DUMP_OBJECT_INTERN (1<<0)
+#define DUMP_OBJECT_RECORD_START (1<<1)
+#define DUMP_OBJECT_DRY_RUN (1<<2)
+#define DUMP_OBJECT_FORCE_WORD_ALIGNMENT (1<<3)
+#define DUMP_OBJECT_PROHIBIT_ENQUEUE (1<<4)
+
+static ptrdiff_t dump_object (struct dump_context *ctx, Lisp_Object object);
+static ptrdiff_t dump_object_1 (struct dump_context *ctx,
+                                Lisp_Object object,
+                                int flags);
+
+static void
+dump_push (Lisp_Object *where, Lisp_Object newelt)
+{
+  *where = Fcons (newelt, *where);
+}
+
+static Lisp_Object
+dump_pop (Lisp_Object *where)
+{
+  Lisp_Object ret = XCAR (*where);
+  *where = XCDR (*where);
+  return ret;
+}
+
+static bool
+dump_tracking_referrers_p (struct dump_context *ctx)
+{
+  return !NILP (ctx->referrers);
+}
+
+static void
+dump_set_have_current_referrer (struct dump_context *ctx, bool have)
+{
+#ifdef ENABLE_CHECKING
+  ctx->have_current_referrer = have;
+#endif
+}
+
+/* Define as a macro so we can avoid evaluating OBJECT
+   if we dont want referrer tracking.  */
+#define DUMP_SET_REFERRER(ctx, object)                   \
+  do {                                                   \
+    struct dump_context *_ctx = (ctx);                   \
+    eassert (!_ctx->have_current_referrer);              \
+    dump_set_have_current_referrer (_ctx, true);         \
+    if (dump_tracking_referrers_p (_ctx))                \
+      ctx->current_referrer = (object);                  \
+  } while (0);
+
+static void
+dump_clear_referrer (struct dump_context *ctx)
+{
+  eassert (ctx->have_current_referrer);
+  dump_set_have_current_referrer (ctx, false);
+  if (dump_tracking_referrers_p (ctx))
+    ctx->current_referrer = Qnil;
+}
+
+static Lisp_Object
+dump_ptr_referrer (const char *label, void *address)
+{
+  char buf[128];
+  buf[0] = '\0';
+  sprintf (buf, "%s @ %p", label, address);
+  return build_string (buf);
+}
+
+static void
+print_paths_to_root (struct dump_context *ctx, Lisp_Object object);
+
+static void dump_remember_cold_op (struct dump_context *ctx,
+                                   enum cold_op op,
+                                   Lisp_Object arg);
+
+_Noreturn
+static void
+error_unsupported_dump_object (struct dump_context *ctx,
+                               Lisp_Object object,
+                               const char* msg)
+{
+  if (dump_tracking_referrers_p (ctx))
+    print_paths_to_root (ctx, object);
+  error ("unsupported object type in dump: %s", msg);
+}
+
+static ptrdiff_t
+emacs_basis (void)
+{
+  return (ptrdiff_t) &Vpurify_flag;
+}
+
+static ptrdiff_t
+emacs_offset (const void *emacs_ptr)
+{
+  /* TODO: assert that emacs_ptr is actually in emacs */
+  eassert (emacs_ptr != NULL);
+  ptrdiff_t emacs_ptr_value = (ptrdiff_t) emacs_ptr;
+  ptrdiff_t emacs_ptr_relative = emacs_ptr_value - emacs_basis ();
+  return emacs_ptr_relative;
+}
+
+/* Return whether OBJECT is a symbol the storage of which is built
+   into Emacs (and so is invariant across ASLR).  */
+static bool
+dump_builtin_symbol_p (Lisp_Object object)
+{
+  if (!SYMBOLP (object))
+    return false;
+  char* bp = (char*) lispsym;
+  struct Lisp_Symbol *s = XSYMBOL (object);
+  char* sp = (char*) s;
+  return bp <= sp && sp < bp + sizeof (lispsym);
+}
+
+/* Return whether OBJECT has the same bit pattern in all Emacs
+   invocations --- i.e., is invariant across a dump.  */
+static bool
+dump_object_self_representing_p (Lisp_Object object)
+{
+  return INTEGERP (object) || dump_builtin_symbol_p (object);
+}
+
+#define DEFINE_FROMLISP_FUNC(fn, type)          \
+  static type                                   \
+  fn (Lisp_Object value)                        \
+  {                                             \
+    type result;                                \
+    CONS_TO_INTEGER (value, type, result);      \
+    return result;                              \
+  }
+
+DEFINE_FROMLISP_FUNC (intmax_t_from_lisp, intmax_t);
+DEFINE_FROMLISP_FUNC (ptrdiff_t_from_lisp, ptrdiff_t);
+DEFINE_FROMLISP_FUNC (dump_off_from_lisp, dump_off_t);
+
+static void
+dump_tailq_init (struct dump_tailq *tailq)
+{
+  tailq->head = tailq->tail = Qnil;
+}
+
+static void
+dump_tailq_append (struct dump_tailq *tailq, Lisp_Object value)
+{
+  Lisp_Object link = Fcons (value, Qnil);
+  if (NILP (tailq->head))
+    {
+      eassert (NILP (tailq->tail));
+      tailq->head = tailq->tail = link;
+    }
+  else
+    {
+      eassert (!NILP (tailq->tail));
+      XSETCDR (tailq->tail, link);
+      tailq->tail = link;
+    }
+}
+
+static bool
+dump_tailq_empty_p (struct dump_tailq *tailq)
+{
+  return NILP (tailq->head);
+}
+
+static Lisp_Object
+dump_tailq_pop (struct dump_tailq *tailq)
+{
+  eassert (!dump_tailq_empty_p (tailq));
+  Lisp_Object value = XCAR (tailq->head);
+  tailq->head = XCDR (tailq->head);
+  if (NILP (tailq->head))
+    tailq->tail = Qnil;
+  return value;
+}
+
+static void
+dump_write (struct dump_context *ctx, const void *buf, ptrdiff_t nbyte)
+{
+  eassert (nbyte == 0 || buf != NULL);
+  eassert (ctx->obj_offset == 0);
+  eassert ((ctx->flags & DUMP_OBJECT_DRY_RUN) == 0);
+  if (emacs_write (ctx->fd, buf, nbyte) < nbyte)
+    report_file_error ("Could not write to dump file", ctx->dump_filename);
+  ctx->offset += nbyte;
+}
+
+static void
+dump_seek (struct dump_context *ctx, ptrdiff_t offset)
+{
+  eassert (ctx->obj_offset == 0);
+  if (lseek (ctx->fd, offset, SEEK_SET) < 0)
+    report_file_error ("Setting file position",
+                       ctx->dump_filename);
+  ctx->offset = offset;
+}
+
+static void
+dump_write_zero (struct dump_context *ctx, ptrdiff_t nbytes)
+{
+  while (nbytes > 0)
+    {
+      ptrdiff_t zero = 0;
+      ptrdiff_t to_write = sizeof (zero);
+      if (to_write > nbytes)
+        to_write = nbytes;
+      dump_write (ctx, &zero, to_write);
+      nbytes -= to_write;
+    }
+}
+
+static void
+dump_align_output (struct dump_context *ctx, ptrdiff_t alignment)
+{
+  if (ctx->offset % alignment != 0)
+    dump_write_zero (ctx, alignment - (ctx->offset % alignment));
+}
+
+static ptrdiff_t
+dump_object_start (struct dump_context *ctx,
+                   int alignment,
+                   void *out,
+                   ptrdiff_t outsz)
+{
+  eassert (ctx->obj_offset == 0);
+  if ((ctx->flags & DUMP_OBJECT_FORCE_WORD_ALIGNMENT) &&
+      alignment > sizeof (void*))
+    alignment = sizeof (void*);
+  if ((ctx->flags & DUMP_OBJECT_DRY_RUN) == 0)
+    dump_align_output (ctx, alignment);
+  ctx->obj_offset = ctx->offset;
+  memset (out, 0, outsz);
+  return ctx->offset;
+}
+
+static ptrdiff_t
+dump_object_finish (struct dump_context *ctx,
+                    const void *out,
+                    ptrdiff_t sz)
+{
+  ptrdiff_t offset = ctx->obj_offset;
+  eassert (offset > 0);
+  eassert (offset == ctx->offset); /* No intervening writes.  */
+  ctx->obj_offset = 0;
+  if ((ctx->flags & DUMP_OBJECT_DRY_RUN) == 0)
+    dump_write (ctx, out, sz);
+  return offset;
+}
+
+/* Return offset at which OBJECT has been dumped, or 0 if OBJECT has
+   not been dumped.  */
+static ptrdiff_t
+dump_recall_object (struct dump_context *ctx, Lisp_Object object)
+{
+  Lisp_Object dumped = ctx->objects_dumped;
+  return ptrdiff_t_from_lisp (Fgethash (object, dumped, make_number (0)));
+}
+
+static void
+dump_remember_object (struct dump_context *ctx,
+                      Lisp_Object object,
+                      ptrdiff_t offset)
+{
+  Fputhash (object, INTEGER_TO_CONS (offset), ctx->objects_dumped);
+}
+
+static void
+dump_note_reachable (struct dump_context *ctx, Lisp_Object object)
+{
+  eassert (ctx->have_current_referrer);
+  if (!dump_tracking_referrers_p (ctx))
+    return;
+  Lisp_Object referrer = ctx->current_referrer;
+  Lisp_Object obj_referrers = Fgethash (object, ctx->referrers, Qnil);
+  if (NILP (Fmemq (referrer, obj_referrers)))
+    Fputhash (object, Fcons (referrer, obj_referrers), ctx->referrers);
+}
+
+/* If this object lives in the Emacs image and not on the heap, return
+   a pointer to the object data.  Otherwise, return NULL.  */
+static void*
+dump_object_emacs_ptr (Lisp_Object lv)
+{
+  if (SUBRP (lv))
+    return XSUBR (lv);
+  if (dump_builtin_symbol_p (lv))
+    return XSYMBOL (lv);
+  return NULL;
+}
+
+static void
+dump_enqueue_object (struct dump_context *ctx, Lisp_Object object)
+{
+  if ((!dump_object_self_representing_p (object) ||
+       dump_object_emacs_ptr (object)) &&
+      dump_recall_object (ctx, object) == 0)
+    {
+      eassert ((ctx->flags & DUMP_OBJECT_PROHIBIT_ENQUEUE) == 0);
+
+      dump_remember_object (ctx, object, -1);
+      if (BOOL_VECTOR_P (object) || FLOATP (object))
+        dump_remember_cold_op (ctx, COLD_OP_OBJECT, object);
+      else
+        dump_tailq_append (&ctx->dump_queue, object);
+    }
+  dump_note_reachable (ctx, object);
+}
+
+static void
+print_paths_to_root_1 (struct dump_context *ctx,
+                       Lisp_Object object,
+                       int level)
+{
+  Lisp_Object referrers = Fgethash (object, ctx->referrers, Qnil);
+  while (!NILP (referrers))
+    {
+      Lisp_Object referrer = XCAR (referrers);
+      referrers = XCDR (referrers);
+      Lisp_Object repr = Fprin1_to_string (referrer, Qnil);
+      for (int i = 0; i < level; ++i)
+        fputc (' ', stderr);
+      fprintf (stderr, "%s\n", SDATA (repr));
+      print_paths_to_root_1 (ctx, referrer, level + 1);
+    }
+}
+
+static void
+print_paths_to_root (struct dump_context *ctx, Lisp_Object object)
+{
+  print_paths_to_root_1 (ctx, object, 0);
+}
+
+static void
+dump_remember_cold_op (struct dump_context *ctx,
+                       enum cold_op op,
+                       Lisp_Object arg)
+{
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+  dump_push (&ctx->cold_queue, Fcons (make_number (op), arg));
+}
+
+/* Add a dump relocation that points into Emacs.
+
+   Add a relocation that updates the pointer stored at DUMP_OFFSET to
+   point into the Emacs binary upon dump load.  The pointer-sized
+   value at DUMP_OFFSET in the dump file should contain a number
+   relative to emacs_basis().  */
+static void
+dump_reloc_dump_to_emacs_raw_ptr (struct dump_context *ctx,
+                                  ptrdiff_t dump_offset)
+{
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+  dump_push (&ctx->dump_relocs,
+             list2 (INTEGER_TO_CONS (RELOC_DUMP_TO_EMACS_RAW_PTR),
+                    INTEGER_TO_CONS (dump_offset)));
+}
+
+/* Add a dump relocation that points a Lisp_Object back at the dump.
+
+   Add a relocation that updates the Lisp_Object at DUMP_OFFSET in the
+   dump to point to another object in the dump.  The Lisp_Object-sized
+   value at DUMP_OFFSET in the dump file should contain the offset of
+   the target object relative to the start of the dump.  */
+static void
+dump_reloc_dump_to_dump_lv (struct dump_context *ctx,
+                            ptrdiff_t dump_offset,
+                            enum Lisp_Type type)
+{
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+
+  ptrdiff_t reloc_type;
+  switch (type)
+    {
+    case Lisp_Symbol:
+    case Lisp_Misc:
+    case Lisp_String:
+    case Lisp_Vectorlike:
+    case Lisp_Cons:
+    case Lisp_Float:
+      reloc_type = RELOC_DUMP_TO_DUMP_LV + type;
+      break;
+    default:
+      emacs_abort ();
+    }
+
+  dump_push (&ctx->dump_relocs,
+             list2 (INTEGER_TO_CONS (reloc_type),
+                    INTEGER_TO_CONS (dump_offset)));
+}
+
+/* Add a dump relocation that points a raw pointer back at the dump.
+
+   Add a relocation that updates the raw pointer at DUMP_OFFSET in the
+   dump to point to another object in the dump.  The pointer-sized
+   value at DUMP_OFFSET in the dump file should contain the offset of
+   the target object relative to the start of the dump.  */
+static void
+dump_reloc_dump_to_dump_raw_ptr (struct dump_context *ctx,
+                                 ptrdiff_t dump_offset)
+{
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+
+  dump_push (&ctx->dump_relocs,
+             list2 (INTEGER_TO_CONS (RELOC_DUMP_TO_DUMP_RAW_PTR),
+                    INTEGER_TO_CONS (dump_offset)));
+}
+
+/* Add a dump relocation that points to a Lisp object in Emacs.
+
+   Add a relocation that updates the Lisp_Object at DUMP_OFFSET in the
+   dump to point to a lisp object in Emacs.  The Lisp_Object-sized
+   value at DUMP_OFFSET in the dump file should contain the offset of
+   the target object relative to emacs_basis().  TYPE is the type of
+   Lisp value.  */
+static void
+dump_reloc_dump_to_emacs_lv (struct dump_context *ctx,
+                             ptrdiff_t dump_offset,
+                             enum Lisp_Type type)
+{
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+
+  ptrdiff_t reloc_type;
+  switch (type)
+    {
+    case Lisp_Misc:
+    case Lisp_String:
+    case Lisp_Vectorlike:
+    case Lisp_Cons:
+    case Lisp_Float:
+      reloc_type = RELOC_DUMP_TO_EMACS_LV + type;
+      break;
+    default:
+      emacs_abort ();
+    }
+
+  dump_push (&ctx->dump_relocs,
+             list2 (INTEGER_TO_CONS (reloc_type),
+                    INTEGER_TO_CONS (dump_offset)));
+}
+
+/* Add an Emacs relocation that copies arbitrary bytes from the dump.
+
+   When the dump is loaded, Emacs copies SIZE bytes from OFFSET in
+   dump to LOCATION in the Emacs data section.  This copying happens
+   after other relocations, so it's all right to, say, copy a
+   Lisp_Value (since by the time we copy the Lisp_Value, it'll have
+   been adjusted to account for the location of the running Emacs and
+   dump file).  */
+static void
+dump_emacs_reloc_copy_from_dump (struct dump_context *ctx,
+                                 ptrdiff_t dump_offset,
+                                 void* emacs_ptr,
+                                 ptrdiff_t size)
+{
+  eassert (size >= 0);
+  eassert (size < (1 << EMACS_RELOC_LENGTH_BITS));
+
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+
+  if (size == 0)
+    return;
+
+  dump_push (&ctx->emacs_relocs,
+             list4 (make_number (RELOC_EMACS_COPY_FROM_DUMP),
+                    INTEGER_TO_CONS (emacs_offset (emacs_ptr)),
+                    INTEGER_TO_CONS (dump_offset),
+                    INTEGER_TO_CONS (size)));
+}
+
+/* Add an Emacs relocation that sets values to arbitrary bytes.
+
+   When the dump is loaded, Emacs copies SIZE bytes from the
+   relocation itself to the adjusted location inside Emacs EMACS_PTR.
+   SIZE is the number of bytes to copy.  See struct emacs_reloc for
+   the maximum size that this mechanism can support.  The value comes
+   from VALUE_PTR.
+ */
+static void
+dump_emacs_reloc_immediate (struct dump_context *ctx,
+                            const void *emacs_ptr,
+                            const void *value_ptr,
+                            ptrdiff_t size)
+{
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+
+  intmax_t value = 0;
+  eassert (size <= sizeof (value));
+  memcpy (&value, value_ptr, size);
+  dump_push (&ctx->emacs_relocs,
+             list4 (make_number (RELOC_EMACS_IMMEDIATE),
+                    INTEGER_TO_CONS (emacs_offset (emacs_ptr)),
+                    INTEGER_TO_CONS (value),
+                    INTEGER_TO_CONS (size)));
+}
+
+#define DEFINE_EMACS_IMMEDIATE_FN(fnname, type)                         \
+  static void                                                           \
+  fnname (struct dump_context *ctx,                                     \
+          const type *emacs_ptr,                                        \
+          type value)                                                   \
+  {                                                                     \
+    dump_emacs_reloc_immediate (                                        \
+      ctx, emacs_ptr, &value, sizeof (value));                          \
+  }
+
+DEFINE_EMACS_IMMEDIATE_FN (dump_emacs_reloc_immediate_lv, Lisp_Object);
+DEFINE_EMACS_IMMEDIATE_FN (dump_emacs_reloc_immediate_ptrdiff_t, ptrdiff_t);
+DEFINE_EMACS_IMMEDIATE_FN (dump_emacs_reloc_immediate_emacs_int, EMACS_INT);
+DEFINE_EMACS_IMMEDIATE_FN (dump_emacs_reloc_immediate_int, int);
+DEFINE_EMACS_IMMEDIATE_FN (dump_emacs_reloc_immediate_bool, bool);
+
+/* Add an emacs relocation that makes a raw pointer in Emacs point
+   into the dump.  */
+static void
+dump_emacs_reloc_to_dump_ptr_raw (struct dump_context *ctx,
+                                  const void* emacs_ptr,
+                                  ptrdiff_t dump_offset)
+{
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+
+  dump_push (&ctx->emacs_relocs,
+             list3 (make_number (RELOC_EMACS_DUMP_PTR_RAW),
+                    INTEGER_TO_CONS (emacs_offset (emacs_ptr)),
+                    INTEGER_TO_CONS (dump_offset)));
+}
+
+/* Add an emacs relocation that points into the dump.
+
+   When the dump is loaded, the Lisp_Object at EMACS_ROOT in Emacs to
+   point to VALUE.  VALUE can be any Lisp value; this function
+   automatically queues the value for dumping if necessary.  */
+static void
+dump_emacs_reloc_to_dump_lv (struct dump_context *ctx,
+                             Lisp_Object *emacs_ptr,
+                             Lisp_Object value)
+{
+  if (dump_object_self_representing_p (value))
+    dump_emacs_reloc_immediate_lv (ctx, emacs_ptr, value);
+  else
+    {
+      if ((ctx->flags & DUMP_OBJECT_DRY_RUN) == 0)
+        dump_push (
+          &ctx->emacs_relocs,
+          list3 (INTEGER_TO_CONS (RELOC_EMACS_DUMP_LV + XTYPE (value)),
+                 INTEGER_TO_CONS (emacs_offset (emacs_ptr)),
+                 value));
+      dump_enqueue_object (ctx, value);
+    }
+}
+
+/* Add an emacs relocation that makes a raw pointer in Emacs point
+   back into the Emacs image.  */
+static void
+dump_emacs_reloc_to_emacs_ptr_raw (struct dump_context *ctx,
+                                   void* emacs_ptr,
+                                   void *target_emacs_ptr)
+{
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+
+  dump_push (&ctx->emacs_relocs,
+             list3 (make_number (RELOC_EMACS_EMACS_PTR_RAW),
+                    INTEGER_TO_CONS (emacs_offset (emacs_ptr)),
+                    INTEGER_TO_CONS (emacs_offset (target_emacs_ptr))));
+}
+
+/* Add an Emacs relocation that makes a raw pointer in Emacs point to
+   a different part of Emacs.  */
+
+enum dump_fixup_type
+  {
+    DUMP_FIXUP_LISP_OBJECT,
+    DUMP_FIXUP_LISP_OBJECT_RAW,
+    DUMP_FIXUP_PTR_DUMP_RAW,
+  };
+
+enum dump_lv_fixup_type
+  {
+    LV_FIXUP_LISP_OBJECT,
+    LV_FIXUP_RAW_POINTER,
+  };
+
+/* Make something in the dump point to a lisp object.
+
+   CTX is a dump context.  DUMP_OFFSET is the location in the dump to
+   fix.  VALUE is the object to which the location in the dump
+   should point.
+
+   If FIXUP_SUBTYPE is LV_FIXUP_LISP_OBJECT, we expect a Lisp_Object
+   at DUMP_OFFSET.  If it's LV_FIXUP_RAW_POINTER, we expect a pointer.
+ */
+static void
+dump_remember_fixup_lv (struct dump_context *ctx,
+                        ptrdiff_t dump_offset,
+                        Lisp_Object value,
+                        enum dump_lv_fixup_type fixup_subtype)
+{
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+
+  dump_push (&ctx->fixups,
+             list3 (
+               make_number (fixup_subtype == LV_FIXUP_LISP_OBJECT
+                            ? DUMP_FIXUP_LISP_OBJECT
+                            : DUMP_FIXUP_LISP_OBJECT_RAW),
+               INTEGER_TO_CONS (dump_offset),
+               value));
+}
+
+/* Remember to fix up the dump file such that the pointer-sized value
+   at DUMP_OFFSET points to NEW_DUMP_OFFSET in the dump file and to
+   its absolute address at runtime.  */
+static void
+dump_remember_fixup_ptr_raw (struct dump_context *ctx,
+                             ptrdiff_t dump_offset,
+                             ptrdiff_t new_dump_offset)
+{
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+
+  dump_push (&ctx->fixups,
+             list3 (
+               make_number (DUMP_FIXUP_PTR_DUMP_RAW),
+               INTEGER_TO_CONS (dump_offset),
+               INTEGER_TO_CONS (new_dump_offset)));
+}
+
+static void
+dump_root_visitor (Lisp_Object *root_ptr, enum gc_root_type type, void *data)
+{
+  struct dump_context *ctx = data;
+  Lisp_Object value = *root_ptr;
+  if (type == GC_ROOT_C_SYMBOL)
+    {
+      eassert (dump_builtin_symbol_p (value));
+      /* Remember to dump the object itself later along with all the
+         rest of the copied-to-Emacs objects.  */
+      DUMP_SET_REFERRER (ctx, build_string ("built-in symbol list"));
+      dump_enqueue_object (ctx, value);
+      dump_clear_referrer (ctx);
+    }
+  else
+    {
+      DUMP_SET_REFERRER (ctx, dump_ptr_referrer ("emacs root", root_ptr));
+      dump_emacs_reloc_to_dump_lv (ctx, root_ptr, *root_ptr);
+      dump_clear_referrer (ctx);
+    }
+}
+
+/* Kick off the dump process by queuing up the static GC roots.  */
+static void
+dump_roots (struct dump_context *ctx)
+{
+  struct gc_root_visitor visitor;
+  memset (&visitor, 0, sizeof (visitor));
+  visitor.visit = dump_root_visitor;
+  visitor.data = ctx;
+  visit_static_gc_roots (visitor);
+}
+
+static ptrdiff_t
+field_relpos (const void *in_start, const void *in_field)
+{
+  ptrdiff_t in_start_val = (ptrdiff_t) in_start;
+  ptrdiff_t in_field_val = (ptrdiff_t) in_field;
+  eassert (in_start_val <= in_field_val);
+  ptrdiff_t relpos = in_field_val - in_start_val;
+  eassert (relpos < 1024); /* Sanity check.  */
+  return relpos;
+}
+
+static void
+cpyptr (void *out, const void *in)
+{
+  memcpy (out, in, sizeof (void *));
+}
+
+/* Convenience macro for regular assignment.  */
+#define DUMP_FIELD_COPY(out, in, name) \
+  do {                                 \
+    (out)->name = (in)->name;          \
+  } while (0)
+
+static void
+dump_field_lv_or_rawptr (struct dump_context *ctx,
+                         void *out,
+                         const void *in_start,
+                         const void *in_field,
+                         /* opt */ const enum Lisp_Type *raw_ptr_type)
+{
+  eassert (ctx->obj_offset > 0);
+
+  Lisp_Object value;
+  ptrdiff_t relpos = field_relpos (in_start, in_field);
+  void *out_field = (char *) out + relpos;
+  if (raw_ptr_type == NULL)
+    {
+      memcpy (&value, in_field, sizeof (value));
+      if (dump_object_self_representing_p (value))
+        {
+          memcpy (out_field, &value, sizeof (value));
+          return;
+        }
+    }
+  else
+    {
+      void *ptrval;
+      cpyptr (&ptrval, in_field);
+      if (ptrval == NULL)
+        return; /* Nothing to do.  */
+      switch (*raw_ptr_type)
+        {
+        case Lisp_Symbol:
+          value = make_lisp_symbol (ptrval);
+          break;
+        case Lisp_Misc:
+        case Lisp_String:
+        case Lisp_Vectorlike:
+        case Lisp_Cons:
+        case Lisp_Float:
+          value = make_lisp_ptr (ptrval, *raw_ptr_type);
+          break;
+        default:
+          emacs_abort ();
+        }
+    }
+
+  /* Now value is the Lisp_Object to which we want to point whether or
+     not the field is a raw pointer (in which case we just synthesized
+     the Lisp_Object outselves) or a Lisp_Object (in which case we
+     just copied the thing).  Add a fixup or relocation.  */
+
+  ptrdiff_t out_value;
+  ptrdiff_t out_field_offset = ctx->obj_offset + relpos;
+  ptrdiff_t target_offset = dump_recall_object (ctx, value);
+  if (target_offset > 0)
+    {
+      /* We've already dumped the referenced object, so we can emit
+         the value and a relocation directly instead of indirecting
+         through a fixup.  */
+      out_value = target_offset;
+      if (raw_ptr_type)
+        dump_reloc_dump_to_dump_raw_ptr (ctx, out_field_offset);
+      else
+        dump_reloc_dump_to_dump_lv (ctx, out_field_offset, XTYPE (value));
+    }
+  else
+    {
+      /* We don't know about the target object yet, so add a fixup.
+         When we process the fixup, we'll have dumped the target
+         object.  */
+      out_value = (ptrdiff_t) 0xDEADF00D;
+      dump_remember_fixup_lv (ctx,
+                              out_field_offset,
+                              value,
+                              ( raw_ptr_type
+                                ? LV_FIXUP_RAW_POINTER
+                                : LV_FIXUP_LISP_OBJECT ));
+      if (target_offset == 0)
+        dump_enqueue_object (ctx, value);
+    }
+
+  memcpy (out_field, &out_value, sizeof (out_value));
+}
+
+/* Set a pointer field on an output object during dump.
+
+   CTX is the dump context.  OFFSET is the offset at which the current
+   object starts.  OUT is a pointer to the dump output object.
+   IN_START is the start of the current Emacs object.  IN_FIELD is a
+   pointer to the field in that object.  TYPE is the type of pointer
+   to which IN_FIELD points.
+ */
+static void
+dump_field_lv_rawptr (struct dump_context *ctx,
+                      void *out,
+                      const void *in_start,
+                      const void *in_field,
+                      enum Lisp_Type type)
+{
+  dump_field_lv_or_rawptr (ctx, out, in_start, in_field, &type);
+}
+
+/* Set a Lisp_Object field on an output object during dump.
+
+   CTX is a dump context.  OFFSET is the offset at which the current
+   object starts.  OUT is a pointer to the dump output object.
+   IN_START is the start of the current Emacs object.  IN_FIELD is a
+   pointer to a Lisp_Object field in that object.
+
+   Arrange for the dump to contain fixups and relocations such that,
+   at load time, the given field of the output object contains a valid
+   Lisp_Object pointing to the same notional object that *IN_FIELD
+   contains now.
+
+   See idomatic usage below.  */
+static void
+dump_field_lv (struct dump_context *ctx,
+               void *out,
+               const void *in_start,
+               const Lisp_Object *in_field)
+{
+  dump_field_lv_or_rawptr (ctx, out, in_start, in_field, NULL);
+}
+
+/* Note that we're going to add a manual fixup for the given field
+   later.  */
+static void
+dump_field_fixup_later (struct dump_context *ctx,
+                        void *out,
+                        const void *in_start,
+                        const void *in_field)
+{
+  // TODO: more error checking
+  (void) field_relpos (in_start, in_field);
+}
+
+/* Mark an output object field, which is as wide as a poiner, as being
+   fixed up to point to a specific offset in the dump.  */
+static void
+dump_field_ptr_to_dump_offset (struct dump_context *ctx,
+                               void *out,
+                               const void *in_start,
+                               const void *in_field,
+                               ptrdiff_t target_dump_offset)
+{
+  eassert (ctx->obj_offset > 0);
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+
+  ptrdiff_t relpos = field_relpos (in_start, in_field);
+  dump_reloc_dump_to_dump_raw_ptr (ctx, ctx->obj_offset + relpos);
+  ptrdiff_t outval = target_dump_offset;
+  memcpy ((char*) out + relpos, &outval, sizeof (outval));
+}
+
+/* Mark a field as pointing to a place inside Emacs.
+
+   CTX is the dump context.  OUT points to the out-object for the
+   current dump function.  IN_START points to the start of the object
+   being dumped.  IN_FIELD points to the field inside the object being
+   dumped that we're dumping.  The contents of this field (which
+   should be as wide as a pointer) are the Emacs pointer to dump.
+
+ */
+static void
+dump_field_emacs_ptr (struct dump_context *ctx,
+                      void *out,
+                      const void *in_start,
+                      const void *in_field)
+{
+  eassert (ctx->obj_offset > 0);
+  if (ctx->flags & DUMP_OBJECT_DRY_RUN)
+    return;
+
+  ptrdiff_t abs_emacs_ptr;
+  cpyptr (&abs_emacs_ptr, in_field);
+  ptrdiff_t rel_emacs_ptr = abs_emacs_ptr - emacs_basis ();
+  ptrdiff_t relpos = field_relpos (in_start, in_field);
+  cpyptr ((char*) out + relpos, &rel_emacs_ptr);
+  dump_reloc_dump_to_emacs_raw_ptr (ctx, ctx->obj_offset + relpos);
+}
+
+static ptrdiff_t
+dump_cons (struct dump_context *ctx, const struct Lisp_Cons *cons)
+{
+  struct Lisp_Cons out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  dump_field_lv (ctx, &out, cons, &cons->car);
+  dump_field_lv (ctx, &out, cons, &cons->u.cdr);
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_interval_tree (struct dump_context *ctx,
+                    INTERVAL tree,
+                    ptrdiff_t parent_offset)
+{
+  // TODO: output tree breadth-first?
+  struct interval out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, tree, total_length);
+  DUMP_FIELD_COPY (&out, tree, position);
+  if (tree->left)
+    dump_field_fixup_later (ctx, &out, tree, &tree->left);
+  if (tree->right)
+    dump_field_fixup_later (ctx, &out, tree, &tree->right);
+  if (!tree->up_obj)
+    {
+      eassert (parent_offset != 0);
+      dump_field_ptr_to_dump_offset (
+        ctx, &out,
+        tree, &tree->up.interval,
+        parent_offset);
+    }
+  else
+    dump_field_lv (ctx, &out, tree, &tree->up.obj);
+  DUMP_FIELD_COPY (&out, tree, up_obj);
+  eassert (tree->gcmarkbit == 0);
+  DUMP_FIELD_COPY (&out, tree, write_protect);
+  DUMP_FIELD_COPY (&out, tree, visible);
+  DUMP_FIELD_COPY (&out, tree, front_sticky);
+  DUMP_FIELD_COPY (&out, tree, rear_sticky);
+  dump_field_lv (ctx, &out, tree, &tree->plist);
+  ptrdiff_t offset = dump_object_finish (ctx, &out, sizeof (out));
+  if (tree->left)
+      dump_remember_fixup_ptr_raw (
+        ctx,
+        offset + offsetof (struct interval, left),
+        dump_interval_tree (ctx, tree->left, offset));
+  if (tree->right)
+      dump_remember_fixup_ptr_raw (
+        ctx,
+        offset + offsetof (struct interval, right),
+        dump_interval_tree (ctx, tree->right, offset));
+  return offset;
+}
+
+static ptrdiff_t
+dump_string (struct dump_context *ctx, const struct Lisp_String *string)
+{
+  /* If we have text properties, write them _after_ the string so that
+     at runtime, the prefetcher and cache will DTRT. (We access the
+     string before its properties.).
+
+     There's special code to dump string data contiguously later on.
+     we seldom write to string data and never relocate it, so lumping
+     it together at the end of the dump saves on COW faults.
+
+     If, however, the string's size_byte field is -1, the string data
+     is actually a pointer to Emacs data segment, so we can do even
+     better by emitting a relocation instead of bothering to copy the
+     string data.  */
+  struct Lisp_String out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, string, size);
+  DUMP_FIELD_COPY (&out, string, size_byte);
+  if (string->intervals)
+    dump_field_fixup_later (ctx, &out, string, &string->intervals);
+
+  if (string->size_byte == -2)
+    /* String literal in Emacs rodata.  */
+    dump_field_emacs_ptr (ctx, &out, string, &string->data);
+  else
+    {
+      dump_field_fixup_later (ctx, &out, string, &string->data);
+      dump_remember_cold_op (ctx,
+                             COLD_OP_STRING,
+                             make_lisp_ptr ((void*) string, Lisp_String));
+    }
+
+  ptrdiff_t offset = dump_object_finish (ctx, &out, sizeof (out));
+  if (string->intervals)
+    dump_remember_fixup_ptr_raw (
+      ctx,
+      offset + offsetof (struct Lisp_String, intervals),
+      dump_interval_tree (ctx, string->intervals, 0));
+
+  return offset;
+}
+
+static ptrdiff_t
+dump_marker (struct dump_context *ctx, const struct Lisp_Marker *marker)
+{
+  struct Lisp_Marker out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, marker, type);
+  eassert (marker->gcmarkbit == 0);
+  (void) marker->spacer; /* Do not write padding.  */
+  DUMP_FIELD_COPY (&out, marker, need_adjustment);
+  DUMP_FIELD_COPY (&out, marker, insertion_type);
+  if (marker->buffer)
+    {
+      dump_field_lv_rawptr (
+        ctx, &out,
+        marker, &marker->buffer,
+        Lisp_Vectorlike);
+      dump_field_lv_rawptr (
+        ctx, &out,
+        marker, &marker->next,
+        Lisp_Misc);
+      DUMP_FIELD_COPY (&out, marker, charpos);
+      DUMP_FIELD_COPY (&out, marker, bytepos);
+    }
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_overlay (struct dump_context *ctx, const struct Lisp_Overlay *overlay)
+{
+  struct Lisp_Overlay out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, overlay, type);
+  eassert (overlay->gcmarkbit == 0);
+  (void) overlay->spacer; /* Do not write padding.  */
+  dump_field_lv_rawptr (ctx, &out, overlay, &overlay->next, Lisp_Misc);
+  dump_field_lv (ctx, &out, overlay, &overlay->start);
+  dump_field_lv (ctx, &out, overlay, &overlay->end);
+  dump_field_lv (ctx, &out, overlay, &overlay->plist);
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_save_value (struct dump_context *ctx,
+                 const struct Lisp_Save_Value *ptr)
+{
+  struct Lisp_Save_Value out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, ptr, type);
+  eassert(ptr->gcmarkbit == 0);
+  (void) ptr->spacer; /* Do not write padding.  */
+  DUMP_FIELD_COPY (&out, ptr, save_type);
+  for (int i = 0; i < SAVE_VALUE_SLOTS; i++)
+    {
+      switch (save_type (&out, i))
+        {
+        case SAVE_UNUSED:
+          break;
+        case SAVE_INTEGER:
+          DUMP_FIELD_COPY (&out, ptr, data[i].integer);
+          break;
+        case SAVE_FUNCPOINTER:
+          dump_field_emacs_ptr (ctx, &out, ptr, &ptr->data[i].funcpointer);
+          break;
+        case SAVE_OBJECT:
+          dump_field_lv (ctx, &out, ptr, &ptr->data[i].object);
+          break;
+        case SAVE_POINTER:
+          error_unsupported_dump_object(
+            ctx, make_lisp_ptr ((void *) ptr, Lisp_Misc), "SAVE_POINTER");
+        default:
+          emacs_abort ();
+        }
+    }
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static void
+dump_field_finalizer_ref (struct dump_context *ctx,
+                          void *out,
+                          const struct Lisp_Finalizer *finalizer,
+                          struct Lisp_Finalizer *const *field)
+{
+  if (*field == &finalizers || *field == &doomed_finalizers)
+    dump_field_emacs_ptr (ctx, out, finalizer, field);
+  else
+    dump_field_lv_rawptr (ctx, out, finalizer, field, Lisp_Misc);
+}
+
+static ptrdiff_t
+dump_finalizer (struct dump_context *ctx,
+                const struct Lisp_Finalizer *finalizer)
+{
+  struct Lisp_Finalizer out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, finalizer, base.type);
+  eassert (finalizer->base.gcmarkbit == 0);
+  (void) finalizer->base.spacer; /* Do not write padding.  */
+  dump_field_finalizer_ref (ctx, &out, finalizer, &finalizer->prev);
+  dump_field_finalizer_ref (ctx, &out, finalizer, &finalizer->next);
+  dump_field_lv (ctx, &out, finalizer, &finalizer->function);
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_misc_any (struct dump_context *ctx, struct Lisp_Misc_Any *misc_any)
+{
+  ptrdiff_t result;
+
+  switch (misc_any->type)
+    {
+    case Lisp_Misc_Marker:
+      result = dump_marker (ctx, (struct Lisp_Marker *) misc_any);
+      break;
+
+    case Lisp_Misc_Overlay:
+      result = dump_overlay (ctx, (struct Lisp_Overlay *) misc_any);
+      break;
+
+    case Lisp_Misc_Save_Value:
+      result = dump_save_value (ctx, (struct Lisp_Save_Value *) misc_any);
+      break;
+
+    case Lisp_Misc_Finalizer:
+      result = dump_finalizer (ctx, (struct Lisp_Finalizer *) misc_any);
+      break;
+
+#ifdef HAVE_MODULES
+    case Lisp_Misc_User_Ptr:
+      error_unsupported_dump_object(
+        ctx,
+        make_lisp_ptr (misc_any, Lisp_Misc),
+        "module user ptr");
+      break;
+#endif
+
+    default:
+    case Lisp_Misc_Float: /* Not used */
+      emacs_abort ();
+    }
+
+  return result;
+}
+
+static ptrdiff_t
+dump_float (struct dump_context *ctx, const struct Lisp_Float *lfloat)
+{
+  struct Lisp_Float out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, lfloat, u.data);
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_fwd_int (struct dump_context *ctx, const struct Lisp_Intfwd *intfwd)
+{
+  dump_emacs_reloc_immediate_emacs_int (ctx, intfwd->intvar, *intfwd->intvar);
+  struct Lisp_Intfwd out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, intfwd, type);
+  dump_field_emacs_ptr (ctx, &out, intfwd, &intfwd->intvar);
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_fwd_bool (struct dump_context *ctx, const struct Lisp_Boolfwd *boolfwd)
+{
+  dump_emacs_reloc_immediate_bool (ctx, boolfwd->boolvar, *boolfwd->boolvar);
+  struct Lisp_Boolfwd out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, boolfwd, type);
+  dump_field_emacs_ptr (ctx, &out, boolfwd, &boolfwd->boolvar);
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_fwd_obj (struct dump_context *ctx, const struct Lisp_Objfwd *objfwd)
+{
+  dump_emacs_reloc_to_dump_lv (ctx, objfwd->objvar, *objfwd->objvar);
+  struct Lisp_Objfwd out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, objfwd, type);
+  dump_field_emacs_ptr (ctx, &out, objfwd, &objfwd->objvar);
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_fwd_buffer_obj (struct dump_context *ctx,
+                     const struct Lisp_Buffer_Objfwd *buffer_objfwd)
+{
+  struct Lisp_Buffer_Objfwd out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, buffer_objfwd, type);
+  DUMP_FIELD_COPY (&out, buffer_objfwd, offset);
+  dump_field_lv (ctx, &out, buffer_objfwd, &buffer_objfwd->predicate);
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_fwd_kboard_obj (struct dump_context *ctx,
+                     const struct Lisp_Kboard_Objfwd *kboard_objfwd)
+{
+  struct Lisp_Kboard_Objfwd out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, kboard_objfwd, type);
+  DUMP_FIELD_COPY (&out, kboard_objfwd, offset);
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_fwd (struct dump_context *ctx, union Lisp_Fwd *fwd)
+{
+  ptrdiff_t offset;
+
+  switch (XFWDTYPE (fwd))
+    {
+    case Lisp_Fwd_Int:
+      offset = dump_fwd_int (ctx, &fwd->u_intfwd);
+      break;
+    case Lisp_Fwd_Bool:
+      offset = dump_fwd_bool (ctx, &fwd->u_boolfwd);
+      break;
+    case Lisp_Fwd_Obj:
+      offset = dump_fwd_obj (ctx, &fwd->u_objfwd);
+      break;
+    case Lisp_Fwd_Buffer_Obj:
+      offset = dump_fwd_buffer_obj (ctx, &fwd->u_buffer_objfwd);
+      break;
+    case Lisp_Fwd_Kboard_Obj:
+      offset = dump_fwd_kboard_obj (ctx, &fwd->u_kboard_objfwd);
+      break;
+    default:
+      emacs_abort ();
+    }
+
+  return offset;
+}
+
+static ptrdiff_t
+dump_blv (struct dump_context *ctx,
+          const struct Lisp_Buffer_Local_Value *blv)
+{
+  struct Lisp_Buffer_Local_Value out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, blv, local_if_set);
+  DUMP_FIELD_COPY (&out, blv, frame_local);
+  DUMP_FIELD_COPY (&out, blv, found);
+  if (blv->fwd)
+    dump_field_fixup_later (ctx, &out, blv, &blv->fwd);
+  dump_field_lv (ctx, &out, blv, &blv->where);
+  dump_field_lv (ctx, &out, blv, &blv->defcell);
+  dump_field_lv (ctx, &out, blv, &blv->valcell);
+  ptrdiff_t offset = dump_object_finish (ctx, &out, sizeof (out));
+  if (blv->fwd)
+    dump_remember_fixup_ptr_raw (
+      ctx,
+      offset + offsetof (struct Lisp_Buffer_Local_Value, fwd),
+      dump_fwd (ctx, blv->fwd));
+  return offset;
+}
+
+static ptrdiff_t
+dump_symbol (struct dump_context *ctx, const struct Lisp_Symbol *symbol)
+{
+  struct Lisp_Symbol out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  eassert (symbol->gcmarkbit == 0);
+  DUMP_FIELD_COPY (&out, symbol, redirect);
+  DUMP_FIELD_COPY (&out, symbol, constant);
+  DUMP_FIELD_COPY (&out, symbol, interned);
+  DUMP_FIELD_COPY (&out, symbol, declared_special);
+  DUMP_FIELD_COPY (&out, symbol, pinned);
+  dump_field_lv (ctx, &out, symbol, &symbol->name);
+  switch (symbol->redirect)
+    {
+    case SYMBOL_PLAINVAL:
+      dump_field_lv (ctx, &out, symbol, &symbol->val.value);
+      break;
+    case SYMBOL_VARALIAS:
+      dump_field_lv_rawptr (ctx, &out, symbol,
+                            &symbol->val.alias, Lisp_Symbol);
+      break;
+    case SYMBOL_LOCALIZED:
+      dump_field_fixup_later (ctx, &out, symbol, &symbol->val.blv);
+      break;
+    case SYMBOL_FORWARDED:
+      dump_field_fixup_later (ctx, &out, symbol, &symbol->val.fwd);
+      break;
+    default:
+      emacs_abort ();
+    }
+  dump_field_lv (ctx, &out, symbol, &symbol->function);
+  dump_field_lv (ctx, &out, symbol, &symbol->plist);
+  dump_field_lv_rawptr (ctx, &out, symbol, &symbol->next, Lisp_Symbol);
+
+  // XXX: linearize symbol chains
+
+  ptrdiff_t offset = dump_object_finish (ctx, &out, sizeof (out));
+
+  switch (symbol->redirect)
+    {
+    case SYMBOL_LOCALIZED:
+      dump_remember_fixup_ptr_raw (
+        ctx,
+        offset + offsetof (struct Lisp_Symbol, val.blv),
+        dump_blv (ctx, symbol->val.blv));
+      break;
+    case SYMBOL_FORWARDED:
+      dump_remember_fixup_ptr_raw (
+        ctx,
+        offset + offsetof (struct Lisp_Symbol, val.fwd),
+        dump_fwd (ctx, symbol->val.fwd));
+      break;
+    default:
+      break;
+    }
+  return offset;
+}
+
+static ptrdiff_t
+dump_vectorlike_generic (
+  struct dump_context *ctx,
+  const struct vectorlike_header *header)
+{
+  const struct Lisp_Vector *v = (const struct Lisp_Vector *) header;
+  ptrdiff_t size = header->size;
+  enum pvec_type pvectype = PSEUDOVECTOR_TYPE (header);
+  ptrdiff_t offset;
+
+  if (size & PSEUDOVECTOR_FLAG)
+    {
+      /* Assert that the pseudovector contains only Lisp values ---
+         but see the PVEC_SUB_CHAR_TABLE special case below.  */
+      eassert (((size & PSEUDOVECTOR_REST_MASK)
+                >> PSEUDOVECTOR_REST_BITS) == 0);
+      size &= PSEUDOVECTOR_SIZE_MASK;
+    }
+
+  dump_align_output (ctx, GCALIGNMENT);
+  ptrdiff_t prefix_start_offset = ctx->offset;
+
+  ptrdiff_t skip;
+  if (pvectype == PVEC_SUB_CHAR_TABLE)
+    {
+      /* PVEC_SUB_CHAR_TABLE has a special case because it's a
+         variable-length vector (unlike other pseudovectors) and has
+         its non-Lisp data _before_ the variable-length Lisp part.  */
+      const struct Lisp_Sub_Char_Table *sct =
+        (const struct Lisp_Sub_Char_Table *) header;
+      struct Lisp_Sub_Char_Table out;
+      dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+      DUMP_FIELD_COPY (&out, sct, header.size);
+      DUMP_FIELD_COPY (&out, sct, depth);
+      DUMP_FIELD_COPY (&out, sct, min_char);
+      offset = dump_object_finish (ctx, &out, sizeof (out));
+      skip = SUB_CHAR_TABLE_OFFSET;
+    }
+  else
+    {
+      struct vectorlike_header out;
+      dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+      DUMP_FIELD_COPY (&out, header, size);
+      offset = dump_object_finish (ctx, &out, sizeof (out));
+      skip = 0;
+    }
+
+  ptrdiff_t prefix_size = ctx->offset - prefix_start_offset;
+  eassert (prefix_size > 0);
+  ptrdiff_t skip_start = (char*) &v->contents[skip] - (char*) v;
+  eassert (skip_start >= prefix_size);
+  dump_write_zero (ctx, skip_start - prefix_size);
+  for (ptrdiff_t i = skip; i < size; ++i)
+    {
+      Lisp_Object out;
+      const Lisp_Object *vslot = &v->contents[i];
+      eassert (ctx->offset % sizeof (out) == 0);
+      dump_object_start (ctx, 1, &out, sizeof (out));
+      dump_field_lv (ctx, &out, vslot, vslot);
+      dump_object_finish (ctx, &out, sizeof (out));
+    }
+
+  return offset;
+}
+
+static void
+dump_object_start_pseudovector (
+  struct dump_context *ctx,
+  struct vectorlike_header *out_hdr,
+  ptrdiff_t out_size,
+  const struct vectorlike_header *in_hdr)
+{
+  const struct Lisp_Vector *in = (const struct Lisp_Vector *) in_hdr;
+  struct Lisp_Vector *out = (struct Lisp_Vector *) out_hdr;
+
+  eassert (vector_nbytes ((struct Lisp_Vector *) in) == out_size);
+
+  dump_object_start (ctx, GCALIGNMENT, out, out_size);
+  DUMP_FIELD_COPY (out, in, header);
+  ptrdiff_t size = in->header.size;
+  eassert (size & PSEUDOVECTOR_FLAG);
+  size &= PSEUDOVECTOR_SIZE_MASK;
+  for (ptrdiff_t i = 0; i < size; ++i)
+    dump_field_lv (ctx, out, in, &in->contents[i]);
+}
+
+/* Determine whether the hash table's hash order is stable
+   across dump and load.  If it is, we don't have to trigger
+   a rehash on access.  */
+static bool
+dump_hash_table_stable_p (struct Lisp_Hash_Table *hash)
+{
+  bool is_eql = hash->test.hashfn == hashfn_eql;
+  bool is_equal = hash->test.hashfn == hashfn_equal;
+  ptrdiff_t size = HASH_TABLE_SIZE (hash);
+  for (ptrdiff_t i = 0; i < size; ++i)
+    if (!NILP (HASH_HASH (hash, i)))
+      {
+        Lisp_Object key =  HASH_KEY (hash, i);
+        if (!(dump_builtin_symbol_p (key) ||
+              INTEGERP (key) ||
+              (is_equal && STRINGP (key)) ||
+              ((is_equal || is_eql) && FLOATP (key))))
+          return false;
+      }
+
+  return true;
+}
+
+static ptrdiff_t
+dump_hash_table (struct dump_context *ctx,
+                 const struct Lisp_Hash_Table *hash_in)
+{
+  struct Lisp_Hash_Table hash_munged = *hash_in;
+  struct Lisp_Hash_Table *hash = &hash_munged;
+
+  /* Remember to rehash this hash table on first access.  After a
+     dump reload, the hash table values will have changed, so we'll
+     need to rebuild the index.
+
+     TODO: for EQ and EQL hash tables, it should be possible to rehash
+     here using the preferred load address of the dump, eliminating
+     the need to rehash-on-access if we can load the dump where we
+     want.  */
+  if (hash->count > 0 && !dump_hash_table_stable_p (hash))
+    hash->count = -hash->count;
+
+  struct Lisp_Hash_Table out;
+  dump_object_start_pseudovector (
+    ctx, &out.header, sizeof (out), &hash->header);
+  DUMP_FIELD_COPY (&out, hash, count);
+  dump_field_lv (ctx, &out, hash, &hash->key_and_value);
+  dump_field_lv (ctx, &out, hash, &hash->test.name);
+  dump_field_lv (ctx, &out, hash, &hash->test.user_hash_function);
+  dump_field_lv (ctx, &out, hash, &hash->test.user_cmp_function);
+  dump_field_emacs_ptr (ctx, &out, hash, &hash->test.cmpfn);
+  dump_field_emacs_ptr (ctx, &out, hash, &hash->test.hashfn);
+  dump_field_lv_rawptr (ctx, &out, hash, &hash->next_weak, Lisp_Vectorlike);
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static ptrdiff_t
+dump_buffer (struct dump_context *ctx, const struct buffer *in_buffer)
+{
+  struct buffer munged_buffer = *in_buffer;
+  struct buffer *buffer = &munged_buffer;
+
+  /* Clear some buffer state for correctness upon load.  */
+  if (buffer->base_buffer == NULL)
+    buffer->window_count = 0;
+  else
+    eassert (buffer->window_count == -1);
+  buffer->last_selected_window_ = Qnil;
+  buffer->display_count_ = make_number (0);
+  buffer->clip_changed = 0;
+  buffer->last_window_start = -1;
+  buffer->point_before_scroll_ = Qnil;
+
+  ptrdiff_t base_offset = 0;
+  if (buffer->base_buffer)
+    {
+      eassert (buffer->base_buffer->base_buffer == NULL);
+      base_offset = dump_object (
+        ctx,
+        make_lisp_ptr (buffer->base_buffer, Lisp_Vectorlike));
+    }
+
+  eassert ((base_offset == 0 && buffer->text == &in_buffer->own_text) ||
+           (base_offset > 0 && buffer->text != &in_buffer->own_text));
+
+  struct buffer out;
+  dump_object_start_pseudovector (
+    ctx, &out.header, sizeof (out), &buffer->header);
+  if (base_offset == 0)
+    base_offset = ctx->obj_offset;
+  eassert (base_offset > 0);
+  if (buffer->base_buffer == NULL)
+    {
+      eassert (base_offset == ctx->obj_offset);
+
+      if (BUFFER_LIVE_P (buffer))
+        {
+          dump_field_fixup_later (ctx, &out, buffer, &buffer->own_text.beg);
+          dump_remember_cold_op (
+            ctx,
+            COLD_OP_BUFFER,
+            make_lisp_ptr ((void*) in_buffer, Lisp_Vectorlike));
+        }
+      else
+        eassert (buffer->own_text.beg == NULL);
+
+      DUMP_FIELD_COPY (&out, buffer, own_text.gpt);
+      DUMP_FIELD_COPY (&out, buffer, own_text.z);
+      DUMP_FIELD_COPY (&out, buffer, own_text.gpt_byte);
+      DUMP_FIELD_COPY (&out, buffer, own_text.z_byte);
+      DUMP_FIELD_COPY (&out, buffer, own_text.gap_size);
+      DUMP_FIELD_COPY (&out, buffer, own_text.modiff);
+      DUMP_FIELD_COPY (&out, buffer, own_text.chars_modiff);
+      DUMP_FIELD_COPY (&out, buffer, own_text.save_modiff);
+      DUMP_FIELD_COPY (&out, buffer, own_text.overlay_modiff);
+      DUMP_FIELD_COPY (&out, buffer, own_text.compact);
+      DUMP_FIELD_COPY (&out, buffer, own_text.beg_unchanged);
+      DUMP_FIELD_COPY (&out, buffer, own_text.end_unchanged);
+      DUMP_FIELD_COPY (&out, buffer, own_text.unchanged_modified);
+      DUMP_FIELD_COPY (&out, buffer, own_text.overlay_unchanged_modified);
+      if (buffer->own_text.intervals)
+        dump_field_fixup_later (ctx, &out, buffer, &buffer->own_text.intervals);
+      dump_field_lv_rawptr (ctx, &out, buffer, &buffer->own_text.markers,
+                            Lisp_Misc);
+      DUMP_FIELD_COPY (&out, buffer, own_text.inhibit_shrinking);
+      DUMP_FIELD_COPY (&out, buffer, own_text.redisplay);
+    }
+
+  eassert (ctx->obj_offset > 0);
+  dump_remember_fixup_ptr_raw (
+    ctx,
+    ctx->obj_offset + offsetof (struct buffer, text),
+    base_offset + offsetof (struct buffer, own_text));
+
+  dump_field_lv_rawptr (ctx, &out, buffer, &buffer->next, Lisp_Vectorlike);
+  DUMP_FIELD_COPY (&out, buffer, pt);
+  DUMP_FIELD_COPY (&out, buffer, pt_byte);
+  DUMP_FIELD_COPY (&out, buffer, begv);
+  DUMP_FIELD_COPY (&out, buffer, begv_byte);
+  DUMP_FIELD_COPY (&out, buffer, zv);
+  DUMP_FIELD_COPY (&out, buffer, zv_byte);
+
+  if (buffer->base_buffer)
+    {
+      eassert (ctx->obj_offset != base_offset);
+      dump_field_ptr_to_dump_offset (
+        ctx, &out, buffer, &buffer->base_buffer,
+        base_offset);
+    }
+
+  DUMP_FIELD_COPY (&out, buffer, indirections);
+  DUMP_FIELD_COPY (&out, buffer, window_count);
+
+  memcpy (&out.local_flags,
+          &buffer->local_flags,
+          sizeof (out.local_flags));
+  DUMP_FIELD_COPY (&out, buffer, modtime);
+  DUMP_FIELD_COPY (&out, buffer, modtime_size);
+  DUMP_FIELD_COPY (&out, buffer, auto_save_modified);
+  DUMP_FIELD_COPY (&out, buffer, display_error_modiff);
+  DUMP_FIELD_COPY (&out, buffer, auto_save_failure_time);
+  DUMP_FIELD_COPY (&out, buffer, last_window_start);
+
+  /* Not worth serializing these caches.  TODO: really? */
+  out.newline_cache = NULL;
+  out.width_run_cache = NULL;
+  out.bidi_paragraph_cache = NULL;
+
+  DUMP_FIELD_COPY (&out, buffer, prevent_redisplay_optimizations_p);
+  DUMP_FIELD_COPY (&out, buffer, clip_changed);
+
+  dump_field_lv_rawptr (ctx, &out, buffer, &buffer->overlays_before,
+                        Lisp_Misc);
+
+  dump_field_lv_rawptr (ctx, &out, buffer, &buffer->overlays_after,
+                        Lisp_Misc);
+
+  DUMP_FIELD_COPY (&out, buffer, overlay_center);
+  dump_field_lv (ctx, &out, buffer, &buffer->undo_list_);
+  ptrdiff_t offset = dump_object_finish (ctx, &out, sizeof (out));
+  if (!buffer->base_buffer && buffer->own_text.intervals)
+    dump_remember_fixup_ptr_raw (
+      ctx,
+      offset + offsetof (struct buffer, own_text.intervals),
+      dump_interval_tree (ctx, buffer->own_text.intervals, 0));
+
+  return offset;
+}
+
+static ptrdiff_t
+dump_bool_vector (struct dump_context *ctx, const struct Lisp_Vector *v)
+{
+  /* No relocation needed, so we don't need dump_object_start.  */
+  dump_align_output (ctx, GCALIGNMENT);
+  eassert (ctx->offset >= ctx->header.hot_end);
+  ptrdiff_t offset = ctx->offset;
+  ptrdiff_t nbytes = vector_nbytes ((struct Lisp_Vector *) v);
+  dump_write (ctx, v, nbytes);
+  return offset;
+}
+
+static ptrdiff_t
+dump_subr (struct dump_context *ctx, const struct Lisp_Subr *subr)
+{
+  struct Lisp_Subr out;
+  dump_object_start (ctx, GCALIGNMENT, &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, subr, header.size);
+  dump_field_emacs_ptr (ctx, &out, subr, &subr->function.a0);
+  DUMP_FIELD_COPY (&out, subr, min_args);
+  DUMP_FIELD_COPY (&out, subr, max_args);
+  dump_field_emacs_ptr (ctx, &out, subr, &subr->symbol_name);
+  dump_field_emacs_ptr (ctx, &out, subr, &subr->intspec);
+  DUMP_FIELD_COPY (&out, subr, doc);
+  return dump_object_finish (ctx, &out, sizeof (out));
+}
+
+static void
+fill_pseudovec (struct vectorlike_header *header, Lisp_Object item)
+{
+  struct Lisp_Vector *v = (struct Lisp_Vector *) header;
+  eassert (v->header.size & PSEUDOVECTOR_FLAG);
+  ptrdiff_t size = v->header.size & PSEUDOVECTOR_SIZE_MASK;
+  for (ptrdiff_t idx = 0; idx < size; idx++)
+    v->contents[idx] = item;
+}
+
+static ptrdiff_t
+dump_nilled_pseudovec (struct dump_context *ctx,
+                       const struct vectorlike_header *in)
+{
+  ptrdiff_t nbytes = vector_nbytes ((struct Lisp_Vector *) in);
+  struct vectorlike_header *in_nilled = alloca (nbytes);
+  memset (in_nilled, 0, nbytes);
+  in_nilled->size = in->size;
+  fill_pseudovec (in_nilled, Qnil);
+  struct vectorlike_header *out = alloca (nbytes);
+  memset (out, 0, nbytes);
+  dump_object_start_pseudovector (ctx, out, nbytes, in_nilled);
+  return dump_object_finish (ctx, out, nbytes);
+}
+
+static ptrdiff_t
+dump_vectorlike (struct dump_context *ctx, const struct Lisp_Vector *v)
+{
+  ptrdiff_t offset;
+  Lisp_Object lv = make_lisp_ptr ((void *) v, Lisp_Vectorlike);
+  switch (PSEUDOVECTOR_TYPE (&v->header))
+    {
+    case PVEC_FONT:
+      /* There are three kinds of font objects that all use PVEC_FONT,
+         distinguished by their size.  Font specs and entities are
+         harmless data carriers that we can dump like other Lisp
+         objects.  Fonts themselves are window-system-specific and
+         need to be recreated on each startup.  */
+      if ((v->header.size & PSEUDOVECTOR_SIZE_MASK) != FONT_SPEC_MAX &&
+          (v->header.size & PSEUDOVECTOR_SIZE_MASK) != FONT_ENTITY_MAX)
+        error_unsupported_dump_object(ctx, lv, "font");
+      /* Fall through */
+    case PVEC_NORMAL_VECTOR:
+    case PVEC_COMPILED:
+    case PVEC_CHAR_TABLE:
+    case PVEC_SUB_CHAR_TABLE:
+      offset = dump_vectorlike_generic (ctx, &v->header);
+      break;
+    case PVEC_BOOL_VECTOR:
+      offset = dump_bool_vector(ctx, v);
+      break;
+    case PVEC_HASH_TABLE:
+      offset = dump_hash_table (ctx, (struct Lisp_Hash_Table *) v);
+      break;
+    case PVEC_BUFFER:
+      offset = dump_buffer (ctx, (struct buffer *) v);
+      break;
+    case PVEC_SUBR:
+      offset = dump_subr(ctx, (const struct Lisp_Subr *) v);
+      break;
+    case PVEC_FRAME:
+    case PVEC_WINDOW:
+    case PVEC_PROCESS:
+    case PVEC_TERMINAL:
+      offset = dump_nilled_pseudovec (ctx, &v->header);
+      break;
+    case PVEC_WINDOW_CONFIGURATION:
+      error_unsupported_dump_object(ctx, lv, "window configuration");
+    case PVEC_OTHER:
+      error_unsupported_dump_object(ctx, lv, "other?!");
+    case PVEC_XWIDGET:
+      error_unsupported_dump_object(ctx, lv, "xwidget");
+    case PVEC_XWIDGET_VIEW:
+      error_unsupported_dump_object(ctx, lv, "xwidget view");
+    default:
+      error_unsupported_dump_object(ctx, lv, "weird pseudovector");
+    }
+
+  return offset;
+}
+
+/* Add an object to the dump.  */
+static ptrdiff_t
+dump_object_1 (struct dump_context *ctx, Lisp_Object object, int flags)
+{
+#ifdef ENABLE_CHECKING
+  eassert (!EQ (object, Vdead));
+#endif
+
+  if (flags & DUMP_OBJECT_DRY_RUN)
+    flags &= ~(DUMP_OBJECT_INTERN | DUMP_OBJECT_RECORD_START);
+
+  int saved_flags = ctx->flags;
+  flags |= ctx->flags;
+  ctx->flags = flags;
+
+  ptrdiff_t offset = dump_recall_object (ctx, object);
+  if (flags & DUMP_OBJECT_INTERN)
+    eassert (!dump_object_self_representing_p (object));
+
+  if (offset <= 0)
+    {
+      DUMP_SET_REFERRER (ctx, object);
+      switch (XTYPE (object))
+        {
+        case Lisp_String:
+          offset = dump_string (ctx, XSTRING (object));
+          break;
+        case Lisp_Vectorlike:
+          offset = dump_vectorlike (ctx, XVECTOR (object));
+          break;
+        case Lisp_Symbol:
+          offset = dump_symbol (ctx, XSYMBOL (object));
+          break;
+        case Lisp_Misc:
+          offset = dump_misc_any (ctx, XMISCANY (object));
+          break;
+        case Lisp_Cons:
+          offset = dump_cons (ctx, XCONS (object));
+          break;
+        case Lisp_Float:
+          offset = dump_float (ctx, XFLOAT (object));
+          break;
+        case_Lisp_Int:
+          eassert (!"should not be dumping int: is self-representing");
+        default:
+          emacs_abort ();
+        }
+      eassert (offset > 0);
+      if (flags & DUMP_OBJECT_INTERN)
+        dump_remember_object (ctx, object, offset);
+      if (flags & DUMP_OBJECT_RECORD_START)
+          dump_push (&ctx->object_starts,
+                     list2 (INTEGER_TO_CONS (XTYPE (object)),
+                            INTEGER_TO_CONS (offset)));
+
+      dump_clear_referrer (ctx);
+
+      /* If we dumped a cons cell, we put its car and cdr on the dump
+         queue; we'll eventually get around to dumping them.  That's
+         fine from a correctness perspective, but but Lisp has lots of
+         lists, and code likes to traverse lists.  Make sure the cons
+         cells for reasonable-sized lists are dumped next to each
+         other.  */
+      if (CONSP (object) &&
+          CONSP (XCDR (object)) &&
+          flags == (DUMP_OBJECT_INTERN | DUMP_OBJECT_RECORD_START) &&
+          ctx->cons_chain_depth < max_cons_chain_depth)
+        {
+          ctx->cons_chain_depth += 1;
+          dump_object (ctx, XCDR (object));
+          ctx->cons_chain_depth -= 1;
+        }
+    }
+
+  ctx->flags = saved_flags;
+  return offset;
+}
+
+static ptrdiff_t
+dump_object (struct dump_context *ctx, Lisp_Object object)
+{
+  ptrdiff_t result;
+  if (dump_object_emacs_ptr (object) != NULL)
+    {
+      result = dump_recall_object (ctx, object);
+      eassert (result < 0);
+      if (result > -2)
+        {
+          dump_object_1 (ctx, object, DUMP_OBJECT_DRY_RUN);
+          dump_push (&ctx->copied_queue, object);
+          result = -2;
+          dump_remember_object (ctx, object, result);
+        }
+    }
+  else
+    result = dump_object_1 (
+      ctx,
+      object,
+      DUMP_OBJECT_INTERN | DUMP_OBJECT_RECORD_START);
+  return result;
+}
+
+static ptrdiff_t
+dump_charset (struct dump_context *ctx, int cs_i)
+{
+  const struct charset *cs = charset_table + cs_i;
+  struct charset out;
+  dump_object_start (ctx, sizeof (int), &out, sizeof (out));
+  DUMP_FIELD_COPY (&out, cs, id);
+  DUMP_FIELD_COPY (&out, cs, hash_index);
+  DUMP_FIELD_COPY (&out, cs, dimension);
+  memcpy (out.code_space, &cs->code_space, sizeof (cs->code_space));
+  if (cs->code_space_mask)
+    dump_field_fixup_later (ctx, &out, cs, &cs->code_space_mask);
+  DUMP_FIELD_COPY (&out, cs, code_linear_p);
+  DUMP_FIELD_COPY (&out, cs, iso_chars_96);
+  DUMP_FIELD_COPY (&out, cs, ascii_compatible_p);
+  DUMP_FIELD_COPY (&out, cs, supplementary_p);
+  DUMP_FIELD_COPY (&out, cs, compact_codes_p);
+  DUMP_FIELD_COPY (&out, cs, unified_p);
+  DUMP_FIELD_COPY (&out, cs, iso_final);
+  DUMP_FIELD_COPY (&out, cs, iso_revision);
+  DUMP_FIELD_COPY (&out, cs, emacs_mule_id);
+  DUMP_FIELD_COPY (&out, cs, method);
+  DUMP_FIELD_COPY (&out, cs, min_code);
+  DUMP_FIELD_COPY (&out, cs, max_code);
+  DUMP_FIELD_COPY (&out, cs, char_index_offset);
+  DUMP_FIELD_COPY (&out, cs, min_char);
+  DUMP_FIELD_COPY (&out, cs, max_char);
+  DUMP_FIELD_COPY (&out, cs, invalid_code);
+  memcpy (out.fast_map, &cs->fast_map, sizeof (cs->fast_map));
+  DUMP_FIELD_COPY (&out, cs, code_offset);
+  ptrdiff_t offset = dump_object_finish (ctx, &out, sizeof (out));
+  if (cs->code_space_mask)
+    dump_remember_cold_op (ctx, COLD_OP_CHARSET,
+                           Fcons (INTEGER_TO_CONS (cs_i),
+                                  INTEGER_TO_CONS (offset)));
+  return offset;
+}
+
+static ptrdiff_t
+dump_charset_table (struct dump_context *ctx)
+{
+  dump_align_output (ctx, GCALIGNMENT);
+  ptrdiff_t offset = ctx->offset;
+  for (int i = 0; i < charset_table_used; ++i)
+    dump_charset (ctx, i);
+  dump_emacs_reloc_to_dump_ptr_raw (ctx, &charset_table, offset);
+  dump_emacs_reloc_immediate_int (
+    ctx, &charset_table_used, charset_table_used);
+  dump_emacs_reloc_immediate_emacs_int (
+    ctx, &charset_table_size, charset_table_used);
+  return offset;
+}
+
+static void
+dump_finalizer_list_head_ptr (struct dump_context *ctx,
+                              struct Lisp_Finalizer **ptr)
+{
+  struct Lisp_Finalizer *value = *ptr;
+  if (value != &finalizers && value != &doomed_finalizers)
+    dump_emacs_reloc_to_dump_ptr_raw (
+      ctx, ptr,
+      dump_object (ctx, make_lisp_ptr (value, Lisp_Misc)));
+}
+
+static void
+dump_metadata_for_pdumper (struct dump_context *ctx)
+{
+  for (int i = 0; i < nr_dump_hooks; ++i)
+    dump_emacs_reloc_to_emacs_ptr_raw (ctx, &dump_hooks[i], dump_hooks[i]);
+  dump_emacs_reloc_immediate_int (ctx, &nr_dump_hooks, nr_dump_hooks);
+
+  for (int i = 0; i < nr_remembered_data; ++i)
+    {
+      dump_emacs_reloc_to_emacs_ptr_raw (
+        ctx,
+        &remembered_data[i].mem,
+        remembered_data[i].mem);
+      dump_emacs_reloc_immediate_int (
+        ctx,
+        &remembered_data[i].sz,
+        remembered_data[i].sz);
+    }
+  dump_emacs_reloc_immediate_int (
+    ctx,
+    &nr_remembered_data,
+    nr_remembered_data);
+}
+
+static void
+dump_copied_objects (struct dump_context *ctx)
+{
+  /* Sort the objects into the order in which they'll appear in Emacs.  */
+  Lisp_Object copied_queue =
+    Fsort (Fnreverse (ctx->copied_queue),
+           Qdump_emacs_portable__sort_predicate_copied);
+  ctx->copied_queue = Qnil;
+
+  /* Dump the objects and generate a copy relocation for each.  We'll
+     merge adjacent copy relocations upon output.  */
+  while (!NILP (copied_queue))
+    {
+      Lisp_Object copied = dump_pop (&copied_queue);
+      void *optr = dump_object_emacs_ptr (copied);
+      eassert (optr != NULL);
+      ptrdiff_t offset = dump_object_1 (
+        ctx,
+        copied,
+        ( DUMP_OBJECT_FORCE_WORD_ALIGNMENT
+          | DUMP_OBJECT_PROHIBIT_ENQUEUE));
+      ptrdiff_t size = ctx->offset - offset;
+      dump_emacs_reloc_copy_from_dump (ctx, offset, optr, size);
+    }
+}
+
+static void
+dump_cold_string (struct dump_context *ctx, Lisp_Object string)
+{
+  /* Dump string contents.  */
+  ptrdiff_t string_offset = dump_recall_object (ctx, string);
+  eassert (string_offset > 0);
+  ptrdiff_t total_size = SBYTES (string) + 1;
+  eassert (total_size > 0);
+  dump_remember_fixup_ptr_raw (
+    ctx,
+    string_offset + offsetof (struct Lisp_String, data),
+    ctx->offset);
+  dump_write (ctx, XSTRING (string)->data, total_size);
+}
+
+static void
+dump_cold_charset (struct dump_context *ctx, Lisp_Object data)
+{
+  /* Dump charset lookup tables.  */
+  int cs_i = XFASTINT (XCAR (data));
+  ptrdiff_t cs_dump_offset = ptrdiff_t_from_lisp (XCDR (data));
+  dump_remember_fixup_ptr_raw (
+    ctx,
+    cs_dump_offset + offsetof (struct charset, code_space_mask),
+    ctx->offset);
+  struct charset *cs = charset_table + cs_i;
+  dump_write (ctx, cs->code_space_mask, 256);
+}
+
+static void
+dump_cold_buffer (struct dump_context *ctx, Lisp_Object data)
+{
+  /* Dump buffer text.  */
+  ptrdiff_t buffer_offset = dump_recall_object (ctx, data);
+  eassert (buffer_offset > 0);
+  struct buffer *b = XBUFFER (data);
+  eassert (b->text == &b->own_text);
+  /* Zero the gap so we don't dump uninitialized bytes.  */
+  memset (BUF_GPT_ADDR (b), 0, BUF_GAP_SIZE (b));
+  /* See buffer.c for this calculation.  */
+  ptrdiff_t nbytes = BUF_Z_BYTE (b) - BUF_BEG_BYTE (b) + BUF_GAP_SIZE (b) + 1;
+  dump_remember_fixup_ptr_raw (
+    ctx,
+    buffer_offset + offsetof (struct buffer, own_text.beg),
+    ctx->offset);
+  dump_write (ctx, b->own_text.beg, nbytes);
+}
+
+static void
+dump_cold_data (struct dump_context *ctx)
+{
+  if (!NILP (ctx->cold_queue))
+    {
+      Lisp_Object cold_queue = Fnreverse (ctx->cold_queue);
+      ctx->cold_queue = Qnil;
+      while (!NILP (cold_queue))
+        {
+          Lisp_Object item = dump_pop (&cold_queue);
+          enum cold_op op = XFASTINT (XCAR (item));
+          Lisp_Object data = XCDR (item);
+          switch (op)
+            {
+            case COLD_OP_STRING:
+              dump_cold_string (ctx, data);
+              break;
+            case COLD_OP_CHARSET:
+              dump_cold_charset (ctx, data);
+              break;
+            case COLD_OP_BUFFER:
+              dump_cold_buffer (ctx, data);
+              break;
+            case COLD_OP_OBJECT:
+              /* Objects that we can put in the cold section
+                 must not refer to other objects.  */
+              eassert (dump_tailq_empty_p (&ctx->dump_queue));
+              dump_object (ctx, data);
+              eassert (dump_tailq_empty_p (&ctx->dump_queue));
+              break;
+            default:
+              emacs_abort ();
+            }
+        }
+    }
+}
+
+static void
+read_raw_ptr_and_lv (const void *mem,
+                     enum Lisp_Type type,
+                     void **out_ptr,
+                     Lisp_Object *out_lv)
+{
+  memcpy (out_ptr, mem, sizeof (*out_ptr));
+  if (*out_ptr != NULL)
+    {
+      switch (type)
+        {
+        case Lisp_Symbol:
+          *out_lv = make_lisp_symbol (*out_ptr);
+          break;
+        case Lisp_Misc:
+        case Lisp_String:
+        case Lisp_Vectorlike:
+        case Lisp_Cons:
+        case Lisp_Float:
+          *out_lv = make_lisp_ptr (*out_ptr, type);
+          break;
+        default:
+          emacs_abort ();
+        }
+    }
+}
+
+/* Enqueue for dumping objects referenced by static non-Lisp_Object
+   pointers inside Emacs.  */
+static void
+dump_user_remembered_data_hot (struct dump_context *ctx)
+{
+  for (int i = 0; i < nr_remembered_data; ++i)
+    {
+      void *mem = remembered_data[i].mem;
+      int sz = remembered_data[i].sz;
+      if (sz <= 0)
+        {
+          enum Lisp_Type type = -sz;
+          void *value;
+          Lisp_Object lv;
+          read_raw_ptr_and_lv (mem, type, &value, &lv);
+          if (value != NULL)
+            {
+              DUMP_SET_REFERRER (ctx, dump_ptr_referrer ("user data", mem));
+              dump_enqueue_object (ctx, lv);
+              dump_clear_referrer (ctx);
+            }
+        }
+    }
+}
+
+/* Dump user-specified non-relocated data.  */
+static void
+dump_user_remembered_data_cold (struct dump_context *ctx)
+{
+  for (int i = 0; i < nr_remembered_data; ++i)
+    {
+      void *mem = remembered_data[i].mem;
+      int sz = remembered_data[i].sz;
+      if (sz > 0)
+        {
+          /* Scalar: try to inline the value into the relocation if
+             it's small enough; if it's bigger than we can fit in a
+             relocation, we have to copy the data into the dump proper
+             and issue a copy relocation.  */
+          if (sz <= sizeof (intmax_t))
+            dump_emacs_reloc_immediate (ctx, mem, mem, sz);
+          else
+            {
+              dump_emacs_reloc_copy_from_dump (ctx, ctx->offset, mem, sz);
+              dump_write (ctx, mem, sz);
+            }
+        }
+      else
+        {
+          /* *mem is a raw pointer to a Lisp object of some sort.
+             The object to which it points should have already been
+             dumped by dump_user_remembered_data_hot.  */
+          void *value;
+          Lisp_Object lv;
+          enum Lisp_Type type = -sz;
+          read_raw_ptr_and_lv (mem, type, &value, &lv);
+          if (value == NULL)
+            /* We can't just ignore NULL: the variable might have
+               transitioned from non-NULL to NULL, and we want to
+               record this fact.  */
+            dump_emacs_reloc_immediate_ptrdiff_t (ctx, mem, 0);
+          else
+            {
+              if (dump_object_emacs_ptr (lv) != NULL)
+                {
+                  /* We have situation like this:
+
+                     static Lisp_Symbol *foo;
+                     ...
+                     foo = XSYMBOL(Qt);
+                     ...
+                     pdumper_remember_lv_raw_ptr (&foo, Lisp_Symbol);
+
+                     Built-in symbols like Qt aren't in the dump!
+                     They're actually in Emacs proper.  We need a
+                     special case to point this value back at Emacs
+                     instead of to something in the dump that
+                     isn't there.
+
+                     An analogous situation applies to subrs, since
+                     Lisp_Subr structures always live in Emacs, not
+                     the dump.
+
+                  */
+                  dump_emacs_reloc_to_emacs_ptr_raw (
+                    ctx, mem, dump_object_emacs_ptr (lv));
+                }
+              else
+                {
+                  eassert (!dump_object_self_representing_p (lv));
+                  ptrdiff_t dump_offset = dump_recall_object (ctx, lv);
+                  if (dump_offset <= 0)
+                    error ("raw-pointer object not dumped?!");
+                  dump_emacs_reloc_to_dump_ptr_raw (ctx, mem, dump_offset);
+                }
+            }
+        }
+    }
+}
+
+static void
+dump_unwind_cleanup (void *data)
+{
+  // XXX: omit relocations that duplicate BSS?
+  // XXX: prevent ralloc moving
+  // XXX: dumb mode for GC
+  struct dump_context *ctx = data;
+  if (ctx->fd >= 0)
+    emacs_close (ctx->fd);
+  Vpurify_flag = ctx->old_purify_flag;
+  unblock_input ();
+}
+
+static Lisp_Object
+make_eq_hash_table (void)
+{
+  return CALLN (Fmake_hash_table, QCtest, Qeq);
+}
+
+static void
+dump_do_fixup (struct dump_context *ctx, Lisp_Object fixup)
+{
+  enum dump_fixup_type type = XFASTINT (XCAR (fixup));
+  fixup = XCDR (fixup);
+  ptrdiff_t dump_fixup_offset = ptrdiff_t_from_lisp (XCAR (fixup));
+  fixup = XCDR (fixup);
+  Lisp_Object arg = XCAR (fixup);
+  eassert (NILP (XCDR (fixup)));
+  dump_seek (ctx, dump_fixup_offset);
+  ptrdiff_t target_offset;
+  bool do_write = true;
+  switch (type)
+    {
+    case DUMP_FIXUP_LISP_OBJECT:
+    case DUMP_FIXUP_LISP_OBJECT_RAW:
+      /* Dump wants a pointer to a Lisp object.
+         If DUMP_FIXUP_LISP_OBJECT_RAW, we should stick a C pointer in
+         the dump; otherwise, a Lisp_Object.  */
+      if (SUBRP (arg))
+        {
+          target_offset = emacs_offset (XSUBR (arg));
+          if (type == DUMP_FIXUP_LISP_OBJECT)
+            dump_reloc_dump_to_emacs_lv (ctx, ctx->offset, XTYPE (arg));
+          else
+            dump_reloc_dump_to_emacs_raw_ptr (ctx, ctx->offset);
+        }
+      else if (dump_builtin_symbol_p (arg))
+        {
+          eassert (dump_object_self_representing_p (arg));
+          /* These symbols are part of Emacs, so point there.  If we
+             want a Lisp_Object, we're set.  If we want a raw pointer,
+             we need to emit a relocation.  */
+          if (type == DUMP_FIXUP_LISP_OBJECT)
+            {
+              do_write = false;
+              dump_write (ctx, &arg, sizeof (arg));
+            }
+          else
+            {
+              target_offset = emacs_offset (XSYMBOL (arg));
+              dump_reloc_dump_to_emacs_raw_ptr (ctx, ctx->offset);
+            }
+        }
+      else
+        {
+          eassert (dump_object_emacs_ptr (arg) == NULL);
+          target_offset = dump_recall_object (ctx, arg);
+          if (target_offset <= 0)
+            error ("fixup object not dumped");
+          if (type == DUMP_FIXUP_LISP_OBJECT)
+            dump_reloc_dump_to_dump_lv (ctx, ctx->offset, XTYPE (arg));
+          else
+            dump_reloc_dump_to_dump_raw_ptr (ctx, ctx->offset);
+        }
+      break;
+    case DUMP_FIXUP_PTR_DUMP_RAW:
+      /* Dump wants a raw pointer to something that's not a lisp
+         object.  It knows the exact location it wants, so just
+         believe it.  */
+      target_offset = ptrdiff_t_from_lisp (arg);
+      dump_reloc_dump_to_dump_raw_ptr (ctx, ctx->offset);
+      break;
+    default:
+      emacs_abort ();
+    }
+  if (do_write)
+    dump_write (ctx, &target_offset, sizeof (target_offset));
+}
+
+static ptrdiff_t
+dump_check_dump_off (struct dump_context *ctx, ptrdiff_t dump_offset)
+{
+  eassert (dump_offset > 0);
+  if (ctx)
+    eassert (dump_offset < ctx->end_heap);
+  return dump_offset;
+}
+
+static void
+dump_check_emacs_off (ptrdiff_t emacs_off)
+{
+  eassert (labs (emacs_off) <= 30*1024*1024);
+}
+
+static void
+dump_emit_dump_reloc (struct dump_context *ctx, Lisp_Object lreloc)
+{
+  struct dump_reloc reloc;
+  dump_object_start (ctx, 1, &reloc, sizeof (reloc));
+  reloc.type = XFASTINT (dump_pop (&lreloc));
+  eassert (reloc.type <= RELOC_DUMP_TO_EMACS_LV + Lisp_Float);
+  reloc.offset = dump_off_from_lisp (dump_pop (&lreloc));
+  dump_check_dump_off (ctx, reloc.offset);
+  eassert (reloc.offset % 4 == 0); // Alignment
+  eassert (NILP (lreloc));
+  dump_object_finish (ctx, &reloc, sizeof (reloc));
+}
+
+static struct emacs_reloc
+decode_emacs_reloc (struct dump_context *ctx, Lisp_Object lreloc)
+{
+  struct emacs_reloc reloc;
+  memset (&reloc, 0, sizeof (reloc));
+  int type = XFASTINT (dump_pop (&lreloc));
+  reloc.emacs_offset = dump_off_from_lisp (dump_pop (&lreloc));
+  dump_check_emacs_off (reloc.emacs_offset);
+  switch (type)
+    {
+    case RELOC_EMACS_COPY_FROM_DUMP:
+      {
+        reloc.type = type;
+        eassert (reloc.type == type);
+        reloc.u.dump_offset = dump_off_from_lisp (dump_pop (&lreloc));
+        dump_check_dump_off (ctx, reloc.u.dump_offset);
+        dump_off_t length = dump_off_from_lisp (dump_pop (&lreloc));
+        reloc.length = length;
+        if (reloc.length != length)
+          error ("relocation copy length too large");
+      }
+      break;
+    case RELOC_EMACS_IMMEDIATE:
+      {
+        reloc.type = type;
+        eassert (reloc.type == type);
+        intmax_t value = intmax_t_from_lisp (dump_pop (&lreloc));
+        ptrdiff_t size = ptrdiff_t_from_lisp (dump_pop (&lreloc));
+        reloc.u.immediate = value;
+        reloc.length = size;
+        eassert (reloc.length == size);
+      }
+      break;
+    default:
+      {
+        eassert (RELOC_EMACS_DUMP_LV <= type);
+        eassert (type <= RELOC_EMACS_DUMP_LV + Lisp_Float);
+        reloc.type = RELOC_EMACS_DUMP_LV;
+        eassert (reloc.type == RELOC_EMACS_DUMP_LV);
+        reloc.length = type - RELOC_EMACS_DUMP_LV;
+        eassert (reloc.length == type - RELOC_EMACS_DUMP_LV);
+        Lisp_Object target_value = dump_pop (&lreloc);
+        /* If the object is self-representing,
+           dump_emacs_reloc_to_dump_lv didn't do its job.
+           dump_emacs_reloc_to_dump_lv should have added a
+           RELOC_EMACS_IMMEDIATE relocation instead.  */
+        eassert (!dump_object_self_representing_p (target_value));
+        reloc.u.dump_offset = dump_recall_object (ctx, target_value);
+        if (reloc.u.dump_offset <= 0)
+          {
+            Lisp_Object repr = Fprin1_to_string (target_value, Qnil);
+            error ("relocation target was not dumped: %s", SDATA (repr));
+          }
+        dump_check_dump_off (ctx, reloc.u.dump_offset);
+      }
+      break;
+    case RELOC_EMACS_EMACS_PTR_RAW:
+      reloc.type = type;
+      eassert (reloc.type == type);
+      reloc.u.emacs_offset2 = dump_off_from_lisp (dump_pop (&lreloc));
+      dump_check_emacs_off (reloc.u.emacs_offset2);
+      break;
+    case RELOC_EMACS_DUMP_PTR_RAW:
+      reloc.type = type;
+      eassert (reloc.type == type);
+      reloc.u.dump_offset = dump_off_from_lisp (dump_pop (&lreloc));
+      dump_check_dump_off (ctx, reloc.u.dump_offset);
+      break;
+    }
+
+  eassert (NILP (lreloc));
+  return reloc;
+}
+
+static void
+dump_emit_emacs_reloc (struct dump_context *ctx, Lisp_Object lreloc)
+{
+  struct emacs_reloc reloc;
+  dump_object_start (ctx, 1, &reloc, sizeof (reloc));
+  reloc = decode_emacs_reloc (ctx, lreloc);
+  dump_object_finish (ctx, &reloc, sizeof (reloc));
+}
+
+static Lisp_Object
+dump_merge_emacs_relocs (Lisp_Object lreloc_a, Lisp_Object lreloc_b)
+{
+  /* Combine copy relocations together if they're copying from
+     adjacent chunks to adjacent chunks.  */
+
+  if (XFASTINT (XCAR (lreloc_a)) != RELOC_EMACS_COPY_FROM_DUMP ||
+      XFASTINT (XCAR (lreloc_b)) != RELOC_EMACS_COPY_FROM_DUMP)
+    return Qnil;
+
+  struct emacs_reloc reloc_a = decode_emacs_reloc (NULL, lreloc_a);
+  struct emacs_reloc reloc_b = decode_emacs_reloc (NULL, lreloc_b);
+
+  eassert (reloc_a.type == RELOC_EMACS_COPY_FROM_DUMP);
+  eassert (reloc_b.type == RELOC_EMACS_COPY_FROM_DUMP);
+
+  if (reloc_a.emacs_offset + reloc_a.length != reloc_b.emacs_offset)
+    return Qnil;
+
+  if (reloc_a.u.dump_offset + reloc_a.length != reloc_b.u.dump_offset)
+    return Qnil;
+
+  ptrdiff_t new_length = reloc_a.length + reloc_b.length;
+  reloc_a.length = new_length;
+  if (reloc_a.length != new_length)
+    return Qnil; /* Overflow */
+
+  return list4 (make_number (RELOC_EMACS_COPY_FROM_DUMP),
+                INTEGER_TO_CONS (reloc_a.emacs_offset),
+                INTEGER_TO_CONS (reloc_a.u.dump_offset),
+                INTEGER_TO_CONS (reloc_a.length));
+}
+
+static void
+drain_reloc_list (struct dump_context *ctx,
+                  void (*handler)(struct dump_context *, Lisp_Object),
+                  Lisp_Object (*merger)(Lisp_Object a, Lisp_Object b),
+                  Lisp_Object *reloc_list,
+                  struct dump_table_locator *out_locator)
+{
+  Lisp_Object relocs = Fsort (Fnreverse (*reloc_list),
+                              Qdump_emacs_portable__sort_predicate);
+  *reloc_list = Qnil;
+  dump_align_output (ctx, sizeof (dump_off_t));
+  struct dump_table_locator locator;
+  memset (&locator, 0, sizeof (locator));
+  locator.offset = ctx->offset;
+  for (; !NILP (relocs); locator.nr_entries += 1)
+    {
+      Lisp_Object reloc = dump_pop (&relocs);
+      Lisp_Object merged;
+      while (merger != NULL &&
+             !NILP (relocs) &&
+             ((merged = merger (reloc, XCAR (relocs))), !NILP (merged)))
+        {
+          reloc = merged;
+          relocs = XCDR (relocs);
+        }
+      handler (ctx, reloc);
+    }
+  *out_locator = locator;
+}
+
+static void
+dump_do_fixups (struct dump_context *ctx)
+{
+  ptrdiff_t saved_offset = ctx->offset;
+  Lisp_Object fixups = Fsort (Fnreverse (ctx->fixups),
+                              Qdump_emacs_portable__sort_predicate);
+  ctx->fixups = Qnil;
+  while (!NILP (fixups))
+    dump_do_fixup (ctx, dump_pop (&fixups));
+  dump_seek (ctx, saved_offset);
+}
+
+DEFUN ("dump-emacs-portable",
+       Fdump_emacs_portable, Sdump_emacs_portable,
+       1, 2, 0,
+       doc: /* Dump current state of Emacs into dump file FILENAME.
+If TRACK-REFERRERS is non-nil, keep additional debugging information
+that can help track down the provenance of unsupported object
+types.  */)
+     (Lisp_Object filename, Lisp_Object track_referrers)
+{
+  eassert (initialized);
+
+  if (will_dump_with_unexec)
+    error ("This Emacs instance was started under the assumption "
+           "that it would be dumped with unexec, not the portable "
+           "dumper.  Dumping with the portable dumper may produce "
+           "unexpected results.");
+
+  ptrdiff_t count = SPECPDL_INDEX ();
+
+  /* Bind `command-line-processed' to nil before dumping,
+     so that the dumped Emacs will process its command line
+     and set up to work with X windows if appropriate.  */
+  Lisp_Object symbol = intern ("command-line-processed");
+  specbind (symbol, Qnil);
+
+  CHECK_STRING (filename);
+  filename = Fexpand_file_name (filename, Qnil);
+  filename = ENCODE_FILE (filename);
+
+  struct dump_context ctx_buf;
+  struct dump_context *ctx = &ctx_buf;
+  memset (ctx, 0, sizeof (*ctx));
+  ctx->fd = -1;
+
+  ctx->objects_dumped = make_eq_hash_table ();
+  dump_tailq_init (&ctx->dump_queue);
+  ctx->fixups = Qnil;
+  ctx->copied_queue = Qnil;
+  ctx->cold_queue = Qnil;
+  ctx->dump_relocs = Qnil;
+  ctx->object_starts = Qnil;
+  ctx->emacs_relocs = Qnil;
+
+  ctx->current_referrer = Qnil;
+  if (!NILP (track_referrers))
+    ctx->referrers = make_eq_hash_table ();
+
+  ctx->dump_filename = filename;
+
+  record_unwind_protect_ptr (dump_unwind_cleanup, ctx);
+  block_input ();
+
+  ctx->old_purify_flag = Vpurify_flag;
+  Vpurify_flag = Qnil;
+
+  /* Make sure various weird things are less likely to happen.  */
+  ctx->old_post_gc_hook = Vpost_gc_hook;
+  Vpost_gc_hook = Qnil;
+
+  ctx->fd = emacs_open (SSDATA (filename), O_RDWR | O_TRUNC | O_CREAT, 0666);
+  if (ctx->fd < 0)
+    report_file_error ("Opening dump output", filename);
+  verify (sizeof (ctx->header.magic) == sizeof (dump_magic));
+  memcpy (&ctx->header.magic, dump_magic, sizeof (dump_magic));
+  ctx->header.magic[0] = '!'; /* Note that dump is incomplete.  */
+
+  verify (sizeof (fingerprint) == sizeof (ctx->header.fingerprint));
+  memcpy (ctx->header.fingerprint, fingerprint, sizeof (fingerprint));
+
+  dump_write (ctx, &ctx->header, sizeof (ctx->header));
+
+  /* Start the dump process by processing the static roots and
+     queuing up the objects to which they refer.   */
+  dump_roots (ctx);
+
+  dump_charset_table (ctx);
+  dump_finalizer_list_head_ptr (ctx, &finalizers.prev);
+  dump_finalizer_list_head_ptr (ctx, &finalizers.next);
+  dump_finalizer_list_head_ptr (ctx, &doomed_finalizers.prev);
+  dump_finalizer_list_head_ptr (ctx, &doomed_finalizers.next);
+  dump_user_remembered_data_hot (ctx);
+
+  /* We've already remembered all of the GC roots themselves, but we
+     have to manually save the list of GC roots.  */
+  dump_metadata_for_pdumper (ctx);
+  for (int i = 0; i < staticidx; ++i)
+    dump_emacs_reloc_to_emacs_ptr_raw (ctx, &staticvec[i], staticvec[i]);
+  dump_emacs_reloc_immediate_int (ctx, &staticidx, staticidx);
+
+  /* Dump until while we keep finding objects to dump.  */
+  while (!dump_tailq_empty_p (&ctx->dump_queue))
+    dump_object (ctx, dump_tailq_pop (&ctx->dump_queue));
+
+  eassert (dump_tailq_empty_p (&ctx->dump_queue));
+  ctx->header.hot_discardable_start = ctx->offset;
+
+  dump_copied_objects (ctx);
+  eassert (dump_tailq_empty_p (&ctx->dump_queue));
+  eassert (NILP (ctx->copied_queue));
+
+  dump_align_output (ctx, getpagesize ());
+  ctx->header.hot_end = ctx->offset;
+  dump_cold_data (ctx);
+   /* dump_user_remembered_data_cold needs to be after dump_cold_data
+      in case dump_cold_data dumps a lisp object to which C code
+      points.  dump_user_remembered_data_cold assumes that all lisp
+      objects have been dumped.  */
+  dump_user_remembered_data_cold (ctx);
+  ctx->end_heap = ctx->offset;
+  dump_do_fixups (ctx);
+  drain_reloc_list (
+    ctx, dump_emit_dump_reloc, NULL,
+    &ctx->dump_relocs,
+    &ctx->header.dump_relocs);
+  drain_reloc_list (
+    ctx, dump_emit_dump_reloc, NULL,
+    &ctx->object_starts,
+    &ctx->header.object_starts);
+  drain_reloc_list (
+    ctx, dump_emit_emacs_reloc, dump_merge_emacs_relocs,
+    &ctx->emacs_relocs,
+    &ctx->header.emacs_relocs);
+
+  eassert (dump_tailq_empty_p (&ctx->dump_queue));
+  eassert (NILP (ctx->fixups));
+  eassert (NILP (ctx->dump_relocs));
+  eassert (NILP (ctx->emacs_relocs));
+
+  ctx->header.magic[0] = dump_magic[0]; /* Note dump is complete.  */
+  dump_seek (ctx, 0);
+  dump_write (ctx, &ctx->header, sizeof (ctx->header));
+
+  return unbind_to (count, Qnil);
+
+  // XXX: consider getting rid of hooks and just rely
+  // on explicit calls?
+
+  // XXX: nullify frame_and_buffer_state
+
+  // XXX: inline stuff in pdumper.h
+
+  // XXX: preferred base address
+
+  // XXX: make offset math non-fwrapv-safe
+
+  // XXX: output symbol chains consecutively
+}
+
+DEFUN ("dump-emacs-portable--sort-predicate",
+       Fdump_emacs_portable__sort_predicate,
+       Sdump_emacs_portable__sort_predicate,
+       2, 2, 0,
+       doc: /* Internal relocation sorting function.  */)
+     (Lisp_Object a, Lisp_Object b)
+{
+  ptrdiff_t a_offset = ptrdiff_t_from_lisp (XCAR (XCDR (a)));
+  ptrdiff_t b_offset = ptrdiff_t_from_lisp (XCAR (XCDR (b)));
+  return a_offset < b_offset ? Qt : Qnil;
+}
+
+DEFUN ("dump-emacs-portable--sort-predicate-copied",
+       Fdump_emacs_portable__sort_predicate_copied,
+       Sdump_emacs_portable__sort_predicate_copied,
+       2, 2, 0,
+       doc: /* Internal relocation sorting function.  */)
+     (Lisp_Object a, Lisp_Object b)
+{
+  eassert (dump_object_emacs_ptr (a));
+  eassert (dump_object_emacs_ptr (b));
+  return dump_object_emacs_ptr (a) < dump_object_emacs_ptr (b) ? Qt : Qnil;
+}
+
+void
+pdumper_do_now_and_after_load (pdumper_hook hook)
+{
+  if (nr_dump_hooks == ARRAYELTS (dump_hooks))
+    fatal ("out of dump hooks: make dump_hooks[] bigger");
+  dump_hooks[nr_dump_hooks++] = hook;
+  hook ();
+}
+
+static void
+pdumper_remember_user_data_1 (void *mem, int nbytes)
+{
+  if (nr_remembered_data == ARRAYELTS (remembered_data))
+    fatal ("out of remembered data slots: make remembered_data[] bigger");
+  remembered_data[nr_remembered_data].mem = mem;
+  remembered_data[nr_remembered_data].sz = nbytes;
+  nr_remembered_data += 1;
+}
+
+void
+pdumper_remember_scalar (void *mem, ptrdiff_t nbytes)
+{
+  eassert (0 <= nbytes && nbytes <= INT_MAX);
+  if (nbytes > 0)
+    pdumper_remember_user_data_1 (mem, nbytes);
+}
+
+void
+pdumper_remember_lv_raw_ptr (void* ptr, enum Lisp_Type type)
+{
+  pdumper_remember_user_data_1 (ptr, -type);
+}
+
+\f
+
+struct loaded_dump
+{
+  char *start;
+  char *end;
+  struct dump_header header;
+  unsigned *mark_bits;
+};
+
+struct loaded_dump loaded_dump;
+
+/* Search for a relocation given a relocation target.
+
+   DUMP is the dump metadata structure.  TABLE is the relocation table
+   to search.  KEY is the dump offset to find.  Return the greatest
+   relocation RELOC such that RELOC.offset <= KEY or NULL if no such
+   relocation exists.  */
+static const struct dump_reloc *
+dump_find_relocation (struct loaded_dump *dump,
+                      const struct dump_table_locator *table,
+                      dump_off_t key)
+{
+  const struct dump_reloc *left = (void *)(dump->start + table->offset);
+  const struct dump_reloc *right = left + table->nr_entries;
+  const struct dump_reloc *found = NULL;
+
+  while (left < right)
+    {
+      const struct dump_reloc *mid = left + (right - left) / 2;
+      if (mid->offset <= key)
+        {
+          found = mid;
+          left = mid + 1;
+          if (left >= right || left->offset > key)
+            break;
+        }
+      else
+        right = mid;
+   }
+
+  return found;
+}
+
+static bool
+dump_loaded_p (void)
+{
+  return loaded_dump.start != NULL;
+}
+
+/* Return whether the OBJ points somewhere into the loaded dump image.
+   Works even when we have no dump loaded --- in this case, it just
+   returns false.  */
+bool
+pdumper_object_p (const void *obj)
+{
+  const char *p = obj;
+  return loaded_dump.start <= p && p < loaded_dump.end;
+}
+
+/* Return whether OBJ points exactly to the start of some object in
+   the loaded dump image.  It is a programming error to call this
+   routine for an OBJ for which pdumper_object_p would return
+   false.  */
+bool
+pdumper_object_p_precise (const void *obj)
+{
+  return pdumper_find_object_type (obj) != PDUMPER_NO_OBJECT;
+}
+
+/* Return the type of the dumped object that starts at OBJ.  It is a
+   programming error to call this routine for an OBJ for which
+   pdumper_object_p would return false.  */
+enum Lisp_Type
+pdumper_find_object_type (const void *obj)
+{
+  eassert (pdumper_object_p (obj));
+  ptrdiff_t offset = (char *) obj - (char *)loaded_dump.start;
+  if (offset % GCALIGNMENT != 0)
+    return PDUMPER_NO_OBJECT;
+  const struct dump_reloc *reloc =
+    dump_find_relocation (&loaded_dump,
+                          &loaded_dump.header.object_starts,
+                          offset);
+  return (reloc != NULL && reloc->offset == offset)
+    ? reloc->type
+    : PDUMPER_NO_OBJECT;
+}
+
+static ptrdiff_t
+dump_mark_bits_nbytes (ptrdiff_t max_offset)
+{
+  ptrdiff_t bits_needed = (max_offset + GCALIGNMENT - 1) / GCALIGNMENT;
+  ptrdiff_t bytes_needed = (bits_needed + CHAR_BIT - 1) / CHAR_BIT;
+  return ROUNDUP (bytes_needed, sizeof (unsigned));
+}
+
+bool
+pdumper_marked_p (const void *obj)
+{
+  eassert (pdumper_object_p (obj));
+  ptrdiff_t offset = (char *) obj - loaded_dump.start;
+  eassert (offset % GCALIGNMENT == 0);
+  eassert (offset < loaded_dump.header.hot_discardable_start);
+  ptrdiff_t bitno = offset / GCALIGNMENT;
+  ptrdiff_t slotno = bitno / (CHAR_BIT * sizeof (unsigned));
+  unsigned *slot = &loaded_dump.mark_bits[slotno];
+  return *slot & (1U << (bitno % (CHAR_BIT * sizeof (unsigned))));
+}
+
+void
+pdumper_set_marked (const void *obj)
+{
+  eassert (pdumper_object_p (obj));
+  ptrdiff_t offset = (char *) obj - loaded_dump.start;
+  eassert (offset % GCALIGNMENT == 0);
+  eassert (offset < loaded_dump.header.hot_discardable_start);
+  ptrdiff_t bitno = offset / GCALIGNMENT;
+  ptrdiff_t slotno = bitno / (CHAR_BIT * sizeof (unsigned));
+  unsigned *slot = &loaded_dump.mark_bits[slotno];
+  *slot |= (1U << (bitno % (CHAR_BIT * sizeof (unsigned))));
+}
+
+void
+pdumper_clear_marks (void)
+{
+  memset (loaded_dump.mark_bits, 0,
+          dump_mark_bits_nbytes (loaded_dump.header.hot_discardable_start));
+}
+
+static ssize_t
+pdumper_read (int fd, void *buf, size_t bytes_to_read)
+{
+  eassert (bytes_to_read <= SSIZE_MAX);
+  size_t bytes_read = 0;
+  while (bytes_read < bytes_to_read)
+    {
+      ssize_t chunk =
+        read (fd, (char*) buf + bytes_read, bytes_to_read - bytes_read);
+      if (chunk < 0)
+        return chunk;
+      if (chunk == 0)
+        break;
+      bytes_read += chunk;
+    }
+
+  return bytes_read;
+}
+
+static void *
+dump_ptr (struct loaded_dump *dump, ptrdiff_t offset)
+{
+  eassert (dump->start + offset < dump->end);
+  return dump->start + offset;
+}
+
+static void *
+emacs_ptr (ptrdiff_t offset)
+{
+  // TODO: assert somehow that offset is actually inside Emacs
+  return (void *) (emacs_basis () + offset);
+}
+
+static void
+dump_do_dump_relocation (struct loaded_dump *dump,
+                         struct dump_reloc reloc)
+{
+  ptrdiff_t *dump_ptr_ptr = dump_ptr (dump, reloc.offset);
+  ptrdiff_t dump_ptr = *dump_ptr_ptr;
+  ptrdiff_t dump_base = (ptrdiff_t) dump->start;
+
+  /* For -O0 debugging: optimizer realizes this variable is dead and
+     optimizes it away.  */
+  ptrdiff_t orig_dump_ptr = dump_ptr;
+  (void) orig_dump_ptr;
+
+  switch (reloc.type)
+    {
+    case RELOC_DUMP_TO_EMACS_RAW_PTR:
+      dump_ptr = dump_ptr + emacs_basis ();
+      *dump_ptr_ptr = dump_ptr;
+      break;
+    case RELOC_DUMP_TO_DUMP_RAW_PTR:
+      dump_ptr = dump_ptr + dump_base;
+      *dump_ptr_ptr = dump_ptr;
+      break;
+    default:
+      {
+        enum Lisp_Type lisp_type;
+        if (RELOC_DUMP_TO_DUMP_LV <= reloc.type &&
+            reloc.type < RELOC_DUMP_TO_EMACS_LV)
+          {
+            lisp_type = reloc.type - RELOC_DUMP_TO_DUMP_LV;
+            dump_ptr = dump_ptr + dump_base;
+          }
+        else
+          {
+            eassert (RELOC_DUMP_TO_EMACS_LV <= reloc.type);
+            eassert (reloc.type < RELOC_DUMP_TO_EMACS_LV + 8);
+            lisp_type = reloc.type - RELOC_DUMP_TO_EMACS_LV;
+            dump_ptr = dump_ptr + emacs_basis ();
+          }
+
+        Lisp_Object lv;
+        if (lisp_type == Lisp_Symbol)
+          lv = make_lisp_symbol ((void *) dump_ptr);
+        else
+          lv = make_lisp_ptr ((void *) dump_ptr, lisp_type);
+
+        * (Lisp_Object *) dump_ptr_ptr = lv;
+        break;
+      }
+    }
+
+  // XXX: raw_ptr or ptr_raw. Pick one.
+}
+
+static void
+dump_do_dump_relocations (struct loaded_dump *dump)
+{
+  struct dump_header *header = &dump->header;
+  struct dump_reloc *r = dump_ptr (dump, header->dump_relocs.offset);
+  dump_off_t nr_entries = header->dump_relocs.nr_entries;
+  for (dump_off_t i = 0; i < nr_entries; ++i)
+    dump_do_dump_relocation (dump, r[i]);
+}
+
+static void
+dump_do_emacs_relocation (struct loaded_dump *dump,
+                          struct emacs_reloc reloc)
+{
+  ptrdiff_t dump_base = (ptrdiff_t) dump->start;
+  ptrdiff_t pval;
+  Lisp_Object lv;
+
+  switch (reloc.type)
+    {
+    case RELOC_EMACS_COPY_FROM_DUMP:
+      eassume (reloc.length > 0);
+      memcpy (emacs_ptr (reloc.emacs_offset),
+              dump_ptr (dump, reloc.u.dump_offset),
+              reloc.length);
+      break;
+    case RELOC_EMACS_IMMEDIATE:
+      eassume (reloc.length > 0);
+      eassume (reloc.length <= sizeof (reloc.u.immediate));
+      memcpy (emacs_ptr (reloc.emacs_offset),
+              &reloc.u.immediate,
+              reloc.length);
+      break;
+    case RELOC_EMACS_DUMP_PTR_RAW:
+      pval = reloc.u.dump_offset + dump_base;
+      memcpy (emacs_ptr (reloc.emacs_offset), &pval, sizeof (pval));
+      break;
+    case RELOC_EMACS_EMACS_PTR_RAW:
+      pval = reloc.u.emacs_offset2 + emacs_basis ();
+      memcpy (emacs_ptr (reloc.emacs_offset), &pval, sizeof (pval));
+      break;
+    case RELOC_EMACS_DUMP_LV:
+      eassume (reloc.length <= Lisp_Float);
+      if (reloc.length == Lisp_Symbol)
+        lv = make_lisp_symbol (dump_ptr (dump, reloc.u.dump_offset));
+      else
+        lv = make_lisp_ptr (dump_ptr (dump, reloc.u.dump_offset),
+                            reloc.length);
+      memcpy (emacs_ptr (reloc.emacs_offset), &lv, sizeof (lv));
+      break;
+    default:
+      fatal ("unrecognied relocation type %d", (int) reloc.type);
+    }
+}
+
+static void
+dump_do_emacs_relocations (struct loaded_dump *dump)
+{
+  struct dump_header *header = &dump->header;
+  struct emacs_reloc *r = dump_ptr (dump, header->emacs_relocs.offset);
+  dump_off_t nr_entries = header->emacs_relocs.nr_entries;
+  for (dump_off_t i = 0; i < nr_entries; ++i)
+    dump_do_emacs_relocation (dump, r[i]);
+}
+
+/* Load a dump from DUMP_FILENAME.  We run very early in
+   initialization, so we can't use lisp, unwinding, xmalloc, and so
+   on.  */
+enum pdumper_load_result
+pdumper_load (const char *dump_filename)
+{
+  int fd = -1;
+  enum pdumper_load_result err = PDUMPER_LOAD_ERROR;
+  struct loaded_dump ndump;
+  struct stat stat;
+  struct dump_header *header = &ndump.header;
+  ptrdiff_t mark_nbytes;
+
+  memset (&ndump, 0, sizeof (ndump));
+  eassert (!initialized);
+  eassert (!dump_loaded_p ());
+
+  err = PDUMPER_LOAD_FILE_NOT_FOUND;
+  fd = emacs_open (dump_filename, O_RDONLY, 0);
+  if (fd < 0)
+    goto out;
+
+  if (fstat (fd, &stat) < 0)
+    goto out;
+
+  err = PDUMPER_LOAD_BAD_FILE_TYPE;
+  if (stat.st_size < sizeof (*header))
+    goto out;
+
+  err = PDUMPER_LOAD_OOM;
+  ndump.start = malloc (stat.st_size);
+  if (ndump.start == NULL)
+    goto out;
+  eassert ((ptrdiff_t) ndump.start % GCALIGNMENT == 0);
+  ndump.end = ndump.start + stat.st_size;
+
+  err = PDUMPER_LOAD_BAD_FILE_TYPE;
+  if (pdumper_read (fd, ndump.start, stat.st_size) < stat.st_size)
+    goto out;
+
+  memcpy (header, ndump.start, sizeof (*header));
+  if (memcmp (header->magic, dump_magic, sizeof (dump_magic) != 0))
+    goto out;
+
+  err = PDUMPER_LOAD_VERSION_MISMATCH;
+  verify (sizeof (header->fingerprint) == sizeof (fingerprint));
+  if (memcmp (header->fingerprint, fingerprint, sizeof (fingerprint)))
+    goto out;
+
+  err = PDUMPER_LOAD_ERROR;
+  mark_nbytes = dump_mark_bits_nbytes (ndump.header.hot_discardable_start);
+  ndump.mark_bits = calloc (1, mark_nbytes);
+  if (ndump.mark_bits == NULL)
+    goto out;
+
+  /* Point of no return.  */
+
+  gflags.dumped_with_pdumper_ = true;
+  loaded_dump = ndump;
+  memset (&ndump, 0, sizeof (ndump));
+  err = PDUMPER_LOAD_SUCCESS;
+
+  dump_do_dump_relocations (&loaded_dump);
+  dump_do_emacs_relocations (&loaded_dump);
+
+  /* Run the functions Emacs registered for doing post-dump-load
+     initialization.  */
+  for (int i = 0; i < nr_dump_hooks; ++i)
+    dump_hooks[i] ();
+  gflags.initialized_ = true;
+
+ out:
+  free (ndump.mark_bits);
+  free (ndump.start);
+  if (0 <= fd)
+    emacs_close (fd);
+  return err;
+}
+
+\f
+
+void
+syms_of_pdumper (void)
+{
+  defsubr (&Sdump_emacs_portable);
+  defsubr (&Sdump_emacs_portable__sort_predicate);
+  defsubr (&Sdump_emacs_portable__sort_predicate_copied);
+  DEFSYM (Qdump_emacs_portable__sort_predicate,
+          "dump-emacs-portable--sort-predicate");
+  DEFSYM (Qdump_emacs_portable__sort_predicate_copied,
+          "dump-emacs-portable--sort-predicate-copied");
+}
diff --git a/src/pdumper.h b/src/pdumper.h
new file mode 100644
index 0000000..d6922b7
--- /dev/null
+++ b/src/pdumper.h
@@ -0,0 +1,115 @@
+/* Header file for the portable dumper.
+
+Copyright (C) 2016 Free Software Foundation,
+Inc.
+
+This file is part of GNU Emacs.
+
+GNU Emacs is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or (at
+your option) any later version.
+
+GNU Emacs is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
+
+#ifndef EMACS_PDUMPER_H
+#define EMACS_PDUMPER_H
+
+#include "lisp.h"
+
+/* The portable dumper automatically preserves the Lisp heap and any C
+   variables to which the Lisp heap points.  It doesn't know anything
+   about other C variables.  The functions below allow code from other
+   parts of Emacs to tell the portable dumper about other bits of
+   information to preserve in dumped images.
+
+   These memory-records are themselves preserved in the dump, so call
+   the functions below only on the !initialized init path, just
+   like staticpro.
+
+   There are no special functions to preserve a global Lisp_Object.
+   You should just staticpro these.  */
+
+/* Indicate in source code that we're deliberately relying on pdumper
+   not preserving the given value.  Compiles to nothing --- for humans
+   only.  */
+#define PDUMPER_IGNORE(thing) ((void) &(thing))
+
+/* Remember the value of THING in dumped images.  THING must not
+   contain any pointers or Lisp_Object variables: these values are not
+   valid across dump and load.  */
+#define PDUMPER_REMEMBER_SCALAR(thing)                  \
+  pdumper_remember_scalar (&(thing), sizeof (thing))
+void pdumper_remember_scalar (void *data, ptrdiff_t nbytes);
+
+/* Remember the pointer at *PTR.  *PTR must be null or point to a Lisp
+   object.  TYPE is the rough type of Lisp object to which *PTR
+   points.  */
+void pdumper_remember_lv_raw_ptr (void* ptr, enum Lisp_Type type);
+
+/* Remember the pointer at *PTR.  *PTR must be null or point to
+   something in the Emacs process image (e.g., a function).  */
+void pdumper_remember_emacs_ptr (void *ptr);
+
+typedef void (*pdumper_hook)(void);
+void pdumper_do_now_and_after_load (pdumper_hook);
+
+/* Macros useful in pdumper callback functions.  Assign a value if
+   we're loading a dump and the value needs to be reset to its
+   original value, and if we're initializing for the first time,
+   assert that the value has the expected original value.  */
+
+#define PDUMPER_RESET(variable, value)         \
+  do {                                         \
+    if (dumped_with_pdumper)                   \
+      (variable) = (value);                    \
+    else                                       \
+      eassert ((variable) == (value));         \
+  } while (0)
+
+#define PDUMPER_RESET_LV(variable, value)         \
+  do {                                            \
+    if (dumped_with_pdumper)                      \
+      (variable) = (value);                       \
+    else                                          \
+      eassert (EQ ((variable), (value)));         \
+  } while (0)
+
+/* Actually load a dump.  */
+
+enum pdumper_load_result
+  {
+    PDUMPER_LOAD_SUCCESS,
+    PDUMPER_NOT_LOADED /* Not returned: useful for callers */,
+    PDUMPER_LOAD_FILE_NOT_FOUND,
+    PDUMPER_LOAD_BAD_FILE_TYPE,
+    PDUMPER_LOAD_OOM,
+    PDUMPER_LOAD_VERSION_MISMATCH,
+    PDUMPER_LOAD_ERROR,
+  };
+
+enum pdumper_load_result pdumper_load (const char *dump_filename);
+
+_GL_ATTRIBUTE_CONST
+bool pdumper_object_p (const void *obj);
+#define PDUMPER_NO_OBJECT ((enum Lisp_Type) -1)
+_GL_ATTRIBUTE_CONST
+enum Lisp_Type pdumper_find_object_type (const void *obj);
+_GL_ATTRIBUTE_CONST
+bool pdumper_object_p_precise (const void *obj);
+
+bool pdumper_marked_p (const void *obj);
+void pdumper_set_marked (const void *obj);
+void pdumper_clear_marks (void);
+
+void syms_of_pdumper (void);
+
+
+
+#endif
diff --git a/src/process.c b/src/process.c
index 49340b1..6bb2c02 100644
--- a/src/process.c
+++ b/src/process.c
@@ -7797,9 +7797,7 @@ init_process_emacs (int sockfd)
 
   inhibit_sentinels = 0;
 
-#ifndef CANNOT_DUMP
-  if (! noninteractive || initialized)
-#endif
+  if (!will_dump_with_unexec)
     {
 #if defined HAVE_GLIB && !defined WINDOWSNT
       /* Tickle glib's child-handling code.  Ask glib to wait for Emacs itself;
diff --git a/src/profiler.c b/src/profiler.c
index 07e21ae..8036fe6 100644
--- a/src/profiler.c
+++ b/src/profiler.c
@@ -21,6 +21,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "lisp.h"
 #include "syssignal.h"
 #include "systime.h"
+#include "pdumper.h"
 
 /* Return A + B, but return the maximum fixnum if the result would overflow.
    Assume A and B are nonnegative and in fixnum range.  */
@@ -572,6 +573,8 @@ hashfn_profiler (struct hash_table_test *ht, Lisp_Object bt)
     return XHASH (bt);
 }
 
+static void syms_of_profiler_for_pdumper (void);
+
 void
 syms_of_profiler (void)
 {
@@ -610,4 +613,22 @@ to make room for new entries.  */);
   defsubr (&Sprofiler_memory_stop);
   defsubr (&Sprofiler_memory_running_p);
   defsubr (&Sprofiler_memory_log);
+
+  pdumper_do_now_and_after_load (syms_of_profiler_for_pdumper);
+}
+
+static void
+syms_of_profiler_for_pdumper (void)
+{
+  if (dumped_with_pdumper)
+    {
+      cpu_log = Qnil;
+      memory_log = Qnil;
+    }
+  else
+    {
+      eassert (NILP (cpu_log));
+      eassert (NILP (memory_log));
+    }
+
 }
diff --git a/src/search.c b/src/search.c
index e597c33..68b788b 100644
--- a/src/search.c
+++ b/src/search.c
@@ -29,6 +29,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "region-cache.h"
 #include "blockinput.h"
 #include "intervals.h"
+#include "pdumper.h"
 
 #include <sys/types.h>
 #include "regex.h"
@@ -3390,25 +3391,18 @@ the buffer.  If the buffer doesn't have a cache, the value is nil.  */)
   return val;
 }
 \f
+
+static void syms_of_search_for_pdumper (void);
+
 void
 syms_of_search (void)
 {
-  register int i;
-
-  for (i = 0; i < REGEXP_CACHE_SIZE; ++i)
+  for (int i = 0; i < REGEXP_CACHE_SIZE; ++i)
     {
-      searchbufs[i].buf.allocated = 100;
-      searchbufs[i].buf.buffer = xmalloc (100);
-      searchbufs[i].buf.fastmap = searchbufs[i].fastmap;
-      searchbufs[i].regexp = Qnil;
-      searchbufs[i].whitespace_regexp = Qnil;
-      searchbufs[i].syntax_table = Qnil;
       staticpro (&searchbufs[i].regexp);
       staticpro (&searchbufs[i].whitespace_regexp);
       staticpro (&searchbufs[i].syntax_table);
-      searchbufs[i].next = (i == REGEXP_CACHE_SIZE-1 ? 0 : &searchbufs[i+1]);
     }
-  searchbuf_head = &searchbufs[0];
 
   /* Error condition used for failing searches.  */
   DEFSYM (Qsearch_failed, "search-failed");
@@ -3466,4 +3460,22 @@ is to bind it with `let' around a small expression.  */);
   defsubr (&Sset_match_data);
   defsubr (&Sregexp_quote);
   defsubr (&Snewline_cache_check);
+
+  pdumper_do_now_and_after_load (syms_of_search_for_pdumper);
+}
+
+static void
+syms_of_search_for_pdumper (void)
+{
+  for (int i = 0; i < REGEXP_CACHE_SIZE; ++i)
+    {
+      searchbufs[i].buf.allocated = 100;
+      searchbufs[i].buf.buffer = xmalloc (100);
+      searchbufs[i].buf.fastmap = searchbufs[i].fastmap;
+      searchbufs[i].regexp = Qnil;
+      searchbufs[i].whitespace_regexp = Qnil;
+      searchbufs[i].syntax_table = Qnil;
+      searchbufs[i].next = (i == REGEXP_CACHE_SIZE-1 ? 0 : &searchbufs[i+1]);
+    }
+  searchbuf_head = &searchbufs[0];
 }
diff --git a/src/sysdep.c b/src/sysdep.c
index 892e976..00f128c 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -1850,7 +1850,7 @@ maybe_fatal_sig (int sig)
 }
 
 void
-init_signals (bool dumping)
+init_signals (void)
 {
   struct sigaction thread_fatal_action;
   struct sigaction action;
@@ -2001,7 +2001,7 @@ init_signals (bool dumping)
   /* Don't alter signal handlers if dumping.  On some machines,
      changing signal handlers sets static data that would make signals
      fail to work right when the dumped Emacs is run.  */
-  if (dumping)
+  if (will_dump)
     return;
 
   sigfillset (&process_fatal_action.sa_mask);
diff --git a/src/syssignal.h b/src/syssignal.h
index 3de83c7..5f9a0da 100644
--- a/src/syssignal.h
+++ b/src/syssignal.h
@@ -22,7 +22,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 
 #include <signal.h>
 
-extern void init_signals (bool);
+extern void init_signals (void);
 extern void block_child_signal (sigset_t *);
 extern void unblock_child_signal (sigset_t const *);
 extern void block_tty_out_signal (sigset_t *);
diff --git a/src/textprop.c b/src/textprop.c
index 7af8c69..fdbb761 100644
--- a/src/textprop.c
+++ b/src/textprop.c
@@ -2340,11 +2340,10 @@ inherits it if NONSTICKINESS is nil.  The `front-sticky' and
   Vtext_property_default_nonsticky
     = list2 (Fcons (Qsyntax_table, Qt), Fcons (Qdisplay, Qt));
 
-  staticpro (&interval_insert_behind_hooks);
-  staticpro (&interval_insert_in_front_hooks);
   interval_insert_behind_hooks = Qnil;
   interval_insert_in_front_hooks = Qnil;
-
+  staticpro (&interval_insert_behind_hooks);
+  staticpro (&interval_insert_in_front_hooks);
 
   /* Common attributes one might give text.  */
 
diff --git a/src/w32fns.c b/src/w32fns.c
index 8c8272b..4c8b651 100644
--- a/src/w32fns.c
+++ b/src/w32fns.c
@@ -56,6 +56,8 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "w32.h"
 #endif
 
+#include "pdumper.h"
+
 #include <basetyps.h>
 #include <unknwn.h>
 #include <commctrl.h>
@@ -9767,6 +9769,7 @@ syms_of_w32fns (void)
   track_mouse_window = NULL;
 
   w32_visible_system_caret_hwnd = NULL;
+  PDUMPER_IGNORE (w32_visible_system_caret_hwnd);
 
   DEFSYM (Qundefined_color, "undefined-color");
   DEFSYM (Qcancel_timer, "cancel-timer");
diff --git a/src/w32font.c b/src/w32font.c
index 4d15cff..69c0763 100644
--- a/src/w32font.c
+++ b/src/w32font.c
@@ -32,6 +32,8 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "w32.h"
 #endif
 
+#include "pdumper.h"
+
 /* Cleartype available on Windows XP, cleartype_natural from XP SP1.
    The latter does not try to fit cleartype smoothed fonts into the
    same bounding box as the non-antialiased version of the font.
@@ -2567,6 +2569,9 @@ struct font_driver w32font_driver =
 
 /* Initialize state that does not change between invocations. This is only
    called when Emacs is dumped.  */
+
+static void syms_of_w32font_for_pdumper (void);
+
 void
 syms_of_w32font (void)
 {
@@ -2746,6 +2751,12 @@ versions of Windows) characters.  */);
 
   defsubr (&Sx_select_font);
 
+  pdumper_do_now_and_after_load (syms_of_w32font_for_pdumper);
+}
+
+static void
+syms_of_w32font_for_pdumper (void)
+{
   w32font_driver.type = Qgdi;
   register_font_driver (&w32font_driver, NULL);
 }
diff --git a/src/w32menu.c b/src/w32menu.c
index 7c66360..bc187e2 100644
--- a/src/w32menu.c
+++ b/src/w32menu.c
@@ -30,6 +30,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "buffer.h"
 #include "coding.h"	/* for ENCODE_SYSTEM */
 #include "menu.h"
+#include "pdumper.h"
 
 /* This may include sys/types.h, and that somehow loses
    if this is not done before the other system files.  */
@@ -1585,6 +1586,7 @@ syms_of_w32menu (void)
   globals_of_w32menu ();
 
   current_popup_menu = NULL;
+  PDUMPER_IGNORE (current_popup_menu);
 
   DEFSYM (Qdebug_on_next_call, "debug-on-next-call");
   DEFSYM (Qunsupported__w32_dialog, "unsupported--w32-dialog");
diff --git a/src/w32uniscribe.c b/src/w32uniscribe.c
index 5f91b50..000adcb 100644
--- a/src/w32uniscribe.c
+++ b/src/w32uniscribe.c
@@ -36,6 +36,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "composite.h"
 #include "font.h"
 #include "w32font.h"
+#include "pdumper.h"
 
 struct uniscribe_font_info
 {
@@ -1168,9 +1169,17 @@ struct font_driver uniscribe_font_driver =
    as it needs to test for the existence of the Uniscribe library.  */
 void syms_of_w32uniscribe (void);
 
+static void syms_of_w32uniscribe_for_pdumper (void);
+
 void
 syms_of_w32uniscribe (void)
 {
+  pdumper_do_now_and_after_load (syms_of_w32uniscribe_for_pdumper);
+}
+
+static void
+syms_of_w32uniscribe_for_pdumper (void)
+{
   HMODULE uniscribe;
 
   /* Don't init uniscribe when dumping */
diff --git a/src/window.c b/src/window.c
index acbefcd..aa87283 100644
--- a/src/window.c
+++ b/src/window.c
@@ -42,6 +42,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #ifdef MSDOS
 #include "msdos.h"
 #endif
+#include "pdumper.h"
 
 static ptrdiff_t count_windows (struct window *);
 static ptrdiff_t get_leaf_windows (struct window *, struct window **,
@@ -7274,10 +7275,56 @@ and scrolling positions.  */)
   return Qnil;
 }
 \f
+
+static void init_window_once_for_pdumper (void);
+
 void
 init_window_once (void)
 {
+  minibuf_window = Qnil;
+  staticpro (&minibuf_window);
+
+  selected_window = Qnil;
+  staticpro (&selected_window);
+
+  Vwindow_list = Qnil;
+  staticpro (&Vwindow_list);
+
+  minibuf_selected_window = Qnil;
+  staticpro (&minibuf_selected_window);
+
+  pdumper_do_now_and_after_load (init_window_once_for_pdumper);
+}
+
+static void init_window_once_for_pdumper (void)
+{
+  window_scroll_pixel_based_preserve_x = -1;
+  window_scroll_pixel_based_preserve_y = -1;
+  window_scroll_preserve_hpos = -1;
+  window_scroll_preserve_vpos = -1;
+  PDUMPER_IGNORE (sequence_number);
+
+  PDUMPER_RESET_LV (minibuf_window, Qnil);
+  PDUMPER_RESET_LV (selected_window, Qnil);
+  PDUMPER_RESET_LV (Vwindow_list, Qnil);
+  PDUMPER_RESET_LV (minibuf_selected_window, Qnil);
+
+  /* Hack: if mode_line_in_non_selected_windows is true (which it may
+     be, if we're restoring from a dump) the guts of
+     make_initial_frame will try to access selected_window, which is
+     invalid at this point, and lose.  For the purposes of creating
+     the initial frame and window, this variable must be false.  */
+  bool old_mode_line_in_non_selected_windows;
+  if (dumped_with_pdumper)
+    {
+      old_mode_line_in_non_selected_windows
+        = mode_line_in_non_selected_windows;
+      mode_line_in_non_selected_windows = false;
+    }
   struct frame *f = make_initial_frame ();
+  if (dumped_with_pdumper)
+    mode_line_in_non_selected_windows =
+      old_mode_line_in_non_selected_windows;
   XSETFRAME (selected_frame, f);
   Vterminal_frame = selected_frame;
   minibuf_window = f->minibuffer_window;
@@ -7324,16 +7371,6 @@ syms_of_window (void)
   DEFSYM (Qfloor, "floor");
   DEFSYM (Qceiling, "ceiling");
 
-  staticpro (&Vwindow_list);
-
-  minibuf_selected_window = Qnil;
-  staticpro (&minibuf_selected_window);
-
-  window_scroll_pixel_based_preserve_x = -1;
-  window_scroll_pixel_based_preserve_y = -1;
-  window_scroll_preserve_hpos = -1;
-  window_scroll_preserve_vpos = -1;
-
   DEFVAR_LISP ("temp-buffer-show-function", Vtemp_buffer_show_function,
 	       doc: /* Non-nil means call as function to display a help buffer.
 The function is called with one argument, the buffer to be displayed.
diff --git a/src/xfont.c b/src/xfont.c
index c2b7317..53d3632 100644
--- a/src/xfont.c
+++ b/src/xfont.c
@@ -31,6 +31,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "character.h"
 #include "charset.h"
 #include "font.h"
+#include "pdumper.h"
 
 \f
 /* X core font driver.  */
@@ -1113,6 +1114,9 @@ xfont_check (struct frame *f, struct font *font)
 }
 
 \f
+
+static void syms_of_xfont_for_pdumper (void);
+
 void
 syms_of_xfont (void)
 {
@@ -1120,6 +1124,12 @@ syms_of_xfont (void)
   xfont_scripts_cache = CALLN (Fmake_hash_table, QCtest, Qequal);
   staticpro (&xfont_scratch_props);
   xfont_scratch_props = Fmake_vector (make_number (8), Qnil);
+  pdumper_do_now_and_after_load (syms_of_xfont_for_pdumper);
+}
+
+static void
+syms_of_xfont_for_pdumper (void)
+{
   xfont_driver.type = Qx;
   register_font_driver (&xfont_driver, NULL);
 }
diff --git a/src/xftfont.c b/src/xftfont.c
index 861ad80..c9efb6d 100644
--- a/src/xftfont.c
+++ b/src/xftfont.c
@@ -32,6 +32,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "composite.h"
 #include "font.h"
 #include "ftfont.h"
+#include "pdumper.h"
 
 /* Xft font driver.  */
 
@@ -751,6 +752,8 @@ xftfont_cached_font_ok (struct frame *f, Lisp_Object font_object,
   return ok;
 }
 
+static void syms_of_xftfont_for_pdumper (void);
+
 void
 syms_of_xftfont (void)
 {
@@ -768,8 +771,12 @@ syms_of_xftfont (void)
 This is needed with some fonts to correct vertical overlap of glyphs.  */);
   xft_font_ascent_descent_override = 0;
 
-  ascii_printable[0] = 0;
+  pdumper_do_now_and_after_load (syms_of_xftfont_for_pdumper);
+}
 
+static void
+syms_of_xftfont_for_pdumper (void)
+{
   xftfont_driver = ftfont_driver;
   xftfont_driver.type = Qxft;
   xftfont_driver.get_cache = xfont_driver.get_cache;
@@ -789,8 +796,8 @@ This is needed with some fonts to correct vertical overlap of glyphs.  */);
   xftfont_driver.shape = xftfont_shape;
 #endif
   /* When using X double buffering, the XftDraw structure we build
-   seems to be useless once a frame is resized, so recreate it on
-   ConfigureNotify and in some other cases.  */
+     seems to be useless once a frame is resized, so recreate it on
+     ConfigureNotify and in some other cases.  */
   xftfont_driver.drop_xrender_surfaces = xftfont_drop_xrender_surfaces;
 
   register_font_driver (&xftfont_driver, NULL);
diff --git a/src/xmenu.c b/src/xmenu.c
index 9ab7bdf..f080ffb 100644
--- a/src/xmenu.c
+++ b/src/xmenu.c
@@ -44,6 +44,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "buffer.h"
 #include "coding.h"
 #include "sysselect.h"
+#include "pdumper.h"
 
 #ifdef MSDOS
 #include "msdos.h"
@@ -2337,15 +2338,12 @@ DEFUN ("menu-or-popup-active-p", Fmenu_or_popup_active_p, Smenu_or_popup_active_
   return (popup_activated ()) ? Qt : Qnil;
 }
 \f
+
+static void syms_of_xmenu_for_pdumper (void);
+
 void
 syms_of_xmenu (void)
 {
-#ifdef USE_X_TOOLKIT
-  enum { WIDGET_ID_TICK_START = 1 << 16 };
-  widget_id_tick = WIDGET_ID_TICK_START;
-  next_menubar_widget_id = 1;
-#endif
-
   DEFSYM (Qdebug_on_next_call, "debug-on-next-call");
   defsubr (&Smenu_or_popup_active_p);
 
@@ -2354,4 +2352,16 @@ syms_of_xmenu (void)
   Ffset (intern_c_string ("accelerate-menu"),
 	 intern_c_string (Sx_menu_bar_open_internal.symbol_name));
 #endif
+
+  pdumper_do_now_and_after_load (syms_of_xmenu_for_pdumper);
+}
+
+static void
+syms_of_xmenu_for_pdumper (void)
+{
+#ifdef USE_X_TOOLKIT
+  enum { WIDGET_ID_TICK_START = 1 << 16 };
+  widget_id_tick = WIDGET_ID_TICK_START;
+  next_menubar_widget_id = 1;
+#endif
 }
diff --git a/src/xselect.c b/src/xselect.c
index b997cc8..bfa0dfb 100644
--- a/src/xselect.c
+++ b/src/xselect.c
@@ -35,6 +35,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "blockinput.h"
 #include "termhooks.h"
 #include "keyboard.h"
+#include "pdumper.h"
 
 #include <X11/Xproto.h>
 
@@ -2620,6 +2621,9 @@ x_send_client_event (Lisp_Object display, Lisp_Object dest, Lisp_Object from,
 }
 
 \f
+
+static void syms_of_xselect_for_pdumper (void);
+
 void
 syms_of_xselect (void)
 {
@@ -2635,17 +2639,9 @@ syms_of_xselect (void)
 
   reading_selection_reply = Fcons (Qnil, Qnil);
   staticpro (&reading_selection_reply);
-  reading_selection_window = 0;
-  reading_which_selection = 0;
 
-  property_change_wait_list = 0;
-  prop_location_identifier = 0;
-  property_change_reply = Fcons (Qnil, Qnil);
   staticpro (&property_change_reply);
 
-  converted_selections = NULL;
-  conversion_fail_tag = None;
-
   /* FIXME: Duplicate definition in nsselect.c.  */
   DEFVAR_LISP ("selection-converter-alist", Vselection_converter_alist,
 	       doc: /* An alist associating X Windows selection-types with functions.
@@ -2724,4 +2720,18 @@ A value of 0 means wait as long as necessary.  This is initialized from the
   DEFSYM (Qforeign_selection, "foreign-selection");
   DEFSYM (Qx_lost_selection_functions, "x-lost-selection-functions");
   DEFSYM (Qx_sent_selection_functions, "x-sent-selection-functions");
+
+  pdumper_do_now_and_after_load (syms_of_xselect_for_pdumper);
+}
+
+static void
+syms_of_xselect_for_pdumper (void)
+{
+  reading_selection_window = 0;
+  reading_which_selection = 0;
+  property_change_wait_list = 0;
+  prop_location_identifier = 0;
+  property_change_reply = Fcons (Qnil, Qnil);
+  converted_selections = NULL;
+  conversion_fail_tag = None;
 }
diff --git a/src/xsettings.c b/src/xsettings.c
index d7af68f..259cc57 100644
--- a/src/xsettings.c
+++ b/src/xsettings.c
@@ -32,6 +32,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "keyboard.h"
 #include "blockinput.h"
 #include "termhooks.h"
+#include "pdumper.h"
 
 #include <X11/Xproto.h>
 
@@ -1008,13 +1009,18 @@ void
 syms_of_xsettings (void)
 {
   current_mono_font = NULL;
+  PDUMPER_IGNORE (current_mono_font);
   current_font = NULL;
+  PDUMPER_IGNORE (current_font);
   first_dpyinfo = NULL;
+  PDUMPER_IGNORE (first_dpyinfo);
 #ifdef HAVE_GSETTINGS
   gsettings_client = NULL;
+  PDUMPER_IGNORE (gsettings_client);
 #endif
 #ifdef HAVE_GCONF
   gconf_client = NULL;
+  PDUMPER_IGNORE (gconf_client);
 #endif
 
   DEFSYM (Qmonospace_font_name, "monospace-font-name");
diff --git a/src/xterm.c b/src/xterm.c
index bdc21e6..ea21e8f 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -81,6 +81,7 @@ along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.  */
 #include "xsettings.h"
 #include "sysselect.h"
 #include "menu.h"
+#include "pdumper.h"
 
 #ifdef USE_X_TOOLKIT
 #include <X11/Shell.h>
@@ -12809,6 +12810,7 @@ void
 syms_of_xterm (void)
 {
   x_error_message = NULL;
+  PDUMPER_IGNORE (x_error_message);
 
   DEFSYM (Qvendor_specific_keysyms, "vendor-specific-keysyms");
   DEFSYM (Qlatin_1, "latin-1");

Re: Preview: portable dumper

[Burton Samograd]

Excellent idea and good work.  Will this also allow for the dumping of Emacs at arbitrary time after it’s been run, or will there still be issues with that?

Burton Samograd

> On Nov 28, 2016, at 12:50 PM, Daniel Colascione <dancol@dancol.org> wrote:
> 
> I've been working on a portable dumper for GNU Emacs. The attached patch is an early version of this work. I'll write up all the usual NEWS entries, changelogs, and (in this case necessary) dedicated documentation before I land it. I want to start getting review comments now that the code has roughly its final shape.
> 
> The point of this gargantuan patch is that we can rip out our unexec implementations and replace them with loading a data file that contains an Emacs heap image. There are no dependencies on executable rewriting, disabling ASLR, or saving and restoring internal malloc state. This system works with fully position-independent executables and with any malloc implementation.
> 
> Basically, there's a new dump-emacs-portable function that walks the Emacs heap and writes the data, along with necessary relocation, to a file called emacs.pdmp. On startup, early in main, we find emacs.pdmp and load it. Once we've loaded the file, we walk the list of relocations contained in the dump and adjust it to account for the runtime locations of Emacs and the dump data (neither of which we know in advance in a PIE world.)
> 
> There are a few subtleties: I've carefully designed the file format to be mmap-able and to minimize the number of on-demand copies the system makes while accessing this file. For example, we stick bool-vectors and string data at the end of the dump in a contiguous block. We follow this block with the relocations, which we can throw away as soon as we've used them.
> 
> An additional optimization follows, although this part isn't implemented yet: we can define a "preferred load address" for the dump and write relocation information such that if the dump and Emacs end up being where we expect them to be, we don't have to perform any relocations at all.
> 
> The system gracefully degrades though. If we can't use mmap or whatever on a given platform, it's possible to just slurp the whole file into a malloced region of memory and access it from there. This approach would benefit from compression, which will reduce IO loads: LZ4 reduces the dump size for me from ~12MB to ~4MB. As in the mmap case, we can throw away
> 
> The code isn't even close to optimized yet --- I've only tested it at -O0, defined GC_CHECK_MARKED_OBJECTS, and not yet inlined frequently-called functions pdumper.h --- but even so, it's within 100ms or so of an unexeced Emacs.
> 
> It's also possible to dump an already-dumped Emacs, so it should be possible for users to have their own dump files.
> 
> If we want to preserve the current model of a single "emacs" executable that contains itself, we can embed emacs.pdmp inside the emacs executable data section pretty easily. There's no behavior change involved.
> 
> If you want to try this code, build CANNOT_DUMP=1, run ./temacs -l loadup pdump, then ./emacs (or if that doesn't work, ./emacs --dump-file=emacs.pdmp).
> <pdumper.diff>

Re: Preview: portable dumper

[Daniel Colascione]

On 11/28/2016 11:58 AM, Burton Samograd wrote:
> Excellent idea and good work.  Will this also allow for the dumping of Emacs at arbitrary time after it’s been run, or will there still be issues with that?
>

In theory. In practice, I want to roll out this work gradually. Step one 
is to land it and keep it off by default --- we don't have to stop using 
unexec just because we want to experiment with a portable dumper.

Later, we'll enable the portable dumper by default, but as an unexec 
replacement, and keep unexec around. Once this arrangement works for a 
few releases, we can delete unexec and talk about dumping at later 
points in Emacs execution history.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 28 Nov 2016 11:50:31 -0800
> 
> I've been working on a portable dumper for GNU Emacs. The attached patch 
> is an early version of this work. I'll write up all the usual NEWS 
> entries, changelogs, and (in this case necessary) dedicated 
> documentation before I land it. I want to start getting review comments 
> now that the code has roughly its final shape.

Thanks for your efforts, but I think this is a wrong direction, that
will eventually get us into the same problem we have with unexec: the
need to know and depend on intimate details of relocations, memory
allocation, etc.  On top of that, adding Lisp objects will now require
writing its dumper back-end, so this will be a constant maintenance
burden of the kind that only a few of us can bear.

Making the initial load of preloaded Lisp files (most probably, a
single large file) fast enough to allow us to dump the dumping phase
altogether is a much more perspective direction.

Re: Preview: portable dumper

[Daniel Colascione]

On 11/28/2016 12:12 PM, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 28 Nov 2016 11:50:31 -0800
>>
>> I've been working on a portable dumper for GNU Emacs. The attached patch
>> is an early version of this work. I'll write up all the usual NEWS
>> entries, changelogs, and (in this case necessary) dedicated
>> documentation before I land it. I want to start getting review comments
>> now that the code has roughly its final shape.
>
> Thanks for your efforts, but I think this is a wrong direction, that
> will eventually get us into the same problem we have with unexec: the
> need to know and depend on intimate details of relocations, memory
> allocation, etc.  On top of that, adding Lisp objects will now require
> writing its dumper back-end, so this will be a constant maintenance
> burden of the kind that only a few of us can bear.

Only details of Emacs, not the system. I insist that this direction is 
the one that will yield adequate performance.

> Making the initial load of preloaded Lisp files (most probably, a
> single large file) fast enough to allow us to dump the dumping phase
> altogether is a much more perspective direction.

That's what this work does. If we disagree, that's a shame. I feel 
strongly enough about this issue to fork over it.

Re: Preview: portable dumper

[Daniel Colascione]

On 11/28/2016 12:14 PM, Daniel Colascione wrote:
> On 11/28/2016 12:12 PM, Eli Zaretskii wrote:
>>> From: Daniel Colascione <dancol@dancol.org>
>>> Date: Mon, 28 Nov 2016 11:50:31 -0800
>>>
>>> I've been working on a portable dumper for GNU Emacs. The attached patch
>>> is an early version of this work. I'll write up all the usual NEWS
>>> entries, changelogs, and (in this case necessary) dedicated
>>> documentation before I land it. I want to start getting review comments
>>> now that the code has roughly its final shape.
>>
>> Thanks for your efforts, but I think this is a wrong direction, that
>> will eventually get us into the same problem we have with unexec: the
>> need to know and depend on intimate details of relocations, memory
>> allocation, etc.  On top of that, adding Lisp objects will now require
>> writing its dumper back-end, so this will be a constant maintenance
>> burden of the kind that only a few of us can bear.
>
> Only details of Emacs, not the system. I insist that this direction is
> the one that will yield adequate performance.
>
>> Making the initial load of preloaded Lisp files (most probably, a
>> single large file) fast enough to allow us to dump the dumping phase
>> altogether is a much more perspective direction.
>
> That's what this work does. If we disagree, that's a shame. I feel
> strongly enough about this issue to fork over it.

Also, this work *exists*. I don't see anyone signing up to implement the 
giant elc approach, which *I* believe is fundamentally the wrong direction.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 28 Nov 2016 12:14:34 -0800
> Cc: emacs-devel@gnu.org
> 
> I feel strongly enough about this issue to fork over it.

Don't be ridiculous.  Emacs is not my pet project.  You just need a
new maintainer, that's all.  Because if Emacs is going in a direction
to which I object, I definitely cannot be it.

Re: Preview: portable dumper

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> Thanks for your efforts, but I think this is a wrong direction, that will
EZ> eventually get us into the same problem we have with unexec: the need to
EZ> know and depend on intimate details of relocations, memory allocation,
EZ> etc. On top of that, adding Lisp objects will now require writing its
EZ> dumper back-end, so this will be a constant maintenance burden of the kind
EZ> that only a few of us can bear.

Is there a way to get away from such a requirement, Eli? If unexec becomes
untenable in the future, is there an alternative that doesn't place the burden
upon us to encode the right amount of information in the dumped file?

If I understand Daniel's contribution, he's proposing the equivalent of a
program loader for Emacs Lisp byte-code, stored within what amounts to a
custom object file format.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Daniel Colascione]

On 11/28/2016 12:20 PM, John Wiegley wrote:
>>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>
> EZ> Thanks for your efforts, but I think this is a wrong direction, that will
> EZ> eventually get us into the same problem we have with unexec: the need to
> EZ> know and depend on intimate details of relocations, memory allocation,
> EZ> etc. On top of that, adding Lisp objects will now require writing its
> EZ> dumper back-end, so this will be a constant maintenance burden of the kind
> EZ> that only a few of us can bear.
>
> Is there a way to get away from such a requirement, Eli? If unexec becomes
> untenable in the future, is there an alternative that doesn't place the burden
> upon us to encode the right amount of information in the dumped file?
>
> If I understand Daniel's contribution, he's proposing the equivalent of a
> program loader for Emacs Lisp byte-code, stored within what amounts to a
> custom object file format.

Yes --- one that *we* control, which frees us from changes to underlying 
systems. Yes, the thing has intimate knowledge of Lisp_Object internals, 
but whoever changes those internals (and these changes are rare) can 
change the dumper too.

Re: Preview: portable dumper

[John Wiegley]

>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:

DC> Yes --- one that *we* control, which frees us from changes to underlying
DC> systems. Yes, the thing has intimate knowledge of Lisp_Object internals,
DC> but whoever changes those internals (and these changes are rare) can
DC> change the dumper too.

Then the difference between .elc and this new -- .eld? -- would be exactly
what?  Relocation information to speed up load time?

I'm just wondering: if it's strictly a speed gain, then is there any other way
to achieve it; if it's a functionality gain, what are the advantages over just
speeding up .elc loading.

Thanks for doing this work, btw!! The value of your exploration is not lost on
me, since you've now made a concrete alternative for us to consider. If the
differences in approach are small enough, there's no reason we can't change
our minds yet again in the future. This is a purely internal matter, after
all.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Daniel Colascione]

On 11/28/2016 12:26 PM, John Wiegley wrote:
>>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:
>
> DC> Yes --- one that *we* control, which frees us from changes to underlying
> DC> systems. Yes, the thing has intimate knowledge of Lisp_Object internals,
> DC> but whoever changes those internals (and these changes are rare) can
> DC> change the dumper too.
>
> Then the difference between .elc and this new -- .eld? -- would be exactly
> what?  Relocation information to speed up load time?
>
> I'm just wondering: if it's strictly a speed gain, then is there any other way
> to achieve it; if it's a functionality gain, what are the advantages over just
> speeding up .elc loading.

The dump can be used as-is without preparation, particularly if it loads 
at its desired base address. Reading an ELC requires lots of parsing, 
interning, and allocation of various sorts. It also prohibits any kind 
of data sharing between differences instances of Emacs on the same 
system, while I expect about a third to half of these dumps to be shared.

It's also possible to dump arbitrarily already-loaded Emacs processes, 
which you don't get with a merged ELC, so that's a functionality difference.

If Lisp layout changes in the future, it's very easy to adjust pdumper 
to match. It's really not a big maintenance burden; read the code.

> Thanks for doing this work, btw!! The value of your exploration is not lost on
> me, since you've now made a concrete alternative for us to consider. If the
> differences in approach are small enough, there's no reason we can't change
> our minds yet again in the future. This is a purely internal matter, after
> all.

Are you rejecting the patch?

Re: Preview: portable dumper

[Burton Samograd]

> It's also possible to dump arbitrarily already-loaded Emacs processes, which you don't get with a merged ELC, so that's a functionality difference.

You can dump a running process and have it come back exactly, GUI and everything? That would be a godsend, but given the dependency on lower level libraries written in C, I’m having a hard time seeing how this would work.

—
Burton Samograd

Re: Preview: portable dumper

[Daniel Colascione]

On November 28, 2016 12:37:52 PM PST, Burton Samograd <burton.samograd@autodesk.com> wrote:
>
>> It's also possible to dump arbitrarily already-loaded Emacs
>processes, which you don't get with a merged ELC, so that's a
>functionality difference.
>
>You can dump a running process and have it come back exactly, GUI and
>everything? That would be a godsend, but given the dependency on lower
>level libraries written in C, I’m having a hard time seeing how this
>would work.


Frame and window objects come back dead after being restored from the dump. Buffers are intact though. If you want to preserve window configuration, you can layer saving and restoring of layout on top, just like desktop.el does.


>
>—
>Burton Samograd

Re: Preview: portable dumper

[Ted Zlatanov]

On Mon, 28 Nov 2016 12:44:50 -0800 Daniel Colascione <dancol@dancol.org> wrote: 

DC> Frame and window objects come back dead after being restored from the dump.
DC> Buffers are intact though. If you want to preserve window configuration, you can
DC> layer saving and restoring of layout on top, just like desktop.el does.

Since this will also be available to users, is there a way to exclude
some variables from dumping explicitly when they are declared? For
instance cache data, especially cached passwords?

Thanks
Ted

Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Nov 29 2016, Ted Zlatanov wrote:
> On Mon, 28 Nov 2016 12:44:50 -0800 Daniel Colascione <dancol@dancol.org> wrote: 
>
> DC> Frame and window objects come back dead after being restored from the dump.
> DC> Buffers are intact though. If you want to preserve window
> DC> configuration, you can
> DC> layer saving and restoring of layout on top, just like desktop.el does.
>
> Since this will also be available to users, is there a way to exclude
> some variables from dumping explicitly when they are declared? For
> instance cache data, especially cached passwords?

Just let-bind any variables you want with the values you'd prefer them
to have in the dump. We can come up with something fancier if needed.

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Frame and window objects come back dead after being restored from the dump.

Can that be fixed by creating new equivalent windows and frames?


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Nov 29 2016, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Frame and window objects come back dead after being restored from the dump.
>
> Can that be fixed by creating new equivalent windows and frames?

Yes: you'd want to reuse desktop's code to save window and frame
configuration. I plan to provide a both-dump-hook and after-dump-hook
mechanism that Lisp code can use for tasks like this.

Re: Preview: portable dumper

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

>
> > Can that be fixed by creating new equivalent windows and frames?
>
> Yes: you'd want to reuse desktop's code to save window and frame
> configuration. I plan to provide a both-dump-hook and after-dump-hook
> mechanism that Lisp code can use for tasks like this.
>

This comment remembered me about a proposal I was waiting to be polished
about the lisp objects representation, but since it's related it kinda
forces me to talk about it earlier :-)

I wrote an utility that lets you see your keyboards macros as emacs lisp (
https://github.com/Silex/elmacro).

In this utility, I have to deal with all kind of lisp objects that needed
to be serialized, and I found myself writing utilities that dumped windows,
frames & buffers differently than was "print" does.

Basically, instead of the traditional "#<window 3 on bla.el>" that does not
eval well, I dump it as "(elmacro-get-window 3)". This function just loops
over "(window-list)" and returns the correct window, that way I can eval
that code and it works (of course it does not work of the window 3 gets
destroyed before the code evaluation, but that's ok). A good example are
mouse clicks: https://github.com/Silex/elmacro#mouse-events-support

During the implementation I often thought "wouldn't it be nice if 'print'
(and prin1/prin1-to-string/etc) dumped ALL lisp objects as something
evalable?".

So, here is my proposal. Make all emacs lisp objects always have an "read"
friendly text representation (or at least as much as possible).

Maybe by doing so it also simplifies the dumper.

Kind regards,
Philippe

[-- Attachment #2: Type: text/html, Size: 2240 bytes --]

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: burton.samograd@autodesk.com, eliz@gnu.org, emacs-devel@gnu.org
> Date: Tue, 29 Nov 2016 11:48:40 -0500
> 
>   > Frame and window objects come back dead after being restored from the dump.
> 
> Can that be fixed by creating new equivalent windows and frames?

I'm actually wondering why frames and windows (and buffers, for that
matter) need to be dumped at all.  The fact we do that now is just an
artifact of how unexec works: it dumps the entire data section.  We
even have special code that runs at startup to undo some of the
peculiarities this causes.

However, a dumper that we control by manually telling it what to dum
and what not doesn't have to do that.  IMO, that's just something that
doesn't need to exist in the brave new unexec-free world.  (Note that
the "one huge .elc file" method has this built-in.)

Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Nov 29 2016, Eli Zaretskii wrote:
>> From: Richard Stallman <rms@gnu.org>
>> CC: burton.samograd@autodesk.com, eliz@gnu.org, emacs-devel@gnu.org
>> Date: Tue, 29 Nov 2016 11:48:40 -0500
>> 
>>   > Frame and window objects come back dead after being restored from the dump.
>> 
>> Can that be fixed by creating new equivalent windows and frames?
>
> I'm actually wondering why frames and windows (and buffers, for that
> matter) need to be dumped at all.  The fact we do that now is just an
> artifact of how unexec works: it dumps the entire data section.  We
> even have special code that runs at startup to undo some of the
> peculiarities this causes.
>
> However, a dumper that we control by manually telling it what to dum
> and what not doesn't have to do that.  IMO, that's just something that
> doesn't need to exist in the brave new unexec-free world.  (Note that
> the "one huge .elc file" method has this built-in.)

Frames and windows aren't dumped. That's why they come back as
dead objects. Lisp objects have to point to *something*, and a dead
object of the right type is a good placeholder.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Cc: rms@gnu.org,  emacs-devel@gnu.org,  burton.samograd@autodesk.com
> Date: Tue, 29 Nov 2016 09:49:59 -0800
> 
> > I'm actually wondering why frames and windows (and buffers, for that
> > matter) need to be dumped at all.  The fact we do that now is just an
> > artifact of how unexec works: it dumps the entire data section.  We
> > even have special code that runs at startup to undo some of the
> > peculiarities this causes.
> >
> > However, a dumper that we control by manually telling it what to dum
> > and what not doesn't have to do that.  IMO, that's just something that
> > doesn't need to exist in the brave new unexec-free world.  (Note that
> > the "one huge .elc file" method has this built-in.)
> 
> Frames and windows aren't dumped. That's why they come back as
> dead objects. Lisp objects have to point to *something*, and a dead
> object of the right type is a good placeholder.

Which Lisp objects need to point to frames and windows?

Re: Preview: portable dumper

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> However, a dumper that we control by manually telling it what to dum and
EZ> what not doesn't have to do that. IMO, that's just something that doesn't
EZ> need to exist in the brave new unexec-free world. (Note that the "one huge
EZ> .elc file" method has this built-in.)

I'd still like to see research into the "one huge .elc file" method, for the
obvious benefits you listed. As far as I understand, the only difference
between the portable dumper, and one-huge-.elc, is efficiency. Given infinite
resources, the .elc method is much to be preferred, since everything we need
"falls out" from existing design.

I wonder what it would take to bridge that efficiency gap, or am I missing
some new feature provided by the portable dumper?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: rms@gnu.org,  dancol@dancol.org,  emacs-devel@gnu.org,  burton.samograd@autodesk.com
> Date: Tue, 29 Nov 2016 10:03:01 -0800
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> However, a dumper that we control by manually telling it what to dum and
> EZ> what not doesn't have to do that. IMO, that's just something that doesn't
> EZ> need to exist in the brave new unexec-free world. (Note that the "one huge
> EZ> .elc file" method has this built-in.)
> 
> I'd still like to see research into the "one huge .elc file" method, for the
> obvious benefits you listed.

Stefan and Ken made some good progress.  IMO, we should actively
continue pursuing that goal.

> As far as I understand, the only difference between the portable
> dumper, and one-huge-.elc, is efficiency. Given infinite resources,
> the .elc method is much to be preferred, since everything we need
> "falls out" from existing design.

Any memory dumper will always be more efficient than loading .elc.
The challenge is to make the difference smaller than the annoyance
level of the users.  Since initial testing already produces load times
below 0.5 sec, I don't see why further optimizations won't deliver
enough speedup to make it fast enough.

Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Nov 29 2016, Eli Zaretskii wrote:
>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: rms@gnu.org, dancol@dancol.org, emacs-devel@gnu.org,
>> burton.samograd@autodesk.com
>> Date: Tue, 29 Nov 2016 10:03:01 -0800
>> 
>> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> EZ> However, a dumper that we control by manually telling it what to dum and
>> EZ> what not doesn't have to do that. IMO, that's just something that doesn't
>> EZ> need to exist in the brave new unexec-free world. (Note that the "one huge
>> EZ> .elc file" method has this built-in.)
>> 
>> I'd still like to see research into the "one huge .elc file" method, for the
>> obvious benefits you listed.
>
> Stefan and Ken made some good progress.  IMO, we should actively
> continue pursuing that goal.
>
>> As far as I understand, the only difference between the portable
>> dumper, and one-huge-.elc, is efficiency. Given infinite resources,
>> the .elc method is much to be preferred, since everything we need
>> "falls out" from existing design.
>
> Any memory dumper will always be more efficient than loading .elc.
> The challenge is to make the difference smaller than the annoyance
> level of the users.  Since initial testing already produces load times
> below 0.5 sec, I don't see why further optimizations won't deliver
> enough speedup to make it fast enough.

Because you can't possible get away, in the ELC approach, from:

 1) reading every damn byte of the file, and
 2) allocating memory for every object you read.

Efficiency is always going to be at least linear in the size of the
"elc dump" --- super-linear complexity if you want read-circle.

A memory dump is linear only in the amount of data you actually
access. Plus, there's no serialization or deserialization overhead. A
memory dump is also inherently gentler on the underlying system, since
we can use many of the pages unmodified (which makes them shareable and
trivially swappable), while every damn byte of the Emacs heap after
loading ELC files will be process-private dirty storage.

I do not think the portable dumper is something that's so complex as to
not be worth the performance, especially since any lread implementation
that's even close to being acceptably fast is going to be very complex
in its own right.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Cc: John Wiegley <jwiegley@gmail.com>,  rms@gnu.org,  emacs-devel@gnu.org,  burton.samograd@autodesk.com
> Date: Tue, 29 Nov 2016 10:49:10 -0800
> 
> > Any memory dumper will always be more efficient than loading .elc.
> [...]
> Because you can't possible get away, in the ELC approach, from:
> 
>  1) reading every damn byte of the file, and
>  2) allocating memory for every object you read.
> 
> Efficiency is always going to be at least linear in the size of the
> "elc dump" --- super-linear complexity if you want read-circle.
> 
> A memory dump is linear only in the amount of data you actually
> access. Plus, there's no serialization or deserialization overhead. A
> memory dump is also inherently gentler on the underlying system, since
> we can use many of the pages unmodified (which makes them shareable and
> trivially swappable), while every damn byte of the Emacs heap after
> loading ELC files will be process-private dirty storage.

Like I said: any memory dumper will always be faster.  But having less
functionality coded in non-trivial C is IMO an important goal, one we
have been pursuing for quite some time, and for a good reason.
Although it should be clear that any adequate C implementation will
always be faster than an equivalent Lisp implementation.

> I do not think the portable dumper is something that's so complex as to
> not be worth the performance

It's complex.  I have read the patch.  I do know my way in the Emacs
internals.

> especially since any lread implementation that's even close to being
> acceptably fast is going to be very complex in its own right.

Even if we accept your opinion on the necessity for changes in lread
(and I don't think I agree), that is a one-off job.

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I do not think the portable dumper is something that's so complex as to
  > not be worth the performance, especially since any lread implementation
  > that's even close to being acceptably fast is going to be very complex
  > in its own right.

Rather than arguing theoretically about speculations,
let's see people do their best to improve the big-elc-file
approach, and then compare facts with facts.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, jwiegley@gmail.com, burton.samograd@autodesk.com,
> 	emacs-devel@gnu.org
> Date: Thu, 01 Dec 2016 04:18:04 -0500
> 
> Rather than arguing theoretically about speculations,
> let's see people do their best to improve the big-elc-file
> approach, and then compare facts with facts.

Agreed.

Ken, will you be able to continue working on your branch in this
direction?  Are all the changes suggested/tried by Stefan already on
that branch?  If not, would it be possible for Stefan or yourself to
update the branch?

Thanks in advance.

Re: Preview: portable dumper

[Ken Raeburn]

> On Dec 1, 2016, at 13:11, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Richard Stallman <rms@gnu.org>
>> CC: eliz@gnu.org, jwiegley@gmail.com, burton.samograd@autodesk.com,
>> 	emacs-devel@gnu.org
>> Date: Thu, 01 Dec 2016 04:18:04 -0500
>> 
>> Rather than arguing theoretically about speculations,
>> let's see people do their best to improve the big-elc-file
>> approach, and then compare facts with facts.
> 
> Agreed.
> 
> Ken, will you be able to continue working on your branch in this
> direction?  Are all the changes suggested/tried by Stefan already on
> that branch?  If not, would it be possible for Stefan or yourself to
> update the branch?

I have an update I’m still working on, to refine the hash table handling I did to generate less garbage while reading normal .el or .elc files.  I hope to get that uploaded soon.  And I think my previous push of the branch was before Stefan’s latest patch (Oct 31?), so I’ll pull that in too, unless he’s done some more work since.

My focus has been on the dumped.elc load performance; the content generation has been pretty much all his work.

Stefan’s work on reducing “#n#” placeholder substitutions tackled the particular case of cons cells, and he suggested extending it to other types, but based on some stats I got looking over the big .elc file, I think cons cells and strings are the majority of the cases.  Stefan’s code addresses the cons cells, and for property-less strings (and certain other types) the substitution pass isn’t needed, so I think we’ve addressed the biggest gains to be had in that area.

The patch I made to use a pair of hash tables for tracking previously read objects seems to have improved performance as well, but I’ve started to refine it a bit to not create a new pair of hash tables for every top-level read done, when the majority of the calls (over 95% during a loadup.el pass I instrumented) don’t actually wind up needing the hash tables.

Regardless of the outcome of this current disagreement and whether big-elc gets used, I think the lread changes may soon be ready to merge, even if the performance benefits are less drastic for normal Lisp code.

As for non-performance issues, last time I tried the big-elc code, it didn’t work for me in batch mode because of a null pointer dereference in face processing; I haven’t had a chance to look into it further but maybe in the next few days I can.

Ken

Re: Preview: portable dumper

[Daniel Colascione]

On 12/01/2016 08:28 PM, Ken Raeburn wrote:
>
>> On Dec 1, 2016, at 13:11, Eli Zaretskii <eliz@gnu.org> wrote:
>>
>>> From: Richard Stallman <rms@gnu.org>
>>> CC: eliz@gnu.org, jwiegley@gmail.com, burton.samograd@autodesk.com,
>>> 	emacs-devel@gnu.org
>>> Date: Thu, 01 Dec 2016 04:18:04 -0500
>>>
>>> Rather than arguing theoretically about speculations,
>>> let's see people do their best to improve the big-elc-file
>>> approach, and then compare facts with facts.
>>
>> Agreed.
>>
>> Ken, will you be able to continue working on your branch in this
>> direction?  Are all the changes suggested/tried by Stefan already on
>> that branch?  If not, would it be possible for Stefan or yourself to
>> update the branch?
>
> I have an update I’m still working on, to refine the hash table handling I did to generate less garbage while reading normal .el or .elc files.  I hope to get that uploaded soon.  And I think my previous push of the branch was before Stefan’s latest patch (Oct 31?), so I’ll pull that in too, unless he’s done some more work since.
>
> My focus has been on the dumped.elc load performance; the content generation has been pretty much all his work.
>
> Stefan’s work on reducing “#n#” placeholder substitutions tackled the particular case of cons cells, and he suggested extending it to other types, but based on some stats I got looking over the big .elc file, I think cons cells and strings are the majority of the cases.  Stefan’s code addresses the cons cells, and for property-less strings (and certain other types) the substitution pass isn’t needed, so I think we’ve addressed the biggest gains to be had in that area.
>
> The patch I made to use a pair of hash tables for tracking previously read objects seems to have improved performance as well, but I’ve started to refine it a bit to not create a new pair of hash tables for every top-level read done, when the majority of the calls (over 95% during a loadup.el pass I instrumented) don’t actually wind up needing the hash tables.
>

Of course all of this work isn't in C and doesn't require specialized 
skills, right?

> Regardless of the outcome of this current disagreement and whether
big-elc gets used, I think the lread changes may soon be ready to merge,
even if the performance benefits are less drastic for normal Lisp code.

Yes, elc loading performance is welcome regardless.

Re: Preview: portable dumper

[Eli Zaretskii]

> Cc: rms@gnu.org, jwiegley@gmail.com, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Thu, 1 Dec 2016 20:41:47 -0800
> 
> Of course all of this work isn't in C and doesn't require specialized 
> skills, right?

It does.  We also make additions in C in other areas.

The differences between this and other C changes, and your proposed
dumper, as was already written several times, are:

  . there's no Lisp-level alternative for them
  . these changes are mostly a one-time job, and the probability of
    them to be actively maintained is very low, if not zero

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Thu, 1 Dec 2016 23:28:27 -0500
> Cc: rms@gnu.org,
>  dancol@dancol.org,
>  jwiegley@gmail.com,
>  emacs-devel@gnu.org
> 
> > Ken, will you be able to continue working on your branch in this
> > direction?  Are all the changes suggested/tried by Stefan already on
> > that branch?  If not, would it be possible for Stefan or yourself to
> > update the branch?
> 
> I have an update I’m still working on, to refine the hash table handling I did to generate less garbage while reading normal .el or .elc files.  I hope to get that uploaded soon.  And I think my previous push of the branch was before Stefan’s latest patch (Oct 31?), so I’ll pull that in too, unless he’s done some more work since.

Thanks.

> Regardless of the outcome of this current disagreement and whether big-elc gets used, I think the lread changes may soon be ready to merge, even if the performance benefits are less drastic for normal Lisp code.

I agree in general, but let's discuss this again when you think the
branch is ready.  I'd like to have the branch reviewed by those who
are interested (I will certainly do that myself) before we merge.

Does the branch include any benchmarks, to compare performance with
the current code, both for dumped.elc and for normal Lisp code?  If
not, can they be included/added, perhaps under test/?

> As for non-performance issues, last time I tried the big-elc code, it didn’t work for me in batch mode because of a null pointer dereference in face processing; I haven’t had a chance to look into it further but maybe in the next few days I can.

Let me know if you need help in that department.

Thanks again for working on this.

Re: Preview: portable dumper

[Ken Raeburn]

On Dec 2, 2016, at 03:03, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Ken Raeburn <raeburn@raeburn.org>
>> 
>> Regardless of the outcome of this current disagreement and whether big-elc gets used, I think the lread changes may soon be ready to merge, even if the performance benefits are less drastic for normal Lisp code.
> 
> I agree in general, but let's discuss this again when you think the
> branch is ready.  I'd like to have the branch reviewed by those who
> are interested (I will certainly do that myself) before we merge.

Certainly.  Some of that code is a bit subtle, and I’d appreciate the review.

> Does the branch include any benchmarks, to compare performance with
> the current code, both for dumped.elc and for normal Lisp code?  If
> not, can they be included/added, perhaps under test/?

Not currently.  My benchmark has usually been along the lines of:

  time ./emacs -nw —eval '(progn (sit-for 0.01) (kill-emacs))’

to initialize everything (including tty display and input checking) and then quit, in a consistent fashion from run to run.  After running it several times and hopefully seeing fairly consistent numbers after the first invocation or two, I estimate the average by eye.  So, yeah, it could be more precise.

Since we’re timing Emacs startup itself, we can’t test it with Lisp helper functions invoked once Emacs has started like we would with many other things.  And it’ll be dependent on system issues, like whether emacs and dumped.elc are in the kernel’s page cache or not.  For simplicity I’ve been testing with the cache warmed up, because it eliminates the storage system hardware and OS read-ahead algorithms and such while I focus on the lread.c code, but in the real world those will sometimes be factors too, and should be considered.

I’ll think about scripting this in some way that may be more robust, and maybe adding hooks for OS-specific cache flushing and such.

Ken

Re: Preview: portable dumper

[John Wiegley]

>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:

DC> Are you rejecting the patch?

No, not at all.  I'll follow up more in my response to Eli.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Burton Samograd]

> I'm just wondering: if it's strictly a speed gain, then is there any other way
> to achieve it; if it's a functionality gain, what are the advantages over just
> speeding up .elc loading.

It sounds like it disconnects the internal unexec format from the underlying system, which could be a good portability gain.  It also allows the emacs executable to load different core runtimes and would make the initial binary quite a bit smaller, if I understand correctly.

This type of patch seems like a good move to me and pulls it closer to almost every CL implementation out there.

—
Burton Samograd

Re: Preview: portable dumper

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Daniel Colascione <dancol@dancol.org>,  emacs-devel@gnu.org
> Date: Mon, 28 Nov 2016 12:20:41 -0800
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> Thanks for your efforts, but I think this is a wrong direction, that will
> EZ> eventually get us into the same problem we have with unexec: the need to
> EZ> know and depend on intimate details of relocations, memory allocation,
> EZ> etc. On top of that, adding Lisp objects will now require writing its
> EZ> dumper back-end, so this will be a constant maintenance burden of the kind
> EZ> that only a few of us can bear.
> 
> Is there a way to get away from such a requirement, Eli?

Yes, by loading Lisp code directly.

> If I understand Daniel's contribution, he's proposing the equivalent of a
> program loader for Emacs Lisp byte-code, stored within what amounts to a
> custom object file format.

No, it's not only the byte code, its the entire state of Emacs after
loading that byte code and initializing variables and subroutines not
in Lisp.  IOW, all of the Lisp objects in the entire heap, no matter
where they came from.

Re: Preview: portable dumper

[Paul Eggert]

On 11/28/2016 12:12 PM, Eli Zaretskii wrote:
> I think this is a wrong direction

If the proposed dumper works reliably and quickly and portably, Emacs 
should be in a significantly better place than it is now. So in that 
sense this is not the wrong direction.

Perhaps you're right and making lread faster would work better. However, 
nobody has implemented that. If someone does implement it we can switch 
to it later, regardless of whether we use the proposed dumper in the 
meantime.

Re: Preview: portable dumper

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 28 Nov 2016 12:21:13 -0800
> 
> On 11/28/2016 12:12 PM, Eli Zaretskii wrote:
> > I think this is a wrong direction
> 
> If the proposed dumper works reliably and quickly and portably, Emacs 
> should be in a significantly better place than it is now. So in that 
> sense this is not the wrong direction.
> 
> Perhaps you're right and making lread faster would work better. However, 
> nobody has implemented that. If someone does implement it we can switch 
> to it later, regardless of whether we use the proposed dumper in the 
> meantime.

I'm not going to argue.  If enough people think I'm mistaken and want
to go the portable dumper way, I will resign right here and now.  It
is very easy to convince me to step down, because I hesitated to take
this job to begin with.

Your call, crowd.

Re: Preview: portable dumper

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> I'm not going to argue. If enough people think I'm mistaken and want to go
EZ> the portable dumper way, I will resign right here and now. It is very easy
EZ> to convince me to step down, because I hesitated to take this job to begin
EZ> with.

EZ> Your call, crowd.

OK, please let's take a few steps back and not make any rash decisions
today. :)

If Daniel's proposing an internal optimization and is willing to maintain it
-- and if it doesn't make work for other contributors -- I suggest we accept
it as long as he, or someone else, is willing to support it. At the very
least, it frees us from dependence on a glibc feature that is doomed to
disappear.

If later the work becomes difficult to support, or if Daniel disappears, won't
we just be returning to the same position as now? If that happeans, we can
take what we've learned and try another approach, such as speeding up .elc
loading. Or, if someone comes up with a better alternative in the meantime, we
can just switch to it.

Accepting this patch doesn't mean we don't try the .elc speedup idea. The only
thing we have to make sure is that it doesn't unduly increase the difficulty
of adding new Lisp objects.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  dancol@dancol.org,  emacs-devel@gnu.org
> Date: Mon, 28 Nov 2016 12:47:10 -0800
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> I'm not going to argue. If enough people think I'm mistaken and want to go
> EZ> the portable dumper way, I will resign right here and now. It is very easy
> EZ> to convince me to step down, because I hesitated to take this job to begin
> EZ> with.
> 
> EZ> Your call, crowd.
> 
> OK, please let's take a few steps back and not make any rash decisions
> today. :)

It's no longer my decision.

> If Daniel's proposing an internal optimization and is willing to maintain it
> -- and if it doesn't make work for other contributors -- I suggest we accept
> it as long as he, or someone else, is willing to support it.

Emacs future cannot depend on a single person.  We had more than once
a situation where key developer(s) left leaving a feature or a whole
area unmaintained.  If some development isn't a one-time job -- and
this one surely isn't -- then experience shows that any arcane code in
C that loses its experts will become unmaintained and unmaintainable.
The only hope for us to escape this fate is to do it in a way that
most contemporary contributors understand and can easily learn to
hack, which means do it mostly in Lisp, reusing the existing C
machinery that is proven by time and doesn't need maintenance --
'load' and its cousins.  That was what I proposed.  The advantage of
that is that Emacs then becomes a normal program, one that doesn't
dump itself in any way, shape, or form, but instead reads in a
byte-compiled file, using the normal memory allocation routines.  No
more need to save any internal state, worry about relocations, ASLR,
etc.  It amazes me that the advantages of that are not immediately
clear to everyone, but I guess I'm biased.

> At the very least, it frees us from dependence on a glibc feature
> that is doomed to disappear.

No one is suggesting to stay with unexec.  The disagreement is about
where to go to get rid of it.

> If later the work becomes difficult to support, or if Daniel disappears, won't
> we just be returning to the same position as now? If that happeans, we can
> take what we've learned and try another approach, such as speeding up .elc
> loading. Or, if someone comes up with a better alternative in the meantime, we
> can just switch to it.

I don't think this is practical.  We cannot go back, certainly not
when unexec requires the expertise it does.  No one will agree to go
back.  Like with unexec, the only way the portable dumper will be
thrown away is if/when it turns out it is an obstacle to Emacs
development, by which time it will be too late.

> Accepting this patch doesn't mean we don't try the .elc speedup idea. The only
> thing we have to make sure is that it doesn't unduly increase the difficulty
> of adding new Lisp objects.

Accepting this patch means we believe in this direction, consider it a
perspective one, enough so that we are willing to invest a non-trivial
amount of energy into making it happen, fix the fallout, debug the
results, etc.  We are talking about at least one major release, maybe
more, and definitely several minor releases.  IOW, we are talking
several years.  I don't think we should gamble with such high stakes,
not unless we sure we want to go there, because this is the best
alternative of ditching unexec.

Anyway, I already wrote all this; there was a long discussion about
these issues, it's in the archives.  I have nothing else to add to
what I wrote back then.  This project needs to decide where it wants
to go, and the rest will be history.

Re: Preview: portable dumper

[Daniel Colascione]

On Mon, Nov 28 2016, Eli Zaretskii wrote:
>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: Paul Eggert <eggert@cs.ucla.edu>,  dancol@dancol.org,  emacs-devel@gnu.org
>> Date: Mon, 28 Nov 2016 12:47:10 -0800
>> 
>> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> EZ> I'm not going to argue. If enough people think I'm mistaken and want to go
>> EZ> the portable dumper way, I will resign right here and now. It is very easy
>> EZ> to convince me to step down, because I hesitated to take this job to begin
>> EZ> with.
>> 
>> EZ> Your call, crowd.
>> 
>> OK, please let's take a few steps back and not make any rash decisions
>> today. :)
>
> It's no longer my decision.
>
>> If Daniel's proposing an internal optimization and is willing to maintain it
>> -- and if it doesn't make work for other contributors -- I suggest we accept
>> it as long as he, or someone else, is willing to support it.
>
> Emacs future cannot depend on a single person.

We are a long way away from the day when only one person left on earth
can simultaneously hold in his mind the concepts of addition, modulus,
and binary search in C. The operating principle of this work is very
simple. If we need internals documentation, I'll write documentation ---
but there's really not much to it.

> We had more than once
> a situation where key developer(s) left leaving a feature or a whole
> area unmaintained.  If some development isn't a one-time job -- and
> this one surely isn't -- then experience shows that any arcane code in
> C that loses its experts will become unmaintained and unmaintainable.
> The only hope for us to escape this fate is to do it in a way that
> most contemporary contributors understand and can easily learn to
> hack, which means do it mostly in Lisp, reusing the existing C
> machinery that is proven by time and doesn't need maintenance --
> 'load' and its cousins.

They don't need maintenance --- except that they're at least an order of
magnitude too slow right now, and nobody's stepped up to do the
maintenance work necessary to make them useful in a new use case.

> That was what I proposed.  The advantage of
> that is that Emacs then becomes a normal program, one that doesn't
> dump itself in any way, shape, or form, but instead reads in a
> byte-compiled file, using the normal memory allocation routines.  No
> more need to save any internal state, worry about relocations, ASLR,
> etc.  It amazes me that the advantages of that are not immediately
> clear to everyone, but I guess I'm biased.

The whole point of the portable dumper is that you don't need to worry
about ASLR, ASAN, TSAN, or whatever libc people dream up next. Emacs
*is* a normal program --- it does only the things normal programs are
allowed to do and that platforms support, things like pointer assignment
and reading from a file. A "relocation" is literally just addition ---
it's not some horrible dynamic programming monstrosity like redisplay,
which I understand used to literally have a skull and crossbones in
ASCII art over the main body of the code.

Using your arguments, we should switch to Pango for text layout, FUSE
for remote access, clang-format for indentation, and --- hell, let's
just make Emacs a webapp. If you want maximum technological
democratization, you want JavaScript. See https://atom.io/.

>> At the very least, it frees us from dependence on a glibc feature
>> that is doomed to disappear.
>
> No one is suggesting to stay with unexec.  The disagreement is about
> where to go to get rid of it.
>
>> If later the work becomes difficult to support, or if Daniel disappears, won't
>> we just be returning to the same position as now? If that happeans, we can
>> take what we've learned and try another approach, such as speeding up .elc
>> loading. Or, if someone comes up with a better alternative in the meantime, we
>> can just switch to it.
>
> I don't think this is practical.  We cannot go back, certainly not
> when unexec requires the expertise it does.  No one will agree to go
> back.  Like with unexec, the only way the portable dumper will be
> thrown away is if/when it turns out it is an obstacle to Emacs
> development, by which time it will be too late.

Unexec will keep working as well as it does today.


>> Accepting this patch doesn't mean we don't try the .elc speedup idea. The only
>> thing we have to make sure is that it doesn't unduly increase the difficulty
>> of adding new Lisp objects.
>
> Accepting this patch means we believe in this direction, consider it a
> perspective one, enough so that we are willing to invest a non-trivial
> amount of energy into making it happen, fix the fallout, debug the
> results, etc.  We are talking about at least one major release, maybe
> more, and definitely several minor releases.  IOW, we are talking
> several years.  I don't think we should gamble with such high stakes,
> not unless we sure we want to go there, because this is the best
> alternative of ditching unexec.

The stakes aren't high. As johnw wrote, unexec will keep working as well
as it does today, and nobody is stopping anyone else making 

Re: Preview: portable dumper

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> Accepting this patch means we believe in this direction, consider it a
> perspective one, enough so that we are willing to invest a non-trivial
> amount of energy into making it happen, fix the fallout, debug the results,
> etc. We are talking about at least one major release, maybe more, and
> definitely several minor releases. IOW, we are talking several years. I
> don't think we should gamble with such high stakes, not unless we sure we
> want to go there, because this is the best alternative of [...]

If I didn't know you were talking about pdumper vs. unexec here, I'd swear
this above paragraph was expressing my own concerns about merging Tromey's
concurrency branch. My issues then were exactly the same: very few people who
understand the C implementation, and a multi-year commitment down a road that
may not be the right solution. Yet you agreed that merging it in was fine,
since we could gain experience and at least there's someone who actually did
the work.

Why is pdumper not the same scenario? We have commitment from someone willing
to work on it, he's even willing to write documentation, and it will give us
experience in an area we're free to change at any time, because it doesn't
expose APIs to the end user.

I say we try it, while keeping unexec in place, until it becomes more obvious
that we shouldn't. If we wait too long for the "best alternative", it might
the same as what happened to concurrency: it never happens, because there's
always a concern about the unexpected-down-the-road.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: eggert@cs.ucla.edu,  dancol@dancol.org,  emacs-devel@gnu.org
> Date: Mon, 28 Nov 2016 14:18:58 -0800
> 
> > Accepting this patch means we believe in this direction, consider it a
> > perspective one, enough so that we are willing to invest a non-trivial
> > amount of energy into making it happen, fix the fallout, debug the results,
> > etc. We are talking about at least one major release, maybe more, and
> > definitely several minor releases. IOW, we are talking several years. I
> > don't think we should gamble with such high stakes, not unless we sure we
> > want to go there, because this is the best alternative of [...]
> 
> If I didn't know you were talking about pdumper vs. unexec here, I'd swear
> this above paragraph was expressing my own concerns about merging Tromey's
> concurrency branch.

Superficially, perhaps.  But as soon as you look at the details, there
are two important differences.

First, the only piece of code in the concurrency patch that is not
entirely trivial is a 60-line function (30% of whose line count is
comments and ifdef's).  That's all there is to understand there.  The
rest is extremely simple, almost trivial: functions that directly map
into the corresponding system threads APIs, or 2-liners for creation,
checking, and otherwise simple management of the few new object types.
By contrast, the proposed patch, even in its initial version, weighs
in at 3200 lines in its core part, and almost none of it is trivial or
simple.

What's more, it comes with the need to make important and non-trivial
decisions wrt which objects and data structures need to be dumped, and
how, something that requires intimate knowledge of the internals and
their use patterns, both at dump time and in interactive sessions.
These decisions will need to be revisited when the related objects are
added or modified, which is the part that averts me the most, because
it means extending the Lisp interpreter, which is already not easy,
will have become even harder.

The other important difference is that threads are an opt-in feature:
if you don't start any threads, Emacs with threads still works as it
did before, and you can just ignore anything related to threads.  IOW,
threads are a feature we offer to Lisp programs that want to use it.
By contrast, dumping code is a central and mandatory part of building
Emacs and of its startup process.  It cannot be ignored or made
optional.

Re: Preview: portable dumper

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> By contrast, dumping code is a central and mandatory part of building Emacs
> and of its startup process. It cannot be ignored or made optional.

Could we ask Daniel to include his code under an #ifdef, so until it's proven
we can always disable it, or back out the changes easily if need be? I'd like
to entertain the experiment, if we can.

I spent a good chunk of my career working on linkers and loaders, so I'm aware
of some of the complexities we're inviting in. My biggest fear is that things
like this always seem simple initially -- but the devil is in the details.
However, if we maintain a way to trivially back out, I don't see how it can
hurt to give Daniel his room to play. Maybe it could turn out better than we
think?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: eggert@cs.ucla.edu,  dancol@dancol.org,  emacs-devel@gnu.org
> Date: Tue, 29 Nov 2016 11:11:15 -0800
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > By contrast, dumping code is a central and mandatory part of building Emacs
> > and of its startup process. It cannot be ignored or made optional.
> 
> Could we ask Daniel to include his code under an #ifdef, so until it's proven
> we can always disable it, or back out the changes easily if need be?

What would that solve?  Don't we still need to review the patches,
compile the code once in a while, debug it as needed, etc.?

> I'd like to entertain the experiment, if we can.

Fine, but why do you need me for that?  What would be my role in that
experiment?

Re: Preview: portable dumper

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> Could we ask Daniel to include his code under an #ifdef, so until it's proven
>> we can always disable it, or back out the changes easily if need be?

> What would that solve? Don't we still need to review the patches, compile
> the code once in a while, debug it as needed, etc.?

It keeps the portable dumper fenced off. We'd compile it in by default, but it
helps future readers, for a while, by clearly demarcating when special
handling is needed for the sake of that dumper.

>> I'd like to entertain the experiment, if we can.

> Fine, but why do you need me for that? What would be my role in that
> experiment?

I guess just say OK at this point, and help with the review if you're
inclined? Daniel will handle the work of integration and documentation.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Nov 29 2016, John Wiegley wrote:
>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Could we ask Daniel to include his code under an #ifdef, so until it's proven
>>> we can always disable it, or back out the changes easily if need be?
>
>> What would that solve? Don't we still need to review the patches, compile
>> the code once in a while, debug it as needed, etc.?
>
> It keeps the portable dumper fenced off. We'd compile it in by default, but it
> helps future readers, for a while, by clearly demarcating when special
> handling is needed for the sake of that dumper.

FWIW, as a logistical matter, it's not really practical to literally
#ifdef out the pdumper integration into the rest of Emacs. We can,
however, make the pdumper constructs compile down to nothing if you opt
not to enable pdumper in the build configuration. I specifically
designed the pdumper.h API to make this arrangement easy and natural.

As a matter of general taste, I _much_ prefer this style:

    #ifdef HAVE_FOO
        void foo_thing (void);
    #else
    # define foo_thing() ((void)0)
    #endif

        void
        function1()
        {
          ...
          foo_thing();
          ...
        }

        void function2()
        {
          ...
          foo_thing();
          ...
        }

        void function3()
        {
          ...
          foo_thing();
          ...
        }

to this style:

        void
        function1()
        {
          ...
    #ifdef HAVE_FOO
          foo_thing();
    #endif
          ...
        }

        void function2()
        {
          ...
    #ifdef HAVE_FOO
          foo_thing();
    #endif
          ...
        }

        void function3()
        {
          ...
    #ifdef HAVE_FOO
          foo_thing();
    #endif
          ...
        }


>
>>> I'd like to entertain the experiment, if we can.
>
>> Fine, but why do you need me for that? What would be my role in that
>> experiment?
>
> I guess just say OK at this point, and help with the review if you're
> inclined? Daniel will handle the work of integration and documentation.

Re: Preview: portable dumper

[John Wiegley]

>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:

DC> As a matter of general taste, I _much_ prefer this style:

Yes, I agree.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

>
> As a matter of general taste, I _much_ prefer this style:
>
>     #ifdef HAVE_FOO
>         void foo_thing (void);
>     #else
>     # define foo_thing() ((void)0)
>     #endif
>
>         void
>         function1()
>         {
>           ...
>           foo_thing();
>           ...
>         }
>
> to this style:
>
>         void
>         function1()
>         {
>           ...
>     #ifdef HAVE_FOO
>           foo_thing();
>     #endif
>           ...
>         }
>
>         void function2()
>         {
>           ...
>     #ifdef HAVE_FOO
>           foo_thing();
>     #endif
>           ...
>         }
>


+1

ifdefs should be as much out of the way of the code as possible in my
honest opinion.

Philippe

[-- Attachment #2: Type: text/html, Size: 1226 bytes --]

Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Nov 29 2016, Eli Zaretskii wrote:
>> From: John Wiegley <jwiegley@gmail.com>
>> Cc: eggert@cs.ucla.edu,  dancol@dancol.org,  emacs-devel@gnu.org
>> Date: Mon, 28 Nov 2016 14:18:58 -0800
>> 
>> > Accepting this patch means we believe in this direction, consider it a
>> > perspective one, enough so that we are willing to invest a non-trivial
>> > amount of energy into making it happen, fix the fallout, debug the results,
>> > etc. We are talking about at least one major release, maybe more, and
>> > definitely several minor releases. IOW, we are talking several years. I
>> > don't think we should gamble with such high stakes, not unless we sure we
>> > want to go there, because this is the best alternative of [...]
>> 
>> If I didn't know you were talking about pdumper vs. unexec here, I'd swear
>> this above paragraph was expressing my own concerns about merging Tromey's
>> concurrency branch.
>
> Superficially, perhaps.  But as soon as you look at the details, there
> are two important differences.
>
> First, the only piece of code in the concurrency patch that is not
> entirely trivial is a 60-line function (30% of whose line count is
> comments and ifdef's).

You're the only person I've ever seen use words like "entirely trivial"
to describe the addition of shared-memory concurrency to a legacy
single-threaded system. Even if the code for the threads system looks
simple, the concepts involved are anything but.

By contrast, the portable dumper is about marshalling, which I think is
comparatively simpler.

> That's all there is to understand there.  The
> rest is extremely simple, almost trivial: functions that directly map
> into the corresponding system threads APIs, or 2-liners for creation,
> checking, and otherwise simple management of the few new object types.
> By contrast, the proposed patch, even in its initial version, weighs
> in at 3200 lines in its core part, and almost none of it is trivial or
> simple.
>
> What's more, it comes with the need to make important and non-trivial
> decisions wrt which objects and data structures need to be dumped, and
> how, something that requires intimate knowledge of the internals and
> their use patterns, both at dump time and in interactive sessions.
> These decisions will need to be revisited when the related objects are
> added or modified, which is the part that averts me the most, because
> it means extending the Lisp interpreter, which is already not easy,
> will have become even harder.

I don't see it as a problem that the internals of Emacs need to have
knowledge of the internals of Emacs.  What matters is whether adding new
object types, which is already a very rare exercise, is made
prohibitively hard. Keep in mind that any programmer who adds a new
object type needs to *also* update the garbage collector, and GC and the
portable dumper require exactly the same kind of object description.

(I looked into actually unifying the GC and pdumper, but I don't think
it's possible without introducing significant new abstractions. It's not
worth it for something that happens so rarely as modification to core
Emacs data type representations.)

If you want to follow this reasoning to its logical conclusion, we
should switch to Boehm GC.

> The other important difference is that threads are an opt-in feature:
> if you don't start any threads, Emacs with threads still works as it
> did before, and you can just ignore anything related to threads.  IOW,
> threads are a feature we offer to Lisp programs that want to use it.
> By contrast, dumping code is a central and mandatory part of building
> Emacs and of its startup process.  It cannot be ignored or made
> optional.

The portable dumper is also optional: you can always use the legacy
unexec code, which isn't going away, or just rely on loadup (and the
hypothetical ultra-fast lread we've discussed).

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Thanks for your efforts, but I think this is a wrong direction, that
  > will eventually get us into the same problem we have with unexec: the
  > need to know and depend on intimate details of relocations, memory
  > allocation, etc.

Maintaining the portable dumper will be far simpler than maintaining
unexec.  Unexec has to write a file in whatever format the system uses
for executables.  If that is ELF format, it is very complicated,
and it wasn't designed for this purpose.  It is necessary to copy
the material from the temacs executable, and modify it along the way.

All those problems will be absent for the portable dumper, which
will make its maintenance much easier.

		      On top of that, adding Lisp objects will now require
  > writing its dumper back-end, so this will be a constant maintenance
  > burden of the kind that only a few of us can bear.

Yes, but how often do we make such changes?  Once every few years,
I think.  And it won't be a big difficulty to do so, if the code of
the dumper is clean.  I am sure it will be a lot cleaner than the code
of unexec.

  > Making the initial load of preloaded Lisp files (most probably, a
  > single large file) fast enough to allow us to dump the dumping phase
  > altogether is a much more perspective direction.

I think this has its own difficulty -- the need to represent everything
in Lisp syntax.

I don't know a priori which one will be cleaner.  It would be good to
implement this alternative so we can compare them for real rather than
just in our imaginations.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: dancol@dancol.org, emacs-devel@gnu.org
> Date: Tue, 29 Nov 2016 11:55:16 -0500
> 
> Maintaining the portable dumper will be far simpler than maintaining
> unexec.  Unexec has to write a file in whatever format the system uses
> for executables.  If that is ELF format, it is very complicated,
> and it wasn't designed for this purpose.  It is necessary to copy
> the material from the temacs executable, and modify it along the way.
> 
> All those problems will be absent for the portable dumper, which
> will make its maintenance much easier.

Easier: yes.  Easy enough: no, not IMO.  Not for most of the Emacs
contributors we have and will have in the near future.

The number of people aboard who can matter-of-factly hack the Emacs
internals on the C level is consistently going down, and is already so
small they can be counted on one hand.  We must make Emacs depend less
on people from this small and diminishing group, if we want the
development pace increased or even kept at its current level.  To me,
that means keep as many features out of low-level C, and limit futzing
with C-level internals of Lisp objects and the Lisp interpreter to the
absolute minimum.  IOW, any feature that can have an alternative
implementation in Lisp should not be done in C, even if the Lisp
implementation will be slightly slower.

> 		      On top of that, adding Lisp objects will now require
>   > writing its dumper back-end, so this will be a constant maintenance
>   > burden of the kind that only a few of us can bear.
> 
> Yes, but how often do we make such changes?  Once every few years,
> I think.

More like once a year, maybe more.

> And it won't be a big difficulty to do so, if the code of the dumper
> is clean.

For you and me, and for a few others, sure.  But Emacs cannot be
developed by a handful of people anymore.  We already fall behind with
frequency of releases, with the speed of delivering new features, with
analyzing and fixing bugs reported to the tracker, etc.  If we don't
attract more core developers, we will never get out of this vicious
cycle.  And bringing C-level core developers on board is a very slow,
painful, and unreliable process, because C programmers are a dying
breed, and Emacs internals are not for the faint at heart.

> I am sure it will be a lot cleaner than the code of unexec.

I'm sure, too, but that's not the issue.

>   > Making the initial load of preloaded Lisp files (most probably, a
>   > single large file) fast enough to allow us to dump the dumping phase
>   > altogether is a much more perspective direction.
> 
> I think this has its own difficulty -- the need to represent everything
> in Lisp syntax.

I don't think I follow: what we preload in loadup already has Lisp
syntax.

Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Nov 29 2016, Eli Zaretskii wrote:
>> From: Richard Stallman <rms@gnu.org>
>> CC: dancol@dancol.org, emacs-devel@gnu.org
>> Date: Tue, 29 Nov 2016 11:55:16 -0500
>> 
>> Maintaining the portable dumper will be far simpler than maintaining
>> unexec.  Unexec has to write a file in whatever format the system uses
>> for executables.  If that is ELF format, it is very complicated,
>> and it wasn't designed for this purpose.  It is necessary to copy
>> the material from the temacs executable, and modify it along the way.
>> 
>> All those problems will be absent for the portable dumper, which
>> will make its maintenance much easier.
>
> Easier: yes.  Easy enough: no, not IMO.  Not for most of the Emacs
> contributors we have and will have in the near future.

Who do you imagine maintaining Emacs? CS 101 students? We have to assume
a certain baseline level of skill on the part of our future
colleagues. I don't want to dumb down our implementation for the sake of
people we don't know and who, for all we know, may never
actually appear.

> The number of people aboard who can matter-of-factly hack the Emacs
> internals on the C level is consistently going down, and is already so
> small they can be counted on one hand.  We must make Emacs depend less
> on people from this small and diminishing group, if we want the
> development pace increased or even kept at its current level.

I've discussed before why I think this argument is ill-founded. We
already have examples over the past few years of people who've
familiarized themselves with Emacs internals in order to add a feature
they want. If there's a need, people will familiarize themselves with
the editor.

Please, present some evidence that the reason we've seen a decline in
core contributions due to the complexity of the current codebase. I
don't agree that we'd see more contributors appear if we simplified our
algorithms. On the contrary, making Emacs slower will drive users away.

The solution to the problem you're raising is better internals
documentation, not oversimplification.

> To me,
> that means keep as many features out of low-level C, and limit futzing
> with C-level internals of Lisp objects and the Lisp interpreter to the
> absolute minimum.  IOW, any feature that can have an alternative
> implementation in Lisp should not be done in C, even if the Lisp
> implementation will be slightly slower.

I agree that what *can* be done in Lisp should be done in Lisp. We're
talking about a facility that doesn't fall under this general
guideline. It's the job of the C core to provide to Lisp facilities
necessary to implement higher-level features.  Fast startup is one of
these platform facilities, one very common in other Lisp systems.

>> 		      On top of that, adding Lisp objects will now require
>>   > writing its dumper back-end, so this will be a constant maintenance
>>   > burden of the kind that only a few of us can bear.
>> 
>> Yes, but how often do we make such changes?  Once every few years,
>> I think.
>
> More like once a year, maybe more.

Like what? The last two new types in core have been finalizers and the
module thing, both of which have been trivial Lisp_Misc types.

By the way: it's very easy to add support for new object types
to pdumper. Have you read the code?

There are certain types of object that are especially easy to support:

Do you have a pseudovector with no non-Lisp allocations? Adding support
to pdumper for this object type is one line of code.

Do you have a pseudovector that you want to come back from dumps "dead",
like frames? Adding support to pdumper for this object type is one line
of code.

How many new objects types are we going to add that don't fall under one
of these two categories? You're optimizing for hypotheticals at the
expense of the concrete present.

>
>> And it won't be a big difficulty to do so, if the code of the dumper
>> is clean.
>
> For you and me, and for a few others, sure.  But Emacs cannot be
> developed by a handful of people anymore.  We already fall behind with
> frequency of releases, with the speed of delivering new features, with
> analyzing and fixing bugs reported to the tracker, etc.  If we don't
> attract more core developers, we will never get out of this vicious
> cycle.  And bringing C-level core developers on board is a very slow,
> painful, and unreliable process, because C programmers are a dying
> breed, and Emacs internals are not for the faint at heart.

So move more functionality to Lisp. We've done a good job of doing that
lately. We're talking about a special case. You can't move eval to
Lisp either.

If you want to pursue this path, how about moving redisplay to Lisp? You
know more about redisplay than anyone else around, I think, and
redisplay is a much hairier blob of C code than the portable dumper is.

>> I am sure it will be a lot cleaner than the code of unexec.
>
> I'm sure, too, but that's not the issue.
>
>>   > Making the initial load of preloaded Lisp files (most probably, a
>>   > single large file) fast enough to allow us to dump the dumping phase
>>   > altogether is a much more perspective direction.
>> 
>> I think this has its own difficulty -- the need to represent everything
>> in Lisp syntax.
>
> I don't think I follow: what we preload in loadup already has Lisp
> syntax.

Yes, but the data structures that we create via loadup don't all have
easy-to-digest Lisp representations, and creating them is on the same
level of complexity as the portable dumper.

*This* code exists. Where is the hypothetical sufficiently-fast lread?

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Cc: rms@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 29 Nov 2016 11:03:35 -0800
> 
> > Easier: yes.  Easy enough: no, not IMO.  Not for most of the Emacs
> > contributors we have and will have in the near future.
> 
> Who do you imagine maintaining Emacs? CS 101 students?

Just look around, they are all here.

> We have to assume a certain baseline level of skill on the part of
> our future colleagues. I don't want to dumb down our implementation
> for the sake of people we don't know and who, for all we know, may
> never actually appear.

You are insulting most of the people on this list, some of them
long-time and fruitful contributors.  They don't deserve that.

> > The number of people aboard who can matter-of-factly hack the Emacs
> > internals on the C level is consistently going down, and is already so
> > small they can be counted on one hand.  We must make Emacs depend less
> > on people from this small and diminishing group, if we want the
> > development pace increased or even kept at its current level.
> 
> I've discussed before why I think this argument is ill-founded. We
> already have examples over the past few years of people who've
> familiarized themselves with Emacs internals in order to add a feature
> they want. If there's a need, people will familiarize themselves with
> the editor.

I don't think this is accurate, but even if it is, it is better not to
have this bar.

> Please, present some evidence that the reason we've seen a decline in
> core contributions due to the complexity of the current codebase. I
> don't agree that we'd see more contributors appear if we simplified our
> algorithms.

You are missing the point: not complexity and algorithms are the
problem, the implementation language is the problem.

> I agree that what *can* be done in Lisp should be done in Lisp. We're
> talking about a facility that doesn't fall under this general
> guideline.

??? loadup.el is Lisp code.

> >> 		      On top of that, adding Lisp objects will now require
> >>   > writing its dumper back-end, so this will be a constant maintenance
> >>   > burden of the kind that only a few of us can bear.
> >> 
> >> Yes, but how often do we make such changes?  Once every few years,
> >> I think.
> >
> > More like once a year, maybe more.
> 
> Like what? The last two new types in core have been finalizers and the
> module thing, both of which have been trivial Lisp_Misc types.

Trivial to some, much less trivial to others.

> By the way: it's very easy to add support for new object types
> to pdumper. Have you read the code?

Yes, I have read the code.  It's easy for you to add new object types,
because you wrote that code, debugged it, and didn't yet have time to
forget how it works.

It is also easy for me to tweak bidi.c to add features required by
Unicode 6.3 and 8.0.  Does that mean anyone else can "easily" do that?

> If you want to pursue this path, how about moving redisplay to Lisp? You
> know more about redisplay than anyone else around, I think, and
> redisplay is a much hairier blob of C code than the portable dumper is.

If you stop arguing with me so much about each and every issue, then
perhaps I will have time to explore this.  As things are, sorry:
arguing with you takes up all of my free time -- or what's left of it
after triaging and fixing bugs, helping people to implement stuff in
C, something you think is "easy" for everyone past CS101, adding and
fixing documentation, including that one Daniel Colascione failed to
write, and whatnot else that is evidently part of the job.

> >> I think this has its own difficulty -- the need to represent everything
> >> in Lisp syntax.
> >
> > I don't think I follow: what we preload in loadup already has Lisp
> > syntax.
> 
> Yes, but the data structures that we create via loadup don't all have
> easy-to-digest Lisp representations, and creating them is on the same
> level of complexity as the portable dumper.

But it already works.

> *This* code exists. Where is the hypothetical sufficiently-fast lread?

In the works, I hope.  It is my job as a maintainer to encourage that
development.

> > First, the only piece of code in the concurrency patch that is not
> > entirely trivial is a 60-line function (30% of whose line count is
> > comments and ifdef's).
> 
> You're the only person I've ever seen use words like "entirely trivial"
> to describe the addition of shared-memory concurrency to a legacy
> single-threaded system. Even if the code for the threads system looks
> simple, the concepts involved are anything but.

Concurrency in Emacs is for people who already know what that is and
are familiar with the concepts.  _After_ you are familiar with the
concepts, the code is surprisingly simple and straightforward.

> By contrast, the portable dumper is about marshalling, which I think is
> comparatively simpler.

The concepts are simpler, but the implementation is not so simple.
And some stuff is done manually, with the PDUMPER_* macros rules for
whose use are only clear to someone who knows a lot about the involved
objects and their use patterns.

> The portable dumper is also optional: you can always use the legacy
> unexec code, which isn't going away, or just rely on loadup (and the
> hypothetical ultra-fast lread we've discussed).

If that is the direction we want to move, then the efforts invested in
the development of the portable dumper are a complete waste of our
resources.

Re: Preview: portable dumper

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

>> We have to assume a certain baseline level of skill on the part of our
>> future colleagues. I don't want to dumb down our implementation for the
>> sake of people we don't know and who, for all we know, may never actually
>> appear.

EZ> You are insulting most of the people on this list, some of them long-time
EZ> and fruitful contributors. They don't deserve that.

Eli, I don't read any insult here, just Daniel questioning the fear that we'd
code ourselves into oblivion if core allowed any more complexity. I think we
all want more Lisp and less C, I've seen you both say as much. The real point
being debated is whether some additional C wouldn't help us in the interim.

Note that I've seen both statements become true: Complex core that repulsed
new contributors, and new contributors willing to learn anything to help a
project.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Paul Eggert]

On 11/29/2016 10:39 AM, Eli Zaretskii wrote:
> If we don't attract more core developers, we will never get out of this vicious cycle.

Although we certainly could use more core developers, it's quite a 
stretch to say that switching from unexec to a portable undumper will 
scare recruits away. On the contrary, the portable approach is simpler 
and easier for non-experts than what we have now.

> C programmers are a dying breed
I don't agree, as C is an essential component of GNU/Linux and is 
robustly represented in the practical and research programming 
communities. (For what it's worth, C's popularity is ranked #7 in PyPL 
and #2 in TIOBE.) Although it'd be fine to replacing the core Emacs 
interpreter with something better if we can scare up developer resources 
to do that, in the meantime it's OK to assume C expertise for people 
modifying the Emacs interpreter.

Re: Preview: portable dumper

[Eli Zaretskii]

> Cc: dancol@dancol.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 29 Nov 2016 11:13:37 -0800
> 
> On 11/29/2016 10:39 AM, Eli Zaretskii wrote:
> > If we don't attract more core developers, we will never get out of this vicious cycle.
> 
> Although we certainly could use more core developers, it's quite a 
> stretch to say that switching from unexec to a portable undumper will 
> scare recruits away.

Which is why I didn't say that.

> On the contrary, the portable approach is simpler and easier for
> non-experts than what we have now.

I didn't compare the proposed portable dumper with what we have now,
so this is a red herring.

> > C programmers are a dying breed
> I don't agree, as C is an essential component of GNU/Linux and is 
> robustly represented in the practical and research programming 
> communities. (For what it's worth, C's popularity is ranked #7 in PyPL 
> and #2 in TIOBE.)

When did you last try to hire a C or C++ programmer?

As another data point, just look in the logs at the names of people
who make C-level changes, and count them.

I didn't say I like this, but leading a project to be viable in the
future needs to look at the reality, not at our wishful thinking.

Re: Preview: portable dumper

[Paul Eggert]

On 11/29/2016 11:35 AM, Eli Zaretskii wrote:

> I didn't compare the proposed portable dumper with what we have now,
That's our fundamental disagreement, then. I am willing to settle for 
real improvements to Emacs even if they are not ideal. And I think that 
this sort of willingness is more likely to attract contributions from 
developers.

> leading a project to be viable in the
> future needs to look at the reality, not at our wishful thinking
Daniel's proposed change consists of code that works and that runs fast. 
Isn't that reality? And in contrast, isn't the lread-based approach 
wishful thinking?

 > When did you last try to hire a C or C++ programmer?

I don't *hire* them, I *create* them. I currently have 450 undergraduate 
students, and C++ and C are the only programming languages that they 
have all written and debugged code in. (This is not just my decision; it 
comes from our faculty's curriculum.) According to Philip Guo's 2014 
survey, the most popular languages used in teaching computer science in 
top U.S. universities are Python (25-30%), Java (20-25%), C/C++ 
(10-15%), and MATLAB (5-10%). So from what I can see, the C pipeline is 
by no means empty.

Anyway, although C has probably passed its peak and will surely die out 
eventually, it's unlikely for this to be anytime soon. The prospect of 
its demise shouldn't materially affect our choice of Emacs dumping 
technology in the meantime.

Re: Preview: portable dumper

[Eli Zaretskii]

> Cc: rms@gnu.org, dancol@dancol.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 29 Nov 2016 12:54:16 -0800
> 
> Daniel's proposed change consists of code that works and that runs fast. 

I don't think "works and is fast" are the only criteria for accepting
patches, or even the most important ones.

> Isn't that reality? And in contrast, isn't the lread-based approach 
> wishful thinking?

Maybe, maybe not.  I don't understand why we need to grasp the first
straw we see around to get rid of unexec.  What's the rush?

And let me remind you that you didn't like this same idea very much
when I proposed it 2 months ago, see the discussion that starts at

  http://lists.gnu.org/archive/html/bug-gnu-emacs/2016-09/msg00227.html

> from what I can see, the C pipeline is by no means empty.

Well, it somehow becomes all but empty on the other end.  Believe me.
And even if we cannot agree on that, the facts of what we see here are
very clear.

> The prospect of [C]s demise shouldn't materially affect our choice
> of Emacs dumping  technology in the meantime.

I think it would be a grave mistake not to be affected by that.  We
would be burying our heads in the sand if we don't.

Re: Preview: portable dumper

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> Maybe, maybe not. I don't understand why we need to grasp the first straw
EZ> we see around to get rid of unexec. What's the rush?

Eli has a good point here, there really is no rush. Daniel, would you be
willing to continue to incubate these changes in a branch, while we also
consider and explore alternatives?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Daniel Colascione]

On Wed, Nov 30 2016, John Wiegley wrote:
>>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>
> EZ> Maybe, maybe not. I don't understand why we need to grasp the first straw
> EZ> we see around to get rid of unexec. What's the rush?
>
> Eli has a good point here, there really is no rush. Daniel, would you be
> willing to continue to incubate these changes in a branch, while we also
> consider and explore alternatives?

No. This code isn't disruptive to the Emacs core if it's used, so
there's no reason to keep it out of master.  Experience with the
concurrency branch tells me that interesting work in some feature branch
ghetto never actually gets merged.

I maintain that Eli's concerns about maintainability are unfounded.

Re: Preview: portable dumper

[John Wiegley]

>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:

DC> No. This code isn't disruptive to the Emacs core if it's used, so there's
DC> no reason to keep it out of master. Experience with the concurrency branch
DC> tells me that interesting work in some feature branch ghetto never
DC> actually gets merged.

You've only tested this on one platform so far, yes? That alone is reason not
to merge it immediately into master. Also, I'd like to see documentation
before it goes in, because that's essential to its maintainability.

Further, I don't like hearing ultimatums, on either side, over what amounts to
an efficiency issue. This matter has already shown that it deserves a little
cooling off and further thought. Call it a ghetto if you will, but you've not
convinced me that incubating code in branches is a bad idea.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Paul Eggert]

On 11/30/2016 01:03 PM, John Wiegley wrote:
> I don't like hearing ultimatums, on either side, over what amounts to
> an efficiency issue.

Yes, I've heard too many ultimatums too.

However, this is not simply an efficiency issue. Emacs currently does 
not run on FreeBSD arm64 because of the unexec problem (Bug#24892), and 
it's quite likely that the set of platforms where Emacs does not run 
will grow as platforms change memory management techniques. We should 
not reject a working fix merely because we hypothesize that there might 
be a better improvement that someone might implement later.

Re: Preview: portable dumper

[John Wiegley]

>>>>> "PE" == Paul Eggert <eggert@cs.ucla.edu> writes:

PE> However, this is not simply an efficiency issue. Emacs currently does not
PE> run on FreeBSD arm64 because of the unexec problem (Bug#24892), and it's
PE> quite likely that the set of platforms where Emacs does not run will grow
PE> as platforms change memory management techniques. We should not reject a
PE> working fix merely because we hypothesize that there might be a better
PE> improvement that someone might implement later.

That's a good point, Paul, I hadn't thought of that. And the Docker issue,
apparently.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 30 Nov 2016 13:06:11 -0800
> 
> However, this is not simply an efficiency issue. Emacs currently does 
> not run on FreeBSD arm64 because of the unexec problem (Bug#24892)

Let's get our facts straight, shall we?  First, Emacs never worked on
that platform, and second, the current reason is that they removed a
certain library function, which happens to be called by unexec.
Removal of a library function can break any part of Emacs.

Re: Preview: portable dumper

[Paul Eggert]

Eli Zaretskii wrote:

> Emacs never worked on that platform

Sure it did. Emacs worked until FreeBSD 11, when the arm64 port modernized its 
memory-management implementation. If Glibc does something similar, it would be a 
mistake to say "that's OK, Emacs never worked on that platform", as this might 
give non-experts the misimpression that Emacs never worked on GNU/Linux.

> they removed a certain library function, which happens to be called by unexec.

This was no accident. They removed it because it was producing bogus values, and 
it didn't reflect how FreeBSD actually allocates memory. Whether the function 
worked for Emacs at all was apparently a matter of luck.

I would rather not tell FreeBSD/arm64 users that they can't run Emacs any more 
because we can't be bothered to install a patch that's been contributed, not 
even a patch that's compiled by default only on FreeBSD/arm64.

I sympathize with Daniel about the politics of this. This is not the sort of 
change that we'd ordinarily create a new branch for, and the way this is being 
handled will likely discourage further contributions.

Re: Preview: portable dumper

[Eli Zaretskii]

> Cc: dancol@dancol.org, rms@gnu.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 1 Dec 2016 01:16:17 -0800
> 
>     Emacs never worked on that platform
> 
> Sure it did. Emacs worked until FreeBSD 11, when the arm64 port modernized its memory-management implementation.

From https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24892#104

  Ed Maste wrote:
  > arm64 support was first available in a release in FreeBSD 11.0, without sbrk, and sbrk never existed on the stable/11 branch.

  Thanks, I didn't know that. So Emacs has never worked in this environment.

Doesn't that mean Emacs on arm64 was never available before FreeBSD
11, and sbrk never existed on FreeBSD/arm64?

>     they removed a certain library function, which happens to be called by unexec.
> 
> This was no accident. They removed it because it was producing bogus values, and it didn't reflect how FreeBSD actually allocates memory.

And they are thinking on bringing it back, see

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24892#92

Re: Preview: portable dumper

[Daniel Colascione]

On 12/01/2016 09:26 AM, Eli Zaretskii wrote:
>> Cc: dancol@dancol.org, rms@gnu.org, emacs-devel@gnu.org
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Date: Thu, 1 Dec 2016 01:16:17 -0800
>>
>>     Emacs never worked on that platform
>>
>> Sure it did. Emacs worked until FreeBSD 11, when the arm64 port modernized its memory-management implementation.
>
> From https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24892#104
>
>   Ed Maste wrote:
>   > arm64 support was first available in a release in FreeBSD 11.0, without sbrk, and sbrk never existed on the stable/11 branch.
>
>   Thanks, I didn't know that. So Emacs has never worked in this environment.
>
> Doesn't that mean Emacs on arm64 was never available before FreeBSD
> 11, and sbrk never existed on FreeBSD/arm64?
>
>>     they removed a certain library function, which happens to be called by unexec.
>>
>> This was no accident. They removed it because it was producing bogus values, and it didn't reflect how FreeBSD actually allocates memory.
>
> And they are thinking on bringing it back, see

They shouldn't bring it back. Reserving a single region in a process as 
"the heap" and doling out parts of it via mmap is an inflexible 
allocation policy. mmap ought to be the only way to get memory from the 
system.

Re: Preview: portable dumper

[Paul Eggert]

On 12/01/2016 09:26 AM, Eli Zaretskii wrote:
>> >This was no accident. They removed it because it was producing bogus values, and it didn't reflect how FreeBSD actually allocates memory.
> And they are thinking on bringing it back, see
>
>    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24892#92

That thread discusses their implementing sbrk as always returning 0 or 
something trivial like that, purely to pacify broken applications that 
use sbrk (0) to estimate memory usage and fail to build if sbrk does not 
exist. Such a workaround obviously won't get Emacs running. In contrast, 
Daniel's proposed patch should work.

Re: Preview: portable dumper

[Daniel Colascione]

On Wed, Nov 30 2016, John Wiegley wrote:
>>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:
>
> DC> No. This code isn't disruptive to the Emacs core if it's used, so there's
> DC> no reason to keep it out of master. Experience with the concurrency branch
> DC> tells me that interesting work in some feature branch ghetto never
> DC> actually gets merged.
>
> You've only tested this on one platform so far, yes? That alone is reason not
> to merge it immediately into master. Also, I'd like to see documentation
> before it goes in, because that's essential to its maintainability.
>
> Further, I don't like hearing ultimatums, on either side, over what amounts to
> an efficiency issue. This matter has already shown that it deserves a little
> cooling off and further thought. Call it a ghetto if you will, but you've not
> convinced me that incubating code in branches is a bad idea.

It's not an ultimatum: I'm merely being up-front about the that work I
am willing to do and the work that I am not willing to do. I am not
willing to work on this code in a branch, because if I do, it is very
likely that this code will never be merged --- just like the concurrency
branch.  If we put this thing on a branch, it will be ignored.  At best,
every few years, someone will bring it the portable dumper --- just like
the concurrency branch --- and ask why we don't use it.  At those times,
we'll repeat the same argument we're having now and and never move
forward with merging the increasingly bitrotted code.

What will change in the future, when we talk about merging the
hypothetical branch?  Nobody has expressed concern about the technical
quality of this feature. Yes, of course the final version will have more
documentation --- but that's something we can do right now, pre-commit,
not something we need to incubate on a branch.  It's not as if we think
this thing might destabilize Emacs, especially if (at first) it's not
actually compiled into Emacs by default.

Putting this code on a branch is burying it.  Do you want to reject code
that *exists* and that solves real problems on the basis of one
developer's philosophical objections and preference for code that does
not exist, that nobody has signed up to write, and that I insist will
never perform adequately?  How disappointed do you want to make people
who've written working code that solves real problems?

Re: Preview: portable dumper

[John Wiegley]

>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:

DC> It's not an ultimatum: I'm merely being up-front about the that work I am
DC> willing to do and the work that I am not willing to do. I am not willing
DC> to work on this code in a branch, because if I do, it is very likely that
DC> this code will never be merged --- just like the concurrency branch.

BTW, the concurrency branch has the green light. If anyone is willing to work
to resolve its remaining issues, they can merge it to master once ready.

DC> Putting this code on a branch is burying it. Do you want to reject code
DC> that *exists* and that solves real problems on the basis of one
DC> developer's philosophical objections and preference for code that does not
DC> exist, that nobody has signed up to write, and that I insist will never
DC> perform adequately? How disappointed do you want to make people who've
DC> written working code that solves real problems?

Until you're code is tested on more platforms, it needs to live someplace
other than master. That's what branches are for.

I agree with Eli that it doesn't need to be merged *today*. However, I also
agree with RMS, and you, that at least it's a working solution (to some
extent), so if we can't think of something better during the time you're
ironing the wrinkles out, it ought to go in until we do.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Daniel Colascione]

On Wed, Nov 30 2016, John Wiegley wrote:
>>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:
>
> DC> It's not an ultimatum: I'm merely being up-front about the that work I am
> DC> willing to do and the work that I am not willing to do. I am not willing
> DC> to work on this code in a branch, because if I do, it is very likely that
> DC> this code will never be merged --- just like the concurrency branch.
>
> BTW, the concurrency branch has the green light. If anyone is willing to work
> to resolve its remaining issues, they can merge it to master once ready.
>
> DC> Putting this code on a branch is burying it. Do you want to reject code
> DC> that *exists* and that solves real problems on the basis of one
> DC> developer's philosophical objections and preference for code that does not
> DC> exist, that nobody has signed up to write, and that I insist will never
> DC> perform adequately? How disappointed do you want to make people who've
> DC> written working code that solves real problems?
>
> Until you're code is tested on more platforms, it needs to live someplace
> other than master. That's what branches are for.

I'll test it on the systems I can before I commit.  We can enable
compilation once on individual systems once we do more thorough testing.
A branch is definitely not the right approach here, because you haven't
established any firm acceptance criteria, and what criteria you have
mentioned (write some documentation, test on more configurations) are
things I can do before the patch lands at all.

The Cairo code doesn't work at all and *that's* in mainline.
That's okay, because it's not enabled by default.  Branches are for
experimental features that can't coexist with regular Emacs use
and development.  The portable dumper is not such a feature.

Re: Preview: portable dumper

[John Wiegley]

>>>>> "DC" == Daniel Colascione <dancol@dancol.org> writes:

DC> The Cairo code doesn't work at all and *that's* in mainline. That's okay,
DC> because it's not enabled by default. Branches are for experimental
DC> features that can't coexist with regular Emacs use and development. The
DC> portable dumper is not such a feature.

You do have a point there. I'll give Eli some more time to think on this, and
then we'll get back to you. Thanks for the patience.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Paul Eggert]

On 11/30/2016 01:50 PM, Daniel Colascione wrote:
> I'll test it on the systems I can before I commit.

Once you get it working I can volunteer to test it on a few unusual 
systems (e.g., Solaris 10 sparc32, AIX 7 POWER64); this shouldn't take 
long if it's not a holiday. We should also test a 32-bit platform built 
--with-wide-int.

Re: Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

I can test the wide int, and help on exotic systems will be much appreciated

On November 30, 2016 5:37:46 PM PST, Paul Eggert <eggert@cs.ucla.edu> wrote:
>On 11/30/2016 01:50 PM, Daniel Colascione wrote:
>> I'll test it on the systems I can before I commit.
>
>Once you get it working I can volunteer to test it on a few unusual 
>systems (e.g., Solaris 10 sparc32, AIX 7 POWER64); this shouldn't take 
>long if it's not a holiday. We should also test a 32-bit platform built
>
>--with-wide-int.

[-- Attachment #2: Type: text/html, Size: 916 bytes --]

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Wed, 30 Nov 2016 13:50:33 -0800
> 
> > Until you're code is tested on more platforms, it needs to live someplace
> > other than master. That's what branches are for.
> 
> I'll test it on the systems I can before I commit.  We can enable
> compilation once on individual systems once we do more thorough testing.
> A branch is definitely not the right approach here, because you haven't
> established any firm acceptance criteria, and what criteria you have
> mentioned (write some documentation, test on more configurations) are
> things I can do before the patch lands at all.

Having new features on a branch is our standard development practice.
It allows the interested people to try the feature in situations no
single person can possibly create, even if that person has access to
several platforms, because usage patterns differ a lot.

We can establish pass criteria, if you like, but the only criteria
until now were "if enough time passed and no one complained, it's
ready".

In this case, we will probably also consider alternative approaches if
they become mature.

> The Cairo code doesn't work at all and *that's* in mainline.
> That's okay, because it's not enabled by default.

The Cairo code works, it just has some redisplay bugs that no one was
able to fix.  It was disabled post-factum, when we have learned about
those problems, unfortunately after its developer has departed.

> Branches are for experimental features that can't coexist with
> regular Emacs use and development.

This is simply not true.

Re: Preview: portable dumper

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> Having new features on a branch is our standard development practice. It
EZ> allows the interested people to try the feature in situations no single
EZ> person can possibly create, even if that person has access to several
EZ> platforms, because usage patterns differ a lot.

EZ> We can establish pass criteria, if you like, but the only criteria until
EZ> now were "if enough time passed and no one complained, it's ready".

EZ> In this case, we will probably also consider alternative approaches if
EZ> they become mature.

I agree with Eli here, this is what branches are for. Daniel, if you think
they have cultural baggage that makes them distasteful, that's a different
problem. Branches are a perfectly acceptable tool, and they give us an easy
way to assess progress as the work develops.

Please submit your code to a branch, and we'll ask everyone who's capable to
begin testing on as many platforms as we can find. Once enough time has passed
and no one has complained, it can move to master.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Daniel Colascione]

On November 30, 2016 8:10:21 PM PST, John Wiegley <jwiegley@gmail.com> wrote:
>>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>
>EZ> Having new features on a branch is our standard development
>practice. It
>EZ> allows the interested people to try the feature in situations no
>single
>EZ> person can possibly create, even if that person has access to
>several
>EZ> platforms, because usage patterns differ a lot.
>
>EZ> We can establish pass criteria, if you like, but the only criteria
>until
>EZ> now were "if enough time passed and no one complained, it's ready".
>
>EZ> In this case, we will probably also consider alternative approaches
>if
>EZ> they become mature.
>
>I agree with Eli here, this is what branches are for. Daniel, if you
>think
>they have cultural baggage that makes them distasteful, that's a
>different
>problem. Branches are a perfectly acceptable tool, and they give us an
>easy
>way to assess progress as the work develops.
>
>Please submit your code to a branch, and we'll ask everyone who's
>capable to
>begin testing on as many platforms as we can find. Once enough time has
>passed
>and no one has complained, it can move to master.

If that's your decision, I'm going to fork.

Re: Preview: portable dumper

[John Wiegley]

>>>>> Daniel Colascione <dancol@dancol.org> writes:

> If that's your decision, I'm going to fork.

I'm quite sorry to hear it, but if branches are unacceptable, then such is
your right as a free software developer.

Just so you know, I'd hope to see your patches on master within six months, if
we can find some willing testers (and one has already stepped forward). The
only way your changes *wouldn't* land is if a better alternative comes along
in the meantime -- which, of course, would mean we all win.

I don't suppose I can encourage you to maintain your fork within the Emacs.git
repository? There this thing, rhymes with "blanch", where you could work on
your version in tandem, even merge changes from us every once in a while... :)

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Daniel Colascione]

On 11/30/2016 08:49 PM, John Wiegley wrote:
>>>>>> Daniel Colascione <dancol@dancol.org> writes:
>
>> If that's your decision, I'm going to fork.
>
> I'm quite sorry to hear it, but if branches are unacceptable, then such is
> your right as a free software developer.

Branches in general are fine. Relegating this specific feature to a 
branch feels political.

I would very much prefer to continue improving GNU Emacs. I'm a longtime 
fan of the GNU Project. But I feel like the current gatekeepers are 
standing in the way of progress. We can't even make the conservative GC 
safe in the face of foreseeable compiler optimizations. This isn't the 
first time that I've gotten a flat "no" in response to a good idea. If I 
do fork, I can incorporate many more kinds of experimental work. One of 
the first things I'd consider would be an LLVM-based tracing JIT.

Let me try one last time to explain my position: this dumping code will 
make no Emacs no worse. It cannot possible destabilize Emacs when it is 
compiled out of Emacs at configure-time. I specifically designed this 
feature to be optional. I am perfectly happy merging my code in such a 
way that it is not compiled into Emacs by default, and I am perfectly 
happy waiting until we've gotten sufficient testing before enabling the 
portable dumper.

In the miraculous case that someone contributes an lread optimization 
that allows that the big-ELC approach to perform as well as a memory 
image, I will happily back out of my code.

But because this code is harmless when compiled away and because we can 
back it out without problems, I see the requirement that we put this 
code into a branch as a way to kill the feature. If this code were more 
fundamentally destabilizing, I would feel different --- but as it is, 
since this code is *not* fundamentally destabilizing, I don't see a 
technical case for a feature branch.

I am perfectly happy with a six month or longer timetable for enabling 
this code at ./configure time. I expected as much. The XEmacs experience 
with their portable dumper (which I have not read) was similar. I 
specifically designed this code to be compatible with the current 
implementation and to allow for a gradual transition.

Is anyone else signing up to make Emacs work better on modern systems?

> Just so you know, I'd hope to see your patches on master within six months,

What specific conditions need to be met before merging? Why can't I meet 
these requirements before landing the diff? This code will never meet 
the "nobody has complained" requirement. It's going to have to land over 
some objections, because some contributors object to the very essence of 
the change. If there are technical requirements, why can't I meet these 
requirements before landing this work in master? If there are political 
requirements, what makes you think that a branch would help me meet them?

> if
> we can find some willing testers (and one has already stepped forward). The
> only way your changes *wouldn't* land is if a better alternative comes along
> in the meantime -- which, of course, would mean we all win.

What is the fundamental difference between merging this code into 
mainline in a disabled-by-default configuration and merging it into a 
branch, except that it's much more convenient to try the feature in the 
former case?

>
> I don't suppose I can encourage you to maintain your fork within the Emacs.git
> repository? There this thing, rhymes with "blanch", where you could work on
> your version in tandem, even merge changes from us every once in a while... :)
>

There have been cries to adopt a more modern development style, one 
focused on GitHub and pull requests. I can certainly accommodate.

Re: Preview: portable dumper

[Matt Armstrong]

Daniel Colascione <dancol@dancol.org> writes:

> On 11/30/2016 08:49 PM, John Wiegley wrote:
>>>>>>> Daniel Colascione <dancol@dancol.org> writes:
>>
>>> If that's your decision, I'm going to fork.
>>
>> I'm quite sorry to hear it, but if branches are unacceptable, then such is
>> your right as a free software developer.
>
> Branches in general are fine. Relegating this specific feature to a 
> branch feels political.
>
> I would very much prefer to continue improving GNU Emacs. I'm a longtime 
> fan of the GNU Project. But I feel like the current gatekeepers are 
> standing in the way of progress.

Hey Daniel, as a mostly-irrelevant observer, I hope you do.  I don't
have time to devote to Emacs the way you do.  I think the best use of my
time is to try to convince you to continue doing so, by being as honest
as I can.  My advice:

 - take a breath, cool down
 - give it time, build trust
 - build on areas of agreement and common interest

I'm not an Emacs maintainer, and can't speak as one.  But, I play a
similar role at work, and can tell you that the technical issues at play
in these conflicts you have with Emacs maintainers are secondary.  At
their core, these discussions are all about trust.  You don't fully
trust the maintainers, and they don't fully trust you (hopefully "yet"
on both accounts).

In my "day job" I have the equivalent of "commit rights" to some widely
used software critical to the company's function.

From what I see, the role is similar to that of an Emacs maintainer.  It
is actually more important that the software remain functional and high
quality, and healthy for the long term, than it is to optimize
development practices for velocity of change.

Like Eli, John, and other maintainers, I'm often put in the position of
saying "no" or "slow down, I'm not sure yet" to a change.  Quite often
those asking for the change don't have the same perspective I do.  They
might see their change as minor.  Or critical.  Or urgent.  Or, if what
they introduce incurs some ongoing maintenance cost, they might
underestimate that cost, or truly believe in their commitment to bearing
that cost for the long term.

What they often don't appreciate is that I've seen *all* the failures.
I've seen the "simiple risk free idea" break horribly, or later became
something we regret.  I've seen authors vanish with half finished ideas,
leaving the *actual* maintainers with a mess.  I've seen "critically
urgent" changes become totally unnecessary as easier, more viable
alternatives come to light (often only after careful analysis, done by
me or others, sometimes to the impatient response of those proposing
change).

In this kind of system, where there is a hierarchy of decision makers,
and quality is often more important than speed, trust is earned over
time.

The quickest way to build trust is slowly, with a demonstrated
willingness and capability to collaborate and be sane over a period of
time.  Technical merit and an ability to make good logical arguments are
also important, but secondary.

True technical "arguments" are always boring.  They're pure pro/con
assessments, dispassionately weighing the knowns, unknowns, risks, etc.

Things get "spicy" when it is issues of trust, power, authority,
morality, etc.  Yet, I think it is human nature to try to "win" such
discussions by continuing to appeal to technical issues alone.  Quite
often, it boils down to "I don't trust you yet" and "Your lack of trust
frustrates me".  There is no getting around that.  A decision being made
for "political reasons" is not actually bad, it is the expected norm.

As a mater of political strategy, I'd encourage you to view "please
commit to a branch" as a victory.  Do it and watch trust grow in both
directions.  It is the foot in the door that gets the project closer to
the place you'd like to see it go.  It is a strategic move!


> Let me try one last time to explain my position: this dumping code
> will make no Emacs no worse. It cannot possible destabilize Emacs when
> it is compiled out of Emacs at configure-time. I specifically designed
> this feature to be optional. I am perfectly happy merging my code in
> such a way that it is not compiled into Emacs by default, and I am
> perfectly happy waiting until we've gotten sufficient testing before
> enabling the portable dumper.

First, let me say these your arguments don't strike me as being wildly
off base.  But, I'm not even an Emacs maintainer and can't help but view
the with some suspicion born from the school of hard knocks (see
scenarios outlined above).  Viewed from my experience, if I pretend to
be an Emacs maintainer, this is actually what I end up thinking:

 - There will be more #ifdefs in Emacs code.

 - You are not likely to admit it, but you may disappear.  If you do, it
   may be my job to remove the #ifdefs.

 - You are particularly forceful and invested in getting your way.  If I
   end up saying "no" in the end, you may walk away, leaving me with a
   mess to clean up.

 - You are less experienced than the maintainers with the code base, but
   claim a change is safe/easy/cheap/harmless.  That is nice, but I've
   seen these claims play out differently before.

 - So, risk is non zero.  I'd like to entertain ways to move this
   forward with less risk.


> But because this code is harmless when compiled away and because we
> can back it out without problems, I see the requirement that we put
> this code into a branch as a way to kill the feature. If this code
> were more fundamentally destabilizing, I would feel different --- but
> as it is, since this code is *not* fundamentally destabilizing, I
> don't see a technical case for a feature branch.

The purpose of a branch is to isolate risk.  Sure, any branch may never
merge back, but the primary reason to commit to a branch is to
*facilitate* future code sharing between the branch and mainline, not to
kill features.


>> Just so you know, I'd hope to see your patches on master within six
>> months,
>
> What specific conditions need to be met before merging?

I think asking for a contract is not unfair, but may be unrealistic.

Have you seen the green light given to the concurrency branch?  That
branch began without a contract.

Again, from a maintainer's point of view, this kind of contractual
question is sometimes unanswerable.  The maintainer may simply feel
reluctance and not know why.

Simply being willing to "play ball" over a period of time, combined with
the absence of any better options coming along, can often tip the scale
in the "okay to merge" direction.  Sometimes people need to actually
"see" a diff on their own screens, run the binary themselves, etc., and
hear a few positive reports from others they trust, before they're
willing to green light something.  Emailing a pre-commited patch around
is one way to do that, but git was invented to support a better workflow
directly.

Reluctance on the part of project owners is not inherently bad.

I have actually been in the position of apologizing to people after I
held off their proposed changes for a long time, only to realize that
"they were right all along" and was "obviously the right thing to do" in
hindsight.  I usually thank them for being patient with my reluctance.
In reality, my reluctance is usually good judgment, and has saved more
work than it has caused harm.  Often, I cannot even initially articulate
why an idea gives me pause.  Sometimes it comes to me some time after
the fact.


>> if we can find some willing testers (and one has already stepped
>> forward). The only way your changes *wouldn't* land is if a better
>> alternative comes along in the meantime -- which, of course, would
>> mean we all win.

The above is John saying "yes".  I can't tell you how many times I've
said yes to 95% of a proposal and people walk away because of a minor
difference of opinion on a non-essential issue.


> What is the fundamental difference between merging this code into 
> mainline in a disabled-by-default configuration and merging it into a 
> branch

1) the feature owner (you) absorbs the entire cost of maintaining the
branch.  The maintenance cost at mainline is zero.  There is zero chance
of ChangeLog, documentation, NEWS, or any mention of the feature
appearing in future snapshots, pretests, releases.  As far as I know,
there is no #ifdef for Emacs documentation.  If the feature is not
eventually accepted, the cost of rejecting it is zero.  There is no
"backing the change out" step.

2) it is easy to review the state of the diff in its entirety against
the master at any point.  Once merged you're stuck with more primitive
tools like grep and git blame.


> , except that it's much more convenient to try the feature in the
> former case?

It is probably worth enumerating the concrete differences.

1) git makes it trivial to check out a branch and build.  Playing around
with the concurrency branch was pretty easy the time I did it.  For the
"I want to try this for a day and report back" use case, it is fine.

2) using a feature branch long term *is* harder.  Only one can be used
at a time (easily), and the feature branch owner must merge from master
regularly to keep it relevant.

3) using a branch *combined* with another branch is a pain.  If I want
to test "concurrency AND portable dumper" I have to merge the two
branches myself.

There is no correct choice, they are just simple tradeoffs.  I see both
options as reasonable.

If the goal is to collect a small-ish number of "I tried it on platform
X and it works" reports, and let the ideas percolate/bake for a time,
then a branch makes a lot of sense.

If the goal is to get the feature used by as many different people as
possible as fast as possible in as many different scenarios as possible,
then a branch makes less sense.

I'll point out that your reluctance to commit to a branch is about trust
and not about the technical issues above: you don't seem to believe John
when he says the intent is to facilitate a future merge.


>> I don't suppose I can encourage you to maintain your fork within the
>> Emacs.git repository? There this thing, rhymes with "blanch", where
>> you could work on your version in tandem, even merge changes from us
>> every once in a while... :)
>
> There have been cries to adopt a more modern development style, one
> focused on GitHub and pull requests. I can certainly accommodate.

I do agree that submitting patches to projects on "non github" systems
is beginning to feel archaic.  I hope the era of emailing patches around
disappears.

If this is a detractor for Emacs it is also a detractor for every
project hosted on savannah.gnu.org.  Rather than fork Emacs to github to
get the "modern" development style, the greater impact, in terms of
benefit to GNU, is adding the desired features to savannah.

I don't know if anybody is working on this or not.

That said, I think you're too quick to suggest forking Emacs.  You'll
get more done working with Emacs folks than in spite of them, and the
stuff you work on will have more impact.  Even if it feels to you like
you could go 10x faster alone, building a real community is pretty hard,
and you'd be at high risk of burn out doing it alone, and the final
product may not result in more than another non-standard Emacs that gets
little use.

Re: Preview: portable dumper

[John Wiegley]

>>>>> "MA" == Matt Armstrong <marmstrong@google.com> writes:

MA> I'm not an Emacs maintainer, and can't speak as one. But, I play a similar
MA> role at work, and can tell you that the technical issues at play in these
MA> conflicts you have with Emacs maintainers are secondary. At their core,
MA> these discussions are all about trust. You don't fully trust the
MA> maintainers, and they don't fully trust you (hopefully "yet" on both
MA> accounts).

This was a really wonderful post, Matt. I hope it's OK if I share it with some
friends who also maintain large projects.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Phillip Lord]

On Thu, December 1, 2016 5:12 am, Daniel Colascione wrote:
> On 11/30/2016 08:49 PM, John Wiegley wrote:
>> if we can find some willing testers (and one has already stepped
>> forward). The only way your changes *wouldn't* land is if a better
>> alternative comes along in the meantime -- which, of course, would mean
>> we all win.
>
> What is the fundamental difference between merging this code into
> mainline in a disabled-by-default configuration and merging it into a
> branch, except that it's much more convenient to try the feature in the
> former case?


I wasn't going to comment on this thread, but alas here I go!

You launched the thread with "preview" in the title, that is you don't
think that your patch is finished yes. So, you are planning on doing more
work on it, before pushing to master? In which case, you already have a
branch. Why not push that up to the main emacs git? This would make is
very easy for people to test straight off the bat. I'll build and test it
on windows for instance.

I agree with your concerns that it will just moulder on a branch, and
that's true. There is a portable dumper out there for Emacs (version 21),
which is perhaps an indicator. But then disabled-by-default work runs the
same risk. No one tests it that much.

There's enough interest in your work that having it on a branch might even
contribute to encouraging more use of feature branches as part of
development rather than long running forks. That would be good, from my
perspective.

Phil

Re: Preview: portable dumper

[Daniel Colascione]

On November 30, 2016 7:47:21 PM PST, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Cc: Paul Eggert <eggert@cs.ucla.edu>,  rms@gnu.org, 
>emacs-devel@gnu.org
>> Date: Wed, 30 Nov 2016 13:50:33 -0800
>> 
>> > Until you're code is tested on more platforms, it needs to live
>someplace
>> > other than master. That's what branches are for.
>> 
>> I'll test it on the systems I can before I commit.  We can enable
>> compilation once on individual systems once we do more thorough
>testing.
>> A branch is definitely not the right approach here, because you
>haven't
>> established any firm acceptance criteria, and what criteria you have
>> mentioned (write some documentation, test on more configurations) are
>> things I can do before the patch lands at all.
>
>Having new features on a branch is our standard development practice.

No it isn't.

>It allows the interested people to try the feature in situations no
>single person can possibly create, even if that person has access to
>several platforms, because usage patterns differ a lot.

No, that's what configuration options do. Branches are seldom touched, because there are much bigger barriers to trying them.


>We can establish pass criteria, if you like, but the only criteria
>until now were "if enough time passed and no one complained, it's
>ready".

You have wanted to kill this feature at any cost, contrary to the wishes of every other Emacs contributor. What kind of fair pass criteria can I expect?

>In this case, we will probably also consider alternative approaches if
>they become mature.

Got it. So we won't consider merging the code until some lread improvement comes along somehow (which might take years) and we'll declare the lread improvement the winner no matter what the performance difference.

>> The Cairo code doesn't work at all and *that's* in mainline.
>> That's okay, because it's not enabled by default.
>
>The Cairo code works, it just has some redisplay bugs that no one was
>able to fix.  

So it doesn't work.

> It was disabled post-factum, when we have learned about
>those problems, unfortunately after its developer has departed.

It's obvious within five minutes of trying the feature that it doesn't work. That's okay, because it's opt-in.

>> Branches are for experimental features that can't coexist with
>> regular Emacs use and development.
>
>This is simply not true.

It is true if you look at the history of things for which we've actually used branches.

Putting this feature on a branch is tantamount to killing it. There is no legitimate reason to keep it out of mainline, because it is harmless unless enabled, and we can keep it disabled for the moment.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Paul Eggert <eggert@cs.ucla.edu>,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Wed, 30 Nov 2016 13:44:14 -0800
> 
> BTW, the concurrency branch has the green light. If anyone is willing to work
> to resolve its remaining issues, they can merge it to master once ready.

I will try to give it a go this weekend.

Re: Preview: portable dumper

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

>
> EZ> Maybe, maybe not. I don't understand why we need to grasp the first
> straw
> EZ> we see around to get rid of unexec. What's the rush?
>
> Eli has a good point here, there really is no rush. Daniel, would you be
> willing to continue to incubate these changes in a branch, while we also
> consider and explore alternatives?


Isn't unexec what prevents emacs from building inside docker containers
(because of the personality syscall)?

As far as I'm concerned, that's a quite annoying emacs limitation. See bug
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23529.

That said, I understand I'm probably a minory opinion, and that because
this is a sensitive topic we should not rush it... but it's not something
that should be put in a wish list (e.g some untested branch), because it's
something that needs to be fixed sooner or later.

Philippe

[-- Attachment #2: Type: text/html, Size: 1312 bytes --]

Re: Preview: portable dumper

[Daniel Colascione]

On Wed, Nov 30 2016, Philippe Vaucher wrote:
>  EZ> Maybe, maybe not. I don't understand why we need to grasp the first straw
>  EZ> we see around to get rid of unexec. What's the rush?
>
>  Eli has a good point here, there really is no rush. Daniel, would you be
>  willing to continue to incubate these changes in a branch, while we also
>  consider and explore alternatives?
>
> Isn't unexec what prevents emacs from building inside docker containers (because of the personality
> syscall)?
>
> As far as I'm concerned, that's a quite annoying emacs limitation. See bug
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23529.

With the portable dumper, we'll no longer need any personality-changing
calls. Emacs will be a normal, boring program.

Re: Preview: portable dumper

[Paul Eggert]

On 11/30/2016 08:38 AM, Eli Zaretskii wrote:

> you didn't like this same idea very much

Yes and I still don't like it much, and if someone implements a better 
solution I will be happy to advocate replacing the currently-proposed 
approach with a better one. The Emacs dumping scheme has always been and 
will remain private and Emacs-version-specific, so we will have a lot of 
flexibility here.

In the meantime, if the currently-proposed approach works and is fast, I 
see no reason to reject the contribution, or to delay it indefinitely by 
putting it into some branch that nobody looks at. If the patch 
adequately fixes a real and growing problem and we have no other working 
fix, we should install it into the master branch.

>> from what I can see, the C pipeline is by no means empty.
> Well, it somehow becomes all but empty on the other end.

Sure, maybe 40 years from now C programmers will be history! That is no 
reason for us to reject a patch that we need today.

If we want to switch the Emacs core from C to some other programming 
environment, fine. We can think about doing that and we will have plenty 
of time to think. In the meantime, the core can and should assume C. 
Every solution to this more-urgent problem will involve some C hacking, 
and that's OK.

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I sympathize with your general point, but I don't think it is
conceivable to make Emacs's C code much simpler or easier to maintain
than it is now.

Also, I don't think the portable dumper would increase the conplexity,
not if it is written cleanly.

To increase the _number_ of modules that look at Lisp objects with a
given level of complexity does not increase the complexity.  When
changing the data structures, it is no harder to update 4 places than
3 places.  It adds a little more work, but it isn't _harder_ work.

I conclude that the portable dumper won't make it any harder to
find people to work on the C code.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: dancol@dancol.org, emacs-devel@gnu.org
> Date: Thu, 01 Dec 2016 04:18:03 -0500
> 
> I sympathize with your general point, but I don't think it is
> conceivable to make Emacs's C code much simpler or easier to maintain
> than it is now.

I agree.

However, this is not about making existing code simpler.  It's about
_adding_ more C code, where an alternative implementation might allow
doing that without such additions.

> Also, I don't think the portable dumper would increase the conplexity,
> not if it is written cleanly.

It's a new and significant mechanism, which is central to building and
starting Emacs.  It needs to be studied, understood, and maintained.
Therefore, it does increase the complexity.  You might argue that the
increase in complexity is not significant enough to worry about, but
it's definitely there.

> To increase the _number_ of modules that look at Lisp objects with a
> given level of complexity does not increase the complexity.

It's not just the number of modules, it's what they do with the
objects.  The dumper does qualitatively different things, unlike any
other module in Emacs.  It's not just "more of the same".

> When changing the data structures, it is no harder to update 4
> places than 3 places.  It adds a little more work, but it isn't
> _harder_ work.

Well, if I need to do more work, the job is harder.

> I conclude that the portable dumper won't make it any harder to
> find people to work on the C code.

I agree with that conclusion.  However, my problem is that, since
finding such people is already hard, making more Emacs modules depend
on that might expose us to more future maintenance problems than if we
don't.  Especially if those modules are so central to Emacs as the one
being discussed here.

Re: Preview: portable dumper

[Stefan Monnier]

> It's a new and significant mechanism, which is central to building and
> starting Emacs.  It needs to be studied, understood, and maintained.

FWIW, I looked at Daniel's code and found it understandable, whereas
I never managed to make any sense of the unexec code, which still just looks
like magical voodoo to me, despite my trying twice to understand it.


        Stefan

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 01 Dec 2016 21:18:04 -0500
> 
> > It's a new and significant mechanism, which is central to building and
> > starting Emacs.  It needs to be studied, understood, and maintained.
> 
> FWIW, I looked at Daniel's code and found it understandable, whereas
> I never managed to make any sense of the unexec code, which still just looks
> like magical voodoo to me, despite my trying twice to understand it.

I understand both (though, admittedly, that excludes unexelf.c, which
I never needed to look at for more than 5 sec).  But this is not about
you or me.  Do you really believe that I'm afraid of the dumper code,
having tweaked the display engine to speak RTL?  However, this is
about the next generation of Emacs maintainers who will inherit this
code and will have to maintain it.

Granted, the proposed dumper is not very complicated.  But it isn't
trivial either.  So if we can achieve a similar effect by using the
"normal" loadup code, which is much simpler and doesn't really require
understanding anything new, I think it's more beneficial for the
project's future.

The key issue is performance, as was stated many times.  If the
performance of loadup after all the speedups is not good enough, then
we will have to conclude that this idea cannot fly, at least not yet.
(That conclusion might be revisited later, if/when disks, filesystems,
and machines become faster.)  It will also not fly if no one will pick
up the gauntlet and work on that idea to get it to completion.

But IMO we should at least try implementing that idea, because its
benefits are so clear.  We shouldn't give up without trying, and we
have no special reason to decide the idea isn't viable until we've
tried.

Once again, if those ideas seem strange, let alone incorrect, to
you-all out there, just say the word, and I will step down.  Then I
can stop worrying about the portable dumper, and you can stop worrying
about my strange ideas.  Emacs is not my private project; I'm only
entitled to promote my ideas if the others either support them or
trust me to DTRT.  Please decide which one is it, and let's move on.

Re: Preview: portable dumper

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 2170 bytes --]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> The key issue is performance, as was stated many times. If the performance
EZ> of loadup after all the speedups is not good enough, then we will have to
EZ> conclude that this idea cannot fly, at least not yet. (That conclusion
EZ> might be revisited later, if/when disks, filesystems, and machines become
EZ> faster.) It will also not fly if no one will pick up the gauntlet and work
EZ> on that idea to get it to completion.

I think Daniel has helped to stimulate us by offering a close-to-working
solution to an ancient problem. The code it replaces is worse; while the
as-yet-unproven idea of a fast .elc loader is better. It's now upon us to
prove that we can beat the portable dumper, by making good on a fast .elc
loader.

That said, I don't want to let Daniel's idea wither on the vine while we prove
or disprove a fast .elc loader. Sure, we can wait a couple of months to give
that idea a chance while we review the dumper, but if we don't get an .elc
loader in that time, we should give Daniel's idea a go, if for no other reason
than that the work is almost done, and we could gain some valuable experience
-- and a way to do some real performance comparisons.

If the .elc idea pans out (and I really hope it can), I think even Daniel will
be relieved at not having to maintain his dumper, and he can help us to
extricate his helpful addition.

EZ> Once again, if those ideas seem strange, let alone incorrect, to you-all
EZ> out there, just say the word, and I will step down. Then I can stop
EZ> worrying about the portable dumper, and you can stop worrying about my
EZ> strange ideas. Emacs is not my private project; I'm only entitled to
EZ> promote my ideas if the others either support them or trust me to DTRT.
EZ> Please decide which one is it, and let's move on.

Please do not step down, Eli. Please don't. It sickens my heart every time I
hear you suggest it, so just stop doing that for my sake. :)

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 658 bytes --]

Re: Preview: portable dumper

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org
> Date: Fri, 02 Dec 2016 00:08:55 -0800
> 
> EZ> Once again, if those ideas seem strange, let alone incorrect, to you-all
> EZ> out there, just say the word, and I will step down. Then I can stop
> EZ> worrying about the portable dumper, and you can stop worrying about my
> EZ> strange ideas. Emacs is not my private project; I'm only entitled to
> EZ> promote my ideas if the others either support them or trust me to DTRT.
> EZ> Please decide which one is it, and let's move on.
> 
> Please do not step down, Eli. Please don't. It sickens my heart every time I
> hear you suggest it, so just stop doing that for my sake. :)

I hear you.  But with all due respect, you cannot have it both ways:
beg me to stay in charge, and then push me to give up one of the few
firm ideas I have about where Emacs should go and how.  It isn't fair.
I arrived at those ideas by carefully watching Emacs development for
many years.  What kind of maintainer will I be if I cannot base my
judgment and decisions on a few principal ideas I have about Emacs
development?  What could replace those ideas to be the base for
decisions I need to make almost every day?

Re: Preview: portable dumper

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 3434 bytes --]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> But with all due respect, you cannot have it both ways: beg me to stay in
> charge, and then push me to give up one of the few firm ideas I have about
> where Emacs should go and how.

I think a part of this job is always political: Balancing the twin goals of
promoting the best ideas, while also keeping everyone involved encouraged and
motivated. When we say "yes" to an idea, we might be hurting Emacs; but when
we say "no", we might be hurting that contributor, and losing their future
involvement. Sometimes that's fine -- after all, everyone else in the world
wants us to care about Emacs more than a limited-time set of contributors --
but we also cannot thrive without active contributors, so "watering the
garden" is part of our task.

We all know the portable dumper is not an ideal solution; we even have some
notion of what the ideal solution looks like. It's indeed a bit frustrating to
knowingly support a technology path that may end up incurring a lot of work,
if a much better solution might be just around the corner.

However, I ask you to consider the merits of Daniel's proposal from several
angles -- angles a regular developer may not care about at all:

 - If we accept the work, we show to others that *code wins*. That is, if a
   problem exists in Emacs, and someone comes to us with actual code, this has
   tremendous weight. If this is our position, it encourages others to
   experiment, since they'll fear less that after so much work, we might just
   say "no" because it's not as good a solution as we'd like.

 - If we accept the work, it encourages Daniel to play a larger role, to learn
   more about the C internals, and likely to contribute even more.

 - If we accept the work, and a better solution comes along later, we can
   revert the change without undue labor.  That is, our end cost is not so
   terribly high that we're panting ourselves into a corner.

 - If we accept the work, despite our disagreement, *precisely because most of
   the other core developers do agree*, we're saying that their opinion holds
   a lot of weight with us, and this encourages frank and open discussion. If
   decisions like this are made by just you or me, against all other
   objections, it diminishes the value of this mailing list in others' eyes.
   It then seems like we're just making the Emacs we want, and not the Emacs
   all of us want.

I think there is a "greater good" here, and it is not unduly damaged by
letting a short-term solution into the code until a longer-term solution
appears. There is much goodwill to be gained, and really not so much harm. On
the other hand, rejecting it -- despite the general sense of agreement is has
gained -- would cause a much greater social damage, which is far more
difficult to undo than one day asking Daniel to revert his work.

All that said, if you feel strongly that this proposal hurts Emacs itself, and
cannot abide it, I understand. I was hoping we could all win here, but I know
that sometimes this isn't realistic. I just hope we can all recognize that
this portable dumper idea is not so terribly important: not to fork over, nor
to resign from the amazing support you provide.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 658 bytes --]

Re: Preview: portable dumper

[Karl Fogel]

[-- Attachment #1: Type: text/plain, Size: 5682 bytes --]

Following up to what John said, with a perspective that may be useful to Eli:

Eli, I have a great deal of respect for your technical judgement and the amount of work you put into Emacs.  But I have a slightly different take on what maintainership means.

When I was in a position similar to yours (one of a small group of core co-maintainers, in the Subversion project), there were a few occasions when a major technical decision went in a direction I didn't agree with.  My disagreement was not of the "this will destroy the project" sort, but I did feel the decisions were poor technical choices that could be a long-term drag on the project -- creating extra maintenance work for existing developers, and creating barriers to entry for new developers.  In other words, objections very similar to yours now.

In each case, discussions eventually made it clear to me that my opinion was not going to prevail.  There were just too many developers who felt differently.  This didn't tempt me to step down from maintainership, though I will admit that it *did* tempt me to keep score a bit, so that later on I might be able to say "Hey folks, remember when I turned out to be right about X, and Y, and Z?  So please listen to me this time."  I think I may even have used that privilege once or twice later :-), though I don't remember clearly if I did.  What I do remember clearly is that I turned out to be wrong in at least one of the cases (at least, I established to my own satisfaction that my warnings had been unwarranted).

Your value as a maintainer is not lessened if a particular decision doesn't go the way you recommended.  (Of course, if maintainership becomes unenjoyable to you because of the decision, that's a different question, and only you can make that call.)  I think there is even a good argument that your continued maintainership actually becomes slightly *more* important to the project: of all the people who could spot whether the negative consequences of the decision are coming true, you would perhaps be the person most likely to spot it, and most likely to point it out to the other developers.

You must make your own decision, of course.  But I hope this perspective is a useful response to your question:

  "...What kind of maintainer will I be if I cannot base my judgment and
  decisions on a few principal ideas I have about Emacs development?
  What could replace those ideas to be the base for decisions I need to
  make almost every day?"

(I pretty much agree with everything John said in his email below, too, for what it's worth.)

Best regards,
-Karl

John Wiegley <jwiegley@gmail.com> writes:
>I think a part of this job is always political: Balancing the twin goals of
>promoting the best ideas, while also keeping everyone involved encouraged and
>motivated. When we say "yes" to an idea, we might be hurting Emacs; but when
>we say "no", we might be hurting that contributor, and losing their future
>involvement. Sometimes that's fine -- after all, everyone else in the world
>wants us to care about Emacs more than a limited-time set of contributors --
>but we also cannot thrive without active contributors, so "watering the
>garden" is part of our task.
>
>We all know the portable dumper is not an ideal solution; we even have some
>notion of what the ideal solution looks like. It's indeed a bit frustrating to
>knowingly support a technology path that may end up incurring a lot of work,
>if a much better solution might be just around the corner.
>
>However, I ask you to consider the merits of Daniel's proposal from several
>angles -- angles a regular developer may not care about at all:
>
> - If we accept the work, we show to others that *code wins*. That is, if a
>   problem exists in Emacs, and someone comes to us with actual code, this has
>   tremendous weight. If this is our position, it encourages others to
>   experiment, since they'll fear less that after so much work, we might just
>   say "no" because it's not as good a solution as we'd like.
>
> - If we accept the work, it encourages Daniel to play a larger role, to learn
>   more about the C internals, and likely to contribute even more.
>
> - If we accept the work, and a better solution comes along later, we can
>   revert the change without undue labor.  That is, our end cost is not so
>   terribly high that we're panting ourselves into a corner.
>
> - If we accept the work, despite our disagreement, *precisely because most of
>   the other core developers do agree*, we're saying that their opinion holds
>   a lot of weight with us, and this encourages frank and open discussion. If
>   decisions like this are made by just you or me, against all other
>   objections, it diminishes the value of this mailing list in others' eyes.
>   It then seems like we're just making the Emacs we want, and not the Emacs
>   all of us want.
>
>I think there is a "greater good" here, and it is not unduly damaged by
>letting a short-term solution into the code until a longer-term solution
>appears. There is much goodwill to be gained, and really not so much harm. On
>the other hand, rejecting it -- despite the general sense of agreement is has
>gained -- would cause a much greater social damage, which is far more
>difficult to undo than one day asking Daniel to revert his work.
>
>All that said, if you feel strongly that this proposal hurts Emacs itself, and
>cannot abide it, I understand. I was hoping we could all win here, but I know
>that sometimes this isn't realistic. I just hope we can all recognize that
>this portable dumper idea is not so terribly important: not to fork over, nor
>to resign from the amazing support you provide.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Preview: portable dumper

[Daniel Colascione]

FWIW, I'll put my patch on a branch this weekend.

On Fri, Dec 02 2016, Karl Fogel wrote:
> Following up to what John said, with a perspective that may be useful to Eli:
>
> Eli, I have a great deal of respect for your technical judgement and
> the amount of work you put into Emacs.  But I have a slightly
> different take on what maintainership means.
>
> When I was in a position similar to yours (one of a small group of
> core co-maintainers, in the Subversion project), there were a few
> occasions when a major technical decision went in a direction I didn't
> agree with.  My disagreement was not of the "this will destroy the
> project" sort, but I did feel the decisions were poor technical
> choices that could be a long-term drag on the project -- creating
> extra maintenance work for existing developers, and creating barriers
> to entry for new developers.  In other words, objections very similar
> to yours now.
>
> In each case, discussions eventually made it clear to me that my
> opinion was not going to prevail.  There were just too many developers
> who felt differently.  This didn't tempt me to step down from
> maintainership, though I will admit that it *did* tempt me to keep
> score a bit, so that later on I might be able to say "Hey folks,
> remember when I turned out to be right about X, and Y, and Z?
> So please listen to me this time."  I think I may even have used that
> privilege once or twice later :-), though I don't remember clearly if
> I did.  What I do remember clearly is that I turned out to be wrong in
> at least one of the cases (at least, I established to my own
> satisfaction that my warnings had been unwarranted).
>
> Your value as a maintainer is not lessened if a particular decision
> doesn't go the way you recommended.  (Of course, if maintainership
> becomes unenjoyable to you because of the decision, that's a different
> question, and only you can make that call.)  I think there is even a
> good argument that your continued maintainership actually becomes
> slightly *more* important to the project: of all the people who could
> spot whether the negative consequences of the decision are coming
> true, you would perhaps be the person most likely to spot it, and most
> likely to point it out to the other developers.
>
> You must make your own decision, of course.  But I hope this
> perspective is a useful response to your question:
>
>   "...What kind of maintainer will I be if I cannot base my judgment and
>   decisions on a few principal ideas I have about Emacs development?
>   What could replace those ideas to be the base for decisions I need to
>   make almost every day?"
>
> (I pretty much agree with everything John said in his email below,
> too, for what it's worth.)
>
> Best regards,
> -Karl
>
> John Wiegley <jwiegley@gmail.com> writes:
>>I think a part of this job is always political: Balancing the twin goals of
>>promoting the best ideas, while also keeping everyone involved encouraged and
>>motivated. When we say "yes" to an idea, we might be hurting Emacs; but when
>>we say "no", we might be hurting that contributor, and losing their future
>>involvement. Sometimes that's fine -- after all, everyone else in the world
>>wants us to care about Emacs more than a limited-time set of contributors --
>>but we also cannot thrive without active contributors, so "watering the
>>garden" is part of our task.
>>
>>We all know the portable dumper is not an ideal solution; we even have some
>>notion of what the ideal solution looks like. It's indeed a bit frustrating to
>>knowingly support a technology path that may end up incurring a lot of work,
>>if a much better solution might be just around the corner.
>>
>>However, I ask you to consider the merits of Daniel's proposal from several
>>angles -- angles a regular developer may not care about at all:
>>
>> - If we accept the work, we show to others that *code wins*. That is, if a
>>   problem exists in Emacs, and someone comes to us with actual code, this has
>>   tremendous weight. If this is our position, it encourages others to
>>   experiment, since they'll fear less that after so much work, we might just
>>   say "no" because it's not as good a solution as we'd like.
>>
>> - If we accept the work, it encourages Daniel to play a larger role, to learn
>>   more about the C internals, and likely to contribute even more.
>>
>> - If we accept the work, and a better solution comes along later, we can
>>   revert the change without undue labor.  That is, our end cost is not so
>>   terribly high that we're panting ourselves into a corner.
>>
>> - If we accept the work, despite our disagreement, *precisely because most of
>>   the other core developers do agree*, we're saying that their opinion holds
>>   a lot of weight with us, and this encourages frank and open discussion. If
>>   decisions like this are made by just you or me, against all other
>>   objections, it diminishes the value of this mailing list in others' eyes.
>>   It then seems like we're just making the Emacs we want, and not the Emacs
>>   all of us want.
>>
>>I think there is a "greater good" here, and it is not unduly damaged by
>>letting a short-term solution into the code until a longer-term solution
>>appears. There is much goodwill to be gained, and really not so much harm. On
>>the other hand, rejecting it -- despite the general sense of agreement is has
>>gained -- would cause a much greater social damage, which is far more
>>difficult to undo than one day asking Daniel to revert his work.
>>
>>All that said, if you feel strongly that this proposal hurts Emacs itself, and
>>cannot abide it, I understand. I was hoping we could all win here, but I know
>>that sometimes this isn't realistic. I just hope we can all recognize that
>>this portable dumper idea is not so terribly important: not to fork over, nor
>>to resign from the amazing support you provide.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Fri, 02 Dec 2016 13:22:56 -0800
> Cc: emacs-devel@gnu.org
> 
> FWIW, I'll put my patch on a branch this weekend.

Thank you.

Re: Preview: portable dumper

[Karl Fogel]

Daniel Colascione <dancol@dancol.org> writes:
>FWIW, I'll put my patch on a branch this weekend.

Thanks, Daniel.  I will build it and try it out.

Best regards,
-Karl

>On Fri, Dec 02 2016, Karl Fogel wrote:
>> Following up to what John said, with a perspective that may be useful to Eli:
>>
>> Eli, I have a great deal of respect for your technical judgement and
>> the amount of work you put into Emacs.  But I have a slightly
>> different take on what maintainership means.
>>
>> When I was in a position similar to yours (one of a small group of
>> core co-maintainers, in the Subversion project), there were a few
>> occasions when a major technical decision went in a direction I didn't
>> agree with.  My disagreement was not of the "this will destroy the
>> project" sort, but I did feel the decisions were poor technical
>> choices that could be a long-term drag on the project -- creating
>> extra maintenance work for existing developers, and creating barriers
>> to entry for new developers.  In other words, objections very similar
>> to yours now.
>>
>> In each case, discussions eventually made it clear to me that my
>> opinion was not going to prevail.  There were just too many developers
>> who felt differently.  This didn't tempt me to step down from
>> maintainership, though I will admit that it *did* tempt me to keep
>> score a bit, so that later on I might be able to say "Hey folks,
>> remember when I turned out to be right about X, and Y, and Z?
>> So please listen to me this time."  I think I may even have used that
>> privilege once or twice later :-), though I don't remember clearly if
>> I did.  What I do remember clearly is that I turned out to be wrong in
>> at least one of the cases (at least, I established to my own
>> satisfaction that my warnings had been unwarranted).
>>
>> Your value as a maintainer is not lessened if a particular decision
>> doesn't go the way you recommended.  (Of course, if maintainership
>> becomes unenjoyable to you because of the decision, that's a different
>> question, and only you can make that call.)  I think there is even a
>> good argument that your continued maintainership actually becomes
>> slightly *more* important to the project: of all the people who could
>> spot whether the negative consequences of the decision are coming
>> true, you would perhaps be the person most likely to spot it, and most
>> likely to point it out to the other developers.
>>
>> You must make your own decision, of course.  But I hope this
>> perspective is a useful response to your question:
>>
>>   "...What kind of maintainer will I be if I cannot base my judgment and
>>   decisions on a few principal ideas I have about Emacs development?
>>   What could replace those ideas to be the base for decisions I need to
>>   make almost every day?"
>>
>> (I pretty much agree with everything John said in his email below,
>> too, for what it's worth.)
>>
>> Best regards,
>> -Karl
>>
>> John Wiegley <jwiegley@gmail.com> writes:
>>>I think a part of this job is always political: Balancing the twin goals of
>>>promoting the best ideas, while also keeping everyone involved encouraged and
>>>motivated. When we say "yes" to an idea, we might be hurting Emacs; but when
>>>we say "no", we might be hurting that contributor, and losing their future
>>>involvement. Sometimes that's fine -- after all, everyone else in the world
>>>wants us to care about Emacs more than a limited-time set of contributors --
>>>but we also cannot thrive without active contributors, so "watering the
>>>garden" is part of our task.
>>>
>>>We all know the portable dumper is not an ideal solution; we even have some
>>>notion of what the ideal solution looks like. It's indeed a bit frustrating to
>>>knowingly support a technology path that may end up incurring a lot of work,
>>>if a much better solution might be just around the corner.
>>>
>>>However, I ask you to consider the merits of Daniel's proposal from several
>>>angles -- angles a regular developer may not care about at all:
>>>
>>> - If we accept the work, we show to others that *code wins*. That is, if a
>>>   problem exists in Emacs, and someone comes to us with actual code, this has
>>>   tremendous weight. If this is our position, it encourages others to
>>>   experiment, since they'll fear less that after so much work, we might just
>>>   say "no" because it's not as good a solution as we'd like.
>>>
>>> - If we accept the work, it encourages Daniel to play a larger role, to learn
>>>   more about the C internals, and likely to contribute even more.
>>>
>>> - If we accept the work, and a better solution comes along later, we can
>>>   revert the change without undue labor.  That is, our end cost is not so
>>>   terribly high that we're panting ourselves into a corner.
>>>
>>> - If we accept the work, despite our disagreement, *precisely because most of
>>>   the other core developers do agree*, we're saying that their opinion holds
>>>   a lot of weight with us, and this encourages frank and open discussion. If
>>>   decisions like this are made by just you or me, against all other
>>>   objections, it diminishes the value of this mailing list in others' eyes.
>>>   It then seems like we're just making the Emacs we want, and not the Emacs
>>>   all of us want.
>>>
>>>I think there is a "greater good" here, and it is not unduly damaged by
>>>letting a short-term solution into the code until a longer-term solution
>>>appears. There is much goodwill to be gained, and really not so much harm. On
>>>the other hand, rejecting it -- despite the general sense of agreement is has
>>>gained -- would cause a much greater social damage, which is far more
>>>difficult to undo than one day asking Daniel to revert his work.
>>>
>>>All that said, if you feel strongly that this proposal hurts Emacs itself, and
>>>cannot abide it, I understand. I was hoping we could all win here, but I know
>>>that sometimes this isn't realistic. I just hope we can all recognize that
>>>this portable dumper idea is not so terribly important: not to fork over, nor
>>>to resign from the amazing support you provide.

Re: Preview: portable dumper

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 292 bytes --]

On Fri, Dec 2, 2016 at 10:22 PM, Daniel Colascione <dancol@dancol.org>
wrote:

> FWIW, I'll put my patch on a branch this weekend.
>

Sorry if I missed it, but was this done? I can't find a relevant branch by
looking at https://github.com/emacs-mirror/emacs/branches/active

Thanks,
Philippe

[-- Attachment #2: Type: text/html, Size: 735 bytes --]

Re: Preview: portable dumper

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 490 bytes --]

@Daniel

Just bumping the thread. Would like to know what came out of this.

On Thu, Dec 15, 2016, 9:31 AM Philippe Vaucher <philippe.vaucher@gmail.com>
wrote:

> On Fri, Dec 2, 2016 at 10:22 PM, Daniel Colascione <dancol@dancol.org>
> wrote:
>
>> FWIW, I'll put my patch on a branch this weekend.
>>
>
> Sorry if I missed it, but was this done? I can't find a relevant branch by
> looking at https://github.com/emacs-mirror/emacs/branches/active
>
> Thanks,
> Philippe
>
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 1375 bytes --]

Re: Preview: portable dumper

[Jeremie Courreges-Anglas]

[-- Attachment #1: Type: text/plain, Size: 934 bytes --]

On Wed, Oct 18 2017, Kaushal Modi <kaushal.modi@gmail.com> wrote:
> @Daniel
>
> Just bumping the thread. Would like to know what came out of this.

Same here.  I don't really care whether the portable dumper or the "big
.elc file" approach is taken, but it would be cool to see emacs out of
the malloc and unexec business.  This code is fragile and tends to break
on less popular platforms (OpenBSD in my case).

> On Thu, Dec 15, 2016, 9:31 AM Philippe Vaucher <philippe.vaucher@gmail.com>
> wrote:
>
>> On Fri, Dec 2, 2016 at 10:22 PM, Daniel Colascione <dancol@dancol.org>
>> wrote:
>>
>>> FWIW, I'll put my patch on a branch this weekend.
>>>
>>
>> Sorry if I missed it, but was this done? I can't find a relevant branch by
>> looking at https://github.com/emacs-mirror/emacs/branches/active
>>
>> Thanks,
>> Philippe
>>

-- 
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Preview: portable dumper

[Daniel Colascione]

On 10/19/2017 03:12 AM, Jeremie Courreges-Anglas wrote:
> On Wed, Oct 18 2017, Kaushal Modi <kaushal.modi@gmail.com> wrote:
>> @Daniel
>>
>> Just bumping the thread. Would like to know what came out of this.
> 
> Same here.  I don't really care whether the portable dumper or the "big
> .elc file" approach is taken, but it would be cool to see emacs out of
> the malloc and unexec business.  This code is fragile and tends to break
> on less popular platforms (OpenBSD in my case).

I've pushed the portable dumper to the pdumper Savannah branch. It 
should support a fully PIC Emacs.

I implemented demand-paging logic and backed all of it out: the demand 
paging worked fine, but I wasn't able to figure out a good way to get GC 
not to page in the whole region almost immediately anyway. I was 
surprised to find that my Emacs GCs almost immediately after start.

Timing on the current code is nevertheless pretty good. Running emacs -Q 
-nw -batch --eval '(progn (garbage-collect) (kill-emacs))' 100 times,

   unexec: 22.47ms per invocation
   pdumper: 28.19ms per invocation

I think this degree of startup overhead is acceptable.

I could implement the non-PIC mode I'd previously mentioned; I'm 
confident it would be faster than unexec. But with PIC doing this well, 
why bother? We want a PIC Emacs for security reasons anyway.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 12 Feb 2018 12:18:36 -0800
> 
> I've pushed the portable dumper to the pdumper Savannah branch. It 
> should support a fully PIC Emacs.

Thanks.  I'd urge people to try this branch and report any issues they
see.

Re: Preview: portable dumper

[Philipp Stephani]

[-- Attachment #1.1: Type: text/plain, Size: 447 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Di., 13. Feb. 2018 um 17:39 Uhr:

> > From: Daniel Colascione <dancol@dancol.org>
> > Date: Mon, 12 Feb 2018 12:18:36 -0800
> >
> > I've pushed the portable dumper to the pdumper Savannah branch. It
> > should support a fully PIC Emacs.
>
> Thanks.  I'd urge people to try this branch and report any issues they
> see.
>
>
Got a crash when building on Debian testing. I've attached the output of
'bt full'.

[-- Attachment #1.2: Type: text/html, Size: 837 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gdb.txt --]
[-- Type: text/plain; charset="Shift_JIS"; name="gdb.txt", Size: 31042 bytes --]

#0  raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:51
        set = {
          __val = {0, 0, 0, 0, 0, 0, 0, 1095216660480, 204351844644765126, 2450524270238223137, 42930436372038613, 47150946645313795, 434264250787087240, 36173497990776353, 140721880134016, 6082251}
        }
        pid = <optimized out>
        tid = <optimized out>
#1  0x0000000000588efb in terminate_due_to_signal (sig=6, backtrace_limit=2147483647) at emacs.c:388
No locals.
#2  0x00000000006233cc in die (msg=0x7762b0 "offset < dump_private.header.discardable_start", file=0x774860 "pdumper.c", line=4718) at alloc.c:7789
No locals.
#3  0x000000000062dfb0 in pdumper_marked_p_impl (obj=0x7fb64a872168) at pdumper.c:4718
        offset = 6930792
        bitno = 29767524975390623
#4  0x0000000000588d83 in pdumper_marked_p (obj=0x7fb64a872168) at pdumper.h:205
No locals.
#5  0x000000000061c21f in cons_marked_p (c=0x7fb64a872168) at alloc.c:4163
No locals.
#6  0x0000000000621b74 in mark_object (arg=XIL(0x7fb64a87216b)) at alloc.c:7077
        ptr = 0x7fb64a872168
        obj = XIL(0x7fb64a872168)
        po = 0x7fb64a872168
        m = 0x7fb64ab440f4
        cdr_count = 0
#7  0x000000000061d978 in mark_maybe_object (obj=XIL(0x7fb64a87216b)) at alloc.c:5018
        po = 0x7fb64a872168
        m = 0x0
#8  0x000000000061ddd9 in mark_memory (start=0x7ffc5dad7590, end=0x7ffc5dad9cb8) at alloc.c:5210
        pp = 0x7ffc5dad8dc8 "k!\207J\266\177"
#9  0x000000000061de0e in mark_stack (bottom=0x7ffc5dad9cb8 " \037\215\002", end=0x7ffc5dad7590 "\240u\255]\374\177") at alloc.c:5417
No locals.
#10 0x00000000006ed837 in mark_one_thread (thread=0xe28540 <main_thread>) at thread.c:617
        stack_top = 0x7ffc5dad7590
#11 0x00000000006ed9cf in mark_threads_callback (ignore=0x0) at thread.c:650
        thread_obj = XIL(0xe28545)
        iter = 0xe28540 <main_thread>
#12 0x000000000061de55 in flush_stack_call_func (func=0x6ed93c <mark_threads_callback>, arg=0x0) at alloc.c:5444
        end = 0x7ffc5dad7590
        self = 0xe28540 <main_thread>
        sentry = {
          o = {
            __max_align_ll = 0, 
            __max_align_ld = <invalid float value>
          }
        }
#13 0x00000000006eda01 in mark_threads () at thread.c:657
No locals.
#14 0x0000000000620162 in garbage_collect_1 (end=0x7ffc5dad7780) at alloc.c:6322
        nextb = 0x0
        stack_top_variable = 0 '\000'
        message_p = false
        count = 25
        start = {
          tv_sec = 1518641881, 
          tv_nsec = 37396085
        }
        retval = XIL(0)
        tot_before = 0
        visitor = {
          visit = 0x61fe06 <mark_object_root_visitor>, 
          data = 0x0
        }
        total = {XIL(0), XIL(0), XIL(0x7ffc5dad7600), XIL(0x5825cb), XIL(0x7ffc5dad7630), make_number(1623929), XIL(0x7ffc5dad7630), XIL(0xe11620), XIL(0xd260), XIL(0xd260)}
#15 0x0000000000620844 in Fgarbage_collect () at alloc.c:6493
        end = 0x7ffc5dad7780
        sentry = {
          o = {
            __max_align_ll = 0, 
            __max_align_ld = <invalid float value>
          }
        }
#16 0x0000000000585c2c in maybe_gc () at lisp.h:5002
No locals.
#17 0x0000000000655136 in eval_sub (form=XIL(0x29d1aa3)) at eval.c:2195
        fun = make_number(1664453)
        val = XIL(0x7ffc5dad78c0)
        original_fun = XIL(0x5dad7888)
        original_args = XIL(0x7fb64a673420)
        funcar = XIL(0x5825cb)
        count = 0
        argvals = {XIL(0x58399f), XIL(0xd8fc45), XIL(0x7ffc5dad7830), XIL(0x5839c4), XIL(0x58399f), XIL(0xe043c0), XIL(0), XIL(0)}
#18 0x000000000068e41d in readevalloop (readcharfun=XIL(0x6a50), infile0=0x7ffc5dad7ad0, sourcename=XIL(0x29622c4), printflag=false, unibyte=XIL(0), readfun=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2073
        count1 = 25
        c = 35
        val = XIL(0x29d1aa3)
        count = 21
        b = 0x0
        continue_reading_p = true
        lex_bound = XIL(0)
        whole_buffer = false
        first_sexp = false
        macroexpand = XIL(0)
#19 0x000000000068c6ea in Fload (file=XIL(0x7fb64a20e0d4), noerror=XIL(0), nomessage=XIL(0xc660), nosuffix=XIL(0), must_suffix=XIL(0xc660)) at lread.c:1457
        input = {
          stream = 0x295ebc0, 
          lookahead = 0 '\000', 
          buf = "\000\000\000"
        }
        stream = 0x295ebc0
        fd = 5
        fd_index = 13
        count = 13
        found = XIL(0x295e854)
        efound = XIL(0x295e854)
        hist_file_name = XIL(0x295e854)
        newer = false
        compiled = true
        handler = XIL(0x2942473)
        safe_p = true
        fmode = 0x7864bb "r"
        version = 23
        is_module = false
        is_elc = 1
#20 0x0000000000654c54 in Fautoload_do_load (fundef=XIL(0x7fb64a731193), funname=XIL(0x7fb64992cda0), macro_only=XIL(0)) at eval.c:2075
        count = 11
        kind = XIL(0)
        ignore_errors = XIL(0)
#21 0x00000000006571d1 in Ffuncall (nargs=1, args=0x7ffc5dad7d00) at eval.c:2839
        fun = XIL(0x7fb64a731193)
        original_fun = XIL(0x7fb64992cda0)
        funcar = XIL(0x2cd0)
        numargs = 0
        val = XIL(0)
        count = 10
#22 0x00000000006ad1ca in exec_byte_code (bytestr=XIL(0x7fb64a6e98b4), vector=XIL(0x7fb64a6e8685), maxdepth=make_number(23), args_template=make_number(257), nargs=1, args=0x7ffc5dad8688) at bytecode.c:632
        op = 0
        type = (unknown: 1251200684)
        targets = {0x6b0161 <exec_byte_code+15833>, 0x6b0189 <exec_byte_code+15873>, 0x6b018b <exec_byte_code+15875>, 0x6b018d <exec_byte_code+15877>, 0x6b018f <exec_byte_code+15879>, 0x6b018f <exec_byte_code+15879>, 0x6b01f4 <exec_byte_code+15980>, 0x6b0268 <exec_byte_code+16096>, 0x6ac8f2 <exec_byte_code+1386>, 0x6ac8f4 <exec_byte_code+1388>, 0x6ac8f6 <exec_byte_code+1390>, 0x6ac8f8 <exec_byte_code+1392>, 0x6ac8fa <exec_byte_code+1394>, 0x6ac8fa <exec_byte_code+1394>, 0x6ac900 <exec_byte_code+1400>, 0x6ac8c1 <exec_byte_code+1337>, 0x6acdd5 <exec_byte_code+2637>, 0x6acdd7 <exec_byte_code+2639>, 0x6acdd9 <exec_byte_code+2641>, 0x6acddb <exec_byte_code+2643>, 0x6acddd <exec_byte_code+2645>, 0x6acddd <exec_byte_code+2645>, 0x6ace12 <exec_byte_code+2698>, 0x6acde3 <exec_byte_code+2651>, 0x6ad0e7 <exec_byte_code+3423>, 0x6ad0e9 <exec_byte_code+3425>, 0x6ad0eb <exec_byte_code+3427>, 0x6ad0ed <exec_byte_code+3429>, 0x6ad0ef <exec_byte_code+3431>, 0x6ad0ef <exec_byte_code+3431>, 0x6ad0a1 <exec_byte_code+3353>, 0x6ad0b8 <exec_byte_code+3376>, 0x6ad197 <exec_byte_code+3599>, 0x6ad199 <exec_byte_code+3601>, 0x6ad19b <exec_byte_code+3603>, 0x6ad19d <exec_byte_code+3605>, 0x6ad19f <exec_byte_code+3607>, 0x6ad19f <exec_byte_code+3607>, 0x6ad151 <exec_byte_code+3529>, 0x6ad168 <exec_byte_code+3552>, 0x6ad24c <exec_byte_code+3780>, 0x6ad24e <exec_byte_code+3782>, 0x6ad250 <exec_byte_code+3784>, 0x6ad252 <exec_byte_code+3786>, 0x6ad254 <exec_byte_code+3788>, 0x6ad254 <exec_byte_code+3788>, 0x6ad206 <exec_byte_code+3710>, 0x6ad21d <exec_byte_code+3733>, 0x6adac7 <exec_byte_code+5951>, 0x6ad9ad <exec_byte_code+5669>, 0x6ad9a4 <exec_byte_code+5660>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6adcf8 <exec_byte_code+6512>, 0x6ade17 <exec_byte_code+6799>, 0x6ade81 <exec_byte_code+6905>, 0x6adeec <exec_byte_code+7012>, 0x6adf58 <exec_byte_code+7120>, 0x6acc22 <exec_byte_code+2202>, 0x6accaa <exec_byte_code+2338>, 0x6adfdc <exec_byte_code+7252>, 0x6acb60 <exec_byte_code+2008>, 0x6acd12 <exec_byte_code+2442>, 0x6ae051 <exec_byte_code+7369>, 0x6ae0b9 <exec_byte_code+7473>, 0x6ae101 <exec_byte_code+7545>, 0x6ae169 <exec_byte_code+7649>, 0x6ae1bb <exec_byte_code+7731>, 0x6ae28c <exec_byte_code+7940>, 0x6ae2d4 <exec_byte_code+8012>, 0x6ae33c <exec_byte_code+8116>, 0x6ae3c1 <exec_byte_code+8249>, 0x6ae409 <exec_byte_code+8321>, 0x6ae451 <exec_byte_code+8393>, 0x6ae4b9 <exec_byte_code+8497>, 0x6ae521 <exec_byte_code+8601>, 0x6ae589 <exec_byte_code+8705>, 0x6ae60e <exec_byte_code+8838>, 0x6ae660 <exec_byte_code+8920>, 0x6ae6b2 <exec_byte_code+9002>, 0x6ae783 <exec_byte_code+9211>, 0x6ae7fd <exec_byte_code+9333>, 0x6ae877 <exec_byte_code+9455>, 0x6aea46 <exec_byte_code+9918>, 0x6aeab3 <exec_byte_code+10027>, 0x6aeb20 <exec_byte_code+10136>, 0x6aeb8d <exec_byte_code+10245>, 0x6aebfa <exec_byte_code+10354>, 0x6aec4c <exec_byte_code+10436>, 0x6aecca <exec_byte_code+10562>, 0x6aed1c <exec_byte_code+10644>, 0x6aed6e <exec_byte_code+10726>, 0x6aedc0 <exec_byte_code+10808>, 0x6aeecc <exec_byte_code+11076>, 0x6ad827 <exec_byte_code+5279>, 0x6aef2a <exec_byte_code+11170>, 0x6aef72 <exec_byte_code+11242>, 0x6af03e <exec_byte_code+11446>, 0x6af0a7 <exec_byte_code+11551>, 0x6af105 <exec_byte_code+11645>, 0x6af14d <exec_byte_code+11717>, 0x6af193 <exec_byte_code+11787>, 0x6af1d9 <exec_byte_code+11857>, 0x6af227 <exec_byte_code+11935>, 0x6b0161 <exec_byte_code+15833>, 0x6af27f <exec_byte_code+12023>, 0x6af2c5 <exec_byte_code+12093>, 0x6af30b <exec_byte_code+12163>, 0x6af351 <exec_byte_code+12233>, 0x6af397 <exec_byte_code+12303>, 0x6af3dd <exec_byte_code+12373>, 0x6ad827 <exec_byte_code+5279>, 0x6b0161 <exec_byte_code+15833>, 0x6af425 <exec_byte_code+12445>, 0x6af47a <exec_byte_code+12530>, 0x6af4c2 <exec_byte_code+12602>, 0x6af50a <exec_byte_code+12674>, 0x6af572 <exec_byte_code+12778>, 0x6af5da <exec_byte_code+12882>, 0x6af622 <exec_byte_code+12954>, 0x6af73f <exec_byte_code+13239>, 0x6af7a7 <exec_byte_code+13343>, 0x6af80f <exec_byte_code+13447>, 0x6af877 <exec_byte_code+13551>, 0x6af8bd <exec_byte_code+13621>, 0x6b0161 <exec_byte_code+15833>, 0x6ad75b <exec_byte_code+5075>, 0x6ad303 <exec_byte_code+3963>, 0x6acace <exec_byte_code+1862>, 0x6ad3b6 <exec_byte_code+4142>, 0x6ad43a <exec_byte_code+4274>, 0x6ad4bb <exec_byte_code+4403>, 0x6ad70f <exec_byte_code+4999>, 0x6ad724 <exec_byte_code+5020>, 0x6ad04e <exec_byte_code+3270>, 0x6ad7de <exec_byte_code+5206>, 0x6ad85e <exec_byte_code+5334>, 0x6ad8ec <exec_byte_code+5476>, 0x6ad935 <exec_byte_code+5549>, 0x6adb13 <exec_byte_code+6027>, 0x6adb90 <exec_byte_code+6152>, 0x6adc15 <exec_byte_code+6285>, 0x6adc75 <exec_byte_code+6381>, 0x6ad2b5 <exec_byte_code+3885>, 0x6af905 <exec_byte_code+13693>, 0x6af98a <exec_byte_code+13826>, 0x6af9d2 <exec_byte_code+13898>, 0x6afa1a <exec_byte_code+13970>, 0x6afa62 <exec_byte_code+14042>, 0x6afaaa <exec_byte_code+14114>, 0x6afb12 <exec_byte_code+14218>, 0x6afb7a <exec_byte_code+14322>, 0x6afbe2 <exec_byte_code+14426>, 0x6afc4a <exec_byte_code+14530>, 0x6afdc0 <exec_byte_code+14904>, 0x6afe28 <exec_byte_code+15008>, 0x6afe90 <exec_byte_code+15112>, 0x6afed8 <exec_byte_code+15184>, 0x6aff40 <exec_byte_code+15288>, 0x6affa8 <exec_byte_code+15392>, 0x6afff0 <exec_byte_code+15464>, 0x6b0038 <exec_byte_code+15536>, 0x6aee12 <exec_byte_code+10890>, 0x6aee64 <exec_byte_code+10972>, 0x6b008a <exec_byte_code+15618>, 0x6b00f6 <exec_byte_code+15726>, 0x6b0161 <exec_byte_code+15833>, 0x6ad53c <exec_byte_code+4532>, 0x6ad559 <exec_byte_code+4561>, 0x6ad5c8 <exec_byte_code+4672>, 0x6ad637 <exec_byte_code+4783>, 0x6ad6a3 <exec_byte_code+4891>, 0x6ae20d <exec_byte_code+7813>, 0x6ae704 <exec_byte_code+9084>, 0x6aefbf <exec_byte_code+11319>, 0x6b02fb <exec_byte_code+16243>, 0x6b0370 <exec_byte_code+16360>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0406 <exec_byte_code+16510>, 0x6b048d <exec_byte_code+16645>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b06b7 <exec_byte_code+17199> <repeats 64 times>}
        const_length = 106
        bytestr_length = 1176
        vectorp = 0x7fb64a6e8688
        quitcounter = 104 'h'
        stack_items = 24
        sa_avail = 15016
        sa_count = 8
        sa_must_free = false
        alloc = 0x7ffc5dad7c90
        item_bytes = 192
        stack_base = 0x7ffc5dad7c90
        top = 0x7ffc5dad7d00
        stack_lim = 0x7ffc5dad7d50
        bytestr_data = 0x7ffc5dad7d50 "\306 \210\b\203\021"
        pc = 0x7ffc5dad7e88 "\210\202d\003\016A㝃m\001\332\026B\001\206O\001\n\211A\022\242\344\001!\211@\001A\211\004GU\204e\001\333\345\005\003\313O\"\210\346\002!\266\005\202d\003\016A睃\345\001\313\350\351\016C\"\003\206\203\001\n\211A\022\242\211\262\r\313\332\036D\322\003\003\003#)\266\203\203\237\001\006\n\327\313O\262\vڲ\001\352\353\006\f!!\262\v\211\203\300\001\314\016E\006\fC\"\026E\006\t\203\341\001\016E\262\n\202\341\001\006\t\203\327\001\006\t\006\v\006\vAB\241\210\006\tA\262\n\202\341\001\006\n\016EB\211\026E\262\n\210\202d\003\016A읃\r\002\353\002\206\366\001\n\211A\022\242!\352\001!\355\001!\203\003\002"...
        count = 8
        result = XIL(0)
#23 0x0000000000657c82 in funcall_lambda (fun=XIL(0x7fb64a6e8655), nargs=1, arg_vector=0x7ffc5dad8680) at eval.c:3023
        size = 5
        val = XIL(0x1200e043c0)
        syms_left = make_number(257)
        next = XIL(0x62ee70)
        lexenv = XIL(0x7ffc5dad85b0)
        count = 8
        i = 0
        optional = false
        rest = false
        previous_optional_or_rest = 127
#24 0x0000000000657086 in Ffuncall (nargs=2, args=0x7ffc5dad8678) at eval.c:2824
        fun = XIL(0x7fb64a6e8655)
        original_fun = XIL(0x7fb6498e4260)
        funcar = make_number(1623929)
        numargs = 1
        val = XIL(0x7ffc5dad8660)
        count = 7
#25 0x00000000006ad1ca in exec_byte_code (bytestr=XIL(0x7fb64a6ed05c), vector=XIL(0x7fb64a6e9af5), maxdepth=make_number(20), args_template=make_number(0), nargs=0, args=0x7ffc5dad9268) at bytecode.c:632
        op = 1
        type = CONDITION_CASE
        targets = {0x6b0161 <exec_byte_code+15833>, 0x6b0189 <exec_byte_code+15873>, 0x6b018b <exec_byte_code+15875>, 0x6b018d <exec_byte_code+15877>, 0x6b018f <exec_byte_code+15879>, 0x6b018f <exec_byte_code+15879>, 0x6b01f4 <exec_byte_code+15980>, 0x6b0268 <exec_byte_code+16096>, 0x6ac8f2 <exec_byte_code+1386>, 0x6ac8f4 <exec_byte_code+1388>, 0x6ac8f6 <exec_byte_code+1390>, 0x6ac8f8 <exec_byte_code+1392>, 0x6ac8fa <exec_byte_code+1394>, 0x6ac8fa <exec_byte_code+1394>, 0x6ac900 <exec_byte_code+1400>, 0x6ac8c1 <exec_byte_code+1337>, 0x6acdd5 <exec_byte_code+2637>, 0x6acdd7 <exec_byte_code+2639>, 0x6acdd9 <exec_byte_code+2641>, 0x6acddb <exec_byte_code+2643>, 0x6acddd <exec_byte_code+2645>, 0x6acddd <exec_byte_code+2645>, 0x6ace12 <exec_byte_code+2698>, 0x6acde3 <exec_byte_code+2651>, 0x6ad0e7 <exec_byte_code+3423>, 0x6ad0e9 <exec_byte_code+3425>, 0x6ad0eb <exec_byte_code+3427>, 0x6ad0ed <exec_byte_code+3429>, 0x6ad0ef <exec_byte_code+3431>, 0x6ad0ef <exec_byte_code+3431>, 0x6ad0a1 <exec_byte_code+3353>, 0x6ad0b8 <exec_byte_code+3376>, 0x6ad197 <exec_byte_code+3599>, 0x6ad199 <exec_byte_code+3601>, 0x6ad19b <exec_byte_code+3603>, 0x6ad19d <exec_byte_code+3605>, 0x6ad19f <exec_byte_code+3607>, 0x6ad19f <exec_byte_code+3607>, 0x6ad151 <exec_byte_code+3529>, 0x6ad168 <exec_byte_code+3552>, 0x6ad24c <exec_byte_code+3780>, 0x6ad24e <exec_byte_code+3782>, 0x6ad250 <exec_byte_code+3784>, 0x6ad252 <exec_byte_code+3786>, 0x6ad254 <exec_byte_code+3788>, 0x6ad254 <exec_byte_code+3788>, 0x6ad206 <exec_byte_code+3710>, 0x6ad21d <exec_byte_code+3733>, 0x6adac7 <exec_byte_code+5951>, 0x6ad9ad <exec_byte_code+5669>, 0x6ad9a4 <exec_byte_code+5660>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6adcf8 <exec_byte_code+6512>, 0x6ade17 <exec_byte_code+6799>, 0x6ade81 <exec_byte_code+6905>, 0x6adeec <exec_byte_code+7012>, 0x6adf58 <exec_byte_code+7120>, 0x6acc22 <exec_byte_code+2202>, 0x6accaa <exec_byte_code+2338>, 0x6adfdc <exec_byte_code+7252>, 0x6acb60 <exec_byte_code+2008>, 0x6acd12 <exec_byte_code+2442>, 0x6ae051 <exec_byte_code+7369>, 0x6ae0b9 <exec_byte_code+7473>, 0x6ae101 <exec_byte_code+7545>, 0x6ae169 <exec_byte_code+7649>, 0x6ae1bb <exec_byte_code+7731>, 0x6ae28c <exec_byte_code+7940>, 0x6ae2d4 <exec_byte_code+8012>, 0x6ae33c <exec_byte_code+8116>, 0x6ae3c1 <exec_byte_code+8249>, 0x6ae409 <exec_byte_code+8321>, 0x6ae451 <exec_byte_code+8393>, 0x6ae4b9 <exec_byte_code+8497>, 0x6ae521 <exec_byte_code+8601>, 0x6ae589 <exec_byte_code+8705>, 0x6ae60e <exec_byte_code+8838>, 0x6ae660 <exec_byte_code+8920>, 0x6ae6b2 <exec_byte_code+9002>, 0x6ae783 <exec_byte_code+9211>, 0x6ae7fd <exec_byte_code+9333>, 0x6ae877 <exec_byte_code+9455>, 0x6aea46 <exec_byte_code+9918>, 0x6aeab3 <exec_byte_code+10027>, 0x6aeb20 <exec_byte_code+10136>, 0x6aeb8d <exec_byte_code+10245>, 0x6aebfa <exec_byte_code+10354>, 0x6aec4c <exec_byte_code+10436>, 0x6aecca <exec_byte_code+10562>, 0x6aed1c <exec_byte_code+10644>, 0x6aed6e <exec_byte_code+10726>, 0x6aedc0 <exec_byte_code+10808>, 0x6aeecc <exec_byte_code+11076>, 0x6ad827 <exec_byte_code+5279>, 0x6aef2a <exec_byte_code+11170>, 0x6aef72 <exec_byte_code+11242>, 0x6af03e <exec_byte_code+11446>, 0x6af0a7 <exec_byte_code+11551>, 0x6af105 <exec_byte_code+11645>, 0x6af14d <exec_byte_code+11717>, 0x6af193 <exec_byte_code+11787>, 0x6af1d9 <exec_byte_code+11857>, 0x6af227 <exec_byte_code+11935>, 0x6b0161 <exec_byte_code+15833>, 0x6af27f <exec_byte_code+12023>, 0x6af2c5 <exec_byte_code+12093>, 0x6af30b <exec_byte_code+12163>, 0x6af351 <exec_byte_code+12233>, 0x6af397 <exec_byte_code+12303>, 0x6af3dd <exec_byte_code+12373>, 0x6ad827 <exec_byte_code+5279>, 0x6b0161 <exec_byte_code+15833>, 0x6af425 <exec_byte_code+12445>, 0x6af47a <exec_byte_code+12530>, 0x6af4c2 <exec_byte_code+12602>, 0x6af50a <exec_byte_code+12674>, 0x6af572 <exec_byte_code+12778>, 0x6af5da <exec_byte_code+12882>, 0x6af622 <exec_byte_code+12954>, 0x6af73f <exec_byte_code+13239>, 0x6af7a7 <exec_byte_code+13343>, 0x6af80f <exec_byte_code+13447>, 0x6af877 <exec_byte_code+13551>, 0x6af8bd <exec_byte_code+13621>, 0x6b0161 <exec_byte_code+15833>, 0x6ad75b <exec_byte_code+5075>, 0x6ad303 <exec_byte_code+3963>, 0x6acace <exec_byte_code+1862>, 0x6ad3b6 <exec_byte_code+4142>, 0x6ad43a <exec_byte_code+4274>, 0x6ad4bb <exec_byte_code+4403>, 0x6ad70f <exec_byte_code+4999>, 0x6ad724 <exec_byte_code+5020>, 0x6ad04e <exec_byte_code+3270>, 0x6ad7de <exec_byte_code+5206>, 0x6ad85e <exec_byte_code+5334>, 0x6ad8ec <exec_byte_code+5476>, 0x6ad935 <exec_byte_code+5549>, 0x6adb13 <exec_byte_code+6027>, 0x6adb90 <exec_byte_code+6152>, 0x6adc15 <exec_byte_code+6285>, 0x6adc75 <exec_byte_code+6381>, 0x6ad2b5 <exec_byte_code+3885>, 0x6af905 <exec_byte_code+13693>, 0x6af98a <exec_byte_code+13826>, 0x6af9d2 <exec_byte_code+13898>, 0x6afa1a <exec_byte_code+13970>, 0x6afa62 <exec_byte_code+14042>, 0x6afaaa <exec_byte_code+14114>, 0x6afb12 <exec_byte_code+14218>, 0x6afb7a <exec_byte_code+14322>, 0x6afbe2 <exec_byte_code+14426>, 0x6afc4a <exec_byte_code+14530>, 0x6afdc0 <exec_byte_code+14904>, 0x6afe28 <exec_byte_code+15008>, 0x6afe90 <exec_byte_code+15112>, 0x6afed8 <exec_byte_code+15184>, 0x6aff40 <exec_byte_code+15288>, 0x6affa8 <exec_byte_code+15392>, 0x6afff0 <exec_byte_code+15464>, 0x6b0038 <exec_byte_code+15536>, 0x6aee12 <exec_byte_code+10890>, 0x6aee64 <exec_byte_code+10972>, 0x6b008a <exec_byte_code+15618>, 0x6b00f6 <exec_byte_code+15726>, 0x6b0161 <exec_byte_code+15833>, 0x6ad53c <exec_byte_code+4532>, 0x6ad559 <exec_byte_code+4561>, 0x6ad5c8 <exec_byte_code+4672>, 0x6ad637 <exec_byte_code+4783>, 0x6ad6a3 <exec_byte_code+4891>, 0x6ae20d <exec_byte_code+7813>, 0x6ae704 <exec_byte_code+9084>, 0x6aefbf <exec_byte_code+11319>, 0x6b02fb <exec_byte_code+16243>, 0x6b0370 <exec_byte_code+16360>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0406 <exec_byte_code+16510>, 0x6b048d <exec_byte_code+16645>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b06b7 <exec_byte_code+17199> <repeats 64 times>}
        const_length = 209
        bytestr_length = 1715
        vectorp = 0x7fb64a6e9af8
        quitcounter = 10 '\n'
        stack_items = 21
        sa_avail = 14501
        sa_count = 7
        sa_must_free = false
        alloc = 0x7ffc5dad8670
        item_bytes = 168
        stack_base = 0x7ffc5dad8670
        top = 0x7ffc5dad8678
        stack_lim = 0x7ffc5dad8718
        bytestr_data = 0x7ffc5dad8718 "\306 \020\307\021\n\023\307\024\310\311!\211\307=\204\060"
        pc = 0x7ffc5dad8d1a "\210\307\016@\211\203L\006\211@\002\204E\006\211;\203E\006\201", <incomplete sequence \310>
        count = 7
        result = XIL(0x7ffc5dad9030)
#26 0x0000000000657c82 in funcall_lambda (fun=XIL(0x7fb64a6e9ac5), nargs=0, arg_vector=0x7ffc5dad9268) at eval.c:3023
        size = 5
        val = XIL(0x1200e043c0)
        syms_left = make_number(0)
        next = XIL(0xd89480)
        lexenv = XIL(0x4)
        count = 7
        i = 0
        optional = false
        rest = false
        previous_optional_or_rest = 127
#27 0x0000000000657086 in Ffuncall (nargs=1, args=0x7ffc5dad9260) at eval.c:2824
        fun = XIL(0x7fb64a6e9ac5)
        original_fun = XIL(0x7fb6498e56d0)
        funcar = make_number(1661816)
        numargs = 0
        val = XIL(0x7ffc5dad9240)
        count = 6
#28 0x00000000006ad1ca in exec_byte_code (bytestr=XIL(0x7fb64a6edc9c), vector=XIL(0x7fb64a6ed21d), maxdepth=make_number(12), args_template=make_number(0), nargs=0, args=0x7ffc5dad98a0) at bytecode.c:632
        op = 0
        type = (unknown: 1571657732)
        targets = {0x6b0161 <exec_byte_code+15833>, 0x6b0189 <exec_byte_code+15873>, 0x6b018b <exec_byte_code+15875>, 0x6b018d <exec_byte_code+15877>, 0x6b018f <exec_byte_code+15879>, 0x6b018f <exec_byte_code+15879>, 0x6b01f4 <exec_byte_code+15980>, 0x6b0268 <exec_byte_code+16096>, 0x6ac8f2 <exec_byte_code+1386>, 0x6ac8f4 <exec_byte_code+1388>, 0x6ac8f6 <exec_byte_code+1390>, 0x6ac8f8 <exec_byte_code+1392>, 0x6ac8fa <exec_byte_code+1394>, 0x6ac8fa <exec_byte_code+1394>, 0x6ac900 <exec_byte_code+1400>, 0x6ac8c1 <exec_byte_code+1337>, 0x6acdd5 <exec_byte_code+2637>, 0x6acdd7 <exec_byte_code+2639>, 0x6acdd9 <exec_byte_code+2641>, 0x6acddb <exec_byte_code+2643>, 0x6acddd <exec_byte_code+2645>, 0x6acddd <exec_byte_code+2645>, 0x6ace12 <exec_byte_code+2698>, 0x6acde3 <exec_byte_code+2651>, 0x6ad0e7 <exec_byte_code+3423>, 0x6ad0e9 <exec_byte_code+3425>, 0x6ad0eb <exec_byte_code+3427>, 0x6ad0ed <exec_byte_code+3429>, 0x6ad0ef <exec_byte_code+3431>, 0x6ad0ef <exec_byte_code+3431>, 0x6ad0a1 <exec_byte_code+3353>, 0x6ad0b8 <exec_byte_code+3376>, 0x6ad197 <exec_byte_code+3599>, 0x6ad199 <exec_byte_code+3601>, 0x6ad19b <exec_byte_code+3603>, 0x6ad19d <exec_byte_code+3605>, 0x6ad19f <exec_byte_code+3607>, 0x6ad19f <exec_byte_code+3607>, 0x6ad151 <exec_byte_code+3529>, 0x6ad168 <exec_byte_code+3552>, 0x6ad24c <exec_byte_code+3780>, 0x6ad24e <exec_byte_code+3782>, 0x6ad250 <exec_byte_code+3784>, 0x6ad252 <exec_byte_code+3786>, 0x6ad254 <exec_byte_code+3788>, 0x6ad254 <exec_byte_code+3788>, 0x6ad206 <exec_byte_code+3710>, 0x6ad21d <exec_byte_code+3733>, 0x6adac7 <exec_byte_code+5951>, 0x6ad9ad <exec_byte_code+5669>, 0x6ad9a4 <exec_byte_code+5660>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6adcf8 <exec_byte_code+6512>, 0x6ade17 <exec_byte_code+6799>, 0x6ade81 <exec_byte_code+6905>, 0x6adeec <exec_byte_code+7012>, 0x6adf58 <exec_byte_code+7120>, 0x6acc22 <exec_byte_code+2202>, 0x6accaa <exec_byte_code+2338>, 0x6adfdc <exec_byte_code+7252>, 0x6acb60 <exec_byte_code+2008>, 0x6acd12 <exec_byte_code+2442>, 0x6ae051 <exec_byte_code+7369>, 0x6ae0b9 <exec_byte_code+7473>, 0x6ae101 <exec_byte_code+7545>, 0x6ae169 <exec_byte_code+7649>, 0x6ae1bb <exec_byte_code+7731>, 0x6ae28c <exec_byte_code+7940>, 0x6ae2d4 <exec_byte_code+8012>, 0x6ae33c <exec_byte_code+8116>, 0x6ae3c1 <exec_byte_code+8249>, 0x6ae409 <exec_byte_code+8321>, 0x6ae451 <exec_byte_code+8393>, 0x6ae4b9 <exec_byte_code+8497>, 0x6ae521 <exec_byte_code+8601>, 0x6ae589 <exec_byte_code+8705>, 0x6ae60e <exec_byte_code+8838>, 0x6ae660 <exec_byte_code+8920>, 0x6ae6b2 <exec_byte_code+9002>, 0x6ae783 <exec_byte_code+9211>, 0x6ae7fd <exec_byte_code+9333>, 0x6ae877 <exec_byte_code+9455>, 0x6aea46 <exec_byte_code+9918>, 0x6aeab3 <exec_byte_code+10027>, 0x6aeb20 <exec_byte_code+10136>, 0x6aeb8d <exec_byte_code+10245>, 0x6aebfa <exec_byte_code+10354>, 0x6aec4c <exec_byte_code+10436>, 0x6aecca <exec_byte_code+10562>, 0x6aed1c <exec_byte_code+10644>, 0x6aed6e <exec_byte_code+10726>, 0x6aedc0 <exec_byte_code+10808>, 0x6aeecc <exec_byte_code+11076>, 0x6ad827 <exec_byte_code+5279>, 0x6aef2a <exec_byte_code+11170>, 0x6aef72 <exec_byte_code+11242>, 0x6af03e <exec_byte_code+11446>, 0x6af0a7 <exec_byte_code+11551>, 0x6af105 <exec_byte_code+11645>, 0x6af14d <exec_byte_code+11717>, 0x6af193 <exec_byte_code+11787>, 0x6af1d9 <exec_byte_code+11857>, 0x6af227 <exec_byte_code+11935>, 0x6b0161 <exec_byte_code+15833>, 0x6af27f <exec_byte_code+12023>, 0x6af2c5 <exec_byte_code+12093>, 0x6af30b <exec_byte_code+12163>, 0x6af351 <exec_byte_code+12233>, 0x6af397 <exec_byte_code+12303>, 0x6af3dd <exec_byte_code+12373>, 0x6ad827 <exec_byte_code+5279>, 0x6b0161 <exec_byte_code+15833>, 0x6af425 <exec_byte_code+12445>, 0x6af47a <exec_byte_code+12530>, 0x6af4c2 <exec_byte_code+12602>, 0x6af50a <exec_byte_code+12674>, 0x6af572 <exec_byte_code+12778>, 0x6af5da <exec_byte_code+12882>, 0x6af622 <exec_byte_code+12954>, 0x6af73f <exec_byte_code+13239>, 0x6af7a7 <exec_byte_code+13343>, 0x6af80f <exec_byte_code+13447>, 0x6af877 <exec_byte_code+13551>, 0x6af8bd <exec_byte_code+13621>, 0x6b0161 <exec_byte_code+15833>, 0x6ad75b <exec_byte_code+5075>, 0x6ad303 <exec_byte_code+3963>, 0x6acace <exec_byte_code+1862>, 0x6ad3b6 <exec_byte_code+4142>, 0x6ad43a <exec_byte_code+4274>, 0x6ad4bb <exec_byte_code+4403>, 0x6ad70f <exec_byte_code+4999>, 0x6ad724 <exec_byte_code+5020>, 0x6ad04e <exec_byte_code+3270>, 0x6ad7de <exec_byte_code+5206>, 0x6ad85e <exec_byte_code+5334>, 0x6ad8ec <exec_byte_code+5476>, 0x6ad935 <exec_byte_code+5549>, 0x6adb13 <exec_byte_code+6027>, 0x6adb90 <exec_byte_code+6152>, 0x6adc15 <exec_byte_code+6285>, 0x6adc75 <exec_byte_code+6381>, 0x6ad2b5 <exec_byte_code+3885>, 0x6af905 <exec_byte_code+13693>, 0x6af98a <exec_byte_code+13826>, 0x6af9d2 <exec_byte_code+13898>, 0x6afa1a <exec_byte_code+13970>, 0x6afa62 <exec_byte_code+14042>, 0x6afaaa <exec_byte_code+14114>, 0x6afb12 <exec_byte_code+14218>, 0x6afb7a <exec_byte_code+14322>, 0x6afbe2 <exec_byte_code+14426>, 0x6afc4a <exec_byte_code+14530>, 0x6afdc0 <exec_byte_code+14904>, 0x6afe28 <exec_byte_code+15008>, 0x6afe90 <exec_byte_code+15112>, 0x6afed8 <exec_byte_code+15184>, 0x6aff40 <exec_byte_code+15288>, 0x6affa8 <exec_byte_code+15392>, 0x6afff0 <exec_byte_code+15464>, 0x6b0038 <exec_byte_code+15536>, 0x6aee12 <exec_byte_code+10890>, 0x6aee64 <exec_byte_code+10972>, 0x6b008a <exec_byte_code+15618>, 0x6b00f6 <exec_byte_code+15726>, 0x6b0161 <exec_byte_code+15833>, 0x6ad53c <exec_byte_code+4532>, 0x6ad559 <exec_byte_code+4561>, 0x6ad5c8 <exec_byte_code+4672>, 0x6ad637 <exec_byte_code+4783>, 0x6ad6a3 <exec_byte_code+4891>, 0x6ae20d <exec_byte_code+7813>, 0x6ae704 <exec_byte_code+9084>, 0x6aefbf <exec_byte_code+11319>, 0x6b02fb <exec_byte_code+16243>, 0x6b0370 <exec_byte_code+16360>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0406 <exec_byte_code+16510>, 0x6b048d <exec_byte_code+16645>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b0161 <exec_byte_code+15833>, 0x6b06b7 <exec_byte_code+17199> <repeats 64 times>}
        const_length = 77
        bytestr_length = 443
        vectorp = 0x7fb64a6ed220
        quitcounter = 39 '\''
        stack_items = 13
        sa_avail = 15837
        sa_count = 5
        sa_must_free = false
        alloc = 0x7ffc5dad9250
        item_bytes = 104
        stack_base = 0x7ffc5dad9250
        top = 0x7ffc5dad9260
        stack_lim = 0x7ffc5dad92b8
        bytestr_data = 0x7ffc5dad92b8 "\b\203\b"
        pc = 0x7ffc5dad942c "\210)\210\375\376\377\"\210\201H"
        count = 5
        result = XIL(0x5)
#29 0x0000000000657c82 in funcall_lambda (fun=XIL(0x7fb64a6ed1ed), nargs=0, arg_vector=0x7ffc5dad98a0) at eval.c:3023
        size = 5
        val = XIL(0)
        syms_left = make_number(0)
        next = XIL(0)
        lexenv = XIL(0)
        count = 5
        i = 6483568
        optional = false
        rest = false
        previous_optional_or_rest = 127
#30 0x00000000006578b9 in apply_lambda (fun=XIL(0x7fb64a6ed1ed), args=XIL(0), count=4) at eval.c:2959
        args_left = XIL(0)
        i = 0
        numargs = 0
        arg_vector = 0x7ffc5dad98a0
        tem = make_number(1444944)
        sa_avail = 16384
        sa_count = 5
        sa_must_free = false
#31 0x0000000000655a66 in eval_sub (form=XIL(0x7fb64a83c86b)) at eval.c:2332
        fun = XIL(0x7fb64a6ed1ed)
        val = XIL(0x28ec250)
        original_fun = XIL(0x7fb6498e8df8)
        original_args = XIL(0)
        funcar = make_number(1663778)
        count = 4
        argvals = {XIL(0x118), XIL(0x7fb64a85ea50), XIL(0x7fb64a1e3500), XIL(0), XIL(0x7ffc5dad99c0), XIL(0xe0bf50), XIL(0x584ae0), XIL(0)}
#32 0x0000000000654dd5 in Feval (form=XIL(0x7fb64a83c86b), lexical=XIL(0)) at eval.c:2107
        count = 3
#33 0x000000000058df4e in top_level_2 () at keyboard.c:1122
No locals.
#34 0x0000000000652e25 in internal_condition_case (bfun=0x58df2b <top_level_2>, handlers=XIL(0x53d0), hfun=0x58d921 <cmd_error>) at eval.c:1336
        val = XIL(0x5825cb)
        c = 0x291b590
#35 0x000000000058df96 in top_level_1 (ignore=XIL(0)) at keyboard.c:1130
No locals.
#36 0x000000000065231b in internal_catch (tag=XIL(0xccc0), func=0x58df50 <top_level_1>, arg=XIL(0)) at eval.c:1101
        val = XIL(0)
        c = 0x291b470
#37 0x000000000058de77 in command_loop () at keyboard.c:1091
No locals.
#38 0x000000000058d408 in recursive_edit_1 () at keyboard.c:698
        count = 1
        val = XIL(0x7ffc5dad9bc0)
#39 0x000000000058d600 in Frecursive_edit () at keyboard.c:769
        count = 0
        buffer = XIL(0)
#40 0x000000000058b446 in main (argc=9, argv=0x7ffc5dad9e68) at emacs.c:1915
        stack_bottom_variable = 0x28d1f20
        do_initial_setlocale = true
        skip_args = 2
        no_loadup = false
        junk = 0x0
        dname_arg = 0x0
        ch_to_dir = 0x0
        original_pwd = 0x0
        argv0_base = 0x7ffc5dadc248 "emacs"
        is_temacs = false
        loaded_dump = 0x28b54d0 "../src/emacs.pdmp"
        dump_mode = 0x0
        disable_aslr = false
        rlim = {
          rlim_cur = 10022912, 
          rlim_max = 18446744073709551615
        }
        sockfd = -1
        module_assertions = false
You can't do that without a process to debug.

Re: Preview: portable dumper

[Daniel Colascione]

On 02/14/2018 01:03 PM, Philipp Stephani wrote:
> 
> 
> Eli Zaretskii <eliz@gnu.org <mailto:eliz@gnu.org>> schrieb am Di., 13. 
> Feb. 2018 um 17:39 Uhr:
> 
>      > From: Daniel Colascione <dancol@dancol.org
>     <mailto:dancol@dancol.org>>
>      > Date: Mon, 12 Feb 2018 12:18:36 -0800
>      >
>      > I've pushed the portable dumper to the pdumper Savannah branch. It
>      > should support a fully PIC Emacs.
> 
>     Thanks.  I'd urge people to try this branch and report any issues they
>     see.
> 
> 
> Got a crash when building on Debian testing. I've attached the output of 
> 'bt full'.

Thanks. There were two bugs, both relating to conservative GC:

1) To make conservative GC work, we record in the dump the start 
position and Lisp type of each object in the dump. The problem was that 
we were recording object-start records for objects in the discardable 
region. With the right stack garbage as input, conservative GC would 
find one of these object-start records, use it as evidence that it was 
looking at a real object, then call mark_object, which died because it 
knows that objects in the discardable region shouldn't be live. Fixed by 
just not emitting these discardable-region object-start records anymore.

2) When evaluating a Lisp_Object on the stack in mark_maybe_object, 
occasionally, we can find a word with a valid pointer component but the 
wrong type tags. Now we check that the Lisp_Object's tagged type matches 
the actual type of the object in the dump.

I don't know why it was so hard to repro these on my machine. I had to 
bootstrap a few times before Emacs began crashing.

Re: Preview: portable dumper

[Ken Brown]

On 2/13/2018 11:37 AM, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 12 Feb 2018 12:18:36 -0800
>>
>> I've pushed the portable dumper to the pdumper Savannah branch. It
>> should support a fully PIC Emacs.
> 
> Thanks.  I'd urge people to try this branch and report any issues they
> see.

I just tried to build on 64-bit Cygwin, and the build fails as follows:

[...]
Dumping under the name bootstrap-emacs.pdmp
dumping fingerprint: 
923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
Dump complete
Byte counts: header=80 hot=13187392 discardable=119424 cold=9086640
Reloc counts: hot=919268 discardable=5790
make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
make[2]: Entering directory '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
   ELC      ../../pdumper/lisp/emacs-lisp/macroexp.elc
   ELC      ../../pdumper/lisp/emacs-lisp/cconv.elc
   ELC      ../../pdumper/lisp/emacs-lisp/byte-opt.elc
   ELC      ../../pdumper/lisp/emacs-lisp/bytecomp.elc
emacs: could not load dump file "../src/bootstrap-emacs.pdmp": out of memory

There's probably some obvious explanation, but I don't see it at the moment.

Ken

Re: Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/html, Size: 2308 bytes --]

Re: Preview: portable dumper

[Ken Brown]

On 2/15/2018 6:36 PM, Daniel Colascione wrote:
> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
> 
>     On 2/13/2018 11:37 AM, Eli Zaretskii wrote:
>      >> From: Daniel Colascione <dancol@dancol.org>
>      >> Date: Mon, 12 Feb 2018 12:18:36 -0800
>      >>
>      >> I've pushed the portable dumper to the pdumper Savannah branch. It
>      >> should support a fully PIC Emacs.
>      >
>      > Thanks.  I'd urge people to try this branch and report any issues
>     they
>      > see.
> 
>     I just tried to build on 64-bit Cygwin, and the build fails as follows:
> 
>     [...]
>     Dumping under the name bootstrap-emacs.pdmp
>     dumping fingerprint:
>     923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>     Dump complete
>     Byte counts: header=80 hot=13187392 discardable=119424 cold=9086640
>     Reloc counts: hot=919268 discardable=5790
>     make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
>     make[2]: Entering directory
>     '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>         ELC      ../../pdumper/lisp/emacs-lisp/macroexp.elc
>         ELC      ../../pdumper/lisp/emacs-lisp/cconv.elc
>         ELC      ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>         ELC      ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>     emacs: could not load dump file "../src/bootstrap-emacs.pdmp": out
>     of memory
> 
>     There's probably some obvious explanation, but I don't see it at the
>     moment.
> 
> 
> I'm not entirely surprised to see Cygwin fall over here. We could just 
> use the Windows memory mapping functions directly, but I'd prefer to 
> stick with the POSIX API if we can make it work.

I agree.

> Any idea where in the 
> mmap sequence we fail? 

I haven't looked at the code yet, so I don't understand the question. 
If you give me some guidance, I'll try to investigate.

Ken

Re: Preview: portable dumper

[Daniel Colascione]

On 02/15/2018 05:56 PM, Ken Brown wrote:
> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
>>
>>     On 2/13/2018 11:37 AM, Eli Zaretskii wrote:
>>      >> From: Daniel Colascione <dancol@dancol.org>
>>      >> Date: Mon, 12 Feb 2018 12:18:36 -0800
>>      >>
>>      >> I've pushed the portable dumper to the pdumper Savannah 
>> branch. It
>>      >> should support a fully PIC Emacs.
>>      >
>>      > Thanks.  I'd urge people to try this branch and report any issues
>>     they
>>      > see.
>>
>>     I just tried to build on 64-bit Cygwin, and the build fails as 
>> follows:
>>
>>     [...]
>>     Dumping under the name bootstrap-emacs.pdmp
>>     dumping fingerprint:
>>     923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>>     Dump complete
>>     Byte counts: header=80 hot=13187392 discardable=119424 cold=9086640
>>     Reloc counts: hot=919268 discardable=5790
>>     make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
>>     make[2]: Entering directory
>>     '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>>         ELC      ../../pdumper/lisp/emacs-lisp/macroexp.elc
>>         ELC      ../../pdumper/lisp/emacs-lisp/cconv.elc
>>         ELC      ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>>         ELC      ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>>     emacs: could not load dump file "../src/bootstrap-emacs.pdmp": out
>>     of memory
>>
>>     There's probably some obvious explanation, but I don't see it at the
>>     moment.
>>
>>
>> I'm not entirely surprised to see Cygwin fall over here. We could just 
>> use the Windows memory mapping functions directly, but I'd prefer to 
>> stick with the POSIX API if we can make it work.
> 
> I agree.
> 
>> Any idea where in the mmap sequence we fail? 
> 
> I haven't looked at the code yet, so I don't understand the question. If 
> you give me some guidance, I'll try to investigate.

Thanks. I think the trouble must be in dump_mmap_contiguous. We report 
all errors from this function as "out of memory". dump_mmap_contiguous 
takes an array of mapping descriptors (think ELF segments or something) 
and maps them all into a single contiguous region of virtual memory. On 
POSIX systems, we reserve a chunk of address space with a big PROT_NONE 
anonymous mapping, then carve it up into the separate mappings that we 
really want. Windows doesn't support atomic mmap replacement, so Cygwin 
has to emulate it, and I wouldn't be surprised if something's going 
wrong in this emulation.

That you're able to compile a few elisp files before bootstrap starts 
failing makes me wonder whether we're dealing with Cygwin's perennial 
problem with address space collisions and some kind of dynamic-base DLL.

Re: Preview: portable dumper

[Ken Brown]

On 2/15/2018 9:36 PM, Daniel Colascione wrote:
> On 02/15/2018 05:56 PM, Ken Brown wrote:
>> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
[...]
>>>     I just tried to build on 64-bit Cygwin, and the build fails as 
>>> follows:
>>>
>>>     [...]
>>>     Dumping under the name bootstrap-emacs.pdmp
>>>     dumping fingerprint:
>>>     923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>>>     Dump complete
>>>     Byte counts: header=80 hot=13187392 discardable=119424 cold=9086640
>>>     Reloc counts: hot=919268 discardable=5790
>>>     make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
>>>     make[2]: Entering directory
>>>     '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>>>         ELC      ../../pdumper/lisp/emacs-lisp/macroexp.elc
>>>         ELC      ../../pdumper/lisp/emacs-lisp/cconv.elc
>>>         ELC      ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>>>         ELC      ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>>>     emacs: could not load dump file "../src/bootstrap-emacs.pdmp": out
>>>     of memory
>>>
>>>     There's probably some obvious explanation, but I don't see it at the
>>>     moment.
>>>
>>>
>>> I'm not entirely surprised to see Cygwin fall over here. We could 
>>> just use the Windows memory mapping functions directly, but I'd 
>>> prefer to stick with the POSIX API if we can make it work.
>>
>> I agree.
>>
>>> Any idea where in the mmap sequence we fail? 
>>
>> I haven't looked at the code yet, so I don't understand the question. 
>> If you give me some guidance, I'll try to investigate.
> 
> Thanks. I think the trouble must be in dump_mmap_contiguous. We report 
> all errors from this function as "out of memory". dump_mmap_contiguous 
> takes an array of mapping descriptors (think ELF segments or something) 
> and maps them all into a single contiguous region of virtual memory. On 
> POSIX systems, we reserve a chunk of address space with a big PROT_NONE 
> anonymous mapping, then carve it up into the separate mappings that we 
> really want. Windows doesn't support atomic mmap replacement, so Cygwin 
> has to emulate it, and I wouldn't be surprised if something's going 
> wrong in this emulation.

You're right that the problem is in dump_mmap_contiguous.  I stepped 
through it in gdb and found the following:

1. The call to dump_anonymous_allocate at pdumper.c:4432 succeeds.

2. We enter the loop at pdumper.c:4451 with i = 0.  dump_map_file is 
called, which calls dump_map_file_posix with protection = 
DUMP_MEMORY_ACCESS_READWRITE.

3. This calls mmap at line 4219 with mem_prot = PROT_READ | PROT_WRITE 
and mem_flags = MAP_PRIVATE | MAP_FIXED.

4. This mmap call fails.

I don't know if this is a bug in Cygwin's mmap or simply a limitation 
that has to be worked around.  Should I take this to the Cygwin mailing 
list, or do you have other ideas as to how to proceed?

> That you're able to compile a few elisp files before bootstrap starts 
> failing...

That's an illusion.  I was doing a parallel build, and the output I 
posted simply showed the first few files that emacs was trying to compile.

Ken

Re: Preview: portable dumper

[Ken Brown]

On 2/17/2018 6:38 PM, Ken Brown wrote:
> On 2/15/2018 9:36 PM, Daniel Colascione wrote:
>> On 02/15/2018 05:56 PM, Ken Brown wrote:
>>> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>>>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
> [...]
>>>>     I just tried to build on 64-bit Cygwin, and the build fails as 
>>>> follows:
>>>>
>>>>     [...]
>>>>     Dumping under the name bootstrap-emacs.pdmp
>>>>     dumping fingerprint:
>>>>     923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>>>>     Dump complete
>>>>     Byte counts: header=80 hot=13187392 discardable=119424 cold=9086640
>>>>     Reloc counts: hot=919268 discardable=5790
>>>>     make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
>>>>     make[2]: Entering directory
>>>>     '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>>>>         ELC      ../../pdumper/lisp/emacs-lisp/macroexp.elc
>>>>         ELC      ../../pdumper/lisp/emacs-lisp/cconv.elc
>>>>         ELC      ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>>>>         ELC      ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>>>>     emacs: could not load dump file "../src/bootstrap-emacs.pdmp": out
>>>>     of memory
>>>>
>>>>     There's probably some obvious explanation, but I don't see it at 
>>>> the
>>>>     moment.
>>>>
>>>>
>>>> I'm not entirely surprised to see Cygwin fall over here. We could 
>>>> just use the Windows memory mapping functions directly, but I'd 
>>>> prefer to stick with the POSIX API if we can make it work.
>>>
>>> I agree.
>>>
>>>> Any idea where in the mmap sequence we fail? 
>>>
>>> I haven't looked at the code yet, so I don't understand the question. 
>>> If you give me some guidance, I'll try to investigate.
>>
>> Thanks. I think the trouble must be in dump_mmap_contiguous. We report 
>> all errors from this function as "out of memory". dump_mmap_contiguous 
>> takes an array of mapping descriptors (think ELF segments or 
>> something) and maps them all into a single contiguous region of 
>> virtual memory. On POSIX systems, we reserve a chunk of address space 
>> with a big PROT_NONE anonymous mapping, then carve it up into the 
>> separate mappings that we really want. Windows doesn't support atomic 
>> mmap replacement, so Cygwin has to emulate it, and I wouldn't be 
>> surprised if something's going wrong in this emulation.
> 
> You're right that the problem is in dump_mmap_contiguous.  I stepped 
> through it in gdb and found the following:
> 
> 1. The call to dump_anonymous_allocate at pdumper.c:4432 succeeds.
> 
> 2. We enter the loop at pdumper.c:4451 with i = 0.  dump_map_file is 
> called, which calls dump_map_file_posix with protection = 
> DUMP_MEMORY_ACCESS_READWRITE.
> 
> 3. This calls mmap at line 4219 with mem_prot = PROT_READ | PROT_WRITE 
> and mem_flags = MAP_PRIVATE | MAP_FIXED.
> 
> 4. This mmap call fails.
> 
> I don't know if this is a bug in Cygwin's mmap or simply a limitation 
> that has to be worked around.  Should I take this to the Cygwin mailing 
> list, or do you have other ideas as to how to proceed?

The following looks relevant:

   https://sourceware.org/ml/cygwin/2013-04/msg00372.html

Ken

Re: Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/html, Size: 5619 bytes --]

Re: Preview: portable dumper

[Ken Brown]

On 2/17/2018 7:02 PM, Daniel Colascione wrote:
> On Feb 17, 2018 3:38 PM, Ken Brown <kbrown@cornell.edu> wrote:
> 
>     On 2/15/2018 9:36 PM, Daniel Colascione wrote:
>      > On 02/15/2018 05:56 PM, Ken Brown wrote:
>      >> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>      >>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
>     [...]
>      >>>     I just tried to build on 64-bit Cygwin, and the build fails as
>      >>> follows:
>      >>>
>      >>>     [...]
>      >>>     Dumping under the name bootstrap-emacs.pdmp
>      >>>     dumping fingerprint:
>      >>>    
>     923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>      >>>     Dump complete
>      >>>     Byte counts: header=80 hot=13187392 discardable=119424
>     cold=9086640
>      >>>     Reloc counts: hot=919268 discardable=5790
>      >>>     make -C ../lisp compile-first
>     EMACS="../src/bootstrap-emacs.exe"
>      >>>     make[2]: Entering directory
>      >>>     '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/macroexp.elc
>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/cconv.elc
>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>      >>>     emacs: could not load dump file
>     "../src/bootstrap-emacs.pdmp": out
>      >>>     of memory
>      >>>
>      >>>     There's probably some obvious explanation, but I don't see
>     it at the
>      >>>     moment.
>      >>>
>      >>>
>      >>> I'm not entirely surprised to see Cygwin fall over here. We could
>      >>> just use the Windows memory mapping functions directly, but I'd
>      >>> prefer to stick with the POSIX API if we can make it work.
>      >>
>      >> I agree.
>      >>
>      >>> Any idea where in the mmap sequence we fail?
>      >>
>      >> I haven't looked at the code yet, so I don't understand the
>     question.
>      >> If you give me some guidance, I'll try to investigate.
>      >
>      > Thanks. I think the trouble must be in dump_mmap_contiguous. We
>     report
>      > all errors from this function as "out of memory".
>     dump_mmap_contiguous
>      > takes an array of mapping descriptors (think ELF segments or
>     something)
>      > and maps them all into a single contiguous region of virtual
>     memory. On
>      > POSIX systems, we reserve a chunk of address space with a big
>     PROT_NONE
>      > anonymous mapping, then carve it up into the separate mappings
>     that we
>      > really want. Windows doesn't support atomic mmap replacement, so
>     Cygwin
>      > has to emulate it, and I wouldn't be surprised if something's going
>      > wrong in this emulation.
> 
>     You're right that the problem is in dump_mmap_contiguous.  I stepped
>     through it in gdb and found the following:
> 
>     1. The call to dump_anonymous_allocate at pdumper.c:4432 succeeds.
> 
>     2. We enter the loop at pdumper.c:4451 with i = 0.  dump_map_file is
>     called, which calls dump_map_file_posix with protection =
>     DUMP_MEMORY_ACCESS_READWRITE.
> 
>     3. This calls mmap at line 4219 with mem_prot = PROT_READ | PROT_WRITE
>     and mem_flags = MAP_PRIVATE | MAP_FIXED.
> 
>     4. This mmap call fails.
> 
>     I don't know if this is a bug in Cygwin's mmap or simply a limitation
>     that has to be worked around.  Should I take this to the Cygwin mailing
>     list, or do you have other ideas as to how to proceed?
> 
> I'd definitely let the Cygwin people know. If the problem is that Cygwin 
> doesn't properly support atomic map replacement, I think we can work 
> around the problem by using the same unmap-before-remap approach we use 
> for the native Windows code, but keep using the POSIX memory functions. 
> I can come up with a patch blind, but you'd be in a better position to 
> iterate quickly.

OK, here's Corinna's answer:

   https://cygwin.com/ml/cygwin/2018-02/msg00202.html

It seems like she's suggesting the same unmap-before-remap approach that 
you mentioned.

Ken

Re: Preview: portable dumper

[Daniel Colascione]

On 02/19/2018 05:30 AM, Ken Brown wrote:
> On 2/17/2018 7:02 PM, Daniel Colascione wrote:
>> On Feb 17, 2018 3:38 PM, Ken Brown <kbrown@cornell.edu> wrote:
>>
>>     On 2/15/2018 9:36 PM, Daniel Colascione wrote:
>>      > On 02/15/2018 05:56 PM, Ken Brown wrote:
>>      >> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>>      >>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
>>     [...]
>>      >>>     I just tried to build on 64-bit Cygwin, and the build 
>> fails as
>>      >>> follows:
>>      >>>
>>      >>>     [...]
>>      >>>     Dumping under the name bootstrap-emacs.pdmp
>>      >>>     dumping fingerprint:
>>      >>>     
>> 923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>>      >>>     Dump complete
>>      >>>     Byte counts: header=80 hot=13187392 discardable=119424
>>     cold=9086640
>>      >>>     Reloc counts: hot=919268 discardable=5790
>>      >>>     make -C ../lisp compile-first
>>     EMACS="../src/bootstrap-emacs.exe"
>>      >>>     make[2]: Entering directory
>>      >>>     '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/macroexp.elc
>>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/cconv.elc
>>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>>      >>>     emacs: could not load dump file
>>     "../src/bootstrap-emacs.pdmp": out
>>      >>>     of memory
>>      >>>
>>      >>>     There's probably some obvious explanation, but I don't see
>>     it at the
>>      >>>     moment.
>>      >>>
>>      >>>
>>      >>> I'm not entirely surprised to see Cygwin fall over here. We 
>> could
>>      >>> just use the Windows memory mapping functions directly, but I'd
>>      >>> prefer to stick with the POSIX API if we can make it work.
>>      >>
>>      >> I agree.
>>      >>
>>      >>> Any idea where in the mmap sequence we fail?
>>      >>
>>      >> I haven't looked at the code yet, so I don't understand the
>>     question.
>>      >> If you give me some guidance, I'll try to investigate.
>>      >
>>      > Thanks. I think the trouble must be in dump_mmap_contiguous. We
>>     report
>>      > all errors from this function as "out of memory".
>>     dump_mmap_contiguous
>>      > takes an array of mapping descriptors (think ELF segments or
>>     something)
>>      > and maps them all into a single contiguous region of virtual
>>     memory. On
>>      > POSIX systems, we reserve a chunk of address space with a big
>>     PROT_NONE
>>      > anonymous mapping, then carve it up into the separate mappings
>>     that we
>>      > really want. Windows doesn't support atomic mmap replacement, so
>>     Cygwin
>>      > has to emulate it, and I wouldn't be surprised if something's 
>> going
>>      > wrong in this emulation.
>>
>>     You're right that the problem is in dump_mmap_contiguous.  I stepped
>>     through it in gdb and found the following:
>>
>>     1. The call to dump_anonymous_allocate at pdumper.c:4432 succeeds.
>>
>>     2. We enter the loop at pdumper.c:4451 with i = 0.  dump_map_file is
>>     called, which calls dump_map_file_posix with protection =
>>     DUMP_MEMORY_ACCESS_READWRITE.
>>
>>     3. This calls mmap at line 4219 with mem_prot = PROT_READ | 
>> PROT_WRITE
>>     and mem_flags = MAP_PRIVATE | MAP_FIXED.
>>
>>     4. This mmap call fails.
>>
>>     I don't know if this is a bug in Cygwin's mmap or simply a limitation
>>     that has to be worked around.  Should I take this to the Cygwin 
>> mailing
>>     list, or do you have other ideas as to how to proceed?
>>
>> I'd definitely let the Cygwin people know. If the problem is that 
>> Cygwin doesn't properly support atomic map replacement, I think we can 
>> work around the problem by using the same unmap-before-remap approach 
>> we use for the native Windows code, but keep using the POSIX memory 
>> functions. I can come up with a patch blind, but you'd be in a better 
>> position to iterate quickly.
> 
> OK, here's Corinna's answer:
> 
>    https://cygwin.com/ml/cygwin/2018-02/msg00202.html
> 
> It seems like she's suggesting the same unmap-before-remap approach that 
> you mentioned.

Thanks. I've pushed a fix, I think. Try it now.

Re: Preview: portable dumper

[Ken Brown]

On 2/19/2018 12:03 PM, Daniel Colascione wrote:
> On 02/19/2018 05:30 AM, Ken Brown wrote:
>> On 2/17/2018 7:02 PM, Daniel Colascione wrote:
>>> On Feb 17, 2018 3:38 PM, Ken Brown <kbrown@cornell.edu> wrote:
>>>
>>>     On 2/15/2018 9:36 PM, Daniel Colascione wrote:
>>>      > On 02/15/2018 05:56 PM, Ken Brown wrote:
>>>      >> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>>>      >>> On Feb 15, 2018 3:31 PM, Ken Brown <kbrown@cornell.edu> wrote:
>>>     [...]
>>>      >>>     I just tried to build on 64-bit Cygwin, and the build 
>>> fails as
>>>      >>> follows:
>>>      >>>
>>>      >>>     [...]
>>>      >>>     Dumping under the name bootstrap-emacs.pdmp
>>>      >>>     dumping fingerprint:
>>>      >>> 
>>> 923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>>>      >>>     Dump complete
>>>      >>>     Byte counts: header=80 hot=13187392 discardable=119424
>>>     cold=9086640
>>>      >>>     Reloc counts: hot=919268 discardable=5790
>>>      >>>     make -C ../lisp compile-first
>>>     EMACS="../src/bootstrap-emacs.exe"
>>>      >>>     make[2]: Entering directory
>>>      >>>     '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>>>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/macroexp.elc
>>>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/cconv.elc
>>>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>>>      >>>         ELC      ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>>>      >>>     emacs: could not load dump file
>>>     "../src/bootstrap-emacs.pdmp": out
>>>      >>>     of memory
>>>      >>>
>>>      >>>     There's probably some obvious explanation, but I don't see
>>>     it at the
>>>      >>>     moment.
>>>      >>>
>>>      >>>
>>>      >>> I'm not entirely surprised to see Cygwin fall over here. We 
>>> could
>>>      >>> just use the Windows memory mapping functions directly, but I'd
>>>      >>> prefer to stick with the POSIX API if we can make it work.
>>>      >>
>>>      >> I agree.
>>>      >>
>>>      >>> Any idea where in the mmap sequence we fail?
>>>      >>
>>>      >> I haven't looked at the code yet, so I don't understand the
>>>     question.
>>>      >> If you give me some guidance, I'll try to investigate.
>>>      >
>>>      > Thanks. I think the trouble must be in dump_mmap_contiguous. We
>>>     report
>>>      > all errors from this function as "out of memory".
>>>     dump_mmap_contiguous
>>>      > takes an array of mapping descriptors (think ELF segments or
>>>     something)
>>>      > and maps them all into a single contiguous region of virtual
>>>     memory. On
>>>      > POSIX systems, we reserve a chunk of address space with a big
>>>     PROT_NONE
>>>      > anonymous mapping, then carve it up into the separate mappings
>>>     that we
>>>      > really want. Windows doesn't support atomic mmap replacement, so
>>>     Cygwin
>>>      > has to emulate it, and I wouldn't be surprised if something's 
>>> going
>>>      > wrong in this emulation.
>>>
>>>     You're right that the problem is in dump_mmap_contiguous.  I stepped
>>>     through it in gdb and found the following:
>>>
>>>     1. The call to dump_anonymous_allocate at pdumper.c:4432 succeeds.
>>>
>>>     2. We enter the loop at pdumper.c:4451 with i = 0.  dump_map_file is
>>>     called, which calls dump_map_file_posix with protection =
>>>     DUMP_MEMORY_ACCESS_READWRITE.
>>>
>>>     3. This calls mmap at line 4219 with mem_prot = PROT_READ | 
>>> PROT_WRITE
>>>     and mem_flags = MAP_PRIVATE | MAP_FIXED.
>>>
>>>     4. This mmap call fails.
>>>
>>>     I don't know if this is a bug in Cygwin's mmap or simply a 
>>> limitation
>>>     that has to be worked around.  Should I take this to the Cygwin 
>>> mailing
>>>     list, or do you have other ideas as to how to proceed?
>>>
>>> I'd definitely let the Cygwin people know. If the problem is that 
>>> Cygwin doesn't properly support atomic map replacement, I think we 
>>> can work around the problem by using the same unmap-before-remap 
>>> approach we use for the native Windows code, but keep using the POSIX 
>>> memory functions. I can come up with a patch blind, but you'd be in a 
>>> better position to iterate quickly.
>>
>> OK, here's Corinna's answer:
>>
>>    https://cygwin.com/ml/cygwin/2018-02/msg00202.html
>>
>> It seems like she's suggesting the same unmap-before-remap approach 
>> that you mentioned.
> 
> Thanks. I've pushed a fix, I think. Try it now.

That fixes it.

Thanks.

Ken

Re: Preview: portable dumper

[Ken Brown]

On 2/19/2018 5:33 PM, Ken Brown wrote:
> On 2/19/2018 12:03 PM, Daniel Colascione wrote:
>> Thanks. I've pushed a fix, I think. Try it now.
> 
> That fixes it.

But then the following commit broke the build:

commit fca8372020248318f1d0007378607a912b19026e
Author: Daniel Colascione <dancol@dancol.org>
Date:   Mon Feb 19 16:35:09 2018 -0800

     eassert -> eassume in pdumper

More precisely, it broke a "standard" build (./configure && make).  The 
build fails as follows:

[...]
Finding pointers to doc strings...done
Dumping under the name bootstrap-emacs.pdmp
dumping fingerprint: 
d3d1723e85f76355cab6880175f65823cbad50b923958d6dcb4b5e78a766a04e
make[1]: *** [Makefile:779: bootstrap-emacs.exe] Segmentation fault 
(core dumped)

If I configure with --enable-checking, or if I build without 
optimization, then the build succeeds again.

This is on 64-bit Cygwin.  I haven't tried any other platform.

Ken

Re: Preview: portable dumper

[Daniel Colascione]

On 02/20/2018 08:32 AM, Ken Brown wrote:
> On 2/19/2018 5:33 PM, Ken Brown wrote:
>> On 2/19/2018 12:03 PM, Daniel Colascione wrote:
>>> Thanks. I've pushed a fix, I think. Try it now.
>>
>> That fixes it.
> 
> But then the following commit broke the build:
> 
> commit fca8372020248318f1d0007378607a912b19026e
> Author: Daniel Colascione <dancol@dancol.org>
> Date:   Mon Feb 19 16:35:09 2018 -0800
> 
>      eassert -> eassume in pdumper
> 
> More precisely, it broke a "standard" build (./configure && make).  The 
> build fails as follows:

Reverted.

Re: Preview: portable dumper

[Andy Moreton]

On Mon 19 Feb 2018, Daniel Colascione wrote:
>
> Thanks. I've pushed a fix, I think. Try it now.

I'm sure Ken will soon report some more rigorous testing, but I have
successfully built a Cygwin w32 build of emacs, which runs gnus well
enough to send this. I haven't tried the X11 build.

    AndyM

Re: Preview: portable dumper

[Clément Pit-Claudel]

On 2018-02-13 11:37, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 12 Feb 2018 12:18:36 -0800
>>
>> I've pushed the portable dumper to the pdumper Savannah branch. It 
>> should support a fully PIC Emacs.
> 
> Thanks.  I'd urge people to try this branch and report any issues they
> see.

Thanks Daniel.  I compiled with the flags suggested in DEBUG and ran for a few minutes.  No issues to report on that test run, which includes heavy face customizations, plenty of subprocesses, and about 100 packages :) Wonderful work!

Some notes:

* After bootstrapping (pbootstrap), load_dump took 77.82ms ± 13.63ms
* After the final dump (pdump), load_dump took 33.70ms ± 10.22ms
* I tried to use dump-emacs-portable, but it crashed.  I've pasted the backtrace at the bottom of this email.

Thanks again!
Clément.

$ gdb ./emacs
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./emacs...done.
SIGINT is used by the debugger.
Are you sure you want to change it? (y or n) [answered Y; input not from terminal]
DISPLAY = :0
TERM = dumb
Breakpoint 1 at 0x59b9a3: file emacs.c, line 357.
Temporary breakpoint 2 at 0x5c92db: file sysdep.c, line 1075.
(gdb) run --batch -l ~/.emacs.d/init.el --eval '(dump-emacs-portable "test.pdump")'
Starting program: /build/emacs/pdumper/src/emacs --batch -l ~/.emacs.d/init.el --eval '(dump-emacs-portable "test.pdump")'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
load_dump completed in 16.418 milliseconds: result=0
[New Thread 0x7fffe49d8700 (LWP 31691)]
dumping fingerprint: 0c14ce6dcf659b419ed01ba19f6847c88519a8d7f04f432885fd65dee2d040de

Thread 1 "emacs" hit Breakpoint 1, terminate_due_to_signal (sig=6, backtrace_limit=40) at emacs.c:357
357	{
(gdb) bt full
#0  terminate_due_to_signal (sig=6, backtrace_limit=40) at emacs.c:357
No locals.
#1  0x00000000005caeed in emacs_abort () at sysdep.c:2437
No locals.
#2  0x00000000005afb2c in unblock_input_to (level=-1) at keyboard.c:7161
No locals.
#3  0x00000000005afb43 in unblock_input () at keyboard.c:7177
No locals.
#4  0x0000000000642c24 in dump_unwind_cleanup (data=0x7fffffffb470) at pdumper.c:3395
        ctx = 0x7fffffffb470
#5  0x00000000006722fb in do_one_unbind (this_binding=0x7fffffffb050, unwinding=true, bindflag=SET_INTERNAL_UNBIND)
    at eval.c:3457
No locals.
#6  0x000000000067272f in unbind_to (count=3, value=XIL(0)) at eval.c:3573
        this_binding = {
          kind = SPECPDL_UNWIND_PTR, 
          unwind = {
            kind = SPECPDL_UNWIND_PTR, 
            func = 0x642be3 <dump_unwind_cleanup>, 
            arg = XIL(0x7fffffffb470)
          }, 
          unwind_ptr = {
            kind = SPECPDL_UNWIND_PTR, 
---Type <return> to continue, or q <return> to quit---
            func = 0x642be3 <dump_unwind_cleanup>, 
            arg = 0x7fffffffb470
          }, 
          unwind_int = {
            kind = SPECPDL_UNWIND_PTR, 
            func = 0x642be3 <dump_unwind_cleanup>, 
            arg = -19344
          }, 
          unwind_void = {
            kind = SPECPDL_UNWIND_PTR, 
            func = 0x642be3 <dump_unwind_cleanup>
          }, 
          let = {
            kind = SPECPDL_UNWIND_PTR, 
            symbol = XIL(0x642be3), 
            old_value = XIL(0x7fffffffb470), 
            where = XIL(0x7fffe53c1505), 
            saved_value = XIL(0)
          }, 
          bt = {
            kind = SPECPDL_UNWIND_PTR, 
            debug_on_exit = false, 
            function = XIL(0x642be3), 
---Type <return> to continue, or q <return> to quit---
            args = 0x7fffffffb470, 
            nargs = 140737039308037
          }
        }
        quitf = XIL(0)
#7  0x000000000066b247 in unwind_to_catch (catch=0xed1690, value=XIL(0x3ac8d23)) at eval.c:1149
        last_time = false
#8  0x000000000066c81b in signal_or_quit (error_symbol=XIL(0x5310), data=XIL(0x3ac8d03), keyboard_quit=false)
    at eval.c:1674
        unwind_data = XIL(0x3ac8d23)
        conditions = XIL(0x7fffe53e3a1b)
        string = XIL(0x3ac8d23)
        real_error_symbol = XIL(0x5310)
        clause = XIL(0xc180)
        h = 0xed1690
#9  0x000000000066c429 in Fsignal (error_symbol=XIL(0x5310), data=XIL(0x3ac8d03)) at eval.c:1569
No locals.
#10 0x000000000059862e in xsignal (error_symbol=XIL(0x5310), data=XIL(0x3ac8d03)) at lisp.h:4109
No locals.
#11 0x000000000066c91c in xsignal1 (error_symbol=XIL(0x5310), arg=XIL(0x25ff544)) at eval.c:1703
No locals.
#12 0x000000000066d1f6 in verror (m=0x79ee58 "unsupported object type in dump: %s", ap=0x7fffffffb240) at eval.c:1888
No locals.
---Type <return> to continue, or q <return> to quit---
#13 0x000000000066d2a7 in error (m=0x79ee58 "unsupported object type in dump: %s") at eval.c:1900
        ap = <error reading variable ap (Attempt to dereference a generic pointer.)>
#14 0x000000000063b107 in error_unsupported_dump_object (ctx=0x7fffffffb470, object=XIL(0x11082b5), 
    msg=0x79fad7 "window configuration") at pdumper.c:635
No locals.
#15 0x0000000000641519 in dump_vectorlike (ctx=0x7fffffffb470, v=0x11082b0) at pdumper.c:2848
        offset = 32767
        lv = XIL(0x11082b5)
#16 0x00000000006416dd in dump_object_1 (ctx=0x7fffffffb470, object=XIL(0x11082b5)) at pdumper.c:2900
        offset = -1
#17 0x0000000000641918 in dump_object (ctx=0x7fffffffb470, object=XIL(0x11082b5)) at pdumper.c:2965
        result = 0
#18 0x000000000064414a in Fdump_emacs_portable (filename=XIL(0x25ff564), track_referrers=XIL(0)) at pdumper.c:3794
        count = 13
        symbol = XIL(0x7fffe4a96068)
        ctx_buf = {
          header = {
            magic = "!UMPEDGNUEMACS\000", 
            fingerprint = "\f\024\316m\317e\233A\236\320\033\241\237hGȅ\031\250\327\360OC(\205\375e\336\342\320", <incomplete sequence \336>, 
            dump_relocs = {
              offset = 0, 
              nr_entries = 0
---Type <return> to continue, or q <return> to quit---
            }, 
            object_starts = {
              offset = 0, 
              nr_entries = 0
            }, 
            emacs_relocs = {
              offset = 0, 
              nr_entries = 0
            }, 
            discardable_start = 0, 
            cold_start = 0
          }, 
          old_purify_flag = XIL(0), 
          old_post_gc_hook = XIL(0), 
          fd = 6, 
          dump_filename = XIL(0x25ff564), 
          offset = 2411504, 
          obj_offset = 0, 
          flags = {
            dump_object_contents = true, 
            dump_object_starts = true, 
            pack_objects = false, 
            assert_already_seen = false, 
---Type <return> to continue, or q <return> to quit---
            defer_hash_tables = true, 
            defer_symbols = false
          }, 
          end_heap = 0, 
          objects_dumped = XIL(0x11141c5), 
          referrers = XIL(0), 
          current_referrer = XIL(0), 
          have_current_referrer = true, 
          dump_queue = {
            zero_weight_objects = {
              head = XIL(0x3a792f3), 
              tail = XIL(0xeb5b83), 
              length = 1565
            }, 
            one_weight_normal_objects = {
              head = XIL(0x3977d73), 
              tail = XIL(0x1dd1a33), 
              length = 388
            }, 
            one_weight_strong_objects = {
              head = XIL(0x3ac9753), 
              tail = XIL(0x17a46c3), 
              length = 42646
---Type <return> to continue, or q <return> to quit---
            }, 
            fancy_weight_objects = {
              head = XIL(0x3acb753), 
              tail = XIL(0x34fa903), 
              length = 164
            }, 
            link_weights = XIL(0x154aee5), 
            sequence_numbers = XIL(0x157ecc5), 
            next_sequence_number = 116616
          }, 
          deferred_hash_tables = XIL(0x3a55c03), 
          deferred_symbols = XIL(0), 
          fixups = XIL(0x3ac8c63), 
          symbol_aux = XIL(0), 
          copied_queue = XIL(0x3aab0e3), 
          cold_queue = XIL(0x3ac9f23), 
          dump_relocs = XIL(0x3ab1603), 
          object_starts = XIL(0x3ac8cf3), 
          emacs_relocs = XIL(0x3aa8fa3), 
          number_hot_relocations = 0, 
          number_discardable_relocations = 0
        }
        ctx = 0x7fffffffb470
---Type <return> to continue, or q <return> to quit---
        header_start = 0
        header_end = 80
        hot_start = 80
        hot_end = 0
        discardable_end = 52608
        number_hot_relocations = 0
        number_discardable_relocations = 52608
        cold_end = 0
#19 0x000000000066e7c9 in eval_sub (form=XIL(0x2783483)) at eval.c:2290
        i = 2
        maxargs = 2
        args_left = XIL(0)
        numargs = make_number(1)
        fun = XIL(0xdb5045)
        val = XIL(0x7fffe53c1500)
        original_fun = XIL(0x7fffe46766b8)
        original_args = XIL(0x2783493)
        funcar = XIL(0x7fffffffb670)
        count = 12
        argvals = {XIL(0x2789394), XIL(0), XIL(0x7fffffffb730), XIL(0x672073), XIL(0), XIL(0xe3c470), XIL(0x7ad0), 
          XIL(0x7ad0)}
#20 0x000000000066dc63 in Feval (form=XIL(0x2783483), lexical=XIL(0)) at eval.c:2107
        count = 11
---Type <return> to continue, or q <return> to quit---
#21 0x0000000000670641 in funcall_subr (subr=0xdb8a00 <Seval>, numargs=1, args=0x7fffffffb948) at eval.c:2899
        internal_argbuf = {XIL(0x2783483), XIL(0), XIL(0x7fffffffb800), XIL(0x59662f), XIL(0xdb8a05), XIL(0xdb8a00), 
          XIL(0x7fffffffb820), XIL(0x596657)}
        internal_args = 0x7fffffffb7d0
#22 0x0000000000670150 in Ffuncall (nargs=2, args=0x7fffffffb940) at eval.c:2822
        fun = XIL(0xdb8a05)
        original_fun = XIL(0x5430)
        funcar = XIL(0x7fffe58c665b)
        numargs = 1
        val = XIL(0)
        count = 10
#23 0x00000000006c3f5c in exec_byte_code (bytestr=XIL(0x7fffe58c6d5c), vector=XIL(0x7fffe58c5b2d), 
    maxdepth=make_number(23), args_template=make_number(257), nargs=1, args=0x7fffffffc4c8) at bytecode.c:632
        op = 1
        type = CATCHER
        targets = {0x6c7447 <exec_byte_code+17448>, 0x6c7476 <exec_byte_code+17495>, 0x6c7478 <exec_byte_code+17497>, 
          0x6c747a <exec_byte_code+17499>, 0x6c747c <exec_byte_code+17501>, 0x6c747c <exec_byte_code+17501>, 
          0x6c74ee <exec_byte_code+17615>, 0x6c7572 <exec_byte_code+17747>, 0x6c35c0 <exec_byte_code+1441>, 
          0x6c35c2 <exec_byte_code+1443>, 0x6c35c4 <exec_byte_code+1445>, 0x6c35c6 <exec_byte_code+1447>, 
          0x6c35c8 <exec_byte_code+1449>, 0x6c35c8 <exec_byte_code+1449>, 0x6c35d1 <exec_byte_code+1458>, 
          0x6c357d <exec_byte_code+1374>, 0x6c3afd <exec_byte_code+2782>, 0x6c3aff <exec_byte_code+2784>, 
          0x6c3b01 <exec_byte_code+2786>, 0x6c3b03 <exec_byte_code+2788>, 0x6c3b05 <exec_byte_code+2790>, 
          0x6c3b05 <exec_byte_code+2790>, 0x6c3b4f <exec_byte_code+2864>, 0x6c3b0e <exec_byte_code+2799>, 
---Type <return> to continue, or q <return> to quit---
          0x6c3e3f <exec_byte_code+3616>, 0x6c3e41 <exec_byte_code+3618>, 0x6c3e43 <exec_byte_code+3620>, 
          0x6c3e45 <exec_byte_code+3622>, 0x6c3e47 <exec_byte_code+3624>, 0x6c3e47 <exec_byte_code+3624>, 
          0x6c3dde <exec_byte_code+3519>, 0x6c3dfe <exec_byte_code+3551>, 0x6c3f1a <exec_byte_code+3835>, 
          0x6c3f1c <exec_byte_code+3837>, 0x6c3f1e <exec_byte_code+3839>, 0x6c3f20 <exec_byte_code+3841>, 
          0x6c3f22 <exec_byte_code+3843>, 0x6c3f22 <exec_byte_code+3843>, 0x6c3eb9 <exec_byte_code+3738>, 
          0x6c3ed9 <exec_byte_code+3770>, 0x6c3ffd <exec_byte_code+4062>, 0x6c3fff <exec_byte_code+4064>, 
          0x6c4001 <exec_byte_code+4066>, 0x6c4003 <exec_byte_code+4068>, 0x6c4005 <exec_byte_code+4070>, 
          0x6c4005 <exec_byte_code+4070>, 0x6c3f9c <exec_byte_code+3965>, 0x6c3fbc <exec_byte_code+3997>, 
          0x6c4981 <exec_byte_code+6498>, 0x6c4854 <exec_byte_code+6197>, 0x6c4848 <exec_byte_code+6185>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c4be0 <exec_byte_code+7105>, 
          0x6c4d17 <exec_byte_code+7416>, 0x6c4d8b <exec_byte_code+7532>, 0x6c4e00 <exec_byte_code+7649>, 
          0x6c4e76 <exec_byte_code+7767>, 0x6c391b <exec_byte_code+2300>, 0x6c39b3 <exec_byte_code+2452>, 
          0x6c4f07 <exec_byte_code+7912>, 0x6c3847 <exec_byte_code+2088>, 0x6c3a28 <exec_byte_code+2569>, 
          0x6c4f86 <exec_byte_code+8039>, 0x6c4ffb <exec_byte_code+8156>, 0x6c504a <exec_byte_code+8235>, 
          0x6c50bf <exec_byte_code+8352>, 0x6c511b <exec_byte_code+8444>, 0x6c520f <exec_byte_code+8688>, 
          0x6c525e <exec_byte_code+8767>, 0x6c52d3 <exec_byte_code+8884>, 0x6c536b <exec_byte_code+9036>, 
          0x6c53ba <exec_byte_code+9115>, 0x6c5409 <exec_byte_code+9194>, 0x6c547e <exec_byte_code+9311>, 
          0x6c54f3 <exec_byte_code+9428>, 0x6c5568 <exec_byte_code+9545>, 0x6c5600 <exec_byte_code+9697>, 
          0x6c565c <exec_byte_code+9789>, 0x6c56b8 <exec_byte_code+9881>, 0x6c57ac <exec_byte_code+10125>, 
          0x6c5836 <exec_byte_code+10263>, 0x6c58c0 <exec_byte_code+10401>, 0x6c5aa2 <exec_byte_code+10883>, 
          0x6c5b1c <exec_byte_code+11005>, 0x6c5b96 <exec_byte_code+11127>, 0x6c5c10 <exec_byte_code+11249>, 
          0x6c5c8a <exec_byte_code+11371>, 0x6c5ce6 <exec_byte_code+11463>, 0x6c5d74 <exec_byte_code+11605>, 
---Type <return> to continue, or q <return> to quit---
          0x6c5dd0 <exec_byte_code+11697>, 0x6c5e2c <exec_byte_code+11789>, 0x6c5e88 <exec_byte_code+11881>, 
          0x6c5fb5 <exec_byte_code+12182>, 0x6c46c7 <exec_byte_code+5800>, 0x6c601a <exec_byte_code+12283>, 
          0x6c6069 <exec_byte_code+12362>, 0x6c6155 <exec_byte_code+12598>, 0x6c61c5 <exec_byte_code+12710>, 
          0x6c622a <exec_byte_code+12811>, 0x6c6279 <exec_byte_code+12890>, 0x6c62c6 <exec_byte_code+12967>, 
          0x6c6313 <exec_byte_code+13044>, 0x6c6368 <exec_byte_code+13129>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c63c7 <exec_byte_code+13224>, 0x6c6414 <exec_byte_code+13301>, 0x6c6461 <exec_byte_code+13378>, 
          0x6c64ae <exec_byte_code+13455>, 0x6c64fb <exec_byte_code+13532>, 0x6c6548 <exec_byte_code+13609>, 
          0x6c46c7 <exec_byte_code+5800>, 0x6c7447 <exec_byte_code+17448>, 0x6c6597 <exec_byte_code+13688>, 
          0x6c65f1 <exec_byte_code+13778>, 0x6c6640 <exec_byte_code+13857>, 0x6c668f <exec_byte_code+13936>, 
          0x6c6704 <exec_byte_code+14053>, 0x6c6779 <exec_byte_code+14170>, 0x6c67c8 <exec_byte_code+14249>, 
          0x6c690b <exec_byte_code+14572>, 0x6c6980 <exec_byte_code+14689>, 0x6c69f5 <exec_byte_code+14806>, 
          0x6c6a6a <exec_byte_code+14923>, 0x6c6ab7 <exec_byte_code+15000>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c45e3 <exec_byte_code+5572>, 0x6c40bf <exec_byte_code+4256>, 0x6c379c <exec_byte_code+1917>, 
          0x6c41a3 <exec_byte_code+4484>, 0x6c4240 <exec_byte_code+4641>, 0x6c42da <exec_byte_code+4795>, 
          0x6c4590 <exec_byte_code+5489>, 0x6c45a8 <exec_byte_code+5513>, 0x6c3d81 <exec_byte_code+3426>, 
          0x6c4682 <exec_byte_code+5731>, 0x6c46ff <exec_byte_code+5856>, 0x6c478c <exec_byte_code+5997>, 
          0x6c47d1 <exec_byte_code+6066>, 0x6c49ce <exec_byte_code+6575>, 0x6c4a4e <exec_byte_code+6703>, 
          0x6c4ae6 <exec_byte_code+6855>, 0x6c4b50 <exec_byte_code+6961>, 0x6c406d <exec_byte_code+4174>, 
          0x6c6b06 <exec_byte_code+15079>, 0x6c6b9e <exec_byte_code+15231>, 0x6c6bed <exec_byte_code+15310>, 
          0x6c6c3c <exec_byte_code+15389>, 0x6c6c8b <exec_byte_code+15468>, 0x6c6cda <exec_byte_code+15547>, 
          0x6c6d4f <exec_byte_code+15664>, 0x6c6dc4 <exec_byte_code+15781>, 0x6c6e39 <exec_byte_code+15898>, 
          0x6c6eae <exec_byte_code+16015>, 0x6c703f <exec_byte_code+16416>, 0x6c70b4 <exec_byte_code+16533>, 
          0x6c7129 <exec_byte_code+16650>, 0x6c7178 <exec_byte_code+16729>, 0x6c71ed <exec_byte_code+16846>, 
---Type <return> to continue, or q <return> to quit---
          0x6c7262 <exec_byte_code+16963>, 0x6c72b1 <exec_byte_code+17042>, 0x6c7300 <exec_byte_code+17121>, 
          0x6c5ee4 <exec_byte_code+11973>, 0x6c5f40 <exec_byte_code+12065>, 0x6c735c <exec_byte_code+17213>, 
          0x6c73d2 <exec_byte_code+17331>, 0x6c7447 <exec_byte_code+17448>, 0x6c4374 <exec_byte_code+4949>, 
          0x6c439a <exec_byte_code+4987>, 0x6c4419 <exec_byte_code+5114>, 0x6c4498 <exec_byte_code+5241>, 
          0x6c4514 <exec_byte_code+5365>, 0x6c5177 <exec_byte_code+8536>, 0x6c5714 <exec_byte_code+9973>, 
          0x6c60bd <exec_byte_code+12446>, 0x6c7621 <exec_byte_code+17922>, 0x6c76a0 <exec_byte_code+18049>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c774c <exec_byte_code+18221>, 
          0x6c77f2 <exec_byte_code+18387>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c79ff <exec_byte_code+18912> <repeats 64 times>}
        const_length = 106
        bytestr_length = 1176
        vectorp = 0x7fffe58c5b30
        quitcounter = 104 'h'
        stack_items = 24
        sa_avail = 15016
        sa_count = 8
        sa_must_free = false
        alloc = 0x7fffffffb8b0
        item_bytes = 192
        stack_base = 0x7fffffffb8b0
        top = 0x7fffffffb940
---Type <return> to continue, or q <return> to quit---
        stack_lim = 0x7fffffffb970
        bytestr_data = 0x7fffffffb970 "\306 \210\b\203\021"
        pc = 0x7fffffffbad8 "\266\005\202d\003\016A睃\345\001\313\350\351\016C\"\003\206\203\001\n\211A\022\242\211\262\r\313\332\036D\322\003\003\003#)\266\203\203\237\001\006\n\327\313O\262\vڲ\001\352\353\006\f!!\262\v\211\203\300\001\314\016E\006\fC\"\026E\006\t\203\341\001\016E\262\n\202\341\001\006\t\203\327\001\006\t\006\v\006\vAB\241\210\006\tA\262\n\202\341\001\006\n\016EB\211\026E\262\n\210\202d\003\016A읃\r\002\353\002\206\366\001\n\211A\022\242!\352\001!\355\001!\203\003\002\211\262\002\356\002\313\332#\266\003\202d\003\016A-\002\353\002\206\036\002\n\211A\022\242!\352\001!\356\001\313ډ$\266\003\202d\003\016", <incomplete sequence \360>...
        count = 8
        result = XIL(0)
#24 0x0000000000670dc0 in funcall_lambda (fun=XIL(0x7fffe58c5afd), nargs=1, arg_vector=0x7fffffffc4c0) at eval.c:3023
        size = 5
        val = XIL(0x7fffe58c5afd)
        syms_left = make_number(257)
        next = XIL(0x1200db6c45)
        lexenv = XIL(0x7fffffffc400)
        count = 8
        i = 77324306848
        optional = false
        rest = false
        previous_optional_or_rest = false
#25 0x0000000000670194 in Ffuncall (nargs=2, args=0x7fffffffc4b8) at eval.c:2824
        fun = XIL(0x7fffe58c5afd)
---Type <return> to continue, or q <return> to quit---
        original_fun = XIL(0x7fffe4a91128)
        funcar = XIL(0x7fffffffc4a0)
        numargs = 1
        val = XIL(0x7fffe4694d90)
        count = 7
#26 0x00000000006c3f5c in exec_byte_code (bytestr=XIL(0x7fffe58ca504), vector=XIL(0x7fffe58c6f9d), 
    maxdepth=make_number(20), args_template=make_number(0), nargs=0, args=0x7fffffffd2c8) at bytecode.c:632
        op = 1
        type = CONDITION_CASE
        targets = {0x6c7447 <exec_byte_code+17448>, 0x6c7476 <exec_byte_code+17495>, 0x6c7478 <exec_byte_code+17497>, 
          0x6c747a <exec_byte_code+17499>, 0x6c747c <exec_byte_code+17501>, 0x6c747c <exec_byte_code+17501>, 
          0x6c74ee <exec_byte_code+17615>, 0x6c7572 <exec_byte_code+17747>, 0x6c35c0 <exec_byte_code+1441>, 
          0x6c35c2 <exec_byte_code+1443>, 0x6c35c4 <exec_byte_code+1445>, 0x6c35c6 <exec_byte_code+1447>, 
          0x6c35c8 <exec_byte_code+1449>, 0x6c35c8 <exec_byte_code+1449>, 0x6c35d1 <exec_byte_code+1458>, 
          0x6c357d <exec_byte_code+1374>, 0x6c3afd <exec_byte_code+2782>, 0x6c3aff <exec_byte_code+2784>, 
          0x6c3b01 <exec_byte_code+2786>, 0x6c3b03 <exec_byte_code+2788>, 0x6c3b05 <exec_byte_code+2790>, 
          0x6c3b05 <exec_byte_code+2790>, 0x6c3b4f <exec_byte_code+2864>, 0x6c3b0e <exec_byte_code+2799>, 
          0x6c3e3f <exec_byte_code+3616>, 0x6c3e41 <exec_byte_code+3618>, 0x6c3e43 <exec_byte_code+3620>, 
          0x6c3e45 <exec_byte_code+3622>, 0x6c3e47 <exec_byte_code+3624>, 0x6c3e47 <exec_byte_code+3624>, 
          0x6c3dde <exec_byte_code+3519>, 0x6c3dfe <exec_byte_code+3551>, 0x6c3f1a <exec_byte_code+3835>, 
          0x6c3f1c <exec_byte_code+3837>, 0x6c3f1e <exec_byte_code+3839>, 0x6c3f20 <exec_byte_code+3841>, 
          0x6c3f22 <exec_byte_code+3843>, 0x6c3f22 <exec_byte_code+3843>, 0x6c3eb9 <exec_byte_code+3738>, 
          0x6c3ed9 <exec_byte_code+3770>, 0x6c3ffd <exec_byte_code+4062>, 0x6c3fff <exec_byte_code+4064>, 
---Type <return> to continue, or q <return> to quit---
          0x6c4001 <exec_byte_code+4066>, 0x6c4003 <exec_byte_code+4068>, 0x6c4005 <exec_byte_code+4070>, 
          0x6c4005 <exec_byte_code+4070>, 0x6c3f9c <exec_byte_code+3965>, 0x6c3fbc <exec_byte_code+3997>, 
          0x6c4981 <exec_byte_code+6498>, 0x6c4854 <exec_byte_code+6197>, 0x6c4848 <exec_byte_code+6185>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c4be0 <exec_byte_code+7105>, 
          0x6c4d17 <exec_byte_code+7416>, 0x6c4d8b <exec_byte_code+7532>, 0x6c4e00 <exec_byte_code+7649>, 
          0x6c4e76 <exec_byte_code+7767>, 0x6c391b <exec_byte_code+2300>, 0x6c39b3 <exec_byte_code+2452>, 
          0x6c4f07 <exec_byte_code+7912>, 0x6c3847 <exec_byte_code+2088>, 0x6c3a28 <exec_byte_code+2569>, 
          0x6c4f86 <exec_byte_code+8039>, 0x6c4ffb <exec_byte_code+8156>, 0x6c504a <exec_byte_code+8235>, 
          0x6c50bf <exec_byte_code+8352>, 0x6c511b <exec_byte_code+8444>, 0x6c520f <exec_byte_code+8688>, 
          0x6c525e <exec_byte_code+8767>, 0x6c52d3 <exec_byte_code+8884>, 0x6c536b <exec_byte_code+9036>, 
          0x6c53ba <exec_byte_code+9115>, 0x6c5409 <exec_byte_code+9194>, 0x6c547e <exec_byte_code+9311>, 
          0x6c54f3 <exec_byte_code+9428>, 0x6c5568 <exec_byte_code+9545>, 0x6c5600 <exec_byte_code+9697>, 
          0x6c565c <exec_byte_code+9789>, 0x6c56b8 <exec_byte_code+9881>, 0x6c57ac <exec_byte_code+10125>, 
          0x6c5836 <exec_byte_code+10263>, 0x6c58c0 <exec_byte_code+10401>, 0x6c5aa2 <exec_byte_code+10883>, 
          0x6c5b1c <exec_byte_code+11005>, 0x6c5b96 <exec_byte_code+11127>, 0x6c5c10 <exec_byte_code+11249>, 
          0x6c5c8a <exec_byte_code+11371>, 0x6c5ce6 <exec_byte_code+11463>, 0x6c5d74 <exec_byte_code+11605>, 
          0x6c5dd0 <exec_byte_code+11697>, 0x6c5e2c <exec_byte_code+11789>, 0x6c5e88 <exec_byte_code+11881>, 
          0x6c5fb5 <exec_byte_code+12182>, 0x6c46c7 <exec_byte_code+5800>, 0x6c601a <exec_byte_code+12283>, 
          0x6c6069 <exec_byte_code+12362>, 0x6c6155 <exec_byte_code+12598>, 0x6c61c5 <exec_byte_code+12710>, 
          0x6c622a <exec_byte_code+12811>, 0x6c6279 <exec_byte_code+12890>, 0x6c62c6 <exec_byte_code+12967>, 
          0x6c6313 <exec_byte_code+13044>, 0x6c6368 <exec_byte_code+13129>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c63c7 <exec_byte_code+13224>, 0x6c6414 <exec_byte_code+13301>, 0x6c6461 <exec_byte_code+13378>, 
---Type <return> to continue, or q <return> to quit---
          0x6c64ae <exec_byte_code+13455>, 0x6c64fb <exec_byte_code+13532>, 0x6c6548 <exec_byte_code+13609>, 
          0x6c46c7 <exec_byte_code+5800>, 0x6c7447 <exec_byte_code+17448>, 0x6c6597 <exec_byte_code+13688>, 
          0x6c65f1 <exec_byte_code+13778>, 0x6c6640 <exec_byte_code+13857>, 0x6c668f <exec_byte_code+13936>, 
          0x6c6704 <exec_byte_code+14053>, 0x6c6779 <exec_byte_code+14170>, 0x6c67c8 <exec_byte_code+14249>, 
          0x6c690b <exec_byte_code+14572>, 0x6c6980 <exec_byte_code+14689>, 0x6c69f5 <exec_byte_code+14806>, 
          0x6c6a6a <exec_byte_code+14923>, 0x6c6ab7 <exec_byte_code+15000>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c45e3 <exec_byte_code+5572>, 0x6c40bf <exec_byte_code+4256>, 0x6c379c <exec_byte_code+1917>, 
          0x6c41a3 <exec_byte_code+4484>, 0x6c4240 <exec_byte_code+4641>, 0x6c42da <exec_byte_code+4795>, 
          0x6c4590 <exec_byte_code+5489>, 0x6c45a8 <exec_byte_code+5513>, 0x6c3d81 <exec_byte_code+3426>, 
          0x6c4682 <exec_byte_code+5731>, 0x6c46ff <exec_byte_code+5856>, 0x6c478c <exec_byte_code+5997>, 
          0x6c47d1 <exec_byte_code+6066>, 0x6c49ce <exec_byte_code+6575>, 0x6c4a4e <exec_byte_code+6703>, 
          0x6c4ae6 <exec_byte_code+6855>, 0x6c4b50 <exec_byte_code+6961>, 0x6c406d <exec_byte_code+4174>, 
          0x6c6b06 <exec_byte_code+15079>, 0x6c6b9e <exec_byte_code+15231>, 0x6c6bed <exec_byte_code+15310>, 
          0x6c6c3c <exec_byte_code+15389>, 0x6c6c8b <exec_byte_code+15468>, 0x6c6cda <exec_byte_code+15547>, 
          0x6c6d4f <exec_byte_code+15664>, 0x6c6dc4 <exec_byte_code+15781>, 0x6c6e39 <exec_byte_code+15898>, 
          0x6c6eae <exec_byte_code+16015>, 0x6c703f <exec_byte_code+16416>, 0x6c70b4 <exec_byte_code+16533>, 
          0x6c7129 <exec_byte_code+16650>, 0x6c7178 <exec_byte_code+16729>, 0x6c71ed <exec_byte_code+16846>, 
          0x6c7262 <exec_byte_code+16963>, 0x6c72b1 <exec_byte_code+17042>, 0x6c7300 <exec_byte_code+17121>, 
          0x6c5ee4 <exec_byte_code+11973>, 0x6c5f40 <exec_byte_code+12065>, 0x6c735c <exec_byte_code+17213>, 
          0x6c73d2 <exec_byte_code+17331>, 0x6c7447 <exec_byte_code+17448>, 0x6c4374 <exec_byte_code+4949>, 
          0x6c439a <exec_byte_code+4987>, 0x6c4419 <exec_byte_code+5114>, 0x6c4498 <exec_byte_code+5241>, 
          0x6c4514 <exec_byte_code+5365>, 0x6c5177 <exec_byte_code+8536>, 0x6c5714 <exec_byte_code+9973>, 
          0x6c60bd <exec_byte_code+12446>, 0x6c7621 <exec_byte_code+17922>, 0x6c76a0 <exec_byte_code+18049>, 
---Type <return> to continue, or q <return> to quit---
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c774c <exec_byte_code+18221>, 
          0x6c77f2 <exec_byte_code+18387>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c79ff <exec_byte_code+18912> <repeats 64 times>}
        const_length = 209
        bytestr_length = 1715
        vectorp = 0x7fffe58c6fa0
        quitcounter = 7 '\a'
        stack_items = 21
        sa_avail = 14501
        sa_count = 7
        sa_must_free = false
        alloc = 0x7fffffffc4b0
        item_bytes = 168
        stack_base = 0x7fffffffc4b0
        top = 0x7fffffffc4b8
        stack_lim = 0x7fffffffc558
        bytestr_data = 0x7fffffffc558 "\306 \020\307\021\n\023\307\024\310\311!\211\307=\204\060"
        pc = 0x7fffffffcb5a "\210\307\016@\211\203L\006\211@\002\204E\006\211;\203E\006\201", <incomplete sequence \310>
        count = 7
        result = XIL(0x7fffffffd0b0)
---Type <return> to continue, or q <return> to quit---
#27 0x0000000000670dc0 in funcall_lambda (fun=XIL(0x7fffe58c6f6d), nargs=0, arg_vector=0x7fffffffd2c8) at eval.c:3023
        size = 5
        val = XIL(0x7fffe58c6f6d)
        syms_left = make_number(0)
        next = XIL(0x1200db4dc5)
        lexenv = XIL(0x7fffffffd200)
        count = 7
        i = 77324306848
        optional = false
        rest = false
        previous_optional_or_rest = false
#28 0x0000000000670194 in Ffuncall (nargs=1, args=0x7fffffffd2c0) at eval.c:2824
        fun = XIL(0x7fffe58c6f6d)
        original_fun = XIL(0x7fffe4a92598)
        funcar = XIL(0xec5215)
        numargs = 0
        val = XIL(0xec5215)
        count = 6
#29 0x00000000006c3f5c in exec_byte_code (bytestr=XIL(0x7fffe58cb144), vector=XIL(0x7fffe58ca6c5), 
    maxdepth=make_number(12), args_template=make_number(0), nargs=0, args=0x7fffffffdb00) at bytecode.c:632
        op = 0
        type = CATCHER
        targets = {0x6c7447 <exec_byte_code+17448>, 0x6c7476 <exec_byte_code+17495>, 0x6c7478 <exec_byte_code+17497>, 
---Type <return> to continue, or q <return> to quit---
          0x6c747a <exec_byte_code+17499>, 0x6c747c <exec_byte_code+17501>, 0x6c747c <exec_byte_code+17501>, 
          0x6c74ee <exec_byte_code+17615>, 0x6c7572 <exec_byte_code+17747>, 0x6c35c0 <exec_byte_code+1441>, 
          0x6c35c2 <exec_byte_code+1443>, 0x6c35c4 <exec_byte_code+1445>, 0x6c35c6 <exec_byte_code+1447>, 
          0x6c35c8 <exec_byte_code+1449>, 0x6c35c8 <exec_byte_code+1449>, 0x6c35d1 <exec_byte_code+1458>, 
          0x6c357d <exec_byte_code+1374>, 0x6c3afd <exec_byte_code+2782>, 0x6c3aff <exec_byte_code+2784>, 
          0x6c3b01 <exec_byte_code+2786>, 0x6c3b03 <exec_byte_code+2788>, 0x6c3b05 <exec_byte_code+2790>, 
          0x6c3b05 <exec_byte_code+2790>, 0x6c3b4f <exec_byte_code+2864>, 0x6c3b0e <exec_byte_code+2799>, 
          0x6c3e3f <exec_byte_code+3616>, 0x6c3e41 <exec_byte_code+3618>, 0x6c3e43 <exec_byte_code+3620>, 
          0x6c3e45 <exec_byte_code+3622>, 0x6c3e47 <exec_byte_code+3624>, 0x6c3e47 <exec_byte_code+3624>, 
          0x6c3dde <exec_byte_code+3519>, 0x6c3dfe <exec_byte_code+3551>, 0x6c3f1a <exec_byte_code+3835>, 
          0x6c3f1c <exec_byte_code+3837>, 0x6c3f1e <exec_byte_code+3839>, 0x6c3f20 <exec_byte_code+3841>, 
          0x6c3f22 <exec_byte_code+3843>, 0x6c3f22 <exec_byte_code+3843>, 0x6c3eb9 <exec_byte_code+3738>, 
          0x6c3ed9 <exec_byte_code+3770>, 0x6c3ffd <exec_byte_code+4062>, 0x6c3fff <exec_byte_code+4064>, 
          0x6c4001 <exec_byte_code+4066>, 0x6c4003 <exec_byte_code+4068>, 0x6c4005 <exec_byte_code+4070>, 
          0x6c4005 <exec_byte_code+4070>, 0x6c3f9c <exec_byte_code+3965>, 0x6c3fbc <exec_byte_code+3997>, 
          0x6c4981 <exec_byte_code+6498>, 0x6c4854 <exec_byte_code+6197>, 0x6c4848 <exec_byte_code+6185>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c4be0 <exec_byte_code+7105>, 
          0x6c4d17 <exec_byte_code+7416>, 0x6c4d8b <exec_byte_code+7532>, 0x6c4e00 <exec_byte_code+7649>, 
          0x6c4e76 <exec_byte_code+7767>, 0x6c391b <exec_byte_code+2300>, 0x6c39b3 <exec_byte_code+2452>, 
          0x6c4f07 <exec_byte_code+7912>, 0x6c3847 <exec_byte_code+2088>, 0x6c3a28 <exec_byte_code+2569>, 
          0x6c4f86 <exec_byte_code+8039>, 0x6c4ffb <exec_byte_code+8156>, 0x6c504a <exec_byte_code+8235>, 
          0x6c50bf <exec_byte_code+8352>, 0x6c511b <exec_byte_code+8444>, 0x6c520f <exec_byte_code+8688>, 
---Type <return> to continue, or q <return> to quit---
          0x6c525e <exec_byte_code+8767>, 0x6c52d3 <exec_byte_code+8884>, 0x6c536b <exec_byte_code+9036>, 
          0x6c53ba <exec_byte_code+9115>, 0x6c5409 <exec_byte_code+9194>, 0x6c547e <exec_byte_code+9311>, 
          0x6c54f3 <exec_byte_code+9428>, 0x6c5568 <exec_byte_code+9545>, 0x6c5600 <exec_byte_code+9697>, 
          0x6c565c <exec_byte_code+9789>, 0x6c56b8 <exec_byte_code+9881>, 0x6c57ac <exec_byte_code+10125>, 
          0x6c5836 <exec_byte_code+10263>, 0x6c58c0 <exec_byte_code+10401>, 0x6c5aa2 <exec_byte_code+10883>, 
          0x6c5b1c <exec_byte_code+11005>, 0x6c5b96 <exec_byte_code+11127>, 0x6c5c10 <exec_byte_code+11249>, 
          0x6c5c8a <exec_byte_code+11371>, 0x6c5ce6 <exec_byte_code+11463>, 0x6c5d74 <exec_byte_code+11605>, 
          0x6c5dd0 <exec_byte_code+11697>, 0x6c5e2c <exec_byte_code+11789>, 0x6c5e88 <exec_byte_code+11881>, 
          0x6c5fb5 <exec_byte_code+12182>, 0x6c46c7 <exec_byte_code+5800>, 0x6c601a <exec_byte_code+12283>, 
          0x6c6069 <exec_byte_code+12362>, 0x6c6155 <exec_byte_code+12598>, 0x6c61c5 <exec_byte_code+12710>, 
          0x6c622a <exec_byte_code+12811>, 0x6c6279 <exec_byte_code+12890>, 0x6c62c6 <exec_byte_code+12967>, 
          0x6c6313 <exec_byte_code+13044>, 0x6c6368 <exec_byte_code+13129>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c63c7 <exec_byte_code+13224>, 0x6c6414 <exec_byte_code+13301>, 0x6c6461 <exec_byte_code+13378>, 
          0x6c64ae <exec_byte_code+13455>, 0x6c64fb <exec_byte_code+13532>, 0x6c6548 <exec_byte_code+13609>, 
          0x6c46c7 <exec_byte_code+5800>, 0x6c7447 <exec_byte_code+17448>, 0x6c6597 <exec_byte_code+13688>, 
          0x6c65f1 <exec_byte_code+13778>, 0x6c6640 <exec_byte_code+13857>, 0x6c668f <exec_byte_code+13936>, 
          0x6c6704 <exec_byte_code+14053>, 0x6c6779 <exec_byte_code+14170>, 0x6c67c8 <exec_byte_code+14249>, 
          0x6c690b <exec_byte_code+14572>, 0x6c6980 <exec_byte_code+14689>, 0x6c69f5 <exec_byte_code+14806>, 
          0x6c6a6a <exec_byte_code+14923>, 0x6c6ab7 <exec_byte_code+15000>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c45e3 <exec_byte_code+5572>, 0x6c40bf <exec_byte_code+4256>, 0x6c379c <exec_byte_code+1917>, 
          0x6c41a3 <exec_byte_code+4484>, 0x6c4240 <exec_byte_code+4641>, 0x6c42da <exec_byte_code+4795>, 
          0x6c4590 <exec_byte_code+5489>, 0x6c45a8 <exec_byte_code+5513>, 0x6c3d81 <exec_byte_code+3426>, 
          0x6c4682 <exec_byte_code+5731>, 0x6c46ff <exec_byte_code+5856>, 0x6c478c <exec_byte_code+5997>, 
---Type <return> to continue, or q <return> to quit---
          0x6c47d1 <exec_byte_code+6066>, 0x6c49ce <exec_byte_code+6575>, 0x6c4a4e <exec_byte_code+6703>, 
          0x6c4ae6 <exec_byte_code+6855>, 0x6c4b50 <exec_byte_code+6961>, 0x6c406d <exec_byte_code+4174>, 
          0x6c6b06 <exec_byte_code+15079>, 0x6c6b9e <exec_byte_code+15231>, 0x6c6bed <exec_byte_code+15310>, 
          0x6c6c3c <exec_byte_code+15389>, 0x6c6c8b <exec_byte_code+15468>, 0x6c6cda <exec_byte_code+15547>, 
          0x6c6d4f <exec_byte_code+15664>, 0x6c6dc4 <exec_byte_code+15781>, 0x6c6e39 <exec_byte_code+15898>, 
          0x6c6eae <exec_byte_code+16015>, 0x6c703f <exec_byte_code+16416>, 0x6c70b4 <exec_byte_code+16533>, 
          0x6c7129 <exec_byte_code+16650>, 0x6c7178 <exec_byte_code+16729>, 0x6c71ed <exec_byte_code+16846>, 
          0x6c7262 <exec_byte_code+16963>, 0x6c72b1 <exec_byte_code+17042>, 0x6c7300 <exec_byte_code+17121>, 
          0x6c5ee4 <exec_byte_code+11973>, 0x6c5f40 <exec_byte_code+12065>, 0x6c735c <exec_byte_code+17213>, 
          0x6c73d2 <exec_byte_code+17331>, 0x6c7447 <exec_byte_code+17448>, 0x6c4374 <exec_byte_code+4949>, 
          0x6c439a <exec_byte_code+4987>, 0x6c4419 <exec_byte_code+5114>, 0x6c4498 <exec_byte_code+5241>, 
          0x6c4514 <exec_byte_code+5365>, 0x6c5177 <exec_byte_code+8536>, 0x6c5714 <exec_byte_code+9973>, 
          0x6c60bd <exec_byte_code+12446>, 0x6c7621 <exec_byte_code+17922>, 0x6c76a0 <exec_byte_code+18049>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c774c <exec_byte_code+18221>, 
          0x6c77f2 <exec_byte_code+18387>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 0x6c7447 <exec_byte_code+17448>, 
          0x6c79ff <exec_byte_code+18912> <repeats 64 times>}
        const_length = 77
        bytestr_length = 443
        vectorp = 0x7fffe58ca6c8
        quitcounter = 41 ')'
        stack_items = 13
---Type <return> to continue, or q <return> to quit---
        sa_avail = 15837
        sa_count = 5
        sa_must_free = false
        alloc = 0x7fffffffd2b0
        item_bytes = 104
        stack_base = 0x7fffffffd2b0
        top = 0x7fffffffd2c0
        stack_lim = 0x7fffffffd318
        bytestr_data = 0x7fffffffd318 "\b\203\b"
        pc = 0x7fffffffd48c "\210)\210\375\376\377\"\210\201H"
        count = 5
        result = XIL(0xcff5ab3b)
#30 0x0000000000670dc0 in funcall_lambda (fun=XIL(0x7fffe58ca695), nargs=0, arg_vector=0x7fffffffdb00) at eval.c:3023
        size = 5
        val = XIL(0x7fffffffdb00)
        syms_left = make_number(0)
        next = XIL(0x7fffffffdab0)
        lexenv = XIL(0)
        count = 5
        i = 6770506
        optional = false
        rest = false
        previous_optional_or_rest = false
---Type <return> to continue, or q <return> to quit---
#31 0x00000000006709e8 in apply_lambda (fun=XIL(0x7fffe58ca695), args=XIL(0), count=4) at eval.c:2959
        args_left = XIL(0)
        i = 0
        numargs = 0
        arg_vector = 0x7fffffffdb00
        tem = XIL(0)
        sa_avail = 16384
        sa_count = 5
        sa_must_free = false
#32 0x000000000066e973 in eval_sub (form=XIL(0x7fffe5a1c16b)) at eval.c:2332
        fun = XIL(0x7fffe58ca695)
        val = XIL(0x7fffe53c1500)
        original_fun = XIL(0x7fffe4a95cc0)
        original_args = XIL(0)
        funcar = XIL(0x7fffffffdc10)
        count = 4
        argvals = {XIL(0x5976c7), XIL(0xeb4920), XIL(0x7fffffffdcd0), XIL(0x672073), XIL(0), XIL(0xe3c470), 
          XIL(0x7ad0), XIL(0x7ad0)}
#33 0x000000000066dc63 in Feval (form=XIL(0x7fffe5a1c16b), lexical=XIL(0)) at eval.c:2107
        count = 3
#34 0x00000000005a0b0e in top_level_2 () at keyboard.c:1122
No locals.
#35 0x000000000066bc8e in internal_condition_case (bfun=0x5a0af1 <top_level_2>, handlers=XIL(0x5310), 
---Type <return> to continue, or q <return> to quit---
    hfun=0x5a04de <cmd_error>) at eval.c:1336
        val = XIL(0xed1570)
        c = 0xed1690
#36 0x00000000005a0b52 in top_level_1 (ignore=XIL(0)) at keyboard.c:1130
No locals.
#37 0x000000000066b12d in internal_catch (tag=XIL(0xc7e0), func=0x5a0b10 <top_level_1>, arg=XIL(0)) at eval.c:1101
        val = XIL(0x7fffffffde00)
        c = 0xed1570
#38 0x00000000005a0a43 in command_loop () at keyboard.c:1091
No locals.
#39 0x000000000059ffd8 in recursive_edit_1 () at keyboard.c:698
        count = 1
        val = XIL(0x7fffffffde60)
#40 0x00000000005a01c6 in Frecursive_edit () at keyboard.c:769
        count = 0
        buffer = XIL(0)
#41 0x000000000059df6b in main (argc=6, argv=0x7fffffffe0b8) at emacs.c:1929
        stack_bottom_variable = 0x0
        do_initial_setlocale = true
        skip_args = 1
        no_loadup = false
        junk = 0x0
        dname_arg = 0x0
---Type <return> to continue, or q <return> to quit---
        ch_to_dir = 0x0
        original_pwd = 0x0
        argv0_base = 0x7fffffffe4d9 "emacs"
        is_temacs = false
        loaded_dump = 0x0
        dump_mode = 0x0
        disable_aslr = false
        rlim = {
          rlim_cur = 10022912, 
          rlim_max = 18446744073709551615
        }
        sockfd = -1

Lisp Backtrace:
"dump-emacs-portable" (0xffffb6d0)
"eval" (0xffffb948)
"command-line-1" (0xffffc4c0)
"command-line" (0xffffd2c8)
"normal-top-level" (0xffffdb00)
(gdb) 

Re: Preview: portable dumper

[Daniel Colascione]

On 02/16/2018 05:01 PM, Clément Pit-Claudel wrote:
> * I tried to use dump-emacs-portable, but it crashed.  I've pasted the backtrace at the bottom of this email.

Should be fixed. Note that the dump will still fail: we can't dump 
window configuration objects. But at least Emacs won't crash.

Re: Preview: portable dumper

[Clément Pit-Claudel]

On 2018-02-19 12:06, Daniel Colascione wrote:
> On 02/16/2018 05:01 PM, Clément Pit-Claudel wrote:
>> * I tried to use dump-emacs-portable, but it crashed.  I've pasted the backtrace at the bottom of this email.
> 
> Should be fixed. Note that the dump will still fail: we can't dump window configuration objects. But at least Emacs won't crash.

Thanks, this seems to have partly done the trick.  I now get "Invalid face: eldoc-highlight-function-argument" when trying to redump, which is indeed better than a crash.  Is that the error you expected?

I tried redumping after loading a smaller init file, but I got assertion failures; here's a repro:

(gdb) run --batch --eval '(dump-emacs-portable "test.pdump")'
Starting program: /build/emacs/pdumper/src/emacs --batch --eval '(dump-emacs-portable "test.pdump")'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
load_dump completed in 14.372 milliseconds: result=0
[New Thread 0x7fffe49d7700 (LWP 9176)]
dumping fingerprint: b0a6626af263696a377e04b184acc7f4ca0786a175a63b064366fda2f150ada3

pdumper.c:2603: Emacs fatal error: assertion failed: XHASH_TABLE (table_rehashed)->count >= 0

Thread 1 "emacs" hit Breakpoint 1, terminate_due_to_signal (sig=6, backtrace_limit=2147483647) at emacs.c:357
357	{
(gdb) bt full
#0  terminate_due_to_signal (sig=6, backtrace_limit=2147483647) at emacs.c:357
No locals.
#1  0x000000000063a821 in die (msg=0x79f9e0 "XHASH_TABLE (table_rehashed)->count >= 0", file=0x79ef40 "pdumper.c", line=2603) at alloc.c:7782
No locals.
#2  0x00000000006400b0 in check_hash_table_rehash (table_orig=XIL(0x7fffe5429495)) at pdumper.c:2603
        table_rehashed = XIL(0xec6465)
        expected_contents = XIL(0x7fffffffb290)
#3  0x00000000006403ce in dump_hash_table (ctx=0x7fffffffb550, hash_in=0x7fffe5429490) at pdumper.c:2651
        is_stable = false
        hash_munged = {
          header = {
            size = 140737488335680, 
            gcaligned = 64 '@'
          }, 
          weak = XIL(0x681095), 
          hash = XIL(0x7fffffffb340), 
          next = XIL(0xbaf), 
          index = XIL(0x7fffdffb6025), 
          count = 15701936, 
          next_free = 140737488335696, 
          pure = 191, 
          rehash_threshold = 0, 
          rehash_size = -3.62270368e+19, 
          key_and_value = XIL(0x8d44), 
          test = {
            name = XIL(0x7fffffffb370), 
            user_hash_function = XIL(0x5970a8), 
            user_cmp_function = make_number(4520), 
            cmpfn = 0xec65e0, 
            hashfn = 0x7fffffffb3d0
          }, 
          next_weak = 0x682490 <hash_lookup+228>
        }
        hash = 0xbac
        out = {
          header = {
            size = 140736951181349, 
            gcaligned = 37 '%'
          }, 
          weak = XIL(0), 
          hash = XIL(0x7fffffffb3b0), 
          next = XIL(0x7fffffff), 
          index = make_number(-1), 
          count = -2147483648, 
          next_free = 140737488335824, 
          pure = 216, 
          rehash_threshold = 2.3509887e-38, 
          rehash_size = 2.53382788e-41, 
          key_and_value = XIL(0xec65e0), 
          test = {
            name = XIL(0xffffffffffffffff), 
            user_hash_function = XIL(0x683ec7), 
            user_cmp_function = make_number(0), 
            cmpfn = 0x46ae3ed52235, 
            hashfn = 0x7fffffffb420
          }, 
          next_weak = 0xe359a0 <lispsym>
        }
#4  0x0000000000641566 in dump_vectorlike (ctx=0x7fffffffb550, v=0x7fffe5429490) at pdumper.c:2919
        offset = 32767
        lv = XIL(0x7fffe5429495)
#5  0x000000000064179b in dump_object_1 (ctx=0x7fffffffb550, object=XIL(0x7fffe5429495)) at pdumper.c:2988
        offset = -1
#6  0x00000000006419d6 in dump_object (ctx=0x7fffffffb550, object=XIL(0x7fffe5429495)) at pdumper.c:3053
        result = 0
#7  0x0000000000644203 in Fdump_emacs_portable (filename=XIL(0xeca154), track_referrers=XIL(0)) at pdumper.c:3884
        count = 13
        symbol = XIL(0x7fffe4a94478)
        ctx_buf = {
          header = {
            magic = "!UMPEDGNUEMACS\000", 
            fingerprint = "\260\246bj\362cij7~\004\261\204\254\307\364\312\a\206\241u\246;\006Cf\375\242\361P\255\243", 
            dump_relocs = {
              offset = 0, 
              nr_entries = 0
            }, 
            object_starts = {
              offset = 0, 
              nr_entries = 0
            }, 
            emacs_relocs = {
              offset = 0, 
              nr_entries = 0
            }, 
            discardable_start = 0, 
            cold_start = 0
          }, 
          old_purify_flag = XIL(0), 
          old_post_gc_hook = XIL(0), 
          fd = 7, 
          dump_filename = XIL(0xeca154), 
          offset = 601000, 
          obj_offset = 0, 
          flags = {
            dump_object_contents = true, 
            dump_object_starts = true, 
            pack_objects = false, 
            assert_already_seen = false, 
            defer_hash_tables = true, 
            defer_symbols = false
          }, 
          end_heap = 0, 
          objects_dumped = XIL(0xec65e5), 
          referrers = XIL(0), 
          current_referrer = XIL(0), 
          have_current_referrer = true, 
          dump_queue = {
            zero_weight_objects = {
              head = XIL(0x145cf53), 
              tail = XIL(0xeb8ec3), 
              length = 1529
            }, 
            one_weight_normal_objects = {
              head = XIL(0x1499113), 
              tail = XIL(0x1499113), 
              length = 1
            }, 
            one_weight_strong_objects = {
              head = XIL(0x1498403), 
              tail = XIL(0xfaa033), 
              length = 1713
            }, 
            fancy_weight_objects = {
              head = XIL(0), 
              tail = XIL(0), 
              length = 0
            }, 
            link_weights = XIL(0xeff225), 
            sequence_numbers = XIL(0xef97b5), 
            next_sequence_number = 18098
          }, 
          deferred_hash_tables = XIL(0x145cf63), 
          deferred_symbols = XIL(0), 
          fixups = XIL(0x1498493), 
          symbol_aux = XIL(0), 
          copied_queue = XIL(0x149b163), 
          cold_queue = XIL(0x1499643), 
          dump_relocs = XIL(0x149b193), 
          object_starts = XIL(0x1498523), 
          emacs_relocs = XIL(0x149bbc3), 
          number_hot_relocations = 0, 
          number_discardable_relocations = 0
        }
        ctx = 0x7fffffffb550
        header_start = 0
        header_end = 80
        hot_start = 80
        hot_end = 0
        discardable_end = 52608
        number_hot_relocations = 0
        number_discardable_relocations = 52608
        cold_end = 0
#8  0x000000000066e7e1 in eval_sub (form=XIL(0xefb043)) at eval.c:2293
        i = 2
        maxargs = 2
        args_left = XIL(0)
        numargs = make_number(1)
        fun = XIL(0xdb6045)
        val = XIL(0x7fffe53c0500)
        original_fun = XIL(0x7fffe46746b8)
        original_args = XIL(0xefb053)
        funcar = XIL(0x7fffffffb750)
        count = 12
        argvals = {XIL(0xf11c44), XIL(0), XIL(0x7fffffffb810), XIL(0x67208b), XIL(0), XIL(0xe3d470), XIL(0x7ad0), XIL(0x7ad0)}
#9  0x000000000066dc7b in Feval (form=XIL(0xefb043), lexical=XIL(0)) at eval.c:2110
        count = 11
#10 0x0000000000670659 in funcall_subr (subr=0xdb9a00 <Seval>, numargs=1, args=0x7fffffffba28) at eval.c:2902
        internal_argbuf = {XIL(0xefb043), XIL(0), XIL(0x7fffffffb8e0), XIL(0x5966ed), XIL(0xdb9a05), XIL(0xdb9a00), XIL(0x7fffffffb900), XIL(0x596715)}
        internal_args = 0x7fffffffb8b0
#11 0x0000000000670168 in Ffuncall (nargs=2, args=0x7fffffffba20) at eval.c:2825
        fun = XIL(0xdb9a05)
        original_fun = XIL(0x5430)
        funcar = XIL(0x7fffe58c57db)
        numargs = 1
        val = XIL(0)
        count = 10
#12 0x00000000006c3f74 in exec_byte_code (bytestr=XIL(0x7fffe58c5edc), vector=XIL(0x7fffe58c4cad), maxdepth=make_number(23), args_template=make_number(257), nargs=1, args=0x7fffffffc5a8) at bytecode.c:632
        op = 1
        type = CONDITION_CASE
        targets = {0x6c745f <exec_byte_code+17448>, 0x6c748e <exec_byte_code+17495>, 0x6c7490 <exec_byte_code+17497>, 0x6c7492 <exec_byte_code+17499>, 0x6c7494 <exec_byte_code+17501>, 0x6c7494 <exec_byte_code+17501>, 
          0x6c7506 <exec_byte_code+17615>, 0x6c758a <exec_byte_code+17747>, 0x6c35d8 <exec_byte_code+1441>, 0x6c35da <exec_byte_code+1443>, 0x6c35dc <exec_byte_code+1445>, 0x6c35de <exec_byte_code+1447>, 0x6c35e0 <exec_byte_code+1449>, 
          0x6c35e0 <exec_byte_code+1449>, 0x6c35e9 <exec_byte_code+1458>, 0x6c3595 <exec_byte_code+1374>, 0x6c3b15 <exec_byte_code+2782>, 0x6c3b17 <exec_byte_code+2784>, 0x6c3b19 <exec_byte_code+2786>, 0x6c3b1b <exec_byte_code+2788>, 
          0x6c3b1d <exec_byte_code+2790>, 0x6c3b1d <exec_byte_code+2790>, 0x6c3b67 <exec_byte_code+2864>, 0x6c3b26 <exec_byte_code+2799>, 0x6c3e57 <exec_byte_code+3616>, 0x6c3e59 <exec_byte_code+3618>, 0x6c3e5b <exec_byte_code+3620>, 
          0x6c3e5d <exec_byte_code+3622>, 0x6c3e5f <exec_byte_code+3624>, 0x6c3e5f <exec_byte_code+3624>, 0x6c3df6 <exec_byte_code+3519>, 0x6c3e16 <exec_byte_code+3551>, 0x6c3f32 <exec_byte_code+3835>, 0x6c3f34 <exec_byte_code+3837>, 
          0x6c3f36 <exec_byte_code+3839>, 0x6c3f38 <exec_byte_code+3841>, 0x6c3f3a <exec_byte_code+3843>, 0x6c3f3a <exec_byte_code+3843>, 0x6c3ed1 <exec_byte_code+3738>, 0x6c3ef1 <exec_byte_code+3770>, 0x6c4015 <exec_byte_code+4062>, 
          0x6c4017 <exec_byte_code+4064>, 0x6c4019 <exec_byte_code+4066>, 0x6c401b <exec_byte_code+4068>, 0x6c401d <exec_byte_code+4070>, 0x6c401d <exec_byte_code+4070>, 0x6c3fb4 <exec_byte_code+3965>, 0x6c3fd4 <exec_byte_code+3997>, 
          0x6c4999 <exec_byte_code+6498>, 0x6c486c <exec_byte_code+6197>, 0x6c4860 <exec_byte_code+6185>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 
          0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c4bf8 <exec_byte_code+7105>, 0x6c4d2f <exec_byte_code+7416>, 0x6c4da3 <exec_byte_code+7532>, 0x6c4e18 <exec_byte_code+7649>, 0x6c4e8e <exec_byte_code+7767>, 
          0x6c3933 <exec_byte_code+2300>, 0x6c39cb <exec_byte_code+2452>, 0x6c4f1f <exec_byte_code+7912>, 0x6c385f <exec_byte_code+2088>, 0x6c3a40 <exec_byte_code+2569>, 0x6c4f9e <exec_byte_code+8039>, 0x6c5013 <exec_byte_code+8156>, 
          0x6c5062 <exec_byte_code+8235>, 0x6c50d7 <exec_byte_code+8352>, 0x6c5133 <exec_byte_code+8444>, 0x6c5227 <exec_byte_code+8688>, 0x6c5276 <exec_byte_code+8767>, 0x6c52eb <exec_byte_code+8884>, 0x6c5383 <exec_byte_code+9036>, 
          0x6c53d2 <exec_byte_code+9115>, 0x6c5421 <exec_byte_code+9194>, 0x6c5496 <exec_byte_code+9311>, 0x6c550b <exec_byte_code+9428>, 0x6c5580 <exec_byte_code+9545>, 0x6c5618 <exec_byte_code+9697>, 0x6c5674 <exec_byte_code+9789>, 
          0x6c56d0 <exec_byte_code+9881>, 0x6c57c4 <exec_byte_code+10125>, 0x6c584e <exec_byte_code+10263>, 0x6c58d8 <exec_byte_code+10401>, 0x6c5aba <exec_byte_code+10883>, 0x6c5b34 <exec_byte_code+11005>, 
          0x6c5bae <exec_byte_code+11127>, 0x6c5c28 <exec_byte_code+11249>, 0x6c5ca2 <exec_byte_code+11371>, 0x6c5cfe <exec_byte_code+11463>, 0x6c5d8c <exec_byte_code+11605>, 0x6c5de8 <exec_byte_code+11697>, 
          0x6c5e44 <exec_byte_code+11789>, 0x6c5ea0 <exec_byte_code+11881>, 0x6c5fcd <exec_byte_code+12182>, 0x6c46df <exec_byte_code+5800>, 0x6c6032 <exec_byte_code+12283>, 0x6c6081 <exec_byte_code+12362>, 
          0x6c616d <exec_byte_code+12598>, 0x6c61dd <exec_byte_code+12710>, 0x6c6242 <exec_byte_code+12811>, 0x6c6291 <exec_byte_code+12890>, 0x6c62de <exec_byte_code+12967>, 0x6c632b <exec_byte_code+13044>, 
          0x6c6380 <exec_byte_code+13129>, 0x6c745f <exec_byte_code+17448>, 0x6c63df <exec_byte_code+13224>, 0x6c642c <exec_byte_code+13301>, 0x6c6479 <exec_byte_code+13378>, 0x6c64c6 <exec_byte_code+13455>, 
          0x6c6513 <exec_byte_code+13532>, 0x6c6560 <exec_byte_code+13609>, 0x6c46df <exec_byte_code+5800>, 0x6c745f <exec_byte_code+17448>, 0x6c65af <exec_byte_code+13688>, 0x6c6609 <exec_byte_code+13778>, 
          0x6c6658 <exec_byte_code+13857>, 0x6c66a7 <exec_byte_code+13936>, 0x6c671c <exec_byte_code+14053>, 0x6c6791 <exec_byte_code+14170>, 0x6c67e0 <exec_byte_code+14249>, 0x6c6923 <exec_byte_code+14572>, 
          0x6c6998 <exec_byte_code+14689>, 0x6c6a0d <exec_byte_code+14806>, 0x6c6a82 <exec_byte_code+14923>, 0x6c6acf <exec_byte_code+15000>, 0x6c745f <exec_byte_code+17448>, 0x6c45fb <exec_byte_code+5572>, 
          0x6c40d7 <exec_byte_code+4256>, 0x6c37b4 <exec_byte_code+1917>, 0x6c41bb <exec_byte_code+4484>, 0x6c4258 <exec_byte_code+4641>, 0x6c42f2 <exec_byte_code+4795>, 0x6c45a8 <exec_byte_code+5489>, 0x6c45c0 <exec_byte_code+5513>, 
          0x6c3d99 <exec_byte_code+3426>, 0x6c469a <exec_byte_code+5731>, 0x6c4717 <exec_byte_code+5856>, 0x6c47a4 <exec_byte_code+5997>, 0x6c47e9 <exec_byte_code+6066>, 0x6c49e6 <exec_byte_code+6575>, 0x6c4a66 <exec_byte_code+6703>, 
          0x6c4afe <exec_byte_code+6855>, 0x6c4b68 <exec_byte_code+6961>, 0x6c4085 <exec_byte_code+4174>, 0x6c6b1e <exec_byte_code+15079>, 0x6c6bb6 <exec_byte_code+15231>, 0x6c6c05 <exec_byte_code+15310>, 
          0x6c6c54 <exec_byte_code+15389>, 0x6c6ca3 <exec_byte_code+15468>, 0x6c6cf2 <exec_byte_code+15547>, 0x6c6d67 <exec_byte_code+15664>, 0x6c6ddc <exec_byte_code+15781>, 0x6c6e51 <exec_byte_code+15898>, 
          0x6c6ec6 <exec_byte_code+16015>, 0x6c7057 <exec_byte_code+16416>, 0x6c70cc <exec_byte_code+16533>, 0x6c7141 <exec_byte_code+16650>, 0x6c7190 <exec_byte_code+16729>, 0x6c7205 <exec_byte_code+16846>, 
          0x6c727a <exec_byte_code+16963>, 0x6c72c9 <exec_byte_code+17042>, 0x6c7318 <exec_byte_code+17121>, 0x6c5efc <exec_byte_code+11973>, 0x6c5f58 <exec_byte_code+12065>, 0x6c7374 <exec_byte_code+17213>, 
          0x6c73ea <exec_byte_code+17331>, 0x6c745f <exec_byte_code+17448>, 0x6c438c <exec_byte_code+4949>, 0x6c43b2 <exec_byte_code+4987>, 0x6c4431 <exec_byte_code+5114>, 0x6c44b0 <exec_byte_code+5241>, 0x6c452c <exec_byte_code+5365>, 
          0x6c518f <exec_byte_code+8536>, 0x6c572c <exec_byte_code+9973>, 0x6c60d5 <exec_byte_code+12446>, 0x6c7639 <exec_byte_code+17922>, 0x6c76b8 <exec_byte_code+18049>, 0x6c745f <exec_byte_code+17448>, 
          0x6c745f <exec_byte_code+17448>, 0x6c7764 <exec_byte_code+18221>, 0x6c780a <exec_byte_code+18387>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 
          0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c7a17 <exec_byte_code+18912> <repeats 64 times>}
        const_length = 106
        bytestr_length = 1176
        vectorp = 0x7fffe58c4cb0
        quitcounter = 103 'g'
        stack_items = 24
        sa_avail = 15016
        sa_count = 8
        sa_must_free = false
        alloc = 0x7fffffffb990
        item_bytes = 192
        stack_base = 0x7fffffffb990
        top = 0x7fffffffba20
        stack_lim = 0x7fffffffba50
        bytestr_data = 0x7fffffffba50 "\306 \210\b\203\021"
        pc = 0x7fffffffbbb8 "\266\005\202d\003\016A睃\345\001\313\350\351\016C\"\003\206\203\001\n\211A\022\242\211\262\r\313\332\036D\322\003\003\003#)\266\203\203\237\001\006\n\327\313O\262\vڲ\001\352\353\006\f!!\262\v\211\203\300\001\314\016E\006\fC\"\026E\006\t\203\341\001\016E\262\n\202\341\001\006\t\203\327\001\006\t\006\v\006\vAB\241\210\006\tA\262\n\202\341\001\006\n\016EB\211\026E\262\n\210\202d\003\016A읃\r\002\353\002\206\366\001\n\211A\022\242!\352\001!\355\001!\203\003\002\211\262\002\356\002\313\332#\266\003\202d\003\016A-\002\353\002\206\036\002\n\211A\022\242!\352\001!\356\001\313ډ$\266\003\202d\003\016", <incomplete sequence \360>...
        count = 8
        result = XIL(0xffff800000003c71)
#13 0x0000000000670dd8 in funcall_lambda (fun=XIL(0x7fffe58c4c7d), nargs=1, arg_vector=0x7fffffffc5a0) at eval.c:3026
        size = 5
        val = XIL(0x7fffe58c4c7d)
        syms_left = make_number(257)
        next = XIL(0x1200efac90)
        lexenv = XIL(0x7fffffffc4e0)
        count = 8
        i = 77324310944
        optional = false
        rest = false
        previous_optional_or_rest = false
#14 0x00000000006701ac in Ffuncall (nargs=2, args=0x7fffffffc598) at eval.c:2827
        fun = XIL(0x7fffe58c4c7d)
        original_fun = XIL(0x7fffe4a8f2a8)
        funcar = XIL(0x7fffffffc580)
        numargs = 1
        val = XIL(0x7fffe4692d90)
        count = 7
#15 0x00000000006c3f74 in exec_byte_code (bytestr=XIL(0x7fffe58c9904), vector=XIL(0x7fffe58c611d), maxdepth=make_number(12), args_template=make_number(0), nargs=0, args=0x7fffffffd2f8) at bytecode.c:632
        op = 1
        type = CONDITION_CASE
        targets = {0x6c745f <exec_byte_code+17448>, 0x6c748e <exec_byte_code+17495>, 0x6c7490 <exec_byte_code+17497>, 0x6c7492 <exec_byte_code+17499>, 0x6c7494 <exec_byte_code+17501>, 0x6c7494 <exec_byte_code+17501>, 
          0x6c7506 <exec_byte_code+17615>, 0x6c758a <exec_byte_code+17747>, 0x6c35d8 <exec_byte_code+1441>, 0x6c35da <exec_byte_code+1443>, 0x6c35dc <exec_byte_code+1445>, 0x6c35de <exec_byte_code+1447>, 0x6c35e0 <exec_byte_code+1449>, 
          0x6c35e0 <exec_byte_code+1449>, 0x6c35e9 <exec_byte_code+1458>, 0x6c3595 <exec_byte_code+1374>, 0x6c3b15 <exec_byte_code+2782>, 0x6c3b17 <exec_byte_code+2784>, 0x6c3b19 <exec_byte_code+2786>, 0x6c3b1b <exec_byte_code+2788>, 
          0x6c3b1d <exec_byte_code+2790>, 0x6c3b1d <exec_byte_code+2790>, 0x6c3b67 <exec_byte_code+2864>, 0x6c3b26 <exec_byte_code+2799>, 0x6c3e57 <exec_byte_code+3616>, 0x6c3e59 <exec_byte_code+3618>, 0x6c3e5b <exec_byte_code+3620>, 
          0x6c3e5d <exec_byte_code+3622>, 0x6c3e5f <exec_byte_code+3624>, 0x6c3e5f <exec_byte_code+3624>, 0x6c3df6 <exec_byte_code+3519>, 0x6c3e16 <exec_byte_code+3551>, 0x6c3f32 <exec_byte_code+3835>, 0x6c3f34 <exec_byte_code+3837>, 
          0x6c3f36 <exec_byte_code+3839>, 0x6c3f38 <exec_byte_code+3841>, 0x6c3f3a <exec_byte_code+3843>, 0x6c3f3a <exec_byte_code+3843>, 0x6c3ed1 <exec_byte_code+3738>, 0x6c3ef1 <exec_byte_code+3770>, 0x6c4015 <exec_byte_code+4062>, 
          0x6c4017 <exec_byte_code+4064>, 0x6c4019 <exec_byte_code+4066>, 0x6c401b <exec_byte_code+4068>, 0x6c401d <exec_byte_code+4070>, 0x6c401d <exec_byte_code+4070>, 0x6c3fb4 <exec_byte_code+3965>, 0x6c3fd4 <exec_byte_code+3997>, 
          0x6c4999 <exec_byte_code+6498>, 0x6c486c <exec_byte_code+6197>, 0x6c4860 <exec_byte_code+6185>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 
          0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c4bf8 <exec_byte_code+7105>, 0x6c4d2f <exec_byte_code+7416>, 0x6c4da3 <exec_byte_code+7532>, 0x6c4e18 <exec_byte_code+7649>, 0x6c4e8e <exec_byte_code+7767>, 
          0x6c3933 <exec_byte_code+2300>, 0x6c39cb <exec_byte_code+2452>, 0x6c4f1f <exec_byte_code+7912>, 0x6c385f <exec_byte_code+2088>, 0x6c3a40 <exec_byte_code+2569>, 0x6c4f9e <exec_byte_code+8039>, 0x6c5013 <exec_byte_code+8156>, 
          0x6c5062 <exec_byte_code+8235>, 0x6c50d7 <exec_byte_code+8352>, 0x6c5133 <exec_byte_code+8444>, 0x6c5227 <exec_byte_code+8688>, 0x6c5276 <exec_byte_code+8767>, 0x6c52eb <exec_byte_code+8884>, 0x6c5383 <exec_byte_code+9036>, 
          0x6c53d2 <exec_byte_code+9115>, 0x6c5421 <exec_byte_code+9194>, 0x6c5496 <exec_byte_code+9311>, 0x6c550b <exec_byte_code+9428>, 0x6c5580 <exec_byte_code+9545>, 0x6c5618 <exec_byte_code+9697>, 0x6c5674 <exec_byte_code+9789>, 
          0x6c56d0 <exec_byte_code+9881>, 0x6c57c4 <exec_byte_code+10125>, 0x6c584e <exec_byte_code+10263>, 0x6c58d8 <exec_byte_code+10401>, 0x6c5aba <exec_byte_code+10883>, 0x6c5b34 <exec_byte_code+11005>, 
          0x6c5bae <exec_byte_code+11127>, 0x6c5c28 <exec_byte_code+11249>, 0x6c5ca2 <exec_byte_code+11371>, 0x6c5cfe <exec_byte_code+11463>, 0x6c5d8c <exec_byte_code+11605>, 0x6c5de8 <exec_byte_code+11697>, 
          0x6c5e44 <exec_byte_code+11789>, 0x6c5ea0 <exec_byte_code+11881>, 0x6c5fcd <exec_byte_code+12182>, 0x6c46df <exec_byte_code+5800>, 0x6c6032 <exec_byte_code+12283>, 0x6c6081 <exec_byte_code+12362>, 
          0x6c616d <exec_byte_code+12598>, 0x6c61dd <exec_byte_code+12710>, 0x6c6242 <exec_byte_code+12811>, 0x6c6291 <exec_byte_code+12890>, 0x6c62de <exec_byte_code+12967>, 0x6c632b <exec_byte_code+13044>, 
          0x6c6380 <exec_byte_code+13129>, 0x6c745f <exec_byte_code+17448>, 0x6c63df <exec_byte_code+13224>, 0x6c642c <exec_byte_code+13301>, 0x6c6479 <exec_byte_code+13378>, 0x6c64c6 <exec_byte_code+13455>, 
          0x6c6513 <exec_byte_code+13532>, 0x6c6560 <exec_byte_code+13609>, 0x6c46df <exec_byte_code+5800>, 0x6c745f <exec_byte_code+17448>, 0x6c65af <exec_byte_code+13688>, 0x6c6609 <exec_byte_code+13778>, 
          0x6c6658 <exec_byte_code+13857>, 0x6c66a7 <exec_byte_code+13936>, 0x6c671c <exec_byte_code+14053>, 0x6c6791 <exec_byte_code+14170>, 0x6c67e0 <exec_byte_code+14249>, 0x6c6923 <exec_byte_code+14572>, 
          0x6c6998 <exec_byte_code+14689>, 0x6c6a0d <exec_byte_code+14806>, 0x6c6a82 <exec_byte_code+14923>, 0x6c6acf <exec_byte_code+15000>, 0x6c745f <exec_byte_code+17448>, 0x6c45fb <exec_byte_code+5572>, 
          0x6c40d7 <exec_byte_code+4256>, 0x6c37b4 <exec_byte_code+1917>, 0x6c41bb <exec_byte_code+4484>, 0x6c4258 <exec_byte_code+4641>, 0x6c42f2 <exec_byte_code+4795>, 0x6c45a8 <exec_byte_code+5489>, 0x6c45c0 <exec_byte_code+5513>, 
          0x6c3d99 <exec_byte_code+3426>, 0x6c469a <exec_byte_code+5731>, 0x6c4717 <exec_byte_code+5856>, 0x6c47a4 <exec_byte_code+5997>, 0x6c47e9 <exec_byte_code+6066>, 0x6c49e6 <exec_byte_code+6575>, 0x6c4a66 <exec_byte_code+6703>, 
          0x6c4afe <exec_byte_code+6855>, 0x6c4b68 <exec_byte_code+6961>, 0x6c4085 <exec_byte_code+4174>, 0x6c6b1e <exec_byte_code+15079>, 0x6c6bb6 <exec_byte_code+15231>, 0x6c6c05 <exec_byte_code+15310>, 
          0x6c6c54 <exec_byte_code+15389>, 0x6c6ca3 <exec_byte_code+15468>, 0x6c6cf2 <exec_byte_code+15547>, 0x6c6d67 <exec_byte_code+15664>, 0x6c6ddc <exec_byte_code+15781>, 0x6c6e51 <exec_byte_code+15898>, 
          0x6c6ec6 <exec_byte_code+16015>, 0x6c7057 <exec_byte_code+16416>, 0x6c70cc <exec_byte_code+16533>, 0x6c7141 <exec_byte_code+16650>, 0x6c7190 <exec_byte_code+16729>, 0x6c7205 <exec_byte_code+16846>, 
          0x6c727a <exec_byte_code+16963>, 0x6c72c9 <exec_byte_code+17042>, 0x6c7318 <exec_byte_code+17121>, 0x6c5efc <exec_byte_code+11973>, 0x6c5f58 <exec_byte_code+12065>, 0x6c7374 <exec_byte_code+17213>, 
          0x6c73ea <exec_byte_code+17331>, 0x6c745f <exec_byte_code+17448>, 0x6c438c <exec_byte_code+4949>, 0x6c43b2 <exec_byte_code+4987>, 0x6c4431 <exec_byte_code+5114>, 0x6c44b0 <exec_byte_code+5241>, 0x6c452c <exec_byte_code+5365>, 
          0x6c518f <exec_byte_code+8536>, 0x6c572c <exec_byte_code+9973>, 0x6c60d5 <exec_byte_code+12446>, 0x6c7639 <exec_byte_code+17922>, 0x6c76b8 <exec_byte_code+18049>, 0x6c745f <exec_byte_code+17448>, 
          0x6c745f <exec_byte_code+17448>, 0x6c7764 <exec_byte_code+18221>, 0x6c780a <exec_byte_code+18387>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 
          0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c7a17 <exec_byte_code+18912> <repeats 64 times>}
        const_length = 207
        bytestr_length = 1604
        vectorp = 0x7fffe58c6120
        quitcounter = 5 '\005'
        stack_items = 13
        sa_avail = 14676
        sa_count = 7
        sa_must_free = false
        alloc = 0x7fffffffc590
        item_bytes = 104
        stack_base = 0x7fffffffc590
        top = 0x7fffffffc598
        stack_lim = 0x7fffffffc5f8
        bytestr_data = 0x7fffffffc5f8 "\306 \020\307\021\n\023\307\024\310\311!\211\307=\204\060"
        pc = 0x7fffffffcb8b "\210\307\016@\211\203\335\005\211@\002\204\326\005\211;\203\326\005\201", <incomplete sequence \304>
        count = 7
        result = XIL(0x7fffffffd0e0)
#16 0x0000000000670dd8 in funcall_lambda (fun=XIL(0x7fffe58c60ed), nargs=0, arg_vector=0x7fffffffd2f8) at eval.c:3026
        size = 5
        val = XIL(0x7fffe58c60ed)
        syms_left = make_number(0)
        next = XIL(0x1200db5dc5)
        lexenv = XIL(0x7fffffffd230)
        count = 7
        i = 77324310944
        optional = false
        rest = false
        previous_optional_or_rest = false
#17 0x00000000006701ac in Ffuncall (nargs=1, args=0x7fffffffd2f0) at eval.c:2827
        fun = XIL(0x7fffe58c60ed)
        original_fun = XIL(0x7fffe4a90718)
        funcar = XIL(0xec6215)
        numargs = 0
        val = XIL(0xec6215)
        count = 6
#18 0x00000000006c3f74 in exec_byte_code (bytestr=XIL(0x7fffe58ca554), vector=XIL(0x7fffe58c9ad5), maxdepth=make_number(12), args_template=make_number(0), nargs=0, args=0x7fffffffdb30) at bytecode.c:632
        op = 0
        type = CATCHER
        targets = {0x6c745f <exec_byte_code+17448>, 0x6c748e <exec_byte_code+17495>, 0x6c7490 <exec_byte_code+17497>, 0x6c7492 <exec_byte_code+17499>, 0x6c7494 <exec_byte_code+17501>, 0x6c7494 <exec_byte_code+17501>, 
          0x6c7506 <exec_byte_code+17615>, 0x6c758a <exec_byte_code+17747>, 0x6c35d8 <exec_byte_code+1441>, 0x6c35da <exec_byte_code+1443>, 0x6c35dc <exec_byte_code+1445>, 0x6c35de <exec_byte_code+1447>, 0x6c35e0 <exec_byte_code+1449>, 
          0x6c35e0 <exec_byte_code+1449>, 0x6c35e9 <exec_byte_code+1458>, 0x6c3595 <exec_byte_code+1374>, 0x6c3b15 <exec_byte_code+2782>, 0x6c3b17 <exec_byte_code+2784>, 0x6c3b19 <exec_byte_code+2786>, 0x6c3b1b <exec_byte_code+2788>, 
          0x6c3b1d <exec_byte_code+2790>, 0x6c3b1d <exec_byte_code+2790>, 0x6c3b67 <exec_byte_code+2864>, 0x6c3b26 <exec_byte_code+2799>, 0x6c3e57 <exec_byte_code+3616>, 0x6c3e59 <exec_byte_code+3618>, 0x6c3e5b <exec_byte_code+3620>, 
          0x6c3e5d <exec_byte_code+3622>, 0x6c3e5f <exec_byte_code+3624>, 0x6c3e5f <exec_byte_code+3624>, 0x6c3df6 <exec_byte_code+3519>, 0x6c3e16 <exec_byte_code+3551>, 0x6c3f32 <exec_byte_code+3835>, 0x6c3f34 <exec_byte_code+3837>, 
          0x6c3f36 <exec_byte_code+3839>, 0x6c3f38 <exec_byte_code+3841>, 0x6c3f3a <exec_byte_code+3843>, 0x6c3f3a <exec_byte_code+3843>, 0x6c3ed1 <exec_byte_code+3738>, 0x6c3ef1 <exec_byte_code+3770>, 0x6c4015 <exec_byte_code+4062>, 
          0x6c4017 <exec_byte_code+4064>, 0x6c4019 <exec_byte_code+4066>, 0x6c401b <exec_byte_code+4068>, 0x6c401d <exec_byte_code+4070>, 0x6c401d <exec_byte_code+4070>, 0x6c3fb4 <exec_byte_code+3965>, 0x6c3fd4 <exec_byte_code+3997>, 
          0x6c4999 <exec_byte_code+6498>, 0x6c486c <exec_byte_code+6197>, 0x6c4860 <exec_byte_code+6185>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 
          0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c4bf8 <exec_byte_code+7105>, 0x6c4d2f <exec_byte_code+7416>, 0x6c4da3 <exec_byte_code+7532>, 0x6c4e18 <exec_byte_code+7649>, 0x6c4e8e <exec_byte_code+7767>, 
          0x6c3933 <exec_byte_code+2300>, 0x6c39cb <exec_byte_code+2452>, 0x6c4f1f <exec_byte_code+7912>, 0x6c385f <exec_byte_code+2088>, 0x6c3a40 <exec_byte_code+2569>, 0x6c4f9e <exec_byte_code+8039>, 0x6c5013 <exec_byte_code+8156>, 
          0x6c5062 <exec_byte_code+8235>, 0x6c50d7 <exec_byte_code+8352>, 0x6c5133 <exec_byte_code+8444>, 0x6c5227 <exec_byte_code+8688>, 0x6c5276 <exec_byte_code+8767>, 0x6c52eb <exec_byte_code+8884>, 0x6c5383 <exec_byte_code+9036>, 
          0x6c53d2 <exec_byte_code+9115>, 0x6c5421 <exec_byte_code+9194>, 0x6c5496 <exec_byte_code+9311>, 0x6c550b <exec_byte_code+9428>, 0x6c5580 <exec_byte_code+9545>, 0x6c5618 <exec_byte_code+9697>, 0x6c5674 <exec_byte_code+9789>, 
          0x6c56d0 <exec_byte_code+9881>, 0x6c57c4 <exec_byte_code+10125>, 0x6c584e <exec_byte_code+10263>, 0x6c58d8 <exec_byte_code+10401>, 0x6c5aba <exec_byte_code+10883>, 0x6c5b34 <exec_byte_code+11005>, 
          0x6c5bae <exec_byte_code+11127>, 0x6c5c28 <exec_byte_code+11249>, 0x6c5ca2 <exec_byte_code+11371>, 0x6c5cfe <exec_byte_code+11463>, 0x6c5d8c <exec_byte_code+11605>, 0x6c5de8 <exec_byte_code+11697>, 
          0x6c5e44 <exec_byte_code+11789>, 0x6c5ea0 <exec_byte_code+11881>, 0x6c5fcd <exec_byte_code+12182>, 0x6c46df <exec_byte_code+5800>, 0x6c6032 <exec_byte_code+12283>, 0x6c6081 <exec_byte_code+12362>, 
          0x6c616d <exec_byte_code+12598>, 0x6c61dd <exec_byte_code+12710>, 0x6c6242 <exec_byte_code+12811>, 0x6c6291 <exec_byte_code+12890>, 0x6c62de <exec_byte_code+12967>, 0x6c632b <exec_byte_code+13044>, 
          0x6c6380 <exec_byte_code+13129>, 0x6c745f <exec_byte_code+17448>, 0x6c63df <exec_byte_code+13224>, 0x6c642c <exec_byte_code+13301>, 0x6c6479 <exec_byte_code+13378>, 0x6c64c6 <exec_byte_code+13455>, 
          0x6c6513 <exec_byte_code+13532>, 0x6c6560 <exec_byte_code+13609>, 0x6c46df <exec_byte_code+5800>, 0x6c745f <exec_byte_code+17448>, 0x6c65af <exec_byte_code+13688>, 0x6c6609 <exec_byte_code+13778>, 
          0x6c6658 <exec_byte_code+13857>, 0x6c66a7 <exec_byte_code+13936>, 0x6c671c <exec_byte_code+14053>, 0x6c6791 <exec_byte_code+14170>, 0x6c67e0 <exec_byte_code+14249>, 0x6c6923 <exec_byte_code+14572>, 
          0x6c6998 <exec_byte_code+14689>, 0x6c6a0d <exec_byte_code+14806>, 0x6c6a82 <exec_byte_code+14923>, 0x6c6acf <exec_byte_code+15000>, 0x6c745f <exec_byte_code+17448>, 0x6c45fb <exec_byte_code+5572>, 
          0x6c40d7 <exec_byte_code+4256>, 0x6c37b4 <exec_byte_code+1917>, 0x6c41bb <exec_byte_code+4484>, 0x6c4258 <exec_byte_code+4641>, 0x6c42f2 <exec_byte_code+4795>, 0x6c45a8 <exec_byte_code+5489>, 0x6c45c0 <exec_byte_code+5513>, 
          0x6c3d99 <exec_byte_code+3426>, 0x6c469a <exec_byte_code+5731>, 0x6c4717 <exec_byte_code+5856>, 0x6c47a4 <exec_byte_code+5997>, 0x6c47e9 <exec_byte_code+6066>, 0x6c49e6 <exec_byte_code+6575>, 0x6c4a66 <exec_byte_code+6703>, 
          0x6c4afe <exec_byte_code+6855>, 0x6c4b68 <exec_byte_code+6961>, 0x6c4085 <exec_byte_code+4174>, 0x6c6b1e <exec_byte_code+15079>, 0x6c6bb6 <exec_byte_code+15231>, 0x6c6c05 <exec_byte_code+15310>, 
          0x6c6c54 <exec_byte_code+15389>, 0x6c6ca3 <exec_byte_code+15468>, 0x6c6cf2 <exec_byte_code+15547>, 0x6c6d67 <exec_byte_code+15664>, 0x6c6ddc <exec_byte_code+15781>, 0x6c6e51 <exec_byte_code+15898>, 
          0x6c6ec6 <exec_byte_code+16015>, 0x6c7057 <exec_byte_code+16416>, 0x6c70cc <exec_byte_code+16533>, 0x6c7141 <exec_byte_code+16650>, 0x6c7190 <exec_byte_code+16729>, 0x6c7205 <exec_byte_code+16846>, 
          0x6c727a <exec_byte_code+16963>, 0x6c72c9 <exec_byte_code+17042>, 0x6c7318 <exec_byte_code+17121>, 0x6c5efc <exec_byte_code+11973>, 0x6c5f58 <exec_byte_code+12065>, 0x6c7374 <exec_byte_code+17213>, 
          0x6c73ea <exec_byte_code+17331>, 0x6c745f <exec_byte_code+17448>, 0x6c438c <exec_byte_code+4949>, 0x6c43b2 <exec_byte_code+4987>, 0x6c4431 <exec_byte_code+5114>, 0x6c44b0 <exec_byte_code+5241>, 0x6c452c <exec_byte_code+5365>, 
          0x6c518f <exec_byte_code+8536>, 0x6c572c <exec_byte_code+9973>, 0x6c60d5 <exec_byte_code+12446>, 0x6c7639 <exec_byte_code+17922>, 0x6c76b8 <exec_byte_code+18049>, 0x6c745f <exec_byte_code+17448>, 
          0x6c745f <exec_byte_code+17448>, 0x6c7764 <exec_byte_code+18221>, 0x6c780a <exec_byte_code+18387>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 
          0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c745f <exec_byte_code+17448>, 0x6c7a17 <exec_byte_code+18912> <repeats 64 times>}
        const_length = 77
        bytestr_length = 443
        vectorp = 0x7fffe58c9ad8
        quitcounter = 41 ')'
        stack_items = 13
        sa_avail = 15837
        sa_count = 5
        sa_must_free = false
        alloc = 0x7fffffffd2e0
        item_bytes = 104
        stack_base = 0x7fffffffd2e0
        top = 0x7fffffffd2f0
        stack_lim = 0x7fffffffd348
        bytestr_data = 0x7fffffffd348 "\b\203\b"
        pc = 0x7fffffffd4bc "\210)\210\375\376\377\"\210\201H"
        count = 5
        result = XIL(0x7ffff56555e0)
#19 0x0000000000670dd8 in funcall_lambda (fun=XIL(0x7fffe58c9aa5), nargs=0, arg_vector=0x7fffffffdb30) at eval.c:3026
        size = 5
        val = XIL(0x7fffffffdb30)
        syms_left = make_number(0)
        next = XIL(0x7fffffffdae0)
        lexenv = XIL(0)
        count = 5
        i = 6770530
        optional = false
        rest = false
        previous_optional_or_rest = false
#20 0x0000000000670a00 in apply_lambda (fun=XIL(0x7fffe58c9aa5), args=XIL(0), count=4) at eval.c:2962
        args_left = XIL(0)
        i = 0
        numargs = 0
        arg_vector = 0x7fffffffdb30
        tem = XIL(0)
        sa_avail = 16384
        sa_count = 5
        sa_must_free = false
#21 0x000000000066e98b in eval_sub (form=XIL(0x7fffe5a1b57b)) at eval.c:2335
        fun = XIL(0x7fffe58c9aa5)
        val = XIL(0x7fffe53c0500)
        original_fun = XIL(0x7fffe4a940d0)
        original_args = XIL(0)
        funcar = XIL(0x7fffffffdc40)
        count = 4
        argvals = {XIL(0x597785), XIL(0xeb5930), XIL(0x7fffffffdd00), XIL(0x67208b), XIL(0), XIL(0xe3d470), XIL(0x7ad0), XIL(0x7ad0)}
#22 0x000000000066dc7b in Feval (form=XIL(0x7fffe5a1b57b), lexical=XIL(0)) at eval.c:2110
        count = 3
#23 0x00000000005a0bcc in top_level_2 () at keyboard.c:1122
No locals.
#24 0x000000000066bc68 in internal_condition_case (bfun=0x5a0baf <top_level_2>, handlers=XIL(0x5310), hfun=0x5a059c <cmd_error>) at eval.c:1336
        val = XIL(0xed25a8)
        c = 0xed2690
#25 0x00000000005a0c10 in top_level_1 (ignore=XIL(0)) at keyboard.c:1130
No locals.
#26 0x000000000066b107 in internal_catch (tag=XIL(0xc7e0), func=0x5a0bce <top_level_1>, arg=XIL(0)) at eval.c:1101
        val = XIL(0x7fffffffde30)
        c = 0xed2570
#27 0x00000000005a0b01 in command_loop () at keyboard.c:1091
No locals.
#28 0x00000000005a0096 in recursive_edit_1 () at keyboard.c:698
        count = 1
        val = XIL(0x7fffffffde90)
#29 0x00000000005a0284 in Frecursive_edit () at keyboard.c:769
        count = 0
        buffer = XIL(0)
#30 0x000000000059e029 in main (argc=4, argv=0x7fffffffe0e8) at emacs.c:1929
        stack_bottom_variable = 0x0
        do_initial_setlocale = true
        skip_args = 1
        no_loadup = false
        junk = 0x0
        dname_arg = 0x0
        ch_to_dir = 0x0
        original_pwd = 0x0
        argv0_base = 0x7fffffffe4fe "emacs"
        is_temacs = false
        loaded_dump = 0x0
        dump_mode = 0x0
        disable_aslr = false
        rlim = {
          rlim_cur = 10022912, 
          rlim_max = 18446744073709551615
        }
        sockfd = -1

Lisp Backtrace:
"dump-emacs-portable" (0xffffb7b0)
"eval" (0xffffba28)
"command-line-1" (0xffffc5a0)
"command-line" (0xffffd2f8)
"normal-top-level" (0xffffdb30)
(gdb) 

Re: Preview: portable dumper

[Charles A. Roelli]

> Date: Tue, 13 Feb 2018 18:37:43 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> CC: emacs-devel@gnu.org
> Reply-to: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Daniel Colascione <dancol@dancol.org>
> > Date: Mon, 12 Feb 2018 12:18:36 -0800
> > 
> > I've pushed the portable dumper to the pdumper Savannah branch. It 
> > should support a fully PIC Emacs.
> 
> Thanks.  I'd urge people to try this branch and report any issues they
> see.

On Apple's version of FSF GCC, I saw these kinds of errors:

  CC       pdumper.o
pdumper.c:63: warning: unknown option after ‘#pragma GCC diagnostic’ kind
pdumper.c: In function ‘dump_reloc_set_offset’:
pdumper.c:289: error: #pragma GCC diagnostic not allowed inside functions
pdumper.c:289: error: #pragma GCC diagnostic not allowed inside functions
pdumper.c:291: error: #pragma GCC diagnostic not allowed inside functions

They're fixed locally with this patch:

diff --git a/src/pdumper.c b/src/pdumper.c
index a05b2c4..844ec25 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -54,7 +54,10 @@
 
 #ifdef HAVE_PDUMPER
 
-#ifdef __GNUC__
+#if defined(__GNUC__) && !defined(__APPLE__) /* Apple's old GCC (based
+                                                on FSF's GCC) does not
+                                                support pragmas in
+                                                functions.  */
 # pragma GCC diagnostic error "-Wconversion"
 # pragma GCC diagnostic error "-Wshadow"
 # define ALLOW_IMPLICIT_CONVERSION                       \

Afterwards I had only this error remaining:

pdumper.c: In function ‘dump_anonymous_allocate_posix’:
pdumper.c:4043: error: ‘MAP_ANONYMOUS’ undeclared (first use in this function)
pdumper.c:4043: error: (Each undeclared identifier is reported only once
pdumper.c:4043: error: for each function it appears in.)

Seems like MAP_ANONYMOUS is not defined on all systems -- I don't know
where to go from there.

Re: Preview: portable dumper

[Alan Third]

On Sat, Feb 17, 2018 at 12:53:48PM +0100, Charles A. Roelli wrote:
> On Apple's version of FSF GCC, I saw these kinds of errors:
> 
>   CC       pdumper.o
> pdumper.c:63: warning: unknown option after ‘#pragma GCC diagnostic’ kind
> pdumper.c: In function ‘dump_reloc_set_offset’:
> pdumper.c:289: error: #pragma GCC diagnostic not allowed inside functions
> pdumper.c:289: error: #pragma GCC diagnostic not allowed inside functions
> pdumper.c:291: error: #pragma GCC diagnostic not allowed inside functions
> 
> They're fixed locally with this patch:
> 
> diff --git a/src/pdumper.c b/src/pdumper.c
> index a05b2c4..844ec25 100644
> --- a/src/pdumper.c
> +++ b/src/pdumper.c
> @@ -54,7 +54,10 @@
>  
>  #ifdef HAVE_PDUMPER
>  
> -#ifdef __GNUC__
> +#if defined(__GNUC__) && !defined(__APPLE__) /* Apple's old GCC (based
> +                                                on FSF's GCC) does not
> +                                                support pragmas in
> +                                                functions.  */
>  # pragma GCC diagnostic error "-Wconversion"
>  # pragma GCC diagnostic error "-Wshadow"
>  # define ALLOW_IMPLICIT_CONVERSION                       \

I’m pretty sure this will disable these pragmas for more recent clang
versions on macOS, and probably any newer version of GCC people have
compiled themselves..
-- 
Alan Third

Re: Preview: portable dumper

[Charles A. Roelli]

> From: Alan Third <alan@idiocy.org>
> 
> On Sat, Feb 17, 2018 at 12:53:48PM +0100, Charles A. Roelli wrote:
> > On Apple's version of FSF GCC, I saw these kinds of errors:
> > 
> >   CC       pdumper.o
> > pdumper.c:63: warning: unknown option after ‘#pragma GCC diagnostic’ kind
> > pdumper.c: In function ‘dump_reloc_set_offset’:
> > pdumper.c:289: error: #pragma GCC diagnostic not allowed inside functions
> > pdumper.c:289: error: #pragma GCC diagnostic not allowed inside functions
> > pdumper.c:291: error: #pragma GCC diagnostic not allowed inside functions
> > 
> > They're fixed locally with this patch:
> > 
> > diff --git a/src/pdumper.c b/src/pdumper.c
> > index a05b2c4..844ec25 100644
> > --- a/src/pdumper.c
> > +++ b/src/pdumper.c
> > @@ -54,7 +54,10 @@
> >  
> >  #ifdef HAVE_PDUMPER
> >  
> > -#ifdef __GNUC__
> > +#if defined(__GNUC__) && !defined(__APPLE__) /* Apple's old GCC (based
> > +                                                on FSF's GCC) does not
> > +                                                support pragmas in
> > +                                                functions.  */
> >  # pragma GCC diagnostic error "-Wconversion"
> >  # pragma GCC diagnostic error "-Wshadow"
> >  # define ALLOW_IMPLICIT_CONVERSION                       \
> 
> I’m pretty sure this will disable these pragmas for more recent clang
> versions on macOS, and probably any newer version of GCC people have
> compiled themselves..
> -- 
> Alan Third

Whoops, my bad.  If there's a feature test for pragmas, that would of
course be better.

Re: Preview: portable dumper

[Andy Moreton]

On Tue 13 Feb 2018, Eli Zaretskii wrote:

>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 12 Feb 2018 12:18:36 -0800
>> 
>> I've pushed the portable dumper to the pdumper Savannah branch. It 
>> should support a fully PIC Emacs.
>
> Thanks.  I'd urge people to try this branch and report any issues they
> see.

The branch currently fails for out of tree builds (tested with mingw64
on msys2 for 64bit Windows). Something like this is needed to fix it:

diff --git a/src/Makefile.in b/src/Makefile.in
index e2950a02eb..0df59b90a8 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -411,11 +411,13 @@ base_obj =
 	$(W32_OBJ) $(WINDOW_SYSTEM_OBJ) $(XGSELOBJ) $(JSON_OBJ)
 obj = $(base_obj) $(NS_OBJC_OBJ)
 
-dmpstruct_headers=lisp.h buffer.h intervals.h charset.h
+dmpstruct_headers = ${srcdir}/lisp.h ${srcdir}/buffer.h ${srcdir}/intervals.h \
+	${srcdir}/charset.h
 pdumper.o: dmpstruct.h
-dmpstruct.h: dmpstruct.awk
+dmpstruct.h: ${srcdir}/dmpstruct.awk
 dmpstruct.h: $(libsrc)/make-fingerprint$(EXEEXT) $(dmpstruct_headers)
-	POSIXLY_CORRECT=1 awk -f dmpstruct.awk $(dmpstruct_headers) > $@
+	POSIXLY_CORRECT=1 awk -f ${srcdir}/dmpstruct.awk \
+	$(dmpstruct_headers) > $@
 
 ## Object files used on some machine or other.
 ## These go in the DOC file on all machines in case they are needed.

Re: Preview: portable dumper

[Stefan Monnier]

> I've pushed the portable dumper to the pdumper Savannah branch. It should
> support a fully PIC Emacs.

Thank you very much Daniel.  I hope this can move to master soon.
I just tried it (on Debian testing, with the latest commit at the time,
i.e. 5232a17c365f871a84603caba81712415a260042) and here's my experience:

- compilation fails with

    In file included from character.h:27:0,
                     from buffer.h:27,
                     from pdumper.c:18:
    pdumper.c: In function ‘intmax_t_to_lisp’:
    pdumper.c:699:29: error: conversion to ‘EMACS_INT {aka int}’ from ‘intmax_t {aka long long int}’ may alter its value [-Werror=conversion]
         return INTEGER_TO_CONS (value);  \
                                 ^
    lisp.h:3572:19: note: in definition of macro ‘INTEGER_TO_CONS’
        ? make_number (i)          \
                       ^
    pdumper.c:703:1: note: in expansion of macro ‘DEFINE_TOLISP_FUNC’
     DEFINE_TOLISP_FUNC (intmax_t_to_lisp, intmax_t);
     ^~~~~~~~~~~~~~~~~~
    pdumper.c: In function ‘pdumper_load’:
    pdumper.c:5049:14: error: conversion to ‘unsigned int’ from ‘off_t {aka long long int}’ may alter its value [-Werror=conversion]
          .size = dump_size - header->cold_start,
                  ^~~~~~~~~
    pdumper.c:5072:21: error: conversion to ‘intptr_t {aka int}’ from ‘off_t {aka long long int}’ may alter its value [-Werror=conversion]
       dump_public.end = dump_public.start + dump_size;
                         ^~~~~~~~~~~
    In file included from character.h:27:0,
                     from buffer.h:27,
                     from pdumper.c:18:

  I fixed those with the patch below, but it seems there's some issues
  with off_t vs size_t (I guess a dump file shouldn't be bigger than the
  available address space, so even though the max size of files can be
  much bigger, we can use smaller integers to refer to size&position
  within dump files).

- after that the dump itself crashed as follows:

    pdumper.c:2468: Emacs fatal error: assertion failed: vector_nbytes ((struct Lisp_Vector *) in) == out_size

  Backtrace below.


        Stefan


Finding pointers to doc strings...
Finding pointers to doc strings...done
Dumping under the name bootstrap-emacs.pdmp
dumping fingerprint: d003f05f220e1040ec992fccec03d65c4934d9181cbde233f17892969593f435

pdumper.c:2468: Emacs fatal error: assertion failed: vector_nbytes ((struct Lisp_Vector *) in) == out_size

Thread 1 "temacs" hit Breakpoint 1, terminate_due_to_signal (sig=6, 
    backtrace_limit=2147483647) at emacs.c:357
(gdb) bt
#0  0x08194a5a in terminate_due_to_signal (sig=6, backtrace_limit=2147483647)
    at emacs.c:357
#1  0x08216a9e in die (msg=0x833c654 "vector_nbytes ((struct Lisp_Vector *) in) == out_size", file=0x833b6f8 "pdumper.c", line=2468) at alloc.c:7796
#2  0x08227321 in dump_object_start_pseudovector (ctx=ctx@entry=0xbfffec08, out_hdr=out_hdr@entry=0xbfffe730, out_size=out_size@entry=536, in_hdr=0xbfffe948)
    at pdumper.c:2468
#3  0x082284d0 in dump_buffer (ctx=0xbfffec08, in_buffer=0x8718d20)
    at pdumper.c:2632
#4  0x08228be1 in dump_vectorlike (ctx=ctx@entry=0xbfffec08, v=0x8718d20)
    at pdumper.c:2825
#5  0x08228dc1 in dump_object_1 (ctx=ctx@entry=0xbfffec08, object=object@entry=XIL(0x8718d25)) at pdumper.c:2889
#6  0x08228f2e in dump_object (ctx=ctx@entry=0xbfffec08, object=XIL(0x8718d25)) at pdumper.c:2954
#7  0x0822981f in Fdump_emacs_portable (filename=XIL(0x922ce54), track_referrers=XIL(0)) at pdumper.c:3780
#8  0x08251349 in eval_sub (form=...) at eval.c:2290
#9  0x08252337 in Fif (args=XIL(0x88f13db)) at eval.c:414
#10 0x08250ff4 in eval_sub (form=...) at eval.c:2239
#11 0x082518e9 in Fprogn (body=...) at eval.c:459
#12 0x08253ef5 in Flet (args=XIL(0x88b029b)) at eval.c:973
#13 0x08250ff4 in eval_sub (form=...) at eval.c:2239
#14 0x08252337 in Fif (args=XIL(0x87f975b)) at eval.c:414
#15 0x08250ff4 in eval_sub (form=...) at eval.c:2239
#16 0x08286d3c in readevalloop (readcharfun=XIL(0x34c8), infile0=infile0@entry=0xbffff1a4, sourcename=..., 
    sourcename@entry=XIL(0x873c434), printflag=false, unibyte=XIL(0), readfun=XIL(0), start=XIL(0), end=XIL(0)) at lread.c:2073
#17 0x08287cbc in Fload (file=XIL(0x8723644), noerror=XIL(0), nomessage=XIL(0), nosuffix=XIL(0), must_suffix=XIL(0)) at lread.c:1457
#18 0x082513c7 in eval_sub (form=...) at eval.c:2301
#19 0x0825450a in Feval (form=XIL(0x87128bb), lexical=XIL(0)) at eval.c:2107
#20 0x081992fe in top_level_2 () at keyboard.c:1122
#21 0x0824c263 in internal_condition_case (bfun=0x81992c8 <top_level_2>, handlers=XIL(0x2988), hfun=0x819eb61 <cmd_error>) at eval.c:1336
#22 0x0819cbaf in top_level_1 (ignore=XIL(0)) at keyboard.c:1130
#23 0x0824c160 in internal_catch (tag=XIL(0x64e0), func=0x819caf0 <top_level_1>, arg=XIL(0)) at eval.c:1101
#24 0x0819b8ef in command_loop () at keyboard.c:1091
#25 0x0819e233 in recursive_edit_1 () at keyboard.c:698
#26 0x0819e896 in Frecursive_edit () at keyboard.c:769
#27 0x08195aaa in main (argc=<optimized out>, argv=<optimized out>)
    at emacs.c:1915

Lisp Backtrace:
"dump-emacs-portable" (0xbfffed5c)
"if" (0xbfffee4c)
"let" (0xbfffef8c)
"if" (0xbffff05c)
"load" (0xbffff28c)
(gdb) 

Re: Preview: portable dumper

[Daniel Colascione]

On 02/14/2018 08:28 PM, Stefan Monnier wrote:
>> I've pushed the portable dumper to the pdumper Savannah branch. It should
>> support a fully PIC Emacs.
> 
> Thank you very much Daniel.  I hope this can move to master soon.
> I just tried it (on Debian testing, with the latest commit at the time,
> i.e. 5232a17c365f871a84603caba81712415a260042) and here's my experience:
> 
> - compilation fails with
> 
>      In file included from character.h:27:0,
>                       from buffer.h:27,
>                       from pdumper.c:18:
>      pdumper.c: In function ‘intmax_t_to_lisp’:
>      pdumper.c:699:29: error: conversion to ‘EMACS_INT {aka int}’ from ‘intmax_t {aka long long int}’ may alter its value [-Werror=conversion]
>           return INTEGER_TO_CONS (value);  \

Fixed. I'm tempted to just convert INTEGER_TO_CONS to a pair of 
functions (one signed, one unsigned) that just take an {,u}intmax_t.

>      pdumper.c:2468: Emacs fatal error: assertion failed: vector_nbytes ((struct Lisp_Vector *) in) == out_size

Modified the code to tolerate end-of-pseudovector padding in 32-bit builds.

Re: Preview: portable dumper

[Paul Eggert]

On 02/15/2018 02:13 PM, Daniel Colascione wrote:
> I'm tempted to just convert INTEGER_TO_CONS to a pair of functions 
> (one signed, one unsigned) that just take an {,u}intmax_t. 

Sometimes one does not know whether an integer is signed. For example, 
the ino_t type is signed on some platforms, and not on others. This is 
why INTEGER_CONS is written the way it is: it is supposed to do the 
right thing regardless of the signedness of its argument.

Re: Preview: portable dumper

[Daniel Colascione]

[-- Attachment #1: Type: text/html, Size: 1139 bytes --]

Re: Preview: portable dumper

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1593 bytes --]

Here are some problems I ran into on Fedora 27 x86-64. Proposed patch 
attached.

   CC       pdumper.o
In file included from character.h:27:0,
                  from buffer.h:27,
                  from pdumper.c:18:
pdumper.c: In function ‘dump_object_1’:
pdumper.c:2920:70: error: left-hand operand of comma expression has no 
effect [-Werror=unused-value]
            eassert (("should not be dumping int: is self-representing", 
0));
                                                                       ^
lisp.h:195:43: note: in definition of macro ‘eassert’
  # define eassert(cond) ((void) (false && (cond))) /* Check COND 
compiles.  */
                                            ^~~~
pdumper.c: In function ‘dump_vectorlike’:
pdumper.c:2823:10: error: this statement may fall through 
[-Werror=implicit-fallthrough=]
        if ((v->header.size & PSEUDOVECTOR_SIZE_MASK) != FONT_SPEC_MAX &&
           ^
pdumper.c:2827:5: note: here
      case PVEC_NORMAL_VECTOR:
      ^~~~
pdumper.c: At top level:
pdumper.c:119:0: error: macro "MAX_OFF_T" is not used 
[-Werror=unused-macros]
  # define MAX_OFF_T ((sizeof (off_t) == sizeof (int32_t)) ?      \

cc1: all warnings being treated as errors
make[1]: *** [Makefile:386: pdumper.o] Error 1
make[1]: Leaving directory '/home/eggert/src/gnu/emacs/pdumper/src'
make: *** [Makefile:420: src] Error 2


[-- Attachment #2: 0001-Pacify-pdumper-for-GCC-7.3.1.patch --]
[-- Type: text/x-patch, Size: 1658 bytes --]

From 990d36af757a3d16b08f438f96431e8d07ae371e Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Thu, 15 Feb 2018 14:54:50 -0800
Subject: [PATCH] Pacify pdumper for GCC 7.3.1

* src/pdumper.c (MAX_OFF_T): Remove.
(dump_vectorlike): Add FALLTHROUGH.
(dump_object_1): Avoid unportable eassert.
---
 src/pdumper.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/src/pdumper.c b/src/pdumper.c
index 499c8dc27c..9a286c0247 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -115,11 +115,6 @@ verify (sizeof (off_t) == sizeof (int32_t) ||
         sizeof (off_t) == sizeof (int64_t));
 verify (CHAR_BIT == 8);
 
-#ifndef MAX_OFF_T
-# define MAX_OFF_T ((sizeof (off_t) == sizeof (int32_t)) ?      \
-                    INT32_MAX : INT64_MAX)
-#endif
-
 #define DIVIDE_ROUND_UP(x, y) (((x) + (y) - 1) / (y))
 
 static const char dump_magic[16] = {
@@ -2823,7 +2818,7 @@ dump_vectorlike (struct dump_context *ctx, const struct Lisp_Vector *v)
       if ((v->header.size & PSEUDOVECTOR_SIZE_MASK) != FONT_SPEC_MAX &&
           (v->header.size & PSEUDOVECTOR_SIZE_MASK) != FONT_ENTITY_MAX)
         error_unsupported_dump_object(ctx, lv, "font");
-      /* Fall through */
+      FALLTHROUGH;
     case PVEC_NORMAL_VECTOR:
     case PVEC_COMPILED:
     case PVEC_CHAR_TABLE:
@@ -2917,7 +2912,7 @@ dump_object_1 (struct dump_context *ctx, Lisp_Object object)
           offset = dump_float (ctx, XFLOAT (object));
           break;
         case_Lisp_Int:
-          eassert (("should not be dumping int: is self-representing", 0));
+	  emacs_abort ();
         default:
           emacs_abort ();
         }
-- 
2.14.3

Re: Preview: portable dumper

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 893 bytes --]

I had the following problem on Fedora 27 when building the pdumper 
branch just now:

   CC       emacs.o
emacs.c: In function ‘load_dump’:
emacs.c:771:5: error: jump skips variable initialization 
[-Werror=jump-misses-init]
      goto out;
      ^~~~
emacs.c:817:2: note: label ‘out’ defined here
   out:
   ^~~
emacs.c:800:15: note: ‘path_exec’ declared here
    const char *path_exec = PATH_EXEC;
                ^~~~~~~~~
emacs.c:788:5: error: jump skips variable initialization 
[-Werror=jump-misses-init]
      goto out;
      ^~~~
emacs.c:817:2: note: label ‘out’ defined here
   out:
   ^~~
emacs.c:800:15: note: ‘path_exec’ declared here
    const char *path_exec = PATH_EXEC;
                ^~~~~~~~~
cc1: all warnings being treated as errors

The attached patch fixes it.


[-- Attachment #2: 0001-src-emacs.c-load_dump-Pacify-GCC-Wjump-misses-init.patch --]
[-- Type: text/x-patch, Size: 1079 bytes --]

From 80c596f9a992c9424a5a15716a6639f5efedbed5 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Thu, 15 Feb 2018 14:34:49 -0800
Subject: [PATCH] * src/emacs.c (load_dump): Pacify GCC -Wjump-misses-init.

---
 src/emacs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/emacs.c b/src/emacs.c
index 0d7c6ed9b3..7ed4f2758e 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -748,6 +748,7 @@ load_dump (int *inout_argc,
 
   /* Look for an explicitly-specified dump file.  */
 
+  const char *path_exec = PATH_EXEC;
   char *dump_file = find_and_remove_dump_file_argument (&argc, &argv);
   if (initialized && dump_file)
     /* TODO: maybe more thoroughly scrub process environment in order
@@ -797,7 +798,6 @@ load_dump (int *inout_argc,
 
      FIXME: this doesn't work with emacs-XX.YY.ZZ.pdmp versioned files.  */
   argv0_base = "emacs";
-  const char *path_exec = PATH_EXEC;
 #ifdef WINDOWSNT
   /* On MS-Windows, PATH_EXEC normally starts with a literal
      "%emacs_dir%", so it will never work without some tweaking.  */
-- 
2.14.3

Re: Preview: portable dumper

[andres.ramirez]

Hello.

Doing make bootstrap on archlinux with cc (GCC) 7.2.1 20180116

I am getting this:
--8<---------------cut here---------------start------------->8---
  CC       pdumper.o
  CC       data.o
In file included from character.h:27:0,
                 from buffer.h:27,
                 from pdumper.c:18:
pdumper.c: In function ‘intmax_t_to_lisp’:
pdumper.c:699:29: error: conversion to ‘EMACS_INT {aka int}’ from ‘intmax_t {aka long long int}’ may alter its value [-Werror=conversion]
     return INTEGER_TO_CONS (value);  \
                             ^
lisp.h:3572:19: note: in definition of macro ‘INTEGER_TO_CONS’
    ? make_number (i)          \
                   ^
pdumper.c:703:1: note: in expansion of macro ‘DEFINE_TOLISP_FUNC’
 DEFINE_TOLISP_FUNC (intmax_t_to_lisp, intmax_t);
 ^~~~~~~~~~~~~~~~~~
pdumper.c: In function ‘pdumper_load’:
pdumper.c:5049:14: error: conversion to ‘unsigned int’ from ‘off_t {aka long long int}’ may alter its value [-Werror=conversion]
      .size = dump_size - header->cold_start,
              ^~~~~~~~~
pdumper.c:5072:21: error: conversion to ‘intptr_t {aka int}’ from ‘off_t {aka long long int}’ may alter its value [-Werror=conversion]
   dump_public.end = dump_public.start + dump_size;
                     ^~~~~~~~~~~
cc1: some warnings being treated as errors
make[2]: *** [Makefile:386: pdumper.o] Error 1
make[2]: *** Waiting for unfinished jobs....
--8<---------------cut here---------------end--------------->8---

Re: Preview: portable dumper

[Daniele Nicolodi]

[-- Attachment #1: Type: text/plain, Size: 405 bytes --]

On 12/02/2018 13:18, Daniel Colascione wrote:
> I've pushed the portable dumper to the pdumper Savannah branch.

Hi Daniel,

the recently introduced `dmpstruct.awk` script fails with a non-GNU awk:
POSIX awk does not like a | occurring just after a grouping ( or just
before a ) in a regular expression.  The attached patch fixes the syntax
preserving the meaning of the regular expressions.

Cheers,
Dan

[-- Attachment #2: dmpstruct.patch --]
[-- Type: text/plain, Size: 509 bytes --]

diff --git a/src/dmpstruct.awk b/src/dmpstruct.awk
index 608c26fcc5..0a22030405 100755
--- a/src/dmpstruct.awk
+++ b/src/dmpstruct.awk
@@ -10,10 +10,10 @@ BEGIN {
   struct_name = $2
   close (tmpfile)
 }
-/^(enum|struct|union) [a-zA-Z0-9_]+([\t ]|\/\*.*\*\/)*$/, /^(|  )};$/ {
+/^(enum|struct|union) [a-zA-Z0-9_]+([\t ]|\/\*.*\*\/)*$/, /^(  )?};$/ {
   print $0 > tmpfile
 }
-/^(|  )};$/ {
+/^(  )?};$/ {
   if (struct_name != "") {
     fflush (tmpfile)
     cmd = "../lib-src/make-fingerprint -r " tmpfile

Re: Preview: portable dumper

[Daniel Colascione]

On 02/19/2018 02:01 PM, Daniele Nicolodi wrote:
> On 12/02/2018 13:18, Daniel Colascione wrote:
>> I've pushed the portable dumper to the pdumper Savannah branch.
> 
> Hi Daniel,
> 
> the recently introduced `dmpstruct.awk` script fails with a non-GNU awk:
> POSIX awk does not like a | occurring just after a grouping ( or just
> before a ) in a regular expression.  The attached patch fixes the syntax
> preserving the meaning of the regular expressions.

Thanks. Applied.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Karl Fogel <kfogel@red-bean.com>
> Date: Fri, 02 Dec 2016 14:11:23 -0600
> 
> Following up to what John said, with a perspective that may be useful to Eli:

Thanks, but it isn't useful.  Why? because, like John, you don't think
that my opinion about the preferred solution to getting rid of unexec
is right.

What would be useful is a perspective from someone who understands the
issues and agrees with me about the preferred solution.  Because only
such a person can see the dilemma through my eyes, and only that
person can find the right words.

> When I was in a position similar to yours (one of a small group of core co-maintainers, in the Subversion project), there were a few occasions when a major technical decision went in a direction I didn't agree with.  My disagreement was not of the "this will destroy the project" sort, but I did feel the decisions were poor technical choices that could be a long-term drag on the project -- creating extra maintenance work for existing developers, and creating barriers to entry for new developers.  In other words, objections very similar to yours now.

No, they aren't similar.  In cases like the one you describe, I
invariably seek a compromise or give in.  It's what you call "destroy
the project" thing that makes the difference.

> Your value as a maintainer is not lessened if a particular decision doesn't go the way you recommended.  (Of course, if maintainership becomes unenjoyable to you because of the decision, that's a different question, and only you can make that call.)

"Unenjoyable"?  It's intolerable.  It would mean that on my watch
something happened that I believe is detrimental to a project which I
took upon myself to improve, and I saw it happening and didn't prevent
it.  Can you understand my plight?

Re: Preview: portable dumper

[Daniel Colascione]

On Sat, Dec 03 2016, Eli Zaretskii wrote:
>> From: Karl Fogel <kfogel@red-bean.com>
>> Date: Fri, 02 Dec 2016 14:11:23 -0600
>> 
>> Following up to what John said, with a perspective that may be useful to Eli:
>
> Thanks, but it isn't useful.  Why? because, like John, you don't think
> that my opinion about the preferred solution to getting rid of unexec
> is right.
>
> What would be useful is a perspective from someone who understands the
> issues and agrees with me about the preferred solution.  Because only
> such a person can see the dilemma through my eyes, and only that
> person can find the right words.

We've all been contributing to Emacs for many years. I think we have a
pretty good grasp of the issues and potential problems.  I understand
that you think there's an existential risk that goes into this change,
but I think you're mistaken, and a lot of other people agree. We all
have access to the same information; it's unlikely (but not impossible)
that you're the only one who can see certain potential problems. Please
give this change the benefit of the doubt.

Even if your worst-case scenario comes to pass, the worst outcome
outcome will just be more fiddly maintenance. I think the pdumper code
is pretty clean; I'm open to suggestions for making it moreso. The code
doesn't depend on arcane details of underlying operating systems --- it
depends only on details of Emacs itself.  We don't need ELF experts.
If there's no one left who can understand pdumper, there's no one left
who can understand the GC either.  In that scenario, we're already in
Mad Max territory anyway.

You won't have to touch this code.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Cc: Karl Fogel <kfogel@red-bean.com>,  emacs-devel@gnu.org
> Date: Fri, 02 Dec 2016 14:28:25 -0800
> 
> We've all been contributing to Emacs for many years. I think we have a
> pretty good grasp of the issues and potential problems.  I understand
> that you think there's an existential risk that goes into this change,
> but I think you're mistaken, and a lot of other people agree. We all
> have access to the same information; it's unlikely (but not impossible)
> that you're the only one who can see certain potential problems.

Assessment of trends and their impact on the future, with all the
uncertainties involved, requires more than just processing the
immediately available information.

> Please give this change the benefit of the doubt.

I thought I was already doing that.  Isn't that why the code will be
on a branch soon?

> Even if your worst-case scenario comes to pass, the worst outcome
> outcome will just be more fiddly maintenance.

We already have such "fiddly maintenance" situation in several areas.
Font selection, font back-ends, character composition, complex script
shaping, and (to some extent) the interaction with X and its window
managers -- all of these are areas where bugs are left for months and
years without being fixed, and no progress is made to adapt Emacs to
the new technologies out there.

IOW, "the worst outcome" is already here.  I'm desperately trying not
to make it worse.

> I think the pdumper code is pretty clean; I'm open to suggestions
> for making it moreso.

I will definitely try to keep this in mind when reviewing the code.
However, code cleanness is not the main problem, based on the above
examples.  There's nothing unclean in the code in any of the areas I
mentioned.  The problems are elsewhere.

> If there's no one left who can understand pdumper, there's no one
> left who can understand the GC either.

GC is stable for many years, and "just works".  Bugs only happen there
when people make changes in GC code, for benefits that are not
essential.  If there's no one on board who understands GC, we can
simply disallow any changes in that area.

By contrast, the portable dumper is new code.  No matter how simple
and clean, it will take some time until it becomes mature enough to
raise to the current status of GC.  Until that happens, we cannot
afford being in the same situation as with GC.

> You won't have to touch this code.

I most probably will, and I have no reason to be afraid of that.  But
this isn't about me.

Re: Preview: portable dumper

[Daniel Colascione]

On Sat, Dec 03 2016, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Cc: Karl Fogel <kfogel@red-bean.com>,  emacs-devel@gnu.org
>> Date: Fri, 02 Dec 2016 14:28:25 -0800
>> 
>> We've all been contributing to Emacs for many years. I think we have a
>> pretty good grasp of the issues and potential problems.  I understand
>> that you think there's an existential risk that goes into this change,
>> but I think you're mistaken, and a lot of other people agree. We all
>> have access to the same information; it's unlikely (but not impossible)
>> that you're the only one who can see certain potential problems.
>
> Assessment of trends and their impact on the future, with all the
> uncertainties involved, requires more than just processing the
> immediately available information.

What information then?

We are both rational creatures.  If we disagree, it's either because one
of us made a leap of logic that other has not or because we're starting
with different postulates. Can we please get at the root of
our disagreement?

What evidence would convince you that you were incorrect? On my part, I
might be swayed by examples of other free software projects that have
failed due to the inclusion of C code. I cannot think of any.

It is very frustrating to be asked to believe radical assertions made
without any evidence; that's what happened on the thread-safe malloc
thread, and it was very unproductive.

>> Please give this change the benefit of the doubt.
>
> I thought I was already doing that.  Isn't that why the code will be
> on a branch soon?

The branch will be short-lived. I will want to merge this code
soon. It's at that time that I will most appreciate being given the
benefit of the doubt.

>> Even if your worst-case scenario comes to pass, the worst outcome
>> outcome will just be more fiddly maintenance.
>
> We already have such "fiddly maintenance" situation in several areas.
> Font selection, font back-ends, character composition, complex script
> shaping, and (to some extent) the interaction with X and its window
> managers -- all of these are areas where bugs are left for months and
> years without being fixed, and no progress is made to adapt Emacs to
> the new technologies out there.

The reason that these bugs aren't getting fixed is that they're not
bothering people enough to provoke fixes. I know that I'm much more
inclined to fix bugs when they bother me personally. It's misleading to
cite unfixed minor bugs as evidence that nobody knows how to fix the
bugs.

Placing restrictions on the people who do know how to fix these bugs is
a good way to ensure that we'll either leave them unfixed (thinking it
not worth the trouble to justify the addition of C code) or just keep
the fixes local (which I confess to doing, for this reason).

You know why I haven't worked on the GTK+-only backend I proposed? I
don't *need* to yet. The X11 one works fine, with fixes here and
there. One day, it won't work fine anymore, and either I or someone else
will make it work fine. This situation is not evidence that we're
forgetting how to program GUIs. It means that we did an adequate job.

As for "adapting Emacs to the new technologies out there": that is
precisely what the portable dumper does! I sent a patch that adapts
Emacs to modern technologies and you're complaining that nobody is doing
that. What do you expect me to think?

> IOW, "the worst outcome" is already here.  I'm desperately trying not
> to make it worse.

Emacs runs fine and has many users. It's influential in the community. I
have no idea how many users we have, but I find Emacs users in every
group of developers I visit. emacs-devel is a high traffic mailing
list. We make the news on every major release. If this situation is "the
worst outcome", the worst outcome is pretty comfortable.

Bugs in the sections of code you mention get fixed when you allow them
to get fixed --- Stefan had to try very hard recently to get the
before-change-functions bug fixed, and that was a simple fix for an
egregious problem.

>> I think the pdumper code is pretty clean; I'm open to suggestions
>> for making it moreso.
>
> I will definitely try to keep this in mind when reviewing the code.
> However, code cleanness is not the main problem, based on the above
> examples.  There's nothing unclean in the code in any of the areas I
> mentioned.  The problems are elsewhere.

Yes. I don't think anyone is concerned about code quality. There is
literally nothing I know how to do to this code to make you feel better
about merging it. I feel like the code's very existence is what bothers
you, and I can do nothing about that except give up, which I won't
do. If my assessment is incorrect, and there is something I can do to
the code to make you feel better about it, please let me know.

>> If there's no one left who can understand pdumper, there's no one
>> left who can understand the GC either.
>
> GC is stable for many years, and "just works".  Bugs only happen there
> when people make changes in GC code, for benefits that are not
> essential.  If there's no one on board who understands GC, we can
> simply disallow any changes in that area.

The surest way to ensure that nobody understands a piece of code in a
free software project is to put up a big sign that says "OFF LIMITS! DO
NOT TOUCH!". People learn by doing. You need to let people experiment.
Moving fast (and breaking things occasionally) beats OWNERS. Emacs is
not pacemaker firmware. It's not flying an airplane. A bug introduced is
not the end of the world. We can bisect. You can't simultaneously lament
that the flow of new core developers is a trickle and keep the spigot
turned off.

Your position on this matter has never made any sense to me.

> By contrast, the portable dumper is new code.  No matter how simple
> and clean, it will take some time until it becomes mature enough to
> raise to the current status of GC.  Until that happens, we cannot
> afford being in the same situation as with GC.

Half the people on emacs-devel understand the GC. The other half could
understand it with a few days of study. The last time we had a major GC
bug (the purespace symbol reference thing), we quickly had two
independent patches for fixing it.  (Mine lost, but that's history.)
Both of those path authors are on this thread. This outcome doesn't
suggest that GC is in danger of becoming abandonware. If pdumper ends up
in the same position GC is in today, that's fine.

Speaking of future-proofing Emacs: I'd appreciate fixing the
conservative GC problem I highlighted a few days ago before it becomes a
real bug. Maybe I misread the messages, but my sense was that you were
firmly against that change as well, and I never figured out why.

>> You won't have to touch this code.
>
> I most probably will, and I have no reason to be afraid of that.  But
> this isn't about me.

You are the only one objecting. Perhaps, generally speaking, your
opinion carries greater weight, but it should not carry infinite
weight. Please defer to the judgment of the rest of the community. This
change is not an existential threat. We are not the enemy.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Cc: kfogel@red-bean.com,  emacs-devel@gnu.org
> Date: Sat, 03 Dec 2016 01:34:42 -0800
> 
> > Assessment of trends and their impact on the future, with all the
> > uncertainties involved, requires more than just processing the
> > immediately available information.
> 
> What information then?
> 
> We are both rational creatures.  If we disagree, it's either because one
> of us made a leap of logic that other has not or because we're starting
> with different postulates. Can we please get at the root of
> our disagreement?

Didn't you ever participate in a meeting discussing a complex issue,
and saw different people suggest different assessments and different
courses of actions, although they all have the same information about
the issue at their disposal?  I'm sure you did.  Why does it happen?
For any number of reasons: different age and experience, different
weight assigned to the same factors involved, different intuition,
different abilities to analyze a complex problem and discern the
important from the unimportant, etc. etc.

It's not about being rational.  Complex problems like that cannot be
solved by deterministic algorithms, and the input data is almost
always incomplete.

> What evidence would convince you that you were incorrect?

That's easy: a significant increase in the number of active developers
working on the C level.

> On my part, I might be swayed by examples of other free software
> projects that have failed due to the inclusion of C code.

There's nothing wrong with including C code in Emacs.  I'm not against
it.  I'm doing that myself all the time, and others do it as well,
with my full blessing.  This isn't about adding C code, as I already
said many times.  This is about adding significant new components
implemented in C that solve problems which could be solved in Lisp
instead.

> It is very frustrating to be asked to believe radical assertions made
> without any evidence

I gave evidence, you just don't agree with my conclusions from that
evidence.

> >> Please give this change the benefit of the doubt.
> >
> > I thought I was already doing that.  Isn't that why the code will be
> > on a branch soon?
> 
> The branch will be short-lived. I will want to merge this code
> soon. It's at that time that I will most appreciate being given the
> benefit of the doubt.

Ah, so you still think the suggestion to put the code on a branch was
a political trick, rather than a step towards its potential admission?
I guess _I_ am the one who is not getting any benefit of the doubt
here.

> > We already have such "fiddly maintenance" situation in several areas.
> > Font selection, font back-ends, character composition, complex script
> > shaping, and (to some extent) the interaction with X and its window
> > managers -- all of these are areas where bugs are left for months and
> > years without being fixed, and no progress is made to adapt Emacs to
> > the new technologies out there.
> 
> The reason that these bugs aren't getting fixed is that they're not
> bothering people enough to provoke fixes. I know that I'm much more
> inclined to fix bugs when they bother me personally.

So you think that bugs should only be fixed by those who are
personally bothered by them in their routine Emacs usage?  That would
leave most of the bugs unfixed.  It would also mean that bugs reported
by users who are not programmers, or who cannot afford investing
enough time to learn the relevant parts of the Emacs code, will never
be fixed; if such users are bothered enough by these bugs, their only
choice is simply to stop using Emacs.

As things are, and as they should be, most bugs are fixed by the core
developers, because we want to fix as many bugs as we can, even if we
personally aren't bothered at all by them.  The problem that bothers
me happens when the core developers lack knowledge for fixing a bug
and cannot afford investing the time to study the relevant code and
the underlying issues/features, to acquire that knowledge.

> It's misleading to cite unfixed minor bugs as evidence that nobody
> knows how to fix the bugs.

"Minor bugs"?  I'm talking about problems that slow down display of
certain fonts to a crawl, or produce badly formatted display, or cause
crashes in GC or in malloc.  These bugs are definitely not minor.

> Placing restrictions on the people who do know how to fix these bugs is
> a good way to ensure that we'll either leave them unfixed (thinking it
> not worth the trouble to justify the addition of C code) or just keep
> the fixes local (which I confess to doing, for this reason).

Code review is not just about obviously wrong code.  It is also about
other less obvious factors, such as how well the submission fits with
the overall design and implementation.  You will find these issues
brought up in any project, and some submissions rejected on those
grounds; Emacs is no exception.  Nor did I invent that here.

> There is literally nothing I know how to do to this code to make you
> feel better about merging it. I feel like the code's very existence
> is what bothers you

That's profoundly false, not to say unfair.  I've stated exactly what
bothers me: I would like to see this problem fixed differently, if
possible and feasible.  I also explained what would be my preferred
solution.  After all this, what you say above is a strawman at best.

> If my assessment is incorrect, and there is something I can do to
> the code to make you feel better about it, please let me know.

It's not the code that bothers me, it's the pressure to accept it
without exploring the possibility of solving the same problem by the
preferred method.  What's the rush?  You said you are going to be
around for maintaining the code, so why can't you wait for us to try
the lread speedup first?  xwidgets was on a branch for months after it
was declared ready; why this one is different?

> >> If there's no one left who can understand pdumper, there's no one
> >> left who can understand the GC either.
> >
> > GC is stable for many years, and "just works".  Bugs only happen there
> > when people make changes in GC code, for benefits that are not
> > essential.  If there's no one on board who understands GC, we can
> > simply disallow any changes in that area.
> 
> The surest way to ensure that nobody understands a piece of code in a
> free software project is to put up a big sign that says "OFF LIMITS! DO
> NOT TOUCH!". People learn by doing.

People learn by doing, but many don't stay long enough to learn
enough.  They disappear with their partial knowledge, and we are left
to deal with the bugs.  If those people were to stay here longer, or
if we had someone on board who'd be able to fix the fallout quickly,
then sure, we could afford to be nonchalant about changes in those
core areas.  As things are, we need to be more cautious.  It's a fine
balance, for sure, which is why changes in those areas are indeed
accepted, as you can clearly see from the logs.  We didn't mark those
areas OFF LIMITS.  You were describing a desperate situation, which
didn't happen yet, so I said that if push comes to shove, we can apply
desperate measures.  We are not there yet, and I hope we never will
be.

> You need to let people experiment. Emacs is not pacemaker
> firmware. It's not flying an airplane.

It isn't a student project, either.  It is a veteran project expected
to be stable, even on the master branch.  People do important things
with it, and their edits are sometimes precious.  There are
distributions out there providing snapshots of master, and people use
those snapshots in their routine work.  We cannot afford having master
broken except for very short times.

> A bug introduced is not the end of the world. We can bisect.

This only works in simple cases.  It doesn't work when the offending
commit was fixing a problem, which we don't want to get back.  It also
doesn't work when removal of the offending commit invalidates a large
series of changes which we find unjustified to back out.

And then there are bugs whose reproduction is itself a problem, for
which bisection is infeasible.

> You can't simultaneously lament that the flow of new core developers
> is a trickle and keep the spigot turned off.

The spigot is not off.  There are a few cases, hopefully rare, where
changes are rejected for these reasons.  But they are a minority, and
your presentation of them as a majority is a significant exaggeration,
probably to make a point.

The truth is that this is a fine balance, and I think we are generally
doing well with walking that thin line.

> Your position on this matter has never made any sense to me.

And yet people want me to continue doing this job.  Which makes
absolutely no sense to me.  Why would they want that if they don't
agree with how I do my job?

Would someone please try answering this question?

> Half the people on emacs-devel understand the GC. The other half could
> understand it with a few days of study.

Really?  Then why did the last 2 serious bugs there wait until yours
truly debugged them and proposed a fix?

> The last time we had a major GC bug (the purespace symbol reference
> thing), we quickly had two independent patches for fixing it.  (Mine
> lost, but that's history.)

No, the last two major bugs that were detected by GC were bug#24478
and bug#24640.  Their root causes were not in GC, but debugging them
required familiarity with GC and related functionality.  May I suggest
that you familiarize yourself with how those bugs were analyzed and
how the fix for them was found?  Because this kind of problems, and
our ability to fix them in the future, is what bothers me.

> Speaking of future-proofing Emacs: I'd appreciate fixing the
> conservative GC problem I highlighted a few days ago before it becomes a
> real bug. Maybe I misread the messages, but my sense was that you were
> firmly against that change as well, and I never figured out why.

You never presented any code, so the discussion was purely
theoretical.  Who knows, perhaps I misunderstood what you were
suggesting.  It happened to me as well: 2 months ago I described the
idea very similar to your portable dumper, and Paul Eggert rejected it
in very firm terms.  Now he supports your implementation of the same
ideas.  Go figure.

> >> You won't have to touch this code.
> >
> > I most probably will, and I have no reason to be afraid of that.  But
> > this isn't about me.
> 
> You are the only one objecting.

Whose job is it to object?

> Perhaps, generally speaking, your opinion carries greater weight,
> but it should not carry infinite weight.

How much weight does it have?  How do you determine whether that
weight is enough to override the opinions of N others?

> Please defer to the judgment of the rest of the community.

Isn't that why I agreed to put the code on a branch?

> This change is not an existential threat.

Not it alone, no.  But it's in that direction which I'd like to avoid
as much as possible.  Add enough such changes, and they might be such
a serious threat.

> We are not the enemy.

Neither am I.

Re: Preview: portable dumper

[Alan Mackenzie]

Hello, Eli.

I haven't really been following this thread, but one tangential thing in
it jumped out at me:

On Sat, Dec 03, 2016 at 02:47:07PM +0200, Eli Zaretskii wrote:
> > From: Daniel Colascione <dancol@dancol.org>
> > Cc: kfogel@red-bean.com,  emacs-devel@gnu.org
> > Date: Sat, 03 Dec 2016 01:34:42 -0800

[ .... ]

> > What evidence would convince you that you were incorrect?

> That's easy: a significant increase in the number of active developers
> working on the C level.

I have made quite a lot of changes at the C level, but I'm not really a
"core C level" developer.

However, it feels that an unusually high proportion of C level changes I
have hacked or proposed have been rejected.  ("Unusual" when compared to
lisp level changes.)  Examples include:

(i) Changing the method of syntax.c scanning backwards over comments.  My
changes found their way into branch comment-cache in 2016-03.  Despite
this change having been extensively discussed in emacs-devel, and sort of
"approved", the final patch was never considered on its merits.  The
ostensible reason was that it used a cache which wasn't the syntax-ppss
cache.

(ii) Around 2015-11-17, I proposed a patch to fix bug #21869 and bug
#21333, with top line of the commit message being "Invoke
window-size-change-functions after changing echo area height.".  The
problem here was that window-size-change-functions was sometimes being
called twice.  You rejected my patch because you were "not keen" on
changing the order of calls in the display engine because we "didn't
fully understand what was going on".  Again, I don't think this proposed
patch was really considered on its merits.

(iii) Earlier this year, we were having problems because some primitives
were not calling before-change-functions and after-change-functions the
way we might wish.  My offer to analyse the code and amend it so that all
primitives would call b-c-f and a-c-f predictably was declined, the
proviso being (if I remember correctly) "unless somebody writes a solid
suite of unit tests".  At the time of this rejection, I'd already
invested some time on the analysis.

It would be unfair not to mention that a lot of changes I've proposed at
the C level, and hacked, have been accepted.  Yet, at the same time, when
I now consider tackling problems which need fixing at the C level, I feel
a doubt in my mind that any proposed fix would be properly considered,
and I weigh up this possibility before committing myself to spending any
time on it.

In short, I feel discouraged from working at the C level because of the
above.  I might not be the only developer who feels this.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Preview: portable dumper

[Eli Zaretskii]

> Date: Sat, 3 Dec 2016 14:36:04 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: kfogel@red-bean.com, Daniel Colascione <dancol@dancol.org>,
> 	emacs-devel@gnu.org
> 
> However, it feels that an unusually high proportion of C level changes I
> have hacked or proposed have been rejected.  ("Unusual" when compared to
> lisp level changes.)

Can you estimate that proportion?  You cited 3 examples, and then said
that "a lot of changes have been accepted".  3 out of many is not a
high proportion.

> (i) Changing the method of syntax.c scanning backwards over comments.  My
> changes found their way into branch comment-cache in 2016-03.  Despite
> this change having been extensively discussed in emacs-devel, and sort of
> "approved", the final patch was never considered on its merits.  The
> ostensible reason was that it used a cache which wasn't the syntax-ppss
> cache.

I don't think I participated in that discussion or reviewed that
patch, so I cannot say anything about that.

> (ii) Around 2015-11-17, I proposed a patch to fix bug #21869 and bug
> #21333, with top line of the commit message being "Invoke
> window-size-change-functions after changing echo area height.".  The
> problem here was that window-size-change-functions was sometimes being
> called twice.  You rejected my patch because you were "not keen" on
> changing the order of calls in the display engine because we "didn't
> fully understand what was going on".  Again, I don't think this proposed
> patch was really considered on its merits.

Please see

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21869#23

It indicates that we already had a patch for those problems, which was
already tested quite a lot by that time.  I think it's only natural to
prefer a well tested and discussed patch to a new one trying to fix
the same problem.  Reading this message further down:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21869#37

I see that you agreed that the problem was fixed by the patch we had,
which allowed me to close the bug.

> (iii) Earlier this year, we were having problems because some primitives
> were not calling before-change-functions and after-change-functions the
> way we might wish.  My offer to analyse the code and amend it so that all
> primitives would call b-c-f and a-c-f predictably was declined, the
> proviso being (if I remember correctly) "unless somebody writes a solid
> suite of unit tests".  At the time of this rejection, I'd already
> invested some time on the analysis.

That's not exactly my recollection.  The analysis was not rejected, I
simply already did it when you proposed that, so it wasn't necessary.

> In short, I feel discouraged from working at the C level because of the
> above.

Please don't be discouraged.  There's no policy of preventing changes
on the C level.  However, for some changes which affect important
functions, I think it's prudent to require a reasonable coverage by
tests, so that we know we didn't break anything.

Re: Preview: portable dumper

[Alan Mackenzie]

Hello, Eli.

On Sat, Dec 03, 2016 at 05:11:33PM +0200, Eli Zaretskii wrote:
> > Date: Sat, 3 Dec 2016 14:36:04 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: kfogel@red-bean.com, Daniel Colascione <dancol@dancol.org>,
> > 	emacs-devel@gnu.org

> > However, it feels that an unusually high proportion of C level changes I
> > have hacked or proposed have been rejected.  ("Unusual" when compared to
> > lisp level changes.)

> Can you estimate that proportion?  You cited 3 examples, and then said
> that "a lot of changes have been accepted".  3 out of many is not a
> high proportion.

I can't really estimate it accurately (or it would take far too much
time to do so).  It could be as high as 1 in 3.  It could be as low as 1
in 10.

> > (i) Changing the method of syntax.c scanning backwards over comments.  My
> > changes found their way into branch comment-cache in 2016-03.  Despite
> > this change having been extensively discussed in emacs-devel, and sort of
> > "approved", the final patch was never considered on its merits.  The
> > ostensible reason was that it used a cache which wasn't the syntax-ppss
> > cache.

> I don't think I participated in that discussion or reviewed that
> patch, so I cannot say anything about that.

I've found that thread, and I misremembered it.  The preceding
discussion was not extensive.  It was in the thread with Subject:
"bug#22884: 25.0.92; C/l mode editing takes waaaayy too long" and
essentially consisted of just three emails:
  (i) Alan -> Eli: Thu, 3 Mar 2016 23:18:23 +0000
  (ii) Eli -> Alan: Fri, 04 Mar 2016 10:32:56 +0200
  (iii) Alan -> Eli: Fri, 4 Mar 2016 09:37:07 +0000
.  In (i), I proposed using text properties to cache the string/comment
state of positions, and asked you whether that might overwhelm the text
property mechanism.  In (ii) you replied that it shouldn't.  In (iii) I
announced my intention to hack it, which I then did.

I don't remember you reviewing the patch, or expressing any further
views on it afterwards, so we agree on that.  The discussion after I
submitted my patch happened in the threads Subject: "Re: [Emacs-diffs]
comment-cache 223d16f 2/3: Apply `comment-depth' text properties when
calling `back_comment'." and Subject: "Permission requested to merge
branch comment-cache into master.".  This discussion was long and (to
me) unexpectedly aggressive, and didn't concern itself with the
correctness of the patch.  It seemed intended to prevent the adoption of
my patch by spreading FUD.  If this was indeed the intention, it was
entirely successful.

> > (ii) Around 2015-11-17, I proposed a patch to fix bug #21869 and bug
> > #21333, with top line of the commit message being "Invoke
> > window-size-change-functions after changing echo area height.".  The
> > problem here was that window-size-change-functions was sometimes being
> > called twice.  You rejected my patch because you were "not keen" on
> > changing the order of calls in the display engine because we "didn't
> > fully understand what was going on".  Again, I don't think this proposed
> > patch was really considered on its merits.

> Please see

>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21869#23

> It indicates that we already had a patch for those problems, which was
> already tested quite a lot by that time.  I think it's only natural to
> prefer a well tested and discussed patch to a new one trying to fix
> the same problem.  Reading this message further down:

>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21869#37

> I see that you agreed that the problem was fixed by the patch we had,
> which allowed me to close the bug.

I've read through that debbugs bug archive.  It was a long time ago.  I
remember feeling discouraged that my patch was not considered.  I think
I felt that my patch fixed the fundamental problem in xdisp.c, whereas
the patch actually applied was more of a workaround.  Or something like
that.  But I didn't push the point at the time.

> > (iii) Earlier this year, we were having problems because some primitives
> > were not calling before-change-functions and after-change-functions the
> > way we might wish.  My offer to analyse the code and amend it so that all
> > primitives would call b-c-f and a-c-f predictably was declined, the
> > proviso being (if I remember correctly) "unless somebody writes a solid
> > suite of unit tests".  At the time of this rejection, I'd already
> > invested some time on the analysis.

> That's not exactly my recollection.  The analysis was not rejected, I
> simply already did it when you proposed that, so it wasn't necessary.

OK.  But the notion of acutally fixing the primitives so that they would
each call b-c-f and a-c-f was strongly discouraged, if I remember
correctly.

> > In short, I feel discouraged from working at the C level because of the
> > above.

> Please don't be discouraged.  There's no policy of preventing changes
> on the C level.  However, for some changes which affect important
> functions, I think it's prudent to require a reasonable coverage by
> tests, so that we know we didn't break anything.

Perhaps it is that last point (that C functions are more likely to be
critical to Emacs's working than lisp functions, and therefore the
testing is more important) that is the thing.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Preview: portable dumper

[Dmitry Gutov]

On 04.12.2016 14:20, Alan Mackenzie wrote:

> This discussion was long and (to
> me) unexpectedly aggressive, and didn't concern itself with the
> correctness of the patch.

It really did. When a patch disregards a whole class of current use 
cases, and is likely to make them worse, that's very much a correctness 
issue.

> It seemed intended to prevent the adoption of
> my patch by spreading FUD.  If this was indeed the intention, it was
> entirely successful.

Thanks for the insult, Alan.

Re: Preview: portable dumper

[Eli Zaretskii]

> Date: Sun, 4 Dec 2016 12:20:33 +0000
> Cc: kfogel@red-bean.com, dancol@dancol.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> I can't really estimate it accurately (or it would take far too much
> time to do so).  It could be as high as 1 in 3.  It could be as low as 1
> in 10.

I hope it's the latter.

> > > (i) Changing the method of syntax.c scanning backwards over comments.  My
> > > changes found their way into branch comment-cache in 2016-03.  Despite
> > > this change having been extensively discussed in emacs-devel, and sort of
> > > "approved", the final patch was never considered on its merits.  The
> > > ostensible reason was that it used a cache which wasn't the syntax-ppss
> > > cache.
> 
> > I don't think I participated in that discussion or reviewed that
> > patch, so I cannot say anything about that.
> 
> I've found that thread, and I misremembered it.  The preceding
> discussion was not extensive.  It was in the thread with Subject:
> "bug#22884: 25.0.92; C/l mode editing takes waaaayy too long" and
> essentially consisted of just three emails:
>   (i) Alan -> Eli: Thu, 3 Mar 2016 23:18:23 +0000
>   (ii) Eli -> Alan: Fri, 04 Mar 2016 10:32:56 +0200
>   (iii) Alan -> Eli: Fri, 4 Mar 2016 09:37:07 +0000
> .  In (i), I proposed using text properties to cache the string/comment
> state of positions, and asked you whether that might overwhelm the text
> property mechanism.  In (ii) you replied that it shouldn't.  In (iii) I
> announced my intention to hack it, which I then did.
> 
> I don't remember you reviewing the patch, or expressing any further
> views on it afterwards, so we agree on that.  The discussion after I
> submitted my patch happened in the threads Subject: "Re: [Emacs-diffs]
> comment-cache 223d16f 2/3: Apply `comment-depth' text properties when
> calling `back_comment'." and Subject: "Permission requested to merge
> branch comment-cache into master.".  This discussion was long and (to
> me) unexpectedly aggressive, and didn't concern itself with the
> correctness of the patch.

I've re-read those threads and found my participation in them to be
almost non-existent.  In the first one it was limited to presenting
benchmarks of the performance that was discussed.  In the second one I
barely said anything at all, and when I did, it was to encourage you
to install your patch on the release branch rather than master.

> >   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21869#23
> 
> > It indicates that we already had a patch for those problems, which was
> > already tested quite a lot by that time.  I think it's only natural to
> > prefer a well tested and discussed patch to a new one trying to fix
> > the same problem.  Reading this message further down:
> 
> >   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21869#37
> 
> > I see that you agreed that the problem was fixed by the patch we had,
> > which allowed me to close the bug.
> 
> I've read through that debbugs bug archive.  It was a long time ago.  I
> remember feeling discouraged that my patch was not considered.  I think
> I felt that my patch fixed the fundamental problem in xdisp.c, whereas
> the patch actually applied was more of a workaround.  Or something like
> that.  But I didn't push the point at the time.

Once again, a patch that was discussed at length and tested by several
people was committed that fixed the problem.  I can understand your
frustration, but we really cannot install more than a single patch for
each problem.

> > > (iii) Earlier this year, we were having problems because some primitives
> > > were not calling before-change-functions and after-change-functions the
> > > way we might wish.  My offer to analyse the code and amend it so that all
> > > primitives would call b-c-f and a-c-f predictably was declined, the
> > > proviso being (if I remember correctly) "unless somebody writes a solid
> > > suite of unit tests".  At the time of this rejection, I'd already
> > > invested some time on the analysis.
> 
> > That's not exactly my recollection.  The analysis was not rejected, I
> > simply already did it when you proposed that, so it wasn't necessary.
> 
> OK.  But the notion of acutally fixing the primitives so that they would
> each call b-c-f and a-c-f was strongly discouraged, if I remember
> correctly.

Yes, because those touched very fundamental functionalities in
manipulating buffer text, which I didn't want to destabilize without
having a test suite with good coverage.

So in sum, I feel that these incidents don't have a lot in common with
the issues discussed here.

> > > In short, I feel discouraged from working at the C level because of the
> > > above.
> 
> > Please don't be discouraged.  There's no policy of preventing changes
> > on the C level.  However, for some changes which affect important
> > functions, I think it's prudent to require a reasonable coverage by
> > tests, so that we know we didn't break anything.
> 
> Perhaps it is that last point (that C functions are more likely to be
> critical to Emacs's working than lisp functions, and therefore the
> testing is more important) that is the thing.

I think treating code in C as more delicate is prudent: changes there
will almost always have wider implications than changes in Lisp, and
the code is frequently harder to understand, so better testing is IMO
justified.

Re: Preview: portable dumper

[Daniel Colascione]

On Sat, Dec 03 2016, Alan Mackenzie wrote:
> Hello, Eli.
>
> I haven't really been following this thread, but one tangential thing in
> it jumped out at me:
>
> On Sat, Dec 03, 2016 at 02:47:07PM +0200, Eli Zaretskii wrote:
>> > From: Daniel Colascione <dancol@dancol.org>
>> > Cc: kfogel@red-bean.com,  emacs-devel@gnu.org
>> > Date: Sat, 03 Dec 2016 01:34:42 -0800
>
> [ .... ]
>
>> > What evidence would convince you that you were incorrect?
>
>> That's easy: a significant increase in the number of active developers
>> working on the C level.
>
> I have made quite a lot of changes at the C level, but I'm not really a
> "core C level" developer.
>
> However, it feels that an unusually high proportion of C level changes I
> have hacked or proposed have been rejected.  ("Unusual" when compared to
> lisp level changes.)  Examples include:
>
> (i) Changing the method of syntax.c scanning backwards over comments.  My
> changes found their way into branch comment-cache in 2016-03.  Despite
> this change having been extensively discussed in emacs-devel, and sort of
> "approved", the final patch was never considered on its merits.  The
> ostensible reason was that it used a cache which wasn't the syntax-ppss
> cache.

I'd forgotten about that patch. It's further evidence of the trouble
lurking here. How about landing that code now?

Re: Preview: portable dumper

[Dmitry Gutov]

On 03.12.2016 19:36, Daniel Colascione wrote:

> I'd forgotten about that patch. It's further evidence of the trouble
> lurking here. How about landing that code now?

How about not being rash and reading up on the conversations that ended 
in that patch not being applied?

Re: Preview: portable dumper

[Stefan Monnier]

>> I'd forgotten about that patch. It's further evidence of the trouble
>> lurking here. How about landing that code now?
> How about not being rash and reading up on the conversations that ended in
> that patch not being applied?

Or applying the patch I suggested instead (much simpler and leverages
the syntax-ppss cache)...


        Stefan

Re: Preview: portable dumper

[Daniel Colascione]

On 12/03/2016 01:09 PM, Stefan Monnier wrote:
>>> I'd forgotten about that patch. It's further evidence of the trouble
>>> lurking here. How about landing that code now?
>> How about not being rash and reading up on the conversations that ended in
>> that patch not being applied?
>
> Or applying the patch I suggested instead (much simpler and leverages
> the syntax-ppss cache)...

I'll have to go back and reread the thread. I do recall Alan having a 
good point about syntax-ppss being incorrect for certain kinds of 
two-character comment starters.

Re: Preview: portable dumper

[Stefan Monnier]

> I'll have to go back and reread the thread. I do recall Alan having a good
> point about syntax-ppss being incorrect for certain kinds of two-character
> comment starters.

That was a problem with parse-partial-sexp, and AFAIK Alan fixed
it recently.


        Stefan

Re: Preview: portable dumper

[Alan Mackenzie]

On Sat, Dec 03, 2016 at 04:09:24PM -0500, Stefan Monnier wrote:
> >> I'd forgotten about that patch. It's further evidence of the trouble
> >> lurking here. How about landing that code now?
> > How about not being rash and reading up on the conversations that ended in
> > that patch not being applied?

> Or applying the patch I suggested instead (much simpler and leverages
> the syntax-ppss cache)...

Your patch was indeed simpler.  Indeed, it was so simple as actually not
to work in general, as well as being ~2 orders of magnitude slower than
the current (forward-comment -1).  syntax-ppss is broken (bug #22983)
and has been for many years, so to base a critical primitive on it seems
unwise.  And one or two other things, too.

Other than that, yes we could apply that patch.

BTW, I've been looking at bug #22983 the last few days, since it seems
nobody else is going to bother fixing it.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Preview: portable dumper

[Dmitry Gutov]

On 04.12.2016 14:34, Alan Mackenzie wrote:

>> Or applying the patch I suggested instead (much simpler and leverages
>> the syntax-ppss cache)...
>
> Your patch was indeed simpler.  Indeed, it was so simple as actually not
> to work in general, as well as being ~2 orders of magnitude slower than
> the current (forward-comment -1).

Which has never been demonstrated to be a problem in practice.

> BTW, I've been looking at bug #22983 the last few days, since it seems
> nobody else is going to bother fixing it.

The patch is there: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22983#47

You're welcome to install it.

I'm still waiting for someone to assist Vitalie with the hard-widen effort.

Re: Preview: portable dumper

[Stefan Monnier]

> Your patch was indeed simpler.  Indeed, it was so simple as actually not
> to work in general, as well as being ~2 orders of magnitude slower than
> the current (forward-comment -1).

You make it sound like it barely ever works and it's always 100
times slower.  I think this qualifies as "gross misrepresentation".


        Stefan

Re: Preview: portable dumper

[Alan Mackenzie]

Hello, Stefan.

On Sun, Dec 04, 2016 at 09:08:26AM -0500, Stefan Monnier wrote:
> > Your patch was indeed simpler.  Indeed, it was so simple as actually not
> > to work in general, as well as being ~2 orders of magnitude slower than
> > the current (forward-comment -1).

> You make it sound like it barely ever works and it's always 100
> times slower.  I think this qualifies as "gross misrepresentation".

I don't think it does.

Your patch won't work, for example, if a lisp program sets syntax-table
text properties (which is _always_ done when
before/after-change-functions are disabled).  This is a very important
scenario.

Consider (forward-comment -1) backwards over a 100 character comment.
The current code will do this in 100n units of time, for some value of
n.  In your patch, we need to calculate the parse-partial-sexp state at
the comment, and this is going to involve a parse-partial-sexp moving
forward over, on average, 10,000 characters.  This is going to take
10,000m units of time, for some m.  10,000m is two orders of magnitude
larger than 100n.

Now, I'm not saying your patch couldn't be improved to handle these and
other things.  Firstly, it then wouldn't be a "simple" patch any longer.
And the next thing, why bother, when the comment-cache code already
exists and works?

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Alan, would you like to become an expert on some areas of the display
code?  If you become the main person working on some area, I am sure
you will get your changes in.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Alan Mackenzie]

Hello, Richard.

On Sat, Dec 03, 2016 at 04:31:35PM -0500, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

> Alan, would you like to become an expert on some areas of the display
> code?  If you become the main person working on some area, I am sure
> you will get your changes in.

I'd like to become an expert on the display code, but I don't think I'm
up to it.  That code is just too difficult for me, unless I gave up
doing other things to be able to spend the time on it.  It's highly
optimised (it has to be), and is thus very long.  I don't think it's
badly written - it's just got a very difficult job to do.

> -- 
> Dr Richard Stallman

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Preview: portable dumper

[Paul Eggert]

Eli Zaretskii wrote:
> the last two major bugs that were detected by GC were bug#24478 and bug#24640.

I don't see how this is counting. I don't see the relationship between bug#24478 
and GC. Bug#24640 was a crash but the bug report was not marked as serious.

If we're merely counting GC-related things that broke Emacs, a 
more-recently-fixed bug related to GC was not reported to bug-gnu-emacs at all; 
it's something I fixed two days ago in the release branch, which fixed a broken 
build with the Oracle Developer Studio C compiler.

On the other hand, if we're talking about truly major bugs detected by GC, then 
Emacs currently doesn't work on FreeBSD 11 arm64 due to GC-related problems that 
Daniel's patches would fix. This is Bug#24892, marked as serious, and it pretty 
clearly outranks all the other bugs mentioned in this thread.

Re: Preview: portable dumper

[Eli Zaretskii]

> Cc: kfogel@red-bean.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 3 Dec 2016 09:41:43 -0800
> 
> Eli Zaretskii wrote:
> > the last two major bugs that were detected by GC were bug#24478 and bug#24640.
> 
> I don't see how this is counting. I don't see the relationship between bug#24478 
> and GC. Bug#24640 was a crash but the bug report was not marked as serious.

Crashes in GC are always serious, because you can never know what
caused them.  In that case, it turned out that reading objects with
the #n=object form was broken, quite a serious problem.

> If we're merely counting GC-related things that broke Emacs, a 
> more-recently-fixed bug related to GC was not reported to bug-gnu-emacs at all; 
> it's something I fixed two days ago in the release branch, which fixed a broken 
> build with the Oracle Developer Studio C compiler.

Indeed, problems that are revealed in GC are not very rare.

> On the other hand, if we're talking about truly major bugs detected by GC, then 
> Emacs currently doesn't work on FreeBSD 11 arm64 due to GC-related problems that 
> Daniel's patches would fix. This is Bug#24892, marked as serious, and it pretty 
> clearly outranks all the other bugs mentioned in this thread.

There's more than one problem, and no single solution can fix them
all.

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So you think that bugs should only be fixed by those who are
  > personally bothered by them in their routine Emacs usage?  That would
  > leave most of the bugs unfixed.  It would also mean that bugs reported
  > by users who are not programmers, or who cannot afford investing
  > enough time to learn the relevant parts of the Emacs code, will never
  > be fixed; if such users are bothered enough by these bugs, their only
  > choice is simply to stop using Emacs.

The measure of real commitment to development of Emacs (or any free program)
is willingness to fix bugs _because that's important_ and not just
because they bother you.

Daniel, after the portable dumper is done, would you like to learn
some area of the display code where bugs are waiting to be fixed?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: dancol@dancol.org, kfogel@red-bean.com, emacs-devel@gnu.org
> Date: Sat, 03 Dec 2016 16:30:37 -0500
> 
> Daniel, after the portable dumper is done, would you like to learn
> some area of the display code where bugs are waiting to be fixed?

"Display code" might be misleading.  There are no bugs I know of in
display code proper (xdisp.c, dispnew.c) that I know of and are not
fixed.

I'm guessing you meant the problems I mentioned in the area of font
selection, font back-ends, and character composition.  Those are the
problematic areas.

Volunteers are more than welcome to work in these areas, whether by
studying the code via bugs that need to be solved, or by any other
means.

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > "Display code" might be misleading.  There are no bugs I know of in
  > display code proper (xdisp.c, dispnew.c) that I know of and are not
  > fixed.

  > I'm guessing you meant the problems I mentioned in the area of font
  > selection, font back-ends, and character composition.  Those are the
  > problematic areas.

I think of all code for putting text on the screen
as the display code.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Paul Eggert]

Daniel Colascione wrote:
> I'd appreciate fixing the
> conservative GC problem I highlighted a few days ago before it becomes a
> real bug.

Although I'm not as worried about that prospect as you are, I would also 
appreciate a fix for the problem, so long as it doesn't hurt performance in the 
common case where the fix isn't needed. As I understand it, you want the the 
conservative object scanner to count any pointer to somewhere in (or just past 
the end of) an object, as opposed to counting only pointers to the start of the 
object. That would be a reasonable thing to add, perhaps conditionally compiled 
so that it doesn't hurt performance on typical architectures today.

Re: Preview: portable dumper

[Stefan Monnier]

> We already have such "fiddly maintenance" situation in several areas.
> Font selection, font back-ends, character composition, complex script
> shaping, and (to some extent) the interaction with X and its window
> managers -- all of these are areas where bugs are left for months and
> years without being fixed, and no progress is made to adapt Emacs to
> the new technologies out there.

From where I stand I have the impression that we have a respectable
"marketshare" among those programmers interested in programming
languages (for example, in my research area (functional programming and
type systems), most new experimental languages come with support for
Emacs first and support for other editors only show up later).
On the contrary I get the impression that our "markethare" among
programmers interested in "GUI stuff" is very much smaller.

So I don't think the lack of people willing to help us with X and font
stuff implies that there's a lack of people able to hack on things like
the GC (and if you're able to hack on the GC, then that probably carries
over to the pdumper since it's closely related).


        Stefan

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >  most new experimental languages come with support for
  > Emacs first and support for other editors only show up later).
  > On the contrary I get the impression that our "markethare" among
  > programmers interested in "GUI stuff" is very much smaller.

This means we have strong support in one area of development,
and insufficient support in another.  However, we need to maintain
Emacs in both areas.

Would anyone like to help us at in the redisplay and graphics C code?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > GC is stable for many years, and "just works".  Bugs only happen there
  > when people make changes in GC code, for benefits that are not
  > essential.  If there's no one on board who understands GC, we can
  > simply disallow any changes in that area.

The issue you raised -- and it is a real consideration -- is that
future changes in Lisp object formats would likely require changes in
the portable dumper (if we use that), just as they would likely
require changes in GC.

If someday nobody can change GC, that would mean nobody can change
Lisp object formats any more.

It won't take very long before the portable dumper becomes as stable
as GC.  Then it would be the same issue for both.  I don't think that
having Lisp object format changes blocked by GC and the portable
dumper is much worse than having them blocked by GC alone.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> Thanks, but it isn't useful. Why? because, like John, you don't think that
EZ> my opinion about the preferred solution to getting rid of unexec is right.

Maybe I'm not understanding fully: I thought we all agreed that your preferred
solution is the best one, provided it can be implemented. In the meantime, we
have an interim measure available that is better than unexec. Since I haven't
heard anyone suggest that we should stick with the portable dumper forever, I
don't fully understand the strength of your objection.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > "Unenjoyable"?  It's intolerable.  It would mean that on my watch
  > something happened that I believe is detrimental to a project which I
  > took upon myself to improve, and I saw it happening and didn't prevent
  > it.  Can you understand my plight?

You have explained a possible drawback of the portable dumper:
it could make maintenance more difficult some years from now.

No one can doubt that this is a possibility.  People do disagree about
how much of a problem it is likely to be.  Many of us don't expect it
will make a big difference.  You forecast that it could.

But you've got to admit that it is uncertain.  We're all making
guesstimates about probabilities.

So if we install the portable dumper, please don't treat it as a
certain disaster.  It would increase some problems quantitatively, and
reduce other problems quantitatively.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: kfogel@red-bean.com, emacs-devel@gnu.org
> Date: Sat, 03 Dec 2016 16:28:58 -0500
> 
> You have explained a possible drawback of the portable dumper:
> it could make maintenance more difficult some years from now.
> 
> No one can doubt that this is a possibility.  People do disagree about
> how much of a problem it is likely to be.  Many of us don't expect it
> will make a big difference.  You forecast that it could.

Thank you for acknowledging that this is a possibility.  I was under
the impression that my opponents didn't agree.

> But you've got to admit that it is uncertain.  We're all making
> guesstimates about probabilities.

IMO, dangerous outcomes should be taken seriously even if their
probability is low, as long as it's non-zero.  "Taken seriously" means
we should do something to try to avoid the danger.  That is what I'm
trying to do: find a solution for the problem that will avoid the
danger.

> So if we install the portable dumper, please don't treat it as a
> certain disaster.

The "disaster" is not the installation of the dumper.  The disaster I
alluded to would be not to try to find a solution that minimizes the
above danger.  If we have tried our best and failed, then it's not a
disaster, it's life: after all, we can only make the best use of the
cards we've been dealt.

Re: Preview: portable dumper

[Daniel Colascione]

On Sun, Dec 04 2016, Eli Zaretskii wrote:
>> From: Richard Stallman <rms@gnu.org>
>> CC: kfogel@red-bean.com, emacs-devel@gnu.org
>> Date: Sat, 03 Dec 2016 16:28:58 -0500
>> 
>> You have explained a possible drawback of the portable dumper:
>> it could make maintenance more difficult some years from now.
>> 
>> No one can doubt that this is a possibility.  People do disagree about
>> how much of a problem it is likely to be.  Many of us don't expect it
>> will make a big difference.  You forecast that it could.
>
> Thank you for acknowledging that this is a possibility.  I was under
> the impression that my opponents didn't agree.

You're misinterpreting this "agreement". Nobody else agrees that a a
significant maintenance burden is likely.  This is how FUD starts.

>> But you've got to admit that it is uncertain.  We're all making
>> guesstimates about probabilities.
>
> IMO, dangerous outcomes should be taken seriously even if their
> probability is low, as long as it's non-zero.

The people who moved humanity forward were not the ones huddling in
caves afraid of the mammoths outside.

We're talking about software. Anything we do, we can undo.  (And please,
stop suggesting that we have some kind of duty to keep master absolutely
stable because someone, somewhere might make a master snapshot tarball
available for something.  Why do we even have releases?)

> "Taken seriously" means
> we should do something to try to avoid the danger.  That is what I'm
> trying to do: find a solution for the problem that will avoid the
> danger.

You keep saying that you'd prefer a "solution" that "avoids" the
"danger".  There *is* no solution that avoids the "danger" you've
highlighted, which is the mere addition of C code.  (No other free
software project views C code as a "danger" to be avoided at all costs.)

Your preferred approach, a magically faster lread, *will* *also*
*involve* *C* *code*.  It will *also* require knowledge of Lisp layout,
since what lread does is transform text into Lisp objects.  It will
*also* need touchups when we add new lisp objects.

The idea that your preferred fast lread (which, again, does not actually
exist) would be a fine, but that the portable dumper is a "danger" ---
it makes no sense.  Your arguments against the portable dumper apply
also to lread.

Please don't tell me that lread is qualitatively different because it's
existing functionality or something: you're talking about complex,
disruptive additions to lread code, and that these additions would go
into a C file that already exists instead of one that doesn't
is immaterial.

>> So if we install the portable dumper, please don't treat it as a
>> certain disaster.
>
> The "disaster" is not the installation of the dumper.  The disaster I
> alluded to would be not to try to find a solution that minimizes the
> above danger.  If we have tried our best and failed, then it's not a
> disaster, it's life: after all, we can only make the best use of the
> cards we've been dealt.

We've known about unexec problems for years and nobody's come up with a
workable alternative.  Please stop obstructing progress because you wish
it weren't so.  The biggest danger here is that you get your way, block
progress, and that we're stuck with unexec and fragile hacks for many
more years.

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You're misinterpreting this "agreement". Nobody else agrees that a a
  > significant maintenance burden is likely.  This is how FUD starts.

Please don't say it with such a belligerent tone.  We need to work together,
so we need to disagree without hostility.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Daniel Colascione]

On 12/04/2016 03:07 PM, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > You're misinterpreting this "agreement". Nobody else agrees that a a
>   > significant maintenance burden is likely.  This is how FUD starts.
>
> Please don't say it with such a belligerent tone.  We need to work together,
> so we need to disagree without hostility.

Sorry. I should have not been as harsh there.

Re: Preview: portable dumper

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 1817 bytes --]

On Sun, Dec 4, 2016 at 6:12 PM, Daniel Colascione <dancol@dancol.org> wrote:

>
> We're talking about software. Anything we do, we can undo.  (And please,
> stop suggesting that we have some kind of duty to keep master absolutely
> stable because someone, somewhere might make a master snapshot tarball
> available for something.  Why do we even have releases?)
>

Keeping master stable depends on the workflow wanted for Emacs... but in
general, having a stable "master" is an accepted practice for any project
of a reasonable size.

The unstability can go in branches, that's what they are for. A stable
master avoids all the unecesserary noise when "master breaks" because that
affects everyone, and it allows for continuous integration as a bonus.

There is a lot of different worflows available and I think it is our job to
adapt to the project's workflow, or maybe propose a different workflow.

Here are different typical used workflows:

   - only one master branch (small projects)
   - the git-flow workflow: one long-lived develop branch meant for
   development (unstable) from which features/releases branches might get
   branched from. The branch master is on the receiving end on the merges and
   anything merged to it should be stable (http://nvie.com/posts/a-
   successful-git-branching-model). Somewhat "long" release cycles.
   - the github workflow: master always "production ready", every
   features/bugfixes gets in a branch, and is merged when ready. Emphasis on
   short release cycles.
   - the git/kernel release model: many topical branches like "next", "pu"
   (proposed updates), "maint" (maintenance), etc.

I think in general having a stable master is desired, now I agree it's not
the end of the world if it gets unstable from time to time as long as it's
fixed quickly.

Philippe

[-- Attachment #2: Type: text/html, Size: 2466 bytes --]

Re: Preview: portable dumper

[Philippe Vaucher]

[-- Attachment #1: Type: text/plain, Size: 1840 bytes --]

>
> Once again, if those ideas seem strange, let alone incorrect, to
> you-all out there, just say the word, and I will step down.  Then I
> can stop worrying about the portable dumper, and you can stop worrying
> about my strange ideas.  Emacs is not my private project; I'm only
> entitled to promote my ideas if the others either support them or
> trust me to DTRT.  Please decide which one is it, and let's move on.
>

Eli, I think you are doing a fantastic job.

The only problem that seems to happen from time to time is that the
discussions starts getting personal and ends up with people saying threats
like "if you put my code in a branch I will fork Emacs" or "if you don't
agree with my ideas I will step down", which is just a way of saying "my
way or the highway". Using these rethorics is not constructive and doesn't
do anyone any good.

To me it looks like the only real problem here is communication, we should
all argue less and try to put ourselves in the other's schoes.

To Daniel, it's important that his work does not bitrot somewhere like the
concurrency branch did. It's also important that he feels there's a strong
chance of his code getting merged before he works more on it.
To Eli, it's important to have Daniel's code more "production ready"
(documentation, etc), and also to give the simpler implementation ideas a
try to avoid having to switch/maintain different systems twice in a short
period of time.

Both needs are understandable and valid. Nobody is wrong here.

To me it's clear that given the requirements, we can plan something that
would satisfy both... for example.

<Eli> Daniel, we'll give other people one month to come up with a sketch of
the alternative idea to see if it's feasible or not. If nothing comes up
after one month, we'll start working on merging your branch.

My 0.02$
Philippe

[-- Attachment #2: Type: text/html, Size: 2399 bytes --]

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Philippe Vaucher <philippe.vaucher@gmail.com>
> Date: Fri, 2 Dec 2016 10:00:28 +0100
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Emacs developers <emacs-devel@gnu.org>
> 
>  Once again, if those ideas seem strange, let alone incorrect, to
>  you-all out there, just say the word, and I will step down. Then I
>  can stop worrying about the portable dumper, and you can stop worrying
>  about my strange ideas. Emacs is not my private project; I'm only
>  entitled to promote my ideas if the others either support them or
>  trust me to DTRT. Please decide which one is it, and let's move on.
> 
> Eli, I think you are doing a fantastic job.
> 
> The only problem that seems to happen from time to time is that the discussions starts getting personal and
> ends up with people saying threats like "if you put my code in a branch I will fork Emacs" or "if you don't agree
> with my ideas I will step down", which is just a way of saying "my way or the highway". Using these rethorics
> is not constructive and doesn't do anyone any good.

What do you expect me to do when I see code which takes Emacs in a
direction I think is wrong?  There are a very few of such directions I
feel strongly we shouldn't take.  What am I supposed to do when I see
one of them proposed for inclusion?

More generally, is the maintainer allowed to reject some code or
course of actions?  If yes, what are the criteria for that?

> To Daniel, it's important that his work does not bitrot somewhere like the concurrency branch did. It's also
> important that he feels there's a strong chance of his code getting merged before he works more on it.
> To Eli, it's important to have Daniel's code more "production ready" (documentation, etc), and also to give the
> simpler implementation ideas a try to avoid having to switch/maintain different systems twice in a short period
> of time.

No, you've misunderstood the nature of our disagreement.  There's no
disagreement about the code being production-ready: Daniel himself
said it isn't yet.

> <Eli> Daniel, we'll give other people one month to come up with a sketch of the alternative idea to see if it's
> feasible or not. If nothing comes up after one month, we'll start working on merging your branch.

That's what I said, with the exception of the "month" part.

If we are talking about better communications, then I can suggest
discussing an idea before it is coded.  As a matter of fact, I
proposed a very similar idea 2 months ago, and it was voted down by
Paul, so I didn't pursue it, also because that discussion brought up
the "one large .elc file" idea, which Paul liked better (and so did
I), the only issue with it being the performance that we could get.
If Daniel have described his idea before coding it, I think at least
some of the aggravation (mine included) could have been spared.

Re: Preview: portable dumper

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]


() Eli Zaretskii <eliz@gnu.org>
() Fri, 02 Dec 2016 12:56:15 +0200

   What do you expect me to do when I see code which takes Emacs
   in a direction I think is wrong?  There are a very few of
   such directions I feel strongly we shouldn't take.  What am I
   supposed to do when I see one of them proposed for inclusion?

Look ahead to see the fork, point it out in ways people can
understand, and help them manage it when they arrive.  If luck
smiles, everyone will remember and learn; if not, only you will.

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Preview: portable dumper

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 366 bytes --]

On Fri, May 26, 2017 at 3:44 PM Thien-Thi Nguyen <ttn@gnu.org> wrote:

> Look ahead to see the fork, point it out in ways people can
> understand, and help them manage it when they arrive.  If luck
> smiles, everyone will remember and learn; if not, only you will.
>

??

I don't think the fork is happening. You are replying to a very old thread.
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 742 bytes --]

Re: Preview: portable dumper

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 533 bytes --]


() Kaushal Modi <kaushal.modi@gmail.com>
() Fri, 26 May 2017 20:26:25 +0000

   I don't think the fork is happening.

"There is no fork."  :-D

   You are replying to a very old thread.

It seems we both are.  Hopefully that's not a problem.

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Preview: portable dumper

[Stefan Monnier]

> Granted, the proposed dumper is not very complicated.  But it isn't
> trivial either.  So if we can achieve a similar effect by using the
> "normal" loadup code, which is much simpler and doesn't really require
> understanding anything new, I think it's more beneficial for the
> project's future.

Ken worked on speeding up the lread.c code, and it got to be
significantly faster, but not fast enough.  AFAIK it's got to the point
where it's not clear exactly how to speed it up further.  Not that it
can't be done, but that it's not obvious how, so it's likely going to
require some serious rethinking and maybe restructuring/rewrite of
the code.

Is it going to happen if we don't merge the pdumper?  I'm not so sure.

The main impetus behind speeding up lread.c is to replace unexec.c, so
I agree with you that merging the pdumper might mean that speeding up
lread.c will simply never happen.  But I think there's also a very
serious risk that even without the pdumper, speeding up lread.c will
still never happen: I have no intention on working at speeding up
lread.c, AFAICT Ken also gave up on it, anyone else?

Personally, I think that maybe we should move in the other direction:
keep lread.c for "source code" and generalize the pdumper code so it can
also be used for the .elc files.


        Stefan

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 02 Dec 2016 08:04:46 -0500
> 
> > Granted, the proposed dumper is not very complicated.  But it isn't
> > trivial either.  So if we can achieve a similar effect by using the
> > "normal" loadup code, which is much simpler and doesn't really require
> > understanding anything new, I think it's more beneficial for the
> > project's future.
> 
> Ken worked on speeding up the lread.c code, and it got to be
> significantly faster, but not fast enough.  AFAIK it's got to the point
> where it's not clear exactly how to speed it up further.  Not that it
> can't be done, but that it's not obvious how, so it's likely going to
> require some serious rethinking and maybe restructuring/rewrite of
> the code.
> 
> Is it going to happen if we don't merge the pdumper?  I'm not so sure.

I'm willing to give that a chance.  I don't see any reason to make the
decision today.

> The main impetus behind speeding up lread.c is to replace unexec.c, so
> I agree with you that merging the pdumper might mean that speeding up
> lread.c will simply never happen.  But I think there's also a very
> serious risk that even without the pdumper, speeding up lread.c will
> still never happen: I have no intention on working at speeding up
> lread.c, AFAICT Ken also gave up on it, anyone else?

Judging by Ken's response, he didn't give up yet.

In any case, it should be clear to anyone that code which isn't
written cannot be used.  So a reality check will get us straight.

> Personally, I think that maybe we should move in the other direction:
> keep lread.c for "source code" and generalize the pdumper code so it can
> also be used for the .elc files.

You mean, move the byte compiler to C?

If not, I don't understand the idea: pdump wants a file in a very
specific format, because it doesn't really understand the entities it
reads.

Re: Preview: portable dumper

[Stefan Monnier]

> You mean, move the byte compiler to C?

No.  I mean that bytecomp.el would generate a file in a format which
pdumper can use (which has various consequences, such as the fact that
.elc files aren't portable any more).


        Stefan

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No.  I mean that bytecomp.el would generate a file in a format which
  > pdumper can use (which has various consequences, such as the fact that
  > .elc files aren't portable any more).

That seems like a big disadvantage to me.  This could be sufficient
reason just by itself to stick with the current format for elc files.
People expect to install one elc file and have it work in multple
Emacs versions.  Only rarely have we made changes that would prevent
this.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Stefan Monnier]

>> No.  I mean that bytecomp.el would generate a file in a format which
>> pdumper can use (which has various consequences, such as the fact that
>> .elc files aren't portable any more).
> That seems like a big disadvantage to me.  This could be sufficient
> reason just by itself to stick with the current format for elc files.
> People expect to install one elc file and have it work in multple
> Emacs versions.  Only rarely have we made changes that would prevent
> this.

Indeed, such a change would mean that "the .elc file" would probably be
auto-generated behind the scene on-demand.


        Stefan

Re: Preview: portable dumper

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, emacs-devel@gnu.org
> Date: Fri, 02 Dec 2016 17:24:55 -0500
> 
>   > No.  I mean that bytecomp.el would generate a file in a format which
>   > pdumper can use (which has various consequences, such as the fact that
>   > .elc files aren't portable any more).
> 
> That seems like a big disadvantage to me.  This could be sufficient
> reason just by itself to stick with the current format for elc files.
> People expect to install one elc file and have it work in multple
> Emacs versions.

That's true, but note that Guile produces non-portable byte-compiled
files since long ago.  The Guile developers did that on purpose and
they think it's TRT.  I was at least initially surprised when I
learned about that.

Re: Preview: portable dumper

[Paul Eggert]

On 12/02/2016 06:51 AM, Stefan Monnier wrote:
> I mean that bytecomp.el would generate a file in a format which
> pdumper can use (which has various consequences, such as the fact that
> .elc files aren't portable any more).

The format could be easy for pdumper to use but still be portable. For 
example, the pdumper format could use 64-bit little-endian 
representation for Lisp_Object values, even on platforms with 32-bit or 
big-endian Lisp_Objects. Although this would complicate the pdumper and 
slow it down on platforms where the native format doesn't match the 
pdumper representation, the pdumper would still be significantly faster 
than lread.c.

Re: Preview: portable dumper

[Daniel Colascione]

On 12/02/2016 06:45 AM, Eli Zaretskii wrote:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 02 Dec 2016 08:04:46 -0500
>>
>>> Granted, the proposed dumper is not very complicated.  But it isn't
>>> trivial either.  So if we can achieve a similar effect by using the
>>> "normal" loadup code, which is much simpler and doesn't really require
>>> understanding anything new, I think it's more beneficial for the
>>> project's future.
>>
>> Ken worked on speeding up the lread.c code, and it got to be
>> significantly faster, but not fast enough.  AFAIK it's got to the point
>> where it's not clear exactly how to speed it up further.  Not that it
>> can't be done, but that it's not obvious how, so it's likely going to
>> require some serious rethinking and maybe restructuring/rewrite of
>> the code.
>>
>> Is it going to happen if we don't merge the pdumper?  I'm not so sure.
>
> I'm willing to give that a chance.  I don't see any reason to make the
> decision today.
>
>> The main impetus behind speeding up lread.c is to replace unexec.c, so
>> I agree with you that merging the pdumper might mean that speeding up
>> lread.c will simply never happen.  But I think there's also a very
>> serious risk that even without the pdumper, speeding up lread.c will
>> still never happen: I have no intention on working at speeding up
>> lread.c, AFAICT Ken also gave up on it, anyone else?
>
> Judging by Ken's response, he didn't give up yet.
>
> In any case, it should be clear to anyone that code which isn't
> written cannot be used.  So a reality check will get us straight.
>
>> Personally, I think that maybe we should move in the other direction:
>> keep lread.c for "source code" and generalize the pdumper code so it can
>> also be used for the .elc files.
>
> You mean, move the byte compiler to C?

Ideally, yes: one way, we'll compile with libgccjit or something. But 
that's a long way away.

For now, nobody's talking about the byte compiler. Stefan's talking 
about the code that *reads* the byte compiler's output. Right now, this 
code is the normal lisp reader, but it doesn't have to be.

> If not, I don't understand the idea: pdump wants a file in a very
> specific format, because it doesn't really understand the entities it
> reads.

pdumper's loader is stupidly simple. It doesn't understand anything 
about what it's loading. That's what makes it fast. Stefan's idea is to 
make the elc format closer to the pdump format. Advantage: it'll be much 
faster that way. Disadvantage: elc files will be closely coupled to the 
precise Emacs builds that generated them.

Re: Preview: portable dumper

[Ken Raeburn]

On Dec 2, 2016, at 09:45, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 02 Dec 2016 08:04:46 -0500
>> 
>>> Granted, the proposed dumper is not very complicated.  But it isn't
>>> trivial either.  So if we can achieve a similar effect by using the
>>> "normal" loadup code, which is much simpler and doesn't really require
>>> understanding anything new, I think it's more beneficial for the
>>> project's future.
>> 
>> Ken worked on speeding up the lread.c code, and it got to be
>> significantly faster, but not fast enough.  AFAIK it's got to the point
>> where it's not clear exactly how to speed it up further.  Not that it
>> can't be done, but that it's not obvious how, so it's likely going to
>> require some serious rethinking and maybe restructuring/rewrite of
>> the code.
>> 
>> Is it going to happen if we don't merge the pdumper?  I'm not so sure.
> 
> I'm willing to give that a chance.  I don't see any reason to make the
> decision today.
> 
>> The main impetus behind speeding up lread.c is to replace unexec.c, so
>> I agree with you that merging the pdumper might mean that speeding up
>> lread.c will simply never happen.  But I think there's also a very
>> serious risk that even without the pdumper, speeding up lread.c will
>> still never happen: I have no intention on working at speeding up
>> lread.c, AFAICT Ken also gave up on it, anyone else?
> 
> Judging by Ken's response, he didn't give up yet.

No, I just got busy with other things for a bit.

Stefan’s right too… since the current time sinks are the “read” functions themselves rather than in support code being called, it’s less immediately obvious where work is needed or how to proceed.  And it’s certainly possible changes will be needed that aren’t as nicely localized as getc() or recursive placeholder substitution.  But, I haven’t tried, yet… :-)

Ken

Re: Preview: portable dumper

[Paul Eggert]

Ken Raeburn wrote:
> Stefan’s right too… since the current time sinks are the “read” functions themselves rather than in support code being called, it’s less immediately obvious where work is needed or how to proceed.

Stefan is right so often that it must embarrass him. :-)

FWIW I recently tuned a private copy of lread.c to use syscalls rather than 
stdio. It didn't help significantly, so I gave that up.

RE: Preview: portable dumper

[Drew Adams]

> People expect to install one elc file and have it work in
> multple Emacs versions.

Would that it still were so.

> Only rarely have we made changes that would prevent this.

That used to be the case.  Now, nearly every major release is
incompatible, AFAICT.  As far as I can see, long gone are the
days when you could byte-compile in an older version and use
the byte-compiled code in a newer version.

Re: Preview: portable dumper

[Eli Zaretskii]

> Date: Fri, 2 Dec 2016 16:07:24 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> 
> > Only rarely have we made changes that would prevent this.
> 
> That used to be the case.  Now, nearly every major release is
> incompatible, AFAICT.  As far as I can see, long gone are the
> days when you could byte-compile in an older version and use
> the byte-compiled code in a newer version.

I'm not aware of such problems.  AFAIK, we only made an incompatible
change once, in Emacs 23.  The byte-compiled files compiled by later
versions should all be compatible.  If you see a problem with that,
please report it as a bug.

Re: Preview: portable dumper

[Joost Kremers]

On Sat, Dec 03 2016, Eli Zaretskii wrote:
>> Date: Fri, 2 Dec 2016 16:07:24 -0800 (PST)
>> From: Drew Adams <drew.adams@oracle.com>
>> Cc: eliz@gnu.org, emacs-devel@gnu.org
>> 
>> > Only rarely have we made changes that would prevent this.
>> 
>> That used to be the case.  Now, nearly every major release is
>> incompatible, AFAICT.  As far as I can see, long gone are the
>> days when you could byte-compile in an older version and use
>> the byte-compiled code in a newer version.
>
> I'm not aware of such problems.  AFAIK, we only made an 
> incompatible
> change once, in Emacs 23.  The byte-compiled files compiled by 
> later
> versions should all be compatible.  If you see a problem with 
> that,
> please report it as a bug.

Well, I may have run into a problem of this kind: a package 
(`expand-region') that started behaving weird when I moved from 
Emacs 24.5 to a 25 pretest. I did not recompile any packages when 
I upgraded Emacs, so when I ran into the problem, I was running 
Emacs 24 byte code in Emacs 25. When I recompiled `expand-region', 
the problem I was experiencing went away.

I didn't investigate further (I know zilch about Emacs' byte code 
and my problem was solved), so I can't be 100% sure it was due to 
a byte code incompatibility, it just seemed likely. (I also didn't 
consider filing a bug with Emacs, because it made sense to me 
*not* to expect byte-code compatibility across major versions.)

Anyway, the details are here (no technical details, just a 
description of the problem):

https://github.com/magnars/expand-region.el/issues/195


-- 
Joost Kremers
Life has its moments

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That used to be the case.  Now, nearly every major release is
  > incompatible, AFAICT.

Do others agree that this is the case?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

RE: Preview: portable dumper

[Drew Adams]

> > > Only rarely have we made changes that would prevent this.
> >
> > That used to be the case.  Now, nearly every major release is
> > incompatible, AFAICT.  As far as I can see, long gone are the
> > days when you could byte-compile in an older version and use
> > the byte-compiled code in a newer version.
> 
> I'm not aware of such problems.  AFAIK, we only made an incompatible
> change once, in Emacs 23.  The byte-compiled files compiled by later
               ^^^^^^^^^^^                           ^^^^^^^^^^^^^^^^^
> versions should all be compatible.  If you see a problem with that,
> please report it as a bug.

A change _in_ 23?  Or a change _after_ 23?  If the change
occurred in 23 then files compiled in 23 should be compatible
in later releases.  It should not be the case only that files
compiled in later releases are compatible with still later releases.

I cannot (always) byte-compile in 23 and use the result in
later releases, at least.  (It depends on the file.)

It may or may not represent a hard incompatibility.  It might
just be a change in a calling sequence or existence of some
macros.  I don't know, and I don't have the time to dig into it.

I do know that I do not hesitate to have conditional code that
(normally) is all that's needed, to deal with such changes,
and that that no longer suffices.

Are you suggesting that if someone can report a true
incompatibility introduced after Emacs 23 you will actually
change the byte compiler to remove it, and that you will
backport that fix to where it was introduced and later
releases?  (Otherwise, the fix doesn't really help.)

You can believe me or not.  I am used to conditionalizing
code to make it work with multiple releases.  And I have
always tried, for my own use, to byte-compile in the oldest
release possible, so that the result works for all of the
releases the particular library supports.  I have had to
stop compiling (except for testing, to pick up warnings)
some libraries because of what I've found to be
byte-compilation incompatibility.

Yes, my claim remains vague.  And yes, I'm anyway glad
to hear that there is some profession of still wanting
to keep compatibility for future releases.  That is a
large consolation, for which I am grateful.

Re: Preview: portable dumper

[Eli Zaretskii]

> Date: Sat, 3 Dec 2016 08:14:35 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: rms@gnu.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > > > Only rarely have we made changes that would prevent this.
> > >
> > > That used to be the case.  Now, nearly every major release is
> > > incompatible, AFAICT.  As far as I can see, long gone are the
> > > days when you could byte-compile in an older version and use
> > > the byte-compiled code in a newer version.
> > 
> > I'm not aware of such problems.  AFAIK, we only made an incompatible
> > change once, in Emacs 23.  The byte-compiled files compiled by later
>                ^^^^^^^^^^^                           ^^^^^^^^^^^^^^^^^
> > versions should all be compatible.  If you see a problem with that,
> > please report it as a bug.
> 
> A change _in_ 23?  Or a change _after_ 23?

The former, AFAIK.

> Are you suggesting that if someone can report a true
> incompatibility introduced after Emacs 23 you will actually
> change the byte compiler to remove it, and that you will
> backport that fix to where it was introduced and later
> releases?  (Otherwise, the fix doesn't really help.)

I don't know, but at least we will know where we stand.

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > However, this is not about making existing code simpler.  It's about
  > _adding_ more C code, where an alternative implementation might allow
  > doing that without such additions.

Yes, of course -- we all know that.

  > It's a new and significant mechanism, which is central to building and
  > starting Emacs.  It needs to be studied, understood, and maintained.
  > Therefore, it does increase the complexity.

It certainly increases the amount of code that understands Lisp object
formats.  However, whether it increases the degree of complexity is
another question.  I don't think it is a problem to have one more place
with the same kind and depth of complexity.  That means _more_ work
on some future rare occasions, but it does not mean _harder_ work.

As long as we have people to maintain allocation and GC, I don't
think we would have trouble finding people to maintain the portable
dumper along with them.

But it's true that I have not seen the code.  Someone should
look at the code and judge whether this is really true.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Paul Eggert]

Thanks for working on this. Although it's not the approach I would have 
taken, there is a real advantage to having running code. A few points:

* The portable dumper doesn't work with modules. Although this should be 
OK for Emacs builds since they don't use modules either, what is your 
thought about this? Will the module API need to change to support dumping?

* I am working on getting Emacs to work with gcc -fcheck-pointer-bounds, 
and will boost the priority of that task and CC: you when I have 
something ready. The portable dumper should work in such an environment. 
I don't see any fundamental problems with this; just giving you a heads-up.

* Functions like do_dump_relocation should use intptr_t or uintptr_t, 
not ptrdiff_t, to access pointers as integers (whether relocated or 
not). In theory this would support architectures where intptr_t and 
ptrdiff_t have different widths. Although Emacs doesn't run on such 
architectures now, conceivably it could and in the meantime the code 
might as well be clear about the distinction between pointers and offsets.

* pdumper_load should check that S_ISREG (stat.st_mode) (I suppose this 
won't be needed if we load from the emacs executable....).

* I suggest using the type bits_word (defined in lisp.h) rather than 
unsigned. This should work better on 64-bit machines (plus, on weird 
platforms).

* For pdumper_read a more-portable limit is MAX_RW_COUNT (see sysdep.c) 
instead of SSIZE_MAX. Maybe that value needs to move into a .h file.

* A nit: pdumper_marked_p relies on implicit conversion to bool, a style 
I find confusing for bool values. In a bool function it's clearer to 
write 'return E != 0;' instead of 'return E;' when E might have values 
other than 0 or 1.

* I haven't thought through all the implications of this and may have 
more comments later.

Re: Preview: portable dumper

[Stefan Monnier]

> I've been working on a portable dumper for GNU Emacs. The attached patch is
> an early version of this work.

Cool!

> An additional optimization follows, although this part isn't implemented
> yet: we can define a "preferred load address" for the dump and write
> relocation information such that if the dump and Emacs end up being where we
> expect them to be, we don't have to perform any relocations at all.

Oh, so the in-file format is also a valid in-memory format?  Great.
AFAIK, that's the crucial difference between a "fast .elc loader" and
your approach.  This should be key to get really good performance
(e.g. good enough that it doesn't slow down bootstrap noticeably).

> would benefit from compression, which will reduce IO loads: LZ4 reduces the
> dump size for me from ~12MB to ~4MB. As in the mmap case, we can throw away

Hmm... interesting the "dump one big .elc" approach I played with (which
kinda works right now, tho with the known performance issues) leads to
a dumped.elc file of a size a bit below 4MB (which gzip compresses down
to ~1MB).

I assume your system is using 64bit Lisp_Object, right?  So on a 32bit
Lisp_Object your dump size is probably closer to 6-7MB.

> functions pdumper.h --- but even so, it's within 100ms or so of an
> unexeced Emacs.

0.1s will still lead to a noticeable slow down of bootstrap, but at
least it should be good enough for the usual end-user who doesn't run
Emacs in batch mode.

> It's also possible to dump an already-dumped Emacs, so it should be possible
> for users to have their own dump files.

[ FWIW in theory my dumped.elc approach could also work on an
  already-dumped Emacs, for use by end-users (it basically just prints the
  contents of the obarray), tho it would indeed need extra work to weed out
  problems printing objects like markers and such.  ]


        Stefan

Re: Preview: portable dumper

[Daniel Colascione]

On Mon, Nov 28 2016, Stefan Monnier wrote:
>> I've been working on a portable dumper for GNU Emacs. The attached patch is
>> an early version of this work.
>
> Cool!
>
>> An additional optimization follows, although this part isn't implemented
>> yet: we can define a "preferred load address" for the dump and write
>> relocation information such that if the dump and Emacs end up being where we
>> expect them to be, we don't have to perform any relocations at all.
>
> Oh, so the in-file format is also a valid in-memory format?  Great.
> AFAIK, that's the crucial difference between a "fast .elc loader" and
> your approach.  This should be key to get really good performance
> (e.g. good enough that it doesn't slow down bootstrap noticeably).

Yes. The file is full of valid memory images of Lisp objects --- that's
why I'm guessing it might be a bit more verbose than ELC,
pre-compression. If you guess correctly at dump time what addresses the
Emacs and dump images will have, you can direct mmap it directly and
run.  With a non-PIC executable, you can do a pretty good job of that
and get performance a bit better than unexec, in theory.

If you load a dump and find that you've guessed wrong on the base
addresses, you need to relocate the image, but that can be done cheaply
and gradually, on access.

>> would benefit from compression, which will reduce IO loads: LZ4 reduces the
>> dump size for me from ~12MB to ~4MB. As in the mmap case, we can throw away
>
> Hmm... interesting the "dump one big .elc" approach I played with (which
> kinda works right now, tho with the known performance issues) leads to
> a dumped.elc file of a size a bit below 4MB (which gzip compresses down
> to ~1MB).
>
> I assume your system is using 64bit Lisp_Object, right?  So on a 32bit
> Lisp_Object your dump size is probably closer to 6-7MB.

Right --- I've tested on 64-bit only. I imagine testing 32-bit and
wide-int might reveal some latent bugs.

>
>> functions pdumper.h --- but even so, it's within 100ms or so of an
>> unexeced Emacs.
>
> 0.1s will still lead to a noticeable slow down of bootstrap, but at
> least it should be good enough for the usual end-user who doesn't run
> Emacs in batch mode.

Oh, of course. But that's without any attention paid to
optimization. I'm sure it can go much faster.

>> It's also possible to dump an already-dumped Emacs, so it should be possible
>> for users to have their own dump files.
>
> [ FWIW in theory my dumped.elc approach could also work on an
>   already-dumped Emacs, for use by end-users (it basically just prints the
>   contents of the obarray), tho it would indeed need extra work to weed out
>   problems printing objects like markers and such.  ]

Yeah --- then both approaches get into the problem of how to dump
arbitrary object graphs, which in practice ends up being "dump the whole
heap except for processes, windows, and frames."

Re: Preview: portable dumper

[Stefan Monnier]

> run.  With a non-PIC executable, you can do a pretty good job of that
> and get performance a bit better than unexec, in theory.

Got me curious: how/why could it be even better than unexec?

> Right --- I've tested on 64-bit only.  I imagine testing 32-bit and
> wide-int might reveal some latent bugs.

I'm not worried about that.

>>> functions pdumper.h --- but even so, it's within 100ms or so of an
>>> unexeced Emacs.
>> 0.1s will still lead to a noticeable slow down of bootstrap, but at
>> least it should be good enough for the usual end-user who doesn't run
>> Emacs in batch mode.
> Oh, of course.  But that's without any attention paid to
> optimization.  I'm sure it can go much faster.

Sure, but it's already fast enough to be usable.
[ Contrary to my dumped.elc, which is currently usable only for
  non-short-lived interactive sessions where you're willing to spend an
  extra second of startup.  ]

> Yeah --- then both approaches get into the problem of how to dump
> arbitrary object graphs, which in practice ends up being "dump the whole
> heap except for processes, windows, and frames."

But the dumped.elc approach has it much more difficult here because it
has to work within the scope of a predefined existing format.


        Stefan

Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Nov 29 2016, Stefan Monnier wrote:
>> run.  With a non-PIC executable, you can do a pretty good job of that
>> and get performance a bit better than unexec, in theory.
>
> Got me curious: how/why could it be even better than unexec?

With unexec, we're stuck with whatever heap layout malloc gives us, and
whenever we modify a byte within this heap image, the operating system
has to make a copy of that byte's host page and map that copy back into
our address space. (That is, we take a copy-on-write fault.) Pure
allocation partially, but not completely, addresses this overhead.

With the portable dumper, *we* choose the layout and we can do things
like put strings and other unlikely-to-change bits in part of the file
that probably won't get COW faults.  We also avoid the need to store
malloc data structures.  At runtime, we store the mark bits for objects
in the pdumper image in a contiguous dedicated memory area instead of
mutating the image, further reducing the number of COW faults we need to
incur.  With pdumper and a non-PIC Emacs, you'll only pay for COW faults
on the lisp objects you actually modify.

Even *with* a PIC Emacs (which I hope is the default, because ASLR
greatly improves security), we can get these savings if
mmap(BASE_ADDRESS_STORED_IN_DUMP, ...) succeeds and we can map the dump
where we want.  If we can't map the dump where we want, we'll just
relocate it.

Alternatively, if we're not memory-mapping, we can compress the dump
file --- that's much harder to do with unexec.

Re: Preview: portable dumper

[Paul Eggert]

On 11/29/2016 01:19 PM, Daniel Colascione wrote:
> Even*with*  a PIC Emacs (which I hope is the default, because ASLR
> greatly improves security), we can get these savings if
> mmap(BASE_ADDRESS_STORED_IN_DUMP, ...) succeeds and we can map the dump
> where we want.  If we can't map the dump where we want, we'll just
> relocate it.

Although your other advantages sound good, this one sounds worrisome. If 
Emacs maps the dump to the same place every time, we are giving up ASLR 
for the dump itself, and won't that pose a potential security risk? If 
so, perhaps it would be better to not mmap to the base address stored in 
the dump (unless we can determine that ASLR is not in use, I suppose).

Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Nov 29 2016, Paul Eggert wrote:
> On 11/29/2016 01:19 PM, Daniel Colascione wrote:
>> Even*with*  a PIC Emacs (which I hope is the default, because ASLR
>> greatly improves security), we can get these savings if
>> mmap(BASE_ADDRESS_STORED_IN_DUMP, ...) succeeds and we can map the dump
>> where we want.  If we can't map the dump where we want, we'll just
>> relocate it.
>
> Although your other advantages sound good, this one sounds
> worrisome. If Emacs maps the dump to the same place every time, we are
> giving up ASLR for the dump itself, and won't that pose a potential
> security risk? If so, perhaps it would be better to not mmap to the
> base address stored in the dump (unless we can determine that ASLR is
> not in use, I suppose).

The Emacs dump isn't PROT_EXEC though. It's pure* data, and this
difference diminishes the risk considerably, I think. I'd at least like
to make this behavior a user option.

* We do store function pointers in the dump, and an attacker could
  theoretically overwrite one of these to point where she wanted --- but
  with all PROT_EXEC code in the process being randomized, where would
  she point the function pointer that's under her control?

Re: Preview: portable dumper

[Paul Eggert]

On 11/29/2016 01:50 PM, Daniel Colascione wrote:
> * We do store function pointers in the dump, and an attacker could
>    theoretically overwrite one of these to point where she wanted --- but
>    with all PROT_EXEC code in the process being randomized, where would
>    she point the function pointer that's under her control?

I'm more worried about the next level up. Although the dump is pure data 
to the machine, it's not pure data to Elisp. Since the dump would 
contain bytecodes, if attackers can alter the bytecodes then they can 
execute whatever Elisp code they want.

Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Nov 29 2016, Paul Eggert wrote:
> On 11/29/2016 01:50 PM, Daniel Colascione wrote:
>> * We do store function pointers in the dump, and an attacker could
>>    theoretically overwrite one of these to point where she wanted --- but
>>    with all PROT_EXEC code in the process being randomized, where would
>>    she point the function pointer that's under her control?
>
> I'm more worried about the next level up. Although the dump is pure
> data to the machine, it's not pure data to Elisp. Since the dump would
> contain bytecodes, if attackers can alter the bytecodes then they can
> execute whatever Elisp code they want.

Good point. How about this? We'll let the PIC-ness of the Emacs
executable (which we know at build time) control whether we try to map
the dump at its preferred load address.  If a user is running non-PIC,
he's already vulnerable.

Re: Preview: portable dumper

[Paul Eggert]

Daniel Colascione wrote:
> How about this? We'll let the PIC-ness of the Emacs
> executable (which we know at build time) control whether we try to map
> the dump at its preferred load address.  If a user is running non-PIC,
> he's already vulnerable.

Yes, that sounds fine, thanks.

Re: Preview: portable dumper

[Stefan Monnier]

>> How about this? We'll let the PIC-ness of the Emacs
>> executable (which we know at build time) control whether we try to map
>> the dump at its preferred load address.  If a user is running non-PIC,
>> he's already vulnerable.
> Yes, that sounds fine, thanks.

I for one would prefer "measurably faster" over "hypothetically ever so
slightly less insecure".


        Stefan

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I'm more worried about the next level up. Although the dump is pure data 
  > to the machine, it's not pure data to Elisp. Since the dump would 
  > contain bytecodes, if attackers can alter the bytecodes then they can 
  > execute whatever Elisp code they want.

If they can change installed system files, they are already in control
of the machine, so why worry about this?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Daniel Colascione]

On Wed, Nov 30 2016, Richard Stallman wrote:
>   > I'm more worried about the next level up. Although the dump is pure data 
>   > to the machine, it's not pure data to Elisp. Since the dump would 
>   > contain bytecodes, if attackers can alter the bytecodes then they can 
>   > execute whatever Elisp code they want.
>
> If they can change installed system files, they are already in control
> of the machine, so why worry about this?

We're not talking about changing system files on disk. We're talking
about changing the image of a system file in memory.

Here's the scenario: suppose I can convince your Emacs to parse a
carefully crafted network packet that triggers a bug in Emacs and lets
me overwrite arbitrary memory in your Emacs process. Today, I win, in
the sense that I gain complete control over your Emacs process and can
do anything Emacs can do.  Address space layout randomization defeats
this attack by randomizing the memory layout of the process: if the
memory layout is randomized, my arbitrary-memory-overwrite vulnerability
doesn't let me do anything interesting, because I don't know *where* and
*what* to write using this vulnerability.

Now, if I know that your Emacs process is randomized *except* for the
location of the dump, I still win, because I can just write to the dump
image in order to make Emacs do what I want it to do. Writing the dump
works because we'll have mapped it copy-on-write, which is needed in
order to make the dump usable as an Emacs heap image. Writes work
whether or not the underlying file is itself writable.  Paul is
absolutely right that we shouldn't defeat ALSR this way.

Maybe we can randomize the dump base address *per* *machine*? That way,
remote attackers would still be frustrated. We'd still be making Emacs
users vulnerable to hostile users running on the same machine.

(Or we could just randomize per-user and dump Emacs the first time it
runs for a particular user?  If we do that after loading ~/.emacs, we
also improve people's startup time.  Invalidating and regenerating the
dump when configuration changes would be a challenge though.)

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Here's the scenario: suppose I can convince your Emacs to parse a
  > carefully crafted network packet that triggers a bug in Emacs and lets
  > me overwrite arbitrary memory in your Emacs process. Today, I win, in
  > the sense that I gain complete control over your Emacs process and can
  > do anything Emacs can do.

That reasoning is logically valid -- but is it really a plausible
scenario that Emacs's parsing of a packet would have a bug that
clobbers other unrelated memory?

What Emacs does with the contents of an incoming packet is mainly
to turn it into Lisp objects and make that available at Lisp level.
That means not much opportunity for such a bug to occur.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Daniel Colascione]

On 12/03/2016 01:32 PM, Richard Stallman wrote:
>   > Here's the scenario: suppose I can convince your Emacs to parse a
>   > carefully crafted network packet that triggers a bug in Emacs and lets
>   > me overwrite arbitrary memory in your Emacs process. Today, I win, in
>   > the sense that I gain complete control over your Emacs process and can
>   > do anything Emacs can do.
>
> That reasoning is logically valid -- but is it really a plausible
> scenario that Emacs's parsing of a packet would have a bug that
> clobbers other unrelated memory?

Bitter experience with other software has shown the answer to be "yes". 
The bug doesn't even have to be in Emacs --- it can be in a library we 
use. For example, we link against libpng when available, and libpng has 
had repeated security problems over the years: 
https://www.cvedetails.com/vulnerability-list/vendor_id-7294/Libpng.html

Gnus uses libpng to display images in news and email. All of this means 
that the scenario I've described can come to pass merely by browsing 
emacs-devel. I'd prefer to protect us against it.

(I don't want to single out libpng --- other libraries have bugs too. 
libpng is just the first one that came to mind.)

> What Emacs does with the contents of an incoming packet is mainly
> to turn it into Lisp objects and make that available at Lisp level.
> That means not much opportunity for such a bug to occur.

If you think so and are willing to take on this risk yourself, you can 
build a non-position-independent Emacs and get security and performance 
equivalent to what you have today. There are lots of hardening measures 
available to modern software, all optional, and we should allow users to 
use them.

Re: Preview: portable dumper

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We seem to be talking at cross purposes.  You raised the issue of
parsing network packets:

  > >   > Here's the scenario: suppose I can convince your Emacs to parse a
  > >   > carefully crafted network packet that triggers a bug in Emacs and lets
  > >   > me overwrite arbitrary memory in your Emacs process. Today, I win, in
  > >   > the sense that I gain complete control over your Emacs process and can
  > >   > do anything Emacs can do.

so I responded about Emacs's very simple parsing of network packets

  > > That reasoning is logically valid -- but is it really a plausible
  > > scenario that Emacs's parsing of a packet would have a bug that
  > > clobbers other unrelated memory?

but now you've changed the subject to libpng.

  > Bitter experience with other software has shown the answer to be "yes". 
  > The bug doesn't even have to be in Emacs --- it can be in a library we 
  > use. For example, we link against libpng when available, 

You might be right about libpng, but if so, that has nothing
particularly to do with parsing network packets in Emacs.
There are various ways the PNG file could get into your Emacs.

Regarding png files and security, there are many ways to view a png
file in various different programs.  It is useful to defend Emacs
better against such bugs, but maybe the better defense is a validator
library, separate from libpng, to carefully validate the format of a
PNG file and detect any inconsistency.  Then programs could call that
validator before showing the PNG file to libpng.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Preview: portable dumper

[Paul Eggert]

Richard Stallman wrote:
> What Emacs does with the contents of an incoming packet is mainly
> to turn it into Lisp objects and make that available at Lisp level.
> That means not much opportunity for such a bug to occur.

Sure, but incoming packets are not random and attackers can design packets to 
exploit Emacs bugs. So any opportunity whatsoever in Lisp conversion is more of 
concern than it would be for more-typical pointer bugs. And as Daniel mentioned, 
attackers will likely focus on the occasions when Emacs converts packet contents 
to non-Lisp objects.

I have been looking into improving checking in this area, by using the 
-fcheck-pointer-bounds option introduced in GCC 5 and supported by some newer 
CPUs. Although -fcheck-pointer-bounds should help, it won't be perfect and it 
will be disabled by default due to its performance cost.

Re: Preview: portable dumper

[Stefan Monnier]

> Although your other advantages sound good, this one sounds worrisome. If
> Emacs maps the dump to the same place every time, we are giving up ASLR for
> the dump itself, and won't that pose a potential security risk? If so,
> perhaps it would be better to not mmap to the base address stored in the
> dump (unless we can determine that ASLR is not in use, I suppose).

I live very happily without ASLR for Emacs right now.  So I'm not
worried about limiting ASLR to a subset of Emacs's heap.


        Stefan

Re: Preview: portable dumper

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Di., 29. Nov. 2016 um 22:35 Uhr:

> On 11/29/2016 01:19 PM, Daniel Colascione wrote:
> > Even*with*  a PIC Emacs (which I hope is the default, because ASLR
> > greatly improves security), we can get these savings if
> > mmap(BASE_ADDRESS_STORED_IN_DUMP, ...) succeeds and we can map the dump
> > where we want.  If we can't map the dump where we want, we'll just
> > relocate it.
>
> Although your other advantages sound good, this one sounds worrisome. If
> Emacs maps the dump to the same place every time, we are giving up ASLR
> for the dump itself


Is that actually the case? Looking at the code, I see nothing that would
prevent dumping with ASLR enabled.

[-- Attachment #2: Type: text/html, Size: 1191 bytes --]

Re: Preview: portable dumper

[Paul Eggert]

On 11/29/2016 02:22 PM, Philipp Stephani wrote:
> I see nothing that would prevent dumping with ASLR enabled.

Yes, that's correct. It's a performance thing, not a correctness thing. 
If Emacs undumps into the same address that it dumped from then it can 
avoid relocating some pointers, which should save some work and avoid 
making some copies (assuming copy-on-write).