Re: Docstring hack

[Lynn Winebarger <owinebar@gmail.com>]

On Sun, Jul 31, 2022 at 5:32 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> Loading `site-load.el` in the first dump is a bad idea because files
> haven't been compiled yet.
>
> Loading `site-load.el` in the second dump is a bad idea because:
> - as currently written, the site-loaded files aren't compiled, so it's
>   too early to dump them.
> - if you change the build to byte-compile them before the second dump,
>   you'll be byte-compiling them with the bootstrap-emacs which might
>   work but will lead to a slower compilation.
>
> So, I suggest something like:
>
>     mv lisp/site-load.el lisp/my-site-load.el
>     make
>     rm src/emacs
>     mv lisp/my-site-load.el  lisp/site-load.el
>     mv lisp/my-site-load.elc lisp/site-load.elc
>     make
>
> So the first 2 dumps are "normal" without any site-loaded files, and
> that's followed by a 3rd dump, where all the ELisp files are already
> byte-compiled.

I have now successfully dumped (byte-compiled only) an additional 511
core emacs files by calling a shell script during the current
second-stage dump to make the actual second stage dump and build the
required elc files, as well as finder-inf and cus-load.
The bad news is, when I just turn off Vpurify_flag at points in the
site-load to avoid a segfault or other problem (and then re-enable it
after the problem libraries), site-load will finish without error, but
the actual dumping process will fail for some reason I haven't
debugged.
The good news is that I was able to resolve the remaining problems
with a handful of changes to the C code in alloc and lread, and
changes to a couple of elisp files.
* alloc - put in a hash table of objects that have been purified
during a call to Fpurecopy, so cycles are not followed.
*           Also changed the "small_amount" in pure_alloc to 20000 and
printed a message on every allocation going over, since I can't rely
on the process actually finishing if many megabytes of additional pure
space are required.
* lread - put in a docstring-hack flag used as an extra and
conditional for the hack, so I could turn that off in site-load
without changing any variables used by the existing code
* easy-mmode - easy-mmode-defmap produces a defconst for a keymap
variable, which I changed to defvar
* tramp-sh - the defconsts for the tramp-completion-function-alist-*
variables changed to defvar, since for some reason the values are
getting modified, probably during tramp-startup-hook

I am going to try native-compilation again.  I can't see why I was
getting an "incoherent" library message, now that I understand the
loadup code that sets the compilation unit file name before the dump.
So I've made the table of compilation units registered by the
load_compilation_unit visible in lisp.  That way I can compare the
method used by loadup, which depends on subrs in compilation units
being accessible through function symbol values, in the site-load file
directly.  Or just do the conversion directly based on that table,
which should guarantee any NCU encountered by pdumper has the file
field set "coherently".

Given the elimination of pure space will not be back ported to 27 and
28, and the problem with pdump when the purify flag is turned off
while loading files,  I think some of these changes (or similar)
should be included maintenance releases of those two major versions,
so there will be some way for users stuck with those versions to
effectively dump significant chunks of core emacs beyond what's in
loadup.

Lynn