bug#57035: Support for site-load.el in native-compile dumping

bug#57035: Support for site-load.el in native-compile dumping

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2458 bytes --]

Version 28.1+
By experimentation I've found the following issues are all that prevent
dumping with large numbers of pre-loaded native compiled units.  The
improvement in start-up performance versus requiring the same libraries in
a .emacs file is ridiculous, even with all libraries pre-compiled.  The
same fixes would provide support for user's construction of personal dump
files.
+ Add additional build stage so a compile phase that only loads loadup is
produced first, with a following stage to compile all the libraries loaded
by site-load.  Originally suggested by Stefan Monnier, it appears
absolutely necessary.  Most libraries not 8n loadup implicitly assume those
libraries are in the compile-time environment.
+. Add a flag variable to turn off the docstring hack in lead.c once loadup
has been processed (or eliminate the hack entirely).
+ Also turn off the prohibition on implicit loading while in dump mode
based on that flag variable.  The file dependencies of site-load can be
determined using load-history once the baseline (loadup only) dump has been
constructed.
+ Turn off purification flag before loading site-load, or fix bugs in
purecopy/pure_alloc (elimination of pure space counts as a bug fix).  Note
that when I've selectively turned off the purify flag in site-load to avoid
purecopy bugs, the process would reach the stage of calling pdump, but that
call would hang indefinitely (I let it go for a couple of hours the first
time, and it just burned cpu cycles without any change in memory use).
+ Enable the standard load path before loading site-load.
+ Make all_loaded_comp_units_h visible as a lisp variable, so loadup can
explicitly fix up all loaded comp units, whether there is a symbol bound to
a subr in that unit or not.
+ Make it possible to cleanly turn off further asynchronous compilation.  I
currently set native-comp-deferred-compilation-deny-list to '("/.*/"), but
I've still seen an occasional job running the compiler on some subr.
+ Ensure the compiler is run using the pdmp file constructed just from
loadup.  It will be impossible to support or debug compiling with arbitrary
compile-time environments.  I've already encountered an issue from having
call-interactive somehow advise itself when compiling with only additional
core emacs libraries in the dump.

I think that covers all the issues I dealt with.  If my employer allows I
can donate code to fix the purecopy issues in versions prior to 29.

Lynn

[-- Attachment #2: Type: text/html, Size: 2870 bytes --]

bug#57035: Support for site-load.el in native-compile dumping

[Lars Ingebrigtsen]

Lynn Winebarger <owinebar@gmail.com> writes:

> +. Add a flag variable to turn off the docstring hack in lead.c once
> loadup has been processed (or eliminate the hack entirely).

The hack has been removed on the trunk.

> I think that covers all the issues I dealt with.  If my employer
> allows I can donate code to fix the purecopy issues in versions prior
> to 29.

As I think Eli has said before, changes in this area are not going to be
backported to earlier Emacs versions, so you should just concentrate on
Emacs 29.

bug#57035: Support for site-load.el in native-compile dumping

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1273 bytes --]

On Sun, Aug 7, 2022, 8:55 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Lynn Winebarger <owinebar@gmail.com> writes:
>
> > +. Add a flag variable to turn off the docstring hack in lead.c once
> > loadup has been processed (or eliminate the hack entirely).
>
> The hack has been removed on the trunk.
>
> > I think that covers all the issues I dealt with.  If my employer
> > allows I can donate code to fix the purecopy issues in versions prior
> > to 29.
>
> As I think Eli has said before, changes in this area are not going to be
> backported to earlier Emacs versions, so you should just concentrate on
> Emacs 29.


The code fixing the limitations and bugs of purecopy (at least the ones
triggered while dumping a large variety of loaded libraries) has already
been implemented for 28.1 on my employer's systems.  If they are willing to
donate it (less than 100 lines of C), I'll post a patch.  Even if it won't
be incorporated in the Emacs distribution, Linux vendors who are
conservative in the version of Emacs they package may choose to use it in
their builds.  Aside from supporting bool vectors and strings with text
properties in pure space, it fixes a couple of fatal bugs that can be
triggered while dumping - one segfault and one runaway allocation.

Lynn

[-- Attachment #2: Type: text/html, Size: 1853 bytes --]