Re: Native-compilation build process (was Re: Loading tramp for dump goes into infinite regress)

[Lynn Winebarger <owinebar@gmail.com>]

On Wed, Aug 3, 2022 at 12:15 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Wed, 3 Aug 2022 10:53:12 -0400
> > Cc: Andrea Corallo <akrl@sdf.org>, Michael Albinus <michael.albinus@gmx.de>,
> >       emacs-devel <emacs-devel@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>
> >
> > > Why are you running "make" only in 'src'?  Some necessary parts of the
> > > build must also run "make" in 'lisp' and in other places.
> >
> > I'm replicating the 3 stage build process Stefan Monnier (added to the
> > cc list) described by invoking a shell script from site-load, when
> > dump-mode is set to pdump.
>
> Well, Stefan definitely didn't mean for you to shoot yourself in the
> foot.  Isn't it possible to run these 2 stages from the top-level
> Makefile?
Not that I can tell - I don't  know what targets I would use from the
top level, but I do know that "emacs.pdmp" and "src/emacs.pdmp"
produce an "unknown target" response.
In any case, I  don't believe I have shot myself in the foot.  I was
just able to produce and use a dump with
* 698 libraries in the preloaded-file-list
    * 136 files listed in lisp.mk
    * 521 files listed in site-load.mk (the file I generate from site-load)
    * I'm not sure where the remainder are from
* 426 native compilation units in "all-loaded-comp-units-h"
* 1126 eln files under native-lisp/28.1-<hash>/preloaded - none appear
directly in 28.1-<hash>
* 1504 elc files under lisp
* 1557 el files under lisp
I'm not sure why some of the files were byte compiled but not native
compiled (or at least, not loaded in the dump).  But at the end of
site-load I loaded cus-start (with dump-mode set to nil), cus-load,
and finally cus-edit.  The result of running customize is promising in
terms of (lack of) lag, but I won't really know if it works until I've
added all the additional site-lisp files in the mix.
My solution was to allow the dump to contain a string with the
absolute path.  That works for my purposes.  When I have something I
want to install permanently, I'll specify the final location as well.
Or I might just create a dump only intended to be run from the
installed location.
I think my total modifications to the distribution source are in the
range of maybe 50 lines of C and a handful of changes of "defconst" to
"defvar" in the lisp source files.  Since one of the goals of this
particular exercise is to be able to minimize the local maintenance of
patches, I consider it to be a win.  Keep in mind I don't control the
rest of the build tools on this system, so autoconf etc may change
without my input.  That's why I rather not do anything that touches
the configure script(s) or Makefile.in.  I'd rather modify the shell
script to accomodate any changes in the process or set of targets
post-configure, and rely on the emacs maintainers expert knowledge of
the configuration process to deal with any changes in the build
toolset.

One change I had to make was to set native-comp-deferred-compilation to nil:
make -C ../lisp EMACS=../src/emacs-1 compile-target
TARGETS="${SL_TARGETS}" BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq
comp-file-preloaded-p t native-comp-deferred-compilation nil)'"
Otherwise I would end up with hundreds of lingering emacs processes
doing something with an argument that was something like
"subr-int-native-comp-<blah>-call-interactively-<hash>...".  The
recursively invoked make process for Stefan's "stage 2" would
nominally complete, but the resulting dump would have a random number
of native compilation units loaded.
Hopefully that's useful information for someone.

> > In any case, once loadup determines there are native compile units
> > loaded while preparing for the dump, either they all should have the
> > names fixed up, or an error should be signaled.
>
> The names cannot be fixed up, because the build doesn't know how.
> These two variables get their values from the configure script, where
> the user specifies where the package will be installed.  Without that,
> the build cannot know their values.

That's your call, I'm just pointing out above that it's usually better
to signal an error when it happens (not  supplying the arguments you
are stating as a requirement) than to signal an error over the
subsequent symptoms of the actual error (in this case, the attempt to
load the incoherent eln file).  I just don't understand why you would
ever want emacs to produce a dump file that will never be loadable
according to the requirements you are enforcing.  Are we actually in
disagreement on that?

> > There could be a flag to
> > dump an image with only the (relative) final installation paths
> > recorded, or it could just be a build that will always be run in place
> > (e.g. an ordinary user build).
>
> Many users never install Emacs, or at least never install some of its
> versions.

I don't doubt that, but this point is a little humorous given the
evident number of daily users of the latest commit in the master
trunk.

>
> Anyway, this is a non-starter: we won't give up this useful
> functionality.

I never said anything about giving it up.  That's distinct from not
making it a hard requirement.
I'm just clarifying the point I made. I don't expect your position to
change, but I'd prefer the rejection to be of what I actually
suggested.

Thanks,
Lynn