Loading tramp for dump goes into infinite regress

Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1892 bytes --]

And I don't even know where to start describing the issue.
It starts with having "(require 'tramp-loaddefs)" in tramp.el.  That's a
nonstarter for dumping, so I extracted all the variable/constant defs from
tramp-loaddefs and put them in a separate file "tramp-consts.el", removed
them from tramp-loaddefs, and then require the tramp-consts in tramp.el at
the same place.  I also added explicit declare-function for all the
functions the compiler complaints about being missing.
Then, at the end of the file, after the provide statement, I've tried
various ways of having the effect of the "require tramp-loaddefs" call
before the tramp init hook functions are run, since those are set in
tramp-loaddefs.  I tried just putting the require tramp-loaddefs there,
loading tramp-loaddefs, explicitly loading the individual files
(conditioned on dump-mode being nil), but all for naught.

If I wrap the "provide 'tramp" call with eval-and-compile, the dump will
successfully load it, but batch compiling it will fail on the "provide"
saying that tramp-methods variable is void.  This happens even if I've
removed the expressions following the provide.  And when I use "message"
calls to track where it's failing, the procedure definitely fails on the
provide.  It seems to be triggering some load procedure that's pulling in
the expressions created by tramp--with-startup, but I can't find the
expression responsible!
If I don't put the provide in an eval-and-compile, then the file compiled,
but when I run the dumping process (via make), it gets to "loading
net/tramp.elc" and just stops until all memory is consumed and the kernel
kills it.  Or I interrupt it.  We're talking tens of GBs of RAM being
consumed here.  If that were "real" and not some runaway meta-recursive
load cycle I wouldn't want the dumped emacs anyway.

Any advice on how I figure out what is going on here?

Thanks,
Lynn

[-- Attachment #2: Type: text/html, Size: 2306 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Michael Albinus]

Lynn Winebarger <owinebar@gmail.com> writes:

Hi Lynn,

> And I don't even know where to start describing the issue.

I don't even understand what you're trying. Could you pls give the
commands you're calling, that I have a chance to reproduce?

> Thanks,
> Lynn

Thanks, and best regards, Michael.

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2785 bytes --]

On Sun, Jul 24, 2022, 3:55 AM Michael Albinus <michael.albinus@gmx.de>
wrote:

> Lynn Winebarger <owinebar@gmail.com> writes:
>
> Hi Lynn,
>
> > And I don't even know where to start describing the issue.
>
> I don't even understand what you're trying. Could you pls give the
> commands you're calling, that I have a chance to reproduce?
>

Good point.  I'm trying to dump a "fully-loaded" emacs 28.1 that includes
several hundred packages from the various elpas.  I've modified those
packages so they reside in site-lisp, use cl-lib instead of cl, and use an
explicit data directory instead of loading files relative to the location
of an installed .el file.
I verified these all "work" by having about 1000 require statements in my
.emacs file to explicitly require them.  I then took the load history and
separated the files into ones from the lisp directory and ones from
site-lisp.  I created a site-load.el and site-lisp-load.el (loaded at the
end of site-load.el) to explicitly load the files in something resembling
dependency order, then running make in the emacs build directory to create
the build.
I've also compiled every ELISP source file using
batch-byte+native-compile.  That was really painful since, as far as I
know, the files can't be native compiled in parallel in arbitrary order
because of potential race conditions, and I haven't made a Makefile for the
.ELN files with the correct dependencies to native compile them in
dependency order using make to build them in parallel.

Now I'm working through the site-load.el file to dump all the required core
emacs ELISP files first (in addition to the ones in loadup) to get the
ordering right.  Tramp is somewhere in the 300s in the list, maybe
halfway-through the list of core emacs files.

I don't really expect anyone to be able to solve this for me.  I'm just
looking for some pointers on how I can figure out what is going on short of
stepping through the dumping process with gdb.

At this point, I'm just trying to figure out why I can native compile
tramp, but the load statement during the load preparing for dumping goes
into infinite regress.  The "require 'tramp-loaddefs" statement was an
obvious problem, so I took the steps I described in my first email to
resolve them.  I just don't understand what's going on when I end the
tramp.el file with "(eval-and-compile (provide 'tramp))" and comment out
the final statements that run the initialization hook.  When I try to
native compile that version of tramp.el, it still seems to attempt to run
the initialization hook when the provide statement is evaluated during the
compile.  I have no idea why that is happening.  I would suspect something
along the lines of "with-eval-on-load 'tramp", but I can't find it.

Any ideas would be appreciated.

Lynn

[-- Attachment #2: Type: text/html, Size: 3877 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Sun, 24 Jul 2022 08:05:30 -0400
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
>  I don't even understand what you're trying. Could you pls give the
>  commands you're calling, that I have a chance to reproduce?
> 
> Good point.  I'm trying to dump a "fully-loaded" emacs 28.1 that includes several hundred packages from the
> various elpas.  I've modified those packages so they reside in site-lisp, use cl-lib instead of cl, and use an
> explicit data directory instead of loading files relative to the location of an installed .el file.  
> I verified these all "work" by having about 1000 require statements in my .emacs file to explicitly require
> them.  I then took the load history and separated the files into ones from the lisp directory and ones from
> site-lisp.  I created a site-load.el and site-lisp-load.el (loaded at the end of site-load.el) to explicitly load the
> files in something resembling dependency order, then running make in the emacs build directory to create
> the build.

Note that Micheal asked you to show specific commands you used, so he
could try reproducing the problems you see, but you haven't provided
any such commands, only a rather general description of what you are
trying to do.

Can you please show the shell commands that you use to do the loading
step?

> I've also compiled every ELISP source file using batch-byte+native-compile.  That was really painful since,
> as far as I know, the files can't be native compiled in parallel in arbitrary order because of potential race
> conditions, and I haven't made a Makefile for the .ELN files with the correct dependencies to native compile
> them in dependency order using make to build them in parallel.

I'm not aware of any such problems.  IME, native compilation can run
in parallel in any order, and no race conditions of any kind are
supposed to happen.

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

On Sun, Jul 24, 2022 at 8:23 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Sun, 24 Jul 2022 08:05:30 -0400
> > Cc: emacs-devel <emacs-devel@gnu.org>
> >
> >  I don't even understand what you're trying. Could you pls give the
> >  commands you're calling, that I have a chance to reproduce?
> >
> > Good point.  I'm trying to dump a "fully-loaded" emacs 28.1 that includes several hundred packages from the
> > various elpas.  I've modified those packages so they reside in site-lisp, use cl-lib instead of cl, and use an
> > explicit data directory instead of loading files relative to the location of an installed .el file.
> > I verified these all "work" by having about 1000 require statements in my .emacs file to explicitly require
> > them.  I then took the load history and separated the files into ones from the lisp directory and ones from
> > site-lisp.  I created a site-load.el and site-lisp-load.el (loaded at the end of site-load.el) to explicitly load the
> > files in something resembling dependency order, then running make in the emacs build directory to create
> > the build.
>
> Note that Micheal asked you to show specific commands you used, so he
> could try reproducing the problems you see, but you haven't provided
> any such commands, only a rather general description of what you are
> trying to do.
>
> Can you please show the shell commands that you use to do the loading
> step?

That would be "make -j$(nproc)", which seems unhelpful.
A more detailed setup (from memory, since I did this on my employer's
system) would be:
------------------------
tar -xzf emacs-28.1-<whatever>.tar.gz
cd emacs-28.1
./autogen.sh
mkdir BUILD
cd BUILD
../configure --prefix=/does/not/exist --with-native-compilation
make -j$(nproc)
find ../lisp -name '*.el' | while read src; do ./src/emacs --batch
--eval "(batch-byte+native-compile)" $(src); done
< follow the steps I described to create the lisp/site-load.el file >
< create a file site-load.mk based on the shell commands in
src/Makefile.in for lisp.mk >
rm etc/DOC
( cd src; make ../etc/DOC )
./lib-src/make-docfile -d lisp -a etc/DOC $(<site-load.mk)
make -j$(nproc)
--------------------------
When the last step stops, I try to fix the issue in the elisp file,
delete the corresponding .elc, then rerun the batch compile command
above before rerunning the last make command.

>
> > I've also compiled every ELISP source file using batch-byte+native-compile.  That was really painful since,
> > as far as I know, the files can't be native compiled in parallel in arbitrary order because of potential race
> > conditions, and I haven't made a Makefile for the .ELN files with the correct dependencies to native compile
> > them in dependency order using make to build them in parallel.
>
> I'm not aware of any such problems.  IME, native compilation can run
> in parallel in any order, and no race conditions of any kind are
> supposed to happen.

Ok, great.  I thought I read, in either the gccemacs blog or the
emacs-devel archive, some concern about race conditions in building
eln files, probably in relation to the async scheduling.  Maybe I am
mistaken or that concern was resolved in the interim.
I'd still like to create that Makefile to do the compilaton in order.
I mean, I have created a file recording the dependencies to the extent
lib-require.el can determine them, but I haven't produced a working
Makefile as of yet.

Lynn

Re: Loading tramp for dump goes into infinite regress

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Sun, 24 Jul 2022 09:30:34 -0400
> Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> 
> > Note that Michael asked you to show specific commands you used, so he
> > could try reproducing the problems you see, but you haven't provided
> > any such commands, only a rather general description of what you are
> > trying to do.
> >
> > Can you please show the shell commands that you use to do the loading
> > step?
> 
> That would be "make -j$(nproc)", which seems unhelpful.

I mean the actual command that loads and dumps all the stuff.  If you
are using the Emacs Makefile's, I guess it's something like this:

  ./temacs -batch -l loadup --temacs=pdump

Or did you modify the Makefile's in the tree?

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

On Sun, Jul 24, 2022 at 9:34 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Sun, 24 Jul 2022 09:30:34 -0400
> > Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> >
> > > Note that Michael asked you to show specific commands you used, so he
> > > could try reproducing the problems you see, but you haven't provided
> > > any such commands, only a rather general description of what you are
> > > trying to do.
> > >
> > > Can you please show the shell commands that you use to do the loading
> > > step?
> >
> > That would be "make -j$(nproc)", which seems unhelpful.
>
> I mean the actual command that loads and dumps all the stuff.  If you
> are using the Emacs Makefile's, I guess it's something like this:
>
>   ./temacs -batch -l loadup --temacs=pdump
>
> Or did you modify the Makefile's in the tree?

I have not modified the Makefiles at this point - hence the explicit
use of make-docfile in my previous reply.

LC_ALL=C ./temacs -batch --no-build-details -l loadup --temacs=pdump
--bin-dest=/does/not/exist/bin
--eln-dest=/does/not/exist/lib/emacs/28.1

This is probably obvious, but I am only running the resulting emacs
from the build directory.

Lynn

Re: Loading tramp for dump goes into infinite regress

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Sun, 24 Jul 2022 10:05:19 -0400
> Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> 
> LC_ALL=C ./temacs -batch --no-build-details -l loadup --temacs=pdump
> --bin-dest=/does/not/exist/bin
> --eln-dest=/does/not/exist/lib/emacs/28.1

Why "28.1"?  Are you building the 28.1 release tarball?  I thought you
were building the master branch of the Git repository, which would be
Emacs 29.0.50.

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

On Sun, Jul 24, 2022, 10:09 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Sun, 24 Jul 2022 10:05:19 -0400
> > Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> >
> > LC_ALL=C ./temacs -batch --no-build-details -l loadup --temacs=pdump
> > --bin-dest=/does/not/exist/bin
> > --eln-dest=/does/not/exist/lib/emacs/28.1
>
> Why "28.1"?  Are you building the 28.1 release tarball?  I thought you
> were building the master branch of the Git repository, which would be
> Emacs 29.0.50.
>

I'm building this on my employer's system, so I start with official release
tarballs and a static set of packages mirrored on a particular date. The
whole thing is sandboxed.

On my personal machines for my personal projects, I'd start with something
closer to master.  That's where I'll be working on compilation units for
byte code based on the pdmp format, that should eventually make this
extended dumping process unecessary.

Lynn

[-- Attachment #2: Type: text/html, Size: 1948 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

On Sun, Jul 24, 2022 at 10:05 AM Lynn Winebarger <owinebar@gmail.com> wrote:
>
> On Sun, Jul 24, 2022 at 9:34 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: Lynn Winebarger <owinebar@gmail.com>
> > > Date: Sun, 24 Jul 2022 09:30:34 -0400
> > > Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> > >
> > > > Note that Michael asked you to show specific commands you used, so he
> > > > could try reproducing the problems you see, but you haven't provided
> > > > any such commands, only a rather general description of what you are
> > > > trying to do.
> > > >
> > > > Can you please show the shell commands that you use to do the loading
> > > > step?
> > >
> > > That would be "make -j$(nproc)", which seems unhelpful.
> >
> > I mean the actual command that loads and dumps all the stuff.  If you
> > are using the Emacs Makefile's, I guess it's something like this:
> >
> >   ./temacs -batch -l loadup --temacs=pdump
> >
> > Or did you modify the Makefile's in the tree?
>
> I have not modified the Makefiles at this point - hence the explicit
> use of make-docfile in my previous reply.
>
> LC_ALL=C ./temacs -batch --no-build-details -l loadup --temacs=pdump
> --bin-dest=/does/not/exist/bin
> --eln-dest=/does/not/exist/lib/emacs/28.1
>
> This is probably obvious, but I am only running the resulting emacs
> from the build directory.

Additional note - if I delete the tramp-<hashes>.eln file from
BUILD/native-lisp/28.1-<hash>/ without touching the corresponding elc
file, the loading proceeds - although there are still issues to
resolve in the subsequent tramp-X files being loaded.

Lynn

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

On Sun, Jul 24, 2022 at 12:00 PM Lynn Winebarger <owinebar@gmail.com> wrote:
>
> On Sun, Jul 24, 2022 at 10:05 AM Lynn Winebarger <owinebar@gmail.com> wrote:
> >
> > On Sun, Jul 24, 2022 at 9:34 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > > From: Lynn Winebarger <owinebar@gmail.com>
> > > > Date: Sun, 24 Jul 2022 09:30:34 -0400
> > > > Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> > > >
> > > > > Note that Michael asked you to show specific commands you used, so he
> > > > > could try reproducing the problems you see, but you haven't provided
> > > > > any such commands, only a rather general description of what you are
> > > > > trying to do.
> > > > >
> > > > > Can you please show the shell commands that you use to do the loading
> > > > > step?
> > > >
> > > > That would be "make -j$(nproc)", which seems unhelpful.
> > >
> > > I mean the actual command that loads and dumps all the stuff.  If you
> > > are using the Emacs Makefile's, I guess it's something like this:
> > >
> > >   ./temacs -batch -l loadup --temacs=pdump
> > >
> > > Or did you modify the Makefile's in the tree?
> >
> > I have not modified the Makefiles at this point - hence the explicit
> > use of make-docfile in my previous reply.
> >
> > LC_ALL=C ./temacs -batch --no-build-details -l loadup --temacs=pdump
> > --bin-dest=/does/not/exist/bin
> > --eln-dest=/does/not/exist/lib/emacs/28.1
> >
> > This is probably obvious, but I am only running the resulting emacs
> > from the build directory.
>
> Additional note - if I delete the tramp-<hashes>.eln file from
> BUILD/native-lisp/28.1-<hash>/ without touching the corresponding elc
> file, the loading proceeds - although there are still issues to
> resolve in the subsequent tramp-X files being loaded.

Ok, after getting the load order of subsequent files corrected
(avoiding errors from an autoload or require during dumping), I got
the same behavior from loading "tramp-sh".  So I removed the eln (but
not elc) file for that library, and now the loading proceeds past the
tramp-X files.
That will work for my immediate purposes, but it seems like something
undesirable is going on between dumping and the native compilation
unit.
That being said, it would also be handy to break out the
tramp-loaddefs file into a "declarations" file that can be safely
required from tramp.el and an "initialization" file that can be loaded
at the end of tramp.el (or externally) to set up the tramp
initialization hooks. The current method of decoupling the
method-specific implementation files from the generic interface file
(by requiring tramp-loaddefs at a certain point in the tramp.el file)
does not seem compatible with including it in the dump.

Lynn

Re: Loading tramp for dump goes into infinite regress

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Sun, 24 Jul 2022 12:20:47 -0400
> Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> 
> it seems like something undesirable is going on between dumping and
> the native compilation unit.

I find this hard to believe.

Btw, you are aware that dumping *.eln files basically dumps just their
names, not their code?  And that, when you start Emacs after dumping,
it will load all of those *.eln files one by one, which takes time
(*.eln files are just shared libraries, like *.so files), and use up
shared-library and handle slots of the Emacs process?  So I'm not even
sure doing this would make sense from the performance POV: it could be
that startup will slower, not faster.

In any case, I'd suggest to get this working with *.elc files in a
build without native compilation support, before you try it with
native compilation.

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 3530 bytes --]

On Sun, Jul 24, 2022 at 12:31 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Sun, 24 Jul 2022 12:20:47 -0400
> > Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> >
> > it seems like something undesirable is going on between dumping and
> > the native compilation unit.
>
> I find this hard to believe.

If I can recreate it in a more direct way on my personal machines, I'll put
in a bug with details including the required modifications to tramp.el (to
remove the problematic require of tramp-loaddefs).
I also observed the same problem/fix with:
  gnus/message
  progmodes/gdb-mi
  calc/calc-ext
  gnus/gnus-sum
  gnus/gnus-art
More troublesome, though is that emacs segfaulted when it attempted to load
nxml/rng-pttrn, whether loading native-compiled, byte-compiled, or source.

Otherwise, I was able to resolve the various issues (including some
extensive surgery on vc/ediff-X files to stamp out the circular loading).
I am going to have to do a real bootstrap to avoid the "incoherent eln"
error, though, and somehow force native compilation for all the libraries
loaded in the dump (that are not on the compiler's "forbidden" list,
anyway).

> Btw, you are aware that dumping *.eln files basically dumps just their
> names, not their code?  And that, when you start Emacs after dumping,
> it will load all of those *.eln files one by one, which takes time
> (*.eln files are just shared libraries, like *.so files), and use up
> shared-library and handle slots of the Emacs process?  So I'm not even
> sure doing this would make sense from the performance POV: it could be
> that startup will slower, not faster.

I am aware that dumping the eln files produces an indirection to the shared
library, but not the details of the implementation.
I would expect (or at least hope) that the effect of the evaluation that is
done on loading to be stored in the dump and for the shared libraries to
get loaded into the process without repeating the evaluation.  So
things like the order of customization groups should be fixed.
I would also expect that whatever the effect is on the emacs process with
.eln files loaded by the dumped file, it is no worse than it would be if
loaded as part of my .emacs file.
Another benefit I expect from native-compilation, dumped or not, is more
efficient memory use when running multiple emacs processes.  With dumping,
I would expect (or hope for) better garbage collector behavior since the
amount of allocation required for the loaded modules should be
pre-determined (whether byte- or native-compiled).  If the image is 300MB
(including the shared libraries), so be it, as long as the memory is shared
between multiple processes.

I'd also like a baseline expectation of performance with native-compiled
libraries in the dumped image.  To the extent that dumping verifies that
there is no cyclic loading, I can always try the approach to linking I
inquired about on this list about a month ago: just catenating the source
files together with some minor alterations.  I'm not sure how large a
source file the native compiler (or libgccjit) will handle, but it would be
interesting to see.

> In any case, I'd suggest to get this working with *.elc files in a
> build without native compilation support, before you try it with
> native compilation.

First I'll see if I can get the ordering straightened out with just the
core emacs files required by these packages, then do a test between
byte-compiled and native-compiled on those.

Lynn

[-- Attachment #2: Type: text/html, Size: 4414 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Mon, 25 Jul 2022 08:49:27 -0400
> Cc: michael.albinus@gmx.de, emacs-devel <emacs-devel@gnu.org>
> 
> > > it seems like something undesirable is going on between dumping and
> > > the native compilation unit.
> >
> > I find this hard to believe.
> 
> If I can recreate it in a more direct way on my personal machines, I'll put in a bug with details including the
> required modifications to tramp.el (to remove the problematic require of tramp-loaddefs).

Please do.

> > Btw, you are aware that dumping *.eln files basically dumps just their
> > names, not their code?  And that, when you start Emacs after dumping,
> > it will load all of those *.eln files one by one, which takes time
> > (*.eln files are just shared libraries, like *.so files), and use up
> > shared-library and handle slots of the Emacs process?  So I'm not even
> > sure doing this would make sense from the performance POV: it could be
> > that startup will slower, not faster.
> 
> I am aware that dumping the eln files produces an indirection to the shared library, but not the details of the
> implementation.
> I would expect (or at least hope) that the effect of the evaluation that is done on loading to be stored in the
> dump and for the shared libraries to get loaded into the process without repeating the evaluation.  So things
> like the order of customization groups should be fixed.

It depends on the top-level forms of the package, and what they do.
If they change values of variables, those values will be dumped, yes.

> I would also expect that whatever the effect is on the emacs process with .eln files loaded by the dumped
> file, it is no worse than it would be if loaded as part of my .emacs file.

How is this different from the previous expectation of yours?

> Another benefit I expect from native-compilation, dumped or not, is more efficient memory use when running
> multiple emacs processes.  With dumping, I would expect (or hope for) better garbage collector behavior
> since the amount of allocation required for the loaded modules should be pre-determined (whether byte- or
> native-compiled).  If the image is 300MB (including the shared libraries), so be it, as long as the memory is
> shared between multiple processes.  

I don't think I understand this expectation, and I don't think
natively-compiled code has any advantages wrt GC over the
byte-compiled code.

> I'd also like a baseline expectation of performance with native-compiled libraries in the dumped image.

What kind of performance?

> > In any case, I'd suggest to get this working with *.elc files in a
> > build without native compilation support, before you try it with
> > native compilation.
> 
> First I'll see if I can get the ordering straightened out with just the core emacs files required by these
> packages, then do a test between byte-compiled and native-compiled on those.

I'm saying that your job could be easier if you did the first step
with byte-compiled files.

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

On Mon, Jul 25, 2022 at 9:56 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > I am aware that dumping the eln files produces an indirection to the shared library, but not the details of the
> > implementation.
> > I would expect (or at least hope) that the effect of the evaluation that is done on loading to be stored in the
> > dump and for the shared libraries to get loaded into the process without repeating the evaluation.  So things
> > like the order of customization groups should be fixed.
>
> It depends on the top-level forms of the package, and what they do.
> If they change values of variables, those values will be dumped, yes.
>
> > I would also expect that whatever the effect is on the emacs process with .eln files loaded by the dumped
> > file, it is no worse than it would be if loaded as part of my .emacs file.
>
> How is this different from the previous expectation of yours?

I was describing my expectation of things like the shared-library and
handle slots of the Emacs process at run-time (factors exogenous to
emacs) versus endogenous factors like the state of the lisp heap.

> > Another benefit I expect from native-compilation, dumped or not, is more efficient memory use when running
> > multiple emacs processes.  With dumping, I would expect (or hope for) better garbage collector behavior
> > since the amount of allocation required for the loaded modules should be pre-determined (whether byte- or
> > native-compiled).  If the image is 300MB (including the shared libraries), so be it, as long as the memory is
> > shared between multiple processes.
>
> I don't think I understand this expectation, and I don't think
> natively-compiled code has any advantages wrt GC over the
> byte-compiled code.

This was a dumped versus non-dumped loading of native compiled
libraries.  Even if they are loaded sequentially in the same way
(except for the load-time evaluation), the creation or modification of
the heap from the pdump skips all the garbage collection cycles and
fragmentation resulting from the initial evaluation.

>
> > I'd also like a baseline expectation of performance with native-compiled libraries in the dumped image.
>
> What kind of performance?
>
Startup time, memory usage, and responsiveness.

> > > In any case, I'd suggest to get this working with *.elc files in a
> > > build without native compilation support, before you try it with
> > > native compilation.
> >
> > First I'll see if I can get the ordering straightened out with just the core emacs files required by these
> > packages, then do a test between byte-compiled and native-compiled on those.
>
> I'm saying that your job could be easier if you did the first step
> with byte-compiled files.

Given that I can't seem to produce a bootstrapped dump without getting
an incoherent eln file, I will be making one more attempt using
byte-compiled versions of the files in site-load for the dump
procedure, but with the native compiler enabled.
In either case, I will also bootstrap with native compilation disabled
for comparison.

Lynn

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

On Tue, Jul 26, 2022 at 7:24 AM Lynn Winebarger <owinebar@gmail.com> wrote:
>
> Given that I can't seem to produce a bootstrapped dump without getting
> an incoherent eln file, I will be making one more attempt using
> byte-compiled versions of the files in site-load for the dump
> procedure, but with the native compiler enabled.
> In either case, I will also bootstrap with native compilation disabled
> for comparison.

I was still seeing some runaway allocation even with native
compilation completely disabled, so I finally fired up gdb and found
the culprit:  "lmalloc", which has a comment saying the "while (true)"
loop shouldn't iterate on a modern platform.  RHEL 7.9 isn't hot off
the presses, but it's not that ancient.
I'll try to replicate on a personal machine and file a bug, but it's
not clear what triggers the condition. When it does occur, it always
occurs at the same place.  One instance was from loading cc-mode in
site-load (after byte-compiling all the dependencies in the correct
order).  I wrapped the "defun c-init-language-vars-for" sexp in an
eval, based on a comment in another one of the cc-* files (i.e.
cargo-cult programming on my part), and that stopped the runaway
allocation.  But then it happened while loading ibuffer.el.

Lynn

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

On Tue, Jul 26, 2022 at 8:58 PM Lynn Winebarger <owinebar@gmail.com> wrote:
> I was still seeing some runaway allocation even with native
> compilation completely disabled, so I finally fired up gdb and found
> the culprit:  "lmalloc", which has a comment saying the "while (true)"
> loop shouldn't iterate on a modern platform.  RHEL 7.9 isn't hot off
> the presses, but it's not that ancient.
> I'll try to replicate on a personal machine and file a bug, but it's
> not clear what triggers the condition. When it does occur, it always
> occurs at the same place.  One instance was from loading cc-mode in
> site-load (after byte-compiling all the dependencies in the correct
> order).  I wrapped the "defun c-init-language-vars-for" sexp in an
> eval, based on a comment in another one of the cc-* files (i.e.
> cargo-cult programming on my part), and that stopped the runaway
> allocation.  But then it happened while loading ibuffer.el.

Wrong again - the infinite loop is in pure_alloc.  It looks like there
is a 10,655 character string in ibuffer.el on which pure_copy is being
called, and pure_alloc only adds increments of 10k.
This I can report as a bug.

Lynn

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1546 bytes --]

On Tue, Jul 26, 2022, 10:48 PM Lynn Winebarger <owinebar@gmail.com> wrote:

> On Tue, Jul 26, 2022 at 8:58 PM Lynn Winebarger <owinebar@gmail.com>
> wrote:
> > I was still seeing some runaway allocation even with native
> > compilation completely disabled, so I finally fired up gdb and found
> > the culprit:  "lmalloc", which has a comment saying the "while (true)"
> > loop shouldn't iterate on a modern platform.  RHEL 7.9 isn't hot off
> > the presses, but it's not that ancient.
> > I'll try to replicate on a personal machine and file a bug, but it's
> > not clear what triggers the condition. When it does occur, it always
> > occurs at the same place.  One instance was from loading cc-mode in
> > site-load (after byte-compiling all the dependencies in the correct
> > order).  I wrapped the "defun c-init-language-vars-for" sexp in an
> > eval, based on a comment in another one of the cc-* files (i.e.
> > cargo-cult programming on my part), and that stopped the runaway
> > allocation.  But then it happened while loading ibuffer.el.
>
> Wrong again - the infinite loop is in pure_alloc.  It looks like there
> is a 10,655 character string in ibuffer.el on which pure_copy is being
> called, and pure_alloc only adds increments of 10k.
> This I can report as a bug.
>

The segmentation faults I've seen are also coming from purecopy.  It's
attempting to traverse a cyclic lisp structure and overflows the stack.

I'm going guess this kind of thing is why Stefan Monnier and Pip Cet were
enthusiastic about ditching pure space.

Lynn

[-- Attachment #2: Type: text/html, Size: 2440 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

I wanted to circle back and answer this now that I have a working "mega-dump".

On Mon, Jul 25, 2022 at 9:56 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > Another benefit I expect from native-compilation, dumped or not, is more efficient memory use when running
> > multiple emacs processes.  With dumping, I would expect (or hope for) better garbage collector behavior
> > since the amount of allocation required for the loaded modules should be pre-determined (whether byte- or
> > native-compiled).  If the image is 300MB (including the shared libraries), so be it, as long as the memory is
> > shared between multiple processes.
>
> I don't think I understand this expectation, and I don't think
> natively-compiled code has any advantages wrt GC over the
> byte-compiled code.

There should be at least one advantage - code representing function
calls to other byte code (even in the same compilation unit) are
represented by references in the heap which must be traced by the GC.
Take this with a block of salt, as I have not verified by inspection
of the natively compiled code, but many of those heap references
should be translated to control transfers to other code addresses,
which are not in the heap allocated or traced by the GC.
But that isn't what I was referring to above.
The first statement was intended with regard to the efficiency of
using shared libraries with code in read-only pages between multiple
processes versus byte code being loaded into modifiable memory regions
along with mutable data, increasing the probability that the memory
will either be not shared from the start or become unshareable due to
a CoW fault.
The second statement was comparing dumped versus undumped performance,
whether native- or byte- compiled.  This isn't a novel observation,
but the dump is basically a one-shot mark-compact GC collection, and
the mmap'ed pdump files would ideally be treated like the old
generation of a generational collector, and so not traced in most (or
all) collection cycles.  This is (or should be) one benefit of pure
space, although the requirement that it be read-only could be dropped
as long as the write-barrier remains and is used to record any
inter-generational pointers to a root set traced by the collector.

> > I'd also like a baseline expectation of performance with native-compiled libraries in the dumped image.
>
> What kind of performance?

Primarily usability.  And I can report that going from undumped to
dumped native-compiled does massively improve performance.  I have not
measured byte-compiled versus native-compiled when both are dumped.
Even with 2100+ eln files being loaded, the startup is very fast even
loading my .init settings (that turn on all the Semantic minor modes,
for example).
When I turned on the profiler for a while, it reported about 45% of
the cpu time was in gc, which I found surprising.  It was a little
laggy in some places, but not "stop-the-world collection of a 200MB
heap" laggy.
I'm tempted to try a quickish hack to see if I can turn pure space
into a never-collected elder generation by making the size a static
variable instead of a define and pointing pure to the start of the
dump image, then modifying the write barrier as described above, just
for my local 28.1 variant.  I don't mind wasting the memory if it
keeps the heap traced by the collector to a reasonable size focused on
shorter-lived constructs than the ones that survive the dump
collection.  I'd be surprised if it worked that easily, though.

>
> I'm saying that your job could be easier if you did the first step
> with byte-compiled files.

And it was - thanks.

Let me know if there are any specific metrics that would be useful.  I
can't promise I'll be able to get back with details quickly (I may
have to create novel implementations of the algorithms on my personal
machines), but I will see what I can do.  The emacs dev team and
contributors have really made tremendous improvements since I last
considered using the emacs source for playing around with some
language implementation ideas somewhere in the 2004-2007 period.  I
thought the massive code base of dynamic scoping and the
implementation of closure with "funvecs" (as they were called then)
was just too big to contemplate, but you guys have done it.  "Closing
the loop" by enabling the dumping of additional natively compiled
libraries is a minor tweak in the code, but it makes a huge difference
if you're trying to make use of all the additional functionality
provided by the available add-on packages.

Lynn

Re: Loading tramp for dump goes into infinite regress

[Po Lu]

Lynn Winebarger <owinebar@gmail.com> writes:

> The second statement was comparing dumped versus undumped performance,
> whether native- or byte- compiled.  This isn't a novel observation,
> but the dump is basically a one-shot mark-compact GC collection, and
> the mmap'ed pdump files would ideally be treated like the old
> generation of a generational collector, and so not traced in most (or
> all) collection cycles.  This is (or should be) one benefit of pure
> space, although the requirement that it be read-only could be dropped
> as long as the write-barrier remains and is used to record any
> inter-generational pointers to a root set traced by the collector.

This (very) minor benefit of pure space is already gone, since it is
unprotected and scanned normally for garbage collection in pdumper
builds.

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1449 bytes --]

On Tue, Aug 9, 2022, 8:36 AM Po Lu <luangruo@yahoo.com> wrote:

> Lynn Winebarger <owinebar@gmail.com> writes:
>
> > The second statement was comparing dumped versus undumped performance,
> > whether native- or byte- compiled.  This isn't a novel observation,
> > but the dump is basically a one-shot mark-compact GC collection, and
> > the mmap'ed pdump files would ideally be treated like the old
> > generation of a generational collector, and so not traced in most (or
> > all) collection cycles.  This is (or should be) one benefit of pure
> > space, although the requirement that it be read-only could be dropped
> > as long as the write-barrier remains and is used to record any
> > inter-generational pointers to a root set traced by the collector.
>
> This (very) minor benefit of pure space is already gone, since it is
> unprotected and scanned normally for garbage collection in pdumper
> builds.
>


I'm not sure you can conclude that the benefit would be minor when the
non-collected space is a 184MB dump file instead of a fixed pure space that
is 2 or 2.4MB as it is in the standard pdump.
At least some vestiges of the write barrier appear to be in place during
dump mode.  I've had to make a lot of minor corrections of "defconst"
keymaps to defvar, for example due to pure_write_error getting called.
Whether that barrier is uniformly enforced or not is one of the reasons I
suspect this quick hack would not be sufficient.

Lynn

[-- Attachment #2: Type: text/html, Size: 2328 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Po Lu]

Lynn Winebarger <owinebar@gmail.com> writes:

> I'm not sure you can conclude that the benefit would be minor when the
> non-collected space is a 184MB dump file instead of a fixed pure space
> that is 2 or 2.4MB as it is in the standard pdump.

AFAIK we don't officially support any non-standard dump.

> At least some vestiges of the write barrier appear to be in place
> during dump mode.  I've had to make a lot of minor corrections of
> "defconst" keymaps to defvar, for example due to pure_write_error
> getting called.  Whether that barrier is uniformly enforced or not is
> one of the reasons I suspect this quick hack would not be sufficient.

I don't think it's very well enforced.  For example, any C code that
calls XSETCAR without checking whether or not the object (i.e. the reuse
argument to Fmatch_end) is writable will result in the invariant being
broken.

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2044 bytes --]

On Tue, Aug 9, 2022, 9:43 AM Po Lu <luangruo@yahoo.com> wrote:

> Lynn Winebarger <owinebar@gmail.com> writes:
>
> > I'm not sure you can conclude that the benefit would be minor when the
> > non-collected space is a 184MB dump file instead of a fixed pure space
> > that is 2 or 2.4MB as it is in the standard pdump.
>
> AFAIK we don't officially support any non-standard dump.
>

I'm not sure how that's relevant, but depending on what you mean by
"support", it may not be totally accurate.  Aside from providing the option
of including site-load and site-init in the system build, there is the
user-level function dump-emacs-portable.   It's accurate to say the
maintainers are not responsible for supporting every random set of
libraries put together in a dump.  But it's also fair to say that the
impact of the system design on nonstandard user dumps is within the remit
of the maintainers as a general matter.  That said, the priority of such
concerns is entirely up to the maintainers.


> At least some vestiges of the write barrier appear to be in place
> > during dump mode.  I've had to make a lot of minor corrections of
> > "defconst" keymaps to defvar, for example due to pure_write_error
> > getting called.  Whether that barrier is uniformly enforced or not is
> > one of the reasons I suspect this quick hack would not be sufficient.
>
> I don't think it's very well enforced.  For example, any C code that
> calls XSETCAR without checking whether or not the object (i.e. the reuse
> argument to Fmatch_end) is writable will result in the invariant being
> broken.
>

True, but if all relevant mutators are effectively contained in lisp.h, the
problem of imposing a write barrier would be effectively contained.  Then
it would just be a matter of determining if the cost of the write barrier
being checked at every mutation is paid for by a perceivable reduction in
gc-related pauses.
But as I said, it's just a hack I'm tempted to try.  I probably won't
unless the gc pauses for this mega-dump build become intolerable.

Lynn

[-- Attachment #2: Type: text/html, Size: 3308 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Stefan Monnier]

> Otherwise, I was able to resolve the various issues (including some
> extensive surgery on vc/ediff-X files to stamp out the circular loading).
> I am going to have to do a real bootstrap to avoid the "incoherent eln"
> error, though, and somehow force native compilation for all the libraries
> loaded in the dump (that are not on the compiler's "forbidden" list,
> anyway).

BTW, I don't really understand why you're bumping into circularity issue
with things like ediff.  I do know it has circularity issues, but they
only affect compilation, AFAIK.

So, I'm wondering: do you include "the world" in both dumps (the
first, done to build `src/bootstrap-emacs.pdmp`, and the second to
build the final `src/emacs.pdmp`) or do you include it only
in the second dump?

Looking at `lisp/loadup.el`, it seems that the normal way is to dump
`site-load.el` in both cases, but that would indeed introduce more
difficulties than needed: we should only load `site-load.el` during the
second dump, so the bootstrap itself is not affected.

IOW, my crystal ball suggests you try the patch below.


        Stefan


diff --git a/lisp/loadup.el b/lisp/loadup.el
index 21a87dbd77b..63c5bc2adec 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -387,15 +386,16 @@
 ;; you may load them with a "site-load.el" file.
 ;; But you must also cause them to be scanned when the DOC file
 ;; is generated.
-(let ((lp load-path))
-  (load "site-load" t)
-  ;; We reset load-path after dumping.
-  ;; For a permanent change in load-path, use configure's
-  ;; --enable-locallisppath option.
-  ;; See https://debbugs.gnu.org/16107 for more details.
-  (or (equal lp load-path)
-      (message "Warning: Change in load-path due to site-load will be \
-lost after dumping")))
+(unless (member dump-mode '("bootstrap" "pbootstrap"))
+  (let ((lp load-path))
+    (load "site-load" t)
+    ;; We reset load-path after dumping.
+    ;; For a permanent change in load-path, use configure's
+    ;; --enable-locallisppath option.
+    ;; See https://debbugs.gnu.org/16107 for more details.
+    (or (equal lp load-path)
+        (message "Warning: Change in load-path due to site-load will be \
+lost after dumping"))))
 
 ;; Used by `kill-buffer', for instance.
 (load "emacs-lisp/rmc")

Re: Loading tramp for dump goes into infinite regress

[Stefan Monnier]

> IOW, my crystal ball suggests you try the patch below.

I think it's simplistic.  The better option is probably to dump 3 times:
1- for the bootstrap, unchanged (i.e. no site-load)
2- the build the "normal" Emacs, unchanged (i.e. no site-load)
3- after compiling all your files with the normal Emacs, finally do
   a monster dump.


        Stefan

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2390 bytes --]

On Mon, Jul 25, 2022, 12:54 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > Otherwise, I was able to resolve the various issues (including some
> > extensive surgery on vc/ediff-X files to stamp out the circular loading).
> > I am going to have to do a real bootstrap to avoid the "incoherent eln"
> > error, though, and somehow force native compilation for all the libraries
> > loaded in the dump (that are not on the compiler's "forbidden" list,
> > anyway).
>
> BTW, I don't really understand why you're bumping into circularity issue
> with things like ediff.  I do know it has circularity issues, but they
> only affect compilation, AFAIK.
>

The C code for the require form hits the panic button if it has to load
something when dump-mode is non-nil.  Something similar happens when an
autoload is invoked during dump-mode, but I haven't looked for the exact
piece of code that does it so I don't know if it's C or lisp.

So, I'm wondering: do you include "the world" in both dumps (the
> first, done to build `src/bootstrap-emacs.pdmp`, and the second to
> build the final `src/emacs.pdmp`) or do you include it only
> in the second dump?
>

I initially was trying to just run the second one, but I got the
"incoherent ELN" error.  So I set up a new build directory, did a fresh
configure followed by "make bootstrap".  I had to further refine the
site-load.el to explicitly load everything required at compile time, wrap
every instance of any expression requiring a generated file with "unless
dump-mode" (since in dump-mode the site-load or loadup file must load all
the required files to work anyway).  That fails when site-load starts
loading bits of cc-mode for the second dump since none of the files in
site-load are even byte-compiled with the bootstrap emacs.  So I generated
a list of .elc targets from site-load (same method used for lisp.mk), then
put together a hacked Makefile in BUILD/lisp that ran the bootstrap-emacs
executable with the suffix rule from the standard Makefile.  That also
failed with the incoherent ELN message.  Then I noticed the rule that uses
the "byte-compile-refresh-preloaded" function, so I'm trying that in the
Makefile.  However, I'm still waiting for the preloaded files to finish
native compiling.

I'd be happy to follow the 3 step path to dumping, but I thought the whole
"no redumping support" prevented that approach.

Lynn

[-- Attachment #2: Type: text/html, Size: 3285 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2789 bytes --]

On Mon, Jul 25, 2022, 8:28 PM Lynn Winebarger <owinebar@gmail.com> wrote:

> On Mon, Jul 25, 2022, 12:54 PM Stefan Monnier <monnier@iro.umontreal.ca>
> wrote:
>
>> > Otherwise, I was able to resolve the various issues (including some
>> > extensive surgery on vc/ediff-X files to stamp out the circular
>> loading).
>> > I am going to have to do a real bootstrap to avoid the "incoherent eln"
>> > error, though, and somehow force native compilation for all the
>> libraries
>> > loaded in the dump (that are not on the compiler's "forbidden" list,
>> > anyway).
>>
>> BTW, I don't really understand why you're bumping into circularity issue
>> with things like ediff.  I do know it has circularity issues, but they
>> only affect compilation, AFAIK.
>>
>
> The C code for the require form hits the panic button if it has to load
> something when dump-mode is non-nil.  Something similar happens when an
> autoload is invoked during dump-mode, but I haven't looked for the exact
> piece of code that does it so I don't know if it's C or lisp.
>
> So, I'm wondering: do you include "the world" in both dumps (the
>> first, done to build `src/bootstrap-emacs.pdmp`, and the second to
>> build the final `src/emacs.pdmp`) or do you include it only
>> in the second dump?
>>
>
> I initially was trying to just run the second one, but I got the
> "incoherent ELN" error.  So I set up a new build directory, did a fresh
> configure followed by "make bootstrap".  I had to further refine the
> site-load.el to explicitly load everything required at compile time, wrap
> every instance of any expression requiring a generated file with "unless
> dump-mode" (since in dump-mode the site-load or loadup file must load all
> the required files to work anyway).  That fails when site-load starts
> loading bits of cc-mode for the second dump since none of the files in
> site-load are even byte-compiled with the bootstrap emacs.  So I generated
> a list of .elc targets from site-load (same method used for lisp.mk),
> then put together a hacked Makefile in BUILD/lisp that ran the
> bootstrap-emacs executable with the suffix rule from the standard
> Makefile.  That also failed with the incoherent ELN message.  Then I
> noticed the rule that uses the "byte-compile-refresh-preloaded" function,
> so I'm trying that in the Makefile.  However, I'm still waiting for the
> preloaded files to finish native compiling.
>
> I'd be happy to follow the 3 step path to dumping, but I thought the whole
> "no redumping support" prevented that approach.
>

Even native compiling using the bootstrap emacs with
byte-compile-refresh-preloaded is failing.  I might try redoing the
bootstrap, then run the native compiler on the site-load files
sequentially. That will take several hours at least, though.

Lynn

[-- Attachment #2: Type: text/html, Size: 4145 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

On Mon, Jul 25, 2022, 12:54 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > Otherwise, I was able to resolve the various issues (including some
> > extensive surgery on vc/ediff-X files to stamp out the circular loading).
> > I am going to have to do a real bootstrap to avoid the "incoherent eln"
> > error, though, and somehow force native compilation for all the libraries
> > loaded in the dump (that are not on the compiler's "forbidden" list,
> > anyway).
>
> BTW, I don't really understand why you're bumping into circularity issue
> with things like ediff.  I do know it has circularity issues, but they
> only affect compilation, AFAIK.
>

I've added another flag variable that allows me to turn off the ban on
implicit loading in dump mode.  I used load history from a session that
required all the packages (that didn't have fatal errors) to generate a
reasonable facsimile of dependency ordering, but allowing autoload and
require to work as intended greatly simplifies the task.
I made a few more corrections to purecopy.  I added support for bool
vectors (used by js2-mode), and strings with text properties.  I was
concerned that the libraries with such string constants are designed to
rely on those properties.
I've finally gotten all the libraries to load, but the dump failed do to
having a dynamic module loaded.

Lynn



>

[-- Attachment #2: Type: text/html, Size: 2089 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 759 bytes --]

On Sat, Aug 6, 2022, 2:07 AM Lynn Winebarger <owinebar@gmail.com> wrote:

> I've added another flag variable that allows me to turn off the ban on
> implicit loading in dump mode.
>

The dump with all ~2500 ELN libraries loaded completed last night.
Unfortunately during startup (before any frame opens) one of them is
evaluating an expression with an argument that should never produce nil but
somehow is nil.
Fortunately I can now redump with just the offending library in site-load.
The error prints fairly quickly when run normally, but shared libraries
evidently open a lot slower under GDB.
Hopefully I'll get the same error with the smaller dump and be able to
determine if there's an issue in the dump file, the dump loading, or
something else.

Lynn

[-- Attachment #2: Type: text/html, Size: 1547 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1299 bytes --]

On Sat, Aug 6, 2022, 8:57 AM Lynn Winebarger <owinebar@gmail.com> wrote:

> On Sat, Aug 6, 2022, 2:07 AM Lynn Winebarger <owinebar@gmail.com> wrote:
>
>> I've added another flag variable that allows me to turn off the ban on
>> implicit loading in dump mode.
>>
>
> The dump with all ~2500 ELN libraries loaded completed last night.
> Unfortunately during startup (before any frame opens) one of them is
> evaluating an expression with an argument that should never produce nil but
> somehow is nil.
> Fortunately I can now redump with just the offending library in
> site-load.  The error prints fairly quickly when run normally, but shared
> libraries evidently open a lot slower under GDB.
> Hopefully I'll get the same error with the smaller dump and be able to
> determine if there's an issue in the dump file, the dump loading, or
> something else.
>

It started up fine when I only explicitly added the one library with the
failing expression, and then again when I dumped after loading the original
list up to that library.
The good news is that emacs starts in a couple of seconds with 1198 ELN
files in the dump.  I haven't been able to compile with profiling enabled
on this system, but I can tell you that is a couple of orders of magnitude
faster than just loading them in .init.

Lynn

[-- Attachment #2: Type: text/html, Size: 2572 bytes --]

Working fully native-compiled "mega dump" (was Re: Loading tramp for dump goes into infinite regress)

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 3328 bytes --]

On Sat, Aug 6, 2022, 11:39 AM Lynn Winebarger <owinebar@gmail.com> wrote:

> On Sat, Aug 6, 2022, 8:57 AM Lynn Winebarger <owinebar@gmail.com> wrote:
>
>> On Sat, Aug 6, 2022, 2:07 AM Lynn Winebarger <owinebar@gmail.com> wrote:
>>
>>> I've added another flag variable that allows me to turn off the ban on
>>> implicit loading in dump mode.
>>>
>>
>> The dump with all ~2500 ELN libraries loaded completed last night.
>> Unfortunately during startup (before any frame opens) one of them is
>> evaluating an expression with an argument that should never produce nil but
>> somehow is nil.
>> Fortunately I can now redump with just the offending library in
>> site-load.  The error prints fairly quickly when run normally, but shared
>> libraries evidently open a lot slower under GDB.
>> Hopefully I'll get the same error with the smaller dump and be able to
>> determine if there's an issue in the dump file, the dump loading, or
>> something else.
>>
>
> It started up fine when I only explicitly added the one library with the
> failing expression, and then again when I dumped after loading the original
> list up to that library.
> The good news is that emacs starts in a couple of seconds with 1198 ELN
> files in the dump.  I haven't been able to compile with profiling enabled
> on this system, but I can tell you that is a couple of orders of magnitude
> faster than just loading them in .init.
>

The bug seems to be from a couple of older (2010-2012) library files that
are dynamically scoped and use copy-face with the "frame" parameter.  All
the other combinations I've tried start up without a problem, so I'm just
going to leave these packages out of the dump.  Also excluded are most
lsp-mode libraries, because they seem to slow down the startup process
(like 4 seconds instead of 2 or less}, vterm and zmq (dynamic modules can't
be dumped), and a few modes that insist on attempting to download things on
startup.
What's included:
All of CEDET
All files in emacs-lisp
Most if not all files in tramp/mail/net
Everything in progmode that doesn't depend on an external utility (namely
moz) being installed
Any TeX or reference/citation functionality (reftex, bibtex, bibliography,
citar, citeproc)
Icon/emoji/image support (e.g alltheicons, major-mode-icons)
All completion and widening/narrowing libraries - helm, ivy, counsel,
company, auto-complete,
Perspective, window-purpose, and similar
Evil packages
Treemacs
Many other miscellaneous packages I'm forgetting.

All in the dump file is about 184 MB for an older Linux on an x86_64
architecture.  That would include all the lsp-mode packages from Melpa, or
at least the ones with more than a nominal number of downloads.  There are
2000-2500 ELN files that will be loaded.  When I loaded these from my init
file I could make a cup of coffee, maybe do a crossword before it would be
ready to use.

One of the packages with the dynamic scoping/face issues is tabbar-ruler,
which I only started using during this exercise and think is fantastic.
Hopefully it will work when required from the init file, as it does
currently.

At any rate it's fantastic at startup.  The customization interface is very
responsive without the usual lag from lazy construction of the
customization groups.  I'll have to see how it behaves after it's been
running a few days.

Lynn

[-- Attachment #2: Type: text/html, Size: 5579 bytes --]

Re: Working fully native-compiled "mega dump" (was Re: Loading tramp for dump goes into infinite regress)

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

On Sat, Aug 6, 2022, 4:23 PM Lynn Winebarger <owinebar@gmail.com> wrote:

> What's included:
> All of CEDET
> All files in emacs-lisp
> Most if not all files in tramp/mail/net
> Everything in progmode that doesn't depend on an external utility (namely
> moz) being installed
> Any TeX or reference/citation functionality (reftex, bibtex, bibliography,
> citar, citeproc)
> Icon/emoji/image support (e.g alltheicons, major-mode-icons)
> All completion and widening/narrowing libraries - helm, ivy, counsel,
> company, auto-complete,
> Perspective, window-purpose, and similar
> Evil packages
> Treemacs
> Many other miscellaneous packages I'm forgetting.
>

I somehow forgot: pretty much every org-related library out there.
The total number of loaded comp units in the final dump: 2104.

Lynn

[-- Attachment #2: Type: text/html, Size: 2177 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Andrea Corallo]

Lynn Winebarger <owinebar@gmail.com> writes:

> On Sun, Jul 24, 2022 at 12:31 PM Eli Zaretskii <eliz@gnu.org> wrote:
>>
>> > From: Lynn Winebarger <owinebar@gmail.com>
>> > Date: Sun, 24 Jul 2022 12:20:47 -0400
>> > Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
>> >
>> > it seems like something undesirable is going on between dumping and
>> > the native compilation unit.
>>
>> I find this hard to believe.
>
> If I can recreate it in a more direct way on my personal machines, I'll put in a bug with details including the required
> modifications to tramp.el (to remove the problematic require of tramp-loaddefs).
> I also observed the same problem/fix with:
>   gnus/message
>   progmodes/gdb-mi
>   calc/calc-ext
>   gnus/gnus-sum
>   gnus/gnus-art
> More troublesome, though is that emacs segfaulted when it attempted to load nxml/rng-pttrn, whether loading
> native-compiled, byte-compiled, or source.
>
> Otherwise, I was able to resolve the various issues (including some extensive surgery on vc/ediff-X files to stamp out
> the circular loading).  I am going to have to do a real bootstrap to avoid the "incoherent eln" error, though, and
> somehow force native compilation for all the libraries loaded in the dump (that are not on the compiler's "forbidden"
> list, anyway).
>
>> Btw, you are aware that dumping *.eln files basically dumps just their
>> names, not their code?  And that, when you start Emacs after dumping,
>> it will load all of those *.eln files one by one, which takes time
>> (*.eln files are just shared libraries, like *.so files), and use up
>> shared-library and handle slots of the Emacs process?  So I'm not even
>> sure doing this would make sense from the performance POV: it could be
>> that startup will slower, not faster.
>
> I am aware that dumping the eln files produces an indirection to the shared library, but not the details of the
> implementation.
> I would expect (or at least hope) that the effect of the evaluation that is done on loading to be stored in the dump and
> for the shared libraries to get loaded into the process without repeating the evaluation.  So things like the order of
> customization groups should be fixed.

While resurrecting from a dump an eln is mapped into memory if (at
least) a native compiled function noted in the dump was loaded from it.
So yeah the dumped state is preserved a no top level re-evaluation is
happening.

BTW I think so far we do not support re-dumping Emacs twice with native
code.  This is not a deep technical limitaiton, it's only because how we
fixup relative eln paths to support installed and non installed builds,
we should probably work on this limitation...

BR

  Andrea

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1398 bytes --]

On Mon, Jul 25, 2022, 4:11 PM Andrea Corallo <akrl@sdf.org> wrote:

> Lynn Winebarger <owinebar@gmail.com> writes:
>
>
>
> While resurrecting from a dump an eln is mapped into memory if (at
> least) a native compiled function noted in the dump was loaded from it.
> So yeah the dumped state is preserved a no top level re-evaluation is
> happening.
>
> BTW I think so far we do not support re-dumping Emacs twice with native
> code.  This is not a deep technical limitaiton, it's only because how we
> fixup relative eln paths to support installed and non installed builds,
> we should probably work on this limitation...


I still can't figure out why building with native compilation and a
site-load file was producing an error about trying to load an incoherent
ELN file.
Now that I have a shell script for running make to compile the included
files during the evaluation of site-init, I'm going to attempt that build
again.
One thing I might do is expose the "all_loaded_comp_units_h" variable in
lisp, so I can check whether all the ELN files were actually loaded and
compare that list against the one computed in loadup based on identifying
native subrs bound to function symbols.  That would provide a sanity check
during the dump.  It might be better to just explicitly base the mapping on
that hash table directly just to ensure no loaded compilation units are
missed while dumping.

Lynn

[-- Attachment #2: Type: text/html, Size: 2019 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

On Sun, Jul 31, 2022, 4:22 PM Lynn Winebarger <owinebar@gmail.com> wrote:

> On Mon, Jul 25, 2022, 4:11 PM Andrea Corallo <akrl@sdf.org> wrote:
>
>>
>> BTW I think so far we do not support re-dumping Emacs twice with native
>> code.  This is not a deep technical limitaiton, it's only because how we
>> fixup relative eln paths to support installed and non installed builds,
>> we should probably work on this limitation...
>
>
> I still can't figure out why building with native compilation and a
> site-load file was producing an error about trying to load an incoherent
> ELN file.
>

I found the problem - when I run make in the src directory, the variables
for the ELN and bin destinations are blank, which causes the code in loadup
that is supposed to make the compilation unit names into pairs to not
bother.
Maybe these two variables should get constructed in src/Makefile.in?
Running make from the src directory shouldn't result in an unusable dump
file.
I'd probably make the code in loadup either throw an error or come up with
some reasonable default.  If you insist on having the final installation
specified, then it should just error during the build instead of producing
a dump guaranteed to fail when loaded.  Otherwise, either use some dummy
values or just treat the build directory as the install directory.

Lynn

[-- Attachment #2: Type: text/html, Size: 2144 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Wed, 3 Aug 2022 05:58:17 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, Michael Albinus <michael.albinus@gmx.de>, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
> I found the problem - when I run make in the src directory, the variables for the ELN and bin destinations are
> blank, which causes the code in loadup that is supposed to make the compilation unit names into pairs to
> not bother.
> Maybe these two variables should get constructed in src/Makefile.in?  Running make from the src directory
> shouldn't result in an unusable dump file.  

Why are you running "make" only in 'src'?  Some necessary parts of the
build must also run "make" in 'lisp' and in other places.

In any case, these variables cannot be defined _only_ in src/Makefile,
but we could perhaps copy the definition from the top-level
Makefile.in to src/Makefile.in.

> I'd probably make the code in loadup either throw an error or come up with some reasonable default.  If you
> insist on having the final installation specified, then it should just error during the build instead of producing a
> dump guaranteed to fail when loaded.  Otherwise, either use some dummy values or just treat the build
> directory as the install directory.

The final installation _must_ be specified, yes.  Using the build
directory won't help because Emacs already does that internally.  IOW,
it only needs help in knowing where the stuff _will_ be installed.

Native-compilation build process (was Re: Loading tramp for dump goes into infinite regress)

[Lynn Winebarger]

On Wed, Aug 3, 2022 at 8:22 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Wed, 3 Aug 2022 05:58:17 -0400
> > Cc: Eli Zaretskii <eliz@gnu.org>, Michael Albinus <michael.albinus@gmx.de>,
> >       emacs-devel <emacs-devel@gnu.org>
> >
> > I found the problem - when I run make in the src directory, the variables for the ELN and bin destinations are
> > blank, which causes the code in loadup that is supposed to make the compilation unit names into pairs to
> > not bother.
> > Maybe these two variables should get constructed in src/Makefile.in?  Running make from the src directory
> > shouldn't result in an unusable dump file.
>
> Why are you running "make" only in 'src'?  Some necessary parts of the
> build must also run "make" in 'lisp' and in other places.

I'm replicating the 3 stage build process Stefan Monnier (added to the
cc list) described by invoking a shell script from site-load, when
dump-mode is set to pdump.  The shell script produces the second stage
build - the one with a dump containing just the files loaded directly
from loadup - then uses it to compile the files loaded by site-load.
It explicitly builds the emacs.pdmp target in src, moves emacs and
emacs.pdmp to emacs-1 and emacs-1.pdmp, then invokes:
make -C ../lisp EMACS=../src/emacs-1 compile-target
TARGETS="${SL_TARGETS}" BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq
comp-file-preloaded-p t)'"
where SL_TARGETS has been extracted from site-load.el using the same
procedure used in constructing lisp.mk

And in the process of getting this script right, I also sometimes
invoke make directly from src after bootstrapping has attempted to
create emacs.pdmp and failed.

>
> In any case, these variables cannot be defined _only_ in src/Makefile,
> but we could perhaps copy the definition from the top-level
> Makefile.in to src/Makefile.in.
>
Whatever you think is best.

> > I'd probably make the code in loadup either throw an error or come up with some reasonable default.  If you
> > insist on having the final installation specified, then it should just error during the build instead of producing a
> > dump guaranteed to fail when loaded.  Otherwise, either use some dummy values or just treat the build
> > directory as the install directory.
>
> The final installation _must_ be specified, yes.  Using the build
> directory won't help because Emacs already does that internally.  IOW,
> it only needs help in knowing where the stuff _will_ be installed

That is true for the dump that is going to be installed, yes, but for
intermediate stages in the build that knowledge is not required for
the resulting dump to be useful.
In any case, once loadup determines there are native compile units
loaded while preparing for the dump, either they all should have the
names fixed up, or an error should be signaled.  It would be way more
helpful to get an error during the dump procedure stating "The
--bin-dir and --eln-dir flags must be specified while dumping with
native-compilation units loaded", than to get "attempt to load an
incoherent eln" file later.  Because that will be the result.

It's not clear to me, though, why the pdmp loading process should
refuse to work if the compilation units recorded in the dump file are
just strings.  It seems to me the "dual location" is more of a "nice
to have" feature than a real requirement.  There could be a flag to
dump an image with only the (relative) final installation paths
recorded, or it could just be a build that will always be run in place
(e.g. an ordinary user build).   Or even if an ordinary user reruns
loadup in dump mode after emacs has been fully built and installed.
Why would the cons in the native compilation unit record be useful at
that point?   For user dumps, recording the absolute paths would
presumably work fine, no?

Lynn

Re: Native-compilation build process (was Re: Loading tramp for dump goes into infinite regress)

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Wed, 3 Aug 2022 10:53:12 -0400
> Cc: Andrea Corallo <akrl@sdf.org>, Michael Albinus <michael.albinus@gmx.de>, 
> 	emacs-devel <emacs-devel@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>
> 
> > Why are you running "make" only in 'src'?  Some necessary parts of the
> > build must also run "make" in 'lisp' and in other places.
> 
> I'm replicating the 3 stage build process Stefan Monnier (added to the
> cc list) described by invoking a shell script from site-load, when
> dump-mode is set to pdump.

Well, Stefan definitely didn't mean for you to shoot yourself in the
foot.  Isn't it possible to run these 2 stages from the top-level
Makefile?

Alternatively, say "make BINDESTDIR=..." etc.

Or, as I suggested, copy the default values of these two variables
from the top-level Makefile.in to src/Makefile.in.  Then it should
probably work from 'src' as well, provided that the other variables
these two reference are also defined there.

>                           The shell script produces the second stage
> build - the one with a dump containing just the files loaded directly
> from loadup - then uses it to compile the files loaded by site-load.
> It explicitly builds the emacs.pdmp target in src, moves emacs and
> emacs.pdmp to emacs-1 and emacs-1.pdmp, then invokes:
> make -C ../lisp EMACS=../src/emacs-1 compile-target
> TARGETS="${SL_TARGETS}" BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq
> comp-file-preloaded-p t)'"
> where SL_TARGETS has been extracted from site-load.el using the same
> procedure used in constructing lisp.mk

You could add BIN_DESTDIR= and ELN_DESTDIR= settings to those
commands.

> > The final installation _must_ be specified, yes.  Using the build
> > directory won't help because Emacs already does that internally.  IOW,
> > it only needs help in knowing where the stuff _will_ be installed
> 
> That is true for the dump that is going to be installed, yes, but for
> intermediate stages in the build that knowledge is not required for
> the resulting dump to be useful.

The Emacs build always produces binaries that can be invoked either
way.  So if you use our Makefiles, you need to play by their rules.

> In any case, once loadup determines there are native compile units
> loaded while preparing for the dump, either they all should have the
> names fixed up, or an error should be signaled.

The names cannot be fixed up, because the build doesn't know how.
These two variables get their values from the configure script, where
the user specifies where the package will be installed.  Without that,
the build cannot know their values.

> It's not clear to me, though, why the pdmp loading process should
> refuse to work if the compilation units recorded in the dump file are
> just strings.

Because it needs to record in the binary where to look for those *.eln
files.  Emacs can be installed in a variety of ways, using all kinds
of "tricks" like symlinks and such, so the names need to include
directories to allow Emacs to find these files quickly at startup.  If
startup is delayed because Emacs looks hight and low for its files,
users will be annoyed.

> It seems to me the "dual location" is more of a "nice
> to have" feature than a real requirement.

No, it's a requirement.  If nothing else, the just-built Emacs binary
is run as part of the build, so it needs to be able to do that.

> There could be a flag to
> dump an image with only the (relative) final installation paths
> recorded, or it could just be a build that will always be run in place
> (e.g. an ordinary user build).

Many users never install Emacs, or at least never install some of its
versions.

Anyway, this is a non-starter: we won't give up this useful
functionality.

> For user dumps, recording the absolute paths would presumably work
> fine, no?

No, because then you cannot relocate the Emacs tree to a different
top-level directory, and have it still work.

There are good reasons behind all these features, they aren't
arbitrary nor redundant.

Re: Native-compilation build process (was Re: Loading tramp for dump goes into infinite regress)

[Lynn Winebarger]

On Wed, Aug 3, 2022 at 12:15 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Wed, 3 Aug 2022 10:53:12 -0400
> > Cc: Andrea Corallo <akrl@sdf.org>, Michael Albinus <michael.albinus@gmx.de>,
> >       emacs-devel <emacs-devel@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>
> >
> > > Why are you running "make" only in 'src'?  Some necessary parts of the
> > > build must also run "make" in 'lisp' and in other places.
> >
> > I'm replicating the 3 stage build process Stefan Monnier (added to the
> > cc list) described by invoking a shell script from site-load, when
> > dump-mode is set to pdump.
>
> Well, Stefan definitely didn't mean for you to shoot yourself in the
> foot.  Isn't it possible to run these 2 stages from the top-level
> Makefile?
Not that I can tell - I don't  know what targets I would use from the
top level, but I do know that "emacs.pdmp" and "src/emacs.pdmp"
produce an "unknown target" response.
In any case, I  don't believe I have shot myself in the foot.  I was
just able to produce and use a dump with
* 698 libraries in the preloaded-file-list
    * 136 files listed in lisp.mk
    * 521 files listed in site-load.mk (the file I generate from site-load)
    * I'm not sure where the remainder are from
* 426 native compilation units in "all-loaded-comp-units-h"
* 1126 eln files under native-lisp/28.1-<hash>/preloaded - none appear
directly in 28.1-<hash>
* 1504 elc files under lisp
* 1557 el files under lisp
I'm not sure why some of the files were byte compiled but not native
compiled (or at least, not loaded in the dump).  But at the end of
site-load I loaded cus-start (with dump-mode set to nil), cus-load,
and finally cus-edit.  The result of running customize is promising in
terms of (lack of) lag, but I won't really know if it works until I've
added all the additional site-lisp files in the mix.
My solution was to allow the dump to contain a string with the
absolute path.  That works for my purposes.  When I have something I
want to install permanently, I'll specify the final location as well.
Or I might just create a dump only intended to be run from the
installed location.
I think my total modifications to the distribution source are in the
range of maybe 50 lines of C and a handful of changes of "defconst" to
"defvar" in the lisp source files.  Since one of the goals of this
particular exercise is to be able to minimize the local maintenance of
patches, I consider it to be a win.  Keep in mind I don't control the
rest of the build tools on this system, so autoconf etc may change
without my input.  That's why I rather not do anything that touches
the configure script(s) or Makefile.in.  I'd rather modify the shell
script to accomodate any changes in the process or set of targets
post-configure, and rely on the emacs maintainers expert knowledge of
the configuration process to deal with any changes in the build
toolset.

One change I had to make was to set native-comp-deferred-compilation to nil:
make -C ../lisp EMACS=../src/emacs-1 compile-target
TARGETS="${SL_TARGETS}" BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq
comp-file-preloaded-p t native-comp-deferred-compilation nil)'"
Otherwise I would end up with hundreds of lingering emacs processes
doing something with an argument that was something like
"subr-int-native-comp-<blah>-call-interactively-<hash>...".  The
recursively invoked make process for Stefan's "stage 2" would
nominally complete, but the resulting dump would have a random number
of native compilation units loaded.
Hopefully that's useful information for someone.

> > In any case, once loadup determines there are native compile units
> > loaded while preparing for the dump, either they all should have the
> > names fixed up, or an error should be signaled.
>
> The names cannot be fixed up, because the build doesn't know how.
> These two variables get their values from the configure script, where
> the user specifies where the package will be installed.  Without that,
> the build cannot know their values.

That's your call, I'm just pointing out above that it's usually better
to signal an error when it happens (not  supplying the arguments you
are stating as a requirement) than to signal an error over the
subsequent symptoms of the actual error (in this case, the attempt to
load the incoherent eln file).  I just don't understand why you would
ever want emacs to produce a dump file that will never be loadable
according to the requirements you are enforcing.  Are we actually in
disagreement on that?

> > There could be a flag to
> > dump an image with only the (relative) final installation paths
> > recorded, or it could just be a build that will always be run in place
> > (e.g. an ordinary user build).
>
> Many users never install Emacs, or at least never install some of its
> versions.

I don't doubt that, but this point is a little humorous given the
evident number of daily users of the latest commit in the master
trunk.

>
> Anyway, this is a non-starter: we won't give up this useful
> functionality.

I never said anything about giving it up.  That's distinct from not
making it a hard requirement.
I'm just clarifying the point I made. I don't expect your position to
change, but I'd prefer the rejection to be of what I actually
suggested.

Thanks,
Lynn

Re: Native-compilation build process (was Re: Loading tramp for dump goes into infinite regress)

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1343 bytes --]

On Wed, Aug 3, 2022, 11:33 PM Lynn Winebarger <owinebar@gmail.com> wrote:

>
> * 698 libraries in the preloaded-file-list
>     * 136 files listed in lisp.mk
>     * 521 files listed in site-load.mk (the file I generate from
> site-load)
>     * I'm not sure where the remainder are from
> * 426 native compilation units in "all-loaded-comp-units-h"
> * 1126 eln files under native-lisp/28.1-<hash>/preloaded - none appear
> directly in 28.1-<hash>
> * 1504 elc files under lisp
> * 1557 el files under lisp
> I'm not sure why some of the files were byte compiled but not native
> compiled (or at least, not loaded in the dump).


It was just from the way native compiled files are not tracked on the
build, and having done a pure byte-compile run to ensure everything would
have compiled dependencies.
Once touched all the source files and redumped, everything that could be
compiled was loaded as native compiled, for a total of 596 NCUs in the dump.

I will now proceed to iron out the load dependency order of the additional
libraries I have set up and compiled in site-lisp.

I found adding call-interactive to the "never-optimize" lists (one
customization variable, one hard coded list in nadvice) resolved the issue
with the runaway async subr compilation.  I deleted close to 100k ELISP
files from /tmp today from jobs I killed.

Lynn

[-- Attachment #2: Type: text/html, Size: 2197 bytes --]

Re: Loading tramp for dump goes into infinite regress

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Sun, 24 Jul 2022 12:00:42 -0400
> Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> 
> Additional note - if I delete the tramp-<hashes>.eln file from
> BUILD/native-lisp/28.1-<hash>/ without touching the corresponding elc
> file, the loading proceeds - although there are still issues to
> resolve in the subsequent tramp-X files being loaded.

If you remove the .eln file, Emacs will load the .elc file instead.
So I'm not really sure what does this factoid teach us.

Are you saying that if you try your "mega-dump" in a build without
native-compilation, it works without problems, and only the build with
native-compilation fails?