Regression in dump-emacs-portable

Regression in dump-emacs-portable

[Lynn Winebarger]

I mentioned redumping a maximal set of core emacs libraries under 28.2
in a recent thread.
When I attempted to build a similar dump for 30.0.50, the dump process
would terminate due to either an abort signal or encountering a
"weird" pseudovector.  For 30.0.50, I was able to get to a successful
redump by moving the files that "caused" these terminations from the
pre-load list to the list of files for which I performed bogus
"provide" prior to pre-dump file loading, then properly "require" in
the after-init-hook.  The same issues arose with 29.0.60, but the
fixes for 30.0.50 did not work for that build.
Details can be found in bug report
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=61447
I have also provided the redumping script that works for 28.2.

Lynn

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Sun, Feb 12, 2023 at 7:51 PM Lynn Winebarger <owinebar@gmail.com> wrote:
> the after-init-hook.  The same issues arose with 29.0.60, but the
> fixes for 30.0.50 did not work for that build.
After more carefully following the script I used in generating the
maximal redump of core emacs libraries for 28.2, I now have a redump
script that works for both 29.0.60 and 30.0.50, although I still have
no idea why the particular libraries (that dump without issue in 28.2)
cause failure in the later two versions.
I did notice that the dump file for 28.2 is 135M, and the dump file
for 30.0.50 and 29.0.60 is only 75M.  (!)  When I exclude the same
additional libraries from the 28.2 dump, the dump file is still 133M,
so the additional exclusions are not enough to explain the size
difference.
I also looked for tests of dump-emacs-portable in the test/
subdirectory, and grep reported nothing.  I can translate my shell
script into elisp to run under ERT, and add testing for each of the
libraries expected to be dumpable (in the sense of not requiring code
evaluated on load that needs to be delayed until run-time)
individually.  How do I submit that?  Are there any restrictions I
should observe?  I have a list of libraries I exclude due to platform
incompatibility with linux (ones with w32 and dos in the names), but I
don't have anything for other platforms.

Lynn

Re: Regression in dump-emacs-portable

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Mon, 13 Feb 2023 20:13:22 -0500
> 
> I also looked for tests of dump-emacs-portable in the test/
> subdirectory, and grep reported nothing.  I can translate my shell
> script into elisp to run under ERT, and add testing for each of the
> libraries expected to be dumpable (in the sense of not requiring code
> evaluated on load that needs to be delayed until run-time)
> individually.  How do I submit that?  Are there any restrictions I
> should observe?

What do these tests actually test?

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Tue, Feb 14, 2023 at 9:23 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Mon, 13 Feb 2023 20:13:22 -0500
> >
> > I also looked for tests of dump-emacs-portable in the test/
> > subdirectory, and grep reported nothing.  I can translate my shell
> > script into elisp to run under ERT, and add testing for each of the
> > libraries expected to be dumpable (in the sense of not requiring code
> > evaluated on load that needs to be delayed until run-time)
> > individually.  How do I submit that?  Are there any restrictions I
> > should observe?
>
> What do these tests actually test?

Whether libraries expected to be redumpable are in fact redumpable.
Almost every library in 28.2 could be redumped, excepting those which
simply failed to load for whatever reason.  Even the ones that need to
be delayed until initialization for practical reasons do not cause
dump-emacs-portable to fail.  A library changing from dumpable to
non-dumpable might be acceptable, but that change has to be known to
evaluate the tradeoff.  Additionally, whether the maximal set of files
expected to be redumpable is redumpable as a whole.  If the maximal
set can be redumped, then presumably any proper subset (closed under
require) could be as well.

Obviously someone's been optimizing something to see the maximal
redump size decrease by 45%.  The total number of libraries (excluding
the obsolete directory) appears to have decreased, but enough to
explain that much of a change in size?

Lynn

Re: Regression in dump-emacs-portable

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Tue, 14 Feb 2023 18:26:07 -0500
> Cc: emacs-devel@gnu.org
> 
> On Tue, Feb 14, 2023 at 9:23 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > What do these tests actually test?
> 
> Whether libraries expected to be redumpable are in fact redumpable.

That's the goal, not the actual testing algorithm.  I asked about the
latter.  How do you intend to test that a dump succeeded (assuming
there's no crash)?

> Almost every library in 28.2 could be redumped, excepting those which
> simply failed to load for whatever reason.

Don't we have Lisp objects that cannot be dumped?  If we do, then not
every library could be dumped even in principle.

Another potential issue with this is (assuming you suggest to actually
try dumping every library) that it will take too long, and thus will
be likely to be skipped in any "normal" run of the test suite, thus
missing the point.

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 4837 bytes --]

On Wed, Feb 15, 2023, 7:43 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Tue, 14 Feb 2023 18:26:07 -0500
> > Cc: emacs-devel@gnu.org
> >
> > On Tue, Feb 14, 2023 at 9:23 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > What do these tests actually test?
> >
> > Whether libraries expected to be redumpable are in fact redumpable.
>
> That's the goal, not the actual testing algorithm.  I asked about the
> latter.  How do you intend to test that a dump succeeded (assuming
> there's no crash)?
>

The dump will have to be performed in a separate emacs process. The easiest
criteria to judge is whether the dump file exists and is greater than 0
bytes.  Emacs appears to create a 0 byte file when dump-emacs-portable is
invoked, which is just not updated if the dump terminates unsuccessfully.
A second criteria is then to invoke emacs with the dump-file and evaluate
some simple expression to verify no unexpected  errors were encountered on
load.
I've started automating my process with some simple shell scripting tracked
at https://github.com/owinebar/emacs-redumping.  It's not much yet, but at
least I was able to align my efforts between 28.2 and 30.0.50.  The next
step will be to create a proper load-time dependency graph, so I can
automate the calculation of the minimal list of features that need to be
provided so that the maximum number of libraries can be loaded for the
dump, with the artificially provided features loaded on an after-init hook
(because before-init happens prior to the X frame initialization).
Once these dependencies are identified and lists are calculated, then
creating a set of canned tests should be straightforward.  Some
makefile-based approach should be adequate for determining which parts of
the dependency graph need to be recalculated after an update.
I want to calculate these dependencies (and compile-time dependencies) to
construct a more robust native-compilation build process anyway.
For a regression test, I would want to record the results from 28.2 as a
basis for measuring 29 and 30, at least as a starting point.   In any case,
I never see an "abort signal" termination in 28, or even a "weird
pseudovector" message.  It's either something incompatible (because I
blindly attempted to load the world) as in the "term" subdirectory or
dos/w32 libraries under linux, or some redefinition of a character table
(which is why I calculate the files loaded in the baseline dump and exclude
them).  I got some very lengthy error messages printing out explicit
objects from some obsolete libraries, so I exclude them as well.
And viper demands user input at startup when it's loaded, so it has to be
excluded from dumping.  There might be some variable to turn off that
annoying behavior, I'm just not interested in investigating.

> Almost every library in 28.2 could be redumped, excepting those which
> > simply failed to load for whatever reason.
>
> Don't we have Lisp objects that cannot be dumped?  If we do, then not
> every library could be dumped even in principle.
>

In 28.2, using dump-emacs-portable, the answer is, not many in the
libraries in included in the Emacs source distribution.  I excluded the
term and obsolete subdirectories from generating the set of libraries to
dump (but not from the final set determined from load-history).  Even
outside of the emacs distribution, the only problematic objects are dynamic
modules.  I assume this is due to dumping in batch-mode.  My exclusion on
wid-edit.el is because dumping it in batch-mode appears to bar it from ever
subsequently creating proper buttons in a graphic terminal.   But dumping
it still succeeds.

Another potential issue with this is (assuming you suggest to actually
> try dumping every library) that it will take too long, and thus will
> be likely to be skipped in any "normal" run of the test suite, thus
> missing the point.
>
My 2017-vintage laptop dumps the 1252 files, including all of leim, in 34
seconds, for a 135MB dump file.
When I added leim to the exclusions list,  1172 libraries are dumped in 24
seconds for a 83MB dump file, which explains why my effort with 30.0.50
produces a 75MB dump.  I excluded leim for 30.0.50 because I was
encountering too many errors to deal with manually, which explains most of
the size reduction.
I'm not sure how the tests are normally run, but I would think anyone
working on pdumper should be interested in a comprehensive test at some
point.  Aside from testing on a per-commit basis, isn't there a more
comprehensive set of regression tests run pre-release? Does emacs have a CI
process regularly running the test suite, or is it more ad hoc?  If nothing
else, failures reported from such a routine run could be used to create a
more targeted test set for someone actively working on pdumper.

Lynn

[-- Attachment #2: Type: text/html, Size: 6441 bytes --]

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Thu, Feb 16, 2023 at 4:31 AM Lynn Winebarger <owinebar@gmail.com> wrote:
> On Wed, Feb 15, 2023, 7:43 AM Eli Zaretskii <eliz@gnu.org> wrote:
>> > From: Lynn Winebarger <owinebar@gmail.com>
>> > Date: Tue, 14 Feb 2023 18:26:07 -0500
>> > Cc: emacs-devel@gnu.org
>> >
>> > On Tue, Feb 14, 2023 at 9:23 AM Eli Zaretskii <eliz@gnu.org> wrote:
>> > >
>> > > What do these tests actually test?
>> >
>> > Whether libraries expected to be redumpable are in fact redumpable.
>>
>> That's the goal, not the actual testing algorithm.  I asked about the
>> latter.  How do you intend to test that a dump succeeded (assuming
>> there's no crash)?
>
>
> The dump will have to be performed in a separate emacs process. The easiest criteria to judge is whether the dump file exists and is greater than 0 bytes.  Emacs appears to create a 0 byte file when dump-emacs-portable is invoked, which is just not updated if the dump terminates unsuccessfully.
> A second criteria is then to invoke emacs with the dump-file and evaluate some simple expression to verify no unexpected  errors were encountered on load.
> I've started automating my process with some simple shell scripting tracked at https://github.com/owinebar/emacs-redumping.  It's not much yet, but at least I was able to align my efforts between 28.2 and 30.0.50.  The next step will be to create a proper load-time dependency graph, so I can automate the calculation of the minimal list of features that need to be provided so that the maximum number of libraries can be loaded for the dump, with the artificially provided features loaded on an after-init hook (because before-init happens prior to the X frame initialization).
> Once these dependencies are identified and lists are calculated, then creating a set of canned tests should be straightforward.  Some makefile-based approach should be adequate for determining which parts of the dependency graph need to be recalculated after an update.
> I want to calculate these dependencies (and compile-time dependencies) to construct a more robust native-compilation build process anyway.
> For a regression test, I would want to record the results from 28.2 as a basis for measuring 29 and 30, at least as a starting point.   In any case, I never see an "abort signal" termination in 28, or even a "weird pseudovector" message.  It's either something incompatible (because I blindly attempted to load the world) as in the "term" subdirectory or dos/w32 libraries under linux, or some redefinition of a character table (which is why I calculate the files loaded in the baseline dump and exclude them).  I got some very lengthy error messages printing out explicit objects from some obsolete libraries, so I exclude them as well.
> And viper demands user input at startup when it's loaded, so it has to be excluded from dumping.  There might be some variable to turn off that annoying behavior, I'm just not interested in investigating.
>
>> > Almost every library in 28.2 could be redumped, excepting those which
>> > simply failed to load for whatever reason.
>>
>> Don't we have Lisp objects that cannot be dumped?  If we do, then not
>> every library could be dumped even in principle.
>
>
> In 28.2, using dump-emacs-portable, the answer is, not many in the libraries in included in the Emacs source distribution.  I excluded the term and obsolete subdirectories from generating the set of libraries to dump (but not from the final set determined from load-history).  Even outside of the emacs distribution, the only problematic objects are dynamic modules.  I assume this is due to dumping in batch-mode.  My exclusion on wid-edit.el is because dumping it in batch-mode appears to bar it from ever subsequently creating proper buttons in a graphic terminal.   But dumping it still succeeds.
>
>> Another potential issue with this is (assuming you suggest to actually
>> try dumping every library) that it will take too long, and thus will
>> be likely to be skipped in any "normal" run of the test suite, thus
>> missing the point.
>
> My 2017-vintage laptop dumps the 1252 files, including all of leim, in 34 seconds, for a 135MB dump file.
> When I added leim to the exclusions list,  1172 libraries are dumped in 24 seconds for a 83MB dump file, which explains why my effort with 30.0.50 produces a 75MB dump.  I excluded leim for 30.0.50 because I was encountering too many errors to deal with manually, which explains most of the size reduction.
> I'm not sure how the tests are normally run, but I would think anyone working on pdumper should be interested in a comprehensive test at some point.  Aside from testing on a per-commit basis, isn't there a more comprehensive set of regression tests run pre-release? Does emacs have a CI process regularly running the test suite, or is it more ad hoc?  If nothing else, failures reported from such a routine run could be used to create a more targeted test set for someone actively working on pdumper.

Just to finish this thought - dumping the full set of libraries,
excluding a few expected to fail, should be the "normal" test.  If
24-34 seconds is too long, there are probably other large subsets that
provide substantial coverage in less time. The more comprehensive
file-by-file approach should be reserved for tracking down the cause
of failures in the normal test.  Theoretically, pdumper might be able
to indicate the source library(ies) associated with particular error,
but the file-by-file approach is always available.

Lynn

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 6631 bytes --]

On Thu, Feb 16, 2023, 4:54 AM Lynn Winebarger <owinebar@gmail.com> wrote:

> On Thu, Feb 16, 2023 at 4:31 AM Lynn Winebarger <owinebar@gmail.com>
> wrote:
> > On Wed, Feb 15, 2023, 7:43 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >> > From: Lynn Winebarger <owinebar@gmail.com>
> >> > Date: Tue, 14 Feb 2023 18:26:07 -0500
> >> > Cc: emacs-devel@gnu.org
> >> >
> >> > On Tue, Feb 14, 2023 at 9:23 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >> > >
> >> > > What do these tests actually test?
> >> >
> >> > Whether libraries expected to be redumpable are in fact redumpable.
> >>
> >> That's the goal, not the actual testing algorithm.  I asked about the
> >> latter.  How do you intend to test that a dump succeeded (assuming
> >> there's no crash)?
> >
> >
> > The dump will have to be performed in a separate emacs process. The
> easiest criteria to judge is whether the dump file exists and is greater
> than 0 bytes.  Emacs appears to create a 0 byte file when
> dump-emacs-portable is invoked, which is just not updated if the dump
> terminates unsuccessfully.
> > A second criteria is then to invoke emacs with the dump-file and
> evaluate some simple expression to verify no unexpected  errors were
> encountered on load.
> > I've started automating my process with some simple shell scripting
> tracked at https://github.com/owinebar/emacs-redumping.  It's not much
> yet, but at least I was able to align my efforts between 28.2 and 30.0.50.
> The next step will be to create a proper load-time dependency graph, so I
> can automate the calculation of the minimal list of features that need to
> be provided so that the maximum number of libraries can be loaded for the
> dump, with the artificially provided features loaded on an after-init hook
> (because before-init happens prior to the X frame initialization).
> > Once these dependencies are identified and lists are calculated, then
> creating a set of canned tests should be straightforward.  Some
> makefile-based approach should be adequate for determining which parts of
> the dependency graph need to be recalculated after an update.
> > I want to calculate these dependencies (and compile-time dependencies)
> to construct a more robust native-compilation build process anyway.
> > For a regression test, I would want to record the results from 28.2 as a
> basis for measuring 29 and 30, at least as a starting point.   In any case,
> I never see an "abort signal" termination in 28, or even a "weird
> pseudovector" message.  It's either something incompatible (because I
> blindly attempted to load the world) as in the "term" subdirectory or
> dos/w32 libraries under linux, or some redefinition of a character table
> (which is why I calculate the files loaded in the baseline dump and exclude
> them).  I got some very lengthy error messages printing out explicit
> objects from some obsolete libraries, so I exclude them as well.
> > And viper demands user input at startup when it's loaded, so it has to
> be excluded from dumping.  There might be some variable to turn off that
> annoying behavior, I'm just not interested in investigating.
> >
> >> > Almost every library in 28.2 could be redumped, excepting those which
> >> > simply failed to load for whatever reason.
> >>
> >> Don't we have Lisp objects that cannot be dumped?  If we do, then not
> >> every library could be dumped even in principle.
> >
> >
> > In 28.2, using dump-emacs-portable, the answer is, not many in the
> libraries in included in the Emacs source distribution.  I excluded the
> term and obsolete subdirectories from generating the set of libraries to
> dump (but not from the final set determined from load-history).  Even
> outside of the emacs distribution, the only problematic objects are dynamic
> modules.  I assume this is due to dumping in batch-mode.  My exclusion on
> wid-edit.el is because dumping it in batch-mode appears to bar it from ever
> subsequently creating proper buttons in a graphic terminal.   But dumping
> it still succeeds.
> >
> >> Another potential issue with this is (assuming you suggest to actually
> >> try dumping every library) that it will take too long, and thus will
> >> be likely to be skipped in any "normal" run of the test suite, thus
> >> missing the point.
> >
> > My 2017-vintage laptop dumps the 1252 files, including all of leim, in
> 34 seconds, for a 135MB dump file.
> > When I added leim to the exclusions list,  1172 libraries are dumped in
> 24 seconds for a 83MB dump file, which explains why my effort with 30.0.50
> produces a 75MB dump.  I excluded leim for 30.0.50 because I was
> encountering too many errors to deal with manually, which explains most of
> the size reduction.
> > I'm not sure how the tests are normally run, but I would think anyone
> working on pdumper should be interested in a comprehensive test at some
> point.  Aside from testing on a per-commit basis, isn't there a more
> comprehensive set of regression tests run pre-release? Does emacs have a CI
> process regularly running the test suite, or is it more ad hoc?  If nothing
> else, failures reported from such a routine run could be used to create a
> more targeted test set for someone actively working on pdumper.
>
> Just to finish this thought - dumping the full set of libraries,
> excluding a few expected to fail, should be the "normal" test.  If
> 24-34 seconds is too long, there are probably other large subsets that
> provide substantial coverage in less time. The more comprehensive
> file-by-file approach should be reserved for tracking down the cause
> of failures in the normal test.  Theoretically, pdumper might be able
> to indicate the source library(ies) associated with particular error,
> but the file-by-file approach is always available.
>

I do see something in the redumped emacs that seems like a bug to me.  The
process I use for creating the dump uses the -Q flag.  But some of the
settings I see in "emacs -Q --dump-file ..." are not the ones I see with
just "emacs -Q".  Some are pretty basic - menu-bar-mode, tool-bar-mode,
global-font-lock-mode, transient-mark-mode are all nil in the redumped
process but not the baseline.  I should check if that happens even when the
only expression evaluated is the call to dump-emacs-portable, with nothing
additional loaded.
A general test would be load the additional files to be dumped, write out
all the symbol properties, variable values, and function values, then load
the dump file and compare everything with equal, including the set of
symbols, variables and functions defined.
Is it fair to say that is the correct expectation of the dumping procedure?

[-- Attachment #2: Type: text/html, Size: 7991 bytes --]

Re: Regression in dump-emacs-portable

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Thu, 16 Feb 2023 10:05:00 -0500
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> I do see something in the redumped emacs that seems like a bug to me.  The process I use for creating the
> dump uses the -Q flag.  But some of the settings I see in "emacs -Q --dump-file ..." are not the ones I see
> with just "emacs -Q".  Some are pretty basic - menu-bar-mode, tool-bar-mode, global-font-lock-mode,
> transient-mark-mode are all nil in the redumped process but not the baseline.

That is exactly the problem with re-dumping: stuff that was
initialized on the first start gets dumped, and then works differently
when Emacs is restarted from the second dump.

It might sound like a minor issue, but it is actually a tip of a very
large iceberg.  When we run temacs to produce the dump file, we jump
through many hoops to avoid these problems, and then when dumped Emacs
is started, it jumps through some more hoops.  Those are all the
init_FOO_once and pdumper_do_now_and_after_load calls.  All this works
differently when Emacs starts from a re-dumped dump file.  For
example, features that are toggled might be turned off in the second
start because the startup code assumes they are originally turned off
and toggling them turns them on.

Figuring this out is the main part of the job of enabling re-dumping.

> A general test would be load the additional files to be dumped, write out all the symbol properties, variable
> values, and function values, then load the dump file and compare everything with equal, including the set of
> symbols, variables and functions defined.
> Is it fair to say that is the correct expectation of the dumping procedure?

I don't think this will work, because at least the defcustoms are
evaluated upon each startup, and a defcustom can have a setter
function.  And there are probably more features which will get in the
way.

Re: Regression in dump-emacs-portable

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Thu, 16 Feb 2023 04:31:10 -0500
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
>  > Almost every library in 28.2 could be redumped, excepting those which
>  > simply failed to load for whatever reason.
> 
>  Don't we have Lisp objects that cannot be dumped?  If we do, then not
>  every library could be dumped even in principle.
> 
> In 28.2, using dump-emacs-portable, the answer is, not many in the libraries in included in the Emacs
> source distribution.  I excluded the term and obsolete subdirectories from generating the set of libraries to
> dump (but not from the final set determined from load-history).

Is exclusion really the way to go, if we want eventually to support
re-dumping at any given moment?

>  Another potential issue with this is (assuming you suggest to actually
>  try dumping every library) that it will take too long, and thus will
>  be likely to be skipped in any "normal" run of the test suite, thus
>  missing the point.
> 
> My 2017-vintage laptop dumps the 1252 files, including all of leim, in 34 seconds, for a 135MB dump file.
> When I added leim to the exclusions list,  1172 libraries are dumped in 24 seconds for a 83MB dump file,
> which explains why my effort with 30.0.50 produces a 75MB dump.

That's a lot.  And I presume your build is optimized?  So an
unoptimized build will be slower.  Running such slow tests routinely
is a PITA, so such a test will probably moved to a category that
doesn't run by default.

> Aside from testing on a per-commit basis, isn't there a
> more comprehensive set of regression tests run pre-release? Does emacs have a CI process regularly
> running the test suite, or is it more ad hoc?

There's EMBA.

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Thu, Feb 16, 2023 at 10:34 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Thu, 16 Feb 2023 10:05:00 -0500
> > Cc: emacs-devel <emacs-devel@gnu.org>
> >
> > I do see something in the redumped emacs that seems like a bug to me.  The process I use for creating the
> > dump uses the -Q flag.  But some of the settings I see in "emacs -Q --dump-file ..." are not the ones I see
> > with just "emacs -Q".  Some are pretty basic - menu-bar-mode, tool-bar-mode, global-font-lock-mode,
> > transient-mark-mode are all nil in the redumped process but not the baseline.
>
> That is exactly the problem with re-dumping: stuff that was
> initialized on the first start gets dumped, and then works differently
> when Emacs is restarted from the second dump.

That appears to be a consequence of setting
custom-delayed-init-variables to t in startup, without saving a copy
to be restored by an after-pdump-load-hook.  Those modes have an
init-value expression involving "(not noninteractive)"  that is
evaluated during batch mode startup but not in the post-redump
startup.

> Figuring this out is the main part of the job of enabling re-dumping.
>
re-dumping in 28.2 works well enough to be useful.  At least I can
explicitly set the custom-delayed-init-variables by setting a hook,
and various kludges to make things work.
 The dump procedure just failing in 29/30 is much more problematic for
me.  I only mentioned the failures with the libraries included in
emacs, but libraries from packages that previously dumped just fine
also fail.  Starting up with 2200 loaded libraries is much faster
after redumping in 28.2.

> > A general test would be load the additional files to be dumped, write out all the symbol properties, variable
> > values, and function values, then load the dump file and compare everything with equal, including the set of
> > symbols, variables and functions defined.
> > Is it fair to say that is the correct expectation of the dumping procedure?
>
> I don't think this will work, because at least the defcustoms are
> evaluated upon each startup, and a defcustom can have a setter
> function.  And there are probably more features which will get in the
> way.
I suppose there are two versions of this test.  One is, what is the
correct behavior, and has it been achieved?  The regression version
is,  is the current behavior no more incorrect than the reference
version, i.e. correct for all elements of the lisp heap for which the
reference version is correct?

Lynn

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Thu, Feb 16, 2023 at 10:46 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >  > Almost every library in 28.2 could be redumped, excepting those which
> >  > simply failed to load for whatever reason.
> >
> >  Don't we have Lisp objects that cannot be dumped?  If we do, then not
> >  every library could be dumped even in principle.
> >
> > In 28.2, using dump-emacs-portable, the answer is, not many in the libraries in included in the Emacs
> > source distribution.  I excluded the term and obsolete subdirectories from generating the set of libraries to
> > dump (but not from the final set determined from load-history).
>
> Is exclusion really the way to go, if we want eventually to support
> re-dumping at any given moment?

At the moment, my concern is just with keeping the functionality 28.2
has for redumping, unless there is some deliberate tradeoff being
made.
I'm just using exclusion until I can automate a more precise
determination of what is dumpable and what isn't.  Also, some of those
libraries may be included in the dump - I'm only excluding them from
the set used to generate the list of libraries to include in the dump.
I probably could have narrowed down the problematic libraries, but I
didn't want to spend a lot of time manually investigating why
libraries classified as obsolete were causing problems.  On the term
side, it seemed that the ones causing issues were either due to
incompatibility (win32 on linux, say) or perhaps required some feature
not available in batch mode, so just attempting to load them caused
problems.  More precise lists of expected failures can be created.

>
> >  Another potential issue with this is (assuming you suggest to actually
> >  try dumping every library) that it will take too long, and thus will
> >  be likely to be skipped in any "normal" run of the test suite, thus
> >  missing the point.
> >
> > My 2017-vintage laptop dumps the 1252 files, including all of leim, in 34 seconds, for a 135MB dump file.
> > When I added leim to the exclusions list,  1172 libraries are dumped in 24 seconds for a 83MB dump file,
> > which explains why my effort with 30.0.50 produces a 75MB dump.
>
> That's a lot.  And I presume your build is optimized?  So an
> unoptimized build will be slower.  Running such slow tests routinely
> is a PITA, so such a test will probably moved to a category that
> doesn't run by default.

It's fine with me if the tests aren't run directly by the developer,
unless they're particularly concerned about it.  I'd expect pdumper
functionality to require testing on multiple platforms anyway, not
just whatever system the developer is using.
I can pick some that are causing failures now, but I have no idea why
they stress the current implementation.  icomplete causes an abort in
131ms.  sh-script generates a "weird pseudovector" message in 376ms
> time emacs -Q -batch --eval '(load "icomplete")' --eval '(dump-emacs-portable "test-icomplete.pdmp")'
Loading icomplete...
Dumping fingerprint:
30c4a01f07c30c32913a3c578aa6871d7af188212b14e2f9c5e3717f80b9ab43
Fatal error 6: Aborted
Backtrace:
emacs(emacs_backtrace+0x37)[0x55ec89e6de67]
emacs(terminate_due_to_signal+0x6f)[0x55ec89d33514]
emacs(nsberror+0x0)[0x55ec89d33726]
emacs(+0x21f9da)[0x55ec89ebb9da]
emacs(+0x21efc7)[0x55ec89ebafc7]
emacs(+0x2203a1)[0x55ec89ebc3a1]
emacs(Fdump_emacs_portable+0xb70)[0x55ec89ebd070]
emacs(eval_sub+0x9b3)[0x55ec89edc1d3]
emacs(Feval+0x3f)[0x55ec89ede22f]
emacs(exec_byte_code+0x482)[0x55ec89f1e182]
emacs(+0x2412fc)[0x55ec89edd2fc]
emacs(eval_sub+0x2bc)[0x55ec89edbadc]
emacs(Feval+0x3f)[0x55ec89ede22f]
emacs(internal_condition_case+0x3e)[0x55ec89ed6c7e]
emacs(+0x1b4618)[0x55ec89e50618]
emacs(internal_catch+0x3d)[0x55ec89ed6bed]
emacs(+0x1b2998)[0x55ec89e4e998]
emacs(recursive_edit_1+0x86)[0x55ec89e54d56]
emacs(Frecursive_edit+0xd0)[0x55ec89e550f0]
emacs(main+0x190f)[0x55ec89d35c5f]
/lib64/libc.so.6(__libc_start_main+0xef)[0x7fb49dd4129d]
emacs(_start+0x2a)[0x55ec89d369ea]
Aborted (core dumped)

real 0m0.131s
user 0m0.102s
sys 0m0.029s
===================================
> time emacs -Q -batch --eval '(load "sh-script")' --eval '(dump-emacs-portable "test-sh-script.pdmp")'
Loading sh-script...
Dumping fingerprint:
30c4a01f07c30c32913a3c578aa6871d7af188212b14e2f9c5e3717f80b9ab43

Error: error ("unsupported object type in dump: weird pseudovector")
  mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode
-0x7c3fa1856517817>))
  debug-early-backtrace()
  debug-early(error (error "unsupported object type in dump: weird
pseudovector"))
  dump-emacs-portable("test-sh-script.pdmp")
  eval((dump-emacs-portable "test-sh-script.pdmp") t)
  command-line-1(("--eval" "(load \"sh-script\")" "--eval"
"(dump-emacs-portable \"test-sh-script.pdmp\")"))
  command-line()
  normal-top-level()
unsupported object type in dump: weird pseudovector

real 0m0.376s
user 0m0.172s
sys 0m0.040s
===================================

> > Aside from testing on a per-commit basis, isn't there a
> > more comprehensive set of regression tests run pre-release? Does emacs have a CI process regularly
> > running the test suite, or is it more ad hoc?
>
> There's EMBA.

Ok, that's the kind of thing I was thinking of. I'll read over the
relevant admin/notes before asking more questions.
I'm going to put some more work into automating generation of
comprehensive redumping scripts this weekend.  Once I have that I can
look into how to use that process to generate regression tests for
specified sets of libraries to redump.

Thanks,
Lynn

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Thu, Feb 16, 2023 at 8:29 PM Lynn Winebarger <owinebar@gmail.com> wrote:
> > time emacs -Q -batch --eval '(load "icomplete")' --eval '(dump-emacs-portable "test-icomplete.pdmp")'
> Loading icomplete...

I was able to cut this down to a single expression that causes the core dump:
=====> icomplete-test.el <==============
(defvar icomplete-overlay (make-overlay (point-min) (point-min) nil t t)
  "Overlay used to display the list of completions.")
===============================
 time emacs -Q -batch --eval '(load "icomplete-test")' --eval
'(dump-emacs-portable "test.pdmp")'
Loading icomplete-test.el (source)...
Dumping fingerprint:
30c4a01f07c30c32913a3c578aa6871d7af188212b14e2f9c5e3717f80b9ab43
Fatal error 6: Aborted
Backtrace:
emacs(emacs_backtrace+0x37)[0x55d13ee16e67]
emacs(terminate_due_to_signal+0x6f)[0x55d13ecdc514]
emacs(nsberror+0x0)[0x55d13ecdc726]
emacs(+0x21f9da)[0x55d13ee649da]
emacs(+0x21efc7)[0x55d13ee63fc7]
emacs(+0x2203a1)[0x55d13ee653a1]
emacs(Fdump_emacs_portable+0xb70)[0x55d13ee66070]
emacs(eval_sub+0x9b3)[0x55d13ee851d3]
emacs(Feval+0x3f)[0x55d13ee8722f]
emacs(exec_byte_code+0x482)[0x55d13eec7182]
emacs(+0x2412fc)[0x55d13ee862fc]
emacs(eval_sub+0x2bc)[0x55d13ee84adc]
emacs(Feval+0x3f)[0x55d13ee8722f]
emacs(internal_condition_case+0x3e)[0x55d13ee7fc7e]
emacs(+0x1b4618)[0x55d13edf9618]
emacs(internal_catch+0x3d)[0x55d13ee7fbed]
emacs(+0x1b2998)[0x55d13edf7998]
emacs(recursive_edit_1+0x86)[0x55d13edfdd56]
emacs(Frecursive_edit+0xd0)[0x55d13edfe0f0]
emacs(main+0x190f)[0x55d13ecdec5f]
/lib64/libc.so.6(__libc_start_main+0xef)[0x7fda0df8529d]
emacs(_start+0x2a)[0x55d13ecdf9ea]
Aborted (core dumped)

real 0m0.137s
user 0m0.096s
sys 0m0.041s

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Thu, Feb 16, 2023 at 8:29 PM Lynn Winebarger <owinebar@gmail.com> wrote:
>  sh-script generates a "weird pseudovector" message in 376ms
> > time emacs -Q -batch --eval '(load "sh-script")' --eval '(dump-emacs-portable "test-sh-script.pdmp")'
This seems to be related to whatever object "treesit-font-lock-rules" creates.
The moral seems to be to create tests of each type of object a
user-visible lisp function might create to get quick test cases.
=============>sh-script-test<======================
(require 'treesit)

(defvar font-lock-comment-face)
(defvar sh-mode--treesit-operators
  '("|" "|&" "||" "&&" ">" ">>" "<" "<<" "<<-" "<<<" "==" "!=" ";&" ";;&")
  "A list of `sh-mode' operators to fontify.")

(defvar sh-mode--treesit-keywords
  '("case" "do" "done" "elif" "else" "esac" "export" "fi" "for"
    "function" "if" "in" "unset" "while" "then")
  "Minimal list of keywords that belong to tree-sitter-bash's grammar.

Some reserved words are not recognize to keep the grammar
simpler.  Those are identified with regex-based filtered queries.

\(See `sh-mode--treesit-other-keywords' and
`sh-mode--treesit-settings').")

(defvar sh-mode--treesit-settings
  (treesit-font-lock-rules
   :feature 'comment
   :language 'bash
   '((comment) @font-lock-comment-face))
  "Tree-sitter font-lock settings for `sh-mode'.")
======================================================================
Loading /home/owinebar/sh-script-test.el (source)...
Dumping fingerprint:
30c4a01f07c30c32913a3c578aa6871d7af188212b14e2f9c5e3717f80b9ab43

Error: error ("unsupported object type in dump: weird pseudovector")
  mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode
-0x13b91d456517831>))
  debug-early-backtrace()
  debug-early(error (error "unsupported object type in dump: weird
pseudovector"))
  dump-emacs-portable("test.pdmp")
  eval((dump-emacs-portable "test.pdmp") t)
  command-line-1(("--eval" "(load \"~/sh-script-test\")" "--eval"
"(dump-emacs-portable \"test.pdmp\")"))
  command-line()
  normal-top-level()
unsupported object type in dump: weird pseudovector

real 0m0.415s
user 0m0.343s
sys 0m0.068s

Re: Regression in dump-emacs-portable

[Po Lu]

Lynn Winebarger <owinebar@gmail.com> writes:

> This seems to be related to whatever object "treesit-font-lock-rules" creates.
> The moral seems to be to create tests of each type of object a
> user-visible lisp function might create to get quick test cases.

    case PVEC_CONDVAR:
      error_unsupported_dump_object (ctx, lv, "condvar");
    case PVEC_SQLITE:
      error_unsupported_dump_object (ctx, lv, "sqlite");
    case PVEC_MODULE_FUNCTION:
      error_unsupported_dump_object (ctx, lv, "module function");
    case PVEC_SYMBOL_WITH_POS:
      error_unsupported_dump_object (ctx, lv, "symbol with pos");
    default:

Please add the all the tree-sitter pvec types here, and see which one it
is.

Thanks.

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Fri, Feb 17, 2023 at 12:23 AM Po Lu <luangruo@yahoo.com> wrote:
>
> Lynn Winebarger <owinebar@gmail.com> writes:
>
> > This seems to be related to whatever object "treesit-font-lock-rules" creates.
> > The moral seems to be to create tests of each type of object a
> > user-visible lisp function might create to get quick test cases.
>
>     case PVEC_CONDVAR:
>       error_unsupported_dump_object (ctx, lv, "condvar");
>     case PVEC_SQLITE:
>       error_unsupported_dump_object (ctx, lv, "sqlite");
>     case PVEC_MODULE_FUNCTION:
>       error_unsupported_dump_object (ctx, lv, "module function");
>     case PVEC_SYMBOL_WITH_POS:
>       error_unsupported_dump_object (ctx, lv, "symbol with pos");
>     default:
>
> Please add the all the tree-sitter pvec types here, and see which one it
> is.
I just pretty-printed the value of the variable this sets instead:
(pp sh-mode--treesit-settings (current-buffer))
((#<treesit-compiled-query> t comment nil)
 (#<treesit-compiled-query> t function nil)
 (#<treesit-compiled-query> t string nil)
 (#<treesit-compiled-query> t heredoc nil)
 (#<treesit-compiled-query> t variable nil)
 (#<treesit-compiled-query> t keyword nil)
 (#<treesit-compiled-query> t command nil)
 (#<treesit-compiled-query> t declaration-command nil)
 (#<treesit-compiled-query> t constant nil)
 (#<treesit-compiled-query> t operator nil)
 (#<treesit-compiled-query> t builtin-variable nil)
 (#<treesit-compiled-query> t number nil)
 (#<treesit-compiled-query> t bracket nil)
 (#<treesit-compiled-query> t delimiter nil)
 (#<treesit-compiled-query> t misc-punctuation nil))

I'm guessing it's whatever #<treeset-compiled-query> represents.
The core dump from pdumping an overlay is a bigger problem to me.
Worst case, I can disable tree-sitter when running ./configure, which
is not an option for overlays.  Also, core dump from running a
function intended for end-users.

Lynn

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Thu, Feb 16, 2023 at 6:45 PM Lynn Winebarger <owinebar@gmail.com> wrote:
>
> On Thu, Feb 16, 2023 at 10:34 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: Lynn Winebarger <owinebar@gmail.com>
> > > Date: Thu, 16 Feb 2023 10:05:00 -0500
> > > Cc: emacs-devel <emacs-devel@gnu.org>
> > >
> > > I do see something in the redumped emacs that seems like a bug to me.  The process I use for creating the
> > > dump uses the -Q flag.  But some of the settings I see in "emacs -Q --dump-file ..." are not the ones I see
> > > with just "emacs -Q".  Some are pretty basic - menu-bar-mode, tool-bar-mode, global-font-lock-mode,
> > > transient-mark-mode are all nil in the redumped process but not the baseline.
> >
> > That is exactly the problem with re-dumping: stuff that was
> > initialized on the first start gets dumped, and then works differently
> > when Emacs is restarted from the second dump.
>
> That appears to be a consequence of setting
> custom-delayed-init-variables to t in startup, without saving a copy
> to be restored by an after-pdump-load-hook.  Those modes have an
> init-value expression involving "(not noninteractive)"  that is
> evaluated during batch mode startup but not in the post-redump
> startup.
Something like this (untested):
diff --git a/lisp/custom.el b/lisp/custom.el
index 0522bdd447b..c16bc8f8560 100644
--- a/lisp/custom.el
+++ b/lisp/custom.el
@@ -138,6 +138,19 @@ custom-delayed-init-variables
   "List of variables whose initialization is pending until startup.
 Once this list has been processed, this var is set to a non-list value.")

+(defvar custom-pdumped-delayed-init-variables nil
+  "List of all variables whose initialization will be pending until
+a startup following a redump.  This list is prepended to
+custom-delayed-init-variables in an after-pdump-load-hook.")
+(defun custom-delayed-init-variables-reset-after-redump ()
+  "Ensure delayed initializations are performed at startup after redumping"
+  (when custom-pdumped-delayed-init-variables
+    (setq custom-delayed-init-variables
+          (nconc custom-pdumped-delayed-init-variables
+                 custom-delayed-init-variables))))
+(add-hook 'after-pdump-load-hook
+          #'custom-delayed-init-variables-reset-after-redump)
+
 (defun custom-initialize-delay (symbol value)
   "Delay initialization of SYMBOL to the next Emacs start.
 This is used in files that are preloaded (or for autoloaded
diff --git a/lisp/startup.el b/lisp/startup.el
index dcc098f84c3..7947f676827 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -1311,6 +1311,7 @@ command-line
             ;; Initialize them in the same order they were loaded, in
             ;; case there are dependencies between them.
             (reverse custom-delayed-init-variables))))
+  (setq custom-pdumped-delayed-init-variables custom-delayed-init-variables)
   (setq custom-delayed-init-variables t)

   ;; Warn for invalid user name.

Re: Regression in dump-emacs-portable

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Fri, 17 Feb 2023 08:22:56 -0500
> Cc: emacs-devel@gnu.org
> 
> On Thu, Feb 16, 2023 at 6:45 PM Lynn Winebarger <owinebar@gmail.com> wrote:
> >
> > On Thu, Feb 16, 2023 at 10:34 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > > From: Lynn Winebarger <owinebar@gmail.com>
> > > > Date: Thu, 16 Feb 2023 10:05:00 -0500
> > > > Cc: emacs-devel <emacs-devel@gnu.org>
> > > >
> > > > I do see something in the redumped emacs that seems like a bug to me.  The process I use for creating the
> > > > dump uses the -Q flag.  But some of the settings I see in "emacs -Q --dump-file ..." are not the ones I see
> > > > with just "emacs -Q".  Some are pretty basic - menu-bar-mode, tool-bar-mode, global-font-lock-mode,
> > > > transient-mark-mode are all nil in the redumped process but not the baseline.
> > >
> > > That is exactly the problem with re-dumping: stuff that was
> > > initialized on the first start gets dumped, and then works differently
> > > when Emacs is restarted from the second dump.
> >
> > That appears to be a consequence of setting
> > custom-delayed-init-variables to t in startup, without saving a copy
> > to be restored by an after-pdump-load-hook.  Those modes have an
> > init-value expression involving "(not noninteractive)"  that is
> > evaluated during batch mode startup but not in the post-redump
> > startup.
> Something like this (untested):
> diff --git a/lisp/custom.el b/lisp/custom.el
> index 0522bdd447b..c16bc8f8560 100644
> --- a/lisp/custom.el
> +++ b/lisp/custom.el
> @@ -138,6 +138,19 @@ custom-delayed-init-variables
>    "List of variables whose initialization is pending until startup.
>  Once this list has been processed, this var is set to a non-list value.")
> 
> +(defvar custom-pdumped-delayed-init-variables nil
> +  "List of all variables whose initialization will be pending until
> +a startup following a redump.  This list is prepended to

I don't understand what will this solve.  Why does it matter when
exactly is a variable initialized, if in any case that will happen
before re-dumping?

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Fri, Feb 17, 2023 at 9:31 AM Eli Zaretskii <eliz@gnu.org> wrote:
> I don't understand what will this solve.  Why does it matter when
> exactly is a variable initialized, if in any case that will happen
> before re-dumping?

You're right with respect to the current definitions of
custom-initialize-* functions that will only set variables if they are
unbound.
The docstring for custom-initialize-delay is:
  "Delay initialization of SYMBOL to the next Emacs start.
This is used in files that are preloaded (or for autoloaded
variables), so that the initialization is done in the run-time
context rather than the build-time context.  This also has the
side-effect that the (delayed) initialization is performed with
the :set function."
But "so the initialization is done in the run-time context" and "to
the *next* Emacs start" (emphasis mine) are contradictory unless
dumping can only happen once.    The first sentence should be "Ensure
SYMBOL is set to initial-value at Emacs startup".  Otherwise, the
"initialization" will only reflect the value of the customization when
it was first set, regardless of whether that value reflects the
*current* run-time context due to redumping.  In particular, currently
redumping has to happen in a noninteractive context, so redumping of a
customization with an initialization expression referencing the value
of 'noninteractive will not be correct when it is later used in an
interactive session.  Also, any customization variables introduced
after a redump that are supposed to be initialized by
#'custom-initialize-delay will not be initialized on the next startup,
since custom-delayed-init-variables is already set to t in the dump.
The current implementation only works for customization variables
defined in the initial dump.
I'll take a shot at a more thorough rewrite of the customization
system to properly support redumping this weekend and send another
patch.

Lynn

Re: Regression in dump-emacs-portable

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Fri, 17 Feb 2023 18:44:18 -0500
> Cc: emacs-devel@gnu.org
> 
> On Fri, Feb 17, 2023 at 9:31 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > I don't understand what will this solve.  Why does it matter when
> > exactly is a variable initialized, if in any case that will happen
> > before re-dumping?
> 
> You're right with respect to the current definitions of
> custom-initialize-* functions that will only set variables if they are
> unbound.
> The docstring for custom-initialize-delay is:
>   "Delay initialization of SYMBOL to the next Emacs start.
> This is used in files that are preloaded (or for autoloaded
> variables), so that the initialization is done in the run-time
> context rather than the build-time context.  This also has the
> side-effect that the (delayed) initialization is performed with
> the :set function."
> But "so the initialization is done in the run-time context" and "to
> the *next* Emacs start" (emphasis mine) are contradictory unless
> dumping can only happen once.    The first sentence should be "Ensure
> SYMBOL is set to initial-value at Emacs startup".

temacs and bootstrap-emacs (and in general any Emacs that is about to
dump itself) also start up, so the change you propose will make the
doc string less accurate.

> I'll take a shot at a more thorough rewrite of the customization
> system to properly support redumping this weekend and send another
> patch.

I'm not sure this is the tree we should be barking up.  The problem is
more general than just delayed-initialization of some defcustoms.

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2079 bytes --]

On Sat, Feb 18, 2023, 2:07 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Fri, 17 Feb 2023 18:44:18 -0500
> > Cc: emacs-devel@gnu.org
> >
> > On Fri, Feb 17, 2023 at 9:31 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > > I don't understand what will this solve.  Why does it matter when
> > > exactly is a variable initialized, if in any case that will happen
> > > before re-dumping?
> >
> > You're right with respect to the current definitions of
> > custom-initialize-* functions that will only set variables if they are
> > unbound.
> > The docstring for custom-initialize-delay is:
> >   "Delay initialization of SYMBOL to the next Emacs start.
> > This is used in files that are preloaded (or for autoloaded
> > variables), so that the initialization is done in the run-time
> > context rather than the build-time context.  This also has the
> > side-effect that the (delayed) initialization is performed with
> > the :set function."
> > But "so the initialization is done in the run-time context" and "to
> > the *next* Emacs start" (emphasis mine) are contradictory unless
> > dumping can only happen once.    The first sentence should be "Ensure
> > SYMBOL is set to initial-value at Emacs startup".
>
> temacs and bootstrap-emacs (and in general any Emacs that is about to
> dump itself) also start up, so the change you propose will make the
> doc string less accurate.
>
> > I'll take a shot at a more thorough rewrite of the customization
> > system to properly support redumping this weekend and send another
> > patch.
>
> I'm not sure this is the tree we should be barking up.  The problem is
> more general than just delayed-initialization of some defcustoms.
>

I wrote about 95% of this over the weekend, most of which is the efficient
solver for dependency ordering of the customization variables set at
startup.  I'll finish it up in the next couple of evenings and post the
patch in a bug report/feature request.

My immediate concern is being able to use the current re-dumping facility
more effectively.

Lynn

[-- Attachment #2: Type: text/html, Size: 3162 bytes --]

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Tue, Feb 21, 2023 at 9:21 AM Lynn Winebarger <owinebar@gmail.com> wrote:
> On Sat, Feb 18, 2023, 2:07 AM Eli Zaretskii <eliz@gnu.org> wrote:
>> > I'll take a shot at a more thorough rewrite of the customization
>> > system to properly support redumping this weekend and send another
>> > patch.
>>
>> I'm not sure this is the tree we should be barking up.  The problem is
>> more general than just delayed-initialization of some defcustoms.
>
>
> I wrote about 95% of this over the weekend, most of which is the efficient solver for dependency ordering of the customization variables set at startup.  I'll finish it up in the next couple of evenings and post the patch in a bug report/feature request.
>
> My immediate concern is being able to use the current re-dumping facility more effectively.

I've posted a patch at
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=61721   It's true there
are not a lot of these customizations in the baseline dump, but there
are many more when additional libraries are redumped.

However, the problem with dumping overlays is very troublesome for
creating comprehensive redumps.

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Thu, Feb 16, 2023 at 10:34 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Thu, 16 Feb 2023 10:05:00 -0500
> > Cc: emacs-devel <emacs-devel@gnu.org>
> >
> > I do see something in the redumped emacs that seems like a bug to me.  The process I use for creating the
> > dump uses the -Q flag.  But some of the settings I see in "emacs -Q --dump-file ..." are not the ones I see
> > with just "emacs -Q".  Some are pretty basic - menu-bar-mode, tool-bar-mode, global-font-lock-mode,
> > transient-mark-mode are all nil in the redumped process but not the baseline.
>
> That is exactly the problem with re-dumping: stuff that was
> initialized on the first start gets dumped, and then works differently
> when Emacs is restarted from the second dump.
>
> It might sound like a minor issue, but it is actually a tip of a very
> large iceberg.  When we run temacs to produce the dump file, we jump
> through many hoops to avoid these problems, and then when dumped Emacs
> is started, it jumps through some more hoops.  Those are all the
> init_FOO_once and pdumper_do_now_and_after_load calls.  All this works
> differently when Emacs starts from a re-dumped dump file.  For
> example, features that are toggled might be turned off in the second
> start because the startup code assumes they are originally turned off
> and toggling them turns them on.

I don't understand why the initialization function for minor mode
variables toggles, or why a minor mode command invoked in startup
without argument would toggle.  That's only supposed to be done when
the mode command is invoked interactively without argument.
It's true that there are only a few customization variables in the
baseline dump file that use custom-initialize-delay.  But one way of
dealing with these issues would be to do a (conservative) analysis of
initialization expressions for dependencies on values that are
dependent on run-time initialization and re-initialize them
accordingly.  The patch I submitted in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=61721 does the first
step.  In my test redump (with nothing additional loaded),
global-font-lock-mode and tooltips-mode are set correctly at least.

I added a bug report specifically about the failure of dumping for
overlays and treesit-compiled-query pseudovectors.
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=61733

Lynn

Re: Regression in dump-emacs-portable

[Gregory Heytings]

>
> I mentioned redumping a maximal set of core emacs libraries under 28.2 
> in a recent thread.
>

Just out of curiosity: what is the purpose of doing that?  You already 
mentioned several times that you are trying to create a "fully-loaded" 
Emacs, with as many packages as possible in the dump file (IIUC), but 
without explaining why (IIRC).

Re: Regression in dump-emacs-portable

[Lynn Winebarger]

On Thu, Feb 23, 2023 at 10:08 AM Gregory Heytings <gregory@heytings.org> wrote:
> > I mentioned redumping a maximal set of core emacs libraries under 28.2
> > in a recent thread.
> >
>
> Just out of curiosity: what is the purpose of doing that?  You already
> mentioned several times that you are trying to create a "fully-loaded"
> Emacs, with as many packages as possible in the dump file (IIUC), but
> without explaining why (IIRC).

I've been an emacs user for a pretty long time though with frequency
depending on how much coding I happen to be doing in my leisure time.
In my day job, I've been using other editors/IDEs.  So while I enjoy
pretending I am a grad student at MIT's AI labs on Symbolics LISP
machines circa the mid-late 1980s, I am curious about the extent to
which Emacs has the features of these other editors,  In terms of
technical design, VSCode is probably the closest, in terms of much of
the interface being implemented in a dynamic language accessible to
the user at run-time.  V8 even provides a form of dumping, although I
don't know that VSCode supports it for end-users.  So, while I can
appreciate the desire for a parsimonious build that can be run with
limited resources, the competitor products regularly use 100-200MBs in
a process and I see no reason to avoid processes using similar amounts
of resources.
I've also historically used the default settings of Emacs for the most
part.  But if I want to see how optional features provided by packages
actually work, I want to be able to turn them on and off in an
options-type menu the way I would in these other editors.  Some
libraries/packages provide customizations via autoload, but to really
be sure you see all the knobs (and get some notion of what the
functionality might be), the packages have to be loaded.  Loading a
bunch of libraries/packages (particularly packages) at startup can be
slow.  If you load 1000 packages plus emacs, that can easily take 2+
minutes to load.  At 2000-2500 packages, 5 minutes is a fair
expectation.  When I was doing this with native-compiled modules, it
was paradoxically even slower.  Dump files, whether build-time dumps
or redumps, are just much much faster to load.  Plus, it's the closest
thing emacs's garbage collector has to a compacting generational
collection.  Loading libraries creates plenty of garbage but also the
longest-living objects in the system, so it makes the GC more
efficient to ensure those objects are compacted.
Ultimately, I think it would be fun to write support for a modular
dump file format for more efficient library loading.  But first, it
makes sense to understand the functionality already available and
measure its performance for benchmarking purposes.  And encourage
others to make use of those facilities, so they do not wither on the
vine.

Lynn

Re: Regression in dump-emacs-portable

[Richard Stallman]

  > So while I enjoy
  > pretending I am a grad student at MIT's AI labs on Symbolics LISP
  > machines circa the mid-late 1980s,

Since the actual author of GNU Emacs (as well as the first Emacs) is me,
to associate them with Symbolics machines is rather the opposite of
the real history.

If you read the last chapter of Hackers, by Steve Levy, you can see how
that is so.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Regression in dump-emacs-portable

[Richard Stallman]

  > I've also historically used the default settings of Emacs for the most
  > part.  But if I want to see how optional features provided by packages
  > actually work, I want to be able to turn them on and off in an
  > options-type menu the way I would in these other editors.  Some
  > libraries/packages provide customizations via autoload, but to really
  > be sure you see all the knobs (and get some notion of what the
  > functionality might be), the packages have to be loaded.  Loading a
  > bunch of libraries/packages (particularly packages) at startup can be
  > slow.

I agree that this is a good area for improvement.  M-x customize was a
stab in that direction, but it isn't as good as what other programs
do.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)