bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Eli Zaretskii]

The current Makefile's still exhibit the problem that when loaddefs.el
is updated, the Emacs executable is not re-dumped to reflect those
updates.  This leads to an annoying and error-prone practice that one
needs to watch the build, detect the telltale "ELC loaddefs.elc"
command (which means loaddefs.el was actually modified), and then
manually invoke "make" again to re-dump Emacs.

AFAICT, the root cause of this is that src/Makefile.in doesn't know
about the dependencies of loaddefs.el, which are spelled out in
lisp/Makefile.in, and therefore it doesn't realize that loaddefs.el
will be modified as part of the current build.



In GNU Emacs 29.0.50 (build 1694, i686-pc-mingw32)
 of 2022-08-12 built on HOME-C4E4A596F7
Repository revision: 687fcc2c40d693bbe342d938e37a4c9a245a1b02
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)

Configured using:
 'configure -C --prefix=/d/usr --with-wide-int
 --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NOTIFY
W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel dos-w32
ls-lisp disp-table term/w32-win w32-win w32-vars term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process emacs)

Memory information:
((conses 16 42199 12256)
 (symbols 48 6234 0)
 (strings 16 16487 2980)
 (string-bytes 1 396347)
 (vectors 16 9214)
 (vector-slots 8 144024 12454)
 (floats 8 23 45)
 (intervals 40 268 96)
 (buffers 888 10))

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> The current Makefile's still exhibit the problem that when loaddefs.el
> is updated, the Emacs executable is not re-dumped to reflect those
> updates.  This leads to an annoying and error-prone practice that one
> needs to watch the build, detect the telltale "ELC loaddefs.elc"
> command (which means loaddefs.el was actually modified), and then
> manually invoke "make" again to re-dump Emacs.
>
> AFAICT, the root cause of this is that src/Makefile.in doesn't know
> about the dependencies of loaddefs.el, which are spelled out in
> lisp/Makefile.in, and therefore it doesn't realize that loaddefs.el
> will be modified as part of the current build.

Yes.  So the next time you say "make", it'll rebuild the executable.

In one way, this is slightly better than what we had before, where
loaddefs.el wouldn't be rebuilt until after VCWITNESS had changed, while
it's now actually updated all the time (that something changes).  It's
more... regular?  But on the other hand, these updates are seldom
actually necessary, so we're doing it a bit too much now, perhaps.

Anyway, back to the problem -- I've poked at this quite a bit, but no
solutions seem really...  nice?  If we updated the loaddefs.el file a
bit earlier (before we build the Emacs executable), that'd fix the
problem, but I could find a way to do that without breaking something
else.  There's probably something trivial I'm overlooking here.

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 57152@debbugs.gnu.org
> Date: Fri, 12 Aug 2022 17:14:26 +0200
> 
> Anyway, back to the problem -- I've poked at this quite a bit, but no
> solutions seem really...  nice?  If we updated the loaddefs.el file a
> bit earlier (before we build the Emacs executable), that'd fix the
> problem, but I could find a way to do that without breaking something
> else.  There's probably something trivial I'm overlooking here.

Something similar to what we do with $(shortlisp), perhaps?

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Something similar to what we do with $(shortlisp), perhaps?

Yes, probably.  loaddefs.el is already in $(shortlisp), but the problem
is that we may later update loaddefs.el again (while doing the scan).

Hm; there must be some easy way to sort of these dependencies.
Unfortunately I don't have much time this weekend, but I'll poke away at
it again next week.

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Lars Ingebrigtsen]

I've spent an hour poking at this.  Here's the recipe to reproduce:

echo ";;;###autoload ;;(+ 1 2)" >> lisp/foo.el
make

This will update loaddefs.el(c), but won't rebuild src/emacs.pdmp.
Then, if you say

make

it'll rebuild src/emacs.pdmp.

I've tried various things, like making autoloads an order-only
prerequisite for $(pdmp) (and various other targets), but the main
problem seems to be that Make has already computed the
timestamps/dependencies at this point, so even if we're running the
update at the "right" time, we don't trigger the $(pdmp) rule.

And moving this earlier results in problems with the boot build.

I think it'd be helpful if somebody with fresh eyes could take a look at
this.

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 57152@debbugs.gnu.org
> Date: Mon, 15 Aug 2022 12:07:15 +0200
> 
> I've tried various things, like making autoloads an order-only
> prerequisite for $(pdmp) (and various other targets), but the main
> problem seems to be that Make has already computed the
> timestamps/dependencies at this point, so even if we're running the
> update at the "right" time, we don't trigger the $(pdmp) rule.

That cannot be the reason, AFAIK.  Make begins by reading all the
rules and dependencies in the Makefile and producing a DAG for
rebuilding the targets; then it walks the DAG invoking commands as it
goes.  Thus, a target that is going to be rebuilt will automatically
trigger the rules of all the targets that depend on it, as long as the
dependencies are known to Make in full.

IOW, Make doesn't update the timestamps of files during the build, it
figures it all out in advance.

So I think the problem here is that the rules which trigger
regeneration of loaddefs.el are in lisp/Makefile, while the rules for
building $(pdmp) are in src/Makefile.

So maybe moving or duplicating the loaddefs.el rules in src/Makefile
will do the trick.

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: 57152@debbugs.gnu.org
>> Date: Mon, 15 Aug 2022 12:07:15 +0200
>> 
>> I've tried various things, like making autoloads an order-only
>> prerequisite for $(pdmp) (and various other targets), but the main
>> problem seems to be that Make has already computed the
>> timestamps/dependencies at this point, so even if we're running the
>> update at the "right" time, we don't trigger the $(pdmp) rule.
>
> So maybe moving or duplicating the loaddefs.el rules in src/Makefile
> will do the trick.

Could it be that src/Makefile is simply not invoked after lisp/Makefile
has built loaddefs.*? In Makefile.in we have

SUBDIR = $(NTDIR) lib lib-src src lisp
...
all: ${SUBDIR} info $(gsettings_SCHEMAS:.xml=.valid)

That is src comes before lisp.  Haching something like a second 'make
-C' at the end seems to do something not entirely unreasonable.

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> I've tried various things, like making autoloads an order-only
>> prerequisite for $(pdmp) (and various other targets), but the main
>> problem seems to be that Make has already computed the
>> timestamps/dependencies at this point, so even if we're running the
>> update at the "right" time, we don't trigger the $(pdmp) rule.

[...]

> IOW, Make doesn't update the timestamps of files during the build, it
> figures it all out in advance.

I think we're saying the same thing.  😀

> So I think the problem here is that the rules which trigger
> regeneration of loaddefs.el are in lisp/Makefile, while the rules for
> building $(pdmp) are in src/Makefile.
>
> So maybe moving or duplicating the loaddefs.el rules in src/Makefile
> will do the trick.

There really are no rules for loaddefs.el -- Emacs itself (via
loaddefs-gen) figures out whether loaddefs.el needs updating.  I.e.,
Make knows nothing about the dependencies loaddefs.el really has.

We could make loaddefs.el depend on all .el files, but then we'd rebuild
the Emacs executable every time we change an .el file, and we don't want
to do that, either.

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Lars Ingebrigtsen]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Could it be that src/Makefile is simply not invoked after lisp/Makefile
> has built loaddefs.*? In Makefile.in we have
>
> SUBDIR = $(NTDIR) lib lib-src src lisp
> ...
> all: ${SUBDIR} info $(gsettings_SCHEMAS:.xml=.valid)
>
> That is src comes before lisp.  Haching something like a second 'make
> -C' at the end seems to do something not entirely unreasonable.

Hm, interesting...  but I think we might end up in a situation where we
first build the Emacs executable, then update the loaddefs.el, and then
build the Emacs executable again.

But perhaps that's OK -- while we're scanning for new loaddefs every
build, there's seldom any new ;;;###autoloads, so the loaddefs.el file
doesn't update all that often.

I'm not quite sure where the second "make -C" would go, though.

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 57152@debbugs.gnu.org
> Date: Wed, 17 Aug 2022 12:34:13 +0200
> 
> We could make loaddefs.el depend on all .el files, but then we'd rebuild
> the Emacs executable every time we change an .el file, and we don't want
> to do that, either.

What about using move-if-change?  That's a standard technique of
avoiding large rebuilds because some dependency's rules cannot be
easily stated.

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Gerd Möllmann]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Could it be that src/Makefile is simply not invoked after lisp/Makefile
>> has built loaddefs.*? In Makefile.in we have
>>
>> SUBDIR = $(NTDIR) lib lib-src src lisp
>> ...
>> all: ${SUBDIR} info $(gsettings_SCHEMAS:.xml=.valid)
>>
>> That is src comes before lisp.  Haching something like a second 'make
>> -C' at the end seems to do something not entirely unreasonable.
>
> Hm, interesting...  but I think we might end up in a situation where we
> first build the Emacs executable, then update the loaddefs.el, and then
> build the Emacs executable again.

I haven't observed that with your 'echo ...>>foo.el; make' example.

But wouldn't that be a hint that there is something fishy in
src/Makefile?  Or incomplete, or something?

>
> But perhaps that's OK -- while we're scanning for new loaddefs every
> build, there's seldom any new ;;;###autoloads, so the loaddefs.el file
> doesn't update all that often.
>
> I'm not quite sure where the second "make -C" would go, though.

I did it this way:

diff --git a/Makefile.in b/Makefile.in
index bf0f52b514..78103f897f 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -358,10 +358,17 @@ ELN_DESTDIR =
 
 gsettings_SCHEMAS = etc/org.gnu.emacs.defaults.gschema.xml
 
-all: ${SUBDIR} info $(gsettings_SCHEMAS:.xml=.valid)
+all: ${SUBDIR} info $(gsettings_SCHEMAS:.xml=.valid) src-depending-on-lisp
 
 .PHONY: all ${SUBDIR} blessmail epaths-force epaths-force-w32 epaths-force-ns-self-contained etc-emacsver
 
+# Changes in lisp may require us to reconsider the build in src.  For
+# example, if loaddefs.{el,elc} were built in lisp, we need a new
+# .pdmp containing the new autoloads.
+.PHONY: src-depending-on-lisp
+src-depending-on-lisp: lisp
+	${MAKE} -C src
+
 # If configure were to just generate emacsver.tex from emacsver.tex.in
 # in the normal way, the timestamp of emacsver.tex would always be
 # newer than that of the pdf files, which are prebuilt in release tarfiles.


The dependency on lisp in src-depending-on-lisp is necessary to make
sure it runs after lisp has been built.  Maybe there is a clever trick
to express "only if loaddefs has been regenerated" in some way, but
'make -C src' is quite fast already if it hasn't to do anything, at
least compared to other things.

There is one pretty strange thing though, which I can't explain.  I once
got an error "trying to load incoherent eln" which is mentioned in
bug#45103.  I can't say much else about this though, not even what the
cause of this might be.  It was a build with -j8 if that matters.  Maybe
elns are modified during dumping, so that a second dump cannot be done?
But that wouldn't explain why it succeeds most of the time.  No idea,
and not easily reproducible, alas.

CC'ing Andrea.

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Lars Ingebrigtsen]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

>> I'm not quite sure where the second "make -C" would go, though.
>
> I did it this way:

[...]

> +all: ${SUBDIR} info $(gsettings_SCHEMAS:.xml=.valid) src-depending-on-lisp

Ah, right.  I've now done some testing with your patch, and it fixes the
problem (without introducing any new extra src/emacs builds in a
bootstrap or otherwise).  So I just went ahead and pushed it.  :-)

> There is one pretty strange thing though, which I can't explain.  I once
> got an error "trying to load incoherent eln" which is mentioned in
> bug#45103.  I can't say much else about this though, not even what the
> cause of this might be.  It was a build with -j8 if that matters.  Maybe
> elns are modified during dumping, so that a second dump cannot be done?
> But that wouldn't explain why it succeeds most of the time.  No idea,
> and not easily reproducible, alas.

Yes, I've seen those, too, but very seldom (like -- once a month?), and
I haven't been able to reproduce it.

bug#57152: 29.0.50; Emacs executable isn't rebuilt when loaddefs.el is modified

[Gerd Möllmann]

On 22-08-18 14:46 , Lars Ingebrigtsen wrote:
> So I just went ahead and pushed it. :-) 

Oh no :-).

> There is one pretty strange thing though, which I can't explain. I once
>> got an error "trying to load incoherent eln" which is mentioned in
>> bug#45103.  I can't say much else about this though, not even what the
>> cause of this might be.  It was a build with -j8 if that matters.  Maybe
>> elns are modified during dumping, so that a second dump cannot be done?
>> But that wouldn't explain why it succeeds most of the time.  No idea,
>> and not easily reproducible, alas.
> Yes, I've seen those, too, but very seldom (like -- once a month?), and
> I haven't been able to reproduce it.

If anyone reading this has any idea, be it just a gut feeling or 
something, how to reproduce this failure, even if not 100% reliable, I'd 
be grateful to hear it.