bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

If you keep rebuilding Emacs from the Git repository using just the
"make -jN" commands, to track the development branches, after some
time loaddefs.el becomes outdated.  To see the outdated parts, rename
loaddefs.el and then say

  $ make -C lisp autoloads-force

Then compare the old loaddefs.el with the newly created one: you will
see many changes, depending on when was the last time you bootstrapped
or otherwise regenerated loaddefs.el from scratch.

This happens because regeneration of the parts of loaddefs.el affected
by Lisp changes is unreliable and misses some changes, in particular
those where autoloads from some Lisp files are redirected to private
*-loaddefs.el files instead of the common loaddefs.el.

These updates should happen automatically, they should not require
people to "make bootstrap" or manually regenerate loaddefs.el.

In GNU Emacs 28.0.50 (build 122, i686-pc-mingw32)
 of 2021-04-30 built on HOME-C4E4A596F7
Repository revision: ab7a61e0efd0684bc37a556d12f36521f9f61782
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)

Configured using:
 'configure -C --prefix=/d/usr --with-wide-int --with-native-compilation
 --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs auth-source eieio eieio-core eieio-loaddefs
password-cache json map text-property-search time-date mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils comp
comp-cstr warnings subr-x rx cl-seq cl-macs cl-extra help-mode seq
byte-opt gv cl-loaddefs cl-lib bytecomp byte-compile cconv iso-transl
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win w32-vars
term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process nativecomp emacs)

Memory information:
((conses 16 84237 12365)
 (symbols 48 8956 1)
 (strings 16 25167 3939)
 (string-bytes 1 768361)
 (vectors 16 16937)
 (vector-slots 8 295589 16641)
 (floats 8 28 124)
 (intervals 40 267 89)
 (buffers 888 11))

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Gregory Heytings]

>
> If you keep rebuilding Emacs from the Git repository using just the 
> "make -jN" commands, to track the development branches, after some time 
> loaddefs.el becomes outdated.  To see the outdated parts, rename 
> loaddefs.el and then say
>
>  $ make -C lisp autoloads-force
>
> Then compare the old loaddefs.el with the newly created one: you will 
> see many changes, depending on when was the last time you bootstrapped 
> or otherwise regenerated loaddefs.el from scratch.
>
> This happens because regeneration of the parts of loaddefs.el affected 
> by Lisp changes is unreliable and misses some changes, in particular 
> those where autoloads from some Lisp files are redirected to private 
> *-loaddefs.el files instead of the common loaddefs.el.
>
> These updates should happen automatically, they should not require 
> people to "make bootstrap" or manually regenerate loaddefs.el.
>

Would it not make sense to include these changes in their respective 
commits?

Another solution would be to have a cron job running somewhere and 
automatically pushing these changes once a day or so.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> Date: Fri, 30 Apr 2021 15:01:15 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 48117@debbugs.gnu.org
> 
> > These updates should happen automatically, they should not require 
> > people to "make bootstrap" or manually regenerate loaddefs.el.
> >
> 
> Would it not make sense to include these changes in their respective 
> commits?

loaddefs.el is a generated file (and so are the other *-loaddefs.el
files).

> Another solution would be to have a cron job running somewhere and 
> automatically pushing these changes once a day or so.

How can we do that for files that aren't versioned?

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Glenn Morris]

This issue has been present forever.
There are some comments in the Makefiles about it.
There are several issues, eg:

1) autoload generation is slow.

2) the dependencies of the loaddefs files are unknown to make,
and are basically "all lisp files". (You can't even say "just those
files with autoload statements", because removing a previously existing
autoload statement changes the output.)

3) Traditionally, re-making loaddefs files could make trivial changes
to the output that weren't important (eg ordering of the "no
autoloads" section, timestamping), but would still trigger re-dumping emacs.
Which could then trigger regeneration of the autoloads, and
re-dumping, etc.  This may be better nowadays, since there is no
longer timestamp information in the loaddefs files (see autoload-timestamps).

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Glenn Morris]

One idea, don't know if workable:
always regenerate loaddefs, but using a temporary file name for the
main loaddefs.
use build-aux/move-if-change to only replace the real loaddefs file if
there have been changes. Although again, it's possible these would not
be "real" (ie significant) changes.

This would mean every invocation of make would generate a loaddefs file,
often for no need, which seems ugly.
It's a trade off between doing things properly (which is a bootstrap)
and efficiently, as has always been the case with Emacs's build.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Gregory Heytings]

>>> These updates should happen automatically, they should not require 
>>> people to "make bootstrap" or manually regenerate loaddefs.el.
>>
>> Would it not make sense to include these changes in their respective 
>> commits?
>
> loaddefs.el is a generated file (and so are the other *-loaddefs.el 
> files).
>

Ah, yes, indeed.

>> Another solution would be to have a cron job running somewhere and 
>> automatically pushing these changes once a day or so.
>
> How can we do that for files that aren't versioned?
>

I have two questions:

Why are these files not versioned?

The generated lisp/loaddefs.el file is AFAICS identical to the 
lisp/ldefs-boot.el (after calling admin/update_autogen -L), which is 
versioned.  Why is the lisp/loaddefs.el generated?

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: 48117@debbugs.gnu.org
> Date: Fri, 30 Apr 2021 11:22:00 -0400
> 
> This issue has been present forever.

Yes, I know.  This isn't trivial, or else it would have been solved
long ago.

> 1) autoload generation is slow.

Based on my latest experience, I think this is somewhat exaggerated,
especially given that our builds became slower lately.

> 2) the dependencies of the loaddefs files are unknown to make,
> and are basically "all lisp files". (You can't even say "just those
> files with autoload statements", because removing a previously existing
> autoload statement changes the output.)
> 
> 3) Traditionally, re-making loaddefs files could make trivial changes
> to the output that weren't important (eg ordering of the "no
> autoloads" section, timestamping), but would still trigger re-dumping emacs.
> Which could then trigger regeneration of the autoloads, and
> re-dumping, etc.  This may be better nowadays, since there is no
> longer timestamp information in the loaddefs files (see autoload-timestamps).

What worries me the most is that when 'autoloads' is run (and it is,
from time to time), we still end up with outdated loaddefs.el.  I
think we could live with outdated loaddefs.el for short periods of
time, but it looks like running 'autoloads' only updates the part(s)
of the file for Lisp files that the build thinks to be responsible for
the update.  Or something like that, because how else to explain that
some parts remain outdated?

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: 48117@debbugs.gnu.org
> Date: Fri, 30 Apr 2021 11:38:43 -0400
> 
> One idea, don't know if workable:
> always regenerate loaddefs, but using a temporary file name for the
> main loaddefs.
> use build-aux/move-if-change to only replace the real loaddefs file if
> there have been changes. Although again, it's possible these would not
> be "real" (ie significant) changes.

Thanks, I think it's worth trying.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Andreas Schwab]

On Apr 30 2021, Glenn Morris wrote:

> always regenerate loaddefs, but using a temporary file name for the
> main loaddefs.
> use build-aux/move-if-change to only replace the real loaddefs file if
> there have been changes.

That's already what batch-update-autoloads does.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> Date: Fri, 30 Apr 2021 15:47:51 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 48117@debbugs.gnu.org
> 
> Why are these files not versioned?

Because we try not to keep in Git any files that are generated by the
build.  It's redundant to have them there, and also causes frequent
unnecessary conflicts.

> The generated lisp/loaddefs.el file is AFAICS identical to the 
> lisp/ldefs-boot.el (after calling admin/update_autogen -L), which is 
> versioned.  Why is the lisp/loaddefs.el generated?

ldefs-boot.el is versioned because it is needed for the initial build
of a fresh clone, and it is not identical to loaddefs.el, especially
not when you are developing.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  48117@debbugs.gnu.org
> Date: Fri, 30 Apr 2021 17:59:45 +0200
> 
> On Apr 30 2021, Glenn Morris wrote:
> 
> > always regenerate loaddefs, but using a temporary file name for the
> > main loaddefs.
> > use build-aux/move-if-change to only replace the real loaddefs file if
> > there have been changes.
> 
> That's already what batch-update-autoloads does.

Judging by the results, that's not so.  Specifically,
batch-update-autoloads doesn't regenerate _all_ of the loaddefs.el,
only some of its part(s).

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Andreas Schwab]

On Apr 30 2021, Eli Zaretskii wrote:

> Judging by the results, that's not so.  Specifically,
> batch-update-autoloads doesn't regenerate _all_ of the loaddefs.el,
> only some of its part(s).

Yes, because it calls update-directory-autoloads for each directory.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Glenn Morris]

Andreas Schwab wrote:

> That's already what batch-update-autoloads does.

Shows how much my memory is worth! :)

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Glenn Morris]

Another idea, FWIW:
In the make rule for $(lisp)/loaddefs.el, if loaddefs.el exists and is
older than ldefs-boot.el, start by copying the latter to the former.
This ought to limit how outdated loaddefs can get?

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Gregory Heytings]

>> The generated lisp/loaddefs.el file is AFAICS identical to the 
>> lisp/ldefs-boot.el (after calling admin/update_autogen -L), which is 
>> versioned.  Why is the lisp/loaddefs.el generated?
>
> ldefs-boot.el is versioned because it is needed for the initial build of 
> a fresh clone, and it is not identical to loaddefs.el, especially not 
> when you are developing.
>

Can ldefs-boot.el not be used to detect whether loaddefs.el needs to be 
regenerated?  ISTM that if one assumes that ldefs-boot.el is kept up to 
date, it is necessary to regenerate the loaddefs file only when 
ldefs-boot.el becomes more recent than the loaddefs file.  This would 
avoid rebuilding the loaddefs file on each make invocation.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: rgm@gnu.org,  48117@debbugs.gnu.org
> Date: Fri, 30 Apr 2021 18:21:07 +0200
> 
> On Apr 30 2021, Eli Zaretskii wrote:
> 
> > Judging by the results, that's not so.  Specifically,
> > batch-update-autoloads doesn't regenerate _all_ of the loaddefs.el,
> > only some of its part(s).
> 
> Yes, because it calls update-directory-autoloads for each directory.

AFAIU, Glenn's proposal was to process all the directories.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: 48117@debbugs.gnu.org
> Date: Fri, 30 Apr 2021 13:25:36 -0400
> 
> Another idea, FWIW:
> In the make rule for $(lisp)/loaddefs.el, if loaddefs.el exists and is
> older than ldefs-boot.el, start by copying the latter to the former.
> This ought to limit how outdated loaddefs can get?

Yes, I think this is a good idea.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> Date: Fri, 30 Apr 2021 17:32:13 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 48117@debbugs.gnu.org
> 
> Can ldefs-boot.el not be used to detect whether loaddefs.el needs to be 
> regenerated?

I believe that's what Glenn suggested a few minutes ago.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Gregory Heytings]

>> Can ldefs-boot.el not be used to detect whether loaddefs.el needs to be 
>> regenerated?
>
> I believe that's what Glenn suggested a few minutes ago.
>

Yes, it's more or less the same idea, our mails crossed each other because 
of the lists.gnu.org delivery time.

Note that, as I said, this won't work without keeping ldefs-boot.el up to 
date.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> Date: Fri, 30 Apr 2021 19:08:48 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 48117@debbugs.gnu.org
> 
> Note that, as I said, this won't work without keeping ldefs-boot.el up to 
> date.

We (read: Glenn) already do.  You can see that in "git log".

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Gregory Heytings]

>> Note that, as I said, this won't work without keeping ldefs-boot.el up 
>> to date.
>
> We (read: Glenn) already do.  You can see that in "git log".
>

I think this should preferably be automated.  The current ldefs-boot.el 
file in the trunk is 25 days old, running admin/update_autogen -L adds 247 
lines and modifies 100 lines.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Glenn Morris]

Gregory Heytings wrote:

> I think this should preferably be automated.  The current
> ldefs-boot.el file in the trunk is 25 days old, running
> admin/update_autogen -L adds 247 lines and modifies 100 lines.

It gets automaticallly updated on the first of the month.
It could be more frequent, but it would seem like churn for no real
reason most of the time.
(It wouldn't make any difference to the issue that prompted this report.)

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Gregory Heytings]

>> I think this should preferably be automated.  The current ldefs-boot.el 
>> file in the trunk is 25 days old, running admin/update_autogen -L adds 
>> 247 lines and modifies 100 lines.
>
> It gets automaticallly updated on the first of the month.  It could be 
> more frequent, but it would seem like churn for no real reason most of 
> the time.
>

IMO once a week would be better.  An outdated ldefs-boot creates 
unnecessary warnings during the build.

>
> (It wouldn't make any difference to the issue that prompted this 
> report.)
>

That depends on the chosen solution.  You suggested copying ldefs-boot.el 
onto loaddefs.el when it is more recent.  I'm not entirely sure, but it 
seems to me that forcing a regeneration of the loaddefs files during make 
whenever ldefs-boot.el is more recent than loaddefs.el would be a better 
solution.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> Date: Sat, 01 May 2021 08:09:42 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, 48117@debbugs.gnu.org
> 
> You suggested copying ldefs-boot.el onto loaddefs.el when it is more
> recent.  I'm not entirely sure, but it seems to me that forcing a
> regeneration of the loaddefs files during make whenever
> ldefs-boot.el is more recent than loaddefs.el would be a better
> solution.

Your proposal would produce marginally better results for a
significantly longer build time, so I don't think it's a net win.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Gregory Heytings]

>> You suggested copying ldefs-boot.el onto loaddefs.el when it is more 
>> recent.  I'm not entirely sure, but it seems to me that forcing a 
>> regeneration of the loaddefs files during make whenever ldefs-boot.el 
>> is more recent than loaddefs.el would be a better solution.
>
> Your proposal would produce marginally better results for a 
> significantly longer build time, so I don't think it's a net win.
>

If ldefs-boot.el is updated, say, once a week, this would force the 
regeneration of the loaddefs files at most once a week.  Wouldn't that be 
a reasonable compromise?  On my computer, regenerating the loaddefs files 
takes about 10 seconds, or ~3% of the time of a make bootstrap.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> Date: Sat, 01 May 2021 09:20:51 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: rgm@gnu.org, 48117@debbugs.gnu.org
> 
> > Your proposal would produce marginally better results for a 
> > significantly longer build time, so I don't think it's a net win.
> 
> If ldefs-boot.el is updated, say, once a week, this would force the 
> regeneration of the loaddefs files at most once a week.  Wouldn't that be 
> a reasonable compromise?

Compromise between which alternatives?

The "marginally better" results in your proposal are that if someone
updates from Git when he/she is in the middle of some development,
then loaddefs.el are made up-to-date immediately, as opposed to
_maybe_ waiting for the next update of ldefs-boot.el.  (I say "maybe"
because in general loaddefs.el _are_ updated as part of routine
builds, just not 100% reliably so.)

> On my computer, regenerating the loaddefs files takes about 10
> seconds, or ~3% of the time of a make bootstrap.

Keep in mind that some people use less powerful machines.  And the
bootstrap time is not relevant, because loaddefs.el is completely
regenerated during bootstrap anyway.  The time that is relevant is the
time of just "make", and that is usually quite short, even on slow
machines.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Gregory Heytings]

>>> Your proposal would produce marginally better results for a 
>>> significantly longer build time, so I don't think it's a net win.
>>
>> If ldefs-boot.el is updated, say, once a week, this would force the 
>> regeneration of the loaddefs files at most once a week.  Wouldn't that 
>> be a reasonable compromise?
>
> Compromise between which alternatives?
>

I see at least the following possible alternatives:

- the current situation, which you describe in your original post, in 
which the loaddefs files need to be regenerated manually (inconvenient)

- regenerating the loaddefs files for each make invocation (inefficient)

- copying the ldefs-boot.el onto loaddefs.el when it is more recent (which 
IIUC could lose local additions to loaddefs.el)

- automatically regenerating the loaddefs files when ldefs-boot.el is more 
recent than loaddefs.el (which I understand could be a bit slow)

- issue only a warning when make is invoked and ldefs-boot.el is more 
recent than loaddefs.el (and perhaps add a autoloads or loaddefs target to 
the main Makefile)

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> Date: Sat, 01 May 2021 12:29:12 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: rgm@gnu.org, 48117@debbugs.gnu.org
> 
> > Compromise between which alternatives?
> >
> 
> I see at least the following possible alternatives:

My point is that the gain from your proposal wrt the next best
alternative is small, but the price is too high.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> 1) autoload generation is slow.
>
> Based on my latest experience, I think this is somewhat exaggerated,
> especially given that our builds became slower lately.

Has it?  I haven't noticed any substantial slow-down...  (Unless you
mean with native-comp switched on.)

In any case, autoload generation takes a substantial chunk of time --
especially since it's single-threaded.  Glenn's suggestion to copy over
ldefs-boot.el (if it's newer) sounds like a good idea to me.  (And I'd
also like if that file was regenerated more often than once a month...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Glenn Morris <rgm@gnu.org>,  48117@debbugs.gnu.org
> Date: Sun, 02 May 2021 09:28:04 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> 1) autoload generation is slow.
> >
> > Based on my latest experience, I think this is somewhat exaggerated,
> > especially given that our builds became slower lately.
> 
> Has it?  I haven't noticed any substantial slow-down...  (Unless you
> mean with native-comp switched on.)

Yes, native-comp, certainly.  But the build became slower a year or so
ago as well, I no longer remember why.

> In any case, autoload generation takes a substantial chunk of time --
> especially since it's single-threaded.  Glenn's suggestion to copy over
> ldefs-boot.el (if it's newer) sounds like a good idea to me.

Yes, I agree.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Glenn Morris]

Glenn Morris wrote:

> Another idea, FWIW:
> In the make rule for $(lisp)/loaddefs.el, if loaddefs.el exists and is
> older than ldefs-boot.el, start by copying the latter to the former.
> This ought to limit how outdated loaddefs can get?

BTW, I think this might actually make things worse in terms of detecting
new secondary loaddefs files like texinfo-loaddefs (not that those
happen very often), by bumping the timestamp on the primary loaddefs.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: 48117@debbugs.gnu.org
> Date: Sun, 02 May 2021 12:59:17 -0400
> 
> Glenn Morris wrote:
> 
> > Another idea, FWIW:
> > In the make rule for $(lisp)/loaddefs.el, if loaddefs.el exists and is
> > older than ldefs-boot.el, start by copying the latter to the former.
> > This ought to limit how outdated loaddefs can get?
> 
> BTW, I think this might actually make things worse in terms of detecting
> new secondary loaddefs files like texinfo-loaddefs (not that those
> happen very often), by bumping the timestamp on the primary loaddefs.

But there's no such thing as detecting new secondary loaddefs, right?
The only remedy for that is "make autoloads-force", right?

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Gregory Heytings]

>> Another idea, FWIW: In the make rule for $(lisp)/loaddefs.el, if 
>> loaddefs.el exists and is older than ldefs-boot.el, start by copying 
>> the latter to the former. This ought to limit how outdated loaddefs can 
>> get?
>
> BTW, I think this might actually make things worse in terms of detecting 
> new secondary loaddefs files like texinfo-loaddefs (not that those 
> happen very often), by bumping the timestamp on the primary loaddefs.
>

Which is one of the reasons why I still believe that my proposed solution 
is better: whenever lisp/ldefs-boot.el is more recent than 
lisp/loaddefs.el, issue a warning when make is invoked to suggest the 
regeneration of _all_ autoload files, and add a command to do this in the 
main Makefile: make autoloads would do find -name '*loaddefs.el' -delete 
&& make -C lisp autoloads.  In theory this should have the same effect as 
make -C lisp autoloads-force, but that command is only marginally faster, 
and I much prefer the clarity / determinicity of a "delete all and 
regenerate" operation.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> In any case, autoload generation takes a substantial chunk of time --
> especially since it's single-threaded.  Glenn's suggestion to copy over
> ldefs-boot.el (if it's newer) sounds like a good idea to me.  (And I'd
> also like if that file was regenerated more often than once a month...)

If Eli agrees, I'm happy to update ldefs-boot.el more frequently.

Just let me know how often you want it updated and I will make it so.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> If Eli agrees, I'm happy to update ldefs-boot.el more frequently.
>
> Just let me know how often you want it updated and I will make it so.

I usually just update it now if I add an ;;;###autoload somewhere --
updating that file kinda goes hand in hand with that.  (I added a new
"make ldefs-boot.el" Makefile target to make it more convenient.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Wed, 8 Dec 2021 01:36:25 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, Glenn Morris <rgm@gnu.org>, 48117@debbugs.gnu.org
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > In any case, autoload generation takes a substantial chunk of time --
> > especially since it's single-threaded.  Glenn's suggestion to copy over
> > ldefs-boot.el (if it's newer) sounds like a good idea to me.  (And I'd
> > also like if that file was regenerated more often than once a month...)
> 
> If Eli agrees, I'm happy to update ldefs-boot.el more frequently.

I see no down-sides to updating it frequently, so it's fine with me.

bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Then compare the old loaddefs.el with the newly created one: you will
> see many changes, depending on when was the last time you bootstrapped
> or otherwise regenerated loaddefs.el from scratch.
>
> This happens because regeneration of the parts of loaddefs.el affected
> by Lisp changes is unreliable and misses some changes, in particular
> those where autoloads from some Lisp files are redirected to private
> *-loaddefs.el files instead of the common loaddefs.el.

This should now be fixed in Emacs 29 -- the private loaddefs files
updates should be handled as well as the main one.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no