unofficial mirror of bug-gnu-emacs@gnu.org 
 help / color / mirror / code / Atom feed
* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
@ 2021-04-30 11:51 Eli Zaretskii
  2021-04-30 15:01 ` Gregory Heytings
                   ` (2 more replies)
  0 siblings, 3 replies; 37+ messages in thread
From: Eli Zaretskii @ 2021-04-30 11:51 UTC (permalink / raw)
  To: 48117

If you keep rebuilding Emacs from the Git repository using just the
"make -jN" commands, to track the development branches, after some
time loaddefs.el becomes outdated.  To see the outdated parts, rename
loaddefs.el and then say

  $ make -C lisp autoloads-force

Then compare the old loaddefs.el with the newly created one: you will
see many changes, depending on when was the last time you bootstrapped
or otherwise regenerated loaddefs.el from scratch.

This happens because regeneration of the parts of loaddefs.el affected
by Lisp changes is unreliable and misses some changes, in particular
those where autoloads from some Lisp files are redirected to private
*-loaddefs.el files instead of the common loaddefs.el.

These updates should happen automatically, they should not require
people to "make bootstrap" or manually regenerate loaddefs.el.

In GNU Emacs 28.0.50 (build 122, i686-pc-mingw32)
 of 2021-04-30 built on HOME-C4E4A596F7
Repository revision: ab7a61e0efd0684bc37a556d12f36521f9f61782
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)

Configured using:
 'configure -C --prefix=/d/usr --with-wide-int --with-native-compilation
 --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs auth-source eieio eieio-core eieio-loaddefs
password-cache json map text-property-search time-date mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils comp
comp-cstr warnings subr-x rx cl-seq cl-macs cl-extra help-mode seq
byte-opt gv cl-loaddefs cl-lib bytecomp byte-compile cconv iso-transl
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win w32-vars
term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process nativecomp emacs)

Memory information:
((conses 16 84237 12365)
 (symbols 48 8956 1)
 (strings 16 25167 3939)
 (string-bytes 1 768361)
 (vectors 16 16937)
 (vector-slots 8 295589 16641)
 (floats 8 28 124)
 (intervals 40 267 89)
 (buffers 888 11))





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 11:51 bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable Eli Zaretskii
@ 2021-04-30 15:01 ` Gregory Heytings
  2021-04-30 15:03   ` Eli Zaretskii
  2021-04-30 15:22 ` Glenn Morris
  2022-06-05 16:25 ` Lars Ingebrigtsen
  2 siblings, 1 reply; 37+ messages in thread
From: Gregory Heytings @ 2021-04-30 15:01 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 48117


>
> If you keep rebuilding Emacs from the Git repository using just the 
> "make -jN" commands, to track the development branches, after some time 
> loaddefs.el becomes outdated.  To see the outdated parts, rename 
> loaddefs.el and then say
>
>  $ make -C lisp autoloads-force
>
> Then compare the old loaddefs.el with the newly created one: you will 
> see many changes, depending on when was the last time you bootstrapped 
> or otherwise regenerated loaddefs.el from scratch.
>
> This happens because regeneration of the parts of loaddefs.el affected 
> by Lisp changes is unreliable and misses some changes, in particular 
> those where autoloads from some Lisp files are redirected to private 
> *-loaddefs.el files instead of the common loaddefs.el.
>
> These updates should happen automatically, they should not require 
> people to "make bootstrap" or manually regenerate loaddefs.el.
>

Would it not make sense to include these changes in their respective 
commits?

Another solution would be to have a cron job running somewhere and 
automatically pushing these changes once a day or so.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 15:01 ` Gregory Heytings
@ 2021-04-30 15:03   ` Eli Zaretskii
  2021-04-30 15:47     ` Gregory Heytings
  0 siblings, 1 reply; 37+ messages in thread
From: Eli Zaretskii @ 2021-04-30 15:03 UTC (permalink / raw)
  To: Gregory Heytings; +Cc: 48117

> Date: Fri, 30 Apr 2021 15:01:15 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 48117@debbugs.gnu.org
> 
> > These updates should happen automatically, they should not require 
> > people to "make bootstrap" or manually regenerate loaddefs.el.
> >
> 
> Would it not make sense to include these changes in their respective 
> commits?

loaddefs.el is a generated file (and so are the other *-loaddefs.el
files).

> Another solution would be to have a cron job running somewhere and 
> automatically pushing these changes once a day or so.

How can we do that for files that aren't versioned?





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 11:51 bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable Eli Zaretskii
  2021-04-30 15:01 ` Gregory Heytings
@ 2021-04-30 15:22 ` Glenn Morris
  2021-04-30 15:38   ` Glenn Morris
  2021-04-30 15:48   ` Eli Zaretskii
  2022-06-05 16:25 ` Lars Ingebrigtsen
  2 siblings, 2 replies; 37+ messages in thread
From: Glenn Morris @ 2021-04-30 15:22 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 48117


This issue has been present forever.
There are some comments in the Makefiles about it.
There are several issues, eg:

1) autoload generation is slow.

2) the dependencies of the loaddefs files are unknown to make,
and are basically "all lisp files". (You can't even say "just those
files with autoload statements", because removing a previously existing
autoload statement changes the output.)

3) Traditionally, re-making loaddefs files could make trivial changes
to the output that weren't important (eg ordering of the "no
autoloads" section, timestamping), but would still trigger re-dumping emacs.
Which could then trigger regeneration of the autoloads, and
re-dumping, etc.  This may be better nowadays, since there is no
longer timestamp information in the loaddefs files (see autoload-timestamps).





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 15:22 ` Glenn Morris
@ 2021-04-30 15:38   ` Glenn Morris
  2021-04-30 15:51     ` Eli Zaretskii
                       ` (2 more replies)
  2021-04-30 15:48   ` Eli Zaretskii
  1 sibling, 3 replies; 37+ messages in thread
From: Glenn Morris @ 2021-04-30 15:38 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 48117


One idea, don't know if workable:
always regenerate loaddefs, but using a temporary file name for the
main loaddefs.
use build-aux/move-if-change to only replace the real loaddefs file if
there have been changes. Although again, it's possible these would not
be "real" (ie significant) changes.

This would mean every invocation of make would generate a loaddefs file,
often for no need, which seems ugly.
It's a trade off between doing things properly (which is a bootstrap)
and efficiently, as has always been the case with Emacs's build.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 15:03   ` Eli Zaretskii
@ 2021-04-30 15:47     ` Gregory Heytings
  2021-04-30 16:00       ` Eli Zaretskii
  0 siblings, 1 reply; 37+ messages in thread
From: Gregory Heytings @ 2021-04-30 15:47 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 48117


>>> These updates should happen automatically, they should not require 
>>> people to "make bootstrap" or manually regenerate loaddefs.el.
>>
>> Would it not make sense to include these changes in their respective 
>> commits?
>
> loaddefs.el is a generated file (and so are the other *-loaddefs.el 
> files).
>

Ah, yes, indeed.

>> Another solution would be to have a cron job running somewhere and 
>> automatically pushing these changes once a day or so.
>
> How can we do that for files that aren't versioned?
>

I have two questions:

Why are these files not versioned?

The generated lisp/loaddefs.el file is AFAICS identical to the 
lisp/ldefs-boot.el (after calling admin/update_autogen -L), which is 
versioned.  Why is the lisp/loaddefs.el generated?





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 15:22 ` Glenn Morris
  2021-04-30 15:38   ` Glenn Morris
@ 2021-04-30 15:48   ` Eli Zaretskii
  2021-05-02  7:28     ` Lars Ingebrigtsen
  1 sibling, 1 reply; 37+ messages in thread
From: Eli Zaretskii @ 2021-04-30 15:48 UTC (permalink / raw)
  To: Glenn Morris; +Cc: 48117

> From: Glenn Morris <rgm@gnu.org>
> Cc: 48117@debbugs.gnu.org
> Date: Fri, 30 Apr 2021 11:22:00 -0400
> 
> This issue has been present forever.

Yes, I know.  This isn't trivial, or else it would have been solved
long ago.

> 1) autoload generation is slow.

Based on my latest experience, I think this is somewhat exaggerated,
especially given that our builds became slower lately.

> 2) the dependencies of the loaddefs files are unknown to make,
> and are basically "all lisp files". (You can't even say "just those
> files with autoload statements", because removing a previously existing
> autoload statement changes the output.)
> 
> 3) Traditionally, re-making loaddefs files could make trivial changes
> to the output that weren't important (eg ordering of the "no
> autoloads" section, timestamping), but would still trigger re-dumping emacs.
> Which could then trigger regeneration of the autoloads, and
> re-dumping, etc.  This may be better nowadays, since there is no
> longer timestamp information in the loaddefs files (see autoload-timestamps).

What worries me the most is that when 'autoloads' is run (and it is,
from time to time), we still end up with outdated loaddefs.el.  I
think we could live with outdated loaddefs.el for short periods of
time, but it looks like running 'autoloads' only updates the part(s)
of the file for Lisp files that the build thinks to be responsible for
the update.  Or something like that, because how else to explain that
some parts remain outdated?





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 15:38   ` Glenn Morris
@ 2021-04-30 15:51     ` Eli Zaretskii
  2021-04-30 15:59     ` Andreas Schwab
  2021-04-30 17:25     ` Glenn Morris
  2 siblings, 0 replies; 37+ messages in thread
From: Eli Zaretskii @ 2021-04-30 15:51 UTC (permalink / raw)
  To: Glenn Morris; +Cc: 48117

> From: Glenn Morris <rgm@gnu.org>
> Cc: 48117@debbugs.gnu.org
> Date: Fri, 30 Apr 2021 11:38:43 -0400
> 
> One idea, don't know if workable:
> always regenerate loaddefs, but using a temporary file name for the
> main loaddefs.
> use build-aux/move-if-change to only replace the real loaddefs file if
> there have been changes. Although again, it's possible these would not
> be "real" (ie significant) changes.

Thanks, I think it's worth trying.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 15:38   ` Glenn Morris
  2021-04-30 15:51     ` Eli Zaretskii
@ 2021-04-30 15:59     ` Andreas Schwab
  2021-04-30 16:03       ` Eli Zaretskii
  2021-04-30 17:10       ` Glenn Morris
  2021-04-30 17:25     ` Glenn Morris
  2 siblings, 2 replies; 37+ messages in thread
From: Andreas Schwab @ 2021-04-30 15:59 UTC (permalink / raw)
  To: Glenn Morris; +Cc: 48117

On Apr 30 2021, Glenn Morris wrote:

> always regenerate loaddefs, but using a temporary file name for the
> main loaddefs.
> use build-aux/move-if-change to only replace the real loaddefs file if
> there have been changes.

That's already what batch-update-autoloads does.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 15:47     ` Gregory Heytings
@ 2021-04-30 16:00       ` Eli Zaretskii
  2021-04-30 17:32         ` Gregory Heytings
  0 siblings, 1 reply; 37+ messages in thread
From: Eli Zaretskii @ 2021-04-30 16:00 UTC (permalink / raw)
  To: Gregory Heytings; +Cc: 48117

> Date: Fri, 30 Apr 2021 15:47:51 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 48117@debbugs.gnu.org
> 
> Why are these files not versioned?

Because we try not to keep in Git any files that are generated by the
build.  It's redundant to have them there, and also causes frequent
unnecessary conflicts.

> The generated lisp/loaddefs.el file is AFAICS identical to the 
> lisp/ldefs-boot.el (after calling admin/update_autogen -L), which is 
> versioned.  Why is the lisp/loaddefs.el generated?

ldefs-boot.el is versioned because it is needed for the initial build
of a fresh clone, and it is not identical to loaddefs.el, especially
not when you are developing.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 15:59     ` Andreas Schwab
@ 2021-04-30 16:03       ` Eli Zaretskii
  2021-04-30 16:21         ` Andreas Schwab
  2021-04-30 17:10       ` Glenn Morris
  1 sibling, 1 reply; 37+ messages in thread
From: Eli Zaretskii @ 2021-04-30 16:03 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: rgm, 48117

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  48117@debbugs.gnu.org
> Date: Fri, 30 Apr 2021 17:59:45 +0200
> 
> On Apr 30 2021, Glenn Morris wrote:
> 
> > always regenerate loaddefs, but using a temporary file name for the
> > main loaddefs.
> > use build-aux/move-if-change to only replace the real loaddefs file if
> > there have been changes.
> 
> That's already what batch-update-autoloads does.

Judging by the results, that's not so.  Specifically,
batch-update-autoloads doesn't regenerate _all_ of the loaddefs.el,
only some of its part(s).





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 16:03       ` Eli Zaretskii
@ 2021-04-30 16:21         ` Andreas Schwab
  2021-04-30 17:54           ` Eli Zaretskii
  0 siblings, 1 reply; 37+ messages in thread
From: Andreas Schwab @ 2021-04-30 16:21 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: rgm, 48117

On Apr 30 2021, Eli Zaretskii wrote:

> Judging by the results, that's not so.  Specifically,
> batch-update-autoloads doesn't regenerate _all_ of the loaddefs.el,
> only some of its part(s).

Yes, because it calls update-directory-autoloads for each directory.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 15:59     ` Andreas Schwab
  2021-04-30 16:03       ` Eli Zaretskii
@ 2021-04-30 17:10       ` Glenn Morris
  1 sibling, 0 replies; 37+ messages in thread
From: Glenn Morris @ 2021-04-30 17:10 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: 48117

Andreas Schwab wrote:

> That's already what batch-update-autoloads does.

Shows how much my memory is worth! :)





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 15:38   ` Glenn Morris
  2021-04-30 15:51     ` Eli Zaretskii
  2021-04-30 15:59     ` Andreas Schwab
@ 2021-04-30 17:25     ` Glenn Morris
  2021-04-30 17:58       ` Eli Zaretskii
  2021-05-02 16:59       ` Glenn Morris
  2 siblings, 2 replies; 37+ messages in thread
From: Glenn Morris @ 2021-04-30 17:25 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 48117


Another idea, FWIW:
In the make rule for $(lisp)/loaddefs.el, if loaddefs.el exists and is
older than ldefs-boot.el, start by copying the latter to the former.
This ought to limit how outdated loaddefs can get?






^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 16:00       ` Eli Zaretskii
@ 2021-04-30 17:32         ` Gregory Heytings
  2021-04-30 18:57           ` Eli Zaretskii
  0 siblings, 1 reply; 37+ messages in thread
From: Gregory Heytings @ 2021-04-30 17:32 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 48117


>> The generated lisp/loaddefs.el file is AFAICS identical to the 
>> lisp/ldefs-boot.el (after calling admin/update_autogen -L), which is 
>> versioned.  Why is the lisp/loaddefs.el generated?
>
> ldefs-boot.el is versioned because it is needed for the initial build of 
> a fresh clone, and it is not identical to loaddefs.el, especially not 
> when you are developing.
>

Can ldefs-boot.el not be used to detect whether loaddefs.el needs to be 
regenerated?  ISTM that if one assumes that ldefs-boot.el is kept up to 
date, it is necessary to regenerate the loaddefs file only when 
ldefs-boot.el becomes more recent than the loaddefs file.  This would 
avoid rebuilding the loaddefs file on each make invocation.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 16:21         ` Andreas Schwab
@ 2021-04-30 17:54           ` Eli Zaretskii
  0 siblings, 0 replies; 37+ messages in thread
From: Eli Zaretskii @ 2021-04-30 17:54 UTC (permalink / raw)
  To: Andreas Schwab; +Cc: rgm, 48117

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: rgm@gnu.org,  48117@debbugs.gnu.org
> Date: Fri, 30 Apr 2021 18:21:07 +0200
> 
> On Apr 30 2021, Eli Zaretskii wrote:
> 
> > Judging by the results, that's not so.  Specifically,
> > batch-update-autoloads doesn't regenerate _all_ of the loaddefs.el,
> > only some of its part(s).
> 
> Yes, because it calls update-directory-autoloads for each directory.

AFAIU, Glenn's proposal was to process all the directories.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 17:25     ` Glenn Morris
@ 2021-04-30 17:58       ` Eli Zaretskii
  2021-05-02 16:59       ` Glenn Morris
  1 sibling, 0 replies; 37+ messages in thread
From: Eli Zaretskii @ 2021-04-30 17:58 UTC (permalink / raw)
  To: Glenn Morris; +Cc: 48117

> From: Glenn Morris <rgm@gnu.org>
> Cc: 48117@debbugs.gnu.org
> Date: Fri, 30 Apr 2021 13:25:36 -0400
> 
> Another idea, FWIW:
> In the make rule for $(lisp)/loaddefs.el, if loaddefs.el exists and is
> older than ldefs-boot.el, start by copying the latter to the former.
> This ought to limit how outdated loaddefs can get?

Yes, I think this is a good idea.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 17:32         ` Gregory Heytings
@ 2021-04-30 18:57           ` Eli Zaretskii
  2021-04-30 19:08             ` Gregory Heytings
  0 siblings, 1 reply; 37+ messages in thread
From: Eli Zaretskii @ 2021-04-30 18:57 UTC (permalink / raw)
  To: Gregory Heytings; +Cc: 48117

> Date: Fri, 30 Apr 2021 17:32:13 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 48117@debbugs.gnu.org
> 
> Can ldefs-boot.el not be used to detect whether loaddefs.el needs to be 
> regenerated?

I believe that's what Glenn suggested a few minutes ago.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 18:57           ` Eli Zaretskii
@ 2021-04-30 19:08             ` Gregory Heytings
  2021-04-30 19:10               ` Eli Zaretskii
  0 siblings, 1 reply; 37+ messages in thread
From: Gregory Heytings @ 2021-04-30 19:08 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 48117


>> Can ldefs-boot.el not be used to detect whether loaddefs.el needs to be 
>> regenerated?
>
> I believe that's what Glenn suggested a few minutes ago.
>

Yes, it's more or less the same idea, our mails crossed each other because 
of the lists.gnu.org delivery time.

Note that, as I said, this won't work without keeping ldefs-boot.el up to 
date.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 19:08             ` Gregory Heytings
@ 2021-04-30 19:10               ` Eli Zaretskii
  2021-04-30 19:20                 ` Gregory Heytings
  0 siblings, 1 reply; 37+ messages in thread
From: Eli Zaretskii @ 2021-04-30 19:10 UTC (permalink / raw)
  To: Gregory Heytings; +Cc: 48117

> Date: Fri, 30 Apr 2021 19:08:48 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 48117@debbugs.gnu.org
> 
> Note that, as I said, this won't work without keeping ldefs-boot.el up to 
> date.

We (read: Glenn) already do.  You can see that in "git log".





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 19:10               ` Eli Zaretskii
@ 2021-04-30 19:20                 ` Gregory Heytings
  2021-05-01  0:16                   ` Glenn Morris
  0 siblings, 1 reply; 37+ messages in thread
From: Gregory Heytings @ 2021-04-30 19:20 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 48117


>> Note that, as I said, this won't work without keeping ldefs-boot.el up 
>> to date.
>
> We (read: Glenn) already do.  You can see that in "git log".
>

I think this should preferably be automated.  The current ldefs-boot.el 
file in the trunk is 25 days old, running admin/update_autogen -L adds 247 
lines and modifies 100 lines.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 19:20                 ` Gregory Heytings
@ 2021-05-01  0:16                   ` Glenn Morris
  2021-05-01  8:09                     ` Gregory Heytings
  0 siblings, 1 reply; 37+ messages in thread
From: Glenn Morris @ 2021-05-01  0:16 UTC (permalink / raw)
  To: Gregory Heytings; +Cc: 48117

Gregory Heytings wrote:

> I think this should preferably be automated.  The current
> ldefs-boot.el file in the trunk is 25 days old, running
> admin/update_autogen -L adds 247 lines and modifies 100 lines.

It gets automaticallly updated on the first of the month.
It could be more frequent, but it would seem like churn for no real
reason most of the time.
(It wouldn't make any difference to the issue that prompted this report.)





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-05-01  0:16                   ` Glenn Morris
@ 2021-05-01  8:09                     ` Gregory Heytings
  2021-05-01  8:25                       ` Eli Zaretskii
  0 siblings, 1 reply; 37+ messages in thread
From: Gregory Heytings @ 2021-05-01  8:09 UTC (permalink / raw)
  To: Glenn Morris; +Cc: 48117


>> I think this should preferably be automated.  The current ldefs-boot.el 
>> file in the trunk is 25 days old, running admin/update_autogen -L adds 
>> 247 lines and modifies 100 lines.
>
> It gets automaticallly updated on the first of the month.  It could be 
> more frequent, but it would seem like churn for no real reason most of 
> the time.
>

IMO once a week would be better.  An outdated ldefs-boot creates 
unnecessary warnings during the build.

>
> (It wouldn't make any difference to the issue that prompted this 
> report.)
>

That depends on the chosen solution.  You suggested copying ldefs-boot.el 
onto loaddefs.el when it is more recent.  I'm not entirely sure, but it 
seems to me that forcing a regeneration of the loaddefs files during make 
whenever ldefs-boot.el is more recent than loaddefs.el would be a better 
solution.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-05-01  8:09                     ` Gregory Heytings
@ 2021-05-01  8:25                       ` Eli Zaretskii
  2021-05-01  9:20                         ` Gregory Heytings
  0 siblings, 1 reply; 37+ messages in thread
From: Eli Zaretskii @ 2021-05-01  8:25 UTC (permalink / raw)
  To: Gregory Heytings; +Cc: rgm, 48117

> Date: Sat, 01 May 2021 08:09:42 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, 48117@debbugs.gnu.org
> 
> You suggested copying ldefs-boot.el onto loaddefs.el when it is more
> recent.  I'm not entirely sure, but it seems to me that forcing a
> regeneration of the loaddefs files during make whenever
> ldefs-boot.el is more recent than loaddefs.el would be a better
> solution.

Your proposal would produce marginally better results for a
significantly longer build time, so I don't think it's a net win.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-05-01  8:25                       ` Eli Zaretskii
@ 2021-05-01  9:20                         ` Gregory Heytings
  2021-05-01  9:34                           ` Eli Zaretskii
  0 siblings, 1 reply; 37+ messages in thread
From: Gregory Heytings @ 2021-05-01  9:20 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: rgm, 48117


>> You suggested copying ldefs-boot.el onto loaddefs.el when it is more 
>> recent.  I'm not entirely sure, but it seems to me that forcing a 
>> regeneration of the loaddefs files during make whenever ldefs-boot.el 
>> is more recent than loaddefs.el would be a better solution.
>
> Your proposal would produce marginally better results for a 
> significantly longer build time, so I don't think it's a net win.
>

If ldefs-boot.el is updated, say, once a week, this would force the 
regeneration of the loaddefs files at most once a week.  Wouldn't that be 
a reasonable compromise?  On my computer, regenerating the loaddefs files 
takes about 10 seconds, or ~3% of the time of a make bootstrap.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-05-01  9:20                         ` Gregory Heytings
@ 2021-05-01  9:34                           ` Eli Zaretskii
  2021-05-01 12:29                             ` Gregory Heytings
  0 siblings, 1 reply; 37+ messages in thread
From: Eli Zaretskii @ 2021-05-01  9:34 UTC (permalink / raw)
  To: Gregory Heytings; +Cc: rgm, 48117

> Date: Sat, 01 May 2021 09:20:51 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: rgm@gnu.org, 48117@debbugs.gnu.org
> 
> > Your proposal would produce marginally better results for a 
> > significantly longer build time, so I don't think it's a net win.
> 
> If ldefs-boot.el is updated, say, once a week, this would force the 
> regeneration of the loaddefs files at most once a week.  Wouldn't that be 
> a reasonable compromise?

Compromise between which alternatives?

The "marginally better" results in your proposal are that if someone
updates from Git when he/she is in the middle of some development,
then loaddefs.el are made up-to-date immediately, as opposed to
_maybe_ waiting for the next update of ldefs-boot.el.  (I say "maybe"
because in general loaddefs.el _are_ updated as part of routine
builds, just not 100% reliably so.)

> On my computer, regenerating the loaddefs files takes about 10
> seconds, or ~3% of the time of a make bootstrap.

Keep in mind that some people use less powerful machines.  And the
bootstrap time is not relevant, because loaddefs.el is completely
regenerated during bootstrap anyway.  The time that is relevant is the
time of just "make", and that is usually quite short, even on slow
machines.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-05-01  9:34                           ` Eli Zaretskii
@ 2021-05-01 12:29                             ` Gregory Heytings
  2021-05-01 13:00                               ` Eli Zaretskii
  0 siblings, 1 reply; 37+ messages in thread
From: Gregory Heytings @ 2021-05-01 12:29 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: rgm, 48117


>>> Your proposal would produce marginally better results for a 
>>> significantly longer build time, so I don't think it's a net win.
>>
>> If ldefs-boot.el is updated, say, once a week, this would force the 
>> regeneration of the loaddefs files at most once a week.  Wouldn't that 
>> be a reasonable compromise?
>
> Compromise between which alternatives?
>

I see at least the following possible alternatives:

- the current situation, which you describe in your original post, in 
which the loaddefs files need to be regenerated manually (inconvenient)

- regenerating the loaddefs files for each make invocation (inefficient)

- copying the ldefs-boot.el onto loaddefs.el when it is more recent (which 
IIUC could lose local additions to loaddefs.el)

- automatically regenerating the loaddefs files when ldefs-boot.el is more 
recent than loaddefs.el (which I understand could be a bit slow)

- issue only a warning when make is invoked and ldefs-boot.el is more 
recent than loaddefs.el (and perhaps add a autoloads or loaddefs target to 
the main Makefile)





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-05-01 12:29                             ` Gregory Heytings
@ 2021-05-01 13:00                               ` Eli Zaretskii
  0 siblings, 0 replies; 37+ messages in thread
From: Eli Zaretskii @ 2021-05-01 13:00 UTC (permalink / raw)
  To: Gregory Heytings; +Cc: rgm, 48117

> Date: Sat, 01 May 2021 12:29:12 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: rgm@gnu.org, 48117@debbugs.gnu.org
> 
> > Compromise between which alternatives?
> >
> 
> I see at least the following possible alternatives:

My point is that the gain from your proposal wrt the next best
alternative is small, but the price is too high.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 15:48   ` Eli Zaretskii
@ 2021-05-02  7:28     ` Lars Ingebrigtsen
  2021-05-02  7:41       ` Eli Zaretskii
  2021-12-08  0:36       ` Stefan Kangas
  0 siblings, 2 replies; 37+ messages in thread
From: Lars Ingebrigtsen @ 2021-05-02  7:28 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: Glenn Morris, 48117

Eli Zaretskii <eliz@gnu.org> writes:

>> 1) autoload generation is slow.
>
> Based on my latest experience, I think this is somewhat exaggerated,
> especially given that our builds became slower lately.

Has it?  I haven't noticed any substantial slow-down...  (Unless you
mean with native-comp switched on.)

In any case, autoload generation takes a substantial chunk of time --
especially since it's single-threaded.  Glenn's suggestion to copy over
ldefs-boot.el (if it's newer) sounds like a good idea to me.  (And I'd
also like if that file was regenerated more often than once a month...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-05-02  7:28     ` Lars Ingebrigtsen
@ 2021-05-02  7:41       ` Eli Zaretskii
  2021-12-08  0:36       ` Stefan Kangas
  1 sibling, 0 replies; 37+ messages in thread
From: Eli Zaretskii @ 2021-05-02  7:41 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: rgm, 48117

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Glenn Morris <rgm@gnu.org>,  48117@debbugs.gnu.org
> Date: Sun, 02 May 2021 09:28:04 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> 1) autoload generation is slow.
> >
> > Based on my latest experience, I think this is somewhat exaggerated,
> > especially given that our builds became slower lately.
> 
> Has it?  I haven't noticed any substantial slow-down...  (Unless you
> mean with native-comp switched on.)

Yes, native-comp, certainly.  But the build became slower a year or so
ago as well, I no longer remember why.

> In any case, autoload generation takes a substantial chunk of time --
> especially since it's single-threaded.  Glenn's suggestion to copy over
> ldefs-boot.el (if it's newer) sounds like a good idea to me.

Yes, I agree.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 17:25     ` Glenn Morris
  2021-04-30 17:58       ` Eli Zaretskii
@ 2021-05-02 16:59       ` Glenn Morris
  2021-05-02 18:54         ` Eli Zaretskii
  2021-05-03  8:41         ` Gregory Heytings
  1 sibling, 2 replies; 37+ messages in thread
From: Glenn Morris @ 2021-05-02 16:59 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 48117

Glenn Morris wrote:

> Another idea, FWIW:
> In the make rule for $(lisp)/loaddefs.el, if loaddefs.el exists and is
> older than ldefs-boot.el, start by copying the latter to the former.
> This ought to limit how outdated loaddefs can get?

BTW, I think this might actually make things worse in terms of detecting
new secondary loaddefs files like texinfo-loaddefs (not that those
happen very often), by bumping the timestamp on the primary loaddefs.






^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-05-02 16:59       ` Glenn Morris
@ 2021-05-02 18:54         ` Eli Zaretskii
  2021-05-03  8:41         ` Gregory Heytings
  1 sibling, 0 replies; 37+ messages in thread
From: Eli Zaretskii @ 2021-05-02 18:54 UTC (permalink / raw)
  To: Glenn Morris; +Cc: 48117

> From: Glenn Morris <rgm@gnu.org>
> Cc: 48117@debbugs.gnu.org
> Date: Sun, 02 May 2021 12:59:17 -0400
> 
> Glenn Morris wrote:
> 
> > Another idea, FWIW:
> > In the make rule for $(lisp)/loaddefs.el, if loaddefs.el exists and is
> > older than ldefs-boot.el, start by copying the latter to the former.
> > This ought to limit how outdated loaddefs can get?
> 
> BTW, I think this might actually make things worse in terms of detecting
> new secondary loaddefs files like texinfo-loaddefs (not that those
> happen very often), by bumping the timestamp on the primary loaddefs.

But there's no such thing as detecting new secondary loaddefs, right?
The only remedy for that is "make autoloads-force", right?





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-05-02 16:59       ` Glenn Morris
  2021-05-02 18:54         ` Eli Zaretskii
@ 2021-05-03  8:41         ` Gregory Heytings
  1 sibling, 0 replies; 37+ messages in thread
From: Gregory Heytings @ 2021-05-03  8:41 UTC (permalink / raw)
  To: Glenn Morris; +Cc: 48117


>> Another idea, FWIW: In the make rule for $(lisp)/loaddefs.el, if 
>> loaddefs.el exists and is older than ldefs-boot.el, start by copying 
>> the latter to the former. This ought to limit how outdated loaddefs can 
>> get?
>
> BTW, I think this might actually make things worse in terms of detecting 
> new secondary loaddefs files like texinfo-loaddefs (not that those 
> happen very often), by bumping the timestamp on the primary loaddefs.
>

Which is one of the reasons why I still believe that my proposed solution 
is better: whenever lisp/ldefs-boot.el is more recent than 
lisp/loaddefs.el, issue a warning when make is invoked to suggest the 
regeneration of _all_ autoload files, and add a command to do this in the 
main Makefile: make autoloads would do find -name '*loaddefs.el' -delete 
&& make -C lisp autoloads.  In theory this should have the same effect as 
make -C lisp autoloads-force, but that command is only marginally faster, 
and I much prefer the clarity / determinicity of a "delete all and 
regenerate" operation.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-05-02  7:28     ` Lars Ingebrigtsen
  2021-05-02  7:41       ` Eli Zaretskii
@ 2021-12-08  0:36       ` Stefan Kangas
  2021-12-08  0:40         ` Lars Ingebrigtsen
  2021-12-08  3:32         ` Eli Zaretskii
  1 sibling, 2 replies; 37+ messages in thread
From: Stefan Kangas @ 2021-12-08  0:36 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: Glenn Morris, 48117

Lars Ingebrigtsen <larsi@gnus.org> writes:

> In any case, autoload generation takes a substantial chunk of time --
> especially since it's single-threaded.  Glenn's suggestion to copy over
> ldefs-boot.el (if it's newer) sounds like a good idea to me.  (And I'd
> also like if that file was regenerated more often than once a month...)

If Eli agrees, I'm happy to update ldefs-boot.el more frequently.

Just let me know how often you want it updated and I will make it so.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-12-08  0:36       ` Stefan Kangas
@ 2021-12-08  0:40         ` Lars Ingebrigtsen
  2021-12-08  3:32         ` Eli Zaretskii
  1 sibling, 0 replies; 37+ messages in thread
From: Lars Ingebrigtsen @ 2021-12-08  0:40 UTC (permalink / raw)
  To: Stefan Kangas; +Cc: Glenn Morris, 48117

Stefan Kangas <stefan@marxist.se> writes:

> If Eli agrees, I'm happy to update ldefs-boot.el more frequently.
>
> Just let me know how often you want it updated and I will make it so.

I usually just update it now if I add an ;;;###autoload somewhere --
updating that file kinda goes hand in hand with that.  (I added a new
"make ldefs-boot.el" Makefile target to make it more convenient.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-12-08  0:36       ` Stefan Kangas
  2021-12-08  0:40         ` Lars Ingebrigtsen
@ 2021-12-08  3:32         ` Eli Zaretskii
  1 sibling, 0 replies; 37+ messages in thread
From: Eli Zaretskii @ 2021-12-08  3:32 UTC (permalink / raw)
  To: Stefan Kangas; +Cc: rgm, larsi, 48117

> From: Stefan Kangas <stefan@marxist.se>
> Date: Wed, 8 Dec 2021 01:36:25 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, Glenn Morris <rgm@gnu.org>, 48117@debbugs.gnu.org
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > In any case, autoload generation takes a substantial chunk of time --
> > especially since it's single-threaded.  Glenn's suggestion to copy over
> > ldefs-boot.el (if it's newer) sounds like a good idea to me.  (And I'd
> > also like if that file was regenerated more often than once a month...)
> 
> If Eli agrees, I'm happy to update ldefs-boot.el more frequently.

I see no down-sides to updating it frequently, so it's fine with me.





^ permalink raw reply	[flat|nested] 37+ messages in thread

* bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable
  2021-04-30 11:51 bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable Eli Zaretskii
  2021-04-30 15:01 ` Gregory Heytings
  2021-04-30 15:22 ` Glenn Morris
@ 2022-06-05 16:25 ` Lars Ingebrigtsen
  2 siblings, 0 replies; 37+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-05 16:25 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 48117

Eli Zaretskii <eliz@gnu.org> writes:

> Then compare the old loaddefs.el with the newly created one: you will
> see many changes, depending on when was the last time you bootstrapped
> or otherwise regenerated loaddefs.el from scratch.
>
> This happens because regeneration of the parts of loaddefs.el affected
> by Lisp changes is unreliable and misses some changes, in particular
> those where autoloads from some Lisp files are redirected to private
> *-loaddefs.el files instead of the common loaddefs.el.

This should now be fixed in Emacs 29 -- the private loaddefs files
updates should be handled as well as the main one.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 37+ messages in thread

end of thread, other threads:[~2022-06-05 16:25 UTC | newest]

Thread overview: 37+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-30 11:51 bug#48117: 28.0.50; Update of loaddefs.el during normal build is unreliable Eli Zaretskii
2021-04-30 15:01 ` Gregory Heytings
2021-04-30 15:03   ` Eli Zaretskii
2021-04-30 15:47     ` Gregory Heytings
2021-04-30 16:00       ` Eli Zaretskii
2021-04-30 17:32         ` Gregory Heytings
2021-04-30 18:57           ` Eli Zaretskii
2021-04-30 19:08             ` Gregory Heytings
2021-04-30 19:10               ` Eli Zaretskii
2021-04-30 19:20                 ` Gregory Heytings
2021-05-01  0:16                   ` Glenn Morris
2021-05-01  8:09                     ` Gregory Heytings
2021-05-01  8:25                       ` Eli Zaretskii
2021-05-01  9:20                         ` Gregory Heytings
2021-05-01  9:34                           ` Eli Zaretskii
2021-05-01 12:29                             ` Gregory Heytings
2021-05-01 13:00                               ` Eli Zaretskii
2021-04-30 15:22 ` Glenn Morris
2021-04-30 15:38   ` Glenn Morris
2021-04-30 15:51     ` Eli Zaretskii
2021-04-30 15:59     ` Andreas Schwab
2021-04-30 16:03       ` Eli Zaretskii
2021-04-30 16:21         ` Andreas Schwab
2021-04-30 17:54           ` Eli Zaretskii
2021-04-30 17:10       ` Glenn Morris
2021-04-30 17:25     ` Glenn Morris
2021-04-30 17:58       ` Eli Zaretskii
2021-05-02 16:59       ` Glenn Morris
2021-05-02 18:54         ` Eli Zaretskii
2021-05-03  8:41         ` Gregory Heytings
2021-04-30 15:48   ` Eli Zaretskii
2021-05-02  7:28     ` Lars Ingebrigtsen
2021-05-02  7:41       ` Eli Zaretskii
2021-12-08  0:36       ` Stefan Kangas
2021-12-08  0:40         ` Lars Ingebrigtsen
2021-12-08  3:32         ` Eli Zaretskii
2022-06-05 16:25 ` Lars Ingebrigtsen

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).