*-loaddefs.el files are not byte-compiled after being regenerated

*-loaddefs.el files are not byte-compiled after being regenerated

[Eli Zaretskii]

Something's wrong with the Makefile rules for these files: I see quite
a lot that they are regenerated, but the results are not subsequently
byte-compiled, causing warnings when I start Emacs after building it.

Re: *-loaddefs.el files are not byte-compiled after being regenerated

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Something's wrong with the Makefile rules for these files: I see quite
> a lot that they are regenerated, but the results are not subsequently
> byte-compiled, causing warnings when I start Emacs after building it.

Sounds like the loaddefs files are being generated too early in some
circumstances to be picked up by compile-main?

Do you have a recipe to reproduce this problem?

Hm...  I got rid of the VCWITNESS rule because I didn't think it should
be necessary any more, but perhaps the ordering doesn't quite work
without it always...

Re: *-loaddefs.el files are not byte-compiled after being regenerated

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Sounds like the loaddefs files are being generated too early in some
> circumstances to be picked up by compile-main?

Never mind, it's simple:

touch lisp/mh-e/mh-junk.el
make

will generate lisp/mh-e/mh-loaddefs.el, but not compile it.

Re: *-loaddefs.el files are not byte-compiled after being regenerated

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Never mind, it's simple:
>
> touch lisp/mh-e/mh-junk.el
> make
>
> will generate lisp/mh-e/mh-loaddefs.el, but not compile it.

OK, I think this should be fixed now.

Re: *-loaddefs.el files are not byte-compiled after being regenerated

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 04 Aug 2022 18:53:36 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Something's wrong with the Makefile rules for these files: I see quite
> > a lot that they are regenerated, but the results are not subsequently
> > byte-compiled, causing warnings when I start Emacs after building it.
> 
> Sounds like the loaddefs files are being generated too early in some
> circumstances to be picked up by compile-main?
> 
> Do you have a recipe to reproduce this problem?

Unfortunately, no.

I use "make -j4", maybe that somehow triggers the problem.

Another thing that perhaps is relevant is that the generation of the
*-loaddefs.el files happens in parallel with (and close to the end of)
the byte-compilation of updated *.el files.  Here's an example:

  Finding pointers to doc strings...
  Finding pointers to doc strings...done
  Growing hash table to: 120000
  Pure-hashed: 18227 strings, 5621 vectors, 50818 conses, 5088 bytecodes, 330 others
  Dumping under the name emacs.pdmp
  Dumping fingerprint: 4d49c5e06c09d7ec789816d97b378812d91a300716e9553969c02f8f43918a9d
  Dump complete
  Byte counts: header=100 hot=7205544 discardable=130440 cold=3728296
  Reloc counts: hot=439088 discardable=5359
  Adding name emacs-29.0.50.1611.exe
  Adding name emacs-29.0.50.1611.pdmp
  cp -f emacs.pdmp bootstrap-emacs.pdmp
  make[1]: Leaving directory `/d/gnu/git/emacs/trunk/src'
  make -C lisp all
  make[1]: Entering directory `/d/gnu/git/emacs/trunk/lisp'
  make -C ../leim all EMACS="../src/emacs.exe"
  make -C ../admin/grammars all EMACS="../../src/emacs.exe"
  make[2]: Entering directory `/d/gnu/git/emacs/trunk/admin/grammars'
  make[2]: Nothing to be done for `all'.
  make[2]: Leaving directory `/d/gnu/git/emacs/trunk/admin/grammars'
  make[2]: Entering directory `/d/gnu/git/emacs/trunk/leim'
  make[2]: Nothing to be done for `all'.
  make[2]: Leaving directory `/d/gnu/git/emacs/trunk/leim'
    GEN      autoloads
  rm ./emacs-lisp/autoload.elc
  rm ./net/netrc.elc
  make[2]: Entering directory `/d/gnu/git/emacs/trunk/lisp'
  make[2]: Nothing to be done for `compile-targets'.
  make[2]: Leaving directory `/d/gnu/git/emacs/trunk/lisp'
  Source file `d:/gnu/git/emacs/trunk/lisp/emacs-lisp/loaddefs-gen.el' newer than byte-compiled file; using older file
    INFO     Scraping files for loaddefs...
  make[2]: Entering directory `/d/gnu/git/emacs/trunk/doc/misc'
  make[2]: `org.texi' is up to date.
  make[2]: `modus-themes.texi' is up to date.
  make[2]: Leaving directory `/d/gnu/git/emacs/trunk/doc/misc'
  make[2]: Entering directory `/d/gnu/git/emacs/trunk/leim'
  make[2]: Nothing to be done for `generate-ja-dic'.
  make[2]: Leaving directory `/d/gnu/git/emacs/trunk/leim'
  make[2]: Entering directory `/d/gnu/git/emacs/trunk/lisp'
  make[2]: Nothing to be done for `compile-targets'.
  make[2]: Leaving directory `/d/gnu/git/emacs/trunk/lisp'
    INFO     Scraping files for loaddefs...10%
  make[2]: Entering directory `/d/gnu/git/emacs/trunk/lisp'
    ELC      auth-source.elc
    ELC      cus-dep.elc
    ELC      dired-aux.elc
    ELC      dired-loaddefs.elc
    ELC      find-dired.elc
    ELC      ibuffer-loaddefs.elc
    ELC      makesum.elc
    ELC      ps-print-loaddefs.elc
    INFO     Scraping files for loaddefs...20%
    ELC      calc/calc-loaddefs.elc
    ELC      calendar/cal-loaddefs.elc
    ELC      calendar/diary-loaddefs.elc
    ELC      calendar/holiday-loaddefs.elc
    ELC      cedet/ede/loaddefs.elc
    ELC      cedet/semantic/loaddefs.elc
    ELC      cedet/srecode/loaddefs.elc
    INFO     Scraping files for loaddefs...30%
  make[2]: Leaving directory `/d/gnu/git/emacs/trunk/lisp'
  make[2]: Entering directory `/d/gnu/git/emacs/trunk/lisp'
    ELC      emacs-lisp/checkdoc.elc
    ELC      emacs-lisp/cl-loaddefs.elc
    ELC      emacs-lisp/loaddefs-gen.elc
    INFO     Scraping files for loaddefs...40%
    ELC      emacs-lisp/package.elc
    ELC      erc/erc-loaddefs.elc
    ELC      eshell/em-extpipe.elc
    ELC      eshell/esh-groups.elc
    ELC      gnus/nnimap.elc
    INFO     Scraping files for loaddefs...50%
    ELC      gnus/nntp.elc
    INFO     Scraping files for loaddefs...60%
  make[2]: Leaving directory `/d/gnu/git/emacs/trunk/lisp'
    INFO     Scraping files for loaddefs...70%
  make[2]: Entering directory `/d/gnu/git/emacs/trunk/lisp'
    ELC      mh-e/mh-loaddefs.elc
    ELC      net/eww.elc
    ELC      net/tramp-adb.elc
    ELC      net/tramp-loaddefs.elc
    INFO     Scraping files for loaddefs...80%
    ELC      net/tramp-sh.elc
    ELC      net/tramp.elc
    ELC      obsolete/autoload.elc
    ELC      obsolete/netrc.elc
    INFO     Scraping files for loaddefs...90%
    ELC      org/ob-lilypond.elc
    ELC      org/org-loaddefs.elc
    ELC      progmodes/meta-mode.elc
    INFO     Scraping files for loaddefs...100%
    INFO     Scraping files for loaddefs...done
    GEN      textmodes/reftex-loaddefs.el
    GEN      textmodes/texinfo-loaddefs.el
    GEN      org/org-loaddefs.el
    GEN      net/tramp-loaddefs.el
    GEN      mh-e/mh-loaddefs.el
    GEN      eshell/esh-groups.el
    GEN      erc/erc-loaddefs.el
    GEN      emacs-lisp/cl-loaddefs.el
    GEN      cedet/srecode/loaddefs.el
    GEN      cedet/semantic/loaddefs.el
    GEN      cedet/ede/loaddefs.el
  make[2]: Leaving directory `/d/gnu/git/emacs/trunk/lisp'
    GEN      calendar/diary-loaddefs.el
    GEN      calendar/holiday-loaddefs.el
    GEN      calendar/cal-loaddefs.el
    GEN      calc/calc-loaddefs.el
    GEN      ps-print-loaddefs.el
    GEN      ibuffer-loaddefs.el
    GEN      dired-loaddefs.el
  make[2]: Entering directory `/d/gnu/git/emacs/trunk/lisp'
    ELC      textmodes/reftex-loaddefs.elc
    ELC      textmodes/texinfo-loaddefs.elc
  make[2]: Leaving directory `/d/gnu/git/emacs/trunk/lisp'
    GEN      loaddefs.el
  make[1]: Leaving directory `/d/gnu/git/emacs/trunk/lisp'
  make -C doc/misc info
  make[1]: Entering directory `/d/gnu/git/emacs/trunk/doc/misc'
  make[1]: Nothing to be done for `info'.
  make[1]: Leaving directory `/d/gnu/git/emacs/trunk/doc/misc'

Shouldn't the loaddefs files be the dependency of every .elc file?

Re: *-loaddefs.el files are not byte-compiled after being regenerated

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 04 Aug 2022 19:10:56 +0200
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Never mind, it's simple:
> >
> > touch lisp/mh-e/mh-junk.el
> > make
> >
> > will generate lisp/mh-e/mh-loaddefs.el, but not compile it.
> 
> OK, I think this should be fixed now.

Thanks.

Re: *-loaddefs.el files are not byte-compiled after being regenerated

[Stefan Monnier]

>     ELC      textmodes/reftex-loaddefs.elc
>     ELC      textmodes/texinfo-loaddefs.elc

Why do we compile those?  I thought we only wanted to compile
`lisp/loaddefs.el`.


        Stefan

Re: *-loaddefs.el files are not byte-compiled after being regenerated

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Why do we compile those?  I thought we only wanted to compile
> `lisp/loaddefs.el`.

To make

;;;###foo-autoload
(progn
  (defun foo ()
    (here's a big bunch of code)))

byte-compiled.

Re: *-loaddefs.el files are not byte-compiled after being regenerated

[Stefan Monnier]

Lars Ingebrigtsen [2022-08-05 00:26:29] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> Why do we compile those?  I thought we only wanted to compile
>> `lisp/loaddefs.el`.
>
> To make
>
> ;;;###foo-autoload
> (progn
>   (defun foo ()
>     (here's a big bunch of code)))
>
> byte-compiled.

Ah, I thought the change was only about the handling of docstrings for
`etc/DOC`, but indeed there are other benefits to byte-compiling
loaddefs files.  Not sure they're important enough, tho (we still don't
compile ELPA packages's autoloads, IIUC).


        Stefan

Re: *-loaddefs.el files are not byte-compiled after being regenerated

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Ah, I thought the change was only about the handling of docstrings for
> `etc/DOC`, but indeed there are other benefits to byte-compiling
> loaddefs files.

It's also nice to have the byte compiler help you get the logic right.
For instance, in Emacs 28, saying

(require 'tramp-loaddefs)

will bug out, but byte-compiling it shows where the problem is, so it
was trivial to fix.

Not that this is a major deal either, but it's just one more thing the
byte-compiler can worry about so that us poor human beans don't have to
work as hard.

Re: *-loaddefs.el files are not byte-compiled after being regenerated

[Stefan Monnier]

> It's also nice to have the byte compiler help you get the logic right.

Indeed.  I guess we should try and compile the autoloads in ELPA
packages as well, then (actually we do compile them when they're put
into the quickstart file).


        Stefan

Re: *-loaddefs.el files are not byte-compiled after being regenerated

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Indeed.  I guess we should try and compile the autoloads in ELPA
> packages as well, then (actually we do compile them when they're put
> into the quickstart file).

Yes, I think that makes sense, but then we'd have to fix the #$
compilation problem first.