Reworking loaddefs.el generation

Reworking loaddefs.el generation

[Lars Ingebrigtsen]

I've been working a bit on making loaddef file generation faster (and
easier to comrehend).  Building loaddefs currently takes about 10
seconds (on my build machine), and I've got it down to 2 seconds.  (For
my "make -j32" build, this means a reduction of 6% time, since that bit
is single-threaded and make doesn't have anything else to schedule while
Emacs is working.)

But there's some bits here I don't quite understand.  We have, for
instance:

# Update MH-E internal autoloads. These are not to be confused with
# the autoloads for the MH-E entry points, which are already in loaddefs.el.
MH_E_DIR = $(lisp)/mh-e
MH_E_SRC = $(sort $(wildcard ${MH_E_DIR}/mh*.el))
MH_E_SRC := $(filter-out ${MH_E_DIR}/mh-loaddefs.el,${MH_E_SRC})

.PHONY: mh-autoloads
mh-autoloads: $(MH_E_DIR)/mh-loaddefs.el
$(MH_E_DIR)/mh-loaddefs.el: $(MH_E_SRC)
	$(emacs) -l autoload \
	   --eval "(setq generate-autoload-cookie \";;;###mh-autoload\")" \
	   --eval "(setq generated-autoload-file (expand-file-name (unmsys--file-name \"$@\")))" \
	   -f batch-update-autoloads $(MH_E_DIR)

But if I say:

$ touch lisp/mh-e/mh-e.el
$ make

The lisp/mh-e/mh-loaddefs.el is not regenerated.  So either this
Makefile snippet doesn't work, or I'm misreading what it's trying to do
(which is quite possible).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Andreas Schwab]

On Mai 27 2022, Lars Ingebrigtsen wrote:

> $ touch lisp/mh-e/mh-e.el
> $ make
>
> The lisp/mh-e/mh-loaddefs.el is not regenerated.

make -C lisp mh-autoloads

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

>> $ touch lisp/mh-e/mh-e.el
>> $ make
>>
>> The lisp/mh-e/mh-loaddefs.el is not regenerated.
>
> make -C lisp mh-autoloads

But that will make the rule run always (that is, until the
mh-loaddefs.el file gets updated because new autoloads have actually
arrived).

If that's the intention, then I guess I understood the rule -- it's just
very well implemented, which I kinda suspected.

In any case, in the new implementation, I've removed those rules --
"make loaddefs.el" will regenerate all these loaddefs files.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Stefan Monnier]

> I've been working a bit on making loaddef file generation faster (and
> easier to comrehend).  Building loaddefs currently takes about 10
> seconds (on my build machine), and I've got it down to 2 seconds.  (For
> my "make -j32" build, this means a reduction of 6% time, since that bit
> is single-threaded and make doesn't have anything else to schedule while
> Emacs is working.)

BTW, I think `autoload.el` should be extended to provide explicit
support for these things like `mh-autoloads.el` so we don't need these
ad-hoc rules.

Eg.g. I was thinking that file-local variables should be able to include
things like:

    ;; Local Variables:
    ;; autoload-extra-cookies: ((mh "mh-loaddefs.el") (tramp "../net/tramp-loaddefs.el"))
    ;; End:

which would tell `autoload.el` to look for
";;;###\\(mh-\\|tramp-\\)?autoload" matches and depending on the exact
match put the resulting entry either in the normal `loaddefs.el`, or in
`mh-loaddefs.el` or in `tramp-loaddefs.el`.


        Stefan

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> BTW, I think `autoload.el` should be extended to provide explicit
> support for these things like `mh-autoloads.el` so we don't need these
> ad-hoc rules.

Yes, my new version has that.  😀

> Eg.g. I was thinking that file-local variables should be able to include
> things like:
>
>     ;; Local Variables:
>     ;; autoload-extra-cookies: ((mh "mh-loaddefs.el") (tramp "../net/tramp-loaddefs.el"))
>     ;; End:
>
> which would tell `autoload.el` to look for
> ";;;###\\(mh-\\|tramp-\\)?autoload" matches and depending on the exact
> match put the resulting entry either in the normal `loaddefs.el`, or in
> `mh-loaddefs.el` or in `tramp-loaddefs.el`.

I'm doing the latter without the former, and that seems to work fine.
That is, tramp.el currently has 

;;;###tramp-autoload
;;;###autoload

that says exactly this, so my new code just uses that.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I've been working a bit on making loaddef file generation faster (and
> easier to comrehend).  Building loaddefs currently takes about 10
> seconds (on my build machine), and I've got it down to 2 seconds.  (For
> my "make -j32" build, this means a reduction of 6% time, since that bit
> is single-threaded and make doesn't have anything else to schedule while
> Emacs is working.)

After testing across Debian/Windows/Macos as well as package
installation, I've now pushed this to the trunk.  Give a shout if it
broke anything.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> After testing across Debian/Windows/Macos as well as package
> installation, I've now pushed this to the trunk.  Give a shout if it
> broke anything.

Found a bug -- it's rebuilding Emacs a bit too much now.  I'll get
fixing...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Found a bug -- it's rebuilding Emacs a bit too much now.  I'll get
> fixing...

Well, it's just a bit more rebuildey now -- it rebuilds Emacs after
pulling even when no .el/.c files have been changed.  But that's pretty
rare, actually, so I'm not fixing that immediately.

But one thing I didn't quite consider what that this used to take:

[larsi@stories ~/src/emacs/trunk]$ touch .git/logs/HEAD lisp/gnus/gnus.el; time make -j32
eal	0m3.210s
user	0m3.055s
sys	0m0.247s

This simulates a "git pull" that only changes gnus.el.  In the new
version, this is slower:

real	0m4.761s
user	0m4.610s
sys	0m0.249s

So while a "make bootstrap" is faster, this common case is now slower.
I'll see what I can do about that.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 31 May 2022 18:53:53 +0200
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Found a bug -- it's rebuilding Emacs a bit too much now.  I'll get
> > fixing...
> 
> Well, it's just a bit more rebuildey now -- it rebuilds Emacs after
> pulling even when no .el/.c files have been changed.  But that's pretty
> rare, actually, so I'm not fixing that immediately.

AFAICS, it also regenerates all the loaddefs files after commit+push.

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> AFAICS, it also regenerates all the loaddefs files after commit+push.

Yes, whenever .git/logs/HEAD is updated (and I think a commit and/or
push does that, too)?

I've got a solution for this now, and is testing it on different
machines, so I'll be pushing that later tonight.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So while a "make bootstrap" is faster, this common case is now slower.
> I'll see what I can do about that.

Should be fixed now -- this case is now also faster than before, but
marginally.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Alan Mackenzie]

Hello, Lars.

On Fri, May 27, 2022 at 12:53:54 +0200, Lars Ingebrigtsen wrote:
> I've been working a bit on making loaddef file generation faster (and
> easier to comrehend).  Building loaddefs currently takes about 10
> seconds (on my build machine), and I've got it down to 2 seconds.  (For
> my "make -j32" build, this means a reduction of 6% time, since that bit
> is single-threaded and make doesn't have anything else to schedule while
> Emacs is working.)

[ .... ]

While on the subject of build times, another frustrating delay happens
with the generation of .texi intermediate files from those doc files
whose source is .org.  There are two of these, modus-themes.texi and
org.texi.

The first isn't too bad, because it's small, but org.texi is taking
~20 seconds to generate on my machine (which is doing nothing else at
the time) plus, perhaps, 3 seconds for org.info.

Would it perhaps be possible to set the documentation generation going
at an earlier stage of the bootstrap, in effect spreading out the
slowness over the entire battery of processor cores?

> -- 
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Alan Mackenzie <acm@muc.de> writes:

> Would it perhaps be possible to set the documentation generation going
> at an earlier stage of the bootstrap, in effect spreading out the
> slowness over the entire battery of processor cores?

Yes, that sounds like a good idea.  We do want some of the .elc files to
be built first (the ones that are used to convert .texi and .org files),
I think?  So somebody has to identify those.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Building loaddefs currently takes about 10
> seconds (on my build machine), and I've got it down to 2 seconds.  (For
> my "make -j32" build, this means a reduction of 6% time, since that bit
> is single-threaded and make doesn't have anything else to schedule while
> Emacs is working.)

Uhm.  That's what I thought, but if I compare a checkout from earlier
today (fbe7ac16d722f786e5f), I get:

time make -j32 bootstrap

real	2m21.046s
user	16m34.325s
sys	0m51.938s

And with the current trunk it's:

real	1m52.976s
user	16m1.060s
sys	0m52.892s

*scratches head*

That's a much bigger reduction -- 20% -- and it sounds just weird.
Well, it's making all the loaddefs files now in under 2 seconds (it used
to take 10 seconds for the main one and then the other ones fired off as
separate compilation processes), but it still sounds oddly...  too much.

I'd take that with a grain of salt, but I can't explain it otherwise
(unless there was something in particular wrong with
fbe7ac16d722f786e5f -- perhaps I should try with a checkout from a
couple days ago instead).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Stefan Monnier]

> The first isn't too bad, because it's small, but org.texi is taking
> ~20 seconds to generate on my machine (which is doing nothing else at
> the time) plus, perhaps, 3 seconds for org.info.

The generation of `org.texi` from `org.org` is excruciatingly
slow, indeed.  Maybe we can better hide that latency with a more careful
scheduling, but maybe we could also try to speed it up: I mean if you
compare it to the time it takes for texi2pdf to process the result it's
rather shameful (after all, the generation of the `org.texi` seems
algorithmically a lot simpler).


        Stefan

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I'd take that with a grain of salt, but I can't explain it otherwise
> (unless there was something in particular wrong with
> fbe7ac16d722f786e5f -- perhaps I should try with a checkout from a
> couple days ago instead).

No, it's correct.  I was comparing with "time make autoloads-force", and
that took ten seconds.  But making the initial loaddefs.el file under a
"make bootstrap" (or initial build) really takes 30 seconds --
presumably because it's using some uncompiled Lisp functions (for
radix-tree, hack-local-variables etc).

So it really is a 20% reduction in build time.  😅

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 31 May 2022 20:09:30 +0200
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > So while a "make bootstrap" is faster, this common case is now slower.
> > I'll see what I can do about that.
> 
> Should be fixed now -- this case is now also faster than before, but
> marginally.

I don't understand how it could be faster, if it regenerates
loaddefs.el from scratch each time.

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't understand how it could be faster, if it regenerates
> loaddefs.el from scratch each time.

It doesn't -- it just regenerates the bits from files that are newer
than loaddefs.el.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 31 May 2022 21:33:22 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I don't understand how it could be faster, if it regenerates
> > loaddefs.el from scratch each time.
> 
> It doesn't -- it just regenerates the bits from files that are newer
> than loaddefs.el.

I see it starting from 10%, and it takes longer than typical updates
before the change, which usually would just show 90% and then "done".

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I see it starting from 10%, and it takes longer than typical updates
> before the change, which usually would just show 90% and then "done".

In which cases?  If I

touch .git/logs/HEAD lisp/gnus/gnus.el; time make -j4

it just says

  INFO     Scraping files for loaddefs... 
  INFO     Scraping files for loaddefs...done
  GEN      loaddefs.el

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Yes, that sounds like a good idea.  We do want some of the .elc files to
> be built first (the ones that are used to convert .texi and .org files),
> I think?  So somebody has to identify those.

I had a brief peek at doing this, but the way the makefiles are arranged
here, it's not at all clear what the best approach would be.  In the
top-level Makefile we have

# This dependency is due to those doc/misc/ manuals that use .org sources.
# Depending on src is sufficient, but ends up being slow, since the
# uncompiled lisp/org/*.el files are used to build the .texi files
# (which can be slow even with the elc files).
misc-info: lisp

and indeed, without org/*.el compiled, building the org manual takes 3x
longer.  So we'd need to rearrange those to be compiled first, and then
have that target depend on that new target?  Perhaps put them in
MAIN_FIRST in lisp/Makefile, and...  er...  something.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The generation of `org.texi` from `org.org` is excruciatingly
> slow, indeed.  Maybe we can better hide that latency with a more careful
> scheduling, but maybe we could also try to speed it up: I mean if you
> compare it to the time it takes for texi2pdf to process the result it's
> rather shameful (after all, the generation of the `org.texi` seems
> algorithmically a lot simpler).

If somebody want to poke at this, then perhaps this is a good starting
point:

(progn
  (require 'ox-texinfo)
  (with-temp-buffer
    (insert-file-contents "~/src/emacs/trunk/doc/misc/org.org")
    (setq default-directory (expand-file-name "~/src/emacs/trunk/doc/misc/"))
    (profiler-start 'cpu)
    (org-export-to-file 'texinfo "/tmp/org.texi"))
  (profiler-stop)
  (profiler-report))


        1564  19%                               - org-export-resolve-fuzzy-link

looks pretty suspicious, for instance.  (And that thing should have an
easier calling convention...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So we'd need to rearrange those to be compiled first, and then
> have that target depend on that new target?  Perhaps put them in
> MAIN_FIRST in lisp/Makefile, and...  er...  something.

Perhaps something along the lines of:

diff --git a/doc/misc/Makefile.in b/doc/misc/Makefile.in
index d9c5173c07..a0788580b7 100644
--- a/doc/misc/Makefile.in
+++ b/doc/misc/Makefile.in
@@ -71,8 +71,8 @@ INFO_COMMON =
 	dbus dired-x ebrowse ede ediff edt eieio \
 	emacs-mime epa erc ert eshell eudc efaq eww \
 	flymake forms gnus emacs-gnutls htmlfontify idlwave ido info.info \
-	mairix-el message mh-e modus-themes newsticker nxml-mode octave-mode \
-	org pcl-cvs pgg rcirc remember reftex sasl \
+	mairix-el message mh-e newsticker nxml-mode octave-mode \
+	pcl-cvs pgg rcirc remember reftex sasl \
 	sc semantic ses sieve smtpmail speedbar srecode todo-mode transient \
 	tramp url vhdl-mode vip viper vtable widget wisent woman
 
@@ -82,7 +82,7 @@ INFO_INSTALL =
 ## Info files to build on current platform.
 ## This is all of them, even though they might not all get installed,
 ## because the info files are pre-built in release tarfiles.
-INFO_TARGETS = $(INFO_COMMON) efaq-w32
+INFO_TARGETS = $(INFO_COMMON) efaq-w32 org modus-themes
 
 ## Some manuals have their source in .org format.
 ## This is discouraged because the .texi files it generates
diff --git a/lisp/Makefile.in b/lisp/Makefile.in
index 5f9ca01694..4e3219bcef 100644
--- a/lisp/Makefile.in
+++ b/lisp/Makefile.in
@@ -98,7 +98,9 @@ COMPILE_FIRST +=
 # to speed things up.
 MAIN_FIRST = ./emacs-lisp/eieio.el ./emacs-lisp/eieio-base.el \
   ./cedet/semantic/db.el ./emacs-lisp/cconv.el \
-  ./international/ja-dic-cnv.el
+  ./international/ja-dic-cnv.el \
+  ./org/ox.el ./org/ox-texinfo.el ./org/org-macro.el ./org/org-element.el \
+  ./org/oc.el ./org/ol.el ./emacs-lisp/cl-lib.el
 
 # Prevent any settings in the user environment causing problems.
 unexport EMACSDATA EMACSDOC EMACSLOADPATH EMACSPATH
@@ -121,11 +123,12 @@ SUBDIRS_SUBDIRS =
 
 # cus-load and finder-inf are not explicitly requested by anything, so
 # we add them here to make sure they get built.
-all: compile-main $(lisp)/cus-load.el $(lisp)/finder-inf.el generate-ja-dic
+all: compile-main $(lisp)/cus-load.el $(lisp)/finder-inf.el generate-ja-dic \
+	org-manuals
 
 PHONY_EXTRAS =
 .PHONY: all custom-deps finder-data autoloads update-subdirs $(PHONY_EXTRAS) \
-	generate-ja-dic
+	generate-ja-dic org-manuals
 
 # custom-deps and finder-data both used to scan _all_ the *.el files.
 # This could lead to problems in parallel builds if automatically
@@ -161,6 +164,9 @@ generate-ja-dic:
 	$(AM_V_at)$(MAKE) -C ../leim generate-ja-dic EMACS="$(EMACS)"
 	$(AM_V_at)$(MAKE) compile-targets TARGETS="./leim/ja-dic/ja-dic.elc"
 
+org-manuals: main-first
+	$(MAKE) -C ../doc/misc org.texi modus-themes.texi
+
 ## Comments on loaddefs generation:
 
 # loaddefs depends on gen-lisp for two reasons:


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Alan Mackenzie <acm@muc.de>,  emacs-devel@gnu.org
> Date: Wed, 01 Jun 2022 07:13:37 +0200
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > The generation of `org.texi` from `org.org` is excruciatingly
> > slow, indeed.  Maybe we can better hide that latency with a more careful
> > scheduling, but maybe we could also try to speed it up: I mean if you
> > compare it to the time it takes for texi2pdf to process the result it's
> > rather shameful (after all, the generation of the `org.texi` seems
> > algorithmically a lot simpler).
> 
> If somebody want to poke at this, then perhaps this is a good starting
> point:
> 
> (progn
>   (require 'ox-texinfo)
>   (with-temp-buffer
>     (insert-file-contents "~/src/emacs/trunk/doc/misc/org.org")
>     (setq default-directory (expand-file-name "~/src/emacs/trunk/doc/misc/"))
>     (profiler-start 'cpu)
>     (org-export-to-file 'texinfo "/tmp/org.texi"))
>   (profiler-stop)
>   (profiler-report))
> 
> 
>         1564  19%                               - org-export-resolve-fuzzy-link
> 
> looks pretty suspicious, for instance.  (And that thing should have an
> easier calling convention...)

Instead of trying to speed up this conversion, we could divide org.org
into 10 files, and prepare a Texinfo template that has the preamble
and a series of 10 @include's, one each for every partial file.  Then
most builds would take 1/10th of the time we see now, since situations
where more than one part will be modified are probably extremely rare.

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Perhaps something along the lines of:

I've now pushed something like that -- I've just altered
lisp/Makefile.in to opportunistically start doing the org -> texi
conversion while doing compile-main.

This shaves off about ten seconds of a "make -j8 bootstrap" build here.
I don't think there's any major sections where a typical machine is
single-threaded during the build process any more, except when doing the
two temacs/bootstrap builds, and I don't think there's much to be done
about that.

While we're tweaking these things, this reminds me -- should we have a
build target for "clean everything but don't require configure to be run
again"?  In 99.7% of all cases I'm saying "make bootstrap", I haven't
changed anything that requires configure to be run again, so perhaps
that would be nice to have.  Which brings me to a question -- just what
is it that triggers the new configure run, anyway?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Which brings me to a question -- just what is it that triggers the new
> configure run, anyway?

Oh, config.status missing.  Hm...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 01 Jun 2022 04:29:02 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I see it starting from 10%, and it takes longer than typical updates
> > before the change, which usually would just show 90% and then "done".
> 
> In which cases?  If I
> 
> touch .git/logs/HEAD lisp/gnus/gnus.el; time make -j4
> 
> it just says
> 
>   INFO     Scraping files for loaddefs... 
>   INFO     Scraping files for loaddefs...done
>   GEN      loaddefs.el

I see this with the current master:

  $ touch lisp/array.el .git/logs/HEAD
  $ make -j4
  [...]
    GEN      loaddefs.el
    INFO     Scraping files for loaddefs...
    INFO     Scraping files for loaddefs...26%
    INFO     Scraping files for loaddefs...53%
    INFO     Scraping files for loaddefs...81%
    INFO     Scraping files for loaddefs...done
    GEN      loaddefs.el

Why so much "scraping" lines for just one file modified?

And the example which was before my eyes when I wrote the above was
actually this:

    ELC      ../lisp/minibuffer.elc
    ELC      ../lisp/newcomment.elc
    ELC      ../lisp/simple.elc
  [...]
    GEN      loaddefs.el
    INFO     Scraping files for loaddefs...
    INFO     Scraping files for loaddefs...10%
    INFO     Scraping files for loaddefs...20%
    INFO     Scraping files for loaddefs...30%
    INFO     Scraping files for loaddefs...40%
    INFO     Scraping files for loaddefs...50%
    INFO     Scraping files for loaddefs...60%
    INFO     Scraping files for loaddefs...70%
    INFO     Scraping files for loaddefs...80%
    INFO     Scraping files for loaddefs...90%
    INFO     Scraping files for loaddefs...100%
    INFO     Scraping files for loaddefs...done
    GEN      textmodes/reftex-loaddefs.el
    GEN      textmodes/texinfo-loaddefs.el
    GEN      org/org-loaddefs.el
    GEN      net/tramp-loaddefs.el
    GEN      mh-e/mh-loaddefs.el
    GEN      mail/rmail-loaddefs.el
    GEN      eshell/esh-groups.el
    GEN      erc/erc-loaddefs.el
    GEN      emacs-lisp/eieio-loaddefs.el
    GEN      emacs-lisp/cl-loaddefs.el
    GEN      cedet/srecode/loaddefs.el
    GEN      cedet/semantic/loaddefs.el
    GEN      cedet/ede/loaddefs.el
    GEN      calendar/diary-loaddefs.el
    GEN      calendar/holiday-loaddefs.el
    GEN      calendar/cal-loaddefs.el
    GEN      calc/calc-loaddefs.el
    GEN      ps-print-loaddefs.el
    GEN      ibuffer-loaddefs.el
    GEN      htmlfontify-loaddefs.el
    GEN      dired-loaddefs.el
    GEN      loaddefs.el
  [...]
    ELC      bookmark.elc
    ELC      emacs-lisp/loaddefs-gen.elc
    ELC      textmodes/tex-mode.elc

IOW, only 6 Lisp files and none of the C files were modified, and yet
all the *loaddefs.el files were regenerated from scratch.  (That was
yesterday at 22:40 UTC+3, so maybe you changed something since then.)

Re: Reworking loaddefs.el generation

[Stefan Monnier]

> and indeed, without org/*.el compiled, building the org manual takes 3x
> longer.  So we'd need to rearrange those to be compiled first, and then
> have that target depend on that new target?  Perhaps put them in
> MAIN_FIRST in lisp/Makefile, and...  er...  something.

FWIW, I've been wishing more and more that we switch to a "single"
Makefile (which `include`s the other ones) so that we can have more
precise/clean dependencies between targets in different directories.


        Stefan

Re: Reworking loaddefs.el generation

[Andreas Schwab]

On Jun 01 2022, Lars Ingebrigtsen wrote:

> Which brings me to a question -- just what is it that triggers the new
> configure run, anyway?

make --debug can tell you.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> FWIW, I've been wishing more and more that we switch to a "single"
> Makefile (which `include`s the other ones) so that we can have more
> precise/clean dependencies between targets in different directories.

Yes, indeed.  It's a pain setting up sensible dependencies when we have
separate Makefiles.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Andreas Schwab <schwab@suse.de> writes:

> make --debug can tell you.

Ah, thanks -- didn't know about that.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I see this with the current master:
>
>   $ touch lisp/array.el .git/logs/HEAD
>   $ make -j4
>   [...]
>     GEN      loaddefs.el
>     INFO     Scraping files for loaddefs...
>     INFO     Scraping files for loaddefs...26%
>     INFO     Scraping files for loaddefs...53%
>     INFO     Scraping files for loaddefs...81%
>     INFO     Scraping files for loaddefs...done
>     GEN      loaddefs.el
>
> Why so much "scraping" lines for just one file modified?

I get

  GEN      loaddefs.el
  INFO     Scraping files for loaddefs... 
  INFO     Scraping files for loaddefs...done
  GEN      loaddefs.el

in that test case.

The loop is:

      (dolist (file files)
        (progress-reporter-update progress (setq file-count (1+ file-count)))
        (when (or (not updating)
                  (file-newer-than-file-p file output-file))
          <scrape>

So it seems like file-newer-than-file-p is very slow on your machine
since progress-reporter manages to output several percentages?

> And the example which was before my eyes when I wrote the above was
> actually this:

[...]

>     GEN      loaddefs.el
>     INFO     Scraping files for loaddefs...
>     INFO     Scraping files for loaddefs...10%
>     INFO     Scraping files for loaddefs...20%
>     INFO     Scraping files for loaddefs...30%
>     INFO     Scraping files for loaddefs...40%

[...]

>     ELC      emacs-lisp/loaddefs-gen.elc

Yes, when loaddefs-gen is newer than lisp/loaddefs.el, it regenerates
all the loaddefs files (both to avoid problems with old-style loaddefs
files and to force updates if there's a bug in loaddefs-gen itself).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Instead of trying to speed up this conversion, we could divide org.org
> into 10 files, and prepare a Texinfo template that has the preamble
> and a series of 10 @include's, one each for every partial file.  Then
> most builds would take 1/10th of the time we see now, since situations
> where more than one part will be modified are probably extremely rare.

I'm not very familiar with ox-texinfo, but I don't think it supports
doing something like that?  (I may well be mistaken.)

In any case, this doesn't help with the "make bootstrap" case, so it
would be nice if somebody had a look at what it's doing.  It seems like
(conceptually) it should be a very easy transform, and using 20 seconds
on a smallish (750K) file like that seems just odd.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

I was curious to see how load times have evolved over the last couple of
years, so I did a loop like this:

git checkout `git rev-list -1 --before="Jun 1 2021" HEAD`; git clean -xf; time make -j32 bootstrap

And got the following result.

Jun 1 2020 1m36.639s
Jan 1 2021 1m49.241s
Jun 1 2021 2m2.398s
Dec 1 2021 2m10.583s
May 1 2022 2m25.506s

So we've been losing about 10s per six months?  :-/

The changes this week basically takes us back two years.  I know what a
couple of things that led to longer build times -- org.org and
symbols-with-positions -- but there's been other things, too, without us
noticing much.  It'd be interesting to see if we could pin-point commits
that's led to compilation performance regressions, but I guess that'd
take quite a while to run...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Reworking loaddefs.el generation

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 01 Jun 2022 14:13:21 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I see this with the current master:
> >
> >   $ touch lisp/array.el .git/logs/HEAD
> >   $ make -j4
> >   [...]
> >     GEN      loaddefs.el
> >     INFO     Scraping files for loaddefs...
> >     INFO     Scraping files for loaddefs...26%
> >     INFO     Scraping files for loaddefs...53%
> >     INFO     Scraping files for loaddefs...81%
> >     INFO     Scraping files for loaddefs...done
> >     GEN      loaddefs.el
> >
> > Why so much "scraping" lines for just one file modified?
> 
> I get
> 
>   GEN      loaddefs.el
>   INFO     Scraping files for loaddefs... 
>   INFO     Scraping files for loaddefs...done
>   GEN      loaddefs.el
> 
> in that test case.
> 
> The loop is:
> 
>       (dolist (file files)
>         (progress-reporter-update progress (setq file-count (1+ file-count)))
>         (when (or (not updating)
>                   (file-newer-than-file-p file output-file))
>           <scrape>
> 
> So it seems like file-newer-than-file-p is very slow on your machine
> since progress-reporter manages to output several percentages?

However slow file-newer-than-file-p is here, it didn't become slower
since these changes were done.  Before those changes, I was used to
see the same as you see now: just two lines with no percents.  So
something did change that made the process slower, at least on this
machine.

Re: Reworking loaddefs.el generation

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> However slow file-newer-than-file-p is here, it didn't become slower
> since these changes were done.  Before those changes, I was used to
> see the same as you see now: just two lines with no percents.  So
> something did change that made the process slower, at least on this
> machine.

Yeah, I guess using file-newer-than-file-p is (as it usually is) a bad
choice here.  I've now switched to time-less-p, like autoload.el uses.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no