bug#62762: Incremental builds and Lisp files dependencies pulling a new version of the code

[Eli Zaretskii <eliz@gnu.org>]

> Date: Sat, 6 May 2023 13:00:38 +0700
> Cc: Ihor Radchenko <yantar92@posteo.net>, Eli Zaretskii <eliz@gnu.org>,
>  bzg@gnu.org, dmitry@gutov.dev, 62762@debbugs.gnu.org,
>  Alan Mackenzie <acm@muc.de>
> From: Max Nikulin <manikulin@gmail.com>
> 
> On 06/05/2023 01:17, Stefan Monnier wrote:
> > I'll refrain from discussing this here.  Incremental recompilation
> > problems should be discussed in another bug report, IMO.
> 
> I will try to explain why this bug report was caused by issues with 
> incremental builds.

I've started a new thread, because I agree with Stefan: this is a
separate issue.  Please reply with this new thread's Subject.

> Let's assume that a trouble with cyclic dependencies is a real one (I 
> have not convinced in that). A way to the correct builds is to 
> *completely* avoid loading of .elc files during incremental builds. When 
> only a few files changed then relying purely on .el files should cause 
> significant performance penalty.

Exactly.  And please don't forget that quite a few files are
preloaded, so using only *.el means to preload only the *.el files,
which will result in a significantly slower bootstrap-emacs binary,
and the build will become annoyingly slow, for no good reason.

> Unfortunately it would not work without 
> describing at least some dependencies. In the `org-assert-version' they 
> should be at least that all lisp/org/*.elc files depend on 
> lisp/org/org-version.el and lisp/org/org-macs.el. Without automatic 
> dependency generation it is a kludge, but it should significantly 
> alleviate the issue.

This doesn't really work, not as long as we use Make: some *.elc files
might not exist yet, for whatever reasons, so Make will try to
generate them first, thus disrupting the order of generation that you
will try to encode in the dependencies.  We already tried that, and it
didn't work, not even just for Org.

> Commits pushed so far trade false positives to false negatives and to 
> reports of bugs due to "undefined" functions and incorrect signatures to 
> Org developers and maintainers.

Those commits only affect byte compilation of Org as part of building
Emacs, so I see no reason why they should cause the above issues for
Orge developers.

> Perhaps `org-assert-version' may be improved, but this report was caused 
> by broken build rules.

The bug#62762 was reported because Org behaved unlike any other Lisp
package in Emacs.  We have other packages that define macros used in
many other *.el files, and none of them does the version-check like
Org did.  The problems with making incompatible changes in one or more
of the macros are well known to Emacs developers who rebuild Emacs
frequently, and we have ways to deal with them.  Some of those ways
(the simplest ones) are encoded in the top-level Makefile, see
ADVICE-ON-FAILURE there.  There are others, more subtle ones, for
those, like myself, who don't bootstrap, ever, except the first time a
fresh repository is cloned.

That Org behaved in a stark different manner was therefore an
unpleasant annoyance, so we now fixed that by making Org behave in
this regard like any other Lisp package in Emacs -- but only when
Emacs is built, because the knob which forces this behavior is off by
default, and is only turned on by lisp/Makefile in the Emacs tree.