bug#62762: circular dependencies in elisp files and make

[Max Nikulin <manikulin@gmail.com>]

On 13/05/2023 21:48, Stefan Monnier wrote:
>> 1. A script reads dependency files (if they exist) created during previous
>>    build and removes stale .elc files.
> 
> `load-prefer-newer` should give us the same result already.

No, because some .elc files may become stale not because of modification 
of the corresponding files, but due to update of a macro inside a 
`require'd file. Removing of outdated .elc files *before* 
byte-compilation pass ensures that no obsolete macro definitions are used.

On 13/05/2023 21:49, Stefan Monnier wrote:
>> Writing "stale" I mean more than comparison timestamps of a.el and
>> a.elc. Dependency files may contain
>>
>> a.elc: a.el d.elc
>> d.elc: d.el
>>
>> the script walks through dependency graph, compares timestamps and when it
>> d.el is newer than d.elc, *both* "a.elc" and "d.elc" are removed.
> 
> Make will already take care of this dependency and recompile `d.elc`
> before `a.elc`.

You are right when dependency graph is a tree. You provided an example 
of circular dependency in lisp/vc/ediff-util.el. It is the case when to 
correctly recompile ediff it is necessary to remove ediff*.elc files 
before compilation (I do not like your approach with breaking cycle at 
an arbitrary edge since outdated macro definition still may be used). 
Files necessary for byte compilation is another example of a dependency 
loop. I assume there are may be more tricky cases I am unaware about, so 
prefer to be at the safe side.

Eli raised a question of rebuilding of an .elc file when a new 
dependency with outdated .elc file is introduced. Since it is almost 
impossible to get dependency list without byte compilation, there are 2 
options: ask make to rebuild .elc files files once more (updated 
dependency graph is available during second byte compilation pass) or 
just remove all stale .elc file before starting byte compilation. I 
prefer the latter approach.

Due to these reasons I prefer 2 stage incremental build.

On 13/05/2023 22:43, Stefan Monnier wrote:
> Not sure about two passes, but if you want it to be "right" (i.e. aim
> to never need `make bootstrap`), then every `.elc` file will transitively
> depend on every preloaded `.el` file, including `loaddefs.el` which is
> itself auto-generated from all the `.el` files.  IOW the only truly
> "right" way is to always recompile all the `.elc` files whenever  `.el`
> (or `.c` or `.h`) file is modified 🙁

I still hope, it is possible to add loaddefs files as order-only 
dependencies. I see just one case when it may lead to incomplete 
rebuild: a.el uses a function from al.el without (require 'al) and 
autoload cookie is missed in al.el. After compilation of a.el dependency 
on al.elc is unknown thus updating al.el is not a reason to recompile 
a.elc. I hope, raising priority of "Warning: the following functions are 
not known to be defined:" to an error preventing creation of .elc file 
will solve the issue.

Despite at first glance loaddefs ruin the idea with partial incremental 
rebuild and dependency tracking, it seems a workaround exists.