bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin <manikulin@gmail.com>]

On 03/05/2023 17:30, Ihor Radchenko wrote:
> Stefan Monnier writes:
> 
>>> This looks interesting, if we replace all the requires in Org with
>>> `my-require-with-shadow-check'.
...
> Max, do you see any obvious downsides in Stefan's idea about consulting
> `load-history' vs. `load-path'?

First of all, I think, incremental builds are broken in Emacs. 
`org-assert-version' is just the most apparent manifestation.

gcc supports generation of dependency files as a side effect of 
preprocessing for decades, see
- (info "(make) Automatic-Prerequisites")
 
https://www.gnu.org/software/make/manual/html_node/Automatic-Prerequisites.html
- https://gcc.gnu.org/news/dependencies.html
   Dependency Generation Improvements. 22 January 2001
- https://gcc.gnu.org/onlinedocs/gcc/Preprocessor-Options.html#index-MF

The idea is that clean build does not require dependency map. For a 
rebuild, .d files are included into Makefile, so make is able to 
determine proper order of compilation.

Perhaps a similar trick may be done in elisp by advicing e.g. `load'.

Instead of dependency tracking Makefile in Emacs uses much more limited 
approach with the main-first target. That is why incremental build may 
easily result in mixed-version compilation. Imprinted Org version just 
detects such case.

By the way, generated dependency map might help to properly reload a 
package update without restarting of Emacs.

It seems, despite "make" in the bug title, it has been decided to 
dedicate this issue to plumbing of `org-assert-version'.

Inspecting `load-history' might be tried, however such approach may 
cause other issues.

I have heard that some users complain concerning Emacs startup time. 
Likely a hashmap will be required in addition to the `load-history' list 
to avoid performance degradation. I am unsure if the following idea have 
some benefits: check shadowing when `load-path' is modified, not for 
each file loaded from the newly added directory.

I am unsure that path comparison is able to detect a problem when a user 
reloads Org after git pull and compiling new version. Files with new 
functions and macros are loaded from the same directory.

In respect to incremental builds, `org-assert-version' is a disaster. 
Any update requires full recompiling. It is a reason why I always 
considered it as a kludge. Unfortunately I do not have a better idea.

We need a tool that clearly informs users that they have issues with 
their configuration files, multiple versions of a package are available 
and it causes mixed version loading with inconsistent function 
definitions or, even worse, mixed version compiling with obsolete macro 
definitions. Emacs restart with fixed loading does not help in the 
latter case.

Before introducing of `org-assert-version', errors were obscure: 
undefined functions or incompatible arguments (for internal functions). 
Unfortunately `org-assert-version' have caused another stream of complains.

The issue is not limited to package.el or replacing built-in package. 
Various ways to load packages are used, so a couple of versions may 
appear in load-path even for non built-in package.

The tool should detect version mismatch during loading, reloading, and 
compiling of an update.

I think, both aspects must be addressed:
- Improving dependency handling during incremental builds
- Informing users about issues with loading of multi-file packages.