bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

> First of all, I think, incremental builds are broken in Emacs.

Indeed, but in practice they usually work fine.  So in the absence of
a better solution, we stay with this hack and tell people to try
`make bootstrap` when there's a problem.

Also, the use of `load-prefer-newer` in lisp/Makefile eliminates most of
the brokenness we used to have in our incremental builds.

> `org-assert-version' is just the most apparent manifestation.

AFAIK `org-assert-version` tries to solve a different problem.  Indeed,
we recently introduced `org--inhibit-version-check` specifically so as
not to use `org-assert-version` for Emacs's broken incremental builds
(i.e. we decided we preferred that brokenness there).

> gcc supports generation of dependency files as a side effect of
> preprocessing for decades, see
[...]
> Perhaps a similar trick may be done in elisp by advicing e.g. `load'.

I encourage people to try that out, yes.
Having been there (many years ago), I can already warn them that an
important problem will be the presence of cyclic dependencies.

> Instead of dependency tracking Makefile in Emacs uses much more limited
> approach with the main-first target.

Hmm... no, `main-first` doesn't have much to do with dependencies (and
things like deciding when to *re*compile a file), it's concerned with
compile-time performance (typically for the first compilation, when the
`.d` dependencies wouldn't be available yet).

> That is why incremental build may easily result in mixed-version
> compilation.

Actually, now that we set `load-prefer-newer`, it is rarely a problem.
[ Of course, it can still be a problem, but only in cases such as when
  a macro is changed such that the old expansion is not compatible with
  code using the new definitions.  This happens (e.g. when we changed
  structs to use `record` instead of `vector`), but not fairly rarely.  ]

> By the way, generated dependency map might help to properly reload a package
> update without restarting of Emacs.

BTW, I was reminded recently that `load-history` keeps track of
`require`s so we already have most of that info at hand.

> I have heard that some users complain concerning Emacs startup time. Likely
> a hashmap will be required in addition to the `load-history' list to avoid
> performance degradation. I am unsure if the following idea have some
> benefits: check shadowing when `load-path' is modified, not for each file
> loaded from the newly added directory.

The idea was to check the shadowing only for one specific file, so as to
try and keep the added cost in check.  Checking shadowings when
`load-path` is modified is harder (because `load-history` doesn't
remember which name was used to load a file, so it could be confused and
think a file like `lisp/progmodes/compile.el` shadows a previously
loaded `lisp/cedet/srecode/compile.el`) and could be more costly
(because it has to check for "all" files).

> I am unsure that path comparison is able to detect a problem when a user
> reloads Org after git pull and compiling new version.

[ AFAICT, there are different situations with related yet different
  problems that manifest slightly differently and that may be solved
  differently as well, so please be specific when you mention
  problematic scenarios.  ]

If you mean `git pull; make` in Emacs's source code, then AFAIK this
should almost never be a problem (thanks to `load-prefer-newer`), and if
it is, it's a general problem that's not specific to Org, and "we"
changed `org-assert-version` so as not to bother trying to solve
this case.

If you mean `git pull; make` in Org's source repository, then I must
admit that I don't have much experience with it, but make Org's make
file could set `load-prefer-newer` (or use some hack to autogenerate
`.d` dependency files :-).

`my-require-with-shadow-check` is instead aimed at the case where the
users try to load a mix of two different Org versions in the same Emacs
sessions, either because of things like:
- they're byte-compiling Org-2 in an Emacs that has Org-1 already loaded.
- they inadvertently end up loading part of Org-1 early in their init
  file before they set their `load-path` to point to Org-2.

I understand Org people have had lots of problems with mixed-versions in
the past, but I personally don't know which circumstances have been more
often at the source of those problems, so maybe my suggestion addresses
a problem that's not very important, indeed.

I my own experience `git pull; make` in Emacs's source code has
virtually never caused me trouble with Org files, and it's never cause
any trouble in my checkout of Org's source code either, which is why
I'm suggesting something like `my-require-with-shadow-check`, which
targets other scenarios.

> In respect to incremental builds, `org-assert-version' is a disaster.
> Any update requires full recompiling.

Indeed, I used to disable it locally (before `org--inhibit-version-check`).

> It is a reason why I always considered it as a kludge.  Unfortunately
> I do not have a better idea.

Maybe a clear set of examples of the kinds of problems that
`org-assert-version` aims to catch would be a good start.
For me it's still not really clear.

> The issue is not limited to package.el or replacing built-in package.
> Various ways to load packages are used, so a couple of versions may
> appear in load-path even for non built-in package.

So far I haven't seen enough of the diversity of situations are (source
of) problems to have a clear idea of what a good solution should look
like, admittedly.

`my-require-with-shadow-check` is only based on my intuition that it
might catch the most common issues where two different Org versions are
used in the same Emacs session, which seems to cover most of the
problems (except for those due to `git pull; make`, which seem to be
qualitatively of a different kind).


        Stefan