bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Dmitry Gutov]

The log excerpt is at the end.

Could someone fix that, so that I don't have to 'make bootstrap' every 
other time after a 'git pull'?

make[3]: Entering directory '/home/dgutov/vc/emacs-master/lisp'
   ELC      org/ox.elc
Warning (emacs): Org version mismatch.  Org loading aborted.
This warning usually appears when a built-in Org version is loaded
prior to the more recent Org version.

Version mismatch is commonly encountered in the following situations:

1. Emacs is loaded using literate Org config and more recent Org
    version is loaded inside the file loaded by ‘org-babel-load-file’.
    ‘org-babel-load-file’ triggers the built-in Org version clashing
    the newer Org version attempt to be loaded later.

    It is recommended to move the Org loading code before the
    ‘org-babel-load-file’ call.

2. New Org version is loaded manually by setting ‘load-path’, but some
    other package depending on Org is loaded before the ‘load-path’ is
    configured.
    This "other package" is triggering built-in Org version, again
    causing the version mismatch.

    It is recommended to set ‘load-path’ as early in the config as
    possible.

3. New Org version is loaded using straight.el package manager and
    other package depending on Org is loaded before straight triggers
    loading of the newer Org version.

    It is recommended to put

     (straight-use-package 'org)

    early in the config.  Ideally, right after the straight.el
    bootstrap.  Moving ‘use-package’ :straight declaration may not be
    sufficient if the corresponding ‘use-package’ statement is
    deferring the loading.

In toplevel form:
org/ox.el:79:2: Error: Org version mismatch.  Make sure that correct 
‘load-path’ is set early in init.el

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Date: Tue, 11 Apr 2023 02:09:45 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> Could someone fix that, so that I don't have to 'make bootstrap' every 
> other time after a 'git pull'?

First, you don't need to "make bootstrap" in this case.  It is enough
to say "touch lisp/org/*.el".

And second, I tried to install a solution to that on the master
branch, but it is not perfect, to say the least: depending on which
*.el files in lisp/org/ were updated by the same pull/merge that
brought the new org-version.el (which is the source of the trouble),
the solution I installed may or may not work.

So a better solution is probably to "touch" all the *.el files in
lisp/org/ (or just some of them, if we can identify them) when
org-version.el is updated.  Patches welcome, or I will get to that
eventually.

Or maybe Org folks (CC'ed) could suggest a better fix.  I understand
the reasons for this behavior in Org, but none of the reasons
described in org-version.el are relevant to Emacs development, when a
new version of Org is merged.  Maybe the abort could be augmented not
to produce such a drastic effect?

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Or maybe Org folks (CC'ed) could suggest a better fix.  I understand
> the reasons for this behavior in Org, but none of the reasons
> described in org-version.el are relevant to Emacs development, when a
> new version of Org is merged.  Maybe the abort could be augmented not
> to produce such a drastic effect?

The fact the Org error is triggered means that Emacs does not re-compile
.el files that refer to macros that have been changed.
See https://orgmode.org/list/jwvsfkv5s7l.fsf-monnier+emacs@gnu.org

Macros not being updated, as one may expect, may break the code using
old macro expansions. So, I cannot say that Org is wrong complaining
about mismatch here. In fact, we already made the check less strict to
work around the same issue on ELPA. See
https://list.orgmode.org/orgmode/87r10ey8ov.fsf@localhost/ and
https://list.orgmode.org/orgmode/87pmfijrvw.fsf@localhost/

Maybe make bootstrap is justified then?

Of course, ideally, Emacs should provide a way to handle macro dependencies.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Dmitry Gutov <dmitry@gutov.dev>, Bastien <bzg@gnu.org>,
>  62762@debbugs.gnu.org
> Date: Tue, 11 Apr 2023 07:18:18 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Or maybe Org folks (CC'ed) could suggest a better fix.  I understand
> > the reasons for this behavior in Org, but none of the reasons
> > described in org-version.el are relevant to Emacs development, when a
> > new version of Org is merged.  Maybe the abort could be augmented not
> > to produce such a drastic effect?
> 
> The fact the Org error is triggered means that Emacs does not re-compile
> .el files that refer to macros that have been changed.

You know this is not really possible in Emacs.

> Macros not being updated, as one may expect, may break the code using
> old macro expansions. So, I cannot say that Org is wrong complaining
> about mismatch here. In fact, we already made the check less strict to
> work around the same issue on ELPA. See
> https://list.orgmode.org/orgmode/87r10ey8ov.fsf@localhost/ and
> https://list.orgmode.org/orgmode/87pmfijrvw.fsf@localhost/

How is this different from any other Lisp file in Emacs?  We have
quite a few that define macros used elsewhere, and we rely on
byte-compilation during the build to tell us when the old definitions
get in the way.  We don't forcefully abort the build, just because
problems _might_ exist.  And people who want to make sure their builds
are 110% perfect always bootstrap to begin with.  Building a tarball
also requires a bootstrap, so the chance of these issues slipping
through cracks are minuscule to say the least.

Anyway, any other words of wisdom?  Like which org/*.el files actually
need to be recompiled in this case?  Or are you saying all of them
need to be recompiled?

I also am not sure I understand the logic of this test: AFAIU it only
tests the version string, not the actual macros that might have been
changed.  Are you saying that Org bumps its version string each time
_any_ Org macro is modified?

Also, how about the alternative of including the version string in
every Org Lisp file that needs to be recompiled when the version
changes (or the macros change)?  Then recompilation will happen
automatically, and this problem will go away.  Does this make sense?

> Maybe make bootstrap is justified then?

No, it isn't.  Not unless files outside of lisp/org/ will start
importing org-version.el.

> Of course, ideally, Emacs should provide a way to handle macro dependencies.

Ideally, this would be a much nicer world, indeed.  But humanity was
expelled from the Garden of Eden long ago, so here we are.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> The fact the Org error is triggered means that Emacs does not re-compile
>> .el files that refer to macros that have been changed.
>
> You know this is not really possible in Emacs.

I don't. Say, why not to store hash of the macro sources in the byte
code and verifying the hash when re-compilation is requested? Though I
am not that much familiar with Emacs source compilation.

>> Macros not being updated, as one may expect, may break the code using
>> old macro expansions. So, I cannot say that Org is wrong complaining
>> about mismatch here. In fact, we already made the check less strict to
>> work around the same issue on ELPA. See
>> https://list.orgmode.org/orgmode/87r10ey8ov.fsf@localhost/ and
>> https://list.orgmode.org/orgmode/87pmfijrvw.fsf@localhost/
>
> How is this different from any other Lisp file in Emacs?  We have
> quite a few that define macros used elsewhere, and we rely on
> byte-compilation during the build to tell us when the old definitions
> get in the way.  We don't forcefully abort the build, just because
> problems _might_ exist.  And people who want to make sure their builds
> are 110% perfect always bootstrap to begin with.  Building a tarball
> also requires a bootstrap, so the chance of these issues slipping
> through cracks are minuscule to say the least.

Sure, in the context of Emacs compilation. The code in question is
mostly aiming at newer Org installed on top of built-in Org. We are
consistently getting issues related to incorrect macro expansion and
mixing different Org versions.

> Anyway, any other words of wisdom?  Like which org/*.el files actually
> need to be recompiled in this case?  Or are you saying all of them
> need to be recompiled?

All of them, because org-assert-version expands into
 (unless (equal "version string" (org-release)) ...)
at compile time. "Version string" contains the return value of
(org-release) for the time the .elc file is compiled.

Note that the reported issue will only happen when Org version string
changes between the builds.

What we might do to work around the problem is detecting Emacs compilation
and disable the check. Is there a way to detect that Emacs source is
being compiled from Elisp?

> I also am not sure I understand the logic of this test: AFAIU it only
> tests the version string, not the actual macros that might have been
> changed.  Are you saying that Org bumps its version string each time
> _any_ Org macro is modified?

No. We originally tested by commit hash. Version string check is a
trade-off between the problem with ELPA builds (see the links in my
previous message) and the need to detect mixed installation problems.

> Also, how about the alternative of including the version string in
> every Org Lisp file that needs to be recompiled when the version
> changes (or the macros change)?  Then recompilation will happen
> automatically, and this problem will go away.  Does this make sense?

No. We will not be able to maintain this manually.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, bzg@gnu.org, 62762@debbugs.gnu.org
> Date: Tue, 11 Apr 2023 18:35:51 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> The fact the Org error is triggered means that Emacs does not re-compile
> >> .el files that refer to macros that have been changed.
> >
> > You know this is not really possible in Emacs.
> 
> I don't. Say, why not to store hash of the macro sources in the byte
> code and verifying the hash when re-compilation is requested? Though I
> am not that much familiar with Emacs source compilation.

We need to do this the "make" way.  That is, we need to be able to
produce dependencies that Make can understand, e.g. similarly to how
the *.d files in src/.deps are produced by GCC.  This is in principle
possible, but we need changes in the byte compiler and in 'load', and
probably more.

> Note that the reported issue will only happen when Org version string
> changes between the builds.

Which happened twice during the last week or two, AFAIR.  Moreover, it
happens on almost every branch, and if someone builds several branches
routinely (I do), the annoyance happens several times in a row.

> What we might do to work around the problem is detecting Emacs compilation
> and disable the check. Is there a way to detect that Emacs source is
> being compiled from Elisp?

Not sure I understand: the byte compiler is a Lisp program, so every
compilation is "from Lisp", no?  Or what am I missing?

> > Also, how about the alternative of including the version string in
> > every Org Lisp file that needs to be recompiled when the version
> > changes (or the macros change)?  Then recompilation will happen
> > automatically, and this problem will go away.  Does this make sense?
> 
> No. We will not be able to maintain this manually.

I guess it means we are on our own, sigh...

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Note that the reported issue will only happen when Org version string
>> changes between the builds.
>
> Which happened twice during the last week or two, AFAIR.  Moreover, it
> happens on almost every branch, and if someone builds several branches
> routinely (I do), the annoyance happens several times in a row.

Fair point.

>> What we might do to work around the problem is detecting Emacs compilation
>> and disable the check. Is there a way to detect that Emacs source is
>> being compiled from Elisp?
>
> Not sure I understand: the byte compiler is a Lisp program, so every
> compilation is "from Lisp", no?  Or what am I missing?

I mean, can we detect when the compiler is invoked from "make" in Emacs
source?

>> > Also, how about the alternative of including the version string in
>> > every Org Lisp file that needs to be recompiled when the version
>> > changes (or the macros change)?  Then recompilation will happen
>> > automatically, and this problem will go away.  Does this make sense?
>> 
>> No. We will not be able to maintain this manually.
>
> I guess it means we are on our own, sigh...

You are not. In the worst case, we may add some flag into Org sources
that are inside Emacs tree. Something like `org-version-built-in-p' that
will be set to t only in the Emacs' version of Org.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, bzg@gnu.org, 62762@debbugs.gnu.org
> Date: Tue, 11 Apr 2023 18:59:17 +0000
> 
> >> What we might do to work around the problem is detecting Emacs compilation
> >> and disable the check. Is there a way to detect that Emacs source is
> >> being compiled from Elisp?
> >
> > Not sure I understand: the byte compiler is a Lisp program, so every
> > compilation is "from Lisp", no?  Or what am I missing?
> 
> I mean, can we detect when the compiler is invoked from "make" in Emacs
> source?

I guess we ccoul define some environment variable in Lisp/Makefile?

> > I guess it means we are on our own, sigh...
> 
> You are not. In the worst case, we may add some flag into Org sources
> that are inside Emacs tree. Something like `org-version-built-in-p' that
> will be set to t only in the Emacs' version of Org.

And what will that mean when org-version.el bumps its version?

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Cc: bzg@gnu.org, dmitry@gutov.dev, 62762@debbugs.gnu.org
> Date: Tue, 11 Apr 2023 22:28:14 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > I mean, can we detect when the compiler is invoked from "make" in Emacs
> > source?
> 
> I guess we ccoul define some environment variable in Lisp/Makefile?
             ^^^^^
That was supposed to be "could", sorry.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I mean, can we detect when the compiler is invoked from "make" in Emacs
>> source?
>
> I guess we ccoul define some environment variable in Lisp/Makefile?

That should be fine.

>> > I guess it means we are on our own, sigh...
>> 
>> You are not. In the worst case, we may add some flag into Org sources
>> that are inside Emacs tree. Something like `org-version-built-in-p' that
>> will be set to t only in the Emacs' version of Org.
>
> And what will that mean when org-version.el bumps its version?

My idea is to change `org-assert-version' in such a way that the check
is completely bypassed when `org-version-built-in-p' is non-nil (of when
some environment variable is set). As I said, `org-assert-version' is
not aiming for built-in Org compilation. Just for Org package
installation.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, bzg@gnu.org, 62762@debbugs.gnu.org
> Date: Wed, 12 Apr 2023 09:03:48 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> You are not. In the worst case, we may add some flag into Org sources
> >> that are inside Emacs tree. Something like `org-version-built-in-p' that
> >> will be set to t only in the Emacs' version of Org.
> >
> > And what will that mean when org-version.el bumps its version?
> 
> My idea is to change `org-assert-version' in such a way that the check
> is completely bypassed when `org-version-built-in-p' is non-nil (of when
> some environment variable is set). As I said, `org-assert-version' is
> not aiming for built-in Org compilation. Just for Org package
> installation.

I think this will be a good change from my POV, thank you.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> My idea is to change `org-assert-version' in such a way that the check
>> is completely bypassed when `org-version-built-in-p' is non-nil (of when
>> some environment variable is set). As I said, `org-assert-version' is
>> not aiming for built-in Org compilation. Just for Org package
>> installation.
>
> I think this will be a good change from my POV, thank you.

So, will you prefer env variable or Elisp variable?
Environment variable is potentially more useful as it might also be used
by other code, not just by Org.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, bzg@gnu.org, 62762@debbugs.gnu.org
> Date: Thu, 13 Apr 2023 08:34:19 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> My idea is to change `org-assert-version' in such a way that the check
> >> is completely bypassed when `org-version-built-in-p' is non-nil (of when
> >> some environment variable is set). As I said, `org-assert-version' is
> >> not aiming for built-in Org compilation. Just for Org package
> >> installation.
> >
> > I think this will be a good change from my POV, thank you.
> 
> So, will you prefer env variable or Elisp variable?
> Environment variable is potentially more useful as it might also be used
> by other code, not just by Org.

I think ELisp variable is better, since environment variables are
inherited by sub-processes in ways that are hard to control.

Thanks.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 262 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> I think ELisp variable is better, since environment variables are
> inherited by sub-processes in ways that are hard to control.

See the attached.
On Emacs side, `org--built-in-p' will need to be set during compilation.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-assert-version-Skip-version-check-when-Org-is-a-.patch --]
[-- Type: text/x-patch, Size: 1545 bytes --]

From 89140b0d747aa127e7b06ed927c09286215e4824 Mon Sep 17 00:00:00 2001
Message-Id: <89140b0d747aa127e7b06ed927c09286215e4824.1681395507.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Thu, 13 Apr 2023 16:18:15 +0200
Subject: [PATCH] org-assert-version: Skip version check when Org is a part of
 Emacs

* lisp/org-macs.el (org--built-in-p): New constant indicating if Org
source is a part of Emacs source tree.
(org-assert-version): Skip check when Org is built-in.
---
 lisp/org-macs.el | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/lisp/org-macs.el b/lisp/org-macs.el
index ad1d3cf14..b4a5d82c8 100644
--- a/lisp/org-macs.el
+++ b/lisp/org-macs.el
@@ -36,6 +36,9 @@ (require 'format-spec)
 
 ;;; Org version verification.
 
+(defconst org--built-in-p nil
+  "When non-nil, assume that Org is a part of Emacs source.
+See Emacs bug #62762.")
 (defmacro org-assert-version ()
   "Assert compile time and runtime version match."
   ;; We intentionally use a more permissive `org-release' instead of
@@ -45,7 +48,7 @@ (defmacro org-assert-version ()
   ;; `org-assert-version' calls would fail using strict
   ;; `org-git-version' check because the generated Org version strings
   ;; will not match.
-  `(unless (equal (org-release) ,(org-release))
+  `(unless (or org--built-in-p (equal (org-release) ,(org-release)))
      (warn "Org version mismatch.  Org loading aborted.
 This warning usually appears when a built-in Org version is loaded
 prior to the more recent Org version.
-- 
2.40.0


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, bzg@gnu.org, 62762@debbugs.gnu.org
> Date: Thu, 13 Apr 2023 14:20:05 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think ELisp variable is better, since environment variables are
> > inherited by sub-processes in ways that are hard to control.
> 
> See the attached.
> On Emacs side, `org--built-in-p' will need to be set during compilation.

LGTM, will soon incorporate this in our build system.

Thanks!

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> See the attached.
>> On Emacs side, `org--built-in-p' will need to be set during compilation.
>
> LGTM, will soon incorporate this in our build system.

Once you confirm that my patch + your build system update fixes the
problem, I will push to Org bugfix and bump a minor bugfix release.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, bzg@gnu.org, 62762@debbugs.gnu.org
> Date: Thu, 13 Apr 2023 15:05:21 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> See the attached.
> >> On Emacs side, `org--built-in-p' will need to be set during compilation.
> >
> > LGTM, will soon incorporate this in our build system.
> 
> Once you confirm that my patch + your build system update fixes the
> problem, I will push to Org bugfix and bump a minor bugfix release.

I've now installed the change on master.

I wonder why org--built-in-p is a defconst, though: shouldn't it be a
defvar instead?

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> I wonder why org--built-in-p is a defconst, though: shouldn't it be a
> defvar instead?

I wanted to make it clear that users must not set this variable.
If you think that it is excessive, I can use defvar instead.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, bzg@gnu.org, 62762@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 11:36:40 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I wonder why org--built-in-p is a defconst, though: shouldn't it be a
> > defvar instead?
> 
> I wanted to make it clear that users must not set this variable.
> If you think that it is excessive, I can use defvar instead.

It's up to you, I don't mind.  (But if you decide to go with defconst,
maybe add a comment there explaining that this is the intent.)

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> It's up to you, I don't mind.  (But if you decide to go with defconst,
> maybe add a comment there explaining that this is the intent.)

I updated the docstring.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?h=bugfix&id=07508fb09

The commit is now included into the new bugfix release.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: dmitry@gutov.dev, bzg@gnu.org, 62762@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 11:57:57 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It's up to you, I don't mind.  (But if you decide to go with defconst,
> > maybe add a comment there explaining that this is the intent.)
> 
> I updated the docstring.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?h=bugfix&id=07508fb09
> 
> The commit is now included into the new bugfix release.

Thanks.  The change I made in Emacs is only on master, FYI.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Cc: bzg@gnu.org, dmitry@gutov.dev, 62762@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 13:48:46 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Ihor Radchenko <yantar92@posteo.net>
> > Cc: dmitry@gutov.dev, bzg@gnu.org, 62762@debbugs.gnu.org
> > Date: Thu, 13 Apr 2023 15:05:21 +0000
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > >> See the attached.
> > >> On Emacs side, `org--built-in-p' will need to be set during compilation.
> > >
> > > LGTM, will soon incorporate this in our build system.
> > 
> > Once you confirm that my patch + your build system update fixes the
> > problem, I will push to Org bugfix and bump a minor bugfix release.
> 
> I've now installed the change on master.
> 
> I wonder why org--built-in-p is a defconst, though: shouldn't it be a
> defvar instead?

This doesn't seem to work as expected here?  Here's what I get with
the latest update from master:

  '../src/emacs.exe' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  --eval "(setq org--built-in-p t)"  -f batch-byte-compile org/org-element.el
  Warning (emacs): Org version mismatch.  Org loading aborted.
  This warning usually appears when a built-in Org version is loaded prior to the more recent Org version.

  Version mismatch is commonly encountered in the following situations:

This seems to indicate that the compilation is aborted even though the
command line sets org--built-in-p non-nil.  What went wrong here?

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> This doesn't seem to work as expected here?  Here's what I get with
> the latest update from master:
>
>   '../src/emacs.exe' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  --eval "(setq org--built-in-p t)"  -f batch-byte-compile org/org-element.el
>   Warning (emacs): Org version mismatch.  Org loading aborted.
>   This warning usually appears when a built-in Org version is loaded prior to the more recent Org version.
>
>   Version mismatch is commonly encountered in the following situations:
>
> This seems to indicate that the compilation is aborted even though the
> command line sets org--built-in-p non-nil.  What went wrong here?

I see nothing wrong in the new version of the `org-assert-version'
macro. The only explanation I can come up with is that
`org-assert-version' from an older Org version somehow got mixed in.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: bzg@gnu.org, dmitry@gutov.dev, 62762@debbugs.gnu.org
> Date: Fri, 21 Apr 2023 15:28:02 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > This doesn't seem to work as expected here?  Here's what I get with
> > the latest update from master:
> >
> >   '../src/emacs.exe' -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t byte-compile-warnings 'all)"  --eval "(setq org--built-in-p t)"  -f batch-byte-compile org/org-element.el
> >   Warning (emacs): Org version mismatch.  Org loading aborted.
> >   This warning usually appears when a built-in Org version is loaded prior to the more recent Org version.
> >
> >   Version mismatch is commonly encountered in the following situations:
> >
> > This seems to indicate that the compilation is aborted even though the
> > command line sets org--built-in-p non-nil.  What went wrong here?
> 
> I see nothing wrong in the new version of the `org-assert-version'
> macro. The only explanation I can come up with is that
> `org-assert-version' from an older Org version somehow got mixed in.

I think it's the defconst thingy.  E.g., with foo.el which says

  (defconst my-foo nil)

I cannot override the value from the command line:

  emacs -Q -batch --eval "(setq my-foo t)" -l foo.el --eval "(message \"%s\" my-foo)"
   => nil

But if I replace defconst with defvar, it yields t, as I'd expect.

So I think org--built-in-p should be a defvar, not defconst.  Or maybe
you can teach me how to override a defconst?  Stefan, any ideas?

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> I think it's the defconst thingy.  E.g., with foo.el which says
>
>   (defconst my-foo nil)
>
> I cannot override the value from the command line:
>
>   emacs -Q -batch --eval "(setq my-foo t)" -l foo.el --eval "(message \"%s\" my-foo)"
>    => nil
>
> But if I replace defconst with defvar, it yields t, as I'd expect.
>
> So I think org--built-in-p should be a defvar, not defconst.  Or maybe
> you can teach me how to override a defconst?  Stefan, any ideas?

Well. I think I missed that `defconst', unlike `defvar', overrides the
existing variable value. Switching to `defvar' is ok. (Or, it might be
something like (eval-after-load 'org-macs ...), but I see no reason to
prefer it over switching to defvar).

I think it will be best to install the change in Emacs and then back-port
it to Org. Will it be ok?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org
> Date: Sat, 22 Apr 2023 12:39:42 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So I think org--built-in-p should be a defvar, not defconst.  Or maybe
> > you can teach me how to override a defconst?  Stefan, any ideas?
> 
> Well. I think I missed that `defconst', unlike `defvar', overrides the
> existing variable value. Switching to `defvar' is ok. (Or, it might be
> something like (eval-after-load 'org-macs ...), but I see no reason to
> prefer it over switching to defvar).
> 
> I think it will be best to install the change in Emacs and then back-port
> it to Org. Will it be ok?

Sure, fine.

Thanks.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Ihor Radchenko [2023-04-22 12:39:42] wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> I think it's the defconst thingy.  E.g., with foo.el which says
>>
>>   (defconst my-foo nil)
>>
>> I cannot override the value from the command line:
>>
>>   emacs -Q -batch --eval "(setq my-foo t)" -l foo.el --eval "(message \"%s\" my-foo)"
>>    => nil
>>
>> But if I replace defconst with defvar, it yields t, as I'd expect.
>>
>> So I think org--built-in-p should be a defvar, not defconst.  Or maybe
>> you can teach me how to override a defconst?  Stefan, any ideas?
>
> Well. I think I missed that `defconst', unlike `defvar', overrides the
> existing variable value. Switching to `defvar' is ok. (Or, it might be
> something like (eval-after-load 'org-macs ...), but I see no reason to
> prefer it over switching to defvar).
>
> I think it will be best to install the change in Emacs and then back-port
> it to Org. Will it be ok?

While you're there: could you rename that var?
I think it should be called `org--inhibit-version-check` (no "-p" since
it's a var and not a predicate, and more importantly no ill-defined
notion of "built-in").


        Stefan

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I think it will be best to install the change in Emacs and then back-port
>> it to Org. Will it be ok?
>
> While you're there: could you rename that var?
> I think it should be called `org--inhibit-version-check` (no "-p" since
> it's a var and not a predicate, and more importantly no ill-defined
> notion of "built-in").

I have nothing against renaming. Feel free to do it together with
changing to `defvar'.

Let me know when you install the change on Emacs side. 

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Eli Zaretskii <eliz@gnu.org>, bzg@gnu.org, dmitry@gutov.dev,
>  62762@debbugs.gnu.org
> Date: Sun, 23 Apr 2023 08:55:45 +0000
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> I think it will be best to install the change in Emacs and then back-port
> >> it to Org. Will it be ok?
> >
> > While you're there: could you rename that var?
> > I think it should be called `org--inhibit-version-check` (no "-p" since
> > it's a var and not a predicate, and more importantly no ill-defined
> > notion of "built-in").
> 
> I have nothing against renaming. Feel free to do it together with
> changing to `defvar'.
> 
> Let me know when you install the change on Emacs side. 

Done.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Let me know when you install the change on Emacs side. 
>
> Done.

Thanks!
Backported: https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=8eb209984
Kyle, let me know if I did anything wrong.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Dmitry Gutov]

On 24/04/2023 15:29, Ihor Radchenko wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>>> Let me know when you install the change on Emacs side.
>> Done.
> Thanks!
> Backported:https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=8eb209984
> Kyle, let me know if I did anything wrong.

Just to give an update on the problem:

I still see the "Org version mismatch" error fairly often when 
rebuilding Emacs (after which I have to invoke 'touch lisp/org/...' 
again), or when trying to use org-mode in an already-built Emacs. All of 
that with the master branch.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Date: Fri, 24 Nov 2023 19:43:53 +0200
> Cc: monnier@iro.umontreal.ca, bzg@gnu.org, 62762@debbugs.gnu.org,
>  Kyle Meyer <kyle@kyleam.com>
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> I still see the "Org version mismatch" error fairly often when 
> rebuilding Emacs (after which I have to invoke 'touch lisp/org/...' 
> again), or when trying to use org-mode in an already-built Emacs. All of 
> that with the master branch.

That's because the version test was not removed, just suppressed in
the most frequent situations during the build.  But when Org files
need to be actually loaded (e.g., to generate a manual out of some
foo.org source), the test will kick in and will abort the command.

The only way to avoid that is to redesign and reimplement the solution
for version mismatch in Org macros.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Dmitry Gutov]

On 24/11/2023 20:54, Eli Zaretskii wrote:
>> Date: Fri, 24 Nov 2023 19:43:53 +0200
>> Cc:monnier@iro.umontreal.ca,bzg@gnu.org,62762@debbugs.gnu.org,
>>   Kyle Meyer<kyle@kyleam.com>
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> I still see the "Org version mismatch" error fairly often when
>> rebuilding Emacs (after which I have to invoke 'touch lisp/org/...'
>> again), or when trying to use org-mode in an already-built Emacs. All of
>> that with the master branch.
> That's because the version test was not removed, just suppressed in
> the most frequent situations during the build.  But when Org files
> need to be actually loaded (e.g., to generate a manual out of some
> foo.org source), the test will kick in and will abort the command.
> 
> The only way to avoid that is to redesign and reimplement the solution
> for version mismatch in Org macros.

I thought the idea was to skip the version check at least when the Org 
version is built-in? That is, comes with Emacs as opposed to being 
installed from ELPA.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Date: Fri, 24 Nov 2023 20:56:31 +0200
> Cc: yantar92@posteo.net, monnier@iro.umontreal.ca, bzg@gnu.org,
>  62762@debbugs.gnu.org, kyle@kyleam.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 24/11/2023 20:54, Eli Zaretskii wrote:
> >> Date: Fri, 24 Nov 2023 19:43:53 +0200
> >> Cc:monnier@iro.umontreal.ca,bzg@gnu.org,62762@debbugs.gnu.org,
> >>   Kyle Meyer<kyle@kyleam.com>
> >> From: Dmitry Gutov<dmitry@gutov.dev>
> >>
> >> I still see the "Org version mismatch" error fairly often when
> >> rebuilding Emacs (after which I have to invoke 'touch lisp/org/...'
> >> again), or when trying to use org-mode in an already-built Emacs. All of
> >> that with the master branch.
> > That's because the version test was not removed, just suppressed in
> > the most frequent situations during the build.  But when Org files
> > need to be actually loaded (e.g., to generate a manual out of some
> > foo.org source), the test will kick in and will abort the command.
> > 
> > The only way to avoid that is to redesign and reimplement the solution
> > for version mismatch in Org macros.
> 
> I thought the idea was to skip the version check at least when the Org 
> version is built-in?

AFAIU, this is impossible, as long as Org uses macros that can change
in incompatible ways, and as long as the decision of Org developers is
in effect to force recompilation each time they bump the version, so
as to avoid weird problems due to changes in macros.

The change we installed in the Emacs build procedure simply delays the
error until Org is actually used (as opposed to just being compiled as
part of the build update).

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Dmitry Gutov]

On 25/11/2023 09:08, Eli Zaretskii wrote:
> The change we installed in the Emacs build procedure simply delays the
> error until Org is actually used (as opposed to just being compiled as
> part of the build update).

In that case, could we have a way to avoid rebuilding the manuals? I 
recompile Emacs much more often than I read the manual, and in most 
cases it doesn't need to be the latest version.

Rebuilding the manual is in general the slowest step of the minor 
rebuilds, and skipping it could avoid this error to (at least until 
org-mode is used, yes).

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Date: Sat, 25 Nov 2023 14:38:36 +0200
> Cc: yantar92@posteo.net, monnier@iro.umontreal.ca, bzg@gnu.org,
>  62762@debbugs.gnu.org, kyle@kyleam.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 25/11/2023 09:08, Eli Zaretskii wrote:
> > The change we installed in the Emacs build procedure simply delays the
> > error until Org is actually used (as opposed to just being compiled as
> > part of the build update).
> 
> In that case, could we have a way to avoid rebuilding the manuals? I 

No one can guarantee that building the manuals is the only situation
where Org will be invoked during the build.  So this doesn't sound
like worth the trouble to me.  But you could always manually touch all
the manuals in the info/ subdirectory, before building, if you don't
care about keeping the manuals up-to-date.

> Rebuilding the manual is in general the slowest step of the minor 
> rebuilds, and skipping it could avoid this error to (at least until 
> org-mode is used, yes).

Manuals are rebuilt relatively rarely, and those in doc/emacs and
doc/lispref are rebuilt in parallel to other build commands, so "the
slowest step" is not accurate IME.  Another thing to keep in mind is
that whenever the Org macros change, frequently so does the Org
manual...

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Dmitry Gutov]

On 25/11/2023 14:59, Eli Zaretskii wrote:
>> Date: Sat, 25 Nov 2023 14:38:36 +0200
>> Cc:yantar92@posteo.net,monnier@iro.umontreal.ca,bzg@gnu.org,
>>   62762@debbugs.gnu.org,kyle@kyleam.com
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 25/11/2023 09:08, Eli Zaretskii wrote:
>>> The change we installed in the Emacs build procedure simply delays the
>>> error until Org is actually used (as opposed to just being compiled as
>>> part of the build update).
>> In that case, could we have a way to avoid rebuilding the manuals? I
> No one can guarantee that building the manuals is the only situation
> where Org will be invoked during the build.  So this doesn't sound
> like worth the trouble to me.  But you could always manually touch all
> the manuals in the info/ subdirectory, before building, if you don't
> care about keeping the manuals up-to-date.

Ok, I'll do that, thank you.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I don't.  Say, why not to store hash of the macro sources in the byte
> code and verifying the hash when re-compilation is requested?  Though I
> am not that much familiar with Emacs source compilation.

In theory it's possible.  Nobody has worked on it :-)
Note that it's not just macros but defsubsts as well.

> Sure, in the context of Emacs compilation. The code in question is
> mostly aiming at newer Org installed on top of built-in Org. We are
> consistently getting issues related to incorrect macro expansion and
> mixing different Org versions.

Someone™ *really* needs to sit down and fix the underlying problem in
`package.el`.  The current "solution" in `package.el` *should* work (it
should reload the new `org-macs.el` on top of the old one, which
I think should avoid all the problems seen in practice for Org), so it
seems we just have a plain bug in the implementation of our "solution".
[ Which is good: it should be simple to fix, compared to trying to come
  up with another solution.  ]

>> Anyway, any other words of wisdom?  Like which org/*.el files actually
>> need to be recompiled in this case?  Or are you saying all of them
>> need to be recompiled?

[ BTW, `rm lisp/org/*.elc` is a better solution since it avoids changing
  timestamps on all those .el files.  ]

> What we might do to work around the problem is detecting Emacs compilation
> and disable the check.  Is there a way to detect that Emacs source is
> being compiled from Elisp?

This sounds like adding more brittle hacks on top of brittle hacks.

>> I also am not sure I understand the logic of this test: AFAIU it only
>> tests the version string, not the actual macros that might have been
>> changed.  Are you saying that Org bumps its version string each time
>> _any_ Org macro is modified?
>
> No. We originally tested by commit hash. Version string check is a
> trade-off between the problem with ELPA builds (see the links in my
> previous message) and the need to detect mixed installation problems.

BTW, have you tried to use a test along the lines of: look through
`load-history` to see if we loaded org-* files from a different directory
than the one in which `load-file-name` resides?
[ We may need to adjust the test to account for the `package.el`
  "solution" which will result in the above test detecting a mixed
  version situation even though it "should" work correctly.  ]


        Stefan

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  bzg@gnu.org,  dmitry@gutov.dev,
>   62762@debbugs.gnu.org
> Date: Sat, 22 Apr 2023 10:30:01 -0400
> 
> [ BTW, `rm lisp/org/*.elc` is a better solution since it avoids changing
>   timestamps on all those .el files.  ]

With current VCSs, the time stamp of versioned files is more or less
random anyway.  So it isn't worth it trying too hard to keep those
random time stamps.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I don't.  Say, why not to store hash of the macro sources in the byte
>> code and verifying the hash when re-compilation is requested?  Though I
>> am not that much familiar with Emacs source compilation.
>
> In theory it's possible.  Nobody has worked on it :-)
> Note that it's not just macros but defsubsts as well.

... and other define-inlines.

>> Sure, in the context of Emacs compilation. The code in question is
>> mostly aiming at newer Org installed on top of built-in Org. We are
>> consistently getting issues related to incorrect macro expansion and
>> mixing different Org versions.
>
> Someone™ *really* needs to sit down and fix the underlying problem in
> `package.el`.  The current "solution" in `package.el` *should* work (it
> should reload the new `org-macs.el` on top of the old one, which
> I think should avoid all the problems seen in practice for Org), so it
> seems we just have a plain bug in the implementation of our "solution".
> [ Which is good: it should be simple to fix, compared to trying to come
>   up with another solution.  ]

AFAIK, the current re-loading approach in package.el does help.
But not all the Org users are using that new Emacs versions.
And not all the Org users are using package.el.

>> What we might do to work around the problem is detecting Emacs compilation
>> and disable the check.  Is there a way to detect that Emacs source is
>> being compiled from Elisp?
>
> This sounds like adding more brittle hacks on top of brittle hacks.

Sure. But I really have no better ideas, especially considering backward
compatibility requirements down to Emacs 26.

>> No. We originally tested by commit hash. Version string check is a
>> trade-off between the problem with ELPA builds (see the links in my
>> previous message) and the need to detect mixed installation problems.
>
> BTW, have you tried to use a test along the lines of: look through
> `load-history` to see if we loaded org-* files from a different directory
> than the one in which `load-file-name` resides?

Yup. Several problems here:
1. Not all the org-* files are a part of Org.
2. This will completely block the possibility for users to provide custom
   versions of Org libraries, when they need to.
3. We actually have `org-version' that is using this approach to
   indicate problems without blocking staff, but it still misses many
   cases.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> BTW, have you tried to use a test along the lines of: look through
>> `load-history` to see if we loaded org-* files from a different directory
>> than the one in which `load-file-name` resides?
>
> Yup. Several problems here:
> 1. Not all the org-* files are a part of Org.

Ah right, we can't just match "org-*", indeed.

Maybe if we can identify a "first org-*.el file" (i.e. a file which
*any* use of some part of Org mode would necessarily load pretty early
on), then we can reduce this check to:

- check if that "first org-*.el file" is already loaded.
- if no, load it
- if yes, compare the output of `locate-file` to what is found in
  `load-history`, and in case it doesn't match (because of a new
  directory in `load-path`), signal an error (or maybe force a reload
  instead?).

We could even provide a new `require-with-shadow-check` function that
does just that.


        Stefan

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Maybe if we can identify a "first org-*.el file" (i.e. a file which
> *any* use of some part of Org mode would necessarily load pretty early
> on), then we can reduce this check to:

Such file is org-macs.el

> - check if that "first org-*.el file" is already loaded.
> - if no, load it
> - if yes, compare the output of `locate-file` to what is found in
>   `load-history`, and in case it doesn't match (because of a new
>   directory in `load-path`), signal an error (or maybe force a reload
>   instead?).
>
> We could even provide a new `require-with-shadow-check` function that
> does just that.

May you provide an approximate code for that function?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> May you provide an approximate code for that function?

Something like (guaranteed 100% untested):

    (defun my-require-with-shadow-check (feature &optional reload)
      (if (not (featurep feature))
          (require feature)
        (let ((file (locate-file (symbol-name feature)
                                 load-path (get-load-suffixes))))
          (if (assoc file load-history)
              ;; The desired file is already loaded, we're clear!
              nil
            ;; The feature is already provided, but from some other file
            ;; than expected.  Presumably because `load-path` has been
            ;; changed since the file was loaded.  This is likely a sign
            ;; that we're loading a mix of files from different versions.
            ;; That can spell trouble.
            (if reload
                ;; Try and fix the situation by force-reloading the file
                ;; This may fix the problem but only if that file doesn't
                ;; `require` any other affected file, or if it also
                ;; uses `my-require-with-shadow-check` to similarly reload
                ;; any other affected file.
                (load file nil t t)
              ;; We could try and find the "other/older" file in `load-history`
              ;; to provide better feedback to the user.
              (error "Mixed versions!!"))))))


-- Stefan

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> May you provide an approximate code for that function?
>
> Something like (guaranteed 100% untested):
>
>     (defun my-require-with-shadow-check (feature &optional reload)

Thanks!
This looks interesting, if we replace all the requires in Org with
`my-require-with-shadow-check'.

Although, AFAIU, it does not address the problem with stale .elc files
kept from the previous Org version in the same folder (AFAIR, it is what
often happens in ELPA builds).

And we somehow need to make sure that `my-require-with-shadow-check' is
available for all its users. Just like with `org-assert-version', older
Org may not provide the function and compilation/loading can fail with
"Symbol function is void".

Your idea with providing built-in `require-with-shadow-check` will
partially solve the latter problem. At least, on the latest Emacs.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> May you provide an approximate code for that function?
>>
>> Something like (guaranteed 100% untested):
>>
>>     (defun my-require-with-shadow-check (feature &optional reload)
>
> Thanks!
> This looks interesting, if we replace all the requires in Org with
> `my-require-with-shadow-check'.

If we make it signal an error, then I think we only need it for the
requires that load "the one first file" (which you suggest could be
`org-macs.el`).

> Although, AFAIU, it does not address the problem with stale .elc files
> kept from the previous Org version in the same folder (AFAIR, it is what
> often happens in ELPA builds).

I don't understand what you mean here: when `package.el` compiles the
files of the Org package, that package has just been untarred into
a fresh new directory, so there should never be any stale `.elc` files
in there.

Stale `.elc` files can happen for "in place builds", such as when doing
`git pull; make` in Emacs's source directory, but we just disabled
the mixed-version check for that case...

> And we somehow need to make sure that `my-require-with-shadow-check' is
> available for all its users.

Ah, the fun of bootstrapping :-)


        Stefan

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> This looks interesting, if we replace all the requires in Org with
>> `my-require-with-shadow-check'.
>
> If we make it signal an error, then I think we only need it for the
> requires that load "the one first file" (which you suggest could be
> `org-macs.el`).

This might work.

Max, do you see any obvious downsides in Stefan's idea about consulting
`load-history' vs. `load-path'?

>> Although, AFAIU, it does not address the problem with stale .elc files
>> kept from the previous Org version in the same folder (AFAIR, it is what
>> often happens in ELPA builds).
>
> I don't understand what you mean here: when `package.el` compiles the
> files of the Org package, that package has just been untarred into
> a fresh new directory, so there should never be any stale `.elc` files
> in there.

I am referring to https://orgmode.org/list/jwvsfkv5s7l.fsf-monnier+emacs@gnu.org

>> And we somehow need to make sure that `my-require-with-shadow-check' is
>> available for all its users.
>
> Ah, the fun of bootstrapping :-)

I guess that the cleanest way could be putting
`require-with-shadow-check' into a new file. Then, we have no chance of
this file being loaded by old Org version.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Alan Mackenzie]

Hello, Ihor.

It occurs to me that perhaps the CC Mode solution to the original
problem might be useful.  It was written by my predecessor at CC Mode,
Martin Stjernholm.  In ~20 years of using it, I've never had problems
with using incorrect versions of macros, or anything like that.

To quote part of the commentary from lisp/progmodes/cc-bytecomp.el:

;; This file is used to ensure that the CC Mode files are correctly
;; compiled regardless the environment (e.g. if an older CC Mode with
;; outdated macros are loaded during compilation).  It also provides
;; features to defeat the compiler warnings for selected symbols.

It works by pushing the correct directories onto load-path at the start
of each compilation, and popping them off again at the end.

Feel free to consider using this approach!

-- 
Alan Mackenzie (Nuremberg, Germany).



On Wed, May 03, 2023 at 10:30:55 +0000, Ihor Radchenko wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:

> >> This looks interesting, if we replace all the requires in Org with
> >> `my-require-with-shadow-check'.
> >
> > If we make it signal an error, then I think we only need it for the
> > requires that load "the one first file" (which you suggest could be
> > `org-macs.el`).

> This might work.

> Max, do you see any obvious downsides in Stefan's idea about consulting
> `load-history' vs. `load-path'?

> >> Although, AFAIU, it does not address the problem with stale .elc files
> >> kept from the previous Org version in the same folder (AFAIR, it is what
> >> often happens in ELPA builds).
> >
> > I don't understand what you mean here: when `package.el` compiles the
> > files of the Org package, that package has just been untarred into
> > a fresh new directory, so there should never be any stale `.elc` files
> > in there.

> I am referring to https://orgmode.org/list/jwvsfkv5s7l.fsf-monnier+emacs@gnu.org

> >> And we somehow need to make sure that `my-require-with-shadow-check' is
> >> available for all its users.
> >
> > Ah, the fun of bootstrapping :-)

> I guess that the cleanest way could be putting
> `require-with-shadow-check' into a new file. Then, we have no chance of
> this file being loaded by old Org version.

> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Date: Wed, 3 May 2023 21:37:11 +0000
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, bzg@gnu.org, dmitry@gutov.dev,
>   Eli Zaretskii <eliz@gnu.org>, 62762@debbugs.gnu.org,
>   Max Nikulin <manikulin@gmail.com>
> From: Alan Mackenzie <acm@muc.de>
> 
> It occurs to me that perhaps the CC Mode solution to the original
> problem might be useful.  It was written by my predecessor at CC Mode,
> Martin Stjernholm.  In ~20 years of using it, I've never had problems
> with using incorrect versions of macros, or anything like that.

No, the situation with CC Mode solution is far from ideal.  It's the
reason that we have had the dependencies below in lisp/Makefile.in for
the past 15 years:

  # https://debbugs.gnu.org/1004
  # CC Mode uses a compile time macro system which causes a compile time
  # dependency in cc-*.elc files on the macros in other cc-*.el and the
  # version string in cc-defs.el.
  $(lisp)/progmodes/cc-align.elc\
   $(lisp)/progmodes/cc-cmds.elc $(lisp)/progmodes/cc-compat.elc\
   $(lisp)/progmodes/cc-engine.elc $(lisp)/progmodes/cc-fonts.elc\
   $(lisp)/progmodes/cc-langs.elc $(lisp)/progmodes/cc-menus.elc\
   $(lisp)/progmodes/cc-mode.elc $(lisp)/progmodes/cc-styles.elc\
   $(lisp)/progmodes/cc-vars.elc: \
     $(lisp)/progmodes/cc-bytecomp.elc $(lisp)/progmodes/cc-defs.elc

  $(lisp)/progmodes/cc-align.elc $(lisp)/progmodes/cc-cmds.elc: \
     $(lisp)/progmodes/cc-vars.elc $(lisp)/progmodes/cc-engine.elc

  $(lisp)/progmodes/cc-compat.elc: \
     $(lisp)/progmodes/cc-vars.elc $(lisp)/progmodes/cc-styles.elc \
     $(lisp)/progmodes/cc-engine.elc

  $(lisp)/progmodes/cc-defs.elc: $(lisp)/progmodes/cc-bytecomp.elc

  $(lisp)/progmodes/cc-engine.elc: $(lisp)/progmodes/cc-langs.elc \
     $(lisp)/progmodes/cc-vars.elc

  $(lisp)/progmodes/cc-fonts.elc: $(lisp)/progmodes/cc-langs.elc \
     $(lisp)/progmodes/cc-vars.elc $(lisp)/progmodes/cc-engine.elc

  $(lisp)/progmodes/cc-langs.elc: $(lisp)/progmodes/cc-vars.elc

  $(lisp)/progmodes/cc-mode.elc: $(lisp)/progmodes/cc-langs.elc \
     $(lisp)/progmodes/cc-vars.elc $(lisp)/progmodes/cc-engine.elc \
     $(lisp)/progmodes/cc-styles.elc $(lisp)/progmodes/cc-cmds.elc \
     $(lisp)/progmodes/cc-align.elc $(lisp)/progmodes/cc-menus.elc

  $(lisp)/progmodes/cc-styles.elc: $(lisp)/progmodes/cc-vars.elc \
     $(lisp)/progmodes/cc-align.elc

Each time some of the cc-*.el files change we recompile all of them.
The difference between this and Org is that Org has many more files,
so spelling out their dependencies is impractical (that was the first
possible solution I thought about when I tried to solve the Org
issue).

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Alan Mackenzie]

Hello, Eli.

On Thu, May 04, 2023 at 08:35:55 +0300, Eli Zaretskii wrote:
> > Date: Wed, 3 May 2023 21:37:11 +0000
> > Cc: Stefan Monnier <monnier@iro.umontreal.ca>, bzg@gnu.org, dmitry@gutov.dev,
> >   Eli Zaretskii <eliz@gnu.org>, 62762@debbugs.gnu.org,
> >   Max Nikulin <manikulin@gmail.com>
> > From: Alan Mackenzie <acm@muc.de>
> > 
> > It occurs to me that perhaps the CC Mode solution to the original
> > problem might be useful.  It was written by my predecessor at CC Mode,
> > Martin Stjernholm.  In ~20 years of using it, I've never had problems
> > with using incorrect versions of macros, or anything like that.

> No, the situation with CC Mode solution is far from ideal.  It's the
> reason that we have had the dependencies below in lisp/Makefile.in for
> the past 15 years:

The dependencies in Lisp/Makefile are there because there are actual
dependencies between the CC Mode source files.  I think this is
orthogonal to the problems in upstream CC Mode which Martin Stjernholm
solved (or, at least, worked around) with cc-bytecomp.el.  They were to
do with getting versions of macro files mixed up.  If I've understood
Ihor correctly (about which I'm far from sure), org was suffering the
same problem as CC Mode was ~20 years ago, and that is the reason for
the version check in org's build system.

Unfortunately, org's fix for its upstream problem leaks downstream into
Emacs and causes build failure, or at the very least _has_ caused such
failure.  My last post was suggesting that the mechanism in
cc-bytecomp.el might be able to take the place of that version check,
causing less aggravation for those building Emacs.

>   # https://debbugs.gnu.org/1004
>   # CC Mode uses a compile time macro system which causes a compile time
>   # dependency in cc-*.elc files on the macros in other cc-*.el and the
>   # version string in cc-defs.el.

[ .... ]

> Each time some of the cc-*.el files change we recompile all of them.
> The difference between this and Org is that Org has many more files,
> so spelling out their dependencies is impractical (that was the first
> possible solution I thought about when I tried to solve the Org
> issue).

There are other subprojects in Emacs with several/many source files,
such as gnus, cedet, and calc.  How do they manage to stay coherent
without special mechanisms?

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Date: Thu, 4 May 2023 14:02:37 +0000
> Cc: yantar92@posteo.net, monnier@iro.umontreal.ca, bzg@gnu.org,
>   dmitry@gutov.dev, 62762@debbugs.gnu.org, manikulin@gmail.com
> From: Alan Mackenzie <acm@muc.de>
> 
> > No, the situation with CC Mode solution is far from ideal.  It's the
> > reason that we have had the dependencies below in lisp/Makefile.in for
> > the past 15 years:
> 
> The dependencies in Lisp/Makefile are there because there are actual
> dependencies between the CC Mode source files.

The same is true for Org.  The reason for the dependency is
immaterial; the result is the same.

> > Each time some of the cc-*.el files change we recompile all of them.
> > The difference between this and Org is that Org has many more files,
> > so spelling out their dependencies is impractical (that was the first
> > possible solution I thought about when I tried to solve the Org
> > issue).
> 
> There are other subprojects in Emacs with several/many source files,
> such as gnus, cedet, and calc.  How do they manage to stay coherent
> without special mechanisms?

The "secret" is not to use macros, or use them as little as possible.
Stale expanded macros from *.elc files cause breakage.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 03/05/2023 17:30, Ihor Radchenko wrote:
> Stefan Monnier writes:
> 
>>> This looks interesting, if we replace all the requires in Org with
>>> `my-require-with-shadow-check'.
...
> Max, do you see any obvious downsides in Stefan's idea about consulting
> `load-history' vs. `load-path'?

First of all, I think, incremental builds are broken in Emacs. 
`org-assert-version' is just the most apparent manifestation.

gcc supports generation of dependency files as a side effect of 
preprocessing for decades, see
- (info "(make) Automatic-Prerequisites")
 
https://www.gnu.org/software/make/manual/html_node/Automatic-Prerequisites.html
- https://gcc.gnu.org/news/dependencies.html
   Dependency Generation Improvements. 22 January 2001
- https://gcc.gnu.org/onlinedocs/gcc/Preprocessor-Options.html#index-MF

The idea is that clean build does not require dependency map. For a 
rebuild, .d files are included into Makefile, so make is able to 
determine proper order of compilation.

Perhaps a similar trick may be done in elisp by advicing e.g. `load'.

Instead of dependency tracking Makefile in Emacs uses much more limited 
approach with the main-first target. That is why incremental build may 
easily result in mixed-version compilation. Imprinted Org version just 
detects such case.

By the way, generated dependency map might help to properly reload a 
package update without restarting of Emacs.

It seems, despite "make" in the bug title, it has been decided to 
dedicate this issue to plumbing of `org-assert-version'.

Inspecting `load-history' might be tried, however such approach may 
cause other issues.

I have heard that some users complain concerning Emacs startup time. 
Likely a hashmap will be required in addition to the `load-history' list 
to avoid performance degradation. I am unsure if the following idea have 
some benefits: check shadowing when `load-path' is modified, not for 
each file loaded from the newly added directory.

I am unsure that path comparison is able to detect a problem when a user 
reloads Org after git pull and compiling new version. Files with new 
functions and macros are loaded from the same directory.

In respect to incremental builds, `org-assert-version' is a disaster. 
Any update requires full recompiling. It is a reason why I always 
considered it as a kludge. Unfortunately I do not have a better idea.

We need a tool that clearly informs users that they have issues with 
their configuration files, multiple versions of a package are available 
and it causes mixed version loading with inconsistent function 
definitions or, even worse, mixed version compiling with obsolete macro 
definitions. Emacs restart with fixed loading does not help in the 
latter case.

Before introducing of `org-assert-version', errors were obscure: 
undefined functions or incompatible arguments (for internal functions). 
Unfortunately `org-assert-version' have caused another stream of complains.

The issue is not limited to package.el or replacing built-in package. 
Various ways to load packages are used, so a couple of versions may 
appear in load-path even for non built-in package.

The tool should detect version mismatch during loading, reloading, and 
compiling of an update.

I think, both aspects must be addressed:
- Improving dependency handling during incremental builds
- Informing users about issues with loading of multi-file packages.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> First of all, I think, incremental builds are broken in Emacs.

Indeed, but in practice they usually work fine.  So in the absence of
a better solution, we stay with this hack and tell people to try
`make bootstrap` when there's a problem.

Also, the use of `load-prefer-newer` in lisp/Makefile eliminates most of
the brokenness we used to have in our incremental builds.

> `org-assert-version' is just the most apparent manifestation.

AFAIK `org-assert-version` tries to solve a different problem.  Indeed,
we recently introduced `org--inhibit-version-check` specifically so as
not to use `org-assert-version` for Emacs's broken incremental builds
(i.e. we decided we preferred that brokenness there).

> gcc supports generation of dependency files as a side effect of
> preprocessing for decades, see
[...]
> Perhaps a similar trick may be done in elisp by advicing e.g. `load'.

I encourage people to try that out, yes.
Having been there (many years ago), I can already warn them that an
important problem will be the presence of cyclic dependencies.

> Instead of dependency tracking Makefile in Emacs uses much more limited
> approach with the main-first target.

Hmm... no, `main-first` doesn't have much to do with dependencies (and
things like deciding when to *re*compile a file), it's concerned with
compile-time performance (typically for the first compilation, when the
`.d` dependencies wouldn't be available yet).

> That is why incremental build may easily result in mixed-version
> compilation.

Actually, now that we set `load-prefer-newer`, it is rarely a problem.
[ Of course, it can still be a problem, but only in cases such as when
  a macro is changed such that the old expansion is not compatible with
  code using the new definitions.  This happens (e.g. when we changed
  structs to use `record` instead of `vector`), but not fairly rarely.  ]

> By the way, generated dependency map might help to properly reload a package
> update without restarting of Emacs.

BTW, I was reminded recently that `load-history` keeps track of
`require`s so we already have most of that info at hand.

> I have heard that some users complain concerning Emacs startup time. Likely
> a hashmap will be required in addition to the `load-history' list to avoid
> performance degradation. I am unsure if the following idea have some
> benefits: check shadowing when `load-path' is modified, not for each file
> loaded from the newly added directory.

The idea was to check the shadowing only for one specific file, so as to
try and keep the added cost in check.  Checking shadowings when
`load-path` is modified is harder (because `load-history` doesn't
remember which name was used to load a file, so it could be confused and
think a file like `lisp/progmodes/compile.el` shadows a previously
loaded `lisp/cedet/srecode/compile.el`) and could be more costly
(because it has to check for "all" files).

> I am unsure that path comparison is able to detect a problem when a user
> reloads Org after git pull and compiling new version.

[ AFAICT, there are different situations with related yet different
  problems that manifest slightly differently and that may be solved
  differently as well, so please be specific when you mention
  problematic scenarios.  ]

If you mean `git pull; make` in Emacs's source code, then AFAIK this
should almost never be a problem (thanks to `load-prefer-newer`), and if
it is, it's a general problem that's not specific to Org, and "we"
changed `org-assert-version` so as not to bother trying to solve
this case.

If you mean `git pull; make` in Org's source repository, then I must
admit that I don't have much experience with it, but make Org's make
file could set `load-prefer-newer` (or use some hack to autogenerate
`.d` dependency files :-).

`my-require-with-shadow-check` is instead aimed at the case where the
users try to load a mix of two different Org versions in the same Emacs
sessions, either because of things like:
- they're byte-compiling Org-2 in an Emacs that has Org-1 already loaded.
- they inadvertently end up loading part of Org-1 early in their init
  file before they set their `load-path` to point to Org-2.

I understand Org people have had lots of problems with mixed-versions in
the past, but I personally don't know which circumstances have been more
often at the source of those problems, so maybe my suggestion addresses
a problem that's not very important, indeed.

I my own experience `git pull; make` in Emacs's source code has
virtually never caused me trouble with Org files, and it's never cause
any trouble in my checkout of Org's source code either, which is why
I'm suggesting something like `my-require-with-shadow-check`, which
targets other scenarios.

> In respect to incremental builds, `org-assert-version' is a disaster.
> Any update requires full recompiling.

Indeed, I used to disable it locally (before `org--inhibit-version-check`).

> It is a reason why I always considered it as a kludge.  Unfortunately
> I do not have a better idea.

Maybe a clear set of examples of the kinds of problems that
`org-assert-version` aims to catch would be a good start.
For me it's still not really clear.

> The issue is not limited to package.el or replacing built-in package.
> Various ways to load packages are used, so a couple of versions may
> appear in load-path even for non built-in package.

So far I haven't seen enough of the diversity of situations are (source
of) problems to have a clear idea of what a good solution should look
like, admittedly.

`my-require-with-shadow-check` is only based on my intuition that it
might catch the most common issues where two different Org versions are
used in the same Emacs session, which seems to cover most of the
problems (except for those due to `git pull; make`, which seem to be
qualitatively of a different kind).


        Stefan

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 05/05/2023 04:53, Stefan Monnier wrote:
>> First of all, I think, incremental builds are broken in Emacs.
> 
> Indeed, but in practice they usually work fine.
...
> Also, the use of `load-prefer-newer` in lisp/Makefile eliminates most of
> the brokenness we used to have in our incremental builds.

`load-prefer-newer' is a kludge as well. E.g. Python people migrated 
from comparison of timestamps to inscribing of .py file hash into byte 
compiled .pyc files. So if the hash in .pyc does not match .py content 
then .pyc file is recompiled or just ignored.

By the way, `load-prefer-newer' make things even worse in the case of 
`org-assert-version' when full dependency tree is not available. I have 
tried to add dependency for all lisp/org/*.elc files on 
lisp/org/org-macs.elc and lisp/org/org-version.el, but it is not enough. 
During recompilation `require' prefers .elc files (.el files not 
changed, so they are older) with inscribed stale `org-version' causing 
compilation error. If .el files were loaded, recompiling would be 
slower, but with up to date macro definitions. So in general, due to 
`load-prefer-newer' the chance to get stale macro during incremental 
build is higher than .el files are preferred. I admit, it is not common 
case when definition of a macro depends on another macro loaded from 
another file.

I am unsure if it is possible to express in make that all lisp/org/*.elc 
files must be removed on change of 
lisp/org/{org-version.el,org-macs.el}. It might alleviate the issue till 
implementation of properly dependency tracking.

>> `org-assert-version' is just the most apparent manifestation.
> 
> AFAIK `org-assert-version` tries to solve a different problem.

Of course `org-assert-version' was introduced to solve another problem, 
but it highlighted issues with dependency handling during incremental 
builds.

> Indeed,
> we recently introduced `org--inhibit-version-check` specifically so as
> not to use `org-assert-version` for Emacs's broken incremental builds
> (i.e. we decided we preferred that brokenness there).

`org--inhibit-version-check' is even uglier kludge that partially 
deactivated `org-assert-version' and just hides the issue with 
dependencies for incremental builds.

>> gcc supports generation of dependency files as a side effect of
>> preprocessing for decades, see
> [...]
>> Perhaps a similar trick may be done in elisp by advicing e.g. `load'.

Another idea: while single file is compiled per emacs invocation 
`load-history' may be compared before and after byte compilation.

> I can already warn them that an
> important problem will be the presence of cyclic dependencies.

In the C and C++ world the solution for cyclic dependencies is forward 
declarations. Some kind of such approach I see in Org as well. 
lisp/org/ol.el and lisp/org/org-element.el are mutually dependent. 
org-element.el requires 'ol, while the latter just declares functions 
from 'org-element.

When dependency files are generated, it is possible to create a report 
on cyclic dependency and to disentangle them.

>> Instead of dependency tracking Makefile in Emacs uses much more limited
>> approach with the main-first target.
> 
> Hmm... no, `main-first` doesn't have much to do with dependencies (and
> things like deciding when to *re*compile a file), it's concerned with
> compile-time performance (typically for the first compilation, when the
> `.d` dependencies wouldn't be available yet).

My impression is that main-first defines order of compilation. In this 
sense it servers for the same purpose as dependencies.

>> By the way, generated dependency map might help to properly reload a package
>> update without restarting of Emacs.
> 
> BTW, I was reminded recently that `load-history` keeps track of
> `require`s so we already have most of that info at hand.

I agree that heuristics based on `load-history' mostly works, but I 
suspect corner cases might exist. I have not inspected `org-reload' 
closely and what Emacs core offers instead.

> The idea was to check the shadowing only for one specific file, so as to
> try and keep the added cost in check.

That specific file must be checked for each file loaded from the updated 
directory. So hash O(1) should be better than scanning the list.

>> I am unsure that path comparison is able to detect a problem when a user
>> reloads Org after git pull and compiling new version.
> 
> If you mean `git pull; make` in Emacs's source code,

I mean reloading of updated Org from a clone of the org-mode repository 
to emacs installed as a system package (or as an independent custom 
built). A recent case where origin of the problem remained obscure:

Colin Baxter to emacs-orgmode. Why am I being told to use "straight.el"? 
Fri, 21 Apr 2023 10:42:27 +0100. 
https://list.orgmode.org/87ildpbmgs.fsf@yandex.com

Notice that mixed version loading may happen without changing of 
`load-path'. New org is loaded from the same directory.

> If you mean `git pull; make` in Org's source repository, then I must
> admit that I don't have much experience with it, but make Org's make
> file could set `load-prefer-newer`

Not enough, see beginning of the message.

> `my-require-with-shadow-check` is instead aimed at the case where the
> users try to load a mix of two different Org versions in the same Emacs
> sessions, either because of things like:
> - they're byte-compiling Org-2 in an Emacs that has Org-1 already loaded.

Does in help in the following case?
1. Base Org part is loaded on opening of some .org file.
2. Org in that directory is updated and recompiled.
3. New Org feature is loaded (autoloaded or by explicit call of e.g.
    (require ob-shell))

> I my own experience `git pull; make` in Emacs's source code has
> virtually never caused me trouble with Org files,

I recall issues after introducing of `org-encode-time' and finally Ihor 
replaced original macro to a less efficient function. However I am not 
sure what was the real reason:
- Some issue with conditional macro definition and native compilation
- Incomplete incremental rebuilds
- Macro was not properly defined so it was disappearing from byte 
compiled code.
However "undefined" complains might be similar to what I saw for 
`org-assert-version'.

>> In respect to incremental builds, `org-assert-version' is a disaster.
>> Any update requires full recompiling.
> 
> Indeed, I used to disable it locally (before `org--inhibit-version-check`).

By disaster I mean increased build time. `org--inhibit-version-check' 
allows mixed version compilation for macros unrelated to 
`org-assert-version', it is another disaster.

> Maybe a clear set of examples of the kinds of problems that
> `org-assert-version` aims to catch would be a good start.
> For me it's still not really clear.

- You are almost certainly going to ignore package.el issues in released 
Emacs versions including 28.
- Besides package.el there are other package managers. I have heard of 
weird issues with straight.el and org, but I have no links describing 
details.
- Some users just clone org-mode repository and add that directory to 
`load-path'. Directory remains the same after updates.
- Users tend to load some part of org before adding new version to 
load-path: explicit require, through dependencies, unsure if org file as 
startup screen may cause it as well
- User wish to update Org without restarting of Emacs. Recompiling may 
be called from Emacs (e.g. package.el) or by make or other scripts.

The goals:
- prevent mixed version compiling
- prevent mixed version loading
- error message should be clear for users explaining what actions they 
should perform for recovery from broken state.

I have not isolated an issue in Emacs-28 with package.el and difference 
with emacs binary is running from source tree or from install tree 
(.el+.elc vs. el.gz+.elc files in the directory with built-in Org and 
preference of .el vs. elc in require), so I can not tell if it affects 
other package managers in newer Emacs. It is the case when 
`org-assert-version' prevents loading of mixed compilation result, but 
user experience is terrible since simple recompilation does not help.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 05/05/2023 11:18, Max Nikulin wrote:
> On 05/05/2023 04:53, Stefan Monnier wrote:
>> Also, the use of `load-prefer-newer` in lisp/Makefile eliminates most of
>> the brokenness we used to have in our incremental builds.
> 
> `load-prefer-newer' is a kludge as well. E.g. Python people migrated 
> from comparison of timestamps to inscribing of .py file hash into byte 
> compiled .pyc files. So if the hash in .pyc does not match .py content 
> then .pyc file is recompiled or just ignored.

I have realized that neither `load-prefer-newer' not checking hash of 
the source .el file can help per se when an .elc file becomes stale due 
to update of a macro in a require'd file.

The robust way is to define compilation order through dependencies, 
preferably autogenerated ones). An alternative is to hope that usually 
it does not hurt and you are ready to remove .elc files (e.g. by make 
bootstrap) when you faced an apparent error or just suspect mixed 
version compilation as the cause of noticed strange behavior (I named 
this broken incremental builds).

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Date: Fri, 5 May 2023 12:27:25 +0700
> From: Max Nikulin <manikulin@gmail.com>
> Cc: Ihor Radchenko <yantar92@posteo.net>, Eli Zaretskii <eliz@gnu.org>,
>  bzg@gnu.org, dmitry@gutov.dev, 62762@debbugs.gnu.org,
>  Alan Mackenzie <acm@muc.de>
> 
> I have realized that neither `load-prefer-newer' not checking hash of 
> the source .el file can help per se when an .elc file becomes stale due 
> to update of a macro in a require'd file.

Yes.

> The robust way is to define compilation order through dependencies, 
> preferably autogenerated ones).

This doesn't work in Emacs, in general, due to circular dependencies.

> An alternative is to hope that usually it does not hurt and you are
> ready to remove .elc files (e.g. by make bootstrap) when you faced
> an apparent error or just suspect mixed version compilation as the
> cause of noticed strange behavior (I named this broken incremental
> builds).

That's what we have been doing for ages, and it generally works well.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 05/05/2023 13:46, Eli Zaretskii wrote:
> Date: Fri, 5 May 2023 12:27:25 +0700 From: Max Nikulin
>> The robust way is to define compilation order through dependencies,
>> preferably autogenerated ones).

> This doesn't work in Emacs, in general, due to circular dependencies.

Could you, please, provide an example where circular dependencies are 
unavoidable or cost of disentangling of mutual dependencies is prohibitive?

My general impression that behavior of code having circular dependencies 
is harder to comprehend.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Date: Fri, 5 May 2023 14:27:29 +0700
> Cc: monnier@iro.umontreal.ca, yantar92@posteo.net, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> On 05/05/2023 13:46, Eli Zaretskii wrote:
> > Date: Fri, 5 May 2023 12:27:25 +0700 From: Max Nikulin
> >> The robust way is to define compilation order through dependencies,
> >> preferably autogenerated ones).
> 
> > This doesn't work in Emacs, in general, due to circular dependencies.
> 
> Could you, please, provide an example where circular dependencies are 
> unavoidable or cost of disentangling of mutual dependencies is prohibitive?

What do you mean by "unavoidable"?

> My general impression that behavior of code having circular dependencies 
> is harder to comprehend.

AFAIR, the problem is that we use 'require', eval-when-compile etc. to
get definitions of macros, functions, and variables.  There's nothing
difficult to comprehend in this.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 05/05/2023 17:38, Eli Zaretskii wrote:
>> Date: Fri, 5 May 2023 14:27:29 +0700 From: Max Nikulin
>>
>> On 05/05/2023 13:46, Eli Zaretskii wrote:
>>> Date: Fri, 5 May 2023 12:27:25 +0700 From: Max Nikulin
>>>> The robust way is to define compilation order through dependencies,
>>>> preferably autogenerated ones).
>>
>>> This doesn't work in Emacs, in general, due to circular dependencies.
>>
>> Could you, please, provide an example where circular dependencies are
>> unavoidable or cost of disentangling of mutual dependencies is prohibitive?
> 
> What do you mean by "unavoidable"?

Ones that do not allow to apply approaches like:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62762#140
Max Nikulin Fri, 5 May 2023 11:18:17 +0700
> In the C and C++ world the solution for cyclic dependencies is forward 
> declarations. Some kind of such approach I see in Org as well. 
> lisp/org/ol.el and lisp/org/org-element.el are mutually dependent. 
> org-element.el requires 'ol, while the latter just declares functions 
> from 'org-element.

On 05/05/2023 17:38, Eli Zaretskii wrote:
>> My general impression that behavior of code having circular dependencies
>> is harder to comprehend.
> 
> AFAIR, the problem is that we use 'require', eval-when-compile etc. to
> get definitions of macros, functions, and variables.  There's nothing
> difficult to comprehend in this.

I still have no idea which way it may be related to determining of order 
of compilation based on dependency tree. That is why I asked for 
particular examples.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Date: Fri, 5 May 2023 18:20:09 +0700
> Cc: monnier@iro.umontreal.ca, yantar92@posteo.net, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> On 05/05/2023 17:38, Eli Zaretskii wrote:
> >>> This doesn't work in Emacs, in general, due to circular dependencies.
> >>
> >> Could you, please, provide an example where circular dependencies are
> >> unavoidable or cost of disentangling of mutual dependencies is prohibitive?
> > 
> > What do you mean by "unavoidable"?
> 
> Ones that do not allow to apply approaches like:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62762#140

That's a huge message.  Which part of it specifically?

> Max Nikulin Fri, 5 May 2023 11:18:17 +0700
> > In the C and C++ world the solution for cyclic dependencies is forward 
> > declarations. Some kind of such approach I see in Org as well. 
> > lisp/org/ol.el and lisp/org/org-element.el are mutually dependent. 
> > org-element.el requires 'ol, while the latter just declares functions 
> > from 'org-element.

If it's this part, then I don't see how it can be relevant to our
case: our problem is to teach Make about these dependencies, not to
teach a Lisp compiler about them.  So forward declarations don't help
here, because Make doesn't have any such declarations, and it figures
out dependencies on its own, given only the prerequisites specified
for targets.

> On 05/05/2023 17:38, Eli Zaretskii wrote:
> >> My general impression that behavior of code having circular dependencies
> >> is harder to comprehend.
> > 
> > AFAIR, the problem is that we use 'require', eval-when-compile etc. to
> > get definitions of macros, functions, and variables.  There's nothing
> > difficult to comprehend in this.
> 
> I still have no idea which way it may be related to determining of order 
> of compilation based on dependency tree. That is why I asked for 
> particular examples.

I don't have ready examples off the top of my head, sorry.  You have
the Emacs Lisp sources, so you have all the information, and can find
those circular dependencies yourself if you don't believe me and
Stefan.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 05/05/2023 18:33, Eli Zaretskii wrote:
>> Date: Fri, 5 May 2023 18:20:09 +0700 From: Max Nikulin
>> On 05/05/2023 17:38, Eli Zaretskii wrote:
>>>>> This doesn't work in Emacs, in general, due to circular dependencies.

Eli, I am still unable to associate this with particular cases of elisp 
code or with how it is split over several files. That is why I am again 
asking for examples illustrating issues you have in mind.

>>> What do you mean by "unavoidable"?
>>
>> Max Nikulin Fri, 5 May 2023 11:18:17 +0700
>>> In the C and C++ world the solution for cyclic dependencies is forward
>>> declarations. Some kind of such approach I see in Org as well.
>>> lisp/org/ol.el and lisp/org/org-element.el are mutually dependent.
>>> org-element.el requires 'ol, while the latter just declares functions
>>> from 'org-element.
> 
> If it's this part, then I don't see how it can be relevant to our
> case: our problem is to teach Make about these dependencies, not to
> teach a Lisp compiler about them.

I can not figure out which kind of cyclic dependencies may cause 
troubles with dependency generation, but I had a hope that it would be 
possible to reorganize code to avoid such kind of dependencies.

> I don't have ready examples off the top of my head, sorry.  You have
> the Emacs Lisp sources, so you have all the information, and can find
> those circular dependencies yourself if you don't believe me and
> Stefan.

It might be even better if you will provide a brief example unrelated to 
real Emacs code. I have no idea which code I should search for.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Date: Fri, 5 May 2023 22:33:48 +0700
> Cc: monnier@iro.umontreal.ca, yantar92@posteo.net, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> On 05/05/2023 18:33, Eli Zaretskii wrote:
> >> Date: Fri, 5 May 2023 18:20:09 +0700 From: Max Nikulin
> >> On 05/05/2023 17:38, Eli Zaretskii wrote:
> >>>>> This doesn't work in Emacs, in general, due to circular dependencies.
> 
> Eli, I am still unable to associate this with particular cases of elisp 
> code or with how it is split over several files. That is why I am again 
> asking for examples illustrating issues you have in mind.
> 
> >>> What do you mean by "unavoidable"?
> >>
> >> Max Nikulin Fri, 5 May 2023 11:18:17 +0700
> >>> In the C and C++ world the solution for cyclic dependencies is forward
> >>> declarations. Some kind of such approach I see in Org as well.
> >>> lisp/org/ol.el and lisp/org/org-element.el are mutually dependent.
> >>> org-element.el requires 'ol, while the latter just declares functions
> >>> from 'org-element.
> > 
> > If it's this part, then I don't see how it can be relevant to our
> > case: our problem is to teach Make about these dependencies, not to
> > teach a Lisp compiler about them.
> 
> I can not figure out which kind of cyclic dependencies may cause 
> troubles with dependency generation, but I had a hope that it would be 
> possible to reorganize code to avoid such kind of dependencies.
> 
> > I don't have ready examples off the top of my head, sorry.  You have
> > the Emacs Lisp sources, so you have all the information, and can find
> > those circular dependencies yourself if you don't believe me and
> > Stefan.
> 
> It might be even better if you will provide a brief example unrelated to 
> real Emacs code. I have no idea which code I should search for.

You should look for all the ways we load one Lisp file from another:

  . require
  . eval-when-compile
  . eval-and-compile

etc.  When fileA.el does any of that to load fileB.el, and fileB.el
does something to load fileA.el (whether directly or indirectly, via
other files), you have a circular dependency.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 05/05/2023 22:49, Eli Zaretskii wrote:
> You should look for all the ways we load one Lisp file from another:
> 
>    . require
>    . eval-when-compile
>    . eval-and-compile

I was aware of these forms before you mentioned them first time in this 
discussion.

> etc.  When fileA.el does any of that to load fileB.el, and fileB.el
> does something to load fileA.el (whether directly or indirectly, via
> other files), you have a circular dependency.

I am sorry, but it is far from a complete example. I still have no idea 
how to combine it in a such way that there would be no problem to 
compile and load files, but attempt to create build dependencies would 
fail due to a cycle.

Perhaps editing some Emacs files you will notice the pattern you are 
writing about. I will be grateful if you provide names of files 
demonstrating cyclic dependency issue.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> Date: Fri, 5 May 2023 23:46:36 +0700
> Cc: monnier@iro.umontreal.ca, yantar92@posteo.net, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> >    . require
> >    . eval-when-compile
> >    . eval-and-compile
> 
> I was aware of these forms before you mentioned them first time in this 
> discussion.
> 
> > etc.  When fileA.el does any of that to load fileB.el, and fileB.el
> > does something to load fileA.el (whether directly or indirectly, via
> > other files), you have a circular dependency.
> 
> I am sorry, but it is far from a complete example. I still have no idea 
> how to combine it in a such way that there would be no problem to 
> compile and load files, but attempt to create build dependencies would 
> fail due to a cycle.
> 
> Perhaps editing some Emacs files you will notice the pattern you are 
> writing about. I will be grateful if you provide names of files 
> demonstrating cyclic dependency issue.

You seem to think it's easy to find these examples and provide them,
and I just stubbornly refuse to reveal them to you?

It isn't easy.  You need to actively find these references and chase
their chains, to find those cases.  I just know they exist because I
bumped into them in the past.

If you want to find them, you will have to write the code to detect
those dependencies, and then look at the graph produced from them.
There's no other way (unless Stefan happens to remember some by
heart, by some luck).

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> If you want to find them, you will have to write the code to detect
> those dependencies, and then look at the graph produced from them.
> There's no other way (unless Stefan happens to remember some by
> heart, by some luck).

I think looking for calls to `provide` that are not at the end of files
is a good first step to finding the obvious ones (if my memory serves,
viper and ediff are two examples in Emacs's own code.  The problem is
more common outside of Emacs because apparently fewer janitors roam
those realms).

Of course, the early files provide many other examples where we "solve"
the dependencies by hand to get things to bootstrap.


        Stefan

bug#62762: circular dependencies in elisp files and make

[Max Nikulin]

On 06/05/2023 01:26, Stefan Monnier wrote:
>> If you want to find them, you will have to write the code to detect
>> those dependencies, and then look at the graph produced from them.
>> There's no other way (unless Stefan happens to remember some by
>> heart, by some luck).
> I think looking for calls to `provide` that are not at the end of files
> is a good first step to finding the obvious ones (if my memory serves,
> viper and ediff are two examples in Emacs's own code.

I see no cyclic dependency in viper, at least based on `require':

viper.elc: viper-init.elc viper-keym.elc viper-cmd.elc
viper-cmd.elc: viper-util.elc viper-keym.elc viper-mous.elc 
viper-macs.elc viper-ex.elc
viper-macs.elc: viper-mous.elc viper-ex.elc viper-util.elc viper-keym.elc
viper-ex.elc: viper-keym.elc viper-util.elc
viper-keym.elc: viper-util.elc
viper-mous.elc: viper-util.elc
viper-util.elc: viper-init.elc

I expect that it should ensure ordering for make allowing parallel 
execution to some degree.

Frankly speaking, I have not realized why `provide' not at the end is 
important. Do you mean the following

lisp/vc/ediff-util.el:(provide 'ediff-util)    ;FIXME: Break cyclic 
dependencies and move to the end!

and mutual `require' 'ediff-util from ediff-init.el and 'ediff-init from 
ediff-util.el? My idea was to suppress loading of .elc files during 
compilation of such files.

bug#62762: circular dependencies in elisp files and make

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> important. Do you mean the following
> lisp/vc/ediff-util.el:(provide 'ediff-util)    ;FIXME: Break cyclic
> dependencies and move to the end!

Sounds about right, yes.

> and mutual `require' 'ediff-util from ediff-init.el and 'ediff-init from
> ediff-util.el? My idea was to suppress loading of .elc files during
> compilation of such files.

When I looked at it, my approach was to break the cycle in an arbitrary
way and say that if it occasionally leads to a miscompilation it's the
package's fault.


        Stefan

bug#62762: circular dependencies in elisp files and make

[Eli Zaretskii]

> Date: Thu, 11 May 2023 22:14:03 +0700
> Cc: yantar92@posteo.net, bzg@gnu.org, dmitry@gutov.dev,
>  62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> My idea was to suppress loading of .elc files during compilation of
> such files.

That idea will make a typical build much slower, for the rare case
where such measures are really needed.  That's the wrong balance in my
book.

bug#62762: circular dependencies in elisp files and make

[Max Nikulin]

On 11/05/2023 22:59, Eli Zaretskii wrote:
>> Date: Thu, 11 May 2023 22:14:03 +0700 From: Max Nikulin
>>
>> My idea was to suppress loading of .elc files during compilation of
>> such files.
> 
> That idea will make a typical build much slower, for the rare case
> where such measures are really needed.  That's the wrong balance in my
> book.

I have noticed that you are afraid that correct rebuild might be too 
slow, but I do not see a reason why it should be noticeable.

1. There is no need to suppress .elc during initial clean build that 
takes enough time. It is for incremental build when usually only a 
fraction of all lisp files are recompiled
2. At first, files necessary to load bytecomp are rebuilt, so during 
compiling majority of files, suffix list is modified immediately before 
`byte-compile-file' call. That is why only files required by source file 
of the current target are affected.
3. I hope, it is possible to suppress loading of .elc only in the case 
of circular dependencies.

bug#62762: circular dependencies in elisp files and make

[Eli Zaretskii]

> Date: Fri, 12 May 2023 21:59:56 +0700
> Cc: monnier@iro.umontreal.ca, yantar92@posteo.net, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> On 11/05/2023 22:59, Eli Zaretskii wrote:
> >> Date: Thu, 11 May 2023 22:14:03 +0700 From: Max Nikulin
> >>
> >> My idea was to suppress loading of .elc files during compilation of
> >> such files.
> > 
> > That idea will make a typical build much slower, for the rare case
> > where such measures are really needed.  That's the wrong balance in my
> > book.
> 
> I have noticed that you are afraid that correct rebuild might be too 
> slow, but I do not see a reason why it should be noticeable.
> 
> 1. There is no need to suppress .elc during initial clean build that 
> takes enough time. It is for incremental build when usually only a 
> fraction of all lisp files are recompiled

I think it will be much more than "a fraction", due to dependencies.
But I will withhold my judgment until I see the actual code and what
it does to the build.

> 2. At first, files necessary to load bytecomp are rebuilt

Why only bytecomp?  Byte compiling needs more than just bytecomp,
AFAIU.

bug#62762: circular dependencies in elisp files and make

[Max Nikulin]

On 12/05/2023 22:10, Eli Zaretskii wrote:
>> Date: Fri, 12 May 2023 21:59:56 +0700 From: Max Nikulin
>>
>> 2. At first, files necessary to load bytecomp are rebuilt
> 
> Why only bytecomp?  Byte compiling needs more than just bytecomp,
> AFAIU.

Because the ".el.elc:" rule uses "-f batch-byte-compile". I expect that

emacs -Q --batch -l bytecomp --eval \
   "(mapc (lambda (x) (princ (car x) (terpri))) load-history)"

gives reasonable approximation to "files necessary to load bytecomp". 
What should be added to prerequisites besides bytecomp dependencies?

bug#62762: circular dependencies in elisp files and make

[Eli Zaretskii]

> Date: Fri, 12 May 2023 22:26:47 +0700
> Cc: monnier@iro.umontreal.ca, yantar92@posteo.net, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> On 12/05/2023 22:10, Eli Zaretskii wrote:
> >>
> > Why only bytecomp?  Byte compiling needs more than just bytecomp,
> > AFAIU.
> 
> Because the ".el.elc:" rule uses "-f batch-byte-compile". I expect that
> 
> emacs -Q --batch -l bytecomp --eval \
>    "(mapc (lambda (x) (princ (car x) (terpri))) load-history)"
> 
> gives reasonable approximation to "files necessary to load bytecomp". 
> What should be added to prerequisites besides bytecomp dependencies?

Anything that bytecomp loads via autoloads while compiling.  Also
byte-opt and its autoloads, I think.

bug#62762: circular dependencies in elisp files and make

[Max Nikulin]

On 12/05/2023 23:01, Eli Zaretskii wrote:
>> Date: Fri, 12 May 2023 22:26:47 +0700 From: Max Nikulin
>>
>> I expect that
>>
>> emacs -Q --batch -l bytecomp --eval \
>>     "(mapc (lambda (x) (princ (car x) (terpri))) load-history)"
>>
>> gives reasonable approximation to "files necessary to load bytecomp".
>> What should be added to prerequisites besides bytecomp dependencies?
> 
> Anything that bytecomp loads via autoloads while compiling.  Also
> byte-opt and its autoloads, I think.

You are right concerning autoloads, while byte-opt is just one extra 
file in the dependency list. However the same expression evaluated when 
`batch-byte-compile' call finished gives list of dependencies for the 
just created .elc files. Basic statistics based on dependency files may 
provide better approximation which files should be compiled at first to 
achieve faster build. I do not think that complications you described 
are blockers for accurate dependency tracking.

My initial idea was to compile files necessary for byte compilation at 
first and for compilation of "regular" files load components required 
for byte compilation and after that suppress loading of .elc files. What 
can be better is removing all stale .elc file before compiling of 
updates sources. A script iterates over dependency files and deletes 
.elc if any its prerequisite has been updated. Following compilation 
pass can safely use .elc files that are either survived from previous 
build as up to date or just refreshed.

bug#62762: circular dependencies in elisp files and make

[Eli Zaretskii]

> Date: Sat, 13 May 2023 10:08:25 +0700
> Cc: monnier@iro.umontreal.ca, yantar92@posteo.net, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> My initial idea was to compile files necessary for byte compilation at 
> first and for compilation of "regular" files load components required 
> for byte compilation and after that suppress loading of .elc files. What 
> can be better is removing all stale .elc file before compiling of 
> updates sources. A script iterates over dependency files and deletes 
> .elc if any its prerequisite has been updated. Following compilation 
> pass can safely use .elc files that are either survived from previous 
> build as up to date or just refreshed.

So you will make each build always do two byte-compilation passes
where today we have just one?  And the first one of these will use
only *.el files, including for those that are preloaded into
bootstrap-emacs?  Doesn't that sound like a significant slow-down of
the build?

Also, what about native-compilation, with or without AOT?

bug#62762: circular dependencies in elisp files and make

[Max Nikulin]

On 13/05/2023 13:50, Eli Zaretskii wrote:
>> Date: Sat, 13 May 2023 10:08:25 +0700 From: Max Nikulin
>>
>> ... What
>> can be better is removing all stale .elc file before compiling of
>> updates sources. A script iterates over dependency files and deletes
>> .elc if any its prerequisite has been updated. Following compilation
>> pass can safely use .elc files that are either survived from previous
>> build as up to date or just refreshed.
> 
> So you will make each build always do two byte-compilation passes
> where today we have just one?  And the first one of these will use
> only *.el files, including for those that are preloaded into
> bootstrap-emacs?  Doesn't that sound like a significant slow-down of
> the build?

I do not see a reason for significant slow-down because I do not suggest 
two passes of *byte-compilataion*.

1. A script reads dependency files (if they exist) created during 
previous build and removes stale .elc files.
2. Normal "make" pass that takes into account dependency between files 
for ordering of compile commands. Dependency files are created or 
updated as a side-effect of compilation.

Likely it is reasonable to split stage 2 into steps similar to current 
targets like main-first and mark most of files as dependent on a target 
that (throw its dependencies) compiles files required for byte compilation.

bug#62762: circular dependencies in elisp files and make

[Eli Zaretskii]

> Date: Sat, 13 May 2023 14:34:06 +0700
> Cc: monnier@iro.umontreal.ca, yantar92@posteo.net, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> 1. A script reads dependency files (if they exist) created during 
> previous build and removes stale .elc files.

This will break if the updated files have different dependencies.  You
need to recreate the dependencies each build.

Also, it is not clear what is the plan for the macros.  If one of the
macros used during byte compilation changes in incompatible ways,
trying to byte-compile using Emacs which preloaded the previous
(outdated) definition of the macro will fail.  So some changes need
also to generate a new Emacs binary, not just byte-compile in the
right order.

> 2. Normal "make" pass that takes into account dependency between files 
> for ordering of compile commands. Dependency files are created or 
> updated as a side-effect of compilation.
> 
> Likely it is reasonable to split stage 2 into steps similar to current 
> targets like main-first and mark most of files as dependent on a target 
> that (throw its dependencies) compiles files required for byte compilation.

What is the plan for the various autoloads files?  They depend on all
the files, and many (all?) files depend on them.

bug#62762: circular dependencies in elisp files and make

[Max Nikulin]

On 13/05/2023 15:46, Eli Zaretskii wrote:
>> Date: Sat, 13 May 2023 14:34:06 +0700 From: Max Nikulin
>>
>> 1. A script reads dependency files (if they exist) created during
>> previous build and removes stale .elc files.
> 
> This will break if the updated files have different dependencies.  You
> need to recreate the dependencies each build.

Let's consider a.el that contains (require 'd) or it autoloads some 
macro from d.el. If a.el changed then a.elc is removed on this stage, so 
updated a.elc will be created on the next stage. Dependency on d.elc is 
known from previous build, so if d.el is changed then both a.elc and 
d.elc are removed. If new (require 'd-new) is added to a.el then it is 
not a problem as well. a.elc is removed due to changed a.el. d-new.elc 
is either up to date or it is removed due to changes in its dependencies 
or in d-new.el.

So no stale files should survive stage 1. The only issue with obsolete 
dependencies is that compilation order may be not optimal.

> Also, it is not clear what is the plan for the macros.  If one of the
> macros used during byte compilation changes in incompatible ways,
> trying to byte-compile using Emacs which preloaded the previous
> (outdated) definition of the macro will fail.

I hope, it is addressed above.

>  So some changes need
> also to generate a new Emacs binary, not just byte-compile in the
> right order.

I expect that dependencies of the script removing stale files are 
minimized. If changes in the breaks this script then it is time for 
"make clean". If emacs binary depends on .elc files then some file may 
be touched to let "make" know that the executable needs rebuild.

>> 2. Normal "make" pass that takes into account dependency between files
>> for ordering of compile commands. Dependency files are created or
>> updated as a side-effect of compilation.
>>
>> Likely it is reasonable to split stage 2 into steps similar to current
>> targets like main-first and mark most of files as dependent on a target
>> that (throw its dependencies) compiles files required for byte compilation.
> 
> What is the plan for the various autoloads files?  They depend on all
> the files, and many (all?) files depend on them.

My plan is that autoloads/loaddefs are order-only dependencies for .elc 
files. However autoloads/loaddefs files depends on all files necessary 
to generate them.

bug#62762: circular dependencies in elisp files and make

[Eli Zaretskii]

> Date: Sat, 13 May 2023 17:25:06 +0700
> Cc: monnier@iro.umontreal.ca, yantar92@posteo.net, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> On 13/05/2023 15:46, Eli Zaretskii wrote:
> >> Date: Sat, 13 May 2023 14:34:06 +0700 From: Max Nikulin
> >>
> >> 1. A script reads dependency files (if they exist) created during
> >> previous build and removes stale .elc files.
> > 
> > This will break if the updated files have different dependencies.  You
> > need to recreate the dependencies each build.
> 
> Let's consider a.el that contains (require 'd) or it autoloads some 
> macro from d.el. If a.el changed then a.elc is removed on this stage, so 
> updated a.elc will be created on the next stage. Dependency on d.elc is 
> known from previous build, so if d.el is changed then both a.elc and 
> d.elc are removed. If new (require 'd-new) is added to a.el then it is 
> not a problem as well. a.elc is removed due to changed a.el. d-new.elc 
> is either up to date or it is removed due to changes in its dependencies 
> or in d-new.el.

I was talking about dependencies, not about stale files.  They will
need to be refreshed every build, because, unlike C sources, the *.el
files change in that aspect much more frequently.

> So no stale files should survive stage 1. The only issue with obsolete 
> dependencies is that compilation order may be not optimal.

Stale files are already not an issue, because the build runs with
load-prefer-newer set non-nil.

> > Also, it is not clear what is the plan for the macros.  If one of the
> > macros used during byte compilation changes in incompatible ways,
> > trying to byte-compile using Emacs which preloaded the previous
> > (outdated) definition of the macro will fail.
> 
> I hope, it is addressed above.

It isn't.

> >  So some changes need
> > also to generate a new Emacs binary, not just byte-compile in the
> > right order.
> 
> I expect that dependencies of the script removing stale files are 
> minimized. If changes in the breaks this script then it is time for 
> "make clean". If emacs binary depends on .elc files then some file may 
> be touched to let "make" know that the executable needs rebuild.

I don't understand how this will work, but don't let me discourage you
from trying to make it work and showing how.

> > What is the plan for the various autoloads files?  They depend on all
> > the files, and many (all?) files depend on them.
> 
> My plan is that autoloads/loaddefs are order-only dependencies for .elc 
> files. However autoloads/loaddefs files depends on all files necessary 
> to generate them.

Same here.

Thanks for working on this, anyway.  If it can work, it could be an
important change.

bug#62762: circular dependencies in elisp files and make

[Max Nikulin]

On 13/05/2023 17:58, Eli Zaretskii wrote:
>> Date: Sat, 13 May 2023 17:25:06 +0700 From: Max Nikulin
>>
>> On 13/05/2023 15:46, Eli Zaretskii wrote:
>>>> Date: Sat, 13 May 2023 14:34:06 +0700 From: Max Nikulin
>>>>
>>>> 1. A script reads dependency files (if they exist) created during
----------^^^^^^^^^^^^^^^^^^^^^^^

>>>> previous build and removes stale .elc files.
>>>
>>> This will break if the updated files have different dependencies.  You
>>> need to recreate the dependencies each build.
>>
>> Let's consider a.el that contains (require 'd) or it autoloads some
>> macro from d.el. If a.el changed then a.elc is removed on this stage, so
>> updated a.elc will be created on the next stage. Dependency on d.elc is
>> known from previous build, so if d.el is changed then both a.elc and
>> d.elc are removed. If new (require 'd-new) is added to a.el then it is
>> not a problem as well. a.elc is removed due to changed a.el. d-new.elc
>> is either up to date or it is removed due to changes in its dependencies
>> or in d-new.el.
> 
> I was talking about dependencies, not about stale files.  They will
> need to be refreshed every build, because, unlike C sources, the *.el
> files change in that aspect much more frequently.

Writing "stale" I mean more than comparison timestamps of a.el and 
a.elc. Dependency files may contain

a.elc: a.el d.elc
d.elc: d.el

the script walks through dependency graph, compares timestamps and when 
it d.el is newer than d.elc, *both* "a.elc" and "d.elc" are removed. It 
is unfortunate that a part of "make" should be implemented in the 
script, but I am unsure if some trick is suitable to leverage "make" 
power for removing of files instead of their creation.

Next compile stage updates dependency files as a side effect and new 
dependencies may appear

d.elc: d.el d-transitive.elc

however absence of d-transitive.elc during dependency removing stage 1 
does not lead to any problem.

bug#62762: circular dependencies in elisp files and make

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Writing "stale" I mean more than comparison timestamps of a.el and
> a.elc. Dependency files may contain
>
> a.elc: a.el d.elc
> d.elc: d.el
>
> the script walks through dependency graph, compares timestamps and when it
> d.el is newer than d.elc, *both* "a.elc" and "d.elc" are removed.

Make will already take care of this dependency and recompile `d.elc`
before `a.elc`.


        Stefan

bug#62762: circular dependencies in elisp files and make

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> 1. A script reads dependency files (if they exist) created during previous
>   build and removes stale .elc files.

`load-prefer-newer` should give us the same result already.


        Stefan

bug#62762: circular dependencies in elisp files and make

[Max Nikulin]

On 13/05/2023 21:48, Stefan Monnier wrote:
>> 1. A script reads dependency files (if they exist) created during previous
>>    build and removes stale .elc files.
> 
> `load-prefer-newer` should give us the same result already.

No, because some .elc files may become stale not because of modification 
of the corresponding files, but due to update of a macro inside a 
`require'd file. Removing of outdated .elc files *before* 
byte-compilation pass ensures that no obsolete macro definitions are used.

On 13/05/2023 21:49, Stefan Monnier wrote:
>> Writing "stale" I mean more than comparison timestamps of a.el and
>> a.elc. Dependency files may contain
>>
>> a.elc: a.el d.elc
>> d.elc: d.el
>>
>> the script walks through dependency graph, compares timestamps and when it
>> d.el is newer than d.elc, *both* "a.elc" and "d.elc" are removed.
> 
> Make will already take care of this dependency and recompile `d.elc`
> before `a.elc`.

You are right when dependency graph is a tree. You provided an example 
of circular dependency in lisp/vc/ediff-util.el. It is the case when to 
correctly recompile ediff it is necessary to remove ediff*.elc files 
before compilation (I do not like your approach with breaking cycle at 
an arbitrary edge since outdated macro definition still may be used). 
Files necessary for byte compilation is another example of a dependency 
loop. I assume there are may be more tricky cases I am unaware about, so 
prefer to be at the safe side.

Eli raised a question of rebuilding of an .elc file when a new 
dependency with outdated .elc file is introduced. Since it is almost 
impossible to get dependency list without byte compilation, there are 2 
options: ask make to rebuild .elc files files once more (updated 
dependency graph is available during second byte compilation pass) or 
just remove all stale .elc file before starting byte compilation. I 
prefer the latter approach.

Due to these reasons I prefer 2 stage incremental build.

On 13/05/2023 22:43, Stefan Monnier wrote:
> Not sure about two passes, but if you want it to be "right" (i.e. aim
> to never need `make bootstrap`), then every `.elc` file will transitively
> depend on every preloaded `.el` file, including `loaddefs.el` which is
> itself auto-generated from all the `.el` files.  IOW the only truly
> "right" way is to always recompile all the `.elc` files whenever  `.el`
> (or `.c` or `.h`) file is modified 🙁

I still hope, it is possible to add loaddefs files as order-only 
dependencies. I see just one case when it may lead to incomplete 
rebuild: a.el uses a function from al.el without (require 'al) and 
autoload cookie is missed in al.el. After compilation of a.el dependency 
on al.elc is unknown thus updating al.el is not a reason to recompile 
a.elc. I hope, raising priority of "Warning: the following functions are 
not known to be defined:" to an error preventing creation of .elc file 
will solve the issue.

Despite at first glance loaddefs ruin the idea with partial incremental 
rebuild and dependency tracking, it seems a workaround exists.

bug#62762: circular dependencies in elisp files and make

[Eli Zaretskii]

> Date: Mon, 15 May 2023 17:11:25 +0700
> Cc: Eli Zaretskii <eliz@gnu.org>, yantar92@posteo.net, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> I see just one case when it may lead to incomplete 
> rebuild: a.el uses a function from al.el without (require 'al) and 
> autoload cookie is missed in al.el.

This can happen when we use 'declare-function'.

bug#62762: circular dependencies in elisp files and make

[Max Nikulin]

On 15/05/2023 18:20, Eli Zaretskii wrote:
>> Date: Mon, 15 May 2023 17:11:25 +0700 From: Max Nikulin
>>
>> I see just one case when it may lead to incomplete
>> rebuild: a.el uses a function from al.el without (require 'al) and
>> autoload cookie is missed in al.el.
> 
> This can happen when we use 'declare-function'.

Sorry, my bad. When a.el uses a *macro* from al.el then, instead of 
macro expansion result, function call appears in a.elc.

Anyway by this sentence I was trying to convince that it is enough to 
have loaddefs as order-only dependency. What would be more interesting 
is an example when an .elc file *must* be recompiled in response to 
update of a loaddefs file. I mean a case of hidden dependency on a file 
that is the source for autoload declaration.

bug#62762: circular dependencies in elisp files and make

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> You are right when dependency graph is a tree.  You provided an
> example of circular dependency in lisp/vc/ediff-util.el.
[...]
> Due to these reasons I prefer 2 stage incremental build.

My intuition tells me it's probably not going to result in a good
trade-off, which is why I haven't worked on it so far.

But you don't need to try and convince me (or anyone else).
Instead, just go and implement it.

If you prove my intuition wrong, then we'll all be happy.


        Stefan

bug#62762: circular dependencies in elisp files and make

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I do not see a reason for significant slow-down because I do not suggest two
> passes of *byte-compilataion*.

Not sure about two passes, but if you want it to be "right" (i.e. aim
to never need `make bootstrap`), then every `.elc` file will transitively
depend on every preloaded `.el` file, including `loaddefs.el` which is
itself auto-generated from all the `.el` files.  IOW the only truly
"right" way is to always recompile all the `.elc` files whenever  `.el`
(or `.c` or `.h`) file is modified :-(

So, in order to avoid this, we need to make concessions, such as
ignoring (transitive) dependencies on preloaded files, and those files
used by the byte compiler.


        Stefan

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Max,

> `load-prefer-newer' is a kludge as well.

I have to point out at this stage that I feel your answers aren't
helping address the very concrete short term question at hand.
Instead you're pointing to well known problems left and right that only
bite in very specific corner cases and with no really clear solution.

The problems `org-assert-version` is trying to fix happen a lot more
often than those corner cases and are much easier to handle.

> E.g. Python people migrated from comparison of timestamps to
> inscribing of .py file hash into byte compiled .pyc files.

Good for them.  But that's unrelated to he problem at hand (and would
only make a difference in cases that are a lot more hypothetical than
the ones we're considering here).

E.g. the mixed-version problems that `org-assert-version` tries to
detect can happen without any `.elc` file in sight at all.

Could we focus on the problem at hand, please?

> By the way, `load-prefer-newer' make things even worse [...] (.el
> files not changed, so they are older)

No, if ".el files not changed, so they are older", then
`load-prefer-newer` has no effect at all, so it can't make things worse.

>>> `org-assert-version' is just the most apparent manifestation.
>> AFAIK `org-assert-version` tries to solve a different problem.
> Of course `org-assert-version' was introduced to solve another
> problem, but it highlighted issues with dependency handling during
> incremental builds.

Yes, ELisp has lot of other problems.  But can we move those discussions
to other bug reports, otherwise I'm afraid we'll never get anywhere.

>> Indeed, we recently introduced `org--inhibit-version-check`
>> specifically so as not to use `org-assert-version` for Emacs's broken
>> incremental builds (i.e. we decided we preferred that brokenness
>> there).
> `org--inhibit-version-check' is even uglier kludge that partially
> deactivated `org-assert-version'

Maybe so, but its addition is still the embodiment of people's judgment
that for `git pull; make` in Emacs's sources, `org-assert-version` is
worse than the problems it tries to address.

> and just hides the issue with dependencies for incremental builds.

No, it doesn't.

>>> Perhaps a similar trick may be done in elisp by advicing e.g. `load'.
> Another idea: while single file is compiled per emacs invocation
> `load-history' may be compared before and after byte compilation.

It's not really "another idea" (unless you meant `advicing` in a very
narrow sense, but since we don't like Emacs to use advice within itself,
it would be a non-starter).
It's just one of the many ways to implement that same idea.

>> I can already warn them that an important problem will be the
>> presence of cyclic dependencies.
> In the C and C++ world the solution for cyclic dependencies is forward
> declarations. Some kind of such approach I see in Org as
> well. lisp/org/ol.el and lisp/org/org-element.el are mutually
> dependent. org-element.el requires 'ol, while the latter just declares
> functions from 'org-element.
>
> When dependency files are generated, it is possible to create a report on
> cyclic dependency and to disentangle them.

As part of my janitorial work, I regularly have to try and disentangle
circular dependencies.  In my experience the hardest part is to convince
the upstream maintainers to accept the churn, even though it brings no
material improvement (and forces them to get used to the new code
organization).

So, yes, technical changes could help, but it would only go so far.

[ I'm stopping here, I have to go.  ]


        Stefan

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 05/05/2023 21:29, Stefan Monnier wrote:
>> `load-prefer-newer' is a kludge as well.
> 
> I have to point out at this stage that I feel your answers aren't
> helping address the very concrete short term question at hand.

It was not me who bring `load-prefer-newer' to this discussion. My point 
is: if timestamps were helpful to solve the issue with using of stale 
.elc files during incremental built after update then check based on .el 
content hashsum would do it even better.

> E.g. the mixed-version problems that `org-assert-version` tries to
> detect can happen without any `.elc` file in sight at all.

Current variant `org-assert-version' can not work when Org update is 
loaded uncompiled, Certainly mixed version loading may happen purely 
with .el files. Mixed compilation issue (the topic of this bug) happens 
namely with .elc files.

>> By the way, `load-prefer-newer' make things even worse [...] (.el
>> files not changed, so they are older)
> 
> No, if ".el files not changed, so they are older", then
> `load-prefer-newer` has no effect at all, so it can't make things worse.

If *.el files had higher priority than newer *.elc files than compiling 
result would be correct. An .elc file may become stale not due to change 
of its .el source, but due to changes in required files. The price is 
slower incremental builds. Initial clean build may use .elc files. This 
is applicable to Emacs builds, things may be more complicated when users 
build Org updates.

> Yes, ELisp has lot of other problems.  But can we move those discussions
> to other bug reports, otherwise I'm afraid we'll never get anywhere.

I see 2 possible routes to avoid make + org-assert-version issues
- to make incremental builds really reliable, rather than just "it 
generally works well"
- improve `org-assert-version'

Unfortunately committed code made `org-assert-version' less efficient. 
The function you suggested may solve some issues, but I am unsure if it 
can replace current variant of `org-assert-version' completely.

I have no ideas how to make `org-assert-version' better.

I had a hope that dependency generation may solve compiling issue, but I 
can not figure out what sort of circular dependencies is troublesome. I 
just must believe you and Eli.

> [ I'm stopping here, I have to go.  ]

Certainly there is no hurry. For me priority is the following:

Example of circular dependencies that may cause trouble during 
determining of compiling order (names of files or complete example, I 
hope it should not be longer that a dozen of lines).

Whether I missed something and your function may handle loading files 
from updated directory.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I have no ideas how to make `org-assert-version' better.

`my-require-with-shadow-check' is a suggestion to make it better.
And this sub-thread of the bug report was presumably your answer to the
question by Ihor:

    Max, do you see any obvious downsides in Stefan's idea about consulting
    `load-history' vs. `load-path'?

So are you saying that you don't think it's better?
If so, do you also think it's worse, and if so, in which respect?

> I had a hope that dependency generation may solve compiling issue,

It won't solve the mixed-version problem.

Incremental recompilation (i.e. `git pull; make`) is a qualitatively
different problem from the mixed-version problem, and by now, I'm pretty
much convinced that there just isn't a solution that fixes both: they
may look somewhat similar from a distance, but they're fundamentally
quite different.

Currently `org-assert-version` has no effect for `git pull; make` in
Emacs sources, so it's presumably focused on the mixed-version problem.
And I think that for this problem `my-require-with-shadow-check' is
a better check (e.g. it will also work when none of the files are
compiled).

> but I can not figure out what sort of circular dependencies is
> troublesome.  I just must believe you and Eli.

I'll refrain from discussing this here.  Incremental recompilation
problems should be discussed in another bug report, IMO.


        Stefan

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

Stefan, my apologizes that I sent first copy exclusively to you.

On 06/05/2023 01:17, Stefan Monnier wrote:
>> I have no ideas how to make `org-assert-version' better.
> 
> `my-require-with-shadow-check' is a suggestion to make it better.

My current opinion is that `my-require-with-shadow-check' may be an 
improvement when *combined* with original `org-assert-version' (that 
inscribes org-version into .elc files).

The reason why `my-require-with-shadow-check' may be significantly less 
efficient if it will *replace* original implementation of 
`org-assert-version' is the following.

Consider a user who cloned org-mode git repository and loads Org mode 
from there with periodic pull and rebuild actions. Forget about built-in 
Org and assume that `load-path' is modified early enough to not cause 
any problem.

Now literally citing my Fri, 5 May 2023 11:18:17 +0700 message
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62762#140

Does in help in the following case?
1. Base Org part is loaded on opening of some .org file.
2. Org in that directory is updated and recompiled.
3. New Org feature is loaded (autoloaded or by explicit call of e.g.
    (require ob-shell))

Additional argument:

When the reporter got the error from `org-assert-version' it was not 
necessary a false positive (however most probable on minor version 
update), user may face an error later while running Org because not all 
files were recompiled after changes in macros.

I am in doubts if `my-require-with-shadow-check' may help in such cases.

> And this sub-thread of the bug report was presumably your answer to the
> question by Ihor:
> 
>      Max, do you see any obvious downsides in Stefan's idea about consulting
>      `load-history' vs. `load-path'?

My excuses:
- message subject has not changed, so it was unclear that it was decided 
to discuss purely mitigation of symptoms instead of fixing of real cause
- participants of discussion are spread across time zones, so a longer 
messages may be preferred
- Configuration of https://debbugs.gnu.org is worse than 
https://bugs.debian.org, reply to specific messages links are missed.
- debbugs and thunderbird are not friends in respect to mbox links (and 
I have not created a wrapper script that should work as a workaround)

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

AFAIU there are 2 main situations:

A) recompilation (i.e. `git pull; make`).
   This can happen either in Org's repository or in Emacs's repository.
   For Emacs repository, Emacs maintainers prefer recompilation to be
   faster even if it implies occasional breakage (clearly, the details
   matter to know which of the two evil we prefer, but we're not
   interested in "fixing of real cause" unless it's cheap enough).
B) mixed versions from different directories.

AFAICT `my-require-with-shadow-check` should be able to catch most/all
problems for (B), without impacting (A), and more reliably than
`org-assert-version` (and in an arguably less ad-hoc way).

So, to me, (assuming the above is true) the remaining question is what
Org developers want to do about `git pull; make` in Org's own repository.


        Stefan

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 06/05/2023 20:02, Stefan Monnier wrote:
> AFAIU there are 2 main situations:
> 
> A) recompilation (i.e. `git pull; make`).
>     This can happen either in Org's repository or in Emacs's repository.
...
Are there brave enough developers and users who do not restart emacs 
after rebuild of emacs? Some Org users hope that `org-reload' works and 
they can update Org without restarting Emacs.

> B) mixed versions from different directories.
> 
> AFAICT `my-require-with-shadow-check` should be able to catch most/all
> problems for (B),

Agree.

> without impacting (A), and more reliably than
> `org-assert-version` (and in an arguably less ad-hoc way).

Sorry, but `org-assert-version' should catch loading from the same 
directory a file compiled for new version since loaded earlier 
org-version.el defined old version that does not match `org-version' 
inscribed into new .elc files. I assume either combination of "make 
clean" and "make" or proper incremental build. That is why I disagree 
with "without impacting (A)".

> So, to me, (assuming the above is true) the remaining question is what
> Org developers want to do about `git pull; make` in Org's own repository.

Perhaps `org-assert-directory' may be added to `org-assert-version'

By the way, if just org-macs should be checked then would not be it more 
efficient in respect to load time to define a variable containing the 
directory and to check in other files if they are loaded from the same 
directory? With such approach there is no need in walking through the 
`load-history' list.

==> tst-macs.el <==

(defun tst-get-load-dir ()
   (file-name-directory
    (or load-file-name
        ;; For `eval-buffer'.
        (buffer-file-name (buffer-base-buffer)))))

(defvar tst-load-dir (tst-get-load-dir))

(defun tst-assert-load-dir ()
   (let ((dir (tst-get-load-dir))
	(url "https://orgmode.org/worg/org-faq.html#mixed-install"))
     (unless (equal tst-load-dir dir)
       (error "Path shadowing detected, see <%s>. %s vs %s"
	     url dir tst-load-dir))))

(provide 'tst-macs)


==> tst-main.el <==

(require 'tst-macs)

(eval-and-compile
   (if (fboundp 'tst-assert-load-dir)
     (tst-assert-load-dir)
    (error "Path shadowing detected, see 
<https://orgmode.org/worg/org-faq.html#mixed-install>")))

(provide 'tst-main)

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> AFAIU there are 2 main situations:
>> A) recompilation (i.e. `git pull; make`).
>>     This can happen either in Org's repository or in Emacs's repository.
> ...
> Are there brave enough developers and users who do not restart emacs after
>  rebuild of Emacs?

I recompile daily, and restart every other month or so?
FWIW, I never bumped into a problem with Org due to that (and never
used `org-reload` either).

>> B) mixed versions from different directories.
>> AFAICT `my-require-with-shadow-check` should be able to catch most/all
>> problems for (B),
>
> Agree.
>
>> without impacting (A), and more reliably than
>> `org-assert-version` (and in an arguably less ad-hoc way).
>
> Sorry, but `org-assert-version' should catch loading from the same directory
> a file compiled for new version since loaded earlier org-version.el defined
> old version that does not match `org-version' inscribed into new .elc
> files.  I assume either combination of "make clean" and "make" or proper
> incremental build.

I'd consider these scenarios to fall into (A) or thereabout.  And users
who do like I do (i.e. recompile Org while a session is using those Org
files) get what they deserve if they encounter a problem.

We have so many more real problems that I'm not interested in
considering such cases.  I can give you a life-long supply of further
hypothetical cases if you're so inclined :-)

> By the way, if just org-macs should be checked then would not be it more
> efficient in respect to load time to define a variable containing the
> directory and to check in other files if they are loaded from the same
> directory? With such approach there is no need in walking through the
> `load-history' list.

You might be right.  I was thinking in more general terms (e.g. I'm
thinking that maybe a test along the lines of the one performed by
`my-require-with-shadow-check` could be (optionally) included in
`require`).


        Stefan

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> So, to me, (assuming the above is true) the remaining question is what
>> Org developers want to do about `git pull; make` in Org's own repository.
>
> Perhaps `org-assert-directory' may be added to `org-assert-version'
>
>
> (defun tst-assert-load-dir ()
>    (let ((dir (tst-get-load-dir))
> 	(url "https://orgmode.org/worg/org-faq.html#mixed-install"))
>      (unless (equal tst-load-dir dir)
>        (error "Path shadowing detected, see <%s>. %s vs %s"
> 	     url dir tst-load-dir))))

Sounds reasonable.
I think we can try to do the following:

1. Use the load-dir test, warn, and try to re-load
2. Use org-assert-version, but downgrade error to simple warning

WDYT?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 08/05/2023 17:05, Ihor Radchenko wrote:
> Max Nikulin writes:
>> (defun tst-assert-load-dir ()
>>     (let ((dir (tst-get-load-dir))
>> 	(url "https://orgmode.org/worg/org-faq.html#mixed-install"))
>>       (unless (equal tst-load-dir dir)
>>         (error "Path shadowing detected, see <%s>. %s vs %s"
>> 	     url dir tst-load-dir))))
> 
> I think we can try to do the following:
> 
> 1. Use the load-dir test, warn, and try to re-load
> 2. Use org-assert-version, but downgrade error to simple warning

Ihor, final decision is up to you.

I think, `org-reload' may be recommended in a warning message. I am 
unsure if `org-reload' called from the assertion while an Org component 
is loaded may not cause a more severe issue. My special concern is 
reloading when new Org is being compiled: is it safe or reloading should 
be suppressed completely?

I believe, during creation of .elc files, assert macro should signal an 
error since a warning is not enough to prevent loading mixed compilation 
result to next Emacs session. I do not like that currently during 
compiling with old Org loaded, `org-assert-version' causes just a 
warning. Users have difficulties with clean recompiling.

While loading, non-fatal error may be a better option when versions do 
not differ too much.

I like that you are not going to throw version check away completely.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 2398 bytes --]

Max Nikulin <manikulin@gmail.com> writes:

>> I think we can try to do the following:
>> 
>> 1. Use the load-dir test, warn, and try to re-load
>> 2. Use org-assert-version, but downgrade error to simple warning
>
> Ihor, final decision is up to you.

> I think, `org-reload' may be recommended in a warning message. I am 
> unsure if `org-reload' called from the assertion while an Org component 
> is loaded may not cause a more severe issue. My special concern is 
> reloading when new Org is being compiled: is it safe or reloading should 
> be suppressed completely?

I am thinking about the following:
1. Use `org-reload' instead of load-dir test. This will re-load all the
   currently loaded Org libraries, forcing them to be consistent. This
   is necessary because Org does not only rely upon explicit (require
   ...) statements - we have way too many `declare-function' expecting
   other libraries to be loaded by other means.
2. Run `org-reload' _after_ loading current file.
   Because (equal (org-release) ,(org-release)) will only ever trigger
   during compilation when we (require 'already-compiled-library),
   running `org-reload' after loading 'already-compiled-library will
   also ensure consistent Org state after (require
   'already-compiled-library) is finished.

> I believe, during creation of .elc files, assert macro should signal an 
> error since a warning is not enough to prevent loading mixed compilation 
> result to next Emacs session. I do not like that currently during 
> compiling with old Org loaded, `org-assert-version' causes just a 
> warning. Users have difficulties with clean recompiling.
>
> While loading, non-fatal error may be a better option when versions do 
> not differ too much.

I do not think that we should throw and error. `org-reload' approach
should hopefully solve the problem with mixed versions. If not, not
throwing an error is still better because mixed compilation actually
works relatively fine in many cases - much better than throwing a hard
error from user perspective. And we will still notify the user that
something is wrong and that random failures are expected.

See the attached tentative patch for bugfix.

P.S. I am not sure how to approach this patch in a view of the Emacs
release process. This is a critical bug, so we aim for bugfix Org
release, but we do not want to make life harder for Emacs devs as well.


[-- Attachment #2: 0001-org-assert-version-Demote-to-warning-and-attempt-to-.patch --]
[-- Type: text/x-patch, Size: 58016 bytes --]

From 5d81ce17ade36135ae405305fb88efd79f97d96f Mon Sep 17 00:00:00 2001
Message-ID: <5d81ce17ade36135ae405305fb88efd79f97d96f.1687521721.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Fri, 23 Jun 2023 14:17:05 +0300
Subject: [PATCH] org-assert-version: Demote to warning and attempt to work
 around mixed versions

* lisp/org-load.el: New library containing load-related functions and commands.
(org-require-autoloaded-modules):
(org-reload):
(org-customize):
(org-create-customize-menu):
(org--inhibit-version-check): Move from other files.
(org-assert-version): Move from lisp/org-macs.el, remove error, and
instead try `org-reload' when version mismatch is detected.  Update
the warning text, explaining extra scenario when we compile from Git
source with stale .elc files.
* lisp/org-macs.el (org--inhibit-version-check):
(org-assert-version):
(org-find-library-dir): Move from lisp/org-compat.el
* lisp/org.el:
(org-require-autoloaded-modules):
(org-reload):
(org-customize):
(org-create-customize-menu): Move to lisp/org-load.el
* lisp/org-compat.el (org-find-library-dir): Move to lisp/org-macs.el

All the callers to `org-assert-version' are now using
(require 'org-load) instead of (require 'org-macs).

The changes in `org-assert-version' are aiming at 3 goals:

1. Move `org-assert-version' to a new file, not available in older
   Org versions.  This makes sure that (require 'org-load) either
   signals an error or guarantees loading `org-assert-version'
   macro.  Previous approach with defining the macro in org-macs.el
   suffered from byte-compilation silently treating the macro as
   undefined function; when org-macs.el was taken from older Org
   version and did not define the macro at all.

2. Be less strict and do not throw an error.  This is necessary
   because mixed Org versions have non-zero chance to produce working
   Org, while throwing an error is always hitting users.  In this
   scenario, pedantic approach makes more harm.

3. Try to recover gracefully from mixed version state by re-loading
   Org.  This should be considered the last resort and we need to
   attract user's attention with a warning, but at least `org-reload'
   has a chance to keep Org operational.

   An alternative to `org-reload' could also be
   "require-with-shadow-check" that ensures that each required library
   is taken from the up-to-date `load-path' ordering [1].  However,
   Org currently does not rely strictly upon `require' statements.
   Org libraries are full of `declare-function' statements that
   implicitly assume that certain libraries would be loaded by other
   means.  So, the alternative approach is less reliable compared to
   full Org re-loading.

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62762#110

Other discussions of the issue with `org-assert-version' and mixed
install:

- https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59882
- https://list.orgmode.org/orgmode/tnm7ui$a8k$1@ciao.gmane.io/
- https://list.orgmode.org/orgmode/87bkonzisl.fsf@gnu.org/
- https://yhetil.org/emacs-devel/00b52cc1-b0ac-a0c0-c91d-33a7213e9881@web.de/
- https://irreal.org/blog/?p=10999
---
 lisp/ob-C.el           |   2 +-
 lisp/ob-R.el           |   2 +-
 lisp/ob-awk.el         |   2 +-
 lisp/ob-calc.el        |   2 +-
 lisp/ob-clojure.el     |   2 +-
 lisp/ob-comint.el      |   2 +-
 lisp/ob-core.el        |   2 +-
 lisp/ob-css.el         |   2 +-
 lisp/ob-ditaa.el       |   2 +-
 lisp/ob-dot.el         |   2 +-
 lisp/ob-emacs-lisp.el  |   2 +-
 lisp/ob-eshell.el      |   2 +-
 lisp/ob-eval.el        |   2 +-
 lisp/ob-exp.el         |   2 +-
 lisp/ob-forth.el       |   2 +-
 lisp/ob-fortran.el     |   2 +-
 lisp/ob-gnuplot.el     |   2 +-
 lisp/ob-groovy.el      |   2 +-
 lisp/ob-haskell.el     |   2 +-
 lisp/ob-java.el        |   2 +-
 lisp/ob-js.el          |   2 +-
 lisp/ob-julia.el       |   2 +-
 lisp/ob-latex.el       |   2 +-
 lisp/ob-lilypond.el    |   2 +-
 lisp/ob-lisp.el        |   2 +-
 lisp/ob-lob.el         |   2 +-
 lisp/ob-lua.el         |   2 +-
 lisp/ob-makefile.el    |   2 +-
 lisp/ob-matlab.el      |   2 +-
 lisp/ob-maxima.el      |   2 +-
 lisp/ob-ocaml.el       |   2 +-
 lisp/ob-octave.el      |   2 +-
 lisp/ob-org.el         |   2 +-
 lisp/ob-perl.el        |   2 +-
 lisp/ob-plantuml.el    |   2 +-
 lisp/ob-processing.el  |   2 +-
 lisp/ob-python.el      |   2 +-
 lisp/ob-ref.el         |   2 +-
 lisp/ob-ruby.el        |   2 +-
 lisp/ob-sass.el        |   2 +-
 lisp/ob-scheme.el      |   2 +-
 lisp/ob-screen.el      |   2 +-
 lisp/ob-sed.el         |   2 +-
 lisp/ob-shell.el       |   2 +-
 lisp/ob-sql.el         |   2 +-
 lisp/ob-sqlite.el      |   2 +-
 lisp/ob-table.el       |   2 +-
 lisp/ob-tangle.el      |   2 +-
 lisp/ob.el             |   2 +-
 lisp/oc-basic.el       |   2 +-
 lisp/oc-biblatex.el    |   2 +-
 lisp/oc-bibtex.el      |   2 +-
 lisp/oc-csl.el         |   2 +-
 lisp/oc-natbib.el      |   2 +-
 lisp/oc.el             |   2 +-
 lisp/ol-bbdb.el        |   2 +-
 lisp/ol-bibtex.el      |   2 +-
 lisp/ol-docview.el     |   2 +-
 lisp/ol-doi.el         |   2 +-
 lisp/ol-eshell.el      |   2 +-
 lisp/ol-eww.el         |   2 +-
 lisp/ol-gnus.el        |   2 +-
 lisp/ol-info.el        |   2 +-
 lisp/ol-irc.el         |   2 +-
 lisp/ol-man.el         |   2 +-
 lisp/ol-mhe.el         |   2 +-
 lisp/ol-rmail.el       |   2 +-
 lisp/ol-w3m.el         |   2 +-
 lisp/ol.el             |   2 +-
 lisp/org-agenda.el     |   2 +-
 lisp/org-archive.el    |   2 +-
 lisp/org-attach-git.el |   2 +-
 lisp/org-attach.el     |   2 +-
 lisp/org-capture.el    |   2 +-
 lisp/org-clock.el      |   2 +-
 lisp/org-compat.el     |   3 -
 lisp/org-crypt.el      |   2 +-
 lisp/org-ctags.el      |   2 +-
 lisp/org-cycle.el      |   2 +-
 lisp/org-datetree.el   |   2 +-
 lisp/org-duration.el   |   2 +-
 lisp/org-element.el    |   2 +-
 lisp/org-entities.el   |   2 +-
 lisp/org-faces.el      |   2 +-
 lisp/org-feed.el       |   2 +-
 lisp/org-fold-core.el  |   2 +-
 lisp/org-fold.el       |   2 +-
 lisp/org-footnote.el   |   2 +-
 lisp/org-goto.el       |   2 +-
 lisp/org-habit.el      |   2 +-
 lisp/org-id.el         |   2 +-
 lisp/org-indent.el     |   2 +-
 lisp/org-inlinetask.el |   2 +-
 lisp/org-keys.el       |   2 +-
 lisp/org-lint.el       |   2 +-
 lisp/org-list.el       |   2 +-
 lisp/org-load.el       | 196 +++++++++++++++++++++++++++++++++++++++++
 lisp/org-macro.el      |   2 +-
 lisp/org-macs.el       |  69 +--------------
 lisp/org-mobile.el     |   2 +-
 lisp/org-mouse.el      |   2 +-
 lisp/org-num.el        |   2 +-
 lisp/org-pcomplete.el  |   2 +-
 lisp/org-persist.el    |   2 +-
 lisp/org-plot.el       |   2 +-
 lisp/org-protocol.el   |   2 +-
 lisp/org-refile.el     |   2 +-
 lisp/org-src.el        |   2 +-
 lisp/org-table.el      |   2 +-
 lisp/org-tempo.el      |   2 +-
 lisp/org-timer.el      |   2 +-
 lisp/org.el            |  85 +-----------------
 lisp/ox-ascii.el       |   2 +-
 lisp/ox-beamer.el      |   2 +-
 lisp/ox-html.el        |   2 +-
 lisp/ox-icalendar.el   |   2 +-
 lisp/ox-koma-letter.el |   2 +-
 lisp/ox-latex.el       |   2 +-
 lisp/ox-man.el         |   2 +-
 lisp/ox-md.el          |   2 +-
 lisp/ox-odt.el         |   2 +-
 lisp/ox-org.el         |   2 +-
 lisp/ox-publish.el     |   2 +-
 lisp/ox-texinfo.el     |   2 +-
 lisp/ox.el             |   2 +-
 125 files changed, 321 insertions(+), 274 deletions(-)
 create mode 100644 lisp/org-load.el

diff --git a/lisp/ob-C.el b/lisp/ob-C.el
index 7763c4c07..4251f86bf 100644
--- a/lisp/ob-C.el
+++ b/lisp/ob-C.el
@@ -33,7 +33,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cc-mode)
diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index f2cc069aa..3d29ebfec 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -29,7 +29,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ob-awk.el b/lisp/ob-awk.el
index e5e42a91d..2add0a329 100644
--- a/lisp/ob-awk.el
+++ b/lisp/ob-awk.el
@@ -33,7 +33,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-calc.el b/lisp/ob-calc.el
index 5572ce818..68ca87025 100644
--- a/lisp/ob-calc.el
+++ b/lisp/ob-calc.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-clojure.el b/lisp/ob-clojure.el
index d993e0cb7..0047f6dbf 100644
--- a/lisp/ob-clojure.el
+++ b/lisp/ob-clojure.el
@@ -46,7 +46,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index adfe31c40..23b457001 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -31,7 +31,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob-core)
diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index e69ce4f1d..a7315130b 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -24,7 +24,7 @@ ;;; ob-core.el --- Working with Code Blocks          -*- lexical-binding: t; -*-
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ob-css.el b/lisp/ob-css.el
index f93c576fc..0dff5525e 100644
--- a/lisp/ob-css.el
+++ b/lisp/ob-css.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-ditaa.el b/lisp/ob-ditaa.el
index 622000828..70f5ed679 100644
--- a/lisp/ob-ditaa.el
+++ b/lisp/ob-ditaa.el
@@ -37,7 +37,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-dot.el b/lisp/ob-dot.el
index ecef071a7..b62266503 100644
--- a/lisp/ob-dot.el
+++ b/lisp/ob-dot.el
@@ -40,7 +40,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-emacs-lisp.el b/lisp/ob-emacs-lisp.el
index f2111d649..3ebc2cf5d 100644
--- a/lisp/ob-emacs-lisp.el
+++ b/lisp/ob-emacs-lisp.el
@@ -27,7 +27,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob-core)
diff --git a/lisp/ob-eshell.el b/lisp/ob-eshell.el
index 95f5777ae..5c82a8f43 100644
--- a/lisp/ob-eshell.el
+++ b/lisp/ob-eshell.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-eval.el b/lisp/ob-eval.el
index 07e530772..071102ac9 100644
--- a/lisp/ob-eval.el
+++ b/lisp/ob-eval.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (eval-when-compile (require 'subr-x))  ; For `string-empty-p', Emacs < 29
diff --git a/lisp/ob-exp.el b/lisp/ob-exp.el
index 4d1b084be..d062c75d1 100644
--- a/lisp/ob-exp.el
+++ b/lisp/ob-exp.el
@@ -24,7 +24,7 @@ ;;; ob-exp.el --- Exportation of Babel Source Blocks -*- lexical-binding: t; -*-
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob-core)
diff --git a/lisp/ob-forth.el b/lisp/ob-forth.el
index e5dcad6d0..e90a6ae13 100644
--- a/lisp/ob-forth.el
+++ b/lisp/ob-forth.el
@@ -33,7 +33,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-fortran.el b/lisp/ob-fortran.el
index 2c050cb18..1e4b62159 100644
--- a/lisp/ob-fortran.el
+++ b/lisp/ob-fortran.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-gnuplot.el b/lisp/ob-gnuplot.el
index e3e42918c..fbd7f1186 100644
--- a/lisp/ob-gnuplot.el
+++ b/lisp/ob-gnuplot.el
@@ -40,7 +40,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-groovy.el b/lisp/ob-groovy.el
index c0da69fcb..7e3315baa 100644
--- a/lisp/ob-groovy.el
+++ b/lisp/ob-groovy.el
@@ -32,7 +32,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-haskell.el b/lisp/ob-haskell.el
index 7185ed61f..495d18d0e 100644
--- a/lisp/ob-haskell.el
+++ b/lisp/ob-haskell.el
@@ -40,7 +40,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-java.el b/lisp/ob-java.el
index 893860526..034034876 100644
--- a/lisp/ob-java.el
+++ b/lisp/ob-java.el
@@ -29,7 +29,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-js.el b/lisp/ob-js.el
index de7ea9131..e0fc0171b 100644
--- a/lisp/ob-js.el
+++ b/lisp/ob-js.el
@@ -39,7 +39,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-julia.el b/lisp/ob-julia.el
index cb5c7fa3b..9895aa10f 100644
--- a/lisp/ob-julia.el
+++ b/lisp/ob-julia.el
@@ -32,7 +32,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ob-latex.el b/lisp/ob-latex.el
index ce39628d6..68b584523 100644
--- a/lisp/ob-latex.el
+++ b/lisp/ob-latex.el
@@ -31,7 +31,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-lilypond.el b/lisp/ob-lilypond.el
index 9693b89e2..75470c71f 100644
--- a/lisp/ob-lilypond.el
+++ b/lisp/ob-lilypond.el
@@ -33,7 +33,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-lisp.el b/lisp/ob-lisp.el
index 048ef883c..268129460 100644
--- a/lisp/ob-lisp.el
+++ b/lisp/ob-lisp.el
@@ -37,7 +37,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-lob.el b/lisp/ob-lob.el
index f77bda73d..e8ceb42e3 100644
--- a/lisp/ob-lob.el
+++ b/lisp/ob-lob.el
@@ -24,7 +24,7 @@ ;;; ob-lob.el --- Functions Supporting the Library of Babel -*- lexical-binding:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ob-lua.el b/lisp/ob-lua.el
index 81521d9e0..6c7aeb2f5 100644
--- a/lisp/ob-lua.el
+++ b/lisp/ob-lua.el
@@ -36,7 +36,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-makefile.el b/lisp/ob-makefile.el
index 25306159b..31c4a509e 100644
--- a/lisp/ob-makefile.el
+++ b/lisp/ob-makefile.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-matlab.el b/lisp/ob-matlab.el
index 43344495d..38c7b539f 100644
--- a/lisp/ob-matlab.el
+++ b/lisp/ob-matlab.el
@@ -36,7 +36,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-maxima.el b/lisp/ob-maxima.el
index d1d7c7424..0673e0b7f 100644
--- a/lisp/ob-maxima.el
+++ b/lisp/ob-maxima.el
@@ -32,7 +32,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-ocaml.el b/lisp/ob-ocaml.el
index 09224b98b..7d89f6141 100644
--- a/lisp/ob-ocaml.el
+++ b/lisp/ob-ocaml.el
@@ -36,7 +36,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-octave.el b/lisp/ob-octave.el
index 1de263a52..9261404d4 100644
--- a/lisp/ob-octave.el
+++ b/lisp/ob-octave.el
@@ -30,7 +30,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-org.el b/lisp/ob-org.el
index 1bf02b461..79084e2f2 100644
--- a/lisp/ob-org.el
+++ b/lisp/ob-org.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-perl.el b/lisp/ob-perl.el
index c8cff5da5..45e31fd32 100644
--- a/lisp/ob-perl.el
+++ b/lisp/ob-perl.el
@@ -29,7 +29,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-plantuml.el b/lisp/ob-plantuml.el
index 3202c6e41..705925cd7 100644
--- a/lisp/ob-plantuml.el
+++ b/lisp/ob-plantuml.el
@@ -35,7 +35,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-processing.el b/lisp/ob-processing.el
index 4eeaf98e0..558b4099d 100644
--- a/lisp/ob-processing.el
+++ b/lisp/ob-processing.el
@@ -52,7 +52,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index c19af0ab3..1512d3234 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -29,7 +29,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-ref.el b/lisp/ob-ref.el
index 2eae3690d..51ecd51fe 100644
--- a/lisp/ob-ref.el
+++ b/lisp/ob-ref.el
@@ -50,7 +50,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob-core)
diff --git a/lisp/ob-ruby.el b/lisp/ob-ruby.el
index b94bc73dd..390e32074 100644
--- a/lisp/ob-ruby.el
+++ b/lisp/ob-ruby.el
@@ -36,7 +36,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-sass.el b/lisp/ob-sass.el
index 1498b931e..c8970b2a6 100644
--- a/lisp/ob-sass.el
+++ b/lisp/ob-sass.el
@@ -39,7 +39,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-scheme.el b/lisp/ob-scheme.el
index ef41a2ec4..0d1ec142a 100644
--- a/lisp/ob-scheme.el
+++ b/lisp/ob-scheme.el
@@ -39,7 +39,7 @@ ;;; Requirements:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-screen.el b/lisp/ob-screen.el
index 269538e79..f7f1a1d67 100644
--- a/lisp/ob-screen.el
+++ b/lisp/ob-screen.el
@@ -35,7 +35,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-sed.el b/lisp/ob-sed.el
index 6469901fd..1b281c80d 100644
--- a/lisp/ob-sed.el
+++ b/lisp/ob-sed.el
@@ -41,7 +41,7 @@ ;;; Usage:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 2c30a2605..b5deaf61b 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -27,7 +27,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-sql.el b/lisp/ob-sql.el
index f73e7003f..377e31f0f 100644
--- a/lisp/ob-sql.el
+++ b/lisp/ob-sql.el
@@ -71,7 +71,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-sqlite.el b/lisp/ob-sqlite.el
index 526b73ebd..533bee2a1 100644
--- a/lisp/ob-sqlite.el
+++ b/lisp/ob-sqlite.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob)
diff --git a/lisp/ob-table.el b/lisp/ob-table.el
index 5c91f33f8..88d74816f 100644
--- a/lisp/ob-table.el
+++ b/lisp/ob-table.el
@@ -54,7 +54,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ob-core)
diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index e63bde89f..824dc963e 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -27,7 +27,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ob.el b/lisp/ob.el
index 4444ced58..0a81584f3 100644
--- a/lisp/ob.el
+++ b/lisp/ob.el
@@ -23,7 +23,7 @@ ;;; ob.el --- Working with Code Blocks in Org        -*- lexical-binding: t; -*-
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org-macs)
diff --git a/lisp/oc-basic.el b/lisp/oc-basic.el
index 1c8c37aa9..105e759ba 100644
--- a/lisp/oc-basic.el
+++ b/lisp/oc-basic.el
@@ -66,7 +66,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'bibtex)
diff --git a/lisp/oc-biblatex.el b/lisp/oc-biblatex.el
index b2d31f0f6..bb6d6dca8 100644
--- a/lisp/oc-biblatex.el
+++ b/lisp/oc-biblatex.el
@@ -63,7 +63,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'map)
diff --git a/lisp/oc-bibtex.el b/lisp/oc-bibtex.el
index d5e5490ed..a5a311224 100644
--- a/lisp/oc-bibtex.el
+++ b/lisp/oc-bibtex.el
@@ -36,7 +36,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'oc)
diff --git a/lisp/oc-csl.el b/lisp/oc-csl.el
index 2b00cb599..65e821593 100644
--- a/lisp/oc-csl.el
+++ b/lisp/oc-csl.el
@@ -112,7 +112,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/oc-natbib.el b/lisp/oc-natbib.el
index 855be2a5c..5900d720b 100644
--- a/lisp/oc-natbib.el
+++ b/lisp/oc-natbib.el
@@ -43,7 +43,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'oc)
diff --git a/lisp/oc.el b/lisp/oc.el
index 8a7b66209..e62088e11 100644
--- a/lisp/oc.el
+++ b/lisp/oc.el
@@ -61,7 +61,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org-compat)
diff --git a/lisp/ol-bbdb.el b/lisp/ol-bbdb.el
index 47bd9d98c..a0ec6a79a 100644
--- a/lisp/ol-bbdb.el
+++ b/lisp/ol-bbdb.el
@@ -93,7 +93,7 @@ ;;; Commentary:
 ;;
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ol-bibtex.el b/lisp/ol-bibtex.el
index fd9517233..dd11b422d 100644
--- a/lisp/ol-bibtex.el
+++ b/lisp/ol-bibtex.el
@@ -107,7 +107,7 @@ ;;; History:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'bibtex)
diff --git a/lisp/ol-docview.el b/lisp/ol-docview.el
index a9cd9b995..fd4525940 100644
--- a/lisp/ol-docview.el
+++ b/lisp/ol-docview.el
@@ -42,7 +42,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'doc-view)
diff --git a/lisp/ol-doi.el b/lisp/ol-doi.el
index 971b05087..e24c4b463 100644
--- a/lisp/ol-doi.el
+++ b/lisp/ol-doi.el
@@ -26,7 +26,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ol)
diff --git a/lisp/ol-eshell.el b/lisp/ol-eshell.el
index b7d3baf38..67b79538f 100644
--- a/lisp/ol-eshell.el
+++ b/lisp/ol-eshell.el
@@ -23,7 +23,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'eshell)
diff --git a/lisp/ol-eww.el b/lisp/ol-eww.el
index ee970b6aa..1d404b6de 100644
--- a/lisp/ol-eww.el
+++ b/lisp/ol-eww.el
@@ -45,7 +45,7 @@ ;;; Commentary:
 \f
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ol)
diff --git a/lisp/ol-gnus.el b/lisp/ol-gnus.el
index 7c07ce045..6a4362bc5 100644
--- a/lisp/ol-gnus.el
+++ b/lisp/ol-gnus.el
@@ -31,7 +31,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'gnus-sum)
diff --git a/lisp/ol-info.el b/lisp/ol-info.el
index ad9e4a12b..7df698daa 100644
--- a/lisp/ol-info.el
+++ b/lisp/ol-info.el
@@ -30,7 +30,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ol)
diff --git a/lisp/ol-irc.el b/lisp/ol-irc.el
index 3760644e8..b14d3c47f 100644
--- a/lisp/ol-irc.el
+++ b/lisp/ol-irc.el
@@ -48,7 +48,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ol)
diff --git a/lisp/ol-man.el b/lisp/ol-man.el
index 40fdc8f92..96c991b21 100644
--- a/lisp/ol-man.el
+++ b/lisp/ol-man.el
@@ -24,7 +24,7 @@ ;;; ol-man.el --- Links to man pages -*- lexical-binding: t; -*-
 ;;
 ;;; Commentary:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ol)
diff --git a/lisp/ol-mhe.el b/lisp/ol-mhe.el
index 403b5a254..16b1967f8 100644
--- a/lisp/ol-mhe.el
+++ b/lisp/ol-mhe.el
@@ -30,7 +30,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org-macs)
diff --git a/lisp/ol-rmail.el b/lisp/ol-rmail.el
index 287779828..f95b6999c 100644
--- a/lisp/ol-rmail.el
+++ b/lisp/ol-rmail.el
@@ -30,7 +30,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ol)
diff --git a/lisp/ol-w3m.el b/lisp/ol-w3m.el
index 6fb01a60d..6d14e6ebc 100644
--- a/lisp/ol-w3m.el
+++ b/lisp/ol-w3m.el
@@ -41,7 +41,7 @@ ;;; Acknowledgments:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ol)
diff --git a/lisp/ol.el b/lisp/ol.el
index 9ad191c8f..e5e8de80f 100644
--- a/lisp/ol.el
+++ b/lisp/ol.el
@@ -27,7 +27,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org-compat)
diff --git a/lisp/org-agenda.el b/lisp/org-agenda.el
index d3e616431..6ff3b3cb0 100644
--- a/lisp/org-agenda.el
+++ b/lisp/org-agenda.el
@@ -45,7 +45,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-archive.el b/lisp/org-archive.el
index e74c59a51..3f74e6d14 100644
--- a/lisp/org-archive.el
+++ b/lisp/org-archive.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org)
diff --git a/lisp/org-attach-git.el b/lisp/org-attach-git.el
index 4763aefd4..2c9242d70 100644
--- a/lisp/org-attach-git.el
+++ b/lisp/org-attach-git.el
@@ -29,7 +29,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org-attach)
diff --git a/lisp/org-attach.el b/lisp/org-attach.el
index e8b27751e..428c12068 100644
--- a/lisp/org-attach.el
+++ b/lisp/org-attach.el
@@ -34,7 +34,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-capture.el b/lisp/org-capture.el
index a696c615b..cf487ac19 100644
--- a/lisp/org-capture.el
+++ b/lisp/org-capture.el
@@ -47,7 +47,7 @@ ;;; TODO
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-clock.el b/lisp/org-clock.el
index d7fe14cd5..77225d2d6 100644
--- a/lisp/org-clock.el
+++ b/lisp/org-clock.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-compat.el b/lisp/org-compat.el
index d5bf2191a..360143018 100644
--- a/lisp/org-compat.el
+++ b/lisp/org-compat.el
@@ -1176,9 +1176,6 @@ (defun org-move-to-column (column &optional force _buffer)
            buffer-invisibility-spec)))
     (move-to-column column force)))
 
-(defmacro org-find-library-dir (library)
-  `(file-name-directory (or (locate-library ,library) "")))
-
 (defun org-count-lines (s)
   "How many lines in string S?"
   (let ((start 0) (n 1))
diff --git a/lisp/org-crypt.el b/lisp/org-crypt.el
index b5df115cf..b6a306371 100644
--- a/lisp/org-crypt.el
+++ b/lisp/org-crypt.el
@@ -54,7 +54,7 @@ ;;; Thanks:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org-macs)
diff --git a/lisp/org-ctags.el b/lisp/org-ctags.el
index 5dd2bfd59..8ff2563e1 100644
--- a/lisp/org-ctags.el
+++ b/lisp/org-ctags.el
@@ -135,7 +135,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (eval-when-compile (require 'cl-lib))
diff --git a/lisp/org-cycle.el b/lisp/org-cycle.el
index 90fc95f41..85e94657f 100644
--- a/lisp/org-cycle.el
+++ b/lisp/org-cycle.el
@@ -29,7 +29,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org-macs)
diff --git a/lisp/org-datetree.el b/lisp/org-datetree.el
index 035ef047a..d3491324d 100644
--- a/lisp/org-datetree.el
+++ b/lisp/org-datetree.el
@@ -30,7 +30,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org)
diff --git a/lisp/org-duration.el b/lisp/org-duration.el
index 2919efd4c..2adddcc1b 100644
--- a/lisp/org-duration.el
+++ b/lisp/org-duration.el
@@ -51,7 +51,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-element.el b/lisp/org-element.el
index df43ebcf0..7397cb6cb 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -58,7 +58,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'avl-tree)
diff --git a/lisp/org-entities.el b/lisp/org-entities.el
index 61083022b..7125a5c77 100644
--- a/lisp/org-entities.el
+++ b/lisp/org-entities.el
@@ -27,7 +27,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (declare-function org-mode "org" ())
diff --git a/lisp/org-faces.el b/lisp/org-faces.el
index 594c9a6e7..5c1bd50a1 100644
--- a/lisp/org-faces.el
+++ b/lisp/org-faces.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (defgroup org-faces nil
diff --git a/lisp/org-feed.el b/lisp/org-feed.el
index f0b294b4a..43c035ad6 100644
--- a/lisp/org-feed.el
+++ b/lisp/org-feed.el
@@ -88,7 +88,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org)
diff --git a/lisp/org-fold-core.el b/lisp/org-fold-core.el
index 6c17b00fa..1ba7601f1 100644
--- a/lisp/org-fold-core.el
+++ b/lisp/org-fold-core.el
@@ -270,7 +270,7 @@ ;;; Performance considerations
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org-macs)
diff --git a/lisp/org-fold.el b/lisp/org-fold.el
index 1b7ca22b0..2e63f5100 100644
--- a/lisp/org-fold.el
+++ b/lisp/org-fold.el
@@ -42,7 +42,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org-macs)
diff --git a/lisp/org-footnote.el b/lisp/org-footnote.el
index 6bdd0b32f..7874a4af1 100644
--- a/lisp/org-footnote.el
+++ b/lisp/org-footnote.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 ;;;; Declarations
diff --git a/lisp/org-goto.el b/lisp/org-goto.el
index 00bf5d46a..2da5a86e2 100644
--- a/lisp/org-goto.el
+++ b/lisp/org-goto.el
@@ -22,7 +22,7 @@ ;;; org-goto.el --- Fast navigation in an Org buffer  -*- lexical-binding: t; -*
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org)
diff --git a/lisp/org-habit.el b/lisp/org-habit.el
index fb6a48b5f..a835ff270 100644
--- a/lisp/org-habit.el
+++ b/lisp/org-habit.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-id.el b/lisp/org-id.el
index 490e4ddee..d7b4dd338 100644
--- a/lisp/org-id.el
+++ b/lisp/org-id.el
@@ -70,7 +70,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org)
diff --git a/lisp/org-indent.el b/lisp/org-indent.el
index fe5233878..0a0019a44 100644
--- a/lisp/org-indent.el
+++ b/lisp/org-indent.el
@@ -36,7 +36,7 @@ ;;; Commentary:
 ;;
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org-macs)
diff --git a/lisp/org-inlinetask.el b/lisp/org-inlinetask.el
index cb7fdf88b..1fb16a2f8 100644
--- a/lisp/org-inlinetask.el
+++ b/lisp/org-inlinetask.el
@@ -78,7 +78,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org)
diff --git a/lisp/org-keys.el b/lisp/org-keys.el
index 37a150bb7..b5c576b2a 100644
--- a/lisp/org-keys.el
+++ b/lisp/org-keys.el
@@ -27,7 +27,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-lint.el b/lisp/org-lint.el
index 0e2967b6c..e6c4d088f 100644
--- a/lisp/org-lint.el
+++ b/lisp/org-lint.el
@@ -85,7 +85,7 @@ ;;; Commentary:
 \f
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-list.el b/lisp/org-list.el
index 1fc99d070..63ff65bca 100644
--- a/lisp/org-list.el
+++ b/lisp/org-list.el
@@ -76,7 +76,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-load.el b/lisp/org-load.el
new file mode 100644
index 000000000..c618452f2
--- /dev/null
+++ b/lisp/org-load.el
@@ -0,0 +1,196 @@
+;;; org-load.el --- Functions and commands that load Org mode         -*- lexical-binding: t; -*-
+
+;; Copyright (C) 2023-2023 Free Software Foundation, Inc.
+
+;; Author: Ihor Radchenko <yantar92 at gmail dot com>
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+;;
+;; This file contains commands and functions dealing with loading
+;; various parts of Org and checking consistency of the Org libraries.
+
+(require 'org-macs)
+
+(defun org-require-autoloaded-modules ()
+  (interactive)
+  (mapc #'require
+	'(org-agenda org-archive org-attach org-clock org-colview org-id
+		     org-table org-timer)))
+
+;;;###autoload
+(defun org-reload (&optional uncompiled)
+  "Reload all Org Lisp files.
+With prefix arg UNCOMPILED, load the uncompiled versions."
+  (interactive "P")
+  (require 'loadhist)
+  (let* ((org-dir     (org-find-library-dir "org"))
+	 (contrib-dir (or (org-find-library-dir "org-contribdir") org-dir))
+	 (feature-re "^\\(org\\|ob\\|ox\\|ol\\|oc\\)\\(-.*\\)?")
+	 (remove-re (format "\\`%s\\'"
+			    (regexp-opt '("org" "org-loaddefs" "org-version"))))
+	 (feats (delete-dups
+		 (mapcar 'file-name-sans-extension
+			 (mapcar 'file-name-nondirectory
+				 (delq nil
+				       (mapcar 'feature-file
+					       features))))))
+	 (lfeat (append
+		 (sort
+		  (setq feats
+			(delq nil (mapcar
+				 (lambda (f)
+				   (if (and (string-match feature-re f)
+					    (not (string-match remove-re f)))
+				       f nil))
+				 feats)))
+		  'string-lessp)
+		 (list "org-version" "org")))
+	 (load-suffixes (if uncompiled (reverse load-suffixes) load-suffixes))
+	 load-uncore load-misses)
+    (setq load-misses
+	  (delq t
+		(mapcar (lambda (f)
+			  (or (org-load-noerror-mustsuffix (concat org-dir f))
+			      (and (string= org-dir contrib-dir)
+				   (org-load-noerror-mustsuffix (concat contrib-dir f)))
+			      (and (org-load-noerror-mustsuffix (concat (org-find-library-dir f) f))
+				   (push f load-uncore)
+				   t)
+			      f))
+			lfeat)))
+    (when load-uncore
+      (message "The following feature%s found in load-path, please check if that's correct:\n%s"
+	       (if (> (length load-uncore) 1) "s were" " was")
+               (reverse load-uncore)))
+    (if load-misses
+	(message "Some error occurred while reloading Org feature%s\n%s\nPlease check *Messages*!\n%s"
+		 (if (> (length load-misses) 1) "s" "") load-misses (org-version nil 'full))
+      (message "Successfully reloaded Org\n%s" (org-version nil 'full)))))
+
+;;;###autoload
+(defun org-customize ()
+  "Call the customize function with org as argument."
+  (interactive)
+  (org-load-modules-maybe)
+  (org-require-autoloaded-modules)
+  (customize-browse 'org))
+
+(defun org-create-customize-menu ()
+  "Create a full customization menu for Org mode, insert it into the menu."
+  (interactive)
+  (org-load-modules-maybe)
+  (org-require-autoloaded-modules)
+  (easy-menu-change
+   '("Org") "Customize"
+   `(["Browse Org group" org-customize t]
+     "--"
+     ,(customize-menu-create 'org)
+     ["Set" Custom-set t]
+     ["Save" Custom-save t]
+     ["Reset to Current" Custom-reset-current t]
+     ["Reset to Saved" Custom-reset-saved t]
+     ["Reset to Standard Settings" Custom-reset-standard t]))
+  (message "\"Org\"-menu now contains full customization menu"))
+
+;;; Org version verification.
+
+(defvar org--inhibit-version-check nil
+  "When non-nil, skip the detection of mixed-versions situations.
+For internal use only.  See Emacs bug #62762.
+This variable is only supposed to be changed by Emacs build scripts.
+When nil, Org tries to detect when Org source files were compiled with
+a different version of Org (which tends to lead to incorrect `.elc' files),
+or when the current Emacs session has loaded a mix of files from different
+Org versions (typically the one bundled with Emacs and another one installed
+from GNU ELPA), which can happen if some parts of Org were loaded before
+`load-path' was changed (e.g. before the GNU-ELPA-installed Org is activated
+by `package-activate-all').")
+(defmacro org-assert-version ()
+  "Check compile time and runtime version match.
+Warn, and attempt to fix mixed version, if any."
+  ;; We intentionally use a more permissive `org-release' instead of
+  ;; `org-git-version' to work around deficiencies in Elisp
+  ;; compilation after pulling latest changes.  Unchanged files will
+  ;; not be re-compiled and thus their macro-expanded
+  ;; `org-assert-version' calls would fail using strict
+  ;; `org-git-version' check because the generated Org version strings
+  ;; will not match.
+  `(unless (or org--inhibit-version-check (equal (org-release) ,(org-release)))
+     (warn "Org version mismatch.  Random Org errors may occur.
+This warning usually appears when a built-in Org version is loaded
+prior to the more recent Org version.
+
+Version mismatch is commonly encountered in the following situations:
+
+1. Some compiled Org libraries are not re-compiled after Org update.
+
+   It is recommended to re-install Org, or, if Org is loaded from git
+   repo, to run
+
+      make clean; make autoloads
+
+2. Emacs is loaded using literate Org config and more recent Org
+   version is loaded inside the file loaded by `org-babel-load-file'.
+   `org-babel-load-file' triggers the built-in Org version clashing
+   the newer Org version attempt to be loaded later.
+
+   It is recommended to move the Org loading code before the
+   `org-babel-load-file' call.
+
+3. New Org version is loaded manually by setting `load-path', but some
+   other package depending on Org is loaded before the `load-path' is
+   configured.
+   This \"other package\" is triggering built-in Org version, again
+   causing the version mismatch.
+
+   It is recommended to set `load-path' as early in the config as
+   possible.
+
+4. New Org version is loaded using straight.el package manager and
+   other package depending on Org is loaded before straight triggers
+   loading of the newer Org version.
+
+   It is recommended to put
+
+    %s
+
+   early in the config.  Ideally, right after the straight.el
+   bootstrap.  Moving `use-package' :straight declaration may not be
+   sufficient if the corresponding `use-package' statement is
+   deferring the loading."
+           ;; Avoid `warn' replacing "'" with "’" (see `format-message').
+           "(straight-use-package 'org)")
+     ;; Try to re-load Org.
+     ;; Do the re-loading after the current, faulty file is loaded.
+     ;; The current file may still fail to load - situation we cannot
+     ;; work around.
+     ;; We re-load the whole Org here because Org commonly uses
+     ;; `declare-function' statements and thus more granular approach,
+     ;; like suggested in
+     ;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62762#110
+     ;; will not be as reliable.
+     ;; Note that this clause will never run during compilation
+     ;; because version check can only trigger in already compiled
+     ;; files with macro expanded differently during compilation and
+     ;; during loading.  (equal (org-release) ,(org-release)) is
+     ;; always t when compiling current `load-file-name'.
+     (eval-after-load load-file-name (org-reload t))))
+
+(provide 'org-load)
+
+;;; org-load.el ends here
diff --git a/lisp/org-macro.el b/lisp/org-macro.el
index 481e431c2..98706f676 100644
--- a/lisp/org-macro.el
+++ b/lisp/org-macro.el
@@ -47,7 +47,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-macs.el b/lisp/org-macs.el
index 1552675f8..984e7c023 100644
--- a/lisp/org-macs.el
+++ b/lisp/org-macs.el
@@ -34,72 +34,6 @@ ;;; Code:
 (require 'cl-lib)
 (require 'format-spec)
 
-;;; Org version verification.
-
-(defvar org--inhibit-version-check nil
-  "When non-nil, skip the detection of mixed-versions situations.
-For internal use only.  See Emacs bug #62762.
-This variable is only supposed to be changed by Emacs build scripts.
-When nil, Org tries to detect when Org source files were compiled with
-a different version of Org (which tends to lead to incorrect `.elc' files),
-or when the current Emacs session has loaded a mix of files from different
-Org versions (typically the one bundled with Emacs and another one installed
-from GNU ELPA), which can happen if some parts of Org were loaded before
-`load-path' was changed (e.g. before the GNU-ELPA-installed Org is activated
-by `package-activate-all').")
-(defmacro org-assert-version ()
-  "Assert compile time and runtime version match."
-  ;; We intentionally use a more permissive `org-release' instead of
-  ;; `org-git-version' to work around deficiencies in Elisp
-  ;; compilation after pulling latest changes.  Unchanged files will
-  ;; not be re-compiled and thus their macro-expanded
-  ;; `org-assert-version' calls would fail using strict
-  ;; `org-git-version' check because the generated Org version strings
-  ;; will not match.
-  `(unless (or org--inhibit-version-check (equal (org-release) ,(org-release)))
-     (warn "Org version mismatch.  Org loading aborted.
-This warning usually appears when a built-in Org version is loaded
-prior to the more recent Org version.
-
-Version mismatch is commonly encountered in the following situations:
-
-1. Emacs is loaded using literate Org config and more recent Org
-   version is loaded inside the file loaded by `org-babel-load-file'.
-   `org-babel-load-file' triggers the built-in Org version clashing
-   the newer Org version attempt to be loaded later.
-
-   It is recommended to move the Org loading code before the
-   `org-babel-load-file' call.
-
-2. New Org version is loaded manually by setting `load-path', but some
-   other package depending on Org is loaded before the `load-path' is
-   configured.
-   This \"other package\" is triggering built-in Org version, again
-   causing the version mismatch.
-
-   It is recommended to set `load-path' as early in the config as
-   possible.
-
-3. New Org version is loaded using straight.el package manager and
-   other package depending on Org is loaded before straight triggers
-   loading of the newer Org version.
-
-   It is recommended to put
-
-    %s
-
-   early in the config.  Ideally, right after the straight.el
-   bootstrap.  Moving `use-package' :straight declaration may not be
-   sufficient if the corresponding `use-package' statement is
-   deferring the loading."
-           ;; Avoid `warn' replacing "'" with "’" (see `format-message').
-           "(straight-use-package 'org)")
-     (error "Org version mismatch.  Make sure that correct `load-path' is set early in init.el")))
-
-;; We rely on org-macs when generating Org version.  Checking Org
-;; version here will interfere with Org build process.
-;; (org-assert-version)
-
 (declare-function org-mode "org" ())
 (declare-function org-agenda-files "org" (&optional unrestricted archives))
 (declare-function org-time-string-to-seconds "org" (s))
@@ -215,6 +149,9 @@ (defmacro org-eval-in-environment (environment form)
   (declare (debug (form form)) (indent 1) (obsolete cl-progv "2021"))
   `(eval (list 'let ,environment ',form)))
 
+(defmacro org-find-library-dir (library)
+  `(file-name-directory (or (locate-library ,library) "")))
+
 ;;;###autoload
 (defmacro org-load-noerror-mustsuffix (file)
   "Load FILE with optional arguments NOERROR and MUSTSUFFIX."
diff --git a/lisp/org-mobile.el b/lisp/org-mobile.el
index 01395a4b6..17cb2f881 100644
--- a/lisp/org-mobile.el
+++ b/lisp/org-mobile.el
@@ -31,7 +31,7 @@ ;;; Commentary:
 ;; iPhone and Android - any external viewer/flagging/editing
 ;; application that uses the same conventions could be used.
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-mouse.el b/lisp/org-mouse.el
index 3b2c4cba6..90779be3f 100644
--- a/lisp/org-mouse.el
+++ b/lisp/org-mouse.el
@@ -136,7 +136,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org)
diff --git a/lisp/org-num.el b/lisp/org-num.el
index cbe5e455e..9d3f14402 100644
--- a/lisp/org-num.el
+++ b/lisp/org-num.el
@@ -61,7 +61,7 @@ ;;; Commentary:
 \f
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-pcomplete.el b/lisp/org-pcomplete.el
index 0567052c0..e5cc707e7 100644
--- a/lisp/org-pcomplete.el
+++ b/lisp/org-pcomplete.el
@@ -26,7 +26,7 @@ ;;; Code:
 
 ;;;; Require other packages
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org-macs)
diff --git a/lisp/org-persist.el b/lisp/org-persist.el
index 01078f459..a43853d3c 100644
--- a/lisp/org-persist.el
+++ b/lisp/org-persist.el
@@ -149,7 +149,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org-compat)
diff --git a/lisp/org-plot.el b/lisp/org-plot.el
index fe61e9ace..0eb603dc4 100644
--- a/lisp/org-plot.el
+++ b/lisp/org-plot.el
@@ -31,7 +31,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-protocol.el b/lisp/org-protocol.el
index 2b07a377e..fc6d23f5e 100644
--- a/lisp/org-protocol.el
+++ b/lisp/org-protocol.el
@@ -128,7 +128,7 @@ ;;; Commentary:
 ;;
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org)
diff --git a/lisp/org-refile.el b/lisp/org-refile.el
index 5a41c022f..947a593df 100644
--- a/lisp/org-refile.el
+++ b/lisp/org-refile.el
@@ -25,7 +25,7 @@ ;;; Commentary:
 ;; Org Refile allows you to refile subtrees to various locations.
 
 ;;; Code:
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'org)
diff --git a/lisp/org-src.el b/lisp/org-src.el
index aadd8eba5..793d89ca1 100644
--- a/lisp/org-src.el
+++ b/lisp/org-src.el
@@ -31,7 +31,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-table.el b/lisp/org-table.el
index 83bad3f3a..4add43e6d 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -34,7 +34,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org-tempo.el b/lisp/org-tempo.el
index 81ec8fdcc..4d0847dcd 100644
--- a/lisp/org-tempo.el
+++ b/lisp/org-tempo.el
@@ -38,7 +38,7 @@ ;;; Commentary:
 ;;
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'tempo)
diff --git a/lisp/org-timer.el b/lisp/org-timer.el
index 919630e69..b4d24f45c 100644
--- a/lisp/org-timer.el
+++ b/lisp/org-timer.el
@@ -35,7 +35,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/org.el b/lisp/org.el
index 2e26c08e5..3966e9b3e 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -71,7 +71,7 @@ (defvar org-inlinetask-min-level)
 
 ;;;; Require other packages
 
-(require 'org-compat)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
@@ -18278,89 +18278,6 @@ (defun org-install-agenda-files-menu ()
 		  ;; Prevent initialization from failing.
 		  (ignore-errors (org-agenda-files t)))))))))
 
-;;;; Documentation
-
-(defun org-require-autoloaded-modules ()
-  (interactive)
-  (mapc #'require
-	'(org-agenda org-archive org-attach org-clock org-colview org-id
-		     org-table org-timer)))
-
-;;;###autoload
-(defun org-reload (&optional uncompiled)
-  "Reload all Org Lisp files.
-With prefix arg UNCOMPILED, load the uncompiled versions."
-  (interactive "P")
-  (require 'loadhist)
-  (let* ((org-dir     (org-find-library-dir "org"))
-	 (contrib-dir (or (org-find-library-dir "org-contribdir") org-dir))
-	 (feature-re "^\\(org\\|ob\\|ox\\|ol\\|oc\\)\\(-.*\\)?")
-	 (remove-re (format "\\`%s\\'"
-			    (regexp-opt '("org" "org-loaddefs" "org-version"))))
-	 (feats (delete-dups
-		 (mapcar 'file-name-sans-extension
-			 (mapcar 'file-name-nondirectory
-				 (delq nil
-				       (mapcar 'feature-file
-					       features))))))
-	 (lfeat (append
-		 (sort
-		  (setq feats
-			(delq nil (mapcar
-				   (lambda (f)
-				     (if (and (string-match feature-re f)
-					      (not (string-match remove-re f)))
-					 f nil))
-				   feats)))
-		  'string-lessp)
-		 (list "org-version" "org")))
-	 (load-suffixes (if uncompiled (reverse load-suffixes) load-suffixes))
-	 load-uncore load-misses)
-    (setq load-misses
-	  (delq t
-		(mapcar (lambda (f)
-			  (or (org-load-noerror-mustsuffix (concat org-dir f))
-			      (and (string= org-dir contrib-dir)
-				   (org-load-noerror-mustsuffix (concat contrib-dir f)))
-			      (and (org-load-noerror-mustsuffix (concat (org-find-library-dir f) f))
-				   (push f load-uncore)
-				   t)
-			      f))
-			lfeat)))
-    (when load-uncore
-      (message "The following feature%s found in load-path, please check if that's correct:\n%s"
-	       (if (> (length load-uncore) 1) "s were" " was")
-               (reverse load-uncore)))
-    (if load-misses
-	(message "Some error occurred while reloading Org feature%s\n%s\nPlease check *Messages*!\n%s"
-		 (if (> (length load-misses) 1) "s" "") load-misses (org-version nil 'full))
-      (message "Successfully reloaded Org\n%s" (org-version nil 'full)))))
-
-;;;###autoload
-(defun org-customize ()
-  "Call the customize function with org as argument."
-  (interactive)
-  (org-load-modules-maybe)
-  (org-require-autoloaded-modules)
-  (customize-browse 'org))
-
-(defun org-create-customize-menu ()
-  "Create a full customization menu for Org mode, insert it into the menu."
-  (interactive)
-  (org-load-modules-maybe)
-  (org-require-autoloaded-modules)
-  (easy-menu-change
-   '("Org") "Customize"
-   `(["Browse Org group" org-customize t]
-     "--"
-     ,(customize-menu-create 'org)
-     ["Set" Custom-set t]
-     ["Save" Custom-save t]
-     ["Reset to Current" Custom-reset-current t]
-     ["Reset to Saved" Custom-reset-saved t]
-     ["Reset to Standard Settings" Custom-reset-standard t]))
-  (message "\"Org\"-menu now contains full customization menu"))
-
 ;;;; Miscellaneous stuff
 
 ;;; Generally useful functions
diff --git a/lisp/ox-ascii.el b/lisp/ox-ascii.el
index 692dd216d..13dd7439f 100644
--- a/lisp/ox-ascii.el
+++ b/lisp/ox-ascii.el
@@ -28,7 +28,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ox)
diff --git a/lisp/ox-beamer.el b/lisp/ox-beamer.el
index 689bf1559..d00baa41d 100644
--- a/lisp/ox-beamer.el
+++ b/lisp/ox-beamer.el
@@ -30,7 +30,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ox-html.el b/lisp/ox-html.el
index b27254b8a..93d8fb399 100644
--- a/lisp/ox-html.el
+++ b/lisp/ox-html.el
@@ -31,7 +31,7 @@ ;;; Code:
 
 ;;; Dependencies
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ox-icalendar.el b/lisp/ox-icalendar.el
index 81a77a770..16d88a6f3 100644
--- a/lisp/ox-icalendar.el
+++ b/lisp/ox-icalendar.el
@@ -32,7 +32,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ox-koma-letter.el b/lisp/ox-koma-letter.el
index 33eb0597b..9cd48c1f5 100644
--- a/lisp/ox-koma-letter.el
+++ b/lisp/ox-koma-letter.el
@@ -165,7 +165,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ox-latex.el b/lisp/ox-latex.el
index 768bf402b..2be4ef34c 100644
--- a/lisp/ox-latex.el
+++ b/lisp/ox-latex.el
@@ -27,7 +27,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ox-man.el b/lisp/ox-man.el
index bb716f7b9..3b703fd1f 100644
--- a/lisp/ox-man.el
+++ b/lisp/ox-man.el
@@ -37,7 +37,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ox-md.el b/lisp/ox-md.el
index 5be0ca22e..9e368ba0f 100644
--- a/lisp/ox-md.el
+++ b/lisp/ox-md.el
@@ -29,7 +29,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ox-odt.el b/lisp/ox-odt.el
index dd96ed8e0..b0c0b234a 100644
--- a/lisp/ox-odt.el
+++ b/lisp/ox-odt.el
@@ -25,7 +25,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ox-org.el b/lisp/ox-org.el
index ed72cf4f2..98e4682b4 100644
--- a/lisp/ox-org.el
+++ b/lisp/ox-org.el
@@ -25,7 +25,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'ox)
diff --git a/lisp/ox-publish.el b/lisp/ox-publish.el
index f9c3877d7..23cd7f325 100644
--- a/lisp/ox-publish.el
+++ b/lisp/ox-publish.el
@@ -38,7 +38,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ox-texinfo.el b/lisp/ox-texinfo.el
index 5befcac1c..0e5e696db 100644
--- a/lisp/ox-texinfo.el
+++ b/lisp/ox-texinfo.el
@@ -26,7 +26,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
diff --git a/lisp/ox.el b/lisp/ox.el
index 6f819def9..f979c3eef 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -72,7 +72,7 @@ ;;; Commentary:
 
 ;;; Code:
 
-(require 'org-macs)
+(require 'org-load)
 (org-assert-version)
 
 (require 'cl-lib)
-- 
2.41.0


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 23/06/2023 19:02, Ihor Radchenko wrote:
> 1. Use `org-reload' instead of load-dir test.

Just a reminder: directory test may catch the case with shadowing by Ort 
that is not compiled. Version comparison is unable to detect it.

> I do not think that we should throw and error. `org-reload' approach
> should hopefully solve the problem with mixed versions.

Have you tested compilation with Org  already loaded to Emacs <= 28?

> P.S. I am not sure how to approach this patch in a view of the Emacs
> release process.
Are there other consequences of your changes for incremental rebuilds in 
Emacs source tree besides converting error to warning?

My impression is that the most wide spread case is installing the Org 
ELPA package should just work in Emacs-29. I am unsure concerning 
precise scenario for spacemacs. I have not idea if shadowing by 
correctly compiled Org is a frequent issue. Are there other cases 
affecting significant number of users and what is behavior for them?

Current warning is so long that it may confuse users who sees only its 
final part.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:
> On 23/06/2023 19:02, Ihor Radchenko wrote:
>> 1. Use `org-reload' instead of load-dir test.
>
> Just a reminder: directory test may catch the case with shadowing by Ort 
> that is not compiled. Version comparison is unable to detect it.

Yeah, you are right. We may need to combine both the approaches with
shadow require and version check.

I will look further into this.

>> I do not think that we should throw and error. `org-reload' approach
>> should hopefully solve the problem with mixed versions.
>
> Have you tested compilation with Org  already loaded to Emacs <= 28?

Do you know how to do it? Our previous reproducers involved M-x
package-install, but I am not sure how to test it without ELPA release.

>> P.S. I am not sure how to approach this patch in a view of the Emacs
>> release process.
> Are there other consequences of your changes for incremental rebuilds in 
> Emacs source tree besides converting error to warning?

There are no consequences at all. The whole check is completely bypassed
in Emacs tree. Even the warning is suppressed. See
`org--inhibit-version-check'.

> My impression is that the most wide spread case is installing the Org 
> ELPA package should just work in Emacs-29. I am unsure concerning 
> precise scenario for spacemacs. I have not idea if shadowing by 
> correctly compiled Org is a frequent issue. Are there other cases 
> affecting significant number of users and what is behavior for them?

Installing is not the concern. My concern is introducing major changes
right ahead for the release. My question was aimed at the Emacs devs
here.

> Current warning is so long that it may confuse users who sees only its 
> final part.

You are right. I will think how to approach this.
I am considering:
1. Move most the warning to dedicated manual section.
2. Display additional short warning at the bottom.

I am now leaning towards (1).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii <eliz@gnu.org>,
>  bzg@gnu.org, dmitry@gutov.dev, 62762@debbugs.gnu.org, Alan Mackenzie
>  <acm@muc.de>
> Date: Tue, 27 Jun 2023 10:59:57 +0000
> 
> > My impression is that the most wide spread case is installing the Org 
> > ELPA package should just work in Emacs-29. I am unsure concerning 
> > precise scenario for spacemacs. I have not idea if shadowing by 
> > correctly compiled Org is a frequent issue. Are there other cases 
> > affecting significant number of users and what is behavior for them?
> 
> Installing is not the concern. My concern is introducing major changes
> right ahead for the release. My question was aimed at the Emacs devs
> here.

I was not aware that this discussion is pertinent to the release
branch.  Is it?  I thought any changes in this matter are for master.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Installing is not the concern. My concern is introducing major changes
>> right ahead for the release. My question was aimed at the Emacs devs
>> here.
>
> I was not aware that this discussion is pertinent to the release
> branch.  Is it?  I thought any changes in this matter are for master.

This is aiming for Org's bugfix branch (as a critical fix).
AFAIK, Org bugfix releases were going into release branch.

At the same time, this change is not aiming at Org distributed within
Emacs tree. The whole issue is with ELPA version of Org.

In any case, if Emacs release is coming really soon, I can just hold on
until Emacs is released.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: manikulin@gmail.com, monnier@iro.umontreal.ca, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> Date: Tue, 27 Jun 2023 12:08:37 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Installing is not the concern. My concern is introducing major changes
> >> right ahead for the release. My question was aimed at the Emacs devs
> >> here.
> >
> > I was not aware that this discussion is pertinent to the release
> > branch.  Is it?  I thought any changes in this matter are for master.
> 
> This is aiming for Org's bugfix branch (as a critical fix).
> AFAIK, Org bugfix releases were going into release branch.
> 
> At the same time, this change is not aiming at Org distributed within
> Emacs tree. The whole issue is with ELPA version of Org.

The above two paragraphs contradict each other, AFAIU.  If a change is
installed on the release branch of the Emacs Git repository, it will
be included in the release tarball, and thus in the Emacs source tree.

OTOH, if this is not about the Emacs Git repository, then why did you
say you were asking Emacs developers these questions?

I guess I'm missing something here.

> In any case, if Emacs release is coming really soon, I can just hold on
> until Emacs is released.

As long as this code will be in Emacs on the master branch, you don't
need to wait for the release (which is done from the release branch).

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> This is aiming for Org's bugfix branch (as a critical fix).
>> AFAIK, Org bugfix releases were going into release branch.
>> 
>> At the same time, this change is not aiming at Org distributed within
>> Emacs tree. The whole issue is with ELPA version of Org.
>
> The above two paragraphs contradict each other, AFAIU.  If a change is
> installed on the release branch of the Emacs Git repository, it will
> be included in the release tarball, and thus in the Emacs source tree.
>
> OTOH, if this is not about the Emacs Git repository, then why did you
> say you were asking Emacs developers these questions?
>
> I guess I'm missing something here.

Normally, every Org release is synchronized with Emacs upstream (master).
During the Emacs release, we have done the same for Emacs emacs-29
branch (Org 9.6.6 is included in emacs-29; see
https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/org/org.el?h=emacs-29#n12).

However, during the Emacs release process, if significant changes have
to be done on Org side, we may consider not syncing the Emacs release
branch and Org bugfix. See
https://orgmode.org/worg/org-maintenance.html#org-version-emacs

But definition of "significant" is subjective. So, I wanted to clarify
with you (Emacs dev).

CCing Kyle Meyer, in case if I misunderstand something.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: manikulin@gmail.com, monnier@iro.umontreal.ca, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de, Kyle Meyer
>  <kyle@kyleam.com>, Bastien <bzg@gnu.org>
> Date: Tue, 27 Jun 2023 12:30:01 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> This is aiming for Org's bugfix branch (as a critical fix).
> >> AFAIK, Org bugfix releases were going into release branch.
> >> 
> >> At the same time, this change is not aiming at Org distributed within
> >> Emacs tree. The whole issue is with ELPA version of Org.
> >
> > The above two paragraphs contradict each other, AFAIU.  If a change is
> > installed on the release branch of the Emacs Git repository, it will
> > be included in the release tarball, and thus in the Emacs source tree.
> >
> > OTOH, if this is not about the Emacs Git repository, then why did you
> > say you were asking Emacs developers these questions?
> >
> > I guess I'm missing something here.
> 
> Normally, every Org release is synchronized with Emacs upstream (master).
> During the Emacs release, we have done the same for Emacs emacs-29
> branch (Org 9.6.6 is included in emacs-29; see
> https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/org/org.el?h=emacs-29#n12).
> 
> However, during the Emacs release process, if significant changes have
> to be done on Org side, we may consider not syncing the Emacs release
> branch and Org bugfix. See
> https://orgmode.org/worg/org-maintenance.html#org-version-emacs
> 
> But definition of "significant" is subjective. So, I wanted to clarify
> with you (Emacs dev).

I'm not sure I understood the above, but just to be clear: the
emacs-29 release branch was not supposed to have any of the changes
meant to prevent aborting the build due to Org version mismatch.  I
now see that the org--inhibit-version-check change did somehow get
installed on emacs-29, and I think it's unfortunate and was never
discussed.  In https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62762#53
and again in https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62762#65 I
clearly said that the change was installed only on master.  I think
the fact that this got backported to emacs-29 was unintentional.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 27/06/2023 17:59, Ihor Radchenko wrote:
> Max Nikulin writes:
>> On 23/06/2023 19:02, Ihor Radchenko wrote:

>>> I do not think that we should throw and error. `org-reload' approach
>>> should hopefully solve the problem with mixed versions.
>>
>> Have you tested compilation with Org  already loaded to Emacs <= 28?
> 
> Do you know how to do it? Our previous reproducers involved M-x
> package-install, but I am not sure how to test it without ELPA release.

I have not tried yet, but I hope, preparing a tar file and using 
`package-install-file' should behave in the same way as installing from 
ELPA. Some spacemacs users believe they have noticed some difference, 
but I am not convinced.

> You are right. I will think how to approach this.
> I am considering:
> 1. Move most the warning to dedicated manual section.
> 2. Display additional short warning at the bottom.
> 
> I am now leaning towards (1).

Since it will be only in new version of the manual, a URL with online 
manual should be provided. Using `info' may give old version with no 
added section. Perhaps the FAQ entry should be revised as well.

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> Do you know how to do it? Our previous reproducers involved M-x
>> package-install, but I am not sure how to test it without ELPA release.
>
> I have not tried yet, but I hope, preparing a tar file and using 
> `package-install-file' should behave in the same way as installing from 
> ELPA. Some spacemacs users believe they have noticed some difference, 
> but I am not convinced.

I tried this and indeed it revealed issues...

With

(defmacro org-require-with-shadowcheck (feature)
  "Load FEATURE making sure that it is loaded using current `load-path'.
When FEATURE is not yet loaded, act like `require' does.  When FEATURE
is loaded, but not consistent with the current value of `load-path',
re-load it."
  `(eval-and-compile
     (if (not (featurep ,feature)) (require ,feature)
       (let ((file (locate-library (symbol-name ,feature))))
         (unless (equal (feature-file ,feature) file)
	   ;; The feature is already provided, but from some other file
	   ;; than expected.  Presumably because `load-path` has been
	   ;; changed since the file was loaded.  This is likely a sign
	   ;; that we're loading a mix of files from different versions.
	   ;; That can spell trouble.
           (load file nil t t))))))

using https://git.sr.ht/~yantar92/org-mode/tree/feature/shadowcheck

1. emacs-30 -Q
2. M-x org-version
3. M-x package-install-file <RET> /path/to/org/lisp

I am getting

ob-C.el:40:2: Error: Eager macro-expansion failure: (error "Eager macro-expansion skipped due to cycle:
  … => (load \"org-element.el\") => (macroexpand-all (org-require-with-shadowcheck 'org …)) => (macroexpand (org-require-with-shadowcheck …)) => (load \"org.el\") => (load \"org-element.el\")")

which is rather curious since org.el is not supposed to load
org-element.el

I feel that I am missing something obvious.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62762: 'make' often errors with "Org version mismatch" after pulling a new version of the code

[Max Nikulin]

On 06/05/2023 01:17, Stefan Monnier wrote:
> I'll refrain from discussing this here.  Incremental recompilation
> problems should be discussed in another bug report, IMO.

I will try to explain why this bug report was caused by issues with 
incremental builds.

Original report:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62762#5

> Could someone fix that, so that I don't have to 'make bootstrap' every 
> other time after a 'git pull'?
> 
> make[3]: Entering directory '/home/dgutov/vc/emacs-master/lisp'
>   ELC      org/ox.elc
> Warning (emacs): Org version mismatch.  Org loading aborted.

It may be a *valid* error, however on minor Org updates it is likely a 
false positive.

ox.el has a number of `require's. Byte compilation loaded an .elc file 
before it was recompiled with new definition of a macro. Notice that an 
.elc file may become stale even if the corresponding .el file has not 
changed. The .el file may require macros from other .el (.elc) files and 
expand them during compiling.

So incremental build must compile files in certain order. It must take 
into account dependencies between .elc and .el files and prevent loading 
of stale .elc files.

`load-prefer-newer' can be a workaround only for direct "%.elc: %.el" 
dependencies. Transitional dependencies still may cause troubles.

Let's assume that a trouble with cyclic dependencies is a real one (I 
have not convinced in that). A way to the correct builds is to 
*completely* avoid loading of .elc files during incremental builds. When 
only a few files changed then relying purely on .el files should cause 
significant performance penalty. Unfortunately it would not work without 
describing at least some dependencies. In the `org-assert-version' they 
should be at least that all lisp/org/*.elc files depend on 
lisp/org/org-version.el and lisp/org/org-macs.el. Without automatic 
dependency generation it is a kludge, but it should significantly 
alleviate the issue. Grepping org sources for defmacro and adding e.g. 
org-element.el dependency will make incremental builds with stale macro 
definitions even more rare.

Commits pushed so far trade false positives to false negatives and to 
reports of bugs due to "undefined" functions and incorrect signatures to 
Org developers and maintainers.

Perhaps `org-assert-version' may be improved, but this report was caused 
by broken build rules.

bug#62762: Incremental builds and Lisp files dependencies pulling a new version of the code

[Eli Zaretskii]

> Date: Sat, 6 May 2023 13:00:38 +0700
> Cc: Ihor Radchenko <yantar92@posteo.net>, Eli Zaretskii <eliz@gnu.org>,
>  bzg@gnu.org, dmitry@gutov.dev, 62762@debbugs.gnu.org,
>  Alan Mackenzie <acm@muc.de>
> From: Max Nikulin <manikulin@gmail.com>
> 
> On 06/05/2023 01:17, Stefan Monnier wrote:
> > I'll refrain from discussing this here.  Incremental recompilation
> > problems should be discussed in another bug report, IMO.
> 
> I will try to explain why this bug report was caused by issues with 
> incremental builds.

I've started a new thread, because I agree with Stefan: this is a
separate issue.  Please reply with this new thread's Subject.

> Let's assume that a trouble with cyclic dependencies is a real one (I 
> have not convinced in that). A way to the correct builds is to 
> *completely* avoid loading of .elc files during incremental builds. When 
> only a few files changed then relying purely on .el files should cause 
> significant performance penalty.

Exactly.  And please don't forget that quite a few files are
preloaded, so using only *.el means to preload only the *.el files,
which will result in a significantly slower bootstrap-emacs binary,
and the build will become annoyingly slow, for no good reason.

> Unfortunately it would not work without 
> describing at least some dependencies. In the `org-assert-version' they 
> should be at least that all lisp/org/*.elc files depend on 
> lisp/org/org-version.el and lisp/org/org-macs.el. Without automatic 
> dependency generation it is a kludge, but it should significantly 
> alleviate the issue.

This doesn't really work, not as long as we use Make: some *.elc files
might not exist yet, for whatever reasons, so Make will try to
generate them first, thus disrupting the order of generation that you
will try to encode in the dependencies.  We already tried that, and it
didn't work, not even just for Org.

> Commits pushed so far trade false positives to false negatives and to 
> reports of bugs due to "undefined" functions and incorrect signatures to 
> Org developers and maintainers.

Those commits only affect byte compilation of Org as part of building
Emacs, so I see no reason why they should cause the above issues for
Orge developers.

> Perhaps `org-assert-version' may be improved, but this report was caused 
> by broken build rules.

The bug#62762 was reported because Org behaved unlike any other Lisp
package in Emacs.  We have other packages that define macros used in
many other *.el files, and none of them does the version-check like
Org did.  The problems with making incompatible changes in one or more
of the macros are well known to Emacs developers who rebuild Emacs
frequently, and we have ways to deal with them.  Some of those ways
(the simplest ones) are encoded in the top-level Makefile, see
ADVICE-ON-FAILURE there.  There are others, more subtle ones, for
those, like myself, who don't bootstrap, ever, except the first time a
fresh repository is cloned.

That Org behaved in a stark different manner was therefore an
unpleasant annoyance, so we now fixed that by making Org behave in
this regard like any other Lisp package in Emacs -- but only when
Emacs is built, because the knob which forces this behavior is off by
default, and is only turned on by lisp/Makefile in the Emacs tree.

bug#62762: Incremental builds and Lisp files dependencies pulling a new version of the code

[Max Nikulin]

On 06/05/2023 14:51, Eli Zaretskii wrote:
> This doesn't really work, not as long as we use Make: some *.elc files
> might not exist yet, for whatever reasons, so Make will try to
> generate them first, thus disrupting the order of generation that you
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> will try to encode in the dependencies.  We already tried that, and it
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> didn't work, not even just for Org.

Could you, please, be more precise? Do you suspect that Make may start 
building a target before completion of all rules for its prerequisites? 
It sounds like a grave bug in Make.

bug#62762: Incremental builds and Lisp files dependencies pulling a new version of the code

[Eli Zaretskii]

> Date: Sat, 6 May 2023 22:50:22 +0700
> Cc: monnier@iro.umontreal.ca, yantar92@posteo.net, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> On 06/05/2023 14:51, Eli Zaretskii wrote:
> > This doesn't really work, not as long as we use Make: some *.elc files
> > might not exist yet, for whatever reasons, so Make will try to
> > generate them first, thus disrupting the order of generation that you
>                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > will try to encode in the dependencies.  We already tried that, and it
>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > didn't work, not even just for Org.
> 
> Could you, please, be more precise? Do you suspect that Make may start 
> building a target before completion of all rules for its prerequisites? 

No, I mean that the order in which Make builds the targets might not
be what you think, if all you tell it that all org/*.elc files depend
on org-macs.elc.

bug#62762: Incremental builds and Lisp files dependencies pulling a new version of the code

[Max Nikulin]

On 06/05/2023 23:26, Eli Zaretskii wrote:
>> Date: Sat, 6 May 2023 22:50:22 +0700 From: Max Nikulin
>>
>> On 06/05/2023 14:51, Eli Zaretskii wrote:
>>> This doesn't really work, not as long as we use Make: some *.elc files
>>> might not exist yet, for whatever reasons, so Make will try to
>>> generate them first, thus disrupting the order of generation that you
>>                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> will try to encode in the dependencies.  We already tried that, and it
>>          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> didn't work, not even just for Org.
>>
>> Could you, please, be more precise? Do you suspect that Make may start
>> building a target before completion of all rules for its prerequisites?
> 
> No, I mean that the order in which Make builds the targets might not
> be what you think, if all you tell it that all org/*.elc files depend
> on org-macs.elc.

Sorry, but it is still not clear to we what "doesn't really work" and I 
feel that it may be important for further discussion.

If a Makefile contains a dependency like

org/ol.elc: org/org-macs.elc

and org/org-macs.elc needs update then the rule for org/ol.elc is 
executed after completion of the rule for org/org-macs.elc. The purpose 
of dependencies is to define order of execution of build rules.

However I suggested dependencies on org-macs.el, not on org-macs.elc:

Sat, 6 May 2023 13:00:38 +0700
>> they 
>> should be at least that all lisp/org/*.elc files depend on 
>> lisp/org/org-version.el and lisp/org/org-macs.el.

bug#62762: Incremental builds and Lisp files dependencies pulling a new version of the code

[Eli Zaretskii]

> Date: Sun, 7 May 2023 09:45:20 +0700
> Cc: monnier@iro.umontreal.ca, yantar92@posteo.net, bzg@gnu.org,
>  dmitry@gutov.dev, 62762@debbugs.gnu.org, acm@muc.de
> From: Max Nikulin <manikulin@gmail.com>
> 
> On 06/05/2023 23:26, Eli Zaretskii wrote:
> >> Date: Sat, 6 May 2023 22:50:22 +0700 From: Max Nikulin
> >>
> > No, I mean that the order in which Make builds the targets might not
> > be what you think, if all you tell it that all org/*.elc files depend
> > on org-macs.elc.
> 
> Sorry, but it is still not clear to we what "doesn't really work" and I 
> feel that it may be important for further discussion.
> 
> If a Makefile contains a dependency like
> 
> org/ol.elc: org/org-macs.elc
> 
> and org/org-macs.elc needs update then the rule for org/ol.elc is 
> executed after completion of the rule for org/org-macs.elc. The purpose 
> of dependencies is to define order of execution of build rules.

My point was that with the above rule, if org-macs.elc doesn't exist,
Make will want to produce it, even though for your purposes there's no
reason to do that.

> However I suggested dependencies on org-macs.el, not on org-macs.elc:
> 
> Sat, 6 May 2023 13:00:38 +0700
> >> they 
> >> should be at least that all lisp/org/*.elc files depend on 
> >> lisp/org/org-version.el and lisp/org/org-macs.el.

I don't really understand in enough detail what is it that you are
suggesting, because you show your suggestion piecemeal.  Can you
describe the suggestion in its entirety, top to bottom, including all
the details?  In particular, since the proposed automatic dependency
generation will be side effect of byte-compilation (right?), how do
you make sure the compilation is in the correct order when
dependencies change due to changes in the source *.el files?

Please also consider parallel builds, something that almost all of the
Emacs developers use all the time.  Thus, if N files depend on foo.el,
when foo.el changes Make will want to run N parallel jobs compiling
each one of the N dependencies.

In addition, some of the *.el files are generated, so they change as
result of changes in other *.el files.  AFAIK, automatic dependency
generation doesn't usually deal with such cases, it only deals with
dependencies on files that are not regenerated as part of the build.
How do you suggest to handle this?

And finally, some of the Lisp files are preloaded, so their changes,
and the changes in their *.elc files as result of changes in their
prerequisites, affect the Emacs binary that is used to byte-compile
the Lisp files.  If the preloaded files are outdated, many
compilations might fail just because of this aspect.

If you have an idea for solving all of these problems (which are
really just the initial tips of the iceberg), please describe your
idea with all of its details.  Otherwise, this discussion is not
really useful.