Permanently fix org versioning breakage during builds?

Permanently fix org versioning breakage during builds?

[T.V Raman]

This version mismatch breakage from org started happening again;
appears to rear its ugly head every few weeks, with the fix being to
clean *.elc files in org. Could we build this know-how into the build
rules, at present, the result is a fairly broken experience if you
load org in your startup.
-- 

-- 

Re: Permanently fix org versioning breakage during builds?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 897 bytes --]

On Fri, Dec 22, 2023, 18:59 T.V Raman <raman@google.com> wrote:

> This version mismatch breakage from org started happening again;
> appears to rear its ugly head every few weeks, with the fix being to
> clean *.elc files in org. Could we build this know-how into the build
> rules, at present, the result is a fairly broken experience if you
> load org in your startup.
>

+1

The message is extremely confusing, and all I remember seeing are
references to "straight" which is a package manager not even in Emacs. At a
minimum, the correct "delete elc" trick should be there.

Another way to fix this issue is for the org developers to not deal with
macro expansion-site recompilation problems so aggressively, and making
more use of the "call-with" idiom whereby a macro immediately expands to a
function that has all the frequently-changing implementation details.

João

[-- Attachment #2: Type: text/html, Size: 1372 bytes --]

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

> Another way to fix this issue is for the org developers to not deal with
> macro expansion-site recompilation problems so aggressively, and making
> more use of the "call-with" idiom whereby a macro immediately expands to a
> function that has all the frequently-changing implementation details.

May you elaborate?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1374 bytes --]

LOn Sat, Dec 23, 2023, 18:03 Ihor Radchenko <yantar92@posteo.net> wrote:

> João Távora <joaotavora@gmail.com> writes:
>
> > Another way to fix this issue is for the org developers to not deal with
> > macro expansion-site recompilation problems so aggressively, and making
> > more use of the "call-with" idiom whereby a macro immediately expands to
> a
> > function that has all the frequently-changing implementation details.
>
> May you elaborate?
>

I have already elaborated a while back, but on a tangent deep down in a
thread and I'm on my phone right now. So maybe these links would give you
an idea of what it looks like.

http://random-state.net/log/3390120648.html
https://www.reddit.com/r/Common_Lisp/comments/okvgf0/examples_of_callwith_style_in_macros/
https://news.ycombinator.com/item?id=12476032

These are all common lisp related, but there are, I'm fairly sure, numerous
examples in Elisp, though not necessarily named "call-with". Let me know
what you understand of the examples before we continue.

You may also want to elaborate yourself on an example of a macro that is
emblematic of the expansion-site recompilation problem that, as I contend,
you fixed somewhat heavy-handedly and lead to these recurring irritations
to many users. We can see if we can fix that example with with the
technique I allude to.

João

[-- Attachment #2: Type: text/html, Size: 2417 bytes --]

Re: Permanently fix org versioning breakage during builds?

[T.V Raman]

But while this gets discussed and investigated, please add a make rule
that just forces a clean rebuild of org, at least in the Emacs tree --
-- 

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: João Távora <joaotavora@gmail.com>,  emacs-devel
>  <emacs-devel@gnu.org>
> Date: Sat, 23 Dec 2023 18:43:29 -0800
> 
> But while this gets discussed and investigated, please add a make rule
> that just forces a clean rebuild of org, at least in the Emacs tree --

We tried, and it didn't work well.  The problem is not as simple to
fix by Make rules as it seems at the first sight.

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

"T.V Raman" <raman@google.com> writes:

> But while this gets discussed and investigated, please add a make rule
> that just forces a clean rebuild of org, at least in the Emacs tree --

I now converted that error into warning, so that it is no longer
blocking.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

>> May you elaborate?
>>
>
> I have already elaborated a while back, but on a tangent deep down in a
> thread and I'm on my phone right now. So maybe these links would give you
> an idea of what it looks like.
>
> http://random-state.net/log/3390120648.html
> https://www.reddit.com/r/Common_Lisp/comments/okvgf0/examples_of_callwith_style_in_macros/
> https://news.ycombinator.com/item?id=12476032
>
> These are all common lisp related, but there are, I'm fairly sure, numerous
> examples in Elisp, though not necessarily named "call-with". Let me know
> what you understand of the examples before we continue.

I do not think that your idea will work.
What you propose is compiling `org-assert-version' once.
However, the whole point behind `org-assert-version' is that it should
be compiled into every .elc file in Org:

   (equal (org-release) ,(org-release))

will only work when ,(org-release) is expanded at compile file thus
recording the Org version used during the compilation inside the
resulting .elc file. This way, we can compare Org version active in
Emacs session vs. Org version used for the compilation.

> You may also want to elaborate yourself on an example of a macro that is
> emblematic of the expansion-site recompilation problem that, as I contend,
> you fixed somewhat heavy-handedly and lead to these recurring irritations
> to many users. We can see if we can fix that example with with the
> technique I allude to.

May you please try to explain what you mean in other words? I am not
sure what you are trying to convey in this paragraph.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> I now converted that error into warning, so that it is no longer
> blocking.

Thanks, but ideally Emacs should not display warnings loading built-in
files, however they were built, unless a definite problem exists that
requires they be compiled afresh.  Org cannot change so dramatically
from one moment to the next, as the frequency of these warnings would
suggest, that each warning or error it signals is justified.

What do you say to disabling the mechanism for version checks in Org as
distributed with Emacs?  Nothing will be lost, since AFAIU the check
exists for the sake of conflicting Org installations set in place by
package managers rather than Emacs itself.

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 1393 bytes --]

Po Lu <luangruo@yahoo.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> I now converted that error into warning, so that it is no longer
>> blocking.
>
> Thanks, but ideally Emacs should not display warnings loading built-in
> files, however they were built, unless a definite problem exists that
> requires they be compiled afresh.  Org cannot change so dramatically
> from one moment to the next, as the frequency of these warnings would
> suggest, that each warning or error it signals is justified.

Yes, Org cannot change so dramatically. However, when Org does change
its macros in breaking ways (for example, to fix a bug), it is very hard
to figure out what goes wrong in the case when we get a mix of old and
new .elc files. That's why `org-assert-version' was written in the first
place.

> What do you say to disabling the mechanism for version checks in Org as
> distributed with Emacs?  Nothing will be lost, since AFAIU the check
> exists for the sake of conflicting Org installations set in place by
> package managers rather than Emacs itself.

Considering the existing stance that it is ok to break things
occasionally due to stale macros when running make on Emacs git
repository, I see no problem disabling these checks for built-in Org
mode.

Tentatively, it may even be done without affecting version check for
out-of-tree builds. See the attached patch.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-assert-version-Disable-version-check-in-.elc-fil.patch --]
[-- Type: text/x-patch, Size: 1375 bytes --]

From eadeaa85336e01a5c3d5ac3d5088a83d627d1fc7 Mon Sep 17 00:00:00 2001
Message-ID: <eadeaa85336e01a5c3d5ac3d5088a83d627d1fc7.1703422144.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sun, 24 Dec 2023 13:47:04 +0100
Subject: [PATCH] org-assert-version: Disable version check in .elc files for
 built-in Org mode

* lisp/org-macs.el (org-assert-version): Record
`org--inhibit-version-check' during compile time inside
`org-assert-version'.  This way, .elc files generated when compiling
built-in Org mode in Emacs git tree never trigger version mismatch
warning.

Link: https://yhetil.org/emacs-devel/87zfxzdbna.fsf@yahoo.com
---
 lisp/org-macs.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/org-macs.el b/lisp/org-macs.el
index 10ef9d06b..fd083ca62 100644
--- a/lisp/org-macs.el
+++ b/lisp/org-macs.el
@@ -56,7 +56,7 @@ (defmacro org-assert-version ()
   ;; `org-assert-version' calls would fail using strict
   ;; `org-git-version' check because the generated Org version strings
   ;; will not match.
-  `(unless (or org--inhibit-version-check (equal (org-release) ,(org-release)))
+  `(unless (or ,org--inhibit-version-check (equal (org-release) ,(org-release)))
      (warn "Org version mismatch.
 This warning usually appears when a built-in Org version is loaded
 prior to the more recent Org version.
-- 
2.42.0


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> Yes, Org cannot change so dramatically. However, when Org does change
> its macros in breaking ways (for example, to fix a bug), it is very hard
> to figure out what goes wrong in the case when we get a mix of old and
> new .elc files. That's why `org-assert-version' was written in the first
> place.

Indeed.  As regards breakage resulting from the accumulation of changes
incremental compilation cannot account for, our practice has always been
to wait for the user to discover the failures and resolve them on his
own.

> Considering the existing stance that it is ok to break things
> occasionally due to stale macros when running make on Emacs git
> repository, I see no problem disabling these checks for built-in Org
> mode.
>
> Tentatively, it may even be done without affecting version check for
> out-of-tree builds. See the attached patch.

If it works, please install.  I'm afraid I can't test it.  Thanks in
advance.

Re: Permanently fix org versioning breakage during builds?

[Stefan Kangas]

Ihor Radchenko <yantar92@posteo.net> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> What do you say to disabling the mechanism for version checks in Org as
>> distributed with Emacs?  Nothing will be lost, since AFAIU the check
>> exists for the sake of conflicting Org installations set in place by
>> package managers rather than Emacs itself.
>
> Considering the existing stance that it is ok to break things
> occasionally due to stale macros when running make on Emacs git
> repository, I see no problem disabling these checks for built-in Org
> mode.
>
> Tentatively, it may even be done without affecting version check for
> out-of-tree builds. See the attached patch.

Could you explain the benefit of doing that?  Is the problem simply that
some users building from emacs.git don't want to say "make bootstrap",
or is it something else?

What will be the consequences for everyone else, if we were to install
the proposed change?

Re: Permanently fix org versioning breakage during builds?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2418 bytes --]

On Sun, Dec 24, 2023, 11:47 Ihor Radchenko <yantar92@posteo.net> wrote:

> João Távora <joaotavora@gmail.com> writes:
>
> >> May you elaborate?
> >>
> >
> > I have already elaborated a while back, but on a tangent deep down in a
> > thread and I'm on my phone right now. So maybe these links would give you
> > an idea of what it looks like.
> >
> > http://random-state.net/log/3390120648.html
> >
> https://www.reddit.com/r/Common_Lisp/comments/okvgf0/examples_of_callwith_style_in_macros/
> > https://news.ycombinator.com/item?id=12476032
> >
> > These are all common lisp related, but there are, I'm fairly sure,
> numerous
> > examples in Elisp, though not necessarily named "call-with". Let me know
> > what you understand of the examples before we continue.
>
> I do not think that your idea will work.
> What you propose is compiling `org-assert-version' once.
>

Hmm? How can that be what I'm proposing if I didn't even know about this
definition. Is it a form, a macro?

What ultimate problem is it solving? What condition do you need to assert
at Emacs master build-time, failing which something else will go wrong?

I just provided ideas on how to solve a very common build-time pitfall in
Lisp. A pitfall that can be solved by requiring a recompilation of
everything, as seems to be the current way, or in other less brutal ways.

Keep in mind something in the build system is causing builds to fail with
cryptic messages even for people who don't use Org or rarely ever do. I've
never touched any Org related files in my life, why should it blow up in my
face? Of course happens to all package maintainers but usually there is a
fix. There should be one here, too.

> You may also want to elaborate yourself on an example of a macro that is
> > emblematic of the expansion-site recompilation problem that, as I
> contend,
> > you fixed somewhat heavy-handedly and lead to these recurring irritations
> > to many users. We can see if we can fix that example with with the
> > technique I allude to.
>
> May you please try to explain what you mean in other words? I am not
> sure what you are trying to convey in this paragraph.
>

Give me (us) an example of a defmacro form whose body is frequently changed
and which requires the expanders of such a macro, which presumably live in
other files, to be recompiled, even if said files haven't changed.

[-- Attachment #2: Type: text/html, Size: 4135 bytes --]

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

>> I do not think that your idea will work.
>> What you propose is compiling `org-assert-version' once.
>>
>
> Hmm? How can that be what I'm proposing if I didn't even know about this
> definition. Is it a form, a macro?

> What ultimate problem is it solving? What condition do you need to assert
> at Emacs master build-time, failing which something else will go wrong?

`org-assert-version' is a macro.
It is solving a very common problem when some files from Org mode are
loaded from Emacs distribution and some are loaded from a newer Org mode
version typically installed from ELPA. This causes various unexpected
breakages.

Or, similarly, when built-in Org mode is updated, some macros are
changed, but their users are not re-compiled, leaving stale macro
expansions inside .elc files. Again, this causes various breakages.

`org-assert-version' macro makes sure that no mixing like the above is
happening.

> I just provided ideas on how to solve a very common build-time pitfall in
> Lisp. A pitfall that can be solved by requiring a recompilation of
> everything, as seems to be the current way, or in other less brutal ways.

> Keep in mind something in the build system is causing builds to fail with
> cryptic messages even for people who don't use Org or rarely ever do. I've
> never touched any Org related files in my life, why should it blow up in my
> face? Of course happens to all package maintainers but usually there is a
> fix. There should be one here, too.

Nothing is blown up in your face. The version breakage we are discussing
in this thread is at runtime, when Org mode is loaded and detects that
some of .elc files for Org libraries are stale (compiled using older Org
mode version).

>> May you please try to explain what you mean in other words? I am not
>> sure what you are trying to convey in this paragraph.
>>
>
> Give me (us) an example of a defmacro form whose body is frequently changed
> and which requires the expanders of such a macro, which presumably live in
> other files, to be recompiled, even if said files haven't changed.

I hope that the above clarified the situation.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

Stefan Kangas <stefankangas@gmail.com> writes:

>> Tentatively, it may even be done without affecting version check for
>> out-of-tree builds. See the attached patch.
>
> Could you explain the benefit of doing that?  Is the problem simply that
> some users building from emacs.git don't want to say "make bootstrap",
> or is it something else?

AFAIU, yes - some users do not want to do make bootstrap and do not like
that Org mode complains about mixed versions of .elc files.

> What will be the consequences for everyone else, if we were to install
> the proposed change?

Org mode may break in unexpected ways due to stale .elc that did not
update their macro expansions.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 24 Dec 2023 14:13:26 +0000
> Cc: "T.V Raman" <raman@google.com>, emacs-devel <emacs-devel@gnu.org>
> 
> Keep in mind something in the build system is causing builds to fail with cryptic messages even for
> people who don't use Org or rarely ever do. I've never touched any Org related files in my life, why
> should it blow up in my face?

Building Emacs requires to build several manuals whose source is in
Org, and thus you need a working Org, otherwise the build will fail.
IOW, even if you personally don't visit Org files and don't use any
Org feature, the build does.

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Po Lu <luangruo@yahoo.com>, "T.V Raman" <raman@google.com>,
>  João Távora
>  <joaotavora@gmail.com>, emacs-devel <emacs-devel@gnu.org>
> Date: Sun, 24 Dec 2023 14:51:48 +0000
> 
> Stefan Kangas <stefankangas@gmail.com> writes:
> 
> > What will be the consequences for everyone else, if we were to install
> > the proposed change?
> 
> Org mode may break in unexpected ways due to stale .elc that did not
> update their macro expansions.

Yes.  That could happen even as part of the build, if the manuals
whose sources are *.org files need to be rebuilt.

But such breakage happens at times anyway, when some macros change in
incompatible ways and files that have the old versions of those macros
compiled into them are not recompiled.  It isn't anything new for
people who regularly update from Git and rebuild Emacs.

Re: Permanently fix org versioning breakage during builds?

[T.V Raman]

\rm lisp/org/*.elc is what I added to the shell script that I use to
build Emacs.


Eli Zaretskii writes:
 > > From: "T.V Raman" <raman@google.com>
 > > Cc: João Távora <joaotavora@gmail.com>,  emacs-devel
 > >  <emacs-devel@gnu.org>
 > > Date: Sat, 23 Dec 2023 18:43:29 -0800
 > > 
 > > But while this gets discussed and investigated, please add a make rule
 > > that just forces a clean rebuild of org, at least in the Emacs tree --
 > 
 > We tried, and it didn't work well.  The problem is not as simple to
 > fix by Make rules as it seems at the first sight.

-- 

Re: Permanently fix org versioning breakage during builds?

[T.V Raman]

make bootstrap is a far heavier hammer than \rm lisp/org/*.elc


Stefan Kangas writes:
 > Ihor Radchenko <yantar92@posteo.net> writes:
 > 
 > > Po Lu <luangruo@yahoo.com> writes:
 > >
 > >> What do you say to disabling the mechanism for version checks in Org as
 > >> distributed with Emacs?  Nothing will be lost, since AFAIU the check
 > >> exists for the sake of conflicting Org installations set in place by
 > >> package managers rather than Emacs itself.
 > >
 > > Considering the existing stance that it is ok to break things
 > > occasionally due to stale macros when running make on Emacs git
 > > repository, I see no problem disabling these checks for built-in Org
 > > mode.
 > >
 > > Tentatively, it may even be done without affecting version check for
 > > out-of-tree builds. See the attached patch.
 > 
 > Could you explain the benefit of doing that?  Is the problem simply that
 > some users building from emacs.git don't want to say "make bootstrap",
 > or is it something else?
 > 
 > What will be the consequences for everyone else, if we were to install
 > the proposed change?

-- 

Re: Permanently fix org versioning breakage during builds?

[João Távora]

On Sun, Dec 24, 2023 at 2:45 PM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> >> I do not think that your idea will work.
> >> What you propose is compiling `org-assert-version' once.
> >>
> >
> > Hmm? How can that be what I'm proposing if I didn't even know about this
> > definition. Is it a form, a macro?
>
> > What ultimate problem is it solving? What condition do you need to assert
> > at Emacs master build-time, failing which something else will go wrong?
>
> `org-assert-version' is a macro.
> It is solving a very common problem when some files from Org mode are
> loaded from Emacs distribution and some are loaded from a newer Org mode
> version typically installed from ELPA. This causes various unexpected
> breakages.

OK, what thing does `org-assert-version` assert and what is one concrete
example of a breakage.  Can this unexpected breakage exist if someone
has never ever installed Org mode from ELPA?

> Or, similarly, when built-in Org mode is updated, some macros are
> changed, but their users are not re-compiled, leaving stale macro
> expansions inside .elc files. Again, this causes various breakages.

Well, this is a problem that is very commonly solved by the technique
I described.  But without an example of one member of this "some macros"
set and the example of one of its users in some other file, it's hard
to justify your previous assertion that "the idea will not work".

Maybe it will, maybe it won't, or maybe it would but it would be
unpractical for some reason (say, if there are hundred such macros,
or if they're created by a macro-generating macro, or whatever: macros
can be hairy).

For example, the technique I described earlier cannot generally solve
a very similar problem with DEFSTRUCT (in our case cl-defstruct), because
all the slot accessors are functions generated with cl-defstruct which have
compiler macros and removing the compiler macros would negate the dispatching
speedup which the (only?) reason to choose structs over classes.

But we won't know any of this until we study the problem with some
concrete code.

> `org-assert-version' macro makes sure that no mixing like the above is
> happening.
>
> > I just provided ideas on how to solve a very common build-time pitfall in
> > Lisp. A pitfall that can be solved by requiring a recompilation of
> > everything, as seems to be the current way, or in other less brutal ways.
>
> > Keep in mind something in the build system is causing builds to fail with
> > cryptic messages even for people who don't use Org or rarely ever do. I've
> > never touched any Org related files in my life, why should it blow up in my
> > face? Of course happens to all package maintainers but usually there is a
> > fix. There should be one here, too.
>
> Nothing is blown up in your face. The version breakage we are discussing
> in this thread is at runtime, when Org mode is loaded and detects that
> some of .elc files for Org libraries are stale (compiled using older Org
> mode version).

I've seen this problem tens of times, even fairly recently, when doing my
usual git pull --rebase && make routine.  It's especially annoying during a
git bisect.  It's only solved by `make bootstrap` or `rm -rf lisp/org/*.elc`.

Is this problem 100% fully solved?

> I hope that the above clarified the situation.

No, it didn't, I'm afraid.  If you know one of these macros (you alluded
earlier to "some macros"), type its name here, please.

João

Re: Permanently fix org versioning breakage during builds?

[T.V Raman]

Also, if the problem org is running into is the result of conflicts
between org files bundled in Emacs vs those coming from an elpa
install: one way to break this knot might be:

1. Files from core Emacs should be considered the truth and stable and
   Emacs should not throw an error as long as user installs nothing
   org from elpa.

   2. If user does install from elpa, those org files are hopefully
      newer -- and that is where the version check should be thrown.
      3. If a user installing from elpa leaves around old org files,
         ie older than what is bundled in emacs, then that user
         deserves the consequences of the breakage with a hard error.

         

João Távora writes:
 > On Sun, Dec 24, 2023 at 2:45 PM Ihor Radchenko <yantar92@posteo.net> wrote:
 > >
 > > João Távora <joaotavora@gmail.com> writes:
 > >
 > > >> I do not think that your idea will work.
 > > >> What you propose is compiling `org-assert-version' once.
 > > >>
 > > >
 > > > Hmm? How can that be what I'm proposing if I didn't even know about this
 > > > definition. Is it a form, a macro?
 > >
 > > > What ultimate problem is it solving? What condition do you need to assert
 > > > at Emacs master build-time, failing which something else will go wrong?
 > >
 > > `org-assert-version' is a macro.
 > > It is solving a very common problem when some files from Org mode are
 > > loaded from Emacs distribution and some are loaded from a newer Org mode
 > > version typically installed from ELPA. This causes various unexpected
 > > breakages.
 > 
 > OK, what thing does `org-assert-version` assert and what is one concrete
 > example of a breakage.  Can this unexpected breakage exist if someone
 > has never ever installed Org mode from ELPA?
 > 
 > > Or, similarly, when built-in Org mode is updated, some macros are
 > > changed, but their users are not re-compiled, leaving stale macro
 > > expansions inside .elc files. Again, this causes various breakages.
 > 
 > Well, this is a problem that is very commonly solved by the technique
 > I described.  But without an example of one member of this "some macros"
 > set and the example of one of its users in some other file, it's hard
 > to justify your previous assertion that "the idea will not work".
 > 
 > Maybe it will, maybe it won't, or maybe it would but it would be
 > unpractical for some reason (say, if there are hundred such macros,
 > or if they're created by a macro-generating macro, or whatever: macros
 > can be hairy).
 > 
 > For example, the technique I described earlier cannot generally solve
 > a very similar problem with DEFSTRUCT (in our case cl-defstruct), because
 > all the slot accessors are functions generated with cl-defstruct which have
 > compiler macros and removing the compiler macros would negate the dispatching
 > speedup which the (only?) reason to choose structs over classes.
 > 
 > But we won't know any of this until we study the problem with some
 > concrete code.
 > 
 > > `org-assert-version' macro makes sure that no mixing like the above is
 > > happening.
 > >
 > > > I just provided ideas on how to solve a very common build-time pitfall in
 > > > Lisp. A pitfall that can be solved by requiring a recompilation of
 > > > everything, as seems to be the current way, or in other less brutal ways.
 > >
 > > > Keep in mind something in the build system is causing builds to fail with
 > > > cryptic messages even for people who don't use Org or rarely ever do. I've
 > > > never touched any Org related files in my life, why should it blow up in my
 > > > face? Of course happens to all package maintainers but usually there is a
 > > > fix. There should be one here, too.
 > >
 > > Nothing is blown up in your face. The version breakage we are discussing
 > > in this thread is at runtime, when Org mode is loaded and detects that
 > > some of .elc files for Org libraries are stale (compiled using older Org
 > > mode version).
 > 
 > I've seen this problem tens of times, even fairly recently, when doing my
 > usual git pull --rebase && make routine.  It's especially annoying during a
 > git bisect.  It's only solved by `make bootstrap` or `rm -rf lisp/org/*.elc`.
 > 
 > Is this problem 100% fully solved?
 > 
 > > I hope that the above clarified the situation.
 > 
 > No, it didn't, I'm afraid.  If you know one of these macros (you alluded
 > earlier to "some macros"), type its name here, please.
 > 
 > João

-- 

Re: Permanently fix org versioning breakage during builds?

[João Távora]

On Sun, Dec 24, 2023 at 4:32 PM T.V Raman <raman@google.com> wrote:
>
> make bootstrap is a far heavier hammer than \rm lisp/org/*.elc

Indeed.  On Windows, the configure script takes ages (especially if
you don't have the luxury of skipping nasty antivirus checks)

And even rm lisp/org/*.elc is a fairly heavy hammer.  org takes a
long time to compile.

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Date: Sun, 24 Dec 2023 08:29:45 -0800
> Cc: raman@google.com,
>     yantar92@posteo.net,
>     joaotavora@gmail.com,
>     emacs-devel@gnu.org
> 
> \rm lisp/org/*.elc is what I added to the shell script that I use to
> build Emacs.

That's not a Make rule.  A Make rule would only do that when it needs
to be done, and detecting when that happens is the non-trivial part.

If you are willing to pay the price of recompiling all Org files on
every build (not everyone wants that), then you already have your
solution.

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 24 Dec 2023 16:39:54 +0000
> Cc: stefankangas@gmail.com, yantar92@posteo.net, luangruo@yahoo.com, 
>  emacs-devel@gnu.org
> 
> On Windows, the configure script takes ages

Not if you use -C.  Then it flies (except for the very first time you
build a fresh clone).

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Org mode may break in unexpected ways due to stale .elc that did not
>> update their macro expansions.
>
> Yes.  That could happen even as part of the build, if the manuals
> whose sources are *.org files need to be rebuilt.
>
> But such breakage happens at times anyway, when some macros change in
> incompatible ways and files that have the old versions of those macros
> compiled into them are not recompiled.  It isn't anything new for
> people who regularly update from Git and rebuild Emacs.

The main problem is that with org-assert-version, "from time to time"
becomes more frequent as the breakage is guaranteed every time a new
version of Org is installed into Emacs tree.

Since we release a new minor Org version every few weeks, the breakage
happens every few weeks.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Date: Sun, 24 Dec 2023 08:36:52 -0800
> Cc: yantar92@posteo.net,
>     raman@google.com,
>     emacs-devel@gnu.org
> 
> 
>    2. If user does install from elpa, those org files are hopefully
>       newer -- and that is where the version check should be thrown.

That doesn't really work, since each "git pull" that brings modified
Org files makes those files "newer" as far as Make is concerned, since
their time stamps are set to the "git pull" time.

Re: Permanently fix org versioning breakage during builds?

[João Távora]

On Sun, Dec 24, 2023 at 4:32 PM João Távora <joaotavora@gmail.com> wrote:

>
> No, it didn't, I'm afraid.  If you know one of these macros (you alluded
> earlier to "some macros"), type its name here, please.

IIn the absence of examples (so far), I went looking for one myself.
I found 'org-export-with-buffer-copy' in lisp/org/ox.el.  It's expanded
in ox-html.el and ox-publish.el in the same directory.

As far as I can see, it falls right into the category of
expansion-site-recompilation problem that may happen if the macro's
implementation details, which are reasonably lengthy, are changed
in ox.el.  If ox-html.el and ox-publish.el aren't changed as well,
our build system will fail to recompile them and Org will malfunction.

[  Tangent: one thing I notice is that the expansion in ox-html.el is
buggy.  It evaluates the keyword arguments to the macros multiple times
for no effect.  The solution would be to change the macro calling
convention to be:

(cl-defmacro org-export-with-buffer-copy ((&key to-buffer drop-visibility
                                          drop-narrowing drop-contents
                                          drop-locals
                                          &allow-other-keys) &body body)
  ...)


But this is not the problem in this thread, of course (though I do
heavily recommend doing so if it's feasible: it might not be if
this macro is used outside of Org) ]

Anyway, let's apply the "CALL-WITH" technique:

(cl-defmacro org-export-with-buffer-copy ( &rest body
                                           &key to-buffer drop-visibility
                                           drop-narrowing drop-contents
                                           drop-locals
                                           &allow-other-keys)
  "..."
  `(org-export--call-with-buffer-copy (lambda () ,@body)
                                      :to-buffer ,to-buffer
                                      :drop-visibility ,drop-visibility
                                      :drop-narrowing ,drop-narrowing
                                      :drop-contents ,drop-contents
                                      :drop-locals ,drop-locals))

(cl-defun org-export--call-with-buffer-copy (fn &key to-buffer drop-visibility
                                                drop-narrowing drop-contents
                                                drop-locals
                                                &allow-other-keys)
  (let ((buf-copy (org-export-copy-buffer
                   :to-buffer to-buffer
                   :drop-visibility drop-visibility
                   :drop-narrowing drop-narrowing
                   :drop-contents drop-contents
                   :drop-locals drop-locals)))
    (unwind-protect
(with-current-buffer buf-copy
 (goto-char (point-min))
          (prog1
     (funcall fn)
            ;; `org-export-copy-buffer' carried the value of
            ;; `buffer-file-name' from the original buffer.  When not
            ;; killed, the new buffer copy may become a target of
            ;; `find-file'.  Prevent this.
            (setq buffer-file-name nil)))
      (and (buffer-live-p buf-copy)
  ;; Kill copy without confirmation.
  (progn (with-current-buffer buf-copy
   (restore-buffer-modified-p nil))
                  (unless to-buffer
   (kill-buffer buf-copy)))))))


As a bonus, no more "with-gensyms" needed or hairy commas.  Easier
debugging.  I did a make -C test org/tests but that doesn't have any
good tests and I don't know how to exercise the "publish" and "html"
functionality, so I ask that you test.

The full patch is after my sig.

João

diff --git a/lisp/org/ox.el b/lisp/org/ox.el
index e9cc0ed8fc7..d742172a4c7 100644
--- a/lisp/org/ox.el
+++ b/lisp/org/ox.el
@@ -2597,30 +2597,39 @@ org-export-with-buffer-copy
 copy.  `:to-buffer', `:drop-visibility', `:drop-narrowing',
 `:drop-contents', and `:drop-locals' are passed as arguments to
 `org-export-copy-buffer'."
-  (declare (debug t))
-  (org-with-gensyms (buf-copy)
-    `(let ((,buf-copy (org-export-copy-buffer
-                       :to-buffer ,to-buffer
-                       :drop-visibility ,drop-visibility
-                       :drop-narrowing ,drop-narrowing
-                       :drop-contents ,drop-contents
-                       :drop-locals ,drop-locals)))
-       (unwind-protect
-   (with-current-buffer ,buf-copy
-     (goto-char (point-min))
-             (prog1
-         (progn ,@body)
-               ;; `org-export-copy-buffer' carried the value of
-               ;; `buffer-file-name' from the original buffer.  When not
-               ;; killed, the new buffer copy may become a target of
-               ;; `find-file'.  Prevent this.
-               (setq buffer-file-name nil)))
- (and (buffer-live-p ,buf-copy)
-      ;; Kill copy without confirmation.
-      (progn (with-current-buffer ,buf-copy
-       (restore-buffer-modified-p nil))
-                     (unless ,to-buffer
-       (kill-buffer ,buf-copy))))))))
+  `(org-export--call-with-buffer-copy (lambda () ,@body)
+                                      :to-buffer ,to-buffer
+                                      :drop-visibility ,drop-visibility
+                                      :drop-narrowing ,drop-narrowing
+                                      :drop-contents ,drop-contents
+                                      :drop-locals ,drop-locals))
+
+(cl-defun org-export--call-with-buffer-copy (fn &key to-buffer drop-visibility
+                                                drop-narrowing drop-contents
+                                                drop-locals
+                                                &allow-other-keys)
+  (let ((buf-copy (org-export-copy-buffer
+                   :to-buffer to-buffer
+                   :drop-visibility drop-visibility
+                   :drop-narrowing drop-narrowing
+                   :drop-contents drop-contents
+                   :drop-locals drop-locals)))
+    (unwind-protect
+ (with-current-buffer buf-copy
+  (goto-char (point-min))
+          (prog1
+      (funcall fn)
+            ;; `org-export-copy-buffer' carried the value of
+            ;; `buffer-file-name' from the original buffer.  When not
+            ;; killed, the new buffer copy may become a target of
+            ;; `find-file'.  Prevent this.
+            (setq buffer-file-name nil)))
+      (and (buffer-live-p buf-copy)
+   ;; Kill copy without confirmation.
+   (progn (with-current-buffer buf-copy
+    (restore-buffer-modified-p nil))
+                  (unless to-buffer
+    (kill-buffer buf-copy)))))))

 (cl-defun org-export--generate-copy-script (buffer
                                             &key

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

> OK, what thing does `org-assert-version` assert and what is one concrete
> example of a breakage.  Can this unexpected breakage exist if someone
> has never ever installed Org mode from ELPA?

The assertion is (equal (org-release) ,(org-release)) - assert run-time
and compile-time versions of Org equal.

And here we are talking about the situation when a user compiles Emacs
from git upstream - make will generate a bunch of .elc files that become
obsolete if git pull pulls in a new Org version. Then, another make will
not re-compile .elc files for the files that remain unchanged, causing
version conflict.

>> Or, similarly, when built-in Org mode is updated, some macros are
>> changed, but their users are not re-compiled, leaving stale macro
>> expansions inside .elc files. Again, this causes various breakages.
>
> Well, this is a problem that is very commonly solved by the technique
> I described.  But without an example of one member of this "some macros"
> set and the example of one of its users in some other file, it's hard
> to justify your previous assertion that "the idea will not work".

My statement about "the idea will not work" was referring to the
`org-assert-version' macro. If we wrap calls to `org-assert-version'
into a function, only the file defining that function will be used to
derive compile-time Org version.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[João Távora]

On Sun, Dec 24, 2023 at 4:58 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Sun, 24 Dec 2023 16:39:54 +0000
> > Cc: stefankangas@gmail.com, yantar92@posteo.net, luangruo@yahoo.com,
> >  emacs-devel@gnu.org
> >
> > On Windows, the configure script takes ages
>
> Not if you use -C.  Then it flies (except for the very first time you
> build a fresh clone).

Oh thank you!  Is is just passing `./configure -C` and that's it?  Do I do
need something else?

João

Re: Permanently fix org versioning breakage during builds?

[João Távora]

On Sun, Dec 24, 2023 at 5:02 PM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > OK, what thing does `org-assert-version` assert and what is one concrete
> > example of a breakage.  Can this unexpected breakage exist if someone
> > has never ever installed Org mode from ELPA?
>
> The assertion is (equal (org-release) ,(org-release)) - assert run-time
> and compile-time versions of Org equal.

OK.  So AFAIU org-assert-version macro is itself a tool that you use to
"check sanity" in the face of the maroexpansion-site recompilation
problems posed by _other_ macros and their expansions.

So it seems to follow that if _those_ problems cause by _other macros_
somehow were solved, then the org-assert-version macro wouldn't be
needed, or at least wouldn't be needed for this job.

> My statement about "the idea will not work" was referring to the
> `org-assert-version' macro. If we wrap calls to `org-assert-version'
> into a function, only the file defining that function will be used to
> derive compile-time Org version.

This is the "XY problem" situation.  org-assert-version is solving a given
problem, it's not useful to consider it as part of the problem itself.

João

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

> On Sun, Dec 24, 2023 at 4:32 PM João Távora <joaotavora@gmail.com> wrote:
>
>>
>> No, it didn't, I'm afraid.  If you know one of these macros (you alluded
>> earlier to "some macros"), type its name here, please.
>
> IIn the absence of examples (so far), I went looking for one myself.
> I found 'org-export-with-buffer-copy' in lisp/org/ox.el.  It's expanded
> in ox-html.el and ox-publish.el in the same directory.

It is not about a specific macro. Any macro may be broken this way.

There may be confusion about `org-assert-version' macro vs. all other
macros. The former is my attempt to detect potential breakage of the
latter.

> [  Tangent: one thing I notice is that the expansion in ox-html.el is
> buggy.  It evaluates the keyword arguments to the macros multiple times
> for no effect.  The solution would be to change the macro calling
> convention to be:
>
> (cl-defmacro org-export-with-buffer-copy ((&key to-buffer drop-visibility
>                                           drop-narrowing drop-contents
>                                           drop-locals
>                                           &allow-other-keys) &body body)
>   ...)

May you elaborate?

> Anyway, let's apply the "CALL-WITH" technique:
>
> (cl-defmacro org-export-with-buffer-copy ( &rest body
>                                            &key to-buffer drop-visibility
>                                            drop-narrowing drop-contents
>                                            drop-locals
>                                            &allow-other-keys)
>   "..."
>   `(org-export--call-with-buffer-copy (lambda () ,@body)

AFAIU, this technique will prevent compiler optimizations, won't it?

> As a bonus, no more "with-gensyms" needed or hairy commas.  Easier
> debugging.  I did a make -C test org/tests but that doesn't have any
> good tests and I don't know how to exercise the "publish" and "html"
> functionality, so I ask that you test.

We do not sync Org tests with Emacs tree. They are only a part of Org
repository: https://git.savannah.gnu.org/cgit/emacs/org-mode.git/
Also, do note that development version of Org mode is not what you see
in Emacs tree - Emacs has only stable version. Your patch will not apply
on the latest Org main branch.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

> OK.  So AFAIU org-assert-version macro is itself a tool that you use to
> "check sanity" in the face of the maroexpansion-site recompilation
> problems posed by _other_ macros and their expansions.

Yes.

> So it seems to follow that if _those_ problems cause by _other macros_
> somehow were solved, then the org-assert-version macro wouldn't be
> needed, or at least wouldn't be needed for this job.

Indeed.


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> OK.  So AFAIU org-assert-version macro is itself a tool that you use to
>> "check sanity" in the face of the maroexpansion-site recompilation
>> problems posed by _other_ macros and their expansions.
>
> Yes.

In addition, `org-assert-version' solves another problem with two
version of Org mode (ELPA/git version/manually installed and built-in)
loaded at the same time. This problem is not about macros per-se, but
about situations like

init.el:

(require 'org) ;; built-in org libraries are loaded
(push "/path/to/newer/org" load-path)
(require 'org-inlinetask) ;; org-inlinetask is loaded from the new path,causing conflicts

Such situations are fairly common in user configurations.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[T.V Raman]

in that case, perhaps we shouldn't bbe installing a ew version in
Emacs this often.



Ihor Radchenko writes:
 > Eli Zaretskii <eliz@gnu.org> writes:
 > 
 > >> Org mode may break in unexpected ways due to stale .elc that did not
 > >> update their macro expansions.
 > >
 > > Yes.  That could happen even as part of the build, if the manuals
 > > whose sources are *.org files need to be rebuilt.
 > >
 > > But such breakage happens at times anyway, when some macros change in
 > > incompatible ways and files that have the old versions of those macros
 > > compiled into them are not recompiled.  It isn't anything new for
 > > people who regularly update from Git and rebuild Emacs.
 > 
 > The main problem is that with org-assert-version, "from time to time"
 > becomes more frequent as the breakage is guaranteed every time a new
 > version of Org is installed into Emacs tree.
 > 
 > Since we release a new minor Org version every few weeks, the breakage
 > happens every few weeks.
 > 
 > -- 
 > Ihor Radchenko // yantar92,
 > Org mode contributor,
 > Learn more about Org mode at <https://orgmode.org/>.
 > Support Org development at <https://liberapay.com/org-mode>,
 > or support my work at <https://liberapay.com/yantar92>

-- 

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

"T.V Raman" <raman@google.com> writes:

> in that case, perhaps we shouldn't bbe installing a ew version in
> Emacs this often.

The versions I am talking about are bugfix releases.
Not installing them means missing out on the containing bug fixes.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[João Távora]

On Sun, Dec 24, 2023 at 5:13 PM Ihor Radchenko <yantar92@posteo.net> wrote:

> It is not about a specific macro. Any macro may be broken this way.

No, that's that not true.  Only macros defined in one file and
expanded in other files.  And not if you use this technique.

My advice is to find a list of macros defined in one file and
expanded in another file.

> There may be confusion about `org-assert-version' macro vs. all other
> macros. The former is my attempt to detect potential breakage of the
> latter.
>
> > [  Tangent: one thing I notice is that the expansion in ox-html.el is
> > buggy.  It evaluates the keyword arguments to the macros multiple times
> > for no effect.  The solution would be to change the macro calling
> > convention to be:
> >
> > (cl-defmacro org-export-with-buffer-copy ((&key to-buffer drop-visibility
> >                                           drop-narrowing drop-contents
> >                                           drop-locals
> >                                           &allow-other-keys) &body body)
> >   ...)
>
> May you elaborate?

Just expand the macro as in ox-html.el and see for yourself.

> > Anyway, let's apply the "CALL-WITH" technique:
> >
> > (cl-defmacro org-export-with-buffer-copy ( &rest body
> >                                            &key to-buffer drop-visibility
> >                                            drop-narrowing drop-contents
> >                                            drop-locals
> >                                            &allow-other-keys)
> >   "..."
> >   `(org-export--call-with-buffer-copy (lambda () ,@body)
>
> AFAIU, this technique will prevent compiler optimizations, won't it?

What compiler optimizations are you talking about?  The only
price to pay is an extra funcall.  "We can solve any problem by
introducing an extra level of indirection."  The function compiled,
where presumably the complicated optimization-worthy logic lies
is still compiled.

> in Emacs tree - Emacs has only stable version. Your patch will not apply
> on the latest Org main branch.

Surely can't be very hard to adapt it now that I've shown how it's done,
it's a localized change to a single definition in a single file.

João

Re: Permanently fix org versioning breakage during builds?

[João Távora]

On Sun, Dec 24, 2023 at 5:21 PM Ihor Radchenko <yantar92@posteo.net> wrote:

> init.el:
>
> (require 'org) ;; built-in org libraries are loaded
> (push "/path/to/newer/org" load-path)
> (require 'org-inlinetask) ;; org-inlinetask is loaded from the new path,causing conflicts

No doubt, but the solution to that problem, whatever it is, which at
any rate doesn't seem to be a compile-time problem, rather a run-time
loading one) shouldn't affect any user that doesn't use ELPA/git Org.

So, again, don't make org-assert-version as a pre-condition.  It is a
(somewhat heavy and defficient, I'm afraid) tool to, among others, to  solve
a given problem.

The problem could for example  be solved by variable-watching load-path
carefully.

Re: Permanently fix org versioning breakage during builds?

[João Távora]

On Sun, Dec 24, 2023 at 5:26 PM T.V Raman <raman@google.com> wrote:
>
> in that case, perhaps we shouldn't bbe installing a ew version in
> Emacs this often.

Indeed.  Yet another tool to solve the problem.  This one works for
me, not sure about Emacs master users wanting to use Org bleeding
edge and not use ELPA/git for that.

Re: Permanently fix org versioning breakage during builds?

[João Távora]

On Sun, Dec 24, 2023 at 5:41 PM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> "T.V Raman" <raman@google.com> writes:
>
> > in that case, perhaps we shouldn't bbe installing a ew version in
> > Emacs this often.
>
> The versions I am talking about are bugfix releases.
> Not installing them means missing out on the containing bug fixes.

User can get these bug fixes with a simple `M-x package-install` or
`M-x package-update` (perhaps with one of more prefix arguments, not sure,
or via an extra variable).  That's the way it works with Eglot, for
example.

You have to weigh what is worse.  A slightly-buggy-in-obscure-corners-Org
that keeps the master build clean and manages to load most org files or
the efforts it takes to update a package from ELPA/git.

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

> The problem could for example  be solved by variable-watching load-path
> carefully.

org-assert-version is the best we came up with.
If you know a concrete idea how to solve it differently, feel free to
share it.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[T.V Raman]

But installing them as at present breaks all of emacs -- so we need a
balance. Perhaps the org-tree can be more rigorous, in that process,
tolerate such breakages and fix them, but only push to emacs when it
doesn't break emacs.


Ihor Radchenko writes:
 > "T.V Raman" <raman@google.com> writes:
 > 
 > > in that case, perhaps we shouldn't bbe installing a ew version in
 > > Emacs this often.
 > 
 > The versions I am talking about are bugfix releases.
 > Not installing them means missing out on the containing bug fixes.
 > 
 > -- 
 > Ihor Radchenko // yantar92,
 > Org mode contributor,
 > Learn more about Org mode at <https://orgmode.org/>.
 > Support Org development at <https://liberapay.com/org-mode>,
 > or support my work at <https://liberapay.com/yantar92>

-- 

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

>> The versions I am talking about are bugfix releases.
>> Not installing them means missing out on the containing bug fixes.
>
> User can get these bug fixes with a simple `M-x package-install` or
> `M-x package-update` (perhaps with one of more prefix arguments, not sure,
> or via an extra variable).  That's the way it works with Eglot, for
> example.

> You have to weigh what is worse.  A slightly-buggy-in-obscure-corners-Org
> that keeps the master build clean and manages to load most org files or
> the efforts it takes to update a package from ELPA/git.

I'd prefer to keep Org version in Emacs tree more up-to-date.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[T.V Raman]

And while I remember, we also need to investigate how badly these
breakages bite native-compile; I'm not enabling native-compile for
various other reasons, so am  not in a position to verify,


João Távora writes:
 > On Sun, Dec 24, 2023 at 5:13 PM Ihor Radchenko <yantar92@posteo.net> wrote:
 > 
 > > It is not about a specific macro. Any macro may be broken this way.
 > 
 > No, that's that not true.  Only macros defined in one file and
 > expanded in other files.  And not if you use this technique.
 > 
 > My advice is to find a list of macros defined in one file and
 > expanded in another file.
 > 
 > > There may be confusion about `org-assert-version' macro vs. all other
 > > macros. The former is my attempt to detect potential breakage of the
 > > latter.
 > >
 > > > [  Tangent: one thing I notice is that the expansion in ox-html.el is
 > > > buggy.  It evaluates the keyword arguments to the macros multiple times
 > > > for no effect.  The solution would be to change the macro calling
 > > > convention to be:
 > > >
 > > > (cl-defmacro org-export-with-buffer-copy ((&key to-buffer drop-visibility
 > > >                                           drop-narrowing drop-contents
 > > >                                           drop-locals
 > > >                                           &allow-other-keys) &body body)
 > > >   ...)
 > >
 > > May you elaborate?
 > 
 > Just expand the macro as in ox-html.el and see for yourself.
 > 
 > > > Anyway, let's apply the "CALL-WITH" technique:
 > > >
 > > > (cl-defmacro org-export-with-buffer-copy ( &rest body
 > > >                                            &key to-buffer drop-visibility
 > > >                                            drop-narrowing drop-contents
 > > >                                            drop-locals
 > > >                                            &allow-other-keys)
 > > >   "..."
 > > >   `(org-export--call-with-buffer-copy (lambda () ,@body)
 > >
 > > AFAIU, this technique will prevent compiler optimizations, won't it?
 > 
 > What compiler optimizations are you talking about?  The only
 > price to pay is an extra funcall.  "We can solve any problem by
 > introducing an extra level of indirection."  The function compiled,
 > where presumably the complicated optimization-worthy logic lies
 > is still compiled.
 > 
 > > in Emacs tree - Emacs has only stable version. Your patch will not apply
 > > on the latest Org main branch.
 > 
 > Surely can't be very hard to adapt it now that I've shown how it's done,
 > it's a localized change to a single definition in a single file.
 > 
 > João

-- 

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

>> It is not about a specific macro. Any macro may be broken this way.
>
> No, that's that not true.  Only macros defined in one file and
> expanded in other files.  And not if you use this technique.

You are indeed right. 

>> > [  Tangent: one thing I notice is that the expansion in ox-html.el is
>> > buggy.  It evaluates the keyword arguments to the macros multiple times
>> > for no effect.  The solution would be to change the macro calling
>> > convention to be:
>> >
>> > (cl-defmacro org-export-with-buffer-copy ((&key to-buffer drop-visibility
>> >                                           drop-narrowing drop-contents
>> >                                           drop-locals
>> >                                           &allow-other-keys) &body body)
>> >   ...)
>>
>> May you elaborate?
>
> Just expand the macro as in ox-html.el and see for yourself.

Hmm... It actually looks like a problem with `cl-defmacro' rather than
with macro definition itself.

>> >   `(org-export--call-with-buffer-copy (lambda () ,@body)
>>
>> AFAIU, this technique will prevent compiler optimizations, won't it?
>
> What compiler optimizations are you talking about?  The only
> price to pay is an extra funcall.  "We can solve any problem by
> introducing an extra level of indirection."  The function compiled,
> where presumably the complicated optimization-worthy logic lies
> is still compiled.

Any use of (funcall 'symbol ...) means that compiler is not able to know
the function slot of 'symbol at compile time, because it may change.
Hence, any optimization that relies upon knowing both the context of the
funcall and the internals of 'symbol will become impossible.

And this will not solve the problem when Org files are loaded in mixture
from Emacs built-in version and from some other version (ELPA, manually
installed Org mode, etc).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[T.V Raman]

Ihor Radchenko writes:

> I'd prefer to keep Org version in Emacs tree more up-to-date.

I understand and appreciate that view-point.

But for those of us asking for a fix, "At what cost?" A broken Emacs
is in my opinion too high a price to pay,

 > 
 > -- 
 > Ihor Radchenko // yantar92,
 > Org mode contributor,
 > Learn more about Org mode at <https://orgmode.org/>.
 > Support Org development at <https://liberapay.com/org-mode>,
 > or support my work at <https://liberapay.com/yantar92>

-- 

Re: Permanently fix org versioning breakage during builds?

[João Távora]

On Sun, Dec 24, 2023 at 6:07 PM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > The problem could for example  be solved by variable-watching load-path
> > carefully.
>
> org-assert-version is the best we came up with.
> If you know a concrete idea how to solve it differently, feel free to
> share it.

This was a concrete idea.  Your example showed load-path being modified
and a subsequent require doing the wrong thing.  Variable watchers
are a thing, see "Watching variables" in the Elisp manual.

João

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

"T.V Raman" <raman@google.com> writes:

> But installing them as at present breaks all of emacs -- so we need a
> balance. Perhaps the org-tree can be more rigorous, in that process,
> tolerate such breakages and fix them, but only push to emacs when it
> doesn't break emacs.

I already downgraded the error to warning. No breakage will present
after the next 2 bugfix Org releases.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

>> org-assert-version is the best we came up with.
>> If you know a concrete idea how to solve it differently, feel free to
>> share it.
>
> This was a concrete idea.  Your example showed load-path being modified
> and a subsequent require doing the wrong thing.  Variable watchers
> are a thing, see "Watching variables" in the Elisp manual.

The problem is that we can only install these watchers in the newer
versions of Org mode. Older versions do not have them.

Then, if the newer version is loaded after the older version is
partially loaded, the watchers will not trigger in useful ways.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

"T.V Raman" <raman@google.com> writes:

> But for those of us asking for a fix, "At what cost?" A broken Emacs
> is in my opinion too high a price to pay,

Emacs will no longer be broken.
See https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=af1821673

From my point of view, the discussion now is not about Emacs being
broken because `org-assert-version' throws an error, but about other
types of breakages - the ones `org-assert-version' is aiming to prevent;
they existed for years.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: stefankangas@gmail.com, luangruo@yahoo.com, raman@google.com,
>  joaotavora@gmail.com, emacs-devel@gnu.org
> Date: Sun, 24 Dec 2023 16:59:11 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > But such breakage happens at times anyway, when some macros change in
> > incompatible ways and files that have the old versions of those macros
> > compiled into them are not recompiled.  It isn't anything new for
> > people who regularly update from Git and rebuild Emacs.
> 
> The main problem is that with org-assert-version, "from time to time"
> becomes more frequent as the breakage is guaranteed every time a new
> version of Org is installed into Emacs tree.

If org-assert-version is the source of breakage, then that's not the
breakage I was talking about.  I was talking about breakage due to
changes in macros that were the motivation for you to introduce
org-assert-version in the first place.

IOW, instead of _inducing_ a breakage via org-assert-version, you
should let stuff break by itself, when an incompatible change is made
in one of the macros which org-assert-version protects.

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 24 Dec 2023 17:05:59 +0000
> Cc: raman@google.com, stefankangas@gmail.com, yantar92@posteo.net, 
> 	luangruo@yahoo.com, emacs-devel@gnu.org
> 
> On Sun, Dec 24, 2023 at 4:58 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: João Távora <joaotavora@gmail.com>
> > > Date: Sun, 24 Dec 2023 16:39:54 +0000
> > > Cc: stefankangas@gmail.com, yantar92@posteo.net, luangruo@yahoo.com,
> > >  emacs-devel@gnu.org
> > >
> > > On Windows, the configure script takes ages
> >
> > Not if you use -C.  Then it flies (except for the very first time you
> > build a fresh clone).
> 
> Oh thank you!  Is is just passing `./configure -C` and that's it?

Yes.

> Do I do need something else?

No, just run "make" therefater.  IOW, you need "configure -C" only the
first time you build a clone, after that just says "make -jN", and it
will run "configure -C" if needed automatically.

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Date: Sun, 24 Dec 2023 09:26:30 -0800
> Cc: eliz@gnu.org,
>     stefankangas@gmail.com,
>     luangruo@yahoo.com,
>     raman@google.com,
>     joaotavora@gmail.com,
>     emacs-devel@gnu.org
> 
> in that case, perhaps we shouldn't bbe installing a ew version in
> Emacs this often.

Sorry, that won't fly.  We cannot forcibly slow down the Emacs
development just because some people who track the development branch
are bothered by this issue.  The development branches exist first and
foremost for developers.

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 24 Dec 2023 17:54:10 +0000
> Cc: "T.V Raman" <raman@google.com>, emacs-devel <emacs-devel@gnu.org>
> 
> On Sun, Dec 24, 2023 at 5:21 PM Ihor Radchenko <yantar92@posteo.net> wrote:
> 
> > init.el:
> >
> > (require 'org) ;; built-in org libraries are loaded
> > (push "/path/to/newer/org" load-path)
> > (require 'org-inlinetask) ;; org-inlinetask is loaded from the new path,causing conflicts
> 
> No doubt, but the solution to that problem, whatever it is, which at
> any rate doesn't seem to be a compile-time problem, rather a run-time
> loading one) shouldn't affect any user that doesn't use ELPA/git Org.

Once again: building Emacs uses Org at run-time (for building some
manuals).

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 24 Dec 2023 18:01:18 +0000
> Cc: "T.V Raman" <raman@google.com>, eliz@gnu.org, stefankangas@gmail.com, 
> 	luangruo@yahoo.com, emacs-devel@gnu.org
> 
> On Sun, Dec 24, 2023 at 5:41 PM Ihor Radchenko <yantar92@posteo.net> wrote:
> >
> > "T.V Raman" <raman@google.com> writes:
> >
> > > in that case, perhaps we shouldn't bbe installing a ew version in
> > > Emacs this often.
> >
> > The versions I am talking about are bugfix releases.
> > Not installing them means missing out on the containing bug fixes.
> 
> User can get these bug fixes with a simple `M-x package-install` or
> `M-x package-update` (perhaps with one of more prefix arguments, not sure,
> or via an extra variable).  That's the way it works with Eglot, for
> example.
> 
> You have to weigh what is worse.  A slightly-buggy-in-obscure-corners-Org
> that keeps the master build clean and manages to load most org files or
> the efforts it takes to update a package from ELPA/git.

This idea is a non-starter, so please drop it.  We will not slow down
the pace of development and fixing bugs because of these problems.
They are the same problems we have when macros in cconv.el or
macroexp.el or gv.el or in other similar files are modified.

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: "T.V Raman" <raman@google.com>, emacs-devel <emacs-devel@gnu.org>
> Date: Sun, 24 Dec 2023 18:10:11 +0000
> 
> João Távora <joaotavora@gmail.com> writes:
> 
> > The problem could for example  be solved by variable-watching load-path
> > carefully.
> 
> org-assert-version is the best we came up with.
> If you know a concrete idea how to solve it differently, feel free to
> share it.

Why not just let things break when incompatible changes in Org macros
are installed?  We do that for any other Lisp file in the repository
that defines macros used by other files.  Why cannot Org do the same
when it is compiled as part of the Emacs build?

IOW, why not simply disable the version test as part of the Emacs
build, and leave the breakage for those cases where (a) changes in
macros are really incompatible and (b) they affect Org code that is
being run as part of the build?

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: "T.V Raman" <raman@google.com>, eliz@gnu.org, stefankangas@gmail.com,
>  luangruo@yahoo.com, emacs-devel@gnu.org
> Date: Sun, 24 Dec 2023 18:12:35 +0000
> 
> João Távora <joaotavora@gmail.com> writes:
> 
> >> The versions I am talking about are bugfix releases.
> >> Not installing them means missing out on the containing bug fixes.
> >
> > User can get these bug fixes with a simple `M-x package-install` or
> > `M-x package-update` (perhaps with one of more prefix arguments, not sure,
> > or via an extra variable).  That's the way it works with Eglot, for
> > example.
> 
> > You have to weigh what is worse.  A slightly-buggy-in-obscure-corners-Org
> > that keeps the master build clean and manages to load most org files or
> > the efforts it takes to update a package from ELPA/git.
> 
> I'd prefer to keep Org version in Emacs tree more up-to-date.

So do I.

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> The main problem is that with org-assert-version, "from time to time"
>> becomes more frequent as the breakage is guaranteed every time a new
>> version of Org is installed into Emacs tree.
>
> If org-assert-version is the source of breakage, then that's not the
> breakage I was talking about.  I was talking about breakage due to
> changes in macros that were the motivation for you to introduce
> org-assert-version in the first place.
>
> IOW, instead of _inducing_ a breakage via org-assert-version, you
> should let stuff break by itself, when an incompatible change is made
> in one of the macros which org-assert-version protects.

Yep. That's what I've done now. `org-assert-version' now throws a
warning instead of error:
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=af1821673

The remaining problem (maybe) is that even the warning may not be
desired:

(from Po Lu's messages)
>>> Thanks, but ideally Emacs should not display warnings loading
>>> built-in files

And I am still interested to discuss further on the alternative ways to
address the problem with macros _and also_ the problem with multiple Org
versions being loaded.

For macros, João Távora suggested wrapping them into a function call and
then use the function call + (lambda () ...) instead of macros.
Although I am not sure how it will affect byte- and native-compilation.
And it feels like very awkward workaround that should not be necessary.

For mixed versions, no suggestions were surfaced yet in this discussion.
(Although, I am still trying to make previous ideas with
require-with-shadowcheck work in a local WIP feature branch).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Date: Sun, 24 Dec 2023 10:11:59 -0800
> Cc: raman@google.com,
>     eliz@gnu.org,
>     stefankangas@gmail.com,
>     luangruo@yahoo.com,
>     joaotavora@gmail.com,
>     emacs-devel@gnu.org
> 
> But installing them as at present breaks all of emacs

No, it doesn't.  If you see that org-version.el was modified, just
remove all the lisp/org/*.elc files, and be done.  And if you don't
pay attention, you will get an error message, and will remember then.

(And if Ihor accepts my suggestion up-thread, this will be even less
in our face.)

Let's not make a mountain out of a molehill, okay?

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Date: Sun, 24 Dec 2023 10:13:11 -0800
> Cc: yantar92@posteo.net,
>     raman@google.com,
>     emacs-devel@gnu.org
> 
> And while I remember, we also need to investigate how badly these
> breakages bite native-compile; I'm not enabling native-compile for
> various other reasons, so am  not in a position to verify,

Building with native-compilation has the same issues with Org as
without it.  There's nothing additional there: you need to recompile
all the Org files when the version is updated, otherwise the build
will fail.

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Date: Sun, 24 Dec 2023 10:16:18 -0800
> Cc: joaotavora@gmail.com,
>     raman@google.com,
>     eliz@gnu.org,
>     stefankangas@gmail.com,
>     luangruo@yahoo.com,
>     emacs-devel@gnu.org
> 
> 
> 
> Ihor Radchenko writes:
> 
> > I'd prefer to keep Org version in Emacs tree more up-to-date.
> 
> I understand and appreciate that view-point.
> 
> But for those of us asking for a fix, "At what cost?" A broken Emacs
> is in my opinion too high a price to pay,

Sorry, no.  The price we pay is not too high, not for the developers.

You have the option of updating less frequently, if those problems
bother you.

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: stefankangas@gmail.com, luangruo@yahoo.com, raman@google.com,
>  joaotavora@gmail.com, emacs-devel@gnu.org
> Date: Sun, 24 Dec 2023 18:50:23 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > IOW, instead of _inducing_ a breakage via org-assert-version, you
> > should let stuff break by itself, when an incompatible change is made
> > in one of the macros which org-assert-version protects.
> 
> Yep. That's what I've done now. `org-assert-version' now throws a
> warning instead of error:
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=af1821673

It shouldn't even display a warning.  We don't display any warning
when, say, cconv.el changes one of its macros.

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> org-assert-version is the best we came up with.
>> If you know a concrete idea how to solve it differently, feel free to
>> share it.
>
> Why not just let things break when incompatible changes in Org macros
> are installed?  We do that for any other Lisp file in the repository
> that defines macros used by other files.  Why cannot Org do the same
> when it is compiled as part of the Emacs build?

Org can. However, breaking Emacs builds is not the main purpose of
org-assert-version. The bigger problem I am trying to solve is obscure
breakage when people upgrade Org mode or mix built-in and newer Org mode
in their init files.

> IOW, why not simply disable the version test as part of the Emacs
> build, and leave the breakage for those cases where (a) changes in
> macros are really incompatible and (b) they affect Org code that is
> being run as part of the build?

That's what the patch I shared is about. But I still need to check that
the patch will not break detection of loading mixed Org versions from
init.el.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: joaotavora@gmail.com, raman@google.com, emacs-devel@gnu.org
> Date: Sun, 24 Dec 2023 18:56:02 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> org-assert-version is the best we came up with.
> >> If you know a concrete idea how to solve it differently, feel free to
> >> share it.
> >
> > Why not just let things break when incompatible changes in Org macros
> > are installed?  We do that for any other Lisp file in the repository
> > that defines macros used by other files.  Why cannot Org do the same
> > when it is compiled as part of the Emacs build?
> 
> Org can. However, breaking Emacs builds is not the main purpose of
> org-assert-version. The bigger problem I am trying to solve is obscure
> breakage when people upgrade Org mode or mix built-in and newer Org mode
> in their init files.

I am talking specifically about what happens when Org is compiled as
part of an Emacs build, using the Emacs Makefile's.  You can keep your
safeguards for the other situations, of course.

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Yep. That's what I've done now. `org-assert-version' now throws a
>> warning instead of error:
>> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=af1821673
>
> It shouldn't even display a warning.  We don't display any warning
> when, say, cconv.el changes one of its macros.

Ok. I will look into testing and installing the patch I shared in this thread.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Stefan Kangas]

Ihor Radchenko <yantar92@posteo.net> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Org mode may break in unexpected ways due to stale .elc that did not
>>> update their macro expansions.
>>
>> Yes.  That could happen even as part of the build, if the manuals
>> whose sources are *.org files need to be rebuilt.
>>
>> But such breakage happens at times anyway, when some macros change in
>> incompatible ways and files that have the old versions of those macros
>> compiled into them are not recompiled.  It isn't anything new for
>> people who regularly update from Git and rebuild Emacs.
>
> The main problem is that with org-assert-version, "from time to time"
> becomes more frequent as the breakage is guaranteed every time a new
> version of Org is installed into Emacs tree.
>
> Since we release a new minor Org version every few weeks, the breakage
> happens every few weeks.

Would it make sense to make `org-assert-version' less strict on master?
It might not be necessary to require bootstrap for every single bug fix
release, but I can see how it might be helpful for larger bumps.

Would that make sense?  I'm not terribly close to the problem, since I
bootstrap pretty often and only run from master.

Re: Permanently fix org versioning breakage during builds?

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@posteo.net>
>> Date: Sun, 24 Dec 2023 18:12:35 +0000
>>
>> I'd prefer to keep Org version in Emacs tree more up-to-date.
>
> So do I.

Yup, very much so.

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

Stefan Kangas <stefankangas@gmail.com> writes:

>> Since we release a new minor Org version every few weeks, the breakage
>> happens every few weeks.
>
> Would it make sense to make `org-assert-version' less strict on master?
> It might not be necessary to require bootstrap for every single bug fix
> release, but I can see how it might be helpful for larger bumps.
>
> Would that make sense?  I'm not terribly close to the problem, since I
> bootstrap pretty often and only run from master.

I thought about it (to compare only major+minor versions), but bugfix
releases sometimes contain critical fixes. So, it might not be ideal.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[T.V Raman]

Thanks for that, I missed the downgrade to warning.


Ihor Radchenko writes:
 > "T.V Raman" <raman@google.com> writes:
 > 
 > > But installing them as at present breaks all of emacs -- so we need a
 > > balance. Perhaps the org-tree can be more rigorous, in that process,
 > > tolerate such breakages and fix them, but only push to emacs when it
 > > doesn't break emacs.
 > 
 > I already downgraded the error to warning. No breakage will present
 > after the next 2 bugfix Org releases.
 > 
 > -- 
 > Ihor Radchenko // yantar92,
 > Org mode contributor,
 > Learn more about Org mode at <https://orgmode.org/>.
 > Support Org development at <https://liberapay.com/org-mode>,
 > or support my work at <https://liberapay.com/yantar92>

-- 

Re: Permanently fix org versioning breakage during builds?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2969 bytes --]

On Sun, Dec 24, 2023 at 6:10 PM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> João Távora <joaotavora@gmail.com> writes:

> >> > [  Tangent: one thing I notice is that the expansion in ox-html.el is
> >> > buggy.  It evaluates the keyword arguments to the macros multiple
times
> >> > for no effect.  The solution would be to change the macro calling
> >> > convention to be:
> >> >
> >> > (cl-defmacro org-export-with-buffer-copy ((&key to-buffer
drop-visibility
> >> >                                           drop-narrowing
drop-contents
> >> >                                           drop-locals
> >> >                                           &allow-other-keys) &body
body)
> >> >   ...)
> >>
> >> May you elaborate?
> >
> > Just expand the macro as in ox-html.el and see for yourself.
>
> Hmm... It actually looks like a problem with `cl-defmacro' rather than
> with macro definition itself.

No, Common Lisp's DEFMACRO works exactly like this. You need to understand
how &rest works with &key and &allow-other-keys. It _also_ keeps the keys.
That's just how it works.

> >> >   `(org-export--call-with-buffer-copy (lambda () ,@body)
> >>
> >> AFAIU, this technique will prevent compiler optimizations, won't it?
> >
> > What compiler optimizations are you talking about?  The only
> > price to pay is an extra funcall.  "We can solve any problem by
> > introducing an extra level of indirection."  The function compiled,
> > where presumably the complicated optimization-worthy logic lies
> > is still compiled.
>
> Any use of (funcall 'symbol ...) means that compiler is not able to know
> the function slot of 'symbol at compile time, because it may change.
> Hence, any optimization that relies upon knowing both the context of the
> funcall and the internals of 'symbol will become impossible.

It's not even a symbol that is in question. And whatever this
function-calling, it is more than probably dwarved by the cost of the
remaining macro expansion. If you were making, say, a sequence processing
library it would have to be measured. But here I'm 99.9% sure there's no
efficiency downside. Just in allocation costs alone, your macro creates
buffers, copies things, etc and then there's the expanded body. Just fixing
the bug I've found will more than pay for it likely.

> And this will not solve the problem when Org files are loaded in mixture
> from Emacs built-in version and from some other version (ELPA, manually
> installed Org mode, etc).

That is a different problem with a different solution, and it only affects
those who update Org from ELPA. There are many problems in the world, you
can't expect a single solution to fix them all. That other problem, for
example, has nothing to do with macro expansion site recompilation. I'm
only targeting _that_ one for now, because that is what directly affects
developers or anyone working with an Emacs from master.

João

[-- Attachment #2: Type: text/html, Size: 3892 bytes --]

Re: Permanently fix org versioning breakage during builds?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

On Sun, Dec 24, 2023, 14:56 Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Távora <joaotavora@gmail.com>
> > Date: Sun, 24 Dec 2023 14:13:26 +0000
> > Cc: "T.V Raman" <raman@google.com>, emacs-devel <emacs-devel@gnu.org>
> >
> > Keep in mind something in the build system is causing builds to fail
> with cryptic messages even for
> > people who don't use Org or rarely ever do. I've never touched any Org
> related files in my life, why
> > should it blow up in my face?
>
> Building Emacs requires to build several manuals whose source is in
> Org, and thus you need a working Org, otherwise the build will fail.
> IOW, even if you personally don't visit Org files and don't use any
> Org feature, the build does.
>

Ok, then that explains why 'make' fails, and one more reason we should work
so it doesn't blow up in users faces when they have just updated their
branch.

João

>

[-- Attachment #2: Type: text/html, Size: 1748 bytes --]

Re: Permanently fix org versioning breakage during builds?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1860 bytes --]

On Sun, Dec 24, 2023, 18:32 Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Ihor Radchenko <yantar92@posteo.net>
> > Cc: stefankangas@gmail.com, luangruo@yahoo.com, raman@google.com,
> >  joaotavora@gmail.com, emacs-devel@gnu.org
> > Date: Sun, 24 Dec 2023 16:59:11 +0000
> >
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> > > But such breakage happens at times anyway, when some macros change in
> > > incompatible ways and files that have the old versions of those macros
> > > compiled into them are not recompiled.  It isn't anything new for
> > > people who regularly update from Git and rebuild Emacs.
> >
> > The main problem is that with org-assert-version, "from time to time"
> > becomes more frequent as the breakage is guaranteed every time a new
> > version of Org is installed into Emacs tree.
>
> If org-assert-version is the source of breakage, then that's not the
> breakage I was talking about.  I was talking about breakage due to
> changes in macros that were the motivation for you to introduce
> org-assert-version in the first place.
>
> IOW, instead of _inducing_ a breakage via org-assert-version, you
> should let stuff break by itself, when an incompatible change is made
> in one of the macros which org-assert-version protects.
>

Exactly the point I was trying to make.

org-assert-version shouldn't become part of the problem.

The breakage due to incompatible changes can be tackled one by one, with
certain programming techniques like the one I exemplified with real Org
code, until they are eradicated or become very infrequent.

I've used that technique tens if not hundreds of times. It's very effective
for precisely this problem, which affects not only Emacs and Elisp but any
large Lisp program. The efficiency argument levied against it only very
rarely holds water.

João

>

[-- Attachment #2: Type: text/html, Size: 3170 bytes --]

&allow-other-keys + &rest body in cl-defmacro (was: Permanently fix org versioning breakage during builds?)

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 501 bytes --]

João Távora <joaotavora@gmail.com> writes:

>> Hmm... It actually looks like a problem with `cl-defmacro' rather than
>> with macro definition itself.
>
> No, Common Lisp's DEFMACRO works exactly like this. You need to understand
> how &rest works with &key and &allow-other-keys. It _also_ keeps the keys.
> That's just how it works.

Now I feel a bit like RMS when he complained about CL-lib introducing
alien concepts to Elisp :)

What about non-breaking change like the attached?


[-- Attachment #2: 0001-org-element-with-buffer-copy-org-export-with-buffer-.patch --]
[-- Type: text/x-patch, Size: 2253 bytes --]

From 3ebf5565196e2dfa3548564cf5a266e51e35cd9d Mon Sep 17 00:00:00 2001
Message-ID: <3ebf5565196e2dfa3548564cf5a266e51e35cd9d.1703503500.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Mon, 25 Dec 2023 12:20:10 +0100
Subject: [PATCH] org-element-with-buffer-copy, org-export-with-buffer-copy:
 Fix expansion
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/org-element.el (org-element-with-buffer-copy):
* lisp/ox.el (org-export-with-buffer-copy): Make sure that keyword
arguments in the macro are not duplicated into BODY, which happens for
&rest body + &allow-other-keys in `cl-defmacro'.

Reported-by: João Távora <joaotavora@gmail.com>
Link: https://yhetil.org/emacs-devel/CALDnm53p7nXdnbA2PuH-VHL7EeGQUXYy3x_87hp3oj2pkyWS9Q@mail.gmail.com/
---
 lisp/org-element.el | 3 +++
 lisp/ox.el          | 3 +++
 2 files changed, 6 insertions(+)

diff --git a/lisp/org-element.el b/lisp/org-element.el
index 0924c56fe..842cf8857 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -669,6 +669,9 @@ (cl-defmacro org-element-with-buffer-copy ( &rest body
 copy.  TO-BUFFER, DROP-VISIBILITY, DROP-NARROWING, DROP-CONTENTS, and
 DROP-LOCALS are passed as arguments to `org-element-copy-buffer'."
   (declare (debug t))
+  ;; Drop keyword arguments from BODY.
+  (while (memq (car body) '(:to-buffer :drop-visibility :drop-narrowing :drop-contents :drop-locals))
+    (pop body) (pop body))
   (org-with-gensyms (buf-copy)
     `(let ((,buf-copy (org-element-copy-buffer
                        :to-buffer ,to-buffer
diff --git a/lisp/ox.el b/lisp/ox.el
index b661b59c3..a54b3d809 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -2599,6 +2599,9 @@ (cl-defmacro org-export-with-buffer-copy ( &rest body
 In addition, buffer-local variables are set according to #+BIND:
 keywords."
   (declare (debug t))
+  ;; Drop keyword arguments from BODY.
+  (while (memq (car body) '(:to-buffer :drop-visibility :drop-narrowing :drop-contents :drop-locals))
+    (pop body) (pop body))
   (org-with-gensyms (bind-variables)
     `(let ((,bind-variables (org-export--list-bound-variables)))
        (org-element-with-buffer-copy
-- 
2.42.0


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: &allow-other-keys + &rest body in cl-defmacro (was: Permanently fix org versioning breakage during builds?)

[João Távora]

On Mon, Dec 25, 2023 at 11:23 AM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> >> Hmm... It actually looks like a problem with `cl-defmacro' rather than
> >> with macro definition itself.
> >
> > No, Common Lisp's DEFMACRO works exactly like this. You need to understand
> > how &rest works with &key and &allow-other-keys. It _also_ keeps the keys.
> > That's just how it works.
>
> Now I feel a bit like RMS when he complained about CL-lib introducing
> alien concepts to Elisp :)

It's not Cl-lib's fault that the Org developers made a mistake and didn't
understand how the Common Lisp tool they selected works.

Even in Elisp a zillion macros are correctly written in this form:

  (defmacro with-foo (options-when-fooing &rest body)

It's exactly the same, only you need to do the destructuring yourself.
cl-defmacro simplifies this for free with

  (cl-defmacro with-foo ((&key option1 option2) &body body)

You (or the Org devs) just made a mistake that's all, happens to everybody.
Don't blame Cl-lib for it.

> What about non-breaking change like the attached?

You should be checking for `keywordp` instead, because &allow-other-keys.

João

Re: &allow-other-keys + &rest body in cl-defmacro (was: Permanently fix org versioning breakage during builds?)

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

>> Now I feel a bit like RMS when he complained about CL-lib introducing
>> alien concepts to Elisp :)
>
> It's not Cl-lib's fault that the Org developers made a mistake and didn't
> understand how the Common Lisp tool they selected works.

I do not blame cl-lib. Mostly grumbling on unexpected concept that is
hard to grasp even after reading the manual (maybe, the manual could be
improved).

> Even in Elisp a zillion macros are correctly written in this form:
>
>   (defmacro with-foo (options-when-fooing &rest body)
>
> It's exactly the same, only you need to do the destructuring yourself.
> cl-defmacro simplifies this for free with
>
>   (cl-defmacro with-foo ((&key option1 option2) &body body)

It would be nice if this approach were described in the manual.
Now, there is just description of destructuring, but it is very generic.

>> What about non-breaking change like the attached?
>
> You should be checking for `keywordp` instead, because &allow-other-keys.

The only reason I used &allow-other-keys is that I couldn't use &rest
without specifying &allow-other-keys.
(I did not know about ((&key ...) &rest body) trick when writing this macro.)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

>> Any use of (funcall 'symbol ...) means that compiler is not able to know
>> the function slot of 'symbol at compile time, because it may change.
>> Hence, any optimization that relies upon knowing both the context of the
>> funcall and the internals of 'symbol will become impossible.
>
> It's not even a symbol that is in question. And whatever this
> function-calling, it is more than probably dwarved by the cost of the
> remaining macro expansion. If you were making, say, a sequence processing
> library it would have to be measured. But here I'm 99.9% sure there's no
> efficiency downside. Just in allocation costs alone, your macro creates
> buffers, copies things, etc and then there's the expanded body. Just fixing
> the bug I've found will more than pay for it likely.

I doubt that you are right about 99.9%, because I know that we have some
macros (like `org-with-wide-buffer') that are utilized in
performance-critical code - org-element-cache.

And it is not the only concern why I do not feel that your suggestion is
a good idea:

- The problem with stale macros is not just Org mode's problem - it is
  the problem with Elisp compiler machinery. A very hard one that is
  non-trivial to solve.

- The rest of Emacs just leaves the problem with macros as is and the
  only reason why Org mode touched it is that it was simply a side
  effect of how org-assert-version is implemented - it is first and
  foremost aiming at handling the issue with multiple Org mode versions
  being loaded at the same time, while the side effect of detecting
  stale macros was a bonus.

- Even if I solve the problem with macros using the technique you
  proposed, the problem with mixed versions still has to be handled.

- Further, fixing the stale macro problem means rewriting all the
  existing, and, more importantly, future Org mode macros - an extra
  maintenance burden which I am very hesitant to take.

- And that maintenance burden also comes with an additional downside of
  potential performance degradation.

So, all-in-all, I do not see the idea with wrapped macros as something
practical for Org mode.

For now, I will go ahead and disable org-assert-version checks when
compiling Org mode within Emacs tree, leaving it enabled when Org is
installed by other means.

This will have a downside of missing detecting mixed versions when ELPA
version of Org mode is loaded first, followed (somehow) by built-in Org
version. But I think that such scenario is uncommon enough - one would
have to delete newer Org mode from load-path, which would be a call for
trouble anyway.

I also hope to find a more robust approach for org-assert-version that
will not annoy users with warnings/errors.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[João Távora]

On Mon, Dec 25, 2023, 13:55 Ihor Radchenko <yantar92@posteo.net> wrote:

> I doubt that you are right about 99.9%, because I know that we have some
> macros (like `org-with-wide-buffer') that are utilized in
> performance-critical code - org-element-cache.

You want to bet on it? You really do believe the cost of an indirect funcall
outweighs multiple buffer allocations, plust whatever &rest body does?

(require 'cl-lib)

(defmacro joaot/progn (&rest body)
  `(progn ,@body))

(defmacro joaot/funcall (&rest body)
  `(joaot/with-funcall (lambda () ,@body)))

(defun joaot/with-funcall (fn) (funcall fn))
(when nil

(benchmark-run-compiled 100000 (joaot/progn (cons nil nil)))
;;(0.03082571 2 0.026723563000004447)
;;(0.030770479000000003 2 0.02662909699999716)
;;(0.030249778 2 0.02616808300000173)
;;(0.025063519000000003 2 0.021807326999997656)
;;(0.022791136 2 0.020065053000003275)

(benchmark-run-compiled 100000 (joaot/funcall (cons nil nil)))
;;(0.028634259999999998 2 0.02006373300000064)
;;(0.028718769 2 0.020229722999999922)
;;(0.02806812 2 0.020030715000004307)
;;(0.025772916 2 0.017697210000001462)
;;(0.026947274 2 0.018522359999998628)

(benchmark-run-compiled 100000 (joaot/progn (with-temp-buffer)))
;;(1.4586596770000002 179 1.2495902690000023)
;;(1.434278908 179 1.228926580999996)
;;(1.444412169 179 1.2385162100000002)
;;(1.445273846 179 1.2392989229999998)
;;(1.464507893 179 1.2581699959999995)

(benchmark-run-compiled 100000 (joaot/funcall (with-temp-buffer)))
;;(1.4344190810000002 179 1.223486917999999)
;;(1.418460156 179 1.211255973)
;;(1.449469906 179 1.232802902000003)
;;(1.419341703 179 1.2082177509999994)
;;(1.517436337 179 1.2952383430000012)

The time is completely dominated by allocation and
garbage collection.

A single cons allocation per macroexpansion is enough
to make the difference unmeasurable.  In fact, to be able
to measure it, you'll have to make body completely empty
and run it tens of milions of iterations to be able to measure
anything perceivable in the tenths of a second.

> And it is not the only concern why I do not feel that your suggestion is
> a good idea:
>
> - The problem with stale macros is not just Org mode's problem - it is
>   the problem with Elisp compiler machinery. A very hard one that is
>   non-trivial to solve.

It doesn't have a solution! It's just what macros give you: syntax
manipulation in exchange for late binding. You must add functions
back to regain Lisp's late binding and deal with Lisp's build systems
which don't usually let you specify dependencies between files.

> - The rest of Emacs just leaves the problem with macros as is and the
>   only reason why Org mode touched it is that it was simply a side
>   effect of how org-assert-version is implemented - it is first and
>   foremost aiming at handling the issue with multiple Org mode versions
>   being loaded at the same time, while the side effect of detecting
>   stale macros was a bonus.
>
> - Even if I solve the problem with macros using the technique you
>   proposed, the problem with mixed versions still has to be handled.

I've already explained that is a different other for people who use ELPA
& git to track bleeding edge.  It has another solution.  It's very confusing
to see you reject a solution to a given real problem just because it
doesn't solve another completely different one.

> - Further, fixing the stale macro problem means rewriting all the
>   existing, and, more importantly, future Org mode macros - an extra
>   maintenance burden which I am very hesitant to take.

So you'd rather bother everyone else with broken builds...
And what maintenance burden.  It's much easier to write macros
like this.  No hygiene concerns, no awkward ',foo' everywhere.

> - And that maintenance burden also comes with an additional downside of
>   potential performance degradation.

This bullet is a repeat / filler. Fallacious argument with no data.  I've given
you hard data. Read it.

> So, all-in-all, I do not see the idea with wrapped macros as something
> practical for Org mode.

I tried my best. Repeatedly in this argument you have made erroneous
statements and logically fallacies. And when I correct them, you just came
back whit some more. I have handed you a solution on a plate to fix a real
problem, and you drag your feet so it can stay "your solution".
Can't beat hubris.  Good luck.

João

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

> (benchmark-run-compiled 100000 (joaot/funcall (with-temp-buffer)))
> ;;(1.4344190810000002 179 1.223486917999999)
> ;;(1.418460156 179 1.211255973)
> ;;(1.449469906 179 1.232802902000003)
> ;;(1.419341703 179 1.2082177509999994)
> ;;(1.517436337 179 1.2952383430000012)
>
> The time is completely dominated by allocation and
> garbage collection.

> A single cons allocation per macroexpansion is enough
> to make the difference unmeasurable.  In fact, to be able
> to measure it, you'll have to make body completely empty
> and run it tens of milions of iterations to be able to measure
> anything perceivable in the tenths of a second.

I now tried to change `org-with-wide-buffer' to use funcall and I now
agree with you that performance degradation is not visible.

>> - The problem with stale macros is not just Org mode's problem - it is
>>   the problem with Elisp compiler machinery. A very hard one that is
>>   non-trivial to solve.
>
> It doesn't have a solution! It's just what macros give you: syntax
> manipulation in exchange for late binding. You must add functions
> back to regain Lisp's late binding and deal with Lisp's build systems
> which don't usually let you specify dependencies between files.

By "non-trivial to solve" I meant that solving it would require
introducing dependencies between files.

>> - The rest of Emacs just leaves the problem with macros as is and the
>>   only reason why Org mode touched it is that it was simply a side
>>   effect of how org-assert-version is implemented - it is first and
>>   foremost aiming at handling the issue with multiple Org mode versions
>>   being loaded at the same time, while the side effect of detecting
>>   stale macros was a bonus.
>>
>> - Even if I solve the problem with macros using the technique you
>>   proposed, the problem with mixed versions still has to be handled.
>
> I've already explained that is a different other for people who use ELPA
> & git to track bleeding edge.  It has another solution.  It's very confusing
> to see you reject a solution to a given real problem just because it
> doesn't solve another completely different one.

That's not what I meant. What I meant is that Org mode solving stale
macro problem in Org code will not fix broken Emacs builds - they can
still be broken because of other macros elsewhere in Emacs tree.
And Emacs maintainers told me that there is no plan to address stale
macros issue in Emacs tree - users are expected to run make bootstrap or
similar commands in case of breakage.

So, unless there is some _additional_ benefit for Org mode, I see no
point spending my time to solve stale macro problem just in Org mode.

Such additional benefit could be solving the mixed version problem, but,
unfortunately, mixed version problem is not about macros.

>> - Further, fixing the stale macro problem means rewriting all the
>>   existing, and, more importantly, future Org mode macros - an extra
>>   maintenance burden which I am very hesitant to take.
>
> So you'd rather bother everyone else with broken builds...

If Emacs maintainers say that it is ok, I do not see why I should try to
solve it just for Org mode.

> And what maintenance burden.  It's much easier to write macros
> like this.  No hygiene concerns, no awkward ',foo' everywhere.

What you described is not how Elisp manual suggest to write macros and
not what people usually submit in patches. If I decide to follow your
suggestion, we will need to maintain an extra rule about macro style - I
(and other Org mode maintainers) will have to watch for this yet another
extra thing. Hence maintenance burden.

>> - And that maintenance burden also comes with an additional downside of
>>   potential performance degradation.
>
> This bullet is a repeat / filler. Fallacious argument with no data.  I've given
> you hard data. Read it.

I agree that this argument is incorrect.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> Considering the existing stance that it is ok to break things
> occasionally due to stale macros when running make on Emacs git
> repository, I see no problem disabling these checks for built-in Org
> mode.
>
> Tentatively, it may even be done without affecting version check for
> out-of-tree builds. See the attached patch.
>
> From eadeaa85336e01a5c3d5ac3d5088a83d627d1fc7 Mon Sep 17 00:00:00 2001
> Message-ID: <eadeaa85336e01a5c3d5ac3d5088a83d627d1fc7.1703422144.git.yantar92@posteo.net>
> From: Ihor Radchenko <yantar92@posteo.net>
> Date: Sun, 24 Dec 2023 13:47:04 +0100
> Subject: [PATCH] org-assert-version: Disable version check in .elc files for
>  built-in Org mode

I now installed the patch.
No more extra breakage should occur in Emacs tree.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=bd9dc3217

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: &allow-other-keys + &rest body in cl-defmacro (was: Permanently fix org versioning breakage during builds?)

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> What about non-breaking change like the attached?
>
> From 3ebf5565196e2dfa3548564cf5a266e51e35cd9d Mon Sep 17 00:00:00 2001
> Message-ID: <3ebf5565196e2dfa3548564cf5a266e51e35cd9d.1703503500.git.yantar92@posteo.net>
> From: Ihor Radchenko <yantar92@posteo.net>
> Date: Mon, 25 Dec 2023 12:20:10 +0100
> Subject: [PATCH] org-element-with-buffer-copy, org-export-with-buffer-copy:
>  Fix expansion

I have installed a version with `keywordp', as suggested.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=3afae6690

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: "T.V Raman" <raman@google.com>, João Távora
>  <joaotavora@gmail.com>, emacs-devel <emacs-devel@gnu.org>
> Date: Mon, 25 Dec 2023 18:59:35 +0000
> 
> I now installed the patch.
> No more extra breakage should occur in Emacs tree.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=bd9dc3217

Only after the next merge from Org to Emacs, right?

Re: Permanently fix org versioning breakage during builds?

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I now installed the patch.
>> No more extra breakage should occur in Emacs tree.
>> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=bd9dc3217
>
> Only after the next merge from Org to Emacs, right?

Next two merges probably - when stale .elc files also get the new
version of org-assert-version.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Permanently fix org versioning breakage during builds?

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: luangruo@yahoo.com, raman@google.com, joaotavora@gmail.com,
>  emacs-devel@gnu.org
> Date: Mon, 25 Dec 2023 20:02:26 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I now installed the patch.
> >> No more extra breakage should occur in Emacs tree.
> >> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=bd9dc3217
> >
> > Only after the next merge from Org to Emacs, right?
> 
> Next two merges probably - when stale .elc files also get the new
> version of org-assert-version.

So people should not expect this problem to disappear in Emacs builds
starting from now, not yet.