Org mode update breaking build?

Org mode update breaking build?

[Tobias Bading]

Grrr. Is it normal that Org updates break builds?

I had a Git worktree with a clean checkout and build of Emacs 29 —
933705d61e according to the reflog. Just now I fetched the latest Emacs 29
changes and tried to build 21ec6c1d5c using the usual “nice make -j4”,
resulting in a *huge* wall of text. Something along the lines of

--- %< ---

Warning (emacs): Org version mismatch.  Org loading aborted.
This warning usually appears when a built-in Org version is loaded
prior to the more recent Org version.

Version mismatch is commonly encountered in the following situations:

[…]

In toplevel form:
org/ob-core.el:31:2: Error: Org version mismatch.  Make sure that correct ‘load-path’ is set early in init.el
make[3]: *** [Makefile:332: org/ob-core.elc] Error 1
make[3]: *** Waiting for unfinished jobs....

--- >% ---

A “git clean -fdx -e .ccls-cache -e compile_commands.json -e TAGS” plus
“./autogen.sh” plus “configure […]” plus “nice make -j4” later I have a
working clean Emacs 29 again.

But of course the same happens again in my hacked worktree. So this time I
try “make distclean” for a change — plus the whole autogen, configure,
make aria again. And guess what… the error is still there!

Is that normal? Why is snowflake Org breaking GNU Make?

Tobias

PS: please keep me CC’d, thanks

Re: Org mode update breaking build?

[Alan Mackenzie]

Hello, Tobias.

On Wed, May 03, 2023 at 17:26:05 +0200, Tobias Bading wrote:
> Grrr. Is it normal that Org updates break builds?

For me, it has become normal.  I'm not happy about it.

> I had a Git worktree with a clean checkout and build of Emacs 29 —
> 933705d61e according to the reflog. Just now I fetched the latest Emacs 29
> changes and tried to build 21ec6c1d5c using the usual “nice make -j4”,
> resulting in a *huge* wall of text. Something along the lines of

> --- %< ---

> Warning (emacs): Org version mismatch.  Org loading aborted.
> This warning usually appears when a built-in Org version is loaded
> prior to the more recent Org version.

> Version mismatch is commonly encountered in the following situations:

I don't use org, and I don't understand why I should be expected to
debug its breaking of the build.

In practice, I have to do make bootstrap every time I update my
repositories.

I would have thought a warning from org, not an error, would be
appropriate, here.

> […]

> In toplevel form:
> org/ob-core.el:31:2: Error: Org version mismatch.  Make sure that correct ‘load-path’ is set early in init.el
> make[3]: *** [Makefile:332: org/ob-core.elc] Error 1
> make[3]: *** Waiting for unfinished jobs....

> --- >% ---

> A “git clean -fdx -e .ccls-cache -e compile_commands.json -e TAGS” plus
> “./autogen.sh” plus “configure […]” plus “nice make -j4” later I have a
> working clean Emacs 29 again.

> But of course the same happens again in my hacked worktree. So this time I
> try “make distclean” for a change — plus the whole autogen, configure,
> make aria again. And guess what… the error is still there!

> Is that normal? Why is snowflake Org breaking GNU Make?

I hope an org maintainer will explain this again.  I've got a feeling
the topic came up in emacs-devel some days ago.

> Tobias

> PS: please keep me CC’d, thanks

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Org mode update breaking build?

[Eli Zaretskii]

> Date: Wed, 3 May 2023 17:26:05 +0200
> From: Tobias Bading <tbading@web.de>
> 
> Warning (emacs): Org version mismatch.  Org loading aborted.
> This warning usually appears when a built-in Org version is loaded
> prior to the more recent Org version.
> 
> Version mismatch is commonly encountered in the following situations:
> 
> […]
> 
> In toplevel form:
> org/ob-core.el:31:2: Error: Org version mismatch.  Make sure that correct ‘load-path’ is set early in init.el
> make[3]: *** [Makefile:332: org/ob-core.elc] Error 1
> make[3]: *** Waiting for unfinished jobs....
> 
> --- >% ---
> 
> A “git clean -fdx -e .ccls-cache -e compile_commands.json -e TAGS” plus
> “./autogen.sh” plus “configure […]” plus “nice make -j4” later I have a
> working clean Emacs 29 again.
> 
> But of course the same happens again in my hacked worktree. So this time I
> try “make distclean” for a change — plus the whole autogen, configure,
> make aria again. And guess what… the error is still there!
> 
> Is that normal?

Yes, on emacs-29.  Well, not "normal" but "expected".  The fix is to
remove lisp/org/*.elc and say "make" again.

There's a solution on master, and an improvement on that solution in
the works, but I decided not to risk that on the release branch, since
the problem happens relatively rarely and the workaround is simple
enough.

Re: Org mode update breaking build?

[Tobias Bading]

On 03.05.23 18:04, Eli Zaretskii wrote:
> Yes, on emacs-29.  Well, not "normal" but "expected".  The fix is to
> remove lisp/org/*.elc and say "make" again.

“make distclean” didn’t help and no *.elc should have survived that, right?

Re: Org mode update breaking build?

[Tobias Bading]

On 03.05.23 17:55, Alan Mackenzie wrote:
> I don't use org, and I don't understand why I should be expected to
> debug its breaking of the build.

+1

In this case the math is: distclean < bug < extraclean

But that’s probably no better than having to “make bootstrap” every time.

Re: Org mode update breaking build?

[Eli Zaretskii]

> Date: Wed, 3 May 2023 18:07:27 +0200
> Cc: emacs-devel@gnu.org
> From: Tobias Bading <tbading@web.de>
> 
> On 03.05.23 18:04, Eli Zaretskii wrote:
> > Yes, on emacs-29.  Well, not "normal" but "expected".  The fix is to
> > remove lisp/org/*.elc and say "make" again.
> 
> “make distclean” didn’t help and no *.elc should have survived that, right?

Wrong.  "make distclean" doesn't remove any *.elc files, because those
come with the tarball, and "make distclean" only removes stuff that
didn't come with the tarball, but was built later.

Re: Org mode update breaking build?

[Eli Zaretskii]

> Date: Wed, 3 May 2023 18:11:30 +0200
> Cc: emacs-devel@gnu.org
> From: Tobias Bading <tbading@web.de>
> 
> On 03.05.23 17:55, Alan Mackenzie wrote:
> > I don't use org, and I don't understand why I should be expected to
> > debug its breaking of the build.
> 
> +1
> 
> In this case the math is: distclean < bug < extraclean
> 
> But that’s probably no better than having to “make bootstrap” every time.

We already fixed that, but only on master.  So you don't need to
convince anyone that this needed fixing.  We are convinced.

Re: Org mode update breaking build?

[Tobias Bading]

On 03.05.23 18:58, Eli Zaretskii wrote:
>> In this case the math is: distclean < bug < extraclean
>>
>> But that’s probably no better than having to “make bootstrap” every time.
>
> We already fixed that, but only on master.  So you don't need to
> convince anyone that this needed fixing.  We are convinced.

:-)

It wasn’t my intention to p*ss off the org-mode and/or Emacs developers or
anything like that. At work I have to deal with Maven-based Java projects
all the time and I simply cannot have build jobs crapping out on me in my
spare time, too. XD

Re: Org mode update breaking build?

[John ff]

⁣​

On 3 May 2023, 17:58, at 17:58, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 3 May 2023 18:07:27 +0200
>> Cc: emacs-devel@gnu.org
>> From: Tobias Bading <tbading@web.de>
>>
>> On 03.05.23 18:04, Eli Zaretskii wrote:
>> > Yes, on emacs-29.  Well, not "normal" but "expected".  The fix is
>to
>> > remove lisp/org/*.elc and say "make" again.
>>
>> “make distclean” didn’t help and no *.elc should have survived that,
>right?
>
>Wrong.  "make distclean" doesn't remove any *.elc files, because those
>come with the tarball, and "make distclean" only removes stuff that
>didn't come with the tarball, but was built later.

Re: Org mode update breaking build?

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

> There's a solution on master, and an improvement on that solution in
> the works,

Is the solution commit 62e4eb8fcf7?  This commit is a week old but I
would swear oath that I've had this org version mismatch error yesterday
(or maybe two days ago?) on two different machines following the master
branch.  On both of them I update my emacs (almost) every morning.

If my memory is correct and I get the error again, what information
should I collect and attach to Bug#62762?

Bye,
Tassilo

Re: Org mode update breaking build?

[Michael Heerdegen]

Tassilo Horn <tsdh@gnu.org> writes:

> Is the solution commit 62e4eb8fcf7?  This commit is a week old but I
> would swear oath that I've had this org version mismatch error yesterday
> (or maybe two days ago?) on two different machines following the master
> branch.  On both of them I update my emacs (almost) every morning.

Same here if it matters: saw the error for the first time on master
yesterday, and I am updating and calling "make" daily as well.

Michael.

Re: Org mode update breaking build?

[Bob Rogers]

   From: Michael Heerdegen <michael_heerdegen@web.de>
   Date: Thu, 04 May 2023 04:00:01 +0200

   Tassilo Horn <tsdh@gnu.org> writes:

   > Is the solution commit 62e4eb8fcf7?  This commit is a week old but I
   > would swear oath that I've had this org version mismatch error yesterday
   > (or maybe two days ago?) on two different machines following the master
   > branch.  On both of them I update my emacs (almost) every morning.

   Same here if it matters: saw the error for the first time on master
   yesterday, and I am updating and calling "make" daily as well.

   Michael.

FWIW, "make bootstrap" worked for me just now on b28d44d4226.

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: Org mode update breaking build?

[Michael Heerdegen]

Bob Rogers <rogers@rgrjr.com> writes:

> FWIW, "make bootstrap" worked for me just now on b28d44d4226.

I mainly wanted to confirm the "yesterday" part.

I didn't try bootstrap; only tried "make" and got the org loading error
state.

Then immediately git clean -xf, autoconf, configure and make, just to be
on the safe side...

Michael.

Re: Org mode update breaking build?

[Bob Rogers]

   From: Michael Heerdegen <michael_heerdegen@web.de>
   Date: Thu, 04 May 2023 05:30:26 +0200

   Bob Rogers <rogers@rgrjr.com> writes:

   > FWIW, "make bootstrap" worked for me just now on b28d44d4226.

   I mainly wanted to confirm the "yesterday" part.

   I didn't try bootstrap; only tried "make" and got the org loading error
   state.

   Then immediately git clean -xf, autoconf, configure and make, just to be
   on the safe side...

   Michael.

OK.  I just thought I'd put that out there because it's my understanding
that if "make bootstrap" fixes the issue, it's not considered a serious
build breakage.

					-- Bob

Re: Org mode update breaking build?

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 3 May 2023 18:11:30 +0200
>> Cc: emacs-devel@gnu.org
>> From: Tobias Bading <tbading@web.de>
>> 
>> On 03.05.23 17:55, Alan Mackenzie wrote:
>> > I don't use org, and I don't understand why I should be expected
>> > to
>> > debug its breaking of the build.
>> 
>> +1
>> 
>> In this case the math is: distclean < bug < extraclean
>> 
>> But that’s probably no better than having to “make bootstrap” every
>> time.
>
> We already fixed that, but only on master.  So you don't need to
> convince anyone that this needed fixing.  We are convinced.

Is there any reason Org Mode needs to signal a compilation error in this
case?  And if there is, why not simply make all the Org elc files depend
on each other in deps.mk or something?

Re: Org mode update breaking build?

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: Tobias Bading <tbading@web.de>, emacs-devel@gnu.org
> Date: Wed, 03 May 2023 21:02:34 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > There's a solution on master, and an improvement on that solution in
> > the works,
> 
> Is the solution commit 62e4eb8fcf7?

It's a major part of that, yes.  Another important part is in commit
d80f959bed (in its part that changed org-macs.el).

> This commit is a week old but I would swear oath that I've had this
> org version mismatch error yesterday (or maybe two days ago?) on two
> different machines following the master branch.  On both of them I
> update my emacs (almost) every morning.

You need to remove all the org/*.elc files, once, after that commit,
to adjust to the renaming of the variable that is an important part of
that fix.

> If my memory is correct and I get the error again, what information
> should I collect and attach to Bug#62762?

Run the failed compilation command manually and print the value of
org--inhibit-version-check while compiling the offending file.

Re: Org mode update breaking build?

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Thu, 04 May 2023 04:00:01 +0200
> 
> Tassilo Horn <tsdh@gnu.org> writes:
> 
> > Is the solution commit 62e4eb8fcf7?  This commit is a week old but I
> > would swear oath that I've had this org version mismatch error yesterday
> > (or maybe two days ago?) on two different machines following the master
> > branch.  On both of them I update my emacs (almost) every morning.
> 
> Same here if it matters: saw the error for the first time on master
> yesterday, and I am updating and calling "make" daily as well.

Same answer.

Re: Org mode update breaking build?

[Eli Zaretskii]

> From: Bob Rogers <rogers@rgrjr.com>
> Date: Wed, 3 May 2023 20:23:26 -0700
> CC: emacs-devel@gnu.org
> 
>    From: Michael Heerdegen <michael_heerdegen@web.de>
>    Date: Thu, 04 May 2023 04:00:01 +0200
> 
>    Tassilo Horn <tsdh@gnu.org> writes:
> 
>    > Is the solution commit 62e4eb8fcf7?  This commit is a week old but I
>    > would swear oath that I've had this org version mismatch error yesterday
>    > (or maybe two days ago?) on two different machines following the master
>    > branch.  On both of them I update my emacs (almost) every morning.
> 
>    Same here if it matters: saw the error for the first time on master
>    yesterday, and I am updating and calling "make" daily as well.
> 
>    Michael.
> 
> FWIW, "make bootstrap" worked for me just now on b28d44d4226.

"make bootstrap" is a blunt weapon, and is not necessary in this case
(and in many others).  But of course, if you'd rather use a single
simple recipe, and don't much care for the longer build time, using
"make bootstrap" is perfectly fine.

Re: Org mode update breaking build?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Tobias Bading <tbading@web.de>,  acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 04 May 2023 12:42:12 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We already fixed that, but only on master.  So you don't need to
> > convince anyone that this needed fixing.  We are convinced.
> 
> Is there any reason Org Mode needs to signal a compilation error in this
> case?

It no longer does, on master.

Please read the thread if you want to know the gory details, I'd
rather we didn't repeat all that.

> And if there is, why not simply make all the Org elc files depend on
> each other in deps.mk or something?

Because it's impractical (too many files, some of which may not exist
or be outdated etc.), and because it doesn't really work in this case.
I tried that as the first attempt at solving the problem.

Re: Org mode update breaking build?

[Bob Rogers]

   From: Eli Zaretskii <eliz@gnu.org>
   Date: Thu, 04 May 2023 08:42:53 +0300

   > From: Bob Rogers <rogers@rgrjr.com>
   > Date: Wed, 3 May 2023 20:23:26 -0700
   > CC: emacs-devel@gnu.org
   > 
   > . . .
   > 
   > FWIW, "make bootstrap" worked for me just now on b28d44d4226.

   "make bootstrap" is a blunt weapon, and is not necessary in this case
   (and in many others).  But of course, if you'd rather use a single
   simple recipe, and don't much care for the longer build time, using
   "make bootstrap" is perfectly fine.

I only recompile roughly once a week, so I'm happy to trade off extra
compile time for greater chance of successful completion.

					-- Bob

Re: Org mode update breaking build?

[Madhu]

* Alan Mackenzie <ZFKEC4EoYZeTjOhK@ACM> :
Wrote on Wed, 3 May 2023 15:55:55 +0000:
> On Wed, May 03, 2023 at 17:26:05 +0200, Tobias Bading wrote:
>> Grrr. Is it normal that Org updates break builds?
> For me, it has become normal.  I'm not happy about it.
> I don't use org, and I don't understand why I should be expected to
> debug its breaking of the build.
>
> In practice, I have to do make bootstrap every time I update my
> repositories.

I haven't update master very recently but a while ago I got into the
habit of running a script before calling `make'. the $1 parameter is the
path to src/lisp. it lists stale elc files, which I then delete by hand.

```
for i in $(find "$1" -name \*.elc); do
	j=${i%c};
	if [ ! -e "$i" ]; then
		echo "MISSING $i" >> /dev/stderr
	elif [ ! -e "$j" ]; then
		echo "MISSING $j" >> /dev/stderr
	else
		# file-newer-p.sh $j $i  # UNTESTED.
		a=$(stat -c '%Y' "$j")
		b=$(stat -c '%Y' "$i")
		if [ $a -gt $a ]; then echo "$i"; fi;
	fi
done
```

Since I started doing this, I don't think I've faced elisp recompilation
problems. But I never understood the rationale for "loading stale elc"
behaviour as a part of recompilation.  If someone can explain that I'd
appreciate it.

Re: Org mode update breaking build?

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Mon, 08 May 2023 17:30:17 +0530
> 
> I haven't update master very recently but a while ago I got into the
> habit of running a script before calling `make'. the $1 parameter is the
> path to src/lisp. it lists stale elc files, which I then delete by hand.
> 
> ```
> for i in $(find "$1" -name \*.elc); do
> 	j=${i%c};
> 	if [ ! -e "$i" ]; then
> 		echo "MISSING $i" >> /dev/stderr
> 	elif [ ! -e "$j" ]; then
> 		echo "MISSING $j" >> /dev/stderr
> 	else
> 		# file-newer-p.sh $j $i  # UNTESTED.
> 		a=$(stat -c '%Y' "$j")
> 		b=$(stat -c '%Y' "$i")
> 		if [ $a -gt $a ]; then echo "$i"; fi;
> 	fi
> done
> ```
> 
> Since I started doing this, I don't think I've faced elisp recompilation
> problems. But I never understood the rationale for "loading stale elc"
> behaviour as a part of recompilation.  If someone can explain that I'd
> appreciate it.

We have switched to "load newer el" behavior quite some time ago, so I
think your script is from before that.

The problem here is not with *.elc files that are older than the
corresponding *.el files, the problem is with _other_ *.elc files that
were produced with outdated definitions of macros, and those macros
are defined in the *.el files other than the one corresponding to the
problematic *.elc.  So just looking at time stamps will not allow to
find the problematic *.elc files.

Re: Org mode update breaking build?

[Sam Steingold]

> * Eli Zaretskii <ryvm@tah.bet> [2023-05-03 19:58:07 +0300]:
>
> We already fixed that, but only on master.  So you don't need to
> convince anyone that this needed fixing.  We are convinced.

I built from git master and did not get any errors.
However, when I visited an org-mode file, I got this:

--8<---------------cut here---------------start------------->8---
Debugger entered--Lisp error: (error "Org version mismatch.  Make sure that correct ‘load-path’ is set early in init.el")
  error("Org version mismatch.  Make sure that correct `load-path' is set early in init.el")
  byte-code(...)
  require(org-keys)
  byte-code(...)
  org-mode()
  set-auto-mode-0(org-mode nil)
  set-auto-mode()
  normal-mode(t)
  after-find-file(nil t)
  find-file-noselect-1(#<buffer xxx.txt> "~/xxx.txt" nil nil "~/xxx.txt" (1180547 66307))
  find-file-noselect("~/xxx.txt" nil nil t)
  find-file("~/xxx.txt" t)
  funcall-interactively(find-file "~/xxx.txt" t)
  command-execute(find-file)
--8<---------------cut here---------------end--------------->8---

(along with the usual long but worthless instructions)

While I agree that those who never use org can be happy with the change,
but I would argue that this is _worse_ for me (and other - willing or
unwilling - org users).

Basically, an apparently successful build no longer guarantees that the
resulting Emacs will be working.

It would seem that if the build process can detect this "Org version
mismatch", it should just `rm -f lisp/org/*.elc` and restart (remembering
that it tried this hack to avoid an infinite loop if it does not work).

Thank you!

-- 
Sam Steingold (https://aphar.dreamwidth.org/) on Pop 22.04 (jammy) X 11.0.12101004
https://lastingimpactpsychology.com https://steingoldpsychology.com
https://ffii.org https://www.memritv.org http://think-israel.org
Don't take life too seriously, you'll never get out of it alive!

Re: Org mode update breaking build?

[Eli Zaretskii]

> From: Sam Steingold <sds@gnu.org>
> Date: Tue, 09 May 2023 12:34:29 -0400
> 
> > * Eli Zaretskii <ryvm@tah.bet> [2023-05-03 19:58:07 +0300]:
> >
> > We already fixed that, but only on master.  So you don't need to
> > convince anyone that this needed fixing.  We are convinced.
> 
> I built from git master and did not get any errors.
> However, when I visited an org-mode file, I got this:
> 
> --8<---------------cut here---------------start------------->8---
> Debugger entered--Lisp error: (error "Org version mismatch.  Make sure that correct ‘load-path’ is set early in init.el")
>   error("Org version mismatch.  Make sure that correct `load-path' is set early in init.el")
>   byte-code(...)
>   require(org-keys)
>   byte-code(...)
>   org-mode()
>   set-auto-mode-0(org-mode nil)
>   set-auto-mode()
>   normal-mode(t)
>   after-find-file(nil t)
>   find-file-noselect-1(#<buffer xxx.txt> "~/xxx.txt" nil nil "~/xxx.txt" (1180547 66307))
>   find-file-noselect("~/xxx.txt" nil nil t)
>   find-file("~/xxx.txt" t)
>   funcall-interactively(find-file "~/xxx.txt" t)
>   command-execute(find-file)
> --8<---------------cut here---------------end--------------->8---
> 
> (along with the usual long but worthless instructions)

That's expected when there's an incompatible change.  It can also
happen with other packages, not just with Org.  It's nothing new.

> Basically, an apparently successful build no longer guarantees that the
> resulting Emacs will be working.

That was never guaranteed.  Not just when Org changed.

> It would seem that if the build process can detect this "Org version
> mismatch", it should just `rm -f lisp/org/*.elc` and restart (remembering
> that it tried this hack to avoid an infinite loop if it does not work).

Feel free to propose how to detect this, and then restart, without
disrupting the whole build.  We are not just compiling Org, we compile
hundreds of Lisp files, so whatever you propose should be consistent
with how the normal build proceeds, including its support for high
parallelism.  If a better solution will be found, we will surely
embrace it.

Re: Org mode update breaking build?

[Sam Steingold]

> * Eli Zaretskii <ryvm@tah.bet> [2023-05-09 19:47:20 +0300]:
>
>> From: Sam Steingold <sds@gnu.org>
>> Date: Tue, 09 May 2023 12:34:29 -0400
>> 
>> > * Eli Zaretskii <ryvm@tah.bet> [2023-05-03 19:58:07 +0300]:
>> >
>> > We already fixed that, but only on master.  So you don't need to
>> > convince anyone that this needed fixing.  We are convinced.
>> 
>> I built from git master and did not get any errors.
>> However, when I visited an org-mode file, I got this:
>> 
>> --8<---------------cut here---------------start------------->8---
>> Debugger entered--Lisp error: (error "Org version mismatch.  Make sure that correct ‘load-path’ is set early in init.el")
>> --8<---------------cut here---------------end--------------->8---
>> 
>
> That's expected when there's an incompatible change.  It can also
> happen with other packages, not just with Org.  It's nothing new.

sad.

>> Basically, an apparently successful build no longer guarantees that
>> the resulting Emacs will be working.
>
> That was never guaranteed.  Not just when Org changed.

I have never seen this before, sorry.

>> It would seem that if the build process can detect this "Org version
>> mismatch", it should just `rm -f lisp/org/*.elc` and restart (remembering
>> that it tried this hack to avoid an infinite loop if it does not work).
>
> Feel free to propose how to detect this, and then restart, without
> disrupting the whole build.  We are not just compiling Org, we compile
> hundreds of Lisp files, so whatever you propose should be consistent
> with how the normal build proceeds, including its support for high
> parallelism.  If a better solution will be found, we will surely
> embrace it.

I can describe how CLISP handles this:
(https://clisp.sourceforge.io/impnotes/require.html#lib-files
and https://lists.gnu.org/archive/html/emacs-devel/2021-10/msg00274.html)

Compilation of `foo.el` should produce 2 files:

1. `foo.elc`, as now - this is the code whose loading is functionally
   equivalent to loading `foo.el`.

2. `foo.elh` ("header") - this contains only the compile-time
   dependencies (i.e., compiled `defvar`, `defconst`, `defmacro`, and
   `defsubst` definitions and function declarations),

When the compiler sees `(require 'foo)`, it will check whether `foo.elh`
has changed since last loaded and will reload it automatically.
Note that `foo.elh` is (probably) much smaller than `foo.elc` and thus
cheaper to load.

For "pre-built" files like `subr.el` (which no other file ever requires):
when `subr.el` is recompiled, we check whether `subr.elh` has changed, and,
if it did, everything needs to be recompiled.
If it did not, no action is necessary.

Thank you for your kind an informative reply.

-- 
Sam Steingold (https://aphar.dreamwidth.org/) on Pop 22.04 (jammy) X 11.0.12101004
https://lastingimpactpsychology.com https://steingoldpsychology.com
https://iris.org.il https://memri.org https://mideasttruth.com
To iterate is human; to recurse, divine.

Re: Org mode update breaking build?

[Eli Zaretskii]

> From: Sam Steingold <sds@gnu.org>
> Date: Tue, 09 May 2023 13:17:28 -0400
> 
> >> Basically, an apparently successful build no longer guarantees that
> >> the resulting Emacs will be working.
> >
> > That was never guaranteed.  Not just when Org changed.
> 
> I have never seen this before, sorry.

I guess you don't rebuild Emacs frequently enough.

> > Feel free to propose how to detect this, and then restart, without
> > disrupting the whole build.  We are not just compiling Org, we compile
> > hundreds of Lisp files, so whatever you propose should be consistent
> > with how the normal build proceeds, including its support for high
> > parallelism.  If a better solution will be found, we will surely
> > embrace it.
> 
> I can describe how CLISP handles this:
> (https://clisp.sourceforge.io/impnotes/require.html#lib-files
> and https://lists.gnu.org/archive/html/emacs-devel/2021-10/msg00274.html)
> 
> Compilation of `foo.el` should produce 2 files:
> 
> 1. `foo.elc`, as now - this is the code whose loading is functionally
>    equivalent to loading `foo.el`.
> 
> 2. `foo.elh` ("header") - this contains only the compile-time
>    dependencies (i.e., compiled `defvar`, `defconst`, `defmacro`, and
>    `defsubst` definitions and function declarations),
> 
> When the compiler sees `(require 'foo)`, it will check whether `foo.elh`
> has changed since last loaded and will reload it automatically.
> Note that `foo.elh` is (probably) much smaller than `foo.elc` and thus
> cheaper to load.
> 
> For "pre-built" files like `subr.el` (which no other file ever requires):
> when `subr.el` is recompiled, we check whether `subr.elh` has changed, and,
> if it did, everything needs to be recompiled.
> If it did not, no action is necessary.

The general idea is simple, indeed, but there are rocks under the
surface.  See

   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62762#206

for the description of some of them.

Re: Org mode update breaking build?

[Sam Steingold]

> * Eli Zaretskii <ryvm@tah.bet> [2023-05-09 22:07:43 +0300]:
>
> The general idea is simple, indeed, but there are rocks under the
> surface.
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62762#206
> for the description of some of them.

Sigh.
I guess there is a reason why most Lisp software is build using
Lisp-based build systems (defsystem, asdf &c)


-- 
Sam Steingold (https://aphar.dreamwidth.org/) on Pop 22.04 (jammy) X 11.0.12101004
https://lastingimpactpsychology.com https://steingoldpsychology.com
https://fairforall.org https://www.peaceandtolerance.org/ https://camera.org
Software is like sex: it's better when it's free.

Re: Org mode update breaking build?

[Madhu]

Hello
*  Eli Zaretskii <eliz@gnu.org> <83a5yfc60y.fsf@gnu.org>
Wrote on Mon, 08 May 2023 16:22:05 +0300
>> From: Madhu <enometh@meer.net>
>> Date: Mon, 08 May 2023 17:30:17 +0530
>>
>> I haven't update master very recently but a while ago I got into the
>> habit of running a script before calling `make'. the $1 parameter is the
>> path to src/lisp. it lists stale elc files, which I then delete by hand.
>>
>> ```
>> for i in $(find "$1" -name \*.elc); do
>> 	j=${i%c};
>> 	if [ ! -e "$i" ]; then
>> 		echo "MISSING $i" >> /dev/stderr
>> 	elif [ ! -e "$j" ]; then
>> 		echo "MISSING $j" >> /dev/stderr
>> 	else
>> 		# file-newer-p.sh $j $i  # UNTESTED.
>> 		a=$(stat -c '%Y' "$j")
>> 		b=$(stat -c '%Y' "$i")
>> 		if [ $a -gt $a ]; then echo "$i"; fi;
>> 	fi

(Obvious typo there: should be $a -gt $b)

>> done
>> ```
>> Since I started doing this, I don't think I've faced elisp recompilation
>> problems. But I never understood the rationale for "loading stale elc"
>> behaviour as a part of recompilation.  If someone can explain that I'd
>> appreciate it.
>
> We have switched to "load newer el" behavior quite some time ago, so I
> think your script is from before that.

Is this just during bootup?

I still see messages on startup like
```
Source file ‘/14/build/emacs/lisp/dired.el’ newer than byte-compiled file; using older file
```

during startup.
```
(load "dired") shows a similar message.
Loading dired (compiled; note, source file is newer)...done
```

> The problem here is not with *.elc files that are older than the
> corresponding *.el files, the problem is with _other_ *.elc files that
> were produced with outdated definitions of macros, and those macros
> are defined in the *.el files other than the one corresponding to the
> problematic *.elc.  So just looking at time stamps will not allow to
> find the problematic *.elc files.

Indeed the script is not useful in this case. I ran into the
org-update problem just now and had to delete all the org elc files by
hand.

Re: Org mode update breaking build?

[Eli Zaretskii]

> Date: Sun, 14 May 2023 11:19:34 +0530 (IST)
> Cc: emacs-devel@gnu.org
> From: Madhu <enometh@meer.net>
> 
> *  Eli Zaretskii <eliz@gnu.org> <83a5yfc60y.fsf@gnu.org>
> > We have switched to "load newer el" behavior quite some time ago, so I
> > think your script is from before that.
> 
> Is this just during bootup?

During the Emacs build (I don't know what you mean by "bootup").  This
thread discusses failures in the Emacs build.

> I still see messages on startup like
> ```
> Source file ‘/14/build/emacs/lisp/dired.el’ newer than byte-compiled file; using older file
> ```
> 
> during startup.
> ```
> (load "dired") shows a similar message.
> Loading dired (compiled; note, source file is newer)...done
> ```

Yes, expected, since load-prefer-newer is by default nil.