bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

CC: Alan Mackenzie, author of the change in 10083e788f7349fa363d100687dc3d94bea88f57

I've seen for a long time Emacs master builds fail from time to time in spectacular
ways after updating the repo, sometimes so badly that `make clean` doesn't help.

I never dug into that though, but I'm attributing this to the occasional build
messages similar to:

    Source file ‘/home/constantine/Projects/emacs/lisp/emacs-lisp/comp.el’ newer than byte-compiled file; using older file
    Source file ‘/home/constantine/Projects/emacs/lisp/emacs-lisp/bytecomp.el’ newer than byte-compiled file; using older file
    Source file ‘/home/constantine/Projects/emacs/lisp/emacs-lisp/comp-cstr.el’ newer than byte-compiled file; using older file

…which makes sense, because if the repo changed `comp.el` API and Emacs during the
build of newer files is trying to make use of older `.elc` file and hence the older
API, it may result in failure.

Got some spare time today, dug into one of the messages. From what I understand it's
caused by this line `lisp/Makefile.in`:

    # ... but we must prefer .elc files for those in the early bootstrap.
    compile-first: BYTE_COMPILE_FLAGS = $(BYTE_COMPILE_EXTRA_FLAGS)

From what I understand, this rewrites BYTE_COMPILE_FLAGS to be an empty variable,
which results in `(setq load-prefer-newer t)` being stripped off of the build.

The straightforward solution is to remove this line. But since the line's commentary
opposes to such solution, I'm starting up a discussion what exactly should be the
behavior here 😊

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Eli Zaretskii]

> Cc: Alan Mackenzie <acm@muc.de>
> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Date: Sat, 16 Nov 2024 18:11:01 +0300
> 
> CC: Alan Mackenzie, author of the change in 10083e788f7349fa363d100687dc3d94bea88f57
> 
> I've seen for a long time Emacs master builds fail from time to time in spectacular
> ways after updating the repo, sometimes so badly that `make clean` doesn't help.
> 
> I never dug into that though, but I'm attributing this to the occasional build
> messages similar to:
> 
>     Source file ‘/home/constantine/Projects/emacs/lisp/emacs-lisp/comp.el’ newer than byte-compiled file; using older file
>     Source file ‘/home/constantine/Projects/emacs/lisp/emacs-lisp/bytecomp.el’ newer than byte-compiled file; using older file
>     Source file ‘/home/constantine/Projects/emacs/lisp/emacs-lisp/comp-cstr.el’ newer than byte-compiled file; using older file
> 
> …which makes sense, because if the repo changed `comp.el` API and Emacs during the
> build of newer files is trying to make use of older `.elc` file and hence the older
> API, it may result in failure.
> 
> Got some spare time today, dug into one of the messages. From what I understand it's
> caused by this line `lisp/Makefile.in`:
> 
>     # ... but we must prefer .elc files for those in the early bootstrap.
>     compile-first: BYTE_COMPILE_FLAGS = $(BYTE_COMPILE_EXTRA_FLAGS)
> 
> >From what I understand, this rewrites BYTE_COMPILE_FLAGS to be an empty variable,
> which results in `(setq load-prefer-newer t)` being stripped off of the build.

They are supposed to be stripped only while processing the
compile-first target, and that is on purpose.  See the section
"Target-specific" in the GNU Make manual, where this feature is
documented.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Alan Mackenzie]

Hello, Konstantin.

On Sat, Nov 16, 2024 at 18:11:01 +0300, Konstantin Kharlamov wrote:
> CC: Alan Mackenzie, author of the change in 10083e788f7349fa363d100687dc3d94bea88f57

> I've seen for a long time Emacs master builds fail from time to time in spectacular
> ways after updating the repo, sometimes so badly that `make clean` doesn't help.

> I never dug into that though, but I'm attributing this to the occasional build
> messages similar to:

>     Source file ‘/home/constantine/Projects/emacs/lisp/emacs-lisp/comp.el’ newer than byte-compiled file; using older file
>     Source file ‘/home/constantine/Projects/emacs/lisp/emacs-lisp/bytecomp.el’ newer than byte-compiled file; using older file
>     Source file ‘/home/constantine/Projects/emacs/lisp/emacs-lisp/comp-cstr.el’ newer than byte-compiled file; using older file

> …which makes sense, because if the repo changed `comp.el` API and Emacs during the
> build of newer files is trying to make use of older `.elc` file and hence the older
> API, it may result in failure.

The idea is that the byte compiler is first built from .el files, giving
..elc files.  The .elc byte compiler is then used for all the other
files, since it's much faster.

> Got some spare time today, dug into one of the messages. From what I understand it's
> caused by this line `lisp/Makefile.in`:

>     # ... but we must prefer .elc files for those in the early bootstrap.
>     compile-first: BYTE_COMPILE_FLAGS = $(BYTE_COMPILE_EXTRA_FLAGS)

> >From what I understand, this rewrites BYTE_COMPILE_FLAGS to be an empty variable,
> which results in `(setq load-prefer-newer t)` being stripped off of the build.

Yes, this is to prefer the faster .elc byte compiler, whose file dates
have been set back to the epoch (1970-01-01:00:00:00 UTC).  I think this
was to ensure that when we come to native compilation, the .el source
files of the compiler will be newer than the .elc files, hence
triggering a native compilation of them.

I honestly don't believe that this mechanism for speeding up the early
build is the cause of the spectacular failures in some of your builds,
but I could be wrong.

> The straightforward solution is to remove this line. But since the line's commentary
> opposes to such solution, I'm starting up a discussion what exactly should be the
> behavior here 😊

Perhaps if you could be more specific about the failures you see, we
might be able to be of more help.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

On Sat, 2024-11-16 at 16:53 +0000, Alan Mackenzie wrote:
> Hello, Konstantin.
>
> On Sat, Nov 16, 2024 at 18:11:01 +0300, Konstantin Kharlamov wrote:
> > CC: Alan Mackenzie, author of the change in
> > 10083e788f7349fa363d100687dc3d94bea88f57
>
> > I've seen for a long time Emacs master builds fail from time to
> > time in spectacular
> > ways after updating the repo, sometimes so badly that `make clean`
> > doesn't help.
>
> > I never dug into that though, but I'm attributing this to the
> > occasional build
> > messages similar to:
>
> >     Source file ‘/home/constantine/Projects/emacs/lisp/emacs-
> > lisp/comp.el’ newer than byte-compiled file; using older file
> >     Source file ‘/home/constantine/Projects/emacs/lisp/emacs-
> > lisp/bytecomp.el’ newer than byte-compiled file; using older file
> >     Source file ‘/home/constantine/Projects/emacs/lisp/emacs-
> > lisp/comp-cstr.el’ newer than byte-compiled file; using older file
>
> > …which makes sense, because if the repo changed `comp.el` API and
> > Emacs during the
> > build of newer files is trying to make use of older `.elc` file and
> > hence the older
> > API, it may result in failure.
>
> The idea is that the byte compiler is first built from .el files,
> giving
> ..elc files.  The .elc byte compiler is then used for all the other
> files, since it's much faster.
>
> > Got some spare time today, dug into one of the messages. From what
> > I understand it's
> > caused by this line `lisp/Makefile.in`:
>
> >     # ... but we must prefer .elc files for those in the early
> > bootstrap.
> >     compile-first: BYTE_COMPILE_FLAGS = $(BYTE_COMPILE_EXTRA_FLAGS)
>
> > > From what I understand, this rewrites BYTE_COMPILE_FLAGS to be an
> > > empty variable,
> > which results in `(setq load-prefer-newer t)` being stripped off of
> > the build.
>
> Yes, this is to prefer the faster .elc byte compiler, whose file
> dates
> have been set back to the epoch (1970-01-01:00:00:00 UTC).  I think
> this
> was to ensure that when we come to native compilation, the .el source
> files of the compiler will be newer than the .elc files, hence
> triggering a native compilation of them.
>
> I honestly don't believe that this mechanism for speeding up the
> early
> build is the cause of the spectacular failures in some of your
> builds,
> but I could be wrong.
>
> > The straightforward solution is to remove this line. But since the
> > line's commentary
> > opposes to such solution, I'm starting up a discussion what exactly
> > should be the
> > behavior here 😊
>
> Perhaps if you could be more specific about the failures you see, we
> might be able to be of more help.

Sure, I just reproduced it after removing all `.elc` files in the repo,
here how:

1. `git checkout f2f13fa630b` (a commit from April)
2. `make -j$(nproc)` to compile. Note: you don't need to wait for build
to finish, I just waited for all files under `lisp/emacs-lisp`
directory to finish compilation, and then ^C'ed it.
3. `git checkout 29098a291f5` (a November commit).
4. `make -j$(nproc)`

## Expected

Build finishes

## Actual

It fails with stacktrace so huge that it goes beyond Konsole scrollback
and with the following message:


    Loading macroexp.elc (compiled; note, source file is newer)...
    Wrong type argument: "../../lisp/emacs-lisp/comp.el", hash-table-p, (#s(byte-to-native-top-level (byte-code "��!���!���!���!���!���!���!���!������%�����DD�����&        �����DD�����&   �����DD���&�����DD�����&������DD�����&������DD�����&������DD�����&�" [require bytecomp cl-lib gv rx subr-x warnings comp-common comp-cstr custom-declare-group ...] 10) t) #s(byte-to-native-top-level (#<symbol defvar at 4425> #<symbol no-native-compile at 4432> nil "Non-nil to prevent native-compiling of Emacs Lisp code.
    Note that when `no-byte-compile' is set to non-nil it overrides the value of
    `no-native-compile'.
    This is normally set in local file variables at the end of the
    Emacs Lisp file:

You can see it fails on the `comp.el`, and I presume the reason is it simply didn't rebuild the necessary .elc files, and instead loaded outdated bytecode.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Eli Zaretskii]

> Cc: 74382@debbugs.gnu.org
> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Date: Sat, 16 Nov 2024 20:45:47 +0300
> 
> ## Expected
> 
> Build finishes
> 
> ## Actual
> 
> It fails with stacktrace so huge that it goes beyond Konsole scrollback
> and with the following message:
> 
> 
>     Loading macroexp.elc (compiled; note, source file is newer)...
>     Wrong type argument: "../../lisp/emacs-lisp/comp.el", hash-table-p, (#s(byte-to-native-top-level (byte-code "��!���!���!���!���!���!���!���!������%�����DD�����&        �����DD�����&   �����DD���&�����DD�����&������DD�����&������DD�����&������DD�����&�" [require bytecomp cl-lib gv rx subr-x warnings comp-common comp-cstr custom-declare-group ...] 10) t) #s(byte-to-native-top-level (#<symbol defvar at 4425> #<symbol no-native-compile at 4432> nil "Non-nil to prevent native-compiling of Emacs Lisp code.
>     Note that when `no-byte-compile' is set to non-nil it overrides the value of
>     `no-native-compile'.
>     This is normally set in local file variables at the end of the
>     Emacs Lisp file:

I don't think this is related to what lisp/Makefile does.  When
macroexp.el is updated, builds are known to fail until you remove
macroexp.elc (or bootstrap).

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

On Sat, 2024-11-16 at 20:38 +0200, Eli Zaretskii wrote:
> > Cc: 74382@debbugs.gnu.org
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Date: Sat, 16 Nov 2024 20:45:47 +0300
> > 
> > ## Expected
> > 
> > Build finishes
> > 
> > ## Actual
> > 
> > It fails with stacktrace so huge that it goes beyond Konsole
> > scrollback
> > and with the following message:
> > 
> > 
> >     Loading macroexp.elc (compiled; note, source file is newer)...
> >     Wrong type argument: "../../lisp/emacs-lisp/comp.el", hash-
> > table-p, (#s(byte-to-native-top-level (byte-code
> > "��!���!���!���!���!���!���!���!������%�����DD�����&       
> > �����DD�����&  
> > �����DD���&�����DD�����&������DD�����&������DD�����&������DD�����&�
> > " [require bytecomp cl-lib gv rx subr-x warnings comp-common comp-
> > cstr custom-declare-group ...] 10) t) #s(byte-to-native-top-level
> > (#<symbol defvar at 4425> #<symbol no-native-compile at 4432> nil
> > "Non-nil to prevent native-compiling of Emacs Lisp code.
> >     Note that when `no-byte-compile' is set to non-nil it overrides
> > the value of
> >     `no-native-compile'.
> >     This is normally set in local file variables at the end of the
> >     Emacs Lisp file:
> 
> I don't think this is related to what lisp/Makefile does.  When
> macroexp.el is updated, builds are known to fail until you remove
> macroexp.elc (or bootstrap).

Okay, but why you need to remove macroexp.elc, isn't it the build
system job to rebuild it when it's changed? I think so, and the
macroexp.el is among the `COMPILE_FIRST` files in lisp/Makefile.in.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Eli Zaretskii]

> Cc: acm@muc.de, 74382@debbugs.gnu.org
> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Date: Sat, 16 Nov 2024 21:43:41 +0300
> 
> > I don't think this is related to what lisp/Makefile does.  When
> > macroexp.el is updated, builds are known to fail until you remove
> > macroexp.elc (or bootstrap).
> 
> Okay, but why you need to remove macroexp.elc

Because macroexp.el contains macros, which might have been already
expanded in the .elc file(s).

> isn't it the build system job to rebuild it when it's changed?

It's impractical, because we have many files with macros.  Tracking
all of those dependencies would mean that changes in any file will
trigger unnecessary recompilation of many other files.  If you don't
mind spending that time waiting for the build, just "make bootstrap"
every time you update from Git, and you will have that.

> I think so, and the
> macroexp.el is among the `COMPILE_FIRST` files in lisp/Makefile.in.

Yes, because it is needed to bootstrap the native-compilation stuff,
and using the .el file makes that extremely slow.

There's no bug here.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

On Sat, 2024-11-16 at 22:00 +0200, Eli Zaretskii wrote:
> > Cc: acm@muc.de, 74382@debbugs.gnu.org
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Date: Sat, 16 Nov 2024 21:43:41 +0300
> > 
> > > I don't think this is related to what lisp/Makefile does.  When
> > > macroexp.el is updated, builds are known to fail until you remove
> > > macroexp.elc (or bootstrap).
> > 
> > Okay, but why you need to remove macroexp.elc
> 
> Because macroexp.el contains macros, which might have been already
> expanded in the .elc file(s).
> 
> > isn't it the build system job to rebuild it when it's changed?
> 
> It's impractical, because we have many files with macros.  Tracking
> all of those dependencies would mean that changes in any file will
> trigger unnecessary recompilation of many other files.  If you don't
> mind spending that time waiting for the build, just "make bootstrap"
> every time you update from Git, and you will have that.

Unless I'm missing something, the problem seems to be with one exact
file, macroexpand.elc, and not with others. So the algorithm is simple:
if `macroexpand.el` was modified, remove its elc file. You don't need
to track any dependencies.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: acm@muc.de, 74382@debbugs.gnu.org
> Date: Sun, 17 Nov 2024 01:54:48 +0300
> 
> > It's impractical, because we have many files with macros.  Tracking
> > all of those dependencies would mean that changes in any file will
> > trigger unnecessary recompilation of many other files.  If you don't
> > mind spending that time waiting for the build, just "make bootstrap"
> > every time you update from Git, and you will have that.
> 
> Unless I'm missing something, the problem seems to be with one exact
> file, macroexpand.elc, and not with others.

No, that's not true.  I had similar problems with basically all the
files in COMPILE_FIRST.

More importantly, what you seem to be missing is that we deliberately
play with the time stamps of the *.elc files in COMPILE_FIRST (search
for "UTC" in the Makefile), so we must not use load-prefer-newer in
this case.  That is the reason why it's removed from
BYTE_COMPILE_FLAGS.

> So the algorithm is simple: if `macroexpand.el` was modified, remove
> its elc file. You don't need to track any dependencies.

How will load-prefer-newer help if this is what you do?  That's the
trigger for this bug report, no?

In any case, this is not the reason why load-prefer-newer is removed
while we COMPILE_FIRST; see above.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Gerd Möllmann]

Konstantin Kharlamov <Hi-Angel@yandex.ru> writes:

> Sure, I just reproduced it after removing all `.elc` files in the repo,
> here how:
>
> 1. `git checkout f2f13fa630b` (a commit from April)
> 2. `make -j$(nproc)` to compile. Note: you don't need to wait for build
> to finish, I just waited for all files under `lisp/emacs-lisp`
> directory to finish compilation, and then ^C'ed it.
> 3. `git checkout 29098a291f5` (a November commit).
> 4. `make -j$(nproc)`

This would always work if lisp/Makefile would rm the .elc files from
COMPILE_FIRST, right? I suspect this isn't done to speed up the build in
the usual case, and because it's a bit difficult to automatically
determine if it has to done or not.

Does a "make clean" after the checkout in (3) make it work?

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

On Sun, 2024-11-17 at 08:25 +0100, Gerd Möllmann wrote:
> Konstantin Kharlamov <Hi-Angel@yandex.ru> writes:
> 
> > Sure, I just reproduced it after removing all `.elc` files in the
> > repo,
> > here how:
> > 
> > 1. `git checkout f2f13fa630b` (a commit from April)
> > 2. `make -j$(nproc)` to compile. Note: you don't need to wait for
> > build
> > to finish, I just waited for all files under `lisp/emacs-lisp`
> > directory to finish compilation, and then ^C'ed it.
> > 3. `git checkout 29098a291f5` (a November commit).
> > 4. `make -j$(nproc)`
> 
> This would always work if lisp/Makefile would rm the .elc files from
> COMPILE_FIRST, right? I suspect this isn't done to speed up the build
> in
> the usual case, and because it's a bit difficult to automatically
> determine if it has to done or not.
> 
> Does a "make clean" after the checkout in (3) make it work?

I don't think so, because `make clean` for some reason doesn't remove
`.elc` artifacts.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

On Sun, 2024-11-17 at 08:44 +0200, Eli Zaretskii wrote:
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Cc: acm@muc.de, 74382@debbugs.gnu.org
> > Date: Sun, 17 Nov 2024 01:54:48 +0300
> > 
> > > It's impractical, because we have many files with macros. 
> > > Tracking
> > > all of those dependencies would mean that changes in any file
> > > will
> > > trigger unnecessary recompilation of many other files.  If you
> > > don't
> > > mind spending that time waiting for the build, just "make
> > > bootstrap"
> > > every time you update from Git, and you will have that.
> > 
> > Unless I'm missing something, the problem seems to be with one
> > exact
> > file, macroexpand.elc, and not with others.
> 
> No, that's not true.  I had similar problems with basically all the
> files in COMPILE_FIRST.
> 
> More importantly, what you seem to be missing is that we deliberately
> play with the time stamps of the *.elc files in COMPILE_FIRST (search
> for "UTC" in the Makefile), so we must not use load-prefer-newer in
> this case.  That is the reason why it's removed from
> BYTE_COMPILE_FLAGS.
> 
> > So the algorithm is simple: if `macroexpand.el` was modified,
> > remove
> > its elc file. You don't need to track any dependencies.
> 
> How will load-prefer-newer help if this is what you do?  That's the
> trigger for this bug report, no?
> 
> In any case, this is not the reason why load-prefer-newer is removed
> while we COMPILE_FIRST; see above.

Alright, for more efficient discussion I think I need to dig into how
this all work in different situations and measure performance, to come
up with some suggestions, but I'm afraid ATM I just don't have the
spare time, so maybe let's close the discussion for now…

I just see that COMPILE_FIRST files should never be used while being
stale, but for performance reasons they are used stale. I don't think
this is the case where the correctness can be traded off for
performance, but it's just my opinion.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Eli Zaretskii]

> Cc: Alan Mackenzie <acm@muc.de>, 74382@debbugs.gnu.org
> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Date: Sun, 17 Nov 2024 18:21:36 +0300
> 
> On Sun, 2024-11-17 at 08:25 +0100, Gerd Möllmann wrote:
> > Konstantin Kharlamov <Hi-Angel@yandex.ru> writes:
> > 
> > > Sure, I just reproduced it after removing all `.elc` files in the
> > > repo,
> > > here how:
> > > 
> > > 1. `git checkout f2f13fa630b` (a commit from April)
> > > 2. `make -j$(nproc)` to compile. Note: you don't need to wait for
> > > build
> > > to finish, I just waited for all files under `lisp/emacs-lisp`
> > > directory to finish compilation, and then ^C'ed it.
> > > 3. `git checkout 29098a291f5` (a November commit).
> > > 4. `make -j$(nproc)`
> > 
> > This would always work if lisp/Makefile would rm the .elc files from
> > COMPILE_FIRST, right? I suspect this isn't done to speed up the build
> > in
> > the usual case, and because it's a bit difficult to automatically
> > determine if it has to done or not.
> > 
> > Does a "make clean" after the checkout in (3) make it work?
> 
> I don't think so, because `make clean` for some reason doesn't remove
> `.elc` artifacts.

And it shouldn't, because *.elc files are part of a release tarball.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Gerd Möllmann]

Konstantin Kharlamov <Hi-Angel@yandex.ru> writes:

> On Sun, 2024-11-17 at 08:25 +0100, Gerd Möllmann wrote:
>> Konstantin Kharlamov <Hi-Angel@yandex.ru> writes:
>> 
>> > Sure, I just reproduced it after removing all `.elc` files in the
>> > repo,
>> > here how:
>> > 
>> > 1. `git checkout f2f13fa630b` (a commit from April)
>> > 2. `make -j$(nproc)` to compile. Note: you don't need to wait for
>> > build
>> > to finish, I just waited for all files under `lisp/emacs-lisp`
>> > directory to finish compilation, and then ^C'ed it.
>> > 3. `git checkout 29098a291f5` (a November commit).
>> > 4. `make -j$(nproc)`
>> 
>> This would always work if lisp/Makefile would rm the .elc files from
>> COMPILE_FIRST, right? I suspect this isn't done to speed up the build
>> in
>> the usual case, and because it's a bit difficult to automatically
>> determine if it has to done or not.
>> 
>> Does a "make clean" after the checkout in (3) make it work?
>
> I don't think so, because `make clean` for some reason doesn't remove
> `.elc` artifacts.

Ah, right, it is bootstrap-clean that removes the .elc files.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

On Sun, 2024-11-17 at 17:37 +0200, Eli Zaretskii wrote:
> > Cc: Alan Mackenzie <acm@muc.de>, 74382@debbugs.gnu.org
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Date: Sun, 17 Nov 2024 18:21:36 +0300
> > 
> > On Sun, 2024-11-17 at 08:25 +0100, Gerd Möllmann wrote:
> > > Konstantin Kharlamov <Hi-Angel@yandex.ru> writes:
> > > 
> > > > Sure, I just reproduced it after removing all `.elc` files in
> > > > the
> > > > repo,
> > > > here how:
> > > > 
> > > > 1. `git checkout f2f13fa630b` (a commit from April)
> > > > 2. `make -j$(nproc)` to compile. Note: you don't need to wait
> > > > for
> > > > build
> > > > to finish, I just waited for all files under `lisp/emacs-lisp`
> > > > directory to finish compilation, and then ^C'ed it.
> > > > 3. `git checkout 29098a291f5` (a November commit).
> > > > 4. `make -j$(nproc)`
> > > 
> > > This would always work if lisp/Makefile would rm the .elc files
> > > from
> > > COMPILE_FIRST, right? I suspect this isn't done to speed up the
> > > build
> > > in
> > > the usual case, and because it's a bit difficult to automatically
> > > determine if it has to done or not.
> > > 
> > > Does a "make clean" after the checkout in (3) make it work?
> > 
> > I don't think so, because `make clean` for some reason doesn't
> > remove
> > `.elc` artifacts.
> 
> And it shouldn't, because *.elc files are part of a release tarball.

But if someone decided to build from a release tarball, sure they can
produce .elc files as well, can't they?

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: gerd.moellmann@gmail.com, acm@muc.de, 74382@debbugs.gnu.org
> Date: Sun, 17 Nov 2024 18:43:17 +0300
> 
> On Sun, 2024-11-17 at 17:37 +0200, Eli Zaretskii wrote:
> > > Cc: Alan Mackenzie <acm@muc.de>, 74382@debbugs.gnu.org
> > > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > > Date: Sun, 17 Nov 2024 18:21:36 +0300
> > > 
> > > On Sun, 2024-11-17 at 08:25 +0100, Gerd Möllmann wrote:
> > > > Konstantin Kharlamov <Hi-Angel@yandex.ru> writes:
> > > > 
> > > > > Sure, I just reproduced it after removing all `.elc` files in
> > > > > the
> > > > > repo,
> > > > > here how:
> > > > > 
> > > > > 1. `git checkout f2f13fa630b` (a commit from April)
> > > > > 2. `make -j$(nproc)` to compile. Note: you don't need to wait
> > > > > for
> > > > > build
> > > > > to finish, I just waited for all files under `lisp/emacs-lisp`
> > > > > directory to finish compilation, and then ^C'ed it.
> > > > > 3. `git checkout 29098a291f5` (a November commit).
> > > > > 4. `make -j$(nproc)`
> > > > 
> > > > This would always work if lisp/Makefile would rm the .elc files
> > > > from
> > > > COMPILE_FIRST, right? I suspect this isn't done to speed up the
> > > > build
> > > > in
> > > > the usual case, and because it's a bit difficult to automatically
> > > > determine if it has to done or not.
> > > > 
> > > > Does a "make clean" after the checkout in (3) make it work?
> > > 
> > > I don't think so, because `make clean` for some reason doesn't
> > > remove
> > > `.elc` artifacts.
> > 
> > And it shouldn't, because *.elc files are part of a release tarball.
> 
> But if someone decided to build from a release tarball, sure they can
> produce .elc files as well, can't they?

No, Emacs release tarballs come with *.elc files, and users shouldn't
recompile them.  For starters, it makes the build significantly
longer, besides being unnecessary.

Recompiling *.elc files is only needed if the corresponding *.el files
are modified, something that doesn't normally happen when you build a
release tarball.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

On Sun, 2024-11-17 at 17:53 +0200, Eli Zaretskii wrote:
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Cc: gerd.moellmann@gmail.com, acm@muc.de, 74382@debbugs.gnu.org
> > Date: Sun, 17 Nov 2024 18:43:17 +0300
> > 
> > On Sun, 2024-11-17 at 17:37 +0200, Eli Zaretskii wrote:
> > > > Cc: Alan Mackenzie <acm@muc.de>, 74382@debbugs.gnu.org
> > > > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > > > Date: Sun, 17 Nov 2024 18:21:36 +0300
> > > > 
> > > > On Sun, 2024-11-17 at 08:25 +0100, Gerd Möllmann wrote:
> > > > > Konstantin Kharlamov <Hi-Angel@yandex.ru> writes:
> > > > > 
> > > > > > Sure, I just reproduced it after removing all `.elc` files
> > > > > > in
> > > > > > the
> > > > > > repo,
> > > > > > here how:
> > > > > > 
> > > > > > 1. `git checkout f2f13fa630b` (a commit from April)
> > > > > > 2. `make -j$(nproc)` to compile. Note: you don't need to
> > > > > > wait
> > > > > > for
> > > > > > build
> > > > > > to finish, I just waited for all files under `lisp/emacs-
> > > > > > lisp`
> > > > > > directory to finish compilation, and then ^C'ed it.
> > > > > > 3. `git checkout 29098a291f5` (a November commit).
> > > > > > 4. `make -j$(nproc)`
> > > > > 
> > > > > This would always work if lisp/Makefile would rm the .elc
> > > > > files
> > > > > from
> > > > > COMPILE_FIRST, right? I suspect this isn't done to speed up
> > > > > the
> > > > > build
> > > > > in
> > > > > the usual case, and because it's a bit difficult to
> > > > > automatically
> > > > > determine if it has to done or not.
> > > > > 
> > > > > Does a "make clean" after the checkout in (3) make it work?
> > > > 
> > > > I don't think so, because `make clean` for some reason doesn't
> > > > remove
> > > > `.elc` artifacts.
> > > 
> > > And it shouldn't, because *.elc files are part of a release
> > > tarball.
> > 
> > But if someone decided to build from a release tarball, sure they
> > can
> > produce .elc files as well, can't they?
> 
> No, Emacs release tarballs come with *.elc files, and users shouldn't
> recompile them.  For starters, it makes the build significantly
> longer, besides being unnecessary.
> 
> Recompiling *.elc files is only needed if the corresponding *.el
> files
> are modified, something that doesn't normally happen when you build a
> release tarball.

Okay. So, how about a compromise here: provide release tarball with
modified Makefiles which upon calling `make clean` would not remove
`.elc` files — but let `make clean` inside git-repository remove elc
files?

Users expect `make clean` to remove non-config-related bulid artifacts.
Which includes `.elc` files. I can't count how many times I was
forgetting about this peculiarity of Emacs build system, and after
finding out that even `make clean` doesn't help with build errors (due
to COMPILE_FIRST files stuff), I had to nuke everything with `git clean
-fdx`. Even Gerd in this discussion forgot about this peculiarity — and
Gerd unlike me is a regular Emacs developer.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: gerd.moellmann@gmail.com, acm@muc.de, 74382@debbugs.gnu.org
> Date: Sun, 17 Nov 2024 19:04:42 +0300
> 
> On Sun, 2024-11-17 at 17:53 +0200, Eli Zaretskii wrote:
> > No, Emacs release tarballs come with *.elc files, and users shouldn't
> > recompile them.  For starters, it makes the build significantly
> > longer, besides being unnecessary.
> > 
> > Recompiling *.elc files is only needed if the corresponding *.el
> > files
> > are modified, something that doesn't normally happen when you build a
> > release tarball.
> 
> Okay. So, how about a compromise here: provide release tarball with
> modified Makefiles which upon calling `make clean` would not remove
> `.elc` files — but let `make clean` inside git-repository remove elc
> files?

We already have a special target for that: maintainer-clean.  There's
no need to make such confusing differences between what "make clean"
does in a tarball and in Git.  That's a standard GNU target, so it
should do what the GNU Coding Standards say, and do it consistently.

Removing all the *.elc files (and a few *.el files that are generated
by the build from Git) makes the build much longer, so doing so should
be harder and rarer, not easier and more frequent.

> Users expect `make clean` to remove non-config-related bulid artifacts.
> Which includes `.elc` files. I can't count how many times I was
> forgetting about this peculiarity of Emacs build system, and after
> finding out that even `make clean` doesn't help with build errors (due
> to COMPILE_FIRST files stuff), I had to nuke everything with `git clean
> -fdx`. Even Gerd in this discussion forgot about this peculiarity — and
> Gerd unlike me is a regular Emacs developer.

You will have to get used to this curiosity of the Emacs build system,
sorry.  The main audience of the build stuff in Git is Emacs
developers, so everyone else have to adapt.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

On Sun, 2024-11-17 at 18:29 +0200, Eli Zaretskii wrote:
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Cc: gerd.moellmann@gmail.com, acm@muc.de, 74382@debbugs.gnu.org
> > Date: Sun, 17 Nov 2024 19:04:42 +0300
> > 
> > On Sun, 2024-11-17 at 17:53 +0200, Eli Zaretskii wrote:
> > > No, Emacs release tarballs come with *.elc files, and users
> > > shouldn't
> > > recompile them.  For starters, it makes the build significantly
> > > longer, besides being unnecessary.
> > > 
> > > Recompiling *.elc files is only needed if the corresponding *.el
> > > files
> > > are modified, something that doesn't normally happen when you
> > > build a
> > > release tarball.
> > 
> > Okay. So, how about a compromise here: provide release tarball with
> > modified Makefiles which upon calling `make clean` would not remove
> > `.elc` files — but let `make clean` inside git-repository remove
> > elc
> > files?
> 
> We already have a special target for that: maintainer-clean.  There's
> no need to make such confusing differences between what "make clean"
> does in a tarball and in Git.  That's a standard GNU target, so it
> should do what the GNU Coding Standards say, and do it consistently.

GNU Coding Standard section for `make clean` says, quoting:

> Delete all files […] that are normally created by building the
program. However, don’t delete the files that record the configuration.
Also preserve files that could be made by building, but normally aren’t
because the distribution comes with them.

The "git distribution" doesn't come with .elc files, hence .elc files
should be removed by `make clean` if run in the git repository. That's
what the standard says.

> Removing all the *.elc files (and a few *.el files that are generated
> by the build from Git) makes the build much longer, so doing so
> should
> be harder and rarer, not easier and more frequent.

You are optimizing for the wrong people. The auditory for release
tarballs are package maintainers, and they couldn't care less if `make
clean` removes all artifacts or not, because: 1. Nowadays the majority
don't package on their personal machines anyway and use CI, and 2. they
don't typically execute `make clean`.

This "don't clean elc files during `make clean`" hurts Emacs devs and
contributors, while gaining nothing in return.

> > Users expect `make clean` to remove non-config-related bulid
> > artifacts.
> > Which includes `.elc` files. I can't count how many times I was
> > forgetting about this peculiarity of Emacs build system, and after
> > finding out that even `make clean` doesn't help with build errors
> > (due
> > to COMPILE_FIRST files stuff), I had to nuke everything with `git
> > clean
> > -fdx`. Even Gerd in this discussion forgot about this peculiarity —
> > and
> > Gerd unlike me is a regular Emacs developer.
> 
> You will have to get used to this curiosity of the Emacs build
> system,
> sorry.  The main audience of the build stuff in Git is Emacs
> developers, so everyone else have to adapt.

I don't think Emacs developers are using release tarballs, so this
"curiosity" isn't helping them.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: gerd.moellmann@gmail.com, acm@muc.de, 74382@debbugs.gnu.org
> Date: Sun, 17 Nov 2024 19:46:25 +0300
> 
> On Sun, 2024-11-17 at 18:29 +0200, Eli Zaretskii wrote:
> > We already have a special target for that: maintainer-clean.  There's
> > no need to make such confusing differences between what "make clean"
> > does in a tarball and in Git.  That's a standard GNU target, so it
> > should do what the GNU Coding Standards say, and do it consistently.
> 
> GNU Coding Standard section for `make clean` says, quoting:
> 
> > Delete all files […] that are normally created by building the
> program. However, don’t delete the files that record the configuration.
> Also preserve files that could be made by building, but normally aren’t
> because the distribution comes with them.
> 
> The "git distribution" doesn't come with .elc files, hence .elc files
> should be removed by `make clean` if run in the git repository. That's
> what the standard says.

There's no "Git distribution", so this doesn't apply.

Once again, it is more important to me that "make clean" does the same
in every case than anything else.

> This "don't clean elc files during `make clean`" hurts Emacs devs and
> contributors, while gaining nothing in return.

I disagree.

> > You will have to get used to this curiosity of the Emacs build
> > system,
> > sorry.  The main audience of the build stuff in Git is Emacs
> > developers, so everyone else have to adapt.
> 
> I don't think Emacs developers are using release tarballs, so this
> "curiosity" isn't helping them.

The curiosity is for those who build from tarballs, whoever they are.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

On Sun, 2024-11-17 at 19:09 +0200, Eli Zaretskii wrote:
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Cc: gerd.moellmann@gmail.com, acm@muc.de, 74382@debbugs.gnu.org
> > Date: Sun, 17 Nov 2024 19:46:25 +0300
> > 
> > On Sun, 2024-11-17 at 18:29 +0200, Eli Zaretskii wrote:
> > > We already have a special target for that: maintainer-clean. 
> > > There's
> > > no need to make such confusing differences between what "make
> > > clean"
> > > does in a tarball and in Git.  That's a standard GNU target, so
> > > it
> > > should do what the GNU Coding Standards say, and do it
> > > consistently.
> > 
> > GNU Coding Standard section for `make clean` says, quoting:
> > 
> > > Delete all files […] that are normally created by building the
> > program. However, don’t delete the files that record the
> > configuration.
> > Also preserve files that could be made by building, but normally
> > aren’t
> > because the distribution comes with them.
> > 
> > The "git distribution" doesn't come with .elc files, hence .elc
> > files
> > should be removed by `make clean` if run in the git repository.
> > That's
> > what the standard says.
> 
> There's no "Git distribution", so this doesn't apply.

The Cambridge Dictionary defines word "distribution" as¹:

> the process of giving things out to several people, or spreading or
supplying something

Git repo provides people with Emacs sources, so that does apply.

> Once again, it is more important to me that "make clean" does the
> same
> in every case than anything else.
> 
> > This "don't clean elc files during `make clean`" hurts Emacs devs
> > and
> > contributors, while gaining nothing in return.
> 
> I disagree.

Well, since we ruled out the distro packagers as the auditory for the
`make clean`, who else do you see would benefit from it?

> > > You will have to get used to this curiosity of the Emacs build
> > > system,
> > > sorry.  The main audience of the build stuff in Git is Emacs
> > > developers, so everyone else have to adapt.
> > 
> > I don't think Emacs developers are using release tarballs, so this
> > "curiosity" isn't helping them.
> 
> The curiosity is for those who build from tarballs, whoever they are.

Here's the thing, the `foo clean` target in any build system in 95% of
cases in my experience is only used to work around the bugs in how the
project set up the build system, more specifically when compilation
command doesn't rebuild project properly. In Emacs we already know that
the bug is only with those COMPILE_FIRST files. A release tarball user
is very unlikely to modify exactly those files and exactly in a way to
would lead to the problem. Hence the user is very unlikely to use `make
clean` whatsoever. And then, even if they do catch the bug, they again
*do* want to get rid of the offending files.

So overall, I just don't see who would ever want `make clean` not to
remove `.elc` files, even among tarball users.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Gerd Möllmann]

Konstantin Kharlamov <Hi-Angel@yandex.ru> writes:

> Even Gerd in this discussion forgot about this peculiarity — and
> Gerd unlike me is a regular Emacs developer.

Even worse, even worse: He wrote that stuff! But I think I'll excuse him
because that was 25 years ago, and he took a 20 year break from Emacs,
after stepping down, and he's old of course :-).

Seriously, maybe knowing a bit of history helps understand the current
situation wrt .elc files? One wouldn't believe it nowadays, but they
were originally in version control, i.e. RCS, and later CVS. I didn't
want that in the public CVS repo we set up for Emacs 21, so I added the
ability to bootstrap and removed the .elc files from CVS. COMPILE_FIRST
and so on are part of the bootstrapping support.

Rewriting the build system to follow some standard (if it existed back
then, which I don't remember), was not important to me. There were so
many other things to do.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

On Mon, 2024-11-18 at 05:06 +0100, Gerd Möllmann wrote:
> Konstantin Kharlamov <Hi-Angel@yandex.ru> writes:
> 
> > Even Gerd in this discussion forgot about this peculiarity — and
> > Gerd unlike me is a regular Emacs developer.
> 
> Even worse, even worse: He wrote that stuff! But I think I'll excuse
> him
> because that was 25 years ago, and he took a 20 year break from
> Emacs,
> after stepping down, and he's old of course :-).
> 
> Seriously, maybe knowing a bit of history helps understand the
> current
> situation wrt .elc files? One wouldn't believe it nowadays, but they
> were originally in version control, i.e. RCS, and later CVS. I didn't
> want that in the public CVS repo we set up for Emacs 21, so I added
> the
> ability to bootstrap and removed the .elc files from CVS.
> COMPILE_FIRST
> and so on are part of the bootstrapping support.
> 
> Rewriting the build system to follow some standard (if it existed
> back
> then, which I don't remember), was not important to me. There were so
> many other things to do.

Haha, well, everything gets improved iteratively, so that's okay 😊

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

On Mon, 2024-11-18 at 05:06 +0100, Gerd Möllmann wrote:
> Konstantin Kharlamov <Hi-Angel@yandex.ru> writes:
> 
> > Even Gerd in this discussion forgot about this peculiarity — and
> > Gerd unlike me is a regular Emacs developer.
> 
> Even worse, even worse: He wrote that stuff! But I think I'll excuse
> him
> because that was 25 years ago, and he took a 20 year break from
> Emacs,
> after stepping down, and he's old of course :-).
> 
> Seriously, maybe knowing a bit of history helps understand the
> current
> situation wrt .elc files? One wouldn't believe it nowadays, but they
> were originally in version control, i.e. RCS, and later CVS. I didn't
> want that in the public CVS repo we set up for Emacs 21, so I added
> the
> ability to bootstrap and removed the .elc files from CVS.
> COMPILE_FIRST
> and so on are part of the bootstrapping support.

Btw, thank you, this bit of history indeed is interesting. During whole
discussion I had a question on the back of my mind: how this
"distribute pre-built elc in tarballs" idea initially came to be. I
mean, it's kind of nice from POV of saving a bit of energy around the
world on CI machines, but I don't see much beyond that. Building elc
files is not *that* bad for elc distribution to be strictly necessary.

Now that you told this, I realize it's just a solution to a problem
from 25 years old back, from times when that actually was a problem.

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: Eli Zaretskii <eliz@gnu.org>, acm@muc.de, 74382@debbugs.gnu.org
> Date: Mon, 18 Nov 2024 13:05:54 +0300
> 
> Btw, thank you, this bit of history indeed is interesting. During whole
> discussion I had a question on the back of my mind: how this
> "distribute pre-built elc in tarballs" idea initially came to be. I
> mean, it's kind of nice from POV of saving a bit of energy around the
> world on CI machines, but I don't see much beyond that. Building elc
> files is not *that* bad for elc distribution to be strictly necessary.

Once again, building all the *.elc files takes a long time, even on
modern systems.  I have a 32-core screamer, and it still takes a few
minutes to byte-compile everything.  On an older system, it used to
take me 15 minutes even in parallel (-j4) builds.

Computers got much faster, but people know that, so they have less
patience.  Thus, avoiding recompilation of the *.elc files (and Info,
and other derived files) is still important to make the build faster.
A release tarball builds in less than 1 min due to these measures.

> Now that you told this, I realize it's just a solution to a problem
> from 25 years old back, from times when that actually was a problem.

I don't understand this conclusion.  What problem existed 25 years ago
that is related to "make clean" or to load-prefer-newer, and doesn't
exist anymore?

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Konstantin Kharlamov]

On Mon, 2024-11-18 at 14:59 +0200, Eli Zaretskii wrote:
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Cc: Eli Zaretskii <eliz@gnu.org>, acm@muc.de, 74382@debbugs.gnu.org
> > Date: Mon, 18 Nov 2024 13:05:54 +0300
> > 
> > Btw, thank you, this bit of history indeed is interesting. During
> > whole
> > discussion I had a question on the back of my mind: how this
> > "distribute pre-built elc in tarballs" idea initially came to be. I
> > mean, it's kind of nice from POV of saving a bit of energy around
> > the
> > world on CI machines, but I don't see much beyond that. Building
> > elc
> > files is not *that* bad for elc distribution to be strictly
> > necessary.
> 
> Once again, building all the *.elc files takes a long time, even on
> modern systems.  I have a 32-core screamer, and it still takes a few
> minutes to byte-compile everything.  On an older system, it used to
> take me 15 minutes even in parallel (-j4) builds.
> 
> Computers got much faster, but people know that, so they have less
> patience.  Thus, avoiding recompilation of the *.elc files (and Info,
> and other derived files) is still important to make the build faster.
> A release tarball builds in less than 1 min due to these measures.

3 and even 15 minutes of compilation once a few months at worst (the
time between Emacs releases) is not a big deal. Besides, the endusers
don't typically compile releases, instead distro packagers do that, and
they are typically using CI.

Emacs by far is not the slowest project to compile from scratch. AFAIR
LibreOffce and Linux Kernel take longer to build.

> > Now that you told this, I realize it's just a solution to a problem
> > from 25 years old back, from times when that actually was a
> > problem.
> 
> I don't understand this conclusion.  What problem existed 25 years
> ago
> that is related to "make clean" or to load-prefer-newer, and doesn't
> exist anymore?

This is tangentially related to `make clean` discussoin. I was just
curious how come that Emacs started distributing elc files in release
tarballs. Like, was there someone who asked for that, or anything… Now
I see it's just a feature from long past. I am not saying it's bad or
good, because that's a separate question (there are both pros and
cons).

bug#74382: `compile-first` Make rule is no longer using `load-prefer-newer`

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: gerd.moellmann@gmail.com, 74382@debbugs.gnu.org
> Date: Mon, 18 Nov 2024 16:12:08 +0300
> 
> > Once again, building all the *.elc files takes a long time, even on
> > modern systems.  I have a 32-core screamer, and it still takes a few
> > minutes to byte-compile everything.  On an older system, it used to
> > take me 15 minutes even in parallel (-j4) builds.
> > 
> > Computers got much faster, but people know that, so they have less
> > patience.  Thus, avoiding recompilation of the *.elc files (and Info,
> > and other derived files) is still important to make the build faster.
> > A release tarball builds in less than 1 min due to these measures.
> 
> 3 and even 15 minutes of compilation once a few months at worst (the
> time between Emacs releases) is not a big deal. Besides, the endusers
> don't typically compile releases, instead distro packagers do that, and
> they are typically using CI.

That's your opinions, not mine.  From my POV, having these files in
the tarball makes the build much faster and also much more reliable
and correct.  That means a lot, even if you don't value that.

> Emacs by far is not the slowest project to compile from scratch. AFAIR
> LibreOffce and Linux Kernel take longer to build.

So we are supposed to judge ourselves by the lowest common
denominator?

> This is tangentially related to `make clean` discussoin. I was just
> curious how come that Emacs started distributing elc files in release
> tarballs.

Any project that doesn't distribute platform-independent files in its
tarball does a disservice to its users.  There's absolutely no reason
not to include them, and more than one to include: time it takes to
build them, tools required for building them that are otherwise not
needed, etc.