Updating autoloads, custom-deps, finder-data

Updating autoloads, custom-deps, finder-data

[Eli Zaretskii]

When any of the Lisp files are modified, typing "make" in the
top-level directory byte-compiles those modified files, as expected,
but does not invoke "make autoloads", "make custom-deps", and "make
finder-data" targets.  Thus the autoloads and the other meta-data
regarding the modified Lisp files slowly bit-rots, unless you manually
invoke those targets or perform a full bootstrap.

Is this a bug in the build procedure, or am I missing something?

Re: Updating autoloads, custom-deps, finder-data

[Eli Zaretskii]

> Date: Sat, 20 Apr 2013 11:10:08 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> When any of the Lisp files are modified, typing "make" in the
> top-level directory byte-compiles those modified files, as expected,
> but does not invoke "make autoloads", "make custom-deps", and "make
> finder-data" targets.  Thus the autoloads and the other meta-data
> regarding the modified Lisp files slowly bit-rots, unless you manually
> invoke those targets or perform a full bootstrap.
> 
> Is this a bug in the build procedure, or am I missing something?

Btw, I did see this comment in lisp/Makefile.in:

  # custom-deps and finder-data both used to scan _all_ the *.el files.
  # This could lead to problems in parallel builds if automatically
  # generated *.el files (eg loaddefs etc) were being changed at the same time.
  # One solution was to add autoloads as a prerequisite:
  # http://lists.gnu.org/archive/html/emacs-pretest-bug/2007-01/msg00469.html
  # http://lists.gnu.org/archive/html/bug-gnu-emacs/2007-12/msg00171.html
  # However, this meant that running these targets modified loaddefs.el,
  # every time (due to time-stamping).  Calling these rules from
  # bootstrap-after would modify loaddefs after src/emacs, resulting
  # in make install remaking src/emacs for no real reason:
  # http://lists.gnu.org/archive/html/emacs-devel/2008-02/msg00311.html
  # Nowadays these commands don't scan automatically generated files,
  # since they will never contain any useful information
  # (see finder-no-scan-regexp and custom-dependencies-no-scan-regexp).

but it is unclear to me what's its conclusion.  Problems with parallel
builds can be taken care of by proper dependencies and/or by judicial
use of temporary files and move-if-change.  And the last sentence
seems to say that at least some of the problems no longer exist(?)

IOW, the status of this issue is unclear.

In addition, there's nothing at all here about autoloads.

Re: Updating autoloads, custom-deps, finder-data

[Glenn Morris]

Eli Zaretskii wrote:

> Btw, I did see this comment in lisp/Makefile.in:
>
>   # custom-deps and finder-data both used to scan _all_ the *.el files.
>   # This could lead to problems in parallel builds if automatically
>   # generated *.el files (eg loaddefs etc) were being changed at the same time.
>   # One solution was to add autoloads as a prerequisite:
>   # http://lists.gnu.org/archive/html/emacs-pretest-bug/2007-01/msg00469.html
>   # http://lists.gnu.org/archive/html/bug-gnu-emacs/2007-12/msg00171.html
>   # However, this meant that running these targets modified loaddefs.el,
>   # every time (due to time-stamping).  Calling these rules from
>   # bootstrap-after would modify loaddefs after src/emacs, resulting
>   # in make install remaking src/emacs for no real reason:
>   # http://lists.gnu.org/archive/html/emacs-devel/2008-02/msg00311.html
>   # Nowadays these commands don't scan automatically generated files,
>   # since they will never contain any useful information
>   # (see finder-no-scan-regexp and custom-dependencies-no-scan-regexp).
>
> but it is unclear to me what's its conclusion.  Problems with parallel
> builds can be taken care of by proper dependencies and/or by judicial
> use of temporary files and move-if-change.  And the last sentence
> seems to say that at least some of the problems no longer exist(?)
>
> IOW, the status of this issue is unclear.

Seems clear to me, but I think I wrote it.
The stuff in the past tense describes the problems that used to exist.
The stuff in the present tense describes why they don't exist now.

(I don't recall if there is an issue with _versioned_ files that contain
autoload information, eg rmail.el. In practice I have never seen one.)

Re: Updating autoloads, custom-deps, finder-data

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 20 Apr 2013 13:58:10 -0400
> 
> The stuff in the past tense describes the problems that used to exist.
> The stuff in the present tense describes why they don't exist now.

So what are the problems, if any, that still exist and preclude us
from writing recipes that would update the autoloads and the rest as
part of a normal "make"?

Re: Updating autoloads, custom-deps, finder-data

[Glenn Morris]

Eli Zaretskii wrote:

> So what are the problems, if any, that still exist and preclude us
> from writing recipes that would update the autoloads and the rest as
> part of a normal "make"?

Sorry, don't know. I'm a happy bootstrapper.
Beware of Emacs getting re-dumped when not necessary is the only thing that
comes to mind (this is a case where the build-number becomes annoying).
http://lists.gnu.org/archive/html/emacs-devel/2008-02/msg00311.html

Re: Updating autoloads, custom-deps, finder-data

[Stefan Monnier]

> When any of the Lisp files are modified, typing "make" in the
> top-level directory byte-compiles those modified files, as expected,
> but does not invoke "make autoloads", "make custom-deps", and "make
> finder-data" targets.  Thus the autoloads and the other meta-data
> regarding the modified Lisp files slowly bit-rots, unless you manually
> invoke those targets or perform a full bootstrap.

> Is this a bug in the build procedure, or am I missing something?

No, I think you're right.  There's a dependency on
.bzr/checkout/dirstate that triggers update of autoloads after some
bzr operations, but that's all.

It's a tradeoff:
- rebuilding those files takes a significant amount of time (and
  triggers more rebuilds down the road).
- rebuilding custom-deps and finder-data is not tremendously important.
- rebuilding loaddefs can be very important, but is often not needed.
- it's basically impossible to know when a rebuild is actually necessary.


        Stefan