Parallel builds with loaddefs removed

Parallel builds with loaddefs removed

[Eli Zaretskii]

When one of the *loaddefs.el files is removed for some reason, the
very next "make -j" fails, because it sometimes compiles a file that
needs, e.g., cl-loaddefs.el before it is recreated.  This happened to
me today because cl-loaddefs.el was removed from bzr, and "bzr up"
deleted it.

Can this be fixed?

Re: Parallel builds with loaddefs removed

[Stefan Monnier]

> When one of the *loaddefs.el files is removed for some reason, the
> very next "make -j" fails, because it sometimes compiles a file that
> needs, e.g., cl-loaddefs.el before it is recreated.  This happened to
> me today because cl-loaddefs.el was removed from bzr, and "bzr up"
> deleted it.
> Can this be fixed?

"make bootstrap" fixes it, AFAIK.


        Stefan

Re: Parallel builds with loaddefs removed

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 21 Jun 2013 11:30:52 -0400
> 
> > When one of the *loaddefs.el files is removed for some reason, the
> > very next "make -j" fails, because it sometimes compiles a file that
> > needs, e.g., cl-loaddefs.el before it is recreated.  This happened to
> > me today because cl-loaddefs.el was removed from bzr, and "bzr up"
> > deleted it.
> > Can this be fixed?
> 
> "make bootstrap" fixes it, AFAIK.

As does just "make" with no -j.  But this means we have a problem in
our dependencies.

Re: Parallel builds with loaddefs removed

[Stefan Monnier]

>> "make bootstrap" fixes it, AFAIK.
> As does just "make" with no -j.

Actually, no.  Just "make" might create a cl-loaddefs.el but it doesn't
guarantee that this file is complete, so it can still fail.


        Stefan

Re: Parallel builds with loaddefs removed

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 21 Jun 2013 14:57:26 -0400
> 
> >> "make bootstrap" fixes it, AFAIK.
> > As does just "make" with no -j.
> 
> Actually, no.  Just "make" might create a cl-loaddefs.el but it doesn't
> guarantee that this file is complete, so it can still fail.

That's even worse.  It doesn't make sense to have to bootstrap just to
regenerate a single file.

Re: Parallel builds with loaddefs removed

[Glenn Morris]

Eli Zaretskii wrote:

> When one of the *loaddefs.el files is removed for some reason, the
> very next "make -j" fails, because it sometimes compiles a file that
> needs, e.g., cl-loaddefs.el before it is recreated.  This happened to
> me today because cl-loaddefs.el was removed from bzr, and "bzr up"
> deleted it.

That specific problem should be impossible now, owing to cl-lib.el's:

(or (load "cl-loaddefs" 'noerror 'quiet)
    ;; When bootstrapping, cl-loaddefs hasn't been built yet!
    (require 'cl-macs))

I guess you had a stale cl-lib.elc around?
So AFAICS, that's just a one-time problem, the usual sort of issue with
stale elc files that we know non-bootstrap builds sometimes have.

Re: Parallel builds with loaddefs removed

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 21 Jun 2013 15:20:22 -0400
> 
> That specific problem should be impossible now, owing to cl-lib.el's:
> 
> (or (load "cl-loaddefs" 'noerror 'quiet)
>     ;; When bootstrapping, cl-loaddefs hasn't been built yet!
>     (require 'cl-macs))
> 
> I guess you had a stale cl-lib.elc around?
> So AFAICS, that's just a one-time problem, the usual sort of issue with
> stale elc files that we know non-bootstrap builds sometimes have.

cl-loaddefs.el has loaddefs not just from cl-macs.el.  So having it
absent or outdated can still fail the build in interesting ways, even
with that change.

Re: Parallel builds with loaddefs removed

[Stefan Monnier]

>> >> "make bootstrap" fixes it, AFAIK.
>> > As does just "make" with no -j.
>> Actually, no.  Just "make" might create a cl-loaddefs.el but it doesn't
>> guarantee that this file is complete, so it can still fail.
> That's even worse.  It doesn't make sense to have to bootstrap just to
> regenerate a single file.

Yet, that's how things have been with all the *-loaddefs.el files for
several years now.  You only notice it now because the "bzr rm" I do
caused your cl-loaddefs.el file to disappear, but otherwise this file
would normally never disappear except you "make bootstrap".

It's a one off transient problem, fixed by "make bootstrap", like many
other such transient problems.

Actually, you can recreate this file without bootstrapping:
"rm lisp/loaddefs.el; make" will do.  Of course, on a machine with many
CPUs, this ends up taking almost as long as a bootstrap ;-(


        Stefan

Re: Parallel builds with loaddefs removed

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 21 Jun 2013 15:56:36 -0400
> 
> Actually, you can recreate this file without bootstrapping:
> "rm lisp/loaddefs.el; make" will do.

"make autoloads" will solve that (as it will solve also the original
problem I described).

But I get the drift: you don't want to fix this.  So be it.

Re: Parallel builds with loaddefs removed

[Stefan Monnier]

> But I get the drift: you don't want to fix this.  So be it.

It's worse: I don't know how to fix it.


        Stefan

Re: Parallel builds with loaddefs removed

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Sat, 22 Jun 2013 11:15:36 -0400
> 
> > But I get the drift: you don't want to fix this.  So be it.
> 
> It's worse: I don't know how to fix it.

Something like this:

$(lisp): cl-loaddefs.el

cl-loaddefs.el: cl-extra.el cl-macs.el cl-seq.el
	$(MAKE) autoloads

where $(lisp) is the list of all the *.elc files, computed like we do
in compile-main.

Re: Parallel builds with loaddefs removed

[Stefan Monnier]

> cl-loaddefs.el: cl-extra.el cl-macs.el cl-seq.el
> 	$(MAKE) autoloads

I don't think that works: if the cl-*.el files haven't been touched
since the last "make autoloads", "make autoloads" won't fetch their
autoloads to put them in cl-loaddefs.el.


        Stefan

Re: Parallel builds with loaddefs removed

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Sat, 22 Jun 2013 15:20:38 -0400
> 
> > cl-loaddefs.el: cl-extra.el cl-macs.el cl-seq.el
> > 	$(MAKE) autoloads
> 
> I don't think that works: if the cl-*.el files haven't been touched
> since the last "make autoloads", "make autoloads" won't fetch their
> autoloads to put them in cl-loaddefs.el.

If they were not touched and cl-loaddefs.el exists, then that's fine.
If they were not touched and cl-loaddefs.el does not exist, the rule
will fire and create it.  If they were touched, the rule will fire and
update cl-loaddefs.el.

Re: Parallel builds with loaddefs removed

[Stefan Monnier]

> If they were not touched and cl-loaddefs.el does not exist, the rule
> will fire and create it.

And what I'm saying is that this is not the case: the rule will fire,
but it will not create cl-loaddefs.el, or maybe it will partly but
not completely, because the autoload.el code will mostly compare the
timestamp in lisp/loaddefs.el with the mtime of the cl-*.el files and
only consult those that are more recent, and only those so consulted
will see their autoloads added to cl-loaddefs.

> If they were touched, the rule will fire and
> update cl-loaddefs.el.

If they were touched, then the comparison with the lisp.loaddefs.el
timestamp will indeed cause them to be scanned, so their autoloads will
end up in cl-loaddefs.el.  But if they were not all touched, then only
some of them will be consulted, so the cl-loaddefs.el will end up
incomplete.


        Stefan