bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Markus Triska]

A parallel compilation (-j 2) with today's CVS version aborted with:

   In toplevel form:
   mh-e/mh-folder.el:34:1:Error: Required feature `mh-e' was not provided
   Wrote /Users/mt/emacs/lisp/mh-e/mh-e.elc
   Compiling /Users/mt/emacs/lisp/mh-e/mh-funcs.el
   make[1]: *** [/Users/mt/emacs/lisp/mh-e/mh-folder.elc] Error 1
   make[1]: *** Waiting for unfinished jobs....
   make[1]: *** Waiting for unfinished jobs....
   No MH variant found on the system
   Wrote /Users/mt/emacs/lisp/mh-e/mh-funcs.elc
   make: *** [lisp] Error 2

After another "$ make -j 2", the compilation proceeded without errors.
The full compilation log is available from:

   http://www.logic.at/prolog/log20081225.txt

I have seen this problem only once so far in many tries.


In GNU Emacs 23.0.60.1 (i386-apple-darwin8.11.1, GTK+ Version 2.12.9)
 of 2008-12-25 on mt-computer.local
Windowing system distributor `The XFree86 Project, Inc', version 11.0.40400000
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_GB.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: nil
  default-enable-multibyte-characters: t

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Eli Zaretskii]

> From: Markus Triska <markus.triska@gmx.at>
> Date: Thu, 25 Dec 2008 23:27:56 +0100 (CET)
> Cc: 
> 
> 
> A parallel compilation (-j 2) with today's CVS version aborted with:
> 
>    In toplevel form:
>    mh-e/mh-folder.el:34:1:Error: Required feature `mh-e' was not provided
>    Wrote /Users/mt/emacs/lisp/mh-e/mh-e.elc
>    Compiling /Users/mt/emacs/lisp/mh-e/mh-funcs.el
>    make[1]: *** [/Users/mt/emacs/lisp/mh-e/mh-folder.elc] Error 1
>    make[1]: *** Waiting for unfinished jobs....
>    make[1]: *** Waiting for unfinished jobs....
>    No MH variant found on the system
>    Wrote /Users/mt/emacs/lisp/mh-e/mh-funcs.elc
>    make: *** [lisp] Error 2

This is because we don't tell Make all the truth about the
dependencies between Lisp files, so it by default assumes that
mh-funcs.elc and mh-e.elc are independent, and compiles them in two
different jobs that run in parallel.  But the truth is that
mh-funcs.el says this:

   (require 'mh-e)

which causes Emacs to look for mh-e.elc when it compiles mh-funcs.el.
So there's a race condition here between the 2 jobs that Make runs in
parallel under "-j 2": one job compiles mh-e.el and writes mh-e.elc,
while the other job, which compiles mh-funcs.el, reads from mh-e.elc.
In this case, the reading job evidently accessed mh-e.elc after it was
already created, but before it was written in its fullness (as you see
above, the "Wrote /Users/mt/emacs/lisp/mh-e/mh-e.elc" message is
printed _after_ the error message).  So the "provide" form was not yet
in mh-e.elc when it was read, and Emacs barfed.

In general, we lack such dependencies all over lisp/Makefile.in, so I
submit that "make -j N" works for us by sheer luck: I'm guessing that
this is due to small values of N and the fact that $(ELCFILES) causes
Make to compile files in alphabetic order, combined with very few
dependencies between Lisp files whose names come close to one another
in that order.  But given a large enough value of N (and a
correspondingly large number of processors on the system), we should
see such failures all the time.

I added to lisp/Makefile.in a series of dependencies that should fix
the case of MH-E compilation.  Please see if this fixes the problem
for you.  If you have more than 2 processors, it would make sense to
try "make -j N" with values of N larger than 2, to increase the
probability of such a clash.

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Stefan Monnier]

> I added to lisp/Makefile.in a series of dependencies that should fix
> the case of MH-E compilation.  Please see if this fixes the problem
> for you.  If you have more than 2 processors, it would make sense to
> try "make -j N" with values of N larger than 2, to increase the
> probability of such a clash.

This won't solve it in general, because there are many circular
dependencies among Elisp files.  Another approach is to just close the
race condition: write the byte-compiled file to foo.elc.temp and then
rename it atomically to foo.elc.


        Stefan

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Bill Wohler]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I added to lisp/Makefile.in a series of dependencies that should fix
>> the case of MH-E compilation.  Please see if this fixes the problem
>> for you.  If you have more than 2 processors, it would make sense to
>> try "make -j N" with values of N larger than 2, to increase the
>> probability of such a clash.
>
> This won't solve it in general, because there are many circular
> dependencies among Elisp files.  Another approach is to just close the
> race condition: write the byte-compiled file to foo.elc.temp and then
> rename it atomically to foo.elc.

Thanks Eli and Stefan. My laptop isn't a multi-processor box (yet) so
I can't be of more assistance. All I can say is that with the last big
MH-E reorganization going into MH-E 8.0, we eliminated all circular
dependencies within MH-E. I can't speak for the packages we require,
however.

-- 
Bill Wohler <wohler@newt.com>  http://www.newt.com/wohler/  GnuPG ID:610BD9AD

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 1705@emacsbugs.donarmstrong.com,  Markus Triska <markus.triska@gmx.at>,  emacs-pretest-bug@gnu.org,  bug-gnu-emacs@gnu.org
> Date: Tue, 30 Dec 2008 21:30:33 -0500
> 
> > I added to lisp/Makefile.in a series of dependencies that should fix
> > the case of MH-E compilation.  Please see if this fixes the problem
> > for you.  If you have more than 2 processors, it would make sense to
> > try "make -j N" with values of N larger than 2, to increase the
> > probability of such a clash.
> 
> This won't solve it in general, because there are many circular
> dependencies among Elisp files.  Another approach is to just close the
> race condition: write the byte-compiled file to foo.elc.temp and then
> rename it atomically to foo.elc.

But adding dependencies is TRT on its own right, even if your
alternative is adopted.  As long as the byte compiler cannot grok
those dependencies by itself, that is.

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Stefan Monnier]

>> This won't solve it in general, because there are many circular
>> dependencies among Elisp files.  Another approach is to just close the
>> race condition: write the byte-compiled file to foo.elc.temp and then
>> rename it atomically to foo.elc.
> But adding dependencies is TRT on its own right,

Personally, I've hacked my Emacs so that when byte-compiling files,
`require'd files get loaded from the .el in preference to the .elc file
if it's more recent.


        Stefan

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 1705@emacsbugs.donarmstrong.com,  markus.triska@gmx.at,  emacs-pretest-bug@gnu.org,  bug-gnu-emacs@gnu.org
> Date: Sun, 04 Jan 2009 23:12:58 -0500
> 
> Personally, I've hacked my Emacs so that when byte-compiling files,
> `require'd files get loaded from the .el in preference to the .elc file
> if it's more recent.

I don't think this is TRT in general.

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Stefan Monnier]

>> Personally, I've hacked my Emacs so that when byte-compiling files,
>> `require'd files get loaded from the .el in preference to the .elc file
>> if it's more recent.

> I don't think this is TRT in general.

Neither is the preference of .elc even if it's older.


        Stefan

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 1705@emacsbugs.donarmstrong.com,  markus.triska@gmx.at,  emacs-pretest-bug@gnu.org,  bug-gnu-emacs@gnu.org
> Date: Mon, 05 Jan 2009 17:21:23 -0500
> 
> >> Personally, I've hacked my Emacs so that when byte-compiling files,
> >> `require'd files get loaded from the .el in preference to the .elc file
> >> if it's more recent.
> 
> > I don't think this is TRT in general.
> 
> Neither is the preference of .elc even if it's older.

I wasn't arguing for that, I was arguing for adding dependencies to
lisp/Makefile.in.  That is always TRT.

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Richard M Stallman]

    Neither is the preference of .elc even if it's older.

In general, it is right to load the .elc even if older,
because that way editing the source code does not
immediately force you to use your new code.
You get to choose (by compiling) when to make it active.

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Stefan Monnier]

>> >> Personally, I've hacked my Emacs so that when byte-compiling files,
>> >> `require'd files get loaded from the .el in preference to the .elc file
>> >> if it's more recent.
>> > I don't think this is TRT in general.
>> Neither is the preference of .elc even if it's older.
> I wasn't arguing for that,

I know.

> I was arguing for adding dependencies to
> lisp/Makefile.in.  That is always TRT.

Except when there are circular dependencies, which (as mentioned) do
exist in fairly large quantities in Emacs.


        Stefan

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 1705@emacsbugs.donarmstrong.com,  markus.triska@gmx.at,  emacs-pretest-bug@gnu.org,  bug-gnu-emacs@gnu.org
> Date: Tue, 06 Jan 2009 10:00:59 -0500
> 
> > I was arguing for adding dependencies to
> > lisp/Makefile.in.  That is always TRT.
> 
> Except when there are circular dependencies

Of course, that goes without saying.  Make will simply barf on
circular dependencies.

bug#1705: 23.0.60; parallel build failure: `mh-e' was not provided

[Stefan Monnier]

>> > I was arguing for adding dependencies to
>> > lisp/Makefile.in.  That is always TRT.
>> Except when there are circular dependencies
> Of course, that goes without saying.  Make will simply barf on
> circular dependencies.

Not only that, but the build order will not be correct (at least
insofar as the dependencies are really needed).  Which is why another
solution is also needed.


        Stefan