parallel make bug

parallel make bug

[Dan Nicolaescu]

Doing make -j8 bootstrap  on a 2 processor machine:

Directories: /home/dann/emacs/Emacs-CVS/emacs/lisp/. /home/dann/emacs/Emacs-CVS/emacs/lisp/./eshell /home/dann/emacs/Emacs-CVS/emacs/lisp/./mail /home/dann/emacs/Emacs-CVS/emacs/lisp/./language /home/dann/emacs/Emacs-CVS/emacs/lisp/./emacs-lisp /home/dann/emacs/Emacs-CVS/emacs/lisp/./calc /home/dann/emacs/Emacs-CVS/emacs/lisp/./gnus /home/dann/emacs/Emacs-CVS/emacs/lisp/./textmodes /home/dann/emacs/Emacs-CVS/emacs/lisp/./net /home/dann/emacs/Emacs-CVS/emacs/lisp/./mh-e /home/dann/emacs/Emacs-CVS/emacs/lisp/./calendar /home/dann/emacs/Emacs-CVS/emacs/lisp/./progmodes /home/dann/emacs/Emacs-CVS/emacs/lisp/./erc /home/dann/emacs/Emacs-CVS/emacs/lisp/./international /home/dann/emacs/Emacs-CVS/emacs/lisp/./emulation /home/dann/emacs/Emacs-CVS/emacs/lisp/./url /home/dann/emacs/Emacs-CVS/emacs/lisp/./play
Compiling /home/dann/emacs/Emacs-CVS/emacs/lisp/emacs-lisp/byte-opt.el
IO error reading /home/dann/emacs/Emacs-CVS/emacs/lisp/subdirs.el: Input/output error
make[2]: *** [autoloads] Error 255
make[2]: *** Waiting for unfinished jobs....

Is there any reason $(lisp)/subdirs.el should start a recursive make
instead of just depending on update-subdirs like in the patch below?


*** Makefile.in      07 Apr 2007 08:37:10 -0700 1.85
--- Makefile.in      26 Jun 2007 13:59:36 -0700 
***************
*** 112,119 ****
    echo Directories: $$wins; \
    $(EMACS) $(EMACSOPT) -l autoload --eval '(setq generated-autoload-file "$(lisp)/loaddefs.el")' -f batch-update-autoloads $$wins
  
! $(lisp)/subdirs.el:
!       $(MAKE) $(MFLAGS) update-subdirs
  update-subdirs: doit
                  wd=$(lisp); $(setwins); \
                  for file in $$wins; do \
--- 112,119 ----
    echo Directories: $$wins; \
    $(EMACS) $(EMACSOPT) -l autoload --eval '(setq generated-autoload-file "$(lisp)/loaddefs.el")' -f batch-update-autoloads $$wins
  
! $(lisp)/subdirs.el: update-subdirs
! 
  update-subdirs: doit
                  wd=$(lisp); $(setwins); \
                  for file in $$wins; do \

Re: parallel make bug

[Stefan Monnier]

> Is there any reason $(lisp)/subdirs.el should start a recursive make
> instead of just depending on update-subdirs like in the patch below?

Because the current form allows it to just check that subdirs.el exists
whereas your patch would force it to always be rebuilt.

The problem being that we have no easy way to check that subdirs.el is
up-to-date, so we can either rebuild it all the time or only rebuild it when
specifically asked to.


        Stefan

Re: parallel make bug

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

  > > Is there any reason $(lisp)/subdirs.el should start a recursive make
  > > instead of just depending on update-subdirs like in the patch below?
  > 
  > Because the current form allows it to just check that subdirs.el exists
  > whereas your patch would force it to always be rebuilt.
  > 
  > The problem being that we have no easy way to check that subdirs.el is
  > up-to-date, so we can either rebuild it all the time or only rebuild it when
  > specifically asked to.

How about always rebuilding and use move-if-change ? 

Re: parallel make bug

[Stefan Monnier]

>> > Is there any reason $(lisp)/subdirs.el should start a recursive make
>> > instead of just depending on update-subdirs like in the patch below?
>> 
>> Because the current form allows it to just check that subdirs.el exists
>> whereas your patch would force it to always be rebuilt.
>> 
>> The problem being that we have no easy way to check that subdirs.el is
>> up-to-date, so we can either rebuild it all the time or only rebuild it when
>> specifically asked to.

> How about always rebuilding and use move-if-change ? 

Try it.  Note that the same kind of problem occurs with autoloads, cus-load
and finder-inf.


        Stefan

Re: parallel make bug

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@ics.uci.edu>
> Date: Tue, 26 Jun 2007 14:10:05 -0700
> 
> 
> Doing make -j8 bootstrap  on a 2 processor machine:
> 
> Directories: /home/dann/emacs/Emacs-CVS/emacs/lisp/. /home/dann/emacs/Emacs-CVS/emacs/lisp/./eshell /home/dann/emacs/Emacs-CVS/emacs/lisp/./mail /home/dann/emacs/Emacs-CVS/emacs/lisp/./language /home/dann/emacs/Emacs-CVS/emacs/lisp/./emacs-lisp /home/dann/emacs/Emacs-CVS/emacs/lisp/./calc /home/dann/emacs/Emacs-CVS/emacs/lisp/./gnus /home/dann/emacs/Emacs-CVS/emacs/lisp/./textmodes /home/dann/emacs/Emacs-CVS/emacs/lisp/./net /home/dann/emacs/Emacs-CVS/emacs/lisp/./mh-e /home/dann/emacs/Emacs-CVS/emacs/lisp/./calendar /home/dann/emacs/Emacs-CVS/emacs/lisp/./progmodes /home/dann/emacs/Emacs-CVS/emacs/lisp/./erc /home/dann/emacs/Emacs-CVS/emacs/lisp/./international /home/dann/emacs/Emacs-CVS/emacs/lisp/./emulation /home/dann/emacs/Emacs-CVS/emacs/lisp/./url /home/dann/emacs/Emacs-CVS/emacs/lisp/./play
> Compiling /home/dann/emacs/Emacs-CVS/emacs/lisp/emacs-lisp/byte-opt.el
> IO error reading /home/dann/emacs/Emacs-CVS/emacs/lisp/subdirs.el: Input/output error
> make[2]: *** [autoloads] Error 255
> make[2]: *** Waiting for unfinished jobs....
> 
> Is there any reason $(lisp)/subdirs.el should start a recursive make
> instead of just depending on update-subdirs like in the patch below?

Is there a reason for the I/O error?  I don't see why it should
happen; do you?

Re: parallel make bug

[Dan Nicolaescu]

Eli Zaretskii <eliz@gnu.org> writes:

  > > From: Dan Nicolaescu <dann@ics.uci.edu>
  > > Date: Tue, 26 Jun 2007 14:10:05 -0700
  > > 
  > > 
  > > Doing make -j8 bootstrap  on a 2 processor machine:
  > > 
  > > Directories: /home/dann/emacs/Emacs-CVS/emacs/lisp/. /home/dann/emacs/Emacs-CVS/emacs/lisp/./eshell /home/dann/emacs/Emacs-CVS/emacs/lisp/./mail /home/dann/emacs/Emacs-CVS/emacs/lisp/./language /home/dann/emacs/Emacs-CVS/emacs/lisp/./emacs-lisp /home/dann/emacs/Emacs-CVS/emacs/lisp/./calc /home/dann/emacs/Emacs-CVS/emacs/lisp/./gnus /home/dann/emacs/Emacs-CVS/emacs/lisp/./textmodes /home/dann/emacs/Emacs-CVS/emacs/lisp/./net /home/dann/emacs/Emacs-CVS/emacs/lisp/./mh-e /home/dann/emacs/Emacs-CVS/emacs/lisp/./calendar /home/dann/emacs/Emacs-CVS/emacs/lisp/./progmodes /home/dann/emacs/Emacs-CVS/emacs/lisp/./erc /home/dann/emacs/Emacs-CVS/emacs/lisp/./international /home/dann/emacs/Emacs-CVS/emacs/lisp/./emulation /home/dann/emacs/Emacs-CVS/emacs/lisp/./url /home/dann/emacs/Emacs-CVS/emacs/lisp/./play
  > > Compiling /home/dann/emacs/Emacs-CVS/emacs/lisp/emacs-lisp/byte-opt.el
  > > IO error reading /home/dann/emacs/Emacs-CVS/emacs/lisp/subdirs.el: Input/output error
  > > make[2]: *** [autoloads] Error 255
  > > make[2]: *** Waiting for unfinished jobs....
  > > 
  > > Is there any reason $(lisp)/subdirs.el should start a recursive make
  > > instead of just depending on update-subdirs like in the patch below?
  > 
  > Is there a reason for the I/O error?  I don't see why it should
  > happen; do you?

For bootstrap, update-subdirs and autoloads can execute in
parallel. autoloads will run another make to do update-subdirs.
If "autoloads" creates subdirs.el first while "update-subdirs" is not
running, it is possible that later "update-subdirs" deletes subdirs.el
while "autoloads" is using it. Am I off base here? 

Re: parallel make bug

[Andreas Schwab]

Dan Nicolaescu <dann@ics.uci.edu> writes:

> For bootstrap, update-subdirs and autoloads can execute in
> parallel. autoloads will run another make to do update-subdirs.
> If "autoloads" creates subdirs.el first while "update-subdirs" is not
> running, it is possible that later "update-subdirs" deletes subdirs.el
> while "autoloads" is using it. Am I off base here? 

But why do you get an I/O error?  That looks like a hardware problem.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: parallel make bug

[Dan Nicolaescu]

Andreas Schwab <schwab@suse.de> writes:

  > Dan Nicolaescu <dann@ics.uci.edu> writes:
  > 
  > > For bootstrap, update-subdirs and autoloads can execute in
  > > parallel. autoloads will run another make to do update-subdirs.
  > > If "autoloads" creates subdirs.el first while "update-subdirs" is not
  > > running, it is possible that later "update-subdirs" deletes subdirs.el
  > > while "autoloads" is using it. Am I off base here? 
  > 
  > But why do you get an I/O error?  That looks like a hardware problem.

Doubtful, the machine I used is quite solid. IMO its probably
something to do with NFS (the source and objs directories are on
NFS).

Anyway, the error does not occur all the time, sometimes the failure is:

Cannot open load file: autoload

Re: parallel make bug

[Andreas Schwab]

Dan Nicolaescu <dann@ics.uci.edu> writes:

> Anyway, the error does not occur all the time, sometimes the failure is:
>
> Cannot open load file: autoload

Does this help?

--- Makefile.in	24 Apr 2007 10:31:30 +0200	1.85
+++ Makefile.in	27 Jun 2007 19:01:20 +0200	
@@ -239,7 +239,7 @@ MH_E_SRC = $(lisp)/mh-e/mh-acros.el $(li
 	$(lisp)/mh-e/mh-xface.el
 
 mh-autoloads: $(lisp)/mh-e/mh-loaddefs.el
-$(lisp)/mh-e/mh-loaddefs.el: $(MH_E_SRC)
+$(lisp)/mh-e/mh-loaddefs.el: $(lisp)/subdirs.el $(MH_E_SRC)
 	echo ";;; mh-loaddefs.el --- automatically extracted autoloads" > $@
 	echo "" >> $@
 	echo ";; Copyright (C) 2003, 2004, 2005, 2006, 2007 Free Software Foundation, Inc." >> $@

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: parallel make bug

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dan Nicolaescu <dann@ics.uci.edu>
> Date: Tue, 26 Jun 2007 22:29:52 -0700
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>   > > From: Dan Nicolaescu <dann@ics.uci.edu>
>   > > Date: Tue, 26 Jun 2007 14:10:05 -0700
>   > > 
>   > > 
>   > > Doing make -j8 bootstrap  on a 2 processor machine:
>   > > 
>   > > Directories: /home/dann/emacs/Emacs-CVS/emacs/lisp/. /home/dann/emacs/Emacs-CVS/emacs/lisp/./eshell /home/dann/emacs/Emacs-CVS/emacs/lisp/./mail /home/dann/emacs/Emacs-CVS/emacs/lisp/./language /home/dann/emacs/Emacs-CVS/emacs/lisp/./emacs-lisp /home/dann/emacs/Emacs-CVS/emacs/lisp/./calc /home/dann/emacs/Emacs-CVS/emacs/lisp/./gnus /home/dann/emacs/Emacs-CVS/emacs/lisp/./textmodes /home/dann/emacs/Emacs-CVS/emacs/lisp/./net /home/dann/emacs/Emacs-CVS/emacs/lisp/./mh-e /home/dann/emacs/Emacs-CVS/emacs/lisp/./calendar /home/dann/emacs/Emacs-CVS/emacs/lisp/./progmodes /home/dann/emacs/Emacs-CVS/emacs/lisp/./erc /home/dann/emacs/Emacs-CVS/emacs/lisp/./international /home/dann/emacs/Emacs-CVS/emacs/lisp/./emulation /home/dann/emacs/Emacs-CVS/emacs/lisp/./url /home/dann/emacs/Emacs-CVS/emacs/lisp/./play
>   > > Compiling /home/dann/emacs/Emacs-CVS/emacs/lisp/emacs-lisp/byte-opt.el
>   > > IO error reading /home/dann/emacs/Emacs-CVS/emacs/lisp/subdirs.el: Input/output error
>   > > make[2]: *** [autoloads] Error 255
>   > > make[2]: *** Waiting for unfinished jobs....
>   > > 
>   > > Is there any reason $(lisp)/subdirs.el should start a recursive make
>   > > instead of just depending on update-subdirs like in the patch below?
>   > 
>   > Is there a reason for the I/O error?  I don't see why it should
>   > happen; do you?
> 
> For bootstrap, update-subdirs and autoloads can execute in
> parallel. autoloads will run another make to do update-subdirs.
> If "autoloads" creates subdirs.el first while "update-subdirs" is not
> running, it is possible that later "update-subdirs" deletes subdirs.el
> while "autoloads" is using it. Am I off base here? 

If you think the conflict is between "autoloads" and "update-subdirs",
then the usual solution is to serialize them: make "update-subdirs"
touch some file, and make "autoloads" depend on that file.

Re: parallel make bug

[Dan Nicolaescu]

Andreas Schwab <schwab@suse.de> writes:

  > Dan Nicolaescu <dann@ics.uci.edu> writes:
  > 
  > > Anyway, the error does not occur all the time, sometimes the failure is:
  > >
  > > Cannot open load file: autoload
  > 
  > Does this help?

Yep, I couldn't make the build fail after applying this patch. 
Please don't forget the emacs-22 branch when you check it in.

Thanks!

parallel make bug

[Chip Coldwell]

I believe I may also have uncovered a parallel make bug, building on
the Fedora 9 GNU/Linux distribution:

make[2]: Leaving directory `/builddir/build/BUILD/emacs-23.0.60/leim'
Compiling /builddir/build/BUILD/emacs-23.0.60/lisp/diff.el
Wrote /builddir/build/BUILD/emacs-23.0.60/lisp/dframe.elc
Compiling /builddir/build/BUILD/emacs-23.0.60/lisp/dired-aux.el
Wrote /builddir/build/BUILD/emacs-23.0.60/lisp/desktop.elc
Compiling /builddir/build/BUILD/emacs-23.0.60/lisp/dired-x.el
Wrote /builddir/build/BUILD/emacs-23.0.60/lisp/diff.elc
Compiling /builddir/build/BUILD/emacs-23.0.60/lisp/dired.el
In toplevel form:
dired-x.el:108:1:Error: Required feature `dired-aux' was not provided
make[2]: *** [/builddir/build/BUILD/emacs-23.0.60/lisp/dired-x.elc] Error 1
make[2]: *** Waiting for unfinished jobs....
Wrote /builddir/build/BUILD/emacs-23.0.60/lisp/dired-aux.elc
Wrote /builddir/build/BUILD/emacs-23.0.60/lisp/diff-mode.elc
Wrote /builddir/build/BUILD/emacs-23.0.60/lisp/dired.elc

Here it appears that dired-aux.el and dired-x.el are being built
simultaneously, despite the dependency on dired-aux by dired-x:

$ egrep -e ^\\\(require dired-x.el
(require 'dired)
(require 'dired-aux)

Indeed, it looks like both dired and dired-aux need to compile before
dired-x, IIUC.

Chip

-- 
Charles M. "Chip" Coldwell
Senior Software Engineer
Red Hat, Inc
978-392-2426

GPG ID:  852E052F
GPG FPR: 77E5 2B51 4907 F08A 7E92  DE80 AFA9 9A8F 852E 052F