Building cvs emacs on a dual-core machine

Building cvs emacs on a dual-core machine

[Eric Lilja]

Hi!
I recently upgraded my computer with a dual-core CPU. I've been enjoying 
nice decreases in compile time in my own projects by specifying --jobs=2 
when invoking make.

However, when I tried that with emacs it started compiling, very fast 
with both cores busy, but the build craps out (sorry, no log) after a 
little while. Maybe because it tries to build something that depends on 
something else that has not been quite finished yet because when 
multiple jobs are specified things doesn't happen in serial anymore? I 
don't know, but I know it works if I exclude --jobs=2. But then emacs is 
built and the total cpu utilization is only 50 percent. :-( Is there a 
way I can utilise both cores when building emacs? I bought the new 
processor mostly to get improved compile time performace. :-)

- Eric

Re: Building cvs emacs on a dual-core machine

[Kim F. Storm]

Eric Lilja <mindcooler@gmail.com> writes:

> However, when I tried that with emacs it started compiling, very fast
> with both cores busy, but the build craps out (sorry, no log) after a
> little while.

Only a fractional part of the build is about compiling C files.

Most of the time is spent inside emacs byte-compiling Lisp files,
and when that is done, it starts loading and undumping.

So most of the time, you have just one process running.

> Maybe because it tries to build something that depends
> on something else that has not been quite finished yet because when
> multiple jobs are specified things doesn't happen in serial anymore? I
> don't know, but I know it works if I exclude --jobs=2. But then emacs
> is built and the total cpu utilization is only 50 percent. :-( Is
> there a way I can utilise both cores when building emacs? 

Maybe you can build a GTK and a non-GTK version at the same time :-)

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Building cvs emacs on a dual-core machine

[Eli Zaretskii]

> From: Eric Lilja <mindcooler@gmail.com>
> Date: Wed, 20 Dec 2006 12:47:13 +0100
> 
> I recently upgraded my computer with a dual-core CPU. I've been enjoying 
> nice decreases in compile time in my own projects by specifying --jobs=2 
> when invoking make.
> 
> However, when I tried that with emacs it started compiling, very fast 
> with both cores busy, but the build craps out (sorry, no log) after a 
> little while.

Any such failures are bugs in the Emacs Makefiles, so please be sure
to report them with full details, including the build log.

If you didn't try this with the CVS version or with the latest
pretest, please do try one of these, because many problems present in
Emacs 21.x are already resolved in the latest development code.

Thanks in advance.

> Maybe because it tries to build something that depends on 
> something else that has not been quite finished yet because when 
> multiple jobs are specified things doesn't happen in serial anymore?

A bug-free Makefile should never cause any such trouble, since all the
dependencies are supposed to be spelled out in it, and Make will never
start building a file until all its prerequisites are built.

Re: Building cvs emacs on a dual-core machine

[Eli Zaretskii]

> From: storm@cua.dk (Kim F. Storm)
> Date: Wed, 20 Dec 2006 15:40:22 +0100
> Cc: emacs-devel@gnu.org
> 
> Only a fractional part of the build is about compiling C files.

Perhaps I'm missing something, but I don't see the relevance of this,
even if it were true that C compilation is only a small fraction of
the build.  There's nothing to prevent a parallel build from compiling
2 Lisp files at the same time, is there?

> Most of the time is spent inside emacs byte-compiling Lisp files,

Btw, byte compilation is only relevant to bootstrapping.  A normal
build of an Emacs tarball spends mots of its time in C compilation, I
think.  Don't you agree?

Re: Building cvs emacs on a dual-core machine

[Eric Lilja]

Eli Zaretskii skrev:
>> From: Eric Lilja <mindcooler@gmail.com>
>> Date: Wed, 20 Dec 2006 12:47:13 +0100
>>
>> I recently upgraded my computer with a dual-core CPU. I've been enjoying 
>> nice decreases in compile time in my own projects by specifying --jobs=2 
>> when invoking make.
>>
>> However, when I tried that with emacs it started compiling, very fast 
>> with both cores busy, but the build craps out (sorry, no log) after a 
>> little while.
> 
> Any such failures are bugs in the Emacs Makefiles, so please be sure
> to report them with full details, including the build log.

I will. This was with the cvs sources I downloaded yesterday (december 
19th). I have been trying a few times since the beginning of november 
(that's when I got the dual core cpu), neither have been able to build 
when specifying --jobs=2. I will do a fresh checkout later today and 
hopefully provided you with the details you need.

> 
> If you didn't try this with the CVS version or with the latest
> pretest, please do try one of these, because many problems present in
> Emacs 21.x are already resolved in the latest development code.
> 
> Thanks in advance.
> 
>> Maybe because it tries to build something that depends on 
>> something else that has not been quite finished yet because when 
>> multiple jobs are specified things doesn't happen in serial anymore?
> 
> A bug-free Makefile should never cause any such trouble, since all the
> dependencies are supposed to be spelled out in it, and Make will never
> start building a file until all its prerequisites are built.

- Eric

Re: Building cvs emacs on a dual-core machine

[Eric Lilja]

Eric Lilja skrev:
> Eli Zaretskii skrev:
>>> From: Eric Lilja <mindcooler@gmail.com>
>>> Date: Wed, 20 Dec 2006 12:47:13 +0100
>>>
>>> I recently upgraded my computer with a dual-core CPU. I've been 
>>> enjoying nice decreases in compile time in my own projects by 
>>> specifying --jobs=2 when invoking make.
>>>
>>> However, when I tried that with emacs it started compiling, very fast 
>>> with both cores busy, but the build craps out (sorry, no log) after a 
>>> little while.
>>
>> Any such failures are bugs in the Emacs Makefiles, so please be sure
>> to report them with full details, including the build log.
> 
> I will. This was with the cvs sources I downloaded yesterday (december 
> 19th). I have been trying a few times since the beginning of november 
> (that's when I got the dual core cpu), neither have been able to build 
> when specifying --jobs=2. I will do a fresh checkout later today and 
> hopefully provided you with the details you need.

Ok, here's where it errors out (same with --jobs=2 and --jobs=4):
gcc -I. -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=1 -c 
-mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1
=_cdecl    -o oo-spd/i386/preprep.o preprep.c
gcc -o oo-spd/i386/preprep.exe \
                     -mno-cygwin  oo-spd/i386/preprep.o

Essential Lisp files seem to be missing.  You should either
do `make bootstrap' or create `lisp/abbrev.elc' somehow.

mingw32-make: *** [maybe-bootstrap-SH] Error 255

Yes, sorry, I'm on Windows. This is from a fresh cvs checkout performed 
just now. My previous steps were:
./configure.bat --with-gcc --no-cygwin --no-debug
mingw32-make --jobs=4 bootstrap
You mentioned build log, what else information can I provide to help 
track this problem down?

> 
>>
>> If you didn't try this with the CVS version or with the latest
>> pretest, please do try one of these, because many problems present in
>> Emacs 21.x are already resolved in the latest development code.
>>
>> Thanks in advance.
>>
>>> Maybe because it tries to build something that depends on something 
>>> else that has not been quite finished yet because when multiple jobs 
>>> are specified things doesn't happen in serial anymore?
>>
>> A bug-free Makefile should never cause any such trouble, since all the
>> dependencies are supposed to be spelled out in it, and Make will never
>> start building a file until all its prerequisites are built.
> 
> - Eric

- Eric

Re: Building cvs emacs on a dual-core machine

[Richard Stallman]

     Maybe because it tries to build something that depends on 
    something else that has not been quite finished yet because when 
    multiple jobs are specified things doesn't happen in serial anymore?

That is quite likely.  We could fix this if we knew precisely what
the unrecorded dependency is.

But we have to depend on you to figure out what it is.

Re: Building cvs emacs on a dual-core machine

[Eli Zaretskii]

> From: Eric Lilja <mindcooler@gmail.com>
> Date: Wed, 20 Dec 2006 22:03:56 +0100
> 
> Ok, here's where it errors out (same with --jobs=2 and --jobs=4):
> gcc -I. -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=1 -c 
> -mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1
> =_cdecl    -o oo-spd/i386/preprep.o preprep.c
> gcc -o oo-spd/i386/preprep.exe \
>                      -mno-cygwin  oo-spd/i386/preprep.o
> 
> Essential Lisp files seem to be missing.  You should either
> do `make bootstrap' or create `lisp/abbrev.elc' somehow.
> 
> mingw32-make: *** [maybe-bootstrap-SH] Error 255
> 
> Yes, sorry, I'm on Windows.

That's an important piece of knowledge ;-)

What version of Make do you have?  What does "mingw32-make --version"
print?  The --jobs option started to work correctly on Windows only in
the latest version 3.81 of GNU Make.  If you don't have that version
of Make, --jobs will be very unreliable on Windows.

Also, note that, since the jobserver feature in Make is currently not
suppoorted on Windows, the sub-Make's are invoked with the implied
"--jobs=1" option, in effect defeating parallelism in recusrive Make
invocations.  You will have to tweak the recursive Make invocations in
the Makefile's to explicitly pass the --jobs=2 option to recursive
Make's, in order to see parallelism in subdirectories.

Also, what port of sh.exe do you have?

> You mentioned build log, what else information can I provide to help 
> track this problem down?

Instead of just "mingw32-make --jobs=4 bootstrap" run this command
(from the CMD prompt):

   mingw32-make --jobs=4 --debug=j bootstrap 2>&1 | tee build.txt

The file build.txt will then capture all the output of the build
session; post that file here.

(I assume that you have a port of GNU `tee'; if not, you will have to
install it, or capture the session output in some other way.)

I see already a few problems with --jobs related to Windows, but I'd
like to see the information requested above to make sure I don't miss
anything.

TIA

Re: Building cvs emacs on a dual-core machine

[Eric Lilja]

[-- Attachment #1: Type: text/plain, Size: 2717 bytes --]

Eli Zaretskii wrote:
>> From: Eric Lilja <mindcooler@gmail.com>
>> Date: Wed, 20 Dec 2006 22:03:56 +0100
>>
>> Ok, here's where it errors out (same with --jobs=2 and --jobs=4):
>> gcc -I. -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=1 -c 
>> -mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1
>> =_cdecl    -o oo-spd/i386/preprep.o preprep.c
>> gcc -o oo-spd/i386/preprep.exe \
>>                      -mno-cygwin  oo-spd/i386/preprep.o
>>
>> Essential Lisp files seem to be missing.  You should either
>> do `make bootstrap' or create `lisp/abbrev.elc' somehow.
>>
>> mingw32-make: *** [maybe-bootstrap-SH] Error 255
>>
>> Yes, sorry, I'm on Windows.
> 
> That's an important piece of knowledge ;-)
> 
> What version of Make do you have?  What does "mingw32-make --version"
> print?  The --jobs option started to work correctly on Windows only in
> the latest version 3.81 of GNU Make.  If you don't have that version
> of Make, --jobs will be very unreliable on Windows.

$ mingw32-make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i686-pc-mingw32

> 
> Also, note that, since the jobserver feature in Make is currently not
> suppoorted on Windows, the sub-Make's are invoked with the implied
> "--jobs=1" option, in effect defeating parallelism in recusrive Make
> invocations.  You will have to tweak the recursive Make invocations in
> the Makefile's to explicitly pass the --jobs=2 option to recursive
> Make's, in order to see parallelism in subdirectories.
> 
> Also, what port of sh.exe do you have?

$ sh.exe --version
GNU bash, version 3.2.9(10)-release (i686-pc-cygwin)
Copyright (C) 2005 Free Software Foundation, Inc.

> 
>> You mentioned build log, what else information can I provide to help 
>> track this problem down?
> 
> Instead of just "mingw32-make --jobs=4 bootstrap" run this command
> (from the CMD prompt):
> 
>    mingw32-make --jobs=4 --debug=j bootstrap 2>&1 | tee build.txt
> 
> The file build.txt will then capture all the output of the build
> session; post that file here.
> 
> (I assume that you have a port of GNU `tee'; if not, you will have to
> install it, or capture the session output in some other way.)
> 
> I see already a few problems with --jobs related to Windows, but I'd
> like to see the information requested above to make sure I don't miss
> anything.

I ran it and I'm attaching build.txt, but I ran it from TCI running bash 
instead of a regular cmd prompt, hope that I still was able to provide 
you with the information you needed.

> 
> TIA

- Eric

[-- Attachment #2: build.txt --]
[-- Type: text/plain, Size: 2621 bytes --]

GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i686-pc-mingw32
CreateProcess(C:\cygwin\bin\echo.exe,echo,...)
CreateProcess(C:\cygwin\bin\sh.exe,C:/cygwin/bin/sh.exe -c "echo [Please ignore a syntax error on the next line - it is intentional] 1>&2",...)
[Please ignore a syntax error on the next line - it is intentional]
CreateProcess(C:\cygwin\bin\sh.exe,C:/cygwin/bin/sh.exe -c "echo foo\"",...)
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
mkdir "oo-spd"
CreateProcess(C:\cygwin\bin\sh.exe,C:/cygwin/bin/sh.exe -c "mkdir \"oo-spd\"",...)
Putting child 0x009e7808 (oo-spd) PID 10413920 on the chain.
CreateProcess(C:\cygwin\bin\echo.exe,echo Using C:/cygwin/bin/sh.exe as shell.,...)
Putting child 0x009ef2c8 (which-sh) PID 10425384 on the chain.
Using C:/cygwin/bin/sh.exe as shell.
CreateProcess(C:\cygwin\bin\sh.exe,C:/cygwin/bin/sh.exe -c "if [ ! -f ../lisp/abbrev.elc ] ; then \
  echo; \
  echo \"Essential Lisp files seem to be missing.  You should either\"; \
  echo \"do \`make bootstrap' or create \`lisp/abbrev.elc' somehow.\";  \
  echo; \
  exit -1; \
fi",...)
Putting child 0x009f2df0 (maybe-bootstrap-SH) PID 10441560 on the chain.
Live child 0x009f2df0 (maybe-bootstrap-SH) PID 10441560 
Live child 0x009ef2c8 (which-sh) PID 10425384 
Live child 0x009e7808 (oo-spd) PID 10413920 
Reaping winning child 0x009ef2c8 PID 10425384 
Removing child 0x009ef2c8 PID 10425384 from chain.
Live child 0x009f2df0 (maybe-bootstrap-SH) PID 10441560 
Live child 0x009e7808 (oo-spd) PID 10413920 

Essential Lisp files seem to be missing.  You should either
do `make bootstrap' or create `lisp/abbrev.elc' somehow.

Reaping winning child 0x009e7808 PID 10413920 
Removing child 0x009e7808 PID 10413920 from chain.
mkdir "oo-spd/i386"
CreateProcess(C:\cygwin\bin\sh.exe,C:/cygwin/bin/sh.exe -c "mkdir \"oo-spd/i386\"",...)
Putting child 0x009f0f98 (oo-spd/i386) PID 10413920 on the chain.
Live child 0x009f0f98 (oo-spd/i386) PID 10413920 
Live child 0x009f2df0 (maybe-bootstrap-SH) PID 10441560 
Reaping losing child 0x009f2df0 PID 10441560 
mingw32-make: *** [maybe-bootstrap-SH] Error 255
Removing child 0x009f2df0 PID 10441560 from chain.
mingw32-make: *** Waiting for unfinished jobs....
Live child 0x009f0f98 (oo-spd/i386) PID 10413920 
Reaping winning child 0x009f0f98 PID 10413920 
Removing child 0x009f0f98 PID 10413920 from chain.

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Building cvs emacs on a dual-core machine

[Kim F. Storm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: storm@cua.dk (Kim F. Storm)
>> Date: Wed, 20 Dec 2006 15:40:22 +0100
>> Cc: emacs-devel@gnu.org
>> 
>> Only a fractional part of the build is about compiling C files.
>
> Perhaps I'm missing something, but I don't see the relevance of this,
> even if it were true that C compilation is only a small fraction of
> the build.  There's nothing to prevent a parallel build from compiling
> 2 Lisp files at the same time, is there?

No.  Forget what I said.

>
>> Most of the time is spent inside emacs byte-compiling Lisp files,
>
> Btw, byte compilation is only relevant to bootstrapping.  A normal
> build of an Emacs tarball spends mots of its time in C compilation, I
> think.  Don't you agree?

That is true, yes.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Building cvs emacs on a dual-core machine

[Chris Moore]

Eli Zaretskii <eliz@gnu.org> writes:

> There's nothing to prevent a parallel build from compiling 2 Lisp
> files at the same time, is there?

Yes, there is, at least on GNU/Linux make isn't used to determine
which Lisp files to compile.

During bootstrapping, it's done using the shell:

  [...] for el in [...] $els; do echo Compiling $el; [...]; done

and during 'make recompile', it's done using emacs itself:

  ../src/emacs -batch --no-site-file --multibyte --eval
  (batch-byte-recompile-directory 0) /home/chris/programs/emacs2/lisp

Neither of these will honour make's --jobs=2 flag.  In fact using
--jobs=2 makes the build ever so slightly slower on this dual-core
machine.

Re: Building cvs emacs on a dual-core machine

[Eli Zaretskii]

> Cc: storm@cua.dk (Kim F. Storm),  mindcooler@gmail.com,  emacs-devel@gnu.org
> From: Chris Moore <dooglus@gmail.com>
> Date: Fri, 22 Dec 2006 15:39:40 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > There's nothing to prevent a parallel build from compiling 2 Lisp
> > files at the same time, is there?
> 
> Yes, there is, at least on GNU/Linux make isn't used to determine
> which Lisp files to compile.
> 
> During bootstrapping, it's done using the shell:
> 
>   [...] for el in [...] $els; do echo Compiling $el; [...]; done
> 
> and during 'make recompile', it's done using emacs itself:
> 
>   ../src/emacs -batch --no-site-file --multibyte --eval
>   (batch-byte-recompile-directory 0) /home/chris/programs/emacs2/lisp
> 
> Neither of these will honour make's --jobs=2 flag.

It's true that lisp/Makefile.in doesn't give much chance to exploit
parallelism, but it's not true that no parallelism is possible there.
`compile', `recompile', and `cvs-update' have prerequisites that can
be produced in parallel.

leim/Makefile.in is better organized for parallel make, and it's not
inconceivable to have Emacs compile in parallel in lisp/ and in leim/.

Re: Building cvs emacs on a dual-core machine

[Eli Zaretskii]

> From: Eric Lilja <mindcooler@gmail.com>
> Date: Thu, 21 Dec 2006 01:14:45 +0100
> 
> Eli Zaretskii wrote:
> >> From: Eric Lilja <mindcooler@gmail.com>
> >> Date: Wed, 20 Dec 2006 22:03:56 +0100
> >>
> >> Ok, here's where it errors out (same with --jobs=2 and --jobs=4):
> >> gcc -I. -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=1 -c 
> >> -mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1
> >> =_cdecl    -o oo-spd/i386/preprep.o preprep.c
> >> gcc -o oo-spd/i386/preprep.exe \
> >>                      -mno-cygwin  oo-spd/i386/preprep.o
> >>
> >> Essential Lisp files seem to be missing.  You should either
> >> do `make bootstrap' or create `lisp/abbrev.elc' somehow.
> >>
> >> mingw32-make: *** [maybe-bootstrap-SH] Error 255
> >>
> >> Yes, sorry, I'm on Windows.
> > 
> > That's an important piece of knowledge ;-)
> > 
> > What version of Make do you have?  What does "mingw32-make --version"
> > print?  The --jobs option started to work correctly on Windows only in
> > the latest version 3.81 of GNU Make.  If you don't have that version
> > of Make, --jobs will be very unreliable on Windows.
> 
> $ mingw32-make --version
> GNU Make 3.81

I found and fixed several problems with Windows makefile's which
prevented parallel builds.  Please resync with CVS and try again, I
hope it will work now.

Note that, since the native Windows build of Make (including
mingw32-make) doesn't yet support the jobserver feature, it by default
invokes sub-Make's with --jobs=1 switch, and thus defeats parallelism
in recursive Make invocations.  I made a provision for working around
this; to use it, invoke the top-level Make in the nt/ subdirectory
like this:

    mingw32-make bootstrap --jobs=2 XMFLAGS="--jobs=2"

The XMFLAGS variable is used to pass an explicit --jobs option to
sub-Make's, overriding the default --jobs=1.  (There's no requirement
that the argument to --jobs on the command line must be identical to
what you put in XMFLAGS, you can use different numbers if you wish.)

Please tell if you have any additional problems with parallel builds
on Windows.

And thanks again for drawing our attention to this.

Re: Building cvs emacs on a dual-core machine

[Eric Lilja]

Eli Zaretskii skrev:
>> From: Eric Lilja <mindcooler@gmail.com>
>> Date: Thu, 21 Dec 2006 01:14:45 +0100
>>
>> Eli Zaretskii wrote:
>>>> From: Eric Lilja <mindcooler@gmail.com>
>>>> Date: Wed, 20 Dec 2006 22:03:56 +0100
>>>>
>>>> Ok, here's where it errors out (same with --jobs=2 and --jobs=4):
>>>> gcc -I. -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=1 -c 
>>>> -mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1
>>>> =_cdecl    -o oo-spd/i386/preprep.o preprep.c
>>>> gcc -o oo-spd/i386/preprep.exe \
>>>>                      -mno-cygwin  oo-spd/i386/preprep.o
>>>>
>>>> Essential Lisp files seem to be missing.  You should either
>>>> do `make bootstrap' or create `lisp/abbrev.elc' somehow.
>>>>
>>>> mingw32-make: *** [maybe-bootstrap-SH] Error 255
>>>>
>>>> Yes, sorry, I'm on Windows.
>>> That's an important piece of knowledge ;-)
>>>
>>> What version of Make do you have?  What does "mingw32-make --version"
>>> print?  The --jobs option started to work correctly on Windows only in
>>> the latest version 3.81 of GNU Make.  If you don't have that version
>>> of Make, --jobs will be very unreliable on Windows.
>> $ mingw32-make --version
>> GNU Make 3.81
> 
> I found and fixed several problems with Windows makefile's which
> prevented parallel builds.  Please resync with CVS and try again, I
> hope it will work now.
> 
> Note that, since the native Windows build of Make (including
> mingw32-make) doesn't yet support the jobserver feature, it by default
> invokes sub-Make's with --jobs=1 switch, and thus defeats parallelism
> in recursive Make invocations.  I made a provision for working around
> this; to use it, invoke the top-level Make in the nt/ subdirectory
> like this:
> 
>     mingw32-make bootstrap --jobs=2 XMFLAGS="--jobs=2"
> 
> The XMFLAGS variable is used to pass an explicit --jobs option to
> sub-Make's, overriding the default --jobs=1.  (There's no requirement
> that the argument to --jobs on the command line must be identical to
> what you put in XMFLAGS, you can use different numbers if you wish.)
> 
> Please tell if you have any additional problems with parallel builds
> on Windows.
> 
> And thanks again for drawing our attention to this.

I tried your changes on a fresh checkout (sorry for using time but I 
thought it would work and wanted to see how big decrease in compile time 
I would get with your fixes):
$ time mingw32-make bootstrap --jobs=2 XMFLAGS="--jobs=2"
[Please ignore a syntax error on the next line - it is intentional]
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
mkdir "oo-spd"
mkdir "oo-spd/i386"
echo oo-spd/i386 > stamp_BLD
gcc -I. -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=1 -c 
-mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1
=_cdecl    -o oo-spd/i386/addsection.o addsection.c
gcc -o oo-spd/i386/addsection.exe \
                     -mno-cygwin  oo-spd/i386/addsection.o   -luser32
c:/mingw32-make-3.81-1/bin/mingw32-make -j 1 --jobs=2 -C ../lisp 
bootstrap-clean
[Please ignore a syntax error on the next line - it is intentional]
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
mingw32-make[1]: Entering directory `c:/cvsemacs/emacs/lisp'
cp c:/cvsemacs/emacs/lisp/ldefs-boot.el c:/cvsemacs/emacs/lisp/loaddefs.el
c:/mingw32-make-3.81-1/bin/mingw32-make -wj 1 loaddefs.el-SH
for dir in . calc calendar emacs-lisp emulation erc eshell gnus 
international language mail mh-e net play progmodes
term textmodes url obsolete; do rm -f $dir/*.elc; done
[Please ignore a syntax error on the next line - it is intentional]
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
mingw32-make[2]: Entering directory `c:/cvsemacs/emacs/lisp'
echo ";;; loaddefs.el --- automatically extracted autoloads" > 
loaddefs.el-SH
echo ";;" >> loaddefs.el-SH; echo ";;; Code:" >> loaddefs.el-SH
echo "(autoload 'define-minor-mode \"easy-mmode\")" >>loaddefs.el-SH
echo "(autoload 'define-ccl-program \"ccl\")" >>loaddefs.el-SH
echo "(autoload 'regexp-opt \"regexp-opt\")" >>loaddefs.el-SH
echo "(autoload 'string-to-list \"mule-util\")" >>loaddefs.el-SH
echo "(autoload 'define-derived-mode \"derived\")" >>loaddefs.el-SH
echo "(autoload 'encoded-kbd-mode \"encoded-kb\")" >>loaddefs.el-SH
echo "(defvar cvs-global-menu nil)" >>loaddefs.el-SH
echo "♀" >> loaddefs.el-SH
echo ";;; Local Variables:" >> loaddefs.el-SH
echo ";;; version-control: never" >> loaddefs.el-SH
echo ";;; no-byte-compile: t" >> loaddefs.el-SH
echo ";;; no-update-autoloads: t" >> loaddefs.el-SH
echo ";;; End:" >> loaddefs.el-SH
echo ";;; loaddefs.el ends here" >> loaddefs.el-SH
mingw32-make[2]: Leaving directory `c:/cvsemacs/emacs/lisp'
cp loaddefs.el-SH c:/cvsemacs/emacs/lisp/loaddefs.el
rm loaddefs.el-SH
mingw32-make[1]: Leaving directory `c:/cvsemacs/emacs/lisp'
c:/mingw32-make-3.81-1/bin/mingw32-make -j 1 --jobs=2 -C ../src clean
[Please ignore a syntax error on the next line - it is intentional]
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
mingw32-make[1]: Entering directory `c:/cvsemacs/emacs/src'
rm *~ "s/*~" "m/*~"
rm: cannot remove `*~': No such file or directory
rm: cannot remove `s/*~': No such file or directory
rm: cannot remove `m/*~': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
rm
rm: missing operand
Try `rm --help' for more information.
mingw32-make[1]: [clean] Error 1 (ignored)
rm -r oo-spd
rm: cannot remove `oo-spd': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
rm stamp_BLD
rm: cannot remove `stamp_BLD': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
mingw32-make[1]: Leaving directory `c:/cvsemacs/emacs/src'
c:/mingw32-make-3.81-1/bin/mingw32-make -j 1 --jobs=2 -C ../lib-src clean
[Please ignore a syntax error on the next line - it is intentional]
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
mingw32-make[1]: Entering directory `c:/cvsemacs/emacs/lib-src'
rm *~ DOC*
rm: cannot remove `*~': No such file or directory
rm: cannot remove `DOC*': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
rm ctags.c
rm: cannot remove `ctags.c': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
rm getopt.h
rm: cannot remove `getopt.h': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
rm -r oo-spd
rm: cannot remove `oo-spd': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
rm stamp_BLD
rm: cannot remove `stamp_BLD': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
mingw32-make[1]: Leaving directory `c:/cvsemacs/emacs/lib-src'
c:/mingw32-make-3.81-1/bin/mingw32-make -j 1 --jobs=2 XMFLAGS="--jobs=2" 
-C ../src bootstrap
[Please ignore a syntax error on the next line - it is intentional]
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
mingw32-make[1]: Entering directory `c:/cvsemacs/emacs/nt'
mingw32-make[1]: Nothing to be done for `../src'.
c:/mingw32-make-3.81-1/bin/mingw32-make -wj 1 --jobs=2 -C -C ../lisp 
bootstrap-clean
mingw32-make: Entering an unknown directory
mingw32-make: *** -C: No such file or directory.  Stop.
mingw32-make: Leaving an unknown directory
mingw32-make[1]: *** [bootstrap-gmake] Error 2
mingw32-make[1]: Leaving directory `c:/cvsemacs/emacs/nt'
mingw32-make: *** [bootstrap-gmake] Error 2

real    0m5.093s
user    0m0.015s
sys     0m0.000s

hivemind@mindcooler /cygdrive/c/cvsemacs/emacs/nt
$


Also, doing an ordinary bootstrap without specifying jobs or XMFLAGS 
doesn't seem to work right now either.

- Eric

Re: Building cvs emacs on a dual-core machine

[Chris Moore]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Chris Moore <dooglus@gmail.com>
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > There's nothing to prevent a parallel build from compiling 2 Lisp
>> > files at the same time, is there?
>> 
>> Yes, there is, at least on GNU/Linux make isn't used to determine
>> which Lisp files to compile.
>> 
> It's true that lisp/Makefile.in doesn't give much chance to exploit
> parallelism, but it's not true that no parallelism is possible there.
> `compile', `recompile', and `cvs-update' have prerequisites that can
> be produced in parallel.

OK maybe it's possible, but it doesn't seem to happen to any
significant extent.

While compiling C code, both cores are used, and while compiling Lisp
code one of them is pretty much idle.

That's what I see, anyway.

Re: Building cvs emacs on a dual-core machine

[Eli Zaretskii]

> From: Eric Lilja <mindcooler@gmail.com>
> Date: Sat, 23 Dec 2006 00:30:54 +0100
> 
> c:/mingw32-make-3.81-1/bin/mingw32-make -wj 1 --jobs=2 -C -C ../lisp bootstrap-clean
> mingw32-make: Entering an unknown directory
> mingw32-make: *** -C: No such file or directory.  Stop.

Fascinating... where did that second -C come from?  Can you try to
find out?

> Also, doing an ordinary bootstrap without specifying jobs or XMFLAGS 
> doesn't seem to work right now either.

Please show a the error message in that case.  Bootstrap works for me,
both with -j 2 and without it, but I don't have a dual core machine,
only a hyper-threaded one.

Also, could you please invoke Make from CMD prompt, not from the
Cygwin shell?

Ordinary bootstrap works for me, I just verified that again.  Did you
try in a clean tree, or in the one where the -j 2 bootstrap failed?
If the latter, perhaps the cleaning targets don't do a good enough
job; please try in a clean tree.

Re: Building cvs emacs on a dual-core machine

[Juanma Barranquero]

On 12/22/06, Eli Zaretskii <eliz@gnu.org> wrote:

> I found and fixed several problems with Windows makefile's which
> prevented parallel builds.

BTW,

C:\...\HEAD> cvs -q update
? lib-src/stamp_BLD
? nt/stamp_BLD
? src/stamp_BLD

Should we add stamp_BLD to a few .cvsignore files?

                    /L/e/k/t/u

Re: Building cvs emacs on a dual-core machine

[Eric Lilja]

Eli Zaretskii skrev:
>> From: Eric Lilja <mindcooler@gmail.com>
>> Date: Sat, 23 Dec 2006 00:30:54 +0100
>>
>> c:/mingw32-make-3.81-1/bin/mingw32-make -wj 1 --jobs=2 -C -C ../lisp bootstrap-clean
>> mingw32-make: Entering an unknown directory
>> mingw32-make: *** -C: No such file or directory.  Stop.
> 
> Fascinating... where did that second -C come from?  Can you try to
> find out?
> 
>> Also, doing an ordinary bootstrap without specifying jobs or XMFLAGS 
>> doesn't seem to work right now either.
> 
> Please show a the error message in that case.  Bootstrap works for me,
> both with -j 2 and without it, but I don't have a dual core machine,
> only a hyper-threaded one.
> 
> Also, could you please invoke Make from CMD prompt, not from the
> Cygwin shell?
> 
> Ordinary bootstrap works for me, I just verified that again.  Did you
> try in a clean tree, or in the one where the -j 2 bootstrap failed?
> If the latter, perhaps the cleaning targets don't do a good enough
> job; please try in a clean tree.


I did a fresh checkout again. I cannot build emacs anymore. I didn't 
specify jobs at all. I tried both cmd.com and cygwin's bash. cmd.com yields:
C:\cvsemacs\emacs\nt>mingw32-make bootstrap
mkdir "oo-spd"
mkdir "oo-spd/i386"
echo oo-spd/i386 > stamp_BLD
gcc -I. -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=1 -c 
-mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1=_cdecl    -o 
oo-spd/i386/addsection.o addsection.c
process_begin: CreateProcess(NULL, gcc -I. -DWIN32_LEAN_AND_MEAN 
-D_WIN32_WINNT=0x0400 -D_X86_=1 -c -mno-cygwin -mtune=pentium4 -O2 
-Di386 -D_CRTAPI1=_cdecl -o
oo-spd/i386/addsection.o addsection.c, ...) failed.
make (e=2): The system cannot find the file specified.
mingw32-make: *** [oo-spd/i386/addsection.o] Error 2

(I had to configure from bash because configure.bat said it needed "cp")

 From bash:
$ mingw32-make bootstrap
[Please ignore a syntax error on the next line - it is intentional]
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
gcc -I. -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=1 -c 
-mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1
=_cdecl    -o oo-spd/i386/addsection.o addsection.c
gcc -o oo-spd/i386/addsection.exe \
                     -mno-cygwin  oo-spd/i386/addsection.o   -luser32
c:/mingw32-make-3.81-1/bin/mingw32-make   -C ../lisp bootstrap-clean
[Please ignore a syntax error on the next line - it is intentional]
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
mingw32-make[1]: Entering directory `c:/cvsemacs/emacs/lisp'
cp c:/cvsemacs/emacs/lisp/ldefs-boot.el c:/cvsemacs/emacs/lisp/loaddefs.el
for dir in . calc calendar emacs-lisp emulation erc eshell gnus 
international language mail mh-e net play progmodes
term textmodes url obsolete; do rm -f $dir/*.elc; done
mingw32-make[1]: Leaving directory `c:/cvsemacs/emacs/lisp'
c:/mingw32-make-3.81-1/bin/mingw32-make   -C ../src clean
[Please ignore a syntax error on the next line - it is intentional]
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
mingw32-make[1]: Entering directory `c:/cvsemacs/emacs/src'
rm *~ "s/*~" "m/*~"
rm: cannot remove `*~': No such file or directory
rm: cannot remove `s/*~': No such file or directory
rm: cannot remove `m/*~': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
rm
rm: missing operand
Try `rm --help' for more information.
mingw32-make[1]: [clean] Error 1 (ignored)
rm -r oo-spd
rm: cannot remove `oo-spd': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
rm stamp_BLD
rm: cannot remove `stamp_BLD': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
mingw32-make[1]: Leaving directory `c:/cvsemacs/emacs/src'
c:/mingw32-make-3.81-1/bin/mingw32-make   -C ../lib-src clean
[Please ignore a syntax error on the next line - it is intentional]
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
mingw32-make[1]: Entering directory `c:/cvsemacs/emacs/lib-src'
rm *~ DOC*
rm: cannot remove `*~': No such file or directory
rm: cannot remove `DOC*': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
rm ctags.c
rm: cannot remove `ctags.c': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
rm getopt.h
rm: cannot remove `getopt.h': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
rm -r oo-spd
rm: cannot remove `oo-spd': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
rm stamp_BLD
rm: cannot remove `stamp_BLD': No such file or directory
mingw32-make[1]: [clean] Error 1 (ignored)
mingw32-make[1]: Leaving directory `c:/cvsemacs/emacs/lib-src'
c:/mingw32-make-3.81-1/bin/mingw32-make   XMFLAGS="" -C ../src bootstrap
[Please ignore a syntax error on the next line - it is intentional]
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
mingw32-make[1]: Entering directory `c:/cvsemacs/emacs/src'
c:/mingw32-make-3.81-1/bin/mingw32-make -w \ temacs CFLAGS='-I. 
-DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=
1 -c  -mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1=_cdecl 
-Demacs=1 -DWINDOWSNT -DDOS_NT -DHAVE_CONFIG_H -I.
./nt/inc -D_UCHAR_T -DHAVE_NTGUI=1 -DUSE_CRT_DLL=1 -DPURESIZE=5000000'
[Please ignore a syntax error on the next line - it is intentional]
/usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/sh: -c: line 1: syntax error: unexpected end of file
mingw32-make[2]: Entering directory `c:/cvsemacs/emacs/src'
mingw32-make[2]: *** No rule to make target ` temacs'.  Stop.
mingw32-make[2]: Leaving directory `c:/cvsemacs/emacs/src'
mingw32-make[1]: *** [bootstrap-temacs] Error 2
mingw32-make[1]: Leaving directory `c:/cvsemacs/emacs/src'
mingw32-make: *** [bootstrap-gmake] Error 2


Hope this helps, I won't be able to respond for a few hours, it's past 3 
AM here and I going to visist my parents for holidays tomorrow.

- Eric

Re: Building cvs emacs on a dual-core machine

[Eli Zaretskii]

> Date: Sat, 23 Dec 2006 02:54:57 +0100
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: "Eric Lilja" <mindcooler@gmail.com>, emacs-devel@gnu.org
> 
> C:\...\HEAD> cvs -q update
> ? lib-src/stamp_BLD
> ? nt/stamp_BLD
> ? src/stamp_BLD
> 
> Should we add stamp_BLD to a few .cvsignore files?

Definitely.  Fixed.

Re: Building cvs emacs on a dual-core machine

[Eli Zaretskii]

> From: Eric Lilja <mindcooler@gmail.com>
> Date: Sat, 23 Dec 2006 03:04:30 +0100
> 
> I cannot build emacs anymore.

Sorry to hear that, but I cannot reproduce any of the failures on my
machine, it still bootstraps and builds for me, both with and without
"--jobs".

> I didn't specify jobs at all. I tried both cmd.com and cygwin's
> bash. cmd.com yields:
> C:\cvsemacs\emacs\nt>mingw32-make bootstrap
> mkdir "oo-spd"
> mkdir "oo-spd/i386"
> echo oo-spd/i386 > stamp_BLD
> gcc -I. -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=1 -c 
> -mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1=_cdecl    -o 
> oo-spd/i386/addsection.o addsection.c
> process_begin: CreateProcess(NULL, gcc -I. -DWIN32_LEAN_AND_MEAN 
> -D_WIN32_WINNT=0x0400 -D_X86_=1 -c -mno-cygwin -mtune=pentium4 -O2 
> -Di386 -D_CRTAPI1=_cdecl -o
> oo-spd/i386/addsection.o addsection.c, ...) failed.
> make (e=2): The system cannot find the file specified.
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I feel terribly stupid, because I cannot figure out what is the file
it doesn't find here.  It cannot be the oo-spd/i386 directory since it
was just created before the compilation command.  And it cannot be
addsection.c, because you are in the nt/ subdirectory, where
addsection.c lives.  Could you please try to find out what file is
missing?

>  From bash:
> $ mingw32-make bootstrap
> [...]
> mingw32-make[1]: Leaving directory `c:/cvsemacs/emacs/lib-src'
> c:/mingw32-make-3.81-1/bin/mingw32-make   XMFLAGS="" -C ../src bootstrap
> [Please ignore a syntax error on the next line - it is intentional]
> /usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
> /usr/bin/sh: -c: line 1: syntax error: unexpected end of file
> mingw32-make[1]: Entering directory `c:/cvsemacs/emacs/src'
> c:/mingw32-make-3.81-1/bin/mingw32-make -w \ temacs CFLAGS='-I. 
                                             ^^^^^^^^
> -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=
> 1 -c  -mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1=_cdecl 
> -Demacs=1 -DWINDOWSNT -DDOS_NT -DHAVE_CONFIG_H -I.
> ./nt/inc -D_UCHAR_T -DHAVE_NTGUI=1 -DUSE_CRT_DLL=1 -DPURESIZE=5000000'
> [Please ignore a syntax error on the next line - it is intentional]
> /usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
> /usr/bin/sh: -c: line 1: syntax error: unexpected end of file
> mingw32-make[2]: Entering directory `c:/cvsemacs/emacs/src'
> mingw32-make[2]: *** No rule to make target ` temacs'.  Stop.
                                               ^^^^^^^^
I think I can understand this one; sounds like some bug in the MinGW
make and/or in its interaction with the Cygwin Bash.  Note the lone
backslash above and the leading blank in ` temacs'.

I made some changes to work around this; please resync and try again.
(Needless to say, it still bootstraps for me, at least without --jobs.
Will try with --jobs shortly.)

Re: Building cvs emacs on a dual-core machine

[Eric Lilja]

Eli Zaretskii skrev:
>> From: Eric Lilja <mindcooler@gmail.com>
>> Date: Sat, 23 Dec 2006 03:04:30 +0100
>>
>> I cannot build emacs anymore.
> 
> Sorry to hear that, but I cannot reproduce any of the failures on my
> machine, it still bootstraps and builds for me, both with and without
> "--jobs".
> 
>> I didn't specify jobs at all. I tried both cmd.com and cygwin's
>> bash. cmd.com yields:
>> C:\cvsemacs\emacs\nt>mingw32-make bootstrap
>> mkdir "oo-spd"
>> mkdir "oo-spd/i386"
>> echo oo-spd/i386 > stamp_BLD
>> gcc -I. -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=1 -c 
>> -mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1=_cdecl    -o 
>> oo-spd/i386/addsection.o addsection.c
>> process_begin: CreateProcess(NULL, gcc -I. -DWIN32_LEAN_AND_MEAN 
>> -D_WIN32_WINNT=0x0400 -D_X86_=1 -c -mno-cygwin -mtune=pentium4 -O2 
>> -Di386 -D_CRTAPI1=_cdecl -o
>> oo-spd/i386/addsection.o addsection.c, ...) failed.
>> make (e=2): The system cannot find the file specified.
>               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> I feel terribly stupid, because I cannot figure out what is the file
> it doesn't find here.  It cannot be the oo-spd/i386 directory since it
> was just created before the compilation command.  And it cannot be
> addsection.c, because you are in the nt/ subdirectory, where
> addsection.c lives.  Could you please try to find out what file is
> missing?
> 
>>  From bash:
>> $ mingw32-make bootstrap
>> [...]
>> mingw32-make[1]: Leaving directory `c:/cvsemacs/emacs/lib-src'
>> c:/mingw32-make-3.81-1/bin/mingw32-make   XMFLAGS="" -C ../src bootstrap
>> [Please ignore a syntax error on the next line - it is intentional]
>> /usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
>> /usr/bin/sh: -c: line 1: syntax error: unexpected end of file
>> mingw32-make[1]: Entering directory `c:/cvsemacs/emacs/src'
>> c:/mingw32-make-3.81-1/bin/mingw32-make -w \ temacs CFLAGS='-I. 
>                                              ^^^^^^^^
>> -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=
>> 1 -c  -mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1=_cdecl 
>> -Demacs=1 -DWINDOWSNT -DDOS_NT -DHAVE_CONFIG_H -I.
>> ./nt/inc -D_UCHAR_T -DHAVE_NTGUI=1 -DUSE_CRT_DLL=1 -DPURESIZE=5000000'
>> [Please ignore a syntax error on the next line - it is intentional]
>> /usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
>> /usr/bin/sh: -c: line 1: syntax error: unexpected end of file
>> mingw32-make[2]: Entering directory `c:/cvsemacs/emacs/src'
>> mingw32-make[2]: *** No rule to make target ` temacs'.  Stop.
>                                                ^^^^^^^^
> I think I can understand this one; sounds like some bug in the MinGW
> make and/or in its interaction with the Cygwin Bash.  Note the lone
> backslash above and the leading blank in ` temacs'.
> 
> I made some changes to work around this; please resync and try again.
> (Needless to say, it still bootstraps for me, at least without --jobs.
> Will try with --jobs shortly.)

I just performed a fresh checkout and managed to complete the entire 
build bootstrap process without problem (this one without specifying 
--jobs=2). The only thing different from last night was that I had 
upgraded TCI (an alternative to rxvt) to the latest beta but that 
shouldn't make a difference I think. I did this checkout before I saw 
the message I'm replying to now, don't know if I got the changes or not 
but it worked now. I will do a rebuild again and time it this time and 
then time another rebuild this time with --jobs=2 and see if both can 
complete successfully (and their respective times will be interesting 
for me to compare).

- Eric

Re: Building cvs emacs on a dual-core machine

[Eric Lilja]

Eric Lilja skrev:
> Eli Zaretskii skrev:
>>> From: Eric Lilja <mindcooler@gmail.com>
>>> Date: Sat, 23 Dec 2006 03:04:30 +0100
>>>
>>> I cannot build emacs anymore.
>>
>> Sorry to hear that, but I cannot reproduce any of the failures on my
>> machine, it still bootstraps and builds for me, both with and without
>> "--jobs".
>>
>>> I didn't specify jobs at all. I tried both cmd.com and cygwin's
>>> bash. cmd.com yields:
>>> C:\cvsemacs\emacs\nt>mingw32-make bootstrap
>>> mkdir "oo-spd"
>>> mkdir "oo-spd/i386"
>>> echo oo-spd/i386 > stamp_BLD
>>> gcc -I. -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=1 -c 
>>> -mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1=_cdecl    -o 
>>> oo-spd/i386/addsection.o addsection.c
>>> process_begin: CreateProcess(NULL, gcc -I. -DWIN32_LEAN_AND_MEAN 
>>> -D_WIN32_WINNT=0x0400 -D_X86_=1 -c -mno-cygwin -mtune=pentium4 -O2 
>>> -Di386 -D_CRTAPI1=_cdecl -o
>>> oo-spd/i386/addsection.o addsection.c, ...) failed.
>>> make (e=2): The system cannot find the file specified.
>>               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> I feel terribly stupid, because I cannot figure out what is the file
>> it doesn't find here.  It cannot be the oo-spd/i386 directory since it
>> was just created before the compilation command.  And it cannot be
>> addsection.c, because you are in the nt/ subdirectory, where
>> addsection.c lives.  Could you please try to find out what file is
>> missing?
>>
>>>  From bash:
>>> $ mingw32-make bootstrap
>>> [...]
>>> mingw32-make[1]: Leaving directory `c:/cvsemacs/emacs/lib-src'
>>> c:/mingw32-make-3.81-1/bin/mingw32-make   XMFLAGS="" -C ../src bootstrap
>>> [Please ignore a syntax error on the next line - it is intentional]
>>> /usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
>>> /usr/bin/sh: -c: line 1: syntax error: unexpected end of file
>>> mingw32-make[1]: Entering directory `c:/cvsemacs/emacs/src'
>>> c:/mingw32-make-3.81-1/bin/mingw32-make -w \ temacs CFLAGS='-I. 
>>                                              ^^^^^^^^
>>> -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x0400 -D_X86_=
>>> 1 -c  -mno-cygwin -mtune=pentium4 -O2  -Di386 -D_CRTAPI1=_cdecl 
>>> -Demacs=1 -DWINDOWSNT -DDOS_NT -DHAVE_CONFIG_H -I.
>>> ./nt/inc -D_UCHAR_T -DHAVE_NTGUI=1 -DUSE_CRT_DLL=1 -DPURESIZE=5000000'
>>> [Please ignore a syntax error on the next line - it is intentional]
>>> /usr/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
>>> /usr/bin/sh: -c: line 1: syntax error: unexpected end of file
>>> mingw32-make[2]: Entering directory `c:/cvsemacs/emacs/src'
>>> mingw32-make[2]: *** No rule to make target ` temacs'.  Stop.
>>                                                ^^^^^^^^
>> I think I can understand this one; sounds like some bug in the MinGW
>> make and/or in its interaction with the Cygwin Bash.  Note the lone
>> backslash above and the leading blank in ` temacs'.
>>
>> I made some changes to work around this; please resync and try again.
>> (Needless to say, it still bootstraps for me, at least without --jobs.
>> Will try with --jobs shortly.)
> 
> I just performed a fresh checkout and managed to complete the entire 
> build bootstrap process without problem (this one without specifying 
> --jobs=2). The only thing different from last night was that I had 
> upgraded TCI (an alternative to rxvt) to the latest beta but that 
> shouldn't make a difference I think. I did this checkout before I saw 
> the message I'm replying to now, don't know if I got the changes or not 
> but it worked now. I will do a rebuild again and time it this time and 
> then time another rebuild this time with --jobs=2 and see if both can 
> complete successfully (and their respective times will be interesting 
> for me to compare).

Another rebuild: Without jobs, ~11 minutes
With jobs: Will not complete. :-(

Oh well, at least I can build myself a working emacs now.

> 
> - Eric

- Eric

Re: Building cvs emacs on a dual-core machine

[Eli Zaretskii]

> From: Eric Lilja <mindcooler@gmail.com>
> Date: Sat, 23 Dec 2006 12:24:21 +0100
> 
> Another rebuild: Without jobs, ~11 minutes

Same here.  I get between 11 and 13 minutes, depending on whether I
use "-j 2" or not.  This is a single-core system with a 3-GHz
hyper-threaded CPU.

> With jobs: Will not complete. :-(

Please show me the error messages which prevented it from finishing.

For repeatability's sake, please always start with a fresh CVS tree
when you try another bootstrap, until we resolve the problems.  To
avoid the need to checkout the entire tree each time, what I do is
immediately after a checkout copy the fresh tree to another place (you
can compress it if you want to preserve space), and then, before each
new build, I remove the old tree, where I built Emacs the previous
time, and copy the fresh tree in its place.  This makes every build go
through the same steps, and makes debugging easier.  Could you please
adopt a similar scheme?

FWIW, I have now finished testing the latest changes in the various
makefile.w32-in files.  I tested all the combinations I could try on
this system:

  . with sh.exe and with CMD
  . with and without '-j 2 XMFLAGS="-j 2"' options
  . both CVS HEAD and the 22.0.92 tarball (with makefile.w32-in files
    and gmake.defs replaced by the ones in CVS HEAD)

All combinations of these were tested, and they all succeeded.

So for further testing, I must rely on your data.  We use different
development tools (e.g., I don't have Cygwin installed, and my gmake
is compiled locally from the official sources, not downloaded from the
MinGW site), so perhaps some problems only rear their ugly heads with
your choice of tools.

TIA

Re: Building cvs emacs on a dual-core machine

[Randal L. Schwartz]

>>>>> "Richard" == Richard Stallman <rms@gnu.org> writes:

Richard>      Maybe because it tries to build something that depends on 
Richard>     something else that has not been quite finished yet because when 
Richard>     multiple jobs are specified things doesn't happen in serial anymore?

Richard> That is quite likely.  We could fix this if we knew precisely what
Richard> the unrecorded dependency is.

Richard> But we have to depend on you to figure out what it is.

Yeah, and that's tough, because it's a timing issue, and those are tricky.

For the record, *I* also tried

        make clean
        make -j3 bootstrap

and it also failed, midway through the build, because some things
weren't ready "in time".

So, there's clearly some dependency not being marked properly in the
Makefiles, but the only solution is to painstakingly go through each and every
Makefile and see where -j3 would have gone parallel, and make sure there's no
sequential items, or any items that use a "temp file" with a fixed name.

It's hard.  Unless the Makefile was designed with -j3 in mind, it generally
*doesn't* work, in my experience.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: Building cvs emacs on a dual-core machine

[Richard Stallman]

    So, there's clearly some dependency not being marked properly in the
    Makefiles, but the only solution is to painstakingly go through each and every
    Makefile and see where -j3 would have gone parallel, and make sure there's no
    sequential items, or any items that use a "temp file" with a fixed name.

First of all, which compilation command failed?  That will give us one
side of the problem.  Second, which targets have not yet been made?
That will narrow down the other side, down to a few dozen
possibilities at most, maybe less.

You could probably then figure out by hand which one it is.

Re: Building cvs emacs on a dual-core machine

[Eric Lilja]

Eli Zaretskii wrote:
>> From: Eric Lilja [snip raw email address quote]
>> Date: Sat, 23 Dec 2006 12:24:21 +0100
>>
>> Another rebuild: Without jobs, ~11 minutes
> 
> Same here.  I get between 11 and 13 minutes, depending on whether I
> use "-j 2" or not.  This is a single-core system with a 3-GHz
> hyper-threaded CPU.
> 
>> With jobs: Will not complete. :-(
> 
> Please show me the error messages which prevented it from finishing.

Sorry for the delay, but I have been gone for most of the holidays. Hope 
you notice my update to this old thread (easy to miss if you use gmane 
like me).

Anyway, I just performed a full checkout and was able to build emacs 
using the following script:

#!/bin/bash
export CVS_RSH="ssh"
touch ~/.cvspass
cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/sources/emacs co emacs
rm ~/.cvspass
cd emacs/nt/
./configure.bat --with-gcc --no-cygwin --no-debug
time mingw32-make bootstrap --jobs=2 XMFLAGS="--jobs=2"
mingw32-make install

Same system as before. So with the changes you made I can now perform a 
full bootstrap when specifying the jobs parameter. :-) I didn't try to 
build without specifying jobs, but I'm sure it works if this script 
works (and it does). Thanks for fixing this! As you mentioned, 
specifying jobs provide no real benefit under windows at the moment but 
it still feels good that this has been fixed.

- Eric

Re: Building cvs emacs on a dual-core machine

[Eli Zaretskii]

> From: Eric Lilja <mindcooler@gmail.com>
> Date: Wed, 03 Jan 2007 21:56:00 +0100
> 
> Anyway, I just performed a full checkout and was able to build emacs 
> using the following script:
> 
> #!/bin/bash
> export CVS_RSH="ssh"
> touch ~/.cvspass
> cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/sources/emacs co emacs
> rm ~/.cvspass
> cd emacs/nt/
> ./configure.bat --with-gcc --no-cygwin --no-debug
> time mingw32-make bootstrap --jobs=2 XMFLAGS="--jobs=2"
> mingw32-make install
> 
> Same system as before. So with the changes you made I can now perform a 
> full bootstrap when specifying the jobs parameter. :-)

Thanks for testing.

> As you mentioned, specifying jobs provide no real benefit under
> windows at the moment but it still feels good that this has been
> fixed.

Yes, it means that at least our Makefile's are now more correct in
stating dependencies between various targets.

Re: Building cvs emacs on a dual-core machine

[Randal L. Schwartz]

>>>>> "Eric" == Eric Lilja <mindcooler@gmail.com> writes:

Eric> Same system as before. So with the changes you made I can now perform a
Eric> full bootstrap when specifying the jobs parameter. :-) I didn't try to
Eric> build without specifying jobs, but I'm sure it works if this script
Eric> works (and it does). Thanks for fixing this! As you mentioned,
Eric> specifying jobs provide no real benefit under windows at the moment but
Eric> it still feels good that this has been fixed.

I just did:
        cvs -q up
        make clean # to start from a known base
        time make -j3 bootstrap

The C compiles at the beginning are much faster, although
the .el => .elc phase appears to be single threaded, but worked fine.

Oddly enough, it seems to have taken twice as long!  Wow.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: Building cvs emacs on a dual-core machine

[Eli Zaretskii]

> From: merlyn@stonehenge.com (Randal L. Schwartz)
> Date: 04 Jan 2007 12:17:35 -0800
> 
> I just did:
>         cvs -q up
>         make clean # to start from a known base

Is "make clean" indeed enough to remove everything generated by
bootstrap?  I think you need maintainer-clean.

>         time make -j3 bootstrap
> 
> The C compiles at the beginning are much faster, although
> the .el => .elc phase appears to be single threaded, but worked fine.

The byte compilation runs in a single Emacs, but Make should run
several other commands in parallel to byte compilation, such as
autoloads, finder-data, and custom-deps.  So at least the first part
of byte compilation runs in parallel with another process

> Oddly enough, it seems to have taken twice as long!  Wow.

What took twice as long, exactly?  The byte compilation stage?