Two strange messages while building Emacs on MS-Windows

Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

Hello,

Lately, I'm seeing two strange messages while building Emacs on MS-Windows.

First message: "conflicting tags" (during a `bzr pull' of the trunk):

  C:\emacs\trunk>bzr pull
  Using saved parent location: bzr://bzr.savannah.gnu.org/emacs/trunk/
  No revisions to pull.
  Conflicting tags:
      mh-e-8.4
      mh-e-8.3.1


Second message: "load-path shadowings" (during a `make'):

  ...
  Pure-hashed: 22211 strings, 3141 vectors, 36433 conses, 2846
bytecodes, 85 others
  Dumping under the name emacs
  Dumping from c:/emacs/trunk/src/oo/i386/temacs.exe
            to c:/emacs/trunk/src/oo/i386/emacs.exe
  1841562 pure bytes used
  "./oo/i386/emacs.exe" -batch -f list-load-path-shadows
  This site has duplicate Lisp libraries with the same name.
  If a locally-installed Lisp library overrides a library in the Emacs release,
  that can cause trouble, and you should probably remove the locally-installed
  version unless you know what you are doing.

  c:/emacs/trunk/src/../lisp/terminal hides
c:/emacs/trunk/lisp/obsolete/terminal
  c:/emacs/trunk/src/../lisp/longlines hides
c:/emacs/trunk/lisp/obsolete/longlines

  2 Emacs Lisp load-path shadowings were found
  ...


Do you see them too?  Are they normal?

TIA.

-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Glenn Morris]

Dani Moncayo wrote:

>   Conflicting tags:
>       mh-e-8.4
>       mh-e-8.3.1

See the previous time this happened:
http://lists.gnu.org/archive/html/emacs-devel/2011-09/msg00490.html

Use something like

bzr tag --force \
  -r 'revid:wohler@newt.com-20121125191128-anxghxzwj2qy6ody' mh-e-8.4

>   c:/emacs/trunk/src/../lisp/terminal hides
> c:/emacs/trunk/lisp/obsolete/terminal
>   c:/emacs/trunk/src/../lisp/longlines hides
> c:/emacs/trunk/lisp/obsolete/longlines

I guess you did not remove the old .elc files; ie did not bootstrap.

Re: Two strange messages while building Emacs on MS-Windows

[Glenn Morris]

Glenn Morris wrote:

> I guess you did not remove the old .elc files; ie did not bootstrap.

PS if you are going to be providing pre-built binaries for general
consumption, I hope you will make them via bootstrap, else there could
be subtle problems from time to time.

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

>> I guess you did not remove the old .elc files; ie did not bootstrap.
>
> PS if you are going to be providing pre-built binaries for general
> consumption, I hope you will make them via bootstrap, else there could
> be subtle problems from time to time.

I see.  Thank you Glenn.

I was following Eli's advice for doing incremental builds
(http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12128#20), since
bootstrapping takes a lot more time.

But, why the makefiles don't take care all necessary recompilations?

Being forced to do a full bootstrap is too bad.  Is not there a better
solution for this?

-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

>>   Conflicting tags:
>>       mh-e-8.4
>>       mh-e-8.3.1
>
> See the previous time this happened:
> http://lists.gnu.org/archive/html/emacs-devel/2011-09/msg00490.html
>
> Use something like
>
> bzr tag --force \
>   -r 'revid:wohler@newt.com-20121125191128-anxghxzwj2qy6ody' mh-e-8.4

That errors out:
  bzr: ERROR: Unsupported protocol for url
"'revid:wohler@newt.com-20121125191128-anxghxzwj2qy6ody'"

But `bzr pull --overwrite' seems to have done the trick.  Now the
"conflicting tags" messages are gone.

Thanks.

>>   c:/emacs/trunk/src/../lisp/terminal hides
>> c:/emacs/trunk/lisp/obsolete/terminal
>>   c:/emacs/trunk/src/../lisp/longlines hides
>> c:/emacs/trunk/lisp/obsolete/longlines
>
> I guess you did not remove the old .elc files; ie did not bootstrap.

No I didn't remove anything, and didn't bootstrap either.

-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Stefan Monnier]

>>> c:/emacs/trunk/src/../lisp/terminal hides
>>> c:/emacs/trunk/lisp/obsolete/terminal
>>> c:/emacs/trunk/src/../lisp/longlines hides
>>> c:/emacs/trunk/lisp/obsolete/longlines
>> I guess you did not remove the old .elc files; ie did not bootstrap.
> No I didn't remove anything, and didn't bootstrap either.

FWIW, the non-w32 build automatically removes these left-over .elc files
(tho it removes them after checking for such shadowing, so you typically
still get those messages when files are moved, but only once).


        Stefan

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> Date: Fri, 7 Dec 2012 21:17:04 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: Emacs development discussions <emacs-devel@gnu.org>
> 
> I was following Eli's advice for doing incremental builds
> (http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12128#20), since
> bootstrapping takes a lot more time.

That advice wasn't meant to be used for producing distributions, only
for your own use.

> But, why the makefiles don't take care all necessary recompilations?

Because we don't have a good way of tracking dependencies between Lisp
files.

> Being forced to do a full bootstrap is too bad.

Why is it too bad for a distribution you do once in a while?  Let it
run in a separate directory, and you can meanwhile do something else.

How much time does it take you to bootstrap, btw?  And on what
machine?

> Is not there a better solution for this?

No.  Which is why admin/make-tarball.txt, the file documenting the
procedure to produce a release tarball, says:

  2.  Bootstrap to make 100% sure all elc files are up-to-date, and to
      make sure that the later tagged version will bootstrap, should it be
      necessary to check it out.

Re: Two strange messages while building Emacs on MS-Windows

[Stefan Monnier]

> But, why the makefiles don't take care all necessary recompilations?

Several reasons:
- there is no tool that extracts dependencies between Elisp files, so we
  don't have the needed dependency data.
- if you try to make such a tool you'll find that we have a lot of
  (apparently) circular dependencies.
- unless the dependency management is very clever, it will tend to
  recompile everything whenever a change is made to one of the
  "core" files.  E.g.: the .elc files depend on bytecomp.elc.
  bytecomp.elc is generated from the "emacs" executable which required
  many elisp files like subr.el(c), mule.el(c), help.el(c), so any
  change to one of these files causes generation of a new "emacs"
  executable, which in turn would cause recompilation of bytecomp.elc
  and (pretty much) all other .elc files.

So the "naive right way" quickly degenerates to something very close to
"make bootstrap".  But in 99% of the cases, only the modified files
require recompilation, so the default is to be incorrect.
Hopefully, at some point, we'll be able to do better.


        Stefan

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

>> I was following Eli's advice for doing incremental builds
>> (http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12128#20), since
>> bootstrapping takes a lot more time.
>
> That advice wasn't meant to be used for producing distributions, only
> for your own use.

Ok, I didn't know it.

>> Being forced to do a full bootstrap is too bad.
>
> Why is it too bad for a distribution you do once in a while?  Let it
> run in a separate directory, and you can meanwhile do something else.
>
> How much time does it take you to bootstrap, btw?  And on what
> machine?

I'm not sure.  About 30 min, I think.  I'll time it next time.

I said it's "too bad" because I'm used to incremental compilations,
which take much less time.  But if there's no better alternative, I'll
have to do it.

>> Is not there a better solution for this?
>
> No.  Which is why admin/make-tarball.txt, the file documenting the
> procedure to produce a release tarball, says:
>
>   2.  Bootstrap to make 100% sure all elc files are up-to-date, and to
>       make sure that the later tagged version will bootstrap, should it be
>       necessary to check it out.

Ok, thanks for the info.

-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Stefan Monnier]

> I'm not sure.  About 30 min, I think.  I'll time it next time.

That's what it takes on my fit-pc2, but on my T61 laptop, it's about
7min (and on my E350 nettop it's around 20min).
So unless you're using underpowered machines like mine, there's probably
room for improvement.


        Stefan

Re: Two strange messages while building Emacs on MS-Windows

[joakim]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 7 Dec 2012 21:17:04 +0100
>> From: Dani Moncayo <dmoncayo@gmail.com>
>> Cc: Emacs development discussions <emacs-devel@gnu.org>
>> 
>> I was following Eli's advice for doing incremental builds
>> (http://debbugs.gnu.org/cgi/bugreport.cgi?bug=12128#20), since
>> bootstrapping takes a lot more time.
>
> That advice wasn't meant to be used for producing distributions, only
> for your own use.
>
>> But, why the makefiles don't take care all necessary recompilations?
>
> Because we don't have a good way of tracking dependencies between Lisp
> files.
>
>> Being forced to do a full bootstrap is too bad.
>
> Why is it too bad for a distribution you do once in a while?  Let it
> run in a separate directory, and you can meanwhile do something else.
>
> How much time does it take you to bootstrap, btw?  And on what
> machine?
>
>> Is not there a better solution for this?
>
> No.  Which is why admin/make-tarball.txt, the file documenting the
> procedure to produce a release tarball, says:
>
>   2.  Bootstrap to make 100% sure all elc files are up-to-date, and to
>       make sure that the later tagged version will bootstrap, should it be
>       necessary to check it out.

Is it possible to compile for windows on gnu/linux using wine or
something? It would be interesting to setup a windows build job on my
new build server. 

-- 
Joakim Verona

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> Date: Fri, 7 Dec 2012 23:11:01 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: rgm@gnu.org, emacs-devel@gnu.org
> 
> > How much time does it take you to bootstrap, btw?  And on what
> > machine?
> 
> I'm not sure.  About 30 min, I think.  I'll time it next time.

You didn't say what machine was that.  If it's anywhere near core i7,
you can slash these times by a factor of 3, if you use a recent CVS
build of GNU Make which supports job-server style parallel builds on
Windows.  Just type "make -j8 bootstrap" instead of just "make bootstrap".
("-j8" because the core i7 CPU has the equivalent of 6.5 processing
units, and the recommendation for Make is to use a number slightly
larger than that.)

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: joakim@verona.se
> Cc: Dani Moncayo <dmoncayo@gmail.com>,  emacs-devel@gnu.org
> Date: Fri, 07 Dec 2012 23:57:57 +0100
> 
> Is it possible to compile for windows on gnu/linux using wine or
> something?

Why don't you try?  I don't think anyone did, or at least we didn't
hear about that.

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

>> > How much time does it take you to bootstrap, btw?  And on what
>> > machine?
>>
>> I'm not sure.  About 30 min, I think.  I'll time it next time.
>
> You didn't say what machine was that.

Sorry.  The processor is an Intel Core i5-2410M @ 2.30GHz.

> If it's anywhere near core i7,
> you can slash these times by a factor of 3, if you use a recent CVS
> build of GNU Make which supports job-server style parallel builds on
> Windows.  Just type "make -j8 bootstrap" instead of just "make bootstrap".
> ("-j8" because the core i7 CPU has the equivalent of 6.5 processing
> units, and the recommendation for Make is to use a number slightly
> larger than that.)

I'll try a parallel build.  BTW, a plain "-j" option seem to be a good
choice;  according to the GNU Make manual it will dynamically
determine how many parallel recipes can be run:

  `-j [JOBS]'
  `--jobs[=JOBS]'
       Specifies the number of recipes (jobs) to run simultaneously.
       With no argument, `make' runs as many recipes simultaneously as
       possible.  If there is more than one `-j' option, the last one is
       effective.  *Note Parallel Execution: Parallel, for more
       information on how recipes are run.  Note that this option is
       ignored on MS-DOS.


-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> Date: Sat, 8 Dec 2012 09:43:22 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> I'll try a parallel build.  BTW, a plain "-j" option seem to be a good
> choice;  according to the GNU Make manual it will dynamically
> determine how many parallel recipes can be run:
> 
>   `-j [JOBS]'
>   `--jobs[=JOBS]'
>        Specifies the number of recipes (jobs) to run simultaneously.
>        With no argument, `make' runs as many recipes simultaneously as
>        possible.  If there is more than one `-j' option, the last one is
>        effective.  *Note Parallel Execution: Parallel, for more
>        information on how recipes are run.  Note that this option is
>        ignored on MS-DOS.

Read more.  In the referenced node, the manual says:

     If the `-j' option is followed by an integer, this is the number of
  recipes to execute at once; this is called the number of "job slots".
  If there is nothing looking like an integer after the `-j' option,
  there is no limit on the number of job slots.
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

That's it.  Nothing as sophisticated as you seem to have concluded.
On Windows, "no limit" means "-j 64", because that's the limitation of
the API used on Windows to watch child processes.

(The referenced node also describes the -l option, which could limit
the effective number of parallel jobs depending on the system load,
but since sensing the system load doesn't work on Windows, that option
is of no use for you.)

So I suggest to time different "-j N" bootstraps and find the one that
is the fastest.  Or just use the recipe I gave ;-)

Re: Two strange messages while building Emacs on MS-Windows

[joakim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: joakim@verona.se
>> Cc: Dani Moncayo <dmoncayo@gmail.com>,  emacs-devel@gnu.org
>> Date: Fri, 07 Dec 2012 23:57:57 +0100
>> 
>> Is it possible to compile for windows on gnu/linux using wine or
>> something?
>
> Why don't you try?  I don't think anyone did, or at least we didn't
> hear about that.

Okay, I'll have a go.

http://www.gnu.org/software/emacs/windows/Getting-Emacs.html

seems to indicate one uses Mingw on windows to compile Emacs.
Is that correct?

I am familiar with cross-compiling for windows on gnu/linux. Would that
work? 

-- 
Joakim Verona

Re: Two strange messages while building Emacs on MS-Windows

[Óscar Fuentes]

joakim@verona.se writes:

>> Why don't you try?  I don't think anyone did, or at least we didn't
>> hear about that.
>
> Okay, I'll have a go.
>
> http://www.gnu.org/software/emacs/windows/Getting-Emacs.html
>
> seems to indicate one uses Mingw on windows to compile Emacs.
> Is that correct?
>
> I am familiar with cross-compiling for windows on gnu/linux. Would that
> work? 

For cross-compiling, there are a few difficult passages:

 * Mixed build mode: you build some pieces with native tools (the
   cross-compiler, etc) but other pieces, like the .elc files, shall be
   built with the products (temacs/emacs executable) on the target
   environment, which implies running some temacs/emacs under wine.
   Dumping should also be done under wine.

 * Most likely, the W32 makefiles are far from supporting
   cross-compiling. They also would need to work on the
   GNU/Linux-MSWindows mixed environment and my impression is that they
   strongly depend on MSWindows features all the way.

 * The "regular" configure&make used elsewhere could be adapted by
   setting the appropriate knobs at configure-time and make-time
   (setting variables like CC, EMACS, etc.) The problem I've seen so far
   is that the makefiles works on the native file system, while wine
   works on its own file system (with different root, volume letters and
   all). So commands executed under wine would take the wrong pathnames.
   Adapting that seems tricky.

OTOH, a MS Windows Emacs install appears to work well enough under Wine.
I managed to lock both Emacs and the KDE desktop (!) with a simple
operation (had to switch to a text-mode console and kill wine from
there) but it seems that it should work for byte compiling and,
hopefully, dumping. This means that, starting from `wine cmd.exe',
installing the *native* mingw compiler (not the cross-compiler) and
building with the MS Windows procedure, maybe a full MS Windows build
can be achieved. Not something that I would advise for an official
release, though.

Re: Two strange messages while building Emacs on MS-Windows

[Óscar Fuentes]

Óscar Fuentes <ofv@wanadoo.es> writes:

> OTOH, a MS Windows Emacs install appears to work well enough under Wine.
> I managed to lock both Emacs and the KDE desktop (!) with a simple
> operation (had to switch to a text-mode console and kill wine from
> there) but it seems that it should work for byte compiling and,
> hopefully, dumping. This means that, starting from `wine cmd.exe',
> installing the *native* mingw compiler (not the cross-compiler) and
> building with the MS Windows procedure, maybe a full MS Windows build
> can be achieved. Not something that I would advise for an official
> release, though.

The nt/configure.bat script complains with errors like this:

D:\exp\emacs\emacs\nt>if ()==(cl) goto compilercheckdone 
Syntax error
File not found.


The script does not end, it seems to enter an infinite loop. Tried
executing cmd.exe with /E:ON or /X but made no difference.


This is what I'm using:

oscar@qcore:~/dev/exp/emacs$ wine --version
wine-1.4.1
oscar@qcore:~/dev/exp/emacs$ wine cmd.exe
CMD Version 1.4.1


What's the oldest OS version supported by the build scripts?

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: joakim@verona.se
> Date: Sat, 08 Dec 2012 11:58:42 +0100
> Cc: emacs-devel@gnu.org, dmoncayo@gmail.com
> 
> http://www.gnu.org/software/emacs/windows/Getting-Emacs.html
> 
> seems to indicate one uses Mingw on windows to compile Emacs.
> Is that correct?

Yes.  MSVC is also supported, but most people who build on Windows use
MinGW32 GCC.

> I am familiar with cross-compiling for windows on gnu/linux. Would that
> work? 

No.  The configure script and the Makefile.in files don't support the
Windows build yet.  See

  http://lists.gnu.org/archive/html/emacs-devel/2012-12/msg00203.html

for more details.

So I suggest to start with the build procedure described in
nt/INSTALL.  That would save you a lot of effort.

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 08 Dec 2012 13:12:37 +0100
> 
> The nt/configure.bat script complains with errors like this:
> 
> D:\exp\emacs\emacs\nt>if ()==(cl) goto compilercheckdone 
> Syntax error
> File not found.
> 
> 
> The script does not end, it seems to enter an infinite loop. Tried
> executing cmd.exe with /E:ON or /X but made no difference.

Your cmd.exe is dysfunctional.

> oscar@qcore:~/dev/exp/emacs$ wine --version
> wine-1.4.1
> oscar@qcore:~/dev/exp/emacs$ wine cmd.exe
> CMD Version 1.4.1
> 
> 
> What's the oldest OS version supported by the build scripts?

Windows NT4.

Re: Two strange messages while building Emacs on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> Your cmd.exe is dysfunctional.

That's what I was suspecting.

>> What's the oldest OS version supported by the build scripts?
>
> Windows NT4.

Thanks.

So Wine is neglecting cmd.exe. Using one from a Windows XP install
allows the script to complete. Then, buildobj.h causes a failure because
it contains too much " and '. Apparently make-buildobj-SH: is used
instead of make-buildobj-CMD. Fixing that by hand allows the process to
reach this point:

mingw32-make[1]: Entering directory `D:/exp/emacs/emacs/lib-src'
mmingw32-make[1]: Leaving directory `D:/exp/emacs/emacs/lib-src'
ingw32-make[1]: *** No rule to make target `../lisp/abbrev.elc', needed by `DOC'.  Stop.
mingw32-make: *** [all-other-dirs-gmake] Error 2


Why DOC depends on .elc files when there is no temacs.exe for building
them?

Then I blindly remove the DOC target from here and there and the build
reaches the point where temacs.bin is created:

gcc -o oo-spd/i386/temacs.bin  -gdwarf-2 -g3   -Ld:/exp/emacs/gnuwin32/lib -Wl,-stack,0x00800000 -Wl,-heap,0x00100000 -Wl,-image-base,0x01000000 -Wl,-subsystem,console -Wl,-entry,__start -Wl,-Map,oo-spd/i386/temacs.map oo-spd/i386/firstfile.o oo-spd/i386/emacs.res oo-spd/i386/temacs0.a oo-spd/i386/temacs1.a oo-spd/i386/temacs2.a oo-spd/i386/lastfile.a ../lib/oo-spd/i386/libgnu.a -lwinmm -ladvapi32 -lgdi32 -lcomdlg32 -luser32 -lmpr -lshell32 -lwinspool -lole32 -lcomctl32 -lusp10  
"./../nt/oo-spd/i386/addsection" "oo-spd/i386/temacs.bin" "oo-spd/i386/temacs.exe" EMHEAP 27
Dumping from oo-spd/i386/temacs.bin
          to oo-spd/i386/temacs.exe
"./oo-spd/i386/temacs.exe" -batch -l loadup dump
mingw32-make[1]: *** [oo-spd/i386/emacs.exe] Error -1
mingw32-make[1]: Leaving directory `D:/exp/emacs/emacs/src'
mingw32-make: *** [all-other-dirs-gmake] Error 2



Well, that's enough for now.

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 08 Dec 2012 14:24:23 +0100
> 
> So Wine is neglecting cmd.exe. Using one from a Windows XP install
> allows the script to complete. Then, buildobj.h causes a failure because
> it contains too much " and '. Apparently make-buildobj-SH: is used
> instead of make-buildobj-CMD.

Maybe the Windows makefiles assume something that is false in Wine.
The test for the shell type is in nt/gmake.defs:

  # Determine whether make is using sh or cmd/command as shell; cmd.exe
  # will output "ECHO is on" when echo is given by itself, while sh will
  # not produce any output.
  sh_output := $(shell echo)
  ifeq "$(findstring ECHO, $(sh_output))" "ECHO"
  THE_SHELL = $(COMSPEC)$(ComSpec)
  SHELLTYPE=CMD
  SWITCHCHAR=/
  else
  USING_SH = 1
  THE_SHELL = $(SHELL)
  SHELLTYPE=SH

Which part(s) of this do(es)n't work in Wine?

> mingw32-make[1]: Entering directory `D:/exp/emacs/emacs/lib-src'
> mmingw32-make[1]: Leaving directory `D:/exp/emacs/emacs/lib-src'
> ingw32-make[1]: *** No rule to make target `../lisp/abbrev.elc', needed by `DOC'.  Stop.
> mingw32-make: *** [all-other-dirs-gmake] Error 2

You need to say "make bootstrap" explicitly, the Windows build scripts
cannot automatically intuit that a bootstrap is needed.

> Then I blindly remove the DOC target from here and there and the build
> reaches the point where temacs.bin is created:
> 
> gcc -o oo-spd/i386/temacs.bin  -gdwarf-2 -g3   -Ld:/exp/emacs/gnuwin32/lib -Wl,-stack,0x00800000 -Wl,-heap,0x00100000 -Wl,-image-base,0x01000000 -Wl,-subsystem,console -Wl,-entry,__start -Wl,-Map,oo-spd/i386/temacs.map oo-spd/i386/firstfile.o oo-spd/i386/emacs.res oo-spd/i386/temacs0.a oo-spd/i386/temacs1.a oo-spd/i386/temacs2.a oo-spd/i386/lastfile.a ../lib/oo-spd/i386/libgnu.a -lwinmm -ladvapi32 -lgdi32 -lcomdlg32 -luser32 -lmpr -lshell32 -lwinspool -lole32 -lcomctl32 -lusp10  
> "./../nt/oo-spd/i386/addsection" "oo-spd/i386/temacs.bin" "oo-spd/i386/temacs.exe" EMHEAP 27
> Dumping from oo-spd/i386/temacs.bin
>           to oo-spd/i386/temacs.exe
> "./oo-spd/i386/temacs.exe" -batch -l loadup dump
> mingw32-make[1]: *** [oo-spd/i386/emacs.exe] Error -1

This means the above dump command failed.  So something is wrong with
the resulting binary.

Thanks.

Re: Two strange messages while building Emacs on MS-Windows

[Stefan Monnier]

> Sorry.  The processor is an Intel Core i5-2410M @ 2.30GHz.

I get about 6min with "make -j2 bootstrap" on a Thinkpad T61, which is
an older machine (T9300 core 2 duo at 2.5GHz), so 30min sounds very
slow, even for sequential compilation.

IIRC the Windows bootstrap code is not as well tuned, and byte-compiles
all the .el files with the "bootstrap-emacs" instead of only compiling
the preloaded files this way.  That might explain it (the
"bootstrap-emacs" is an emacs binary built with the non-compiled Elisp
files, so its Elisp functions are slower).


        Stefan

Re: Two strange messages while building Emacs on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Sat, 08 Dec 2012 14:24:23 +0100
>> 
>> So Wine is neglecting cmd.exe. Using one from a Windows XP install
>> allows the script to complete. Then, buildobj.h causes a failure because
>> it contains too much " and '. Apparently make-buildobj-SH: is used
>> instead of make-buildobj-CMD.
>
> Maybe the Windows makefiles assume something that is false in Wine.
> The test for the shell type is in nt/gmake.defs:
>
>   # Determine whether make is using sh or cmd/command as shell; cmd.exe
>   # will output "ECHO is on" when echo is given by itself, while sh will
>   # not produce any output.
>   sh_output := $(shell echo)
>   ifeq "$(findstring ECHO, $(sh_output))" "ECHO"
>   THE_SHELL = $(COMSPEC)$(ComSpec)
>   SHELLTYPE=CMD
>   SWITCHCHAR=/
>   else
>   USING_SH = 1
>   THE_SHELL = $(SHELL)
>   SHELLTYPE=SH
>
> Which part(s) of this do(es)n't work in Wine?

cmd.exe from Wine outputs

Echo is off

(note the capitalization), while the one taken from the Spanish Windows
XP outputs

ECHO está activado

I don't know why the test fails with that cmd.exe, it was copied from
the virtual machine where I build Emacs. Possibly my mistake, because at
some point I realized that COMSPEC was not set and restarted the whole
process, but maybe some generated makefile was not deleted.

BTW, Wine's cmd.exe seems unable to get the quoting right while
generating buildobj.h. I tried multiple combinations, but haven't found
a method for outputting one double-quote to the file.

>> mingw32-make[1]: Entering directory `D:/exp/emacs/emacs/lib-src'
>> mmingw32-make[1]: Leaving directory `D:/exp/emacs/emacs/lib-src'
>> ingw32-make[1]: *** No rule to make target `../lisp/abbrev.elc', needed by `DOC'.  Stop.
>> mingw32-make: *** [all-other-dirs-gmake] Error 2
>
> You need to say "make bootstrap" explicitly, the Windows build scripts
> cannot automatically intuit that a bootstrap is needed.

Yes, I started with `make bootstrap' but, after fixing buildobj.h,
just executed `make'.

[snip]

After undoing the workarounds for the DOC target mentioned on my
previous message and commenting out the test for the shell type quoted
at the beginning of this one, hence forcing all makefiles to use the
copied cmd.exe, the build continued all along, except for about 30 .el
files which failed to compile crashing emacs but didn't stop the build.
Then `make install' failed with

mingw32-make[2]: Entering directory `D:/exp/emacs/emacs/lisp'
cp -f *.el "d:\exp\emacs\installed/lisp"
fixme:process:CmdBatNotification 1
cp: cannot stat `*.el': Invalid argument
fixme:process:CmdBatNotification 0
mingw32-make[2]: *** [install-lisp-CMD] Error 1
mingw32-make[2]: Leaving directory `D:/exp/emacs/emacs/lisp'

The `fixme' comes from Wine, there are a lot of them about
CmdBatNotification. The second most popular is

fixme:msvcrt:MSVCRT__sopen_s : pmode 0x81b6 ignored

At this point I gave out, as Wine seems too unreliable for the task.
Also I had the impression that the Wine build is slower than the
virtualized Windows XP build running on the same physical machine.

Re: Two strange messages while building Emacs on MS-Windows

[Jason Rumney]

Dani Moncayo <dmoncayo@gmail.com> writes:

> But, why the makefiles don't take care all necessary recompilations?

They do, but not neccesarily in the correct order if there are
interdependencies between changed files.  Also, as in this case, a
recompile will not remove the .elc files for .el files that have moved
in the bzr tree.

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sat, 08 Dec 2012 18:06:14 -0500
> 
> IIRC the Windows bootstrap code is not as well tuned, and byte-compiles
> all the .el files with the "bootstrap-emacs" instead of only compiling
> the preloaded files this way.  That might explain it (the
> "bootstrap-emacs" is an emacs binary built with the non-compiled Elisp
> files, so its Elisp functions are slower).

The more important reason is that byte compilation on Windows during
bootstrap is done sequentially, whereas on Unix it's done in parallel
(under "make -j").

Re: Two strange messages while building Emacs on MS-Windows

[Stefan Monnier]

> The more important reason is that byte compilation on Windows during
> bootstrap is done sequentially, whereas on Unix it's done in parallel
> (under "make -j").

I do "make -j2" and get about %170 of cpu utilisation (most of the
less-than-100% is due to phases in the build process where we only have
one process running, such as when running `configure'), so presumably my
6min would turn into no more than 10min if run sequentially.
Still pretty far from his 30min, so while "make -j" would help, the
inefficiency of the bootstrap is at least as important in his specific
case, I think.


        Stefan

Re: Two strange messages while building Emacs on MS-Windows

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I do "make -j2" and get about %170 of cpu utilisation (most of the
> less-than-100% is due to phases in the build process where we only have
> one process running, such as when running `configure'), so presumably my
> 6min would turn into no more than 10min if run sequentially.
> Still pretty far from his 30min, so while "make -j" would help, the
> inefficiency of the bootstrap is at least as important in his specific
> case, I think.

Starting an Emacs process for byte-compiling one .el file is *much*
slower on MSWindows than on GNU/Linux. In general, starting a process is
much slower on Windows than *nix, but I don't know if there are other
factors involved.

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 09 Dec 2012 19:15:13 +0100
> 
> Starting an Emacs process for byte-compiling one .el file is *much*
> slower on MSWindows than on GNU/Linux. In general, starting a process is
> much slower on Windows than *nix

Do you have any numbers to put "slow" and "fast" into perspective?

Re: Two strange messages while building Emacs on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Starting an Emacs process for byte-compiling one .el file is *much*
>> slower on MSWindows than on GNU/Linux. In general, starting a process is
>> much slower on Windows than *nix
>
> Do you have any numbers to put "slow" and "fast" into perspective?

Good question. This is on the same machine, average of five runs
after warming the cache with one run:

time emacs --batch --eval '(message "Hello")'

Kubuntu 12.10 x86_64  : 0.026 s
Windows XP SP3 32 bits: 0.240 s

Bootstrapping compiles ~1400 el files. The difference on process startup
accounts for >5 minutes. And that's only for byte-compiling. Add
invocations to gcc, make, cmd, fileutils, etc.

Still, most of the difference on bootstrap time is unexplained. Maybe
it's the filesystem, maybe Emacs is super-optimized for GNU/Linux and
not so much for MSWindows...

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dmoncayo@gmail.com,  emacs-devel@gnu.org
> Date: Sun, 09 Dec 2012 10:50:31 -0500
> 
> > The more important reason is that byte compilation on Windows during
> > bootstrap is done sequentially, whereas on Unix it's done in parallel
> > (under "make -j").
> 
> I do "make -j2" and get about %170 of cpu utilisation (most of the
> less-than-100% is due to phases in the build process where we only have
> one process running, such as when running `configure'), so presumably my
> 6min would turn into no more than 10min if run sequentially.

Indeed, I get 10min 40sec for sequential compilation with Emacs whose
byte compiler was byte-compiled.  This is on a 3.4GHz Core i7, which
is probably faster than your cores.

With the changes I committed in revision 111166, lisp/makefile.w32-in
now allows for 4-way parallelism while byte-compiling, which slashes
the byte compilation times from 10:40 to 3:35 if I use "make -jN" with
N >= 4.

The above is with byte-compiled byte compiler.  If I do the same with
the interpreted byte compiler, the sequential compilation takes 12:30
while the parallel compilation takes 4:40 -- slower, but not by a
large margin, only 25% slowdown.  A full bootstrap with "make -j8"
takes about 6 minutes.

> Still pretty far from his 30min, so while "make -j" would help, the
> inefficiency of the bootstrap is at least as important in his specific
> case, I think.

I cannot explain 30 min, since I never get such long times here.  One
possible reason could be that Dani uses an MSYS Make, which doesn't
support parallel execution (the MinGW Make does in its latest
builds).  Another potential difference between what I do and what Dani
does is that Dani uses MSYS Bash, while I use cmd.exe.  But I can
hardly believe that MSYS Bash slows down things so much.

Yet another possible reason could be that Dani runs a 64-bit Windows 7
while I did the above measurements on a 32-bit XPSP3.  Recently I
found out that running 32-bit executables on 64-bit Windows OS incurs
a tangible penalty.  But my own measurements of byte compilation on
Windows 7 show only 2-fold slowdown, which still brings us only to
around 20 min or so, not 30.  A mystery.

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 09 Dec 2012 20:31:51 +0100
> 
> > Do you have any numbers to put "slow" and "fast" into perspective?
> 
> Good question. This is on the same machine, average of five runs
> after warming the cache with one run:
> 
> time emacs --batch --eval '(message "Hello")'
> 
> Kubuntu 12.10 x86_64  : 0.026 s
> Windows XP SP3 32 bits: 0.240 s

On what hardware?

I get 0.080s on my Core i7 running 32-bit XP SP3.

Re: Two strange messages while building Emacs on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> time emacs --batch --eval '(message "Hello")'
>> 
>> Kubuntu 12.10 x86_64  : 0.026 s
>> Windows XP SP3 32 bits: 0.240 s
>
> On what hardware?
>
> I get 0.080s on my Core i7 running 32-bit XP SP3.

Intel Q6600 2.4GHz.

Please note that GNU/Linux on my old machine still is 3x faster than
Windows XP on your newer one.

(Why your i7 is 4x faster than my Q6600? No idea. There are no running
antivirus and Emacs was built with default settings.)

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

>> Still pretty far from his 30min, so while "make -j" would help, the
>> inefficiency of the bootstrap is at least as important in his specific
>> case, I think.
>
> I cannot explain 30 min, since I never get such long times here.  One
> possible reason could be that Dani uses an MSYS Make, which doesn't
> support parallel execution (the MinGW Make does in its latest
> builds).

I'm using "mingw32-make" (C:\MinGW\bin\mingw32-make.exe), not "MSYS
Make" (C:\MinGW\msys\1.0\bin\make.exe).

It seems to me that the current version of mingw32-make (GNU Make
3.82.90) does support parallel execution, because I sometimes see
mixed messages (from different processes) in the terminal when doing a
bootstrap with "-jN".

>  Another potential difference between what I do and what Dani
> does is that Dani uses MSYS Bash, while I use cmd.exe.  But I can
> hardly believe that MSYS Bash slows down things so much.

I've just bootstrapped (trunk revno 111165) using cmd.exe as shell,
and it took 30 min. and 29sec.  I.e., the shell doesn't seem to be
relevant here.

> Yet another possible reason could be that Dani runs a 64-bit Windows 7
> while I did the above measurements on a 32-bit XPSP3.  Recently I
> found out that running 32-bit executables on 64-bit Windows OS incurs
> a tangible penalty.  But my own measurements of byte compilation on
> Windows 7 show only 2-fold slowdown, which still brings us only to
> around 20 min or so, not 30.  A mystery.

Yes, my current OS is Windows 7 64-bit.

Next time I'll try your changes to `lisp/makefile.w32-in', and see if
they are effective here.

Thanks.

-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 09 Dec 2012 23:44:45 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> time emacs --batch --eval '(message "Hello")'
> >> 
> >> Kubuntu 12.10 x86_64  : 0.026 s
> >> Windows XP SP3 32 bits: 0.240 s
> >
> > On what hardware?
> >
> > I get 0.080s on my Core i7 running 32-bit XP SP3.
> 
> Intel Q6600 2.4GHz.
> 
> Please note that GNU/Linux on my old machine still is 3x faster than
> Windows XP on your newer one.

Noted.

> (Why your i7 is 4x faster than my Q6600? No idea. There are no running
> antivirus and Emacs was built with default settings.)

How did you time things on Windows?

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> Date: Mon, 10 Dec 2012 00:31:10 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> >  Another potential difference between what I do and what Dani
> > does is that Dani uses MSYS Bash, while I use cmd.exe.  But I can
> > hardly believe that MSYS Bash slows down things so much.
> 
> I've just bootstrapped (trunk revno 111165) using cmd.exe as shell,
> and it took 30 min. and 29sec.  I.e., the shell doesn't seem to be
> relevant here.
> 
> > Yet another possible reason could be that Dani runs a 64-bit Windows 7
> > while I did the above measurements on a 32-bit XPSP3.  Recently I
> > found out that running 32-bit executables on 64-bit Windows OS incurs
> > a tangible penalty.  But my own measurements of byte compilation on
> > Windows 7 show only 2-fold slowdown, which still brings us only to
> > around 20 min or so, not 30.  A mystery.
> 
> Yes, my current OS is Windows 7 64-bit.
> 
> Next time I'll try your changes to `lisp/makefile.w32-in', and see if
> they are effective here.

The mystery still stands.

I suggest to separate variables here, and time the byte compilation
alone.  Try this:

  cd lisp
  find . -name "*.elc" -delete
  make compile

and time the last command.

With the current trunk, you can do "make -j6 compile" instead of the
last command, and it should be about 3 to 4 times faster.

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 09 Dec 2012 10:50:31 -0500
> Cc: emacs-devel@gnu.org, dmoncayo@gmail.com
> 
> I do "make -j2" and get about %170 of cpu utilisation (most of the
> less-than-100% is due to phases in the build process where we only have
> one process running, such as when running `configure'), so presumably my
> 6min would turn into no more than 10min if run sequentially.

FWIW, "make -j2 bootstrap" was timed as follows:

  real    17m2.832s
  user    23m4.760s
  sys     4m45.010s

on this machine:

  Linux fencepost.gnu.org 2.6.32-42-server #1trisquel3 SMP Sat Aug 11 18:02:12 UTC 2012 x86_64 GNU/Linux

whose /proc/cpuinfo says, inter alia:

  vendor_id       : AuthenticAMD
  cpu family      : 16
  model           : 9
  model name      : AMD Opteron(tm) Processor 6128
  stepping        : 1
  cpu MHz         : 2000.078
  cache size      : 512 KB

This machine was 98% idle when I timed the bootstrap.

This command:

   time emacs --batch --eval '(message "Hello")'

(with an unoptimized build of Emacs) takes between 0.085s and 0.123s
on this machine.  (Why this is 5 times slower than Óscar's slower
machine is also unclear.)

The command "cd lisp && make compile-always", which sequentially
compiles every Lisp file, was timed on this machine as follows:

  real    23m11.277s
  user    16m50.820s
  sys     2m45.910s

The same command with "make -j2" times like this:

  real    11m51.381s
  user    17m15.850s
  sys     2m49.410s

which is an entirely reasonable 2-fold speedup in the elapsed time
(and seems to reveal some bug in accounting for user- and
system-times).

According to http://www.cpubenchmark.net, the CPU marks of the
machines mentioned in this thread are, top to bottom:

  Intel Core i7-2600      8,628
  AMD Opteron 6128        4,999
  Intel Core i5-2410M     3,186
  Intel Core2 Quad Q6600  2,941
  Intel Core 2 duo T9300  1,709

So I have no idea how Stefan gets a 6-min bootstrap on his 2.5-GHz
T9300 Core 2 duo...  Probably missing some factor here.  I thought
maybe the filesystem used on fencepost could be it, but "df -hT ."
there says:

  Filesystem    Type    Size  Used Avail Use% Mounted on
  /dev/xvdb1    ext4    493G  378G  115G  77% /srv/data

and my, perhaps incorrect, reading of that is that this is a local
ext4 filesystem.

Re: Two strange messages while building Emacs on MS-Windows

[Romain Francoise]

Eli Zaretskii <eliz@gnu.org> writes:

> This machine was 98% idle when I timed the bootstrap.

fencepost is a Xen virtual machine so even if it's idle, the hardware
itself may not be.

> According to http://www.cpubenchmark.net, the CPU marks of the
> machines mentioned in this thread are, top to bottom:

>   Intel Core i7-2600      8,628
>   AMD Opteron 6128        4,999
>   Intel Core i5-2410M     3,186
>   Intel Core2 Quad Q6600  2,941
>   Intel Core 2 duo T9300  1,709

FWIW, on my Core i7-2600 (4 cores, 8 threads), `make -j16 bootstrap'
runs in about 1m30s:

  403.12s user 20.83s system 463% cpu 1:31.37 total

Fast enough to always do a clean build from a pristine source tree and be
spared strange bootstrap errors...

Re: Two strange messages while building Emacs on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> How did you time things on Windows?

From a MSYS bash shell:

time <command>


Your reports motivated me to do more research. Booted again the machine
with the XP partition with the intention of timing with other methods.
The first try was with MSYS again, just a sanity check, and this is the
result for the second and successive runs:

time /c/apps/emacs/bin/emacs --batch --eval '(message "Hello")'

0.047 s

I can only speculate about what happened. That XP partition was used by
the first time on about two years for the timings I posted the first
time you asked for data. This last time I booted XP it was idle for
several minutes before I ran the tests. Maybe the first time Windows
Update or other background programs were doing their chores, although
Task Manager didn't show significant activity.

The timings I originally posted seemed right because they were about 10%
better than the ones I get on the virtualized XP machine I use for
building Emacs. Now I'm surprised to see how big the performance penalty
virtualization imposes on some tasks. (Now I recall that long time ago I
did quite a bit of experimentation and observed that the overhead is
less if you assign just one CPU to the virtual machine.)

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 10 Dec 2012 12:11:22 +0100
> 
> time /c/apps/emacs/bin/emacs --batch --eval '(message "Hello")'
> 
> 0.047 s

So what does this say about the efficiency of starting programs on
Windows as compared to GNU/Linux?

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Romain Francoise <romain@orebokech.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  dmoncayo@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 10 Dec 2012 11:37:37 +0100
> 
> FWIW, on my Core i7-2600 (4 cores, 8 threads), `make -j16 bootstrap'
> runs in about 1m30s:
> 
>   403.12s user 20.83s system 463% cpu 1:31.37 total

What about "make -j2"?  That was the basis for comparison, so going to
higher parallelization gets in the way of comparison.

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> Date: Mon, 10 Dec 2012 11:59:14 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: dmoncayo@gmail.com, emacs-devel@gnu.org
> 
> The command "cd lisp && make compile-always", which sequentially
> compiles every Lisp file, was timed on this machine as follows:
> 
>   real    23m11.277s
>   user    16m50.820s
>   sys     2m45.910s
> 
> The same command with "make -j2" times like this:
> 
>   real    11m51.381s
>   user    17m15.850s
>   sys     2m49.410s
> 
> which is an entirely reasonable 2-fold speedup in the elapsed time
> (and seems to reveal some bug in accounting for user- and
> system-times).

Actually, there's no bug: user and system times are sums of all the
cores that participated in the job, so for -j2 these numbers are
roughly doubled.

Re: Two strange messages while building Emacs on MS-Windows

[Romain Francoise]

Eli Zaretskii <eliz@gnu.org> writes:

> What about "make -j2"?  That was the basis for comparison, so going to
> higher parallelization gets in the way of comparison.

make -j2 bootstrap:

 246.51s user 15.08s system 124% cpu 3:30.54 total

Re: Two strange messages while building Emacs on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> time /c/apps/emacs/bin/emacs --batch --eval '(message "Hello")'
>> 
>> 0.047 s
>
> So what does this say about the efficiency of starting programs on
> Windows as compared to GNU/Linux?

It says that MS Windows is ~80% slower than GNU/Linux at running emacs
with for evaling a simple expression :-)

Serioulsy, though, Emacs does quite a few things at startup. My day to
day experience is that GNU/Linux efficiency at starting processes make a
difference when running lots of instances of short-lived programs, as,
for instance, a compiler test suite. Years ago, I worked on MS-Windows
native and on a virtualized Debian install: running the test suite for
the compiler I was working on required more than twice the time on
MS-Windows than on the Debian virtual machine.

As for emacs build times, the startup time is small enough to not make
it all that relevant while running >1000 times emacs.exe for compiling
all the .el files.

Still, I don't understand how your i7 is 90% slower than my old Q6600,
running the same OS. IIRC you mentioned that your build is unoptimized.
Mine is with default configuration, which defaults to -O2.

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Romain Francoise <romain@orebokech.com>
> Cc: monnier@iro.umontreal.ca,  dmoncayo@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 10 Dec 2012 16:33:16 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What about "make -j2"?  That was the basis for comparison, so going to
> > higher parallelization gets in the way of comparison.
> 
> make -j2 bootstrap:
> 
>  246.51s user 15.08s system 124% cpu 3:30.54 total

Thanks.

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 10 Dec 2012 16:42:15 +0100
> 
> It says that MS Windows is ~80% slower than GNU/Linux at running emacs
> with for evaling a simple expression :-)

That's more reasonable, and probably even expected, see below.

> Serioulsy, though, Emacs does quite a few things at startup.

Indeed.  And on Windows it does more.  What's more, quite a few of
those things involves calling 'stat', which on Windows is much slower.

> Still, I don't understand how your i7 is 90% slower than my old Q6600,
> running the same OS. IIRC you mentioned that your build is unoptimized.
> Mine is with default configuration, which defaults to -O2.

A more important difference is that I do have an anti-virus installed
(although the few tens of milliseconds that takes, if that's the
reason, I'm gladly willing to pay).  And my system is heavily used for
years (the motherboard is new, but the OS installation is 7-year old),
so it has quite a lot of baggage in its Registry etc.

On top of that, I didn't use MSYS to time the commands.  I have a
clone of the 'time' command (called 'timep', because 'time' is taken
by an internal cmd command that does something very different) which
uses Windows APIs to report times used by its child processes.  This
alone can explain the difference.

Anyway, a 4 to 6 min bootstrap is something I can easily live with,
especially since I almost never do it.

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

> The mystery still stands.
>
> I suggest to separate variables here, and time the byte compilation
> alone.  Try this:
>
>   cd lisp
>   find . -name "*.elc" -delete
>   make compile
>
> and time the last command.
>
> With the current trunk, you can do "make -j6 compile" instead of the
> last command, and it should be about 3 to 4 times faster.

"mingw32-make [-j4] compile" takes:
  - With revno 111165 (before your changes in `lisp/makefile.w32-in'):
    . without "-jN": 15:57
    . with -j4: 15:34
  - With revno 111175:
    . without "-jN": 16:21
    . with -j4: 6:23


A full bootstrap with "-j4" (of the last trunk) has taken 13:53.

So it seems that I've got an speedup of 50% approximately.  (Not bad!)

And for the record, from a MSYS bash:

  $ time /c/emacs/trunk/bin/emacs --batch --eval '(message "Hello")'
  Hello

  real    0m1.189s
  user    0m0.031s
  sys     0m0.000s



-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> Date: Mon, 10 Dec 2012 21:57:32 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> "mingw32-make [-j4] compile" takes:
>   - With revno 111165 (before your changes in `lisp/makefile.w32-in'):
>     . without "-jN": 15:57
>     . with -j4: 15:34
>   - With revno 111175:
>     . without "-jN": 16:21
>     . with -j4: 6:23
> 
> 
> A full bootstrap with "-j4" (of the last trunk) has taken 13:53.

So now the new mystery is: why does it take you a whopping 7.5 min to
compile the C sources of Emacs?  It takes about 30 sec for me with GCC
3.4.x; GCC 4.7.2 that you use should be about 2 times slower, so I'd
expect something like 1 min, not 7.5.

Try this:

  cd nt
  touch ../src/config.h
  make -j6

and time the last command.

> So it seems that I've got an speedup of 50% approximately.  (Not bad!)

You should be able to do better, like 8min at most.  Something is
still getting in your way.

> And for the record, from a MSYS bash:
> 
>   $ time /c/emacs/trunk/bin/emacs --batch --eval '(message "Hello")'
>   Hello
> 
>   real    0m1.189s
>   user    0m0.031s
>   sys     0m0.000s

Maybe that's your problem.  Does the time go down if you repeat that
command?  If it doesn't, you may wish to try from cmd.exe.

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

> So now the new mystery is: why does it take you a whopping 7.5 min to
> compile the C sources of Emacs?  It takes about 30 sec for me with GCC
> 3.4.x; GCC 4.7.2 that you use should be about 2 times slower, so I'd
> expect something like 1 min, not 7.5.
>
> Try this:
>
>   cd nt
>   touch ../src/config.h
>   make -j6
>
> and time the last command.

That errors out here with the current trunk:
  mingw32-make[1]: *** No rule to make target
'../lisp/term/tty-colors.elc', needed by 'DOC'.  Stop.
  mingw32-make[1]: *** Waiting for unfinished jobs....
  mingw32-make[1]: Leaving directory 'C:/emacs/trunk/lib-src'
  makefile:453: recipe for target 'all-other-dirs-gmake' failed
  mingw32-make: *** [all-other-dirs-gmake] Error 2


>> So it seems that I've got an speedup of 50% approximately.  (Not bad!)
>
> You should be able to do better, like 8min at most.  Something is
> still getting in your way.
>
>> And for the record, from a MSYS bash:
>>
>>   $ time /c/emacs/trunk/bin/emacs --batch --eval '(message "Hello")'
>>   Hello
>>
>>   real    0m1.189s
>>   user    0m0.031s
>>   sys     0m0.000s
>
> Maybe that's your problem.  Does the time go down if you repeat that
> command?  If it doesn't, you may wish to try from cmd.exe.



-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> Date: Mon, 10 Dec 2012 22:33:46 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> >   cd nt
> >   touch ../src/config.h
> >   make -j6
> >
> > and time the last command.
> 
> That errors out here with the current trunk:
>   mingw32-make[1]: *** No rule to make target
> '../lisp/term/tty-colors.elc', needed by 'DOC'.  Stop.
>   mingw32-make[1]: *** Waiting for unfinished jobs....
>   mingw32-make[1]: Leaving directory 'C:/emacs/trunk/lib-src'
>   makefile:453: recipe for target 'all-other-dirs-gmake' failed
>   mingw32-make: *** [all-other-dirs-gmake] Error 2

Why don't you have tty-colors.elc?  It should be produced by the
bootstrap or by "make bzr-update", whichever you ran last.

What is going on in your builds?

Anyway, just compile that file by hand and try again.

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

>> >   cd nt
>> >   touch ../src/config.h
>> >   make -j6
>> >
>> > and time the last command.
>>
>> That errors out here with the current trunk:
>>   mingw32-make[1]: *** No rule to make target
>> '../lisp/term/tty-colors.elc', needed by 'DOC'.  Stop.
>>   mingw32-make[1]: *** Waiting for unfinished jobs....
>>   mingw32-make[1]: Leaving directory 'C:/emacs/trunk/lib-src'
>>   makefile:453: recipe for target 'all-other-dirs-gmake' failed
>>   mingw32-make: *** [all-other-dirs-gmake] Error 2
>
> Why don't you have tty-colors.elc?  It should be produced by the
> bootstrap or by "make bzr-update", whichever you ran last.
>
> What is going on in your builds?
>
> Anyway, just compile that file by hand and try again.

I don't know Eli, but I tried even a distclean + configure + bootstrap
and the bootstrap ended up failing with the same error above.  I'll
try that when I have time.  Sorry for the trouble.


-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> Date: Tue, 11 Dec 2012 08:18:40 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> >> >   cd nt
> >> >   touch ../src/config.h
> >> >   make -j6
> >> >
> >> > and time the last command.
> >>
> >> That errors out here with the current trunk:
> >>   mingw32-make[1]: *** No rule to make target
> >> '../lisp/term/tty-colors.elc', needed by 'DOC'.  Stop.
> >>   mingw32-make[1]: *** Waiting for unfinished jobs....
> >>   mingw32-make[1]: Leaving directory 'C:/emacs/trunk/lib-src'
> >>   makefile:453: recipe for target 'all-other-dirs-gmake' failed
> >>   mingw32-make: *** [all-other-dirs-gmake] Error 2
> >
> > Why don't you have tty-colors.elc?  It should be produced by the
> > bootstrap or by "make bzr-update", whichever you ran last.
> >
> > What is going on in your builds?
> >
> > Anyway, just compile that file by hand and try again.
> 
> I don't know Eli, but I tried even a distclean + configure + bootstrap
> and the bootstrap ended up failing with the same error above.  I'll
> try that when I have time.  Sorry for the trouble.

Sorry, it was my bad.  There was a typo in lisp/makefile.w32-in, now
fixed, which would cause 'lisp/term' to be skipped, but only of a
Unixy shell was used.

Re: Two strange messages while building Emacs on MS-Windows

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 369 bytes --]

On 12/10/2012 12:57 PM, Dani Moncayo wrote:
> And for the record, from a MSYS bash:
> 
>   $ time /c/emacs/trunk/bin/emacs --batch --eval '(message "Hello")'
>   Hello
> 
>   real    0m1.189s

Can you try cp /c/emacs/trunk/bin/emacs.exe /c/emacs/trunk/bin/emacs2.exe &&
time /c/emacs/trunk/bin/emacs2.exe? Can you try running the time command a few
times?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

> Sorry, it was my bad.  There was a typo in lisp/makefile.w32-in, now
> fixed, which would cause 'lisp/term' to be skipped, but only of a
> Unixy shell was used.

Thanks, now I can bootstrap again.

It seems like nowadays I'm about the only person who build Emacs on
Windows using a shell different to the native cmd.exe.  OTOH, having
to support two completely different shells for Windows implies (for
you) an extra effort that perhaps could be saved for other (better)
tasks.

IOW: I'd agree to simplify the MS-Windows makefiles by removing the
SH-related stuff.

-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

> So now the new mystery is: why does it take you a whopping 7.5 min to
> compile the C sources of Emacs?  It takes about 30 sec for me with GCC
> 3.4.x; GCC 4.7.2 that you use should be about 2 times slower, so I'd
> expect something like 1 min, not 7.5.
>
> Try this:
>
>   cd nt
>   touch ../src/config.h
>   make -j6
>
> and time the last command.

0:51

>>   $ time /c/emacs/trunk/bin/emacs --batch --eval '(message "Hello")'
>>   Hello
>>
>>   real    0m1.189s
>>   user    0m0.031s
>>   sys     0m0.000s
>
> Maybe that's your problem.  Does the time go down if you repeat that
> command?

I've tried that now, and I observe that the "real" component varies
between 145 and 200 ms (without decreasing or increasing tendency;
it's kinda random).

I don't understand such big difference with the previous measure.

In any case, the total bootstrap time keeps about the same: about 15 min.

> If it doesn't, you may wish to try from cmd.exe.

I don't know what's the best way to measure execution time from a
cmd.exe shell.  I've written this script:
  @echo Start %time%
  @C:\emacs\trunk\bin\emacs --batch --eval '(message "Hello")'
  @echo End %time%

And I get this output when I run it:
  Start 21:33:04.25
  Loading macroexp.elc...
  End of file during parsing
  End 21:33:04.42

-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

>>   $ time /c/emacs/trunk/bin/emacs --batch --eval '(message "Hello")'
>>   Hello
>>
>>   real    0m1.189s
>
> Can you try cp /c/emacs/trunk/bin/emacs.exe /c/emacs/trunk/bin/emacs2.exe &&
> time /c/emacs/trunk/bin/emacs2.exe? Can you try running the time command a few
> times?

I guess you mean to execute
  time /c/emacs/trunk/bin/emacs2 --batch --eval '(message "Hello")'

The result is the same as invoking "emacs" instead of "emacs2", i.e.,
between 145 and 200 ms.

-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

On 12/11/2012 12:42 PM, Dani Moncayo wrote:
>>>   $ time /c/emacs/trunk/bin/emacs --batch --eval '(message "Hello")'
>>>   Hello
>>>
>>>   real    0m1.189s
>>
>> Can you try cp /c/emacs/trunk/bin/emacs.exe /c/emacs/trunk/bin/emacs2.exe &&
>> time /c/emacs/trunk/bin/emacs2.exe? Can you try running the time command a few
>> times?
> 
> I guess you mean to execute
>   time /c/emacs/trunk/bin/emacs2 --batch --eval '(message "Hello")'
> 
> The result is the same as invoking "emacs" instead of "emacs2", i.e.,
> between 145 and 200 ms.
> 

What do you mean? The emacs run above took 1189ms.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

>>>>   $ time /c/emacs/trunk/bin/emacs --batch --eval '(message "Hello")'
>>>>   Hello
>>>>
>>>>   real    0m1.189s
>>>
>>> Can you try cp /c/emacs/trunk/bin/emacs.exe /c/emacs/trunk/bin/emacs2.exe &&
>>> time /c/emacs/trunk/bin/emacs2.exe? Can you try running the time command a few
>>> times?
>>
>> I guess you mean to execute
>>   time /c/emacs/trunk/bin/emacs2 --batch --eval '(message "Hello")'
>>
>> The result is the same as invoking "emacs" instead of "emacs2", i.e.,
>> between 145 and 200 ms.
>>
>
> What do you mean? The emacs run above took 1189ms.

I only say what I see.  Apparently, yesterday it took indeed much
longer to start.  No idea why.



-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

On 12/11/2012 12:58 PM, Dani Moncayo wrote:
>>>>>   $ time /c/emacs/trunk/bin/emacs --batch --eval '(message "Hello")'
>>>>>   Hello
>>>>>
>>>>>   real    0m1.189s
>>>>
>>>> Can you try cp /c/emacs/trunk/bin/emacs.exe /c/emacs/trunk/bin/emacs2.exe &&
>>>> time /c/emacs/trunk/bin/emacs2.exe? Can you try running the time command a few
>>>> times?
>>>
>>> I guess you mean to execute
>>>   time /c/emacs/trunk/bin/emacs2 --batch --eval '(message "Hello")'
>>>
>>> The result is the same as invoking "emacs" instead of "emacs2", i.e.,
>>> between 145 and 200 ms.
>>>
>>
>> What do you mean? The emacs run above took 1189ms.
> 
> I only say what I see.  Apparently, yesterday it took indeed much
> longer to start.  No idea why.

Have you rebooted in the interim without rebuilding that emacs binary?



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> Date: Tue, 11 Dec 2012 21:41:44 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> IOW: I'd agree to simplify the MS-Windows makefiles by removing the
> SH-related stuff.

Thanks.  It will go one day, but we need to deprecate it first.

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

>> I only say what I see.  Apparently, yesterday it took indeed much
>> longer to start.  No idea why.
>
> Have you rebooted in the interim without rebuilding that emacs binary?

I have rebooted, but this emacs binary is different, since I've
bootstrapped about an hour ago.

-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

On 12/11/2012 1:01 PM, Dani Moncayo wrote:
>>> I only say what I see.  Apparently, yesterday it took indeed much
>>> longer to start.  No idea why.
>>
>> Have you rebooted in the interim without rebuilding that emacs binary?
> 
> I have rebooted, but this emacs binary is different, since I've
> bootstrapped about an hour ago.
> 

Hrm. It's not what I thought, then. With Cygwin, I'm seeing the OS sometimes
decide to re-read all Emacs code pages from disk instead of retrieving them from
cache, but it only when executing a freshly-dumped Emacs binary.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> Date: Tue, 11 Dec 2012 21:41:55 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > So now the new mystery is: why does it take you a whopping 7.5 min to
> > compile the C sources of Emacs?  It takes about 30 sec for me with GCC
> > 3.4.x; GCC 4.7.2 that you use should be about 2 times slower, so I'd
> > expect something like 1 min, not 7.5.
> >
> > Try this:
> >
> >   cd nt
> >   touch ../src/config.h
> >   make -j6
> >
> > and time the last command.
> 
> 0:51

As expected.  So, if compiling the C sources takes 1 min and compiling
the Lisp files takes 6.5 min, how come the bootstrap takes 15 min?
What else takes 7.5 min?

> >>   $ time /c/emacs/trunk/bin/emacs --batch --eval '(message "Hello")'
> >>   Hello
> >>
> >>   real    0m1.189s
> >>   user    0m0.031s
> >>   sys     0m0.000s
> >
> > Maybe that's your problem.  Does the time go down if you repeat that
> > command?
> 
> I've tried that now, and I observe that the "real" component varies
> between 145 and 200 ms (without decreasing or increasing tendency;
> it's kinda random).

Most Windows time measurement have inherent 15-msec granularity.

> I don't know what's the best way to measure execution time from a
> cmd.exe shell.

I attach a program that you could use, call it timep.exe.

/* timep: Emulate Unix `time' command. */
#include <stdio.h>
#include <windows.h>
#include <tchar.h>

LPTSTR SkipArg (const LPTSTR args)
{
   LPTSTR argp = args;

   while (*argp != _T (' '))
     ++argp;
   while (*argp == _T (' '))
     ++argp;
   return argp;
}

int _tmain (int argc, LPTSTR argv [])
{
   STARTUPINFO StartUp;
   PROCESS_INFORMATION ProcInfo;
   union { /* Structure required for file time arithmetic. */
      LONGLONG li;
      FILETIME ft;
   } CreateTime, ExitTime, ElapsedTime;
   FILETIME KernelTime, UserTime;
   SYSTEMTIME ElTiSys, KeTiSys, UsTiSys, StartTimeSys, ExitTimeSys;
   LPTSTR targv = SkipArg (GetCommandLine ());
   OSVERSIONINFO OSVer;
   BOOL IsNT;
   HANDLE hProc;
   DWORD status;

   OSVer.dwOSVersionInfoSize = sizeof(OSVERSIONINFO);
   GetVersionEx (&OSVer);
   IsNT = (OSVer.dwPlatformId == VER_PLATFORM_WIN32_NT);
   /* NT (all versions) returns VER_PLATFORM_WIN32_NT. */
   GetStartupInfo (&StartUp);
   GetSystemTime (&StartTimeSys);

   /* Execute the command line; wait for process to complete. */
   CreateProcess (NULL, targv, NULL, NULL, TRUE,
         NORMAL_PRIORITY_CLASS, NULL, NULL, &StartUp, &ProcInfo);

   /* Assure that we have all REQUIRED access to the process. */
   DuplicateHandle (GetCurrentProcess (), ProcInfo.hProcess,
         GetCurrentProcess (), &hProc,
         PROCESS_QUERY_INFORMATION | SYNCHRONIZE, FALSE, 0);
   WaitForSingleObject (hProc, INFINITE);
   GetExitCodeProcess (hProc, &status);
   GetSystemTime (&ExitTimeSys);

   if (status >= 0 && status < 256 && IsNT) { /* W NT. Elapsed, Kernel, & User times. */
      GetProcessTimes (hProc, &CreateTime.ft,
            &ExitTime.ft, &KernelTime, &UserTime);
      ElapsedTime.li = ExitTime.li - CreateTime.li;
      FileTimeToSystemTime (&ElapsedTime.ft, &ElTiSys);
      FileTimeToSystemTime (&KernelTime, &KeTiSys);
      FileTimeToSystemTime (&UserTime, &UsTiSys);
      _ftprintf (stderr, _T ("\nreal\t%02dh%02dm%02d.%03ds\n"),
            ElTiSys.wHour, ElTiSys.wMinute, ElTiSys.wSecond,
            ElTiSys.wMilliseconds);
      _ftprintf (stderr, _T ("user\t%02dh%02dm%02d.%03ds\n"),
            UsTiSys.wHour, UsTiSys.wMinute, UsTiSys.wSecond,
            UsTiSys.wMilliseconds);
      _ftprintf (stderr, _T ("sys\t%02dh%02dm%02d.%03ds\n"),
            KeTiSys.wHour, KeTiSys.wMinute, KeTiSys.wSecond,
            KeTiSys.wMilliseconds);
   } else {
      /* Windows 9x and CE. Elapsed time only. */
   }
   CloseHandle (ProcInfo.hThread); CloseHandle (ProcInfo.hProcess);
   CloseHandle (hProc);
   return 0;
}

Re: Two strange messages while building Emacs on MS-Windows

[Dani Moncayo]

>> > Try this:
>> >
>> >   cd nt
>> >   touch ../src/config.h
>> >   make -j6
>> >
>> > and time the last command.
>>
>> 0:51
>
> As expected.  So, if compiling the C sources takes 1 min and compiling
> the Lisp files takes 6.5 min, how come the bootstrap takes 15 min?
> What else takes 7.5 min?

I don't know.  It seems that a bootstrap is not equivalent to the sum
of the two commands we've measured.

>> I don't know what's the best way to measure execution time from a
>> cmd.exe shell.

Thanks.

-- 
Dani Moncayo

Re: Two strange messages while building Emacs on MS-Windows

[Stefan Monnier]

>> As expected.  So, if compiling the C sources takes 1 min and compiling
>> the Lisp files takes 6.5 min, how come the bootstrap takes 15 min?
>> What else takes 7.5 min?
> I don't know.  It seems that a bootstrap is not equivalent to the sum
> of the two commands we've measured.

Indeed, there are various other things like leim, info, and whatnot.
Still 7½min seems like a long time for it.

Could it be that when you compiled the files, you used an `emacs' that
was dumped from .elc files, whereas when you bootstrap, you compile the
files with an `emacs' that was dumped from .el files?


        Stefan

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> Date: Tue, 11 Dec 2012 22:58:33 +0100
> From: Dani Moncayo <dmoncayo@gmail.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > As expected.  So, if compiling the C sources takes 1 min and compiling
> > the Lisp files takes 6.5 min, how come the bootstrap takes 15 min?
> > What else takes 7.5 min?
> 
> I don't know.  It seems that a bootstrap is not equivalent to the sum
> of the two commands we've measured.

The only other thing is deleting files at the beginning of bootstrap.

I suggest to bootstrap with a clock in your hand, and make notes about
the times each part takes.  There are at least 5 min more to gain
here, so I think you should be motivated enough ;-)

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Tue, 11 Dec 2012 17:55:58 -0500
> 
> >> As expected.  So, if compiling the C sources takes 1 min and compiling
> >> the Lisp files takes 6.5 min, how come the bootstrap takes 15 min?
> >> What else takes 7.5 min?
> > I don't know.  It seems that a bootstrap is not equivalent to the sum
> > of the two commands we've measured.
> 
> Indeed, there are various other things like leim, info, and whatnot.

Leim takes much less than a minute, and info is not built as part of
the Windows bootstrap.

> Could it be that when you compiled the files, you used an `emacs' that
> was dumped from .elc files, whereas when you bootstrap, you compile the
> files with an `emacs' that was dumped from .el files?

Yes, but that can only explain a 25% increase in time.

Re: Two strange messages while building Emacs on MS-Windows

[Stefan Monnier]

>> Could it be that when you compiled the files, you used an `emacs' that
>> was dumped from .elc files, whereas when you bootstrap, you compile the
>> files with an `emacs' that was dumped from .el files?
> Yes, but that can only explain a 25% increase in time.

Where do you get that 25% from?


        Stefan

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: dmoncayo@gmail.com, emacs-devel@gnu.org
> Date: Wed, 12 Dec 2012 09:26:48 -0500
> 
> >> Could it be that when you compiled the files, you used an `emacs' that
> >> was dumped from .elc files, whereas when you bootstrap, you compile the
> >> files with an `emacs' that was dumped from .el files?
> > Yes, but that can only explain a 25% increase in time.
> 
> Where do you get that 25% from?

I reported that here:

  http://lists.gnu.org/archive/html/emacs-devel/2012-12/msg00247.html

But it now seems my measurements in this area were not entirely
accurate, to say the least.  Stay tuned.

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> Date: Wed, 12 Dec 2012 05:48:51 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > Date: Tue, 11 Dec 2012 22:58:33 +0100
> > From: Dani Moncayo <dmoncayo@gmail.com>
> > Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> > 
> > > As expected.  So, if compiling the C sources takes 1 min and compiling
> > > the Lisp files takes 6.5 min, how come the bootstrap takes 15 min?
> > > What else takes 7.5 min?
> > 
> > I don't know.  It seems that a bootstrap is not equivalent to the sum
> > of the two commands we've measured.
> 
> The only other thing is deleting files at the beginning of bootstrap.
> 
> I suggest to bootstrap with a clock in your hand, and make notes about
> the times each part takes.  There are at least 5 min more to gain
> here, so I think you should be motivated enough ;-)

This is a bit longish message, sorry.  The bottom line is that I think
I know where those 5 min come from, and that Dani should get used to
the 15-min mark for the time being.

First, on my Core i7 running XP, and with "make -j8", the detailed
time schedule of a full bootstrap are:

  1:10 min for the initial compilation of C sources
  0:30 min for generating loaddefs
  4:10 min for compiling Lisp files
  0:26 min for the final compilation of C sources
  6:46 min total

After the bootstrap, removing all the *.elc files and recompiling them
with emacs.exe, which includes a byte-compiled byte compiler, takes
about 2:30 to 3 min with "make -j8", which is 60% of the time it takes
using an interpreted byte compiler.  Not the 25% I somehow measured a
day or two ago.

I also did measurements on a Windows 7 machine.  This one has a Core 2
duo E7300 CPU, so it is slightly less powerful than what Dani has.
Here's the bootstrap schedule on that machine using "make -j2":

  1:45 min for the initial compilation of C sources
  0:50 min for generating loaddefs
 11:35 min for compiling Lisp files
  0:50 min for the final compilation of C sources
 16:00 min total

Removing *.elc files and recompiling all of them with byte-compiled
compiler and "make -j2" takes 6:00 minutes, which is again almost
twice faster.  IOW, the missing 5 min are the difference between
compiling Lisp files with an interpreted byte compiler and a
byte-compiled one.

So I guess the 15 min that Dani gets is what is possible on a 2-core
machine, and any further improvements are possible only if Emacs is
re-dumped with byte-compiled preloaded filed early on and then that
binary used to compile the rest of Lisp files, like we do on Unix.

Re: Two strange messages while building Emacs on MS-Windows

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Tue, 11 Dec 2012 17:55:58 -0500
> 
> Could it be that when you compiled the files, you used an `emacs' that
> was dumped from .elc files, whereas when you bootstrap, you compile the
> files with an `emacs' that was dumped from .el files?

That is how bootstrap works on Windows, yes.