bootstrap compiles C files twice

bootstrap compiles C files twice

[Glenn Morris]

make bootstrap compiles all thes src/*.c files twice over (at least it
does for me).

In the top-level Makefile, bootstrap-build does:

(cd src; $(MAKE) $(MFLAGS) bootstrap)

which compiles them all once, before compiling the lisp files. Then it does:

(cd src; $(MAKE) $(MFLAGS) mostlyclean)

which deletes temacs, *.o, and some other stuff. Then it does:

$(MAKE) $(MFLAGS) all

which requires temacs and so causes all the .o files to be rebuilt.
(Even if temacs is retained at this point, some time-stamping seems to
cause it to need a spurious rebuild, but that is probably a separate
issue.)

Can anyone explain why `mostlyclean' is called after compiling the
lisp files? Things seem to work fine (and obviously more quickly)
without doing this.


Thanks.

Re: bootstrap compiles C files twice

[Stefan Monnier]

> Can anyone explain why `mostlyclean' is called after compiling the
> lisp files? Things seem to work fine (and obviously more quickly)
> without doing this.

I do not have an answer for that.  I was also surprised when this was
done (when I originally worked on the bootstrap code to remove the .elc
files from the CVS repository, I didn't force regeneration of the .o
and temacs file and I cannot remember any bug report about it).

But recently someone mentioned the issue of cross-compiling and in this
context it makes sense: the bootstrap-emacs (and its temacs) should be
compiled for the host platform so we can generate the .elc files, and
then the final `emacs' (and its `temacs') need to be compiled for the
target platform.

Not that cross-compiling is a common occurrence (it's definitely not
supported out-of-the-box for example).


        Stefan "who rarely bootstraps anyway"

Re: bootstrap compiles C files twice

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

  > > Can anyone explain why `mostlyclean' is called after compiling the
  > > lisp files? Things seem to work fine (and obviously more quickly)
  > > without doing this.
  > 
  > I do not have an answer for that.  I was also surprised when this was
  > done (when I originally worked on the bootstrap code to remove the .elc
  > files from the CVS repository, I didn't force regeneration of the .o
  > and temacs file and I cannot remember any bug report about it).
  > 
  > But recently someone mentioned the issue of cross-compiling and in this
  > context it makes sense: the bootstrap-emacs (and its temacs) should be
  > compiled for the host platform so we can generate the .elc files, and
  > then the final `emacs' (and its `temacs') need to be compiled for the
  > target platform.

You need to rerun configure if bootstrap-emacs is built for the host and
then build emacs for the target, so removing the object files is
probably not related to that ... 


  > Not that cross-compiling is a common occurrence (it's definitely not
  > supported out-of-the-box for example).

Meaning that it does not work at all :-)

Re: bootstrap compiles C files twice

[Jason Rumney]

Glenn Morris wrote:
> Can anyone explain why `mostlyclean' is called after compiling the
> lisp files? Things seem to work fine (and obviously more quickly)
> without doing this.
>   

I think the size of purespace differs for bootstrap builds, at least on 
some platforms. Although it only affects some .o files, so it shouldn't 
be necessary to remove all of them.

Re: bootstrap compiles C files twice

[Eli Zaretskii]

> Date: Tue, 03 Jun 2008 08:55:12 +0100
> From: Jason Rumney <jasonr@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Glenn Morris wrote:
> > Can anyone explain why `mostlyclean' is called after compiling the
> > lisp files? Things seem to work fine (and obviously more quickly)
> > without doing this.
> >   
> 
> I think the size of purespace differs for bootstrap builds, at least on 
> some platforms.

It used to be on all platforms, but I see that the compiler switch to
set PURESIZE to a very large value was removed on Posix systems.  I
guess they don't care about pure space overflow in bootstrap-emacs?
If that's true, then we do need to rebuild with correct pure size once
the *.el files are compiled and bootstrap-emacs can be tossed.

> Although it only affects some .o files, so it shouldn't be necessary
> to remove all of them.

Yes, but do we really want to maintain a list of those who do depend
on the pure size?  The gains are minimal: compiling C files takes a
small fraction of the total bootstrap time, especially under -j.

Re: bootstrap compiles C files twice

[Glenn Morris]

Eli Zaretskii wrote:

> It used to be on all platforms, but I see that the compiler switch to
> set PURESIZE to a very large value was removed on Posix systems.  I
> guess they don't care about pure space overflow in bootstrap-emacs?
> If that's true, then we do need to rebuild with correct pure size once
> the *.el files are compiled and bootstrap-emacs can be tossed.

I see PURESIZE being messed about with in src/makefile.w32-in, but it
has not been in src/Makefile.in since rev 1.251, over six years ago.

> Yes, but do we really want to maintain a list of those who do depend
> on the pure size?

It looks like there is absolutely no need.

Since the top-level Makefile.in is not used by the Windows build (?),
there seems no reason to compile all the C files twice.

Re: bootstrap compiles C files twice

[Glenn Morris]

Glenn Morris wrote:

> It looks like there is absolutely no need.

(for those of us not using Windows, I meant.)

Re: bootstrap compiles C files twice

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Jason Rumney <jasonr@gnu.org>,  emacs-devel@gnu.org
> Date: Tue, 03 Jun 2008 14:40:54 -0400
> 
> Eli Zaretskii wrote:
> 
> > It used to be on all platforms, but I see that the compiler switch to
> > set PURESIZE to a very large value was removed on Posix systems.  I
> > guess they don't care about pure space overflow in bootstrap-emacs?
> > If that's true, then we do need to rebuild with correct pure size once
> > the *.el files are compiled and bootstrap-emacs can be tossed.
> 
> I see PURESIZE being messed about with in src/makefile.w32-in, but it
> has not been in src/Makefile.in since rev 1.251, over six years ago.
> 
> > Yes, but do we really want to maintain a list of those who do depend
> > on the pure size?
> 
> It looks like there is absolutely no need.
> 
> Since the top-level Makefile.in is not used by the Windows build (?),
> there seems no reason to compile all the C files twice.

See my remarks above about overflowing the pure storage while building
bootstrap-emacs.  I'd advise to understand that issue and its
implications first, before we eliminate the second compilation.  But
that's me.

Re: bootstrap compiles C files twice

[Dan Nicolaescu]

Eli Zaretskii <eliz@gnu.org> writes:

  > > From: Glenn Morris <rgm@gnu.org>
  > > Cc: Jason Rumney <jasonr@gnu.org>,  emacs-devel@gnu.org
  > > Date: Tue, 03 Jun 2008 14:40:54 -0400
  > > 
  > > Eli Zaretskii wrote:
  > > 
  > > > It used to be on all platforms, but I see that the compiler switch to
  > > > set PURESIZE to a very large value was removed on Posix systems.  I
  > > > guess they don't care about pure space overflow in bootstrap-emacs?
  > > > If that's true, then we do need to rebuild with correct pure size once
  > > > the *.el files are compiled and bootstrap-emacs can be tossed.
  > > 
  > > I see PURESIZE being messed about with in src/makefile.w32-in, but it
  > > has not been in src/Makefile.in since rev 1.251, over six years ago.
  > > 
  > > > Yes, but do we really want to maintain a list of those who do depend
  > > > on the pure size?
  > > 
  > > It looks like there is absolutely no need.
  > > 
  > > Since the top-level Makefile.in is not used by the Windows build (?),
  > > there seems no reason to compile all the C files twice.
  > 
  > See my remarks above about overflowing the pure storage while building
  > bootstrap-emacs. 

Is that an issue?  We don't seem to be even close to that, at least on a
GNU/Linux x86 machine:

[snip]
((286525 . 10109) (13904 . 2) (653 . 157) 2024114 1310655 (105 . 2) (24 . 49) (30703 . 11822))
Finding pointers to doc strings...
Finding pointers to doc strings...done
Finding pointers to doc strings...done
Dumping under names emacs and emacs-23.0.60
71704 pure bytes used
^^^^^
mv -f emacs bootstrap-emacs

Re: bootstrap compiles C files twice

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@ics.uci.edu>
> Cc: Glenn Morris <rgm@gnu.org>, emacs-devel@gnu.org, jasonr@gnu.org
> Date: Tue, 03 Jun 2008 13:52:34 -0700
> 
>   > See my remarks above about overflowing the pure storage while building
>   > bootstrap-emacs. 
> 
> Is that an issue?  We don't seem to be even close to that, at least on a
> GNU/Linux x86 machine:
> 
> [snip]
> ((286525 . 10109) (13904 . 2) (653 . 157) 2024114 1310655 (105 . 2) (24 . 49) (30703 . 11822))
> Finding pointers to doc strings...
> Finding pointers to doc strings...done
> Finding pointers to doc strings...done
> Dumping under names emacs and emacs-23.0.60
> 71704 pure bytes used
> ^^^^^

Isn't this small number a clear sign of pure space overflow?

Re: bootstrap compiles C files twice

[Dan Nicolaescu]

Eli Zaretskii <eliz@gnu.org> writes:

  > > From: Dan Nicolaescu <dann@ics.uci.edu>
  > > Cc: Glenn Morris <rgm@gnu.org>, emacs-devel@gnu.org, jasonr@gnu.org
  > > Date: Tue, 03 Jun 2008 13:52:34 -0700
  > > 
  > >   > See my remarks above about overflowing the pure storage while building
  > >   > bootstrap-emacs. 
  > > 
  > > Is that an issue?  We don't seem to be even close to that, at least on a
  > > GNU/Linux x86 machine:
  > > 
  > > [snip]
  > > ((286525 . 10109) (13904 . 2) (653 . 157) 2024114 1310655 (105 . 2) (24 . 49) (30703 . 11822))
  > > Finding pointers to doc strings...
  > > Finding pointers to doc strings...done
  > > Finding pointers to doc strings...done
  > > Dumping under names emacs and emacs-23.0.60
  > > 71704 pure bytes used
  > > ^^^^^
  > 
  > Isn't this small number a clear sign of pure space overflow?

There's usually a message when that occurs.  But after adding * 10 to
PURESIZE exactly the same number is printed.  So no overflow.

Re: bootstrap compiles C files twice

[Stefan Monnier]

>> > See my remarks above about overflowing the pure storage while building
>> > bootstrap-emacs. 
>> 
>> Is that an issue?  We don't seem to be even close to that, at least on a
>> GNU/Linux x86 machine:
>> 
>> [snip]
>> ((286525 . 10109) (13904 . 2) (653 . 157) 2024114 1310655 (105 . 2) (24 . 49) (30703 . 11822))
>> Finding pointers to doc strings...
>> Finding pointers to doc strings...done
>> Finding pointers to doc strings...done
>> Dumping under names emacs and emacs-23.0.60
>> 71704 pure bytes used
>> ^^^^^

> Isn't this small number a clear sign of pure space overflow?

I think it's a sign that loadup.el does

(if (or (equal (nth 3 command-line-args) "bootstrap")
    [...]
      ;; We'll probably overflow the pure space.
      (setq purify-flag nil)


-- Stefan

Re: bootstrap compiles C files twice

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Dan Nicolaescu <dann@ics.uci.edu>, rgm@gnu.org, jasonr@gnu.org,
>         emacs-devel@gnu.org
> Date: Tue, 03 Jun 2008 23:57:48 -0400
> 
> >> Dumping under names emacs and emacs-23.0.60
> >> 71704 pure bytes used
> >> ^^^^^
> 
> > Isn't this small number a clear sign of pure space overflow?
> 
> I think it's a sign that loadup.el does
> 
> (if (or (equal (nth 3 command-line-args) "bootstrap")
>     [...]
>       ;; We'll probably overflow the pure space.
>       (setq purify-flag nil)

Yes, exactly.

Re: bootstrap compiles C files twice

[Glenn Morris]

Eli Zaretskii wrote:

> See my remarks above about overflowing the pure storage while building
> bootstrap-emacs.  I'd advise to understand that issue and its
> implications first, before we eliminate the second compilation.  But
> that's me.

I don't understand what your point is.

I'm talking about re-using temacs, not the dumped emacs it creates.

Currently, we compile src/*.c to *.o, then link to make temacs. And we
do this twice. Each time is absolutely identical. I have checked this
by diffing the relevant section of the build logs, and the *.o and
temacs files generated each time.

I can see that in makefile.w32-in, you use a different PURESPACE
option to build bootstrap-temacs; but as I said, this has not been
true for six years in Makefile.in. I have no interest in making any
changes to makefile.w32-in. Perhaps you mean Makefile.in might have
been doing it wrong for six years?

The cross-compile argument would make sense to me, but Dan says it
doesn't work anyway.

Apologies if I have the wrong end of the stick...

Re: bootstrap compiles C files twice

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: jasonr@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 03 Jun 2008 18:18:54 -0400
> 
> Eli Zaretskii wrote:
> 
> > See my remarks above about overflowing the pure storage while building
> > bootstrap-emacs.  I'd advise to understand that issue and its
> > implications first, before we eliminate the second compilation.  But
> > that's me.
> 
> I don't understand what your point is.

I have no point.  I was saying that if I needed to decide whether to
eliminate the second round of compilation, I'd first want to
understand why was the -DPURESIZE= option removed from src/Makefile.in
during the first compilation, even though the produced bootstrap-emacs
loads Lisp files in source form, which uses more pure space.  Without
a clear understanding of that, I couldn't be sure why -DPURESIZE= was
removed, but the second compilation wasn't.  Maybe it was just by
omission, but maybe it wasn't.

> I can see that in makefile.w32-in, you use a different PURESPACE
> option to build bootstrap-temacs; but as I said, this has not been
> true for six years in Makefile.in. I have no interest in making any
> changes to makefile.w32-in. Perhaps you mean Makefile.in might have
> been doing it wrong for six years?

No, I meant that perhaps Makefile.in was doing for six years two
compilations for a good reason.

> The cross-compile argument would make sense to me, but Dan says it
> doesn't work anyway.

It never did.  It would be nice to have that, but that would be a
different project, and I didn't mean that.

Re: bootstrap compiles C files twice

[Glenn Morris]

Eli Zaretskii wrote:

> No, I meant that perhaps Makefile.in was doing for six years two
> compilations for a good reason.

I'm sorry, I can't understand this.

Re: bootstrap compiles C files twice

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: jasonr@gnu.org,  emacs-devel@gnu.org
> Date: Wed, 04 Jun 2008 15:20:31 -0400
> 
> Eli Zaretskii wrote:
> 
> > No, I meant that perhaps Makefile.in was doing for six years two
> > compilations for a good reason.
> 
> I'm sorry, I can't understand this.

Sorry for being unclear.  I meant to say that there might be a good
reason for compiling the sources twice, and that understanding how and
why did we get rid of the -DPURESIZE= switch on the first of these two
compilations might help us understand that good reason, or realize
that there isn't one.

Re: bootstrap compiles C files twice

[Glenn Morris]

Eli Zaretskii wrote:

> I meant to say that there might be a good reason for compiling the
> sources twice, and that understanding how and why did we get rid of
> the -DPURESIZE= switch on the first of these two compilations might
> help us understand that good reason, or realize that there isn't
> one.

I can't imagine that you mean there's a reason to do two identical
compilations, so you can only mean (?) that you are concerned the
-DPURESIZE difference might need to be put back. To which I can only
say again that it's worked fine without it for years. I guess the
reason is the already mentioned contemporaneous change to set
purify-flag nil when making bootstrap-emacs.

The fact that dumping bootstrap-emacs reports: 109423 pure bytes used

whereas dumping emacs proper reports: 1961272 pure bytes used

makes me feel pretty relaxed about PURESIZE issues.

Re: bootstrap compiles C files twice

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: jasonr@gnu.org,  emacs-devel@gnu.org
> Date: Thu, 05 Jun 2008 15:29:56 -0400
> 
> Eli Zaretskii wrote:
> 
> > I meant to say that there might be a good reason for compiling the
> > sources twice, and that understanding how and why did we get rid of
> > the -DPURESIZE= switch on the first of these two compilations might
> > help us understand that good reason, or realize that there isn't
> > one.
> 
> I can't imagine that you mean there's a reason to do two identical
> compilations

An identical compiler command line does not necessarily mean an
identical object file, if, for example, the source file includes
something that was changed in between.

Anyway, now you know what I meant; time for me to stop talking and for
you to make a decision.

> The fact that dumping bootstrap-emacs reports: 109423 pure bytes used

As Stefan pointed out, that number should not be used as a measure of
how large should the pure[] array be, since bootstrap-emacs overflows
the pure space (on Posix platforms) and uses malloc'ed space instead.