Yet another bootstrap failure: Required feature `esh-groups' was not provided

Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Alan Mackenzie]

Hi, Emacs!

Yet another bootstrap failure:

Friday 2006-06-06, ~15:00 UCT
% cvs update
% ./configure --with-gif=no --with-tiff=no
% make bootstrap

.......
.......

Compiling /home/acm/emacs/emacs/lisp/eshell/em-alias.el

In toplevel form:
eshell/em-alias.el:96:1:Error: Required feature `esh-groups' was not provided
make[3]: *** [/home/acm/emacs/emacs/lisp/eshell/em-alias.elc] Error 1
make[3]: Leaving directory `/home/acm/emacs/emacs/lisp'
make[2]: *** [compile] Error 2
make[2]: Leaving directory `/home/acm/emacs/emacs/lisp'
make[1]: *** [bootstrap-build] Error 2
make[1]: Leaving directory `/home/acm/emacs/emacs'
make: *** [bootstrap] Error 2

Am I doing something wrong?

Unless one has the energy to debug build failures, it's hardly worth the
hassle of cvs-updating Emacs at the moment.  It _never_ builds cleanly -
unless the moon is currently blue.  It seems it's not just me.

This constant brokenness of the CVS head saps infinite energy from me.
This probably isn't just me, either.  I really can't be bothered to debug
the latest commission every time I do a CVS update.  And yes, I'm aware
I've broken the CVS build in the past, too.

There was some talk a while ago of commissions triggering an automatic
build.  This would be nice.

A makefile with proper dependencies might also be a solution.

Please can we fix this with some urgency?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Glenn Morris]

Alan Mackenzie wrote:

> eshell/em-alias.el:96:1:Error: Required feature `esh-groups' was not provided
[...]
> Am I doing something wrong?

Yes. Please read INSTALL.CVS, or search the list archives.

(make autogen-clean; make autoloads _may_ help you in this case)

> This constant brokenness of the CVS head saps infinite energy from me.

The constant reporting of the same non-bugs saps my energy (but
fortunately not an "infinite" amount).

> There was some talk a while ago of commissions triggering an automatic
> build.  This would be nice.

It wouldn't help all the people who persist in building Emacs by a
shortcut, then getting confused when it breaks.

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Dan Nicolaescu]

Glenn Morris <rgm@gnu.org> writes:

  > Alan Mackenzie wrote:
  > 
  > > eshell/em-alias.el:96:1:Error: Required feature `esh-groups' was not provided
  > [...]
  > > Am I doing something wrong?
  > 
  > Yes. Please read INSTALL.CVS, or search the list archives.
  > 
  > (make autogen-clean; make autoloads _may_ help you in this case)
  > 
  > > This constant brokenness of the CVS head saps infinite energy from me.
  > 
  > The constant reporting of the same non-bugs saps my energy (but
  > fortunately not an "infinite" amount).

Given that people seem to expect that "make bootstrap" works all the
time, would it be a good idea to make "make bootstrap" to first clean
everything first so that it starts from a clean slate and then it has a
greater chance of meeting people's expectations?
This might reduce the amount of time you have to spend responding to
these types of messages... 

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Stefan Monnier]

> Unless one has the energy to debug build failures, it's hardly worth the
> hassle of cvs-updating Emacs at the moment.  It _never_ builds cleanly -
> unless the moon is currently blue.  It seems it's not just me.

Unless you have extra time to waste, I strongly recommend not to
bootstrap all the time.  But it wouldn't solve your problem in this
case, admittedly.

> Given that people seem to expect that "make bootstrap" works all the
> time, would it be a good idea to make "make bootstrap" to first clean
> everything first so that it starts from a clean slate and then it has a
> greater chance of meeting people's expectations?
> This might reduce the amount of time you have to spend responding to
> these types of messages... 

100% agreement.


        Stefan

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Alan Mackenzie]

'Evening, Glenn!

On Fri, Jun 06, 2008 at 01:22:09PM -0400, Glenn Morris wrote:
> Alan Mackenzie wrote:

> > eshell/em-alias.el:96:1:Error: Required feature `esh-groups' was not provided
> [...]
> > Am I doing something wrong?

> Yes. Please read INSTALL.CVS, or search the list archives.

Did that over 5 years ago.  It says:

    Therefore, to build from CVS you must run "make bootstrap"
    instead of just "make":

      $ ./configure
      $ make bootstrap

    The bootstrap process makes sure all necessary files are rebuilt
    before it builds the final Emacs binary.

ALL NECESSARY FILES is what it says.  Oops, my mistake, that's the Emacs
22 version of INSTALL.CVS.

NO, it is NOT my mistake.  It's an incompatible change, and the person
who made that change should be strung up, even if that person is you,
Glenn.  ;-)   It's a gratuitous, unnecessary, incompatible change, since
the documented meaning of "make bootstrap", i.e. "rebuild everything
needed" could quite easily have been kept.  Would it be possible to bring
back that meaning?

> (make autogen-clean; make autoloads _may_ help you in this case)

Thanks, but it didn't.  :-(  The makefile couldn't find the target
autogen-clean.  So I followed the instructions in the new INSTALL.CVS
and first did % make maintainer-clean; ./configure .... ; make
bootstrap.

This failed as follows:

    make[3]: *** No rule to make target `/home/acm/emacs/emacs/lisp/nxml/nxml-enc.elc', needed by `compile-main'.  Stop.
    make[3]: Leaving directory `/home/acm/emacs/emacs/lisp'

<rant>
Our makefile system is broken.  The whole point of the make utility is to
build software automatically, so that hackers don't need to waste their
time messing with arcane minutiae.  Is it really to much to expect to be
able to type

    % ./configure <params>; make some-target

and have the software build (modulo errors in the files.{c,el}?

> > This constant brokenness of the CVS head saps infinite energy from me.

> The constant reporting of the same non-bugs saps my energy (but
> fortunately not an "infinite" amount).

Forgive my sarcasm, but am I supposed to read through (or diff) the
entire Emacs process documentation every time I update Emacs, just in
case some crazed lunatic, er sorry, I mean some conscientious hacker, has
"enhanced" it?

If a "non-bug" is reported often enough by enough people, then it is an
actual bug; it's a UI bug, or a documentation bug, or a process bug, or
whatever, but it's unmistakably a bug.

Just as a matter of interest, in the last few months on emacs-devel
there have been approximately these numbers of threads complaining about
Emacs CVS not building:

May:       7
April:     9
March:     9
February:  2

Whatever the reason, this is a horrendous time sink.  Failure to build
the CVS head should essentially _never_ happen - possibly once or twice
a year at most.  Our process is badly broken.
</rant>

Recently, I proposed installing a tool on savannah which would trigger a
test build every time source files were committed.  The proposal didn't
meet with much enthusiasm.

> > There was some talk a while ago of commissions triggering an automatic
> > build.  This would be nice.

> It wouldn't help all the people who persist in building Emacs by a
> shortcut, then getting confused when it breaks.

Since when has "./configure; make bootstrap" been a shortcut?  (That's
both a rhetorical and a real question.)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Eli Zaretskii]

> Date: Fri, 6 Jun 2008 20:35:41 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: emacs-devel@gnu.org
> 
> Our makefile system is broken.  The whole point of the make utility is to
> build software automatically, so that hackers don't need to waste their
> time messing with arcane minutiae.  Is it really to much to expect to be
> able to type
> 
>     % ./configure <params>; make some-target
> 
> and have the software build (modulo errors in the files.{c,el}?

This does work, just not in an arbitrary CVS-updated sandbox.  That
is, in a release tarball, the configury behaves exactly as you want it
to.

> Forgive my sarcasm, but am I supposed to read through (or diff) the
> entire Emacs process documentation every time I update Emacs, just in
> case some crazed lunatic, er sorry, I mean some conscientious hacker, has
> "enhanced" it?

Sorry, Alan, but your expectations are too high.  AFAIK, a bootstrap
build is guaranteed to work only in a fresh CVS checkout.  It is not
guaranteed to work in a sandbox littered by stale files.  I agree that
it would be great to have more, but it's a lot of work, and the
results cannot be reasonably tested in practice, since the number of
different ways you can screw up your sandbox is infinite.

People who regularly update from CVS trunk are expected to be able to
tinker with their files, and have enough energy for that.

> Just as a matter of interest, in the last few months on emacs-devel
> there have been approximately these numbers of threads complaining about
> Emacs CVS not building:
> 
> May:       7
> April:     9
> March:     9
> February:  2

That's expected, in a trunk that is actively developed by many
contributors at once.

> Whatever the reason, this is a horrendous time sink.  Failure to build
> the CVS head should essentially _never_ happen - possibly once or twice
> a year at most.

Sorry, but it's hopelessly unrealistic to request that.  Given the
number of different platforms we support, it is impractical even to
request that any given change will compile on all of them, let alone
build without a hitch.

> Recently, I proposed installing a tool on savannah which would trigger a
> test build every time source files were committed.  The proposal didn't
> meet with much enthusiasm.

Well, how about volunteering to do it, then?  It obviously bugs you
enough to make you a motivated individual.

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Vinicius Jose Latorre]

Alan Mackenzie wrote:
> Hi, Emacs!
>
> Yet another bootstrap failure:
>
> Friday 2006-06-06, ~15:00 UCT
> % cvs update
> % ./configure --with-gif=no --with-tiff=no
> % make bootstrap
>
> .......
> .......
>
> Compiling /home/acm/emacs/emacs/lisp/eshell/em-alias.el
>
> In toplevel form:
> eshell/em-alias.el:96:1:Error: Required feature `esh-groups' was not provided
> make[3]: *** [/home/acm/emacs/emacs/lisp/eshell/em-alias.elc] Error 1
> make[3]: Leaving directory `/home/acm/emacs/emacs/lisp'
> make[2]: *** [compile] Error 2
> make[2]: Leaving directory `/home/acm/emacs/emacs/lisp'
> make[1]: *** [bootstrap-build] Error 2
> make[1]: Leaving directory `/home/acm/emacs/emacs'
> make: *** [bootstrap] Error 2
>
> Am I doing something wrong?
>   

Did you try the steps below?

% cvs update
% make maintainer-clean
% ./configure --with-gif=no --with-tiff=no
% make boostrap


> Unless one has the energy to debug build failures, it's hardly worth the
> hassle of cvs-updating Emacs at the moment.  It _never_ builds cleanly -
> unless the moon is currently blue.  It seems it's not just me.
>
> This constant brokenness of the CVS head saps infinite energy from me.
> This probably isn't just me, either.  I really can't be bothered to debug
> the latest commission every time I do a CVS update.  And yes, I'm aware
> I've broken the CVS build in the past, too.
>
> There was some talk a while ago of commissions triggering an automatic
> build.  This would be nice.
>
> A makefile with proper dependencies might also be a solution.
>
> Please can we fix this with some urgency?
>
>   

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Glenn Morris]

Alan Mackenzie wrote:

> It's an incompatible change,

What is, precisely?

> Thanks, but it didn't.  :-(  The makefile couldn't find the target
> autogen-clean.

In other words, your lisp/Makefile wasn't even up to date.

>     make[3]: *** No rule to make target `/home/acm/emacs/emacs/lisp/nxml/nxml-enc.elc', needed by `compile-main'.  Stop.

Can't help with that.

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Glenn Morris]

>> Recently, I proposed installing a tool on savannah which would trigger a
>> test build every time source files were committed.

I do this every day anyway. Obviously it wouldn't help any of the
people who have been writing to this list with their own particular
problems.

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Alan Mackenzie]

Hi, Glenn!

On Fri, Jun 06, 2008 at 10:51:36PM -0400, Glenn Morris wrote:
> Alan Mackenzie wrote:

> > It's an incompatible change,

> What is, precisely?

Er, the change in the meaning of "make bootstrap" which I saw last night.
I think I saw something which wasn't there.  Sorry for shouting at
everybody about this.

> > Thanks, but it didn't.  :-(  The makefile couldn't find the target
> > autogen-clean.

> In other words, your lisp/Makefile wasn't even up to date.

No, autogen-clean exists only in .../lisp/Makefile, not the top level
one.  :-)

> >     make[3]: *** No rule to make target `/home/acm/emacs/emacs/lisp/nxml/nxml-enc.elc', needed by `compile-main'.  Stop.

> Can't help with that.

There've been changes to nxml-mode recently.  Maybe the Makefile just
needs tweaking.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Alan Mackenzie]

Hi, Glenn,

On Fri, Jun 06, 2008 at 10:53:36PM -0400, Glenn Morris wrote:

> >> Recently, I proposed installing a tool on savannah which would trigger a
> >> test build every time source files were committed.

> I do this every day anyway.

Which of my verbs does "this" reference?

Assuming you mean that you regularly test the build process somehow,
what happens when the build fails?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Alan Mackenzie]

Hi, Eli!

On Fri, Jun 06, 2008 at 11:41:36PM +0300, Eli Zaretskii wrote:
> > Date: Fri, 6 Jun 2008 20:35:41 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: emacs-devel@gnu.org

> > Our makefile system is broken.  The whole point of the make utility
> > is to build software automatically, so that hackers don't need to
> > waste their time messing with arcane minutiae.  Is it really to much
> > to expect to be able to type

> >     % ./configure <params>; make some-target

> > and have the software build (modulo errors in the files.{c,el}?

> This does work, just not in an arbitrary CVS-updated sandbox.  That
> is, in a release tarball, the configure behaves exactly as you want it
> to.

That's not the point here.  Why shouldn't make be able to work in an
arbitrary CVS setup?  I think it could, that it should, just that it
doesn't at the moment.  I can't see any feature of Emacs which renders
the make paradigm (putting dependencies into Makefile and checking
timestamps) non functional.  Our Makefiles simply don't record the
dependencies properly.

> Sorry, Alan, but your expectations are too high.  AFAIK, a bootstrap
> build is guaranteed to work only in a fresh CVS checkout.  It is not
> guaranteed to work in a sandbox littered by stale files.

OK, I didn't know that.  So a workaround for me could be to copy all
source files to a temporary directory structure, build there with make
bootstrap, then copy the resulting files back to my main directory.

"Stale files" means, for example, .elc files whose source hasn't changed,
but use defuns at byte-compilation time or macros which have changed.

> I agree that it would be great to have more, but it's a lot of work,
> and the results cannot be reasonably tested in practice, since the
> number of different ways you can screw up your sandbox is infinite.

Agree, agree, disagree, agree.  It may not be possible for a build to
work all the time, but it could be made to work nearly all the time.  I
think; I hope.

> People who regularly update from CVS trunk are expected to be able to
> tinker with their files, and have enough energy for that.

:-)  I conjecture that it's updating sporadically rather than continually
that causes the pain.

> > Just as a matter of interest, in the last few months on emacs-devel
> > there have been approximately these numbers of threads complaining
> > about Emacs CVS not building:

> > May:       7
> > April:     9
> > March:     9
> > February:  2

> That's expected, in a trunk that is actively developed by many
> contributors at once.

?????  Do you know whether it happens in other projects with ~20 active
developers?

> > Whatever the reason, this is a horrendous time sink.  Failure to
> > build the CVS head should essentially _never_ happen - possibly once
> > or twice a year at most.

> Sorry, but it's hopelessly unrealistic to request that.  Given the
> number of different platforms we support, it is impractical even to
> request that any given change will compile on all of them, let alone
> build without a hitch.

OK.  I have an utterly standard, if somewhat old, GNU/Linux box, probably
the most popular setup.  I think it's reasonable to expect the trunk to
build on my box nearly all the time.  Possibly on w32, too.

> > Recently, I proposed installing a tool on savannah which would
> > trigger a test build every time source files were committed.  The
> > proposal didn't meet with much enthusiasm.

> Well, how about volunteering to do it, then?  It obviously bugs you
> enough to make you a motivated individual.

Yes.  I think I will try to enhance Makefile.in to record more
dependencies.  

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Eli Zaretskii]

> Date: Sat, 7 Jun 2008 09:50:24 +0000
> Cc: rgm@gnu.org, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > This does work, just not in an arbitrary CVS-updated sandbox.  That
> > is, in a release tarball, the configure behaves exactly as you want it
> > to.
> 
> That's not the point here.  Why shouldn't make be able to work in an
> arbitrary CVS setup?

Because development adds and removes files (which doesn't happen with
releases), and stale files can get in the way in unpredictable ways.

> Our Makefiles simply don't record the dependencies properly.

That might be one reason for the problems, but it isn't the only one.
And if you are talking about Lisp files, their dependencies are not
easy to find out, due to lots of macros implicitly imported at compile
time via `require'.

Perhaps we should first improve our infrastructure: how about a switch
to Emacs, like the -MD switch to GCC, that would cause it to generate
a Make include file with dependencies as a side effect of byte
compilation?  Without help from the byte compiler, I'd consider any
additional dependencies for Lisp files an unjustified maintenance
burden, because those dependencies will need to be updated any time
some `require' line somewhere is added, deleted, or modified.

> > I agree that it would be great to have more, but it's a lot of work,
> > and the results cannot be reasonably tested in practice, since the
> > number of different ways you can screw up your sandbox is infinite.
> 
> Agree, agree, disagree, agree.  It may not be possible for a build to
> work all the time, but it could be made to work nearly all the time.  I
> think; I hope.

It's fruitless to argue with hopes, so I won't.

> > People who regularly update from CVS trunk are expected to be able to
> > tinker with their files, and have enough energy for that.
> 
> :-)  I conjecture that it's updating sporadically rather than continually
> that causes the pain.

"Regularly" doesn't mean every day; it could mean once a week or once
in a fortnight.

> > > May:       7
> > > April:     9
> > > March:     9
> > > February:  2
> 
> > That's expected, in a trunk that is actively developed by many
> > contributors at once.
> 
> ?????  Do you know whether it happens in other projects with ~20 active
> developers?

None of the projects I'm involved with that have something similar to
"bootstrap" use the kind of ``fire at will'' commit policy we use in
Emacs.  Those other projects all have some kind of mandatory
review-before-commit policy for all but a few extremely trusted
developers.  At peer review time, problems can be detected before they
do any harm.

> OK.  I have an utterly standard, if somewhat old, GNU/Linux box, probably
> the most popular setup.  I think it's reasonable to expect the trunk to
> build on my box nearly all the time.

What is a ``standard GNU/Linux box''?  Is it 32-bit or 64-bit?  What
kernel version do you use, and what version of glibc?  What compiler
version?

Any one of the above factors, and perhaps more, could cause a change
that compiles on the contributor's machine not to compile on yours.

> > > Recently, I proposed installing a tool on savannah which would
> > > trigger a test build every time source files were committed.  The
> > > proposal didn't meet with much enthusiasm.
> 
> > Well, how about volunteering to do it, then?  It obviously bugs you
> > enough to make you a motivated individual.
> 
> Yes.  I think I will try to enhance Makefile.in to record more
> dependencies.  

Thank you.

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Glenn Morris]

Alan Mackenzie wrote:

> Assuming you mean that you regularly test the build process somehow,
> what happens when the build fails?

I fix it, if I can. Otherwise wait a bit before reporting, because 56
other people are probably composing "My Emacs does not work!11!!!"
emails at the same time.

But I'm only testing a clean build, so people not starting from a
clean slate may see other problems. It's not reasonable to expect
every possible starting point to work flawlessly.

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Alan Mackenzie]

Hi, Eli!

On Sat, Jun 07, 2008 at 03:19:31PM +0300, Eli Zaretskii wrote:
> > Date: Sat, 7 Jun 2008 09:50:24 +0000
> > Cc: rgm@gnu.org, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> > Our Makefiles simply don't record the dependencies properly.

> That might be one reason for the problems, but it isn't the only one.
> And if you are talking about Lisp files, their dependencies are not
> easy to find out, due to lots of macros implicitly imported at compile
> time via `require'.

It can't be that difficult; lisp is designed for this type of
manipulation.

> Perhaps we should first improve our infrastructure: how about a switch
> to Emacs, like the -MD switch to GCC, that would cause it to generate
> a Make include file with dependencies as a side effect of byte
> compilation?  Without help from the byte compiler, I'd consider any
> additional dependencies for Lisp files an unjustified maintenance
> burden, because those dependencies will need to be updated any time
> some `require' line somewhere is added, deleted, or modified.

The dependencies absolutely must be generated automatically.  Whether
this should be done by the byte-compiler or a separate script isn't clear
(yet).  Such a separate script could probably be run in temacs, creating
the dependencies very early in the build process.

> > > I agree that it would be great to have more, but it's a lot of work,
> > > and the results cannot be reasonably tested in practice, since the
> > > number of different ways you can screw up your sandbox is infinite.

> > Agree, agree, disagree, agree.  It may not be possible for a build to
> > work all the time, but it could be made to work nearly all the time.  I
> > think; I hope.

> It's fruitless to argue with hopes, so I won't.

Oh, you cynical person.  ;-)

> > :-)  I conjecture that it's updating sporadically rather than
> > continually that causes the pain.

> "Regularly" doesn't mean every day; it could mean once a week or once
> in a fortnight.

"Sporadically", for me, means once a month or once every two months.

> > > > May:       7
> > > > April:     9
> > > > March:     9
> > > > February:  2

> > > That's expected, in a trunk that is actively developed by many
> > > contributors at once.

> > ?????  Do you know whether it happens in other projects with ~20
> > active developers?

> None of the projects I'm involved with that have something similar to
> "bootstrap" use the kind of ``fire at will'' commit policy we use in
> Emacs.  Those other projects all have some kind of mandatory
> review-before-commit policy for all but a few extremely trusted
> developers.  At peer review time, problems can be detected before they
> do any harm.

One kind of "peer review" we could do is doing a build test as part of
the commission process.  This might be a bit heavy on server CPU time.

> > OK.  I have an utterly standard, if somewhat old, GNU/Linux box,
> > probably the most popular setup.  I think it's reasonable to expect
> > the trunk to build on my box nearly all the time.

> What is a ``standard GNU/Linux box''?  Is it 32-bit or 64-bit?  What
> kernel version do you use, and what version of glibc?  What compiler
> version?

For what it's worth:
32-bit;
Linux acm 2.6.8 #7 Wed May 23 18:12:53 BST 2007 i686 GNU/Linux.
glibc: don't know, how do you display the version number?
gcc (GCC) 3.3.5 (Debian 1:3.3.5-13)

I can't honestly see that it's worth that much.  Won't Emacs compile with
pretty much any Linux version, any GCC and any GLIBC?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Stephen J. Turnbull]

Alan Mackenzie writes:

 alan> It can't be that difficult; lisp is designed for this type of
 alan> manipulation.

It's not easy.  Several people at XEmacs have tried over a period of
years.  What hasn't been tried yet is adapting the byte-compiler to
the task.  It's been suggested, but people who hack byte generally
don't have a problem with debugging the makefile problem.

 alan> The dependencies absolutely must be generated automatically.
 alan> Whether this should be done by the byte-compiler or a separate
 alan> script isn't clear (yet).  Such a separate script could
 alan> probably be run in temacs, creating the dependencies very early
 alan> in the build process.

I don't think that makes a lot of sense.  The problem is that to
generate the Lisp dependencies you have to do this parsing of *all*
the Lisp files.  If you're going to do that, why not just do "find
. -name '*.elc' -exec rm '{}'" and recompile?  (Especially on Windows
just reading that many files costs a lot of time, although the
byte-optimizer is quite time-consuming, too.)

Also, there's no real reason why (non-dependency-breaking) users and
3rd party developers should have to run this.  Rather, have the
committers run it as part of their precommit testing (although they
mostly won't, at least you have an appropriate person to take out your
frustration on).

 alan> One kind of "peer review" we could do is doing a build test as
 alan> part of the commission process.  This might be a bit heavy on
 alan> server CPU time.

That's not really helpful, it's only one machine.  Eg, I misdoubt that
Glenn sees very many broken builds, that's why he committed those
changes.  The useful aspect of "one-time" build testing is to insist
that committers do a "make" just to ensure that anything they touched
has no syntax errors in it, that's about the maximum effect you can
ask for.

"Build-bot" has been mentioned; that would be a much more useful
infrastructure improvement.  I suspect it would cost about as much
hacker effort to set up a build-bot master as to hack the commit-hook
to run a build, it needn't be redone when (medatashi, medetashi!) CVS
bites the dust, and it won't involve repo downtime at all.

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Romain Francoise]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> "Build-bot" has been mentioned; that would be a much more useful
> infrastructure improvement.  I suspect it would cost about as much
> hacker effort to set up a build-bot master as to hack the commit-hook
> to run a build, it needn't be redone when (medatashi, medetashi!) CVS
> bites the dust, and it won't involve repo downtime at all.

I run an unofficial buildbot at <URL:http://build.orebokech.com/>,
it's not hooked into CVS but does regular builds (every 6 hours), on
GNU/Linux.  When it finds bugs I report them to the person who
committed the offending change.  So far it's been pretty rare event;
my logs show that bootstrap has broken only five times in the past
few months: on Apr 12, May 13, May 15, May 18 and Jun 03.  (There
may have been more but they got fixed in less than 6 hours...)

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > None of the projects I'm involved with that have something similar to
 > "bootstrap" use the kind of ``fire at will'' commit policy we use in
 > Emacs.  Those other projects all have some kind of mandatory
 > review-before-commit policy for all but a few extremely trusted
 > developers.  At peer review time, problems can be detected before they
 > do any harm.

Interesting theory.  However, the way it actually works in some
projects is that the review-before-commit policy is mandatory for the
extremely trusted developers, too.  They're not trusted to do it right
the first time, they're trusted to get it right by the time they
commit.  The difference between them and the common herd is that they
are trusted to know when they need peer review (changes in the build
process are one example), and when they can review their own code
(straightforward bug fixes are most common).

Eg, Python has at least as many people who can commit on their own
responsibility as Emacs does, but build breakage on the half-dozen
"common" platforms is rare[1], and I've never seen threads on
regressions in the build for parts of the project implemented in
Python.  The practice of having trusted developers do self-review is
enabled by infrastructure including a test suite containing about 500
files and a "buildbot" network.


Footnotes: 
[1]  Admittedly, they typically lag about 3 years on supporting
Microsoft "native" compilers.  I can understand that, though.<wink>

Build-time dependencies

[Stefan Monnier]

> The dependencies absolutely must be generated automatically.  Whether
> this should be done by the byte-compiler or a separate script isn't clear
> (yet).  Such a separate script could probably be run in temacs, creating
> the dependencies very early in the build process.

I think it should be easy to generate the dependencies from bytecomp.el
(just record macros as we expand them, and then try and figure out
which file they come from).  This way they could be kept uptodate
100% automatically.


        Stefan

Re: Yet another bootstrap failure: Required feature `esh-groups' was not provided

[Stefan Monnier]

> GNU/Linux.  When it finds bugs I report them to the person who
> committed the offending change.  So far it's been pretty rare event;
> my logs show that bootstrap has broken only five times in the past
> few months: on Apr 12, May 13, May 15, May 18 and Jun 03.  (There
> may have been more but they got fixed in less than 6 hours...)

Now, that's what I like to hear.


        Stefan ;-)

Re: Build-time dependencies

[Glenn Morris]

Stefan Monnier wrote:

> I think it should be easy to generate the dependencies from bytecomp.el
> (just record macros as we expand them, and then try and figure out
> which file they come from).  This way they could be kept uptodate
> 100% automatically.

What exactly is the point of doing this supposed to be?
To make non-bootstrap builds more reliable?

Re: Build-time dependencies

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Alan Mackenzie <acm@muc.de>,  Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sun, 08 Jun 2008 15:03:07 -0400
> 
> Stefan Monnier wrote:
> 
> > I think it should be easy to generate the dependencies from bytecomp.el
> > (just record macros as we expand them, and then try and figure out
> > which file they come from).  This way they could be kept uptodate
> > 100% automatically.
> 
> What exactly is the point of doing this supposed to be?
> To make non-bootstrap builds more reliable?

To make dependencies between Lips files explicit in lisp/Makefile.in,
and thus causing Make to compile Lisp files in the right order.  Right
now, whenever some Lisp files change, I frequently need to "make
recompile" several times, sometimes removing stale *.elc files
manually, until it finally succeeds, because it basically compiles
them in a random order.

Re: Build-time dependencies

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Glenn Morris <rgm@gnu.org>
>> Cc: Alan Mackenzie <acm@muc.de>,  Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Sun, 08 Jun 2008 15:03:07 -0400
>> 
>> Stefan Monnier wrote:
>> 
>> > I think it should be easy to generate the dependencies from bytecomp.el
>> > (just record macros as we expand them, and then try and figure out
>> > which file they come from).  This way they could be kept uptodate
>> > 100% automatically.
>> 
>> What exactly is the point of doing this supposed to be?
>> To make non-bootstrap builds more reliable?
>
> To make dependencies between Lips files explicit in lisp/Makefile.in,
> and thus causing Make to compile Lisp files in the right order.  Right
> now, whenever some Lisp files change, I frequently need to "make
> recompile" several times, sometimes removing stale *.elc files
> manually, until it finally succeeds, because it basically compiles
> them in a random order.

Personally, I would appreciate it if we could get of the bootstrap
target altogether, and when Emacs chose to do a bootstrap on "make"
whenever things could require it.

Similar with the "clean" target.  Having special targets that will _not_
bootstrap even when it _could_ be necessary in order to make debugging
faster/easier is nice.

But I think without extra measures, it would be nice if "make" did
everything that might be required.  The combination of "recompile" and
"cvs-update" and "bootstrap" and "whatever-clean" in what ever
directories that one has to try whenever something looks like not
working:

That is not really the idea behind using make.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Build-time dependencies

[Glenn Morris]

Eli Zaretskii wrote:

>> What exactly is the point of doing this supposed to be?
>> To make non-bootstrap builds more reliable?
>
> To make dependencies between Lips files explicit in lisp/Makefile.in,
> and thus causing Make to compile Lisp files in the right order.  Right
> now, whenever some Lisp files change, I frequently need to "make
> recompile" several times, sometimes removing stale *.elc files
> manually, until it finally succeeds, because it basically compiles
> them in a random order.

I think that's what I said.

My solution is just to delete all the *.elc and not think about it.
The time it takes to fully rebuild Emacs is nothing special compared
to other modern applications.

I would just discourage the use of `recompile' et al except by those
who know what they are doing, have slow machines, and need to rebuild
Emacs frequently. But I know a lot of people seem to feel differently.

Re: Build-time dependencies

[Glenn Morris]

Glenn Morris wrote:

> I would just discourage the use of `recompile' et al except by those
> who know what they are doing, have slow machines, and need to rebuild
> Emacs frequently.

PS as a data point, it takes less than 10 minutes to bootstrap on a ~2
year old dual core machine. It's getting hard to buy anything now that
has only one core, even in a laptop. But I know support for older
hardware is important to Emacs.

Re: Build-time dependencies

[Stefan Monnier]

> To make dependencies between Lips files explicit in lisp/Makefile.in,

No, they'd be put into a separate file not under CVS control.

> and thus causing Make to compile Lisp files in the right order.  Right
> now, whenever some Lisp files change, I frequently need to "make
> recompile" several times, sometimes removing stale *.elc files
> manually, until it finally succeeds, because it basically compiles
> them in a random order.

Indeed.


        Stefan

Re: Build-time dependencies

[Stephen J. Turnbull]

Glenn Morris writes:

 > I would just discourage the use of `recompile' et al except by those
 > who know what they are doing, have slow machines, and need to rebuild
 > Emacs frequently. But I know a lot of people seem to feel differently.

The simple thing to do is to put "bootstrap:" as the first line in the
Makefile.

Re: Build-time dependencies

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: acm@muc.de,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 08 Jun 2008 21:49:02 -0400
> 
> Glenn Morris wrote:
> 
> > I would just discourage the use of `recompile' et al except by those
> > who know what they are doing, have slow machines, and need to rebuild
> > Emacs frequently.
> 
> PS as a data point, it takes less than 10 minutes to bootstrap on a ~2
> year old dual core machine.

It takes 30 minutes on mine, maybe because it was shining and new more
than 3 years ago, has only 1 core, and doesn't run GNU/Linux.

> It's getting hard to buy anything now that has only one core, even
> in a laptop.

Well, no one goes and buys a new machine just to build Emacs.

Re: Build-time dependencies

[Robert J. Chassell]

    PS as a data point, it takes less than 10 minutes to bootstrap ...

For me it takes more than 40 minutes on my most commonly used machine,
a laptop, and I still have not found another screen with the number of
pixels and the quality of this one, although I expect such within five
years.  Bootstraping once took much longer and I am glad the process
has been made more efficient.

In any case, I hardly ever bootstrap although I try to update every day.

People's milage varies ...

I am not against the various new targets that have been suggested and
if they are installed, I might modify my lisp expression.

-- 
    Robert J. Chassell                          GnuPG Key ID: 004B4AC8
    bob@rattlesnake.com                         bob@gnu.org
    http://www.rattlesnake.com                  http://www.teak.cc

Re: Build-time dependencies

[Ted Zlatanov]

On Mon, 09 Jun 2008 11:26:18 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

EZ> Well, no one goes and buys a new machine just to build Emacs.

Of course, we do it to *run* Emacs :)

Ted

Re: Build-time dependencies

[David Kastrup]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Mon, 09 Jun 2008 11:26:18 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 
>
> EZ> Well, no one goes and buys a new machine just to build Emacs.
>
> Of course, we do it to *run* Emacs :)

So we do it to build character?

-- 
David Kastrup

Re: Build-time dependencies

[Alan Mackenzie]

Hi, Eli!

On Sun, Jun 08, 2008 at 10:25:11PM +0300, Eli Zaretskii wrote:
> > From: Glenn Morris <rgm@gnu.org>
> > Cc: Alan Mackenzie <acm@muc.de>,  Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> > Date: Sun, 08 Jun 2008 15:03:07 -0400

> > Stefan Monnier wrote:

> > > I think it should be easy to generate the dependencies from
> > > bytecomp.el (just record macros as we expand them, and then try and
> > > figure out which file they come from).  This way they could be kept
> > > uptodate 100% automatically.

> > What exactly is the point of doing this supposed to be?
> > To make non-bootstrap builds more reliable?

> To make dependencies between Lips files explicit in lisp/Makefile.in,
> and thus causing Make to compile Lisp files in the right order.  Right
> now, whenever some Lisp files change, I frequently need to "make
> recompile" several times, sometimes removing stale *.elc files
> manually, until it finally succeeds, because it basically compiles them
> in a random order.

OK, let me propose a test of this future dependency generator.  It must
determine that a change to cc-langs.el requires cc-{mode,engine}.el to be
recompiled.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Build-time dependencies

[Richard M Stallman]

Bootstrapping takes a long time on the OLPC.
I don't like the idea of making bootstrapping an unavoidable
part of building Emacs.

BTW, the suggestion I made yesterday also automatically assures
a bootstrap the first time you build from CVS.
CVS will contain only the source file bootstrap,el, not the
compiled file.

Re: Build-time dependencies

[Glenn Morris]

Richard M Stallman wrote:

> Bootstrapping takes a long time on the OLPC.
> I don't like the idea of making bootstrapping an unavoidable
> part of building Emacs.

There have been no changes towards that.

What's being proposed re dependencies is definitely the Right Thing to
do. Personally I view it very much as a wishlist item, but if someone
implements it, great.

It has no influence on _releases_ of Emacs of course. Perhaps people
with slow machines might be better served by the emacs-snapshot
packages provided by some distributions, or by nightly
pre-bootstrapped snapshots on Savannah? But then there are bandwidth
issues I suppose...

A better solution: more frequent releases! :)

Re: Build-time dependencies

[Miles Bader]

Glenn Morris <rgm@gnu.org> writes:
> Perhaps people with slow machines might be better served by the
> emacs-snapshot packages provided by some distributions, or by nightly
> pre-bootstrapped snapshots on Savannah?

Some people with slow machines actually develop Emacs...

-miles

-- 
"1971 pickup truck; will trade for guns"

Re: Build-time dependencies

[Glenn Morris]

Miles Bader wrote:

> Some people with slow machines actually develop Emacs...

I'm assuming they fall into the category I already mentioned of "those
who know what they are doing".

It will be great if one of them indeed implements a smarter recompile.

Re: Build-time dependencies

[Ted Zlatanov]

On Mon, 09 Jun 2008 15:53:16 -0400 Glenn Morris <rgm@gnu.org> wrote: 

GM> It will be great if one of [the smart people] indeed implements a
GM> smarter recompile.

Is SCons a possibility?  I looked at it 3 months ago and really liked
it, but it's supposed to be slow for large projects and uses Python.  It
would probably work all right for Emacs.

Ted

Re: Build-time dependencies

[Stefan Monnier]

GM> It will be great if one of [the smart people] indeed implements a
GM> smarter recompile.

> Is SCons a possibility?  I looked at it 3 months ago and really liked
> it, but it's supposed to be slow for large projects and uses Python.  It
> would probably work all right for Emacs.

It doesn't look like it would help much with the Elisp side of
recompiling, which is the tricky part.


        Stefan