eshell-defgroup. Do we really need this?

eshell-defgroup. Do we really need this?

[Alan Mackenzie]

Hi, emacs,

I've just done a cvs update in CVS head.  I really wish I hadn't.

I tried unadorned 'make'.  This told me "can't find cl-compile-time-init" on an
(include cl).  What????

So back to 'make bootstrap'.  It says "can't find eshell-defgroup".

Hey, guys, 'make bootstrap' should work.

   MAKE BOOTSTRAP SHOULD WORK.

   MAKE BOOTSTRAP SHOULD WORK.

Or, SOMETHING should - the last fallback before shouting naughty words
at high volume.  make bootstrap _is_ that last, guaranteed to work
fallback, isn't it?  Isn't it?  Hey, I'm tired and fed up, and really
ought to be in bed.  I'm hacking off you, the reader.  I can't even
think straight, I feel like I'm drunk, but I've not touched any alcohol
for days.  I'm not even sleep deprived.

Hey, make bootstrap should ALWAYS work, shouldn't it?

A quick 

So, here's another hour lost to bootstrap hell.  This just isn't fun.

A quick find . -name '*.el' | xargs grep eshell-defgroup gives me 
(defalias 'eshell-defgroup 'defgroup) somewhere.  Hey, is this what's
breaking make bootstrap????

WHAT'S WRONG WITH DEFGROUP, that you need to call it eshell-defgroup??


Hey, guys, THIS ISN'T FUN!!!!  I want to hack Emacs.

I know, I've caused this myself before too, so I'm not claiming I'm
better than anybody else.  But I thought we'd got this fixed a few weeks
back.

Why won't our CVS head build NEARLY ALL THE TIME!!!

I wish I hadn't.  It's update hell again, in eshell hell.
Back to update hell, in eshell.

update autogen-clean, as recommended by 

Heck, I'm going to bed.

WE CAN DO BETTER THAN THIS, SURELY?????

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: eshell-defgroup. Do we really need this?

[Ted Zlatanov]

On Tue, 29 Jul 2008 22:27:54 +0000 Alan Mackenzie <acm@muc.de> wrote: 

AM> Why won't our CVS head build NEARLY ALL THE TIME!!!

It's not hard to set up automated builds; I wouldn't use something like
CruiseControl but a simple tool like buildbot or even a shell script
would be nice (cvs up && configure && make bootstrap etc) Failure
notifications can be sent to a central server and more than N (maybe 5?)
failures in the same place would trigger a bug report or some other
notification.

CPAN has a nice way to do this with a whole network of testers, so when
you submit a module failed tests go back to you.

Sorry if this has been discussed before but I didn't see such a thread.

Ted

Re: eshell-defgroup. Do we really need this?

[Lennart Borgman (gmail)]

Ted Zlatanov wrote:
> On Tue, 29 Jul 2008 22:27:54 +0000 Alan Mackenzie <acm@muc.de> wrote: 
> 
> AM> Why won't our CVS head build NEARLY ALL THE TIME!!!
> 
> It's not hard to set up automated builds; I wouldn't use something like
> CruiseControl but a simple tool like buildbot or even a shell script
> would be nice (cvs up && configure && make bootstrap etc) Failure
> notifications can be sent to a central server and more than N (maybe 5?)
> failures in the same place would trigger a bug report or some other
> notification.
> 
> CPAN has a nice way to do this with a whole network of testers, so when
> you submit a module failed tests go back to you.
> 
> Sorry if this has been discussed before but I didn't see such a thread.


It was discussed before, but I do not know what happened then

  http://lists.gnu.org/archive/html/emacs-devel/2008-04/msg01011.html

Re: eshell-defgroup. Do we really need this?

[Ted Zlatanov]

On Thu, 31 Jul 2008 21:18:49 +0200 "Lennart Borgman (gmail)" <lennart.borgman@gmail.com> wrote: 


LB> [automating builds] was discussed before, but I do not know what
LB> happened then

LB>  http://lists.gnu.org/archive/html/emacs-devel/2008-04/msg01011.html

I see it working at

http://build.orebokech.com/

Romain Francoise is the maintainer, it seems.

Romain, can you set up build failures to be aggregated?  I can set up
buildbot (send me your setup if you can) for a confirmation run; if both
our builds fail then the Emacs CVS trunk is probably broken.  If
buildbot doesn't support this, I can write a simple mail filter to
handle it.  The important thing is to send a notice only when the build
breaks (after 15 minutes or so, in case we miss an immediate fix commit)
and another one when it's back.  We definitely don't want e-mails on
every commit.

Ted

Re: eshell-defgroup. Do we really need this?

[Romain Francoise]

Hi Ted,

Ted Zlatanov <tzz@lifelogs.com> writes:

> Romain, can you set up build failures to be aggregated?  I can set
> up buildbot (send me your setup if you can) for a confirmation
> run; if both our builds fail then the Emacs CVS trunk is probably
> broken.

My experience with running this buildbot (and others) suggests that
there is little value in doing this; buildbot does a clean build
every time so if it fails then we can be fairly sure that CVS is
broken.

I'm not sure which platform you were offering to build on, but the
standard way to set up a buildbot is to have a master instance with
one builder per platform, say one for Debian, one for Windows, one
for Mac OS X, etc. The instance at http://emacsbuild.orebokech.com/
only has builders for Debian etch, having more would be nice.

> The important thing is to send a notice only when the build breaks
> (after 15 minutes or so, in case we miss an immediate fix commit)
> and another one when it's back.  We definitely don't want e-mails
> on every commit.

Builds are time-based at the moment: every six hours.  And it
already sends mail only when the state changes, I made that change
at Martin's request (it's not standard buildbot behavior).

While we're on this topic, I should note that a few weeks ago I
changed the configuration of the builders to do an additional build
with --without-x, it breaks often and can go unnoticed for some time
because relatively few people configure Emacs that way.

Re: eshell-defgroup. Do we really need this?

[Ted Zlatanov]

On Fri, 01 Aug 2008 19:08:04 +0200 Romain Francoise <romain@orebokech.com> wrote: 

RF> Hi Ted,
RF> Ted Zlatanov <tzz@lifelogs.com> writes:

>> Romain, can you set up build failures to be aggregated?  I can set
>> up buildbot (send me your setup if you can) for a confirmation
>> run; if both our builds fail then the Emacs CVS trunk is probably
>> broken.

RF> My experience with running this buildbot (and others) suggests that
RF> there is little value in doing this; buildbot does a clean build
RF> every time so if it fails then we can be fairly sure that CVS is
RF> broken.

You think so even considering the large amount of people that would get
this report?  I'd rather be cautious and have at least one confirmation
of the failure before reporting it.  But, of course, it's your
choice--as long as we report something.

RF> I'm not sure which platform you were offering to build on, but the
RF> standard way to set up a buildbot is to have a master instance with
RF> one builder per platform, say one for Debian, one for Windows, one
RF> for Mac OS X, etc. The instance at http://emacsbuild.orebokech.com/
RF> only has builders for Debian etch, having more would be nice.

I can set mine up on Ubuntu 6.x and 7.x and MacOS X.  Send me your
buildbot config so I know we're doing the same things.  It's probably
good to publicize these, actually, so others can set up Emacs buildbots
easily.  Maybe the Emacs wiki would be a good place to post the config?

RF> Builds are time-based at the moment: every six hours.  And it
RF> already sends mail only when the state changes, I made that change
RF> at Martin's request (it's not standard buildbot behavior).

Great :)

RF> While we're on this topic, I should note that a few weeks ago I
RF> changed the configuration of the builders to do an additional build
RF> with --without-x, it breaks often and can go unnoticed for some time
RF> because relatively few people configure Emacs that way.

Excellent.  I'll do --with-ns, --without-x, and --with-x on MacOS X and
just the latter two on the Ubuntu systems.

Ted

Re: eshell-defgroup. Do we really need this?

[Stephen J. Turnbull]

Ted Zlatanov writes:
 > On Fri, 01 Aug 2008 19:08:04 +0200 Romain Francoise <romain@orebokech.com> wrote: 

 > RF> My experience with running this buildbot (and others) suggests that
 > RF> there is little value in doing this; buildbot does a clean build
 > RF> every time so if it fails then we can be fairly sure that CVS is
 > RF> broken.
 > 
 > You think so even considering the large amount of people that would get
 > this report?  I'd rather be cautious and have at least one confirmation
 > of the failure before reporting it.  But, of course, it's your
 > choice--as long as we report something.

Romain's right, you don't need confirmation.  If a clean build breaks,
it's broke.  What to do about it is another question.

XEmacs has a separate list for build reports, whether user-contributed
or automatic.  From Richard's comments about the BTS, I'd put money on
him wanting a separate list for this, too.  (That's 'cause I really
like the odds, not because I speak for Richard.)  Works for us.  (We
don't use buildbot, yet.)

Python core just assumes that people (and in particular the release
engineers) will be watching the buildbot's waterfall URL.  Works for
them.

Python also has a system of "community" (ie, apps written in Python)
buildbots with the intent of notifying somebody that the dev lines of
Python are breaking stable builds.  Current status is "failing
miserably", as nobody pays attention to them.  That is For reasons
that I don't think apply to Emacs, but for the sake of completeness I
include the case here.

Re: eshell-defgroup. Do we really need this?

[Ted Zlatanov]

On Sat, 02 Aug 2008 13:18:15 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> On Fri, 01 Aug 2008 19:08:04 +0200 Romain Francoise <romain@orebokech.com> wrote: 

RF> My experience with running this buildbot (and others) suggests that
RF> there is little value in doing this; buildbot does a clean build
RF> every time so if it fails then we can be fairly sure that CVS is
RF> broken.
>> 
>> You think so even considering the large amount of people that would get
>> this report?  I'd rather be cautious and have at least one confirmation
>> of the failure before reporting it.  But, of course, it's your
>> choice--as long as we report something.

SJT> Romain's right, you don't need confirmation.  If a clean build breaks,
SJT> it's broke.  What to do about it is another question.

Builds can break for many reasons, some local (e.g. disk full).  Why
bother many people with a false report?  It would condition them to
ignore truly broken builds.  That's my concern.

SJT> XEmacs has a separate list for build reports, whether user-contributed
SJT> or automatic.  From Richard's comments about the BTS, I'd put money on
SJT> him wanting a separate list for this, too.  (That's 'cause I really
SJT> like the odds, not because I speak for Richard.)  Works for us.  (We
SJT> don't use buildbot, yet.)

SJT> Python core just assumes that people (and in particular the release
SJT> engineers) will be watching the buildbot's waterfall URL.  Works for
SJT> them.

SJT> Python also has a system of "community" (ie, apps written in Python)
SJT> buildbots with the intent of notifying somebody that the dev lines of
SJT> Python are breaking stable builds.  Current status is "failing
SJT> miserably", as nobody pays attention to them.  That is For reasons
SJT> that I don't think apply to Emacs, but for the sake of completeness I
SJT> include the case here.

Thanks for explaining.  I think the case for Emacs is simpler, but I am
certainly not the one to make any decisions about it.  I just want to
provide the service because I want to do something about broken builds.

Ted

Re: eshell-defgroup. Do we really need this?

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > SJT> Romain's right, you don't need confirmation.  If a clean build breaks,
 > SJT> it's broke.  What to do about it is another question.
 > 
 > Builds can break for many reasons, some local (e.g. disk full).  Why
 > bother many people with a false report?

Because they don't happen much in practice on well-maintained 'bots,
which is what you want.

 > It would condition them to ignore truly broken builds.

Excuse me, but isn't the problem that they already do??  (Yes, I know
that's specious.  It's still true.  Fix the bigger problem first. :-)

"Snappy Answers" aside, and acknowledging that broken builds are taken
very seriously by the maintainers, there's a real problem in the Emacs
build system.  XEmacs and SXEmacs see *way* fewer "broken build"
reports, and when we do, the response is almost always that the
responsible developer pipes up with "oops, my bad, fixed" within 24
hours.  I've *never* seen the kind of "Did you wait until the goat
died?  You can't start the build before the sacrificial goat is dead!" 
threads that are so common on emacs-devel.

 > That's my concern.

There are sufficient broken builds in Emacs that that is not a worry.
If there is a 'bot spewing because of disk full, sentence the 'bot
owner to some public service like reading the entire Collected Works
of Richard Stallman (including the source code to all his programs)
out loud at the main gate of Microsoft.

If and when the rate of disk full reports reaches 10% of the rate of
genuine breakage, start forwarding them as bug reports to buildbot.

Also, it shouldn't be hard to construct a filter that recognizes
such and pings the 'bot owner.  If you have access to the Mailman
pipeline, it can easily be installed in the list config (ie, without
risk to other Mailman lists) and set up to ping only interested
parties, and not forward it to the list.

Re: eshell-defgroup. Do we really need this?

[Ted Zlatanov]

On Mon, 4 Aug 2008 17:49:01 +0000 (UTC) "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
SJT> Romain's right, you don't need confirmation.  If a clean build breaks,
SJT> it's broke.  What to do about it is another question.
>> 
>> Builds can break for many reasons, some local (e.g. disk full).  Why
>> bother many people with a false report?

SJT> Because they don't happen much in practice on well-maintained 'bots,
SJT> which is what you want.

I guess I come from a background of sysadmin, where things that can go
wrong will, so I'd rather not assume this.  I've had enough experience
with "this should never happen" happening at 3 AM.

>> It would condition them to ignore truly broken builds.

SJT> Excuse me, but isn't the problem that they already do??  (Yes, I know
SJT> that's specious.  It's still true.  Fix the bigger problem first. :-)

No.  You're confusing issues.  What I'm trying to help provide is a
proactive mechanism.  The current broken build detection is reactive:
users report busted builds or developers find out when they do a build.
So broken builds just don't emerge as a problem quickly.

SJT> XEmacs and SXEmacs see *way* fewer "broken build" reports, and when
SJT> we do, the response is almost always that the responsible developer
SJT> pipes up with "oops, my bad, fixed" within 24 hours.  I've *never*
SJT> seen the kind of "Did you wait until the goat died?  You can't
SJT> start the build before the sacrificial goat is dead!"  threads that
SJT> are so common on emacs-devel.

Well, maybe that will change when we can identify the change that broke
the builds.  I am not trying to change social dynamics, regardless.
It's neither my target nor my interest, and the Emacs maintainers should
address that side of the process.

SJT> If there is a 'bot spewing because of disk full, sentence the 'bot
SJT> owner to some public service like reading the entire Collected Works
SJT> of Richard Stallman (including the source code to all his programs)
SJT> out loud at the main gate of Microsoft.

SJT> If and when the rate of disk full reports reaches 10% of the rate of
SJT> genuine breakage, start forwarding them as bug reports to buildbot.

I was giving an example.  By definition you can't anticipate every
failure mode, so I'd rather not assume builds will always work.  Here's
some other causes: transient network outage, missing/broken libraries,
misconfiguration, race conditions, OS limitations, ACLs, memory/CPU
usage limits, swap space/process table/memory exhaustion, power outage,
filesystem corruption...  I could go on, but the point is a broken build
from a single system can be caused by too many factors external to the
build process.

SJT> Also, it shouldn't be hard to construct a filter that recognizes
SJT> such and pings the 'bot owner.  If you have access to the Mailman
SJT> pipeline, it can easily be installed in the list config (ie, without
SJT> risk to other Mailman lists) and set up to ping only interested
SJT> parties, and not forward it to the list.

My suggestion was to look for 5 or more broken build reports from
buildbots in the community.  I think that's better than the workarounds
you suggest, because it's not dependent on anything other than agreement
between separate systems.  Your solution recognizes potential failure
modes *after* they occur and works backwards after the annoyance has
already happened.

Ted

Re: eshell-defgroup. Do we really need this?

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > I guess I come from a background of sysadmin, where things that can go
 > wrong will, so I'd rather not assume this.  I've had enough experience
 > with "this should never happen" happening at 3 AM.

That's just an argument for never doing any testing, since *any* test
could fail due to a flaky memory chip.

 > What I'm trying to help provide is a proactive mechanism.

Then look elsewhere than buildbot, which is just an attempt to speed
up the reaction.  A proactive solution would be to convince the Scons
people to join GNU.<wink>

 > SJT> XEmacs and SXEmacs see *way* fewer "broken build" reports, and when
 > SJT> we do, the response is almost always that the responsible developer
 > SJT> pipes up with "oops, my bad, fixed" within 24 hours.  I've *never*
 > SJT> seen the kind of "Did you wait until the goat died?  You can't
 > SJT> start the build before the sacrificial goat is dead!"  threads that
 > SJT> are so common on emacs-devel.
 > 
 > Well, maybe that will change when we can identify the change that broke
 > the builds.  I am not trying to change social dynamics, regardless.

That wasn't a diagnosis, that was a description of a painful symptom.
We've got a patient with a migraine, and he's *not* holding a hammer.

That is, this thread was occasioned by breakage that happened to *one*
person, and we do not yet know what caused that problem.  Since the
details the OP has since given "shouldn't happen" the maintainers are
almost certainly going to table the matter and wait for more
evidence.  The buildbots won't help with that.

What buildbots can do is point out the easy ones quickly.  Note that
Alan used several methods to ensure a "clean" build; none of them
worked.  If there were a buildbot for his platform and configuration,
then you could say, "OK, bot is green, so let's see what is 'dirty'
about your environment."

 > My suggestion was to look for 5 or more broken build reports from
 > buildbots in the community.

I think you vastly overestimate the number of buildbots that will be
forthcoming.

 > I could go on, but the point is a broken build from a single system
 > can be caused by too many factors external to the build process.

Surely you can distinguish between the number of ways to go wrong and
the probability of going wrong?  The evidence I've seen in the Python
project is that nobody has ever complained in about 2 years of running
buildbots of spurious reports.  The vast majority of red bots are bugs
in the Python build or regressions.

buildbots (was: eshell-defgroup. Do we really need this?)

[Ted Zlatanov]

(subject changed, sorry I didn't do it sooner)

On Tue, 05 Aug 2008 17:20:00 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> I guess I come from a background of sysadmin, where things that can go
>> wrong will, so I'd rather not assume this.  I've had enough experience
>> with "this should never happen" happening at 3 AM.

SJT> That's just an argument for never doing any testing, since *any* test
SJT> could fail due to a flaky memory chip.

My point is simple: redundant testing reduces the chance of false
positives.  How is anticipating a system failure an argument for never
testing?  I can't follow your reasoning, sorry.

>> What I'm trying to help provide is a proactive mechanism.

SJT> Then look elsewhere than buildbot, which is just an attempt to speed
SJT> up the reaction.  A proactive solution would be to convince the Scons
SJT> people to join GNU.<wink>

I think you're mistaking the automated *process* with the tools that
implement it.  I want to provide the former, and don't care about the
latter (though buildbot seems easiest to set up).

SJT> That is, this thread was occasioned by breakage that happened to *one*
SJT> person, and we do not yet know what caused that problem.  Since the
SJT> details the OP has since given "shouldn't happen" the maintainers are
SJT> almost certainly going to table the matter and wait for more
SJT> evidence.  The buildbots won't help with that.

Buildbots provide independent verification of breakage.  We will have a
testing baseline, a date when things broke, and the knowledge that
things used to work before that date.  User reports can't provide all
three with the same degree of assurance (if at all).

>> My suggestion was to look for 5 or more broken build reports from
>> buildbots in the community.

SJT> I think you vastly overestimate the number of buildbots that will be
SJT> forthcoming.

It works for CPAN testers, why not for Emacs?  I can contribute 3
buildbots easily, and I'm sure others can do it too.  5 is a good number
but it can go down to 2 if needed.  The point is to avoid false positives.

>> I could go on, but the point is a broken build from a single system
>> can be caused by too many factors external to the build process.

SJT> Surely you can distinguish between the number of ways to go wrong and
SJT> the probability of going wrong?  The evidence I've seen in the Python
SJT> project is that nobody has ever complained in about 2 years of running
SJT> buildbots of spurious reports.  The vast majority of red bots are bugs
SJT> in the Python build or regressions.

You mentioned the situation with Python is different.  I think it's a
worthwhile experiment in the Emacs community.

Stefan or Chong, can you suggest a place to send the build failure
reports?

Ted

buildbots (was: eshell-defgroup. Do we really need this?)

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > My point is simple: redundant testing reduces the chance of false
 > positives.  How is anticipating a system failure an argument for never
 > testing?  I can't follow your reasoning, sorry.

Redundant testing also increases the chance of false negatives; those
opposed effects go hand in hand.  I see no reason to suppose that
false positives are more harmful than false negatives at this stage,
quite the reverse, that's all.

place to send build failure reports? (was: buildbots)

[Ted Zlatanov]

On Tue, 05 Aug 2008 08:57:12 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> Stefan or Chong, can you suggest a place to send the build failure
TZ> reports?

Ping...  The note was buried at the end of a message last time :)

Ted

Re: place to send build failure reports?

[Chong Yidong]

Ted Zlatanov <tzz@lifelogs.com> writes:

> TZ> Stefan or Chong, can you suggest a place to send the build failure
> TZ> reports?
>
> Ping...  The note was buried at the end of a message last time :)

I would suggest sending it to the private email of whoever volunteers to
handle this.  Then, when a build fails, that volunteer would forward the
email to emacs-devel.

(The human intervention is to prevent unnecessary spam from
malfunctioning buildbots, repeated build failures during tree merges,
etc.)

Re: eshell-defgroup. Do we really need this?

[Romain Francoise]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I can set mine up on Ubuntu 6.x and 7.x and MacOS X.  [...]  I'll
> do --with-ns, --without-x, and --with-x on MacOS X and just the
> latter two on the Ubuntu systems.

There seems to be some confusion here: are you proposing to set up a
new master instance, or to provide build slaves for my master?

If the former, your buildbot will live at a separate URLs, which may
not be ideal.
If the latter, the commands are defined by the master and you don't
need any config file.

Actually, now that I think about it, we should probably redo the
configuration from scratch and have separate masters for the trunk
and the 22 branch, as Python does...

Re: eshell-defgroup. Do we really need this?

[Ted Zlatanov]

On Sun, 03 Aug 2008 21:02:07 +0200 Romain Francoise <romain@orebokech.com> wrote: 

RF> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I can set mine up on Ubuntu 6.x and 7.x and MacOS X.  [...]  I'll
>> do --with-ns, --without-x, and --with-x on MacOS X and just the
>> latter two on the Ubuntu systems.

RF> There seems to be some confusion here: are you proposing to set up a
RF> new master instance, or to provide build slaves for my master?

New master.

RF> If the latter, the commands are defined by the master and you don't
RF> need any config file.

I want to use the commands you use for your master, to ensure we are
consistently testing the same things.

RF> Actually, now that I think about it, we should probably redo the
RF> configuration from scratch and have separate masters for the trunk
RF> and the 22 branch, as Python does...

Works for me too.

Ted

Progress: eshell-defgroup. Do we really need this?

[Alan Mackenzie]

Hi, Emacs!

On Tue, Jul 29, 2008 at 10:27:54PM +0000, Alan Mackenzie wrote:
> Hi, emacs,

> So back to 'make bootstrap'.  It says "can't find eshell-defgroup".

make bootstrap is generating loaddefs.el with lines like

    (eshell-defgroup eshell-alias nil "Docstring" :tag "Command aliases" :gr\oup 'eshell-module)

, and AFTER them the line which should have come first:

    (defalias 'eshell-defgroup 'defgroup)

.  It seems that the loaddefs.el generator first builds these
(eshell-defgroup....) lines in eshell/esh-group.el, and then, somehow,
copies them to loaddefs.el.  The code isn't easy to understand.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Progress: eshell-defgroup. Do we really need this?

[Stefan Monnier]

>> So back to 'make bootstrap'.  It says "can't find eshell-defgroup".

> make bootstrap is generating loaddefs.el with lines like

>     (eshell-defgroup eshell-alias nil "Docstring" :tag "Command aliases" :gr\oup 'eshell-module)

> , and AFTER them the line which should have come first:

>     (defalias 'eshell-defgroup 'defgroup)

> .  It seems that the loaddefs.el generator first builds these
> (eshell-defgroup....) lines in eshell/esh-group.el, and then, somehow,
> copies them to loaddefs.el.  The code isn't easy to understand.

Those "(eshell-defgroup" should not be in loaddefs.el but in
lisp/eshell/esh-groups.el.  This is done by autoload.el, so either we
have a bug in autoload.el that sometimes makes it ignore the

  generated-autoload-file: "esh-groups.el"

in the files, or there's some other autoload.el that gets in the way.


        Stefan