Problem with building Emacs diagnosed. Help please!

Problem with building Emacs diagnosed. Help please!

[Alan Mackenzie]

Hi, all!

As I've said once or twice recently, whilst 'make bootstrap'ping, I've
been getting the error message

    Loading loaddefs.el (source)...
    Symbol's function definition is void: eshell-defgroup
    make[1]: *** [emacs] Error 255
    make[1]: Leaving directory `/home/acm/emacs/emacs/src'
    make: *** [src] Error 2

.  I've diagnosed now what is going on.  The pertinent lines of
loaddefs.el look like this:

########################################################################
;;;### (autoloads nil "em-alias" "eshell/em-alias.el" (18494 62417))
;;; Generated autoloads from eshell/em-alias.el

(eshell-defgroup eshell-alias nil "Command aliases allow for easy definition of alternate commands." :tag "Command aliases" :gr\oup 'eshell-module)   ;  <======================================================================

;;;***

<plus several more of the like>
.....
.....

;;;### (autoloads (eshell-command-result eshell-command eshell) "eshell"
;;;;;;  "eshell/eshell.el" (18578 7465))
;;; Generated autoloads from eshell/eshell.el

(defalias 'eshell-defgroup 'defgroup)     ; <========================================================================
########################################################################

, i.e. the problem is that `eshell-defgroup' is defined after it is used.

As far as I can make out (in .../lisp/emacs-lisp/autoload.el), when the
autoload processing is done, the order in which elisp files are processed
is random (for some value of "random").  So having elisp files set up so
that they depend on the processing order is a Bad Thing.

Greg, when you changed the eshell source to use `eshell-defgroup' instead
of just plain old `defgroup' (to which it is aliased anyway) on 21st May
2008, was there some overwhelmingly important reason?  It looks like you
intend (or intended?) to add some extra functionality to
`eshell-defgroup' at some stage.  Could that, perhaps, be achieved some
other way?

However, given that we've passed the deadline for new features, I would
like to revert that change for Emacs 23, removing `eshell-defgroup' so as
to firm up the build process.

Perhaps I should also amend the Elisp manual to warn against dependencies
between Elisp files in the autoload bits.

Maybe, at some stage, we could introduce a new construct, something like

    ;;;###autoload-requires eshell.el

, but my gut feeling is that this would be unnecessary complexity.

What do people think?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Problem with building Emacs diagnosed. Help please!

[Stefan Monnier]

> As far as I can make out (in .../lisp/emacs-lisp/autoload.el), when the
> autoload processing is done, the order in which elisp files are processed
> is random (for some value of "random").

IIRC there's some effort to try and keep it lexicographically ordered,
but IIRC it's not 100% reliable.

> So having elisp files set up so
> that they depend on the processing order is a Bad Thing.

Agreed.  And relying on the lexicographic ordering is not a good
idea either.

> However, given that we've passed the deadline for new features, I would
> like to revert that change for Emacs 23, removing `eshell-defgroup' so as
> to firm up the build process.

Sounds good to me.

> Maybe, at some stage, we could introduce a new construct, something like
>     ;;;###autoload-requires eshell.el
> , but my gut feeling is that this would be unnecessary complexity.

Agreed.

> What do people think?

Thanks for tracking it down.


        Stefan

Re: Problem with building Emacs diagnosed. Help please!

[Glenn Morris]

Alan Mackenzie wrote:

> As I've said once or twice recently, whilst 'make bootstrap'ping, I've
> been getting the error message

Has anyone else seen this? Do you see it in a fresh checkout?
It works for me, and I don't see that there is anything wrong here.
Sorry you're having problems.

>     Loading loaddefs.el (source)...
>     Symbol's function definition is void: eshell-defgroup

eshell-defgroup does not appear in my loaddefs.el.

>     make[1]: *** [emacs] Error 255
>     make[1]: Leaving directory `/home/acm/emacs/emacs/src'
>     make: *** [src] Error 2
>
> .  I've diagnosed now what is going on.  The pertinent lines of
> loaddefs.el look like this:
>
> ########################################################################
> ;;;### (autoloads nil "em-alias" "eshell/em-alias.el" (18494 62417))
> ;;; Generated autoloads from eshell/em-alias.el
>
> (eshell-defgroup eshell-alias nil "Command aliases allow for easy definition of alternate commands." :tag "Command aliases" :gr\oup 'eshell-module)   ;  <======================================================================
>
> ;;;***
>
> <plus several more of the like>

I'll assume you meant esh-groups.el here, not loaddefs.el.

> (defalias 'eshell-defgroup 'defgroup)     ; <========================================================================
> ########################################################################
>
> , i.e. the problem is that `eshell-defgroup' is defined after it is used.

eshell-defgroup is not autoloaded. It is defined in eshell.el, which
all the files that use it require. The above line you quote should not
be in any generated file, not is it for me. I think your build is
messed up.

> is random (for some value of "random").  So having elisp files set up so
> that they depend on the processing order is a Bad Thing.

They are not so set up.

> Greg, when you changed the eshell source to use `eshell-defgroup' instead
> of just plain old `defgroup' (to which it is aliased anyway) on 21st May
> 2008, was there some overwhelmingly important reason? 

I can't speak for Greg, but I made the change for a reason, yes. To
replace homegrown eshell code with the normal autoload mechanism.
Autoload treats defgroup specially, eshell needs the whole definition
in esh-groups, hence the alias subterfuge. I believe it works fine.

> It looks like you intend (or intended?) to add some extra
> functionality to `eshell-defgroup' at some stage.

No, it works as is.

> However, given that we've passed the deadline for new features, I would
> like to revert that change for Emacs 23, removing `eshell-defgroup' so as
> to firm up the build process.

I think your build tree is messed up somehow.

> Perhaps I should also amend the Elisp manual to warn against dependencies
> between Elisp files in the autoload bits.

I think you have misdiagnosed the problem. Any problem that might
exist must be due to the build not starting from a clean state for
some reason.

Sorry, I'm on vacation and it's hard for me to investigate.

Re: Problem with building Emacs diagnosed. Help please!

[Glenn Morris]

Actually, you would get this problem if those eshell autoloads went
into loaddefs,el, as you say they are (?). But they should be going
into eshell/esh-groups.el, as per the file-specific value of
generated-autoload-file. Dunno why that's being ignored in your build.

Re: Problem with building Emacs diagnosed. Help please!

[Alan Mackenzie]

Hi, Glenn,

On Sun, Aug 03, 2008 at 02:59:05PM -0400, Glenn Morris wrote:
> Alan Mackenzie wrote:

> > As I've said once or twice recently, whilst 'make bootstrap'ping, I've
> > been getting the error message

> Has anyone else seen this? Do you see it in a fresh checkout?

A fresh checkout builds "OK", for the following value of "OK": After
doing ./configure and make bootstrap, the build crashes with
(presumably) a segmentation fault (it said "error 6", or something
similar).  After doing "make" ~10 times, emacs was built.  At each
attempt, it crashed with a seg fault.  I think there's a memory leak
here, but I've said that already.

> It works for me, and I don't see that there is anything wrong here.
> Sorry you're having problems.

Thanks!

> >     Loading loaddefs.el (source)...
> >     Symbol's function definition is void: eshell-defgroup

> eshell-defgroup does not appear in my loaddefs.el.

It doesn't appear in my new-checkout build, but it does in my usual
Emacs-23, in respect of files
em-{alias,banner,basic,cmpl,dirs,glob,hist,ls,pred,prompt,rebind,script,smart,term,unix,xtra}.el.
In fact, that's pretty much every lisp file which has ;;;###autoloads.

Somehow, autoload.el is failing to redirect the autoloads to the correct
output file (here, eshell/esh-groups.el).

[ ..... ]

> > , i.e. the problem is that `eshell-defgroup' is defined after it is used.

> eshell-defgroup is not autoloaded. It is defined in eshell.el, which
> all the files that use it require. The above line you quote should not
> be in any generated file, not is it for me. I think your build is
> messed up.

Apologies.  I inserted ";;;###autoload" on eshell-defgroup myself, in an
attempt to get things working, then forgot about it and got thoroughly
confused.

> > Glenn, when you changed the eshell source to use `eshell-defgroup'
> > instead of just plain old `defgroup' (to which it is aliased anyway)
> > on 21st May 2008, was there some overwhelmingly important reason? 

> ... when I made the change for a reason, yes. To replace homegrown
> eshell code with the normal autoload mechanism.  Autoload treats
> defgroup specially, eshell needs the whole definition in esh-groups,
> hence the alias subterfuge. I believe it works fine.

Ah, right, I see.

> I think your build tree is messed up somehow.

> I think you have misdiagnosed the problem. Any problem that might
> exist must be due to the build not starting from a clean state for
> some reason.

Yes.  But I'm so sick and fed up of not being able to build routinely, I
am going to track down this problem even if it takes me weeks to do it.
I am not prepared to check out the sources from scratch every time I
can't get a build working.

Refactoring autoload.el might be a good first move.

> Sorry, I'm on vacation and it's hard for me to investigate.

Thanks for taking the time to put me right!  I hope you're enjoying it!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Problem with building Emacs diagnosed. Help please!

[Chong Yidong]

Alan Mackenzie <acm@muc.de> writes:

> A fresh checkout builds "OK", for the following value of "OK": After
> doing ./configure and make bootstrap, the build crashes with
> (presumably) a segmentation fault (it said "error 6", or something
> similar).  After doing "make" ~10 times, emacs was built.  At each
> attempt, it crashed with a seg fault.  I think there's a memory leak
> here, but I've said that already.

That's very strange.  I've never seen anything like this on my machine
(x86_64-unknown-linux-gnu, GTK+ Version 2.12.9).  Could you post your
system specs?

Re: Problem with building Emacs diagnosed. Help please!

[Alan Mackenzie]

On Mon, Aug 04, 2008 at 02:45:29PM -0400, Chong Yidong wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > A fresh checkout builds "OK", for the following value of "OK": After
> > doing ./configure and make bootstrap, the build crashes with
> > (presumably) a segmentation fault (it said "error 6", or something
> > similar).  After doing "make" ~10 times, emacs was built.  At each
> > attempt, it crashed with a seg fault.  I think there's a memory leak
> > here, but I've said that already.

> That's very strange.  I've never seen anything like this on my machine
> (x86_64-unknown-linux-gnu, GTK+ Version 2.12.9).  Could you post your
> system specs?

Athlon 1.2 GHz, 768 Mb RAM.
Swap partition: ~2 Gb.

Kernel:           Linux acm 2.6.8 #7 Wed May 23 18:12:53 BST 2007 i686 GNU/Linux
gcc --version:    gcc (GCC) 3.3.5 (Debian 1:3.3.5-13)

Emacs configuration:  ./configure --with-gif=no --with-tiff=no

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Problem with building Emacs diagnosed. Help please!

[Glenn Morris]

Alan Mackenzie wrote:

> I am not prepared to check out the sources from scratch every time I
> can't get a build working.

I didn't mean that. I meant that you seem at the moment to have gotten
into a very weird state somehow, and rather than trying to figure it
out, it would be easier to throw it away and start again. If you get
into the weird state again, then it makes more sense to investigate
it. But for some hopefully transient, never-repeatable glitch, why
bother?

(Perhaps the "emacs" that your build invokes to make the autoloads is
messed up somehow, or it finds the wrong one somehow.)

I update and build from CVS every day and almost never see build
failures, and they are always rapidly fixed. I'm sure the same is true
for others.

Re: Problem with building Emacs diagnosed. Help please!

[Lennart Borgman (gmail)]

Glenn Morris wrote:
> Alan Mackenzie wrote:
> 
>> I am not prepared to check out the sources from scratch every time I
>> can't get a build working.
> 
> I didn't mean that. I meant that you seem at the moment to have gotten
> into a very weird state somehow, and rather than trying to figure it
> out, it would be easier to throw it away and start again. If you get
> into the weird state again, then it makes more sense to investigate
> it. But for some hopefully transient, never-repeatable glitch, why
> bother?
> 
> (Perhaps the "emacs" that your build invokes to make the autoloads is
> messed up somehow, or it finds the wrong one somehow.)
> 
> I update and build from CVS every day and almost never see build
> failures, and they are always rapidly fixed. I'm sure the same is true
> for others.

I have a problem right now that I hope is solved, but I am not sure. 
Since I have quite a lot of patches to Emacs I can't just download the 
sources again. I tried bootstrap-clean, but it did not seem to help. Now 
I have manually deleted diary-loaddefs.el (or what it was called).

It would be nice if there were a file somewhere in the distribution that 
explains what to do in strange situations. No need to write it, but 
every time something stranges pops up and gets solved a note could be 
added. (Of course the notes must be reviewed now and then to see if they 
are still valid, but I still think it could save time.)

Re: Problem with building Emacs diagnosed. Help please!

[Alan Mackenzie]

Hi, everyone!

On Fri, Aug 08, 2008 at 06:53:39PM -0400, Glenn Morris wrote:
> Alan Mackenzie wrote:

> > I am not prepared to check out the sources from scratch every time I
> > can't get a build working.

> I didn't mean that. I meant that you seem at the moment to have gotten
> into a very weird state somehow, and rather than trying to figure it
> out, it would be easier to throw it away and start again.

OK, I _was_ in a wierd state, and I've now tracked it down and fixed it.
The wierdness was a broken `hack-local-variables', which had the effect
that the setting of generated-autoload-file in the eshell/*.el files was
being ignored.  I'd hacked hack-local-variables a bit in the recent past,
but it was Yidong who committed the change, slightly altered.  My fault
for not being more careful!

In tracking down that problem, I've refactored autoload.el quite a bit.
I think it'd be worth committing, since it's simpler than the current
version.  I'll post it soon.

> If you get into the weird state again, then it makes more sense to
> investigate it. But for some hopefully transient, never-repeatable
> glitch, why bother?

Because I get build problems nearly every time I update.  Even though
I've managed to build Emacs now, I'm still getting seg faults on every
build.  I can just restart a build with 'make' a few times, and it gets
to the end in the end.  I've reached a point where I just can't take the
constant uncertainty and unreliability I see.  So I am going to get into
the innards of this, no matter how painful, until I can sort out build
failures rapidly and with only moderate pain.

> I update and build from CVS every day and almost never see build
> failures, and they are always rapidly fixed. I'm sure the same is true
> for others.

I update only every few weeks.  That's probably got a lot to do with it.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Problem with building Emacs diagnosed. Help please!

[Lennart Borgman (gmail)]

Alan Mackenzie wrote:
>> I update and build from CVS every day and almost never see build
>> failures, and they are always rapidly fixed. I'm sure the same is true
>> for others.
> 
> I update only every few weeks.  That's probably got a lot to do with it.

Some time ago I noticed this trouble and decided to do checkouts very 
often - without building. That helped quite a bit.