build broken: no defun org-float-time. Who's guilty, and what does he propose?

build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Alan Mackenzie]

Hi, Emacs,

I've just cvs updated, and the build breaks with:

  org/org-ascii.el:29:1:Error: Symbol's function definition is void: org-float-time

Could the person who's responsible please fix this.

Here we go again.  The endless tread mill of the broken build.  It feels
like being spat at.  Sorry to be so "unhelpful", but I've got no desire
or energy to keep debugging broken builds, or to keep "try make
bootstrap"ing.  I'm going through a very bad patch in my personal life,
and I just don't have the energy any more, and I'm not prepared to do it.

"cvs update" followed by "make" MUST WORK NEARLY ALL THE TIME.
Alternatively "bzr update", "make" working nearly all the time would be
OK.

This is a serious process bug we have, we've had for years, and we MUST
fix.  If it's not fixed soon, I'm just going to give up on Emacs
maintenance and bid the project goodbye.  The pain level is too high.

Sorry, and all that.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Andreas Schwab]

Alan Mackenzie <acm@muc.de> writes:

> I've just cvs updated, and the build breaks with:
>
>   org/org-ascii.el:29:1:Error: Symbol's function definition is void: org-float-time

I cannot reproduce that.  Please make sure you don't have any outdated
elc file around.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[joakim]

Alan Mackenzie <acm@muc.de> writes:

> Hi, Emacs,
>
> I've just cvs updated, and the build breaks with:
>
>   org/org-ascii.el:29:1:Error: Symbol's function definition is void: org-float-time
>
> Could the person who's responsible please fix this.
>
> Here we go again.  The endless tread mill of the broken build.  It feels
> like being spat at.  Sorry to be so "unhelpful", but I've got no desire
> or energy to keep debugging broken builds, or to keep "try make
> bootstrap"ing.  I'm going through a very bad patch in my personal life,
> and I just don't have the energy any more, and I'm not prepared to do it.
>
> "cvs update" followed by "make" MUST WORK NEARLY ALL THE TIME.
> Alternatively "bzr update", "make" working nearly all the time would be
> OK.
>
> This is a serious process bug we have, we've had for years, and we MUST
> fix.  If it's not fixed soon, I'm just going to give up on Emacs
> maintenance and bid the project goodbye.  The pain level is too high.

I've been interested in setting up an automatic build test environment
for Emacs for some time. It would basically checkout the emacs sources
on every checkin, build the sources and report the build
quality(probably with some existing continuous integration software).

I would also store a TB or something of past builds.

Would that be of any help to you?


> Sorry, and all that.
-- 
Joakim Verona

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Lennart Borgman]

On Mon, Sep 7, 2009 at 11:52 AM, <joakim@verona.se> wrote:

> I've been interested in setting up an automatic build test environment
> for Emacs for some time. It would basically checkout the emacs sources
> on every checkin, build the sources and report the build
> quality(probably with some existing continuous integration software).

Please set it up.

Would this test build also run unit tests for Emacs?

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[joakim]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Mon, Sep 7, 2009 at 11:52 AM, <joakim@verona.se> wrote:
>
>> I've been interested in setting up an automatic build test environment
>> for Emacs for some time. It would basically checkout the emacs sources
>> on every checkin, build the sources and report the build
>> quality(probably with some existing continuous integration software).
>
> Please set it up.
>
> Would this test build also run unit tests for Emacs?

I dont think we have any unit tests in the codebase right?
If we do I would expect them to run during the build.

Heres some issues I see:

- I have lots of disk but just ADSL uprate. Should be enough if you just
want to quickly see the build quality history. Possibly not quite enough
if you want to bisect some bug by downloading pre-made binaries and test
them. 

- I dont really know how to trigger builds from cvs checkins. I guess I
will have to poll now and then.

I'm mostly familiar with Java based continuous integration tools, where
you normaly get a lot of features like these out of the box.

-- 
Joakim Verona

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Jan D.]

I think most problems comes from doing
cvs update
and then you get old .elc files left behind or you need to do
make bootstrap
again.

A make that removed old .elc files and did bootstrap when needed is  
the ideal solution. The second part is probably very hard.

     Jan D.

7 sep 2009 kl. 11.52 skrev joakim@verona.se:

> Alan Mackenzie <acm@muc.de> writes:
>
>> Hi, Emacs,
>>
>> I've just cvs updated, and the build breaks with:
>>
>>  org/org-ascii.el:29:1:Error: Symbol's function definition is void:  
>> org-float-time
>>
>> Could the person who's responsible please fix this.
>>
>> Here we go again.  The endless tread mill of the broken build.  It  
>> feels
>> like being spat at.  Sorry to be so "unhelpful", but I've got no  
>> desire
>> or energy to keep debugging broken builds, or to keep "try make
>> bootstrap"ing.  I'm going through a very bad patch in my personal  
>> life,
>> and I just don't have the energy any more, and I'm not prepared to  
>> do it.
>>
>> "cvs update" followed by "make" MUST WORK NEARLY ALL THE TIME.
>> Alternatively "bzr update", "make" working nearly all the time  
>> would be
>> OK.
>>
>> This is a serious process bug we have, we've had for years, and we  
>> MUST
>> fix.  If it's not fixed soon, I'm just going to give up on Emacs
>> maintenance and bid the project goodbye.  The pain level is too high.
>
> I've been interested in setting up an automatic build test environment
> for Emacs for some time. It would basically checkout the emacs sources
> on every checkin, build the sources and report the build
> quality(probably with some existing continuous integration software).
>
> I would also store a TB or something of past builds.
>
> Would that be of any help to you?
>
>
>> Sorry, and all that.
> -- 
> Joakim Verona
>

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Eli Zaretskii]

> From: "Jan D." <jan.h.d@swipnet.se>
> Date: Mon, 7 Sep 2009 12:30:00 +0200
> Cc: Alan Mackenzie <acm@muc.de>, "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> 
> A make that removed old .elc files and did bootstrap when needed is  
> the ideal solution. The second part is probably very hard.

What we need IMO is a way to scan all the *.el files, look for
`require', and generate Make dependencies between Lisp files.  Then
this problem should be gone for good.

Any takers?

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Ken Raeburn]

On Sep 7, 2009, at 13:42, Eli Zaretskii wrote:
>> From: "Jan D." <jan.h.d@swipnet.se>
>> Date: Mon, 7 Sep 2009 12:30:00 +0200
>> Cc: Alan Mackenzie <acm@muc.de>, "emacs-devel@gnu.org" <emacs-devel@gnu.org 
>> >
>>
>> A make that removed old .elc files and did bootstrap when needed is
>> the ideal solution. The second part is probably very hard.
>
> What we need IMO is a way to scan all the *.el files, look for
> `require', and generate Make dependencies between Lisp files.  Then
> this problem should be gone for good.
>
> Any takers?

I just spent part of the last couple of hours thinking over this  
problem too.  Other files could be autoloaded during compilation, and  
should be listed as dependencies too.

The byte compiler has rather a lot of information available while it's  
doing its job.  Perhaps it could look at the files that were loaded as  
part of the compilation, and generate a list of dependencies.  The  
byte compiler code itself, and files loaded via loadup.el, would still  
present a problem; a conservative approach would be to always list all  
of them as dependencies, though that would lead to excessive  
rebuilding still.

Ken

RE: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Drew Adams]

> > What we need IMO is a way to scan all the *.el files, look for
> > `require', and generate Make dependencies between Lisp files.  Then
> > this problem should be gone for good. Any takers?
> 
> I just spent part of the last couple of hours thinking over this  
> problem too.  Other files could be autoloaded during 
> compilation, and should be listed as dependencies too.
> 
> The byte compiler has rather a lot of information available 
> while it's doing its job.  Perhaps it could look at the files
> that were loaded as part of the compilation, and generate a
> list of dependencies.  The byte compiler code itself, and
> files loaded via loadup.el, would still present a problem;
> a conservative approach would be to always list all  
> of them as dependencies, though that would lead to excessive  
> rebuilding still.

Dunno if any of this code would help, but you might be able to adapt some of it,
depending on what you're trying to do:

http://www.emacswiki.org/emacs/LibraryDependencies

http://www.emacswiki.org/emacs/lib-requires.el
http://www.emacswiki.org/emacs/elisp-depend.el

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Lennart Borgman]

On Mon, Sep 7, 2009 at 8:32 PM, Drew Adams<drew.adams@oracle.com> wrote:
>> > What we need IMO is a way to scan all the *.el files, look for
>> > `require', and generate Make dependencies between Lisp files.  Then
>> > this problem should be gone for good. Any takers?
>>
>> I just spent part of the last couple of hours thinking over this
>> problem too.  Other files could be autoloaded during
>> compilation, and should be listed as dependencies too.
>>
>> The byte compiler has rather a lot of information available
>> while it's doing its job.  Perhaps it could look at the files
>> that were loaded as part of the compilation, and generate a
>> list of dependencies.  The byte compiler code itself, and
>> files loaded via loadup.el, would still present a problem;
>> a conservative approach would be to always list all
>> of them as dependencies, though that would lead to excessive
>> rebuilding still.
>
> Dunno if any of this code would help, but you might be able to adapt some of it,
> depending on what you're trying to do:
>
> http://www.emacswiki.org/emacs/LibraryDependencies
>
> http://www.emacswiki.org/emacs/lib-requires.el
> http://www.emacswiki.org/emacs/elisp-depend.el


A further question:

What can the Emacs that compiles the libraries do? Can it hold the
dependency tree and start subprocesses for compilation? Or is that a
bad idea?

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Ken Raeburn]

On Sep 7, 2009, at 14:42, Lennart Borgman wrote:
> A further question:
>
> What can the Emacs that compiles the libraries do? Can it hold the
> dependency tree and start subprocesses for compilation? Or is that a
> bad idea?

My initial thought is that we should let make do that, since it's  
already got the machinery for dealing with multiple processes running  
in parallel (necessary if you want to take full advantage of modern  
multicore machines), implementing limits on the number of active  
processes, etc.  Having the Emacs process just generate or update the  
list of dependencies in the makefile is probably good enough.

Ken

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Glenn Morris]

>>> A make that removed old .elc files and did bootstrap when needed is
>>> the ideal solution. The second part is probably very hard.
>>
>> What we need IMO is a way to scan all the *.el files, look for
>> `require', and generate Make dependencies between Lisp files.  Then
>> this problem should be gone for good.

Or we could just ask Stefan to install the patch he has to prefer .el
files while building:

http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=2061#92

The level of hyperbole in the OP and the subject is rather extreme.
Always run `make bootstrap' if you don't want to deal with these issues.

Automated build testing is no use, since it cannot come round to your
house and type `make bootstrap' for you.

RE: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Drew Adams]

> it cannot come round to your house and type `make bootstrap' for you.

That feature's scheduled for Emacs 25, no?

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Mon, 07 Sep 2009 15:03:29 -0400
> 
> Or we could just ask Stefan to install the patch he has to prefer .el
> files while building:
> 
> http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=2061#92

But that would mean loadup loads un-compiled files, right?  That
doesn't sound like a good idea.

more reliable `make'

[Glenn Morris]

[subject changed]

Eli Zaretskii wrote:

> But that would mean loadup loads un-compiled files, right?  That
> doesn't sound like a good idea.

I'll wait to hear the details of Stefans's implementation, but I
imagine it was preferring the .el files only if newer. The standard
make rules would ensure the loadup dependencies were all recompiled
before dumping.

Re: more reliable `make'

[Stefan Monnier]

> I'll wait to hear the details of Stefans's implementation, but I
> imagine it was preferring the .el files only if newer.

And only during byte-compilation, yes.


        Stefan

Re: more reliable `make'

[Glenn Morris]

Stefan Monnier wrote:

>> I'll wait to hear the details of Stefans's implementation, but I
>> imagine it was preferring the .el files only if newer.
>
> And only during byte-compilation, yes.

So if I understand correctly, while not as correct/fast as generating
all the dependency info, it's a lot simpler to implement, and should
render issues like this a thing of the past?

(The ability to prefer uncompiled files has been requested a number of
times, so it also seems like an option worth having in general.)

Re: more reliable `make'

[Stefan Monnier]

>>> I'll wait to hear the details of Stefans's implementation, but I
>>> imagine it was preferring the .el files only if newer.
>> And only during byte-compilation, yes.
> So if I understand correctly, while not as correct/fast as generating
> all the dependency info, it's a lot simpler to implement, and should
> render issues like this a thing of the past?

It won't make them a thing of the past.  It just makes them a bit
less frequent.


        Stefan

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[joakim]

Glenn Morris <rgm@gnu.org> writes:

>>>> A make that removed old .elc files and did bootstrap when needed is
>>>> the ideal solution. The second part is probably very hard.
>>>
>>> What we need IMO is a way to scan all the *.el files, look for
>>> `require', and generate Make dependencies between Lisp files.  Then
>>> this problem should be gone for good.
>
> Or we could just ask Stefan to install the patch he has to prefer .el
> files while building:
>
> http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=2061#92
>
> The level of hyperbole in the OP and the subject is rather extreme.
> Always run `make bootstrap' if you don't want to deal with these issues.
>
> Automated build testing is no use, since it cannot come round to your
> house and type `make bootstrap' for you.

Would it not be possible to have 2 sets of automated build tests? One
that tests a clean checkout, and one that shows the latest state of an
incremental build? I think automated build tests would be useful even
without testing incremental builds.

>
-- 
Joakim Verona

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Glenn Morris]

joakim@verona.se wrote:

> Would it not be possible to have 2 sets of automated build tests? One
> that tests a clean checkout, and one that shows the latest state of an
> incremental build? I think automated build tests would be useful even
> without testing incremental builds.

Sure. Make the latter send Alan an email I guess.

I'm fairly sure there are several people who already do clean builds
every day anyway (there is certainly at least one), and report any
real breakage.

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[joakim]

Glenn Morris <rgm@gnu.org> writes:

> joakim@verona.se wrote:
>
>> Would it not be possible to have 2 sets of automated build tests? One
>> that tests a clean checkout, and one that shows the latest state of an
>> incremental build? I think automated build tests would be useful even
>> without testing incremental builds.
>
> Sure. Make the latter send Alan an email I guess.

He could subscribe to the build rss feed if he so wished.

> I'm fairly sure there are several people who already do clean builds
> every day anyway (there is certainly at least one), and report any
> real breakage.

Yes of course. I would mostly set continuous integration up for my own
personal needs, and was just gauging if there were other people that
might be interested in my particular setup.

-- 
Joakim Verona

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Alan Mackenzie]

Hi, Glenn,

On Mon, Sep 07, 2009 at 03:03:29PM -0400, Glenn Morris wrote:

> >>> A make that removed old .elc files and did bootstrap when needed is
> >>> the ideal solution. The second part is probably very hard.

> >> What we need IMO is a way to scan all the *.el files, look for
> >> `require', and generate Make dependencies between Lisp files.  Then
> >> this problem should be gone for good.

> Or we could just ask Stefan to install the patch he has to prefer .el
> files while building:

> http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=2061#92

> Always run `make bootstrap' if you don't want to deal with these issues.

That's horrible.  It's very slow, and if the build breaks for any other
reason (careless commission), you haven't even got the bulk of your old
build left.

> Automated build testing is no use, since it cannot come round to your
> house and type `make bootstrap' for you.

Automated build testing is of some use; it can often detect an incomplete
(or otherwise misjudged) commission.  It should also be cheap - the
entire cost is in setting it up.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Stefan Monnier]

>> Always run `make bootstrap' if you don't want to deal with these issues.

> That's horrible.  It's very slow, and if the build breaks for any other
> reason (careless commission), you haven't even got the bulk of your old
> build left.

The problem is that we don't have the necessary code right now to
generate the needed dependency graph, so wile "cvs update; make" often
works, it sometimes fails for lack of dependency info, in which case
"make bootstrap" works around the problem.


        Stefan

Re: build broken: no defun org-float-time. A workaround.

[Alan Mackenzie]

Hi, everybody,

On Tue, Sep 08, 2009 at 12:48:50PM -0400, Stefan Monnier wrote:
> >> Always run `make bootstrap' if you don't want to deal with these issues.

> > That's horrible.  It's very slow, and if the build breaks for any
> > other reason (careless commission), you haven't even got the bulk of
> > your old build left.

> The problem is that we don't have the necessary code right now to
> generate the needed dependency graph, so wile "cvs update; make" often
> works, it sometimes fails for lack of dependency info, in which case
> "make bootstrap" works around the problem.

How about this for a workaround?  Before doing any byte compilation, you
scan the .../lisp directory for stale .elc files, and rename them all.
That way, the byte compiler is forced to load the new .el files in
response to `require', and the like.  At the end of the process, you
either delete or unrename the stale files as appropriate.

Here's a script I've hacked together which does this.  Is it easy to
implement this idea in the makefiles?

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
#! /bin/bash
##############################################################################
#
#   m a k e - e m a c s . s h
#
# Invoke make, having first renamed stale .elc files to prevent them being
# loaded spuriously by the byte compiler.  Afterwards, delete or restore each
# renamed file appropriately.
#
# Call this script from the top level of an Emacs tree.
##############################################################################
OLD=ksiwrjcy

STALES=$(for f in `find . -name '*.el'` ; do
            if [ -f ${f}c -a ${f}c -ot $f ] ; then echo ${f}c ; fi
        done)
echo "Stale files: $STALES" >&2
sleep 3
echo

for f in $STALES ; do mv ${f}{,.$OLD} ; done

make

for f in $STALES ; do
    if [ -f $f ]
    then rm ${f}.$OLD
    else mv ${f}{.$OLD,}
        echo $f wasn\'t remade >&2
    fi
done
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: build broken: no defun org-float-time. A workaround.

[Ken Raeburn]

On Sep 8, 2009, at 14:17, Alan Mackenzie wrote:
> How about this for a workaround?  Before doing any byte compilation,  
> you
> scan the .../lisp directory for stale .elc files, and rename them all.
> That way, the byte compiler is forced to load the new .el files in
> response to `require', and the like.  At the end of the process, you
> either delete or unrename the stale files as appropriate.

It's not quite enough.  It'll prevent the use of outdated .elc files,  
but if macro definitions have changed, and the macros are used in  
other lisp files, it won't trigger recompilation of those files.

Ken

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Mon, 7 Sep 2009 13:59:07 -0400
> Cc: "Jan D." <jan.h.d@swipnet.se>, acm@muc.de, joakim@verona.se,
>         emacs-devel@gnu.org
> 
> Other files could be autoloaded during compilation, and should be
> listed as dependencies too.

We have `features' and `load-history' to help us.

> The byte compiler code itself, and files loaded via loadup.el, would
> still present a problem; a conservative approach would be to always
> list all of them as dependencies, though that would lead to
> excessive rebuilding still.

Files loaded by loadup.el are easily extracted by simple text
scanning, and the byte compiler itself indeed _is_ a prerequisite for
every other compilation.

Btw, I don't understand what problem do you see with files preloaded
by loadup.  They should simply be all prerequisites of temacs, and
that's it, right?

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Ken Raeburn]

On Sep 7, 2009, at 16:06, Eli Zaretskii wrote:
> We have `features' and `load-history' to help us.

Yes.

> Files loaded by loadup.el are easily extracted by simple text
> scanning, and the byte compiler itself indeed _is_ a prerequisite for
> every other compilation.

But if a minor comment change is made in the compiler source, must we  
recompile everything?  We don't generally make .o files explicitly  
depend on the C compiler.  Though, in the gcc project they probably  
do... so, yeah, maybe it's the way to go.

> Btw, I don't understand what problem do you see with files preloaded
> by loadup.  They should simply be all prerequisites of temacs, and
> that's it, right?


Well, "emacs", not "temacs" which is just linked from the C code, but  
also I was assuming we wouldn't make the emacs binary an explicit  
dependency for the .elc files; otherwise, someone downloading and  
building a release will have to recompile all the elisp code even  
though the results will differ from everyone else's only in the first  
few comment lines.

And I'm not sure if we want to list all those files as dependencies  
anyways.  We might indeed want to recompile everything if subr.el  
changes, if we can't figure out what actually used stuff from that  
file.  On the other hand, we probably don't want to recompile  
everything because lisp/language/georgian.el changed.  (Conservatively  
speaking, I suppose we should, in case someone decided to redefine  
'defcustom' there.)  Actually, I think we want to rebuild if subr.elc  
changes, other than the comments indicating who compiled it and when,  
not just for all subr.el changes, and likewise for the compiler code....

Ken

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Stephen J. Turnbull]

Ken Raeburn writes:

 > But if a minor comment change is made in the compiler source, must we  
 > recompile everything?  We don't generally make .o files explicitly  
 > depend on the C compiler.

C++ ABIs ....

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Mon, 7 Sep 2009 19:39:32 -0400
> Cc: Emacs development discussions <emacs-devel@gnu.org>
> 
> > Files loaded by loadup.el are easily extracted by simple text
> > scanning, and the byte compiler itself indeed _is_ a prerequisite for
> > every other compilation.
> 
> But if a minor comment change is made in the compiler source, must we  
> recompile everything?

Yes, why not?

> We don't generally make .o files explicitly depend on the C
> compiler.

If a compiler can change the ABI, you must.

> > Btw, I don't understand what problem do you see with files preloaded
> > by loadup.  They should simply be all prerequisites of temacs, and
> > that's it, right?
> 
> 
> Well, "emacs", not "temacs" which is just linked from the C code, but  
> also I was assuming we wouldn't make the emacs binary an explicit  
> dependency for the .elc files

Not all of them, just those that are preloaded.  We already have that
dependency:

  emacs${EXEEXT}: temacs${EXEEXT} ${etc}DOC ${lisp} ${SOME_MACHINE_LISP}

> otherwise, someone downloading and  
> building a release will have to recompile all the elisp code

I don't see why.  Please explain.

> On the other hand, we probably don't want to recompile  
> everything because lisp/language/georgian.el changed.

No, we don't, but I don't see how would that happen.  You just need to
re-dump.

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Ken Raeburn]

On Sep 7, 2009, at 23:20, Eli Zaretskii wrote:
>> But if a minor comment change is made in the compiler source, must we
>> recompile everything?
>
> Yes, why not?

Excessive time spent rebuilding things that aren't going to change?

>> We don't generally make .o files explicitly depend on the C
>> compiler.
>
> If a compiler can change the ABI, you must.

True.  Is that something we need to worry about with Emacs?  It's  
changed a handful of times in the history of Emacs, but hasn't it  
generally been backwards-compatible?  Don't older .elc files generally  
still work?  Do we really need to force a recompile and *make*  
everything take advantage of the latest and greatest optimizer tweaks  
or whatever?

>>> Btw, I don't understand what problem do you see with files preloaded
>>> by loadup.  They should simply be all prerequisites of temacs, and
>>> that's it, right?
>>
>> Well, "emacs", not "temacs" which is just linked from the C code, but
>> also I was assuming we wouldn't make the emacs binary an explicit
>> dependency for the .elc files
>
> Not all of them, just those that are preloaded.  We already have that
> dependency:
>
>  emacs${EXEEXT}: temacs${EXEEXT} ${etc}DOC ${lisp} $ 
> {SOME_MACHINE_LISP}

So "emacs" gets rebuilt if subr.el(c) is updated, yes.  But I meant  
the other way around -- that we wouldn't want to list foobar.elc as  
depending on the emacs binary as a way of expressing dependence on all  
the macros and functions that were preloaded into that binary.

>> otherwise, someone downloading and
>> building a release will have to recompile all the elisp code
>
> I don't see why.  Please explain.

If foobar.el uses a macro that's defined in subr.el or something else  
that's preloaded, and the macro's definition (or that of some function  
in yet another file that gets invoked during expansion of the macro)  
is changed, we want foobar.el to be recompiled, right?  What should  
foobar.elc be listed as depending on that would trigger this?  The  
choices seem to be subr.{el,elc} (and by extension all the preloaded  
files, unless we can either compute more precise dependencies or  
maintain them properly by hand) or the emacs binary itself.

Compiling a freshly downloaded source tree obviously updates the emacs  
binary's timestamp, so if that's how we express the dependency on  
"whatever is preloaded that we can't compute precise dependencies on",  
everything gets byte compiled at build time, millions of people waste  
millions of extra cycles, electricity usage soars, more oil is burned,  
the greenhouse effect is amplified, and we all die.  (I'm sorry, was  
someone talking about hyperbole earlier?)  So, I think we probably  
don't want to do that.

Which, I think, leaves us indicating that foobar.elc depends on  
subr.elc and friends.

>> On the other hand, we probably don't want to recompile
>> everything because lisp/language/georgian.el changed.
>
> No, we don't, but I don't see how would that happen.  You just need to
> re-dump.

Depends if georgian.el(c) gets listed among the dependencies for all  
the other .elc files, just in case someday it redefines "defcustom",  
kind of like how things would depend on the compiler just in case the  
ABI changes.

Ken

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Tue, 8 Sep 2009 03:54:28 -0400
> Cc: emacs-devel@gnu.org
> 
> On Sep 7, 2009, at 23:20, Eli Zaretskii wrote:
> >> But if a minor comment change is made in the compiler source, must we
> >> recompile everything?
> >
> > Yes, why not?
> 
> Excessive time spent rebuilding things that aren't going to change?

This is why we want to avoid it as much as possible.  What I asked is
why do you think we might be able to get away without it.

> >> We don't generally make .o files explicitly depend on the C
> >> compiler.
> >
> > If a compiler can change the ABI, you must.
> 
> True.  Is that something we need to worry about with Emacs?  It's  
> changed a handful of times in the history of Emacs, but hasn't it  
> generally been backwards-compatible?

Mostly yes, but if you want to be sure...

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Ken Raeburn]

On Sep 8, 2009, at 13:47, Eli Zaretskii wrote:
>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Tue, 8 Sep 2009 03:54:28 -0400
>> Cc: emacs-devel@gnu.org
>>
>> On Sep 7, 2009, at 23:20, Eli Zaretskii wrote:
>>>> But if a minor comment change is made in the compiler source,  
>>>> must we
>>>> recompile everything?
>>>
>>> Yes, why not?
>>
>> Excessive time spent rebuilding things that aren't going to change?
>
> This is why we want to avoid it as much as possible.  What I asked is
> why do you think we might be able to get away without it.

Ah.  Well, in the abstract, if it really is just a comment change (or  
whitespace change, or minor coding change that the optimizer boils  
down to the same byte code as the previous version), it won't affect  
the operation of the compiler and thus won't affect the result.

Concretely, there are ways we can check for that -- see if the "real"  
content of the .elc file for the compiler source has changed, by which  
I mean stuff other than the compilation info at the front.  We could  
compare most of the file content with the previous version, or get rid  
of those comments and compare entire files, and retain the old file  
versions and old timestamps on unchanged files for further dependency  
checks, if we keep relying on make to do the grunt work.  I think  
GCC's build system has some mechanisms for doing that sort of thing,  
though it's a bit awkward.

Bypassing make, or at least its timestamp checking as the primary  
mechanism, we could compute and record md5 (or sha1, or sha2) sums of  
the important parts of the files, and compare them to see if things  
have changed in any way we care about.  (E.g., foobar.elc could  
include in its compilation-info comments, "foobar.el:47ecb138...  
subr.elc:3feb9814... quux.elc:4857fabd...".  For performance, it could  
even cache its own md5 sum so we don't have to keep recomputing it.)   
The recompilation command could check the dependencies against the  
recorded sums, and if they all match, just update the timestamp on the  
output file, to keep make from re-invoking that compilation command.   
Touching subr.el could still cause a cascade of recompilation  
commands, but they'd just be updating timestamps.

Ken

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Stefan Monnier]

>> Other files could be autoloaded during compilation, and should be
>> listed as dependencies too.
> We have `features' and `load-history' to help us.

It's doable, yes.  Note that just looking at the `require's and
`provide's will give you a graph with cycles.

> Files loaded by loadup.el are easily extracted by simple text
> scanning, and the byte compiler itself indeed _is_ a prerequisite for
> every other compilation.

Note that a basic hypothesis here, is that "make bootstrap" is not
a good solution, so any solution that ends recompiling similar amounts
of code is not good either.  But if you look in detail, you'll see that
only bootstrap does it really right: the byte-compiler is
a prerequisitie of all elc files, and the byte-compiler itself uses
potentially any code in loadup.el (and then some), so any change to
a preloaded file should fundamentally require recompiling all
Lisp files.  Anything short of that is an optimistic optimization.

> Btw, I don't understand what problem do you see with files preloaded
> by loadup.  They should simply be all prerequisites of temacs, and
> that's it, right?

Of temacs, no, but of `emacs' or `emacs-bootstrap', yes.
A recent case to ponder is the addition of define-obsolete-face-alias.
Maybe we can handle this one in the following way:

  When byte-compiling elc files with a preexisting emacs-bootstrap
  binary, re-load all the files that were preloaded but that have been
  changed since building emacs-bootstrap, before doing the actual
  recompilation (i.e. when starting emacs-bootstrap, check the
  load-history to see if any of those files have changed and reload them
  if needed).

Of course, this will bump into a few bugs at first, since some of the
preloaded files do not like to be re-loaded.


-- Stefan

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Ken Raeburn <raeburn@raeburn.org>, acm@muc.de, jan.h.d@swipnet.se,
>         joakim@verona.se, emacs-devel@gnu.org
> Date: Tue, 08 Sep 2009 13:01:21 -0400
> 
> Note that a basic hypothesis here, is that "make bootstrap" is not
> a good solution, so any solution that ends recompiling similar amounts
> of code is not good either.  But if you look in detail, you'll see that
> only bootstrap does it really right: the byte-compiler is
> a prerequisitie of all elc files, and the byte-compiler itself uses
> potentially any code in loadup.el (and then some), so any change to
> a preloaded file should fundamentally require recompiling all
> Lisp files.

I don't think this is quite that bad.  There's a similar issue in many
GNU packages with config.h files produced by a configure script, and
the solution there is to output the new version into a temporary file
and use move-if-change to replace the original file.  We could do the
same with bytecomp.elc, and that would solve the above problem.

>   When byte-compiling elc files with a preexisting emacs-bootstrap
>   binary, re-load all the files that were preloaded but that have been
>   changed since building emacs-bootstrap, before doing the actual
>   recompilation (i.e. when starting emacs-bootstrap, check the
>   load-history to see if any of those files have changed and reload them
>   if needed).

Sounds like a good plan to me.

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Stefan Monnier]

>> When byte-compiling elc files with a preexisting emacs-bootstrap
>> binary, re-load all the files that were preloaded but that have been
>> changed since building emacs-bootstrap, before doing the actual
>> recompilation (i.e. when starting emacs-bootstrap, check the
>> load-history to see if any of those files have changed and reload them
>> if needed).

> Sounds like a good plan to me.

You can try the patch below (actually: first apply the second hunk,
recompile all, then install the first hunk).


        Stefan


Index: lisp/Makefile.in
===================================================================
RCS file: /sources/emacs/emacs/lisp/Makefile.in,v
retrieving revision 1.188
diff -u -r1.188 Makefile.in
--- lisp/Makefile.in	27 Aug 2009 18:35:24 -0000	1.188
+++ lisp/Makefile.in	9 Sep 2009 03:13:25 -0000
@@ -1287,7 +1287,7 @@
 # src/Makefile.in to rebuild a particular Lisp file, no questions asked.
 compile-onefile:
 	@echo Compiling $(THEFILE)
-	@$(emacs) $(BYTE_COMPILE_EXTRA_FLAGS) -f batch-byte-compile $(THEFILE)
+	@$(emacs) $(BYTE_COMPILE_EXTRA_FLAGS) -f byte-compile-refresh-preloaded -f batch-byte-compile $(THEFILE)
 
 # Files MUST be compiled one by one. If we compile several files in a
 # row (i.e., in the same instance of Emacs) we can't make sure that
Index: lisp/emacs-lisp/bytecomp.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/emacs-lisp/bytecomp.el,v
retrieving revision 2.257
diff -u -r2.257 bytecomp.el
--- lisp/emacs-lisp/bytecomp.el	5 Sep 2009 19:10:40 -0000	2.257
+++ lisp/emacs-lisp/bytecomp.el	9 Sep 2009 03:13:26 -0000
@@ -4362,6 +4363,24 @@
        nil))))
 
 ;;;###autoload
+(defun byte-compile-refresh-preloaded ()
+  (let* ((argv0 (car command-line-args))
+         (emacs-file (executable-find argv0)))
+    (if (not (and emacs-file (file-executable-p emacs-file)))
+        (message "Can't find %s to refresh preloaded Lisp files" argv0)
+      (dolist (f (reverse load-history))
+        (setq f (car f))
+        (if (string-match "elc\\'" f) (setq f (substring f 0 -1)))
+        (when (and (file-readable-p f)
+                   (file-newer-than-file-p f emacs-file))
+          (message "Reloading stale %s" (file-name-nondirectory f))
+          (condition-case nil
+              (load f 'noerror nil 'nosuffix)
+            ;; Probably shouldn't happen, but in case of an error, it seems
+            ;; at least as useful to ignore it as it is to stop compilation.
+            (error nil)))))))
+
+;;;###autoload
 (defun batch-byte-recompile-directory (&optional arg)
   "Run `byte-recompile-directory' on the dirs remaining on the command line.
 Must be used only with `-batch', and kills Emacs on completion.

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > What we need IMO is a way to scan all the *.el files, look for
 > `require', and generate Make dependencies between Lisp files.  Then
 > this problem should be gone for good.

It's both harder and easier than that.  The harder part: Charles
Wilson tried to do this for XEmacs about 10 years ago and gave up.
You also need to handle conditional requires, explicit loads,
autoloads, and there was some other weirdness (files like cl-macs.el,
which in XEmacs at least can't be required, and maybe even more arcane
stuff ... at least you guys don't support DLLs so you don't have to
worry about that).

The easier part: we can assume that the .el files are good for the
purposes of eliminating errors from stray out-of-date .elcs.  But .el
files are just as good as .elc files (as long as you're willing to
accept a few extra seconds for bootstrapping operations).

The way XEmacs handles this is

1. build temacs, and start it
2. load enough .el files to bootstrap the compiler (I think this is
   the whole to-dump list in practice)
3. compile the compiler, and reload it as compiled
4. recompile all out of date .elc files in the to-dump list
5. restart temacs, load the .elc files, and dump
6. start xemacs and recompile any remaining out-of-date elcs.

We still get a few .el/.elc confusions, but they're all PEBKACs.

Alan: there *is* a good reason for preferring the .elc to the .el in
normal operation.  Much surgical experimentation on elisp is done "in
vitro", so the .elc may be considered "stable", while definitions from
the .el are pulled in with C-x C-e for testing.  As long as you
haven't trashed M-x load somehow, you can restore the stable Lisp
environment with a quick M-x load.

HTH

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: "Jan D." <jan.h.d@swipnet.se>,
>     acm@muc.de,
>     joakim@verona.se,
>     emacs-devel@gnu.org
> Date: Tue, 08 Sep 2009 11:37:54 +0900
> 
> You also need to handle conditional requires, explicit loads,
> autoloads, and there was some other weirdness (files like cl-macs.el,
> which in XEmacs at least can't be required, and maybe even more arcane
> stuff ... at least you guys don't support DLLs so you don't have to
> worry about that).

Again, I think `features' and `load-history' should overcome those
difficulties.  And if needed, we could add some code to the byte
compiler to output dependencies, like GCC does.

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Stefan Monnier]

> I've been interested in setting up an automatic build test environment
> for Emacs for some time. It would basically checkout the emacs sources
> on every checkin, build the sources and report the build
> quality(probably with some existing continuous integration software).

I think such a thing would be good for 2 things:
1- maintain a CVS tag pointing at the last-known-good (well, known
   good for your particular test enviornment, of course).
   So people like Alan can simply follow this tag.
2- auto-detect cases where bootstrap is necessary.  Once detected, the
   script could adding a special thingy somewhere (e.g. a file
   emacs/admin/need-bootstrap) so that "make"
   automatically does "make bootstrap" for you when facing this problem.


-- Stefan

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Miles Bader]

Alan Mackenzie <acm@muc.de> writes:
> This is a serious process bug we have, we've had for years, and we MUST
> fix.  If it's not fixed soon, I'm just going to give up on Emacs
> maintenance and bid the project goodbye.  The pain level is too high.

I agree that this sort of thing can be annoying, but the procedure for
dealing with such things is well known and not particular onerous:
"make bootstrap" (could be a bit onerous if you've got a very slow
machine though)

Moreover, it happens very rarely -- I'd guess only a few times a year,
and far less often than the more usual sort of problem where somebody
just committed a bogus change that simply doesn't compile (though that
kind of problem is fairly rare too).

I don't know how you get "the pain level is too high" out of that...

-Miles

-- 
Arrest, v. Formally to detain one accused of unusualness.

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[joakim]

Miles Bader <miles@gnu.org> writes:

> Alan Mackenzie <acm@muc.de> writes:
>> This is a serious process bug we have, we've had for years, and we MUST
>> fix.  If it's not fixed soon, I'm just going to give up on Emacs
>> maintenance and bid the project goodbye.  The pain level is too high.
>
> I agree that this sort of thing can be annoying, but the procedure for
> dealing with such things is well known and not particular onerous:
> "make bootstrap" (could be a bit onerous if you've got a very slow
> machine though)
>
> Moreover, it happens very rarely -- I'd guess only a few times a year,
> and far less often than the more usual sort of problem where somebody
> just committed a bogus change that simply doesn't compile (though that
> kind of problem is fairly rare too).
>
> I don't know how you get "the pain level is too high" out of that...

It does kind of interrupt ones workflow. I scratched my head a lot last
time, because I thought the error was on my end and didnt think of
bootstraping. 

>
> -Miles
-- 
Joakim Verona

Re: build broken: no defun org-float-time. Who's guilty, and what does he propose?

[Alan Mackenzie]

Hi, Miles,

On Mon, Sep 07, 2009 at 07:05:32PM +0900, Miles Bader wrote:
> Alan Mackenzie <acm@muc.de> writes:
> > This is a serious process bug we have, we've had for years, and we
> > MUST fix.  If it's not fixed soon, I'm just going to give up on Emacs
> > maintenance and bid the project goodbye.  The pain level is too high.

> I agree that this sort of thing can be annoying, but the procedure for
> dealing with such things is well known and not particular onerous:
> "make bootstrap" (could be a bit onerous if you've got a very slow
> machine though)

Maybe I'm not as much of a real man as you are.

I find the process onerous indeed.  The build fails, you've got to read
error messages, possibly grep the source code for a missing symbol, have
a quick check of ChangeLog (MOST of us put details of changes there), ...

And yes, make bootstrap takes a long time, where a long time is perhaps
20 minutes or half an hour.  Very slow PCs haven't been sold since the
1990s, so there probably aren't too many of them being used by Emacsers.

The point is, make should work.  Not best out of ten, not all but a "few
times a year", but always, modulo the occasional real screwup.

> Moreover, it happens very rarely -- I'd guess only a few times a year,

No, it happens frequently -- I'd guess several times a year.

> and far less often than the more usual sort of problem where somebody
> just committed a bogus change that simply doesn't compile (though that
> kind of problem is fairly rare too).

Er, excuse me, but that comes into the category of build failure too.
There's no justice in committing build-breaking changes, it's anything
but just.  It's a window in the donkey for everybody.  That our process
allows it is a bug.  We earnestly discussed fixes about a year ago.

> I don't know how you get "the pain level is too high" out of that...

Because I've been sensitised to it by the repeated instances of it over
the last few years.  It happened to me this morning, only the second or
third time I've cvs updated since the release of 23.1.  Because it
completely throws my train of thought, destroying my morning's work.

Above all, because it forces me to do dumb, brainless, moronic stuff
which a computer program, for example a build script, could do far
faster, painlessly, with less effort than me.

Having forced myself to look at it, it's fairly obvious.  The build
process is loading the wrong file in response to (require 'org-exp).
It's loading an out of date file, because our makefiles are broken.
That's not news to anybody.  And make bootstrap is a ridiculous
overreaction to one out of date file.elc.

Maybe the solution is, somehow, to load the more up to date of foo.el and
foo.elc during the build process.  In fact, the build process even admits
it knows that "source file foo.el newrt than byte-compiled file".
There's probably a terribly good reason why it doesn't fix the problem
rather than just reporting it.

> -Miles

-- 
Alan Mackenzie (Nuremberg, Germany).