experiences running a Hudson CI server on the Emacs codebase

experiences running a Hudson CI server on the Emacs codebase

[joakim]

I'm running a Hudson CI server on the Emacs codebase. I'm not able to
make it public yet, but might this autumn. Anyway, the experiences
should be valid for any CI setup.

The CI server has tried to do 110 builds since I set it up. It polls bzr
to see if a change has been made, then tries to build. Out of 110,
about 40 has failed.

Some of the failed builds were as expected due to the ansification of
the trunk.

Unexpectedly, a lot of failures were due to bzr. I use bzr 2.0.5 from
the Fedora distro, and the bzr plugin of Hudson. Theres lots of free
disk space.

Heres one type of failure:
,----
| Started by an SCM change
| $ bzr revision-info -d /var/lib/hudson/jobs/emacs-trunk/workspace
| [workspace] $ bzr pull --overwrite http://bzr.savannah.gnu.org/r/emacs/trunk
| http://bzr.savannah.gnu.org/r/emacs/trunk is permanently redirected to http://bzr.savannah.gnu.org/r/emacs/trunk/
| Unable to obtain lock file:///var/lib/hudson/jobs/emacs-trunk/workspace/.bzr/branch/lock
| held by hudson@exodia on host exodia [process #29264]
| locked 3 hours, 0 minutes ago
| Will continue to try until 10:11:13, unless you press Ctrl-C
| If you're sure that it's not being modified, use bzr break-lock file:///var/lib/hudson/jobs/emacs-trunk/workspace/.bzr/branch/lock
| bzr: ERROR: Could not acquire lock "LockDir(file:///var/lib/hudson/jobs/emacs-trunk/workspace/.bzr/branch/lock)": 
| ERROR: Failed to pull
| Getting local revision...
| $ bzr revision-info -d /var/lib/hudson/jobs/emacs-trunk/workspace
| kfogel@red-bean.com-20100713221751-w0z5dw4b8cpu1a3h
| Finished: FAILURE
`----

Heres another one:

,----
| Started by an SCM change
| $ bzr revision-info -d /var/lib/hudson/jobs/emacs-trunk/workspace
| [workspace] $ bzr pull --overwrite http://bzr.savannah.gnu.org/r/emacs/trunk
| bzr: ERROR: exceptions.EOFError: EOF read where object expected
| 
| Traceback (most recent call last):
|   File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 842, in exception_to_return_code
|     return the_callable(*args, **kwargs)
|   File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 1037, in run_bzr
|     ret = run(*run_argv)
|   File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 654, in run_argv_aliases
|     return self.run(**all_cmd_args)
|   File "/usr/lib/python2.6/site-packages/bzrlib/builtins.py", line 970, in run
|     possible_transports=possible_transports)
|   File "/usr/lib/python2.6/site-packages/bzrlib/bundle/__init__.py", line 49, in read_mergeable_from_url
|     _do_directive)
|   File "/usr/lib/python2.6/site-packages/bzrlib/bundle/__init__.py", line 71, in read_mergeable_from_transport
|     redirected_transport)
|   File "/usr/lib/python2.6/site-packages/bzrlib/lazy_import.py", line 125, in __call__
|     return obj(*args, **kwargs)
|   File "/usr/lib/python2.6/site-packages/bzrlib/transport/__init__.py", line 1646, in do_catching_redirections
|     transport = redirected(transport, e, redirection_notice)
|   File "/usr/lib/python2.6/site-packages/bzrlib/bundle/__init__.py", line 62, in redirected_transport
|     note(redirection_notice)
|   File "/usr/lib/python2.6/site-packages/bzrlib/trace.py", line 117, in note
|     _bzr_logger.info(*args, **kwargs)
|   File "/usr/lib/python2.6/logging/__init__.py", line 1030, in info
|     self._log(INFO, msg, args, **kwargs)
|   File "/usr/lib/python2.6/logging/__init__.py", line 1142, in _log
|     record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, func, extra)
|   File "/usr/lib/python2.6/logging/__init__.py", line 1117, in makeRecord
|     rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
|   File "/usr/lib/python2.6/logging/__init__.py", line 272, in __init__
|     from multiprocessing import current_process
|   File "/usr/lib/python2.6/multiprocessing/__init__.py", line 63, in <module>
|     from multiprocessing.process import Process, current_process, active_children
| EOFError: EOF read where object expected
| 
| bzr 2.0.5 on python 2.6.2 (Linux-2.6.32.11-99.fc12.i686-i686-athlon-with-fedora-12-Constantine)
| arguments: ['/usr/bin/bzr', 'pull', '--overwrite', 'http://bzr.savannah.gnu.org/r/emacs/trunk']
| encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
| plugins:
|   launchpad            /usr/lib/python2.6/site-packages/bzrlib/plugins/launchpad [2.0.5]
|   netrc_credential_store /usr/lib/python2.6/site-packages/bzrlib/plugins/netrc_credential_store [2.0.5]
| 
| *** Bazaar has encountered an internal error.  This probably indicates a
|     bug in Bazaar.  You can help us fix it by filing a bug report at
|         https://bugs.launchpad.net/bzr/+filebug
|     including this traceback and a description of the problem.
| ERROR: Failed to pull
| Getting local revision...
| $ bzr revision-info -d /var/lib/hudson/jobs/emacs-trunk/workspace
| cyd@stupidchicken.com-20100711221453-4vuxwof0928ooq2v
| Finished: FAILURE
`----



-- 
Joakim Verona

Re: experiences running a Hudson CI server on the Emacs codebase

[Richard Stallman]

    Unexpectedly, a lot of failures were due to bzr.

If you encounter possible bzr bugs, would you please send bzr bug
reports?  We are not the ones who will try to fix these bugs,
so reporting them here is not constructive.

Re: experiences running a Hudson CI server on the Emacs codebase

[joakim]

Richard Stallman <rms@gnu.org> writes:

>     Unexpectedly, a lot of failures were due to bzr.
>
> If you encounter possible bzr bugs, would you please send bzr bug
> reports?  We are not the ones who will try to fix these bugs,
> so reporting them here is not constructive.

I guess my email was a bit truncated. I think the bzr bugs have since been
fixed, but with a vanilla Fedora setup you will experience them. Someone
setting up a similar build environment will hopefully now know to
install a newer bzr which isnt default in current Fedora.

My Hudson has now done 121 Emacs builds, the last ones have not had bzr
issues since ive upgraded bzr.

Anyway it would also be nice to have some code quality statistics.

To that end I installed cppcheck, which is a static code analyzer. It
detects 13 errors on the emacs codebase, most of which seem to be false
positives.

Also it would be nice to check that code formating is ok, and code
documentation is ok. I dont know how to do that yet, but I was thinking
on maybe using "indent" and "checkdoc-batch".


-- 
Joakim Verona

Re: experiences running a Hudson CI server on the Emacs codebase

[Ted Zlatanov]

On Thu, 15 Jul 2010 21:39:28 +0200 joakim@verona.se wrote: 

j> I guess my email was a bit truncated. I think the bzr bugs have since been
j> fixed, but with a vanilla Fedora setup you will experience them. Someone
j> setting up a similar build environment will hopefully now know to
j> install a newer bzr which isnt default in current Fedora.

j> My Hudson has now done 121 Emacs builds, the last ones have not had bzr
j> issues since ive upgraded bzr.

If you send me your setup I can try putting that up on a public server.
I was going to do a BuildBot but it seems like Hudson is a better choice
based on the Bazaar integration.  I'll also document the steps I took so
others can follow them.

j> Also it would be nice to check that code formating is ok, and code
j> documentation is ok. I dont know how to do that yet, but I was thinking
j> on maybe using "indent" and "checkdoc-batch".

For the ELisp code, elint-directory may be a nice way to check
everything in lisp/*

Ted

Re: experiences running a Hudson CI server on the Emacs codebase

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 15 Jul 2010 21:39:28 +0200 joakim@verona.se wrote: 
>
> j> I guess my email was a bit truncated. I think the bzr bugs have since been
> j> fixed, but with a vanilla Fedora setup you will experience them. Someone
> j> setting up a similar build environment will hopefully now know to
> j> install a newer bzr which isnt default in current Fedora.
>
> j> My Hudson has now done 121 Emacs builds, the last ones have not had bzr
> j> issues since ive upgraded bzr.
>
> If you send me your setup I can try putting that up on a public server.
> I was going to do a BuildBot but it seems like Hudson is a better choice
> based on the Bazaar integration.  I'll also document the steps I took so
> others can follow them.

I'm going to try to make my server public as well, I'm going to get a
100mbit line soon hopefully.

Anyway, heres what I have:
- Stock fedora 12, but install a newer bzr from rawhide
- Install Hudson from the Hudson yum repos
- Install the Hudson bzr plugin
- Configure a "free form" project in Hudson
- trigger build by poll bzr every 30 minutes
- ./configure && make -j4

I havent looked at it in a while, but now I see that it was 11 days ago
I had a sucessful build. The build fails on:

org/org-docbook.el:76:1:Error: Symbol's function definition is void: org-make-overlay

so aparently my build command isnt robust.

>
> j> Also it would be nice to check that code formating is ok, and code
> j> documentation is ok. I dont know how to do that yet, but I was thinking
> j> on maybe using "indent" and "checkdoc-batch".
>
> For the ELisp code, elint-directory may be a nice way to check
> everything in lisp/*
>
> Ted
>
-- 
Joakim Verona

Re: experiences running a Hudson CI server on the Emacs codebase

[Jan Djärv]

joakim@verona.se skrev 2010-07-30 20.44:

> Anyway, heres what I have:
> - Stock fedora 12, but install a newer bzr from rawhide

Just curious as I simetimes run Fedora 12 myself, what does the stock bzr lack?

	Jan D.

Re: experiences running a Hudson CI server on the Emacs codebase

[joakim]

Jan Djärv <jan.h.d@swipnet.se> writes:

> joakim@verona.se skrev 2010-07-30 20.44:
>
>> Anyway, heres what I have:
>> - Stock fedora 12, but install a newer bzr from rawhide
>
> Just curious as I simetimes run Fedora 12 myself, what does the stock bzr lack?

The stock bzr gave me this error:

bzr revision-info -d /var/lib/hudson/jobs/emacs-trunk/workspace
[workspace] $ bzr pull --overwrite http://bzr.savannah.gnu.org/r/emacs/trunk
bzr: ERROR: exceptions.EOFError: EOF read where object expected

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/site-packages/bzrlib/builtins.py", line 970, in run
    possible_transports=possible_transports)
  File "/usr/lib/python2.6/site-packages/bzrlib/bundle/__init__.py", line 49, in read_mergeable_from_url
    _do_directive)
  File "/usr/lib/python2.6/site-packages/bzrlib/bundle/__init__.py", line 71, in read_mergeable_from_transport
    redirected_transport)
  File "/usr/lib/python2.6/site-packages/bzrlib/lazy_import.py", line 125, in __call__
    return obj(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/transport/__init__.py", line 1646, in do_catching_redirections
    transport = redirected(transport, e, redirection_notice)
  File "/usr/lib/python2.6/site-packages/bzrlib/bundle/__init__.py", line 62, in redirected_transport
    note(redirection_notice)
  File "/usr/lib/python2.6/site-packages/bzrlib/trace.py", line 117, in note
    _bzr_logger.info(*args, **kwargs)
  File "/usr/lib/python2.6/logging/__init__.py", line 1030, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python2.6/logging/__init__.py", line 1142, in _log
    record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, func, extra)
  File "/usr/lib/python2.6/logging/__init__.py", line 1117, in makeRecord
    rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
  File "/usr/lib/python2.6/logging/__init__.py", line 272, in __init__
    from multiprocessing import current_process
  File "/usr/lib/python2.6/multiprocessing/__init__.py", line 63, in <module>
    from multiprocessing.process import Process, current_process, active_children
EOFError: EOF read where object expected

bzr 2.0.5 on python 2.6.2 (Linux-2.6.32.11-99.fc12.i686-i686-athlon-with-fedora-12-Constantine)
arguments: ['/usr/bin/bzr', 'pull', '--overwrite', 'http://bzr.savannah.gnu.org/r/emacs/trunk']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  launchpad            /usr/lib/python2.6/site-packages/bzrlib/plugins/launchpad [2.0.5]
  netrc_credential_store /usr/lib/python2.6/site-packages/bzrlib/plugins/netrc_credential_store [2.0.5]

*** Bazaar has encountered an internal error.  This probably indicates a
    bug in Bazaar.  You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.
ERROR: Failed to pull
Getting local revision...
$ bzr revision-info -d /var/lib/hudson/jobs/emacs-trunk/workspace
cyd@stupidchicken.com-20100711221453-4vuxwof0928ooq2v
Finished: FAILURE


>
> 	Jan D.
-- 
Joakim Verona