experiences running a Hudson CI server on the Emacs codebase

[joakim@verona.se]

I'm running a Hudson CI server on the Emacs codebase. I'm not able to
make it public yet, but might this autumn. Anyway, the experiences
should be valid for any CI setup.

The CI server has tried to do 110 builds since I set it up. It polls bzr
to see if a change has been made, then tries to build. Out of 110,
about 40 has failed.

Some of the failed builds were as expected due to the ansification of
the trunk.

Unexpectedly, a lot of failures were due to bzr. I use bzr 2.0.5 from
the Fedora distro, and the bzr plugin of Hudson. Theres lots of free
disk space.

Heres one type of failure:
,----
| Started by an SCM change
| $ bzr revision-info -d /var/lib/hudson/jobs/emacs-trunk/workspace
| [workspace] $ bzr pull --overwrite http://bzr.savannah.gnu.org/r/emacs/trunk
| http://bzr.savannah.gnu.org/r/emacs/trunk is permanently redirected to http://bzr.savannah.gnu.org/r/emacs/trunk/
| Unable to obtain lock file:///var/lib/hudson/jobs/emacs-trunk/workspace/.bzr/branch/lock
| held by hudson@exodia on host exodia [process #29264]
| locked 3 hours, 0 minutes ago
| Will continue to try until 10:11:13, unless you press Ctrl-C
| If you're sure that it's not being modified, use bzr break-lock file:///var/lib/hudson/jobs/emacs-trunk/workspace/.bzr/branch/lock
| bzr: ERROR: Could not acquire lock "LockDir(file:///var/lib/hudson/jobs/emacs-trunk/workspace/.bzr/branch/lock)": 
| ERROR: Failed to pull
| Getting local revision...
| $ bzr revision-info -d /var/lib/hudson/jobs/emacs-trunk/workspace
| kfogel@red-bean.com-20100713221751-w0z5dw4b8cpu1a3h
| Finished: FAILURE
`----

Heres another one:

,----
| Started by an SCM change
| $ bzr revision-info -d /var/lib/hudson/jobs/emacs-trunk/workspace
| [workspace] $ bzr pull --overwrite http://bzr.savannah.gnu.org/r/emacs/trunk
| bzr: ERROR: exceptions.EOFError: EOF read where object expected
| 
| Traceback (most recent call last):
|   File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 842, in exception_to_return_code
|     return the_callable(*args, **kwargs)
|   File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 1037, in run_bzr
|     ret = run(*run_argv)
|   File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 654, in run_argv_aliases
|     return self.run(**all_cmd_args)
|   File "/usr/lib/python2.6/site-packages/bzrlib/builtins.py", line 970, in run
|     possible_transports=possible_transports)
|   File "/usr/lib/python2.6/site-packages/bzrlib/bundle/__init__.py", line 49, in read_mergeable_from_url
|     _do_directive)
|   File "/usr/lib/python2.6/site-packages/bzrlib/bundle/__init__.py", line 71, in read_mergeable_from_transport
|     redirected_transport)
|   File "/usr/lib/python2.6/site-packages/bzrlib/lazy_import.py", line 125, in __call__
|     return obj(*args, **kwargs)
|   File "/usr/lib/python2.6/site-packages/bzrlib/transport/__init__.py", line 1646, in do_catching_redirections
|     transport = redirected(transport, e, redirection_notice)
|   File "/usr/lib/python2.6/site-packages/bzrlib/bundle/__init__.py", line 62, in redirected_transport
|     note(redirection_notice)
|   File "/usr/lib/python2.6/site-packages/bzrlib/trace.py", line 117, in note
|     _bzr_logger.info(*args, **kwargs)
|   File "/usr/lib/python2.6/logging/__init__.py", line 1030, in info
|     self._log(INFO, msg, args, **kwargs)
|   File "/usr/lib/python2.6/logging/__init__.py", line 1142, in _log
|     record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, func, extra)
|   File "/usr/lib/python2.6/logging/__init__.py", line 1117, in makeRecord
|     rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
|   File "/usr/lib/python2.6/logging/__init__.py", line 272, in __init__
|     from multiprocessing import current_process
|   File "/usr/lib/python2.6/multiprocessing/__init__.py", line 63, in <module>
|     from multiprocessing.process import Process, current_process, active_children
| EOFError: EOF read where object expected
| 
| bzr 2.0.5 on python 2.6.2 (Linux-2.6.32.11-99.fc12.i686-i686-athlon-with-fedora-12-Constantine)
| arguments: ['/usr/bin/bzr', 'pull', '--overwrite', 'http://bzr.savannah.gnu.org/r/emacs/trunk']
| encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
| plugins:
|   launchpad            /usr/lib/python2.6/site-packages/bzrlib/plugins/launchpad [2.0.5]
|   netrc_credential_store /usr/lib/python2.6/site-packages/bzrlib/plugins/netrc_credential_store [2.0.5]
| 
| *** Bazaar has encountered an internal error.  This probably indicates a
|     bug in Bazaar.  You can help us fix it by filing a bug report at
|         https://bugs.launchpad.net/bzr/+filebug
|     including this traceback and a description of the problem.
| ERROR: Failed to pull
| Getting local revision...
| $ bzr revision-info -d /var/lib/hudson/jobs/emacs-trunk/workspace
| cyd@stupidchicken.com-20100711221453-4vuxwof0928ooq2v
| Finished: FAILURE
`----



-- 
Joakim Verona