Continuous Integration for package.el Packages

Continuous Integration for package.el Packages

[Moritz Ulrich]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256


Hello,

During my efforts to integrate package.el packages into the Nix[1]
package manager, I started running automatic builds on all packages
available in GNU ELPA and Marmalade packages.

Nix is a pretty good fit for this kind of tests because it limits access
to all resources. This prevents leaking of the host environment into the
build, preventing implicit dependencies.

The results are available on my Hydra instance[2]. Most build errors are
caused by missing 'Version' headers. Some others fail because they
depend on packages neither available in ELPA nor in Marmalade (visible
in the 'Evaluation Errors' tab).

The code implementing all this is available on Github[3]. A cronjob
fetches new releases every four hours, Hydra starts building soon after.

What do you think of this? Is it useful? Does your build fails, even if
it works on your local Emacs instance? Please tell me!


Cheers,
Moritz Ulrich


[1]: http://nixos.org/
[2]: http://hydra.tarn-vedra.de/jobset/package-el-packages/next#tabs-status
[3]: https://github.com/NixOS/nixpkgs/pull/664

- -- 
Moritz Ulrich
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)

iQIcBAEBCAAGBQJR2VJCAAoJEKnhzHnsv6QyyRkQAKTWZLRLialJeWKBgOu4TTft
A0xN772SNDgfe775O+RFNVpI850npM72JgTAVUtphJtE9amd7ovl7uBOZ3bo6TPF
XdsQzDRZtV5pMNur6C8JQijyNtVwcMYivOboZeTe/z1qsuGGVsxaAnUI2wYeYVRr
babNWMkgaMN82U7xPdLdBvk9hqelpPRhuOiqF4A9DMxj55OJolRuFGssBSlBMFpn
ql3UsumfhgoZ1ZFbLl8qr0X/g92Rivr6FHLZvcetnpL0p6eckxQMRH/fopKbrEJU
5Ou6luTTVHpi8E7obUpChEerXC+RJ/4OJuXe4zXPGoBHbWYBCfyKj2kK8w+8KwWA
Ub3cF1A5jTpPPx4HrCU9TwVonSmJ8H2STTDc/9Qy5Ry5ElyHexPTKzuZoETe6uKV
t+GCOUAfH4KjCviDHyw70PAt3ccmqq8b5kzfNmRTSnnHaGuK1+jUdJYfrYrYUs9y
cJikZm+QmpAvj4fUkAmoCBf0wVtMHIwuNaxtJFQ3bLGOJ/YTCxQgw2SvE3YqSFmD
oPUo9vIWs2cnom91YTF+YEUtnwqbROF4IEu0EsvQMbHl0rtp6Ww3k9oUxUbVEm2c
yV6l8QLz58PQnn4vaOs9rVPr7qJOzDb/Z1W6b5Lr55r663n78ofy5SL3Jmz+TPtZ
5Zkr0ZsNi2QOcyd6+dUp
=a7Ad
-----END PGP SIGNATURE-----

Re: Continuous Integration for package.el Packages

[Dmitry Gutov]

Moritz Ulrich <moritz@tarn-vedra.de> writes:
> What do you think of this? Is it useful? Does your build fails, even if
> it works on your local Emacs instance? Please tell me!

My packages seem to build fine, thanks. I don't know how useful this is
going to be, though, without email notifications to package authors, or
something similar.

I do have to ask: where does this package [version] come from?

http://hydra.tarn-vedra.de/build/13352

It's not on Marmalade or GNU ELPA. 

Re: Continuous Integration for package.el Packages

[Moritz Ulrich]

Dmitry Gutov writes:

> Moritz Ulrich <moritz@tarn-vedra.de> writes:
>> What do you think of this? Is it useful? Does your build fails, even if
>> it works on your local Emacs instance? Please tell me!
>
> My packages seem to build fine, thanks. I don't know how useful this is
> going to be, though, without email notifications to package authors, or
> something similar.

Hydra can do this, it's just a matter of extracting the authors & mail
addresses and adding them to the `meta' property of the package.

> I do have to ask: where does this package [version] come from?
>
> http://hydra.tarn-vedra.de/build/13352
>
> It's not on Marmalade or GNU ELPA. 

Sorry, forgot to mention this: Nix includes some Emacs packages which
are maintained by hand. I decided to keep them for legacy. They have a
lower priority when installing compared to package.el packages, so
keeping them won't hurt (package.el packages will always override them).

Cheers,
Moritz

-- 
Moritz Ulrich

Re: Continuous Integration for package.el Packages

[Sebastian Wiesner]

2013/7/7 Moritz Ulrich <moritz@tarn-vedra.de>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
>
> Hello,
>
> During my efforts to integrate package.el packages into the Nix[1]
> package manager, I started running automatic builds on all packages
> available in GNU ELPA and Marmalade packages.

Do you intend to test MELPA, too?   It builds snapshot versions, which
are more interesting to test.

> The results are available on my Hydra instance[2]. Most build errors are
> caused by missing 'Version' headers. Some others fail because they
> depend on packages neither available in ELPA nor in Marmalade (visible
> in the 'Evaluation Errors' tab).
>
> The code implementing all this is available on Github[3]. A cronjob
> fetches new releases every four hours, Hydra starts building soon after.
>
> What do you think of this? Is it useful? Does your build fails, even if
> it works on your local Emacs instance? Please tell me!

To be honest, it is not useful for me.

I already have a more powerful setup for automatically build and test
my Flycheck package.  It's based on Carton and Vagrant to create
automated and repeatable VMs for local unit testing and Travis CI to
build my package and run my ERT tests after every single push.

As long as you do not run my ERT tests with all necessary dependencies
(beyond just package.el deps), you don't tell me anything I didn't
already know.

Re: Continuous Integration for package.el Packages

[Richard Stallman]

        [ To any NSA and FBI agents reading my email: please consider
        [ whether defending the US Constitution against all enemies,
        [ foreign or domestic, requires you to follow Snowden's example.

    Nix is a pretty good fit for this kind of tests because it limits access
    to all resources. This prevents leaking of the host environment into the
    build, preventing implicit dependencies.

How about using Guix?  Guix is supposed to be similar to Nix,
but using Guile, and it is meant for GNU.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Continuous Integration for package.el Packages

[Moritz Ulrich]

Richard Stallman writes:

>         [ To any NSA and FBI agents reading my email: please consider
>         [ whether defending the US Constitution against all enemies,
>         [ foreign or domestic, requires you to follow Snowden's example.
>
>     Nix is a pretty good fit for this kind of tests because it limits access
>     to all resources. This prevents leaking of the host environment into the
>     build, preventing implicit dependencies.
>
> How about using Guix?  Guix is supposed to be similar to Nix,
> but using Guile, and it is meant for GNU.

The tests running on the Hydra instance are just a side effect of my
efforts to add the emacs packages to Nix (I'm a NixOS user). Using Guix
should be easy and almost a straight translation. I'm just not (yet)
familar with Guix to give this a try myself.

Maybe I will in the future.


-- 
Moritz Ulrich

Re: Continuous Integration for package.el Packages

[Moritz Ulrich]

Sebastian Wiesner writes:

> 2013/7/7 Moritz Ulrich <moritz@tarn-vedra.de>:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>>
>> Hello,
>>
>> During my efforts to integrate package.el packages into the Nix[1]
>> package manager, I started running automatic builds on all packages
>> available in GNU ELPA and Marmalade packages.
>
> Do you intend to test MELPA, too?   It builds snapshot versions, which
> are more interesting to test.

I'll see if I find time to setup MELPA too. You're right - it's more
interesting.

>> The results are available on my Hydra instance[2]. Most build errors are
>> caused by missing 'Version' headers. Some others fail because they
>> depend on packages neither available in ELPA nor in Marmalade (visible
>> in the 'Evaluation Errors' tab).
>>
>> The code implementing all this is available on Github[3]. A cronjob
>> fetches new releases every four hours, Hydra starts building soon after.
>>
>> What do you think of this? Is it useful? Does your build fails, even if
>> it works on your local Emacs instance? Please tell me!
>
> To be honest, it is not useful for me.
>
> I already have a more powerful setup for automatically build and test
> my Flycheck package.  It's based on Carton and Vagrant to create
> automated and repeatable VMs for local unit testing and Travis CI to
> build my package and run my ERT tests after every single push.
>
> As long as you do not run my ERT tests with all necessary dependencies
> (beyond just package.el deps), you don't tell me anything I didn't
> already know.

The code currently loads all .el files with 'test' in their name and
runs ert afterwards. This won't work with non-package.el dependencies
though. Nix blocks access on environment which isn't explicitly listed
in the buildInputs.

To add native dependencies, package.el would need a new dependency field
listing native dependencies. This won't be easy, but would make
installations way more reproducible.

-- 
Moritz Ulrich

Re: Continuous Integration for package.el Packages

[Moritz Ulrich]

Moritz Ulrich writes:

> Richard Stallman writes:
>
>>         [ To any NSA and FBI agents reading my email: please consider
>>         [ whether defending the US Constitution against all enemies,
>>         [ foreign or domestic, requires you to follow Snowden's example.
>>
>>     Nix is a pretty good fit for this kind of tests because it limits access
>>     to all resources. This prevents leaking of the host environment into the
>>     build, preventing implicit dependencies.
>>
>> How about using Guix?  Guix is supposed to be similar to Nix,
>> but using Guile, and it is meant for GNU.
>
> The tests running on the Hydra instance are just a side effect of my
> efforts to add the emacs packages to Nix (I'm a NixOS user). Using Guix
> should be easy and almost a straight translation. I'm just not (yet)
> familar with Guix to give this a try myself.
>
> Maybe I will in the future.

Also, guix currently can't live without Nix. Guix can do the high-level
work (generating a derivation file) but won't do all the builds and
stuff.


-- 
Moritz Ulrich

Re: Continuous Integration for package.el Packages

[Glenn Morris]

Just wanted to say that this looks very nice, thank you.

For the elpa.gnu.org packages, would it be worth having this on 
the official hydra instance, http://hydra.nixos.org/project/gnu ?

Maybe building each package from the emacs elpa branch against current
emacs trunk and emacs-24 branches. And separate "coverage" builds for
those packages with self-tests?

Re: Continuous Integration for package.el Packages

[Bastien]

Hi Moritz,

Moritz Ulrich <moritz@tarn-vedra.de> writes:

> The tests running on the Hydra instance are just a side effect of my
> efforts to add the emacs packages to Nix (I'm a NixOS user). Using Guix
> should be easy and almost a straight translation. I'm just not (yet)
> familar with Guix to give this a try myself.
>
> Maybe I will in the future.

I hope to learn more about Guix packages on the GNU hackers meeting,
this friday in Paris.  Then perhaps I will be able to help with this,
if you can guide me through the necessary steps.

Best,

-- 
 Bastien