[GSoC] Rewrite Hydra to be more integrated with Guix.

[GSoC] Rewrite Hydra to be more integrated with Guix.

[Mathieu Lirzin]

Hello fellow Guix Hackers,

Being currently a student which is due to make an internship, I intend
to turn this boring administrative injunction into an opportunity to
contribute to Guix by applying to Google Summer of Code which is
considered as an internship by University of Bordeaux standards.

Hydra is a Nix-based continuous build system which is used by Guix to
compile packages on different platforms and to distribute packages
substitutes.  With time, nix-daemon and guix-daemon are evolving
differently.  Hydra being heavily dependent on nix-daemon, Guix is not
able to use its newest versions.  Moreover there are some software
related performance issues (among others) in the current Guix
infrastructure that are unlikely to be solved considering the
foreignness of Hydra Perl implementation to Guix hackers.

In that context, I am willing to work on implementing a continous build
system similar to Hydra in Guile.

This GSoC will not likely succeed in implementing every features Hydra
is currently providing.  The objective is rather to create the basis
which will then allow further developpements to overcomes the present
difficulties.  To achieve this the following milestones (suggested by
Ludo) will be followed:

- Implementing a simple loop pulling Guix Git repository and building
  every packages.
   
- Adding a “job” abstraction to be able to build different Git branches.

- Adding support for a database to keep track of the build results with
  their associated commit, derivation and output.

- Adding a API over HTTP to get the build results remotely (ideally
  through an Emacs interface).

- Adding support for configuring and launching jobs remotely.

Everyone is welcome to provide its input in order to improve the roadmap
or commenting on the global picture.

I have added an entry for this project in the GSoC idea list:

  https://libreplanet.org/wiki/Group:Guix/GSoC-2016#Rewrite_Hydra_to_be_more_integrated_with_Guix

-- 
Mathieu Lirzin

Re: [GSoC] Rewrite Hydra to be more integrated with Guix.

[Alex Kost]

Mathieu Lirzin (2016-03-18 00:38 +0300) wrote:

> Hello fellow Guix Hackers,
>
> Being currently a student which is due to make an internship, I intend
> to turn this boring administrative injunction into an opportunity to
> contribute to Guix by applying to Google Summer of Code which is
> considered as an internship by University of Bordeaux standards.
>
> Hydra is a Nix-based continuous build system which is used by Guix to
> compile packages on different platforms and to distribute packages
> substitutes.  With time, nix-daemon and guix-daemon are evolving
> differently.  Hydra being heavily dependent on nix-daemon, Guix is not
> able to use its newest versions.  Moreover there are some software
> related performance issues (among others) in the current Guix
> infrastructure that are unlikely to be solved considering the
> foreignness of Hydra Perl implementation to Guix hackers.
>
> In that context, I am willing to work on implementing a continous build
> system similar to Hydra in Guile.

Aaaah!  It would be really great!  Thank you so much for beginning this
project!

> This GSoC will not likely succeed in implementing every features Hydra
> is currently providing.  The objective is rather to create the basis
> which will then allow further developpements to overcomes the present
> difficulties.  To achieve this the following milestones (suggested by
> Ludo) will be followed:
>
> - Implementing a simple loop pulling Guix Git repository and building
>   every packages.
>
> - Adding a “job” abstraction to be able to build different Git branches.
>
> - Adding support for a database to keep track of the build results with
>   their associated commit, derivation and output.
>
> - Adding a API over HTTP to get the build results remotely (ideally
>   through an Emacs interface).

I dream of a more feature-full API, as the current "M-x guix-hydra-…"
stuff is not very useful.

-- 
Alex

Re: [GSoC] Rewrite Hydra to be more integrated with Guix.

[Ludovic Courtès]

Alex Kost <alezost@gmail.com> skribis:

> Mathieu Lirzin (2016-03-18 00:38 +0300) wrote:

[...]

>> This GSoC will not likely succeed in implementing every features Hydra
>> is currently providing.  The objective is rather to create the basis
>> which will then allow further developpements to overcomes the present
>> difficulties.  To achieve this the following milestones (suggested by
>> Ludo) will be followed:
>>
>> - Implementing a simple loop pulling Guix Git repository and building
>>   every packages.
>>
>> - Adding a “job” abstraction to be able to build different Git branches.
>>
>> - Adding support for a database to keep track of the build results with
>>   their associated commit, derivation and output.
>>
>> - Adding a API over HTTP to get the build results remotely (ideally
>>   through an Emacs interface).
>
> I dream of a more feature-full API, as the current "M-x guix-hydra-…"
> stuff is not very useful.

What features do you have in mind?

Now is a good time to throw ideas at Mathieu!  ;-)

Ludo’.

Re: [GSoC] Rewrite Hydra to be more integrated with Guix.

[Alex Kost]

Ludovic Courtès (2016-03-18 23:55 +0300) wrote:

> Alex Kost <alezost@gmail.com> skribis:
>
>> Mathieu Lirzin (2016-03-18 00:38 +0300) wrote:
>
> [...]
>
>>> This GSoC will not likely succeed in implementing every features Hydra
>>> is currently providing.  The objective is rather to create the basis
>>> which will then allow further developpements to overcomes the present
>>> difficulties.  To achieve this the following milestones (suggested by
>>> Ludo) will be followed:
>>>
>>> - Implementing a simple loop pulling Guix Git repository and building
>>>   every packages.
>>>
>>> - Adding a “job” abstraction to be able to build different Git branches.
>>>
>>> - Adding support for a database to keep track of the build results with
>>>   their associated commit, derivation and output.
>>>
>>> - Adding a API over HTTP to get the build results remotely (ideally
>>>   through an Emacs interface).
>>
>> I dream of a more feature-full API, as the current "M-x guix-hydra-…"
>> stuff is not very useful.
>
> What features do you have in mind?

For example, the Hydra API does not provide a way to search for jobs as
we can do with web interface like this:

  http://hydra.gnu.org/search?query=emacs

Also there is no way to get any info on evaluations, so it's not
possible to compare evals: i.e., to see what new jobs were added, what
was failed or succeeded, etc.

-- 
Alex

Re: [GSoC] Rewrite Hydra to be more integrated with Guix.

[Ludovic Courtès]

Alex Kost <alezost@gmail.com> skribis:

> For example, the Hydra API does not provide a way to search for jobs as
> we can do with web interface like this:
>
>   http://hydra.gnu.org/search?query=emacs
>
> Also there is no way to get any info on evaluations, so it's not
> possible to compare evals: i.e., to see what new jobs were added, what
> was failed or succeeded, etc.

Indeed.  The web interface provides access to that, but not the API
apparently.

This part (the HTTP API) is almost outside the scope of this GSoC
proposal I think, but good to keep in mind.

Ludo’.

Re: [GSoC] Rewrite Hydra to be more integrated with Guix.

[Ludovic Courtès]

Hi!

Mathieu Lirzin <mthl@gnu.org> skribis:

> This GSoC will not likely succeed in implementing every features Hydra
> is currently providing.  The objective is rather to create the basis
> which will then allow further developpements to overcomes the present
> difficulties.  To achieve this the following milestones (suggested by
> Ludo) will be followed:
>
> - Implementing a simple loop pulling Guix Git repository and building
>   every packages.
>    
> - Adding a “job” abstraction to be able to build different Git branches.
>
> - Adding support for a database to keep track of the build results with
>   their associated commit, derivation and output.
>
> - Adding a API over HTTP to get the build results remotely (ideally
>   through an Emacs interface).
>
> - Adding support for configuring and launching jobs remotely.

As we discussed earlier, I think this is pretty cool!  :-)

The nice thing is that it should be possible to incrementally improve
things, always having something working and providing a real service.

The idea is to assume we’d keep using ‘guix offload’ to distribute work
to the build machines.  We know that it’s not perfect in terms of
efficiency because every store item has to travel back and forth between
the front-end and the build machines.  However, there’s no fully-baked
design to improve on this AFAIK, so it’s reasonable to assume that
improving that part is out of the scope of your project.

Anyway, I think everyone is welcome to make suggestions on how this
thing should work, or mistakes to avoid.  Feedback from Mark, Andreas,
and others who’ve dealt with Hydra or other CI systems is particularly
welcome!

Thoughts?

Ludo’.

Re: [GSoC] Rewrite Hydra to be more integrated with Guix.

[Andreas Enge]

Hello,

over some lunch I have been chatting with a famous Debian developer to under-
stand a bit how Debian handles their build process. What is interesting with
their approach is that the master manages a queue of packages to be built,
and the slaves connect to fetch work when they are ready (for instance, when
they have empty resources). With our offload mechanism, the master contacts
all the build slaves to schedule the work. This is not totally different,
but I wonder whether the Debian approach is not more flexible and more geared
towards a distributed approach. (Of course, in both cases the master machine
needs to keep a list of authorised build machines, so the degree of centra-
lisation is ultimately the same.)

In the end, I think it would be interesting to not use the offload approach,
where the offloading machine sends all inputs to the build machine, which
returns the result, so that more or less the complete set of binary packages
flows along the tree of build machines to the root and the other way round.
(As I recently understood during discussions with Ludovic, currently we have
a tree of height one: hydra is the root and all build machines are the leaves;
but the offloading mechanism could transparently work with trees of bigger
height, where hydra offloads to a root of a smaller build cluster, which
itself offloads to one of its subordinate build machines. And so on. The same
thing would also work in the Debian setting, assuming that build requests
come from the leaves and are forwarded towards the root, while a work package
is handed down the other way.) Now it would be interesting if machines just
got build jobs (or made requests) and looked for the necessary inputs inside
a big magma/cloud/gnunet/ipfs/mirror network, compiled the package, and
dropped the result into the same magma. Already in our current set-up, it
would be useful if hydra need not send the build inputs, but build machines
could just fetch them from an arbitrary nginx mirror, where they are cached.

Andreas

Re: [GSoC] Rewrite Hydra to be more integrated with Guix.

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> In the end, I think it would be interesting to not use the offload approach,
> where the offloading machine sends all inputs to the build machine, which
> returns the result, so that more or less the complete set of binary packages
> flows along the tree of build machines to the root and the other way round.
> (As I recently understood during discussions with Ludovic, currently we have
> a tree of height one: hydra is the root and all build machines are the leaves;
> but the offloading mechanism could transparently work with trees of bigger
> height, where hydra offloads to a root of a smaller build cluster, which
> itself offloads to one of its subordinate build machines. And so on. The same
> thing would also work in the Debian setting, assuming that build requests
> come from the leaves and are forwarded towards the root, while a work package
> is handed down the other way.) Now it would be interesting if machines just
> got build jobs (or made requests) and looked for the necessary inputs inside
> a big magma/cloud/gnunet/ipfs/mirror network, compiled the package, and
> dropped the result into the same magma. Already in our current set-up, it
> would be useful if hydra need not send the build inputs, but build machines
> could just fetch them from an arbitrary nginx mirror, where they are cached.

I agree.  However, I think there are a lot of unknowns, which makes it
unsuitable for a GSoC project.

That’s why I was suggesting that Mathieu do not consider workload
distribution (meaning we would keep using the existing offload
mechanism) as part of this project.

Ludo’.