Guix, Hydra & Nix

Guix, Hydra & Nix

[Sree Harsha Totakura]

Hi!

I have a couple of questions regarding Guix integration with Hydra server.

From the sources, I found that build-aux/hydra/*.scm convert packages into a
format hydra understands for building them.

Hydra refuses to install without nix-store.  So Nix has to be installed as well.
I'm guessing Hydra uses nix-store command to build and retrieve store paths for
packages given to it.  Is it true?  If so, how can hydra nix-store compile Guix
packages?

Also, does Hydra serve the narinfo files automatically for packages which are
built through it?  Or does it serve those files for all packages present in the
store, even the ones present in store before Hydra was installed?

The current git master of Hydra requires current git master of Nix.  But, it
FTBS for me.  See http://pastebin.com/3bdCUrJw and http://pastebin.com/Sa0bezZc.
 Any hints?

Regards,
Sree

Re: Guix, Hydra & Nix

[Ludovic Courtès]

Sree Harsha Totakura <sreeharsha@totakura.in> skribis:

> From the sources, I found that build-aux/hydra/*.scm convert packages into a
> format hydra understands for building them.

Yes.  Specifically, build-aux/hydra/*.scm return a list of “jobs”, where
each job is represented as an alist with a number of compulsory items,
such as ‘derivation’ (see ‘package->alist’.)

The result of those scripts is then consumed by Hydra’s
hydra-eval-guile-jobs:

  https://github.com/NixOS/hydra/blob/master/src/script/hydra-eval-guile-jobs.in

hydra-eval-guile-jobs is itself called by hydra-evaluator, from here:

  https://github.com/NixOS/hydra/blob/master/src/lib/Hydra/Helper/AddBuilds.pm

> Hydra refuses to install without nix-store.  So Nix has to be installed as well.
> I'm guessing Hydra uses nix-store command to build and retrieve store paths for
> packages given to it.  Is it true?

Yes.  ‘nix-store’ is essentially a client of {guix,nix}-daemon, just
like ‘guix build’.  Hydra passes ‘nix-store’ a .drv file name, a
low-level representation of the build process; ‘nix-store’ doesn’t have
to care whether it was produced by Guix or Nix, and can’t tell the
difference.  (Similarly, ‘guix build’ can be passed a .drv produced by
Nix.)

> If so, how can hydra nix-store compile Guix packages?

Just like this.  :-)

> Also, does Hydra serve the narinfo files automatically for packages which are
> built through it?  Or does it serve those files for all packages present in the
> store, even the ones present in store before Hydra was installed?

The latter, it serves everything; see:

  https://github.com/NixOS/hydra/blob/master/src/lib/Hydra/View/NARInfo.pm

> The current git master of Hydra requires current git master of Nix.  But, it
> FTBS for me.  See http://pastebin.com/3bdCUrJw and http://pastebin.com/Sa0bezZc.
>  Any hints?

(Please copy logs inline.)

I ignored the Docbook issue, and used a tarball from
<http://hydra.nixos.org/job/nix/trunk/tarball/latest> to work around the
Bison issue.

HTH!

Ludo’.

Re: Guix, Hydra & Nix

[Sree Harsha Totakura]

Hi Ludo,

Thank you very much.  Please ACK my understanding.

On 03/28/2014 10:26 PM, Ludovic Courtès wrote:
> Sree Harsha Totakura <sreeharsha@totakura.in> skribis:
> 
>> From the sources, I found that build-aux/hydra/*.scm convert packages into a
>> format hydra understands for building them.
> 
> Yes.  Specifically, build-aux/hydra/*.scm return a list of “jobs”, where
> each job is represented as an alist with a number of compulsory items,
> such as ‘derivation’ (see ‘package->alist’.)
> 

OK, so the list of jobs is what is termed as a jobset in Hydra.  Jobset is a
collection of jobs + configuration for when to rebuild the jobset and where to
get the jobset's inputs (sources).

The file build-aux/hydra/gnu-system.scm prepares the jobset which contains all
packages currently in Guix.  This jobset is configured in hydra.gnu.org to
initiate a build whenever there is a new commit in Guix source repository.

The file build-aux/hydra/guix.scm is a jobset to build Guix iteself.

> The result of those scripts is then consumed by Hydra’s
> hydra-eval-guile-jobs:
> 
>   https://github.com/NixOS/hydra/blob/master/src/script/hydra-eval-guile-jobs.in
> 
> hydra-eval-guile-jobs is itself called by hydra-evaluator, from here:
> 
>   https://github.com/NixOS/hydra/blob/master/src/lib/Hydra/Helper/AddBuilds.pm
> 

Aye, this is where the magic happens.  So it looks at the file extension of the
jobset's recipe and if it is .scm, then it uses Guix.

>> Hydra refuses to install without nix-store.  So Nix has to be installed as well.
>> I'm guessing Hydra uses nix-store command to build and retrieve store paths for
>> packages given to it.  Is it true?
> 
> Yes.  ‘nix-store’ is essentially a client of {guix,nix}-daemon, just
> like ‘guix build’.  Hydra passes ‘nix-store’ a .drv file name, a
> low-level representation of the build process; ‘nix-store’ doesn’t have
> to care whether it was produced by Guix or Nix, and can’t tell the
> difference.  (Similarly, ‘guix build’ can be passed a .drv produced by
> Nix.)
> 
>> If so, how can hydra nix-store compile Guix packages?
> 
> Just like this.  :-)

Ah, so the derivations, when sent to {guix,nix}-daemon are expressed in the same
format?  `guix build' converts our guile code into this format.

So, when Hydra sees a .scm file as input, it asks Guix to convert the jobset
into derivations which are then sent to the {guix,nix}-daemon via 'nix-store'.

Does this mean that {guix,nix}-daemon are interchangeable?  Can Hydra run with
the setup: guix clients, nix clients, guix-daemon, Hydra?  Or does it need a
nix-daemon?

Sree

Re: Guix, Hydra & Nix

[Ludovic Courtès]

Hello!

Sree Harsha Totakura <sreeharsha@totakura.in> skribis:

> On 03/28/2014 10:26 PM, Ludovic Courtès wrote:
>> Sree Harsha Totakura <sreeharsha@totakura.in> skribis:
>> 
>>> From the sources, I found that build-aux/hydra/*.scm convert packages into a
>>> format hydra understands for building them.
>> 
>> Yes.  Specifically, build-aux/hydra/*.scm return a list of “jobs”, where
>> each job is represented as an alist with a number of compulsory items,
>> such as ‘derivation’ (see ‘package->alist’.)
>> 
>
> OK, so the list of jobs is what is termed as a jobset in Hydra.  Jobset is a
> collection of jobs + configuration for when to rebuild the jobset and where to
> get the jobset's inputs (sources).
>
> The file build-aux/hydra/gnu-system.scm prepares the jobset which contains all
> packages currently in Guix.  This jobset is configured in hydra.gnu.org to
> initiate a build whenever there is a new commit in Guix source repository.
>
> The file build-aux/hydra/guix.scm is a jobset to build Guix iteself.

Yes.

>> The result of those scripts is then consumed by Hydra’s
>> hydra-eval-guile-jobs:
>> 
>>   https://github.com/NixOS/hydra/blob/master/src/script/hydra-eval-guile-jobs.in
>> 
>> hydra-eval-guile-jobs is itself called by hydra-evaluator, from here:
>> 
>>   https://github.com/NixOS/hydra/blob/master/src/lib/Hydra/Helper/AddBuilds.pm
>> 
>
> Aye, this is where the magic happens.  So it looks at the file extension of the
> jobset's recipe and if it is .scm, then it uses Guix.

Exactly (strictly speaking, it uses Guile, Guix being just a Guile
library.)

>>> Hydra refuses to install without nix-store.  So Nix has to be installed as well.
>>> I'm guessing Hydra uses nix-store command to build and retrieve store paths for
>>> packages given to it.  Is it true?
>> 
>> Yes.  ‘nix-store’ is essentially a client of {guix,nix}-daemon, just
>> like ‘guix build’.  Hydra passes ‘nix-store’ a .drv file name, a
>> low-level representation of the build process; ‘nix-store’ doesn’t have
>> to care whether it was produced by Guix or Nix, and can’t tell the
>> difference.  (Similarly, ‘guix build’ can be passed a .drv produced by
>> Nix.)
>> 
>>> If so, how can hydra nix-store compile Guix packages?
>> 
>> Just like this.  :-)
>
> Ah, so the derivations, when sent to {guix,nix}-daemon are expressed in the same
> format?  `guix build' converts our guile code into this format.

Right.  Try cat $(guix build -d emacs) to see what it looks like.  Quoth
the manual (info "(guix) Programming Interface"):

     Lower-level APIs are available to interact with the daemon and the
  store.  To instruct the daemon to perform a build action, users actually
  provide it with a "derivation".  A derivation is a low-level
  representation of the build actions to be taken, and the environment in
  which they should occur—derivations are to package definitions what
  assembly is to C programs.

> So, when Hydra sees a .scm file as input, it asks Guix to convert the jobset
> into derivations which are then sent to the {guix,nix}-daemon via 'nix-store'.

Yes.

> Does this mean that {guix,nix}-daemon are interchangeable?

Yes and no.  The core functionality is the same, but the command-line
interface is different, and the default settings is different (for
instance, guix-daemon uses chroot builds by default, with nothing in
/bin, whereas nix-daemon uses non-chroot builds by default.)

> Can Hydra run with the setup: guix clients, nix clients, guix-daemon,
> Hydra?  Or does it need a nix-daemon?

No, no need for nix-daemon.

On hydra.gnu.org, we have guix-daemon running, and Guix and Nix clients,
the latter being there for Hydra.

HTH!

Ludo’.