Using cuirass to build your own manifest.

Using cuirass to build your own manifest.

[Mathieu Othacehe]

Hi,

Here's a small tutorial on how to setup cuirass to build your own
manifest.

I see two major reasons for this kind of setup:

* When you pull latest guix, hydra and bayfront may not have finished
  building all the packages you use.

* Hydra and bayfront won't build your custom packages.

For this reasons I installed cuirass to build my manifest, using a
configuration very similar to the one used on bayfront
(http://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/bayfront.scm).

I guess it may not be the easier way to do it so don't hesiste to
comment. Anyway, here's how I set it up.

1) config.scm

--8<---------------cut here---------------start------------->8---
(define %cuirass-specs
  ;; Cuirass specifications to build Guix.
  #~(list `((#:name . "guix")
            (#:url . "git://git.savannah.gnu.org/guix.git")
            (#:branch . "master")
            (#:no-compile? #t)
            (#:load-path . ".")
            ;; Eval function drv-list in file guix-drv.scm
            (#:proc . drv-list)
            (#:file . #$(local-file "/home/mathieu/conf/guix/cuirass/guix-drv.scm")))))
(services
 ...
 (service cuirass-service-type
          (cuirass-configuration
           (interval 30) ;; git pull guix repo every 30 seconds.
           (use-substitutes? #t)
           (port 8082)
           (load-path '("/home/mathieu/conf/guix/packages"
                        "/home/mathieu/conf/guix/common_packages"))
           (specifications %cuirass-specs))))
--8<---------------cut here---------------end--------------->8---

With this configuration, cuirass service will wake-up every 30
seconds. It will pull branch master of guix.git. If new commits have
appeared since last evaluation, a build will be triggered.

Cuirass will evaluate the function specified with #:proc in the file
#:file. This function is supposed to return the list of derivations you
want to build.

Note that if this list contains custom packages, you have to setup
load-path to point to the directories containing the custom packages
definitions.

2) guix-drv.scm

--8<---------------cut here---------------start------------->8---
(use-modules (guix config)
             (guix store)
             (guix grafts)
             (guix packages)
             (guix ui)
             (guix derivations)
             (guix monads)
             (guix profiles)
             (gnu packages)
             (srfi srfi-1))

(define (drv-package store package)
  (lambda ()
    `((#:job-name . ,(string-append
                      (package-name package)
                      "-"
                      (package-version package)
                      "-job"))
      (#:derivation . ,(derivation-file-name
                        (parameterize ((%graft? #f))
                          (package-derivation store package #:graft? #f)))))))

(define (drv-list store arguments)
  (let* ((manifest
         (load* "/home/mathieu/conf/guix/manifest.scm"
                (make-user-module
                 '((guix profiles) (gnu)))))
         (packages
          (map manifest-entry-item
               (manifest-entries manifest))))
    (parameterize ((%graft? #f))
      (map (lambda (package)
             (drv-package store package))
           (delete-duplicates! packages)))))
--8<---------------cut here---------------end--------------->8---

The drv-list procedure loads the file manifest.scm which content is
detailed below. The list produced by drv-list looks like :

--8<---------------cut here---------------start------------->8---
(((#:job-name . "acpi-1.7-job") (#:derivation
. "/gnu/store/r9s5x0ksj02hsw4n3acdxab8ggjp4z7y-acpi-1.7.drv")) ...)
--8<---------------cut here---------------end--------------->8---

3) manifest.scm

--8<---------------cut here---------------start------------->8---
(define (spec->packages spec)
  (call-with-values (lambda ()
                      (specification->package+output spec)) list))

(define packages-list
  '("acpi"
    "acpica"
    ...
    "yasm"
    "zip"
  ))

(packages->manifest
 (map spec->packages packages-list))
--8<---------------cut here---------------end--------------->8---

Now you may want to run cuirass on a dedicated machine and access
packages built by cuirass on other machines. You just need to run guix
publish to do that.

4) config.scm

--8<---------------cut here---------------start------------->8---
(services
    ...
    (guix-publish-service #:host "0.0.0.0"))
--8<---------------cut here---------------end--------------->8---

5) other machines config.scm

--8<---------------cut here---------------start------------->8---
(services
 (modify-services %base-services
                  (guix-service-type
                   config =>
                   (guix-configuration
                    (inherit config)
                    (substitute-urls '("https://bayfront.guixsd.org"
                                       "https://mirror.hydra.gnu.org"
                                       "https://cuirass-machine-name"))))))
--8<---------------cut here---------------end--------------->8---

Where "cuirass-machine-name" is the URL of the machine running cuirass.

Note that the order in substitute list has an importance. Here, the
substitutes from cuirass-machine-name are only downloaded if they do not
exist on bayfront and hydra.

Depending on the upload speed of your build machine you may want to put
it on top of the list or not ...

At last, you need to authorize "cuirass-machine-name" key with this
command :

--8<---------------cut here---------------start------------->8---
guix archive --authorize < cuirass-machine-name.pub
--8<---------------cut here---------------end--------------->8---

You can refer to
https://www.gnu.org/software/guix/manual/html_node/Substitutes.html#Substitutes
for precisions.

That's it, I hope it will be helful.

To finish, the downsides of this setup are :
* You need to keep your manifest up-to-date on your build machine.
* There are no easy ways to track cuirass status but to do some sql on
cuirass database.

Besides that, it's working fine.

Happy building,

Mathieu

Re: Using cuirass to build your own manifest.

[ng0]

Mathieu Othacehe transcribed 5.9K bytes:
> 
> Hi,
> 
> Here's a small tutorial on how to setup cuirass to build your own
> manifest.

Nice :) I am working on a similar documentation, it would just extend
this by providing examples of how to do this via .onion, via OpenNIC dns
name, and inside the freifunk meshnetwork.
If I mention you as the original author may I just extend the text, and
add my changes?

> I see two major reasons for this kind of setup:
> 
> * When you pull latest guix, hydra and bayfront may not have finished
>   building all the packages you use.
> 
> * Hydra and bayfront won't build your custom packages.
> 
> For this reasons I installed cuirass to build my manifest, using a
> configuration very similar to the one used on bayfront
> (http://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/bayfront.scm).
> 
> I guess it may not be the easier way to do it so don't hesiste to
> comment. Anyway, here's how I set it up.
> 
> 1) config.scm
> 
> --8<---------------cut here---------------start------------->8---
> (define %cuirass-specs
>   ;; Cuirass specifications to build Guix.
>   #~(list `((#:name . "guix")
>             (#:url . "git://git.savannah.gnu.org/guix.git")
>             (#:branch . "master")
>             (#:no-compile? #t)
>             (#:load-path . ".")
>             ;; Eval function drv-list in file guix-drv.scm
>             (#:proc . drv-list)
>             (#:file . #$(local-file "/home/mathieu/conf/guix/cuirass/guix-drv.scm")))))
> (services
>  ...
>  (service cuirass-service-type
>           (cuirass-configuration
>            (interval 30) ;; git pull guix repo every 30 seconds.
>            (use-substitutes? #t)
>            (port 8082)
>            (load-path '("/home/mathieu/conf/guix/packages"
>                         "/home/mathieu/conf/guix/common_packages"))
>            (specifications %cuirass-specs))))
> --8<---------------cut here---------------end--------------->8---
> 
> With this configuration, cuirass service will wake-up every 30
> seconds. It will pull branch master of guix.git. If new commits have
> appeared since last evaluation, a build will be triggered.
> 
> Cuirass will evaluate the function specified with #:proc in the file
> #:file. This function is supposed to return the list of derivations you
> want to build.
> 
> Note that if this list contains custom packages, you have to setup
> load-path to point to the directories containing the custom packages
> definitions.
> 
> 2) guix-drv.scm
> 
> --8<---------------cut here---------------start------------->8---
> (use-modules (guix config)
>              (guix store)
>              (guix grafts)
>              (guix packages)
>              (guix ui)
>              (guix derivations)
>              (guix monads)
>              (guix profiles)
>              (gnu packages)
>              (srfi srfi-1))
> 
> (define (drv-package store package)
>   (lambda ()
>     `((#:job-name . ,(string-append
>                       (package-name package)
>                       "-"
>                       (package-version package)
>                       "-job"))
>       (#:derivation . ,(derivation-file-name
>                         (parameterize ((%graft? #f))
>                           (package-derivation store package #:graft? #f)))))))
> 
> (define (drv-list store arguments)
>   (let* ((manifest
>          (load* "/home/mathieu/conf/guix/manifest.scm"
>                 (make-user-module
>                  '((guix profiles) (gnu)))))
>          (packages
>           (map manifest-entry-item
>                (manifest-entries manifest))))
>     (parameterize ((%graft? #f))
>       (map (lambda (package)
>              (drv-package store package))
>            (delete-duplicates! packages)))))
> --8<---------------cut here---------------end--------------->8---
> 
> The drv-list procedure loads the file manifest.scm which content is
> detailed below. The list produced by drv-list looks like :
> 
> --8<---------------cut here---------------start------------->8---
> (((#:job-name . "acpi-1.7-job") (#:derivation
> . "/gnu/store/r9s5x0ksj02hsw4n3acdxab8ggjp4z7y-acpi-1.7.drv")) ...)
> --8<---------------cut here---------------end--------------->8---
> 
> 3) manifest.scm
> 
> --8<---------------cut here---------------start------------->8---
> (define (spec->packages spec)
>   (call-with-values (lambda ()
>                       (specification->package+output spec)) list))
> 
> (define packages-list
>   '("acpi"
>     "acpica"
>     ...
>     "yasm"
>     "zip"
>   ))
> 
> (packages->manifest
>  (map spec->packages packages-list))
> --8<---------------cut here---------------end--------------->8---
> 
> Now you may want to run cuirass on a dedicated machine and access
> packages built by cuirass on other machines. You just need to run guix
> publish to do that.
> 
> 4) config.scm
> 
> --8<---------------cut here---------------start------------->8---
> (services
>     ...
>     (guix-publish-service #:host "0.0.0.0"))
> --8<---------------cut here---------------end--------------->8---
> 
> 5) other machines config.scm
> 
> --8<---------------cut here---------------start------------->8---
> (services
>  (modify-services %base-services
>                   (guix-service-type
>                    config =>
>                    (guix-configuration
>                     (inherit config)
>                     (substitute-urls '("https://bayfront.guixsd.org"
>                                        "https://mirror.hydra.gnu.org"
>                                        "https://cuirass-machine-name"))))))
> --8<---------------cut here---------------end--------------->8---
> 
> Where "cuirass-machine-name" is the URL of the machine running cuirass.
> 
> Note that the order in substitute list has an importance. Here, the
> substitutes from cuirass-machine-name are only downloaded if they do not
> exist on bayfront and hydra.
> 
> Depending on the upload speed of your build machine you may want to put
> it on top of the list or not ...
> 
> At last, you need to authorize "cuirass-machine-name" key with this
> command :
> 
> --8<---------------cut here---------------start------------->8---
> guix archive --authorize < cuirass-machine-name.pub
> --8<---------------cut here---------------end--------------->8---
> 
> You can refer to
> https://www.gnu.org/software/guix/manual/html_node/Substitutes.html#Substitutes
> for precisions.
> 
> That's it, I hope it will be helful.
> 
> To finish, the downsides of this setup are :
> * You need to keep your manifest up-to-date on your build machine.
> * There are no easy ways to track cuirass status but to do some sql on
> cuirass database.
> 
> Besides that, it's working fine.
> 
> Happy building,
> 
> Mathieu
> 

Re: Using cuirass to build your own manifest.

[Mathieu Othacehe]

> If I mention you as the original author may I just extend the text, and
> add my changes?

Sure, fell free to do it !

Mathieu

Re: Using cuirass to build your own manifest.

[Ludovic Courtès]

Hi!

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> Here's a small tutorial on how to setup cuirass to build your own
> manifest.
>
> I see two major reasons for this kind of setup:
>
> * When you pull latest guix, hydra and bayfront may not have finished
>   building all the packages you use.
>
> * Hydra and bayfront won't build your custom packages.
>
> For this reasons I installed cuirass to build my manifest, using a
> configuration very similar to the one used on bayfront
> (http://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/bayfront.scm).

This is a very cool hack!

It might make sense to turn it into a service in GuixSD proper, because
I suspect it’s a use case that people may be interested in.  So you
would write, say:

  (service auto-build-service
           (auto-build-service-configuration
             (manifest (local-file …))
             (load-path '("/home/alice/…"))))

and it would do the right thing.  WDYT?

BTW, I think it’s the kind of experience report that would be worth
turning into a blog post on our web site.  It’s always interesting to
see how people solve specific problems with the tools.

Thanks!

Ludo’.

Re: Using cuirass to build your own manifest.

[Mathieu Othacehe]

> This is a very cool hack!

Thanks you !

>
> It might make sense to turn it into a service in GuixSD proper, because
> I suspect it’s a use case that people may be interested in.  So you
> would write, say:
>
>   (service auto-build-service
>            (auto-build-service-configuration
>              (manifest (local-file …))
>              (load-path '("/home/alice/…"))))
>
> and it would do the right thing.  WDYT?

Sure, it could be nice, I'll try to come up with a patch for this service.

>
> BTW, I think it’s the kind of experience report that would be worth
> turning into a blog post on our web site.  It’s always interesting to
> see how people solve specific problems with the tools.

I'll be glad to turn this small tuto into an article now or when the
service will be available, as you wish.

Mathieu

Re: Using cuirass to build your own manifest.

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1696 bytes --]

Hi Mathieu!

Cool stuff!  It's very interesting to see how others do things.  I have
a basic Scheme question about what you did, though.

Mathieu Othacehe <m.othacehe@gmail.com> writes:

> (define (drv-package store package)
>   (lambda ()
>     `((#:job-name . ,(string-append
>                       (package-name package)
>                       "-"
>                       (package-version package)
>                       "-job"))
>       (#:derivation . ,(derivation-file-name
>                         (parameterize ((%graft? #f))
>                           (package-derivation store package #:graft? #f)))))))

It looks like drv-package is a procedure which returns a procedure.  Is
that correct?

> (define (drv-list store arguments)
>   (let* ((manifest
>          (load* "/home/mathieu/conf/guix/manifest.scm"
>                 (make-user-module
>                  '((guix profiles) (gnu)))))
>          (packages
>           (map manifest-entry-item
>                (manifest-entries manifest))))
>     (parameterize ((%graft? #f))
>       (map (lambda (package)
>              (drv-package store package))
>            (delete-duplicates! packages)))))
>
>
> The drv-list procedure loads the file manifest.scm which content is
> detailed below. The list produced by drv-list looks like :
>
> (((#:job-name . "acpi-1.7-job") (#:derivation
> . "/gnu/store/r9s5x0ksj02hsw4n3acdxab8ggjp4z7y-acpi-1.7.drv")) ...)

If drv-package is a procedure that returns a procedure, then it seems to
me like the the list produced by drv-list should look like a list of
procedures, but that isn't what you wrote.  What am I misunderstanding?

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Using cuirass to build your own manifest.

[Ludovic Courtès]

Hi Mathieu,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

>> This is a very cool hack!
>
> Thanks you !
>
>>
>> It might make sense to turn it into a service in GuixSD proper, because
>> I suspect it’s a use case that people may be interested in.  So you
>> would write, say:
>>
>>   (service auto-build-service
>>            (auto-build-service-configuration
>>              (manifest (local-file …))
>>              (load-path '("/home/alice/…"))))
>>
>> and it would do the right thing.  WDYT?
>
> Sure, it could be nice, I'll try to come up with a patch for this service.
>
>>
>> BTW, I think it’s the kind of experience report that would be worth
>> turning into a blog post on our web site.  It’s always interesting to
>> see how people solve specific problems with the tools.
>
> I'll be glad to turn this small tuto into an article now or when the
> service will be available, as you wish.

Excellent.  You can write the article anytime you wish, as you see fit.

The blog posts are located at:

  https://git.savannah.gnu.org/cgit/guix/guix-artwork.git/tree/website/posts

and you can write it as Markdown (.md file).

When you write it you can email it to interested readers (count me in)
if you’d rather avoid spoiling it by posting it here.  :-)

Thanks,
Ludo’.

Re: Using cuirass to build your own manifest.

[Mathieu Othacehe]

Hi Chris,

> Hi Mathieu!
>
> Cool stuff!  It's very interesting to see how others do things.  I have
> a basic Scheme question about what you did, though.

Thank you !

> It looks like drv-package is a procedure which returns a procedure.  Is
> that correct?

Yes it is.

>> The drv-list procedure loads the file manifest.scm which content is
>> detailed below. The list produced by drv-list looks like :
>>
>> (((#:job-name . "acpi-1.7-job") (#:derivation
>> . "/gnu/store/r9s5x0ksj02hsw4n3acdxab8ggjp4z7y-acpi-1.7.drv")) ...)
>
> If drv-package is a procedure that returns a procedure, then it seems to
> me like the the list produced by drv-list should look like a list of
> procedures, but that isn't what you wrote.  What am I misunderstanding?

Yes you're right, the list produced by drv-list is in fact a list of
procedures, as dictated by cuirass:

((lambda () '((#:job-name . "acpi-1.7-job") (#:derivation . "/gnu/store/r9s5x0ksj02hsw4n3acdxab8ggjp4z7y-acpi-1.7.drv"))) ...)

Sorry for the mistake,

Mathieu

Re: Using cuirass to build your own manifest.

[Mathieu Lirzin]

Hi,

Mathieu Othacehe <m.othacehe@gmail.com> writes:

> Here's a small tutorial on how to setup cuirass to build your own
> manifest.

Amazing work! :)

> I see two major reasons for this kind of setup:
>
> * When you pull latest guix, hydra and bayfront may not have finished
>   building all the packages you use.
>
> * Hydra and bayfront won't build your custom packages.

This is definitely a use case that I expect to be common so I agree with
Ludo that would make sense to make a service for that.

> To finish, the downsides of this setup are :
> * You need to keep your manifest up-to-date on your build machine.

Do you have ideas/suggestions on how to improve that?

> * There are no easy ways to track cuirass status but to do some sql on
> cuirass database.

With the HTTP API that I hope will be implemented this summer, the
situation would hopefully be better.

Thank you for this tutorial.  This is really useful given the current
state of Cuirass documentation.  ;)

-- 
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

Re: Using cuirass to build your own manifest.

[Mathieu Othacehe]

Hi,

> Amazing work! :)

Thank you !

>> To finish, the downsides of this setup are :
>> * You need to keep your manifest up-to-date on your build machine.
>
> Do you have ideas/suggestions on how to improve that?

Well there is the option to pull a git repository containing the
manifest with mcron. Or maybe drv-list procedure could do the git pull
before loading the manifest ?

>
>> * There are no easy ways to track cuirass status but to do some sql on
>> cuirass database.
>
> With the HTTP API that I hope will be implemented this summer, the
> situation would hopefully be better.

I read your plans for GSOC, I'm really eager to see the result :)

Mathieu