Cuirass changes

Cuirass changes

[Clément Lassieur]

[-- Attachment #1: Type: text/plain, Size: 2462 bytes --]

Hi,

I'm currently working on Cuirass, so that it looks a bit more like
Hydra.  The changes include:

1. Adding support for multiple inputs.  Currently Cuirass only supports
one input per specification, which has to be the Guix git repository.
But one might want inputs for:
  - a repository containing the 'proc' that returns the jobs,
  - a repository containing extra packages (GUIX_PACKAGE_PATH).
Those inputs would be fetched at INTERVAL as well, and any change would
trigger an evaluation.  This is a follow-up of
https://lists.gnu.org/archive/html/guix-patches/2018-06/msg00311.html.

2. Removing the notion of 'project'.  Cuirass really has no project, and
what is called a 'project' is in fact a 'specification' (which Hydra
calls 'jobset').  And what is called a 'jobset' by Cuirass is in fact
the branch of the Guix input.  So the patch associates to the name
'jobset' what it really is: a specification.  A consequence is that the
'jobset' filtering of the API now filters by specification name, and it
is not possible anymore to filter by branch.  (But it was useless IMHO.)

3. Allowing any input to be in Guile's %load-path when 'proc' is called.
Usually, the only input that is in Guile's %load-path is Guix.  But
someone might want to use a custom library in 'proc'.  The alternative
is to just specify the Guix input that should be appended to Guile's
%load-path.

(I'm unsure about 3., I can't find any real use case.  That would just
make it more flexible I guess.)

Also, some specification fields (#:load-path-inputs,
#:package-path-inputs, #:proc-input) refer to the inputs by their name,
as Hydra does (with nixExprInput).  This makes it easy to do
configuration mistakes, and would ultimately require a configuration
syntax checker.  The only alternative I can think of involves
duplicating the inputs, which isn't good.

I'm sending this email because even though the patches are already done
and they work on my Cuirass instance, there are a few other things to do
before I can send them:
  - add a mechanism to update the database,
  - update the documentation.
So if you think I'm not on the right track, please let me know as soon
as possible :-)

I attached my configuration before those patches and the one after them,
so that you can see how the new configuration would look like.  The work
in progress is available at
https://git.lassieur.org/cgit/cuirass.git/log/.

Clément

[-- Attachment #2: old.scm --]
[-- Type: application/octet-stream, Size: 1145 bytes --]

(list
 '((#:name . "guix-modular-savannah")
   (#:url . "git://git.savannah.gnu.org/guix.git")
   (#:load-path . ".")
   (#:file . "build-aux/cuirass/guix-modular.scm")
   (#:no-compile? #t)
   (#:proc . cuirass-jobs)
   (#:arguments (systems "x86_64-linux"))
   (#:branch . "master"))
 '((#:name . "guix-modular-clem")
   (#:url . "git://git.lassieur.org/guix.git")
   (#:load-path . ".")
   (#:file . "build-aux/cuirass/guix-modular.scm")
   (#:no-compile? #t)
   (#:proc . cuirass-jobs)
   (#:arguments (systems "x86_64-linux"))
   (#:branch . "master"))
 '((#:name . "guix-manifest-savannah")
   (#:url . "git://git.savannah.gnu.org/guix.git")
   (#:load-path . ".")
   (#:file .
    #$(local-file
       "/home/clement/.clem/guix/cuirass-derivations.scm"))
   (#:no-compile? #t)
   (#:proc . drv-list)
   (#:arguments . ())
   (#:branch . "master"))
 '((#:name . "guix-manifest-clem")
   (#:url . "git://git.lassieur.org/guix.git")
   (#:load-path . ".")
   (#:file .
    #$(local-file
       "/home/clement/.clem/guix/cuirass-derivations.scm"))
   (#:no-compile? #t)
   (#:proc . drv-list)
   (#:arguments . ())
   (#:branch . "master")))

[-- Attachment #3: new.scm --]
[-- Type: application/octet-stream, Size: 2825 bytes --]

(list
 '((#:name . "guix-modular-savannah")
   (#:load-path-inputs . ("guix-modular"))
   (#:package-path-inputs . ())
   (#:proc-input . "guix-modular")
   (#:proc-path . "build-aux/cuirass/guix-modular.scm")
   (#:proc . cuirass-jobs)
   (#:proc-args (systems "x86_64-linux"))
   (#:inputs . (((#:name . "guix-modular") ;used by guix-modular.scm
                 (#:url . "git://git.savannah.gnu.org/guix.git")
                 (#:load-path . ".")
                 (#:branch . "master")
                 (#:no-compile? . #t)))))
 '((#:name . "guix-modular-clem")
   (#:load-path-inputs . ("guix-modular"))
   (#:package-path-inputs . ())
   (#:proc-input . "guix-modular")
   (#:proc-path . "build-aux/cuirass/guix-modular.scm")
   (#:proc . cuirass-jobs)
   (#:proc-args (systems "x86_64-linux"))
   (#:inputs . (((#:name . "guix-modular") ;used by guix-modular.scm
                 (#:url . "git://git.lassieur.org/guix.git")
                 (#:load-path . ".")
                 (#:branch . "master")
                 (#:no-compile? . #t)))))
 '((#:name . "guix-manifest-savannah")
   (#:load-path-inputs . ("guix"))
   (#:package-path-inputs . ("packages"))
   (#:proc-input . "conf")
   (#:proc-path . "guix/cuirass-derivations.scm")
   (#:proc . drv-list)
   (#:proc-args . ())
   (#:inputs . (((#:name . "guix")
                 (#:url . "git://git.savannah.gnu.org/guix.git")
                 (#:load-path . ".")
                 (#:branch . "master")
                 (#:no-compile? . #t))
                ((#:name . "conf")
                 (#:url . "git://git.lassieur.org/emacs.git")
                 (#:load-path . ".")
                 (#:branch . "master")
                 (#:no-compile? . #t))
                ((#:name . "packages")
                 (#:url . "git://git.lassieur.org/packages.git")
                 (#:load-path . ".")
                 (#:branch . "master")
                 (#:no-compile? . #t)))))
 '((#:name . "guix-manifest-clem")
   (#:load-path-inputs . ("guix-clem"))
   (#:package-path-inputs . ("packages"))
   (#:proc-input . "conf")
   (#:proc-path . "guix/cuirass-derivations.scm")
   (#:proc . drv-list)
   (#:proc-args . ())
   (#:inputs . (((#:name . "guix-clem")
                 (#:url . "git://git.lassieur.org/guix.git")
                 (#:load-path . ".")
                 (#:branch . "master")
                 (#:no-compile? . #t))
                ((#:name . "conf")
                 (#:url . "git://git.lassieur.org/emacs.git")
                 (#:load-path . ".")
                 (#:branch . "master")
                 (#:no-compile? . #t))
                ((#:name . "packages")
                 (#:url . "git://git.lassieur.org/packages.git")
                 (#:load-path . ".")
                 (#:branch . "master")
                 (#:no-compile? . #t))))))

Re: Cuirass changes

[Ludovic Courtès]

Hi Clément,

Sorry for the delay.  Lots of exciting stuff in this message!

Clément Lassieur <clement@lassieur.org> skribis:

> 1. Adding support for multiple inputs.  Currently Cuirass only supports
> one input per specification, which has to be the Guix git repository.
> But one might want inputs for:
>   - a repository containing the 'proc' that returns the jobs,
>   - a repository containing extra packages (GUIX_PACKAGE_PATH).
> Those inputs would be fetched at INTERVAL as well, and any change would
> trigger an evaluation.  This is a follow-up of
> https://lists.gnu.org/archive/html/guix-patches/2018-06/msg00311.html.

Excellent.  I think the polling interval should be per input, while
we’re at it.

> 2. Removing the notion of 'project'.  Cuirass really has no project, and
> what is called a 'project' is in fact a 'specification' (which Hydra
> calls 'jobset').  And what is called a 'jobset' by Cuirass is in fact
> the branch of the Guix input.  So the patch associates to the name
> 'jobset' what it really is: a specification.  A consequence is that the
> 'jobset' filtering of the API now filters by specification name, and it
> is not possible anymore to filter by branch.  (But it was useless IMHO.)

That makes sense to me.

> 3. Allowing any input to be in Guile's %load-path when 'proc' is called.
> Usually, the only input that is in Guile's %load-path is Guix.  But
> someone might want to use a custom library in 'proc'.  The alternative
> is to just specify the Guix input that should be appended to Guile's
> %load-path.
>
> (I'm unsure about 3., I can't find any real use case.  That would just
> make it more flexible I guess.)

We can also wait until there’s a real use case, it’s often a good idea.
:-)

> Also, some specification fields (#:load-path-inputs,
> #:package-path-inputs, #:proc-input) refer to the inputs by their name,
> as Hydra does (with nixExprInput).

Nice!

> This makes it easy to do configuration mistakes, and would ultimately
> require a configuration syntax checker.  The only alternative I can
> think of involves duplicating the inputs, which isn't good.

Good.

> I'm sending this email because even though the patches are already done
> and they work on my Cuirass instance, there are a few other things to do
> before I can send them:
>   - add a mechanism to update the database,
>   - update the documentation.
> So if you think I'm not on the right track, please let me know as soon
> as possible :-)

Looks like you’re on the right track.  You submitted changes to
guix-patches in the meantime so I’ll take a look at these.

> I attached my configuration before those patches and the one after them,
> so that you can see how the new configuration would look like.  The work
> in progress is available at
> https://git.lassieur.org/cgit/cuirass.git/log/.

Neat.

>  '((#:name . "guix-manifest-savannah")
>    (#:load-path-inputs . ("guix"))
>    (#:package-path-inputs . ("packages"))
>    (#:proc-input . "conf")
>    (#:proc-path . "guix/cuirass-derivations.scm")
>    (#:proc . drv-list)
>    (#:proc-args . ())
>    (#:inputs . (((#:name . "guix")
>                  (#:url . "git://git.savannah.gnu.org/guix.git")
>                  (#:load-path . ".")
>                  (#:branch . "master")
>                  (#:no-compile? . #t))
>                 ((#:name . "conf")
>                  (#:url . "git://git.lassieur.org/emacs.git")
>                  (#:load-path . ".")
>                  (#:branch . "master")
>                  (#:no-compile? . #t))
>                 ((#:name . "packages")
>                  (#:url . "git://git.lassieur.org/packages.git")
>                  (#:load-path . ".")
>                  (#:branch . "master")
>                  (#:no-compile? . #t)))))

Really cool example, and exactly what we were missing from Hydra.

Thanks a lot for working on this!

Ludo’.

Re: Cuirass changes

[Ricardo Wurmus]

Hi Clément,

Ludovic Courtès <ludo@gnu.org> writes:

> Lots of exciting stuff in this message!

Indeed!

I’m supportive of all of the proposed changes.  My only concern is about
potential conflicts with Tatiana’s work on the web interface.  Do you
think we could merge Tatiana’s patches first?

-- 
Ricardo

Re: Cuirass changes

[Clément Lassieur]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Clément,
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Lots of exciting stuff in this message!
>
> Indeed!

Thank you :-)

> I’m supportive of all of the proposed changes.  My only concern is about
> potential conflicts with Tatiana’s work on the web interface.  Do you
> think we could merge Tatiana’s patches first?

I think we should merge whichever patchset is ready first.  I will
surely help with the conflicts resolution, but it shouldn't be too
difficult, because we didn't work on the same parts.  It's a matter of
renaming a few database fields, that's all.  (There is no git conflict.)

Clément

Re: Cuirass changes

[Clément Lassieur]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

>> 1. Adding support for multiple inputs.  Currently Cuirass only supports
>> one input per specification, which has to be the Guix git repository.
>> But one might want inputs for:
>>   - a repository containing the 'proc' that returns the jobs,
>>   - a repository containing extra packages (GUIX_PACKAGE_PATH).
>> Those inputs would be fetched at INTERVAL as well, and any change would
>> trigger an evaluation.  This is a follow-up of
>> https://lists.gnu.org/archive/html/guix-patches/2018-06/msg00311.html.
>
> Excellent.  I think the polling interval should be per input, while
> we’re at it.

Indeed, but that would complicate things.  Would it make sense to do it
in another commit?

>> 3. Allowing any input to be in Guile's %load-path when 'proc' is called.
>> Usually, the only input that is in Guile's %load-path is Guix.  But
>> someone might want to use a custom library in 'proc'.  The alternative
>> is to just specify the Guix input that should be appended to Guile's
>> %load-path.
>>
>> (I'm unsure about 3., I can't find any real use case.  That would just
>> make it more flexible I guess.)
>
> We can also wait until there’s a real use case, it’s often a good idea.
> :-)

I finally opted for a flexible %load-path.  I think it also makes the
structure of the specification more transparent and understandable: it's
very clear that the input will be used as the load path, it's less
magical.  But if you object I'd happily change it.

>> I'm sending this email because even though the patches are already done
>> and they work on my Cuirass instance, there are a few other things to do
>> before I can send them:
>>   - add a mechanism to update the database,
>>   - update the documentation.
>> So if you think I'm not on the right track, please let me know as soon
>> as possible :-)
>
> Looks like you’re on the right track.  You submitted changes to
> guix-patches in the meantime so I’ll take a look at these.

Cool!  Thank you :-)

Clément

Re: Cuirass changes

[Ludovic Courtès]

Clément Lassieur <clement@lassieur.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>>> 1. Adding support for multiple inputs.  Currently Cuirass only supports
>>> one input per specification, which has to be the Guix git repository.
>>> But one might want inputs for:
>>>   - a repository containing the 'proc' that returns the jobs,
>>>   - a repository containing extra packages (GUIX_PACKAGE_PATH).
>>> Those inputs would be fetched at INTERVAL as well, and any change would
>>> trigger an evaluation.  This is a follow-up of
>>> https://lists.gnu.org/archive/html/guix-patches/2018-06/msg00311.html.
>>
>> Excellent.  I think the polling interval should be per input, while
>> we’re at it.
>
> Indeed, but that would complicate things.  Would it make sense to do it
> in another commit?

Sure, this can definitely be done later, I was just mentioning it in
passing.

Ludo’.