[bug#32547] [PATCH] hydra: Add support for manifests.

[bug#32547] [PATCH] hydra: Add support for manifests.

[Clément Lassieur]

* build-aux/hydra/gnu-system.scm (arguments->manifests, manifests->packages):
New procedures.
(hydra-jobs): Add a "manifests" subset.
* doc/guix.texi (Continuous Integration): Update accordingly.
---
 build-aux/hydra/gnu-system.scm | 34 +++++++++++++++++++++
 doc/guix.texi                  | 56 +++++++++++++++++++++++-----------
 2 files changed, 72 insertions(+), 18 deletions(-)

diff --git a/build-aux/hydra/gnu-system.scm b/build-aux/hydra/gnu-system.scm
index b1554ced4..e8ce8eada 100644
--- a/build-aux/hydra/gnu-system.scm
+++ b/build-aux/hydra/gnu-system.scm
@@ -56,6 +56,7 @@
              (guix packages)
              (guix derivations)
              (guix monads)
+             (guix ui)
              ((guix licenses) #:select (gpl3+))
              ((guix utils) #:select (%current-system))
              ((guix scripts system) #:select (read-operating-system))
@@ -311,6 +312,30 @@ valid."
                           packages)))
                  #:select? (const #t)))           ;include hidden packages
 
+(define (arguments->manifests arguments)
+  "Return the list of manifests extracted from ARGUMENTS."
+  (map (match-lambda
+         ((input-name . relative-path)
+          (let* ((checkout (assq-ref arguments (string->symbol input-name)))
+                 (base (assq-ref checkout 'file-name)))
+            (in-vicinity base relative-path))))
+       (assq-ref arguments 'manifests)))
+
+(define (manifests->packages store manifests)
+  "Return the list of packages found in MANIFESTS."
+  (define (load-manifest manifest)
+    (save-module-excursion
+     (lambda ()
+       (set-current-module (make-user-module '((guix profiles) (gnu))))
+       (primitive-load manifest))))
+
+  (parameterize ((%graft? #f))
+    (delete-duplicates!
+     (map manifest-entry-item
+          (append-map (compose manifest-entries
+                               load-manifest)
+                      manifests)))))
+
 \f
 ;;;
 ;;; Hydra entry point.
@@ -323,6 +348,7 @@ valid."
       ("core" 'core)                              ; only build core packages
       ("hello" 'hello)                            ; only build hello
       (((? string?) (? string?) ...) 'list)       ; only build selected list of packages
+      ("manifests" 'manifests)                    ; only build packages in the list of manifests
       (_ 'all)))                                  ; build everything
 
   (define systems
@@ -419,6 +445,14 @@ valid."
                                                  package system))
                                   packages))
                          '()))
+                    ((manifests)
+                     ;; Build packages in the list of manifests.
+                     (let* ((manifests (arguments->manifests arguments))
+                            (packages (manifests->packages store manifests)))
+                       (map (lambda (package)
+                              (package-job store (job-name package)
+                                           package system))
+                            packages)))
                     (else
                      (error "unknown subset" subset))))
                 systems)))
diff --git a/doc/guix.texi b/doc/guix.texi
index d2d278df4..2b2fe6c93 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -17704,23 +17704,43 @@ The type of the Cuirass service.  Its value must be a
 @code{cuirass-configuration} object, as described below.
 @end defvr
 
-To add build jobs, you have to set the @code{specifications} field of
-the configuration.  Here is an example of a service defining a build job
-based on a specification that can be found in Cuirass source tree.  This
-service polls the Guix repository and builds a subset of the Guix
-packages, as prescribed in the @file{gnu-system.scm} example spec:
-
-@example
-(let ((spec #~((#:name . "guix")
-               (#:url . "git://git.savannah.gnu.org/guix.git")
-               (#:load-path . ".")
-               (#:file . "build-aux/cuirass/gnu-system.scm")
-               (#:proc . cuirass-jobs)
-               (#:arguments (subset . "hello"))
-               (#:branch . "master"))))
-  (service cuirass-service-type
-           (cuirass-configuration
-            (specifications #~(list '#$spec)))))
+To add build jobs, you have to set the @code{specifications} field of the
+configuration.  Here is an example of a service that polls the Guix repository
+and builds the packages from a manifest.  Some of the packages are defined in
+the @code{"custom-packages"} input, which is the equivalent of
+@code{GUIX_PACKAGE_PATH}.
+
+@example
+(define %cuirass-specs
+  #~(list
+     '((#:name . "my-manifest")
+       (#:load-path-inputs . ("guix"))
+       (#:package-path-inputs . ("custom-packages"))
+       (#:proc-input . "guix")
+       (#:proc-file . "build-aux/cuirass/gnu-system.scm")
+       (#:proc . cuirass-jobs)
+       (#:proc-args . ((subset . "manifests")
+                       (systems . ("x86_64-linux"))
+                       (manifests . (("config" . "guix/manifest.scm")))))
+       (#:inputs . (((#:name . "guix")
+                     (#:url . "git://git.savannah.gnu.org/guix.git")
+                     (#:load-path . ".")
+                     (#:branch . "master")
+                     (#:no-compile? . #t))
+                    ((#:name . "config")
+                     (#:url . "git://git.example.org/config.git")
+                     (#:load-path . ".")
+                     (#:branch . "master")
+                     (#:no-compile? . #t))
+                    ((#:name . "custom-packages")
+                     (#:url . "git://git.example.org/custom-packages.git")
+                     (#:load-path . ".")
+                     (#:branch . "master")
+                     (#:no-compile? . #t)))))))
+
+(service cuirass-service-type
+         (cuirass-configuration
+          (specifications %cuirass-specs)))
 @end example
 
 While information related to build jobs is located directly in the
@@ -17747,7 +17767,7 @@ Owner's group of the @code{cuirass} process.
 Number of seconds between the poll of the repositories followed by the
 Cuirass jobs.
 
-@item @code{database} (default: @code{"/var/run/cuirass/cuirass.db"})
+@item @code{database} (default: @code{"/var/lib/cuirass/cuirass.db"})
 Location of sqlite database which contains the build results and previously
 added specifications.
 
-- 
2.18.0

bug#32547: [PATCH] hydra: Add support for manifests.

[Clément Lassieur]

Clément Lassieur <clement@lassieur.org> writes:

> * build-aux/hydra/gnu-system.scm (arguments->manifests, manifests->packages):
> New procedures.
> (hydra-jobs): Add a "manifests" subset.
> * doc/guix.texi (Continuous Integration): Update accordingly.

Pushed!

[bug#32547] [PATCH] hydra: Add support for manifests.

[Ludovic Courtès]

Hi Clément,

Clément Lassieur <clement@lassieur.org> skribis:

> * build-aux/hydra/gnu-system.scm (arguments->manifests, manifests->packages):
> New procedures.
> (hydra-jobs): Add a "manifests" subset.
> * doc/guix.texi (Continuous Integration): Update accordingly.

Thanks for pushing the patch, and sorry for not commenting earlier!  I
agree this can be very useful.

I’m slightly concerned about the complexity of this file.  Maybe
sometime down the road we could split it in several proper modules, like
(guix ci jobs) that would provide procedures to define “package jobs”,
“cross-compilation jobs”, and “VM jobs”.

Thoughts?

Ludo’.

[bug#32547] [PATCH] hydra: Add support for manifests.

[Clément Lassieur]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Clément,
>
> Clément Lassieur <clement@lassieur.org> skribis:
>
>> * build-aux/hydra/gnu-system.scm (arguments->manifests, manifests->packages):
>> New procedures.
>> (hydra-jobs): Add a "manifests" subset.
>> * doc/guix.texi (Continuous Integration): Update accordingly.
>
> Thanks for pushing the patch, and sorry for not commenting earlier!  I
> agree this can be very useful.
>
> I’m slightly concerned about the complexity of this file.  Maybe
> sometime down the road we could split it in several proper modules, like
> (guix ci jobs) that would provide procedures to define “package jobs”,
> “cross-compilation jobs”, and “VM jobs”.

Yes, I agree!  And "test jobs" maybe?

Clément

[bug#32547] [PATCH] hydra: Add support for manifests.

[Ludovic Courtès]

Clément Lassieur <clement@lassieur.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi Clément,
>>
>> Clément Lassieur <clement@lassieur.org> skribis:
>>
>>> * build-aux/hydra/gnu-system.scm (arguments->manifests, manifests->packages):
>>> New procedures.
>>> (hydra-jobs): Add a "manifests" subset.
>>> * doc/guix.texi (Continuous Integration): Update accordingly.
>>
>> Thanks for pushing the patch, and sorry for not commenting earlier!  I
>> agree this can be very useful.
>>
>> I’m slightly concerned about the complexity of this file.  Maybe
>> sometime down the road we could split it in several proper modules, like
>> (guix ci jobs) that would provide procedures to define “package jobs”,
>> “cross-compilation jobs”, and “VM jobs”.
>
> Yes, I agree!  And "test jobs" maybe?

Indeed.  Well, food for thought!

Ludo’.