[bug#28484] Some steps and questions for Cuirass

[bug#28484] Some steps and questions for Cuirass

[Jan Nieuwenhuizen]

When Cuirass creates a new working directory from a guix.git clone, it tries
to build guix but fails like this

    /gnu/store/jqs80jq4j9z06p2vb9f7djv9k7jmc3d4-profile/bin/msgmerge: cannot create output file "de.po": Permission denied
    make[3]: *** [Makefile:255: de.po] Error 1
    make[3]: *** Waiting for unfinished jobs....

...although that may be a feature?

Patch 1/3 fixes this build problem...however now Cuirass rebuilds *all* of
guix (.go files) for each update of guix.git that it tracks.  When playing
with Cuirass to set it up, it's handy to only build `hello'.  Rebuilding all
of guix for each commit even if it fast-forwards, do we need/want that?

Patch 2/3 allows to test with other packages than "hello" or everything by
using a spec like

     ((#:name . "guix")
      (#:url . "git://git.savannah.gnu.org/guix.git")
      (#:load-path . ".")
      (#:file . "examples/gnu-system.scm")
      (#:proc . hydra-jobs)
      (#:arguments (subset . ("hello" "grep")))
      (#:branch . "master"))

Because this build recipe and thus this feature belongs in the repository
we're building, I also added a patch (the last one) for Guix itself that
implements this.

Patch 3/3 is the one I'm least happy with.  When building many more packages
than only `hello', Cuirass will not give any feedback until after everything
is built.  The Emacs interface under M-x guix-hydra-latests-builds is very
nice, but polling for that less so.

As the comment in the file already says

    ;; Register the results in the database.
    ;; XXX: The 'build-derivations' call is blocking so we end updating the
    ;; database potentially long after things have been built.

we probably want feedback (or guix-hydra-latests-builds availability) per
package...but any ideas how to do that?  We'd have to wrap/closure that into
building the derivations?

Greetings,
janneke

[bug#28485] [PATCH 1/3] base: Make working copy writable.

[Jan Nieuwenhuizen]

* src/cuirass/base.scm (copy-repository-cache): Make working copy writable.
Fixes guix .po build errors.
---
 src/cuirass/base.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm
index 02e587a..1711c30 100644
--- a/src/cuirass/base.scm
+++ b/src/cuirass/base.scm
@@ -143,7 +143,8 @@ directory and the sha1 of the top level commit in this directory."
       (let ((name (assq-ref spec #:name)))
         ;; Flush any directory with the same name.
         (false-if-exception (delete-file-recursively name))
-        (copy-recursively repo name)))))
+        (copy-recursively repo name)
+        (system* "chmod" "-R" "+w" name)))))
 
 (define (compile dir)
   ;; Required for fetching Guix bootstrap tarballs.
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#28485: Acknowledgement ([PATCH 1/3] base: Make working copy writable.)

[Jan Nieuwenhuizen]

GNU bug Tracking System writes:

as discussed in other thread, pushed to guix-cuirass master as

88363faa83631feb7af2395b9a33beb267ec050c

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#28486] [PATCH 2/3] gnu-system: Accept subset with list of package names.

[Jan Nieuwenhuizen]

* examples/gnu-system.scm (hydra-jobs): Accept subset with list of package
names.
---
 examples/gnu-system.scm | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/examples/gnu-system.scm b/examples/gnu-system.scm
index 4076786..ee2b571 100644
--- a/examples/gnu-system.scm
+++ b/examples/gnu-system.scm
@@ -2,6 +2,7 @@
 ;;;
 ;;; Copyright © 2012, 2013, 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
+;;; Copyright © 2017 Jan Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of Cuirass.
 ;;;
@@ -163,9 +164,10 @@ valid."
   "Return Hydra jobs."
   (define subset
     (match (assoc-ref arguments 'subset)
-      ("core" 'core)                    ; only build core packages
-      ("hello" 'hello)                  ; only build hello
-      (_ 'all)))                        ; build everything
+      ("core" 'core)                        ; only build core packages
+      ("hello" 'hello)                      ; only build hello
+      (((? string?) (? string?) ...) 'list) ; only build selected list of packages
+      (_ 'all)))                            ; build everything
 
   (define (cross-jobs system)
     (define (from-32-to-64? target)
@@ -226,6 +228,16 @@ valid."
                          (let ((hello (specification->package "hello")))
                            (list (package-job store (%job-name hello) hello system)))
                          '()))
+                    ((list)
+                     ;; Build selected list of packages only.
+                     (if (string=? system (%current-system))
+                         (let* ((names (assoc-ref arguments 'subset))
+                                (packages (map specification->package names)))
+                           (map (lambda (package)
+                                    (package-job store (%job-name package)
+                                                 package system))
+                                  packages))
+                         '()))
                     (else
                      (error "unknown subset" subset))))
                 %hydra-supported-systems)))
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#28486: Acknowledgement ([PATCH 2/3] gnu-system: Accept subset with list of package names.)

[Jan Nieuwenhuizen]

GNU bug Tracking System writes:

As discussed in other thread... pushed to master as bbab646926714ad6cdaac3e4578a14b408b65a84
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#28488] [PATCH 3/3] base: Show feedback when build has finished.

[Jan Nieuwenhuizen]

* src/cuirass/base.scm (build-packages): Print some feedback when derivations
have been built.
---
 src/cuirass/base.scm | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm
index 1711c30..d1c8282 100644
--- a/src/cuirass/base.scm
+++ b/src/cuirass/base.scm
@@ -35,6 +35,7 @@
   #:use-module (ice-9 threads)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-19)
+  #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-35)
   #:export (;; Procedures.
@@ -218,13 +219,15 @@ directory and the sha1 of the top level commit in this directory."
                      (#:timestamp . ,cur-time)
                      (#:starttime . ,cur-time)
                      (#:stoptime . ,cur-time))))
-        (db-add-build db build))
-      build))
+        (db-add-build db build)
+        build)))
 
   ;; Pass all the jobs at once so we benefit from as much parallelism as
   ;; possible (we must be using #:keep-going? #t).  Swallow build logs (the
   ;; daemon keeps them anyway), and swallow build errors.
   (guard (c ((nix-protocol-error? c) #t))
+    (format #t "load-path=~s\n" %load-path)
+    (format #t "load-compiled-path=~s\n" %load-compiled-path)
     (format #t "building ~a derivations...~%" (length jobs))
     (parameterize ((current-build-output-port (%make-void-port "w")))
       (build-derivations store
@@ -235,7 +238,15 @@ directory and the sha1 of the top level commit in this directory."
   ;; Register the results in the database.
   ;; XXX: The 'build-derivations' call is blocking so we end updating the
   ;; database potentially long after things have been built.
-  (map register jobs))
+  (let* ((results (map register jobs))
+         (status (map (cut assq-ref <> #:status) results))
+         (success (length (filter zero? status)))
+         (outputs (map (cut assq-ref <> #:outputs) results))
+         (outs (filter-map (cut assoc-ref <> "out") outputs))
+         (fail (- (length jobs) success)))
+    (format #t "outputs:\n~a\n" (string-join outs "\n"))
+    (format #t "success: ~a, fail: ~a\n" success fail)
+    results))
 
 (define (process-specs db jobspecs)
   "Evaluate and build JOBSPECS and store results in DB."
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

bug#28488: Acknowledgement ([PATCH 3/3] base: Show feedback when build has finished.)

[Jan Nieuwenhuizen]

GNU bug Tracking System writes:

as discussed in other thread, pushed to guix-cuirass master as
87ad259dba7de38b6e3ab954cd7b2f655358d877

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#28487] [PATCH] cuirass: Add gnu-system build spec.

[Jan Nieuwenhuizen]

* build-aux/cuirass/gnu-system.scm: New file.  Combines
build-aux/hydra/gnu-system.scm, guix-cuirass/examples/gnu-system.scm and supports
building a named subset.

     (let ((spec #~((#:name . "guix")
                    (#:url . "git://git.savannah.gnu.org/guix.git")
                    (#:load-path . ".")
                    (#:file . "build-aux/cuirass/gnu-system.scm")
                    (#:proc . hydra-jobs)
                    (#:arguments (subset . ("hello" "grep")))
                    (#:branch . "master"))))
       (service cuirass-service-type
                (cuirass-configuration
                 (specifications #~(list '#$spec)))))

---
 build-aux/cuirass/gnu-system.scm | 376 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 376 insertions(+)
 create mode 100644 build-aux/cuirass/gnu-system.scm

diff --git a/build-aux/cuirass/gnu-system.scm b/build-aux/cuirass/gnu-system.scm
new file mode 100644
index 000000000..15fdd3e31
--- /dev/null
+++ b/build-aux/cuirass/gnu-system.scm
@@ -0,0 +1,376 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
+;;; Copyright © 2017 Jan Nieuwenhuizen <janneke@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+;;;
+;;; This file defines build jobs for the Hydra continuation integration
+;;; tool.
+;;;
+
+;; Attempt to use our very own Guix modules.
+(eval-when (compile load eval)
+
+  ;; Ignore any available .go, and force recompilation.  This is because our
+  ;; checkout in the store has mtime set to the epoch, and thus .go files look
+  ;; newer, even though they may not correspond.
+  (set! %fresh-auto-compile #t)
+
+  (and=> (assoc-ref (current-source-location) 'filename)
+         (lambda (file)
+           (let ((dir (string-append (dirname file) "/../..")))
+             (format (current-error-port) "prepending ~s to the load path~%"
+                     dir)
+             (set! %load-path (cons dir %load-path))))))
+
+(use-modules (guix config)
+             (guix store)
+             (guix grafts)
+             (guix profiles)
+             (guix packages)
+             (guix derivations)
+             (guix monads)
+             ((guix licenses)
+              #:select (gpl3+ license-name license-uri license-comment))
+             ((guix utils) #:select (%current-system))
+             ((guix scripts system) #:select (read-operating-system))
+             ((guix scripts pack)
+              #:select (lookup-compressor self-contained-tarball))
+             (gnu packages)
+             (gnu packages gcc)
+             (gnu packages base)
+             (gnu packages gawk)
+             (gnu packages guile)
+             (gnu packages gettext)
+             (gnu packages compression)
+             (gnu packages multiprecision)
+             (gnu packages make-bootstrap)
+             (gnu packages package-management)
+             (gnu system)
+             (gnu system vm)
+             (gnu system install)
+             (gnu tests)
+             (srfi srfi-1)
+             (srfi srfi-26)
+             (ice-9 match))
+
+;; XXX: Debugging hack: since `hydra-eval-guile-jobs' redirects the output
+;; port to the bit bucket, let us write to the error port instead.
+(setvbuf (current-error-port) _IOLBF)
+(set-current-output-port (current-error-port))
+
+(define (license->alist lcs)
+  "Return LCS <license> object as an alist."
+  ;; Sometimes 'license' field is a list of licenses.
+  (if (list? lcs)
+      (map license->alist lcs)
+      `((name . ,(license-name lcs))
+        (uri . ,(license-uri lcs))
+        (comment . ,(license-comment lcs)))))
+
+(define (package-metadata package)
+  "Convert PACKAGE to an alist suitable for Hydra."
+  `((#:description . ,(package-synopsis package))
+    (#:long-description . ,(package-description package))
+    (#:license . ,(license->alist (package-license package)))
+    (#:home-page . ,(package-home-page package))
+    (#:maintainers . ("bug-guix@gnu.org"))
+    (#:max-silent-time . ,(or (assoc-ref (package-properties package)
+                                         'max-silent-time)
+                              3600))      ;1 hour by default
+    (#:timeout . ,(or (assoc-ref (package-properties package) 'timeout)
+                      72000))))           ;20 hours by default
+
+(define (package-job store job-name package system)
+  "Return a job called JOB-NAME that builds PACKAGE on SYSTEM."
+  (lambda ()
+    `((#:job-name . ,(string-append (symbol->string job-name) "." system))
+      (#:derivation . ,(derivation-file-name
+                        (parameterize ((%graft? #f))
+                          (package-derivation store package system
+                                              #:graft? #f))))
+      ,@(package-metadata package))))
+
+(define (package-cross-job store job-name package target system)
+  "Return a job called TARGET.JOB-NAME that cross-builds PACKAGE
+for TARGET on SYSTEM."
+  (lambda ()
+    `((#:job-name . ,(string-join (list target (symbol->string job-name) system)
+                                  "."))
+      (#:derivation . ,(derivation-file-name
+                        (parameterize ((%graft? #f))
+                          (package-cross-derivation store package target system
+                                                    #:graft? #f))))
+      ,@(package-metadata package))))
+
+(define %core-packages
+  ;; Note: Don't put the '-final' package variants because (1) that's
+  ;; implicit, and (2) they cannot be cross-built (due to the explicit input
+  ;; chain.)
+  (list gcc-4.8 gcc-4.9 gcc-5 glibc binutils
+        gmp mpfr mpc coreutils findutils diffutils patch sed grep
+        gawk gnu-gettext hello guile-2.0 guile-2.2 zlib gzip xz
+        %bootstrap-binaries-tarball
+        %binutils-bootstrap-tarball
+        (%glibc-bootstrap-tarball)
+        %gcc-bootstrap-tarball
+        %guile-bootstrap-tarball
+        %bootstrap-tarballs))
+
+(define %packages-to-cross-build
+  %core-packages)
+
+(define %cross-targets
+  '("mips64el-linux-gnu"
+    "mips64el-linux-gnuabi64"
+    "arm-linux-gnueabihf"
+    "aarch64-linux-gnu"
+    "powerpc-linux-gnu"
+    "i586-pc-gnu"                                 ;aka. GNU/Hurd
+    "i686-w64-mingw32"))
+
+(define %guixsd-supported-systems
+  '("x86_64-linux" "i686-linux"))
+
+(define (qemu-jobs store system)
+  "Return a list of jobs that build QEMU images for SYSTEM."
+  (define (->alist drv)
+    `((derivation . ,(derivation-file-name drv))
+      (description . "Stand-alone QEMU image of the GNU system")
+      (long-description . "This is a demo stand-alone QEMU image of the GNU
+system.")
+      (license . ,gpl3+)
+      (home-page . ,%guix-home-page-url)
+      (maintainers . ("bug-guix@gnu.org"))))
+
+  (define (->job name drv)
+    (let ((name (symbol-append name (string->symbol ".")
+                               (string->symbol system))))
+      `(,name . ,(lambda ()
+                   (parameterize ((%graft? #f))
+                     (->alist drv))))))
+
+  (define MiB
+    (expt 2 20))
+
+  (if (member system %guixsd-supported-systems)
+      (list (->job 'usb-image
+                   (run-with-store store
+                     (mbegin %store-monad
+                       (set-guile-for-build (default-guile))
+                       (system-disk-image installation-os
+                                          #:disk-image-size
+                                          (* 1024 MiB)))))
+            (->job 'iso9660-image
+                   (run-with-store store
+                     (mbegin %store-monad
+                       (set-guile-for-build (default-guile))
+                       (system-disk-image installation-os
+                                          #:file-system-type
+                                          "iso9660")))))
+      '()))
+
+(define (system-test-jobs store system)
+  "Return a list of jobs for the system tests."
+  (define (test->thunk test)
+    (lambda ()
+      (define drv
+        (run-with-store store
+          (mbegin %store-monad
+            (set-current-system system)
+            (set-grafting #f)
+            (set-guile-for-build (default-guile))
+            (system-test-value test))))
+
+      `((derivation . ,(derivation-file-name drv))
+        (description . ,(format #f "GuixSD '~a' system test"
+                                (system-test-name test)))
+        (long-description . ,(system-test-description test))
+        (license . ,gpl3+)
+        (home-page . ,%guix-home-page-url)
+        (maintainers . ("bug-guix@gnu.org")))))
+
+  (define (->job test)
+    (let ((name (string->symbol
+                 (string-append "test." (system-test-name test)
+                                "." system))))
+      (cons name (test->thunk test))))
+
+  (if (member system %guixsd-supported-systems)
+      (map ->job (all-system-tests))
+      '()))
+
+(define (tarball-jobs store system)
+  "Return Hydra jobs to build the self-contained Guix binary tarball."
+  (define (->alist drv)
+    `((derivation . ,(derivation-file-name drv))
+      (description . "Stand-alone binary Guix tarball")
+      (long-description . "This is a tarball containing binaries of Guix and
+all its dependencies, and ready to be installed on non-GuixSD distributions.")
+      (license . ,gpl3+)
+      (home-page . ,%guix-home-page-url)
+      (maintainers . ("bug-guix@gnu.org"))))
+
+  (define (->job name drv)
+    (let ((name (symbol-append name (string->symbol ".")
+                               (string->symbol system))))
+      `(,name . ,(lambda ()
+                   (parameterize ((%graft? #f))
+                     (->alist drv))))))
+
+  ;; XXX: Add a job for the stable Guix?
+  (list (->job 'binary-tarball
+               (run-with-store store
+                 (mbegin %store-monad
+                   (set-guile-for-build (default-guile))
+                   (>>= (profile-derivation (packages->manifest (list guix)))
+                        (lambda (profile)
+                          (self-contained-tarball "guix-binary" profile
+                                                  #:localstatedir? #t
+                                                  #:compressor
+                                                  (lookup-compressor "xz")))))
+                 #:system system))))
+
+(define job-name
+  ;; Return the name of a package's job.
+  (compose string->symbol package-full-name))
+
+(define package->job
+  (let ((base-packages
+         (delete-duplicates
+          (append-map (match-lambda
+                       ((_ package _ ...)
+                        (match (package-transitive-inputs package)
+                          (((_ inputs _ ...) ...)
+                           inputs))))
+                      (%final-inputs)))))
+    (lambda (store package system)
+      "Return a job for PACKAGE on SYSTEM, or #f if this combination is not
+valid."
+      (cond ((member package base-packages)
+             #f)
+            ((supported-package? package system)
+             (let ((drv (package-derivation store package system
+                                            #:graft? #f)))
+               (and (substitutable-derivation? drv)
+                    (package-job store (job-name package)
+                                 package system))))
+            (else
+             #f)))))
+
+\f
+;;;
+;;; Hydra entry point.
+;;;
+
+(define (hydra-jobs store arguments)
+  "Return Hydra jobs."
+  (define subset
+    (match (assoc-ref arguments 'subset)
+      ("core" 'core)                              ; only build core packages
+      ("hello" 'hello)                            ; only build hello
+      (((? string?) (? string?) ...) 'list)       ; only build selected list of packages
+      (_ 'all)))                                  ; build everything
+
+  (define (cross-jobs system)
+    (define (from-32-to-64? target)
+      ;; Return true if SYSTEM is 32-bit and TARGET is 64-bit.  This hack
+      ;; prevents known-to-fail cross-builds from i686-linux or armhf-linux to
+      ;; mips64el-linux-gnuabi64.
+      (and (or (string-prefix? "i686-" system)
+               (string-prefix? "i586-" system)
+               (string-prefix? "armhf-" system))
+           (string-contains target "64")))    ;x86_64, mips64el, aarch64, etc.
+
+    (define (same? target)
+      ;; Return true if SYSTEM and TARGET are the same thing.  This is so we
+      ;; don't try to cross-compile to 'mips64el-linux-gnu' from
+      ;; 'mips64el-linux'.
+      (or (string-contains target system)
+          (and (string-prefix? "armhf" system)    ;armhf-linux
+               (string-prefix? "arm" target))))   ;arm-linux-gnueabihf
+
+    (define (pointless? target)
+      ;; Return #t if it makes no sense to cross-build to TARGET from SYSTEM.
+      (and (string-contains target "mingw")
+           (not (string=? "x86_64-linux" system))))
+
+    (define (either proc1 proc2 proc3)
+      (lambda (x)
+        (or (proc1 x) (proc2 x) (proc3 x))))
+
+    (append-map (lambda (target)
+                  (map (lambda (package)
+                         (package-cross-job store (job-name package)
+                                            package target system))
+                       %packages-to-cross-build))
+                (remove (either from-32-to-64? same? pointless?)
+                        %cross-targets)))
+
+  ;; Turn off grafts.  Grafting is meant to happen on the user's machines.
+  (parameterize ((%graft? #f))
+    ;; Return one job for each package, except bootstrap packages.
+    (append-map (lambda (system)
+                  (case subset
+                    ((all)
+                     ;; Build everything, including replacements.
+                     (let ((all (fold-packages
+                                 (lambda (package result)
+                                   (cond ((package-replacement package)
+                                          (cons* package
+                                                 (package-replacement package)
+                                                 result))
+                                         ((package-superseded package)
+                                          result) ;don't build it
+                                         (else
+                                          (cons package result))))
+                                 '()))
+                           (job (lambda (package)
+                                  (package->job store package
+                                                system))))
+                       (append (filter-map job all)
+                               (qemu-jobs store system)
+                               (system-test-jobs store system)
+                               (tarball-jobs store system)
+                               (cross-jobs system))))
+                    ((core)
+                     ;; Build core packages only.
+                     (append (map (lambda (package)
+                                    (package-job store (job-name package)
+                                                 package system))
+                                  %core-packages)
+                             (cross-jobs system)))
+                    ((hello)
+                     ;; Build hello package only.
+                     (if (string=? system (%current-system))
+                         (let ((hello (specification->package "hello")))
+                           (list (package-job store (job-name hello) hello system)))
+                         '()))
+                    ((list)
+                     ;; Build selected list of packages only.
+                     (if (string=? system (%current-system))
+                         (let* ((names (assoc-ref arguments 'subset))
+                                (packages (map specification->package names)))
+                           (map (lambda (package)
+                                    (package-job store (job-name package)
+                                                 package system))
+                                  packages))
+                         '()))
+                    (else
+                     (error "unknown subset" subset))))
+                %hydra-supported-systems)))
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#28487] [PATCH] cuirass: Add gnu-system build spec.

[Ludovic Courtès]

Hello!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> * build-aux/cuirass/gnu-system.scm: New file.  Combines
> build-aux/hydra/gnu-system.scm, guix-cuirass/examples/gnu-system.scm and supports
> building a named subset.
>
>      (let ((spec #~((#:name . "guix")
>                     (#:url . "git://git.savannah.gnu.org/guix.git")
>                     (#:load-path . ".")
>                     (#:file . "build-aux/cuirass/gnu-system.scm")
>                     (#:proc . hydra-jobs)
>                     (#:arguments (subset . ("hello" "grep")))
>                     (#:branch . "master"))))
>        (service cuirass-service-type
>                 (cuirass-configuration
>                  (specifications #~(list '#$spec)))))

I like this but… while we’re at it, could we arrange to have a single
file that works for both Hydra and Cuirass?

Concretely, we currently have these two files:

  https://git.savannah.gnu.org/cgit/guix.git/tree/build-aux/hydra/gnu-system.scm
  https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/cuirass-jobs.scm

There are very close to one another.  The main difference is that one
returns an alist with symbols, whereas the other returns an alist with
keywords.

If we could instead add, say, build-aux/hydra/gnu-system-cuirass.scm,
which simply loads gnu-system.scm and “converts” the resulting alist
appropriately, that would be awesome.

I understand this is not quite what you were asking for ;-), but it
would achieve the same result.

How does that sound?

Thanks,
Ludo’.

[bug#28487] [PATCH] cuirass: Add gnu-system build spec.

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 1208 bytes --]

Ludovic Courtès writes:

> I like this but… while we’re at it, could we arrange to have a single
> file that works for both Hydra and Cuirass?

I like that.  It bothered me a bit to have similar files (not DRY) but
didn't see how that could be easily avoided.

> The main difference is that one returns an alist with symbols, whereas
> the other returns an alist with keywords.

That helps a lot...I kind of vaguely knew this but not well enough to see it!

> If we could instead add, say, build-aux/hydra/gnu-system-cuirass.scm,
> which simply loads gnu-system.scm and “converts” the resulting alist
> appropriately, that would be awesome.

I opted for build-aux/cuiras/gnu-cuirass.scm.

> I understand this is not quite what you were asking for ;-), but it
> would achieve the same result.

Attached is a new proposal, untested.  I will test it tomorrow, please
shoot on things that are already visibly foo.

One question: Assuming we intend to move to Cuirass and away from Hydra,
do we want to reverse the translation: have Cuirass be the untranslated
version and keep the translation to Hydra alive for as long as we need
to support Hydra?

Greetings,
janneke


[-- Attachment #2: 0001-cuirass-Add-gnu-system-build-spec.patch --]
[-- Type: text/x-patch, Size: 5034 bytes --]

From 8f10a707b0bbf6949388d85de175ed98974a8df4 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Sat, 16 Sep 2017 12:57:37 +0200
Subject: [PATCH] cuirass: Add gnu-system build spec.

* build-aux/hydra/gnu-system.scm (hydra-jobs): Add subsets "hello"
and list of packages: ("name[@version" ...).
* build-aux/cuirass/gnu-system.scm: New file.
* doc/guix.texi (Continuous Integration): Update example spec.
---
 build-aux/cuirass/gnu-system.scm | 36 ++++++++++++++++++++++++++++++++++++
 build-aux/hydra/gnu-system.scm   | 18 ++++++++++++++++++
 doc/guix.texi                    |  4 ++--
 3 files changed, 56 insertions(+), 2 deletions(-)
 create mode 100644 build-aux/cuirass/gnu-system.scm

diff --git a/build-aux/cuirass/gnu-system.scm b/build-aux/cuirass/gnu-system.scm
new file mode 100644
index 000000000..b545323f6
--- /dev/null
+++ b/build-aux/cuirass/gnu-system.scm
@@ -0,0 +1,36 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2017 Jan Nieuwenhuizen <janneke@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+;;;
+;;; This file defines build jobs for the Cuirass continuation integration
+;;; tool.
+;;;
+
+(include-from-path "build-aux/hydra/gnu-system.scm")
+
+(define (cuirass-jobs store arguments)
+  "Return Cuirass jobs."
+  (pk (map hydra-job->cuirass-job (hydra-jobs store arguments))))
+
+(define (hydra-job->cuirass-job hydra-job)
+  (let ((name (car hydra-job))
+        (job ((cdr hydra-job))))
+    (cons name (lambda _ (map symbol-alist-entry->keyword-alist-entry job)))))
+
+(define (symbol-alist-entry->keyword-alist-entry entry)
+  (cons (symbol->keyword (car entry)) (cdr entry)))
diff --git a/build-aux/hydra/gnu-system.scm b/build-aux/hydra/gnu-system.scm
index 73bd566f7..9968fc3da 100644
--- a/build-aux/hydra/gnu-system.scm
+++ b/build-aux/hydra/gnu-system.scm
@@ -270,6 +270,8 @@ valid."
   (define subset
     (match (assoc-ref arguments 'subset)
       ("core" 'core)                              ; only build core packages
+      ("hello" 'hello)                            ; only build hello
+      (((? string?) (? string?) ...) 'list)       ; only build selected list of packages
       (_ 'all)))                                  ; build everything
 
   (define (cross-jobs system)
@@ -340,6 +342,22 @@ valid."
                                                  package system))
                                   %core-packages)
                              (cross-jobs system)))
+                    ((hello)
+                     ;; Build hello package only.
+                     (if (string=? system (%current-system))
+                         (let ((hello (specification->package "hello")))
+                           (list (package-job store (job-name hello) hello system)))
+                         '()))
+                    ((list)
+                     ;; Build selected list of packages only.
+                     (if (string=? system (%current-system))
+                         (let* ((names (assoc-ref arguments 'subset))
+                                (packages (map specification->package names)))
+                           (map (lambda (package)
+                                    (package-job store (job-name package)
+                                                 package system))
+                                  packages))
+                         '()))
                     (else
                      (error "unknown subset" subset))))
                 %hydra-supported-systems)))
diff --git a/doc/guix.texi b/doc/guix.texi
index fff3fbd5f..c1391ee4b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -15375,8 +15375,8 @@ packages, as prescribed in the @file{gnu-system.scm} example spec:
                             cuirass
                             "/tests/gnu-system.scm"))
 
-               (#:proc . hydra-jobs)
-               (#:arguments (subset . "hello"))
+               (#:proc . cuirass-jobs)
+               (#:arguments (subset . ("hello" "grep")))
                (#:branch . "master"))))
   (service cuirass-service-type
            (cuirass-configuration
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com


[-- Attachment #3: Type: text/plain, Size: 154 bytes --]



-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#28487] [PATCH] cuirass: Add gnu-system build spec.

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

>> If we could instead add, say, build-aux/hydra/gnu-system-cuirass.scm,
>> which simply loads gnu-system.scm and “converts” the resulting alist
>> appropriately, that would be awesome.
>
> I opted for build-aux/cuiras/gnu-cuirass.scm.

ehh.. as the patch shows: build-aux/cuirass/gnu-system.scm.

and I'll add add (C) notice for myself to to hydra/gnu-system.scm

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#28487] [PATCH] cuirass: Add gnu-system build spec.

[Ludovic Courtès]

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Ludovic Courtès writes:
>
>> I like this but… while we’re at it, could we arrange to have a single
>> file that works for both Hydra and Cuirass?
>
> I like that.  It bothered me a bit to have similar files (not DRY) but
> didn't see how that could be easily avoided.
>
>> The main difference is that one returns an alist with symbols, whereas
>> the other returns an alist with keywords.
>
> That helps a lot...I kind of vaguely knew this but not well enough to see it!
>
>> If we could instead add, say, build-aux/hydra/gnu-system-cuirass.scm,
>> which simply loads gnu-system.scm and “converts” the resulting alist
>> appropriately, that would be awesome.
>
> I opted for build-aux/cuiras/gnu-cuirass.scm.

Sounds good!

> Attached is a new proposal, untested.  I will test it tomorrow, please
> shoot on things that are already visibly foo.
>
> One question: Assuming we intend to move to Cuirass and away from Hydra,
> do we want to reverse the translation: have Cuirass be the untranslated
> version and keep the translation to Hydra alive for as long as we need
> to support Hydra?
>
> Greetings,
> janneke
>
> From 8f10a707b0bbf6949388d85de175ed98974a8df4 Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen <janneke@gnu.org>
> Date: Sat, 16 Sep 2017 12:57:37 +0200
> Subject: [PATCH] cuirass: Add gnu-system build spec.
>
> * build-aux/hydra/gnu-system.scm (hydra-jobs): Add subsets "hello"
> and list of packages: ("name[@version" ...).
> * build-aux/cuirass/gnu-system.scm: New file.
> * doc/guix.texi (Continuous Integration): Update example spec.
> ---
>  build-aux/cuirass/gnu-system.scm | 36 ++++++++++++++++++++++++++++++++++++
>  build-aux/hydra/gnu-system.scm   | 18 ++++++++++++++++++
>  doc/guix.texi                    |  4 ++--
>  3 files changed, 56 insertions(+), 2 deletions(-)
>  create mode 100644 build-aux/cuirass/gnu-system.scm
>
> diff --git a/build-aux/cuirass/gnu-system.scm b/build-aux/cuirass/gnu-system.scm
> new file mode 100644
> index 000000000..b545323f6
> --- /dev/null
> +++ b/build-aux/cuirass/gnu-system.scm
> @@ -0,0 +1,36 @@
> +;;; GNU Guix --- Functional package management for GNU
> +;;; Copyright © 2017 Jan Nieuwenhuizen <janneke@gnu.org>
> +;;;
> +;;; This file is part of GNU Guix.
> +;;;
> +;;; GNU Guix is free software; you can redistribute it and/or modify it
> +;;; under the terms of the GNU General Public License as published by
> +;;; the Free Software Foundation; either version 3 of the License, or (at
> +;;; your option) any later version.
> +;;;
> +;;; GNU Guix is distributed in the hope that it will be useful, but
> +;;; WITHOUT ANY WARRANTY; without even the implied warranty of
> +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +;;; GNU General Public License for more details.
> +;;;
> +;;; You should have received a copy of the GNU General Public License
> +;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
> +
> +;;;
> +;;; This file defines build jobs for the Cuirass continuation integration
> +;;; tool.
> +;;;
> +
> +(include-from-path "build-aux/hydra/gnu-system.scm")
> +
> +(define (cuirass-jobs store arguments)
> +  "Return Cuirass jobs."
> +  (pk (map hydra-job->cuirass-job (hydra-jobs store arguments))))
> +
> +(define (hydra-job->cuirass-job hydra-job)
> +  (let ((name (car hydra-job))
> +        (job ((cdr hydra-job))))
> +    (cons name (lambda _ (map symbol-alist-entry->keyword-alist-entry job)))))
> +
> +(define (symbol-alist-entry->keyword-alist-entry entry)
> +  (cons (symbol->keyword (car entry)) (cdr entry)))

I *think* that’s correct, though we’ll need to double check.

> --- a/build-aux/hydra/gnu-system.scm
> +++ b/build-aux/hydra/gnu-system.scm
> @@ -270,6 +270,8 @@ valid."
>    (define subset
>      (match (assoc-ref arguments 'subset)
>        ("core" 'core)                              ; only build core packages
> +      ("hello" 'hello)                            ; only build hello
> +      (((? string?) (? string?) ...) 'list)       ; only build selected list of packages
>        (_ 'all)))                                  ; build everything

This part could be added separately.

(It’s not usuable via Hydra since its UIs does not support passing
list-of-strings arguments.)

Thanks a lot!

Ludo’.

[bug#28487] [PATCH] cuirass: Add gnu-system build spec.

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 1922 bytes --]

Ludovic Courtès writes:

Hey Ludo',

>> I opted for build-aux/cuiras/gnu-cuirass.scm.
>
> Sounds good!

Ok, grand.

>> +(define (symbol-alist-entry->keyword-alist-entry entry)
>> +  (cons (symbol->keyword (car entry)) (cdr entry)))
>
> I *think* that’s correct, though we’ll need to double check.

I tested today and there were minor problems.  Cuirass actually doesn't
take an alist; instead takes a list that includes (#:job-name . "name").

Also, Cuirass performs an sexp-read and thus #<<license> ...> needs to
get sexp'ified.

Anyway, what I proposed was close and attached is a tested, working
version (that may need some work, see below).

>> --- a/build-aux/hydra/gnu-system.scm
>> +++ b/build-aux/hydra/gnu-system.scm
>> @@ -270,6 +270,8 @@ valid."
>>    (define subset
>>      (match (assoc-ref arguments 'subset)
>>        ("core" 'core)                              ; only build core packages
>> +      ("hello" 'hello)                            ; only build hello
>> +      (((? string?) (? string?) ...) 'list)       ; only build selected list of packages
>>        (_ 'all)))                                  ; build everything
>
> This part could be added separately.

Yes...it could.  Do you mean a separate patch, or ...

> (It’s not usuable via Hydra since its UIs does not support passing
> list-of-strings arguments.)

...I don't quite understand what you propose here.  I appreciate that
I'm adding functionality for Cuirass to the hydra file where in itself
that does not make much sense...

Otoh, I don't see how to move this functionality to
cuirass/gnu-system.scm only without duplicating much of `hydra-jobs'; so
that's probably not what you mean... / somewhat confused here. ;-)

I'd be happy to update or split this patch if once I understand what you
want or feel free to do it for me if that's easier for you.

Thanks!
janneke


[-- Attachment #2: 0001-cuirass-Add-gnu-system-build-spec.patch --]
[-- Type: text/x-patch, Size: 6134 bytes --]

From bb82d85df2380cd0c25de0f659a7010f2d88ee4f Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Sat, 16 Sep 2017 12:57:37 +0200
Subject: [PATCH] cuirass: Add gnu-system build spec.

* build-aux/hydra/gnu-system.scm (hydra-jobs): Add subsets "hello"
and list of packages: ("name[@version" ...).
* guix/licenses.scm: Export <license>.
* build-aux/cuirass/gnu-system.scm: New file.
* doc/guix.texi (Continuous Integration): Update example spec.
---
 build-aux/cuirass/gnu-system.scm | 45 ++++++++++++++++++++++++++++++++++++++++
 build-aux/hydra/gnu-system.scm   | 19 +++++++++++++++++
 doc/guix.texi                    |  4 ++--
 guix/licenses.scm                |  3 ++-
 4 files changed, 68 insertions(+), 3 deletions(-)
 create mode 100644 build-aux/cuirass/gnu-system.scm

diff --git a/build-aux/cuirass/gnu-system.scm b/build-aux/cuirass/gnu-system.scm
new file mode 100644
index 000000000..0028d9d8e
--- /dev/null
+++ b/build-aux/cuirass/gnu-system.scm
@@ -0,0 +1,45 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2017 Jan Nieuwenhuizen <janneke@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+;;;
+;;; This file defines build jobs for the Cuirass continuation integration
+;;; tool.
+;;;
+
+(include-from-path "build-aux/hydra/gnu-system.scm")
+
+(use-modules ((guix licenses) #:select (<license>)))
+
+(define (cuirass-jobs store arguments)
+  "Return Cuirass jobs."
+  (map hydra-job->cuirass-job (hydra-jobs store arguments)))
+
+(define (hydra-job->cuirass-job hydra-job)
+  (let ((name (car hydra-job))
+        (job ((cdr hydra-job))))
+    (lambda _ (acons #:job-name (symbol->string name)
+                     (map symbol-alist-entry->keyword-alist-entry job)))))
+
+(define (symbol-alist-entry->keyword-alist-entry entry)
+  (cons (symbol->keyword (car entry)) (entry->sexp-entry (cdr entry))))
+
+(define (entry->sexp-entry o)
+  (match o
+    (($ <license> name uri comment)
+     `((name . ,name) (uri . ,uri) (comment . ,comment)))
+    (_ o)))
diff --git a/build-aux/hydra/gnu-system.scm b/build-aux/hydra/gnu-system.scm
index 73bd566f7..146d929f9 100644
--- a/build-aux/hydra/gnu-system.scm
+++ b/build-aux/hydra/gnu-system.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017 Jan Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -270,6 +271,8 @@ valid."
   (define subset
     (match (assoc-ref arguments 'subset)
       ("core" 'core)                              ; only build core packages
+      ("hello" 'hello)                            ; only build hello
+      (((? string?) (? string?) ...) 'list)       ; only build selected list of packages
       (_ 'all)))                                  ; build everything
 
   (define (cross-jobs system)
@@ -340,6 +343,22 @@ valid."
                                                  package system))
                                   %core-packages)
                              (cross-jobs system)))
+                    ((hello)
+                     ;; Build hello package only.
+                     (if (string=? system (%current-system))
+                         (let ((hello (specification->package "hello")))
+                           (list (package-job store (job-name hello) hello system)))
+                         '()))
+                    ((list)
+                     ;; Build selected list of packages only.
+                     (if (string=? system (%current-system))
+                         (let* ((names (assoc-ref arguments 'subset))
+                                (packages (map specification->package names)))
+                           (map (lambda (package)
+                                    (package-job store (job-name package)
+                                                 package system))
+                                  packages))
+                         '()))
                     (else
                      (error "unknown subset" subset))))
                 %hydra-supported-systems)))
diff --git a/doc/guix.texi b/doc/guix.texi
index fff3fbd5f..c1391ee4b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -15375,8 +15375,8 @@ packages, as prescribed in the @file{gnu-system.scm} example spec:
                             cuirass
                             "/tests/gnu-system.scm"))
 
-               (#:proc . hydra-jobs)
-               (#:arguments (subset . "hello"))
+               (#:proc . cuirass-jobs)
+               (#:arguments (subset . ("hello" "grep")))
                (#:branch . "master"))))
   (service cuirass-service-type
            (cuirass-configuration
diff --git a/guix/licenses.scm b/guix/licenses.scm
index 6de611da2..144dd06f0 100644
--- a/guix/licenses.scm
+++ b/guix/licenses.scm
@@ -31,7 +31,8 @@
 
 (define-module (guix licenses)
   #:use-module (srfi srfi-9)
-  #:export (license? license-name license-uri license-comment
+  #:export (<license>
+            license? license-name license-uri license-comment
             agpl1 agpl3 agpl3+
             asl1.1 asl2.0
             boost1.0
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com


[-- Attachment #3: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#28487] [PATCH] cuirass: Add gnu-system build spec.

[Ludovic Courtès]

Hi!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

>>> +(define (symbol-alist-entry->keyword-alist-entry entry)
>>> +  (cons (symbol->keyword (car entry)) (cdr entry)))
>>
>> I *think* that’s correct, though we’ll need to double check.
>
> I tested today and there were minor problems.  Cuirass actually doesn't
> take an alist; instead takes a list that includes (#:job-name . "name").
>
> Also, Cuirass performs an sexp-read and thus #<<license> ...> needs to
> get sexp'ified.
>
> Anyway, what I proposed was close and attached is a tested, working
> version (that may need some work, see below).

Oh good points, thanks for testing!

BTW, there’s “make hydra-jobs.scm”, which I occasionally use to test
gnu-system.scm (it spits a raw alist, which is enough to make sure that
it works.)  We could similarly add “make cuirass-jobs.scm” eventually.

>>> --- a/build-aux/hydra/gnu-system.scm
>>> +++ b/build-aux/hydra/gnu-system.scm
>>> @@ -270,6 +270,8 @@ valid."
>>>    (define subset
>>>      (match (assoc-ref arguments 'subset)
>>>        ("core" 'core)                              ; only build core packages
>>> +      ("hello" 'hello)                            ; only build hello
>>> +      (((? string?) (? string?) ...) 'list)       ; only build selected list of packages
>>>        (_ 'all)))                                  ; build everything
>>
>> This part could be added separately.
>
> Yes...it could.  Do you mean a separate patch, or ...

Yes, separate patch for clarity: first patch does the Hydra/Cuirass
split, second patch adds the ability to select a list of packages.

Would that be OK?

>> (It’s not usuable via Hydra since its UIs does not support passing
>> list-of-strings arguments.)
>
> ...I don't quite understand what you propose here.  I appreciate that
> I'm adding functionality for Cuirass to the hydra file where in itself
> that does not make much sense...
>
> Otoh, I don't see how to move this functionality to
> cuirass/gnu-system.scm only without duplicating much of `hydra-jobs'; so
> that's probably not what you mean... / somewhat confused here. ;-)

Never mind, it *was* confusing.  :-)

Last nitpicking:

> +(define (entry->sexp-entry o)
> +  (match o
> +    (($ <license> name uri comment)
> +     `((name . ,name) (uri . ,uri) (comment . ,comment)))
> +    (_ o)))

[...]

> --- a/guix/licenses.scm
> +++ b/guix/licenses.scm
> @@ -31,7 +31,8 @@
>  
>  (define-module (guix licenses)
>    #:use-module (srfi srfi-9)
> -  #:export (license? license-name license-uri license-comment
> +  #:export (<license>

I prefer not to export record type descriptors in general, because that
exposes too much of the internals and makes it harder to change the code
afterwards.  For instance, if we change the layout of <license>, and if
<license> is exported, we have to carefully check all users and it’s
easy to get it wrong.  (There’s also the problem that exposing the RTD
makes records forgeable: we no longer control how <license> objects are
created, so we cannot make sure that <license> objects we get are
“genuine.”)

Anyway, with this fixed, OK to push.

After that we should update the config on berlin.guixsd.org to use this
file directly.

Thanks a lot!

Ludo’.

[bug#28487] [PATCH] cuirass: Add gnu-system build spec.

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

>> Anyway, what I proposed was close and attached is a tested, working
>> version (that may need some work, see below).
>
> Oh good points, thanks for testing!

np...as I'm using this myself it's nice if it works ;-)

> BTW, there’s “make hydra-jobs.scm”, which I occasionally use to test
> gnu-system.scm (it spits a raw alist, which is enough to make sure that
> it works.)  We could similarly add “make cuirass-jobs.scm” eventually.

...looked into this briefly but this would mean adding some option to
evaluate.scm, something like

    ;; Without further ado...
    (match (command-line)
-     ((command file)
+     ((command file cuirass? ...))
...
-        (match ((module-ref %user-module 'hydra-jobs) store '())
+        (match ((module-ref %user-module
+           (if (equal? cuirass? "cuirass") cuirass-jobs 'hydra-jobs)
?           store '())

> Yes, separate patch for clarity: first patch does the Hydra/Cuirass
> split, second patch adds the ability to select a list of packages.
>
> Would that be OK?

Sure, done.

>> +  #:export (<license>
>
> I prefer not to export record type descriptors in general, because that
> exposes too much of the internals

Sure, I agree.  Using license? and accessors now.

As an aside: Interesting, I've been using GOOPS a lot with match and we have
standardised on only matching class/record type, like so

    (match o
      (($ <license>) ...)

so I was wondering why you didn't export <license>.  Now it makes sense:
you simply cannot use record destructors if you don't export <..> in the
first place.  For GOOPS that's a bit different, you don't have class?
for each <class>.

> Anyway, with this fixed, OK to push.

Thanks, pushed with all this fixed to master as 516b53828e90018126b79e7600cae9aa531e06d7

> After that we should update the config on berlin.guixsd.org to use this
> file directly.

Ricardo: ping?

> Thanks a lot!

Thank you!  Most welcome,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#28487] [PATCH] cuirass: Add gnu-system build spec.

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 541 bytes --]

Jan Nieuwenhuizen writes:

> ...looked into this briefly but this would mean adding some option to
> evaluate.scm, something like
>
>     ;; Without further ado...
>     (match (command-line)
> -     ((command file)
> +     ((command file cuirass? ...))
> ...
> -        (match ((module-ref %user-module 'hydra-jobs) store '())
> +        (match ((module-ref %user-module
> +           (if (equal? cuirass? "cuirass") cuirass-jobs 'hydra-jobs)
> ?           store '())

Hmm, that could work---actual working prototype attached :-)

janneke


[-- Attachment #2: 0001-cuirass-Add-cuirass-jobs.scm-target-to-compute-the-C.patch --]
[-- Type: text/x-patch, Size: 3425 bytes --]

From 8c547d1d6a76dfb5e3b59f7fce404d0b60c6d929 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Wed, 27 Sep 2017 22:44:56 +0200
Subject: [PATCH] cuirass: Add 'cuirass-jobs.scm' target to compute the Cuirass
 jobs.

* build-aux/hydra/evaluate.scm: Support "cuirass" command line option.
* Makefile.am (cuirass-jobs.scm): New target.
---
 Makefile.am                  | 10 ++++++++++
 build-aux/hydra/evaluate.scm | 13 +++++++++----
 2 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/Makefile.am b/Makefile.am
index a178d53cb..2c87e4fe3 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -7,6 +7,7 @@
 # Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
 # Copyright © 2017 Leo Famulari <leo@famulari.name>
 # Copyright © 2017 Ricardo Wurmus <rekado@elephly.net>
+# Copyright © 2017 Jan Nieuwenhuizen <janneke@gnu.org>
 #
 # This file is part of GNU Guix.
 #
@@ -708,6 +709,15 @@ hydra-jobs.scm: $(GOBJECTS)
 	  "$(top_srcdir)/build-aux/hydra/gnu-system.scm" > "$@.tmp"
 	$(AM_V_at)mv "$@.tmp" "$@"
 
+# Compute the Cuirass jobs and write them in the target file.
+cuirass-jobs.scm: $(GOBJECTS)
+	$(AM_V_at)$(MKDIR_P) "`dirname "$@"`"
+	$(AM_V_GEN)$(top_builddir)/pre-inst-env "$(GUILE)"		\
+	  "$(top_srcdir)/build-aux/hydra/evaluate.scm"			\
+	  "$(top_srcdir)/build-aux/cuirass/gnu-system.scm" 		\
+	  cuirass > "$@.tmp"
+	$(AM_V_at)mv "$@.tmp" "$@"
+
 .PHONY: gen-ChangeLog gen-AUTHORS gen-tarball-version
 .PHONY: assert-no-store-file-names assert-binaries-available
 .PHONY: assert-final-inputs-self-contained
diff --git a/build-aux/hydra/evaluate.scm b/build-aux/hydra/evaluate.scm
index cc6a4b949..604022abc 100644
--- a/build-aux/hydra/evaluate.scm
+++ b/build-aux/hydra/evaluate.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017 Jan Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -70,7 +71,7 @@ Otherwise return THING."
 \f
 ;; Without further ado...
 (match (command-line)
-  ((command file)
+  ((command file cuirass? ...)
    ;; Load FILE, a Scheme file that defines Hydra jobs.
    (let ((port (current-output-port)))
      (save-module-excursion
@@ -96,7 +97,11 @@ Otherwise return THING."
 
        ;; Call the entry point of FILE and print the resulting job sexp.
        (pretty-print
-        (match ((module-ref %user-module 'hydra-jobs) store '())
+        (match ((module-ref %user-module
+                            (if (equal? cuirass? "cuirass")
+                                'cuirass-jobs
+                                'hydra-jobs))
+                store '())
           (((names . thunks) ...)
            (map (lambda (job thunk)
                   (format (current-error-port) "evaluating '~a'... " job)
@@ -107,8 +112,8 @@ Otherwise return THING."
                 names thunks)))
         port))))
   ((command _ ...)
-   (format (current-error-port) "Usage: ~a FILE
-Evaluate the Hydra jobs defined in FILE.~%"
+   (format (current-error-port) "Usage: ~a FILE [cuirass]
+Evaluate the Hydra or Cuirass jobs defined in FILE.~%"
            command)
    (exit 1)))
 
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com


[-- Attachment #3: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#28487] [PATCH] cuirass: Add gnu-system build spec.

[Ludovic Courtès]

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> From 8c547d1d6a76dfb5e3b59f7fce404d0b60c6d929 Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen <janneke@gnu.org>
> Date: Wed, 27 Sep 2017 22:44:56 +0200
> Subject: [PATCH] cuirass: Add 'cuirass-jobs.scm' target to compute the Cuirass
>  jobs.
>
> * build-aux/hydra/evaluate.scm: Support "cuirass" command line option.
> * Makefile.am (cuirass-jobs.scm): New target.

Perfect, go for it.

Thank you!

Ludo’.

bug#28487: [PATCH] cuirass: Add gnu-system build spec.

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> From 8c547d1d6a76dfb5e3b59f7fce404d0b60c6d929 Mon Sep 17 00:00:00 2001
>> From: Jan Nieuwenhuizen <janneke@gnu.org>
>> Date: Wed, 27 Sep 2017 22:44:56 +0200
>> Subject: [PATCH] cuirass: Add 'cuirass-jobs.scm' target to compute the Cuirass
>>  jobs.
>>
>> * build-aux/hydra/evaluate.scm: Support "cuirass" command line option.
>> * Makefile.am (cuirass-jobs.scm): New target.
>
> Perfect, go for it.

Pushed to master as 454caca8e6db34d7777ddb4ba84680d41e381b32

Thanks!
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#28487] [PATCH] cuirass: Add gnu-system build spec.

[Ludovic Courtès]

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> As an aside: Interesting, I've been using GOOPS a lot with match and we have
> standardised on only matching class/record type, like so
>
>     (match o
>       (($ <license>) ...)

I would make it:

  (match o
    ((? license?)
     ...))

> so I was wondering why you didn't export <license>.  Now it makes sense:
> you simply cannot use record destructors if you don't export <..> in the
> first place.  For GOOPS that's a bit different, you don't have class?
> for each <class>.

Yes.  I think the “spirit of GOOPS” is that class objects are public.

Ludo’.

[bug#28487] [PATCH] cuirass: Add gnu-system build spec.

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

>>     (match o
>>       (($ <license>) ...)
>
> I would make it:
>
>   (match o
>     ((? license?)
>      ...))

Oops, something like this was already pushed here:

     > Anyway, with this fixed, OK to push.

     Thanks, pushed with all this fixed to master as 516b53828e90018126b79e7600cae9aa531e06d7

Sure hope I didn't interpret the OK to push too lightly.  Mentioned
commit above is the second patch, the first is here

    http://git.savannah.gnu.org/cgit/guix.git/commit/?id=66bc1d2aaf74fc7eb4ef9b3519c69bd37142ffb3

>> so I was wondering why you didn't export <license>.  Now it makes sense:
>> you simply cannot use record destructors if you don't export <..> in the
>> first place.  For GOOPS that's a bit different, you don't have class?
>> for each <class>.
>
> Yes.  I think the “spirit of GOOPS” is that class objects are public.

Yeah...and then you have to be `careful'; not so nice come to think of it.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#28484] Some steps and questions for Cuirass

[Mathieu Othacehe]

Hi Jan,

> ...although that may be a feature?

I don't think so, patch 1 & 2 LGTM.

> Patch 3/3 is the one I'm least happy with.  When building many more packages
> than only `hello', Cuirass will not give any feedback until after everything
> is built.  The Emacs interface under M-x guix-hydra-latests-builds is very
> nice, but polling for that less so.
>
> As the comment in the file already says
>
>     ;; Register the results in the database.
>     ;; XXX: The 'build-derivations' call is blocking so we end updating the
>     ;; database potentially long after things have been built.
>
> we probably want feedback (or guix-hydra-latests-builds availability) per
> package...but any ideas how to do that?  We'd have to wrap/closure that into
> building the derivations?

Now that Cuirass supports M-x guix-hydra-latests-builds, the next step
is to detect build start/stop (maybe via guix build log parsing see:
https://lists.gnu.org/archive/html/guix-devel/2017-07/msg00221.html).

This way, for each new commit in guix, cuirass will only insert in
database the derivation that were built. Plus, it will be possible to
insert the build in db when it starts and update its status when it
stops/fails.

Until this is developed, patch 3 LGTM too !

Mathieu

bug#28484: Some steps and questions for Cuirass

[Jan Nieuwenhuizen]

Mathieu Othacehe writes:

> I don't think so, patch 1 & 2 LGTM.

Thanks!

> Now that Cuirass supports M-x guix-hydra-latests-builds, the next step
> is to detect build start/stop (maybe via guix build log parsing see:
> https://lists.gnu.org/archive/html/guix-devel/2017-07/msg00221.html).
>
> This way, for each new commit in guix, cuirass will only insert in
> database the derivation that were built. Plus, it will be possible to
> insert the build in db when it starts and update its status when it
> stops/fails.

Ah that makes sense.  Thanks for sharing this thread.

> Until this is developed, patch 3 LGTM too !

Thanks, pushed to guix-cuirass master as 87ad259dba7de38b6e3ab954cd7b2f655358d877

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com