[bug#65732] [PATCH 0/2] Add the Build Farm Front-End

[bug#65732] [PATCH 0/2] Add the Build Farm Front-End

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 648 bytes --]

This is the package and service for the Build Farm Front-End, intended
to replace the Guix Build Coordinator queue builds script, and also
provide a web interface for build farms.


Christopher Baines (2):
  gnu: Add bffe.
  services: guix: Add bffe-service-type.

 doc/guix.texi                       |  59 +++++++++++++
 gnu/packages/package-management.scm |  51 +++++++++++
 gnu/services/guix.scm               | 131 +++++++++++++++++++++++++++-
 gnu/tests/guix.scm                  |  81 ++++++++++++++++-
 4 files changed, 320 insertions(+), 2 deletions(-)


base-commit: 8d70ff3a49f7cdf0eab93b3a1c54ec2f016afc4a
-- 
2.41.0

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

[bug#65732] [PATCH 1/2] gnu: Add bffe.

[Christopher Baines]

* gnu/packages/package-management.scm (bffe): New variable.
---
 gnu/packages/package-management.scm | 51 +++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index aa24798071..a49f302bb7 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -961,6 +961,57 @@ (define-public rpm
     ;; The whole is GPLv2+; librpm itself is dual-licensed LGPLv2+ | GPLv2+.
     (license license:gpl2+)))
 
+(define-public bffe
+  (let ((commit "3ce4613908bb4a42494323ef0597f6c3ae2dee24")
+        (revision "1"))
+    (package
+      (name "bffe")
+      (version (git-version "0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://git.cbaines.net/guix/bffe")
+                      (commit commit)))
+                (sha256
+                 (base32
+                  "0g7arfl8rwybqpmw8vc16ilvrva1bb435lqgbqypx0nphbcv2pxn"))
+                (file-name (string-append name "-" version "-checkout"))))
+      (build-system gnu-build-system)
+      (native-inputs
+       (list pkg-config
+             autoconf
+             automake
+
+             ;; Guile libraries are needed here for cross-compilation.
+             (car (assoc-ref (package-native-inputs guix) "guile"))
+             guile-gnutls
+             guile-json-4
+             guix
+             guix-data-service
+             guix-build-coordinator
+             guile-fibers-1.3
+             guile-prometheus
+             guile-lib))
+      (propagated-inputs
+       (list guile-gnutls
+             guile-json-4
+             guix
+             guix-data-service
+             guix-build-coordinator
+             guile-fibers-1.3
+             guile-prometheus
+             guile-lib))
+      (home-page "https://git.cbaines.net/guix/bffe")
+      (synopsis "Build Farm Front-end for Guix")
+      (description
+       "The BFFE of Build Farm Front-end is an experimental frontend for Guix
+build farms.  It works together with the Guix Data Service and Guix Build
+Coordinator to submit builds and monitor the activity.
+
+It functions as a Guile library, with the @code{run-bffe-service} procedure in
+the @code{(bffe)} module as the entry point.")
+      (license license:gpl3+))))
+
 (define-public python-anaconda-client
   (package
     (name "python-anaconda-client")

base-commit: 8d70ff3a49f7cdf0eab93b3a1c54ec2f016afc4a
-- 
2.41.0

[bug#65732] [PATCH 2/2] services: guix: Add bffe-service-type.

[Christopher Baines]

This is intended to replace the functionality of the Guix Build Coordinator
queue builds script, and also provide a web interface for build farms.

* gnu/services/guix.scm (<bffe-configuration>): New record type.
(bffe-configuration, bffe-configuration?,
bffe-configuration-package,
bffe-configuration-user,
bffe-configuration-group,
bffe-configuration-arguments
bffe-configuration-extra-environment-variables): New procedures.
(bffe-service-type): New variable.
* gnu/tests/guix.scm (%test-bffe): New variable.
* doc/guix.texi (Guix Services): Document the new service.
---
 doc/guix.texi         |  59 +++++++++++++++++++
 gnu/services/guix.scm | 131 +++++++++++++++++++++++++++++++++++++++++-
 gnu/tests/guix.scm    |  81 +++++++++++++++++++++++++-
 3 files changed, 269 insertions(+), 2 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index a6b74ce9c7..bf12f8945d 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -38083,6 +38083,65 @@ PAM Mount Service
 @node Guix Services
 @subsection Guix Services
 
+@subsubheading Build Farm Front-End (BFFE)
+The @uref{https://git.cbaines.net/guix/bffe/,Build Farm Front-End}
+assists with building Guix packages in bulk.  It's responsible for
+submitting builds and displaying the status of the build farm.
+
+@defvar bffe-service-type
+Service type for the Build Farm Front-End.  Its value must be a
+@code{bffe-configuration} object.
+@end defvar
+
+@deftp {Data Type} bffe-configuration
+Data type representing the configuration of the Build Farm Front-End.
+
+@table @asis
+@item @code{package} (default: @code{bffe})
+The Build Farm Front-End package to use.
+
+@item @code{user} (default: @code{"bffe"})
+The system user to run the service as.
+
+@item @code{group} (default: @code{"bffe"})
+The system group to run the service as.
+
+@item @code{arguments}
+A list of arguments to the Build Farm Front-End.  These are passed to
+the @code{run-bffe-service} procedure when starting the service.
+
+For example, the following value directs the Build Farm Front-End to
+submit builds for derivations available from @code{data.guix.gnu.org} to
+the Build Coordinator instance assumed to be running on the same
+machine.
+
+@example
+(list
+ #:build
+ (list
+  (build-from-guix-data-service
+   (data-service-url "https://data.guix.gnu.org")
+   (build-coordinator-url "http://127.0.0.1:8746")
+   (branches '("master"))
+   (systems '("x86_64-linux" "i686-linux"))
+   (systems-and-targets
+    (map (lambda (target)
+           (cons "x86_64-linux" target))
+         '("aarch64-linux-gnu"
+           "i586-pc-gnu")))
+   (build-priority (const 0))))
+ #:web-server-args
+ '(#:event-source "https://example.com"
+   #:controller-args
+   (#:title "example.com build farm")))
+@end example
+
+@item @code{extra-environment-variables} (default: @var{'()})
+Extra environment variables to set via the shepherd service.
+
+@end table
+@end deftp
+
 @subsubheading Guix Build Coordinator
 The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build
 Coordinator} aids in distributing derivation builds among machines
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index 99b21f52d8..e9db2a231d 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -140,7 +140,17 @@ (define-module (gnu services guix)
             nar-herder-cached-compression-configuration-type
             nar-herder-cached-compression-configuration-level
             nar-herder-cached-compression-configuration-directory
-            nar-herder-cached-compression-configuration-directory-max-size))
+            nar-herder-cached-compression-configuration-directory-max-size
+
+            bffe-configuration
+            bffe-configuration?
+            bffe-configuration-package
+            bffe-configuration-user
+            bffe-configuration-group
+            bffe-configuration-arguments
+            bffe-configuration-extra-environment-variables
+
+            bffe-service-type))
 
 ;;;; Commentary:
 ;;;
@@ -1030,3 +1040,122 @@ (define nar-herder-service-type
                         nar-herder-account)))
    (description
     "Run a Nar Herder server.")))
+
+\f
+;;;
+;;; Build Farm Front-end (BFFE)
+;;;
+
+(define-record-type* <bffe-configuration>
+  bffe-configuration make-bffe-configuration
+  bffe-configuration?
+  (package       bffe-configuration-package
+                 (default bffe))
+  (user          bffe-configuration-user
+                 (default "bffe"))
+  (group         bffe-configuration-group
+                 (default "bffe"))
+  (arguments     bffe-configuration-arguments)
+  (extra-environment-variables
+   bffe-configuration-extra-environment-variables
+   (default '())))
+
+(define (bffe-shepherd-services config)
+  (define bffe-package
+    (bffe-configuration-package config))
+
+  (define start-script
+    (program-file
+     "run-bffe"
+     (with-extensions (cons
+                       bffe-package
+                       ;; This is a poorly constructed Guile load path,
+                       ;; since it contains things that aren't Guile
+                       ;; libraries, but it means that the Guile
+                       ;; libraries needed for BFFE don't need to be
+                       ;; individually specified here.
+                       (map second (package-transitive-propagated-inputs
+                                    bffe-package)))
+       #~(begin
+           (use-modules (bffe)
+                        (bffe manage-builds))
+
+           (setvbuf (current-output-port) 'line)
+           (setvbuf (current-error-port) 'line)
+
+           (simple-format #t "starting the bffe:\n  ~A\n"
+                          (current-filename))
+
+           (apply run-bffe-service
+                  (append
+                   (list #:pid-file "/var/run/bffe/pid")
+                   #$(bffe-configuration-arguments config)))))
+     #:guile guile-3.0))
+
+  (match-record config <bffe-configuration>
+    (package user group arguments extra-environment-variables)
+
+    (list
+     (shepherd-service
+      (documentation "Build Farm Front-end")
+      (provision '(bffe))
+      (requirement '(networking))
+      (start #~(make-forkexec-constructor
+                (list #$start-script)
+                #:user #$user
+                #:group #$group
+                #:pid-file "/var/run/bffe/pid"
+                #:directory "/var/lib/bffe"
+                #:environment-variables
+                `(,(string-append
+                    "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+                  "LC_ALL=en_US.utf8"
+                  #$@extra-environment-variables)
+                #:log-file "/var/log/bffe/server.log"))
+      (stop #~(make-kill-destructor))))))
+
+(define (bffe-activation config)
+  #~(begin
+      (use-modules (guix build utils))
+
+      (define %user
+        (getpw #$(bffe-configuration-user
+                  config)))
+
+      (chmod "/var/lib/bffe" #o755)
+
+      (mkdir-p "/var/log/bffe")
+
+      ;; Allow writing the PID file
+      (mkdir-p "/var/run/bffe")
+      (chown "/var/run/bffe"
+             (passwd:uid %user)
+             (passwd:gid %user))))
+
+(define (bffe-account config)
+  (match-record config <bffe-configuration>
+    (user group)
+    (list (user-group
+           (name group)
+           (system? #t))
+          (user-account
+           (name user)
+           (group group)
+           (system? #t)
+           (comment "BFFE user")
+           (home-directory "/var/lib/bffe")
+           (shell (file-append shadow "/sbin/nologin"))))))
+
+(define bffe-service-type
+  (service-type
+   (name 'bffe)
+   (extensions
+    (list
+     (service-extension shepherd-root-service-type
+                        bffe-shepherd-services)
+     (service-extension activation-service-type
+                        bffe-activation)
+     (service-extension account-service-type
+                        bffe-account)))
+   (description
+    "Run the Build Farm Front-end.")))
diff --git a/gnu/tests/guix.scm b/gnu/tests/guix.scm
index ad0980a10c..240ded4825 100644
--- a/gnu/tests/guix.scm
+++ b/gnu/tests/guix.scm
@@ -37,7 +37,8 @@ (define-module (gnu tests guix)
   #:use-module (ice-9 match)
   #:export (%test-guix-build-coordinator
             %test-guix-data-service
-            %test-nar-herder))
+            %test-nar-herder
+            %test-bffe))
 
 ;;;
 ;;; Guix Build Coordinator
@@ -325,3 +326,81 @@ (define %test-nar-herder
    (name "nar-herder")
    (description "Connect to a running Nar Herder server.")
    (value (run-nar-herder-test))))
+
+\f
+;;;
+;;; Build Farm Front-end
+;;;
+
+(define %bffe-os
+  (simple-operating-system
+   (service dhcp-client-service-type)
+   (service guix-build-coordinator-service-type)
+   (service bffe-service-type
+            (bffe-configuration
+             (arguments
+              #~(list
+                 #:web-server-args
+                 '(#:port 8767
+                   #:controller-args
+                   (#:title "Test title"))))))))
+
+(define (run-bffe-test)
+  (define os
+    (marionette-operating-system
+     %bffe-os
+     #:imported-modules '((gnu services herd)
+                          (guix combinators))))
+
+  (define forwarded-port 8767)
+
+  (define vm
+    (virtual-machine
+     (operating-system os)
+     (memory-size 1024)
+     (port-forwardings `((,forwarded-port . 8767)))))
+
+  (define test
+    (with-imported-modules '((gnu build marionette))
+      #~(begin
+          (use-modules (srfi srfi-11) (srfi srfi-64)
+                       (gnu build marionette)
+                       (web uri)
+                       (web client)
+                       (web response))
+
+          (define marionette
+            (make-marionette (list #$vm)))
+
+          (test-runner-current (system-test-runner #$output))
+          (test-begin "bffe")
+
+          (test-assert "service running"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (match (start-service 'bffe)
+                  (#f #f)
+                  (('service response-parts ...)
+                   (match (assq-ref response-parts 'running)
+                     ((pid) (number? pid))))))
+             marionette))
+
+          (test-equal "http-get"
+            200
+            (let-values
+                (((response text)
+                  (http-get #$(simple-format
+                               #f "http://localhost:~A/" forwarded-port)
+                            #:decode-body? #t)))
+              (response-code response)))
+
+          (test-end))))
+
+  (gexp->derivation "bffe-test" test))
+
+(define %test-bffe
+  (system-test
+   (name "bffe")
+   (description "Connect to a running Build Farm Front-end.")
+   (value (run-bffe-test))))
-- 
2.41.0

[bug#65732] [PATCH 0/2] Add the Build Farm Front-End

[Ludovic Courtès]

Hello,

Christopher Baines <mail@cbaines.net> skribis:

> * gnu/packages/package-management.scm (bffe): New variable.


[...]

> +             ;; Guile libraries are needed here for cross-compilation.
> +             (car (assoc-ref (package-native-inputs guix) "guile"))

Rather: (lookup-package-native-input guix "guile")

Otherwise LGTM.

> +      (home-page "https://git.cbaines.net/guix/bffe")
> +      (synopsis "Build Farm Front-end for Guix")
> +      (description
> +       "The BFFE of Build Farm Front-end is an experimental frontend for Guix
> +build farms.  It works together with the Guix Data Service and Guix Build
> +Coordinator to submit builds and monitor the activity.

Looks nice!  Have you already deployed it somewhere?

Ludo’.

[bug#65732] [PATCH 0/2] Add the Build Farm Front-End

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> This is intended to replace the functionality of the Guix Build Coordinator
> queue builds script, and also provide a web interface for build farms.
>
> * gnu/services/guix.scm (<bffe-configuration>): New record type.
> (bffe-configuration, bffe-configuration?,
> bffe-configuration-package,
> bffe-configuration-user,
> bffe-configuration-group,
> bffe-configuration-arguments
> bffe-configuration-extra-environment-variables): New procedures.
> (bffe-service-type): New variable.
> * gnu/tests/guix.scm (%test-bffe): New variable.
> * doc/guix.texi (Guix Services): Document the new service.

Overall LGTM.

Nitpick: you can make lines a bit longer, for instance:

> +(define (bffe-activation config)
> +  #~(begin
> +      (use-modules (guix build utils))
> +
> +      (define %user
> +        (getpw #$(bffe-configuration-user
> +                  config)))

‘config’ can be moved to the previous line.

> +      (chown "/var/run/bffe"
> +             (passwd:uid %user)
> +             (passwd:gid %user))))

This can be a single line.

> +   (extensions
> +    (list
> +     (service-extension shepherd-root-service-type
> +                        bffe-shepherd-services)

And: (list (service-extension …

Ludo’.

[bug#65732] [PATCH 0/2] Add the Build Farm Front-End

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1296 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hello,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> * gnu/packages/package-management.scm (bffe): New variable.
>
>
> [...]
>
>> +             ;; Guile libraries are needed here for cross-compilation.
>> +             (car (assoc-ref (package-native-inputs guix) "guile"))
>
> Rather: (lookup-package-native-input guix "guile")

I've actually changed this to guile-next now, as the bffe needs the new
scheme custom/soft ports that don't cause issues with fibers.

>> +      (home-page "https://git.cbaines.net/guix/bffe")
>> +      (synopsis "Build Farm Front-end for Guix")
>> +      (description
>> +       "The BFFE of Build Farm Front-end is an experimental frontend for Guix
>> +build farms.  It works together with the Guix Data Service and Guix Build
>> +Coordinator to submit builds and monitor the activity.
>
> Looks nice!  Have you already deployed it somewhere?

It's been running on bayfront for a while, and manages pages like:

  https://bordeaux.guix.gnu.org/
  https://bordeaux.guix.gnu.org/activity
  https://bordeaux.guix.gnu.org/build/21fa2ed8-b4d0-43ca-8531-e1fa33266708

As well as taking over the job of querying the data service for builds
and submitting them to the coordinator.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#65732: [PATCH 0/2] Add the Build Farm Front-End

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1544 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Christopher Baines <mail@cbaines.net> skribis:
>
>> This is intended to replace the functionality of the Guix Build Coordinator
>> queue builds script, and also provide a web interface for build farms.
>>
>> * gnu/services/guix.scm (<bffe-configuration>): New record type.
>> (bffe-configuration, bffe-configuration?,
>> bffe-configuration-package,
>> bffe-configuration-user,
>> bffe-configuration-group,
>> bffe-configuration-arguments
>> bffe-configuration-extra-environment-variables): New procedures.
>> (bffe-service-type): New variable.
>> * gnu/tests/guix.scm (%test-bffe): New variable.
>> * doc/guix.texi (Guix Services): Document the new service.
>
> Overall LGTM.
>
> Nitpick: you can make lines a bit longer, for instance:
>
>> +(define (bffe-activation config)
>> +  #~(begin
>> +      (use-modules (guix build utils))
>> +
>> +      (define %user
>> +        (getpw #$(bffe-configuration-user
>> +                  config)))
>
> ‘config’ can be moved to the previous line.
>
>> +      (chown "/var/run/bffe"
>> +             (passwd:uid %user)
>> +             (passwd:gid %user))))
>
> This can be a single line.
>
>> +   (extensions
>> +    (list
>> +     (service-extension shepherd-root-service-type
>> +                        bffe-shepherd-services)
>
> And: (list (service-extension …

I've tweaked the formatting as suggested above and pushed this to master
as 82abf6ddadc6139148660440a064e60ae68f238e.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

[bug#65732] [PATCH 0/2] Add the Build Farm Front-End

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> It's been running on bayfront for a while, and manages pages like:
>
>   https://bordeaux.guix.gnu.org/
>   https://bordeaux.guix.gnu.org/activity
>   https://bordeaux.guix.gnu.org/build/21fa2ed8-b4d0-43ca-8531-e1fa33266708
>
> As well as taking over the job of querying the data service for builds
> and submitting them to the coordinator.

Oh, neat!

Ludo’.