* [bug#43494] [PATCH 0/4] Add package and services for the Guix Build Coordinator
@ 2020-09-18 18:34 Christopher Baines
2020-09-18 18:40 ` [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator Christopher Baines
2020-09-19 10:10 ` [bug#43494] [PATCH v2 " Christopher Baines
0 siblings, 2 replies; 23+ messages in thread
From: Christopher Baines @ 2020-09-18 18:34 UTC (permalink / raw)
To: 43494
[-- Attachment #1: Type: text/plain, Size: 694 bytes --]
This patch series is not particularly well tested, like the Guix Build
Coordinator itself, but it should hopefully still work at least in some
configurations.
Christopher Baines (4):
gnu: Add guix-build-coordinator.
services: guix: Add guix-build-coordinator-service-type.
services: guix: Add guix-build-coordinator-agent-service-type.
services: guix: Add guix-build-coordinator-queue-builds-service-type.
doc/guix.texi | 143 +++++++++
gnu/packages/package-management.scm | 89 ++++++
gnu/services/guix.scm | 433 +++++++++++++++++++++++++++-
gnu/tests/guix.scm | 75 ++++-
4 files changed, 736 insertions(+), 4 deletions(-)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator.
2020-09-18 18:34 [bug#43494] [PATCH 0/4] Add package and services for the Guix Build Coordinator Christopher Baines
@ 2020-09-18 18:40 ` Christopher Baines
2020-09-18 18:40 ` [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type Christopher Baines
` (3 more replies)
2020-09-19 10:10 ` [bug#43494] [PATCH v2 " Christopher Baines
1 sibling, 4 replies; 23+ messages in thread
From: Christopher Baines @ 2020-09-18 18:40 UTC (permalink / raw)
To: 43494
* gnu/packages/package-management.scm (guix-build-coordinator): New variable.
---
gnu/packages/package-management.scm | 89 +++++++++++++++++++++++++++++
1 file changed, 89 insertions(+)
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index ec87226197..4272d3134d 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -50,6 +50,7 @@
#:use-module (gnu packages cpio)
#:use-module (gnu packages crypto)
#:use-module (gnu packages curl)
+ #:use-module (gnu packages databases)
#:use-module (gnu packages dbm)
#:use-module (gnu packages docbook)
#:use-module (gnu packages file)
@@ -981,6 +982,94 @@ environments.")
;; and the fonts included in this package are licensed OFL1.1.
(license (list license:gpl3+ license:agpl3+ license:silofl1.1))))
+(define-public guix-build-coordinator
+ (let ((commit "5e8408c833e209efddfa0159114b90400c1aaf4d")
+ (revision "0"))
+ (package
+ (name "guix-build-coordinator")
+ (version (git-version "0" revision commit))
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://git.cbaines.net/git/guix/build-coordinator")
+ (commit commit)))
+ (sha256
+ (base32
+ "0f7m1zg9mlb2m22qyblglaa36h8f49b810jc9j5b0hsb42ijwh4b"))
+ (file-name (string-append name "-" version "-checkout"))))
+ (build-system gnu-build-system)
+ (arguments
+ '(#:modules ((guix build utils)
+ (guix build gnu-build-system)
+ (ice-9 ftw)
+ (ice-9 match)
+ (ice-9 rdelim)
+ (ice-9 popen))
+ #:phases
+ (modify-phases %standard-phases
+ (add-before 'build 'set-GUILE_AUTO_COMPILE
+ (lambda _
+ ;; To avoid warnings relating to 'guild'.
+ (setenv "GUILE_AUTO_COMPILE" "0")
+ #t))
+ (add-after 'install 'wrap-executable
+ (lambda* (#:key inputs outputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (bin (string-append out "/bin"))
+ (guile (assoc-ref inputs "guile"))
+ (guile-effective-version
+ (read-line
+ (open-pipe* OPEN_READ
+ (string-append guile "/bin/guile")
+ "-c" "(display (effective-version))")))
+ (scm (string-append out "/share/guile/site/"
+ guile-effective-version))
+ (go (string-append out "/lib/guile/"
+ guile-effective-version
+ "/site-ccache")))
+ (for-each
+ (lambda (file)
+ (simple-format (current-error-port)
+ "wrapping: ~A\n"
+ (string-append bin "/" file))
+ (wrap-program (string-append bin "/" file)
+ `("PATH" ":" prefix
+ (,bin ,(assoc-ref inputs "sqitch")))
+ `("PERL5LIB" ":" prefix
+ (,(getenv "PERL5LIB")))
+ `("GUILE_LOAD_PATH" ":" prefix
+ (,scm ,(getenv "GUILE_LOAD_PATH")))
+ `("GUILE_LOAD_COMPILED_PATH" ":" prefix
+ (,go ,(getenv "GUILE_LOAD_COMPILED_PATH")))))
+ (scandir bin
+ (match-lambda
+ ((or "." "..") #f)
+ (_ #t))))
+ #t)))
+ (delete 'strip)))) ; As the .go files aren't compatible
+ (native-inputs
+ `(("pkg-config" ,pkg-config)
+ ("autoconf" ,autoconf)
+ ("automake" ,automake)
+ ("perl" ,perl)))
+ (inputs
+ `(("guile" ,guile-3.0-latest)
+ ("guile-fibers" ,guile-fibers)
+ ("guile-prometheus" ,guile-prometheus)
+ ("guile-gcrypt" ,guile-gcrypt)
+ ("guile-json" ,guile-json-3)
+ ("guile-lzlib" ,guile-lzlib)
+ ("guile-sqlite3" ,guile-sqlite3)
+ ("guix" ,guix)
+ ("sqlite" ,sqlite)
+ ("sqitch" ,sqitch)
+ ("perl-dbd-sqlite" ,perl-dbd-sqlite)))
+ (home-page "https://git.cbaines.net/guile/guix/build-coordinator")
+ (synopsis "")
+ (description
+ "")
+ (license license:gpl3+))))
+
(define-public guix-jupyter
(package
(name "guix-jupyter")
--
2.28.0
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
2020-09-18 18:40 ` [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator Christopher Baines
@ 2020-09-18 18:40 ` Christopher Baines
2020-09-18 20:20 ` Ludovic Courtès
2020-09-18 18:40 ` [bug#43494] [PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type Christopher Baines
` (2 subsequent siblings)
3 siblings, 1 reply; 23+ messages in thread
From: Christopher Baines @ 2020-09-18 18:40 UTC (permalink / raw)
To: 43494
* gnu/services/guix.scm (<guix-build-coordinator-configuration>): New record
type.
(guix-build-coordinator-configuration, guix-build-coordinator-configuration?,
guix-build-coordinator-configuration-package,
guix-build-coordinator-configuration-user,
guix-build-coordinator-configuration-group,
guix-build-coordinator-configuration-datastore-uri-string,
guix-build-coordinator-configuration-agent-communication-uri-string,
guix-build-coordinator-configuration-client-communication-uri-string,
guix-build-coordinator-configuration-allocation-strategy,
guix-build-coordinator-configuration-hooks,
guix-build-coordinator-configuration-guile,
make-guix-build-coordinator-start-script,
guix-build-coordinator-shepherd-services, guix-build-coordinator-activation,
guix-build-coordinator-account): New procedures.
(guix-build-coordinator-service-type): New variable.
* gnu/tests/guix.scm (%test-guix-build-coordinator): New variable.
* doc/guix.texi (Guix Services): Document it.
---
doc/guix.texi | 55 ++++++++++++
gnu/services/guix.scm | 204 +++++++++++++++++++++++++++++++++++++++++-
gnu/tests/guix.scm | 75 +++++++++++++++-
3 files changed, 330 insertions(+), 4 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index 88128a4b3a..e83008c177 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -27486,6 +27486,61 @@ The complete list of possible options can be found in the man page for
@node Guix Services
@subsection Guix Services
+@subsubheading Guix Build Coordinator
+The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build
+Coordinator} aids in building derivations. The Guix Daemon is still
+used to build the derivations, but the Guix Build Coordinator provides
+ways of performing builds across many machines and working with the
+results of builds.
+
+There is a script to run the coordinator component of the Guix Build
+Coordinator, but the Guix service uses a custom Guile script instead, to
+provide better integration with G-expressions used in the configuration.
+
+@defvar {Scheme Variable} guix-build-coordinator-service-type
+Service type for the Guix Build Coordinator. Its value must be a
+@code{guix-build-coordinator-configuration} object.
+@end defvar
+
+@deftp {Data Type} guix-build-coordinator-configuration
+Data type representing the configuration of the Guix Build Coordinator.
+
+@table @asis
+@item @code{package} (default: @code{guix-build-coordinator})
+The Guix Build Coordinator package to use.
+
+@item @code{user} (default: @code{"guix-build-coordinator"})
+The system user to run the service as.
+
+@item @code{group} (default: @code{"guix-build-coordinator"})
+The system group to run the service as.
+
+@item @code{database-uri-string} (default: @code{"sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"})
+The URI to use for the database.
+
+@item @code{agent-communication-uri} (default: @code{"http://0.0.0.0:8745"})
+The URI describing how to listen to requests from agent processes.
+
+@item @code{client-communication-uri} (default: @code{"http://127.0.0.1:8746"})
+The URI describing how to listen to requests from clients. The client
+API allows submitting builds and currently isn't authenticated, so take
+care when configuring this value.
+
+@item @code{allocation-strategy} (default: @code{#~basic-build-allocation-strategy})
+A G-expression for the allocation strategy to be used. This is a
+procedure that takes the datastore as an argument and populates the
+allocation plan in the database.
+
+@item @code{hooks} (default: @var{'()})
+An association list of hooks. These provide a way to execute arbitrary
+code upon certian events, like a build result being processed.
+
+@item @code{guile} (default: @code{guile-3.0-latest})
+The Guile package with which to run the Guix Build Coordinator.
+
+@end table
+@end deftp
+
@subsubheading Guix Data Service
The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores
and provides data about GNU Guix. This includes information about
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index 10a8581a62..de14fcadb2 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -17,20 +17,40 @@
;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
(define-module (gnu services guix)
+ #:use-module (srfi srfi-1)
#:use-module (ice-9 match)
#:use-module (guix gexp)
#:use-module (guix records)
+ #:use-module (guix packages)
#:use-module ((gnu packages base)
#:select (glibc-utf8-locales))
#:use-module (gnu packages admin)
+ #:use-module (gnu packages databases)
#:use-module (gnu packages web)
+ #:use-module (gnu packages guile)
+ #:use-module (gnu packages guile-xyz)
+ #:use-module (gnu packages package-management)
#:use-module (gnu services)
#:use-module (gnu services base)
#:use-module (gnu services admin)
#:use-module (gnu services shepherd)
#:use-module (gnu services getmail)
#:use-module (gnu system shadow)
- #:export (<guix-data-service-configuration>
+ #:export (guix-build-coordinator-configuration
+ guix-build-coordinator-configuration?
+ guix-build-coordinator-configuration-package
+ guix-build-coordinator-configuration-user
+ guix-build-coordinator-configuration-group
+ guix-build-coordinator-configuration-datastore-uri-string
+ guix-build-coordinator-configuration-agent-communication-uri-string
+ guix-build-coordinator-configuration-client-communication-uri-string
+ guix-build-coordinator-configuration-allocation-strategy
+ guix-build-coordinator-configuration-hooks
+ guix-build-coordinator-configuration-guile
+
+ guix-build-coordinator-service-type
+
+ <guix-data-service-configuration>
guix-data-service-configuration
guix-data-service-configuration?
guix-data-service-package
@@ -45,11 +65,189 @@
;;;; Commentary:
;;;
-;;; This module implements a service that to run instances of the Guix Data
-;;; Service, which provides data about Guix over time.
+;;; Services specifically related to GNU Guix.
;;;
;;;; Code:
+(define-record-type* <guix-build-coordinator-configuration>
+ guix-build-coordinator-configuration make-guix-build-coordinator-configuration
+ guix-build-coordinator-configuration?
+ (package guix-build-coordinator-configuration-package
+ (default guix-build-coordinator))
+ (user guix-build-coordinator-configuration-user
+ (default "guix-build-coordinator"))
+ (group guix-build-coordinator-configuration-group
+ (default "guix-build-coordinator"))
+ (database-uri-string
+ guix-build-coordinator-configuration-datastore-uri-string
+ (default "sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"))
+ (agent-communication-uri-string
+ guix-build-coordinator-configuration-agent-communication-uri-string
+ (default "http://0.0.0.0:8745"))
+ (client-communication-uri-string
+ guix-build-coordinator-configuration-client-communication-uri-string
+ (default "http://127.0.0.1:8746"))
+ (allocation-strategy
+ guix-build-coordinator-configuration-allocation-strategy
+ (default #~basic-build-allocation-strategy))
+ (hooks guix-build-coordinator-configuration-hooks
+ (default '()))
+ (guile guix-build-coordinator-configuration-guile
+ (default guile-3.0-latest)))
+
+(define* (make-guix-build-coordinator-start-script database-uri-string
+ allocation-strategy
+ pid-file
+ guix-build-coordinator-package
+ #:key
+ agent-communication-uri-string
+ client-communication-uri-string
+ (hooks '())
+ (guile guile-3.0))
+ (program-file
+ "start-guix-build-coordinator"
+ (with-extensions (cons guix-build-coordinator-package
+ ;; This is a poorly constructed Guile load path,
+ ;; since it contains things that aren't Guile
+ ;; libraries, but it means that the Guile libraries
+ ;; needed for the Guix Build Coordinator don't need
+ ;; to be individually specified here.
+ (map second (package-inputs
+ guix-build-coordinator-package)))
+ #~(begin
+ (use-modules (srfi srfi-1)
+ (ice-9 match)
+ (web uri)
+ (prometheus)
+ (guix-build-coordinator hooks)
+ (guix-build-coordinator datastore)
+ (guix-build-coordinator build-allocator)
+ (guix-build-coordinator coordinator))
+
+ ;; Otherwise sqitch can't use SQlite
+ (setenv "PERL5LIB"
+ #$(file-append perl-dbd-sqlite "/lib/perl5/site_perl"))
+
+ (let* ((metrics-registry (make-metrics-registry
+ #:namespace
+ "guixbuildcoordinator_"))
+ (datastore (database-uri->datastore
+ #$database-uri-string
+ #:metrics-registry metrics-registry))
+ (hooks
+ (list #$@(map (match-lambda
+ ((name . hook-gexp)
+ #~(cons name #$hook-gexp)))
+ hooks)))
+ (hooks-with-defaults
+ `(,@hooks
+ ,@(remove (match-lambda
+ ((name . _) (assq-ref hooks name)))
+ %default-hooks)))
+ (build-coordinator (make-build-coordinator
+ #:datastore datastore
+ #:hooks hooks-with-defaults
+ #:metrics-registry metrics-registry
+ #:allocation-strategy #$allocation-strategy)))
+
+ (run-coordinator-service
+ build-coordinator
+ #:update-datastore? #t
+ #:pid-file #$pid-file
+ #:agent-communication-uri (string->uri
+ #$agent-communication-uri-string)
+ #:client-communication-uri (string->uri
+ #$client-communication-uri-string)))))
+ #:guile guile))
+
+(define (guix-build-coordinator-shepherd-services config)
+ (match-record config <guix-build-coordinator-configuration>
+ (package user group database-uri-string
+ agent-communication-uri-string
+ client-communication-uri-string
+ allocation-strategy
+ hooks
+ guile)
+ (list
+ (shepherd-service
+ (documentation "Guix Build Coordinator")
+ (provision '(guix-build-coordinator))
+ (requirement '(networking))
+ (start #~(make-forkexec-constructor
+ (list #$(make-guix-build-coordinator-start-script
+ database-uri-string
+ allocation-strategy
+ "/var/run/guix-build-coordinator/pid"
+ package
+ #:agent-communication-uri-string
+ agent-communication-uri-string
+ #:client-communication-uri-string
+ client-communication-uri-string
+ #:hooks hooks
+ #:guile guile))
+ #:user #$user
+ #:group #$group
+ #:pid-file "/var/run/guix-build-coordinator/pid"
+ ;; Allow time for migrations to run
+ #:pid-file-timeout 60
+ #:environment-variables
+ `(,(string-append
+ "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+ "LC_ALL=en_US.utf8")
+ #:log-file "/var/log/guix-build-coordinator/coordinator.log"))
+ (stop #~(make-kill-destructor))))))
+
+(define (guix-build-coordinator-activation config)
+ #~(begin
+ (use-modules (guix build utils))
+
+ (define %user (getpw "guix-build-coordinator"))
+
+ (chmod "/var/lib/guix-build-coordinator" #o755)
+
+ (mkdir-p "/var/log/guix-build-coordinator")
+
+ ;; Allow writing the PID file
+ (mkdir-p "/var/run/guix-build-coordinator")
+ (chown "/var/run/guix-build-coordinator"
+ (passwd:uid %user)
+ (passwd:gid %user))))
+
+(define (guix-build-coordinator-account config)
+ (match-record config <guix-build-coordinator-configuration>
+ (user group)
+ (list (user-group
+ (name group)
+ (system? #t))
+ (user-account
+ (name user)
+ (group group)
+ (system? #t)
+ (comment "Guix Build Coordinator user")
+ (home-directory "/var/lib/guix-build-coordinator")
+ (shell (file-append shadow "/sbin/nologin"))))))
+
+(define guix-build-coordinator-service-type
+ (service-type
+ (name 'guix-build-coordinator)
+ (extensions
+ (list
+ (service-extension shepherd-root-service-type
+ guix-build-coordinator-shepherd-services)
+ (service-extension activation-service-type
+ guix-build-coordinator-activation)
+ (service-extension account-service-type
+ guix-build-coordinator-account)))
+ (default-value
+ (guix-build-coordinator-configuration))
+ (description
+ "Run an instance of the Guix Build Coordinator.")))
+
+\f
+;;;
+;;; Guix Data Service
+;;;
+
(define-record-type* <guix-data-service-configuration>
guix-data-service-configuration make-guix-data-service-configuration
guix-data-service-configuration?
diff --git a/gnu/tests/guix.scm b/gnu/tests/guix.scm
index 6139e31cf0..20b67d55d3 100644
--- a/gnu/tests/guix.scm
+++ b/gnu/tests/guix.scm
@@ -35,7 +35,80 @@
#:use-module (guix store)
#:use-module (guix utils)
#:use-module (ice-9 match)
- #:export (%test-guix-data-service))
+ #:export (%test-guix-build-coordinator
+ %test-guix-data-service))
+
+;;;
+;;; Guix Build Coordinator
+;;;
+
+(define %guix-build-coordinator-os
+ (simple-operating-system
+ (service dhcp-client-service-type)
+ (service guix-build-coordinator-service-type)))
+
+(define (run-guix-build-coordinator-test)
+ (define os
+ (marionette-operating-system
+ %guix-build-coordinator-os
+ #:imported-modules '((gnu services herd)
+ (guix combinators))))
+
+ (define forwarded-port 8745)
+
+ (define vm
+ (virtual-machine
+ (operating-system os)
+ (memory-size 1024)
+ (port-forwardings `((,forwarded-port . 8745)))))
+
+ (define test
+ (with-imported-modules '((gnu build marionette))
+ #~(begin
+ (use-modules (srfi srfi-11) (srfi srfi-64)
+ (gnu build marionette)
+ (web uri)
+ (web client)
+ (web response))
+
+ (define marionette
+ (make-marionette (list #$vm)))
+
+ (mkdir #$output)
+ (chdir #$output)
+
+ (test-begin "guix-build-coordinator")
+
+ (test-assert "service running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (match (start-service 'guix-build-coordinator)
+ (#f #f)
+ (('service response-parts ...)
+ (match (assq-ref response-parts 'running)
+ ((pid) (number? pid))))))
+ marionette))
+
+ (test-equal "http-get"
+ 200
+ (let-values
+ (((response text)
+ (http-get #$(simple-format
+ #f "http://localhost:~A/metrics" forwarded-port)
+ #:decode-body? #t)))
+ (response-code response)))
+
+ (test-end)
+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+
+ (gexp->derivation "guix-build-coordinator-test" test))
+
+(define %test-guix-build-coordinator
+ (system-test
+ (name "guix-build-coordinator")
+ (description "Connect to a running Guix Build Coordinator.")
+ (value (run-guix-build-coordinator-test))))
\f
;;;
--
2.28.0
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type.
2020-09-18 18:40 ` [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator Christopher Baines
2020-09-18 18:40 ` [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type Christopher Baines
@ 2020-09-18 18:40 ` Christopher Baines
2020-09-18 20:25 ` Ludovic Courtès
2020-09-18 18:40 ` [bug#43494] [PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type Christopher Baines
2020-09-18 20:08 ` [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator Ludovic Courtès
3 siblings, 1 reply; 23+ messages in thread
From: Christopher Baines @ 2020-09-18 18:40 UTC (permalink / raw)
To: 43494
* gnu/services/guix.scm (<guix-build-coordinator-agent-configuration>): New
record type.
(guix-build-coordinator-agent-configuration,
guix-build-coordinator-agent-configuration?,
guix-build-coordinator-agent-configuration-package,
guix-build-coordinator-agent-configuration-user,
guix-build-coordinator-agent-configuration-coordinator,
guix-build-coordinator-agent-configuration-uuid),
guix-build-coordinator-agent-configuration-password,
guix-build-coordinator-agent-configuration-password-file,
guix-build-coordinator-agent-configuration-systems,
guix-build-coordinator-agent-configuration-max-parallel-builds,
guix-build-coordinator-agent-configuration-derivation-substitute-urls,
guix-build-coordinator-agent-configuration-non-derivation-substitute-urls,
guix-build-coordinator-agent-shepherd-services,
guix-build-coordinator-agent-activation,
guix-build-coordinator-agent-account): New procedures.
(guix-build-coordinator-agent-service-type): New variable.
* doc/guix.texi (Guix Services): Document it.
---
doc/guix.texi | 50 +++++++++++++++++
gnu/services/guix.scm | 121 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 171 insertions(+)
diff --git a/doc/guix.texi b/doc/guix.texi
index e83008c177..db91831197 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -27541,6 +27541,56 @@ The Guile package with which to run the Guix Build Coordinator.
@end table
@end deftp
+@defvar {Scheme Variable} guix-build-coordinator-agent-service-type
+Service type for a Guix Build Coordinator agent. Its value must be a
+@code{guix-build-coordinator-agent-configuration} object.
+@end defvar
+
+@deftp {Data Type} guix-build-coordinator-agent-configuration
+Data type representing the configuration a Guix Build Coordinator agent.
+
+@table @asis
+@item @code{package} (default: @code{guix-build-coordinator})
+The Guix Build Coordinator package to use.
+
+@item @code{user} (default: @code{"guix-build-coordinator-agent"})
+The system user to run the service as.
+
+@item @code{coordinator} (default: @code{"http://localhost:8745"})
+The URI to use when connecting to the coordinator.
+
+@item @code{uuid}
+The UUID of the agent. This should be generated by the coordinator
+process, stored in the coordinator database, and used by the intended
+agent.
+
+@item @code{password} (default: @code{#f})
+The password to use when connecting to the coordinator. A file to read
+the password from can also be specified, and this is more secure.
+
+@item @code{password-file} (default: @code{#f})
+A file containing the password to use when connecting to the
+coordinator.
+
+@item @code{systems} (default: @var{#f})
+The systems for which this agent should fetch builds. The agent process
+will use the current system it's running on as the default.
+
+@item @code{max-parallel-builds} (default: @code{1})
+The number of builds to perform in parallel.
+
+@item @code{derivation-substitute-urls} (default: @code{1})
+URLs from which to attempt to fetch substitutes for derivations, if the
+derivations aren't already available.
+
+@item @code{non-derivation-substitute-urls} (default: @code{1})
+URLs from which to attempt to fetch substitutes for build inputs, if the
+input store items aren't already available.
+
+@end table
+@end deftp
+
+
@subsubheading Guix Data Service
The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores
and provides data about GNU Guix. This includes information about
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index de14fcadb2..71e58237da 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -50,6 +50,21 @@
guix-build-coordinator-service-type
+ guix-build-coordinator-agent-configuration
+ guix-build-coordinator-agent-configuration?
+ guix-build-coordinator-agent-configuration-package
+ guix-build-coordinator-agent-configuration-user
+ guix-build-coordinator-agent-configuration-coordinator
+ guix-build-coordinator-agent-configuration-uuid)
+ guix-build-coordinator-agent-configuration-password
+ guix-build-coordinator-agent-configuration-password-file
+ guix-build-coordinator-agent-configuration-systems
+ guix-build-coordinator-agent-configuration-max-parallel-builds
+ guix-build-coordinator-agent-configuration-derivation-substitute-urls
+ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls
+
+ guix-build-coordinator-agent-service-type
+
<guix-data-service-configuration>
guix-data-service-configuration
guix-data-service-configuration?
@@ -95,6 +110,33 @@
(guile guix-build-coordinator-configuration-guile
(default guile-3.0-latest)))
+(define-record-type* <guix-build-coordinator-agent-configuration>
+ guix-build-coordinator-agent-configuration
+ make-guix-build-coordinator-agent-configuration
+ guix-build-coordinator-agent-configuration?
+ (package guix-build-coordinator-agent-configuration-package
+ (default guix-build-coordinator))
+ (user guix-build-coordinator-agent-configuration-user
+ (default "guix-build-coordinator-agent"))
+ (coordinator guix-build-coordinator-agent-configuration-coordinator
+ (default "http://localhost:8745"))
+ (uuid guix-build-coordinator-agent-configuration-uuid)
+ (password guix-build-coordinator-agent-configuration-password
+ (default #f))
+ (password-file guix-build-coordinator-agent-configuration-password-file
+ (default #f))
+ (systems guix-build-coordinator-agent-configuration-systems
+ (default #f))
+ (max-parallel-builds
+ guix-build-coordinator-agent-configuration-max-parallel-builds
+ (default 1))
+ (derivation-substitute-urls
+ guix-build-coordinator-agent-configuration-derivation-substitute-urls
+ (default #f))
+ (non-derivation-substitute-urls
+ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls
+ (default #f)))
+
(define* (make-guix-build-coordinator-start-script database-uri-string
allocation-strategy
pid-file
@@ -243,6 +285,85 @@
(description
"Run an instance of the Guix Build Coordinator.")))
+(define (guix-build-coordinator-agent-shepherd-services config)
+ (match-record config <guix-build-coordinator-agent-configuration>
+ (package user coordinator uuid password password-file max-parallel-builds
+ derivation-substitute-urls non-derivation-substitute-urls
+ systems)
+ (list
+ (shepherd-service
+ (documentation "Guix Build Coordinator Agent")
+ (provision '(guix-build-coordinator-agent))
+ (requirement '(networking))
+ (start #~(make-forkexec-constructor
+ (list #$(file-append package "/bin/guix-build-coordinator-agent")
+ #$(string-append "--coordinator=" coordinator)
+ #$(string-append "--uuid=" uuid)
+ #$@(if password
+ #~(#$(string-append "--password=" password))
+ #~())
+ #$@(if password-file
+ #~(#$(string-append "--password-file=" password-file))
+ #~())
+ #$(simple-format #f "--max-parallel-builds=~A"
+ max-parallel-builds)
+ #$@(if derivation-substitute-urls
+ #~(#$(string-append
+ "--derivation-substitute-urls="
+ (string-join derivation-substitute-urls " ")))
+ #~())
+ #$@(if non-derivation-substitute-urls
+ #~(#$(string-append
+ "--non-derivation-substitute-urls="
+ (string-join derivation-substitute-urls " ")))
+ #~())
+ #$@(map (lambda (system)
+ (string-append "--system=" system))
+ (or systems '())))
+ #:user #$user
+ #:pid-file "/var/run/guix-build-coordinator-agent/pid"
+ #:environment-variables
+ `(,(string-append
+ "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+ "LC_ALL=en_US.utf8")
+ #:log-file "/var/log/guix-build-coordinator/agent.log"))
+ (stop #~(make-kill-destructor))))))
+
+(define (guix-build-coordinator-agent-activation config)
+ #~(begin
+ (use-modules (guix build utils))
+
+ (mkdir-p "/var/log/guix-build-coordinator")
+
+ ;; Allow writing the PID file
+ (mkdir-p "/var/run/guix-build-coordinator-agent")
+ (chown "/var/run/guix-build-coordinator-agent"
+ (passwd:uid %user)
+ (passwd:gid %user))))
+
+(define (guix-build-coordinator-agent-account config)
+ (list (user-account
+ (name (guix-build-coordinator-agent-configuration-user config))
+ (group "nogroup")
+ (system? #t)
+ (comment "Guix Build Coordinator agent user")
+ (home-directory "/var/empty")
+ (shell (file-append shadow "/sbin/nologin")))))
+
+(define guix-build-coordinator-agent-service-type
+ (service-type
+ (name 'guix-build-coordinator-agent)
+ (extensions
+ (list
+ (service-extension shepherd-root-service-type
+ guix-build-coordinator-agent-shepherd-services)
+ (service-extension activation-service-type
+ guix-build-coordinator-agent-activation)
+ (service-extension account-service-type
+ guix-build-coordinator-agent-account)))
+ (description
+ "Run an instance of the Guix Build Coordinator.")))
+
\f
;;;
;;; Guix Data Service
--
2.28.0
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
2020-09-18 18:40 ` [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator Christopher Baines
2020-09-18 18:40 ` [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type Christopher Baines
2020-09-18 18:40 ` [bug#43494] [PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type Christopher Baines
@ 2020-09-18 18:40 ` Christopher Baines
2020-09-18 20:31 ` Ludovic Courtès
2020-09-18 20:08 ` [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator Ludovic Courtès
3 siblings, 1 reply; 23+ messages in thread
From: Christopher Baines @ 2020-09-18 18:40 UTC (permalink / raw)
To: 43494
* gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):
New record type.
(guix-build-coordinator-queue-builds-configuration,
guix-build-coordinator-queue-builds-configuration?,
guix-build-coordinator-queue-builds-configuration-package,
guix-build-coordinator-queue-builds-configuration-user,
guix-build-coordinator-queue-builds-coordinator,
guix-build-coordinator-queue-builds-configuration-systems,
guix-build-coordinator-queue-builds-configuration-system-and-targets,
guix-build-coordinator-queue-builds-configuration-guix-data-service,
guix-build-coordinator-queue-builds-configuration-processed-commits-file,
guix-build-coordinator-queue-builds-shepherd-services,
guix-build-coordinator-queue-builds-activation,
guix-build-coordinator-queue-builds-account): New procedures.
(guix-build-coordinator-queue-builds-service-type): New variable.
---
doc/guix.texi | 38 +++++++++++++++
gnu/services/guix.scm | 108 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 146 insertions(+)
diff --git a/doc/guix.texi b/doc/guix.texi
index db91831197..1b47a01217 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -27590,6 +27590,44 @@ input store items aren't already available.
@end table
@end deftp
+@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type
+Service type for the
+guix-build-coordinator-queue-builds-from-guix-data-service script. Its
+value must be a @code{guix-build-coordinator-queue-builds-configuration}
+object.
+@end defvar
+
+@deftp {Data Type} guix-build-coordinator-queue-builds-configuration
+Data type representing the options to the queue builds from guix data
+service script.
+
+@table @asis
+@item @code{package} (default: @code{guix-build-coordinator})
+The Guix Build Coordinator package to use.
+
+@item @code{user} (default: @code{"guix-build-coordinator-queue-builds"})
+The system user to run the service as.
+
+@item @code{coordinator} (default: @code{"http://localhost:8745"})
+The URI to use when connecting to the coordinator.
+
+@item @code{systems} (default: @code{#f})
+The systems for which to fetch derivations to build.
+
+@item @code{systems-and-targets} (default: @code{#f})
+An association list of system and target pairs for which to fetch
+derivations to build.
+
+@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"})
+The Guix Data Service instance from which to query to find out about
+derivations to build.
+
+@item @code{processed-commits-file} (default: @code{"/var/lib/guix-build-coordinator-queue-builds/processed-commits"})
+A file to record which commits have been processed, to avoid needlessly
+processing them again if the service is restarted.
+
+@end table
+@end deftp
@subsubheading Guix Data Service
The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index 71e58237da..975febebd1 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -65,6 +65,18 @@
guix-build-coordinator-agent-service-type
+ guix-build-coordinator-queue-builds-configuration
+ guix-build-coordinator-queue-builds-configuration?
+ guix-build-coordinator-queue-builds-configuration-package
+ guix-build-coordinator-queue-builds-configuration-user
+ guix-build-coordinator-queue-builds-coordinator
+ guix-build-coordinator-queue-builds-configuration-systems
+ guix-build-coordinator-queue-builds-configuration-system-and-targets
+ guix-build-coordinator-queue-builds-configuration-guix-data-service
+ guix-build-coordinator-queue-builds-configuration-processed-commits-file
+
+ guix-build-coordinator-queue-builds-service-type
+
<guix-data-service-configuration>
guix-data-service-configuration
guix-data-service-configuration?
@@ -137,6 +149,28 @@
guix-build-coordinator-agent-configuration-non-derivation-substitute-urls
(default #f)))
+(define-record-type* <guix-build-coordinator-queue-builds-configuration>
+ guix-build-coordinator-queue-builds-configuration
+ make-guix-build-coordinator-queue-builds-configuration
+ guix-build-coordinator-queue-builds-configuration?
+ (package guix-build-coordinator-queue-builds-configuration-package
+ (default guix-build-coordinator))
+ (user guix-build-coordinator-queue-builds-configuration-user
+ (default "guix-build-coordinator-queue-builds"))
+ (coordinator guix-build-coordinator-queue-builds-coordinator
+ (default "http://localhost:8745"))
+ (systems guix-build-coordinator-queue-builds-configuration-systems
+ (default #f))
+ (systems-and-targets
+ guix-build-coordinator-queue-builds-configuration-system-and-targets
+ (default #f))
+ (guix-data-service
+ guix-build-coordinator-queue-builds-configuration-guix-data-service
+ (default "https://data.guix.gnu.org"))
+ (processed-commits-file
+ guix-build-coordinator-queue-builds-configuration-processed-commits-file
+ (default "/var/lib/guix-build-coordinator-queue-builds/processed-commits")))
+
(define* (make-guix-build-coordinator-start-script database-uri-string
allocation-strategy
pid-file
@@ -364,6 +398,80 @@
(description
"Run an instance of the Guix Build Coordinator.")))
+(define (guix-build-coordinator-queue-builds-shepherd-services config)
+ (match-record config <guix-build-coordinator-queue-builds-configuration>
+ (package user coordinator systems systems-and-targets
+ guix-data-service processed-commits-file)
+ (list
+ (shepherd-service
+ (documentation "Guix Build Coordinator queue builds from Guix Data Service")
+ (provision '(guix-build-coordinator-queue-builds))
+ (requirement '(networking))
+ (start
+ #~(make-forkexec-constructor
+ (list
+ #$(file-append
+ package
+ "/bin/guix-build-coordinator-queue-builds-from-guix-data-service")
+ #$(string-append "--coordinator=" coordinator)
+ #$@(map (lambda (system)
+ (string-append "--system=" system))
+ (or systems '()))
+ #$@(map (match-lambda
+ ((system . target)
+ (string-append "--system-and-target=" system "=" target)))
+ (or systems-and-targets '()))
+ #$@(if guix-data-service
+ #~(#$(string-append "--guix-data-service=" guix-data-service))
+ #~())
+ #$@(if processed-commits-file
+ #~(#$(string-append "--processed-commits-file="
+ processed-commits-file))
+ #~()))
+ #:user #$user
+ #:pid-file "/var/run/guix-build-coordinator-queue-builds/pid"
+ #:environment-variables
+ `(,(string-append
+ "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+ "LC_ALL=en_US.utf8")
+ #:log-file "/var/log/guix-build-coordinator/queue-builds.log"))
+ (stop #~(make-kill-destructor))))))
+
+(define (guix-build-coordinator-queue-builds-activation config)
+ #~(begin
+ (use-modules (guix build utils))
+
+ (mkdir-p "/var/log/guix-build-coordinator")
+
+ ;; Allow writing the PID file
+ (mkdir-p "/var/run/guix-build-coordinator-queue-builds")
+ (chown "/var/run/guix-build-coordinator-queue-builds"
+ (passwd:uid %user)
+ (passwd:gid %user))))
+
+(define (guix-build-coordinator-queue-builds-account config)
+ (list (user-account
+ (name (guix-build-coordinator-queue-builds-configuration-user config))
+ (group "nogroup")
+ (system? #t)
+ (comment "Guix Build Coordinator queue-builds user")
+ (home-directory "/var/empty")
+ (shell (file-append shadow "/sbin/nologin")))))
+
+(define guix-build-coordinator-queue-builds-service-type
+ (service-type
+ (name 'guix-build-coordinator-queue-builds)
+ (extensions
+ (list
+ (service-extension shepherd-root-service-type
+ guix-build-coordinator-queue-builds-shepherd-services)
+ (service-extension activation-service-type
+ guix-build-coordinator-queue-builds-activation)
+ (service-extension account-service-type
+ guix-build-coordinator-queue-builds-account)))
+ (description
+ "Run the Guix Build Coordinator queue builds script.")))
+
\f
;;;
;;; Guix Data Service
--
2.28.0
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator.
2020-09-18 18:40 ` [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator Christopher Baines
` (2 preceding siblings ...)
2020-09-18 18:40 ` [bug#43494] [PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type Christopher Baines
@ 2020-09-18 20:08 ` Ludovic Courtès
2020-09-18 23:34 ` Jonathan Brielmaier
2020-09-19 9:05 ` Christopher Baines
3 siblings, 2 replies; 23+ messages in thread
From: Ludovic Courtès @ 2020-09-18 20:08 UTC (permalink / raw)
To: Christopher Baines; +Cc: 43494
Hi!
Christopher Baines <mail@cbaines.net> skribis:
> * gnu/packages/package-management.scm (guix-build-coordinator): New variable.
[...]
> + (lambda* (#:key inputs outputs #:allow-other-keys)
> + (let* ((out (assoc-ref outputs "out"))
> + (bin (string-append out "/bin"))
> + (guile (assoc-ref inputs "guile"))
> + (guile-effective-version
> + (read-line
> + (open-pipe* OPEN_READ
> + (string-append guile "/bin/guile")
> + "-c" "(display (effective-version))")))
Maybe use ‘target-guile-effective-version’ from (guix build
guile-build-system).
> + (wrap-program (string-append bin "/" file)
> + `("PATH" ":" prefix
> + (,bin ,(assoc-ref inputs "sqitch")))
> + `("PERL5LIB" ":" prefix
> + (,(getenv "PERL5LIB")))
Do we really need PERL5LIB here? Shouldn’t it be done in the wrapper of
‘sqitch’ directly?
> + (scandir bin
> + (match-lambda
> + ((or "." "..") #f)
> + (_ #t))))
I think you could just use (find-files bin).
> + (inputs
> + `(("guile" ,guile-3.0-latest)
Rather ,@(assoc-ref (package-native-inputs guix) "guile").
> + ("sqitch" ,sqitch)
> + ("perl-dbd-sqlite" ,perl-dbd-sqlite)))
Shouldn’t perl-dbd-sqlite be a dependency of sqitch?
> + (home-page "https://git.cbaines.net/guile/guix/build-coordinator")
> + (synopsis "")
> + (description
> + "")
Missing! :-)
LGTM with changes along these lines!
Ludo’.
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
2020-09-18 18:40 ` [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type Christopher Baines
@ 2020-09-18 20:20 ` Ludovic Courtès
2020-09-19 9:13 ` Christopher Baines
0 siblings, 1 reply; 23+ messages in thread
From: Ludovic Courtès @ 2020-09-18 20:20 UTC (permalink / raw)
To: Christopher Baines; +Cc: 43494
Christopher Baines <mail@cbaines.net> skribis:
> * gnu/services/guix.scm (<guix-build-coordinator-configuration>): New record
> type.
> (guix-build-coordinator-configuration, guix-build-coordinator-configuration?,
> guix-build-coordinator-configuration-package,
> guix-build-coordinator-configuration-user,
> guix-build-coordinator-configuration-group,
> guix-build-coordinator-configuration-datastore-uri-string,
> guix-build-coordinator-configuration-agent-communication-uri-string,
> guix-build-coordinator-configuration-client-communication-uri-string,
> guix-build-coordinator-configuration-allocation-strategy,
> guix-build-coordinator-configuration-hooks,
> guix-build-coordinator-configuration-guile,
> make-guix-build-coordinator-start-script,
> guix-build-coordinator-shepherd-services, guix-build-coordinator-activation,
> guix-build-coordinator-account): New procedures.
> (guix-build-coordinator-service-type): New variable.
> * gnu/tests/guix.scm (%test-guix-build-coordinator): New variable.
> * doc/guix.texi (Guix Services): Document it.
Yay!
> +@subsubheading Guix Build Coordinator
> +The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build
> +Coordinator} aids in building derivations. The Guix Daemon is still
^
Maybe something like: “in distributing derivation builds among machines
running an @dfn{agent}”.
Also, s/Guix Daemon/build daemon/ or similar.
(In general I’m in favor of avoiding “brands” in documentation and
concept names.)
Maybe add a “@quotation Note” stating that it’s work in progress and
subject to change.
> +(define* (make-guix-build-coordinator-start-script database-uri-string
> + allocation-strategy
> + pid-file
> + guix-build-coordinator-package
> + #:key
> + agent-communication-uri-string
> + client-communication-uri-string
> + (hooks '())
> + (guile guile-3.0))
> + (program-file
> + "start-guix-build-coordinator"
> + (with-extensions (cons guix-build-coordinator-package
> + ;; This is a poorly constructed Guile load path,
> + ;; since it contains things that aren't Guile
> + ;; libraries, but it means that the Guile libraries
> + ;; needed for the Guix Build Coordinator don't need
> + ;; to be individually specified here.
> + (map second (package-inputs
> + guix-build-coordinator-package)))
Perhaps there should eventually be a ‘guix-build-coordinator’ command in
the package itself?
Otherwise LGTM, thanks!
One thing we discussed on IRC is the name and ways to set up one’s
monitor so that
guix-build-coordinator-configuration-client-communication-uri-string
fits on one line. :-)
An idea that came to mind was “(Guix) dispatch(er)”, which is
synonymous. You could use the (guix dispatch …) name space.
Food for thought! (No rush, of course!)
Ludo’.
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type.
2020-09-18 18:40 ` [bug#43494] [PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type Christopher Baines
@ 2020-09-18 20:25 ` Ludovic Courtès
2020-09-19 9:49 ` Christopher Baines
0 siblings, 1 reply; 23+ messages in thread
From: Ludovic Courtès @ 2020-09-18 20:25 UTC (permalink / raw)
To: Christopher Baines; +Cc: 43494
Christopher Baines <mail@cbaines.net> skribis:
> * gnu/services/guix.scm (<guix-build-coordinator-agent-configuration>): New
> record type.
> (guix-build-coordinator-agent-configuration,
> guix-build-coordinator-agent-configuration?,
> guix-build-coordinator-agent-configuration-package,
> guix-build-coordinator-agent-configuration-user,
> guix-build-coordinator-agent-configuration-coordinator,
> guix-build-coordinator-agent-configuration-uuid),
> guix-build-coordinator-agent-configuration-password,
> guix-build-coordinator-agent-configuration-password-file,
> guix-build-coordinator-agent-configuration-systems,
> guix-build-coordinator-agent-configuration-max-parallel-builds,
> guix-build-coordinator-agent-configuration-derivation-substitute-urls,
> guix-build-coordinator-agent-configuration-non-derivation-substitute-urls,
> guix-build-coordinator-agent-shepherd-services,
> guix-build-coordinator-agent-activation,
> guix-build-coordinator-agent-account): New procedures.
> (guix-build-coordinator-agent-service-type): New variable.
> * doc/guix.texi (Guix Services): Document it.
[...]
> +@defvar {Scheme Variable} guix-build-coordinator-agent-service-type
> +Service type for a Guix Build Coordinator agent. Its value must be a
Perhaps “coordinator” and “agent” should be defined in a few sentences
above to clarify what this is about.
> +@item @code{derivation-substitute-urls} (default: @code{1})
> +URLs from which to attempt to fetch substitutes for derivations, if the
> +derivations aren't already available.
> +
> +@item @code{non-derivation-substitute-urls} (default: @code{1})
> +URLs from which to attempt to fetch substitutes for build inputs, if the
> +input store items aren't already available.
This is interesting, I wonder how you can distinguish between the two in
code. You have to open different sessions, right? Or to call
‘set-build-options’ again?
> + (description
> + "Run an instance of the Guix Build Coordinator.")))
+ “agent”
Otherwise LGTM!
Ludo’.
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
2020-09-18 18:40 ` [bug#43494] [PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type Christopher Baines
@ 2020-09-18 20:31 ` Ludovic Courtès
2020-09-19 10:05 ` Christopher Baines
0 siblings, 1 reply; 23+ messages in thread
From: Ludovic Courtès @ 2020-09-18 20:31 UTC (permalink / raw)
To: Christopher Baines; +Cc: 43494
Christopher Baines <mail@cbaines.net> skribis:
> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):
> New record type.
> (guix-build-coordinator-queue-builds-configuration,
> guix-build-coordinator-queue-builds-configuration?,
> guix-build-coordinator-queue-builds-configuration-package,
> guix-build-coordinator-queue-builds-configuration-user,
> guix-build-coordinator-queue-builds-coordinator,
> guix-build-coordinator-queue-builds-configuration-systems,
> guix-build-coordinator-queue-builds-configuration-system-and-targets,
> guix-build-coordinator-queue-builds-configuration-guix-data-service,
> guix-build-coordinator-queue-builds-configuration-processed-commits-file,
> guix-build-coordinator-queue-builds-shepherd-services,
> guix-build-coordinator-queue-builds-activation,
> guix-build-coordinator-queue-builds-account): New procedures.
> (guix-build-coordinator-queue-builds-service-type): New variable.
[...]
> +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type
> +Service type for the
> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its
Oh! :-)
‘guix-build-queue’ maybe? Because after all, it can queue builds coming
from anywhere, not just from the Data Service, right?
It’d be nice to add a sentence or two explaining how the “queue” fits in
the big picture (I’m not quite sure :-)).
> +@item @code{systems-and-targets} (default: @code{#f})
> +An association list of system and target pairs for which to fetch
> +derivations to build.
Cross-compilation target triplets are a notion that’s not available at
the derivation level; it only exists for packages.
Are we mixing things here?
> +@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"})
> +The Guix Data Service instance from which to query to find out about
> +derivations to build.
Ah so the queue pulls from the Data Service. Got it!
> +@item @code{processed-commits-file} (default: @code{"/var/lib/guix-build-coordinator-queue-builds/processed-commits"})
> +A file to record which commits have been processed, to avoid needlessly
> +processing them again if the service is restarted.
Maybe in /var/cache by default, no?
> + (processed-commits-file
> + guix-build-coordinator-queue-builds-configuration-processed-commits-file
> + (default "/var/lib/guix-build-coordinator-queue-builds/processed-commits")))
Maybe “state-file”? In addition to saving space :-), it’d leave room
for additional bits of state.
That’s all, thank you!
Ludo’.
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator.
2020-09-18 20:08 ` [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator Ludovic Courtès
@ 2020-09-18 23:34 ` Jonathan Brielmaier
2020-09-19 9:00 ` Christopher Baines
2020-09-19 9:05 ` Christopher Baines
1 sibling, 1 reply; 23+ messages in thread
From: Jonathan Brielmaier @ 2020-09-18 23:34 UTC (permalink / raw)
To: Ludovic Courtès, Christopher Baines; +Cc: 43494
On 18.09.20 22:08, Ludovic Courtès wrote:
>> + (home-page "https://git.cbaines.net/guile/guix/build-coordinator")
It should be https://git.cbaines.net/guix/build-coordinator/
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator.
2020-09-18 23:34 ` Jonathan Brielmaier
@ 2020-09-19 9:00 ` Christopher Baines
0 siblings, 0 replies; 23+ messages in thread
From: Christopher Baines @ 2020-09-19 9:00 UTC (permalink / raw)
To: Jonathan Brielmaier; +Cc: Ludovic Courtès, 43494
[-- Attachment #1: Type: text/plain, Size: 295 bytes --]
Jonathan Brielmaier <jonathan.brielmaier@web.de> writes:
> On 18.09.20 22:08, Ludovic Courtès wrote:
>>> + (home-page "https://git.cbaines.net/guile/guix/build-coordinator")
>
> It should be https://git.cbaines.net/guix/build-coordinator/
Thanks Jonathan, I've fixed this now :)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator.
2020-09-18 20:08 ` [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator Ludovic Courtès
2020-09-18 23:34 ` Jonathan Brielmaier
@ 2020-09-19 9:05 ` Christopher Baines
1 sibling, 0 replies; 23+ messages in thread
From: Christopher Baines @ 2020-09-19 9:05 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 43494
[-- Attachment #1: Type: text/plain, Size: 2797 bytes --]
Ludovic Courtès <ludo@gnu.org> writes:
> Hi!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> * gnu/packages/package-management.scm (guix-build-coordinator): New variable.
>
> [...]
>
>> + (lambda* (#:key inputs outputs #:allow-other-keys)
>> + (let* ((out (assoc-ref outputs "out"))
>> + (bin (string-append out "/bin"))
>> + (guile (assoc-ref inputs "guile"))
>> + (guile-effective-version
>> + (read-line
>> + (open-pipe* OPEN_READ
>> + (string-append guile "/bin/guile")
>> + "-c" "(display (effective-version))")))
>
> Maybe use ‘target-guile-effective-version’ from (guix build
> guile-build-system).
I've changed this. I'll also try and remember and update the
guix-data-service package accordingly, as that's where I copied this
from.
>> + (wrap-program (string-append bin "/" file)
>> + `("PATH" ":" prefix
>> + (,bin ,(assoc-ref inputs "sqitch")))
>> + `("PERL5LIB" ":" prefix
>> + (,(getenv "PERL5LIB")))
>
> Do we really need PERL5LIB here? Shouldn’t it be done in the wrapper of
> ‘sqitch’ directly?
Hmm, when I packaged sqitch, I didn't add all the database drivers which
are optional dependencies, but perl-dbd-pg did end up in there somehow
(probably because that's needed by the guix-data-service).
I've now gone ahead and added the database drivers which Guix has
packages for [1], it does increase the closure size a bit, but it was
quite big anyway.
1:
http://git.savannah.gnu.org/cgit/guix.git/commit/?id=6a1c3a908c8aadfde5b5fe405132839683a01776
All that means I can remove the wrapping here.
>> + (scandir bin
>> + (match-lambda
>> + ((or "." "..") #f)
>> + (_ #t))))
>
> I think you could just use (find-files bin).
Done.
>> + (inputs
>> + `(("guile" ,guile-3.0-latest)
>
> Rather ,@(assoc-ref (package-native-inputs guix) "guile").
Done.
>> + ("sqitch" ,sqitch)
>> + ("perl-dbd-sqlite" ,perl-dbd-sqlite)))
>
> Shouldn’t perl-dbd-sqlite be a dependency of sqitch?
See above.
>> + (home-page "https://git.cbaines.net/guile/guix/build-coordinator")
>> + (synopsis "")
>> + (description
>> + "")
>
> Missing! :-)
Ah, yeah, I should have remembered to lint the package. I've attempted
to write something now, I'm still working on trying to explain what the
Guix Build Coordinator is meant to be.
> LGTM with changes along these lines!
Thanks!
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
2020-09-18 20:20 ` Ludovic Courtès
@ 2020-09-19 9:13 ` Christopher Baines
2020-09-25 9:42 ` Ludovic Courtès
0 siblings, 1 reply; 23+ messages in thread
From: Christopher Baines @ 2020-09-19 9:13 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 43494
[-- Attachment #1: Type: text/plain, Size: 4260 bytes --]
Ludovic Courtès <ludo@gnu.org> writes:
> Christopher Baines <mail@cbaines.net> skribis:
>
>> * gnu/services/guix.scm (<guix-build-coordinator-configuration>): New record
>> type.
>> (guix-build-coordinator-configuration, guix-build-coordinator-configuration?,
>> guix-build-coordinator-configuration-package,
>> guix-build-coordinator-configuration-user,
>> guix-build-coordinator-configuration-group,
>> guix-build-coordinator-configuration-datastore-uri-string,
>> guix-build-coordinator-configuration-agent-communication-uri-string,
>> guix-build-coordinator-configuration-client-communication-uri-string,
>> guix-build-coordinator-configuration-allocation-strategy,
>> guix-build-coordinator-configuration-hooks,
>> guix-build-coordinator-configuration-guile,
>> make-guix-build-coordinator-start-script,
>> guix-build-coordinator-shepherd-services, guix-build-coordinator-activation,
>> guix-build-coordinator-account): New procedures.
>> (guix-build-coordinator-service-type): New variable.
>> * gnu/tests/guix.scm (%test-guix-build-coordinator): New variable.
>> * doc/guix.texi (Guix Services): Document it.
>
> Yay!
>
>> +@subsubheading Guix Build Coordinator
>> +The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build
>> +Coordinator} aids in building derivations. The Guix Daemon is still
> ^
> Maybe something like: “in distributing derivation builds among machines
> running an @dfn{agent}”.
>
> Also, s/Guix Daemon/build daemon/ or similar.
>
> (In general I’m in favor of avoiding “brands” in documentation and
> concept names.)
>
> Maybe add a “@quotation Note” stating that it’s work in progress and
> subject to change.
I've made these changes now.
>> +(define* (make-guix-build-coordinator-start-script database-uri-string
>> + allocation-strategy
>> + pid-file
>> + guix-build-coordinator-package
>> + #:key
>> + agent-communication-uri-string
>> + client-communication-uri-string
>> + (hooks '())
>> + (guile guile-3.0))
>> + (program-file
>> + "start-guix-build-coordinator"
>> + (with-extensions (cons guix-build-coordinator-package
>> + ;; This is a poorly constructed Guile load path,
>> + ;; since it contains things that aren't Guile
>> + ;; libraries, but it means that the Guile libraries
>> + ;; needed for the Guix Build Coordinator don't need
>> + ;; to be individually specified here.
>> + (map second (package-inputs
>> + guix-build-coordinator-package)))
>
> Perhaps there should eventually be a ‘guix-build-coordinator’ command in
> the package itself?
There actually is, one thing I've had in mind for a while now though is
to use a scheme script constructed by the Guix service to run the
coordinator.
For guix.cbaines.net, I'm using the script, but with the hooks passed in
on the command line, the command is rather long, and it means that
backtraces don't work well with the hooks.
I'm unsure how well this has worked out, I didn't anticipate the issues
with the Guile load path, and because of the formatting stripping, both
the code and backtraces will be unreadable anyway... but it is useful in
allowing G-expressions to be used as part of the configuration.
> One thing we discussed on IRC is the name and ways to set up one’s
> monitor so that
> guix-build-coordinator-configuration-client-communication-uri-string
> fits on one line. :-)
>
> An idea that came to mind was “(Guix) dispatch(er)”, which is
> synonymous. You could use the (guix dispatch …) name space.
> Food for thought! (No rush, of course!)
Indeed, I think that's a strong candidate :)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type.
2020-09-18 20:25 ` Ludovic Courtès
@ 2020-09-19 9:49 ` Christopher Baines
0 siblings, 0 replies; 23+ messages in thread
From: Christopher Baines @ 2020-09-19 9:49 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 43494
[-- Attachment #1: Type: text/plain, Size: 2946 bytes --]
Ludovic Courtès <ludo@gnu.org> writes:
> Christopher Baines <mail@cbaines.net> skribis:
>
>> * gnu/services/guix.scm (<guix-build-coordinator-agent-configuration>): New
>> record type.
>> (guix-build-coordinator-agent-configuration,
>> guix-build-coordinator-agent-configuration?,
>> guix-build-coordinator-agent-configuration-package,
>> guix-build-coordinator-agent-configuration-user,
>> guix-build-coordinator-agent-configuration-coordinator,
>> guix-build-coordinator-agent-configuration-uuid),
>> guix-build-coordinator-agent-configuration-password,
>> guix-build-coordinator-agent-configuration-password-file,
>> guix-build-coordinator-agent-configuration-systems,
>> guix-build-coordinator-agent-configuration-max-parallel-builds,
>> guix-build-coordinator-agent-configuration-derivation-substitute-urls,
>> guix-build-coordinator-agent-configuration-non-derivation-substitute-urls,
>> guix-build-coordinator-agent-shepherd-services,
>> guix-build-coordinator-agent-activation,
>> guix-build-coordinator-agent-account): New procedures.
>> (guix-build-coordinator-agent-service-type): New variable.
>> * doc/guix.texi (Guix Services): Document it.
>
> [...]
>
>> +@defvar {Scheme Variable} guix-build-coordinator-agent-service-type
>> +Service type for a Guix Build Coordinator agent. Its value must be a
>
> Perhaps “coordinator” and “agent” should be defined in a few sentences
> above to clarify what this is about.
I've added a bit more information, I want to at some point actually give
a workable minimal example configuration, but I need to do some more
thinking about that.
>> +@item @code{derivation-substitute-urls} (default: @code{1})
>> +URLs from which to attempt to fetch substitutes for derivations, if the
>> +derivations aren't already available.
>> +
>> +@item @code{non-derivation-substitute-urls} (default: @code{1})
>> +URLs from which to attempt to fetch substitutes for build inputs, if the
>> +input store items aren't already available.
>
> This is interesting, I wonder how you can distinguish between the two in
> code. You have to open different sessions, right? Or to call
> ‘set-build-options’ again?
Yeah, it's not strictly necessary to have some separation. In fact, I
can't remember exactly, but I think it should be possible to deploy the
coordinator and agents without explicit configuration of the substitute
URLs, but I need to check that.
These options were added mostly because it useful if you have two
seperate sources of substitutes, one for just derivations (say an
instance of the Guix Data Service), and one just for build outputs (say
populated by one of the hooks) [1].
1: https://git.cbaines.net/guix/build-coordinator/commit/?id=bda7d58853ed4fba976cac92a70c4dc68db263aa
>> + (description
>> + "Run an instance of the Guix Build Coordinator.")))
>
> + “agent”
>
> Otherwise LGTM!
Thanks!
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
2020-09-18 20:31 ` Ludovic Courtès
@ 2020-09-19 10:05 ` Christopher Baines
0 siblings, 0 replies; 23+ messages in thread
From: Christopher Baines @ 2020-09-19 10:05 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 43494
[-- Attachment #1: Type: text/plain, Size: 4083 bytes --]
Ludovic Courtès <ludo@gnu.org> writes:
> Christopher Baines <mail@cbaines.net> skribis:
>
>> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):
>> New record type.
>> (guix-build-coordinator-queue-builds-configuration,
>> guix-build-coordinator-queue-builds-configuration?,
>> guix-build-coordinator-queue-builds-configuration-package,
>> guix-build-coordinator-queue-builds-configuration-user,
>> guix-build-coordinator-queue-builds-coordinator,
>> guix-build-coordinator-queue-builds-configuration-systems,
>> guix-build-coordinator-queue-builds-configuration-system-and-targets,
>> guix-build-coordinator-queue-builds-configuration-guix-data-service,
>> guix-build-coordinator-queue-builds-configuration-processed-commits-file,
>> guix-build-coordinator-queue-builds-shepherd-services,
>> guix-build-coordinator-queue-builds-activation,
>> guix-build-coordinator-queue-builds-account): New procedures.
>> (guix-build-coordinator-queue-builds-service-type): New variable.
>
> [...]
>
>> +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type
>> +Service type for the
>> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its
>
> Oh! :-)
>
> ‘guix-build-queue’ maybe? Because after all, it can queue builds coming
> from anywhere, not just from the Data Service, right?
>
> It’d be nice to add a sentence or two explaining how the “queue” fits in
> the big picture (I’m not quite sure :-)).
So, this got me thinking a bit, maybe this script should actually exist
outside of the guix-build-coordinator repository...
You can run `guix-build-coordinator build ...` to build a derivation,
and what this does is just make the relevant HTTP request to the
coordinator.
What this script does is just fetch derivation names from an instance of
the Guix Data Service, and ask the Guix Build Coordinator to build
them.
>> +@item @code{systems-and-targets} (default: @code{#f})
>> +An association list of system and target pairs for which to fetch
>> +derivations to build.
>
> Cross-compilation target triplets are a notion that’s not available at
> the derivation level; it only exists for packages.
>
> Are we mixing things here?
This setting is basically configuing the query parameters to use for
this page for example [1]. You're right that it's not available at the
derivation level, but you can ask the Guix Data Service for derivations
for packages which were computed for a specific system and target.
1: http://data.guix.gnu.org/revision/70ef8b24550c54cc8e9f20026bfd24b8680499b4/package-derivations
>> +@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"})
>> +The Guix Data Service instance from which to query to find out about
>> +derivations to build.
>
> Ah so the queue pulls from the Data Service. Got it!
I wouldn't read too much in to "queue" in the name here, "submit" might
be a better word to use, since that's the term used on the Guix Build
Coordinator side. There's also nothing resemling a queue anywhere...
>> +@item @code{processed-commits-file} (default: @code{"/var/lib/guix-build-coordinator-queue-builds/processed-commits"})
>> +A file to record which commits have been processed, to avoid needlessly
>> +processing them again if the service is restarted.
>
> Maybe in /var/cache by default, no?
Ah, yeah, I've changed it to be in /var/cache, as that's fine.
>> + (processed-commits-file
>> + guix-build-coordinator-queue-builds-configuration-processed-commits-file
>> + (default "/var/lib/guix-build-coordinator-queue-builds/processed-commits")))
>
> Maybe “state-file”? In addition to saving space :-), it’d leave room
> for additional bits of state.
Currently, it's just a file with each commit on its own line. Given
that, if there's a need to store more state, it might just end up in
different files. If that happens, this setting could become a directory,
rather than a specific file to avoid adding extra configuration options.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH v2 1/4] gnu: Add guix-build-coordinator.
2020-09-18 18:34 [bug#43494] [PATCH 0/4] Add package and services for the Guix Build Coordinator Christopher Baines
2020-09-18 18:40 ` [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator Christopher Baines
@ 2020-09-19 10:10 ` Christopher Baines
2020-09-19 10:10 ` [bug#43494] [PATCH v2 2/4] services: guix: Add guix-build-coordinator-service-type Christopher Baines
` (2 more replies)
1 sibling, 3 replies; 23+ messages in thread
From: Christopher Baines @ 2020-09-19 10:10 UTC (permalink / raw)
To: 43494
* gnu/packages/package-management.scm (guix-build-coordinator): New variable.
---
gnu/packages/package-management.scm | 74 +++++++++++++++++++++++++++++
1 file changed, 74 insertions(+)
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index ec87226197..afa45d6789 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -50,6 +50,7 @@
#:use-module (gnu packages cpio)
#:use-module (gnu packages crypto)
#:use-module (gnu packages curl)
+ #:use-module (gnu packages databases)
#:use-module (gnu packages dbm)
#:use-module (gnu packages docbook)
#:use-module (gnu packages file)
@@ -981,6 +982,79 @@ environments.")
;; and the fonts included in this package are licensed OFL1.1.
(license (list license:gpl3+ license:agpl3+ license:silofl1.1))))
+(define-public guix-build-coordinator
+ (let ((commit "5e8408c833e209efddfa0159114b90400c1aaf4d")
+ (revision "0"))
+ (package
+ (name "guix-build-coordinator")
+ (version (git-version "0" revision commit))
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://git.cbaines.net/git/guix/build-coordinator")
+ (commit commit)))
+ (sha256
+ (base32
+ "0f7m1zg9mlb2m22qyblglaa36h8f49b810jc9j5b0hsb42ijwh4b"))
+ (file-name (string-append name "-" version "-checkout"))))
+ (build-system gnu-build-system)
+ (arguments
+ `(#:modules (((guix build guile-build-system)
+ #:select (target-guile-effective-version))
+ ,@%gnu-build-system-modules)
+ #:imported-modules ((guix build guile-build-system)
+ ,@%gnu-build-system-modules)
+ #:phases
+ (modify-phases %standard-phases
+ (add-before 'build 'set-GUILE_AUTO_COMPILE
+ (lambda _
+ ;; To avoid warnings relating to 'guild'.
+ (setenv "GUILE_AUTO_COMPILE" "0")
+ #t))
+ (add-after 'install 'wrap-executable
+ (lambda* (#:key inputs outputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (bin (string-append out "/bin"))
+ (guile (assoc-ref inputs "guile"))
+ (version (target-guile-effective-version))
+ (scm (string-append out "/share/guile/site/" version))
+ (go (string-append out "/lib/guile/" version "/site-ccache")))
+ (for-each
+ (lambda (file)
+ (simple-format (current-error-port) "wrapping: ~A\n" file)
+ (wrap-program file
+ `("PATH" ":" prefix
+ (,bin ,(assoc-ref inputs "sqitch")))
+ `("GUILE_LOAD_PATH" ":" prefix
+ (,scm ,(getenv "GUILE_LOAD_PATH")))
+ `("GUILE_LOAD_COMPILED_PATH" ":" prefix
+ (,go ,(getenv "GUILE_LOAD_COMPILED_PATH")))))
+ (find-files bin)))
+ #t))
+ (delete 'strip)))) ; As the .go files aren't compatible
+ (native-inputs
+ `(("pkg-config" ,pkg-config)
+ ("autoconf" ,autoconf)
+ ("automake" ,automake)))
+ (inputs
+ `(("guile" ,@(assoc-ref (package-native-inputs guix) "guile"))
+ ("guile-fibers" ,guile-fibers)
+ ("guile-prometheus" ,guile-prometheus)
+ ("guile-gcrypt" ,guile-gcrypt)
+ ("guile-json" ,guile-json-3)
+ ("guile-lzlib" ,guile-lzlib)
+ ("guile-sqlite3" ,guile-sqlite3)
+ ("guix" ,guix)
+ ("sqlite" ,sqlite)
+ ("sqitch" ,sqitch)))
+ (home-page "https://git.cbaines.net/guix/build-coordinator/")
+ (synopsis "Tool to help build derivations")
+ (description
+ "The Guix Build Coordinator helps with performing lots of builds across
+potentially many machines, and with doing something with the results and
+outputs of those builds.")
+ (license license:gpl3+))))
+
(define-public guix-jupyter
(package
(name "guix-jupyter")
--
2.28.0
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH v2 2/4] services: guix: Add guix-build-coordinator-service-type.
2020-09-19 10:10 ` [bug#43494] [PATCH v2 " Christopher Baines
@ 2020-09-19 10:10 ` Christopher Baines
2020-09-19 10:10 ` [bug#43494] [PATCH v2 3/4] services: guix: Add guix-build-coordinator-agent-service-type Christopher Baines
2020-09-19 10:10 ` [bug#43494] [PATCH v2 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type Christopher Baines
2 siblings, 0 replies; 23+ messages in thread
From: Christopher Baines @ 2020-09-19 10:10 UTC (permalink / raw)
To: 43494
* gnu/services/guix.scm (<guix-build-coordinator-configuration>): New record
type.
(guix-build-coordinator-configuration, guix-build-coordinator-configuration?,
guix-build-coordinator-configuration-package,
guix-build-coordinator-configuration-user,
guix-build-coordinator-configuration-group,
guix-build-coordinator-configuration-datastore-uri-string,
guix-build-coordinator-configuration-agent-communication-uri-string,
guix-build-coordinator-configuration-client-communication-uri-string,
guix-build-coordinator-configuration-allocation-strategy,
guix-build-coordinator-configuration-hooks,
guix-build-coordinator-configuration-guile,
make-guix-build-coordinator-start-script,
guix-build-coordinator-shepherd-services, guix-build-coordinator-activation,
guix-build-coordinator-account): New procedures.
(guix-build-coordinator-service-type): New variable.
* gnu/tests/guix.scm (%test-guix-build-coordinator): New variable.
* doc/guix.texi (Guix Services): Document it.
---
doc/guix.texi | 67 ++++++++++++++
gnu/services/guix.scm | 200 +++++++++++++++++++++++++++++++++++++++++-
gnu/tests/guix.scm | 75 +++++++++++++++-
3 files changed, 338 insertions(+), 4 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index f7e2204b53..4f3f0e851f 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -27487,6 +27487,73 @@ The complete list of possible options can be found in the man page for
@node Guix Services
@subsection Guix Services
+@subsubheading Guix Build Coordinator
+The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build
+Coordinator} aids in distributing derivation builds among machines
+running an @dfn{agent}. The build daemon is still used to build the
+derivations, but the Guix Build Coordinator manages allocating builds
+and working with the results.
+
+@quotation Note
+This service is considered experimental. Configuration options may be
+changed in a backwards-incompatible manner, and not all features have
+been thorougly tested.
+@end quotation
+
+The Guix Build Coordinator consists of one @dfn{coordinator}, and one or
+more connected @dfn{agent} processes. The coordinator process handles
+clients submitting builds, and allocating builds to agents. The agent
+processes talk to a build daemon to actually perform the builds, then
+send the results back to the coordinator.
+
+There is a script to run the coordinator component of the Guix Build
+Coordinator, but the Guix service uses a custom Guile script instead, to
+provide better integration with G-expressions used in the configuration.
+
+@defvar {Scheme Variable} guix-build-coordinator-service-type
+Service type for the Guix Build Coordinator. Its value must be a
+@code{guix-build-coordinator-configuration} object.
+@end defvar
+
+@deftp {Data Type} guix-build-coordinator-configuration
+Data type representing the configuration of the Guix Build Coordinator.
+
+@table @asis
+@item @code{package} (default: @code{guix-build-coordinator})
+The Guix Build Coordinator package to use.
+
+@item @code{user} (default: @code{"guix-build-coordinator"})
+The system user to run the service as.
+
+@item @code{group} (default: @code{"guix-build-coordinator"})
+The system group to run the service as.
+
+@item @code{database-uri-string} (default: @code{"sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"})
+The URI to use for the database.
+
+@item @code{agent-communication-uri} (default: @code{"http://0.0.0.0:8745"})
+The URI describing how to listen to requests from agent processes.
+
+@item @code{client-communication-uri} (default: @code{"http://127.0.0.1:8746"})
+The URI describing how to listen to requests from clients. The client
+API allows submitting builds and currently isn't authenticated, so take
+care when configuring this value.
+
+@item @code{allocation-strategy} (default: @code{#~basic-build-allocation-strategy})
+A G-expression for the allocation strategy to be used. This is a
+procedure that takes the datastore as an argument and populates the
+allocation plan in the database.
+
+@item @code{hooks} (default: @var{'()})
+An association list of hooks. These provide a way to execute arbitrary
+code upon certian events, like a build result being processed.
+
+@item @code{guile} (default: @code{guile-3.0-latest})
+The Guile package with which to run the Guix Build Coordinator.
+
+@end table
+@end deftp
+
@subsubheading Guix Data Service
The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores
and provides data about GNU Guix. This includes information about
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index 10a8581a62..1bacd61190 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -17,20 +17,40 @@
;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
(define-module (gnu services guix)
+ #:use-module (srfi srfi-1)
#:use-module (ice-9 match)
#:use-module (guix gexp)
#:use-module (guix records)
+ #:use-module (guix packages)
#:use-module ((gnu packages base)
#:select (glibc-utf8-locales))
#:use-module (gnu packages admin)
+ #:use-module (gnu packages databases)
#:use-module (gnu packages web)
+ #:use-module (gnu packages guile)
+ #:use-module (gnu packages guile-xyz)
+ #:use-module (gnu packages package-management)
#:use-module (gnu services)
#:use-module (gnu services base)
#:use-module (gnu services admin)
#:use-module (gnu services shepherd)
#:use-module (gnu services getmail)
#:use-module (gnu system shadow)
- #:export (<guix-data-service-configuration>
+ #:export (guix-build-coordinator-configuration
+ guix-build-coordinator-configuration?
+ guix-build-coordinator-configuration-package
+ guix-build-coordinator-configuration-user
+ guix-build-coordinator-configuration-group
+ guix-build-coordinator-configuration-datastore-uri-string
+ guix-build-coordinator-configuration-agent-communication-uri-string
+ guix-build-coordinator-configuration-client-communication-uri-string
+ guix-build-coordinator-configuration-allocation-strategy
+ guix-build-coordinator-configuration-hooks
+ guix-build-coordinator-configuration-guile
+
+ guix-build-coordinator-service-type
+
+ <guix-data-service-configuration>
guix-data-service-configuration
guix-data-service-configuration?
guix-data-service-package
@@ -45,11 +65,185 @@
;;;; Commentary:
;;;
-;;; This module implements a service that to run instances of the Guix Data
-;;; Service, which provides data about Guix over time.
+;;; Services specifically related to GNU Guix.
;;;
;;;; Code:
+(define-record-type* <guix-build-coordinator-configuration>
+ guix-build-coordinator-configuration make-guix-build-coordinator-configuration
+ guix-build-coordinator-configuration?
+ (package guix-build-coordinator-configuration-package
+ (default guix-build-coordinator))
+ (user guix-build-coordinator-configuration-user
+ (default "guix-build-coordinator"))
+ (group guix-build-coordinator-configuration-group
+ (default "guix-build-coordinator"))
+ (database-uri-string
+ guix-build-coordinator-configuration-datastore-uri-string
+ (default "sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"))
+ (agent-communication-uri-string
+ guix-build-coordinator-configuration-agent-communication-uri-string
+ (default "http://0.0.0.0:8745"))
+ (client-communication-uri-string
+ guix-build-coordinator-configuration-client-communication-uri-string
+ (default "http://127.0.0.1:8746"))
+ (allocation-strategy
+ guix-build-coordinator-configuration-allocation-strategy
+ (default #~basic-build-allocation-strategy))
+ (hooks guix-build-coordinator-configuration-hooks
+ (default '()))
+ (guile guix-build-coordinator-configuration-guile
+ (default guile-3.0-latest)))
+
+(define* (make-guix-build-coordinator-start-script database-uri-string
+ allocation-strategy
+ pid-file
+ guix-build-coordinator-package
+ #:key
+ agent-communication-uri-string
+ client-communication-uri-string
+ (hooks '())
+ (guile guile-3.0))
+ (program-file
+ "start-guix-build-coordinator"
+ (with-extensions (cons guix-build-coordinator-package
+ ;; This is a poorly constructed Guile load path,
+ ;; since it contains things that aren't Guile
+ ;; libraries, but it means that the Guile libraries
+ ;; needed for the Guix Build Coordinator don't need
+ ;; to be individually specified here.
+ (map second (package-inputs
+ guix-build-coordinator-package)))
+ #~(begin
+ (use-modules (srfi srfi-1)
+ (ice-9 match)
+ (web uri)
+ (prometheus)
+ (guix-build-coordinator hooks)
+ (guix-build-coordinator datastore)
+ (guix-build-coordinator build-allocator)
+ (guix-build-coordinator coordinator))
+
+ (let* ((metrics-registry (make-metrics-registry
+ #:namespace
+ "guixbuildcoordinator_"))
+ (datastore (database-uri->datastore
+ #$database-uri-string
+ #:metrics-registry metrics-registry))
+ (hooks
+ (list #$@(map (match-lambda
+ ((name . hook-gexp)
+ #~(cons name #$hook-gexp)))
+ hooks)))
+ (hooks-with-defaults
+ `(,@hooks
+ ,@(remove (match-lambda
+ ((name . _) (assq-ref hooks name)))
+ %default-hooks)))
+ (build-coordinator (make-build-coordinator
+ #:datastore datastore
+ #:hooks hooks-with-defaults
+ #:metrics-registry metrics-registry
+ #:allocation-strategy #$allocation-strategy)))
+
+ (run-coordinator-service
+ build-coordinator
+ #:update-datastore? #t
+ #:pid-file #$pid-file
+ #:agent-communication-uri (string->uri
+ #$agent-communication-uri-string)
+ #:client-communication-uri (string->uri
+ #$client-communication-uri-string)))))
+ #:guile guile))
+
+(define (guix-build-coordinator-shepherd-services config)
+ (match-record config <guix-build-coordinator-configuration>
+ (package user group database-uri-string
+ agent-communication-uri-string
+ client-communication-uri-string
+ allocation-strategy
+ hooks
+ guile)
+ (list
+ (shepherd-service
+ (documentation "Guix Build Coordinator")
+ (provision '(guix-build-coordinator))
+ (requirement '(networking))
+ (start #~(make-forkexec-constructor
+ (list #$(make-guix-build-coordinator-start-script
+ database-uri-string
+ allocation-strategy
+ "/var/run/guix-build-coordinator/pid"
+ package
+ #:agent-communication-uri-string
+ agent-communication-uri-string
+ #:client-communication-uri-string
+ client-communication-uri-string
+ #:hooks hooks
+ #:guile guile))
+ #:user #$user
+ #:group #$group
+ #:pid-file "/var/run/guix-build-coordinator/pid"
+ ;; Allow time for migrations to run
+ #:pid-file-timeout 60
+ #:environment-variables
+ `(,(string-append
+ "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+ "LC_ALL=en_US.utf8")
+ #:log-file "/var/log/guix-build-coordinator/coordinator.log"))
+ (stop #~(make-kill-destructor))))))
+
+(define (guix-build-coordinator-activation config)
+ #~(begin
+ (use-modules (guix build utils))
+
+ (define %user (getpw "guix-build-coordinator"))
+
+ (chmod "/var/lib/guix-build-coordinator" #o755)
+
+ (mkdir-p "/var/log/guix-build-coordinator")
+
+ ;; Allow writing the PID file
+ (mkdir-p "/var/run/guix-build-coordinator")
+ (chown "/var/run/guix-build-coordinator"
+ (passwd:uid %user)
+ (passwd:gid %user))))
+
+(define (guix-build-coordinator-account config)
+ (match-record config <guix-build-coordinator-configuration>
+ (user group)
+ (list (user-group
+ (name group)
+ (system? #t))
+ (user-account
+ (name user)
+ (group group)
+ (system? #t)
+ (comment "Guix Build Coordinator user")
+ (home-directory "/var/lib/guix-build-coordinator")
+ (shell (file-append shadow "/sbin/nologin"))))))
+
+(define guix-build-coordinator-service-type
+ (service-type
+ (name 'guix-build-coordinator)
+ (extensions
+ (list
+ (service-extension shepherd-root-service-type
+ guix-build-coordinator-shepherd-services)
+ (service-extension activation-service-type
+ guix-build-coordinator-activation)
+ (service-extension account-service-type
+ guix-build-coordinator-account)))
+ (default-value
+ (guix-build-coordinator-configuration))
+ (description
+ "Run an instance of the Guix Build Coordinator.")))
+
+\f
+;;;
+;;; Guix Data Service
+;;;
+
(define-record-type* <guix-data-service-configuration>
guix-data-service-configuration make-guix-data-service-configuration
guix-data-service-configuration?
diff --git a/gnu/tests/guix.scm b/gnu/tests/guix.scm
index 6139e31cf0..20b67d55d3 100644
--- a/gnu/tests/guix.scm
+++ b/gnu/tests/guix.scm
@@ -35,7 +35,80 @@
#:use-module (guix store)
#:use-module (guix utils)
#:use-module (ice-9 match)
- #:export (%test-guix-data-service))
+ #:export (%test-guix-build-coordinator
+ %test-guix-data-service))
+
+;;;
+;;; Guix Build Coordinator
+;;;
+
+(define %guix-build-coordinator-os
+ (simple-operating-system
+ (service dhcp-client-service-type)
+ (service guix-build-coordinator-service-type)))
+
+(define (run-guix-build-coordinator-test)
+ (define os
+ (marionette-operating-system
+ %guix-build-coordinator-os
+ #:imported-modules '((gnu services herd)
+ (guix combinators))))
+
+ (define forwarded-port 8745)
+
+ (define vm
+ (virtual-machine
+ (operating-system os)
+ (memory-size 1024)
+ (port-forwardings `((,forwarded-port . 8745)))))
+
+ (define test
+ (with-imported-modules '((gnu build marionette))
+ #~(begin
+ (use-modules (srfi srfi-11) (srfi srfi-64)
+ (gnu build marionette)
+ (web uri)
+ (web client)
+ (web response))
+
+ (define marionette
+ (make-marionette (list #$vm)))
+
+ (mkdir #$output)
+ (chdir #$output)
+
+ (test-begin "guix-build-coordinator")
+
+ (test-assert "service running"
+ (marionette-eval
+ '(begin
+ (use-modules (gnu services herd))
+ (match (start-service 'guix-build-coordinator)
+ (#f #f)
+ (('service response-parts ...)
+ (match (assq-ref response-parts 'running)
+ ((pid) (number? pid))))))
+ marionette))
+
+ (test-equal "http-get"
+ 200
+ (let-values
+ (((response text)
+ (http-get #$(simple-format
+ #f "http://localhost:~A/metrics" forwarded-port)
+ #:decode-body? #t)))
+ (response-code response)))
+
+ (test-end)
+ (exit (= (test-runner-fail-count (test-runner-current)) 0)))))
+
+ (gexp->derivation "guix-build-coordinator-test" test))
+
+(define %test-guix-build-coordinator
+ (system-test
+ (name "guix-build-coordinator")
+ (description "Connect to a running Guix Build Coordinator.")
+ (value (run-guix-build-coordinator-test))))
\f
;;;
--
2.28.0
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH v2 3/4] services: guix: Add guix-build-coordinator-agent-service-type.
2020-09-19 10:10 ` [bug#43494] [PATCH v2 " Christopher Baines
2020-09-19 10:10 ` [bug#43494] [PATCH v2 2/4] services: guix: Add guix-build-coordinator-service-type Christopher Baines
@ 2020-09-19 10:10 ` Christopher Baines
2020-09-19 10:10 ` [bug#43494] [PATCH v2 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type Christopher Baines
2 siblings, 0 replies; 23+ messages in thread
From: Christopher Baines @ 2020-09-19 10:10 UTC (permalink / raw)
To: 43494
* gnu/services/guix.scm (<guix-build-coordinator-agent-configuration>): New
record type.
(guix-build-coordinator-agent-configuration,
guix-build-coordinator-agent-configuration?,
guix-build-coordinator-agent-configuration-package,
guix-build-coordinator-agent-configuration-user,
guix-build-coordinator-agent-configuration-coordinator,
guix-build-coordinator-agent-configuration-uuid),
guix-build-coordinator-agent-configuration-password,
guix-build-coordinator-agent-configuration-password-file,
guix-build-coordinator-agent-configuration-systems,
guix-build-coordinator-agent-configuration-max-parallel-builds,
guix-build-coordinator-agent-configuration-derivation-substitute-urls,
guix-build-coordinator-agent-configuration-non-derivation-substitute-urls,
guix-build-coordinator-agent-shepherd-services,
guix-build-coordinator-agent-activation,
guix-build-coordinator-agent-account): New procedures.
(guix-build-coordinator-agent-service-type): New variable.
* doc/guix.texi (Guix Services): Document it.
---
doc/guix.texi | 50 +++++++++++++++++
gnu/services/guix.scm | 121 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 171 insertions(+)
diff --git a/doc/guix.texi b/doc/guix.texi
index 4f3f0e851f..b2dde30ddb 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -27554,6 +27554,56 @@ The Guile package with which to run the Guix Build Coordinator.
@end table
@end deftp
+@defvar {Scheme Variable} guix-build-coordinator-agent-service-type
+Service type for a Guix Build Coordinator agent. Its value must be a
+@code{guix-build-coordinator-agent-configuration} object.
+@end defvar
+
+@deftp {Data Type} guix-build-coordinator-agent-configuration
+Data type representing the configuration a Guix Build Coordinator agent.
+
+@table @asis
+@item @code{package} (default: @code{guix-build-coordinator})
+The Guix Build Coordinator package to use.
+
+@item @code{user} (default: @code{"guix-build-coordinator-agent"})
+The system user to run the service as.
+
+@item @code{coordinator} (default: @code{"http://localhost:8745"})
+The URI to use when connecting to the coordinator.
+
+@item @code{uuid}
+The UUID of the agent. This should be generated by the coordinator
+process, stored in the coordinator database, and used by the intended
+agent.
+
+@item @code{password} (default: @code{#f})
+The password to use when connecting to the coordinator. A file to read
+the password from can also be specified, and this is more secure.
+
+@item @code{password-file} (default: @code{#f})
+A file containing the password to use when connecting to the
+coordinator.
+
+@item @code{systems} (default: @var{#f})
+The systems for which this agent should fetch builds. The agent process
+will use the current system it's running on as the default.
+
+@item @code{max-parallel-builds} (default: @code{1})
+The number of builds to perform in parallel.
+
+@item @code{derivation-substitute-urls} (default: @code{1})
+URLs from which to attempt to fetch substitutes for derivations, if the
+derivations aren't already available.
+
+@item @code{non-derivation-substitute-urls} (default: @code{1})
+URLs from which to attempt to fetch substitutes for build inputs, if the
+input store items aren't already available.
+
+@end table
+@end deftp
+
+
@subsubheading Guix Data Service
The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores
and provides data about GNU Guix. This includes information about
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index 1bacd61190..e4f5cf57b9 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -50,6 +50,21 @@
guix-build-coordinator-service-type
+ guix-build-coordinator-agent-configuration
+ guix-build-coordinator-agent-configuration?
+ guix-build-coordinator-agent-configuration-package
+ guix-build-coordinator-agent-configuration-user
+ guix-build-coordinator-agent-configuration-coordinator
+ guix-build-coordinator-agent-configuration-uuid
+ guix-build-coordinator-agent-configuration-password
+ guix-build-coordinator-agent-configuration-password-file
+ guix-build-coordinator-agent-configuration-systems
+ guix-build-coordinator-agent-configuration-max-parallel-builds
+ guix-build-coordinator-agent-configuration-derivation-substitute-urls
+ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls
+
+ guix-build-coordinator-agent-service-type
+
<guix-data-service-configuration>
guix-data-service-configuration
guix-data-service-configuration?
@@ -95,6 +110,33 @@
(guile guix-build-coordinator-configuration-guile
(default guile-3.0-latest)))
+(define-record-type* <guix-build-coordinator-agent-configuration>
+ guix-build-coordinator-agent-configuration
+ make-guix-build-coordinator-agent-configuration
+ guix-build-coordinator-agent-configuration?
+ (package guix-build-coordinator-agent-configuration-package
+ (default guix-build-coordinator))
+ (user guix-build-coordinator-agent-configuration-user
+ (default "guix-build-coordinator-agent"))
+ (coordinator guix-build-coordinator-agent-configuration-coordinator
+ (default "http://localhost:8745"))
+ (uuid guix-build-coordinator-agent-configuration-uuid)
+ (password guix-build-coordinator-agent-configuration-password
+ (default #f))
+ (password-file guix-build-coordinator-agent-configuration-password-file
+ (default #f))
+ (systems guix-build-coordinator-agent-configuration-systems
+ (default #f))
+ (max-parallel-builds
+ guix-build-coordinator-agent-configuration-max-parallel-builds
+ (default 1))
+ (derivation-substitute-urls
+ guix-build-coordinator-agent-configuration-derivation-substitute-urls
+ (default #f))
+ (non-derivation-substitute-urls
+ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls
+ (default #f)))
+
(define* (make-guix-build-coordinator-start-script database-uri-string
allocation-strategy
pid-file
@@ -239,6 +281,85 @@
(description
"Run an instance of the Guix Build Coordinator.")))
+(define (guix-build-coordinator-agent-shepherd-services config)
+ (match-record config <guix-build-coordinator-agent-configuration>
+ (package user coordinator uuid password password-file max-parallel-builds
+ derivation-substitute-urls non-derivation-substitute-urls
+ systems)
+ (list
+ (shepherd-service
+ (documentation "Guix Build Coordinator Agent")
+ (provision '(guix-build-coordinator-agent))
+ (requirement '(networking))
+ (start #~(make-forkexec-constructor
+ (list #$(file-append package "/bin/guix-build-coordinator-agent")
+ #$(string-append "--coordinator=" coordinator)
+ #$(string-append "--uuid=" uuid)
+ #$@(if password
+ #~(#$(string-append "--password=" password))
+ #~())
+ #$@(if password-file
+ #~(#$(string-append "--password-file=" password-file))
+ #~())
+ #$(simple-format #f "--max-parallel-builds=~A"
+ max-parallel-builds)
+ #$@(if derivation-substitute-urls
+ #~(#$(string-append
+ "--derivation-substitute-urls="
+ (string-join derivation-substitute-urls " ")))
+ #~())
+ #$@(if non-derivation-substitute-urls
+ #~(#$(string-append
+ "--non-derivation-substitute-urls="
+ (string-join derivation-substitute-urls " ")))
+ #~())
+ #$@(map (lambda (system)
+ (string-append "--system=" system))
+ (or systems '())))
+ #:user #$user
+ #:pid-file "/var/run/guix-build-coordinator-agent/pid"
+ #:environment-variables
+ `(,(string-append
+ "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+ "LC_ALL=en_US.utf8")
+ #:log-file "/var/log/guix-build-coordinator/agent.log"))
+ (stop #~(make-kill-destructor))))))
+
+(define (guix-build-coordinator-agent-activation config)
+ #~(begin
+ (use-modules (guix build utils))
+
+ (mkdir-p "/var/log/guix-build-coordinator")
+
+ ;; Allow writing the PID file
+ (mkdir-p "/var/run/guix-build-coordinator-agent")
+ (chown "/var/run/guix-build-coordinator-agent"
+ (passwd:uid %user)
+ (passwd:gid %user))))
+
+(define (guix-build-coordinator-agent-account config)
+ (list (user-account
+ (name (guix-build-coordinator-agent-configuration-user config))
+ (group "nogroup")
+ (system? #t)
+ (comment "Guix Build Coordinator agent user")
+ (home-directory "/var/empty")
+ (shell (file-append shadow "/sbin/nologin")))))
+
+(define guix-build-coordinator-agent-service-type
+ (service-type
+ (name 'guix-build-coordinator-agent)
+ (extensions
+ (list
+ (service-extension shepherd-root-service-type
+ guix-build-coordinator-agent-shepherd-services)
+ (service-extension activation-service-type
+ guix-build-coordinator-agent-activation)
+ (service-extension account-service-type
+ guix-build-coordinator-agent-account)))
+ (description
+ "Run a Guix Build Coordinator agent.")))
+
\f
;;;
;;; Guix Data Service
--
2.28.0
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH v2 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
2020-09-19 10:10 ` [bug#43494] [PATCH v2 " Christopher Baines
2020-09-19 10:10 ` [bug#43494] [PATCH v2 2/4] services: guix: Add guix-build-coordinator-service-type Christopher Baines
2020-09-19 10:10 ` [bug#43494] [PATCH v2 3/4] services: guix: Add guix-build-coordinator-agent-service-type Christopher Baines
@ 2020-09-19 10:10 ` Christopher Baines
2020-10-05 8:00 ` Ludovic Courtès
2 siblings, 1 reply; 23+ messages in thread
From: Christopher Baines @ 2020-09-19 10:10 UTC (permalink / raw)
To: 43494
* gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):
New record type.
(guix-build-coordinator-queue-builds-configuration,
guix-build-coordinator-queue-builds-configuration?,
guix-build-coordinator-queue-builds-configuration-package,
guix-build-coordinator-queue-builds-configuration-user,
guix-build-coordinator-queue-builds-coordinator,
guix-build-coordinator-queue-builds-configuration-systems,
guix-build-coordinator-queue-builds-configuration-system-and-targets,
guix-build-coordinator-queue-builds-configuration-guix-data-service,
guix-build-coordinator-queue-builds-configuration-processed-commits-file,
guix-build-coordinator-queue-builds-shepherd-services,
guix-build-coordinator-queue-builds-activation,
guix-build-coordinator-queue-builds-account): New procedures.
(guix-build-coordinator-queue-builds-service-type): New variable.
---
doc/guix.texi | 38 +++++++++++++++
gnu/services/guix.scm | 108 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 146 insertions(+)
diff --git a/doc/guix.texi b/doc/guix.texi
index b2dde30ddb..767c99c5cb 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -27603,6 +27603,44 @@ input store items aren't already available.
@end table
@end deftp
+@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type
+Service type for the
+guix-build-coordinator-queue-builds-from-guix-data-service script. Its
+value must be a @code{guix-build-coordinator-queue-builds-configuration}
+object.
+@end defvar
+
+@deftp {Data Type} guix-build-coordinator-queue-builds-configuration
+Data type representing the options to the queue builds from guix data
+service script.
+
+@table @asis
+@item @code{package} (default: @code{guix-build-coordinator})
+The Guix Build Coordinator package to use.
+
+@item @code{user} (default: @code{"guix-build-coordinator-queue-builds"})
+The system user to run the service as.
+
+@item @code{coordinator} (default: @code{"http://localhost:8745"})
+The URI to use when connecting to the coordinator.
+
+@item @code{systems} (default: @code{#f})
+The systems for which to fetch derivations to build.
+
+@item @code{systems-and-targets} (default: @code{#f})
+An association list of system and target pairs for which to fetch
+derivations to build.
+
+@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"})
+The Guix Data Service instance from which to query to find out about
+derivations to build.
+
+@item @code{processed-commits-file} (default: @code{"/var/cache/guix-build-coordinator-queue-builds/processed-commits"})
+A file to record which commits have been processed, to avoid needlessly
+processing them again if the service is restarted.
+
+@end table
+@end deftp
@subsubheading Guix Data Service
The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores
diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm
index e4f5cf57b9..1493a24be5 100644
--- a/gnu/services/guix.scm
+++ b/gnu/services/guix.scm
@@ -65,6 +65,18 @@
guix-build-coordinator-agent-service-type
+ guix-build-coordinator-queue-builds-configuration
+ guix-build-coordinator-queue-builds-configuration?
+ guix-build-coordinator-queue-builds-configuration-package
+ guix-build-coordinator-queue-builds-configuration-user
+ guix-build-coordinator-queue-builds-coordinator
+ guix-build-coordinator-queue-builds-configuration-systems
+ guix-build-coordinator-queue-builds-configuration-system-and-targets
+ guix-build-coordinator-queue-builds-configuration-guix-data-service
+ guix-build-coordinator-queue-builds-configuration-processed-commits-file
+
+ guix-build-coordinator-queue-builds-service-type
+
<guix-data-service-configuration>
guix-data-service-configuration
guix-data-service-configuration?
@@ -137,6 +149,28 @@
guix-build-coordinator-agent-configuration-non-derivation-substitute-urls
(default #f)))
+(define-record-type* <guix-build-coordinator-queue-builds-configuration>
+ guix-build-coordinator-queue-builds-configuration
+ make-guix-build-coordinator-queue-builds-configuration
+ guix-build-coordinator-queue-builds-configuration?
+ (package guix-build-coordinator-queue-builds-configuration-package
+ (default guix-build-coordinator))
+ (user guix-build-coordinator-queue-builds-configuration-user
+ (default "guix-build-coordinator-queue-builds"))
+ (coordinator guix-build-coordinator-queue-builds-coordinator
+ (default "http://localhost:8745"))
+ (systems guix-build-coordinator-queue-builds-configuration-systems
+ (default #f))
+ (systems-and-targets
+ guix-build-coordinator-queue-builds-configuration-system-and-targets
+ (default #f))
+ (guix-data-service
+ guix-build-coordinator-queue-builds-configuration-guix-data-service
+ (default "https://data.guix.gnu.org"))
+ (processed-commits-file
+ guix-build-coordinator-queue-builds-configuration-processed-commits-file
+ (default "/var/cache/guix-build-coordinator-queue-builds/processed-commits")))
+
(define* (make-guix-build-coordinator-start-script database-uri-string
allocation-strategy
pid-file
@@ -360,6 +394,80 @@
(description
"Run a Guix Build Coordinator agent.")))
+(define (guix-build-coordinator-queue-builds-shepherd-services config)
+ (match-record config <guix-build-coordinator-queue-builds-configuration>
+ (package user coordinator systems systems-and-targets
+ guix-data-service processed-commits-file)
+ (list
+ (shepherd-service
+ (documentation "Guix Build Coordinator queue builds from Guix Data Service")
+ (provision '(guix-build-coordinator-queue-builds))
+ (requirement '(networking))
+ (start
+ #~(make-forkexec-constructor
+ (list
+ #$(file-append
+ package
+ "/bin/guix-build-coordinator-queue-builds-from-guix-data-service")
+ #$(string-append "--coordinator=" coordinator)
+ #$@(map (lambda (system)
+ (string-append "--system=" system))
+ (or systems '()))
+ #$@(map (match-lambda
+ ((system . target)
+ (string-append "--system-and-target=" system "=" target)))
+ (or systems-and-targets '()))
+ #$@(if guix-data-service
+ #~(#$(string-append "--guix-data-service=" guix-data-service))
+ #~())
+ #$@(if processed-commits-file
+ #~(#$(string-append "--processed-commits-file="
+ processed-commits-file))
+ #~()))
+ #:user #$user
+ #:pid-file "/var/run/guix-build-coordinator-queue-builds/pid"
+ #:environment-variables
+ `(,(string-append
+ "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale")
+ "LC_ALL=en_US.utf8")
+ #:log-file "/var/log/guix-build-coordinator/queue-builds.log"))
+ (stop #~(make-kill-destructor))))))
+
+(define (guix-build-coordinator-queue-builds-activation config)
+ #~(begin
+ (use-modules (guix build utils))
+
+ (mkdir-p "/var/log/guix-build-coordinator")
+
+ ;; Allow writing the PID file
+ (mkdir-p "/var/run/guix-build-coordinator-queue-builds")
+ (chown "/var/run/guix-build-coordinator-queue-builds"
+ (passwd:uid %user)
+ (passwd:gid %user))))
+
+(define (guix-build-coordinator-queue-builds-account config)
+ (list (user-account
+ (name (guix-build-coordinator-queue-builds-configuration-user config))
+ (group "nogroup")
+ (system? #t)
+ (comment "Guix Build Coordinator queue-builds user")
+ (home-directory "/var/empty")
+ (shell (file-append shadow "/sbin/nologin")))))
+
+(define guix-build-coordinator-queue-builds-service-type
+ (service-type
+ (name 'guix-build-coordinator-queue-builds)
+ (extensions
+ (list
+ (service-extension shepherd-root-service-type
+ guix-build-coordinator-queue-builds-shepherd-services)
+ (service-extension activation-service-type
+ guix-build-coordinator-queue-builds-activation)
+ (service-extension account-service-type
+ guix-build-coordinator-queue-builds-account)))
+ (description
+ "Run the Guix Build Coordinator queue builds script.")))
+
\f
;;;
;;; Guix Data Service
--
2.28.0
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
2020-09-19 9:13 ` Christopher Baines
@ 2020-09-25 9:42 ` Ludovic Courtès
2020-09-26 8:43 ` Christopher Baines
0 siblings, 1 reply; 23+ messages in thread
From: Ludovic Courtès @ 2020-09-25 9:42 UTC (permalink / raw)
To: Christopher Baines; +Cc: 43494
Hi!
Christopher Baines <mail@cbaines.net> skribis:
> Ludovic Courtès <ludo@gnu.org> writes:
[...]
>>> + (program-file
>>> + "start-guix-build-coordinator"
>>> + (with-extensions (cons guix-build-coordinator-package
>>> + ;; This is a poorly constructed Guile load path,
>>> + ;; since it contains things that aren't Guile
>>> + ;; libraries, but it means that the Guile libraries
>>> + ;; needed for the Guix Build Coordinator don't need
>>> + ;; to be individually specified here.
>>> + (map second (package-inputs
>>> + guix-build-coordinator-package)))
>>
>> Perhaps there should eventually be a ‘guix-build-coordinator’ command in
>> the package itself?
>
> There actually is, one thing I've had in mind for a while now though is
> to use a scheme script constructed by the Guix service to run the
> coordinator.
>
> For guix.cbaines.net, I'm using the script, but with the hooks passed in
> on the command line, the command is rather long, and it means that
> backtraces don't work well with the hooks.
You mean because the hooks are interpreted, and so all you see in the
backtrace is a bunch of ‘eval’ calls?
> I'm unsure how well this has worked out, I didn't anticipate the issues
> with the Guile load path, and because of the formatting stripping, both
> the code and backtraces will be unreadable anyway... but it is useful in
> allowing G-expressions to be used as part of the configuration.
OK.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type.
2020-09-25 9:42 ` Ludovic Courtès
@ 2020-09-26 8:43 ` Christopher Baines
0 siblings, 0 replies; 23+ messages in thread
From: Christopher Baines @ 2020-09-26 8:43 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 43494
[-- Attachment #1: Type: text/plain, Size: 1509 bytes --]
Ludovic Courtès <ludo@gnu.org> writes:
> Hi!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>
> [...]
>
>>>> + (program-file
>>>> + "start-guix-build-coordinator"
>>>> + (with-extensions (cons guix-build-coordinator-package
>>>> + ;; This is a poorly constructed Guile load path,
>>>> + ;; since it contains things that aren't Guile
>>>> + ;; libraries, but it means that the Guile libraries
>>>> + ;; needed for the Guix Build Coordinator don't need
>>>> + ;; to be individually specified here.
>>>> + (map second (package-inputs
>>>> + guix-build-coordinator-package)))
>>>
>>> Perhaps there should eventually be a ‘guix-build-coordinator’ command in
>>> the package itself?
>>
>> There actually is, one thing I've had in mind for a while now though is
>> to use a scheme script constructed by the Guix service to run the
>> coordinator.
>>
>> For guix.cbaines.net, I'm using the script, but with the hooks passed in
>> on the command line, the command is rather long, and it means that
>> backtraces don't work well with the hooks.
>
> You mean because the hooks are interpreted, and so all you see in the
> backtrace is a bunch of ‘eval’ calls?
Yeah, I haven't done much testing of this, but that's my assumption.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [bug#43494] [PATCH v2 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
2020-09-19 10:10 ` [bug#43494] [PATCH v2 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type Christopher Baines
@ 2020-10-05 8:00 ` Ludovic Courtès
2020-10-05 17:16 ` bug#43494: " Christopher Baines
0 siblings, 1 reply; 23+ messages in thread
From: Ludovic Courtès @ 2020-10-05 8:00 UTC (permalink / raw)
To: Christopher Baines; +Cc: 43494
Hi Chris,
Sorry for the delay. Patches 1–3 LGTM! One minor comment on patch 4:
Christopher Baines <mail@cbaines.net> skribis:
> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):
> New record type.
> (guix-build-coordinator-queue-builds-configuration,
> guix-build-coordinator-queue-builds-configuration?,
> guix-build-coordinator-queue-builds-configuration-package,
> guix-build-coordinator-queue-builds-configuration-user,
> guix-build-coordinator-queue-builds-coordinator,
> guix-build-coordinator-queue-builds-configuration-systems,
> guix-build-coordinator-queue-builds-configuration-system-and-targets,
> guix-build-coordinator-queue-builds-configuration-guix-data-service,
> guix-build-coordinator-queue-builds-configuration-processed-commits-file,
> guix-build-coordinator-queue-builds-shepherd-services,
> guix-build-coordinator-queue-builds-activation,
> guix-build-coordinator-queue-builds-account): New procedures.
> (guix-build-coordinator-queue-builds-service-type): New variable.
[...]
>
> +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type
> +Service type for the
> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its
> +value must be a @code{guix-build-coordinator-queue-builds-configuration}
> +object.
> +@end defvar
Could you add a paragraph right above this explaining what this service
is about?
> +(define guix-build-coordinator-queue-builds-service-type
> + (service-type
> + (name 'guix-build-coordinator-queue-builds)
> + (extensions
> + (list
> + (service-extension shepherd-root-service-type
> + guix-build-coordinator-queue-builds-shepherd-services)
> + (service-extension activation-service-type
> + guix-build-coordinator-queue-builds-activation)
> + (service-extension account-service-type
> + guix-build-coordinator-queue-builds-account)))
> + (description
> + "Run the Guix Build Coordinator queue builds script.")))
Likewise it’d be great if you could add a few works here.
OK to push with these changes, thank you!
Ludo’.
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#43494: [PATCH v2 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type.
2020-10-05 8:00 ` Ludovic Courtès
@ 2020-10-05 17:16 ` Christopher Baines
0 siblings, 0 replies; 23+ messages in thread
From: Christopher Baines @ 2020-10-05 17:16 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 43494-done
[-- Attachment #1: Type: text/plain, Size: 2799 bytes --]
Ludovic Courtès <ludo@gnu.org> writes:
> Hi Chris,
>
> Sorry for the delay. Patches 1–3 LGTM! One minor comment on patch 4:
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> * gnu/services/guix.scm (<guix-build-coordinator-queue-builds-configuration>):
>> New record type.
>> (guix-build-coordinator-queue-builds-configuration,
>> guix-build-coordinator-queue-builds-configuration?,
>> guix-build-coordinator-queue-builds-configuration-package,
>> guix-build-coordinator-queue-builds-configuration-user,
>> guix-build-coordinator-queue-builds-coordinator,
>> guix-build-coordinator-queue-builds-configuration-systems,
>> guix-build-coordinator-queue-builds-configuration-system-and-targets,
>> guix-build-coordinator-queue-builds-configuration-guix-data-service,
>> guix-build-coordinator-queue-builds-configuration-processed-commits-file,
>> guix-build-coordinator-queue-builds-shepherd-services,
>> guix-build-coordinator-queue-builds-activation,
>> guix-build-coordinator-queue-builds-account): New procedures.
>> (guix-build-coordinator-queue-builds-service-type): New variable.
>
> [...]
>
>>
>> +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type
>> +Service type for the
>> +guix-build-coordinator-queue-builds-from-guix-data-service script. Its
>> +value must be a @code{guix-build-coordinator-queue-builds-configuration}
>> +object.
>> +@end defvar
>
> Could you add a paragraph right above this explaining what this service
> is about?
>
>> +(define guix-build-coordinator-queue-builds-service-type
>> + (service-type
>> + (name 'guix-build-coordinator-queue-builds)
>> + (extensions
>> + (list
>> + (service-extension shepherd-root-service-type
>> + guix-build-coordinator-queue-builds-shepherd-services)
>> + (service-extension activation-service-type
>> + guix-build-coordinator-queue-builds-activation)
>> + (service-extension account-service-type
>> + guix-build-coordinator-queue-builds-account)))
>> + (description
>> + "Run the Guix Build Coordinator queue builds script.")))
>
> Likewise it’d be great if you could add a few works here.
>
> OK to push with these changes, thank you!
I've tried to add some clarity in the places you mention. I'm still very
tempted to make a separate repository/package for the queue builds from
Guix Data Service script... I'm not quite decided yet though.
Anyway, thanks for taking another look. I've gone ahead and pushed these
patches as c14714cbbfe9239410aec2ed73282192a64fbbdc. I did tweak the
package definition a bit though to move closer to being able to
cross-compile it (you can now if you tweak the inputs).
Thanks again,
Chris
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2020-10-05 17:17 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-09-18 18:34 [bug#43494] [PATCH 0/4] Add package and services for the Guix Build Coordinator Christopher Baines
2020-09-18 18:40 ` [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator Christopher Baines
2020-09-18 18:40 ` [bug#43494] [PATCH 2/4] services: guix: Add guix-build-coordinator-service-type Christopher Baines
2020-09-18 20:20 ` Ludovic Courtès
2020-09-19 9:13 ` Christopher Baines
2020-09-25 9:42 ` Ludovic Courtès
2020-09-26 8:43 ` Christopher Baines
2020-09-18 18:40 ` [bug#43494] [PATCH 3/4] services: guix: Add guix-build-coordinator-agent-service-type Christopher Baines
2020-09-18 20:25 ` Ludovic Courtès
2020-09-19 9:49 ` Christopher Baines
2020-09-18 18:40 ` [bug#43494] [PATCH 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type Christopher Baines
2020-09-18 20:31 ` Ludovic Courtès
2020-09-19 10:05 ` Christopher Baines
2020-09-18 20:08 ` [bug#43494] [PATCH 1/4] gnu: Add guix-build-coordinator Ludovic Courtès
2020-09-18 23:34 ` Jonathan Brielmaier
2020-09-19 9:00 ` Christopher Baines
2020-09-19 9:05 ` Christopher Baines
2020-09-19 10:10 ` [bug#43494] [PATCH v2 " Christopher Baines
2020-09-19 10:10 ` [bug#43494] [PATCH v2 2/4] services: guix: Add guix-build-coordinator-service-type Christopher Baines
2020-09-19 10:10 ` [bug#43494] [PATCH v2 3/4] services: guix: Add guix-build-coordinator-agent-service-type Christopher Baines
2020-09-19 10:10 ` [bug#43494] [PATCH v2 4/4] services: guix: Add guix-build-coordinator-queue-builds-service-type Christopher Baines
2020-10-05 8:00 ` Ludovic Courtès
2020-10-05 17:16 ` bug#43494: " Christopher Baines
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).