[bug#71038] [PATCH 0/2] Enable specifying the available builtin builders.

[bug#71038] [PATCH 0/2] Enable specifying the available builtin builders.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

This will enable addressing [11 by having the data service specify
'("download") for the builtin builders.

1: https://issues.guix.gnu.org/67250


Christopher Baines (2):
  guix: store: Enable specifying the available builtin builders.
  guix: channels: Enable specifiying available builtin builders.

 build-aux/build-self.scm | 28 ++++++++++++++++++--------
 guix/channels.scm        | 43 +++++++++++++++++++++++++++++-----------
 guix/store.scm           | 13 ++++++++----
 3 files changed, 60 insertions(+), 24 deletions(-)


base-commit: 0846eaecd45783bf40e8dc67b0c16f71068524b7
-- 
2.41.0

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

[bug#71038] [PATCH 1/2] guix: store: Enable specifying the available builtin builders.

[Christopher Baines]

To open-connection and port->connection.  This overrides the discovered
builtin builders that the daemon says it provides.

This is useful when you want to generate compatible derivations that can be
run with a daemon that potentially doesn't support builtin builders that the
daemon you're using to generate the derivations has.

I'm looking at this in particular because I want to use this in the data
service, since it provides substitutes for derivations, and since these can be
built on other machines, it's useful to control which builtin builders they
depend on.

* guix/store.scm (open-connection, port->connection): Accept
 #:assume-available-builtin-builders and use this instead of
%built-in-builders.

Fixes: <https://issues.guix.gnu.org/67250>.

Change-Id: I45d58ab93b6d276d280552858fc81ebc2b58828a
---
 guix/store.scm | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/guix/store.scm b/guix/store.scm
index 58ddaa8d15..0c734cdca7 100644
--- a/guix/store.scm
+++ b/guix/store.scm
@@ -571,7 +571,7 @@ (define* (connect-to-daemon uri #:key non-blocking?)
 
 (define* (open-connection #:optional (uri (%daemon-socket-uri))
                           #:key port (reserve-space? #t) cpu-affinity
-                          non-blocking?)
+                          non-blocking? assume-available-builtin-builders)
   "Connect to the daemon at URI (a string), or, if PORT is not #f, use it as
 the I/O port over which to communicate to a build daemon.
 
@@ -616,7 +616,9 @@ (define* (open-connection #:optional (uri (%daemon-socket-uri))
           (when (>= (protocol-minor v) 11)
             (write-int (if reserve-space? 1 0) port))
           (letrec* ((built-in-builders
-                     (delay (%built-in-builders conn)))
+                     (if assume-available-builtin-builders
+                         (delay assume-available-builtin-builders)
+                         (delay (%built-in-builders conn))))
                     (caches
                      (make-vector
                       (atomic-box-ref %store-connection-caches)
@@ -635,7 +637,8 @@ (define* (open-connection #:optional (uri (%daemon-socket-uri))
             conn))))))
 
 (define* (port->connection port
-                           #:key (version %protocol-version))
+                           #:key (version %protocol-version)
+                           assume-available-builtin-builders)
   "Assimilate PORT, an input/output port, and return a connection to the
 daemon, assuming the given protocol VERSION.
 
@@ -654,7 +657,9 @@ (define* (port->connection port
                               (make-vector
                                (atomic-box-ref %store-connection-caches)
                                vlist-null)
-                              (delay (%built-in-builders connection))))
+                              (if assume-available-builtin-builders
+                                  (delay assume-available-builtin-builders)
+                                  (delay (%built-in-builders connection)))))
 
     connection))
 

base-commit: 0846eaecd45783bf40e8dc67b0c16f71068524b7
-- 
2.41.0

[bug#71038] [PATCH 2/2] guix: channels: Enable specifiying available builtin builders.

[Christopher Baines]

When computing channel instance derivations.

This is useful when you want to generate compatible derivations that can be
run with a daemon that potentially doesn't support builtin builders that the
daemon you're using to generate the derivations has.

I'm looking at this in particular because I want to use this in the data
service, since it provides substitutes for derivations, and since these can be
built on other machines, it's useful to control which builtin builders they
depend on.

Fixes: <https://issues.guix.gnu.org/67250>.

* build-aux/build-self.scm (build-program): Accept
 #:assume-available-builtin-builders and pass along to port->connection or
open-connection as approriate.
(build): Accept and pass on #:assume-available-builtin-builders.
* guix/channels.scm (build-from-source, build-channel-instance,
channel-instance-derivations, channel-instances->manifest,
channel-instances->derivation): Accept and pass on
 #:assume-available-builtin-builders.

Change-Id: I315c990de66c6f7dca25a859165a5568abe385ea
---
 build-aux/build-self.scm | 28 ++++++++++++++++++--------
 guix/channels.scm        | 43 +++++++++++++++++++++++++++++-----------
 2 files changed, 51 insertions(+), 20 deletions(-)

diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index 02822a2ee8..7afb13c1e4 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -241,7 +241,8 @@ (define guile-gcrypt
 
 (define* (build-program source version
                         #:optional (guile-version (effective-version))
-                        #:key (pull-version 0) (channel-metadata #f))
+                        #:key (pull-version 0) (channel-metadata #f)
+                        assume-available-builtin-builders)
   "Return a program that computes the derivation to build Guix from SOURCE."
   (define select?
     ;; Select every module but (guix config) and non-Guix modules.
@@ -331,11 +332,20 @@ (define* (build-program source version
                          ;; case, attempt to open a new connection.
                          (let* ((proto (string->number protocol-version))
                                 (store (if (integer? proto)
-                                           (port->connection (duplicate-port
-                                                              (current-input-port)
-                                                              "w+0")
-                                                             #:version proto)
-                                           (open-connection)))
+                                           (port->connection
+                                            (duplicate-port
+                                             (current-input-port)
+                                             "w+0")
+                                            #:version proto
+                                            #$@(if assume-available-builtin-builders
+                                                   #~(#:assume-available-builtin-builders
+                                                      '(#$@assume-available-builtin-builders))
+                                                   '()))
+                                           (open-connection
+                                            #$@(if assume-available-builtin-builders
+                                                   #~(#:assume-available-builtin-builders
+                                                      '(#$@assume-available-builtin-builders))
+                                                   '()))))
                                 (sock  (socket AF_UNIX SOCK_STREAM 0)))
                            ;; Connect to BUILD-OUTPUT and send it the raw
                            ;; build output.
@@ -406,7 +416,7 @@ (define* (build source
                 (guile-version (if (> pull-version 0)
                                    "3.0"
                                    (effective-version)))
-
+                assume-available-builtin-builders
                 #:allow-other-keys
                 #:rest rest)
   "Return a derivation that unpacks SOURCE into STORE and compiles Scheme
@@ -415,7 +425,9 @@ (define* (build source
   ;; SOURCE.
   (mlet %store-monad ((build  (build-program source version guile-version
                                              #:channel-metadata channel-metadata
-                                             #:pull-version pull-version))
+                                             #:pull-version pull-version
+                                             #:assume-available-builtin-builders
+                                             assume-available-builtin-builders))
                       (system (if system (return system) (current-system)))
                       (home -> (getenv "HOME"))
 
diff --git a/guix/channels.scm b/guix/channels.scm
index 0d7bc541cc..6469d08bb9 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -704,7 +704,8 @@ (define (with-trivial-build-handler mvalue)
               store))))
 
 (define* (build-from-source instance
-                            #:key core verbose? (dependencies '()) system)
+                            #:key core verbose? (dependencies '()) system
+                            assume-available-builtin-builders)
   "Return a derivation to build Guix from INSTANCE, using the self-build
 script contained therein.  When CORE is true, build package modules under
 SOURCE using CORE, an instance of Guix.  By default, build for the current
@@ -750,20 +751,25 @@ (define* (build-from-source instance
                   #:verbose? verbose? #:version commit
                   #:system system
                   #:channel-metadata (channel-instance->sexp instance)
-                  #:pull-version %pull-version))))
+                  #:pull-version %pull-version
+                  #:assume-available-builtin-builders
+                  assume-available-builtin-builders))))
 
       ;; Build a set of modules that extend Guix using the standard method.
       (standard-module-derivation name source core dependencies)))
 
 (define* (build-channel-instance instance system
-                                 #:optional core (dependencies '()))
+                                 #:optional core (dependencies '())
+                                 #:key assume-available-builtin-builders)
   "Return, as a monadic value, the derivation for INSTANCE, a channel
 instance, for SYSTEM.  DEPENDENCIES is a list of extensions providing Guile
 modules that INSTANCE depends on."
   (build-from-source instance
                      #:core core
                      #:dependencies dependencies
-                     #:system system))
+                     #:system system
+                     #:assume-available-builtin-builders
+                     assume-available-builtin-builders))
 
 (define (resolve-dependencies instances)
   "Return a procedure that, given one of the elements of INSTANCES, returns
@@ -793,7 +799,8 @@ (define (resolve-dependencies instances)
   (lambda (instance)
     (vhash-foldq* cons '() instance edges)))
 
-(define* (channel-instance-derivations instances #:key system)
+(define* (channel-instance-derivations instances #:key system
+                                       assume-available-builtin-builders)
   "Return the list of derivations to build INSTANCES, in the same order as
 INSTANCES.  Build for the current system by default, or SYSTEM if specified."
   (define core-instance
@@ -809,11 +816,15 @@ (define* (channel-instance-derivations instances #:key system)
   (define (instance->derivation instance)
     (mlet %store-monad ((system (if system (return system) (current-system))))
       (mcached (if (eq? instance core-instance)
-                   (build-channel-instance instance system)
+                   (build-channel-instance instance system
+                                           #:assume-available-builtin-builders
+                                           assume-available-builtin-builders)
                    (mlet %store-monad ((core (instance->derivation core-instance))
                                        (deps (mapm %store-monad instance->derivation
                                                    (edges instance))))
-                     (build-channel-instance instance system core deps)))
+                     (build-channel-instance instance system core deps
+                                             #:assume-available-builtin-builders
+                                             assume-available-builtin-builders)))
                instance
                system)))
 
@@ -915,7 +926,8 @@ (define (channel-instance->sexp instance)
                     intro))))))
             '()))))
 
-(define* (channel-instances->manifest instances #:key system)
+(define* (channel-instances->manifest instances #:key system
+                                      assume-available-builtin-builders)
   "Return a profile manifest with entries for all of INSTANCES, a list of
 channel instances.  By default, build for the current system, or SYSTEM if
 specified."
@@ -934,8 +946,11 @@ (define* (channel-instances->manifest instances #:key system)
         (properties
          `((source ,(channel-instance->sexp instance)))))))
 
-  (mlet* %store-monad ((derivations (channel-instance-derivations instances
-                                                                  #:system system))
+  (mlet* %store-monad ((derivations (channel-instance-derivations
+                                     instances
+                                     #:system system
+                                     #:assume-available-builtin-builders
+                                     assume-available-builtin-builders))
                        (entries ->  (map instance->entry instances derivations)))
     (return (manifest entries))))
 
@@ -990,10 +1005,14 @@ (define %channel-profile-hooks
   ;; The default channel profile hooks.
   (cons package-cache-file %default-profile-hooks))
 
-(define (channel-instances->derivation instances)
+(define* (channel-instances->derivation instances
+                                        #:key assume-available-builtin-builders)
   "Return the derivation of the profile containing INSTANCES, a list of
 channel instances."
-  (mlet %store-monad ((manifest (channel-instances->manifest instances)))
+  (mlet %store-monad ((manifest (channel-instances->manifest
+                                 instances
+                                 #:assume-available-builtin-builders
+                                 assume-available-builtin-builders)))
     ;; Emit a profile in format version so that, if INSTANCES denotes an old
     ;; Guix, it can still read that profile, for instance for the purposes of
     ;; 'guix describe'.
-- 
2.41.0

[bug#71038] [PATCH 1/2] guix: store: Enable specifying the available builtin builders.

[Simon Tournier]

Hi Chris,

On sam., 18 mai 2024 at 14:19, Christopher Baines <mail@cbaines.net> wrote:

> diff --git a/guix/store.scm b/guix/store.scm
> index 58ddaa8d15..0c734cdca7 100644
> --- a/guix/store.scm
> +++ b/guix/store.scm
> @@ -571,7 +571,7 @@ (define* (connect-to-daemon uri #:key non-blocking?)
>
>  (define* (open-connection #:optional (uri (%daemon-socket-uri))
>                            #:key port (reserve-space? #t) cpu-affinity
> -                          non-blocking?)
> +                          non-blocking? assume-available-builtin-builders)

Why add the variable %assume-available-builtin-builders and default to
it?

Something like:

--8<---------------cut here---------------start------------->8---
(define %assume-available-builtin-builders
  "List of builtin builders supported by the builder Guix daemon."
  (list "download" "git-download"))

(define* (open-connection #:optional (uri (%daemon-socket-uri))
                          #:key port (reserve-space? #t) cpu-affinity
                          non-blocking?)
                          non-blocking?
                          (assume-available-builtin-builders %assume-available-builtin-builders))
--8<---------------cut here---------------end--------------->8---

And then default to this %assume-available-builtin-builders elsewhere in
[PATCH 2/2].  IMHO, it changes almost nothing but it would help to know
(document) what to pass as argument.

Cheers,
simon

[bug#71038] [PATCH 1/2] guix: store: Enable specifying the available builtin builders.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1612 bytes --]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> On sam., 18 mai 2024 at 14:19, Christopher Baines <mail@cbaines.net> wrote:
>
>> diff --git a/guix/store.scm b/guix/store.scm
>> index 58ddaa8d15..0c734cdca7 100644
>> --- a/guix/store.scm
>> +++ b/guix/store.scm
>> @@ -571,7 +571,7 @@ (define* (connect-to-daemon uri #:key non-blocking?)
>>
>>  (define* (open-connection #:optional (uri (%daemon-socket-uri))
>>                            #:key port (reserve-space? #t) cpu-affinity
>> -                          non-blocking?)
>> +                          non-blocking? assume-available-builtin-builders)
>
> Why add the variable %assume-available-builtin-builders and default to
> it?
>
> Something like:
>
> --8<---------------cut here---------------start------------->8---
> (define %assume-available-builtin-builders
>   "List of builtin builders supported by the builder Guix daemon."
>   (list "download" "git-download"))
>
> (define* (open-connection #:optional (uri (%daemon-socket-uri))
>                           #:key port (reserve-space? #t) cpu-affinity
>                           non-blocking?)
>                           non-blocking?
>                           (assume-available-builtin-builders %assume-available-builtin-builders))
> --8<---------------cut here---------------end--------------->8---
>
> And then default to this %assume-available-builtin-builders elsewhere in
> [PATCH 2/2].  IMHO, it changes almost nothing but it would help to know
> (document) what to pass as argument.

I think it's sensible to not use a fixed list by default, but check what
the daemon supports.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

[bug#71038] [PATCH 1/2] guix: store: Enable specifying the available builtin builders.

[Simon Tournier]

Hi Chris,

On Sun, 26 May 2024 at 09:10, Christopher Baines <mail@cbaines.net> wrote:

>>>  (define* (open-connection #:optional (uri (%daemon-socket-uri))
>>>                            #:key port (reserve-space? #t) cpu-affinity
>>> -                          non-blocking?)
>>> +                          non-blocking? assume-available-builtin-builders)
>>
>> Why add the variable %assume-available-builtin-builders and default to
>> it?
>>
>> Something like:
>>
>> --8<---------------cut here---------------start------------->8---
>> (define %assume-available-builtin-builders
>>   "List of builtin builders supported by the builder Guix daemon."
>>   (list "download" "git-download"))
>>
>> (define* (open-connection #:optional (uri (%daemon-socket-uri))
>>                           #:key port (reserve-space? #t) cpu-affinity
>>                           non-blocking?)
>>                           non-blocking?
>>                           (assume-available-builtin-builders %assume-available-builtin-builders))
>> --8<---------------cut here---------------end--------------->8---
>>
>> And then default to this %assume-available-builtin-builders elsewhere in
>> [PATCH 2/2].  IMHO, it changes almost nothing but it would help to know
>> (document) what to pass as argument.
>
> I think it's sensible to not use a fixed list by default, but check what
> the daemon supports.

Do you mean dynamically construct the proposal of
%assume-available-builtin-builders?  Why not.

Aside, my point is to provide a default value for the new argument and
not let it free.  Because when reading the source code, not knowing its
type, neither any meaningful value make it hard to remember what it use
or how to use it, IMHO.  That’s why I am suggesting something like
%assume-available-builtin-builders that collects the acceptable values
– for the most recent daemon, indeed; well it would simplify the
documentation of this new parameter / argument.

Considering your patch, how do I know that I could used it, as [1]:

--8<---------------cut here---------------start------------->8---
As in:

  (open-connection
    #:assume-available-builtin-builders '("download"))
--8<---------------cut here---------------end--------------->8---

Because it is not clear from the docstring.  And there is many
procedures that would require some docstring update with this new
procedure argument. :-)


1: bug#67250: builtin:git-download capability detection not working for the bordeaux build farm
Ludovic Courtès <ludo@gnu.org>
Wed, 22 Nov 2023 11:19:42 +0100
id:87bkbmm6o1.fsf@gnu.org
https://issues.guix.gnu.org/67250
https://issues.guix.gnu.org/msgid/87bkbmm6o1.fsf@gnu.org
https://yhetil.org/guix/87bkbmm6o1.fsf@gnu.org

Cheers,
simon

[bug#71038] [PATCH 1/2] guix: store: Enable specifying the available builtin builders.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2785 bytes --]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> On Sun, 26 May 2024 at 09:10, Christopher Baines <mail@cbaines.net> wrote:
>
>>>>  (define* (open-connection #:optional (uri (%daemon-socket-uri))
>>>>                            #:key port (reserve-space? #t) cpu-affinity
>>>> -                          non-blocking?)
>>>> +                          non-blocking? assume-available-builtin-builders)
>>>
>>> Why add the variable %assume-available-builtin-builders and default to
>>> it?
>>>
>>> Something like:
>>>
>>> --8<---------------cut here---------------start------------->8---
>>> (define %assume-available-builtin-builders
>>>   "List of builtin builders supported by the builder Guix daemon."
>>>   (list "download" "git-download"))
>>>
>>> (define* (open-connection #:optional (uri (%daemon-socket-uri))
>>>                           #:key port (reserve-space? #t) cpu-affinity
>>>                           non-blocking?)
>>>                           non-blocking?
>>>                           (assume-available-builtin-builders %assume-available-builtin-builders))
>>> --8<---------------cut here---------------end--------------->8---
>>>
>>> And then default to this %assume-available-builtin-builders elsewhere in
>>> [PATCH 2/2].  IMHO, it changes almost nothing but it would help to know
>>> (document) what to pass as argument.
>>
>> I think it's sensible to not use a fixed list by default, but check what
>> the daemon supports.
>
> Do you mean dynamically construct the proposal of
> %assume-available-builtin-builders?  Why not.
>
> Aside, my point is to provide a default value for the new argument and
> not let it free.  Because when reading the source code, not knowing its
> type, neither any meaningful value make it hard to remember what it use
> or how to use it, IMHO.  That’s why I am suggesting something like
> %assume-available-builtin-builders that collects the acceptable values
> – for the most recent daemon, indeed; well it would simplify the
> documentation of this new parameter / argument.

I'm not sure I follow. I guess open-connection could have a
#:build-in-builders argument that expects a procedure that takes the
store connection, and returns the list of strings. That would allow it
to have the default of %built-in-builders from (guix store).

While this adds the flexibility for users to provide their own way of
setting the builtin builders enabled on a connection, I'm not sure
there's a need for it currently and I don't think it addresses your
concern about not knowing what value to provide.

It sounds easier just to make it clear from the docstrings that you
provide a list of strings, and have it default to #f to indicate to the
daemon's buildin builders.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

[bug#71038] [PATCH v2 1/2] guix: store: Enable specifying the available builtin builders.

[Christopher Baines]

To open-connection and port->connection.  This overrides the discovered
builtin builders that the daemon says it provides.

This is useful when you want to generate compatible derivations that can be
run with a daemon that potentially doesn't support builtin builders that the
daemon you're using to generate the derivations has.

I'm looking at this in particular because I want to use this in the data
service, since it provides substitutes for derivations, and since these can be
built on other machines, it's useful to control which builtin builders they
depend on.

* guix/store.scm (open-connection, port->connection): Accept
 #:assume-available-builtin-builders and use this instead of
%built-in-builders.

Fixes: <https://issues.guix.gnu.org/67250>.

Change-Id: I45d58ab93b6d276d280552858fc81ebc2b58828a
---
 guix/store.scm | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/guix/store.scm b/guix/store.scm
index 58ddaa8d15..8c823a2185 100644
--- a/guix/store.scm
+++ b/guix/store.scm
@@ -571,7 +571,7 @@ (define* (connect-to-daemon uri #:key non-blocking?)
 
 (define* (open-connection #:optional (uri (%daemon-socket-uri))
                           #:key port (reserve-space? #t) cpu-affinity
-                          non-blocking?)
+                          non-blocking? assume-available-builtin-builders)
   "Connect to the daemon at URI (a string), or, if PORT is not #f, use it as
 the I/O port over which to communicate to a build daemon.
 
@@ -580,8 +580,10 @@ (define* (open-connection #:optional (uri (%daemon-socket-uri))
 should the disk become full.  When CPU-AFFINITY is true, it must be an integer
 corresponding to an OS-level CPU number to which the daemon's worker process
 for this connection will be pinned.  If NON-BLOCKING?, use a non-blocking
-socket when using the file, unix or guix URI schemes.  Return a server
-object."
+socket when using the file, unix or guix URI schemes.  If
+ASSUME-AVAILABLE-BUILTIN-BUILDERS is provided, it should be a list of strings
+and this will be used instead of the builtin builders provided by the build
+daemon.  Return a server object."
   (define (handshake-error)
     (raise (condition
             (&store-connection-error (file (or port uri))
@@ -616,7 +618,9 @@ (define* (open-connection #:optional (uri (%daemon-socket-uri))
           (when (>= (protocol-minor v) 11)
             (write-int (if reserve-space? 1 0) port))
           (letrec* ((built-in-builders
-                     (delay (%built-in-builders conn)))
+                     (if assume-available-builtin-builders
+                         (delay assume-available-builtin-builders)
+                         (delay (%built-in-builders conn))))
                     (caches
                      (make-vector
                       (atomic-box-ref %store-connection-caches)
@@ -635,9 +639,13 @@ (define* (open-connection #:optional (uri (%daemon-socket-uri))
             conn))))))
 
 (define* (port->connection port
-                           #:key (version %protocol-version))
+                           #:key (version %protocol-version)
+                           assume-available-builtin-builders)
   "Assimilate PORT, an input/output port, and return a connection to the
-daemon, assuming the given protocol VERSION.
+daemon, assuming the given protocol VERSION.  If
+ASSUME-AVAILABLE-BUILTIN-BUILDERS is provided, it should be a list of strings
+and this will be used instead of the builtin builders provided by the build
+daemon.
 
 Warning: this procedure assumes that the initial handshake with the daemon has
 already taken place on PORT and that we're just continuing on this established
@@ -654,7 +662,9 @@ (define* (port->connection port
                               (make-vector
                                (atomic-box-ref %store-connection-caches)
                                vlist-null)
-                              (delay (%built-in-builders connection))))
+                              (if assume-available-builtin-builders
+                                  (delay assume-available-builtin-builders)
+                                  (delay (%built-in-builders connection)))))
 
     connection))
 

base-commit: 018f2781d5aa301c156c81981f1fd3df56e438b9
-- 
2.45.1

[bug#71038] [PATCH v2 2/2] guix: channels: Enable specifiying available builtin builders.

[Christopher Baines]

When computing channel instance derivations.

This is useful when you want to generate compatible derivations that can be
run with a daemon that potentially doesn't support builtin builders that the
daemon you're using to generate the derivations has.

I'm looking at this in particular because I want to use this in the data
service, since it provides substitutes for derivations, and since these can be
built on other machines, it's useful to control which builtin builders they
depend on.

Fixes: <https://issues.guix.gnu.org/67250>.

* build-aux/build-self.scm (build-program): Accept
 #:assume-available-builtin-builders and pass along to port->connection or
open-connection as approriate.
(build): Accept and pass on #:assume-available-builtin-builders.
* guix/channels.scm (build-from-source, build-channel-instance,
channel-instance-derivations, channel-instances->manifest,
channel-instances->derivation): Accept and pass on
 #:assume-available-builtin-builders.

Change-Id: I315c990de66c6f7dca25a859165a5568abe385ea
---
 build-aux/build-self.scm | 33 ++++++++++++++------
 guix/channels.scm        | 67 ++++++++++++++++++++++++++++++----------
 2 files changed, 74 insertions(+), 26 deletions(-)

diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index 02822a2ee8..c14bebd0b1 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -241,8 +241,12 @@ (define guile-gcrypt
 
 (define* (build-program source version
                         #:optional (guile-version (effective-version))
-                        #:key (pull-version 0) (channel-metadata #f))
-  "Return a program that computes the derivation to build Guix from SOURCE."
+                        #:key (pull-version 0) (channel-metadata #f)
+                        assume-available-builtin-builders)
+  "Return a program that computes the derivation to build Guix from SOURCE.
+If ASSUME-AVAILABLE-BUILTIN-BUILDERS is provided, it should be a list of
+strings and this will be used instead of the builtin builders provided by the
+build daemon, from within the generated build program."
   (define select?
     ;; Select every module but (guix config) and non-Guix modules.
     ;; Also exclude (guix channels): it is autoloaded by (guix describe), but
@@ -331,11 +335,20 @@ (define* (build-program source version
                          ;; case, attempt to open a new connection.
                          (let* ((proto (string->number protocol-version))
                                 (store (if (integer? proto)
-                                           (port->connection (duplicate-port
-                                                              (current-input-port)
-                                                              "w+0")
-                                                             #:version proto)
-                                           (open-connection)))
+                                           (port->connection
+                                            (duplicate-port
+                                             (current-input-port)
+                                             "w+0")
+                                            #:version proto
+                                            #$@(if assume-available-builtin-builders
+                                                   #~(#:assume-available-builtin-builders
+                                                      '(#$@assume-available-builtin-builders))
+                                                   '()))
+                                           (open-connection
+                                            #$@(if assume-available-builtin-builders
+                                                   #~(#:assume-available-builtin-builders
+                                                      '(#$@assume-available-builtin-builders))
+                                                   '()))))
                                 (sock  (socket AF_UNIX SOCK_STREAM 0)))
                            ;; Connect to BUILD-OUTPUT and send it the raw
                            ;; build output.
@@ -406,7 +419,7 @@ (define* (build source
                 (guile-version (if (> pull-version 0)
                                    "3.0"
                                    (effective-version)))
-
+                assume-available-builtin-builders
                 #:allow-other-keys
                 #:rest rest)
   "Return a derivation that unpacks SOURCE into STORE and compiles Scheme
@@ -415,7 +428,9 @@ (define* (build source
   ;; SOURCE.
   (mlet %store-monad ((build  (build-program source version guile-version
                                              #:channel-metadata channel-metadata
-                                             #:pull-version pull-version))
+                                             #:pull-version pull-version
+                                             #:assume-available-builtin-builders
+                                             assume-available-builtin-builders))
                       (system (if system (return system) (current-system)))
                       (home -> (getenv "HOME"))
 
diff --git a/guix/channels.scm b/guix/channels.scm
index 0d7bc541cc..dfdf6cfe3f 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -704,11 +704,15 @@ (define (with-trivial-build-handler mvalue)
               store))))
 
 (define* (build-from-source instance
-                            #:key core verbose? (dependencies '()) system)
+                            #:key core verbose? (dependencies '()) system
+                            assume-available-builtin-builders)
   "Return a derivation to build Guix from INSTANCE, using the self-build
 script contained therein.  When CORE is true, build package modules under
 SOURCE using CORE, an instance of Guix.  By default, build for the current
-system, or SYSTEM if specified."
+system, or SYSTEM if specified.  If ASSUME-AVAILABLE-BUILTIN-BUILDERS is
+provided, it should be a list of strings and this will be used instead of the
+builtin builders provided by the build daemon for store connections used
+during this process."
   (define name
     (symbol->string
      (channel-name (channel-instance-channel instance))))
@@ -750,20 +754,28 @@ (define* (build-from-source instance
                   #:verbose? verbose? #:version commit
                   #:system system
                   #:channel-metadata (channel-instance->sexp instance)
-                  #:pull-version %pull-version))))
+                  #:pull-version %pull-version
+                  #:assume-available-builtin-builders
+                  assume-available-builtin-builders))))
 
       ;; Build a set of modules that extend Guix using the standard method.
       (standard-module-derivation name source core dependencies)))
 
 (define* (build-channel-instance instance system
-                                 #:optional core (dependencies '()))
+                                 #:optional core (dependencies '())
+                                 #:key assume-available-builtin-builders)
   "Return, as a monadic value, the derivation for INSTANCE, a channel
 instance, for SYSTEM.  DEPENDENCIES is a list of extensions providing Guile
-modules that INSTANCE depends on."
+modules that INSTANCE depends on.  If ASSUME-AVAILABLE-BUILTIN-BUILDERS is
+provided, it should be a list of strings and this will be used instead of the
+builtin builders provided by the build daemon for store connections used
+during this process."
   (build-from-source instance
                      #:core core
                      #:dependencies dependencies
-                     #:system system))
+                     #:system system
+                     #:assume-available-builtin-builders
+                     assume-available-builtin-builders))
 
 (define (resolve-dependencies instances)
   "Return a procedure that, given one of the elements of INSTANCES, returns
@@ -793,9 +805,13 @@ (define (resolve-dependencies instances)
   (lambda (instance)
     (vhash-foldq* cons '() instance edges)))
 
-(define* (channel-instance-derivations instances #:key system)
+(define* (channel-instance-derivations instances #:key system
+                                       assume-available-builtin-builders)
   "Return the list of derivations to build INSTANCES, in the same order as
-INSTANCES.  Build for the current system by default, or SYSTEM if specified."
+INSTANCES.  Build for the current system by default, or SYSTEM if specified.
+If ASSUME-AVAILABLE-BUILTIN-BUILDERS is provided, it should be a list of
+strings and this will be used instead of the builtin builders provided by the
+build daemon for store connections used during this process."
   (define core-instance
     ;; The 'guix' channel is treated specially: it's an implicit dependency of
     ;; all the other channels.
@@ -809,11 +825,15 @@ (define* (channel-instance-derivations instances #:key system)
   (define (instance->derivation instance)
     (mlet %store-monad ((system (if system (return system) (current-system))))
       (mcached (if (eq? instance core-instance)
-                   (build-channel-instance instance system)
+                   (build-channel-instance instance system
+                                           #:assume-available-builtin-builders
+                                           assume-available-builtin-builders)
                    (mlet %store-monad ((core (instance->derivation core-instance))
                                        (deps (mapm %store-monad instance->derivation
                                                    (edges instance))))
-                     (build-channel-instance instance system core deps)))
+                     (build-channel-instance instance system core deps
+                                             #:assume-available-builtin-builders
+                                             assume-available-builtin-builders)))
                instance
                system)))
 
@@ -915,10 +935,13 @@ (define (channel-instance->sexp instance)
                     intro))))))
             '()))))
 
-(define* (channel-instances->manifest instances #:key system)
+(define* (channel-instances->manifest instances #:key system
+                                      assume-available-builtin-builders)
   "Return a profile manifest with entries for all of INSTANCES, a list of
 channel instances.  By default, build for the current system, or SYSTEM if
-specified."
+specified.  If ASSUME-AVAILABLE-BUILTIN-BUILDERS is provided, it should be a
+list of strings and this will be used instead of the builtin builders provided
+by the build daemon for store connections used during this process."
   (define (instance->entry instance drv)
     (let ((commit  (channel-instance-commit instance))
           (channel (channel-instance-channel instance)))
@@ -934,8 +957,11 @@ (define* (channel-instances->manifest instances #:key system)
         (properties
          `((source ,(channel-instance->sexp instance)))))))
 
-  (mlet* %store-monad ((derivations (channel-instance-derivations instances
-                                                                  #:system system))
+  (mlet* %store-monad ((derivations (channel-instance-derivations
+                                     instances
+                                     #:system system
+                                     #:assume-available-builtin-builders
+                                     assume-available-builtin-builders))
                        (entries ->  (map instance->entry instances derivations)))
     (return (manifest entries))))
 
@@ -990,10 +1016,17 @@ (define %channel-profile-hooks
   ;; The default channel profile hooks.
   (cons package-cache-file %default-profile-hooks))
 
-(define (channel-instances->derivation instances)
+(define* (channel-instances->derivation instances
+                                        #:key assume-available-builtin-builders)
   "Return the derivation of the profile containing INSTANCES, a list of
-channel instances."
-  (mlet %store-monad ((manifest (channel-instances->manifest instances)))
+channel instances.  If ASSUME-AVAILABLE-BUILTIN-BUILDERS is provided, it
+should be a list of strings and this will be used instead of the builtin
+builders provided by the build daemon for store connections used during this
+process."
+  (mlet %store-monad ((manifest (channel-instances->manifest
+                                 instances
+                                 #:assume-available-builtin-builders
+                                 assume-available-builtin-builders)))
     ;; Emit a profile in format version so that, if INSTANCES denotes an old
     ;; Guix, it can still read that profile, for instance for the purposes of
     ;; 'guix describe'.
-- 
2.45.1

[bug#71038] [PATCH v2 1/2] guix: store: Enable specifying the available builtin builders.

[Ludovic Courtès via Guix-patches]

Hello!

Summarizing our live review session here…

Christopher Baines <mail@cbaines.net> skribis:

> To open-connection and port->connection.  This overrides the discovered
> builtin builders that the daemon says it provides.
>
> This is useful when you want to generate compatible derivations that can be
> run with a daemon that potentially doesn't support builtin builders that the
> daemon you're using to generate the derivations has.
>
> I'm looking at this in particular because I want to use this in the data
> service, since it provides substitutes for derivations, and since these can be
> built on other machines, it's useful to control which builtin builders they
> depend on.
>
> * guix/store.scm (open-connection, port->connection): Accept
>  #:assume-available-builtin-builders and use this instead of
> %built-in-builders.
>
> Fixes: <https://issues.guix.gnu.org/67250>.
>
> Change-Id: I45d58ab93b6d276d280552858fc81ebc2b58828a

This is nice.  Nitpick: you can drop “guix:” from the commit message subject.

>  (define* (open-connection #:optional (uri (%daemon-socket-uri))
>                            #:key port (reserve-space? #t) cpu-affinity
> -                          non-blocking?)
> +                          non-blocking? assume-available-builtin-builders)

My only suggestion here is to rename ‘assume-available-builtin-builders’
to just ‘built-in-builders’ (so that it’s a noun rather than a phrase).

Ludo’.

[bug#71038] [PATCH v2 2/2] guix: channels: Enable specifiying available builtin builders.

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> When computing channel instance derivations.
>
> This is useful when you want to generate compatible derivations that can be
> run with a daemon that potentially doesn't support builtin builders that the
> daemon you're using to generate the derivations has.
>
> I'm looking at this in particular because I want to use this in the data
> service, since it provides substitutes for derivations, and since these can be
> built on other machines, it's useful to control which builtin builders they
> depend on.
>
> Fixes: <https://issues.guix.gnu.org/67250>.
>
> * build-aux/build-self.scm (build-program): Accept
>  #:assume-available-builtin-builders and pass along to port->connection or
> open-connection as approriate.
> (build): Accept and pass on #:assume-available-builtin-builders.
> * guix/channels.scm (build-from-source, build-channel-instance,
> channel-instance-derivations, channel-instances->manifest,
> channel-instances->derivation): Accept and pass on
>  #:assume-available-builtin-builders.
>
> Change-Id: I315c990de66c6f7dca25a859165a5568abe385ea

I would have hoped we could avoid this, because it’s make tricky code a
bit harder to read, but as you said, it’s a real issue that needs
fixing: on build farms or for anyone using offloading and potentially
not upgrading all the machines in lockstep.

> +                                            #:version proto
> +                                            #$@(if assume-available-builtin-builders
> +                                                   #~(#:assume-available-builtin-builders
> +                                                      '(#$@assume-available-builtin-builders))
> +                                                   '()))

Here (in ‘build-program’), we can assume we’re using the current (guix
store) module, so we can safely pass #:built-in-builders unconditionally.

That’s it!

The series LGTM with these changes.

Thank you! :-)

[bug#71038] [PATCH v3 1/2] guix: store: Enable specifying the builtin builders.

[Christopher Baines]

To open-connection and port->connection.  This overrides the discovered
builtin builders that the daemon says it provides.

This is useful when you want to generate compatible derivations that can be
run with a daemon that potentially doesn't support builtin builders that the
daemon you're using to generate the derivations has.

I'm looking at this in particular because I want to use this in the data
service, since it provides substitutes for derivations, and since these can be
built on other machines, it's useful to control which builtin builders they
depend on.

* guix/store.scm (open-connection, port->connection): Accept
 #:built-in-builders and use this instead of %built-in-builders.

Fixes: <https://issues.guix.gnu.org/67250>.

Change-Id: I45d58ab93b6d276d280552858fc81ebc2b58828a
---
 guix/store.scm | 28 +++++++++++++++++++---------
 1 file changed, 19 insertions(+), 9 deletions(-)

diff --git a/guix/store.scm b/guix/store.scm
index 4070b686cb..cf5848e580 100644
--- a/guix/store.scm
+++ b/guix/store.scm
@@ -566,7 +566,7 @@ (define* (connect-to-daemon uri-or-filename #:key non-blocking?)
 
 (define* (open-connection #:optional (uri (%daemon-socket-uri))
                           #:key port (reserve-space? #t) cpu-affinity
-                          non-blocking?)
+                          non-blocking? built-in-builders)
   "Connect to the daemon at URI (a string), or, if PORT is not #f, use it as
 the I/O port over which to communicate to a build daemon.
 
@@ -575,8 +575,10 @@ (define* (open-connection #:optional (uri (%daemon-socket-uri))
 should the disk become full.  When CPU-AFFINITY is true, it must be an integer
 corresponding to an OS-level CPU number to which the daemon's worker process
 for this connection will be pinned.  If NON-BLOCKING?, use a non-blocking
-socket when using the file, unix or guix URI schemes.  Return a server
-object."
+socket when using the file, unix or guix URI schemes.  If
+BUILT-IN-BUILDERS is provided, it should be a list of strings
+and this will be used instead of the builtin builders provided by the build
+daemon.  Return a server object."
   (define (handshake-error)
     (raise (condition
             (&store-connection-error (file (or port uri))
@@ -610,8 +612,10 @@ (define* (open-connection #:optional (uri (%daemon-socket-uri))
               (write-int cpu-affinity port)))
           (when (>= (protocol-minor v) 11)
             (write-int (if reserve-space? 1 0) port))
-          (letrec* ((built-in-builders
-                     (delay (%built-in-builders conn)))
+          (letrec* ((actual-built-in-builders
+                     (if built-in-builders
+                         (delay built-in-builders)
+                         (delay (%built-in-builders conn))))
                     (caches
                      (make-vector
                       (atomic-box-ref %store-connection-caches)
@@ -624,15 +628,19 @@ (define* (open-connection #:optional (uri (%daemon-socket-uri))
                                              (make-hash-table 100)
                                              (make-hash-table 100)
                                              caches
-                                             built-in-builders)))
+                                             actual-built-in-builders)))
             (let loop ((done? (process-stderr conn)))
               (or done? (process-stderr conn)))
             conn))))))
 
 (define* (port->connection port
-                           #:key (version %protocol-version))
+                           #:key (version %protocol-version)
+                           built-in-builders)
   "Assimilate PORT, an input/output port, and return a connection to the
-daemon, assuming the given protocol VERSION.
+daemon, assuming the given protocol VERSION.  If
+BUILT-IN-BUILDERS is provided, it should be a list of strings
+and this will be used instead of the builtin builders provided by the build
+daemon.
 
 Warning: this procedure assumes that the initial handshake with the daemon has
 already taken place on PORT and that we're just continuing on this established
@@ -649,7 +657,9 @@ (define* (port->connection port
                               (make-vector
                                (atomic-box-ref %store-connection-caches)
                                vlist-null)
-                              (delay (%built-in-builders connection))))
+                              (if built-in-builders
+                                  (delay built-in-builders)
+                                  (delay (%built-in-builders connection)))))
 
     connection))
 

base-commit: 5f1e4e4c0242af6bcba656aedf8b49afbe7247b7
-- 
2.45.2

[bug#71038] [PATCH v3 2/2] guix: channels: Enable specifiying available builtin builders.

[Christopher Baines]

When computing channel instance derivations.

This is useful when you want to generate compatible derivations that can be
run with a daemon that potentially doesn't support builtin builders that the
daemon you're using to generate the derivations has.

I'm looking at this in particular because I want to use this in the data
service, since it provides substitutes for derivations, and since these can be
built on other machines, it's useful to control which builtin builders they
depend on.

Fixes: <https://issues.guix.gnu.org/67250>.

* build-aux/build-self.scm (build-program): Accept
 #:built-in-builders and pass along to port->connection or
open-connection as approriate.
(build): Accept and pass on #:built-in-builders.
* guix/channels.scm (build-from-source, build-channel-instance,
channel-instance-derivations, channel-instances->manifest,
channel-instances->derivation): Accept and pass on
 #:built-in-builders.

Change-Id: I315c990de66c6f7dca25a859165a5568abe385ea
---
 build-aux/build-self.scm | 29 +++++++++++------
 guix/channels.scm        | 67 ++++++++++++++++++++++++++++++----------
 2 files changed, 70 insertions(+), 26 deletions(-)

diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index 02822a2ee8..d9299b9af2 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -241,8 +241,12 @@ (define guile-gcrypt
 
 (define* (build-program source version
                         #:optional (guile-version (effective-version))
-                        #:key (pull-version 0) (channel-metadata #f))
-  "Return a program that computes the derivation to build Guix from SOURCE."
+                        #:key (pull-version 0) (channel-metadata #f)
+                        built-in-builders)
+  "Return a program that computes the derivation to build Guix from SOURCE.
+If BUILT-IN-BUILDERS is provided, it should be a list of
+strings and this will be used instead of the builtin builders provided by the
+build daemon, from within the generated build program."
   (define select?
     ;; Select every module but (guix config) and non-Guix modules.
     ;; Also exclude (guix channels): it is autoloaded by (guix describe), but
@@ -331,11 +335,16 @@ (define* (build-program source version
                          ;; case, attempt to open a new connection.
                          (let* ((proto (string->number protocol-version))
                                 (store (if (integer? proto)
-                                           (port->connection (duplicate-port
-                                                              (current-input-port)
-                                                              "w+0")
-                                                             #:version proto)
-                                           (open-connection)))
+                                           (port->connection
+                                            (duplicate-port
+                                             (current-input-port)
+                                             "w+0")
+                                            #:version proto
+                                            #:built-in-builders
+                                            '#$built-in-builders)
+                                           (open-connection
+                                            #:built-in-builders
+                                            '#$built-in-builders)))
                                 (sock  (socket AF_UNIX SOCK_STREAM 0)))
                            ;; Connect to BUILD-OUTPUT and send it the raw
                            ;; build output.
@@ -406,7 +415,7 @@ (define* (build source
                 (guile-version (if (> pull-version 0)
                                    "3.0"
                                    (effective-version)))
-
+                built-in-builders
                 #:allow-other-keys
                 #:rest rest)
   "Return a derivation that unpacks SOURCE into STORE and compiles Scheme
@@ -415,7 +424,9 @@ (define* (build source
   ;; SOURCE.
   (mlet %store-monad ((build  (build-program source version guile-version
                                              #:channel-metadata channel-metadata
-                                             #:pull-version pull-version))
+                                             #:pull-version pull-version
+                                             #:built-in-builders
+                                             built-in-builders))
                       (system (if system (return system) (current-system)))
                       (home -> (getenv "HOME"))
 
diff --git a/guix/channels.scm b/guix/channels.scm
index 0d7bc541cc..34f63eb833 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -704,11 +704,15 @@ (define (with-trivial-build-handler mvalue)
               store))))
 
 (define* (build-from-source instance
-                            #:key core verbose? (dependencies '()) system)
+                            #:key core verbose? (dependencies '()) system
+                            built-in-builders)
   "Return a derivation to build Guix from INSTANCE, using the self-build
 script contained therein.  When CORE is true, build package modules under
 SOURCE using CORE, an instance of Guix.  By default, build for the current
-system, or SYSTEM if specified."
+system, or SYSTEM if specified.  If BUILT-IN-BUILDERS is
+provided, it should be a list of strings and this will be used instead of the
+builtin builders provided by the build daemon for store connections used
+during this process."
   (define name
     (symbol->string
      (channel-name (channel-instance-channel instance))))
@@ -750,20 +754,28 @@ (define* (build-from-source instance
                   #:verbose? verbose? #:version commit
                   #:system system
                   #:channel-metadata (channel-instance->sexp instance)
-                  #:pull-version %pull-version))))
+                  #:pull-version %pull-version
+                  #:built-in-builders
+                  built-in-builders))))
 
       ;; Build a set of modules that extend Guix using the standard method.
       (standard-module-derivation name source core dependencies)))
 
 (define* (build-channel-instance instance system
-                                 #:optional core (dependencies '()))
+                                 #:optional core (dependencies '())
+                                 #:key built-in-builders)
   "Return, as a monadic value, the derivation for INSTANCE, a channel
 instance, for SYSTEM.  DEPENDENCIES is a list of extensions providing Guile
-modules that INSTANCE depends on."
+modules that INSTANCE depends on.  If BUILT-IN-BUILDERS is
+provided, it should be a list of strings and this will be used instead of the
+builtin builders provided by the build daemon for store connections used
+during this process."
   (build-from-source instance
                      #:core core
                      #:dependencies dependencies
-                     #:system system))
+                     #:system system
+                     #:built-in-builders
+                     built-in-builders))
 
 (define (resolve-dependencies instances)
   "Return a procedure that, given one of the elements of INSTANCES, returns
@@ -793,9 +805,13 @@ (define (resolve-dependencies instances)
   (lambda (instance)
     (vhash-foldq* cons '() instance edges)))
 
-(define* (channel-instance-derivations instances #:key system)
+(define* (channel-instance-derivations instances #:key system
+                                       built-in-builders)
   "Return the list of derivations to build INSTANCES, in the same order as
-INSTANCES.  Build for the current system by default, or SYSTEM if specified."
+INSTANCES.  Build for the current system by default, or SYSTEM if specified.
+If BUILT-IN-BUILDERS is provided, it should be a list of
+strings and this will be used instead of the builtin builders provided by the
+build daemon for store connections used during this process."
   (define core-instance
     ;; The 'guix' channel is treated specially: it's an implicit dependency of
     ;; all the other channels.
@@ -809,11 +825,15 @@ (define* (channel-instance-derivations instances #:key system)
   (define (instance->derivation instance)
     (mlet %store-monad ((system (if system (return system) (current-system))))
       (mcached (if (eq? instance core-instance)
-                   (build-channel-instance instance system)
+                   (build-channel-instance instance system
+                                           #:built-in-builders
+                                           built-in-builders)
                    (mlet %store-monad ((core (instance->derivation core-instance))
                                        (deps (mapm %store-monad instance->derivation
                                                    (edges instance))))
-                     (build-channel-instance instance system core deps)))
+                     (build-channel-instance instance system core deps
+                                             #:built-in-builders
+                                             built-in-builders)))
                instance
                system)))
 
@@ -915,10 +935,13 @@ (define (channel-instance->sexp instance)
                     intro))))))
             '()))))
 
-(define* (channel-instances->manifest instances #:key system)
+(define* (channel-instances->manifest instances #:key system
+                                      built-in-builders)
   "Return a profile manifest with entries for all of INSTANCES, a list of
 channel instances.  By default, build for the current system, or SYSTEM if
-specified."
+specified.  If BUILT-IN-BUILDERS is provided, it should be a
+list of strings and this will be used instead of the builtin builders provided
+by the build daemon for store connections used during this process."
   (define (instance->entry instance drv)
     (let ((commit  (channel-instance-commit instance))
           (channel (channel-instance-channel instance)))
@@ -934,8 +957,11 @@ (define* (channel-instances->manifest instances #:key system)
         (properties
          `((source ,(channel-instance->sexp instance)))))))
 
-  (mlet* %store-monad ((derivations (channel-instance-derivations instances
-                                                                  #:system system))
+  (mlet* %store-monad ((derivations (channel-instance-derivations
+                                     instances
+                                     #:system system
+                                     #:built-in-builders
+                                     built-in-builders))
                        (entries ->  (map instance->entry instances derivations)))
     (return (manifest entries))))
 
@@ -990,10 +1016,17 @@ (define %channel-profile-hooks
   ;; The default channel profile hooks.
   (cons package-cache-file %default-profile-hooks))
 
-(define (channel-instances->derivation instances)
+(define* (channel-instances->derivation instances
+                                        #:key built-in-builders)
   "Return the derivation of the profile containing INSTANCES, a list of
-channel instances."
-  (mlet %store-monad ((manifest (channel-instances->manifest instances)))
+channel instances.  If BUILT-IN-BUILDERS is provided, it
+should be a list of strings and this will be used instead of the builtin
+builders provided by the build daemon for store connections used during this
+process."
+  (mlet %store-monad ((manifest (channel-instances->manifest
+                                 instances
+                                 #:built-in-builders
+                                 built-in-builders)))
     ;; Emit a profile in format version so that, if INSTANCES denotes an old
     ;; Guix, it can still read that profile, for instance for the purposes of
     ;; 'guix describe'.
-- 
2.45.2

[bug#71038] [PATCH v3 2/2] guix: channels: Enable specifiying available builtin builders.

[Ludovic Courtès]

Hello!

v3 LGTM, thanks!

Ludo’.

[bug#71038] [PATCH v4 1/3] guix: store: Enable specifying the builtin builders.

[Christopher Baines]

To open-connection and port->connection.  This overrides the discovered
builtin builders that the daemon says it provides.

This is useful when you want to generate compatible derivations that can be
run with a daemon that potentially doesn't support builtin builders that the
daemon you're using to generate the derivations has.

I'm looking at this in particular because I want to use this in the data
service, since it provides substitutes for derivations, and since these can be
built on other machines, it's useful to control which builtin builders they
depend on.

* guix/store.scm (open-connection, port->connection): Accept
 #:built-in-builders and use this instead of %built-in-builders.

Fixes: <https://issues.guix.gnu.org/67250>.

Change-Id: I45d58ab93b6d276d280552858fc81ebc2b58828a
---
 guix/store.scm | 28 +++++++++++++++++++---------
 1 file changed, 19 insertions(+), 9 deletions(-)

diff --git a/guix/store.scm b/guix/store.scm
index 4070b686cb..cf5848e580 100644
--- a/guix/store.scm
+++ b/guix/store.scm
@@ -566,7 +566,7 @@ (define* (connect-to-daemon uri-or-filename #:key non-blocking?)
 
 (define* (open-connection #:optional (uri (%daemon-socket-uri))
                           #:key port (reserve-space? #t) cpu-affinity
-                          non-blocking?)
+                          non-blocking? built-in-builders)
   "Connect to the daemon at URI (a string), or, if PORT is not #f, use it as
 the I/O port over which to communicate to a build daemon.
 
@@ -575,8 +575,10 @@ (define* (open-connection #:optional (uri (%daemon-socket-uri))
 should the disk become full.  When CPU-AFFINITY is true, it must be an integer
 corresponding to an OS-level CPU number to which the daemon's worker process
 for this connection will be pinned.  If NON-BLOCKING?, use a non-blocking
-socket when using the file, unix or guix URI schemes.  Return a server
-object."
+socket when using the file, unix or guix URI schemes.  If
+BUILT-IN-BUILDERS is provided, it should be a list of strings
+and this will be used instead of the builtin builders provided by the build
+daemon.  Return a server object."
   (define (handshake-error)
     (raise (condition
             (&store-connection-error (file (or port uri))
@@ -610,8 +612,10 @@ (define* (open-connection #:optional (uri (%daemon-socket-uri))
               (write-int cpu-affinity port)))
           (when (>= (protocol-minor v) 11)
             (write-int (if reserve-space? 1 0) port))
-          (letrec* ((built-in-builders
-                     (delay (%built-in-builders conn)))
+          (letrec* ((actual-built-in-builders
+                     (if built-in-builders
+                         (delay built-in-builders)
+                         (delay (%built-in-builders conn))))
                     (caches
                      (make-vector
                       (atomic-box-ref %store-connection-caches)
@@ -624,15 +628,19 @@ (define* (open-connection #:optional (uri (%daemon-socket-uri))
                                              (make-hash-table 100)
                                              (make-hash-table 100)
                                              caches
-                                             built-in-builders)))
+                                             actual-built-in-builders)))
             (let loop ((done? (process-stderr conn)))
               (or done? (process-stderr conn)))
             conn))))))
 
 (define* (port->connection port
-                           #:key (version %protocol-version))
+                           #:key (version %protocol-version)
+                           built-in-builders)
   "Assimilate PORT, an input/output port, and return a connection to the
-daemon, assuming the given protocol VERSION.
+daemon, assuming the given protocol VERSION.  If
+BUILT-IN-BUILDERS is provided, it should be a list of strings
+and this will be used instead of the builtin builders provided by the build
+daemon.
 
 Warning: this procedure assumes that the initial handshake with the daemon has
 already taken place on PORT and that we're just continuing on this established
@@ -649,7 +657,9 @@ (define* (port->connection port
                               (make-vector
                                (atomic-box-ref %store-connection-caches)
                                vlist-null)
-                              (delay (%built-in-builders connection))))
+                              (if built-in-builders
+                                  (delay built-in-builders)
+                                  (delay (%built-in-builders connection)))))
 
     connection))
 

base-commit: bf6ab0e0f5066d999e027a7eb8ecf05db71123ce
-- 
2.45.2

[bug#71038] [PATCH v4 2/3] guix: channels: Enable specifiying available builtin builders.

[Christopher Baines]

When computing channel instance derivations.

This is useful when you want to generate compatible derivations that can be
run with a daemon that potentially doesn't support builtin builders that the
daemon you're using to generate the derivations has.

I'm looking at this in particular because I want to use this in the data
service, since it provides substitutes for derivations, and since these can be
built on other machines, it's useful to control which builtin builders they
depend on.

Fixes: <https://issues.guix.gnu.org/67250>.

* build-aux/build-self.scm (build-program): Accept
 #:built-in-builders and pass along to port->connection or
open-connection as approriate.
(build): Accept and pass on #:built-in-builders.
* guix/channels.scm (build-from-source, build-channel-instance,
channel-instance-derivations, channel-instances->manifest,
channel-instances->derivation): Accept and pass on
 #:built-in-builders.

Change-Id: I315c990de66c6f7dca25a859165a5568abe385ea
---
 build-aux/build-self.scm | 29 +++++++++++------
 guix/channels.scm        | 67 ++++++++++++++++++++++++++++++----------
 2 files changed, 70 insertions(+), 26 deletions(-)

diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index 02822a2ee8..d9299b9af2 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -241,8 +241,12 @@ (define guile-gcrypt
 
 (define* (build-program source version
                         #:optional (guile-version (effective-version))
-                        #:key (pull-version 0) (channel-metadata #f))
-  "Return a program that computes the derivation to build Guix from SOURCE."
+                        #:key (pull-version 0) (channel-metadata #f)
+                        built-in-builders)
+  "Return a program that computes the derivation to build Guix from SOURCE.
+If BUILT-IN-BUILDERS is provided, it should be a list of
+strings and this will be used instead of the builtin builders provided by the
+build daemon, from within the generated build program."
   (define select?
     ;; Select every module but (guix config) and non-Guix modules.
     ;; Also exclude (guix channels): it is autoloaded by (guix describe), but
@@ -331,11 +335,16 @@ (define* (build-program source version
                          ;; case, attempt to open a new connection.
                          (let* ((proto (string->number protocol-version))
                                 (store (if (integer? proto)
-                                           (port->connection (duplicate-port
-                                                              (current-input-port)
-                                                              "w+0")
-                                                             #:version proto)
-                                           (open-connection)))
+                                           (port->connection
+                                            (duplicate-port
+                                             (current-input-port)
+                                             "w+0")
+                                            #:version proto
+                                            #:built-in-builders
+                                            '#$built-in-builders)
+                                           (open-connection
+                                            #:built-in-builders
+                                            '#$built-in-builders)))
                                 (sock  (socket AF_UNIX SOCK_STREAM 0)))
                            ;; Connect to BUILD-OUTPUT and send it the raw
                            ;; build output.
@@ -406,7 +415,7 @@ (define* (build source
                 (guile-version (if (> pull-version 0)
                                    "3.0"
                                    (effective-version)))
-
+                built-in-builders
                 #:allow-other-keys
                 #:rest rest)
   "Return a derivation that unpacks SOURCE into STORE and compiles Scheme
@@ -415,7 +424,9 @@ (define* (build source
   ;; SOURCE.
   (mlet %store-monad ((build  (build-program source version guile-version
                                              #:channel-metadata channel-metadata
-                                             #:pull-version pull-version))
+                                             #:pull-version pull-version
+                                             #:built-in-builders
+                                             built-in-builders))
                       (system (if system (return system) (current-system)))
                       (home -> (getenv "HOME"))
 
diff --git a/guix/channels.scm b/guix/channels.scm
index 0d7bc541cc..34f63eb833 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -704,11 +704,15 @@ (define (with-trivial-build-handler mvalue)
               store))))
 
 (define* (build-from-source instance
-                            #:key core verbose? (dependencies '()) system)
+                            #:key core verbose? (dependencies '()) system
+                            built-in-builders)
   "Return a derivation to build Guix from INSTANCE, using the self-build
 script contained therein.  When CORE is true, build package modules under
 SOURCE using CORE, an instance of Guix.  By default, build for the current
-system, or SYSTEM if specified."
+system, or SYSTEM if specified.  If BUILT-IN-BUILDERS is
+provided, it should be a list of strings and this will be used instead of the
+builtin builders provided by the build daemon for store connections used
+during this process."
   (define name
     (symbol->string
      (channel-name (channel-instance-channel instance))))
@@ -750,20 +754,28 @@ (define* (build-from-source instance
                   #:verbose? verbose? #:version commit
                   #:system system
                   #:channel-metadata (channel-instance->sexp instance)
-                  #:pull-version %pull-version))))
+                  #:pull-version %pull-version
+                  #:built-in-builders
+                  built-in-builders))))
 
       ;; Build a set of modules that extend Guix using the standard method.
       (standard-module-derivation name source core dependencies)))
 
 (define* (build-channel-instance instance system
-                                 #:optional core (dependencies '()))
+                                 #:optional core (dependencies '())
+                                 #:key built-in-builders)
   "Return, as a monadic value, the derivation for INSTANCE, a channel
 instance, for SYSTEM.  DEPENDENCIES is a list of extensions providing Guile
-modules that INSTANCE depends on."
+modules that INSTANCE depends on.  If BUILT-IN-BUILDERS is
+provided, it should be a list of strings and this will be used instead of the
+builtin builders provided by the build daemon for store connections used
+during this process."
   (build-from-source instance
                      #:core core
                      #:dependencies dependencies
-                     #:system system))
+                     #:system system
+                     #:built-in-builders
+                     built-in-builders))
 
 (define (resolve-dependencies instances)
   "Return a procedure that, given one of the elements of INSTANCES, returns
@@ -793,9 +805,13 @@ (define (resolve-dependencies instances)
   (lambda (instance)
     (vhash-foldq* cons '() instance edges)))
 
-(define* (channel-instance-derivations instances #:key system)
+(define* (channel-instance-derivations instances #:key system
+                                       built-in-builders)
   "Return the list of derivations to build INSTANCES, in the same order as
-INSTANCES.  Build for the current system by default, or SYSTEM if specified."
+INSTANCES.  Build for the current system by default, or SYSTEM if specified.
+If BUILT-IN-BUILDERS is provided, it should be a list of
+strings and this will be used instead of the builtin builders provided by the
+build daemon for store connections used during this process."
   (define core-instance
     ;; The 'guix' channel is treated specially: it's an implicit dependency of
     ;; all the other channels.
@@ -809,11 +825,15 @@ (define* (channel-instance-derivations instances #:key system)
   (define (instance->derivation instance)
     (mlet %store-monad ((system (if system (return system) (current-system))))
       (mcached (if (eq? instance core-instance)
-                   (build-channel-instance instance system)
+                   (build-channel-instance instance system
+                                           #:built-in-builders
+                                           built-in-builders)
                    (mlet %store-monad ((core (instance->derivation core-instance))
                                        (deps (mapm %store-monad instance->derivation
                                                    (edges instance))))
-                     (build-channel-instance instance system core deps)))
+                     (build-channel-instance instance system core deps
+                                             #:built-in-builders
+                                             built-in-builders)))
                instance
                system)))
 
@@ -915,10 +935,13 @@ (define (channel-instance->sexp instance)
                     intro))))))
             '()))))
 
-(define* (channel-instances->manifest instances #:key system)
+(define* (channel-instances->manifest instances #:key system
+                                      built-in-builders)
   "Return a profile manifest with entries for all of INSTANCES, a list of
 channel instances.  By default, build for the current system, or SYSTEM if
-specified."
+specified.  If BUILT-IN-BUILDERS is provided, it should be a
+list of strings and this will be used instead of the builtin builders provided
+by the build daemon for store connections used during this process."
   (define (instance->entry instance drv)
     (let ((commit  (channel-instance-commit instance))
           (channel (channel-instance-channel instance)))
@@ -934,8 +957,11 @@ (define* (channel-instances->manifest instances #:key system)
         (properties
          `((source ,(channel-instance->sexp instance)))))))
 
-  (mlet* %store-monad ((derivations (channel-instance-derivations instances
-                                                                  #:system system))
+  (mlet* %store-monad ((derivations (channel-instance-derivations
+                                     instances
+                                     #:system system
+                                     #:built-in-builders
+                                     built-in-builders))
                        (entries ->  (map instance->entry instances derivations)))
     (return (manifest entries))))
 
@@ -990,10 +1016,17 @@ (define %channel-profile-hooks
   ;; The default channel profile hooks.
   (cons package-cache-file %default-profile-hooks))
 
-(define (channel-instances->derivation instances)
+(define* (channel-instances->derivation instances
+                                        #:key built-in-builders)
   "Return the derivation of the profile containing INSTANCES, a list of
-channel instances."
-  (mlet %store-monad ((manifest (channel-instances->manifest instances)))
+channel instances.  If BUILT-IN-BUILDERS is provided, it
+should be a list of strings and this will be used instead of the builtin
+builders provided by the build daemon for store connections used during this
+process."
+  (mlet %store-monad ((manifest (channel-instances->manifest
+                                 instances
+                                 #:built-in-builders
+                                 built-in-builders)))
     ;; Emit a profile in format version so that, if INSTANCES denotes an old
     ;; Guix, it can still read that profile, for instance for the purposes of
     ;; 'guix describe'.
-- 
2.45.2

[bug#71038] [PATCH v4 3/3] inferior: Use the host built-in-builders with inferior.

[Christopher Baines]

Rather than querying the built-in-builders from the inferior, as using the
host value allows specifying it when opening the connection.

* guix/inferior.scm (port->inferior): Have cached-store-connection take the
built-in-builders.
(inferior-eval-with-store): Call cached-store-connection with the store
connection built-in-builders.

Change-Id: I27c20732355c0c6aa646748a02df39db302cd568
---
 guix/inferior.scm | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)

diff --git a/guix/inferior.scm b/guix/inferior.scm
index 190ba01b3c..b60bf1ab01 100644
--- a/guix/inferior.scm
+++ b/guix/inferior.scm
@@ -253,7 +253,8 @@ (define* (port->inferior pipe #:optional (close close-port))
                       result)
        (inferior-eval '(begin
                          (define %store-table (make-hash-table))
-                         (define (cached-store-connection store-id version)
+                         (define (cached-store-connection store-id version
+                                                          built-in-builders)
                            ;; Cache connections to store ID.  This ensures that
                            ;; the caches within <store-connection> (in
                            ;; particular the object cache) are reused across
@@ -268,9 +269,19 @@ (define* (port->inferior pipe #:optional (close close-port))
                                ;; risk of talking to the wrong daemon or having
                                ;; our build result reclaimed (XXX).
                                (let ((store (if (defined? 'port->connection)
-                                                (port->connection %bridge-socket
-                                                                  #:version
-                                                                  version)
+                                                ;; #:built-in-builders was
+                                                ;; added in 2024
+                                                (catch 'keyword-argument-error
+                                                  (lambda ()
+                                                    (port->connection %bridge-socket
+                                                                      #:version
+                                                                      version
+                                                                      #:built-in-builders
+                                                                      built-in-builders))
+                                                  (lambda _
+                                                    (port->connection %bridge-socket
+                                                                      #:version
+                                                                      version)))
                                                 (open-connection))))
                                  (hashv-set! %store-table store-id store)
                                  store))))
@@ -690,11 +701,13 @@ (define (inferior-eval-with-store inferior store code)
          ;; The address of STORE itself is not a good identifier because it
          ;; keeps changing through the use of "functional caches".  The
          ;; address of its socket port makes more sense.
-         (store-id (object-address (store-connection-socket store))))
+         (store-id (object-address (store-connection-socket store)))
+         (store-built-in-builders (built-in-builders store)))
     (ensure-store-bridge! inferior)
     (send-inferior-request
      `(let ((proc  ,code)
-            (store (cached-store-connection ,store-id ,proto)))
+            (store (cached-store-connection ,store-id ,proto
+                                            ',store-built-in-builders)))
         ;; Serialize '&store-protocol-error' conditions.  The exception
         ;; serialization mechanism that 'read-repl-response' expects is
         ;; unsuitable for SRFI-35 error conditions, hence this special case.
-- 
2.45.2

[bug#71038] [PATCH v3 2/2] guix: channels: Enable specifiying available builtin builders.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello!
>
> v3 LGTM, thanks!

I was going to push last night, but then I realised that this doesn't
actually go far enough for the data service as the built-in-builders
value needs passing through to port->connection when called in
port->inferior.

I've sent a v4 now which adds an additional commit to address this.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

[bug#71038] [PATCH v4 3/3] inferior: Use the host built-in-builders with inferior.

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> Rather than querying the built-in-builders from the inferior, as using the
> host value allows specifying it when opening the connection.
>
> * guix/inferior.scm (port->inferior): Have cached-store-connection take the
> built-in-builders.
> (inferior-eval-with-store): Call cached-store-connection with the store
> connection built-in-builders.
>
> Change-Id: I27c20732355c0c6aa646748a02df39db302cd568

LGTM as well!

Thanks,
Ludo'.

bug#71038: [PATCH v4 3/3] inferior: Use the host built-in-builders with inferior.

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Christopher Baines <mail@cbaines.net> skribis:
>
>> Rather than querying the built-in-builders from the inferior, as using the
>> host value allows specifying it when opening the connection.
>>
>> * guix/inferior.scm (port->inferior): Have cached-store-connection take the
>> built-in-builders.
>> (inferior-eval-with-store): Call cached-store-connection with the store
>> connection built-in-builders.
>>
>> Change-Id: I27c20732355c0c6aa646748a02df39db302cd568
>
> LGTM as well!

Thanks for taking a look, I've pushed this to master as
f3e17f9ff1bf77b4ebddf0de3340e77e2ec8a830.

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]