all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Expensive builds when computing channel instance derivations
@ 2022-02-20 12:06 Christopher Baines
  2022-02-28 13:24 ` Ludovic Courtès
  0 siblings, 1 reply; 4+ messages in thread
From: Christopher Baines @ 2022-02-20 12:06 UTC (permalink / raw)
  To: guix-devel


[-- Attachment #1.1: Type: text/plain, Size: 2627 bytes --]

Hey,

Back in early 2021, I was trying to address the issues the Guix Data
Service has when trying to compute channel instance derivations for
various systems [1].

1: https://issues.guix.gnu.org/47989

Some changes did some out of that, and I believe the helped, but even
with those changes, being able to build things for the system you want
to compute the channel instance derivation for seemed to remain
necessary.

This poses an operational issue for things like data.guix.gnu.org, since
it can only compute channel instance derivations for systems it can
build for, which in practice means that it's limited by the systems
which QEMU emulation is enabled for. Even for those systems it can build
for, because builds can happen when attempting to compute these
derivations, it means that data.guix.gnu.org spends a lot of time
waiting for builds to complete, just so it can store these derivations.

I have a suspicion that this issue is a big contributor to the
data.guix.gnu.org slowness when processing new revisions, so I tried to
dig in to it more recently as I didn't know why these builds were
happening.

I think I figured out that it's related to grafts. I've attached a test
script which computes the channel instance derivation for mips64el-linux
which I'm not set up to build things for (if you can build for
mips64el-linux, replace this with a system you can't build for). You'll
need to use the attached patch (also present on [2]) when running this
script.

2: https://git.cbaines.net/guix/log/?h=channel-instances-manifest-graft-control

When I run this script locally, it first succeeds in computing the
channel instance derivation when grafts are disabled, but then fails
when grafts are enabled:

  while setting up the build environment: a `mips64el-linux' is required
  to build
  `/gnu/store/g40shyhsd00r5dq3mm76c2b1krnr1njh-guile-bootstrap-2.0.drv',
  but I am a `x86_64-linux'

Even though I think this shows that grafts are involved, I'm not sure
what this means? I'm guessing that the effects of grafts aren't as clear
cut as for packages here, since the grafts are happening here as part of
computing a derivation I'm guessing that the derivation is actually
built with the grafted outputs, which differs from how grafts affect
packages. Given this, I guess computing the derivation without grafts
means that the replacements that would be grafted in are just not used
at all.

To recap on my aim here, I really just want to be able to compute
channel instance derivations without performing any expensive builds, or
if that's not possible, it would be good to understand why?

Thanks,

Chris



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-channels-Allow-disabling-grafts-when-computing-deriv.patch --]
[-- Type: text/x-patch, Size: 7881 bytes --]

From 38a12f0f22841b76050a0cf5163cdc65b7f92193 Mon Sep 17 00:00:00 2001
From: Christopher Baines <mail@cbaines.net>
Date: Fri, 18 Feb 2022 23:06:57 +0000
Subject: [PATCH] channels: Allow disabling grafts when computing derivations.

---
 build-aux/build-self.scm | 23 +++++++++++++++--------
 guix/channels.scm        | 19 +++++++++++--------
 2 files changed, 26 insertions(+), 16 deletions(-)

diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm
index 02822a2ee8..0e7fc2907d 100644
--- a/build-aux/build-self.scm
+++ b/build-aux/build-self.scm
@@ -241,7 +241,8 @@ (define guile-gcrypt
 
 (define* (build-program source version
                         #:optional (guile-version (effective-version))
-                        #:key (pull-version 0) (channel-metadata #f))
+                        #:key (pull-version 0) (channel-metadata #f)
+                        (graft? #t))
   "Return a program that computes the derivation to build Guix from SOURCE."
   (define select?
     ;; Select every module but (guix config) and non-Guix modules.
@@ -316,6 +317,8 @@ (define fake-git
                         (read-disable 'positions))
 
                       (use-modules (guix store)
+                                   (guix grafts)
+                                   (guix monads)
                                    (guix self)
                                    (guix derivations)
                                    (srfi srfi-1))
@@ -348,12 +351,14 @@ (define fake-git
                                              (%make-void-port "w"))
                                             (current-build-output-port sock))
                                (run-with-store store
-                                 (guix-derivation source version
-                                                  #$guile-version
-                                                  #:channel-metadata
-                                                  '#$channel-metadata
-                                                  #:pull-version
-                                                  #$pull-version)
+                                 (mbegin %store-monad
+                                   (set-grafting #$graft?)
+                                   (guix-derivation source version
+                                                    #$guile-version
+                                                    #:channel-metadata
+                                                    '#$channel-metadata
+                                                    #:pull-version
+                                                    #$pull-version))
                                  #:system system))
                              derivation-file-name))))))
                   #:module-path (list source))))
@@ -398,6 +403,7 @@ (define* (build source
                 #:key verbose?
                 (version (date-version-string)) channel-metadata
                 system
+                (graft? #t)
                 (pull-version 0)
 
                 ;; For the standalone Guix, default to Guile 3.0.  For old
@@ -415,7 +421,8 @@ (define* (build source
   ;; SOURCE.
   (mlet %store-monad ((build  (build-program source version guile-version
                                              #:channel-metadata channel-metadata
-                                             #:pull-version pull-version))
+                                             #:pull-version pull-version
+                                             #:graft? graft?))
                       (system (if system (return system) (current-system)))
                       (home -> (getenv "HOME"))
 
diff --git a/guix/channels.scm b/guix/channels.scm
index 5f47834c10..3aba677534 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -658,7 +658,7 @@ (define (with-trivial-build-handler mvalue)
               store))))
 
 (define* (build-from-source instance
-                            #:key core verbose? (dependencies '()) system)
+                            #:key core verbose? (dependencies '()) system graft?)
   "Return a derivation to build Guix from INSTANCE, using the self-build
 script contained therein.  When CORE is true, build package modules under
 SOURCE using CORE, an instance of Guix.  By default, build for the current
@@ -703,13 +703,14 @@ (define script
            (build source
                   #:verbose? verbose? #:version commit
                   #:system system
+                  #:graft? graft?
                   #:channel-metadata (channel-instance->sexp instance)
                   #:pull-version %pull-version))))
 
       ;; Build a set of modules that extend Guix using the standard method.
       (standard-module-derivation name source core dependencies)))
 
-(define* (build-channel-instance instance system
+(define* (build-channel-instance instance system graft?
                                  #:optional core (dependencies '()))
   "Return, as a monadic value, the derivation for INSTANCE, a channel
 instance, for SYSTEM.  DEPENDENCIES is a list of extensions providing Guile
@@ -717,7 +718,8 @@ (define* (build-channel-instance instance system
   (build-from-source instance
                      #:core core
                      #:dependencies dependencies
-                     #:system system))
+                     #:system system
+                     #:graft? graft?))
 
 (define (resolve-dependencies instances)
   "Return a procedure that, given one of the elements of INSTANCES, returns
@@ -747,7 +749,7 @@ (define edges
   (lambda (instance)
     (vhash-foldq* cons '() instance edges)))
 
-(define* (channel-instance-derivations instances #:key system)
+(define* (channel-instance-derivations instances #:key system graft?)
   "Return the list of derivations to build INSTANCES, in the same order as
 INSTANCES.  Build for the current system by default, or SYSTEM if specified."
   (define core-instance
@@ -763,11 +765,11 @@ (define edges
   (define (instance->derivation instance)
     (mlet %store-monad ((system (if system (return system) (current-system))))
       (mcached (if (eq? instance core-instance)
-                   (build-channel-instance instance system)
+                   (build-channel-instance instance system graft?)
                    (mlet %store-monad ((core (instance->derivation core-instance))
                                        (deps (mapm %store-monad instance->derivation
                                                    (edges instance))))
-                     (build-channel-instance instance system core deps)))
+                     (build-channel-instance instance system graft? core deps)))
                instance
                system)))
 
@@ -869,7 +871,7 @@ (define (channel-instance->sexp instance)
                     intro))))))
             '()))))
 
-(define* (channel-instances->manifest instances #:key system)
+(define* (channel-instances->manifest instances #:key system (graft? #t))
   "Return a profile manifest with entries for all of INSTANCES, a list of
 channel instances.  By default, build for the current system, or SYSTEM if
 specified."
@@ -889,7 +891,8 @@ (define (instance->entry instance drv)
          `((source ,(channel-instance->sexp instance)))))))
 
   (mlet* %store-monad ((derivations (channel-instance-derivations instances
-                                                                  #:system system))
+                                                                  #:system system
+                                                                  #:graft? graft?))
                        (entries ->  (map instance->entry instances derivations)))
     (return (manifest entries))))
 
-- 
2.34.0


[-- Attachment #1.3: test script --]
[-- Type: text/plain, Size: 1696 bytes --]

(use-modules (srfi srfi-1)
             (guix store)
             (guix profiles)
             (guix derivations)
             (guix utils)
             (guix status)
             (guix grafts)
             (guix channels))

(define channels
  (list (channel
         (name 'guix)
         (branch "channel-instances-manifest-graft-control")
         (url "https://git.cbaines.net/git/guix.git"))))

(let ((drv
       (with-store store
         (let* ((channel-instances
                 (latest-channel-instances store channels #:authenticate? #f))
                (manifest
                 ((channel-instances->manifest channel-instances
                                               #:system
                                               "mips64el-linux"
                                               #:graft? #f)
                  store)))
           (derivation-file-name
            (manifest-entry-item
             (first
              (manifest-entries manifest))))))))
  (peek "Successfully computed derivation without grafts" drv))

(newline)
(newline)
(peek "Now trying with grafts")
(newline)

(let ((drv
       (with-store store
         (let* ((channel-instances
                 (latest-channel-instances store channels #:authenticate? #f))
                (manifest
                 ((channel-instances->manifest channel-instances
                                               #:system
                                               "mips64el-linux")
                  store)))
           (derivation-file-name
            (manifest-entry-item
             (first
              (manifest-entries manifest))))))))
  (peek "Successfully computed derivation with grafts" drv))

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply related	[flat|nested] 4+ messages in thread
end of thread, other threads:[~2023-02-08  7:55 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-20 12:06 Expensive builds when computing channel instance derivations Christopher Baines
2022-02-28 13:24 ` Ludovic Courtès
2023-01-21 14:55   ` Christopher Baines
2023-02-08  7:42     ` Christopher Baines

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.