From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 0DzJHxY4EmKQMAAAgWs5BA (envelope-from ) for ; Sun, 20 Feb 2022 13:46:14 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id KESjGBY4EmIdegEAG6o9tA (envelope-from ) for ; Sun, 20 Feb 2022 13:46:14 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id ADF3825399 for ; Sun, 20 Feb 2022 13:46:13 +0100 (CET) Received: from localhost ([::1]:56730 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nLlbQ-0000f8-8L for larch@yhetil.org; Sun, 20 Feb 2022 07:46:12 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55122) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nLlX4-00081V-OQ for guix-devel@gnu.org; Sun, 20 Feb 2022 07:41:43 -0500 Received: from [2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27] (port=50407 helo=mira.cbaines.net) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nLlX2-0003Ei-I6 for guix-devel@gnu.org; Sun, 20 Feb 2022 07:41:42 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 0019227BBE9 for ; Sun, 20 Feb 2022 12:41:33 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id bdfdaa9a for ; Sun, 20 Feb 2022 12:41:32 +0000 (UTC) User-agent: mu4e 1.6.10; emacs 27.2 From: Christopher Baines To: guix-devel@gnu.org Subject: Expensive builds when computing channel instance derivations Date: Sun, 20 Feb 2022 12:06:40 +0000 Message-ID: <87h78tbthh.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27 (failed) Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, RDNS_NONE=0.793, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1645361173; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=DWRAnvbK0rTZDIAVrOOEE2mFQwBF8hq+QSYNZRLzFUg=; b=sW8x0iR7GkPRT1l33tXyCkYrQrpKbIYISmDs3SmrxpiDOyrAo1AgU7w/NG+bsd1Af8Rp60 YMbKveS6vSEjzps6RXMQxGCsW2e0pN8nRlRILbJyL/CK/rUJKJr+1oV66OJjFlPdTIO+Q0 r2tj0hHmyXozVwXXWiDm0ibtVolNahAI+bdPfuMUzrTWADvknJdwA/si0Yy8tsdnPGpQ+v lcLmkXXOm7RdahziczCzZtBiEI9pDgylmkOMpWYJ0SSyLVI0UNzZb7LC6VxvUzI/OgpLH+ OpOHA/+//xgXPPMOHf9C6MaaA6Z5LFHVARuKfl4k1aFv1FV96IzzUZPcXfbRSg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1645361173; a=rsa-sha256; cv=none; b=FsiQlRPdOQhFDyeCfjNb1O/OQ9N2a0CPcUtfa9MZylLwwdPjPQh6P+b5+yigx4eIXFa9Qq MvPDMYrj+qOMtkOR6l+lE1/O0vdjtANjcMe5+4xjbuLvhjDTrfclgJ5P0uQZeNNtEmbu1u JUz95638ekKFpKp2tu27ZMDTGfbXWVA8sx6u3zMXtVtOPKojM4NQtkiZ76RareNvH+Qsno fd+I4PHwquiIxdcLeMMS5algHfE9LkskZljEADDxGlaApaSorb6tIDJZok9/771mWPLPpp LtJhBD1FlHk9ZxTdW5XEiTKbHA3pw4+WmqslepEuSsNL+33HwosdV+WsgWNk8g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.23 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: ADF3825399 X-Spam-Score: -5.23 X-Migadu-Scanner: scn1.migadu.com X-TUID: Glbwo0dXmVBO --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain Hey, Back in early 2021, I was trying to address the issues the Guix Data Service has when trying to compute channel instance derivations for various systems [1]. 1: https://issues.guix.gnu.org/47989 Some changes did some out of that, and I believe the helped, but even with those changes, being able to build things for the system you want to compute the channel instance derivation for seemed to remain necessary. This poses an operational issue for things like data.guix.gnu.org, since it can only compute channel instance derivations for systems it can build for, which in practice means that it's limited by the systems which QEMU emulation is enabled for. Even for those systems it can build for, because builds can happen when attempting to compute these derivations, it means that data.guix.gnu.org spends a lot of time waiting for builds to complete, just so it can store these derivations. I have a suspicion that this issue is a big contributor to the data.guix.gnu.org slowness when processing new revisions, so I tried to dig in to it more recently as I didn't know why these builds were happening. I think I figured out that it's related to grafts. I've attached a test script which computes the channel instance derivation for mips64el-linux which I'm not set up to build things for (if you can build for mips64el-linux, replace this with a system you can't build for). You'll need to use the attached patch (also present on [2]) when running this script. 2: https://git.cbaines.net/guix/log/?h=channel-instances-manifest-graft-control When I run this script locally, it first succeeds in computing the channel instance derivation when grafts are disabled, but then fails when grafts are enabled: while setting up the build environment: a `mips64el-linux' is required to build `/gnu/store/g40shyhsd00r5dq3mm76c2b1krnr1njh-guile-bootstrap-2.0.drv', but I am a `x86_64-linux' Even though I think this shows that grafts are involved, I'm not sure what this means? I'm guessing that the effects of grafts aren't as clear cut as for packages here, since the grafts are happening here as part of computing a derivation I'm guessing that the derivation is actually built with the grafted outputs, which differs from how grafts affect packages. Given this, I guess computing the derivation without grafts means that the replacements that would be grafted in are just not used at all. To recap on my aim here, I really just want to be able to compute channel instance derivations without performing any expensive builds, or if that's not possible, it would be good to understand why? Thanks, Chris --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-channels-Allow-disabling-grafts-when-computing-deriv.patch Content-Transfer-Encoding: quoted-printable From=2038a12f0f22841b76050a0cf5163cdc65b7f92193 Mon Sep 17 00:00:00 2001 From: Christopher Baines Date: Fri, 18 Feb 2022 23:06:57 +0000 Subject: [PATCH] channels: Allow disabling grafts when computing derivation= s. =2D-- build-aux/build-self.scm | 23 +++++++++++++++-------- guix/channels.scm | 19 +++++++++++-------- 2 files changed, 26 insertions(+), 16 deletions(-) diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm index 02822a2ee8..0e7fc2907d 100644 =2D-- a/build-aux/build-self.scm +++ b/build-aux/build-self.scm @@ -241,7 +241,8 @@ (define guile-gcrypt =20 (define* (build-program source version #:optional (guile-version (effective-version)) =2D #:key (pull-version 0) (channel-metadata #f)) + #:key (pull-version 0) (channel-metadata #f) + (graft? #t)) "Return a program that computes the derivation to build Guix from SOURCE= ." (define select? ;; Select every module but (guix config) and non-Guix modules. @@ -316,6 +317,8 @@ (define fake-git (read-disable 'positions)) =20 (use-modules (guix store) + (guix grafts) + (guix monads) (guix self) (guix derivations) (srfi srfi-1)) @@ -348,12 +351,14 @@ (define fake-git (%make-void-port "w")) (current-build-output-port soc= k)) (run-with-store store =2D (guix-derivation source version =2D #$guile-version =2D #:channel-metadata =2D '#$channel-metadata =2D #:pull-version =2D #$pull-version) + (mbegin %store-monad + (set-grafting #$graft?) + (guix-derivation source version + #$guile-version + #:channel-metadata + '#$channel-metadata + #:pull-version + #$pull-version)) #:system system)) derivation-file-name)))))) #:module-path (list source)))) @@ -398,6 +403,7 @@ (define* (build source #:key verbose? (version (date-version-string)) channel-metadata system + (graft? #t) (pull-version 0) =20 ;; For the standalone Guix, default to Guile 3.0. For old @@ -415,7 +421,8 @@ (define* (build source ;; SOURCE. (mlet %store-monad ((build (build-program source version guile-version #:channel-metadata channel-me= tadata =2D #:pull-version pull-version= )) + #:pull-version pull-version + #:graft? graft?)) (system (if system (return system) (current-system))) (home -> (getenv "HOME")) =20 diff --git a/guix/channels.scm b/guix/channels.scm index 5f47834c10..3aba677534 100644 =2D-- a/guix/channels.scm +++ b/guix/channels.scm @@ -658,7 +658,7 @@ (define (with-trivial-build-handler mvalue) store)))) =20 (define* (build-from-source instance =2D #:key core verbose? (dependencies '()) syste= m) + #:key core verbose? (dependencies '()) system = graft?) "Return a derivation to build Guix from INSTANCE, using the self-build script contained therein. When CORE is true, build package modules under SOURCE using CORE, an instance of Guix. By default, build for the current @@ -703,13 +703,14 @@ (define script (build source #:verbose? verbose? #:version commit #:system system + #:graft? graft? #:channel-metadata (channel-instance->sexp instance) #:pull-version %pull-version)))) =20 ;; Build a set of modules that extend Guix using the standard method. (standard-module-derivation name source core dependencies))) =20 =2D(define* (build-channel-instance instance system +(define* (build-channel-instance instance system graft? #:optional core (dependencies '())) "Return, as a monadic value, the derivation for INSTANCE, a channel instance, for SYSTEM. DEPENDENCIES is a list of extensions providing Guile @@ -717,7 +718,8 @@ (define* (build-channel-instance instance system (build-from-source instance #:core core #:dependencies dependencies =2D #:system system)) + #:system system + #:graft? graft?)) =20 (define (resolve-dependencies instances) "Return a procedure that, given one of the elements of INSTANCES, returns @@ -747,7 +749,7 @@ (define edges (lambda (instance) (vhash-foldq* cons '() instance edges))) =20 =2D(define* (channel-instance-derivations instances #:key system) +(define* (channel-instance-derivations instances #:key system graft?) "Return the list of derivations to build INSTANCES, in the same order as INSTANCES. Build for the current system by default, or SYSTEM if specifie= d." (define core-instance @@ -763,11 +765,11 @@ (define edges (define (instance->derivation instance) (mlet %store-monad ((system (if system (return system) (current-system= )))) (mcached (if (eq? instance core-instance) =2D (build-channel-instance instance system) + (build-channel-instance instance system graft?) (mlet %store-monad ((core (instance->derivation core-in= stance)) (deps (mapm %store-monad instance->= derivation (edges instance)))) =2D (build-channel-instance instance system core deps))) + (build-channel-instance instance system graft? core d= eps))) instance system))) =20 @@ -869,7 +871,7 @@ (define (channel-instance->sexp instance) intro)))))) '())))) =20 =2D(define* (channel-instances->manifest instances #:key system) +(define* (channel-instances->manifest instances #:key system (graft? #t)) "Return a profile manifest with entries for all of INSTANCES, a list of channel instances. By default, build for the current system, or SYSTEM if specified." @@ -889,7 +891,8 @@ (define (instance->entry instance drv) `((source ,(channel-instance->sexp instance))))))) =20 (mlet* %store-monad ((derivations (channel-instance-derivations instances =2D #:syst= em system)) + #:system= system + #:graft?= graft?)) (entries -> (map instance->entry instances derivat= ions))) (return (manifest entries)))) =20 =2D-=20 2.34.0 --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=test-channel-instances-manifest-graft-control.scm Content-Description: test script (use-modules (srfi srfi-1) (guix store) (guix profiles) (guix derivations) (guix utils) (guix status) (guix grafts) (guix channels)) (define channels (list (channel (name 'guix) (branch "channel-instances-manifest-graft-control") (url "https://git.cbaines.net/git/guix.git")))) (let ((drv (with-store store (let* ((channel-instances (latest-channel-instances store channels #:authenticate? #f)) (manifest ((channel-instances->manifest channel-instances #:system "mips64el-linux" #:graft? #f) store))) (derivation-file-name (manifest-entry-item (first (manifest-entries manifest)))))))) (peek "Successfully computed derivation without grafts" drv)) (newline) (newline) (peek "Now trying with grafts") (newline) (let ((drv (with-store store (let* ((channel-instances (latest-channel-instances store channels #:authenticate? #f)) (manifest ((channel-instances->manifest channel-instances #:system "mips64el-linux") store))) (derivation-file-name (manifest-entry-item (first (manifest-entries manifest)))))))) (peek "Successfully computed derivation with grafts" drv)) --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmISNvpfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XdB4A//VtbyQ82HrmEIVdJhe9AxjkJ1cIFMWMaJ wrRjCWhQwUU375hI3PVDhLCxMTylrH7L5hjzpakucS+tfrI8O6QCwt3YbdijQXwU 56YHmOTQevjWXbyu7DweHxOs2qe0W1zmqt52r4JV7jdObacVO9KLiRSZjOyYwg3t 4MiYAmpCiVydMZ/gXfXLeb9fFtud6JZT1DedlDPQs2Sf6+auRRDfTRN78K7qKJ7P abtqKyJ0nTRjnjY9BgFcNqPgawKhkdc5xBGNoIJ5HutTudAmSYuJ0Hqz1x5qZuKr w+MGIy5XFqGxyAXCHJSoZNNVYf/oQBs4coUZFsQEYWylOfhSgNj/fs/c9Qgnb5Ti D0fTSLOg1aTsPWMNwudCTsiw84Z2l9Uba9b4QqjaHYZPdRer+ZeBACw6x75vrFrx XRH7PsOKYAyaMn+dKYFjYly0MPV5AwQBCVSIYaeliI0RI7Yo4k6sTWkGG4yh7bpo Njr3xftEfImqyj+k3TRn8BFSynLMaPqXevj4OGttMcjAr3kRWutQd4TEvM3P5eHj 0TljYLFLL5+QqUmVIR12hIgwcUjb6ttA5gX/ReSA+5W3mA3a1PruRgS1UzlAlNZo GdaiAmrHXS4qVR4ySr8EERbcNIHisL+Lx9/cDKfp8ugD9WfFLY1rWbsjn/mjdqrP rt9PJKwxU/U= =HyrT -----END PGP SIGNATURE----- --==-=-=--