From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id cMI9OtoCzGN4hgAAbAwnHQ (envelope-from ) for ; Sat, 21 Jan 2023 16:20:59 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id YNtiOdoCzGOQSAEAG6o9tA (envelope-from ) for ; Sat, 21 Jan 2023 16:20:58 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 847F316A23 for ; Sat, 21 Jan 2023 16:20:58 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pJFfC-0002Cj-JI; Sat, 21 Jan 2023 10:20:14 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pJFfA-0002CU-SL for guix-devel@gnu.org; Sat, 21 Jan 2023 10:20:12 -0500 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pJFf8-0005t4-Ke; Sat, 21 Jan 2023 10:20:12 -0500 Received: from localhost (unknown [89.207.171.75]) by mira.cbaines.net (Postfix) with ESMTPSA id EFF3F27BBE9; Sat, 21 Jan 2023 15:20:05 +0000 (GMT) Received: from phact (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 911ca446; Sat, 21 Jan 2023 15:20:05 +0000 (UTC) References: <87h78tbthh.fsf@cbaines.net> <87zgmbcefg.fsf@gnu.org> User-agent: mu4e 1.6.11; emacs 28.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Subject: Re: Expensive builds when computing channel instance derivations Date: Sat, 21 Jan 2023 14:55:40 +0000 In-reply-to: <87zgmbcefg.fsf@gnu.org> Message-ID: <87zgabrk2k.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: 14 X-Spam_score: 1.4 X-Spam_bar: + X-Spam_report: (1.4 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_SBL_CSS=3.335, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1674314458; a=rsa-sha256; cv=none; b=PTFvLvLh0/Oyvj7geYONG/NFRQMXSNdQYyno2Z9E4h6Q82Okgjl7yK87/c71Jm4xP8//5n rfbb34QeLMP7oWJ7cUd6aQf79mGL09msD0RPKeBTUA9l8gHEZpeI+p1XRRC8s9UvSx9gE7 p3M1iAbBztRTz31GrZLv5XiNv6qV0pimp0FQf9omtUpqeydWm3qrLtrVWICFksw2DKQfFK zjMNHmf1grS59NHjFEBia8vnQXtEzRGmwUHkuBDoeAxTTFAQMIe4VjTshzHBndgYH9stkH tqfkKtxpyLCjKwrRQfv29UzxMEEPSIaFVHCvgk4AvuYV3Kgqz3PPubR2BkGsoA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1674314458; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=JdRn55LcoLeeRAQxN9pU/ywodwu+dSRMlbNJsPkmJzE=; b=qgTr8vugtRd20NSZwmv2U2rBOLYwSobj+XTNBFWGa34FFsMAeZYg7PoWG7YGl9H8KYdtxN EpOzlXPnY+5XacK4j76mD/Cet+/iRFW2YVm35HOJc+H+CIknTg5UAmuGdir3nuzPR57zaY YoMJKz/JNm4ZaXwLZbKwZ9he2gCTtxIReghn53NDOOZW56XJZ/pRNkvty5pi34pooa8yV9 GpAD7C3ILy6PZGDq10ebApyvS6vrSastPzb+rh7RcQ6E3hiGTLRdESlJP9wwE6q8tSqy2U JJE/HqS6SxMnqrnlnRYY2YokvFoDA3wcYpEVy6SmseOuCcKG+Pwb3YLhCEzrkg== X-Spam-Score: -4.30 X-Migadu-Queue-Id: 847F316A23 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Scanner: scn0.migadu.com X-Migadu-Spam-Score: -4.30 X-TUID: WmJ8jX1hfIhg --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Christopher Baines skribis: > >> I think I figured out that it's related to grafts. I've attached a test >> script which computes the channel instance derivation for mips64el-linux >> which I'm not set up to build things for (if you can build for >> mips64el-linux, replace this with a system you can't build for). You'll >> need to use the attached patch (also present on [2]) when running this >> script. >> >> 2: https://git.cbaines.net/guix/log/?h=3Dchannel-instances-manifest-graf= t-control >> >> When I run this script locally, it first succeeds in computing the >> channel instance derivation when grafts are disabled, but then fails >> when grafts are enabled: >> >> while setting up the build environment: a `mips64el-linux' is required >> to build >> `/gnu/store/g40shyhsd00r5dq3mm76c2b1krnr1njh-guile-bootstrap-2.0.drv', >> but I am a `x86_64-linux' >> >> Even though I think this shows that grafts are involved, I'm not sure >> what this means? I'm guessing that the effects of grafts aren't as clear >> cut as for packages here, since the grafts are happening here as part of >> computing a derivation I'm guessing that the derivation is actually >> built with the grafted outputs, which differs from how grafts affect >> packages. Given this, I guess computing the derivation without grafts >> means that the replacements that would be grafted in are just not used >> at all. >> >> To recap on my aim here, I really just want to be able to compute >> channel instance derivations without performing any expensive builds, or >> if that's not possible, it would be good to understand why? For some reason I didn't follow up on this thread at the time, I can't remember now it's been so long. This issue has remained at the back of my mind though since within the Guix Data Service, I think it's becoming more impactful. > In general, computing the derivation of a package may incur > builds/downloads due to the way grafts work: depending on the *build > result* of a package, =E2=80=98package->derivation=E2=80=99 might return = a grafting > derivation or it might return the original derivation. > > The =E2=80=9CComputing Guix derivation=E2=80=9D bit of =E2=80=98guix pull= =E2=80=99 is no exception: it > computes the derivation of objects that depend on packages that may need > to be grafted. Thus, it ends up triggering builds as part of this > process. While that's a way of thinking about it to highlight the similarities, it's the differences I'm interested in here. For package derivations, I think about grafts as a layer on top. First you build the packages normally, then you apply any grafts including building any replacements as required. I don't know if that's an entirely correct way of thinking about it, but it seems to work as a mental model. For bordeaux.guix.gnu.org to provide substitues, it's sufficient to build the packages (including replacements) without any grafting, because that's a separate concern. I don't think the same can be said for the channel instance derivations though, probably because of the difference you highlight above regarding grafts actually affecting how the derivations involved are constructed, rather than acting as a layer on top. > It would be tempting to turn off grafts in this case. But then, =E2=80= =98guix > pull=E2=80=99 would always give us an ungrafted Guix, which is undesirabl= e. For > the purposes of the Data Service though, you could consider turning off > grafts globally. I get that not applying grafts to guix could be undesirable, but I'm still don't see why there aren't other options? What would be the implications of treating it like package grafting. As in, you'd always build guix without grafts, then you'd apply the grafting process to the ungrafted output? That would be more consistent with the way it works for packages, and it would probably go all or some of the way to resolve the issues for the Guix Data Service. There's maybe other options as well, although as I raised in my initial email, it's hard for me to think about this since I haven't worked out why the grafting here doesn't work like packages grafting for packages in the first place. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmPMAqNfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xdvvg/9EGnTjkwgP/3oRpBlKVUzAr7I0tWNsYw0 F7h1+M+5xGm7OURyIAp2yageKArJwoh7fJZHOq2eynn2Mj98HoBGe/fXRl7QMIQy eiKpU0ZRl83fTpsPgaWsKdGrAahWAJXTjKwEdth3UFCxb0Lc1dSGQOalz40f4ODr r0x0n/pkJGnlaUblp4jXNpJp9CTh4J5aBZWvVhM+CKLZzHD2GZA4oar+OBxHHzHC 8wNGIGlGqOteeHxxnQoZh3abgP7pbVi3P8Wtju+byZjjJ/RZ7OJNjoBauysLyoSm /rIPp+qgDkBshT+esHxJqGUv6s3JjAiOQSRc2/PXs9Ota2ugbSsk8jgo1yzChxJw nsAwYTd2lpX15SzWMgZTHEP0bY6Sgi/QVYV9ubmKo6PBRjZG0cKmfvG7L/ydclMw sNCu+Fv8EGkzgWPPdzXPRHdmZ+q+a2pAV4sJG62UT+dWTuZdQgo43mvX+ZFxrkzx Y2mm2NzvAztePKL63/SHQ4clY5YS8i+I0jAiA9HpUoTMCFxi2aqhfmJchAaCXe2O 1pejjvaCWDTwJgkd/bWlVSZrvE7eCd88NVez8q1nCMbLTL8XIpVbJuZc/5xgYVD2 DLPTd7h8ZZl1zsE8+vMr1fsQxJw6p6NKa4o5MQQ41P2iKorVNxEm3PTbDMFZd/aa xSassuNRMyI= =qAp6 -----END PGP SIGNATURE----- --=-=-=--