From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id GFIOHuIv9mPfOwEAbAwnHQ (envelope-from ) for ; Wed, 22 Feb 2023 16:08:18 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id GBwdHeIv9mPKcQEAG6o9tA (envelope-from ) for ; Wed, 22 Feb 2023 16:08:18 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DCB1731126 for ; Wed, 22 Feb 2023 16:08:17 +0100 (CET) Authentication-Results: aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Seal: i=1; s=key1; d=yhetil.org; t=1677078498; a=rsa-sha256; cv=none; b=RlPJi4JIZaz4biaXpVIEIDoehHj1aFHx/17pSsGgQ6dnCwlcadDlOlxZPKKJs0q7+enNfv OELStwxjjfRQtqezep2IUBKG7F1tn3XMHMgKjnQvK95/aF/69ijwOFnuRYaFa9bCJuQNIQ OriHI+yCpkWf5zaz5tfwduh9SjSxpkG3QnGqFsABIYl0sdh9eBSv/1qaDhFQwE4qIV7bxl BVGCZymaU6uapHjdjrQczcpEfUIEFLKHJaLAkUVNMOc1VV4GtbOCLDfQkEOdxaFCE4RqE+ ICCu52kZczBlsCio+1k4ouZuCmzVmJyhtdztZaxnmtsIkkhXolceRSNyRUpNjw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1677078498; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=ATVwnTCQiIZJInHTOIqsjfn7LsQaNiX1ffl3oZ6NBwk=; b=AXndE9V17Ga8V8Ip1V1ct8TYEh4RK0eKvD5Hs97psixZFCOBKNmoVmHZjcavne+jeN5OVy rEDGKnPQF/EESKcu3zGW4yY0n24lWZ5BOQ8KlewNOhy4fhwIkZjTAJ9ysj6qZ2XZI2ft+o 8n3UBMFz17/IEMNsLtjS60yaToSo2bowzHcL1C++GMhu2whPiW0DeboXOSU4/nS/mbPRjK X3GFzJNddJgvwuq+nR9oVUXrfEXa4sfzQIiy6xYA672PZ9cL+fxQfB3tI/yJgMclQw8dze bAodg/wwkescXmc3fYIgaZEEvIPkzmureGg9dwQxpK4TmS7ar0swRV6RPrXGNQ== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pUqj4-00019k-Re; Wed, 22 Feb 2023 10:08:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pUqiw-00018d-Pj for guix-patches@gnu.org; Wed, 22 Feb 2023 10:08:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pUqiw-00046u-C0 for guix-patches@gnu.org; Wed, 22 Feb 2023 10:08:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pUqiv-0005XM-PY for guix-patches@gnu.org; Wed, 22 Feb 2023 10:08:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#61363] [PATCH 2/2] self: Apply grafts to the outputs of the guix derivation. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 22 Feb 2023 15:08:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 61363 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 61363@debbugs.gnu.org Received: via spool by 61363-submit@debbugs.gnu.org id=B61363.167707845821256 (code B ref 61363); Wed, 22 Feb 2023 15:08:01 +0000 Received: (at 61363) by debbugs.gnu.org; 22 Feb 2023 15:07:38 +0000 Received: from localhost ([127.0.0.1]:60064 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pUqiX-0005Wl-I7 for submit@debbugs.gnu.org; Wed, 22 Feb 2023 10:07:38 -0500 Received: from mira.cbaines.net ([212.71.252.8]:42286) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pUqiT-0005Wa-8Q for 61363@debbugs.gnu.org; Wed, 22 Feb 2023 10:07:35 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:54d1:d5d4:280e:f699]) by mira.cbaines.net (Postfix) with ESMTPSA id 2118416B9E; Wed, 22 Feb 2023 15:07:31 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 32853173; Wed, 22 Feb 2023 15:07:29 +0000 (UTC) References: <20230208075403.11788-1-mail@cbaines.net> <20230208075403.11788-2-mail@cbaines.net> <87sfey9i1t.fsf@gnu.org> User-agent: mu4e 1.8.13; emacs 28.2 From: Christopher Baines Date: Wed, 22 Feb 2023 11:17:48 +0000 In-reply-to: <87sfey9i1t.fsf@gnu.org> Message-ID: <878rgpeo28.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: X-Migadu-Queue-Id: DCB1731126 X-Spam-Score: -6.06 X-Migadu-Spam-Score: -6.06 X-Migadu-Scanner: scn0.migadu.com List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-TUID: YAkePR/fScul --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: >> I'm looking at this as it'll allow the Guix Data Service to compute the >> derivations without grafts, and for these to be useful for substitutes >> regardless of whether users are using grafts. > > How does it help exactly? By disabling grafts in that context? So the Guix Data Service is somewhat built on the assumption that it's cheap to compute derivations, at least with grafts disabled. That's always been the case for packages, but for channel instance derivations it's not reliably the case, since currently disabling grafts doesn't apply to the whole process, and even if it did, the derivations you'd get out wouldn't be that useful (since you can't transform the outputs from those derivations to the outputs you'd get if using grafts). With these changes, it's always relatively cheap to compute the channel instance derivations, and it's always possible to compute the derivations for any system without needing to be able to perform builds for that system. You can see this in how the data service has processed Guix before and after these patches. This is the channel instances before: https://data.qa.guix.gnu.org/revision/a582d863465990642d331bc05bf073f47fb= 80908/channel-instances and this is after: https://data.qa.guix.gnu.org/revision/9cfbb22b556d28a0af345824ae5b3e00eb3= f4a15/channel-instances Given data.qa.guix.gnu.org is running on an x86_64-linux system, that and i686-linux isn't generally a problem, but I'm guessing it only managed to compute the powerpc64le-linux and aarch64-linux derivations because it was able to substitute the necessary store items. For other system computing the derivations would have failed. I believe this change will also mean that the build farms will go from performing the grafting for these builds, to being able to not do so, in line with how builds for packages are handled. This isn't a big thing, but I think it makes sense. >> +++ b/guix/self.scm >> @@ -752,7 +752,8 @@ (define* (compiled-guix source #:key >> (gzip (specification->package "gzip")) >> (bzip2 (specification->package "bzip2")) >> (xz (specification->package "xz")) >> - (guix (specification->package "guix"))) >> + (guix (specification->package "guix")) >> + (graft? #t)) >> "Return a file-like object that contains a compiled Guix." >> (define guile-avahi >> (specification->package "guile-avahi")) >> @@ -802,6 +803,12 @@ (define dependencies >> guile-json guile-semver guile-ssh guile-sqlite3 >> guile-lib guile-zlib guile-lzlib guile-zstd))) >>=20=20 >> + (define packages >> + (cons* gzip >> + bzip2 >> + xz >> + dependencies)) >> + > > [...] > >> + (let ((obj (built-modules (lambda (node) >> + (list (node-source node) >> + (node-compiled node)))))) >> + (if graft? >> + (explicit-grafting obj packages) >> + obj))) > > There are two things I=E2=80=99m not comfortable with: > > 1. Having in (guix packages); it looks misplaced. I didn't put it there at first, but I think it makes sense since grafting is currently specific to packages, as is this additional code. > 2. More importantly, manually listing packages that might require > grafting looks like a slippery slope (=E2=80=9Coops! we=E2=80=99re n= ot getting the > GnuTLS graft for that CVE, too bad=E2=80=9D). > > I designed and implemented several variants to try and delay grafting. > One of them consisted in carrying graft information in gexps: > > https://git.savannah.gnu.org/cgit/guix.git/log?h=3Dwip-gexp-grafts > > It=E2=80=99s kinda similar to what you=E2=80=99re proposing in that graft= information is > carried as far as possible. The main difference is that it=E2=80=99s aut= omated. That's interesting, I think that making grafting not specific to packages, and something where the replacement is handled at a lower level (e.g. gexps) would be an alternative way to handle this. Given that this approach works though, maybe the explicit-grafting functionality could just sit and be used inside of (guix self). Given that module is very explicit about what packages are used, it should be possible to arrange the code so it's very hard to miss a package out, which should address your concern about manually listing packages (maybe specification->package can be tweaked so that it's possible to get all the packages, and that can be the list considered for grafting). I don't know of any other places where this approach would be useful, so while it would be nice to have a more general grafting mechanism eventually, I'd also like to be able to make these changes to channel instance grafts sooner rather than later. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmP2L69fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xf3pw/9EtEo0UOool4+8PfEFZ/Df9MYigLmm/y9 MWN0gQrfhosBdRVVGGZlCDlz9Hgh1ix07pJKSUELHxaoLGwhyKgPvxghz7A9BbBi DN4D+AJL/8Xe8aPblQvoXWwMnGViHdK1HW9uxey8UJwCl2Np/0jrYq3H30WzXPCU XWCSEz4hcFV4DTb7VDaLEINtkthv0lEPkKnaMP0ojGNoSztPxaUtCzNVc78zMxCV 6wMxa7HK7ew5131RYnGpn13p1hHFhhfnnY8EbmhkxfnZ9huZOCCNez4ifkO5jjjP CpztgnQPBe+/EgApp71Vwmx5s2ZrQMIhNykLg6e1wtRhDygtEYmI7RlzXMWVrYTn XY0XZ9d5xlc9K4hvFbvl3BA8yzjRXNSeSJMDKlvEGqy8qpl3BxH1MxXdGHkN0yGc 9M7cv9hMqwFcE7rVUKrEKfHwASDGwMwe9yTnE7eSlLhmO/xv2h+aHsmQQfrGe3y2 g5KnoOlBO26bbfXIf5rGOOSOCAmQUHMb25KFMMhdfTgP0Z6BLqNtLMbIKptEr6cM dGhaxv47kEXEEP8TcuCezdAd3N3oTwPneUM/BKmUVEKmktPlRBkR2beKxp35tFb8 ngxgSLlUj8xhf7wTf22OzaUeuXmGNmQkaycpxwQhtuEfCtZXF6G6XUU5xouXO7RE tUzp7aAE1oU= =1/IL -----END PGP SIGNATURE----- --=-=-=--