From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:306:2d92::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id gM41CLq1KGWAnQAAG6o9tA:P1 (envelope-from ) for ; Fri, 13 Oct 2023 05:12:58 +0200 Received: from aspmx1.migadu.com ([2001:41d0:306:2d92::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id gM41CLq1KGWAnQAAG6o9tA (envelope-from ) for ; Fri, 13 Oct 2023 05:12:58 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E9D8B4723E for ; Fri, 13 Oct 2023 05:12:56 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=tilde.club header.s=mail header.b=poGPQlGC; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=tilde.club (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1697166777; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=OiovU/oawodSDfPIxMcQtVK2e90VvuOP+2m20q66iyE=; b=LIl6DwXc3ygVWcXYEL6XEqa8qjnXzJLVF0fOknr+vraxYARyfOpixpl+a1DAAiVW12kWmx 1JQ1yLO1FFQKip0tEkMIoiAJfq+zzWQ5Lbt27IjNE9X1wHI06djGtVvpkYqe32ab63K8ky NA+C3118Sc7St4yyc+yiPHcqrWyfD+cKDOPlowiR5J0qvsybYW1ib2tocp7r/GjTThO2L8 ZJ6pF4HxVlbT9VM1JUkQXJCIFgDKOl9F7dBerbH28My73q12hqVxcpBzl74Yk92sXuM6qu Bn1RibPXjsDSOMjnYUECDZj4YVZjM1pr/TGFXRIQ1zg32XW3OxsLsg/Jj3rr8Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=tilde.club header.s=mail header.b=poGPQlGC; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=tilde.club (policy=none) ARC-Seal: i=1; s=key1; d=yhetil.org; t=1697166777; a=rsa-sha256; cv=none; b=nC5GYOzSiYpHjdww/MQLu30Q3kO/t5EjTlieAz3sEIqF+KT6fmGxgTc/FC1xcGvrzUK0cw 6fkeemzRL0jnNZ6wfIMGmhk8JCEI+5vcatHaXyvUnKdrNXptK+PsGIiHkLVyRABRWoB4fK ecNr4RhRUhNCdoqzQ1GmpxREmaQZlYnqIwbrvJMMAIdBtWjlFYz2dIxcl3BVruxHfsK08u kOiumkkQVV1lbFsVGsBVRa9FGzMEJaDDB4+dqVnQCnxohmICf/F1ZWc+F70PWaQ8k9JkAi IPigkn256ZqmPNX4xqCIlP9ziPGLSitgR4pNcQJtdp92svY1LDQlBC37k+GTqw== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qr8bS-000071-6F; Thu, 12 Oct 2023 23:12:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qr8bQ-00005H-Ud for bug-guix@gnu.org; Thu, 12 Oct 2023 23:12:40 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qr8bQ-00065d-1U for bug-guix@gnu.org; Thu, 12 Oct 2023 23:12:40 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qr8bm-0007MW-H5 for bug-guix@gnu.org; Thu, 12 Oct 2023 23:13:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#65665: package-mapping with #:deep? #t doesn't get all the implicit inputs Resent-From: Ulf Herrman Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Fri, 13 Oct 2023 03:13:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65665 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Ulf Herrman , 65665@debbugs.gnu.org Received: via spool by 65665-submit@debbugs.gnu.org id=B65665.169716673728245 (code B ref 65665); Fri, 13 Oct 2023 03:13:02 +0000 Received: (at 65665) by debbugs.gnu.org; 13 Oct 2023 03:12:17 +0000 Received: from localhost ([127.0.0.1]:44518 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qr8b2-0007LU-LS for submit@debbugs.gnu.org; Thu, 12 Oct 2023 23:12:17 -0400 Received: from tilde.club ([2607:5300:204:4340::114]:39164 ident=postfix) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qr8az-0007LJ-Tw for 65665@debbugs.gnu.org; Thu, 12 Oct 2023 23:12:15 -0400 Received: by tilde.club (Postfix, from userid 5378) id F17102204ECD3; Fri, 13 Oct 2023 03:11:48 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 tilde.club F17102204ECD3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tilde.club; s=mail; t=1697166708; bh=Y2J1UJmc8+760t42UzsPJ0oJkdYYEYKE+KFervDcklM=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=poGPQlGCUDBaAgvlvMoLJxptJLxUZw9nr/9POKMqSSUragHwhmNnFRW3HPH+6yGLl eL0PmZetDrERfTT6C+j4oje6TJoXcO5LrsTVlWz6Dt+PPfQko+gZEp9oCEx+LjXEa2 FD1YpgG9c2yAE5wrOVcolofDwNn0VCyG0ITRXujg= From: Ulf Herrman References: <87h6ofufy5.fsf@tilde.club> <87msxmqwng.fsf@tilde.club> <871qe0lytk.fsf_-_@gnu.org> Date: Thu, 12 Oct 2023 22:11:32 -0500 In-Reply-To: <871qe0lytk.fsf_-_@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Thu, 12 Oct 2023 16:06:47 +0200") Message-ID: <87wmvrdxnf.fsf@tilde.club> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: 0.74 X-Spam-Score: 0.74 X-Migadu-Queue-Id: E9D8B4723E X-Migadu-Scanner: mx2.migadu.com X-TUID: QHzyAENgwhxP --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Ulf Herrman skribis: > >> -(define (build-system-with-package-mapping bs rewrite) >> +(define (build-system-with-package-mapping bs rewrite-input rewrite-arg= ument) >> "Return a variant of BS, a build system, that rewrites a bag's inputs= by >> passing them through REWRITE, a procedure that takes an input tuplet and >> returns a \"rewritten\" input tuplet." >> @@ -1442,9 +1442,10 @@ (define (build-system-with-package-mapping bs rew= rite) >> (let ((lowered (apply lower args))) >> (bag >> (inherit lowered) >> - (build-inputs (map rewrite (bag-build-inputs lowered))) >> - (host-inputs (map rewrite (bag-host-inputs lowered))) >> - (target-inputs (map rewrite (bag-target-inputs lowered)))))) >> + (build-inputs (map rewrite-input (bag-build-inputs lowered))) >> + (host-inputs (map rewrite-input (bag-host-inputs lowered))) >> + (target-inputs (map rewrite-input (bag-target-inputs lowered))) >> + (arguments (map rewrite-argument (bag-arguments lowered)))))) > > Aah, now I understand. :-) > > It=E2=80=99s indeed the case that arguments can capture references to pac= kages > that won=E2=80=99t be caught by =E2=80=98package-mapping=E2=80=99. For i= nstance, if you write: > > (package > =E2=80=A6 > (arguments (list =E2=80=A6 #~(whatever #$coreutils)))) > > =E2=80=A6 then =E2=80=98coreutils=E2=80=99 here cannot be replaced. > > To address this, the recommendation is to always add dependencies to > input fields and to use self-references within arguments. The example > above should be written like this: > > (package > =E2=80=A6 > (arguments > (list =E2=80=A6 #~(whatever #$(this-package-input "coreutils"))))) > > It=E2=80=99s just a recommendation and one can perfectly ignore it, and I > suppose that was the impetus for this patch. That and a growing thirst for a nuclear option for package rewriting brought about by trying to debug deep transformations while simultaneously experimenting with rewriting a manifest of around 270 packages. There are really several distinct issues at play here: 1. The case of #:qtbase and #:guile being invisible to package-mapping. This is what I first noticed, and cannot be fixed without modifying the build systems. This is what prompted looking for packages in package and bag arguments, and recursing into lists therein (just in case an argument took a list of packages). 2. The (perceived) case of packages hiding inside arguments. In hindsight, this was probably actually (3) causing this, though it's hard to tell because I discovered it last. I attempted to resolve this by recursing through s and s. 3. `this-package' referring to the inherited package in thunked fields, rather than the package inheriting them. This is what prompted the use of package-{inputs,arguments,etc}-with-package. (1) could be resolved in several ways. I'm partial to looking for arguments whose values are packages and transforming them (when #:deep? #t is specified), and adjusting the build systems to make that work consistently, which is what I've done. (2) is probably not an issue, though it occurs to me that the technique of recursively searching through arguments looking for packages could be used to implement a sort of automated "transformability" check, which could help a lot when trying to debug transformations. (3) is a major issue; the entire strategy of using `this-package-input' to enable transformations breaks because of it. My fix works for me at least, though it requires exposing additional low-level procedures and transformation authors using them. I'll open another bug about it, as requested. > This is one of the things discussed while designing this change: > > https://guix.gnu.org/en/blog/2021/the-big-change/ > (search for =E2=80=9Cself-referential records=E2=80=9D) > > https://issues.guix.gnu.org/49169 > > My take was and still is that it=E2=80=99s an acceptable limitation. Pac= kagers > need to follow the guideline above if they want proper support for > rewriting, =E2=80=98guix graph=E2=80=99, and other tools. > > WDYT? I think it's probably reasonable, though I would like it if there were tooling in place to detect cases where said guidelines are not followed, so as to aid in debugging and verifying that a transformation worked as intended. =2D Ulf --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQHIBAEBCAAyFiEEn6BUn0yca1D9JsMa1lV76sJM9mgFAmUotWQUHHN0cmluZXNz QHRpbGRlLmNsdWIACgkQ1lV76sJM9mgcHQv/cLA17ks/VPFW1MKvakcWKjbhqOya ZoJcYoUsCporAu8xONfRKpZ3sSWPwbL4I/WS57DClxkYO5Xf3f5pgH5xJo24CT3h gkorqQikxsoAUhfuLeoG9HchGrA+SX0MJIEQkCv0Fg7YXc4imsHAH97EOPnwy3L3 dslCDKLFLTJMDhnFLHOksaeAFlQUqhDNHnu6v8AhGVcvS5Vqxq0KkJwWS4knggzX jvC+tO/JVRyXrxvvUVTtCCyXGvv6r9VA0KAXbwRT2FbII7DkfQBA1/xA8tkKOmCi QgC4VddTd2poemWeUKL9RPeWv7VhhrTXyxFKssu5SMyo0Y1apcbBCstk9Py+FvSP YUWzjZwUy1eKFax7WIXpTIE8KpRWjkOX0Rnb9ES3f4T+lbQ13O9Mb0Kjg6HLxV8K Q7mvQ8P9H5rXWSE7AbnGXB+UlO+Jmlkxc2XBzxc7Zv7QHCinSiG04efZdSlPPSW+ wfA+Q7UdrkjBqZ7lWoa8PXNgKSWtvlb3YP/j =lzX7 -----END PGP SIGNATURE----- --=-=-=--