From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id OAeoEx4xU2OveQAAbAwnHQ (envelope-from ) for ; Sat, 22 Oct 2022 01:54:06 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id aEvFEx4xU2N5jwAA9RJhRA (envelope-from ) for ; Sat, 22 Oct 2022 01:54:06 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 0609339DBE for ; Sat, 22 Oct 2022 01:54:06 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1om0De-0004EC-Eb; Fri, 21 Oct 2022 18:10:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1olzFY-0007HY-A8 for guix-devel@gnu.org; Fri, 21 Oct 2022 17:08:16 -0400 Received: from mail-wm1-x32c.google.com ([2a00:1450:4864:20::32c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1olzFW-0001DN-A2 for guix-devel@gnu.org; Fri, 21 Oct 2022 17:08:16 -0400 Received: by mail-wm1-x32c.google.com with SMTP id v130-20020a1cac88000000b003bcde03bd44so5970873wme.5 for ; Fri, 21 Oct 2022 14:08:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=beadling-co-uk.20210112.gappssmtp.com; s=20210112; h=mime-version:date:message-id:in-reply-to:subject:cc:to:from :user-agent:references:from:to:cc:subject:date:message-id:reply-to; bh=2jajNFWtHOstMHDemEFEpP9UI7b7aLfhl/d1SgjH96g=; b=Dvcc3XUzxAuGvvJKJTOQy4hgK24cNCDuqXIcL1aVqoKIaEtipit+b8d14PfC5pOCeG gwp5bcbuVzR1tmmrymgpXj3kXJhCmXAr79OM2SwCwxw1bva82fWZRNu0MGtnIdyY5qRw hCdJVBQQZ1YhQvvntm/q1vAhSvRM71vhVQA7Q2srWlRlIK3x2jyNp+xNpEjiLOxc16gm YkSvV+fH8VLdOy2Qjx8bKuyEZii0nX0/3XQr7dMGdI7TCoOZu4wKtNg9gus/rNMRpKUU ZrnKnExl0cLeRLzhwyXYcNpmUwg3T0p//s85cD6uyUxDHL1q6WIMrnF4EiqcnyEYzQr9 mvQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:date:message-id:in-reply-to:subject:cc:to:from :user-agent:references:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=2jajNFWtHOstMHDemEFEpP9UI7b7aLfhl/d1SgjH96g=; b=D6tpKmafTSEPOwHZSb2ff5kTT2N/uo/rl1JtbnWC5aUq2yXY04vPVYiBGwq1i9xlLr CKb34nzrDJvsTAXXmJk/h2SeOak2W0HS6EOywNz66+GvN4VgXolfk3UClhKrVVORIjNM DSWzs99OrDowIA271JNcutiN6USOj/ROG2l8F95KMhm20oBjZLJnebe554w22FTPzUC8 zytfE195aIz5lGwjHu5wYArEmpVXE5+/goMSuz4D9fPTZIMs4YKth9BzqnMewqQAbrZy E2fQqwVEfVgFfYne6g/cT5Vm1qIG8aA93lmUlla0GaP+e6KiqkbwVCFLOpFGVIpbj0s0 Q3yA== X-Gm-Message-State: ACrzQf2bFxuFVOsR0bmR8led77S/UIE2VEhBZmaiGf3y5f5bc9drUlZc A4Nj2ZSkvQxO3OWLS36cGV+4ksvK0i/xD9DO X-Google-Smtp-Source: AMsMyM718OjIGp6yk8GBUa9Jz3EstyJHWE/Kt1cnbjIw8Kwv7V6NwMSlISIFI3HfplJiTAOfRflBxQ== X-Received: by 2002:a05:600c:2b94:b0:3c6:f941:a26f with SMTP id j20-20020a05600c2b9400b003c6f941a26fmr14267819wmc.8.1666386491912; Fri, 21 Oct 2022 14:08:11 -0700 (PDT) Received: from xps13 ([2a02:6b61:74a5:0:3e3e:75ff:de88:7a90]) by smtp.gmail.com with ESMTPSA id g19-20020a05600c4ed300b003c409244bb0sm4282755wmq.6.2022.10.21.14.08.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Oct 2022 14:08:11 -0700 (PDT) References: <87czam9nxq.fsf@beadling.co.uk> <86r0z11psa.fsf@gmail.com> User-agent: mu4e 1.4.15; emacs 27.2 From: Phil To: zimoun Cc: guix-devel@gnu.org Subject: Re: Pinning package inputs using inferiors? In-reply-to: <86r0z11psa.fsf@gmail.com> Message-ID: <87czakdgw5.fsf@beadling.co.uk> Date: Fri, 21 Oct 2022 22:08:10 +0100 MIME-Version: 1.0 Content-Type: text/plain Received-SPF: none client-ip=2a00:1450:4864:20::32c; envelope-from=phil@beadling.co.uk; helo=mail-wm1-x32c.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Guix-devel" Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1666396446; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=2jajNFWtHOstMHDemEFEpP9UI7b7aLfhl/d1SgjH96g=; b=dPj/jYdHm9w5pTbgwkDXgOVAtTNLiAnVeV1rSDfmi4BQJhtKxz54qCHksTV6BONeDwlmf3 mT7kpRnbZRX4bDWGTaDQZF6L065bEgTBZmOjJw6pYfCLO083BReAzv2P7Z4h7Qwfyx0snV 11iZgW4phtRZbzT5z0C3/V3Eye3Y36ybZif46YdPmAZp4prXrIuBuHiuAgABm6LeXHbQp3 s+3UI//xlcypHvYGgL0Zr0Vd08rvaHeXp/diaVSmf75ZKePSZtntkrkVaDWntx0TtPnuXa tZGJfLzok223w4DOmVSeEqYJfkOJg5D5gvprCUkeMuZh1oEdxAauu0abR4j5dA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1666396446; a=rsa-sha256; cv=none; b=fwc0fFYuD3n3r96wXQbFw0zboJbh5ECDx896HfgakXWg3XE2RlBO2inzPCiZ5ZkFlFmekT 8oA5DJCs+2P3WY8IV+j/P15VFtyA8KwkVPwaxhT4fzhGvPES/x2AMbn2adMv8cA6nIC/JU fhWyZftr1TRwo9wG/moGtl7TZKFvDkSRFjU5Yy1OTsKSFajjLf+N4N2JYODU4VpLAM2cpK 2vZwR+1YBhUB+KiQVRjhjMM2I4GRDPCuw4iq9LwzTLTUrc6F85GMxRYKm1FLuYws8+9Fkw kvL3H0Z0uTrLsNxmWuCH4CXDGI0in+Mt2f9NP+N9nqgBj2B8uY8GQwqviJ4vGQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=Dvcc3XUz; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -0.94 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=Dvcc3XUz; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 0609339DBE X-Spam-Score: -0.94 X-Migadu-Scanner: scn1.migadu.com X-TUID: aTAEJiOF0/78 Thanks Simon - I've given an example below. zimoun writes: > For an example, see python-numpy and python-numpy-next in (gnu packages > python-xyz). This was my original way of handling this but in what is perhaps a niche use of Guix by my department - it ultimately doesn't scale well, for our use-case. Originally the department was small enough that there was only a handful of applications sharing one or two common in-house libraries. As we've scaled-up we now have the situation where 3 or 4 common libraries are being used by say 10 applications. We have rapid release schedules - and want to be able to release the common libraries on a weekly basis. But the time to sign-off on a common library takes a few days per application, so it's not practical for every project to bump version every week - they have other priorities. In an ideal world automated unit and regression testing would be comprehensive enough that we could move aggressively each week, but at least for now that's not practical given the complex nature of signing off the libraries and the applications which use the libraries. So, ideally, what we'd like to do is just have each common library churn-out releases every week, and have the releases available in Guix, but without having an obligation on dependent applications to adopt these changes if they don't want to. Note all libraries and applications share the same channel - one solution would be to have each library in their own channel, but this feels ugly to me. Our solution (somewhat tricky to explain without a whiteboard - apologies!) is to co-locate the package definition of the common library in the common library repo itself - we call it something like .requirements.scm and it is naturally kept in lockstep with the code in that repo that the definition will build. This is very different to traditional Guix where channels contain definitions separately in a different repo to the code those definitions describe how to build. We then have a job in our CI/CD system that allows us to give a tag on the common library repo, and the name of an application that uses the common library. The job will copy the .requirements.scm into our channel inside a private module specific to the application that uses the common library. The idea is that you can have many versions of .requirements.scm private to every application package definition that references it. You could even read .requirements.scm using a function that clones the application repo on-the-fly rather than statically storing it in the channel - we haven't gone this far yet, as it makes the system even more complex to reason about. This is basically the same idea as the python-numpy-next but allows for many versions of python-numpy to co-exist by keeping them all in private modules so they don't clash. It's a cool idea and works pretty well, but requires us to augment Guix with a set of extra tools to lift and shift these private definitions around, which complicates our setup considerably. It feels like wanting to make many versions of a library available at once isn't an unreasonable way to work at-scale. However, it also feels like a departure from the philosophy of Guix to decentralise package definitions and to allow for a potentially large range of versions to co-exist in the same channel commit. We could try to further integrate the idea into guix by writing new guix commands to support it - we're still working out the details ourselves, but if it works well we'd love to present it at a future Guix Days or similar! In the meantime I was wondering if anyone else had a similar use-case for Guix and if they had tried something similar or different to handle many versions in an automated way in the same channel commit? Apologies that's more than I was intending to write - but hopefully that makes some sense! If it doesn't I can try to flesh out specific example?