From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id UFGtEEM6VmOakwAAbAwnHQ (envelope-from ) for ; Mon, 24 Oct 2022 09:09:55 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id eJXsD0M6VmNxRQAAG6o9tA (envelope-from ) for ; Mon, 24 Oct 2022 09:09:55 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DE06739B74 for ; Mon, 24 Oct 2022 09:09:54 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1omj9h-0005Ph-5a for larch@yhetil.org; Sun, 23 Oct 2022 18:09:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1omJns-0007a8-J5 for guix-devel@gnu.org; Sat, 22 Oct 2022 15:05:04 -0400 Received: from mail-wr1-x431.google.com ([2a00:1450:4864:20::431]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1omJnq-0001mg-2u for guix-devel@gnu.org; Sat, 22 Oct 2022 15:05:04 -0400 Received: by mail-wr1-x431.google.com with SMTP id bu30so9656966wrb.8 for ; Sat, 22 Oct 2022 12:05:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:sender:from:to :cc:subject:date:message-id:reply-to; bh=OeTik7czB/xfQA+Z6r8MO3Y8YVwrfqeGRtnSvyfPIP8=; b=UoV9k9KZYbgWfc4c/bl3qCGPSWGZNAtb+BIoTIBs09G+h4AH0c9YOZsPM+bbWTr0sR pLERzPP5yhPrrQqFPeJ3iK1cIEcptcn4sAe6o1GIFTydN2qCXHVMF1Ko52fkyhpgG6/7 KDJ3exJFXBeQhZEG6assSZfFevBVWyIVWXmEHWdXYAv4axqxeselFrBPB0VIMKnimsk3 v4pz6zgSiBvHDUTZgG4V2M8+bbhxMPcQz5jXiNY4/DQ+NGjf1JpAqJPMqVIxoh9tFRDA HoasTOJ3Ux3D7kjFVU0JEd+AIwXuusD2EZkkOnhgiTi9CxF5Kj6FMFGpQ4NmbPzG273i fDrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OeTik7czB/xfQA+Z6r8MO3Y8YVwrfqeGRtnSvyfPIP8=; b=Fs/OLOWXOLwe1Egsq39j/AS19jGhRxc9ltYAuZgeVs91KigKGz4GANI01uGPDnC2QO vxXRdgO6d/mivkN+1k6JrjahYUhCSvMaxTI5+d9lX/UBaAfq26pjtr+FqGrwkuZG4aTF lMTtFAOvmzdmLHC4k/JeYCF3ves5agpAP+qR6QYxVUWv6UjeFVTSDCzyffrSXI6pogh3 FdA+1GNpngaBI6+GMvQJm6o/MH/DgCYYLdsHjHwm5hx8jhi48rnKJm93qrPjvICqlGDv hvZAb6I2+vWthz1D08kjX+3cQT7AdTSW9wdScemVQngUxcv/nIUj69msEdTzf/6pn7FP QqVw== X-Gm-Message-State: ACrzQf1HjHcndLmPiVB0QkZ6dS7DzReSxA7yRt0jdWLpvZ799wh+UcYg X23PbnN4vtNWZL3/fAR7pGc= X-Google-Smtp-Source: AMsMyM76+n+6SG9sjDBuri0o/fBB00b4wvYKCSmwdzmVU53ahfVbmSff6iRS4gw8ISWAo1xZpBfuOA== X-Received: by 2002:a5d:6da1:0:b0:231:c189:e077 with SMTP id u1-20020a5d6da1000000b00231c189e077mr16407745wrs.114.1666465500360; Sat, 22 Oct 2022 12:05:00 -0700 (PDT) Received: from localhost ([141.226.13.62]) by smtp.gmail.com with ESMTPSA id m11-20020adff38b000000b0023658168944sm3678569wro.114.2022.10.22.12.04.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Oct 2022 12:04:59 -0700 (PDT) Date: Sat, 22 Oct 2022 22:04:46 +0300 From: Efraim Flashner To: Phil Cc: zimoun , guix-devel@gnu.org Subject: Re: Pinning package inputs using inferiors? Message-ID: Mail-Followup-To: Phil , zimoun , guix-devel@gnu.org References: <87czam9nxq.fsf@beadling.co.uk> <86r0z11psa.fsf@gmail.com> <87czakdgw5.fsf@beadling.co.uk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="w0SZTCi6LPAlY1J7" Content-Disposition: inline In-Reply-To: <87czakdgw5.fsf@beadling.co.uk> X-PGP-Key-ID: 0x41AAE7DCCA3D8351 X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc X-PGP-Fingerprint: A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Received-SPF: pass client-ip=2a00:1450:4864:20::431; envelope-from=efraim.flashner@gmail.com; helo=mail-wr1-x431.google.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1666595395; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=OeTik7czB/xfQA+Z6r8MO3Y8YVwrfqeGRtnSvyfPIP8=; b=fzsOe52+gSpw8IxUvpobwTxK6LlgpVFn6c7X7rLFAUvhgDtUyDdijMv3vU0FFuXGzLSP12 onBfA0QWV3CN5m4liaWBOKxOV5SgrbToR5O3m1yreaPZIwbBjOUPCUYlm8LMFMCRZSU50+ cJAJfQAOSt5/lFHZdLnZeV4yS9DlLsFSLYcPtAopaOiHTbGJ6dLH7ejkFo8+RLgS6nPRgc AmcH7H8gwGxZ8zSRPJ1w2M4LpYnJuRFECosTdpBvVOlVYdABtkpLbAwhFKRHSQNdb5LnTu eULXscDrlzyZLL4WiU6AHMt58bqooEt/ZvF/zKk+4NKgYcbc3aSIJRVnczxoeA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1666595395; a=rsa-sha256; cv=none; b=MRBaaQi75k3D5OS1mLnEh3+6P8aYZ1BxHiLj2cH++A9YN5TFAAEziehGZUNuGXGgEc3AUC mRkwHf22Ep9DV2IX1YhR3267zF93oaxSpZqhPhfzm4XUFemYonpWjErutZdXqF+p8Dz9Kw XOf81zU+DSdVaPoUrxXMIYepfKQJYHX3jWqEXapDDWZzMKQJ9HAlYYhaPNf2RJu+nMQHT5 ac5IQC3yJLXnglPwISeiY0J2MfdTxGRA8G5MMUR7hBcEvDI2lyDnxexUrkFgdYKDsp78Kb es2ISE94bdpOPLhhi9zpgeUUnOJnw1QMzjlZ8l1MtpV6rxh3p3THL9p8T8CKxw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=UoV9k9KZ; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 6.50 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=UoV9k9KZ; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: DE06739B74 X-Spam-Score: 6.50 X-Migadu-Scanner: scn0.migadu.com X-TUID: 8CVjKs0nwbeL --w0SZTCi6LPAlY1J7 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Oct 21, 2022 at 10:08:10PM +0100, Phil wrote: > Thanks Simon - I've given an example below. >=20 > zimoun writes: >=20 > > For an example, see python-numpy and python-numpy-next in (gnu packages > > python-xyz). >=20 > This was my original way of handling this but in what is perhaps a niche > use of Guix by my department - it ultimately doesn't scale well, for our > use-case. >=20 > Originally the department was small enough that there was only a handful > of applications sharing one or two common in-house libraries. >=20 > As we've scaled-up we now have the situation where 3 or 4 common > libraries are being used by say 10 applications. >=20 > We have rapid release schedules - and want to be able to release the > common libraries on a weekly basis. But the time to sign-off on a > common library takes a few days per application, so it's not practical for > every project to bump version every week - they have other priorities. >=20 > In an ideal world automated unit and regression testing would be > comprehensive enough that we could move aggressively each week, but at > least for now that's not practical given the complex nature of signing > off the libraries and the applications which use the libraries. >=20 > So, ideally, what we'd like to do is just have each common library > churn-out releases every week, and have the releases available in Guix, > but without having an obligation on dependent applications to adopt > these changes if they don't want to. >=20 > Note all libraries and applications share the same channel - one > solution would be to have each library in their own channel, but this > feels ugly to me. >=20 > Our solution (somewhat tricky to explain without a whiteboard - > apologies!) is to co-locate the package definition of the common library > in the common library repo itself - we call it something like > .requirements.scm and it is naturally kept in lockstep with the code in > that repo that the definition will build. This is very different to > traditional Guix where channels contain definitions separately in a > different repo to the code those definitions describe how to build.=20 >=20 > We then have a job in our CI/CD system that allows us to give a tag on the > common library repo, and the name of an application that uses the common > library. >=20 > The job will copy the .requirements.scm into our channel inside a > private module specific to the application that uses the common library. >=20 > The idea is that you can have many versions of .requirements.scm private > to every application package definition that references it. >=20 > You could even read .requirements.scm using a function that clones the > application repo on-the-fly rather than statically storing it in the > channel - we haven't gone this far yet, as it makes the system even more > complex to reason about. >=20 > This is basically the same idea as the python-numpy-next but allows for > many versions of python-numpy to co-exist by keeping them all in private > modules so they don't clash. >=20 > It's a cool idea and works pretty well, but requires us to augment Guix > with a set of extra tools to lift and shift these private definitions > around, which complicates our setup considerably. >=20 > It feels like wanting to make many versions of a library available at > once isn't an unreasonable way to work at-scale. However, it also feels > like a departure from the philosophy of Guix to decentralise package > definitions and to allow for a potentially large range of versions to > co-exist in the same channel commit. >=20 > We could try to further integrate the idea into guix by writing new guix > commands to support it - we're still working out the details ourselves, > but if it works well we'd love to present it at a future Guix Days or > similar! >=20 > In the meantime I was wondering if anyone else had a similar use-case > for Guix and if they had tried something similar or different to handle > many versions in an automated way in the same channel commit? >=20 > Apologies that's more than I was intending to write - but hopefully that > makes some sense! If it doesn't I can try to flesh out specific example? Apologies for not slotting in the reply inline, I'm not sure exactly where to put it. This might be a good use for package transformations. Imagine the following: ;;; Python-dep-1 (define-publid python-dep1-week1 ...) (define-publid python-dep1-week2 ...) (define-publid python-dep1-week3 ...) ;;; Python-dep-2 (define-publid python-dep2-week1 ...) (define-publid python-dep2-week2 ...) (define-publid python-dep2-week3 ...) ;;; Python package (define my-python-package-base (name "my-python-package-base") ... (inputs (list python-dep1 python-dep2))) (define-public my-python-package (inherit my-python-package-base) (name "my-python-package") (inputs (modify-inputs (package-inputs python-package-base) (replace python-dep1 python-dep1-week3) (replace python-dep2 python-dep2-week2)))) or if you wanted to do it recursively (define package-inputs-for-my-python-package (package-input-rewriting/spec `(("python-dep1" . ,(const python-dep1-week3)) ("python-dep2" . ,(const python-dep2-week2))))) (define-public python-package-with-correct-inputs (package (inherit (package-inputs-for-ython-package my-python-package-base)) (name "my-python-package"))) Both ideas use 'placeholder packages' so that you can swap them out for your actual dependency, the first one only for itself and the second one working recursively. I use the second one myself at work to replace tensorflow with a version built for the machine it's running on, and at home to use some newer golang libraries. --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --w0SZTCi6LPAlY1J7 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmNUPsoACgkQQarn3Mo9 g1HPwxAAvcdTPftm4hkIT8kKiu3mwYewOmBHT4dgATp91Qs7sAxTILnlUgw5TTp+ HslGqDuP6SO3x+qtTe1MFwa8hjNv3HQOn3gJquTljG6NJxaS8KqNkzrgf+IatUFZ bsmeCEWI22IzA4RmsBUE/n4T66d55YbOOzeWRUOnBSPKz4CppYn//uyvUe4+I0kt 5y4CcZcRKUIPvDzkfvejOC7VDz/4IzzxjKx7Ym3lDwt8S8lJ/T3UVM32dru6/hai pTiqNgZfBt+VGe4ILNTIS/a4yvypvtwuSECNPAd9L8UEEb8G9xOf0vGglElBVL2v Z9HEEHF+Tod7XrhMvWVTAGMHNx5Le16J6COt45+rTJvI+jauqLOTu1zY8pflKL62 F1zUjb2JWYDo8viwTnJlFG9vfVMmrVFI7xJTTBXhIVrBx2zi5miPRNFnIP/neWXR GLVwU2fJHCOk/5SI6z47DVuQD3tms9f2Q7fh7w2lnCJ+qfyjg/TrdX2fjW2iMwMA JYYyXST81maEzYIyRS/fhum85fZnQb6lJTJ2j1D3L4UYipl62o1RdDmjq5nTnlmw kuHOVCjuNcuRNe+Kq/rTTgOvSMc1tNyA9F2giv5iMiUIG9FUbAs7LUJncVj55IVd 38Pd6F7WtABuikfvoQ6K4/wozpNxSNoPHs62icHjhKtNn5aBM7o= =gYI3 -----END PGP SIGNATURE----- --w0SZTCi6LPAlY1J7--