From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id EDRkKpFNdWGIUQAAgWs5BA (envelope-from ) for ; Sun, 24 Oct 2021 14:12:01 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 4EATJpFNdWGmDgAAB5/wlQ (envelope-from ) for ; Sun, 24 Oct 2021 12:12:01 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 51DE82DD2A for ; Sun, 24 Oct 2021 14:12:01 +0200 (CEST) Received: from localhost ([::1]:47412 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mecM4-0005EB-Cx for larch@yhetil.org; Sun, 24 Oct 2021 08:12:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38840) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mecLY-0005CS-DU for guix-devel@gnu.org; Sun, 24 Oct 2021 08:11:29 -0400 Received: from mail-io1-xd36.google.com ([2607:f8b0:4864:20::d36]:44889) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mecLV-0000if-PH for guix-devel@gnu.org; Sun, 24 Oct 2021 08:11:28 -0400 Received: by mail-io1-xd36.google.com with SMTP id r134so11764069iod.11 for ; Sun, 24 Oct 2021 05:11:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=beadling-co-uk.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=LOMcex2F7sAnvBYY5jJBzIWrYvZKXti3v9YXZA4xASM=; b=VCPenDKOK7wn3u1T+wLAuGtshr+mUf955xorh33Tx8UlWpkN1A5xx4yxFNaoVP6+M1 C6EOGLdd6jcDQgXTF/r8WtnKqPa7DOHwSRSOGkOfR532WuzvTmMFewOHPQT5kQildG/j SGvGGJbmQ7bDOQiAudPeXBf0j9I4ukup6e+UIZw0k+yAB8grTpIapXpGKmMLkdMERugR YQheHTMmoGvEIEJn8hV3s30CL7lgGUv3xFsnvLjSTtrVjQ1pUwrKyb+3SVHHSVwzward yAcAhT0TyHeQpRXJ8Vpm4bbuDGjbruSMdvqhG5OdzxBQG9gEdgnr8I6qC8VMQbTw3p18 Q8Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=LOMcex2F7sAnvBYY5jJBzIWrYvZKXti3v9YXZA4xASM=; b=Wo4XHZi7EjyLeHZCyrk4kj8vHoKYp1EWLaS61SeZDzbi7GX3KhPQi7lzkwA6XxfMBD MWiArRL4bEOjTyc1vqmeH4uB5VRkng6akTDAcNqfj5sMxbf1t2bx4wsFd+eMXoGDLwK4 XrV443QkmrDEEgBBriOSaGVPGGzvI+FOQSovindEtvJD2uS5xmokW22E864wzdgs44Y2 O3IzS5TPk7mdl+iLHH/BQO4B56EiqnstWhN6JDK+ULgrWy+CyZ4LnukfirAQwbpEu322 GGUHOig6L76nWE2Rf1Athq+umBrYU/88dZnkEXPx9qDu9lfcJIFJV5t6hT/23upk5zcI +rgQ== X-Gm-Message-State: AOAM532FROScm1oGLRM6gs9LQ6Jo/vPetxUvNDIFpJqM0d6/T3UxWTjE CEosC+4uX1K+3d0pAj5z3ErW5NmLLGhrEcQPc52CDQ== X-Google-Smtp-Source: ABdhPJw4Z0QRBAX2dMh2ZnvijaKfMYsRc12Fu2guc9ulXacC+AMe94AxiU3z41cNXTvCWvOtH0B/tkSTdkG6zbIhdks= X-Received: by 2002:a05:6602:2c88:: with SMTP id i8mr6975803iow.48.1635077483882; Sun, 24 Oct 2021 05:11:23 -0700 (PDT) MIME-Version: 1.0 References: <87cznzbhss.fsf@beadling.co.uk> <87sfwuw2kr.fsf@gnu.org> <87mtn0f45b.fsf@beadling.co.uk> In-Reply-To: <87mtn0f45b.fsf@beadling.co.uk> From: Phil Beadling Date: Sun, 24 Oct 2021 13:11:13 +0100 Message-ID: Subject: Re: --with-source version not honored? To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= Content-Type: multipart/alternative; boundary="0000000000009b829d05cf1824e8" Received-SPF: none client-ip=2607:f8b0:4864:20::d36; envelope-from=phil@beadling.co.uk; helo=mail-io1-xd36.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guix Devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1635077521; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=LOMcex2F7sAnvBYY5jJBzIWrYvZKXti3v9YXZA4xASM=; b=WjEzAoZwtLUN1o45VCJ8vKxFXZwlLDURaIQr9Rm4SnUrQYOBl4X27y4KnREIBFpzIhhKVV EW41uO69E32ZVfXG82mqIEMqWOy5cafXNSW5fY77aV6wLBVs/FNmg6h6jJIGKjoEA8HybD u05M9eMyIw28eLqDgWQLb2Z68PugIxCYNQEqaMuq6PShsAsJUjC/bQyNlBCf9MoJsEJEUU v4WEXQ1wS3QPvCjN5T0MCTUfecbNuvdfXRo4YHql50jeQsUCYahdHpb5vVBJ+Gf55YHqDB IZLc+3UzEqbsh6pg451hTefAb73UFJRl0bfycOnujPi2PD0jfFNDbU77v+qFaw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1635077521; a=rsa-sha256; cv=none; b=VjBD/9rFObqrM/wSBr6c0xBh5u12q+xMbECDPlBDO4xKIFcJdIgQ4QEdB6lBjELLyrPfwa irTX0xVHzTTLr68NTbiVa45IuwwTaqzHR2g1yqUwnj/LNe5ia8IbnTph4fR8COHR12/esv A4tZWHA96reesS4XgdLpgbMIA++oj4/zJZqnIi+SssSCjy08hrtm/PbBCuP30RD7+IQUeQ MlN9z2nvCs2dWvkV7jI0uqE7zG7PnATypdJLk2OeqyIiGwBszCadm/nN4gsZZmYE6bTnB3 +hkTa4yemyv7GW3QkXFqN19r5xgHvdzCZm+dFTuZRmL/bBjaVpJ1qHM1VFbaog== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=VCPenDKO; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -0.42 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=beadling-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=VCPenDKO; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 51DE82DD2A X-Spam-Score: -0.42 X-Migadu-Scanner: scn0.migadu.com X-TUID: R37dX/Tt55nG --0000000000009b829d05cf1824e8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I was able to write a short manifest that avoided the overwriting of the original "foobarpy" package in my case, and instead cleanly replace it with a newer version specified using "with-source". By setting, for example, GUIX_FOOBAR_VERSION=3D1.23 and GUIX_TEST_PACKAGE=3Dmy-package - I can now create an environment which buil= ds the version of foobarpy I want from an https link generated from GUIX_FOOBAR_VERSION, and then create a modified version of "my-package" with foobarpy replaced, rather than overwritten. This is much better than my original hack, but it's still not recursive so whilst it does rebuild my-package with the new version of foobarpy as a propagated input, it isn't rebuilding the whole dependency tree - other inputs to my-package could also depend on foobarpy, and should be rebuilt too - so what I've done is more of a graft, buthappens to be good enough for my specific use case (the foobarpy is not used anywhere else in my case). Anyway thought I'd share this here as a template for simple non-recursive cases. I suspect it's not a massive effort to make it recursive, when I have a moment I'll have a think about that too. Any thoughts on how to improve are welcome! (use-modules (guix transformations) ;; options->transformation (guix packages) ;; package/inherit (srfi srfi-1) ;; fold (srfi srfi-98)) ;; get-environment-variable (define (generate-foobar-link version) (string-append "https://packages.foobar.com/" (substring version 0 3) "/foobar" version "_linux64.tar.gz")) (define transform ;; The package transformation procedure. (let* ((foobar-version (get-environment-variable "GUIX_FOOBAR_VERSION")) (foobar-link (string-append "foobarpy@" foobar-version "=3D" (generate-foobar-link foobar-version)))) (format #t "~%Setting Foobar Version: ~a~%" foobar-version) (options->transformation `((with-source . ,foobar-link))))) (define my-foobar-package (transform (specification->package "foobarpy"))) (define-public my-package-with-my-foobar (let* ((test-package-string (get-environment-variable "GUIX_TEST_PACKAGE")) (test-package (specification->package test-package-string))) (package/inherit test-package (propagated-inputs `(("foobarpy" ,my-foobar-package) ,@(fold alist-delete (package-propagated-inputs test-package) '("foobarpy"))))))) (packages->manifest (list my-package-with-my-foobar)) On Sat, 23 Oct 2021 at 11:04, Phil wrote: > Hi, > > Ludovic Court=C3=A8s writes: > > > For historical reasons, =E2=80=98--with-source=E2=80=99 only applies to= leaf packages, > > unlike most (all?) other transformation inputs. Concretely, this works= : > > Yes I think this is what I'm hitting. I want to reference a package > using --with-source, but then immediately build an environment which > feeds this source as an input into a 2nd package - forcing it to > rebuild. > > My hack works but for propagated dependencies in python - but it's not a > proper solution, and is brittle. I was going to try to improve > it by writing a Guile script to avoid the overwriting - I can post any > solution I find if I get something working. > > Currently I can only do 2 things at the command line it seems: > > 1. I can try package X without writing a package but only in isolation > or as a leaf. > > 2. I can replace package X as a dependency of package Y, if I write a > package > for it. > > Being able to combine 1 and 2 at the command line - would be very > useful. > > Cheers, > Phil. > --0000000000009b829d05cf1824e8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I was able to write a short manifest that avoided the= overwriting of the original "foobarpy" package in my case, and i= nstead cleanly replace it with a newer version specified using "with-s= ource".

By setting, for example, GUIX_FOOBAR_= VERSION=3D1.23 and GUIX_TEST_PACKAGE=3Dmy-package - I can now create an env= ironment which builds the version of foobarpy I want from an https link gen= erated from GUIX_FOOBAR_VERSION, and then create a modified version of &quo= t;my-package" with foobarpy replaced, rather than overwritten.

This is much better than my original hack, but it's s= till not recursive so whilst it does rebuild my-package with the new versio= n of foobarpy as a propagated input, it isn't rebuilding the whole depe= ndency tree - other inputs to my-package could also depend on foobarpy, and= should be rebuilt too - so what I've done is more of a graft, buthappe= ns to be good enough for my specific use case (the foobarpy is not used any= where else in my case).

Anyway thought I'd= share this here as a template for simple non-recursive cases.=C2=A0 I susp= ect it's not a massive effort to make it recursive, when I have a momen= t I'll have a think about that too.=C2=A0 Any thoughts on how to improv= e are welcome!

=C2=A0(use-modules (guix transf= ormations) ;; options->transformation
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 (guix packages) ;; package/inherit
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (srfi srfi-1) ;; fold
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (srfi srfi-98)) ;; get-environment-variable=


=C2=A0(define (generate-foobar-link version)
=C2=A0 =C2=A0(s= tring-append "https://package= s.foobar.com/"
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 (substring version 0 3) "/foobar"
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 version "_linux64.ta= r.gz"))

=C2=A0(define transform
=C2=A0 =C2=A0;; The package = transformation procedure.
=C2=A0 =C2=A0(let* ((foobar-version (get-envir= onment-variable "GUIX_FOOBAR_VERSION"))
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 (foobar-link (string-append
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "foobarpy@&quo= t; foobar-version
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "=3D" (generate-foobar-link fooba= r-version))))
=C2=A0 =C2=A0 =C2=A0(format #t "~%Setting Foobar Vers= ion: ~a~%" foobar-version)
=C2=A0 =C2=A0 =C2=A0(options->transfo= rmation
=C2=A0 =C2=A0 =C2=A0 `((with-source . ,foobar-link)))))

= =C2=A0(define my-foobar-package (transform (specification->package "= ;foobarpy")))

=C2=A0(define-public my-package-with-my-foobar=C2=A0 =C2=A0(let* ((test-package-string (get-environment-variable "G= UIX_TEST_PACKAGE"))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (test-packag= e (specification->package test-package-string)))
=C2=A0 =C2=A0 =C2=A0= (package/inherit test-package
=C2=A0 =C2=A0 =C2=A0 =C2=A0(propagated-inp= uts
=C2=A0 =C2=A0 =C2=A0 =C2=A0 `(("foobarpy" ,my-foobar-packa= ge)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ,@(fold alist-delete (package-pro= pagated-inputs test-package) '("foobarpy")))))))

=C2= =A0(packages->manifest (list my-package-with-my-foobar))

<= /div>


On Sat, 23 Oct 2021 at 11:04, Phil <phil@beadling.co.uk> wrote:
Hi,

Ludovic Court=C3=A8s writes:

> For historical reasons, =E2=80=98--with-source=E2=80=99 only applies t= o leaf packages,
> unlike most (all?) other transformation inputs.=C2=A0 Concretely, this= works:

Yes I think this is what I'm hitting.=C2=A0 I want to reference a packa= ge
using --with-source, but then immediately build an environment which
feeds this source as an input into a 2nd package - forcing it to
rebuild.

My hack works but for propagated dependencies in python - but it's not = a
proper solution, and is brittle.=C2=A0 I was going to try to improve
it by writing a Guile script to avoid the overwriting - I can post any
solution I find if I get something working.

Currently I can only do 2 things at the command line it seems:

1. I can try package X without writing a package but only in isolation
or as a leaf.

2. I can replace package X as a dependency of package Y, if I write a packa= ge
for it.

Being able to combine 1 and 2 at the command line - would be very
useful.

Cheers,
Phil.
--0000000000009b829d05cf1824e8--