* --with-source version not honored? @ 2021-10-20 9:18 Phil Beadling 2021-10-20 11:08 ` Julien Lepiller 0 siblings, 1 reply; 11+ messages in thread From: Phil Beadling @ 2021-10-20 9:18 UTC (permalink / raw) To: guix-devel [-- Attachment #1: Type: text/plain, Size: 2347 bytes --] Hi all, I'm using the following incantation: guix build --with-source=foobar@9.5.0=/opt/thirdparty/foobar/foobar950_beta/linux64 <--with-source=gurobipy@9.5.0=/opt/thirdparty/gurobi/gurobi950_beta/linux64> foobar However the package build is failing with: (copy-file "lib/libfoobar.so.9.0.1" "/gnu/store/gkawzac…") In procedure copy-file: No such file or directory That is the new version number 9.5.0 is not written to every place when transforming the original package (version 9.0.1). I think only the package-version is updated, but the other package components are not then regenerated, meaning that if they use the package-version as an input we get a disjoint package. In the example above I use version like so: *(add-after 'install 'install-foobar-library (lambda* (#:key outputs #:allow-other-keys) (let* ((dir (string-append (assoc-ref outputs "out") "/lib/python3.8/site-packages/foobar/")) (lib-to-install (string-append "libfoobar.so." ,version))* But ",version" is not updated, nor is it updated if I change this to call (package-version foobar). If I drop into Guile I can see this a bit more clearly by writing a manifest - the code below gives exactly the same error however when the package-version is displayed it correctly responds with 9.5.0. Not sure if this should be considered a bug, or if there is a lazy way of evaluating version so avoid the problem - I think it's unexpected from a practicioners point of view as packages end up inconsistent. I presume I can manually replace the arguments section of the package in the manifest to workaround this - is there a standard way of doing this? Any ideas or clarifications welcome! Cheers, Phil. *(use-modules (guix transformations) (guix packages))(define transform ;; The package transformation procedure. (options->transformation '((with-source . "gurobipy@9.5.0=/opt/thirdparty/foobar/foobar950_beta/linux64"))))(define my-package (transform (specification->package "foobar")))(display (package-version my-package)) ;; this will display version 9.5.0(newline)(packages->manifest (list my-package)) ;; building this will fail because copy-file still looks for 9.0.1* [-- Attachment #2: Type: text/html, Size: 4382 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: --with-source version not honored? 2021-10-20 9:18 --with-source version not honored? Phil Beadling @ 2021-10-20 11:08 ` Julien Lepiller 2021-10-20 11:48 ` zimoun 2021-10-20 19:25 ` Phil 0 siblings, 2 replies; 11+ messages in thread From: Julien Lepiller @ 2021-10-20 11:08 UTC (permalink / raw) To: guix-devel, Phil Beadling [-- Attachment #1: Type: text/plain, Size: 2838 bytes --] I think your incantation is incorrect: you build foobar@9.0.1, and you replace the source of foobar@9.5.0 only. For the rest of your question, I think there is actually no way to fix that: when you use ",version", it gets evaluated before you can import the package. Maybe (package-version this-package) would work? Le 20 octobre 2021 05:18:02 GMT-04:00, Phil Beadling <phil@beadling.co.uk> a écrit : >Hi all, > >I'm using the following incantation: > >guix build >--with-source=foobar@9.5.0=/opt/thirdparty/foobar/foobar950_beta/linux64 ><--with-source=gurobipy@9.5.0=/opt/thirdparty/gurobi/gurobi950_beta/linux64> >foobar > > >However the package build is failing with: > >(copy-file "lib/libfoobar.so.9.0.1" "/gnu/store/gkawzac…") > >In procedure copy-file: No such file or directory > > >That is the new version number 9.5.0 is not written to every place when >transforming the original package (version 9.0.1). I think only the >package-version is updated, but the other package components are not then >regenerated, meaning that if they use the package-version as an input we >get a disjoint package. > >In the example above I use version like so: > > > > > > >*(add-after 'install 'install-foobar-library > (lambda* (#:key outputs #:allow-other-keys) > (let* ((dir (string-append (assoc-ref outputs "out") > >"/lib/python3.8/site-packages/foobar/")) > (lib-to-install (string-append >"libfoobar.so." ,version))* > >But ",version" is not updated, nor is it updated if I change this to call >(package-version foobar). > > >If I drop into Guile I can see this a bit more clearly by writing a >manifest - the code below gives exactly the same error however when the >package-version is displayed it correctly responds with 9.5.0. > >Not sure if this should be considered a bug, or if there is a lazy way of >evaluating version so avoid the problem - I think it's unexpected from a >practicioners point of view as packages end up inconsistent. > >I presume I can manually replace the arguments section of the package in >the manifest to workaround this - is there a standard way of doing this? > >Any ideas or clarifications welcome! > >Cheers, >Phil. > > > > > > > > > > > > > > > > >*(use-modules (guix transformations) (guix packages))(define >transform ;; The package transformation procedure. >(options->transformation '((with-source . >"gurobipy@9.5.0=/opt/thirdparty/foobar/foobar950_beta/linux64"))))(define >my-package (transform (specification->package "foobar")))(display >(package-version my-package)) ;; this will display version >9.5.0(newline)(packages->manifest (list my-package)) ;; building this will >fail because copy-file still looks for 9.0.1* [-- Attachment #2: Type: text/html, Size: 5442 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: --with-source version not honored? 2021-10-20 11:08 ` Julien Lepiller @ 2021-10-20 11:48 ` zimoun 2021-10-20 19:46 ` Phil 2021-10-20 19:25 ` Phil 1 sibling, 1 reply; 11+ messages in thread From: zimoun @ 2021-10-20 11:48 UTC (permalink / raw) To: Julien Lepiller; +Cc: Guix Devel Hi, On Wed, 20 Oct 2021 at 13:27, Julien Lepiller <julien@lepiller.eu> wrote: > > I think your incantation is incorrect: you build foobar@9.0.1, and you replace the source of foobar@9.5.0 only. It reminds me this thread: <https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00022.html> <https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00110.html> All the best, simon ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: --with-source version not honored? 2021-10-20 11:48 ` zimoun @ 2021-10-20 19:46 ` Phil 2021-10-21 10:18 ` zimoun 2021-10-21 20:22 ` Ludovic Courtès 0 siblings, 2 replies; 11+ messages in thread From: Phil @ 2021-10-20 19:46 UTC (permalink / raw) To: zimoun; +Cc: Guix Devel Hi zimoun writes: > It reminds me this thread: > > <https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00022.html> Thanks this is an interesting thread - I stumbled upon a quirk trying to find the right combination of switches. I found that if I do this (which I think is bad): guix environment --with-source=foobar@9.5.0=/path/to/package some-package-that-depends-on-foobar --ad-hoc foobar This gives me the warning that with-source will have no effect on some-package-that-depends-on-foobar, but that's not strictly true here. In this case the depending package's dependencies are installed into the environment including the original version of foobar (9.0.1) first, but this is then overwritten by foobar@9.0.5 - I can see files from both packages under the site-packages for foobar. Weirdly this is almost what I was trying to do: "Install some-package-that-depends-on-foobar, but rebuilt with the version of foobar from with-source." However the reason it works is probably not intended behavior. I couldn't work out an incantation that provided this facility properly - my best guess was: guix environment --with-source=foobar@9.5.0=/path/to/package --with-inputs=foobar=foobar@9.5.0 some-package-that-depends-on-foobar Which returns that it can't find foobar@9.5.0. I figure at this point I might have more luck if I drop into Guile to do this, but it's always a win when I can do it as a one-liner in shell. Any ideas if I can create a new package with --with-source and then substitute it in the same command for an input of another package? ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: --with-source version not honored? 2021-10-20 19:46 ` Phil @ 2021-10-21 10:18 ` zimoun 2021-10-21 20:22 ` Ludovic Courtès 1 sibling, 0 replies; 11+ messages in thread From: zimoun @ 2021-10-21 10:18 UTC (permalink / raw) To: Phil; +Cc: Guix Devel Hi Phil, On Wed, 20 Oct 2021 at 20:46, Phil <phil@beadling.co.uk> wrote: > guix environment --with-source=foobar@9.5.0=/path/to/package > some-package-that-depends-on-foobar --ad-hoc foobar Well, I do not know what you are trying to achieve. > This gives me the warning that with-source will have no effect on > some-package-that-depends-on-foobar, but that's not strictly true here. > > In this case the depending package's dependencies are installed into the > environment including the original version of foobar (9.0.1) first, but this > is then overwritten by foobar@9.0.5 - I can see files from > both packages under the site-packages for foobar. Usually, the transformation reads: guix build foobar --with-source=foobar=/path/to/foobar/v9.5.0 or guix build barbaz --with-inputs=foobar=foobar@9.5.0 The transformation ’with-source’ overwrites the origin of the target package. Target package means the one specified after ’--with-source=’. If you have 2 packages ’foobar’, one at version 9.0.1 and one at 9.5.0, and you want to overwrite the source of 9.5.0, from my understanding, you should write: guix build foobar@9.5.0 --with-source=foobar@9.5.0=/path/to/foobar/v9.5.0-modif Does it not work? The transformation ’with-inputs’ overwrites the inputs. The package ’barbaz’ depends on the package ’foobar’ at one specific version; in the input lists, there is ("foobar" ,foobar-9.0.1) and here ,foobar-9.0.1 refers to a symbol defining a package at a specific version. This symbol is probably just ’foobar’, but if all is named foobar, the explanations are made harder. ;-) If you have another version of foobar, you have probably another symbol, say foobar-9.5.0. Then ’with-inputs=foobar=foobar@9.5.0’ replaces the symbol listed as inputs of ’barbaz’ by the symbol referring to “foobar@9.5.0“. In the example, somehow, the symbol ’foobar-9.0.1’ will be replaced by the symbol ’foobar-9.5.1’. Now, be careful with ("identifier" ,symbol). I do not remember if ’with-inputs’ uses ‘identifier’ or the “name” that ’symbol’ defines. I guess it is “name”. I agree that it can appear confusing; a change in core-updates should remove this confusing ’identifier’, I guess. :-) > Any ideas if I can create a new package with --with-source and then > substitute it in the same command for an input of another package? Is this command-line not working guix build barbaz --with-source=foobar=/path/to/replaced/foobar ? Assuming the package ’barbaz’ depends on the package “foobar”. Otherwise, maybe you can provide an concrete example defining the package ’barbazz’ and ’foobar’. :-) Cheers, simon ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: --with-source version not honored? 2021-10-20 19:46 ` Phil 2021-10-21 10:18 ` zimoun @ 2021-10-21 20:22 ` Ludovic Courtès 2021-10-21 21:17 ` zimoun 2021-10-23 10:04 ` Phil 1 sibling, 2 replies; 11+ messages in thread From: Ludovic Courtès @ 2021-10-21 20:22 UTC (permalink / raw) To: Phil; +Cc: Guix Devel Hi, Phil <phil@beadling.co.uk> skribis: > Any ideas if I can create a new package with --with-source and then > substitute it in the same command for an input of another package? For historical reasons, ‘--with-source’ only applies to leaf packages, unlike most (all?) other transformation inputs. Concretely, this works: guix build guile --with-source=guile=… but this has no effect (Guix depends on Guile, but that Guile is left unchanged): guix build guix --with-source=guile=… Could it be that it’s what you’re stumbling upon? Jesse Gibbons provided a patch last year (!) to address that: https://issues.guix.gnu.org/43193 Maybe it’s time to pick it up where we left off. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: --with-source version not honored? 2021-10-21 20:22 ` Ludovic Courtès @ 2021-10-21 21:17 ` zimoun 2021-10-23 10:04 ` Phil 1 sibling, 0 replies; 11+ messages in thread From: zimoun @ 2021-10-21 21:17 UTC (permalink / raw) To: Ludovic Courtès, Phil; +Cc: Guix Devel Hi, On Thu, 21 Oct 2021 at 22:22, Ludovic Courtès <ludo@gnu.org> wrote: > For historical reasons, ‘--with-source’ only applies to leaf packages, > unlike most (all?) other transformation inputs. Oh, good to know! :-) Therefore, what I wrote before is partially wrong. Cheers, simon ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: --with-source version not honored? 2021-10-21 20:22 ` Ludovic Courtès 2021-10-21 21:17 ` zimoun @ 2021-10-23 10:04 ` Phil 2021-10-24 12:11 ` Phil Beadling 1 sibling, 1 reply; 11+ messages in thread From: Phil @ 2021-10-23 10:04 UTC (permalink / raw) To: Ludovic Courtès; +Cc: Guix Devel Hi, Ludovic Courtès writes: > For historical reasons, ‘--with-source’ only applies to leaf packages, > unlike most (all?) other transformation inputs. Concretely, this works: Yes I think this is what I'm hitting. I want to reference a package using --with-source, but then immediately build an environment which feeds this source as an input into a 2nd package - forcing it to rebuild. My hack works but for propagated dependencies in python - but it's not a proper solution, and is brittle. I was going to try to improve it by writing a Guile script to avoid the overwriting - I can post any solution I find if I get something working. Currently I can only do 2 things at the command line it seems: 1. I can try package X without writing a package but only in isolation or as a leaf. 2. I can replace package X as a dependency of package Y, if I write a package for it. Being able to combine 1 and 2 at the command line - would be very useful. Cheers, Phil. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: --with-source version not honored? 2021-10-23 10:04 ` Phil @ 2021-10-24 12:11 ` Phil Beadling 2021-10-29 14:02 ` Ludovic Courtès 0 siblings, 1 reply; 11+ messages in thread From: Phil Beadling @ 2021-10-24 12:11 UTC (permalink / raw) To: Ludovic Courtès; +Cc: Guix Devel [-- Attachment #1: Type: text/plain, Size: 3744 bytes --] I was able to write a short manifest that avoided the overwriting of the original "foobarpy" package in my case, and instead cleanly replace it with a newer version specified using "with-source". By setting, for example, GUIX_FOOBAR_VERSION=1.23 and GUIX_TEST_PACKAGE=my-package - I can now create an environment which builds the version of foobarpy I want from an https link generated from GUIX_FOOBAR_VERSION, and then create a modified version of "my-package" with foobarpy replaced, rather than overwritten. This is much better than my original hack, but it's still not recursive so whilst it does rebuild my-package with the new version of foobarpy as a propagated input, it isn't rebuilding the whole dependency tree - other inputs to my-package could also depend on foobarpy, and should be rebuilt too - so what I've done is more of a graft, buthappens to be good enough for my specific use case (the foobarpy is not used anywhere else in my case). Anyway thought I'd share this here as a template for simple non-recursive cases. I suspect it's not a massive effort to make it recursive, when I have a moment I'll have a think about that too. Any thoughts on how to improve are welcome! (use-modules (guix transformations) ;; options->transformation (guix packages) ;; package/inherit (srfi srfi-1) ;; fold (srfi srfi-98)) ;; get-environment-variable (define (generate-foobar-link version) (string-append "https://packages.foobar.com/" (substring version 0 3) "/foobar" version "_linux64.tar.gz")) (define transform ;; The package transformation procedure. (let* ((foobar-version (get-environment-variable "GUIX_FOOBAR_VERSION")) (foobar-link (string-append "foobarpy@" foobar-version "=" (generate-foobar-link foobar-version)))) (format #t "~%Setting Foobar Version: ~a~%" foobar-version) (options->transformation `((with-source . ,foobar-link))))) (define my-foobar-package (transform (specification->package "foobarpy"))) (define-public my-package-with-my-foobar (let* ((test-package-string (get-environment-variable "GUIX_TEST_PACKAGE")) (test-package (specification->package test-package-string))) (package/inherit test-package (propagated-inputs `(("foobarpy" ,my-foobar-package) ,@(fold alist-delete (package-propagated-inputs test-package) '("foobarpy"))))))) (packages->manifest (list my-package-with-my-foobar)) On Sat, 23 Oct 2021 at 11:04, Phil <phil@beadling.co.uk> wrote: > Hi, > > Ludovic Courtès writes: > > > For historical reasons, ‘--with-source’ only applies to leaf packages, > > unlike most (all?) other transformation inputs. Concretely, this works: > > Yes I think this is what I'm hitting. I want to reference a package > using --with-source, but then immediately build an environment which > feeds this source as an input into a 2nd package - forcing it to > rebuild. > > My hack works but for propagated dependencies in python - but it's not a > proper solution, and is brittle. I was going to try to improve > it by writing a Guile script to avoid the overwriting - I can post any > solution I find if I get something working. > > Currently I can only do 2 things at the command line it seems: > > 1. I can try package X without writing a package but only in isolation > or as a leaf. > > 2. I can replace package X as a dependency of package Y, if I write a > package > for it. > > Being able to combine 1 and 2 at the command line - would be very > useful. > > Cheers, > Phil. > [-- Attachment #2: Type: text/html, Size: 4617 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: --with-source version not honored? 2021-10-24 12:11 ` Phil Beadling @ 2021-10-29 14:02 ` Ludovic Courtès 0 siblings, 0 replies; 11+ messages in thread From: Ludovic Courtès @ 2021-10-29 14:02 UTC (permalink / raw) To: Phil Beadling; +Cc: Guix Devel Hi, Phil Beadling <phil@beadling.co.uk> skribis: > I was able to write a short manifest that avoided the overwriting of the > original "foobarpy" package in my case, and instead cleanly replace it with > a newer version specified using "with-source". > > By setting, for example, GUIX_FOOBAR_VERSION=1.23 and > GUIX_TEST_PACKAGE=my-package - I can now create an environment which builds > the version of foobarpy I want from an https link generated from > GUIX_FOOBAR_VERSION, and then create a modified version of "my-package" > with foobarpy replaced, rather than overwritten. Nice! For the record, there’s a ‘--with-latest’ option that I initially intended to be ‘--with-version’, but the importer/updater code isn’t capable enough yet. > Anyway thought I'd share this here as a template for simple non-recursive > cases. I suspect it's not a massive effort to make it recursive, when I > have a moment I'll have a think about that too. Any thoughts on how to > improve are welcome! Thanks for sharing! Ludo’. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: --with-source version not honored? 2021-10-20 11:08 ` Julien Lepiller 2021-10-20 11:48 ` zimoun @ 2021-10-20 19:25 ` Phil 1 sibling, 0 replies; 11+ messages in thread From: Phil @ 2021-10-20 19:25 UTC (permalink / raw) To: Julien Lepiller; +Cc: guix-devel Thanks for the reply. Julien Lepiller writes: > evaluated before you can import the package. Maybe (package-version this-package) would work? Yes! (package-version this-package) worked perfectly - thanks for your help. ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-10-29 14:02 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-10-20 9:18 --with-source version not honored? Phil Beadling 2021-10-20 11:08 ` Julien Lepiller 2021-10-20 11:48 ` zimoun 2021-10-20 19:46 ` Phil 2021-10-21 10:18 ` zimoun 2021-10-21 20:22 ` Ludovic Courtès 2021-10-21 21:17 ` zimoun 2021-10-23 10:04 ` Phil 2021-10-24 12:11 ` Phil Beadling 2021-10-29 14:02 ` Ludovic Courtès 2021-10-20 19:25 ` Phil
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/guix.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.