--with-source version not honored?

--with-source version not honored?

[Phil Beadling]

[-- Attachment #1: Type: text/plain, Size: 2347 bytes --]

Hi all,

I'm using the following incantation:

guix build
--with-source=foobar@9.5.0=/opt/thirdparty/foobar/foobar950_beta/linux64
<--with-source=gurobipy@9.5.0=/opt/thirdparty/gurobi/gurobi950_beta/linux64>
foobar


However the package build is failing with:

(copy-file "lib/libfoobar.so.9.0.1" "/gnu/store/gkawzac…")

In procedure copy-file: No such file or directory


That is the new version number 9.5.0 is not written to every place when
transforming the original package (version 9.0.1).  I think only the
package-version is updated, but the other package components are not then
regenerated, meaning that if they use the package-version as an input we
get a disjoint package.

In the example above I use version like so:






*(add-after 'install 'install-foobar-library
      (lambda* (#:key outputs #:allow-other-keys)
              (let* ((dir (string-append (assoc-ref outputs "out")

"/lib/python3.8/site-packages/foobar/"))
                                             (lib-to-install (string-append
"libfoobar.so." ,version))*

But ",version" is not updated, nor is it updated if I change this to call
(package-version foobar).


If I drop into Guile I can see this a bit more clearly by writing a
manifest - the code below gives exactly the same error however when the
package-version is displayed it correctly responds with 9.5.0.

Not sure if this should be considered a bug, or if there is a lazy way of
evaluating version so avoid the problem - I think it's unexpected from a
practicioners point of view as packages end up inconsistent.

I presume I can manually replace the arguments section of the package in
the manifest to workaround this - is there a standard way of doing this?

Any ideas or clarifications welcome!

Cheers,
Phil.
















*(use-modules (guix transformations)             (guix packages))(define
transform  ;; The package transformation procedure.
(options->transformation   '((with-source .
"gurobipy@9.5.0=/opt/thirdparty/foobar/foobar950_beta/linux64"))))(define
my-package (transform (specification->package "foobar")))(display
(package-version my-package)) ;; this will display version
9.5.0(newline)(packages->manifest (list my-package)) ;; building this will
fail because copy-file still looks for 9.0.1*

[-- Attachment #2: Type: text/html, Size: 4382 bytes --]

Re: --with-source version not honored?

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 2838 bytes --]

I think your incantation is incorrect: you build foobar@9.0.1, and you replace the source of foobar@9.5.0 only.

For the rest of your question, I think there is actually no way to fix that: when you use ",version", it gets evaluated before you can import the package. Maybe (package-version this-package) would work?

Le 20 octobre 2021 05:18:02 GMT-04:00, Phil Beadling <phil@beadling.co.uk> a écrit :
>Hi all,
>
>I'm using the following incantation:
>
>guix build
>--with-source=foobar@9.5.0=/opt/thirdparty/foobar/foobar950_beta/linux64
><--with-source=gurobipy@9.5.0=/opt/thirdparty/gurobi/gurobi950_beta/linux64>
>foobar
>
>
>However the package build is failing with:
>
>(copy-file "lib/libfoobar.so.9.0.1" "/gnu/store/gkawzac…")
>
>In procedure copy-file: No such file or directory
>
>
>That is the new version number 9.5.0 is not written to every place when
>transforming the original package (version 9.0.1).  I think only the
>package-version is updated, but the other package components are not then
>regenerated, meaning that if they use the package-version as an input we
>get a disjoint package.
>
>In the example above I use version like so:
>
>
>
>
>
>
>*(add-after 'install 'install-foobar-library
>      (lambda* (#:key outputs #:allow-other-keys)
>              (let* ((dir (string-append (assoc-ref outputs "out")
>
>"/lib/python3.8/site-packages/foobar/"))
>                                             (lib-to-install (string-append
>"libfoobar.so." ,version))*
>
>But ",version" is not updated, nor is it updated if I change this to call
>(package-version foobar).
>
>
>If I drop into Guile I can see this a bit more clearly by writing a
>manifest - the code below gives exactly the same error however when the
>package-version is displayed it correctly responds with 9.5.0.
>
>Not sure if this should be considered a bug, or if there is a lazy way of
>evaluating version so avoid the problem - I think it's unexpected from a
>practicioners point of view as packages end up inconsistent.
>
>I presume I can manually replace the arguments section of the package in
>the manifest to workaround this - is there a standard way of doing this?
>
>Any ideas or clarifications welcome!
>
>Cheers,
>Phil.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>*(use-modules (guix transformations)             (guix packages))(define
>transform  ;; The package transformation procedure.
>(options->transformation   '((with-source .
>"gurobipy@9.5.0=/opt/thirdparty/foobar/foobar950_beta/linux64"))))(define
>my-package (transform (specification->package "foobar")))(display
>(package-version my-package)) ;; this will display version
>9.5.0(newline)(packages->manifest (list my-package)) ;; building this will
>fail because copy-file still looks for 9.0.1*

[-- Attachment #2: Type: text/html, Size: 5442 bytes --]

Re: --with-source version not honored?

[zimoun]

Hi,

On Wed, 20 Oct 2021 at 13:27, Julien Lepiller <julien@lepiller.eu> wrote:
>
> I think your incantation is incorrect: you build foobar@9.0.1, and you replace the source of foobar@9.5.0 only.

It reminds me this thread:

<https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00022.html>
<https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00110.html>

All the best,
simon

Re: --with-source version not honored?

[Phil]

Thanks for the reply.

Julien Lepiller writes:

> evaluated before you can import the package. Maybe (package-version this-package) would work?

Yes! (package-version this-package) worked perfectly - thanks for your help.

Re: --with-source version not honored?

[Phil]

Hi

zimoun writes:

> It reminds me this thread:
>
> <https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00022.html>

Thanks this is an interesting thread - I stumbled upon a quirk trying to
find the right combination of switches.

I found that if I do this (which I think is bad):

guix environment --with-source=foobar@9.5.0=/path/to/package
some-package-that-depends-on-foobar --ad-hoc foobar

This gives me the warning that with-source will have no effect on
some-package-that-depends-on-foobar, but that's not strictly true here.

In this case the depending package's dependencies are installed into the
environment including the original version of foobar (9.0.1) first, but this
is then overwritten by foobar@9.0.5 - I can see files from
both packages under the site-packages for foobar.

Weirdly this is almost what I was trying to do:
"Install some-package-that-depends-on-foobar, but rebuilt with the version of
foobar from with-source."

However the reason it works is probably not intended behavior.

I couldn't work out an incantation that provided this facility properly
- my best guess was:

guix environment --with-source=foobar@9.5.0=/path/to/package
--with-inputs=foobar=foobar@9.5.0 some-package-that-depends-on-foobar

Which returns that it can't find foobar@9.5.0.

I figure at this point I might have more luck if I drop into Guile to do
this, but it's always a win when I can do it as a one-liner in shell.

Any ideas if I can create a new package with --with-source and then
substitute it in the same command for an input of another package? 

Re: --with-source version not honored?

[zimoun]

Hi Phil,

On Wed, 20 Oct 2021 at 20:46, Phil <phil@beadling.co.uk> wrote:

> guix environment --with-source=foobar@9.5.0=/path/to/package
> some-package-that-depends-on-foobar --ad-hoc foobar

Well, I do not know what you are trying to achieve.


> This gives me the warning that with-source will have no effect on
> some-package-that-depends-on-foobar, but that's not strictly true here.
>
> In this case the depending package's dependencies are installed into the
> environment including the original version of foobar (9.0.1) first, but this
> is then overwritten by foobar@9.0.5 - I can see files from
> both packages under the site-packages for foobar.

Usually, the transformation reads:

  guix build foobar --with-source=foobar=/path/to/foobar/v9.5.0

or

  guix build barbaz --with-inputs=foobar=foobar@9.5.0


The transformation ’with-source’ overwrites the origin of the target
package.  Target package means the one specified after ’--with-source=’.

If you have 2 packages ’foobar’, one at version 9.0.1 and one at 9.5.0,
and you want to overwrite the source of 9.5.0, from my understanding,
you should write:

 guix build foobar@9.5.0 --with-source=foobar@9.5.0=/path/to/foobar/v9.5.0-modif

Does it not work?


The transformation ’with-inputs’ overwrites the inputs.  The package
’barbaz’ depends on the package ’foobar’ at one specific version; in the
input lists, there is ("foobar" ,foobar-9.0.1) and here ,foobar-9.0.1
refers to a symbol defining a package at a specific version.  This
symbol is probably just ’foobar’, but if all is named foobar, the
explanations are made harder. ;-)

If you have another version of foobar, you have probably another symbol,
say foobar-9.5.0.  Then ’with-inputs=foobar=foobar@9.5.0’ replaces the
symbol listed as inputs of ’barbaz’ by the symbol referring to
“foobar@9.5.0“.  In the example, somehow, the symbol ’foobar-9.0.1’ will
be replaced by the symbol ’foobar-9.5.1’.

Now, be careful with ("identifier" ,symbol).  I do not remember if
’with-inputs’ uses ‘identifier’ or the “name” that ’symbol’ defines.  I
guess it is “name”.  I agree that it can appear confusing; a change in
core-updates should remove this confusing ’identifier’, I guess. :-)

> Any ideas if I can create a new package with --with-source and then
> substitute it in the same command for an input of another package? 

Is this command-line not working

  guix build barbaz --with-source=foobar=/path/to/replaced/foobar

?  Assuming the package ’barbaz’ depends on the package “foobar”.


Otherwise, maybe you can provide an concrete example defining the
package ’barbazz’ and ’foobar’. :-)


Cheers,
simon

Re: --with-source version not honored?

[Ludovic Courtès]

Hi,

Phil <phil@beadling.co.uk> skribis:

> Any ideas if I can create a new package with --with-source and then
> substitute it in the same command for an input of another package? 

For historical reasons, ‘--with-source’ only applies to leaf packages,
unlike most (all?) other transformation inputs.  Concretely, this works:

  guix build guile --with-source=guile=…

but this has no effect (Guix depends on Guile, but that Guile is left
unchanged):

  guix build guix --with-source=guile=…

Could it be that it’s what you’re stumbling upon?

Jesse Gibbons provided a patch last year (!) to address that:

  https://issues.guix.gnu.org/43193

Maybe it’s time to pick it up where we left off.

Thanks,
Ludo’.

Re: --with-source version not honored?

[zimoun]

Hi,

On Thu, 21 Oct 2021 at 22:22, Ludovic Courtès <ludo@gnu.org> wrote:

> For historical reasons, ‘--with-source’ only applies to leaf packages,
> unlike most (all?) other transformation inputs.

Oh, good to know! :-)

Therefore, what I wrote before is partially wrong.

Cheers,
simon

Re: --with-source version not honored?

[Phil]

Hi,

Ludovic Courtès writes:

> For historical reasons, ‘--with-source’ only applies to leaf packages,
> unlike most (all?) other transformation inputs.  Concretely, this works:

Yes I think this is what I'm hitting.  I want to reference a package
using --with-source, but then immediately build an environment which
feeds this source as an input into a 2nd package - forcing it to
rebuild.

My hack works but for propagated dependencies in python - but it's not a
proper solution, and is brittle.  I was going to try to improve
it by writing a Guile script to avoid the overwriting - I can post any
solution I find if I get something working.

Currently I can only do 2 things at the command line it seems:

1. I can try package X without writing a package but only in isolation
or as a leaf.

2. I can replace package X as a dependency of package Y, if I write a package
for it.

Being able to combine 1 and 2 at the command line - would be very
useful.

Cheers,
Phil.

Re: --with-source version not honored?

[Phil Beadling]

[-- Attachment #1: Type: text/plain, Size: 3744 bytes --]

I was able to write a short manifest that avoided the overwriting of the
original "foobarpy" package in my case, and instead cleanly replace it with
a newer version specified using "with-source".

By setting, for example, GUIX_FOOBAR_VERSION=1.23 and
GUIX_TEST_PACKAGE=my-package - I can now create an environment which builds
the version of foobarpy I want from an https link generated from
GUIX_FOOBAR_VERSION, and then create a modified version of "my-package"
with foobarpy replaced, rather than overwritten.

This is much better than my original hack, but it's still not recursive so
whilst it does rebuild my-package with the new version of foobarpy as a
propagated input, it isn't rebuilding the whole dependency tree - other
inputs to my-package could also depend on foobarpy, and should be rebuilt
too - so what I've done is more of a graft, buthappens to be good enough
for my specific use case (the foobarpy is not used anywhere else in my
case).

Anyway thought I'd share this here as a template for simple non-recursive
cases.  I suspect it's not a massive effort to make it recursive, when I
have a moment I'll have a think about that too.  Any thoughts on how to
improve are welcome!

 (use-modules (guix transformations) ;; options->transformation
              (guix packages) ;; package/inherit
              (srfi srfi-1) ;; fold
              (srfi srfi-98)) ;; get-environment-variable


 (define (generate-foobar-link version)
   (string-append "https://packages.foobar.com/"
                  (substring version 0 3) "/foobar"
                  version "_linux64.tar.gz"))

 (define transform
   ;; The package transformation procedure.
   (let* ((foobar-version (get-environment-variable "GUIX_FOOBAR_VERSION"))
          (foobar-link (string-append
                        "foobarpy@" foobar-version
                        "=" (generate-foobar-link foobar-version))))
     (format #t "~%Setting Foobar Version: ~a~%" foobar-version)
     (options->transformation
      `((with-source . ,foobar-link)))))

 (define my-foobar-package (transform (specification->package "foobarpy")))

 (define-public my-package-with-my-foobar
   (let* ((test-package-string (get-environment-variable
"GUIX_TEST_PACKAGE"))
          (test-package (specification->package test-package-string)))
     (package/inherit test-package
       (propagated-inputs
        `(("foobarpy" ,my-foobar-package)
          ,@(fold alist-delete (package-propagated-inputs test-package)
'("foobarpy")))))))

 (packages->manifest (list my-package-with-my-foobar))



On Sat, 23 Oct 2021 at 11:04, Phil <phil@beadling.co.uk> wrote:

> Hi,
>
> Ludovic Courtès writes:
>
> > For historical reasons, ‘--with-source’ only applies to leaf packages,
> > unlike most (all?) other transformation inputs.  Concretely, this works:
>
> Yes I think this is what I'm hitting.  I want to reference a package
> using --with-source, but then immediately build an environment which
> feeds this source as an input into a 2nd package - forcing it to
> rebuild.
>
> My hack works but for propagated dependencies in python - but it's not a
> proper solution, and is brittle.  I was going to try to improve
> it by writing a Guile script to avoid the overwriting - I can post any
> solution I find if I get something working.
>
> Currently I can only do 2 things at the command line it seems:
>
> 1. I can try package X without writing a package but only in isolation
> or as a leaf.
>
> 2. I can replace package X as a dependency of package Y, if I write a
> package
> for it.
>
> Being able to combine 1 and 2 at the command line - would be very
> useful.
>
> Cheers,
> Phil.
>

[-- Attachment #2: Type: text/html, Size: 4617 bytes --]

Re: --with-source version not honored?

[Ludovic Courtès]

Hi,

Phil Beadling <phil@beadling.co.uk> skribis:

> I was able to write a short manifest that avoided the overwriting of the
> original "foobarpy" package in my case, and instead cleanly replace it with
> a newer version specified using "with-source".
>
> By setting, for example, GUIX_FOOBAR_VERSION=1.23 and
> GUIX_TEST_PACKAGE=my-package - I can now create an environment which builds
> the version of foobarpy I want from an https link generated from
> GUIX_FOOBAR_VERSION, and then create a modified version of "my-package"
> with foobarpy replaced, rather than overwritten.

Nice!  For the record, there’s a ‘--with-latest’ option that I initially
intended to be ‘--with-version’, but the importer/updater code isn’t
capable enough yet.

> Anyway thought I'd share this here as a template for simple non-recursive
> cases.  I suspect it's not a massive effort to make it recursive, when I
> have a moment I'll have a think about that too.  Any thoughts on how to
> improve are welcome!

Thanks for sharing!

Ludo’.