Package variant defined in manifest not visible by Guix

Package variant defined in manifest not visible by Guix

[Philippe SWARTVAGHER]

Hello,

I'm trying to use a manifest file to define a package variant which adds
an option to the configure phase:

```

(use-modules (inria storm))

(define starpu-maxnodes1
  (package
    (inherit starpu)
    (arguments
      (substitute-keyword-arguments (package-arguments starpu)
                                    ((#:configure-flags flags '())
                                     `(cons "--enable-maxnodes=1"
,flags))))))

```

When I'm trying to build this variant, guix can't find it:

```

% guix build -c 2 -m starpu-maxnodes1.scm starpu-maxnodes1
guix build: erreur : starpu-maxnodes1 : paquet inconnu

```

I don't see what I'm doing wrong. Any idea ?

Just if you want to try these exact commands, StarPU is a package
defined in this channel:

```

(channel
        (name 'guix-hpc)
        (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git")
        (commit
          "fa8cc70bc428c08a625a533f778537314be64ce7"))

```

(I didn't subscribe to help-guix mailing list, so please keep me as
recipient of your answers.)

-- 
Philippe SWARTVAGHER

PhD Student
TADaaM team, Inria Bordeaux Sud-Ouest

Re: Package variant defined in manifest not visible by Guix

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1954 bytes --]

On Mon, Aug 16, 2021 at 10:08:34AM +0200, Philippe SWARTVAGHER wrote:
> Hello,
> 
> I'm trying to use a manifest file to define a package variant which adds
> an option to the configure phase:
> 
> ```
> 
> (use-modules (inria storm))
> 
> (define starpu-maxnodes1
>   (package
>     (inherit starpu)
>     (arguments
>       (substitute-keyword-arguments (package-arguments starpu)
>                                     ((#:configure-flags flags '())
>                                      `(cons "--enable-maxnodes=1"
> ,flags))))))
> 
> ```
> 
> When I'm trying to build this variant, guix can't find it:
> 
> ```
> 
> % guix build -c 2 -m starpu-maxnodes1.scm starpu-maxnodes1
> guix build: erreur : starpu-maxnodes1 : paquet inconnu
> 
> ```
> 
> I don't see what I'm doing wrong. Any idea ?
> 
> Just if you want to try these exact commands, StarPU is a package
> defined in this channel:

Inherit inherits everything except for what you override, so your
package can be refered to as starpu-maxnodes1 when using it as an input
for another package (for example), but to build it from the command line
it still has the original name of "starpu".

Also, I haven't tried running 'guix build -m manifest.scm
package-in-manifest', so I don't know if that works or not.

> ```
> 
> (channel
>         (name 'guix-hpc)
>         (url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git")
>         (commit
>           "fa8cc70bc428c08a625a533f778537314be64ce7"))
> 
> ```
> 
> (I didn't subscribe to help-guix mailing list, so please keep me as
> recipient of your answers.)
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Package variant defined in manifest not visible by Guix

[Ricardo Wurmus]

Hi,

> I'm trying to use a manifest file to define a package variant 
> which adds
> an option to the configure phase:
>
> ```
>
> (use-modules (inria storm))
>
> (define starpu-maxnodes1
>   (package
>     (inherit starpu)
>     (arguments
>       (substitute-keyword-arguments (package-arguments starpu)
>                                     ((#:configure-flags flags 
>                                     '())
>                                      `(cons 
>                                      "--enable-maxnodes=1"
> ,flags))))))
>
> ```
>
> When I'm trying to build this variant, guix can't find it:
>
> ```
>
> % guix build -c 2 -m starpu-maxnodes1.scm starpu-maxnodes1
> guix build: erreur : starpu-maxnodes1 : paquet inconnu
>
> ```
>
> I don't see what I'm doing wrong. Any idea ?

There seems to be a small misunderstanding.  You defined a package 
variant, but that’s not a manifest.  A manifest describes the 
complete contents of a profile; i.e. it’s a list of packages that 
Guix should install.

Here are three different ways that should work for you:

1) Build a single package from a file.

Here we define the package variant in a file and make sure that 
the file evaluates to the package value:

--8<---------------cut here---------------start------------->8---
(use-modules (inria storm))

(define starpu-maxnodes1
  (package
    (inherit starpu)
    (arguments
      (substitute-keyword-arguments (package-arguments starpu)
                                    ((#:configure-flags flags '())
                                     `(cons "--enable-maxnodes=1" 
                                     ,flags))))))

;; Make sure the file evaluates to the package value.
;; The definition alone has no specified return value.
starpu-maxnodes1
--8<---------------cut here---------------end--------------->8---

Now we can build the package specified in the file:

    guix build --file=this-file.scm


2) Build a whole manifest from a file.

Here the file must evaluate to a manifest value, not just a single 
package.

--8<---------------cut here---------------start------------->8---
(use-modules (inria storm))

(define starpu-maxnodes1
  (package
    (inherit starpu)
    (arguments
      (substitute-keyword-arguments (package-arguments starpu)
                                    ((#:configure-flags flags '())
                                     `(cons "--enable-maxnodes=1" 
                                     ,flags))))))

(packages->manifest (list starpu-maxnodes1))
--8<---------------cut here---------------end--------------->8---

Then build the profile from the manifest file:

    guix build --manifest=this-file.scm


3) Create a module and use it however you want.

You can make the custom package available to any Guix command by 
putting it into a Guile module and then informing Guix about the 
module.  This is a little more effort as you need more boilerplate 
code to define the module (and the file name needs to match the 
moule header, etc).


--8<---------------cut here---------------start------------->8---
(define-module (my packages storm)
 #:use-module (gnu packages)
 #:use-module (guix packages)
 #:use-module (guix utils)
 ;; …
 #:use-module (inria storm)
 #:export (starpu-maxnodes1))

(define starpu-maxnodes1
  (package
    (inherit starpu)
    (arguments
      (substitute-keyword-arguments (package-arguments starpu)
                                    ((#:configure-flags flags '())
                                     `(cons "--enable-maxnodes=1" 
                                     ,flags))))))
--8<---------------cut here---------------end--------------->8---

Put this in a file “my/packages/storm.scm” and then set the 
GUIX_PACKAGE_PATH environment variable to the directory containing 
“my” (e.g. $HOME/code/guix/custom containing 
$HOME/code/guix/custom/my/packages/storm.scm).

Note that you haven’t overridden the “name” field of the original 
“starpu” package, so on the command line the package cannot be 
distinguished (unless you use “-e '(@ (my packages storm) 
starpu-maxnodes1)'”).  Do this to also change the name on the 
command line:

--8<---------------cut here---------------start------------->8---
(define starpu-maxnodes1
  (package
    (inherit starpu)
    (name "starpu-maxnodes1")
    (arguments
      (substitute-keyword-arguments (package-arguments starpu)
                                    ((#:configure-flags flags '())
                                     `(cons "--enable-maxnodes=1" 
                                     ,flags))))))
--8<---------------cut here---------------end--------------->8---

Hope this helps!

-- 
Ricardo

Re: Package variant defined in manifest not visible by Guix

[zimoun]

Hi Philippe,

On Mon, 16 Aug 2021 at 16:34, Ricardo Wurmus <rekado@elephly.net> wrote:

> 3) Create a module and use it however you want.

[...]

> Put this in a file “my/packages/storm.scm” and then set the 
> GUIX_PACKAGE_PATH environment variable to the directory containing 
> “my” (e.g. $HOME/code/guix/custom containing 
> $HOME/code/guix/custom/my/packages/storm.scm).

Do not miss the CLI option ’--load-path’.  Instead of setting the
environment variable GUIX_PACKAGE_PATH, you can also do:

  guix build --load-path=my/packages/storm.scm <your-variant-package-name>


> Note that you haven’t overridden the “name” field of the original 
> “starpu” package, so on the command line the package cannot be 
> distinguished (unless you use “-e '(@ (my packages storm) 
> starpu-maxnodes1)'”).  Do this to also change the name on the 
> command line:
>
> --8<---------------cut here---------------start------------->8---
> (define starpu-maxnodes1
>   (package
>     (inherit starpu)
>     (name "starpu-maxnodes1")
>     (arguments
>       (substitute-keyword-arguments (package-arguments starpu)
>                                     ((#:configure-flags flags '())
>                                      `(cons "--enable-maxnodes=1" 
>                                      ,flags))))))
> --8<---------------cut here---------------end--------------->8---

In the initial error message, the package is unknown because the package
name ’starpu-maxnodes1’ does not exist, it is the name of the symbol.
And the symbol ’starpu-maxnodes1’ has the name ’starpu’.  Does it make
sense?


Cheers,
simon

Re: Package variant defined in manifest not visible by Guix

[Philippe SWARTVAGHER]

Hello,

Le 16/08/2021 à 16:34, Ricardo Wurmus a écrit :
> There seems to be a small misunderstanding.  You defined a package
> variant, but that’s not a manifest.  A manifest describes the complete
> contents of a profile; i.e. it’s a list of packages that Guix should
> install.
>
> Here are three different ways that should work for you:
>
> 1) Build a single package from a file.
>
> Now we can build the package specified in the file:
>
>    guix build --file=this-file.scm
>
>
> 2) Build a whole manifest from a file.
>
> Here the file must evaluate to a manifest value, not just a single
> package.

Ok, I was missing the line to evaluate to a manifest, instead of a
single package.


> Then build the profile from the manifest file:
>
>    guix build --manifest=this-file.scm
>
>
> 3) Create a module and use it however you want.
>
> You can make the custom package available to any Guix command by
> putting it into a Guile module and then informing Guix about the
> module.  This is a little more effort as you need more boilerplate
> code to define the module (and the file name needs to match the moule
> header, etc).
>
> Hope this helps!

Sure, it helps ! Thanks a lot ! However, I still have some questions:

- I don't really see the difference, or more precisely: the difference
of goal, between defining the package in a simple file (solution 1) and
using a manifest (solution 2).

- The following command builds the package as defined in the *file*,
without running tests:

guix build -c 2 -f starpu-maxnodes1-file.scm
--without-tests=starpu-maxnodes1

So package transformations seem to work. However:

guix build -c 2 -f starpu-maxnodes1-file.scm chameleon
--with-input=starpu=starpu-maxnodes1 --with-input=openblas=mkl
--without-tests=starpu-maxnodes1

doesn't work: starpu-maxnodes1 is an unknown package...

The same happens if I use a manifest instead of a file to define
starpu-maxnodes1:

guix build -c 2 -m starpu-maxnodes1.scm --with-input=openmpi=nmad
--without-tests=starpu-maxnodes1

works (without running tests), however:

guix build -c 2 -m starpu-maxnodes1.scm chameleon
--with-input=starpu=starpu-maxnodes1 --with-input=openblas=mkl
--with-input=openmpi=nmad --without-tests=starpu-maxnodes1

doesn't.

I guess I have to put all transformations of several packages in a
single manifest, one cannot combine manifest and CLI options; packages
defined in manifest aren't available for use in CLI transformations
(although the working --without-tests seems to be a counter argument).
Am I right ?

Thanks again for your help !

-- 
Philippe SWARTVAGHER

PhD Student
TADaaM team, Inria Bordeaux Sud-Ouest

Re: Package variant defined in manifest not visible by Guix

[zimoun]

Hi Philippe,


On Wed, 18 Aug 2021 at 10:57, Philippe SWARTVAGHER <philippe.swartvagher@inria.fr> wrote:
> Le 16/08/2021 à 16:34, Ricardo Wurmus a écrit :

>> There seems to be a small misunderstanding.  You defined a package
>> variant, but that’s not a manifest.  A manifest describes the complete
>> contents of a profile; i.e. it’s a list of packages that Guix should
>> install.
>>
>> Here are three different ways that should work for you:
>>
>> 1) Build a single package from a file.
>>
>> Now we can build the package specified in the file:
>>
>>    guix build --file=this-file.scm
>>
>>
>> 2) Build a whole manifest from a file.
>>
>> Here the file must evaluate to a manifest value, not just a single
>> package.
>
> Ok, I was missing the line to evaluate to a manifest, instead of a
> single package.
>
>
>> Then build the profile from the manifest file:
>>
>>    guix build --manifest=this-file.scm
>>
>>
>> 3) Create a module and use it however you want.
>>
>> You can make the custom package available to any Guix command by
>> putting it into a Guile module and then informing Guix about the
>> module.  This is a little more effort as you need more boilerplate
>> code to define the module (and the file name needs to match the moule
>> header, etc).
>>
>> Hope this helps!
>
> Sure, it helps ! Thanks a lot ! However, I still have some questions:
>
> - I don't really see the difference, or more precisely: the difference
> of goal, between defining the package in a simple file (solution 1) and
> using a manifest (solution 2).

Solution #1 (option --file) builds the package defined by FILE.
Solution #2 (option --manifest) builds the packages listed in FILE.

> - The following command builds the package as defined in the *file*,
> without running tests:
>
> guix build -c 2 -f starpu-maxnodes1-file.scm
> --without-tests=starpu-maxnodes1
>
> So package transformations seem to work. However:
>
> guix build -c 2 -f starpu-maxnodes1-file.scm chameleon
> --with-input=starpu=starpu-maxnodes1 --with-input=openblas=mkl
> --without-tests=starpu-maxnodes1
>
> doesn't work: starpu-maxnodes1 is an unknown package...
>
> The same happens if I use a manifest instead of a file to define
> starpu-maxnodes1:
>
> guix build -c 2 -m starpu-maxnodes1.scm --with-input=openmpi=nmad
> --without-tests=starpu-maxnodes1
>
> works (without running tests), however:
>
> guix build -c 2 -m starpu-maxnodes1.scm chameleon
> --with-input=starpu=starpu-maxnodes1 --with-input=openblas=mkl
> --with-input=openmpi=nmad --without-tests=starpu-maxnodes1
>
> doesn't.

I think what you want is the option --load-path.  Create a folder and
put the file starpu-maxnodes1.scm inside; for instance
’/tmp/example/starpu-maxnodes1.scm’.  Then, the command-line:

  guix build -c 2 -L /tmp/example chameleon --with-input=starpu=starpu-maxnodes1

should do the job.  In this case, the file
’/tmp/example/starpu-maxnodes1.scm’ does not have to return a package or
a manifest but instead define a module.  Does it make sense?

> I guess I have to put all transformations of several packages in a
> single manifest, one cannot combine manifest and CLI options; packages
> defined in manifest aren't available for use in CLI transformations
> (although the working --without-tests seems to be a counter argument).
> Am I right ?

As Ricardo explained, the aim manifest is to describe the complete
contents of a profile, so from my understanding, the manifest should
also contain the transformation.

Well, I think what you want is the option ’--load-path’ and not a
manifest or the option ’--file’.

Hope that helps,
simon