'guix install' downloads all outputs for packages with cmake build system.

'guix install' downloads all outputs for packages with cmake build system.

[nigko]

Hello Guix!

I have found that a command 'guix install <package>' with a package having
  (1) cmake build system
  (2) (length outputs) > 1
would download all outputs of the package from a substitute server, not
just "out" output as it should.

For example, a package 'x265', which has 'static' and 'out' outputs and
cmake build system gives:
------------------------------------------------------------------------
# don't forget to insure that /gnu/store does not already contain
# x265's static output before running this command:
$ guix install x265
The following package will be installed:
   x265 3.5
...
2,8 MB will be downloaded
 x265-3.5-static  1.3MiB
 x265-3.5  1.4MiB
...
------------------------------------------------------------------------

There are 84 packages in the main guix repository satisfying (1) and (2)
criteria . Here is the full list of their names for you to try:

apache-arrow
argagg
arx-libertatis
bctoolbox
bcunit
belcard
belle-sip
belr
ceph
cmake
dealii
dealii-openmpi
dhtnet
dino
discregrid
fcitx
fcitx5-chinese-addons
fcitx5-gtk
gdcm
gemmi
gnucash
grpc
insight-toolkit
java-antlr4-runtime-cpp
libavif
libdecaf
libftdi
libgit2
liblinphone
libssh
lime
llvm
llvm-for-mesa
llvm-for-rocm
mariadb
mediastreamer2
mmg
mympd
openscenegraph
ortp
protobuf
pt-scotch
pt-scotch-shared
pt-scotch32
qtbase
qtdeclarative
scotch
scotch-shared
scotch32
sortmerna
supercollider
tidy
tidy-html
transmission
webkitgtk
webkitgtk-for-gtk3
webkitgtk-with-libsoup2
weechat
woff2
wpewebkit
x265

An interesting bit about this abnormal behavior was gained from the
discusstion on IRC. It turns out that the command
'guix install --no-grafts <package>' with --no-grafts flag does not
download all the outputs of the package but only the "out" output,
as one would expect from 'guix install' command. So the issue seems
to be related to grafts. What do you think?

Regards,
Nigko

Re: 'guix install' downloads all outputs for packages with cmake build system.

[Simon Tournier]

Hi,

On Wed, 09 Oct 2024 at 18:55, nigko <nigko.yerden@gmail.com> wrote:

> ------------------------------------------------------------------------
> # don't forget to insure that /gnu/store does not already contain
> # x265's static output before running this command:
> $ guix install x265
> The following package will be installed:
>    x265 3.5
> ...
> 2,8 MB will be downloaded
>  x265-3.5-static  1.3MiB
>  x265-3.5  1.4MiB
> ...
> ------------------------------------------------------------------------

Hum. I am not able to reproduce.

First, let be sure nothing is already in the store.

--8<---------------cut here---------------start------------->8---
$ guix gc --list-dead | grep x265-3.5-static
finding garbage collector roots...
determining live/dead paths...

$ guix gc --list-live | grep x265-3.5-static
finding garbage collector roots...
determining live/dead paths...

$ find /gnu/store -maxdepth 1 -type d -name "x256-3.5-static" -print
--8<---------------cut here---------------end--------------->8---

Second, try to install:

--8<---------------cut here---------------start------------->8---
$ guix install x265 --dry-run
guix install: warning: Your Guix installation is 92 days old.
guix install: warning: Consider running 'guix pull' followed by
'guix package -u' to get up-to-date packages and security updates.

The following package would be installed:
   x265 3.5
--8<---------------cut here---------------end--------------->8---

And ’static’ output does not seem required.

> An interesting bit about this abnormal behavior was gained from the
> discusstion on IRC. It turns out that the command
> 'guix install --no-grafts <package>' with --no-grafts flag does not
> download all the outputs of the package but only the "out" output,
> as one would expect from 'guix install' command. So the issue seems
> to be related to grafts. What do you think?

Could you give your Guix revision?  Mine is 056910e; please note x265
had not be modified since a while and the package isn’t grafted.

--8<---------------cut here---------------start------------->8---
$ guix build x265 -d
/gnu/store/5v0d3wf3rhy69i1fl0jdchpmwy5wxbas-x265-3.5.drv

$ guix build x265 -d --no-grafts
/gnu/store/5v0d3wf3rhy69i1fl0jdchpmwy5wxbas-x265-3.5.drv
--8<---------------cut here---------------end--------------->8---

Hum, I don’t know where could the observed behaviour come from?

Cheers,
simon

Re: 'guix install' downloads all outputs for packages with cmake build system.

[Nigko Yerden]

Simon Tournier wrote:
> Hum. I am not able to reproduce.
> 
> First, let be sure nothing is already in the store.
> 
> --8<---------------cut here---------------start------------->8---
> $ guix gc --list-dead | grep x265-3.5-static
> finding garbage collector roots...
> determining live/dead paths...
> 
> $ guix gc --list-live | grep x265-3.5-static
> finding garbage collector roots...
> determining live/dead paths...
> 
> $ find /gnu/store -maxdepth 1 -type d -name "x256-3.5-static" -print
> --8<---------------cut here---------------end--------------->8---
> 
> Second, try to install:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix install x265 --dry-run
> guix install: warning: Your Guix installation is 92 days old.
> guix install: warning: Consider running 'guix pull' followed by
> 'guix package -u' to get up-to-date packages and security updates.
> 
> The following package would be installed:
>     x265 3.5
> --8<---------------cut here---------------end--------------->8---
> 
> And ’static’ output does not seem required.
> 
>> An interesting bit about this abnormal behavior was gained from the
>> discusstion on IRC. It turns out that the command
>> 'guix install --no-grafts <package>' with --no-grafts flag does not
>> download all the outputs of the package but only the "out" output,
>> as one would expect from 'guix install' command. So the issue seems
>> to be related to grafts. What do you think?
> 
> Could you give your Guix revision?  Mine is 056910e; please note x265
> had not be modified since a while and the package isn’t grafted.
> 
> --8<---------------cut here---------------start------------->8---
> $ guix build x265 -d
> /gnu/store/5v0d3wf3rhy69i1fl0jdchpmwy5wxbas-x265-3.5.drv
> 
> $ guix build x265 -d --no-grafts
> /gnu/store/5v0d3wf3rhy69i1fl0jdchpmwy5wxbas-x265-3.5.drv
> --8<---------------cut here---------------end--------------->8---
> 
> Hum, I don’t know where could the observed behaviour come from?
> 
> Cheers,
> simon

Hello Simon,

Now I'm on 47ab553178eb00d02ba9588ed1f7df44fb2f7c4c guix commit and
still have the problem:
----begin-----------------------------------------------------------
$ guix build x265 -d
3,9 MB will be downloaded:
   /gnu/store/1h81psijhx3l8p88ds66q7a001rgvcl2-x265-3.5
   /gnu/store/4zw8h8za4z1cg4k5iz5lha6v1s9wzwkz-x265-3.5-static
substituting /gnu/store/4zw8h8za4z1cg4k5iz5lha6v1s9wzwkz-x265-3.5-static...
downloading from https://cuirass.genenetwork.org/nar/zstd/4zw8h8za4z1cg4k5iz5lha6v1s9wzwkz-x265-3.5-static ...
  x265-3.5-static  2.2MiB

substituting /gnu/store/1h81psijhx3l8p88ds66q7a001rgvcl2-x265-3.5...
downloading from https://cuirass.genenetwork.org/nar/zstd/1h81psijhx3l8p88ds66q7a001rgvcl2-x265-3.5 ...
  x265-3.5  1.5MiB

/gnu/store/7555d85ldgn3vgby3sdgl0rf9ac1b6mv-x265-3.5.drv
----end-------------------------------------------------------------

Moreover, since then I have realized that the problem is not specific
to packages with cmake build system but concerns all packages with
multiple outputs.

Your Guix installation is 92 days old. Probably this is the reason why
you wasn't able to reproduce the problem. As far as I remember this
problem have arisen since the latest core packages update.

Regards,
Nigko

Re: 'guix install' downloads all outputs for packages with cmake build system.

[Rutherther]

Hello Nigko,

> Hello Simon,

> Now I'm on 47ab553178eb00d02ba9588ed1f7df44fb2f7c4c guix commit and
> still have the problem:
> ----begin-----------------------------------------------------------
> $ guix build x265 -d

You are building the package. I think I've already explained it on
the IRC few weeks ago: due to how guix is made, build always builds
all outputs. That is because when you are building the package,
there is just one build process and in middle of that process
some files are copied to other outputs. With `guix build`, you cannot
even specify the outputs. So even on substitution, you get all
of them. That's what you would get if nothing was substituted
and the build ran on your system.

> Moreover, since then I have realized that the problem is not specific
> to packages with cmake build system but concerns all packages with
> multiple outputs.

Yes, with guix build it is for all packages, is expected,
and there is likely no way around it. Please provide guix install
example, where this happens, that's what you reported as an issue earlier.
With install, you no longer need to get all outputs,
when you do not need to build the package,
so you should be able to get just one output as long as
it is substituted. So indeed, if install substitutes all outputs
for some packages, something is wrong and it should be possible to fix it.

Rutherther
Date: Wed, 11 Dec 2024 09:54:36 +0100
Message-ID: <878qsmppkj.fsf@ditigal.xyz>

Re: 'guix install' downloads all outputs for packages with cmake build system.

[nigko]

Hello Rutherther,

On Wed, Dec 11, 2024 at 1:54 PM Rutherther <rutherther@ditigal.xyz> wrote:
> You are building the package. I think I've already explained it on
> the IRC few weeks ago: due to how guix is made, build always builds
> all outputs. That is because when you are building the package,
> there is just one build process and in middle of that process
> some files are copied to other outputs. With `guix build`, you cannot
> even specify the outputs. So even on substitution, you get all
> of them. That's what you would get if nothing was substituted
> and the build ran on your system.
>
> > Moreover, since then I have realized that the problem is not specific
> > to packages with cmake build system but concerns all packages with
> > multiple outputs.
>
> Yes, with guix build it is for all packages, is expected,
> and there is likely no way around it. Please provide guix install
> example, where this happens, that's what you reported as an issue earlier.
> With install, you no longer need to get all outputs,
> when you do not need to build the package,
> so you should be able to get just one output as long as
> it is substituted. So indeed, if install substitutes all outputs
> for some packages, something is wrong and it should be possible to fix it.

You are right. Here is the output of guix install:
-------------------------begin--------------------------------------------------
$ guix install x265
guix install: warning: Your Guix installation is 16 days old.
guix install: warning: Consider running 'guix pull' followed by
'guix package -u' to get up-to-date packages and security updates.

The following package will be installed:
   x265 3.5

substitute: updating substitutes from
'https://cuirass.genenetwork.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
3.9 MB will be downloaded
 x265-3.5-static  2.2MiB
 x265-3.5  1.5MiB
The following derivation will be built:
  /gnu/store/5vpq2pqp6s4ig6m0mcwrcbhv22ks2i9j-profile.drv
...
-------------------------end----------------------------------------------------

And here is the version with --no-grafts option (after cleaning the
store, of course):
-------------------------begin--------------------------------------------------
$ guix install --no-grafts x265
guix install: warning: Your Guix installation is 16 days old.
guix install: warning: Consider running 'guix pull' followed by
'guix package -u' to get up-to-date packages and security updates.

The following package will be installed:
   x265 3.5

The following derivation will be built:
  /gnu/store/vzs886g803pzp0wfg94z6ckqlyl2z16v-profile.drv

1.6 MB will be downloaded
 x265-3.5  1.5MiB
generating GLib schema cache...
...
-------------------------end----------------------------------------------------

Regards,
Nigko

Re: 'guix install' downloads all outputs for packages with cmake build system.

[nigko]

Here is an example of problematic package with gnu-build-system:

----------------------begin--------------------------------------------
$ guix install libsmf
guix install: warning: Your Guix installation is 16 days old.
guix install: warning: Consider running 'guix pull' followed by
'guix package -u' to get up-to-date packages and security updates.

The following package will be installed:
   libsmf 1.3

substitute: updating substitutes from
'https://cuirass.genenetwork.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
0.1 MB will be downloaded
 libsmf-1.3-static  21KiB
 libsmf-1.3  36KiB
The following derivation will be built:
  /gnu/store/hkbg1bppzchxzcc9z3mx7hmn1mz8v97p-profile.drv

building CA certificate bundle...
...
----------------------end----------------------------------------------

With --no-grafts option:
----------------------begin--------------------------------------------
$ guix install --no-grafts libsmf
guix install: warning: Your Guix installation is 16 days old.
guix install: warning: Consider running 'guix pull' followed by
'guix package -u' to get up-to-date packages and security updates.

The following package will be installed:
   libsmf 1.3

substitute: updating substitutes from
'https://cuirass.genenetwork.org'... 100.0%
The following derivation will be built:
  /gnu/store/qhxnykr6r0zkw5s030yg2ham9fsvjw04-profile.drv

0.0 MB will be downloaded
 libsmf-1.3  36KiB
generating GLib schema cache...
...
----------------------end----------------------------------------------

Regards,
Nigko