(propagated) 'inputs' depends on 'outputs'?

(propagated) 'inputs' depends on 'outputs'?

[zimoun]

Dear,

My question is: is it possible to specify 'inputs' only for specific 'outputs'?

Well, I think that the answer is no because build-time vs install-time
but I could have wrong and/or someone could have a fix for me. :-)


As an exemple, give a look at the package "git".  Because it has
several 'outputs' ("send-email", "svn", etc.), the list of "inputs"
provides e.g., "subversion" even if I am only interested by
"git:send-email".  This matters about closure (bandwidth, rebuild,
etc.).

I do not have an example at hand, but I think that if one package has
2 outputs '("out" "doc")' and the documentation requires heavy TeX
then I cannot work only on the "out" without downloading all the
documentation dependencies, i.e.,

   guix environment foo

could eat all my resources even if I am not interested by the
documentation part.


Another example is the package "emacs-helm-bibtex" which propagated
both Helm and Ivy completion backends.  Well, for example if the
'outputs' list is '("out" "helm" "ivy")' then the user could install
only the Ivy backend with:

   guix install emacs-helm-bibtex:ivy

without propagating the full 'emacs-helm' package.  Well, from my
understanding, the way is to create another package using 'inherit'
and remove the unnecessary part and then install with:

   guix install emacs-helm-bibtex-ivy

And same strategy the former Git example.


Well, it is already possible to test of some variables and set
different inputs,  See for example the package 'ncurse'.  Hopefully it
is possible because it is intensively used for cross-compiling. :-)


Thank you in advance for any tips. :-)

All the best,
simon

Re: (propagated) 'inputs' depends on 'outputs'?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2426 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> My question is: is it possible to specify 'inputs' only for specific
> 'outputs'?

No, but only because the two concepts are not directly related.

A derivation has inputs, an output doesn't, but it does have references.

Take git as an example.

→ guix build git
/gnu/store/76f7iracvzjqwzxldcdr9055ia1vdgw7-git-2.26.2-credential-netrc
/gnu/store/fmxq1pawy32zw0a0qb371a36lr58rwzf-git-2.26.2-gui
/gnu/store/axbf1db65bk316c44csnf6bgkzb4ilvx-git-2.26.2
/gnu/store/826y3r09kjl0fg1c51xwrlxvjd4kb8ff-git-2.26.2-send-email
/gnu/store/dw4c975zfnvabpzl0sg7n7fv8spdflv6-git-2.26.2-subtree
/gnu/store/sz27ibw6f3icyvwjvhlvqjn2kp2x6mbp-git-2.26.2-svn

→ guix gc --references /gnu/store/sz27ibw6f3icyvwjvhlvqjn2kp2x6mbp-git-2.26.2-svn
/gnu/store/0bnp4bh1cid0xb415n6h4iyl52cmwpgd-perl-term-readkey-2.38
/gnu/store/8zvc5mvk0xm3ygrxsgpyy5ilxb5rzjry-perl-5.30.2
/gnu/store/axbf1db65bk316c44csnf6bgkzb4ilvx-git-2.26.2
/gnu/store/fvhj74pghapbjvsvj27skvkra1by1965-bash-minimal-5.0.16
/gnu/store/sz27ibw6f3icyvwjvhlvqjn2kp2x6mbp-git-2.26.2-svn
/gnu/store/wn07ns1i8pqjfz92xkbicm5azg5jx2lc-subversion-1.14.0

→ guix gc --references /gnu/store/axbf1db65bk316c44csnf6bgkzb4ilvx-git-2.26.2
/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib
/gnu/store/09a5iq080g9b641jyl363dr5jkkvnhcn-python-3.8.2
/gnu/store/85lvkhmvr3a75j6rl4y7xj67qdnb9ssr-perl-cgi-4.47
/gnu/store/8zvc5mvk0xm3ygrxsgpyy5ilxb5rzjry-perl-5.30.2
/gnu/store/axbf1db65bk316c44csnf6bgkzb4ilvx-git-2.26.2
/gnu/store/d48261l1a5xvyibj5vlnravwlkwcil8c-perl-html-parser-3.72
/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31
/gnu/store/fvhj74pghapbjvsvj27skvkra1by1965-bash-minimal-5.0.16
/gnu/store/hcxpkksmbql6s4al8yy2myr25kh4cic0-openssl-1.1.1g
/gnu/store/imh5xxqw10dql4crlngbbjh4r24raf4j-expat-2.2.9
/gnu/store/qvahafxrr2mcl4anjxdkkprrvd4k0xjj-pcre2-10.34
/gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11
/gnu/store/wlz0dsvc00s59cbjlyb04v9rvxiprqsz-curl-7.69.1


Looking at different outputs, the references are different. If you're
just using the "out" output, then you don't need subversion in your
store, but if you're using the "svn" output, then you do, as that output
references an output for subversion.

The references for an output isn't something specified, but something
decided by what references that output actually contains.

Hope that helps,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: (propagated) 'inputs' depends on 'outputs'?

[Julien Lepiller]

Le 13 juin 2020 07:01:01 GMT-04:00, zimoun <zimon.toutoune@gmail.com> a écrit :
>Dear,
>
>My question is: is it possible to specify 'inputs' only for specific
>'outputs'?
>
>Well, I think that the answer is no because build-time vs install-time
>but I could have wrong and/or someone could have a fix for me. :-)

Exactly, no. You cannot separate inputs from outputs, because they are part of the same derivation. When you build an output, you actually build the complete derivation and there's no way to separate that in "this part builds out" and "this part builds doc", etc.

For propagated-inputs it would make some sense. A propagated input is just a normal input when building the package (so it contributes to every output). Its propagation behaviour only takes place when you want to install it in a profile though.

If foo propagates bar, anl you install foo:out, you will also get bar:out in your profile. Same if you install foo:bin.

It would make sense to only propagate for some outputs: suppose at runtime only foo:bin requires the propagation of bar. Since foo and bar are already built, it should be possible to restrict the propagation behaviour to that output. Foo:out would not bring in bar anymore, reducing the closure size. 

That's not implemented though.

>
>All the best,
>simon

Re: (propagated) 'inputs' depends on 'outputs'?

[zimoun]

Hi Chris,

Thank you for explaining.  It still miss a point.


On Sat, 13 Jun 2020 at 12:16, Christopher Baines <mail@cbaines.net> wrote:

> Looking at different outputs, the references are different. If you're
> just using the "out" output, then you don't need subversion in your
> store, but if you're using the "svn" output, then you do, as that output
> references an output for subversion.
>
> The references for an output isn't something specified, but something
> decided by what references that output actually contains.

I understand.

Back to the Git's example, there is still something I miss: I cannot
build Git from source (e.g. no substitutes) using "guix build git"
without downloading -- and possibly building too -- all the Subversion
stuff.

Other said, if I run "guix install git --no-substitutes" then I will
build all the Git outputs, so build all Subversion and other, and
finally only install the references of "out" (the rest could be garbage
collected).


Cheers,
simon

Re: (propagated) 'inputs' depends on 'outputs'?

[zimoun]

Bonjour Julien,

On Sat, 13 Jun 2020 at 07:38, Julien Lepiller <julien@lepiller.eu> wrote:

> Exactly, no. You cannot separate inputs from outputs, because they are
> part of the same derivation. When you build an output, you actually
> build the complete derivation and there's no way to separate that in
> "this part builds out" and "this part builds doc", etc.

So it means that I need to build all the outputs even if I am interested
in only one, right?

If I run "guix install foo:out --no-substitutes" then I potentially
build any other "outputs"" of foo, e.g., "doc" i.e., potentially
download a lot of TeX stuff, or in the case of Git, all the Subversion
stuff.  Right?

Even if at the end, only the references used by "foo:out" will be
tracked and all the others potentially garbage collected.  Right?


> It would make sense to only propagate for some outputs: suppose at
> runtime only foo:bin requires the propagation of bar. Since foo and
> bar are already built, it should be possible to restrict the
> propagation behaviour to that output. Foo:out would not bring in bar
> anymore, reducing the closure size.

Yes, it seems making sense to only propagated if the output needs it.
Well, if it is not implemented yet maybe it is because it is not really
necessary. :-)


Thank you for explaining.  It is clearer for me now.

Cheers,
simon

Re: (propagated) 'inputs' depends on 'outputs'?

[Julien Lepiller]

Le 16 juin 2020 08:41:58 GMT-04:00, zimoun <zimon.toutoune@gmail.com> a écrit :
>Bonjour Julien,
>
>On Sat, 13 Jun 2020 at 07:38, Julien Lepiller <julien@lepiller.eu>
>wrote:
>
>> Exactly, no. You cannot separate inputs from outputs, because they
>are
>> part of the same derivation. When you build an output, you actually
>> build the complete derivation and there's no way to separate that in
>> "this part builds out" and "this part builds doc", etc.
>
>So it means that I need to build all the outputs even if I am
>interested
>in only one, right?

Absolutely, but you shouldn't thenk of it as building multiple outputs, because you are really just building one thing: there is only one package object, one derivation. It happens to create multiple directories in the store, but there's only one thing you build.

>
>If I run "guix install foo:out --no-substitutes" then I potentially
>build any other "outputs"" of foo, e.g., "doc" i.e., potentially
>download a lot of TeX stuff, or in the case of Git, all the Subversion
>stuff.  Right?

Yes, because building foo:out doesn't make sense. You build foo and guix is nice enough to understand that's wgat you mean :)

>
>Even if at the end, only the references used by "foo:out" will be
>tracked and all the others potentially garbage collected.  Right?
>
>
>> It would make sense to only propagate for some outputs: suppose at
>> runtime only foo:bin requires the propagation of bar. Since foo and
>> bar are already built, it should be possible to restrict the
>> propagation behaviour to that output. Foo:out would not bring in bar
>> anymore, reducing the closure size.
>
>Yes, it seems making sense to only propagated if the output needs it.
>Well, if it is not implemented yet maybe it is because it is not really
>necessary. :-)

That cannot be automated because usually we use propagation when there is no direct reference. It wouldn't be useful otherwise.

>
>
>Thank you for explaining.  It is clearer for me now.
>
>Cheers,
>simon

Re: (propagated) 'inputs' depends on 'outputs'?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> On Sat, 13 Jun 2020 at 12:16, Christopher Baines <mail@cbaines.net> wrote:
>
>> Looking at different outputs, the references are different. If you're
>> just using the "out" output, then you don't need subversion in your
>> store, but if you're using the "svn" output, then you do, as that output
>> references an output for subversion.
>>
>> The references for an output isn't something specified, but something
>> decided by what references that output actually contains.
>
> I understand.
>
> Back to the Git's example, there is still something I miss: I cannot
> build Git from source (e.g. no substitutes) using "guix build git"
> without downloading -- and possibly building too -- all the Subversion
> stuff.

That’s correct.  Outputs are buckets into which we drop build
artifacts.  We will build everything with all inputs in one derivation
and then move stuff into output buckets.  This means that you can
download independent outputs individually, but they are still all the
result of that *single* derivation.  So to build any of the outputs you
will need to build that derivation, even if it also results in other
outputs that you don’t care about.

-- 
Ricardo

Re: (propagated) 'inputs' depends on 'outputs'?

[zimoun]

On Tue, 16 Jun 2020 at 10:28, Julien Lepiller <julien@lepiller.eu> wrote:

>>If I run "guix install foo:out --no-substitutes" then I potentially
>>build any other "outputs"" of foo, e.g., "doc" i.e., potentially
>>download a lot of TeX stuff, or in the case of Git, all the Subversion
>>stuff.  Right?
>
> Yes, because building foo:out doesn't make sense. You build foo and
> guix is nice enough to understand that's wgat you mean :)

I understand.  I do not know if it does not make sense but I
understand. :-)

Well, in this lockdown period, my bandwith and computing resources were
limited and building e.g. all Subversion (or any heavy doc packages)
when I wanted only small output e.g. Git:out (without the svn support)
appeared to me frustrating.

Now it is clear for me, it is by design.  Thank you for the
explanations.


>>> It would make sense to only propagate for some outputs: suppose at
>>> runtime only foo:bin requires the propagation of bar. Since foo and
>>> bar are already built, it should be possible to restrict the
>>> propagation behaviour to that output. Foo:out would not bring in bar
>>> anymore, reducing the closure size.
>>
>>Yes, it seems making sense to only propagated if the output needs it.
>>Well, if it is not implemented yet maybe it is because it is not really
>>necessary. :-)
>
> That cannot be automated because usually we use propagation when there
> is no direct reference. It wouldn't be useful otherwise.

I am not sure to get the point.  From my understanding, it could be
possible to add information to native-inputs, inputs and
propagated-inputs, e.g.,

      ;; For 'git-svn'.
      ("subversion" ,subversion "svn")
      ("perl-term-readkey" ,perl-term-readkey "svn")

or whatever other mean.  Then it becomes possible to only build e.g.,
git:svn and/or propagate specific inputs depending on the outputs.

In the light of your explanation, it does not make sense for
native-inputs and inputs because Guix is building only one thing
(package object, derivation).  But it could make sense for the
propagated-inputs.  Well, the same way that "outputs" are not automatic
but specified somehow by "arguments".

However, it is not implemented and now after the explanations, I do not
know it is is worth.  Well, it is another way to see package
parameters. :-)


Thank you again for the explanations.


Cheers,
simon

Re: (propagated) 'inputs' depends on 'outputs'?

[zimoun]

Hi Ricardo,

On Tue, 16 Jun 2020 at 17:34, Ricardo Wurmus <rekado@elephly.net> wrote:

> That’s correct.  Outputs are buckets into which we drop build
> artifacts.  We will build everything with all inputs in one derivation
> and then move stuff into output buckets.  This means that you can
> download independent outputs individually, but they are still all the
> result of that *single* derivation.  So to build any of the outputs you
> will need to build that derivation, even if it also results in other
> outputs that you don’t care about.

Thank you for the explanations.  I understand even if time to time I
feel frustrated to download and/or build a lot of unrelated packages
that I am really interested in.  But hey, it is how the thing works. :-)

Now all is clear.

Cheers,
simon

Re: (propagated) 'inputs' depends on 'outputs'?

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 548 bytes --]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> I am not sure to get the point.  From my understanding, it could be
> possible to add information to native-inputs, inputs and
> propagated-inputs, e.g.,
>
>       ;; For 'git-svn'.
>       ("subversion" ,subversion "svn")
>       ("perl-term-readkey" ,perl-term-readkey "svn")
>
> or whatever other mean.  Then it becomes possible to only build e.g.,
> git:svn and/or propagate specific inputs depending on the outputs.

The ‘TODO’ file, which is really a museum, has this:


[-- Attachment #2: Type: text/x-org, Size: 258 bytes --]

** extend ‘propagated-build-inputs’ with support for multiple outputs

#+BEGIN_SRC scheme
  (outputs '("out" "include"))
  (propagated-build-inputs
    `(((("i1" ,p1 "o1")
        ("i2" ,p2))
       => "include")
      ("i3" ,p3)))
#+END_SRC


[-- Attachment #3: Type: text/plain, Size: 46 bytes --]


Is this what you have in mind?

Ludo’.

Re: (propagated) 'inputs' depends on 'outputs'?

[zimoun]

Hi Ludo,

On Fri, 19 Jun 2020 at 22:47, Ludovic Courtès <ludo@gnu.org> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:

>>       ;; For 'git-svn'.
>>       ("subversion" ,subversion "svn")
>>       ("perl-term-readkey" ,perl-term-readkey "svn")

> The ‘TODO’ file, which is really a museum, has this:
>
> ** extend ‘propagated-build-inputs’ with support for multiple outputs
>
> #+BEGIN_SRC scheme
>   (outputs '("out" "include"))
>   (propagated-build-inputs
>     `(((("i1" ,p1 "o1")
>         ("i2" ,p2))
>        => "include")
>       ("i3" ,p3)))
> #+END_SRC
>
>
> Is this what you have in mind?

Yes.  That's why it is important to go to museum. ;-)
But if the idea is old and has not happened yet, maybe it is not
worth... For Git, for sure! :-)


All the best,
simon