Design decision behind inputs/native-inputs/propagated-inputs

Design decision behind inputs/native-inputs/propagated-inputs

[Steven Allen]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

All,

I just attended David Thompson's talk about Guix and asked some
questions about the difference between inputs, propagated-inputs, and
native-inputs. I believe I now understand what each does but am unclear
as to why.

Currently, I use Arch. On Arch, we have makedepends and depends where
only depends are kept at runtime. On Guix, this distinction is detected
at build time by searching the output files for references into the
inputs. However, unless I'm mistaken, native-inputs are *usually* build
dependencies and inputs are *usually* runtime dependencies. So, my
question is why not:

1. Get rid of the automagic run/build dependency detection.
2. Have:
  a. private-inputs -- runtime dependencies not linked into the environment
  b. propagated-inputs -- no change
  c. build-inputs -- always native, never included in the final output

Specifically, what is the use case for non-native build-only dependencies 
(inputs that are not included in the final package) and native runtime
dependencies (native-inputs that *are* included in the final package)?
Alternatively, am I completely missing the point?

P.S. Please CC, I'm not subscribed.

-- 
Steven Allen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Design decision behind inputs/native-inputs/propagated-inputs

[Ludovic Courtès]

Hello,

Steven Allen <steven@stebalien.com> skribis:

> I just attended David Thompson's talk about Guix and asked some

Cool, glad that it led you to contact us!  ;-)

> Currently, I use Arch. On Arch, we have makedepends and depends where
> only depends are kept at runtime. On Guix, this distinction is detected
> at build time by searching the output files for references into the
> inputs. However, unless I'm mistaken, native-inputs are *usually* build
> dependencies and inputs are *usually* runtime dependencies. So, my
> question is why not:
>
> 1. Get rid of the automagic run/build dependency detection.
> 2. Have:
>   a. private-inputs -- runtime dependencies not linked into the environment
>   b. propagated-inputs -- no change
>   c. build-inputs -- always native, never included in the final output
>
> Specifically, what is the use case for non-native build-only dependencies 
> (inputs that are not included in the final package) and native runtime
> dependencies (native-inputs that *are* included in the final package)?
> Alternatively, am I completely missing the point?

The description of these three things has been improved in the manual on
‘master’, compared to what’s on the on-line manual:

--8<---------------cut here---------------start------------->8---
     ‘inputs’ (default: ‘'()’)
     ‘native-inputs’ (default: ‘'()’)
     ‘propagated-inputs’ (default: ‘'()’)
          These fields list dependencies of the package.  Each one is a
          list of tuples, where each tuple has a label for the input (a
          string) as its first element, a package, origin, or derivation
          as its second element, and optionally the name of the output
          thereof that should be used, which defaults to ‘"out"’ (*note
          Packages with Multiple Outputs::, for more on package
          outputs).  For example, the list below specifies 3 inputs:

               `(("libffi" ,libffi)
                 ("libunistring" ,libunistring)
                 ("glib:bin" ,glib "bin"))  ;the "bin" output of Glib

          The distinction between ‘native-inputs’ and ‘inputs’ is
          necessary when considering cross-compilation.  When
          cross-compiling, dependencies listed in ‘inputs’ are built for
          the _target_ architecture; conversely, dependencies listed in
          ‘native-inputs’ are built for the architecture of the _build_
          machine.

          ‘native-inputs’ is typically where you would list tools needed
          at build time but not at run time, such as Autoconf, Automake,
          pkg-config, Gettext, or Bison.  ‘guix lint’ can report likely
          mistakes in this area (*note Invoking guix lint::).

          Lastly, ‘propagated-inputs’ is similar to ‘inputs’, but the
          specified packages will be force-installed alongside the
          package they belong to (*note ‘guix package’:
          package-cmd-propagated-inputs, for information on how ‘guix
          package’ deals with propagated inputs.)

          For example this is necessary when a C/C++ library needs
          headers of another library to compile, or when a pkg-config
          file refers to another one via its ‘Requires’ field.

          Another example where ‘propagated-inputs’ is useful is for
          languages that lack a facility to record the run-time search
          path akin to ELF’s ‘RUNPATH’; this includes Guile, Python,
          Perl, GHC, and more.  To ensure that libraries written in
          those languages can find library code they depend on at run
          time, run-time dependencies must be listed in
          ‘propagated-inputs’ rather than ‘inputs’.
--8<---------------cut here---------------end--------------->8---

In short, the distinction between ‘native-inputs’ and ‘inputs’ exists
solely because Guix supports cross-compilation.  Otherwise it would be
unneeded.

Propagated inputs are a way to manually say: “I want this package to
automatically pull in those other packages.”  This is necessary in the
cases given above.

Does this clarify the rationale?

Thanks for your feedback,
Ludo’.

Re: Design decision behind inputs/native-inputs/propagated-inputs

[Steven Allen]

[-- Attachment #1: Type: text/plain, Size: 2129 bytes --]

On 01-21-16, Ludovic Courtès wrote:
> In short, the distinction between ‘native-inputs’ and ‘inputs’ exists
> solely because Guix supports cross-compilation.  Otherwise it would be
> unneeded.
> 
> Propagated inputs are a way to manually say: “I want this package to
> automatically pull in those other packages.”  This is necessary in the
> cases given above.
> 
> Does this clarify the rationale?

I believe I understand what they mean and how they solve the problem. My
question is more "why autodetect runtime dependencies"? That is,
dependencies are currently available at runtime as follows:

 Type              | Run Time
 ------------------+----------------------------------------------
 inputs            | autodetect (if referenced)
 propagated-inputs | always (included in the profile)
 native-inputs     | autodetect (if referenced)

However, I can't think of any cases where the following wouldn't suffice:

 Type              | Run Time                         | In Arch
 ------------------+----------------------------------+-----------
 inputs            | always                           | (pseudo-)depends
 propagated-inputs | always (included in the profile) | depends
 native-inputs     | never                            | makedepends

The key difference being the lack of autodetection in the second case.

In words, I can't think of any cases where one *would* want
native-inputs at runtime and/or *not* want (non-native) inputs at
runtime.

I ask because I hate automagic (magical autodetection). To illustrate,
given:

    (package
        (name "foo")
        (inputs `(("bar", bar)))
        (native-inputs `(("baz", baz)))
        ...
    )

I know that the default output of bar, will be available at build time
but whether or not it remains available at runtime depends on whether or
not my package includes the string `/gnu/store/$hash-bar/...` somewhere
in its output. Why not just say "bar will be available at runtime and
baz will not" (really "baz may not" because it might be an input to
another package).

-- 
Steven Allen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Design decision behind inputs/native-inputs/propagated-inputs

[Ben Woodcroft]

Hi Steven,

On 22/01/16 02:08, Steven Allen wrote:
> On 01-21-16, Ludovic Courtès wrote:
>> In short, the distinction between ‘native-inputs’ and ‘inputs’ exists
>> solely because Guix supports cross-compilation.  Otherwise it would be
>> unneeded.
>>
>> Propagated inputs are a way to manually say: “I want this package to
>> automatically pull in those other packages.”  This is necessary in the
>> cases given above.
>>
>> Does this clarify the rationale?
> I believe I understand what they mean and how they solve the problem. My
> question is more "why autodetect runtime dependencies"?
Can I ask, what do you mean by "autodetect" ?

Thanks,
ben

Re: Design decision behind inputs/native-inputs/propagated-inputs

[Steven Allen]

[-- Attachment #1: Type: text/plain, Size: 1345 bytes --]

Ben,

Replying back on list because I managed to take us off...

On 01-22-16, Ben Woodcroft wrote:
> On 22/01/16 08:13, Steven Allen wrote:
> >Most distros distinguish between build dependencies and runtime
> >dependencies. Guix doesn't appear to have this distinction and I am
> >under the impression that it autodetects which inputs are needed at
> >runtime (as apposed to at compile time only) by scanning the output
> >files for references to files in the inputs. Is this not the case?
> In short, no.

> You could say it scans the derivation which includes a list of "runtime"
> dependencies i.e. inputs and propagated-inputs. Derivations are in a
> well-defined format that guix generates, and are just a way of keeping track
> of dependencies that are needed at runtime. Guix doesn't scan the binaries
> or shared libraries for instance. Indeed that would be mayhem as you
> suggest.

Thanks! That makes so much more sense. Just to be clear, given:

    (package
        (name "foo")
        (inputs `(("bar", bar)))
        (native-inputs `(("baz", baz)))
        ...
    )

If I install "foo", "foo" will **always** be able to use the files in "bar"
but will not (may not?) be able to use the files in "baz"?

Sorry about the confusion...

-- 
Steven Allen
((Do Not Email <honeypot@stebalien.com>))

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Design decision behind inputs/native-inputs/propagated-inputs

[Ben Woodcroft]

Hi,

On 22/01/16 09:19, Steven Allen wrote:
> Just to be clear, given:
>
>      (package
>          (name "foo")
>          (inputs `(("bar", bar)))
>          (native-inputs `(("baz", baz)))
>          ...
>      )
>
> If I install "foo", "foo" will **always** be able to use the files in "bar"
> but will not (may not?) be able to use the files in "baz"?
That's true for some definition of 'use', yes. If you add foo to your 
profile and it is downloaded from a substitute (as opposed to being 
built from source on your machine), then you'll have to download bar as 
well, but not necessarily baz.
> Sorry about the confusion...
Not at all.

ben

Re: Design decision behind inputs/native-inputs/propagated-inputs

[Steven Allen]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

On 01-22-16, Ben Woodcroft wrote:
> On 22/01/16 09:19, Steven Allen wrote:
> >Just to be clear, given:
> >
> >     (package
> >         (name "foo")
> >         (inputs `(("bar", bar)))
> >         (native-inputs `(("baz", baz)))
> >         ...
> >     )
> >
> >If I install "foo", "foo" will **always** be able to use the files in "bar"
> >but will not (may not?) be able to use the files in "baz"?
> That's true for some definition of 'use', yes. If you add foo to your
> profile and it is downloaded from a substitute (as opposed to being built
> from source on your machine), then you'll have to download bar as well, but
> not necessarily baz.

Got it. Thanks.

-- 
Steven Allen
((Do Not Email <honeypot@stebalien.com>))

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Design decision behind inputs/native-inputs/propagated-inputs

[Steven Allen]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

Sorry to bug you again but I was going through some reports on the bug
tracker and came across https://debbugs.gnu.org/cgi/bugreport.cgi?bug=20765:

> Python .egg files must not be compressed
> ========================================
> ...
> Because it is compressed, the daemon’s conservative scanning fails to
> see what store items it refers to; in particular Ricardo noted that
> on his machine, python-pillow refers to a non-existent store item for
> OpenJPEG.
> ...
    
This kind of behavior is what initially gave me the impression that
runtime dependencies were auto detected.

Can anyone tell me what's going on here or point me to the relevant
documentation? Specifically, why does the daemon (assuming guix-daemon)
need to see what store items the package refers to?

-- 
Steven Allen
((Do Not Email <honeypot@stebalien.com>))

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Design decision behind inputs/native-inputs/propagated-inputs

[Ludovic Courtès]

Steven Allen <steven@stebalien.com> skribis:

> Can anyone tell me what's going on here or point me to the relevant
> documentation? Specifically, why does the daemon (assuming guix-daemon)
> need to see what store items the package refers to?

At the end of a build, guix-daemon scans all the produced file in search
of references to other store items.  It then records those run-time
references (so a subset of the build-time dependencies) in the
/var/guix/db/db.sqlite database, which can be queried with
‘guix gc --references’.

Scanning is also made from ‘registerOutputs’ in build.cc.

For background, see Eelco Dolstra’s thesis at
<http://nixos.org/~eelco/pubs/phd-thesis.pdf>.

HTH!

Ludo’.

Re: Design decision behind inputs/native-inputs/propagated-inputs

[Steven Allen]

[-- Attachment #1: Type: text/plain, Size: 2371 bytes --]

Ludovic,

On 01-23-16, Ludovic Courtès wrote:
> At the end of a build, guix-daemon scans all the produced file in search
> of references to other store items.  It then records those run-time
> references (so a subset of the build-time dependencies) in the
> /var/guix/db/db.sqlite database, which can be queried with
> ‘guix gc --references’.

So Guix does auto detect which inputs are needed at runtime! Back to my
original question: why? I've read the relevant parts of the thesis (2.1,
3.3, 7.1.5) and none of them actually answer this question. The thesis
answers "why is this not a horrible idea?" with "it appears to work in
practice" (2.1, 7.1.5) which:

1. Isn't very satisfying.
2. Is wrong. It works in practice until some unexpected case comes up
   (e.g. python eggs) and the package manager itself has to be changed.
3. Doesn't actually motivate this feature.

The only motivation I can think of is that it makes packaging (1) easier
(2) less error prone. You don't have to care about what inputs are
needed at runtime, you just have to write down all inputs and the
package manager will detect which ones are needed at runtime for you.
However, this Guix can't fully take advantage of this because it needs
to support cross compilation. Therefore, all *platform-dependent* inputs
needed at build time need to be specially designated.

My proposal is to:

1. Keep everything in the inputs list at runtime (write this list down
   somewhere).
2. Move all build-only inputs into native-inputs.
3. Turn this auto detection into a lint that validates the inputs list
   (verify that the inputs list isn't *missing* an input).
4. Optionally rename native-inputs to build-inputs.

Motivation:

1. One can tell which inputs will be kept at runtime and which will not
   by looking at the package definition.
2. One can package any program without ever having to modify the build
   system or play games (e.g. decompress random files).
3. It's less error prone because it's explicit. Unlike the current
   system, it won't break if some file happens to be compressed or
   encoded in some unexpected format.

And my question is: are there any use cases that this doesn't support?

Sorry for the wall of text and thanks for the thesis/explanation!

-- 
Steven Allen
((Do Not Email <honeypot@stebalien.com>))

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]