inputs vs. native-inputs vs. propagated-inputs

inputs vs. native-inputs vs. propagated-inputs

[Hartmut Goebel]

Hi,

I'm a bit confused about the difference between inputs, native-inputs
and propagated-inputs.

The manual states:

          The distinction between ‘native-inputs’ and ‘inputs’ is
          necessary when considering cross-compilation.  When
          cross-compiling, dependencies listed in ‘inputs’ are built for
          the _target_ architecture; conversely, dependencies listed in
          ‘native-inputs’ are built for the architecture of the _build_
          machine.

For for I understand. But then the manual says:

          ‘native-inputs’ is typically used to list tools needed at
          build time, but not at run time, such as Autoconf, Automake,
          pkg-config, Gettext, or Bison.

The first sentence implies that "inputs" are treated as needed at run time.

Now consider libAAA (supporting the famous Amiga AAA chipset ;-), which
requires the headers of libBBB to compile (but only to fetch some
constant definitions) and libCCC at run-time. Without libCCC, libAAA
could not work. And libAAA uses pkg-config to find the header files.

So for me this would be:
  libBBB: inputs
  libCCC: propagates inputs
  pkg-confg: native inputs


Is this correct?

If so, how can I as a packager find out if eg. libBBB is only used at
build time and libCCC need to be a propagated input?

Same for pkg-config: How to determine if this is only needed ar build
time (as I would always expect)? The manual says:

          … propagated-inputs …
          For example this is necessary when a C/C++ library needs
          headers of another library to compile, or when a pkg-config
          file refers to another one via its ‘Requires’ field.

  For me this is confusing.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: inputs vs. native-inputs vs. propagated-inputs

[宋文武]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Hi,
>
> I'm a bit confused about the difference between inputs, native-inputs
> and propagated-inputs.
>
> The manual states:
>
>           The distinction between ‘native-inputs’ and ‘inputs’ is
>           necessary when considering cross-compilation.  When
>           cross-compiling, dependencies listed in ‘inputs’ are built for
>           the _target_ architecture; conversely, dependencies listed in
>           ‘native-inputs’ are built for the architecture of the _build_
>           machine.
Yes, _native_ means the same architecture as the build machine.

>
> For for I understand. But then the manual says:
>
>           ‘native-inputs’ is typically used to list tools needed at
>           build time, but not at run time, such as Autoconf, Automake,
>           pkg-config, Gettext, or Bison.
>
> The first sentence implies that "inputs" are treated as needed at run
> time.
No, as _native_ inputs usually are tools for building (and testing),
most time they’re not needed at run time.

The dependencies of a store item (an output of the package) is
computed by scan all its files for store paths (as reported
by ‘guix gc –-references …’).  When you download a store item by
substitutes, its dependencise will be downloaded too.  When you
install a store item by ‘guix package -i /gnu/store/…’, only one
path will be added to the profile.


The runtime dependencies are only for _package_ objects.
When you add a _package_ object into profile (or inputs for
building other package), its ‘propagated-inputs’ will be added too.


>
> Now consider libAAA (supporting the famous Amiga AAA chipset ;-), which
> requires the headers of libBBB to compile (but only to fetch some
> constant definitions) and libCCC at run-time. Without libCCC, libAAA
> could not work. And libAAA uses pkg-config to find the header files.
>
> So for me this would be:
>   libBBB: inputs
>   libCCC: propagates inputs
>   pkg-confg: native inputs
>
>
> Is this correct?
No, libCCC not necessary be propagated, it’s a dependency of libAAA,
but this dependency maybe transparent if the use of libAAA doesn’t
requires libAAA.

- To build libAAA natively, be inputs is enougth.
- To build libAAA for other architecture, pkg-config needed to
  be ‘native-inputs’, due to it’s an ELF executable to be invoked by the
  build system.
- To make the users of libAAA happy, libBBB or libCCC should be propagated
  if the use of libAAA require them (eg: its headers include libBBB’s
  headers or its .pc file list libCCC in the ‘Requires’ field.).


>
> If so, how can I as a packager find out if eg. libBBB is only used at
> build time and libCCC need to be a propagated input?
By looking the output of ‘guix gc –-references …’, the build time ones
are ones not there.  If you think something shouldn’t be there (eg:
build a library, but gcc end up within its references), then it’s time
to patch it to reduce closure size.

For propagated inputs, it’s for the users.  So if (for programs) we run
it fine or (for libraries) include its headers and link it fine, nothing
need to be propagated.

>
> Same for pkg-config: How to determine if this is only needed ar build
> time (as I would always expect)? The manual says:
>
>           … propagated-inputs …
>           For example this is necessary when a C/C++ library needs
>           headers of another library to compile, or when a pkg-config
>           file refers to another one via its ‘Requires’ field.
>
>   For me this is confusing.
Many build systems use ‘pkg-config’ to check for libraries and get
flags, a pc file usually list some other packages in its ‘Requires’
field, if one of these packages is missing (doesn’t have a <package>.pc
file in PKG_CONFIG_PATH), this pc file will be treated as broken, and
the package will be reported as ‘not found’.  So propageted these
packages make ‘pkg-config’ works, reduce the work for packaging its
users (otherwise, those packages need to added as inputs even they’re
not used directly).

Re: inputs vs. native-inputs vs. propagated-inputs

[Hartmut Goebel]

Hi,

Thanks for your answer

Am 12.06.2016 um 14:38 schrieb 宋文武:
> Hartmut Goebel <h.goebel@crazy-compilers.com> writes:
>> For for I understand. But then the manual says:
>>
>>           ‘native-inputs’ is typically used to list tools needed at
>>           build time, but not at run time, such as Autoconf, Automake,
>>           pkg-config, Gettext, or Bison.
>>
>> The first sentence implies that "inputs" are treated as needed at run
>> time.
> No, as _native_ inputs usually are tools for building (and testing),
> most time they’re not needed at run time.

This paragraph is only talking about "native-inputs" and about "needed …
not at run time". While the paragraph just above this sentence is
talking about both "inputs" and native-inputs", this "needed … not at
run time" implies "inputs" are needed at run time.

I suggest rephrasing this into something like: "Both inputs and
native-inputs are used for stuff needed at build time, not at run time.
'inputs' are for ..., e.g. library headers, ..., while 'native-inputs'
are for tools such as Autoconf, Automake, pkg-config, Gettext, or Bison."


>
>> If so, how can I as a packager find out if eg. libBBB is only used at
>> build time and libCCC need to be a propagated input?
> By looking the output of ‘guix gc –-references …’, the build time ones
> are ones not there.  

I'm the packager, so I'm the one who needs to *define* the dependencies.
There is no ‘guix gc –-references …’ I can query. So *I* need a way to
determine whether an input needs to be propagated or not.

>
>> Same for pkg-config: How to determine if this is only needed ar build
>> time (as I would always expect)? The manual says:
>>
>>           … propagated-inputs …
>>           For example this is necessary when a C/C++ library needs
>>           headers of another library to compile, or when a pkg-config
>>           file refers to another one via its ‘Requires’ field.
>>
>>   For me this is confusing.
> Many build systems use ‘pkg-config’ to check for libraries and get
> flags, a pc file usually list some other packages in its ‘Requires’
> field, if one of these packages is missing (doesn’t have a <package>.pc
> file in PKG_CONFIG_PATH), this pc file will be treated as broken, and
> the package will be reported as ‘not found’.  So propageted these
> packages make ‘pkg-config’ works, reduce the work for packaging its
> users (otherwise, those packages need to added as inputs even they’re
> not used directly).

Thanks, I misunderstood the example. I though it was talking about
"pkg-config", while it is talking about .pc files.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: inputs vs. native-inputs vs. propagated-inputs

[Leo Famulari]

On Sun, Jun 12, 2016 at 05:50:29PM +0200, Hartmut Goebel wrote:
> Am 12.06.2016 um 14:38 schrieb 宋文武:
> > Hartmut Goebel <h.goebel@crazy-compilers.com> writes:
> >> For for I understand. But then the manual says:
> >>
> >>           ‘native-inputs’ is typically used to list tools needed at
> >>           build time, but not at run time, such as Autoconf, Automake,
> >>           pkg-config, Gettext, or Bison.
> >>
> >> The first sentence implies that "inputs" are treated as needed at run
> >> time.
> > No, as _native_ inputs usually are tools for building (and testing),
> > most time they’re not needed at run time.
> 
> This paragraph is only talking about "native-inputs" and about "needed …
> not at run time". While the paragraph just above this sentence is
> talking about both "inputs" and native-inputs", this "needed … not at
> run time" implies "inputs" are needed at run time.
> 
> I suggest rephrasing this into something like: "Both inputs and
> native-inputs are used for stuff needed at build time, not at run time.
> 'inputs' are for ..., e.g. library headers, ..., while 'native-inputs'
> are for tools such as Autoconf, Automake, pkg-config, Gettext, or Bison."

'Inputs' do typically get used at run-time, as do propagated-inputs.

I found it hard to understand the distinctions by reading. It was only
when I had been making packages for a while that I understood.

I've tried to improve this text but I haven't come up with anything yet.

> >> If so, how can I as a packager find out if eg. libBBB is only used at
> >> build time and libCCC need to be a propagated input?

You will need at least a little knowledge about the programs you are
packaging and how they are supposed to build and run. I read a bit about
each program to guess about how libAAA uses it.

> I'm the packager, so I'm the one who needs to *define* the dependencies.
> There is no ‘guix gc –-references …’ I can query. So *I* need a way to
> determine whether an input needs to be propagated or not.

Test the program in an isolated environment and see if it works without
propagating the inputs.

You can set up an environment with only libAAA ...

$ guix environment --container --ad-hoc libAAA

... and then somehow exercise libAAA to see if it can find its
dependencies.

Also, once you've built the package, using `guix gc --references` is a
good way to inspect it. 

The type of input chosen by the packager does not dictate how libAAA
uses the dependency. The package could erroneously retain a reference to
a native-input like Automake, and `guix gc --references` will show you
this.

So, if libCCC appears in `guix gc --references /gnu/store/...-libAAA`,
it's reasonable to guess that libCCC does not need to be propagated.

Or, the package could lack a reference to something you *know* is needed
at run-time. So you can address that with propagated-inputs or setting
some build-time configuration.

Is it making more sense?

Re: inputs vs. native-inputs vs. propagated-inputs

[Hartmut Goebel]

Hallo Leo,

thanks for you answer.

Am 12.06.2016 um 21:53 schrieb Leo Famulari:
> 'Inputs' do typically get used at run-time, as do propagated-inputs.
>
> I found it hard to understand the distinctions by reading. It was only
> when I had been making packages for a while that I understood.
>
> I've tried to improve this text but I haven't come up with anything yet.

I'd try a text, but I did not really understand the difference yet.

>>> If so, how can I as a packager find out if eg. libBBB is only used at
>>> build time and libCCC need to be a propagated input?
> You will need at least a little knowledge about the programs you are
> packaging and how they are supposed to build and run. I read a bit about
> each program to guess about how libAAA uses it.

IC. I was hoping. I could just package some stuff without any knowledge.
E.g  to make basic libraries and programs available so others can add
more programs. Obviously I was wrong here :-(

>
>> I'm the packager, so I'm the one who needs to *define* the dependencies.
>> There is no ‘guix gc –-references …’ I can query. So *I* need a way to
>> determine whether an input needs to be propagated or not.
> Test the program in an isolated environment and see if it works without
> propagating the inputs.

Thanks, this is a helpful tip.

> Also, once you've built the package, using `guix gc --references` is a
> good way to inspect it. 

This is what I do not get: If I do not specify some dependency, how will
it be listed with `gc --references` (except the case there is another
dependency). And if `gc --references` would be able to find dependencies
I missed, why at all should one list dependencies?

> The type of input chosen by the packager does not dictate how libAAA
> uses the dependency. The package could erroneously retain a reference to
> a native-input like Automake, and `guix gc --references` will show you
> this.
>
> So, if libCCC appears in `guix gc --references /gnu/store/...-libAAA`,
> it's reasonable to guess that libCCC does not need to be propagated.
>
> Or, the package could lack a reference to something you *know* is needed
> at run-time. So you can address that with propagated-inputs or setting
> some build-time configuration.
>
> Is it making more sense?

It opens a window in the dust of incomprehension :-) But I need to thing
(and test) on this whole topic a lot more.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: inputs vs. native-inputs vs. propagated-inputs

[Leo Famulari]

On Fri, Jun 17, 2016 at 10:49:59PM +0200, Hartmut Goebel wrote:
> >>> If so, how can I as a packager find out if eg. libBBB is only used at
> >>> build time and libCCC need to be a propagated input?
> > You will need at least a little knowledge about the programs you are
> > packaging and how they are supposed to build and run. I read a bit about
> > each program to guess about how libAAA uses it.
> 
> IC. I was hoping. I could just package some stuff without any knowledge.
> E.g  to make basic libraries and programs available so others can add
> more programs. Obviously I was wrong here :-(

To clarify: it does not require expert knowledge. If the upstream
maintainers have used a standard build system like Python's setuptools
or Automake, packaging for Guix is almost as easy as filling out a form.

> > Also, once you've built the package, using `guix gc --references` is a
> > good way to inspect it. 
> 
> This is what I do not get: If I do not specify some dependency, how will
> it be listed with `gc --references` (except the case there is another
> dependency). And if `gc --references` would be able to find dependencies
> I missed, why at all should one list dependencies?

`gc --references` is a diagnostic tool that looks at the result of
building the package, and reports what it finds. It's up to us to decide
if the result is what we expected.

I _think_ that `gc --references` works by querying the database for a
list of references in the store item you give it as an argument. I think
that the information in the database is created after building, by
scanning the files of the package's output in /gnu/store, looking for
strings that appear to be paths in /gnu/store. I say "I think" because I
am not sure. I don't understand that part of the code very well yet.

Here's an example of how I use it:
http://lists.gnu.org/archive/html/guix-devel/2016-06/msg00367.html

That package, msgpack, requires zlib to build and run.  After adding
zlib to msgpack's inputs, msgpack's build system is able to find zlib
and build msgpack successfully.

But, I used `guix gc --references` to list the references in
'/gnu/store/...-msgpack' to other store items, and there are no
references to zlib. That is, there are no strings in the msgpack output
that are paths like '/gnu/store/...-zlib'.

For some reason, the build process is not recording the location of zlib
into the built msgpack binaries. So, msgpack will probably not work
until we fix that.

I hope this helps.

Re: inputs vs. native-inputs vs. propagated-inputs

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> I _think_ that `gc --references` works by querying the database for a
> list of references in the store item you give it as an argument. I think
> that the information in the database is created after building, by
> scanning the files of the package's output in /gnu/store, looking for
> strings that appear to be paths in /gnu/store. I say "I think" because I
> am not sure. I don't understand that part of the code very well yet.

That’s 100% correct!  :-)  Namely, ‘guix gc --references’ makes an RPC to
the daemon, which then looks things up in the database (see the ‘Refs’
table in nix/libstore/schema.sql.)

Scanning for references indeed happens at the end of a successful build,
in ‘scanForReferences’ in libstore/references.cc.  Since scanning is
expensive (I/O-intensive), the result is stored in the database.

Ludo’.

Re: inputs vs. native-inputs vs. propagated-inputs

[Lukas Gradl]

ludo@gnu.org (Ludovic Courtès) writes:

> Leo Famulari <leo@famulari.name> skribis:
>
>> I _think_ that `gc --references` works by querying the database for a
>> list of references in the store item you give it as an argument. I think
>> that the information in the database is created after building, by
>> scanning the files of the package's output in /gnu/store, looking for
>> strings that appear to be paths in /gnu/store. I say "I think" because I
>> am not sure. I don't understand that part of the code very well yet.
>
> That’s 100% correct!  :-)  Namely, ‘guix gc --references’ makes an RPC to
> the daemon, which then looks things up in the database (see the ‘Refs’
> table in nix/libstore/schema.sql.)
>
> Scanning for references indeed happens at the end of a successful build,
> in ‘scanForReferences’ in libstore/references.cc.  Since scanning is
> expensive (I/O-intensive), the result is stored in the database.

Sorry, this might be a dumb question, I don't quite understand the
concept of these references.  Why are references important?  Shouldn't
every store item know which other store-items are related to it from the
"inputs"-field in its definition?  Why is it necessary to keep track of
the references?

Thank you!

Re: inputs vs. native-inputs vs. propagated-inputs

[Ludovic Courtès]

Lukas Gradl <lgradl@openmailbox.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Leo Famulari <leo@famulari.name> skribis:
>>
>>> I _think_ that `gc --references` works by querying the database for a
>>> list of references in the store item you give it as an argument. I think
>>> that the information in the database is created after building, by
>>> scanning the files of the package's output in /gnu/store, looking for
>>> strings that appear to be paths in /gnu/store. I say "I think" because I
>>> am not sure. I don't understand that part of the code very well yet.
>>
>> That’s 100% correct!  :-)  Namely, ‘guix gc --references’ makes an RPC to
>> the daemon, which then looks things up in the database (see the ‘Refs’
>> table in nix/libstore/schema.sql.)
>>
>> Scanning for references indeed happens at the end of a successful build,
>> in ‘scanForReferences’ in libstore/references.cc.  Since scanning is
>> expensive (I/O-intensive), the result is stored in the database.
>
> Sorry, this might be a dumb question, I don't quite understand the
> concept of these references.  Why are references important?  Shouldn't
> every store item know which other store-items are related to it from the
> "inputs"-field in its definition?  Why is it necessary to keep track of
> the references?

The “references” of a store item are its run-time dependencies, a subset
of the ‘inputs’ etc. fields, which are themselves the compile-time
dependencies.

If those run-time dependencies were not inferred automatically by the
daemon, we’d have to maintain them individually, and this would be
error-prone and imprecise.

Run-time dependency information is what allows Guix to know which
substitutes need to be downloaded when installing from substitutes, and
it’s what allows the garbage collector to determine which store items
are “live”, and which ones are not.

HTH!

Ludo’.

Re: inputs vs. native-inputs vs. propagated-inputs

[Lukas Gradl]

ludo@gnu.org (Ludovic Courtès) writes:

> Lukas Gradl <lgradl@openmailbox.org> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> Leo Famulari <leo@famulari.name> skribis:
>>>
>>>> I _think_ that `gc --references` works by querying the database for a
>>>> list of references in the store item you give it as an argument. I think
>>>> that the information in the database is created after building, by
>>>> scanning the files of the package's output in /gnu/store, looking for
>>>> strings that appear to be paths in /gnu/store. I say "I think" because I
>>>> am not sure. I don't understand that part of the code very well yet.
>>>
>>> That’s 100% correct!  :-)  Namely, ‘guix gc --references’ makes an RPC to
>>> the daemon, which then looks things up in the database (see the ‘Refs’
>>> table in nix/libstore/schema.sql.)
>>>
>>> Scanning for references indeed happens at the end of a successful build,
>>> in ‘scanForReferences’ in libstore/references.cc.  Since scanning is
>>> expensive (I/O-intensive), the result is stored in the database.
>>
>> Sorry, this might be a dumb question, I don't quite understand the
>> concept of these references.  Why are references important?  Shouldn't
>> every store item know which other store-items are related to it from the
>> "inputs"-field in its definition?  Why is it necessary to keep track of
>> the references?
>
> The “references” of a store item are its run-time dependencies, a subset
> of the ‘inputs’ etc. fields, which are themselves the compile-time
> dependencies.
>
> If those run-time dependencies were not inferred automatically by the
> daemon, we’d have to maintain them individually, and this would be
> error-prone and imprecise.
>
> Run-time dependency information is what allows Guix to know which
> substitutes need to be downloaded when installing from substitutes, and
> it’s what allows the garbage collector to determine which store items
> are “live”, and which ones are not.
>

OK, Thank you for the explanation, this makes more sense to me now!

Best,
Lukas

Re: inputs vs. native-inputs vs. propagated-inputs

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2789 bytes --]

Lukas Gradl <lgradl@openmailbox.org> writes:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Lukas Gradl <lgradl@openmailbox.org> skribis:
>>
>>> ludo@gnu.org (Ludovic Courtès) writes:
>>>
>>>> Leo Famulari <leo@famulari.name> skribis:
>>>>
>>>>> I _think_ that `gc --references` works by querying the database for a
>>>>> list of references in the store item you give it as an argument. I think
>>>>> that the information in the database is created after building, by
>>>>> scanning the files of the package's output in /gnu/store, looking for
>>>>> strings that appear to be paths in /gnu/store. I say "I think" because I
>>>>> am not sure. I don't understand that part of the code very well yet.
>>>>
>>>> That’s 100% correct!  :-)  Namely, ‘guix gc --references’ makes an RPC to
>>>> the daemon, which then looks things up in the database (see the ‘Refs’
>>>> table in nix/libstore/schema.sql.)
>>>>
>>>> Scanning for references indeed happens at the end of a successful build,
>>>> in ‘scanForReferences’ in libstore/references.cc.  Since scanning is
>>>> expensive (I/O-intensive), the result is stored in the database.
>>>
>>> Sorry, this might be a dumb question, I don't quite understand the
>>> concept of these references.  Why are references important?  Shouldn't
>>> every store item know which other store-items are related to it from the
>>> "inputs"-field in its definition?  Why is it necessary to keep track of
>>> the references?
>>
>> The “references” of a store item are its run-time dependencies, a subset
>> of the ‘inputs’ etc. fields, which are themselves the compile-time
>> dependencies.
>>
>> If those run-time dependencies were not inferred automatically by the
>> daemon, we’d have to maintain them individually, and this would be
>> error-prone and imprecise.
>>
>> Run-time dependency information is what allows Guix to know which
>> substitutes need to be downloaded when installing from substitutes, and
>> it’s what allows the garbage collector to determine which store items
>> are “live”, and which ones are not.
>>
>
> OK, Thank you for the explanation, this makes more sense to me now!
>
> Best,
> Lukas

If you are curious, there is a more detailed explanation of "references"
in Eelco Dolstra's PhD thesis, "The Purely Functional Software
Deployment Model", available here:

https://nixos.org/%7Eeelco/pubs/phd-thesis.pdf

You'll find more information about Nix here, much of which applies to
Guix also:

http://nixos.org/docs/papers.html

You'll find more links about Guix here:

https://www.gnu.org/software/guix/contribute/

And this repo contains talks etc. about Guix:

http://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]