Finding Dependencies at Run Time

Finding Dependencies at Run Time

[Peter Polidoro]

I apologize for a very basic question, but I could not find where this is documented. If a paper or manual page describes this I would be happy to read it.

How, in general, does code in a Guix package find its dependency packages at run time?

Does it potentially work differently for each build system or is the same approach used for all packages?

Are absolute store paths of the dependencies saved at build time in environment variables or symbolic links or something that are accessible to the run time code?

If code in a package looks for dependencies using relative paths, are copies or links to the dependencies saved within the package? Or is that sort of duplication always avoided? Are relative paths converted to absolute store paths somehow?

Is the Guix daemon needed at run time to help packages find each other or is the information needed to resolve all of the dependencies saved into the package at build time?

Thanks!

Re: Finding Dependencies at Run Time

[Julien Lepiller]

The information is all in the package. You can see for instance store paths that are embedded form a store path with:

guix gc --references /gnu/store/…

The mecanism depends a bit on the build system but for C programs, it's embeded at build-time in its RPATH. For applications, there are wrappers (https://guix.gnu.org/manual/devel/en/html_node/Build-Utilities.html#Wrappers). Otherwise, we have to propagate run-time dependencies.

The daemon is no longer involved once packages are built/substituted.

Le 13 juillet 2022 16:56:27 GMT+02:00, Peter Polidoro <peter@polidoro.io> a écrit :
>I apologize for a very basic question, but I could not find where this is documented. If a paper or manual page describes this I would be happy to read it.
>
>How, in general, does code in a Guix package find its dependency packages at run time?
>
>Does it potentially work differently for each build system or is the same approach used for all packages?
>
>Are absolute store paths of the dependencies saved at build time in environment variables or symbolic links or something that are accessible to the run time code?
>
>If code in a package looks for dependencies using relative paths, are copies or links to the dependencies saved within the package? Or is that sort of duplication always avoided? Are relative paths converted to absolute store paths somehow?
>
>Is the Guix daemon needed at run time to help packages find each other or is the information needed to resolve all of the dependencies saved into the package at build time?
>
>Thanks!

Re: Finding Dependencies at Run Time

[Peter Polidoro]

> The mecanism depends a bit on the build system but for C 
> programs, it's embeded at build-time in its RPATH. For 
> applications, there are
> wrappers 
> (https://guix.gnu.org/manual/devel/en/html_node/Build-Utilities.html#Wrappers).

So in general, packages use environment variables to find the 
absolute paths of their dependencies in the store?

In a package definition, are those dependency paths always set 
using wrap-program? Are search-paths, native-search-paths, and 
setenv also used to set those environment variables or are those 
used for something else?

> Otherwise, we have to propagate run-time dependencies.

So how else can packages propagate run-time dependencies besides 
environment variables?

Re: Finding Dependencies at Run Time

[Julien Lepiller]

Maybe the general case is C programs rather than wrappers.

During the build, search-paths and native-search-paths are used to set up environment variables. If you use --keep-failed and interrupt a build you'l find them in /tmp/guix-build-…/environment-variables.

For C programs, LIBRARY_PATH is embedded as RPATH in the resulting binary or library.

For others, the required search paths can be embedded in a wrapper, which defines environment variables before calling the actual program.

For propagation, dependencies are found in the environment. It's less "pure" than the other ways, so we try to avoid resorting to that. Unfortunately some programming languages don't really leave us a choice (like python…).

Le 13 juillet 2022 19:47:45 GMT+02:00, Peter Polidoro <peter@polidoro.io> a écrit :
>
>> The mecanism depends a bit on the build system but for C programs, it's embeded at build-time in its RPATH. For applications, there are
>> wrappers (https://guix.gnu.org/manual/devel/en/html_node/Build-Utilities.html#Wrappers).
>
>So in general, packages use environment variables to find the absolute paths of their dependencies in the store?
>
>In a package definition, are those dependency paths always set using wrap-program? Are search-paths, native-search-paths, and setenv also used to set those environment variables or are those used for something else?
>
>> Otherwise, we have to propagate run-time dependencies.
>
>So how else can packages propagate run-time dependencies besides environment variables?

Re: Finding Dependencies at Run Time

[Peter Polidoro]

Your explanations are very helpful, thank you, and your links made 
me realize that devel version of the manual has lots of 
information that I could not find in the stable version of the 
manual.

> During the build, search-paths and native-search-paths are used 
> to set up environment variables. If you use --keep-failed and 
> interrupt a
> build you'l find them in 
> /tmp/guix-build-…/environment-variables.

So search-paths and native-search-paths are set before the build 
and unset after the build so they are unavailable during run-time? 
Or are native-search-paths only available at build-time and 
search-paths available at both build-time and run-time?

Are the search-paths and native-search-paths absolute path values 
found by automatically searching the directories in all of the 
inputs or native inputs, looking for files or directories that 
match a pattern? So they are a way to map relative paths into 
absolute paths to the dependency packages?

> For others, the required search paths can be embedded in a 
> wrapper, which defines environment variables before calling the 
> actual program.

So if a package needs run-time environment variables, then a 
package should use wrap-program to attach them to a command? These 
are not found automatically by searching the inputs, they must be 
manually defined using explicit input paths?

> For propagation, dependencies are found in the environment. It's 
> less "pure" than the other ways, so we try to avoid resorting to 
> that.
> Unfortunately some programming languages don't really leave us a 
> choice (like python…).

So python packages are not using an environment variable, such as 
PYTHONPATH, to find dependencies? How are they placed together 
into an environment so they can find each other?

Re: Finding Dependencies at Run Time

[Julien Lepiller]

My explanation must not have been clear. You can read more on search-paths at https://guix.gnu.org/manual/devel/en/html_node/Search-Paths.html#Search-Paths

Basically, it's a declaration on packages that specify which path-like environment variables they honor.

When building an environment/profile, guix will look at all the variable declarations of packages of that environment and create the corresponding variables.

So when building a package, guix creates an environment with all the dependencies (inputs) of the package, and the corresponding environment variables.

Similarly, when you install say python (that brings in its search-path definition) and python-numpy (that provides a python library), guix will know to set GUIX_PYTHONPATH properly for using numpy with python. Since numpy propagates its dependencies, they are also part of the environment and that's how numpy can find them.

Remember the difference between inputs and propagated inputs: they're the same, but when you create a profile, inputs are not part of the profile (so they need a direct store reference, such as RPATH or a wrapper), whereas propagated inputs are part of the profile, so an environment variable allows to find them.

Le 13 juillet 2022 20:51:18 GMT+02:00, Peter Polidoro <peter@polidoro.io> a écrit :
>Your explanations are very helpful, thank you, and your links made me realize that devel version of the manual has lots of information that I could not find in the stable version of the manual.
>
>> During the build, search-paths and native-search-paths are used to set up environment variables. If you use --keep-failed and interrupt a
>> build you'l find them in /tmp/guix-build-…/environment-variables.
>
>So search-paths and native-search-paths are set before the build and unset after the build so they are unavailable during run-time? Or are native-search-paths only available at build-time and search-paths available at both build-time and run-time?
>
>Are the search-paths and native-search-paths absolute path values found by automatically searching the directories in all of the inputs or native inputs, looking for files or directories that match a pattern? So they are a way to map relative paths into absolute paths to the dependency packages?
>
>> For others, the required search paths can be embedded in a wrapper, which defines environment variables before calling the actual program.
>
>So if a package needs run-time environment variables, then a package should use wrap-program to attach them to a command? These are not found automatically by searching the inputs, they must be manually defined using explicit input paths?
>
>> For propagation, dependencies are found in the environment. It's less "pure" than the other ways, so we try to avoid resorting to that.
>> Unfortunately some programming languages don't really leave us a choice (like python…).
>
>So python packages are not using an environment variable, such as PYTHONPATH, to find dependencies? How are they placed together into an environment so they can find each other?

Re: Finding Dependencies at Run Time

[Peter Polidoro]

> Remember the difference between inputs and propagated inputs: 
> they're the same, but when you create a profile, inputs are not 
> part of the
> profile (so they need a direct store reference, such as RPATH or 
> a wrapper), whereas propagated inputs are part of the profile, 
> so an
> environment variable allows to find them.

Thank you for patiently explaining, I think I am starting to 
understand now, but please correct me if I am still wrong.

So there is package build-time, profile build-time, and run-time.

Wrappers should be used to set environment variables when the 
value can be determined at package build-time, such as absolute 
paths to inputs.

Search paths should be used to set environment variables when the 
value will be determined at profile build-time, such as relative 
paths to either propagated-inputs or packages that also installed 
in the same profile, but not listed explicitly as 
propagated-inputs to the package declaring the search paths.

I am still a little unclear about the difference between 
search-paths and native-search-paths, though. If not 
cross-compiling then they act the same? If you are 
cross-compiling, though, then it only finds packages that are 
listed as native-inputs to the package declaring the search paths? 
Native-inputs are not normally installed in a profile, though, 
correct? Does that mean native-search-paths will only be found 
when using a development shell when cross-compiling?

I guess I do not properly understand cross-compiling. So when 
cross-compiling, there is a package build-time, where you are 
building packages for a target architecture on a host machine, 
profile build-time, where you are building a profile on the 
target, and run-time, when you are running on the target? If 
search-paths and native-search-paths are determined at profile 
build-time, how do they behave differently on the target machine?

Re: Finding Dependencies at Run Time

[Thiago Jung Bauermann]

Hello Peter,

Peter Polidoro <peter@polidoro.io> writes:

>> Remember the difference between inputs and propagated inputs: they're
>> the same, but when you create a profile, inputs are not part of the
>> profile (so they need a direct store reference, such as RPATH or a
>> wrapper), whereas propagated inputs are part of the profile, so an
>> environment variable allows to find them.
>
> Thank you for patiently explaining, I think I am starting to
> understand now, but please correct me if I am still wrong.

I think there are still some misunderstandings, or perhaps at least to
me it's not clear what your question is. I think if you could provide a
concrete example of what you are trying to understand it would be
helpful.

> So there is package build-time, profile build-time, and run-time.
>
> Wrappers should be used to set environment variables when the value
> can be determined at package build-time, such as absolute paths to
> inputs.

No. Ideally wrappers shouldn't be necessary and wouldn't be used, but
for some software they are a necessary/useful workaround. At least
that's my understanding.

In the ideal situation, a Guix-package program uses absolute paths to
find the resources (libraries, scripts, other files) that it needs. For
example: how does a compiled program finds the library it was linked
with? It has the full path to glibc hard-coded into the ELF's library
runpath:

popigai: readelf --dynamic ~/.guix-profile/bin/htop | grep runpath
 0x000000000000001d (RUNPATH)            Library runpath: [/gnu/store/9rrnm5hdjw7cy96a2a9rfgh6y08wsbmf-ncurses-6.2.20210619/lib:/gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib:/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib:/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../..]

Or if it's a shell script, it has the full path to its interpreter:

popigai: head -n1 ~/.guix-profile/bin/vcsh
#!/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/sh

Sometimes the original program doesn't use absolute paths, but we can
patch it so that it does. This is the case for example in public-inbox,
which invokes “xapian-compact” and expects it to be found in PATH, and
we change it to use the full path (this is in
guix/gnu/packages/mail.scm, line 4111 in the revision of Guix I'm
looking at):

--8<---------------cut here---------------start------------->8---
  ;; Use absolute paths for 'xapian-compact'.
  (substitute* "lib/PublicInbox/Xapcmd.pm"
    (("'xapian-compact'")
    (format #f "'~a'" (search-input-file inputs
                                          "/bin/xapian-compact"))))
--8<---------------cut here---------------end--------------->8---

But sometimes such changes aren't possible so in that case we do use
wrappers to configure environment variables that will cause the program
to find the resource. This also happens in public-inbox, which around
line 4160 has a ‘wrap-programs’ phase that creates a wrapper so that it
can find Perl modules and also the git and curl executables.

One thing that is implicit from the above is that at runtime, Guix
doesn't do anything to help a program find its dependencies. It has
already done its job at build time by hard-coding paths so that the
program can find them on its own or setting up a wrapper, and at profile
build-time, by setting up any relevant environment variables.

> Search paths should be used to set environment variables when the
> value will be determined at profile build-time, such as relative paths
> to either propagated-inputs or packages that also installed in the
> same profile, but not listed explicitly as propagated-inputs to the
> package declaring the search paths.

I'd say that search paths should be used when the set of resources that
a program will need can't be determined at build time. For example, the
Python interpreter can load a huge number of libraries. It doesn't make
sense to hard-code the absolute paths of all of them in the Python
interpreter. Rather, it should look for the ones that are installed in
the profile.

Another use case is when a program's dependency is optional and it can
work well enough without the dependency being available. That way, the
user can install the dependencies they want and the program will use
them.

> I am still a little unclear about the difference between search-paths
> and native-search-paths, though. If not cross-compiling then they act
> the same?

Yes, that's correct.

> If you are cross-compiling, though, then it only finds packages that
> are listed as native-inputs to the package declaring the search paths?

I don't know about cross-compilation profiles, but hopefully it's
similar to the build environment set up by Guix when cross-compiling a
package. Because of that, I'll answer some of your questions in this
section of the email but not all of them.

When Guix cross-compiles a package, native inputs are the ones that will
be used during the process of building the package (and not at runtime),
so they should be built for the architecture of the build machine. For
example, the compiler itself, the make tool, etc. The regular inputs
will be used by the package at runtime and thus will be built for the
target architecture.

> Native-inputs are not normally installed in a profile, though,
> correct?

In the case of a non-cross build, the native inputs will also be
installed but will be of the same architecture as the normal inputs.
In this case, it doesn't make any difference whether a dependency is
listed in the native-inputs or inputs field of a package.

I hope I haven't added to the confusion. :-)

-- 
Thanks
Thiago

Re: Finding Dependencies at Run Time

[Ricardo Wurmus]

Peter Polidoro <peter@polidoro.io> writes:

> Your explanations are very helpful, thank you, and your links made me
> realize that devel version of the manual has lots of information that
> I could not find in the stable version of the manual.

We recommend using the manual that comes with Guix, because it matches
the particular version of Guix you have installed.  You can use any Info
reader to view the manual and the cookbook.

-- 
Ricardo

Re: Finding Dependencies at Run Time

[Philip McGrath]

On Thu, Jul 14, 2022, at 4:25 AM, Ricardo Wurmus wrote:
> Peter Polidoro <peter@polidoro.io> writes:
>
>> Your explanations are very helpful, thank you, and your links made me
>> realize that devel version of the manual has lots of information that
>> I could not find in the stable version of the manual.
>
> We recommend using the manual that comes with Guix, because it matches
> the particular version of Guix you have installed.  You can use any Info
> reader to view the manual and the cookbook.
>

I mentioned this somewhere before, but I would love to be able to install a local copy of the HTML manual with the nice styling used on the website.

-Philip

Re: Finding Dependencies at Run Time

[Ricardo Wurmus]

"Philip McGrath" <philip@philipmcgrath.com> writes:

> On Thu, Jul 14, 2022, at 4:25 AM, Ricardo Wurmus wrote:
>> Peter Polidoro <peter@polidoro.io> writes:
>>
>>> Your explanations are very helpful, thank you, and your links made me
>>> realize that devel version of the manual has lots of information that
>>> I could not find in the stable version of the manual.
>>
>> We recommend using the manual that comes with Guix, because it matches
>> the particular version of Guix you have installed.  You can use any Info
>> reader to view the manual and the cookbook.
>>
>
> I mentioned this somewhere before, but I would love to be able to install a local copy of the HTML manual with the nice styling used on the website.

You can build all HTML and PDF versions of the manual with

   ./pre-inst-env guix build -f doc/build.scm

from a Guix checkout.  This will take a little while.

The convenience of using the Info document is that it is installed with
“guix pull”, so it will always match your current version of Guix.  The
Info document is also more accessible than the HTML version because of
the manually curated index (hit “i” in your Info reader and type away to
get to a topic more quickly).

-- 
Ricardo

Re: Finding Dependencies at Run Time

[zimoun]

Hi,

(I am an avid Emacs user but to be honest I barely use Info manuals.  Even
after many attempts to use them, I find the Info interface
counter-intuitive.  Initially, I thought it was because I do not have
the habits and I forced myself to exclusively use it.  The conclusions
are:

 1. it is easier for me to open
        https://guix.gnu.org/manual/devel/en/guix.html
 even sometimes eww (which is a bit slow, BTW) and just directly
 search.  It is also really easy to reach the index.

 2. aside the case where I exactly know beforehand what I am looking
 for, I do not use the index.  Instead, I use:

  a) directly query a keyword, say C-f with Chromium or else
  b) query with a search engine

end of the parenthesis. :-))


On Thu, 14 Jul 2022 at 13:28, Ricardo Wurmus <rekado@elephly.net> wrote:

> The convenience of using the Info document is that it is installed with
> “guix pull”, so it will always match your current version of Guix.  The
> Info document is also more accessible than the HTML version because of
> the manually curated index (hit “i” in your Info reader and type away to
> get to a topic more quickly).

Why not also distribute the HTML manual along with the Info one?  For
instance, on Debian,

        $ find /usr/share/doc/ -type f -name "*.html" | wc -l
        681

and I do not remember installing some *-doc packages.

For instance, why not just have

    ~/.config/guix/current/share/doc/guix/manual.html

at first, then populate this /doc/guix folder with all the fancy things
used for the online rendering.  WDYT?


Cheers,
simon