* Finding Dependencies at Run Time
@ 2022-07-13 14:56 Peter Polidoro
2022-07-13 16:03 ` Julien Lepiller
0 siblings, 1 reply; 12+ messages in thread
From: Peter Polidoro @ 2022-07-13 14:56 UTC (permalink / raw)
To: help-guix
I apologize for a very basic question, but I could not find where this is documented. If a paper or manual page describes this I would be happy to read it.
How, in general, does code in a Guix package find its dependency packages at run time?
Does it potentially work differently for each build system or is the same approach used for all packages?
Are absolute store paths of the dependencies saved at build time in environment variables or symbolic links or something that are accessible to the run time code?
If code in a package looks for dependencies using relative paths, are copies or links to the dependencies saved within the package? Or is that sort of duplication always avoided? Are relative paths converted to absolute store paths somehow?
Is the Guix daemon needed at run time to help packages find each other or is the information needed to resolve all of the dependencies saved into the package at build time?
Thanks!
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Finding Dependencies at Run Time
2022-07-13 14:56 Finding Dependencies at Run Time Peter Polidoro
@ 2022-07-13 16:03 ` Julien Lepiller
2022-07-13 17:47 ` Peter Polidoro
0 siblings, 1 reply; 12+ messages in thread
From: Julien Lepiller @ 2022-07-13 16:03 UTC (permalink / raw)
To: help-guix, Peter Polidoro
The information is all in the package. You can see for instance store paths that are embedded form a store path with:
guix gc --references /gnu/store/…
The mecanism depends a bit on the build system but for C programs, it's embeded at build-time in its RPATH. For applications, there are wrappers (https://guix.gnu.org/manual/devel/en/html_node/Build-Utilities.html#Wrappers). Otherwise, we have to propagate run-time dependencies.
The daemon is no longer involved once packages are built/substituted.
Le 13 juillet 2022 16:56:27 GMT+02:00, Peter Polidoro <peter@polidoro.io> a écrit :
>I apologize for a very basic question, but I could not find where this is documented. If a paper or manual page describes this I would be happy to read it.
>
>How, in general, does code in a Guix package find its dependency packages at run time?
>
>Does it potentially work differently for each build system or is the same approach used for all packages?
>
>Are absolute store paths of the dependencies saved at build time in environment variables or symbolic links or something that are accessible to the run time code?
>
>If code in a package looks for dependencies using relative paths, are copies or links to the dependencies saved within the package? Or is that sort of duplication always avoided? Are relative paths converted to absolute store paths somehow?
>
>Is the Guix daemon needed at run time to help packages find each other or is the information needed to resolve all of the dependencies saved into the package at build time?
>
>Thanks!
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Finding Dependencies at Run Time
2022-07-13 16:03 ` Julien Lepiller
@ 2022-07-13 17:47 ` Peter Polidoro
2022-07-13 18:39 ` Julien Lepiller
0 siblings, 1 reply; 12+ messages in thread
From: Peter Polidoro @ 2022-07-13 17:47 UTC (permalink / raw)
To: Julien Lepiller; +Cc: help-guix
> The mecanism depends a bit on the build system but for C
> programs, it's embeded at build-time in its RPATH. For
> applications, there are
> wrappers
> (https://guix.gnu.org/manual/devel/en/html_node/Build-Utilities.html#Wrappers).
So in general, packages use environment variables to find the
absolute paths of their dependencies in the store?
In a package definition, are those dependency paths always set
using wrap-program? Are search-paths, native-search-paths, and
setenv also used to set those environment variables or are those
used for something else?
> Otherwise, we have to propagate run-time dependencies.
So how else can packages propagate run-time dependencies besides
environment variables?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Finding Dependencies at Run Time
2022-07-13 17:47 ` Peter Polidoro
@ 2022-07-13 18:39 ` Julien Lepiller
2022-07-13 18:51 ` Peter Polidoro
0 siblings, 1 reply; 12+ messages in thread
From: Julien Lepiller @ 2022-07-13 18:39 UTC (permalink / raw)
To: Peter Polidoro; +Cc: help-guix
Maybe the general case is C programs rather than wrappers.
During the build, search-paths and native-search-paths are used to set up environment variables. If you use --keep-failed and interrupt a build you'l find them in /tmp/guix-build-…/environment-variables.
For C programs, LIBRARY_PATH is embedded as RPATH in the resulting binary or library.
For others, the required search paths can be embedded in a wrapper, which defines environment variables before calling the actual program.
For propagation, dependencies are found in the environment. It's less "pure" than the other ways, so we try to avoid resorting to that. Unfortunately some programming languages don't really leave us a choice (like python…).
Le 13 juillet 2022 19:47:45 GMT+02:00, Peter Polidoro <peter@polidoro.io> a écrit :
>
>> The mecanism depends a bit on the build system but for C programs, it's embeded at build-time in its RPATH. For applications, there are
>> wrappers (https://guix.gnu.org/manual/devel/en/html_node/Build-Utilities.html#Wrappers).
>
>So in general, packages use environment variables to find the absolute paths of their dependencies in the store?
>
>In a package definition, are those dependency paths always set using wrap-program? Are search-paths, native-search-paths, and setenv also used to set those environment variables or are those used for something else?
>
>> Otherwise, we have to propagate run-time dependencies.
>
>So how else can packages propagate run-time dependencies besides environment variables?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Finding Dependencies at Run Time
2022-07-13 18:39 ` Julien Lepiller
@ 2022-07-13 18:51 ` Peter Polidoro
2022-07-13 21:02 ` Julien Lepiller
2022-07-14 8:25 ` Ricardo Wurmus
0 siblings, 2 replies; 12+ messages in thread
From: Peter Polidoro @ 2022-07-13 18:51 UTC (permalink / raw)
To: Julien Lepiller; +Cc: help-guix
Your explanations are very helpful, thank you, and your links made
me realize that devel version of the manual has lots of
information that I could not find in the stable version of the
manual.
> During the build, search-paths and native-search-paths are used
> to set up environment variables. If you use --keep-failed and
> interrupt a
> build you'l find them in
> /tmp/guix-build-…/environment-variables.
So search-paths and native-search-paths are set before the build
and unset after the build so they are unavailable during run-time?
Or are native-search-paths only available at build-time and
search-paths available at both build-time and run-time?
Are the search-paths and native-search-paths absolute path values
found by automatically searching the directories in all of the
inputs or native inputs, looking for files or directories that
match a pattern? So they are a way to map relative paths into
absolute paths to the dependency packages?
> For others, the required search paths can be embedded in a
> wrapper, which defines environment variables before calling the
> actual program.
So if a package needs run-time environment variables, then a
package should use wrap-program to attach them to a command? These
are not found automatically by searching the inputs, they must be
manually defined using explicit input paths?
> For propagation, dependencies are found in the environment. It's
> less "pure" than the other ways, so we try to avoid resorting to
> that.
> Unfortunately some programming languages don't really leave us a
> choice (like python…).
So python packages are not using an environment variable, such as
PYTHONPATH, to find dependencies? How are they placed together
into an environment so they can find each other?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Finding Dependencies at Run Time
2022-07-13 18:51 ` Peter Polidoro
@ 2022-07-13 21:02 ` Julien Lepiller
2022-07-14 17:47 ` Peter Polidoro
2022-07-14 8:25 ` Ricardo Wurmus
1 sibling, 1 reply; 12+ messages in thread
From: Julien Lepiller @ 2022-07-13 21:02 UTC (permalink / raw)
To: Peter Polidoro; +Cc: help-guix
My explanation must not have been clear. You can read more on search-paths at https://guix.gnu.org/manual/devel/en/html_node/Search-Paths.html#Search-Paths
Basically, it's a declaration on packages that specify which path-like environment variables they honor.
When building an environment/profile, guix will look at all the variable declarations of packages of that environment and create the corresponding variables.
So when building a package, guix creates an environment with all the dependencies (inputs) of the package, and the corresponding environment variables.
Similarly, when you install say python (that brings in its search-path definition) and python-numpy (that provides a python library), guix will know to set GUIX_PYTHONPATH properly for using numpy with python. Since numpy propagates its dependencies, they are also part of the environment and that's how numpy can find them.
Remember the difference between inputs and propagated inputs: they're the same, but when you create a profile, inputs are not part of the profile (so they need a direct store reference, such as RPATH or a wrapper), whereas propagated inputs are part of the profile, so an environment variable allows to find them.
Le 13 juillet 2022 20:51:18 GMT+02:00, Peter Polidoro <peter@polidoro.io> a écrit :
>Your explanations are very helpful, thank you, and your links made me realize that devel version of the manual has lots of information that I could not find in the stable version of the manual.
>
>> During the build, search-paths and native-search-paths are used to set up environment variables. If you use --keep-failed and interrupt a
>> build you'l find them in /tmp/guix-build-…/environment-variables.
>
>So search-paths and native-search-paths are set before the build and unset after the build so they are unavailable during run-time? Or are native-search-paths only available at build-time and search-paths available at both build-time and run-time?
>
>Are the search-paths and native-search-paths absolute path values found by automatically searching the directories in all of the inputs or native inputs, looking for files or directories that match a pattern? So they are a way to map relative paths into absolute paths to the dependency packages?
>
>> For others, the required search paths can be embedded in a wrapper, which defines environment variables before calling the actual program.
>
>So if a package needs run-time environment variables, then a package should use wrap-program to attach them to a command? These are not found automatically by searching the inputs, they must be manually defined using explicit input paths?
>
>> For propagation, dependencies are found in the environment. It's less "pure" than the other ways, so we try to avoid resorting to that.
>> Unfortunately some programming languages don't really leave us a choice (like python…).
>
>So python packages are not using an environment variable, such as PYTHONPATH, to find dependencies? How are they placed together into an environment so they can find each other?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Finding Dependencies at Run Time
2022-07-13 18:51 ` Peter Polidoro
2022-07-13 21:02 ` Julien Lepiller
@ 2022-07-14 8:25 ` Ricardo Wurmus
2022-07-14 8:48 ` Philip McGrath
1 sibling, 1 reply; 12+ messages in thread
From: Ricardo Wurmus @ 2022-07-14 8:25 UTC (permalink / raw)
To: Peter Polidoro; +Cc: Julien Lepiller, help-guix
Peter Polidoro <peter@polidoro.io> writes:
> Your explanations are very helpful, thank you, and your links made me
> realize that devel version of the manual has lots of information that
> I could not find in the stable version of the manual.
We recommend using the manual that comes with Guix, because it matches
the particular version of Guix you have installed. You can use any Info
reader to view the manual and the cookbook.
--
Ricardo
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Finding Dependencies at Run Time
2022-07-14 8:25 ` Ricardo Wurmus
@ 2022-07-14 8:48 ` Philip McGrath
2022-07-14 11:28 ` Ricardo Wurmus
0 siblings, 1 reply; 12+ messages in thread
From: Philip McGrath @ 2022-07-14 8:48 UTC (permalink / raw)
To: Ricardo Wurmus, Peter Polidoro; +Cc: Julien Lepiller, help-guix
On Thu, Jul 14, 2022, at 4:25 AM, Ricardo Wurmus wrote:
> Peter Polidoro <peter@polidoro.io> writes:
>
>> Your explanations are very helpful, thank you, and your links made me
>> realize that devel version of the manual has lots of information that
>> I could not find in the stable version of the manual.
>
> We recommend using the manual that comes with Guix, because it matches
> the particular version of Guix you have installed. You can use any Info
> reader to view the manual and the cookbook.
>
I mentioned this somewhere before, but I would love to be able to install a local copy of the HTML manual with the nice styling used on the website.
-Philip
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Finding Dependencies at Run Time
2022-07-14 8:48 ` Philip McGrath
@ 2022-07-14 11:28 ` Ricardo Wurmus
2022-07-14 13:27 ` zimoun
0 siblings, 1 reply; 12+ messages in thread
From: Ricardo Wurmus @ 2022-07-14 11:28 UTC (permalink / raw)
To: Philip McGrath; +Cc: Peter Polidoro, Julien Lepiller, help-guix
"Philip McGrath" <philip@philipmcgrath.com> writes:
> On Thu, Jul 14, 2022, at 4:25 AM, Ricardo Wurmus wrote:
>> Peter Polidoro <peter@polidoro.io> writes:
>>
>>> Your explanations are very helpful, thank you, and your links made me
>>> realize that devel version of the manual has lots of information that
>>> I could not find in the stable version of the manual.
>>
>> We recommend using the manual that comes with Guix, because it matches
>> the particular version of Guix you have installed. You can use any Info
>> reader to view the manual and the cookbook.
>>
>
> I mentioned this somewhere before, but I would love to be able to install a local copy of the HTML manual with the nice styling used on the website.
You can build all HTML and PDF versions of the manual with
./pre-inst-env guix build -f doc/build.scm
from a Guix checkout. This will take a little while.
The convenience of using the Info document is that it is installed with
“guix pull”, so it will always match your current version of Guix. The
Info document is also more accessible than the HTML version because of
the manually curated index (hit “i” in your Info reader and type away to
get to a topic more quickly).
--
Ricardo
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Finding Dependencies at Run Time
2022-07-14 11:28 ` Ricardo Wurmus
@ 2022-07-14 13:27 ` zimoun
0 siblings, 0 replies; 12+ messages in thread
From: zimoun @ 2022-07-14 13:27 UTC (permalink / raw)
To: Ricardo Wurmus, Philip McGrath; +Cc: Peter Polidoro, Julien Lepiller, help-guix
Hi,
(I am an avid Emacs user but to be honest I barely use Info manuals. Even
after many attempts to use them, I find the Info interface
counter-intuitive. Initially, I thought it was because I do not have
the habits and I forced myself to exclusively use it. The conclusions
are:
1. it is easier for me to open
https://guix.gnu.org/manual/devel/en/guix.html
even sometimes eww (which is a bit slow, BTW) and just directly
search. It is also really easy to reach the index.
2. aside the case where I exactly know beforehand what I am looking
for, I do not use the index. Instead, I use:
a) directly query a keyword, say C-f with Chromium or else
b) query with a search engine
end of the parenthesis. :-))
On Thu, 14 Jul 2022 at 13:28, Ricardo Wurmus <rekado@elephly.net> wrote:
> The convenience of using the Info document is that it is installed with
> “guix pull”, so it will always match your current version of Guix. The
> Info document is also more accessible than the HTML version because of
> the manually curated index (hit “i” in your Info reader and type away to
> get to a topic more quickly).
Why not also distribute the HTML manual along with the Info one? For
instance, on Debian,
$ find /usr/share/doc/ -type f -name "*.html" | wc -l
681
and I do not remember installing some *-doc packages.
For instance, why not just have
~/.config/guix/current/share/doc/guix/manual.html
at first, then populate this /doc/guix folder with all the fancy things
used for the online rendering. WDYT?
Cheers,
simon
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Finding Dependencies at Run Time
2022-07-13 21:02 ` Julien Lepiller
@ 2022-07-14 17:47 ` Peter Polidoro
2022-07-16 21:53 ` Thiago Jung Bauermann
0 siblings, 1 reply; 12+ messages in thread
From: Peter Polidoro @ 2022-07-14 17:47 UTC (permalink / raw)
To: Julien Lepiller; +Cc: help-guix
> Remember the difference between inputs and propagated inputs:
> they're the same, but when you create a profile, inputs are not
> part of the
> profile (so they need a direct store reference, such as RPATH or
> a wrapper), whereas propagated inputs are part of the profile,
> so an
> environment variable allows to find them.
Thank you for patiently explaining, I think I am starting to
understand now, but please correct me if I am still wrong.
So there is package build-time, profile build-time, and run-time.
Wrappers should be used to set environment variables when the
value can be determined at package build-time, such as absolute
paths to inputs.
Search paths should be used to set environment variables when the
value will be determined at profile build-time, such as relative
paths to either propagated-inputs or packages that also installed
in the same profile, but not listed explicitly as
propagated-inputs to the package declaring the search paths.
I am still a little unclear about the difference between
search-paths and native-search-paths, though. If not
cross-compiling then they act the same? If you are
cross-compiling, though, then it only finds packages that are
listed as native-inputs to the package declaring the search paths?
Native-inputs are not normally installed in a profile, though,
correct? Does that mean native-search-paths will only be found
when using a development shell when cross-compiling?
I guess I do not properly understand cross-compiling. So when
cross-compiling, there is a package build-time, where you are
building packages for a target architecture on a host machine,
profile build-time, where you are building a profile on the
target, and run-time, when you are running on the target? If
search-paths and native-search-paths are determined at profile
build-time, how do they behave differently on the target machine?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Finding Dependencies at Run Time
2022-07-14 17:47 ` Peter Polidoro
@ 2022-07-16 21:53 ` Thiago Jung Bauermann
0 siblings, 0 replies; 12+ messages in thread
From: Thiago Jung Bauermann @ 2022-07-16 21:53 UTC (permalink / raw)
To: Peter Polidoro; +Cc: Julien Lepiller, help-guix
Hello Peter,
Peter Polidoro <peter@polidoro.io> writes:
>> Remember the difference between inputs and propagated inputs: they're
>> the same, but when you create a profile, inputs are not part of the
>> profile (so they need a direct store reference, such as RPATH or a
>> wrapper), whereas propagated inputs are part of the profile, so an
>> environment variable allows to find them.
>
> Thank you for patiently explaining, I think I am starting to
> understand now, but please correct me if I am still wrong.
I think there are still some misunderstandings, or perhaps at least to
me it's not clear what your question is. I think if you could provide a
concrete example of what you are trying to understand it would be
helpful.
> So there is package build-time, profile build-time, and run-time.
>
> Wrappers should be used to set environment variables when the value
> can be determined at package build-time, such as absolute paths to
> inputs.
No. Ideally wrappers shouldn't be necessary and wouldn't be used, but
for some software they are a necessary/useful workaround. At least
that's my understanding.
In the ideal situation, a Guix-package program uses absolute paths to
find the resources (libraries, scripts, other files) that it needs. For
example: how does a compiled program finds the library it was linked
with? It has the full path to glibc hard-coded into the ELF's library
runpath:
popigai: readelf --dynamic ~/.guix-profile/bin/htop | grep runpath
0x000000000000001d (RUNPATH) Library runpath: [/gnu/store/9rrnm5hdjw7cy96a2a9rfgh6y08wsbmf-ncurses-6.2.20210619/lib:/gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib:/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib:/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../..]
Or if it's a shell script, it has the full path to its interpreter:
popigai: head -n1 ~/.guix-profile/bin/vcsh
#!/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/sh
Sometimes the original program doesn't use absolute paths, but we can
patch it so that it does. This is the case for example in public-inbox,
which invokes “xapian-compact” and expects it to be found in PATH, and
we change it to use the full path (this is in
guix/gnu/packages/mail.scm, line 4111 in the revision of Guix I'm
looking at):
--8<---------------cut here---------------start------------->8---
;; Use absolute paths for 'xapian-compact'.
(substitute* "lib/PublicInbox/Xapcmd.pm"
(("'xapian-compact'")
(format #f "'~a'" (search-input-file inputs
"/bin/xapian-compact"))))
--8<---------------cut here---------------end--------------->8---
But sometimes such changes aren't possible so in that case we do use
wrappers to configure environment variables that will cause the program
to find the resource. This also happens in public-inbox, which around
line 4160 has a ‘wrap-programs’ phase that creates a wrapper so that it
can find Perl modules and also the git and curl executables.
One thing that is implicit from the above is that at runtime, Guix
doesn't do anything to help a program find its dependencies. It has
already done its job at build time by hard-coding paths so that the
program can find them on its own or setting up a wrapper, and at profile
build-time, by setting up any relevant environment variables.
> Search paths should be used to set environment variables when the
> value will be determined at profile build-time, such as relative paths
> to either propagated-inputs or packages that also installed in the
> same profile, but not listed explicitly as propagated-inputs to the
> package declaring the search paths.
I'd say that search paths should be used when the set of resources that
a program will need can't be determined at build time. For example, the
Python interpreter can load a huge number of libraries. It doesn't make
sense to hard-code the absolute paths of all of them in the Python
interpreter. Rather, it should look for the ones that are installed in
the profile.
Another use case is when a program's dependency is optional and it can
work well enough without the dependency being available. That way, the
user can install the dependencies they want and the program will use
them.
> I am still a little unclear about the difference between search-paths
> and native-search-paths, though. If not cross-compiling then they act
> the same?
Yes, that's correct.
> If you are cross-compiling, though, then it only finds packages that
> are listed as native-inputs to the package declaring the search paths?
I don't know about cross-compilation profiles, but hopefully it's
similar to the build environment set up by Guix when cross-compiling a
package. Because of that, I'll answer some of your questions in this
section of the email but not all of them.
When Guix cross-compiles a package, native inputs are the ones that will
be used during the process of building the package (and not at runtime),
so they should be built for the architecture of the build machine. For
example, the compiler itself, the make tool, etc. The regular inputs
will be used by the package at runtime and thus will be built for the
target architecture.
> Native-inputs are not normally installed in a profile, though,
> correct?
In the case of a non-cross build, the native inputs will also be
installed but will be of the same architecture as the normal inputs.
In this case, it doesn't make any difference whether a dependency is
listed in the native-inputs or inputs field of a package.
I hope I haven't added to the confusion. :-)
--
Thanks
Thiago
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2022-07-16 23:37 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-13 14:56 Finding Dependencies at Run Time Peter Polidoro
2022-07-13 16:03 ` Julien Lepiller
2022-07-13 17:47 ` Peter Polidoro
2022-07-13 18:39 ` Julien Lepiller
2022-07-13 18:51 ` Peter Polidoro
2022-07-13 21:02 ` Julien Lepiller
2022-07-14 17:47 ` Peter Polidoro
2022-07-16 21:53 ` Thiago Jung Bauermann
2022-07-14 8:25 ` Ricardo Wurmus
2022-07-14 8:48 ` Philip McGrath
2022-07-14 11:28 ` Ricardo Wurmus
2022-07-14 13:27 ` zimoun
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.