Specifying dependencies among package outputs?

Specifying dependencies among package outputs?

[Simon South]

Am I right in thinking there is no way to specify dependencies among the
outputs of a single package? To specify that a package's "out" output
depends on its "lib" output, for instance.

I ask because the Knot package (in gnu/package/dns.scm) builds a number
of logically distinct targets---daemon, libraries, administrative
utilities, general-purpose utilities, and documentation---and it would
be nice to separate at least some of these into individual outputs, in
part because we could then specify only the libraries as a dependency of
Knot Resolver.

However, Knot's daemon and utilities have the same dependency on its own
libraries, so pulling those into a separate "lib" output would be liable
to break everything else.

I've searched and can't find an example of this being done, nor can I
find any mention of it in the documentation. So I assume it's simply not
possible, and you would need to define an entirely separate package that
builds from the same source code---right?

-- 
Simon South
simon@simonsouth.net

Re: Specifying dependencies among package outputs?

[Julien Lepiller]

Le 14 octobre 2020 18:32:27 GMT-04:00, Simon South <simon@simonsouth.net> a écrit :
>Am I right in thinking there is no way to specify dependencies among
>the
>outputs of a single package? To specify that a package's "out" output
>depends on its "lib" output, for instance.

The notion of dependencies do not make a lot of sense in guix. We talk about inputs: things that are accessible during the build, but don't really declare which are needed at runtime and which are needed at build-time (granted usually the distinction between native and non native does that job).

For tracking runtime dependencies, guix uses another mechanism: when a package holds a reference to another package (by embedding the store path of that other package), it depends on it, anl guix will pull them both.

>
>I ask because the Knot package (in gnu/package/dns.scm) builds a number
>of logically distinct targets---daemon, libraries, administrative
>utilities, general-purpose utilities, and documentation---and it would
>be nice to separate at least some of these into individual outputs, in
>part because we could then specify only the libraries as a dependency
>of
>Knot Resolver.
>
>However, Knot's daemon and utilities have the same dependency on its
>own
>libraries, so pulling those into a separate "lib" output would be
>liable
>to break everything else.
>
>I've searched and can't find an example of this being done, nor can I
>find any mention of it in the documentation. So I assume it's simply
>not
>possible, and you would need to define an entirely separate package
>that
>builds from the same source code---right?

We actually do that for a few programs. Look at mariadb for an example of splitting lependencies to different outputs. Mariadb has binaries that require the libraries. During the build, the library path is recorded in the binaries (using the rpath mechanism). Since the path to the lib output is present in the binary, it dependg on it and that output is pulled by guix.

Knot could work the same. Specify the libdir and make sure the libdir path is embedded in the binaries. Then, guix will know that a lependency exists and will pull the libraries.

Any package that depends on the library rather than the binary will need to be adjusted to use knot:lib as an input. Other than that, no breakage.

Re: Specifying dependencies among package outputs?

[Tobias Geerinckx-Rice]

[-- Attachment #1.1: Type: text/plain, Size: 3816 bytes --]

Simon,

Simon South 写道：
> Am I right in thinking there is no way to specify dependencies 
> among the
> outputs of a single package?

Well, yes, but probably not in the way you mean: they aren't 
specified at all.  Oh dear, nckx's responding to a question about 
‘dependencies’.  Apologies to those who know what's coming.

I avoid the word dependency when discussing Guix and urge you to 
do the same.  It's too ambiguous to be useful.  Some distributions 
use terms like ‘run-time dependency’ or ‘build dependency’ as if 
they're similar, but to me Guix/Nix prove that they're unrelated 
by treating them very differently.  We have inputs, which are 
specified by humans (or implicitly by the build system), and 
references -- which are not[0].

Inputs are built or downloaded when building a package, but only 
references -- the actual occurence of the string 
/gnu/store/<hash>-<package>-<version> in another 
/gnu/store/<subdirectory> -- determine which of those inputs are 
actually relevant after the build has finished.  Only referenced 
store items will be kept around during the next GC or downloaded 
when installing pre-built substitutes.

> To specify that a package's "out" output depends on its "lib" 
> output, for instance.

So it's the build process itself that specifies this: when it's 
complete, /gnu/store/<...>-knot-3.0.1 (:out) may or may not retain 
a reference to /gnu/store/<...>-knot-3.0.1-lib (:lib), and that's 
what determines whether or not :out depends on :lib.

I suspect the majority of packages with a :lib output do so.

If you apply the patch below you'll see (e.g., with ‘guix size’) 
that installing only knot:tools will pull in knot{:out,:lib} 
without any human-made hints to that effect.

> I ask because the Knot package (in gnu/package/dns.scm) builds a 
> number
> of logically distinct targets---daemon, libraries, 
> administrative
> utilities, general-purpose utilities, and documentation---and it 
> would
> be nice to separate at least some of these into individual 
> outputs

Separating tools into administrative vs. general-purpose goes too 
far.

Attached patch:

  $ guix size /gnu/store/...-knot-3.0.1-doc
  total: 0.2 MiB (no references)
  $ guix size /gnu/store/...-knot-3.0.1-lib
  total: 145.0 MiB (self: 2.4 MiB)
  $ guix size /gnu/store/...-knot-3.0.1
  total: 171.1 MiB (self: 5.2 MiB; refers to :lib)
  $ guix size /gnu/store/...-knot-3.0.1-tools
  total: 164.9 MiB (self: 0.4 MiB; refers to :lib)

Old monolithic knot:

  $ guix size /gnu/store/...-knot-3.0.1
  total: 171.5 MiB (self: 8.0 MiB)

> [only the libraries end up a dependency of] Knot Resolver.

Correct.  Building knot-resolver with only ("knot:lib" ,knot 
"lib"):

  $ guix size /gnu/store/...-knot-resolver-5.1.3
  total: 169.1 MiB

Old monolithic knot:

  $ guix size /gnu/store/...-knot-resolver-5.1.3
  total: 183.0 MiB

A saving of 13.9 MiB or <10% is not considered impressive.  Some 
might say it's not worth the complexity although I'm OK with it.

> However, Knot's daemon and utilities have the same dependency on 
> its own
> libraries, so pulling those into a separate "lib" output would 
> be liable
> to break everything else.

Why?

> I've searched and can't find an example of this being done, nor 
> can I
> find any mention of it in the documentation. So I assume it's 
> simply not
> possible, and you would need to define an entirely separate 
> package that
> builds from the same source code---right?

Which other examples or documentation have you read?  How would 
you consider Knot more complex or problematic?

Kind regards,

T G-R

[0]: I'm ignoring propagated-inputs, both to simplify things and 
because it makes me happy.


[-- Attachment #1.2: 0001-gnu-knot-Build-separate-outputs.patch --]
[-- Type: text/x-patch, Size: 3718 bytes --]

From 38ae89365b9bff6676d771c74589af391e53283b Mon Sep 17 00:00:00 2001
From: Tobias Geerinckx-Rice <me@tobias.gr>
Date: Thu, 15 Oct 2020 02:36:02 +0200
Subject: [PATCH] gnu: knot: Build separate outputs.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* gnu/packages/dns.scm (knot)[outputs]: New field adding :doc, :lib,
and :tools outputs.
[arguments]: Add #:configure-flags to install into :doc and :lib.
Add a new ‘split-:tools’ phase to install into :tools.
Add a new ‘break-circular-:lib->:out-reference’ phase to do just that.
---
 gnu/packages/dns.scm | 38 +++++++++++++++++++++++++++++++++-----
 1 file changed, 33 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/dns.scm b/gnu/packages/dns.scm
index 1775660162..c566e7260e 100644
--- a/gnu/packages/dns.scm
+++ b/gnu/packages/dns.scm
@@ -827,13 +827,21 @@ Extensions} (DNSSEC).")
            (delete-file-recursively "src/contrib/libbpf")
            #t))))
     (build-system gnu-build-system)
+    (outputs (list "out" "doc" "lib" "tools"))
     (arguments
      `(#:configure-flags
-       (list "--sysconfdir=/etc"
+       (list (string-append "--docdir=" (assoc-ref %outputs "doc")
+                            "/share/" ,name "-" ,version)
+             (string-append "--mandir=" (assoc-ref %outputs "doc")
+                            "/share/man")
+             (string-append "--infodir=" (assoc-ref %outputs "doc")
+                            "/share/info")
+             (string-append "--libdir=" (assoc-ref %outputs "lib") "/lib")
+             "--sysconfdir=/etc"
              "--localstatedir=/var"
-             "--enable-dnstap"          ; let tools read/write capture files
-             "--enable-fastparser"      ; disabled by default when .git/ exists
-             "--enable-xdp=auto"        ; XXX [=yes] currently means =embedded
+             "--enable-dnstap"         ; let tools read/write capture files
+             "--enable-fastparser"     ; disabled by default when .git/ exists
+             "--enable-xdp=auto"       ; XXX [=yes] currently means =embedded
              "--with-module-dnstap=yes") ; detailed query capturing & logging
        #:phases
        (modify-phases %standard-phases
@@ -868,7 +876,27 @@ Extensions} (DNSSEC).")
                        "install"))))
          (add-after 'install 'install-info
            (lambda _
-             (invoke "make" "install-info"))))))
+             (invoke "make" "install-info")))
+         (add-after 'install 'break-circular-:lib->:out-reference
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (assoc-ref outputs "lib")))
+               (for-each (lambda (file)
+                           (substitute* file
+                             (("(prefix=).*" _ assign)
+                              (string-append assign lib "\n"))))
+                         (find-files lib "\\.pc$"))
+               #t)))
+         (add-after 'install 'split-:tools
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out   (assoc-ref outputs "out"))
+                    (tools (assoc-ref outputs "tools")))
+               (for-each (lambda (command)
+                           (mkdir-p (string-append tools (dirname command)))
+                           (rename-file (string-append out command)
+                                        (string-append tools command)))
+                         (list "/bin/kdig"
+                               "/bin/khost"))
+               #t))))))
     (native-inputs
      `(("autoconf" ,autoconf)
        ("automake" ,automake)
-- 
2.28.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Specifying dependencies among package outputs?

[Tobias Geerinckx-Rice]

[-- Attachment #1.1: Type: text/plain, Size: 232 bytes --]

Tobias Geerinckx-Rice 写道：
> Correct.  Building knot-resolver with only ("knot:lib" ,knot 
> "lib"):

Might's well send that trivial part too.  (And yes, the comment 
shuffling in the previous one was of course bogus.)


[-- Attachment #1.2: 0001-gnu-knot-resolver-Build-with-only-knot-lib.patch --]
[-- Type: text/x-patch, Size: 971 bytes --]

From 0b3ce4cb3f2618714fd91123a38eb679ed26dc84 Mon Sep 17 00:00:00 2001
From: Tobias Geerinckx-Rice <me@tobias.gr>
Date: Thu, 15 Oct 2020 02:40:38 +0200
Subject: [PATCH] gnu: knot-resolver: Build with only knot:lib.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This saves 13.9 MiB (~7.5%) of total ‘guix size’.

* gnu/packages/dns.scm (knot-resolver)[inputs]: Replace knot with
knot:lib.
---
 gnu/packages/dns.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/dns.scm b/gnu/packages/dns.scm
index 6e39234367..139b19a4f5 100644
--- a/gnu/packages/dns.scm
+++ b/gnu/packages/dns.scm
@@ -989,7 +989,7 @@ synthesis, and on-the-fly re-configuration.")
     (inputs
      `(("fstrm" ,fstrm)
        ("gnutls" ,gnutls)
-       ("knot" ,knot)
+       ("knot:lib" ,knot "lib")
        ("libuv" ,libuv)
        ("lmdb" ,lmdb)
        ("luajit" ,luajit)
-- 
2.28.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Specifying dependencies among package outputs?

[zimoun]

Hi Tobias,

On Thu, 15 Oct 2020 at 02:38, Tobias Geerinckx-Rice <me@tobias.gr> wrote:

> Well, yes, but probably not in the way you mean: they aren't
> specified at all.  Oh dear, nckx's responding to a question about
> ‘dependencies’.  Apologies to those who know what's coming.

[...]

> If you apply the patch below you'll see (e.g., with ‘guix size’)
> that installing only knot:tools will pull in knot{:out,:lib}
> without any human-made hints to that effect.

Wow!  Thank you for the detailed explanation.  I have also been
puzzled by this and my mind was still a bit foggy on the topic.  Now
all clear! :-)


> Attached patch:
>
>   $ guix size /gnu/store/...-knot-3.0.1-doc
>   total: 0.2 MiB (no references)
>   $ guix size /gnu/store/...-knot-3.0.1-lib
>   total: 145.0 MiB (self: 2.4 MiB)
>   $ guix size /gnu/store/...-knot-3.0.1
>   total: 171.1 MiB (self: 5.2 MiB; refers to :lib)
>   $ guix size /gnu/store/...-knot-3.0.1-tools
>   total: 164.9 MiB (self: 0.4 MiB; refers to :lib)
>
> Old monolithic knot:
>
>   $ guix size /gnu/store/...-knot-3.0.1
>   total: 171.5 MiB (self: 8.0 MiB)

These numbers are self explanatory for me.  Maybe this example could
go to the Cookbook?



All the best,
simon

Re: Specifying dependencies among package outputs?

[Simon South]

Thank you Brett, Julien and Tobias for your responses. They've been
super helpful and have cleared things up for me quite a bit.

Julien Lepiller <julien@lepiller.eu> writes:
> For tracking runtime dependencies, guix uses another mechanism: when a
> package holds a reference to another package (by embedding the store
> path of that other package), it depends on it, anl guix will pull them
> both.

This was the key piece of information I was missing: Guix packages don't
really deal with "dependencies" per se. Instead a package has inputs and
outputs, plus a build process that transforms one set to the other; and
the references among these items that result naturally from the
transformation process are what Guix uses to determine which packages
should be installed alongside which.

So a packager's focus ought to be on specifying the package's inputs,
outputs and transformation correctly, after which Guix can generally be
relied on to do the right thing automatically with regard to
"dependencies".

This makes sense, and even seems painfully obvious now that I've written
it out (hello, functional programming). But it wasn't obvious before.

This also clears up for me a bit of remaining confusion around the
distinction between "inputs" and "propagated inputs": I wondered why, if
a package's inputs are its dependencies, the "propagated-inputs" field
is needed at all since surely a package's inputs would always be
installed alongside it. The explanation is that a package's inputs are
_not_ its dependencies; they are merely inputs to its build process, and
whether one becomes a "dependency" depends entirely on whether a
reference to it remains in any of the package's outputs. The
"propagated-input" field is used to ensure this association is made,
even when there is no apparent reference (that Guix can find) in the
outputs.

Tobias Geerinckx-Rice <me@tobias.gr> writes:
> If you apply the patch below you'll see (e.g., with ‘guix size’) that
> installing only knot:tools will pull in knot{:out,:lib} without any
> human-made hints to that effect.

Thank you for this! This is amazing, and exactly the sort of thing I had
in mind. (Though I wonder if the "tools" output would better be called
"utils", to match the isc-bind package?)

Are you planning on committing these changes? I think they're great.

> A saving of 13.9 MiB or <10% is not considered impressive.

It's not that bad either, and anyway it seems logical to me for the
package to be split up this way considering each of the outputs serves a
distinct purpose.

>> However, Knot's daemon and utilities have the same dependency on its
>> own libraries, so pulling those into a separate "lib" output would be
>> liable to break everything else.
>
> Why?

Assuming you didn't mean this rhetorically: My assumption was that
without an explicitly stated dependency between the "out" and "lib"
packages, Guix wouldn't know to install the two together. So a user
running "guix install knot" would get only a binary that aborts at
startup with a complaint about missing libraries.

Not being aware of "references" in this context is what had me confused.

> Which other examples or documentation have you read?

I actually did a fair bit of searching in multiple places but because I
was looking for "dependencies" and not "references", none of what I
found seemed very helpful. The manual could certainly do more to
highlight this distinction; in its "package" reference for instance it
says quite plainly regarding the "input" fields, "These fields list
dependencies of the package."[0]

That said, looking at the manual again this morning I quickly found
this[1]:

       The outputs of derivations—i.e., the build results—have a set of
    “references”, as reported by the ‘references’ RPC or the ‘guix gc
    --references’ command (*note Invoking guix gc::).  References are
    the set of run-time dependencies of the build results.  References
    are a subset of the inputs of the derivation; this subset is
    automatically computed by the build daemon by scanning all the files
    in the outputs.

That more-or-less lays it out, but I wouldn't have found it since

- It's in the page on "Derivations", which I'm accustomed to thinking of
  as a low-level detail that can safely be ignored;

- A quick scan of the paragraph gives the impression it has more to do
  with the "guix gc" command than with writing package definitions; and

- The paragraph makes no mention of "dependencies" (nor how they and
  "references" relate to one another), so I wouldn't have landed on it
  with a plain-text search.

To be fair, that paragraph _is_ linked to in the concept index under
"dependencies, run-time", so I perhaps should have found it a bit sooner
than I did.

Even so I think it might be good to add a brief section to the manual
that addresses directly how "dependencies" are managed by Guix, since I
doubt I'll be the last person who arrives with a traditional
package-management mindset and is confused about how Guix handles
things.

> How would you consider Knot more complex or problematic?

Knot seemed to be unique in that its distribution contains the source
code for both its executables and the libraries on which they depend,
all of which are meant to be built together in a single run.

That didn't strike me as unusual at first but searching through
gnu/packages, I found plenty of packages whose associated libraries were
distributed in a completely separate source distribution and thus were
naturally built as a separate package, which made clear the dependency
between them.

Not finding any obvious examples of packages dependent on their own
outputs made me start to think this was perhaps actually very uncommon
and not supported by Guix for this reason---though now I understand I
didn't find anything because stating this kind of dependency is
generally unnecessary, so I was basically searching for something that
doesn't exist.

[0] https://guix.gnu.org/manual/en/html_node/package-Reference.html#package-Reference
[1] https://guix.gnu.org/manual/en/html_node/Derivations.html#Derivations

-- 
Simon South
simon@simonsouth.net

Re: Specifying dependencies among package outputs?

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 2868 bytes --]

Simon,

Simon South 写道：
> Thank you Brett, Julien and Tobias for your responses. They've 
> been
> super helpful and have cleared things up for me quite a bit.

Very happy to hear that!

> This also clears up for me a bit of remaining confusion around 
> the
> distinction between "inputs" and "propagated inputs": I wondered 
> why, if
> a package's inputs are its dependencies, the "propagated-inputs" 
> field
> is needed at all since surely a package's inputs would always be
> installed alongside it. The explanation is that a package's 
> inputs are
> _not_ its dependencies; they are merely inputs to its build 
> process, and
> whether one becomes a "dependency" depends entirely on whether a
> reference to it remains in any of the package's outputs. The
> "propagated-input" field is used to ensure this association is 
> made,
> even when there is no apparent reference (that Guix can find) in 
> the
> outputs.

Yes, P-Is are a bit of a hack outside of the simple functional 
model.

Note that they're not equivalent to (say) simply echoing some 
store references into a /gnu/store/foo/.propz text file.  They 
affect the resulting profile as if they had been explicitly 
installed into it.  Knot keeps a reference to fstrm, but ‘guix 
install knot’ will not make ‘fstrm_capture’ appear in your $PATH. 
If it were propagated, it would.

Propagation is evil and indispensable.

> Tobias Geerinckx-Rice <me@tobias.gr> writes:
>> If you apply the patch below you'll see (e.g., with ‘guix 
>> size’) that
>> installing only knot:tools will pull in knot{:out,:lib} without 
>> any
>> human-made hints to that effect.
>
> Thank you for this! This is amazing, and exactly the sort of 
> thing I had
> in mind. (Though I wonder if the "tools" output would better be 
> called
> "utils", to match the isc-bind package?)

Rather not.  I dislike pointless abbreviation.  I'm not alone[0]. 
I don't think matching BIND provides value.

> Are you planning on committing these changes? I think they're 
> great.

Sure.  I'll move all of /bin to :tools, since I'm defining :tools 
as ‘reasons I install knot on my laptop’.

I'll keep /sbin in :out.  Some of its commands could be useful 
even without a [running] knotd but I don't think it's likely.

Sound good?

>>> However, Knot's daemon and utilities have the same dependency 
>>> on its
>>> own libraries, so pulling those into a separate "lib" output 
>>> would be
>>> liable to break everything else.
>>
>> Why?
>
> Assuming you didn't mean this rhetorically:

Not at all.  I don't consider rhetorical ‘Why?’s useful and am 
always interested in the answers.  Thanks for taking the time to 
respond at length.  I'll read it at my leisure.

Kind regards,

T G-R

[0]: http://issues.guix.gnu.org/43881#3

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Specifying dependencies among package outputs?

[Simon South]

Tobias Geerinckx-Rice <me@tobias.gr> writes:
> Sound good?

Yep, that feels like a natural division.

Thanks again, and thanks also for the notes on
"propagated-inputs". Onwards and upwards.

-- 
Simon South
simon@simonsouth.net

Re: Specifying dependencies among package outputs?

[Tobias Geerinckx-Rice]

On 2020-10-16 1:45, Simon South wrote:
> Tobias Geerinckx-Rice <me@tobias.gr> writes:
>> Sound good?
> 
> Yep, that feels like a natural division.

I've pushed a polished version of the example to master: man pages now 
end up in the same output as their command.  Man section numbers align 
with the {s,}bin convention so that was easy.

The Texinfo manual less so, as it covers all of Knot.  I see no better 
solution than to ship it separately.  For 'info knot' to work one must 
install (the tiny) knot:doc alongside the desired output(s).

Kind regards,

T G-R

Sent from a Web browser. Excuse or enjoy my brevity.