Don't add propagated-inputs for all outputs

Don't add propagated-inputs for all outputs

[宋文武]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]

Hello, we have a TODO for "extend `propagated-inputs` with support for
multiple outputs".  I try to do it for a while, but unable to find a
clear way, since add a new syntax for specify output in
propagated-inputs require changes in too many places..

Think about the intention, what we want is to avoid unwanted
propagated-inputs for building a package or user profiles, and
propagated-inputs is mostly used for development packages which
requiring headers/libraries from its inputs.  It seems I can hardcode
the choice here to the "dev" output (if a package have both "dev" and
"out", its "out" should considered be an application) or the "out"
output (a library/development package).


Then the change is straight:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-packages-Don-t-propagate-inputs-for-non-development-.patch --]
[-- Type: text/x-patch, Size: 1610 bytes --]

From 98a8666a0cbf33e24efff615243b98144a92c950 Mon Sep 17 00:00:00 2001
Message-ID: <98a8666a0cbf33e24efff615243b98144a92c950.1693047369.git.iyzsong@member.fsf.org>
From: =?UTF-8?q?=E5=AE=8B=E6=96=87=E6=AD=A6?= <iyzsong@member.fsf.org>
Date: Sat, 26 Aug 2023 18:27:09 +0800
Subject: [PATCH] packages: Don't propagate inputs for non-development package
 outputs.

* guix/packages.scm (transitive-inputs): Only include propagated inputs from a
package for its "dev" output, or its "out" output if the package doesn't have
a "dev" one.
---
 guix/packages.scm | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/guix/packages.scm b/guix/packages.scm
index ba98bb0fb4..435d55de71 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -1143,7 +1143,13 @@ (define (transitive-inputs inputs)
            (loop rest result propagated first? seen)
            (loop rest
                  (cons input result)
-                 (cons (package-propagated-inputs package) propagated)
+                 ;; Only add propagated inputs for PACKAGE:dev, or PACKAGE:out
+                 ;; when PACKAGE doesn't have a "dev" output.
+                 (if (if (member "dev" (package-outputs package))
+                         (member "dev" outputs)
+                         (or (null? outputs) (member "out" outputs)))
+                     (cons (package-propagated-inputs package) propagated)
+                     propagated)
                  first?
                  (vhash-consq package outputs seen))))
       ((input rest ...)

base-commit: eeb71d778f149834015858467fbeeb1276d96d1d
-- 
2.41.0


[-- Attachment #3: Type: text/plain, Size: 230 bytes --]



Not much benifits now, but i think it will helps when we have more
mulitple outputs packages.  Also how about add a slimming team aiming to
reduce the closure size of packages and system, anyone interested?

Thanks. 🥳

[bug#65550] [PATCH] profiles: Don't propagate inputs for non-development packages.

[iyzsong--- via Guix-patches via]

From: 宋文武 <iyzsong@member.fsf.org>

* guix/profiles.scm (package->manifest-entry): Only include propagated inputs
from a package for its "dev" output, or its "out" output if the package
doesn't have a "dev" one.
---
 guix/profiles.scm | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/guix/profiles.scm b/guix/profiles.scm
index fea766879d..9ed81dec4d 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -390,7 +390,14 @@ (define* (package->manifest-entry package #:optional (output "out")
                           ((label package output)
                            (package->manifest-entry package output
                                                     #:parent (delay entry))))
-                        (package-propagated-inputs package)))
+                        ;; Only add propagated inputs for PACKAGE:dev, or
+                        ;; PACKAGE:out when PACKAGE doesn't have a "dev"
+                        ;; output.
+                        (if (if (member "dev" (package-outputs package))
+                                (equal? "dev" output)
+                                (equal? "out" output))
+                            (package-propagated-inputs package)
+                            '())))
             (entry (manifest-entry
                      (name (package-name package))
                      (version (package-version package))

base-commit: 98a8666a0cbf33e24efff615243b98144a92c950
-- 
2.41.0

[bug#65550] [PATCH] profiles: Don't propagate inputs for non-development packages.

[Liliana Marie Prikler]

Am Samstag, dem 26.08.2023 um 21:53 +0800 schrieb iyzsong@envs.net:
> From: 宋文武 <iyzsong@member.fsf.org>
> 
> * guix/profiles.scm (package->manifest-entry): Only include
> propagated inputs from a package for its "dev" output, or its "out"
> output if the package doesn't have a "dev" one.
> ---
I think this patch misses the most obvious use case of the out:lib
split.  I also think that hardcoding this list is bound to fail.

Instead, we could for the time being solve this with yet another
package property.
  '((propagate-inputs-from "lib")) ; but not out
  '((propagate-inputs-from . ("lib"))) ; same meaning, different style
  '((propagate-inputs-from "out" "lib")) ; but not doc
If the property is missing, we still propagate from all outputs, as is
currently done.

WDYT?

[bug#65550] [PATCH] profiles: Don't propagate inputs for non-development packages.

[宋文武 via Guix-patches via]

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

> Am Samstag, dem 26.08.2023 um 21:53 +0800 schrieb iyzsong@envs.net:
>> From: 宋文武 <iyzsong@member.fsf.org>
>> 
>> * guix/profiles.scm (package->manifest-entry): Only include
>> propagated inputs from a package for its "dev" output, or its "out"
>> output if the package doesn't have a "dev" one.
>> ---
> I think this patch misses the most obvious use case of the out:lib
> split.  I also think that hardcoding this list is bound to fail.

Ah, that's true.  We currently put both headers and pkgconfig files in
the "lib" output, I think we should put those files into "dev" instead,
only leave shared libraries in "lib" (then we don't need propagated it
anymore).

And I did a test to find packages with "lib" and propagated-inputs:
--8<---------------cut here---------------start------------->8---
(use-modules (gnu packages)
             (guix packages)
             (srfi srfi-1)
             (ice-9 pretty-print))

(define x
  (delete-duplicates
   (fold-packages
    (lambda (package result)
      (if (and (member "lib" (package-outputs package))
               (not (null? (package-propagated-inputs package))))
          (cons (package-name package) result)
          result))
    '())))

(pretty-print x)
--8<---------------cut here---------------end--------------->8---
Only hwloc and apache-arrow, and 24 packages (uniqued by name) have
a "lib" output, seems doable.


> Instead, we could for the time being solve this with yet another
> package property.
>   '((propagate-inputs-from "lib")) ; but not out
>   '((propagate-inputs-from . ("lib"))) ; same meaning, different style
>   '((propagate-inputs-from "out" "lib")) ; but not doc
> If the property is missing, we still propagate from all outputs, as is
> currently done.
>
> WDYT?

Yes, a property is more flexible here, haven't think about it.  Well I
wishful think if we always follow a good convention (put development
files in "dev" or "out"), then maybe we can saving some bytes in code?
I'd like to split more packages with multiple outputs with "dev" to see
how the convention works out.  If not then we could take this 'property'
way.

Thanks!

[bug#65550] [PATCH] profiles: Don't propagate inputs for non-development packages.

[Liliana Marie Prikler]

Am Sonntag, dem 27.08.2023 um 15:30 +0800 schrieb 宋文武:
> > Instead, we could for the time being solve this with yet another
> > package property.
> >   '((propagate-inputs-from "lib")) ; but not out
> >   '((propagate-inputs-from . ("lib"))) ; same meaning, different
> > style
> >   '((propagate-inputs-from "out" "lib")) ; but not doc
> > If the property is missing, we still propagate from all outputs, as
> > is currently done.
> > 
> > WDYT?
> 
> Yes, a property is more flexible here, haven't think about it.  Well
> I wishful think if we always follow a good convention (put
> development files in "dev" or "out"), then maybe we can saving some
> bytes in code?  I'd like to split more packages with multiple outputs
> with "dev" to see how the convention works out.  If not then we could
> take this 'property' way.
I'd really like to avoid yet another special output, when "bin", "lib",
etc. have already been given clear meanings, one of which strongly
overlaps with "stuff that wants propagated inputs for pkg-config
reasons".

The only package that currently has a "dev" output is mariadb, and it's
quite an odd one, as "dev" has been introduced to reduce closure size,
not to propagate even more.  I don't think that hardcoding output names
works given this precedent.  At the very least, "dev" is the wrong
choice.

Cheers

[bug#65550] [PATCH] profiles: Don't propagate inputs for non-development packages.

[宋文武 via Guix-patches via]

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

> I'd really like to avoid yet another special output, when "bin", "lib",
> etc. have already been given clear meanings, one of which strongly
> overlaps with "stuff that wants propagated inputs for pkg-config
> reasons".

The benefit to put headers files and libraries files into seperated
outputs is to reduce the runtime closure size of packages, for example
my home profile contains xfce, emacs, fonts, etc. has total 5GiB (by
guix size), and they have 300MiB under include.

calculated by:
--8<---------------cut here---------------start------------->8---
x=/gnu/store/0fyhci5kv03rfb9430jqs9wki2ifq5ja-profile
guix size $x
for i in `guix size $x`;
  do [ -e $i/include ] && du -sb $i/include;
done | awk '{ sum += $1 } END { print sum / 1024 / 1024 }'
--8<---------------cut here---------------end--------------->8---

If put headers and other development files into a "dev" output, then
those 300MiB can be saved (won't need to be substituted if substitutes
available).  Note that use a "include" output won't help here if you
leave pkg-config files in "lib", since pkg-config files need reference
its include and binaries need reference its libraries.

So it seems to me a "dev" output is unavoidable, also both Debian and
Alpine Linux use '-dev' packages for the same reason, it should be
familiar to learn..

[bug#65550] [PATCH] profiles: Don't propagate inputs for non-development packages.

[Liliana Marie Prikler]

Am Dienstag, dem 29.08.2023 um 18:24 +0800 schrieb 宋文武:
> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> 
> > I'd really like to avoid yet another special output, when "bin",
> > "lib", etc. have already been given clear meanings, one of which
> > strongly overlaps with "stuff that wants propagated inputs for pkg-
> > config reasons".
> 
> The benefit to put headers files and libraries files into seperated
> outputs is to reduce the runtime closure size of packages, for
> example my home profile contains xfce, emacs, fonts, etc. has total
> 5GiB (by guix size), and they have 300MiB under include.
> 
> calculated by:
> --8<---------------cut here---------------start------------->8---
> x=/gnu/store/0fyhci5kv03rfb9430jqs9wki2ifq5ja-profile
> guix size $x
> for i in `guix size $x`;
>   do [ -e $i/include ] && du -sb $i/include;
> done | awk '{ sum += $1 } END { print sum / 1024 / 1024 }'
> --8<---------------cut here---------------end--------------->8---
> 
> If put headers and other development files into a "dev" output, then
> those 300MiB can be saved (won't need to be substituted if
> substitutes available).  Note that use a "include" output won't help
> here if you leave pkg-config files in "lib", since pkg-config files
> need reference its include and binaries need reference its libraries.
> 
> So it seems to me a "dev" output is unavoidable, also both Debian and
> Alpine Linux use '-dev' packages for the same reason, it should be
> familiar to learn..
There is little point in a dev package when you blow up its size by
propagating inputs…

[bug#65550] [PATCH] profiles: Don't propagate inputs for non-development packages.

[宋文武 via Guix-patches via]

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

> Am Dienstag, dem 29.08.2023 um 18:24 +0800 schrieb 宋文武:
>> [...]
>> If put headers and other development files into a "dev" output, then
>> those 300MiB can be saved (won't need to be substituted if
>> substitutes available).  Note that use a "include" output won't help
>> here if you leave pkg-config files in "lib", since pkg-config files
>> need reference its include and binaries need reference its libraries.
>> 
>> So it seems to me a "dev" output is unavoidable, also both Debian and
>> Alpine Linux use '-dev' packages for the same reason, it should be
>> familiar to learn..
> There is little point in a dev package when you blow up its size by
> propagating inputs…

Well propagated-inputs won't increase size of a package, they bring some
of its inputs when build other one to remove the need to list those some
inputs as inputs of the other one.

Consider gtk+ and mousepad:
  gtk+ propagated cairo, freetype, pango, etc.
  
  so I don't need add those inputs to mousepad, and if gtk+ doesn't
  propagated them then I must add them to mousepad because those are
  actually needed (because pkg-config check them and need link flags
  from those dependencies of gtk+).

But for runtime, when substitute of mousepad is available, if we split
thoes packages with "dev" outputs, guix size mousepad would still be
something like:

...
/gnu/store/xxxx-mesa-23.1.4
/gnu/store/xxxx-freetype-2.13.0
/gnu/store/xxxx-gtk+-3.24.37
...

It won't include any "dev" output of its dependencies packages, since
development files are not referenced by mousepad's binary, so the size
of headers and development files is saved (won't be substituted).

Only when substitute of mousepad is not available, you'll start by download
mesa-23.1.4-dev, gtk+3.24.37-dev, etc.  And after build, those "dev"
outputs can be GCed immediately.


Hope it's clear now!

[bug#65550] Don't add propagated-inputs for all outputs

[Maxim Cournoyer]

Hi,

宋文武 <iyzsong@envs.net> writes:

> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
>
>> I'd really like to avoid yet another special output, when "bin", "lib",
>> etc. have already been given clear meanings, one of which strongly
>> overlaps with "stuff that wants propagated inputs for pkg-config
>> reasons".
>
> The benefit to put headers files and libraries files into seperated
> outputs is to reduce the runtime closure size of packages, for example
> my home profile contains xfce, emacs, fonts, etc. has total 5GiB (by
> guix size), and they have 300MiB under include.
>
> calculated by:
>
> x=/gnu/store/0fyhci5kv03rfb9430jqs9wki2ifq5ja-profile
> guix size $x
> for i in `guix size $x`;
>   do [ -e $i/include ] && du -sb $i/include;
> done | awk '{ sum += $1 } END { print sum / 1024 / 1024 }'
>
> If put headers and other development files into a "dev" output, then
> those 300MiB can be saved (won't need to be substituted if substitutes
> available).  Note that use a "include" output won't help here if you
> leave pkg-config files in "lib", since pkg-config files need reference
> its include and binaries need reference its libraries.
>
> So it seems to me a "dev" output is unavoidable, also both Debian and
> Alpine Linux use '-dev' packages for the same reason, it should be
> familiar to learn..

I'm not convinced that saving 300 MiB on 5 GiB is worth making using
Guix packages more complicated, especially in this age of compressed
file systems (text files compress very well).  We'd also have to change
a couple things to make it convenient and not have to plaster the code
base with ugly `(,lib "dev") inputs ^^'.

I reckon it could all be automated, but I'm not sure it's worth the
effort at this stage (there bigger low hanging fruits to pick in my
opinion -- such as moving static libraries automatically in the gnu
build system, paying more attention to large documentation or unecessary
installed files, etc.).

-- 
Thanks,
Maxim

[bug#65550] [PATCH] profiles: Don't propagate inputs for non-development packages.

[Josselin Poiret via Guix-patches via]

[-- Attachment #1: Type: text/plain, Size: 875 bytes --]

Hi everyone,

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

> I think this patch misses the most obvious use case of the out:lib
> split.  I also think that hardcoding this list is bound to fail.
>
> Instead, we could for the time being solve this with yet another
> package property.
>   '((propagate-inputs-from "lib")) ; but not out
>   '((propagate-inputs-from . ("lib"))) ; same meaning, different style
>   '((propagate-inputs-from "out" "lib")) ; but not doc
> If the property is missing, we still propagate from all outputs, as is
> currently done.
>
> WDYT?

I don't really like the hackiness of this (yet another weird thing for
independent contributors to learn), but I also understand the need.
This one LGTM, but maybe we should think a bit more about simpler
solutions before committing to this.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

[bug#65550] Don't add propagated-inputs for all outputs

[Maxim Cournoyer]

Hi Liliana and 宋文武,

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

> Am Samstag, dem 26.08.2023 um 21:53 +0800 schrieb iyzsong@envs.net:
>> From: 宋文武 <iyzsong@member.fsf.org>
>>
>> * guix/profiles.scm (package->manifest-entry): Only include
>> propagated inputs from a package for its "dev" output, or its "out"
>> output if the package doesn't have a "dev" one.
>> ---
> I think this patch misses the most obvious use case of the out:lib
> split.  I also think that hardcoding this list is bound to fail.
>
> Instead, we could for the time being solve this with yet another
> package property.
>   '((propagate-inputs-from "lib")) ; but not out
>   '((propagate-inputs-from . ("lib"))) ; same meaning, different style
>   '((propagate-inputs-from "out" "lib")) ; but not doc
> If the property is missing, we still propagate from all outputs, as is
> currently done.

I'm rather against the proposed changes as it stands.  I think it'd lead
to a lot of noise in the code base and favor more complex splits of
packages, which I'm doubtful is worth the cost (in terms of
hackiness/complexity).

Since I've known Guix, I've appreciated that a simple 'find $(guix build
some-package) typically shows me the package whole, except perhaps for
its doc and debug symbols.  Having to know which packages have a "lib"
outputs when using them in package definitions is also not fun in my
experience (you start by adding the package, then it fails, then you
verify its outputs, then you add `(,package "lib")), and I fear going to
a "dev" output would bring the same kind of annoyance but at larger
scale.

I think we'd need to evaluate what we'd gain in terms of size reduction
a bit more carefully before moving in this direction and how it'd impact
the user experience.  E.g., if we can reduce the minimal Guix image size
by maybe 30%, that'd be nice, but if we're talking of about 5% like in
your example profile, it doesn't seem worth the complexity to me.

-- 
Thanks,
Maxim

[bug#65550] Don't add propagated-inputs for all outputs

[宋文武 via Guix-patches via]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi Liliana and 宋文武,
>
> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
>
>> Am Samstag, dem 26.08.2023 um 21:53 +0800 schrieb iyzsong@envs.net:
>>> From: 宋文武 <iyzsong@member.fsf.org>
>>>
>>> * guix/profiles.scm (package->manifest-entry): Only include
>>> propagated inputs from a package for its "dev" output, or its "out"
>>> output if the package doesn't have a "dev" one.
>>> ---
>> I think this patch misses the most obvious use case of the out:lib
>> split.  I also think that hardcoding this list is bound to fail.
>>
>> Instead, we could for the time being solve this with yet another
>> package property.
>>   '((propagate-inputs-from "lib")) ; but not out
>>   '((propagate-inputs-from . ("lib"))) ; same meaning, different style
>>   '((propagate-inputs-from "out" "lib")) ; but not doc
>> If the property is missing, we still propagate from all outputs, as is
>> currently done.
>
> [...]
>
> I think we'd need to evaluate what we'd gain in terms of size reduction
> a bit more carefully before moving in this direction and how it'd impact
> the user experience.  E.g., if we can reduce the minimal Guix image size
> by maybe 30%, that'd be nice, but if we're talking of about 5% like in
> your example profile, it doesn't seem worth the complexity to me.

I agree, thanks Maxim and Liliana for the inputs!

[bug#65550] Don't add propagated-inputs for all outputs

[Maxim Cournoyer]

Hi,

宋文武 <iyzsong@envs.net> writes:

[...]

>> I think we'd need to evaluate what we'd gain in terms of size reduction
>> a bit more carefully before moving in this direction and how it'd impact
>> the user experience.  E.g., if we can reduce the minimal Guix image size
>> by maybe 30%, that'd be nice, but if we're talking of about 5% like in
>> your example profile, it doesn't seem worth the complexity to me.
>
> I agree, thanks Maxim and Liliana for the inputs!

Should we close this issue for now?

-- 
Thanks,
Maxim

bug#65550: Don't add propagated-inputs for all outputs

[宋文武 via Guix-patches via]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi,
>
> 宋文武 <iyzsong@envs.net> writes:
>
> [...]
>
>>> I think we'd need to evaluate what we'd gain in terms of size reduction
>>> a bit more carefully before moving in this direction and how it'd impact
>>> the user experience.  E.g., if we can reduce the minimal Guix image size
>>> by maybe 30%, that'd be nice, but if we're talking of about 5% like in
>>> your example profile, it doesn't seem worth the complexity to me.
>>
>> I agree, thanks Maxim and Liliana for the inputs!
>
> Should we close this issue for now?

Yes, closing.

[bug#65550] [PATCH] profiles: Don't propagate inputs for non-development packages.

[Ludovic Courtès]

Hello!

iyzsong@envs.net skribis:

> From: 宋文武 <iyzsong@member.fsf.org>
>
> * guix/profiles.scm (package->manifest-entry): Only include propagated inputs
> from a package for its "dev" output, or its "out" output if the package
> doesn't have a "dev" one.

What’s the rationale?

Right now, there’s only one package with an output called “dev”:

--8<---------------cut here---------------start------------->8---
$ guix package -A |cut -f3|grep dev|wc -l
1
--8<---------------cut here---------------end--------------->8---

So the focus on “dev” is debatable.

More importantly, this would break things like:

  guix shell python python-scipy -C -- python3 -c 'import numpy'

… where ‘python-numpy’ is propagated by ‘python-scipy’.

Definitely not something we want to change.

WDYT?

Ludo’.