Guix size reduction work group

Guix size reduction work group

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1034 bytes --]

Hi!

At FOSDEM 2020 Ludovic explained to the world why Guix is better than
other container formats (to put it mildly ;p).

Possibly the only counter-argument to "Guix  > Snap / Flatpak" is
the size of the Guix containers.

I also talked to Mathieu who has been working relentlessly on
cross-compilation and embedded device support recently.  He gave a
presentation on Guix vs. Yocto at FOSDEM.  But again, the main
counter-argument is that Yocto allows for much smaller systems.  (I
think Mathieu has some figures.)

Shall we start a work group to fix the issue?

- Write a blog article to explain the issue and a detailed process on
  how to fix it.  (Embed it to the manual.)

- Improve the tooling.  In my experience, guix graph is quickly unusable
  with a high number of nodes.  Maybe d3.js could be leveraged to add a
  filtering system, or a way to click on nodes to hide them and all
  their children.

- How do we compare to Nix?

Other suggestions?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix size reduction work group

[zimoun]

Hi Pierre,

On Tue, 4 Feb 2020 at 16:57, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> Shall we start a work group to fix the issue?

One of the obvious bottleneck is the documentation. Some packages come
with a full doc and sometimes with Pandoc and/or TeX stuff. In
general, it is required but when speaking about more minimalism...
So, the only way is to examine package by package and to check the
status of the documentation, IMHO.


All the best,
simon

Re: Guix size reduction work group

[Julien Lepiller]

Le 4 février 2020 11:22:10 GMT-05:00, zimoun <zimon.toutoune@gmail.com> a écrit :
>Hi Pierre,
>
>On Tue, 4 Feb 2020 at 16:57, Pierre Neidhardt <mail@ambrevar.xyz>
>wrote:
>
>> Shall we start a work group to fix the issue?
>
>One of the obvious bottleneck is the documentation. Some packages come
>with a full doc and sometimes with Pandoc and/or TeX stuff. In
>general, it is required but when speaking about more minimalism...
>So, the only way is to examine package by package and to check the
>status of the documentation, IMHO.
>
>
>All the best,
>simon

Another thing to address is the package closure size by pruning some elements from packages' closures. The obvious one is gcc, that appears in some closure because of a file that registers compilation commands for instance. Another situation is when a package requires a part of another package, and another part of that package (unused) requires more stuff.
This happened with mariadb: some software needed mariadb's library, but the mariadb also came with binaries that required more dependencies. Splitting mariadb reduced the closure of qt5 by 200 MB for instance, for a combination of these two reasons, and the fact that binaries in mariadb are really big.

Re: Guix size reduction work group

[Ludovic Courtès]

Hi!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Shall we start a work group to fix the issue?
>
> - Write a blog article to explain the issue and a detailed process on
>   how to fix it.  (Embed it to the manual.)

The “Submitting Patches” section mentions closure size specifically.  Is
there anything you think we should add there?

> - Improve the tooling.  In my experience, guix graph is quickly unusable
>   with a high number of nodes.  Maybe d3.js could be leveraged to add a
>   filtering system, or a way to click on nodes to hide them and all
>   their children.

‘guix size’ is key here: it’s a profiler, exactly what we need IMO.
WDYT?

> - How do we compare to Nix?

A few years back we were doing better because we used separate outputs
in key places where Nixpkgs didn’t.  Later on Nixpkgs had a large part
of its packages split in several outputs (more than we do).  From what I
heard, it wasn’t as fruitful as they had hoped it would be in terms of
closure size, but it might still be better than what we have, dunno.

The thing is, I think it’s something that requires constant care, every
time we add a package or modify an existing one.  It’s very easy to lose
benefits that had been previously obtained through hard work!

At any rate, I agree we need to improve!

Ludo’.

Re: Guix size reduction work group

[zimoun]

Hi Ludo,

On Wed, 5 Feb 2020 at 16:18, Ludovic Courtès <ludo@gnu.org> wrote:

> > - Improve the tooling.  In my experience, guix graph is quickly unusable
> >   with a high number of nodes.  Maybe d3.js could be leveraged to add a
> >   filtering system, or a way to click on nodes to hide them and all
> >   their children.
>
> ‘guix size’ is key here: it’s a profiler, exactly what we need IMO.
> WDYT?

It is hard to examine the graph with "guix size".
Sometimes, I am doing: "guix graph foo | grep"; especially with 'bag'
and friends. (I do not want to speak for Mathieu but he told me that
he is doing the same time to time. :-))

I agree that using "guix size" is already enough to find the culprits.


> > - How do we compare to Nix?
>
> A few years back we were doing better because we used separate outputs
> in key places where Nixpkgs didn’t.  Later on Nixpkgs had a large part
> of its packages split in several outputs (more than we do).  From what I
> heard, it wasn’t as fruitful as they had hoped it would be in terms of
> closure size, but it might still be better than what we have, dunno.

IMHO, there is 2 directions: add more outputs or duplicate package variants.

For example Emacs:

for pkg in emacs-xwidgets emacs emacs-no-x-toolkit emacs-no-x emacs-minimal;
do
      echo $pkg
     guix size $pkg | tail -n1
     echo ""
done

emacs-xwidgets
total: 1222.4 MiB

emacs
total: 1003.7 MiB

emacs-no-x-toolkit
total: 522.6 MiB

emacs-no-x
total: 275.6 MiB

emacs-minimal
total: 204.2 MiB


And from my opinion, not enough packages are variant-ed.


> The thing is, I think it’s something that requires constant care, every
> time we add a package or modify an existing one.  It’s very easy to lose
> benefits that had been previously obtained through hard work!

I have never thought, neither tried but is it possible to find and/or
build all the packages that 'inherit' from a specific one?


Cheers,
simon

Re: Guix size reduction work group

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

>> The thing is, I think it’s something that requires constant care, every
>> time we add a package or modify an existing one.  It’s very easy to lose
>> benefits that had been previously obtained through hard work!
>
> I have never thought, neither tried but is it possible to find and/or
> build all the packages that 'inherit' from a specific one?

Nope (‘inherit’ is purely syntactic, it doesn’t “live on” at run time.)
What would it buy you, though?

Ludo’.

Re: Guix size reduction work group

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]

Hello Ludo,

Ludovic Courtès <ludo@gnu.org> ezt írta (időpont: 2020. febr. 7., Pén
22:36):

> Hi,
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
> >> The thing is, I think it’s something that requires constant care, every
> >> time we add a package or modify an existing one.  It’s very easy to lose
> >> benefits that had been previously obtained through hard work!
> >
> > I have never thought, neither tried but is it possible to find and/or
> > build all the packages that 'inherit' from a specific one?
>
> Nope (‘inherit’ is purely syntactic, it doesn’t “live on” at run time.)
> What would it buy you, though?
>

It is currently easy to break packages by updating a package that is
inherited from. You have no way to know about that relationship by simple
inspection, and is not discoverable by current tooling either. So I also
believe that it would be useful to at least be able to list them. One place
where this becomes painful is in bootstrap chains.

Do you think this can be achieved somehow without complicating
implementation?


> Ludo’.
>

Best regards,
g_bor

>

[-- Attachment #2: Type: text/html, Size: 2053 bytes --]

Re: Guix size reduction work group

[zimoun]

Hi,

On Fri, 7 Feb 2020 at 23:31, Gábor Boskovits <boskovits@gmail.com> wrote:
> Ludovic Courtès <ludo@gnu.org> ezt írta (időpont: 2020. febr. 7., Pén 22:36):
>> zimoun <zimon.toutoune@gmail.com> skribis:

>> >> The thing is, I think it’s something that requires constant care, every
>> >> time we add a package or modify an existing one.  It’s very easy to lose
>> >> benefits that had been previously obtained through hard work!
>> >
>> > I have never thought, neither tried but is it possible to find and/or
>> > build all the packages that 'inherit' from a specific one?
>>
>> Nope (‘inherit’ is purely syntactic, it doesn’t “live on” at run time.)
>> What would it buy you, though?
>
> It is currently easy to break packages by updating a package that is inherited from. You have no way to know about that relationship by simple inspection, and is not discoverable by current tooling either. So I also believe that it would be useful to at least be able to list them. One place where this becomes painful is in bootstrap chains.

Yes, exactly that. :-)


> Do you think this can be achieved somehow without complicating implementation?

Maybe grouping all the packages with the same 'origin'. It should
possible, isn't it?


Cheers,
simon

Re: Guix size reduction work group

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2016 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi!
>
> Pierre Neidhardt <mail@ambrevar.xyz> skribis:
>
>> Shall we start a work group to fix the issue?
>>
>> - Write a blog article to explain the issue and a detailed process on
>>   how to fix it.  (Embed it to the manual.)
>
> The “Submitting Patches” section mentions closure size specifically.  Is
> there anything you think we should add there?

It does not give much details, e.g. the ones that Julien mentioned.
Also I don't think `guix size' is enough, see below.

>> - Improve the tooling.  In my experience, guix graph is quickly unusable
>>   with a high number of nodes.  Maybe d3.js could be leveraged to add a
>>   filtering system, or a way to click on nodes to hide them and all
>>   their children.
>
> ‘guix size’ is key here: it’s a profiler, exactly what we need IMO.
> WDYT?

Maybe I'm missing something, but in general I think that guix size only
gives a birds eye view and does not allow for closer inspection.

Say FOO has BAR in its closure, but not in the explicit inputs, how can
I figure out which of the indirect inputs drags BAR in?

With an extensive guix graph, it quickly becomes impossible to follow
the millions of arrows.

What I'd like to have is an interactive graph that I can trim to links
between given nodes.  This would allow me to ask "give me the dependency
chain that links FOO and BAR".

> The thing is, I think it’s something that requires constant care, every
> time we add a package or modify an existing one.  It’s very easy to lose
> benefits that had been previously obtained through hard work!

This is a good point.  "Adding" a package is less critical since it does
not impact the closure size of the rest.  For "updates" maybe we could
leverage the continuous integration to flag a warning when a new build
has increased the size of the package compared to a previous build by
some threshold.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix size reduction work group

[zimoun]

Hi Pierre,

On Sat, 8 Feb 2020 at 17:43, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> >> - Improve the tooling.  In my experience, guix graph is quickly unusable
> >>   with a high number of nodes.  Maybe d3.js could be leveraged to add a
> >>   filtering system, or a way to click on nodes to hide them and all
> >>   their children.
> >
> > ‘guix size’ is key here: it’s a profiler, exactly what we need IMO.
> > WDYT?
>
> Maybe I'm missing something, but in general I think that guix size only
> gives a birds eye view and does not allow for closer inspection.
>
> Say FOO has BAR in its closure, but not in the explicit inputs, how can
> I figure out which of the indirect inputs drags BAR in?

I do not understand what you are looking for, but there is already:

   guix graph -t reverse-package
    guix graph -t reverse-bag

that does more or less what I understand you want.


> With an extensive guix graph, it quickly becomes impossible to follow
> the millions of arrows.

Using a static graphviz, I agree. And I generally end up by grepping
directly in the 'dot' output file.

Maybe something more dynamic using 'd3.js' or similar to view *big*
graphs could help.



> What I'd like to have is an interactive graph that I can trim to links
> between given nodes.  This would allow me to ask "give me the dependency
> chain that links FOO and BAR".

I agree that tools to work with the graph would be welcome.
Currently, the graph is here but under-used, IMHO.

The first step seems to list what operations and filtering is missing.



> > The thing is, I think it’s something that requires constant care, every
> > time we add a package or modify an existing one.  It’s very easy to lose
> > benefits that had been previously obtained through hard work!
>
> This is a good point.  "Adding" a package is less critical since it does
> not impact the closure size of the rest.  For "updates" maybe we could
> leverage the continuous integration to flag a warning when a new build
> has increased the size of the package compared to a previous build by
> some threshold.

You could propose such feature to the Guix Data Service.
For example, on this webpage [1], the history of all the Git package
in Guix is shown. The closure size could be reported.

[1] http://data.guix.gnu.org/repository/1/branch/master/package/git



All the best,
simon

Re: Guix size reduction work group

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1375 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

>> Say FOO has BAR in its closure, but not in the explicit inputs, how can
>> I figure out which of the indirect inputs drags BAR in?
>
> I do not understand what you are looking for, but there is already:
>
>    guix graph -t reverse-package
>     guix graph -t reverse-bag

Yes, but this produces way too big a graph, which was my point below.

> Maybe something more dynamic using 'd3.js' or similar to view *big*
> graphs could help.

Indeed!

> The first step seems to list what operations and filtering is missing.

I'm not sure what d3.js can do, but I can think of the following:

- The ability to "hide" nodes and all their edges by making them
  translucent (so that they can be unhidden).

- Hide/show all nodes inside the mouse rectangular selection.

- Hide/show all nodes matching some text.

- Close the current selection, that is, if some intermediary nodes are
  hidden for the currently showing nodes, show them.

> You could propose such feature to the Guix Data Service.
> For example, on this webpage [1], the history of all the Git package
> in Guix is shown. The closure size could be reported.
>
> [1] http://data.guix.gnu.org/repository/1/branch/master/package/git

This is a great idea!  Christopher, what do you think?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix size reduction work group

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1520 bytes --]


Pierre Neidhardt <mail@ambrevar.xyz> writes:

> zimoun <zimon.toutoune@gmail.com> writes:
>
>> You could propose such feature to the Guix Data Service.
>> For example, on this webpage [1], the history of all the Git package
>> in Guix is shown. The closure size could be reported.
>>
>> [1] http://data.guix.gnu.org/repository/1/branch/master/package/git
>
> This is a great idea!  Christopher, what do you think?

I don't think adding to that particular page is feasible, but I think
having a page just of this information would be great :)

I'm not quite sure how that would work, but the Guix Data Service can
query for narinfo files, and from those it can store the size of
specific outputs, as well as there references.

So the closure size (I think?) is the size of an output, plus the
closure size of all of its references (the definition is recursive).

Then you could compute the closure size for the output (or set of
outputs) for a package, and then graph that over the different revisions
on a branch. Because this is using information from builds, it's a
little complicated by the information potentially varying between build
server, for example, if the nar non-deterministically includes a
reference.

Going back to the point about preventing regressions though, there I
think it would be good to enhance the comparision functionality of the
Guix Data Service so that it can compare derivation outputs, and say how
the closure size has changed. Then you could look at this when reviewing
patches.

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Guix size reduction work group

[zimoun]

On Mon, 10 Feb 2020 at 09:09, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> zimoun <zimon.toutoune@gmail.com> writes:

> > I do not understand what you are looking for, but there is already:
> >
> >    guix graph -t reverse-package
> >     guix graph -t reverse-bag
>
> Yes, but this produces way too big a graph, which was my point below.

Fact of life. ;-)



> > The first step seems to list what operations and filtering is missing.
>
> I'm not sure what d3.js can do, but I can think of the following:

I do not know what is the state of art to visualize "big" graph with
hundreds or more nodes.
I agree that something reactive could be really cool and helpful.


Cheers,
simon

Re: Guix size reduction work group

[Julien Lepiller]

Le 10 février 2020 03:09:57 GMT-05:00, Pierre Neidhardt <mail@ambrevar.xyz> a écrit :
>zimoun <zimon.toutoune@gmail.com> writes:
>
>>> Say FOO has BAR in its closure, but not in the explicit inputs, how
>can
>>> I figure out which of the indirect inputs drags BAR in?
>>
>> I do not understand what you are looking for, but there is already:
>>
>>    guix graph -t reverse-package
>>     guix graph -t reverse-bag
>
>Yes, but this produces way too big a graph, which was my point below.
>
>> Maybe something more dynamic using 'd3.js' or similar to view *big*
>> graphs could help.
>
>Indeed!
>
>> The first step seems to list what operations and filtering is
>missing.
>
>I'm not sure what d3.js can do, but I can think of the following:
>
>- The ability to "hide" nodes and all their edges by making them
>  translucent (so that they can be unhidden).
>
>- Hide/show all nodes inside the mouse rectangular selection.
>
>- Hide/show all nodes matching some text.
>
>- Close the current selection, that is, if some intermediary nodes are
>  hidden for the currently showing nodes, show them.
>
>> You could propose such feature to the Guix Data Service.
>> For example, on this webpage [1], the history of all the Git package
>> in Guix is shown. The closure size could be reported.
>>
>> [1] http://data.guix.gnu.org/repository/1/branch/master/package/git
>
>This is a great idea!  Christopher, what do you think?

I remember there was a script for graphviz floating around that could do exactly that: given two nodes it would only keep those in between them. Then you could generate a much smaller graph.

Re: Guix size reduction work group

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

Ludovic Courtès 写道：
> Nope (‘inherit’ is purely syntactic, it doesn’t “live on” at run 
> time.)
> What would it buy you, though?

In addition to what Gábor mentioned, ‘guix refresh -l’ rebuild 
numbers can be dangerously misleading when inheritance is 
involved.

Would this not be useful information to record?  When I was new 
here I played around with doing just that but my Guile-foo was too 
weak.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Guix size reduction work group

[Ludovic Courtès]

Hi,

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

> Ludovic Courtès 写道：
>> Nope (‘inherit’ is purely syntactic, it doesn’t “live on” at run
>> time.)
>> What would it buy you, though?
>
> In addition to what Gábor mentioned, ‘guix refresh -l’ rebuild numbers
> can be dangerously misleading when inheritance is involved.
>
> Would this not be useful information to record?  When I was new here I
> played around with doing just that but my Guile-foo was too weak.

For this specific use case, I agree it would be useful info.

For the more general record use case, I’m skeptical because:

  1. (foo (inherit (foo (a 1)))) would no longer be equal? to
     (foo (a 1)), which makes little sense from a programming viewpoint.

  2. The whole inheritance chain would be kept in memory, when in fact
     ‘inherit’ is nothing more than syntactic sugar to copy a record.

Maybe we could go fancy and augment ‘define-record-type*’ such that one
can ask for “inheritance reification”, but #2 sounds like that could
quickly make it impractical.

Ludo’.

Re: Guix size reduction work group

[Ludovic Courtès]

Hello!

Christopher Baines <mail@cbaines.net> skribis:

> I'm not quite sure how that would work, but the Guix Data Service can
> query for narinfo files, and from those it can store the size of
> specific outputs, as well as there references.
>
> So the closure size (I think?) is the size of an output, plus the
> closure size of all of its references (the definition is recursive).

Yes.  So the Data Service could pick that info from narinfo files and
have per-package pages with a graph showing the closure/self size
evolution—IOW, ‘guix size’ over time.

Doable, but easier said than done!

Ludo’.

Re: Guix size reduction work group

[Ludovic Courtès]

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi!
>>
>> Pierre Neidhardt <mail@ambrevar.xyz> skribis:
>>
>>> Shall we start a work group to fix the issue?
>>>
>>> - Write a blog article to explain the issue and a detailed process on
>>>   how to fix it.  (Embed it to the manual.)
>>
>> The “Submitting Patches” section mentions closure size specifically.  Is
>> there anything you think we should add there?
>
> It does not give much details, e.g. the ones that Julien mentioned.
> Also I don't think `guix size' is enough, see below.

‘guix size’ is a profiler, so when one is looking for ways to optimize
the size of package closures, it’s a good tool.

A graphical and interactive rendering would surely be an improvement,
but honestly, I don’t think that’s what’s preventing us from reducing
the size of our packages.  :-)

Thanks,
Ludo’.