How many bytes do we add (closure of guix) when adding one new package?

[Simon Tournier <zimon.toutoune@gmail.com>]

Hi,

The initial discussion was about the closure of Guix and that “guix pull”
brings graphical libraries.  See #63050 [1].

Here, I would like to open a discussion about how Guix scales,
i.e. about its size.  I am trying to answer the question I am asking as
the subject. ;-)

It’s another angle to view Andreas and Ludo discussion: :-)

>> Note that I do not care so much about the closure size, but about the
>> number of packages that are needed to just build guix (although of course
>> the two are related). Or otherwise said, the dependencies for "guix pull".
>
> Yes, understood.  Graphviz is not in the closure anyway, it’s a
> build-only dependency.

Somehow, the closure is increasing:

--8<---------------cut here---------------start------------->8---
$ for i in $(seq 1 4); do guix time-machine --commit=v1.$i.0 -- size guix | grep 'total:' ;done
total: 410.9 MiB
total: 496.0 MiB
total: 564.8 MiB
total: 637.2 MiB

$ guix size guix | grep 'total:'
total: 611.2 MiB
--8<---------------cut here---------------end--------------->8---

(Yeah, the package guix is not exactly the same as guix itself, but it
appears to me a good enough approximation.  And my current revision is
14c0380.)

Compare:

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=v1.1.0 -- size guix --sort=self | wc -l
44

$ guix time-machine --commit=v1.4.0 -- size guix --sort=self | wc -l
72

$ guix size guix --sort=self | wc -l
70
--8<---------------cut here---------------end--------------->8---

which is the Andreas’s concern for “exotic” architectures.  Moreover,
the inflation (in size) is about some packages that are just becoming
bigger.

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=v1.1.0 -- size guix --sort=self | head
store item                                                       total    self
/gnu/store/fp16m5hkzql7jwhvnkm1j1i5qch0arhx-guix-1.1.0rc2-1.9d0d27f   410.9   221.6  53.9%
/gnu/store/1mkkv2caiqbdbbd256c4dirfi4kwsacv-guile-2.2.6            123.9    44.4  10.8%
/gnu/store/ahqgl4h89xqj695lgqvsaf6zh2nhy4pj-glibc-2.29              37.4    35.8   8.7%
/gnu/store/2plcy91lypnbbysb18ymnhaw3zwk8pg1-gcc-7.4.0-lib           70.0    32.6   7.9%
/gnu/store/n79cf8bvy3k96gjk1rf18d36w40lkwlr-glibc-utf8-locales-2.29    13.9    13.9   3.4%
/gnu/store/k2m4q2av9hw73hw2jx6qrxqdyh855398-openssl-1.1.1c          76.4     6.4   1.6%
/gnu/store/gzp4ig4rdb1qf4i5dy1d9nl0zmj5q09y-ncurses-6.1-20190609    75.9     5.9   1.4%
/gnu/store/hfvz18igm68p5yz7z4asn6ph363blp1z-gnutls-3.6.9           130.6     5.1   1.2%
/gnu/store/b5vjmib411m74lbpf051fnwz3s9zcw79-guile-git-0.3.0         98.8     4.4   1.1%

$ guix time-machine --commit=v1.4.0 -- size guix --sort=self | head
store item                                                       total    self
/gnu/store/9nvx97hr8kkr26gzwni2fblfn0yq0xjw-guix-1.4.0rc2          637.2   330.1  51.8%
/gnu/store/qlmpcy5zi84m6dikq3fnx5dz38qpczlc-guile-3.0.8            130.0    53.0   8.3%
/gnu/store/cnfsv9ywaacyafkqdqsv2ry8f01yr7a9-guile-3.0.7            129.1    52.0   8.2%
/gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33              38.3    36.6   5.7%
/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib          71.7    33.4   5.2%
/gnu/store/96srhmpmxa20wmsck95g3iq4hb3lz4a0-glib-2.70.2             98.1    15.3   2.4%
/gnu/store/mw3py6smb1pk8yx298hd9ivz9lzbksqi-glibc-utf8-locales-2.33    13.9    13.9   2.2%
/gnu/store/5583c2za2jsn9g6az79rnksgvigwnsk7-util-linux-2.37.2-lib    80.7     9.0   1.4%
/gnu/store/9rrnm5hdjw7cy96a2a9rfgh6y08wsbmf-ncurses-6.2.20210619    77.6     5.9   0.9%

$ guix size guix --sort=self | head
store item                                                       total    self
/gnu/store/cgjddvw9zay626z8hyxl0zmn1354c24k-guix-1.4.0-6.dc5430c   611.2   350.2  57.3%
/gnu/store/4gvgcfdiz67wv04ihqfa8pqwzsb0qpv5-guile-3.0.9            135.0    53.1   8.7%
/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35              40.6    38.8   6.3%
/gnu/store/930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib          75.3    34.7   5.7%
/gnu/store/nb40pwd37v6i1g4b1fq4l6q4h9px3asr-glib-2.72.3            101.3    14.9   2.4%
/gnu/store/5fmqijrs5f7vx8mc2q2pmq26yvhb74sm-glibc-utf8-locales-2.35    13.9    13.9   2.3%
/gnu/store/gwx2sf5wl9bsl21lwv35g5la63bwyy95-util-linux-2.37.4-lib    84.3     9.0   1.5%
/gnu/store/69wd3pd1hd3j84xr965jj2fk2qmxn0hl-openssl-3.0.8           83.4     8.1   1.3%
/gnu/store/bcc053jvsbspdjr17gnnd9dg85b3a0gy-ncurses-6.2.20210619    81.2     5.9   1.0%
--8<---------------cut here---------------end--------------->8---

Considering Guix itself, one explanation for the increase is the number
of packages – assuming the services and other are negligible; “git diff
--shortstat” is a good indicator at first sight.  Well, we could be more
precise about the documentation.  Hum, this ugly,

--8<---------------cut here---------------start------------->8---
$ for doc in $(for ci in $(for t in $(git tag | grep v1 | grep -v rc ); do git --no-pager show $t | grep commit ;done); do for d in $(find ~/.cache/guix/inferiors/ -type l -print); do printf "$d "; $d/bin/guix --version 2>/dev/null ;done | grep $ci ;done | cut -f1 -d' '); do du -sh $(readlink -f $doc)/share/* ;done | grep info
172K	/gnu/store/5pa1706ckwhn6x4mn5kl2b7h15k3in9x-profile/share/info
200K	/gnu/store/z1icpkfbz59dr7k7rnb0jd8j1ii8mdph-profile/share/info
376K	/gnu/store/hm0rwgcvrs85y3hgjsw8616cxy61h6si-profile/share/info
304K	/gnu/store/zbrgzk7l0j7805i82sl3gmx6y2b0iz9q-profile/share/info
--8<---------------cut here---------------end--------------->8---

is probably providing a clue about the assumption.

Ok, let assume that the packages are the main source of size increasing.

The question is: can we evaluate the size for one package?  How many
bytes do we add to the whole Guix when we add one package?  On average
and roughly.

We have the number of packages and the whole size for successive
versions.  Therefore, we can do the difference between the two.  We get
[2078, 1848, 4704, 1532] which means 2078 packages had been added
between v1.1.0 and v1.2.0, 1848 between v1.2.0 and v1.3.0, etc.

We can do the same for the size, [22.9, 19.4, 66.2, 20.1] and then we
can compute the ratio: the size per package.  Something like:

[0.011020211742059676, 0.010497835497835485, 0.01407312925170069, 0.013120104438642276]

Let get an average: 0.012177820232559531.

Now, let take the number of packages for v1.1.0 and do the
multiplication.  We get: 159.6 MiB.

Ok, it means that the difference is more or less the core of Guix – what
we are assuming that is slowly growing.  It reads 62 MiB.

Therefore, we can predict the size for the other versions using this
linear model based on the previous evaluated average.

    size = mean * number_packages + core

It reads:

[246.9, 269.4, 326.7, 345.3] compared to [244.5, 263.9, 330.1, 350.2].

Hum, this quick back-to-the-envelope computation does not seem too bad.
I guess.

Conclusions:

 1. the addition of one package leads to an increase of ~ 12 KiB

 2. the core of Guix is about ~ 62 MiB

 3. doubling the number of packages is doubling the size to download at
    “guix pull” time.


Maybe, we should re-think (guix self).  Especially the *package-modules*
part and re-discuss if we could split that part.  From my understanding.


Cheers,
simon

1: https://issues.guix.gnu.org/issue/63050