Avoiding rebuilds (e.g. of mariadb, the entire Rust chain, etc.)?

Avoiding rebuilds (e.g. of mariadb, the entire Rust chain, etc.)?

[Matthew Brooks]

Is there any way to avoid rebuilding stuff like mariadb, the entire Rust chain, etc. unless one of those packages *actually* changes? It seems like every few days every single package needs to rebuild for some reason, including many packages that spend unbelievably long times running tests that will never actually be of use to me, so I'm usually only able to update every couple of weeks since so much constantly needs to be rebuilt and everything takes so long.

It seems to have gotten worse over time as well, as more and more base packages pull in extremely computation-hungry dependency chains. At this very moment, for example, I'm waiting for Rust & co. to compile simply to delete old system generations from the bootloader.

I've got substitutes turned off (primarily because I like the idea of every package being "home built" as it were), but I can't imagine that even the official substitiute servers would be able to keep up with the constant rebuilds my system seems to want unless I've got something horribly misconfigured somewhere.

The definition for the guix service in my config is:

(modify-services
 %base-services
 (guix-service-type
  config =>
  (guix-configuration
   (inherit config)
   (use-substitutes? #f)
   (authorized-keys (list
         (local-file "./laptop-key.pub")
         (local-file "./tablet-key.pub")
         ))
   (substitute-urls '())
   (extra-options '("--gc-keep-derivations=yes" "--gc-keep-outputs=yes" "--no-substitutes"))
   ))
 )

I've virtually zero experience with Scheme or Guile, so maybe there's something obvious I'm missing, or some obvious way to change the behavior, but I've never been able to find anything about it.

Re: Avoiding rebuilds (e.g. of mariadb, the entire Rust chain, etc.)?

[David Lecompte via]

[-- Attachment #1.1: Type: text/plain, Size: 1897 bytes --]

Le dimanche 26 décembre 2021 à 23:41 -0600, Matthew Brooks a écrit :
> Is there any way to avoid rebuilding stuff like mariadb, the entire
> Rust chain, etc. unless one of those packages *actually* changes? It
> seems like every few days every single package needs to rebuild for
> some reason, including many packages that spend unbelievably long
> times running tests that will never actually be of use to me, so I'm
> usually only able to update every couple of weeks since so much
> constantly needs to be rebuilt and everything takes so long.
> 

I don't have a solution, just similar experience but with a different
setup.

I use Guix on Trisquel on my X200, I have substitues enabled, my
installed packages are glibc-locales, fontconfig, font-adobe-source-
han-sans, gajim, gajim-omemo, gajim-openpgp, nheko, syncthing,
darktable,  and ungoogled-chromium.

Often, "guix pull" and "guix package update" take between 1h and 2h.
This is not due to the download time. Apparently, even with substitues,
some CPU-heavy work is needed.

> It seems to have gotten worse over time as well, as more and more
> base packages pull in extremely computation-hungry dependency chains.
> At this very moment, for example, I'm waiting for Rust & co. to
> compile simply to delete old system generations from the bootloader.
> 

I started "guix package -u" about 40h ago, it has been on ungoogled-
chromium-96.0.4664.110-1.drv for more than 30h now and it is not
finished.

For a long time, both cores of the CPU have been at 100% load. Now, it
is much lower and I see a lot of disk activity but still working.
During the whole time, I have almost not used the computer at all.

I will have to travel in two hours and I need to switch my laptop off
first. If the update is not finished, I will stop the process. I guess
I will remove ungoogled chromium as well.


[-- Attachment #1.2: publickey - david.lecompte@metani.info - 292b3e27.asc --]
[-- Type: application/pgp-keys, Size: 3212 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 855 bytes --]

Re: Avoiding rebuilds (e.g. of mariadb, the entire Rust chain, etc.)?

[Leo Famulari]

On Sun, Dec 26, 2021 at 11:41:30PM -0600, Matthew Brooks wrote:
> Is there any way to avoid rebuilding stuff like mariadb, the entire Rust chain, etc. unless one of those packages *actually* changes? It seems like every few days every single package needs to rebuild for some reason, including many packages that spend unbelievably long times running tests that will never actually be of use to me, so I'm usually only able to update every couple of weeks since so much constantly needs to be rebuilt and everything takes so long.

In Guix, packages are considered to have changed when any part of their
dependency graph changes. So, no, there is not a way to avoid rebuilding
a package when one of its transitive dependencies has changed.

We have guidelines about how to make changes to the distro that are
based on how many rebuilds they cause. See item 8:

https://guix.gnu.org/manual/en/html_node/Submitting-Patches.html#Submitting-Patches

Both MariaDB and Rust have several thousand packages that depend on
them, so we change these packages seldomly, and use the build farm to
rebuild everything in advance so that users don't have to.

If users choose to avoid substitutes, that's fine too, but there's no
way to avoid doing the work.

Recently, we deployed the results of a "core-updates" project, which
updated core packages, and caused every single package to be rebuilt.
So, that's why you are having to rebuild more than usual.

> It seems to have gotten worse over time as well, as more and more base packages pull in extremely computation-hungry dependency chains. At this very moment, for example, I'm waiting for Rust & co. to compile simply to delete old system generations from the bootloader.

Yes, Rust continues to grow in importance and has become a relatively
low-level dependency of the distro on the x86_64 platform. We are
continually finding ways to shorten the "bootstrap chain" of Rust, but
it's still fairly long.

> I've got substitutes turned off (primarily because I like the idea of every package being "home built" as it were), but I can't imagine that even the official substitiute servers would be able to keep up with the constant rebuilds my system seems to want unless I've got something horribly misconfigured somewhere.

The build farm has ~2800 powerful CPU cores for the x86_64 architecture,
so it can keep up. Other architectures are less well-resourced. And of
course very few users have access to those kinds of resources.

>    (extra-options '("--gc-keep-derivations=yes" "--gc-keep-outputs=yes" "--no-substitutes"))

I would have suggested adding those first two options. Beyond that,
maybe others can suggest something that will help.

Re: Avoiding rebuilds (e.g. of mariadb, the entire Rust chain, etc.)?

[zimoun]

Hi,

On Mon, 27 Dec 2021 at 09:23, David Lecompte via <help-guix@gnu.org> wrote:
> Le dimanche 26 décembre 2021 à 23:41 -0600, Matthew Brooks a écrit :
>> Is there any way to avoid rebuilding stuff like mariadb, the entire
>> Rust chain, etc. unless one of those packages *actually* changes? It
>> seems like every few days every single package needs to rebuild for
>> some reason, including many packages that spend unbelievably long
>> times running tests that will never actually be of use to me, so I'm
>> usually only able to update every couple of weeks since so much
>> constantly needs to be rebuilt and everything takes so long.
>> 
>
> I don't have a solution, just similar experience but with a different
> setup.
>
> I use Guix on Trisquel on my X200, I have substitues enabled, my
> installed packages are glibc-locales, fontconfig, font-adobe-source-
> han-sans, gajim, gajim-omemo, gajim-openpgp, nheko, syncthing,
> darktable,  and ungoogled-chromium.
>
> Often, "guix pull" and "guix package update" take between 1h and 2h.
> This is not due to the download time. Apparently, even with substitues,
> some CPU-heavy work is needed.

Maybe I miss something with your requirements.  To avoid such
disagreements, I do 3 things:

 1) enable ’channel-with-substitutes-available’ [1]

 2) split profiles: essentials under ~/.guix-profile (nss-certs, git,
 recutils, glibc-locales, tree, etc.); emacs under
 ~/.config/guix/profiles/emacs, python under
 ~/.config/guix/profiles/python, etc.  All the profiles managed using
 manifest.scm [2]

 3) run “guix weather -m manifest.scm” before updating


Basically, most of the time, I do not need the very last version and I
can wait that the build farm has it.  The only annoyance is security
fixes and substitutes not ready yet.  In this case, in the meantime, I
run the package with “guix shell -C” which is a weak mitigation or “guix
system vm” if I am in paranoid mode. :-)


1: <https://guix.gnu.org/manual/devel/en/guix.html#Channels-with-Substitutes>
2: <https://guix.gnu.org/cookbook/en/guix-cookbook.html#Guix-Profiles-in-Practice>


Cheers,
simon

Re: Avoiding rebuilds (e.g. of mariadb, the entire Rust chain, etc.)?

[zimoun]

Hi,

On Sun, 26 Dec 2021 at 23:41, Matthew Brooks <matthewfbrooks@mailbox.org> wrote:

> I've got substitutes turned off (primarily because I like the idea of
> every package being "home built" as it were), but I can't imagine that
> even the official substitiute servers would be able to keep up with
> the constant rebuilds my system seems to want unless I've got
> something horribly misconfigured somewhere. 

The only “home built” are the “tunable“ [1] packages, which are very
rare.  Otherwise, Guix tries very hard to build portable packages [2].

Personally, I see only 2 cases to have substitutes turned off:

  1) you often challenge your builds against substitutes ones and report
     reproducibility or performances issues,
  2) you inspect all the sources for trusting them before building.

If not, for what my opinion is worth, I do not understand the aim for
burning such energy in this world of climate change.

1: <https://issues.guix.gnu.org/52283#27>
2:<https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/>


Cheers,
simon

Re: Avoiding rebuilds (e.g. of mariadb, the entire Rust chain, etc.)?

[David Lecompte]

[-- Attachment #1.1: Type: text/plain, Size: 1319 bytes --]

Le mercredi 29 décembre 2021 à 10:12 +0100, zimoun a écrit :
> Maybe I miss something with your requirements.  To avoid such
> disagreements, I do 3 things:
> 
>  1) enable ’channel-with-substitutes-available’ [1]
> 
>  2) split profiles: essentials under ~/.guix-profile (nss-certs, git,
>  recutils, glibc-locales, tree, etc.); emacs under
>  ~/.config/guix/profiles/emacs, python under
>  ~/.config/guix/profiles/python, etc.  All the profiles managed using
>  manifest.scm [2]
> 
>  3) run “guix weather -m manifest.scm” before updating
> 

Thanks for the suggestions!

I created a channels.scm file to enable ’channel-with-substitutes-
available’ following what is indicated in [1]. Does that have an effect
even without the split profiles?

I am still reading [2] on the split profiles, I will try when I become
confident I somehow understood (I never used a manifest.scm file so
far).

I saw that, even without a manifest.scm file, I can run "guix weather"
with a list of packages as argument, so I can do that with the
installed packages to get a clue before running an update, and wait in
case some substitute is not available yet.

My (still very low) understanding of Guix is getting slightly higher
thanks to your advice, thanks again!

David.


[-- Attachment #1.2: publickey - david.lecompte@metani.info - 292b3e27.asc --]
[-- Type: application/pgp-keys, Size: 3212 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 855 bytes --]

Re: Avoiding rebuilds (e.g. of mariadb, the entire Rust chain, etc.)?

[zimoun]

Hi,

On Sun, 02 Jan 2022 at 17:36, David Lecompte <david.lecompte@metani.info> wrote:
> Le mercredi 29 décembre 2021 à 10:12 +0100, zimoun a écrit :

> I saw that, even without a manifest.scm file, I can run "guix weather"
> with a list of packages as argument, so I can do that with the
> installed packages to get a clue before running an update, and wait in
> case some substitute is not available yet.

In addition, you can go to CI and give a look at the status.

The entry point is <https://ci.guix.gnu.org/> then you can click on the
dashboard (row master, on the right the “screen“ icon).

Then you are on the last evaluation.  All green bullets are substituable
packages and red are failure.  In the field ‘search for jobs’, you can
filter, for instance type ‘emacs-’ and you see all green except 2.  Let
click on emacs-md4rd and you are there:

<https://ci.guix.gnu.org/build/240317/details>

You can click on the raw log and then read it:

<https://ci.guix.gnu.org/build/240317/log/raw>


Last, sometimes it fails because a dependency failed, as here:

<https://ci.guix.gnu.org/build/265498/details>

Therefore, click on “Show More” and follow the links.


Hope that helps,
simon