Guix pull: avoiding "Computing Guix derivation"

Guix pull: avoiding "Computing Guix derivation"

[Edouard Klein]

Hi Guix,

First, I'd like to apologize for not having taken the time to answer
those who helped me on a previous guix performance issue (with
containers), the reason is tied to the topic of this email: the store
has eaten all the space of my server, and solving that takes precedence
over everything else, because no space == no services.

So, I need to clear some space, and to do that I need to have every user
run guix pull (and by that I mean root will sudo -u $user guix pull),
then update all of their profiles, and then guix gc
--delete-generations.

This ought to turn deduplication up to 11, and enjoy a reduced store
size.

I've already solved the cache size problem:
mount -t overlay overlay -o \
lowerdir="/root/.cache/guix",upperdir="/home/$user/.cache/guix-overlay",workdir="/home/$user/.cache/guix-workdir" \
"/home/$user/.cache/guix"

Then
bindfs --mirror=$user /home/$user/.cache/guix /home/$user/.cache/guix

This lets root (who just ran guix pull) share its cache with every user,
and avoid blasting away 700MB of disk space in every $HOME to reproduce
the cache.

However, now, I'm facing the previously addressed problem of guix pull
being slow and hungry:
https://www.mail-archive.com/guix-devel@gnu.org/msg66442.html (Guix pull speed)
https://yhetil.org/guix/87h6mwf4u3.fsf@lapenas.dev/T/ (guix pull
performance)

On my server, in order to run guix pull, I have to stop all other
services otherwise I run out of ram.

Then, one root has pulled, I need to wait 4 minutes /per user/ for guix
pull to finish its "Computing Guix derivation" step.

I would like to know two things, one is for the sake of knowledge, and
the other is to solve the problem at hand:

- Why is this step not substitutable ? The inputs are known, a hash can
be derived, a substitute server could be queried for an output of that
hash ? What am I missing ? Does the guix derivation not end up in the
store ? What makes it so special that it can't be served by a substitute
server ?

- Is there a way (even a very dirty one, like hand copying stuff accross
  /var/guix/profiles/per-user/*/current-guix) I can stop paying this 4
  minutes per user price ? As I said, this is downtime on my server, as
  I need to stop all other services to let guix pull finish.

Thanks in advance.

Sorry for beating a dead horse, it's just that I can't scale anything up
until I solve these performance issues. Sure I could rent a bigger
server, but that's just kicking the can down the road.


Cheers,

Edouard.

Re: Guix pull: avoiding "Computing Guix derivation"

[Ryan Sundberg]

Hi Edouard, I have couple of ideas for you, which may help with your deployment process:

1) if possible, mount /gnu on a btrfs filesystem with zstd compression enabled. This will let you fill up at least 3x of your physical disk space because the store compresses very well.

2) for production servers you should try using `guix deploy` to push rather than pull profiles. This will offload most of the process to the machine pushing the deployment.

With regards to multi user profiles, if this is a typical server (not a terminal/shell server) I wonder why you need to have per user profiles at all. I only say this because I am not sure about running `guix deploy` on an individual user basis.

Sincerely,
Ryan Sundberg

May 13, 2024 8:08:35 AM Edouard Klein <edou@rdklein.fr>:

> Hi Guix,
> 
> First, I'd like to apologize for not having taken the time to answer
> those who helped me on a previous guix performance issue (with
> containers), the reason is tied to the topic of this email: the store
> has eaten all the space of my server, and solving that takes precedence
> over everything else, because no space == no services.
> 
> So, I need to clear some space, and to do that I need to have every user
> run guix pull (and by that I mean root will sudo -u $user guix pull),
> then update all of their profiles, and then guix gc
> --delete-generations.
> 
> This ought to turn deduplication up to 11, and enjoy a reduced store
> size.
> 
> I've already solved the cache size problem:
> mount -t overlay overlay -o \
> lowerdir="/root/.cache/guix",upperdir="/home/$user/.cache/guix-overlay",workdir="/home/$user/.cache/guix-workdir" \
> "/home/$user/.cache/guix"
> 
> Then
> bindfs --mirror=$user /home/$user/.cache/guix /home/$user/.cache/guix
> 
> This lets root (who just ran guix pull) share its cache with every user,
> and avoid blasting away 700MB of disk space in every $HOME to reproduce
> the cache.
> 
> However, now, I'm facing the previously addressed problem of guix pull
> being slow and hungry:
> https://www.mail-archive.com/guix-devel@gnu.org/msg66442.html (Guix pull speed)
> https://yhetil.org/guix/87h6mwf4u3.fsf@lapenas.dev/T/ (guix pull
> performance)
> 
> On my server, in order to run guix pull, I have to stop all other
> services otherwise I run out of ram.
> 
> Then, one root has pulled, I need to wait 4 minutes /per user/ for guix
> pull to finish its "Computing Guix derivation" step.
> 
> I would like to know two things, one is for the sake of knowledge, and
> the other is to solve the problem at hand:
> 
> - Why is this step not substitutable ? The inputs are known, a hash can
> be derived, a substitute server could be queried for an output of that
> hash ? What am I missing ? Does the guix derivation not end up in the
> store ? What makes it so special that it can't be served by a substitute
> server ?
> 
> - Is there a way (even a very dirty one, like hand copying stuff accross
>   /var/guix/profiles/per-user/*/current-guix) I can stop paying this 4
>   minutes per user price ? As I said, this is downtime on my server, as
>   I need to stop all other services to let guix pull finish.
> 
> Thanks in advance.
> 
> Sorry for beating a dead horse, it's just that I can't scale anything up
> until I solve these performance issues. Sure I could rent a bigger
> server, but that's just kicking the can down the road.
> 
> 
> Cheers,
> 
> Edouard.

Re: Guix pull: avoiding "Computing Guix derivation"

[Simon Tournier]

Hi,

On lun., 13 mai 2024 at 17:04, Edouard Klein <edou@rdklein.fr> wrote:

> - Why is this step not substitutable ? The inputs are known, a hash can
> be derived, a substitute server could be queried for an output of that
> hash ? What am I missing ? Does the guix derivation not end up in the
> store ? What makes it so special that it can't be served by a substitute
> server ?

Assume we are running two different Guix revisions, say A and B.  And at
the end of our respective “guix pull”, we expect to have the same
revision, say C.  We expect to then run the same Guix.

Other said, how can we “compile” the code of C using one machinery from
A and another potentially different from B and expect to have the same
result?

Somehow, we need an intermediary step: something minimal that is
independent of A and B but produces the same C.  And it’s the aim of
“Computing derivation” with the script build-aux/build-self.scm.

The inputs are known, indeed.  However, the ones from A and from B are
not necessary the sames.  For instance, Guile of A might be different of
Guile of B.  Somehow, that “Computing derivation” is what allows to time
travel.

Well, that’s my understanding and I could have missing something.


For instance, I am running:

        $ guix describe
        Generation 4	mars 25 2024 18:22:25	(current)
          guix 929ddec
            repository URL: https://git.savannah.gnu.org/git/guix.git
            branch: master
            commit: 929ddec8f4a181be653152c7436581c2adc54eee

Assume it is revision A.  From there, let run ’guix pull’.

--8<---------------cut here---------------start------------->8---
$ time guix pull
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Authenticating channel 'guix', commits 9edb3f6 to 6ba29e0 (51 new commits)...
Building from this channel:
  guix      https://git.savannah.gnu.org/git/guix.git	6ba29e0
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
 module-import  2KiB               305KiB/s 00:00 ▕██████████████████▏ 100.0%
 module-import-compiled  1.2MiB    306KiB/s 00:04 ▕██████████████████▏ 100.0%
 compute-guix-derivation  1015B    1.8MiB/s 00:00 ▕██████████████████▏ 100.0%
Computing Guix derivation for 'x86_64-linux'... /
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/115prkj2rp58gl2lr0qy6higpliw8xpy-profile.drv
  /gnu/store/56dsm4f5snfmnfgkki4x41fj0xcl394m-guix-daemon.drv
  /gnu/store/77r2c33r1cq03qrrrfsf3g6mvmjp0w92-guix-command.drv
  /gnu/store/0y0fq1lmz3dwqi0fpbq4g6swkc9yljmp-guix-cli-modules.drv
  /gnu/store/bbczy39rid3q6j191mj397ak589s1101-guix-system-modules.drv
  /gnu/store/gx3ngbxmb6ayzjpxv9afsigvajc5d7h4-guix-extra-modules.drv
  /gnu/store/i0b6inna8a8bxg75k3dxr4xl6jrn53cm-guix-cli-core-modules.drv
  /gnu/store/iww6bi0ciaqkjjbqp26iclq2ijd1bgck-guix-system-tests-modules.drv
  /gnu/store/x0ai22yrq99kp3il0x3nf1d3yj5fydqy-guix-packages-base-modules.drv

70,7 MB will be downloaded
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%

[...]

 guix-package-cache  872KiB        303KiB/s 00:03 ▕██████████████████▏ 100.0%
 guix-cli  315KiB                  241KiB/s 00:01 ▕██████████████████▏ 100.0%
 guix-cli-core  922KiB             194KiB/s 00:05 ▕██████████████████▏ 100.0%
 guix-cli-modules  260B            253KiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-cli-core-modules  261B       450KiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-extra  2.8MiB                363KiB/s 00:08 ▕██████████████████▏ 100.0%
 guix-manual  5.2MiB               273KiB/s 00:19 ▕██████████████████▏ 100.0%
 guix-extra-modules  261B          187KiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-packages-base  17.4MiB       271KiB/s 01:06 ▕██████████████████▏ 100.0%
 guix-system  7.9MiB               264KiB/s 00:31 ▕██████████████████▏ 100.0%
 guix-packages-base-modules  265B  279KiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-system-modules  262B         282KiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-system-tests  921KiB         332KiB/s 00:03 ▕██████████████████▏ 100.0%
 guix-system-tests-modules  267B   279KiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-6ba29e021-modules  31.2MiB   299KiB/s 01:47 ▕██████████████████▏ 100.0%
 guix-module-union  2KiB           385KiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-command  655B                1.0MiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-daemon  402B                 711KiB/s 00:00 ▕██████████████████▏ 100.0%
 guix-6ba29e021  802B              499KiB/s 00:00 ▕██████████████████▏ 100.0%

[...]

real	5m8,633s
user	1m53,567s
sys	0m1,530s
--8<---------------cut here---------------end--------------->8---

Ok, all is substitutable (wow! 70MiB that’s not nothing… another story)
except the part “Computing derivation”.

Now, from another revision B (here I choose C to make things more
confusing ;-)), let run:

--8<---------------cut here---------------start------------->8---
simon@lili$ time guix time-machine --commit=6ba29e0 -- describe 
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://guix.bordeaux.inria.fr'... 100.0%
[...]
 config.scm  507B                  377KiB/s 00:00 ▕██████████████████▏ 100.0%
 git.scm  105B                     127KiB/s 00:00 ▕██████████████████▏ 100.0%
 hash.scm  130B                    267KiB/s 00:00 ▕██████████████████▏ 100.0%
 module-import  2KiB               557KiB/s 00:00 ▕██████████████████▏ 100.0%
 module-import  2KiB               577KiB/s 00:00 ▕██████████████████▏ 100.0%
 module-import-compiled  1.2MiB    302KiB/s 00:04 ▕██████████████████▏ 100.0%
 module-import-compiled  1.2MiB    244KiB/s 00:05 ▕██████████████████▏ 100.0%
building /gnu/store/icka5f4977darnyw75is2v716k0agp48-compute-guix-derivation.drv...
Computing Guix derivation for 'x86_64-linux'... |
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://guix.bordeaux.inria.fr'... 100.0%
The following derivations will be built:
  /gnu/store/iq1xhzdjc2lhvfmg5ap65a1wx2mwbynw-profile.drv
  /gnu/store/jkslyhj4m35bqrg0am8y5qlh9gsdamyv-guix-6ba29e021.drv
  /gnu/store/8nc0vjaz96n3rixxdi57w0zk7ja5nh7h-guix-daemon.drv
  /gnu/store/fryzibmg9vrc542qra59qmzf8d6n86sw-guix-command.drv
  /gnu/store/qqnxqvqsbjm1mglisnmm9sb0n6043c9i-guix-module-union.drv
  /gnu/store/q9p35jxy0jr4zaldm6nmkj91zkhsv0an-guix-6ba29e021-modules.drv
  /gnu/store/5fz6m02fgjnyqf359wv47hwwfhfwk64b-guix-packages-modules.drv
  /gnu/store/jm5il7lsxigw0d63cv1vwx1kxgm06580-guix-home-modules.drv
  /gnu/store/jz26dqj8kai936h2fj0szdnkmnlhmfl3-guix-config-modules.drv
  /gnu/store/lf4k886h1cqnj39774hmnpfsxxi40cnb-guix-config.drv
  /gnu/store/lh77nxg1hl3snzfd8f5fngzfj6zbr8jn-guix-config-source.drv
  /gnu/store/593nr6hirjgkvmpfyvfw8laqha2ycgfw-config.scm.drv
  /gnu/store/jjqy49vzcfcvdkg59cpssvg643bsilkl-inferior-script.scm.drv
  /gnu/store/pbfh548cj96kd7gxdyrs1q4ljdgdb1gn-profile.drv

2,4 MB will be downloaded
 module-import-compiled  107KiB    145KiB/s 00:01 ▕██████████████████▏ 100.0%
 guix-home  751KiB                 180KiB/s 00:04 ▕██████████████████▏ 100.0%
 guix-packages  1.5MiB             210KiB/s 00:07 ▕██████████████████▏ 100.0%
building /gnu/store/593nr6hirjgkvmpfyvfw8laqha2ycgfw-config.scm.drv...
building /gnu/store/jm5il7lsxigw0d63cv1vwx1kxgm06580-guix-home-modules.drv...
building /gnu/store/lh77nxg1hl3snzfd8f5fngzfj6zbr8jn-guix-config-source.drv...
building /gnu/store/5fz6m02fgjnyqf359wv47hwwfhfwk64b-guix-packages-modules.drv...
building /gnu/store/lf4k886h1cqnj39774hmnpfsxxi40cnb-guix-config.drv...
building /gnu/store/jz26dqj8kai936h2fj0szdnkmnlhmfl3-guix-config-modules.drv...
building /gnu/store/q9p35jxy0jr4zaldm6nmkj91zkhsv0an-guix-6ba29e021-modules.drv...
building /gnu/store/qqnxqvqsbjm1mglisnmm9sb0n6043c9i-guix-module-union.drv...
building /gnu/store/fryzibmg9vrc542qra59qmzf8d6n86sw-guix-command.drv...
building /gnu/store/8nc0vjaz96n3rixxdi57w0zk7ja5nh7h-guix-daemon.drv...
building /gnu/store/jkslyhj4m35bqrg0am8y5qlh9gsdamyv-guix-6ba29e021.drv...
[...]

  guix 6ba29e0
    repository URL: https://git.savannah.gnu.org/git/guix.git
    commit: 6ba29e02108ed144da1234b4b5512ee03865dcf6

real	1m34,408s
user	1m50,315s
sys	0m1,421s
--8<---------------cut here---------------end--------------->8---

Similar story.  All is substitutatble except “Computing derivation”.


Now, if we scrutinize, let compare the script we run when type “guix”:

(A) /gnu/store/77r2c33r1cq03qrrrfsf3g6mvmjp0w92-guix-command.drv

    and
    
(B) /gnu/store/fryzibmg9vrc542qra59qmzf8d6n86sw-guix-command.drv


The outputs are different (I did not expect that ;-))

(A) /gnu/store/mwv6xzx8ggyzb67jwj1pmxyr8zndq12x-guix-command

    and 

(B) /gnu/store/4jjd4zfyvp2g8gnp3wqzaxnn9xs9sdan-guix-command


Well, I was expecting the same guix-command.  Why not the same?  Because
one thing is different (aside the order of listed inputs):

(A) /gnu/store/39l6dl5q5287wnn063nx4qgd3gx1l6zy-guix-module-union.drv

    and
    
(B) /gnu/store/qqnxqvqsbjm1mglisnmm9sb0n6043c9i-guix-module-union.drv


Let dive in more.  The two outputs of these derivations look similar;
except one file: config.scm.  (Aside .go binary compiled files which are
known to not be reproducible.)  The difference between revision A and B
reads:

--8<---------------cut here---------------start------------->8---
(define %channel-metadata
  (quote
   (repository
    (version 0)
    (url "https://git.savannah.gnu.org/git/guix.git")
(A)    (branch "master")
(B)    (branch #f)
    (commit "6ba29e02108ed144da1234b4b5512ee03865dcf6")
    (name guix)
    (introduction
     (channel-introduction
      (version 0)
      (commit "9edb3f66fd807b096b48283debdcddccfea34bad")
      (signer "BBB0 2DDF 2CEA F6A8 0D1D  E643 A2A0 6DF2 A33A 54FA"))))))
--8<---------------cut here---------------end--------------->8---

and that’s because one is from regular “guix pull”, and the other is
from “guix time-machine”.  Well, maybe that could be considered as a
bug, that’s another story. ;-)


All the point is to build the same result starting from two different
machinery (different Guile, modules, etc) and ending to the same output.
Considering the combinatorial, it would not be possible to substitutes;
it would mean compute all the couples A-C and B-C.

That’s an hard topic and speed it up would be very nice.  Bah it would
require to redesign stuff and perhaps find a process to build the same
output using two different toolchains, somehow.

Again, maybe I am missing something.


> - Is there a way (even a very dirty one, like hand copying stuff accross
>   /var/guix/profiles/per-user/*/current-guix) I can stop paying this 4
>   minutes per user price ? As I said, this is downtime on my server, as
>   I need to stop all other services to let guix pull finish.

Nothing I am aware.

For this situation with different users, are they all running the same
Guix revision and targeting the same final Guix revision?

If yes, indeed ’guix pull’ should be costly only once and then instantly
fast for all the others; if that’s not already the case.

If no, as explained, it seems hard to avoid the cost of “Computing
derivation“ for each user using a different Guix revision.

Well, maybe I am missing something.

Cheers,
simon

Re: Guix pull: avoiding "Computing Guix derivation"

[Richard Sent]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On lun., 13 mai 2024 at 17:04, Edouard Klein <edou@rdklein.fr> wrote:
>
>> - Why is this step not substitutable ? The inputs are known, a hash can
>> be derived, a substitute server could be queried for an output of that
>> hash ? What am I missing ? Does the guix derivation not end up in the
>> store ? What makes it so special that it can't be served by a substitute
>> server ?
>
> Assume we are running two different Guix revisions, say A and B.  And at
> the end of our respective “guix pull”, we expect to have the same
> revision, say C.  We expect to then run the same Guix.
>
> Other said, how can we “compile” the code of C using one machinery from
> A and another potentially different from B and expect to have the same
> result?
>
> Somehow, we need an intermediary step: something minimal that is
> independent of A and B but produces the same C.  And it’s the aim of
> “Computing derivation” with the script build-aux/build-self.scm.
>
> The inputs are known, indeed.  However, the ones from A and from B are
> not necessary the sames.  For instance, Guile of A might be different of
> Guile of B.  Somehow, that “Computing derivation” is what allows to time
> travel.
> ...
> All the point is to build the same result starting from two different
> machinery (different Guile, modules, etc) and ending to the same output.
> Considering the combinatorial, it would not be possible to substitutes;
> it would mean compute all the couples A-C and B-C.
>
> That’s an hard topic and speed it up would be very nice.  Bah it would
> require to redesign stuff and perhaps find a process to build the same
> output using two different toolchains, somehow.

Thanks for the detailed explanation Simon!

Given that Guix packages itself as a package, I wonder if there is room
to utilize the "Guix package" Guix as a uniform base to build "Guix, the
collection of channels" and skip "computing Guix derivation".

For instance,

--8<---------------cut here---------------start------------->8---
~ $ guix shell guix -- guix describe
The following derivation will be built:
  /gnu/store/lk0qkgsz33qzhi6c83bw4nbc3sbzjb3s-profile.drv

building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
  guix 4c94b9e
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 4c94b9e983bc51d9504655f1e7727c4f6d14b6b7
--8<---------------cut here---------------end--------------->8---

Instead of A and B building C directly, A and B download the
substitutable Guix package D, then use D to build C. Because D is a
reproducible package, it should be substitutable from both A and B.
Then, because D->C is the same for everyone, we could substitute the
Guix derivation computation for C.

My understanding is this would resolve the question of "how to build C
given that A and B are different without a long derivation computation?"

--8<---------------cut here---------------start------------->8---
~ $ guix shell guix -- guix time-machine -q --branch=master -- describe
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Computing Guix derivation for 'x86_64-linux'... /
substitute: updating substitutes from 'http://10.1.1.101:80'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/yp694ahf5fb8sm06r9llmgciax8yq20j-profile.drv
  /gnu/store/dw2gzlz8n93dmvkpb9wbr45p7awsixal-profile.drv
  /gnu/store/x59ywl8rncs0aqjqnxmjyhyzcznnm643-inferior-script.scm.drv

building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
building /gnu/store/x59ywl8rncs0aqjqnxmjyhyzcznnm643-inferior-script.scm.drv...
building package cache...
building profile with 1 package...
  guix 0edbb93
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 0edbb93130651f29dc59fe4ca546a5065670ac8a
--8<---------------cut here---------------end--------------->8---

I imagine there's a large amount of behind the scenes work that would
need to be done for this to work, but if nothing else it's an
interesting thought experiment!

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

Re: Guix pull: avoiding "Computing Guix derivation"

[Simon Tournier]

Hi,

On lun., 13 mai 2024 at 17:11, Richard Sent <richard@freakingpenguin.com> wrote:

> Instead of A and B building C directly, A and B download the
> substitutable Guix package D, then use D to build C. Because D is a
> reproducible package, it should be substitutable from both A and B.
> Then, because D->C is the same for everyone, we could substitute the
> Guix derivation computation for C.

Maybe I am missing some details.  From my understanding, your D is the
result of the “Computing derivation” dance.  And it is a minimalist
build because it must work for both cases: with or without substitutes.

Somehow, my understanding of the current process is:

  A -> D  -> C
  B -> D* -> C

And, D or D* can be the same script.  Here, the property is a kind of
idempotency.  Hence, C is substitutable.

IIUC, your proposal is to fix D (the intermediary step).  You propose
the package named ’guix’ that changes barely, but it could be something
more minimalist.  The requirements is: susbtitutable.  The problem is
transferred to the first arrow, no?

How can we be sure that A and B points to the same D?

Other said, A lives in the past.  Later, we changed D becoming D*
because some bugs.  The other revision B lives after this change.
Therefore, we have the same picture:

  A -> D
  B -> D*

But then how do we go to C since D and D* does not have a kind of
idempotent property.

From my understanding, the current situation/process is very similar
with the one for compiling compiler: you have your new Source of
compiler and an OldCompiler binary, so you run:

  OldCompiler(Source) -> Compiler
  and again Compiler(Source) -> NewCompiler

Somehow, whatever the choice of some OldCompiler binary, you get the
same NewCompiler binary (aside trusting trust attack and friends,
obviously ;-))

The story of “guix pull” is not so different; from my understanding.

Again, maybe I am missing something.


Cheers,
simon

Re: Guix pull: avoiding "Computing Guix derivation"

[Richard Sent]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On lun., 13 mai 2024 at 17:11, Richard Sent <richard@freakingpenguin.com> wrote:
>
>> Instead of A and B building C directly, A and B download the
>> substitutable Guix package D, then use D to build C. Because D is a
>> reproducible package, it should be substitutable from both A and B.
>> Then, because D->C is the same for everyone, we could substitute the
>> Guix derivation computation for C.
>
> Maybe I am missing some details.  From my understanding, your D is the
> result of the “Computing derivation” dance.  And it is a minimalist
> build because it must work for both cases: with or without substitutes.
>
> Somehow, my understanding of the current process is:
>
>   A -> D  -> C
>   B -> D* -> C
>
> And, D or D* can be the same script.  Here, the property is a kind of
> idempotency.  Hence, C is substitutable.
>
> IIUC, your proposal is to fix D (the intermediary step).  You propose
> the package named ’guix’ that changes barely, but it could be something
> more minimalist.  The requirements is: susbtitutable.  The problem is
> transferred to the first arrow, no?
>
> How can we be sure that A and B points to the same D?
>
> Other said, A lives in the past.  Later, we changed D becoming D*
> because some bugs.  The other revision B lives after this change.
> Therefore, we have the same picture:
>
>   A -> D
>   B -> D*
>
> But then how do we go to C since D and D* does not have a kind of
> idempotent property.
>
> From my understanding, the current situation/process is very similar
> with the one for compiling compiler: you have your new Source of
> compiler and an OldCompiler binary, so you run:
>
>   OldCompiler(Source) -> Compiler
>   and again Compiler(Source) -> NewCompiler
>
> Somehow, whatever the choice of some OldCompiler binary, you get the
> same NewCompiler binary (aside trusting trust attack and friends,
> obviously ;-))

You're correct. This solution wouldn't be sufficient to avoid "Computing
Guix Derivation" for every possible A or B. To my understanding it could
reduce the frequency this occurs.

(To change letters yet again, I'll use Z to refer to the output Guix.)

As a package (either "guix" or a minimalist subset), D, D*, D**, etc.
should all be substitutable. If we consider input Guix's A, B, C, E, F,
..., our original "Compute Guix Derivation" graph would look like:

--8<---------------cut here---------------start------------->8---
     Z
    / \
   /| |\
 /| | | |\
/ | | | | \
A B C E F G
--8<---------------cut here---------------end--------------->8---

i.e. for every possible input Guix (and output Z), we'd need to store a
substitutable intermediate step. Not feasible.

By having A, B, C, ..., use a substitutable Guix package D, our graph
would look like this instead:

--8<---------------cut here---------------start------------->8---
      Z
     /| \
    / |  \
  D   D*  D**
 /|\  |\  |
A B C E F G
--8<---------------cut here---------------end--------------->8---

Assuming D changes significantly less frequently than A, B, C..., I
would think this should be something we could feasibly substitute (at
least for recent D and Z).

Perhaps this isn't viable. I'll confess to not being super familiar with
how build-self.scm operates. At a naive level, $ guix build guix doesn't
need to run "Computing Guix derivation", so if A->D doesn't need to
compute the derivation, and D->Z is substitutable for some subset of D
and Z, I'd think we can skip "Computing Guix derivation" as long as
someone isn't trying to either

a) Use an A that refers to a D********** that's since been GCd

b) Try to get a Z that hasn't been built by D, D*, D**, etc.

Does that sound correct?

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

Re: Guix pull: avoiding "Computing Guix derivation"

[Simon Tournier]

Hi Richard,

On lun., 13 mai 2024 at 20:52, Richard Sent <richard@freakingpenguin.com> wrote:

> You're correct. This solution wouldn't be sufficient to avoid "Computing
> Guix Derivation" for every possible A or B. To my understanding it could
> reduce the frequency this occurs.

[...]

> Assuming D changes significantly less frequently than A, B, C..., I
> would think this should be something we could feasibly substitute (at
> least for recent D and Z).

Well, the package ’guix’ has changed 14 times over the past year.
Therefore, this D cannot be this package ’guix’, IMHO.

--8<---------------cut here---------------start------------->8---
$ git log --format="%cd %s" --since="1 year ago" | grep 'gnu: guix: Update' 
Mon May 13 18:22:53 2024 +0200 gnu: guix: Update to 7ca9809.
Tue Mar 12 14:27:01 2024 +0100 gnu: guix: Update to 4c94b9e.
Mon Mar 11 23:14:37 2024 +0100 gnu: guix: Update to 8f4ffb3.
Sat Dec 2 15:37:44 2023 +0100 gnu: guix: Update to 1.4.0-16.aeb494322c.
Thu Nov 30 07:15:36 2023 +0100 gnu: guix: Update to 1.4.0-15.e0885fcfbb.
Thu Nov 9 10:42:55 2023 +0200 gnu: guix: Update to a60ff46.
Fri Oct 6 12:26:44 2023 +0200 gnu: guix: Update to e863274.
Thu Sep 28 11:44:08 2023 +0200 gnu: guix: Update to d0438fc.
Mon Sep 18 12:31:52 2023 +0200 Revert "gnu: guix: Update to 1.4.0-11.658de25e99."
Mon Sep 18 06:49:46 2023 +0200 gnu: guix: Update to 1.4.0-11.658de25e99.
Tue Aug 22 21:30:49 2023 +0200 gnu: guix: Update to 1.4.0-10.4dfdd82210.
Tue Aug 22 11:17:52 2023 +0200 gnu: guix: Update to 30355c1.
Mon Oct 2 09:28:02 2023 +0200 gnu: guix: Update to 1.4.0-12.b9fae146d6.
Mon Aug 21 18:44:49 2023 +0200 gnu: guix: Update to 0e6215a.
Fri Jun 9 22:11:14 2023 +0200 gnu: guix: Update to 44bbfc2.
--8<---------------cut here---------------end--------------->8---

Maybe I have a bad practise but here my “guix pull” history:

--8<---------------cut here---------------start------------->8---
$ guix pull -l | grep Generation
Generation 1	nov. 17 2023 13:18:58
Generation 2	déc. 11 2023 10:55:51
Generation 3	févr. 02 2024 01:56:52
Generation 4	mars 25 2024 18:22:25
Generation 5	mai 13 2024 19:28:31	(current)
--8<---------------cut here---------------end--------------->8---

Therefore, I am not convinced that replacing "Computing Guix derivation"
(build-aux/build-self.scm) by the package ’guix’ would be robust enough.

(Assuming another package ’guix’, lighter e.g., without requiring the
test suite, etc.)

All that said, any experiment – even if it appears at first clunky – is
very welcome!  This part will be improved only if there is a collective
effort / discussion / try, IMHO, i.e., by challenging the status quo. :-)

Cheers,
simon

Re: Guix pull: avoiding "Computing Guix derivation"

[Ludovic Courtès]

Hi,

Edouard Klein <edou@rdklein.fr> skribis:

> - Why is this step not substitutable ? The inputs are known, a hash can
> be derived, a substitute server could be queried for an output of that
> hash ? What am I missing ? Does the guix derivation not end up in the
> store ? What makes it so special that it can't be served by a substitute
> server ?

It’s not substitutable because it’s not a derivation.  It’s not a
derivation because it needs to access the store to “compute the Guix
derivation”.

The latter could be addressed with “nested” or “recursive” derivations
as Nix calls them, but it’s quite a beast.

Anther option would be to have a built-in derivation builder that would
do that process on behalf of users.  That would turn the process into a
derivation, which is thus cachable, substitutable, etc.

> - Is there a way (even a very dirty one, like hand copying stuff accross
>   /var/guix/profiles/per-user/*/current-guix) I can stop paying this 4
>   minutes per user price ? As I said, this is downtime on my server, as
>   I need to stop all other services to let guix pull finish.

I don’t know of any quick hack for that.

Thanks,
Ludo’.

Re: Guix pull: avoiding "Computing Guix derivation"

[Richard Sent]

>> - Why is this step not substitutable ? The inputs are known, a hash can
>> be derived, a substitute server could be queried for an output of that
>> hash ? What am I missing ? Does the guix derivation not end up in the
>> store ? What makes it so special that it can't be served by a substitute
>> server ?
>
>It’s not substitutable because it’s not a derivation.  It’s not a
>derivation because it needs to access the store to “compute the Guix
>derivation”.

I see that build in build-self.scm isn't a derivation, but most of the time spent in build is, to my knowledge, while the daemon is building the build-program derivation. build-program in build-self.scm returns a gexp->script file-like object and I see various /gnu/store/*-compute-guix-derivation.drv files in my store.

Ergo, if more of the inputs to build-program were locked, the compute-guix-derivation output should be substitutable. To my knowledge the problem largely lies in that the modules imported in build-self.scm are from the current Guix, which vary wildly between machines and is rarely, if ever, substitutable. In theory if those modules were pinned to particular versions, the bulk of the time spent in build would be removed.

Obviously there are challenges with this approach. For one, if we were to use the "guix" package to run build and generate compute-guix-derivation.drv, what would we do if a user on Guix A wanted to upgrade/downgrade to B, but the Guix package was changed in-between?

I think the various problems are solvable, perhaps by making the solution opt-in. e.g. $ guix pull --quick, which attempts to build channels using an infrequently updated Guix/Guix-subset to more regularly match an existing compute-guix-derivation in a substitute server's store.

Sorry if the formatting on this is off, sent from my phone.

Re: Guix pull: avoiding "Computing Guix derivation"

[Edouard Klein]

Hi,


Good advice, thank you ! I just began playing with btrfs, and I see that
you can convert an existing ext4 to a btrfs, so I may try that (I'll
just run some backups first).

I was not aware that guix deploy does the computation on the local
computer rather than on the target. I'll look more into it, thanks !

As for the user profiles, those exist because the users are real human
users. The server in question is https://the-dam.org, a GNU Guix public
access server.

I'll keep this thread posted on how it went :)

Cheers,

Edouard.

Ryan Sundberg <ryan@arctype.co> writes:

> Hi Edouard, I have couple of ideas for you, which may help with your deployment process:
>
> 1) if possible, mount /gnu on a btrfs filesystem with zstd compression enabled. This will let you fill up at least 3x of your physical disk space because the store compresses very well.
>
> 2) for production servers you should try using `guix deploy` to push rather than pull profiles. This will offload most of the process to the machine pushing the deployment.
>
> With regards to multi user profiles, if this is a typical server (not a
> terminal/shell server) I wonder why you need to have per user profiles at all. I
> only say this because I am not sure about running `guix deploy` on an individual
> user basis.
>
> Sincerely,
> Ryan Sundberg
>
> May 13, 2024 8:08:35 AM Edouard Klein <edou@rdklein.fr>:
>
>> Hi Guix,
>>
>> First, I'd like to apologize for not having taken the time to answer
>> those who helped me on a previous guix performance issue (with
>> containers), the reason is tied to the topic of this email: the store
>> has eaten all the space of my server, and solving that takes precedence
>> over everything else, because no space == no services.
>>
>> So, I need to clear some space, and to do that I need to have every user
>> run guix pull (and by that I mean root will sudo -u $user guix pull),
>> then update all of their profiles, and then guix gc
>> --delete-generations.
>>
>> This ought to turn deduplication up to 11, and enjoy a reduced store
>> size.
>>
>> I've already solved the cache size problem:
>> mount -t overlay overlay -o \
>> lowerdir="/root/.cache/guix",upperdir="/home/$user/.cache/guix-overlay",workdir="/home/$user/.cache/guix-workdir" \
>> "/home/$user/.cache/guix"
>>
>> Then
>> bindfs --mirror=$user /home/$user/.cache/guix /home/$user/.cache/guix
>>
>> This lets root (who just ran guix pull) share its cache with every user,
>> and avoid blasting away 700MB of disk space in every $HOME to reproduce
>> the cache.
>>
>> However, now, I'm facing the previously addressed problem of guix pull
>> being slow and hungry:
>> https://www.mail-archive.com/guix-devel@gnu.org/msg66442.html (Guix pull speed)
>> https://yhetil.org/guix/87h6mwf4u3.fsf@lapenas.dev/T/ (guix pull
>> performance)
>>
>> On my server, in order to run guix pull, I have to stop all other
>> services otherwise I run out of ram.
>>
>> Then, one root has pulled, I need to wait 4 minutes /per user/ for guix
>> pull to finish its "Computing Guix derivation" step.
>>
>> I would like to know two things, one is for the sake of knowledge, and
>> the other is to solve the problem at hand:
>>
>> - Why is this step not substitutable ? The inputs are known, a hash can
>> be derived, a substitute server could be queried for an output of that
>> hash ? What am I missing ? Does the guix derivation not end up in the
>> store ? What makes it so special that it can't be served by a substitute
>> server ?
>>
>> - Is there a way (even a very dirty one, like hand copying stuff accross
>>   /var/guix/profiles/per-user/*/current-guix) I can stop paying this 4
>>   minutes per user price ? As I said, this is downtime on my server, as
>>   I need to stop all other services to let guix pull finish.
>>
>> Thanks in advance.
>>
>> Sorry for beating a dead horse, it's just that I can't scale anything up
>> until I solve these performance issues. Sure I could rent a bigger
>> server, but that's just kicking the can down the road.
>>
>>
>> Cheers,
>>
>> Edouard.

Re: Guix pull: avoiding "Computing Guix derivation"

[Edouard Klein]

Thank you both for this detailed explanations.

I think I have a better grasp at the problem. Keep in mind that I never
not dive into guix's internals. I just

- had previous experience with
Gentoo and Source Mage (anyone remember this distro ?) who did some
interesting steps towards reproductibility back in the day
- knew what a Merkle dag is
- read the manual

and dived in with an overconfidence in my understanding of how this all
works.

To simplify the problem somewhat, bear in mind that a very good first
step would be to avoid the "Computing Guix derivation" step, once it's
been done already once and its result is "nothing to do". If that could
happen, I could just run it as root once (4 minutes of downtime a week
is manageable) and let the other users enjoy my cache.




Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi Richard,
>
> On lun., 13 mai 2024 at 20:52, Richard Sent <richard@freakingpenguin.com> wrote:
>
>> You're correct. This solution wouldn't be sufficient to avoid "Computing
>> Guix Derivation" for every possible A or B. To my understanding it could
>> reduce the frequency this occurs.
>
> [...]
>
>> Assuming D changes significantly less frequently than A, B, C..., I
>> would think this should be something we could feasibly substitute (at
>> least for recent D and Z).
>
> Well, the package ’guix’ has changed 14 times over the past year.
> Therefore, this D cannot be this package ’guix’, IMHO.
>
> $ git log --format="%cd %s" --since="1 year ago" | grep 'gnu: guix: Update'
> Mon May 13 18:22:53 2024 +0200 gnu: guix: Update to 7ca9809.
> Tue Mar 12 14:27:01 2024 +0100 gnu: guix: Update to 4c94b9e.
> Mon Mar 11 23:14:37 2024 +0100 gnu: guix: Update to 8f4ffb3.
> Sat Dec 2 15:37:44 2023 +0100 gnu: guix: Update to 1.4.0-16.aeb494322c.
> Thu Nov 30 07:15:36 2023 +0100 gnu: guix: Update to 1.4.0-15.e0885fcfbb.
> Thu Nov 9 10:42:55 2023 +0200 gnu: guix: Update to a60ff46.
> Fri Oct 6 12:26:44 2023 +0200 gnu: guix: Update to e863274.
> Thu Sep 28 11:44:08 2023 +0200 gnu: guix: Update to d0438fc.
> Mon Sep 18 12:31:52 2023 +0200 Revert "gnu: guix: Update to 1.4.0-11.658de25e99."
> Mon Sep 18 06:49:46 2023 +0200 gnu: guix: Update to 1.4.0-11.658de25e99.
> Tue Aug 22 21:30:49 2023 +0200 gnu: guix: Update to 1.4.0-10.4dfdd82210.
> Tue Aug 22 11:17:52 2023 +0200 gnu: guix: Update to 30355c1.
> Mon Oct 2 09:28:02 2023 +0200 gnu: guix: Update to 1.4.0-12.b9fae146d6.
> Mon Aug 21 18:44:49 2023 +0200 gnu: guix: Update to 0e6215a.
> Fri Jun 9 22:11:14 2023 +0200 gnu: guix: Update to 44bbfc2.
>
>
> Maybe I have a bad practise but here my “guix pull” history:
>
> $ guix pull -l | grep Generation
> Generation 1	nov. 17 2023 13:18:58
> Generation 2	déc. 11 2023 10:55:51
> Generation 3	févr. 02 2024 01:56:52
> Generation 4	mars 25 2024 18:22:25
> Generation 5	mai 13 2024 19:28:31	(current)
>
> Therefore, I am not convinced that replacing "Computing Guix derivation"
> (build-aux/build-self.scm) by the package ’guix’ would be robust enough.
>
> (Assuming another package ’guix’, lighter e.g., without requiring the
> test suite, etc.)
>
> All that said, any experiment – even if it appears at first clunky – is
> very welcome!  This part will be improved only if there is a collective
> effort / discussion / try, IMHO, i.e., by challenging the status quo. :-)
>
> Cheers,
> simon

Re: Guix pull: avoiding "Computing Guix derivation"

[Edouard Klein]

Hi !

I may be a bit dense, but there's one part I don't understand:

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Edouard Klein <edou@rdklein.fr> skribis:
>
>> - Why is this step not substitutable ? The inputs are known, a hash can
>> be derived, a substitute server could be queried for an output of that
>> hash ? What am I missing ? Does the guix derivation not end up in the
>> store ? What makes it so special that it can't be served by a substitute
>> server ?
>
> It’s not substitutable because it’s not a derivation.  It’s not a
> derivation because it needs to access the store to “compute the Guix
> derivation”.
>

Doesn't it have hashable inputs ? Like the different git hashes of the
different channels, plus the hash of whatever guix+guile revision is actually
doing the work ?

I think my misunderstanding boils down to a bad mental model of how guix
works.

What I imagine the "Computing guix derivation" step to be is no
different than building a package whose inputs are the channels'
sources, guile, and the current guix, and whose output is the new guix.

Simon and Richard pointed out that the combinatorial combination of all
possible input combination makes this hard to substitute, but I don't
understand how its result, once in the store, can't lead to a simple
linking from the profile to the store, instead of re-running the
"Computing guix derivation" step when guix pull is called with the same
inputs.

Surely my mental model is wrong somehow ?





> The latter could be addressed with “nested” or “recursive” derivations
> as Nix calls them, but it’s quite a beast.
>
> Anther option would be to have a built-in derivation builder that would
> do that process on behalf of users.  That would turn the process into a
> derivation, which is thus cachable, substitutable, etc.
>
>> - Is there a way (even a very dirty one, like hand copying stuff accross
>>   /var/guix/profiles/per-user/*/current-guix) I can stop paying this 4
>>   minutes per user price ? As I said, this is downtime on my server, as
>>   I need to stop all other services to let guix pull finish.
>
> I don’t know of any quick hack for that.
>
> Thanks,
> Ludo’.

Re: Guix pull: avoiding "Computing Guix derivation"

[Richard Sent]

> I see that build in build-self.scm isn't a derivation, but most of the time spent in build is, to my knowledge, while the daemon is building the build-program derivation. build-program in build-self.scm returns a gexp->script file-like object and I see various /gnu/store *-compute-guix-derivation.drv files in my store.

Sorry, saying "most of the time is spent on building the build-program derivation" is not correct. Most of the time is spent on running the compute-guix-derivation script. D'oh.

I'll spend more time trying to understand the current system.

Re: Guix pull: avoiding "Computing Guix derivation"

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 1021 bytes --]

Hi everyone,

Edouard Klein <edou@rdklein.fr> writes:

> I think my misunderstanding boils down to a bad mental model of how guix
> works.
>
> What I imagine the "Computing guix derivation" step to be is no
> different than building a package whose inputs are the channels'
> sources, guile, and the current guix, and whose output is the new guix.
>
> Simon and Richard pointed out that the combinatorial combination of all
> possible input combination makes this hard to substitute, but I don't
> understand how its result, once in the store, can't lead to a simple
> linking from the profile to the store, instead of re-running the
> "Computing guix derivation" step when guix pull is called with the same
> inputs.
>
> Surely my mental model is wrong somehow ?

You might be interested in this thread [1], which tries to explain the
terrible performance of `guix pull`.

[1] mid:87wmxe3vsa.fsf@jpoiret.xyz
    https://yhetil.org/guix/87wmxe3vsa.fsf@jpoiret.xyz/

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Guix pull: avoiding "Computing Guix derivation"

[Edouard Klein]

Hi,


Josselin Poiret <dev@jpoiret.xyz> writes:

> [[PGP Signed Part:Undecided]]
> Hi everyone,
>
> Edouard Klein <edou@rdklein.fr> writes:
>
>> I think my misunderstanding boils down to a bad mental model of how guix
>> works.
>>
>> What I imagine the "Computing guix derivation" step to be is no
>> different than building a package whose inputs are the channels'
>> sources, guile, and the current guix, and whose output is the new guix.
>>
>> Simon and Richard pointed out that the combinatorial combination of all
>> possible input combination makes this hard to substitute, but I don't
>> understand how its result, once in the store, can't lead to a simple
>> linking from the profile to the store, instead of re-running the
>> "Computing guix derivation" step when guix pull is called with the same
>> inputs.
>>
>> Surely my mental model is wrong somehow ?
>
> You might be interested in this thread [1], which tries to explain the
> terrible performance of `guix pull`.
>
> [1] mid:87wmxe3vsa.fsf@jpoiret.xyz
>     https://yhetil.org/guix/87wmxe3vsa.fsf@jpoiret.xyz/
>

I reread it, but I still don't understand why this step is not
substitutable. It has a fixed set of input and it is a deterministic
process. Unless I don't understand what is going on, which is probable,
but I can't seem to find where I'm mistaken.



> Best,