Guix pull speed

Guix pull speed

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 1698 bytes --]

Hi everyone,

After looking a bit more into guix pull speed, or to be more precise the
"Computing Guix derivation..." step, which is not substitutable.  I've
come to the conclusion that the thing that takes the majority of the
time is loading the files that define the packages that the new Guix
needs to build itself.  These files are not compiled yet, and worse,
loading just (gnu packages guile) ends up loading 361 other package
files.  You can generate a package graph in GraphML with `guix graph -t
module -b graphml guile`, and use e.g. networkx to analyze it.

You can compare with a compiled check-out of guix by just running the
following in a `guix repl`:
--8<---------------cut here---------------start------------->8---
(use-modules (guix self) (guix monad-repl))
,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1)
--8<---------------cut here---------------end--------------->8---
which takes at most 5 seconds on my laptop.

One idea I had was to move all the packages that are looked up in (guix
self) to their own little bubble, to avoid having to load extra stuff.
However, this is not currently possible because some of them do have
non-trivial dependency graphs.  I've identified these problematic
inputs: guile-avahi guile-ssh guile-git guile-gnutls guix-daemon (it
pulls in all other dependencies itself) po4a graphviz

What could be done about this?  Another solution would be to somehow
build Guix without any of the dependencies and then add them in later,
similar to what is done with build-aux/build-self.scm to be able to load
(guix self) in the first place.  That seems quite complex though.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Guix pull speed

[Simon Tournier]

Hi Josselin,

On Tue, 29 Aug 2023 at 13:58, Josselin Poiret <dev@jpoiret.xyz> wrote:

> After looking a bit more into guix pull speed, or to be more precise the
> "Computing Guix derivation..." step, which is not substitutable.  I've
> come to the conclusion that the thing that takes the majority of the
> time is loading the files that define the packages that the new Guix
> needs to build itself.

Well, on my machine, the bigger bottleneck seems the procedure name
’proxy’ which copies stuff around, IIUC. See [1].


>                         These files are not compiled yet, and worse,
> loading just (gnu packages guile) ends up loading 361 other package
> files.  You can generate a package graph in GraphML with `guix graph -t
> module -b graphml guile`, and use e.g. networkx to analyze it.

Hum, is this ’graphml’ something you have not submitted?  Or am I
missing a point?  Last time I played with “guix graph”, I wrote a small
script for bridging with networkx.  See [2].


> You can compare with a compiled check-out of guix by just running the
> following in a `guix repl`:
> --8<---------------cut here---------------start------------->8---
> (use-modules (guix self) (guix monad-repl))
> ,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1)
> --8<---------------cut here---------------end--------------->8---
> which takes at most 5 seconds on my laptop.

Yeah, that’s fast. :-)

For comparing, what would be the corresponding derivations that “guix
pull” is building?


> One idea I had was to move all the packages that are looked up in (guix
> self) to their own little bubble, to avoid having to load extra stuff.
> However, this is not currently possible because some of them do have
> non-trivial dependency graphs.  I've identified these problematic
> inputs: guile-avahi guile-ssh guile-git guile-gnutls guix-daemon (it
> pulls in all other dependencies itself) po4a graphviz
>
> What could be done about this?  Another solution would be to somehow
> build Guix without any of the dependencies and then add them in later,
> similar to what is done with build-aux/build-self.scm to be able to load
> (guix self) in the first place.  That seems quite complex though.

Thanks for looking at this.  Well, as reported in [1], I am missing what
is the aim of the procedure ’proxy’ and if it could be avoided.

Cheers,
simon


1: Slow guix pull
Simon Tournier <zimon.toutoune@gmail.com>
Wed, 23 Aug 2023 15:48:06 +0200
id:86o7ix3m5l.fsf@gmail.com
https://lists.gnu.org/archive/html/guix-devel/2023-08
https://yhetil.org/guix/86o7ix3m5l.fsf@gmail.com

2: Some stats about the graph of dependencies
zimoun <zimon.toutoune@gmail.com>
Fri, 09 Dec 2022 18:29:43 +0100
id:874ju4qyd4.fsf@gmail.com
https://lists.gnu.org/archive/html/guix-devel/2022-12
https://yhetil.org/guix/874ju4qyd4.fsf@gmail.com

Re: Guix pull speed

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 2002 bytes --]

Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi Josselin,
>
> On Tue, 29 Aug 2023 at 13:58, Josselin Poiret <dev@jpoiret.xyz> wrote:
>
>> After looking a bit more into guix pull speed, or to be more precise the
>> "Computing Guix derivation..." step, which is not substitutable.  I've
>> come to the conclusion that the thing that takes the majority of the
>> time is loading the files that define the packages that the new Guix
>> needs to build itself.
>
> Well, on my machine, the bigger bottleneck seems the procedure name
> ’proxy’ which copies stuff around, IIUC. See [1].

Proxy is on the helper script side, it's just waiting for the actual
build script to do its thing.

>>                         These files are not compiled yet, and worse,
>> loading just (gnu packages guile) ends up loading 361 other package
>> files.  You can generate a package graph in GraphML with `guix graph -t
>> module -b graphml guile`, and use e.g. networkx to analyze it.
>
> Hum, is this ’graphml’ something you have not submitted?  Or am I
> missing a point?  Last time I played with “guix graph”, I wrote a small
> script for bridging with networkx.  See [2].

I've committed it pretty recently, and should work OOTB now.

>> You can compare with a compiled check-out of guix by just running the
>> following in a `guix repl`:
>> --8<---------------cut here---------------start------------->8---
>> (use-modules (guix self) (guix monad-repl))
>> ,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1)
>> --8<---------------cut here---------------end--------------->8---
>> which takes at most 5 seconds on my laptop.
>
> Yeah, that’s fast. :-)
>
> For comparing, what would be the corresponding derivations that “guix
> pull” is building?

It's building the same!  Just that building the derivation takes way
longer because it has first to load a bunch of uncompiled guile files.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Guix pull speed

[Simon Tournier]

Hi Josselin,

On Wed, 06 Sep 2023 at 11:45, Josselin Poiret <dev@jpoiret.xyz> wrote:

>> Well, on my machine, the bigger bottleneck seems the procedure name
>> ’proxy’ which copies stuff around, IIUC. See [1].
>
> Proxy is on the helper script side, it's just waiting for the actual
> build script to do its thing.

Do it copy on the fly?  I mean, is it first written somewhere then moved?

>> Hum, is this ’graphml’ something you have not submitted?  Or am I
>> missing a point?  Last time I played with “guix graph”, I wrote a small
>> script for bridging with networkx.  See [2].
>
> I've committed it pretty recently, and should work OOTB now.

Cool!


>>> You can compare with a compiled check-out of guix by just running the
>>> following in a `guix repl`:
>>> --8<---------------cut here---------------start------------->8---
>>> (use-modules (guix self) (guix monad-repl))
>>> ,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1)
>>> --8<---------------cut here---------------end--------------->8---
>>> which takes at most 5 seconds on my laptop.
>>
>> Yeah, that’s fast. :-)
>>
>> For comparing, what would be the corresponding derivations that “guix
>> pull” is building?
>
> It's building the same!  Just that building the derivation takes way
> longer because it has first to load a bunch of uncompiled guile files.

Well, I am not sure to follow.  Is your point not about why it
takes longer?

If it is the same, these 5 seconds is not why “guix pull” is slow, no?


Cheers,
simon

Re: Guix pull speed

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 917 bytes --]


-- 

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Do it copy on the fly?  I mean, is it first written somewhere then moved?

It just copies the output of the building process, which shouldn't be
too big.  It's waiting for build output though, hence why it's taking so
much time.

>> It's building the same!  Just that building the derivation takes way
>> longer because it has first to load a bunch of uncompiled guile files.
>
> Well, I am not sure to follow.  Is your point not about why it
> takes longer?
>
> If it is the same, these 5 seconds is not why “guix pull” is slow, no?

It's the same process, but in one case it's done with an already
compiled tree, so loading the files is instantaneous, but on `guix pull`
the tree is not compiled yet (that's the whole point of `guix pull`!)
and so loading these 300+ guile files is extremely slow.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Guix pull speed

[Simon Tournier]

Hi Josselin,

On Thu, 7 Sept 2023 at 10:37, Josselin Poiret <dev@jpoiret.xyz> wrote:

[...]

Thanks for explaining.

Cheers,
simon

Re: Guix pull speed

[Ludovic Courtès]

Hello!

Josselin Poiret <dev@jpoiret.xyz> skribis:

> After looking a bit more into guix pull speed, or to be more precise the
> "Computing Guix derivation..." step, which is not substitutable.  I've
> come to the conclusion that the thing that takes the majority of the
> time is loading the files that define the packages that the new Guix
> needs to build itself.  These files are not compiled yet, and worse,
> loading just (gnu packages guile) ends up loading 361 other package
> files.  You can generate a package graph in GraphML with `guix graph -t
> module -b graphml guile`, and use e.g. networkx to analyze it.

Nice (we should add this trick under “Invoking guix graph” for instance;
we already mention xdot there.)

> You can compare with a compiled check-out of guix by just running the
> following in a `guix repl`:
>
> (use-modules (guix self) (guix monad-repl))
> ,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1)
>
> which takes at most 5 seconds on my laptop.
>
> One idea I had was to move all the packages that are looked up in (guix
> self) to their own little bubble, to avoid having to load extra stuff.
> However, this is not currently possible because some of them do have
> non-trivial dependency graphs.  I've identified these problematic
> inputs: guile-avahi guile-ssh guile-git guile-gnutls guix-daemon (it
> pulls in all other dependencies itself) po4a graphviz

Yeah, I don’t think the idea of an independent subset of the module
graph is achievable, because sooner or later glibc starts depending on
Python, which starts depending on Rust, which pulls in Qt.  Compiling
with ‘-Wunused-modules’ might allow us to clean up the module graph a
bit though.

> What could be done about this?  Another solution would be to somehow
> build Guix without any of the dependencies and then add them in later,
> similar to what is done with build-aux/build-self.scm to be able to load
> (guix self) in the first place.  That seems quite complex though.

Most of the time is spent evaluating (gnu packages …) modules;
evaluation is pretty slow, and from what I remember of previous
profiling sessions, this is largely due to psyntax.  We should
concentrate on that.


Another approach: the resulting .drv is a deterministic result.
Wouldn’t it be nice if we could make the whole “Computing Guix
derivation” process a derivation itself?  That way it could be
substituted.

Nix introduced (or considered introducing) “recursive Nix”, whereby a
derivation build process can talk to the daemon “from the inside”,
creating additional derivations on the way.  This could be one
possibility, but it’s a lot of work with unclear outcomes.

In Guix, we can use the (guix …) modules inside the build process, and
we have almost everything we need to compute a set of derivations “in
the abstract” (we’d need a <store-connection> backend that, instead of
making RPCs, would store .drv and “sources” in memory or in a file).  So
we could create a derivation that returns let’s say a nar containing the
closure of the .drv (the “Guix derivation”), which we would then import.

There’s “just” one gotcha with this plan: grafts.  To compute grafts, we
potentially need to build things, and for that we’d need to actually
talk to the daemon.


Maybe there are other possibilities, such as adding a
“builtin:build-self” derivation builder in the daemon.

Food for thought!

Ludo’.

Re: Guix pull speed

[Simon Tournier]

Hi,

On Thu, 14 Sep 2023 at 11:58, Ludovic Courtès <ludo@gnu.org> wrote:

>> What could be done about this?  Another solution would be to somehow
>> build Guix without any of the dependencies and then add them in later,
>> similar to what is done with build-aux/build-self.scm to be able to load
>> (guix self) in the first place.  That seems quite complex though.
>
> Most of the time is spent evaluating (gnu packages …) modules;
> evaluation is pretty slow, and from what I remember of previous
> profiling sessions, this is largely due to psyntax.  We should
> concentrate on that.

On my machine the step “Computing Guix derivation” is single-threaded.
Is it not possible to exploit multi-thread here?

Cheers,
simon