Re: Can we speed it up? Prev: compiling guix is too slow?

[ludo@gnu.org (Ludovic Courtès)]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> What will it be like with 15K packages? We will get there. We can
> actually try it now by doubling the package tree - anyone wants to try
> and create a simulation? I.e., not just double the tree, make sure
> there are cross references between the two graphs by modifying some of
> the inputs at random.
>
> This leads to the following thought: why don't we create a 'lazy'
> build. There are multiple ways to go about it (I think). One would be
> to parse scheme files for package names and only compile those that
> are needed when someone invokes a guix command (and have not been
> compiled yet). Or generate a meta list for a source tree.

Note that there are several things we could do:

  • Not compile gnu/packages/*.scm at all and instead turn on Guile’s
    auto-compilation.  As things are currently, the first ‘guix package’
    invocation would take ages though.

  • Never compile gnu/packages/*.scm and instead interpret it, though
    that’s currently relatively slow and probably more memory-consuming.

I think with ‘wip-pull-reload’ (I’ll resume work on it, I promise!)
things should already be nicer, and then, we should keep improving the
compiler (Andy already significantly improved CPU consumption in Guile
2.2.3).

> Or subcategorize packages so only those packages get included that are
> asked for (assuming there are no deeper dependencies). For example,
> few people need the bioinformatics packages. We could have the sub
> section of the graph split out and have people do:
>
>   guix package --topic=bio -i samtools
>
> for example and compile the contents gnu/packages/bio/ directory when
> that happens the first time for a specific checkout.

I think that should be the last resort, but yeah.

Another thing we could do to speed up lookup-by-name is to maintain a
cache that maps package names to modules (currently we always traverse
all the package modules with ‘fold-packages’.)

> I think scalability is a good goal and instant compilation another ;).
> A few years back it just took 30 seconds to build Guix.

Yep, scalability is this year’s challenge!

Ludo’.