From: ludo@gnu.org (Ludovic Courtès)
To: Pjotr Prins <pjotr.public12@thebird.nl>
Cc: guix-guix-devel <guix-devel@gnu.org>
Subject: Re: Can we speed it up? Prev: compiling guix is too slow?
Date: Mon, 05 Feb 2018 11:15:04 +0100 [thread overview]
Message-ID: <87h8qvivyv.fsf@gnu.org> (raw)
In-Reply-To: <20180205073410.GA9752@thebird.nl> (Pjotr Prins's message of "Mon, 5 Feb 2018 08:34:10 +0100")
Pjotr Prins <pjotr.public12@thebird.nl> skribis:
> What will it be like with 15K packages? We will get there. We can
> actually try it now by doubling the package tree - anyone wants to try
> and create a simulation? I.e., not just double the tree, make sure
> there are cross references between the two graphs by modifying some of
> the inputs at random.
>
> This leads to the following thought: why don't we create a 'lazy'
> build. There are multiple ways to go about it (I think). One would be
> to parse scheme files for package names and only compile those that
> are needed when someone invokes a guix command (and have not been
> compiled yet). Or generate a meta list for a source tree.
Note that there are several things we could do:
• Not compile gnu/packages/*.scm at all and instead turn on Guile’s
auto-compilation. As things are currently, the first ‘guix package’
invocation would take ages though.
• Never compile gnu/packages/*.scm and instead interpret it, though
that’s currently relatively slow and probably more memory-consuming.
I think with ‘wip-pull-reload’ (I’ll resume work on it, I promise!)
things should already be nicer, and then, we should keep improving the
compiler (Andy already significantly improved CPU consumption in Guile
2.2.3).
> Or subcategorize packages so only those packages get included that are
> asked for (assuming there are no deeper dependencies). For example,
> few people need the bioinformatics packages. We could have the sub
> section of the graph split out and have people do:
>
> guix package --topic=bio -i samtools
>
> for example and compile the contents gnu/packages/bio/ directory when
> that happens the first time for a specific checkout.
I think that should be the last resort, but yeah.
Another thing we could do to speed up lookup-by-name is to maintain a
cache that maps package names to modules (currently we always traverse
all the package modules with ‘fold-packages’.)
> I think scalability is a good goal and instant compilation another ;).
> A few years back it just took 30 seconds to build Guix.
Yep, scalability is this year’s challenge!
Ludo’.
prev parent reply other threads:[~2018-02-05 10:15 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-06-28 22:58 compiling guix is too slow? Feng Shu
2017-06-29 1:23 ` Adonay Felipe Nogueira
2017-06-29 2:56 ` Leo Famulari
2017-07-01 13:33 ` Ludovic Courtès
2017-07-03 7:50 ` Andy Wingo
2017-07-03 8:06 ` ng0
2017-07-03 10:19 ` Ludovic Courtès
2017-07-03 12:15 ` ng0
2017-07-04 9:17 ` Ricardo Wurmus
2017-07-04 9:23 ` ng0
2017-07-03 10:18 ` Ludovic Courtès
2017-06-29 4:28 ` Alex Vong
2017-07-02 20:38 ` Alex Kost
2017-07-04 8:16 ` Alex Vong
2017-07-05 7:35 ` Alex Kost
2017-07-05 14:39 ` Alex Vong
2017-07-06 3:49 ` Chris Marusich
2018-02-05 7:34 ` Can we speed it up? Prev: " Pjotr Prins
2018-02-05 8:15 ` Konrad Hinsen
2018-02-05 8:55 ` Pjotr Prins
2018-02-05 9:56 ` Konrad Hinsen
2018-02-05 10:12 ` Pjotr Prins
2018-02-05 10:15 ` Ludovic Courtès [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87h8qvivyv.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=guix-devel@gnu.org \
--cc=pjotr.public12@thebird.nl \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.