From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([209.51.188.92]:55036) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gihxa-0004mb-Rp for guix-patches@gnu.org; Sun, 13 Jan 2019 10:46:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gihxZ-0003c6-V3 for guix-patches@gnu.org; Sun, 13 Jan 2019 10:46:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:58703) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gihxZ-0003c0-Qk for guix-patches@gnu.org; Sun, 13 Jan 2019 10:46:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1gihxZ-0008Ht-OD for guix-patches@gnu.org; Sun, 13 Jan 2019 10:46:01 -0500 Subject: [bug#34060] [PATCH 00/10] Add a cache for package lookups Resent-Message-ID: Received: from eggs.gnu.org ([209.51.188.92]:54962) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gihxL-0004ft-Tf for guix-patches@gnu.org; Sun, 13 Jan 2019 10:45:48 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gihxL-0003Ti-2A for guix-patches@gnu.org; Sun, 13 Jan 2019 10:45:47 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Sun, 13 Jan 2019 16:45:32 +0100 Message-Id: <20190113154532.29606-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: 34060@debbugs.gnu.org Hello! This patch adjust =E2=80=98guix pull=E2=80=99 & co. such that Guix comes = with a cache to speed up package lookups. The end result is that: guix build coreutils becomes equivalent to: guix build -e '(@ (gnu packages base) coreutils)' That means we have fewer files to open (only the closure of (gnu packages base) instead of all the (gnu packages =E2=80=A6) modules), and thus less I/O and a smaller memory footprint. Without cache, we have: --8<---------------cut here---------------start------------->8--- $ GUIX_PROFILING=3Dgc time ./pre-inst-env guix build coreutils -nd /gnu/store/ddlgwpishzb8985gwg5fdrydvllg254a-coreutils-8.30.drv Garbage collection statistics: heap size: 46.18 MiB allocated: 102.57 MiB GC times: 14 time spent in GC: 0.25 seconds (25% of user time) 1.01user 0.06system 0:00.96elapsed 112%CPU (0avgtext+0avgdata 158916maxre= sident)k 0inputs+0outputs (0major+14602minor)pagefaults 0swaps --8<---------------cut here---------------end--------------->8--- With the cache, we get: --8<---------------cut here---------------start------------->8--- $ GUIX_PROFILING=3Dgc time ./foo/bin/guix build coreutils -nd /gnu/store/ddlgwpishzb8985gwg5fdrydvllg254a-coreutils-8.30.drv Garbage collection statistics: heap size: 36.52 MiB allocated: 87.30 MiB GC times: 14 time spent in GC: 0.24 seconds (27% of user time) 0.89user 0.06system 0:00.84elapsed 112%CPU (0avgtext+0avgdata 124168maxre= sident)k 0inputs+0outputs (0major+11423minor)pagefaults 0swaps --8<---------------cut here---------------end--------------->8--- As a bonus, we take advantage of the cache to speed up =E2=80=98guix package -A=E2=80=99 (which now runs in ~0.5s instead of ~1.7s) and =E2=80=98guix edit=E2=80=99. The cache is populated by packages of all the channels specified during =E2=80=98guix pull=E2=80=99. The cache is *not* used at all when GUIX_PACKAGE_PATH is set or a =E2=80=98-L=E2=80=99 flag is passed, or ./pre-inst-env is used. Feedback welcome! Ludo=E2=80=99. Ludovic Court=C3=A8s (10): profiling: Add a "gc" profiling component. guix package: Avoid 'find-newest-available-packages'. packages: Remove 'find-newest-available-packages'. inferior: Add 'gexp->derivation-in-inferior'. discovery: Add 'fold-module-public-variables*'. pull: Build profile with 'channel-instances->derivation'. channels: Compute a package cache and use it. edit: Use 'specification->location' to read information from the cache. guix package: '--list-available' can use data from the cache. status: Distinguish 'package-cache' profile hook. gnu/packages.scm | 253 ++++++++++++++++++++++++++++++++++----- guix/channels.scm | 32 ++++- guix/discovery.scm | 28 ++++- guix/inferior.scm | 26 ++++ guix/profiling.scm | 25 +++- guix/scripts/edit.scm | 29 ++--- guix/scripts/package.scm | 137 +++++++++++---------- guix/scripts/pull.scm | 4 +- guix/status.scm | 2 + tests/packages.scm | 77 ++++++++++-- 10 files changed, 491 insertions(+), 122 deletions(-) --=20 2.20.1