From: zimoun <zimon.toutoune@gmail.com>
To: 39258@debbugs.gnu.org
Cc: ludo@gnu.org, arunisaac@systemreboot.net,
zimoun <zimon.toutoune@gmail.com>
Subject: [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances
Date: Thu, 15 Jul 2021 09:33:26 +0200 [thread overview]
Message-ID: <20210715073328.212123-1-zimon.toutoune@gmail.com> (raw)
In-Reply-To: <cu7pnfaar36.fsf@systemreboot.net>
Hi,
This is an attempt to improve the performance of "guix search". It is still
half baked but it allows to discuss further the idea about expanding the
current '/lib/guix/package.cache' and avoids to forget an IRL discussion. ;-)
Let start by what needs to be improved: the part when cache is not
authoritative. It is slower than the current approach because the package is
read twice, i.e., the module is indeed loaded twice, once by
'fold-available-packages' via 'fold-module-public-variables*' and then again
by 'find-packages-by-description' via 'read-package-from'. The issue is to
have a common interface for both cases (cache and no-cache). More thoughts
are required. ;-)
Then, using the cache is slower than expected. Therefore, something is maybe
twisted -- quick implementation before holidays ;-) -- with the use of
'fold-avaibale-packages' as proposed by Ludo [1]. Note that instead another
'fold-packages' (say 'fold-packages*') using the new cache should be used. As
it is done with v4 and the performances were as expected:
<http://issues.guix.gnu.org/39258#89>
1: <http://issues.guix.gnu.org/39258#93>
From my understanding, the issue that 'package-relevance' accepts a 'package'
(and then all the chain until displaying) and 'fold-avaibale-packages' does
not return a package. Well, I do not know; especially where to put something
similiar to 'read-package-from'.
To test, after applying the patches, the command is:
./pre-inst-env guix pull --allow-downgrades --disable-authentication \
--url=$(pwd) --branch=search-v6 -p /tmp/new
Let compare only for cold cache and time this cache building (Guix 7db8fd6):
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix build --check $(guix gc --derivers $(readlink -f ~/.config/guix/current/lib/guix/package.cache))
real 0m28,848s
user 0m1,481s
sys 0m0,252s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix build --check $(guix gc --derivers $(readlink -f /tmp/new/lib/guix/package.cache))
real 0m40,279s
user 0m1,582s
sys 0m0,232s
It seems longer but compared to the time of "guix pull" completion, it seems
acceptable. However, maybe it could become an issue when running a lot of
"guix time-machine"... Well, hard trade-off. ;-)
Let compare for some queries:
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search game | recsel -C -P name | wc -l
371
real 0m7,561s
user 0m3,525s
sys 0m0,391s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search game | recsel -C -P name | wc -l
371
real 0m9,814s
user 0m3,240s
sys 0m0,363s
--8<---------------cut here---------------end--------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search strategy game | recsel -C -P name | wc -l
16
real 0m8,565s
user 0m2,803s
sys 0m0,430s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search strategy game | recsel -C -P name | wc -l
16
real 0m9,679s
user 0m2,370s
sys 0m0,334s
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search strategy game caesar | recsel -C -P name | wc -l
0
real 0m8,307s
user 0m2,388s
sys 0m0,366s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search strategy game caesar | recsel -C -P name | wc -l
0
real 0m3,626s
user 0m0,948s
sys 0m0,101s
--8<---------------cut here---------------end--------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search game strategy the | recsel -C -P name | wc -l
4
real 0m8,776s
user 0m2,903s
sys 0m0,454s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search game strategy the | recsel -C -P name | wc -l
4
real 0m9,495s
user 0m2,546s
sys 0m0,313s
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search the game strategy | recsel -C -P name | wc -l
4
real 0m8,502s
user 0m2,534s
sys 0m0,388s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search the game strategy | recsel -C -P name | wc -l
4
real 0m9,508s
user 0m2,254s
sys 0m0,363s
--8<---------------cut here---------------end--------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search crypto library | recsel -C -P name | grep libb2
libb2
real 0m8,744s
user 0m2,875s
sys 0m0,374s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search crypto library | recsel -C -P name | grep libb2
libb2
real 0m9,229s
user 0m2,448s
sys 0m0,397s
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search cuirass integration | recsel -C -P name | wc -l
1
real 0m8,132s
user 0m2,343s
sys 0m0,407s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search cuirass integration | recsel -C -P name | wc -l
1
real 0m8,940s
user 0m2,036s
sys 0m0,369s
--8<---------------cut here---------------end--------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search cuirass | recsel -C -P name | wc -l
2
real 0m8,240s
user 0m2,461s
sys 0m0,367s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search cuirass | recsel -C -P name | wc -l
2
real 0m8,863s
user 0m2,019s
sys 0m0,377s
--8<---------------cut here---------------start------------->8---
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time guix search cuirass integration foo | recsel -C -P name | wc -l
0
real 0m8,258s
user 0m2,418s
sys 0m0,521s
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search cuirass integration foo | recsel -C -P name | wc -l
0
real 0m3,358s
user 0m0,867s
sys 0m0,139s
--8<---------------cut here---------------end--------------->8---
This last example suggests that 'read-package-from' is the slowdown.
(On a side note, maybe I am doing wrong, but there is no improvement by the
recent introduction of 'cut' for multi-terms as the query "the game strategy"
and "game strategy the". Another story. :-))
When cache is not authoritative, it is worse, as expected:
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
time /tmp/new/bin/guix search -L /tmp/my-pkgs cuirass integration foo | recsel -C -P name | wc -l
0
real 0m12,503s
user 0m7,807s
sys 0m0,529s
and note that currently the performances of "guix search" is the same for both
cases (authoritative and not authoritative); i.e., see previous timing.
Last, two minor remarks about previous comments.
1. Ludo commented:
> Therefore the cache '/lib/guix/package.cache' contains more
> information.
This breaks the binary interface, so we’ll have to analyze the impact of
such a change and devise a strategy.
<http://issues.guix.gnu.org/39258#93>
and after some checking, this should be fine, IIUC. The '--news' is ok
because of '#:allow-other-keys'. And other parts are modified accordingly.
Guix revision N creates a cache that Guix revision N+1 will read but it should
not be an issue; see 'inferior-available-packages'.
2. And Ludo wrote:
I realize the other cache also has that problem, but it would be nice to add a
version tag to the cache. Basically emit something like:
(package-metadata-cache (version 0) VECTOR …)
instead of just:
(VECTOR …)
<http://issues.guix.gnu.org/39258#93>
which is, after discussions, not necessary. Versioning does not make sense
here because the cache is read by the Guix which generates it. Therefore,
specify a version is extraneous here.
Comments are welcome for this work-in-progress. ;-)
Cheers,
simon
zimoun (2): DRAFT packages: Add fields to packages cache. DRAFT scripts:
package: Use cache in 'find-packages-by-description'.
gnu/packages.scm | 52 +++++++++++++++++++++++++++-------------
guix/scripts/package.scm | 46 +++++++++++++++++++++++++----------
2 files changed, 70 insertions(+), 28 deletions(-)
base-commit: 4196087f3d6fc254db5b4c47658e5679c835516f
--
2.32.0
next prev parent reply other threads:[~2021-07-15 7:34 UTC|newest]
Thread overview: 126+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-23 19:51 [bug#39258] Faster guix search using an sqlite cache Arun Isaac
2020-01-29 23:33 ` zimoun
2020-01-30 13:48 ` Arun Isaac
2020-01-31 12:48 ` zimoun
2020-02-02 21:16 ` Arun Isaac
2020-02-04 10:19 ` zimoun
2020-02-06 1:58 ` Arun Isaac
2020-02-11 16:29 ` Ludovic Courtès
2020-02-11 18:21 ` zimoun
2020-02-11 18:39 ` Ludovic Courtès
2020-02-11 19:07 ` Arun Isaac
2020-02-11 20:20 ` zimoun
2020-02-15 14:50 ` Arun Isaac
2020-02-11 20:13 ` zimoun
2020-02-27 20:41 ` [bug#39258] [PATCH 0/4] Xapian for Guix package search Arun Isaac
2020-02-27 20:41 ` [bug#39258] [PATCH 1/4] gnu: Add guile-xapian Arun Isaac
2020-03-03 16:29 ` zimoun
2020-02-27 20:41 ` [bug#39258] [PATCH 2/4] build-self: Add guile-xapian to Guix dependencies Arun Isaac
2020-02-27 20:41 ` [bug#39258] [PATCH 3/4] gnu: Generate xapian package search index Arun Isaac
2020-02-28 8:04 ` Pierre Neidhardt
2020-03-05 20:26 ` Arun Isaac
2020-03-03 18:29 ` zimoun
2020-02-27 20:41 ` [bug#39258] [PATCH 4/4] gnu: Use xapian index for package search Arun Isaac
2020-02-28 8:11 ` Pierre Neidhardt
2020-03-03 19:21 ` zimoun
2020-03-03 19:51 ` zimoun
2020-02-28 8:13 ` [bug#39258] [PATCH 0/4] Xapian for Guix " Pierre Neidhardt
2020-02-28 12:39 ` zimoun
2020-02-28 12:49 ` Pierre Neidhardt
2020-02-28 15:36 ` Arun Isaac
2020-02-28 16:04 ` Arun Isaac
2020-03-02 18:37 ` zimoun
2020-03-02 19:13 ` zimoun
2020-03-03 20:04 ` zimoun
2020-02-29 8:25 ` Arun Isaac
2020-03-02 18:27 ` zimoun
2020-02-28 12:36 ` zimoun
2020-03-05 16:46 ` Ludovic Courtès
2020-03-07 13:31 ` [bug#39258] [PATCH v2 0/3] " Arun Isaac
2020-03-07 13:31 ` [bug#39258] [PATCH v2 1/3] build-self: Add guile-xapian to Guix dependencies Arun Isaac
2020-03-09 18:14 ` zimoun
2020-03-09 23:40 ` Jonathan Brielmaier
2020-03-10 5:24 ` Arun Isaac
2020-03-07 13:31 ` [bug#39258] [PATCH v2 2/3] gnu: Generate Xapian package search index Arun Isaac
2020-03-09 18:19 ` zimoun
2020-03-07 13:31 ` [bug#39258] [PATCH v2 3/3] gnu: Use Xapian index for package search Arun Isaac
2020-03-07 20:33 ` [bug#39258] [PATCH v2 0/3] Xapian for Guix " Ludovic Courtès
2020-03-08 9:01 ` Arun Isaac
2020-03-08 11:33 ` Ludovic Courtès
2020-03-08 20:27 ` Arun Isaac
2020-03-09 7:42 ` Pierre Neidhardt
2020-03-09 12:50 ` zimoun
2020-03-09 10:35 ` Ludovic Courtès
2020-03-10 14:17 ` Arun Isaac
2020-03-10 14:33 ` zimoun
2020-03-11 13:50 ` Ludovic Courtès
2020-03-13 5:37 ` Arun Isaac
2020-03-15 20:40 ` Ludovic Courtès
2020-03-09 7:50 ` Pierre Neidhardt
2020-03-09 10:28 ` Ludovic Courtès
2020-03-09 13:03 ` zimoun
2020-03-09 12:53 ` zimoun
2020-03-09 12:47 ` zimoun
2020-03-09 12:40 ` zimoun
2020-03-09 12:34 ` zimoun
2020-03-08 20:27 ` zimoun
2020-03-08 20:40 ` Arun Isaac
2020-03-09 12:28 ` zimoun
2020-03-27 16:26 ` [bug#39258] [PATCH v3 0/3] Package metadata cache for guix search Arun Isaac
2020-03-27 16:26 ` [bug#39258] [PATCH v3 1/3] guix: Generate package metadata cache Arun Isaac
2020-04-24 20:48 ` Ludovic Courtès
2020-04-26 9:48 ` zimoun
2020-04-26 14:35 ` Ludovic Courtès
2020-04-26 14:54 ` Pierre Neidhardt
2020-04-26 15:33 ` Ludovic Courtès
2020-04-26 15:05 ` zimoun
2020-03-27 16:26 ` [bug#39258] [PATCH v3 2/3] guix: Search " Arun Isaac
2020-04-24 20:58 ` Ludovic Courtès
2020-03-27 16:26 ` [bug#39258] [PATCH v3 3/3] guix: Use package metadata cache for package search Arun Isaac
2020-04-24 21:03 ` Ludovic Courtès
2020-04-05 14:08 ` [bug#39258] [PATCH v3 0/3] Package metadata cache for guix search Ludovic Courtès
2020-04-24 21:05 ` Ludovic Courtès
2020-04-26 3:54 ` [bug#39258] benchmark search: default vs v2 vs v3 zimoun
2020-04-26 7:29 ` Pierre Neidhardt
2020-04-26 15:49 ` Ludovic Courtès
2020-04-26 17:01 ` zimoun
2020-04-26 20:22 ` Ludovic Courtès
2020-04-30 13:10 ` zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 0/3] Faster cache generation (similar as v3) zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 1/3] DRAFT packages: Add fields to packages cache zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 2/3] DRAFT packages: Add new procedure 'fold-packages*' zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 3/3] DRAFT guix package: Use cache in 'find-packages-by-description' zimoun
2020-05-03 16:43 ` [bug#39258] [PATCH v4 0/3] Faster cache generation (similar as v3) Ludovic Courtès
2020-05-03 18:10 ` zimoun
2020-05-03 19:49 ` Ludovic Courtès
2020-06-01 0:00 ` [bug#39258] [PATCH 0/4] Optimize guix search Arun Isaac
2020-06-01 0:00 ` [bug#39258] [PATCH 1/4] ui: Cut off search early if any regexp does not match Arun Isaac
2020-06-09 8:29 ` Ludovic Courtès
2020-06-01 0:00 ` [bug#39258] [PATCH 2/4] ui: Use string matching with literal search strings Arun Isaac
2020-06-09 8:33 ` Ludovic Courtès
2020-06-09 9:55 ` zimoun
2020-06-13 12:37 ` Arun Isaac
2020-06-13 13:36 ` zimoun
2020-06-13 17:21 ` Arun Isaac
2020-06-14 19:14 ` zimoun
2020-06-13 19:32 ` Ludovic Courtès
2020-06-15 20:18 ` Arun Isaac
2020-06-01 0:00 ` [bug#39258] [PATCH 3/4] ui: Do not translate package synopsis a second time Arun Isaac
2020-06-09 8:33 ` Ludovic Courtès
2020-06-01 0:00 ` [bug#39258] [PATCH 4/4] ui: Use package-description-string Arun Isaac
2020-06-09 8:34 ` Ludovic Courtès
2020-06-01 1:25 ` [bug#39258] [PATCH v5 0/4] Optimize guix search zimoun
2020-06-01 2:24 ` Arun Isaac
2020-06-01 10:01 ` zimoun
2020-06-01 10:11 ` [bug#39258] KMP string search algorithm? zimoun
2020-06-01 22:24 ` Leo Famulari
2020-06-01 23:48 ` Arun Isaac
2020-06-02 8:49 ` Ludovic Courtès
2021-07-15 7:33 ` zimoun [this message]
2021-07-15 7:33 ` [bug#39258] [PATCH v6 1/2] DRAFT packages: Add fields to packages cache zimoun
2021-07-17 8:31 ` Arun Isaac
2021-07-23 15:30 ` Ludovic Courtès
2021-08-17 14:03 ` zimoun
2021-07-15 7:33 ` [bug#39258] [PATCH v6 2/2] DRAFT scripts: package: Use cache in 'find-packages-by-description' zimoun
2021-07-23 15:43 ` [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances Ludovic Courtès
2021-08-20 15:42 ` zimoun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210715073328.212123-1-zimon.toutoune@gmail.com \
--to=zimon.toutoune@gmail.com \
--cc=39258@debbugs.gnu.org \
--cc=arunisaac@systemreboot.net \
--cc=ludo@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).