From: "Ludovic Courtès" <ludo@gnu.org>
To: Arun Isaac <arunisaac@systemreboot.net>
Cc: mail@ambrevar.xyz, 39258@debbugs.gnu.org, zimon.toutoune@gmail.com
Subject: [bug#39258] [PATCH v2 0/3] Xapian for Guix package search
Date: Sat, 07 Mar 2020 21:33:16 +0100 [thread overview]
Message-ID: <87sgijgb1v.fsf@gnu.org> (raw)
In-Reply-To: <20200307133116.11443-1-arunisaac@systemreboot.net> (Arun Isaac's message of "Sat, 7 Mar 2020 19:01:13 +0530")
Hello,
Arun Isaac <arunisaac@systemreboot.net> skribis:
> Here is the second iteration of my Xapian Guix package search patchset. I have
> found the reason the earlier patchset did not show significant speedup. It
> turns out that most of the time is spent in printing and texinfo rendering of
> the search results. So, in this patchset, I pre-render the search results
> while building the Xapian index and stuff them into the Xapian database
> itself. Therefore, during `guix search`, I just pull out the pre-rendered
> search results and print it on the screen. This is much faster. See comparison
> below.
>
> With a warm cache,
> $ time guix search inkscape
>
> real 0m1.787s
> user 0m1.745s
> sys 0m0.111s
>
> $ time /tmp/test/bin/guix search inkscape
>
> real 0m0.199s
> user 0m0.182s
> sys 0m0.024s
Nice!
In general, pre-rendering doesn’t seem practical to me: the output of
‘guix search’ is locale-dependent (it speaks the user’s language) and
adjusts to the terminal width (well, this is temporarily broken on
Guile 3.0.0, but see ‘%text-width’ in (guix ui)).
Also, if the 12K+ descriptions need to be rendered at the time the user
runs ‘guix pull’, the experience may not be great, because it could take
a bit of time.
WDYT?
> Why not use a simpler package search results format like Arch Linux or Debian
> does? We could just display the package name, version and synopsis like so.
>
> inkscape 0.92.4
> Vector graphics editor
> inklingreader 0.8
> Wacom Inkling sketch format conversion and manipulation
>
> Why do we need the entire recutils format? If the user is interested, they can
> always use `guix package --show` to get the full recutils formatted
> info. Having shorter search results will make everything even faster and much
> more readable. WDYT?
What I like about the recutils format in this context is that it’s both
human- and machine-readable. The examples in the manual show how it can
be useful to select the information displayed or to refine the search
(info "(guix) Invoking guix package").
Also: I’d recommend tackling one thing at a time. :-)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Note that ‘guix search’ time is largely dominated by I/O.
>
> Yes, `guix search` is I/O intensive. That is why I expect Xapian to do better
> since it only needs to access matching packages not all packages. Also, the
> Xapian index is fast at all times. It is not very dependent on a warm
> filesystem cache.
Yes, indeed.
>> On my laptop,
>> I get (first measurement is cold cache, second one is warm cache):
>>
>> --8<---------------cut here---------------start------------->8---
>> $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
>> $ time guix search foo >/dev/null
>>
>> real 0m2.631s
>> user 0m1.134s
>> sys 0m0.124s
>> $ time guix search foo >/dev/null
>>
>> real 0m0.836s
>> user 0m1.027s
>> sys 0m0.053s
>> --8<---------------cut here---------------end--------------->8---
>>
>> It’s hard to do better on the warm cache case because at this level,
>> there may be other things to optimize having little to do with searching
>> itself.
>>
>> Note that this is on an SSD; the cold-cache case must be worse on NFS or
>> on a spinning disk, and there we could gain a lot.
>
> My laptop is quite old with a particularly slow HDD. Hence my motivation to
> improve guix search performance!
Were you able to measure the cost of rendering specifically?
Here’s what I see when I turn ‘package->recutils’ into a no-op:
--8<---------------cut here---------------start------------->8---
$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time ./pre-inst-env guix search foo
real 0m1.617s
user 0m0.812s
sys 0m0.094s
$ time ./pre-inst-env guix search foo
real 0m0.595s
user 0m0.747s
sys 0m0.043s
--8<---------------cut here---------------end--------------->8---
To compare with:
--8<---------------cut here---------------start------------->8---
$ time ./pre-inst-env guix search foo >/dev/null
real 0m0.829s
user 0m1.026s
sys 0m0.046s
--8<---------------cut here---------------end--------------->8---
I think we should look at a profile of ‘package->recutils’, there’s
probably room for improvement there.
Thoughts?
Ludo’.
next prev parent reply other threads:[~2020-03-07 20:34 UTC|newest]
Thread overview: 126+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-23 19:51 [bug#39258] Faster guix search using an sqlite cache Arun Isaac
2020-01-29 23:33 ` zimoun
2020-01-30 13:48 ` Arun Isaac
2020-01-31 12:48 ` zimoun
2020-02-02 21:16 ` Arun Isaac
2020-02-04 10:19 ` zimoun
2020-02-06 1:58 ` Arun Isaac
2020-02-11 16:29 ` Ludovic Courtès
2020-02-11 18:21 ` zimoun
2020-02-11 18:39 ` Ludovic Courtès
2020-02-11 19:07 ` Arun Isaac
2020-02-11 20:20 ` zimoun
2020-02-15 14:50 ` Arun Isaac
2020-02-11 20:13 ` zimoun
2020-02-27 20:41 ` [bug#39258] [PATCH 0/4] Xapian for Guix package search Arun Isaac
2020-02-27 20:41 ` [bug#39258] [PATCH 1/4] gnu: Add guile-xapian Arun Isaac
2020-03-03 16:29 ` zimoun
2020-02-27 20:41 ` [bug#39258] [PATCH 2/4] build-self: Add guile-xapian to Guix dependencies Arun Isaac
2020-02-27 20:41 ` [bug#39258] [PATCH 3/4] gnu: Generate xapian package search index Arun Isaac
2020-02-28 8:04 ` Pierre Neidhardt
2020-03-05 20:26 ` Arun Isaac
2020-03-03 18:29 ` zimoun
2020-02-27 20:41 ` [bug#39258] [PATCH 4/4] gnu: Use xapian index for package search Arun Isaac
2020-02-28 8:11 ` Pierre Neidhardt
2020-03-03 19:21 ` zimoun
2020-03-03 19:51 ` zimoun
2020-02-28 8:13 ` [bug#39258] [PATCH 0/4] Xapian for Guix " Pierre Neidhardt
2020-02-28 12:39 ` zimoun
2020-02-28 12:49 ` Pierre Neidhardt
2020-02-28 15:36 ` Arun Isaac
2020-02-28 16:04 ` Arun Isaac
2020-03-02 18:37 ` zimoun
2020-03-02 19:13 ` zimoun
2020-03-03 20:04 ` zimoun
2020-02-29 8:25 ` Arun Isaac
2020-03-02 18:27 ` zimoun
2020-02-28 12:36 ` zimoun
2020-03-05 16:46 ` Ludovic Courtès
2020-03-07 13:31 ` [bug#39258] [PATCH v2 0/3] " Arun Isaac
2020-03-07 13:31 ` [bug#39258] [PATCH v2 1/3] build-self: Add guile-xapian to Guix dependencies Arun Isaac
2020-03-09 18:14 ` zimoun
2020-03-09 23:40 ` Jonathan Brielmaier
2020-03-10 5:24 ` Arun Isaac
2020-03-07 13:31 ` [bug#39258] [PATCH v2 2/3] gnu: Generate Xapian package search index Arun Isaac
2020-03-09 18:19 ` zimoun
2020-03-07 13:31 ` [bug#39258] [PATCH v2 3/3] gnu: Use Xapian index for package search Arun Isaac
2020-03-07 20:33 ` Ludovic Courtès [this message]
2020-03-08 9:01 ` [bug#39258] [PATCH v2 0/3] Xapian for Guix " Arun Isaac
2020-03-08 11:33 ` Ludovic Courtès
2020-03-08 20:27 ` Arun Isaac
2020-03-09 7:42 ` Pierre Neidhardt
2020-03-09 12:50 ` zimoun
2020-03-09 10:35 ` Ludovic Courtès
2020-03-10 14:17 ` Arun Isaac
2020-03-10 14:33 ` zimoun
2020-03-11 13:50 ` Ludovic Courtès
2020-03-13 5:37 ` Arun Isaac
2020-03-15 20:40 ` Ludovic Courtès
2020-03-09 7:50 ` Pierre Neidhardt
2020-03-09 10:28 ` Ludovic Courtès
2020-03-09 13:03 ` zimoun
2020-03-09 12:53 ` zimoun
2020-03-09 12:47 ` zimoun
2020-03-09 12:40 ` zimoun
2020-03-09 12:34 ` zimoun
2020-03-08 20:27 ` zimoun
2020-03-08 20:40 ` Arun Isaac
2020-03-09 12:28 ` zimoun
2020-03-27 16:26 ` [bug#39258] [PATCH v3 0/3] Package metadata cache for guix search Arun Isaac
2020-03-27 16:26 ` [bug#39258] [PATCH v3 1/3] guix: Generate package metadata cache Arun Isaac
2020-04-24 20:48 ` Ludovic Courtès
2020-04-26 9:48 ` zimoun
2020-04-26 14:35 ` Ludovic Courtès
2020-04-26 14:54 ` Pierre Neidhardt
2020-04-26 15:33 ` Ludovic Courtès
2020-04-26 15:05 ` zimoun
2020-03-27 16:26 ` [bug#39258] [PATCH v3 2/3] guix: Search " Arun Isaac
2020-04-24 20:58 ` Ludovic Courtès
2020-03-27 16:26 ` [bug#39258] [PATCH v3 3/3] guix: Use package metadata cache for package search Arun Isaac
2020-04-24 21:03 ` Ludovic Courtès
2020-04-05 14:08 ` [bug#39258] [PATCH v3 0/3] Package metadata cache for guix search Ludovic Courtès
2020-04-24 21:05 ` Ludovic Courtès
2020-04-26 3:54 ` [bug#39258] benchmark search: default vs v2 vs v3 zimoun
2020-04-26 7:29 ` Pierre Neidhardt
2020-04-26 15:49 ` Ludovic Courtès
2020-04-26 17:01 ` zimoun
2020-04-26 20:22 ` Ludovic Courtès
2020-04-30 13:10 ` zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 0/3] Faster cache generation (similar as v3) zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 1/3] DRAFT packages: Add fields to packages cache zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 2/3] DRAFT packages: Add new procedure 'fold-packages*' zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 3/3] DRAFT guix package: Use cache in 'find-packages-by-description' zimoun
2020-05-03 16:43 ` [bug#39258] [PATCH v4 0/3] Faster cache generation (similar as v3) Ludovic Courtès
2020-05-03 18:10 ` zimoun
2020-05-03 19:49 ` Ludovic Courtès
2020-06-01 0:00 ` [bug#39258] [PATCH 0/4] Optimize guix search Arun Isaac
2020-06-01 0:00 ` [bug#39258] [PATCH 1/4] ui: Cut off search early if any regexp does not match Arun Isaac
2020-06-09 8:29 ` Ludovic Courtès
2020-06-01 0:00 ` [bug#39258] [PATCH 2/4] ui: Use string matching with literal search strings Arun Isaac
2020-06-09 8:33 ` Ludovic Courtès
2020-06-09 9:55 ` zimoun
2020-06-13 12:37 ` Arun Isaac
2020-06-13 13:36 ` zimoun
2020-06-13 17:21 ` Arun Isaac
2020-06-14 19:14 ` zimoun
2020-06-13 19:32 ` Ludovic Courtès
2020-06-15 20:18 ` Arun Isaac
2020-06-01 0:00 ` [bug#39258] [PATCH 3/4] ui: Do not translate package synopsis a second time Arun Isaac
2020-06-09 8:33 ` Ludovic Courtès
2020-06-01 0:00 ` [bug#39258] [PATCH 4/4] ui: Use package-description-string Arun Isaac
2020-06-09 8:34 ` Ludovic Courtès
2020-06-01 1:25 ` [bug#39258] [PATCH v5 0/4] Optimize guix search zimoun
2020-06-01 2:24 ` Arun Isaac
2020-06-01 10:01 ` zimoun
2020-06-01 10:11 ` [bug#39258] KMP string search algorithm? zimoun
2020-06-01 22:24 ` Leo Famulari
2020-06-01 23:48 ` Arun Isaac
2020-06-02 8:49 ` Ludovic Courtès
2021-07-15 7:33 ` [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances zimoun
2021-07-15 7:33 ` [bug#39258] [PATCH v6 1/2] DRAFT packages: Add fields to packages cache zimoun
2021-07-17 8:31 ` Arun Isaac
2021-07-23 15:30 ` Ludovic Courtès
2021-08-17 14:03 ` zimoun
2021-07-15 7:33 ` [bug#39258] [PATCH v6 2/2] DRAFT scripts: package: Use cache in 'find-packages-by-description' zimoun
2021-07-23 15:43 ` [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances Ludovic Courtès
2021-08-20 15:42 ` zimoun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87sgijgb1v.fsf@gnu.org \
--to=ludo@gnu.org \
--cc=39258@debbugs.gnu.org \
--cc=arunisaac@systemreboot.net \
--cc=mail@ambrevar.xyz \
--cc=zimon.toutoune@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).