all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
From: zimoun <zimon.toutoune@gmail.com>
To: "Ludovic Courtès" <ludo@gnu.org>
Cc: Arun Isaac <arunisaac@systemreboot.net>, 39258@debbugs.gnu.org
Subject: [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances
Date: Fri, 20 Aug 2021 17:42:44 +0200	[thread overview]
Message-ID: <CAJ3okZ3ijny997BMMZ2yqwHTQJG7C2YZo+QMHOE+z0xhspsSug@mail.gmail.com> (raw)
In-Reply-To: <875yx1ave7.fsf@gnu.org>

Hi Ludo,

On Fri, 23 Jul 2021 at 17:43, Ludovic Courtès <ludo@gnu.org> wrote:

> > From my understanding, the issue that 'package-relevance' accepts a 'package'
> > (and then all the chain until displaying) and 'fold-avaibale-packages' does
> > not return a package.  Well, I do not know; especially where to put something
> > similiar to 'read-package-from'.
>
> Yeah that’s annoying.  Perhaps we need <proto-package> or
> <package-metadata>.  With some trickery we could have record type
> inheritance or something, maybe.  Dunno.
>
> It would be good if we could arrange so that ‘fold-available-packages’
> doesn’t allocate anything though.

It does not allocate, the allocation is done by 'find-packages-by-description'.

Well, I think v4/v3 [0] is the direction to follow.  Therefore, I
would revisit this version and try to address two of Ludo's comments
[1] and the other ones in v3.

BTW, I have not investigated from where the slowness comes:
 - allocation
 - garbage collection
 - '(module-ref (resolve-interface module) symbol)'
because I have been a bit disappointed by the performance of this v6.

0: <http://issues.guix.gnu.org/39258#89>
1: <http://issues.guix.gnu.org/39258#93>

> > Let compare only for cold cache and time this cache building (Guix 7db8fd6):
> >
> >   sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
> >   time guix build --check $(guix gc --derivers $(readlink -f ~/.config/guix/current/lib/guix/package.cache))
> >
> >   real        0m28,848s
> >   user        0m1,481s
> >   sys 0m0,252s
> >
> >   sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
> >   time guix build --check $(guix gc --derivers $(readlink -f /tmp/new/lib/guix/package.cache))
> >
> >   real        0m40,279s
> >   user        0m1,582s
> >   sys 0m0,232s
> >
> > It seems longer but compared to the time of "guix pull" completion, it seems
> > acceptable.
>
> Both the initial timing and the target are waaay too much.  :-/

Yes, but that's another story. :-)  We cannot fix all in the same time, IMHO.

Here the point is to speedup "guix search" and to accept to pay a
little more at "guix pull" time; then we could optimize the cache
generation.  Considering the overall time of "guix pull", this extra
time appears to me acceptable---if "guix search" is faster. :-)

> On my i7 laptop I have:
>
> --8<---------------cut here---------------start------------->8---
> $ time ./pre-inst-env  guile -c '(use-modules (gnu packages)) (generate-package-cache "/tmp/t.cache")'
>
> real    0m20.738s
> user    0m44.413s
> sys     0m0.341s
> --8<---------------cut here---------------end--------------->8---
>
> It’s CPU-bound; we should probably start by optimizing that.
>
> In Guile 3.0.7 there was a change that improved this noticeably:
>
>   https://git.savannah.gnu.org/cgit/guile.git/commit/?id=05614f792bfabbc33798863edd0bb67c744e9299
>
> We should prolly look for similar optimization opportunities in the
> assembler…

Yes, but this kind of optimization will not provide a faster "guix
search" but a faster "guix pull".

--8<---------------cut here---------------start------------->8---
$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time guile -c '(use-modules (gnu packages)) (generate-package-cache
"/tmp/t1.cache")'

real    0m15,728s
user    0m23,940s
sys     0m0,826s

$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time guile -c '(load-compiled "/tmp/t1.cache/lib/guix/package.cache")'

real    0m1,026s
user    0m0,258s
sys     0m0,051s

$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time ./pre-inst-env  guile -c '(use-modules (gnu packages))
(generate-package-cache "/tmp/t2.cache")'

real    0m35,570s
user    3m12,951s
sys     0m3,807s

$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time guile -c '(load-compiled "/tmp/t2.cache/lib/guix/package.cache")'

real    0m1,055s
user    0m0,283s
sys     0m0,055s
--8<---------------cut here---------------end--------------->8---

(And if we speak about performance, raw loading the cache takes ~1s
but "guix package -A >/dev/null" takes ~8s.  It is a big gap for
parsing the CLI and sorting.  Worse, "guix --version >/dev/null" takes
~2s on cold cache.  We should probably start by reduce this overhead.)

> > Let compare for some queries:
> >
> >   sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
> >   time guix search game | recsel -C -P name | wc -l
> >   371
> >
> >   real        0m7,561s
> >   user        0m3,525s
> >   sys 0m0,391s
>
> I think you should run:
>
>   time guix search game > /dev/null
>
> otherwise Bash’s built-in ‘time’ shows the wall-clock time of the whole
> pipeline, and the processing time of ‘recsel’ is probably not negligible
> here.

First, I am confused... If the formatter (recsel) is not negligible,
then it should be dropped and replaced by an internal (fast)
formatter.  Well, I mean that as an end-user I am interested by the
time of the whole pipeline.

Second, on my machine the time is somehow negligible*. ;-)  On cold
cache, 10 runs using the pipe or using the redirection, keeping the
max and the min for each:

--8<---------------cut here---------------start------------->8---
real    0m9,598s
user    0m3,961s
sys     0m0,415s

real    0m8,744s
user    0m3,772s
sys     0m0,431s
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
real    0m8,755s
user    0m3,869s
sys     0m0,540s

real    0m8,390s
user    0m3,767s
sys     0m0,416s
--8<---------------cut here---------------end--------------->8---

*negligible: better said, it does not give the bad impression.  Even
if it is roughly 5% of difference.


Cheers,
simon




      reply	other threads:[~2021-08-20 15:44 UTC|newest]

Thread overview: 126+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-23 19:51 [bug#39258] Faster guix search using an sqlite cache Arun Isaac
2020-01-29 23:33 ` zimoun
2020-01-30 13:48   ` Arun Isaac
2020-01-31 12:48     ` zimoun
2020-02-02 21:16       ` Arun Isaac
2020-02-04 10:19         ` zimoun
2020-02-06  1:58           ` Arun Isaac
2020-02-11 16:29             ` Ludovic Courtès
2020-02-11 18:21               ` zimoun
2020-02-11 18:39                 ` Ludovic Courtès
2020-02-11 19:07                   ` Arun Isaac
2020-02-11 20:20                     ` zimoun
2020-02-15 14:50                     ` Arun Isaac
2020-02-11 20:13                   ` zimoun
2020-02-27 20:41 ` [bug#39258] [PATCH 0/4] Xapian for Guix package search Arun Isaac
2020-02-27 20:41   ` [bug#39258] [PATCH 1/4] gnu: Add guile-xapian Arun Isaac
2020-03-03 16:29     ` zimoun
2020-02-27 20:41   ` [bug#39258] [PATCH 2/4] build-self: Add guile-xapian to Guix dependencies Arun Isaac
2020-02-27 20:41   ` [bug#39258] [PATCH 3/4] gnu: Generate xapian package search index Arun Isaac
2020-02-28  8:04     ` Pierre Neidhardt
2020-03-05 20:26       ` Arun Isaac
2020-03-03 18:29     ` zimoun
2020-02-27 20:41   ` [bug#39258] [PATCH 4/4] gnu: Use xapian index for package search Arun Isaac
2020-02-28  8:11     ` Pierre Neidhardt
2020-03-03 19:21     ` zimoun
2020-03-03 19:51       ` zimoun
2020-02-28  8:13   ` [bug#39258] [PATCH 0/4] Xapian for Guix " Pierre Neidhardt
2020-02-28 12:39     ` zimoun
2020-02-28 12:49       ` Pierre Neidhardt
2020-02-28 15:36     ` Arun Isaac
2020-02-28 16:04       ` Arun Isaac
2020-03-02 18:37         ` zimoun
2020-03-02 19:13           ` zimoun
2020-03-03 20:04             ` zimoun
2020-02-29  8:25       ` Arun Isaac
2020-03-02 18:27         ` zimoun
2020-02-28 12:36   ` zimoun
2020-03-05 16:46   ` Ludovic Courtès
2020-03-07 13:31 ` [bug#39258] [PATCH v2 0/3] " Arun Isaac
2020-03-07 13:31   ` [bug#39258] [PATCH v2 1/3] build-self: Add guile-xapian to Guix dependencies Arun Isaac
2020-03-09 18:14     ` zimoun
2020-03-09 23:40     ` Jonathan Brielmaier
2020-03-10  5:24       ` Arun Isaac
2020-03-07 13:31   ` [bug#39258] [PATCH v2 2/3] gnu: Generate Xapian package search index Arun Isaac
2020-03-09 18:19     ` zimoun
2020-03-07 13:31   ` [bug#39258] [PATCH v2 3/3] gnu: Use Xapian index for package search Arun Isaac
2020-03-07 20:33   ` [bug#39258] [PATCH v2 0/3] Xapian for Guix " Ludovic Courtès
2020-03-08  9:01     ` Arun Isaac
2020-03-08 11:33       ` Ludovic Courtès
2020-03-08 20:27         ` Arun Isaac
2020-03-09  7:42           ` Pierre Neidhardt
2020-03-09 12:50             ` zimoun
2020-03-09 10:35           ` Ludovic Courtès
2020-03-10 14:17             ` Arun Isaac
2020-03-10 14:33               ` zimoun
2020-03-11 13:50               ` Ludovic Courtès
2020-03-13  5:37                 ` Arun Isaac
2020-03-15 20:40                   ` Ludovic Courtès
2020-03-09  7:50         ` Pierre Neidhardt
2020-03-09 10:28           ` Ludovic Courtès
2020-03-09 13:03             ` zimoun
2020-03-09 12:53           ` zimoun
2020-03-09 12:47         ` zimoun
2020-03-09 12:40       ` zimoun
2020-03-09 12:34     ` zimoun
2020-03-08 20:27   ` zimoun
2020-03-08 20:40     ` Arun Isaac
2020-03-09 12:28   ` zimoun
2020-03-27 16:26 ` [bug#39258] [PATCH v3 0/3] Package metadata cache for guix search Arun Isaac
2020-03-27 16:26   ` [bug#39258] [PATCH v3 1/3] guix: Generate package metadata cache Arun Isaac
2020-04-24 20:48     ` Ludovic Courtès
2020-04-26  9:48       ` zimoun
2020-04-26 14:35         ` Ludovic Courtès
2020-04-26 14:54           ` Pierre Neidhardt
2020-04-26 15:33             ` Ludovic Courtès
2020-04-26 15:05           ` zimoun
2020-03-27 16:26   ` [bug#39258] [PATCH v3 2/3] guix: Search " Arun Isaac
2020-04-24 20:58     ` Ludovic Courtès
2020-03-27 16:26   ` [bug#39258] [PATCH v3 3/3] guix: Use package metadata cache for package search Arun Isaac
2020-04-24 21:03     ` Ludovic Courtès
2020-04-05 14:08   ` [bug#39258] [PATCH v3 0/3] Package metadata cache for guix search Ludovic Courtès
2020-04-24 21:05   ` Ludovic Courtès
2020-04-26  3:54 ` [bug#39258] benchmark search: default vs v2 vs v3 zimoun
2020-04-26  7:29   ` Pierre Neidhardt
2020-04-26 15:49   ` Ludovic Courtès
2020-04-26 17:01     ` zimoun
2020-04-26 20:22       ` Ludovic Courtès
2020-04-30 13:10     ` zimoun
2020-05-03 15:01 ` [bug#39258] [PATCH v4 0/3] Faster cache generation (similar as v3) zimoun
2020-05-03 15:01   ` [bug#39258] [PATCH v4 1/3] DRAFT packages: Add fields to packages cache zimoun
2020-05-03 15:01   ` [bug#39258] [PATCH v4 2/3] DRAFT packages: Add new procedure 'fold-packages*' zimoun
2020-05-03 15:01   ` [bug#39258] [PATCH v4 3/3] DRAFT guix package: Use cache in 'find-packages-by-description' zimoun
2020-05-03 16:43   ` [bug#39258] [PATCH v4 0/3] Faster cache generation (similar as v3) Ludovic Courtès
2020-05-03 18:10     ` zimoun
2020-05-03 19:49       ` Ludovic Courtès
2020-06-01  0:00 ` [bug#39258] [PATCH 0/4] Optimize guix search Arun Isaac
2020-06-01  0:00   ` [bug#39258] [PATCH 1/4] ui: Cut off search early if any regexp does not match Arun Isaac
2020-06-09  8:29     ` Ludovic Courtès
2020-06-01  0:00   ` [bug#39258] [PATCH 2/4] ui: Use string matching with literal search strings Arun Isaac
2020-06-09  8:33     ` Ludovic Courtès
2020-06-09  9:55       ` zimoun
2020-06-13 12:37       ` Arun Isaac
2020-06-13 13:36         ` zimoun
2020-06-13 17:21           ` Arun Isaac
2020-06-14 19:14             ` zimoun
2020-06-13 19:32         ` Ludovic Courtès
2020-06-15 20:18           ` Arun Isaac
2020-06-01  0:00   ` [bug#39258] [PATCH 3/4] ui: Do not translate package synopsis a second time Arun Isaac
2020-06-09  8:33     ` Ludovic Courtès
2020-06-01  0:00   ` [bug#39258] [PATCH 4/4] ui: Use package-description-string Arun Isaac
2020-06-09  8:34     ` Ludovic Courtès
2020-06-01  1:25   ` [bug#39258] [PATCH v5 0/4] Optimize guix search zimoun
2020-06-01  2:24     ` Arun Isaac
2020-06-01 10:01     ` zimoun
2020-06-01 10:11 ` [bug#39258] KMP string search algorithm? zimoun
2020-06-01 22:24   ` Leo Famulari
2020-06-01 23:48     ` Arun Isaac
2020-06-02  8:49       ` Ludovic Courtès
2021-07-15  7:33 ` [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances zimoun
2021-07-15  7:33   ` [bug#39258] [PATCH v6 1/2] DRAFT packages: Add fields to packages cache zimoun
2021-07-17  8:31     ` Arun Isaac
2021-07-23 15:30       ` Ludovic Courtès
2021-08-17 14:03         ` zimoun
2021-07-15  7:33   ` [bug#39258] [PATCH v6 2/2] DRAFT scripts: package: Use cache in 'find-packages-by-description' zimoun
2021-07-23 15:43   ` [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances Ludovic Courtès
2021-08-20 15:42     ` zimoun [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAJ3okZ3ijny997BMMZ2yqwHTQJG7C2YZo+QMHOE+z0xhspsSug@mail.gmail.com \
    --to=zimon.toutoune@gmail.com \
    --cc=39258@debbugs.gnu.org \
    --cc=arunisaac@systemreboot.net \
    --cc=ludo@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.