[bug#39258] benchmark search: default vs v2 vs v3

[zimoun <zimon.toutoune@gmail.com>]

Hi Ludo,

On Sun, 26 Apr 2020 at 17:49, Ludovic Courtès <ludo@gnu.org> wrote:

> It does seem like Arun’s v3 (or maybe even v2) would work nicely.

The v3 is more interesting because it does not change the relevance
scoring and does not add other dependency.
However v2 is interesting to easily test BM25 which is another
relevance scoring... work in progress. :-)


> > The question is the tradeoff between: the slowdown of pull vs the
> > speedup of search. What is acceptable?
>
> That’s only one criterion among others.  I hear the argument that 25s is
> “nothing” compared to the rest, but it’s really a tradeoff.  Like, if I
> spent a day optimizing ‘guix pull’ and managed to save 25s, I would find
> it nice.  :-)

And I expect that the middle-term roadmap would even decrease more the
computations of derivations. ;-)



> > $ time guix pull -C ~/.config/guix/default-channels.scm
>
> It also depends on what’s in that file, of course.

Contains only one line: %default-channels

See my wishlist ;-)
https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00393.html



me:  2m13.693s
you: 0m57.916s

As we already discussed elsewhere, it is hard to "test" 'guix pull'.
Does it make sense to measure "guix pull"? As Chris (Marusich) did for
CDN.


> > Well, let remove the profiles and garbage collect the index files:
> >
> > rm /tmp/default /tmp/v{2,3}*
> > guix gc -D \
> >    /gnu/store/g5c08vqsv31nkn2r0hr32dbrkhf3cvd8-guix-package-cache \
> >    /gnu/store/8xbzhn81hmshagbgazmnr7xfps1cdsa3-guix-package-search-index \
> >    /gnu/store/8j78b5c4ddic21gcx7wpbq2akjn7x7mr-guix-package-metadata-cache
>
> Could you do, for v2 and v3:
>
>   time guix build /gnu/store/…-package-metadata-cache.drv --check

Newbie me! :-)

Two points:

   1. It may not be reproducible... I am checking.
   2. The time seems similar (v2=26s and v3=29s) considering the time
to start Guile and so on.

--8<---------------cut here---------------start------------->8---
guix gc --list-live | grep metadata
time /tmp/v3/bin/guix build
/gnu/store/jxs0abica8kjz1ppym95df97jk0qa9by-guix-package-metadata-cache.drv
--check
The following profile hook will be built:
   /gnu/store/jxs0abica8kjz1ppym95df97jk0qa9by-guix-package-metadata-cache.drv
building package cache...
(repl-version 0 1 1)
Generating package metadata cache for
'/gnu/store/95mi525syinh08jmcd3q7a7a8mr1sykb-profile'...
(values (value "/gnu/store/zhp7wv87vr6iis0fa3ff925i5r04i08q-guix-package-metadata-cache/lib/guix/package-metadata.cache"))
guix build: error: derivation
`/gnu/store/jxs0abica8kjz1ppym95df97jk0qa9by-guix-package-metadata-cache.drv'
may not be deterministic: output
`/gnu/store/zhp7wv87vr6iis0fa3ff925i5r04i08q-guix-package-metadata-cache'
differs

real    0m29.788s
user    0m0.535s
sys    0m0.025s
--8<---------------cut here---------------end--------------->8---


> That we’ll give us the exact cost of that part.  It’ll be interesting
> especially in the Xapian case, which we expected to be higher.

--8<---------------cut here---------------start------------->8---
time /tmp/v2/bin/guix build
/gnu/store/w0dhl2n3ngi4v2ld8lprkqjl1g1q2m4p-guix-package-search-index.drv
--check
The following profile hook will be built:
   /gnu/store/w0dhl2n3ngi4v2ld8lprkqjl1g1q2m4p-guix-package-search-index.drv
running profile hook of type 'package-search-index'...
(repl-version 0 1 1)
Generating package search index for
'/gnu/store/wiinj9nrb45wlf2cgbgkjl9chxz9cb9b-profile'...
(values (value "/gnu/store/8xbzhn81hmshagbgazmnr7xfps1cdsa3-guix-package-search-index/lib/guix/package-search.index"))
guix build: error: derivation
`/gnu/store/w0dhl2n3ngi4v2ld8lprkqjl1g1q2m4p-guix-package-search-index.drv'
may not be deterministic: output
`/gnu/store/8xbzhn81hmshagbgazmnr7xfps1cdsa3-guix-package-search-index'
differs

real    0m26.552s
user    0m0.626s
sys    0m0.046s
--8<---------------cut here---------------end--------------->8---

It is not higher. Why should it be?


Considering aside the issue of reproducibility -- which should be one!
-- well, should be possible to download the index file as any other
substitute?


Cheers,
simon