Faster “guix search” (was Re: How many bytes do we add (closure of guix) when adding one new package?)

[Simon Tournier <zimon.toutoune@gmail.com>]

Hi,

On Tue, 30 May 2023 at 21:10, Csepp <raingloom@riseup.net> wrote:

>              It makes zero sense to load full package definitions from
> disk for most queries, such as guix search, with an SoA representation
> we could load only the fields that we care about.

That’s already the case; see
~/.config/guix/current/lib/guix/package.cache.

For instance, “guix package -A” exploits it and the performances are
acceptable.  Two past summers, wow already! I tried to augment it and
exploit it for “guix search”.  The implementation and benchmark is in
#39258 [1].  Well, the whole thread of #39258 appears to me worth to
consider because it spots various bottleneck specific to “guix search”
and explains why the improvement is not straightforward.

Well, I have started months ago to write a Guix extension using
guile-xapian.  My aim is to tackle two annoyances: 1. the speed and
2. the relevance.

About the relevance #2, the issue is that the current scoring considers
only the local information of one package without considering the global
information of all the others.  Well, see [2,3,4] for some details. :-)

1: https://issues.guix.gnu.org/39258#119
2: https://yhetil.org/guix/CAJ3okZ3E3bhZ5pROZS68wEKdKOcZ8SpXsvdi-bnB=9Jz3mPahA@mail.gmail.com
3: https://yhetil.org/guix/CAJ3okZ3+hn0nJP98OhnZYLWJvhLGpdTUK+jB0hoM5JArQxO=zw@mail.gmail.com
4: https://yhetil.org/guix/CAJ3okZ0LaJzWDBA7bjqZew_jAmtt1rj9PJhevwrtBiA_COXENg@mail.gmail.com


> ps.: Now I'm even more glad that I'm using a file system with
> transparent compression on all my Guix systems.

Did you benchmarked the performances for some Guix operations on these
compressed vs uncompressed file system?


Cheers,
simon