Re: Inverted index to accelerate guix package search

[Arun Isaac <arunisaac@systemreboot.net>]

[-- Attachment #1: Type: text/plain, Size: 2032 bytes --]


I've replaced the cache building code in gnu/packages.scm with code that
builds a sqlite database instead. I haven't finished hooking this up to
the guix search code. I'll have it ready in another day or two.

> To test "guix pull", simple "make as-derivation". Disclaim: can take
> some time :-)
>
> Then the issue is more to avoid to pollute your ~/.cache/guix and
> ~/.config/guix :-)
>
> 1. Update Guix with the result in /tmp/test
>
> guix pull -p /tmp/test --url=/path/to/guix/repo
>
> 2. Create your SQL index
>
> /tmp/test/bin/guix pull -p /tmp/trash
>
> Now your index should be created with all the packages currently in master.
> To have something reproducible (and faster), I suggest to add
> --commit= and always pull against the same commit.
>
> 3. Test the index
>
> /tmp/test/bin/guix search foo
>
> I mean something along these lines. ;-)

Thanks, I got the idea. I'll try it out.

>> I think it is not possible to search using regular expressions in sqlite
>
> I think it is possible. I imagine something using multiple query.
> I will give a look at the Guile module.

Sure, let me know if you find something.

> I disagree. We should keep the regexp. Otherwise we cannot include
> under "guix search" or "guix package --search=" because arguments
> about backward compatibility.

I agree about backward compatibility.

>> About sqlite versus an inverted index using vhashes, I don't know if it
>> is possible to serialize a vhash onto disk. Even if that were possible,
>> we'll have to load the entire vhash based inverted index into memory for
>> every invocation of guix search, and that could hit
>> performance. Something like guile-gdbm could have helped, but that's
>> another story.
>
> And your first test was not fair. ;-)
> Because you compared when the hash table was already in memory.
> I mean to know the real performance, only timing can talk. :-)

Yes, it wasn't completely fair. :-P I was assuming there was some way to
efficiently serialize to disk and read from it without too much overhead.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]