Re: File search

["Antoine R. Dumont (@ardumont)" <ardumont@softwareheritage.org>]

[-- Attachment #1: Type: text/plain, Size: 4343 bytes --]

Hello,

> So we went from 413s to 11s (on the Guix System node) for only 6% fewer
> files in the latter case?  Do I get that right?  That’s pretty cool.

Not 6% of loss, a bit more, around half is only detected between the
first and second round. Here is the summary [1] (org-mode) table I should
have sent to ease reading.

I don't have a better dataset to test against so you tell me if it is
still worth pushing in that direction ;)

[1]

|-----------+-------------+----------+----------|
| Iteration | Host System | Time (s) | Packages |
|-----------+-------------+----------+----------|
| 1st       | Debian      |   121.88 |      284 |
|           | Guix System |   413.55 |      749 |
|-----------+-------------+----------+----------|
| 2nd       | Debian      |      1.3 |      101 |
|           | Guix System |    11.54 |      354 |
|-----------+-------------+----------+----------|

> It should instead show “git@2.38.1:send-email”.  We probably need an
> ‘output’ field in the ‘Packages’ table.

Why must that be "git@2.38.1:send-email", what does it mean?

Providing we continue on that direction (see first question above), I'll
check what i can do (I'm not sure how to properly do that just yet).

> Also going forward we’ll need a schema version, as in:

yes, that I see how to do.

> Oh, and progress bars too.

I'm a bit unsettled on this. Hopefully it was mostly a joke ;)

If it's serious, how can we implement this as we don't know in advance
how many packages we'll discover locally (if we don't change the current
approach, that is, to avoid incurring too much time penalty).

And also, what implem should be used, I know the "pv" package provide
some pipe utility for that but that's about it. Do you have some example
in the guix codebase that does some progress bar already?

> And a pony.  :-)

;p

Cheers,
--
tony / Antoine R. Dumont (@ardumont)

-----------------------------------------------------------------
gpg fingerprint BF00 203D 741A C9D5 46A8 BE07 52E2 E984 0D10 C3B8


Ludovic Courtès <ludo@gnu.org> writes:

> Howdy!
>
> "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com>
> skribis:
>
>> Please, find enclosed the latest implementation as a patch (somewhat vcs
>> code ;). I've edited commits to mark Ludo as author with his
>> started/amended implementations first [0] (that should be in the patch).
>
> Nice!
>
>> For information, I extracted some number from runs to compare our
>> iterations (see the org-file attachment). The first iteration being
>> "extracts packages from the store" and the second one "extracts packages
>> from the system manifest". Those runs happened both on a guixified
>> debian host and a raw guix host (more packages).
>
> So we went from 413s to 11s (on the Guix System node) for only 6% fewer
> files in the latter case?  Do I get that right?  That’s pretty cool.
>
> The implementation based on manifests can of course miss packages, so
> it’s a tradeoff.  Purely local indexing will only find packages you
> already have anyway, so eventually we’ll need a second mode that would
> download a database.
>
> BTW, I noticed outputs are not properly handled so far, as in this
> example:
>
> --8<---------------cut here---------------start------------->8---
> $ GUIX_EXTENSIONS_PATH=$HOME/tmp/guix-index guix index search git-send-email
> git@2.38.1           /gnu/store/g3lgyzr749l76qma7srycclgsm0f78iq-git-2.38.1-send-email/libexec/git-core/git-send-email
> git@2.37.1           /gnu/store/n3hkzz5ydm0qm1c2jja2pwy2v19mq1k0-git-2.37.1-send-email/libexec/git-core/git-send-email
> --8<---------------cut here---------------end--------------->8---
>
> It should instead show “git@2.38.1:send-email”.  We probably need an
> ‘output’ field in the ‘Packages’ table.
>
> Also going forward we’ll need a schema version, as in:
>
> --8<---------------cut here---------------start------------->8---
> create table SchemaVersion (
>     version     integer not null;
> );
> --8<---------------cut here---------------end--------------->8---
>
> so that the tool can upgrade or discard databases that have the wrong
> version.
>
> Oh, and progress bars too.  And a pony.  :-)
>
> Thanks for your work!
>
> Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 873 bytes --]