guix package --search slow ?

guix package --search slow ?

[zimoun]

Dear,

When searching packages with `guix search`, it appears to me slow;
especially compared to `aptitude` of Debian.

Say on my machine:

$ time guix package --search=python-numpy | recsel -p name -
name: python-numpy
name: python-numpy-documentation
name: python-numpydoc
name: python2-numpy-documentation

real    0m11.864s
user    0m1.160s
sys     0m0.212s


$ time aptitude search python-numpy
i A python-numpy                    - Numerical Python adds a fast array facilit
v   python-numpy-abi9               -
v   python-numpy-api10              -
p   python-numpy-dbg                - Fast array facility to the Python language
v   python-numpy-dev                -
p   python-numpy-doc                - NumPy documentation
p   python-numpydoc                 - Sphinx extension to support docstrings in

real    0m3.219s
user    0m0.656s
sys     0m0.088s


I am not sure to exactly understand how the search works.
Basically, a "big" `fold' is done on `%package-module-path' and there
is no caching. Right?

What should be done to improve the search speed?


All the best,
simon

Re: guix package --search slow ?

[Ricardo Wurmus]

For me this command

  time guix package --search=python-numpy

when run for the first time takes

  real	0m4.729s
  user	0m1.332s
  sys	0m0.815s

All following invocations finish in about a second:

  real	0m1.469s
  user	0m1.459s
  sys	0m0.108s

-- 
Ricardo

Re: guix package --search slow ?

[Meiyo Peng]

Hi zimoun,

zimoun writes:

> When searching packages with `guix search`, it appears to me slow;
> especially compared to `aptitude` of Debian.
> ...
> What should be done to improve the search speed?

I remember "guix package --search" has always been pretty fast since
this patch series of Ludovic is merged into guix:
https://lists.gnu.org/archive/html/guix-patches/2019-01/msg00254.html.

Before that, "guix package --search" is slow.

Please check your guix's version.  If you are not using an old Guix.  I
am not sure why your experience is different from us.


--
Meiyo Peng
https://www.pengmeiyu.com/

Re: guix package --search slow ?

[zimoun]

On Tue, 14 May 2019 at 22:17, Ricardo Wurmus <rekado@elephly.net> wrote:
>
> For me this command
>
>   time guix package --search=python-numpy
>
> when run for the first time takes
>
>   real  0m4.729s
>   user  0m1.332s
>   sys   0m0.815s
>
> All following invocations finish in about a second:
>
>   real  0m1.469s
>   user  0m1.459s
>   sys   0m0.108s

Your machine is probably faster than mine ;-)

Yes, once the cache is warm, then it is faster.
By "All following invocations" you mean a search for a non related
previous one. Right?
It is expected to be faster because the `fold' over all the package
has warmed the memory cache (kernel stuff). Right?

On my machine, I have:

 $ time guix package --search=numpy
real    0m7.353s
user    0m1.256s
sys     0m0.136s

$  time guix package --search=conduit
real    0m0.728s
user    0m0.764s
sys     0m0.044s

compared to Debian `aptitude` for example:

 $ time aptitude search numpy
real    0m3.513s
user    0m0.692s
sys     0m0.092s

 $ time aptitude search conduit
real    0m0.622s
user    0m0.540s
sys     0m0.048s


So yes, it depends on how the memory cache is (echo 1 >
/proc/sys/vm/drop_caches or echo 3 > /proc/sys/vm/drop_caches) , and
once the cache is warmed, it is fast enough.

My point is that the first search is (often)at least twice longer than
the `aptitude` counter part. And even it is longer than a `git grep`
on the raw .scm files.

 $ time git --no-pager grep numpy
real    0m4.369s
user    0m0.112s
sys     0m0.132s

 $ time git --no-pager grep conduit
real    0m0.069s
user    0m0.108s
sys     0m0.012s


Well, maybe I miss a point.


All the best,
simon

Re: guix package --search slow ?

[zimoun]

Dear,

On Wed, 15 May 2019 at 04:22, Meiyo Peng <meiyo@riseup.net> wrote:
>
> zimoun writes:
>
> > When searching packages with `guix search`, it appears to me slow;
> > especially compared to `aptitude` of Debian.
> > ...
> > What should be done to improve the search speed?
>
> I remember "guix package --search" has always been pretty fast since
> this patch series of Ludovic is merged into guix:
> https://lists.gnu.org/archive/html/guix-patches/2019-01/msg00254.html.
>
> Before that, "guix package --search" is slow.

Thank you for the pointer. Always interesting. :-)
And you mean: was slower. ;-)

>
> Please check your guix's version.  If you are not using an old Guix.  I
> am not sure why your experience is different from us.

Hum? I am doing on regular basis `guix pull`. This updates the version, right?
I mean, `guix describe` tells:

guix describe
Generation 22   May 13 2019 18:46:26    (current)
  bimsb d1cba4a
    repository URL: file:///home/simon/work/guix/bimsb
    branch: master
    commit: d1cba4a2cba1eb6c6c33328fea511b75dfcffe39
  guix ccabb66
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: ccabb664ed55d44767e037bb73169be2ecf9449d

Hum? maybe I am missing something.


All the best,
simon

Re: guix package --search slow ?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> Your machine is probably faster than mine ;-)

I ran this on a Thinkpad T60 with an Intel Core2 CPU (T7600) @ 2.33GHz.

> Yes, once the cache is warm, then it is faster.
> By "All following invocations" you mean a search for a non related
> previous one. Right?

I meant: the very same search.

> My point is that the first search is (often)at least twice longer than
> the `aptitude` counter part. And even it is longer than a `git grep`
> on the raw .scm files.

Yeah, it would be good to figure out a way to make it faster.

--
Ricardo

Re: guix package --search slow ?

[Ludovic Courtès]

Hi zimoun,

zimoun <zimon.toutoune@gmail.com> skribis:

> Your machine is probably faster than mine ;-)

For the record, commit 94aeec0aef03ab44e41bfc3e77c3b623cb3d607c, shortly
before 1.0, make ‘guix search’ faster (as in less CPU usage.)

>  $ time guix package --search=numpy
> real    0m7.353s
> user    0m1.256s
> sys     0m0.136s
>
> $  time guix package --search=conduit
> real    0m0.728s
> user    0m0.764s
> sys     0m0.044s
>
> compared to Debian `aptitude` for example:
>
>  $ time aptitude search numpy
> real    0m3.513s
> user    0m0.692s
> sys     0m0.092s
>
>  $ time aptitude search conduit
> real    0m0.622s
> user    0m0.540s
> sys     0m0.048s

The cold-cache timings are worse for Guix, which ends up doing a lot of
I/O to traverse all these .scm files; APT only has to go through a
couple of files I guess.

The warm-cache timings are comparable though, which I think is good.
:-)

We could probably improve the cold-cache timings by adding synopses and
descriptions to the package cache that ‘guix pull’ builds in
~/.config/guix/current/lib/guix/package.cache, but that would also make
that cache bigger.  Not sure if it’s worth it.

Thanks,
Ludo’.

Re: guix package --search slow ?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 671 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> We could probably improve the cold-cache timings by adding synopses and
> descriptions to the package cache that ‘guix pull’ builds in
> ~/.config/guix/current/lib/guix/package.cache, but that would also make
> that cache bigger.  Not sure if it’s worth it.

To be clear, did you mean it might not be worth it because of the size
it takes on the user's drive or because the increase in size might slow
down the lookup?

For the former, currently my cache is 3.5 MiB big so I think it's safe
to go ahead and try if we get faster lookups with embedded synopses.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: guix package --search slow ?

[Jonathan Brielmaier]

On 5/15/19 3:34 PM, Ludovic Courtès wrote:
> We could probably improve the cold-cache timings by adding synopses and
> descriptions to the package cache that ‘guix pull’ builds in
> ~/.config/guix/current/lib/guix/package.cache, but that would also make
> that cache bigger.  Not sure if it’s worth it.

Another idea would be to have some kind of simple search which only
includes the package name and maybe the synopsis. So this could go to
the cache and I would assume that a cold-cache search will also be
faster. And as a nice give away, it doesn't include so much scrolling :)

So something like:
$ guix search --simple numpy
python-numpy                 | Fundamental package for scientific...
python-numpy-documentation   | Documentation for the python-numpy...
python-numpydoc              |
python2-numpy                |
python2-numpy                |
python2-numpy-documentation  |
python2-numpydoc             |

But in general I don't find the search so slow. Sometimes zypper for
example has to refresh the repos and than "guix search" is faster, even
with cold cache...

Re: guix package --search slow ?

[zimoun]

Hi,

On Wed, 15 May 2019 at 04:22, Meiyo Peng <meiyo@riseup.net> wrote:
>
> I remember "guix package --search" has always been pretty fast since
> this patch series of Ludovic is merged into guix:
> https://lists.gnu.org/archive/html/guix-patches/2019-01/msg00254.html.
>
> Before that, "guix package --search" is slow.

From the message of the pointed patch, only `guix package -A` uses
this new cache. If I understand well.
I mean, this cache is loaded in the function `fold-available-packages'
used by the option `--list-available` but not in the functions used by
the option `--search`.
Ludo pointed the commit 94aeec0aef03ab44e41bfc3e77c3b623cb3d607c which
improves the search by doing directly on the raw Texinfo instead.
Maybe you remembered this one.

Thank you to pointing me this patch, now I know a bit more about the
internals. Not sure to understand well. :-)


All the ebst,
simon

Re: guix package --search slow ?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 395 bytes --]

> Another idea would be to have some kind of simple search which only
> includes the package name and maybe the synopsis.

For Emacs users, it's what helm-system-packages does.  It implements its
own names+synopses cache.  helm-system-packages takes some seconds on
first start to build the cache; after that you get instant live search! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: guix package --search slow ?

[zimoun]

Hi,

On Wed, 15 May 2019 at 15:34, Ludovic Courtès <ludo@gnu.org> wrote:
>
> For the record, commit 94aeec0aef03ab44e41bfc3e77c3b623cb3d607c, shortly
> before 1.0, make ‘guix search’ faster (as in less CPU usage.)

Six time faster the commit message claims ;-)
Awesome how only 2 lines change things.


> The cold-cache timings are worse for Guix, which ends up doing a lot of
> I/O to traverse all these .scm files; APT only has to go through a
> couple of files I guess.

Hum? I do not know how APT (say the `aptitude` implementation) works.
I suppose that `aptitude update` pulls then hashes somehow a database/cache.

I mean reduce the I/O with all these scattered .scm files should
improve the timings.

> The warm-cache timings are comparable though, which I think is good.
> :-)

Yes! :-)
However, because the first search is slow, the final feeling is that
`guix search` is slow. :-(


> We could probably improve the cold-cache timings by adding synopses and
> descriptions to the package cache that ‘guix pull’ builds in
> ~/.config/guix/current/lib/guix/package.cache, but that would also make
> that cache bigger.  Not sure if it’s worth it.

Hum? and why not use SQLite for this file?


All the best,
simon

Re: guix package --search slow ?

[zimoun]

On Wed, 15 May 2019 at 15:48, Jonathan Brielmaier
<jonathan.brielmaier@web.de> wrote:
>
> On 5/15/19 3:34 PM, Ludovic Courtès wrote:
> > We could probably improve the cold-cache timings by adding synopses and
> > descriptions to the package cache that ‘guix pull’ builds in
> > ~/.config/guix/current/lib/guix/package.cache, but that would also make
> > that cache bigger.  Not sure if it’s worth it.
>
> Another idea would be to have some kind of simple search which only
> includes the package name and maybe the synopsis. So this could go to
> the cache and I would assume that a cold-cache search will also be
> faster. And as a nice give away, it doesn't include so much scrolling :)

`guix package -A` already searches only for the names and uses the cache.
From my opinion, the difference between adding the synopsis or
synopsis+description to the cache is similar. Qui peut le plus peut le
moins (Who can do more can do less too ;-)

> But in general I don't find the search so slow. Sometimes zypper for
> example has to refresh the repos and than "guix search" is faster, even
> with cold cache...

I agree that now the performance are not so "horrible", but even, can
we do better? :-)


All the best,
simon

Re: guix package --search slow ?

[zimoun]

Hi Pierre,

On Wed, 15 May 2019 at 16:08, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> > Another idea would be to have some kind of simple search which only
> > includes the package name and maybe the synopsis.
>
> For Emacs users, it's what helm-system-packages does.  It implements its
> own names+synopses cache.  helm-system-packages takes some seconds on
> first start to build the cache; after that you get instant live search! :)

You are doing ads for your package ;-)

Well, I cite your doctring [1], that's the point.

Guix is extremely slow to list everything, thus the cache is
persisted on drive. It's only updated whenever
`helm-system-packages-guix--last-pull-commits' is different from the
cache commit.

And what do mean by this comment [2]

;; We build both caches at the same time. We could also build just-in-time, but
;; benchmarks show that it only saves less than 20% when building one cache.



Last, the "slowness" comes from `guix package --search` doing there
[3], I guess.


What is the size of the persistent cache?



[1] https://github.com/emacs-helm/helm-system-packages/blob/master/helm-system-packages-guix.el#L141
[2] https://github.com/emacs-helm/helm-system-packages/blob/master/helm-system-packages-guix.el#L148
[3] https://github.com/emacs-helm/helm-system-packages/blob/master/helm-system-packages-guix.el#L163


Cheers,
simon

Re: guix package --search slow ?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 915 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

>> For Emacs users, it's what helm-system-packages does.  It implements its
>> own names+synopses cache.  helm-system-packages takes some seconds on
>> first start to build the cache; after that you get instant live search! :)
>
> You are doing ads for your package ;-)

Helm, what else... :p

> And what do mean by this comment [2]
>
> ;; We build both caches at the same time. We could also build just-in-time, but
> ;; benchmarks show that it only saves less than 20% when building one cache.

Here "caches" refer to the internal in-memory caches of
helm-system-packages, this has nothing to do with the "guix search..."
cache.  Poor phrasing, admittedly. :/

> Last, the "slowness" comes from `guix package --search` doing there
> [3], I guess.

Yes, helm-system-packages relies on "guix search".

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: guix package --search slow ?

[zimoun]

Hi,

My bad !

On Wed, 15 May 2019 at 15:34, Ludovic Courtès <ludo@gnu.org> wrote:

> The warm-cache timings are comparable though, which I think is good.
> :-)

`aptitude search` is not equivalent to `guix package --search` but to
`guix package --list-available`.
The `aptitude` equivalence should be `aptitude search '?term(numpy)'`

And now, Guix is better. ;-)

$ time guix package -s numpy
real    0m7.924s
user    0m1.240s
sys     0m0.144s

$ time guix package -s conduit
real    0m0.758s
user    0m0.796s
sys     0m0.044s

compared to:

$ time aptitude search '?term(numpy)'
real    0m7.393s
user    0m1.556s
sys     0m0.628s

$ time aptitude search '?term(conduit)'
real    0m1.824s
user    0m1.352s
sys     0m0.464s


However, I still think that a SQL caching file containing also
descriptions should improve the searching experience with cold cache.

Thanks! Now I have better understood some internals. :-)

All the best,
simon

Re: guix package --search slow ?

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 469 bytes --]

Hello Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

[...]

> We could probably improve the cold-cache timings by adding synopses and
> descriptions to the package cache that ‘guix pull’ builds in
> ~/.config/guix/current/lib/guix/package.cache, but that would also make
> that cache bigger.  Not sure if it’s worth it.

AFAIU it's worth it, at least in terms of disk space

Thanks,
Gio'.

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: guix package --search slow ?

[Ludovic Courtès]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> We could probably improve the cold-cache timings by adding synopses and
>> descriptions to the package cache that ‘guix pull’ builds in
>> ~/.config/guix/current/lib/guix/package.cache, but that would also make
>> that cache bigger.  Not sure if it’s worth it.
>
> To be clear, did you mean it might not be worth it because of the size
> it takes on the user's drive or because the increase in size might slow
> down the lookup?

If the cache is bigger, loading it may take a bit more time.

On an SSD, I find the timings to be acceptable now, so I’m also a bit
biased.

Ludo’.