Faster!

Faster!

[Ludovic Courtès]

Hello!

With commits a7b6ffe and bdcf35a, you should notice that ‘guix-build’
and similar are more than twice as fast as two days ago.

This was achieved by adding caches in the now obvious places, which
noticeably reduces the number of round trips and the amount of data sent
to the daemon.

Ludo’.

Re: Faster!

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 278 bytes --]

Am Mittwoch, 30. Januar 2013 schrieb Ludovic Courtès:
> With commits a7b6ffe and bdcf35a, you should notice that ‘guix-build’
> and similar are more than twice as fast as two days ago.

Excellent! Is "guix-package --roll-back" also faster now? It feels so!

Andreas

[-- Attachment #2: Type: text/html, Size: 1797 bytes --]

Re: Faster!

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Am Mittwoch, 30. Januar 2013 schrieb Ludovic Courtès:
>> With commits a7b6ffe and bdcf35a, you should notice that ‘guix-build’
>> and similar are more than twice as fast as two days ago.

There’ve been additional optimizations in the meantime, so it’s now 4
times as fast as 3 days ago.  :-)

> Excellent! Is "guix-package --roll-back" also faster now? It feels so!

C’m’on, this one was already fast, because all it does is to create a
symlink.

Ludo’.

Re: Faster!

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 267 bytes --]

Am Mittwoch, 30. Januar 2013 schrieb Ludovic Courtès:
> C’m’on, this one was already fast, because all it does is to create a
> symlink.

That is why I wondered. But when I rolled back 20 generations the other 
day, it did not feel fast at all.

Andreas

[-- Attachment #2: Type: text/html, Size: 1774 bytes --]

Re: Faster!

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Am Mittwoch, 30. Januar 2013 schrieb Ludovic Courtès:
>> C’m’on, this one was already fast, because all it does is to create a
>> symlink.
>
> That is why I wondered. But when I rolled back 20 generations the other 
> day, it did not feel fast at all.

Really?  That’s weird.  Building the empty profile can take a bit of
time the first time (a few seconds at most), but other than that, it
should be fast.

Ludo’.

Re: Faster!

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]

Am Mittwoch, 30. Januar 2013 schrieb Ludovic Courtès:
> Really?  That’s weird.  Building the empty profile can take a bit of
> time the first time (a few seconds at most), but other than that, it
> should be fast.

Well, it depends on your definition of fast. A second, I would say, which 
felt slow given that it was only a symlink. Maybe this was just the time 
for starting guix-package; did it become faster?

Andreas

[-- Attachment #2: Type: text/html, Size: 2084 bytes --]

Re: Faster!

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Am Mittwoch, 30. Januar 2013 schrieb Ludovic Courtès:
>> Really?  That’s weird.  Building the empty profile can take a bit of
>> time the first time (a few seconds at most), but other than that, it
>> should be fast.
>
> Well, it depends on your definition of fast. A second, I would say, which 
> felt slow given that it was only a symlink. Maybe this was just the time 
> for starting guix-package; did it become faster?

No, but it’s true that it’s surprisingly slow to start:

--8<---------------cut here---------------start------------->8---
$ time ./pre-inst-env guix-package

real    0m1.084s
user    0m0.904s
sys     0m0.058s

$ time ./pre-inst-env guix-build

real    0m0.101s
user    0m0.081s
sys     0m0.017s
--8<---------------cut here---------------end--------------->8---

Worth investigating...

Ludo’.

Re: Faster!

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> No, but it’s true that it’s surprisingly slow to start:
>
> $ time ./pre-inst-env guix-package
>
> real    0m1.084s
> user    0m0.904s
> sys     0m0.058s

This is actually the time it takes to connect to the daemon, and to
compute the derivation of Guile (which is then used to build the
profile, if a profile gets built.)

Queries (-I and -A) don’t need this, so I just rearranged the code
accordingly.  Both now take less than .2s on my laptop.

In the roll-back case, neither is necessary, except when rolling back to
generation 0 for the first time.  All other actions need it.  So I just
kept it for all the actions, including roll-back.

Thanks,
Ludo’.

Re: Faster!

[Nikita Karetnikov]

[-- Attachment #1: Type: text/plain, Size: 910 bytes --]

While we are at it, will it be better to rewrite 'fold-packages' and
similar things (like 'search') using streams?

For example:

  #:use-module (ice-9 streams)

[...]

(define (fold-packages-lazy proc init)
  "Call (PROC PACKAGE RESULT) for each available package, using INIT as
the initial value of RESULT."
  (stream-fold
   (lambda (module result)
     (stream-fold (lambda (var result)
                    (if (package? var)
                        (proc var result)
                        result))
                  result
                  (list->stream
                   (module-map (lambda (sym var)
                                 (false-if-exception (variable-ref var)))
                               module))))
   init
   (list->stream (package-modules))))

'time' shows that the above is a bit slower than the current version.
Will it be faster when we have more packages?  What about memory?

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Faster!

[Ludovic Courtès]

Nikita Karetnikov <nikita@karetnikov.org> skribis:

> While we are at it, will it be better to rewrite 'fold-packages' and
> similar things (like 'search') using streams?

Good question.

[...]

> 'time' shows that the above is a bit slower than the current version.
> Will it be faster when we have more packages?  What about memory?

It will necessarily use slightly more memory.

Performance depends on our use cases.  Currently, all our use cases
(finding packages by name or by description, listing them, etc.)
traverse the whole list.  Thus, it would not buy us anything to use
streams here.

Perhaps we can revisit that if/when we have an application where it
would help, but that doesn’t come to mind here.

WDYT?

Ludo’.

Re: Faster!

[Ludovic Courtès]

The saga continues!

With today’s commit, the execution time of

  guix build -e '(@ (gnu packages emacs) emacs)' -d

is down from 1.54s to 1.08s on my 2.6 GHz laptop.  That’s 30%.  :-)

The head of the profile is now like this:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,pr (package-derivation s emacs)
%     cumulative   self
time   seconds     seconds      name
 29.07      0.35      0.33  write
  8.14      0.09      0.09  search-path
  6.98      0.09      0.08  open-file
  5.81      0.07      0.07  read
  4.65      0.51      0.05  call-with-output-string
  4.65      0.05      0.05  %after-gc-thunk
  3.49      0.04      0.04  hash-ref
  2.33      0.24      0.03  write-derivation
  2.33      0.07      0.03  bytevector-quintet-fold-right
  2.33      0.03      0.03  #<procedure 23dc1c0 (_ _ _ _)>
  2.33      0.03      0.03  hash-set!
  2.33      0.03      0.03  display
[...]
  0.00      0.79      0.00  build-expression->derivation
[...]
---
Sample count: 86
Total time: 1.125903012 seconds (0.152722833 seconds in GC)
--8<---------------cut here---------------end--------------->8---

IOW, the main bottleneck is in Guile’s ‘write’ and port output (as
called from ‘build-expression->derivation’).

We also do a lot of GC.  This could be improved by optimizing the RPCs
in store.scm to allocate less.

Ludo’.