Performance of computing cross derivations

Performance of computing cross derivations

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2740 bytes --]

Hey!

When asked by the data service, it seems to take Guix around 3 minutes
to compute cross derivations for all packages (to a single
target). Here's a simple script that replicates this:

  (use-modules (srfi srfi-34)
               (gnu packages)
               (guix grafts)
               (guix packages)
               (guix store)
               (statprof))
  
  (define (all-cross system target)
    (with-store store
      (%graft? #f)
      (fold-packages
       (lambda (package result)
         (with-exception-handler
             (lambda (exn)
               (unless (package-cross-build-system-error? exn)
                 (peek exn))
               result)
           (lambda ()
             (package-cross-derivation store
                                       package
                                       target
                                       system)
             (+ 1 result))
           #:unwind? #t))
       0)))
  
  (statprof
   (lambda ()
     (peek "COUNT"
           (all-cross "x86_64-linux"
                      "i586-pc-gnu")))
   #:count-calls? #t)


Here's some relevant output:

  %     cumulative   self             
  time   seconds    seconds   calls   procedure
   50.48    126.68    102.40                            ice-9/vlist.scm:502:0:vhash-foldq*
   11.49     23.31     23.31                            hashq
    5.16     10.52     10.47                            write
    2.79     14.28      5.65                            ice-9/vlist.scm:494:0:vhash-fold*
    2.28      4.63      4.63                            equal?
    2.14      4.35      4.35                            hash
    1.85      4.67      3.75                            guix/packages.scm:1874:0:input=?
    1.78      3.68      3.61                            put-string
    1.77      7.16      3.59                            guix/derivations.scm:736:0:derivation/masked-inputs
    0.93      1.90      1.90                            get-bytevector-n
    0.78      1.58      1.58                            put-char
    0.67      1.36      1.36                            search-path

  ...
  
  Total time: 202.872232073 seconds (30.927648399 seconds in GC)


Over 3 minutes seems like a long time for this, especially since it only
computes around 10000 derivations.

I don't know how to use statprof, but looking at vhash-foldq* being at
the top of the output, is this suggesting that around a third of the CPU
time is being spent looking for things in various caches?

I had a go at using the Guix profiling stuff and I did get some output,
but I couldn't figure out how to get it to show all the caching going
on.

Any ideas?

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Performance of computing cross derivations

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> When asked by the data service, it seems to take Guix around 3 minutes
> to compute cross derivations for all packages (to a single
> target). Here's a simple script that replicates this:

To understand the cost of computing a package’s derivation, I generally
start looking at caches and memoization:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING="object-cache" guix build gcc-toolchain -d --no-grafts
/gnu/store/iwn6frqqcyw808sgsnjv26dn6rq7mijd-gcc-toolchain-13.2.0.drv
Object Cache:
  fresh caches:    19
  lookups:       3667
  hits:          3342 (91.1%)
  cache size:     323 entries
$ GUIX_PROFILING="object-cache" guix build sed -d --no-grafts --target=aarch64-linux-gnu
/gnu/store/yxakl87wizwzcqapx4sdkp56652cxb4m-sed-4.8.drv
Object Cache:
  fresh caches:    20
  lookups:       5420
  hits:          4919 (90.8%)
  cache size:     500 entries
--8<---------------cut here---------------end--------------->8---

Caches are critical: since we’re dealing with huge package graphs, we
need to make sure we don’t end up computing the same thing several
times.  (You can also add “memoization” to the ‘GUIX_PROFILING’ variable
above.)

One idiom that defeats caching is:

  (define (make-me-a-package x y z)
    (package
      …))

Such a procedure returns a fresh package every time it’s called,
preventing caching from happening (because cache entries are compared
with ‘eq?’).  That typically leads to lower hit rates.

Anyway, lots of words to say that I don’t see anything immediately
obvious with cross-compilation, yet I wouldn’t be surprised if some of
these cache-defeating idioms were used because we’ve payed less
attention to this.

An even better thing to start with: compare the timing of ‘guix build -d
--no-grafts $PKG --target=aarch64-linux-gnu’ for all valid values of
$PKG, and investigate those that take the most time.

HTH!

Ludo’.