Re: [PATCH] git-download: Speed up 'git-predicate'.

[Christopher Baines <mail@cbaines.net>]

[-- Attachment #1.1: Type: text/plain, Size: 2295 bytes --]

To provide some anecdotal evidence on the difference this makes, I wrote
a little test script [1]. It should roughly follow how git-predicate is
used with the local-file gexp.

Running the current implementation of git-predicate on the guix
repository with 50 trials takes:

  real    0m5.374s
  user    0m4.052s
  sys     0m1.484s

Compared to the following for the patched implementation:

  real    0m8.398s
  user    0m5.040s
  sys     0m2.016s

This shows that the patched implementation has decreased the
performance. Using the real value, the test time has increased by ~3
seconds. This corresponds roughly to a change from ~0.11 seconds
previously, to ~0.17 seconds for each individual traversal of the entire
repository.

I encountered a performance issue when trying to use git-predicate with
this repository [2], which I have written a Guix package for here [3].
Instead of the 1463 files that git ls-files reports locally for the guix
repository, smart-answers contains 26732 files.

The running time with smart-answers is left as an exercise to the
reader, all I can say is that on my machine, it takes more than 40
minutes for just 1 trial.

Using this patch, running the test script with 1 trial gives:

  real    0m4.917s
  user    0m3.640s
  sys     0m1.428s

1:

(use-modules (srfi srfi-1)
             (srfi srfi-26)
             (ice-9 ftw)
             (guix git-download))

(define (test file select?)
  (let dump ((f file) (s (lstat file)))
    (case (stat:type s)
      ((regular)
       (display "."))
      ((directory)
       (for-each
        (lambda (e)
          (let* ((f (string-append f "/" e))
                 (s (lstat f)))
            (if (select? f s)
                (dump f s))))
        (scandir f (negate (cut member <> '("." ".."))) string<?)))
      ((symlink)
       (display ".")))))

(define (run-test dir trials)
  (chdir dir)
  (for-each
   (lambda (trial)
     (test dir (git-predicate dir)))
   (iota trials)))

(run-test (second (command-line))
          (string->number (third (command-line))))


2: https://github.com/alphagov/smart-answers
3:
https://github.com/alphagov/govuk-guix/blob/65c6b8f3a0f01cd6ae4b51f356b74d4472b08e70/gds/packages/govuk.scm#L1136-L1153


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 858 bytes --]