bug#44390: Cuirass reports succesful builds as failed in /var/log/cuirass.log and cuirass-web reports them as scheduled

bug#44390: Cuirass reports succesful builds as failed in /var/log/cuirass.log and cuirass-web reports them as scheduled

[david larsson]

Hi!

In /var/log/cuirass.log:

2020-11-02T15:41:34 build succeeded: 
'/gnu/store/aqn31rymjs3qybsd1d2ssn93f8s0jlz5-git-checkout.drv'
2020-11-02T15:41:35 build started: 
'/gnu/store/q7q6s24h05kxib5wvjmnn5zmy6vh3m63-git-checkout.drv'
2020-11-02T15:41:39 build succeeded: 
'/gnu/store/q7q6s24h05kxib5wvjmnn5zmy6vh3m63-git-checkout.drv'
2020-11-02T15:41:39 build started: 
'/gnu/store/hwkdx9qabrp8q3anc1d0brnxfr7xdj2n-orgmk-v0.0.2.drv'
2020-11-02T15:41:40 build succeeded: 
'/gnu/store/hwkdx9qabrp8q3anc1d0brnxfr7xdj2n-orgmk-v0.0.2.drv'
2020-11-02T15:41:41 build started: 
'/gnu/store/2lxjrb5sfi0mxj1871zb5g47nkg76j6z-fwknop-3.0.0.drv'
2020-11-02T15:42:19 build succeeded: 
'/gnu/store/2lxjrb5sfi0mxj1871zb5g47nkg76j6z-fwknop-3.0.0.drv'
2020-11-02T15:42:20 done with 10 derivations
2020-11-02T15:42:20 outputs:
/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10
/gnu/store/p02qaingwm7hmlax4zh9af8d72jivdnr-cowsay-3.04
/gnu/store/bj30r24xjv55nh1lx0r24psrlrm5haif-orgmk-v0.0.2
/gnu/store/nbknii5yq7djc6vg90kay7jiplk65n2i-fwknop-3.0.0
/gnu/store/bprgycch5x2bh8gch32nksw9cvrii2lc-python-on-guile-1.2.3.5
/gnu/store/97aggnqbar8gzwfqqxf8mk10rpia9244-libxml2-xpath0-1.1.1-0.0.0.6c7a378
/gnu/store/9bszaplxdxg3l6wid9l7lf6vqnnmkyic-libxml2-xpath0-1.1.1-0.0.0.6c7a378-static
/gnu/store/8nw4nz1l72yxalni2xnfdna2qpxr87r8-guile-bash-parallel-v0.0.7
/gnu/store/cqdqaa0nsvjqv7003sfkfiv9b8r2080k-guile-base64-v0.0.1
/gnu/store/1sq6cnpnpzpxmv9nkpfvim9wild6c303-perl-data-ical-0.24
/gnu/store/4ijsrmkb02z2s18bgkw5kyvf4kpwsdyg-python-pydotplus-2.0.2
2020-11-02T15:42:20 success: 0, fail: 10
2020-11-02T15:42:34 heap: 44.49 MiB; threads: 11; file descriptors: 4



And in the web interface they are reported as "Scheduled":

Input	Commit
git://git.savannah.gnu.org/guix.git	28e20c53530671ece8421ccb4ac815fd1beaf9fe
file:///home/cuirass/my-guix-packages	9f9c644076b088aabee76626598115c0fe6bfbde
Pending builds of evaluation #1

All 10
  Scheduled 10
  Succeeded 0
  Failed 0
ID	Specification	Completion time	Job	Name	System	Log
10	my-pkgs	—	python-pydotplus-2.0.2.x86_64-linux	python-pydotplus-2.0.2	x86_64-linux	—
9	my-pkgs	—	perl-data-ical-0.24.x86_64-linux	perl-data-ical-0.24	x86_64-linux	—
8	my-pkgs	—	guile-base64-v0.0.1.x86_64-linux	guile-base64-v0.0.1	x86_64-linux	—
7	my-pkgs	—	guile-bash-parallel-v0.0.7.x86_64-linux	guile-bash-parallel-v0.0.7	x86_64-linux	—
6	my-pkgs	—	libxml2-xpath0-1.1.1-0.0.0.6c7a378.x86_64-linux	libxml2-xpath0-1.1.1-0.0.0.6c7a378	x86_64-linux	—
5	my-pkgs	—	python-on-guile-1.2.3.5.x86_64-linux	python-on-guile-1.2.3.5	x86_64-linux	—
4	my-pkgs	—	fwknop-3.0.0.x86_64-linux	fwknop-3.0.0	x86_64-linux	—
3	my-pkgs	—	orgmk-v0.0.2.x86_64-linux	orgmk-v0.0.2	x86_64-linux	—
2	my-pkgs	—	cowsay-3.04.x86_64-linux	cowsay-3.04	x86_64-linux	—
1	my-pkgs	—	hello-2.10.x86_64-linux	hello-2.10	x86_64-linux	—


Cuirass config:

(define %cuirass-specs
   #~(
      ;; spec two
      list
   '((#:name . "my-pkgs")
     (#:load-path-inputs . ("guix"))
     ;;(#:package-path-inputs . ("my-guixsd-config.sh"))
     (#:package-path-inputs . ("my-guix-packages"))
     ;;(#:package-path-inputs . ())
     ;;(#:package-path-inputs . ())
     (#:proc-input . "guix")
     (#:proc-file . "build-aux/cuirass/gnu-system.scm")
     (#:proc . cuirass-jobs)
     (#:proc-args .
      (;;(subset . manifests)
       ;;(subset . ("hello" "cowsay" "perl-moose" "jupyter" 
"enlightenment"))
       (subset . ("hello" "cowsay" "orgmk" "fwknop" "python-on-guile" 
"libxml2-xpath0" "guile-bash-parallel" "guile-base64" "perl-data-ical" 
"python-pydotplus"))
       (systems . ("x86_64-linux"))
       ;;(manifests . (("my-guix-packages" . "manifest.scm")))
       ))
     (#:inputs . (
                  ((#:name . "guix")
                   (#:url . "git://git.savannah.gnu.org/guix.git")
                   (#:load-path . ".")
                   (#:branch . "master")
                   (#:no-compile? . #t))

                  ((#:name . "my-guix-packages")
                   (#:url . "file:///home/cuirass/my-guix-packages")
                   (#:load-path . "packages")
                   (#:branch . "master")
                   (#:no-compile? . #t))

                  ;; ((#:name . "my-guix-packages")
                  ;;  (#:url . 
"https://github.com/methuselah-0/my-guix-packages.git")
                  ;;  (#:load-path . "packages")
                  ;;  (#:branch . "master")
                  ;;  (#:no-compile? . #t))

                  ))
     (#:build-outputs . ()))

))


Best regards,
David

bug#44390: Cuirass reports succesful builds as failed in /var/log/cuirass.log and cuirass-web reports them as scheduled

[Mathieu Othacehe]

Hello David,

> /gnu/store/4ijsrmkb02z2s18bgkw5kyvf4kpwsdyg-python-pydotplus-2.0.2
> 2020-11-02T15:42:20 success: 0, fail: 10
> 2020-11-02T15:42:34 heap: 44.49 MiB; threads: 11; file descriptors: 4

Thanks for the bug report. This bug is the result of some recent
optimizations. The database requests updating the build statuses are
queued and issues by batches of 100 requests.

As you only have 10 builds, those requests are only queued. That's why
the web interface reports them as "scheduled" while the log shows that
they were built.

The "fail: 10" is also a consequence of that problem. Forcing the
pending queries once a batch of builds is done could possibly solve this
problem but I need to think more about it.

Thanks,

Mathieu

bug#44390: Cuirass reports succesful builds as failed in /var/log/cuirass.log and cuirass-web reports them as scheduled

[Jonathan Brielmaier]

Nice bug report David! I encounter the same problem :P

@Mathieu: Would it be an option to just make it configurable? It seems
that it makes a significant performance impact on larger cuirass
servers. But small servers still could set it to 1 or so...

bug#44390: Cuirass reports succesful builds as failed in /var/log/cuirass.log and cuirass-web reports them as scheduled

[Jonathan Brielmaier]

In the mean time I did a quick fix on my system in config.scm:

```
(use-modules (gnu)

             (gnu packages ci) (guix packages) (guix utils))


(define-public cuirass/queue-1

  (package/inherit cuirass

    (name "cuirass-queue-1")

    (arguments

     (substitute-keyword-arguments (package-arguments cuirass)

       ((#:phases phases)

        `(modify-phases ,phases

           (add-after 'disable-repo-tests 'set-queue-size

             (lambda _

               (substitute* "src/cuirass/database.scm"

                 (("queue-size 100") "queue-size 1"))

               #t))))))))

```

Plus (cuirass cuirass/queue-1)
 in the cuirass-configuration. So no need from my side to hurry anmyore :)

bug#44390: Cuirass reports succesful builds as failed in /var/log/cuirass.log and cuirass-web reports them as scheduled

[david larsson]

On 2020-11-06 15:07, Jonathan Brielmaier wrote:
> In the mean time I did a quick fix on my system in config.scm:
> 
> ```
> (use-modules (gnu)
> 
>              (gnu packages ci) (guix packages) (guix utils))
> 
> 
> (define-public cuirass/queue-1
> 
>   (package/inherit cuirass
> 
>     (name "cuirass-queue-1")
> 
>     (arguments
> 
>      (substitute-keyword-arguments (package-arguments cuirass)
> 
>        ((#:phases phases)
> 
>         `(modify-phases ,phases
> 
>            (add-after 'disable-repo-tests 'set-queue-size
> 
>              (lambda _
> 
>                (substitute* "src/cuirass/database.scm"
> 
>                  (("queue-size 100") "queue-size 1"))
> 
>                #t))))))))
> 
> ```
> 
> Plus (cuirass cuirass/queue-1)
>  in the cuirass-configuration. So no need from my side to hurry anmyore 
> :)

Thank you for this. It works well!

Best regards,
David

bug#44390: Cuirass reports succesful builds as failed in /var/log/cuirass.log and cuirass-web reports them as scheduled

[Mathieu Othacehe]

Hello,

Cuirass commit 697fa14584551d9595cd042f1ffeba240e45a127 adds a
"--queue-size" argument.

With Guix commit 770fb659a62d8adab9a4ba7b44e2760722703879, Cuirass is
passed the "queue-size" argument, defaulting to 1.

Thanks,

Mathieu

bug#44390: Cuirass reports succesful builds as failed in /var/log/cuirass.log and cuirass-web reports them as scheduled

[Jonathan Brielmaier]

On 08.12.20 12:09, Mathieu Othacehe wrote:
>
> Hello,
>
> Cuirass commit 697fa14584551d9595cd042f1ffeba240e45a127 adds a
> "--queue-size" argument.
>
> With Guix commit 770fb659a62d8adab9a4ba7b44e2760722703879, Cuirass is
> passed the "queue-size" argument, defaulting to 1.

Fix confirmed on my personal Cuirass server :)

Merci Mathieu!