Running individual system tests

Running individual system tests

[Daniel Schäfer]

Hi Guix,

Guix System has nice system tests, as described in [1] but they don't 
easily allow to run a single test.

The manual says to run `make check-system TESTS="basic mcron"` in the 
guix source tree.
That, however, (apparently) requires to be bootstrapped, configured and 
all Scheme files to be fully built.

Is there something as simple as:

$ guix build -f gnu/tests/rsync.scm
guix build: error: #<unspecified>: not something we can build

---

I hacked together a solution that works but is far from nice.
It is based off of build-aux/run-system-tests.scm and I tried to strip 
it down to the core.

$ cat run-test.scm
(use-modules (gnu tests rsync)
              (gnu tests)
              (guix store)
              (guix status)
              (guix monads)
              (guix derivations)
              (guix ui)
              (srfi srfi-1)
              (srfi srfi-34))

(define (built-derivations* drv)
   (lambda (store)
     (guard (c ((store-protocol-error? c)
                (values #f store)))
       (values (build-derivations store drv) store))))

(system-test-value %test-rsync)
   (with-store store
     (with-status-verbosity 2
       (run-with-store store
         (mlet* %store-monad ((drv (mapm %store-monad system-test-value 
(list %test-rsync)))
                              (out -> (map derivation->output-path drv)))
           (mbegin %store-monad
             (show-what-to-build* drv)
             (set-build-options* #:keep-going? #t #:keep-failed? #t
                                 #:print-build-trace #t
                                 #:print-extended-build-trace? #t
                                 #:fallback? #t)
             (built-derivations* drv)
             (display out))))))

$ guile -s run-test.scm
(/gnu/store/scifaigbvcj20lplj3mvqj3ndr9w0nlv-rsync-test)Backtrace:
            8 (apply-smob/1 #<catch-closure 7353e0>)
In ice-9/boot-9.scm:
     705:2  7 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
     619:8  6 (_ #(#(#<directory (guile-user) 7cd140>)))
In ice-9/boot-9.scm:
    2312:4  5 (save-module-excursion _)
   3831:12  4 (_)
In /home/zoid/guix/gnu/tests/run-test.scm:
      19:4  3 (_)
In guix/status.scm:
     810:4  2 (call-with-status-report _ _)
In guix/store.scm:
   1737:24  1 (run-with-store _ _ #:guile-for-build _ #:system _ # _)
In unknown file:
            0 (_ #<store-connection 256.99 3b8dd20>)

ERROR: Wrong type to apply: #<unspecified>


It shows the derivation output path before the backtrace and thus I can 
inspect the log.
Can I do this in a nicer way?
It would be very helpful for out-of-tree system tests.

Thanks

- Daniel

[1] https://guix.gnu.org/blog/2016/guixsd-system-tests/

Fwd: Running individual system tests

[Daniel Schäfer]

[-- Attachment #1: Type: text/plain, Size: 3585 bytes --]

Hi Guix developers,

I'm still trying to figure out, how to run a single Guix system test 
without building the entirety of Guix.
Does anybody do this? Has anybody written a Guix system test for 
something not in Guix? Maybe a personal or work-related project.

CC'ing some people who've added guix system tests.

Thanks,
Daniel

-------- Forwarded Message --------
Subject: 	Running individual system tests
Date: 	Tue, 15 Oct 2019 13:21:45 +0200
From: 	Daniel Schäfer <daniel@danielschaefer.me>
To: 	help-guix@gnu.org



Hi Guix,

Guix System has nice system tests, as described in [1] but they don't 
easily allow to run a single test.

The manual says to run `make check-system TESTS="basic mcron"` in the 
guix source tree.
That, however, (apparently) requires to be bootstrapped, configured and 
all Scheme files to be fully built.

Is there something as simple as:

$ guix build -f gnu/tests/rsync.scm
guix build: error: #<unspecified>: not something we can build

---

I hacked together a solution that works but is far from nice.
It is based off of build-aux/run-system-tests.scm and I tried to strip 
it down to the core.

$ cat run-test.scm
(use-modules (gnu tests rsync)
              (gnu tests)
              (guix store)
              (guix status)
              (guix monads)
              (guix derivations)
              (guix ui)
              (srfi srfi-1)
              (srfi srfi-34))

(define (built-derivations* drv)
   (lambda (store)
     (guard (c ((store-protocol-error? c)
                (values #f store)))
       (values (build-derivations store drv) store))))

(system-test-value %test-rsync)
   (with-store store
     (with-status-verbosity 2
       (run-with-store store
         (mlet* %store-monad ((drv (mapm %store-monad system-test-value 
(list %test-rsync)))
                              (out -> (map derivation->output-path drv)))
           (mbegin %store-monad
             (show-what-to-build* drv)
             (set-build-options* #:keep-going? #t #:keep-failed? #t
                                 #:print-build-trace #t
                                 #:print-extended-build-trace? #t
                                 #:fallback? #t)
             (built-derivations* drv)
             (display out))))))

$ guile -s run-test.scm
(/gnu/store/scifaigbvcj20lplj3mvqj3ndr9w0nlv-rsync-test)Backtrace:
            8 (apply-smob/1 #<catch-closure 7353e0>)
In ice-9/boot-9.scm:
     705:2  7 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
     619:8  6 (_ #(#(#<directory (guile-user) 7cd140>)))
In ice-9/boot-9.scm:
    2312:4  5 (save-module-excursion _)
   3831:12  4 (_)
In /home/zoid/guix/gnu/tests/run-test.scm:
      19:4  3 (_)
In guix/status.scm:
     810:4  2 (call-with-status-report _ _)
In guix/store.scm:
   1737:24  1 (run-with-store _ _ #:guile-for-build _ #:system _ # _)
In unknown file:
            0 (_ #<store-connection 256.99 3b8dd20>)

ERROR: Wrong type to apply: #<unspecified>


It shows the derivation output path before the backtrace and thus I can 
inspect the log.
Can I do this in a nicer way?
It would be very helpful for out-of-tree system tests.

Thanks

- Daniel

[1] https://guix.gnu.org/blog/2016/guixsd-system-tests/



[-- Attachment #2: Type: text/html, Size: 5704 bytes --]

Re: Fwd: Running individual system tests

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

Hello Daniel,

Daniel Schäfer <git@danielschaefer.me> writes:

> I'm still trying to figure out, how to run a single Guix system test 
> without building the entirety of Guix.
> Does anybody do this? Has anybody written a Guix system test for 
> something not in Guix? Maybe a personal or work-related project.

You only have to build the entirety of Guix once.  Later invokations of
"make check-system" will re-use the compiled modules from the previous
"make" invokation, and only compile those that changed in the mean time.

Note that 'run-basic-test' also builds a derivation of your Guix
checkout to make it available in the VM.  I suspect this is what you are
referring to?

For out-of-tree tests, you can try to hack 'gnu/tests.scm' and
'build-aux/run-system-tests.scm' to your needs.

HTH!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Fwd: Running individual system tests

[Ricardo Wurmus]

Hi Daniel,

I don’t understand why you would want to run a system test without
building Guix first.

Could you elaborate?

-- 
Ricardo

Re: Fwd: Running individual system tests

[Daniel Schäfer]

Hi Ricardo,

For a test of an external system, like something I develop at work, I 
don't want/need to have Guix with all modules compiled present.
I don't want unnecessary steps if they are not necessary.

Please refer to my first mail with an example of running a test standalone.
I'd like some assistance to make my snippet nicer and not return an 
error after successfully running the test.

Thanks,
Daniel

On 10/19/19 7:35 PM, Ricardo Wurmus wrote:
> Hi Daniel,
>
> I don’t understand why you would want to run a system test without
> building Guix first.
>
> Could you elaborate?
>

Re: Fwd: Running individual system tests

[Daniel Schäfer]

So you're saying, I have to run

```
./bootstrap
./configure
make
./pre-inst-env guix build my-new-package
```

which builds the daemon, and compiles all Guile modules?
And I have to do it, whenever I want to run any system test, or build an 
in-tree package (which I'm going to submit for inclusion)?
Or more precisely: It has to be run, once, before I do those things. 
Question: Doesn't it ever have to be rerun? How are new modules compiled?

It takes 33 minutes on my laptop and is only going to take longer, when 
Guix gets bigger.
Is there maybe an interpreted mode? Or could this be better with Guile 
3.0 JIT?

- Daniel

On 10/21/19 8:43 AM, Daniel Schäfer wrote:
> Hi Ricardo,
>
> For a test of an external system, like something I develop at work, I 
> don't want/need to have Guix with all modules compiled present.
> I don't want unnecessary steps if they are not necessary.
>
> Please refer to my first mail with an example of running a test 
> standalone.
> I'd like some assistance to make my snippet nicer and not return an 
> error after successfully running the test.
>
> Thanks,
> Daniel
>
> On 10/19/19 7:35 PM, Ricardo Wurmus wrote:
>> Hi Daniel,
>>
>> I don’t understand why you would want to run a system test without
>> building Guix first.
>>
>> Could you elaborate?
>>

Re: Running individual system tests

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1490 bytes --]

Hi,

On Wed, 23 Oct 2019 14:03:20 +0200
Daniel Schäfer <git@danielschaefer.me> wrote:

> And I have to do it, whenever I want to run any system test, or build an 
> in-tree package (which I'm going to submit for inclusion)?

Yes, in the end you have to do that.  I agree that that's a very high
bar of entry (for example compared to Gentoo ebuilds).  But yes.

> Or more precisely: It has to be run, once, before I do those things. 
> Question: Doesn't it ever have to be rerun? How are new modules compiled?

Yes.  Ideally (in order to have a consistent state), delete the entire git
clone of guix, then clone guix again and do it all over again.

In practice I only invoke "make -j5" again.
If that doesn't work, I invoke "./configure --localstatedir=/var" again
and then "make -j5".
This will incrementally compile new things so it will be quite fast.

If the version of Guile got swapped before the second run of that stuff, the
final result will be Frankenstein's monster, though, so I'm under no illusion
that that's safe.

> It takes 33 minutes on my laptop and is only going to take longer, when 
> Guix gets bigger.

-j5

> Is there maybe an interpreted mode?

There is, but it is buggy.  Ideally, nobody has to run "make" before
any of this stuff (Guile can just interpret the scm files) but there's
a strange bug where Guile then will use the wrong files (probably old
compiled files from ~/.cache) if you do that, so I never do it anymore.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Fwd: Running individual system tests

[Ricardo Wurmus]

Daniel Schäfer <git@danielschaefer.me> writes:

> Is there maybe an interpreted mode?

Yes, if you don’t compile the Scheme files they will be interpreted.
It’s much slower than using the compiled Guix.

--
Ricardo

Re: Fwd: Running individual system tests

[Ludovic Courtès]

Hi Daniel,

Daniel Schäfer <git@danielschaefer.me> skribis:

> I hacked together a solution that works but is far from nice.
> It is based off of build-aux/run-system-tests.scm and I tried to strip
> it down to the core.

There’s a simpler solution:

--8<---------------cut here---------------start------------->8---
$ cat /tmp/test.scm
(use-modules (gnu tests) (gnu tests rsync))

(lambda ()
  (system-test-value %test-rsync))
$ guix build -f /tmp/test.scm 
…
--8<---------------cut here---------------end--------------->8---

However, note that the (gnu tests …) API could change anytime.  In
particular, I’d like to avoid monadic style there.

HTH!

Ludo’.