Problems running 'check-system'

Problems running 'check-system'

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 9708 bytes --]

Hi,

I'm having trouble running the system tests.  There are three issues:

Issue #1: I've noticed that I have to run 'make' before running 'make
check-system'.  An example will illustrate what I mean.

The following invocation fails in a clean checkout of master (in an
environment created via 'guix environment --pure guix'):

  ./bootstrap
  ./configure --localstatedir=/var
  make -j check-system TESTS=installed-os

It fails with this error message:

--8<---------------cut here---------------start------------->8---
[... some output omitted for brevity ...]

GUILEC   guix/scripts/copy.go
  GUILEC   guix/store/ssh.go
  GUILEC   guix/scripts/offload.go
  GUILEC   guix/config.go
  GUILEC   guix/tests.go
  GUILEC   guix/tests/http.go
;;; note: source file /home/marusich/guix/./build-aux/run-system-tests.scm
;;;       newer than compiled /home/marusich/.cache/guile/ccache/2.0-LE-8-2.0/home/marusich/guix/build-aux/run-system-tests.scm.go
In execvp of git: No such file or directory
Running 1 system tests...
Backtrace:
In ice-9/boot-9.scm:
  66: 19 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
 432: 18 [eval # #]
In ice-9/r4rs.scm:
  90: 17 [dynamic-wind #<procedure 3816e20 at ice-9/boot-9.scm:1143:2 _> ...]
In guix/store.scm:
1332: 16 [run-with-store # #<procedure 42f0d50 at ice-9/eval.scm:416:20 (a)> # ...]
In ice-9/r4rs.scm:
  39: 15 [call-with-values #<procedure 42f0ae0 at ice-9/eval.scm:416:20 ()> ...]
In guix/monads.scm:
 376: 14 [#<procedure 42f0db0 at guix/monads.scm:373:2 (state)> #]
 376: 13 [#<procedure 42f0de0 at guix/monads.scm:373:2 (state)> #]
 376: 12 [#<procedure 42f0e40 at guix/monads.scm:373:2 (state)> #]
In gnu/tests/install.scm:
 230: 11 [#<procedure 389bca0 at gnu/tests/install.scm:230:4 (state)> #]
 142: 10 [#<procedure 48017e0 at gnu/tests/install.scm:142:2 (state)> #]
In gnu/services.scm:
 268: 9 [#<procedure 4801810 at gnu/services.scm:268:2 (state)> #]
In gnu/system.scm:
 275: 8 [#<procedure 4327240 at gnu/system.scm:275:2 (state)> #]
In guix/gexp.scm:
 604: 7 [#<procedure 480b900 at guix/gexp.scm:604:2 (state)> #]
 487: 6 [#<procedure 47ff680 at guix/gexp.scm:487:4 (state)> #]
 490: 5 [#<procedure 4803600 at guix/gexp.scm:490:22 (state)> #]
In guix/store.scm:
1252: 4 [#<procedure 4803630 at guix/store.scm:1251:28 (store)> #]
In guix/packages.scm:
 806: 3 [cache! #<weak-key-hash-table 1cc0dc0 0/113> # # ...]
1121: 2 [thunk]
1053: 1 [bag->derivation # # #]
In srfi/srfi-1.scm:
 575: 0 [map #<procedure 480c570 at guix/packages.scm:1055:30 (t-9e7228c7bc8-578a)> ...]

srfi/srfi-1.scm:575:27: In procedure map:
srfi/srfi-1.scm:575:27: Throw to key `srfi-34' with args `(#<condition &message [message: "could not find bootstrap binary 'guile-2.0.9.tar.xz' for system 'x86_64-linux'"] 4b62d80>)'.
make: *** [Makefile:5066: check-system] Error 1
[2] [env] marusich@garuda:~/guix
$ 
--8<---------------cut here---------------end--------------->8---

However, if I invoke 'make' first, then the "could not find bootstrap
binary" message does not show up, and the test proceeds to be run:

  ./bootstrap
  ./configure --localstatedir=/var
  make -j
  make -j check-system TESTS=installed-os

Is this expected?  I thought one of the benefits of using different
targets in a Makefile was so that you could invoke a specific target
(e.g., 'check-system') and only build the things necessary to run that
target.  It surprised me that I needed to also run the default target
first in this case.

Issue #2: even when I run 'make' first, the test fails.  It fails with
this message:

--8<---------------cut here---------------start------------->8---
[... some output omitted for brevity ...]

starting phase `copy-bootstrap-guile'
Backtrace:
In ice-9/boot-9.scm:
 160: 13 [catch #t #<catch-closure 8c5dc0> ...]
In unknown file:
   ?: 12 [apply-smob/1 #<catch-closure 8c5dc0>]
In ice-9/boot-9.scm:
  66: 11 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
 432: 10 [eval # #]
In ice-9/boot-9.scm:
2412: 9 [save-module-excursion #<procedure 8e6840 at ice-9/boot-9.scm:4084:3 ()>]
4089: 8 [#<procedure 8e6840 at ice-9/boot-9.scm:4084:3 ()>]
1734: 7 [%start-stack load-stack #<procedure 8f6e20 at ice-9/boot-9.scm:4080:10 ()>]
1739: 6 [#<procedure 8f8960 ()>]
In unknown file:
   ?: 5 [primitive-load "/gnu/store/wgh83kqjif20wfdg56iz7bxk9d4xmlk0-guix-0.12.0-9.25a4+-guile-builder"]
In ice-9/eval.scm:
 387: 4 [eval # ()]
In srfi/srfi-1.scm:
 827: 3 [every1 #<procedure f9fa40 at /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:649:9 (expr)> ...]
In /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:
 653: 2 [#<procedure f9fa40 at /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:649:9 (expr)> #]
In ice-9/eval.scm:
 432: 1 [eval # #]
In unknown file:
   ?: 0 [copy-file "/gnu/store/dgncc5wmw8prxq09y71hqjc6g7rxqvvb-guile-2.0.9.tar.xz" ...]

ERROR: In procedure copy-file:
ERROR: In procedure copy-file: Permission denied
note: keeping build directory `/tmp/guix-build-guix-0.12.0-9.25a4+.drv-1'
builder for `/gnu/store/hm9bfka04gjbzjah7ilk3hp998bzdvqv-guix-0.12.0-9.25a4+.drv' failed with exit code 1
@ build-failed /gnu/store/hm9bfka04gjbzjah7ilk3hp998bzdvqv-guix-0.12.0-9.25a4+.drv - 1 builder for `/gnu/store/hm9bfka04gjbzjah7ilk3hp998bzdvqv-guix-0.12.0-9.25a4+.drv' failed with exit code 1
cannot build derivation `/gnu/store/1wc6ajd6mazb084ddl2mkpzcnkn42x2n-activate-service.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/h1aw5zf5z7n5f51xa4dm0hz5b5n75gdm-ca-certificate-bundle.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/lqkb3fxvqq7adakpgy5j4n6v05ycvd3n-fonts-dir.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/xvjycblb25lyd80mpxhf4azikzjz74bv-info-dir.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/n1dv7my6awklaplsk1j3j34dx5gqh3s1-manual-database.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/khj9qjc52vgqmqkrwmmhr5iqqnz5h19g-shepherd-guix-daemon.scm.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/azw92b3nf95gyqwx2lijzmqka43xfpn9-activate.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/91mp0bgpknv9i0x426m9w76pz4026qvn-profile.drv': 5 dependencies couldn't be built
cannot build derivation `/gnu/store/sl8mknzf3wsdcjjlp8w3zv1ng1s1kwhx-shepherd.conf.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/318yazn1vzp3814nny0x65jva7bn4ck7-boot.drv': 2 dependencies couldn't be built
cannot build derivation `/gnu/store/igcd9wffmvnvyigjkxs2g277yfjahy8p-system.drv': 2 dependencies couldn't be built
cannot build derivation `/gnu/store/i34d0nscnns16amzkir0lly8763dhqdr-grub.cfg.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/c2vwrvygc5bnk1lh21qc4065m5ac9yx3-builder-in-linux-vm.drv': 2 dependencies couldn't be built
cannot build derivation `/gnu/store/yfxmvgis6l03543c7yyqsq4awv1hl7m8-linux-vm-loader.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/s4c5n2jf753vqk0bxmidjs4wxvzrjngr-disk-image.drv': 3 dependencies couldn't be built
cannot build derivation `/gnu/store/lji1bz94mgqd3v3cjb3h1pgbl6d23glv-installation.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/d074m8w00lgglwh84lrj7rasm703viw3-installed-os.drv': 1 dependencies couldn't be built
TOTAL: 1
FAIL: /gnu/store/cjqs0mc6vpkgzx7r7x24hz4r5hyqgink-installed-os
make: *** [Makefile:5066: check-system] Error 1
[2] [env] marusich@garuda:~/guix
$
--8<---------------cut here---------------end--------------->8---

Any idea why that's failing?  The file
/gnu/store/dgncc5wmw8prxq09y71hqjc6g7rxqvvb-guile-2.0.9.tar.xz is
readable by all users.  I tried to insert a "pk" in the file
guix/build/gnu-build-system.scm to see what the target of the copy
command was, but after about 24 hours (!!) of waiting for the subsequent
build to finish (for some reason this single-line change caused many
things to be rebuilt), I found that the build failed for a different,
apparently unrelated reason.  Quite unfortunate.

This brings me to Issue #3: the system tests appear to be unreliable and
impractical for day-to-day development.  Let me be clear: I really,
really want to like these system tests.  It's fantastic that the entire
system, its installation process, and other features can be tested
automatically.  I was hoping to hack around with them and eventually add
a system test for booting and installing from an ISO-9669 image.  But
with an iteration time of over 24 hours (like above) for a single-line
change, I don't know how helpful these tests really are for day-to-day
development.  I feel like I must be missing some kind of trick here to
speed up the iteration time.  Does anyone have any tips on how to speed
it up?  Maybe I'm missing something that's obvious to everyone else.

Finally, it seems like the system tests are not being run automatically
during the build of the Guix package, so I wonder if they're ever being
run by anyone at all.  Every time I've tried to run them recently,
they've failed, and my attempts to troubleshoot are frustrated by the
incredibly slow iteration time.  I have yet to find a commit on which
the system tests succeed, including the commit tagged for the v0.12.0
release.  I was hoping to find one so I could automatically bisect to
find out which commit introduced the I saw above, but at this rate I
doubt I'll be able to find one soon.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Problems running 'check-system'

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

>   make -j check-system TESTS=installed-os

[...]

> srfi/srfi-1.scm:575:27: Throw to key `srfi-34' with args `(#<condition &message [message: "could not find bootstrap binary 'guile-2.0.9.tar.xz' for system 'x86_64-linux'"] 4b62d80>)'.
> make: *** [Makefile:5066: check-system] Error 1
> [2] [env] marusich@garuda:~/guix
> $ 
>
>
> However, if I invoke 'make' first, then the "could not find bootstrap
> binary" message does not show up, and the test proceeds to be run:

That’s not surprising: ‘make’ triggers a download of guile-2.0.9.tar.xz
in gnu/packages/bootstrap.  See Makefile.am.

However, ‘check-system’ should depend on this target.  Fixed in
693f12ce2326f82020e90e58e69cf2b54808c19b.

> Issue #2: even when I run 'make' first, the test fails.  It fails with
> this message:
>
> [... some output omitted for brevity ...]
>
> starting phase `copy-bootstrap-guile'
> Backtrace:
> In ice-9/boot-9.scm:
>  160: 13 [catch #t #<catch-closure 8c5dc0> ...]
> In unknown file:
>    ?: 12 [apply-smob/1 #<catch-closure 8c5dc0>]
> In ice-9/boot-9.scm:
>   66: 11 [call-with-prompt prompt0 ...]
> In ice-9/eval.scm:
>  432: 10 [eval # #]
> In ice-9/boot-9.scm:
> 2412: 9 [save-module-excursion #<procedure 8e6840 at ice-9/boot-9.scm:4084:3 ()>]
> 4089: 8 [#<procedure 8e6840 at ice-9/boot-9.scm:4084:3 ()>]
> 1734: 7 [%start-stack load-stack #<procedure 8f6e20 at ice-9/boot-9.scm:4080:10 ()>]
> 1739: 6 [#<procedure 8f8960 ()>]
> In unknown file:
>    ?: 5 [primitive-load "/gnu/store/wgh83kqjif20wfdg56iz7bxk9d4xmlk0-guix-0.12.0-9.25a4+-guile-builder"]
> In ice-9/eval.scm:
>  387: 4 [eval # ()]
> In srfi/srfi-1.scm:
>  827: 3 [every1 #<procedure f9fa40 at /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:649:9 (expr)> ...]
> In /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:
>  653: 2 [#<procedure f9fa40 at /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:649:9 (expr)> #]
> In ice-9/eval.scm:
>  432: 1 [eval # #]
> In unknown file:
>    ?: 0 [copy-file "/gnu/store/dgncc5wmw8prxq09y71hqjc6g7rxqvvb-guile-2.0.9.tar.xz" ...]
>
> ERROR: In procedure copy-file:
> ERROR: In procedure copy-file: Permission denied

What this means is that the target of ‘copy-file’ is read-only.

> Any idea why that's failing?  The file
> /gnu/store/dgncc5wmw8prxq09y71hqjc6g7rxqvvb-guile-2.0.9.tar.xz is
> readable by all users.  I tried to insert a "pk" in the file
> guix/build/gnu-build-system.scm to see what the target of the copy
> command was, but after about 24 hours (!!) of waiting for the subsequent
> build to finish (for some reason this single-line change caused many
> things to be rebuilt), I found that the build failed for a different,
> apparently unrelated reason.  Quite unfortunate.

(guix build gnu-build-system) is used by everything, which is why
modifying it triggers a rebuild of everything.

> This brings me to Issue #3: the system tests appear to be unreliable and
> impractical for day-to-day development.  Let me be clear: I really,
> really want to like these system tests.  It's fantastic that the entire
> system, its installation process, and other features can be tested
> automatically.  I was hoping to hack around with them and eventually add
> a system test for booting and installing from an ISO-9669 image.  But
> with an iteration time of over 24 hours (like above) for a single-line
> change, I don't know how helpful these tests really are for day-to-day
> development.  I feel like I must be missing some kind of trick here to
> speed up the iteration time.  Does anyone have any tips on how to speed
> it up?  Maybe I'm missing something that's obvious to everyone else.

No, I think you stumbled upon a bunch of rough edges, and I can
understand your frustration.

> Finally, it seems like the system tests are not being run automatically
> during the build of the Guix package, so I wonder if they're ever being
> run by anyone at all.  Every time I've tried to run them recently,
> they've failed, and my attempts to troubleshoot are frustrated by the
> incredibly slow iteration time.  I have yet to find a commit on which
> the system tests succeed, including the commit tagged for the v0.12.0
> release.  I was hoping to find one so I could automatically bisect to
> find out which commit introduced the I saw above, but at this rate I
> doubt I'll be able to find one soon.

The system tests are being run on Hydra at each commit:

  https://hydra.gnu.org/job/gnu/master/test.installed-os.x86_64-linux
  https://hydra.gnu.org/job/gnu/master/test.encrypted-root-os.i686-linux
  https://hydra.gnu.org/job/gnu/master/test.dicod.x86_64-linux
  …

(See build-aux/hydra/gnu-system.scm for details.)

I run some of the tests locally or check the results online from
time-to-time, but I admit that checking all of them is still mostly
manual.

However, while the “normal” system tests (mcron, dicod, nginx, openssh,
etc.) run quickly, the installation tests are expensive: they always
start by rebuilding Guix from the current source, proceed with
installation in a VM, and finally boot the resulting system.  Thus,
working with “normal” system tests is convenient, but working with
installation tests is more tiresome.

I think it’s inherent to the nature of installation tests, though if
anyone has ideas on how to make them run faster, that’d be great!

Good news: as of v0.12.0-3433-g4aabc8eaa, “installed-os” passes for me.
:-)

HTH!

Ludo’.

Re: Problems running 'check-system'

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 8875 bytes --]

Hi Ludo,

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Chris,
>
> Chris Marusich <cmmarusich@gmail.com> skribis:
>
>>   make -j check-system TESTS=installed-os
>
> [...]
>
>> srfi/srfi-1.scm:575:27: Throw to key `srfi-34' with args
>> `(#<condition &message [message: "could not find bootstrap binary
>> 'guile-2.0.9.tar.xz' for system 'x86_64-linux'"] 4b62d80>)'.
>> make: *** [Makefile:5066: check-system] Error 1
>> [2] [env] marusich@garuda:~/guix
>> $ 
>>
>>
>> However, if I invoke 'make' first, then the "could not find bootstrap
>> binary" message does not show up, and the test proceeds to be run:
>
> That’s not surprising: ‘make’ triggers a download of guile-2.0.9.tar.xz
> in gnu/packages/bootstrap.  See Makefile.am.

I didn't know that.  Thank you for clarifying it!

> However, ‘check-system’ should depend on this target.  Fixed in
> 693f12ce2326f82020e90e58e69cf2b54808c19b.

Great!

>> Issue #2: even when I run 'make' first, the test fails.  It fails with
>> this message:
>>
>> [... some output omitted for brevity ...]
>>
>> starting phase `copy-bootstrap-guile'
>> Backtrace:
>> In ice-9/boot-9.scm:
>>  160: 13 [catch #t #<catch-closure 8c5dc0> ...]
>> In unknown file:
>>    ?: 12 [apply-smob/1 #<catch-closure 8c5dc0>]
>> In ice-9/boot-9.scm:
>>   66: 11 [call-with-prompt prompt0 ...]
>> In ice-9/eval.scm:
>>  432: 10 [eval # #]
>> In ice-9/boot-9.scm:
>> 2412: 9 [save-module-excursion #<procedure 8e6840 at ice-9/boot-9.scm:4084:3 ()>]
>> 4089: 8 [#<procedure 8e6840 at ice-9/boot-9.scm:4084:3 ()>]
>> 1734: 7 [%start-stack load-stack #<procedure 8f6e20 at ice-9/boot-9.scm:4080:10 ()>]
>> 1739: 6 [#<procedure 8f8960 ()>]
>> In unknown file:
>>    ?: 5 [primitive-load "/gnu/store/wgh83kqjif20wfdg56iz7bxk9d4xmlk0-guix-0.12.0-9.25a4+-guile-builder"]
>> In ice-9/eval.scm:
>>  387: 4 [eval # ()]
>> In srfi/srfi-1.scm:
>>  827: 3 [every1 #<procedure f9fa40 at /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:649:9 (expr)> ...]
>> In /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:
>>  653: 2 [#<procedure f9fa40 at /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:649:9 (expr)> #]
>> In ice-9/eval.scm:
>>  432: 1 [eval # #]
>> In unknown file:
>>    ?: 0 [copy-file "/gnu/store/dgncc5wmw8prxq09y71hqjc6g7rxqvvb-guile-2.0.9.tar.xz" ...]
>>
>> ERROR: In procedure copy-file:
>> ERROR: In procedure copy-file: Permission denied
>
> What this means is that the target of ‘copy-file’ is read-only.

Why is this test is trying to copy a file into a read-only target?  I've
tried debugging this myself by inserting pk statements in the relevant
code, but the long iteration time makes it difficult.

>> I tried to insert a "pk" in the file guix/build/gnu-build-system.scm
>> to see what the target of the copy command was, but after about 24
>> hours (!!) of waiting for the subsequent build to finish (for some
>> reason this single-line change caused many things to be rebuilt), I
>> found that the build failed for a different, apparently unrelated
>> reason.  Quite unfortunate.
>
> (guix build gnu-build-system) is used by everything, which is why
> modifying it triggers a rebuild of everything.

Although in retrospect this seems obvious, I did not realize it at the
time.  I will remember this for the future.  Thank you for the reminder!

> The system tests are being run on Hydra at each commit:
>
>   https://hydra.gnu.org/job/gnu/master/test.installed-os.x86_64-linux
>   https://hydra.gnu.org/job/gnu/master/test.encrypted-root-os.i686-linux
>   https://hydra.gnu.org/job/gnu/master/test.dicod.x86_64-linux
>   …
>
> (See build-aux/hydra/gnu-system.scm for details.)

I looked on Hydra but didn't find these entries.  Thank you for pointing
them out; I'm glad the tests are being run!

Am I correct in interpreting the following page to mean that the
installed-os tests have not passed even once in the last few weeks?

  https://hydra.gnu.org/job/gnu/master/test.installed-os.x86_64-linux

> Good news: as of v0.12.0-3433-g4aabc8eaa, “installed-os” passes for
>me.
> :-)

Sadly, from v0.12.0-3433-g4aabc8eaa, it fails for me with the same kind
of error as before.  I ran the following commands from that commit (the
workspace began from a clean slate, and my guix version is 20170416.04):

  guix environment --pure guix -- ./bootstrap
  guix environment --pure guix -- ./configure --localstatedir=/var
  guix environment --pure guix -- make -j check-system TESTS=installed-os

I got the following error:

--8<---------------cut here---------------start------------->8---
starting phase `copy-bootstrap-guile'
Backtrace:
In ice-9/boot-9.scm:
 160: 13 [catch #t #<catch-closure 8c5b40> ...]
In unknown file:
   ?: 12 [apply-smob/1 #<catch-closure 8c5b40>]
In ice-9/boot-9.scm:
  66: 11 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
 432: 10 [eval # #]
In ice-9/boot-9.scm:
2412: 9 [save-module-excursion #<procedure 8e6840 at ice-9/boot-9.scm:4084:3 ()>]
4089: 8 [#<procedure 8e6840 at ice-9/boot-9.scm:4084:3 ()>]
1734: 7 [%start-stack load-stack #<procedure 8f6ba0 at ice-9/boot-9.scm:4080:10 ()>]
1739: 6 [#<procedure 8f8960 ()>]
In unknown file:
   ?: 5 [primitive-load "/gnu/store/3c1f44lc4spzb4k5syyyynv4rq6l3h53-guix-0.12.0-9.25a4+-guile-builder"]
In ice-9/eval.scm:
 387: 4 [eval # ()]
In srfi/srfi-1.scm:
 827: 3 [every1 #<procedure ae67a0 at /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:649:9 (expr)> ...]
In /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:
 653: 2 [#<procedure ae67a0 at /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:649:9 (expr)> #]
In ice-9/eval.scm:
 432: 1 [eval # #]
In unknown file:
   ?: 0 [copy-file "/gnu/store/dgncc5wmw8prxq09y71hqjc6g7rxqvvb-guile-2.0.9.tar.xz" ...]

ERROR: In procedure copy-file:
ERROR: In procedure copy-file: Permission denied
note: keeping build directory `/tmp/guix-build-guix-0.12.0-9.25a4+.drv-5'
builder for `/gnu/store/w9rx1wx5synd4jxfxzacy6xjl2522r93-guix-0.12.0-9.25a4+.drv' failed with exit code 1
@ build-failed /gnu/store/w9rx1wx5synd4jxfxzacy6xjl2522r93-guix-0.12.0-9.25a4+.drv - 1 builder for `/gnu/store/w9rx1wx5synd4jxfxzacy6xjl2522r93-guix-0.12.0-9.25a4+.drv' failed with exit code 1
cannot build derivation `/gnu/store/zqb17z81s8101g09rib4dsag53xcfksh-activate-service.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/dlqsn6i1i2zbzb0msg00jy5aqva7h223-ca-certificate-bundle.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/jqssa6ii471rysyipnx6fllijmxxza5r-fonts-dir.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/b5ac1f536k923fx73c6cw9p8c3fzmlsf-info-dir.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/i3cmj7c47fz0vw6a3sxaf4b3z44pr9bc-manual-database.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/65k0zpznczsy7zqb9q5jy9ii66ikjfgi-shepherd-guix-daemon.scm.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/mxs4c49giinid8hm6k1923hsbxh2xjb9-activate.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/6f1i6sfgm8y7wh6p5bk4yllkivlzd69n-profile.drv': 5 dependencies couldn't be built
cannot build derivation `/gnu/store/q1f3fy9ip5m3b5kc8mvz9wwj9p3fccbv-shepherd.conf.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/j9rarvlbik25gmwdpis6h2snmvkd5jw8-boot.drv': 2 dependencies couldn't be built
cannot build derivation `/gnu/store/qxmkv4xqb2lb2p1kjwkg4jd5xg6apvk9-system.drv': 2 dependencies couldn't be built
cannot build derivation `/gnu/store/m7zi83z60fq58bbsr9wra8sv6x27zqxk-grub.cfg.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/qyjj078rr8xbc3q2yzqlc8zic2asmwpz-builder-in-linux-vm.drv': 2 dependencies couldn't be built
cannot build derivation `/gnu/store/3hg377qbh2frjn95gcarck4vwjdwldlv-linux-vm-loader.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/ld0hg8jsg7qx8k8l0h0hvapfq2qa823k-disk-image.drv': 3 dependencies couldn't be built
cannot build derivation `/gnu/store/3ssys1pnpmbv88jch7181flsbk8889as-installation.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/a112fvz9xqpczrpf8dp2sr28p8bjyb1q-installed-os.drv': 1 dependencies couldn't be built
TOTAL: 1
FAIL: /gnu/store/52z746p3jwz2d2jxrizrbrr09wh9n2vb-installed-os
make: *** [Makefile:5075: check-system] Error 1
--8<---------------cut here---------------end--------------->8---

I'll let you know if I make any progress on debugging this.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Problems running 'check-system'

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>>> Issue #2: even when I run 'make' first, the test fails.  It fails with
>>> this message:
>>>
>>> [... some output omitted for brevity ...]
>>>
>>> starting phase `copy-bootstrap-guile'
>>> Backtrace:
>>> In ice-9/boot-9.scm:
>>>  160: 13 [catch #t #<catch-closure 8c5dc0> ...]
>>> In unknown file:
>>>    ?: 12 [apply-smob/1 #<catch-closure 8c5dc0>]
>>> In ice-9/boot-9.scm:
>>>   66: 11 [call-with-prompt prompt0 ...]
>>> In ice-9/eval.scm:
>>>  432: 10 [eval # #]
>>> In ice-9/boot-9.scm:
>>> 2412: 9 [save-module-excursion #<procedure 8e6840 at ice-9/boot-9.scm:4084:3 ()>]
>>> 4089: 8 [#<procedure 8e6840 at ice-9/boot-9.scm:4084:3 ()>]
>>> 1734: 7 [%start-stack load-stack #<procedure 8f6e20 at ice-9/boot-9.scm:4080:10 ()>]
>>> 1739: 6 [#<procedure 8f8960 ()>]
>>> In unknown file:
>>>    ?: 5 [primitive-load "/gnu/store/wgh83kqjif20wfdg56iz7bxk9d4xmlk0-guix-0.12.0-9.25a4+-guile-builder"]
>>> In ice-9/eval.scm:
>>>  387: 4 [eval # ()]
>>> In srfi/srfi-1.scm:
>>>  827: 3 [every1 #<procedure f9fa40 at /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:649:9 (expr)> ...]
>>> In /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:
>>>  653: 2 [#<procedure f9fa40 at /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:649:9 (expr)> #]
>>> In ice-9/eval.scm:
>>>  432: 1 [eval # #]
>>> In unknown file:
>>>    ?: 0 [copy-file "/gnu/store/dgncc5wmw8prxq09y71hqjc6g7rxqvvb-guile-2.0.9.tar.xz" ...]
>>>
>>> ERROR: In procedure copy-file:
>>> ERROR: In procedure copy-file: Permission denied
>>
>> What this means is that the target of ‘copy-file’ is read-only.
>
> Why is this test is trying to copy a file into a read-only target?  I've
> tried debugging this myself by inserting pk statements in the relevant
> code, but the long iteration time makes it difficult.

This is coming from the ‘copy-bootstrap-guile’ phase of the ‘guix’
package, which happens before ‘configure’.

I wonder why you’re seeing that happening.  Does “./pre-inst-env guix
build guix” fail as well?

If not, one way to debug is to use “guix build -K” and inspect the files
under gnu/packages/bootstrap in the failed build tree.

And I don’t why we’re getting different results here.

HTH,
Ludo’.

Re: Problems running 'check-system'

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 4610 bytes --]

Hi Ludo,

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Chris,
>
> Chris Marusich <cmmarusich@gmail.com> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>>>> Issue #2: even when I run 'make' first, the test fails.  It fails with
>>>> this message:
>>>>
>>>> [... some output omitted for brevity ...]
>>>>
>>>> starting phase `copy-bootstrap-guile'
>>>> Backtrace:
>>>> In ice-9/boot-9.scm:
>>>>  160: 13 [catch #t #<catch-closure 8c5dc0> ...]
>>>> In unknown file:
>>>>    ?: 12 [apply-smob/1 #<catch-closure 8c5dc0>]
>>>> In ice-9/boot-9.scm:
>>>>   66: 11 [call-with-prompt prompt0 ...]
>>>> In ice-9/eval.scm:
>>>>  432: 10 [eval # #]
>>>> In ice-9/boot-9.scm:
>>>> 2412: 9 [save-module-excursion #<procedure 8e6840 at ice-9/boot-9.scm:4084:3 ()>]
>>>> 4089: 8 [#<procedure 8e6840 at ice-9/boot-9.scm:4084:3 ()>]
>>>> 1734: 7 [%start-stack load-stack #<procedure 8f6e20 at ice-9/boot-9.scm:4080:10 ()>]
>>>> 1739: 6 [#<procedure 8f8960 ()>]
>>>> In unknown file:
>>>>    ?: 5 [primitive-load "/gnu/store/wgh83kqjif20wfdg56iz7bxk9d4xmlk0-guix-0.12.0-9.25a4+-guile-builder"]
>>>> In ice-9/eval.scm:
>>>>  387: 4 [eval # ()]
>>>> In srfi/srfi-1.scm:
>>>>  827: 3 [every1 #<procedure f9fa40 at /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:649:9 (expr)> ...]
>>>> In /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:
>>>>  653: 2 [#<procedure f9fa40 at /gnu/store/a42pfdz8w5qxdkp6xz8783ydywmp0p8p-module-import/guix/build/gnu-build-system.scm:649:9 (expr)> #]
>>>> In ice-9/eval.scm:
>>>>  432: 1 [eval # #]
>>>> In unknown file:
>>>>    ?: 0 [copy-file "/gnu/store/dgncc5wmw8prxq09y71hqjc6g7rxqvvb-guile-2.0.9.tar.xz" ...]
>>>>
>>>> ERROR: In procedure copy-file:
>>>> ERROR: In procedure copy-file: Permission denied
>>>
>>> What this means is that the target of ‘copy-file’ is read-only.
>>
>> Why is this test is trying to copy a file into a read-only target?  I've
>> tried debugging this myself by inserting pk statements in the relevant
>> code, but the long iteration time makes it difficult.
>
> This is coming from the ‘copy-bootstrap-guile’ phase of the ‘guix’
> package, which happens before ‘configure’.
>
> I wonder why you’re seeing that happening.  Does “./pre-inst-env guix
> build guix” fail as well?

No, it doesn't fail.  When I tried this, I needed to invoke it with
--check, and the build failed due to non-determinism, but the actual
build and check phases succeeded (I ran this on my GuixSD machine).

> And I don’t why we’re getting different results here.

I think I might have finally found the reason.  When I remove the
"--pure" from the invocation, the "installed-os" test runs successfully
on my GuixSD machine.  It seems that, for some reason, "--pure" was the
cause of the failure.  That's surprising to me, since my expectation was
that Guix's tests should work just fine in a pure environment.  Could
this be a bug?

This surprising behavior is not limited to my single GuixSD system.
I've tried running the same invocation on my Ubuntu machine, and it
fails with the same "ERROR: In procedure copy-file: Permission denied"
message.  So I think you ought to be able to reproduce it, too, as long
as you use the same invocation (and the same commit:
4aabc8eaa22314b412d8c5517e05085aa1b38498).  Curiously, on my Ubuntu
system, when I remove "--pure", the "copy-file" error does NOT occur.
(The "installed-os" test fails anyway due to KVM issues because I
haven't set up KVM properly on my Ubuntu machine yet, but that's not
relevant to this discussion.)

To summarize, on my GuixSD system, the "copy-file" error occurs when I
run:

  guix environment --pure guix -- make check-system TESTS=installed-os

On my GuixSD system, it also occurs when I run:

  guix environment --pure guix

followed by (in the environment):

  make check-system TESTS=installed-os

However, on my Ubuntu system, the "copy-file" error only occurs when I
run:

  guix environment --pure guix -- make check-system TESTS=installed-os

On my Ubuntu system, it does NOT occur when I run:

  guix environment --pure guix

followed by (in the environment):

  make check-system TESTS=installed-os

In conclusion, I can now run the tests, which is great.  However, I'm
surprised that "--pure" was causing the "copy-file" error.  I'm also
unable to explain the difference in behavior between GuixSD and Ubuntu
when using "--pure".  Could this be a bug?

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Problems running 'check-system'

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> I think I might have finally found the reason.  When I remove the
> "--pure" from the invocation, the "installed-os" test runs successfully
> on my GuixSD machine.  It seems that, for some reason, "--pure" was the
> cause of the failure.  That's surprising to me, since my expectation was
> that Guix's tests should work just fine in a pure environment.  Could
> this be a bug?

When using ‘--pure’, there’s no ‘git’ command in $PATH.  So my guess
would be that this changes the content of (current-guix), which uses
‘git-predicate’.

In particular, your ‘current-guix’ package ends up including
guile-2.0.9.tar.xz file, because it cannot determine that it’s not under
version control, and then the build process of the ‘guix’ package fails
because this file already exists and is read-only.

QED!  :-)

That’s a limitation of ‘git-predicate’ in its current form.  Eventually
we’ll require ‘guile-git’ and we won’t have that problem.

In the meantime, the workaround is to add Git to your environment:

  guix environment guix --ad-hoc git --pure

Thanks for debugging it, and I HTH!

Ludo’.

Re: Problems running 'check-system'

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1505 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Chris,
>
> Chris Marusich <cmmarusich@gmail.com> skribis:
>
>> I think I might have finally found the reason.  When I remove the
>> "--pure" from the invocation, the "installed-os" test runs successfully
>> on my GuixSD machine.  It seems that, for some reason, "--pure" was the
>> cause of the failure.  That's surprising to me, since my expectation was
>> that Guix's tests should work just fine in a pure environment.  Could
>> this be a bug?
>
> When using ‘--pure’, there’s no ‘git’ command in $PATH.  So my guess
> would be that this changes the content of (current-guix), which uses
> ‘git-predicate’.
>
> In particular, your ‘current-guix’ package ends up including
> guile-2.0.9.tar.xz file, because it cannot determine that it’s not under
> version control, and then the build process of the ‘guix’ package fails
> because this file already exists and is read-only.
>
> QED!  :-)
>
> That’s a limitation of ‘git-predicate’ in its current form.  Eventually
> we’ll require ‘guile-git’ and we won’t have that problem.
>
> In the meantime, the workaround is to add Git to your environment:
>
>   guix environment guix --ad-hoc git --pure
>
> Thanks for debugging it, and I HTH!
>
> Ludo’.

You're right.  The test passed when I added "--ad-hoc git".  Your
explanation makes sense.  Thank you for bearing with me and taking the
time to help me figure this out!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]