From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Marusich Subject: bug#21097: verify-store test failure on armhf-linux Date: Fri, 08 Jun 2018 01:21:33 -0700 Message-ID: <87fu1xy9gy.fsf@garuda.local.i-did-not-set--mail-host-address--so-tickle-me> References: <87k2tu3b10.fsf@netris.org> <877flybt9v.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:43692) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fRCet-00075X-4K for bug-guix@gnu.org; Fri, 08 Jun 2018 04:22:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fRCep-0007c7-41 for bug-guix@gnu.org; Fri, 08 Jun 2018 04:22:07 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:58996) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fRCeo-0007c1-Vy for bug-guix@gnu.org; Fri, 08 Jun 2018 04:22:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1fRCen-00076f-Mm for bug-guix@gnu.org; Fri, 08 Jun 2018 04:22:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <877flybt9v.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Tue, 03 Nov 2015 23:41:16 +0100") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 21097@debbugs.gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable ludo@gnu.org (Ludovic Court=C3=A8s) writes: > I=E2=80=99ve become convinced that this is due to parallelism: several > guix-daemon processes run at the same time. In this case, I bet this > process tries to remove an item from the ValidPaths table while another > is trying to add it in the Refs table or something. > > In dc57d527 I added #:parallel-tests? #f for =E2=80=98guix-devel=E2=80=99= . Eventually > we should fix the makefile to run this test alone, as is done for > =E2=80=98guix-gc.sh=E2=80=99. In the 2 years and 7 months since we disabled parallel tests in commit dc57d527aee4eb18ec5fb345f90d6637bbd1a4d2 to work around this bug, we may have allowed other parallelism bugs to quietly creep in. Today, I observed a parallel test failure that seems unrelated to the original bug reported here. And anecdotally, I feel that the tests frequently fail spuriously when I run them in parallel. Until we get to the bottom of this, I agree that the best thing to do is to always run the tests in serial. For completeness, below I'll report the failure I observed today. On my x86_64-linux GuixSD machine, using Guix version 0ec430f79530ee343c175347952f91a78adca5ec (this is what my ~/.config/guix/latest points to), I entered a Guix development environment via "guix environment guix". In Guix's Git repository, I checked out commit 4dd91dff477b9717b3fa494b23976e4d69ab7dfc (the current tip of core-updates) and ran the following commands: ./bootstrap && ./configure --localstatedir=3D/var && make -j \ && make -j check The following tests failed: FAIL: tests/guix-hash.sh FAIL: tests/guix-download.sh FAIL: tests/guix-build.sh FAIL: tests/guix-package.sh FAIL: tests/guix-system.sh When I immediately ran "make recheck" without making any changes, the same 5 tests passed. Note that this ran the tests in serial because I omitted -j. When I ran the same 5 tests again in parallel using the following command, they all passed: make -j check TESTS=3D"tests/guix-hash.sh tests/guix-download.sh \ tests/guix-build.sh tests/guix-package.sh tests/guix-system.sh" I also tried running just tests/guix-hash.sh and tests/guix-download.sh together 10 times in serial and then 10 times in parallel. Unfortunately, this didn't reproduce the failure, either (i.e., all 20 test runs passed). All in all, this seems to suggest that the failures I observed might be caused by a parallelism bug when running the entire test suite. Regarding the cause of failure, the 5 tests all failed with a message like the following: =2D-8<---------------cut here---------------start------------->8--- ERROR: In procedure canonicalize-path: In procedure canonicalize-path: No such file or directory + guix download --version Backtrace: In ice-9/boot-9.scm: 2875:24 19 (_) 222:17 18 (map1 (((guix utils)) ((guix config)) ((guix #)) ((=E2=80=A6))= =E2=80=A6)) 2788:17 17 (resolve-interface (guix utils) #:select _ #:hide _ # _ =E2=80= =A6) 2714:10 16 (_ (guix utils) _ _ #:ensure _) 2982:16 15 (try-module-autoload _ _) 2312:4 14 (save-module-excursion #) 3002:22 13 (_) In unknown file: 12 (primitive-load-path "guix/utils" #) In guix/utils.scm: 26:0 11 (_) In ice-9/boot-9.scm: 2862:4 10 (define-module* _ #:filename _ #:pure _ #:version _ # _ =E2=80= =A6) 2875:24 9 (_) 222:17 8 (map1 (((guix config)) ((srfi srfi-1)) ((srfi #)) (#) =E2=80= =A6)) 2788:17 7 (resolve-interface (guix config) #:select _ #:hide _ # _ =E2= =80=A6) 2714:10 6 (_ (guix config) _ _ #:ensure _) 2982:16 5 (try-module-autoload _ _) 2312:4 4 (save-module-excursion #) 3002:22 3 (_) In unknown file: 2 (primitive-load-path "guix/config" #) In guix/config.scm: 86:6 1 (_) In unknown file: 0 (canonicalize-path "/home/marusich/guix/test-tmp/db") =2D-8<---------------cut here---------------end--------------->8--- All the test failures looked the same, except that instead of "guix download --version", the equivalent command (e.g., "guix system =2D-version") was invoked. I realize this information doesn't help solve the original bug reported here. However, it's a real failure, so I hope it'll be useful. In any case, it shows that there are probably multiple parallelism bugs lurking in our code now. We're going to have to solve all those parallelism bugs before we can reliably run the tests in parallel again. =2D-=20 Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAlsaPI0ACgkQ3UCaFdgi Rp3+/g//VWvLRC78ldJ/Xxf1ErllS8hiYsHl5jtTxaU3hm/EnPqOe34P6nKWPh67 DMicUYcNgRWSkx5qEP19WuSjq9c21HQo6NTgZFhrkZR5SLzw44HqO1OZOACuGtRu exbkD774mbgEJyDbEZwhN/6G/V9myyjF07/h1ZkSqV58Rh2e9DVA2Wdjojr+OoIK RcM0WtcJwHa5XNdsJF6Mlgwsjjmv6vtE9gkS27iL59f2U0TlSJUQMkylLxa6uq2r bMUfxspURkk0nyC62kFGeP2g7m7rIT3bNj2C751sDRLh9VJeu+5D6g1klRb1YIlA 2WFgqU39B1TGIMO9a/JG/PaRAxpHsdQt+hS9Urf7i/E88eEQ1xbxF3CuUzqZ2kBm 07nHpx1yI0ek+MdD/gnL8wbGL/1ClZ20IWLuSgMStNa1V2hE/QFzoeKDt+KUZecG +tAaxBj4TKlCQCvf24OZiyskH9gjwobFOD86JWb4MZ1rmeMdInUTN2HK+2DOCUvG 3qfaFa64LpqqeRqyXQtirN8q33vlonzBUesxfu4IIKxhvpVbAbYPOkzLfoK9khac m9I5JdAbfXs/Vr7ZJ+bTJEcz3OgX4g0+pl6ZT4+qDLf/4LiTKKVxEjfghE7xiHrQ Z7KH2D5mBnjhdI1dJYHxkNj4jv4+d6LgGQKkYkAQXbPqQjh/ijo= =vzSU -----END PGP SIGNATURE----- --=-=-=--