From: Pjotr Prins <pjotr.public12@thebird.nl>
To: "Ludovic Courtès" <ludo@gnu.org>
Cc: guix-devel@gnu.org
Subject: Re: Treating tests as special case
Date: Thu, 5 Apr 2018 16:59:29 +0200 [thread overview]
Message-ID: <20180405145929.GA345@thebird.nl> (raw)
In-Reply-To: <87efjtzqo4.fsf@gnu.org>
On Thu, Apr 05, 2018 at 04:14:19PM +0200, Ludovic Courtès wrote:
> I sympathize with what you write about the inconvenience of running
> tests, when substitutes aren’t available. However, I do think running
> tests has real value.
>
> Of course sometimes we just spend time fiddling with the tests so they
> would run in the isolated build environment, and they do run flawlessly
> once we’ve done the usual adjustments (no networking, no /bin/sh, etc.)
>
> However, in many packages we found integration issues that we would just
> have missed had we not run the tests; that in turn can lead to very bad
> user experience. In other cases we found real upstream bugs and were
> able to report them
> (cf. <https://github.com/TaylanUB/scheme-bytestructures/issues/30> for
> an example from today.) Back when I contributed to Nixpkgs, tests were
> not run by default and I think that it had a negative impact on QA.
>
> So to me, not running tests is not an option.
I am *not* suggesting we stop testing and stop writing tests. They are
extremely important for integration (thought we could do with a lot
less and more focussed integration tests - ref Hickey). What I am
writing is that we don't have to rerun tests for everyone *once* they
succeed *somewhere*. If you have a successful reproducible build and
tests on a platform there is really no point in rerunning tests
everywhere for the exact same setup. It is a nice property of our FP
approach. Proof that it is not necessary is the fact that we
distribute substitute binaries without running tests there. What I am
proposing in essence is 'substitute tests'.
Ricardo is suggesting an implementation. I think it is simpler. When
building a derivation we know the hash. If we have a list of hashes in
the database for successful tests (hash-tests-passed) it is
essentially queriable and done. Even when the substitute gets removed,
that item can still remain at almost no cost.
Ludo, I think we need to do this. There is no point in running tests
that already have been run. Hickey is right. I have reached
enlightment. Almost everything I thought about testing is wrong. If
all the inputs are the same the test will *always* pass. There is no
point to it! The only way such a test won't pass it by divine
intervention or real hardware problems. Both we don't want to test
for.
If tests are so important to rerun: tell me why we are not running
tests when substituting binaries?
> The problem I’m more interested in is: can we provide substitutes more
> quickly? Can we grow an infrastructure such that ‘master’, by default,
> contains software that has already been built?
Sure, that is another challenge and an important one.
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
> > An idea that came up on #guix several months ago was to separate the
> > building of packages from testing. Testing would be a continuation of
> > the build, like grafts could be envisioned as a continuation of the
> > build.
>
> I agree it would be nice, but I think there’s a significant technical
> issue: test suites usually expect to run from the build tree.
What I understand is that Nix already does something like this. they
have split testing out to allow for network access. I don't propose to
split the process. I propose to cache testing as part of the build.
Pj.
next prev parent reply other threads:[~2018-04-05 15:04 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-04-05 5:24 Treating tests as special case Pjotr Prins
2018-04-05 6:05 ` Gábor Boskovits
2018-04-05 8:39 ` Pjotr Prins
2018-04-05 8:58 ` Hartmut Goebel
2018-04-05 6:21 ` Björn Höfling
2018-04-05 8:43 ` Pjotr Prins
2018-04-06 8:58 ` Chris Marusich
2018-04-06 18:36 ` David Pirotte
2018-04-05 10:14 ` Ricardo Wurmus
2018-04-05 12:19 ` Björn Höfling
2018-04-05 14:10 ` Ricardo Wurmus
2018-04-05 10:26 ` Ricardo Wurmus
2018-04-05 14:14 ` Ludovic Courtès
2018-04-05 14:59 ` Pjotr Prins [this message]
2018-04-05 15:17 ` Ricardo Wurmus
2018-04-05 15:24 ` Ludovic Courtès
2018-04-05 16:41 ` Pjotr Prins
2018-04-05 18:35 ` Pjotr Prins
2018-04-06 7:57 ` Retaining substitutes Ludovic Courtès
2018-04-05 20:26 ` Treating tests as special case Mark H Weaver
2018-04-06 6:06 ` Pjotr Prins
2018-04-06 8:27 ` Ricardo Wurmus
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180405145929.GA345@thebird.nl \
--to=pjotr.public12@thebird.nl \
--cc=guix-devel@gnu.org \
--cc=ludo@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).