Re: Treating tests as special case

[Pjotr Prins <pjotr.public12@thebird.nl>]

On Thu, Apr 05, 2018 at 05:24:12PM +0200, Ludovic Courtès wrote:
> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
> 
> > I am *not* suggesting we stop testing and stop writing tests. They are
> > extremely important for integration (thought we could do with a lot
> > less and more focussed integration tests - ref Hickey). What I am
> > writing is that we don't have to rerun tests for everyone *once* they
> > succeed *somewhere*. If you have a successful reproducible build and
> > tests on a platform there is really no point in rerunning tests
> > everywhere for the exact same setup. It is a nice property of our FP
> > approach. Proof that it is not necessary is the fact that we
> > distribute substitute binaries without running tests there. What I am
> > proposing in essence is 'substitute tests'. 
> 
> Understood.
> 
> > If tests are so important to rerun: tell me why we are not running
> > tests when substituting binaries?
> 
> Because you have a substitute if and only those tests already passed
> somewhere.  This is exactly the property we’re interested in, right?

Yup. Problem is substitutes go away. We don't retain them and I often
encounter that use case.

Providing test-substitutes is much lighter and can be retained
forever.

When tests ever pass on a build server, we don't have to repeat them.
That is my story.

Pj.