Turning off tests leads to a different store item (was: Unable to build "Ten Years Reproducibility Challenge" paper)

Turning off tests leads to a different store item (was: Unable to build "Ten Years Reproducibility Challenge" paper)

[Suhail]

Felix Lechner via <help-guix@gnu.org> writes:

> On Tue, Oct 31 2023, Julien Lepiller wrote:
>
>> You could skip tests, but that would build a different derivation
>> tree, so it might affect reproducibility
>
> Those shortfalls are a consequence of our packaging habits. In an ideal
> world, would running a test suite ever build a different derivation?

Tests can fail for a number of reasons. At times tests may have some
failure modes that aren't so easily reliably overcome (as setting the
system clock).

By default, do we capture the test log and include it as part of the
build artifact? I'm not asking rhetorically.

If so, is there an existing discussion noting the pros/cons of that
decision that I can read to be better informed?

If not, why should skipping the tests result in a different derivation
tree? And if it shouldn't, but does, is there an existing bug report for
my reference?

> Please bear with me as I again belabor the same point without
> receiving public support. Building packages should be separate from
> testing them.

FWIW, I believe I agree with you. Perhaps following Simon's recently
proposed (not yet accepted) RFC process could help in this regard:
<https://issues.guix.gnu.org/issue/66844>.

Is there an existing issue which notes some of the drawbacks of the
existing approach and perhaps proposes some alternatives?

I do not know enough to know how trivial or not it would be to alter the
present behaviour (and what some of the unintended consequences might
be), but I believe this is an important topic that needs some thought
and consideration.

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an intention
to enter into an agreement, has no operative effect until a definitive agreement
is signed in writing by both parties, and that no party should act in reliance
on the email or any representations of the sender until a definitive agreement
is signed in writing by both parties.

This email may contain information that is privileged, confidential and/or
exempt from disclosure.  No waiver whatsoever is intended by sending this e-mail
which is intended only for the named recipient(s).  Unauthorized use,
dissemination or copying is prohibited.  If you receive this email in error,
please notify the sender and destroy all copies of this email.

Re: Turning off tests leads to a different store item (was: Unable to build "Ten Years Reproducibility Challenge" paper)

[Simon Tournier]

Hi,

On Wed, 01 Nov 2023 at 17:52, Suhail <suhail@bayesians.ca> wrote:

> If not, why should skipping the tests result in a different derivation
> tree?

The store path is different because it hashes all the inputs, included
the builder script; from my understanding.

For instance, consider the package ’hello’.  The derivation with and
without tests are:

--8<---------------cut here---------------start------------->8---
$ guix build hello -d --no-grafts
/gnu/store/cvdzgd9glngrb98jqfa6bpm6nj6j3q55-hello-2.12.1.drv

$ guix build hello -d --without-tests=hello --no-grafts
/gnu/store/my14b3qwzqjawzadvcc30p2z5iv53amx-hello-2.12.1.drv
--8<---------------cut here---------------end--------------->8---

Now, let compare them,

--8<---------------cut here---------------start------------->8---
$ cat $(guix build hello -d --no-grafts) \
      | sed 's/,/\n/g' | grep '/gnu/'| sort > /tmp/with
$ cat $(guix build hello -d --no-grafts --without-tests=hello) \
      | sed 's/,/\n/g' | grep '/gnu/' | sort > /tmp/without
$ diff /tmp/with /tmp/without
4a5,6
> "/gnu/store/3hsisq3553z3gcy0yf270qy7ds9rzmis-hello-2.12.1"
> "/gnu/store/3hsisq3553z3gcy0yf270qy7ds9rzmis-hello-2.12.1")])
6,7d7
< "/gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1"
< "/gnu/store/5mqwac3zshjjn1ig82s12rbi7whqm4n8-hello-2.12.1")])
21a22,23
> "/gnu/store/jd6566z3xyyzfnyvsyi95ybk39xhbybr-hello-2.12.1-builder"]
> "/gnu/store/jd6566z3xyyzfnyvsyi95ybk39xhbybr-hello-2.12.1-builder"]
25,26d26
< "/gnu/store/pspxyd64587jq55jhbz3z92j4lnqm857-hello-2.12.1-builder"]
< "/gnu/store/pspxyd64587jq55jhbz3z92j4lnqm857-hello-2.12.1-builder"]
--8<---------------cut here---------------end--------------->8---

Well, basically the difference is from the builder: it is the Guile
script that builds.  One contains ’#:tests? #t’ and the other contains
’#:tests? #f’.

--8<---------------cut here---------------start------------->8---
$ cat /gnu/store/jd6566z3xyyzfnyvsyi95ybk39xhbybr-hello-2.12.1-builder \
   | sed 's/ /\n/g' > /tmp/without-builder
$ cat /gnu/store/pspxyd64587jq55jhbz3z92j4lnqm857-hello-2.12.1-builder
   | sed 's/ /\n/g' > /tmp/with-builder
$ diff /tmp/with-builder /tmp/without-builder
148c148
< #t
---
> #f
--8<---------------cut here---------------end--------------->8---

Well, from my understanding, get the same the store path for the same
package built with or without its tests would mean a deep change of the
build systems.

Somehow, it would mean to have a derivation for building and another
derivation – referring to the former – for testing.  Well, taking this
direction, one could imagine a derivation for each phases; somehow
extend to all phases what it is already done for fetching source, build
and grafts (guix build --source && guix build --no-grafts && guix build).
That would be a piece of work. :-)

Cheers,
simon