Re: Turning off tests leads to a different store item

Re: Turning off tests leads to a different store item

[Suhail]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> On Wed, 01 Nov 2023 at 17:52, Suhail <suhail@bayesians.ca> wrote:
>
>> If not, why should skipping the tests result in a different
>> derivation tree?
>
> The store path is different because it hashes all the inputs, included
> the builder script; from my understanding.

It certainly seems to be the case. Would you know the specific place(s)
in the source code (in addition to guix/derivations.scm and
guix/store.scm) that would be relevant for this discussion?

On a related note, is Dolstra's phd thesis relevant for these aspects,
or do you know if Guix diverges from Nix in some areas?

> Well, from my understanding, get the same the store path for the same
> package built with or without its tests ...  would mean to have a
> derivation for building and another derivation – referring to the
> former – for testing.

Yes, with the test derivation being something like a "fixed-output
derivation". [[info:guix#Derivations][From the manual]]:

#+begin_quote
Operations such as file downloads and version-control checkouts for
which the expected content hash is known in advance are modeled as
fixed-output derivations. Unlike regular derivations, the outputs of a
fixed-output derivation are independent of its inputs—e.g., a source
code download produces the same result regardless of the download method
and tools being used.
#+end_quote

The hypothetical test derivation leaves the build artifact unchanged,
but does communicate some "side" information. It's like a fixed-output
derivation carrying some metadata (further elaboration below).

> Well, taking this direction, one could imagine a derivation for each
> phases; somehow extend to all phases what it is already done for
> fetching source, build and grafts (guix build --source && guix build
> --no-grafts && guix build).

Perhaps not all. The thing that sets the "check" phase (#:tests?) apart
from the rest is that it's an identity transform with a
side-effect. i.e., it simply reports on the state of its input (i.e.,
the build artifact) leaving the build artifact unchanged. The only other
phase in the gnu-build-system that is similar to the "check phase" is
the "validate-runpath phase".

An alternative (possibly future) version of Guix might allow us to delay
the interpretation/meaning associated with the following cases:

- whether or not the tests were run, and
- whether or not the tests, if run, passed

To be able to do so, we need the "check derivation" be able to
communicate two things (downstream): a summary of the test phase (could
simply be a 3-valued datatype, like a 'Maybe Bool' in Haskell, but could
also be something more elaborate), and the input build artifact (which
remains unchanged). Similarly for the "validate-runpath derivation".

I do not know what the best way to communicate the "side-effect
information" (did the tests run and if so what was the conclusion) would
be. Specifically, should the output of "check derivation" (similarly for
"validate-runpath derivation") include the unchanged build artifact or
not?

Btw, is this still the appropriate mailing list (and more generally,
forum) for this discussion?

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

This email may contain information that is privileged, confidential
and/or exempt from disclosure.  No waiver whatsoever is intended by
sending this e-mail which is intended only for the named recipient(s).
Unauthorized use, dissemination or copying is prohibited.  If you
receive this email in error, please notify the sender and destroy all
copies of this email.

Re: Turning off tests leads to a different store item

[Greg Hogan]

On Thu, Nov 2, 2023 at 11:26 AM Suhail <suhail@bayesians.ca> wrote:
> Perhaps not all. The thing that sets the "check" phase (#:tests?) apart
> from the rest is that it's an identity transform with a
> side-effect. i.e., it simply reports on the state of its input (i.e.,
> the build artifact) leaving the build artifact unchanged. The only other
> phase in the gnu-build-system that is similar to the "check phase" is
> the "validate-runpath phase".

Suhail,

Should this say "without side-effects"? How does one guarantee that
the test phase is free of side-effects?

Perhaps it might be possible to add a new, throwaway layer to the
OverlayFS so that the tests could still write but would not modify the
filesystem.

Alternatively, could we make it easier to record a timestamp in the
manifest for Guix to use as the build clock time? Several packages
already make use of libfaketime, but this could be integrated into the
build process. Not sure of any issues regarding faketime, performance
or otherwise.

Greg

Re: Turning off tests leads to a different store item

[Simon Tournier]

[-- Attachment #1: Type: text/plain, Size: 6167 bytes --]

Hi,

On Thu, 02 Nov 2023 at 15:25, Suhail <suhail@bayesians.ca> wrote:

> It certainly seems to be the case. Would you know the specific place(s)
> in the source code (in addition to guix/derivations.scm and
> guix/store.scm) that would be relevant for this discussion?

I guess all the build systems: guix/build/*.scm and
guix/build-system/*/scm too.


> On a related note, is Dolstra's phd thesis relevant for these aspects,
> or do you know if Guix diverges from Nix in some areas?

I do not know.


> Yes, with the test derivation being something like a "fixed-output
> derivation". [[info:guix#Derivations][From the manual]]:

No, it cannot be a “fixed-output” derivation…

> #+begin_quote
> Operations such as file downloads and version-control checkouts for
> which the expected content hash is known in advance are modeled as
> fixed-output derivations. Unlike regular derivations, the outputs of a
> fixed-output derivation are independent of its inputs—e.g., a source
> code download produces the same result regardless of the download method
> and tools being used.
> #+end_quote

…because we cannot know in advance the expected content hash of the
tests output.


> Perhaps not all. The thing that sets the "check" phase (#:tests?) apart
> from the rest is that it's an identity transform with a
> side-effect. i.e., it simply reports on the state of its input (i.e.,
> the build artifact) leaving the build artifact unchanged. The only other
> phase in the gnu-build-system that is similar to the "check phase" is
> the "validate-runpath phase".

I am not sure to follow.

Well, I wrote earlier:

        Well, from my understanding, get the same the store path for the same
        package built with or without its tests ...  would mean to have a
        derivation for building and another derivation – referring to the
        former – for testing.

And from my understanding, one solution would be to have something as
below.  One “object” for building and another “object” for testing.

    ( Here, I am using the same object namely <package> for the both;
just to make concrete the discussion. :-) )

The point is: we have two derivations; one for the build (without tests)
and another for the tests (without build).

--8<---------------cut here---------------start------------->8---
$ guix build -L . hi-test --no-grafts -K

The following derivations will be built:
  /gnu/store/…-hi-test-2.10.drv
  /gnu/store/…-hi-build-2.10.drv

building /gnu/store/…-hi-build-2.10.drv...
starting phase `set-SOURCE-DATE-EPOCH'
phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
starting phase `set-paths'
phase `set-paths' succeeded after 0.0 seconds
starting phase `install-locale'
phase `install-locale' succeeded after 0.0 seconds
starting phase `unpack'
phase `unpack' succeeded after 0.0 seconds
starting phase `bootstrap'
phase `bootstrap' succeeded after 0.0 seconds
starting phase `patch-usr-bin-file'
phase `patch-usr-bin-file' succeeded after 0.1 seconds
starting phase `patch-source-shebangs'
phase `patch-source-shebangs' succeeded after 0.0 seconds
starting phase `configure'
phase `configure' succeeded after 14.8 seconds
starting phase `patch-generated-file-shebangs'
phase `patch-generated-file-shebangs' succeeded after 0.0 seconds

starting phase `build'

 … do stuff …
 
phase `build' succeeded after 1.2 seconds

starting phase `install'
phase `install' succeeded after 0.5 seconds
starting phase `patch-shebangs'
phase `patch-shebangs' succeeded after 0.0 seconds
starting phase `strip'
phase `strip' succeeded after 0.0 seconds
starting phase `validate-runpath'
phase `validate-runpath' succeeded after 0.0 seconds
starting phase `validate-documentation-location'
phase `validate-documentation-location' succeeded after 0.0 seconds
starting phase `delete-info-dir-file'
phase `delete-info-dir-file' succeeded after 0.0 seconds
starting phase `patch-dot-desktop-files'
phase `patch-dot-desktop-files' succeeded after 0.0 seconds
starting phase `make-dynamic-linker-cache'
starting phase `install-license-files'
phase `install-license-files' succeeded after 0.0 seconds
starting phase `reset-gzip-timestamps'
phase `reset-gzip-timestamps' succeeded after 0.0 seconds
starting phase `compress-documentation'
phase `compress-documentation' succeeded after 0.0 seconds
successfully built /gnu/store/…-hi-build-2.10.drv


building /gnu/store/gnjj4hq5pk890l211b28nkd1dwx4z09k-hi-test-2.10.drv...
starting phase `set-SOURCE-DATE-EPOCH'
phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
starting phase `set-paths'
phase `set-paths' succeeded after 0.0 seconds
starting phase `install-locale'
phase `install-locale' succeeded after 0.0 seconds
starting phase `unpack'
phase `unpack' succeeded after 0.0 seconds
starting phase `bootstrap'
phase `bootstrap' succeeded after 0.0 seconds
starting phase `patch-usr-bin-file'
phase `patch-usr-bin-file' succeeded after 0.0 seconds
starting phase `patch-source-shebangs'
phase `patch-source-shebangs' succeeded after 0.0 seconds
starting phase `configure'
phase `configure' succeeded after 8.9 seconds
starting phase `patch-generated-file-shebangs'
phase `patch-generated-file-shebangs' succeeded after 0.0 seconds

starting phase `check'
[...]

============================================================================
Testsuite summary for GNU Hello 2.10
============================================================================
# TOTAL: 5
# PASS:  4
# SKIP:  1
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
phase `check' succeeded after 0.4 seconds

starting phase `install'
phase `install' succeeded after 0.3 seconds
successfully built /gnu/store/gnjj4hq5pk890l211b28nkd1dwx4z09k-hi-test-2.10.drv

/gnu/store/7fc3933yqq9hnp4rrxp84gxdpg270k7v-hi-build-2.10
/gnu/store/7fc3933yqq9hnp4rrxp84gxdpg270k7v-hi-test-2.10
--8<---------------cut here---------------end--------------->8---

with the file attached.


[-- Attachment #2: hello.scm --]
[-- Type: text/plain, Size: 1572 bytes --]

(define-module (hello)
  #:use-module (guix packages)
  #:use-module (guix download)
  #:use-module (guix build-system gnu)
  #:use-module (guix licenses)
  #:use-module (gnu packages gawk))

(define-public hi
  (package
    (name "hi-build")
    (version "2.10")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
    (build-system gnu-build-system)
    (arguments
     `(#:configure-flags '("--enable-silent-rules")
       #:phases
       (modify-phases %standard-phases
         (delete 'check))))
    (inputs (list gawk))
    (synopsis "Hello, GNU world: An example GNU package")
    (description "Guess what GNU Hello prints!")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))

(define-public hi-test
  (package/inherit hi
    (name "hi-test")
    (build-system gnu-build-system)
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (delete 'build)

         (delete 'patch-shebangs)
         (delete 'strip)
         (delete 'validate-runpath)
         (delete 'reset-gzip-timestamps)
         (delete 'validate-documentation-location)
         (delete 'delete-info-dir-file)
         (delete 'patch-dot-desktop-files)
         (delete 'make-dynamic-linker-cache)
         (delete 'install-license-files)
         (delete 'compress-documentation))))
    (inputs (list hi))))

[-- Attachment #3: Type: text/plain, Size: 301 bytes --]


Somehow, we could have a “build” build-system and a “test” build-system.
And the “build object” would be an inputs of the “test object“.  Well,
somehow perhaps some revamp of the <package> record.

Hum, I do not know… I am not convinced by my own words. :-)

Cheers,
simon

Re: Turning off tests leads to a different store item

[Suhail]

Greg Hogan <code@greghogan.com> writes:

> On Thu, Nov 2, 2023 at 11:26 AM Suhail <suhail@bayesians.ca> wrote:
>> Perhaps not all. The thing that sets the "check" phase (#:tests?) apart
>> from the rest is that it's an identity transform with a
>> side-effect. i.e., it simply reports on the state of its input (i.e.,
>> the build artifact) leaving the build artifact unchanged. The only other
>> phase in the gnu-build-system that is similar to the "check phase" is
>> the "validate-runpath phase".
>
> Should this say "without side-effects"?

No, I meant "with side-effect", but we have a notational
difference. What is termed side-effect is context dependent, so I'll fix
my terminology and make clearer what I meant. To me, the "check" phase
is something that reports on the state of its input. Said report (i.e.,
whether or not the check phase was run, and if run what its output was)
is what I was using the term "side-effect" for. Perhaps "test state
metadata" would be a better term.

> How does one guarantee that the test phase is free of side-effects?

Translation to remove ambiguity of the term "side-effect": How does one
guarantee that the test phase doesn't modify its input build artifact?

This could be enforced using an overlay layer as you suggest, but we
could also take a different approach. It would be sufficient to be able
to distinguish well-behaved tests (those that don't alter the input
build-artifact) from those that aren't (i.e., those that do alter the
input build-artifact).

> Alternatively, could we make it easier to record a timestamp in the
> manifest for Guix to use as the build clock time?

While that would address the original issue that motivated this
discussion thread, it's not an alternative in the context of the current
discussion. Regardless, I agree that having such a facility would be
useful.

While I do believe the defaults in gnu-build-system could benefit from
revisiting our current treatment of the check and validate-runpath
phases, advocating for that is not my primary goal with this
discussion.

As someone who's not well-versed in Guix internals, I am simply trying
to understand what an alternate build-system (say, ds-build-system)
would look like. One that allows the user/caller of a package to
determine how to handle:

- whether or not the tests were run, and
- whether or not the tests, if run, passed

Similarly for other such "phases".

One application for such a build-system (which is of interest to me)
would be to "package" the result of some "data"-dependent computations
where the notion of what constitutes a "passing build artifact" is less
clearly defined (e.g., models used for algorithmic decision-making).

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

This email may contain information that is privileged, confidential
and/or exempt from disclosure.  No waiver whatsoever is intended by
sending this e-mail which is intended only for the named recipient(s).
Unauthorized use, dissemination or copying is prohibited.  If you
receive this email in error, please notify the sender and destroy all
copies of this email.

Re: Turning off tests leads to a different store item

[Simon Tournier]

On Thu, 2 Nov 2023 at 18:02, Simon Tournier <zimon.toutoune@gmail.com> wrote:

> /gnu/store/7fc3933yqq9hnp4rrxp84gxdpg270k7v-hi-build-2.10
> /gnu/store/7fc3933yqq9hnp4rrxp84gxdpg270k7v-hi-test-2.10

Here is a typo when copy/pasting from my Shell to my Email composer
and editing for more clearity.  Obviously, the two items does not have
the same hash.

Cheers,
simon

Re: Turning off tests leads to a different store item

[Suhail]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> On Thu, 02 Nov 2023 at 15:25, Suhail <suhail@bayesians.ca> wrote:
>
>> Yes, with the test derivation being something like a "fixed-output
>> derivation". [[info:guix#Derivations][From the manual]]:
>
> No, it cannot be a “fixed-output” derivation…
>
> …because we cannot know in advance the expected content hash of the
> tests output.

Yes, hence the "something like". It's similar, but also different in
important ways.

> And from my understanding, one solution would be to have something as
> below.  One “object” for building and another “object” for testing.
>
>     ( Here, I am using the same object namely <package> for the both;
> just to make concrete the discussion. :-) )
>
> The point is: we have two derivations; one for the build (without tests)
> and another for the tests (without build).
>
> ...
>
> Somehow, we could have a “build” build-system and a “test” build-system.
> And the “build object” would be an inputs of the “test object“.

You are trying to reconcile our discussion with what you know about Guix
internals which is both useful and necessary (eventually). However,
since I have less knowledge of said internals at present, I am not able
to meaningfully contribute to what the implementation may look like
yet. What I can do, instead, is to articulate some
invariances/properties that I believe are both desirable and reasonable
(without considering how feasible or not they may be).

First some notation. Let's say the "test metadata" captures the
information of interest: were the tests run, and if so what was their
outcome. A very simple (but sufficient for our purposes) datatype would
be the union of null, #t and #f (where the test outcome is a
boolean). It's possible that in practice, "test metadata" may have
additional information (for instance a reference to the "build output"
in the store), but we'll ignore that for now. I'll use "test output" to
mean the combination of "test metadata" and "build output" where "build
output" is also the input to the "thing that generates the 'test
metadata'". So we have the property that the "test output" extends the
"build output" by providing some companion information (i.e., the "test
metadata").

The invariants of interest are about what things are considered
equivalent. To my understanding this is captured in the Guix notion of
what is and isn't considered a Substitute.

1. It should be possible to discard test metadata: We should be able to
   use "test output" as a substitute for "build output". I.e., a
   derivation that doesn't demand that we run the tests ought not to
   care whether or not we did.

2. "build output" can be used as a substitute for "test output" with
   null "test metadata".

3. It shouldn't be possible to vacuously manufacture test outcomes:
   "build output" cannot be used as a substitute for "test output" with
   either #t or #f "test metadata".

If our hypothetical build system (say, ds-build-system) were to admit
the above invariances, do you foresee some complications that may arise
that need to be addressed?

> Well, somehow perhaps some revamp of the <package> record.

Perhaps, but I am not quite there yet to consider how it might be
implemented, because what "it" is is still not sufficiently clear to me.

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

This email may contain information that is privileged, confidential
and/or exempt from disclosure.  No waiver whatsoever is intended by
sending this e-mail which is intended only for the named recipient(s).
Unauthorized use, dissemination or copying is prohibited.  If you
receive this email in error, please notify the sender and destroy all
copies of this email.

Re: Turning off tests leads to a different store item

[Simon Tournier]

Hi,

On Thu, 02 Nov 2023 at 18:54, Suhail <suhail@bayesians.ca> wrote:

> If our hypothetical build system (say, ds-build-system) were to admit
> the above invariances, do you foresee some complications that may arise
> that need to be addressed?

Instead of trying to answer to this question, what comes to my mind when
reading is:

https://www.joachim-breitner.de/blog/743-Build_tool_semantic_aware_build_systems

and also the paper: Build à la carte

    https://dl.acm.org/doi/pdf/10.1145/3236774

Cheers,
simon

Re: Turning off tests leads to a different store item

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

On 2023-11-02 15:25:33 +0000, Suhail wrote:
> [..]
> 
> The hypothetical test derivation leaves the build artifact unchanged,
> but does communicate some "side" information. It's like a fixed-output
> derivation carrying some metadata (further elaboration below).

I am not aware of any, but I wonder if there are test suites that actually
modify the built binary, for example by patching a bool flag `was_tested' to #t.
Runtime behavior could then differ based on the flag.  Do you have some ideas
how to detect/deal with this?

T.
-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Turning off tests leads to a different store item

[Suhail]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> On Thu, 02 Nov 2023 at 18:54, Suhail <suhail@bayesians.ca> wrote:
>
>> If our hypothetical build system (say, ds-build-system) were to admit
>> the above invariances, do you foresee some complications that may arise
>> that need to be addressed?
>
> Instead of trying to answer to this question, what comes to my mind when
> reading is:
>
> https://www.joachim-breitner.de/blog/743-Build_tool_semantic_aware_build_systems

Thank you for that reference. Yes, that's in the right spirit. The
motivational scenario (packaging data science applications) as well as
other specifics are of particular interest to me. From the blog:

> A build system like nix is designed around the idea that builds are
> relatively cheap, and that any change in a dependency ought to trigger
> a rebuild, just to be sure that all build outputs are up-to-date.

In addition to the above assumption, software engineering makes another
assumption that (at times) needs to be relaxed in data science. In
software engineering, the tests are assumed to have a definitive notion
of "pass" vs "fail".  Whereas in data science applications, the "tests"
may not have a definitive "pass" or "fail" semantics and/or may
entertain multiple such interpretations. While I have been calling these
tests so far, perhaps it may be better to call these something other
than that. If so, "statistical report" may be more descriptive.

Regardless, we reach the point where it would help us if we had some
control over defining what is or isn't substitutable for another.

While the blog post is considering this more generally via the use of
"equivalence proxies" (e.g., treat A as being substitutable for B if the
value of the proxy evaluated at A is the same as the value of the proxy
evaluated at B), I am currently limiting myself to the more specific
case where I want to be able to treat A as being substitutable for B
when I am able to define a computation that takes A as an input and is
able to generate B. Note that I am not requiring that I be able to
generate A from B.

So, in Guix, if I want to create a "type of substitute" which instead of
downloading over HTTP/HTTPS runs some other computation how would I go
about doing that?

> and also the paper: Build à la carte
>
>     https://dl.acm.org/doi/pdf/10.1145/3236774

Thank you for that useful reference.

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

This email may contain information that is privileged, confidential
and/or exempt from disclosure.  No waiver whatsoever is intended by
sending this e-mail which is intended only for the named recipient(s).
Unauthorized use, dissemination or copying is prohibited.  If you
receive this email in error, please notify the sender and destroy all
copies of this email.

Re: Turning off tests leads to a different store item

[Suhail]

"Tomas Volf" <~@wolfsden.cz> writes:

> On 2023-11-02 15:25:33 +0000, Suhail wrote:
>> [..]
>> 
>> The hypothetical test derivation leaves the build artifact unchanged,
>> but does communicate some "side" information. It's like a fixed-output
>> derivation carrying some metadata (further elaboration below).
>
> I am not aware of any, but I wonder if there are test suites that actually
> modify the built binary, for example by patching a bool flag `was_tested' to #t.
> Runtime behavior could then differ based on the flag.  Do you have some ideas
> how to detect/deal with this?

When you say "built binary" do you mean as an input to the "hypothetical
test derivation", or the purported output of the test derivation which
was supposed to be identical to the input but has been modified? If
former, you'd need to ensure that the derivation has read-only access to
the input (I believe/hope Guix already does this). If the latter, what's
material is whether or not the original input can be recovered (with
verification based on the comparison of some signature).

To add more context, I am presently interested in situations where a
"hypothetical test derivation" (say, A) passes its input (say, input_1
which was generated as the output of another derivation B) as part of
its output (say, output_A), but adds additional metadata (say,
metadata_A) in the process.

I.e., let output_A = (input_1, metadata_A)

I am also interested in being able to treat output_A as a substitute for
input_1 given a function (say, f) which describes how to obtain input_1
from output_A and whose result matches the known signature of input_1.

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

Re: Turning off tests leads to a different store item

[Simon Tournier]

Hi,

On Fri, 03 Nov 2023 at 14:13, Suhail <suhail@bayesians.ca> wrote:

> In addition to the above assumption, software engineering makes another
> assumption that (at times) needs to be relaxed in data science. In
> software engineering, the tests are assumed to have a definitive notion
> of "pass" vs "fail".  Whereas in data science applications, the "tests"
> may not have a definitive "pass" or "fail" semantics and/or may
> entertain multiple such interpretations. While I have been calling these
> tests so far, perhaps it may be better to call these something other
> than that. If so, "statistical report" may be more descriptive.

Maybe, what you would like is the description of a workflow, e.g.,

    https://guixwl.org/

Cheers,
simon

Re: Turning off tests leads to a different store item

[Maxim Cournoyer]

Hi,

Suhail <suhail@bayesians.ca> writes:

> Felix Lechner via <help-guix@gnu.org> writes:
>
>> On Tue, Oct 31 2023, Julien Lepiller wrote:
>>
>>> You could skip tests, but that would build a different derivation
>>> tree, so it might affect reproducibility
>>
>> Those shortfalls are a consequence of our packaging habits. In an ideal
>> world, would running a test suite ever build a different derivation?

You seem to be confusing 'derivation', which is the build recipe
description that a package "compiles" to, with the package output, the
result of executing the derivation.

While I wouldn't expect the package output to change unless tests are
installed when disabling the test suite, the derivation changes to
account for the different build arguments used (e.g. that #:tests? #t vs
#:tests? #f).

Running tests *may* cause the output derivation to be different.  I
remember a bug in Python that would cause some time stamps to be
embedded in the .pyc when running the test suite (!).  I think it's
safer to assume that a derivation with test enabled is not the same as a
derivation with tests disabled.

There was the "Intensional Store" RFC in Nix that tried to address that
[0]; it may be worth reading but it looks like it comes with a lot of
caveats and complexity.

[0]  https://github.com/NixOS/rfcs/pull/17

-- 
Thanks,
Maxim

Re: Turning off tests leads to a different store item

[Csepp]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi,
>
> Suhail <suhail@bayesians.ca> writes:
>
>> Felix Lechner via <help-guix@gnu.org> writes:
>>
>>> On Tue, Oct 31 2023, Julien Lepiller wrote:
>>>
>>>> You could skip tests, but that would build a different derivation
>>>> tree, so it might affect reproducibility
>>>
>>> Those shortfalls are a consequence of our packaging habits. In an ideal
>>> world, would running a test suite ever build a different derivation?
>
> You seem to be confusing 'derivation', which is the build recipe
> description that a package "compiles" to, with the package output, the
> result of executing the derivation.
>
> While I wouldn't expect the package output to change unless tests are
> installed when disabling the test suite, the derivation changes to
> account for the different build arguments used (e.g. that #:tests? #t vs
> #:tests? #f).
>
> Running tests *may* cause the output derivation to be different.  I
> remember a bug in Python that would cause some time stamps to be
> embedded in the .pyc when running the test suite (!).  I think it's
> safer to assume that a derivation with test enabled is not the same as a
> derivation with tests disabled.
>
> There was the "Intensional Store" RFC in Nix that tried to address that
> [0]; it may be worth reading but it looks like it comes with a lot of
> caveats and complexity.
>
> [0]  https://github.com/NixOS/rfcs/pull/17

Another way would be to run tests separately, but that might be even
more complicated.
I had the idea of simply exporting the build directory into a separate
output, or maybe only that would be an output and the regular package
just takes that as an input and only runs the install step, and the
test "package" just runs the test target.

Re: Turning off tests leads to a different store item

[Felix Lechner via]

Hi,

On Tue, Nov 07 2023, Maxim Cournoyer wrote:

>> Felix Lechner via <help-guix@gnu.org> writes:
>>
>>> Those shortfalls are a consequence of our packaging habits. In an ideal
>>> world, would running a test suite ever build a different derivation?
>
> You seem to be confusing 'derivation', which is the build recipe
> description that a package "compiles" to, with the package output, the
> result of executing the derivation.

Maybe that comment was not a response to mine. Either way, please allow
me to restate what I meant:

A source tree should produce the same derivation regardless of whether
build-time tests ran or didn't. It should also pull the same substitute.

Kind regards
Felix

Re: Turning off tests leads to a different store item

[Maxim Cournoyer]

Hi Felix,

Felix Lechner <felix.lechner@lease-up.com> writes:

> Hi,
>
> On Tue, Nov 07 2023, Maxim Cournoyer wrote:
>
>>> Felix Lechner via <help-guix@gnu.org> writes:
>>>
>>>> Those shortfalls are a consequence of our packaging habits. In an ideal
>>>> world, would running a test suite ever build a different derivation?
>>
>> You seem to be confusing 'derivation', which is the build recipe
>> description that a package "compiles" to, with the package output, the
>> result of executing the derivation.

[...]

> A source tree should produce the same derivation regardless of whether
> build-time tests ran or didn't. It should also pull the same substitute.

A source tree doesn't produce a derivation.  A derivation is the
complete build recipe that captures the source and the package
definition, that when built by the daemon produces a store item.

Earlier in the thread, I've mentioned that running a test suite can have
side effects such as happened in the past in Python with timestamps
being written to the loaded .pyc files when running test suites; we
can't ensure the test suite has no side-effects on the rest of the
package, so we can't simply assume building with #:tests? #f produces
the same as when building with #:tests? #t.

And to come back to the original idea: building with the tests run makes
sense for a distribution -- it's one of the rare things we have at our
disposal to ensure things are properly integrated and keep working.

-- 
Thanks,
Maxim

Re: Turning off tests leads to a different store item

[Felix Lechner via]

Hi Maxim,

On Wed, Nov 08 2023, Maxim Cournoyer wrote:

> A source tree doesn't produce a derivation.  A derivation is the
> complete build recipe that captures the source and the package
> definition, that when built by the daemon produces a store item.

Okay, thanks! Now I'm going to get it right:

The store item that is produced should not change whether build-time
tests run or not.

It does not make sense (and wastes resources) to rebuild a consuming
package because build-time tests were enabled or disabled in an input.

The historical version of openssl gave rise to this thread. It did not
build anymore because the tests no longer worked with the certificates
shipped in that release (a common problem in TLS libraries). Rebuilding
openssl without running the tests rendered the rebuild useless because
it produced a different store item. That should not happen.

Does that make more sense?

Kind regards
Felix

Re: Turning off tests leads to a different store item

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

On 2023-11-08, Felix Lechner via wrote:
> On Wed, Nov 08 2023, Maxim Cournoyer wrote:
>> A source tree doesn't produce a derivation.  A derivation is the
>> complete build recipe that captures the source and the package
>> definition, that when built by the daemon produces a store item.
>
> Okay, thanks! Now I'm going to get it right:
>
> The store item that is produced should not change whether build-time
> tests run or not.
>
> It does not make sense (and wastes resources) to rebuild a consuming
> package because build-time tests were enabled or disabled in an input.
>
> The historical version of openssl gave rise to this thread. It did not
> build anymore because the tests no longer worked with the certificates
> shipped in that release (a common problem in TLS libraries). Rebuilding
> openssl without running the tests rendered the rebuild useless because
> it produced a different store item. That should not happen.
>
> Does that make more sense?

I do not really think people are misunderstanding you, more that your
*should* does not align with reality in a way that guix can depend on;
build-time tests *do* affect the build result in some cases.

You can write tests that can be run outside the build environment, like
Debian's autopkgtest, but those may need to actually be different tests
and most upstreams do not provide them yet...

The only way to ensure they do not change the build results is even more
resource-intensive; systematically run the build without tests and then
run the build with tests and compare the store items... assuming the
build is reproducible in the first place, which is not yet 100% reliable
either, unfortunately.


This does seem well outside the context of help-guix (drop from CC in
replies?) at this point, and should perhaps be moved to guix-devel(added
to CC), as it may require significant changes to guix; it is not a
simple matter of helping someone figure out how to do something with
guix.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Turning off tests leads to a different store item

[Saku Laesvuori]

[-- Attachment #1: Type: text/plain, Size: 1818 bytes --]

On Wed, Nov 08, 2023 at 10:18:40AM -0800, Vagrant Cascadian wrote:
> On 2023-11-08, Felix Lechner via wrote:
> > On Wed, Nov 08 2023, Maxim Cournoyer wrote:
> >> A source tree doesn't produce a derivation.  A derivation is the
> >> complete build recipe that captures the source and the package
> >> definition, that when built by the daemon produces a store item.
> >
> > Okay, thanks! Now I'm going to get it right:
> >
> > The store item that is produced should not change whether build-time
> > tests run or not.
> >
> > It does not make sense (and wastes resources) to rebuild a consuming
> > package because build-time tests were enabled or disabled in an input.
> 
> I do not really think people are misunderstanding you, more that your
> *should* does not align with reality in a way that guix can depend on;
> build-time tests *do* affect the build result in some cases.
> 
> [...]
> 
> The only way to ensure they do not change the build results is even more
> resource-intensive; systematically run the build without tests and then
> run the build with tests and compare the store items... assuming the
> build is reproducible in the first place, which is not yet 100% reliable
> either, unfortunately.

There is another way: simply preventing the tests from changing the
resulting store item. For example, the package could first be built
without tests and then that build tree could be copied to the build tree
of the build with tests enabled. The result of that build could then
just be copied from the testless build, ignoring any changes the test
suite has made to the build tree. I'm not confident enough in my
understanding of how Guix builds things to say for sure that this
specific method would work, but I am quite sure that the general idea is
implementable.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Turning off tests leads to a different store item

[Simon Tournier]

Hi,

On Wed, 8 Nov 2023 at 20:20, Saku Laesvuori <saku@laesvuori.fi> wrote:

> There is another way: simply preventing the tests from changing the
> resulting store item. For example, the package could first be built
> without tests and then that build tree could be copied to the build tree
> of the build with tests enabled.

Somehow, the store would need to keep all the build intermediary
artifacts produced, right?  For instance, consider the extreme case
where the build phase produces .o artifact files and the tests for
whatever reasons relies on these temporary artifacts.

Well, we had a quick chat with Josselin and Andreas about separating
the tests at https://hpc.guix.info/events/2023/workshop/program/
And my understanding of the rough conclusion we had: it is not easy
and the evil are in all the details.  For example autotools: somehow
"make check" is connected in one way or the other to "make" and/or
"make install".  Somehow, the complete build tree (with intermediary
artifacts as .o) should keep in the store.

From a pragmatical point of view, there is packages where the tests
cannot be totally separated from from the temporary build, therefore
the question seems: how do these cases compare to the other regular
cases?  What is the ratio?  Is the rule about many corner cases
without a clear "regular"?  Or are they just few corner cases?

Cheers,
simon

Re: Turning off tests leads to a different store item

[Maxim Cournoyer]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Wed, 8 Nov 2023 at 20:20, Saku Laesvuori <saku@laesvuori.fi> wrote:
>
>> There is another way: simply preventing the tests from changing the
>> resulting store item. For example, the package could first be built
>> without tests and then that build tree could be copied to the build tree
>> of the build with tests enabled.
>
> Somehow, the store would need to keep all the build intermediary
> artifacts produced, right?  For instance, consider the extreme case
> where the build phase produces .o artifact files and the tests for
> whatever reasons relies on these temporary artifacts.
>
> Well, we had a quick chat with Josselin and Andreas about separating
> the tests at https://hpc.guix.info/events/2023/workshop/program/
> And my understanding of the rough conclusion we had: it is not easy
> and the evil are in all the details.  For example autotools: somehow
> "make check" is connected in one way or the other to "make" and/or
> "make install".  Somehow, the complete build tree (with intermediary
> artifacts as .o) should keep in the store.
>
> From a pragmatical point of view, there is packages where the tests
> cannot be totally separated from from the temporary build, therefore
> the question seems: how do these cases compare to the other regular
> cases?  What is the ratio?  Is the rule about many corner cases
> without a clear "regular"?  Or are they just few corner cases?

One easy-ish way, which would be kind of ugly because coupled to the
specific file system capabilities (e.g Btrfs), would be to leverage CoW
features and create a Btrfs snapshot at the beginning of the test suite,
and reverting to it after it's run.

But even that is not fullproof; that'd only protect the build directory,
say, not the store location (some check phases are moved after
installation).

I agree it looks tricky to get it right (and even trickier to prove/test
for it) :-).

I think the lower fruits are in looking at making the test suite of the
few common offenders more robust (using libfaketime or the likes) to
prevent (re)occurrences of time bombs in the future.

-- 
Thanks,
Maxim

Re: Turning off tests leads to a different store item

[Simon Tournier]

Hi,

On Wed, 08 Nov 2023 at 22:17, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> I agree it looks tricky to get it right (and even trickier to prove/test
> for it) :-).

Yeah.  I have tried a rough “proof-of-concept” i.e., two derivations:
one which deletes ’check’ phase and the other which deletes ’build’
phase and depends on the former.

        Re: Turning off tests leads to a different store item
        Simon Tournier <zimon.toutoune@gmail.com>
        Thu, 02 Nov 2023 18:02:18 +0100
        id:86y1fgm6lh.fsf@gmail.com
        https://lists.gnu.org/archive/html/help-guix/2023-11
        https://yhetil.org/guix/86y1fgm6lh.fsf@gmail.com

In this simple case, it works. :-)

But to have something robust, IMHO, it would probably mean 1. create
other objects (record) different from <package> and revamp the build
systems.  And then 2. rebuild many packages for 3. evaluate the ratio
between the number of packages that works this way vs the number of
packages that fails this way.  Oof!  That’s a fun project… :-)


> I think the lower fruits are in looking at making the test suite of the
> few common offenders more robust (using libfaketime or the likes) to
> prevent (re)occurrences of time bombs in the future.

I agree.  On a side note, one of the issue is the time of some tests.
Sometimes, packaging is frustrating: build takes ages, then you fix some
tests, think it will be good, re-launch “guix build”, another test
failing, repeat.  It could nice to be able to cache the result of the
build phase.

Cheers,
simon

Re: Turning off tests leads to a different store item

[Maxim Cournoyer]

Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

[...]

>> I think the lower fruits are in looking at making the test suite of the
>> few common offenders more robust (using libfaketime or the likes) to
>> prevent (re)occurrences of time bombs in the future.
>
> I agree.  On a side note, one of the issue is the time of some tests.
> Sometimes, packaging is frustrating: build takes ages, then you fix some
> tests, think it will be good, re-launch “guix build”, another test
> failing, repeat.  It could nice to be able to cache the result of the
> build phase.

I usually end up working in the /tmp/guix-build-* failed build of a
package for these situations; then I don't need to rebuild the whole
thing for each test suite run, and can test changes directly without
proper patches while working toward a fix.

-- 
Thanks,
Maxim

Re: Turning off tests leads to a different store item

[Simon Tournier]

Hi Maxim,

On Thu, 09 Nov 2023 at 10:04, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

>> I agree.  On a side note, one of the issue is the time of some tests.
>> Sometimes, packaging is frustrating: build takes ages, then you fix some
>> tests, think it will be good, re-launch “guix build”, another test
>> failing, repeat.  It could nice to be able to cache the result of the
>> build phase.
>
> I usually end up working in the /tmp/guix-build-* failed build of a
> package for these situations; then I don't need to rebuild the whole
> thing for each test suite run, and can test changes directly without
> proper patches while working toward a fix.

Do you know all the command lines equivalent for each phase run by all
the build systems? ;-)

Well, this workflow seems appealing but it never works for me on
concrete situations.  Most of the time, my issue does not come from
“what to do from command line” but “how to do it inside the Guix
recipe”.

For example, this test suite adjustment:

          (add-before 'install 'disable-network-tests
            (lambda _
              (substitute* "test/runtests.jl"
                (("\"async.jl") "# \"async.jl")
                (("\"client.jl") "# \"client.jl"))
              (substitute* "test/aws4.jl"
                (("@testset.*HTTP.request with AWS authentication.*" all)
                 (string-append all "return\n")))
              (substitute* "test/insert_layers.jl"
                (("@testset.*Inserted final layer runs handler.*" all)
                 (string-append all "return\n")))
              (substitute* "test/multipart.jl"
                (("@testset \"Setting of Content-Type.*" all)
                 (string-append all "return\n"))
                (("@testset \"Deprecation of .*" all)
                 (string-append all "return\n")))
              (substitute* "test/websockets.jl"
                (("@testset.*External Host.*" all)
                 (string-append all "return\n")))
              (substitute* "test/messages.jl"
                (("@testset.*Read methods.*" all)
                 (string-append all "return\n"))
                (("@testset.*Body - .*" all)
                 (string-append all "return\n"))
                (("@testset.*Write to file.*" all)
                 (string-append all "return\n")))
              (substitute* "test/cookies.jl"
                (("@testset.*Set-Cookie casing.*" all)
                 (string-append all "return\n"))))))))

To have the correct replacement rules, I almost never get it right at
the very first try, and the feedback loop is poor.  In some case, the
replacement is done after 'unpack, so I can just kill the build process
and check inside /tmp/guix-build-* if the output is the expected one.

However, depending on the test suite, sometimes, it errors at the first
failing test and does not report all the failing cases.  For example, in
the previous snippet, I could have to build then check, fail and repeat
something like 11 times, adding some try-error attempts for the correct
replacement rule.

Cheers,
simon

Re: Turning off tests leads to a different store item

[Maxim Cournoyer]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi Maxim,
>
> On Thu, 09 Nov 2023 at 10:04, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>>> I agree.  On a side note, one of the issue is the time of some tests.
>>> Sometimes, packaging is frustrating: build takes ages, then you fix some
>>> tests, think it will be good, re-launch “guix build”, another test
>>> failing, repeat.  It could nice to be able to cache the result of the
>>> build phase.
>>
>> I usually end up working in the /tmp/guix-build-* failed build of a
>> package for these situations; then I don't need to rebuild the whole
>> thing for each test suite run, and can test changes directly without
>> proper patches while working toward a fix.
>
> Do you know all the command lines equivalent for each phase run by all
> the build systems? ;-)

It depends of the build system, but for gnu-build-system, it's typically
just sourcing environment-variables then running 'make check', I think.
Note that 'live' environment variables are captured in the
'environment-variables' file, which is handy.

> Well, this workflow seems appealing but it never works for me on
> concrete situations.  Most of the time, my issue does not come from
> “what to do from command line” but “how to do it inside the Guix
> recipe”.
>
> For example, this test suite adjustment:
>
>           (add-before 'install 'disable-network-tests
>             (lambda _
>               (substitute* "test/runtests.jl"
>                 (("\"async.jl") "# \"async.jl")
>                 (("\"client.jl") "# \"client.jl"))

[...]
>                 (("@testset.*Set-Cookie casing.*" all)
>                  (string-append all "return\n"))))))))
>
> To have the correct replacement rules, I almost never get it right at
> the very first try, and the feedback loop is poor.  In some case, the
> replacement is done after 'unpack, so I can just kill the build process
> and check inside /tmp/guix-build-* if the output is the expected one.

I've used 'guix repl' in a failed build before, running the same
commands directly in the tree to observe their effects (e.g.,
substitutions) directly.

-- 
Thanks,
Maxim