unofficial mirror of help-guix@gnu.org 
 help / color / mirror / Atom feed
* Re: Turning off tests leads to a different store item
@ 2023-11-02 18:54 Suhail
  2023-11-03  9:33 ` Simon Tournier
  0 siblings, 1 reply; 24+ messages in thread
From: Suhail @ 2023-11-02 18:54 UTC (permalink / raw)
  To: Simon Tournier; +Cc: Suhail, Felix Lechner via, Julien Lepiller, Felix Lechner

Simon Tournier <zimon.toutoune@gmail.com> writes:

> On Thu, 02 Nov 2023 at 15:25, Suhail <suhail@bayesians.ca> wrote:
>
>> Yes, with the test derivation being something like a "fixed-output
>> derivation". [[info:guix#Derivations][From the manual]]:
>
> No, it cannot be a “fixed-output” derivation…
>
> …because we cannot know in advance the expected content hash of the
> tests output.

Yes, hence the "something like". It's similar, but also different in
important ways.

> And from my understanding, one solution would be to have something as
> below.  One “object” for building and another “object” for testing.
>
>     ( Here, I am using the same object namely <package> for the both;
> just to make concrete the discussion. :-) )
>
> The point is: we have two derivations; one for the build (without tests)
> and another for the tests (without build).
>
> ...
>
> Somehow, we could have a “build” build-system and a “test” build-system.
> And the “build object” would be an inputs of the “test object“.

You are trying to reconcile our discussion with what you know about Guix
internals which is both useful and necessary (eventually). However,
since I have less knowledge of said internals at present, I am not able
to meaningfully contribute to what the implementation may look like
yet. What I can do, instead, is to articulate some
invariances/properties that I believe are both desirable and reasonable
(without considering how feasible or not they may be).

First some notation. Let's say the "test metadata" captures the
information of interest: were the tests run, and if so what was their
outcome. A very simple (but sufficient for our purposes) datatype would
be the union of null, #t and #f (where the test outcome is a
boolean). It's possible that in practice, "test metadata" may have
additional information (for instance a reference to the "build output"
in the store), but we'll ignore that for now. I'll use "test output" to
mean the combination of "test metadata" and "build output" where "build
output" is also the input to the "thing that generates the 'test
metadata'". So we have the property that the "test output" extends the
"build output" by providing some companion information (i.e., the "test
metadata").

The invariants of interest are about what things are considered
equivalent. To my understanding this is captured in the Guix notion of
what is and isn't considered a Substitute.

1. It should be possible to discard test metadata: We should be able to
   use "test output" as a substitute for "build output". I.e., a
   derivation that doesn't demand that we run the tests ought not to
   care whether or not we did.

2. "build output" can be used as a substitute for "test output" with
   null "test metadata".

3. It shouldn't be possible to vacuously manufacture test outcomes:
   "build output" cannot be used as a substitute for "test output" with
   either #t or #f "test metadata".

If our hypothetical build system (say, ds-build-system) were to admit
the above invariances, do you foresee some complications that may arise
that need to be addressed?

> Well, somehow perhaps some revamp of the <package> record.

Perhaps, but I am not quite there yet to consider how it might be
implemented, because what "it" is is still not sufficiently clear to me.

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

This email may contain information that is privileged, confidential
and/or exempt from disclosure.  No waiver whatsoever is intended by
sending this e-mail which is intended only for the named recipient(s).
Unauthorized use, dissemination or copying is prohibited.  If you
receive this email in error, please notify the sender and destroy all
copies of this email.



^ permalink raw reply	[flat|nested] 24+ messages in thread
[parent not found: <65429087.0c0a0220.5908c.4d60SMTPIN_ADDED_BROKEN@mx.google.com>]
* Re: Turning off tests leads to a different store item
@ 2023-11-03 14:13 Suhail
  2023-11-05 12:07 ` Simon Tournier
  0 siblings, 1 reply; 24+ messages in thread
From: Suhail @ 2023-11-03 14:13 UTC (permalink / raw)
  To: Simon Tournier; +Cc: Suhail, Felix Lechner via, Julien Lepiller, Felix Lechner

Simon Tournier <zimon.toutoune@gmail.com> writes:

> On Thu, 02 Nov 2023 at 18:54, Suhail <suhail@bayesians.ca> wrote:
>
>> If our hypothetical build system (say, ds-build-system) were to admit
>> the above invariances, do you foresee some complications that may arise
>> that need to be addressed?
>
> Instead of trying to answer to this question, what comes to my mind when
> reading is:
>
> https://www.joachim-breitner.de/blog/743-Build_tool_semantic_aware_build_systems

Thank you for that reference. Yes, that's in the right spirit. The
motivational scenario (packaging data science applications) as well as
other specifics are of particular interest to me. From the blog:

> A build system like nix is designed around the idea that builds are
> relatively cheap, and that any change in a dependency ought to trigger
> a rebuild, just to be sure that all build outputs are up-to-date.

In addition to the above assumption, software engineering makes another
assumption that (at times) needs to be relaxed in data science. In
software engineering, the tests are assumed to have a definitive notion
of "pass" vs "fail".  Whereas in data science applications, the "tests"
may not have a definitive "pass" or "fail" semantics and/or may
entertain multiple such interpretations. While I have been calling these
tests so far, perhaps it may be better to call these something other
than that. If so, "statistical report" may be more descriptive.

Regardless, we reach the point where it would help us if we had some
control over defining what is or isn't substitutable for another.

While the blog post is considering this more generally via the use of
"equivalence proxies" (e.g., treat A as being substitutable for B if the
value of the proxy evaluated at A is the same as the value of the proxy
evaluated at B), I am currently limiting myself to the more specific
case where I want to be able to treat A as being substitutable for B
when I am able to define a computation that takes A as an input and is
able to generate B. Note that I am not requiring that I be able to
generate A from B.

So, in Guix, if I want to create a "type of substitute" which instead of
downloading over HTTP/HTTPS runs some other computation how would I go
about doing that?

> and also the paper: Build à la carte
>
>     https://dl.acm.org/doi/pdf/10.1145/3236774

Thank you for that useful reference.

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

This email may contain information that is privileged, confidential
and/or exempt from disclosure.  No waiver whatsoever is intended by
sending this e-mail which is intended only for the named recipient(s).
Unauthorized use, dissemination or copying is prohibited.  If you
receive this email in error, please notify the sender and destroy all
copies of this email.



^ permalink raw reply	[flat|nested] 24+ messages in thread
* Re: Turning off tests leads to a different store item
@ 2023-11-02 17:25 Suhail
  0 siblings, 0 replies; 24+ messages in thread
From: Suhail @ 2023-11-02 17:25 UTC (permalink / raw)
  To: Greg Hogan
  Cc: Suhail, Simon Tournier, Felix Lechner via, Julien Lepiller,
	Felix Lechner

Greg Hogan <code@greghogan.com> writes:

> On Thu, Nov 2, 2023 at 11:26 AM Suhail <suhail@bayesians.ca> wrote:
>> Perhaps not all. The thing that sets the "check" phase (#:tests?) apart
>> from the rest is that it's an identity transform with a
>> side-effect. i.e., it simply reports on the state of its input (i.e.,
>> the build artifact) leaving the build artifact unchanged. The only other
>> phase in the gnu-build-system that is similar to the "check phase" is
>> the "validate-runpath phase".
>
> Should this say "without side-effects"?

No, I meant "with side-effect", but we have a notational
difference. What is termed side-effect is context dependent, so I'll fix
my terminology and make clearer what I meant. To me, the "check" phase
is something that reports on the state of its input. Said report (i.e.,
whether or not the check phase was run, and if run what its output was)
is what I was using the term "side-effect" for. Perhaps "test state
metadata" would be a better term.

> How does one guarantee that the test phase is free of side-effects?

Translation to remove ambiguity of the term "side-effect": How does one
guarantee that the test phase doesn't modify its input build artifact?

This could be enforced using an overlay layer as you suggest, but we
could also take a different approach. It would be sufficient to be able
to distinguish well-behaved tests (those that don't alter the input
build-artifact) from those that aren't (i.e., those that do alter the
input build-artifact).

> Alternatively, could we make it easier to record a timestamp in the
> manifest for Guix to use as the build clock time?

While that would address the original issue that motivated this
discussion thread, it's not an alternative in the context of the current
discussion. Regardless, I agree that having such a facility would be
useful.

While I do believe the defaults in gnu-build-system could benefit from
revisiting our current treatment of the check and validate-runpath
phases, advocating for that is not my primary goal with this
discussion.

As someone who's not well-versed in Guix internals, I am simply trying
to understand what an alternate build-system (say, ds-build-system)
would look like. One that allows the user/caller of a package to
determine how to handle:

- whether or not the tests were run, and
- whether or not the tests, if run, passed

Similarly for other such "phases".

One application for such a build-system (which is of interest to me)
would be to "package" the result of some "data"-dependent computations
where the notion of what constitutes a "passing build artifact" is less
clearly defined (e.g., models used for algorithmic decision-making).

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

This email may contain information that is privileged, confidential
and/or exempt from disclosure.  No waiver whatsoever is intended by
sending this e-mail which is intended only for the named recipient(s).
Unauthorized use, dissemination or copying is prohibited.  If you
receive this email in error, please notify the sender and destroy all
copies of this email.



^ permalink raw reply	[flat|nested] 24+ messages in thread
[parent not found: <6543bf92.d40a0220.bbcd0.1118SMTPIN_ADDED_BROKEN@mx.google.com>]
* Re: Turning off tests leads to a different store item
@ 2023-11-02 15:25 Suhail
  2023-11-02 17:02 ` Simon Tournier
  2023-11-03 13:08 ` Tomas Volf
  0 siblings, 2 replies; 24+ messages in thread
From: Suhail @ 2023-11-02 15:25 UTC (permalink / raw)
  To: Simon Tournier; +Cc: Suhail, Felix Lechner via, Julien Lepiller, Felix Lechner

Simon Tournier <zimon.toutoune@gmail.com> writes:

> On Wed, 01 Nov 2023 at 17:52, Suhail <suhail@bayesians.ca> wrote:
>
>> If not, why should skipping the tests result in a different
>> derivation tree?
>
> The store path is different because it hashes all the inputs, included
> the builder script; from my understanding.

It certainly seems to be the case. Would you know the specific place(s)
in the source code (in addition to guix/derivations.scm and
guix/store.scm) that would be relevant for this discussion?

On a related note, is Dolstra's phd thesis relevant for these aspects,
or do you know if Guix diverges from Nix in some areas?

> Well, from my understanding, get the same the store path for the same
> package built with or without its tests ...  would mean to have a
> derivation for building and another derivation – referring to the
> former – for testing.

Yes, with the test derivation being something like a "fixed-output
derivation". [[info:guix#Derivations][From the manual]]:

#+begin_quote
Operations such as file downloads and version-control checkouts for
which the expected content hash is known in advance are modeled as
fixed-output derivations. Unlike regular derivations, the outputs of a
fixed-output derivation are independent of its inputs—e.g., a source
code download produces the same result regardless of the download method
and tools being used.
#+end_quote

The hypothetical test derivation leaves the build artifact unchanged,
but does communicate some "side" information. It's like a fixed-output
derivation carrying some metadata (further elaboration below).

> Well, taking this direction, one could imagine a derivation for each
> phases; somehow extend to all phases what it is already done for
> fetching source, build and grafts (guix build --source && guix build
> --no-grafts && guix build).

Perhaps not all. The thing that sets the "check" phase (#:tests?) apart
from the rest is that it's an identity transform with a
side-effect. i.e., it simply reports on the state of its input (i.e.,
the build artifact) leaving the build artifact unchanged. The only other
phase in the gnu-build-system that is similar to the "check phase" is
the "validate-runpath phase".

An alternative (possibly future) version of Guix might allow us to delay
the interpretation/meaning associated with the following cases:

- whether or not the tests were run, and
- whether or not the tests, if run, passed

To be able to do so, we need the "check derivation" be able to
communicate two things (downstream): a summary of the test phase (could
simply be a 3-valued datatype, like a 'Maybe Bool' in Haskell, but could
also be something more elaborate), and the input build artifact (which
remains unchanged). Similarly for the "validate-runpath derivation".

I do not know what the best way to communicate the "side-effect
information" (did the tests run and if so what was the conclusion) would
be. Specifically, should the output of "check derivation" (similarly for
"validate-runpath derivation") include the unchanged build artifact or
not?

Btw, is this still the appropriate mailing list (and more generally,
forum) for this discussion?

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

This email may contain information that is privileged, confidential
and/or exempt from disclosure.  No waiver whatsoever is intended by
sending this e-mail which is intended only for the named recipient(s).
Unauthorized use, dissemination or copying is prohibited.  If you
receive this email in error, please notify the sender and destroy all
copies of this email.



^ permalink raw reply	[flat|nested] 24+ messages in thread
end of thread, other threads:[~2023-11-18  4:39 UTC | newest]

Thread overview: 24+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-11-02 18:54 Turning off tests leads to a different store item Suhail
2023-11-03  9:33 ` Simon Tournier
     [not found] <65429087.0c0a0220.5908c.4d60SMTPIN_ADDED_BROKEN@mx.google.com>
2023-11-07 18:58 ` Maxim Cournoyer
2023-11-07 21:58   ` Csepp
2023-11-08  2:53   ` Felix Lechner via
2023-11-08 14:45     ` Maxim Cournoyer
2023-11-08 17:07       ` Felix Lechner via
2023-11-08 18:18         ` Vagrant Cascadian
2023-11-08 19:20           ` Saku Laesvuori
2023-11-08 22:21             ` Simon Tournier
2023-11-09  3:17               ` Maxim Cournoyer
2023-11-09  7:37                 ` Simon Tournier
2023-11-09 15:04                   ` Maxim Cournoyer
2023-11-16  9:31                     ` Simon Tournier
2023-11-18  4:38                       ` Maxim Cournoyer
  -- strict thread matches above, loose matches on Subject: below --
2023-11-03 14:13 Suhail
2023-11-05 12:07 ` Simon Tournier
2023-11-02 17:25 Suhail
     [not found] <6543bf92.d40a0220.bbcd0.1118SMTPIN_ADDED_BROKEN@mx.google.com>
2023-11-02 16:03 ` Greg Hogan
2023-11-02 15:25 Suhail
2023-11-02 17:02 ` Simon Tournier
2023-11-02 17:46   ` Simon Tournier
2023-11-03 13:08 ` Tomas Volf
2023-11-03 20:44   ` Suhail

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).