Re: Turning off tests leads to a different store item

Re: Turning off tests leads to a different store item

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

On 2023-11-08, Felix Lechner via wrote:
> On Wed, Nov 08 2023, Maxim Cournoyer wrote:
>> A source tree doesn't produce a derivation.  A derivation is the
>> complete build recipe that captures the source and the package
>> definition, that when built by the daemon produces a store item.
>
> Okay, thanks! Now I'm going to get it right:
>
> The store item that is produced should not change whether build-time
> tests run or not.
>
> It does not make sense (and wastes resources) to rebuild a consuming
> package because build-time tests were enabled or disabled in an input.
>
> The historical version of openssl gave rise to this thread. It did not
> build anymore because the tests no longer worked with the certificates
> shipped in that release (a common problem in TLS libraries). Rebuilding
> openssl without running the tests rendered the rebuild useless because
> it produced a different store item. That should not happen.
>
> Does that make more sense?

I do not really think people are misunderstanding you, more that your
*should* does not align with reality in a way that guix can depend on;
build-time tests *do* affect the build result in some cases.

You can write tests that can be run outside the build environment, like
Debian's autopkgtest, but those may need to actually be different tests
and most upstreams do not provide them yet...

The only way to ensure they do not change the build results is even more
resource-intensive; systematically run the build without tests and then
run the build with tests and compare the store items... assuming the
build is reproducible in the first place, which is not yet 100% reliable
either, unfortunately.


This does seem well outside the context of help-guix (drop from CC in
replies?) at this point, and should perhaps be moved to guix-devel(added
to CC), as it may require significant changes to guix; it is not a
simple matter of helping someone figure out how to do something with
guix.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Turning off tests leads to a different store item

[Saku Laesvuori]

[-- Attachment #1: Type: text/plain, Size: 1818 bytes --]

On Wed, Nov 08, 2023 at 10:18:40AM -0800, Vagrant Cascadian wrote:
> On 2023-11-08, Felix Lechner via wrote:
> > On Wed, Nov 08 2023, Maxim Cournoyer wrote:
> >> A source tree doesn't produce a derivation.  A derivation is the
> >> complete build recipe that captures the source and the package
> >> definition, that when built by the daemon produces a store item.
> >
> > Okay, thanks! Now I'm going to get it right:
> >
> > The store item that is produced should not change whether build-time
> > tests run or not.
> >
> > It does not make sense (and wastes resources) to rebuild a consuming
> > package because build-time tests were enabled or disabled in an input.
> 
> I do not really think people are misunderstanding you, more that your
> *should* does not align with reality in a way that guix can depend on;
> build-time tests *do* affect the build result in some cases.
> 
> [...]
> 
> The only way to ensure they do not change the build results is even more
> resource-intensive; systematically run the build without tests and then
> run the build with tests and compare the store items... assuming the
> build is reproducible in the first place, which is not yet 100% reliable
> either, unfortunately.

There is another way: simply preventing the tests from changing the
resulting store item. For example, the package could first be built
without tests and then that build tree could be copied to the build tree
of the build with tests enabled. The result of that build could then
just be copied from the testless build, ignoring any changes the test
suite has made to the build tree. I'm not confident enough in my
understanding of how Guix builds things to say for sure that this
specific method would work, but I am quite sure that the general idea is
implementable.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Turning off tests leads to a different store item

[Simon Tournier]

Hi,

On Wed, 8 Nov 2023 at 20:20, Saku Laesvuori <saku@laesvuori.fi> wrote:

> There is another way: simply preventing the tests from changing the
> resulting store item. For example, the package could first be built
> without tests and then that build tree could be copied to the build tree
> of the build with tests enabled.

Somehow, the store would need to keep all the build intermediary
artifacts produced, right?  For instance, consider the extreme case
where the build phase produces .o artifact files and the tests for
whatever reasons relies on these temporary artifacts.

Well, we had a quick chat with Josselin and Andreas about separating
the tests at https://hpc.guix.info/events/2023/workshop/program/
And my understanding of the rough conclusion we had: it is not easy
and the evil are in all the details.  For example autotools: somehow
"make check" is connected in one way or the other to "make" and/or
"make install".  Somehow, the complete build tree (with intermediary
artifacts as .o) should keep in the store.

From a pragmatical point of view, there is packages where the tests
cannot be totally separated from from the temporary build, therefore
the question seems: how do these cases compare to the other regular
cases?  What is the ratio?  Is the rule about many corner cases
without a clear "regular"?  Or are they just few corner cases?

Cheers,
simon

Re: Turning off tests leads to a different store item

[Maxim Cournoyer]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Wed, 8 Nov 2023 at 20:20, Saku Laesvuori <saku@laesvuori.fi> wrote:
>
>> There is another way: simply preventing the tests from changing the
>> resulting store item. For example, the package could first be built
>> without tests and then that build tree could be copied to the build tree
>> of the build with tests enabled.
>
> Somehow, the store would need to keep all the build intermediary
> artifacts produced, right?  For instance, consider the extreme case
> where the build phase produces .o artifact files and the tests for
> whatever reasons relies on these temporary artifacts.
>
> Well, we had a quick chat with Josselin and Andreas about separating
> the tests at https://hpc.guix.info/events/2023/workshop/program/
> And my understanding of the rough conclusion we had: it is not easy
> and the evil are in all the details.  For example autotools: somehow
> "make check" is connected in one way or the other to "make" and/or
> "make install".  Somehow, the complete build tree (with intermediary
> artifacts as .o) should keep in the store.
>
> From a pragmatical point of view, there is packages where the tests
> cannot be totally separated from from the temporary build, therefore
> the question seems: how do these cases compare to the other regular
> cases?  What is the ratio?  Is the rule about many corner cases
> without a clear "regular"?  Or are they just few corner cases?

One easy-ish way, which would be kind of ugly because coupled to the
specific file system capabilities (e.g Btrfs), would be to leverage CoW
features and create a Btrfs snapshot at the beginning of the test suite,
and reverting to it after it's run.

But even that is not fullproof; that'd only protect the build directory,
say, not the store location (some check phases are moved after
installation).

I agree it looks tricky to get it right (and even trickier to prove/test
for it) :-).

I think the lower fruits are in looking at making the test suite of the
few common offenders more robust (using libfaketime or the likes) to
prevent (re)occurrences of time bombs in the future.

-- 
Thanks,
Maxim

Re: Turning off tests leads to a different store item

[Simon Tournier]

Hi,

On Wed, 08 Nov 2023 at 22:17, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> I agree it looks tricky to get it right (and even trickier to prove/test
> for it) :-).

Yeah.  I have tried a rough “proof-of-concept” i.e., two derivations:
one which deletes ’check’ phase and the other which deletes ’build’
phase and depends on the former.

        Re: Turning off tests leads to a different store item
        Simon Tournier <zimon.toutoune@gmail.com>
        Thu, 02 Nov 2023 18:02:18 +0100
        id:86y1fgm6lh.fsf@gmail.com
        https://lists.gnu.org/archive/html/help-guix/2023-11
        https://yhetil.org/guix/86y1fgm6lh.fsf@gmail.com

In this simple case, it works. :-)

But to have something robust, IMHO, it would probably mean 1. create
other objects (record) different from <package> and revamp the build
systems.  And then 2. rebuild many packages for 3. evaluate the ratio
between the number of packages that works this way vs the number of
packages that fails this way.  Oof!  That’s a fun project… :-)


> I think the lower fruits are in looking at making the test suite of the
> few common offenders more robust (using libfaketime or the likes) to
> prevent (re)occurrences of time bombs in the future.

I agree.  On a side note, one of the issue is the time of some tests.
Sometimes, packaging is frustrating: build takes ages, then you fix some
tests, think it will be good, re-launch “guix build”, another test
failing, repeat.  It could nice to be able to cache the result of the
build phase.

Cheers,
simon

Re: Turning off tests leads to a different store item

[Maxim Cournoyer]

Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

[...]

>> I think the lower fruits are in looking at making the test suite of the
>> few common offenders more robust (using libfaketime or the likes) to
>> prevent (re)occurrences of time bombs in the future.
>
> I agree.  On a side note, one of the issue is the time of some tests.
> Sometimes, packaging is frustrating: build takes ages, then you fix some
> tests, think it will be good, re-launch “guix build”, another test
> failing, repeat.  It could nice to be able to cache the result of the
> build phase.

I usually end up working in the /tmp/guix-build-* failed build of a
package for these situations; then I don't need to rebuild the whole
thing for each test suite run, and can test changes directly without
proper patches while working toward a fix.

-- 
Thanks,
Maxim

Re: Turning off tests leads to a different store item

[Simon Tournier]

Hi Maxim,

On Thu, 09 Nov 2023 at 10:04, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

>> I agree.  On a side note, one of the issue is the time of some tests.
>> Sometimes, packaging is frustrating: build takes ages, then you fix some
>> tests, think it will be good, re-launch “guix build”, another test
>> failing, repeat.  It could nice to be able to cache the result of the
>> build phase.
>
> I usually end up working in the /tmp/guix-build-* failed build of a
> package for these situations; then I don't need to rebuild the whole
> thing for each test suite run, and can test changes directly without
> proper patches while working toward a fix.

Do you know all the command lines equivalent for each phase run by all
the build systems? ;-)

Well, this workflow seems appealing but it never works for me on
concrete situations.  Most of the time, my issue does not come from
“what to do from command line” but “how to do it inside the Guix
recipe”.

For example, this test suite adjustment:

          (add-before 'install 'disable-network-tests
            (lambda _
              (substitute* "test/runtests.jl"
                (("\"async.jl") "# \"async.jl")
                (("\"client.jl") "# \"client.jl"))
              (substitute* "test/aws4.jl"
                (("@testset.*HTTP.request with AWS authentication.*" all)
                 (string-append all "return\n")))
              (substitute* "test/insert_layers.jl"
                (("@testset.*Inserted final layer runs handler.*" all)
                 (string-append all "return\n")))
              (substitute* "test/multipart.jl"
                (("@testset \"Setting of Content-Type.*" all)
                 (string-append all "return\n"))
                (("@testset \"Deprecation of .*" all)
                 (string-append all "return\n")))
              (substitute* "test/websockets.jl"
                (("@testset.*External Host.*" all)
                 (string-append all "return\n")))
              (substitute* "test/messages.jl"
                (("@testset.*Read methods.*" all)
                 (string-append all "return\n"))
                (("@testset.*Body - .*" all)
                 (string-append all "return\n"))
                (("@testset.*Write to file.*" all)
                 (string-append all "return\n")))
              (substitute* "test/cookies.jl"
                (("@testset.*Set-Cookie casing.*" all)
                 (string-append all "return\n"))))))))

To have the correct replacement rules, I almost never get it right at
the very first try, and the feedback loop is poor.  In some case, the
replacement is done after 'unpack, so I can just kill the build process
and check inside /tmp/guix-build-* if the output is the expected one.

However, depending on the test suite, sometimes, it errors at the first
failing test and does not report all the failing cases.  For example, in
the previous snippet, I could have to build then check, fail and repeat
something like 11 times, adding some try-error attempts for the correct
replacement rule.

Cheers,
simon

Re: Turning off tests leads to a different store item

[Maxim Cournoyer]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi Maxim,
>
> On Thu, 09 Nov 2023 at 10:04, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>>> I agree.  On a side note, one of the issue is the time of some tests.
>>> Sometimes, packaging is frustrating: build takes ages, then you fix some
>>> tests, think it will be good, re-launch “guix build”, another test
>>> failing, repeat.  It could nice to be able to cache the result of the
>>> build phase.
>>
>> I usually end up working in the /tmp/guix-build-* failed build of a
>> package for these situations; then I don't need to rebuild the whole
>> thing for each test suite run, and can test changes directly without
>> proper patches while working toward a fix.
>
> Do you know all the command lines equivalent for each phase run by all
> the build systems? ;-)

It depends of the build system, but for gnu-build-system, it's typically
just sourcing environment-variables then running 'make check', I think.
Note that 'live' environment variables are captured in the
'environment-variables' file, which is handy.

> Well, this workflow seems appealing but it never works for me on
> concrete situations.  Most of the time, my issue does not come from
> “what to do from command line” but “how to do it inside the Guix
> recipe”.
>
> For example, this test suite adjustment:
>
>           (add-before 'install 'disable-network-tests
>             (lambda _
>               (substitute* "test/runtests.jl"
>                 (("\"async.jl") "# \"async.jl")
>                 (("\"client.jl") "# \"client.jl"))

[...]
>                 (("@testset.*Set-Cookie casing.*" all)
>                  (string-append all "return\n"))))))))
>
> To have the correct replacement rules, I almost never get it right at
> the very first try, and the feedback loop is poor.  In some case, the
> replacement is done after 'unpack, so I can just kill the build process
> and check inside /tmp/guix-build-* if the output is the expected one.

I've used 'guix repl' in a failed build before, running the same
commands directly in the tree to observe their effects (e.g.,
substitutions) directly.

-- 
Thanks,
Maxim