"random check" approach to Guix QA?

"random check" approach to Guix QA?

[Andy Tai]

Hi, currently Guix QA tries to rebuild all dependent packages impacted
by a change (in majority of cases updates to a new version of a
package).  When there are a large number of dependent packages like
say 500 or 1000 or more, rebuilding all that takes a long time and
often the rebuild cannot complete, leaving the patch submission
dangling with no results.

For some things where safety is important, for example import food
safety, the regulatory authority may use random check or random
sampling as a way to check for safety, in case that there is no
resource to physically check every package.

Can the same approach be borrowed here, so when there is large number
of impacted packages from a patch, say larger than 200, Guix QA just
randomly select a subset sample out of these packages and build them,
and in case of (new) failures ask the submitter to fix the (new)
failure? And repeat this as needed.   This way patches can go thru the
QA process more quickly.

Re: "random check" approach to Guix QA?

[Csepp]

Andy Tai <atai@atai.org> writes:

> Hi, currently Guix QA tries to rebuild all dependent packages impacted
> by a change (in majority of cases updates to a new version of a
> package).  When there are a large number of dependent packages like
> say 500 or 1000 or more, rebuilding all that takes a long time and
> often the rebuild cannot complete, leaving the patch submission
> dangling with no results.
>
> For some things where safety is important, for example import food
> safety, the regulatory authority may use random check or random
> sampling as a way to check for safety, in case that there is no
> resource to physically check every package.
>
> Can the same approach be borrowed here, so when there is large number
> of impacted packages from a patch, say larger than 200, Guix QA just
> randomly select a subset sample out of these packages and build them,
> and in case of (new) failures ask the submitter to fix the (new)
> failure? And repeat this as needed.   This way patches can go thru the
> QA process more quickly.

I think it should always do that.  Assign a weight to all derivations
that have to be built and choose from them randomly.
Could use a combination of dependents and estimated build time to comput
the weight.

Re: "random check" approach to Guix QA?

[Andreas Enge]

Hello,

Am Mon, Nov 20, 2023 at 03:11:29PM -0800 schrieb Andy Tai:
> Can the same approach be borrowed here, so when there is large number
> of impacted packages from a patch, say larger than 200, Guix QA just
> randomly select a subset sample out of these packages and build them,
> and in case of (new) failures ask the submitter to fix the (new)
> failure? And repeat this as needed.   This way patches can go thru the
> QA process more quickly.

notice that this approach requires to build a complete connected subgraph
starting with the changed package, so simple random sampling will not work.

For instance, if B depends on A, then C_1 to C_1000 depend on B, one needs
to always include B. Or if B_1 depends on A, B_2 on B_1, and so on until
B_1000 depends on B_999, one would need to only build a few steps
on the path.

Maybe it would make sense, one day when we have lots of idle build
power, to start by building at least the immediate dependents to get
an idea?

Andreas