Build reproducibility metrics

Build reproducibility metrics

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2847 bytes --]

Hey,

So there's the guix challenge command for looking at reproducibility
issues, but I've been wanting to more continuously and automatically
monitoring the reproducibility of packages.

The Guix Data Service has had the ability to do this for a few months
now, but there's not been enough data on substitutes. That's starting to
change though.

ci.guix.gnu.org has been doing a reasonable job of building packages,
but the proportion of packages built by bayfront.guix.gnu.org hasn't
always been very high. I'm hoping just having bayfront not build
core-updates and staging will help with this.

I've also been writing and trying to use the Guix Build Coordinator [1]
to build Guix packages and provide substitutes. That has got to the
point where it's not getting stuck every day at least, and there's more
than 80% of packages are available.

1: https://git.cbaines.net/guix/build-coordinator/about/

Combining that with the substitute server operated by Tobias, which has
a pretty awesome substitute availability of over 90% for recent
revisions, not only is there data from 4 different substitute servers to
use in the comparison, but the proportion of packages where there isn't
sufficient data is pretty low, below 10%.

I'm currently using the data.guix-patches.cbaines.net instance of the
Guix Data Service, you can see the package substitute availability for
the latest revision using this URL [1], and the package reproducibility
at this URL [2].

1: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-substitute-availability
2: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-reproducibility

Some caution is needed when interpreting this data. It's most probably
less up to date than what you'd get through running the guix weather or
guix challenge commands, as it takes the Guix Data Service time to query
the data, that querying process isn't very reliable at the moment
either. Additionally, the "matching" percentage could easily go down if
that output is built with a different hash in the future.

While the number itself maybe isn't the most useful thing, I like that
clicking through to the "Not matching" outputs will show a list of
outputs which didn't build reproducibly, which is something that could
help identify reproducibility issues to investigate and fix.

I think things are coming together on the substitute server side. The
goal I have in mind for this is for users of Guix to be able to have
greater trust in the substitutes they use, through trusting substitutes
only if it's been built reproducibly on multiple substitute servers. It
would be great to see work start soon on how guix as a client to
substitute servers might be enhanced to check for reproducibility when
fetching substitutes.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Build reproducibility metrics

[Ludovic Courtès]

Hello!

Christopher Baines <mail@cbaines.net> skribis:

> I've also been writing and trying to use the Guix Build Coordinator [1]
> to build Guix packages and provide substitutes. That has got to the
> point where it's not getting stuck every day at least, and there's more
> than 80% of packages are available.
>
> 1: https://git.cbaines.net/guix/build-coordinator/about/

Well done!

> Combining that with the substitute server operated by Tobias, which has
> a pretty awesome substitute availability of over 90% for recent
> revisions, not only is there data from 4 different substitute servers to
> use in the comparison, but the proportion of packages where there isn't
> sufficient data is pretty low, below 10%.
>
> I'm currently using the data.guix-patches.cbaines.net instance of the
> Guix Data Service, you can see the package substitute availability for
> the latest revision using this URL [1], and the package reproducibility
> at this URL [2].
>
> 1: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-substitute-availability
> 2: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-reproducibility

That’s really good!  It’s the first time we have this good an overview
of the package reproducibility status.

> Some caution is needed when interpreting this data. It's most probably
> less up to date than what you'd get through running the guix weather or
> guix challenge commands, as it takes the Guix Data Service time to query
> the data, that querying process isn't very reliable at the moment
> either. Additionally, the "matching" percentage could easily go down if
> that output is built with a different hash in the future.
>
> While the number itself maybe isn't the most useful thing, I like that
> clicking through to the "Not matching" outputs will show a list of
> outputs which didn't build reproducibly, which is something that could
> help identify reproducibility issues to investigate and fix.

Yes, definitely.  There’s also always the option of running ‘guix
challenge’ locally.

> I think things are coming together on the substitute server side. The
> goal I have in mind for this is for users of Guix to be able to have
> greater trust in the substitutes they use, through trusting substitutes
> only if it's been built reproducibly on multiple substitute servers. It
> would be great to see work start soon on how guix as a client to
> substitute servers might be enhanced to check for reproducibility when
> fetching substitutes.

Agreed!

I think between that, the reduced bootstrap seeds, and authenticated
checkouts, we’re starting to have a good security story.

Thank you!

Ludo’.

Re: Build reproducibility metrics

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 2263 bytes --]

On 2020-06-03, Christopher Baines wrote:
> Combining that with the substitute server operated by Tobias, which has
> a pretty awesome substitute availability of over 90% for recent
> revisions, not only is there data from 4 different substitute servers to
> use in the comparison, but the proportion of packages where there isn't
> sufficient data is pretty low, below 10%.
>
> I'm currently using the data.guix-patches.cbaines.net instance of the
> Guix Data Service, you can see the package substitute availability for
> the latest revision using this URL [1], and the package reproducibility
> at this URL [2].
>
> 1: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-substitute-availability
> 2: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-reproducibility
>
> Some caution is needed when interpreting this data. It's most probably
> less up to date than what you'd get through running the guix weather or
> guix challenge commands, as it takes the Guix Data Service time to query
> the data, that querying process isn't very reliable at the moment
> either. Additionally, the "matching" percentage could easily go down if
> that output is built with a different hash in the future.
>
> While the number itself maybe isn't the most useful thing, I like that
> clicking through to the "Not matching" outputs will show a list of
> outputs which didn't build reproducibly, which is something that could
> help identify reproducibility issues to investigate and fix.
>
> I think things are coming together on the substitute server side. The
> goal I have in mind for this is for users of Guix to be able to have
> greater trust in the substitutes they use, through trusting substitutes
> only if it's been built reproducibly on multiple substitute servers. It
> would be great to see work start soon on how guix as a client to
> substitute servers might be enhanced to check for reproducibility when
> fetching substitutes.

Really glad to see great progress on this!

I've CC'ed the reproducible builds list, as others might be interested
to see too! original post:

  https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00034.html

live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]