all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Build reproducibility metrics
@ 2020-06-03 20:38 Christopher Baines
  2020-06-04 12:44 ` Ludovic Courtès
  2020-06-04 14:54 ` Vagrant Cascadian
  0 siblings, 2 replies; 3+ messages in thread
From: Christopher Baines @ 2020-06-03 20:38 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 2847 bytes --]

Hey,

So there's the guix challenge command for looking at reproducibility
issues, but I've been wanting to more continuously and automatically
monitoring the reproducibility of packages.

The Guix Data Service has had the ability to do this for a few months
now, but there's not been enough data on substitutes. That's starting to
change though.

ci.guix.gnu.org has been doing a reasonable job of building packages,
but the proportion of packages built by bayfront.guix.gnu.org hasn't
always been very high. I'm hoping just having bayfront not build
core-updates and staging will help with this.

I've also been writing and trying to use the Guix Build Coordinator [1]
to build Guix packages and provide substitutes. That has got to the
point where it's not getting stuck every day at least, and there's more
than 80% of packages are available.

1: https://git.cbaines.net/guix/build-coordinator/about/

Combining that with the substitute server operated by Tobias, which has
a pretty awesome substitute availability of over 90% for recent
revisions, not only is there data from 4 different substitute servers to
use in the comparison, but the proportion of packages where there isn't
sufficient data is pretty low, below 10%.

I'm currently using the data.guix-patches.cbaines.net instance of the
Guix Data Service, you can see the package substitute availability for
the latest revision using this URL [1], and the package reproducibility
at this URL [2].

1: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-substitute-availability
2: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-reproducibility

Some caution is needed when interpreting this data. It's most probably
less up to date than what you'd get through running the guix weather or
guix challenge commands, as it takes the Guix Data Service time to query
the data, that querying process isn't very reliable at the moment
either. Additionally, the "matching" percentage could easily go down if
that output is built with a different hash in the future.

While the number itself maybe isn't the most useful thing, I like that
clicking through to the "Not matching" outputs will show a list of
outputs which didn't build reproducibly, which is something that could
help identify reproducibility issues to investigate and fix.

I think things are coming together on the substitute server side. The
goal I have in mind for this is for users of Guix to be able to have
greater trust in the substitutes they use, through trusting substitutes
only if it's been built reproducibly on multiple substitute servers. It
would be great to see work start soon on how guix as a client to
substitute servers might be enhanced to check for reproducibility when
fetching substitutes.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Build reproducibility metrics
  2020-06-03 20:38 Build reproducibility metrics Christopher Baines
@ 2020-06-04 12:44 ` Ludovic Courtès
  2020-06-04 14:54 ` Vagrant Cascadian
  1 sibling, 0 replies; 3+ messages in thread
From: Ludovic Courtès @ 2020-06-04 12:44 UTC (permalink / raw)
  To: Christopher Baines; +Cc: guix-devel

Hello!

Christopher Baines <mail@cbaines.net> skribis:

> I've also been writing and trying to use the Guix Build Coordinator [1]
> to build Guix packages and provide substitutes. That has got to the
> point where it's not getting stuck every day at least, and there's more
> than 80% of packages are available.
>
> 1: https://git.cbaines.net/guix/build-coordinator/about/

Well done!

> Combining that with the substitute server operated by Tobias, which has
> a pretty awesome substitute availability of over 90% for recent
> revisions, not only is there data from 4 different substitute servers to
> use in the comparison, but the proportion of packages where there isn't
> sufficient data is pretty low, below 10%.
>
> I'm currently using the data.guix-patches.cbaines.net instance of the
> Guix Data Service, you can see the package substitute availability for
> the latest revision using this URL [1], and the package reproducibility
> at this URL [2].
>
> 1: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-substitute-availability
> 2: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-reproducibility

That’s really good!  It’s the first time we have this good an overview
of the package reproducibility status.

> Some caution is needed when interpreting this data. It's most probably
> less up to date than what you'd get through running the guix weather or
> guix challenge commands, as it takes the Guix Data Service time to query
> the data, that querying process isn't very reliable at the moment
> either. Additionally, the "matching" percentage could easily go down if
> that output is built with a different hash in the future.
>
> While the number itself maybe isn't the most useful thing, I like that
> clicking through to the "Not matching" outputs will show a list of
> outputs which didn't build reproducibly, which is something that could
> help identify reproducibility issues to investigate and fix.

Yes, definitely.  There’s also always the option of running ‘guix
challenge’ locally.

> I think things are coming together on the substitute server side. The
> goal I have in mind for this is for users of Guix to be able to have
> greater trust in the substitutes they use, through trusting substitutes
> only if it's been built reproducibly on multiple substitute servers. It
> would be great to see work start soon on how guix as a client to
> substitute servers might be enhanced to check for reproducibility when
> fetching substitutes.

Agreed!

I think between that, the reduced bootstrap seeds, and authenticated
checkouts, we’re starting to have a good security story.

Thank you!

Ludo’.


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Build reproducibility metrics
  2020-06-03 20:38 Build reproducibility metrics Christopher Baines
  2020-06-04 12:44 ` Ludovic Courtès
@ 2020-06-04 14:54 ` Vagrant Cascadian
  1 sibling, 0 replies; 3+ messages in thread
From: Vagrant Cascadian @ 2020-06-04 14:54 UTC (permalink / raw)
  To: Christopher Baines, guix-devel; +Cc: rb-general

[-- Attachment #1: Type: text/plain, Size: 2263 bytes --]

On 2020-06-03, Christopher Baines wrote:
> Combining that with the substitute server operated by Tobias, which has
> a pretty awesome substitute availability of over 90% for recent
> revisions, not only is there data from 4 different substitute servers to
> use in the comparison, but the proportion of packages where there isn't
> sufficient data is pretty low, below 10%.
>
> I'm currently using the data.guix-patches.cbaines.net instance of the
> Guix Data Service, you can see the package substitute availability for
> the latest revision using this URL [1], and the package reproducibility
> at this URL [2].
>
> 1: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-substitute-availability
> 2: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-reproducibility
>
> Some caution is needed when interpreting this data. It's most probably
> less up to date than what you'd get through running the guix weather or
> guix challenge commands, as it takes the Guix Data Service time to query
> the data, that querying process isn't very reliable at the moment
> either. Additionally, the "matching" percentage could easily go down if
> that output is built with a different hash in the future.
>
> While the number itself maybe isn't the most useful thing, I like that
> clicking through to the "Not matching" outputs will show a list of
> outputs which didn't build reproducibly, which is something that could
> help identify reproducibility issues to investigate and fix.
>
> I think things are coming together on the substitute server side. The
> goal I have in mind for this is for users of Guix to be able to have
> greater trust in the substitutes they use, through trusting substitutes
> only if it's been built reproducibly on multiple substitute servers. It
> would be great to see work start soon on how guix as a client to
> substitute servers might be enhanced to check for reproducibility when
> fetching substitutes.

Really glad to see great progress on this!

I've CC'ed the reproducible builds list, as others might be interested
to see too! original post:

  https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00034.html

live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2020-06-04 14:55 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-06-03 20:38 Build reproducibility metrics Christopher Baines
2020-06-04 12:44 ` Ludovic Courtès
2020-06-04 14:54 ` Vagrant Cascadian

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.