bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked

bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]


I found this when trying to build guile3.0-gnutls:

  guix time-machine --commit=94585fffb23079fe71110e2bf99782eb4ccfa12b -- build --no-grafts --check guile3.0-gnutls
  

FAIL: status-request-revoked
============================

trying NORMAL:-VERS-ALL:+VERS-TLS1.2
received status request
received status request
cert_verify_callback:263: certificate verify status doesn't match: 100402 != 22FAIL status-request-revoked (exit status: 1)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> I found this when trying to build guile3.0-gnutls:
>
>   guix time-machine --commit=94585fffb23079fe71110e2bf99782eb4ccfa12b -- build --no-grafts --check guile3.0-gnutls
>   
>
> FAIL: status-request-revoked
> ============================
>
> trying NORMAL:-VERS-ALL:+VERS-TLS1.2
> received status request
> received status request
> cert_verify_callback:263: certificate verify status doesn't match: 100402 != 22FAIL status-request-revoked (exit status: 1)

This was fixed upstream between 3.6.12 and 3.6.14 with this patch by
Bernhard (it’s a small world!):

--8<---------------cut here---------------start------------->8---
commit ed208fe55f31478732fd6cc394f9576b315a42cd
Author: Bernhard M. Wiedemann <bwiedemann@suse.de>
Date:   Sun Apr 5 15:09:57 2020 +0200

    tests: Fix status-request-revoked after 2020-10-24
    
    included certs expire 2020-10-24 so this test fails after that date.
    
    Fixes #967
    
    This patch was done while working on reproducible builds for openSUSE.
    
    Signed-off-by: Bernhard M. Wiedemann <bwiedemann@suse.de>
--8<---------------cut here---------------end--------------->8---

The question for us becomes how to ensure long-term reproducibility in
the presence of such bugs.

In this case, I think the only solution would be to change the system
clock when one rebuilds GnuTLS (or to use ‘--without-tests=gnutls’, but
you end up with different derivations, which is not necessarily
desirable).

Thoughts?

Ludo’.

bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> The question for us becomes how to ensure long-term reproducibility in
> the presence of such bugs.
>
> In this case, I think the only solution would be to change the system
> clock when one rebuilds GnuTLS (or to use ‘--without-tests=gnutls’, but
> you end up with different derivations, which is not necessarily
> desirable).
>
> Thoughts?

There is a related bug report here:

  https://issues.guix.gnu.org/39310

Perhaps we could make a "--with-system-clock" option for 'guix build'
that instructs the daemon to fake the system time?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked

[Ludovic Courtès]

Hi,

Marius Bakke <marius@gnu.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> The question for us becomes how to ensure long-term reproducibility in
>> the presence of such bugs.
>>
>> In this case, I think the only solution would be to change the system
>> clock when one rebuilds GnuTLS (or to use ‘--without-tests=gnutls’, but
>> you end up with different derivations, which is not necessarily
>> desirable).
>>
>> Thoughts?
>
> There is a related bug report here:
>
>   https://issues.guix.gnu.org/39310
>
> Perhaps we could make a "--with-system-clock" option for 'guix build'
> that instructs the daemon to fake the system time?

How would it fake it though?

There are time_namespaces(7), but it’s only for CLOCK_MONOTONIC and
CLOCK_BOOTTIME.

LD_PRELOAD like ‘datefudge’ does is probably not a viable option.

Ludo’.

bug#44559: Solution

[jeremiah]

I created a procedure to work around the build failure and enable a
successful build:


# when gnutls-3.6.12 build breaks you need to do:
# run the following as root
# turn off networking
ip link set enp0s25 down
# Fixup the time so that the build will succeed
timedatectl set-ntp false
timedatectl set-time '2020-10-01'
# Try to build but it will absolutely fail by lack of source if you
# don't enable networking or because you enable networking.
# But turn on networking when specified below
guix time-machine --commit=94585fffb23079fe71110e2bf99782eb4ccfa12b --
# build --no-grafts --check guile3.0-gnutls
# after it finishes building
# /gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv
# and starts building
# /gnu/store/akm0wl58avib46g3d9razlfzfgfg8m6m-python-3.8.2.drv...
# but before it begins building
# /gnu/store/bja7gqzxr62a0akid0rpzmynzy78nkwg-zstd-1.4.4.tar.gz.drv.
# Fix the time and turn on networking as it has additional things to
# download.
# specifically
# https://github.com/facebook/zstd/releases/download/v1.4.4/zstd-1.4.4.tar.gz
# and substitutes for some reason
# failing to do so will result in you needing to repeat the above steps
# again.
timedatectl set-time '$current_date'
timedatectl set-ntp true
# turn on networking
ip link set enp0s25 up
# it'll fail building because the time is current again
# But that is fine, we now will not need networking for this build cycle
# and thus the altered time will be fine and the build will be
# successful this time
# turn off networking
ip link set enp0s25 down
# Fixup the time so that the build will succeed
timedatectl set-ntp false
timedatectl set-time '2020-10-01'
#guix build gnutls@3.6.12 finally
guix time-machine --commit=94585fffb23079fe71110e2bf99782eb4ccfa12b --
# build --no-grafts --check guile3.0-gnutls
# wait until it completes.
# Then we can put the system back in a working state
timedatectl set-time '$current_date'
timedatectl set-ntp true
# turn on networking
ip link set enp0s25 up

bug#44559:

[Carl Dong]

Hi all,

As bitcoin core begins the planning to officially transition to Guix-based releases, I've had many community members build guix v1.2.0 from source and afterward attempt `--bootstrap --no-substitutes` builds. As you may imagine, they are getting stuck on this gnutls problem and cannot proceed further.

I'm wondering:

1. Is there a workaround that does not involve changing the system time? We have attempted several flags:
	1. --with-graft=gnutls=gnutls@3.6.14
	2. --without-tests=gnutls
	3. --with-input=gnutls=gnutls@3.6.14
	These attempts all failed to work around this bug, and I’m curious as to why that would be. My guess would be that when we do `--bootstrap`, Guix bootstraps itself first without taking into account these flags?

2. Since bootstrappability is one of the core tenets of Guix, might it be appropriate to cut a v1.2.1 release with this problem (and any other potential bootstrap problems) fixed? (Happy to discuss in separate thread if more appropriate)

Cheers,
Carl Dong
contact@carldong.me
"I fight for the users"

bug#44559:

[Leo Famulari]

On Tue, Feb 16, 2021 at 04:00:11PM -0500, Carl Dong wrote:
> Hi all,
> 
> As bitcoin core begins the planning to officially transition to Guix-based releases, I've had many community members build guix v1.2.0 from source and afterward attempt `--bootstrap --no-substitutes` builds. As you may imagine, they are getting stuck on this gnutls problem and cannot proceed further.
> 
> I'm wondering:
> 
> 1. Is there a workaround that does not involve changing the system time? We have attempted several flags:
> 	1. --with-graft=gnutls=gnutls@3.6.14
> 	2. --without-tests=gnutls
> 	3. --with-input=gnutls=gnutls@3.6.14
> 	These attempts all failed to work around this bug, and I’m curious as to why that would be. My guess would be that when we do `--bootstrap`, Guix bootstraps itself first without taking into account these flags?
> 
> 2. Since bootstrappability is one of the core tenets of Guix, might it be appropriate to cut a v1.2.1 release with this problem (and any other potential bootstrap problems) fixed? (Happy to discuss in separate thread if more appropriate)

You should see what the Guix maintainers say about this.

My personal opinion is that you should fork Guix your use case. If you
are building from the bootstrap, there is little added cost to making
minor adjustments like disabling this test. You can periodically re-sync
your fork with GNU Guix as convenient. And it's probably more in tune
with your threat model. [0]

This problem of "expiring software" has occurred several times in Guix's
history and I'm sure it will happen again. In general, users are
expected to use substitutes to work around it. They are no worse off
than with traditional binary distros in that case.

[0] Savannah is great but lacking the resources to devote to security.

bug#44559:

[Ludovic Courtès]

Hi Carl,

Carl Dong <contact@carldong.me> skribis:

> As bitcoin core begins the planning to officially transition to Guix-based releases, I've had many community members build guix v1.2.0 from source and afterward attempt `--bootstrap --no-substitutes` builds. As you may imagine, they are getting stuck on this gnutls problem and cannot proceed further.

Yeah.  :-/

> I'm wondering:
>
> 1. Is there a workaround that does not involve changing the system time? We have attempted several flags:
> 	1. --with-graft=gnutls=gnutls@3.6.14
> 	2. --without-tests=gnutls
> 	3. --with-input=gnutls=gnutls@3.6.14
> 	These attempts all failed to work around this bug, and I’m curious as to why that would be. My guess would be that when we do `--bootstrap`, Guix bootstraps itself first without taking into account these flags?

‘--without-tests’ should work, but you need to pass the right version
number I guess?

> 2. Since bootstrappability is one of the core tenets of Guix, might it be appropriate to cut a v1.2.1 release with this problem (and any other potential bootstrap problems) fixed? (Happy to discuss in separate thread if more appropriate)

I agree it’s a problem, and yes, it would probably be a good idea to
release 1.2.1 with the upgraded GnuTLS we now have in ‘master’.

Longer-term, we need to find a way to address or avoid this issue.  A
brute-force approach would be to have the build machines at ci.guix run
with a clock ten years ahead.  That should generally be fine since the
only place where timestamps matter are unmodified upstream tarballs.  In
all other cases, mtime is set to 1.

Perhaps we could start by testing this hypothesis on a separate build
farm.  Chris, Mathieu, WDYT?

Thanks,
Ludo’.

bug#44559:

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1316 bytes --]

Hi Guix,

On Fri, 2021-02-19 at 16:33 +0100, Ludovic Courtès wrote:
> [...]
> Longer-term, we need to find a way to address or avoid this issue.  A
> brute-force approach would be to have the build machines at ci.guix run
> with a clock ten years ahead.  That should generally be fine since the
> only place where timestamps matter are unmodified upstream tarballs.  In
> all other cases, mtime is set to 1.

Alternatively, could the build container be adjusted to always begin at
1970-01-01, using ‘time namespaces’?

Linux: https://lwn.net/Articles/766089/
Hurd analogue: https://www.gnu.org/software/hurd/gnumach-doc/Host-Interface.html#Host-Interface

(Of course, someone needs to find the time to write the patches first.  Maybe
I'll have a try at it eventually, but probably not anytime soon.)

Also, is there any particular reason to set the clock only ten years ahead,
and not, say, a millenia or two?  Some possible reasons:

* year 2038,2446 problem: the ext2 and ext4 filesystems have a restricted
  date range
* year 2038 problem: https://www.gnu.org/software/hurd/gnumach-doc/Host-Interface.html#Host-Interface

  IMO, the year 2038 problem is a bug and affected packages should simply be fixed.
  But perhaps reality is a little more complicated.

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#44559:

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> On Fri, 2021-02-19 at 16:33 +0100, Ludovic Courtès wrote:
>> [...]
>> Longer-term, we need to find a way to address or avoid this issue.  A
>> brute-force approach would be to have the build machines at ci.guix run
>> with a clock ten years ahead.  That should generally be fine since the
>> only place where timestamps matter are unmodified upstream tarballs.  In
>> all other cases, mtime is set to 1.
>
> Alternatively, could the build container be adjusted to always begin at
> 1970-01-01, using ‘time namespaces’?
>
> Linux: https://lwn.net/Articles/766089/

Unfortunately, time namespaces are just for CLOCK_{MONOTONIC,BOOTTIME},
which I think is of little use here:

  https://issues.guix.gnu.org/44559#3

> Also, is there any particular reason to set the clock only ten years ahead,
> and not, say, a millenia or two?  Some possible reasons:
>
> * year 2038,2446 problem: the ext2 and ext4 filesystems have a restricted
>   date range
> * year 2038 problem: https://www.gnu.org/software/hurd/gnumach-doc/Host-Interface.html#Host-Interface
>
>   IMO, the year 2038 problem is a bug and affected packages should simply be fixed.
>   But perhaps reality is a little more complicated.

Yeah, one problem at a time.  :-)

Setting it 10 years ahead would cache the kind of issue we’re talking
about, while not opening the Y2038 can of worms.  I think we need to try
that out and see how it goes.

Ludo’.

bug#44559: Detecting “expiring” builds

[Ludovic Courtès]

It occurred to me that, just like we have childhurds, we could provide a
service that sets up a sub-Guix System running in a VM with its clock
set years ahead, and you would offload to that.  That’s not perfect, but
it’s a rather easy addition.

Another option would be to have built-in support in the daemon.  If you
turn on some option, it would transparently run derivation builds in
qemu-user (does that support changing the system date?) or similar, but
that’s more work.

Ludo’.

bug#44559:

[Carl Dong]

Hi Guix!

Thanks to all of you for your thoughtful replies!

On Feb 19, 2021, at 10:33 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> I agree it’s a problem, and yes, it would probably be a good idea to
> release 1.2.1 with the upgraded GnuTLS we now have in ‘master’.
I’m very heartened by your affirmation of the project’s support of bootstrappability and building from source. :-)

In addition, I think it would be good to make sure that the package transformation options are powerful enough to allow users to sidestep these problems in their own workflow and decrease the pressure on maintainers.

On Feb 19, 2021, at 10:33 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> ‘--without-tests’ should work, but you need to pass the right version
> number I guess?

Oh! That may be the case. I am using `guix time-machine` however, and that does not yet have the `--without-tests` flag, I have opened bug#46650 so that we can discuss that issue there.

On Feb 19, 2021, at 1:32 PM, Maxime Devos <maximedevos@telenet.be> wrote:
> Alternatively, could the build container be adjusted to always begin at
> 1970-01-01, using ‘time namespaces’?
Unfortunately, as Ludovic mentioned earlier in this thread, time_namespaces(7) is only for CLOCK_MONOTONIC and. CLOCK_BOOTTIME. :-(

Carl Dong
contact@carldong.me
"I fight for the users"

> On Feb 19, 2021, at 10:33 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> 
> Hi Carl,
> 
> Carl Dong <contact@carldong.me> skribis:
> 
>> As bitcoin core begins the planning to officially transition to Guix-based releases, I've had many community members build guix v1.2.0 from source and afterward attempt `--bootstrap --no-substitutes` builds. As you may imagine, they are getting stuck on this gnutls problem and cannot proceed further.
> 
> Yeah.  :-/
> 
>> I'm wondering:
>> 
>> 1. Is there a workaround that does not involve changing the system time? We have attempted several flags:
>> 	1. --with-graft=gnutls=gnutls@3.6.14
>> 	2. --without-tests=gnutls
>> 	3. --with-input=gnutls=gnutls@3.6.14
>> 	These attempts all failed to work around this bug, and I’m curious as to why that would be. My guess would be that when we do `--bootstrap`, Guix bootstraps itself first without taking into account these flags?
> 
> ‘--without-tests’ should work, but you need to pass the right version
> number I guess?
> 
>> 2. Since bootstrappability is one of the core tenets of Guix, might it be appropriate to cut a v1.2.1 release with this problem (and any other potential bootstrap problems) fixed? (Happy to discuss in separate thread if more appropriate)
> 
> I agree it’s a problem, and yes, it would probably be a good idea to
> release 1.2.1 with the upgraded GnuTLS we now have in ‘master’.
> 
> Longer-term, we need to find a way to address or avoid this issue.  A
> brute-force approach would be to have the build machines at ci.guix run
> with a clock ten years ahead.  That should generally be fine since the
> only place where timestamps matter are unmodified upstream tarballs.  In
> all other cases, mtime is set to 1.
> 
> Perhaps we could start by testing this hypothesis on a separate build
> farm.  Chris, Mathieu, WDYT?
> 
> Thanks,
> Ludo’.

bug#44559:

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 830 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Perhaps we could start by testing this hypothesis on a separate build
> farm.  Chris, Mathieu, WDYT?

I'm currently thinking about attempting these kind of things (testing
building derivations under different conditions) through the agent tags
in the Guix Build Coordinator.

I haven't used this functionality yet, but it's mostly implemented. The
idea is that agents have tags, that describe various attributes that are
important (time=normal, time=future, maybe for example), and builds can
also be targeted at specific agents by tagging the builds with those
same tags.

Where I'm going with this is that I'm not sure a separate build farm is
needed, it would be good to just incorperate this in to the build farm
used for testing patches and non-master branches.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#44559:

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Perhaps we could start by testing this hypothesis on a separate build
>> farm.  Chris, Mathieu, WDYT?
>
> I'm currently thinking about attempting these kind of things (testing
> building derivations under different conditions) through the agent tags
> in the Guix Build Coordinator.
>
> I haven't used this functionality yet, but it's mostly implemented. The
> idea is that agents have tags, that describe various attributes that are
> important (time=normal, time=future, maybe for example), and builds can
> also be targeted at specific agents by tagging the builds with those
> same tags.

Sounds nice!  Also varying kernels I guess.

> Where I'm going with this is that I'm not sure a separate build farm is
> needed, it would be good to just incorperate this in to the build farm
> used for testing patches and non-master branches.

Sure.  For the build-in-the-future thing, I think we could just do that
by default; what I meant is that we just need to double-check beforehand
that nothing breaks badly.

Thanks,
Ludo’.

bug#44559:

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1492 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Perhaps we could start by testing this hypothesis on a separate build
>>> farm.  Chris, Mathieu, WDYT?
>>
>> I'm currently thinking about attempting these kind of things (testing
>> building derivations under different conditions) through the agent tags
>> in the Guix Build Coordinator.
>>
>> I haven't used this functionality yet, but it's mostly implemented. The
>> idea is that agents have tags, that describe various attributes that are
>> important (time=normal, time=future, maybe for example), and builds can
>> also be targeted at specific agents by tagging the builds with those
>> same tags.
>
> Sounds nice!  Also varying kernels I guess.

Yeah, there's a whole range of variations it would be nice to
methodically test against (filesystems, Linux versions, system time, one
vs many cores, ...).

>> Where I'm going with this is that I'm not sure a separate build farm is
>> needed, it would be good to just incorperate this in to the build farm
>> used for testing patches and non-master branches.
>
> Sure.  For the build-in-the-future thing, I think we could just do that
> by default; what I meant is that we just need to double-check beforehand
> that nothing breaks badly.

Ah, I guess that might work, if I have some time, I'll have a look in to
making the necessary machine changes.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked

[Maxim Cournoyer]

Hi everyone,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Perhaps we could start by testing this hypothesis on a separate build
>>> farm.  Chris, Mathieu, WDYT?
>>
>> I'm currently thinking about attempting these kind of things (testing
>> building derivations under different conditions) through the agent tags
>> in the Guix Build Coordinator.
>>
>> I haven't used this functionality yet, but it's mostly implemented. The
>> idea is that agents have tags, that describe various attributes that are
>> important (time=normal, time=future, maybe for example), and builds can
>> also be targeted at specific agents by tagging the builds with those
>> same tags.
>
> Sounds nice!  Also varying kernels I guess.
>
>> Where I'm going with this is that I'm not sure a separate build farm is
>> needed, it would be good to just incorperate this in to the build farm
>> used for testing patches and non-master branches.
>
> Sure.  For the build-in-the-future thing, I think we could just do that
> by default; what I meant is that we just need to double-check beforehand
> that nothing breaks badly.

I lost track of the details -- is this problem still likely to bite us
in the future, or was something committed to mitigate against it?

Thanks,

Maxim

bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked

[Ludovic Courtès]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> I lost track of the details -- is this problem still likely to bite us
> in the future, or was something committed to mitigate against it?

“Time bombs” are the worst of non-reproducibility bugs because (1) we
don’t have workarounds other than changing the system time on the
machine that performs the build, and (2) we currently lack a way to
detect them.

In this thread, Chris and I proposed addressing #2 by offering ways to
automatically or manually run builds “in the future”¹.  We could actually
similarly address #1 by running builds “in the past”.

In any case, the first step is probably to offer a VM service for Guix
System, similar to childhurd, with an option to specify the starting
time.

So we’re not done yet, but the discussion has drifted beyond the
original GnuTLS bug report, so perhaps we can close it and take it
elsewhere?

Thanks,
Ludo’.

¹ QEMU can be passed an option like “-rtc base=2020-01-01”.

bug#44559: gnutls 3.6.12 fails to build: FAIL: status-request-revoked

[Maxim Cournoyer]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> I lost track of the details -- is this problem still likely to bite us
>> in the future, or was something committed to mitigate against it?
>
> “Time bombs” are the worst of non-reproducibility bugs because (1) we
> don’t have workarounds other than changing the system time on the
> machine that performs the build, and (2) we currently lack a way to
> detect them.

I see.  Thanks for explaining!

> In this thread, Chris and I proposed addressing #2 by offering ways to
> automatically or manually run builds “in the future”¹.  We could actually
> similarly address #1 by running builds “in the past”.
>
> In any case, the first step is probably to offer a VM service for Guix
> System, similar to childhurd, with an option to specify the starting
> time.
>
> So we’re not done yet, but the discussion has drifted beyond the
> original GnuTLS bug report, so perhaps we can close it and take it
> elsewhere?

I think so!  Few seems to visit these years old reports.  If there's
something to brainstorm about, I'd rather see it happen in a new thread
on guix-devel.

Thanks for the reply!

Closing.

Maxim