Time travel accident

Time travel accident

[Konrad Hinsen]

Hi everyone,

For doing some experiments with Python 3.8.2, I tried to use the commit
that introduced that version into Guix;

  guix time-machine --commit=ce35dc84a10b05dc891bfae03f613b907337945e \
   -- shell --pure python \
   -- python3

That fails, due to a build failure in OpenSSL:

   building /gnu/store/nh63q12x95irxyqzls0sfalf8ih5qass-openssl-1.1.1d.drv...
   | 'check' phasebuilder for `/gnu/store/nh63q12x95irxyqzls0sfalf8ih5qass-openssl-1.1.1d.drv' failed with exit code 1
   build of /gnu/store/nh63q12x95irxyqzls0sfalf8ih5qass-openssl-1.1.1d.drv failed
   View build log at '/var/log/guix/drvs/nh/63q12x95irxyqzls0sfalf8ih5qass-openssl-1.1.1d.drv.bz2'.

Inspecting the build log shows that the cause is a failing test:

   Test Summary Report
   -------------------
   ../test/recipes/80-test_ssl_new.t                (Wstat: 256 Tests: 29 Failed: 1)

That problem has been reported elsewhere and identified as caused by an
expired certificate:

   https://github.com/openssl/openssl/issues/18441


I guess there is nothing we can do retroactively to fix this, but can we
do something to prevent such issues in the future?

One idea is to allow disabling tests at the command line. I'd then run
"guix build" for that specific package with tests disabled, and
continue. That should be doable with a suitable package transformation.

Cheers,
  Konrad

Re: Time travel accident

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> I guess there is nothing we can do retroactively to fix this, but can we
> do something to prevent such issues in the future?
>
> One idea is to allow disabling tests at the command line. I'd then run
> "guix build" for that specific package with tests disabled, and
> continue. That should be doable with a suitable package transformation.

We have --without-tests=package already, see --help-transform for all
available package transformations.  The one annoying thing is that
disabling tests will change the derivation and you thus will not recover
the same store item (it might be bit-for-bit equivalent, but its path
will not be the same), preventing you from using substitutes either.

Though, I'm not sure it will help you here because openssl is built as
part of the `guix time-machine`'s build process, which afaik cannot be
transformed.

By the way, we can also "fix the past" by using guix/quirks.scm.  Since
that version of openssl doesn't build anymore, I wonder if we could just
change its derivation retroactively to at least make it build.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Time travel accident

[Julien Lepiller]

Changing your system date should let it build.

Le 7 avril 2023 10:30:11 GMT+02:00, Josselin Poiret <dev@jpoiret.xyz> a écrit :
>Hi Konrad,
>
>Konrad Hinsen <konrad.hinsen@fastmail.net> writes:
>
>> I guess there is nothing we can do retroactively to fix this, but can we
>> do something to prevent such issues in the future?
>>
>> One idea is to allow disabling tests at the command line. I'd then run
>> "guix build" for that specific package with tests disabled, and
>> continue. That should be doable with a suitable package transformation.
>
>We have --without-tests=package already, see --help-transform for all
>available package transformations.  The one annoying thing is that
>disabling tests will change the derivation and you thus will not recover
>the same store item (it might be bit-for-bit equivalent, but its path
>will not be the same), preventing you from using substitutes either.
>
>Though, I'm not sure it will help you here because openssl is built as
>part of the `guix time-machine`'s build process, which afaik cannot be
>transformed.
>
>By the way, we can also "fix the past" by using guix/quirks.scm.  Since
>that version of openssl doesn't build anymore, I wonder if we could just
>change its derivation retroactively to at least make it build.
>
>Best,

Re: Time travel accident

[Konrad Hinsen]

Hi Josselin and Julien,

Thanks to both of you for your suggestions!

Josselin Poiret <dev@jpoiret.xyz> writes:

> We have --without-tests=package already, see --help-transform for all
> available package transformations.  The one annoying thing is that
> disabling tests will change the derivation and you thus will not recover
> the same store item (it might be bit-for-bit equivalent, but its path
> will not be the same), preventing you from using substitutes either.

For my case, that sounds OK. There are no substitutes for that
three-year old commit any more, so I am building everything. And I don't
care about bit-for-bit equivalence, I just want to run Python 3.8.2.

> Though, I'm not sure it will help you here because openssl is built as
> part of the `guix time-machine`'s build process, which afaik cannot be
> transformed.

Ahhh... there's the bad news.

> By the way, we can also "fix the past" by using guix/quirks.scm.  Since

Oohhh... There's always one more surprise in Guix!


Julien Lepiller <julien@lepiller.eu> writes:

> Changing your system date should let it build.

Interesting idea! I tried, but it doesn't work: Guix itself complains
about a certificate failure if I set the clock three years back.

Maybe "guix time-machine" should have an option for setting the clock to
the commit timestamp just for the build process...

Cheers,
  Konrad.

Re: Time travel accident

[Julien Lepiller]

If you're able to find the derivation, you could set your system time and guix build /gnu/store/that.drv. this should not require network at all, so guix shouldn't complain.

Le 7 avril 2023 18:10:13 GMT+02:00, Konrad Hinsen <konrad.hinsen@fastmail.net> a écrit :
>Hi Josselin and Julien,
>
>Thanks to both of you for your suggestions!
>
>Josselin Poiret <dev@jpoiret.xyz> writes:
>
>> We have --without-tests=package already, see --help-transform for all
>> available package transformations.  The one annoying thing is that
>> disabling tests will change the derivation and you thus will not recover
>> the same store item (it might be bit-for-bit equivalent, but its path
>> will not be the same), preventing you from using substitutes either.
>
>For my case, that sounds OK. There are no substitutes for that
>three-year old commit any more, so I am building everything. And I don't
>care about bit-for-bit equivalence, I just want to run Python 3.8.2.
>
>> Though, I'm not sure it will help you here because openssl is built as
>> part of the `guix time-machine`'s build process, which afaik cannot be
>> transformed.
>
>Ahhh... there's the bad news.
>
>> By the way, we can also "fix the past" by using guix/quirks.scm.  Since
>
>Oohhh... There's always one more surprise in Guix!
>
>
>Julien Lepiller <julien@lepiller.eu> writes:
>
>> Changing your system date should let it build.
>
>Interesting idea! I tried, but it doesn't work: Guix itself complains
>about a certificate failure if I set the clock three years back.
>
>Maybe "guix time-machine" should have an option for setting the clock to
>the commit timestamp just for the build process...
>
>Cheers,
>  Konrad.
>

Re: Time travel accident

[Konrad Hinsen]

Julien Lepiller <julien@lepiller.eu> writes:

> If you're able to find the derivation, you could set your system time
> and guix build /gnu/store/that.drv. this should not require network at
> all, so guix shouldn't complain.

Sounds good. The error message contains the path to the derivation,
so... let's just do it!

A few minutes later: I can't do anything at all with time-machine on
that commit any more. Maybe my attempt to build with a modified system
time broke something in the store?

   $ guix time-machine --commit=ce35dc84a10b05dc891bfae03f613b907337945e --
shell --pure python  -- python3
...
   building /gnu/store/6126jqfx0rk9cccm518f53li2nj1q1h0-inferior-script.scm.drv...
   building package cache...
   building profile with 1 package...
   ;;; WARNING: loading compiled file /gnu/store/41zsnwsk02549kqb5njd3fadgnmkzww8-guix-module-union/lib/guile/3.0/site-ccache/guix/ui.go failed:
   ;;; In procedure load-thunk-from-memory: incompatible bytecode kind
...
   ;;; WARNING: loading compiled file /gnu/store/41zsnwsk02549kqb5njd3fadgnmkzww8-guix-module-union/lib/guile/3.0/site-ccache/guix/licenses.go failed:
   ;;; In procedure load-thunk-from-memory: incompatible bytecode kind
   WARNING: Use of `load' in declarative module (guix ui).  Add #:declarative? #f to your define-module invocation.
   Backtrace:
              5 (primitive-load "/home/hinsen/.cache/guix/inferiors/wisrwo5p2aq7o6r…")
   In ice-9/eval.scm:
       619:8  4 (_ #(#(#<directory (guix ui) 7f7fc35115a0>) "/home/hinsen/.cache…" …))
       619:8  3 (_ #(#(#<directory (guix ui) 7f7fc35115a0>)))
       159:9  2 (_ #(#(#<directory (guix ui) 7f7fc35115a0>)))
      223:20  1 (proc #(#(#<directory (guix ui) 7f7fc35115a0>)))
   In unknown file:
              0 (%resolve-variable (7 . _IOLBF) #<directory (guix ui) 7f7fc35115a0>)

   ERROR: In procedure %resolve-variable:
   _IOLBF: unbound variable

In an attempt to clean up, I deleted ~/.cache/guix/inferiors, and then
did a "guix gc". But that makes no difference.

Time travelers live dangerously!

Cheers,
  Konrad.

Re: Time travel accident

[Konrad Hinsen]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

>    building profile with 1 package...
>    ;;; WARNING: loading compiled file /gnu/store/41zsnwsk02549kqb5njd3fadgnmkzww8-guix-module-union/lib/guile/3.0/site-ccache/guix/ui.go failed:
>    ;;; In procedure load-thunk-from-memory: incompatible bytecode kind

I tried to track down why this file remains in the store even after
clearing the "inferiors" cache and doing a gc. Tracking the chain of
referrers, I find:

  /gnu/store/41zsnwsk02549kqb5njd3fadgnmkzww8-guix-module-union
  –> /gnu/store/7r72vknkpnpgp143ckh5kfbg5zan3xsp-guix-command
  –> /gnu/store/cwqaas3mwlx2rhc0ckzmqvygmy7n2s7k-guix-ce35dc8
  –> /gnu/store/2h03nwj2r2jywjk5sk7sxn2hgm979v2m-profile

This last item has two referrers:

 1. /gnu/store/9gfmn1yra7rzavxb9wppqi4lpdvqid8c-inferior-script.scm
 2. itself!

The only items I have encountered so far that have themselves as
referrers are gc roots. But this profile item is not in the list of gc
roots.

The inferior-script item has no referrers at all. It's not a gc root
either. Nevertheless, it is on the output of "guix gc --list-live".
According to my understanding of the store, that shouldn't be possible!

Cheers,
  Konrad.

Re: Time travel accident

[Simon Tournier]

Hi,

On mer., 12 avril 2023 at 12:17, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

>   /gnu/store/41zsnwsk02549kqb5njd3fadgnmkzww8-guix-module-union
>   –> /gnu/store/7r72vknkpnpgp143ckh5kfbg5zan3xsp-guix-command
>   –> /gnu/store/cwqaas3mwlx2rhc0ckzmqvygmy7n2s7k-guix-ce35dc8
>   –> /gnu/store/2h03nwj2r2jywjk5sk7sxn2hgm979v2m-profile
>
> This last item has two referrers:
>
>  1. /gnu/store/9gfmn1yra7rzavxb9wppqi4lpdvqid8c-inferior-script.scm
>  2. itself!

I have,

--8<---------------cut here---------------start------------->8---
$ guix gc --list-dead | grep 9gfmn1yra7rzavxb9wppqi4lpdvqid8c
finding garbage collector roots...
determining live/dead paths...
/gnu/store/9gfmn1yra7rzavxb9wppqi4lpdvqid8c-inferior-script.scm

$ guix gc --references /gnu/store/9gfmn1yra7rzavxb9wppqi4lpdvqid8c-inferior-script.scm
/gnu/store/2h03nwj2r2jywjk5sk7sxn2hgm979v2m-profile

$ guix gc --referrers /gnu/store/2h03nwj2r2jywjk5sk7sxn2hgm979v2m-profile
/gnu/store/2h03nwj2r2jywjk5sk7sxn2hgm979v2m-profile
/gnu/store/9gfmn1yra7rzavxb9wppqi4lpdvqid8c-inferior-script.scm

$ guix gc --list-dead | grep 2h03nwj2r2jywjk5sk7sxn2hgm979v2m
finding garbage collector roots...
determining live/dead paths...
/gnu/store/2h03nwj2r2jywjk5sk7sxn2hgm979v2m-profile

$ guix gc -D /gnu/store/9gfmn1yra7rzavxb9wppqi4lpdvqid8c-inferior-script.scm
finding garbage collector roots...
[0 MiB] deleting '/gnu/store/9gfmn1yra7rzavxb9wppqi4lpdvqid8c-inferior-script.scm'
deleting `/gnu/store/trash'
deleting unused links...
note: currently hard linking saves 27283.40 MiB

$ guix gc --list-dead | grep 9gfmn1yra7rzavxb9wppqi4lpdvqid8c
finding garbage collector roots...
determining live/dead paths...

$ guix gc --referrers /gnu/store/2h03nwj2r2jywjk5sk7sxn2hgm979v2m-profile
/gnu/store/2h03nwj2r2jywjk5sk7sxn2hgm979v2m-profile
--8<---------------cut here---------------end--------------->8---

Do you have special options for guix-daemon or the default ones?

Well, can you run,

    guix gc -D /gnu/store/2h03nwj2r2jywjk5sk7sxn2hgm979v2m-profile

?

Cheers,
simon

Re: Time travel accident

[Konrad Hinsen]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> I have,
>
> --8<---------------cut here---------------start------------->8---
> $ guix gc --list-dead | grep 9gfmn1yra7rzavxb9wppqi4lpdvqid8c
> finding garbage collector roots...
> determining live/dead paths...
> /gnu/store/9gfmn1yra7rzavxb9wppqi4lpdvqid8c-inferior-script.scm

For me it is (well, was, see below) under "–list-live". But it has no
referrers. And...

> $ guix gc -D /gnu/store/9gfmn1yra7rzavxb9wppqi4lpdvqid8c-inferior-script.scm

fails for me (as it should for a live item).

Yesterday, I did "rm -rf ~/.cache/guix/inferiors" followed by "guix gc".
Right now, after the above tests, I did another "guix gc". And now...
the two items we are testing here are "dead" and I could remove them.

So it seems that two garbage collections are better than one, even
though nothing much happened in between (just a few "guix shell"
executions that added a few items to the store).

Cheers,
  Konrad

Re: Time travel accident

[Simon Tournier]

Hi Konrad,

On mar., 11 avril 2023 at 17:20, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

>    Backtrace:
>               5 (primitive-load "/home/hinsen/.cache/guix/inferiors/wisrwo5p2aq7o6r…")
>    In ice-9/eval.scm:
>        619:8  4 (_ #(#(#<directory (guix ui) 7f7fc35115a0>) "/home/hinsen/.cache…" …))
>        619:8  3 (_ #(#(#<directory (guix ui) 7f7fc35115a0>)))
>        159:9  2 (_ #(#(#<directory (guix ui) 7f7fc35115a0>)))
>       223:20  1 (proc #(#(#<directory (guix ui) 7f7fc35115a0>)))
>    In unknown file:
>               0 (%resolve-variable (7 . _IOLBF) #<directory (guix ui) 7f7fc35115a0>)
>
>    ERROR: In procedure %resolve-variable:
>    _IOLBF: unbound variable

I get the same thing without doing anything special.

--8<---------------cut here---------------start------------->8---
$ date
mer. 12 avril 2023 12:09:47 CEST

$ guix time-machine --commit=ce35dc84a10b05dc891bfae03f613b907337945e -- help
;;; WARNING: loading compiled file /gnu/store/41zsnwsk02549kqb5njd3fadgnmkzww8-guix-module-union/lib/guile/3.0/site-ccache/guix/ui.go failed:
;;; In procedure load-thunk-from-memory: incompatible bytecode kind

[...]

;;; In procedure load-thunk-from-memory: incompatible bytecode version
WARNING: Use of `load' in declarative module (guix ui).  Add #:declarative? #f to your define-module invocation.
Backtrace:
           5 (primitive-load "/home/simon/.cache/guix/inferiors/wisrwo5p2aq7…")
In ice-9/eval.scm:
    619:8  4 (_ #(#(#<directory (guix ui) 7fe0f84a85a0>) "/home/simon/.ca…" …))
    619:8  3 (_ #(#(#<directory (guix ui) 7fe0f84a85a0>)))
    159:9  2 (_ #(#(#<directory (guix ui) 7fe0f84a85a0>)))
   223:20  1 (proc #(#(#<directory (guix ui) 7fe0f84a85a0>)))
In unknown file:
           0 (%resolve-variable (7 . _IOLBF) #<directory (guix ui) 7fe0f84a8…>)

ERROR: In procedure %resolve-variable:
_IOLBF: unbound variable
--8<---------------cut here---------------end--------------->8---

Well, it seems related to the Guile version and not about the host
date.  Hum, wait, is it possible to time-travel to:

        ce35dc84a10b05dc891bfae03f613b907337945e
        Author:     Marius Bakke <marius@gnu.org>
        AuthorDate: Thu Mar 29 02:56:00 2018 +0200
        Commit:     Marius Bakke <marius@gnu.org>
        CommitDate: Thu Mar 29 02:56:00 2018 +0200

?  It pre-dates v1.0 from May 2019.  And inferiors had been introduced
around v0.15 from July 2018 [1,2,3].

Well, v1.0 appears to me as the zero for time-travel – soft limit.  And
the introduction of inferiors is the hard limit, so I guess you are
hitting that limit, no?


Cheers,
simon

1: https://guix.gnu.org/blog/2018/gnu-guix-and-guixsd-0.15.0-released/
2: https://guix.gnu.org/en/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/
3: https://issues.guix.gnu.org/32115#8

Re: Time travel accident

[Konrad Hinsen]

Hi Simon,

> I get the same thing without doing anything special.

Interesting. Before my clock-changing experiment, the same command line
got me much further: the old Guix started to work and only failed when
building OpenSSL.

> Well, v1.0 appears to me as the zero for time-travel – soft limit.  And
> the introduction of inferiors is the hard limit, so I guess you are
> hitting that limit, no?

Another good question. I have hit it often enough in the past, but the
error message I got looked different. But then, I guess the error
message for illicit time travel is not part of the documented behavior
of Guix.

Cheers,
  Konrad.

Re: Time travel accident

[Simon Tournier]

Hi Konrad,

On Fri, 07 Apr 2023 at 09:47, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> That fails, due to a build failure in OpenSSL:

Yeah, time bomb!

Somehow, it is a know issue: https://issues.guix.gnu.org/58650

As Julien pointed, replacing the date of the host system can fix such
issue.  Somehow, one idea is to use ‘datefudge’ to detect such time
bomb and/or maybe fake an older time to pass some tests… but I am not
aware of any attempt to tackle these time bomb.

Aside, from my point of view, the 4 assumptions for time traveling with
Guix are:

 1. source code availability
 2. Linux kernel compatibility
 3. hardware compatibility
 4. no time bomb

and a lot of work remains for insuring a kind of validity of these.

Cheers,
simon

Re: Time travel accident

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> That problem has been reported elsewhere and identified as caused by an
> expired certificate:
>
>    https://github.com/openssl/openssl/issues/18441
>
>
> I guess there is nothing we can do retroactively to fix this, but can we
> do something to prevent such issues in the future?

This has been discussed on several occasions, such as:

  https://issues.guix.gnu.org/58650

Unfortunately there’s no bullet-proof way to avoid these time bombs.

My plan is to write a service that makes it easy to offload a build to a
VM that runs with a different time (“in the past”) or something along
these lines to mitigate the problem.

Ludo’.

Re: Time travel accident

[Simon Tournier]

Hi,

On lun., 17 avril 2023 at 15:32, Ludovic Courtès <ludo@gnu.org> wrote:

> My plan is to write a service that makes it easy to offload a build to a
> VM that runs with a different time (“in the past”) or something along
> these lines to mitigate the problem.

Do you mean “in the future” instead?  We need to know if the package
will still build with a different time from the future, no?

Wording aside :-) What do Reproducible Builds for that?  Because
time-bomb seems similar as timestamp…


Cheers,
simon

Re: Time travel accident

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1718 bytes --]

On 2023-04-17, Simon Tournier wrote:
> On lun., 17 avril 2023 at 15:32, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> My plan is to write a service that makes it easy to offload a build to a
>> VM that runs with a different time (“in the past”) or something along
>> these lines to mitigate the problem.
>
> Do you mean “in the future” instead?  We need to know if the package
> will still build with a different time from the future, no?

I can see "in the past" being useful to handle builds with time-bombs
that already slipped through the cracks, and "in the future" to detect
these time-bombs while there is still time to fix them...


> Wording aside :-) What do Reproducible Builds for that?  Because
> time-bomb seems similar as timestamp…

At least for https://tests.reproducible-builds.org/debian on
amd64/x86_64, I think one of the builds runs approximately 1 year, 1
month and 1 day in the future (+397 days?), which pretty much maximizes
the chance of a difference in year, month or day, while sommewhat
minimizing detection of time bombs...

For detecting time-bombs, I would guess you want to test even further
into the future, maybe 5-10 years or so. Much longer, and you are
getting pretty close to 2038, which is an extra-special set of timebombs
that will need to be addressed at some point!

If guix someday has a long-term support release model, catching these
sorts of time-bombs as early as possible becomes even more important,
though with the current release model of once or twice a year or so, it
is a bit less problematic... although if it happens in something low on
the dependency graph, it of course gets more urgent!


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Time travel accident

[Simon Tournier]

Hi,

On Mon, 17 Apr 2023 at 15:16, Vagrant Cascadian <vagrant@reproducible-builds.org> wrote:
> On 2023-04-17, Simon Tournier wrote:

>>> My plan is to write a service that makes it easy to offload a build to a
>>> VM that runs with a different time (“in the past”) or something along
>>> these lines to mitigate the problem.

[...]

> I can see "in the past" being useful to handle builds with time-bombs
> that already slipped through the cracks

My use-case is a researcher trying to redo in the future of 5 years some
computations from a published (3 years ago) article providing some
channels.scm and manifest.scm files.

In this use-case, this researcher often runs Guix on the top of some
GNU/Linux distro and probably on some shared machine in some University.

Last, I envision for my use-case that the Guix infrastructure ecosystem
would not be there to run this “offloading service”.  Otherwise, it
appears to me far easier to just store the current substitutes – or say
part of the current substitutes.


>> Wording aside :-) What do Reproducible Builds for that?  Because
>> time-bomb seems similar as timestamp…
>
> At least for https://tests.reproducible-builds.org/debian on
> amd64/x86_64, I think one of the builds runs approximately 1 year, 1
> month and 1 day in the future (+397 days?), which pretty much maximizes
> the chance of a difference in year, month or day, while sommewhat
> minimizing detection of time bombs...

Since we are somehow building (at least) twice (Bordeaux and Berlin),
maybe we could exploit this fact and so build “in the future“.

I mean, similarly as we are doing world-rebuild with core-updates
cycles, we could do a feature branch with a different time (”in the
future“) more or less around the release.  Somehow, similarly as we are
tagging some packages with ‘non reproducible’, we could tag the ones
with ’time bombs’.

> For detecting time-bombs, I would guess you want to test even further
> into the future, maybe 5-10 years or so. Much longer, and you are
> getting pretty close to 2038, which is an extra-special set of timebombs
> that will need to be addressed at some point!

Yeah, 2038 is something [1]…

1: <https://en.wikipedia.org/wiki/Year_2038_problem>


Cheers,
simon