A real-life test of long-term reproducibility

A real-life test of long-term reproducibility

[Konrad Hinsen]

Hi everyone,

One of our claims is that Guix can rebuild code identically as long as
we have a machine with a Linux kernel and a POSIX filesystem. This week
I had an occasion to put this to a real-life test. So far it's a
failure. I can guess reasons for my failed attempts, but I don't think
they were unreasonable to try. So I'd like to document something that
works, to avoid others falling into the same traps. I just don't know
yet what the Right Way To Do It is!

The package I want to rebuild and use is "nmoldyn" from Guix commit
f250a868d8c687df08559682fa68fb4ea2a1ea69. That's the commit referenced
in my notes, obtained via "guix describe" in early 2018. I am pretty
sure it worked fine back then.

First attempt:

   $ guix time-machine --commit=f250a868d8c687df08559682fa68fb4ea2a1ea69 -- build nmoldyn
   Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
   Backtrace:
   In guix/store.scm:
       672:3 19 (_)
   In ice-9/boot-9.scm:
     1752:10 18 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
   In guix/store.scm:
      659:37 17 (thunk)
   In guix/status.scm:
       815:4 16 (call-with-status-report _ _)
   In guix/store.scm:
      1298:8 15 (call-with-build-handler #<procedure 7ff03a85b960 at guix/ui.scm:1…> …)
     2168:25 14 (run-with-store #<store-connection 256.99 7ff03b715c30> _ # _ # _ # _)
   In guix/inferior.scm:
       903:8 13 (_ _)
   In guix/channels.scm:
       944:2 12 (_ _)
       891:2 11 (_ _)
   In ./guix/monads.scm:
       487:9 10 (_ _)
   In guix/store.scm:
      1996:8  9 (_ _)
   In guix/channels.scm:
      642:36  8 (_ #<store-connection 256.99 7ff03b715c30>)
      703:11  7 (_)
   In ice-9/eval.scm:
       619:8  6 (_ #(#(#(#<directory (build-self) 7ff03a8678c0>) "/gnu/store/…" …) …))
      626:19  5 (_ #(#(#(#<directory (build-self) 7ff03a8678c0>) "/gnu/store/…" …) …))
       155:9  4 (_ #(#(#(#<directory (build-self) 7ff03a8678c0>) "/gnu/store/…" …) …))
      223:20  3 (proc #(#(#(#<directory (build-self) 7ff03a8678c0>) "/gnu/sto…" …) …))
   In unknown file:
              2 (%resolve-variable (7 . %guix-register-program) #<directory (build-…>)
   In ice-9/boot-9.scm:
     1685:16  1 (raise-exception _ #:continuable? _)
     1685:16  0 (raise-exception _ #:continuable? _)

   ice-9/boot-9.scm:1685:16: In procedure raise-exception:
   error: %guix-register-program: unbound variable

I don't understand what is going wrong here, but it may be related to
the fact that the commit I am trying to go back to is older than "guix
time-machine". If that's the explanation, it would help if Guix showed
some clear error message instead of crashing.

Next I tried checking out the source code for commit
f250a868d8c687df08559682fa68fb4ea2a1ea69, and building it from
source. This is a bit tricky because 2018 Guix cannot be built in
today's Guix build environment. For example, today we have Guile 3, but
back then we had Guile 2.2. So I need to do "guix environment guix" in
an older Guix, before the Guile 3 transition, but later than the
introduction of time-machine. I picked one somewhat at random:

   $ guix time-machine --commit=e2293cbbe0cd20ddeb932e6f5616565ab468c087
   -- environment –pure guix

Then I did "bootstrap", "configure –localstatedir=/var", "make
check". The latter shows 15 failures, some of which look important:

   FAIL: tests/builders.scm
   FAIL: tests/derivations.scm
   FAIL: tests/packages.scm
   FAIL: tests/guix-environment.sh
   FAIL: tests/guix-daemon.sh

And indeed I cannot build much with my compiled guix:

   $ ./pre-inst-env guix build nmoldyn

hangs after a while, running a binary called "test-lock" for hours.

Given the time lapse, I suppose there have been incompatible changes in
the build daemon, making the old Guix incompatible with the rather
recent build daemon running on my machine. But is there a way around
this, other than installing an old Guix in a fully isolated VM?

And if installing the old Guix in a VM is the only solution, what would
be the best way to do that? I can't think of much else than starting
from another distribution (e.g. Debian) and following the installation
instructions. That's already a lot of work, but it's made worse by the
installation instructions being hidden inside the manual of that old
commit, which I cannot easily consult.

I'd be grateful for any suggestions!

Cheers,
  Konrad

Re: A real-life test of long-term reproducibility

[blake]

Hi Konrad, 

Just one quick comment:

August 4, 2022 8:43 AM, "Konrad Hinsen" <konrad.hinsen@fastmail.net> wrote:

> Hi everyone,
> 
> One of our claims is that Guix can rebuild code identically as long as
> we have a machine with a Linux kernel and a POSIX filesystem.

This actually isn't the claim. Reproducibility is only guaranteed on
Guix systems. It's important that you reproduce the entire system, which
means you will have needed to have saved the commit of your version of
the Guix package manager in order to return to that system.

See the section of the Guix manual "Replicating Guix" for more info.
https://guix.gnu.org/en/manual/devel/en/html_node/Replicating-Guix.html

Happy hacking!

> This week
> I had an occasion to put this to a real-life test. So far it's a
> failure. I can guess reasons for my failed attempts, but I don't think
> they were unreasonable to try. So I'd like to document something that
> works, to avoid others falling into the same traps. I just don't know
> yet what the Right Way To Do It is!
> 
> The package I want to rebuild and use is "nmoldyn" from Guix commit
> f250a868d8c687df08559682fa68fb4ea2a1ea69. That's the commit referenced
> in my notes, obtained via "guix describe" in early 2018. I am pretty
> sure it worked fine back then.
> 
> First attempt:
> 
> $ guix time-machine --commit=f250a868d8c687df08559682fa68fb4ea2a1ea69 -- build nmoldyn
> Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
> Backtrace:
> In guix/store.scm:
> 672:3 19 (_)
> In ice-9/boot-9.scm:
> 1752:10 18 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
> In guix/store.scm:
> 659:37 17 (thunk)
> In guix/status.scm:
> 815:4 16 (call-with-status-report _ _)
> In guix/store.scm:
> 1298:8 15 (call-with-build-handler #<procedure 7ff03a85b960 at guix/ui.scm:1…> …)
> 2168:25 14 (run-with-store #<store-connection 256.99 7ff03b715c30> _ # _ # _ # _)
> In guix/inferior.scm:
> 903:8 13 (_ _)
> In guix/channels.scm:
> 944:2 12 (_ _)
> 891:2 11 (_ _)
> In ./guix/monads.scm:
> 487:9 10 (_ _)
> In guix/store.scm:
> 1996:8 9 (_ _)
> In guix/channels.scm:
> 642:36 8 (_ #<store-connection 256.99 7ff03b715c30>)
> 703:11 7 (_)
> In ice-9/eval.scm:
> 619:8 6 (_ #(#(#(#<directory (build-self) 7ff03a8678c0>) "/gnu/store/…" …) …))
> 626:19 5 (_ #(#(#(#<directory (build-self) 7ff03a8678c0>) "/gnu/store/…" …) …))
> 155:9 4 (_ #(#(#(#<directory (build-self) 7ff03a8678c0>) "/gnu/store/…" …) …))
> 223:20 3 (proc #(#(#(#<directory (build-self) 7ff03a8678c0>) "/gnu/sto…" …) …))
> In unknown file:
> 2 (%resolve-variable (7 . %guix-register-program) #<directory (build-…>)
> In ice-9/boot-9.scm:
> 1685:16 1 (raise-exception _ #:continuable? _)
> 1685:16 0 (raise-exception _ #:continuable? _)
> 
> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> error: %guix-register-program: unbound variable
> 
> I don't understand what is going wrong here, but it may be related to
> the fact that the commit I am trying to go back to is older than "guix
> time-machine". If that's the explanation, it would help if Guix showed
> some clear error message instead of crashing.
> 
> Next I tried checking out the source code for commit
> f250a868d8c687df08559682fa68fb4ea2a1ea69, and building it from
> source. This is a bit tricky because 2018 Guix cannot be built in
> today's Guix build environment. For example, today we have Guile 3, but
> back then we had Guile 2.2. So I need to do "guix environment guix" in
> an older Guix, before the Guile 3 transition, but later than the
> introduction of time-machine. I picked one somewhat at random:
> 
> $ guix time-machine --commit=e2293cbbe0cd20ddeb932e6f5616565ab468c087
> -- environment –pure guix
> 
> Then I did "bootstrap", "configure –localstatedir=/var", "make
> check". The latter shows 15 failures, some of which look important:
> 
> FAIL: tests/builders.scm
> FAIL: tests/derivations.scm
> FAIL: tests/packages.scm
> FAIL: tests/guix-environment.sh
> FAIL: tests/guix-daemon.sh
> 
> And indeed I cannot build much with my compiled guix:
> 
> $ ./pre-inst-env guix build nmoldyn
> 
> hangs after a while, running a binary called "test-lock" for hours.
> 
> Given the time lapse, I suppose there have been incompatible changes in
> the build daemon, making the old Guix incompatible with the rather
> recent build daemon running on my machine. But is there a way around
> this, other than installing an old Guix in a fully isolated VM?
> 
> And if installing the old Guix in a VM is the only solution, what would
> be the best way to do that? I can't think of much else than starting
> from another distribution (e.g. Debian) and following the installation
> instructions. That's already a lot of work, but it's made worse by the
> installation instructions being hidden inside the manual of that old
> commit, which I cannot easily consult.
> 
> I'd be grateful for any suggestions!
> 
> Cheers,
> Konrad

Re: A real-life test of long-term reproducibility

[zimoun]

Hi Blake,

I am late to the party. :-)

On jeu., 04 août 2022 at 15:35, blake@reproduciblemedia.com wrote:
> August 4, 2022 8:43 AM, "Konrad Hinsen" <konrad.hinsen@fastmail.net> wrote:

>> One of our claims is that Guix can rebuild code identically as long as
>> we have a machine with a Linux kernel and a POSIX filesystem.

This claim is correct, AFAIU.


> This actually isn't the claim. Reproducibility is only guaranteed on
> Guix systems. It's important that you reproduce the entire system, which
> means you will have needed to have saved the commit of your version of
> the Guix package manager in order to return to that system.

It is possible to rebuild identically on any foreign GNU/Linux distro
running the Guix package manager.  The assumptions, between the 2 points
in time, are:

 1. compatibility of the Linux kernel
 2. availability of all the source code
 3. compatibility of the hardware

AFAIK, #1 and #3 are satisfied.  About #2, it depends and many corner
cases are around.

Running Guix System would allow to easily satisfy #1, but, AFAIK, no one
reported an incompatibility of the Linux kernel defeating “guix
time-machine -- build”, and thus, it appears to me still hypothetical
(although possible on the paper) that, in this case, Guix System would
still allow the travel back in time.


> See the section of the Guix manual "Replicating Guix" for more info.
> https://guix.gnu.org/en/manual/devel/en/html_node/Replicating-Guix.html

Guix System is not mentioned, IIRC. :-)


Cheers,
simon

Re: A real-life test of long-term reproducibility

[Ludovic Courtès]

Hi Konrad!

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> The package I want to rebuild and use is "nmoldyn" from Guix commit
> f250a868d8c687df08559682fa68fb4ea2a1ea69. That's the commit referenced
> in my notes, obtained via "guix describe" in early 2018. I am pretty
> sure it worked fine back then.

That’s a commit from January 2018, which is a few months before the
release of 0.15.0, the first release with proper ‘guix pull’ support:

  https://guix.gnu.org/en/blog/2018/gnu-guix-and-guixsd-0.15.0-released/

In other words: warranty void.  :-)

We can go back to 1.0.0, and presumably to 0.15.0, but anything older
than this is unknown territory.

> First attempt:
>
>    $ guix time-machine --commit=f250a868d8c687df08559682fa68fb4ea2a1ea69 -- build nmoldyn
>    Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...

[...]

>    ice-9/boot-9.scm:1685:16: In procedure raise-exception:
>    error: %guix-register-program: unbound variable

It’s a variable that used to exist in (guix config) but was removed long
ago (in ea0a06cee2ba05451f94714a4f913db02efbe92c, shortly before
0.15.0.)

> I don't understand what is going wrong here, but it may be related to
> the fact that the commit I am trying to go back to is older than "guix
> time-machine". If that's the explanation, it would help if Guix showed
> some clear error message instead of crashing.

It could check whether the target commit is in the closure of the
v0.15.0 commit, but then that would give special treatment to the
existing commit history.  Maybe that’s OK though?

> Next I tried checking out the source code for commit
> f250a868d8c687df08559682fa68fb4ea2a1ea69, and building it from
> source. This is a bit tricky because 2018 Guix cannot be built in
> today's Guix build environment. For example, today we have Guile 3, but
> back then we had Guile 2.2. So I need to do "guix environment guix" in
> an older Guix, before the Guile 3 transition, but later than the
> introduction of time-machine. I picked one somewhat at random:
>
>    $ guix time-machine --commit=e2293cbbe0cd20ddeb932e6f5616565ab468c087
>    -- environment –pure guix
>
> Then I did "bootstrap", "configure –localstatedir=/var", "make
> check". The latter shows 15 failures, some of which look important:
>
>    FAIL: tests/builders.scm
>    FAIL: tests/derivations.scm
>    FAIL: tests/packages.scm
>    FAIL: tests/guix-environment.sh
>    FAIL: tests/guix-daemon.sh

We’d have to check what’s in ‘test-suite.log’; it might as well be an
environment issue more than a “real” problem.

> And indeed I cannot build much with my compiled guix:
>
>    $ ./pre-inst-env guix build nmoldyn
>
> hangs after a while, running a binary called "test-lock" for hours.

IIRC that was a bug in a Gnulib test (bundled in several GNU packages)
that would hang on machines with maybe more than 4 cores.  (See commit
acc2dab7f2f50c9169d6388007c770878eae4a9c for example.)  There might be
hints on how to work around it in the mailing list archive…

> Given the time lapse, I suppose there have been incompatible changes in
> the build daemon, making the old Guix incompatible with the rather
> recent build daemon running on my machine. But is there a way around
> this, other than installing an old Guix in a fully isolated VM?
>
> And if installing the old Guix in a VM is the only solution, what would
> be the best way to do that? I can't think of much else than starting
> from another distribution (e.g. Debian) and following the installation
> instructions. That's already a lot of work, but it's made worse by the
> installation instructions being hidden inside the manual of that old
> commit, which I cannot easily consult.

Again, none of this should be necessary as long as the target commit is
after 0.15.0 (or 1.0.0).

That said, it may be possible to build that Jan. 2018 Guix using an
environment created from Guix 0.15.0 or 1.0.0; it’s likely to have the
right versions of dependencies, and it should be reachable with
time-machine.

I’d be curious to hear how it goes!

Besides, I recently added ‘etc/time-travel-manifest.scm’ and added a
corresponding jobset at <https://ci.guix.gnu.org/jobset/time-travel>
(the jobset is not working well, but that’s because there are several
processes concurrently accessing the same cached checkout; we need a
trick to work around that…).  I’d like to consolidate this so we can
make sure that traveling back in time (after 0.15.0 or 1.0.0) remains a
possibility.

Thanks,
Ludo’.

Re: A real-life test of long-term reproducibility

[Timothy Sample]

[-- Attachment #1: Type: text/plain, Size: 5385 bytes --]

Hi Konrad,

Ludovic Courtès <ludo@gnu.org> writes:

> Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
>
>> The package I want to rebuild and use is "nmoldyn" from Guix commit
>> f250a868d8c687df08559682fa68fb4ea2a1ea69. That's the commit referenced
>> in my notes, obtained via "guix describe" in early 2018. I am pretty
>> sure it worked fine back then.
>
> That’s a commit from January 2018, which is a few months before the
> release of 0.15.0, the first release with proper ‘guix pull’ support:
>
>   https://guix.gnu.org/en/blog/2018/gnu-guix-and-guixsd-0.15.0-released/
>
> In other words: warranty void.  :-)

Agreed.  That’s probably too early in Guix’s history to get good
results.  Keep in mind that the Software Heritage crawlers only started
indexing Guix packages in 2019.  That being said, I was too curious to
leave this alone, so I tried my hand at building that package from that
commit.

> [...]

> Again, none of this should be necessary as long as the target commit is
> after 0.15.0 (or 1.0.0).
>
> That said, it may be possible to build that Jan. 2018 Guix using an
> environment created from Guix 0.15.0 or 1.0.0; it’s likely to have the
> right versions of dependencies, and it should be reachable with
> time-machine.
>
> I’d be curious to hear how it goes!

It turns out it’s pretty easy, apart from having to boil the ocean.
Here’s what I did.

I decided to try the easiest thing first, which is checkout f250a86 and
try to build it with an environment from modern Guix.  We still have
Guile 2.2 packages, after all.  Here’s the environment that I used:

    $ guix shell --pure guile@2.2 guile2.2-gnutls guile2.2-gcrypt \
          gcc-toolchain coreutils bash autoconf automake guile2.2-git \
          sqlite bzip2 gettext libtool pkg-config grep sed patch \
          diffutils m4 findutils tar xz gzip libgcrypt gawk make\
          texinfo help2man

From there, I was able to build Guix normally.  Some of those packages
were guesses, and some were added to placate “configure”, I’m not sure
that all of them are necessary, but they are sufficient.

After that, I tried

    $ ./pre-inst-env guix build nmoldyn

It hummed along for a long time, and then my desktop crashed due to the
kernel’s out-of-memory process killer.  For whatever reason, and even
with 16G of RAM and 16G of swap, the Python 3 build gets smitten by the
OOM killer.  It happens when running the test suite, so I disabled it.
(From what I read, the test suite is used to generate profiles to train
the optimizer, so this is probably a bad idea.  I decided to forge ahead
instead of worry about this too much.)

Next, it couldn’t find the source code for ImageMagick 6.9.9-30.  It
took a minute, but I tracked it down hidden in the following RPM:

    http://mirror.speedpartner.de/remi-archive/SRPMS/ImageMagick6-6.9.9.30-1.remi.src.rpm

After extracting the tarball and adding it to the store with
“guix download”, the build continued.

There was one other source code issue, which was Mesa 17.2.1.  It was
easy to solve, as they just moved it to the “older-versions” directory
on their server.

The only other two issues were test failures due to certificate expiry.
It happened for Python 2 and NSS.  I remember there was a discussion
about this problem in general recently, but I couldn’t track it down.
For Python 2, I disabled the offending tests, and for NSS, I just
disabled the whole test suite.

After that, I got:

    @ build-succeeded /gnu/store/40x598cvpp16m87z52sj98c62dfbmmka-nmoldyn-3.0.11.drv -
    /gnu/store/khf2pf29vfq6b3dcca9p9mvz387jvdl7-nmoldyn-3.0.11

I’m honestly shocked that it worked so well.  I wish I had a better way
to keep track of where the sources came from.  For example, I’m curious
how many came from the build farm or other fallback options.

Overall, I give Guix two thumbs up!  Other than the Python 3 optimizer
bit (which might be solvable), nothing substantive had to be changed to
make this happen.

For best practices, I do have one suggestion.  The Guix package
collection is not uniformly reproducible or archived.  The best thing
you can do to ensure the long-term prospects of your projects is to
actually check how much of the source code is archived and how many of
the builds are reproducible.  There is no turn-key solution for this
(though maybe there should be), but you can check the source code
against the Preservation of Guix database [1], and you can check
reproducibility using the Guix Data Service or with “guix check”.
Ideally everything would be archived and reproducible, but it’s a work
in progress.  By checking, you might be able to fix some of the issues
now while it’s easy instead of in five or ten years when it’s
impossible!

Thanks Konrad for the interesting experiment.  While testing this out, I
came to really appreciate how hackable Guix is.  Even if I couldn’t find
Mesa 17.2.1, say, I could proceed with a similar version or try to build
it with Git.  It’s not ideal to have to make changes, but it’s nice to
know that Guix fails gracefully.

In case it’s useful, I’ve attached the changes I made.


-- Tim

[1] It’s at <https://ngyro.com/pog-reports/latest/>, and yes, I know I
ought to update it!  :)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch.diff --]
[-- Type: text/x-patch, Size: 2727 bytes --]

diff --git a/gnu/packages/gl.scm b/gnu/packages/gl.scm
index c42b4a00cb..83629d30e4 100644
--- a/gnu/packages/gl.scm
+++ b/gnu/packages/gl.scm
@@ -223,7 +223,10 @@ also known as DXTn or DXTC) for Mesa.")
     (source
       (origin
         (method url-fetch)
-        (uri (list (string-append "https://mesa.freedesktop.org/archive/"
+        (uri (list (string-append "https://archive.mesa3d.org/"
+                                  "older-versions/17.x/"
+                                  "mesa-" version ".tar.xz")
+                   (string-append "https://mesa.freedesktop.org/archive/"
                                   "mesa-" version ".tar.xz")
                    (string-append "ftp://ftp.freedesktop.org/pub/mesa/"
                                   "mesa-" version ".tar.xz")
diff --git a/gnu/packages/gnuzilla.scm b/gnu/packages/gnuzilla.scm
index a31e93d499..da9317cb42 100644
--- a/gnu/packages/gnuzilla.scm
+++ b/gnu/packages/gnuzilla.scm
@@ -336,7 +336,8 @@ in the Mozilla clients.")
              (setenv "DOMSUF" "(none)")
              (setenv "USE_IP" "TRUE")
              (setenv "IP_ADDRESS" "127.0.0.1")
-             (zero? (system* "./nss/tests/all.sh"))))
+             ;;(zero? (system* "./nss/tests/all.sh"))
+             #t))
            (replace 'install
              (lambda* (#:key outputs #:allow-other-keys)
                (let* ((out (assoc-ref outputs "out"))
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 29fa2809ab..f11f405331 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -162,7 +162,10 @@
                       "Lib/test/test_import.py"
                       "Lib/test/test_shutil.py"
                       "Lib/test/test_socket.py"
-                      "Lib/test/test_subprocess.py"))
+                      "Lib/test/test_subprocess.py"
+                      "Lib/test/test_httplib.py"
+                      "Lib/test/test_ssl.py"
+                      "Lib/test/test_urllib2_localnet.py"))
           #t))))
     (outputs '("out"
                "tk"))                     ;tkinter; adds 50 MiB to the closure
@@ -355,7 +358,12 @@ data types.")
                               "Lib/test/test_fcntl.py"))
                   #t))))
     (arguments (substitute-keyword-arguments (package-arguments python-2)
-                 ((#:tests? _) #t)))
+                 ((#:tests? _) #t)
+                 ((#:phases phases)
+                  `(modify-phases ,phases
+                     (add-before 'build 'set-extratestopts
+                       (lambda _
+                         (setenv "EXTRATESTOPTS" "--help")))))))
     (native-search-paths
      (list (search-path-specification
             (variable "PYTHONPATH")

Re: A real-life test of long-term reproducibility

[Konrad Hinsen]

Hi Ludo and Tim,

Thanks for your comments and experiments!


Ludovic Courtès <ludo@gnu.org> writes:

> That’s a commit from January 2018, which is a few months before the
> release of 0.15.0, the first release with proper ‘guix pull’ support:

Ouch. I wasn't aware that even "guix pull" happened later!

>> I don't understand what is going wrong here, but it may be related to
>> the fact that the commit I am trying to go back to is older than "guix
>> time-machine". If that's the explanation, it would help if Guix showed
>> some clear error message instead of crashing.
>
> It could check whether the target commit is in the closure of the
> v0.15.0 commit, but then that would give special treatment to the
> existing commit history.  Maybe that’s OK though?

I'd say yes, because the only problem is the existing commit history.
At least until we extend time-machine to actually change the past ;-)

> IIRC that was a bug in a Gnulib test (bundled in several GNU packages)
> that would hang on machines with maybe more than 4 cores.  (See commit
> acc2dab7f2f50c9169d6388007c770878eae4a9c for example.)  There might be
> hints on how to work around it in the mailing list archive…

Hardware dependencies...

> That said, it may be possible to build that Jan. 2018 Guix using an
> environment created from Guix 0.15.0 or 1.0.0; it’s likely to have the
> right versions of dependencies, and it should be reachable with
> time-machine.

I'll try with 1.0.0, that looks nicer than 0.15.0 ;-)

> Besides, I recently added ‘etc/time-travel-manifest.scm’ and added a
> corresponding jobset at <https://ci.guix.gnu.org/jobset/time-travel>

Nice! Guix will be certified for time travel!


Timothy Sample <samplet@ngyro.com> writes:

> It turns out it’s pretty easy, apart from having to boil the ocean.
> Here’s what I did.

Very interesting, thanks!

> I’m honestly shocked that it worked so well.  I wish I had a better way
> to keep track of where the sources came from.  For example, I’m curious
> how many came from the build farm or other fallback options.
>
> Overall, I give Guix two thumbs up!  Other than the Python 3 optimizer
> bit (which might be solvable), nothing substantive had to be changed to
> make this happen.

Indeed. And yet, from the point of view of a non-expert user, even the
slightest fix required is a show stopper.

> For best practices, I do have one suggestion.  The Guix package
> collection is not uniformly reproducible or archived.  The best thing
> you can do to ensure the long-term prospects of your projects is to
> actually check how much of the source code is archived and how many of
> the builds are reproducible.  There is no turn-key solution for this

Yes, that's a good idea, and I have done it for my most recent packages.
Time will tell if this is enough.

> Thanks Konrad for the interesting experiment.  While testing this out, I
> came to really appreciate how hackable Guix is.  Even if I couldn’t find
> Mesa 17.2.1, say, I could proceed with a similar version or try to build
> it with Git.  It’s not ideal to have to make changes, but it’s nice to
> know that Guix fails gracefully.

Definitely!

Cheers,
  Konrad.

Re: A real-life test of long-term reproducibility

[zimoun]

Hi Konrad,

On lun., 08 août 2022 at 10:44, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

>> Besides, I recently added ‘etc/time-travel-manifest.scm’ and added a
>> corresponding jobset at <https://ci.guix.gnu.org/jobset/time-travel>
>
> Nice! Guix will be certified for time travel!

[...]

>> For best practices, I do have one suggestion.  The Guix package
>> collection is not uniformly reproducible or archived.  The best thing
>> you can do to ensure the long-term prospects of your projects is to
>> actually check how much of the source code is archived and how many of
>> the builds are reproducible.  There is no turn-key solution for this
>
> Yes, that's a good idea, and I have done it for my most recent packages.
> Time will tell if this is enough.

Now, Guix is checking that “guix time-machine” does not break; i.e, be
able to rebuild a previous Guix using inferior.  That’s cool!

However, many things can be out of rail.  This claim about
reproducibility over the time assumes:

 1. compatibility of the Linux kernel
 2. availability of all the source code
 3. compatibility of the hardware

Well, until now, nothing had been reported about #1.  But, we have
examples of issues about #2 and #3.

For instance, about #2, Timothy reported the loss of the source code of
ImageMagick 6.9.9-30 – some time ago, I reported another issue with
another ImageMagick version from 2020
(<https://yhetil.org/guix/87tuebwlrc.fsf@gmail.com>).  Although the
project is making many efforts to archive all the source, the coverage
is not 100%:

    https://ngyro.com/pog-reports/latest/

and only one tiny loss of only one node in the graph of dependencies,
then all the efforts are ruined.

About #3, the new NVMe disks leads to an issue with bootstrapping; as
reported by <https://issues.guix.gnu.org/41264>.  It means that, if the
binary substitutes are lost and I have only a machine with NVMe, then I
cannot rebuild from scratch.


All that said, Guix is the best and most advanced solution on the market
for reproducible time-traveling. :-)  For most of the cases, it is
awesome to just type “guix time-machine” and rebuild a complete
computational environment exactly as it was 2 or 3 years ago.


On lun., 08 août 2022 at 10:49, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> Even 1.0.0 isn't obvious:
>
>   $ guix time-machine --commit=version-1.0.0 -- environment guix
>   guix time-machine: error: Git error: unable to parse OID - contains invalid characters
>
> OK, so let's try the commit hash:
>
>   $ guix time-machine --commit=48aa30ce73d45dc5f126f42f01e65f1be4a9b578 -- environment guix
>   Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>   Authenticating channel 'guix', commits 9edb3f6 to 48aa30c (6 new commits)...
>   guix time-machine: error: commit
>   48aa30ce73d45dc5f126f42f01e65f1be4a9b578 is not a descendant of
>   introductory commit 9edb3f66fd807b096b48283debdcddccfea34bad

That’s because version-1.0.0 (48aa30ce73) is a branch and indeed not a
descendant.

--8<---------------cut here---------------start------------->8---
* 746ac457cc Merge branch 'version-1.0.0' 
|\ 
* | c457f109be gnu: php: Update to 7.3.5. 

[...]

* | 1a8984536f gnu: Add sdl2-net. 
| |  * 48aa30ce73 build: Add 'doc/build.scm' to build on-line copies of the manual.  (origin/version-1.0.0)
| |  * adf577dcc4 doc: Update htmlxref.cnf. 
| |  * 1a9fc8e228 doc: Warn about missing entries in htmlxref.cnf. 
| |  * 2921b6a611 doc: Adjust cross-references for GNU triplets. 
| |  * 3aa11dfbed doc: Provide the actual URL to the VM image. 
| |  * 542e7fb57f doc: Add note about <https://bugs.gnu.org/35541>. 
| | /  
| |/   
| * 3a3e9f2bb5 guix-install.sh: Update URL. 
| * 9c941364bf vm: Build ISOs and VM images in a UTF-8 environment. 
| * 17acc215bf gnu: guix: Update to 326dcbf. 
| * 326dcbf1b3 gnu: guix: Update to 1.0.0. 
| * 6298c3ffd9 Update NEWS.  (tag: v1.0.0)
--8<---------------cut here---------------end--------------->8---

What you want is tag v1.0.0 (6298c3ffd9).  Otherwise, you need the
option ’--disable-authentication’. 


Cheers,
simon

Re: A real-life test of long-term reproducibility

[Konrad Hinsen]

Hi Simon,

> However, many things can be out of rail.  This claim about
> reproducibility over the time assumes:
>
>  1. compatibility of the Linux kernel
>  2. availability of all the source code
>  3. compatibility of the hardware
>
> Well, until now, nothing had been reported about #1.  But, we have
> examples of issues about #2 and #3.

Thanks for pointing this out. I have had #3 issues myself, with a
package (OpenBLAS or something related) failing to build on processors
later than the software release date. But I guess #2 is the main issue
for reviving old code. Software Heritage should settle that for the
future.

> All that said, Guix is the best and most advanced solution on the market
> for reproducible time-traveling. :-)  For most of the cases, it is
> awesome to just type “guix time-machine” and rebuild a complete
> computational environment exactly as it was 2 or 3 years ago.

Indeed!

> That’s because version-1.0.0 (48aa30ce73) is a branch and indeed not a
> descendant.
...
> What you want is tag v1.0.0 (6298c3ffd9).  Otherwise, you need the
> option ’--disable-authentication’. 

OK, thanks, I will try with that!

Cheers,
  Konrad.

Re: A real-life test of long-term reproducibility

[Konrad Hinsen]

Hi Ludo,

> We can go back to 1.0.0, and presumably to 0.15.0, but anything older
> than this is unknown territory.

Even 1.0.0 isn't obvious:

  $ guix time-machine --commit=version-1.0.0 -- environment guix
  guix time-machine: error: Git error: unable to parse OID - contains invalid characters

OK, so let's try the commit hash:

  $ guix time-machine --commit=48aa30ce73d45dc5f126f42f01e65f1be4a9b578 -- environment guix
  Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
  Authenticating channel 'guix', commits 9edb3f6 to 48aa30c (6 new commits)...
  guix time-machine: error: commit
  48aa30ce73d45dc5f126f42f01e65f1be4a9b578 is not a descendant of
  introductory commit 9edb3f66fd807b096b48283debdcddccfea34bad

Cheers,
  Konrad.

Re: A real-life test of long-term reproducibility

[Ludovic Courtès]

Hi,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>   $ guix time-machine --commit=48aa30ce73d45dc5f126f42f01e65f1be4a9b578 -- environment guix
>   Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>   Authenticating channel 'guix', commits 9edb3f6 to 48aa30c (6 new commits)...
>   guix time-machine: error: commit
>   48aa30ce73d45dc5f126f42f01e65f1be4a9b578 is not a descendant of
>   introductory commit 9edb3f66fd807b096b48283debdcddccfea34bad

Here you would need ‘--allow-downgrades’.

Ludo’.

Re: A real-life test of long-term reproducibility

[zimoun]

Hi Ludo,

On Fri, 02 Sep 2022 at 15:17, Ludovic Courtès <ludo@gnu.org> wrote:

> Here you would need ‘--allow-downgrades’.

Maybe time-machine could advertise of this option?


Well aside, it can be indeed confusing that the version-1.x branch does
not match the tag v1.x; I understand why.

--8<---------------cut here---------------start------------->8---
$ git log --graph --format="%h %s %d" v1.0.0^..origin/version-1.0.0
* 48aa30ce73 build: Add 'doc/build.scm' to build on-line copies of the manual.  (origin/version-1.0.0)
* adf577dcc4 doc: Update htmlxref.cnf. 
* 1a9fc8e228 doc: Warn about missing entries in htmlxref.cnf. 
* 2921b6a611 doc: Adjust cross-references for GNU triplets. 
* 3aa11dfbed doc: Provide the actual URL to the VM image. 
* 542e7fb57f doc: Add note about <https://bugs.gnu.org/35541>. 
* 3a3e9f2bb5 guix-install.sh: Update URL. 
* 9c941364bf vm: Build ISOs and VM images in a UTF-8 environment. 
* 17acc215bf gnu: guix: Update to 326dcbf. 
* 326dcbf1b3 gnu: guix: Update to 1.0.0. 
* 6298c3ffd9 Update NEWS.  (tag: v1.0.0)
--8<---------------cut here---------------end--------------->8---

Does it make sense to document it?  And does it make sense to merge to
master right after 326dcbf1b3?  And not after some commits:

--8<---------------cut here---------------start------------->8---
$ for i in $(seq 0 3); do \
   git log --graph --format="%h %s %d" v1.$i.0^..origin/version-1.$i.0 \
   | wc -l ;done   
11
6
warning: refname 'v1.2.0' is ambiguous.
10
7
--8<---------------cut here---------------end--------------->8---



Cheers,
simon

Re: A real-life test of long-term reproducibility

[Konrad Hinsen]

Hi Ludo and Simon,

> On Fri, 02 Sep 2022 at 15:17, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Here you would need ‘--allow-downgrades’.
>
> Maybe time-machine could advertise of this option?

And explain what it's about. I don't consider myself an absolute
beginner with Guix, but I don't understand what's going on here!

> Well aside, it can be indeed confusing that the version-1.x branch does
> not match the tag v1.x; I understand why.

OK, now I do a well... but should users need to understand the
developers' git workflow?

Ultimately, the issue here is that guix works on the implementation
level of git commit references, whereas the development process uses an
abstraction layer on top of it. Maybe "guix pull" and "guix
time-machine" should acquire some built-in intelligence when given a tag
instead of a hash? Issue a warning when an inappropriate tag is given,
and suggest a correction?

Cheers,
  Konrad.

Re: A real-life test of long-term reproducibility

[zimoun]

Hi Konrad,

On lun., 05 sept. 2022 at 09:49, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
> Hi Ludo and Simon,
>
>> On Fri, 02 Sep 2022 at 15:17, Ludovic Courtès <ludo@gnu.org> wrote:
>>
>>> Here you would need ‘--allow-downgrades’.
>>
>> Maybe time-machine could advertise of this option?
>
> And explain what it's about. I don't consider myself an absolute
> beginner with Guix, but I don't understand what's going on here!

Well, it is documented. ;-) The entry of ’time-machine’ [1] is the
manual says:

        The general syntax is:

            guix time-machine options… -- command arg…

        where command and arg… are passed unmodified to the guix command
        of the specified revision. The options that define this revision
        are the same as for guix pull (see Invoking guix pull):

then the entry of ’guix pull’ [2] explains the option:

        --allow-downgrades

          Allow pulling older or unrelated revisions of channels than
          those currently in use.

          By default, guix pull protects against so-called “downgrade
          attacks” whereby the Git repository of a channel would be
          reset to an earlier or unrelated revision of itself,
          potentially leading you to install older, known-vulnerable
          versions of software packages.

Last, a blog post [3] provides some details.

I agree it could be improved.  Any suggestion? :-)


1: https://guix.gnu.org/manual/devel/en/guix.html#Invoking-guix-time_002dmachine
2: https://guix.gnu.org/manual/devel/en/guix.html#Invoking-guix-pull
3: https://guix.gnu.org/en/blog/2020/securing-updates/


> OK, now I do a well... but should users need to understand the
> developers' git workflow?

I agree.


> Ultimately, the issue here is that guix works on the implementation
> level of git commit references, whereas the development process uses an
> abstraction layer on top of it. Maybe "guix pull" and "guix
> time-machine" should acquire some built-in intelligence when given a tag
> instead of a hash? Issue a warning when an inappropriate tag is given,
> and suggest a correction?

Well, the tags are there and usable.  But, Guix relies on guile-git and
uses string->oid; which supporting hash-commit and not string-tag.
That’s a technical limitation which could be easily improved, I guess.

Even, we could imagine local Git-tags using “guix git tag” and reuse
them via the time-machine.  Some code are around for “guix git log” (see
branch wip-guix-log [4]).

(However, although convenient, Git-tag is extrinsic (mutable) and thus it
is not suitable for long-term, where intrinsic (immutable) seems
preferred.)

4: https://git.savannah.gnu.org/cgit/guix.git/log/?h=wip-guix-log


Cheers,
simon

Re: A real-life test of long-term reproducibility

[Ludovic Courtès]

Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

> Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:
>
>>   $ guix time-machine --commit=48aa30ce73d45dc5f126f42f01e65f1be4a9b578‘> -- environment guix
>>   Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
>>   Authenticating channel 'guix', commits 9edb3f6 to 48aa30c (6 new commits)...
>>   guix time-machine: error: commit
>>   48aa30ce73d45dc5f126f42f01e65f1be4a9b578 is not a descendant of
>>   introductory commit 9edb3f66fd807b096b48283debdcddccfea34bad
>
> Here you would need ‘--allow-downgrades’.

Sorry I replied to quickly; it should have been
‘--disable-authentication’.

Why?  Because here you’re trying to travel to a commit that’s not part
of the “authenticated history”—that is, it’s not a descendant of the
“introductory commit”, which is the first commit in history starting
from which the “authentication invariant” holds.

Hopefully this is clearer; maybe?  :-)

Cheers,
Ludo’.

Re: A real-life test of long-term reproducibility

[Konrad Hinsen]

Hi Ludo and Simon,

> Sorry I replied to quickly; it should have been
> ‘--disable-authentication’.
>
> Why?  Because here you’re trying to travel to a commit that’s not part
> of the “authenticated history”—that is, it’s not a descendant of the
> “introductory commit”, which is the first commit in history starting
> from which the “authentication invariant” holds.
>
> Hopefully this is clearer; maybe?  :-)

It's perfectly clear. I just wonder how I would find the solution
starting from the error message.

zimoun <zimon.toutoune@gmail.com> writes:

>> And explain what it's about. I don't consider myself an absolute
>> beginner with Guix, but I don't understand what's going on here!
>
> Well, it is documented. ;-) The entry of ’time-machine’ [1] is the
> manual says:
...

Same remark: How would I get to that section of the manual from the
error message?

> I agree it could be improved.  Any suggestion? :-)

My impression is that the error message I quoted is very specific to
this situation, so it could actually say "Retry with
--disable-authentication if you are sure that there is no security
issue". Right?

Cheers,
  Konrad.