weird OpenBLAS time-machine

weird OpenBLAS time-machine

[zimoun]

Hi,

Investigating [0], I note that, for these 2 commits,

        $ git log --oneline 4b1538e6ef^..ab0cf06244
        ab0cf06244 gnu: rust: Remove #:rust ,rust-1.52 arguments.
        4b1538e6ef gnu: kexec-tools: Fix build on i686-linux.

where the latter is simply the parent of the former,

        $ git cat-file -p ab0cf06244 | grep parent
        parent 4b1538e6ef76bf46993f0a368a0abbe2f6eb8ffb

then the version of OpenBLAS is different.

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=ab0cf06244 -- show openblas | recsel -P version
0.3.18

$ guix time-machine --commit=4b1538e6ef -- show openblas | recsel -P version
0.3.9
--8<---------------cut here---------------end--------------->8---

The update to 0.3.18 from 0.3.13 is done by bd771edd6c, and as far I
see, there is no merge around; from bd771edd6c to ab0cf06244, all are
direct parents.

Sadly, it is not possible to time-machine to bd771edd6c (Update to
0.3.18) because an unrelated error ((value "Unbound variable: ~S")
(value (rust-1.51))).  For some commits between this range, the
time-machine is also raising an error because (value "Unbound variable:
~S") (value (rust-1.52)).  Anyway, another story. :-)


What is really weird is:

--8<---------------cut here---------------start------------->8---
$ cd $CHECKOUT
$ git show 4b1538e6ef:gnu/packages/maths.scm \
      | grep -n -A 5 'define-public openblas'
4402:(define-public openblas
4403-  (package
4404-    (name "openblas")
4405-    (version "0.3.18")
4406-    (source
4407-     (origin
--
4486:(define-public openblas-ilp64
4487-  (package/inherit openblas
4488-    (name "openblas-ilp64")
4489-    (supported-systems '("x86_64-linux" "aarch64-linux" "mips64el-linux"))
4490-    (arguments
4491-     (substitute-keyword-arguments (package-arguments openblas)
--8<---------------cut here---------------end--------------->8---

For 4b1538e6ef the version should be 0.3.18 but then:

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=4b1538e6ef -- show openblas
name: openblas
version: 0.3.9
outputs: out
systems: x86_64-linux i686-linux
dependencies: cunit@2.1-3 gfortran@7.5.0 perl@5.30.2
location: gnu/packages/maths.scm:3655:2
homepage: https://www.openblas.net/
license: Modified BSD
synopsis: Optimized BLAS library based on GotoBLAS
description: OpenBLAS is a BLAS library forked from the GotoBLAS2-1.13 BSD version.
--8<---------------cut here---------------end--------------->8---


Other said, the package defined in
/gnu/store/…-guix-4b1538e6e-modules/share/guile/site/3.0/gnu/packages
does not match the version is the checkout.


Concretely, compare the file gnu/packages/maths.scm from:

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=4b1538e6ef -- edit openblas
$ git -C ~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq/ \
      show 4b1538e6ef:gnu/packages/maths.scm
--8<---------------cut here---------------end--------------->8---

Here the diff for only OpenBLAS.

--8<---------------cut here---------------start------------->8---
$ diff /tmp/maths-modules.scm /tmp/maths-checkout.scm | grep -A 45 -B 3 '0.3.18'
3657c4405
<     (version "0.3.9")
---
>     (version "0.3.18")
3660,3663c4408,4412
<        (method url-fetch)
<        (uri (string-append "mirror://sourceforge/openblas/v" version "/OpenBLAS%20"
<                            version "%20version.tar.gz"))
<        (file-name (string-append name "-" version ".tar.gz"))
---
>        (method git-fetch)
>        (uri (git-reference
>              (url "https://github.com/xianyi/OpenBLAS")
>              (commit (string-append "v" version))))
>        (file-name (git-file-name name version))
3666c4415
<          "14iz9xnrb9xiwgj84j94mc74gg0zn2vsy9fmsijxxma1n7dck4w3"))))
---
>          "17zdd8asylz2w71hczrz5y344p6d5ds1jn4901maw7zcp3dbk63g"))))
3682a4432
>              "NO_STATIC=1"              ;avoid a 67 MiB static archive
3693c4443
<              ;; Unfortunately, this is not supported on non-x86 architectures,
---
>              ;; Unfortunately, this is not supported on all architectures,
3698,3699c4448,4454
<                        (string-prefix? "i686" system))
<                    '("DYNAMIC_ARCH=1"))
---
>                        (string-prefix? "i686" system)
>                        (string-prefix? "powerpc64le" system)
>                        (string-prefix? "aarch64" system))
>                    ;; Dynamic older enables a few extra CPU architectures that
>                    ;; were released before 2010.
>                    '("DYNAMIC_ARCH=1" "DYNAMIC_OLDER=1" "TARGET=GENERIC"))
>                   ;; On some of these architectures the CPU can't be detected.
3705,3707c4460,4462
<                   ;; On aarch64 force the generic 'armv8-a' target
<                   ((string-prefix? "aarch64" system)
<                    '("TARGET=ARMV8"))
---
>                   ;; Failed to detect CPU.
>                   ((string-prefix? "armhf" system)
>                    '("TARGET=ARMV7"))
3718,3719c4473
<                                     "/lib"))
<              #t)))))
---
>                                     "/lib")))))))
--8<---------------cut here---------------end--------------->8---


What could be wrong?


Cheers,
simon


0: <https://lists.gnu.org/archive/html/guix-devel/2022-02/msg00000.html>

Re: weird OpenBLAS time-machine

[zimoun]

Hi,

On Wed, 02 Feb 2022 at 18:00, zimoun <zimon.toutoune@gmail.com> wrote:

> --8<---------------cut here---------------start------------->8---

[...]

> $ guix time-machine --commit=4b1538e6ef -- show openblas | recsel -P version
> 0.3.9
> --8<---------------cut here---------------end--------------->8---

The issue is because concurrency.  If two time-machines are run
concurrently, they both update ~/.cache/guix/checkouts/ and the end
result is hard to predict.

Well, I probably ran inside one terminal “guix time-machine
--commit=XXXX -- help” where XXXX a commit with openblas@0.3.9; and in
the same time inside another terminal, “guix time-machine
--commit=4b1538e6ef -- help”.  Depending on “the same time”, the state
of the checkout
~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
used by the time-machine at 4b1538e6ef was the state for the commit
XXXX.

Then, all is messed!

Somehow, IIUC the code behind, what is missing is a lock when updating
the cached checkout to prevent concurrent unrelated updates.


For sure, on 3 independent other machines, and after GC the initial
machine, I get the same error for this commit 4b1538e6ef:

--8<---------------cut here---------------start------------->8---
(repl-version 0 1 1)
Generating package cache for '/gnu/store/9zq283291ak72xrvlhax89gyl578kbbg-profile'...
(exception unbound-variable (value #f) (value "Unbound variable: ~S") (value (rust-1.52)) (value #f))
--8<---------------cut here---------------end--------------->8---


Sorry for the noise.

Cheers,
simon

Re: weird OpenBLAS time-machine

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 985 bytes --]

zimoun schreef op do 03-02-2022 om 03:19 [+0100]:
> The issue is because concurrency.  If two time-machines are run
> concurrently, they both update ~/.cache/guix/checkouts/ and the end
> result is hard to predict.
> 
> Well, I probably ran inside one terminal “guix time-machine
> --commit=XXXX -- help” where XXXX a commit with openblas@0.3.9; and in
> the same time inside another terminal, “guix time-machine
> --commit=4b1538e6ef -- help”.  Depending on “the same time”, the state
> of the checkout
> ~/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
> used by the time-machine at 4b1538e6ef was the state for the commit
> XXXX.
> 
> Then, all is messed!

FWIW it's a known issue but I can't find it on issues.guix.gnu.org.
The (unimplemented) fix is to use worktrees, or don't checkout and use
the libgit2 / (guix git) equivalent of
"git show 46fc72b2bfee2a30a3c3f3320e7d84b4b2fd646e some-file".

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: weird OpenBLAS time-machine

[zimoun]

Hi,

On Thu, 3 Feb 2022 at 10:21, Maxime Devos <maximedevos@telenet.be> wrote:
> zimoun schreef op do 03-02-2022 om 03:19 [+0100]:

> > The issue is because concurrency.  If two time-machines are run
> > concurrently, they both update ~/.cache/guix/checkouts/ and the end
> > result is hard to predict.

[...]

> FWIW it's a known issue but I can't find it on issues.guix.gnu.org.
> The (unimplemented) fix is to use worktrees, or don't checkout and use
> the libgit2 / (guix git) equivalent of
> "git show 46fc72b2bfee2a30a3c3f3320e7d84b4b2fd646e some-file".

Is Guile-Git supporting git-wortree?

Cheers,
simon

Re: weird OpenBLAS time-machine

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

> zimoun schreef op do 03-02-2022 om 11:42 [+0100]:
> [...]
> 
> > FWIW it's a known issue but I can't find it on issues.guix.gnu.org.
> > The (unimplemented) fix is to use worktrees, or don't checkout and use
> > the libgit2 / (guix git) equivalent of
> > "git show 46fc72b2bfee2a30a3c3f3320e7d84b4b2fd646e some-file".
> 
> Is Guile-Git supporting git-wortree?

I don't know.  In any case, libgit2 supports worktrees
(https://libgit2.org/libgit2/#HEAD/group/worktree/git_worktree_add), so
adding Guile bindings should be doable if it hasn't been done already.

CC'ing ludo because ludo commented on the issue IIRC.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: weird OpenBLAS time-machine

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> zimoun schreef op do 03-02-2022 om 03:19 [+0100]:
>> The issue is because concurrency.  If two time-machines are run
>> concurrently, they both update ~/.cache/guix/checkouts/ and the end
>> result is hard to predict.

Ah, glad you found out the culprit!

> FWIW it's a known issue but I can't find it on issues.guix.gnu.org.
> The (unimplemented) fix is to use worktrees, or don't checkout and use
> the libgit2 / (guix git) equivalent of
> "git show 46fc72b2bfee2a30a3c3f3320e7d84b4b2fd646e some-file".

Hmm I cannot find it either.  But yeah, probably better than worktrees
would be reading blobs directly off Git.

Thanks,
Ludo’.