guix build --repair might not always work

guix build --repair might not always work

[Nicolò Balzarotti]

#-*- org -*-

Hello Guix!

Sorry for the long mail, but I added all details I could think of in
order to try to debug this error ().

What happened: After a power failure during a guix pull + guix reconfigure, my /gnu/store ended up in a bad state.

Emacs failed to run with errors like:
#+begin_quote
/gnu/store/9i4lhk2g1f2k6sfwk9pyc8idx1ffgilw-emacs-26.3/bin/emacs-26.3: error while loading shared libraries: /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3/lib/libXft.so.2:
 file too short                                                                                                                                                                     
#+end_quote

I run
#+begin_src bash :results list
# This is the broken system
guix gc --verify=repair,contents
#+end_src

#+RESULTS:
- reading the store...
- checking path existence...
- checking hashes...

And nothing else.  So, let's check the hash

#+begin_src bash
# This is the broken system
guix hash -r /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3
#+end_src
#+RESULTS:
: 08fxn4xasrvv5ljydvy6xzxyzpd93f7rk2w0czp70z9fi3qdng9f

This is different than a hash I get on another system:
#+begin_src bash
# This is the sane system
guix hash -r /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3
#+end_src

#+RESULTS:
: 12lx9ar5vb2l4dwgw1cfjqyrfscca4rs6vmnxnza7l8qys97zps3

#+begin_src bash
# This is the broken system
sha1sum /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3/lib/libXft.so.2.3.3
#+end_src

#+RESULTS:
: da39a3ee5e6b4b0d3255bfef95601890afd80709  /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3/lib/libXft.so.2.3.3

#+begin_src bash
# This is the sane system
sha1sum /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3/lib/libXft.so.2.3.3
#+end_src

#+RESULTS:
: ce886196f0ab2bab3d27796b61627e9efcbd9eb0  /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3/lib/libXft.so.2.3.3

#+begin_src bash :results table
# This is the broken system
ls -lah  /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3/lib/libXft.so.2.3.3 | sed -s 's/ /|/g'
#+end_src

#+RESULTS:
| -r-xr-xr-x | 4 | root | root | 0 | Jan |   | 1 |   | 1970 | /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3/lib/libXft.so.2.3.3 |


#+begin_src bash :results table
# This is the sane system
ls -lah  /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3/lib/libXft.so.2.3.3 | sed -s 's/ /|/g'
#+end_src

#+RESULTS:
| -r-xr-xr-x | 2 | root | root | 109K | Jan |   | 1 |   | 1970 | /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3/lib/libXft.so.2.3.3 |

The manual also suggests this command:
#+begin_src bash :results table :dir /sudo:localhost:
# This is the broken system
guix build --repair libxft
#+end_src

#+RESULTS:
| /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3 |

Same on the sane system.

Do you have any idea on why is it failing? On IRC, <bricewge> said he had the
same problem and "solved" it by re-install guix.  Not being able to find which path is broken is a serious problem.

Last thing I could do is to check the database (file:/var/guix/db/db.sqlite)
#+begin_src sqlite :eval never
select * from ValidPaths where path = "/gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3";
#+end_src

#+RESULTS:
| 24700 | /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3 | sha256:2e3ddbf0882e7d70ee67808b998f1ba9ddeffbefc6efe6252d7b67ad3ab1dd21 | 1579715645 | /gnu/store/ajkyhj4qsbzw7kqc4wp9yfzci7r5xgyy-libxft-2.3.3.drv | 10104 |

# Sane System:
#+begin_src sqlite :eval never
select * from ValidPaths where path = "/gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3";
#+end_src
| 162188 | /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3 | sha256:43df7f92f618d1a3beedb66ea333518c69973d968e05fe782354ac5db24a9d8a | 1571144877 | /gnu/store/kv1xa04z0z8rmn5g365gg4glrgxp4cvr-libxft-2.3.3.drv | 272672 |

#+begin_src sqlite :eval never
cat /gnu/store/ajkyhj4qsbzw7kqc4wp9yfzci7r5xgyy-libxft-2.3.3.drv
#+end_src

#+RESULTS:
#+begin_example
Derive([("out","/gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3","","")],[("/gnu/store/8qakp7k3ayv335z5cz3v0gmm2wsc2l0l-expat-2.2.9.drv",["out"]),("/gnu/store/ac3g11x2nznnd93wbhlgllbmwmcvrci3-module-import-compiled.drv",["out"]),("/gnu/store/bv9py3f2dsa5iw0aijqjv9zxwprcy1nb-fontconfig-2.13.1.drv",["out"]),("/gnu/store/fbclj50yav35pdn6lkkadj3vxdiz5ja6-libxft-2.3.3.drv",["out"]),("/gnu/store/il86vp1ir9bqvpn9cnx3b9qbfbjzjvqa-expat-2.2.7.drv",["out"]),("/gnu/store/nvg11k85cvyw96dj8470wp6c1bcwf5mp-guile-2.0.14.drv",["out"]),("/gnu/store/xv0kp47vd6qhn63py0lz6nq1fniczmmr-module-import.drv",["out"]),("/gnu/store/ydb37apsc4rc24npva269vi7gszcddv8-fontconfig-2.13.1.drv",["out"])],["/gnu/store/g4l2y8in2ksvm58xvyn657vpyycd7gsw-libxft-2.3.3-guile-builder"],"x86_64-linux","/gnu/store/qqdjb8vq2kdknfflkfdkqh6vp3mwgwl8-guile-2.0.14/bin/guile",["--no-auto-compile","-L","/gnu/store/7myzmxzapx7jnvbsyzv4xhfi2gg58d1w-module-import","/gnu/store/g4l2y8in2ksvm58xvyn657vpyycd7gsw-libxft-2.3.3-guile-builder"],[("GUILE_LOAD_COMPILED_PATH","/gnu/store/cn5v2g0zihw9hdsgcnbl6knq4zincfdk-module-import-compiled"),("allowSubstitutes","0"),("guix properties","((type . graft) (graft (count . 2)))"),("out","/gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3"),("preferLocalBuild","1")])
#+end_example

#+begin_src bash :results raw :wrap example
# Sane system
cat /gnu/store/kv1xa04z0z8rmn5g365gg4glrgxp4cvr-libxft-2.3.3.drv
#+end_src

#+RESULTS:
#+begin_example
Derive([("out","/gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3","","")],[("/gnu/store/5abb4chfjhln2z6d02859assm6q9ywfm-guile-2.0.14.drv",["out"]),("/gnu/store/5yzc9yzx5icb9ymsw23d33jpyd4hqf5y-libxft-2.3.3.drv",["out"]),("/gnu/store/iwxyyn37rdh712v7y5fphmidrna5cwq2-module-import-compiled.drv",["out"]),("/gnu/store/kg7qgva4bmii2xp2bi2diqyd9khafxvr-fontconfig-2.13.1.drv",["out"]),("/gnu/store/rp8qh5j8z3x6kz2nkmji8vh1sp1m091d-expat-2.2.9.drv",["out"]),("/gnu/store/skba4njas8mnvvsk2jb87nrfirdlniyk-module-import.drv",["out"]),("/gnu/store/x6idp5bk60y095350k4gf1q45hc8bja1-expat-2.2.7.drv",["out"]),("/gnu/store/xzb84yvlcpgi6f2hidf62147v1hah29a-fontconfig-2.13.1.drv",["out"])],["/gnu/store/g4l2y8in2ksvm58xvyn657vpyycd7gsw-libxft-2.3.3-guile-builder"],"x86_64-linux","/gnu/store/qqdjb8vq2kdknfflkfdkqh6vp3mwgwl8-guile-2.0.14/bin/guile",["--no-auto-compile","-L","/gnu/store/7myzmxzapx7jnvbsyzv4xhfi2gg58d1w-module-import","/gnu/store/g4l2y8in2ksvm58xvyn657vpyycd7gsw-libxft-2.3.3-guile-builder"],[("GUILE_LOAD_COMPILED_PATH","/gnu/store/cn5v2g0zihw9hdsgcnbl6knq4zincfdk-module-import-compiled"),("allowSubstitutes","0"),("guix properties","((type . graft) (graft (count . 2)))"),("out","/gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3"),("preferLocalBuild","1")])
#+end_example

They are completely different so I don't know how to further investigate it.

Last thing I did:
#+begin_src bash :eval never
sudo mount -o remount,rw /gnu/store
echo "hello" > /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3/lib/libXft.so.2.3.3
guix build --repair libxft
#+end_src

Results:
#+begin_example
path `/gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3' is corrupted or missing!
applying 2 grafts for /gnu/store/ajkyhj4qsbzw7kqc4wp9yfzci7r5xgyy-libxft-2.3.3.drv...
grafting '/gnu/store/844ngi6h6icrc3kkmxc1rni4pvwijhzg-libxft-2.3.3' -> '/gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3'...
successfully built /gnu/store/ajkyhj4qsbzw7kqc4wp9yfzci7r5xgyy-libxft-2.3.3.drv
/gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3
#+end_example

Any suggestions on why this happens, how to check for other paths of the store, and if this can be fixed?

Thanks, Nicolò

Re: guix build --repair might not always work

[Ludovic Courtès]

Hi,

anothersms@gmail.com (Nicolò Balzarotti) skribis:

> #+begin_src bash :results list
> # This is the broken system
> guix gc --verify=repair,contents
> #+end_src
>
>
> #+RESULTS:
> - reading the store...
> - checking path existence...
> - checking hashes...
>
> And nothing else.  So, let's check the hash

In particular, you should compare the actual hash returned by “guix hash
-r” with that stored in the local database.  If they match, then the
store item is not considered corrupt.

> The manual also suggests this command:
>
> #+begin_src bash :results table :dir /sudo:localhost:
> # This is the broken system
> guix build --repair libxft
> #+end_src

You should pass ‘--no-grafts’ if you want to repair the ungrafted
variant.

> #+begin_src sqlite :eval never
> select * from ValidPaths where path = "/gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3";
> #+end_src
>
>
> #+RESULTS:
> | 24700 | /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3 | sha256:2e3ddbf0882e7d70ee67808b998f1ba9ddeffbefc6efe6252d7b67ad3ab1dd21 | 1579715645 | /gnu/store/ajkyhj4qsbzw7kqc4wp9yfzci7r5xgyy-libxft-2.3.3.drv | 10104 |

So that’s indeed different from what “guix hash -r” returned:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix base32)
scheme@(guile-user)> ,use(guix base16)
scheme@(guile-user)> (bytevector->base16-string (nix-base32-string->bytevector "12lx9ar5vb2l4dwgw1cfjqyrfscca4rs6vmnxnza7l8qys97zps3"))
$5 = "43df7f92f618d1a3beedb66ea333518c69973d968e05fe782354ac5db24a9d8a"
--8<---------------cut here---------------end--------------->8---

Thus, ‘guix gc --verify=contents’ should report it as corrupt.

> #+begin_src bash :eval never
> sudo mount -o remount,rw /gnu/store
> echo "hello" > /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3/lib/libXft.so.2.3.3
> guix build --repair libxft
> #+end_src
>
>
> Results:
>
> #+begin_example
> path `/gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3' is corrupted or missing!
> applying 2 grafts for /gnu/store/ajkyhj4qsbzw7kqc4wp9yfzci7r5xgyy-libxft-2.3.3.drv...
> grafting '/gnu/store/844ngi6h6icrc3kkmxc1rni4pvwijhzg-libxft-2.3.3' -> '/gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3'...
> successfully built /gnu/store/ajkyhj4qsbzw7kqc4wp9yfzci7r5xgyy-libxft-2.3.3.drv
> /gnu/store/mwfb8a9mkcn65nfnsnvj39xly9d8qa90-libxft-2.3.3
> #+end_example

This seems good, no?

I guess we need tests for the repair functionality (repairing itself is
currently untested for the reasons explained in tests/store.scm:901, but
we could arrange to test it somehow.)

Thanks,
Ludo’.

Re: guix build --repair might not always work

[zimoun]

Hi,

On Sat, 25 Jan 2020 at 19:06, Ludovic Courtès <ludo@gnu.org> wrote:

> > And nothing else.  So, let's check the hash
>
> In particular, you should compare the actual hash returned by “guix hash
> -r” with that stored in the local database.  If they match, then the
> store item is not considered corrupt.

It is not possible to do that at the CLI level, right?
And how do you do that at the Scheme level?

All the best,
simon