bug#48462: Corrupt, un-repairable store after power cut during upgrade

bug#48462: Corrupt, un-repairable store after power cut during upgrade

[elaexuotee--- via Bug reports for GNU Guix]

Hey Guix,

Recently, during in the middle of a `guix package -m .guix/manifest.scm`, my
machine had power failure. After rebotting, I simply reran the above and
it completed without error.

However, it appears that some of files in the store items are corrupt:

    $ vim
    vim: error while loading shared libraries: /gnu/store/6z2g9ybl3fkx6bdfg226mz8rpc0g53jx-gtk+-2.24.32/lib/libgdk-x11-2.0.so.0: file too short

    $ file -L /gnu/store/6z2g9ybl3fkx6bdfg226mz8rpc0g53jx-gtk+-2.24.32/lib/libgdk-x11-2.0.so.0
    /gnu/store/6z2g9ybl3fkx6bdfg226mz8rpc0g53jx-gtk+-2.24.32/lib/libgdk-x11-2.0.so.0: empty

As an extra bit of annoyance, the handy verify and repair tools fail to see a
problem:

    $ command -v vim | xargs readlink -f
    /gnu/store/h9q2fik60axj9sai5b66cjk2sfl37ij1-vim-full-8.2.2689/bin/vim

    $ sudo guix build --repair vim-full
    /gnu/store/h9q2fik60axj9sai5b66cjk2sfl37ij1-vim-full-8.2.2689

    $ sudo guix build --repair --no-grafts vim-full
    path `/gnu/store/73p5rzka1cb9dx9fy059lmkf1jg6sv95-gtk+-2.24.32' is corrupted or missing!
    found corrupted or missing path `/gnu/store/73p5rzka1cb9dx9fy059lmkf1jg6sv95-gtk+-2.24.32' in the output closure of `/gnu/store/qyp10sss1ascp70yshs87zdzr1lg71f8-vim-full-8.2.2689.drv'
    substituting /gnu/store/73p5rzka1cb9dx9fy059lmkf1jg6sv95-gtk+-2.24.32...
    downloading from https://mirror.brielmaier.net/nar/lzip/73p5rzka1cb9dx9fy059lmkf1jg6sv95-gtk%2B-2.24.32 ...
     gtk%2B-2.24.32  6.4MiB          152KiB/s 00:43 [##################] 100.0%

    /gnu/store/d21m63am0pxw77883h0y1jmy9iiazvvx-vim-full-8.2.2689

    $ sudo guix gc --verify=repair,contents
    ...

    $ vim
    vim: error while loading shared libraries: /gnu/store/6z2g9ybl3fkx6bdfg226mz8rpc0g53jx-gtk+-2.24.32/lib/libgdk-x11-2.0.so.0: file too short

Even going in with a crude hammer doesn't fix things:

    $ sudo mount -o rw,remount /gnu/store

    $ sudo mv /gnu/store/6z2g9ybl3fkx6bdfg226mz8rpc0g53jx-gtk+-2.24.32 .

    $ sudo guix build --repair vim-full
    path `/gnu/store/6z2g9ybl3fkx6bdfg226mz8rpc0g53jx-gtk+-2.24.32' is corrupted or missing!
    found corrupted or missing path `/gnu/store/6z2g9ybl3fkx6bdfg226mz8rpc0g53jx-gtk+-2.24.32' in the output closure of `/gnu/store/z2q3zna20cn32if3v271x4xavd70ml5z-vim-full-8.2.2689.drv'
    path `/gnu/store/73p5rzka1cb9dx9fy059lmkf1jg6sv95-gtk+-2.24.32' is corrupted or missing!
    substituting /gnu/store/73p5rzka1cb9dx9fy059lmkf1jg6sv95-gtk+-2.24.32...
    downloading from https://mirror.brielmaier.net/nar/lzip/73p5rzka1cb9dx9fy059lmkf1jg6sv95-gtk%2B-2.24.32 ...
     gtk%2B-2.24.32  6.4MiB          134KiB/s 00:49 [##################] 100.0%

    applying 17 grafts for /gnu/store/lm9xqz82q4cwlqs79i0pq77z6cfsx3rw-gtk+-2.24.32.drv ...
    grafting '/gnu/store/73p5rzka1cb9dx9fy059lmkf1jg6sv95-gtk+-2.24.32' -> '/gnu/store/6z2g9ybl3fkx6bdfg226mz8rpc0g53jx-gtk+-2.24.32'...
    successfully built /gnu/store/lm9xqz82q4cwlqs79i0pq77z6cfsx3rw-gtk+-2.24.32.drv
    gnu/store/h9q2fik60axj9sai5b66cjk2sfl37ij1-vim-full-8.2.2689

    $ vim
    vim: error while loading shared libraries: /gnu/store/6z2g9ybl3fkx6bdfg226mz8rpc0g53jx-gtk+-2.24.32/lib/libgdk-x11-2.0.so.0: file too short

It seems like the above graft is going badly:

    $ file -L /gnu/store/73p5rzka1cb9dx9fy059lmkf1jg6sv95-gtk+-2.24.32/lib/libgtk-x11-2.0.so.0
    /gnu/store/73p5rzka1cb9dx9fy059lmkf1jg6sv95-gtk+-2.24.32/lib/libgtk-x11-2.0.so.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), statically linked, not stripped

    $ file -L /gnu/store/6z2g9ybl3fkx6bdfg226mz8rpc0g53jx-gtk+-2.24.32/lib/libgdk-x11-2.0.so.0
    /gnu/store/6z2g9ybl3fkx6bdfg226mz8rpc0g53jx-gtk+-2.24.32/lib/libgdk-x11-2.0.so.0: empty

Some questions:

  1) Am I just missing something obvious?
  2) How could grafting end up producing empty files in the output?
  3) Is the store supposed to be robust against power failures?

bug#48462: Corrupt, un-repairable store after power cut during upgrade

[Liliana Marie Prikler]

Hi Guix,

Am Sonntag, den 16.05.2021, 17:59 +0900 schrieb elaexuotee@wilsonb.com:
> Hey Guix,
> 
> Recently, during in the middle of a `guix package -m
> .guix/manifest.scm`, my
> machine had power failure. After rebotting, I simply reran the above
> and it completed without error.
> 
> However, it appears that some of files in the store items are
> corrupt:
> 
> [...]
> As an extra bit of annoyance, the handy verify and repair tools fail
> to see a problem:
> 
> [...]
> 
> Some questions:
> 
>   1) Am I just missing something obvious?
>   2) How could grafting end up producing empty files in the output?
>   3) Is the store supposed to be robust against power failures?
After dealing with some softer issues related to power outage during
`guix package', I think I'm now able to say that Guix does not process
grafts correctly during repair.

What happened for me, was that certain store files were empty and when
they were later grafted, those empty files were carried over.  Guix,
upon repairing the store, did then not regraft those, but rather kept
them as-is.  I have since fixed said issue by manually deleting the
graft and rebuilding (this was possible, because nothing referred to it
– I'm not sure why `guix gc' didn't pick it up as dead earlier,
though).  Stuff in my profiles seem to also have been rebuilt after
roll-back+gc.

So I think content repair needs to have at least two steps:
1. checking and repairing all "normal" derivations.
2. fixing grafts of packages that were repaired in step 1.

WDYT?