Corrupt/missing store items

Corrupt/missing store items

[Jack Hill]

Hi Guix,

I could use some help figuring out what's happened to my store and how to 
recover.

I first noticed the problem when trying to check if a package build 
reproducably. Unfortunately, I did not capture the output from that, but 
did capture subsequent commands. It seems as if the metadata has gotten 
out of sync with the store:

jackhill@kalessin ~$ guix gc --verify
reading the store...
checking path existence...
path `/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16' disappeared, but it still has valid referrers!

jackhill@kalessin ~$ guix build /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
The following graft will be made:
    /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
applying 1 graft for /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv ...
grafting '/gnu/store/199npi1hcv7zn0r19vl29np6ccshii4p-gnutls-3.6.16-debug' -> '/gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug'...
grafting '/gnu/store/8ixa3p7hwb26warjinffcrvzl064wbcg-gnutls-3.6.16-doc' -> '/gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc'...
grafting '/gnu/store/akc7l65z459pnifrr6bcm97cjvmpvp9k-gnutls-3.6.16' -> '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16'...
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/certtool'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/gnutls-cli'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/gnutls-cli-debug'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/gnutls-serv'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/ocsptool'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/psktool'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/srptool'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/guile/3.0/extensions/guile-gnutls-v-2.so.0.0.0'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/guile/3.0/site-ccache/gnutls.go'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/guile/3.0/site-ccache/gnutls/extra.go'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/libgnutls.so.30.28.2'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/libgnutlsxx.so.28.1.0'
successfully built /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
/gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
/gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
jackhill@kalessin ~$ guix gc --verify                                                      reading the store...
checking path existence...
path `/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16' disappeared, but it still has valid referrers!
jackhill@kalessin ~$ ls -l /gnu/store/^C
^C^C
jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory

Then, I ran `guix gc` to collect all the garbage. Trying to rebuild the gnutls derivation results in:

jackhill@kalessin ~$ guix build /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
18.6 MB will be downloaded:
    /gnu/store/2wrp7x9aclqsapm58dz5i654qds8nbb8-guile-2.0.14
    /gnu/store/mz5fvdfks10rmwxf29n95bp9bim6wq7g-nettle-3.5.1
    /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31
    /gnu/store/s3dcqzwqaakv1yx37by9chksdbkgih17-glibc-2.31-static
    /gnu/store/199npi1hcv7zn0r19vl29np6ccshii4p-gnutls-3.6.16-debug
    /gnu/store/8ixa3p7hwb26warjinffcrvzl064wbcg-gnutls-3.6.16-doc
    /gnu/store/akc7l65z459pnifrr6bcm97cjvmpvp9k-gnutls-3.6.16
The following graft will be made:
    /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
substituting /gnu/store/2wrp7x9aclqsapm58dz5i654qds8nbb8-guile-2.0.14...
Backtrace:
In ice-9/boot-9.scm:
   1752:10 19 (with-exception-handler _ _ #:unwind? _ # _)
   1752:10 18 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
    771:15 17 (_)
In ice-9/exceptions.scm:
    406:15 16 (process-substitution #<output: file 4> _ "/gnu/store/…" …)
In ice-9/boot-9.scm:
   1752:10 15 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
     479:9 14 (_)
In ice-9/boot-9.scm:
   1747:15 13 (with-exception-handler #<procedure 7fb7171635a0 at ic…> …)
In guix/scripts/substitute.scm:
    408:14 12 (_)
    382:28 11 (_ #<<uri> scheme: https userinfo: #f host: "ci.guix.g…> …)
In guix/build/download.scm:
     453:4 10 (open-connection-for-uri _ #:timeout _ # _)
     251:2  9 (tls-wrap #<input-output: socket 14> "ci.guix.gnu.org" # …)
    164:15  8 (load-gnutls)
In ice-9/boot-9.scm:
    3329:6  7 (resolve-interface (gnutls) #:select _ #:hide _ #:prefix …)
   1685:16  6 (raise-exception _ #:continuable? _)
   1683:16  5 (raise-exception _ #:continuable? _)
   1685:16  4 (raise-exception _ #:continuable? _)
   1780:13  3 (_ #<&compound-exception components: (#<&error> #<&orig…>)
   1685:16  2 (raise-exception _ #:continuable? _)
   1683:16  1 (raise-exception _ #:continuable? _)
   1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
no code for module (gnutls)
substitution of /gnu/store/2wrp7x9aclqsapm58dz5i654qds8nbb8-guile-2.0.14 failed
guix build: error: corrupt input while restoring archive from #<closed: file 7f6110e16150>

Both /gnu and /var/guix are on the same btrfs filesystem. I ran a btrfs 
scrub, and that did not turn up any errors. The other "interesting" thing 
about this system, is that it once had deduplication turned off, but that 
has since been turned back on because of bug 47115 [0].

[0] https://issues.guix.gnu.org/47115

Best,
Jack

Re: Corrupt/missing store items

[Jack Hill]

Given what i know about the problem: substitute operations are failing 
because gnutls is not available, I tried:

jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
/gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
/gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory

So Guix think's it's sucessfully built gnutls, but it's still not present 
in the store. I've also tried copying the store items from other hosts. No 
luck so far…

Re: Corrupt/missing store items

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 917 bytes --]


Jack Hill <jackhill@jackhill.us> writes:

> Given what i know about the problem: substitute operations are failing
> because gnutls is not available, I tried:
>
> jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
> /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
> /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
> jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
> ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory
>
> So Guix think's it's sucessfully built gnutls, but it's still not
> present in the store. I've also tried copying the store items from
> other hosts. No luck so far…

Does guix gc --verify=repair say anything?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Corrupt/missing store items

[Jack Hill]

On Tue, 22 Jun 2021, Christopher Baines wrote:

>
> Jack Hill <jackhill@jackhill.us> writes:
>
>> Given what i know about the problem: substitute operations are failing
>> because gnutls is not available, I tried:
>>
>> jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
>> /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
>> /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
>> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
>> jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
>> ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory
>>
>> So Guix think's it's sucessfully built gnutls, but it's still not
>> present in the store. I've also tried copying the store items from
>> other hosts. No luck so far…
>
> Does guix gc --verify=repair say anything?

It fails, becasue without gnutls my substitute servers can't be contacted. 
I also can't reconfigure with http:// substitute servers. Unfortunatley, 
gc doesn't accept a substitute-urls option.

jackhill@kalessin ~$ sudo guix gc --verify=repair
reading the store...
checking path existence...
path `/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16' disappeared, but it still has valid referrers!
substitute: updating substitutes from 'https://ci.guix.gnu.org'...   0.0%Backtrace:
substitute: In ice-9/boot-9.scm:
substitute:   1752:10 16 (with-exception-handler _ _ #:unwind? _ # _)
substitute: In unknown file:
substitute:           15 (apply-smob/0 #<thunk 7fd90657af60>)
substitute: In ice-9/boot-9.scm:
substitute:     724:2 14 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
substitute: In ice-9/eval.scm:
substitute:     619:8 13 (_ #(#(#<directory (guile-user) 7fd906574c80>)))
substitute: In guix/ui.scm:
substitute:   2147:12 12 (run-guix-command _ . _)
substitute: In ice-9/boot-9.scm:
substitute:   1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
substitute:   1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
substitute: In guix/scripts/substitute.scm:
substitute:    757:18  9 (_)
substitute:    348:26  8 (process-query #<output: file 4> _ #:cache-urls _ #:acl _)
substitute: In guix/substitutes.scm:
substitute:    358:27  7 (lookup-narinfos/diverse _ _ #<procedure 7fd90402eba0 …> …)
substitute:    315:31  6 (lookup-narinfos "https://ci.guix.gnu.org" _ # _ # _)
substitute:    238:26  5 (fetch-narinfos _ _ #:open-connection _ # _)
substitute: In ice-9/boot-9.scm:
substitute:   1685:16  4 (raise-exception _ #:continuable? _)
substitute:   1685:16  3 (raise-exception _ #:continuable? _)
substitute:   1780:13  2 (_ #<&compound-exception components: (#<&error> #<&orig…>)
substitute:   1685:16  1 (raise-exception _ #:continuable? _)
substitute:   1685:16  0 (raise-exception _ #:continuable? _)
substitute:
substitute: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
substitute: no code for module (gnutls)
warning: `/gnu/store/3qbmgxrgml2bbi2hzg61nq0amyvg9sgq-guix-1.3.0-4.4985a42/bin/guix substitute' died unexpectedly

Thanks!
Jack

Re: Corrupt/missing store items

[Jack Hill]

On Tue, 22 Jun 2021, Jack Hill wrote:

> On Tue, 22 Jun 2021, Christopher Baines wrote:
>
>> 
>> Jack Hill <jackhill@jackhill.us> writes:
>> 
>>> Given what i know about the problem: substitute operations are failing
>>> because gnutls is not available, I tried:
>>> 
>>> jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org 
>>> /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
>>> /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
>>> /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
>>> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
>>> jackhill@kalessin ~$ ls -l 
>>> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
>>> ls: cannot access 
>>> '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file 
>>> or directory
>>> 
>>> So Guix think's it's sucessfully built gnutls, but it's still not
>>> present in the store. I've also tried copying the store items from
>>> other hosts. No luck so far…
>> 
>> Does guix gc --verify=repair say anything?
>
> It fails, becasue without gnutls my substitute servers can't be contacted. I 
> also can't reconfigure with http:// substitute servers. Unfortunatley, gc 
> doesn't accept a substitute-urls option.

Ok, I started the daemon by hand with http-only substitute servers, and 
--verify=repair appears to have solved my problem as far as I can tell so 
far.

Thanks to you and nckx!
Jack

Re: Corrupt/missing store items

[Ricardo Wurmus]

Jack Hill <jackhill@jackhill.us> writes:

> Given what i know about the problem: substitute operations are 
> failing
> because gnutls is not available, I tried:
>
> jackhill@kalessin ~$ guix build 
> --substitute-urls=http://ci.guix.gnu.org 
> /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
> /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
> /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
> jackhill@kalessin ~$ ls -l 
> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
> ls: cannot access 
> '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No 
> such file or directory

What happens if you try “guix build --check --no-grafts 
/gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv” or 
similar?  The database probably states that 
/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 has 
already been built, so with “--check” you might be able to make it 
build it again.

> So Guix think's it's sucessfully built gnutls, but it's still 
> not
> present in the store. I've also tried copying the store items 
> from
> other hosts. No luck so far…

How did that fail?

How could the directory have disappeared?  Guix appears to be very 
surprised that the directory is missing.

-- 
Ricardo

Re: Corrupt/missing store items

[Jack Hill]

On Tue, 22 Jun 2021, Ricardo Wurmus wrote:

> Jack Hill <jackhill@jackhill.us> writes:
>
> What happens if you try “guix build --check --no-grafts 
> /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv” or similar? 
> The database probably states that 
> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 has already been 
> built, so with “--check” you might be able to make it build it again.

`guix gc --verify=repair` fixed the problem, so I won't be able to test 
this (unless this recurs, but I hope it doesn't).

>> So Guix think's it's sucessfully built gnutls, but it's still not
>> present in the store. I've also tried copying the store items from
>> other hosts. No luck so far…
>
> How did that fail?

I didn't capture verbose output, but I believe nothing was sent because 
the problematic host reported it already had that item.

> How could the directory have disappeared?  Guix appears to be very surprised 
> that the directory is missing.

Apologies for leaving out more details about this host earlier. The 
filesystem is btrfs and its a virtual machine running in someone else's VM 
cluster. I had to reset and boot into older generations a few times before 
I noticed the problem because of a misconfiguration I made. Some of these 
older generation involved downgrading the kernel from 5.12 to 5.11. I 
suppose that could have caused file system corruption, although I would 
not have expected the store to be written to at the time of the resets.

Best,
Jack