unofficial mirror of help-guix@gnu.org 
 help / color / mirror / Atom feed
* Corrupt/missing store items
@ 2021-06-21 18:30 Jack Hill
  2021-06-21 20:28 ` Jack Hill
  0 siblings, 1 reply; 7+ messages in thread
From: Jack Hill @ 2021-06-21 18:30 UTC (permalink / raw)
  To: help-guix

Hi Guix,

I could use some help figuring out what's happened to my store and how to 
recover.

I first noticed the problem when trying to check if a package build 
reproducably. Unfortunately, I did not capture the output from that, but 
did capture subsequent commands. It seems as if the metadata has gotten 
out of sync with the store:

jackhill@kalessin ~$ guix gc --verify
reading the store...
checking path existence...
path `/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16' disappeared, but it still has valid referrers!

jackhill@kalessin ~$ guix build /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
The following graft will be made:
    /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
applying 1 graft for /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv ...
grafting '/gnu/store/199npi1hcv7zn0r19vl29np6ccshii4p-gnutls-3.6.16-debug' -> '/gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug'...
grafting '/gnu/store/8ixa3p7hwb26warjinffcrvzl064wbcg-gnutls-3.6.16-doc' -> '/gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc'...
grafting '/gnu/store/akc7l65z459pnifrr6bcm97cjvmpvp9k-gnutls-3.6.16' -> '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16'...
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/certtool'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/gnutls-cli'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/gnutls-cli-debug'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/gnutls-serv'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/ocsptool'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/psktool'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/srptool'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/guile/3.0/extensions/guile-gnutls-v-2.so.0.0.0'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/guile/3.0/site-ccache/gnutls.go'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/guile/3.0/site-ccache/gnutls/extra.go'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/libgnutls.so.30.28.2'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/libgnutlsxx.so.28.1.0'
successfully built /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
/gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
/gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
jackhill@kalessin ~$ guix gc --verify                                                      reading the store...
checking path existence...
path `/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16' disappeared, but it still has valid referrers!
jackhill@kalessin ~$ ls -l /gnu/store/^C
^C^C
jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory

Then, I ran `guix gc` to collect all the garbage. Trying to rebuild the gnutls derivation results in:

jackhill@kalessin ~$ guix build /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
18.6 MB will be downloaded:
    /gnu/store/2wrp7x9aclqsapm58dz5i654qds8nbb8-guile-2.0.14
    /gnu/store/mz5fvdfks10rmwxf29n95bp9bim6wq7g-nettle-3.5.1
    /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31
    /gnu/store/s3dcqzwqaakv1yx37by9chksdbkgih17-glibc-2.31-static
    /gnu/store/199npi1hcv7zn0r19vl29np6ccshii4p-gnutls-3.6.16-debug
    /gnu/store/8ixa3p7hwb26warjinffcrvzl064wbcg-gnutls-3.6.16-doc
    /gnu/store/akc7l65z459pnifrr6bcm97cjvmpvp9k-gnutls-3.6.16
The following graft will be made:
    /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
substituting /gnu/store/2wrp7x9aclqsapm58dz5i654qds8nbb8-guile-2.0.14...
Backtrace:
In ice-9/boot-9.scm:
   1752:10 19 (with-exception-handler _ _ #:unwind? _ # _)
   1752:10 18 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
    771:15 17 (_)
In ice-9/exceptions.scm:
    406:15 16 (process-substitution #<output: file 4> _ "/gnu/store/…" …)
In ice-9/boot-9.scm:
   1752:10 15 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
     479:9 14 (_)
In ice-9/boot-9.scm:
   1747:15 13 (with-exception-handler #<procedure 7fb7171635a0 at ic…> …)
In guix/scripts/substitute.scm:
    408:14 12 (_)
    382:28 11 (_ #<<uri> scheme: https userinfo: #f host: "ci.guix.g…> …)
In guix/build/download.scm:
     453:4 10 (open-connection-for-uri _ #:timeout _ # _)
     251:2  9 (tls-wrap #<input-output: socket 14> "ci.guix.gnu.org" # …)
    164:15  8 (load-gnutls)
In ice-9/boot-9.scm:
    3329:6  7 (resolve-interface (gnutls) #:select _ #:hide _ #:prefix …)
   1685:16  6 (raise-exception _ #:continuable? _)
   1683:16  5 (raise-exception _ #:continuable? _)
   1685:16  4 (raise-exception _ #:continuable? _)
   1780:13  3 (_ #<&compound-exception components: (#<&error> #<&orig…>)
   1685:16  2 (raise-exception _ #:continuable? _)
   1683:16  1 (raise-exception _ #:continuable? _)
   1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
no code for module (gnutls)
substitution of /gnu/store/2wrp7x9aclqsapm58dz5i654qds8nbb8-guile-2.0.14 failed
guix build: error: corrupt input while restoring archive from #<closed: file 7f6110e16150>

Both /gnu and /var/guix are on the same btrfs filesystem. I ran a btrfs 
scrub, and that did not turn up any errors. The other "interesting" thing 
about this system, is that it once had deduplication turned off, but that 
has since been turned back on because of bug 47115 [0].

[0] https://issues.guix.gnu.org/47115

Best,
Jack

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Corrupt/missing store items
  2021-06-21 18:30 Corrupt/missing store items Jack Hill
@ 2021-06-21 20:28 ` Jack Hill
  2021-06-22  8:20   ` Christopher Baines
  2021-06-22 18:46   ` Ricardo Wurmus
  0 siblings, 2 replies; 7+ messages in thread
From: Jack Hill @ 2021-06-21 20:28 UTC (permalink / raw)
  To: help-guix

Given what i know about the problem: substitute operations are failing 
because gnutls is not available, I tried:

jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
/gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
/gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory

So Guix think's it's sucessfully built gnutls, but it's still not present 
in the store. I've also tried copying the store items from other hosts. No 
luck so far…

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Corrupt/missing store items
  2021-06-21 20:28 ` Jack Hill
@ 2021-06-22  8:20   ` Christopher Baines
  2021-06-22 13:21     ` Jack Hill
  2021-06-22 18:46   ` Ricardo Wurmus
  1 sibling, 1 reply; 7+ messages in thread
From: Christopher Baines @ 2021-06-22  8:20 UTC (permalink / raw)
  To: Jack Hill; +Cc: help-guix

[-- Attachment #1: Type: text/plain, Size: 917 bytes --]


Jack Hill <jackhill@jackhill.us> writes:

> Given what i know about the problem: substitute operations are failing
> because gnutls is not available, I tried:
>
> jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
> /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
> /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
> jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
> ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory
>
> So Guix think's it's sucessfully built gnutls, but it's still not
> present in the store. I've also tried copying the store items from
> other hosts. No luck so far…

Does guix gc --verify=repair say anything?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Corrupt/missing store items
  2021-06-22  8:20   ` Christopher Baines
@ 2021-06-22 13:21     ` Jack Hill
  2021-06-22 13:45       ` Jack Hill
  0 siblings, 1 reply; 7+ messages in thread
From: Jack Hill @ 2021-06-22 13:21 UTC (permalink / raw)
  To: Christopher Baines; +Cc: help-guix

On Tue, 22 Jun 2021, Christopher Baines wrote:

>
> Jack Hill <jackhill@jackhill.us> writes:
>
>> Given what i know about the problem: substitute operations are failing
>> because gnutls is not available, I tried:
>>
>> jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
>> /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
>> /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
>> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
>> jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
>> ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory
>>
>> So Guix think's it's sucessfully built gnutls, but it's still not
>> present in the store. I've also tried copying the store items from
>> other hosts. No luck so far…
>
> Does guix gc --verify=repair say anything?

It fails, becasue without gnutls my substitute servers can't be contacted. 
I also can't reconfigure with http:// substitute servers. Unfortunatley, 
gc doesn't accept a substitute-urls option.

jackhill@kalessin ~$ sudo guix gc --verify=repair
reading the store...
checking path existence...
path `/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16' disappeared, but it still has valid referrers!
substitute: updating substitutes from 'https://ci.guix.gnu.org'...   0.0%Backtrace:
substitute: In ice-9/boot-9.scm:
substitute:   1752:10 16 (with-exception-handler _ _ #:unwind? _ # _)
substitute: In unknown file:
substitute:           15 (apply-smob/0 #<thunk 7fd90657af60>)
substitute: In ice-9/boot-9.scm:
substitute:     724:2 14 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
substitute: In ice-9/eval.scm:
substitute:     619:8 13 (_ #(#(#<directory (guile-user) 7fd906574c80>)))
substitute: In guix/ui.scm:
substitute:   2147:12 12 (run-guix-command _ . _)
substitute: In ice-9/boot-9.scm:
substitute:   1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
substitute:   1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
substitute: In guix/scripts/substitute.scm:
substitute:    757:18  9 (_)
substitute:    348:26  8 (process-query #<output: file 4> _ #:cache-urls _ #:acl _)
substitute: In guix/substitutes.scm:
substitute:    358:27  7 (lookup-narinfos/diverse _ _ #<procedure 7fd90402eba0 …> …)
substitute:    315:31  6 (lookup-narinfos "https://ci.guix.gnu.org" _ # _ # _)
substitute:    238:26  5 (fetch-narinfos _ _ #:open-connection _ # _)
substitute: In ice-9/boot-9.scm:
substitute:   1685:16  4 (raise-exception _ #:continuable? _)
substitute:   1685:16  3 (raise-exception _ #:continuable? _)
substitute:   1780:13  2 (_ #<&compound-exception components: (#<&error> #<&orig…>)
substitute:   1685:16  1 (raise-exception _ #:continuable? _)
substitute:   1685:16  0 (raise-exception _ #:continuable? _)
substitute:
substitute: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
substitute: no code for module (gnutls)
warning: `/gnu/store/3qbmgxrgml2bbi2hzg61nq0amyvg9sgq-guix-1.3.0-4.4985a42/bin/guix substitute' died unexpectedly

Thanks!
Jack

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Corrupt/missing store items
  2021-06-22 13:21     ` Jack Hill
@ 2021-06-22 13:45       ` Jack Hill
  0 siblings, 0 replies; 7+ messages in thread
From: Jack Hill @ 2021-06-22 13:45 UTC (permalink / raw)
  To: Christopher Baines; +Cc: help-guix

On Tue, 22 Jun 2021, Jack Hill wrote:

> On Tue, 22 Jun 2021, Christopher Baines wrote:
>
>> 
>> Jack Hill <jackhill@jackhill.us> writes:
>> 
>>> Given what i know about the problem: substitute operations are failing
>>> because gnutls is not available, I tried:
>>> 
>>> jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org 
>>> /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
>>> /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
>>> /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
>>> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
>>> jackhill@kalessin ~$ ls -l 
>>> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
>>> ls: cannot access 
>>> '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file 
>>> or directory
>>> 
>>> So Guix think's it's sucessfully built gnutls, but it's still not
>>> present in the store. I've also tried copying the store items from
>>> other hosts. No luck so far…
>> 
>> Does guix gc --verify=repair say anything?
>
> It fails, becasue without gnutls my substitute servers can't be contacted. I 
> also can't reconfigure with http:// substitute servers. Unfortunatley, gc 
> doesn't accept a substitute-urls option.

Ok, I started the daemon by hand with http-only substitute servers, and 
--verify=repair appears to have solved my problem as far as I can tell so 
far.

Thanks to you and nckx!
Jack

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Corrupt/missing store items
  2021-06-21 20:28 ` Jack Hill
  2021-06-22  8:20   ` Christopher Baines
@ 2021-06-22 18:46   ` Ricardo Wurmus
  2021-06-22 19:29     ` Jack Hill
  1 sibling, 1 reply; 7+ messages in thread
From: Ricardo Wurmus @ 2021-06-22 18:46 UTC (permalink / raw)
  To: Jack Hill; +Cc: help-guix


Jack Hill <jackhill@jackhill.us> writes:

> Given what i know about the problem: substitute operations are 
> failing
> because gnutls is not available, I tried:
>
> jackhill@kalessin ~$ guix build 
> --substitute-urls=http://ci.guix.gnu.org 
> /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
> /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
> /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
> jackhill@kalessin ~$ ls -l 
> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
> ls: cannot access 
> '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No 
> such file or directory

What happens if you try “guix build --check --no-grafts 
/gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv” or 
similar?  The database probably states that 
/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 has 
already been built, so with “--check” you might be able to make it 
build it again.

> So Guix think's it's sucessfully built gnutls, but it's still 
> not
> present in the store. I've also tried copying the store items 
> from
> other hosts. No luck so far…

How did that fail?

How could the directory have disappeared?  Guix appears to be very 
surprised that the directory is missing.

-- 
Ricardo


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Corrupt/missing store items
  2021-06-22 18:46   ` Ricardo Wurmus
@ 2021-06-22 19:29     ` Jack Hill
  0 siblings, 0 replies; 7+ messages in thread
From: Jack Hill @ 2021-06-22 19:29 UTC (permalink / raw)
  To: Ricardo Wurmus; +Cc: help-guix

On Tue, 22 Jun 2021, Ricardo Wurmus wrote:

> Jack Hill <jackhill@jackhill.us> writes:
>
> What happens if you try “guix build --check --no-grafts 
> /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv” or similar? 
> The database probably states that 
> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 has already been 
> built, so with “--check” you might be able to make it build it again.

`guix gc --verify=repair` fixed the problem, so I won't be able to test 
this (unless this recurs, but I hope it doesn't).

>> So Guix think's it's sucessfully built gnutls, but it's still not
>> present in the store. I've also tried copying the store items from
>> other hosts. No luck so far…
>
> How did that fail?

I didn't capture verbose output, but I believe nothing was sent because 
the problematic host reported it already had that item.

> How could the directory have disappeared?  Guix appears to be very surprised 
> that the directory is missing.

Apologies for leaving out more details about this host earlier. The 
filesystem is btrfs and its a virtual machine running in someone else's VM 
cluster. I had to reset and boot into older generations a few times before 
I noticed the problem because of a misconfiguration I made. Some of these 
older generation involved downgrading the kernel from 5.12 to 5.11. I 
suppose that could have caused file system corruption, although I would 
not have expected the store to be written to at the time of the resets.

Best,
Jack

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2021-06-22 19:30 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-21 18:30 Corrupt/missing store items Jack Hill
2021-06-21 20:28 ` Jack Hill
2021-06-22  8:20   ` Christopher Baines
2021-06-22 13:21     ` Jack Hill
2021-06-22 13:45       ` Jack Hill
2021-06-22 18:46   ` Ricardo Wurmus
2021-06-22 19:29     ` Jack Hill

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).