* Corrupt/missing store items
@ 2021-06-21 18:30 Jack Hill
2021-06-21 20:28 ` Jack Hill
0 siblings, 1 reply; 7+ messages in thread
From: Jack Hill @ 2021-06-21 18:30 UTC (permalink / raw)
To: help-guix
Hi Guix,
I could use some help figuring out what's happened to my store and how to
recover.
I first noticed the problem when trying to check if a package build
reproducably. Unfortunately, I did not capture the output from that, but
did capture subsequent commands. It seems as if the metadata has gotten
out of sync with the store:
jackhill@kalessin ~$ guix gc --verify
reading the store...
checking path existence...
path `/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16' disappeared, but it still has valid referrers!
jackhill@kalessin ~$ guix build /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
The following graft will be made:
/gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
applying 1 graft for /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv ...
grafting '/gnu/store/199npi1hcv7zn0r19vl29np6ccshii4p-gnutls-3.6.16-debug' -> '/gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug'...
grafting '/gnu/store/8ixa3p7hwb26warjinffcrvzl064wbcg-gnutls-3.6.16-doc' -> '/gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc'...
grafting '/gnu/store/akc7l65z459pnifrr6bcm97cjvmpvp9k-gnutls-3.6.16' -> '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16'...
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/certtool'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/gnutls-cli'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/gnutls-cli-debug'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/gnutls-serv'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/ocsptool'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/psktool'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/bin/srptool'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/guile/3.0/extensions/guile-gnutls-v-2.so.0.0.0'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/guile/3.0/site-ccache/gnutls.go'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/guile/3.0/site-ccache/gnutls/extra.go'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/libgnutls.so.30.28.2'
updating '.gnu_debuglink' CRC in '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16/lib/libgnutlsxx.so.28.1.0'
successfully built /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
/gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
/gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
jackhill@kalessin ~$ guix gc --verify reading the store...
checking path existence...
path `/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16' disappeared, but it still has valid referrers!
jackhill@kalessin ~$ ls -l /gnu/store/^C
^C^C
jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory
Then, I ran `guix gc` to collect all the garbage. Trying to rebuild the gnutls derivation results in:
jackhill@kalessin ~$ guix build /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
18.6 MB will be downloaded:
/gnu/store/2wrp7x9aclqsapm58dz5i654qds8nbb8-guile-2.0.14
/gnu/store/mz5fvdfks10rmwxf29n95bp9bim6wq7g-nettle-3.5.1
/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31
/gnu/store/s3dcqzwqaakv1yx37by9chksdbkgih17-glibc-2.31-static
/gnu/store/199npi1hcv7zn0r19vl29np6ccshii4p-gnutls-3.6.16-debug
/gnu/store/8ixa3p7hwb26warjinffcrvzl064wbcg-gnutls-3.6.16-doc
/gnu/store/akc7l65z459pnifrr6bcm97cjvmpvp9k-gnutls-3.6.16
The following graft will be made:
/gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
substituting /gnu/store/2wrp7x9aclqsapm58dz5i654qds8nbb8-guile-2.0.14...
Backtrace:
In ice-9/boot-9.scm:
1752:10 19 (with-exception-handler _ _ #:unwind? _ # _)
1752:10 18 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
771:15 17 (_)
In ice-9/exceptions.scm:
406:15 16 (process-substitution #<output: file 4> _ "/gnu/store/…" …)
In ice-9/boot-9.scm:
1752:10 15 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
479:9 14 (_)
In ice-9/boot-9.scm:
1747:15 13 (with-exception-handler #<procedure 7fb7171635a0 at ic…> …)
In guix/scripts/substitute.scm:
408:14 12 (_)
382:28 11 (_ #<<uri> scheme: https userinfo: #f host: "ci.guix.g…> …)
In guix/build/download.scm:
453:4 10 (open-connection-for-uri _ #:timeout _ # _)
251:2 9 (tls-wrap #<input-output: socket 14> "ci.guix.gnu.org" # …)
164:15 8 (load-gnutls)
In ice-9/boot-9.scm:
3329:6 7 (resolve-interface (gnutls) #:select _ #:hide _ #:prefix …)
1685:16 6 (raise-exception _ #:continuable? _)
1683:16 5 (raise-exception _ #:continuable? _)
1685:16 4 (raise-exception _ #:continuable? _)
1780:13 3 (_ #<&compound-exception components: (#<&error> #<&orig…>)
1685:16 2 (raise-exception _ #:continuable? _)
1683:16 1 (raise-exception _ #:continuable? _)
1685:16 0 (raise-exception _ #:continuable? _)
ice-9/boot-9.scm:1685:16: In procedure raise-exception:
no code for module (gnutls)
substitution of /gnu/store/2wrp7x9aclqsapm58dz5i654qds8nbb8-guile-2.0.14 failed
guix build: error: corrupt input while restoring archive from #<closed: file 7f6110e16150>
Both /gnu and /var/guix are on the same btrfs filesystem. I ran a btrfs
scrub, and that did not turn up any errors. The other "interesting" thing
about this system, is that it once had deduplication turned off, but that
has since been turned back on because of bug 47115 [0].
[0] https://issues.guix.gnu.org/47115
Best,
Jack
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: Corrupt/missing store items
2021-06-21 18:30 Corrupt/missing store items Jack Hill
@ 2021-06-21 20:28 ` Jack Hill
2021-06-22 8:20 ` Christopher Baines
2021-06-22 18:46 ` Ricardo Wurmus
0 siblings, 2 replies; 7+ messages in thread
From: Jack Hill @ 2021-06-21 20:28 UTC (permalink / raw)
To: help-guix
Given what i know about the problem: substitute operations are failing
because gnutls is not available, I tried:
jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
/gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
/gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory
So Guix think's it's sucessfully built gnutls, but it's still not present
in the store. I've also tried copying the store items from other hosts. No
luck so far…
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Corrupt/missing store items
2021-06-21 20:28 ` Jack Hill
@ 2021-06-22 8:20 ` Christopher Baines
2021-06-22 13:21 ` Jack Hill
2021-06-22 18:46 ` Ricardo Wurmus
1 sibling, 1 reply; 7+ messages in thread
From: Christopher Baines @ 2021-06-22 8:20 UTC (permalink / raw)
To: Jack Hill; +Cc: help-guix
[-- Attachment #1: Type: text/plain, Size: 917 bytes --]
Jack Hill <jackhill@jackhill.us> writes:
> Given what i know about the problem: substitute operations are failing
> because gnutls is not available, I tried:
>
> jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
> /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
> /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
> jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
> ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory
>
> So Guix think's it's sucessfully built gnutls, but it's still not
> present in the store. I've also tried copying the store items from
> other hosts. No luck so far…
Does guix gc --verify=repair say anything?
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Corrupt/missing store items
2021-06-22 8:20 ` Christopher Baines
@ 2021-06-22 13:21 ` Jack Hill
2021-06-22 13:45 ` Jack Hill
0 siblings, 1 reply; 7+ messages in thread
From: Jack Hill @ 2021-06-22 13:21 UTC (permalink / raw)
To: Christopher Baines; +Cc: help-guix
On Tue, 22 Jun 2021, Christopher Baines wrote:
>
> Jack Hill <jackhill@jackhill.us> writes:
>
>> Given what i know about the problem: substitute operations are failing
>> because gnutls is not available, I tried:
>>
>> jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
>> /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
>> /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
>> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
>> jackhill@kalessin ~$ ls -l /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
>> ls: cannot access '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file or directory
>>
>> So Guix think's it's sucessfully built gnutls, but it's still not
>> present in the store. I've also tried copying the store items from
>> other hosts. No luck so far…
>
> Does guix gc --verify=repair say anything?
It fails, becasue without gnutls my substitute servers can't be contacted.
I also can't reconfigure with http:// substitute servers. Unfortunatley,
gc doesn't accept a substitute-urls option.
jackhill@kalessin ~$ sudo guix gc --verify=repair
reading the store...
checking path existence...
path `/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16' disappeared, but it still has valid referrers!
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 0.0%Backtrace:
substitute: In ice-9/boot-9.scm:
substitute: 1752:10 16 (with-exception-handler _ _ #:unwind? _ # _)
substitute: In unknown file:
substitute: 15 (apply-smob/0 #<thunk 7fd90657af60>)
substitute: In ice-9/boot-9.scm:
substitute: 724:2 14 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
substitute: In ice-9/eval.scm:
substitute: 619:8 13 (_ #(#(#<directory (guile-user) 7fd906574c80>)))
substitute: In guix/ui.scm:
substitute: 2147:12 12 (run-guix-command _ . _)
substitute: In ice-9/boot-9.scm:
substitute: 1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
substitute: 1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
substitute: In guix/scripts/substitute.scm:
substitute: 757:18 9 (_)
substitute: 348:26 8 (process-query #<output: file 4> _ #:cache-urls _ #:acl _)
substitute: In guix/substitutes.scm:
substitute: 358:27 7 (lookup-narinfos/diverse _ _ #<procedure 7fd90402eba0 …> …)
substitute: 315:31 6 (lookup-narinfos "https://ci.guix.gnu.org" _ # _ # _)
substitute: 238:26 5 (fetch-narinfos _ _ #:open-connection _ # _)
substitute: In ice-9/boot-9.scm:
substitute: 1685:16 4 (raise-exception _ #:continuable? _)
substitute: 1685:16 3 (raise-exception _ #:continuable? _)
substitute: 1780:13 2 (_ #<&compound-exception components: (#<&error> #<&orig…>)
substitute: 1685:16 1 (raise-exception _ #:continuable? _)
substitute: 1685:16 0 (raise-exception _ #:continuable? _)
substitute:
substitute: ice-9/boot-9.scm:1685:16: In procedure raise-exception:
substitute: no code for module (gnutls)
warning: `/gnu/store/3qbmgxrgml2bbi2hzg61nq0amyvg9sgq-guix-1.3.0-4.4985a42/bin/guix substitute' died unexpectedly
Thanks!
Jack
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Corrupt/missing store items
2021-06-22 13:21 ` Jack Hill
@ 2021-06-22 13:45 ` Jack Hill
0 siblings, 0 replies; 7+ messages in thread
From: Jack Hill @ 2021-06-22 13:45 UTC (permalink / raw)
To: Christopher Baines; +Cc: help-guix
On Tue, 22 Jun 2021, Jack Hill wrote:
> On Tue, 22 Jun 2021, Christopher Baines wrote:
>
>>
>> Jack Hill <jackhill@jackhill.us> writes:
>>
>>> Given what i know about the problem: substitute operations are failing
>>> because gnutls is not available, I tried:
>>>
>>> jackhill@kalessin ~$ guix build --substitute-urls=http://ci.guix.gnu.org
>>> /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
>>> /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
>>> /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
>>> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
>>> jackhill@kalessin ~$ ls -l
>>> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
>>> ls: cannot access
>>> '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No such file
>>> or directory
>>>
>>> So Guix think's it's sucessfully built gnutls, but it's still not
>>> present in the store. I've also tried copying the store items from
>>> other hosts. No luck so far…
>>
>> Does guix gc --verify=repair say anything?
>
> It fails, becasue without gnutls my substitute servers can't be contacted. I
> also can't reconfigure with http:// substitute servers. Unfortunatley, gc
> doesn't accept a substitute-urls option.
Ok, I started the daemon by hand with http-only substitute servers, and
--verify=repair appears to have solved my problem as far as I can tell so
far.
Thanks to you and nckx!
Jack
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Corrupt/missing store items
2021-06-21 20:28 ` Jack Hill
2021-06-22 8:20 ` Christopher Baines
@ 2021-06-22 18:46 ` Ricardo Wurmus
2021-06-22 19:29 ` Jack Hill
1 sibling, 1 reply; 7+ messages in thread
From: Ricardo Wurmus @ 2021-06-22 18:46 UTC (permalink / raw)
To: Jack Hill; +Cc: help-guix
Jack Hill <jackhill@jackhill.us> writes:
> Given what i know about the problem: substitute operations are
> failing
> because gnutls is not available, I tried:
>
> jackhill@kalessin ~$ guix build
> --substitute-urls=http://ci.guix.gnu.org
> /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv
> /gnu/store/2vjql2vd6srqxhr0r9xnhclqgc5kxlh1-gnutls-3.6.16-debug
> /gnu/store/ads39f4czri8s1k43qd0qpxi6dr3w0zn-gnutls-3.6.16-doc
> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
> jackhill@kalessin ~$ ls -l
> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16
> ls: cannot access
> '/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16': No
> such file or directory
What happens if you try “guix build --check --no-grafts
/gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv” or
similar? The database probably states that
/gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 has
already been built, so with “--check” you might be able to make it
build it again.
> So Guix think's it's sucessfully built gnutls, but it's still
> not
> present in the store. I've also tried copying the store items
> from
> other hosts. No luck so far…
How did that fail?
How could the directory have disappeared? Guix appears to be very
surprised that the directory is missing.
--
Ricardo
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Corrupt/missing store items
2021-06-22 18:46 ` Ricardo Wurmus
@ 2021-06-22 19:29 ` Jack Hill
0 siblings, 0 replies; 7+ messages in thread
From: Jack Hill @ 2021-06-22 19:29 UTC (permalink / raw)
To: Ricardo Wurmus; +Cc: help-guix
On Tue, 22 Jun 2021, Ricardo Wurmus wrote:
> Jack Hill <jackhill@jackhill.us> writes:
>
> What happens if you try “guix build --check --no-grafts
> /gnu/store/zkhymfsbrv0s4y7l778g78k6y65nidxd-gnutls-3.6.16.drv” or similar?
> The database probably states that
> /gnu/store/jlk67v3nddhv0z963wfvahk8fc8gqcz8-gnutls-3.6.16 has already been
> built, so with “--check” you might be able to make it build it again.
`guix gc --verify=repair` fixed the problem, so I won't be able to test
this (unless this recurs, but I hope it doesn't).
>> So Guix think's it's sucessfully built gnutls, but it's still not
>> present in the store. I've also tried copying the store items from
>> other hosts. No luck so far…
>
> How did that fail?
I didn't capture verbose output, but I believe nothing was sent because
the problematic host reported it already had that item.
> How could the directory have disappeared? Guix appears to be very surprised
> that the directory is missing.
Apologies for leaving out more details about this host earlier. The
filesystem is btrfs and its a virtual machine running in someone else's VM
cluster. I had to reset and boot into older generations a few times before
I noticed the problem because of a misconfiguration I made. Some of these
older generation involved downgrading the kernel from 5.12 to 5.11. I
suppose that could have caused file system corruption, although I would
not have expected the store to be written to at the time of the resets.
Best,
Jack
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-06-22 19:30 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-21 18:30 Corrupt/missing store items Jack Hill
2021-06-21 20:28 ` Jack Hill
2021-06-22 8:20 ` Christopher Baines
2021-06-22 13:21 ` Jack Hill
2021-06-22 13:45 ` Jack Hill
2021-06-22 18:46 ` Ricardo Wurmus
2021-06-22 19:29 ` Jack Hill
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).