bug#44735: gilbc of the running system got wiped while building a package, system broken

[Stefan <stefan-guix@vodafonemail.de>]

Hi!

After trying to just build a package (with a modified guix, but this is certainly unrelated), the system broke catastrophically:

stefan@guix ~/development/guix$ sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild &
stefan@guix ~/development/guix$ /home/stefan/development/guix/pre-inst-env guix build -L /home/stefan/guix u-boot-rpi-3
accepted connection from pid 23848, user stefan
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
12,5 MB will be downloaded:
   /gnu/store/ldg4jqfan2vp01lm255zz7zrb4vllixp-libxau-1.0.9
   /gnu/store/m1r4jwmc56q44x31xcnvg1hcijf0lq88-libxcb-1.14
   /gnu/store/8b75zmsyxc5qghfrxhyqi6g23bq993b1-libbsd-0.10.0
   /gnu/store/z18hwxwgk551y4a0f6j1dxhmp208i4ha-bash-static-5.0.16
   /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31
   /gnu/store/z7hanmdmdalqh1v0y7z8ilinfhyfh91d-glibc-2.31-static
   /gnu/store/pcsl88vd66k62sk1g4wcc9i985xn369m-libxdmcp-1.1.3
   /gnu/store/785ldh00ix897pamyg5p6fpjls6ddwzz-libx11-1.6.A-doc
   /gnu/store/x10mk7ri4ny013km57d3h5093270r7pg-libx11-1.6.A
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substituting /gnu/store/785ldh00ix897pamyg5p6fpjls6ddwzz-libx11-1.6.A-doc...
downloading from https://ci.guix.gnu.org/nar/lzip/785ldh00ix897pamyg5p6fpjls6ddwzz-libx11-1.6.A-doc ...
 libx11-1.6.A-doc  1.2MiB                                          309KiB/s 00:04 [##################] 100.0%

substituting /gnu/store/z18hwxwgk551y4a0f6j1dxhmp208i4ha-bash-static-5.0.16...
downloading from https://ci.guix.gnu.org/nar/lzip/z18hwxwgk551y4a0f6j1dxhmp208i4ha-bash-static-5.0.16 ...
 bash-static-5.0.16  502KiB                                        976KiB/s 00:01 [##################] 100.0%

guix build: error: cannot unlink `/gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/share/locale/de/LC_MESSAGES': Directory not empty
stefan@guix ~/development/guix$ /home/stefan/development/guix/pre-inst-env guix build -L /home/stefan/guix u-boot-rpi-3
accepted connection from pid 23911, user stefan
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
12,0 MB will be downloaded:
   /gnu/store/ldg4jqfan2vp01lm255zz7zrb4vllixp-libxau-1.0.9
   /gnu/store/m1r4jwmc56q44x31xcnvg1hcijf0lq88-libxcb-1.14
   /gnu/store/8b75zmsyxc5qghfrxhyqi6g23bq993b1-libbsd-0.10.0
   /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31
   /gnu/store/z7hanmdmdalqh1v0y7z8ilinfhyfh91d-glibc-2.31-static
   /gnu/store/pcsl88vd66k62sk1g4wcc9i985xn369m-libxdmcp-1.1.3
   /gnu/store/785ldh00ix897pamyg5p6fpjls6ddwzz-libx11-1.6.A-doc
   /gnu/store/x10mk7ri4ny013km57d3h5093270r7pg-libx11-1.6.A
guix build: error: cannot unlink `/gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib/gconv': Directory not empty
stefan@guix ~/development/guix$ /home/stefan/development/guix/pre-inst-env guix build -L /home/stefan/guix u-boot-rpi-3
-bash: /home/stefan/development/guix/pre-inst-env: /bin/sh: Defekter Interpreter: No such file or directory
stefan@guix ~/development/guix$ ls
-bash: /run/current-system/profile/bin/ls: No such file or directory
stefan@guix ~/development/guix$ echo $PATH
/run/setuid-programs:/home/stefan/.config/guix/current/bin:/home/stefan/.guix-profile/bin:/run/current-system/profile/bin:/run/current-system/profile/sbin
stefan@guix ~/development/guix$ ls
-bash: /run/current-system/profile/bin/ls: No such file or directory


The problem that the unlink was not successful was certainly due to a deleted but still opened file on the NFS share. There may be an intermediate hidden .nfs… file, which get created in such a case (“delete on last close”, “silly rename”), However, the RFC-5661 for NFS demands even if OPEN4_RESULT_PRESERVE_UNLINKED is supported, that the directory entry of an open file must not be removed in this case, thus preventing a directory removal.

Taking a look on the nfs-server, this is now the content of the glibc:

# ls gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib -l
total 12
drwxr-xr-x 2 root root 12288 Nov 18 19:41 gconv
lrwxrwxrwx 7 root root    11 Jan  1  1970 libanl.so -> libanl.so.1

Everything is missing.

Of course I tried to reboot the system, but because of the missing ld-linux-aarch64.so.1 the system is not booting properly.


This leaves some questions:

If the whole system (maybe not the booted/running one, but a reconfigured one) was already using /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31, why has it been downloaded again when building a package? Was this due to grafting?

How can it be that the store is kind of “state“ being modified on the fly?

How does it come that removing files for /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31 happens before the download even started? Could this render a system unusable if a network loss happens at the right point in time?

Booting the previous system generation luckily worked – it must be using a different glibc version. But now what about that broken glibc version? How to repair it? The garbage collector will not remove it, as it is still referenced by the latest system generation. Actually I don’t want to delete that generation. The database certainly believes, that this glibc package is installed correctly. What to do now?

The used guix version used when building the package was f6a42ac946edccc7de5e93ee247487cbec40072b.


Bye

Stefan