bug#57838: failing to boot, probably due to guix gc

bug#57838: failing to boot, probably due to guix gc

[Attila Lendvai]

[-- Attachment #1: Type: text/plain, Size: 1194 bytes --]

dear Guixers,

on one of my installs i ran the following two commands as root:

guix gc --delete-generations=60d
guix system delete-generations 60d

i think i ran a reboot pretty soon after this, and the machine is failing to boot with the error "no code for module (ice-9 popen)".

i'm attaching a photo of the kernel panic screen.

the delete-generations left 3 generations, but all the three fail the same way.

i'll keep the machine as-is for a while if some inspection would help by providing logs/data. i also want to fix it and have it running soonish, so let me know if there's anything valuable i should extract from it before attempting to fix it. i hope a chroot and a reconfigure will fix it.

i think i ran the commands multiple times with different time ranges, and in different order.

i did something similar on another install, but IIRC i also ran a `guix system reconfigure` prior to the reboot, and that install didn't break.

HTH,

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Politicians never accuse you of 'greed' for wanting other people's money - only for wanting to keep your own money.”
	— Joseph Sobran (1946–2010)

[-- Attachment #2: kernelpanic.png --]
[-- Type: image/png, Size: 58195 bytes --]

bug#57838: (No Subject)

[Attila Lendvai]

i have fixed it by running a reconfigure in a chroot.

surprisingly, this has also fixed the old, previously broken system generations, not only the newly created one.

maybe some boot related references are not traversed while GC'ing?

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
If you expect the world to be fair with you because you are fair, you're fooling yourself. Stop bitching about the lion, and get busy learning how not to get eaten.

bug#57838: (No Subject)

[Josselin Poiret via Bug reports for GNU Guix]

Hi Attila,
Attila Lendvai <attila@lendvai.name> writes:

> i have fixed it by running a reconfigure in a chroot.
>
> surprisingly, this has also fixed the old, previously broken system generations, not only the newly created one.
>
> maybe some boot related references are not traversed while GC'ing?

I just checked a couple of things on my own system generation:
* the boot script execl's shepherd, which specifies the (full) guile
  interpreter as a full path in the shebang line;
* that guile interpreter does contain the (ice-9 popen) module;
* that same guile interpreter does appear as a requisite of the system
  generation, so shouldn't be gc'd as long as it's a gc root.

I've got no clue :(

-- 
Josselin Poiret

bug#57838: failing to boot, probably due to guix gc

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 788 bytes --]

For these kind of errors, I think I've an idea what's the cause in 
<https://issues.guix.gnu.org/58035>.


On 15-09-2022 21:44, Attila Lendvai wrote:
> dear Guixers,
> 
> on one of my installs i ran the following two commands as root:
> 
> guix gc --delete-generations=60d
> guix system delete-generations 60d
> 
> i think i ran a reboot pretty soon after this, and the machine is failing to boot with the error "no code for module (ice-9 popen)".

How did you reboot?  Maybe whatever rebooting mechanism you use doesn't 
do 'sync' first or doesn't wait for 'sync' to complete.

To test the hypothesis that there is store corruption, could you do 
"guix gc --verify=contents" (assuming there are some old system 
generations you can boot from)?

Greetings,
Maxime.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

bug#57838: failing to boot, probably due to guix gc

[Attila Lendvai]

> > on one of my installs i ran the following two commands as root:
> > 
> > guix gc --delete-generations=60d
> > guix system delete-generations 60d
> > 
> > i think i ran a reboot pretty soon after this, and the machine is failing to boot with the error "no code for module (ice-9 popen)".
> 
> 
> How did you reboot? Maybe whatever rebooting mechanism you use doesn't
> do 'sync' first or doesn't wait for 'sync' to complete.


i'm pretty sure i have issued a `reboot` in an ssh session.


> To test the hypothesis that there is store corruption, could you do
> "guix gc --verify=contents" (assuming there are some old system
> generations you can boot from)?


as i have explained in an earlier mail, all my old generations (3 remained after the GC) got broken, but they all got repaired by a subsequent reconfigure from a chroot. it has probably installed some files that were missing.

# guix gc --verify=contents
reading the store...
checking path existence...
checking hashes...
#

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Self knowledge is always bad news.”
	— John Barth (1930–)

bug#57838:

[Attila Lendvai]

close 57838
--

i just found out that this is a bug in that codebase that we don't
talk about here.

sorry about the noise!

https://gitlab.com/nonguix/nonguix/-/issues/111

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
In the end, we only regret the chances we didn’t take, relationships
we were afraid to have, and the decisions we waited too long to make.