Guix System ext4 index full

Guix System ext4 index full

[Roel Janssen]

Dear Guix,

I'm running Guix System and I'm getting the following messages in
dmesg:
[42457.660237] EXT4-fs warning: 7076 callbacks suppressed
[42457.660240] EXT4-fs warning (device sdd3): ext4_dx_add_entry:2335:
Directory (ino: 19796947) index full, reach max htree level :2
[42457.660242] EXT4-fs warning (device sdd3): ext4_dx_add_entry:2339:
Large directory feature is not enabled on this filesystem
[42457.697840] EXT4-fs warning (device sdd3): ext4_dx_add_entry:2335:
Directory (ino: 19796947) index full, reach max htree level :2
[42457.697842] EXT4-fs warning (device sdd3): ext4_dx_add_entry:2339:
Large directory feature is not enabled on this filesystem
[42457.697930] EXT4-fs warning (device sdd3): ext4_dx_add_entry:2335:
Directory (ino: 19796947) index full, reach max htree level :2
[42457.697931] EXT4-fs warning (device sdd3): ext4_dx_add_entry:2339:
Large directory feature is not enabled on this filesystem
[42457.702879] EXT4-fs warning (device sdd3): ext4_dx_add_entry:2335:
Directory (ino: 19796947) index full, reach max htree level :2
[42457.702880] EXT4-fs warning (device sdd3): ext4_dx_add_entry:2339:
Large directory feature is not enabled on this filesystem
[42457.703994] EXT4-fs warning (device sdd3): ext4_dx_add_entry:2335:
Directory (ino: 19796947) index full, reach max htree level :2
[42457.703995] EXT4-fs warning (device sdd3): ext4_dx_add_entry:2339:
Large directory feature is not enabled on this filesystem

I used the graphical installer to install the Guix System.

Also, I cannot write to the root filesystem because the system thinks
it's full (while df shows it has ~125G free space).

Any help would be appreciated.

Kind regards,
Roel Janssen

Re: Guix System ext4 index full

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 192 bytes --]

Roel,

Roel Janssen 写道：
> Large directory feature is not enabled on this filesystem

Try setting the ‘large_dir’ feature on /dev/sdd3 with tune2fs -O

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Guix System ext4 index full

[Vincent Legoll]

Hello,

On 03/06/2020 22:21, Roel Janssen wrote:
> Also, I cannot write to the root filesystem because the system thinks
> it's full (while df shows it has ~125G free space).

T
I think the filesystem (or directory) is full of inodes.

There are many forms of foulness ;-)

-- 
Vincent Legoll

Re: Guix System ext4 index full

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 759 bytes --]

Vincent Legoll 写道：
> I think the filesystem (or directory) is full of inodes.

No, but it's a similar hard limit, and one that not even ‘df -i’ 
will warn you about.

Ext4's dir_index feature uses hash tables to look up directory 
entries, so that for directories with a very large number of items 
(like /gnu/store!), the kernel doesn't have to do the horribly 
slow equivalent of:

  for i in *; do …; done

Unfortunately, once that hash table fills up, the premier stable 
Linux file system just… gives up and refuses to write any more 
data.  In a very cryptic way.

The large_dir flag ‘increases the limit’ (the man page does not 
say by how much) but it doesn't go away.

Your hash table is full of eels,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Guix System ext4 index full

[Leo Famulari]

On Wed, Jun 03, 2020 at 11:14:23PM +0200, Tobias Geerinckx-Rice wrote:
> Unfortunately, once that hash table fills up, the premier stable Linux file
> system just… gives up and refuses to write any more data.  In a very cryptic
> way.

It's so cryptic... I wonder how often people hit this limit in practice?
Is Guix the only place it happens often? Maybe we can ask the kernel
devs to improve the error reporting here.

Re: Guix System ext4 index full

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 137 bytes --]

I believe similar ext4 issues were mentioned recently.

A good time to test Btrfs! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Guix System ext4 index full

[Roel Janssen]

On Wed, 2020-06-03 at 22:35 +0200, Tobias Geerinckx-Rice wrote:
> Roel,
> 
> Roel Janssen 写道：
> > Large directory feature is not enabled on this filesystem
> 
> Try setting the ‘large_dir’ feature on /dev/sdd3 with tune2fs -O
> 
> Kind regards,
> 
> T G-R

Thanks for the suggestion.  So I did:
$ tune2fs -O large_dir /dev/sdd3

.. and after a reboot I get booted into a GRUB resque shell.

After booting from a live USB, I chrooted into the Guix system and when
running the following command, I get the following error:
$ guix system reconfigure /etc/config.scm
...
guix system: error: while setting up the build environment: cannot
pivot old root directory onto
'/gnu/store/4q1vpidsawmfhmqdwy4mxvjc720jxcjm-slim-
1.3.6.drv.chroot/real-root': Invalid argument

So.. do you have a suggestion for how I could boot into the Guix system
again?

Kind regards,
Roel Janssen

Re: Guix System ext4 index full

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1913 bytes --]

Roel,

Roel Janssen 写道：
> Thanks for the suggestion.  So I did:
> $ tune2fs -O large_dir /dev/sdd3
>
> .. and after a reboot I get booted into a GRUB resque shell.

Shart.  It seems that GRUB still[0] can't read ext4 file systems 
with features newer than 2017 :-/  I'll never recommend this again 
(I use it myself…).

The flag can't be cleared once set:

  $ sudo tune2fs -O ^large_dir img
  tune2fs 1.45.6 (20-Mar-2020)
  Clearing filesystem feature 'large_dir' not supported.

Your data's perfectly safe.  This is just a GRUB limitation.

> So.. do you have a suggestion for how I could boot into the Guix 
> system
> again?

I think realistic options are:

- Migrate to any other file system like btrfs, then use the live 
  USB's GRUB to bootstrap it using ‘configfile 
  (hdX,Y)/your/grub.cfg’ so you don't have to reinstall anything, 
  just copy.  This can even be done-in place[1], assuming you have 
  back-ups.

- Use a separate /boot partition if you want to keep using ext4 
  for large /gnus.  You'll have to manually copy each kernel & 
  friends to /boot but this can be automated in your system .scm.

> After booting from a live USB, I chrooted into the Guix system 
> and when
> running the following command, I get the following error:
> $ guix system reconfigure /etc/config.scm
> ...
> guix system: error: while setting up the build environment: 
> cannot
> pivot old root directory onto
> '/gnu/store/4q1vpidsawmfhmqdwy4mxvjc720jxcjm-slim-
> 1.3.6.drv.chroot/real-root': Invalid argument

I think your in-chroot guix is talking to the live system's 
guix-daemon.  Perhaps stopping the out-of-chroot daemon and 
starting a ‘guix-daemon --disable-chroot’ inside the chroot will 
work.

Kind regards,

T G-R

[0]: https://savannah.gnu.org/bugs/?56897
[1]: https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-convert

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Guix System ext4 index full

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 391 bytes --]

Tobias Geerinckx-Rice 写道：
> I think your in-chroot guix is talking to the live system's
> guix-daemon.  Perhaps stopping the out-of-chroot daemon and 
> starting a
> ‘guix-daemon --disable-chroot’ inside the chroot will work.

If that doesn't help, perhaps this[0] will.

Kind regards,

T G-R

[0]: 
https://lists.gnu.org/archive/html/help-guix/2018-03/msg00116.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Guix System ext4 index full

[Roel Janssen]

Hi Tobias,

Thanks for your help.  Don't worry, I figured I had to reinstall it,
which is okay. :)

Unfortunately, after reinstalling Guix using Btrfs as the root
filesystem I cannot seem to boot either.  The last line printed on the
screen is:
[   10.830469 ] Error: Driver 'pcspkr' is already registered,
aborting...

I think it has something to do with the "compatibility support module"
option in the BIOS, because without it I couldn't boot the LiveUSB.  It
seems that booting and installing Fedora 32 works just fine with the
same hardware, so this is definitely an area where I'd like to see Guix
improve.

Kind regards,
Roel Janssen


On Thu, 2020-06-04 at 15:05 +0200, Tobias Geerinckx-Rice wrote:
> Roel,
> 
> Roel Janssen 写道：
> > Thanks for the suggestion.  So I did:
> > $ tune2fs -O large_dir /dev/sdd3
> > 
> > .. and after a reboot I get booted into a GRUB resque shell.
> 
> Shart.  It seems that GRUB still[0] can't read ext4 file systems 
> with features newer than 2017 :-/  I'll never recommend this again 
> (I use it myself…).
> 
> The flag can't be cleared once set:
> 
>   $ sudo tune2fs -O ^large_dir img
>   tune2fs 1.45.6 (20-Mar-2020)
>   Clearing filesystem feature 'large_dir' not supported.
> 
> Your data's perfectly safe.  This is just a GRUB limitation.
> 
> > So.. do you have a suggestion for how I could boot into the Guix 
> > system
> > again?
> 
> I think realistic options are:
> 
> - Migrate to any other file system like btrfs, then use the live 
>   USB's GRUB to bootstrap it using ‘configfile 
>   (hdX,Y)/your/grub.cfg’ so you don't have to reinstall anything, 
>   just copy.  This can even be done-in place[1], assuming you have 
>   back-ups.
> 
> - Use a separate /boot partition if you want to keep using ext4 
>   for large /gnus.  You'll have to manually copy each kernel & 
>   friends to /boot but this can be automated in your system .scm.
> 
> > After booting from a live USB, I chrooted into the Guix system 
> > and when
> > running the following command, I get the following error:
> > $ guix system reconfigure /etc/config.scm
> > ...
> > guix system: error: while setting up the build environment: 
> > cannot
> > pivot old root directory onto
> > '/gnu/store/4q1vpidsawmfhmqdwy4mxvjc720jxcjm-slim-
> > 1.3.6.drv.chroot/real-root': Invalid argument
> 
> I think your in-chroot guix is talking to the live system's 
> guix-daemon.  Perhaps stopping the out-of-chroot daemon and 
> starting a ‘guix-daemon --disable-chroot’ inside the chroot will 
> work.
> 
> Kind regards,
> 
> T G-R
> 
> [0]: https://savannah.gnu.org/bugs/?56897
> [1]: https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-convert

Re: Guix System ext4 index full

[Vincent Legoll]

Hello Roel,

On 04/06/2020 20:55, Roel Janssen wrote:
> Thanks for your help.  Don't worry, I figured I had to reinstall it,
> which is okay. :)
> 
> Unfortunately, after reinstalling Guix using Btrfs as the root
> filesystem I cannot seem to boot either.  The last line printed on the
> screen is:
> [   10.830469 ] Error: Driver 'pcspkr' is already registered,
> aborting...
> 
> I think it has something to do with the "compatibility support module"
> option in the BIOS, because without it I couldn't boot the LiveUSB.
Did you try reverting this option back to the state it was in before
doing the USB installation ?

This may be a BIOS / UEFI bootability difference between the USB flash
disk and the installed system.

-- 
Vincent Legoll