No more space left on device issue

No more space left on device issue

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 723 bytes --]

Hello Guix,

I've had this issue before but couldn't understand what was happening at
the time. At that time, I simply resized my filesystem then ran the
garbage collector.

$ touch /tmp/test.txt
touch: cannot touch '/tmp/test.txt': No space left on device

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
none            1.9G     0  1.9G   0% /dev
/dev/sda1        37G   25G   11G  71% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
cgroup          1.9G     0  1.9G   0% /sys/fs/cgroup
none            1.9G   12K  1.9G   1% /run/systemd
none            1.9G     0  1.9G   0% /run/user
tmpfs           376M     0  376M   0% /run/user/1000

In the #guix channel, jmd suggested that this might be an inode issue:


[-- Attachment #2: Type: text/plain, Size: 1788 bytes --]

$ sudo tune2fs -l /dev/sda1
Password: 
tune2fs 1.42.13 (17-May-2015)
Filesystem volume name:   my-root
Last mounted on:          /
Filesystem UUID:          2b2b35a6-8f47-4fc2-9700-bda7fc1044cd
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              2445984
Block count:              9764864
Reserved block count:     488242
Free blocks:              3183059
Free inodes:              23
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8208
Inode blocks per group:   513
Flex block group size:    16
Filesystem created:       Tue Nov  1 08:38:50 2016
Last mount time:          Sun Dec 18 10:47:37 2016
Last write time:          Sun Dec 18 10:47:37 2016
Mount count:              20
Maximum mount count:      -1
Last checked:             Tue Nov  1 08:38:50 2016
Check interval:           0 (<none>)
Lifetime writes:          146 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      330c34f4-9c29-4b59-bb21-d8d00770e2e7
Journal backup:           inode blocks

[-- Attachment #3: Type: text/plain, Size: 381 bytes --]


It does seem like it is; the "Free inodes" count is very low at 23 and
this is only refreshed at the time the filesystem is mounted (since I
last booted).

jmd also requests that I post this information here so that the
configuration of the GuixSD filesystem settings can be discussed. With
the store being very inode intensive, there might be something to tweak?

Thanks,

Maxim

Re: No more space left on device issue

[Ludovic Courtès]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> I've had this issue before but couldn't understand what was happening at
> the time. At that time, I simply resized my filesystem then ran the
> garbage collector.
>
> $ touch /tmp/test.txt
> touch: cannot touch '/tmp/test.txt': No space left on device
>
> $ df -h
> Filesystem      Size  Used Avail Use% Mounted on
> none            1.9G     0  1.9G   0% /dev
> /dev/sda1        37G   25G   11G  71% /
> tmpfs           1.9G     0  1.9G   0% /dev/shm
> cgroup          1.9G     0  1.9G   0% /sys/fs/cgroup
> none            1.9G   12K  1.9G   1% /run/systemd
> none            1.9G     0  1.9G   0% /run/user
> tmpfs           376M     0  376M   0% /run/user/1000
>
> In the #guix channel, jmd suggested that this might be an inode issue:
>
> $ sudo tune2fs -l /dev/sda1
> Password: 
> tune2fs 1.42.13 (17-May-2015)
> Filesystem volume name:   my-root
> Last mounted on:          /
> Filesystem UUID:          2b2b35a6-8f47-4fc2-9700-bda7fc1044cd
> Filesystem magic number:  0xEF53
> Filesystem revision #:    1 (dynamic)
> Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize
> Filesystem flags:         signed_directory_hash 
> Default mount options:    user_xattr acl
> Filesystem state:         clean
> Errors behavior:          Continue
> Filesystem OS type:       Linux
> Inode count:              2445984
> Block count:              9764864
> Reserved block count:     488242
> Free blocks:              3183059
> Free inodes:              23
> First block:              0
> Block size:               4096
> Fragment size:            4096
> Group descriptor size:    64
> Reserved GDT blocks:      1024
> Blocks per group:         32768
> Fragments per group:      32768
> Inodes per group:         8208
> Inode blocks per group:   513
> Flex block group size:    16
> Filesystem created:       Tue Nov  1 08:38:50 2016
> Last mount time:          Sun Dec 18 10:47:37 2016
> Last write time:          Sun Dec 18 10:47:37 2016
> Mount count:              20
> Maximum mount count:      -1
> Last checked:             Tue Nov  1 08:38:50 2016
> Check interval:           0 (<none>)
> Lifetime writes:          146 GB
> Reserved blocks uid:      0 (user root)
> Reserved blocks gid:      0 (group root)
> First inode:              11
> Inode size:               256
> Required extra isize:     32
> Desired extra isize:      32
> Journal inode:            8
> Default directory hash:   half_md4
> Directory Hash Seed:      330c34f4-9c29-4b59-bb21-d8d00770e2e7
> Journal backup:           inode blocks
>
>
> It does seem like it is; the "Free inodes" count is very low at 23 and
> this is only refreshed at the time the filesystem is mounted (since I
> last booted).

Indeed, it’s low on free inodes.  :-)

That said, 37 G is not that much (my laptop’s root partition, which
includes the store, is 64 G, and I expect most users are in this
ballpark).  So I wonder if there might be something else consuming
inodes on this file system.  Any ideas?

My root partition has different settings though (it’s really ext3):

--8<---------------cut here---------------start------------->8---
$ sudo tune2fs -l /dev/sda2
tune2fs 1.42.13 (17-May-2015)
Filesystem volume name:   root
Last mounted on:          /
Filesystem UUID:          c5307e6b-d1ba-499d-89c5-cb0b143577c4
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
--8<---------------cut here---------------end--------------->8---

Could it be that one of these explains the difference?

FWIW mirror.hydra.gnu.org and hydra.gnu.org both have a 1.5TB /gnu as ext4.

In the past we had issues where the directory index of /gnu/store/.links
(used for deduplication) would be full (that’s really the directory
index, not a lack of inodes), which was fixed in
12b6c951cf5ca6055a22a2eec85665353f5510e5.

Ludo’.

Re: No more space left on device issue

[Maxim Cournoyer]

Hi Ludovic!

ludo@gnu.org (Ludovic Courtès) writes:

> Indeed, it’s low on free inodes.  :-)
>
> That said, 37 G is not that much (my laptop’s root partition, which
> includes the store, is 64 G, and I expect most users are in this
> ballpark).

I understand that my partition size is not very large; This is a 2011
era laptop with a 64 GB SSD. Although quite dated and behind todays'
standard, it's useful in exposing the limits of the software faster ;).

What I don't understand is why the all the inodes are used at only 71%
of disk usage (11GB left!). There's not much else than Guix there; I
have a documents folder (700 MB with 4k files) and some git
repositories (2.6 GB, 38k files), but that's it.

Assuming that the problem is related to the Guix store being very
file/link intensive and that the Hydra servers deplete their inodes at a
similar ratio, that would lead to their 1.5 TB EXT4 filesystem being
more like 1 TB of usable storage.

> My root partition has different settings though (it’s really ext3):
>
> $ sudo tune2fs -l /dev/sda2
> tune2fs 1.42.13 (17-May-2015)
> Filesystem volume name:   root
> Last mounted on:          /
> Filesystem UUID:          c5307e6b-d1ba-499d-89c5-cb0b143577c4
> Filesystem magic number:  0xEF53
> Filesystem revision #:    1 (dynamic)
> Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
>
> Could it be that one of these explains the difference?
>

I guess we can't really compare ext3 and ext4. We'd need a filesystem
versed person to shed some light here. I'd be more interested to know if
the Hydra servers can (nearly) max their ext4 filesystem without running
out of inodes.

Thanks for your reply,

Maxim

Re: No more space left on device issue

[Ludovic Courtès]

Hello Maxim!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Indeed, it’s low on free inodes.  :-)
>>
>> That said, 37 G is not that much (my laptop’s root partition, which
>> includes the store, is 64 G, and I expect most users are in this
>> ballpark).
>
> I understand that my partition size is not very large; This is a 2011
> era laptop with a 64 GB SSD. Although quite dated and behind todays'
> standard, it's useful in exposing the limits of the software faster ;).

Right; what I meant is that 37 G shouldn’t cause any problems because
it’s relatively small.

> What I don't understand is why the all the inodes are used at only 71%
> of disk usage (11GB left!). There's not much else than Guix there; I
> have a documents folder (700 MB with 4k files) and some git
> repositories (2.6 GB, 38k files), but that's it.

Not sure, I don’t know of inode usage profiling tools.

Does running “guix gc” help?

> Assuming that the problem is related to the Guix store being very
> file/link intensive and that the Hydra servers deplete their inodes at a
> similar ratio, that would lead to their 1.5 TB EXT4 filesystem being
> more like 1 TB of usable storage.

[...]

> I guess we can't really compare ext3 and ext4. We'd need a filesystem
> versed person to shed some light here. I'd be more interested to know if
> the Hydra servers can (nearly) max their ext4 filesystem without running
> out of inodes.

FWIW, on hydra.gnu.org we have this:

--8<---------------cut here---------------start------------->8---
$ df -i /gnu/store
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvda1           98304000 6864127 91439873    7% /
$ df -h /gnu/store
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            1.5T  521G  881G  38% /
--8<---------------cut here---------------end--------------->8---

On mirror.hydra.gnu.org:

--8<---------------cut here---------------start------------->8---
$ df -i /gnu/store
Filesystem               Inodes    IUsed    IFree IUse% Mounted on
/dev/mapper/vg0-store 100663296 10988332 89674964   11% /gnu
$ df -h /gnu/store
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/vg0-store  1.5T  1.4T  103G  93% /gnu
--8<---------------cut here---------------end--------------->8---

Ludo’.

Re: No more space left on device issue

[Tobias Geerinckx-Rice]

[-- Attachment #1.1: Type: text/plain, Size: 605 bytes --]

Maxim,

On 21/12/16 22:48, Ludovic Courtès wrote:
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>> What I don't understand is why the all the inodes are used at only 71%
>> of disk usage (11GB left!). There's not much else than Guix there; I
>> have a documents folder (700 MB with 4k files) and some git
>> repositories (2.6 GB, 38k files), but that's it.
> 
> Not sure, I don’t know of inode usage profiling tools.

There's always ‘du --inode’ (in combination with ‘-s’ if needed) to
quickly point the finger o' blame in the right direction.

Kind regards,

T G-R


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 476 bytes --]

Re: No more space left on device issue

[Maxim Cournoyer]

Hi Ludovic!

ludo@gnu.org (Ludovic Courtès) writes:

> Does running “guix gc” help?

Yes; that's been my fix for now. Running the garbage collector purged 17
GB worth of data from the store.

> FWIW, on hydra.gnu.org we have this:
>
> On mirror.hydra.gnu.org:
>
> $ df -i /gnu/store
> Filesystem               Inodes    IUsed    IFree IUse% Mounted on
> /dev/mapper/vg0-store 100663296 10988332 89674964   11% /gnu
> $ df -h /gnu/store
> Filesystem             Size  Used Avail Use% Mounted on
> /dev/mapper/vg0-store  1.5T  1.4T  103G  93% /gnu

That one is very interesting. The filesystem is nearly full but still it
is only using a tenth of its available inodes. Unless this hard drive is
filled in big part by large single file blobs such as a backups or
similar, it defeats the theory that the currently used EXT4 filesystem
parameters are not optimal for the needs of Guix.

Thanks for this useful info!

Maxim

Re: No more space left on device issue

[Maxim Cournoyer]

Hello Tobias,

Tobias Geerinckx-Rice <me@tobias.gr> writes:

> Maxim,
>
> On 21/12/16 22:48, Ludovic Courtès wrote:
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>>> What I don't understand is why the all the inodes are used at only 71%
>>> of disk usage (11GB left!). There's not much else than Guix there; I
>>> have a documents folder (700 MB with 4k files) and some git
>>> repositories (2.6 GB, 38k files), but that's it.
>> 
>> Not sure, I don’t know of inode usage profiling tools.
>
> There's always ‘du --inode’ (in combination with ‘-s’ if needed) to
> quickly point the finger o' blame in the right direction.

My 37 GiB ext4 partition apparently has a total of around 2.5 M
inodes. Currently (after recently running the garbage collector), its
inodes usage is at 15%:

df -i /dev/sda1
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda1      2445984 358647 2087337   15% /

My home dir only uses 56 k inodes:

du --inodes -s ~
56002   /home/maxim

This would suggest that in my case it really was the Guix store which
was responsible for consuming all the available inodes. I'll keep
monitoring it as it progressively fills up again.

Thanks for pointing at the --inodes option of du, I didn't know about
it!

Maxim