Re: No more space left on device issue

[ludo@gnu.org (Ludovic Courtès)]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> I've had this issue before but couldn't understand what was happening at
> the time. At that time, I simply resized my filesystem then ran the
> garbage collector.
>
> $ touch /tmp/test.txt
> touch: cannot touch '/tmp/test.txt': No space left on device
>
> $ df -h
> Filesystem      Size  Used Avail Use% Mounted on
> none            1.9G     0  1.9G   0% /dev
> /dev/sda1        37G   25G   11G  71% /
> tmpfs           1.9G     0  1.9G   0% /dev/shm
> cgroup          1.9G     0  1.9G   0% /sys/fs/cgroup
> none            1.9G   12K  1.9G   1% /run/systemd
> none            1.9G     0  1.9G   0% /run/user
> tmpfs           376M     0  376M   0% /run/user/1000
>
> In the #guix channel, jmd suggested that this might be an inode issue:
>
> $ sudo tune2fs -l /dev/sda1
> Password: 
> tune2fs 1.42.13 (17-May-2015)
> Filesystem volume name:   my-root
> Last mounted on:          /
> Filesystem UUID:          2b2b35a6-8f47-4fc2-9700-bda7fc1044cd
> Filesystem magic number:  0xEF53
> Filesystem revision #:    1 (dynamic)
> Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize
> Filesystem flags:         signed_directory_hash 
> Default mount options:    user_xattr acl
> Filesystem state:         clean
> Errors behavior:          Continue
> Filesystem OS type:       Linux
> Inode count:              2445984
> Block count:              9764864
> Reserved block count:     488242
> Free blocks:              3183059
> Free inodes:              23
> First block:              0
> Block size:               4096
> Fragment size:            4096
> Group descriptor size:    64
> Reserved GDT blocks:      1024
> Blocks per group:         32768
> Fragments per group:      32768
> Inodes per group:         8208
> Inode blocks per group:   513
> Flex block group size:    16
> Filesystem created:       Tue Nov  1 08:38:50 2016
> Last mount time:          Sun Dec 18 10:47:37 2016
> Last write time:          Sun Dec 18 10:47:37 2016
> Mount count:              20
> Maximum mount count:      -1
> Last checked:             Tue Nov  1 08:38:50 2016
> Check interval:           0 (<none>)
> Lifetime writes:          146 GB
> Reserved blocks uid:      0 (user root)
> Reserved blocks gid:      0 (group root)
> First inode:              11
> Inode size:               256
> Required extra isize:     32
> Desired extra isize:      32
> Journal inode:            8
> Default directory hash:   half_md4
> Directory Hash Seed:      330c34f4-9c29-4b59-bb21-d8d00770e2e7
> Journal backup:           inode blocks
>
>
> It does seem like it is; the "Free inodes" count is very low at 23 and
> this is only refreshed at the time the filesystem is mounted (since I
> last booted).

Indeed, it’s low on free inodes.  :-)

That said, 37 G is not that much (my laptop’s root partition, which
includes the store, is 64 G, and I expect most users are in this
ballpark).  So I wonder if there might be something else consuming
inodes on this file system.  Any ideas?

My root partition has different settings though (it’s really ext3):

--8<---------------cut here---------------start------------->8---
$ sudo tune2fs -l /dev/sda2
tune2fs 1.42.13 (17-May-2015)
Filesystem volume name:   root
Last mounted on:          /
Filesystem UUID:          c5307e6b-d1ba-499d-89c5-cb0b143577c4
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
--8<---------------cut here---------------end--------------->8---

Could it be that one of these explains the difference?

FWIW mirror.hydra.gnu.org and hydra.gnu.org both have a 1.5TB /gnu as ext4.

In the past we had issues where the directory index of /gnu/store/.links
(used for deduplication) would be full (that’s really the directory
index, not a lack of inodes), which was fixed in
12b6c951cf5ca6055a22a2eec85665353f5510e5.

Ludo’.