bug#47379: "statfs" test in tests/syscall.scm fails with BTRFS file systems.

bug#47379: "statfs" test in tests/syscall.scm fails with BTRFS file systems.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1077 bytes --]

Hi Guix,

Version:
guix (GNU Guix) 1155a88308df7649fe74bd5bb8279a4d103ce386

The following test fails:

(start snip)
test-name: statfs
location: $HOME/guix/git/guix/tests/syscalls.scm:123
source:
+ (test-assert
+   "statfs"
+   (let ((fs (statfs "/")))
+     (and (file-system? fs)
+          (> (file-system-block-size fs) 0)
+          (>= (file-system-blocks-available fs) 0)
+          (>= (file-system-blocks-free fs)
+              (file-system-blocks-available fs)))))
actual-value: #f
result: FAIL
(end snip)

Evaluating (statfs "/") from a REPL gives:

scheme@(guix-user)> ((@ (guix build syscalls) statfs) "/")
$2 = #<<file-system> type: 2435016766 block-size: 4096 blocks: 244189696 blocks-free: 178549974 blocks-available: 178571318 files: 0 free-
files: 0 identifier: (1111009624 2088757363) name-length: 255 fragment-size: 4096 mount-flags: 1056 spare: (0 0 0 0)>

It seems the following does not hold on my system:
+          (>= (file-system-blocks-free fs)
+              (file-system-blocks-available fs))

Greetings, Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#47379: "statfs" test in tests/syscall.scm fails with BTRFS file systems.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2199 bytes --]

On Thu, Mar 25, 2021 at 09:49:08AM +0100, Maxime Devos wrote:
> Hi Guix,
> 
> Version:
> guix (GNU Guix) 1155a88308df7649fe74bd5bb8279a4d103ce386
> 
> The following test fails:
> 
> (start snip)
> test-name: statfs
> location: $HOME/guix/git/guix/tests/syscalls.scm:123
> source:
> + (test-assert
> +   "statfs"
> +   (let ((fs (statfs "/")))
> +     (and (file-system? fs)
> +          (> (file-system-block-size fs) 0)
> +          (>= (file-system-blocks-available fs) 0)
> +          (>= (file-system-blocks-free fs)
> +              (file-system-blocks-available fs)))))
> actual-value: #f
> result: FAIL
> (end snip)
> 
> Evaluating (statfs "/") from a REPL gives:
> 
> scheme@(guix-user)> ((@ (guix build syscalls) statfs) "/")
> $2 = #<<file-system> type: 2435016766 block-size: 4096 blocks: 244189696 blocks-free: 178549974 blocks-available: 178571318 files: 0 free-
> files: 0 identifier: (1111009624 2088757363) name-length: 255 fragment-size: 4096 mount-flags: 1056 spare: (0 0 0 0)>
> 
> It seems the following does not hold on my system:
> +          (>= (file-system-blocks-free fs)
> +              (file-system-blocks-available fs))
> 

I'm also running on btrfs
(ins)scheme@(guile-user)> (statfs "/")
$1 = #<<file-system> type: 2435016766 block-size: 4096 blocks: 244049664 blocks-free: 165299045 blocks-available: 165178857 files: 0 free-files: 0 identifier:
(2930232877 3283401406) name-length: 255 fragment-size: 4096 mount-flags: 4128 spare: (0 0 0 0)>
(cmd)scheme@(guile-user)> (>= (file-system-blocks-free (statfs "/")) (file-system-blocks-available (statfs "/")))
$5 = #t

(ins)efraim@3900XT ~$ btrfs filesystem df /
Data, single: total=289.00GiB, used=287.83GiB
System, single: total=32.00MiB, used=48.00KiB
Metadata, single: total=13.00GiB, used=12.07GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

When was the last time you ran btrfs balance? Regardless, we still want
this test to pass on btrfs.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#47379: "statfs" test in tests/syscall.scm fails with BTRFS file systems.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 889 bytes --]

On Thu, 2021-03-25 at 12:04 +0200, Efraim Flashner wrote:
> [...]
> I'm also running on btrfs
> (ins)scheme@(guile-user)> (statfs "/")
> [...]
> 
> (ins)efraim@3900XT ~$ btrfs filesystem df /
> Data, single: total=289.00GiB, used=287.83GiB
> System, single: total=32.00MiB, used=48.00KiB
> Metadata, single: total=13.00GiB, used=12.07GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> When was the last time you ran btrfs balance?

... never? My output of "btrfs filesystem df" FWIW:

$ btrfs filesystem df /
Data, single: total=224.01GiB, used=223.14GiB
System, DUP: total=8.00MiB, used=48.00KiB
Metadata, DUP: total=14.00GiB, used=13.81GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

Are there any reasons for running "btrfs balance"?
If so, what command do you recommend (I assume something
only touching meta data?)

Greetings,
Maxime.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#47379: "statfs" test in tests/syscall.scm fails with BTRFS file systems.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1646 bytes --]

On Thu, Mar 25, 2021 at 11:16:25AM +0100, Maxime Devos wrote:
> On Thu, 2021-03-25 at 12:04 +0200, Efraim Flashner wrote:
> > [...]
> > I'm also running on btrfs
> > (ins)scheme@(guile-user)> (statfs "/")
> > [...]
> > 
> > (ins)efraim@3900XT ~$ btrfs filesystem df /
> > Data, single: total=289.00GiB, used=287.83GiB
> > System, single: total=32.00MiB, used=48.00KiB
> > Metadata, single: total=13.00GiB, used=12.07GiB
> > GlobalReserve, single: total=512.00MiB, used=0.00B
> > 
> > When was the last time you ran btrfs balance?
> 
> ... never? My output of "btrfs filesystem df" FWIW:
> 
> $ btrfs filesystem df /
> Data, single: total=224.01GiB, used=223.14GiB
> System, DUP: total=8.00MiB, used=48.00KiB
> Metadata, DUP: total=14.00GiB, used=13.81GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> Are there any reasons for running "btrfs balance"?
> If so, what command do you recommend (I assume something
> only touching meta data?)
> 

btrfs balance moves the free space around so that you have fewer blocks
with extra freed space. I normally run 'btrfs balance start -dusage=70
-musage=80 $mountpoint'. (unless I have it backwards) this takes any
blocks with data which are less than 70% used and metadata blocks with
less than 80% used and re-arranges them to effectively make the total
space allocated closer to the actual space used. Running it with 50 is
normally plenty.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#47379: "statfs" test in tests/syscall.scm fails with BTRFS file systems.

[raid5atemyhomework via Bug reports for GNU Guix]

> btrfs balance moves the free space around so that you have fewer blocks
> with extra freed space. I normally run 'btrfs balance start -dusage=70
> -musage=80 $mountpoint'. (unless I have it backwards)

I think you do?  Usually the numbers for `musage` are smaller I think.

There is some old advice that you should only balance data and never balance metadata, i.e. `btrfs balance start -dusage=70 $mountpoint`.  This is because 1Gb blocks are assigned to either data or metadata, and it's possible for excessive balances to result in a situation where the metadata gets only a single 1Gb block and the rest of storage is assigned to data.  Then the single metadata 1Gb block gets filled, and when new metadata is needed --- such as to rebalance the large number of data blocks so they take up fewer 1Gb blocks and more blocks can be assigned to metadata --- the filesystem is unable to continue operating due to the lack of metadata space and you are stuck in a condition where you cannot delete data, delete snapshots, and rebalance data.

This is old advice since the new "GlobalReserve" (not so new I think, it was added way back in 4.x? 3.x?) should provide space for temporary metadata operations in such a case.  Personally I'd rather just let metadata be "loose" and unbalanced to avoid the situation altogether; metadata is fairly tiny so it taking up more than one 1Gb block usually means it has two 1Gb blocks, maybe three at a stretch if you've been doing a lot of file creation and deletion events.

Another piece of old advice is to regularly balance.  For example, have a daily `btrfs balance start -dusage=50 -dlimit=2 $mountpoint` --- the `dlimit` makes it so that balancing stops when two 1Gb blocks of data have been merged into some other half-filled 1Gb blocks of data.  If you have never balanced your BTRFS system, you might want to wait for some low-utilization time period, do a full `btrfs balance start -dusage=90 $mountpoint` without a `dlimit`, then schedule a daily balance of `-dusage=50 -dlimit=2` afterwards.  On the other hand, if you're using SSDs, be aware that balancing leads to writing, which lowers your drive's longevity (but the point of `dlimit` is to prevent excessive amounts of daily work, and if you're regularly writing to your disk (almost) everyday anyway, a small `dusage` and `dlimit` would be within the noise of your daily-work-activity writes).

You also want to do regular `btrfs scrub start $mountpoint`.  Once a week for consumer-quality drives, once a month for enterprise-quality drives, if you're not sure which one you have, go weekly.  This is advice typical from ZFS but should still apply to BTRFS.

On SSD (or other storage with TRIM commands) you might want to do scheduled trim regularly once a week or once every two weeks, in order to take allocation pressure off the SSD and let it get better wear-levelling.  This is generally done via `fstrim` without any BTRFS-specific commands.  Old advice is to avoid the `discard` mount option (in some cases it can trim so often that the SSD lifetime is significantly reduced) but that's supposed to be fixed so maybe with a recent version you can mount `-o discard`, maybe.   Personally I'd use explicit scheduled trim still.  Do try to schedule this at low-activity times, though; unless you've got SATA 3.1 (hard to check, most drives/controllers just say "SATA 3" or "SATA III" which may or may not mean including SATA 3.1 support), or SAS, or real SCSI, trim commands are slow.

Finally you might also want to do explicit defragmentation (which is a separate issue from balancing --- balancing ensures you don't have lots of half-used blocks, defragging means files try to have as much of their data in the same 1Gb block) periodically, like once a week or two weeks.


See also https://github.com/kdave/btrfsmaintenance for a package that does btrfs maintenance for you, including balance, scrubbing, trimming, and defragging, and schedules those in "recommended" times as well.  I think it might also have auto-snapshotting, though that is a bit more fraught as snapshots are fairly heavyweight on BTRFS.  Do note that it's crontab/SystemD-based though, so needs a good amount of glue code if you want to use it in Guix.  It's available on Debian as `btrfsmaintenance` package.  It's also got a lot of settings, so you'd be up for a fairly comprehensive configuration system to adapt it for Guix.


Going back on topic... It looks like the test assumes "free" should equal "available", but that is something that is likely not to work on ***all*** copy-on-write filesystems --- including ZFS and bcachefs, not just BTRFS.  In particular, most copy-on-write filesystems (BTRFS, ZFS, and bcachefs) support transparent compression, meaning "available" is often an estimated multiple of "free".  Probably the test should either explicitly use a specific filesystem (maybe `tmpfs` would work? Or create a 1Gb "dd if=/dev/zero` file in `/tmp` and bind-mount `ext4` onto it) that is simple enough that "free" == "available" most of the time, or it should just remove that particular test.


Thanks
raid5atemyhomework