bug#51787: Disk performance on ci.guix.gnu.org

[Ricardo Wurmus <rekado@elephly.net>]

Ricardo Wurmus <rekado@elephly.net> writes:

> Today we discovered a few more things and discussed them on IRC.  Here’s
> a summary.
>
> /var/cache sits on the same storage as /gnu.  We mounted the 5TB ext4
> file system that’s hosted by the SAN at /mnt_test and started copying
> over /var/cache to /mnt_test/var/cache.  Transfer speed was considerably
> faster (not *great*, but reasonably fast) than the copy of
> /gnu/store/trash to the same target.

Turns out that space on the SAN is insufficient for a full copy of
/var/cache.  We’ve hit ENOSPC after 4.2TB.  The SAN enforces some
headroom to remain free, so it denies us full access to the 5TB slice.
Bummer.

I guess we’ll have to wait for the SAN extension some time early 2022
before we can relocate the substitutes cache.

Should we attempt to overwrite /gnu/store and rely exclusively on
substitutes from the cache?

No matter how we look at it, the huge store is a performance problem for
us.  Today I had to kill ’guix gc’ after the GC lock had been held for
about 24 hours.  We will keep having this problem.

-- 
Ricardo