bug#42001: “SQLite database is busy”: contention on the store database

["Ludovic Courtès" <ludo@gnu.org>]

Hi,

Mathieu Othacehe <othacehe@gnu.org> skribis:

>> utimensat(AT_FDCWD, "/gnu/store/whhc80cx1zpi9cj3bil3sp8wwzcf8p9p-image-root/gnu/store/n6l40ygclxfysnc2nxwrslnn5ab6wv3v-profile/lib/modules/5.4.47-gnu/kernel/net/netfilter", [{tv_sec=1592814617, tv_nsec=396083395} /* 2020-06-22T10:30:17.396083395+0200 */, {tv_sec=1, tv_nsec=0} /* 1970-01-01T01:00:01+0100 */], 0) = 0
>> chmod("/gnu/store/whhc80cx1zpi9cj3bil3sp8wwzcf8p9p-image-root/gnu/store/n6l40ygclxfysnc2nxwrslnn5ab6wv3v-profile/lib/modules/5.4.47-gnu/kernel/net/netfilter", 040555) = 0
>> getrusage(RUSAGE_SELF, {ru_utime={tv_sec=1306, tv_usec=334062}, ru_stime={tv_sec=143, tv_usec=819675}, ...}) = 0
>> clock_gettime(CLOCK_PROCESS_CPUTIME_ID, {tv_sec=1450, tv_nsec=153748295}) = 0 
>> lstat("/gnu/store/whhc80cx1zpi9cj3bil3sp8wwzcf8p9p-image-root/gnu/store/n6l40ygclxfysnc2nxwrslnn5ab6wv3v-profile/lib/modules/5.4.47-gnu/kernel/net/netfilter/xt_connlimit.ko", {st_mode=S_IFLNK|0777, st_size=125, ...}) = 0
>
> As discussed on IRC, those "image-root" derivations do exercise store
> deduplication a lot, are could participate to the observed database
> contention.
>
> Turns out I used those derivations as "staging" directories that are
> passed to GNU Xorriso when producing ISO images and "mke2fs" and
> "mkdosfs" when producing raw disk-images. Using the store was also
> convenient to ensure root ownership of the image files.
>
> With 1cb9effc3e875ec7bae31bb06069d16ac89f7e1d and
> 7f75a7ec08975eb6d6e01db61bd6b91f447f655e, I remove those "image-root"
> derivations, and proceed to image creation in a single derivation.

Nice, thanks for the prompt change!

I’ve been monitoring berlin a bit and I’ve become quite convinced that
these big directories are what triggered contention.  The good thing is
that this has allowed us to find weaknesses in (guix store database) and
(guix store deduplicate).

> The later commit is a bit more tricky because "mke2fs" is not able to
> override file permissions. I used "fakeroot" to make sure that the raw
> disk-image files are owned by "root" and not "guixbuilder".

OK, makes sense.

Maybe at some point we’ll have to use libext2fs directly or patch mke2fs
to get the desired flexibility.

Thank you!

Ludo’.