From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ricardo Wurmus Subject: bug#24937: "deleting unused links" GC phase is too slow Date: Thu, 16 Apr 2020 16:27:27 +0200 Message-ID: <87eesnmrow.fsf@elephly.net> References: <87wpg7ffbm.fsf@gnu.org> <87lgvm4lzu.fsf@gnu.org> <87twaaa6j9.fsf@netris.org> <87twaa2vjx.fsf@gnu.org> <87lgvm9sgq.fsf@netris.org> <87d1gwvgu0.fsf@gnu.org> <87wpf4yoz0.fsf@netris.org> <87fulrsqxx.fsf@gnu.org> <87vaunbvcu.fsf@mdc-berlin.de> <87ftd3muhp.fsf@elephly.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:41516) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jP5Uq-0002uK-47 for bug-guix@gnu.org; Thu, 16 Apr 2020 10:28:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jP5Uo-0002GA-JR for bug-guix@gnu.org; Thu, 16 Apr 2020 10:28:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:55747) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jP5Uo-0002G3-Ft for bug-guix@gnu.org; Thu, 16 Apr 2020 10:28:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jP5Uo-0001Ro-DO for bug-guix@gnu.org; Thu, 16 Apr 2020 10:28:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-reply-to: <87ftd3muhp.fsf@elephly.net> List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane-mx.org@gnu.org Sender: "bug-Guix" To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 24937@debbugs.gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Here are more benchmarks on one of the build nodes. It doesn=E2=80=99t nea= rly have as many used inodes as ci.guix.gnu.org, but I could fill it up if necessary. root@hydra-guix-127 ~# df -i /gnu/ Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda3 28950528 2796829 26153699 10% / root@hydra-guix-127 ~# ls -1 /gnu/store/.links | wc -l 2017395 I tested all three modes with statx and with lstat. The links-traversal-statx.c is attached below. * mode 1 + statx --8<---------------cut here---------------start------------->8--- root@hydra-guix-127 ~ [env]# gcc -Wall -std=3Dc99 links-traversal-statx.c -= DMODE=3D1 -D_GNU_SOURCE=3D1 -o links-traversal links-traversal-statx.c:53:8: warning: =EF=BF=BDstat_entries=EF=BF=BD defin= ed but not used [-Wunused-function] 53 | void stat_entries (void) | ^~~~~~~~~~~~ root@hydra-guix-127 ~ [env]# echo 3 > /proc/sys/vm/drop_caches root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 9 seconds (including stat) real 0m9.176s user 0m0.801s sys 0m4.236s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 4 seconds (including stat) real 0m3.556s user 0m0.708s sys 0m2.848s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 4 seconds (including stat) real 0m3.553s user 0m0.599s sys 0m2.954s root@hydra-guix-127 ~ [env]#=20 --8<---------------cut here---------------end--------------->8--- * mode 2 + statx --8<---------------cut here---------------start------------->8--- root@hydra-guix-127 ~ [env]# gcc -Wall -std=3Dc99 links-traversal-statx.c -= DMODE=3D2 -D_GNU_SOURCE=3D1 -o links-traversal root@hydra-guix-127 ~ [env]# echo 3 > /proc/sys/vm/drop_caches root@hydra-guix-127 ~ [env]# time ./links-traversal=20 17377 dir_entries, 10 seconds (including stat) real 0m9.598s user 0m1.210s sys 0m4.257s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 17377 dir_entries, 4 seconds (including stat) real 0m4.094s user 0m0.988s sys 0m3.107s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 17377 dir_entries, 4 seconds (including stat) real 0m4.095s user 0m0.933s sys 0m3.162s root@hydra-guix-127 ~ [env]#=20 --8<---------------cut here---------------end--------------->8--- * mode 3 + statx --8<---------------cut here---------------start------------->8--- root@hydra-guix-127 ~ [env]# gcc -Wall -std=3Dc99 links-traversal-statx.c -= DMODE=3D3 -D_GNU_SOURCE=3D1 -o links-traversal^C root@hydra-guix-127 ~ [env]# echo 3 > /proc/sys/vm/drop_caches root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 7 seconds stat took 3 seconds real 0m9.992s user 0m1.411s sys 0m4.221s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 1 seconds stat took 2 seconds real 0m4.265s user 0m1.120s sys 0m3.145s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 2 seconds stat took 2 seconds real 0m4.267s user 0m1.072s sys 0m3.195s root@hydra-guix-127 ~ [env]#=20 --8<---------------cut here---------------end--------------->8--- Now with just lstat: * mode 1 + lstat --8<---------------cut here---------------start------------->8--- root@hydra-guix-127 ~ [env]# gcc -Wall -std=3Dc99 links-traversal.c -DMODE= =3D1 -D_GNU_SOURCE=3D1 -o links-traversal links-traversal.c:49:8: warning: =EF=BF=BDstat_entries=EF=BF=BD defined but= not used [-Wunused-function] 49 | void stat_entries (void) | ^~~~~~~~~~~~ root@hydra-guix-127 ~ [env]# echo 3 > /proc/sys/vm/drop_caches root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 9 seconds (including stat) real 0m9.303s user 0m0.748s sys 0m4.397s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 4 seconds (including stat) real 0m3.526s user 0m0.540s sys 0m2.987s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 3 seconds (including stat) real 0m3.519s user 0m0.600s sys 0m2.919s root@hydra-guix-127 ~ [env]#=20 --8<---------------cut here---------------end--------------->8--- * mode 2 + lstat --8<---------------cut here---------------start------------->8--- root@hydra-guix-127 ~ [env]# gcc -Wall -std=3Dc99 links-traversal.c -DMODE= =3D2 -D_GNU_SOURCE=3D1 -o links-traversal root@hydra-guix-127 ~ [env]# echo 3 > /proc/sys/vm/drop_caches root@hydra-guix-127 ~ [env]# time ./links-traversal=20 17377 dir_entries, 9 seconds (including stat) real 0m9.614s user 0m1.205s sys 0m4.250s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 17377 dir_entries, 4 seconds (including stat) real 0m4.060s user 0m1.052s sys 0m3.008s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 17377 dir_entries, 4 seconds (including stat) real 0m4.057s user 0m0.984s sys 0m3.073s root@hydra-guix-127 ~ [env]#=20 --8<---------------cut here---------------end--------------->8--- * mode 3 + lstat --8<---------------cut here---------------start------------->8--- root@hydra-guix-127 ~ [env]# gcc -Wall -std=3Dc99 links-traversal.c -DMODE= =3D3 -D_GNU_SOURCE=3D1 -o links-traversal root@hydra-guix-127 ~ [env]# echo 3 > /proc/sys/vm/drop_caches root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 6 seconds stat took 3 seconds real 0m9.767s user 0m1.270s sys 0m4.339s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 2 seconds stat took 2 seconds real 0m4.234s user 0m1.136s sys 0m3.097s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 1 seconds stat took 2 seconds real 0m4.222s user 0m1.052s sys 0m3.170s root@hydra-guix-127 ~ [env]#=20 --8<---------------cut here---------------end--------------->8--- They are all very close, so I think I need to work with a bigger store to see a difference. Or perhaps I did something silly because I don=E2=80=99t know C=E2=80=A6 I= f so please let me know. -- Ricardo --=-=-= Content-Type: application/octet-stream Content-Disposition: attachment; filename=links-traversal-statx.c Content-Transfer-Encoding: base64 I2luY2x1ZGUgPHVuaXN0ZC5oPgojaW5jbHVkZSA8ZGlyZW50Lmg+CiNpbmNsdWRlIDxzeXMvdHlw ZXMuaD4KI2luY2x1ZGUgPHN0ZGxpYi5oPgojaW5jbHVkZSA8c3RkaW8uaD4KI2luY2x1ZGUgPHN5 cy90aW1lLmg+CiNpbmNsdWRlIDxzdHJpbmcuaD4KI2luY2x1ZGUgPHN5cy9zdGF0Lmg+CiNpbmNs dWRlIDx1bmlzdGQuaD4KI2luY2x1ZGUgPGZjbnRsLmg+CiNpbmNsdWRlIDxhc3NlcnQuaD4KCgoK I2RlZmluZSBTVEFUX0lOVEVSTEVBVkVEIDEKI2RlZmluZSBTVEFUX1NFTUlfSU5URVJMRUFWRUQg MgojZGVmaW5lIFNUQVRfT1BUSU1BTCAzCgpzdHJ1Y3QgZW50cnkKewogIGNoYXIgKm5hbWU7CiAg aW5vX3QgaW5vZGU7Cn07CgojZGVmaW5lIE1BWF9FTlRSSUVTIDEzNTAwMDAwMApzdGF0aWMgc3Ry dWN0IGVudHJ5IGRpcl9lbnRyaWVzW01BWF9FTlRSSUVTXTsKCmludAptYWluICgpCnsKICBzdHJ1 Y3QgdGltZXZhbCBzdGFydCwgZW5kOwoKICAvKiBGb3IgdXNlZnVsIHRpbWluZ3MsIGRvOgogICAg IHN1ZG8gc2ggLWMgJ2VjaG8gMyA+IC9wcm9jL3N5cy92bS9kcm9wX2NhY2hlcycgICovCiAgZ2V0 dGltZW9mZGF5ICgmc3RhcnQsIE5VTEwpOwogIERJUiAqbGlua3MgPSBvcGVuZGlyICgiL2dudS9z dG9yZS8ubGlua3MiKTsKCiAgc2l6ZV90IGNvdW50ID0gMDsKCiNpZiBNT0RFICE9IFNUQVRfSU5U RVJMRUFWRUQKICB2b2lkIHNvcnRfZW50cmllcyAodm9pZCkKICB7CiAgICBpbnQgZW50cnlfbG93 ZXIgKGNvbnN0IHZvaWQgKmEsIGNvbnN0IHZvaWQgKmIpCiAgICB7CiAgICAgIHJldHVybiAoKHN0 cnVjdCBlbnRyeSAqKWEpLT5pbm9kZSA8ICgoc3RydWN0IGVudHJ5ICopYiktPmlub2RlOwogICAg fQoKICAgIHFzb3J0IChkaXJfZW50cmllcywgY291bnQsIHNpemVvZiAoc3RydWN0IGVudHJ5KSwK CSAgIGVudHJ5X2xvd2VyKTsKICB9CiNlbmRpZgoKICB2b2lkIHN0YXRfZW50cmllcyAodm9pZCkK ICB7CiAgICBmb3IgKHNpemVfdCBpID0gMDsgaSA8IGNvdW50OyBpKyspCiAgICAgIHsKCXN0cnVj dCBzdGF0eCBzdDsKCXN0YXR4KEFUX0ZEQ1dELCBkaXJfZW50cmllc1tpXS5uYW1lLAogICAgICAg ICAgICAgIEFUX1NZTUxJTktfTk9GT0xMT1cgfCBBVF9TVEFUWF9ET05UX1NZTkMsCiAgICAgICAg ICAgICAgU1RBVFhfU0laRSB8IFNUQVRYX05MSU5LLCAmc3QpOwoJLy9sc3RhdCAoZGlyX2VudHJp ZXNbaV0ubmFtZSwgJnN0KTsKICAgICAgfQogIH0KCiAgZm9yIChzdHJ1Y3QgZGlyZW50ICplbnRy eSA9IHJlYWRkaXIgKGxpbmtzKTsKICAgICAgIGVudHJ5ICE9IE5VTEw7CiAgICAgICBlbnRyeSA9 IHJlYWRkaXIgKGxpbmtzKSkKICAgIHsKICAgICAgYXNzZXJ0IChjb3VudCA8IE1BWF9FTlRSSUVT KTsKICAgICAgZGlyX2VudHJpZXNbY291bnRdLm5hbWUgPSBzdHJkdXAgKGVudHJ5LT5kX25hbWUp OwogICAgICBkaXJfZW50cmllc1tjb3VudF0uaW5vZGUgPSBlbnRyeS0+ZF9pbm87CiNpZiBNT0RF ID09IFNUQVRfSU5URVJMRUFWRUQKICAgICAgc3RydWN0IHN0YXR4IHN0OwogICAgICBzdGF0eChB VF9GRENXRCwgZW50cnktPmRfbmFtZSwKICAgICAgICAgICAgQVRfU1lNTElOS19OT0ZPTExPVyB8 IEFUX1NUQVRYX0RPTlRfU1lOQywgU1RBVFhfU0laRSB8IFNUQVRYX05MSU5LLCAmc3QpOwoKICAg ICAgLy9sc3RhdCAoZW50cnktPmRfbmFtZSwgJnN0KTsKI2VuZGlmCgojaWYgTU9ERSA9PSBTVEFU X1NFTUlfSU5URVJMRUFWRUQKICAgICAgaWYgKGNvdW50KysgPj0gMTAwMDAwKQoJewoJICBzb3J0 X2VudHJpZXMgKCk7CgkgIHN0YXRfZW50cmllcyAoKTsKCSAgY291bnQgPSAwOwoJfQojZWxzZQog ICAgICBjb3VudCsrOwojZW5kaWYKICAgIH0KCiNpZiBNT0RFID09IFNUQVRfU0VNSV9JTlRFUkxF QVZFRAogIHNvcnRfZW50cmllcyAoKTsKICBzdGF0X2VudHJpZXMgKCk7CiNlbmRpZgoKICBnZXR0 aW1lb2ZkYXkgKCZlbmQsIE5VTEwpOwogIHByaW50ZiAoIiV6aSBkaXJfZW50cmllcywgJXppIHNl Y29uZHMiCiNpZiBNT0RFICE9IFNUQVRfT1BUSU1BTAoJICAiIChpbmNsdWRpbmcgc3RhdCkiCiNl bmRpZgoJICAiXG4iLCBjb3VudCwKCSAgZW5kLnR2X3NlYyAtIHN0YXJ0LnR2X3NlYyk7CgojaWYg TU9ERSA9PSBTVEFUX09QVElNQUwKICBzb3J0X2VudHJpZXMgKCk7CiAgZ2V0dGltZW9mZGF5ICgm c3RhcnQsIE5VTEwpOwogIHN0YXRfZW50cmllcyAoKTsKICBnZXR0aW1lb2ZkYXkgKCZlbmQsIE5V TEwpOwoKICBwcmludGYgKCJzdGF0IHRvb2sgJXppIHNlY29uZHNcbiIsIGVuZC50dl9zZWMgLSBz dGFydC50dl9zZWMpOwojZW5kaWYKCiAgcmV0dXJuIEVYSVRfU1VDQ0VTUzsKfQo= --=-=-=--