From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id KNmqK6yzlGIEBwEAbAwnHQ (envelope-from ) for ; Mon, 30 May 2022 14:08:12 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id kLylK6yzlGLkPQAA9RJhRA (envelope-from ) for ; Mon, 30 May 2022 14:08:12 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3761921685 for ; Mon, 30 May 2022 14:08:12 +0200 (CEST) Received: from localhost ([::1]:37790 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nveBu-0003nu-MK for larch@yhetil.org; Mon, 30 May 2022 08:08:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56798) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nvdgo-0006KJ-Lu for bug-guix@gnu.org; Mon, 30 May 2022 07:36:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:49057) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nvdgo-0002sa-Bf for bug-guix@gnu.org; Mon, 30 May 2022 07:36:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nvdgo-0003oP-73 for bug-guix@gnu.org; Mon, 30 May 2022 07:36:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#55721: Low cache usage in the presence of grafts Resent-From: Efraim Flashner Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 30 May 2022 11:36:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 55721 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 55721@debbugs.gnu.org Received: via spool by 55721-submit@debbugs.gnu.org id=B55721.165391053214593 (code B ref 55721); Mon, 30 May 2022 11:36:02 +0000 Received: (at 55721) by debbugs.gnu.org; 30 May 2022 11:35:32 +0000 Received: from localhost ([127.0.0.1]:42954 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvdgJ-0003nJ-Cv for submit@debbugs.gnu.org; Mon, 30 May 2022 07:35:31 -0400 Received: from flashner.co.il ([178.62.234.194]:51146) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nvdgE-0003mu-Kj for 55721@debbugs.gnu.org; Mon, 30 May 2022 07:35:30 -0400 Received: from localhost (unknown [141.226.14.232]) by flashner.co.il (Postfix) with ESMTPSA id A3547401C5; Mon, 30 May 2022 11:35:20 +0000 (UTC) Date: Mon, 30 May 2022 14:34:40 +0300 From: Efraim Flashner Message-ID: Mail-Followup-To: Efraim Flashner , Ludovic =?UTF-8?Q?Court=C3=A8s?= , 55721@debbugs.gnu.org References: <87y1yj5s1p.fsf@inria.fr> <87pmjv5pto.fsf@gnu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="el+JekaWj0IAG2lS" Content-Disposition: inline In-Reply-To: <87pmjv5pto.fsf@gnu.org> X-PGP-Key-ID: 0x41AAE7DCCA3D8351 X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc X-PGP-Fingerprint: A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1653912492; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=EIwOOHXafc+QKaiQ/ZNloCP3m6ke5xheidDOc8Ch+cc=; b=M7ZSQ59HqOtokXY2qHk3BJm2M5yviln2jN8NdIwREG4H8jUKVjacQCtSV3Q10GCa7mXsne gfYGHST5K9qvZh/eQCfF8wpR0cm2CDu0nAud7bbRgGN1+/NKY45l+hwWRT5p61nnTpRgDD p1/xDlyZdmSXVmuQHBiPHWyr9VxPhIyLkdAbQ4l0CZ8X8dOLqxzPAxtakz8ErqGPP+xcG3 JfB1agOc+TW5WEdjq+VBKTay6cE1yAX7JzrAhd0e40wkuT0iXODFvxakghPB+NQhkjLmAM t/DvTSrPqrcErH87nsqBkfBq+Ho9DdbQ9Pv6QbT3gbZgTPz9XzDgGZlWK6mjHg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1653912492; a=rsa-sha256; cv=none; b=HofgZSB0l1YRxAIxGLFBSkq8zmqvSjRtEgf6qamxE2m1BM836StEl6cnsB1MF2EJhjNsUu 6J12+tm/ozSASZdQFkcTijECdMixsvlMPVLwNwmnL+bPvxj/HtvTvlQ5l8pPnY/gpL7kA5 JOiLoKJ65lR2sfbFnWLGL7z4ry6lAf3QUIWQSaGwUiCgoIBGLqu3gZDeR7/NG4dvVx48ja jdFsDb+cSVm+xqLF8ytDvIs3bcN2Cj6z0q6uQlKR8dWFrDFdU19YEcw4mTdvZRh8S6ozQH 22Oz/suH3BVdtyokVjtFq0szrMmwVDR/J4mWga59DrmQtHZ01Nw0meEGEjdOlQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.73 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 3761921685 X-Spam-Score: -5.73 X-Migadu-Scanner: scn0.migadu.com X-TUID: 40CtkVmHnZ9O --el+JekaWj0IAG2lS Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 30, 2022 at 11:31:47AM +0200, Ludovic Court=C3=A8s wrote: > Ludovic Court=C3=A8s skribis: >=20 > > $ time GUIX_PROFILING=3D"rpc reference-cache derivation-graft-cache" ./= pre-inst-env guix system vm gnu/system/examples/desktop.tmpl=20 > > 0.2 MB will be downloaded > > polkit-0.121 198KiB = 1.4MiB/s 00:00 [##################] 100.0% > > /gnu/store/sk1571rnh5jl2ilp0v1k8bqwsdrngwxy-run-vm.sh > > Reference Cache: > > fresh caches: 8 > > lookups: 77581 > > hits: 52890 (68.2%) > > cache size: 838 entries > > Derivation graft cache: > > fresh caches: 39 > > lookups: 704635 > > hits: 627819 (89.1%) > > cache size: 2946 entries > > Remote procedure call summary: 33071 RPCs > > built-in-builders ... 1 > > query-substitutable-path-infos ... 2 > > build-things ... 2 > > add-to-store/tree ... 56 > > valid-path? ... 59 > > add-to-store ... 373 > > add-text-to-store ... 7887 > > query-references ... 24691 > > > > real 5m6.727s > > user 6m59.438s > > sys 0m1.633s >=20 > With this patch: >=20 > diff --git a/guix/store.scm b/guix/store.scm > index efba07bdcd..88cab1bb6a 100644 > --- a/guix/store.scm > +++ b/guix/store.scm > @@ -1337,7 +1337,12 @@ (define (build-accumulator expected-store) > (if (and (eq? (store-connection-socket store) > (store-connection-socket expected-store)) > (=3D mode (build-mode normal))) > - (unresolved things continue) > + (unresolved things > + (lambda (new-store value) > + ;; Borrow caches from NEW-STORE. > + (set-store-connection-caches! > + store (store-connection-caches new-store)) > + (continue value))) > (continue #t)))) > =20 > (define default-cutoff > @@ -1397,7 +1402,8 @@ (define-values (result rest) > (if (unresolved? obj) > ;; Pass #f because 'build-thin= gs' is now > ;; unnecessary. > - ((unresolved-continuation obj)= #f) > + ((unresolved-continuation obj) > + store #f) > obj)) > result #:cutoff cutoff) > (map/accumulate-builds store proc rest #:cutoff cutoff))))) >=20 > =E2=80=A6 we=E2=80=99re doing better: >=20 > --8<---------------cut here---------------start------------->8--- > $ guix gc -D $(guix build polkit --no-grafts) > finding garbage collector roots... > [0 MiB] deleting '/gnu/store/8vsfk8312m5p461wl4dq5lxjagfbwfhz-polkit-0.12= 1' > deleting `/gnu/store/trash' > deleting unused links... > note: currently hard linking saves 72397.13 MiB > $ time GUIX_PROFILING=3D"rpc reference-cache derivation-graft-cache" ./pr= e-inst-env guix system vm gnu/system/examples/desktop.tmpl=20 > 0.2 MB will be downloaded > polkit-0.121 198KiB = 1.6MiB/s 00:00 [##################] 100.0% > /gnu/store/wp3lv4xrh6vw79gnkyi5471c1l8j624n-run-vm.sh > Reference Cache: > fresh caches: 8 > lookups: 17232 > hits: 14816 (86.0%) > cache size: 784 entries > Derivation graft cache: > fresh caches: 39 > lookups: 128402 > hits: 111336 (86.7%) > cache size: 328 entries > Remote procedure call summary: 10796 RPCs > built-in-builders ... 1 > query-substitutable-path-infos ... 2 > build-things ... 2 > add-to-store/tree ... 56 > valid-path? ... 59 > add-to-store ... 373 > query-references ... 2416 > add-text-to-store ... 7887 >=20 > real 1m31.550s > user 2m7.107s > sys 0m0.637s > --8<---------------cut here---------------end--------------->8--- >=20 > There=E2=80=99s still room for improvement though. >=20 > Ludo=E2=80=99. With that change you've dropped the query-references from 24691 to 2416, more than a 90% drop. If you want to take aim at improving it further I'd say take a look at add-text-to-store and see if you can shrink that number. --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --el+JekaWj0IAG2lS Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmKUq80ACgkQQarn3Mo9 g1GdUQ/+N9nCWMi99+De+8gC7kx0N9TfrtbVt0W6vNuzuhc2g5CjDV0/Wm6foo6W 08Mn7bqwMdXJZ4zQshYq5WTKo38Yeie5C7xQ6dFp8cyrWMkicBPhig/XGfo3Whto A3BZ2xOtmpDclUlpwaF4GOxXbcgAEPxLutAjtZYsK4SJdxysAnKEbcWmDQqhFp4G wNZH0ubpnIp2LODdBDi9iL8RmrhUNIhIbFucNYM+QKUGM0A5717vNjS4p/BLPkGu Bw4uqPutweFuvVushzobTctLBN4C9jaizsneukKwBqeZyQSzWqk0j9b1UuUg+3Ev 8bqBL4l0+Jpw/KNl37MNAQ8WZ5gjp92NxfWNeMUDdEweN2eN0sFlEjcDLzVAK5Uc 99+6L4ZsNdlt6cpmmS8zI5pdEvtQCVl9uIzXjXYvexkOjk0cbElXctNv6sPSPhpg HvImJbQTqyGCafQdBaS8V4L5xp7c2TalGooCDgA4nb+vKLy8o5v/36sGA6OjcNp4 uLVV5uzI+U4Ts3UrcHO59/hFy8ZHpE0FKkFzD1PChIIFa5Yt3sMB+d9xjRFsW8pg +iwL85Hgc0DbPhYr9YlB3Onmzj1N01xikLbX9fgJ/2hKNjADxb3c7bqX+nUOBQIa /nvX/GZJlSq0g20lHiPAkRo3w0P1iTBRZxzSs7A5YnfskaqePAk= =f62t -----END PGP SIGNATURE----- --el+JekaWj0IAG2lS--