From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:700:3204::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id gDQMIVKLoWUkTgEAkFu2QA (envelope-from ) for ; Fri, 12 Jan 2024 19:56:18 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id +BPEG1KLoWV9WwEAe85BDQ (envelope-from ) for ; Fri, 12 Jan 2024 19:56:18 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1705085778; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=4h6JbLGC2TqGnackkOjeVbyboxqo6+2cpevNL/VwKhw=; b=JkBnVqHNAjikx6xtF5+0E7p5RKrWpzFDaGml1M46MJlCWcmhP+E03tDd+2yjUIR1nW60gl 4RdqhSuD6G2xOKwCEQ8gLjeyNVuTL/yTEczUPOUfoDZpghXJclRyFT+8o6Bc8KTkemmE7x lob2WzC/sXYebUC05cCn9rwo6qK6rYe1YZEI2QzzFQero2TW9q/yYELoAbgg/nCGKu/99G bbuSoNBYJ3qb866rjI63VR0X7Fs6CHyoxZlDITLn7LBcYslRcdvRXCz+hrocn0QAyunDFs uSU8HJsloK9AZSEcHEMBsa/pgphKruP/dMd+xn9PnzOSkgA23bHP/UvBHhtSvA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1705085778; a=rsa-sha256; cv=none; b=Yx0nxmOpm2bFoVUjuYv+YrRz0DklyYls64CbH0hGgpPYNqiPc5+Y1LQ7goiBIR3yMAWEl+ DKrtIFyhjS7U3zPjp7uCwXNPWofDVFZ2XbxTSat0dqQi8Nl5eRab8/RSNHCbehxaPlCNko /hoCeaotKZENSyN/NNkqFrPFkajIZm3wVzBk4+ioQwFDNsNeUDDgO6rHXo7sMQuA+jc6bG I4nOBy3sWsAxNK9kLldBr0AFuQJVsnjC90GyuYScsDMPy5hu98SAwfRZ+6WmLbOmRJSM1+ mgrkh2R+dWTou+EMD+MKh/29Pfojg8ZrN9d952/4miFcQYRuhhTd+p7UELH3mA== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2A5A16181C for ; Fri, 12 Jan 2024 19:56:18 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rOMhK-0005ay-Fv; Fri, 12 Jan 2024 13:56:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rOMhI-0005al-S5 for guix-patches@gnu.org; Fri, 12 Jan 2024 13:56:04 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rOMhI-0008LS-Id for guix-patches@gnu.org; Fri, 12 Jan 2024 13:56:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rOMhG-0003us-Ku for guix-patches@gnu.org; Fri, 12 Jan 2024 13:56:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#68266] [PATCH v2] guix: store: Add report-object-cache-duplication. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 12 Jan 2024 18:56:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 68266 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Josselin Poiret , Tobias Geerinckx-Rice , Simon Tournier , Mathieu Othacehe , 68266@debbugs.gnu.org, Ricardo Wurmus , Christopher Baines Received: via spool by 68266-submit@debbugs.gnu.org id=B68266.170508572114989 (code B ref 68266); Fri, 12 Jan 2024 18:56:02 +0000 Received: (at 68266) by debbugs.gnu.org; 12 Jan 2024 18:55:21 +0000 Received: from localhost ([127.0.0.1]:37770 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOMga-0003th-Qy for submit@debbugs.gnu.org; Fri, 12 Jan 2024 13:55:21 -0500 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:51301) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOMgY-0003tY-HE for 68266@debbugs.gnu.org; Fri, 12 Jan 2024 13:55:19 -0500 Received: from localhost (unknown [217.155.61.229]) by mira.cbaines.net (Postfix) with ESMTPSA id 5F5A827BBE2; Fri, 12 Jan 2024 18:55:19 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 61a53281; Fri, 12 Jan 2024 18:55:18 +0000 (UTC) References: <87plyfrb2x.fsf@cbaines.net> <89c875f974d1ad81ddd03f664ef08e397771d224.1704891443.git.mail@cbaines.net> <871qamk4xc.fsf@gnu.org> User-agent: mu4e 1.10.7; emacs 29.1 From: Christopher Baines Date: Fri, 12 Jan 2024 18:26:24 +0000 In-reply-to: <871qamk4xc.fsf@gnu.org> Message-ID: <87v87yfkkr.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -7.30 X-Spam-Score: -7.30 X-Migadu-Queue-Id: 2A5A16181C X-Migadu-Scanner: mx12.migadu.com X-TUID: SuanxOKWKMah --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Christopher Baines skribis: > >> This is intended to help with spotting duplication in the object cache, = so >> where many keys, for example package records map to the same derivation.= This >> represents an opportunity for improved performance if you can reduce this >> duplication in the cache, and better take advantage of the already prese= nt >> cache entries. > > Another way to detect this is by looking at =E2=80=98add-data-to-store-ca= che=E2=80=99 > stats: > > $ GUIX_PROFILING=3D"add-data-to-store-cache object-cache" guix build --no= -grafts greetd wlgreet du-dust circtools --target=3Daarch64-linux-gnu -d=20 > /gnu/store/mivzv83wryv9gp5bjncg5m1831dx2xwr-circtools-1.0.0.drv > /gnu/store/4xf7kh9mi0vpvs8m1ak4x8w1rpsdpv6z-du-dust-0.8.6.drv > /gnu/store/ayk54gvlbc1qam6irzf9kaig56dhzni0-wlgreet-0.4.1.drv > /gnu/store/r601i40cii9ic5w1k4hy5c2yngfayh64-greetd-0.9.0.drv > Object Cache: > fresh caches: 22 > lookups: 40435 > hits: 36821 (91.1%) > cache size: 3613 entries > > 'add-data-to-store' cache: > lookups: 4090 > hits: 958 (23.4%) > .drv files: 3062 (74.9%) > Scheme files: 916 (22.4%) > $ GUIX_PROFILING=3D"add-data-to-store-cache object-cache" ./pre-inst-env = guix build --no-grafts greetd wlgreet du-dust circtools --target=3Daarch64-= linux-gnu -d=20 > /gnu/store/1wsldmvigjb8w2gk418npbnfznlb0ck1-circtools-1.0.0.drv > /gnu/store/b5c73fawjdvkgy431qxz9l6l9y9a9lhz-du-dust-0.8.6.drv > /gnu/store/zwc7qzsbzf62dgbbzy74lki4hsr406bw-wlgreet-0.4.1.drv > /gnu/store/vjdd23hc82701afb132z1ajcqa7hfd74-greetd-0.9.0.drv > Object Cache: > fresh caches: 22 > lookups: 37942 > hits: 34523 (91.0%) > cache size: 3418 entries > > 'add-data-to-store' cache: > lookups: 3895 > hits: 763 (19.6%) > .drv files: 2957 (75.9%) > Scheme files: 826 (21.2%) > > Ideally, the hit rate there would be 0% and we could remove it. > > If there=E2=80=99s a positive hit rate, it means we keep adding the same = .drv > and/or *-builder files to the store, meaning that the object cache was > ineffective. > >> +(define* (report-object-cache-duplication store #:key (threshold 10) >> + (port (current-error-port))) > > Do you have an example output of this? > > How helpful does it look to you in practice? Yep, so here's some output I get from computing all the cross derivations to i586-pc-gnu: value # /gnu/store/92avphfdcrcaxx8m5a6ihmw558bj3np8-subversion-1.14.2 7fcfbe5f9= 280> cached 4174 times example keys: - # - # - # "/bin/svn"> - # "/bin/svn"> So it's not immediately obvious what the issue is, but if you search for "/bin/svn", then you find the file-append calls in svn-fetch, so there's a couple of new file-append records added to the cache for each svn-fetch. value /gnu/store/qi1km3qlv5hdsql6h3ibwvz6v4z8lqbz-ld-wrapper.in cached 75= 5 times example keys: - #< file: "/home/chris/Projects/Guix/guix/gnu/packages/ld-= wrapper.in" absolute: # name: "ld-wrapper.in" recursive?: #t select?: #> - #< file: "/home/chris/Projects/Guix/guix/gnu/packages/ld-= wrapper.in" absolute: # name: "ld-wrapper.in" recursive?: #t select?: #> - #< file: "/home/chris/Projects/Guix/guix/gnu/packages/ld-= wrapper.in" absolute: # name: "ld-wrapper.in" recursive?: #t select?: #> - #< file: "/home/chris/Projects/Guix/guix/gnu/packages/ld-= wrapper.in" absolute: # name: "ld-wrapper.in" recursive?: #t select?: #> Similar to file-append, it's not immediately obvious where all these local-file's are coming from, but searching for ld-wrapper.in suggests make-ld-wrapper. My thinking here is that maybe it's worth not caching everything in the object cache. Particularly for file-append, I'm not sure what the cache is actually doing. As for local-file, given there's lower level caching, maybe it's worth leaning on that and not caching local-file either. I tried this out, and it seemed to not make performance worse at least, and it did remove the duplication from the cache. Other things that show up include: value # /gnu/store/lkmwq399jllig2a5r323v6y9nflpn6gn-= binutils-cross-i586-pc-gnu-2.38 7fcfbd358000> cached 1228 times example keys: - # - # - # - # --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmWhixRfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XcrGxAAr+CZGwnbA4IEkRH6kmHCBa1UYeT5mVat FvOKIINgMqv7es0DMBEPy1g9Yi223BpBEHlqSloE7HUANk3gq/MASNJJd/Aeh6tR UjRzzWlG9GfuS3QH8u2gmMSPRElCsd6v9OY9WhaFQvw4Zq/EO45jmEw/QbvMzrHW 3K3w/nG0ss+tT1dYB8U+VWgFpzxg+XlAit5QVR1vW0B0r8zuAJ8SLe11iikx3ZmO X7m/kf6uC4WQU0ptIPVu84oxAjDckmuPMahA4pw+QIY2Ne2ud/fL3rcbUA7d91Ha 3sbdcfO1wClQNJLCsROEEiYUW0+hrfTHZJhlKGHdm4rAP9k6yx4cdiVhNRhu7fwE HoDmnqreW4Hji4Tgbi+hLu9Bv99l2Y0HJ9KdjKD0WmOzuWkNhhP4WrlOqEW1NWHm p72CI4zjMXzCOFosdWqCgTw0nuDoXdb4AMbKgegCbddPCizXcnH5n9upRBfhNSA1 VCl1wr/ZhLtT8JEAJp34YXUESWZSKne+pzVHW8F1oHG9H/Uogx5FP2RvSi4OYjhp 3RrzBdIjyMKfQmlp3zQmrq5OrEXXv4WGflKlrZubRCgV4Z1R4O4eK0Q/TDNhbLmY nvxz9tboQUW1xJAvDbXV/BKZgEf6AjdKEy/Qk3nMUXSx9TFb5BnQ9vBed1B5WTld h+C2I0Q5Qhk= =pIg/ -----END PGP SIGNATURE----- --=-=-=--