From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:1008:1e59::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id 0IE+FVxLoWVCmQAAkFu2QA (envelope-from ) for ; Fri, 12 Jan 2024 15:23:24 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id uN3uEVxLoWWQZAAA62LTzQ (envelope-from ) for ; Fri, 12 Jan 2024 15:23:24 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=ZJ+Tyfis; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1705069404; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=JNydbxIjPmQqI3QLFUAB6uyyUwBUF7mM/yQwyX9GPdA=; b=agSWLlpLuWwKC62xOAQkiZ65noOTnM2E86NUuvLpVPDMvS7j220zOUOLItXhTzU5KDKu9e U/z4OPucCumMfU/CYqevTRlQ3Od6Cmges3wCnbhCH9Kbq/ehpKc5/VkbnvxsUqrkXVFori diBMivTY0+4yHUsNPwNWWIxnbu5zpyYcQ1hM+/zfKY45AvTSF6mbFv94Nhb4IP18tU/oBh RCPSIRLNPXu0aYCJ8GVuGWkDootNJWX8EjxPFGUXp6uUR9TtbXAFIgDokCKTG7xjpnusYf 9Hyu5KhWmdZ6igq+zPEX6KpQf3TKaCq6TrcabR1QFLJFlH2GmoIk0uEc9iMfwQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=ZJ+Tyfis; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1705069404; a=rsa-sha256; cv=none; b=amQUUyiWY7Jv4CPQ7Vk83Aem1i+AqxRAsWdbVyVlKCVfE2TzOWCV9po59DWzSOBi4YrEwF 6637YI22LemVGkdORPaYOxWXSoH5eyz/hs4xJAhXbUakTniGEUIrpkpfdi9jV/8MH05Ocg hGbrdr2MG74sop0IoF7LtSUCP8AWRhbqWIB2t3C6ega08NdfPerFuyoOTDTq+RCrqENBdX YchkcDXvt7pTsft8S+1S+lOOoChWPFin9C1HGO5Vf+rKJuCHpXSw6lY9XPZNYZEBDPCTzV LniqT3grZnqmnb9HZAtPcbd4miNJ09tKeIoaqepTPQFUytjsKp2YOKChiNgjLA== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 354F714C8D for ; Fri, 12 Jan 2024 15:23:24 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rOIR8-0005gE-Rx; Fri, 12 Jan 2024 09:23:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rOIR6-0005da-WB for guix-patches@gnu.org; Fri, 12 Jan 2024 09:23:05 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rOIR6-0007fm-NE for guix-patches@gnu.org; Fri, 12 Jan 2024 09:23:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rOIR3-0005xA-Ps for guix-patches@gnu.org; Fri, 12 Jan 2024 09:23:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#68266] [PATCH v2] guix: store: Add report-object-cache-duplication. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 12 Jan 2024 14:23:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 68266 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Christopher Baines Cc: Josselin Poiret , Tobias Geerinckx-Rice , Simon Tournier , Mathieu Othacehe , 68266@debbugs.gnu.org, Ricardo Wurmus , Christopher Baines Received: via spool by 68266-submit@debbugs.gnu.org id=B68266.170506936022833 (code B ref 68266); Fri, 12 Jan 2024 14:23:01 +0000 Received: (at 68266) by debbugs.gnu.org; 12 Jan 2024 14:22:40 +0000 Received: from localhost ([127.0.0.1]:35260 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOIQh-0005wC-GQ for submit@debbugs.gnu.org; Fri, 12 Jan 2024 09:22:39 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:39110) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOIQe-0005vz-RB for 68266@debbugs.gnu.org; Fri, 12 Jan 2024 09:22:38 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rOIQa-0007Lu-Dm; Fri, 12 Jan 2024 09:22:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=JNydbxIjPmQqI3QLFUAB6uyyUwBUF7mM/yQwyX9GPdA=; b=ZJ+TyfisP9vav6sXWc7F ex67mHDaA8GYKqDiiRH3T0HDaR/fnfU7S7cpgJb0Psi3Hx6CPlRYRwuIR1DPuzqvpINlJBNeTsdrU bW2HgLKgJZXHP7rdLsqRCZQdpecHZQS0HJ960LhLkPBo3xOnFVU44yFpk/ZH8SVPhTXnMgc5rOVV9 A1snS3LMSTrPYzfyVzaEQI5M6Cr1RttrbKp0cr0HXYJ4yuElohhLysRhOjWjpGrhKiDpmKEmOAHl2 KQWou8xdDAH3tim0V59d550nChU8Lgsdbw0GVjdwg8O3PHdD+IM3tDiCiZcFCo8xj7F1Bi5kYQAKq kg3V8MqYAxI44Q==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= In-Reply-To: <89c875f974d1ad81ddd03f664ef08e397771d224.1704891443.git.mail@cbaines.net> (Christopher Baines's message of "Wed, 10 Jan 2024 12:57:23 +0000") References: <87plyfrb2x.fsf@cbaines.net> <89c875f974d1ad81ddd03f664ef08e397771d224.1704891443.git.mail@cbaines.net> Date: Fri, 12 Jan 2024 15:22:07 +0100 Message-ID: <871qamk4xc.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Spam-Score: -5.89 X-Migadu-Queue-Id: 354F714C8D X-Migadu-Scanner: mx13.migadu.com X-Migadu-Spam-Score: -5.89 X-TUID: JGnotBwbI6G+ Christopher Baines skribis: > This is intended to help with spotting duplication in the object cache, so > where many keys, for example package records map to the same derivation. = This > represents an opportunity for improved performance if you can reduce this > duplication in the cache, and better take advantage of the already present > cache entries. Another way to detect this is by looking at =E2=80=98add-data-to-store-cach= e=E2=80=99 stats: --8<---------------cut here---------------start------------->8--- $ GUIX_PROFILING=3D"add-data-to-store-cache object-cache" guix build --no-g= rafts greetd wlgreet du-dust circtools --target=3Daarch64-linux-gnu -d=20 /gnu/store/mivzv83wryv9gp5bjncg5m1831dx2xwr-circtools-1.0.0.drv /gnu/store/4xf7kh9mi0vpvs8m1ak4x8w1rpsdpv6z-du-dust-0.8.6.drv /gnu/store/ayk54gvlbc1qam6irzf9kaig56dhzni0-wlgreet-0.4.1.drv /gnu/store/r601i40cii9ic5w1k4hy5c2yngfayh64-greetd-0.9.0.drv Object Cache: fresh caches: 22 lookups: 40435 hits: 36821 (91.1%) cache size: 3613 entries 'add-data-to-store' cache: lookups: 4090 hits: 958 (23.4%) .drv files: 3062 (74.9%) Scheme files: 916 (22.4%) $ GUIX_PROFILING=3D"add-data-to-store-cache object-cache" ./pre-inst-env gu= ix build --no-grafts greetd wlgreet du-dust circtools --target=3Daarch64-li= nux-gnu -d=20 /gnu/store/1wsldmvigjb8w2gk418npbnfznlb0ck1-circtools-1.0.0.drv /gnu/store/b5c73fawjdvkgy431qxz9l6l9y9a9lhz-du-dust-0.8.6.drv /gnu/store/zwc7qzsbzf62dgbbzy74lki4hsr406bw-wlgreet-0.4.1.drv /gnu/store/vjdd23hc82701afb132z1ajcqa7hfd74-greetd-0.9.0.drv Object Cache: fresh caches: 22 lookups: 37942 hits: 34523 (91.0%) cache size: 3418 entries 'add-data-to-store' cache: lookups: 3895 hits: 763 (19.6%) .drv files: 2957 (75.9%) Scheme files: 826 (21.2%) --8<---------------cut here---------------end--------------->8--- Ideally, the hit rate there would be 0% and we could remove it. If there=E2=80=99s a positive hit rate, it means we keep adding the same .d= rv and/or *-builder files to the store, meaning that the object cache was ineffective. > +(define* (report-object-cache-duplication store #:key (threshold 10) > + (port (current-error-port))) Do you have an example output of this? How helpful does it look to you in practice? I guess it we=E2=80=99d need to add a =E2=80=98register-profiling-hook!=E2= =80=99 call somewhere? If we were to add this, I=E2=80=99d suggest adding a docstring and followin= g the same style as the rest of the code (no long identifiers for local variables and no car/cdr, for example). Thanks, Ludo=E2=80=99.