From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id MD0VBT2GuGCLVQAAgWs5BA (envelope-from ) for ; Thu, 03 Jun 2021 09:35:25 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id qE4YAD2GuGCBMAAAbx9fmQ (envelope-from ) for ; Thu, 03 Jun 2021 07:35:25 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8C1771D487 for ; Thu, 3 Jun 2021 09:35:24 +0200 (CEST) Received: from localhost ([::1]:46930 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lohsx-00050U-Mo for larch@yhetil.org; Thu, 03 Jun 2021 03:35:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58854) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lohsf-0004id-Bq for guix-patches@gnu.org; Thu, 03 Jun 2021 03:35:05 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:58715) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lohsf-0001wt-3X for guix-patches@gnu.org; Thu, 03 Jun 2021 03:35:05 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lohsf-0001XE-0j for guix-patches@gnu.org; Thu, 03 Jun 2021 03:35:05 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#48806] [PATCH 7/7] grafts: Cache the derivation/graft mapping for the whole session. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 03 Jun 2021 07:35:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 48806 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 48806@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 48806-submit@debbugs.gnu.org id=B48806.16227056915830 (code B ref 48806); Thu, 03 Jun 2021 07:35:04 +0000 Received: (at 48806) by debbugs.gnu.org; 3 Jun 2021 07:34:51 +0000 Received: from localhost ([127.0.0.1]:42022 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lohsQ-0001Vy-MI for submit@debbugs.gnu.org; Thu, 03 Jun 2021 03:34:50 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36942) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lohsP-0001Uw-HO for 48806@debbugs.gnu.org; Thu, 03 Jun 2021 03:34:49 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:60220) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lohsK-0001br-BW; Thu, 03 Jun 2021 03:34:44 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=57488 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lohsI-0005sA-II; Thu, 03 Jun 2021 03:34:44 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 3 Jun 2021 09:34:01 +0200 Message-Id: <20210603073401.13629-7-ludo@gnu.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210603073401.13629-1-ludo@gnu.org> References: <20210603073401.13629-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1622705724; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=Ee7Gxj+xFwk1R89S3lgpc4E7sSW+aSIi7eBy70Uoguk=; b=aY92CgKD52UsX6R8rVodxWU25ikGiCCMQn06pekTsLl7CwlKp9/Iv+rZKMKVY7XsrikDWn LDPFTazwmWY4kSvdMgNuOIa16JN9GlqRYBbOx5rYhjlCE2pR6q9bnYuA0ZRjRYBKhY3fhQ UDdYqxxgNXqKRNSoPWQTEbmGf75HnHZlWspL+eGFMK4Fi+IzldLmzbrY7h5S0s/nORmJeJ OTMhW6Prjxsb1uLyKiEX/LHkI0sZ6LdKEPYVgKRuaz1oejCulopWCzgkCudI6RSCqKk23j A7t/ry+LWNuahcItDCIQiuMtnwLdNeZBvw05w2sRvqXps5ZVla+BKfOBmfiPXQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1622705724; a=rsa-sha256; cv=none; b=u6DHnqErCUlpg7cAbF2vhI0U6VxYmVmaYnSeF9XzDaccCYV2Vu+b/HROcHyXdsbdzIZXRJ B+yY51OE/RL/YE7/xrmuyrm/n14cxDQlB0XLJ7iRYCR2ibUuDqnvTlS4T/OCvis1y5h/VT n4t7GBcWEH9A6f/Ph7ogrzF7BfDgGmjGas3DwXkl3OloJbUjh9eU+93ZNM8Si6oxM3TI9T reCBBoTrAax8ZbY9bAeiHg2WSd/2cuEY7F0FSrqwJ44d3SEbijgPLY24i7lBgFJjuG791j 9/iauxpEBUFnv1GvE763qthO7+clz5nY0NNJ+QqWMEZ3Cl2rqXoVoGXPFGw7Mg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: 2.07 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 8C1771D487 X-Spam-Score: 2.07 X-Migadu-Scanner: scn0.migadu.com X-TUID: 2pX2d18eMg2B Partly fixes . Reported by Lars-Dominik Braun . Previously, 'graft-derivation' would start anew at every call. When creating a profile with lots of packages, it would potentially do the same work multiple times. The per-session cache addresses this. It increases the derivation-graft-cache hit rate from 77.9% to 80.1% on: GUIX_PROFILING="derivation-graft-cache" ./pre-inst-env \ guix environment --ad-hoc libreoffice inkscape krita darktable -n The effect is more visible on the pathological case below, where cache hit rate goes from 75% to 87% and wall-clock time from 5.0s to 3.5s: GUIX_PROFILING="derivation-graft-cache" ./pre-inst-env \ guix environment --ad-hoc r-learnr --search-paths * guix/grafts.scm (%graft-cache): New variable. (graft-derivation): Add calls to 'store-connection-cache' and 'set-store-connection-cache!'. --- guix/grafts.scm | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/guix/grafts.scm b/guix/grafts.scm index e5672268b1..4c69eb35a2 100644 --- a/guix/grafts.scm +++ b/guix/grafts.scm @@ -172,6 +172,10 @@ references." items)))) (remove (cut member <> self) refs))) +(define %graft-cache + ;; Cache that maps derivation/outputs/grafts tuples to lists of grafts. + (allocate-store-connection-cache 'grafts)) + (define record-cache-lookup! (cache-lookup-recorder "derivation-graft-cache" "Derivation graft cache")) @@ -271,7 +275,7 @@ derivations to the corresponding set of grafts." #:system system))))) (reference-origins drv items))) - (with-cache (cons (derivation-file-name drv) outputs) + (with-cache (list (derivation-file-name drv) outputs grafts) (match (non-self-references store drv outputs) (() ;no dependencies (return grafts)) @@ -309,17 +313,25 @@ derivations to the corresponding set of grafts." "Apply GRAFTS to the OUTPUTS of DRV and all their dependencies, recursively. That is, if GRAFTS apply only indirectly to DRV, graft the dependencies of DRV, and graft DRV itself to refer to those grafted dependencies." - (match (run-with-state - (cumulative-grafts store drv grafts - #:outputs outputs - #:guile guile #:system system) - vlist-null) ;the initial cache - ((first . rest) - ;; If FIRST is not a graft for DRV, it means that GRAFTS are not - ;; applicable to DRV and nothing needs to be done. - (if (equal? drv (graft-origin first)) - (graft-replacement first) - drv)))) + (let ((grafts cache + (run-with-state + (cumulative-grafts store drv grafts + #:outputs outputs + #:guile guile #:system system) + (store-connection-cache store %graft-cache)))) + + ;; Save CACHE in STORE to benefit from it on the next call. + ;; XXX: Ideally we'd use %STORE-MONAD and 'mcached' and avoid mutating + ;; STORE. + (set-store-connection-cache! store %graft-cache cache) + + (match grafts + ((first . rest) + ;; If FIRST is not a graft for DRV, it means that GRAFTS are not + ;; applicable to DRV and nothing needs to be done. + (if (equal? drv (graft-origin first)) + (graft-replacement first) + drv))))) ;; The following might feel more at home in (guix packages) but since (guix -- 2.31.1