From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id AD7EKn/n1WHReAEAgWs5BA (envelope-from ) for ; Wed, 05 Jan 2022 19:46:23 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id UE9lI3/n1WGZGwEAG6o9tA (envelope-from ) for ; Wed, 05 Jan 2022 19:46:23 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E02B02DAAA for ; Wed, 5 Jan 2022 19:46:22 +0100 (CET) Received: from localhost ([::1]:48978 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1n5BIj-0004hn-V8 for larch@yhetil.org; Wed, 05 Jan 2022 13:46:22 -0500 Received: from eggs.gnu.org ([209.51.188.92]:47176) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n5BIT-0004gc-Nf for guix-patches@gnu.org; Wed, 05 Jan 2022 13:46:05 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:58463) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n5BIP-0004DS-NR for guix-patches@gnu.org; Wed, 05 Jan 2022 13:46:05 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1n5BIP-0008W6-N7 for guix-patches@gnu.org; Wed, 05 Jan 2022 13:46:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#53034] [PATCH] shell: Cache profiles even when using package specs. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 05 Jan 2022 18:46:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 53034 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 53034@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.164140834032709 (code B ref -1); Wed, 05 Jan 2022 18:46:01 +0000 Received: (at submit) by debbugs.gnu.org; 5 Jan 2022 18:45:40 +0000 Received: from localhost ([127.0.0.1]:41776 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n5BI4-0008VU-4l for submit@debbugs.gnu.org; Wed, 05 Jan 2022 13:45:40 -0500 Received: from lists.gnu.org ([209.51.188.17]:50228) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n5BI2-0008VL-5B for submit@debbugs.gnu.org; Wed, 05 Jan 2022 13:45:39 -0500 Received: from eggs.gnu.org ([209.51.188.92]:47050) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n5BI1-0004OZ-LW for guix-patches@gnu.org; Wed, 05 Jan 2022 13:45:37 -0500 Received: from [2001:470:142:3::e] (port=49096 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1n5BHx-00049T-Tz; Wed, 05 Jan 2022 13:45:37 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=Q+OH6D10WJ8/bP2nHn6UsLKCfNVY1L4EzeDsHEOL/R4=; b=rgyRh2yjA5nBal nRV4wQgSVZkgD+UuIGvj6SDOpnyIqD9Tbc7lri7t+n2gUQbhvGBdIX5QtGrTXTx1PxAaH8451zT3u Wy6irnvS4KMvfnkB9W8aDjvrjHht1MwNSJ6MZ98quPK9f14LcmsismTUdaGXFegh2WAJDaYgNMLB6 PIIMd5yzE6vviCr62DVjSg8MrUvYT4Cv+keCKRpS0CBxcxr+BwWAzjhlDOUFWPv3Oxeri+dgmoqvJ U5FZNTpP6AzvrfJbY1rLE5Q/Q6qWK112htS8QXgvpRSm1k/1YfAx64WhY5w2521WcAC5yz14Mtkr+ wzYzBO/A/PLJpEcNbTLw==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:52378 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1n5BHZ-0000Yj-PX; Wed, 05 Jan 2022 13:45:33 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Wed, 5 Jan 2022 19:45:00 +0100 Message-Id: <20220105184500.12171-1-ludo@gnu.org> X-Mailer: git-send-email 2.33.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1641408383; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=9h/yC+1o7VLwaMlfXY5WS72PBftza7uA5k8DoSbKPA4=; b=dEO4LLAJ2hMG03ZxHw3V25fCCzlfdfewIIjzeAm8UZiC6+lTVxxD0OZECWrtU6/gicCT52 iy8TqWSwRdV7cWTD/JAb7kyuP9msQX49oEMzJIt9uT6/CNSKUPZxR73spPT8ocUiozHuLz pjjIOcGdZZZvtSs8Um18F2nhkNwPL8cXjwzQ0giEycOdTBDbKH8/Sn+e3eQOfSlufJxB/7 mne+ecGow1nVxMcm51/Z4ECvQJ1lRSbx0ons8kZsUS8FAosHDhmHIOSPKXgd+jIr8FOxlF FW2lSuhZkksBNtWDssMGJwpI/kEfc/ApujXr8HM2OH3fIVmwj5heXOKexe4RKQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1641408383; a=rsa-sha256; cv=none; b=XO0bxVe+ZPCbs5M2gG6qtaZa5czOt+jz207RWniWmbEZxSgnO+tu6enjSXJL+gmd1giYba sbQfORMSwKyGxAwGtnrdWttYGoK7++tmT/ZkQq5u+QtMah96nTUAzojYp6AkBVlAnqlhmh VhP5RL6o1aX20ffytLfttW6bejy7p0IVM15B7HhCmrukqbN3yyt2AE+LkYiojHnPmEU9uD Gb1JUztKVwjSv3bPyGYClaN5LUeR4LvQCgZkrge1qyeMswsWdrOEKaKD/gbm44Qf0sPbzH 06p3B0H2RHe27hPBM9FuGFWQQgxJa3sswh0ivkbCmKdHH939gQA69yE049SdzQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gnu.org header.s=fencepost-gnu-org header.b=rgyRh2yj; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.50 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gnu.org header.s=fencepost-gnu-org header.b=rgyRh2yj; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: E02B02DAAA X-Spam-Score: -1.50 X-Migadu-Scanner: scn1.migadu.com X-TUID: FVvJAYUK+JqX This enables profile caching not just when '-m' or '-f' is used, but also when package specs are passed on the command line, as in: guix shell -D guix git It also changes profile cache keys to include the system type, which was previously ignored. * guix/scripts/shell.scm (options-with-caching)[single-file-for-caching]: Remove. Call 'profile-cached-gc-root' instead; adjust to accept two values. (profile-cache-primary-key): New procedure. (profile-cache-key): Remove. (profile-file-cache-key, profile-spec-cache-key): New procedures. (profile-cached-gc-root): Rewrite to include functionality formally in 'single-file-for-caching', but extend to handle package specs. * gnu/packages.scm (cache-is-authoritative?): Export. * guix/transformations.scm (transformation-option-key?): New procedure. --- gnu/packages.scm | 3 +- guix/scripts/shell.scm | 161 +++++++++++++++++++++++++-------------- guix/transformations.scm | 9 ++- 3 files changed, 113 insertions(+), 60 deletions(-) Hi there! I was frustrated that I would not benefit from the profile cache when running: guix shell -D guix or, more interestingly: guix shell supertuxkart -- supertuxkart This patch fixes that. Thus, on cache hits, any such command runs in ~0.1s. Well, supertuxkart might run in more like ~1h, but that’s another story. There are still cases where caching is disabled: when the package cache is not authoritative (i.e., -L is used or ‘GUIX_PACKAGE_PATH’ is set), when transformation options are used (because options such as ‘--with-branch’ depend on external state that may change behind our back), and when using ‘-e’ (because we don’t know what the given expression is doing). Thoughts? Ludo’. diff --git a/gnu/packages.scm b/gnu/packages.scm index ccfc83dd11..65ab7a7c1e 100644 --- a/gnu/packages.scm +++ b/gnu/packages.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2012-2020, 2022 Ludovic Courtès ;;; Copyright © 2013 Mark H Weaver ;;; Copyright © 2014 Eric Bavier ;;; Copyright © 2016, 2017 Alex Kost @@ -51,6 +51,7 @@ (define-module (gnu packages) %auxiliary-files-path %package-module-path %default-package-module-path + cache-is-authoritative? fold-packages fold-available-packages diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm index 546639818f..12b6f18200 100644 --- a/guix/scripts/shell.scm +++ b/guix/scripts/shell.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2021 Ludovic Courtès +;;; Copyright © 2021-2022 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -21,7 +21,8 @@ (define-module (guix scripts shell) #:use-module ((guix diagnostics) #:select (location)) #:use-module (guix scripts environment) #:autoload (guix scripts build) (show-build-options-help) - #:autoload (guix transformations) (show-transformation-options-help) + #:autoload (guix transformations) (transformation-option-key? + show-transformation-options-help) #:use-module (guix scripts) #:use-module (guix packages) #:use-module (guix profiles) @@ -40,6 +41,7 @@ (define-module (guix scripts shell) #:use-module ((guix build utils) #:select (mkdir-p)) #:use-module (guix cache) #:use-module ((ice-9 ftw) #:select (scandir)) + #:autoload (gnu packages) (cache-is-authoritative?) #:export (guix-shell)) (define (show-help) @@ -201,51 +203,35 @@ (define (authorized-shell-directory? directory) (const #f))) (define (options-with-caching opts) - "If OPTS contains exactly one 'load' or one 'manifest' key, automatically -add a 'profile' key (when a profile for that file is already in cache) or a -'gc-root' key (to add the profile to cache)." - (define (single-file-for-caching opts) - (let loop ((opts opts) - (file #f)) - (match opts - (() file) - ((('package . _) . _) #f) - ((('load . ('package candidate)) . rest) - (and (not file) (loop rest candidate))) - ((('manifest . candidate) . rest) - (and (not file) (loop rest candidate))) - ((('expression . _) . _) #f) - ((_ . rest) (loop rest file))))) - - ;; Check whether there's a single 'load' or 'manifest' option. When that is - ;; the case, arrange to automatically cache the resulting profile. - (match (single-file-for-caching opts) - (#f opts) - (file - (let* ((root (profile-cached-gc-root file)) - (stat (and root (false-if-exception (lstat root))))) - (if (and (not (assoc-ref opts 'rebuild-cache?)) - stat - (<= (stat:mtime ((@ (guile) stat) file)) - (stat:mtime stat))) - (let ((now (current-time))) - ;; Update the atime on ROOT to reflect usage. - (utime root - now (stat:mtime stat) 0 (stat:mtimensec stat) - AT_SYMLINK_NOFOLLOW) - (alist-cons 'profile root - (remove (match-lambda - (('load . _) #t) - (('manifest . _) #t) - (_ #f)) - opts))) ;load right away - (if (and root (not (assq-ref opts 'gc-root))) - (begin - (if stat - (delete-file root) - (mkdir-p (dirname root))) - (alist-cons 'gc-root root opts)) - opts)))))) + "If OPTS contains only options that allow us to compute a cache key, +automatically add a 'profile' key (when a profile for that file is already in +cache) or a 'gc-root' key (to add the profile to cache)." + ;; Attempt to compute a file name for use as the cached profile GC root. + (let* ((root timestamp (profile-cached-gc-root opts)) + (stat (and root (false-if-exception (lstat root))))) + (if (and (not (assoc-ref opts 'rebuild-cache?)) + stat + (<= timestamp (stat:mtime stat))) + (let ((now (current-time))) + ;; Update the atime on ROOT to reflect usage. + (utime root + now (stat:mtime stat) 0 (stat:mtimensec stat) + AT_SYMLINK_NOFOLLOW) + (alist-cons 'profile root + (remove (match-lambda + (('load . _) #t) + (('manifest . _) #t) + (('package . _) #t) + (('ad-hoc-package . _) #t) + (_ #f)) + opts))) ;load right away + (if (and root (not (assq-ref opts 'gc-root))) + (begin + (if stat + (delete-file root) + (mkdir-p (dirname root))) + (alist-cons 'gc-root root opts)) + opts)))) (define (auto-detect-manifest opts) "If OPTS do not specify packages or a manifest, load a \"guix.scm\" or @@ -308,28 +294,87 @@ (define %profile-cache-directory (make-parameter (string-append (cache-directory #:ensure? #f) "/profiles"))) -(define (profile-cache-key file) +(define (profile-cache-primary-key) + "Return the \"primary key\" used when computing keys for the profile cache. +Return #f if no such key can be obtained and caching cannot be +performed--e.g., because the package cache is not authoritative." + (and (cache-is-authoritative?) + (match (current-channels) + (() + #f) + (((= channel-commit commits) ...) + (string-join commits))))) + +(define (profile-file-cache-key file system) "Return the cache key for the profile corresponding to FILE, a 'guix.scm' or 'manifest.scm' file, or #f if we lack channel information." - (match (current-channels) - (() #f) - (((= channel-commit commits) ...) + (match (profile-cache-primary-key) + (#f #f) + (primary-key (let ((stat (stat file))) (bytevector->base32-string ;; Since FILE is not canonicalized, only include the device/inode ;; numbers. XXX: In some rare cases involving Btrfs and NFS, this can ;; be insufficient: . (sha256 (string->utf8 - (string-append (string-join commits) ":" + (string-append primary-key ":" system ":" (number->string (stat:dev stat)) ":" (number->string (stat:ino stat)))))))))) -(define (profile-cached-gc-root file) - "Return the cached GC root for FILE, a 'guix.scm' or 'manifest.scm' file, or -#f if we lack information to cache it." - (match (profile-cache-key file) - (#f #f) - (key (string-append (%profile-cache-directory) "/" key)))) +(define (profile-spec-cache-key specs system) + "Return the cache key corresponding to SPECS built for SYSTEM, where SPECS +is a list of package specs. Return #f if caching is not possible." + (match (profile-cache-primary-key) + (#f #f) + (primary-key + (bytevector->base32-string + (sha256 (string->utf8 + (string-append primary-key ":" system ":" + (object->string specs)))))))) + +(define (profile-cached-gc-root opts) + "Return two values: the file name of a GC root for use as a profile cache +for the options in OPTS, and a timestamp which, if greater than the GC root's +mtime, indicates that the GC root is stale. If OPTS do not permit caching, +return #f and #f." + (define (key->file key) + (string-append (%profile-cache-directory) "/" key)) + + (let loop ((opts opts) + (system (%current-system)) + (file #f) + (specs '())) + (match opts + (() + (if file + (values (and=> (profile-file-cache-key file system) key->file) + (stat:mtime file)) + (values (and=> (profile-spec-cache-key specs system) key->file) + 0))) + (((and spec ('package . _)) . rest) + (if (not file) + (loop rest system file (cons spec specs)) + (values #f #f))) + ((('load . ('package candidate)) . rest) + (if (and (not file) (null? specs)) + (loop rest system candidate specs) + (values #f #f))) + ((('manifest . candidate) . rest) + (if (and (not file) (null? specs)) + (loop rest system candidate specs) + (values #f #f))) + ((('expression . _) . _) + ;; Arbitrary expressions might be non-deterministic or otherwise depend + ;; on external state so do not cache when they're used. + (values #f #f)) + ((((? transformation-option-key?) . _) . _) + ;; Transformation options are potentially "non-deterministic", or at + ;; least depending on external state (with-source, with-commit, etc.), + ;; so do not cache anything when they're used. + (values #f #f)) + ((('system . system) . rest) + (loop rest system file specs)) + ((_ . rest) (loop rest system file specs))))) ;;; diff --git a/guix/transformations.scm b/guix/transformations.scm index c43c00cdd3..0976f0d824 100644 --- a/guix/transformations.scm +++ b/guix/transformations.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès +;;; Copyright © 2016-2022 Ludovic Courtès ;;; Copyright © 2021 Marius Bakke ;;; ;;; This file is part of GNU Guix. @@ -56,6 +56,7 @@ (define-module (guix transformations) tuned-package show-transformation-options-help + transformation-option-key? %transformation-options)) ;;; Commentary: @@ -796,6 +797,12 @@ (define (transformation-procedure key) (and (eq? k key) proc))) %transformations)) +(define (transformation-option-key? key) + "Return true if KEY is an option key (as returned while parsing options with +%TRANSFORMATION-OPTIONS) corresponding to a package transformation option. +For example, (transformation-option-key? 'with-input) => #t." + (->bool (transformation-procedure key))) + ;;; ;;; Command-line handling. base-commit: 861bac1dfbeaaf40b9c11a287ef7607f0fd105a8 -- 2.33.0