From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id GBduIerQbWHchwAAgWs5BA (envelope-from ) for ; Mon, 18 Oct 2021 21:54:18 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id kPszHerQbWF6EQAA1q6Kng (envelope-from ) for ; Mon, 18 Oct 2021 19:54:18 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1DEA4365F2 for ; Mon, 18 Oct 2021 21:54:18 +0200 (CEST) Received: from localhost ([::1]:35442 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mcYi9-0008PZ-7f for larch@yhetil.org; Mon, 18 Oct 2021 15:54:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55982) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mcYhx-0008Ny-2N for guix-patches@gnu.org; Mon, 18 Oct 2021 15:54:05 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:37143) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mcYhw-00040q-Qp for guix-patches@gnu.org; Mon, 18 Oct 2021 15:54:04 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mcYhw-0000Rd-Q2 for guix-patches@gnu.org; Mon, 18 Oct 2021 15:54:04 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50960] [PATCH v3 10/10] shell: Maintain a profile cache. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 18 Oct 2021 19:54:04 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50960 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 50960@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 50960-submit@debbugs.gnu.org id=B50960.16345868191624 (code B ref 50960); Mon, 18 Oct 2021 19:54:04 +0000 Received: (at 50960) by debbugs.gnu.org; 18 Oct 2021 19:53:39 +0000 Received: from localhost ([127.0.0.1]:48680 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mcYhW-0000Q6-N4 for submit@debbugs.gnu.org; Mon, 18 Oct 2021 15:53:39 -0400 Received: from eggs.gnu.org ([209.51.188.92]:53472) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mcYgn-0000Mm-7K for 50960@debbugs.gnu.org; Mon, 18 Oct 2021 15:52:54 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51644) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mcYgh-0002pY-Uz; Mon, 18 Oct 2021 15:52:47 -0400 Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:60778 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mcYgh-0007lj-Dv; Mon, 18 Oct 2021 15:52:47 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Mon, 18 Oct 2021 21:52:19 +0200 Message-Id: <20211018195219.13898-11-ludo@gnu.org> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211018195219.13898-1-ludo@gnu.org> References: <20211011213809.17482-1-ludo@gnu.org> <20211018195219.13898-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1634586858; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=AWrp6jV/cCWg75UsU1QGXZJsOKOt1XwC93U+/XXE7nw=; b=khmUDImqCOrh/eImX4iBZExaAO0JFyp1qJZ696brrL7c3Oz3VogdLlQrg6M3CVWrrY4xz0 ARO1v3i/AaX/YV/6/MN14jr/r4JkKWGfChJJ992Bv3p+FYOH7d3P4mOfVAs1rrjfnmEQOX wKcFPiApJmsM8iJ2Wm2FjsfLhECiKlsA6ead0/y/phTntzBz8SAE6F9acKKXLSHIUeDdk9 5njzoM6jWavqC7mqOhyWXimzx2vG2NCxDEEdSeE8tmN9KxuB3qoz4TC/SQkB2ruMIX6n1+ /BQKCBiBA5dSVk5gPUGPujGi+vdmeyoP9bpzTDqiHalu11J3HiACTMHANLOOZw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1634586858; a=rsa-sha256; cv=none; b=ABw1LMInstUn2o7k03vOPOvvX8VfakMChvYSOzeOijy30M4FRYZAxEM+nMEDrSBXQHllrR Bfj8hSfgSGiftqwK1af8aWqhcDan8ivO43LkaXp/udVpr4wFLDdmT8T0qaGxvcj6Riu9BQ rwFDXEIeIoZrrqBRBA9qxYC8PoIrAgVCfRALjVOr2mD0chcazmD+XL/pBFCXGb5Ep32YjB /WEyVCLcCJu6WipJE2AwGZhEmyr7vWuEiqDfR+Td53mn0v6JZUbdg3IjCUnJ0Q1+d9aOs3 z0D6kp44DLgOdzBLoxZPkPiGojfk9ejFVZ0my/krm04Wlq2ZWioTH2fviqgKXw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: 2.77 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 1DEA4365F2 X-Spam-Score: 2.77 X-Migadu-Scanner: scn1.migadu.com X-TUID: HPhQ4l8Rkoc8 shell: Maintain a profile cache. With this change, running "guix shell" (no arguments) is equivalent to: guix environment -r ~/.cache/guix/profiles/some-root -l guix.scm This is the cache miss. On cache hit, it's equivalent to: guix environment -p ~/.cache/guix/profiles/some-root ... which can run in 0.1s. * guix/scripts/shell.scm (options-with-caching): New procedure. (parse-args): Use it. (%profile-cache-directory): New variable. (profile-cache-key, profile-cached-gc-root): New procedures. (show-help, %options): Add '--rebuild-cache'. (guix-shell)[cache-entries, entry-expiration]: New procedures. Add call to 'maybe-remove-expired-cache-entries'. * doc/guix.texi (Invoking guix shell): Document '--rebuild-cache'. --- doc/guix.texi | 11 ++++ guix/scripts/shell.scm | 127 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 130 insertions(+), 8 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 7c8f0c1f9b..dabd7fea1e 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -5769,6 +5769,17 @@ This is similar to the same-named option in @command{guix package} (@pxref{profile-manifest, @option{--manifest}}) and uses the same manifest files. +@item --rebuild-cache +When using @option{--manifest}, @option{--file}, or when invoked without +arguments, @command{guix shell} caches the environment so that +subsequent uses are instantaneous. The cache is invalidated anytime the +file is modified. + +The @option{--rebuild-cache} forces the cached environment to be +refreshed even if the file has not changed. This is useful if the +@command{guix.scm} or @command{manifest.scm} has external dependencies, +or if its behavior depends, say, on environment variables. + @item --pure Unset existing environment variables when building the new environment, except those specified with @option{--preserve} (see below). This has the effect of diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm index 45fd536145..4062e8155d 100644 --- a/guix/scripts/shell.scm +++ b/guix/scripts/shell.scm @@ -31,7 +31,15 @@ (define-module (guix scripts shell) #:use-module (srfi srfi-71) #:use-module (ice-9 match) #:autoload (ice-9 rdelim) (read-line) - #:autoload (guix utils) (config-directory) + #:autoload (guix base32) (bytevector->base32-string) + #:autoload (rnrs bytevectors) (string->utf8) + #:autoload (guix utils) (config-directory cache-directory) + #:autoload (guix describe) (current-channels) + #:autoload (guix channels) (channel-commit) + #:autoload (gcrypt hash) (sha256) + #:use-module ((guix build utils) #:select (mkdir-p)) + #:use-module (guix cache) + #:use-module ((ice-9 ftw) #:select (scandir)) #:export (guix-shell)) (define (show-help) @@ -48,6 +56,8 @@ (define (show-help) FILE evaluates to")) (display (G_ " -q inhibit loading of 'guix.scm' and 'manifest.scm'")) + (display (G_ " + --rebuild-cache rebuild cached environment, if any")) (show-environment-options-help) (newline) @@ -109,7 +119,10 @@ (define %options result))) (option '(#\q) #f #f (lambda (opt name arg result) - (alist-cons 'explicit-loading? #t result)))) + (alist-cons 'explicit-loading? #t result))) + (option '("rebuild-cache") #f #f + (lambda (opt name arg result) + (alist-cons 'rebuild-cache? #t result)))) (filter-map (lambda (opt) (and (not (any (lambda (name) (member name to-remove)) @@ -132,11 +145,12 @@ (define (handle-argument arg result) (let ((args command (break (cut string=? "--" <>) args))) (let ((opts (parse-command-line args %options (list %default-options) #:argument-handler handle-argument))) - (auto-detect-manifest - (match command - (() opts) - (("--") opts) - (("--" command ...) (alist-cons 'exec command opts))))))) + (options-with-caching + (auto-detect-manifest + (match command + (() opts) + (("--") opts) + (("--" command ...) (alist-cons 'exec command opts)))))))) (define (find-file-in-parent-directories candidates) "Find one of CANDIDATES in the current directory or one of its ancestors." @@ -187,6 +201,53 @@ (define (authorized-shell-directory? directory) line)))))))))) (const #f))) +(define (options-with-caching opts) + "If OPTS contains exactly one 'load' or one 'manifest' key, automatically +add a 'profile' key (when a profile for that file is already in cache) or a +'gc-root' key (to add the profile to cache)." + (define (single-file-for-caching opts) + (let loop ((opts opts) + (file #f)) + (match opts + (() file) + ((('package . _) . _) #f) + ((('load . ('package candidate)) . rest) + (and (not file) (loop rest candidate))) + ((('manifest . candidate) . rest) + (and (not file) (loop rest candidate))) + ((('expression . _) . _) #f) + ((_ . rest) (loop rest file))))) + + ;; Check whether there's a single 'load' or 'manifest' option. When that is + ;; the case, arrange to automatically cache the resulting profile. + (match (single-file-for-caching opts) + (#f opts) + (file + (let* ((root (profile-cached-gc-root file)) + (stat (and root (false-if-exception (lstat root))))) + (if (and (not (assoc-ref opts 'rebuild-cache?)) + stat + (<= (stat:mtime ((@ (guile) stat) file)) + (stat:mtime stat))) + (let ((now (current-time))) + ;; Update the atime on ROOT to reflect usage. + (utime root + now (stat:mtime stat) 0 (stat:mtimensec stat) + AT_SYMLINK_NOFOLLOW) + (alist-cons 'profile root + (remove (match-lambda + (('load . _) #t) + (('manifest . _) #t) + (_ #f)) + opts))) ;load right away + (if (and root (not (assq-ref opts 'gc-root))) + (begin + (if stat + (delete-file root) + (mkdir-p (dirname root))) + (alist-cons 'gc-root root opts)) + opts)))))) + (define (auto-detect-manifest opts) "If OPTS do not specify packages or a manifest, load a \"guix.scm\" or \"manifest.scm\" file from the current directory or one of its ancestors. @@ -236,9 +297,59 @@ (define disallow-implicit-load? (authorized-directory-file))) opts)))))) + +;;; +;;; Profile cache. +;;; + +(define %profile-cache-directory + ;; Directory where profiles created by 'guix shell' alone (without extra + ;; options) are cached. + (make-parameter (string-append (cache-directory #:ensure? #f) + "/profiles"))) + +(define (profile-cache-key file) + "Return the cache key for the profile corresponding to FILE, a 'guix.scm' or +'manifest.scm' file, or #f if we lack channel information." + (match (current-channels) + (() #f) + (((= channel-commit commits) ...) + (let ((stat (stat file))) + (bytevector->base32-string + ;; Since FILE is not canonicalized, only include the device/inode + ;; numbers. XXX: In some rare cases involving Btrfs and NFS, this can + ;; be insufficient: . + (sha256 (string->utf8 + (string-append (string-join commits) ":" + (number->string (stat:dev stat)) ":" + (number->string (stat:ino stat)))))))))) + +(define (profile-cached-gc-root file) + "Return the cached GC root for FILE, a 'guix.scm' or 'manifest.scm' file, or +#f if we lack information to cache it." + (match (profile-cache-key file) + (#f #f) + (key (string-append (%profile-cache-directory) "/" key)))) + (define-command (guix-shell . args) (category development) (synopsis "spawn one-off software environments") - (guix-environment* (parse-args args))) + (define (cache-entries directory) + (filter-map (match-lambda + ((or "." "..") #f) + (file (string-append directory "/" file))) + (or (scandir directory) '()))) + + (define* (entry-expiration file) + ;; Return the time at which FILE, a cached profile, is considered expired. + (match (false-if-exception (lstat file)) + (#f 0) ;FILE may have been deleted in the meantime + (st (+ (stat:atime st) (* 60 60 24 7))))) + + (let ((result (guix-environment* (parse-args args)))) + (maybe-remove-expired-cache-entries (%profile-cache-directory) + cache-entries + #:entry-expiration entry-expiration) + result)) -- 2.33.0