From: "Ludovic Courtès" <ludo@gnu.org>
To: 50960@debbugs.gnu.org
Cc: "Ludovic Courtès" <ludo@gnu.org>
Subject: [bug#50960] [PATCH v2 11/11] shell: Maintain a profile cache.
Date: Mon, 11 Oct 2021 23:38:09 +0200 [thread overview]
Message-ID: <20211011213809.17482-12-ludo@gnu.org> (raw)
In-Reply-To: <20211011213809.17482-1-ludo@gnu.org>
shell: Maintain a profile cache.
With this change, running "guix shell" (no arguments) is equivalent to:
guix environment -r ~/.cache/guix/profiles/some-root -l guix.scm
This is the cache miss. On cache hit, it's equivalent to:
guix environment -p ~/.cache/guix/profiles/some-root
... which can run in 0.1s.
* guix/scripts/shell.scm (options-with-caching): New procedure.
(parse-args): Use it.
(%profile-cache-directory): New variable.
(profile-cache-key, profile-cached-gc-root): New procedures.
(show-help, %options): Add '--rebuild-cache'.
(guix-shell)[cache-entries, entry-expiration]: New procedures.
Add call to 'maybe-remove-expired-cache-entries'.
* doc/guix.texi (Invoking guix shell): Document '--rebuild-cache'.
---
doc/guix.texi | 11 ++++
guix/scripts/shell.scm | 127 ++++++++++++++++++++++++++++++++++++++---
2 files changed, 130 insertions(+), 8 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index b95025a39f..f3be6b5085 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -5768,6 +5768,17 @@ This is similar to the same-named option in @command{guix package}
(@pxref{profile-manifest, @option{--manifest}}) and uses the same
manifest files.
+@item --rebuild-cache
+When using @option{--manifest}, @option{--file}, or when invoked without
+arguments, @command{guix shell} caches the environment so that
+subsequent uses are instantaneous. The cache is invalidated anytime the
+file is modified.
+
+The @option{--rebuild-cache} forces the cached environment to be
+refreshed even if the file has not changed. This is useful if the
+@command{guix.scm} or @command{manifest.scm} has external dependencies,
+or if its behavior depends, say, on environment variables.
+
@item --pure
Unset existing environment variables when building the new environment, except
those specified with @option{--preserve} (see below). This has the effect of
diff --git a/guix/scripts/shell.scm b/guix/scripts/shell.scm
index 45fd536145..4062e8155d 100644
--- a/guix/scripts/shell.scm
+++ b/guix/scripts/shell.scm
@@ -31,7 +31,15 @@ (define-module (guix scripts shell)
#:use-module (srfi srfi-71)
#:use-module (ice-9 match)
#:autoload (ice-9 rdelim) (read-line)
- #:autoload (guix utils) (config-directory)
+ #:autoload (guix base32) (bytevector->base32-string)
+ #:autoload (rnrs bytevectors) (string->utf8)
+ #:autoload (guix utils) (config-directory cache-directory)
+ #:autoload (guix describe) (current-channels)
+ #:autoload (guix channels) (channel-commit)
+ #:autoload (gcrypt hash) (sha256)
+ #:use-module ((guix build utils) #:select (mkdir-p))
+ #:use-module (guix cache)
+ #:use-module ((ice-9 ftw) #:select (scandir))
#:export (guix-shell))
(define (show-help)
@@ -48,6 +56,8 @@ (define (show-help)
FILE evaluates to"))
(display (G_ "
-q inhibit loading of 'guix.scm' and 'manifest.scm'"))
+ (display (G_ "
+ --rebuild-cache rebuild cached environment, if any"))
(show-environment-options-help)
(newline)
@@ -109,7 +119,10 @@ (define %options
result)))
(option '(#\q) #f #f
(lambda (opt name arg result)
- (alist-cons 'explicit-loading? #t result))))
+ (alist-cons 'explicit-loading? #t result)))
+ (option '("rebuild-cache") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'rebuild-cache? #t result))))
(filter-map (lambda (opt)
(and (not (any (lambda (name)
(member name to-remove))
@@ -132,11 +145,12 @@ (define (handle-argument arg result)
(let ((args command (break (cut string=? "--" <>) args)))
(let ((opts (parse-command-line args %options (list %default-options)
#:argument-handler handle-argument)))
- (auto-detect-manifest
- (match command
- (() opts)
- (("--") opts)
- (("--" command ...) (alist-cons 'exec command opts)))))))
+ (options-with-caching
+ (auto-detect-manifest
+ (match command
+ (() opts)
+ (("--") opts)
+ (("--" command ...) (alist-cons 'exec command opts))))))))
(define (find-file-in-parent-directories candidates)
"Find one of CANDIDATES in the current directory or one of its ancestors."
@@ -187,6 +201,53 @@ (define (authorized-shell-directory? directory)
line))))))))))
(const #f)))
+(define (options-with-caching opts)
+ "If OPTS contains exactly one 'load' or one 'manifest' key, automatically
+add a 'profile' key (when a profile for that file is already in cache) or a
+'gc-root' key (to add the profile to cache)."
+ (define (single-file-for-caching opts)
+ (let loop ((opts opts)
+ (file #f))
+ (match opts
+ (() file)
+ ((('package . _) . _) #f)
+ ((('load . ('package candidate)) . rest)
+ (and (not file) (loop rest candidate)))
+ ((('manifest . candidate) . rest)
+ (and (not file) (loop rest candidate)))
+ ((('expression . _) . _) #f)
+ ((_ . rest) (loop rest file)))))
+
+ ;; Check whether there's a single 'load' or 'manifest' option. When that is
+ ;; the case, arrange to automatically cache the resulting profile.
+ (match (single-file-for-caching opts)
+ (#f opts)
+ (file
+ (let* ((root (profile-cached-gc-root file))
+ (stat (and root (false-if-exception (lstat root)))))
+ (if (and (not (assoc-ref opts 'rebuild-cache?))
+ stat
+ (<= (stat:mtime ((@ (guile) stat) file))
+ (stat:mtime stat)))
+ (let ((now (current-time)))
+ ;; Update the atime on ROOT to reflect usage.
+ (utime root
+ now (stat:mtime stat) 0 (stat:mtimensec stat)
+ AT_SYMLINK_NOFOLLOW)
+ (alist-cons 'profile root
+ (remove (match-lambda
+ (('load . _) #t)
+ (('manifest . _) #t)
+ (_ #f))
+ opts))) ;load right away
+ (if (and root (not (assq-ref opts 'gc-root)))
+ (begin
+ (if stat
+ (delete-file root)
+ (mkdir-p (dirname root)))
+ (alist-cons 'gc-root root opts))
+ opts))))))
+
(define (auto-detect-manifest opts)
"If OPTS do not specify packages or a manifest, load a \"guix.scm\" or
\"manifest.scm\" file from the current directory or one of its ancestors.
@@ -236,9 +297,59 @@ (define disallow-implicit-load?
(authorized-directory-file)))
opts))))))
+\f
+;;;
+;;; Profile cache.
+;;;
+
+(define %profile-cache-directory
+ ;; Directory where profiles created by 'guix shell' alone (without extra
+ ;; options) are cached.
+ (make-parameter (string-append (cache-directory #:ensure? #f)
+ "/profiles")))
+
+(define (profile-cache-key file)
+ "Return the cache key for the profile corresponding to FILE, a 'guix.scm' or
+'manifest.scm' file, or #f if we lack channel information."
+ (match (current-channels)
+ (() #f)
+ (((= channel-commit commits) ...)
+ (let ((stat (stat file)))
+ (bytevector->base32-string
+ ;; Since FILE is not canonicalized, only include the device/inode
+ ;; numbers. XXX: In some rare cases involving Btrfs and NFS, this can
+ ;; be insufficient: <https://lwn.net/Articles/866582/>.
+ (sha256 (string->utf8
+ (string-append (string-join commits) ":"
+ (number->string (stat:dev stat)) ":"
+ (number->string (stat:ino stat))))))))))
+
+(define (profile-cached-gc-root file)
+ "Return the cached GC root for FILE, a 'guix.scm' or 'manifest.scm' file, or
+#f if we lack information to cache it."
+ (match (profile-cache-key file)
+ (#f #f)
+ (key (string-append (%profile-cache-directory) "/" key))))
+
\f
(define-command (guix-shell . args)
(category development)
(synopsis "spawn one-off software environments")
- (guix-environment* (parse-args args)))
+ (define (cache-entries directory)
+ (filter-map (match-lambda
+ ((or "." "..") #f)
+ (file (string-append directory "/" file)))
+ (or (scandir directory) '())))
+
+ (define* (entry-expiration file)
+ ;; Return the time at which FILE, a cached profile, is considered expired.
+ (match (false-if-exception (lstat file))
+ (#f 0) ;FILE may have been deleted in the meantime
+ (st (+ (stat:atime st) (* 60 60 24 7)))))
+
+ (let ((result (guix-environment* (parse-args args))))
+ (maybe-remove-expired-cache-entries (%profile-cache-directory)
+ cache-entries
+ #:entry-expiration entry-expiration)
+ result))
--
2.33.0
next prev parent reply other threads:[~2021-10-11 21:44 UTC|newest]
Thread overview: 108+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-02 10:21 [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment' Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 01/10] packages: Add 'package-development-inputs' Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 02/10] profiles: Add 'package->development-manifest' Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 03/10] DRAFT Add 'guix shell' Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 04/10] DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm' file Ludovic Courtès
2021-10-02 11:52 ` Liliana Marie Prikler
2021-10-02 13:43 ` [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment' Ludovic Courtès
2021-10-05 7:50 ` Maxime Devos
2021-10-08 7:44 ` Ludovic Courtès
2021-10-02 14:15 ` [bug#50960] [PATCH 04/10] DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm' file Maxime Devos
2021-10-04 8:07 ` Ludovic Courtès
2021-10-05 7:51 ` Maxime Devos
2021-10-02 10:22 ` [bug#50960] [PATCH 05/10] environment: Add tests for '--profile' Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 06/10] environment: Skip derivation computation when '--profile' is used Ludovic Courtès
2021-10-02 11:39 ` Liliana Marie Prikler
2021-10-02 13:46 ` [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment' Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 07/10] environment: Do not connect to the daemon when '--profile' is used Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 08/10] environment: Autoload some modules Ludovic Courtès
2021-10-02 10:22 ` [bug#50960] [PATCH 09/10] cache: Gracefully handle non-existent cache Ludovic Courtès
2021-10-02 13:28 ` Maxime Devos
2021-10-02 10:22 ` [bug#50960] [PATCH 10/10] shell: Maintain a profile cache Ludovic Courtès
2021-10-02 13:43 ` Maxime Devos
2021-10-02 14:12 ` Ludovic Courtès
2021-10-02 14:47 ` Maxime Devos
2021-10-04 8:19 ` Ludovic Courtès
2021-10-04 14:20 ` zimoun
2021-10-04 15:58 ` Maxime Devos
2021-10-08 7:37 ` Ludovic Courtès
2021-10-02 13:52 ` Maxime Devos
2021-10-02 14:14 ` Ludovic Courtès
2021-10-02 14:22 ` Maxime Devos
2021-10-04 8:08 ` Ludovic Courtès
2021-10-02 10:50 ` [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment' Jelle Licht
2021-10-02 13:52 ` Ludovic Courtès
2021-10-02 12:10 ` pelzflorian (Florian Pelz)
2021-10-02 13:40 ` Ludovic Courtès
2021-10-02 15:08 ` pelzflorian (Florian Pelz)
2021-10-04 8:22 ` Ludovic Courtès
2021-10-04 9:23 ` pelzflorian (Florian Pelz)
2021-10-04 16:50 ` Maxime Devos
2021-10-02 13:03 ` Christine Lemmer-Webber
2021-10-02 14:00 ` [bug#50960] ‘guix shell’ shebangs Ludovic Courtès
2021-10-03 22:50 ` Katherine Cox-Buday
2021-10-02 23:57 ` [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment' Vagrant Cascadian
2021-10-03 8:36 ` Nicolò Balzarotti
2021-10-04 8:34 ` Ludovic Courtès
2021-10-04 17:12 ` Maxime Devos
2021-10-04 6:56 ` zimoun
2021-10-04 8:39 ` Ludovic Courtès
2021-10-04 10:40 ` zimoun
2021-10-04 12:23 ` Ludovic Courtès
2021-10-04 13:42 ` zimoun
2021-10-04 17:38 ` Leo Famulari
2021-10-08 7:43 ` Ludovic Courtès
2021-10-08 16:16 ` Leo Famulari
2021-10-09 13:38 ` Ludovic Courtès
2021-10-11 0:29 ` Leo Famulari
2021-10-04 21:29 ` [bug#50960] [EXT] " Thompson, David
2021-10-07 9:26 ` Ludovic Courtès
2021-10-07 10:52 ` pelzflorian (Florian Pelz)
2021-10-07 11:17 ` [bug#50960] [EXT] " Thompson, David
2021-10-07 12:01 ` pelzflorian (Florian Pelz)
2021-10-08 14:24 ` Katherine Cox-Buday
2021-10-11 9:13 ` zimoun
2021-10-06 8:12 ` Konrad Hinsen
2021-10-07 8:34 ` Ludovic Courtès
2021-10-07 9:15 ` Liliana Marie Prikler
2021-10-08 15:45 ` Konrad Hinsen
2021-10-09 7:45 ` Liliana Marie Prikler
2021-10-11 8:32 ` Ludovic Courtès
2021-10-09 8:07 ` Stefan
2021-10-11 21:37 ` [bug#50960] [PATCH v2 00/11] 'guix shell' strikes again Ludovic Courtès
2021-10-11 21:37 ` [bug#50960] [PATCH v2 01/11] packages: Add 'package-development-inputs' Ludovic Courtès
2021-10-12 6:39 ` zimoun
2021-10-12 9:54 ` Ludovic Courtès
2021-10-12 11:52 ` zimoun
2021-10-11 21:38 ` [bug#50960] [PATCH v2 02/11] profiles: Add 'package->development-manifest' Ludovic Courtès
2021-10-12 6:43 ` zimoun
2021-10-12 9:27 ` Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 03/11] Add 'guix shell' Ludovic Courtès
2021-10-13 16:51 ` pelzflorian (Florian Pelz)
2021-10-11 21:38 ` [bug#50960] [PATCH v2 04/11] DRAFT shell: By default load the local 'guix.scm' or 'manifest.scm' file Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 05/11] DRAFT shell: Honor in ~/.config/guix/shell-authorized-directories Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 06/11] environment: Add tests for '--profile' Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 07/11] environment: Skip derivation computation when '--profile' is used Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 08/11] environment: Do not connect to the daemon " Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 09/11] environment: Autoload some modules Ludovic Courtès
2021-10-11 21:38 ` [bug#50960] [PATCH v2 10/11] cache: Gracefully handle non-existent cache Ludovic Courtès
2021-10-11 21:38 ` Ludovic Courtès [this message]
2021-10-12 8:53 ` [bug#50960] [PATCH v2 00/11] 'guix shell' strikes again pelzflorian (Florian Pelz)
2021-10-12 8:57 ` pelzflorian (Florian Pelz)
2021-10-12 9:55 ` Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 00/10] Adding 'guix shell': last call! Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 01/10] packages: Add 'package-development-inputs' Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 02/10] profiles: Add 'package->development-manifest' Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 03/10] Add 'guix shell' Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 04/10] shell: By default load the local 'guix.scm' or 'manifest.scm' file Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 05/10] environment: Add tests for '--profile' Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 06/10] environment: Skip derivation computation when '--profile' is used Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 07/10] environment: Do not connect to the daemon " Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 08/10] environment: Autoload some modules Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 09/10] cache: Gracefully handle non-existent cache Ludovic Courtès
2021-10-18 19:52 ` [bug#50960] [PATCH v3 10/10] shell: Maintain a profile cache Ludovic Courtès
2021-10-19 8:43 ` [bug#50960] [PATCH v3 00/10] Adding 'guix shell': last call! zimoun
2021-10-25 13:41 ` [bug#50960] [PATCH 00/10] Add 'guix shell' to subsume 'guix environment' zimoun
2021-10-25 18:19 ` Ludovic Courtès
2021-10-25 19:45 ` zimoun
2021-10-25 18:25 ` Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211011213809.17482-12-ludo@gnu.org \
--to=ludo@gnu.org \
--cc=50960@debbugs.gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).