Cache eviction policy of ‘guix publish’

Cache eviction policy of ‘guix publish’

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

Hello,

(Continued from a thread on guix-sysadmin on how to make sure ‘guix
publish’ does not remove from its own cache things that correspond to
store items that are still valid.)

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver <mhw@netris.org> skribis:

[...]

>>> especially to ensure that all substitutes from the most recent release
>>> are always kept.  Furthermore, I'd like to see all NARs and NARINFOs
>>> kept for as long as the corresponding items are present in the store, so
>>> that users of less popular packages (or less popular systems) are not
>>> penalized for being in the minority.
>>
>> Agreed.  I’m not sure how to do this; perhaps some sort of a “GC root”
>> mechanism?
>
> How about simply adding an option to 'guix publish' that prevents any
> NAR/NARINFO from being expired from the cache as long as the
> corresponding item is present in the store?  It could be called
> something like --never-expire-valid-store-items.
>
> More concretely, at the point in the code where a cache item is expired
> in the current code, insert an additional check: if the
> --never-expire-valid-store-items option is enabled and the corresponding
> store path is still valid, then do not expire it.
>
> What do you think?

Good idea.  The patch below implements this policy.  I’m tempted to just
make that policy the default and not even add a command-line option to
switch to the current policy.  WDYT?

Thank you!

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1894 bytes --]

diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index a7e3e6d62..775ba62b0 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -385,6 +385,24 @@ at a time."
                     (string-suffix? ".narinfo" file)))
       '()))
 
+(define (nar-expiration-time ttl)
+  "Return the narinfo expiration time (in seconds since the Epoch).  The
+expiration time is +inf.0 when passed an item that is still in the store; in
+other cases, it is the last-access time of the item plus TTL.
+
+This policy allows us to keep cached nars that correspond to valid store
+items.  Failing that, we could eventually have to recompute them and return
+404 in the meantime."
+  (let ((expiration-time (file-expiration-time ttl)))
+    (lambda (file)
+      (let ((item (string-append (%store-prefix) "/"
+                                 (basename file ".narinfo"))))
+        ;; Note: We don't need to use 'valid-path?' here because FILE would
+        ;; not exist if ITEM were not valid in the first place.
+        (if (file-exists? item)
+            +inf.0
+            (expiration-time file))))))
+
 (define* (render-narinfo/cached store request hash
                                 #:key ttl (compression %no-compression)
                                 (nar-path "nar")
@@ -435,7 +453,7 @@ requested using POOL."
                  (maybe-remove-expired-cache-entries cache
                                                      narinfo-files
                                                      #:entry-expiration
-                                                     (file-expiration-time ttl)
+                                                     (nar-expiration-time ttl)
                                                      #:delete-entry delete-entry
                                                      #:cleanup-period ttl))))
            (not-found request

Re: Cache eviction policy of ‘guix publish’

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> How about simply adding an option to 'guix publish' that prevents any
>> NAR/NARINFO from being expired from the cache as long as the
>> corresponding item is present in the store?  It could be called
>> something like --never-expire-valid-store-items.
>>
>> More concretely, at the point in the code where a cache item is expired
>> in the current code, insert an additional check: if the
>> --never-expire-valid-store-items option is enabled and the corresponding
>> store path is still valid, then do not expire it.
>>
>> What do you think?
>
> Good idea.  The patch below implements this policy.  I’m tempted to just
> make that policy the default and not even add a command-line option to
> switch to the current policy.  WDYT?

Sure, sounds good to me.

    Thanks!
      Mark

Re: Cache eviction policy of ‘guix publish’

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> How about simply adding an option to 'guix publish' that prevents any
>>> NAR/NARINFO from being expired from the cache as long as the
>>> corresponding item is present in the store?  It could be called
>>> something like --never-expire-valid-store-items.
>>>
>>> More concretely, at the point in the code where a cache item is expired
>>> in the current code, insert an additional check: if the
>>> --never-expire-valid-store-items option is enabled and the corresponding
>>> store path is still valid, then do not expire it.
>>>
>>> What do you think?
>>
>> Good idea.  The patch below implements this policy.  I’m tempted to just
>> make that policy the default and not even add a command-line option to
>> switch to the current policy.  WDYT?
>
> Sure, sounds good to me.

Pushed as c95644f0172ba87822ee7ecee3d2743ebd2c84bc.

I’ll update ‘guix publish’ on hydra.gnu.org sometime next week if nobody
beats me at it.

Thanks!

Ludo’.

Re: Cache eviction policy of ‘guix publish’

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> Mark H Weaver <mhw@netris.org> skribis:
>>>
>>>> How about simply adding an option to 'guix publish' that prevents any
>>>> NAR/NARINFO from being expired from the cache as long as the
>>>> corresponding item is present in the store?  It could be called
>>>> something like --never-expire-valid-store-items.
>>>>
>>>> More concretely, at the point in the code where a cache item is expired
>>>> in the current code, insert an additional check: if the
>>>> --never-expire-valid-store-items option is enabled and the corresponding
>>>> store path is still valid, then do not expire it.
>>>>
>>>> What do you think?
>>>
>>> Good idea.  The patch below implements this policy.  I’m tempted to just
>>> make that policy the default and not even add a command-line option to
>>> switch to the current policy.  WDYT?
>>
>> Sure, sounds good to me.
>
> Pushed as c95644f0172ba87822ee7ecee3d2743ebd2c84bc.
>
> I’ll update ‘guix publish’ on hydra.gnu.org sometime next week if nobody
> beats me at it.

Done!

Ludo’.