all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* [bug#71357] [PATCH] substitute: Don’t keep cache entries more than a few days.
@ 2024-06-04  8:06 Ludovic Courtès
  2024-06-10 10:17 ` [bug#71357] Plus 1 Andreas Enge
  2024-06-10 11:57 ` [bug#71357] [PATCH] substitute: Don’t keep cache entries more than a few days Christopher Baines
  0 siblings, 2 replies; 5+ messages in thread
From: Ludovic Courtès @ 2024-06-04  8:06 UTC (permalink / raw)
  To: 71357
  Cc: Ludovic Courtès, Christopher Baines, Josselin Poiret,
	Ludovic Courtès, Mathieu Othacehe, Ricardo Wurmus,
	Simon Tournier, Tobias Geerinckx-Rice

Experience has shown that keeping too many entries increases disk usage
and, more importantly, leads to long delays when cleaning up the cache,
measured in minutes on slow or busy HDDs with hundreds of thousands of
cache entries, as is common on build machines.  In those cases, the cost
of the cache outweighs its benefit.

* guix/scripts/substitute.scm (%narinfo-expired-cache-entry-removal-delay):
Reduce to 5 days.
(cached-narinfo-expiration-time)[max-ttl]: Reduce to 2 days.

Change-Id: Iab212f572ee9041be61716423a3c014f93fe81ed
---
 guix/scripts/substitute.scm | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

Hello,

Chris mentioned it before and I experienced it the hard way on bayfront:

  https://lists.gnu.org/archive/html/guix-devel/2024-05/msg00177.html

A big narinfo cache is a significant performance hit on spinning HDDs
when the time comes to remove expired entries.

This change makes the cache more ephemeral (2 to 5 days).  I still think
some caching is needed: one will often run several Guix commands in a
day that will query the same narinfos and will only download/build a
small subset (keep in mind that that ‘substitution-oracle’, used by
‘derivation-build-plan’, query narinfos for the closure of the requested
derivations, minus those already valid); it would be wasteful and
inefficient to download them over and over again.  I’d like to have
metrics to estimate that, but I don’t.

Thoughts?

Ludo’.

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index a7ad56dbcd5..8bcbca5e7aa 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -77,7 +77,7 @@ (define-module (guix scripts substitute)
 
 (define %narinfo-expired-cache-entry-removal-delay
   ;; How often we want to remove files corresponding to expired cache entries.
-  (* 7 24 3600))
+  (* 5 24 3600))
 
 (define (warn-about-missing-authentication)
   (warning (G_ "authentication and authorization of substitutes \
@@ -169,8 +169,9 @@ (define (cached-narinfo-expiration-time file)
   "Return the expiration time for FILE, which is a cached narinfo."
   (define max-ttl
     ;; Upper bound on the TTL used to avoid keeping around cached narinfos for
-    ;; too long, which makes the cache bigger and more expensive to traverse.
-    (* 2 30 24 60 60))                            ;2 months
+    ;; too long, which makes the cache bigger and more expensive to traverse
+    ;; when deleting old entries.
+    (* 2 24 60 60))
 
   (catch 'system-error
     (lambda ()

base-commit: 85ac164c41fc4c93d3cb2a5d3321c63598c2855f
-- 
2.45.1





^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [bug#71357] Plus 1
  2024-06-04  8:06 [bug#71357] [PATCH] substitute: Don’t keep cache entries more than a few days Ludovic Courtès
@ 2024-06-10 10:17 ` Andreas Enge
  2024-06-10 11:57 ` [bug#71357] [PATCH] substitute: Don’t keep cache entries more than a few days Christopher Baines
  1 sibling, 0 replies; 5+ messages in thread
From: Andreas Enge @ 2024-06-10 10:17 UTC (permalink / raw)
  To: 71357

This looks good!

Andreas





^ permalink raw reply	[flat|nested] 5+ messages in thread

* [bug#71357] [PATCH] substitute: Don’t keep cache entries more than a few days.
  2024-06-04  8:06 [bug#71357] [PATCH] substitute: Don’t keep cache entries more than a few days Ludovic Courtès
  2024-06-10 10:17 ` [bug#71357] Plus 1 Andreas Enge
@ 2024-06-10 11:57 ` Christopher Baines
  2024-06-13  9:50   ` Ludovic Courtès
  1 sibling, 1 reply; 5+ messages in thread
From: Christopher Baines @ 2024-06-10 11:57 UTC (permalink / raw)
  To: Ludovic Courtès
  Cc: 71357, Josselin Poiret, Simon Tournier, Mathieu Othacehe,
	Tobias Geerinckx-Rice, Ricardo Wurmus, Christopher Baines

[-- Attachment #1: Type: text/plain, Size: 2112 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Experience has shown that keeping too many entries increases disk usage
> and, more importantly, leads to long delays when cleaning up the cache,
> measured in minutes on slow or busy HDDs with hundreds of thousands of
> cache entries, as is common on build machines.  In those cases, the cost
> of the cache outweighs its benefit.
>
> * guix/scripts/substitute.scm (%narinfo-expired-cache-entry-removal-delay):
> Reduce to 5 days.
> (cached-narinfo-expiration-time)[max-ttl]: Reduce to 2 days.
>
> Change-Id: Iab212f572ee9041be61716423a3c014f93fe81ed
> ---
>  guix/scripts/substitute.scm | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
>
> Hello,
>
> Chris mentioned it before and I experienced it the hard way on bayfront:
>
>   https://lists.gnu.org/archive/html/guix-devel/2024-05/msg00177.html
>
> A big narinfo cache is a significant performance hit on spinning HDDs
> when the time comes to remove expired entries.
>
> This change makes the cache more ephemeral (2 to 5 days).  I still think
> some caching is needed: one will often run several Guix commands in a
> day that will query the same narinfos and will only download/build a
> small subset (keep in mind that that ‘substitution-oracle’, used by
> ‘derivation-build-plan’, query narinfos for the closure of the requested
> derivations, minus those already valid); it would be wasteful and
> inefficient to download them over and over again.  I’d like to have
> metrics to estimate that, but I don’t.
>
> Thoughts?

This sounds good to me.

I think one of the problems on bayfront is that each substitute process
looks and decides it's time to remove the expired cache entries. For
every new process that starts and decides to join it, it probably slows
them all down. This is very similar to a "thundering herd" since the
processes trip over each other trying to delete the same files.

This change won't directly address that part of the issue, but maybe
keeping the cache smaller will help reduce the impact when this happens.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [bug#71357] [PATCH] substitute: Don’t keep cache entries more than a few days.
  2024-06-10 11:57 ` [bug#71357] [PATCH] substitute: Don’t keep cache entries more than a few days Christopher Baines
@ 2024-06-13  9:50   ` Ludovic Courtès
  2024-06-16 21:15     ` Ludovic Courtès
  0 siblings, 1 reply; 5+ messages in thread
From: Ludovic Courtès @ 2024-06-13  9:50 UTC (permalink / raw)
  To: Christopher Baines
  Cc: 71357, Josselin Poiret, Simon Tournier, Mathieu Othacehe,
	Tobias Geerinckx-Rice, Ricardo Wurmus, Andreas Enge,
	Christopher Baines

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> This sounds good to me.

Thanks, pushed as 7e00fb9f31f51ac2f9fa67b71a3eb8aaa23efdb6.  I’ll update
the ‘guix’ package soonish.

> I think one of the problems on bayfront is that each substitute process
> looks and decides it's time to remove the expired cache entries. For
> every new process that starts and decides to join it, it probably slows
> them all down. This is very similar to a "thundering herd" since the
> processes trip over each other trying to delete the same files.

Oh right.  I’m not sure if this is what I was seeing, but it can
definitely be a problem on build machines.  We should change (guix
cache) to avoid that.

Thanks!

Ludo’.




^ permalink raw reply	[flat|nested] 5+ messages in thread

* [bug#71357] [PATCH] substitute: Don’t keep cache entries more than a few days.
  2024-06-13  9:50   ` Ludovic Courtès
@ 2024-06-16 21:15     ` Ludovic Courtès
  0 siblings, 0 replies; 5+ messages in thread
From: Ludovic Courtès @ 2024-06-16 21:15 UTC (permalink / raw)
  To: Christopher Baines
  Cc: 71357, Josselin Poiret, Simon Tournier, Mathieu Othacehe,
	Tobias Geerinckx-Rice, Ricardo Wurmus, Andreas Enge,
	Christopher Baines

Ludovic Courtès <ludo@gnu.org> skribis:

> Thanks, pushed as 7e00fb9f31f51ac2f9fa67b71a3eb8aaa23efdb6.  I’ll update
> the ‘guix’ package soonish.

Done in bd5c61781c13611ed16686513980907c6ee34ae6.




^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2024-06-16 21:16 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-06-04  8:06 [bug#71357] [PATCH] substitute: Don’t keep cache entries more than a few days Ludovic Courtès
2024-06-10 10:17 ` [bug#71357] Plus 1 Andreas Enge
2024-06-10 11:57 ` [bug#71357] [PATCH] substitute: Don’t keep cache entries more than a few days Christopher Baines
2024-06-13  9:50   ` Ludovic Courtès
2024-06-16 21:15     ` Ludovic Courtès

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.