From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id oAPGCXADgmBhmwAAgWs5BA (envelope-from ) for ; Fri, 23 Apr 2021 01:14:56 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id cPJ5BXADgmBxAQAA1q6Kng (envelope-from ) for ; Thu, 22 Apr 2021 23:14:56 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B35681BFD9 for ; Fri, 23 Apr 2021 01:14:55 +0200 (CEST) Received: from localhost ([::1]:46520 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZiX8-0005NU-Ts for larch@yhetil.org; Thu, 22 Apr 2021 19:14:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41842) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZiWr-0005NN-I8 for guix-devel@gnu.org; Thu, 22 Apr 2021 19:14:37 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:58547) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZiWk-0006jZ-O5; Thu, 22 Apr 2021 19:14:37 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 0603627BC7C; Fri, 23 Apr 2021 00:14:28 +0100 (BST) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id c7eb486c; Thu, 22 Apr 2021 23:14:27 +0000 (UTC) References: <87mtvhnsn6.fsf@cbaines.net> <874kfyufzg.fsf@gnu.org> User-agent: mu4e 1.4.15; emacs 27.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Narinfo negative and transient error caching In-reply-to: <874kfyufzg.fsf@gnu.org> Date: Fri, 23 Apr 2021 00:14:24 +0100 Message-ID: <87wnsthpyn.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org, 47897@debbugs.gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1619133295; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=6ukP/VZR6KCfHfur1QlGwZscBCmxYYCny1HDcYMZZeM=; b=J4umdEurxEUl/6TGRF9nxL63lPeirqoTP3HtlNP8nUWfmJ6YeOw4xQBgN0tceTEg3gXZfF 5XCvTx6wEMWdsLdfQ2Lj288VHDAghDUqp8KQhoew2YiUSHdLJwkBTcKMyyiIebVdDCLbSO ke3vb4MGKlUEnVYY7qdoyhXRDX9d6iiE5GdJCkRYUpkLC+igsKIOvb4A2lzkKaJgrhDxIp 7pjFebPmMzIVftSbnNgnafCJrqX+QbZwF/GA1xo0KtZ8jc6ssIJ+FmVbfV+MFTibE7yOyu qUzWzU2RxmScUf0bO7TnXMJgczMEXW2/Ki0eK6TzZt+evZjQ3ZIDFVIemki5IA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1619133295; a=rsa-sha256; cv=none; b=dgcVyrM91W3kbjCHifrqLJgNPYZC6gW+ZCTzBz9AodHmM2uCaSOi856IPEykxP5i4b+V12 bynelGFB/u2HOib55mtppWwogoBBi3lzd/48p4WW9Zh4bxw0x0H4U60f7sAHneU+pUrzuc MqB9MbwlvOb06qiNn13JAzNl2CwLDokHWICKLHw7B1YWSYNaVn0IFW5FlT71+mS47nGhip 2RiRuD1ZHiJJwaU8sF4V3nmp09mBqt6c+szgZhq65l/8AZKeNNPmJ/+UOPpy8Ssb0GdMBP zRbZvTjyUILS4Nj79wrjyu+6FtTO8HJSa5i9N7UW0aRw3OVHp0Nwszn55yOJmw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -4.55 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: B35681BFD9 X-Spam-Score: -4.55 X-Migadu-Scanner: scn0.migadu.com X-TUID: 8KyN/Kh/awb6 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi! > > (=E2=80=9CSorry for the long delay=E2=80=9D is officially my motto at thi= s point.) > > Christopher Baines skribis: > >> This has been on my mind for a while, as I wonder what effect it has on >> users fetching substitues. >> >> The narinfo caching as I understand it works as follows: >> >> Default success TTL =3D> 36 hours >> Negative TTL =3D> 1 hour >> Transient error TTL =3D> 10 minutes >> >> I'm ignoring the success TTL, I'm just interested in the negative and >> transient error values. Negative means that when a server says it >> doesn't have an output, that response will be cached for an >> hour. Transient errors are for other HTTP response codes, like 504. > > You=E2=80=99re looking at the default TTLs, which are not the actual TTLs. > Specifically, servers can include a =E2=80=98Cache-Control=E2=80=99 heade= r in their > reply specifying the TTL of their choice, and =E2=80=98guix substitute=E2= =80=99 honors > that: > > https://git.savannah.gnu.org/cgit/guix.git/tree/guix/substitutes.scm#n2= 00 > https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/publish.sc= m#n371 > > =E2=80=98guix publish=E2=80=99 returns 404 with a TTL of 5mn when the req= uested item is > in store but needs to be =E2=80=9Cbaked=E2=80=9D. > > However, =E2=80=98guix publish=E2=80=99 does not set =E2=80=98Cache-Contr= ol=E2=80=99 when the request > item is not in store. In that case, clients use =E2=80=98%narinfo-negati= ve-ttl=E2=80=99 > (1h). You're right that the negative ttl is just a default, so it's possible to override the default behaviour in the success and negative lookup cases, but I don't believe the Cache-Control header is used for transient errors. >> I had a look through the Git history, caching negative lookups has been >> a thing for a while. Caching transient errors was added, but I couldn't >> see why. > > Transient error caching was most likely added in the days of > hydra.gnu.org, that VM that was extremely slow. When overloaded, you=E2= =80=99d > get 500 or similar, and at that point it was safer for clients to wait > and come back later, possibly much later. :-) > >> Personally I don't see a reason to keep either behaviours? > > The main arguments for these negative TTLs are: > > 1. Reducing server load: if the server doesn=E2=80=99t have libreoffice= , don=E2=80=99t > come back asking every 10s, it=E2=80=99s prolly useless. You could = easily > have =E2=80=9CGET storms=E2=80=9D for libreoffice if clients don=E2= =80=99t restrain > themselves. > > 2. Improving client performance: don=E2=80=99t GET things that are like= ly to > fail. As you say, for the negative TTL, the question here is really what's the best default value, if a server isn't specifying one. Given that most narinfo requests precede a build for that thing if the response is negative, I have my doubts about those two arguments above. This is assuming the most common case is users asking guix to install and upgrade things. If a user gets a negative response, they'll just build it instead and not check for that narinfo again. Even if they cancel that build when they realise they don't want to build libreoffice, they'll wait a bit anyway before retrying. > Now, the penalty it imposes is annoying. I=E2=80=99ve sometimes found my= self > working around it, too (because I knew the server was going to have the > store item sooner than 1h). > > Rather than removing it entirely, I can think of these options: > > 1. Reduce the default negative timeouts. I think reducing it is good, as you say, it's possible to override the default from the server side. Just in case someone wants caching behaviour, it might be worth keeping that functionality at least. > 2. Add an option to =E2=80=98guix publish=E2=80=99 (and to the Coordina= tor?) so they > send a =E2=80=98Cache-Control=E2=80=99 header with the chosen TTL on= 404. That > way, if the server operator doesn=E2=80=99t mind extra load, they ca= n run > =E2=80=9Cguix publish --negative-ttl=3D0=E2=80=9D. That sounds sensible. The Guix Build Coordinator doesn't do any serving, that's left to something else like nginx. For the deployments I maintain though, I don't think I'm setting the relevant headers, but I'll look at changing that. Going back to the %narinfo-transient-error-ttl, if I'm correct in saying that it's not possible to override that, maybe that should also use the relevant header value if set? --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmCCA1BfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XettQ//Ytdi5P4uajikIBpDpdEehCyYayl3awpY /KimsScWS6QbULGBPCHFZx92glZYjM3osEZWoBrUApVa5cODKSZ61/R1CzA+rGUO 2ed2Drqvr5OJPzZnf1chojFHqnM3AHpbffJpcqyF3mTZqAhQPD6nW1l6MITpOpLI BG0Af/qZYm4/dKgsJNeTGLqcGiGkew/zne9yzMNwMfirOoaDKgoycJTrJ3PdCCfT MBpkE7UgFMuYagIVqpNnHk+RDnZFf3Ayb23b/Kcrbeo3xcGyDwy4H4raf1uw67G9 xXzTM5WnhlpRsZqM3MKO9aNf6s7PN3x6KiyhmuulwArCzVyI6bF8jTlzuPEE/PSJ 5RrlGQRzxCUQ/mjfL95Xi2/nGWpF8X0I3H3IPdntBjj2lqfyQMww9mna3Byqldwj CEuDwvBFeiwNuNRJaXhQf6jZXPBsZ6MjWdkZNGtyYi+3Zy+43ww8Xvl8RHvKHcuI qSxC4TWAIfXiB8ZmWWPGxVnGVNIGgVl+0N/UB4+AXDuJuXtKGhkY0k2ZZdq3LQ37 /X14mJTYDpLnkvZEvOJzfTpJnL61Q5YBNdRmz0iTlpBMPymZQtHYKbdOtDFoXg5X AmMa+39Mv2+Ep6rMH0c9CPlnmNz97qELsivQ4VmUJohIl5wYGnsWwt1kKPHrA1bq vvGRj5HahDE= =s2Q0 -----END PGP SIGNATURE----- --=-=-=--