From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id sC9EIn5+CmJQcAAAgWs5BA (envelope-from ) for ; Mon, 14 Feb 2022 17:08:30 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id iLWiH35+CmLU4gAA9RJhRA (envelope-from ) for ; Mon, 14 Feb 2022 17:08:30 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 501A04462B for ; Mon, 14 Feb 2022 17:08:25 +0100 (CET) Received: from localhost ([::1]:60540 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nJdto-0000VA-AQ for larch@yhetil.org; Mon, 14 Feb 2022 11:08:24 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57532) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nJdtc-0008VY-IE for guix-patches@gnu.org; Mon, 14 Feb 2022 11:08:12 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:48448) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nJdtS-0003ls-Rd for guix-patches@gnu.org; Mon, 14 Feb 2022 11:08:12 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nJdtS-0004vW-FT for guix-patches@gnu.org; Mon, 14 Feb 2022 11:08:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#53818] [PATCH 0/3] Add Repology updater Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 14 Feb 2022 16:08:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 53818 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Nicolas Goaziou , Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 53818@debbugs.gnu.org, Xinglu Chen Received: via spool by 53818-submit@debbugs.gnu.org id=B53818.164485484118865 (code B ref 53818); Mon, 14 Feb 2022 16:08:02 +0000 Received: (at 53818) by debbugs.gnu.org; 14 Feb 2022 16:07:21 +0000 Received: from localhost ([127.0.0.1]:42345 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nJdsm-0004uD-Kp for submit@debbugs.gnu.org; Mon, 14 Feb 2022 11:07:20 -0500 Received: from laurent.telenet-ops.be ([195.130.137.89]:57814) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nJdsg-0004u0-UX for 53818@debbugs.gnu.org; Mon, 14 Feb 2022 11:07:19 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by laurent.telenet-ops.be with bizsmtp id v47D2600G4UW6Th0147D0m; Mon, 14 Feb 2022 17:07:13 +0100 Message-ID: <0c9466812aeafc33df51270feed16f8f16c531e7.camel@telenet.be> From: Maxime Devos Date: Mon, 14 Feb 2022 17:07:07 +0100 In-Reply-To: <87pmnprasd.fsf@nicolasgoaziou.fr> References: <87pmnx7ynw.fsf@gnu.org> <87y22kxkv3.fsf@yoctocell.xyz> <87leyi4fcv.fsf@gnu.org> <87pmnprasd.fsf@nicolasgoaziou.fr> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-hNi2PQ24BAUsVe0erF0z" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1644854833; bh=qyrbr8XFRt+3mSPbP6wsSjot67U7LkcGNhY4r6KKd24=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=eXyQN+laVt1ft2kRYxLrBQLcIZhZFFh/tTY1hWO4K77u4RcWsyE15O6X41BCIm9xu vs3ATBk0AnvqE27N9qUIh3bbNv8GtB2XjFHjCN1wOnGMT9SP+uXhQDyutmj4srvxky GnyU1iEv3xyc2BHJyu2kw5ihkAbtoE2wiCqOrvgZxr4w9Bj9+eXA3XGeNcCPPFQ4HT pedHy/4ZnOZ2DChbeGlySiEituotYERh+w7lUXJrKCtd6qOBqf4cOa3cPnUuFW062R S95rFlZs2b2dGPkdumgbTKhhcdlyFnpJlM1at0X+HCgC8SLTch7MiQ8JtiLLfPmmsN g8txrD9w/9Abw== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1644854905; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=qyrbr8XFRt+3mSPbP6wsSjot67U7LkcGNhY4r6KKd24=; b=JpsGP7g3whpLTxAci3VefXC1VlWC5pWMQSpjc/NwDb8oZZyvqTppTvck6AfWEKeIdy0Wvv 5UUbIPd6kN83zCkxKmhqQpKDWOSLHsQz1lyDVmgT0vL8oCzRM6bvxFrPSKyICk5c1FakrO Gs8ohtCCnxmti/CVArX2WOsL2vq+Zfx+nFP8XNwGkY9KPZ9A4BpFhS8O5b2rVh+yeyK4ZM 67vqPhcemiwZ0xDETqOeqyE2RfPWtDHd8l9E6toleHJkV0vs2ZfGnt48a1TBMpaE+laxI4 v4sJ3NdeCp65f2PI654wpmtyTuILId6U1U2R0nIwfKsa+wFfCtFaHFRAnETshg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1644854905; a=rsa-sha256; cv=none; b=t+ZXlzd0Lgv6cxVUqOoiSJamCpHSNQRw3+bo/fZdkrgd9s6kRHWMbmGm81vTlnEMXnnqpq ZBuRfaKSAZS/XlDDVH0YuOiAoKjUrDzQJ25txLqQtn2AvkxbTJ6Y1gAuKdf3728PbeuiBQ jNZn/6xpraZGof8+MkbFmO7bTUj/BTCjFVN0ULXcFofTVB7TPkW79pgNUruZguLaAKoLnz HjRu5Yjnk4RuGopR6hCBtHO7+0AP9oyAuatJBWmt8eMSOaia78lEs1ui68yoJv1Y8Drk9h WEG723P2WPHiWO8dpy+rpoKpynRfn1c0Su1/lcj4Bnc5iA1Mph6pMbfVhrfKFw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r22 header.b=eXyQN+la; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.23 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r22 header.b=eXyQN+la; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 501A04462B X-Spam-Score: -4.23 X-Migadu-Scanner: scn1.migadu.com X-TUID: +6kQXq1+S25j --=-hNi2PQ24BAUsVe0erF0z Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Nicolas Goaziou schreef op ma 14-02-2022 om 11:40 [+0100]: > [...]. Assuming I could manage this, > fetching all version information would take considerable time, and, > since many packages are from GitHub, the party would stop early anyway > with GitHub refusing to proceed and requesting some token I don't have. >=20 > OTOH, using Repology API, I get the information I want in about ten > seconds. Sure, I need to eyeball through the results, filtering false > positives (around 4% in my case), but it still is a practical solution. >=20 > IMO, to be useful, updaters may need to rely on an external service, > which may, or may not, belong to the Guix ecosystem. They also need > a good UI. To avoid exceeding API limits and reduce network traffic, I suggest the following change: Cache HTTP responses, using http-fetch/cached instead of http-fetch. When something is in the cache and not expired, this avoids some network traffic and does not bring us closer to the API limits. When it is expired (and in the cache), then at least http-fetch/cached makes a conditional request with If-Modified-Since, which GitHub does not count against the rate limit, assuming a =E2=80=98304 Not Modified=E2=80=99 response! That does not address all your concerns but it should help I think. Greetings, Maxime. --=-hNi2PQ24BAUsVe0erF0z Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYgp+KxccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7rs8AQCPb2Nc7ojg6vJ8KIfvTrVdwhjO A2RbswK28kuzJzYJVgEA0XaZ4XfcThz6CWWQf9GERfNDQsfmDBW/l1rCGt0pZwo= =XkiL -----END PGP SIGNATURE----- --=-hNi2PQ24BAUsVe0erF0z--