From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id CHgPNKwvDmICUgAAgWs5BA (envelope-from ) for ; Thu, 17 Feb 2022 12:21:16 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id YBeJMawvDmLuUQAA9RJhRA (envelope-from ) for ; Thu, 17 Feb 2022 12:21:16 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4F0CE16A94 for ; Thu, 17 Feb 2022 12:21:16 +0100 (CET) Received: from localhost ([::1]:53468 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nKeqZ-00025h-C5 for larch@yhetil.org; Thu, 17 Feb 2022 06:21:15 -0500 Received: from eggs.gnu.org ([209.51.188.92]:59520) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nKeqR-00025V-7g for guix-patches@gnu.org; Thu, 17 Feb 2022 06:21:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:56400) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nKeqM-0003cI-9g for guix-patches@gnu.org; Thu, 17 Feb 2022 06:21:06 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nKeqM-0004dE-4g for guix-patches@gnu.org; Thu, 17 Feb 2022 06:21:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#53818] Improving updaters and =?UTF-8?Q?=E2=80=98guix_?= =?UTF-8?Q?refresh=E2=80=99?= Resent-From: zimoun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 17 Feb 2022 11:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 53818 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Nicolas Goaziou Cc: 53818@debbugs.gnu.org, Xinglu Chen Received: via spool by 53818-submit@debbugs.gnu.org id=B53818.164509684517760 (code B ref 53818); Thu, 17 Feb 2022 11:21:02 +0000 Received: (at 53818) by debbugs.gnu.org; 17 Feb 2022 11:20:45 +0000 Received: from localhost ([127.0.0.1]:50297 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nKeq4-0004cO-T9 for submit@debbugs.gnu.org; Thu, 17 Feb 2022 06:20:45 -0500 Received: from mail-wm1-f42.google.com ([209.85.128.42]:46860) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nKeq3-0004c9-Vo for 53818@debbugs.gnu.org; Thu, 17 Feb 2022 06:20:44 -0500 Received: by mail-wm1-f42.google.com with SMTP id l67-20020a1c2546000000b00353951c3f62so3796459wml.5 for <53818@debbugs.gnu.org>; Thu, 17 Feb 2022 03:20:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-transfer-encoding; bh=gULrkgwQ1fsi80ryHvjJkOsjzEJH9/6f7+CBn2KhQRo=; b=XDIBqvWKqppP5GMiuPSM1XBeLO0EQS7cUjUJEN9Z3XHOqsZCggDLnHjCEFVnURzOHO wL/eIBhtQzqHMt78ANs8Eu7pSh9DKrb9P+lr5Pr7aHGbtNYJdfWExL9T4d5kCmcwLxsN 7ZRIxLr9scuRdkzh6bXTJJbt9I2JYWISh3xdDoALdUg5GIS5vHPTRoePsaY7Eym78C3G hfwCZfac+tb3RUGQcAofPEa5wIQH1XxYEJGXGmKB9Ot+lxtlr6uYZZKgAJ18A1KQrnXS 8xe5p1SpgHU9K059Uv5HLmzSSCITGCPsfOeVdMHeIH5pb8PA8es2VMP9XLtQyjjmZm+4 D1ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=gULrkgwQ1fsi80ryHvjJkOsjzEJH9/6f7+CBn2KhQRo=; b=Z+fAWJLv+81m3HObMolsuoCasnb86XpJvPRfYvVlp5pUGdhEMhBObXmnBAAT1aqrFq vwMNs1wyTjTl5qAfek6MRZHeXm+5aTNvdXWq6E4gDTdEKxjvW2kqXJk5+w5+XyQgVers nyL9C9ZlUbAOmr4gZ6wrURaeQ2Fkl86dIjqv2zpo2zjf4yy5cymlFVvSc4zq0zpjC+Ur 24HL/fY5BIiOyGIvtaWZRp/Ore300nH86YINW+Q8acgzVyov7BViQFdbasG02qumxL+l tL5mi6MG220pCkE7tjosHAyJryQGhdZM1UJR7J5nFhGMM8kZi/ZTP7+Z7m/LO4xdmvA9 lO4A== X-Gm-Message-State: AOAM532P29xmOM4FGZyeKJ6Y73ezAOC4ZW2vwV4CKqGF8p6ef/LWnONu if4RudAjBkDSquZzF6WKD1s= X-Google-Smtp-Source: ABdhPJwYGEPSjqjSL9byaYbxIR+rNRALtu/cKg5dboWp03I013DjQ1O/U660kX+8+myHfDhXXku65g== X-Received: by 2002:a05:600c:3b04:b0:37d:342c:3696 with SMTP id m4-20020a05600c3b0400b0037d342c3696mr5738050wms.62.1645096837920; Thu, 17 Feb 2022 03:20:37 -0800 (PST) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id k28sm1085241wms.23.2022.02.17.03.20.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Feb 2022 03:20:37 -0800 (PST) From: zimoun In-Reply-To: <87mtipok57.fsf@gnu.org> References: <87pmnx7ynw.fsf@gnu.org> <87y22kxkv3.fsf@yoctocell.xyz> <87leyi4fcv.fsf@gnu.org> <87pmnprasd.fsf@nicolasgoaziou.fr> <87fsolv109.fsf@gnu.org> <87leydqohm.fsf@nicolasgoaziou.fr> <87ee44pi4p.fsf_-_@gnu.org> <87a6eroubv.fsf@nicolasgoaziou.fr> <87mtipok57.fsf@gnu.org> Date: Thu, 17 Feb 2022 12:17:14 +0100 Message-ID: <86sfsh4u9x.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1645096876; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=gULrkgwQ1fsi80ryHvjJkOsjzEJH9/6f7+CBn2KhQRo=; b=V4+/5x1KZlCH/e7bfmSWGvdVQSHizDpkTWxeuikAjk04M5aeANVbAczy+9jxhXmbB7E3Pd 8Vt22tZYZzwjjE6anpt0Xlkj8LIVyMZrgUjCNEKRLw47uXd48c3kYxY98xApuy3CIozsUp yYC9FPZMcfx++C6cnM4BF8p7TMVGcQvSn3AUHfuy41Qk8RUw0j0FxXiITxYCtFKw/A4fMP 8gkj4LhaeAD44vJ98jxRsxkBJQMQewDxQMR+1zCtu2sasAsHPxX6S87I+1B8orZuKhZwXL QXItYYKjJHS+Q4EvMaJtBwqjUXXYCjhw9LfK0dev4Fm4fGQ7fWlZJ0Xgarls5g== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1645096876; a=rsa-sha256; cv=none; b=K3Ul7s7ShpzshYju+ibORU9GniIDyHcLit3Tj8Nmp6FAL/DVgym22qokNT8ZHMZGOAMngk +efov9h+kaBnYziWGejG7vnk1C88pBcLBfvJoQITtV6r8BOZY8fwsta2YEsipaYhuUbgIN fYR9C9cIfDYFqRJX2xlj900g7U45asvqAF4nDvK/P5Q45rvsg0DwMAglJ5d3FEwyZtqz21 phpODLu9en0WCbRLRki2iSYQglpjJvG05u3Q0utlitLBmF6TepFL8pSyF4Qf5B/vw3BDJo /99WPwcCikpzsRCcZAvbZkSo3DxQ8CNAqP3GCEHJVz1vaqScqJv5JQQpie2PTA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=XDIBqvWK; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.03 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=XDIBqvWK; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 4F0CE16A94 X-Spam-Score: -2.03 X-Migadu-Scanner: scn1.migadu.com X-TUID: TgAzVXfpQr87 Hi, On Thu, 17 Feb 2022 at 11:35, Ludovic Court=C3=A8s wrote: >>> =E2=80=A2 guix refresh $(guix package -A ^emacs- | cut -f1) >> >> This one is interesting. This illustrates that the UI is, from my point >> of view, a bit lacking. It would be a nice improvement to add a regexp >> mechanism built-in, like in "guix search". > > Makes sense, we can do that. I agree the UI is not nice. Well, at the command line, I never read the complete output of =E2=80=9Cguix package -A=E2=80=9D and I always pipe it w= ith =E2=80=9Ccut -f1=E2=80=9D. Well, I think this complete display is only useful for third-party; the only one I have in mind is emacs-guix. Therefore, are we maintaining this CLI for backward compatibility when we could change both? Something more useful as output would be: name version synopsis Whatever. :-) Even the internal etc/completion/bash/guix has to pipe: --8<---------------cut here---------------start------------->8--- _guix_complete_available_package () { local prefix=3D"$1" if [ -z "$_guix_available_packages" ] then # Cache the complete list because it rarely changes and makes # completion much faster. _guix_available_packages=3D"$(${COMP_WORDS[0]} package -A 2> /dev/null \ | cut -f1)" fi COMPREPLY+=3D($(compgen -W "$_guix_available_packages" -- "$prefix")) } --8<---------------cut here---------------end--------------->8--- Last, I am not convinced that =E2=80=9Cguix search=E2=80=9D would be help h= ere. Because: 1. the output requires to pipe with recsel, 2. it is much slower than =E2=80=9Cpackage -A=E2=80=9D [1]. 1: >>> =E2=80=A2 guix refresh -m packages-i-care-about.scm >> >> Yes, obviously, this is a nice, too. However, it doesn't scale if you >> need to specify 1000+ packages. [...] >> In any case, this fails after reporting status of around 50 packages, >> with this time: >> >> real 0m41,881s >> user 0m12,155s >> sys 0m0,726s > > How does it fail? If it=E2=80=99s the GitHub rate limit, then there=E2= =80=99s only one > answer: you have to provide a token. Let mimick a collection if 1000+ packages I care about. Consider this manifest for packages using r-build-system only=E2=80=A6 --8<---------------cut here---------------start------------->8--- (use-modules (guix packages) (gnu packages) (guix build-system r)) (packages->manifest (fold-packages (lambda (package result) (if (eq? (package-build-system package) r-build-system) (cons package result) result)) '())) --8<---------------cut here---------------end--------------->8--- =E2=80=A6it hits the issue of Github token=E2=80=A6 --8<---------------cut here---------------start------------->8--- gnu/packages/bioconductor.scm:6034:13: 1.66.0 is already the latest version= of r-plgem gnu/packages/bioconductor.scm:6011:13: 1.22.0 is already the latest version= of r-rots gnu/packages/bioconductor.scm:12614:2: warning: 'bioconductor' updater fail= ed to determine available releases for r-fourcseq Backtrace: 13 (primitive-load "/home/simon/.config/guix/current/bin/guix") [...] ice-9/boot-9.scm:1685:16: In procedure raise-exception: Error downloading release information through the GitHub API. This may be fixed by using an access token and setting the environment variable GUIX_GITHUB_TOKEN, for instance one procured from https://github.com/settings/tokens real 10m27.306s user 4m14.077s sys 0m12.467s --8<---------------cut here---------------end--------------->8--- =E2=80=A6when most R packages come from CRAN or Bioconductor archives. Basically, ~5000 packages come from Github which represents ~25% of overall. Therefore, one needs to be really lucky when updating many package and not hit the Github rate limit. Yes, large collection of packages cannot be updated easily. Somehow, it is an issue from upstream and it is hard to fix=E2=80=A6 except by duplicating upstream or provide a token. :-) Well, using the external centralized Repology service is a first step to update at scale, no? A second step could be to have this feature included in the Data Service; but before we have other fishes to fry, IMHO. :-) >> Assuming I don't get the "rate limit exceeded" error, at this rate, it >> would take more than 15 minutes to check all the packages in >> "emacs-xyz.scm". This is a bit long. >> >> I don't see how this could reasonably be made faster without relying on >> an external centralized service doing the checks regularly (e.g., once >> a day) before the user actually requests them. > > Maybe you=E2=80=99re right, but before jumping to the conclusion, we have= to > investigate a bit. Like I wrote, the =E2=80=98gnu=E2=80=99 updater for i= nstance fetches > a single file that remains in cache afterwards=E2=80=94the cost is consta= nt. Repology acts as this =E2=80=9Cexternal centralized service=E2=80=9D, no? = On one hand, it is a practical solution; especially by being fast enough. On the other hand, it serves few false positives (say 4% to fix the ideas). Nicolas, considering the complexity of packages and their origins, do you think it would be possible to do better (fast and accurate) than Repology at scale? >>> =E2=80=A2 guix refresh -m packages-i-care-about.scm >> >> Yes, obviously, this is a nice, too. However, it doesn't scale if you >> need to specify 1000+ packages. > > You can use =E2=80=98fold-packages=E2=80=99 and have three lines that ret= urn a manifest > of 10K packages if you want it. Yes, see example above. >>> If not, what kind of selection mechanism could help? =E2=80=98-s=E2=80= =99 currently >>> accepts only two values, but we could augment it. >> >> Besides regexp matching, it may be useful to filter packages per module, >> or source file name. Package categories is a bit awkward, tho, and >> probably not satisfying. > > We can add options to make it more convenient, but it=E2=80=99s already > possible: Since these features are advanced, why not keep the CLI simple and instead on rely manifest files for complex filtering? >>> I realize this is going off-topic, but let=E2=80=99s see if we can impr= ove the >>> existing infrastructure to make it more convenient. [...] > I think we have nice infrastructure but you raise important > shortcomings. What Xinglu Chen did might in fact be one way to address > it, and there may also be purely UI issues that we could address. All the points raised here are important but appears to me orthogonal with the patch series. :-) Cheers, simon