[bug#53818] Improving updaters and ‘guix refresh’

[Nicolas Goaziou <mail@nicolasgoaziou.fr>]

Hello,

zimoun <zimon.toutoune@gmail.com> writes:

> On Thu, 17 Feb 2022 at 11:35, Ludovic Courtès <ludo@gnu.org> wrote:


>> How does it fail?  If it’s the GitHub rate limit, then there’s only one
>> answer: you have to provide a token.

IIUC, I have to register on GitHub to create this token. This is a bit
sad as a prerequisite to use one core feature of Guix.

> Let mimick a collection if 1000+ packages I care about.  Consider this
> manifest for packages using r-build-system only…
>
> --8<---------------cut here---------------start------------->8---
> (use-modules (guix packages)
>              (gnu packages)
>              (guix build-system r))
>
> (packages->manifest
>  (fold-packages (lambda (package result)
>                   (if (eq? (package-build-system package) r-build-system)
>                       (cons package result)
>                       result))
>                 '()))
> --8<---------------cut here---------------end--------------->8---

I have to learn about fold-packages.

> Nicolas, considering the complexity of packages and their origins, do
> you think it would be possible to do better (fast and accurate) than
> Repology at scale?

It's not about doing better, but doing differently. We do not need all
of Repology's features. 

As far as the updater part is concerned, Repology pokes at various
package repositories, which are usually not upstream, extracts package
versions, applies some heuristics to normalize and compare them, then
decides what is the newest version and which repositories provide
outdated packages. This has a two obvious shortcomings:

1. the version number usually doesn't come from an official source, so
   it may be wrong—e.g., our emacs-csv-mode is "outdated" because Funtoo
   1.4 chose a non-existing higher version number for the same package.

2. version comparison does not understand every local versioning
   scheme—e.g., our emacs-fold-dwim packages is currently at
   "1.2-0.c46f4bb", which is, in Guix parlance, after "1.2", yet
   Repology thinks this is actually older than "1.2".

Therefore, I think a (theoretical) centralized Guix-centric version
checker could be fast: it would only poke at what our packages consider
to be upstream, and accurate, since it would know about our versioning
rules. Basically it could boil down to calling current "guix refresh" on
every package daily, and serializing the results.

>>>>   • guix refresh -m packages-i-care-about.scm
>>>
>>> Yes, obviously, this is a nice, too. However, it doesn't scale if you
>>> need to specify 1000+ packages.
>>
>> You can use ‘fold-packages’ and have three lines that return a manifest
>> of 10K packages if you want it.
>
> Yes, see example above.

Point taken.

Regards,
-- 
Nicolas Goaziou