From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 6NLCDtDrAGKBfQEAgWs5BA (envelope-from ) for ; Mon, 07 Feb 2022 10:52:16 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id GNI7DNDrAGI0yQAA9RJhRA (envelope-from ) for ; Mon, 07 Feb 2022 10:52:16 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6ED35432B5 for ; Mon, 7 Feb 2022 10:52:15 +0100 (CET) Received: from localhost ([::1]:60868 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nH0gw-0006RB-J0 for larch@yhetil.org; Mon, 07 Feb 2022 04:52:14 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40376) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nH00B-00068F-R1 for guix-patches@gnu.org; Mon, 07 Feb 2022 04:08:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:47044) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nH00B-0004LJ-04 for guix-patches@gnu.org; Mon, 07 Feb 2022 04:08:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nH00A-0002vA-SZ; Mon, 07 Feb 2022 04:08:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#53818] [PATCH v2 5/7] import: Add 'repology' updater. Resent-From: Xinglu Chen Original-Sender: "Debbugs-submit" Resent-CC: maximedevos@telenet.be, guix-patches@gnu.org Resent-Date: Mon, 07 Feb 2022 09:08:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 53818 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 53818@debbugs.gnu.org Cc: Maxime Devos X-Debbugs-Original-Xcc: Maxime Devos Received: via spool by 53818-submit@debbugs.gnu.org id=B53818.164422484111131 (code B ref 53818); Mon, 07 Feb 2022 09:08:02 +0000 Received: (at 53818) by debbugs.gnu.org; 7 Feb 2022 09:07:21 +0000 Received: from localhost ([127.0.0.1]:40929 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nGzzT-0002tM-Vk for submit@debbugs.gnu.org; Mon, 07 Feb 2022 04:07:20 -0500 Received: from h178-251-242-94.cust.a3fiber.se ([178.251.242.94]:55300 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nGzzO-0002si-3V for 53818@debbugs.gnu.org; Mon, 07 Feb 2022 04:07:15 -0500 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yoctocell.xyz; s=mail; t=1644224828; bh=tM2bgZ0c95q9tm79KzPQw0F7/5o5pTKGvJyzdbNuSOQ=; h=From:To:Subject:In-Reply-To:References:Date; b=k/rcqoMhqU81atv1BjUpLmBtl6WwkMUJjnOrdcweAuAQsGnnQ9XTMbQTNKguz+MI+ rEQTYKS+ChBFYRSfA9vNuph3Dhem3f8Cksa7Xls8IPMoz38nYD0G1MdmJaPN5u3Rzy Z0QlTySUbpa7GIV3gORwRtZdEazJB9vhM/MIMftw= In-Reply-To: References: Message-Id: <98726379214702d0745f56d9f946e792e803d326.1644224421.git.public@yoctocell.xyz> Date: Mon, 07 Feb 2022 10:07:07 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1644227535; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=QCs+V7FylXMII1muXHWSzlQFlb1lcLGAqHLIU2cvPX4=; b=pEzLu3KV+MOZoZoc/D1D6lebuQmp4WoruPWAIElZQtxx8XdTGRwLrzIVC9Iw07jwHrDnQl GgPfNJnJv5x2kjgeyYLa17yyFhVyQM5o4PirbQCK/fvgmuq+MXKMvCb7HmTW+6mL9TmNQb mLNPhG3iVhxaCwAS5NzeETeiOHC36Qzv3ZK++EuBK+DnPmD8tjzbAhxMQ88o1oLV/F6dGP yecMcEM28w3hkAVQt5e3EHQVWXd08m9Aq9vZb81hq7OXeksUT1v8apYtuqQvxniw94RGLQ 0bRuMQpH+i+eH2Nfzey+YdMrH6l4h2/Dcp2zJFdrE2m9A7JTGrjDpOGpXcQYiQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1644227535; a=rsa-sha256; cv=none; b=UdgyfykpGO3ulFLjrLIHnuz3ydUgFFG9WA6QDVrf53IiFkTNQVhurXMqWsTFSd5kTkPwHL itEVF6c2BPWVCGE/RQrTnQ7zVtad93MgK7itSj1nSTPOpUQpBo+ytiHAecBBTp9Qofxxvy AngpDTektvUfVs55JmtxrPH1U5mv5uoBK0C8iyL95OsrR3brx30QnolvEZ0uI4CrMFO5QI /+GGfn0FATFyFrmTqqaJsJPxbdy+TFy+e5N6YfXglmZUtBulme8BBsT8q7khMKpi9plsH0 pgyMu7iIj89fLeOUr6a4Ok4hyxHop1bZkilAZrrM5jCBmVuSjFMmmtlLpX86Ww== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=yoctocell.xyz header.s=mail header.b="k/rcqoMh"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=yoctocell.xyz (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.03 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=yoctocell.xyz header.s=mail header.b="k/rcqoMh"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=yoctocell.xyz (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 6ED35432B5 X-Spam-Score: -1.03 X-Migadu-Scanner: scn1.migadu.com X-TUID: +QKG909OMLCU * guix/import/repology.scm * tests/import-repology.scm: New files. * Makefile.am (MODULES): Register them. * doc/guix.texi (Invoking guix refresh): Document it. --- Makefile.am | 3 + doc/guix.texi | 7 ++ guix/import/repology.scm | 235 ++++++++++++++++++++++++++++++++++++++ tests/import-repology.scm | 145 +++++++++++++++++++++++ 4 files changed, 390 insertions(+) create mode 100644 guix/import/repology.scm create mode 100644 tests/import-repology.scm diff --git a/Makefile.am b/Makefile.am index 7463606d20..6792917b59 100644 --- a/Makefile.am +++ b/Makefile.am @@ -16,6 +16,7 @@ # Copyright =C2=A9 2019 Efraim Flashner # Copyright =C2=A9 2021 Chris Marusich # Copyright =C2=A9 2021 Andrew Tropin +# Copyright =C2=A9 2022 Xinglu Chen # # This file is part of GNU Guix. # @@ -271,6 +272,7 @@ MODULES =3D \ guix/import/opam.scm \ guix/import/print.scm \ guix/import/pypi.scm \ + guix/import/repology.scm \ guix/import/stackage.scm \ guix/import/texlive.scm \ guix/import/utils.scm \ @@ -488,6 +490,7 @@ SCM_TESTS =3D \ tests/home-import.scm \ tests/import-git.scm \ tests/import-github.scm \ + tests/import-repology.scm \ tests/import-utils.scm \ tests/inferior.scm \ tests/lint.scm \ diff --git a/doc/guix.texi b/doc/guix.texi index 0cf865a672..15d215dd48 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -12932,6 +12932,13 @@ (release-tag-version-delimiter . ":")))) @end lisp =20 +@item repology +an updater that scans @uref{https://repology.org, Repology}, a website +that tracks packages on various package repositories, for updates. + +The name of a package in Guix is not always that same as the name on +Repology; users can set the @code{repology-name} package property to +make the updater use the correct name. =20 @end table =20 diff --git a/guix/import/repology.scm b/guix/import/repology.scm new file mode 100644 index 0000000000..28f3a3af5f --- /dev/null +++ b/guix/import/repology.scm @@ -0,0 +1,235 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2022 Xinglu Chen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix import repology) + #:use-module (guix diagnostics) + #:use-module (guix diagnostics) + #:use-module (guix git-download) + #:use-module (guix i18n) + #:use-module (guix import json) + #:use-module (guix import utils) + #:use-module (guix memoization) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module (ice-9 match) + #:use-module (json) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-2) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-43) + #:export (repology-latest-release + %repology-updater)) + +;;; Commentary: +;;; +;;; This module provides an updater which scans Repology, a site that moni= tors +;;; several package repolsitories, for updates. This means that if any ot= her +;;; package repository has a version of a package that is newer than the +;;; version in Guix, the package should be able to be updated. The updater +;;; should in theory work for all packages in Guix, but the names of some +;;; packages on Repology don't match the name in Guix. The 'repology-name' +;;; package property can be used to fix this. +;;; +;;; Guix already has many different updaters for language-specific package= s, +;;; and these typically provide more accurate data, e.g., input changes, +;;; signature URLs. The Repology updater should really be treated as a la= st +;;; resort for those packages that don't have any other updater to rely on. +;;; +;;; See for the API. +;;; +;;; Code: + +(define %repology-url + "https://repology.org/api/v1/project") + +(define* (package-name->repology-name name #:key (attempt 1)) + "Convert NAME, the name of a Guix package, to the name of the package on +Repology. It doesn't always guess the correct name on the first attempt, = so +on the second attempt it will try to guess another name." + (match attempt + (1 (cond + ((string-prefix? "ghc-" name) + (string-append "haskell:" + (string-drop name (string-length "ghc-")))) + ((string-prefix? "ocaml-" name) + (string-append "ocaml:" + (string-drop name (string-length "ocaml-")))) + ((string-prefix? "perl-" name) + (string-append "perl:" + (string-drop name (string-length "perl-")))) + ((string-prefix? "emacs-" name) + (string-append "emacs:" + (string-drop name (string-length "emacs-")))) + ((string-prefix? "go-" name) + (string-append "go:" + (string-drop name (string-length "go-")))) + ((string-prefix? "rust-" name) + (string-append "rust:" + (string-drop name (string-length "rust-")))) + ((string-prefix? "lua-" name) + (string-append "lua:" + (string-drop name (string-length "lua-")))) + ((string-prefix? "node-" name) + (string-append "node:" + (string-drop name (string-length "node-")))) + ((string-prefix? "python-" name) + (string-append "python:" + (string-drop name (string-length "python-")))) + ((string-prefix? "java-" name) + (string-append "java:" + (string-drop name (string-length "java-")))) + ((string-prefix? "r-" name) + (string-append "r:" + (string-drop name (string-length "r-")))) + ((string-prefix? "ruby-" name) + (string-append "ruby:" + (string-drop name (string-length "ruby-")))) + ((string-prefix? "xf86-" name) + (string-append "xdrv:" + (string-drop name (string-length "xf86-")))) + ((string-prefix? "font-" name) + (string-append "fonts:" + (string-drop name (string-length "font-")))) + ((string-suffix? "-minimal" name) + (string-drop-right name (string-length "-minimal"))) + (else name))) + (2 (cond + ((string-prefix? "xf86-video" name) + (string-append "xdrv:" + (string-drop name (string-length "xf86-video-")))) + ((string-prefix? "xf86-input" name) + (string-append "xdrv:" + (string-drop name (string-length "xf86-input-")))) + ((string-prefix? "minetest-" name) + (string-append "minetest-mod-" + (string-drop name (string-length "minetest-")))) + (else name)))))=20=20 + + +;;; JSON mappings. + +(define-json-mapping make-repology-package + repology-package? + json->repology-package + (repository repology-package-repository "repo") + (src-name repology-package-src-name "srcname") + (binary-name repology-package-binary-name "binname") + (visible-name repology-package-visible-name "visiblename") + (version repology-package-version) + (original-version repology-package-original-version "origversion") + (status repology-package-status) + (summary repology-package-summary) + (categories repology-package-categories) + (licenses repology-package-licenses) + (maintainers repology-package-maintainers)) + + +;;; Updater. + +(define repology-fetch-info + (memoize + (lambda (package) + "Fetch information about PACKAGE using the Repology API." + (define (name->info name) + (let ((url (string-append %repology-url "/" name))) + (and=3D> (json-fetch url #:cached? #t) + (lambda (url) + (vector-map (lambda (a b) + (json->repology-package b)) + url))))) +=20=20=20=20=20 + (let* ((name (or (assoc-ref (package-properties package) + 'repology-name) + (package-name->repology-name (package-name package))= )) + (info (name->info name))) + (if (and info (not (vector-empty? info))) + info + (let ((info (name->info (package-name->repology-name + (package-name package) + #:attempt 2)))) + (if (and info (not (vector-empty? info))) + info + (begin + (warning (G_ "package not found on Repology: ~a\n") + (package-name package)) + #f)))))))) + +(define (update-version string old-version new-version) + "Replace OLD-VERSION in STRING with NEW-VERSION. This assumes that STRI= NG +contains OLD-VERSION verbatim; if it doesn't, #f is returned." + (match (factorize-uri string old-version) + ((? string?) #f) + ((factorized ...) + (apply string-append + (map (lambda (component) + (match component + ('version new-version) + ((? string?) component))) + factorized))))) + +(define (package-source-urls package version) + "Return a list of URLs for PACKAGE at VERSION. If no URL was successful= ly constructed, return #f." + (let ((old-version (package-version package))) + ;; XXX: (guix upstream) only supports tarballs and Git repos for now. + (match (origin-uri (package-source package)) + ((? git-reference? reference) + (and-let* ((old-commit (git-reference-commit reference)) + (new-commit (if (string=3D? old-version old-commit) + version + (update-version old-commit + old-version + version)))) + (git-reference + (inherit reference) + (commit new-commit)))) + ((? string? url) + (list (update-version url old-version version))) + ((? list? urls) + (map (cut update-version <> old-version version) urls)) + (_ #f)))) + +(define (latest-version? repology-package) + "Return the latest released version of REPOLOGY-PACKAGE. If none are fo= und, +return #f." + (and (or (equal? "newest" (repology-package-status repology-package)) + (equal? "unique" (repology-package-status repology-package))) + (repology-package-version repology-package))) + +;; XXX: We use 'pkg' because 'package' will clash with the 'package' field= of +;; 'upstream-source'. +(define (repology-latest-release pkg) + "Return the latest release of the PKG on Repology named NAME." + (and-let* ((packages (repology-fetch-info pkg)) + (versions (filter-map latest-version? + (vector->list packages))) + (latest-version (and (pair? versions) (car versions)))) + ;; TODO: set 'signature-urls'. + (upstream-source + (package (package-name pkg)) + (version latest-version) + (urls (package-source-urls pkg latest-version))))) + +(define %repology-updater + (upstream-updater + (name 'repology) + (description "Updater for packages on Repology") + (pred (const #t)) + (latest repology-latest-release))) + +;;; repology.scm ends here diff --git a/tests/import-repology.scm b/tests/import-repology.scm new file mode 100644 index 0000000000..2d366db283 --- /dev/null +++ b/tests/import-repology.scm @@ -0,0 +1,145 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2022 Xinglu Chen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (test-import-repology) + #:use-module (guix download) + #:use-module (guix git-download) + #:use-module (guix import repology) + #:use-module (guix packages) + #:use-module (guix tests) + #:use-module (guix upstream) + #:use-module (json) + #:use-module (srfi srfi-64)) + +(test-begin "repology") + +(define package-using-git-repository + (dummy-package + "foo" + (version "1.0") + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://git.example.org/foo") + (commit "1.0"))) + (sha256 #f))))) + +(define package-using-tarball + (dummy-package + "foo" + (version "1.0") + (source + (origin + (method git-fetch) + (uri (string-append "https://example.org/foo-" version ".tar.gz")) + (sha256 #f))))) + +(define package-using-tarball-multiple-urls + (dummy-package + "foo" + (version "1.0") + (source + (origin + (method git-fetch) + (uri (list (string-append "https://example.org/foo-" + version ".tar.gz") + (string-append "https://mirror.example.org/foo-" + version ".tar.gz"))) + (sha256 #f))))) + +(define %test-json +"[ + { + \"repo\": \"aur\", + \"srcname\": \"foo\", + \"binname\": \"foo\", + \"visiblename\": \"foo\", + \"version\": \"1.0.r25.gb86405a\", + \"maintainers\": [ + \"bob@aur\" + ], + \"licenses\": [ + \"LGPL3+\" + ], + \"summary\": \"foo bar\" + \"status\": \"rolling\", + \"origversion\": \"1.0.r25.gb86405a-1\" + }, + { + \"repo\": \"gnuguix\", + \"srcname\": \"foo\", + \"binname\": \"foo\", + \"visiblename\": \"foo\", + \"version\": \"1.0\", + \"summary\": \"foo bar\", + \"status\": \"outdated\", + \"origversion\": null + }, + { + \"repo\": \"nix_unstable\", + \"name\": \"foo\", + \"visiblename\": \"foo\", + \"version\": \"2.0\", + \"maintainers\": [ + \"bob@example.org\" + ], + \"licenses\": [ + \"LGPL-3.0-or-later\" + ], + \"summary\": \"foo bar\", + \"status\": \"newest\", + \"origversion\": null + } +]") + +(define (latest-release package) + (mock ((guix import json) json-fetch + (lambda* (url #:key cached?) + (json-string->scm %test-json))) + (repology-latest-release package))) + +(test-equal "package using Git repo: version" + "2.0" + (upstream-source-version + (latest-release package-using-git-repository))) + +(test-equal "package using Git repo: git-reference" + (git-reference + (url "https://git.example.org/foo") + (commit "2.0")) + (upstream-source-urls + (latest-release package-using-git-repository))) + +(test-equal "package using tarball: version" + "2.0" + (upstream-source-version + (latest-release package-using-tarball))) + +(test-equal "package using tarball: URL" + (list "https://example.org/foo-2.0.tar.gz") + (upstream-source-urls + (latest-release package-using-tarball))) + +(test-equal "package using tarball: multiple URLs" + (list "https://example.org/foo-2.0.tar.gz" + "https://mirror.example.org/foo-2.0.tar.gz") + (upstream-source-urls + (latest-release package-using-tarball-multiple-urls))) + +(test-end "repology") --=20 2.34.1