From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id kGtqA1/DA2JrOgAAgWs5BA (envelope-from ) for ; Wed, 09 Feb 2022 14:36:31 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id uMSgN17DA2Lu9AAAG6o9tA (envelope-from ) for ; Wed, 09 Feb 2022 14:36:30 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2B59813EEF for ; Wed, 9 Feb 2022 14:36:30 +0100 (CET) Received: from localhost ([::1]:49302 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nHn90-0003tS-Tu for larch@yhetil.org; Wed, 09 Feb 2022 08:36:26 -0500 Received: from eggs.gnu.org ([209.51.188.92]:56324) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nHmyz-00059R-4V for guix-patches@gnu.org; Wed, 09 Feb 2022 08:26:06 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:56304) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nHmyw-0004hD-DV for guix-patches@gnu.org; Wed, 09 Feb 2022 08:26:04 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nHmyw-00079L-B6; Wed, 09 Feb 2022 08:26:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#53818] [PATCH v3 5/7] import: Add 'repology' updater. Resent-From: Xinglu Chen Original-Sender: "Debbugs-submit" Resent-CC: maximedevos@telenet.be, guix-patches@gnu.org Resent-Date: Wed, 09 Feb 2022 13:26:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 53818 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 53818@debbugs.gnu.org Cc: Maxime Devos X-Debbugs-Original-Xcc: Maxime Devos Received: via spool by 53818-submit@debbugs.gnu.org id=B53818.164441312627387 (code B ref 53818); Wed, 09 Feb 2022 13:26:02 +0000 Received: (at 53818) by debbugs.gnu.org; 9 Feb 2022 13:25:26 +0000 Received: from localhost ([127.0.0.1]:50187 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nHmyI-00077Z-1F for submit@debbugs.gnu.org; Wed, 09 Feb 2022 08:25:26 -0500 Received: from h178-251-242-94.cust.a3fiber.se ([178.251.242.94]:44880 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nHmyG-00077J-5n for 53818@debbugs.gnu.org; Wed, 09 Feb 2022 08:25:21 -0500 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yoctocell.xyz; s=mail; t=1644413113; bh=QJPtJwE9OIg++PiCiG6RNGbOTMFbkV9On/88KoOPwYk=; h=From:To:Subject:In-Reply-To:References:Date; b=Od2NqCRJsnDuAOOVNtc6UkS14BlrtgucrnL0jQJ5tGq5AGj2O2VVNScxLZP0toQhu YVUXvhatGK+HdipY+Kv3tRUknMRPVEt88qkn/e0Og/iAr6UR1NIr4I+A7uD2P6B+7Y CmZsLyoehp/qd4b2/cEBVc0OODfeQwF5jYBZFUeA= In-Reply-To: References: Message-Id: <452009a171299629965ab860eac2a1fdbe8a3554.1644412701.git.public@yoctocell.xyz> Date: Wed, 09 Feb 2022 14:25:13 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1644413790; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=/flF/eLkg/gGFKry3+eThg4pJAGTmc4I8rduTtOfcXk=; b=CRP1yE9n9MgOimiIQ9VHp8tJQfueqykje9aDwTP0MHx7TPG+jT637LxGRJYsDo6K/d17EI ocDToORbosJ/YHis4fI5ruwfLCV8IuAIQjGdLIc1BseXWo7B4iK4GGwJPziJsG0t5+tn9i /Y7awL12B6KsA2rx20dOWfS9vxWTk2XcFhKHMOKD6+OL72wrapYeCEsLX6MBgMdZr6aflp xKJwbHuMvCPCkDxF1nmblgbCyB4u5UjqT+P6uNL4YoJROT29IAKFXJOn+oMoy/ObvBZ6rM Gb1EC2Z5HBSU58AYwIFoeUy1txA2nACCyx/m1Wh71uPwkXVLZkthrMhf1b2l4A== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1644413790; a=rsa-sha256; cv=none; b=tamezAoVTONQvaIJFNTQyIshhh+AbY5jLRZUHmwYMiPDQZ274zVU4qBpOXcPU1vhoHbO28 J4KzzoWZnXcmCokvHU2BqQknooXQn56Jr1gUFPF+9X4dKovl5p4T0kahxzm+7hwEDezIIm 7RzArnT1h1i8R+mQNX7J6Zav/rA6kuKr7zBS040xTDtEAZHedV/7aRG7COoVoN1Ty1QDcR MoFIrxpBDc05a1dwE/xGK+I1XxN0TTb1+WbXJZUYp9Dk6O89G3nqWLCXbZMJThIDC6D1iG AYdfVGrlisApSl51C3R0yuUVtEU8CMCpHG3t4nc04VWL9L55xCLz5FsvgoVWRg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=yoctocell.xyz header.s=mail header.b=Od2NqCRJ; dmarc=fail reason="SPF not aligned (relaxed)" header.from=yoctocell.xyz (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.03 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=yoctocell.xyz header.s=mail header.b=Od2NqCRJ; dmarc=fail reason="SPF not aligned (relaxed)" header.from=yoctocell.xyz (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 2B59813EEF X-Spam-Score: -1.03 X-Migadu-Scanner: scn1.migadu.com X-TUID: HkN2Ds3eT87h * guix/import/repology.scm * tests/import-repology.scm: New files. * Makefile.am (MODULES): Register them. * doc/guix.texi (Invoking guix refresh): Document it. --- Makefile.am | 3 + doc/guix.texi | 8 ++ guix/import/repology.scm | 249 ++++++++++++++++++++++++++++++++++++++ tests/import-repology.scm | 150 +++++++++++++++++++++++ 4 files changed, 410 insertions(+) create mode 100644 guix/import/repology.scm create mode 100644 tests/import-repology.scm diff --git a/Makefile.am b/Makefile.am index 7463606d20..6792917b59 100644 --- a/Makefile.am +++ b/Makefile.am @@ -16,6 +16,7 @@ # Copyright =C2=A9 2019 Efraim Flashner # Copyright =C2=A9 2021 Chris Marusich # Copyright =C2=A9 2021 Andrew Tropin +# Copyright =C2=A9 2022 Xinglu Chen # # This file is part of GNU Guix. # @@ -271,6 +272,7 @@ MODULES =3D \ guix/import/opam.scm \ guix/import/print.scm \ guix/import/pypi.scm \ + guix/import/repology.scm \ guix/import/stackage.scm \ guix/import/texlive.scm \ guix/import/utils.scm \ @@ -488,6 +490,7 @@ SCM_TESTS =3D \ tests/home-import.scm \ tests/import-git.scm \ tests/import-github.scm \ + tests/import-repology.scm \ tests/import-utils.scm \ tests/inferior.scm \ tests/lint.scm \ diff --git a/doc/guix.texi b/doc/guix.texi index 583ba1c61d..2d7612b09a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -12932,6 +12932,14 @@ (release-tag-version-delimiter . ":")))) @end lisp =20 +@item repology +an updater that scans @uref{https://repology.org, Repology}, a website +that tracks packages on various package repositories, for updates. + +The name of a package in Guix is not always that same as the name on +Repology. In most cases, the updater will be able to guess the name +correctly. If it doesn=E2=80=99t, users can set the @code{repology-name} +package property. =20 @end table =20 diff --git a/guix/import/repology.scm b/guix/import/repology.scm new file mode 100644 index 0000000000..87fbd2ee6f --- /dev/null +++ b/guix/import/repology.scm @@ -0,0 +1,249 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2022 Xinglu Chen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix import repology) + #:use-module (guix diagnostics) + #:use-module (guix git-download) + #:use-module (guix http-client) + #:use-module (guix i18n) + #:use-module (guix import json) + #:use-module (guix import utils) + #:use-module (guix memoization) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module (ice-9 match) + #:use-module (json) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-2) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-43) + #:export (%repology-url + repology-fetch-info + repology-latest-release + %repology-updater)) + +;;; Commentary: +;;; +;;; This module provides an updater which scans Repology, a site that moni= tors +;;; several package repolsitories, for updates. This means that if any ot= her +;;; package repository has a version of a package that is newer than the +;;; version in Guix, the package should be able to be updated. The updater +;;; should in theory work for all packages in Guix, but the names of some +;;; packages on Repology don't match the name in Guix. The 'repology-name' +;;; package property can be used to fix this. +;;; +;;; Guix already has many different updaters for language-specific package= s, +;;; and these typically provide more accurate data, e.g., input changes, +;;; signature URLs. The Repology updater should really be treated as a la= st +;;; resort for those packages that don't have any other updater to rely on. +;;; +;;; See for the API. +;;; +;;; Code: + +(define %repology-url + "https://repology.org/api/v1/project") + +(define* (package-name->repology-name name #:key (attempt 1)) + "Convert NAME, the name of a Guix package, to the name of the package on +Repology. It doesn't always guess the correct name on the first attempt, = so +on the second attempt it will try to guess another name." + (match attempt + (1 (cond + ((string-prefix? "ghc-" name) + (string-append "haskell:" + (string-drop name (string-length "ghc-")))) + ((string-prefix? "ocaml-" name) + (string-append "ocaml:" + (string-drop name (string-length "ocaml-")))) + ((string-prefix? "perl-" name) + (string-append "perl:" + (string-drop name (string-length "perl-")))) + ((string-prefix? "emacs-" name) + (string-append "emacs:" + (string-drop name (string-length "emacs-")))) + ((string-prefix? "go-" name) + (string-append "go:" + (string-drop name (string-length "go-")))) + ((string-prefix? "rust-" name) + (string-append "rust:" + (string-drop name (string-length "rust-")))) + ((string-prefix? "lua-" name) + (string-append "lua:" + (string-drop name (string-length "lua-")))) + ((string-prefix? "node-" name) + (string-append "node:" + (string-drop name (string-length "node-")))) + ((string-prefix? "python-" name) + (string-append "python:" + (string-drop name (string-length "python-")))) + ((string-prefix? "java-" name) + (string-append "java:" + (string-drop name (string-length "java-")))) + ((string-prefix? "r-" name) + (string-append "r:" + (string-drop name (string-length "r-")))) + ((string-prefix? "ruby-" name) + (string-append "ruby:" + (string-drop name (string-length "ruby-")))) + ((string-prefix? "xf86-" name) + (string-append "xdrv:" + (string-drop name (string-length "xf86-")))) + ((string-prefix? "font-" name) + (string-append "fonts:" + (string-drop name (string-length "font-")))) + ((string-prefix? "trytond-" name) + (string-append "tryton:" + (string-drop name (string-length "trytond-")))) + ((string-prefix? "python-trytond-" name) + (string-append "tryton:" + (string-drop name (string-length "python-trytond-"= )))) + ((string-suffix? "-minimal" name) + (string-drop-right name (string-length "-minimal")))=20=20=20=20= =20=20=20=20 + (else name))) + (2 (cond + ((string-prefix? "xf86-video" name) + (string-append "xdrv:" + (string-drop name (string-length "xf86-video-")))) + ((string-prefix? "xf86-input" name) + (string-append "xdrv:" + (string-drop name (string-length "xf86-input-")))) + ((string-prefix? "minetest-" name) + (string-append "minetest-mod-" + (string-drop name (string-length "minetest-")))) + ((string-prefix? "lib" name) + (string-drop name (string-length "lib"))) + ((string-prefix? "vim-" name) + (string-append "vim:" + (string-drop name (string-length "vim-")))) + (else name)))))=20=20 + + +;;; JSON mappings. + +(define-json-mapping make-repology-package + repology-package? + json->repology-package + (repository repology-package-repository "repo") + (src-name repology-package-src-name "srcname") + (binary-name repology-package-binary-name "binname") + (visible-name repology-package-visible-name "visiblename") + (version repology-package-version) + (original-version repology-package-original-version "origversion") + (status repology-package-status) + (summary repology-package-summary) + (categories repology-package-categories) + (licenses repology-package-licenses) + (maintainers repology-package-maintainers)) + + +;;; Updater. + +(define repology-fetch-info + (memoize + (lambda (package) + "Fetch information about PACKAGE using the Repology API." + (define (name->info name) + (let ((url (string-append %repology-url "/" name))) + (and=3D> (json-fetch url #:http-fetch http-fetch/cached) + (lambda (url) + (vector-map (lambda (a b) + (json->repology-package b)) + url))))) +=20=20=20=20=20 + (let* ((name (or (assoc-ref (package-properties package) + 'repology-name) + (package-name->repology-name (package-name package))= )) + (info (name->info name))) + (if (and info (not (vector-empty? info))) + info + (let ((info (name->info (package-name->repology-name + (package-name package) + #:attempt 2)))) + (if (and info (not (vector-empty? info))) + info + (begin + (warning (G_ "package not found on Repology: ~a\n") + (package-name package)) + #f)))))))) + +(define (update-version string old-version new-version) + "Replace OLD-VERSION in STRING with NEW-VERSION. This assumes that STRI= NG +contains OLD-VERSION verbatim; if it doesn't, #f is returned." + (match (factorize-uri string old-version) + ((? string?) #f) + ((factorized ...) + (apply string-append + (map (lambda (component) + (match component + ('version new-version) + ((? string?) component))) + factorized))))) + +(define (package-source-urls package version) + "Return a list of URLs for PACKAGE at VERSION. If no URL was successful= ly constructed, return #f." + (and-let* ((old-version (package-version package)) + (source (package-source package))) + ;; XXX: (guix upstream) only supports tarballs and Git repos for now. + (match (origin-uri source) + ((? git-reference? reference) + (and-let* ((old-commit (git-reference-commit reference)) + (new-commit (if (string=3D? old-version old-commit) + version + (update-version old-commit + old-version + version)))) + (git-reference + (inherit reference) + (commit new-commit)))) + ((? string? url) + (list (update-version url old-version version))) + ((? list? urls) + (map (cut update-version <> old-version version) urls)) + (_ #f)))) + +(define (latest-version? repology-package) + "Return the latest released version of REPOLOGY-PACKAGE. If none are fo= und, +return #f." + (and (or (equal? "newest" (repology-package-status repology-package)) + (equal? "unique" (repology-package-status repology-package))) + (repology-package-version repology-package))) + +;; XXX: We use 'pkg' because 'package' will clash with the 'package' field= of +;; 'upstream-source'. +(define (repology-latest-release pkg) + "Return the latest release of the PKG on Repology named NAME." + (and-let* ((packages (repology-fetch-info pkg)) + (versions (filter-map latest-version? + (vector->list packages))) + (latest-version (and (pair? versions) (car versions)))) + ;; TODO: set 'signature-urls'. + (upstream-source + (package (package-name pkg)) + (version latest-version) + (urls (package-source-urls pkg latest-version))))) + +(define %repology-updater + (upstream-updater + (name 'repology) + (description "Updater for packages on Repology") + (pred (const #t)) + (latest repology-latest-release))) + +;;; repology.scm ends here diff --git a/tests/import-repology.scm b/tests/import-repology.scm new file mode 100644 index 0000000000..4da01a4106 --- /dev/null +++ b/tests/import-repology.scm @@ -0,0 +1,150 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2022 Xinglu Chen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (test-import-repology) + #:use-module (guix download) + #:use-module (guix git-download) + #:use-module (guix import repology) + #:use-module (guix memoization) + #:use-module (guix packages) + #:use-module (guix tests) + #:use-module (guix upstream) + #:use-module (json) + #:use-module (srfi srfi-64)) + +(test-begin "repology") + +(define package-using-git-repository + (dummy-package + "foo" + (version "1.0") + (source + (origin + (method git-fetch) + (uri (git-reference + (url "https://git.example.org/foo") + (commit "1.0"))) + (sha256 #f))))) + +(define package-using-tarball + (dummy-package + "foo" + (version "1.0") + (source + (origin + (method git-fetch) + (uri (string-append "https://example.org/foo-" version ".tar.gz")) + (sha256 #f))))) + +(define package-using-tarball-multiple-urls + (dummy-package + "foo" + (version "1.0") + (source + (origin + (method git-fetch) + (uri (list (string-append "https://example.org/foo-" + version ".tar.gz") + (string-append "https://mirror.example.org/foo-" + version ".tar.gz"))) + (sha256 #f))))) + +(define %test-json +"[ + { + \"repo\": \"aur\", + \"srcname\": \"foo\", + \"binname\": \"foo\", + \"visiblename\": \"foo\", + \"version\": \"1.0.r25.gb86405a\", + \"maintainers\": [ + \"bob@aur\" + ], + \"licenses\": [ + \"LGPL3+\" + ], + \"summary\": \"foo bar\" + \"status\": \"rolling\", + \"origversion\": \"1.0.r25.gb86405a-1\" + }, + { + \"repo\": \"gnuguix\", + \"srcname\": \"foo\", + \"binname\": \"foo\", + \"visiblename\": \"foo\", + \"version\": \"1.0\", + \"summary\": \"foo bar\", + \"status\": \"outdated\", + \"origversion\": null + }, + { + \"repo\": \"nix_unstable\", + \"name\": \"foo\", + \"visiblename\": \"foo\", + \"version\": \"2.0\", + \"maintainers\": [ + \"bob@example.org\" + ], + \"licenses\": [ + \"LGPL-3.0-or-later\" + ], + \"summary\": \"foo bar\", + \"status\": \"newest\", + \"origversion\": null + } +]") + +(define (latest-release package) + (invalidate-memoization! repology-fetch-info) + (mock ((guix import json) json-fetch + (lambda* (url #:key http-fetch) + (if (string=3D? url + (string-append %repology-url "/foo")) + (json-string->scm %test-json) + (error "the URL is not correct")))) + (repology-latest-release package))) + +(test-equal "package using Git repo: version" + "2.0" + (upstream-source-version + (latest-release package-using-git-repository))) + +(test-equal "package using Git repo: git-reference" + (git-reference + (url "https://git.example.org/foo") + (commit "2.0")) + (upstream-source-urls + (latest-release package-using-git-repository))) + +(test-equal "package using tarball: version" + "2.0" + (upstream-source-version + (latest-release package-using-tarball))) + +(test-equal "package using tarball: URL" + (list "https://example.org/foo-2.0.tar.gz") + (upstream-source-urls + (latest-release package-using-tarball))) + +(test-equal "package using tarball: multiple URLs" + (list "https://example.org/foo-2.0.tar.gz" + "https://mirror.example.org/foo-2.0.tar.gz") + (upstream-source-urls + (latest-release package-using-tarball-multiple-urls))) + +(test-end "repology") --=20 2.34.1