From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id WG7LCZ4hFmHCDwEAgWs5BA (envelope-from ) for ; Fri, 13 Aug 2021 09:39:10 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id +CpfBZ4hFmEIeQAAbx9fmQ (envelope-from ) for ; Fri, 13 Aug 2021 07:39:10 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id F156F2CDD for ; Fri, 13 Aug 2021 09:39:08 +0200 (CEST) Received: from localhost ([::1]:38940 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mERmW-0007j7-0N for larch@yhetil.org; Fri, 13 Aug 2021 03:39:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44478) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mERmP-0007iz-W8 for guix-patches@gnu.org; Fri, 13 Aug 2021 03:39:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:57145) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mERmP-0003nh-Q0 for guix-patches@gnu.org; Fri, 13 Aug 2021 03:39:01 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mERmP-0002Jv-NM for guix-patches@gnu.org; Fri, 13 Aug 2021 03:39:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#49958] [PATCH] More flexibility in opam importer Resent-From: Xinglu Chen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 13 Aug 2021 07:39:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49958 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Alice BRENON , 49958@debbugs.gnu.org Received: via spool by 49958-submit@debbugs.gnu.org id=B49958.16288402848847 (code B ref 49958); Fri, 13 Aug 2021 07:39:01 +0000 Received: (at 49958) by debbugs.gnu.org; 13 Aug 2021 07:38:04 +0000 Received: from localhost ([127.0.0.1]:40458 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mERlO-0002IG-RZ for submit@debbugs.gnu.org; Fri, 13 Aug 2021 03:38:04 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:59548 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mERlK-0002Hy-2f for 49958@debbugs.gnu.org; Fri, 13 Aug 2021 03:37:57 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1628840264; bh=HQlzVtAUqSZvrM/g7PTOCxhUWRwPrqPTB0Gn61Y/eOM=; h=From:To:Subject:In-Reply-To:References:Date; b=SboMHInNfIc2IWUCBAl/1VkI4C9SCN/8JQaeig+nnNHaO3IzdDZCKluobOnipo8RP cXWpnc386Cf1oeIKzVfoqXPJTKFINqUXXp3wICQhptDuB97yBzdmno0T8UNsODGCPh CThIFv1PbgEhj2GmtgrXU1UaCiMzL3AXD+73yw0g= In-Reply-To: <20210810184826.17d14aab@ens-lyon.fr> References: <20210809140407.748fa019@ens-lyon.fr> <20210809171935.05fac773@ens-lyon.fr> <20210810140413.2f7d2f1b@ens-lyon.fr> <20210810184826.17d14aab@ens-lyon.fr> Date: Fri, 13 Aug 2021 09:37:43 +0200 Message-ID: <87pmuhhk1k.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1628840349; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=+yfdZE2PgvgjjMcwdJeakyq8ohzzXK3jlhhHJB34Orc=; b=lEf246edgfL3h/4UeayIwVm1Ccu1G6sY7IvLz4oL34Uz9gu5Tqea+8MNZjB3XrGonY4noh XkGx0hvXUbLQTsCp5rbg2equ9/jehMzHXIPZHAJYm8lW0E6Vi3CNZLOEHZ0tBQZuAcKATH uqCevQ0+YMlPzSZCVs8zjbHsy6ao8xHjPLs1UJbo7sPbZ9ux0EddLPZWHYSV6YRWAwQE4R 8+mYM8+E5NAIFXPB3toxaHtrLfRr3tAyVEnr1PEBVWmC7sCPHgQJkrzlPHTvEByK1KKcTr XRE3LmynFG2eUVnIAkB+k8oE/Z5xOG3b853w+VZy0TncjG/pP+rAkjzEYzEGOA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1628840349; a=rsa-sha256; cv=none; b=izECm8bjGkzRgxnHcTb/+H+K8sYoMljBCbwuLGDA0HBg311Avb00yBVlt5ON9dFYo4iMk8 NDSPZdoGgn6UUDfLhHzAMDZselASvydBPdFueHZ23xUB1RwEMizFsLQJaYPAafR/b/Rp7O kReMZmejepf2o/bLa7NY+S2kEHrsQvZMvwC4YnroHIM/6aBzmuXqvx6RTCZ52ewy/2BVQ9 yN0qRb13RtZxY7jcko8zrEMH1IPr5/l0hqxiYW1aTWdHv/6PzG2yY5hZLM+hBCMRHluP4o 3J2lVGpaBNXNdwk2Z6q6ExtYhtA3RxrS4xbF2iUbuS+kZNAAAM3Vz9fJtm9jWQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=yoctocell.xyz header.s=mail header.b=SboMHInN; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -3.41 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=yoctocell.xyz header.s=mail header.b=SboMHInN; dmarc=fail reason="SPF not aligned (relaxed)" header.from=yoctocell.xyz (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: F156F2CDD X-Spam-Score: -3.41 X-Migadu-Scanner: scn0.migadu.com X-TUID: jq1K4lnFDEAL --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Tue, Aug 10 2021, Alice BRENON wrote: > - rephrase the documentation. > - remove unnecessary #:ensure #f key in call to (cache-directory). > - clarify what happens when string->uri fails and yields a warning by > passing a 'bad-repo symbol and commenting at its elimination site. > - make a separate function of get-uri now that it's become as large as > its caller repo-type. > - mention the possibility to call --repo several times in help message. > - add points at the end of sentences in commit message and in this > e-mail. > From 49e8236f81462501e89aa40d4e1b77bcc3fbb0ad Mon Sep 17 00:00:00 2001 > From: Alice BRENON > Date: Sat, 7 Aug 2021 19:50:10 +0200 > Subject: [PATCH] guix: opam: More flexibility in the importer. > > * guix/scripts/import/opam.scm: pass all instances of --repo as a list > to the importer. > * guix/import/opam.scm (opam-fetch): stop expecting "expanded" > repositories and call get-opam-repository instead to keep values > "symbolic" as long as possible and factorize. > (get-opam-repository): use the same repository source as CLI opam does > (i.e. HTTP-served index.tar.gz instead of git repositories). > (find-latest-version): be more flexible on the repositories structure > instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/. > * tests/opam.scm: update the call to opam->guix-package since repo is > now expected to be a list and remove the mocked get-opam-repository > deprecated by the support for local folders by the actual > implementation. > * doc/guix.texi: document the new semantics and valid arguments for the > --repo option. > --- > doc/guix.texi | 25 ++++-- > guix/import/opam.scm | 158 +++++++++++++++++++++-------------- > guix/scripts/import/opam.scm | 8 +- > tests/opam.scm | 68 ++++++++------- > 4 files changed, 155 insertions(+), 104 deletions(-) > > diff --git a/doc/guix.texi b/doc/guix.texi > index 4eb5324b51..4a911e4c0f 100644 > --- a/doc/guix.texi > +++ b/doc/guix.texi > @@ -94,6 +94,7 @@ Copyright @copyright{} 2021 Xinglu Chen@* > Copyright @copyright{} 2021 Raghav Gururajan@* > Copyright @copyright{} 2021 Domagoj Stolfa@* > Copyright @copyright{} 2021 Hui Lu@* > +Copyright @copyright{} 2021 Alice Brenon@* >=20=20 > Permission is granted to copy, distribute and/or modify this document > under the terms of the GNU Free Documentation License, Version 1.3 or > @@ -11612,14 +11613,26 @@ Traverse the dependency graph of the given upst= ream package recursively > and generate package expressions for all those packages that are not yet > in Guix. > @item --repo > -Select the given repository (a repository name). Possible values includ= e: > + > +By default, packages are searched in the official OPAM repository. This > +option, which can be used more than once, lets you add other > +repositories where to look for packages. =E2=80=9Clets you add other repositories where to look for package=E2=80=9D= sounds a bit weird, maybe lets you add other repositories which will be used to lookup packages. ? > @itemize > -@item @code{opam}, the default opam repository, > -@item @code{coq} or @code{coq-released}, the stable repository for coq p= ackages, > -@item @code{coq-core-dev}, the repository that contains development vers= ions of coq, > -@item @code{coq-extra-dev}, the repository that contains development ver= sions > - of coq packages. > +@item the name of a known repository - can be one of @code{opam}, > + @code{coq} (equivalent to @code{coq-released}), > + @code{coq-core-dev}, @code{coq-extra-dev} or @code{grew}. > +@item the URL of a repository as expected by the @code{opam repository > + add} command (for instance, the URL equivalent of the above > + @code{opam} name would be @uref{https://opam.ocaml.org}). > +@item the path to a local copy of a repository (a directory containing a > + @file{packages/} sub-directory). > @end itemize > + > +Please note that repositories added with this option do not replace the > +default @code{opam} repository, so calling this importer with the option > +@code{--repo=3Dopam} is redundant. What happens if I specify an additional repository, and a package exists in that repository _and_ the default opam repository? From which repository will the package be imported from? > diff --git a/guix/import/opam.scm b/guix/import/opam.scm > index a35b01d277..0e6cae72c4 100644 > --- a/guix/import/opam.scm > +++ b/guix/import/opam.scm > @@ -2,6 +2,7 @@ > ;;; Copyright =C2=A9 2018 Julien Lepiller > ;;; Copyright =C2=A9 2020 Martin Becze > ;;; Copyright =C2=A9 2021 Xinglu Chen > +;;; Copyright =C2=A9 2021 Alice Brenon > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -22,21 +23,24 @@ > #:use-module (ice-9 ftw) > #:use-module (ice-9 match) > #:use-module (ice-9 peg) > + #:use-module ((ice-9 popen) #:select (open-pipe*)) > #:use-module (ice-9 receive) > - #:use-module ((ice-9 rdelim) #:select (read-line)) > #:use-module (ice-9 textual-ports) > #:use-module (ice-9 vlist) > #:use-module (srfi srfi-1) > #:use-module (srfi srfi-2) > - #:use-module (web uri) > + #:use-module ((srfi srfi-26) #:select (cut)) > + #:use-module ((web uri) #:select (string->uri uri->string)) > + #:use-module ((guix build utils) #:select (dump-port find-files mkdir-= p)) > #:use-module (guix build-system) > #:use-module (guix build-system ocaml) > #:use-module (guix http-client) > - #:use-module (guix git) > #:use-module (guix ui) > #:use-module (guix packages) > #:use-module (guix upstream) > - #:use-module (guix utils) > + #:use-module ((guix utils) #:select (cache-directory > + version>? > + call-with-temporary-output-file)) > #:use-module (guix import utils) > #:use-module ((guix licenses) #:prefix license:) > #:export (opam->guix-package > @@ -121,51 +125,83 @@ > (define-peg-pattern condition-string all (and QUOTE (* STRCHR) QUOTE)) > (define-peg-pattern condition-var all (+ (or (range #\a #\z) "-" ":"))) >=20=20 > -(define* (get-opam-repository #:optional repo) > +(define (opam-cache-directory path) > + (string-append (cache-directory) "/opam/" path)) > + > +(define known-repositories > + '((opam . "https://opam.ocaml.org") > + (coq . "https://coq.inria.fr/opam/released") > + (coq-released . "https://coq.inria.fr/opam/released") > + (coq-core-dev . "https://coq.inria.fr/opam/core-dev") > + (coq-extra-dev . "https://coq.inria.fr/opam/extra-dev") > + (grew . "http://opam.grew.fr"))) > + > +(define (get-uri repo-root) > + (let ((archive-file (string-append repo-root "/index.tar.gz"))) > + (or (string->uri archive-file) > + (begin > + (warning (G_ "'~a' is not a valid URI~%") archive-file) > + 'bad-repo)))) > + > +(define (repo-type repo) > + (match (assoc-ref known-repositories (string->symbol repo)) > + (#f (if (file-exists? repo) > + `(local ,repo) > + `(remote ,(get-uri repo)))) > + (url `(remote ,(get-uri url))))) > + > +(define (update-repository input) > + "Make sure the cache for opam repository INPUT is up-to-date" > + (let* ((output (opam-cache-directory (basename (port-filename input)))) > + (cached-date (if (file-exists? output) > + (stat:mtime (stat output)) > + (begin (mkdir-p output) 0)))) > + (when (> (stat:mtime (stat input)) cached-date) > + (call-with-port > + (open-pipe* OPEN_WRITE "tar" "xz" "-C" output "-f" "-") > + (cut dump-port input <>))) > + output)) > + > +(define* (get-opam-repository #:optional (repo "opam")) > "Update or fetch the latest version of the opam repository and return = the > path to the repository." > - (let ((url (cond > - ((or (not repo) (equal? repo 'opam)) > - "https://github.com/ocaml/opam-repository") > - ((string-prefix? "coq-" (symbol->string repo)) > - "https://github.com/coq/opam-coq-archive") > - ((equal? repo 'coq) "https://github.com/coq/opam-coq-arch= ive") > - (else (throw 'unknown-repository repo))))) > - (receive (location commit _) > - (update-cached-checkout url) > - (cond > - ((or (not repo) (equal? repo 'opam)) > - location) > - ((equal? repo 'coq) > - (string-append location "/released")) > - ((string-prefix? "coq-" (symbol->string repo)) > - (string-append location "/" (substring (symbol->string repo) 4)= )) > - (else location))))) > + (match (repo-type repo) > + (('local p) p) > + (('remote 'bad-repo) #f) ; to weed it out during filter-map in opam-= fetch > + (('remote r) (call-with-port (http-fetch/cached r) update-repository= )))) >=20=20 > ;; Prevent Guile 3 from inlining this procedure so we can mock it in tes= ts. > (set! get-opam-repository get-opam-repository) >=20=20 > -(define (latest-version versions) > - "Find the most recent version from a list of versions." > - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) versions)) > +(define (get-version-and-file path) > + "Analyse a candidate path and return an list containing information fo= r proper > + version comparison as well as the source path for metadata." > + (and-let* ((metadata-file (string-append path "/opam")) > + (filename (basename path)) > + (version (string-join (cdr (string-split filename #\.)) "."= ))) > + (and (file-exists? metadata-file) > + (eq? 'regular (stat:type (stat metadata-file))) > + (if (string-prefix? "v" version) > + `(V ,(substring version 1) ,metadata-file) > + `(digits ,version ,metadata-file))))) What happens if some other prefix is used, e.g., =E2=80=9Crelease-=E2=80=9D= or =E2=80=9CV-=E2=80=9D? Also, why not just return the version number and the metadata file; we don=E2=80=99t really care about the prefix do we? > +(define (keep-max-version a b) > + "Version comparison on the lists returned by the previous function tak= ing the > + janestreet re-versioning into account (v-prefixed come first)." > + (match (cons a b) > + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) > + ((('V _ _) . _) a) > + ((_ . ('V _ _)) b) > + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a b)))) >=20=20 > (define (find-latest-version package repository) > "Get the latest version of a package as described in the given reposit= ory." > - (let* ((dir (string-append repository "/packages/" package)) > - (versions (scandir dir (lambda (name) (not (string-prefix? "." = name)))))) > - (if versions > - (let ((versions (map > - (lambda (dir) > - (string-join (cdr (string-split dir #\.)) ".")) > - versions))) > - ;; Workaround for janestreet re-versionning > - (let ((v-versions (filter (lambda (version) (string-prefix? "v" = version)) versions))) > - (if (null? v-versions) > - (latest-version versions) > - (string-append "v" (latest-version (map (lambda (version) (s= ubstring version 1)) v-versions)))))) > - (begin > - (format #t (G_ "Package not found in opam repository: ~a~%") pac= kage) > - #f)))) > + (let ((packages (string-append repository "/packages")) > + (filter (make-regexp (string-append "^" package "\\.")))) > + (reduce keep-max-version #f > + (filter-map > + get-version-and-file > + (find-files packages filter #:directories? #t))))) >=20=20 > (define (get-metadata opam-file) > (with-input-from-file opam-file > @@ -266,28 +302,30 @@ path to the repository." >=20=20 > (define (depends->native-inputs depends) > (filter (lambda (name) (not (equal? "" name))) > - (map dependency->native-input depends))) > + (map dependency->native-input depends))) >=20=20 > (define (dependency-list->inputs lst) > (map > - (lambda (dependency) > - (list dependency (list 'unquote (string->symbol dependency)))) > - (ocaml-names->guix-names lst))) > - > -(define* (opam-fetch name #:optional (repository (get-opam-repository))) > - (and-let* ((repository repository) > - (version (find-latest-version name repository)) > - (file (string-append repository "/packages/" name "/" name = "." version "/opam"))) > - `(("metadata" ,@(get-metadata file)) > - ("version" . ,(if (string-prefix? "v" version) > - (substring version 1) > - version))))) > - > -(define* (opam->guix-package name #:key (repo 'opam) version) > - "Import OPAM package NAME from REPOSITORY (a directory name) or, if > -REPOSITORY is #f, from the official OPAM repository. Return a 'package'= sexp > + (lambda (dependency) > + (list dependency (list 'unquote (string->symbol dependency)))) > + (ocaml-names->guix-names lst))) > + > +(define* (opam-fetch name #:optional (repositories-specs '("opam"))) > + (or (fold (lambda (repository others) > + (match (find-latest-version name repository) > + ((_ version file) `(("metadata" ,@(get-metadata file)) > + ("version" . ,version))) > + (_ others))) > + #f > + (filter-map get-opam-repository repositories-specs)) > + (leave (G_ "Package '~a' not found~%") name))) Nit: I wouldn=E2=80=99t capitalize =E2=80=9Cpackage=E2=80=9D, otherwise the= error message looks like this guix import: error: Package 'equations' not found Otherwise, LGTM! Great to see more work on the importers! :-) --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmEWIUcVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5sloP/j9GlNL6cVOqlrmzWPOSblnPHGlz Ds5mRFshWiBa52F8BlmRW4p3XnDoahHqewPVMOQef8xxOYzKSKhNo+qUmqfO22Mj z1qFD4gX6c3O9YMuRXfcDynvTaesFLkU0ddu+OaP634fYk62uP922p3JVk8cV8VZ /IgIjermbTNMtY7AfEMBwr37To4Hdq7J8lyq1VQdKxXKbmyqDGBoQC0LELNfmwlp oKjWmUxGC31kxuzZU0iBq84eEcE0kkuXc+kThXZZ6coIoVShMrFB6HoxLM+PiQev br39GgJzlhW2ypFDbYVzGpQdcuiVPr5zPIZuaBFXm40NYMWpLZw+Xhuv9kKHChb6 puRFQlywrxBre6gfx4rUstRWgOjZPUUdeRM+yhn2Kt5pSQat+9bpR8TnoMeMX0hD BkO+hxBqKc072g9WSZ3jgAh40WJiMxv6H34BO8RgTtJ72hS81l7PIw5Mxl+JpCRa iQAPgOrJ9rLE3h59WIueGJ129n5sGc3JxFMwR4Vp0h6OnaSM3uClWDZmlsmhUP34 o0KsI7KU0ErkL3q4gZESK8wrHvT88inCmaW+r4cS5+qk4TSpFe3MgSFtEEM4m3Z0 dHkhusWz87r3/A5P238Qku6MAsKwgutx8PmNzcuyuyvSl8tjDM4GeF9kT+Rmi6eF 5sqW8OLvBFGlzFqF =34CG -----END PGP SIGNATURE----- --=-=-=--