From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id kCxvOIxTFmEwgQEAgWs5BA (envelope-from ) for ; Fri, 13 Aug 2021 13:12:12 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id cIksNIxTFmF8TgAAB5/wlQ (envelope-from ) for ; Fri, 13 Aug 2021 11:12:12 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 82CD679B7 for ; Fri, 13 Aug 2021 13:12:11 +0200 (CEST) Received: from localhost ([::1]:41192 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mEV6g-00089g-9C for larch@yhetil.org; Fri, 13 Aug 2021 07:12:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49918) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mEV6Y-00089I-Nc for guix-patches@gnu.org; Fri, 13 Aug 2021 07:12:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:57280) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mEV6Y-0003Hl-HW for guix-patches@gnu.org; Fri, 13 Aug 2021 07:12:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mEV6Y-0001lj-Dd for guix-patches@gnu.org; Fri, 13 Aug 2021 07:12:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#49958] [PATCH] More flexibility in opam importer Resent-From: Alice BRENON Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 13 Aug 2021 11:12:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49958 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Xinglu Chen Cc: 49958@debbugs.gnu.org Received: via spool by 49958-submit@debbugs.gnu.org id=B49958.16288530826750 (code B ref 49958); Fri, 13 Aug 2021 11:12:02 +0000 Received: (at 49958) by debbugs.gnu.org; 13 Aug 2021 11:11:22 +0000 Received: from localhost ([127.0.0.1]:40593 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mEV5n-0001kf-FX for submit@debbugs.gnu.org; Fri, 13 Aug 2021 07:11:22 -0400 Received: from lxc-smtp2.ens-lyon.fr ([140.77.167.81]:39012) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mEV5j-0001kV-Vn for 49958@debbugs.gnu.org; Fri, 13 Aug 2021 07:11:13 -0400 Received: from localhost (localhost [127.0.0.1]) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTP id 91280E2DBD; Fri, 13 Aug 2021 13:11:10 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.11.0 (20160426) (Debian) at ens-lyon.fr Received: from lxc-smtp2.ens-lyon.fr ([127.0.0.1]) by localhost (lxc-smtp2.ens-lyon.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6D4NyVWOfuI3; Fri, 13 Aug 2021 13:11:10 +0200 (CEST) Received: from localhost (unknown [78.194.167.103]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTPSA id 5001EE2CFA; Fri, 13 Aug 2021 13:11:10 +0200 (CEST) Date: Fri, 13 Aug 2021 13:11:09 +0200 From: Alice BRENON Message-ID: <20210813131109.14c25204@ens-lyon.fr> In-Reply-To: <87pmuhhk1k.fsf@yoctocell.xyz> References: <20210809140407.748fa019@ens-lyon.fr> <20210809171935.05fac773@ens-lyon.fr> <20210810140413.2f7d2f1b@ens-lyon.fr> <20210810184826.17d14aab@ens-lyon.fr> <87pmuhhk1k.fsf@yoctocell.xyz> Organization: ENS de Lyon X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/CNH00mJos/vwxi9.UARMLah" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1628853132; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=fCuoPb7y5HsWhLslgLPpXhHHI1Yi6PTMnFdkmbymjr4=; b=SNE9adDK5XfBxCRLljpVwvHvzIS/WX0y2yauYxGNPL+B3vNRVB747fMzXESTQvxWHHupk3 cQ8+BYJCHN1ePoa5ZumSjFG/tkKmIVckjXTdApu9RLZCAE8djjkhoYaLSl2/sTqFw/05gX 0LDaGkfD0lUwVu3loU5mRO1X6yFkAUBtO1+GYHpRMLeXxt6uacWMv0h1xfqe4VO51IBhLT c2UacN789zkI2L89cpADYRXmU07W+y8oPA4JxkPLVp+9Iia7zB9PzPTstZkAaxfOsweZU5 gdOS0k91roYrVCLVX9nvPR/fHWYNPE9MmPLv2O1Nj71WEOWDG3EAyMOKN/j26Q== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1628853132; a=rsa-sha256; cv=none; b=Izf0p8aelZuEEawtPTa5fU+lfHOKsiX/2nBekfT5lhhcpB4uimm71UT5MdABm7JFoBGKgb BIRxFHQIzQsgDiDs/8bVlGwQE22PCgf2Z2vJLZcHP9w4Sk+QRWZOtXhCpsG0Xr6hydjIYW TXi/qZnZ9Q9csPpm2BEGye61MSEGrze88k87TQywl9cMmZD7Kuf8lXg6losmgnVk/qldmk pb+66nJeojEh4kXWGMV1gRLPxakfHpzxmffzL6Hij2ASn5IOOiMj11ns0vcR35xIN3z0v9 5uAVxXx0DTKG3lVQofkQ2cOojIrMf/A8yaqZL1TUgOs2SXc6Vx5DUJa9N8HcxQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -2.41 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 82CD679B7 X-Spam-Score: -2.41 X-Migadu-Scanner: scn0.migadu.com X-TUID: wkmNpbxTr4vQ --MP_/CNH00mJos/vwxi9.UARMLah Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Le Fri, 13 Aug 2021 09:37:43 +0200, Xinglu Chen a =C3=A9crit : > On Tue, Aug 10 2021, Alice BRENON wrote: >=20 > [=E2=80=A6] > > include: + > > +By default, packages are searched in the official OPAM repository. > > This +option, which can be used more than once, lets you add other > > +repositories where to look for packages. =20 >=20 > =E2=80=9Clets you add other repositories where to look for package=E2=80= =9D sounds a > bit weird, maybe >=20 > lets you add other repositories which will be used to lookup > packages. >=20 > ? Ok, as discussed on IRC, trying "lets you add other repositories which will be searched for packages". > What happens if I specify an additional repository, and a package > exists in that repository _and_ the default opam repository? From > which repository will the package be imported from? That is the beauty of it: the repositories are assumed to be passed by order of preference, defaulting to the official opam repositories only if packages haven't been found anywhere else. Writing this makes me realize that indeed, starting with --repo=3Dopam isn't entirely redundant: it could be used to prevent an otherwise interesting repo from overriding stuff if opam already provides it (let's assume some "super-opam" with a couple additional packages, and custom versions of existing opam packages). Calling `--repo=3Dsuper-opam` would use the super-opam versions as soon as a package exists in super-opam, while `--repo=3Dopam --repo=3Dsuper-opam` would take the super-opam versions only when none exist in opam. A much simpler use-case would be to locally override only some packages in a repo, and pass --repo=3Doverriden-repo --repo=3Dnormal-repo. This behaviour relies on the implementation of opam-fetch and how folds work in guile. Since in the importer script options are stacked as they are retrieved from the CLI arguments, and repositories are then just filter-maped from that list, they end up in a list by reverse order of preference. In opam->guix-package, 'opam gets push on the top if it's not already there somewhere. So what we get as input of opam-fetch is a list of repositories-specs by reverse order of preference. Now fold applies the accumulator to each item in order, so, last elements has the final say, i.e. the last elements which yield results in find-latests are preferred over the earlier elements of the list. This works for the same reason why `(lambda (l) (fold cons '() l)` will reverse its input list. It's slightly inefficient because it means all repositories are searched, in reverse order of preference, but I haven't figured how to get a lazy fold in guile. Granted, I could have written the recursion explicitly to get that. Will fix if performance matters. Also, versions are not compared between repositories, as soon as a repo provides one version of a given package, the latest of all the versions this one provides is used in the output, no matter the contents of other repositories. This is useful to allow "downgrades" by masking parts of repository which have too recent versions. So, thanks for your remark, the documentation deserved a clearer explanation of it. > [=E2=80=A6] > > -(define (latest-version versions) > > - "Find the most recent version from a list of versions." > > - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) > > versions)) +(define (get-version-and-file path) > > + "Analyse a candidate path and return an list containing > > information for proper > > + version comparison as well as the source path for metadata." > > + (and-let* ((metadata-file (string-append path "/opam")) > > + (filename (basename path)) > > + (version (string-join (cdr (string-split filename > > #\.)) "."))) > > + (and (file-exists? metadata-file) > > + (eq? 'regular (stat:type (stat metadata-file))) > > + (if (string-prefix? "v" version) > > + `(V ,(substring version 1) ,metadata-file) > > + `(digits ,version ,metadata-file))))) =20 >=20 > What happens if some other prefix is used, e.g., =E2=80=9Crelease-=E2=80= =9D or =E2=80=9CV-=E2=80=9D? >=20 It would get marked as a 'digit. In a previous draft before I started sending this series of patches, it was called 'R, standing for "regular", then I thought it was not very meaningful, and, since the versions were to my knowledge supposed to be either v[0-9]+(\.[0-9]+)* or [0-9]+(\.[0-9]+)*, I thought I could call that default case "digits" to clearly indicate what I was trying to refer to. I could change it to 'other if it matters too much, but the important thing here is that we distinguish between v-prefixed (the so-called "janestreet re-versionning" mentioned inside the implementation of find-latest on current d87d6d6 master) and other versions because =E2=AC=87=EF=B8=8F > Also, why not just return the version number and the metadata file; we > don=E2=80=99t really care about the prefix do we? >=20 yes we do ! the former latest-version finder handled strings, and dropped this prefix or put it back on the fly, but the logic implemented was: if there are v-prefixed versions, find the greatest of them, without the initial "v", if there aren't, just find the greatest of all versions. This implies that v-prefixed versions are considered more important and automatically greater than non-prefixed versions, no matter what the numbers, which is why this information must be kept. I'm just playing ADTs in guile here, "parsing" the version string only once to retain a symbolic representation of it that will at first glance allow to identify the type of version used and access the relevant digits for comparison. The comparison is used right after: > > +(define (keep-max-version a b) > > + "Version comparison on the lists returned by the previous > > function taking the > > + janestreet re-versioning into account (v-prefixed come first)." > > + (match (cons a b) > > + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) > > + ((('V _ _) . _) a) > > + ((_ . ('V _ _)) b) > > + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a > > b))))=20 and used in the reduce in find-latest-version. So keeping this 'V is what will help janestreet-re-versionned packages "skip the line" by being automatically greater than any non v-prefixed package (thus, v0.0.1 is greater than 13.2, which is the current behaviour). > > (define (find-latest-version package repository) > > "Get the latest version of a package as described in the given > > repository." > > - (let* ((dir (string-append repository "/packages/" package)) > > - (versions (scandir dir (lambda (name) (not > > (string-prefix? "." name)))))) > > - (if versions > > - (let ((versions (map > > - (lambda (dir) > > - (string-join (cdr (string-split dir > > #\.)) ".")) > > - versions))) > > - ;; Workaround for janestreet re-versionning > > - (let ((v-versions (filter (lambda (version) > > (string-prefix? "v" version)) versions))) > > - (if (null? v-versions) > > - (latest-version versions) > > - (string-append "v" (latest-version (map (lambda > > (version) (substring version 1)) v-versions)))))) > > - (begin > > - (format #t (G_ "Package not found in opam repository: > > ~a~%") package) > > - #f)))) > > + (let ((packages (string-append repository "/packages")) > > + (filter (make-regexp (string-append "^" package "\\.")))) > > + (reduce keep-max-version #f > > + (filter-map > > + get-version-and-file > > + (find-files packages filter #:directories? #t))))) > > =20 > [=E2=80=A6] > > + (filter-map get-opam-repository repositories-specs)) > > + (leave (G_ "Package '~a' not found~%") name))) =20 >=20 > Nit: I wouldn=E2=80=99t capitalize =E2=80=9Cpackage=E2=80=9D, otherwise t= he error message > looks like this >=20 > guix import: error: Package 'equations' not found a very neat tip, thank you ! : ) --MP_/CNH00mJos/vwxi9.UARMLah Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=0001-guix-opam-More-flexibility-in-the-importer.patch =46rom cde8b2a5d88d89bfea31c86d3ae94d37c1d3c83f Mon Sep 17 00:00:00 2001 From: Alice BRENON Date: Sat, 7 Aug 2021 19:50:10 +0200 Subject: [PATCH] guix: opam: More flexibility in the importer. * guix/scripts/import/opam.scm: pass all instances of --repo as a list to the importer. * guix/import/opam.scm (opam-fetch): stop expecting "expanded" repositories and call get-opam-repository instead to keep values "symbolic" as long as possible and factorize. (get-opam-repository): use the same repository source as CLI opam does (i.e. HTTP-served index.tar.gz instead of git repositories). (find-latest-version): be more flexible on the repositories structure instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/. * tests/opam.scm: update the call to opam->guix-package since repo is now expected to be a list and remove the mocked get-opam-repository deprecated by the support for local folders by the actual implementation. * doc/guix.texi: document the new semantics and valid arguments for the --repo option. --- doc/guix.texi | 30 +++++-- guix/import/opam.scm | 158 +++++++++++++++++++++-------------- guix/scripts/import/opam.scm | 8 +- tests/opam.scm | 68 ++++++++------- 4 files changed, 160 insertions(+), 104 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 78c1c09858..2d36561186 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -94,6 +94,7 @@ Copyright @copyright{} 2021 Xinglu Chen@* Copyright @copyright{} 2021 Raghav Gururajan@* Copyright @copyright{} 2021 Domagoj Stolfa@* Copyright @copyright{} 2021 Hui Lu@* +Copyright @copyright{} 2021 Alice Brenon@* =20 Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or @@ -11612,14 +11613,31 @@ Traverse the dependency graph of the given upstre= am package recursively and generate package expressions for all those packages that are not yet in Guix. @item --repo -Select the given repository (a repository name). Possible values include: + +By default, packages are searched in the official OPAM repository. This +option, which can be used more than once, lets you add other +repositories which will be searched for packages. It accepts as valid +arguments: + @itemize -@item @code{opam}, the default opam repository, -@item @code{coq} or @code{coq-released}, the stable repository for coq pac= kages, -@item @code{coq-core-dev}, the repository that contains development versio= ns of coq, -@item @code{coq-extra-dev}, the repository that contains development versi= ons - of coq packages. +@item the name of a known repository - can be one of @code{opam}, + @code{coq} (equivalent to @code{coq-released}), + @code{coq-core-dev}, @code{coq-extra-dev} or @code{grew}. +@item the URL of a repository as expected by the @code{opam repository + add} command (for instance, the URL equivalent of the above + @code{opam} name would be @uref{https://opam.ocaml.org}). +@item the path to a local copy of a repository (a directory containing a + @file{packages/} sub-directory). @end itemize + +Repositories must be passed to this option by order of preference and do +not replace the default @code{opam} which is always failed-back to. + +Also, please note that versions are not compared accross repositories. +The first repository (from left to right) that has at least one version +of a given package will prevail over any others and the version imported +will be the latest one found @emph{in this repository only}. + @end table =20 @item go diff --git a/guix/import/opam.scm b/guix/import/opam.scm index a35b01d277..fe13d29f03 100644 --- a/guix/import/opam.scm +++ b/guix/import/opam.scm @@ -2,6 +2,7 @@ ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2020 Martin Becze ;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,21 +23,24 @@ #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module ((ice-9 popen) #:select (open-pipe*)) #:use-module (ice-9 receive) - #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (ice-9 textual-ports) #:use-module (ice-9 vlist) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) - #:use-module (web uri) + #:use-module ((srfi srfi-26) #:select (cut)) + #:use-module ((web uri) #:select (string->uri uri->string)) + #:use-module ((guix build utils) #:select (dump-port find-files mkdir-p)) #:use-module (guix build-system) #:use-module (guix build-system ocaml) #:use-module (guix http-client) - #:use-module (guix git) #:use-module (guix ui) #:use-module (guix packages) #:use-module (guix upstream) - #:use-module (guix utils) + #:use-module ((guix utils) #:select (cache-directory + version>? + call-with-temporary-output-file)) #:use-module (guix import utils) #:use-module ((guix licenses) #:prefix license:) #:export (opam->guix-package @@ -121,51 +125,83 @@ (define-peg-pattern condition-string all (and QUOTE (* STRCHR) QUOTE)) (define-peg-pattern condition-var all (+ (or (range #\a #\z) "-" ":"))) =20 -(define* (get-opam-repository #:optional repo) +(define (opam-cache-directory path) + (string-append (cache-directory) "/opam/" path)) + +(define known-repositories + '((opam . "https://opam.ocaml.org") + (coq . "https://coq.inria.fr/opam/released") + (coq-released . "https://coq.inria.fr/opam/released") + (coq-core-dev . "https://coq.inria.fr/opam/core-dev") + (coq-extra-dev . "https://coq.inria.fr/opam/extra-dev") + (grew . "http://opam.grew.fr"))) + +(define (get-uri repo-root) + (let ((archive-file (string-append repo-root "/index.tar.gz"))) + (or (string->uri archive-file) + (begin + (warning (G_ "'~a' is not a valid URI~%") archive-file) + 'bad-repo)))) + +(define (repo-type repo) + (match (assoc-ref known-repositories (string->symbol repo)) + (#f (if (file-exists? repo) + `(local ,repo) + `(remote ,(get-uri repo)))) + (url `(remote ,(get-uri url))))) + +(define (update-repository input) + "Make sure the cache for opam repository INPUT is up-to-date" + (let* ((output (opam-cache-directory (basename (port-filename input)))) + (cached-date (if (file-exists? output) + (stat:mtime (stat output)) + (begin (mkdir-p output) 0)))) + (when (> (stat:mtime (stat input)) cached-date) + (call-with-port + (open-pipe* OPEN_WRITE "tar" "xz" "-C" output "-f" "-") + (cut dump-port input <>))) + output)) + +(define* (get-opam-repository #:optional (repo "opam")) "Update or fetch the latest version of the opam repository and return the path to the repository." - (let ((url (cond - ((or (not repo) (equal? repo 'opam)) - "https://github.com/ocaml/opam-repository") - ((string-prefix? "coq-" (symbol->string repo)) - "https://github.com/coq/opam-coq-archive") - ((equal? repo 'coq) "https://github.com/coq/opam-coq-archiv= e") - (else (throw 'unknown-repository repo))))) - (receive (location commit _) - (update-cached-checkout url) - (cond - ((or (not repo) (equal? repo 'opam)) - location) - ((equal? repo 'coq) - (string-append location "/released")) - ((string-prefix? "coq-" (symbol->string repo)) - (string-append location "/" (substring (symbol->string repo) 4))) - (else location))))) + (match (repo-type repo) + (('local p) p) + (('remote 'bad-repo) #f) ; to weed it out during filter-map in opam-fe= tch + (('remote r) (call-with-port (http-fetch/cached r) update-repository))= )) =20 ;; Prevent Guile 3 from inlining this procedure so we can mock it in tests. (set! get-opam-repository get-opam-repository) =20 -(define (latest-version versions) - "Find the most recent version from a list of versions." - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) versions)) +(define (get-version-and-file path) + "Analyse a candidate path and return an list containing information for = proper + version comparison as well as the source path for metadata." + (and-let* ((metadata-file (string-append path "/opam")) + (filename (basename path)) + (version (string-join (cdr (string-split filename #\.)) "."))) + (and (file-exists? metadata-file) + (eq? 'regular (stat:type (stat metadata-file))) + (if (string-prefix? "v" version) + `(V ,(substring version 1) ,metadata-file) + `(digits ,version ,metadata-file))))) + +(define (keep-max-version a b) + "Version comparison on the lists returned by the previous function takin= g the + janestreet re-versioning into account (v-prefixed come first)." + (match (cons a b) + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) + ((('V _ _) . _) a) + ((_ . ('V _ _)) b) + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a b)))) =20 (define (find-latest-version package repository) "Get the latest version of a package as described in the given repositor= y." - (let* ((dir (string-append repository "/packages/" package)) - (versions (scandir dir (lambda (name) (not (string-prefix? "." na= me)))))) - (if versions - (let ((versions (map - (lambda (dir) - (string-join (cdr (string-split dir #\.)) ".")) - versions))) - ;; Workaround for janestreet re-versionning - (let ((v-versions (filter (lambda (version) (string-prefix? "v" ve= rsion)) versions))) - (if (null? v-versions) - (latest-version versions) - (string-append "v" (latest-version (map (lambda (version) (sub= string version 1)) v-versions)))))) - (begin - (format #t (G_ "Package not found in opam repository: ~a~%") packa= ge) - #f)))) + (let ((packages (string-append repository "/packages")) + (filter (make-regexp (string-append "^" package "\\.")))) + (reduce keep-max-version #f + (filter-map + get-version-and-file + (find-files packages filter #:directories? #t))))) =20 (define (get-metadata opam-file) (with-input-from-file opam-file @@ -266,28 +302,30 @@ path to the repository." =20 (define (depends->native-inputs depends) (filter (lambda (name) (not (equal? "" name))) - (map dependency->native-input depends))) + (map dependency->native-input depends))) =20 (define (dependency-list->inputs lst) (map - (lambda (dependency) - (list dependency (list 'unquote (string->symbol dependency)))) - (ocaml-names->guix-names lst))) - -(define* (opam-fetch name #:optional (repository (get-opam-repository))) - (and-let* ((repository repository) - (version (find-latest-version name repository)) - (file (string-append repository "/packages/" name "/" name ".= " version "/opam"))) - `(("metadata" ,@(get-metadata file)) - ("version" . ,(if (string-prefix? "v" version) - (substring version 1) - version))))) - -(define* (opam->guix-package name #:key (repo 'opam) version) - "Import OPAM package NAME from REPOSITORY (a directory name) or, if -REPOSITORY is #f, from the official OPAM repository. Return a 'package' s= exp + (lambda (dependency) + (list dependency (list 'unquote (string->symbol dependency)))) + (ocaml-names->guix-names lst))) + +(define* (opam-fetch name #:optional (repositories-specs '("opam"))) + (or (fold (lambda (repository others) + (match (find-latest-version name repository) + ((_ version file) `(("metadata" ,@(get-metadata file)) + ("version" . ,version))) + (_ others))) + #f + (filter-map get-opam-repository repositories-specs)) + (leave (G_ "package '~a' not found~%") name))) + +(define* (opam->guix-package name #:key (repo '()) version) + "Import OPAM package NAME from REPOSITORIES (a list of names, URLs or lo= cal +paths, always including OPAM's official repository). Return a 'package' s= exp or #f on failure." - (and-let* ((opam-file (opam-fetch name (get-opam-repository repo))) + (and-let* ((with-opam (if (member "opam" repo) repo (cons "opam" repo))) + (opam-file (opam-fetch name with-opam)) (version (assoc-ref opam-file "version")) (opam-content (assoc-ref opam-file "metadata")) (url-dict (metadata-ref opam-content "url")) @@ -312,9 +350,7 @@ or #f on failure." (values `(package (name ,(ocaml-name->guix-name name)) - (version ,(if (string-prefix? "v" version) - (substring version 1) - version)) + (version ,version) (source (origin (method url-fetch) diff --git a/guix/scripts/import/opam.scm b/guix/scripts/import/opam.scm index 64164e7cc4..834ac34cb0 100644 --- a/guix/scripts/import/opam.scm +++ b/guix/scripts/import/opam.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -46,7 +47,8 @@ Import and convert the opam package for PACKAGE-NAME.\n")) (display (G_ " -r, --recursive import packages recursively")) (display (G_ " - --repo import packages from this opam repository")) + --repo import packages from this opam repository (name, = URL or local path) + can be used more than once")) (display (G_ " -V, --version display version information and exit")) (newline) @@ -81,7 +83,9 @@ Import and convert the opam package for PACKAGE-NAME.\n")) #:build-options? #f)) =20 (let* ((opts (parse-options)) - (repo (and=3D> (assoc-ref opts 'repo) string->symbol)) + (repo (filter-map (match-lambda + (('repo . name) name) + (_ #f)) opts)) (args (filter-map (match-lambda (('argument . value) value) diff --git a/tests/opam.scm b/tests/opam.scm index f1e3b70cb0..1536b74339 100644 --- a/tests/opam.scm +++ b/tests/opam.scm @@ -82,41 +82,39 @@ url { (set! test-source-hash (call-with-input-file file-name port-sha256)))) (_ (error "Unexpected URL: " url))))) - (mock ((guix import opam) get-opam-repository - (const test-repo)) - (let ((my-package (string-append test-repo - "/packages/foo/foo.1.0.0"))) - (mkdir-p my-package) - (with-output-to-file (string-append my-package "/opam") - (lambda _ - (format #t "~a" test-opam-file)))) - (match (opam->guix-package "foo" #:repo test-repo) - (('package - ('name "ocaml-foo") - ('version "1.0.0") - ('source ('origin - ('method 'url-fetch) - ('uri "https://example.org/foo-1.0.0.tar.gz") - ('sha256 - ('base32 - (? string? hash))))) - ('build-system 'ocaml-build-system) - ('propagated-inputs - ('quasiquote - (("ocaml-zarith" ('unquote 'ocaml-zarith))))) - ('native-inputs - ('quasiquote - (("ocaml-alcotest" ('unquote 'ocaml-alcotest)) - ("ocamlbuild" ('unquote 'ocamlbuild))))) - ('home-page "https://example.org/") - ('synopsis "Some example package") - ('description "This package is just an example.") - ('license 'license:bsd-3)) - (string=3D? (bytevector->nix-base32-string - test-source-hash) - hash)) - (x - (pk 'fail x #f)))))) + (let ((my-package (string-append test-repo + "/packages/foo/foo.1.0.0"))) + (mkdir-p my-package) + (with-output-to-file (string-append my-package "/opam") + (lambda _ + (format #t "~a" test-opam-file)))) + (match (opam->guix-package "foo" #:repo (list test-repo)) + (('package + ('name "ocaml-foo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri "https://example.org/foo-1.0.0.tar.gz") + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'ocaml-build-system) + ('propagated-inputs + ('quasiquote + (("ocaml-zarith" ('unquote 'ocaml-zarith))))) + ('native-inputs + ('quasiquote + (("ocaml-alcotest" ('unquote 'ocaml-alcotest)) + ("ocamlbuild" ('unquote 'ocamlbuild))))) + ('home-page "https://example.org/") + ('synopsis "Some example package") + ('description "This package is just an example.") + ('license 'license:bsd-3)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (pk 'fail x #f))))) =20 ;; Test the opam file parser ;; We fold over some test cases. Each case is a pair of the string to pars= e and the --=20 2.32.0 --MP_/CNH00mJos/vwxi9.UARMLah--