From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id GElmGmhIEWHrLAEAgWs5BA (envelope-from ) for ; Mon, 09 Aug 2021 17:23:20 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id uIEPFmhIEWHJGwAAB5/wlQ (envelope-from ) for ; Mon, 09 Aug 2021 15:23:20 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 62F273AE2 for ; Mon, 9 Aug 2021 17:23:18 +0200 (CEST) Received: from localhost ([::1]:45318 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mD77V-0000F0-KK for larch@yhetil.org; Mon, 09 Aug 2021 11:23:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38216) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mD77H-0000DB-2R for guix-patches@gnu.org; Mon, 09 Aug 2021 11:23:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:45986) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mD77G-00045H-S9 for guix-patches@gnu.org; Mon, 09 Aug 2021 11:23:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mD77G-0004qD-K7 for guix-patches@gnu.org; Mon, 09 Aug 2021 11:23:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#49958] [PATCH] More flexibility in opam importer Resent-From: Alice BRENON Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 09 Aug 2021 15:23:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49958 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 49958@debbugs.gnu.org Received: via spool by 49958-submit@debbugs.gnu.org id=B49958.162852257418581 (code B ref 49958); Mon, 09 Aug 2021 15:23:02 +0000 Received: (at 49958) by debbugs.gnu.org; 9 Aug 2021 15:22:54 +0000 Received: from localhost ([127.0.0.1]:57532 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mD777-0004pc-4S for submit@debbugs.gnu.org; Mon, 09 Aug 2021 11:22:54 -0400 Received: from lxc-smtp2.ens-lyon.fr ([140.77.167.81]:51792) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mD73x-0004kc-KF for 49958@debbugs.gnu.org; Mon, 09 Aug 2021 11:19:38 -0400 Received: from localhost (localhost [127.0.0.1]) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTP id 66B16E1F28 for <49958@debbugs.gnu.org>; Mon, 9 Aug 2021 17:19:36 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.11.0 (20160426) (Debian) at ens-lyon.fr Received: from lxc-smtp2.ens-lyon.fr ([127.0.0.1]) by localhost (lxc-smtp2.ens-lyon.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id X8LKfq-iJ4W6 for <49958@debbugs.gnu.org>; Mon, 9 Aug 2021 17:19:36 +0200 (CEST) Received: from localhost (unknown [78.194.167.103]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTPSA id 4D2D8E1A2D for <49958@debbugs.gnu.org>; Mon, 9 Aug 2021 17:19:36 +0200 (CEST) Date: Mon, 9 Aug 2021 17:19:35 +0200 From: Alice BRENON Message-ID: <20210809171935.05fac773@ens-lyon.fr> In-Reply-To: References: <20210809140407.748fa019@ens-lyon.fr> Organization: ENS de Lyon X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/cz_QVk7./OCdDAm5=mk01Dh" X-Mailman-Approved-At: Mon, 09 Aug 2021 11:22:52 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1628522599; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=ejLAf3L8Uch/XXxANNMePAFwiHiGDcGQ4x/art0qtDY=; b=ND1JbRV8f7hWdIubltdhEozEHHbVBaNW1NJzWfFkkfUR1xmKdFPKoKCyKRmutgIB80ohUV HsFGaO4L6pKSh6rLc7TOBoD+Wp0eUyRh0n8Zue7J9Jwoli1JLryll04HKt/N3L2BSfFoPb OZ8zOCA+sqUTPNmUL73mX02Tq/qtxSPNxDtI4uURxyHTzhxhg66JygD6pHLGz1TXCQxyZ/ IfbDB4Q241c/sO8h9Vh+oopxhw/0RHb3kZmWUieS6D2cfmgHIIQzKUmPaYMVoHFHny5ANh szPeCLezy1F9Tm4KCeZK3HBPutLNxlQvfRu3ras5k+y5tv44jtIeEW3mGr1uLw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1628522599; a=rsa-sha256; cv=none; b=aoQdArbJI8poaQJGBmL2Z97OhSjBUuGwwgTwKxo9Y5DCEQ9BjDO2soQ2mgSS2fS2vpxA58 bT+L4R3jwenK6fsOSZDnNmCsa6jbvRqUBfd76KIdIpLU1ABOvatrtG1DIcfTY9A7LuqwlM kkZtpyTdyej4UqosgWGtY0W88mt03QQ854DV1RaIuHNoyfk7ChXK7S/5YU4Vt6U5W9cmsP 8tjKOfH+1wJPmJds5f7e97lwi5ybhAwQTezLWVs7u/XLfFOUJm41yWRhXnjZR71kJU8THw 2nxARgbvY5mustP8bC27ZhFVmVCEk86n1CUWqPISWElnMAJ7QE+nz6ItMwOd/Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -2.41 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 62F273AE2 X-Spam-Score: -2.41 X-Migadu-Scanner: scn1.migadu.com X-TUID: pkVfsF8TaIwq --MP_/cz_QVk7./OCdDAm5=mk01Dh Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline First update of my patch thanks to the excellent remarks from roptat on IRC: the signature change for opam->guix-package (namely that the #:key parameter `repo` now expects a list instead of a single element) broke the opam test, so here's a version passing a list containing `test-repo` instead of `test-repo` itself directly to fix the corresponding test. Alice Le Mon, 09 Aug 2021 15:04:02 +0000, help-debbugs@gnu.org (GNU bug Tracking System) a =C3=A9crit : > Thank you for filing a new bug report with debbugs.gnu.org. >=20 > This is an automatically generated reply to let you know your message > has been received. >=20 > Your message is being forwarded to the package maintainers and other > interested parties for their attention; they will reply in due course. >=20 > Your message has been sent to the package maintainer(s): > guix-patches@gnu.org >=20 > If you wish to submit further information on this problem, please > send it to 49958@debbugs.gnu.org. >=20 > Please do not send mail to help-debbugs@gnu.org unless you wish > to report a problem with the Bug-tracking system. >=20 --MP_/cz_QVk7./OCdDAm5=mk01Dh Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=0001-guix-opam.patch =46rom 8897e419e92fa9e6e5b91c554bf72722b2934953 Mon Sep 17 00:00:00 2001 From: Alice BRENON Date: Sat, 7 Aug 2021 19:50:10 +0200 Subject: [PATCH] guix: opam: - Add support for importing from several repositories - Add support for custom repositories, either from a URL or= a local path - Use actual opam repositories as defined by the opam tool as source * guix/scripts/import/opam.scm: pass all instances of --repo as a list to the importer * guix/import/opam.scm: - delay repo resolution (call to get-opam-repository from within opam-fetch instead of outside) - use the same repository source as CLI opam does (i.e. HTTP-served index.tar.gz instead of git repositories) - be more flexible on the repositories structure instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/ * tests/opam.scm: update the opam->guix-package test since repo is now expected to be a list --- guix/import/opam.scm | 145 +++++++++++++++++++++-------------- guix/scripts/import/opam.scm | 5 +- tests/opam.scm | 2 +- 3 files changed, 93 insertions(+), 59 deletions(-) diff --git a/guix/import/opam.scm b/guix/import/opam.scm index a35b01d277..c3fad7db65 100644 --- a/guix/import/opam.scm +++ b/guix/import/opam.scm @@ -2,6 +2,7 @@ ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2020 Martin Becze ;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,13 +23,16 @@ #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module (ice-9 popen) #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (ice-9 textual-ports) #:use-module (ice-9 vlist) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) + #:use-module (srfi srfi-26) #:use-module (web uri) + #:use-module ((guix build utils) #:select (dump-port find-files mkdir-p)) #:use-module (guix build-system) #:use-module (guix build-system ocaml) #:use-module (guix http-client) @@ -121,51 +125,78 @@ (define-peg-pattern condition-string all (and QUOTE (* STRCHR) QUOTE)) (define-peg-pattern condition-var all (+ (or (range #\a #\z) "-" ":"))) =20 -(define* (get-opam-repository #:optional repo) +(define (opam-cache-directory path) + (string-append (cache-directory #:ensure? #f) "/opam/" path)) + +(define known-repositories + '((opam . "https://opam.ocaml.org") + (coq . "https://coq.inria.fr/opam/released") + (grew . "http://opam.grew.fr"))) + +(define (repo-type repo) + (define (get-uri repo-root) + (let ((archive-file (string-append repo-root "/index.tar.gz"))) + (or (string->uri archive-file) (throw 'bad-uri archive-file)))) + (match (assoc-ref known-repositories (string->symbol repo)) + (#f (if (file-exists? repo) + `(local ,repo) + `(remote ,(get-uri repo)))) + (url `(remote ,(get-uri url))))) + +(define (update-repository-at output-folder input) + "Make sure the opam repository at OUTPUT-FOLDER is up-to-date with INPUT" + (let ((cached-date (if (file-exists? output-folder) + (stat:mtime (stat output-folder)) + (begin (mkdir-p output-folder) 0)))) + (begin + (and (> (stat:mtime (stat input)) cached-date) + (call-with-port + (open-pipe* OPEN_WRITE "tar" "xz" "-C" output-folder "-f" "-") + (cut dump-port input <>))) + output-folder))) + +(define* (get-opam-repository #:optional (repo "opam")) "Update or fetch the latest version of the opam repository and return the path to the repository." - (let ((url (cond - ((or (not repo) (equal? repo 'opam)) - "https://github.com/ocaml/opam-repository") - ((string-prefix? "coq-" (symbol->string repo)) - "https://github.com/coq/opam-coq-archive") - ((equal? repo 'coq) "https://github.com/coq/opam-coq-archiv= e") - (else (throw 'unknown-repository repo))))) - (receive (location commit _) - (update-cached-checkout url) - (cond - ((or (not repo) (equal? repo 'opam)) - location) - ((equal? repo 'coq) - (string-append location "/released")) - ((string-prefix? "coq-" (symbol->string repo)) - (string-append location "/" (substring (symbol->string repo) 4))) - (else location))))) + (match (repo-type repo) + (('local p) p) + (('remote source) (let ((cache (opam-cache-directory (uri-host source)= ))) + (call-with-port + (http-fetch/cached source) + (cut update-repository-at cache <>)))))) =20 ;; Prevent Guile 3 from inlining this procedure so we can mock it in tests. (set! get-opam-repository get-opam-repository) =20 -(define (latest-version versions) - "Find the most recent version from a list of versions." - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) versions)) +(define (get-version-and-file path) + "Analyse a candidate path and return an list containing information for = proper + version comparison as well as the source path for metadata." + (and-let* ((metadata-file (string-append path "/opam")) + (filename (basename path)) + (version (string-join (cdr (string-split filename #\.)) "."))) + (and (file-exists? metadata-file) + (eq? 'regular (stat:type (stat metadata-file))) + (if (string-prefix? "v" version) + `(V ,(substring version 1) ,metadata-file) + `(digits ,version ,metadata-file))))) + +(define (keep-max-version a b) + "Version comparison on the lists returned by the previous function takin= g the + janestreet re-versioning into account (v-prefixed come first)." + (match (cons a b) + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) + ((('V _ _) . _) a) + ((_ . ('V _ _)) b) + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a b)))) =20 (define (find-latest-version package repository) "Get the latest version of a package as described in the given repositor= y." - (let* ((dir (string-append repository "/packages/" package)) - (versions (scandir dir (lambda (name) (not (string-prefix? "." na= me)))))) - (if versions - (let ((versions (map - (lambda (dir) - (string-join (cdr (string-split dir #\.)) ".")) - versions))) - ;; Workaround for janestreet re-versionning - (let ((v-versions (filter (lambda (version) (string-prefix? "v" ve= rsion)) versions))) - (if (null? v-versions) - (latest-version versions) - (string-append "v" (latest-version (map (lambda (version) (sub= string version 1)) v-versions)))))) - (begin - (format #t (G_ "Package not found in opam repository: ~a~%") packa= ge) - #f)))) + (let ((packages (string-append repository "/packages")) + (filter (make-regexp (string-append "^" package "\\.")))) + (reduce keep-max-version #f + (filter-map + get-version-and-file + (find-files packages filter #:directories? #t))))) =20 (define (get-metadata opam-file) (with-input-from-file opam-file @@ -266,28 +297,30 @@ path to the repository." =20 (define (depends->native-inputs depends) (filter (lambda (name) (not (equal? "" name))) - (map dependency->native-input depends))) + (map dependency->native-input depends))) =20 (define (dependency-list->inputs lst) (map - (lambda (dependency) - (list dependency (list 'unquote (string->symbol dependency)))) - (ocaml-names->guix-names lst))) - -(define* (opam-fetch name #:optional (repository (get-opam-repository))) - (and-let* ((repository repository) - (version (find-latest-version name repository)) - (file (string-append repository "/packages/" name "/" name ".= " version "/opam"))) - `(("metadata" ,@(get-metadata file)) - ("version" . ,(if (string-prefix? "v" version) - (substring version 1) - version))))) - -(define* (opam->guix-package name #:key (repo 'opam) version) - "Import OPAM package NAME from REPOSITORY (a directory name) or, if -REPOSITORY is #f, from the official OPAM repository. Return a 'package' s= exp + (lambda (dependency) + (list dependency (list 'unquote (string->symbol dependency)))) + (ocaml-names->guix-names lst))) + +(define* (opam-fetch name #:optional (repositories-specs '("opam"))) + (or (fold (lambda (repository others) + (match (find-latest-version name repository) + ((_ version file) `(("metadata" ,@(get-metadata file)) + ("version" . ,version))) + (_ others))) + #f + (map get-opam-repository repositories-specs)) + (throw 'package-not-found repositories-specs))) + +(define* (opam->guix-package name #:key (repo '()) version) + "Import OPAM package NAME from REPOSITORIES (a list of names, URLs or lo= cal +paths, always including OPAM's official repository). Return a 'package' s= exp or #f on failure." - (and-let* ((opam-file (opam-fetch name (get-opam-repository repo))) + (and-let* ((with-opam (if (member "opam" repo) repo (cons "opam" repo))) + (opam-file (opam-fetch name with-opam)) (version (assoc-ref opam-file "version")) (opam-content (assoc-ref opam-file "metadata")) (url-dict (metadata-ref opam-content "url")) @@ -312,9 +345,7 @@ or #f on failure." (values `(package (name ,(ocaml-name->guix-name name)) - (version ,(if (string-prefix? "v" version) - (substring version 1) - version)) + (version ,version) (source (origin (method url-fetch) diff --git a/guix/scripts/import/opam.scm b/guix/scripts/import/opam.scm index 64164e7cc4..837a6ef40f 100644 --- a/guix/scripts/import/opam.scm +++ b/guix/scripts/import/opam.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -81,7 +82,9 @@ Import and convert the opam package for PACKAGE-NAME.\n")) #:build-options? #f)) =20 (let* ((opts (parse-options)) - (repo (and=3D> (assoc-ref opts 'repo) string->symbol)) + (repo (filter-map (match-lambda + (('repo . name) name) + (_ #f)) opts)) (args (filter-map (match-lambda (('argument . value) value) diff --git a/tests/opam.scm b/tests/opam.scm index f1e3b70cb0..dcfbd50e5a 100644 --- a/tests/opam.scm +++ b/tests/opam.scm @@ -90,7 +90,7 @@ url { (with-output-to-file (string-append my-package "/opam") (lambda _ (format #t "~a" test-opam-file)))) - (match (opam->guix-package "foo" #:repo test-repo) + (match (opam->guix-package "foo" #:repo `(,test-repo)) (('package ('name "ocaml-foo") ('version "1.0.0") --=20 2.32.0 --MP_/cz_QVk7./OCdDAm5=mk01Dh--