From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id EGs7F+RGEWHEIwEAgWs5BA (envelope-from ) for ; Mon, 09 Aug 2021 17:16:52 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 9XwkE+RGEWGNFwAAB5/wlQ (envelope-from ) for ; Mon, 09 Aug 2021 15:16:52 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2DF0E37FE for ; Mon, 9 Aug 2021 17:16:51 +0200 (CEST) Received: from localhost ([::1]:41738 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mD71G-0005sj-0G for larch@yhetil.org; Mon, 09 Aug 2021 11:16:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33882) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mD6os-0001OF-Uh for guix-patches@gnu.org; Mon, 09 Aug 2021 11:04:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:45933) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mD6os-0007qF-OD for guix-patches@gnu.org; Mon, 09 Aug 2021 11:04:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mD6os-00029f-Gb for guix-patches@gnu.org; Mon, 09 Aug 2021 11:04:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#49958] [PATCH] More flexibility in opam importer Resent-From: Alice BRENON Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 09 Aug 2021 15:04:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 49958 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 49958@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16285213978219 (code B ref -1); Mon, 09 Aug 2021 15:04:02 +0000 Received: (at submit) by debbugs.gnu.org; 9 Aug 2021 15:03:17 +0000 Received: from localhost ([127.0.0.1]:57477 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mD6o7-00028S-OY for submit@debbugs.gnu.org; Mon, 09 Aug 2021 11:03:17 -0400 Received: from lists.gnu.org ([209.51.188.17]:52684) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mD40t-0001HO-Fm for submit@debbugs.gnu.org; Mon, 09 Aug 2021 08:04:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49860) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mD40s-0006OB-No for guix-patches@gnu.org; Mon, 09 Aug 2021 08:04:14 -0400 Received: from lxc-smtp2.ens-lyon.fr ([140.77.167.81]:51116) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mD40p-0001ra-E5 for guix-patches@gnu.org; Mon, 09 Aug 2021 08:04:14 -0400 Received: from localhost (localhost [127.0.0.1]) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTP id 08232E28D6 for ; Mon, 9 Aug 2021 14:04:08 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.11.0 (20160426) (Debian) at ens-lyon.fr Received: from lxc-smtp2.ens-lyon.fr ([127.0.0.1]) by localhost (lxc-smtp2.ens-lyon.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id weeoDZ23OQLH for ; Mon, 9 Aug 2021 14:04:07 +0200 (CEST) Received: from localhost (unknown [78.194.167.103]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTPSA id E2994E0E67 for ; Mon, 9 Aug 2021 14:04:07 +0200 (CEST) Date: Mon, 9 Aug 2021 14:04:07 +0200 From: Alice BRENON Message-ID: <20210809140407.748fa019@ens-lyon.fr> Organization: ENS de Lyon X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/MP2=ddjP20LDcZckf5S4+Rd" Received-SPF: pass client-ip=140.77.167.81; envelope-from=alice.brenon@ens-lyon.fr; helo=lxc-smtp2.ens-lyon.fr X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Mon, 09 Aug 2021 11:03:14 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Mailman-Approved-At: Mon, 09 Aug 2021 11:16:41 -0400 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1628522211; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:list-id:list-help:list-unsubscribe:list-subscribe: list-post; bh=ZTvFdmNB3zLKJRjHLoL/bFaDNZzchEnH11BWwurVpVs=; b=d46LMuF1watbf5JjM+iVo/pvEpzSXdWxwnILuKDTApTLLserozztP5wgbcnnuOmTjKlVhz zGDp+/NB/0Nsq45QC5kwKsaouw1w8XXij/6iAYdKMx55EVTFbrzscV4JqiHML0spRdQSoy VEVID3pQwzdGwugYrhMnSSTXHtna+7oHieVvZJlySXuA28lg3vZ40DQl/PE7UzJZGocNZ1 3H/sIBUMLQGPqxC3/U/YZEpdIit4jrulYKhMcf0kyZScykGqVRjW/cnXFkM4MWkdYlfV2A Tn85X4iXRTCeI2RuzGDg4oWuk/aq+klAUDxMXAzT/4HtYv0GSqlymPNcPQlxWw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1628522211; a=rsa-sha256; cv=none; b=Cgv8D/zJ76ZMqFiSTnAEqIDf7FbZRFgLNJwkBLZsjDyMDnvF5PU8HGMhIRBc0alGmTaQkJ JnOZHslWOEHbkJzCQ1t3x96IS4jloT9I5SbybEC5M86YAHw4nXEAqH0Gw4MGTHK/YqQVOV flqvuRV0X8VXj2IpNDjenPsE3JeX/mkmLOfSQ+jCGMorplaE501CQKNzh5xmweCVpXm88Q dmO9NiXytqaFsy7/239AiPocCEa8FlfBXWqta4DdyBv4QkeQwQmA9oca9WZoeWwm93jd0j wNekNAfekRxQJ7QU7l+pTN3V8QZLOekBVM6EbufekmKY0xXXlMW7pdAu5sXrgg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -2.41 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 2DF0E37FE X-Spam-Score: -2.41 X-Migadu-Scanner: scn0.migadu.com X-TUID: 0z+nxKmDceYW --MP_/MP2=ddjP20LDcZckf5S4+Rd Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline Hello, I'd like to submit this patch for review, discussion and hopefully inclusion to guix' code. I recently tried to import grew[0], a NLP tool written in ocaml and distributed with opam but from a custom repository. The current importer prevented me to do so for several reasons: - the available repositories were hard-coded to be either opam's official repository or coq's - the repositories were expected to be distributed with git: while public git repositories do exist for coq and opam's official repository, they are not the source of truth for the opam tool one can use in an imperative setup like this: `opam repo add coq-released https://coq.inria.fr/opam/released` it entailed that assumptions were made about the freshness of the git repositories and the actual files served to opam, hence differences could theoretically be observed - it appears that the opam tool doesn't enforce as strict a structure on its repositories as its current documentation[1] suggests. Grew's own repository has all versions of each package directly under `/packages/` instead of in a separate subdirectory. While this deserves a clarification from opam's part, this patch hardens guix opam importer against such exotic layouts. - the unability to query several repositories at once rendered recursive imports inefficient, as some packages on a custom opam repository may still need dependencies from the official opam repository even if no guix package has been imported for it yet (this was the case with ocaml-ANSITerminal in my case) The current proposal attempts to solve these difficulties, and allowed me to actually import the guix declaration for grew and its dependencies. I'm still fixing the imported declaration and intend to submit a separate patch to add it to guix packages when it works. I added grew's custom opam repository to the list of known-repositories because it was my immediate target but this is of course not important and could be reverted in favour of coq and opam's official repository only. I used the emacs scripted auto-indenter to save a little time to my reviewers and had to discard many changes that weren't related to my changes to cut the noise. Maybe this file should be reindented in a separate commit following the approval or rejection of this patch proposal. Cheers, Alice [0]: https://grew.fr/ [1]: https://opam.ocaml.org/doc/Manual.html#Repositories --MP_/MP2=ddjP20LDcZckf5S4+Rd Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=0001-guix-opam.patch =46rom 3beac4d8caf3c6693237bc4f975029e58a20e0f4 Mon Sep 17 00:00:00 2001 From: Alice BRENON Date: Sat, 7 Aug 2021 19:50:10 +0200 Subject: [PATCH] guix: opam: - Add support for importing from several repositories - Add support for custom repositories, either from a URL or= a local path - Use actual opam repositories as defined by the opam tool as source * guix/scripts/import/opam.scm: pass all instances of --repo as a list to the importer * guix/import/opam.scm: - delay repo resolution (call to get-opam-repository from within opam-fetch instead of outside) - use the same repository source as CLI opam does (i.e. HTTP-served index.tar.gz instead of git repositories) - be more flexible on the repositories structure instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/ --- guix/import/opam.scm | 145 +++++++++++++++++++++-------------- guix/scripts/import/opam.scm | 5 +- 2 files changed, 92 insertions(+), 58 deletions(-) diff --git a/guix/import/opam.scm b/guix/import/opam.scm index a35b01d277..c3fad7db65 100644 --- a/guix/import/opam.scm +++ b/guix/import/opam.scm @@ -2,6 +2,7 @@ ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2020 Martin Becze ;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,13 +23,16 @@ #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module (ice-9 popen) #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (ice-9 textual-ports) #:use-module (ice-9 vlist) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) + #:use-module (srfi srfi-26) #:use-module (web uri) + #:use-module ((guix build utils) #:select (dump-port find-files mkdir-p)) #:use-module (guix build-system) #:use-module (guix build-system ocaml) #:use-module (guix http-client) @@ -121,51 +125,78 @@ (define-peg-pattern condition-string all (and QUOTE (* STRCHR) QUOTE)) (define-peg-pattern condition-var all (+ (or (range #\a #\z) "-" ":"))) =20 -(define* (get-opam-repository #:optional repo) +(define (opam-cache-directory path) + (string-append (cache-directory #:ensure? #f) "/opam/" path)) + +(define known-repositories + '((opam . "https://opam.ocaml.org") + (coq . "https://coq.inria.fr/opam/released") + (grew . "http://opam.grew.fr"))) + +(define (repo-type repo) + (define (get-uri repo-root) + (let ((archive-file (string-append repo-root "/index.tar.gz"))) + (or (string->uri archive-file) (throw 'bad-uri archive-file)))) + (match (assoc-ref known-repositories (string->symbol repo)) + (#f (if (file-exists? repo) + `(local ,repo) + `(remote ,(get-uri repo)))) + (url `(remote ,(get-uri url))))) + +(define (update-repository-at output-folder input) + "Make sure the opam repository at OUTPUT-FOLDER is up-to-date with INPUT" + (let ((cached-date (if (file-exists? output-folder) + (stat:mtime (stat output-folder)) + (begin (mkdir-p output-folder) 0)))) + (begin + (and (> (stat:mtime (stat input)) cached-date) + (call-with-port + (open-pipe* OPEN_WRITE "tar" "xz" "-C" output-folder "-f" "-") + (cut dump-port input <>))) + output-folder))) + +(define* (get-opam-repository #:optional (repo "opam")) "Update or fetch the latest version of the opam repository and return the path to the repository." - (let ((url (cond - ((or (not repo) (equal? repo 'opam)) - "https://github.com/ocaml/opam-repository") - ((string-prefix? "coq-" (symbol->string repo)) - "https://github.com/coq/opam-coq-archive") - ((equal? repo 'coq) "https://github.com/coq/opam-coq-archiv= e") - (else (throw 'unknown-repository repo))))) - (receive (location commit _) - (update-cached-checkout url) - (cond - ((or (not repo) (equal? repo 'opam)) - location) - ((equal? repo 'coq) - (string-append location "/released")) - ((string-prefix? "coq-" (symbol->string repo)) - (string-append location "/" (substring (symbol->string repo) 4))) - (else location))))) + (match (repo-type repo) + (('local p) p) + (('remote source) (let ((cache (opam-cache-directory (uri-host source)= ))) + (call-with-port + (http-fetch/cached source) + (cut update-repository-at cache <>)))))) =20 ;; Prevent Guile 3 from inlining this procedure so we can mock it in tests. (set! get-opam-repository get-opam-repository) =20 -(define (latest-version versions) - "Find the most recent version from a list of versions." - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) versions)) +(define (get-version-and-file path) + "Analyse a candidate path and return an list containing information for = proper + version comparison as well as the source path for metadata." + (and-let* ((metadata-file (string-append path "/opam")) + (filename (basename path)) + (version (string-join (cdr (string-split filename #\.)) "."))) + (and (file-exists? metadata-file) + (eq? 'regular (stat:type (stat metadata-file))) + (if (string-prefix? "v" version) + `(V ,(substring version 1) ,metadata-file) + `(digits ,version ,metadata-file))))) + +(define (keep-max-version a b) + "Version comparison on the lists returned by the previous function takin= g the + janestreet re-versioning into account (v-prefixed come first)." + (match (cons a b) + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) + ((('V _ _) . _) a) + ((_ . ('V _ _)) b) + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a b)))) =20 (define (find-latest-version package repository) "Get the latest version of a package as described in the given repositor= y." - (let* ((dir (string-append repository "/packages/" package)) - (versions (scandir dir (lambda (name) (not (string-prefix? "." na= me)))))) - (if versions - (let ((versions (map - (lambda (dir) - (string-join (cdr (string-split dir #\.)) ".")) - versions))) - ;; Workaround for janestreet re-versionning - (let ((v-versions (filter (lambda (version) (string-prefix? "v" ve= rsion)) versions))) - (if (null? v-versions) - (latest-version versions) - (string-append "v" (latest-version (map (lambda (version) (sub= string version 1)) v-versions)))))) - (begin - (format #t (G_ "Package not found in opam repository: ~a~%") packa= ge) - #f)))) + (let ((packages (string-append repository "/packages")) + (filter (make-regexp (string-append "^" package "\\.")))) + (reduce keep-max-version #f + (filter-map + get-version-and-file + (find-files packages filter #:directories? #t))))) =20 (define (get-metadata opam-file) (with-input-from-file opam-file @@ -266,28 +297,30 @@ path to the repository." =20 (define (depends->native-inputs depends) (filter (lambda (name) (not (equal? "" name))) - (map dependency->native-input depends))) + (map dependency->native-input depends))) =20 (define (dependency-list->inputs lst) (map - (lambda (dependency) - (list dependency (list 'unquote (string->symbol dependency)))) - (ocaml-names->guix-names lst))) - -(define* (opam-fetch name #:optional (repository (get-opam-repository))) - (and-let* ((repository repository) - (version (find-latest-version name repository)) - (file (string-append repository "/packages/" name "/" name ".= " version "/opam"))) - `(("metadata" ,@(get-metadata file)) - ("version" . ,(if (string-prefix? "v" version) - (substring version 1) - version))))) - -(define* (opam->guix-package name #:key (repo 'opam) version) - "Import OPAM package NAME from REPOSITORY (a directory name) or, if -REPOSITORY is #f, from the official OPAM repository. Return a 'package' s= exp + (lambda (dependency) + (list dependency (list 'unquote (string->symbol dependency)))) + (ocaml-names->guix-names lst))) + +(define* (opam-fetch name #:optional (repositories-specs '("opam"))) + (or (fold (lambda (repository others) + (match (find-latest-version name repository) + ((_ version file) `(("metadata" ,@(get-metadata file)) + ("version" . ,version))) + (_ others))) + #f + (map get-opam-repository repositories-specs)) + (throw 'package-not-found repositories-specs))) + +(define* (opam->guix-package name #:key (repo '()) version) + "Import OPAM package NAME from REPOSITORIES (a list of names, URLs or lo= cal +paths, always including OPAM's official repository). Return a 'package' s= exp or #f on failure." - (and-let* ((opam-file (opam-fetch name (get-opam-repository repo))) + (and-let* ((with-opam (if (member "opam" repo) repo (cons "opam" repo))) + (opam-file (opam-fetch name with-opam)) (version (assoc-ref opam-file "version")) (opam-content (assoc-ref opam-file "metadata")) (url-dict (metadata-ref opam-content "url")) @@ -312,9 +345,7 @@ or #f on failure." (values `(package (name ,(ocaml-name->guix-name name)) - (version ,(if (string-prefix? "v" version) - (substring version 1) - version)) + (version ,version) (source (origin (method url-fetch) diff --git a/guix/scripts/import/opam.scm b/guix/scripts/import/opam.scm index 64164e7cc4..837a6ef40f 100644 --- a/guix/scripts/import/opam.scm +++ b/guix/scripts/import/opam.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -81,7 +82,9 @@ Import and convert the opam package for PACKAGE-NAME.\n")) #:build-options? #f)) =20 (let* ((opts (parse-options)) - (repo (and=3D> (assoc-ref opts 'repo) string->symbol)) + (repo (filter-map (match-lambda + (('repo . name) name) + (_ #f)) opts)) (args (filter-map (match-lambda (('argument . value) value) --=20 2.32.0 --MP_/MP2=ddjP20LDcZckf5S4+Rd--