From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:470:142:3::10]:35967) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1isrAl-0003ph-Lo for guix-patches@gnu.org; Sat, 18 Jan 2020 11:42:12 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1isrAg-0001d1-H8 for guix-patches@gnu.org; Sat, 18 Jan 2020 11:42:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:36118) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1isrAg-0001cs-7h for guix-patches@gnu.org; Sat, 18 Jan 2020 11:42:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1isrAg-0007ko-5l for guix-patches@gnu.org; Sat, 18 Jan 2020 11:42:02 -0500 Subject: [bug#38408] [PATCH v6] Semantic version aware recusive importer for crates Resent-Message-ID: References: <877e2sm0vv.fsf@gnu.org> <6e56589639ea75bfec2c97f7e9e31ad9@riseup.net> <87blrtfwbx.fsf@gnu.org> From: Martin Becze Message-ID: <4c06cf99-30bc-2440-e833-52781376e302@riseup.net> Date: Sat, 18 Jan 2020 11:40:37 -0500 MIME-Version: 1.0 In-Reply-To: <87blrtfwbx.fsf@gnu.org> Content-Type: multipart/mixed; boundary="------------F23251DF520B064EFB5C0863" Content-Language: en-US List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 38408@debbugs.gnu.org This is a multi-part message in MIME format. --------------F23251DF520B064EFB5C0863 Content-Type: multipart/alternative; boundary="------------D19E3F775F4780461428CFBF" --------------D19E3F775F4780461428CFBF Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit okkkie! finally got this rewote! Patches attached. > As noted in my previous message, I think this interface is too > complex, and since it’s used in a single importer, it would be best > to have it directly in (guix import crate). This is now done! > In the meantime, there are probably semver-related things that could > naturally to a helper (guix import semver) module, although perhaps > most of that is already provided by guile-semver? guile-semver is pretty easy to work with so I didn't need to use many helpers. The one I did use I think I will try to submit to guile-semver itself. On 12/27/19 1:38 PM, Ludovic Courtès wrote: > Hi Martin, > > Sorry for the late reply. > > Martin Becze skribis: > >>> Providing an explicit cache bypassing method also sounds >>> worrying to me: the cache is supposed to be transparent and >>> semantics-preserving. >>> >>> More generally, I think adding new features to an importer >>> shouldn’t require modifications in this area, as a matter of >>> separating concerns. >>> >>> WDYT? >>> >>> Thanks, Ludo’. >> >> yes I agree, I removed that in the last version! Which also >> rebases off the new topological sort procedure. I'll attach the >> latest patches here, in case you missed it. > > Thanks! > >> From eeffdf569c4d7fbfd843e0b48404b6a2f3d46343 Mon Sep 17 00:00:00 >> 2001 From: Martin Becze Date: Mon, 16 Dec >> 2019 17:08:16 -0500 Subject: [PATCH v5 1/4] guix: import: added >> recusive-import-semver >> >> * guix/import/utils.scm (recusive-import-semver): New Varible * >> guix/import/utils.scm (package->definition)[arguments]: Add >> append-verions option > > [...] > >> +(define* (recursive-import-semver #:key + name + (range "*") + >> name->metadata + metadata->package + metadata-versions + >> package-dependencies + dependency-name + dependency-range + >> guix-name + make-sexp) + "Generates a list of package expressions >> for the dependencies of the given +NAME and version RANGE. The >> dependencies will be resolved using semantic versioning. +This >> procedure makes the assumption that most package repositories will, >> for a +given package provide some on that package that >> includes what +versions of the package that are available and a >> list of dependencies for each +version. Dependencies are assumed to >> be composed of a NAME, a semantic RANGE and +other data. + +This >> procedure takes the following keys: + NAME - The name of the >> package to import + RANGE - The version range of the package to >> import + NAME->METADATA - A procedure that takes a NAME of a >> package and returns that +package's + METADATA->PACKAGE >> A procedure that takes a package's and VERSION +and >> returns the for the given VERSION + METADATA-VERSIONS A >> procedure that that takes a packages and +returns a >> list of version as strings that are available for the given package >> + PACKAGE-DEPENDENCIES a procedure that returns a list of >> given a + + DEPENDENCY-NAME A procedure >> that takes a and returns the its name + >> DEPENDENCY-RANGE A procedure that takes a and returns >> that +decency's range as a string + GUIX-NAME A procedure that >> take a NAME and returns the Guix version of it + MAKE-SEXP A >> procedure that takes , and a list of pairs >> +containing (EXPORT-NAME ), returning the package >> expression as an +s-expression" > > As noted in my previous message, I think this interface is too > complex, and since it’s used in a single importer, it would be best > to have it directly in (guix import crate). > > If/when there’s a need to share that logic among several importers, > then we can look for ways to factorize whatever needs to be > factorized. > > In the meantime, there are probably semver-related things that could > naturally to a helper (guix import semver) module, although perhaps > most of that is already provided by guile-semver? > >> + (define-record-type > > Also, as a rule of thumb, we wouldn’t want to duplicate these data > types and related code. > > WDYT? > > Thanks, Ludo’. --------------D19E3F775F4780461428CFBF Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit okkkie! finally got this rewote! Patches attached.

> As noted in my previous message, I think this interface is too > complex, and since it’s used in a single importer, it would be best > to have it directly in (guix import crate).
This is now done!

> In the meantime, there are probably semver-related things that could > naturally to a helper (guix import semver) module, although perhaps > most of that is already provided by guile-semver?
guile-semver is pretty easy to work with so I didn't need to use many helpers. The one I did use I think I will try to submit to guile-semver itself.

On 12/27/19 1:38 PM, Ludovic Courtès wrote:
> Hi Martin, > > Sorry for the late reply. > > Martin Becze <mjbecze@riseup.net> skribis: > >>> Providing an explicit cache bypassing method also sounds >>> worrying to me: the cache is supposed to be transparent and >>> semantics-preserving. >>> >>> More generally, I think adding new features to an importer >>> shouldn’t require modifications in this area, as a matter of >>> separating concerns. >>> >>> WDYT? >>> >>> Thanks, Ludo’. >> >> yes I agree, I removed that in the last version! Which also >> rebases off the new topological sort procedure. I'll attach the >> latest patches here, in case you missed it. > > Thanks! > >> From eeffdf569c4d7fbfd843e0b48404b6a2f3d46343 Mon Sep 17 00:00:00 >> 2001 From: Martin Becze <mjbecze@riseup.net> Date: Mon, 16 Dec >> 2019 17:08:16 -0500 Subject: [PATCH v5 1/4] guix: import: added >> recusive-import-semver >> >> * guix/import/utils.scm (recusive-import-semver): New Varible * >> guix/import/utils.scm (package->definition)[arguments]: Add >> append-verions option > > [...] > >> +(define* (recursive-import-semver #:key + name + (range "*") + >> name->metadata + metadata->package + metadata-versions + >> package-dependencies + dependency-name + dependency-range + >> guix-name + make-sexp) + "Generates a list of package expressions >> for the dependencies of the given +NAME and version RANGE. The >> dependencies will be resolved using semantic versioning. +This >> procedure makes the assumption that most package repositories will, >> for a +given package provide some <metadata> on that package that >> includes what +versions of the package that are available and a >> list of dependencies for each +version. Dependencies are assumed to >> be composed of a NAME, a semantic RANGE and +other data. + +This >> procedure takes the following keys: + NAME - The name of the >> package to import + RANGE - The version range of the package to >> import + NAME->METADATA - A procedure that takes a NAME of a >> package and returns that +package's <metadata> + METADATA->PACKAGE >> A procedure that takes a package's <metadata> and VERSION +and >> returns the <package> for the given VERSION + METADATA-VERSIONS A >> procedure that that takes a packages <metadata> and +returns a >> list of version as strings that are available for the given package >> + PACKAGE-DEPENDENCIES a procedure that returns a list of >> <dependency> given a +<package> + DEPENDENCY-NAME A procedure >> that takes a <dependency> and returns the its name + >> DEPENDENCY-RANGE A procedure that takes a <dependency> and returns >> that +decency's range as a string + GUIX-NAME A procedure that >> take a NAME and returns the Guix version of it + MAKE-SEXP A >> procedure that takes <metadata>, <package> and a list of pairs >> +containing (EXPORT-NAME <dependency>), returning the package >> expression as an +s-expression" > > As noted in my previous message, I think this interface is too > complex, and since it’s used in a single importer, it would be best > to have it directly in (guix import crate). > > If/when there’s a need to share that logic among several importers, > then we can look for ways to factorize whatever needs to be > factorized. > > In the meantime, there are probably semver-related things that could > naturally to a helper (guix import semver) module, although perhaps > most of that is already provided by guile-semver? > >> + (define-record-type <node-dependency> > > Also, as a rule of thumb, we wouldn’t want to duplicate these data > types and related code. > > WDYT? > > Thanks, Ludo’.
--------------D19E3F775F4780461428CFBF-- --------------F23251DF520B064EFB5C0863 Content-Type: text/x-patch; charset=UTF-8; name="v6-0001-guix-import-recursive-import-Allow-for-version-nu.patch" Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename*0="v6-0001-guix-import-recursive-import-Allow-for-version-nu.pa"; filename*1="tch" >From 7747d27ed32069acb193b3811b30a5a3cc8cfd5b Mon Sep 17 00:00:00 2001 From: Martin Becze Date: Sat, 18 Jan 2020 05:05:03 -0500 Subject: [PATCH v6 1/2] guix: import: (recursive-import) Allow for version numbers * guix/import/utils.scm (package->definition) [arguments] added optional `append-version?` * guix/import/utils.scm (recursive-import) [arguments] added key `version` and moved `repo` to be a key * tests/import-utils.scm * guix/import/cran.scm (cran->guix-package) [argument]: change `repo` to a key * guix/import/cran.scm (cran-recursive-import) [argument]: change `repo` to a key * guix/scripts/import/cran.scm: change `repo` to a key * guix/import/elpa.scm (elpa->guix-pakcage) [argumnets]: change `repo` to a key * guix/import/elpa.scm (elpa-recursive-import) [argumnets]: change `repo` to a key * guix/scripts/import/elpa.scm: change `repo` to a key * guix/import/gem.scm (gem->guix-package) [arguments]: change `repo` to a key * guix/import/gem.scm (recursive-import) [arguments]: change `repo` to a key * guix/import/opam.scm (opam-recurive-import) [arguments]: change `repo` to a key * guix/import/pypi.scm (pypi-recursive-import) [arguments]: change `repo` to a key * guix/import/stackage.scm (stackage-recursive-import) [arguments]: change `repo` to a key --- guix/import/cran.scm | 8 +++-- guix/import/elpa.scm | 6 ++-- guix/import/gem.scm | 6 ++-- guix/import/opam.scm | 5 ++-- guix/import/pypi.scm | 5 ++-- guix/import/stackage.scm | 5 ++-- guix/import/utils.scm | 57 +++++++++++++++++++++++------------- guix/scripts/import/cran.scm | 5 ++-- guix/scripts/import/elpa.scm | 4 ++- tests/import-utils.scm | 8 +++-- 10 files changed, 69 insertions(+), 40 deletions(-) diff --git a/guix/import/cran.scm b/guix/import/cran.scm index bcb37ed250..9e05dfcba8 100644 --- a/guix/import/cran.scm +++ b/guix/import/cran.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2015, 2016, 2017, 2018, 2019 Ricardo Wurmus ;;; Copyright © 2015, 2016, 2017, 2019, 2020 Ludovic Courtès ;;; Copyright © 2017 Mathieu Othacehe +;;; Copyright © 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -506,7 +507,7 @@ from the alist META, which was derived from the R package's DESCRIPTION file." (define cran->guix-package (memoize - (lambda* (package-name #:optional (repo 'cran)) + (lambda* (package-name #:key (repo 'cran) #:allow-other-keys) "Fetch the metadata for PACKAGE-NAME from REPO and return the `package' s-expression corresponding to that package, or #f on failure." (let ((description (fetch-description repo package-name))) @@ -521,8 +522,9 @@ s-expression corresponding to that package, or #f on failure." (cran->guix-package package-name 'cran)) (else (values #f '())))))))) -(define* (cran-recursive-import package-name #:optional (repo 'cran)) - (recursive-import package-name repo +(define* (cran-recursive-import package-name #:key (repo 'cran)) + (recursive-import package-name + #:repo repo #:repo->guix-package cran->guix-package #:guix-name cran-guix-name)) diff --git a/guix/import/elpa.scm b/guix/import/elpa.scm index 2d4487dba0..9140bcdc34 100644 --- a/guix/import/elpa.scm +++ b/guix/import/elpa.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2015 Federico Beffa ;;; Copyright © 2015, 2016, 2017, 2018, 2020 Ludovic Courtès ;;; Copyright © 2018 Oleg Pykhalov +;;; Copyright © 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -245,7 +246,7 @@ type ''." (license ,license)) dependencies-names))) -(define* (elpa->guix-package name #:optional (repo 'gnu)) +(define* (elpa->guix-package name #:key (repo 'gnu) #:allow-other-keys) "Fetch the package NAME from REPO and produce a Guix package S-expression." (match (fetch-elpa-package name repo) (#f #f) @@ -301,7 +302,8 @@ type ''." (define elpa-guix-name (cut guix-name "emacs-" <>)) (define* (elpa-recursive-import package-name #:optional (repo 'gnu)) - (recursive-import package-name repo + (recursive-import package-name + #:repo repo #:repo->guix-package elpa->guix-package #:guix-name elpa-guix-name)) diff --git a/guix/import/gem.scm b/guix/import/gem.scm index 0bf9ff2552..e744d9e69d 100644 --- a/guix/import/gem.scm +++ b/guix/import/gem.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2015 David Thompson ;;; Copyright © 2016 Ben Woodcroft ;;; Copyright © 2018 Oleg Pykhalov +;;; Copyright © 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -117,7 +118,7 @@ VERSION, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES." ((license) (license->symbol license)) (_ `(list ,@(map license->symbol licenses))))))) -(define* (gem->guix-package package-name #:optional (repo 'rubygems) version) +(define* (gem->guix-package package-name #:key (repo 'rubygems) version) "Fetch the metadata for PACKAGE-NAME from rubygems.org, and return the `package' s-expression corresponding to that package, or #f on failure." (let ((package (rubygems-fetch package-name))) @@ -201,6 +202,7 @@ package on RubyGems." (latest latest-release))) (define* (gem-recursive-import package-name #:optional version) - (recursive-import package-name '() + (recursive-import package-name + #:repo '() #:repo->guix-package gem->guix-package #:guix-name ruby-package-name)) diff --git a/guix/import/opam.scm b/guix/import/opam.scm index 394415fdd4..87c823a98c 100644 --- a/guix/import/opam.scm +++ b/guix/import/opam.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2018 Julien Lepiller +;;; Copyright © 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -311,8 +312,8 @@ or #f on failure." dependencies)))))))) (define (opam-recursive-import package-name) - (recursive-import package-name #f - #:repo->guix-package (lambda (name repo) + (recursive-import package-name + #:repo->guix-package (lambda (name . _) (opam->guix-package name)) #:guix-name ocaml-name->guix-name)) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 354cae9c4c..f0702d6403 100644 --- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2017 Mathieu Othacehe ;;; Copyright © 2018 Ricardo Wurmus ;;; Copyright © 2019 Maxim Cournoyer +;;; Copyright © 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -415,8 +416,8 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE." description license)))))))) (define (pypi-recursive-import package-name) - (recursive-import package-name #f - #:repo->guix-package (lambda (name repo) + (recursive-import package-name + #:repo->guix-package (lambda (name . _) (pypi->guix-package name)) #:guix-name python->package-name)) diff --git a/guix/import/stackage.scm b/guix/import/stackage.scm index 14150201b5..6091cf2c64 100644 --- a/guix/import/stackage.scm +++ b/guix/import/stackage.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017 Federico Beffa ;;; Copyright © 2018 Ricardo Wurmus +;;; Copyright © 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -108,8 +109,8 @@ included in the Stackage LTS release." (leave-with-message "~a: Stackage package not found" package-name)))))) (define (stackage-recursive-import package-name . args) - (recursive-import package-name #f - #:repo->guix-package (lambda (name repo) + (recursive-import package-name + #:repo->guix-package (lambda (name . _) (apply stackage->guix-package (cons name args))) #:guix-name hackage-name->package-name)) diff --git a/guix/import/utils.scm b/guix/import/utils.scm index d17d400ddf..59430d3e66 100644 --- a/guix/import/utils.scm +++ b/guix/import/utils.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2017, 2019 Ricardo Wurmus ;;; Copyright © 2018 Oleg Pykhalov ;;; Copyright © 2019 Robert Vollmert +;;; Copyright © 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -44,6 +45,7 @@ #:use-module (srfi srfi-9) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-71) #:export (factorize-uri flatten @@ -258,13 +260,15 @@ package definition." ((package-inputs ...) `((native-inputs (,'quasiquote ,package-inputs)))))) -(define (package->definition guix-package) +(define* (package->definition guix-package #:optional append-version?) (match guix-package - (('package ('name (? string? name)) _ ...) - `(define-public ,(string->symbol name) - ,guix-package)) - (('let anything ('package ('name (? string? name)) _ ...)) - `(define-public ,(string->symbol name) + ((or + ('package ('name name) ('version version) . rest) + ('let _ ('package ('name name) ('version version) . rest))) + + `(define-public ,(string->symbol (if append-version? + (string-append name "-" version) + version)) ,guix-package)))) (define (build-system-modules) @@ -399,32 +403,43 @@ obtain a node's uniquely identifying \"key\"." (cons head result) (set-insert (node-name head) visited)))))))) -(define* (recursive-import package-name repo - #:key repo->guix-package guix-name +(define* (recursive-import package-name + #:key repo->guix-package guix-name version repo #:allow-other-keys) "Return a list of package expressions for PACKAGE-NAME and all its dependencies, sorted in topological order. For each package, -call (REPO->GUIX-PACKAGE NAME REPO), which should return a package expression -and a list of dependencies; call (GUIX-NAME NAME) to obtain the Guix package -name corresponding to the upstream name." +call (REPO->GUIX-PACKAGE NAME :KEYS version repo), which should return a +package expression and a list of dependencies; call (GUIX-NAME NAME) to +obtain the Guix package name corresponding to the upstream name." (define-record-type - (make-node name package dependencies) + (make-node name version package dependencies) node? (name node-name) + (version node-version) (package node-package) (dependencies node-dependencies)) - (define (exists? name) - (not (null? (find-packages-by-name (guix-name name))))) + (define (exists? name version) + (not (null? (find-packages-by-name (guix-name name) version)))) - (define (lookup-node name) - (receive (package dependencies) (repo->guix-package name repo) - (make-node name package dependencies))) + (define (lookup-node name version) + (let* ((package dependencies (repo->guix-package name + #:version version + #:repo repo)) + (normilizied-deps (map (match-lambda + ((name version) (list name version)) + (name (list name #f))) dependencies))) + (make-node name version package normilizied-deps))) (map node-package - (topological-sort (list (lookup-node package-name)) + (topological-sort (list (lookup-node package-name version)) (lambda (node) - (map lookup-node - (remove exists? + (map (lambda (name-version) + (apply lookup-node name-version)) + (remove (lambda (name-version) + (apply exists? name-version)) (node-dependencies node)))) - node-name))) + (lambda (node) + (string-append + (node-name node) + (or (node-version node) "")))))) diff --git a/guix/scripts/import/cran.scm b/guix/scripts/import/cran.scm index d6f371ef3a..bc266ad9da 100644 --- a/guix/scripts/import/cran.scm +++ b/guix/scripts/import/cran.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014 Eric Bavier ;;; Copyright © 2015, 2017, 2019 Ricardo Wurmus +;;; Copyright © 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -98,10 +99,10 @@ Import and convert the CRAN package for PACKAGE-NAME.\n")) ;; Recursive import (map package->definition (cran-recursive-import package-name - (or (assoc-ref opts 'repo) 'cran))) + #:repo (or (assoc-ref opts 'repo) 'cran))) ;; Single import (let ((sexp (cran->guix-package package-name - (or (assoc-ref opts 'repo) 'cran)))) + #:repo (or (assoc-ref opts 'repo) 'cran)))) (unless sexp (leave (G_ "failed to download description for package '~a'~%") package-name)) diff --git a/guix/scripts/import/elpa.scm b/guix/scripts/import/elpa.scm index d270d2b4bc..07ac07a3d5 100644 --- a/guix/scripts/import/elpa.scm +++ b/guix/scripts/import/elpa.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 Federico Beffa ;;; Copyright © 2018 Oleg Pykhalov +;;; Copyright © 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -102,7 +103,8 @@ Import the latest package named PACKAGE-NAME from an ELPA repository.\n")) (_ #f)) (elpa-recursive-import package-name (or (assoc-ref opts 'repo) 'gnu))) - (let ((sexp (elpa->guix-package package-name (assoc-ref opts 'repo)))) + (let ((sexp (elpa->guix-package package-name + #:repo (assoc-ref opts 'repo)))) (unless sexp (leave (G_ "failed to download package '~a'~%") package-name)) sexp))) diff --git a/tests/import-utils.scm b/tests/import-utils.scm index 87dda3238f..2357ea5c40 100644 --- a/tests/import-utils.scm +++ b/tests/import-utils.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015, 2017 Ricardo Wurmus ;;; Copyright © 2016 Ben Woodcroft +;;; Copyright © 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -48,15 +49,16 @@ (package (name "foo") (inputs `(("bar" ,bar))))) - (recursive-import "foo" 'repo + (recursive-import "foo" + #:repo 'repo #:repo->guix-package (match-lambda* - (("foo" 'repo) + (("foo" #:version #f #:repo 'repo) (values '(package (name "foo") (inputs `(("bar" ,bar)))) '("bar"))) - (("bar" 'repo) + (("bar" #:version #f #:repo 'repo) (values '(package (name "bar")) '()))) -- 2.25.0 --------------F23251DF520B064EFB5C0863 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-guix-import-crate-Use-semver-to-resovle-module-ve.patch" Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename*0="v6-0002-guix-import-crate-Use-semver-to-resovle-module-ve.pa"; filename*1="tch" >From a405e3ffdf2716b9920f6b74e4690c9b7731f67a Mon Sep 17 00:00:00 2001 From: Martin Becze Date: Sat, 18 Jan 2020 10:44:18 -0500 Subject: [PATCH v6 2/2] guix: import: crate: Use semver to resovle module versions * guix/import/crate.scm (make-crate-sexp): formatting, added '#:skip-build?' to build system args; added package definition geneation * guix/import/crate.scm (crate->guix-package): [arguments] moved `verions` to a key. Use semver to resolve the correct module versions * guix/import/crate.scm (crate-name->package0name): [arguments] add #:optional `version` arguement * guix/scripts/import/crate.scm remove package definition generation; changed `version` to a key * tests/crate.scm: added version data to (recursuve-import) test --- guix/import/crate.scm | 140 +++++++++------- guix/scripts/import/crate.scm | 11 +- tests/crate.scm | 290 +++++++++++++++++++--------------- tests/elpa.scm | 3 +- 4 files changed, 258 insertions(+), 186 deletions(-) diff --git a/guix/import/crate.scm b/guix/import/crate.scm index 57823c3639..6847a7046b 100644 --- a/guix/import/crate.scm +++ b/guix/import/crate.scm @@ -1,7 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2016 David Craven ;;; Copyright © 2019, 2020 Ludovic Courtès -;;; Copyright © 2019 Martin Becze +;;; Copyright © 2019, 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -28,6 +28,7 @@ #:use-module (guix import json) #:use-module (guix import utils) #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix memoization) #:use-module (guix monads) #:use-module (guix packages) #:use-module (guix upstream) @@ -35,9 +36,12 @@ #:use-module (ice-9 match) #:use-module (ice-9 regex) #:use-module (json) + #:use-module (semver) + #:use-module (semver ranges) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-71) #:export (crate->guix-package guix-package->crate-name string->license @@ -86,7 +90,7 @@ crate-dependency? json->crate-dependency (id crate-dependency-id "crate_id") ;string - (kind crate-dependency-kind "kind" ;'normal | 'dev + (kind crate-dependency-kind "kind" ;'normal | 'dev | 'build string->symbol) (requirement crate-dependency-requirement "req")) ;string @@ -105,6 +109,8 @@ record or #f if it was not found." (json->crate `(,@alist ("actual_versions" . ,versions)))))))) +(define mem-lookup-crate (memoize lookup-crate)) + (define (crate-version-dependencies version) "Return the list of records of VERSION, a ." @@ -150,34 +156,40 @@ VERSION, CARGO-INPUTS, CARGO-DEVELOPMENT-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTIO and LICENSE." (let* ((port (http-fetch (crate-uri name version))) (guix-name (crate-name->package-name name)) - (cargo-inputs (map crate-name->package-name cargo-inputs)) - (cargo-development-inputs (map crate-name->package-name - cargo-development-inputs)) + (cargo-inputs + (map + (lambda (name-version) + (apply crate-name->package-name name-version)) cargo-inputs)) + (cargo-development-inputs + (map + (lambda (name-version) + (apply crate-name->package-name name-version)) cargo-development-inputs)) (pkg `(package - (name ,guix-name) - (version ,version) - (source (origin - (method url-fetch) - (uri (crate-uri ,name version)) - (file-name (string-append name "-" version ".tar.gz")) - (sha256 - (base32 - ,(bytevector->nix-base32-string (port-sha256 port)))))) - (build-system cargo-build-system) - ,@(maybe-arguments (append (maybe-cargo-inputs cargo-inputs) - (maybe-cargo-development-inputs + (name ,guix-name) + (version ,version) + (source (origin + (method url-fetch) + (uri (crate-uri ,name version)) + (file-name (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + ,(bytevector->nix-base32-string (port-sha256 port)))))) + (build-system cargo-build-system) + ,@(maybe-arguments (append `(#:skip-build? #t) + (maybe-cargo-inputs cargo-inputs) + (maybe-cargo-development-inputs cargo-development-inputs))) - (home-page ,(match home-page - (() "") - (_ home-page))) - (synopsis ,synopsis) - (description ,(beautify-description description)) - (license ,(match license - (() #f) - ((license) license) - (_ `(list ,@license))))))) - (close-port port) - pkg)) + (home-page ,(match home-page + (() "") + (_ home-page))) + (synopsis ,synopsis) + (description ,(beautify-description description)) + (license ,(match license + (() #f) + ((license) license) + (_ `(list ,@license))))))) + (close-port port) + (package->definition pkg #t))) (define (string->license string) (filter-map (lambda (license) @@ -188,37 +200,60 @@ and LICENSE." 'unknown-license!))) (string-split string (string->char-set " /")))) -(define* (crate->guix-package crate-name #:optional version) +(define* (crate->guix-package crate-name #:key version #:allow-other-keys) "Fetch the metadata for CRATE-NAME from crates.io, and return the `package' s-expression corresponding to that package, or #f on failure. When VERSION is specified, attempt to fetch that version; otherwise fetch the latest version of CRATE-NAME." + (define (semver-range-contains-string? range version) + (semver-range-contains? (string->semver-range range) + (string->semver version))) + (define (normal-dependency? dependency) - (eq? (crate-dependency-kind dependency) 'normal)) + (or (eq? (crate-dependency-kind dependency) 'build) + (eq? (crate-dependency-kind dependency) 'normal))) (define crate - (lookup-crate crate-name)) + (mem-lookup-crate crate-name)) (define version-number (or version (crate-latest-version crate))) - (define version* + (define (find-version crate range) + "finds the a vesion of a crate that fulfils the semver " (find (lambda (version) - (string=? (crate-version-number version) - version-number)) + (semver-range-contains-string? + range + (crate-version-number version))) (crate-versions crate))) + (define version* + (find-version crate version-number)) + + (define (sort-map-deps deps) + "sorts the dependencies and maps the dependencies to a list + containing pairs of (name version)" + (sort (map (lambda (dep) + (let* ((name (crate-dependency-id dep)) + (crate (mem-lookup-crate name)) + (req (crate-dependency-requirement dep)) + (ver (find-version crate req))) + (list name + (crate-version-number ver)))) + deps) + (match-lambda* (((_ name) ...) + (apply string-ci (crate-version-license version*) string->license)) - (append cargo-inputs cargo-development-inputs))))) + cargo-inputs)))) -(define* (crate-recursive-import crate-name #:optional version) - (recursive-import crate-name #f - #:repo->guix-package - (lambda (name repo) - (let ((version (and (string=? name crate-name) - version))) - (crate->guix-package name version))) +(define* (crate-recursive-import crate-name #:key version) + (recursive-import crate-name + #:repo->guix-package crate->guix-package + #:version version #:guix-name crate-name->package-name)) (define (guix-package->crate-name package) @@ -252,8 +284,11 @@ latest version of CRATE-NAME." (match parts ((name _ ...) name)))) -(define (crate-name->package-name name) - (string-append "rust-" (string-join (string-split name #\_) "-"))) +(define* (crate-name->package-name name #:optional version) + (let ((name (guix-name "rust-" name))) + (if version + (string-append name "-" version) + name))) ;;; @@ -288,4 +323,3 @@ latest version of CRATE-NAME." (description "Updater for crates.io packages") (pred crate-package?) (latest latest-release))) - diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scm index d834518c18..552628cfc7 100644 --- a/guix/scripts/import/crate.scm +++ b/guix/scripts/import/crate.scm @@ -2,7 +2,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2014 David Thompson ;;; Copyright © 2016 David Craven -;;; Copyright © 2019 Martin Becze +;;; Copyright © 2019, 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -95,13 +95,8 @@ Import and convert the crate.io package for PACKAGE-NAME.\n")) (package-name->name+version spec)) (if (assoc-ref opts 'recursive) - (map (match-lambda - ((and ('package ('name name) . rest) pkg) - `(define-public ,(string->symbol name) - ,pkg)) - (_ #f)) - (crate-recursive-import name version)) - (let ((sexp (crate->guix-package name version))) + (crate-recursive-import name #:version version) + (let ((sexp (crate->guix-package name #:version version))) (unless sexp (leave (G_ "failed to download meta-data for package '~a'~%") (if version diff --git a/tests/crate.scm b/tests/crate.scm index aa51faebf9..39561d5745 100644 --- a/tests/crate.scm +++ b/tests/crate.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2014 David Thompson ;;; Copyright © 2016 David Craven ;;; Copyright © 2019, 2020 Ludovic Courtès +;;; Copyright © 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -54,8 +55,9 @@ "{ \"dependencies\": [ { - \"crate_id\": \"bar\", + \"crate_id\": \"leaf-alice\", \"kind\": \"normal\", + \"req\": \"1.0.0\", } ] }") @@ -88,18 +90,22 @@ { \"crate_id\": \"intermediate-1\", \"kind\": \"normal\", + \"req\": \"1.0.0\", }, { \"crate_id\": \"intermediate-2\", \"kind\": \"normal\", + \"req\": \"1.0.0\", } { \"crate_id\": \"leaf-alice\", \"kind\": \"normal\", + \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\", + \"req\": \"1.0.0\", }, ] }") @@ -132,14 +138,17 @@ { \"crate_id\": \"intermediate-2\", \"kind\": \"normal\", + \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-alice\", \"kind\": \"normal\", + \"req\": \"1.0.0\", }, { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\", + \"req\": \"1.0.0\", } ] }") @@ -172,6 +181,7 @@ { \"crate_id\": \"leaf-bob\", \"kind\": \"normal\", + \"req\": \"1.0.0\", }, ] }") @@ -252,34 +262,48 @@ (open-input-string test-foo-crate)) ("https://crates.io/api/v1/crates/foo/1.0.0/download" (set! test-source-hash - (bytevector->nix-base32-string - (sha256 (string->bytevector "empty file\n" "utf-8")))) + (bytevector->nix-base32-string + (sha256 (string->bytevector "empty file\n" "utf-8")))) (open-input-string "empty file\n")) ("https://crates.io/api/v1/crates/foo/1.0.0/dependencies" (open-input-string test-foo-dependencies)) + ("https://crates.io/api/v1/crates/leaf-alice" + (open-input-string test-leaf-alice-crate)) + ("https://crates.io/api/v1/crates/leaf-alice/1.0.0/download" + (set! test-source-hash + (bytevector->nix-base32-string + (sha256 (string->bytevector "empty file\n" "utf-8")))) + (open-input-string "empty file\n")) + ("https://crates.io/api/v1/crates/leaf-alice/1.0.0/dependencies" + (open-input-string test-leaf-alice-dependencies)) (_ (error "Unexpected URL: " url))))) - (match (crate->guix-package "foo") - (('package - ('name "rust-foo") - ('version "1.0.0") - ('source ('origin - ('method 'url-fetch) - ('uri ('crate-uri "foo" 'version)) - ('file-name ('string-append 'name "-" 'version ".tar.gz")) - ('sha256 - ('base32 - (? string? hash))))) - ('build-system 'cargo-build-system) - ('arguments - ('quasiquote - ('#:cargo-inputs (("rust-bar" ('unquote rust-bar)))))) - ('home-page "http://example.com") - ('synopsis "summary") - ('description "summary") - ('license ('list 'license:expat 'license:asl2.0))) - (string=? test-source-hash hash)) - (x - (pk 'fail x #f))))) + + (match (crate->guix-package "foo") + ((define-public rust-foo-1.0.0 + (package (name "rust-foo") + (version "1.0.0") + (source + (origin + (method url-fetch) + (uri (crate-uri "foo" 'version)) + (file-name (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system 'cargo-build-system) + (arguments + ('quasiquote + (#:skip-build? #t + #:cargo-inputs + (("rust-leaf-alice-1.0.0" ('unquote rust-leaf-alice-1.0.0)))))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + + (string=? test-source-hash hash)) + (x + (pk 'fail x #f))))) (test-assert "cargo-recursive-import" ;; Replace network resources with sample data. @@ -334,105 +358,123 @@ (_ (error "Unexpected URL: " url))))) (match (crate-recursive-import "root") ;; rust-intermediate-2 has no dependency on the rust-leaf-alice package, so this is a valid ordering - ((('package - ('name "rust-leaf-alice") - ('version (? string? ver)) - ('source - ('origin - ('method 'url-fetch) - ('uri ('crate-uri "leaf-alice" 'version)) - ('file-name - ('string-append 'name "-" 'version ".tar.gz")) - ('sha256 - ('base32 - (? string? hash))))) - ('build-system 'cargo-build-system) - ('home-page "http://example.com") - ('synopsis "summary") - ('description "summary") - ('license ('list 'license:expat 'license:asl2.0))) - ('package - ('name "rust-leaf-bob") - ('version (? string? ver)) - ('source - ('origin - ('method 'url-fetch) - ('uri ('crate-uri "leaf-bob" 'version)) - ('file-name - ('string-append 'name "-" 'version ".tar.gz")) - ('sha256 - ('base32 - (? string? hash))))) - ('build-system 'cargo-build-system) - ('home-page "http://example.com") - ('synopsis "summary") - ('description "summary") - ('license ('list 'license:expat 'license:asl2.0))) - ('package - ('name "rust-intermediate-2") - ('version (? string? ver)) - ('source - ('origin - ('method 'url-fetch) - ('uri ('crate-uri "intermediate-2" 'version)) - ('file-name - ('string-append 'name "-" 'version ".tar.gz")) - ('sha256 - ('base32 - (? string? hash))))) - ('build-system 'cargo-build-system) - ('arguments - ('quasiquote - ('#:cargo-inputs (("rust-leaf-bob" ('unquote rust-leaf-bob)))))) - ('home-page "http://example.com") - ('synopsis "summary") - ('description "summary") - ('license ('list 'license:expat 'license:asl2.0))) - ('package - ('name "rust-intermediate-1") - ('version (? string? ver)) - ('source - ('origin - ('method 'url-fetch) - ('uri ('crate-uri "intermediate-1" 'version)) - ('file-name - ('string-append 'name "-" 'version ".tar.gz")) - ('sha256 - ('base32 - (? string? hash))))) - ('build-system 'cargo-build-system) - ('arguments - ('quasiquote - ('#:cargo-inputs (("rust-intermediate-2" ('unquote rust-intermediate-2)) - ("rust-leaf-alice" ('unquote rust-leaf-alice)) - ("rust-leaf-bob" ('unquote rust-leaf-bob)))))) - ('home-page "http://example.com") - ('synopsis "summary") - ('description "summary") - ('license ('list 'license:expat 'license:asl2.0))) - ('package - ('name "rust-root") - ('version (? string? ver)) - ('source - ('origin - ('method 'url-fetch) - ('uri ('crate-uri "root" 'version)) - ('file-name - ('string-append 'name "-" 'version ".tar.gz")) - ('sha256 - ('base32 - (? string? hash))))) - ('build-system 'cargo-build-system) - ('arguments - ('quasiquote - ('#:cargo-inputs (("rust-intermediate-1" ('unquote rust-intermediate-1)) - ("rust-intermediate-2" ('unquote rust-intermediate-2)) - ("rust-leaf-alice" ('unquote rust-leaf-alice)) - ("rust-leaf-bob" ('unquote rust-leaf-bob)))))) - ('home-page "http://example.com") - ('synopsis "summary") - ('description "summary") - ('license ('list 'license:expat 'license:asl2.0)))) + (((define-public rust-leaf-alice-1.0.0 + (package + (name "rust-leaf-alice") + (version (? string? ver)) + (source + (origin + (method url-fetch) + (uri (crate-uri "leaf-alice" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (arguments ('quasiquote (#:skip-build? #t))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public rust-leaf-bob-1.0.0 + (package + (name "rust-leaf-bob") + (version (? string? ver)) + (source + (origin + (method url-fetch) + (uri (crate-uri "leaf-bob" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (arguments ('quasiquote (#:skip-build? #t))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public rust-intermediate-2-1.0.0 + (package + (name "rust-intermediate-2") + (version (? string? ver)) + (source + (origin + (method url-fetch) + (uri (crate-uri "intermediate-2" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (arguments + ('quasiquote (#:skip-build? #t + #:cargo-inputs + (("rust-leaf-bob-1.0.0" + ('unquote rust-leaf-bob-1.0.0)))))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public rust-intermediate-1-1.0.0 + (package + (name "rust-intermediate-1") + (version (? string? ver)) + (source + (origin + (method url-fetch) + (uri (crate-uri "intermediate-1" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (arguments + ('quasiquote (#:skip-build? #t + #:cargo-inputs + (("rust-intermediate-2-1.0.0" + ,rust-intermediate-2-1.0.0) + ("rust-leaf-alice-1.0.0" + ('unquote rust-leaf-alice-1.0.0)) + ("rust-leaf-bob-1.0.0" + ('unquote rust-leaf-bob-1.0.0)))))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0)))) + (define-public rust-root-1.0.0 + (package + (name "rust-root") + (version (? string? ver)) + (source + (origin + (method url-fetch) + (uri (crate-uri "root" version)) + (file-name + (string-append name "-" version ".tar.gz")) + (sha256 + (base32 + (? string? hash))))) + (build-system cargo-build-system) + (arguments + ('quasiquote (#:skip-build? + #t #:cargo-inputs + (("rust-intermediate-1-1.0.0" + ('unquote rust-intermediate-1-1.0.0)) + ("rust-intermediate-2-1.0.0" + ('unquote rust-intermediate-2-1.0.0)) + ("rust-leaf-alice-1.0.0" + ('unquote rust-leaf-alice-1.0.0)) + ("rust-leaf-bob-1.0.0" + ('unquote rust-leaf-bob-1.0.0)))))) + (home-page "http://example.com") + (synopsis "summary") + (description "summary") + (license (list license:expat license:asl2.0))))) #t) (x (pk 'fail x #f))))) diff --git a/tests/elpa.scm b/tests/elpa.scm index b70539bda6..a008cf993c 100644 --- a/tests/elpa.scm +++ b/tests/elpa.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2015 Federico Beffa ;;; Copyright © 2020 Ludovic Courtès +;;; Copyright © 2020 Martin Becze ;;; ;;; This file is part of GNU Guix. ;;; @@ -51,7 +52,7 @@ (200 "This is the description.") (200 "fake tarball contents")) (parameterize ((current-http-proxy (%local-url))) - (match (elpa->guix-package pkg 'gnu/http) + (match (elpa->guix-package pkg #:repo 'gnu/http) (('package ('name "emacs-auctex") ('version "11.88.6") -- 2.25.0 --------------F23251DF520B064EFB5C0863--