From: David Thompson <dthompson2@worcester.edu>
To: "Ludovic Courtès" <ludo@gnu.org>
Cc: guix-devel@gnu.org
Subject: Re: [PATCH] import: Add PyPI importer.
Date: Sun, 28 Sep 2014 19:48:52 -0400 [thread overview]
Message-ID: <8738bbmhln.fsf@izanagi.i-did-not-set--mail-host-address--so-tickle-me> (raw)
In-Reply-To: <87eguwentt.fsf@gnu.org>
[-- Attachment #1: Type: text/plain, Size: 2636 bytes --]
Ludovic Courtès <ludo@gnu.org> writes:
> I wonder if there may be shared options between all the importers
> (like an option for import & live build.)
>
> That can still be addressed by exporting an option list from (guix
> scripts import), like (guix scripts build) does, I think.
>
Done. Added an empty options list for now. To avoid circular
dependencies, I had to lazily resolve the importer procedures in
guix/scripts/import.scm.
>> * Makefile.am (MODULES): Add new files and remove 'guix/snix.scm'.
>> * guix/scripts/import.scm (%default-options, %options): Delete.
>> (importers): New variable.
>> (show-help): List importers.
>> (guix-import): Factor out Nix-specific logic. Delegate to correct importer
>> based upon first argument.
>
> Make sure to adjust tests/snix.scm.
>
> Also, it’d be cool to have tests for (guix import pypi), at least for
> the part that is concerned with parsing JSON and producing a package
> object.
>
Done.
>> +(define tarball-url->string-append
>> + (let ((tar.gz-regex (make-regexp "\\.tar\\.gz$"))
>> + (tarball-regex (make-regexp ".*-(.*)\\.tar\\.gz")))
>> + (lambda (url name version)
>> + "Return a `string-append' s-expression used for building a generic form
>> +of URL for the package NAME where VERSION is replaced by a `version'
>> +variable."
>
> This is similar to what snix has, and i think it should be shared (see
> below.)
>
>> +(define (make-pypi-sexp name version source-url home-page synopsis
>> + description license)
>> + "Return the `package' s-expression for a python package with the given NAME,
>
> Namely, what do you think of having importers return directly a
> ‘package’ object? Then there could be a shared ‘package->sexp’
> procedure, that would to the fancy ‘string-append’ thing like above.
>
> And, eventually, we can add an option to do live builds of the generated
> package objects.
>
> That can also be done in the next iteration, though.
>
I think this is best saved for another patch since it involves modifying
the snix code. I agree that it is a good idea.
>> +(define (factorize-uri uri version)
>> + "Factorize URI, a package tarball URI as a string, such that any occurrences
>> +of the string VERSION is replaced by the symbol 'version."
>
> This one from snix is redundant with ‘tarball-url->string-append’ (and
> maybe less sophisticated?).
>
'factorize-uri' is better, so I'm using that now. I created a new
module '(guix import utils)' for commonly used import procedures.
New patch attached.
[-- Attachment #2: 0001-import-Add-PyPI-importer.patch --]
[-- Type: text/x-diff, Size: 64490 bytes --]
From 0696b70fd31630791b5125625a3a15ea0dfdaf1d Mon Sep 17 00:00:00 2001
From: David Thompson <dthompson2@worcester.edu>
Date: Sat, 27 Sep 2014 10:16:23 -0400
Subject: [PATCH] import: Add PyPI importer.
* guix/snix.scm: Delete.
* guix/import/snix.scm: New file.
* guix/import/pypi.scm: New file.
* guix/import/utils.scm: New file.
* guix/scripts/import/nix.scm: New file.
* guix/scripts/import/pypi.scm: New file.
* tests/pypi.scm: New file.
* tests/snix.scm: Import (guix import snix) module.
* Makefile.am (MODULES): Add new files and remove 'guix/snix.scm'.
(SCM_TESTS): Add 'tests/pypi.scm'.
* guix/scripts/import.scm (%default-options, %options): Delete.
(%standard-import-options, importers): New variables.
(show-help): List importers.
(guix-import): Factor out Nix-specific logic. Delegate to correct importer
based upon first argument.
---
Makefile.am | 9 +-
guix/import/pypi.scm | 169 +++++++++++++++
guix/import/snix.scm | 447 ++++++++++++++++++++++++++++++++++++++++
guix/import/utils.scm | 51 +++++
guix/scripts/import.scm | 85 ++++----
guix/scripts/import/nix.scm | 89 ++++++++
guix/scripts/import/pypi.scm | 83 ++++++++
guix/snix.scm | 474 -------------------------------------------
tests/pypi.scm | 102 ++++++++++
tests/snix.scm | 4 +-
10 files changed, 989 insertions(+), 524 deletions(-)
create mode 100644 guix/import/pypi.scm
create mode 100644 guix/import/snix.scm
create mode 100644 guix/import/utils.scm
create mode 100644 guix/scripts/import/nix.scm
create mode 100644 guix/scripts/import/pypi.scm
delete mode 100644 guix/snix.scm
create mode 100644 tests/pypi.scm
diff --git a/Makefile.am b/Makefile.am
index 3c22a77..2925171 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -75,7 +75,9 @@ MODULES = \
guix/build/syscalls.scm \
guix/build/emacs-utils.scm \
guix/packages.scm \
- guix/snix.scm \
+ guix/import/utils.scm \
+ guix/import/snix.scm \
+ guix/import/pypi.scm \
guix/scripts/download.scm \
guix/scripts/build.scm \
guix/scripts/archive.scm \
@@ -89,6 +91,8 @@ MODULES = \
guix/scripts/refresh.scm \
guix/scripts/system.scm \
guix/scripts/lint.scm \
+ guix/scripts/import/nix.scm \
+ guix/scripts/import/pypi.scm \
guix.scm \
$(GNU_SYSTEM_MODULES)
@@ -160,7 +164,8 @@ SCM_TESTS = \
tests/union.scm \
tests/profiles.scm \
tests/syscalls.scm \
- tests/lint.scm
+ tests/lint.scm \
+ tests/pypi.scm
SH_TESTS = \
tests/guix-build.sh \
diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
new file mode 100644
index 0000000..d0e776e
--- /dev/null
+++ b/guix/import/pypi.scm
@@ -0,0 +1,169 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2014 David Thompson <davet@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import pypi)
+ #:use-module (ice-9 binary-ports)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 pretty-print)
+ #:use-module (ice-9 regex)
+ #:use-module (srfi srfi-1)
+ #:use-module (rnrs bytevectors)
+ #:use-module (json)
+ #:use-module (web uri)
+ #:use-module (guix utils)
+ #:use-module (guix import utils)
+ #:use-module (guix base32)
+ #:use-module (guix hash)
+ #:use-module (guix packages)
+ #:use-module (guix licenses)
+ #:use-module (guix build-system python)
+ #:use-module ((guix build download) #:prefix build:)
+ #:use-module (gnu packages python)
+ #:export (pypi->guix-package))
+
+(define (hash-table->alist table)
+ "Return an alist represenation of TABLE."
+ (map (match-lambda
+ ((key . (lst ...))
+ (cons key
+ (map (lambda (x)
+ (if (hash-table? x)
+ (hash-table->alist x)
+ x))
+ lst)))
+ ((key . (? hash-table? table))
+ (cons key (hash-table->alist table)))
+ (pair pair))
+ (hash-map->list cons table)))
+
+(define (flatten lst)
+ "Return a list that recursively concatenates all sub-lists of LIST."
+ (fold-right
+ (match-lambda*
+ (((sub-list ...) memo)
+ (append (flatten sub-list) memo))
+ ((elem memo)
+ (cons elem memo)))
+ '() lst))
+
+(define (join lst delimiter)
+ "Return a list that contains the elements of LST, each separated by
+DELIMETER."
+ (match lst
+ (() '())
+ ((elem)
+ (list elem))
+ ((elem . rest)
+ (cons* elem delimiter (join rest delimiter)))))
+
+(define (assoc-ref* alist key . rest)
+ "Return the value for KEY from ALIST. For each additional key specified,
+recursively apply the procedure to the sub-list."
+ (if (null? rest)
+ (assoc-ref alist key)
+ (apply assoc-ref* (assoc-ref alist key) rest)))
+
+(define string->license
+ (match-lambda
+ ("GNU LGPL" lgpl2.0)
+ ("GPL" gpl3)
+ ((or "BSD" "BSD License") bsd-3)
+ ((or "MIT" "MIT license" "Expat license") expat)
+ ("Public domain" public-domain)
+ (_ #f)))
+
+(define (url-fetch url file-name)
+ "Save the contents of URL to FILE-NAME."
+ (parameterize ((current-output-port (current-error-port)))
+ (build:url-fetch url file-name)))
+
+(define (json-fetch url)
+ "Return an alist representation of the JSON resource URL."
+ (call-with-temporary-output-file
+ (lambda (temp port)
+ (and (url-fetch url temp)
+ (hash-table->alist
+ (call-with-input-file temp json->scm))))))
+
+(define (pypi-fetch name)
+ "Return an alist representation of the PyPI metadata for the package NAME."
+ (json-fetch (string-append "https://pypi.python.org/pypi/" name "/json")))
+
+(define (latest-source-release pypi-package)
+ "Return the latest source release for PYPI-PACKAGE."
+ (let ((releases (assoc-ref* pypi-package "releases"
+ (assoc-ref* pypi-package "info" "version"))))
+ (or (find (lambda (release)
+ (string=? "sdist" (assoc-ref release "packagetype")))
+ releases)
+ (error "No source release found for pypi package: "
+ (assoc-ref* pypi-package "info" "name")
+ (assoc-ref* pypi-package "info" "version")))))
+
+(define (snake-case str)
+ "Return a downcased version of the string STR where dashes are replaced with
+underscores."
+ (string-join (string-split (string-downcase str) #\_) "-"))
+
+(define (guix-hash-url url)
+ "Download the resource at URL and return the hash in nix-base32 format."
+ (call-with-temporary-output-file
+ (lambda (temp port)
+ (and (url-fetch url temp)
+ (bytevector->nix-base32-string
+ (call-with-input-file temp port-sha256))))))
+
+(define (make-pypi-sexp name version source-url home-page synopsis
+ description license)
+ "Return the `package' s-expression for a python package with the given NAME,
+VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
+ `(package
+ (name ,(string-append "python-" (snake-case name)))
+ (version ,version)
+ (source (origin
+ (method url-fetch)
+ (uri (string-append ,@(factorize-uri source-url version)))
+ (sha256
+ (base32
+ ,(guix-hash-url source-url)))))
+ (build-system python-build-system)
+ (inputs
+ `(("python-setuptools" ,python-setuptools)))
+ (home-page ,home-page)
+ (synopsis ,synopsis)
+ (description ,description)
+ (license ,(assoc-ref `((,lgpl2.0 . lgpl2.0)
+ (,gpl3 . gpl3)
+ (,bsd-3 . bsd-3)
+ (,expat . expat)
+ (,public-domain . public-domain))
+ license))))
+
+(define (pypi->guix-package package-name)
+ "Fetch the metadata for PACKAGE-NAME from pypi.python.org, and return the
+`package' s-expression corresponding to that package."
+ (let ((package (pypi-fetch package-name)))
+ (let ((name (assoc-ref* package "info" "name"))
+ (version (assoc-ref* package "info" "version"))
+ (release (assoc-ref (latest-source-release package) "url"))
+ (synopsis (assoc-ref* package "info" "summary"))
+ (description (assoc-ref* package "info" "summary"))
+ (home-page (assoc-ref* package "info" "home_page"))
+ (license (string->license (assoc-ref* package "info" "license"))))
+ (make-pypi-sexp name version release home-page synopsis
+ description license))))
diff --git a/guix/import/snix.scm b/guix/import/snix.scm
new file mode 100644
index 0000000..bcc4d6b
--- /dev/null
+++ b/guix/import/snix.scm
@@ -0,0 +1,447 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2010, 2011, 2012, 2013 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import snix)
+ #:use-module (sxml ssax)
+ #:use-module (ice-9 popen)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 rdelim)
+ #:use-module (ice-9 format)
+ #:use-module (ice-9 regex)
+ #:use-module (ice-9 vlist)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-26)
+ #:use-module (srfi srfi-37)
+ #:use-module (system foreign)
+ #:use-module (rnrs bytevectors)
+ #:use-module (guix utils)
+ #:use-module (guix import utils)
+ #:use-module (guix base32)
+ #:use-module (guix config)
+ #:use-module (guix gnu-maintenance)
+ #:export (open-nixpkgs
+ xml->snix
+ nixpkgs->guix-package))
+
+;;; Commentary:
+;;;
+;;; Converting Nix code to s-expressions, and then to Guix `package'
+;;; declarations, using the XML output of `nix-instantiate'.
+;;;
+;;; Code:
+
+\f
+;;;
+;;; SNix.
+;;;
+
+;; Nix object types visible in the XML output of `nix-instantiate' and
+;; mapping to S-expressions (we map to sexps, not records, so that we
+;; can do pattern matching):
+;;
+;; at (at varpat attrspat)
+;; attr (attribute loc name value)
+;; attrs (attribute-set attributes)
+;; attrspat (attribute-set-pattern patterns)
+;; bool #f|#t
+;; derivation (derivation drv-path out-path attributes)
+;; ellipsis '...
+;; expr (snix loc body ...)
+;; function (function loc at|attrspat|varpat)
+;; int int
+;; list list
+;; null 'null
+;; path string
+;; string string
+;; unevaluated 'unevaluated
+;; varpat (varpat name)
+;;
+;; Initially ATTRIBUTES in `derivation' and `attribute-set' was a promise;
+;; however, handling `repeated' nodes makes it impossible to do anything
+;; lazily because the whole SXML tree has to be traversed to maintain the
+;; list of known derivations.
+
+(define (xml-element->snix elem attributes body derivations)
+ "Return an SNix element corresponding to XML element ELEM."
+
+ (define (loc)
+ (location (assq-ref attributes 'path)
+ (assq-ref attributes 'line)
+ (assq-ref attributes 'column)))
+
+ (case elem
+ ((at)
+ (values `(at ,(car body) ,(cadr body)) derivations))
+ ((attr)
+ (let ((name (assq-ref attributes 'name)))
+ (cond ((null? body)
+ (values `(attribute-pattern ,name) derivations))
+ ((and (pair? body) (null? (cdr body)))
+ (values `(attribute ,(loc) ,name ,(car body))
+ derivations))
+ (else
+ (error "invalid attribute body" name (loc) body)))))
+ ((attrs)
+ (values `(attribute-set ,(reverse body)) derivations))
+ ((attrspat)
+ (values `(attribute-set-pattern ,body) derivations))
+ ((bool)
+ (values (string-ci=? "true" (assq-ref attributes 'value))
+ derivations))
+ ((derivation)
+ (let ((drv-path (assq-ref attributes 'drvPath))
+ (out-path (assq-ref attributes 'outPath)))
+ (if (equal? body '(repeated))
+ (let ((body (vhash-assoc drv-path derivations)))
+ (if (pair? body)
+ (values `(derivation ,drv-path ,out-path ,(cdr body))
+ derivations)
+
+ ;; DRV-PATH hasn't been encountered yet but may be later
+ ;; (see <http://article.gmane.org/gmane.linux.distributions.nixos/5946>.)
+ ;; Return an `unresolved' node.
+ (values `(unresolved
+ ,(lambda (derivations)
+ (let ((body (vhash-assoc drv-path derivations)))
+ (if (pair? body)
+ `(derivation ,drv-path ,out-path
+ ,(cdr body))
+ (error "no previous occurrence of derivation"
+ drv-path)))))
+ derivations)))
+ (values `(derivation ,drv-path ,out-path ,body)
+ (vhash-cons drv-path body derivations)))))
+ ((ellipsis)
+ (values '... derivations))
+ ((expr)
+ (values `(snix ,(loc) ,@body) derivations))
+ ((function)
+ (values `(function ,(loc) ,body) derivations))
+ ((int)
+ (values (string->number (assq-ref attributes 'value))
+ derivations))
+ ((list)
+ (values body derivations))
+ ((null)
+ (values 'null derivations))
+ ((path)
+ (values (assq-ref attributes 'value) derivations))
+ ((repeated)
+ (values 'repeated derivations))
+ ((string)
+ (values (assq-ref attributes 'value) derivations))
+ ((unevaluated)
+ (values 'unevaluated derivations))
+ ((varpat)
+ (values `(varpat ,(assq-ref attributes 'name)) derivations))
+ (else (error "unhandled Nix XML element" elem))))
+
+(define (resolve snix derivations)
+ "Return a new SNix tree where `unresolved' nodes from SNIX have been
+replaced by the result of their application to DERIVATIONS, a vhash."
+ (let loop ((node snix)
+ (seen vlist-null))
+ (if (vhash-assq node seen)
+ (values node seen)
+ (match node
+ (('unresolved proc)
+ (let ((n (proc derivations)))
+ (values n seen)))
+ ((tag body ...)
+ (let ((body+seen (fold (lambda (n body+seen)
+ (call-with-values
+ (lambda ()
+ (loop n (cdr body+seen)))
+ (lambda (n* seen)
+ (cons (cons n* (car body+seen))
+ (vhash-consq n #t seen)))))
+ (cons '() (vhash-consq node #t seen))
+ body)))
+ (values (cons tag (reverse (car body+seen)))
+ (vhash-consq node #t (cdr body+seen)))))
+ (anything
+ (values anything seen))))))
+
+(define xml->snix
+ (let ((parse
+ (ssax:make-parser NEW-LEVEL-SEED
+ (lambda (elem-gi attributes namespaces expected-content
+ seed)
+ (cons '() (cdr seed)))
+
+ FINISH-ELEMENT
+ (lambda (elem-gi attributes namespaces parent-seed
+ seed)
+ (let ((snix (car seed))
+ (derivations (cdr seed)))
+ (let-values (((snix derivations)
+ (xml-element->snix elem-gi
+ attributes
+ snix
+ derivations)))
+ (cons (cons snix (car parent-seed))
+ derivations))))
+
+ CHAR-DATA-HANDLER
+ (lambda (string1 string2 seed)
+ ;; Discard inter-node strings, which are blanks.
+ seed))))
+ (lambda (port)
+ "Return the SNix represention of TREE, an SXML tree as returned by
+parsing the XML output of `nix-instantiate' on Nixpkgs."
+ (match (parse port (cons '() vlist-null))
+ (((snix) . derivations)
+ (resolve snix derivations))))))
+
+(define (attribute-value attribute)
+ "Return the value of ATTRIBUTE."
+ (match attribute
+ (('attribute _ _ value) value)))
+
+(define (derivation-source derivation)
+ "Return the \"src\" attribute of DERIVATION or #f if not found."
+ (match derivation
+ (('derivation _ _ (attributes ...))
+ (find-attribute-by-name "src" attributes))))
+
+(define (derivation-output-path derivation)
+ "Return the output path of DERIVATION."
+ (match derivation
+ (('derivation _ out-path _)
+ out-path)
+ (_ #f)))
+
+(define (source-output-path src)
+ "Return the output path of SRC, the \"src\" attribute of a derivation."
+ (derivation-output-path (attribute-value src)))
+
+(define (source-urls src)
+ "Return the URLs of SRC, the \"src\" attribute of a derivation."
+ (match src
+ (('attribute _ _ ('derivation _ _ (attributes ...)))
+ (match (find-attribute-by-name "urls" attributes)
+ (('attribute _ _ value)
+ value)))
+ (_ #f)))
+
+(define (source-sha256 src)
+ "Return the sha256 of SRC, the \"src\" attribute of a derivation, as a
+bytevector."
+ (match src
+ (('attribute _ _ ('derivation _ _ (attributes ...)))
+ (match (find-attribute-by-name "outputHash" attributes)
+ (('attribute _ _ value)
+ (match value
+ ((= string-length 52)
+ (nix-base32-string->bytevector value))
+ ((= string-length 64)
+ (base16-string->bytevector value))
+ (_
+ (error "unsupported hash format" value))))))
+ (_ #f)))
+
+(define (derivation-source-output-path derivation)
+ "Return the output path of the \"src\" attribute of DERIVATION or #f
+if DERIVATION lacks an \"src\" attribute."
+ (and=> (derivation-source derivation) source-output-path))
+
+(define* (open-nixpkgs nixpkgs #:optional attribute)
+ "Return an input pipe to the XML representation of Nixpkgs. When
+ATTRIBUTE is true, only that attribute is considered."
+ (with-fluids ((%default-port-encoding "UTF-8"))
+ (let ((cross-system (format #f "{
+ config = \"i686-guix-linux-gnu\";
+ libc = \"glibc\";
+ arch = \"guix\";
+ withTLS = true;
+ float = \"hard\";
+ openssl.system = \"linux-generic32\";
+ platform = (import ~a/pkgs/top-level/platforms.nix).sheevaplug;
+}" nixpkgs)))
+ (apply open-pipe* OPEN_READ
+ %nix-instantiate "--strict" "--eval-only" "--xml"
+
+ ;; Pass a dummy `crossSystem' argument so that `buildInputs' and
+ ;; `nativeBuildInputs' are not coalesced.
+ ;; XXX: This is hacky and has other problems.
+ ;"--arg" "crossSystem" cross-system
+
+ `(,@(if attribute
+ `("-A" ,attribute)
+ '())
+ ,nixpkgs)))))
+
+(define (pipe-failed? pipe)
+ "Close pipe and return its status if it failed."
+ (let ((status (close-pipe pipe)))
+ (if (or (status:term-sig status)
+ (not (= (status:exit-val status) 0)))
+ status
+ #f)))
+
+(define (find-attribute-by-name name attributes)
+ "Return attribute NAME in ATTRIBUTES, an attribute set or list of SNix
+attributes, or #f if NAME cannot be found."
+ (find (lambda (a)
+ (match a
+ (('attribute _ (? (cut string=? <> name)) _)
+ a)
+ (_ #f)))
+ (match attributes
+ (('attribute-set (attributes ...))
+ attributes)
+ (_
+ attributes))))
+
+(define (package-source-output-path package)
+ "Return the output path of the \"src\" derivation of PACKAGE."
+ (derivation-source-output-path (attribute-value package)))
+
+\f
+;;;
+;;; Conversion of "Nix expressions" to "Guix expressions".
+;;;
+
+(define (snix-derivation->guix-package derivation)
+ "Return the `package' s-expression corresponding to SNix DERIVATION, a
+Nixpkgs `stdenv.mkDerivation'-style derivation, and the original source
+location of DERIVATION."
+ (match derivation
+ (('derivation _ _ (attributes ...))
+ (let*-values (((full-name loc)
+ (match (find-attribute-by-name "name" attributes)
+ (('attribute loc _ value)
+ (values value loc))
+ (_
+ (values #f #f))))
+ ((name version)
+ (package-name->name+version full-name)))
+ (define (convert-inputs type)
+ ;; Convert the derivation's input from a list of SNix derivations to
+ ;; a list of name/variable pairs.
+ (match (and=> (find-attribute-by-name type attributes)
+ attribute-value)
+ (#f
+ '())
+ ((inputs ...)
+ ;; Inputs can be either derivations or the null value.
+ (filter-map (match-lambda
+ (('derivation _ _ (attributes ...))
+ (let* ((full-name
+ (attribute-value
+ (find-attribute-by-name "name" attributes)))
+ (name (package-name->name+version full-name)))
+ (list name
+ (list 'unquote (string->symbol name)))))
+ ('null #f))
+ inputs))))
+
+ (define (maybe-inputs guix-name inputs)
+ (match inputs
+ (()
+ '())
+ ((inputs ...)
+ (list (list guix-name
+ (list 'quasiquote inputs))))))
+
+ (define (pretty-uri uri version)
+ (if version
+ (match (factorize-uri uri version)
+ ((items ...)
+ `(string-append ,@items))
+ (x x))
+ uri))
+
+ (define (license-variable license)
+ ;; Return the name of the (guix licenses) variable for LICENSE.
+ (match license
+ ("GPLv2+" 'gpl2+)
+ ("GPLv3+" 'gpl3+)
+ ("LGPLv2+" 'lgpl2.1+)
+ ("LGPLv2.1+" 'lgpl2.1+)
+ ("LGPLv3+" 'lgpl3+)
+ (_ license)))
+
+ (let* ((source (find-attribute-by-name "src" attributes))
+ (urls (source-urls source))
+ (sha256 (source-sha256 source))
+ (meta (and=> (find-attribute-by-name "meta" attributes)
+ attribute-value)))
+ (values
+ `(package
+ (name ,name)
+ (version ,version)
+ (source (origin
+ (method url-fetch)
+ (uri ,(pretty-uri (car urls) version))
+ (sha256
+ (base32
+ ,(bytevector->nix-base32-string sha256)))))
+ (build-system gnu-build-system)
+
+ ;; When doing a native Nixpkgs build, `buildInputs' is empty and
+ ;; everything is in `nativeBuildInputs'. So we can't distinguish
+ ;; between both, here.
+ ;;
+ ;; Note that `nativeBuildInputs' was renamed from
+ ;; `buildNativeInputs' in Nixpkgs sometime around March 2013.
+ ,@(maybe-inputs 'inputs
+ (convert-inputs "nativeBuildInputs"))
+ ,@(maybe-inputs 'propagated-inputs
+ (convert-inputs "propagatedNativeBuildInputs"))
+
+ (home-page ,(and=> (find-attribute-by-name "homepage" meta)
+ attribute-value))
+ (synopsis
+ ;; For GNU packages, prefer the official synopsis.
+ ,(or (false-if-exception
+ (and=> (find (lambda (gnu-package)
+ (equal? (gnu-package-name gnu-package)
+ name))
+ (official-gnu-packages))
+ gnu-package-doc-summary))
+ (and=> (find-attribute-by-name "description" meta)
+ attribute-value)))
+ (description
+ ;; Likewise, prefer the official description of GNU packages.
+ ,(or (false-if-exception
+ (and=> (find (lambda (gnu-package)
+ (equal? (gnu-package-name gnu-package)
+ name))
+ (official-gnu-packages))
+ gnu-package-doc-description))
+ (and=> (find-attribute-by-name "longDescription" meta)
+ attribute-value)))
+ (license ,(and=> (find-attribute-by-name "license" meta)
+ (compose license-variable attribute-value))))
+ loc))))))
+
+(define (nixpkgs->guix-package nixpkgs attribute)
+ "Evaluate ATTRIBUTE in NIXPKGS, the file name of a Nixpkgs checkout,
+and return the `package' s-expression corresponding to that package."
+ (let ((port (open-nixpkgs nixpkgs attribute)))
+ (match (xml->snix port)
+ (('snix loc (and drv ('derivation _ ...)))
+ (and (not (pipe-failed? port))
+ (snix-derivation->guix-package drv)))
+ (_
+ (not (pipe-failed? port))))))
+
+;;; snix.scm ends here
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
new file mode 100644
index 0000000..062cfc5
--- /dev/null
+++ b/guix/import/utils.scm
@@ -0,0 +1,51 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2012, 2013 Ludovic Courtès <ludo@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import utils)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 regex)
+ #:use-module (srfi srfi-1)
+ #:export (factorize-uri))
+
+(define (factorize-uri uri version)
+ "Factorize URI, a package tarball URI as a string, such that any occurrences
+of the string VERSION is replaced by the symbol 'version."
+ (let ((version-rx (make-regexp (regexp-quote version))))
+ (match (regexp-exec version-rx uri)
+ (#f
+ uri)
+ (_
+ (let ((indices (fold-matches version-rx uri
+ '((0))
+ (lambda (m result)
+ (match result
+ (((start) rest ...)
+ `((,(match:end m))
+ (,start . ,(match:start m))
+ ,@rest)))))))
+ (fold (lambda (index result)
+ (match index
+ ((start)
+ (cons (substring uri start)
+ result))
+ ((start . end)
+ (cons* (substring uri start end)
+ 'version
+ result))))
+ '()
+ indices))))))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 6f75017..e9576ba 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -1,5 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2012, 2013 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014 David Thompson <davet@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -18,15 +19,16 @@
(define-module (guix scripts import)
#:use-module (guix ui)
- #:use-module (guix snix)
#:use-module (guix utils)
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-11)
#:use-module (srfi srfi-26)
#:use-module (srfi srfi-37)
+ #:use-module (ice-9 format)
#:use-module (ice-9 match)
#:use-module (ice-9 pretty-print)
- #:export (guix-import))
+ #:export (%standard-import-options
+ guix-import))
\f
;;;
@@ -61,15 +63,30 @@ rather than \\n."
\f
;;;
-;;; Command-line options.
+;;; Command line options.
;;;
-(define %default-options
- '())
+(define %standard-import-options '())
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define importers '("nix" "pypi"))
+
+(define (resolve-importer name)
+ (let ((module (resolve-interface
+ `(guix scripts import ,(string->symbol name))))
+ (proc (string->symbol (string-append "guix-import-" name))))
+ (module-ref module proc)))
(define (show-help)
- (display (_ "Usage: guix import NIXPKGS ATTRIBUTE
-Import and convert the Nix expression ATTRIBUTE of NIXPKGS.\n"))
+ (display (_ "Usage: guix import IMPORTER ARGS ...
+Run IMPORTER with ARGS.\n"))
+ (newline)
+ (display (_ "IMPORTER must be one of the importers listed below:\n"))
+ (format #t "~{ ~a~%~}" importers)
(display (_ "
-h, --help display this help and exit"))
(display (_ "
@@ -77,43 +94,19 @@ Import and convert the Nix expression ATTRIBUTE of NIXPKGS.\n"))
(newline)
(show-bug-report-information))
-(define %options
- ;; Specification of the command-line options.
- (list (option '(#\h "help") #f #f
- (lambda args
- (show-help)
- (exit 0)))
- (option '(#\V "version") #f #f
- (lambda args
- (show-version-and-exit "guix import")))))
-
-\f
-;;;
-;;; Entry point.
-;;;
-
(define (guix-import . args)
- (define (parse-options)
- ;; Return the alist of option values.
- (args-fold* args %options
- (lambda (opt name arg result)
- (leave (_ "~A: unrecognized option~%") name))
- (lambda (arg result)
- (alist-cons 'argument arg result))
- %default-options))
-
- (let* ((opts (parse-options))
- (args (filter-map (match-lambda
- (('argument . value)
- value)
- (_ #f))
- (reverse opts))))
- (match args
- ((nixpkgs attribute)
- (let-values (((expr loc)
- (nixpkgs->guix-package nixpkgs attribute)))
- (format #t ";; converted from ~a:~a~%~%"
- (location-file loc) (location-line loc))
- (pretty-print expr (newline-rewriting-port (current-output-port)))))
- (_
- (leave (_ "wrong number of arguments~%"))))))
+ (match args
+ (()
+ (format (current-error-port)
+ (_ "guix import: missing importer name~%")))
+ ((or ("-h") ("--help"))
+ (show-help)
+ (exit 0))
+ (("--version")
+ (show-version-and-exit "guix import"))
+ ((importer args ...)
+ (if (member importer importers)
+ (let ((expr (apply (resolve-importer importer) args)))
+ (pretty-print expr (newline-rewriting-port (current-output-port))))
+ (format (current-error-port)
+ (_ "guix import: invalid importer~%"))))))
diff --git a/guix/scripts/import/nix.scm b/guix/scripts/import/nix.scm
new file mode 100644
index 0000000..2dc2677
--- /dev/null
+++ b/guix/scripts/import/nix.scm
@@ -0,0 +1,89 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2012, 2013 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014 David Thompson <davet@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import nix)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix import snix)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (ice-9 match)
+ #:export (guix-import-nix))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (_ "Usage: guix import nix NIXPKGS ATTRIBUTE
+Import and convert the Nix expression ATTRIBUTE of NIXPKGS.\n"))
+ (display (_ "
+ -h, --help display this help and exit"))
+ (display (_ "
+ -V, --version display version information and exit"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import nix")))
+ %standard-import-options))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-nix . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (args-fold* args %options
+ (lambda (opt name arg result)
+ (leave (_ "~A: unrecognized option~%") name))
+ (lambda (arg result)
+ (alist-cons 'argument arg result))
+ %default-options))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (match args
+ ((nixpkgs attribute)
+ (let-values (((expr loc)
+ (nixpkgs->guix-package nixpkgs attribute)))
+ (format #t ";; converted from ~a:~a~%~%"
+ (location-file loc) (location-line loc))
+ expr))
+ (_
+ (leave (_ "wrong number of arguments~%"))))))
diff --git a/guix/scripts/import/pypi.scm b/guix/scripts/import/pypi.scm
new file mode 100644
index 0000000..0aaa23a
--- /dev/null
+++ b/guix/scripts/import/pypi.scm
@@ -0,0 +1,83 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2014 David Thompson <davet@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import pypi)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix import pypi)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:export (guix-import-pypi))
+
+\f
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (_ "Usage: guix import pypi PACKAGE-NAME
+Import and convert the PyPI package for PACKAGE-NAME.\n"))
+ (display (_ "
+ -h, --help display this help and exit"))
+ (display (_ "
+ -V, --version display version information and exit"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import pypi")))
+ %standard-import-options))
+
+\f
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-pypi . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (args-fold* args %options
+ (lambda (opt name arg result)
+ (leave (_ "~A: unrecognized option~%") name))
+ (lambda (arg result)
+ (alist-cons 'argument arg result))
+ %default-options))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (match args
+ ((package-name)
+ (pypi->guix-package package-name)))))
diff --git a/guix/snix.scm b/guix/snix.scm
deleted file mode 100644
index a77433b..0000000
--- a/guix/snix.scm
+++ /dev/null
@@ -1,474 +0,0 @@
-;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2010, 2011, 2012, 2013 Ludovic Courtès <ludo@gnu.org>
-;;;
-;;; This file is part of GNU Guix.
-;;;
-;;; GNU Guix is free software; you can redistribute it and/or modify it
-;;; under the terms of the GNU General Public License as published by
-;;; the Free Software Foundation; either version 3 of the License, or (at
-;;; your option) any later version.
-;;;
-;;; GNU Guix is distributed in the hope that it will be useful, but
-;;; WITHOUT ANY WARRANTY; without even the implied warranty of
-;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-;;; GNU General Public License for more details.
-;;;
-;;; You should have received a copy of the GNU General Public License
-;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
-
-(define-module (guix snix)
- #:use-module (sxml ssax)
- #:use-module (ice-9 popen)
- #:use-module (ice-9 match)
- #:use-module (ice-9 rdelim)
- #:use-module (ice-9 format)
- #:use-module (ice-9 regex)
- #:use-module (ice-9 vlist)
- #:use-module (srfi srfi-1)
- #:use-module (srfi srfi-9)
- #:use-module (srfi srfi-11)
- #:use-module (srfi srfi-26)
- #:use-module (srfi srfi-37)
- #:use-module (system foreign)
- #:use-module (rnrs bytevectors)
- #:use-module (guix utils)
- #:use-module (guix base32)
- #:use-module (guix config)
- #:use-module (guix gnu-maintenance)
- #:export (open-nixpkgs
- xml->snix
- nixpkgs->guix-package))
-
-;;; Commentary:
-;;;
-;;; Converting Nix code to s-expressions, and then to Guix `package'
-;;; declarations, using the XML output of `nix-instantiate'.
-;;;
-;;; Code:
-
-\f
-;;;
-;;; SNix.
-;;;
-
-;; Nix object types visible in the XML output of `nix-instantiate' and
-;; mapping to S-expressions (we map to sexps, not records, so that we
-;; can do pattern matching):
-;;
-;; at (at varpat attrspat)
-;; attr (attribute loc name value)
-;; attrs (attribute-set attributes)
-;; attrspat (attribute-set-pattern patterns)
-;; bool #f|#t
-;; derivation (derivation drv-path out-path attributes)
-;; ellipsis '...
-;; expr (snix loc body ...)
-;; function (function loc at|attrspat|varpat)
-;; int int
-;; list list
-;; null 'null
-;; path string
-;; string string
-;; unevaluated 'unevaluated
-;; varpat (varpat name)
-;;
-;; Initially ATTRIBUTES in `derivation' and `attribute-set' was a promise;
-;; however, handling `repeated' nodes makes it impossible to do anything
-;; lazily because the whole SXML tree has to be traversed to maintain the
-;; list of known derivations.
-
-(define (xml-element->snix elem attributes body derivations)
- "Return an SNix element corresponding to XML element ELEM."
-
- (define (loc)
- (location (assq-ref attributes 'path)
- (assq-ref attributes 'line)
- (assq-ref attributes 'column)))
-
- (case elem
- ((at)
- (values `(at ,(car body) ,(cadr body)) derivations))
- ((attr)
- (let ((name (assq-ref attributes 'name)))
- (cond ((null? body)
- (values `(attribute-pattern ,name) derivations))
- ((and (pair? body) (null? (cdr body)))
- (values `(attribute ,(loc) ,name ,(car body))
- derivations))
- (else
- (error "invalid attribute body" name (loc) body)))))
- ((attrs)
- (values `(attribute-set ,(reverse body)) derivations))
- ((attrspat)
- (values `(attribute-set-pattern ,body) derivations))
- ((bool)
- (values (string-ci=? "true" (assq-ref attributes 'value))
- derivations))
- ((derivation)
- (let ((drv-path (assq-ref attributes 'drvPath))
- (out-path (assq-ref attributes 'outPath)))
- (if (equal? body '(repeated))
- (let ((body (vhash-assoc drv-path derivations)))
- (if (pair? body)
- (values `(derivation ,drv-path ,out-path ,(cdr body))
- derivations)
-
- ;; DRV-PATH hasn't been encountered yet but may be later
- ;; (see <http://article.gmane.org/gmane.linux.distributions.nixos/5946>.)
- ;; Return an `unresolved' node.
- (values `(unresolved
- ,(lambda (derivations)
- (let ((body (vhash-assoc drv-path derivations)))
- (if (pair? body)
- `(derivation ,drv-path ,out-path
- ,(cdr body))
- (error "no previous occurrence of derivation"
- drv-path)))))
- derivations)))
- (values `(derivation ,drv-path ,out-path ,body)
- (vhash-cons drv-path body derivations)))))
- ((ellipsis)
- (values '... derivations))
- ((expr)
- (values `(snix ,(loc) ,@body) derivations))
- ((function)
- (values `(function ,(loc) ,body) derivations))
- ((int)
- (values (string->number (assq-ref attributes 'value))
- derivations))
- ((list)
- (values body derivations))
- ((null)
- (values 'null derivations))
- ((path)
- (values (assq-ref attributes 'value) derivations))
- ((repeated)
- (values 'repeated derivations))
- ((string)
- (values (assq-ref attributes 'value) derivations))
- ((unevaluated)
- (values 'unevaluated derivations))
- ((varpat)
- (values `(varpat ,(assq-ref attributes 'name)) derivations))
- (else (error "unhandled Nix XML element" elem))))
-
-(define (resolve snix derivations)
- "Return a new SNix tree where `unresolved' nodes from SNIX have been
-replaced by the result of their application to DERIVATIONS, a vhash."
- (let loop ((node snix)
- (seen vlist-null))
- (if (vhash-assq node seen)
- (values node seen)
- (match node
- (('unresolved proc)
- (let ((n (proc derivations)))
- (values n seen)))
- ((tag body ...)
- (let ((body+seen (fold (lambda (n body+seen)
- (call-with-values
- (lambda ()
- (loop n (cdr body+seen)))
- (lambda (n* seen)
- (cons (cons n* (car body+seen))
- (vhash-consq n #t seen)))))
- (cons '() (vhash-consq node #t seen))
- body)))
- (values (cons tag (reverse (car body+seen)))
- (vhash-consq node #t (cdr body+seen)))))
- (anything
- (values anything seen))))))
-
-(define xml->snix
- (let ((parse
- (ssax:make-parser NEW-LEVEL-SEED
- (lambda (elem-gi attributes namespaces expected-content
- seed)
- (cons '() (cdr seed)))
-
- FINISH-ELEMENT
- (lambda (elem-gi attributes namespaces parent-seed
- seed)
- (let ((snix (car seed))
- (derivations (cdr seed)))
- (let-values (((snix derivations)
- (xml-element->snix elem-gi
- attributes
- snix
- derivations)))
- (cons (cons snix (car parent-seed))
- derivations))))
-
- CHAR-DATA-HANDLER
- (lambda (string1 string2 seed)
- ;; Discard inter-node strings, which are blanks.
- seed))))
- (lambda (port)
- "Return the SNix represention of TREE, an SXML tree as returned by
-parsing the XML output of `nix-instantiate' on Nixpkgs."
- (match (parse port (cons '() vlist-null))
- (((snix) . derivations)
- (resolve snix derivations))))))
-
-(define (attribute-value attribute)
- "Return the value of ATTRIBUTE."
- (match attribute
- (('attribute _ _ value) value)))
-
-(define (derivation-source derivation)
- "Return the \"src\" attribute of DERIVATION or #f if not found."
- (match derivation
- (('derivation _ _ (attributes ...))
- (find-attribute-by-name "src" attributes))))
-
-(define (derivation-output-path derivation)
- "Return the output path of DERIVATION."
- (match derivation
- (('derivation _ out-path _)
- out-path)
- (_ #f)))
-
-(define (source-output-path src)
- "Return the output path of SRC, the \"src\" attribute of a derivation."
- (derivation-output-path (attribute-value src)))
-
-(define (source-urls src)
- "Return the URLs of SRC, the \"src\" attribute of a derivation."
- (match src
- (('attribute _ _ ('derivation _ _ (attributes ...)))
- (match (find-attribute-by-name "urls" attributes)
- (('attribute _ _ value)
- value)))
- (_ #f)))
-
-(define (source-sha256 src)
- "Return the sha256 of SRC, the \"src\" attribute of a derivation, as a
-bytevector."
- (match src
- (('attribute _ _ ('derivation _ _ (attributes ...)))
- (match (find-attribute-by-name "outputHash" attributes)
- (('attribute _ _ value)
- (match value
- ((= string-length 52)
- (nix-base32-string->bytevector value))
- ((= string-length 64)
- (base16-string->bytevector value))
- (_
- (error "unsupported hash format" value))))))
- (_ #f)))
-
-(define (derivation-source-output-path derivation)
- "Return the output path of the \"src\" attribute of DERIVATION or #f
-if DERIVATION lacks an \"src\" attribute."
- (and=> (derivation-source derivation) source-output-path))
-
-(define* (open-nixpkgs nixpkgs #:optional attribute)
- "Return an input pipe to the XML representation of Nixpkgs. When
-ATTRIBUTE is true, only that attribute is considered."
- (with-fluids ((%default-port-encoding "UTF-8"))
- (let ((cross-system (format #f "{
- config = \"i686-guix-linux-gnu\";
- libc = \"glibc\";
- arch = \"guix\";
- withTLS = true;
- float = \"hard\";
- openssl.system = \"linux-generic32\";
- platform = (import ~a/pkgs/top-level/platforms.nix).sheevaplug;
-}" nixpkgs)))
- (apply open-pipe* OPEN_READ
- %nix-instantiate "--strict" "--eval-only" "--xml"
-
- ;; Pass a dummy `crossSystem' argument so that `buildInputs' and
- ;; `nativeBuildInputs' are not coalesced.
- ;; XXX: This is hacky and has other problems.
- ;"--arg" "crossSystem" cross-system
-
- `(,@(if attribute
- `("-A" ,attribute)
- '())
- ,nixpkgs)))))
-
-(define (pipe-failed? pipe)
- "Close pipe and return its status if it failed."
- (let ((status (close-pipe pipe)))
- (if (or (status:term-sig status)
- (not (= (status:exit-val status) 0)))
- status
- #f)))
-
-(define (find-attribute-by-name name attributes)
- "Return attribute NAME in ATTRIBUTES, an attribute set or list of SNix
-attributes, or #f if NAME cannot be found."
- (find (lambda (a)
- (match a
- (('attribute _ (? (cut string=? <> name)) _)
- a)
- (_ #f)))
- (match attributes
- (('attribute-set (attributes ...))
- attributes)
- (_
- attributes))))
-
-(define (package-source-output-path package)
- "Return the output path of the \"src\" derivation of PACKAGE."
- (derivation-source-output-path (attribute-value package)))
-
-\f
-;;;
-;;; Conversion of "Nix expressions" to "Guix expressions".
-;;;
-
-(define (factorize-uri uri version)
- "Factorize URI, a package tarball URI as a string, such that any occurrences
-of the string VERSION is replaced by the symbol 'version."
- (let ((version-rx (make-regexp (regexp-quote version))))
- (match (regexp-exec version-rx uri)
- (#f
- uri)
- (_
- (let ((indices (fold-matches version-rx uri
- '((0))
- (lambda (m result)
- (match result
- (((start) rest ...)
- `((,(match:end m))
- (,start . ,(match:start m))
- ,@rest)))))))
- (fold (lambda (index result)
- (match index
- ((start)
- (cons (substring uri start)
- result))
- ((start . end)
- (cons* (substring uri start end)
- 'version
- result))))
- '()
- indices))))))
-
-(define (snix-derivation->guix-package derivation)
- "Return the `package' s-expression corresponding to SNix DERIVATION, a
-Nixpkgs `stdenv.mkDerivation'-style derivation, and the original source
-location of DERIVATION."
- (match derivation
- (('derivation _ _ (attributes ...))
- (let*-values (((full-name loc)
- (match (find-attribute-by-name "name" attributes)
- (('attribute loc _ value)
- (values value loc))
- (_
- (values #f #f))))
- ((name version)
- (package-name->name+version full-name)))
- (define (convert-inputs type)
- ;; Convert the derivation's input from a list of SNix derivations to
- ;; a list of name/variable pairs.
- (match (and=> (find-attribute-by-name type attributes)
- attribute-value)
- (#f
- '())
- ((inputs ...)
- ;; Inputs can be either derivations or the null value.
- (filter-map (match-lambda
- (('derivation _ _ (attributes ...))
- (let* ((full-name
- (attribute-value
- (find-attribute-by-name "name" attributes)))
- (name (package-name->name+version full-name)))
- (list name
- (list 'unquote (string->symbol name)))))
- ('null #f))
- inputs))))
-
- (define (maybe-inputs guix-name inputs)
- (match inputs
- (()
- '())
- ((inputs ...)
- (list (list guix-name
- (list 'quasiquote inputs))))))
-
- (define (pretty-uri uri version)
- (if version
- (match (factorize-uri uri version)
- ((items ...)
- `(string-append ,@items))
- (x x))
- uri))
-
- (define (license-variable license)
- ;; Return the name of the (guix licenses) variable for LICENSE.
- (match license
- ("GPLv2+" 'gpl2+)
- ("GPLv3+" 'gpl3+)
- ("LGPLv2+" 'lgpl2.1+)
- ("LGPLv2.1+" 'lgpl2.1+)
- ("LGPLv3+" 'lgpl3+)
- (_ license)))
-
- (let* ((source (find-attribute-by-name "src" attributes))
- (urls (source-urls source))
- (sha256 (source-sha256 source))
- (meta (and=> (find-attribute-by-name "meta" attributes)
- attribute-value)))
- (values
- `(package
- (name ,name)
- (version ,version)
- (source (origin
- (method url-fetch)
- (uri ,(pretty-uri (car urls) version))
- (sha256
- (base32
- ,(bytevector->nix-base32-string sha256)))))
- (build-system gnu-build-system)
-
- ;; When doing a native Nixpkgs build, `buildInputs' is empty and
- ;; everything is in `nativeBuildInputs'. So we can't distinguish
- ;; between both, here.
- ;;
- ;; Note that `nativeBuildInputs' was renamed from
- ;; `buildNativeInputs' in Nixpkgs sometime around March 2013.
- ,@(maybe-inputs 'inputs
- (convert-inputs "nativeBuildInputs"))
- ,@(maybe-inputs 'propagated-inputs
- (convert-inputs "propagatedNativeBuildInputs"))
-
- (home-page ,(and=> (find-attribute-by-name "homepage" meta)
- attribute-value))
- (synopsis
- ;; For GNU packages, prefer the official synopsis.
- ,(or (false-if-exception
- (and=> (find (lambda (gnu-package)
- (equal? (gnu-package-name gnu-package)
- name))
- (official-gnu-packages))
- gnu-package-doc-summary))
- (and=> (find-attribute-by-name "description" meta)
- attribute-value)))
- (description
- ;; Likewise, prefer the official description of GNU packages.
- ,(or (false-if-exception
- (and=> (find (lambda (gnu-package)
- (equal? (gnu-package-name gnu-package)
- name))
- (official-gnu-packages))
- gnu-package-doc-description))
- (and=> (find-attribute-by-name "longDescription" meta)
- attribute-value)))
- (license ,(and=> (find-attribute-by-name "license" meta)
- (compose license-variable attribute-value))))
- loc))))))
-
-(define (nixpkgs->guix-package nixpkgs attribute)
- "Evaluate ATTRIBUTE in NIXPKGS, the file name of a Nixpkgs checkout,
-and return the `package' s-expression corresponding to that package."
- (let ((port (open-nixpkgs nixpkgs attribute)))
- (match (xml->snix port)
- (('snix loc (and drv ('derivation _ ...)))
- (and (not (pipe-failed? port))
- (snix-derivation->guix-package drv)))
- (_
- (not (pipe-failed? port))))))
-
-;;; snix.scm ends here
diff --git a/tests/pypi.scm b/tests/pypi.scm
new file mode 100644
index 0000000..da4d03f
--- /dev/null
+++ b/tests/pypi.scm
@@ -0,0 +1,102 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2014 David Thompson <davet@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (test-pypi)
+ #:use-module (guix import pypi)
+ #:use-module (guix base32)
+ #:use-module (guix hash)
+ #:use-module (srfi srfi-64)
+ #:use-module (ice-9 match))
+
+(define-syntax-rule (stub (module proc replacement) body ...)
+ (let* ((m (resolve-module 'module))
+ (original (module-ref m 'proc)))
+ (dynamic-wind
+ (lambda () (module-set! m 'proc replacement))
+ (lambda () body ...)
+ (lambda () (module-set! m 'proc original)))))
+
+(define test-json
+ "{
+ \"info\": {
+ \"version\": \"1.0.0\",
+ \"name\": \"foo\",
+ \"license\": \"GNU LGPL\",
+ \"summary\": \"summary\",
+ \"home_page\": \"http://example.com\",
+ },
+ \"releases\": {
+ \"1.0.0\": [
+ {
+ \"url\": \"https://example.com/foo-1.0.0.egg\",
+ \"packagetype\": \"bdist_egg\",
+ }, {
+ \"url\": \"https://example.com/foo-1.0.0.tar.gz\",
+ \"packagetype\": \"sdist\",
+ }
+ ]
+ }
+}")
+
+(define test-source
+ "foobar")
+
+(test-begin "pypi")
+
+(test-assert "pypi->guix-package"
+ ;; Replace network resources with sample data.
+ (stub ((guix import pypi) url-fetch
+ (lambda (url file-name)
+ (with-output-to-file file-name
+ (lambda ()
+ (display
+ (match url
+ ("https://pypi.python.org/pypi/foo/json"
+ test-json)
+ ("https://example.com/foo-1.0.0.tar.gz"
+ test-source)
+ (_ (error "Unexpected URL: " url))))))))
+ (match (pypi->guix-package "foo")
+ (('package
+ ('name "python-foo")
+ ('version "1.0.0")
+ ('source ('origin
+ ('method 'url-fetch)
+ ('uri ('string-append "https://example.com/foo-"
+ 'version ".tar.gz"))
+ ('sha256
+ ('base32
+ (? string? hash)))))
+ ('build-system 'python-build-system)
+ ('inputs
+ ('quasiquote
+ (("python-setuptools" ('unquote 'python-setuptools)))))
+ ('home-page "http://example.com")
+ ('synopsis "summary")
+ ('description "summary")
+ ('license 'lgpl2.0))
+ (string=? (bytevector->nix-base32-string
+ (call-with-input-string test-source port-sha256))
+ hash))
+ (x
+ (pk 'fail x #f)))))
+
+(test-end "pypi")
+
+\f
+(exit (= (test-runner-fail-count (test-runner-current)) 0))
diff --git a/tests/snix.scm b/tests/snix.scm
index 9d692e9..2318780 100644
--- a/tests/snix.scm
+++ b/tests/snix.scm
@@ -17,14 +17,14 @@
;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
(define-module (test-snix)
- #:use-module (guix snix)
+ #:use-module (guix import snix)
#:use-module ((guix utils) #:select (%nixpkgs-directory))
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-64)
#:use-module (ice-9 match))
(define factorize-uri
- (@@ (guix snix) factorize-uri))
+ (@@ (guix import snix) factorize-uri))
(define-syntax-rule (every? proc lists ...)
(not (not (every proc lists ...))))
--
2.1.0
[-- Attachment #3: Type: text/plain, Size: 136 bytes --]
--
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate
next prev parent reply other threads:[~2014-09-28 23:49 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-09-27 21:15 [PATCH] import: Add PyPI importer David Thompson
2014-09-27 21:19 ` David Thompson
2014-09-27 21:35 ` Ludovic Courtès
2014-09-29 11:23 ` David Thompson
2014-09-29 12:24 ` Ludovic Courtès
2014-09-29 23:31 ` David Thompson
2014-09-30 5:49 ` Alex Kost
2014-09-30 7:13 ` Ludovic Courtès
2014-09-30 11:55 ` David Thompson
2014-09-27 21:49 ` Ludovic Courtès
2014-09-28 23:48 ` David Thompson [this message]
2014-09-29 7:04 ` Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8738bbmhln.fsf@izanagi.i-did-not-set--mail-host-address--so-tickle-me \
--to=dthompson2@worcester.edu \
--cc=guix-devel@gnu.org \
--cc=ludo@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).