Re: [PATCH 1/4] gnu: Add python-avro.

[Marius Bakke <mbakke@fastmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2339 bytes --]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 25.09.2016 um 23:17 schrieb Marius Bakke:
>> +;;; Avro uses a single source repository for all implementations. The individual
>> +;;; released versions often have missing or incomplete test data, so we define
>> +;;; the common source here for use in all avro packages.
>> +(define (avro-version) "1.8.1")
>> +(define (avro-source version)
>
> With "Avro uses a single source repository for all implementations " you
> mean for both Python 2 and Python 3? Well, this is the common case :-)
> Any other packages do have this and we never use a separate definition
> for source or version. Please have a look at e.g. python-graphql-relay
> (picked as example at random).

This is a rather special case. I've updated the comment to make it more
clear. The Avro repository contains one folder for each language
implementation, and py2 and py3 are developed separately:

https://www-eu.apache.org/dist/avro/avro-1.8.1/

The source in this case contains all those subfolders. I'm only building
the two different Python implementations for now, but expect the other
interfaces to be able to use the same source when packaged.

There are release tarballs for the subdirectories, but at least the py3
version was missing tests, which was also the case on PyPi:

https://pypi.python.org/pypi/avro
https://pypi.python.org/pypi/avro-python3

> Am 25.09.2016 um 23:17 schrieb Marius Bakke:
>> +     `(#:configure-flags '("--single-version-externally-managed" "--root=/")
>
> Same here, please add a comment,
>
> BTW: What should --root=/ be goof for? AFAIK this is the default?!

When using --single-version-externally-managed without --root, the
install phase fails with:

error: You must specify --record or --root when building system packages

As to why it is needed, there is another side effect in
python-build-system that is not mentioned in the bug report, but the
same fix works. When not specified, "bin" inputs may end up in the "bin"
output of the package! E.g. for pbcommand, $out/bin contains
"jsonschema" from the python-jsonschema package, whereas
sphinx-bootstrap-theme has "easy_install" in $out/bin and
sphinx-argparse/bin contains various executables from python-sphinx.

I've updated the patch series addressing these and your other comments.

Thanks for the feedback!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-python-avro.patch --]
[-- Type: text/x-patch, Size: 3202 bytes --]

From 9bb2dac10478efc420a97ac70892be0fe0356080 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sun, 25 Sep 2016 20:23:39 +0100
Subject: [PATCH 1/4] gnu: Add python-avro.

* gnu/packages/serialization.scm (avro-version): New variable (private).
(avro-source): New variable (private).
(python-avro, python2-avro): New variables.
---
 gnu/packages/serialization.scm | 50 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/gnu/packages/serialization.scm b/gnu/packages/serialization.scm
index 4a3278f..e80f5fe 100644
--- a/gnu/packages/serialization.scm
+++ b/gnu/packages/serialization.scm
@@ -25,6 +25,7 @@
   #:use-module (guix download)
   #:use-module (guix build-system cmake)
   #:use-module (guix build-system gnu)
+  #:use-module (guix build-system python)
   #:use-module (gnu packages)
   #:use-module (gnu packages autotools)
   #:use-module (gnu packages boost)
@@ -200,3 +201,52 @@ it a convenient format to store user input files.")
      "Cap'n Proto is a very fast data interchange format and capability-based
 RPC system.  Think JSON, except binary.  Or think Protocol Buffers, except faster.")
     (license license:expat)))
+
+;;; Avro uses a single source repository for all language implementations. The
+;;; individual released versions some times have missing or incomplete test data,
+;;; so we use the full source tarball for all Avro packages.
+(define (avro-version) "1.8.1")
+(define (avro-source version)
+  (origin
+    (method url-fetch)
+    (uri (string-append "mirror://apache/avro/avro-" version
+                        "/avro-src-" version ".tar.gz"))
+    (sha256 (base32 "0bplj4qmh7d6p987qd6a13g59awrc5cbx25n5brcrwq8yjik21av"))
+    (modules '((guix build utils)))
+    (snippet
+     '(begin
+        ;; Drop bundled dependencies.
+        (delete-file-recursively "lang/py/lib/simplejson")
+        #t))))
+
+(define-public python-avro
+  (package
+    (name "python-avro")
+    (version (avro-version))
+    (source (avro-source version))
+    (build-system python-build-system)
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'enter-source
+           (lambda _ (chdir "lang/py3") #t)))))
+    (propagated-inputs
+     `(("python-simplejson" ,python-simplejson)))
+    (home-page "https://avro.apache.org/")
+    (synopsis "Python implementation of the Avro data serialization system")
+    (description
+     "Avro is a data serialization system and RPC framework with rich data
+structures.  This package provides the Python interface.")
+    (license license:asl2.0)
+    (properties `((python2-variant . ,(delay python2-avro))))))
+
+(define-public python2-avro
+  (let ((base (package-with-python2 (strip-python2-variant python-avro))))
+    (package (inherit base)
+             (arguments
+              `(#:tests? #f ; TODO: Enable when Apache Ivy is packaged.
+                #:python ,python-2 ; Needed when overriding inherited args.
+                #:phases
+                (modify-phases %standard-phases
+                  (add-after 'unpack 'enter-source
+                    (lambda _ (chdir "lang/py") #t))))))))
-- 
2.10.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-gnu-Add-python-sphinx-bootstrap-theme.patch --]
[-- Type: text/x-patch, Size: 2257 bytes --]

From f460c88f853911618e77f869f31f4465923dbdd8 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sun, 25 Sep 2016 20:40:52 +0100
Subject: [PATCH 2/4] gnu: Add python-sphinx-bootstrap-theme.

* gnu/packages/python.scm (python-sphinx-bootstrap-theme,
  python2-sphinx-bootstrap-theme): New variables.
---
 gnu/packages/python.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 688a5d4..c649bee 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -10939,3 +10939,33 @@ with an associated set of resolve methods that know how to fetch data.")
 provide extendible implementations of common aspects of a cloud so that you can
 focus on building massively scalable web applications.")
     (license license:expat)))
+
+(define-public python-sphinx-bootstrap-theme
+  (package
+    (name "python-sphinx-bootstrap-theme")
+    (version "0.4.12")
+    (source (origin
+              (method url-fetch)
+              (uri (pypi-uri "sphinx-bootstrap-theme" version))
+              (sha256
+               (base32
+                "0wmm292rpfzxaib7zf2j6kdl1dl2xzx303hx8sx8qsdy0pkmrk65"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f ; No tests.
+       ;; Without this flag, various artifacts from the build inputs may end up
+       ;; in the final output. It also works around https://bugs.gnu.org/20765 .
+       #:configure-flags '("--single-version-externally-managed" "--root=/")))
+    (home-page "https://ryan-roemer.github.io/sphinx-bootstrap-theme/")
+    (synopsis "Bootstrap theme for Sphinx")
+    (description "Sphinx theme that integrates the Bootstrap CSS / JavaScript
+framework with various layout options, hierarchical menu navigation, and
+mobile-friendly responsive design.")
+    (license license:expat)
+    (properties `((python2-variant . ,(delay python2-sphinx-bootstrap-theme))))))
+
+(define-public python2-sphinx-bootstrap-theme
+  (package (inherit (package-with-python2 (strip-python2-variant
+                                           python-sphinx-bootstrap-theme)))
+           (native-inputs
+            `(("python2-setuptools" ,python2-setuptools)))))
-- 
2.10.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-gnu-Add-python-sphinx-argparse.patch --]
[-- Type: text/x-patch, Size: 2816 bytes --]

From e81dcaa9ad6ffd507ddbd19ab8d1583b626d3dc1 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sun, 25 Sep 2016 20:45:07 +0100
Subject: [PATCH 3/4] gnu: Add python-sphinx-argparse.

* gnu/packages/python.scm (python-sphinx-argparse,
  python2-sphinx-argparse): New variables.
---
 gnu/packages/python.scm | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index c649bee..8ed0bc5 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -10969,3 +10969,50 @@ mobile-friendly responsive design.")
                                            python-sphinx-bootstrap-theme)))
            (native-inputs
             `(("python2-setuptools" ,python2-setuptools)))))
+
+(define-public python-sphinx-argparse
+  (package
+    (name "python-sphinx-argparse")
+    (version "0.1.15")
+    (source (origin
+              (method url-fetch)
+              (uri (pypi-uri "sphinx-argparse" version))
+              (sha256
+               (base32
+                "14wdxq379xxnhw0kgf8z6jqdi0rd4k5y20jllyar9mxwwjblayvq"))))
+    (build-system python-build-system)
+    (arguments
+     `(;; Without this flag, various artifacts from the build inputs may end up
+       ;; in the final output. It also works around https://bugs.gnu.org/20765 .
+       #:configure-flags '("--single-version-externally-managed" "--root=/")
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'adjust-tests
+           ;; Two tests compare the output of "py.test --help" and fail
+           ;; when it gets ".py.test-real" back, so we substitute it here.
+           (lambda _
+             (substitute* "test/test_parser.py"
+               (("py.test") ".py.test-real"))
+             #t))
+         (delete 'check)
+         (add-after 'install 'check
+           (lambda _ (zero? (system* "py.test" "-vv")))))))
+    (native-inputs
+     `(("python-pytest" ,python-pytest)))
+    (propagated-inputs
+     `(("python-sphinx" ,python-sphinx)
+       ("python-docutils" ,python-docutils)))
+    (home-page "https://github.com/ribozz/sphinx-argparse")
+    (synopsis "Sphinx extension to document argparse commands")
+    (description "Sphinx extension that automatically documents @code{argparse}
+commands and options.")
+    (license license:expat)
+    (properties `((python2-variant . ,(delay python2-sphinx-argparse))))))
+
+(define-public python2-sphinx-argparse
+  (let ((base (package-with-python2 (strip-python2-variant
+                                     python-sphinx-argparse))))
+    (package (inherit base)
+             (native-inputs
+              `(("python2-setuptools" ,python2-setuptools)
+                ,@(package-native-inputs base))))))
-- 
2.10.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-gnu-Add-python2-pbcommand.patch --]
[-- Type: text/x-patch, Size: 3309 bytes --]

From 4d49796615cfbd8c34890835689eea74ddcfca60 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sun, 25 Sep 2016 21:11:58 +0100
Subject: [PATCH 4/4] gnu: Add python2-pbcommand.

* gnu/packages/bioinformatics.scm (python2-pbcommand): New variable.
---
 gnu/packages/bioinformatics.scm | 52 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index 1bf91a9..f5a5756 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -3373,6 +3373,58 @@ interrupted by stop codons.  OrfM finds and prints these ORFs.")
     (home-page "https://github.com/wwood/OrfM")
     (license license:lgpl3+)))
 
+(define-public python2-pbcommand
+  ;; Upstream does not tag git releases and PyPi is out of date.
+  ;; See https://github.com/PacificBiosciences/pbcommand/issues/116
+  (let ((revision "1")
+        (commit "e83a3443a1cd42b4da0d210201d711ede789917f"))
+    (package
+      (name "python2-pbcommand")
+      (version (string-append "0.4.11-" revision "." (string-take commit 7)))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/PacificBiosciences/pbcommand.git")
+                      (commit commit)))
+                (file-name (string-append name "-" version "-checkout"))
+                (sha256
+                 (base32
+                  "1kwg84f23l7ik65qy7cxa6g5nipc2y23mppigd4j3vlzam18v52h"))))
+      (build-system python-build-system)
+      (arguments
+       `(#:python ,python-2
+         ;; Without this flag, various artifacts from the build inputs may end up
+         ;; in the final output. It also works around https://bugs.gnu.org/20765 .
+         #:configure-flags '("--single-version-externally-managed" "--root=/")
+         #:phases
+         (modify-phases %standard-phases
+           (add-after 'unpack 'substitute-/bin/bash
+             (lambda _
+               ;; Fully qualify /bin/bash for running external commands.
+               (substitute* "pbcommand/engine/runner.py"
+                 (("/bin/bash") (which "bash")))
+               #t))
+           (replace 'check
+             (lambda _ (zero? (system* "nosetests" "-v")))))))
+      (native-inputs
+       `(("python-nose" ,python2-nose)
+         ("python-sphinx-bootstrap-theme" ,python2-sphinx-bootstrap-theme)
+         ("python-sphinx-argparse" ,python2-sphinx-argparse)
+         ("python-tox" ,python2-tox)
+         ("python-setuptools" ,python2-setuptools)))
+      (propagated-inputs
+       `(("python-functools32" ,python2-functools32)
+         ("python-jsonschema" ,python2-jsonschema)
+         ("python-numpy" ,python2-numpy "out")
+         ("python-avro" ,python2-avro)
+         ("python-requests" ,python2-requests)
+         ("python-iso8601" ,python2-iso8601)))
+      (home-page "https://github.com/PacificBiosciences/pbcommand")
+      (synopsis "PacBio common models and CLI tool contract interface")
+      (description "PacBio library for common utils, models, and tools
+to interface with pbsmrtpipe workflow engine.")
+      (license license:bsd-3))))
+
 (define-public python2-pbcore
   (package
     (name "python2-pbcore")
-- 
2.10.0