Re: [PATCH 2/3] gnu: Add python-pyxb.

[Marius Bakke <mbakke@fastmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2900 bytes --]

Ben Woodcroft <b.woodcroft@uq.edu.au> writes:

>> Subject: [PATCH 1/3] gnu: python-pysam: Update to 0.9.1.4.
> I'm not sure whether this is a product of the upgrade or not, but I 
> notice this in the build log. I think it is harmless though, WDYT?
>
> starting phase `validate-runpath'
> validating RUNPATH of 10 binaries in 
> "/gnu/store/bpiq3lm6b1kpf54i1vj2dl09ff293wic-python-pysam-0.9.1.4/lib"...
> /gnu/store/bpiq3lm6b1kpf54i1vj2dl09ff293wic-python-pysam-0.9.1.4/lib/python3.4/site-packages/pysam-0.9.1.4-py3.4-linux-x86_64.egg/pysam/libchtslib.cpython-34m.so: 
> warning: RUNPATH contains bogus entries: ("pysam" "." 
> "build/lib.linux-x86_64-3.4/pysam")

I don't see this in the previous version, so it is a regression.
However, it should be mostly harmless. Readelf reports (when compiled
with external htslib, see below):

 0x000000000000001d (RUNPATH)            Library runpath: [/gnu/store/m4gc2wx4q9if1vrhgclpspdil7rqsn21-python-3.4.3/lib:/gnu/store/ba22myqvxccwmmjwwq665rc43hanycxy-htslib-1.3.1/lib:build/lib.linux-x86_64-3.4/pysam:$ORIGIN:/gnu/store/m9vxvhdj691bq1f85lpflvnhcvrdilih-glibc-2.23/lib:/gnu/store/9nifwk709wajpyfwa0jzaa3p6mf10vxs-gcc-4.9.3-lib/lib:/gnu/store/xl19qrfzga52vrvp4ncccwjlnrjqwj95-ncurses-6.0/lib:/gnu/store/5992iq1v7arqa14ym3di58n4la0893nv-zlib-1.2.8/lib:/gnu/store/9nifwk709wajpyfwa0jzaa3p6mf10vxs-gcc-4.9.3-lib/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/../../..]

Compared to the runpath of the same file currently in Guix:

 0x000000000000001d (RUNPATH)            Library runpath: [/gnu/store/m4gc2wx4q9if1vrhgclpspdil7rqsn21-python-3.4.3/lib:/gnu/store/m9vxvhdj691bq1f85lpflvnhcvrdilih-glibc-2.23/lib:/gnu/store/9nifwk709wajpyfwa0jzaa3p6mf10vxs-gcc-4.9.3-lib/lib:/gnu/store/xl19qrfzga52vrvp4ncccwjlnrjqwj95-ncurses-6.0/lib:/gnu/store/5992iq1v7arqa14ym3di58n4la0893nv-zlib-1.2.8/lib:/gnu/store/9nifwk709wajpyfwa0jzaa3p6mf10vxs-gcc-4.9.3-lib/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/../../..]

If a folder named "$CWD/build/lib.linux-x86_64-3.4/pysam exists, it
could potentially allow for code injection, which is troubling.

I opened an issue on their tracker, but don't think it's worth holding
the patch: https://github.com/pysam-developers/pysam/issues/347

>
> Also, I notice that pysam bundles htslib, bcftools and samtools C code. 
> Hopefully it should be straightforward enough to remove htslib as there 
> are install instructions, I'm not sure about the other two. This 
> shouldn't block the patch here, but would you mind taking a look?
> http://pysam.readthedocs.io/en/latest/installation.html#installation

I had a go at this, and also enabled tests since I was reading the build
system anyway. Samtools and bcftools does not seem possible to un-bundle
at this time, but htslib was straightforward.

Enabling tests required packaging bcftools as well. I've attached the
full patch series below. The first three commits are unchanged.

Thanks!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-python-pysam-Update-to-0.9.1.4.patch --]
[-- Type: text/x-patch, Size: 1174 bytes --]

From 0a6220a664ecf3aa967211db5bf9e9b9e539220d Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Fri, 16 Sep 2016 18:05:15 +0100
Subject: [PATCH 1/6] gnu: python-pysam: Update to 0.9.1.4.

* gnu/packages/bioinformatics.scm (python-pysam, python2-pysam): Update
to 0.9.1.4.
---
 gnu/packages/bioinformatics.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index 7a27e0b..47b884e 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -1290,13 +1290,13 @@ multiple sequence alignments.")
 (define-public python-pysam
   (package
     (name "python-pysam")
-    (version "0.8.4")
+    (version "0.9.1.4")
     (source (origin
               (method url-fetch)
               (uri (pypi-uri "pysam" version))
               (sha256
                (base32
-                "1slx5mb94mzm5qzk52q270sab0sar95j67w1g1k452nz3s9j7krh"))))
+                "1i1djacqbr88y7w18b4aa78zxnsyr4sz7yqdq2spi7gs0y6pzvjn"))))
     (build-system python-build-system)
     (arguments
      `(#:tests? #f ; tests are excluded in the manifest
-- 
2.10.0


[-- Attachment #3: 0002-gnu-Add-python-pyxb.patch --]
[-- Type: text/x-patch, Size: 1994 bytes --]

From 6a8ddd17d7f11e0687217c8649a452a1a56c596c Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sat, 17 Sep 2016 10:33:28 +0100
Subject: [PATCH 2/6] gnu: Add python-pyxb.

* gnu/packages/xml.scm (python-pyxb, python2-pyxb): New variables.
---
 gnu/packages/xml.scm | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/gnu/packages/xml.scm b/gnu/packages/xml.scm
index 7befad5..cd3426b 100644
--- a/gnu/packages/xml.scm
+++ b/gnu/packages/xml.scm
@@ -13,6 +13,7 @@
 ;;; Copyright © 2016 Jan Nieuwenhuizen <janneke@gnu.org>
 ;;; Copyright © 2016 ng0 <ng0@we.make.ritual.n0.is>
 ;;; Copyright © 2016 Tobias Geerinckx-Rice <me@tobias.gr>
+;;; Copyright © 2016 Marius Bakke <mbakke@fastmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -729,6 +730,30 @@ different Unicode encodings which happen automatically during
 parsing/saving.")
     (license license:expat)))
 
+(define-public python-pyxb
+  (package
+    (name "python-pyxb")
+    (version "1.2.5")
+    (source (origin
+              (method url-fetch)
+              (uri (pypi-uri "PyXB" version))
+              (sha256
+               (base32
+                "0rzzwibfqa28gxgcxx4cybx1qcg0g6fand06ykj3gz7z5kp653sf"))))
+    (build-system python-build-system)
+    (home-page "http://pyxb.sourceforge.net/")
+    (synopsis "Python XML Schema Bindings")
+    (description
+     "PyXB (\"pixbee\") is a pure Python package that generates Python source
+code for classes that correspond to data structures defined by XMLSchema.")
+    (license (list license:asl2.0    ; Most files.
+                   license:expat     ; pyxb/utils/six.py
+                   license:gpl2      ; bundled jquery in doc is dual MIT/GPL2
+                   license:psfl))))  ; pyxb/utils/activestate.py
+
+(define-public python2-pyxb
+  (package-with-python2 python-pyxb))
+
 (define-public xmlto
   (package
     (name "xmlto")
-- 
2.10.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-gnu-python2-pbcore-Update-to-1.2.10.patch --]
[-- Type: text/x-patch, Size: 1690 bytes --]

From 396cbcbfda24ce5cd1caecedf01407bf6a1f70a1 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sat, 17 Sep 2016 10:42:56 +0100
Subject: [PATCH 3/6] gnu: python2-pbcore: Update to 1.2.10.

* gnu/packages/bioinformatics.scm (python2-pbcore): Update to 1.2.10.
[propagated-inputs]: New field. Add python2-pyxb.
---
 gnu/packages/bioinformatics.scm | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index 47b884e..a598351 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -3376,13 +3376,13 @@ interrupted by stop codons.  OrfM finds and prints these ORFs.")
 (define-public python2-pbcore
   (package
     (name "python2-pbcore")
-    (version "1.2.8")
+    (version "1.2.10")
     (source (origin
               (method url-fetch)
               (uri (pypi-uri "pbcore" version))
               (sha256
                (base32
-                "02pfn5raa3zf739672bg0dkx7z3j2c4nx7vmpfjqy5b12jrqpymk"))))
+                "1kjmv891d6qbpp4shhhvkl02ff4q5xlpnls2513sm2cjcrs52f1i"))))
     (build-system python-build-system)
     (arguments `(#:python ,python-2)) ; pbcore requires Python 2.7
     (inputs
@@ -3395,6 +3395,8 @@ interrupted by stop codons.  OrfM finds and prints these ORFs.")
        ("python-nose" ,python2-nose)
        ("python-setuptools" ,python2-setuptools)
        ("python-sphinx" ,python2-sphinx)))
+    (propagated-inputs
+     `(("python-pyxb" ,python2-pyxb)))
     (home-page "http://pacificbiosciences.github.io/pbcore/")
     (synopsis "Library for reading and writing PacBio data files")
     (description
-- 
2.10.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-gnu-Add-bcftools.patch --]
[-- Type: text/x-patch, Size: 3295 bytes --]

From 2f61ecb17e155c41624a81d4d95d2e80823da02c Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Thu, 22 Sep 2016 13:47:53 +0100
Subject: [PATCH 4/6] gnu: Add bcftools.

* gnu/packages/bioinformatics.scm (bcftools): New variable.
---
 gnu/packages/bioinformatics.scm | 60 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index a598351..e9ab05f 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -265,6 +265,66 @@ instance, it implements several methods to assess contig-wise read coverage.")
 BAM files.")
     (license license:expat)))
 
+(define-public bcftools
+  (package
+    (name "bcftools")
+    (version "1.3.1")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+                    "https://github.com/samtools/bcftools/releases/download/"
+                    version "/bcftools-" version ".tar.bz2"))
+              (sha256
+               (base32
+                "095ry68vmz9q5s1scjsa698dhgyvgw5aicz24c19iwfbai07mhqj"))
+              (modules '((guix build utils)))
+              (snippet
+               ;; Delete bundled htslib.
+               '(delete-file-recursively "htslib-1.3.1"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:test-target "test"
+       #:make-flags (list
+                     "USE_GPL=1"
+                     (string-append "prefix=" (assoc-ref %outputs "out"))
+                     (string-append
+                      "HTSDIR=" (assoc-ref %build-inputs "htslib") "/include")
+                     (string-append
+                      "HTSLIB=" (assoc-ref %build-inputs "htslib") "/lib/libhts.a")
+                     (string-append
+                      "BGZIP=" (assoc-ref %build-inputs "htslib") "/bin/bgzip")
+                     (string-append
+                      "TABIX=" (assoc-ref %build-inputs "htslib") "/bin/tabix"))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'patch-Makefile
+           (lambda _
+             (substitute* "Makefile"
+               ;; Do not attempt to build htslib.
+               (("^include \\$\\(HTSDIR\\)/htslib\\.mk") "")
+               ;; Link against GSL cblas.
+               (("-lcblas") "-lgslcblas"))
+             #t))
+         (delete 'configure)
+         (add-before 'check 'patch-/bin/bash
+           (lambda _
+             (substitute* "test/test.pl"
+               (("/bin/bash") (which "bash"))))))))
+    (native-inputs
+     `(("htslib" ,htslib)
+       ("perl" ,perl)))
+    (inputs
+     `(("gsl" ,gsl)
+       ("zlib" ,zlib)))
+    (home-page "https://samtools.github.io/bcftools/")
+    (synopsis "Utilities for variant calling and manipulating VCFs and BCFs")
+    (description
+     "BCFtools is a set of utilities that manipulate variant calls in the
+Variant Call Format (VCF) and its binary counterpart BCF.  All commands work
+transparently with both VCFs and BCFs, both uncompressed and BGZF-compressed.")
+    ;; The sources are dual MIT/GPL, but becomes GPL-only when USE_GPL=1.
+    (license (list license:gpl3+ license:expat))))
+
 (define-public bedops
   (package
     (name "bedops")
-- 
2.10.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-gnu-python-pysam-Delete-bundled-htslib.patch --]
[-- Type: text/x-patch, Size: 2280 bytes --]

From 29d8aab712b611a6c786aa9b2c47f45e51203ca1 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Thu, 22 Sep 2016 14:08:05 +0100
Subject: [PATCH 5/6] gnu: python-pysam: Delete bundled htslib.

* gnu/packages/bioinformatics.scm (python-pysam)[source]: Add snippet
to delete htslib.
[arguments]: Adjust to modify-phases. Add htslib flags in 'set-flags.
[propagated-inputs]: New field. Add htslib.
---
 gnu/packages/bioinformatics.scm | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index e9ab05f..edfb943 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -1356,17 +1356,27 @@ multiple sequence alignments.")
               (uri (pypi-uri "pysam" version))
               (sha256
                (base32
-                "1i1djacqbr88y7w18b4aa78zxnsyr4sz7yqdq2spi7gs0y6pzvjn"))))
+                "1i1djacqbr88y7w18b4aa78zxnsyr4sz7yqdq2spi7gs0y6pzvjn"))
+              (modules '((guix build utils)))
+              (snippet
+               ;; Drop bundled htslib. TODO: Also remove samtools and bcftools.
+               '(delete-file-recursively "htslib"))))
     (build-system python-build-system)
     (arguments
      `(#:tests? #f ; tests are excluded in the manifest
        #:phases
-       (alist-cons-before
-        'build 'set-flags
-        (lambda _
-          (setenv "LDFLAGS" "-lncurses")
-          (setenv "CFLAGS" "-D_CURSES_LIB=1"))
-        %standard-phases)))
+       (modify-phases %standard-phases
+         (add-before 'build 'set-flags
+           (lambda* (#:key inputs #:allow-other-keys)
+             (setenv "LDFLAGS" "-lncurses")
+             (setenv "HTSLIB_MODE" "external")
+             (setenv "HTSLIB_LIBRARY_DIR" (string-append
+                                           (assoc-ref inputs "htslib") "/lib"))
+             (setenv "HTSLIB_INCLUDE_DIR" (string-append
+                                           (assoc-ref inputs "htslib") "/include"))
+             (setenv "CFLAGS" "-D_CURSES_LIB=1"))))))
+    (propagated-inputs
+     `(("htslib"            ,htslib)))
     (inputs
      `(("ncurses"           ,ncurses)
        ("zlib"              ,zlib)))
-- 
2.10.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-gnu-python-pysam-Enable-tests.patch --]
[-- Type: text/x-patch, Size: 3624 bytes --]

From ab4f0a5174554a7f2b7b243b5488654de7d81ac1 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Thu, 22 Sep 2016 14:11:59 +0100
Subject: [PATCH 6/6] gnu: python-pysam: Enable tests.

* gnu/packages/bioinformatics.scm (python-pysam)[source]: Change from
PyPi to source archive due to missing test data.
[arguments]: Add check phase after install.
[native-inputs]: Add python-nose, samtools and bcftools.
---
 gnu/packages/bioinformatics.scm | 33 ++++++++++++++++++++++++++++-----
 1 file changed, 28 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index edfb943..abbe601 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -1353,18 +1353,21 @@ multiple sequence alignments.")
     (version "0.9.1.4")
     (source (origin
               (method url-fetch)
-              (uri (pypi-uri "pysam" version))
+              ;; Test data is missing on PyPi.
+              (uri (string-append
+                    "https://github.com/pysam-developers/pysam/archive/v"
+                    version ".tar.gz"))
+              (file-name (string-append name "-" version ".tar.gz"))
               (sha256
                (base32
-                "1i1djacqbr88y7w18b4aa78zxnsyr4sz7yqdq2spi7gs0y6pzvjn"))
+                "0y41ssbg6nvn2jgcbnrvkzblpjcwszaiv1rgyd8dwzjkrbfsgsmc"))
               (modules '((guix build utils)))
               (snippet
                ;; Drop bundled htslib. TODO: Also remove samtools and bcftools.
                '(delete-file-recursively "htslib"))))
     (build-system python-build-system)
     (arguments
-     `(#:tests? #f ; tests are excluded in the manifest
-       #:phases
+     `(#:phases
        (modify-phases %standard-phases
          (add-before 'build 'set-flags
            (lambda* (#:key inputs #:allow-other-keys)
@@ -1374,7 +1377,24 @@ multiple sequence alignments.")
                                            (assoc-ref inputs "htslib") "/lib"))
              (setenv "HTSLIB_INCLUDE_DIR" (string-append
                                            (assoc-ref inputs "htslib") "/include"))
-             (setenv "CFLAGS" "-D_CURSES_LIB=1"))))))
+             (setenv "CFLAGS" "-D_CURSES_LIB=1")))
+         (delete 'check)
+         (add-after 'install 'check
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (setenv "PYTHONPATH"
+                     (string-append
+                      (getenv "PYTHONPATH")
+                      ":" (assoc-ref outputs "out")
+                      "/lib/python"
+                      (string-take (string-take-right
+                                    (assoc-ref inputs "python") 5) 3)
+                      "/site-packages"))
+             ;; Step out of source dir so python does not import from CWD.
+             (chdir "tests")
+             (setenv "HOME" "/tmp")
+             (and (zero? (system* "make" "-C" "pysam_data"))
+                  (zero? (system* "make" "-C" "cbcf_data"))
+                  (zero? (system* "nosetests" "-v"))))))))
     (propagated-inputs
      `(("htslib"            ,htslib)))
     (inputs
@@ -1382,6 +1402,9 @@ multiple sequence alignments.")
        ("zlib"              ,zlib)))
     (native-inputs
      `(("python-cython"     ,python-cython)
+       ("python-nose"       ,python-nose)
+       ("samtools"          ,samtools)
+       ("bcftools"          ,bcftools)
        ("python-setuptools" ,python-setuptools)))
     (home-page "https://github.com/pysam-developers/pysam")
     (synopsis "Python bindings to the SAMtools C API")
-- 
2.10.0