From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marius Bakke Subject: Re: [PATCH 2/3] gnu: Add python-pyxb. Date: Thu, 22 Sep 2016 16:15:15 +0100 Message-ID: <87eg4cyni4.fsf@ike.i-did-not-set--mail-host-address--so-tickle-me> References: <20160917101047.4597-1-mbakke@fastmail.com> <20160917101047.4597-3-mbakke@fastmail.com> <3bd16b58-b3d1-d47c-2433-c3a721681463@uq.edu.au> <39d81c47-a96d-1f1c-ad1d-a80e7b7f109d@uq.edu.au> <1a9d61d6-0ee2-1161-25b4-9ffd32396039@uq.edu.au> <87h999ymzk.fsf@ike.i-did-not-set--mail-host-address--so-tickle-me> <679cc096-9eed-152f-0a01-f4a1d85c422e@uq.edu.au> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:52917) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bn5in-0007x3-V6 for guix-devel@gnu.org; Thu, 22 Sep 2016 11:15:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bn5ii-0006rd-QW for guix-devel@gnu.org; Thu, 22 Sep 2016 11:15:32 -0400 Received: from out2-smtp.messagingengine.com ([66.111.4.26]:34855) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bn5ig-0006nw-8T for guix-devel@gnu.org; Thu, 22 Sep 2016 11:15:28 -0400 In-Reply-To: <679cc096-9eed-152f-0a01-f4a1d85c422e@uq.edu.au> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ben Woodcroft , "guix-devel@gnu.org" --=-=-= Content-Type: text/plain Ben Woodcroft writes: >> Subject: [PATCH 1/3] gnu: python-pysam: Update to 0.9.1.4. > I'm not sure whether this is a product of the upgrade or not, but I > notice this in the build log. I think it is harmless though, WDYT? > > starting phase `validate-runpath' > validating RUNPATH of 10 binaries in > "/gnu/store/bpiq3lm6b1kpf54i1vj2dl09ff293wic-python-pysam-0.9.1.4/lib"... > /gnu/store/bpiq3lm6b1kpf54i1vj2dl09ff293wic-python-pysam-0.9.1.4/lib/python3.4/site-packages/pysam-0.9.1.4-py3.4-linux-x86_64.egg/pysam/libchtslib.cpython-34m.so: > warning: RUNPATH contains bogus entries: ("pysam" "." > "build/lib.linux-x86_64-3.4/pysam") I don't see this in the previous version, so it is a regression. However, it should be mostly harmless. Readelf reports (when compiled with external htslib, see below): 0x000000000000001d (RUNPATH) Library runpath: [/gnu/store/m4gc2wx4q9if1vrhgclpspdil7rqsn21-python-3.4.3/lib:/gnu/store/ba22myqvxccwmmjwwq665rc43hanycxy-htslib-1.3.1/lib:build/lib.linux-x86_64-3.4/pysam:$ORIGIN:/gnu/store/m9vxvhdj691bq1f85lpflvnhcvrdilih-glibc-2.23/lib:/gnu/store/9nifwk709wajpyfwa0jzaa3p6mf10vxs-gcc-4.9.3-lib/lib:/gnu/store/xl19qrfzga52vrvp4ncccwjlnrjqwj95-ncurses-6.0/lib:/gnu/store/5992iq1v7arqa14ym3di58n4la0893nv-zlib-1.2.8/lib:/gnu/store/9nifwk709wajpyfwa0jzaa3p6mf10vxs-gcc-4.9.3-lib/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/../../..] Compared to the runpath of the same file currently in Guix: 0x000000000000001d (RUNPATH) Library runpath: [/gnu/store/m4gc2wx4q9if1vrhgclpspdil7rqsn21-python-3.4.3/lib:/gnu/store/m9vxvhdj691bq1f85lpflvnhcvrdilih-glibc-2.23/lib:/gnu/store/9nifwk709wajpyfwa0jzaa3p6mf10vxs-gcc-4.9.3-lib/lib:/gnu/store/xl19qrfzga52vrvp4ncccwjlnrjqwj95-ncurses-6.0/lib:/gnu/store/5992iq1v7arqa14ym3di58n4la0893nv-zlib-1.2.8/lib:/gnu/store/9nifwk709wajpyfwa0jzaa3p6mf10vxs-gcc-4.9.3-lib/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/../../..] If a folder named "$CWD/build/lib.linux-x86_64-3.4/pysam exists, it could potentially allow for code injection, which is troubling. I opened an issue on their tracker, but don't think it's worth holding the patch: https://github.com/pysam-developers/pysam/issues/347 > > Also, I notice that pysam bundles htslib, bcftools and samtools C code. > Hopefully it should be straightforward enough to remove htslib as there > are install instructions, I'm not sure about the other two. This > shouldn't block the patch here, but would you mind taking a look? > http://pysam.readthedocs.io/en/latest/installation.html#installation I had a go at this, and also enabled tests since I was reading the build system anyway. Samtools and bcftools does not seem possible to un-bundle at this time, but htslib was straightforward. Enabling tests required packaging bcftools as well. I've attached the full patch series below. The first three commits are unchanged. Thanks! --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-gnu-python-pysam-Update-to-0.9.1.4.patch >From 0a6220a664ecf3aa967211db5bf9e9b9e539220d Mon Sep 17 00:00:00 2001 From: Marius Bakke Date: Fri, 16 Sep 2016 18:05:15 +0100 Subject: [PATCH 1/6] gnu: python-pysam: Update to 0.9.1.4. * gnu/packages/bioinformatics.scm (python-pysam, python2-pysam): Update to 0.9.1.4. --- gnu/packages/bioinformatics.scm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm index 7a27e0b..47b884e 100644 --- a/gnu/packages/bioinformatics.scm +++ b/gnu/packages/bioinformatics.scm @@ -1290,13 +1290,13 @@ multiple sequence alignments.") (define-public python-pysam (package (name "python-pysam") - (version "0.8.4") + (version "0.9.1.4") (source (origin (method url-fetch) (uri (pypi-uri "pysam" version)) (sha256 (base32 - "1slx5mb94mzm5qzk52q270sab0sar95j67w1g1k452nz3s9j7krh")))) + "1i1djacqbr88y7w18b4aa78zxnsyr4sz7yqdq2spi7gs0y6pzvjn")))) (build-system python-build-system) (arguments `(#:tests? #f ; tests are excluded in the manifest -- 2.10.0 --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: attachment; filename=0002-gnu-Add-python-pyxb.patch Content-Transfer-Encoding: quoted-printable >From 6a8ddd17d7f11e0687217c8649a452a1a56c596c Mon Sep 17 00:00:00 2001 From: Marius Bakke Date: Sat, 17 Sep 2016 10:33:28 +0100 Subject: [PATCH 2/6] gnu: Add python-pyxb. * gnu/packages/xml.scm (python-pyxb, python2-pyxb): New variables. --- gnu/packages/xml.scm | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/gnu/packages/xml.scm b/gnu/packages/xml.scm index 7befad5..cd3426b 100644 --- a/gnu/packages/xml.scm +++ b/gnu/packages/xml.scm @@ -13,6 +13,7 @@ ;;; Copyright =C2=A9 2016 Jan Nieuwenhuizen ;;; Copyright =C2=A9 2016 ng0 ;;; Copyright =C2=A9 2016 Tobias Geerinckx-Rice +;;; Copyright =C2=A9 2016 Marius Bakke ;;; ;;; This file is part of GNU Guix. ;;; @@ -729,6 +730,30 @@ different Unicode encodings which happen automatically= during parsing/saving.") (license license:expat))) =20 +(define-public python-pyxb + (package + (name "python-pyxb") + (version "1.2.5") + (source (origin + (method url-fetch) + (uri (pypi-uri "PyXB" version)) + (sha256 + (base32 + "0rzzwibfqa28gxgcxx4cybx1qcg0g6fand06ykj3gz7z5kp653sf")))) + (build-system python-build-system) + (home-page "http://pyxb.sourceforge.net/") + (synopsis "Python XML Schema Bindings") + (description + "PyXB (\"pixbee\") is a pure Python package that generates Python sou= rce +code for classes that correspond to data structures defined by XMLSchema.") + (license (list license:asl2.0 ; Most files. + license:expat ; pyxb/utils/six.py + license:gpl2 ; bundled jquery in doc is dual MIT/G= PL2 + license:psfl)))) ; pyxb/utils/activestate.py + +(define-public python2-pyxb + (package-with-python2 python-pyxb)) + (define-public xmlto (package (name "xmlto") --=20 2.10.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0003-gnu-python2-pbcore-Update-to-1.2.10.patch >From 396cbcbfda24ce5cd1caecedf01407bf6a1f70a1 Mon Sep 17 00:00:00 2001 From: Marius Bakke Date: Sat, 17 Sep 2016 10:42:56 +0100 Subject: [PATCH 3/6] gnu: python2-pbcore: Update to 1.2.10. * gnu/packages/bioinformatics.scm (python2-pbcore): Update to 1.2.10. [propagated-inputs]: New field. Add python2-pyxb. --- gnu/packages/bioinformatics.scm | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm index 47b884e..a598351 100644 --- a/gnu/packages/bioinformatics.scm +++ b/gnu/packages/bioinformatics.scm @@ -3376,13 +3376,13 @@ interrupted by stop codons. OrfM finds and prints these ORFs.") (define-public python2-pbcore (package (name "python2-pbcore") - (version "1.2.8") + (version "1.2.10") (source (origin (method url-fetch) (uri (pypi-uri "pbcore" version)) (sha256 (base32 - "02pfn5raa3zf739672bg0dkx7z3j2c4nx7vmpfjqy5b12jrqpymk")))) + "1kjmv891d6qbpp4shhhvkl02ff4q5xlpnls2513sm2cjcrs52f1i")))) (build-system python-build-system) (arguments `(#:python ,python-2)) ; pbcore requires Python 2.7 (inputs @@ -3395,6 +3395,8 @@ interrupted by stop codons. OrfM finds and prints these ORFs.") ("python-nose" ,python2-nose) ("python-setuptools" ,python2-setuptools) ("python-sphinx" ,python2-sphinx))) + (propagated-inputs + `(("python-pyxb" ,python2-pyxb))) (home-page "http://pacificbiosciences.github.io/pbcore/") (synopsis "Library for reading and writing PacBio data files") (description -- 2.10.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0004-gnu-Add-bcftools.patch >From 2f61ecb17e155c41624a81d4d95d2e80823da02c Mon Sep 17 00:00:00 2001 From: Marius Bakke Date: Thu, 22 Sep 2016 13:47:53 +0100 Subject: [PATCH 4/6] gnu: Add bcftools. * gnu/packages/bioinformatics.scm (bcftools): New variable. --- gnu/packages/bioinformatics.scm | 60 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm index a598351..e9ab05f 100644 --- a/gnu/packages/bioinformatics.scm +++ b/gnu/packages/bioinformatics.scm @@ -265,6 +265,66 @@ instance, it implements several methods to assess contig-wise read coverage.") BAM files.") (license license:expat))) +(define-public bcftools + (package + (name "bcftools") + (version "1.3.1") + (source (origin + (method url-fetch) + (uri (string-append + "https://github.com/samtools/bcftools/releases/download/" + version "/bcftools-" version ".tar.bz2")) + (sha256 + (base32 + "095ry68vmz9q5s1scjsa698dhgyvgw5aicz24c19iwfbai07mhqj")) + (modules '((guix build utils))) + (snippet + ;; Delete bundled htslib. + '(delete-file-recursively "htslib-1.3.1")))) + (build-system gnu-build-system) + (arguments + `(#:test-target "test" + #:make-flags (list + "USE_GPL=1" + (string-append "prefix=" (assoc-ref %outputs "out")) + (string-append + "HTSDIR=" (assoc-ref %build-inputs "htslib") "/include") + (string-append + "HTSLIB=" (assoc-ref %build-inputs "htslib") "/lib/libhts.a") + (string-append + "BGZIP=" (assoc-ref %build-inputs "htslib") "/bin/bgzip") + (string-append + "TABIX=" (assoc-ref %build-inputs "htslib") "/bin/tabix")) + #:phases + (modify-phases %standard-phases + (add-after 'unpack 'patch-Makefile + (lambda _ + (substitute* "Makefile" + ;; Do not attempt to build htslib. + (("^include \\$\\(HTSDIR\\)/htslib\\.mk") "") + ;; Link against GSL cblas. + (("-lcblas") "-lgslcblas")) + #t)) + (delete 'configure) + (add-before 'check 'patch-/bin/bash + (lambda _ + (substitute* "test/test.pl" + (("/bin/bash") (which "bash")))))))) + (native-inputs + `(("htslib" ,htslib) + ("perl" ,perl))) + (inputs + `(("gsl" ,gsl) + ("zlib" ,zlib))) + (home-page "https://samtools.github.io/bcftools/") + (synopsis "Utilities for variant calling and manipulating VCFs and BCFs") + (description + "BCFtools is a set of utilities that manipulate variant calls in the +Variant Call Format (VCF) and its binary counterpart BCF. All commands work +transparently with both VCFs and BCFs, both uncompressed and BGZF-compressed.") + ;; The sources are dual MIT/GPL, but becomes GPL-only when USE_GPL=1. + (license (list license:gpl3+ license:expat)))) + (define-public bedops (package (name "bedops") -- 2.10.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0005-gnu-python-pysam-Delete-bundled-htslib.patch >From 29d8aab712b611a6c786aa9b2c47f45e51203ca1 Mon Sep 17 00:00:00 2001 From: Marius Bakke Date: Thu, 22 Sep 2016 14:08:05 +0100 Subject: [PATCH 5/6] gnu: python-pysam: Delete bundled htslib. * gnu/packages/bioinformatics.scm (python-pysam)[source]: Add snippet to delete htslib. [arguments]: Adjust to modify-phases. Add htslib flags in 'set-flags. [propagated-inputs]: New field. Add htslib. --- gnu/packages/bioinformatics.scm | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm index e9ab05f..edfb943 100644 --- a/gnu/packages/bioinformatics.scm +++ b/gnu/packages/bioinformatics.scm @@ -1356,17 +1356,27 @@ multiple sequence alignments.") (uri (pypi-uri "pysam" version)) (sha256 (base32 - "1i1djacqbr88y7w18b4aa78zxnsyr4sz7yqdq2spi7gs0y6pzvjn")))) + "1i1djacqbr88y7w18b4aa78zxnsyr4sz7yqdq2spi7gs0y6pzvjn")) + (modules '((guix build utils))) + (snippet + ;; Drop bundled htslib. TODO: Also remove samtools and bcftools. + '(delete-file-recursively "htslib")))) (build-system python-build-system) (arguments `(#:tests? #f ; tests are excluded in the manifest #:phases - (alist-cons-before - 'build 'set-flags - (lambda _ - (setenv "LDFLAGS" "-lncurses") - (setenv "CFLAGS" "-D_CURSES_LIB=1")) - %standard-phases))) + (modify-phases %standard-phases + (add-before 'build 'set-flags + (lambda* (#:key inputs #:allow-other-keys) + (setenv "LDFLAGS" "-lncurses") + (setenv "HTSLIB_MODE" "external") + (setenv "HTSLIB_LIBRARY_DIR" (string-append + (assoc-ref inputs "htslib") "/lib")) + (setenv "HTSLIB_INCLUDE_DIR" (string-append + (assoc-ref inputs "htslib") "/include")) + (setenv "CFLAGS" "-D_CURSES_LIB=1")))))) + (propagated-inputs + `(("htslib" ,htslib))) (inputs `(("ncurses" ,ncurses) ("zlib" ,zlib))) -- 2.10.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0006-gnu-python-pysam-Enable-tests.patch >From ab4f0a5174554a7f2b7b243b5488654de7d81ac1 Mon Sep 17 00:00:00 2001 From: Marius Bakke Date: Thu, 22 Sep 2016 14:11:59 +0100 Subject: [PATCH 6/6] gnu: python-pysam: Enable tests. * gnu/packages/bioinformatics.scm (python-pysam)[source]: Change from PyPi to source archive due to missing test data. [arguments]: Add check phase after install. [native-inputs]: Add python-nose, samtools and bcftools. --- gnu/packages/bioinformatics.scm | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm index edfb943..abbe601 100644 --- a/gnu/packages/bioinformatics.scm +++ b/gnu/packages/bioinformatics.scm @@ -1353,18 +1353,21 @@ multiple sequence alignments.") (version "0.9.1.4") (source (origin (method url-fetch) - (uri (pypi-uri "pysam" version)) + ;; Test data is missing on PyPi. + (uri (string-append + "https://github.com/pysam-developers/pysam/archive/v" + version ".tar.gz")) + (file-name (string-append name "-" version ".tar.gz")) (sha256 (base32 - "1i1djacqbr88y7w18b4aa78zxnsyr4sz7yqdq2spi7gs0y6pzvjn")) + "0y41ssbg6nvn2jgcbnrvkzblpjcwszaiv1rgyd8dwzjkrbfsgsmc")) (modules '((guix build utils))) (snippet ;; Drop bundled htslib. TODO: Also remove samtools and bcftools. '(delete-file-recursively "htslib")))) (build-system python-build-system) (arguments - `(#:tests? #f ; tests are excluded in the manifest - #:phases + `(#:phases (modify-phases %standard-phases (add-before 'build 'set-flags (lambda* (#:key inputs #:allow-other-keys) @@ -1374,7 +1377,24 @@ multiple sequence alignments.") (assoc-ref inputs "htslib") "/lib")) (setenv "HTSLIB_INCLUDE_DIR" (string-append (assoc-ref inputs "htslib") "/include")) - (setenv "CFLAGS" "-D_CURSES_LIB=1")))))) + (setenv "CFLAGS" "-D_CURSES_LIB=1"))) + (delete 'check) + (add-after 'install 'check + (lambda* (#:key inputs outputs #:allow-other-keys) + (setenv "PYTHONPATH" + (string-append + (getenv "PYTHONPATH") + ":" (assoc-ref outputs "out") + "/lib/python" + (string-take (string-take-right + (assoc-ref inputs "python") 5) 3) + "/site-packages")) + ;; Step out of source dir so python does not import from CWD. + (chdir "tests") + (setenv "HOME" "/tmp") + (and (zero? (system* "make" "-C" "pysam_data")) + (zero? (system* "make" "-C" "cbcf_data")) + (zero? (system* "nosetests" "-v")))))))) (propagated-inputs `(("htslib" ,htslib))) (inputs @@ -1382,6 +1402,9 @@ multiple sequence alignments.") ("zlib" ,zlib))) (native-inputs `(("python-cython" ,python-cython) + ("python-nose" ,python-nose) + ("samtools" ,samtools) + ("bcftools" ,bcftools) ("python-setuptools" ,python-setuptools))) (home-page "https://github.com/pysam-developers/pysam") (synopsis "Python bindings to the SAMtools C API") -- 2.10.0 --=-=-=--