[bug#44613] [PATCH] Fix build for bedtools

[bug#44613] [PATCH] Fix build for bedtools

[Roel Janssen]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

Dear Guix,

By updating samtools to 1.11, I introduced a build failure for
bedtools. More precisely, the tests for intersect break in precisely
this way:
https://github.com/arq5x/bedtools2/issues/814

With the following patches, I'd like to add samtools-1.9, htslib-1.9
(samtools depends on that) to fix this problem with bedtools.

Alternatively we could add a patch to disable the failing bedtools
tests.  I manually inspected the test results, and seem to match
perfectly (indicating that there's no problem with bedtools).

Kind regards,
Roel Janssen


[-- Attachment #2: 0003-gnu-bedtools-Use-samtools-1.9.patch --]
[-- Type: text/x-patch, Size: 1018 bytes --]

From 2541b3c44e695ff1a6b56128b6efa72c463c0b0a Mon Sep 17 00:00:00 2001
From: Roel Janssen <roel@gnu.org>
Date: Fri, 13 Nov 2020 11:29:38 +0100
Subject: [PATCH 3/3] gnu: bedtools: Use samtools-1.9.

The build for bedtools with samtools 1.11 triggers a testsuite
failure which is reported here:
https://github.com/arq5x/bedtools2/issues/814

* gnu/packages/bioinformatics.scm (bedtools): Use samtools-1.9.
---
 gnu/packages/bioinformatics.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index e047aebd1d..8ad38ac498 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -437,7 +437,7 @@ computational cluster.")
     (native-inputs
      `(("python" ,python-wrapper)))
     (inputs
-     `(("samtools" ,samtools)
+     `(("samtools" ,samtools-1.9)
        ("zlib" ,zlib)))
     (home-page "https://github.com/arq5x/bedtools2")
     (synopsis "Tools for genome analysis and arithmetic")
-- 
2.29.2


[-- Attachment #3: 0002-gnu-Add-samtools-1.9.patch --]
[-- Type: text/x-patch, Size: 1635 bytes --]

From 7881fd95bcddc0fee4717fab30dbb276745825d8 Mon Sep 17 00:00:00 2001
From: Roel Janssen <roel@gnu.org>
Date: Fri, 13 Nov 2020 11:29:20 +0100
Subject: [PATCH 2/3] gnu: Add samtools-1.9.

* gnu/packages/bioinformatics.scm (samtools-1.9): New variable.
---
 gnu/packages/bioinformatics.scm | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index ba86333fc3..e047aebd1d 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -5699,6 +5699,31 @@ variant calling (in conjunction with bcftools), and a simple alignment
 viewer.")
     (license license:expat)))
 
+(define-public samtools-1.9
+  (package (inherit samtools)
+    (name "samtools")
+    (version "1.9")
+    (source
+     (origin
+       (method url-fetch)
+       (uri
+        (string-append "mirror://sourceforge/samtools/samtools/"
+                       version "/samtools-" version ".tar.bz2"))
+       (sha256
+        (base32
+         "10ilqbmm7ri8z431sn90lvbjwizd0hhkf9rcqw8j823hf26nhgq8"))
+       (modules '((guix build utils)))
+       (snippet '(begin
+                   ;; Delete bundled htslib.
+                   (delete-file-recursively "htslib-1.9")
+                   #t))))
+    (inputs
+     `(("htslib" ,htslib-1.9)
+       ("ncurses" ,ncurses)
+       ("perl" ,perl)
+       ("python" ,python)
+       ("zlib" ,zlib)))))
+
 (define-public samtools-0.1
   ;; This is the most recent version of the 0.1 line of samtools.  The input
   ;; and output formats differ greatly from that used and produced by samtools
-- 
2.29.2


[-- Attachment #4: 0001-gnu-Add-htslib-1.9.patch --]
[-- Type: text/x-patch, Size: 1291 bytes --]

From 220604159383b09e11a7b0bed51237257c7c0442 Mon Sep 17 00:00:00 2001
From: Roel Janssen <roel@gnu.org>
Date: Fri, 13 Nov 2020 11:28:43 +0100
Subject: [PATCH 1/3] gnu: Add htslib-1.9.

* gnu/packages/bioinformatics.scm (htslib-1.9): New variable.
---
 gnu/packages/bioinformatics.scm | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index 06972dee51..ba86333fc3 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -4250,6 +4250,19 @@ data.  It also provides the @command{bgzip}, @command{htsfile}, and
     ;; the rest is released under the Expat license
     (license (list license:expat license:bsd-3))))
 
+(define-public htslib-1.9
+  (package (inherit htslib)
+    (name "htslib")
+    (version "1.9")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+                    "https://github.com/samtools/htslib/releases/download/"
+                    version "/htslib-" version ".tar.bz2"))
+              (sha256
+               (base32
+                "16ljv43sc3fxmv63w7b2ff8m1s7h89xhazwmbm1bicz8axq8fjz0"))))))
+
 ;; This package should be removed once no packages rely upon it.
 (define htslib-1.3
   (package
-- 
2.29.2

[bug#44613] [PATCH] Fix build for bedtools

[zimoun]

Hi Roel,

On Fri, 13 Nov 2020 at 12:01, Roel Janssen <roel@gnu.org> wrote:

> With the following patches, I'd like to add samtools-1.9, htslib-1.9
> (samtools depends on that) to fix this problem with bedtools.

Recently, investigating why the substitute of ’python-pysam’ was not
available, I decided then to give a try at fixing the TODO:

--8<---------------cut here---------------start------------->8---
              (snippet '(begin
                          ;; Drop bundled htslib. TODO: Also remove samtools
                          ;; and bcftools.
                          (delete-file-recursively "htslib")
                          #t))))
--8<---------------cut here---------------end--------------->8---

And the bundled version is 1.9 (if I remember correctly), therefore
because of:

--8<---------------cut here---------------start------------->8---
    (native-inputs
     `(("python-cython" ,python-cython)
       ;; Dependencies below are are for tests only.
       ("samtools" ,samtools)
       ("bcftools" ,bcftools)
       ("python-nose" ,python-nose)))
--8<---------------cut here---------------end--------------->8---

some tests are unhappy.

That’s said, these additions seem fine with me. :-)


All the best,
simon

[bug#44613] [PATCH] Fix build for bedtools

[Roel Janssen]

Hi Simon,

On Fri, 2020-11-13 at 13:39 +0100, zimoun wrote:
> Hi Roel,
> 
> On Fri, 13 Nov 2020 at 12:01, Roel Janssen <roel@gnu.org> wrote:
> 
> > With the following patches, I'd like to add samtools-1.9, htslib-
> > 1.9
> > (samtools depends on that) to fix this problem with bedtools.
> 
> Recently, investigating why the substitute of ’python-pysam’ was not
> available, I decided then to give a try at fixing the TODO:
> 
> --8<---------------cut here---------------start------------->8---
>               (snippet '(begin
>                           ;; Drop bundled htslib. TODO: Also remove
> samtools
>                           ;; and bcftools.
>                           (delete-file-recursively "htslib")
>                           #t))))
> --8<---------------cut here---------------end--------------->8---
> 
> And the bundled version is 1.9 (if I remember correctly), therefore
> because of:
> 
> --8<---------------cut here---------------start------------->8---
>     (native-inputs
>      `(("python-cython" ,python-cython)
>        ;; Dependencies below are are for tests only.
>        ("samtools" ,samtools)
>        ("bcftools" ,bcftools)
>        ("python-nose" ,python-nose)))
> --8<---------------cut here---------------end--------------->8---
> 
> some tests are unhappy.
> 
> That’s said, these additions seem fine with me. :-)
> 

I also tried removing the bundled htslib for bedtools, but didn't go
this route for two reasons:
- The bundled htslib for bedtools seems "slightly modified" (I didn't
investigate further)
- Replacing the references to libhts.a with $(pkg-config htslib --
cflags --libs) produced various linker errors. So I stopped right
there.

I'm sure more tools will likely have failed because of the htslib
upgrade (sorry about this!), so having htslib-1.9 around for some time
may be a good fallback for now.

Just to double-check: Is it OK to push the proposed patches?

Kind regards,
Roel Janssen

[bug#44613] [PATCH] Fix build for bedtools

[zimoun]

Hi Roel,

On Fri, 13 Nov 2020 at 13:55, Roel Janssen <roel@gnu.org> wrote:

> I also tried removing the bundled htslib for bedtools, but didn't go
> this route for two reasons:
> - The bundled htslib for bedtools seems "slightly modified" (I didn't
> investigate further)
> - Replacing the references to libhts.a with $(pkg-config htslib --
> cflags --libs) produced various linker errors. So I stopped right
> there.
>
> I'm sure more tools will likely have failed because of the htslib
> upgrade (sorry about this!), so having htslib-1.9 around for some time
> may be a good fallback for now.

Thank for your explanations.


> Just to double-check: Is it OK to push the proposed patches?

I have not tried them but they LGTM.

All the best,
simon

PS:
I am always confused if the removal should be done in ’origin’ or in the
’add-after 'unpack’ phase; especially when the bundle is free software.
Other said, what should an user expect when fetching with “guix build -S”?
Anyway! :-)

bug#44613: [PATCH] Fix build for bedtools

[Roel Janssen]

Hi Simon,

On Fri, 2020-11-13 at 14:34 +0100, zimoun wrote:
> Hi Roel,
> 
> On Fri, 13 Nov 2020 at 13:55, Roel Janssen <roel@gnu.org> wrote:
> 
> > I also tried removing the bundled htslib for bedtools, but didn't
> > go
> > this route for two reasons:
> > - The bundled htslib for bedtools seems "slightly modified" (I
> > didn't
> > investigate further)
> > - Replacing the references to libhts.a with $(pkg-config htslib --
> > cflags --libs) produced various linker errors. So I stopped right
> > there.
> > 
> > I'm sure more tools will likely have failed because of the htslib
> > upgrade (sorry about this!), so having htslib-1.9 around for some
> > time
> > may be a good fallback for now.
> 
> Thank for your explanations.
> 
> 
> > Just to double-check: Is it OK to push the proposed patches?
> 
> I have not tried them but they LGTM.
> 

Thanks for the quick review. I pushed the patches in
c3232fcc7785abc1057a0d4b5b1832f1e39c9c1b,
da4a38edad52f7bb5a8d41465d09f3f0197fd0b7, and
3ede804f6d4c38ff0b9a5705544a8c35f6827ff1.

Kind regards,
Roel Janssen

[bug#44613] [PATCH] Fix build for bedtools

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 734 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> PS:
> I am always confused if the removal should be done in ’origin’ or in the
> ’add-after 'unpack’ phase; especially when the bundle is free software.
> Other said, what should an user expect when fetching with “guix build -S”?
> Anyway! :-)

Unbundling is always better to do in a snippet.  It leads to less
bandwidth usage, and users can more easily inspect the (actual) code.

For other kinds of patching the boundary is less clear.  Generally,
Guix-specific tweaks should be in a phase, but "universal" bug fixes may
well be in a snippet.

I sometimes imagine a downstream distribution that use Guix sources, but
not the build scripts, to draw the line.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

[bug#44613] what “guix build -S” should return?

[zimoun]

Hi Marius,

Thank you for the explanations.


On Fri, 13 Nov 2020 at 16:08, Marius Bakke <marius@gnu.org> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:
>
>> PS:
>> I am always confused if the removal should be done in ’origin’ or in the
>> ’add-after 'unpack’ phase; especially when the bundle is free software.
>> Other said, what should an user expect when fetching with “guix build -S”?
>> Anyway! :-)
>
> Unbundling is always better to do in a snippet.  It leads to less
> bandwidth usage, and users can more easily inspect the (actual) code.

Well, I do not know.  For example, I could do this workflow:

 guix environment bedtools
 tar -xvf $(guix build -S bedtools)
 make

which probably fails because removing the bundles often needs some extra
tweaks.  Concretely, see python-pysam for instance:

--8<---------------cut here---------------start------------->8---
              (snippet '(begin
                          ;; Drop bundled htslib. TODO: Also remove samtools
                          ;; and bcftools.
                          (delete-file-recursively "htslib")
                          #t))))
[...]
       #:phases
       (modify-phases %standard-phases
         (add-before 'build 'set-flags
           (lambda* (#:key inputs #:allow-other-keys)
             (setenv "HTSLIB_MODE" "external")
             (setenv "HTSLIB_LIBRARY_DIR"
                     (string-append (assoc-ref inputs "htslib") "/lib"))
             (setenv "HTSLIB_INCLUDE_DIR"
                     (string-append (assoc-ref inputs "htslib") "/include"))
--8<---------------cut here---------------end--------------->8---

Then, I am not convince that:

  guix build bedtools --with-git-url=http://example.org

works too.  Or ’--with-source=’ as well.  I remember a discussion
initiated by Mark and Maxim about this: snippet vs phases but I am not
able to reach it.


> For other kinds of patching the boundary is less clear.  Generally,
> Guix-specific tweaks should be in a phase, but "universal" bug fixes may
> well be in a snippet.

I agree that non-free and bug fixes should go to snippet.  Then I am
still confused and my feelings are mixed about Guix specific tweaks.


> I sometimes imagine a downstream distribution that use Guix sources, but
> not the build scripts, to draw the line.

It seems a good criteria to draw the line.  And in the case of bedtools
or python-pysam or many others, ’snippet’ removes (free software)
bundles because of an implicit and non-uniform Guix policy that a
downstream distribution could choose differently.

Well, my mind is not clear about this topic. :-)

All the best,
simon