bioinformatics.scm vs bioconductor.scm ?

bioinformatics.scm vs bioconductor.scm ?

[zimoun]

Dear,

Thank you the nice importers.

I am not sure to understand what is the rule to attribute a package to
bioinformatics.scm or to bioconductor.scm when it comes from
Bioconductor.

For example, the package DeSeq2 from Bioconductor is in bioinformatics.scm.
To be concrete,
  $ grep -e bioconductor-uri bioconductor.scm | wc -l
33
  $ grep -e bioconductor-uri bioinformatics.scm  | wc -l
116

Last, there is not alphabetical order of the packages in the files.
How the order is determined ?


Thank you in advance.

All the best,
simon

Re: bioinformatics.scm vs bioconductor.scm ?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> I am not sure to understand what is the rule to attribute a package to
> bioinformatics.scm or to bioconductor.scm when it comes from
> Bioconductor.

bioinformatics.scm was there first.  Later I added bioconductor.scm
because I didn’t want bioinformatics.scm to eventually be full of R
packages.

New Bioconductor packages should go to bioconductor.scm.  Eventually we
may move all remaining R packages from bioinformatics to
bioconductor.scm.

> Last, there is not alphabetical order of the packages in the files.
> How the order is determined ?

Whichever package is added first comes first, later packages are usually
added to the bottom of the file.  The order has no significance.

--
Ricardo

Re: bioinformatics.scm vs bioconductor.scm ?

[zimoun]

Thank you the explanations.


> New Bioconductor packages should go to bioconductor.scm.  Eventually we
> may move all remaining R packages from bioinformatics to
> bioconductor.scm.

I am a bit confused.
The file bioconductor.scm contains (or will contain) all R packages
from Bioconductor, right?
But R packages from CRAN used in Bioinformatics ? bioconductor.scm or
bioinformatics.scm?

And I am asking myself if a massive import from Bioconductor should be
possible ?

Re: bioinformatics.scm vs bioconductor.scm ?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> Thank you the explanations.
>
>
>> New Bioconductor packages should go to bioconductor.scm.  Eventually we
>> may move all remaining R packages from bioinformatics to
>> bioconductor.scm.
>
> I am a bit confused.
> The file bioconductor.scm contains (or will contain) all R packages
> from Bioconductor, right?

Correct.

> But R packages from CRAN used in Bioinformatics ? bioconductor.scm or
> bioinformatics.scm?

Neither :)  We put them in cran.scm.  At least that’s the new way of
doing this.  Previously it was all ad-hoc, meaning that packages would
end up in bioinformatics.scm…

Ideally, bioinformatics.scm would only contain non-R tools like
samtools, bamtools, bioinfo pipelines, etc.

> And I am asking myself if a massive import from Bioconductor should be
> possible ?

Certainly!  I’ve done this before actually, but I hit two minor
problems:

1. the bioconductor recursive importer does not *automatically* switch
   to “CRAN mode” when a dependent package isn’t found on Bioconductor.
   Not a big problem, but it means that teh import isn’t fully
   automatic.

2. compiling big Guile modules (such as a future (gnu packages cran))
   require lots of memory since Guile 2.2(?), so I didn’t add all these
   packages.  This is a bug and we’d have to split the module, probably,
   to work around it.

--
Ricardo

Re: bioinformatics.scm vs bioconductor.scm ?

[zimoun]

Dear Ricardo,

Thank you for your explanations.

> > And I am asking myself if a massive import from Bioconductor should be
> > possible ?
>
> Certainly!  I’ve done this before actually, but I hit two minor
> problems:
>
> 1. the bioconductor recursive importer does not *automatically* switch
>    to “CRAN mode” when a dependent package isn’t found on Bioconductor.
>    Not a big problem, but it means that teh import isn’t fully
>    automatic.

I am not sure to understand.
Is the bioconductor importer usable from `guix import` ?

I have tried once by hand; to understand step by step. :-)
Thank Pierre for the nice tutorial !
see the package definition below.


> 2. compiling big Guile modules (such as a future (gnu packages cran))
>    require lots of memory since Guile 2.2(?), so I didn’t add all these
>    packages.  This is a bug and we’d have to split the module, probably,
>    to work around it.

Ok, even if I have no clue to work around.



So as I need the FlowCore package to process data from cytometry, let
start my first attempt with this one. :-)

This package is on Bioconductor:
https://bioconductor.org/packages/release/bioc/html/flowCore.html
Then define the package by hand was straightforward! :-)
I am not sure to be compliant... Basically, I have just copied/pasted
the modules from bioinformatics.scm (or bioconductor.scm), then I have
look for if the dependency was already there. Missing r-corpcor from
CRAN, so `guix import cran`.

Hum, the package BiocGenerics needs the version >= 0.1.14, and it is
not defined in the package.
Then, the package grDevices, graphics, methods, stats, stats4 are
required (see bioconductor webpage) but not defined elsewhere. Is it
good ?

What is the convention about license ?
(license name) or (license license:name)

Well, then I have run guix package --install-from-file=my.scm and it
seems to works, I mean the first examples from the vignette do not
complain. ;-)


Thank you for any comment.


All the best
simon

--

(define-module (gnu packages my-bioinformatics)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (guix packages)
  #:use-module (guix utils)
  #:use-module (guix download)
  #:use-module (guix git-download)
  #:use-module (guix hg-download)
  #:use-module (guix build-system ant)
  #:use-module (guix build-system gnu)
  #:use-module (guix build-system cmake)
  #:use-module (guix build-system haskell)
  #:use-module (guix build-system ocaml)
  #:use-module (guix build-system perl)
  #:use-module (guix build-system python)
  #:use-module (guix build-system r)
  #:use-module (guix build-system ruby)
  #:use-module (guix build-system scons)
  #:use-module (guix build-system trivial)
  #:use-module (gnu packages)
  #:use-module (gnu packages autotools)
  #:use-module (gnu packages algebra)
  #:use-module (gnu packages base)
  #:use-module (gnu packages bash)
  #:use-module (gnu packages bison)
  #:use-module (gnu packages bioconductor)
  #:use-module (gnu packages bioinformatics)
  #:use-module (gnu packages boost)
  #:use-module (gnu packages check)
  #:use-module (gnu packages compression)
  #:use-module (gnu packages cpio)
  #:use-module (gnu packages cran)
  #:use-module (gnu packages curl)
  #:use-module (gnu packages documentation)
  #:use-module (gnu packages databases)
  #:use-module (gnu packages datastructures)
  #:use-module (gnu packages file)
  #:use-module (gnu packages flex)
  #:use-module (gnu packages gawk)
  #:use-module (gnu packages gcc)
  #:use-module (gnu packages gd)
  #:use-module (gnu packages gtk)
  #:use-module (gnu packages glib)
  #:use-module (gnu packages graph)
  #:use-module (gnu packages groff)
  #:use-module (gnu packages guile)
  #:use-module (gnu packages haskell)
  #:use-module (gnu packages haskell-check)
  #:use-module (gnu packages haskell-web)
  #:use-module (gnu packages image)
  #:use-module (gnu packages imagemagick)
  #:use-module (gnu packages java)
  #:use-module (gnu packages jemalloc)
  #:use-module (gnu packages dlang)
  #:use-module (gnu packages linux)
  #:use-module (gnu packages logging)
  #:use-module (gnu packages machine-learning)
  #:use-module (gnu packages man)
  #:use-module (gnu packages maths)
  #:use-module (gnu packages mpi)
  #:use-module (gnu packages ncurses)
  #:use-module (gnu packages ocaml)
  #:use-module (gnu packages pcre)
  #:use-module (gnu packages parallel)
  #:use-module (gnu packages pdf)
  #:use-module (gnu packages perl)
  #:use-module (gnu packages perl-check)
  #:use-module (gnu packages pkg-config)
  #:use-module (gnu packages popt)
  #:use-module (gnu packages protobuf)
  #:use-module (gnu packages python)
  #:use-module (gnu packages python-web)
  #:use-module (gnu packages readline)
  #:use-module (gnu packages ruby)
  #:use-module (gnu packages serialization)
  #:use-module (gnu packages shells)
  #:use-module (gnu packages statistics)
  #:use-module (gnu packages swig)
  #:use-module (gnu packages tbb)
  #:use-module (gnu packages tex)
  #:use-module (gnu packages texinfo)
  #:use-module (gnu packages textutils)
  #:use-module (gnu packages time)
  #:use-module (gnu packages tls)
  #:use-module (gnu packages vim)
  #:use-module (gnu packages web)
  #:use-module (gnu packages xml)
  #:use-module (gnu packages xorg)
  #:use-module (srfi srfi-1)
  #:use-module (ice-9 match))


(define-public r-corpcor
  (package
   (name "r-corpcor")
   (version "1.6.9")
   (source
    (origin
     (method url-fetch)
     (uri (cran-uri "corpcor" version))
     (sha256
      (base32
       "1hi3i9d3841snppq1ks5pd8cliq1b4rm4dpsczmfqvwksg8snkrf"))))
   (build-system r-build-system)
   (home-page
    "http://strimmerlab.org/software/corpcor/")
   (synopsis
    "Efficient Estimation of Covariance and (Partial) Correlation")
   (description
    "Implements a James-Stein-type shrinkage estimator for the
covariance matrix, with separate shrinkage for variances and
correlations.  The details of the method are explained in Schafer and
Strimmer (2005) <DOI:10.2202/1544-6115.1175> and Opgen-Rhein and
Strimmer (2007) <DOI:10.2202/1544-6115.1252>.  The approach is both
computationally as well as statistically very efficient, it is
applicable to \"small n, large p\" data, and always returns a positive
definite and well-conditioned covariance matrix.  In addition to
inferring the covariance matrix the package also provides shrinkage
estimators for partial correlations and partial variances.  The
inverse of the covariance and correlation matrix can be efficiently
computed, as well as any arbitrary power of the shrinkage correlation
matrix.  Furthermore, functions are available for fast singular value
decomposition, for computing the pseudoinverse, and for checking the
rank and positive definiteness of a matrix.")
   (license license:gpl3+)))


(define-public r-flowcore
  (package
    (name "r-flowcore")
    (version "1.48.0")
    (source
     (origin
       (method url-fetch)
       (uri (bioconductor-uri "flowCore" version))
       (sha256
        (base32
     "16mh3xlrcxkrqvhv3pry325jzsz97yg84ya8rpvd2lvlpqrz6k3h"))))
    (build-system r-build-system)
    (propagated-inputs
     `(
       ("r-biobase" ,r-biobase)
       ("r-biocgenerics" ,r-biocgenerics)
       ("r-biocmanager" ,r-biocmanager)
       ("r-bh" ,r-bh)
       ("r-graph" ,r-graph)
       ("r-rrcov" ,r-rrcov)
       ("r-r-utils" ,r-r-utils)
       ("r-corpcor" ,r-corpcor)
       ("r-rcpp" ,r-rcpp)
       ("r-matrixstats" ,r-matrixstats)
       ("r-mass" ,r-mass)
       ))
    (inputs
     `(("zlib" ,zlib)))
    (home-page "https://bioconductor.org/packages/flowCore")
    (synopsis "Basic structures for flow cytometry data")
    (description
     "Provides S4 data structures and basic functions to deal with
flow cytometry data.")
    (license license:artistic2.0)))


;;BiocGenerics(>= 0.1.14),
;; grDevices,
;; graphics, methods,
;; stats, stats4,


r-flowcore

Re: bioinformatics.scm vs bioconductor.scm ?

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

On Tue, 18 Dec 2018 06:31:44 -0500
zimoun <zimon.toutoune@gmail.com> wrote:

> What is the convention about license ?
> (license name) or (license license:name)

Just about this point: This is not a "convention", this is part of the
language definition of Guile, the underlying Scheme implementation:

In the module gnu/packes/cran.scm (and many others too) you find:

(define-module (gnu packages cran)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (guix packages)
[...]
)

That means: use everything from module "guix licenses" and prefix it
with "license:". So, in the cran module, you must use "license:name" to
use the publicly defined "name" from the "guix licenses" module. 

In other packages that import "guix licenses" without the prefix, you
use "name" directly. See gnu/packages/scsi.scm for an example.

Björn


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: bioinformatics.scm vs bioconductor.scm ?

[zimoun]

Dear,

Thank you for your explanations.
And sorry if I am still slow to understand.

> > What is the convention about license ?
> > (license name) or (license license:name)
>
> Just about this point: This is not a "convention", this is part of the
> language definition of Guile, the underlying Scheme implementation:

I understand this. :-)

>
> In the module gnu/packes/cran.scm (and many others too) you find:
>
> (define-module (gnu packages cran)
>   #:use-module ((guix licenses) #:prefix license:)
>   #:use-module (guix packages)
> [...]
> )
>
> That means: use everything from module "guix licenses" and prefix it
> with "license:". So, in the cran module, you must use "license:name" to
> use the publicly defined "name" from the "guix licenses" module.

Ok, but for example this convention about CRAN is not consistent with
the importer. :-)
  guix import cran corpcor -r
fills the license field with (license gpl3+) and not (license license:gpl3+)

In other words, why the cran.scm needs a prefix for the license field?

> In other packages that import "guix licenses" without the prefix, you
> use "name" directly. See gnu/packages/scsi.scm for an example.

Ok.
But there is a convention or an explanation why some packages use a
prefix e.g. cran.scm and other not e.g scsi.scm?

Thank you again for your explanations.

Best regards,
simon

Re: bioinformatics.scm vs bioconductor.scm ?

[Ricardo Wurmus]

Hi,

> Is the bioconductor importer usable from `guix import` ?

yes.  You may encounter minor problems when using the recursive
bioconductor importer, as it may try to look up CRAN packages on
Bioconductor.

> This package is on Bioconductor:
> https://bioconductor.org/packages/release/bioc/html/flowCore.html

I’d do

    ./pre-inst-env guix import cran -a bioconductor -r flowCore

This fails because it wants corpcor from CRAN.  So we do:

    ./pre-inst-env guix import cran -r corpcor

We dump the result (with minor changes) in (gnu packages cran) and try
again to import flowCore.  This time it succeeds.

> Hum, the package BiocGenerics needs the version >= 0.1.14, and it is
> not defined in the package.

We have r-biocgenerics 0.28.0 in gnu/packages/bioinformatics.scm.
That’s one of the packages that should move eventually.

> Then, the package grDevices, graphics, methods, stats, stats4 are
> required (see bioconductor webpage) but not defined elsewhere. Is it
> good ?

These are all default packages that are part of R itself.  The importer
skips them.

> What is the convention about license ?
> (license name) or (license license:name)

This depends on the target module.  cran.scm, bioinformatics.scm, and
bioconductor.scm all use the “license:” prefix.  web.scm on the other
hand uses the “l:” prefix.  Take a look at the #:use-module clause at
the top of the module.

--
Ricardo

Re: bioinformatics.scm vs bioconductor.scm ?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> Ok, but for example this convention about CRAN is not consistent with
> the importer. :-)
>   guix import cran corpcor -r
> fills the license field with (license gpl3+) and not (license license:gpl3+)

That’s right.  The importer does not know where the generated package
definition is supposed to be used.

> In other words, why the cran.scm needs a prefix for the license field?

It uses a prefix because we use the “zlib” package often, but not the
“zlib” license.  We could exclude the “zlib” license from (guix
licenses), or import only a specified list of licenses, or we can solve
this naming conflict by prefixing all values from (guix licenses) with
“license:” (or anything else, really).

Really small modules often don’t have this problem in the first place,
so they don’t need to find a solution to work around the naming
conflicts.

--
Ricardo

Re: bioinformatics.scm vs bioconductor.scm ?

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

>> Is the bioconductor importer usable from `guix import` ?
>
> yes.  You may encounter minor problems when using the recursive
> bioconductor importer, as it may try to look up CRAN packages on
> Bioconductor.

This is now fixed in commit 10a1cacb1.  The importer will retry the
import from CRAN when the package cannot be found on Bioconductor.

So this works now:

    ./pre-inst-env guix import cran -a bioconductor -r flowCore


-- 
Ricardo