[PATCH] gnu: Add Scikit-learn.

[PATCH] gnu: Add Scikit-learn.

[Ricardo Wurmus]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: 0001-gnu-Add-Scikit-learn.patch --]
[-- Type: text/x-patch, Size: 3325 bytes --]

From d14dbf6e006b29b93152134c724265cb0b65e41f Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
Date: Mon, 23 Feb 2015 17:22:29 +0100
Subject: [PATCH] gnu: Add Scikit-learn.

* gnu/packages/python.scm (python-scikit-learn, python2-scikit-learn): New
  variables.
---
 gnu/packages/python.scm | 65 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index e19d707..88e99d7 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -1960,6 +1960,71 @@ sources.")
 (define-public python2-sphinx-rtd-theme
   (package-with-python2 python-sphinx-rtd-theme))
 
+(define-public python-scikit-learn
+  (package
+    (name "python-scikit-learn")
+    (version "0.15.2")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append
+             "https://github.com/scikit-learn/scikit-learn/archive/"
+             version ".tar.gz"))
+       (sha256
+        (base32
+         "1rb93h2q15f219sz60sczjb65rg70xjmnp0q4pkkaairx5s05q55"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:phases
+       (alist-cons-before
+        'build 'set-environment-variables
+        (lambda* (#:key inputs #:allow-other-keys)
+          (let* ((atlas-threaded
+                  (string-append (assoc-ref inputs "atlas")
+                                 "/lib/libtatlas.so"))
+                 ;; On single core CPUs only the serial library is created.
+                 (atlas-lib
+                  (if (file-exists? atlas-threaded)
+                      atlas-threaded
+                      (string-append (assoc-ref inputs "atlas")
+                                     "/lib/libsatlas.so"))))
+            (setenv "ATLAS" atlas-lib)))
+        (alist-cons-before
+         'check 'set-HOME
+         ;; some tests require access to "$HOME"
+         (lambda _ (setenv "HOME" "/tmp"))
+         ;; Tests can only be run after the library has been installed and not
+         ;; within the source directory.
+         (alist-cons-after
+          'install 'check
+          (lambda _
+            (with-directory-excursion "/tmp"
+              ;; With Python 3 one test of 3334 fails
+              ;; (sklearn.tests.test_common.test_transformers); see
+              ;; https://github.com/scikit-learn/scikit-learn/issues/3693
+              (system* "nosetests" "-v" "sklearn")))
+          (alist-delete 'check %standard-phases))))))
+    (inputs
+     `(("atlas" ,atlas)
+       ("python-numpy" ,python-numpy)
+       ("python-scipy" ,python-scipy)
+       ("python-nose" ,python-nose)))
+    (home-page "http://scikit-learn.org/")
+    (synopsis "Machine Learning in Python")
+    (description
+     "Scikit-learn provides simple and efficient tools for data
+mining and data analysis.")
+    (license bsd-3)))
+
+(define-public python2-scikit-learn
+  (let ((scikit (package-with-python2 python-scikit-learn)))
+    (package (inherit python-scikit-learn)
+      (name "python2-scikit-learn")
+      (propagated-inputs
+       `(("python2-py2cairo" ,python2-py2cairo)
+         ,@(alist-delete "python-pycairo"
+                         (package-propagated-inputs scikit)))))))
+
 (define-public python-cython
   (package
     (name "python-cython")
-- 
2.1.0

Re: [PATCH] gnu: Add Scikit-learn.

[Ricardo Wurmus]

The python2-scikit-learn package isn't working as it should.  It does
not actually install python2.7 libraries but just the 3.3 libs.

Do you have any hints as to how this can be fixed?

I could not just use the package-with-python2 wrapper because of the
propagated "python-pycairo" input that turns into "python2-pycairo"
rather than "python2-py2cairo".  In my patch I addressed this by
explicitly removing "python-pycairo" from the list of propagated inputs
and adding "python2-py2cairo".

Will I also have to explicitly change the arguments such that python-2
is used?  If so, why isn't this done in the following packages:

  - python2-numpy
  - python2-matplotlib
  - python2-scipy

?

~~ Ricardo

Re: [PATCH] gnu: Add Scikit-learn.

[Andreas Enge]

On Mon, Feb 23, 2015 at 06:17:47PM +0100, Ricardo Wurmus wrote:
> The python2-scikit-learn package isn't working as it should.  It does
> not actually install python2.7 libraries but just the 3.3 libs.

I think you did things the wrong way around: You inherit from the python3
package, then swap its inputs against those from the python2-wrapped one.
In this way, the "#:python" field of "arguments" remains at python3.

Instead, you need to inherit from the python2-wrapped package and then
modify its inputs; in this case, this means
   (inherit scikit)
instead of
   (inherit python-scikit-learn)

Have a look at python2-oauthlib (which actually defines too many inherited
fields, but it illustrates the point).

Andreas

Re: [PATCH] gnu: Add Scikit-learn.

[Andreas Enge]

On Mon, Feb 23, 2015 at 06:17:47PM +0100, Ricardo Wurmus wrote:
> I could not just use the package-with-python2 wrapper because of the
> propagated "python-pycairo" input that turns into "python2-pycairo"
> rather than "python2-py2cairo".  In my patch I addressed this by
> explicitly removing "python-pycairo" from the list of propagated inputs
> and adding "python2-py2cairo".

I wonder how to handle this case. Probably we should rename python2-py2cairo
to python2-pycairo. The include files and the pkg-config files speak only of
"pycairo", not "py2cairo". So it looks like this is also a case where the
tarball name differs from the module name.

What do the specialists think?

Andreas

Re: [PATCH] gnu: Add Scikit-learn.

[Andreas Enge]

On Mon, Feb 23, 2015 at 06:52:38PM +0100, Andreas Enge wrote:
> I wonder how to handle this case. Probably we should rename python2-py2cairo
> to python2-pycairo. The include files and the pkg-config files speak only of
> "pycairo", not "py2cairo". So it looks like this is also a case where the
> tarball name differs from the module name.

Actually, the python module is named "cairo", not "pycairo" or "py2cairo":
   /gnu/store/nql1qlyhy4dbhq260vybp1523lyn2gsz-python2-py2cairo-1.10.0/lib/python2.7/site-packages/cairo

So we should rename python-pycairo to python-cairo, and python2-py2cairo
to python2-cairo.

Andreas

Re: [PATCH] gnu: Add Scikit-learn.

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 106 bytes --]

And attached a patch to the packaging guidelines to clarify how to name
python module packages.

Andreas


[-- Attachment #2: 0001-doc-Clarify-wording-for-naming-of-python-module-pack.patch --]
[-- Type: text/plain, Size: 1619 bytes --]

From 9f44c7a847fb9122efc546ae4ad73e53f4fb9214 Mon Sep 17 00:00:00 2001
From: Andreas Enge <andreas@enge.fr>
Date: Wed, 25 Feb 2015 23:55:07 +0100
Subject: [PATCH] doc: Clarify wording for naming of python module packages.

* doc/guix.texi (Python Modules): Make clear that the name of a python
  module package is to be derived from the module name.
---
 doc/guix.texi | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 81b9353..2f6f35e 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -2477,7 +2477,7 @@ To illustrate the idea, here is an example of a gexp:
   #~(begin
       (mkdir #$output)
       (chdir #$output)
-      (symlink (string-append #$coreutils "/bin/ls") 
+      (symlink (string-append #$coreutils "/bin/ls")
                "list-files")))
 @end example
 
@@ -5521,12 +5521,12 @@ seems desirable that the name of a package for a Python module contains
 the word @code{python}.
 
 Some modules are compatible with only one version of Python, others with both.
-If the package Foo compiles only with Python 3, we name it
+If the module Foo compiles only with Python 3, we name it
 @code{python-foo}; if it compiles only with Python 2, we name it
 @code{python2-foo}. If it is compatible with both versions, we create two
 packages with the corresponding names.
 
-If a project already contains the word @code{python}, we drop this;
+If a module name already contains the word @code{python}, we drop this;
 for instance, the module python-dateutil is packaged under the names
 @code{python-dateutil} and @code{python2-dateutil}.
 
-- 
2.2.1

Re: [PATCH] gnu: Add Scikit-learn.

[Ricardo Wurmus]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]


Andreas Enge writes:

> On Mon, Feb 23, 2015 at 06:17:47PM +0100, Ricardo Wurmus wrote:
>> The python2-scikit-learn package isn't working as it should.  It does
>> not actually install python2.7 libraries but just the 3.3 libs.
>
> I think you did things the wrong way around: You inherit from the python3
> package, then swap its inputs against those from the python2-wrapped one.
> In this way, the "#:python" field of "arguments" remains at python3.

Yeah, I screwed up.  Attached is a better version that actually works.
I explicitly delete python-scipy and python-numpy, both of which have
matplotlib (and thus the problematic python-pycairo) as inputs, from the
propagated inputs and add their python2 versions manually.

~~ Ricardo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Add-Scikit-learn.patch --]
[-- Type: text/x-patch, Size: 3381 bytes --]

From bc1f142127c734342cd308476ce9a7540f8326ec Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
Date: Mon, 23 Feb 2015 17:22:29 +0100
Subject: [PATCH] gnu: Add Scikit-learn.

* gnu/packages/python.scm (python-scikit-learn, python2-scikit-learn): New
  variables.
---
 gnu/packages/python.scm | 68 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 6515237..7128182 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -1952,6 +1952,74 @@ sources.")
 (define-public python2-sphinx-rtd-theme
   (package-with-python2 python-sphinx-rtd-theme))
 
+(define-public python-scikit-learn
+  (package
+    (name "python-scikit-learn")
+    (version "0.15.2")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append
+             "https://github.com/scikit-learn/scikit-learn/archive/"
+             version ".tar.gz"))
+       (sha256
+        (base32
+         "1rb93h2q15f219sz60sczjb65rg70xjmnp0q4pkkaairx5s05q55"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:phases
+       (alist-cons-before
+        'build 'set-environment-variables
+        (lambda* (#:key inputs #:allow-other-keys)
+          (let* ((atlas-threaded
+                  (string-append (assoc-ref inputs "atlas")
+                                 "/lib/libtatlas.so"))
+                 ;; On single core CPUs only the serial library is created.
+                 (atlas-lib
+                  (if (file-exists? atlas-threaded)
+                      atlas-threaded
+                      (string-append (assoc-ref inputs "atlas")
+                                     "/lib/libsatlas.so"))))
+            (setenv "ATLAS" atlas-lib)))
+        (alist-cons-before
+         'check 'set-HOME
+         ;; some tests require access to "$HOME"
+         (lambda _ (setenv "HOME" "/tmp"))
+         ;; Tests can only be run after the library has been installed and not
+         ;; within the source directory.
+         (alist-cons-after
+          'install 'check
+          (lambda _
+            (with-directory-excursion "/tmp"
+              ;; With Python 3 one test of 3334 fails
+              ;; (sklearn.tests.test_common.test_transformers); see
+              ;; https://github.com/scikit-learn/scikit-learn/issues/3693
+              (system* "nosetests" "-v" "sklearn")))
+          (alist-delete 'check %standard-phases))))))
+    (inputs
+     `(("atlas" ,atlas)
+       ("python-nose" ,python-nose)))
+    (propagated-inputs
+     `(("python-numpy" ,python-numpy)
+       ("python-scipy" ,python-scipy)))
+    (home-page "http://scikit-learn.org/")
+    (synopsis "Machine Learning in Python")
+    (description
+     "Scikit-learn provides simple and efficient tools for data
+mining and data analysis.")
+    (license bsd-3)))
+
+(define-public python2-scikit-learn
+  (let ((scikit (package-with-python2 python-scikit-learn)))
+    (package (inherit scikit)
+      (propagated-inputs
+       `(("python2-numpy" ,python2-numpy)
+         ("python2-scipy" ,python2-scipy)
+         ,@(alist-delete
+            "python-numpy"
+            (alist-delete
+             "python-scipy" (package-propagated-inputs scikit))))))))
+
 (define-public python-cython
   (package
     (name "python-cython")
-- 
2.1.0

Re: [PATCH] gnu: Add Scikit-learn.

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

>  Some modules are compatible with only one version of Python, others with both.
> -If the package Foo compiles only with Python 3, we name it
> +If the module Foo compiles only with Python 3, we name it
>  @code{python-foo}; if it compiles only with Python 2, we name it
>  @code{python2-foo}. If it is compatible with both versions, we create two
>  packages with the corresponding names.
>  
> -If a project already contains the word @code{python}, we drop this;
> +If a module name already contains the word @code{python}, we drop this;
>  for instance, the module python-dateutil is packaged under the names
>  @code{python-dateutil} and @code{python2-dateutil}.

I think a Python “module” is a single .py file.  In that case “package”
seems more accurate, no?

Ludo’.

Re: [PATCH] gnu: Add Scikit-learn.

[Andreas Enge]

On Thu, Feb 26, 2015 at 06:11:03PM +0100, Ludovic Courtès wrote:
> I think a Python “module” is a single .py file.  In that case “package”
> seems more accurate, no?

Indeed, that is true. Still I am looking for a justification to rename
"python2-py2cairo" to "python2-cairo", that is, use the module name for
a package containing a single module. Below is a proposed new text for
this section (more readable than a patch), which also replaces another
occurrence of "module" by "package".

What do you think?

Andreas


@node Python Modules
@subsection Python Modules

We currently package Python 2 and Python 3, under the Scheme variable names
@code{python-2} and @code{python} as explained in @ref{Version Numbers}.
Python programs standing for themselves are named as any other package,
using the lowercase upstream name.

To avoid confusion and naming clashes with other programming languages, it
seems desirable that the name of a package for Python modules contains
the word @code{python}.
As base name, we use the lowercase module name for Python packages
containing a single module, or the lowercase upstream package name
otherwise; from this, we drop all occurrences of the word @code{python}.
Some packages are compatible with only one version of Python, others with
both.
If the package with base name @code{foo} compiles only with Python 3, we name
it @code{python-foo}; if it compiles only with Python 2, we name it
@code{python2-foo}. If it is compatible with both versions, we create two
packages with the corresponding names.

For instance, the upstream package python-dateutil is packaged under the
names @code{python-dateutil} and @code{python2-dateutil}.

Re: [PATCH] gnu: Add Scikit-learn.

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> As base name, we use the lowercase module name for Python packages
> containing a single module, or the lowercase upstream package name
> otherwise; from this, we drop all occurrences of the word @code{python}.

Would be good to add the python-cairo example here to clarify it, but
other than that it LGTM.

Thanks,
Ludo’.

Re: [PATCH] gnu: Add Scikit-learn.

[Andreas Enge]

On Thu, Feb 26, 2015 at 07:20:55PM +0100, Ludovic Courtès wrote:
> Would be good to add the python-cairo example here to clarify it, but
> other than that it LGTM.

Okay, will do.

Andreas

Re: [PATCH] gnu: Add Scikit-learn.

[Eric Bavier]

On 2015-02-26 11:59, Andreas Enge wrote:
> On Thu, Feb 26, 2015 at 06:11:03PM +0100, Ludovic Courtès wrote:
>> I think a Python “module” is a single .py file.  In that case 
>> “package”
>> seems more accurate, no?
> 
> Indeed, that is true. Still I am looking for a justification to rename
> "python2-py2cairo" to "python2-cairo"

The issue in this case is that py2cairo and pycairo are actually 
different projects.  I chose to use the name python2-py2cairo, to go 
with our "least divergence from upstream" method.

> that is, use the module name for
> a package containing a single module.

This could become error-prone.  The packager needs to check the 
interface of the library before deciding on a package name?  It might 
also not be future-proof, in the event that a project updates its public 
interface.

-- 
`~Eric

Re: [PATCH] gnu: Add Scikit-learn.

[Andreas Enge]

On Thu, Feb 26, 2015 at 02:03:37PM -0600, Eric Bavier wrote:
> The issue in this case is that py2cairo and pycairo are actually different
> projects.  I chose to use the name python2-py2cairo, to go with our "least
> divergence from upstream" method.

Both seem to belong to the "pycairo" project:
   http://cairographics.org/pycairo/
which produces two different tarballs, one for Python 2 and one for Python 3,
that are released with the same version.

So I think it would make sense to in any case choose a common base name, be it
"pycairo". That would also avoid the problems with dependencies that Ricardo
faced. We recently discussed that we can choose the "project name" over
the "tarball name" in another example.

> >that is, use the module name for
> >a package containing a single module.
> This could become error-prone.  The packager needs to check the interface of
> the library before deciding on a package name?  It might also not be
> future-proof, in the event that a project updates its public interface.

Indeed, the new rule would require to have a look at the output of a package
to count the number of modules... I copied it from the Perl case; are things
different there, and each module is distributed in a separate tarball for Perl?

I think the question of being future proof is not such a big problem, we can
always modify package names (with a bit of work, of course).

So you would suggest to always use the "project name" and not the "module
name" for Python packages?

Andreas

Re: [PATCH] gnu: Add Scikit-learn.

[Andreas Enge]

Hello,

On Thu, Feb 26, 2015 at 12:28:10PM +0100, Ricardo Wurmus wrote:
> Attached is a better version that actually works.
> I explicitly delete python-scipy and python-numpy, both of which have
> matplotlib (and thus the problematic python-pycairo) as inputs, from the
> propagated inputs and add their python2 versions manually.

maybe wait a little bit until we sort out the final name of
python(2)-py(2)cairo; hopefully we will find a solution that simplifies
the packaging task.

Andreas

Re: [PATCH] gnu: Add Scikit-learn.

[Eric Bavier]

On 2015-02-26 14:43, Andreas Enge wrote:
> On Thu, Feb 26, 2015 at 02:03:37PM -0600, Eric Bavier wrote:
>> The issue in this case is that py2cairo and pycairo are actually 
>> different
>> projects.  I chose to use the name python2-py2cairo, to go with our 
>> "least
>> divergence from upstream" method.
> 
> Both seem to belong to the "pycairo" project:
>    http://cairographics.org/pycairo/
> which produces two different tarballs, one for Python 2 and one for 
> Python 3,
> that are released with the same version.
> 
> So I think it would make sense to in any case choose a common base 
> name, be it
> "pycairo". That would also avoid the problems with dependencies that 
> Ricardo
> faced. We recently discussed that we can choose the "project name" over
> the "tarball name" in another example.

Yes, that makes sense.

> 
>> >that is, use the module name for
>> >a package containing a single module.
>> This could become error-prone.  The packager needs to check the 
>> interface of
>> the library before deciding on a package name?  It might also not be
>> future-proof, in the event that a project updates its public 
>> interface.
> 
> Indeed, the new rule would require to have a look at the output of a 
> package
> to count the number of modules... I copied it from the Perl case; are 
> things
> different there, and each module is distributed in a separate tarball 
> for Perl?

No, perl has what are called "distributions".  Many distributions 
contain a single module and possibly submodules, but in many cases there 
are more than one module in a distribution.  Our perl packages are named 
after the distribution.

> I think the question of being future proof is not such a big problem, 
> we can
> always modify package names (with a bit of work, of course).
> 
> So you would suggest to always use the "project name" and not the 
> "module
> name" for Python packages?

Yes.

-- 
`~Eric

Re: [PATCH] gnu: Add Scikit-learn.

[Andreas Enge]

On Thu, Feb 26, 2015 at 03:10:51PM -0600, Eric Bavier wrote:
> On 2015-02-26 14:43, Andreas Enge wrote:
> >So you would suggest to always use the "project name" and not the "module
> >name" for Python packages?
> Yes.

Okay, you are definitely more knowledgeable in that matter, so I would like
to follow your advice. I will add a sentence stating that one should thrive
to choose a common base name independently of the tarball name.

So let us rename python(2)-py(2)cairo to python(2)-pycairo. If nobody beats
me to it, I can do it tomorrow.

Andreas

Re: [PATCH] gnu: Add Scikit-learn.

[Andreas Enge]

On Fri, Feb 27, 2015 at 10:41:51AM +0100, Andreas Enge wrote:
> So let us rename python(2)-py(2)cairo to python(2)-pycairo. If nobody beats
> me to it, I can do it tomorrow.

Well, if I am not mistaken, renaming does not solve the problem really. I had
forgotten how 'package-with-python2' works: It does not replace the input
'python-foo' with 'python2-foo', but with '(package-with-python2 python-foo)'
(and this recursively).

Now, 'python2-pycairo' is not the same as '(package-with-python2 python-pycairo)'
(in particular the source fields are changed).

This means that if one has a package with 'python-pycairo' as input, then
its python2 version cannot simply be created using 'package-with-python2'.
And worse, if one has a package with input a package with input 'python-pycairo',
one needs to unravel even one level more. And so on. This is very unpleasant.
(See the definition of 'python2-matplotlib', in which some inputs are rewritten
by hand. And 'python2-scipy' inherits the problem, since it has as input
'python2-matplotlib'.) This has actually been discussed in November
on the list...

Now, as also discussed before, we cannot look at variable names and just
rewrite them syntactically. But can we adapt the package definition to depend
on the 'name' field, which changes in a predictible way?

For instance: 'python2-pycairo' differs from '(package-with-python2 python-pycairo)'
only in the 'source' field (and the 'license' field, which is a minor point
for the sake of building). So could we redefine the source depending on the
'name' field, which is 'python-pycairo' in one case and 'python2-pycairo'
in the other? Can we use in the 'source' field something like
(if (string=? name "python-pycairo"
              source-x
              source-y))
?
I assume that yes - after all, we already use the 'version' field in the
'source' field. This can only work if the fields are evaluated in the
right order. Is it the order in the definition of the 'package' record?

Andreas

Re: [PATCH] gnu: Add Scikit-learn.

[Andreas Enge]

On Fri, Feb 27, 2015 at 11:48:34PM +0100, Andreas Enge wrote:
> Can we use in the 'source' field something like
> (if (string=? name "python-pycairo"
>               source-x
>               source-y))

We can, but it does not work. The second package inherits not the code to
construct the source, but the already constructed source itself.
One would need a more complicated construction, like putting into this field
a procedure that is evaluated at build time and not at package creation. But
this would modify the basics of our package format (and be rather complicated).

Andreas