[PATCH 1/2] gnu: Add numpy

[PATCH 1/2] gnu: Add numpy

[Federico Beffa]

[-- Attachment #1: Type: text/plain, Size: 59 bytes --]

Here the first packages depending on ATLAS.

Regards,
Fede

[-- Attachment #2: 0001-gnu-Add-numpy.patch --]
[-- Type: text/x-patch, Size: 5578 bytes --]

From 063b33562c4fee2ea4e3fd1b53a27770047e9201 Mon Sep 17 00:00:00 2001
From: Federico Beffa <beffa@fbengineering.ch>
Date: Sun, 26 Oct 2014 15:03:17 +0100
Subject: [PATCH 1/2] gnu: Add numpy

* gnu/packages/python.scm(python-numpy, python2-numpy): New variables.
---
 gnu/packages/python.scm | 112 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 112 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index c995c1b..b39957a 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -37,6 +37,8 @@
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages databases)
   #:use-module (gnu packages zip)
+  #:use-module (gnu packages maths)
+  #:use-module (gnu packages gcc)
   #:use-module (gnu packages multiprecision)
   #:use-module (guix packages)
   #:use-module (guix download)
@@ -1832,3 +1834,113 @@ sources.")
 
 (define-public python2-sphinx
   (package-with-python2 python-sphinx))
+
+(define python-numpy-reference
+  (origin
+    (method url-fetch)
+    (uri (string-append "mirror://sourceforge/numpy/reference.pdf"))
+    (sha256 
+     (base32
+      "0izsmzazhd042ra6p7as2jhwx8713sc5gpn9s5nifwm68biwn0lp"))))
+
+(define python-numpy-userguide
+  (origin
+    (method url-fetch)
+    (uri (string-append "mirror://sourceforge/numpy/userguide.pdf"))
+    (sha256 
+     (base32
+      "1z2xjlxldv01ad2775k9birg1hkqay5wbi5vmgz3rlr6dczvk77k"))))
+
+(define-public python-numpy
+  (package
+    (name "python-numpy")
+    (version "1.9.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append "mirror://sourceforge/numpy"
+                           "/numpy-" version ".tar.gz"))
+       (sha256
+        (base32
+         "1xbzsb84wg32zx5ilgvkdp0lwfqrycwhh5a2q9nx5iwyjpgcxrpp"))))
+    (build-system python-build-system)
+    (inputs
+     `(("python" ,python) ; otherwise ld does not find libpython3.3m
+       ("python-numpy-userguide" ,python-numpy-userguide)
+       ("python-numpy-reference" ,python-numpy-reference)
+       ("atlas" ,atlas)))
+    (native-inputs
+     `(("python-nose" ,python-nose)  ; used by tests
+       ("gfortran" ,gfortran-4.8)))
+    (outputs '("out" "doc"))         ; doc is ca. 9MB
+    (arguments
+     `(#:phases
+       (alist-replace
+        'build
+        (lambda* (#:key outputs inputs 
+                  #:allow-other-keys #:rest args)
+          (let* ((build (assoc-ref %standard-phases 'build))
+                 (atlas-threaded
+                  (string-append (assoc-ref inputs "atlas") 
+                                 "/lib/libtatlas.so"))
+                 ;; On single core CPUs only the serial library is created.
+                 (atlas-lib
+                  (if (file-exists? atlas-threaded)
+                      atlas-threaded
+                      (string-append (assoc-ref inputs "atlas") 
+                                     "/lib/libsatlas.so"))))
+            (setenv "ATLAS" atlas-lib)
+            (apply build args)))
+        (alist-cons-after
+         'install 'install-doc
+         (lambda* (#:key outputs inputs #:allow-other-keys)
+           ;; procedure from gnu-build-system.scm
+           (define (package-name)
+             (let* ((out  (assoc-ref outputs "out"))
+                    (base (basename out))
+                    (dash (string-rindex base #\-)))
+               (string-drop (if dash
+                                (substring base 0 dash)
+                                base)
+                            (+ 1 (string-index base #\-)))))
+
+           (let ((doc (string-append (assoc-ref outputs "doc") 
+                                     "/share/doc/" (package-name))))
+             (mkdir-p doc)
+             (and (zero? 
+                   (system* "cp" 
+                            (assoc-ref inputs "python-numpy-reference") 
+                            (string-append doc "/reference.pdf")))
+                  (zero? 
+                   (system* "cp" 
+                            (assoc-ref inputs "python-numpy-userguide")
+                            (string-append doc "/userguide.pdf"))))))
+         ;; Tests can only be run after the library has been installed and not
+         ;; within the source directory.
+         (alist-cons-after
+          'install 'check
+          (lambda _ 
+            (with-directory-excursion "/tmp"
+              (zero? (system* "python" "-c" "import numpy; numpy.test()"))))
+          (alist-delete 
+           'check 
+           %standard-phases))))))
+    (home-page "http://www.numpy.org/")
+    (synopsis "Fundamental package for scientific computing with Python")
+    (description "NumPy is the fundamental package for scientific computing
+with Python. It contains among other things: a powerful N-dimensional array
+object, sophisticated (broadcasting) functions, tools for integrating C/C++
+and Fortran code, useful linear algebra, Fourier transform, and random number
+capabilities.")
+    (license bsd-3)))
+
+(define-public python2-numpy
+  (package (inherit (package-with-python2 python-numpy))
+           ;; If we don't redefine the inputs here, then python (version 3) is
+           ;; also imported and its libraries come first in PYTHONPATH.  This
+           ;; causes the 'check phase to fail.
+           (inputs
+            `(("python-2" ,python-2) ; otherwise ld does not find libpython2.7
+              ("python-numpy-userguide" ,python-numpy-userguide)
+              ("python-numpy-reference" ,python-numpy-reference)
+              ("atlas" ,atlas)))))
-- 
1.8.4

Re: [PATCH 1/2] gnu: Add numpy

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1325 bytes --]

Federico Beffa <beffa@ieee.org> skribis:

> From 063b33562c4fee2ea4e3fd1b53a27770047e9201 Mon Sep 17 00:00:00 2001
> From: Federico Beffa <beffa@fbengineering.ch>
> Date: Sun, 26 Oct 2014 15:03:17 +0100
> Subject: [PATCH 1/2] gnu: Add numpy
>
> * gnu/packages/python.scm(python-numpy, python2-numpy): New variables.
                           ^
Missing space.

> +(define python-numpy-reference
> +  (origin
> +    (method url-fetch)
> +    (uri (string-append "mirror://sourceforge/numpy/reference.pdf"))
> +    (sha256 
> +     (base32
> +      "0izsmzazhd042ra6p7as2jhwx8713sc5gpn9s5nifwm68biwn0lp"))))
> +
> +(define python-numpy-userguide
> +  (origin
> +    (method url-fetch)
> +    (uri (string-append "mirror://sourceforge/numpy/userguide.pdf"))
> +    (sha256 
> +     (base32
> +      "1z2xjlxldv01ad2775k9birg1hkqay5wbi5vmgz3rlr6dczvk77k"))))

Can these manuals be built from source?

If they can’t, then that’s non-free documentation.

If they can but the process is tedious, then it’s OK to leave it that
way with a TODO, and also a comment stating what its license is.

> +    (inputs
> +     `(("python" ,python) ; otherwise ld does not find libpython3.3m

This is because Python is not added to ‘LIBRARY_PATH’, right?

I think this is fixed by this patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 624 bytes --]

--- a/guix/build-system/python.scm
+++ b/guix/build-system/python.scm
@@ -110,10 +110,11 @@ prepended to the name."
                               '())
                         ,@inputs
 
+                        ("python" ,python)
+
                         ;; Keep the standard inputs of 'gnu-build-system'.
                         ,@(standard-packages)))
-         (build-inputs `(("python" ,python)
-                         ,@native-inputs))
+         (build-inputs native-inputs)
          (outputs outputs)
          (build python-build)
          (arguments (strip-keyword-arguments private-keywords arguments)))))

[-- Attachment #3: Type: text/plain, Size: 2872 bytes --]


Can you confirm?

> +       ("python-numpy-userguide" ,python-numpy-userguide)
> +       ("python-numpy-reference" ,python-numpy-reference)
> +       ("atlas" ,atlas)))

As discussed in the other thread, this should probably be the generic
(unoptimized) ATLAS here.

> +       (alist-replace
> +        'build
> +        (lambda* (#:key outputs inputs 
> +                  #:allow-other-keys #:rest args)
> +          (let* ((build (assoc-ref %standard-phases 'build))

Use ‘alist-cons-before’ instead of calling the original ‘build’ phase (I
think this was discussed before and there was rough consensus on that.)

> +        (alist-cons-after
> +         'install 'install-doc
> +         (lambda* (#:key outputs inputs #:allow-other-keys)
> +           ;; procedure from gnu-build-system.scm
> +           (define (package-name)
> +             (let* ((out  (assoc-ref outputs "out"))
> +                    (base (basename out))
> +                    (dash (string-rindex base #\-)))
> +               (string-drop (if dash
> +                                (substring base 0 dash)
> +                                base)
> +                            (+ 1 (string-index base #\-)))))
> +
> +           (let ((doc (string-append (assoc-ref outputs "doc") 
> +                                     "/share/doc/" (package-name))))

Just use (string-append ,name "-" ,version) so the ‘package-name’
procedure isn’t needed.

> +             (and (zero? 
> +                   (system* "cp" 
> +                            (assoc-ref inputs "python-numpy-reference") 
> +                            (string-append doc "/reference.pdf")))
> +                  (zero? 
> +                   (system* "cp" 
> +                            (assoc-ref inputs "python-numpy-userguide")
> +                            (string-append doc "/userguide.pdf"))))))

Use the ‘copy-file’ procedure instead of calling ‘cp’.

> +         ;; Tests can only be run after the library has been installed and not
> +         ;; within the source directory.
> +         (alist-cons-after
> +          'install 'check
> +          (lambda _ 
> +            (with-directory-excursion "/tmp"
> +              (zero? (system* "python" "-c" "import numpy; numpy.test()"))))
> +          (alist-delete 
> +           'check 
> +           %standard-phases))))))

Just (alist-replace 'check ...) ?

> +(define-public python2-numpy
> +  (package (inherit (package-with-python2 python-numpy))
> +           ;; If we don't redefine the inputs here, then python (version 3) is
> +           ;; also imported and its libraries come first in PYTHONPATH.  This
> +           ;; causes the 'check phase to fail.

Normally this will no longer be needed if the patch above solves the
problem.

Thanks!

Ludo’.

Re: [PATCH 1/2] gnu: Add numpy

[Federico Beffa]

On Sun, Oct 26, 2014 at 7:56 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Can these manuals be built from source?

I get error messages that matplotlib is missing.  I started looking at
matplotlib as well, but I've found that, for the TkAgg back-end, it
needs TKinter which is part of the standard python libraries and it is
built if, during the build phase, tk is available. The current python
package does not build that library (no tk).  In fact, I would like to
add it to be able to move forward with matplotlib :-)

For the other back-ends some packages are missing (python-gtk, Qt, ...)

Another annoying problem is that matplotlib needs numpy, thereby
creating a dependency loop (although only for the documentation).

> If they can but the process is tedious, then it’s OK to leave it that
> way with a TODO, and also a comment stating what its license is.

I've not seen any license statement specific for the documentation.
But, given that the source is distributed with the code, can't we just
assume it is the same license?

>> +    (inputs
>> +     `(("python" ,python) ; otherwise ld does not find libpython3.3m
>
> This is because Python is not added to ‘LIBRARY_PATH’, right?
>
> I think this is fixed by this patch:
>
>
>
> Can you confirm?

Actually it does not.  The "...-python-numpy-1.9.0-guile-builder"
still shows that only the python-wrapper is imported. The wrapper does
not include the libraries.

> As discussed in the other thread, this should probably be the generic
> (unoptimized) ATLAS here.

I think that as a temporary situation this is OK.  However, in my
opinion it does not make much sense to have a separate sub-optimal
package.  I would like to propose to add specific flags to a single
atlas package and, once the new "local-build" flag will be available,
we will remove those flags.

>
>> +       (alist-replace
>> +        'build
>> +        (lambda* (#:key outputs inputs
>> +                  #:allow-other-keys #:rest args)
>> +          (let* ((build (assoc-ref %standard-phases 'build))
>
> Use ‘alist-cons-before’ instead of calling the original ‘build’ phase (I
> think this was discussed before and there was rough consensus on that.)

OK.

> Just use (string-append ,name "-" ,version) so the ‘package-name’
> procedure isn’t needed.

that's much better :-)

> Use the ‘copy-file’ procedure instead of calling ‘cp’.

The reason that made me use 'cp' instead of 'copy-file' is that the
documentation states that the return value of the latter is not
specified and I'm still not familiar with Guile exception handling.
However, I see many places where 'copy-file' is used without any
'catch'.  In principle this should never fail.  If it is OK, I will do
the same.

>
>> +         ;; Tests can only be run after the library has been installed and not
>> +         ;; within the source directory.
>> +         (alist-cons-after
>> +          'install 'check
>> +          (lambda _
>> +            (with-directory-excursion "/tmp"
>> +              (zero? (system* "python" "-c" "import numpy; numpy.test()"))))
>> +          (alist-delete
>> +           'check
>> +           %standard-phases))))))
>
> Just (alist-replace 'check ...) ?

Actually the check phase needs to be moved after the install one (see
comment).  That's the reason for this two step approach.

Thanks for the comments.
Regards,
Fede

Re: [PATCH 1/2] gnu: Add numpy

[Ludovic Courtès]

Federico Beffa <beffa@ieee.org> skribis:

> On Sun, Oct 26, 2014 at 7:56 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>> Can these manuals be built from source?
>
> I get error messages that matplotlib is missing.  I started looking at
> matplotlib as well, but I've found that, for the TkAgg back-end, it
> needs TKinter which is part of the standard python libraries and it is
> built if, during the build phase, tk is available. The current python
> package does not build that library (no tk).  In fact, I would like to
> add it to be able to move forward with matplotlib :-)

Heh, OK.  Then can you just summarize that in a comment with TODO, to
give the reader an incentive to look at it?  :-)

> For the other back-ends some packages are missing (python-gtk, Qt, ...)
>
> Another annoying problem is that matplotlib needs numpy, thereby
> creating a dependency loop (although only for the documentation).

If it’s only to build the documentation, we should be able to handle it
easily (by first building a documentation-less variant of numpy, for
instance.)

>> If they can but the process is tedious, then it’s OK to leave it that
>> way with a TODO, and also a comment stating what its license is.
>
> I've not seen any license statement specific for the documentation.
> But, given that the source is distributed with the code, can't we just
> assume it is the same license?

Yes.

>>> +    (inputs
>>> +     `(("python" ,python) ; otherwise ld does not find libpython3.3m
>>
>> This is because Python is not added to ‘LIBRARY_PATH’, right?
>>
>> I think this is fixed by this patch:
>>
>>
>>
>> Can you confirm?
>
> Actually it does not.  The "...-python-numpy-1.9.0-guile-builder"
> still shows that only the python-wrapper is imported. The wrapper does
> not include the libraries.

Ah right.  And what if you again remove Python from ‘inputs’, and add

  #:python ,python

to the arguments?

That means it will use the actual Python 3.x package, not the wrapper,
so everything will be visible.  The downside is that there will be no
‘python’ command, only ‘python3’.

Perhaps the right fix will be to change ‘python-wrapper’ to symlink the
‘lib’ sub-directory of ‘python’.

>> As discussed in the other thread, this should probably be the generic
>> (unoptimized) ATLAS here.
>
> I think that as a temporary situation this is OK.  However, in my
> opinion it does not make much sense to have a separate sub-optimal
> package.  I would like to propose to add specific flags to a single
> atlas package and, once the new "local-build" flag will be available,
> we will remove those flags.

Yeah.

After some more thought, I’ve finally bit the bullet:

  1. Commit 77b0ac9 adds the #:substitutable? flag to gnu-build-system.

  2. Commit f15615b uses it for ATLAS.

It may still make sense to have the non-optimized version for those who
do not need performance and do not want to build locally maybe, WDYT?

>> Use the ‘copy-file’ procedure instead of calling ‘cp’.
>
> The reason that made me use 'cp' instead of 'copy-file' is that the
> documentation states that the return value of the latter is not
> specified and I'm still not familiar with Guile exception handling.
> However, I see many places where 'copy-file' is used without any
> 'catch'.  In principle this should never fail.  If it is OK, I will do
> the same.

‘copy-file’ will throw an exception if something goes wrong; since the
exception is uncaught, the build process will fail.

That’s usually the behavior that we want, so I think it’s best.

>>> +         ;; Tests can only be run after the library has been installed and not
>>> +         ;; within the source directory.
>>> +         (alist-cons-after
>>> +          'install 'check
>>> +          (lambda _
>>> +            (with-directory-excursion "/tmp"
>>> +              (zero? (system* "python" "-c" "import numpy; numpy.test()"))))
>>> +          (alist-delete
>>> +           'check
>>> +           %standard-phases))))))
>>
>> Just (alist-replace 'check ...) ?
>
> Actually the check phase needs to be moved after the install one (see
> comment).  That's the reason for this two step approach.

Ah OK.

Thanks!

Ludo’.

Re: [PATCH 1/2] gnu: Add numpy

[Federico Beffa]

On Tue, Oct 28, 2014 at 10:34 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> Ah right.  And what if you again remove Python from ‘inputs’, and add
>
>   #:python ,python
>
> to the arguments?
>
> That means it will use the actual Python 3.x package, not the wrapper,
> so everything will be visible.  The downside is that there will be no
> ‘python’ command, only ‘python3’.

Well, as you say, it does not find the 'python' command and stops with
an error.  I think that fixing the command name is even worse that
having python as input.

>
> Perhaps the right fix will be to change ‘python-wrapper’ to symlink the
> ‘lib’ sub-directory of ‘python’.

It was also suggested to make python a propagating input of the wrapper.
https://lists.gnu.org/archive/html/guix-devel/2014-10/msg00303.html

In my opinion, one of these two fixes would be desirable.

> After some more thought, I’ve finally bit the bullet:
>
>   1. Commit 77b0ac9 adds the #:substitutable? flag to gnu-build-system.
>
>   2. Commit f15615b uses it for ATLAS.

Wow! That was quick. Thanks!

> It may still make sense to have the non-optimized version for those who
> do not need performance and do not want to build locally maybe, WDYT?

Personally my main use of ATLAS is through numpy/scipy. Therefore I
would like to be able to use a good performing ATLAS with those
packages.  If that needs a manual installation step, that's fine with
me.

In principle we could make a second package and add a suffix to the
version number such that the general binary version takes precedence.
To get a locally built library one would then have to give an explicit
ATLAS version to guix.  Or, do you have other approaches in mind?

Regards,
Fede

Re: [PATCH 1/2] gnu: Add numpy

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> After some more thought, I’ve finally bit the bullet:
>
>   1. Commit 77b0ac9 adds the #:substitutable? flag to gnu-build-system.
>
>   2. Commit f15615b uses it for ATLAS.

There was a related bug: ‘guix build atlas’ and similar would tell you
that atlas will be downloaded, but in fact it would be built locally as
intended.

Commit d2d0514 fixes that.

Ludo’.

Re: [PATCH 1/2] gnu: Add numpy

[Ludovic Courtès]

Federico Beffa <beffa@ieee.org> skribis:

> On Tue, Oct 28, 2014 at 10:34 AM, Ludovic Courtès <ludo@gnu.org> wrote:

[...]

>> Perhaps the right fix will be to change ‘python-wrapper’ to symlink the
>> ‘lib’ sub-directory of ‘python’.
>
> It was also suggested to make python a propagating input of the wrapper.
> https://lists.gnu.org/archive/html/guix-devel/2014-10/msg00303.html

Yes, that’s even better.

Ludo’.

Re: [PATCH 1/2] gnu: Add numpy

[Andreas Enge]

On Mon, Oct 27, 2014 at 09:26:41PM +0100, Federico Beffa wrote:
> I get error messages that matplotlib is missing.  I started looking at
> matplotlib as well, but I've found that, for the TkAgg back-end, it
> needs TKinter which is part of the standard python libraries and it is
> built if, during the build phase, tk is available.

So we should add tk as an input to the python package?

Andreas

Re: [PATCH 1/2] gnu: Add numpy

[Andreas Enge]

On Tue, Oct 28, 2014 at 10:34:48AM +0100, Ludovic Courtès wrote:
> Ah right.  And what if you again remove Python from ‘inputs’, and add
>   #:python ,python
> to the arguments?
> That means it will use the actual Python 3.x package, not the wrapper,
> so everything will be visible.  The downside is that there will be no
> ‘python’ command, only ‘python3’.
> 
> Perhaps the right fix will be to change ‘python-wrapper’ to symlink the
> ‘lib’ sub-directory of ‘python’.

As already stated, I think we should add python as a propagated input to
python-wrapper. The symlinks are just needed between the differently named
python and python3.

Should we give this a try?

Andreas

Changing python-wrapper to handle lib/ etc.

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Tue, Oct 28, 2014 at 10:34:48AM +0100, Ludovic Courtès wrote:
>> Ah right.  And what if you again remove Python from ‘inputs’, and add
>>   #:python ,python
>> to the arguments?
>> That means it will use the actual Python 3.x package, not the wrapper,
>> so everything will be visible.  The downside is that there will be no
>> ‘python’ command, only ‘python3’.
>> 
>> Perhaps the right fix will be to change ‘python-wrapper’ to symlink the
>> ‘lib’ sub-directory of ‘python’.
>
> As already stated, I think we should add python as a propagated input to
> python-wrapper. The symlinks are just needed between the differently named
> python and python3.
>
> Should we give this a try?

I think our messages crossed each other; I wrote:

  The other option would have been to add ‘python’ to ‘propagated-inputs’,
  which would also have taken care of sub-directories other than lib/.

  As a side-effect, it would also pull in the ‘python3’, ‘pydoc3’,
  etc. commands, but I think it cannot hurt.

So yes, I think we should try this.

What about making this change in a branch, letting Hydra build it, and
then merge it if nothing broke?

We’ll also need the patch from
<http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00364.html> in
this branch.

Do you want to take care of it?  :-)

TIA,
Ludo’.

Re: [PATCH 1/2] gnu: Add numpy

[Federico Beffa]

On Wed, Oct 29, 2014 at 11:20 PM, Andreas Enge <andreas@enge.fr> wrote:
> On Mon, Oct 27, 2014 at 09:26:41PM +0100, Federico Beffa wrote:
>> I get error messages that matplotlib is missing.  I started looking at
>> matplotlib as well, but I've found that, for the TkAgg back-end, it
>> needs TKinter which is part of the standard python libraries and it is
>> built if, during the build phase, tk is available.
>
> So we should add tk as an input to the python package?

That would be great!  In this way the python package should build the
Tkinter library which currently is missing.

Re: [PATCH 1/2] gnu: Add numpy

[Andreas Enge]

On Thu, Oct 30, 2014 at 05:56:53PM +0100, Federico Beffa wrote:
> On Wed, Oct 29, 2014 at 11:20 PM, Andreas Enge <andreas@enge.fr> wrote:
> > So we should add tk as an input to the python package?
> 
> That would be great!  In this way the python package should build the
> Tkinter library which currently is missing.

Okay. I think I should do this together with the changes to python-wrapper
in the python branch to be created. For both python-2 and python-3?

Andreas

Re: [PATCH 1/2] gnu: Add numpy

[Federico Beffa]

On Thu, Oct 30, 2014 at 6:03 PM, Andreas Enge <andreas@enge.fr> wrote:
> Okay. I think I should do this together with the changes to python-wrapper
> in the python branch to be created. For both python-2 and python-3?

Yes, please.  That will allow making a matplotlib package for both
versions of python.

Regards,
Fede

Re: Changing python-wrapper to handle lib/ etc.

[Andreas Enge]

On Thu, Oct 30, 2014 at 02:12:58PM +0100, Ludovic Courtès wrote:
> We’ll also need the patch from
> <http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00364.html> in
> this branch.

Maybe not, we will see. Once the python library is propagated, should it not
appear automatically in the LIBRARY_PATH?

I created the branch, see its progress at
   http://hydra.gnu.org/jobset/gnu/wip-python
Personally, I would like to go slowly with one modification at a time, but
each of them rebuilds about a quarter of the archive. If this poses problems,
we might wish to disable building it on mipsel64 up to the last step.

Andreas

Re: [PATCH 1/2] gnu: Add numpy

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> This is because Python is not added to ‘LIBRARY_PATH’, right?
>
> I think this is fixed by this patch:
>
> --- a/guix/build-system/python.scm
> +++ b/guix/build-system/python.scm
> @@ -110,10 +110,11 @@ prepended to the name."
>                                '())
>                          ,@inputs
>  
> +                        ("python" ,python)
> +
>                          ;; Keep the standard inputs of 'gnu-build-system'.
>                          ,@(standard-packages)))
> -         (build-inputs `(("python" ,python)
> -                         ,@native-inputs))
> +         (build-inputs native-inputs)
>           (outputs outputs)
>           (build python-build)
>           (arguments (strip-keyword-arguments private-keywords arguments)))))

FTR, this patch isn’t needed at all.  What mattered here was to fix
‘python-wrapper’ to have a lib/ sub-directory, which Andreas has just
done in ‘wip-python’.

Ludo’.

Re: Changing python-wrapper to handle lib/ etc.

[Andreas Enge]

The first problem occurs. Swig does not compile any more:
   http://hydra.gnu.org/build/134889/log/raw

There are lots of error messages of the form:
checking testcase namespace_nested under python
namespace_enum_wrap.cxx:154:21: fatal error: Python.h: No such file or directory
 # include <Python.h>

If one compares with the version in master:
   http://hydra.gnu.org/build/131472/log/raw
one realises that there python was simply not found, so the tests were
probably not executed:
checking for Python header files... 
checking for Python library... Not found
checking for python3... no

With the new python wrapper, something is found:
checking for python3-config... python3-config
checking for Python 3.x prefix... /gnu/store/r6614z0w2inxn78wzaa7ic5sai8s7l9f-python-3.3.5
checking for Python 3.x exec-prefix... /gnu/store/r6614z0w2inxn78wzaa7ic5sai8s7l9f-python-3.3.5
checking for Python 3.x version... python3.3
checking for Python 3.x lib dir... lib
checking for Python 3.x header files... -I/gnu/store/r6614z0w2inxn78wzaa7ic5sai8s7l9f-python-3.3.5/include/python3.3m -I/gnu/store/r6614z0w2inxn78wzaa7ic5sai8s7l9f-python-3.3.5/include/python3.3m
checking for Python 3.x library... Not found

Whatever this means! The header files are correctly located in the python3.3m
subdirectory of the include directory. However, the CPATH includes only
the include/ directory itself (logic!). And apparently the tests do not
use the correct include path.

Could maybe a swig specialist have a look?

Andreas

Re: Changing python-wrapper to handle lib/ etc.

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Thu, Oct 30, 2014 at 02:12:58PM +0100, Ludovic Courtès wrote:
>> We’ll also need the patch from
>> <http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00364.html> in
>> this branch.
>
> Maybe not

Indeed.  (I replied in the other thread.)

> I created the branch, see its progress at
>    http://hydra.gnu.org/jobset/gnu/wip-python

Apparently the main regression is SWIG, which fails to find Python.h:

  http://hydra.gnu.org/build/134431

Ludo’.

Re: Changing python-wrapper to handle lib/ etc.

[Federico Beffa]

On Sun, Nov 2, 2014 at 10:32 PM, Andreas Enge <andreas@enge.fr> wrote:
> With the new python wrapper, something is found:
> checking for python3-config... python3-config
> checking for Python 3.x prefix... /gnu/store/r6614z0w2inxn78wzaa7ic5sai8s7l9f-python-3.3.5
> checking for Python 3.x exec-prefix... /gnu/store/r6614z0w2inxn78wzaa7ic5sai8s7l9f-python-3.3.5
> checking for Python 3.x version... python3.3
> checking for Python 3.x lib dir... lib
> checking for Python 3.x header files... -I/gnu/store/r6614z0w2inxn78wzaa7ic5sai8s7l9f-python-3.3.5/include/python3.3m -I/gnu/store/r6614z0w2inxn78wzaa7ic5sai8s7l9f-python-3.3.5/include/python3.3m
> checking for Python 3.x library... Not found
>
> Whatever this means! The header files are correctly located in the python3.3m
> subdirectory of the include directory. However, the CPATH includes only
> the include/ directory itself (logic!). And apparently the tests do not
> use the correct include path.
>
> Could maybe a swig specialist have a look?

I've just noticed the following sentence in the release of version 2.0.12:

- Compilation fixes on some systems for the generated Lua, PHP, Python
and R wrappers.

See http://www.swig.org/

Maybe, it was an upstream bug?

Regards,
Fede

Re: Changing python-wrapper to handle lib/ etc.

[Andreas Enge]

On Mon, Nov 03, 2014 at 11:11:22AM +0100, Federico Beffa wrote:
> - Compilation fixes on some systems for the generated Lua, PHP, Python
> and R wrappers.
> Maybe, it was an upstream bug?

Thanks, I will try to update and keep you updated.

Andreas

Re: Changing python-wrapper to handle lib/ etc.

[Andreas Enge]

On Mon, Nov 03, 2014 at 11:13:45AM +0100, Andreas Enge wrote:
> Thanks, I will try to update and keep you updated.

This was it not. There is also a newer swig-3.0.2, but it also does not
solve the problem. My suspicion is that swig has never worked with python
for us. So my current solution would be to disable it and push this change
if all dependent packages build. Someone interested in swig and python can
then try to fix this.

Andreas

Re: Changing python-wrapper to handle lib/ etc.

[Andreas Enge]

On Mon, Nov 03, 2014 at 09:19:28PM +0100, Andreas Enge wrote:
> This was it not. There is also a newer swig-3.0.2, but it also does not
> solve the problem. My suspicion is that swig has never worked with python
> for us. So my current solution would be to disable it and push this change
> if all dependent packages build. Someone interested in swig and python can
> then try to fix this.

In 2.0.12 without python, now the guile tests fail... 3.0.2 without python
compiles well, so I am trying to build the swig dependencies now.

Andreas

Re: Changing python-wrapper to handle lib/ etc.

[Andreas Enge]

On Mon, Nov 03, 2014 at 09:49:21PM +0100, Andreas Enge wrote:
> In 2.0.12 without python, now the guile tests fail... 3.0.2 without python
> compiles well, so I am trying to build the swig dependencies now.

Everything worked well, so I pushed this solution to wip-python.

Andreas

Re: Changing python-wrapper to handle lib/ etc.

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Mon, Nov 03, 2014 at 09:49:21PM +0100, Andreas Enge wrote:
>> In 2.0.12 without python, now the guile tests fail... 3.0.2 without python
>> compiles well, so I am trying to build the swig dependencies now.
>
> Everything worked well, so I pushed this solution to wip-python.

Nice, thanks!

Ludo’.

Re: Changing python-wrapper to handle lib/ etc.

[Andreas Enge]

Next step: When adding tk as an input to python, I get the followiing error.
tcl in place of tk poses no problem; is there another variable named tk
somewhere?

Backtrace:
In srfi/srfi-1.scm:
 578: 19 [map #<procedure 2bd4e10 at guix/packages.scm:796:30 (t-7629)> (# # # ...)]
In guix/packages.scm:
 655: 18 [expand-input # # # ...]
 613: 17 [cache # # #<procedure thunk ()>]
 862: 16 [thunk]
 794: 15 [bag->derivation # # #]
In srfi/srfi-1.scm:
 576: 14 [map #<procedure 2bf93c0 at guix/packages.scm:796:30 (t-7629)> (# # # ...)]
In guix/packages.scm:
 655: 13 [expand-input # # # ...]
 613: 12 [cache # # #<procedure thunk ()>]
 862: 11 [thunk]
 794: 10 [bag->derivation # # #]
In srfi/srfi-1.scm:
 578: 9 [map #<procedure 2bf91e0 at guix/packages.scm:796:30 (t-7629)> (# # # ...)]
In guix/packages.scm:
 670: 8 [expand-input # # # ...]
In guix/monads.scm:
 469: 7 [run-with-store # #<procedure 2cec1e0 at guix/gexp.scm:173:2 (store)> # ...]
In guix/gexp.scm:
 173: 6 [#<procedure 2bfae70 at guix/gexp.scm:173:2 (store)> #]
 173: 5 [#<procedure 3e5d900 at guix/gexp.scm:173:2 (natives)> ()]
 319: 4 [gexp->sexp #<gexp (begin # # #) 2bf9150> #:system "x86_64-linux" ...]
In srfi/srfi-1.scm:
 578: 3 [map #<procedure reference->sexp (ref #:optional native?)> #]
In guix/gexp.scm:
 312: 2 [reference->sexp (# # # # ...) #f]
In srfi/srfi-1.scm:
 578: 1 [map #<procedure 2bfae10 at guix/gexp.scm:313:24 (t-25791)> (# # # # ...)]
In guix/gexp.scm:
 287: 0 [reference->sexp # #f # ...]

guix/gexp.scm:287:2: In procedure reference->sexp:
guix/gexp.scm:287:2: Throw to key `vm-error' with args `(vm-run "VM: Stack overflow" ())'.


Andreas

Re: Changing python-wrapper to handle lib/ etc.

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Next step: When adding tk as an input to python, I get the followiing error.
> tcl in place of tk poses no problem; is there another variable named tk
> somewhere?

Doesn’t that create a circular dependency between the ‘python’ package
and the ‘tk’ package, somehow?

(Ideally ‘package-derivation’ would detect it and raise and error.)

Ludo’.

Re: Changing python-wrapper to handle lib/ etc.

[Andreas Enge]

On Tue, Nov 04, 2014 at 05:48:56PM +0100, Ludovic Courtès wrote:
> Doesn’t that create a circular dependency between the ‘python’ package
> and the ‘tk’ package, somehow?

I checked only the direct inputs, the key is in "somehow"!

tk depends on libx11, depends on libxcb, depends on python.

How do we break this dependency? By creating one version of python with
input tk (used inside the xorg module), and one with input tk (served to
the outside world)?

Andreas

Re: Changing python-wrapper to handle lib/ etc.

[Andreas Enge]

On Tue, Nov 04, 2014 at 06:08:30PM +0100, Andreas Enge wrote:
> How do we break this dependency? By creating one version of python with
> input tk (used inside the xorg module), and one with input tk (served to
> the outside world)?

The first "with" should have been "without", of course.

Andreas

Re: Changing python-wrapper to handle lib/ etc.

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Tue, Nov 04, 2014 at 05:48:56PM +0100, Ludovic Courtès wrote:
>> Doesn’t that create a circular dependency between the ‘python’ package
>> and the ‘tk’ package, somehow?
>
> I checked only the direct inputs, the key is in "somehow"!
>
> tk depends on libx11, depends on libxcb, depends on python.

This:

  guix gc --references $(guix build libxcb) | grep python

shows nothing.  So what about creating a bare-bones Python for that
(perhaps with other optional dependencies removed, while we’re at it),
and use that as a dependency for libxcb?

Thanks,
Ludo’.

Re: Changing python-wrapper to handle lib/ etc.

[Andreas Enge]

On Tue, Nov 04, 2014 at 10:40:14PM +0100, Ludovic Courtès wrote:
>   guix gc --references $(guix build libxcb) | grep python
> shows nothing.  So what about creating a bare-bones Python for that
> (perhaps with other optional dependencies removed, while we’re at it),
> and use that as a dependency for libxcb?

There is indeed no trace of python scripts in the installed files, but it is
used as a native input. I had hoped we might simply drop it, but it is
required for building. So adding a minimal python sounds like an option.
Or maybe just a clone of the current python (without tk), to minimise the
development work...

Andreas