RFC: building numpy against OpenBLAS.

RFC: building numpy against OpenBLAS.

[Ricardo Wurmus]

[-- Attachment #1: Type: text/plain, Size: 1923 bytes --]

Hi Guix,

python-numpy currently depends on Atlas, which means that it cannot be
substituted with a binary built elsewhere.  OpenBLAS is an alternative
to Atlas and the binary can be used on all supported CPUs at runtime.
This makes it possible for us to make numpy substitutable.

We currently do not have a working OpenBLAS on MIPS, so the attached
patch selects OpenBLAS as an input only when not on MIPS.  Some
additional configuration happens only unless "atlas" is among the
inputs.

I have successfully compiled numpy and python-scikit-image for both
versions of Python on x86_64 with these changes.  I should say, however,
that there's a note in the numpy sources that warns about a bug under
certain conditions:

    # **Warning**: OpenBLAS, by default, is built in multithreaded mode. Due to the
    # way Python's multiprocessing is implemented, a multithreaded OpenBLAS can
    # cause programs using both to hang as soon as a worker process is forked on
    # POSIX systems (Linux, Mac).
    # This is fixed in Openblas 0.2.9 for the pthread build, the OpenMP build using
    # GNU openmp is as of gcc-4.9 not fixed yet.
    # Python 3.4 will introduce a new feature in multiprocessing, called the
    # "forkserver", which solves this problem. For older versions, make sure
    # OpenBLAS is built using pthreads or use Python threads instead of
    # multiprocessing.
    # (This problem does not exist with multithreaded ATLAS.)

I do not know if this is still a problem and I haven't yet tried to
reproduce this.  Our OpenBLAS is not built with openmp, so I think this
problem does not affect us.

Anyway, I just wanted to post this here to ask for opinions.  Maybe this
is a bad idea.  (In my case it makes sense not to use Atlas, because the
compile-time tuning is useless when a shared store is used and clients
use Atlas on machines other than the build host.)

Comments are very welcome!

~~ Ricardo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-python-numpy-Build-against-OpenBLAS.patch --]
[-- Type: text/x-patch, Size: 3268 bytes --]

From 4b56608674a9c7977cad88ce3f03a9fcb72c93bc Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
Date: Fri, 22 May 2015 16:48:05 +0200
Subject: [PATCH] gnu: python-numpy: Build against OpenBLAS.

* gnu/packages/python.scm (python-numpy)[inputs]: Use "openblas" instead of
  "atlas" when not on MIPS.
* gnu/packages/python.scm (python-numpy)[arguments]: Configure build against
  OpenBLAS unless "atlas" is among the inputs.
---
 gnu/packages/python.scm | 38 ++++++++++++++++++++++++++++----------
 1 file changed, 28 insertions(+), 10 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 0916a75..c96296f 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -2283,7 +2283,9 @@ writing C extensions for Python as easy as Python itself.")
     (build-system python-build-system)
     (inputs
      `(("python-nose" ,python-nose)
-       ("atlas" ,atlas)))
+       ,(if (string-prefix? "mips" (%current-system))
+            `("atlas" ,atlas)
+            `("openblas" ,openblas))))
     (native-inputs
      `(("gfortran" ,gfortran-4.8)))
     (arguments
@@ -2291,16 +2293,32 @@ writing C extensions for Python as easy as Python itself.")
        (alist-cons-before
         'build 'set-environment-variables
         (lambda* (#:key inputs #:allow-other-keys)
-          (let* ((atlas-threaded
-                  (string-append (assoc-ref inputs "atlas")
-                                 "/lib/libtatlas.so"))
-                 ;; On single core CPUs only the serial library is created.
-                 (atlas-lib
-                  (if (file-exists? atlas-threaded)
-                      atlas-threaded
+          (if (assoc-ref inputs "atlas")
+              ;; Link against Atlas
+              (let* ((atlas-threaded
                       (string-append (assoc-ref inputs "atlas")
-                                     "/lib/libsatlas.so"))))
-            (setenv "ATLAS" atlas-lib)))
+                                     "/lib/libtatlas.so"))
+                     ;; On single core CPUs only the serial library is created.
+                     (atlas-lib
+                      (if (file-exists? atlas-threaded)
+                          atlas-threaded
+                          (string-append (assoc-ref inputs "atlas")
+                                         "/lib/libsatlas.so"))))
+                (setenv "ATLAS" atlas-lib))
+              ;; Link against OpenBLAS
+              (begin
+                (call-with-output-file "site.cfg"
+                  (lambda (port)
+                    (format port "[openblas]
+libraries = openblas
+library_dirs = ~a/lib
+include_dirs = ~a/include
+" (assoc-ref inputs "openblas") (assoc-ref inputs "openblas"))))
+                ;; Use "gcc" executable, not "cc".
+                (substitute* "numpy/distutils/system_info.py"
+                  (("c = distutils\\.ccompiler\\.new_compiler\\(\\)")
+                   "c = distutils.ccompiler.new_compiler(); c.set_executables(compiler='gcc',compiler_so='gcc',linker_exe='gcc',linker_so='gcc -shared')"))))
+          #t)
         ;; Tests can only be run after the library has been installed and not
         ;; within the source directory.
         (alist-cons-after
-- 
2.1.0

Re: RFC: building numpy against OpenBLAS.

[Eric Bavier]

----- Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> wrote:
> python-numpy currently depends on Atlas, which means that it cannot be
> substituted with a binary built elsewhere.  OpenBLAS is an alternative
> to Atlas and the binary can be used on all supported CPUs at runtime.
> This makes it possible for us to make numpy substitutable.
[...]
> Anyway, I just wanted to post this here to ask for opinions.  Maybe this
> is a bad idea.  (In my case it makes sense not to use Atlas, because the
> compile-time tuning is useless when a shared store is used and clients
> use Atlas on machines other than the build host.)
> 

I would very much like to see OpenBLAS substituted for atlas whereever possible.  The runtime performance of OpenBLAS is quite a bit better than ATLAS.  I've seen e.g. OpenBLAS to be roughly 20% faster than ATLAS for GEMM calls on Sandybridge platforms.

`~Eric

RFC: building numpy against OpenBLAS.

[Federico Beffa]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

> Hi Guix,
>
> python-numpy currently depends on Atlas, which means that it cannot be
> substituted with a binary built elsewhere.  OpenBLAS is an alternative
> to Atlas and the binary can be used on all supported CPUs at runtime.
> This makes it possible for us to make numpy substitutable.

Out of curiosity, could you outline how OpenBLAS is optimized for a
specific CPU architecture while being compiled on a different CPU (and
hence allowing to be substituted)?

Thanks,
Fede

Re: RFC: building numpy against OpenBLAS.

[Ricardo Wurmus]

Federico Beffa writes:

> Out of curiosity, could you outline how OpenBLAS is optimized for a
> specific CPU architecture while being compiled on a different CPU (and
> hence allowing to be substituted)?

The Quick Install instructions[1] say that when OpenBLAS is compiled
with DYNAMIC_ARCH=1

   "All kernel will be included in the library and dynamically switched
    the best architecutre at run time."

It seems that unlike ATLAS, OpenBLAS does not perform any self-tuning
but relies on hand-optimised code (e.g. by using CPU-specific
instructions).

~~ Ricardo

[1]: https://github.com/xianyi/OpenBLAS/blob/80bf3e6a3525f558a9fde2514622313c991c091f/GotoBLAS_02QuickInstall.txt#L67

Re: RFC: building numpy against OpenBLAS.

[Federico Beffa]

On Wed, May 27, 2015 at 5:50 PM, Ricardo Wurmus
<ricardo.wurmus@mdc-berlin.de> wrote:
>
> Federico Beffa writes:
>
>> Out of curiosity, could you outline how OpenBLAS is optimized for a
>> specific CPU architecture while being compiled on a different CPU (and
>> hence allowing to be substituted)?
>
> The Quick Install instructions[1] say that when OpenBLAS is compiled
> with DYNAMIC_ARCH=1
>
>    "All kernel will be included in the library and dynamically switched
>     the best architecutre at run time."
>
> It seems that unlike ATLAS, OpenBLAS does not perform any self-tuning
> but relies on hand-optimised code (e.g. by using CPU-specific
> instructions).

Sounds good. Thanks!

Fede

Re: RFC: building numpy against OpenBLAS.

[Ricardo Wurmus]

Ricardo Wurmus writes:

> python-numpy currently depends on Atlas, which means that it cannot be
> substituted with a binary built elsewhere.  OpenBLAS is an alternative
> to Atlas and the binary can be used on all supported CPUs at runtime.
> This makes it possible for us to make numpy substitutable.
>
> We currently do not have a working OpenBLAS on MIPS, so the attached
> patch selects OpenBLAS as an input only when not on MIPS.  Some
> additional configuration happens only unless "atlas" is among the
> inputs.

If there are no objections I'd like to go ahead and push this patch to
build numpy with OpenBLAS.  From the comments I gather that it's not as
controversial a change as I suspected.

~~ Ricardo

Re: RFC: building numpy against OpenBLAS.

[Mark H Weaver]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

> Ricardo Wurmus writes:
>
>> python-numpy currently depends on Atlas, which means that it cannot be
>> substituted with a binary built elsewhere.  OpenBLAS is an alternative
>> to Atlas and the binary can be used on all supported CPUs at runtime.
>> This makes it possible for us to make numpy substitutable.
>>
>> We currently do not have a working OpenBLAS on MIPS, so the attached
>> patch selects OpenBLAS as an input only when not on MIPS.  Some
>> additional configuration happens only unless "atlas" is among the
>> inputs.
>
> If there are no objections I'd like to go ahead and push this patch to
> build numpy with OpenBLAS.  From the comments I gather that it's not as
> controversial a change as I suspected.

Yes, please do!

    Thanks,
      Mark

Re: RFC: building numpy against OpenBLAS.

[Mark H Weaver]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

> Ricardo Wurmus writes:
>
>> python-numpy currently depends on Atlas, which means that it cannot be
>> substituted with a binary built elsewhere.  OpenBLAS is an alternative
>> to Atlas and the binary can be used on all supported CPUs at runtime.
>> This makes it possible for us to make numpy substitutable.
>>
>> We currently do not have a working OpenBLAS on MIPS, so the attached
>> patch selects OpenBLAS as an input only when not on MIPS.  Some
>> additional configuration happens only unless "atlas" is among the
>> inputs.
>
> If there are no objections I'd like to go ahead and push this patch to
> build numpy with OpenBLAS.  From the comments I gather that it's not as
> controversial a change as I suspected.

I think we should go ahead and switch to OpenBLAS on _all_ platforms,
not just on MIPS.  Were there any objections to that?  If not, I'd
advocate just doing it.

     Thanks!
       Mark

Re: RFC: building numpy against OpenBLAS.

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:
>
>> Ricardo Wurmus writes:
>>
>>> python-numpy currently depends on Atlas, which means that it cannot be
>>> substituted with a binary built elsewhere.  OpenBLAS is an alternative
>>> to Atlas and the binary can be used on all supported CPUs at runtime.
>>> This makes it possible for us to make numpy substitutable.
>>>
>>> We currently do not have a working OpenBLAS on MIPS, so the attached
>>> patch selects OpenBLAS as an input only when not on MIPS.  Some
>>> additional configuration happens only unless "atlas" is among the
>>> inputs.
>>
>> If there are no objections I'd like to go ahead and push this patch to
>> build numpy with OpenBLAS.  From the comments I gather that it's not as
>> controversial a change as I suspected.
>
> I think we should go ahead and switch to OpenBLAS on _all_ platforms,
> not just on MIPS.  Were there any objections to that?  If not, I'd
> advocate just doing it.

Libreoffice depends on python2-numpy, so if we don't switch to OpenBLAS
on all platforms, then we cannot offer binary substitutes for
Libreoffice.

Also, the persistent build failures of python-numpy on Hydra are
probably due to its use of ATLAS, when ALTAS is built on a different
build slave than python2-numpy.

So, at this point we have several strong incentives to phase out the use
of ATLAS.

      Mark

Re: RFC: building numpy against OpenBLAS.

[Ricardo Wurmus]

>>> We currently do not have a working OpenBLAS on MIPS, so the attached
>>> patch selects OpenBLAS as an input only when not on MIPS.  Some
>>> additional configuration happens only unless "atlas" is among the
>>> inputs.
>>
>> If there are no objections I'd like to go ahead and push this patch to
>> build numpy with OpenBLAS.  From the comments I gather that it's not as
>> controversial a change as I suspected.
>
> I think we should go ahead and switch to OpenBLAS on _all_ platforms,
> not just on MIPS.  Were there any objections to that?  If not, I'd
> advocate just doing it.

There may have been a misunderstanding: We can switch to OpenBLAS for
all platforms _except_ MIPS because OpenBLAS currently does not build on
MIPS.

~~ Ricardo

Re: RFC: building numpy against OpenBLAS.

[Mark H Weaver]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

>>>> We currently do not have a working OpenBLAS on MIPS, so the attached
>>>> patch selects OpenBLAS as an input only when not on MIPS.  Some
>>>> additional configuration happens only unless "atlas" is among the
>>>> inputs.
>>>
>>> If there are no objections I'd like to go ahead and push this patch to
>>> build numpy with OpenBLAS.  From the comments I gather that it's not as
>>> controversial a change as I suspected.
>>
>> I think we should go ahead and switch to OpenBLAS on _all_ platforms,
>> not just on MIPS.  Were there any objections to that?  If not, I'd
>> advocate just doing it.
>
> There may have been a misunderstanding: We can switch to OpenBLAS for
> all platforms _except_ MIPS because OpenBLAS currently does not build on
> MIPS.

I was indeed confused.  However, it seems that ATLAS doesn't build on
MIPS either, based on its 'supported-systems' field.

So, I come again to the same conclusion: that we should switch to
OpenBLAS on _all_ platforms, and strip away the ATLAS-handling code in
our python-numpy package.  We can work on fixing OpenBLAS to MIPS later.

What do you think?

    Thanks!
      Mark

Re: RFC: building numpy against OpenBLAS.

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> So, I come again to the same conclusion: that we should switch to
> OpenBLAS on _all_ platforms, and strip away the ATLAS-handling code in
> our python-numpy package.  We can work on fixing OpenBLAS to MIPS later.

Agreed, we can’t afford not to have substitutes for things like
LibreOffice.

Mark or Ricardo, can you push the patch?

Thanks!

Ludo’.

Re: RFC: building numpy against OpenBLAS.

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> So, I come again to the same conclusion: that we should switch to
>> OpenBLAS on _all_ platforms, and strip away the ATLAS-handling code in
>> our python-numpy package.  We can work on fixing OpenBLAS to MIPS later.
>
> Agreed, we can’t afford not to have substitutes for things like
> LibreOffice.
>
> Mark or Ricardo, can you push the patch?

Done, thanks!

    Mark