From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ricardo Wurmus Subject: RFC: building numpy against OpenBLAS. Date: Fri, 22 May 2015 17:00:55 +0200 Message-ID: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:35877) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YvoRf-0005c1-CS for guix-devel@gnu.org; Fri, 22 May 2015 11:01:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YvoRc-0005Yc-3N for guix-devel@gnu.org; Fri, 22 May 2015 11:01:07 -0400 Received: from sinope.bbbm.mdc-berlin.de ([141.80.25.23]:58136) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YvoRb-0005YG-MP for guix-devel@gnu.org; Fri, 22 May 2015 11:01:04 -0400 Received: from localhost (localhost [127.0.0.1]) by sinope.bbbm.mdc-berlin.de (Postfix) with ESMTP id E2C63280829 for ; Fri, 22 May 2015 17:01:01 +0200 (CEST) Received: from sinope.bbbm.mdc-berlin.de ([127.0.0.1]) by localhost (sinope.bbbm.mdc-berlin.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LAAVuS_HItdJ for ; Fri, 22 May 2015 17:00:54 +0200 (CEST) Received: from HTCAONE.mdc-berlin.net (mab.citx.mdc-berlin.de [141.80.36.102]) by sinope.bbbm.mdc-berlin.de (Postfix) with ESMTP for ; Fri, 22 May 2015 17:00:54 +0200 (CEST) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: guix --=-=-= Content-Type: text/plain Hi Guix, python-numpy currently depends on Atlas, which means that it cannot be substituted with a binary built elsewhere. OpenBLAS is an alternative to Atlas and the binary can be used on all supported CPUs at runtime. This makes it possible for us to make numpy substitutable. We currently do not have a working OpenBLAS on MIPS, so the attached patch selects OpenBLAS as an input only when not on MIPS. Some additional configuration happens only unless "atlas" is among the inputs. I have successfully compiled numpy and python-scikit-image for both versions of Python on x86_64 with these changes. I should say, however, that there's a note in the numpy sources that warns about a bug under certain conditions: # **Warning**: OpenBLAS, by default, is built in multithreaded mode. Due to the # way Python's multiprocessing is implemented, a multithreaded OpenBLAS can # cause programs using both to hang as soon as a worker process is forked on # POSIX systems (Linux, Mac). # This is fixed in Openblas 0.2.9 for the pthread build, the OpenMP build using # GNU openmp is as of gcc-4.9 not fixed yet. # Python 3.4 will introduce a new feature in multiprocessing, called the # "forkserver", which solves this problem. For older versions, make sure # OpenBLAS is built using pthreads or use Python threads instead of # multiprocessing. # (This problem does not exist with multithreaded ATLAS.) I do not know if this is still a problem and I haven't yet tried to reproduce this. Our OpenBLAS is not built with openmp, so I think this problem does not affect us. Anyway, I just wanted to post this here to ask for opinions. Maybe this is a bad idea. (In my case it makes sense not to use Atlas, because the compile-time tuning is useless when a shared store is used and clients use Atlas on machines other than the build host.) Comments are very welcome! ~~ Ricardo --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename="0001-gnu-python-numpy-Build-against-OpenBLAS.patch" >From 4b56608674a9c7977cad88ce3f03a9fcb72c93bc Mon Sep 17 00:00:00 2001 From: Ricardo Wurmus Date: Fri, 22 May 2015 16:48:05 +0200 Subject: [PATCH] gnu: python-numpy: Build against OpenBLAS. * gnu/packages/python.scm (python-numpy)[inputs]: Use "openblas" instead of "atlas" when not on MIPS. * gnu/packages/python.scm (python-numpy)[arguments]: Configure build against OpenBLAS unless "atlas" is among the inputs. --- gnu/packages/python.scm | 38 ++++++++++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 10 deletions(-) diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 0916a75..c96296f 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -2283,7 +2283,9 @@ writing C extensions for Python as easy as Python itself.") (build-system python-build-system) (inputs `(("python-nose" ,python-nose) - ("atlas" ,atlas))) + ,(if (string-prefix? "mips" (%current-system)) + `("atlas" ,atlas) + `("openblas" ,openblas)))) (native-inputs `(("gfortran" ,gfortran-4.8))) (arguments @@ -2291,16 +2293,32 @@ writing C extensions for Python as easy as Python itself.") (alist-cons-before 'build 'set-environment-variables (lambda* (#:key inputs #:allow-other-keys) - (let* ((atlas-threaded - (string-append (assoc-ref inputs "atlas") - "/lib/libtatlas.so")) - ;; On single core CPUs only the serial library is created. - (atlas-lib - (if (file-exists? atlas-threaded) - atlas-threaded + (if (assoc-ref inputs "atlas") + ;; Link against Atlas + (let* ((atlas-threaded (string-append (assoc-ref inputs "atlas") - "/lib/libsatlas.so")))) - (setenv "ATLAS" atlas-lib))) + "/lib/libtatlas.so")) + ;; On single core CPUs only the serial library is created. + (atlas-lib + (if (file-exists? atlas-threaded) + atlas-threaded + (string-append (assoc-ref inputs "atlas") + "/lib/libsatlas.so")))) + (setenv "ATLAS" atlas-lib)) + ;; Link against OpenBLAS + (begin + (call-with-output-file "site.cfg" + (lambda (port) + (format port "[openblas] +libraries = openblas +library_dirs = ~a/lib +include_dirs = ~a/include +" (assoc-ref inputs "openblas") (assoc-ref inputs "openblas")))) + ;; Use "gcc" executable, not "cc". + (substitute* "numpy/distutils/system_info.py" + (("c = distutils\\.ccompiler\\.new_compiler\\(\\)") + "c = distutils.ccompiler.new_compiler(); c.set_executables(compiler='gcc',compiler_so='gcc',linker_exe='gcc',linker_so='gcc -shared')")))) + #t) ;; Tests can only be run after the library has been installed and not ;; within the source directory. (alist-cons-after -- 2.1.0 --=-=-=--