From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id XHidFbGrlGTPNwAASxT56A (envelope-from ) for ; Thu, 22 Jun 2023 22:14:41 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id qEDGFLGrlGQjMwEAauVa8A (envelope-from ) for ; Thu, 22 Jun 2023 22:14:41 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E878B17107 for ; Thu, 22 Jun 2023 22:14:40 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qCQgv-00066K-Ct; Thu, 22 Jun 2023 16:14:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qCQgt-00065x-12 for bug-guix@gnu.org; Thu, 22 Jun 2023 16:14:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qCQgr-0007Fj-QG for bug-guix@gnu.org; Thu, 22 Jun 2023 16:14:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qCQgr-0007Cv-Lg for bug-guix@gnu.org; Thu, 22 Jun 2023 16:14:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#63986: Julia is very slow Resent-From: Efraim Flashner Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 22 Jun 2023 20:14:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 63986 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Simon Tournier Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 63986@debbugs.gnu.org, Nicolas Graves , csantosb@inventati.org Received: via spool by 63986-submit@debbugs.gnu.org id=B63986.168746480027642 (code B ref 63986); Thu, 22 Jun 2023 20:14:01 +0000 Received: (at 63986) by debbugs.gnu.org; 22 Jun 2023 20:13:20 +0000 Received: from localhost ([127.0.0.1]:36858 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qCQgB-0007Bm-TL for submit@debbugs.gnu.org; Thu, 22 Jun 2023 16:13:20 -0400 Received: from mail-wm1-f44.google.com ([209.85.128.44]:51400) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qCQg8-0007BU-Nh for 63986@debbugs.gnu.org; Thu, 22 Jun 2023 16:13:18 -0400 Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-3f900cd3f96so66563295e9.2 for <63986@debbugs.gnu.org>; Thu, 22 Jun 2023 13:13:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687464790; x=1690056790; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:sender:from:to :cc:subject:date:message-id:reply-to; bh=Ky8Xs7f6tCQ0/1DiEKVSUMPCsNJgIfwZsJoAStEQ9zY=; b=UFASvvkEbF+0bklbcOwKUqVd98Zd6DV95YcFBvZUj6LfSoCdXrcu2ZA9tIMzxhQCZZ 9+uNZWCwW9s2q0sSw+dOL6JeTcjt7UUbSJK8c6+MdwZZ16ZqPJ2i5iNeAZ6KmDToYaDn 28gGCE79h6kf2HsdA0hbUJs9MneCMP+rk+MPXD1kR+N4HoGnM5t37a6lKkF1y0msBzr5 6xLZ5laY9RV2E0OHx4rzsr4eXvs+9VoQ7UaoZmH12ivqGhqhw5AD4fPJezwjAFXN/oCC g2kYgt1goR8YOailxbIpy/GurBalgca1TUOwjEQ0aEUZFxksKnuP0/BNmyoF5sBtgO9p eK/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687464790; x=1690056790; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ky8Xs7f6tCQ0/1DiEKVSUMPCsNJgIfwZsJoAStEQ9zY=; b=Ny1LLnvS8E4IdMkqssdX2vq5s3Gj8nYQbq3oBzTHXP5nNnyHpfYzq3olb1jxWimWFH /71SwSUYbs7eSE40y+o3qAUMdNRIoIWL4HZXEW372MX7isGocIS+4pO2Ulzrph8NpQdR Q1RcTyNAuI6o/hk0aKryeKTYLZ8gAjg4cyyQtn2Ko26oIS1XM8fVF/jnKIXpH4r41917 +aEoAawyLxkdkr/xZGHBws/YvGnUGl9oDFSdPU7t0BYHa4vNv59uodZrOxyGGc2SjVA0 QdvX5+B2t2koLzlMe/BbGrqspFHlM/9XmqBlP13QStPDfdW4/lWfKr8SWaUdTAcL9yYB 7FzQ== X-Gm-Message-State: AC+VfDwqRUTqVxkRI3KCxMTruUpNgG5Zi8Yo4tXNgQqtuj15FrBAVwBc B6D6EX+zQXzubk7O2AUZasI= X-Google-Smtp-Source: ACHHUZ5ocX4oVNLHhXzvPZLSWWXwMRNrsl5ykIIZ4lSRx3CqPAalMcquNulOQuwK6lZ9IiK7K/5e6g== X-Received: by 2002:adf:f3d0:0:b0:30d:7c09:b677 with SMTP id g16-20020adff3d0000000b0030d7c09b677mr16203892wrp.39.1687464789644; Thu, 22 Jun 2023 13:13:09 -0700 (PDT) Received: from localhost ([188.120.129.124]) by smtp.gmail.com with ESMTPSA id p1-20020adfcc81000000b0030ada01ca78sm7791448wrj.10.2023.06.22.13.13.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jun 2023 13:13:08 -0700 (PDT) Date: Thu, 22 Jun 2023 23:13:07 +0300 From: Efraim Flashner Message-ID: Mail-Followup-To: Efraim Flashner , Simon Tournier , Ludovic =?UTF-8?Q?Court=C3=A8s?= , 63986@debbugs.gnu.org, Nicolas Graves , csantosb@inventati.org References: <87legsl3cm.fsf@inventati.org> <87jzw6rq48.fsf@gnu.org> <871qi7a0qr.fsf@gmail.com> <87pm5o7uh9.fsf@gnu.org> <87ttv0mth3.fsf@inventati.org> <87wmzvmxt8.fsf@inventati.org> <87h6qz8pg0.fsf@gmail.com> <875y7f8hbf.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="gdMZlgtagU6t38UE" Content-Disposition: inline In-Reply-To: <875y7f8hbf.fsf@gmail.com> X-PGP-Key-ID: 0x41AAE7DCCA3D8351 X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc X-PGP-Fingerprint: A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1687464881; a=rsa-sha256; cv=none; b=F1VWjrOnOq1dwvnVUaimYorPvWPEL/Ig34M7rRJ27eoGZ6G4v1yX8tZiBflMxDeIb8IIKu 5e74niFnsMuk6SeTKUcKSLZMbYpJcSOnJC1bI8zheAcDUFYpCUOvqZHnizbMOmD4Pw7MKH aquelPYdkbtrIqDJX/5ayhGYtq+kk5HoWXeRS/B0tLSAa4HKkJ05lWqLIlu6GZUuyImGnt AyDBdHw6M/x6BxrFKYxzAxK4L+mtUkFi4B+9e0OS6C6YICht/BCpdypZo3GDy9SV6FBmwT LraA7IoJxvzcKONwl7Tccuw/Qdho9/1JQXYhdf5D5Hapdni0U3ODvCGyttL0bw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20221208 header.b=UFASvvkE; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1687464881; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=Ky8Xs7f6tCQ0/1DiEKVSUMPCsNJgIfwZsJoAStEQ9zY=; b=k9Mjnq5dxVqz5TfwD0J9R5tUESL16D+lpim0g9ohT+nVZMtsRzWniKbWX5fpyP2EcrA3iG vqy5tNDdLpJGVHc3wkbsASxpDPBF4ZoE5xBbDXgCsYskf8+iKqK7aMoXLirOpivu5x2UBO 5dpkhieXffZk7jpebtibbaeDPk30rJyU1thp4vgGf6flL5t0VAIuvdWj4FsvLHNzO9BZEF SnRDfptIay9/JuySRwUD3orbzNk+rUt/yis2+jRnQCvVb1pjsHbI1+XHY7U5NgCJx4P7bV 7LtGxvTLwGwk38umGF8FU4XmLUPh+1Uloyv1Anqv8hFCqJjwoJccZnRW4WQ24Q== Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20221208 header.b=UFASvvkE; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Scanner: scn0.migadu.com X-Migadu-Spam-Score: -1.95 X-Spam-Score: -1.95 X-Migadu-Queue-Id: E878B17107 X-TUID: +wp6Rh4rVyXG --gdMZlgtagU6t38UE Content-Type: multipart/mixed; boundary="VONVFpOeYgn05Vzw" Content-Disposition: inline --VONVFpOeYgn05Vzw Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jun 22, 2023 at 08:47:48PM +0200, Simon Tournier wrote: > Hi, >=20 > On Thu, 22 Jun 2023 at 19:25, Efraim Flashner wro= te: >=20 > > (ins)efraim@3900XT ~/workspace/guix$ cat /gnu/store/v6z5ykkjfzbc72x1x90= 0xflspqc5wd5r-openblas-ilp64-0.3.20/lib/pkgconfig/openblas.pc > > libdir=3D/gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3= =2E20/lib > > includedir=3D/gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64= -0.3.20/include > > openblas_config=3D USE_64BITINT=3D DYNAMIC_ARCH=3D1 DYNAMIC_OLDER=3D1 N= O_CBLAS=3D NO_LAPACK=3D NO_LAPACKE=3D NO_AFFINITY=3D1 USE_OPENMP=3D generic= MAX_THREADS=3D128 > > version=3D0.3.20 > > extralib=3D-lm -lpthread -lgfortran -lm -lpthread -lgfortran > > Name: openblas > > Description: OpenBLAS is an optimized BLAS library based on GotoBLAS2 1= =2E13 BSD version > > Version: ${version} > > URL: https://github.com/xianyi/OpenBLAS > > Libs: -L${libdir} -lopenblas > > Libs.private: ${extralib} > > Cflags: -I${includedir} > > > > Looks like it should be "LIBBLAS=3D-lopenblas" >=20 > I propose to tweak openblas-ilp64. Currently it looks like: >=20 > --8<---------------cut here---------------start------------->8--- > $ tree $(guix build openblas-ilp64)/lib > /gnu/store/v6z5ykkjfzbc72x1x900xflspqc5wd5r-openblas-ilp64-0.3.20/lib > =E2=94=9C=E2=94=80=E2=94=80 cmake > =E2=94=82=C2=A0=C2=A0 =E2=94=94=E2=94=80=E2=94=80 openblas > =E2=94=82=C2=A0=C2=A0 =E2=94=9C=E2=94=80=E2=94=80 OpenBLASConfig.cmake > =E2=94=82=C2=A0=C2=A0 =E2=94=94=E2=94=80=E2=94=80 OpenBLASConfigVersi= on.cmake > =E2=94=9C=E2=94=80=E2=94=80 libopenblas_ilp64p-r0.3.20.so > =E2=94=9C=E2=94=80=E2=94=80 libopenblas_ilp64.so -> libopenblas_ilp64p-r0= =2E3.20.so > =E2=94=9C=E2=94=80=E2=94=80 libopenblas_ilp64.so.0 -> libopenblas_ilp64p-= r0.3.20.so > =E2=94=94=E2=94=80=E2=94=80 pkgconfig > =E2=94=94=E2=94=80=E2=94=80 openblas.pc > --8<---------------cut here---------------end--------------->8--- >=20 > which is inconsistent with pkgconfig as you noticed above. Therefore, I > am proposing the addition of a symlink of libopenblas_ilp64p.so to > libopenblas.so. For instance this attached patch. I've attached the patch that I've made it to. julia is built with openblas-ilp64 on 64-bit architectures, openblas-ilp64 has its configure-flags adjusted to match what other programs are expecting, and I've also patched python-numpy to use openblas-ilp64 as another test target. --=20 Efraim Flashner =D7=A8=D7=A0=D7=A9=D7=9C=D7=A4 = =D7=9D=D7=99=D7=A8=D7=A4=D7=90 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --VONVFpOeYgn05Vzw Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="julia-openblas-ilp64.patch" Content-Transfer-Encoding: quoted-printable diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm index ba54175822..e96131dfc0 100644 --- a/gnu/packages/julia.scm +++ b/gnu/packages/julia.scm @@ -269,7 +269,10 @@ (define-public julia (substitute* (jlpath "nghttp2") (((from "libnghttp2")) (to "libnghttp2" "libnghttp2"))) (substitute* (jlpath "OpenBLAS") - (((from "libopenblas")) (to "openblas" "libopenblas"))) + (((from "libopenblas")) + ,@(if (target-64bit?) + `((to "openblas" "libopenblas64_" "libopenblas")) + `((to "openblas" "libopenblas"))))) (substitute* (jlpath "OpenLibm") (((from "libopenlibm")) (to "openlibm" "libopenlibm"))) (substitute* (jlpath "PCRE2") @@ -479,12 +482,12 @@ (define-public julia "NO_GIT=3D1" ; build from release tarball. "USE_GPL_LIBS=3D1" ; proudly =20 - ,@(if (target-aarch64?) - `("USE_BLAS64=3D0") - '()) - - "LIBBLAS=3D-lopenblas" - "LIBBLASNAME=3Dlibopenblas" + ,@(if (target-64bit?) + `("USE_BLAS64=3D1" + "LIBBLAS=3D-lopenblas64_" + "LIBBLASNAME=3Dlibopenblas64_") + `("LIBBLAS=3D-lopenblas" + "LIBBLASNAME=3Dlibopenblas")) =20 (string-append "UTF8PROC_INC=3D" (assoc-ref %build-inputs "utf8proc") @@ -513,7 +516,9 @@ (define-public julia ("llvm" ,llvm-julia) ("mbedtls-apache" ,mbedtls-apache) ("mpfr" ,mpfr) - ("openblas" ,openblas) + ,@(if (target-64bit?) + `(("openblas" ,openblas-ilp64)) + `(("openblas" ,openblas))) ("openlibm" ,openlibm) ("p7zip" ,p7zip) ("pcre2" ,pcre2) diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm index f5a2181905..2d3ce41cb7 100644 --- a/gnu/packages/maths.scm +++ b/gnu/packages/maths.scm @@ -4645,7 +4645,9 @@ (define-public openblas-ilp64 (arguments (substitute-keyword-arguments (package-arguments openblas) ((#:make-flags flags #~'()) - #~(append (list "INTERFACE64=3D1" "LIBNAMESUFFIX=3Dilp64") + #~(append (list "INTERFACE64=3D1" + "SYMBOLSUFFIX=3D64_" + "LIBPREFIX=3Dlibopenblas64_") #$flags)))) (synopsis "Optimized BLAS library based on GotoBLAS (ILP64 version)") (license license:bsd-3))) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index b650b71f3b..deeffb67ae 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -6953,19 +6953,36 @@ (define-public python-numpy (lambda _ (setenv "NPY_NUM_BUILD_JOBS" (number->string (parallel-job-count))))) - (add-before 'build 'configure-blas - (lambda* (#:key inputs #:allow-other-keys) - (call-with-output-file "site.cfg" - (lambda (port) - (format port - "\ + #$@(if (target-64bit?) + #~((add-before 'build 'configure-blas + (lambda* (#:key inputs #:allow-other-keys) + (setenv "NPY_USE_BLAS_ILP64" "1") + (setenv "NPY_BLAS_ILP64_ORDER" "openblas64_") + (setenv "NPY_LAPACK_ILP64_ORDER" "openblas64_") + (call-with-output-file "site.cfg" + (lambda (port) + (format port + "\ +[openblas64_] +libraries =3D openblas64_ +library_dirs =3D ~a/lib +include_dirs =3D ~:*~a/include~%" + (dirname (dirname + (search-input-file + inputs "include/openblas_config.h")))= )))))) + #~((add-before 'build 'configure-blas + (lambda* (#:key inputs #:allow-other-keys) + (call-with-output-file "site.cfg" + (lambda (port) + (format port + "\ [openblas] libraries =3D openblas library_dirs =3D ~a/lib include_dirs =3D ~:*~a/include~%" (dirname (dirname (search-input-file - inputs "include/openblas_config.h")))= ))))) + inputs "include/openblas_config.h")))= ))))))) (add-before 'build 'fix-executable-paths (lambda* (#:key inputs #:allow-other-keys) ;; Make /gnu/store/...-bash-.../bin/sh the default shell, @@ -7021,7 +7038,11 @@ (define-public python-numpy python-pytest-xdist python-typing-extensions gfortran)) - (inputs (list bash openblas)) + (inputs + (list bash + (if (target-64bit?) + openblas-ilp64 + openblas))) (home-page "https://numpy.org") (synopsis "Fundamental package for scientific computing with Python") (description "NumPy is the fundamental package for scientific computing --VONVFpOeYgn05Vzw-- --gdMZlgtagU6t38UE Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmSUq1AACgkQQarn3Mo9 g1EKKA/+MMJCz0bec5w1dDBBZFaPNvmamno2xZD+sjeZCIWwruLAjnShYynJ8+qz I9GoOQc7Yzgd52VKd6DekCFXPTlviKRTzpR1wdNmWPgVTz8VKm0kEpjZdGzcz9aq aCoTnZrQs15aEY0B5yDXyXwtWjx4D8FbZLRs/QlBk0a0WRGHm9gxfCRzMOJJS3Iw hFPWftIET4YVtF5W2W/cZqVHJO1OrgFPdfKPglYJGQl2U4WVxpMWOe5xjyjlM27c /uV5l4BzAOhIUDFweIHmNthfG8X+Bn7T2XqKbmWLolOB+rT1ScETWMxiHs/uH6V2 kWaU9v6BhVOVYshqGvxdDzSyqF3YtXiEt1DG9ZwSvWnb3OtzRwzEMSJV4OrVeJzQ 82u822DLRw0IYylFOmhr39Qx4ZV85ekmlPkyz4BI6Hzw38xsorjN18CrnJxYmKDy FVeVRXoQHW6IZJPIYv9XW8LziGeRych2vJWMaU9DkJUbpWFLO2xDA7FZr7Ra5tag JCqoETfRDovpiUeJ89oa3fwR5OseriXJcLYBn8NYET9xsgNtUnQI+7lQfiExNzBN X6iwRqNJ3JqB0pn1iNyms4uX+RlCEokBYS2xZFzTYKI2C/eVme0eEhtWHod6UrEs kFhIfqMGUhO94rwQCoUvheL1YruAuok0br6/4daiZe2gg6E1RYQ= =wake -----END PGP SIGNATURE----- --gdMZlgtagU6t38UE--