[bug#74572] [PATCH 0/6] ] Python bindings, reduced size for OpenPMIx/OpenMPI.

[bug#74572] [PATCH 0/6] ] Python bindings, reduced size for OpenPMIx/OpenMPI.

[Maxim Cournoyer]

Maxim Cournoyer (6):
  gnu: openmpi-5: Update to 5.0.6.
  gnu: openpmix: Enable Python bindings.
  gnu: openpmix: Fix indentation.
  gnu: openpmix: Do not install HTML documentation.
  gnu: openmpi-5: Reformat.
  gnu: openmpi-5: Streamline source and do not install HTML doc.

 gnu/packages/mpi.scm      | 134 +++++++++++++++++++++++---------------
 gnu/packages/parallel.scm | 105 ++++++++++++++++-------------
 2 files changed, 139 insertions(+), 100 deletions(-)


base-commit: 5ff9e51e2c4dae0dc74f5c85ec553e89fe1824fd
-- 
2.46.0

[bug#74572] [PATCH 1/6] gnu: openmpi-5: Update to 5.0.6.

[Maxim Cournoyer]

* gnu/packages/mpi.scm (openmpi-5): Update to 5.0.6.

Change-Id: I132b1aaeee4d7a760301582013575a5ea99f03a1
---
 gnu/packages/mpi.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/mpi.scm b/gnu/packages/mpi.scm
index 20497242e5..41e57c3012 100644
--- a/gnu/packages/mpi.scm
+++ b/gnu/packages/mpi.scm
@@ -318,7 +318,7 @@ (define-public openmpi openmpi-4)
 (define-public openmpi-5
   (package
     (inherit openmpi)
-    (version "5.0.3")
+    (version "5.0.6")
     (source
      (origin
        (method url-fetch)
@@ -326,7 +326,7 @@ (define-public openmpi-5
                            (version-major+minor version)
                            "/downloads/openmpi-" version ".tar.bz2"))
        (sha256
-        (base32 "02x9xmpggw77mdpikjjx83j6i4v3gkqbncda73lk5axk0vr841cr"))))
+        (base32 "0mw1z4ppnlvxngwd58kl5q26qmvf3bgjkd4r8wjpqis3pky86hdx"))))
 
     (inputs (modify-inputs (package-inputs openmpi)
               ;; As of Open MPI 5.0.X, PMIx is used to communicate
-- 
2.46.0

[bug#74572] [PATCH 2/6] gnu: openpmix: Enable Python bindings.

[Maxim Cournoyer]

* gnu/packages/parallel.scm (openpmix)
[arguments] <configure-flags>: Add '--enable-python-bindings'.
<phases>: Add set-LDFLAGS phase.
[native-inputs]: Add python-cython.

Change-Id: I6910ed9c3c7b47b3a76085b95b1f251286a48077
---
 gnu/packages/parallel.scm | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index e508dc6493..6a16f0e4c0 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -16,6 +16,7 @@
 ;;; Copyright © 2024 David Elsing <david.elsing@posteo.net>
 ;;; Copyright © 2024 Romain Garbage <romain.garbage@inria.fr>
 ;;; Copyright © 2024 Arun Isaac <arunisaac@systemreboot.net>
+;;; Copyright © 2024 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -709,14 +710,20 @@ (define-public openpmix
    (arguments
     (list #:configure-flags
           #~(list (string-append "--with-hwloc="
-                                 (ungexp (this-package-input "hwloc") "lib")))
+                                 (ungexp (this-package-input "hwloc") "lib"))
+                  "--enable-python-bindings") ;disabled by default
 
           ;; Don't keep a reference to GCC.
           #:disallowed-references (and (not (%current-target-system))
                                        (list (canonical-package gcc)))
-
           #:phases
           #~(modify-phases %standard-phases
+              (add-after 'unpack 'set-LDFLAGS
+                (lambda _
+                  ;; The Cython-compiled shared library would fail the
+                  ;; validate-runpath phase otherwise.
+                  (setenv "LDFLAGS"
+                          (string-append "-Wl,-rpath=" #$output "/lib"))))
               (add-before 'configure 'strip-pmix-cc-absolute
                 (lambda _
                   ;; The 'pmix_info' program prints the 'configure' command
@@ -733,7 +740,7 @@ (define-public openpmix
                     (("#define PMIX_CONFIGURE_CLI .*")
                      "#define PMIX_CONFIGURE_CLI \"[scrubbed]\"\n")))))))
    (inputs (list libevent `(,hwloc "lib") zlib))
-   (native-inputs (list perl python))
+   (native-inputs (list perl python python-cython))
    (synopsis "PMIx library")
    (description
     "PMIx is an application programming interface standard that provides
-- 
2.46.0

[bug#74572] [PATCH 2/6] gnu: openpmix: Enable Python bindings.

[Ludovic Courtès]

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> * gnu/packages/parallel.scm (openpmix)
> [arguments] <configure-flags>: Add '--enable-python-bindings'.
> <phases>: Add set-LDFLAGS phase.
> [native-inputs]: Add python-cython.
>
> Change-Id: I6910ed9c3c7b47b3a76085b95b1f251286a48077

LGTM, provided this does not incur a significant hit on the closure size
of openmpi (we routinely use ‘guix pack’ to send package sets that
include openmpi to supercomputers).

Ludo’.

[bug#74572] [PATCH 3/6] gnu: openpmix: Fix indentation.

[Maxim Cournoyer]

* gnu/packages/parallel.scm (openpmix): Fix indentation.

Change-Id: I05751408847343ac593e666afc5f1efbbef94a79
---
 gnu/packages/parallel.scm | 106 +++++++++++++++++++-------------------
 1 file changed, 53 insertions(+), 53 deletions(-)

diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index 6a16f0e4c0..3e50a75047 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -696,61 +696,61 @@ (define-public psimd
 
 (define-public openpmix
   (package
-   (name "openpmix")
-   (version "4.2.8")
-   (source (origin
-            (method url-fetch)
-            (uri (string-append
-                  "https://github.com/openpmix/openpmix/releases/download/v"
-                  version "/pmix-" version ".tar.bz2"))
-            (sha256
-             (base32
-              "1j9xlhqrrmgjdkwakamn78y5gj756adi53hn25zksgr3is3l5d09"))))
-   (build-system gnu-build-system)
-   (arguments
-    (list #:configure-flags
-          #~(list (string-append "--with-hwloc="
-                                 (ungexp (this-package-input "hwloc") "lib"))
-                  "--enable-python-bindings") ;disabled by default
-
-          ;; Don't keep a reference to GCC.
-          #:disallowed-references (and (not (%current-target-system))
-                                       (list (canonical-package gcc)))
-          #:phases
-          #~(modify-phases %standard-phases
-              (add-after 'unpack 'set-LDFLAGS
-                (lambda _
-                  ;; The Cython-compiled shared library would fail the
-                  ;; validate-runpath phase otherwise.
-                  (setenv "LDFLAGS"
-                          (string-append "-Wl,-rpath=" #$output "/lib"))))
-              (add-before 'configure 'strip-pmix-cc-absolute
-                (lambda _
-                  ;; The 'pmix_info' program prints the 'configure' command
-                  ;; line, compiler absolute file name, etc., which causes it
-                  ;; to keep references to many build-time packages.  Scrub
-                  ;; these.
-                  (substitute* "configure"
-                    (("PMIX_CC_ABSOLUTE=\"(.*)\"" _ cc)
-                     (string-append "PMIX_CC_ABSOLUTE=\"$(basename \""
-                                    cc "\")\"\n")))))
-              (add-after 'configure 'strip-pmix-config-header
-                (lambda _
-                  (substitute* "src/include/pmix_config.h"
-                    (("#define PMIX_CONFIGURE_CLI .*")
-                     "#define PMIX_CONFIGURE_CLI \"[scrubbed]\"\n")))))))
-   (inputs (list libevent `(,hwloc "lib") zlib))
-   (native-inputs (list perl python python-cython))
-   (synopsis "PMIx library")
-   (description
-    "PMIx is an application programming interface standard that provides
+    (name "openpmix")
+    (version "4.2.8")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+                    "https://github.com/openpmix/openpmix/releases/download/v"
+                    version "/pmix-" version ".tar.bz2"))
+              (sha256
+               (base32
+                "1j9xlhqrrmgjdkwakamn78y5gj756adi53hn25zksgr3is3l5d09"))))
+    (build-system gnu-build-system)
+    (arguments
+     (list
+      #:configure-flags
+      #~(list (string-append "--with-hwloc="
+                             (ungexp (this-package-input "hwloc") "lib"))
+              "--enable-python-bindings") ;disabled by default
+
+      ;; Don't keep a reference to GCC.
+      #:disallowed-references (and (not (%current-target-system))
+                                   (list (canonical-package gcc)))
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-after 'unpack 'set-LDFLAGS
+            (lambda _
+              ;; The Cython-compiled shared library would fail the
+              ;; validate-runpath phase otherwise.
+              (setenv "LDFLAGS"
+                      (string-append "-Wl,-rpath=" #$output "/lib"))))
+          (add-before 'configure 'strip-pmix-cc-absolute
+            (lambda _
+              ;; The 'pmix_info' program prints the 'configure' command line,
+              ;; compiler absolute file name, etc., which causes it to keep
+              ;; references to many build-time packages.  Scrub these.
+              (substitute* "configure"
+                (("PMIX_CC_ABSOLUTE=\"(.*)\"" _ cc)
+                 (string-append "PMIX_CC_ABSOLUTE=\"$(basename \""
+                                cc "\")\"\n")))))
+          (add-after 'configure 'strip-pmix-config-header
+            (lambda _
+              (substitute* "src/include/pmix_config.h"
+                (("#define PMIX_CONFIGURE_CLI .*")
+                 "#define PMIX_CONFIGURE_CLI \"[scrubbed]\"\n")))))))
+    (inputs (list libevent `(,hwloc "lib") zlib))
+    (native-inputs (list perl python python-cython))
+    (synopsis "PMIx library")
+    (description
+     "PMIx is an application programming interface standard that provides
 libraries and programming models with portable and well-defined access to
 commonly needed services in distributed and parallel computing systems.")
-   (home-page "https://pmix.org/")
-   ;; configure: WARNING: PMIx does not support 32 bit builds.
-   (supported-systems %64bit-supported-systems)
-   ;; The provided license is kind of BSD-style but specific.
-   (license (license:fsf-free "https://github.com/openpmix/openpmix?tab=License-1-ov-file#License-1-ov-file"))))
+    (home-page "https://pmix.org/")
+    ;; configure: WARNING: PMIx does not support 32 bit builds.
+    (supported-systems %64bit-supported-systems)
+    ;; The provided license is kind of BSD-style but specific.
+    (license (license:fsf-free "https://github.com/openpmix/openpmix?tab=License-1-ov-file#License-1-ov-file"))))
 
 (define-public prrte
   (package
-- 
2.46.0

[bug#74572] [PATCH 4/6] gnu: openpmix: Do not install HTML documentation.

[Maxim Cournoyer]

This reduces the package size from 9 MiB to 5 MiB.

* gnu/packages/parallel.scm (openpmix) [source]: Add snippet.

Change-Id: I36e4a1e456225588df96f5eb70ed607505dae493
---
 gnu/packages/parallel.scm | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/parallel.scm b/gnu/packages/parallel.scm
index 3e50a75047..a6119743ce 100644
--- a/gnu/packages/parallel.scm
+++ b/gnu/packages/parallel.scm
@@ -705,7 +705,11 @@ (define-public openpmix
                     version "/pmix-" version ".tar.bz2"))
               (sha256
                (base32
-                "1j9xlhqrrmgjdkwakamn78y5gj756adi53hn25zksgr3is3l5d09"))))
+                "1j9xlhqrrmgjdkwakamn78y5gj756adi53hn25zksgr3is3l5d09"))
+              (snippet
+               '(begin (use-modules (guix build utils))
+                       ;; Remove ~5 MiB of pre-built HTML doc.
+                       (delete-file-recursively "docs/_build/html")))))
     (build-system gnu-build-system)
     (arguments
      (list
-- 
2.46.0

[bug#74572] [PATCH 5/6] gnu: openmpi-5: Reformat.

[Maxim Cournoyer]

Better comply to our 80 characters maximum width guideline.

* gnu/packages/mpi.scm (openmpi-5): Reformat.

Change-Id: Ib164c4d7dae7bece60738787fd950003fb75e057
---
 gnu/packages/mpi.scm | 103 ++++++++++++++++++++++---------------------
 1 file changed, 53 insertions(+), 50 deletions(-)

diff --git a/gnu/packages/mpi.scm b/gnu/packages/mpi.scm
index 41e57c3012..67e12657db 100644
--- a/gnu/packages/mpi.scm
+++ b/gnu/packages/mpi.scm
@@ -328,59 +328,62 @@ (define-public openmpi-5
        (sha256
         (base32 "0mw1z4ppnlvxngwd58kl5q26qmvf3bgjkd4r8wjpqis3pky86hdx"))))
 
-    (inputs (modify-inputs (package-inputs openmpi)
-              ;; As of Open MPI 5.0.X, PMIx is used to communicate
-              ;; with SLURM, so SLURM'S PMI is no longer needed.
-              (delete "slurm")
-              (append ucx)              ;for Infiniband support
-              (append openpmix)         ;for PMI support (launching via "srun")
-              (append prrte)))          ;for PMI support (launching via "srun")
-    (native-inputs (modify-inputs (package-native-inputs openmpi)
-                     (append python)))
+    (inputs
+     (modify-inputs (package-inputs openmpi)
+       ;; As of Open MPI 5.0.X, PMIx is used to communicate with SLURM, so
+       ;; SLURM'S PMI is no longer needed.
+       (delete "slurm")
+       (append ucx)                    ;for Infiniband support
+       (append openpmix)               ;for PMI support (launching via "srun")
+       (append prrte)))                ;for PMI support (launching via "srun")
+    (native-inputs
+     (modify-inputs (package-native-inputs openmpi)
+       (append python)))
 
     (outputs '("out" "debug"))
     (arguments
-     (list #:configure-flags
-           #~(list "--enable-mpi-ext=affinity"         ;cr doesn't work
-                   "--with-sge"
-                   "--disable-static"
-
-                   #$@(if (package? (this-package-input "valgrind"))
-                          #~("--enable-memchecker"
-                             "--with-valgrind")
-                          #~("--without-valgrind"))
-
-                   "--with-hwloc=external"
-                   "--with-libevent"
-
-                   ;; This replaces --enable-mpirun-prefix-by-default wich is deprecated
-                   ;; since 5.x.
-                   "--enable-prte-prefix-by-default"
-
-                   ;; Enable support for the 'Process Management Interface for Exascale'
-                   ;; (PMIx) used e.g. by Slurm for the management communication and
-                   ;; coordination of MPI processes.
-                   (string-append "--with-pmix=" #$(this-package-input "openpmix"))
-                   (string-append "--with-prrte=" #$(this-package-input "prrte"))
-
-                   ;; Since 5.x, Infiniband support is provided by ucx.
-                   ;; See https://docs.open-mpi.org/en/main/release-notes/networks.html#miscellaneous-network-notes
-                   (string-append "--with-ucx=" #$(this-package-input "ucx")))
-
-           #:phases
-           #~(modify-phases %standard-phases
-               (add-before 'build 'remove-absolute
-                 (lambda _
-                   ;; Remove compiler absolute file names (OPAL_FC_ABSOLUTE
-                   ;; etc.) to reduce the closure size.  See
-                   ;; <https://lists.gnu.org/archive/html/guix-devel/2017-07/msg00388.html>
-                   ;; and
-                   ;; <https://www.mail-archive.com/users@lists.open-mpi.org//msg31397.html>.
-                   (substitute* '("oshmem/tools/oshmem_info/param.c"
-                                  "ompi/tools/ompi_info/param.c")
-                     (("_ABSOLUTE") "")))))
-
-           #:disallowed-references (list (canonical-package gcc))))))
+     (list
+      #:configure-flags
+      #~(list "--enable-mpi-ext=affinity" ;cr doesn't work
+              "--with-sge"
+              "--disable-static"
+
+              #$@(if (package? (this-package-input "valgrind"))
+                     #~("--enable-memchecker"
+                        "--with-valgrind")
+                     #~("--without-valgrind"))
+
+              "--with-hwloc=external"
+              "--with-libevent"
+
+              ;; This replaces --enable-mpirun-prefix-by-default wich is
+              ;; deprecated since 5.x.
+              "--enable-prte-prefix-by-default"
+
+              ;; Enable support for the 'Process Management Interface for
+              ;; Exascale' (PMIx) used e.g. by Slurm for the management
+              ;; communication and coordination of MPI processes.
+              (string-append "--with-pmix=" #$(this-package-input "openpmix"))
+              (string-append "--with-prrte=" #$(this-package-input "prrte"))
+
+              ;; Since 5.x, Infiniband support is provided by ucx.  See
+              ;; https://docs.open-mpi.org/en/main/release-notes/networks.html#miscellaneous-network-notes
+              (string-append "--with-ucx=" #$(this-package-input "ucx")))
+
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-before 'build 'remove-absolute
+            (lambda _
+              ;; Remove compiler absolute file names (OPAL_FC_ABSOLUTE etc.)
+              ;; to reduce the closure size.  See
+              ;; <https://lists.gnu.org/archive/html/guix-devel/2017-07/msg00388.html>
+              ;; and
+              ;; <https://www.mail-archive.com/users@lists.open-mpi.org//msg31397.html>.
+              (substitute* '("oshmem/tools/oshmem_info/param.c"
+                             "ompi/tools/ompi_info/param.c")
+                (("_ABSOLUTE") "")))))
+
+      #:disallowed-references (list (canonical-package gcc))))))
 
 (define-public openmpi-c++
   (package/inherit openmpi
-- 
2.46.0

[bug#74572] [PATCH 6/6] gnu: openmpi-5: Streamline source and do not install HTML doc.

[Maxim Cournoyer]

This reduces the installed package size by 63 MiB.

* gnu/packages/mpi.scm (openmpi-5) [source]: Add modules, snippet fields.

Change-Id: Ie3dc365a2e4e48dc4c385dd447d3c64b7db065ca
---
 gnu/packages/mpi.scm | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/gnu/packages/mpi.scm b/gnu/packages/mpi.scm
index 67e12657db..2f37364ef3 100644
--- a/gnu/packages/mpi.scm
+++ b/gnu/packages/mpi.scm
@@ -325,6 +325,31 @@ (define-public openmpi-5
        (uri (string-append "https://www.open-mpi.org/software/ompi/v"
                            (version-major+minor version)
                            "/downloads/openmpi-" version ".tar.bz2"))
+       (modules '((guix build utils)
+                  (ice-9 ftw)
+                  (srfi srfi-26)))
+       (snippet
+        '(begin
+           ;; XXX: 'delete-all-but' is copied from the turbovnc package.
+           (define (delete-all-but directory . preserve)
+             (define (directory? x)
+               (and=> (stat x #f)
+                      (compose (cut eq? 'directory <>) stat:type)))
+             (with-directory-excursion directory
+               (let* ((pred
+                       (negate (cut member <> (append '("." "..") preserve))))
+                      (items (scandir "." pred)))
+                 (for-each (lambda (item)
+                             (if (directory? item)
+                                 (delete-file-recursively item)
+                                 (delete-file item)))
+                           items))))
+           ;; Delete as many bundled libraries as permitted by the build
+           ;; system.
+           (delete-all-but "3rd-party" "treematch" "Makefile.in" "Makefile.am")
+           ;; Do not install 64 MiB worth fo pre-generated HTML
+           ;; documentation.
+           (delete-file-recursively "docs/html")))
        (sha256
         (base32 "0mw1z4ppnlvxngwd58kl5q26qmvf3bgjkd4r8wjpqis3pky86hdx"))))
 
-- 
2.46.0

[bug#74572] [PATCH 6/6] gnu: openmpi-5: Streamline source and do not install HTML doc.

[Ludovic Courtès]

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> This reduces the installed package size by 63 MiB.
>
> * gnu/packages/mpi.scm (openmpi-5) [source]: Add modules, snippet fields.
>
> Change-Id: Ie3dc365a2e4e48dc4c385dd447d3c64b7db065ca

This and the rest LGTM.  Thanks!

[bug#74572] [PATCH 0/6] ] Python bindings, reduced size for OpenPMIx/OpenMPI.

[Ludovic Courtès]

Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

>   gnu: openmpi-5: Update to 5.0.6.
>   gnu: openpmix: Enable Python bindings.
>   gnu: openpmix: Fix indentation.
>   gnu: openpmix: Do not install HTML documentation.
>   gnu: openmpi-5: Reformat.
>   gnu: openmpi-5: Streamline source and do not install HTML doc.

I’m afraid you’ll have to rebase: I just pushed
<https://issues.guix.gnu.org/74546>, which also updated openmpi@5.  :-)

BTW, I was telling Efraim we should probably create an HPC team to
coordinate on critical components like MPI implementations and their
drivers (at work we usually look for performance regressions against
high-speed interconnects before submitting changes to openmpi & co.).
How does that sound?

Ludo’.