[bug#47237] [PATCH] gnu: metis: Use 64-bit floating pointer numbers on 64-bit architectures.

[bug#47237] [PATCH] gnu: metis: Use 64-bit floating pointer numbers on 64-bit architectures.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* gnu/packages/maths.scm (metis)[arguments]: Add #:modules and #:phases.
---
 gnu/packages/maths.scm | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

Hi!

Metis defaults to 32-bit floating point numbers.  However, on 64-bit
platforms, users probably expect 64-bit floating point numbers, hence
this patch.

We could make it configurable through different variants or via package
parameters when they’re available, but I wonder about the usefulness of
a variant that uses 32-bit floats on 64-bit architectures.  Can we keep
just this one variant?

Thanks,
Ludo’.

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index 565f109f47..4cd9112be9 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -12,7 +12,7 @@
 ;;; Copyright © 2015 Fabian Harfert <fhmgufs@web.de>
 ;;; Copyright © 2016 Roel Janssen <roel@gnu.org>
 ;;; Copyright © 2016, 2018, 2020 Kei Kebreau <kkebreau@posteo.net>
-;;; Copyright © 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2016 Leo Famulari <leo@famulari.name>
 ;;; Copyright © 2016, 2017 Thomas Danckaert <post@thomasdanckaert.be>
 ;;; Copyright © 2017, 2018, 2019, 2020 Paul Garlick <pgarlick@tourbillion-technology.com>
@@ -3312,7 +3312,26 @@ YACC = bison -pscotchyy -y -b y
      `(#:tests? #f                      ;no tests
        #:configure-flags `("-DSHARED=ON"
                            ,(string-append "-DGKLIB_PATH=" (getcwd)
-                                           "/metis-" ,version "/GKlib"))))
+                                           "/metis-" ,version "/GKlib"))
+
+       #:modules ((system base target)
+                  (guix build cmake-build-system)
+                  (guix build utils))
+       #:phases (modify-phases %standard-phases
+                  (add-after 'unpack 'set-real-type-width
+                    (lambda* (#:key build target #:allow-other-keys)
+                      ;; Enable 64-bit floating point numbers on 64-bit
+                      ;; architectures.  Leave the default 32-bit width on
+                      ;; other architectures.
+                      (let ((word-size
+                             (with-target (or target build %host-type)
+                               (lambda ()
+                                 (target-word-size)))))
+                        (when (= 8 word-size)
+                          (display "setting REALTYPEWIDTH to 64...\n")
+                          (substitute* "include/metis.h"
+                            (("define REALTYPEWIDTH.*$")
+                             "define REALTYPEWIDTH 64\n")))))))))
     (home-page "http://glaros.dtc.umn.edu/gkhome/metis/metis/overview")
     (synopsis "Graph partitioning and fill-reducing matrix ordering library")
     (description
-- 
2.30.2

[bug#47237] [PATCH] gnu: metis: Use 64-bit floating pointer numbers on 64-bit architectures.

[Emmanuel Agullo]

Hello Ludo, hello everybody,

Thank you very much for this patch and discussion.

In numerical linear algebra in particular and hpc in
general, the trend is more and more to use reduced 
floating point arithmetic whenever possible (for memory 
footprint, communication volume and possibly computational 
time). I have no experience with using floating point 
numbers in partitioners such as metis. But I strongly 
suspect that a flexible usage will be requested (the 
same way it is often requested for integers). All in all, 
I would say that the number of bits for the architecture, 
the floating point numbers and the integers shall not 
(a priori) be related. In particular, I would say, in general, 
if a (high-quality) library (such as metis) exposes multiple 
choices, there are chances that users may want to
choose between them (for good reasons).

In the long run, it would be nice to have a naming
convention all over guix to make clear (and explicit)
for libraries that expose it which floating point
and integer arithmetic is being used. There can
hardly be a general rule (as some codes may even
decide to expose, say, part of the integers in
32 bits and another part in 64 bits), but it would
already be a great step to have a convention for
the "main" integer choice and "main" floating-point
choice. Something like mypackage_i32_r64 (if there
is a general convention (beyond guix) to follow,
even better). [It may even be great to use two versions
of the same package to facilitate mixed arithmetic, 
but we can certainly pospone that discussion as it
would certainly open two many issues for now].

In the short run, if there can be only one, I
let metis users playing with floating point
numbers comment on which one shall it be.

Hope this helps.

Thank you again.

Best regards,

Manu


----- Mail original -----
> De: "Ludovic Courtès" <ludo@gnu.org>
> À: guix-patches@gnu.org
> Cc: "Eric Bavier" <bavier@member.fsf.org>, "Emmanuel Agullo" <emmanuel.agullo@inria.fr>, "Paul Garlick"
> <pgarlick@tourbillion-technology.com>, "Ludovic Courtes" <ludovic.courtes@inria.fr>
> Envoyé: Jeudi 18 Mars 2021 16:02:46
> Objet: [PATCH] gnu: metis: Use 64-bit floating pointer numbers on 64-bit architectures.

> From: Ludovic Courtès <ludovic.courtes@inria.fr>
> 
> * gnu/packages/maths.scm (metis)[arguments]: Add #:modules and #:phases.
> ---
> gnu/packages/maths.scm | 23 +++++++++++++++++++++--
> 1 file changed, 21 insertions(+), 2 deletions(-)
> 
> Hi!
> 
> Metis defaults to 32-bit floating point numbers.  However, on 64-bit
> platforms, users probably expect 64-bit floating point numbers, hence
> this patch.
> 
> We could make it configurable through different variants or via package
> parameters when they’re available, but I wonder about the usefulness of
> a variant that uses 32-bit floats on 64-bit architectures.  Can we keep
> just this one variant?
> 
> Thanks,
> Ludo’.
> 
> diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
> index 565f109f47..4cd9112be9 100644
> --- a/gnu/packages/maths.scm
> +++ b/gnu/packages/maths.scm
> @@ -12,7 +12,7 @@
> ;;; Copyright © 2015 Fabian Harfert <fhmgufs@web.de>
> ;;; Copyright © 2016 Roel Janssen <roel@gnu.org>
> ;;; Copyright © 2016, 2018, 2020 Kei Kebreau <kkebreau@posteo.net>
> -;;; Copyright © 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
> +;;; Copyright © 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès
> <ludo@gnu.org>
> ;;; Copyright © 2016 Leo Famulari <leo@famulari.name>
> ;;; Copyright © 2016, 2017 Thomas Danckaert <post@thomasdanckaert.be>
> ;;; Copyright © 2017, 2018, 2019, 2020 Paul Garlick
> <pgarlick@tourbillion-technology.com>
> @@ -3312,7 +3312,26 @@ YACC = bison -pscotchyy -y -b y
>      `(#:tests? #f                      ;no tests
>        #:configure-flags `("-DSHARED=ON"
>                            ,(string-append "-DGKLIB_PATH=" (getcwd)
> -                                           "/metis-" ,version "/GKlib"))))
> +                                           "/metis-" ,version "/GKlib"))
> +
> +       #:modules ((system base target)
> +                  (guix build cmake-build-system)
> +                  (guix build utils))
> +       #:phases (modify-phases %standard-phases
> +                  (add-after 'unpack 'set-real-type-width
> +                    (lambda* (#:key build target #:allow-other-keys)
> +                      ;; Enable 64-bit floating point numbers on 64-bit
> +                      ;; architectures.  Leave the default 32-bit width on
> +                      ;; other architectures.
> +                      (let ((word-size
> +                             (with-target (or target build %host-type)
> +                               (lambda ()
> +                                 (target-word-size)))))
> +                        (when (= 8 word-size)
> +                          (display "setting REALTYPEWIDTH to 64...\n")
> +                          (substitute* "include/metis.h"
> +                            (("define REALTYPEWIDTH.*$")
> +                             "define REALTYPEWIDTH 64\n")))))))))
>     (home-page "http://glaros.dtc.umn.edu/gkhome/metis/metis/overview")
>     (synopsis "Graph partitioning and fill-reducing matrix ordering library")
>     (description
> --
> 2.30.2

[bug#47237] [PATCH] gnu: metis: Use 64-bit floating pointer numbers on 64-bit architectures.

[zimoun]

Hi,

On Thu, 18 Mar 2021 at 16:27, Emmanuel Agullo <emmanuel.agullo@inria.fr> wrote:

> In numerical linear algebra in particular and hpc in
> general, the trend is more and more to use reduced 
> floating point arithmetic whenever possible (for memory 
> footprint, communication volume and possibly computational 
> time). I have no experience with using floating point 
> numbers in partitioners such as metis. But I strongly 

[...]

> choice. Something like mypackage_i32_r64 (if there
> is a general convention (beyond guix) to follow,
> even better). [It may even be great to use two versions
> of the same package to facilitate mixed arithmetic, 
> but we can certainly pospone that discussion as it
> would certainly open two many issues for now].

I am probably out-of-scope, and it does probably not make sense for
partitioners as Metis, how are considered mixed precision solvers?
i.e., computing some part using 64 bits floating points and other part
using 32 bits.  (My memory is not clear but I remember reading couple of
weeks ago an acceleration with 3 levels of float-pointing precision.)

In the i32_r64 example of convention, what should be considered as rXY?
The final accuracy, whatever the internals?

Are these mixed precision solvers used and in the trend?

Well, I do not know if these questions make sense.


Cheers,
simon

[bug#47237] [PATCH] gnu: metis: Use 64-bit floating pointer numbers on 64-bit architectures.

[Emmanuel Agullo]

Hi Simon,

> I am probably out-of-scope, and it does probably not make sense for
> partitioners as Metis, how are considered mixed precision solvers?
> i.e., computing some part using 64 bits floating points and other part
> using 32 bits.  (My memory is not clear but I remember reading couple of
> weeks ago an acceleration with 3 levels of float-pointing precision.)
> 
> In the i32_r64 example of convention, what should be considered as rXY?
> The final accuracy, whatever the internals?

You are right, in principle, there are many combinations we would
like to consider and I don't think combinatorial or numerical library 
developers have followed a unique common rule.

I would say that most fully-featured numerical libraries have an
interface for all four main floating-point arithmetic (s, d, c,
z, to follow blas/lapack convention). Therefore, for most of them,
a single package should be enough regarding floating-point 
arithmetic. The same is true for mixed precision routines
(such as the dsgesv mixed precision lapack solver with iterative refinement
http://www.netlib.org/lapack/explore-html/d7/d3b/group__double_g_esolve_ga05bea3dc0386868e4720f22c969cb9f5.html ).

On the other hand, as I understand, currently, integers are not
handled the same way. There is of course always the option
to force the compiler to turn all integers in a give arithmetic
but advanced combinatorial and numerical codes provider finer
strategies. In fully-featured combinatorial codes such scotch 
(and if I understand metis), the selection is made at compile
time. In guix, for instance, the scotch and scotch32 package 
position INTSIZE64 and INTSIZE32, respectively. [In this
case, symbol renaming would be necessary to implement
algorithms dealing with subgraphs in 32 bits and other
subgraphs in 64 bits (a "mixed integer arithmetic") but 
I would say that is out of the scope of immediate needs 
for the guix community, or, that should at least be discussed
with library developers ahead of time before being packaged
as they may provide facilities in upcoming releases]. The most 
advanced fully-featured numerical codes may also provide selective 
64-bit integers in order to handle large problems while maintaining 
most arrays in 32-bits (one may want to have a look for instance at the
Section 2.6.2 of the 5.3.5 mumps user's guide
for such an example
http://mumps.enseeiht.fr/doc/userguide_5.3.5.pdf ). This
is typically useful for sparse direct solvers (such as mumps)
which may have to cope with very large matrices with billions
of entries. On the other hand, it is much less common to process
such large dense matrices in dense (except possibly for level-1 
blas routines), this is why large integer arrays are most often
consitently 32 bits in dense routines.

To come back to your original question:

> In the i32_r64 example of convention, what should be considered as rXY?
> The final accuracy, whatever the internals?

I would say, at least for now, to focus only on the options
that library developers have proposed to position at compile
time in their makefiles. For scotch, it is already distinguished 
through the scotch and scotch32 pakages (however we may still want
to make sure the naming convention is clear). For metis, *if I 
understand correctly*, there is the option to position both integers and
floating-point numbers, hence maybe the four combinations
that could be considered (but I have no experience in dealing 
with floating-point numbers in metis).

Hope this helps.

Best regards,

Emmanuel

[bug#47237] [PATCH] gnu: metis: Use 64-bit floating pointer numbers on 64-bit architectures.

[Paul Garlick]

Hi Ludo,

I think it is a good idea to allow for different variants, even on 64-
bit architectures.

In one direction there is a performance advantage using 32-bit floats. 
In the other direction accuracy improves using the 64-bit data type.

There is a similar situation with the scotch library.  The scotch32
package variant is preferable, even on 64-bit systems, except for the
case of very large models.

Best regards,

Paul.

On Thu, 2021-03-18 at 16:02 +0100, Ludovic Courtès wrote:
> From: Ludovic Courtès <ludovic.courtes@inria.fr>
> 
> * gnu/packages/maths.scm (metis)[arguments]: Add #:modules and #:phases.
> ---
>  gnu/packages/maths.scm | 23 +++++++++++++++++++++--
>  1 file changed, 21 insertions(+), 2 deletions(-)
> 
> Hi!
> 
> Metis defaults to 32-bit floating point numbers.  However, on 64-bit
> platforms, users probably expect 64-bit floating point numbers, hence
> this patch.
> 
> We could make it configurable through different variants or via package
> parameters when they’re available, but I wonder about the usefulness of
> a variant that uses 32-bit floats on 64-bit architectures.  Can we keep
> just this one variant?
> 
> Thanks,
> Ludo’.
> 
> diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
> index 565f109f47..4cd9112be9 100644
> --- a/gnu/packages/maths.scm
> +++ b/gnu/packages/maths.scm
> @@ -12,7 +12,7 @@
>  ;;; Copyright © 2015 Fabian Harfert <fhmgufs@web.de>
>  ;;; Copyright © 2016 Roel Janssen <roel@gnu.org>
>  ;;; Copyright © 2016, 2018, 2020 Kei Kebreau <kkebreau@posteo.net>
> -;;; Copyright © 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
> +;;; Copyright © 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
>  ;;; Copyright © 2016 Leo Famulari <leo@famulari.name>
>  ;;; Copyright © 2016, 2017 Thomas Danckaert <post@thomasdanckaert.be>
>  ;;; Copyright © 2017, 2018, 2019, 2020 Paul Garlick <pgarlick@tourbillion-technology.com>
> @@ -3312,7 +3312,26 @@ YACC = bison -pscotchyy -y -b y
>       `(#:tests? #f                      ;no tests
>         #:configure-flags `("-DSHARED=ON"
>                             ,(string-append "-DGKLIB_PATH=" (getcwd)
> -                                           "/metis-" ,version "/GKlib"))))
> +                                           "/metis-" ,version "/GKlib"))
> +
> +       #:modules ((system base target)
> +                  (guix build cmake-build-system)
> +                  (guix build utils))
> +       #:phases (modify-phases %standard-phases
> +                  (add-after 'unpack 'set-real-type-width
> +                    (lambda* (#:key build target #:allow-other-keys)
> +                      ;; Enable 64-bit floating point numbers on 64-bit
> +                      ;; architectures.  Leave the default 32-bit width on
> +                      ;; other architectures.
> +                      (let ((word-size
> +                             (with-target (or target build %host-type)
> +                               (lambda ()
> +                                 (target-word-size)))))
> +                        (when (= 8 word-size)
> +                          (display "setting REALTYPEWIDTH to 64...\n")
> +                          (substitute* "include/metis.h"
> +                            (("define REALTYPEWIDTH.*$")
> +                             "define REALTYPEWIDTH 64\n")))))))))
>      (home-page "http://glaros.dtc.umn.edu/gkhome/metis/metis/overview")
>      (synopsis "Graph partitioning and fill-reducing matrix ordering library")
>      (description
> 

[bug#47237] [PATCH] gnu: metis: Use 64-bit floating pointer numbers on 64-bit architectures.

[Ludovic Courtès]

Hi Paul and all,

Paul Garlick <pgarlick@tourbillion-technology.com> skribis:

> I think it is a good idea to allow for different variants, even on 64-
> bit architectures.
>
> In one direction there is a performance advantage using 32-bit floats. 
> In the other direction accuracy improves using the 64-bit data type.

True.  That concurs with what Emmanuel explains.

> There is a similar situation with the scotch library.  The scotch32
> package variant is preferable, even on 64-bit systems, except for the
> case of very large models.

Though ‘scotch32’ is about 32-bit integers, not floating point numbers.
So I guess it’s not the same kind of tradeoff?

Anyway, it sounds like the best course of action is to withdraw this
patch and instead come up with multiple variants or package parameters.

Thanks for your feedback!

Ludo’.

bug#47237: [PATCH] gnu: metis: Use 64-bit floating pointer numbers on 64-bit architectures.

[Ludovic Courtès]

Hi again,

For now I’ve added a ‘metis-r64’ package with 64-bit reals to the
‘guix-hpc’ channel:

  https://gitlab.inria.fr/guix-hpc/guix-hpc/-/commit/f4a77fe95f0286bd63b07af6b9fbef25f3262682

Let’s see later if we can come up with a more generic solution in Guix
proper.

Ludo’.