Open MPI keeps references to GCC, GFortran, etc.

Open MPI keeps references to GCC, GFortran, etc.

[Ludovic Courtès]

Hello,

Open MPI retains references to GCC, GFortran, etc., which significantly
increases its closure size.

The references come from cpp macros such as OMPI_FC_ABSOLUTE (absolute
file name of the Fortran compiler), defined in opal_config.h and
returned by command-line tools ‘ompi_info’ and ‘oshmem_info’.

What would be the impact of changing these OMPI_*_ABSOLUTE uses to just
the basename of the compiler?  Are there tools or practices that rely on
the info provided by these two commands?

Thanks,
Ludo’.

Re: Open MPI keeps references to GCC, GFortran, etc.

[Dave Love]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Hello,
>
> Open MPI retains references to GCC, GFortran, etc., which significantly
> increases its closure size.

My query about cycles from separating the lib output was from looking at
basically this.  There should be a runtime package for compute nodes and
a development one (as "openmpi" and "openmpi-devel" in Fedora).

[It's not the only example of monstrous dependencies I've noticed
trying to build things.]

> The references come from cpp macros such as OMPI_FC_ABSOLUTE (absolute
> file name of the Fortran compiler), defined in opal_config.h and
> returned by command-line tools ‘ompi_info’ and ‘oshmem_info’.

I'm confused what the wrappers are doing, and would have to dig into the
code to find out.  "strings" on a version on our cluster doesn't actually
show gfortran, and "mpifort -showme" doesn't have a path name for it.  I
think I've been here before, but have forgotten.

> What would be the impact of changing these OMPI_*_ABSOLUTE uses to just
> the basename of the compiler?  Are there tools or practices that rely on
> the info provided by these two commands?

There should be no dependence on the C compiler.  I can't say about C++,
past the standard library version, but I think it's deprecated anyhow.
Apart from minor ABI incompatibilities, as far as I know (and when they
change the libfortran soname, sigh), the dependence on a specific
version of gfortran is down to the highly annoying backwards
incompatibility of the module file formats (e.g. mpi.mod).  You can
change the compiler at runtime with an environment variable (OMPI_FC
(?)) and it will either work or fail according to the version of .mod
files involved.  It's annoying that the only real difference between
painstakingly-built mpi packages for different gfortran versions is the
.mod files.

There are plenty of things which don't just use the compiler wrappers as
they should and, I think some try their "-showme" options to get
information.  I can't immediately think of which do, and how, but I can
have a look around at things I have packaged.  I doubt it's worth
worrying about, though.

I was intending to look at parameterizing the build on gfortran version,
which is basically a case of setting "FC" appropriately for configure
(possibly rebuilding just the Fortran part for a list of FCs within one
package build, and installing the module files in different places,
which I've never got round to looking at with rpm).  I'm not clear how
dependencies are, or should be, propagated for different compilers, but
guess it will be trivial.  To support multiple gfortran versions, this
also needs doing for the development part of any library packages that
provide Fortran modules.  [I can rant at length on the combinatorial
build explosion that typically infects HPC installations.  That's part
of the interest in Guix, and I was initially looking at improving MPI
stuff as something I understand from Fedora-land.]

Should I carry on looking at separating out and parameterizing parts of
openmpi, or will a Guix expert work on it?  I'm currently stuck with
figuring out the cycle from the build, but haven't put much effort into
it, and it's not as simple as just turning on the "lib" output.

I don't know whether that helps much, but I can probably answer
questions on anything I might know more about than others.

Re: Open MPI keeps references to GCC, GFortran, etc.

[Ludovic Courtès]

Hi,

Dave Love <fx@gnu.org> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>> Hello,
>>
>> Open MPI retains references to GCC, GFortran, etc., which significantly
>> increases its closure size.
>
> My query about cycles from separating the lib output was from looking at
> basically this.  There should be a runtime package for compute nodes and
> a development one (as "openmpi" and "openmpi-devel" in Fedora).

Interesting.  It’s not a “should” though IMO, in the sense that we add
additional inputs only when we have a good reason to do so.  For
example, in most cases, a “doc” output would only save you a few KiB of
man pages, representing .1% of the package size, so we don’t add a “doc”
output in those cases.  This is different from the Fedora/Debian
approach where packages are systematically split in several outputs.
The “Submitting Patches” section of the manual mentions this.

In the case of Open MPI, adding a “lib” output wouldn’t help: “out”
would still refer to gfortran via ‘ompi_info’, and “lib” would refer to
gfortran via opal_config.h.  Hence my suggestion to simply remove the
reference to gfortran & co.

> [It's not the only example of monstrous dependencies I've noticed
> trying to build things.]

Yeah I think it’s easy to overlook this when working on packages, but we
should constantly pay attention to this with ‘guix size’.

>> The references come from cpp macros such as OMPI_FC_ABSOLUTE (absolute
>> file name of the Fortran compiler), defined in opal_config.h and
>> returned by command-line tools ‘ompi_info’ and ‘oshmem_info’.
>
> I'm confused what the wrappers are doing,

The two commands return the actual compiler names, not the names of
wrappers:

--8<---------------cut here---------------start------------->8---
$ $(guix build openmpi)/bin/ompi_info

[...]

              C compiler: gcc
     C compiler absolute: /gnu/store/zlyn8z138s2vv86bdx9d72kchjczv0xn-gcc-5.4.0/bin/gcc
  C compiler family name: GNU
      C compiler version: 5.4.0
            C++ compiler: g++
   C++ compiler absolute: /gnu/store/zlyn8z138s2vv86bdx9d72kchjczv0xn-gcc-5.4.0/bin/g++
           Fort compiler: gfortran
       Fort compiler abs: /gnu/store/vcx02xlp6s92bxrcafnpw7hnqa4kc32b-gfortran-5.4.0/bin/gfortran
         Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::)
--8<---------------cut here---------------end--------------->8---

AIUI, this is “purely informational”.  Replacing the absolute file names
with just the base name won’t break anything.  Except that there’s
always the possibility of scripts or tools out there parsing the output
of ‘ompi_info’.  Thus my question was: do people really parse the output
of these commands, or can we safely assume that third-party programs
don’t rely on these absolute file names?

> I was intending to look at parameterizing the build on gfortran version,

I suppose you could to:

  (define openmpi-with-gfortran7
    (package
      (inherit openmpi)
      (name "openmpi-gfortran7")
      (inputs `(("gfortran" ,gfortran-7)
                ,@(alist-delete "gfortran" (package-inputs openmpi))))))

(That said, if the .mod files are compatible among gfortran versions, it
probably doesn’t make sense to do this.)

WDYT?

Ludo’.

Re: Open MPI keeps references to GCC, GFortran, etc.

[Dave Love]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Hi,
>
> Dave Love <fx@gnu.org> skribis:
>
>> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>>
>>> Hello,
>>>
>>> Open MPI retains references to GCC, GFortran, etc., which significantly
>>> increases its closure size.
>>
>> My query about cycles from separating the lib output was from looking at
>> basically this.  There should be a runtime package for compute nodes and
>> a development one (as "openmpi" and "openmpi-devel" in Fedora).
>
> Interesting.  It’s not a “should” though IMO, in the sense that we add
> additional inputs only when we have a good reason to do so.

I think I was misunderstanding.  Is the intention actually to get rid of
dependencies on the compilers?  (I assume that should apply to C as well
as Fortran.)  I guess that's arguable, but at least the compilers are
used by mpicc etc., and Fedora and Debian development packages depend
on, or recommend the compilers.

Anyhow, surely the first things to purge are perl, python, gdb, and
others that are currently in the closure.

Looking at the packaging more closely, I think it needs, or should have,
various changes.  --enable-static clobbers dynamically-loaded MCA
components, which I think is is a non-starter.  One question I have is
why are builtin atomics turned on?  They normally aren't, and I don't
know what the consequences are.

You can reduce the closure with other changes I made to bring it roughly
into line with Debian and Fedora and how I'd build it otherwise.
Removing the obsolete vampirtrace support, and the devel headers (which
are only for building external MCA components, which I've never seen
done), and replacing the valgrind integration with the library wrapper,
brings the "size" output down to:

  store item                                                       total    self
  /gnu/store/dws3a11p4s2qhnmapc4p1nm7g36hr3p4-openmpi-1.10.7         438.6     9.7   2.2%
  /gnu/store/vcx02xlp6s92bxrcafnpw7hnqa4kc32b-gfortran-5.4.0         262.9   105.4  24.0%
  /gnu/store/zlyn8z138s2vv86bdx9d72kchjczv0xn-gcc-5.4.0              198.2    92.1  21.0%
  /gnu/store/hdc5cqig58rc1093rx1l6ii8psdlgrav-hwloc-1.11.7-nogui      90.0     2.9   0.7%
  /gnu/store/padj9dg2p0dl82fhyhp73z2nqs5pyfrv-ncurses-6.0             74.3     5.7   1.3%
  /gnu/store/sg7rcm6s5iqjsmnviq9gcjw85bjzf5cn-mpc-1.0.3               73.2     0.4   0.1%
  /gnu/store/dxmkc0cq22dqnvidi0lqf1acrifn8l02-gfortran-5.4.0-lib      73.0    34.5   7.9%
  /gnu/store/w1mrskd2ddgvkr727r9241g8dlkf0rlf-gfortran-5.4.0-lib      73.0    34.5   7.9%
  /gnu/store/p5vnjcz57mgg20461c4ml71chy5b79rq-mpfr-3.1.5              72.8     1.6   0.4%
  /gnu/store/62d6n9qw68m37xavyv79g6nzpij5795k-gmp-6.1.2               71.2     2.6   0.6%
  /gnu/store/yd7bplsvf9nj72wn2z6n38rq9hfmjgd9-zlib-1.2.11             69.0     0.4   0.1%
  /gnu/store/4xkbjwf1iqmcz5is5apzlj6qhk47bz7d-numactl-2.0.11          68.9     0.3   0.1%
  /gnu/store/9sgscigfvrf6wklirgnmhnjqhajljhbv-libpciaccess-0.13.5     68.7     0.1   0.0%
  /gnu/store/m4svrffkzxg88av9rv4gzcpzl2cnvg8r-gcc-5.4.0-lib           68.6    30.1   6.9%
  /gnu/store/dhc2iy059hi91fk55dcv79z09kp6500y-gcc-5.4.0-lib           68.6    30.1   6.9%
  /gnu/store/58nhnjff2yahyxa1yjh2m8k93fb955z5-bash-4.4.12             50.9     5.4   1.2%
  /gnu/store/aiazy6r7flh8jk8fdaxp049g5lvsnkix-readline-7.0            45.5     1.3   0.3%
  /gnu/store/zhg9ds8madp5nk3pb9krabagr1vffygm-ncurses-6.0             44.2     5.7   1.3%
  /gnu/store/yz9yzv9nlvjpyw4vzjklhgk8dpl7hd7f-zlib-1.2.11             38.9     0.4   0.1%
  /gnu/store/ybpgv1v7606xw7mafda66w10hiynpiw2-glibc-2.25              38.5    37.1   8.5%
  /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25              38.5    37.1   8.5%
  /gnu/store/02426nwiy32cscm4h83729vn5ws1gs2i-bash-static-4.4.12       1.4     1.4   0.3%
  total: 438.6 MiB

I don't understand why gfortran and gfortran-lib are so large anyway --
they seem to duplicate C stuff, and gcc-lib bundles things I wouldn't
expect it to.  The RHEL gcc-5 equivalents are ~10 and 2 MB intrinsically.

I'll send the changes I have for consideration.

If you don't mean you want to remove the compiler dependencies, do you
mean outputs, rather than inputs above?  I was assuming a development
package depends on compilers.  Then the reason for separation is that
compute nodes, the way I've run systems, don't generally have a
development environment past what's required for dkms.  Also a user
installing some program in home shouldn't pull in the build environment,
which might clash with what they actually want.

I assume the store is intended to be on a shared filesystem which
compute nodes don't duplicate, which helps with space, but I don't think
that should be required.  The stateless systems I've set up used a
separate compute node image which was much smaller than the login node
one by omitting non-runtime rpms.

> For
> example, in most cases, a “doc” output would only save you a few KiB of
> man pages, representing .1% of the package size, so we don’t add a “doc”
> output in those cases.  This is different from the Fedora/Debian
> approach where packages are systematically split in several outputs.
> The “Submitting Patches” section of the manual mentions this.

I don't remember the Debian guidelines, but Fedora says to split out doc
files if they're "large" in total (~1MB?).  There is a requirement for a
separate -devel package for libraries (and for things like MPI with
compiler wrappers), and debug output is automatically separated.

> In the case of Open MPI, adding a “lib” output wouldn’t help: “out”
> would still refer to gfortran via ‘ompi_info’, and “lib” would refer to
> gfortran via opal_config.h.  Hence my suggestion to simply remove the
> reference to gfortran & co.

Yes, I now understand what the cycles are (thanks!), but I don't
understand why some of the strings are used in the source, or even what
the compiler paths are meant to be -- the package build (cross?)
compiler or the one invoked by mpifort?  [Cross-compilation might just
involve optimizing for a different micro-architecture, of course.]
Perhaps I should ask on the openmpi list, but I doubt I'll get far.

I think it's fine to remove the path from the point of view of (not)
breaking things, and other information strings could go, like overall
romio configure options.  (The relevant info about romio from ompi_info
is just the filesystem types supported.)

>> I'm confused what the wrappers are doing,
>
> The two commands return the actual compiler names, not the names of
> wrappers:
>
> --8<---------------cut here---------------start------------->8---
> $ $(guix build openmpi)/bin/ompi_info

I actually meant mpicc et al, which invoke the compiler.  I don't think
ompi_info should be in any separated runtime package, although it is in
RHEL.

> AIUI, this is “purely informational”.  Replacing the absolute file names
> with just the base name won’t break anything.  Except that there’s
> always the possibility of scripts or tools out there parsing the output
> of ‘ompi_info’.

I'd expect them to use "mpifort --showme" etc. for any configuration,
not ompi_info -- especially as it's not clear what that path means, or
why it matters.

> Thus my question was: do people really parse the output
> of these commands, or can we safely assume that third-party programs
> don’t rely on these absolute file names?

I'd say they shouldn't, and I can't think of any that do that I've
built, but I haven't tried to search lots of source.

>> I was intending to look at parameterizing the build on gfortran version,
>
> I suppose you could to:
>
>   (define openmpi-with-gfortran7
>     (package
>       (inherit openmpi)
>       (name "openmpi-gfortran7")
>       (inputs `(("gfortran" ,gfortran-7)
>                 ,@(alist-delete "gfortran" (package-inputs openmpi))))))

Right.

> (That said, if the .mod files are compatible among gfortran versions, it
> probably doesn’t make sense to do this.)

But they're not compatible, which is a real problem I'd have tried hard
to avoid if I was a maintainer.  All the gfortran versions I can see in
Guix have incompatible module formats.  It would probably be good to
provide symlinks like "gfortran-6", and even "gcc-6", as Debian does.

By the way, I don't want to be an HPC bigot, but HPC requirements seem
to be largely a superset of most others, and applicable in other areas.

Hope that helps a bit.

Re: Open MPI keeps references to GCC, GFortran, etc.

[Ludovic Courtès]

Hello Dave,

Dave Love <fx@gnu.org> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:

[...]

>> Interesting.  It’s not a “should” though IMO, in the sense that we add
>> additional inputs only when we have a good reason to do so.
>
> I think I was misunderstanding.  Is the intention actually to get rid of
> dependencies on the compilers?  (I assume that should apply to C as well
> as Fortran.)  I guess that's arguable, but at least the compilers are
> used by mpicc etc., and Fedora and Debian development packages depend
> on, or recommend the compilers.

My intent was to remove the *run-time* dependency of openmpi on gcc &
co. (as returned by ‘guix gc --references’ or ‘guix size openmpi’.)

> Looking at the packaging more closely, I think it needs, or should have,
> various changes.  --enable-static clobbers dynamically-loaded MCA
> components, which I think is is a non-starter.  One question I have is
> why are builtin atomics turned on?  They normally aren't, and I don't
> know what the consequences are.

No idea, you probably know better than me.

That said, I suggest addressing one problem at a time.  :-)

> You can reduce the closure with other changes I made to bring it roughly
> into line with Debian and Fedora and how I'd build it otherwise.
> Removing the obsolete vampirtrace support, and the devel headers (which
> are only for building external MCA components, which I've never seen
> done), and replacing the valgrind integration with the library wrapper,
> brings the "size" output down to:
>
>   store item                                                       total    self
>   /gnu/store/dws3a11p4s2qhnmapc4p1nm7g36hr3p4-openmpi-1.10.7         438.6     9.7   2.2%

Sounds good!

> I don't understand why gfortran and gfortran-lib are so large anyway --
> they seem to duplicate C stuff, and gcc-lib bundles things I wouldn't
> expect it to.  The RHEL gcc-5 equivalents are ~10 and 2 MB intrinsically.

Yeah, gfortran is actually a GCC with support for C/C++/Fortran.  That
also deserves to be optimized but it’s not trivial I think.

> I assume the store is intended to be on a shared filesystem which
> compute nodes don't duplicate, which helps with space, but I don't think
> that should be required.  The stateless systems I've set up used a
> separate compute node image which was much smaller than the login node
> one by omitting non-runtime rpms.

Yeah the store is typically meant to be shared over NFS or similar.  On
the topic of setting up Guix on a cluster, you might want to check:

  https://elephly.net/posts/2015-04-17-gnu-guix.html
  https://hal.inria.fr/hal-01161771/en

More on that later…

[...]

> I think it's fine to remove the path from the point of view of (not)
> breaking things, and other information strings could go, like overall
> romio configure options.  (The relevant info about romio from ompi_info
> is just the filesystem types supported.)

OK.

[...]

>>> I was intending to look at parameterizing the build on gfortran version,
>>
>> I suppose you could to:
>>
>>   (define openmpi-with-gfortran7
>>     (package
>>       (inherit openmpi)
>>       (name "openmpi-gfortran7")
>>       (inputs `(("gfortran" ,gfortran-7)
>>                 ,@(alist-delete "gfortran" (package-inputs openmpi))))))
>
> Right.
>
>> (That said, if the .mod files are compatible among gfortran versions, it
>> probably doesn’t make sense to do this.)
>
> But they're not compatible, which is a real problem

It shouldn’t be a problem if you do something like shown above, then.

> By the way, I don't want to be an HPC bigot, but HPC requirements seem
> to be largely a superset of most others, and applicable in other areas.

Agreed!  And I think Guix also makes it easier to meet some of the HPC
requirements, from what I’ve seen.

Thanks,
Ludo’.

Re: Open MPI keeps references to GCC, GFortran, etc.

[Dave Love]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> My intent was to remove the *run-time* dependency of openmpi on gcc &
> co. (as returned by ‘guix gc --references’ or ‘guix size openmpi’.)

OK, I can send that, though there might still be a case for a separate
runtime output.

>> Looking at the packaging more closely, I think it needs, or should have,
>> various changes.  --enable-static clobbers dynamically-loaded MCA
>> components, which I think is is a non-starter.  One question I have is
>> why are builtin atomics turned on?  They normally aren't, and I don't
>> know what the consequences are.
>
> No idea, you probably know better than me.

I'm afraid not.  It's a question for whoever enabled it.

>> I assume the store is intended to be on a shared filesystem which
>> compute nodes don't duplicate, which helps with space, but I don't think
>> that should be required.  The stateless systems I've set up used a
>> separate compute node image which was much smaller than the login node
>> one by omitting non-runtime rpms.
>
> Yeah the store is typically meant to be shared over NFS or similar.  On
> the topic of setting up Guix on a cluster, you might want to check:
>
>   https://elephly.net/posts/2015-04-17-gnu-guix.html
>   https://hal.inria.fr/hal-01161771/en

I think I've read both of them, and I'd argue with some of what they say
-- but I shouldn't here.

>>> I suppose you could to:
>>>
>>>   (define openmpi-with-gfortran7
>>>     (package
>>>       (inherit openmpi)
>>>       (name "openmpi-gfortran7")
>>>       (inputs `(("gfortran" ,gfortran-7)
>>>                 ,@(alist-delete "gfortran" (package-inputs openmpi))))))
>>
>> Right.
>>
>>> (That said, if the .mod files are compatible among gfortran versions, it
>>> probably doesn’t make sense to do this.)
>>
>> But they're not compatible, which is a real problem
>
> It shouldn’t be a problem if you do something like shown above, then.

It's a general problem that means you have to do something like the
above (which presumably makes the closure big again).

>> By the way, I don't want to be an HPC bigot, but HPC requirements seem
>> to be largely a superset of most others, and applicable in other areas.
>
> Agreed!  And I think Guix also makes it easier to meet some of the HPC
> requirements, from what I’ve seen.

Yes, but there are clear trades-off, and from what I've seen the promise
isn't really realized yet.  I hope it will be, with more engineering
effort, and thanks for all the effort on it.