* Open MPI keeps references to GCC, GFortran, etc. @ 2017-07-27 9:15 Ludovic Courtès 2017-07-27 13:45 ` Dave Love 0 siblings, 1 reply; 6+ messages in thread From: Ludovic Courtès @ 2017-07-27 9:15 UTC (permalink / raw) To: guix-devel; +Cc: Ricardo Wurmus, Eric Bavier, Dave Love Hello, Open MPI retains references to GCC, GFortran, etc., which significantly increases its closure size. The references come from cpp macros such as OMPI_FC_ABSOLUTE (absolute file name of the Fortran compiler), defined in opal_config.h and returned by command-line tools ‘ompi_info’ and ‘oshmem_info’. What would be the impact of changing these OMPI_*_ABSOLUTE uses to just the basename of the compiler? Are there tools or practices that rely on the info provided by these two commands? Thanks, Ludo’. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Open MPI keeps references to GCC, GFortran, etc. 2017-07-27 9:15 Open MPI keeps references to GCC, GFortran, etc Ludovic Courtès @ 2017-07-27 13:45 ` Dave Love 2017-07-28 8:10 ` Ludovic Courtès 0 siblings, 1 reply; 6+ messages in thread From: Dave Love @ 2017-07-27 13:45 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel, Ricardo Wurmus, Eric Bavier Ludovic Courtès <ludovic.courtes@inria.fr> writes: > Hello, > > Open MPI retains references to GCC, GFortran, etc., which significantly > increases its closure size. My query about cycles from separating the lib output was from looking at basically this. There should be a runtime package for compute nodes and a development one (as "openmpi" and "openmpi-devel" in Fedora). [It's not the only example of monstrous dependencies I've noticed trying to build things.] > The references come from cpp macros such as OMPI_FC_ABSOLUTE (absolute > file name of the Fortran compiler), defined in opal_config.h and > returned by command-line tools ‘ompi_info’ and ‘oshmem_info’. I'm confused what the wrappers are doing, and would have to dig into the code to find out. "strings" on a version on our cluster doesn't actually show gfortran, and "mpifort -showme" doesn't have a path name for it. I think I've been here before, but have forgotten. > What would be the impact of changing these OMPI_*_ABSOLUTE uses to just > the basename of the compiler? Are there tools or practices that rely on > the info provided by these two commands? There should be no dependence on the C compiler. I can't say about C++, past the standard library version, but I think it's deprecated anyhow. Apart from minor ABI incompatibilities, as far as I know (and when they change the libfortran soname, sigh), the dependence on a specific version of gfortran is down to the highly annoying backwards incompatibility of the module file formats (e.g. mpi.mod). You can change the compiler at runtime with an environment variable (OMPI_FC (?)) and it will either work or fail according to the version of .mod files involved. It's annoying that the only real difference between painstakingly-built mpi packages for different gfortran versions is the .mod files. There are plenty of things which don't just use the compiler wrappers as they should and, I think some try their "-showme" options to get information. I can't immediately think of which do, and how, but I can have a look around at things I have packaged. I doubt it's worth worrying about, though. I was intending to look at parameterizing the build on gfortran version, which is basically a case of setting "FC" appropriately for configure (possibly rebuilding just the Fortran part for a list of FCs within one package build, and installing the module files in different places, which I've never got round to looking at with rpm). I'm not clear how dependencies are, or should be, propagated for different compilers, but guess it will be trivial. To support multiple gfortran versions, this also needs doing for the development part of any library packages that provide Fortran modules. [I can rant at length on the combinatorial build explosion that typically infects HPC installations. That's part of the interest in Guix, and I was initially looking at improving MPI stuff as something I understand from Fedora-land.] Should I carry on looking at separating out and parameterizing parts of openmpi, or will a Guix expert work on it? I'm currently stuck with figuring out the cycle from the build, but haven't put much effort into it, and it's not as simple as just turning on the "lib" output. I don't know whether that helps much, but I can probably answer questions on anything I might know more about than others. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Open MPI keeps references to GCC, GFortran, etc. 2017-07-27 13:45 ` Dave Love @ 2017-07-28 8:10 ` Ludovic Courtès 2017-07-31 10:06 ` Dave Love 0 siblings, 1 reply; 6+ messages in thread From: Ludovic Courtès @ 2017-07-28 8:10 UTC (permalink / raw) To: Dave Love; +Cc: guix-devel, Ricardo Wurmus, Eric Bavier Hi, Dave Love <fx@gnu.org> skribis: > Ludovic Courtès <ludovic.courtes@inria.fr> writes: > >> Hello, >> >> Open MPI retains references to GCC, GFortran, etc., which significantly >> increases its closure size. > > My query about cycles from separating the lib output was from looking at > basically this. There should be a runtime package for compute nodes and > a development one (as "openmpi" and "openmpi-devel" in Fedora). Interesting. It’s not a “should” though IMO, in the sense that we add additional inputs only when we have a good reason to do so. For example, in most cases, a “doc” output would only save you a few KiB of man pages, representing .1% of the package size, so we don’t add a “doc” output in those cases. This is different from the Fedora/Debian approach where packages are systematically split in several outputs. The “Submitting Patches” section of the manual mentions this. In the case of Open MPI, adding a “lib” output wouldn’t help: “out” would still refer to gfortran via ‘ompi_info’, and “lib” would refer to gfortran via opal_config.h. Hence my suggestion to simply remove the reference to gfortran & co. > [It's not the only example of monstrous dependencies I've noticed > trying to build things.] Yeah I think it’s easy to overlook this when working on packages, but we should constantly pay attention to this with ‘guix size’. >> The references come from cpp macros such as OMPI_FC_ABSOLUTE (absolute >> file name of the Fortran compiler), defined in opal_config.h and >> returned by command-line tools ‘ompi_info’ and ‘oshmem_info’. > > I'm confused what the wrappers are doing, The two commands return the actual compiler names, not the names of wrappers: --8<---------------cut here---------------start------------->8--- $ $(guix build openmpi)/bin/ompi_info [...] C compiler: gcc C compiler absolute: /gnu/store/zlyn8z138s2vv86bdx9d72kchjczv0xn-gcc-5.4.0/bin/gcc C compiler family name: GNU C compiler version: 5.4.0 C++ compiler: g++ C++ compiler absolute: /gnu/store/zlyn8z138s2vv86bdx9d72kchjczv0xn-gcc-5.4.0/bin/g++ Fort compiler: gfortran Fort compiler abs: /gnu/store/vcx02xlp6s92bxrcafnpw7hnqa4kc32b-gfortran-5.4.0/bin/gfortran Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::) --8<---------------cut here---------------end--------------->8--- AIUI, this is “purely informational”. Replacing the absolute file names with just the base name won’t break anything. Except that there’s always the possibility of scripts or tools out there parsing the output of ‘ompi_info’. Thus my question was: do people really parse the output of these commands, or can we safely assume that third-party programs don’t rely on these absolute file names? > I was intending to look at parameterizing the build on gfortran version, I suppose you could to: (define openmpi-with-gfortran7 (package (inherit openmpi) (name "openmpi-gfortran7") (inputs `(("gfortran" ,gfortran-7) ,@(alist-delete "gfortran" (package-inputs openmpi)))))) (That said, if the .mod files are compatible among gfortran versions, it probably doesn’t make sense to do this.) WDYT? Ludo’. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Open MPI keeps references to GCC, GFortran, etc. 2017-07-28 8:10 ` Ludovic Courtès @ 2017-07-31 10:06 ` Dave Love 2017-07-31 13:57 ` Ludovic Courtès 0 siblings, 1 reply; 6+ messages in thread From: Dave Love @ 2017-07-31 10:06 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel, Ricardo Wurmus, Eric Bavier Ludovic Courtès <ludovic.courtes@inria.fr> writes: > Hi, > > Dave Love <fx@gnu.org> skribis: > >> Ludovic Courtès <ludovic.courtes@inria.fr> writes: >> >>> Hello, >>> >>> Open MPI retains references to GCC, GFortran, etc., which significantly >>> increases its closure size. >> >> My query about cycles from separating the lib output was from looking at >> basically this. There should be a runtime package for compute nodes and >> a development one (as "openmpi" and "openmpi-devel" in Fedora). > > Interesting. It’s not a “should” though IMO, in the sense that we add > additional inputs only when we have a good reason to do so. I think I was misunderstanding. Is the intention actually to get rid of dependencies on the compilers? (I assume that should apply to C as well as Fortran.) I guess that's arguable, but at least the compilers are used by mpicc etc., and Fedora and Debian development packages depend on, or recommend the compilers. Anyhow, surely the first things to purge are perl, python, gdb, and others that are currently in the closure. Looking at the packaging more closely, I think it needs, or should have, various changes. --enable-static clobbers dynamically-loaded MCA components, which I think is is a non-starter. One question I have is why are builtin atomics turned on? They normally aren't, and I don't know what the consequences are. You can reduce the closure with other changes I made to bring it roughly into line with Debian and Fedora and how I'd build it otherwise. Removing the obsolete vampirtrace support, and the devel headers (which are only for building external MCA components, which I've never seen done), and replacing the valgrind integration with the library wrapper, brings the "size" output down to: store item total self /gnu/store/dws3a11p4s2qhnmapc4p1nm7g36hr3p4-openmpi-1.10.7 438.6 9.7 2.2% /gnu/store/vcx02xlp6s92bxrcafnpw7hnqa4kc32b-gfortran-5.4.0 262.9 105.4 24.0% /gnu/store/zlyn8z138s2vv86bdx9d72kchjczv0xn-gcc-5.4.0 198.2 92.1 21.0% /gnu/store/hdc5cqig58rc1093rx1l6ii8psdlgrav-hwloc-1.11.7-nogui 90.0 2.9 0.7% /gnu/store/padj9dg2p0dl82fhyhp73z2nqs5pyfrv-ncurses-6.0 74.3 5.7 1.3% /gnu/store/sg7rcm6s5iqjsmnviq9gcjw85bjzf5cn-mpc-1.0.3 73.2 0.4 0.1% /gnu/store/dxmkc0cq22dqnvidi0lqf1acrifn8l02-gfortran-5.4.0-lib 73.0 34.5 7.9% /gnu/store/w1mrskd2ddgvkr727r9241g8dlkf0rlf-gfortran-5.4.0-lib 73.0 34.5 7.9% /gnu/store/p5vnjcz57mgg20461c4ml71chy5b79rq-mpfr-3.1.5 72.8 1.6 0.4% /gnu/store/62d6n9qw68m37xavyv79g6nzpij5795k-gmp-6.1.2 71.2 2.6 0.6% /gnu/store/yd7bplsvf9nj72wn2z6n38rq9hfmjgd9-zlib-1.2.11 69.0 0.4 0.1% /gnu/store/4xkbjwf1iqmcz5is5apzlj6qhk47bz7d-numactl-2.0.11 68.9 0.3 0.1% /gnu/store/9sgscigfvrf6wklirgnmhnjqhajljhbv-libpciaccess-0.13.5 68.7 0.1 0.0% /gnu/store/m4svrffkzxg88av9rv4gzcpzl2cnvg8r-gcc-5.4.0-lib 68.6 30.1 6.9% /gnu/store/dhc2iy059hi91fk55dcv79z09kp6500y-gcc-5.4.0-lib 68.6 30.1 6.9% /gnu/store/58nhnjff2yahyxa1yjh2m8k93fb955z5-bash-4.4.12 50.9 5.4 1.2% /gnu/store/aiazy6r7flh8jk8fdaxp049g5lvsnkix-readline-7.0 45.5 1.3 0.3% /gnu/store/zhg9ds8madp5nk3pb9krabagr1vffygm-ncurses-6.0 44.2 5.7 1.3% /gnu/store/yz9yzv9nlvjpyw4vzjklhgk8dpl7hd7f-zlib-1.2.11 38.9 0.4 0.1% /gnu/store/ybpgv1v7606xw7mafda66w10hiynpiw2-glibc-2.25 38.5 37.1 8.5% /gnu/store/rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25 38.5 37.1 8.5% /gnu/store/02426nwiy32cscm4h83729vn5ws1gs2i-bash-static-4.4.12 1.4 1.4 0.3% total: 438.6 MiB I don't understand why gfortran and gfortran-lib are so large anyway -- they seem to duplicate C stuff, and gcc-lib bundles things I wouldn't expect it to. The RHEL gcc-5 equivalents are ~10 and 2 MB intrinsically. I'll send the changes I have for consideration. If you don't mean you want to remove the compiler dependencies, do you mean outputs, rather than inputs above? I was assuming a development package depends on compilers. Then the reason for separation is that compute nodes, the way I've run systems, don't generally have a development environment past what's required for dkms. Also a user installing some program in home shouldn't pull in the build environment, which might clash with what they actually want. I assume the store is intended to be on a shared filesystem which compute nodes don't duplicate, which helps with space, but I don't think that should be required. The stateless systems I've set up used a separate compute node image which was much smaller than the login node one by omitting non-runtime rpms. > For > example, in most cases, a “doc” output would only save you a few KiB of > man pages, representing .1% of the package size, so we don’t add a “doc” > output in those cases. This is different from the Fedora/Debian > approach where packages are systematically split in several outputs. > The “Submitting Patches” section of the manual mentions this. I don't remember the Debian guidelines, but Fedora says to split out doc files if they're "large" in total (~1MB?). There is a requirement for a separate -devel package for libraries (and for things like MPI with compiler wrappers), and debug output is automatically separated. > In the case of Open MPI, adding a “lib” output wouldn’t help: “out” > would still refer to gfortran via ‘ompi_info’, and “lib” would refer to > gfortran via opal_config.h. Hence my suggestion to simply remove the > reference to gfortran & co. Yes, I now understand what the cycles are (thanks!), but I don't understand why some of the strings are used in the source, or even what the compiler paths are meant to be -- the package build (cross?) compiler or the one invoked by mpifort? [Cross-compilation might just involve optimizing for a different micro-architecture, of course.] Perhaps I should ask on the openmpi list, but I doubt I'll get far. I think it's fine to remove the path from the point of view of (not) breaking things, and other information strings could go, like overall romio configure options. (The relevant info about romio from ompi_info is just the filesystem types supported.) >> I'm confused what the wrappers are doing, > > The two commands return the actual compiler names, not the names of > wrappers: > > --8<---------------cut here---------------start------------->8--- > $ $(guix build openmpi)/bin/ompi_info I actually meant mpicc et al, which invoke the compiler. I don't think ompi_info should be in any separated runtime package, although it is in RHEL. > AIUI, this is “purely informational”. Replacing the absolute file names > with just the base name won’t break anything. Except that there’s > always the possibility of scripts or tools out there parsing the output > of ‘ompi_info’. I'd expect them to use "mpifort --showme" etc. for any configuration, not ompi_info -- especially as it's not clear what that path means, or why it matters. > Thus my question was: do people really parse the output > of these commands, or can we safely assume that third-party programs > don’t rely on these absolute file names? I'd say they shouldn't, and I can't think of any that do that I've built, but I haven't tried to search lots of source. >> I was intending to look at parameterizing the build on gfortran version, > > I suppose you could to: > > (define openmpi-with-gfortran7 > (package > (inherit openmpi) > (name "openmpi-gfortran7") > (inputs `(("gfortran" ,gfortran-7) > ,@(alist-delete "gfortran" (package-inputs openmpi)))))) Right. > (That said, if the .mod files are compatible among gfortran versions, it > probably doesn’t make sense to do this.) But they're not compatible, which is a real problem I'd have tried hard to avoid if I was a maintainer. All the gfortran versions I can see in Guix have incompatible module formats. It would probably be good to provide symlinks like "gfortran-6", and even "gcc-6", as Debian does. By the way, I don't want to be an HPC bigot, but HPC requirements seem to be largely a superset of most others, and applicable in other areas. Hope that helps a bit. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Open MPI keeps references to GCC, GFortran, etc. 2017-07-31 10:06 ` Dave Love @ 2017-07-31 13:57 ` Ludovic Courtès 2017-07-31 18:49 ` Dave Love 0 siblings, 1 reply; 6+ messages in thread From: Ludovic Courtès @ 2017-07-31 13:57 UTC (permalink / raw) To: Dave Love; +Cc: guix-devel, Ricardo Wurmus, Eric Bavier Hello Dave, Dave Love <fx@gnu.org> skribis: > Ludovic Courtès <ludovic.courtes@inria.fr> writes: [...] >> Interesting. It’s not a “should” though IMO, in the sense that we add >> additional inputs only when we have a good reason to do so. > > I think I was misunderstanding. Is the intention actually to get rid of > dependencies on the compilers? (I assume that should apply to C as well > as Fortran.) I guess that's arguable, but at least the compilers are > used by mpicc etc., and Fedora and Debian development packages depend > on, or recommend the compilers. My intent was to remove the *run-time* dependency of openmpi on gcc & co. (as returned by ‘guix gc --references’ or ‘guix size openmpi’.) > Looking at the packaging more closely, I think it needs, or should have, > various changes. --enable-static clobbers dynamically-loaded MCA > components, which I think is is a non-starter. One question I have is > why are builtin atomics turned on? They normally aren't, and I don't > know what the consequences are. No idea, you probably know better than me. That said, I suggest addressing one problem at a time. :-) > You can reduce the closure with other changes I made to bring it roughly > into line with Debian and Fedora and how I'd build it otherwise. > Removing the obsolete vampirtrace support, and the devel headers (which > are only for building external MCA components, which I've never seen > done), and replacing the valgrind integration with the library wrapper, > brings the "size" output down to: > > store item total self > /gnu/store/dws3a11p4s2qhnmapc4p1nm7g36hr3p4-openmpi-1.10.7 438.6 9.7 2.2% Sounds good! > I don't understand why gfortran and gfortran-lib are so large anyway -- > they seem to duplicate C stuff, and gcc-lib bundles things I wouldn't > expect it to. The RHEL gcc-5 equivalents are ~10 and 2 MB intrinsically. Yeah, gfortran is actually a GCC with support for C/C++/Fortran. That also deserves to be optimized but it’s not trivial I think. > I assume the store is intended to be on a shared filesystem which > compute nodes don't duplicate, which helps with space, but I don't think > that should be required. The stateless systems I've set up used a > separate compute node image which was much smaller than the login node > one by omitting non-runtime rpms. Yeah the store is typically meant to be shared over NFS or similar. On the topic of setting up Guix on a cluster, you might want to check: https://elephly.net/posts/2015-04-17-gnu-guix.html https://hal.inria.fr/hal-01161771/en More on that later… [...] > I think it's fine to remove the path from the point of view of (not) > breaking things, and other information strings could go, like overall > romio configure options. (The relevant info about romio from ompi_info > is just the filesystem types supported.) OK. [...] >>> I was intending to look at parameterizing the build on gfortran version, >> >> I suppose you could to: >> >> (define openmpi-with-gfortran7 >> (package >> (inherit openmpi) >> (name "openmpi-gfortran7") >> (inputs `(("gfortran" ,gfortran-7) >> ,@(alist-delete "gfortran" (package-inputs openmpi)))))) > > Right. > >> (That said, if the .mod files are compatible among gfortran versions, it >> probably doesn’t make sense to do this.) > > But they're not compatible, which is a real problem It shouldn’t be a problem if you do something like shown above, then. > By the way, I don't want to be an HPC bigot, but HPC requirements seem > to be largely a superset of most others, and applicable in other areas. Agreed! And I think Guix also makes it easier to meet some of the HPC requirements, from what I’ve seen. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Open MPI keeps references to GCC, GFortran, etc. 2017-07-31 13:57 ` Ludovic Courtès @ 2017-07-31 18:49 ` Dave Love 0 siblings, 0 replies; 6+ messages in thread From: Dave Love @ 2017-07-31 18:49 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel, Ricardo Wurmus, Eric Bavier Ludovic Courtès <ludovic.courtes@inria.fr> writes: > My intent was to remove the *run-time* dependency of openmpi on gcc & > co. (as returned by ‘guix gc --references’ or ‘guix size openmpi’.) OK, I can send that, though there might still be a case for a separate runtime output. >> Looking at the packaging more closely, I think it needs, or should have, >> various changes. --enable-static clobbers dynamically-loaded MCA >> components, which I think is is a non-starter. One question I have is >> why are builtin atomics turned on? They normally aren't, and I don't >> know what the consequences are. > > No idea, you probably know better than me. I'm afraid not. It's a question for whoever enabled it. >> I assume the store is intended to be on a shared filesystem which >> compute nodes don't duplicate, which helps with space, but I don't think >> that should be required. The stateless systems I've set up used a >> separate compute node image which was much smaller than the login node >> one by omitting non-runtime rpms. > > Yeah the store is typically meant to be shared over NFS or similar. On > the topic of setting up Guix on a cluster, you might want to check: > > https://elephly.net/posts/2015-04-17-gnu-guix.html > https://hal.inria.fr/hal-01161771/en I think I've read both of them, and I'd argue with some of what they say -- but I shouldn't here. >>> I suppose you could to: >>> >>> (define openmpi-with-gfortran7 >>> (package >>> (inherit openmpi) >>> (name "openmpi-gfortran7") >>> (inputs `(("gfortran" ,gfortran-7) >>> ,@(alist-delete "gfortran" (package-inputs openmpi)))))) >> >> Right. >> >>> (That said, if the .mod files are compatible among gfortran versions, it >>> probably doesn’t make sense to do this.) >> >> But they're not compatible, which is a real problem > > It shouldn’t be a problem if you do something like shown above, then. It's a general problem that means you have to do something like the above (which presumably makes the closure big again). >> By the way, I don't want to be an HPC bigot, but HPC requirements seem >> to be largely a superset of most others, and applicable in other areas. > > Agreed! And I think Guix also makes it easier to meet some of the HPC > requirements, from what I’ve seen. Yes, but there are clear trades-off, and from what I've seen the promise isn't really realized yet. I hope it will be, with more engineering effort, and thanks for all the effort on it. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-07-31 18:49 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-07-27 9:15 Open MPI keeps references to GCC, GFortran, etc Ludovic Courtès 2017-07-27 13:45 ` Dave Love 2017-07-28 8:10 ` Ludovic Courtès 2017-07-31 10:06 ` Dave Love 2017-07-31 13:57 ` Ludovic Courtès 2017-07-31 18:49 ` Dave Love
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).