* forcing local build from a package definition @ 2014-10-24 16:10 Federico Beffa 2014-10-24 18:57 ` Eric Bavier 0 siblings, 1 reply; 8+ messages in thread From: Federico Beffa @ 2014-10-24 16:10 UTC (permalink / raw) To: Guix-devel Hi, I'm preparing a package for ATLAS (Automatically Tuned Linear Algebra Software). The library provides functions from LAPACK which are automatically speed optimized for the hardware on which the library is built (the optimization takes place during the build phase). My question is the following: is there a way to force a local build from a package definition (without having to pass '--no-substitutes' to guix)? If not, what do you think about explicitly stating in the description of the package that it MUST be installed with the '--no-substitutes' flag and, if the user does not do so, hydra may provide a package which does not run on the user hardware? Most other distributions deliver binary packages which have been compiled for a very general CPU. However, this not satisfactory since it results in a much slower library. Guix is in a nice position to provide a good performance version of the library and I think we should do so. Regards, Fede ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: forcing local build from a package definition 2014-10-24 16:10 forcing local build from a package definition Federico Beffa @ 2014-10-24 18:57 ` Eric Bavier 2014-10-24 21:41 ` Federico Beffa 2014-10-25 6:23 ` Federico Beffa 0 siblings, 2 replies; 8+ messages in thread From: Eric Bavier @ 2014-10-24 18:57 UTC (permalink / raw) To: Federico Beffa; +Cc: Guix-devel Federico Beffa writes: > Hi, > > I'm preparing a package for ATLAS (Automatically Tuned Linear Algebra > Software). The library provides functions from LAPACK which are automatically > speed optimized for the hardware on which the library is built (the > optimization takes place during the build phase). ATLAS provides only a handful of the LAPACK routines. The primary focus is on BLAS routines. In order for ATLAS to provide a full lapack experience, it needs to be configured with the netlib lapack (which is where our current lapack package comes from). > My question is the following: is there a way to force a local build from a > package definition (without having to pass '--no-substitutes' to > guix)? I don't think this is necessary... > If not, what do you think about explicitly stating in the description of the > package that it MUST be installed with the '--no-substitutes' flag and, if the > user does not do so, hydra may provide a package which does not run on the > user hardware? ATLAS frequently pulls its tuning configuration information from a table of known values at configuration time. I think there is also a way to tell it which architecture it should be built for. > Most other distributions deliver binary packages which have been compiled for > a very general CPU. However, this not satisfactory since it results in a much > slower library. Guix is in a nice position to provide a good performance > version of the library and I think we should do so. You may be interested in exploring the BLIS library, https://code.google.com/p/blis/. It doesn't require build-time performance-tuning, and often significantly outperforms ATLAS http://www.cs.utexas.edu/users/flame/pubs/BLISTOMSrev2.pdf -- Eric Bavier Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: forcing local build from a package definition 2014-10-24 18:57 ` Eric Bavier @ 2014-10-24 21:41 ` Federico Beffa 2014-10-25 6:23 ` Federico Beffa 1 sibling, 0 replies; 8+ messages in thread From: Federico Beffa @ 2014-10-24 21:41 UTC (permalink / raw) To: Eric Bavier; +Cc: Guix-devel On Fri, Oct 24, 2014 at 8:57 PM, Eric Bavier <ericbavier@gmail.com> wrote: > ATLAS provides only a handful of the LAPACK routines. The primary > focus is on BLAS routines. In order for ATLAS to provide a full lapack > experience, it needs to be configured with the netlib lapack (which is > where our current lapack package comes from). yes, I know. But you can tell it to build a full lapack library (making use of the lapack source tar). But this is not the point. > ATLAS frequently pulls its tuning configuration information from a table > of known values at configuration time. I think there is also a way to > tell it which architecture it should be built for. Not really. You can specify the architecture at configure time. But, then it does not exploits the full instruction set and features of your CPU and results in a suboptimal library. For optimal performance it needs to run timing tests during build time. Note that since some time Debian also provides a was to build ATLAS on your own machine. http://math-atlas.sourceforge.net/atlas_install/atlas_install.html There may be other faster libraries, but ATLAS is used by many applications, including numpy/scipy which is actually the reason for which I'm working on ATLAS. Regards, Fede ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: forcing local build from a package definition 2014-10-24 18:57 ` Eric Bavier 2014-10-24 21:41 ` Federico Beffa @ 2014-10-25 6:23 ` Federico Beffa 2014-10-25 22:25 ` Ludovic Courtès 1 sibling, 1 reply; 8+ messages in thread From: Federico Beffa @ 2014-10-25 6:23 UTC (permalink / raw) To: Eric Bavier; +Cc: Guix-devel On Fri, Oct 24, 2014 at 8:57 PM, Eric Bavier <ericbavier@gmail.com> wrote: > ATLAS frequently pulls its tuning configuration information from a table > of known values at configuration time. I think there is also a way to > tell it which architecture it should be built for. Let me try to better explain the problem: ATLAS needs to know the exact type of CPU on which it is running for best performance. This is because it tries to make best use of all available features: Say, you have a recent CPU with 8 cores and instructions set including SSE1 SSE2 SSE3. If you specify this type of CPU, ATLAS will build a library with procedures which make use of 8 parallel threads and try to exploit all available instructions at best. However, the library will not run on an older CPU, say, with no SSE3. And perform poorly on a two cores CPU. To avoid this type of problem you may specify the simplest type of CPU that you want to support at configure time (and this is the approach of most distributions which are only delivering binary). However, if you then install the "low" spec generic library on your brand new CPU with 8 cores it will perform poorly. If we force the library to be built on the machine on which it will be used, then everyone will get the most out of the library. That is the reason for asking if there is a way to force a local build from within a package definition. Hope this better explains the reason for my question. Regards, Fede ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: forcing local build from a package definition 2014-10-25 6:23 ` Federico Beffa @ 2014-10-25 22:25 ` Ludovic Courtès 2014-10-26 8:34 ` Federico Beffa 0 siblings, 1 reply; 8+ messages in thread From: Ludovic Courtès @ 2014-10-25 22:25 UTC (permalink / raw) To: Federico Beffa; +Cc: Guix-devel Federico Beffa <beffa@ieee.org> skribis: > Let me try to better explain the problem: ATLAS needs to know the > exact type of CPU on which it is running for best performance. This is > because it tries to make best use of all available features: > Say, you have a recent CPU with 8 cores and instructions set including > SSE1 SSE2 SSE3. If you specify this type of CPU, ATLAS will build a > library with procedures which make use of 8 parallel threads and try > to exploit all available instructions at best. However, the library > will not run on an older CPU, say, with no SSE3. And perform poorly on > a two cores CPU. I think ATLAS should do like GMP, libc, etc.: build all the possible variants, and then use IFUNC or a similar mechanism to select the right variant at load time. That doesn’t answer your initial question, though. For now, I think it’s OK to let it do its configure-time tuning and add a statement in the description about substitutes, unless other packages depend on it (in the former case, people would be installing it explicitly, so they would most likely know what they’re doing.) WDYT? Thanks, Ludo’. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: forcing local build from a package definition 2014-10-25 22:25 ` Ludovic Courtès @ 2014-10-26 8:34 ` Federico Beffa 2014-10-26 13:59 ` Ludovic Courtès 0 siblings, 1 reply; 8+ messages in thread From: Federico Beffa @ 2014-10-26 8:34 UTC (permalink / raw) To: Ludovic Courtès; +Cc: Guix-devel On Sun, Oct 26, 2014 at 12:25 AM, Ludovic Courtès <ludo@gnu.org> wrote: > I think ATLAS should do like GMP, libc, etc.: build all the possible > variants, and then use IFUNC or a similar mechanism to select the right > variant at load time. I would like to highlight two points: 1. there are many variants (see below). 2. even for the same configure time configuration, the build phase does not necessarily produce identical libraries all the time. The build phase tries many variants of algorithms, times them and picks the best performing one. The result may be different depending on the overall performance of the system such as on memory access time, and several other factors. Here the list of configure time variants: $ ./xprint_enums Architectural enums (Config's enum MACHTYPE): 0 = 'UNKNOWN' 1 = 'POWER3' 2 = 'POWER4' 3 = 'POWER5' 4 = 'PPCG4' 5 = 'PPCG5' 6 = 'POWER6' 7 = 'POWER7' 8 = 'POWERe6500' 9 = 'IBMz9' 10 = 'IBMz10' 11 = 'IBMz196' 12 = 'x86x87' 13 = 'x86SSE1' 14 = 'x86SSE2' 15 = 'x86SSE3' 16 = 'P5' 17 = 'P5MMX' 18 = 'PPRO' 19 = 'PII' 20 = 'PIII' 21 = 'PM' 22 = 'CoreSolo' 23 = 'CoreDuo' 24 = 'Core2Solo' 25 = 'Core2' 26 = 'Corei1' 27 = 'Corei2' 28 = 'Corei3' 29 = 'Atom' 30 = 'P4' 31 = 'P4E' 32 = 'Efficeon' 33 = 'K7' 34 = 'HAMMER' 35 = 'AMD64K10h' 36 = 'AMDLLANO' 37 = 'AMDDOZER' 38 = 'AMDDRIVER' 39 = 'UNKNOWNx86' 40 = 'IA64Itan' 41 = 'IA64Itan2' 42 = 'USI' 43 = 'USII' 44 = 'USIII' 45 = 'USIV' 46 = 'UST1' 47 = 'UST2' 48 = 'UnknownUS' 49 = 'MIPSR1xK' 50 = 'MIPSICE9' 51 = 'ARMv7' Operating System enums (Config's enum OSTYPE): 0 = 'UNKNOWN' 1 = 'Linux' 2 = 'SunOS' 3 = 'SunOS4' 4 = 'OSF1' 5 = 'IRIX' 6 = 'AIX' 7 = 'Win9x' 8 = 'WinNT' 9 = 'Win64' 10 = 'HPUX' 11 = 'FreeBSD' 12 = 'OSX' Compiler integer defines: 0 = 'ICC' 1 = 'SMC' 2 = 'DMC' 3 = 'SKC' 4 = 'DKC' 5 = 'XCC' 6 = 'GCC' 7 = 'F77' ISA extensions are combined by adding their values together (bitvector): none: 1 VSX: 2 AltiVec: 4 AVXMAC: 8 AVXFMA4: 16 AVX: 32 SSE3: 64 SSE2: 128 SSE1: 256 3DNow: 512 NEON: 1024 > That doesn’t answer your initial question, though. For now, I think > it’s OK to let it do its configure-time tuning and add a statement in > the description about substitutes, unless other packages depend on it > (in the former case, people would be installing it explicitly, so they > would most likely know what they’re doing.) I'm preparing numpy/scipy and they depend on ATLAS. But initially it will be independent. Regards, Fede ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: forcing local build from a package definition 2014-10-26 8:34 ` Federico Beffa @ 2014-10-26 13:59 ` Ludovic Courtès 2014-10-27 20:28 ` Federico Beffa 0 siblings, 1 reply; 8+ messages in thread From: Ludovic Courtès @ 2014-10-26 13:59 UTC (permalink / raw) To: Federico Beffa; +Cc: Guix-devel Federico Beffa <beffa@ieee.org> skribis: > On Sun, Oct 26, 2014 at 12:25 AM, Ludovic Courtès <ludo@gnu.org> wrote: >> I think ATLAS should do like GMP, libc, etc.: build all the possible >> variants, and then use IFUNC or a similar mechanism to select the right >> variant at load time. > > I would like to highlight two points: > > 1. there are many variants (see below). I think this is not too different from libc and GMP. > 2. even for the same configure time configuration, the build phase > does not necessarily produce identical libraries all the time. The > build phase tries many variants of algorithms, times them and picks > the best performing one. The result may be different depending on the > overall performance of the system such as on memory access time, and > several other factors. Oh, OK; that’s definitely different. >> That doesn’t answer your initial question, though. For now, I think >> it’s OK to let it do its configure-time tuning and add a statement in >> the description about substitutes, unless other packages depend on it >> (in the former case, people would be installing it explicitly, so they >> would most likely know what they’re doing.) > > I'm preparing numpy/scipy and they depend on ATLAS. But initially it > will be independent. In that case, what about making a generic version (one that can be substituted, at the expense of being less optimized), which would be the one used by NumPy, and then the optimized version? In the not-too-distant future, we’ll be able to mark the package as non-substitutable, which will be best. There’s already a #:local-build? flag around, but its semantics are a bit fuzzy at the moment because it determines both whether to offload and whether to substitute. I would like that to be improved, by having two different options, before we start relying on it in packages. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: forcing local build from a package definition 2014-10-26 13:59 ` Ludovic Courtès @ 2014-10-27 20:28 ` Federico Beffa 0 siblings, 0 replies; 8+ messages in thread From: Federico Beffa @ 2014-10-27 20:28 UTC (permalink / raw) To: Ludovic Courtès; +Cc: Guix-devel On Sun, Oct 26, 2014 at 2:59 PM, Ludovic Courtès <ludo@gnu.org> wrote: > In the not-too-distant future, we’ll be able to mark the package as > non-substitutable, which will be best. There’s already a #:local-build? > flag around, but its semantics are a bit fuzzy at the moment because it > determines both whether to offload and whether to substitute. I would > like that to be improved, by having two different options, before we > start relying on it in packages. That sounds great! Thanks, Fede ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2014-10-27 20:28 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-10-24 16:10 forcing local build from a package definition Federico Beffa 2014-10-24 18:57 ` Eric Bavier 2014-10-24 21:41 ` Federico Beffa 2014-10-25 6:23 ` Federico Beffa 2014-10-25 22:25 ` Ludovic Courtès 2014-10-26 8:34 ` Federico Beffa 2014-10-26 13:59 ` Ludovic Courtès 2014-10-27 20:28 ` Federico Beffa
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/guix.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.