CPU-specific builds

CPU-specific builds

[Ben Woodcroft]

Hi,

Currently, we build a single set of x86_64 packages assuming SSE but not 
SSE2 instructions, but sometimes it would be nice to use more recent 
instructions like AVX. So I'm wondering if there is some way to specify 
a system more specific than 'X86_64'? I tried simply adding 
'--with-arch=haswell' as a configure argument in gcc-4.9 so that flag 
became the default for gcc usage and saw some performance improvements, 
though I did have to disable tests in gnutls.

Hardcoding that configure flag is definitely less than ideal, I'm 
wondering if there is some better way that would enable us to share 
package updates and even substitutes for these systems? My initial 
thought is extending the triplet 'x86_64-unknown-linux-gnu' somehow, but 
I suspect others have better ideas?

Thanks, ben.

Re: CPU-specific builds

[Ludovic Courtès]

Hi Ben,

Ben Woodcroft <b.woodcroft@uq.edu.au> skribis:

> Currently, we build a single set of x86_64 packages assuming SSE but
> not SSE2 instructions, but sometimes it would be nice to use more
> recent instructions like AVX.

Isn’t SSE2 part of the x86_64 base spec?  I always forget.

> So I'm wondering if there is some way to specify a system more
> specific than 'X86_64'? I tried simply adding '--with-arch=haswell' as
> a configure argument in gcc-4.9 so that flag became the default for
> gcc usage and saw some performance improvements, though I did have to
> disable tests in gnutls.

Do you have performance figures for some CPU-intensive applications?

What software are you most interested in?

> Hardcoding that configure flag is definitely less than ideal, I'm
> wondering if there is some better way that would enable us to share
> package updates and even substitutes for these systems? My initial
> thought is extending the triplet 'x86_64-unknown-linux-gnu' somehow,
> but I suspect others have better ideas?

I’m not sure how to do this.  Having, say, an x86_64avx-linux system
type (not triplet) would be impractical because it would be entirely
separate from x86_64-linux (different derivations).

Ideally, software for which using these CPU extensions makes a
significant difference would do what glibc does, which is to provide
several implementations of the relevant functions (one for SSE2, one for
AVX, etc.) and have the right one be selected at load time via an IFUNC
or similar mechanism.

Thanks,
Ludo’.

Re: CPU-specific builds

[Eric Bavier]

On 2016-10-02 08:33, ludo@gnu.org wrote:
> Hi Ben,
> 
> Ben Woodcroft <b.woodcroft@uq.edu.au> skribis:
> 
>> Currently, we build a single set of x86_64 packages assuming SSE but
>> not SSE2 instructions, but sometimes it would be nice to use more
>> recent instructions like AVX.
> 
> Ideally, software for which using these CPU extensions makes a
> significant difference would do what glibc does, which is to provide
> several implementations of the relevant functions (one for SSE2, one 
> for
> AVX, etc.) and have the right one be selected at load time via an IFUNC
> or similar mechanism.

Our openblas package does this, too.

-- 
`~Eric

Re: CPU-specific builds

[Ben Woodcroft]

[-- Attachment #1: Type: text/plain, Size: 2886 bytes --]

Thanks for the responses Ludo and Eric.


On 02/10/16 23:33, Ludovic Courtès wrote:
>> Currently, we build a single set of x86_64 packages assuming SSE but
>> not SSE2 instructions, but sometimes it would be nice to use more
>> recent instructions like AVX.
> Isn’t SSE2 part of the x86_64 base spec?  I always forget.
I don't think so. I always refer to Mark's comment when I forget:
https://lists.gnu.org/archive/html/guix-devel/2016-07/msg01534.html

>> So I'm wondering if there is some way to specify a system more
>> specific than 'X86_64'? I tried simply adding '--with-arch=haswell' as
>> a configure argument in gcc-4.9 so that flag became the default for
>> gcc usage and saw some performance improvements, though I did have to
>> disable tests in gnutls.
> Do you have performance figures for some CPU-intensive applications?
>
> What software are you most interested in?
We tend to run software whose runtime is dependent on the input data, so 
it is hard to say. But up to days or weeks of walltime in some cases. It 
takes a lot of power researching climate change..

As an anecdote, adding "-march=haswell" shaved 13% off the runtime of 
diamond, ~20% if the CPUs were contended.

>> Hardcoding that configure flag is definitely less than ideal, I'm
>> wondering if there is some better way that would enable us to share
>> package updates and even substitutes for these systems? My initial
>> thought is extending the triplet 'x86_64-unknown-linux-gnu' somehow,
>> but I suspect others have better ideas?
> I’m not sure how to do this.  Having, say, an x86_64avx-linux system
> type (not triplet) would be impractical because it would be entirely
> separate from x86_64-linux (different derivations).
Yes my original plan was to rebuild all packages locally, but that was 
probably too optimistic as I reckon there will be too much work involved 
in maintaining it.

> Ideally, software for which using these CPU extensions makes a
> significant difference would do what glibc does, which is to provide
> several implementations of the relevant functions (one for SSE2, one for
> AVX, etc.) and have the right one be selected at load time via an IFUNC
> or similar mechanism.
That sounds useful in some cases, but it is probably too much of a 
stretch for most bioinformatics packages.

In the end I think I'll just compile the specific packages we are 
specifically interested in. I attached some example code in case anyone 
is interested. But this brought up a few questions:

1) I also tried using --expression e.g. guix build --expression '(@@ (my 
packages cpu-specific) diamond-cpu-specific)' but that fails to compile 
as if the GUIX_PACKAGE_PATH is ignored, is that unexpected?

2) Is something amiss with gcc-toolchain-6? Compiling with it, diamond 
complains of a missing stdlib.h.

Thanks,
ben

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: cpu-specific.scm --]
[-- Type: text/x-scheme; name="cpu-specific.scm", Size: 4704 bytes --]

;;; Copyright © 2016 Ben Woodcroft <donttrustben@gmail.com>
;;;
;;; This code is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(define-module (ben packages cpu-specific)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (guix packages)
  #:use-module (guix utils)
  #:use-module (guix download)
  #:use-module (guix git-download)
  #:use-module (guix hg-download)
  #:use-module (guix build-system ant)
  #:use-module (guix build-system gnu)
  #:use-module (guix build-system cmake)
  #:use-module (guix build-system perl)
  #:use-module (guix build-system python)
  #:use-module (guix build-system r)
  #:use-module (guix build-system ruby)
  #:use-module (guix build-system trivial)
  #:use-module (gnu packages)
  #:use-module (gnu packages autotools)
  #:use-module (gnu packages algebra)
  #:use-module (gnu packages base)
  #:use-module (gnu packages bash)
  #:use-module (gnu packages bison)
  #:use-module (gnu packages boost)
  #:use-module (gnu packages commencement)
  #:use-module (gnu packages compression)
  #:use-module (gnu packages cpio)
  #:use-module (gnu packages curl)
  #:use-module (gnu packages documentation)
  #:use-module (gnu packages datastructures)
  #:use-module (gnu packages file)
  #:use-module (gnu packages gawk)
  #:use-module (gnu packages gcc)
  #:use-module (gnu packages gd)
  #:use-module (gnu packages gtk)
  #:use-module (gnu packages glib)
  #:use-module (gnu packages groff)
  #:use-module (gnu packages image)
  #:use-module (gnu packages imagemagick)
  #:use-module (gnu packages java)
  #:use-module (gnu packages linux)
  #:use-module (gnu packages logging)
  #:use-module (gnu packages machine-learning)
  #:use-module (gnu packages man)
  #:use-module (gnu packages maths)
  #:use-module (gnu packages mpi)
  #:use-module (gnu packages ncurses)
  #:use-module (gnu packages pcre)
  #:use-module (gnu packages parallel)
  #:use-module (gnu packages pdf)
  #:use-module (gnu packages perl)
  #:use-module (gnu packages pkg-config)
  #:use-module (gnu packages popt)
  #:use-module (gnu packages protobuf)
  #:use-module (gnu packages python)
  #:use-module (gnu packages readline)
  #:use-module (gnu packages ruby)
  #:use-module (gnu packages serialization)
  #:use-module (gnu packages statistics)
  #:use-module (gnu packages tbb)
  #:use-module (gnu packages tex)
  #:use-module (gnu packages texinfo)
  #:use-module (gnu packages textutils)
  #:use-module (gnu packages time)
  #:use-module (gnu packages tls)
  #:use-module (gnu packages vim)
  #:use-module (gnu packages web)
  #:use-module (gnu packages xml)
  #:use-module (gnu packages xorg)
  #:use-module (gnu packages zip)
  #:use-module (srfi srfi-1)
  #:use-module (gnu packages bioinformatics))

;; "sandybridge" for Ben's laptop
(define cpu "sandybridge")

(define-public gcc-cpu-specific
  (let ((base gcc-5)) ; gcc-6 does not seem to work.
    (package
     (inherit base)
     (name "gcc-cpu-specific")
     (arguments
      (substitute-keyword-arguments (package-arguments base)
        ((#:configure-flags configure-flags)
         `(append ,configure-flags
                  (list (string-append
                         "--with-arch=" ,cpu)))))))))

(define-public (cpu-specific-package base-package)
  (package
    (inherit base-package)
    (name (package-name base-package))
    ;; We must set a higher package version so this package is used instead of
    ;; the package in Guix proper.
    (version (string-append (package-version base-package) "-cpu-specific"))
    (inputs
     `(,@(package-inputs base-package)
       ("gcc" ,((@@ (gnu packages commencement)
                    gcc-toolchain) gcc-cpu-specific))))))

(define-public diamond-cpu-specific (cpu-specific-package diamond))
(define-public fasttree-cpu-specific (cpu-specific-package fasttree))
(define-public blast+-cpu-specific (cpu-specific-package blast+))
(define-public bwa-cpu-specific (cpu-specific-package bwa))
(define-public metabat-cpu-specific (cpu-specific-package metabat))

Re: CPU-specific builds

[Ludovic Courtès]

Ben Woodcroft <b.woodcroft@uq.edu.au> skribis:

> On 02/10/16 23:33, Ludovic Courtès wrote:

[...]

>>> So I'm wondering if there is some way to specify a system more
>>> specific than 'X86_64'? I tried simply adding '--with-arch=haswell' as
>>> a configure argument in gcc-4.9 so that flag became the default for
>>> gcc usage and saw some performance improvements, though I did have to
>>> disable tests in gnutls.
>> Do you have performance figures for some CPU-intensive applications?
>>
>> What software are you most interested in?
> We tend to run software whose runtime is dependent on the input data,
> so it is hard to say. But up to days or weeks of walltime in some
> cases. It takes a lot of power researching climate change..
>
> As an anecdote, adding "-march=haswell" shaved 13% off the runtime of
> diamond, ~20% if the CPUs were contended.

That’s more than I expected.  It’d be useful to have a way to specify
this.

>> Ideally, software for which using these CPU extensions makes a
>> significant difference would do what glibc does, which is to provide
>> several implementations of the relevant functions (one for SSE2, one for
>> AVX, etc.) and have the right one be selected at load time via an IFUNC
>> or similar mechanism.
> That sounds useful in some cases, but it is probably too much of a
> stretch for most bioinformatics packages.

Yeah.

> In the end I think I'll just compile the specific packages we are
> specifically interested in. I attached some example code in case
> anyone is interested. But this brought up a few questions:
>
> 1) I also tried using --expression e.g. guix build --expression '(@@
> (my packages cpu-specific) diamond-cpu-specific)' but that fails to
> compile as if the GUIX_PACKAGE_PATH is ignored, is that unexpected?

What error did you get?

> 2) Is something amiss with gcc-toolchain-6? Compiling with it, diamond
> complains of a missing stdlib.h.

Everything’s fine AFAICS.  However, note that ‘gnu-build-system’ pulls
in GCC 4.9, glibc, etc.  If you add ‘gcc-toolchain’ to the inputs, that
surely conflicts, but I’m unsure which one “wins”; could you check the
‘environment-variables’ file in a build tree?

> (define-public gcc-cpu-specific
>   (let ((base gcc-5)) ; gcc-6 does not seem to work.
>     (package
>      (inherit base)
>      (name "gcc-cpu-specific")
>      (arguments
>       (substitute-keyword-arguments (package-arguments base)
>         ((#:configure-flags configure-flags)
>          `(append ,configure-flags
>                   (list (string-append
>                          "--with-arch=" ,cpu)))))))))
>
> (define-public (cpu-specific-package base-package)
>   (package
>     (inherit base-package)
>     (name (package-name base-package))
>     ;; We must set a higher package version so this package is used instead of
>     ;; the package in Guix proper.
>     (version (string-append (package-version base-package) "-cpu-specific"))
>     (inputs
>      `(,@(package-inputs base-package)
>        ("gcc" ,((@@ (gnu packages commencement)
>                     gcc-toolchain) gcc-cpu-specific))))))
>
> (define-public diamond-cpu-specific (cpu-specific-package diamond))
> (define-public fasttree-cpu-specific (cpu-specific-package fasttree))
> (define-public blast+-cpu-specific (cpu-specific-package blast+))
> (define-public bwa-cpu-specific (cpu-specific-package bwa))
> (define-public metabat-cpu-specific (cpu-specific-package metabat))

Looks like the right way to start.

We should start thinking about what the ideal interface would look like,
though.  At the highest level, I imagine we’d want something like:

  guix package --tune=haswell -i diamond

Under the hood that could work by having an arbitrary list of key/value
options passed to build systems, rather than just the system type and
cross-compilation target as is currently the case.

Food for thought…

Ludo’.

Re: CPU-specific builds

[Ludovic Courtès]

Ben Woodcroft <b.woodcroft@uq.edu.au> skribis:

> On 02/10/16 23:33, Ludovic Courtès wrote:

[...]

>>> So I'm wondering if there is some way to specify a system more
>>> specific than 'X86_64'? I tried simply adding '--with-arch=haswell' as
>>> a configure argument in gcc-4.9 so that flag became the default for
>>> gcc usage and saw some performance improvements, though I did have to
>>> disable tests in gnutls.
>> Do you have performance figures for some CPU-intensive applications?
>>
>> What software are you most interested in?
> We tend to run software whose runtime is dependent on the input data,
> so it is hard to say. But up to days or weeks of walltime in some
> cases. It takes a lot of power researching climate change..
>
> As an anecdote, adding "-march=haswell" shaved 13% off the runtime of
> diamond, ~20% if the CPUs were contended.

That’s more than I expected.  It’d be useful to have a way to specify
this.

>> Ideally, software for which using these CPU extensions makes a
>> significant difference would do what glibc does, which is to provide
>> several implementations of the relevant functions (one for SSE2, one for
>> AVX, etc.) and have the right one be selected at load time via an IFUNC
>> or similar mechanism.
> That sounds useful in some cases, but it is probably too much of a
> stretch for most bioinformatics packages.

Yeah.

> In the end I think I'll just compile the specific packages we are
> specifically interested in. I attached some example code in case
> anyone is interested. But this brought up a few questions:
>
> 1) I also tried using --expression e.g. guix build --expression '(@@
> (my packages cpu-specific) diamond-cpu-specific)' but that fails to
> compile as if the GUIX_PACKAGE_PATH is ignored, is that unexpected?

What error did you get?

> 2) Is something amiss with gcc-toolchain-6? Compiling with it, diamond
> complains of a missing stdlib.h.

Everything’s fine AFAICS.  However, note that ‘gnu-build-system’ pulls
in GCC 4.9, glibc, etc.  If you add ‘gcc-toolchain’ to the inputs, that
surely conflicts, but I’m unsure which one “wins”; could you check the
‘environment-variables’ file in a build tree?

> (define-public gcc-cpu-specific
>   (let ((base gcc-5)) ; gcc-6 does not seem to work.
>     (package
>      (inherit base)
>      (name "gcc-cpu-specific")
>      (arguments
>       (substitute-keyword-arguments (package-arguments base)
>         ((#:configure-flags configure-flags)
>          `(append ,configure-flags
>                   (list (string-append
>                          "--with-arch=" ,cpu)))))))))
>
> (define-public (cpu-specific-package base-package)
>   (package
>     (inherit base-package)
>     (name (package-name base-package))
>     ;; We must set a higher package version so this package is used instead of
>     ;; the package in Guix proper.
>     (version (string-append (package-version base-package) "-cpu-specific"))
>     (inputs
>      `(,@(package-inputs base-package)
>        ("gcc" ,((@@ (gnu packages commencement)
>                     gcc-toolchain) gcc-cpu-specific))))))
>
> (define-public diamond-cpu-specific (cpu-specific-package diamond))
> (define-public fasttree-cpu-specific (cpu-specific-package fasttree))
> (define-public blast+-cpu-specific (cpu-specific-package blast+))
> (define-public bwa-cpu-specific (cpu-specific-package bwa))
> (define-public metabat-cpu-specific (cpu-specific-package metabat))

Looks like the right way to start.

We should start thinking about what the ideal interface would look like,
though.  At the highest level, I imagine we’d want something like:

  guix package --tune=haswell -i diamond

Under the hood that could work by having an arbitrary list of key/value
options passed to build systems, rather than just the system type and
cross-compilation target as is currently the case.

I once proposed something quite similar, but in the context of providing
something equivalent to Gentoo “USE flags”:

  https://lists.gnu.org/archive/html/guix-devel/2015-01/msg00177.html

Food for thought…

Ludo’.