From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id sf3NN43Uq2F/dgEAgWs5BA (envelope-from ) for ; Sat, 04 Dec 2021 21:50:21 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id iNi8Mo3Uq2HidQAAbx9fmQ (envelope-from ) for ; Sat, 04 Dec 2021 20:50:21 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 545D7B473 for ; Sat, 4 Dec 2021 21:50:21 +0100 (CET) Received: from localhost ([::1]:37916 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mtbzA-0006Oi-GS for larch@yhetil.org; Sat, 04 Dec 2021 15:50:20 -0500 Received: from eggs.gnu.org ([209.51.188.92]:35578) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mtbyt-0005rq-8I for guix-patches@gnu.org; Sat, 04 Dec 2021 15:50:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:43685) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mtbys-0001kH-V0 for guix-patches@gnu.org; Sat, 04 Dec 2021 15:50:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mtbys-0003c2-TV for guix-patches@gnu.org; Sat, 04 Dec 2021 15:50:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52283] [PATCH 02/10] transformations: Add '--tune'. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 04 Dec 2021 20:50:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52283 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52283@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 52283-submit@debbugs.gnu.org id=B52283.163865098913790 (code B ref 52283); Sat, 04 Dec 2021 20:50:02 +0000 Received: (at 52283) by debbugs.gnu.org; 4 Dec 2021 20:49:49 +0000 Received: from localhost ([127.0.0.1]:55211 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mtbye-0003a8-11 for submit@debbugs.gnu.org; Sat, 04 Dec 2021 15:49:49 -0500 Received: from eggs.gnu.org ([209.51.188.92]:34708) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mtbya-0003Z8-Ma for 52283@debbugs.gnu.org; Sat, 04 Dec 2021 15:49:45 -0500 Received: from [2001:470:142:3::e] (port=45328 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mtbyU-0001fp-PQ; Sat, 04 Dec 2021 15:49:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=U17giIwh3cjH9FKsmqtHZ+X+JKCiNEMVWUdbymhoL4k=; b=XhaA9B6trOtwpwh9zNpi DPxn2CYmrUU5EflprwxrPRrIYPgHcih46kvAoYWrwZnX8evPCF3BqjP4Vovd601PvuIxh+SCesC2Q VQtichHZLDyNy4HVftyu8v5M6OvLlYjcNl5D7O9gezeO3ct2op71jjqvQ0rvGhjYYIbZpOXZ+Wru6 QH+iPoZjO9XULQCMl/HAClMqK8lah7vyNIJXCaC714LnXAeNdEA4wAqcg2pfvjdY4M1f6OYKtIr+L nzqSBVRdbZZ3Ep4tlRkFfCigqyrIUHf8pNyF1E8H48jhWckKTbTlrZLQzO96eUeRPaqovci2kG9TI rPRjClslCQDsLQ==; Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=45592 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mtbyU-00059z-Lc; Sat, 04 Dec 2021 15:49:38 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Sat, 4 Dec 2021 21:49:16 +0100 Message-Id: <20211204204924.15581-2-ludo@gnu.org> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211204204924.15581-1-ludo@gnu.org> References: <20211204204924.15581-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1638651021; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=U17giIwh3cjH9FKsmqtHZ+X+JKCiNEMVWUdbymhoL4k=; b=mm1zY5D5tD/gjo+O/hNj7uJAtx94jnGpTuMI/GV02QmgMUIECna+YWFB76Or+VdZrmpK/e FZvV0s1PTWh/HWr5BI1fVqGzsrLAJA/IX2r2E9TpRLEG/bAvN3YYSdC7QNtFN+4eTdm9Tm jakCEh9zbqIkBmWPwzXVCoYw43ZCCYTCt98mSmmwIG+cEzVCZnVc/OWudw50GTLKYkbNkK 4APicEdO3EwkAxcoLeVeGg3foFHP4RtUn/PNKe94L6R4YWcpTVk6k1fymKkRVX9HbHL80h 4+ze7TMEIKcufkYZ+zNwFHc2di5rbI8ViEL2l5wjukO4vQm1PITxhJKKBuYJrw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1638651021; a=rsa-sha256; cv=none; b=rMzZwe7ioHvzut8+I1yku8K+GqgCI6DIXJ6Qp0TQB48SKW/GY9BtdwgUg8h102PAtQcF5E cKylFqUxugvKK+7El+HjDQHgxXfQ+4SR3LJx9+4EO1P4INHTSPbpzeS25t0e6RtYAaEjFL oKAixkSOQPZNJRA+qLLaz5zUbpJROvnFyoccfeQKclmxiKWiKwgJe4MSv4UPIISAW0MTko 01kjOe8OQM2XvIck+0SeG//PqAjRfyVXNygTwdRNTkRllU3ft1sLTp5BpsAxQ7/h1PPyOx JDog5iJGGlUzxVxzvw3MtQvNwehEyrwR4lzW3T4zdgeZik904tSWR9UK6mstug== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=XhaA9B6t; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -2.63 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=XhaA9B6t; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 545D7B473 X-Spam-Score: -2.63 X-Migadu-Scanner: scn0.migadu.com X-TUID: jjTzcUpHiF8S From: Ludovic Courtès * guix/transformations.scm (tuning-compiler) (tuned-package, tunable-package?, package-tuning) (transform-package-tuning): New procedures. (%transformations): Add 'tune'. (%transformation-options): Add "--tune". * tests/transformations.scm ("options->transformation, tune"): New test. * doc/guix.texi (Package Transformation Options): Document '--tune'. --- doc/guix.texi | 54 +++++++++++++++ guix/transformations.scm | 134 ++++++++++++++++++++++++++++++++++++++ tests/transformations.scm | 20 ++++++ 3 files changed, 208 insertions(+) diff --git a/doc/guix.texi b/doc/guix.texi index a675631b79..e3aca8fd3b 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -10906,6 +10906,60 @@ available options and a synopsis (these options are not shown in the @table @code +@cindex performance, tuning code +@cindex optimization, of package code +@cindex tuning, of package code +@cindex SIMD support +@cindex tunable packages +@cindex package multi-versioning +@item --tune[=@var{cpu}] +Use versions of the packages marked as ``tunable'' optimized for +@var{cpu}. When @var{cpu} is @code{native}, or when it is omitted, tune +for the CPU on which the @command{guix} command is running. + +Valid @var{cpu} names are those recognized by GCC, the GNU Compiler +Collection. On x86_64 processors, this includes CPU names such as +@code{nehalem}, @code{haswell}, and @code{skylake} (@pxref{x86 Options, +@code{-march},, gcc, Using the GNU Compiler Collection (GCC)}). + +As new generations of CPUs come out, they augment the standard +instruction set architecture (ISA) with additional instructions, in +particular instructions for single-instruction/multiple-data (SIMD) +parallel processing. For example, while Core2 and Skylake CPUs both +implement the x86_64 ISA, only the latter supports AVX2 SIMD +instructions. + +The primary gain one can expect from @option{--tune} is for programs +that can make use of those SIMD capabilities @emph{and} that do not +already have a mechanism to select the right optimized code at run time. +Packages that have the @code{tunable?} property set are considered +@dfn{tunable packages} by the @option{--tune} option; a package +definition with the property set looks like this: + +@lisp +(package + (name "hello-simd") + ;; ... + + ;; This package may benefit from SIMD extensions so + ;; mark it as "tunable". + (properties '((tunable? . #t)))) +@end lisp + +Other packages are not considered tunable. This allows Guix to use +generic binaries in the cases where tuning for a specific CPU is +unlikely to provide any gain. + +Tuned packages are @emph{grafted} onto packages that depend on them +(@pxref{Security Updates, grafts}). Thus, using @option{--no-grafts} +annihilates the effect of @option{--tune}. + +We call this technique @dfn{package multi-versioning}: several variants +of tunable packages may be built, one for each CPU variant. It is the +coarse-grain counterpart of @dfn{function multi-versioning} as +implemented by the GNU tool chain (@pxref{Function Multiversioning,,, +gcc, Using the GNU Compiler Collection (GCC)}). + @item --with-source=@var{source} @itemx --with-source=@var{package}=@var{source} @itemx --with-source=@var{package}@@@var{version}=@var{source} diff --git a/guix/transformations.scm b/guix/transformations.scm index 5ae1977cb2..3be02179ef 100644 --- a/guix/transformations.scm +++ b/guix/transformations.scm @@ -29,6 +29,7 @@ (define-module (guix transformations) #:autoload (guix upstream) (package-latest-release upstream-source-version upstream-source-signature-urls) + #:autoload (guix cpu) (current-cpu cpu->gcc-architecture) #:use-module (guix utils) #:use-module (guix memoization) #:use-module (guix gexp) @@ -49,6 +50,9 @@ (define-module (guix transformations) #:export (options->transformation manifest-entry-with-transformations + tunable-package? + tuned-package + show-transformation-options-help %transformation-options)) @@ -419,6 +423,120 @@ (define replacements obj) obj))) +(define tuning-compiler + (mlambda (micro-architecture) + "Return a compiler wrapper that passes '-march=MICRO-ARCHITECTURE' to the +actual compiler." + (define wrapper + #~(begin + (use-modules (ice-9 match)) + + (define* (search-next command + #:optional + (path (string-split (getenv "PATH") + #\:))) + ;; Search the next COMMAND on PATH, a list of + ;; directories representing the executable search path. + (define this + (stat (car (command-line)))) + + (let loop ((path path)) + (match path + (() + (match command + ("cc" (search-next "gcc")) + (_ #f))) + ((directory rest ...) + (let* ((file (string-append + directory "/" command)) + (st (stat file #f))) + (if (and st (not (equal? this st))) + file + (loop rest))))))) + + (match (command-line) + ((command arguments ...) + (match (search-next (basename command)) + (#f (exit 127)) + (next + (apply execl next + (append (cons next arguments) + (list (string-append "-march=" + #$micro-architecture)))))))))) + + (define program + (program-file (string-append "tuning-compiler-wrapper-" micro-architecture) + wrapper)) + + (computed-file (string-append "tuning-compiler-" micro-architecture) + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils)) + + (define bin (string-append #$output "/bin")) + (mkdir-p bin) + + (for-each (lambda (program) + (symlink #$program + (string-append bin "/" program))) + '("cc" "gcc" "clang" "g++" "c++" "clang++"))))))) + +(define (tuned-package p micro-architecture) + "Return package P tuned for MICRO-ARCHITECTURE." + (define compiler + (tuning-compiler micro-architecture)) + + (package + (inherit p) + (native-inputs + ;; Arrange so that COMPILER comes first in $PATH. + `(("tuning-compiler" ,compiler) + ,@(package-native-inputs p))) + (arguments + (substitute-keyword-arguments (package-arguments p) + ((#:tests? _ #f) #f))) + (properties + `((cpu-tuning . ,micro-architecture) + ,@(package-properties p))))) + +(define (tunable-package? package) + "Return true if package PACKAGE is \"tunable\"--i.e., if tuning it for the +host CPU is worthwhile." + (assq 'tunable? (package-properties package))) + +(define package-tuning + (mlambda (micro-architecture) + "Return a procedure that maps the given package to its counterpart tuned +for MICRO-ARCHITECTURE, a string suitable for GCC's '-march'." + (define rewriting-property + (gensym " package-tuning")) + + (package-mapping (lambda (p) + (cond ((assq rewriting-property (package-properties p)) + p) + ((assq 'tunable? (package-properties p)) + (package/inherit p + (replacement (tuned-package p micro-architecture)) + (properties `((,rewriting-property . #t) + ,@(package-properties p))))) + (else + p))) + (lambda (p) + (assq rewriting-property (package-properties p))) + #:deep? #t))) + +(define (transform-package-tuning micro-architectures) + "Return a procedure that, when " + (match micro-architectures + ((micro-architecture _ ...) + (info (G_ "tuning for CPU micro-architecture ~a~%") + micro-architecture) + (let ((rewrite (package-tuning micro-architecture))) + (lambda (obj) + (if (package? obj) + (rewrite obj) + obj)))))) + (define (transform-package-with-debug-info specs) "Return a procedure that, when passed a package, set its 'replacement' field to the same package but with #:strip-binaries? #f in its 'arguments' field." @@ -601,6 +719,7 @@ (define %transformations (with-commit . ,transform-package-source-commit) (with-git-url . ,transform-package-source-git-url) (with-c-toolchain . ,transform-package-toolchain) + (tune . ,transform-package-tuning) (with-debug-info . ,transform-package-with-debug-info) (without-tests . ,transform-package-tests) (with-patch . ,transform-package-patches) @@ -640,6 +759,21 @@ (define %transformation-options (parser 'with-git-url)) (option '("with-c-toolchain") #t #f (parser 'with-c-toolchain)) + (option '("tune") #f #t + (lambda (opt name arg result . rest) + (define micro-architecture + (match arg + ((or #f "native") + (cpu->gcc-architecture (current-cpu))) + ("generic" #f) + (_ arg))) + + (apply values + (if micro-architecture + (alist-cons 'tune micro-architecture + result) + (alist-delete 'tune result)) + rest))) (option '("with-debug-info") #t #f (parser 'with-debug-info)) (option '("without-tests") #t #f diff --git a/tests/transformations.scm b/tests/transformations.scm index 09839dc1c5..760b523e6e 100644 --- a/tests/transformations.scm +++ b/tests/transformations.scm @@ -465,6 +465,26 @@ (define (package-name* obj) `((with-latest . "foo"))))) (package-version (t p))))) +(test-equal "options->transformation, tune" + '(cpu-tuning . "superfast") + (let* ((p0 (dummy-package "p0")) + (p1 (dummy-package "p1" + (inputs `(("p0" ,p0))) + (properties '((tunable? . #t))))) + (p2 (dummy-package "p2" + (inputs `(("p1" ,p1))))) + (t (options->transformation '((tune . "superfast")))) + (p3 (t p2))) + (and (not (package-replacement p3)) + (match (package-inputs p3) + ((("p1" tuned)) + (match (package-inputs tuned) + ((("p0" p0)) + (and (not (package-replacement p0)) + (assq 'cpu-tuning + (package-properties + (package-replacement tuned))))))))))) + (test-equal "options->transformation + package->manifest-entry" '((transformations . ((without-tests . "foo")))) (let* ((p (dummy-package "foo")) -- 2.33.0