[bug#43679] [PATCH 0/5] Add '--with-toolchain' package transformation option

[bug#43679] [PATCH 0/5] Add '--with-toolchain' package transformation option

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

Hello!

This patch series adds the ‘--with-toolchain’ option.  I’ve
tested it with gcc-toolchain@10 and clang-toolchain, and I can say
it works as advertised.  :-)

One thing I wasn’t entirely sure about: ‘--with-toolchain’ changes
the toolchain of the specified package, not that of its dependents.
This assumes that the toolchains all follow the same ABI.  This is
the case for C, apparently, maybe not for C++.  Should it instead
change to toolchain of the package’s dependents as well?

Something like:

  guix build guile --with-toolchain=guile@3.0.4=clang-toolchain 

generates working code.

Another issue is that since we use ‘package-input-rewriting/spec’,
we can’t change the toolchain of core packages like Guile or Perl
without rebuilding the world.  For example, if we omit “@3.0.4”
in the example above, we rebuild a “guile” package deep down and
everything that follows (aka. “the world”).

Another option I considered was to graft the package that
‘--with-toolchain’ targets instead of rebuilding its dependents.
Again that’d only work if the resulting binaries are ABI-compatible,
but maybe that’s a reasonable assumption.  It would definitely save
build time.  Should it be grafted, or should there be a separate
option to do that?  Thoughts?

Last, when doing ‘--with-toolchain=foo=gcc-toolchain’, I noticed
that ‘foo’ would keep a reference to ‘gcc-toolchain’ for some obscure
reasons:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build hello --with-toolchain=hello=gcc-toolchain
/gnu/store/qi7pqqsxhbwmy75hl43j7l0aw1xr7r42-hello-2.10
$ grep -r $(guix build gcc-toolchain | head -2 |tail -1) /gnu/store/qi7pqqsxhbwmy75hl43j7l0aw1xr7r42-hello-2.10
Duuma dosiero /gnu/store/qi7pqqsxhbwmy75hl43j7l0aw1xr7r42-hello-2.10/bin/hello kongruas
$ strings /gnu/store/qi7pqqsxhbwmy75hl43j7l0aw1xr7r42-hello-2.10/bin/hello | grep $(guix build gcc-toolchain | head -2 |tail -1)
/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib:/gnu/store/qj38f3vi4q1d7z30hkpaxyajv49rwamb-gcc-10.2.0-lib/lib:/gnu/store/qj38f3vi4q1d7z30hkpaxyajv49rwamb-gcc-10.2.0-lib/lib/gcc/x86_64-unknown-linux-gnu/10.2.0/../../..:/gnu/store/pknm43xsza6nlc7bn27djip8fis92akd-gcc-toolchain-10.2.0/lib
--8<---------------cut here---------------end--------------->8---

Not a showstopper but would be nice to address.

Feedback welcome!

Ludo’.

Ludovic Courtès (5):
  gnu: gcc-toolchain: Add 'GUIX_LOCPATH' to the search paths.
  gnu: clang-toolchain: Add 'GUIX_LOCPATH' to the search paths.
  gnu: clang-toolchain: Create 'cc' and 'c++' symlinks.
  packages: Add 'package-with-toolchain'.
  guix build: Add '--with-toolchain'.

 doc/guix.texi                 | 61 +++++++++++++++++++++++++++++++++++
 gnu/packages/commencement.scm |  8 +++--
 gnu/packages/llvm.scm         | 12 ++++++-
 guix/build-system.scm         | 35 ++++++++++++++++++--
 guix/packages.scm             |  9 ++++++
 guix/scripts/build.scm        | 40 +++++++++++++++++++++++
 tests/packages.scm            | 20 ++++++++++++
 tests/scripts-build.scm       | 30 +++++++++++++++++
 8 files changed, 210 insertions(+), 5 deletions(-)

-- 
2.28.0

[bug#43679] [PATCH 1/5] gnu: gcc-toolchain: Add 'GUIX_LOCPATH' to the search paths.

[Ludovic Courtès]

This allows 'gcc-toolchain' to be used as a drop-in replacement for the
default tool chain through '--with-toolchain=gcc-toolchain'.

* gnu/packages/commencement.scm (make-gcc-toolchain)[native-search-paths]
[search-paths]: Append LIBC's search paths.
---
 gnu/packages/commencement.scm | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 4041d5bb89..0938bc3d46 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -3856,8 +3856,12 @@ COREUTILS-FINAL vs. COREUTILS, etc."
                                                      "libc-static")))
                        #t))))
 
-      (native-search-paths (package-native-search-paths gcc))
-      (search-paths (package-search-paths gcc))
+      (native-search-paths
+       (append (package-native-search-paths gcc)
+               (package-native-search-paths libc))) ;GUIX_LOCPATH
+      (search-paths
+       (append (package-search-paths gcc)
+               (package-search-paths libc)))
 
       (license (package-license gcc))
       (synopsis "Complete GCC tool chain for C/C++ development")
-- 
2.28.0

[bug#43679] [PATCH 2/5] gnu: clang-toolchain: Add 'GUIX_LOCPATH' to the search paths.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* gnu/packages/llvm.scm (make-clang-toolchain)[native-search-paths]: Add
'GUIX_LOCPATH'.
---
 gnu/packages/llvm.scm | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
index 15078a1168..4b42c4921a 100644
--- a/gnu/packages/llvm.scm
+++ b/gnu/packages/llvm.scm
@@ -484,7 +484,11 @@ code analysis tools.")
                                                    "libc-static")))
                      #t))))
 
-    (native-search-paths (package-native-search-paths clang))
+    (native-search-paths
+     (append (package-native-search-paths clang)
+             (list (search-path-specification     ;copied from glibc
+                    (variable "GUIX_LOCPATH")
+                    (files '("lib/locale"))))))
     (search-paths (package-search-paths clang))
 
     (license (package-license clang))
-- 
2.28.0

[bug#43679] [PATCH 3/5] gnu: clang-toolchain: Create 'cc' and 'c++' symlinks.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* gnu/packages/llvm.scm (make-clang-toolchain)[arguments]: Create 'cc'
and 'c++' symlinks.
---
 gnu/packages/llvm.scm | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
index 4b42c4921a..361b39710b 100644
--- a/gnu/packages/llvm.scm
+++ b/gnu/packages/llvm.scm
@@ -476,6 +476,12 @@ code analysis tools.")
                        (((names . directories) ...)
                         (union-build out directories)))
 
+                     ;; Create 'cc' and 'c++' so that one can use it as a
+                     ;; drop-in replacement for the default tool chain and
+                     ;; have configure scripts find the compiler.
+                     (symlink "clang" (string-append out "/bin/cc"))
+                     (symlink "clang++" (string-append out "/bin/c++"))
+
                      (union-build (assoc-ref %outputs "debug")
                                   (list (assoc-ref %build-inputs
                                                    "libc-debug")))
-- 
2.28.0

[bug#43679] [PATCH 3/5] gnu: clang-toolchain: Create 'cc' and 'c++' symlinks.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1530 bytes --]

On Mon, Sep 28, 2020 at 09:56:46PM +0200, Ludovic Courtès wrote:
> From: Ludovic Courtès <ludovic.courtes@inria.fr>
> 
> * gnu/packages/llvm.scm (make-clang-toolchain)[arguments]: Create 'cc'
> and 'c++' symlinks.
> ---
>  gnu/packages/llvm.scm | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
> index 4b42c4921a..361b39710b 100644
> --- a/gnu/packages/llvm.scm
> +++ b/gnu/packages/llvm.scm
> @@ -476,6 +476,12 @@ code analysis tools.")
>                         (((names . directories) ...)
>                          (union-build out directories)))
>  
> +                     ;; Create 'cc' and 'c++' so that one can use it as a
> +                     ;; drop-in replacement for the default tool chain and
> +                     ;; have configure scripts find the compiler.
> +                     (symlink "clang" (string-append out "/bin/cc"))
> +                     (symlink "clang++" (string-append out "/bin/c++"))
> +
>                       (union-build (assoc-ref %outputs "debug")
>                                    (list (assoc-ref %build-inputs
>                                                     "libc-debug")))
> -- 
> 2.28.0

Isn't this something we've turned down patches for in the past with gcc?

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#43679] [PATCH 3/5] gnu: clang-toolchain: Create 'cc' and 'c++' symlinks.

[Ludovic Courtès]

Hi Efraim,

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Mon, Sep 28, 2020 at 09:56:46PM +0200, Ludovic Courtès wrote:
>> From: Ludovic Courtès <ludovic.courtes@inria.fr>
>> 
>> * gnu/packages/llvm.scm (make-clang-toolchain)[arguments]: Create 'cc'
>> and 'c++' symlinks.
>> ---
>>  gnu/packages/llvm.scm | 6 ++++++
>>  1 file changed, 6 insertions(+)
>> 
>> diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
>> index 4b42c4921a..361b39710b 100644
>> --- a/gnu/packages/llvm.scm
>> +++ b/gnu/packages/llvm.scm
>> @@ -476,6 +476,12 @@ code analysis tools.")
>>                         (((names . directories) ...)
>>                          (union-build out directories)))
>>  
>> +                     ;; Create 'cc' and 'c++' so that one can use it as a
>> +                     ;; drop-in replacement for the default tool chain and
>> +                     ;; have configure scripts find the compiler.
>> +                     (symlink "clang" (string-append out "/bin/cc"))
>> +                     (symlink "clang++" (string-append out "/bin/c++"))
>> +
>>                       (union-build (assoc-ref %outputs "debug")
>>                                    (list (assoc-ref %build-inputs
>>                                                     "libc-debug")))
>> -- 
>> 2.28.0
>
> Isn't this something we've turned down patches for in the past with gcc?

That is true, and, ahem, I even know a person who was against it.

But!  I think it’s a bit different here: (1) we’re only changing
‘clang-toolchain’, not ‘clang’, (2) while most build systems look for
‘gcc’ in addition to ‘cc’, few of them look for ‘clang’.

‘--with-toolchain’ is useful even if ‘clang-toolchain’ cannot be dropped
in (it allows you to try out different GCC versions, for instance), but
I think it’s more useful if one can use it to build their code with a
different free tool chain.  One use case that comes to mind is
portability testing for C/C++ developers.

WDYT?

Thanks,
Ludo’.

[bug#43679] [PATCH 3/5] gnu: clang-toolchain: Create 'cc' and 'c++' symlinks.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2680 bytes --]

On Tue, Sep 29, 2020 at 09:53:18AM +0200, Ludovic Courtès wrote:
> Hi Efraim,
> 
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > On Mon, Sep 28, 2020 at 09:56:46PM +0200, Ludovic Courtès wrote:
> >> From: Ludovic Courtès <ludovic.courtes@inria.fr>
> >> 
> >> * gnu/packages/llvm.scm (make-clang-toolchain)[arguments]: Create 'cc'
> >> and 'c++' symlinks.
> >> ---
> >>  gnu/packages/llvm.scm | 6 ++++++
> >>  1 file changed, 6 insertions(+)
> >> 
> >> diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
> >> index 4b42c4921a..361b39710b 100644
> >> --- a/gnu/packages/llvm.scm
> >> +++ b/gnu/packages/llvm.scm
> >> @@ -476,6 +476,12 @@ code analysis tools.")
> >>                         (((names . directories) ...)
> >>                          (union-build out directories)))
> >>  
> >> +                     ;; Create 'cc' and 'c++' so that one can use it as a
> >> +                     ;; drop-in replacement for the default tool chain and
> >> +                     ;; have configure scripts find the compiler.
> >> +                     (symlink "clang" (string-append out "/bin/cc"))
> >> +                     (symlink "clang++" (string-append out "/bin/c++"))
> >> +
> >>                       (union-build (assoc-ref %outputs "debug")
> >>                                    (list (assoc-ref %build-inputs
> >>                                                     "libc-debug")))
> >> -- 
> >> 2.28.0
> >
> > Isn't this something we've turned down patches for in the past with gcc?
> 
> That is true, and, ahem, I even know a person who was against it.
> 
> But!  I think it’s a bit different here: (1) we’re only changing
> ‘clang-toolchain’, not ‘clang’, (2) while most build systems look for
> ‘gcc’ in addition to ‘cc’, few of them look for ‘clang’.
> 
> ‘--with-toolchain’ is useful even if ‘clang-toolchain’ cannot be dropped
> in (it allows you to try out different GCC versions, for instance), but
> I think it’s more useful if one can use it to build their code with a
> different free tool chain.  One use case that comes to mind is
> portability testing for C/C++ developers.
> 
> WDYT?

Do we want to have gcc-toolchain provide a cc binary then too? It seems
like it would also help people who have a project they want to build by
hand which has the compiler set as cc and they install the
gcc-toolchain.

> 
> Thanks,
> Ludo’.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#43679] [PATCH 3/5] gnu: clang-toolchain: Create 'cc' and 'c++' symlinks.

[Ludovic Courtès]

Hi,

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Tue, Sep 29, 2020 at 09:53:18AM +0200, Ludovic Courtès wrote:

[...]

>> >> +                     ;; Create 'cc' and 'c++' so that one can use it as a
>> >> +                     ;; drop-in replacement for the default tool chain and
>> >> +                     ;; have configure scripts find the compiler.
>> >> +                     (symlink "clang" (string-append out "/bin/cc"))
>> >> +                     (symlink "clang++" (string-append out "/bin/c++"))
>> >> +
>> >>                       (union-build (assoc-ref %outputs "debug")
>> >>                                    (list (assoc-ref %build-inputs
>> >>                                                     "libc-debug")))
>> >> -- 
>> >> 2.28.0
>> >
>> > Isn't this something we've turned down patches for in the past with gcc?
>> 
>> That is true, and, ahem, I even know a person who was against it.
>> 
>> But!  I think it’s a bit different here: (1) we’re only changing
>> ‘clang-toolchain’, not ‘clang’, (2) while most build systems look for
>> ‘gcc’ in addition to ‘cc’, few of them look for ‘clang’.
>> 
>> ‘--with-toolchain’ is useful even if ‘clang-toolchain’ cannot be dropped
>> in (it allows you to try out different GCC versions, for instance), but
>> I think it’s more useful if one can use it to build their code with a
>> different free tool chain.  One use case that comes to mind is
>> portability testing for C/C++ developers.
>> 
>> WDYT?
>
> Do we want to have gcc-toolchain provide a cc binary then too? It seems
> like it would also help people who have a project they want to build by
> hand which has the compiler set as cc and they install the
> gcc-toolchain.

I’m not enthusiastic (I think it’s good that build systems and tools
keep referring to ‘gcc’) but I guess I’ll have a more difficult time
arguing against it now…

Ludo’.

[bug#43679] [PATCH 4/5] packages: Add 'package-with-toolchain'.

[Ludovic Courtès]

* guix/build-system.scm (build-system-with-toolchain): New procedure.
* guix/packages.scm (package-with-toolchain): New procedure.
* tests/packages.scm ("package-with-toolchain"): New test.
* doc/guix.texi (package Reference): Document 'package-with-toolchain'.
(Build Systems): Mention it.
---
 doc/guix.texi         | 32 ++++++++++++++++++++++++++++++++
 guix/build-system.scm | 35 +++++++++++++++++++++++++++++++++--
 guix/packages.scm     |  9 +++++++++
 tests/packages.scm    | 20 ++++++++++++++++++++
 4 files changed, 94 insertions(+), 2 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index c600d577ac..03836bbf7b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6506,6 +6506,35 @@ cross-compiling:
 It is an error to refer to @code{this-package} outside a package definition.
 @end deffn
 
+Because packages are regular Scheme objects that capture a complete
+dependency graph and associated build procedures, it is often useful to
+write procedures that take a package and return a modified version
+thereof according to some parameters.  Below are a few examples.
+
+@cindex tool chain, choosing a package's tool chain
+@deffn {Scheme Procedure} package-with-toolchain @var{package} @var{toolchain}
+Return a variant of @var{package} that uses @var{toolchain} instead of
+the default GNU toolchain.  @var{toolchain} must be a list of inputs
+(label/package tuples) providing equivalent functionality, such as the
+@code{gcc-toolchain} package.
+
+The example below returns a variant of the @code{hello} package built
+with GCC@tie{}10.x and the rest of the GNU tool chain (Binutils and the
+GNU C Library) instead of the default tool chain:
+
+@lisp
+(let ((toolchain (specification->package "gcc-toolchain@@10")))
+  (package-with-toolchain hello `(("toolchain" ,toolchain))))
+@end lisp
+
+The build tool chain is part of the @dfn{implicit inputs} of
+packages---it's usually not listed as part of the various ``inputs''
+fields and is instead pulled in by the build system.  Consequently, this
+procedure works by changing the build system of @var{package} so that it
+pulls in @var{toolchain} instead of the defaults.  @ref{Build Systems},
+for more on build systems.
+@end deffn
+
 @node origin Reference
 @subsection @code{origin} Reference
 
@@ -6642,6 +6671,9 @@ ornamentation---in other words, a bag is a lower-level representation of
 a package, which includes all the inputs of that package, including some
 that were implicitly added by the build system.  This intermediate
 representation is then compiled to a derivation (@pxref{Derivations}).
+The @code{package-with-toolchain} is an example of a way to change the
+implicit inputs that a package's build system pulls in (@pxref{package
+Reference, @code{package-with-toolchain}}).
 
 Build systems accept an optional list of @dfn{arguments}.  In package
 definitions, these are passed @i{via} the @code{arguments} field
diff --git a/guix/build-system.scm b/guix/build-system.scm
index 4174972b98..0b33b91aa0 100644
--- a/guix/build-system.scm
+++ b/guix/build-system.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012, 2013, 2014 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012, 2013, 2014, 2020 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -18,6 +18,7 @@
 
 (define-module (guix build-system)
   #:use-module (guix records)
+  #:use-module (srfi srfi-1)
   #:use-module (ice-9 match)
   #:export (build-system
             build-system?
@@ -37,7 +38,9 @@
             bag-arguments
             bag-build
 
-            make-bag))
+            make-bag
+
+            build-system-with-toolchain))
 
 (define-record-type* <build-system> build-system make-build-system
   build-system?
@@ -98,3 +101,31 @@ intermediate representation just above derivations."
             #:outputs outputs
             #:target target
             arguments))))
+
+(define (build-system-with-toolchain bs toolchain)
+  "Return a variant of BS, a build system, that uses TOOLCHAIN instead of the
+default GNU toolchain.  TOOLCHAIN must be a list of inputs (label/package
+tuples) providing equivalent functionality, such as the 'gcc-toolchain'
+package."
+  (define lower
+    (build-system-lower bs))
+
+  (define toolchain-packages
+    ;; These are the GNU toolchain packages pulled in by GNU-BUILD-SYSTEM and
+    ;; all the build systems that inherit from it.  Keep the list in sync with
+    ;; 'standard-packages' in (guix build-system gnu).
+    '("gcc" "binutils" "libc" "libc:static" "ld-wrapper"))
+
+  (define (lower* . args)
+    (let ((lowered (apply lower args)))
+      (bag
+        (inherit lowered)
+        (build-inputs
+         (append (fold alist-delete
+                       (bag-build-inputs lowered)
+                       toolchain-packages)
+                 toolchain)))))
+
+  (build-system
+    (inherit bs)
+    (lower lower*)))
diff --git a/guix/packages.scm b/guix/packages.scm
index 4f2bb432be..4764461949 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -124,6 +124,7 @@
             package-patched-vulnerabilities
             package-with-patches
             package-with-extra-patches
+            package-with-toolchain
             package/inherit
 
             transitive-input-references
@@ -790,6 +791,14 @@ specifies modules in scope when evaluating SNIPPET."
                         (append (origin-patches (package-source original))
                                 patches)))
 
+(define (package-with-toolchain package toolchain)
+  "Return a variant of PACKAGE that uses TOOLCHAIN instead of the default GNU
+toolchain.  TOOLCHAIN must be a list of inputs (label/package tuples)
+providing equivalent functionality, such as the 'gcc-toolchain' package."
+  (let ((bs (package-build-system package)))
+    (package/inherit package
+      (build-system (build-system-with-toolchain bs toolchain)))))
+
 (define (transitive-inputs inputs)
   "Return the closure of INPUTS when considering the 'propagated-inputs'
 edges.  Omit duplicate inputs, except for those already present in INPUTS
diff --git a/tests/packages.scm b/tests/packages.scm
index af8941c2e2..0d032005f5 100644
--- a/tests/packages.scm
+++ b/tests/packages.scm
@@ -1407,6 +1407,26 @@
           (derivation-file-name
            (package-derivation %store coreutils))))))))
 
+(test-assert "package-with-toolchain"
+  (let* ((dep (dummy-package "chbouib"
+                (build-system gnu-build-system)
+                (native-inputs `(("x" ,grep)))))
+         (p0  (dummy-package "thingie"
+                (build-system gnu-build-system)
+                (inputs `(("foo" ,grep)
+                          ("bar" ,dep)))))
+         (tc  (dummy-package "my-toolchain"))
+         (p1  (package-with-toolchain p0 `(("toolchain" ,tc)))))
+    (define toolchain-packages
+      '("gcc" "binutils" "glibc" "ld-wrapper"))
+
+    (match (bag-build-inputs (package->bag p1))
+      ((("foo" foo) ("bar" bar) (_ (= package-name packages) . _) ...)
+       (and (not (any (cut member <> packages) toolchain-packages))
+            (member "my-toolchain" packages)
+            (eq? foo grep)
+            (eq? bar dep))))))
+
 (test-equal "package-patched-vulnerabilities"
   '(("CVE-2015-1234")
     ("CVE-2016-1234" "CVE-2018-4567")
-- 
2.28.0

[bug#43679] [PATCH 5/5] guix build: Add '--with-toolchain'.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* guix/scripts/build.scm (transform-package-toolchain): New procedure.
(%transformations): Add it.
(%transformation-options, show-transformation-options-help): Add
'--with-toolchain'.
* tests/scripts-build.scm ("options->transformation, with-toolchain"):
New test.
* doc/guix.texi (Package Transformation Options): Document it.
---
 doc/guix.texi           | 29 +++++++++++++++++++++++++++++
 guix/scripts/build.scm  | 40 ++++++++++++++++++++++++++++++++++++++++
 tests/scripts-build.scm | 30 ++++++++++++++++++++++++++++++
 3 files changed, 99 insertions(+)

diff --git a/doc/guix.texi b/doc/guix.texi
index 03836bbf7b..049958b55f 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -9310,6 +9310,35 @@ must be compatible.  If @var{replacement} is somehow incompatible with
 @var{package}, then the resulting package may be unusable.  Use with
 care!
 
+@cindex tool chain, changing the build tool chain of a package
+@item --with-toolchain=@var{package}=@var{toolchain}
+This option changes @var{package} so that it gets built with
+@var{toolchain} instead of the default GNU tool chain for C/C++.
+
+Consider this example:
+
+@example
+guix build octave-cli \
+  --with-toolchain=fftw=gcc-toolchain@@10 \
+  --with-toolchain=fftwf=gcc-toolchain@@10
+@end example
+
+The command above builds a variant of the @code{fftw} and @code{fftwf}
+packages using version 10 of @code{gcc-toolchain} instead of the default
+tool chain, and then builds a variant of the GNU@tie{}Octave
+command-line interface using them.  GNU@tie{}Octave itself is built
+using the default tool chain.
+
+This other example builds the Hardware Locality (@code{hwloc}) library
+with the Clang C compiler, and then builds its dependents up to
+@code{intel-mpi-benchmarks} using the default tool chain, but linking
+them against the Clang-built @code{hwloc}:
+
+@example
+guix build --with-toolchain=hwloc=clang-toolchain \
+           intel-mpi-benchmarks
+@end example
+
 @item --with-git-url=@var{package}=@var{url}
 @cindex Git, using the latest commit
 @cindex latest commit, building
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index f238e9b876..290046a808 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -393,6 +393,40 @@ a checkout of the Git repository at the given URL."
         (rewrite obj)
         obj)))
 
+(define (transform-package-toolchain replacement-specs)
+  "Return a procedure that, when passed a package, changes its toolchain or
+that of its dependencies according to REPLACEMENT-SPECS.  REPLACEMENT-SPECS is
+a list of strings like \"fftw=gcc-toolchain@10\" meaning that the package to
+the left of the equal sign must be built with the toolchain to the right of
+the equal sign."
+  (define split-on-commas
+    (cute string-tokenize <> (char-set-complement (char-set #\,))))
+
+  (define (specification->input spec)
+    (let ((package (specification->package spec)))
+      (list (package-name package) package)))
+
+  (define replacements
+    (map (lambda (spec)
+           (match (string-tokenize spec %not-equal)
+             ((spec (= split-on-commas toolchain))
+              (cons spec
+                    (lambda (old)
+                      (let ((toolchain (map specification->input toolchain)))
+                        (package-with-toolchain old toolchain)))))
+             (_
+              (leave (G_ "~a: invalid toolchain replacement specification~%")
+                     spec))))
+         replacement-specs))
+
+  (define rewrite
+    (package-input-rewriting/spec replacements))
+
+  (lambda (store obj)
+    (if (package? obj)
+        (rewrite obj)
+        obj)))
+
 (define (transform-package-tests specs)
   "Return a procedure that, when passed a package, sets #:tests? #f in its
 'arguments' field."
@@ -423,6 +457,7 @@ a checkout of the Git repository at the given URL."
     (with-branch . ,transform-package-source-branch)
     (with-commit . ,transform-package-source-commit)
     (with-git-url . ,transform-package-source-git-url)
+    (with-toolchain . ,transform-package-toolchain)
     (without-tests . ,transform-package-tests)))
 
 (define %transformation-options
@@ -444,6 +479,8 @@ a checkout of the Git repository at the given URL."
                   (parser 'with-commit))
           (option '("with-git-url") #t #f
                   (parser 'with-git-url))
+          (option '("with-toolchain") #t #f
+                  (parser 'with-toolchain))
           (option '("without-tests") #t #f
                   (parser 'without-tests)))))
 
@@ -466,6 +503,9 @@ a checkout of the Git repository at the given URL."
   (display (G_ "
       --with-git-url=PACKAGE=URL
                          build PACKAGE from the repository at URL"))
+  (display (G_ "
+      --with-toolchain=PACKAGE=TOOLCHAIN
+                         build package with TOOLCHAIN"))
   (display (G_ "
       --without-tests=PACKAGE
                          build PACKAGE without running its tests")))
diff --git a/tests/scripts-build.scm b/tests/scripts-build.scm
index 5f91360953..2dd1315722 100644
--- a/tests/scripts-build.scm
+++ b/tests/scripts-build.scm
@@ -22,6 +22,8 @@
   #:use-module (guix derivations)
   #:use-module (guix packages)
   #:use-module (guix git-download)
+  #:use-module (guix build-system)
+  #:use-module (guix build-system gnu)
   #:use-module (guix scripts build)
   #:use-module (guix ui)
   #:use-module (guix utils)
@@ -30,6 +32,8 @@
   #:use-module (gnu packages base)
   #:use-module (gnu packages busybox)
   #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-64))
 
 \f
@@ -270,6 +274,32 @@
                        ((("x" dep3))
                         (map package-source (list dep1 dep3))))))))))))
 
+(test-equal "options->transformation, with-toolchain"
+  '("gcc-toolchain")
+  (let* ((p (dummy-package "thingie"
+              (build-system gnu-build-system)
+              (inputs `(("foo" ,grep)
+                        ("bar" ,(dummy-package "chbouib"
+                                  (build-system gnu-build-system)
+                                  (native-inputs `(("x" ,grep)))))))))
+         (t (options->transformation '((with-toolchain
+                                        . "chbouib=gcc-toolchain")))))
+    (define toolchain-packages
+      '("gcc" "binutils" "glibc" "ld-wrapper"))
+
+    (with-store store
+      (let ((new (t store p)))
+        (match (bag-build-inputs (package->bag new))
+          ((("foo" _) ("bar" dep) (_ (= package-name packages) . _) ...)
+           (and (every (cut member <> packages)
+                       toolchain-packages)
+                (match (bag-build-inputs (package->bag dep))
+                  ((("x" dep0) (_ (= package-name packages) . _) ...)
+                   (and (eq? dep0 grep)           ;this one is unchanged
+                        (not (any (cut member <> packages)
+                                  toolchain-packages))
+                        (member "gcc-toolchain" packages)))))))))))
+
 (test-assert "options->transformation, without-tests"
   (let* ((dep (dummy-package "dep"))
          (p   (dummy-package "foo"
-- 
2.28.0

[bug#43679] [PATCH 0/5] Add '--with-toolchain' package transformation option

[zimoun]

Hi,

On Mon, 28 Sep 2020 at 21:53, Ludovic Courtès <ludo@gnu.org> wrote:
> From: Ludovic Courtès <ludovic.courtes@inria.fr>

> One thing I wasn’t entirely sure about: ‘--with-toolchain’ changes
> the toolchain of the specified package, not that of its dependents.
> This assumes that the toolchains all follow the same ABI.  This is
> the case for C, apparently, maybe not for C++.  Should it instead
> change to toolchain of the package’s dependents as well?
>
> Something like:
>
>   guix build guile --with-toolchain=guile@3.0.4=clang-toolchain 
>
> generates working code.

Really cool!  Playing yesterday with the new ’package-mapping’ &
co. (checking ’package-with-explicit-ocaml’), a kind of new
’–with-toolchain’ option was my conclusion. :-)


However, ’–with-toolchain’ can be misleading since it is
’gnu-build-system’ and C/C++ software specific.  I mean, the patch #4
adding ’build-system-with-toolchain’ contains:

--8<---------------cut here---------------start------------->8---
+  (define toolchain-packages
+    ;; These are the GNU toolchain packages pulled in by GNU-BUILD-SYSTEM and
+    ;; all the build systems that inherit from it.  Keep the list in sync with
+    ;; 'standard-packages' in (guix build-system gnu).
+    '("gcc" "binutils" "libc" "libc:static" "ld-wrapper"))
+
+  (define (lower* . args)
+    (let ((lowered (apply lower args)))
+      (bag
+        (inherit lowered)
+        (build-inputs
+         (append (fold alist-delete
+                       (bag-build-inputs lowered)
+                       toolchain-packages)
+                 toolchain)))))
--8<---------------cut here---------------end--------------->8---

And for example, it will not remove ’default-ocaml’ and
’default-findlib’ in the ’ocaml-build-system’.  Even if it would be easy
to specify the options “–with-input=ocaml=ocaml-variant
–with-input=findlib=findlib-variant”.  But for the
’clojure-build-system’ it is 3 packages.

Another example a bit out-of-scope is to rebuild all the Emacs stack
using the package ’emacs-next’ instead of ’emacs’.  The
’emacs-build-system’ depends on ’emacs-minimal’ but some packages (see
’emacs-magit’) rewrite that using instead ’emacs-no-x’.  It could be
nice to be able to write:

  guix build -m manifest.m --with-toolchain=emacs-next-toolchain



In summary, does it make sense, either:

  - change the ’–with-toolchain’ to ’–with-gcc-toolchain’
or
  - tweak ’build-system-with-toolchain’ to pass ’toolchain-packages’ as
  parameter somehow and be able to run:
  
     guix build coq --with-toolchain=coq=ocaml-toolchain4.07
     
?     


> Another issue is that since we use ‘package-input-rewriting/spec’,
> we can’t change the toolchain of core packages like Guile or Perl
> without rebuilding the world.  For example, if we omit “@3.0.4”
> in the example above, we rebuild a “guile” package deep down and
> everything that follows (aka. “the world”).

Yeah but that’s maybe what people want: rebuild the world with another
toolchain, probably optimized for some specific machine (HPC cluster).


> Another option I considered was to graft the package that
> ‘--with-toolchain’ targets instead of rebuilding its dependents.
> Again that’d only work if the resulting binaries are ABI-compatible,
> but maybe that’s a reasonable assumption.  It would definitely save
> build time.  Should it be grafted, or should there be a separate
> option to do that?  Thoughts?

From my perspective, it should be another option.  For example, I
imagine people want to rebuild all the stack with Name-It© compiler.  Or
the Name-It© compiler could be not-ABI compatible.


All the best,
simon

[bug#43679] [PATCH 0/5] Add '--with-toolchain' package transformation option

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Mon, 28 Sep 2020 at 21:53, Ludovic Courtès <ludo@gnu.org> wrote:
>> From: Ludovic Courtès <ludovic.courtes@inria.fr>
>
>> One thing I wasn’t entirely sure about: ‘--with-toolchain’ changes
>> the toolchain of the specified package, not that of its dependents.
>> This assumes that the toolchains all follow the same ABI.  This is
>> the case for C, apparently, maybe not for C++.  Should it instead
>> change to toolchain of the package’s dependents as well?
>>
>> Something like:
>>
>>   guix build guile --with-toolchain=guile@3.0.4=clang-toolchain 
>>
>> generates working code.

[...]

> However, ’–with-toolchain’ can be misleading since it is
> ’gnu-build-system’ and C/C++ software specific.  I mean, the patch #4
> adding ’build-system-with-toolchain’ contains:
>
> +  (define toolchain-packages
> +    ;; These are the GNU toolchain packages pulled in by GNU-BUILD-SYSTEM and
> +    ;; all the build systems that inherit from it.  Keep the list in sync with
> +    ;; 'standard-packages' in (guix build-system gnu).
> +    '("gcc" "binutils" "libc" "libc:static" "ld-wrapper"))
> +
> +  (define (lower* . args)
> +    (let ((lowered (apply lower args)))
> +      (bag
> +        (inherit lowered)
> +        (build-inputs
> +         (append (fold alist-delete
> +                       (bag-build-inputs lowered)
> +                       toolchain-packages)
> +                 toolchain)))))

Yeah this option is meant for C/C++ as I wrote above and (I think) in
the documentation.

> Another example a bit out-of-scope is to rebuild all the Emacs stack
> using the package ’emacs-next’ instead of ’emacs’.  The
> ’emacs-build-system’ depends on ’emacs-minimal’ but some packages (see
> ’emacs-magit’) rewrite that using instead ’emacs-no-x’.  It could be
> nice to be able to write:
>
>   guix build -m manifest.m --with-toolchain=emacs-next-toolchain

Here you’d use ‘--with-input’, though package transformation options
have no effect when using a manifest.

> In summary, does it make sense, either:
>
>   - change the ’–with-toolchain’ to ’–with-gcc-toolchain’

‘--with-gcc-toolchain=clang-toolchain’ would look strange.  :-)

>   - tweak ’build-system-with-toolchain’ to pass ’toolchain-packages’ as
>   parameter somehow and be able to run:
>   
>      guix build coq --with-toolchain=coq=ocaml-toolchain4.07

Can’t you use ‘--with-input=ocamlX.Y=ocamlA.B’ in this case?  If not, we
could devise a separate option rather than overload this one.

>> Another issue is that since we use ‘package-input-rewriting/spec’,
>> we can’t change the toolchain of core packages like Guile or Perl
>> without rebuilding the world.  For example, if we omit “@3.0.4”
>> in the example above, we rebuild a “guile” package deep down and
>> everything that follows (aka. “the world”).
>
> Yeah but that’s maybe what people want: rebuild the world with another
> toolchain, probably optimized for some specific machine (HPC cluster).

Yes, though it doesn’t necessarily make sense.  :-)

But yeah, perhaps rebuilding everything above the given package would be
more in line with what people expect.

>> Another option I considered was to graft the package that
>> ‘--with-toolchain’ targets instead of rebuilding its dependents.
>> Again that’d only work if the resulting binaries are ABI-compatible,
>> but maybe that’s a reasonable assumption.  It would definitely save
>> build time.  Should it be grafted, or should there be a separate
>> option to do that?  Thoughts?
>
> From my perspective, it should be another option.  For example, I
> imagine people want to rebuild all the stack with Name-It© compiler.  Or
> the Name-It© compiler could be not-ABI compatible.

I’m not interested in proprietary compilers if that’s what you have in
mind.  Besides, the SysV ABI is defined for C, so normally all C
compilers produce ABI-compatible code.  There are exceptions such as
OpenMP (Clang is moving to their own libomp, I think, whereas GCC has
libgomp.)

Thanks for your feedback!

Ludo’.

[bug#43679] [PATCH 0/5] Add '--with-toolchain' package transformation option

[zimoun]

Hi,

On Wed, 30 Sep 2020 at 10:46, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> > However, ’–with-toolchain’ can be misleading since it is
> > ’gnu-build-system’ and C/C++ software specific.  I mean, the patch #4
> > adding ’build-system-with-toolchain’ contains:

[...]

> Yeah this option is meant for C/C++ as I wrote above and (I think) in
> the documentation.

Yes in the manual, not in the command line helper.

Without bikeshedding, I find '--with-toolchain' a bad name since it is
only 'gnu-build-system' related.  And from my point of view, it is
also a bad name for the procedures 'build-system-with-toolchain' and
'package-with-toolchain' -- but it does not matter since they are not
written in stone, contrary to command line options harder to change.


> >   - change the ’–with-toolchain’ to ’–with-gcc-toolchain’
>
> ‘--with-gcc-toolchain=clang-toolchain’ would look strange.  :-)

Why not? :-)
My point is: '--with-toolchain' is not specific enough.  Maybe
'--with-gnu-toolchain'?


> >   - tweak ’build-system-with-toolchain’ to pass ’toolchain-packages’ as
> >   parameter somehow and be able to run:
> >
> >      guix build coq --with-toolchain=coq=ocaml-toolchain4.07
>
> Can’t you use ‘--with-input=ocamlX.Y=ocamlA.B’ in this case?  If not, we
> could devise a separate option rather than overload this one.

No, in this case one should use:

   guix build coq \
          --with-input=ocaml=ocaml@4.07 \
          --with-input=ocaml-findlib=ocaml4.07-findlib

to recompile the package 'coq' with the 4.07 'ocaml-build-system'.
For the 'clojure-build-system', there are 3 inputs to specify.  (I
have not checked all the build systems :-)).  And note it works only
if the tools used by the build system are not hidden.

For consistency, it appears to me easier to have one "toolchain" per
build system, say ocaml-toolchain, gcc-toolchain, haskell-toolchain,
and then provides this toolchain to the option '--with-toolchain'.
However, it is complicated to remove the 'build-inputs' since they are
not hard coded -- as it is the case in 'build-system-with-toolchain'.
Or another option is to have one command line option per build system:
--with-gnu-toolchain, --with-ocaml-toolchain, --with-cargo-toolchain,
etc..


> > Yeah but that’s maybe what people want: rebuild the world with another
> > toolchain, probably optimized for some specific machine (HPC cluster).
>
> Yes, though it doesn’t necessarily make sense.  :-)

Sadly!

> But yeah, perhaps rebuilding everything above the given package would be
> more in line with what people expect.

Yeah...maybe providing "what people expect" could reduce the gap in
the HPC community.


> >> Another option I considered was to graft the package that
> >> ‘--with-toolchain’ targets instead of rebuilding its dependents.
> >> Again that’d only work if the resulting binaries are ABI-compatible,
> >> but maybe that’s a reasonable assumption.  It would definitely save
> >> build time.  Should it be grafted, or should there be a separate
> >> option to do that?  Thoughts?
> >
> > From my perspective, it should be another option.  For example, I
> > imagine people want to rebuild all the stack with Name-It© compiler.  Or
> > the Name-It© compiler could be not-ABI compatible.
>
> I’m not interested in proprietary compilers if that’s what you have in
> mind.  Besides, the SysV ABI is defined for C, so normally all C
> compilers produce ABI-compatible code.  There are exceptions such as
> OpenMP (Clang is moving to their own libomp, I think, whereas GCC has
> libgomp.)

It was what I have in mind. :-)
But do the exceptions you point not imply another option?

All the best,
simon

[bug#43679] [PATCH 0/5] Add '--with-toolchain' package transformation option

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

> On Wed, 30 Sep 2020 at 10:46, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>> > However, ’–with-toolchain’ can be misleading since it is
>> > ’gnu-build-system’ and C/C++ software specific.  I mean, the patch #4
>> > adding ’build-system-with-toolchain’ contains:
>
> [...]
>
>> Yeah this option is meant for C/C++ as I wrote above and (I think) in
>> the documentation.
>
> Yes in the manual, not in the command line helper.
>
> Without bikeshedding, I find '--with-toolchain' a bad name since it is
> only 'gnu-build-system' related.  And from my point of view, it is
> also a bad name for the procedures 'build-system-with-toolchain' and
> 'package-with-toolchain' -- but it does not matter since they are not
> written in stone, contrary to command line options harder to change.

I agree that C/C++ don’t have a monopoly on tool chains, no argument
here.  The term “tool chain” is widely used for C/C++ though, much less
for other languages (often the “tool chain” is a single package for
other languages).

We could change the name to ‘--with-c-toolchain’ maybe?  Then someone
might come and suggest that this doesn’t account for C++, Objective-C,
and FORTRAN.

>> Can’t you use ‘--with-input=ocamlX.Y=ocamlA.B’ in this case?  If not, we
>> could devise a separate option rather than overload this one.
>
> No, in this case one should use:
>
>    guix build coq \
>           --with-input=ocaml=ocaml@4.07 \
>           --with-input=ocaml-findlib=ocaml4.07-findlib

Hmm I think the second one is unnecessary since
‘--with-input=ocaml=ocaml@4.07’ effectively gives an ‘ocaml-findlib’
built against OCaml 4.07.

Anyway, we’re drifting off-topic; let’s address OCaml separately if
something needs to be addressed.

> For consistency, it appears to me easier to have one "toolchain" per
> build system, say ocaml-toolchain, gcc-toolchain, haskell-toolchain,
> and then provides this toolchain to the option '--with-toolchain'.
> However, it is complicated to remove the 'build-inputs' since they are
> not hard coded -- as it is the case in 'build-system-with-toolchain'.
> Or another option is to have one command line option per build system:
> --with-gnu-toolchain, --with-ocaml-toolchain, --with-cargo-toolchain,
> etc..

If there’s a need for that, yes.  We’ll see!

>> I’m not interested in proprietary compilers if that’s what you have in
>> mind.  Besides, the SysV ABI is defined for C, so normally all C
>> compilers produce ABI-compatible code.  There are exceptions such as
>> OpenMP (Clang is moving to their own libomp, I think, whereas GCC has
>> libgomp.)
>
> It was what I have in mind. :-)
> But do the exceptions you point not imply another option?

We can’t completely prevent people from shooting themselves in the foot
with transformations, but yeah, maybe we should rebuild everything
higher in the stack with the same toolchain.

Thanks,
Ludo’.

[bug#43679] [PATCH v2 0/5] Add '--with-c-toolchain'

[Ludovic Courtès]

Hi!

This is v2 of this patch, with these changes:

  1. ‘with-toolchain’ is replaced by ‘with-c-toolchain’ everywhere,
     with the understanding that it’s about the C/C++ toolchain
     in practice.  In the end I’m sympathetic with the argument
     that C/C++ don’t have a monopoly on toolchains.  ;-)

  2. ‘--with-c-toolchain=PACKAGE=TOOLCHAIN’ rebuilds not just
     PACKAGE with TOOLCHAIN, but also everything above PACKAGE
     with TOOLCHAIN (in v1, only PACKAGE was rebuilt with TOOLCHAIN
     but everything above it had to be rebuilt anyway.)

     The main motivation here is to reduce the changes that we’re
     introducing ABI incompatibilities that users would have to work
     around by passing on ‘--with-c-toolchain’ for each package in
     the chain.  I think it also more closely matches user
     expectations: when you see things are being rebuilt, you’re
     likely to think that’s because they’re rebuilt with the new
     toolchain, not the default one.

Feedback welcome!

Ludo’.

Ludovic Courtès (5):
  gnu: gcc-toolchain: Add 'GUIX_LOCPATH' to the search paths.
  gnu: clang-toolchain: Add 'GUIX_LOCPATH' to the search paths.
  gnu: clang-toolchain: Create 'cc' and 'c++' symlinks.
  packages: Add 'package-with-c-toolchain'.
  guix build: Add '--with-c-toolchain'.

 doc/guix.texi                 | 70 +++++++++++++++++++++++++++++
 gnu/packages/commencement.scm |  8 +++-
 gnu/packages/llvm.scm         | 12 ++++-
 guix/build-system.scm         | 35 ++++++++++++++-
 guix/packages.scm             |  9 ++++
 guix/scripts/build.scm        | 84 +++++++++++++++++++++++++++++++++++
 tests/packages.scm            | 20 +++++++++
 tests/scripts-build.scm       | 82 ++++++++++++++++++++++++++++++++++
 8 files changed, 315 insertions(+), 5 deletions(-)

-- 
2.28.0

[bug#43679] [PATCH v2 1/5] gnu: gcc-toolchain: Add 'GUIX_LOCPATH' to the search paths.

[Ludovic Courtès]

This allows 'gcc-toolchain' to be used as a drop-in replacement for the
default tool chain through '--with-toolchain=gcc-toolchain'.

* gnu/packages/commencement.scm (make-gcc-toolchain)[native-search-paths]
[search-paths]: Append LIBC's search paths.
---
 gnu/packages/commencement.scm | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 4041d5bb89..0938bc3d46 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -3856,8 +3856,12 @@ COREUTILS-FINAL vs. COREUTILS, etc."
                                                      "libc-static")))
                        #t))))
 
-      (native-search-paths (package-native-search-paths gcc))
-      (search-paths (package-search-paths gcc))
+      (native-search-paths
+       (append (package-native-search-paths gcc)
+               (package-native-search-paths libc))) ;GUIX_LOCPATH
+      (search-paths
+       (append (package-search-paths gcc)
+               (package-search-paths libc)))
 
       (license (package-license gcc))
       (synopsis "Complete GCC tool chain for C/C++ development")
-- 
2.28.0

[bug#43679] [PATCH v2 2/5] gnu: clang-toolchain: Add 'GUIX_LOCPATH' to the search paths.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* gnu/packages/llvm.scm (make-clang-toolchain)[native-search-paths]: Add
'GUIX_LOCPATH'.
---
 gnu/packages/llvm.scm | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
index 15078a1168..4b42c4921a 100644
--- a/gnu/packages/llvm.scm
+++ b/gnu/packages/llvm.scm
@@ -484,7 +484,11 @@ code analysis tools.")
                                                    "libc-static")))
                      #t))))
 
-    (native-search-paths (package-native-search-paths clang))
+    (native-search-paths
+     (append (package-native-search-paths clang)
+             (list (search-path-specification     ;copied from glibc
+                    (variable "GUIX_LOCPATH")
+                    (files '("lib/locale"))))))
     (search-paths (package-search-paths clang))
 
     (license (package-license clang))
-- 
2.28.0

[bug#43679] [PATCH v2 3/5] gnu: clang-toolchain: Create 'cc' and 'c++' symlinks.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* gnu/packages/llvm.scm (make-clang-toolchain)[arguments]: Create 'cc'
and 'c++' symlinks.
---
 gnu/packages/llvm.scm | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
index 4b42c4921a..361b39710b 100644
--- a/gnu/packages/llvm.scm
+++ b/gnu/packages/llvm.scm
@@ -476,6 +476,12 @@ code analysis tools.")
                        (((names . directories) ...)
                         (union-build out directories)))
 
+                     ;; Create 'cc' and 'c++' so that one can use it as a
+                     ;; drop-in replacement for the default tool chain and
+                     ;; have configure scripts find the compiler.
+                     (symlink "clang" (string-append out "/bin/cc"))
+                     (symlink "clang++" (string-append out "/bin/c++"))
+
                      (union-build (assoc-ref %outputs "debug")
                                   (list (assoc-ref %build-inputs
                                                    "libc-debug")))
-- 
2.28.0

[bug#43679] [PATCH v2 4/5] packages: Add 'package-with-c-toolchain'.

[Ludovic Courtès]

* guix/build-system.scm (build-system-with-c-toolchain): New procedure.
* guix/packages.scm (package-with-c-toolchain): New procedure.
* tests/packages.scm ("package-with-c-toolchain"): New test.
* doc/guix.texi (package Reference): Document 'package-with-c-toolchain'.
(Build Systems): Mention it.
---
 doc/guix.texi         | 32 ++++++++++++++++++++++++++++++++
 guix/build-system.scm | 35 +++++++++++++++++++++++++++++++++--
 guix/packages.scm     |  9 +++++++++
 tests/packages.scm    | 20 ++++++++++++++++++++
 4 files changed, 94 insertions(+), 2 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index a6260a12aa..38a8e5d4a6 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6530,6 +6530,35 @@ cross-compiling:
 It is an error to refer to @code{this-package} outside a package definition.
 @end deffn
 
+Because packages are regular Scheme objects that capture a complete
+dependency graph and associated build procedures, it is often useful to
+write procedures that take a package and return a modified version
+thereof according to some parameters.  Below are a few examples.
+
+@cindex tool chain, choosing a package's tool chain
+@deffn {Scheme Procedure} package-with-c-toolchain @var{package} @var{toolchain}
+Return a variant of @var{package} that uses @var{toolchain} instead of
+the default GNU C/C++ toolchain.  @var{toolchain} must be a list of
+inputs (label/package tuples) providing equivalent functionality, such
+as the @code{gcc-toolchain} package.
+
+The example below returns a variant of the @code{hello} package built
+with GCC@tie{}10.x and the rest of the GNU tool chain (Binutils and the
+GNU C Library) instead of the default tool chain:
+
+@lisp
+(let ((toolchain (specification->package "gcc-toolchain@@10")))
+  (package-with-c-toolchain hello `(("toolchain" ,toolchain))))
+@end lisp
+
+The build tool chain is part of the @dfn{implicit inputs} of
+packages---it's usually not listed as part of the various ``inputs''
+fields and is instead pulled in by the build system.  Consequently, this
+procedure works by changing the build system of @var{package} so that it
+pulls in @var{toolchain} instead of the defaults.  @ref{Build Systems},
+for more on build systems.
+@end deffn
+
 @node origin Reference
 @subsection @code{origin} Reference
 
@@ -6666,6 +6695,9 @@ ornamentation---in other words, a bag is a lower-level representation of
 a package, which includes all the inputs of that package, including some
 that were implicitly added by the build system.  This intermediate
 representation is then compiled to a derivation (@pxref{Derivations}).
+The @code{package-with-c-toolchain} is an example of a way to change the
+implicit inputs that a package's build system pulls in (@pxref{package
+Reference, @code{package-with-c-toolchain}}).
 
 Build systems accept an optional list of @dfn{arguments}.  In package
 definitions, these are passed @i{via} the @code{arguments} field
diff --git a/guix/build-system.scm b/guix/build-system.scm
index 4174972b98..76d670995c 100644
--- a/guix/build-system.scm
+++ b/guix/build-system.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012, 2013, 2014 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012, 2013, 2014, 2020 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -18,6 +18,7 @@
 
 (define-module (guix build-system)
   #:use-module (guix records)
+  #:use-module (srfi srfi-1)
   #:use-module (ice-9 match)
   #:export (build-system
             build-system?
@@ -37,7 +38,9 @@
             bag-arguments
             bag-build
 
-            make-bag))
+            make-bag
+
+            build-system-with-c-toolchain))
 
 (define-record-type* <build-system> build-system make-build-system
   build-system?
@@ -98,3 +101,31 @@ intermediate representation just above derivations."
             #:outputs outputs
             #:target target
             arguments))))
+
+(define (build-system-with-c-toolchain bs toolchain)
+  "Return a variant of BS, a build system, that uses TOOLCHAIN instead of the
+default GNU C/C++ toolchain.  TOOLCHAIN must be a list of
+inputs (label/package tuples) providing equivalent functionality, such as the
+'gcc-toolchain' package."
+  (define lower
+    (build-system-lower bs))
+
+  (define toolchain-packages
+    ;; These are the GNU toolchain packages pulled in by GNU-BUILD-SYSTEM and
+    ;; all the build systems that inherit from it.  Keep the list in sync with
+    ;; 'standard-packages' in (guix build-system gnu).
+    '("gcc" "binutils" "libc" "libc:static" "ld-wrapper"))
+
+  (define (lower* . args)
+    (let ((lowered (apply lower args)))
+      (bag
+        (inherit lowered)
+        (build-inputs
+         (append (fold alist-delete
+                       (bag-build-inputs lowered)
+                       toolchain-packages)
+                 toolchain)))))
+
+  (build-system
+    (inherit bs)
+    (lower lower*)))
diff --git a/guix/packages.scm b/guix/packages.scm
index 4f2bb432be..24d6417065 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -124,6 +124,7 @@
             package-patched-vulnerabilities
             package-with-patches
             package-with-extra-patches
+            package-with-c-toolchain
             package/inherit
 
             transitive-input-references
@@ -790,6 +791,14 @@ specifies modules in scope when evaluating SNIPPET."
                         (append (origin-patches (package-source original))
                                 patches)))
 
+(define (package-with-c-toolchain package toolchain)
+  "Return a variant of PACKAGE that uses TOOLCHAIN instead of the default GNU
+C/C++ toolchain.  TOOLCHAIN must be a list of inputs (label/package tuples)
+providing equivalent functionality, such as the 'gcc-toolchain' package."
+  (let ((bs (package-build-system package)))
+    (package/inherit package
+      (build-system (build-system-with-c-toolchain bs toolchain)))))
+
 (define (transitive-inputs inputs)
   "Return the closure of INPUTS when considering the 'propagated-inputs'
 edges.  Omit duplicate inputs, except for those already present in INPUTS
diff --git a/tests/packages.scm b/tests/packages.scm
index 5d5abcbd76..2d13d91344 100644
--- a/tests/packages.scm
+++ b/tests/packages.scm
@@ -1430,6 +1430,26 @@
           (derivation-file-name
            (package-derivation %store coreutils))))))))
 
+(test-assert "package-with-c-toolchain"
+  (let* ((dep (dummy-package "chbouib"
+                (build-system gnu-build-system)
+                (native-inputs `(("x" ,grep)))))
+         (p0  (dummy-package "thingie"
+                (build-system gnu-build-system)
+                (inputs `(("foo" ,grep)
+                          ("bar" ,dep)))))
+         (tc  (dummy-package "my-toolchain"))
+         (p1  (package-with-c-toolchain p0 `(("toolchain" ,tc)))))
+    (define toolchain-packages
+      '("gcc" "binutils" "glibc" "ld-wrapper"))
+
+    (match (bag-build-inputs (package->bag p1))
+      ((("foo" foo) ("bar" bar) (_ (= package-name packages) . _) ...)
+       (and (not (any (cut member <> packages) toolchain-packages))
+            (member "my-toolchain" packages)
+            (eq? foo grep)
+            (eq? bar dep))))))
+
 (test-equal "package-patched-vulnerabilities"
   '(("CVE-2015-1234")
     ("CVE-2016-1234" "CVE-2018-4567")
-- 
2.28.0

[bug#43679] [PATCH v2 5/5] guix build: Add '--with-c-toolchain'.

[Ludovic Courtès]

From: Ludovic Courtès <ludovic.courtes@inria.fr>

* guix/scripts/build.scm (package-dependents/spec)
(package-toolchain-rewriting, transform-package-toolchain): New procedures.
(%transformations): Add it.
(%transformation-options, show-transformation-options-help): Add
'--with-c-toolchain'.
* tests/scripts-build.scm (depends-on-toolchain?): New procedure.
("options->transformation, with-c-toolchain")
("options->transformation, with-c-toolchain twice")
New test.
("options->transformation, with-c-toolchain, no effect"): New tests.
* doc/guix.texi (Package Transformation Options): Document it.
---
 doc/guix.texi           | 38 +++++++++++++++++++
 guix/scripts/build.scm  | 84 +++++++++++++++++++++++++++++++++++++++++
 tests/scripts-build.scm | 82 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 204 insertions(+)

diff --git a/doc/guix.texi b/doc/guix.texi
index 38a8e5d4a6..c955041c61 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -9338,6 +9338,44 @@ must be compatible.  If @var{replacement} is somehow incompatible with
 @var{package}, then the resulting package may be unusable.  Use with
 care!
 
+@cindex tool chain, changing the build tool chain of a package
+@item --with-c-toolchain=@var{package}=@var{toolchain}
+This option changes the compilation of @var{package} and everything that
+depends on it so that they get built with @var{toolchain} instead of the
+default GNU tool chain for C/C++.
+
+Consider this example:
+
+@example
+guix build octave-cli \
+  --with-c-toolchain=fftw=gcc-toolchain@@10 \
+  --with-c-toolchain=fftwf=gcc-toolchain@@10
+@end example
+
+The command above builds a variant of the @code{fftw} and @code{fftwf}
+packages using version 10 of @code{gcc-toolchain} instead of the default
+tool chain, and then builds a variant of the GNU@tie{}Octave
+command-line interface using them.  GNU@tie{}Octave itself is also built
+with @code{gcc-toolchain@@10}.
+
+This other example builds the Hardware Locality (@code{hwloc}) library
+and its dependents up to @code{intel-mpi-benchmarks} with the Clang C
+compiler:
+
+@example
+guix build --with-c-toolchain=hwloc=clang-toolchain \
+           intel-mpi-benchmarks
+@end example
+
+@quotation Note
+There can be application binary interface (ABI) incompatibilities among
+tool chains.  This is particularly true of the C++ standard library and
+run-time support libraries such as that of OpenMP.  By rebuilding all
+dependents with the same tool chain, @option{--with-c-toolchain} minimizes
+the risks of incompatibility but cannot entirely eliminate them.  Choose
+@var{package} wisely.
+@end quotation
+
 @item --with-git-url=@var{package}=@var{url}
 @cindex Git, using the latest commit
 @cindex latest commit, building
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 72a5d46347..e59e0ee67f 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -26,6 +26,7 @@
   #:use-module (guix store)
   #:use-module (guix derivations)
   #:use-module (guix packages)
+  #:use-module (guix memoization)
   #:use-module (guix grafts)
 
   #:use-module (guix utils)
@@ -396,6 +397,83 @@ a checkout of the Git repository at the given URL."
         (rewrite obj)
         obj)))
 
+(define (package-dependents/spec top bottom)
+  "Return the list of dependents of BOTTOM, a spec string, that are also
+dependencies of TOP, a package."
+  (define-values (name version)
+    (package-name->name+version bottom))
+
+  (define dependent?
+    (mlambda (p)
+      (and (package? p)
+           (or (and (string=? name (package-name p))
+                    (or (not version)
+                        (version-prefix? version (package-version p))))
+               (match (bag-direct-inputs (package->bag p))
+                 (((labels dependencies . _) ...)
+                  (any dependent? dependencies)))))))
+
+  (filter dependent? (package-closure (list top))))
+
+(define (package-toolchain-rewriting p bottom toolchain)
+  "Return a procedure that, when passed a package that's either BOTTOM or one
+of its dependents up to P so, changes it so it is built with TOOLCHAIN.
+TOOLCHAIN must be an input list."
+  (define rewriting-property
+    (gensym " package-toolchain-rewriting"))
+
+  (match (package-dependents/spec p bottom)
+    (()                                           ;P does not depend on BOTTOM
+     identity)
+    (set
+     ;; SET is the list of packages "between" P and BOTTOM (included) whose
+     ;; toolchain needs to be changed.
+     (package-mapping (lambda (p)
+                        (if (or (assq rewriting-property
+                                      (package-properties p))
+                                (not (memq p set)))
+                            p
+                            (let ((p (package-with-c-toolchain p toolchain)))
+                              (package/inherit p
+                                (properties `((,rewriting-property . #t)
+                                              ,@(package-properties p)))))))
+                      (lambda (p)
+                        (or (assq rewriting-property (package-properties p))
+                            (not (memq p set))))
+                      #:deep? #t))))
+
+(define (transform-package-toolchain replacement-specs)
+  "Return a procedure that, when passed a package, changes its toolchain or
+that of its dependencies according to REPLACEMENT-SPECS.  REPLACEMENT-SPECS is
+a list of strings like \"fftw=gcc-toolchain@10\" meaning that the package to
+the left of the equal sign must be built with the toolchain to the right of
+the equal sign."
+  (define split-on-commas
+    (cute string-tokenize <> (char-set-complement (char-set #\,))))
+
+  (define (specification->input spec)
+    (let ((package (specification->package spec)))
+      (list (package-name package) package)))
+
+  (define replacements
+    (map (lambda (spec)
+           (match (string-tokenize spec %not-equal)
+             ((spec (= split-on-commas toolchain))
+              (cons spec (map specification->input toolchain)))
+             (_
+              (leave (G_ "~a: invalid toolchain replacement specification~%")
+                     spec))))
+         replacement-specs))
+
+  (lambda (store obj)
+    (if (package? obj)
+        (or (any (match-lambda
+                   ((bottom . toolchain)
+                    ((package-toolchain-rewriting obj bottom toolchain) obj)))
+                 replacements)
+            obj)
+        obj)))
+
 (define (transform-package-tests specs)
   "Return a procedure that, when passed a package, sets #:tests? #f in its
 'arguments' field."
@@ -426,6 +504,7 @@ a checkout of the Git repository at the given URL."
     (with-branch . ,transform-package-source-branch)
     (with-commit . ,transform-package-source-commit)
     (with-git-url . ,transform-package-source-git-url)
+    (with-c-toolchain . ,transform-package-toolchain)
     (without-tests . ,transform-package-tests)))
 
 (define (transformation-procedure key)
@@ -455,6 +534,8 @@ a checkout of the Git repository at the given URL."
                   (parser 'with-commit))
           (option '("with-git-url") #t #f
                   (parser 'with-git-url))
+          (option '("with-c-toolchain") #t #f
+                  (parser 'with-c-toolchain))
           (option '("without-tests") #t #f
                   (parser 'without-tests)))))
 
@@ -477,6 +558,9 @@ a checkout of the Git repository at the given URL."
   (display (G_ "
       --with-git-url=PACKAGE=URL
                          build PACKAGE from the repository at URL"))
+  (display (G_ "
+      --with-c-toolchain=PACKAGE=TOOLCHAIN
+                         build PACKAGE and its dependents with TOOLCHAIN"))
   (display (G_ "
       --without-tests=PACKAGE
                          build PACKAGE without running its tests")))
diff --git a/tests/scripts-build.scm b/tests/scripts-build.scm
index 5f91360953..6925374baa 100644
--- a/tests/scripts-build.scm
+++ b/tests/scripts-build.scm
@@ -22,6 +22,8 @@
   #:use-module (guix derivations)
   #:use-module (guix packages)
   #:use-module (guix git-download)
+  #:use-module (guix build-system)
+  #:use-module (guix build-system gnu)
   #:use-module (guix scripts build)
   #:use-module (guix ui)
   #:use-module (guix utils)
@@ -30,6 +32,8 @@
   #:use-module (gnu packages base)
   #:use-module (gnu packages busybox)
   #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
   #:use-module (srfi srfi-64))
 
 \f
@@ -270,6 +274,80 @@
                        ((("x" dep3))
                         (map package-source (list dep1 dep3))))))))))))
 
+(define* (depends-on-toolchain? p #:optional (toolchain "gcc-toolchain"))
+  "Return true if P depends on TOOLCHAIN instead of the default tool chain."
+  (define toolchain-packages
+    '("gcc" "binutils" "glibc" "ld-wrapper"))
+
+  (define (package-name* obj)
+    (and (package? obj) (package-name obj)))
+
+  (match (bag-build-inputs (package->bag p))
+    (((_ (= package-name* packages) . _) ...)
+     (and (not (any (cut member <> packages) toolchain-packages))
+          (member toolchain packages)))))
+
+(test-assert "options->transformation, with-c-toolchain"
+  (let* ((dep0 (dummy-package "chbouib"
+                 (build-system gnu-build-system)
+                 (native-inputs `(("y" ,grep)))))
+         (dep1 (dummy-package "stuff"
+                 (native-inputs `(("x" ,dep0)))))
+         (p    (dummy-package "thingie"
+                 (build-system gnu-build-system)
+                 (inputs `(("foo" ,grep)
+                           ("bar" ,dep1)))))
+         (t    (options->transformation
+                '((with-c-toolchain . "chbouib=gcc-toolchain")))))
+    ;; Here we check that the transformation applies to DEP0 and all its
+    ;; dependents: DEP0 must use GCC-TOOLCHAIN, DEP1 must use GCC-TOOLCHAIN
+    ;; and the DEP0 that uses GCC-TOOLCHAIN, and so on.
+    (with-store store
+      (let ((new (t store p)))
+        (and (depends-on-toolchain? new "gcc-toolchain")
+             (match (bag-build-inputs (package->bag new))
+               ((("foo" dep0) ("bar" dep1) _ ...)
+                (and (depends-on-toolchain? dep1 "gcc-toolchain")
+                     (not (depends-on-toolchain? dep0 "gcc-toolchain"))
+                     (string=? (package-full-name dep0)
+                               (package-full-name grep))
+                     (match (bag-build-inputs (package->bag dep1))
+                       ((("x" dep) _ ...)
+                        (and (depends-on-toolchain? dep "gcc-toolchain")
+                             (match (bag-build-inputs (package->bag dep))
+                               ((("y" dep) _ ...) ;this one is unchanged
+                                (eq? dep grep))))))))))))))
+
+(test-equal "options->transformation, with-c-toolchain twice"
+  (package-full-name grep)
+  (let* ((dep0 (dummy-package "chbouib"))
+         (dep1 (dummy-package "stuff"))
+         (p    (dummy-package "thingie"
+                 (build-system gnu-build-system)
+                 (inputs `(("foo" ,dep0)
+                           ("bar" ,dep1)
+                           ("baz" ,grep)))))
+         (t    (options->transformation
+                '((with-c-toolchain . "chbouib=clang-toolchain")
+                  (with-c-toolchain . "stuff=clang-toolchain")))))
+    (with-store store
+      (let ((new (t store p)))
+        (and (depends-on-toolchain? new "clang-toolchain")
+             (match (bag-build-inputs (package->bag new))
+               ((("foo" dep0) ("bar" dep1) ("baz" dep2) _ ...)
+                (and (depends-on-toolchain? dep0 "clang-toolchain")
+                     (depends-on-toolchain? dep1 "clang-toolchain")
+                     (not (depends-on-toolchain? dep2 "clang-toolchain"))
+                     (package-full-name dep2)))))))))
+
+(test-assert "options->transformation, with-c-toolchain, no effect"
+  (let ((p (dummy-package "thingie"))
+        (t (options->transformation
+            '((with-c-toolchain . "does-not-exist=gcc-toolchain")))))
+    ;; When it has no effect, '--with-c-toolchain' returns P.
+    (with-store store
+      (eq? (t store p) p))))
+
 (test-assert "options->transformation, without-tests"
   (let* ((dep (dummy-package "dep"))
          (p   (dummy-package "foo"
@@ -286,3 +364,7 @@
                    '(#:tests? #f))))))))
 
 (test-end)
+
+;;; Local Variables:
+;;; eval: (put 'dummy-package 'scheme-indent-function 1)
+;;; End:
-- 
2.28.0

bug#43679: [PATCH v2 0/5] Add '--with-c-toolchain'

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> This is v2 of this patch, with these changes:
>
>   1. ‘with-toolchain’ is replaced by ‘with-c-toolchain’ everywhere,
>      with the understanding that it’s about the C/C++ toolchain
>      in practice.  In the end I’m sympathetic with the argument
>      that C/C++ don’t have a monopoly on toolchains.  ;-)
>
>   2. ‘--with-c-toolchain=PACKAGE=TOOLCHAIN’ rebuilds not just
>      PACKAGE with TOOLCHAIN, but also everything above PACKAGE
>      with TOOLCHAIN (in v1, only PACKAGE was rebuilt with TOOLCHAIN
>      but everything above it had to be rebuilt anyway.)
>
>      The main motivation here is to reduce the changes that we’re
>      introducing ABI incompatibilities that users would have to work
>      around by passing on ‘--with-c-toolchain’ for each package in
>      the chain.  I think it also more closely matches user
>      expectations: when you see things are being rebuilt, you’re
>      likely to think that’s because they’re rebuilt with the new
>      toolchain, not the default one.
>
> Feedback welcome!
>
> Ludo’.
>
> Ludovic Courtès (5):
>   gnu: gcc-toolchain: Add 'GUIX_LOCPATH' to the search paths.
>   gnu: clang-toolchain: Add 'GUIX_LOCPATH' to the search paths.
>   gnu: clang-toolchain: Create 'cc' and 'c++' symlinks.
>   packages: Add 'package-with-c-toolchain'.
>   guix build: Add '--with-c-toolchain'.

Pushed as a2ed00f79fd5bf69c6cca3fa7bdc62726bf848fa!

Ludo’.