Cross-compilation broken on canonical packages.

Cross-compilation broken on canonical packages.

[Mathieu Othacehe]

Hello,

This command fails:

--8<---------------cut here---------------start------------->8---
guix build -e "((@ (gnu packages base) canonical-package) (@ (gnu packages base) grep))"
--8<---------------cut here---------------end--------------->8---

with this output:

--8<---------------cut here---------------start------------->8---
building /gnu/store/nlq0iw749hnyy7nk3cq6pjwqkxv437ca-make-boot0-4.2.1.drv...
starting phase `set-SOURCE-DATE-EPOCH'
phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
starting phase `set-paths'
environment variable `PATH' set to `/gnu/store/36zgy21hfy185nmdgj87ll6q4hj5riiw-gcc-cross-aarch64-linux-gnu-7.4.0/bin:/gnu/store/4d68mhjjwqc95lyrm63yrsnrlqzkbpyr-binutils-cross-aarch64-linux-gnu-2.32/bin'
environment variable `CROSS_LIBRARY_PATH' set to `/gnu/store/pgg0nxh1l39dybb5pwqyskb9qxqz57r9-glibc-mesboot-2.16.0/lib:/gnu/store/p0s4wym402gsnqzicr87zrli43dvfbms-binutils-mesboot-2.20.1a/lib:/gnu/store/bb6s2y6121pxvhwyr1n6dg7pwj88bsbj-gcc-mesboot-4.9.4/lib:/gnu/store/p5r7gprzg774vq8l94924m598i86ryya-glibc-cross-aarch64-linux-gnu-2.29/lib:/gnu/store/z86g7zwj2mdl1328qpgmim262sppjxqp-glibc-cross-aarch64-linux-gnu-2.29-static/lib'
environment variable `CROSS_CPATH' set to `/gnu/store/pgg0nxh1l39dybb5pwqyskb9qxqz57r9-glibc-mesboot-2.16.0/include:/gnu/store/p0s4wym402gsnqzicr87zrli43dvfbms-binutils-mesboot-2.20.1a/include:/gnu/store/bb6s2y6121pxvhwyr1n6dg7pwj88bsbj-gcc-mesboot-4.9.4/include:/gnu/store/bp1ap2whzmwvh2myj31hfmg9pi4sly0r-linux-libre-headers-bootstrap-0/include:/gnu/store/p5r7gprzg774vq8l94924m598i86ryya-glibc-cross-aarch64-linux-gnu-2.29/include:/gnu/store/f3izsng5fzwh6zf2z7aaj5v04jp5hmcy-linux-libre-headers-cross-aarch64-linux-gnu-4.19.56/include'
environment variable `GUIX_LOCPATH' unset
environment variable `C_INCLUDE_PATH' set to `/gnu/store/36zgy21hfy185nmdgj87ll6q4hj5riiw-gcc-cross-aarch64-linux-gnu-7.4.0/include'
environment variable `CPLUS_INCLUDE_PATH' set to `/gnu/store/36zgy21hfy185nmdgj87ll6q4hj5riiw-gcc-cross-aarch64-linux-gnu-7.4.0/include'
environment variable `LIBRARY_PATH' set to `/gnu/store/36zgy21hfy185nmdgj87ll6q4hj5riiw-gcc-cross-aarch64-linux-gnu-7.4.0/lib'
environment variable `C_INCLUDE_PATH' set to `/gnu/store/36zgy21hfy185nmdgj87ll6q4hj5riiw-gcc-cross-aarch64-linux-gnu-7.4.0/include'
phase `set-paths' succeeded after 0.0 seconds
starting phase `install-locale'
warning: failed to install 'en_US.utf8' locale: Invalid argument
phase `install-locale' succeeded after 0.0 seconds
starting phase `unpack'
In execvp of tar: No such file or directory
command "tar" "xvf" "/gnu/store/cr04i9xi5nbkn7lqb1nnxkqvpwy92m78-make-4.2.1.tar.xz" failed with status 127
builder for `/gnu/store/nlq0iw749hnyy7nk3cq6pjwqkxv437ca-make-boot0-4.2.1.drv' failed with exit code 1
build of /gnu/store/nlq0iw749hnyy7nk3cq6pjwqkxv437ca-make-boot0-4.2.1.drv failed
View build log at '/var/log/guix/drvs/nl/q0iw749hnyy7nk3cq6pjwqkxv437ca-make-boot0-4.2.1.drv.gz'.
cannot build derivation `/gnu/store/xpb7jhgjghxy624gz7rfy5zb8rsdri3i-grep-3.3.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/xpb7jhgjghxy624gz7rfy5zb8rsdri3i-grep-3.3.drv' failed
--8<---------------cut here---------------end--------------->8---

This is topic has already been discussed here[1][2]. Usually I work around it
by passing "--no-grafts" or avoiding canonical-packages.

Now that cross compiling a system is possible, avoiding canonical
packages is hard (used in %base-packages and "system" derivation).

Any ideas where to start on this problem?

Thanks,

Mathieu

[1]: https://guix-devel.gnu.narkive.com/LsByvXrK/cross-building-bootstrap-binaries-fail-in-current-master#post5
[2]: https://lists.gnu.org/archive/html/bug-guix/2017-10/msg00101.html

Re: Cross-compilation broken on canonical packages.

[Mathieu Othacehe]

Small mistake sorry. This fails:

--8<---------------cut here---------------start------------->8---
guix build --target=aarch64-linux-gnu -e "((@ (gnu packages base) canonical-package) (@ (gnu packages base) grep))"
--8<---------------cut here---------------end--------------->8---

while this succeeds:

--8<---------------cut here---------------start------------->8---
guix build -e "((@ (gnu packages base) canonical-package) (@ (gnu packages base) grep))"
--8<---------------cut here---------------end--------------->8---

Mathieu

Re: Cross-compilation broken on canonical packages.

[Ludovic Courtès]

Hi Mathieu,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> Small mistake sorry. This fails:
>
> guix build --target=aarch64-linux-gnu -e "((@ (gnu packages base) canonical-package) (@ (gnu packages base) grep))"
>
>
> while this succeeds:
>
> guix build -e "((@ (gnu packages base) canonical-package) (@ (gnu packages base) grep))"

This is expected: packages in ‘%final-inputs’ (those returned by
‘canonical-package’) are rooted in the bootstrap graph and cannot be
cross-compiled.

Perhaps ‘%base-packages’ should use packages form (gnu packages base)
instead?  The reason it uses ‘canonical-package’ is just to avoid
introducing redundant packages in the system, but I don’t think it makes
that much sense.

Thanks,
Ludo’.

Re: Cross-compilation broken on canonical packages.

[Mathieu Othacehe]

[-- Attachment #1: Type: text/plain, Size: 1855 bytes --]


Hello Ludo,

Thanks for your explanation :)

> This is expected: packages in ‘%final-inputs’ (those returned by
> ‘canonical-package’) are rooted in the bootstrap graph and cannot be
> cross-compiled.

Looking at canonical-package in (gnu packages commencement), I see that
there's already a switch on (%current-target-system). The given package
is directly returned if (%current-target-system) is set, which appears
to be what we want!

--8<---------------cut here---------------start------------->8---
         ;; In general we want CANON, except if we're cross-compiling: CANON
         ;; uses explicit inputs, so it is "anchored" in the bootstrapped
         ;; process, with dependencies on things that cannot be
         ;; cross-compiled.
         (if (%current-target-system)
             package
             canon))
--8<---------------cut here---------------end--------------->8---

But, this doesn't work as expected. I guess it is because of
(%current-target-system) evaluation time.

As I'm not fully understand everything here, I would propose to define a
gexp-compiler for "canonical-packages", but there's maybe a better thing
to do?

Anyway, the snippet below works fine with a gexp-compiler (patch
attached) and doesn't work with the current implementation of
canonical-package.

--8<---------------cut here---------------start------------->8---
(use-modules (guix)
             (gnu packages base))

(run-with-store (open-connection)
  (mlet* %store-monad
      ((gexp    -> #~(#$(canonical-package grep)))
       (drv     (gexp->script "test.scm" gexp
                               #:target "aarch64-linux-gnu"))
       (build   (built-derivations (list drv))))
    (return #t)))
--8<---------------cut here---------------end--------------->8---

WDYT?

Thanks,

Mathieu

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-wip-Add-a-canonical-package-gexp-compiler.patch --]
[-- Type: text/x-diff, Size: 3698 bytes --]

From 6266c46181d2880684c89999519423be8d8a8ea3 Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <m.othacehe@gmail.com>
Date: Sun, 22 Dec 2019 17:25:59 +0100
Subject: [PATCH] wip: Add a canonical-package gexp compiler.

---
 gnu/packages/base.scm         |  6 ++++++
 gnu/packages/commencement.scm | 40 +++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)

diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index e8150708c0..22b6e05dae 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -1337,6 +1337,12 @@ package needs iconv ,@(libiconv-if-needed) should be added."
          (proc  (module-ref iface 'canonical-package)))
     (proc package)))
 
+(define-public (canonical-package* package)
+  ;; Avoid circular dependency by lazily resolving 'commencement'.
+  (let* ((iface (resolve-interface '(gnu packages commencement)))
+         (proc  (module-ref iface 'make-canonical-package)))
+    (proc package)))
+
 (define-public (%final-inputs)
   "Return the list of \"final inputs\"."
   ;; Avoid circular dependency by lazily resolving 'commencement'.
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 14ecf246d4..41ab430388 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -54,8 +54,11 @@
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system trivial)
   #:use-module (guix memoization)
+  #:use-module (guix gexp)
+  #:use-module (guix records)
   #:use-module (guix utils)
   #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-9)
   #:use-module (srfi srfi-26)
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 match)
@@ -2613,5 +2616,42 @@ Fortran development to be installed in user profiles.  This includes
 gfortran, as well as libc (headers and binaries, plus debugging symbols
 in the @code{debug} output), and binutils.")))
 
+(define-record-type <canonical-package>
+  (%make-canonical-package name)
+  canonical-package?
+  (name canonical-package-name))
+
+(define-public (make-canonical-package package)
+  (%make-canonical-package package))
+
+(define-gexp-compiler (canonical-package-compiler
+                       (canonical-package <canonical-package>) system target)
+  ;; Return the 'canonical' variant of PACKAGE---i.e., if PACKAGE is one of
+  ;; the implicit inputs of 'gnu-build-system', return that one, otherwise
+  ;; return PACKAGE.
+  ;;
+  ;; The goal is to avoid duplication in cases like GUILE-FINAL vs. GUILE-2.2,
+  ;; COREUTILS-FINAL vs. COREUTILS, etc.
+  (let ((name->package (fold (lambda (input result)
+                               (match input
+                                 ((_ package . outputs)
+                                  (vhash-cons (package-full-name package)
+                                              package result))))
+                             vlist-null
+                             `(("guile" ,guile-final)
+                               ,@%final-inputs)))
+        (package (canonical-package-name canonical-package)))
+    ;; XXX: This doesn't handle dependencies of the final inputs, such as
+    ;; libunistring, GMP, etc.
+    (match (vhash-assoc (package-full-name package) name->package)
+      ((_ . canon)
+       ;; In general we want CANON, except if we're cross-compiling: CANON
+       ;; uses explicit inputs, so it is "anchored" in the bootstrapped
+       ;; process, with dependencies on things that cannot be
+       ;; cross-compiled.
+       (if target
+           (package->cross-derivation package target system)
+           (package->derivation (pk canon) system)))
+      (_ (package->derivation package system)))))
 
 ;;; commencement.scm ends here
-- 
2.24.0

Re: Cross-compilation broken on canonical packages.

[Ludovic Courtès]

Hi,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

>> This is expected: packages in ‘%final-inputs’ (those returned by
>> ‘canonical-package’) are rooted in the bootstrap graph and cannot be
>> cross-compiled.
>
> Looking at canonical-package in (gnu packages commencement), I see that
> there's already a switch on (%current-target-system). The given package
> is directly returned if (%current-target-system) is set, which appears
> to be what we want!
>
>          ;; In general we want CANON, except if we're cross-compiling: CANON
>          ;; uses explicit inputs, so it is "anchored" in the bootstrapped
>          ;; process, with dependencies on things that cannot be
>          ;; cross-compiled.
>          (if (%current-target-system)
>              package
>              canon))
>
>
> But, this doesn't work as expected. I guess it is because of
> (%current-target-system) evaluation time.

Ah yeah.  The hack works when you use ‘canonical-package’ in a package
input, because package input fields are “thunked”, so they’re evaluated
in a dynamic context where ‘%current-target-system’ has the right value.
  
However, the hack doesn’t work for things like the ‘packages’ field of
<operating-system> because it’s not a thunked field.

Two simple solutions here, I think:

  1. Make ‘packages’ a thunked field.

  2. Stop using ‘canonical-package’ altogether in ‘%base-packages’.

I actually have a preference for #2.  We’d need to check what impact it
has on the system closure size, but I suspect it’s going to be minimal.

Thoughts?

(The <canonical-package> trick you posted should work of course, but
it’s even better if we can avoid it!)

Thanks,
Ludo’.

Re: Cross-compilation broken on canonical packages.

[Mathieu Othacehe]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]


Hola,

> However, the hack doesn’t work for things like the ‘packages’ field of
> <operating-system> because it’s not a thunked field.

I see! Thanks for explaining.

>
> Two simple solutions here, I think:
>
>   1. Make ‘packages’ a thunked field.
>
>   2. Stop using ‘canonical-package’ altogether in ‘%base-packages’.
>
> I actually have a preference for #2.  We’d need to check what impact it
> has on the system closure size, but I suspect it’s going to be minimal.
>
> Thoughts?

Option #2 seems nicer. There are other canonical-package calls when
building services. I have a patch attached that removes all those calls.

The image produced when running "guix system disk-image bare-bones.tmpl"
increases by:

* 30MB when removing the canonical-package of %base-packages
* 30MB for all other calls.

So altogether, that's a 60MB increase on a 1.5GB image (4%). I find it
acceptable, WDYT?

Thanks,

Mathieu

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-system-Stop-using-canonical-package.patch --]
[-- Type: text/x-diff, Size: 8022 bytes --]

From a55eb86975c5f1c2783b0af265364821259e3351 Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <m.othacehe@gmail.com>
Date: Tue, 31 Dec 2019 09:56:51 +0100
Subject: [PATCH] system: Stop using canonical-package.

Usage of canonical-package outside of thunked fields breaks cross-compilation,
see: https://lists.gnu.org/archive/html/guix-devel/2019-12/msg00410.html.

* gnu/installer.scm (installer-program): Remove canonical-package.
* gnu/services/base.scm (<nscd-cache>): Ditto,
(%base-services): ditto.
* gnu/services/xorg.scm: Remove useless canonical-package import.
* gnu/system.scm (%base-packages): Remove canonical-package.
* gnu/system/install.scm (%installation-services): Ditto,
(installation-os): ditto.
* gnu/system/locale.scm (single-locale-directory): Ditto,
(%default-locale-libcs): ditto.
---
 gnu/installer.scm      | 22 +++++++++++-----------
 gnu/services/base.scm  | 10 ++++------
 gnu/services/xorg.scm  |  2 +-
 gnu/system.scm         | 11 ++---------
 gnu/system/install.scm |  7 +++----
 gnu/system/locale.scm  |  4 ++--
 6 files changed, 23 insertions(+), 33 deletions(-)

diff --git a/gnu/installer.scm b/gnu/installer.scm
index 167653263f..7f12b39ffc 100644
--- a/gnu/installer.scm
+++ b/gnu/installer.scm
@@ -290,17 +290,17 @@ selected keymap."
   (define set-installer-path
     ;; Add the specified binary to PATH for later use by the installer.
     #~(let* ((inputs
-              '#$(append (list bash ;start subshells
-                               connman ;call connmanctl
-                               cryptsetup
-                               dosfstools ;mkfs.fat
-                               e2fsprogs ;mkfs.ext4
-                               btrfs-progs
-                               kbd ;chvt
-                               guix ;guix system init call
-                               util-linux ;mkwap
-                               shadow)
-                         (map canonical-package (list coreutils)))))
+              '#$(list bash ;start subshells
+                       connman ;call connmanctl
+                       cryptsetup
+                       dosfstools ;mkfs.fat
+                       e2fsprogs ;mkfs.ext4
+                       btrfs-progs
+                       kbd ;chvt
+                       guix ;guix system init call
+                       util-linux ;mkwap
+                       shadow
+                       coreutils)))
         (with-output-to-port (%make-void-port "w")
           (lambda ()
             (set-path-environment-variable "PATH" '("bin" "sbin") inputs)))))
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index b1eff89ecc..b7cde09611 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -44,7 +44,7 @@
                 #:select (alsa-utils crda eudev e2fsprogs fuse gpm kbd lvm2 rng-tools))
   #:use-module (gnu packages bash)
   #:use-module ((gnu packages base)
-                #:select (canonical-package coreutils glibc glibc-utf8-locales))
+                #:select (coreutils glibc glibc-utf8-locales))
   #:use-module (gnu packages package-management)
   #:use-module ((gnu packages gnupg) #:select (guile-gcrypt))
   #:use-module (gnu packages linux)
@@ -1208,7 +1208,7 @@ the tty to run, among other things."
   (name-services nscd-configuration-name-services ;list of <packages>
                  (default '()))
   (glibc      nscd-configuration-glibc            ;<package>
-              (default (canonical-package glibc))))
+              (default glibc)))
 
 (define-record-type* <nscd-cache> nscd-cache make-nscd-cache
   nscd-cache?
@@ -2441,9 +2441,7 @@ to handle."
                    (rules (list lvm2 fuse alsa-utils crda))))
 
         (service special-files-service-type
-                 `(("/bin/sh" ,(file-append (canonical-package bash)
-                                            "/bin/sh"))
-                   ("/usr/bin/env" ,(file-append (canonical-package coreutils)
-                                                 "/bin/env"))))))
+                 `(("/bin/sh" ,(file-append bash "/bin/sh"))
+                   ("/usr/bin/env" ,(file-append coreutils "/bin/env"))))))
 
 ;;; base.scm ends here
diff --git a/gnu/services/xorg.scm b/gnu/services/xorg.scm
index 9c84f7413f..4f6a8aec1e 100644
--- a/gnu/services/xorg.scm
+++ b/gnu/services/xorg.scm
@@ -29,7 +29,7 @@
   #:use-module (gnu system keyboard)
   #:use-module (gnu services base)
   #:use-module (gnu services dbus)
-  #:use-module ((gnu packages base) #:select (canonical-package))
+  #:use-module (gnu packages base)
   #:use-module (gnu packages guile)
   #:use-module (gnu packages xorg)
   #:use-module (gnu packages fonts)
diff --git a/gnu/system.scm b/gnu/system.scm
index abdbb081e6..3efa496194 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -591,18 +591,11 @@ of PROVENANCE-SERVICE-TYPE to its services."
 
          bash-completion
 
-         ;; XXX: We don't use (canonical-package guile-2.2) here because that
-         ;; would create a collision in the global profile between the GMP
-         ;; variant propagated by 'guile-final' and the GMP variant propagated
-         ;; by 'gnutls', itself propagated by 'guix'.
          guile-2.2
          guile-readline guile-colorized
 
-         ;; The packages below are also in %FINAL-INPUTS, so take them from
-         ;; there to avoid duplication.
-         (map canonical-package
-              (list bash coreutils findutils grep sed
-                    diffutils patch gawk tar gzip bzip2 xz lzip))))
+         bash coreutils findutils grep sed
+         diffutils patch gawk tar gzip bzip2 xz lzip))
 
 (define %default-issue
   ;; Default contents for /etc/issue.
diff --git a/gnu/system/install.scm b/gnu/system/install.scm
index 4d1612ac7f..64fd1d1747 100644
--- a/gnu/system/install.scm
+++ b/gnu/system/install.scm
@@ -383,8 +383,7 @@ Access documentation at any time by pressing Alt-F2.\x1b[0m
           ;; Having /bin/sh is a good idea.  In particular it allows Tramp
           ;; connections to this system to work.
           (service special-files-service-type
-                   `(("/bin/sh" ,(file-append (canonical-package bash)
-                                              "/bin/sh"))))
+                   `(("/bin/sh" ,(file-append bash "/bin/sh"))))
 
           ;; Loopback device, needed by OpenSSH notably.
           (service static-networking-service-type
@@ -407,7 +406,7 @@ Access documentation at any time by pressing Alt-F2.\x1b[0m
                    (list bare-bones-os
                          glibc-utf8-locales
                          texinfo
-                         (canonical-package guile-2.2))))))
+                         guile-2.2)))))
 
 (define %issue
   ;; Greeting.
@@ -479,7 +478,7 @@ Access documentation at any time by pressing Alt-F2.\x1b[0m
      ;; Explicitly allow for empty passwords.
      (base-pam-services #:allow-empty-passwords? #t))
 
-    (packages (cons* (canonical-package glibc) ;for 'tzselect' & co.
+    (packages (cons* glibc ;for 'tzselect' & co.
                      parted gptfdisk ddrescue
                      fontconfig
                      font-dejavu font-gnu-unifont
diff --git a/gnu/system/locale.scm b/gnu/system/locale.scm
index 8466d5b07d..c7a371e9bf 100644
--- a/gnu/system/locale.scm
+++ b/gnu/system/locale.scm
@@ -86,7 +86,7 @@ or #f on failure."
      #f)))
 
 (define* (single-locale-directory locales
-                                  #:key (libc (canonical-package glibc)))
+                                  #:key (libc glibc))
   "Return a directory containing all of LOCALES for LIBC compiled.
 
 Because locale data formats are incompatible when switching from one libc to
@@ -147,7 +147,7 @@ data format changes between libc versions."
 
 (define %default-locale-libcs
   ;; The libcs for which we build locales by default.
-  (list (canonical-package glibc)))
+  (list glibc))
 
 (define %default-locale-definitions
   ;; Arbitrary set of locales that are built by default.  They are here mostly
-- 
2.24.1

Re: Cross-compilation broken on canonical packages.

[Ludovic Courtès]

Hola!

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

>> Two simple solutions here, I think:
>>
>>   1. Make ‘packages’ a thunked field.
>>
>>   2. Stop using ‘canonical-package’ altogether in ‘%base-packages’.
>>
>> I actually have a preference for #2.  We’d need to check what impact it
>> has on the system closure size, but I suspect it’s going to be minimal.
>>
>> Thoughts?
>
> Option #2 seems nicer. There are other canonical-package calls when
> building services. I have a patch attached that removes all those calls.
>
> The image produced when running "guix system disk-image bare-bones.tmpl"
> increases by:
>
> * 30MB when removing the canonical-package of %base-packages
> * 30MB for all other calls.

OK.  (It may be slightly more accurate to look at the output of:
guix size $(guix system build …).)

> So altogether, that's a 60MB increase on a 1.5GB image (4%). I find it
> acceptable, WDYT?

It’s more than I thought but I think it’s OK.  (Also, how come
bare-bones takes 1.5 GiB?!)

> From a55eb86975c5f1c2783b0af265364821259e3351 Mon Sep 17 00:00:00 2001
> From: Mathieu Othacehe <m.othacehe@gmail.com>
> Date: Tue, 31 Dec 2019 09:56:51 +0100
> Subject: [PATCH] system: Stop using canonical-package.
>
> Usage of canonical-package outside of thunked fields breaks cross-compilation,
> see: https://lists.gnu.org/archive/html/guix-devel/2019-12/msg00410.html.
>
> * gnu/installer.scm (installer-program): Remove canonical-package.
> * gnu/services/base.scm (<nscd-cache>): Ditto,
> (%base-services): ditto.
> * gnu/services/xorg.scm: Remove useless canonical-package import.
> * gnu/system.scm (%base-packages): Remove canonical-package.
> * gnu/system/install.scm (%installation-services): Ditto,
> (installation-os): ditto.
> * gnu/system/locale.scm (single-locale-directory): Ditto,
> (%default-locale-libcs): ditto.

[...]

> --- a/gnu/system/locale.scm
> +++ b/gnu/system/locale.scm
> @@ -86,7 +86,7 @@ or #f on failure."
>       #f)))
>  
>  (define* (single-locale-directory locales
> -                                  #:key (libc (canonical-package glibc)))
> +                                  #:key (libc glibc))
>    "Return a directory containing all of LOCALES for LIBC compiled.
>  
>  Because locale data formats are incompatible when switching from one libc to
> @@ -147,7 +147,7 @@ data format changes between libc versions."
>  
>  (define %default-locale-libcs
>    ;; The libcs for which we build locales by default.
> -  (list (canonical-package glibc)))
> +  (list glibc))

I would leave this bit unchanged if possible, because that way the glibc
used would really match that programs are linked against.

The rest LGTM, thanks!

Ludo’.

Re: Cross-compilation broken on canonical packages.

[Mathieu Othacehe]

Hi,

> It’s more than I thought but I think it’s OK.  (Also, how come
> bare-bones takes 1.5 GiB?!)

That's one of my next subject of investigation :)

>>  (define %default-locale-libcs
>>    ;; The libcs for which we build locales by default.
>> -  (list (canonical-package glibc)))
>> +  (list glibc))
>
> I would leave this bit unchanged if possible, because that way the glibc
> used would really match that programs are linked against.

operating-system-locale-libcs defaults to %default-locale-libcs, so
keeping (canonical-package glibc) prevents cross-compilation :(

We could use a thunk field to write something like:

--8<---------------cut here---------------start------------->8---
(define (default-locale-libcs ...)
  (if target glibc (canonical-package glibc))
--8<---------------cut here---------------end--------------->8---

But I'd like to avoid that. I'll see if I can figure out something
better after a good night.

Thanks,

Mathieu

Re: Cross-compilation broken on canonical packages.

[Mathieu Othacehe]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]


Hello,

> We could use a thunk field to write something like:
>
> --8<---------------cut here---------------start------------->8---
> (define (default-locale-libcs ...)
>   (if target glibc (canonical-package glibc))
> --8<---------------cut here---------------end--------------->8---
>
> But I'd like to avoid that. I'll see if I can figure out something
> better after a good night.

Couldn't find much better, here's a patch, tell my what you think!

Thanks,

Mathieu

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-defaut-locale-libc.patch --]
[-- Type: text/x-diff, Size: 2253 bytes --]

From e1728d6c79f63ad72bcfb4e45111ccb9f43f1251 Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <m.othacehe@gmail.com>
Date: Fri, 3 Jan 2020 09:23:51 +0100
Subject: [PATCH] defaut locale libc.

---
 gnu/system.scm        |  3 ++-
 gnu/system/locale.scm | 10 +++++-----
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/gnu/system.scm b/gnu/system.scm
index 0842bd47c7..faff0320d3 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -211,7 +211,8 @@
   (locale-definitions operating-system-locale-definitions ; list of <locale-definition>
                       (default %default-locale-definitions))
   (locale-libcs operating-system-locale-libcs     ; list of <packages>
-                (default %default-locale-libcs))
+                (thunked)
+                (default (default-locale-libcs)))
   (name-service-switch operating-system-name-service-switch ; <name-service-switch>
                        (default %default-nss))
 
diff --git a/gnu/system/locale.scm b/gnu/system/locale.scm
index c7a371e9bf..361c024eeb 100644
--- a/gnu/system/locale.scm
+++ b/gnu/system/locale.scm
@@ -37,7 +37,7 @@
             locale-name->definition
             locale-directory
 
-            %default-locale-libcs
+            default-locale-libcs
             %default-locale-definitions
 
             glibc-supported-locales))
@@ -123,7 +123,7 @@ of LIBC."
   (computed-file (string-append "locale-" version) build))
 
 (define* (locale-directory locales
-                           #:key (libcs %default-locale-libcs))
+                           #:key (libcs (default-locale-libcs)))
   "Return a locale directory containing all of LOCALES for each libc package
 listed in LIBCS.
 
@@ -145,9 +145,9 @@ data format changes between libc versions."
                       #:options '(#:local-build? #t
                                   #:substitutable? #f))))))
 
-(define %default-locale-libcs
-  ;; The libcs for which we build locales by default.
-  (list glibc))
+(define* (default-locale-libcs
+          #:key (target (%current-target-system)))
+  (if target (list glibc) (list (canonical-package glibc))))
 
 (define %default-locale-definitions
   ;; Arbitrary set of locales that are built by default.  They are here mostly
-- 
2.24.1

Re: Cross-compilation broken on canonical packages.

[Mathieu Othacehe]

Hey Ludo,

>> But I'd like to avoid that. I'll see if I can figure out something
>> better after a good night.
>
> Couldn't find much better, here's a patch, tell my what you think!

I pushed a patch removing all canonical-packages calls, except the one
for the default-locales-libcs variable.

About this variable, I don't know if the dirty patch I proposed is
better than accepting that the glibc used for locales is different from
the one programs are linked against. WDYT?

Thanks,

Mathieu

Re: Cross-compilation broken on canonical packages.

[Ludovic Courtès]

Hi,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

>>> But I'd like to avoid that. I'll see if I can figure out something
>>> better after a good night.
>>
>> Couldn't find much better, here's a patch, tell my what you think!
>
> I pushed a patch removing all canonical-packages calls, except the one
> for the default-locales-libcs variable.
>
> About this variable, I don't know if the dirty patch I proposed is
> better than accepting that the glibc used for locales is different from
> the one programs are linked against. WDYT?

On second thought, I think removing ‘canonical-package’ calls from (gnu
system locale) should work.  What matters there is that we build locales
with the matching libc (same version, not necessarily same store item.)

The only downside will be potentially an extra glibc download/build to
build the locale set, but that’s probably OK.

WDYT?

Thanks,
Ludo’.

Re: Cross-compilation broken on canonical packages.

[Mathieu Othacehe]

[-- Attachment #1: Type: text/plain, Size: 566 bytes --]


Hello,

> The only downside will be potentially an extra glibc download/build to
> build the locale set, but that’s probably OK.
>
> WDYT?

I think you're right! Pushed it as
f30d84d32db0f4f6cb84e139868e1727a7dc0a51 on core-updates.

Now the good news is that we are one patch away from having core-updates
to cross-compile bare-bones.tmpl system! This is the patch attached,
we've been discussing it here[1] without reaching a conclusion.

WDYT?

Thanks,

Mathieu

[1]: https://lists.gnu.org/archive/html/guix-patches/2019-12/msg00751.html

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gexp-Default-to-current-target.patch --]
[-- Type: text/x-diff, Size: 7340 bytes --]

From f5d773dc0a627ba6059f1025189d33a9e0d083b5 Mon Sep 17 00:00:00 2001
From: Mathieu Othacehe <m.othacehe@gmail.com>
Date: Sat, 28 Dec 2019 21:29:06 +0100
Subject: [PATCH 1/2] gexp: Default to current target.

* guix/gexp.scm (lower-object): Set target argument to 'current by default and
look for the current target system at bind time if needed,
(gexp->file): ditto,
(gexp->script): ditto.
---
 guix/gexp.scm | 87 ++++++++++++++++++++++++++++++---------------------
 1 file changed, 51 insertions(+), 36 deletions(-)

diff --git a/guix/gexp.scm b/guix/gexp.scm
index 12331052a6..11d4e86037 100644
--- a/guix/gexp.scm
+++ b/guix/gexp.scm
@@ -215,7 +215,7 @@ procedure to expand it; otherwise return #f."
 
 (define* (lower-object obj
                        #:optional (system (%current-system))
-                       #:key target)
+                       #:key (target 'current))
   "Return as a value in %STORE-MONAD the derivation or store item
 corresponding to OBJ for SYSTEM, cross-compiling for TARGET if TARGET is true.
 OBJ must be an object that has an associated gexp compiler, such as a
@@ -225,7 +225,10 @@ OBJ must be an object that has an associated gexp compiler, such as a
      (raise (condition (&gexp-input-error (input obj)))))
     (lower
      ;; Cache in STORE the result of lowering OBJ.
-     (mlet %store-monad ((graft? (grafting?)))
+     (mlet %store-monad ((target (if (eq? target 'current)
+                                     (current-target-system)
+                                     (return target)))
+                         (graft? (grafting?)))
        (mcached (let ((lower (lookup-compiler obj)))
                   (lower obj system target))
                 obj
@@ -1571,16 +1574,19 @@ are searched for in PATH.  Return #f when MODULES and EXTENSIONS are empty."
                        #:key (guile (default-guile))
                        (module-path %load-path)
                        (system (%current-system))
-                       target)
+                       (target 'current))
   "Return an executable script NAME that runs EXP using GUILE, with EXP's
 imported modules in its search path.  Look up EXP's modules in MODULE-PATH."
-  (mlet %store-monad ((set-load-path
-                       (load-path-expression (gexp-modules exp)
-                                             module-path
-                                             #:extensions
-                                             (gexp-extensions exp)
-                                             #:system system
-                                             #:target target)))
+  (mlet* %store-monad ((target (if (eq? target 'current)
+                                   (current-target-system)
+                                   (return target)))
+                       (set-load-path
+                        (load-path-expression (gexp-modules exp)
+                                              module-path
+                                              #:extensions
+                                              (gexp-extensions exp)
+                                              #:system system
+                                              #:target target)))
     (gexp->derivation name
                       (gexp
                        (call-with-output-file (ungexp output)
@@ -1609,7 +1615,7 @@ imported modules in its search path.  Look up EXP's modules in MODULE-PATH."
                      (module-path %load-path)
                      (splice? #f)
                      (system (%current-system))
-                     target)
+                     (target 'current))
   "Return a derivation that builds a file NAME containing EXP.  When SPLICE?
 is true, EXP is considered to be a list of expressions that will be spliced in
 the resulting file.
@@ -1620,36 +1626,45 @@ Lookup EXP's modules in MODULE-PATH."
   (define modules (gexp-modules exp))
   (define extensions (gexp-extensions exp))
 
-  (if (or (not set-load-path?)
-          (and (null? modules) (null? extensions)))
-      (gexp->derivation name
-                        (gexp
-                         (call-with-output-file (ungexp output)
-                           (lambda (port)
-                             (for-each (lambda (exp)
-                                         (write exp port))
-                                       '(ungexp (if splice?
-                                                    exp
-                                                    (gexp ((ungexp exp)))))))))
-                        #:local-build? #t
-                        #:substitutable? #f
-                        #:system system
-                        #:target target)
-      (mlet %store-monad ((set-load-path
-                           (load-path-expression modules module-path
-                                                 #:extensions extensions
-                                                 #:system system
-                                                 #:target target)))
+  (mlet* %store-monad
+      ((target (if (eq? target 'current)
+                   (current-target-system)
+                   (return target)))
+       (no-load-path? -> (or (not set-load-path?)
+                             (and (null? modules)
+                                  (null? extensions))))
+       (set-load-path
+        (or (return no-load-path?)
+            (load-path-expression modules module-path
+                                  #:extensions extensions
+                                  #:system system
+                                  #:target target))))
+    (if no-load-path?
+        (gexp->derivation name
+                          (gexp
+                           (call-with-output-file (ungexp output)
+                             (lambda (port)
+                               (for-each
+                                (lambda (exp)
+                                  (write exp port))
+                                '(ungexp (if splice?
+                                             exp
+                                             (gexp ((ungexp exp)))))))))
+                          #:local-build? #t
+                          #:substitutable? #f
+                          #:system system
+                          #:target target)
         (gexp->derivation name
                           (gexp
                            (call-with-output-file (ungexp output)
                              (lambda (port)
                                (write '(ungexp set-load-path) port)
-                               (for-each (lambda (exp)
-                                           (write exp port))
-                                         '(ungexp (if splice?
-                                                      exp
-                                                      (gexp ((ungexp exp)))))))))
+                               (for-each
+                                (lambda (exp)
+                                  (write exp port))
+                                '(ungexp (if splice?
+                                             exp
+                                             (gexp ((ungexp exp)))))))))
                           #:module-path module-path
                           #:local-build? #t
                           #:substitutable? #f
-- 
2.24.1

Re: Cross-compilation broken on canonical packages.

[Ludovic Courtès]

Hi!

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> I think you're right! Pushed it as
> f30d84d32db0f4f6cb84e139868e1727a7dc0a51 on core-updates.
>
> Now the good news is that we are one patch away from having core-updates
> to cross-compile bare-bones.tmpl system! This is the patch attached,
> we've been discussing it here[1] without reaching a conclusion.

Yay!

> From f5d773dc0a627ba6059f1025189d33a9e0d083b5 Mon Sep 17 00:00:00 2001
> From: Mathieu Othacehe <m.othacehe@gmail.com>
> Date: Sat, 28 Dec 2019 21:29:06 +0100
> Subject: [PATCH 1/2] gexp: Default to current target.
>
> * guix/gexp.scm (lower-object): Set target argument to 'current by default and
> look for the current target system at bind time if needed,
> (gexp->file): ditto,
> (gexp->script): ditto.

Do we need to do it for ‘lower-object’?  I think that one is (almost)
always called from a gexp compiler where it’s explicitly passed ‘system’
and ‘target’.

Also, a test or two would be welcome as it’s easy to break it without
noticing.

Otherwise LGTM, thank you, and sorry for the delay!

Ludo’.

Re: Cross-compilation broken on canonical packages.

[Mathieu Othacehe]

Hey Ludo,

> Do we need to do it for ‘lower-object’?  I think that one is (almost)
> always called from a gexp compiler where it’s explicitly passed ‘system’
> and ‘target’.

Yes because of lower-object call in "system-derivation" procedure of
(gnu services).

> Also, a test or two would be welcome as it’s easy to break it without
> noticing.
>
> Otherwise LGTM, thank you, and sorry for the delay!

Pushed with a few tests on core-updates, thanks for reviewing!

Mathieu