bug#50493: Update go-build-system to use Go 1.17.

bug#50493: Update go-build-system to use Go 1.17.

[Sarah Morgensen]

Hello Guix,

This is intended to be a tracking issue for updating go-build-system to
use Go 1.17 by default.  The build system currently uses Go 1.14, which
no longer receives security updates [0].

Here's what needs to be done:

1. Address the aarch64 bug [1], either by verifying that it only happens
on emulated systems, or by fixing it if not.

2. Make some much-needed improvements to go-build-system [2]:

* replace the *slow* phase 'remove-go-references with -trimpath option

* avoid building a directory union of Go inputs (and then copying it!)

* add a "go-std-cache-for-build" package (hidden; name to be
bikeshedded), which will be an implicit input in go-build-system (needed
since we removed the pre-built standard library in go-1.17)

3. Some/all Docker packages still require Go 1.14; make them use go-1.14
explicitly.  Alternatively, update Docker (significant effort involved).

4. Update or patch broken packages.  I've got patches for four so far:

go-github-com-urfave-cli
go-github-com-urfave-cli-v2
go-github-com-sevlyar-go-daemon
go-github-com-keybase-go-ps

I'll submit them as a separate patch and block this with it.  If anyone
else would like to see if their favorite package works with Go 1.17, you
can build it with:

  go build --with-input=go@1.14=go@1.17 <package>

[0] https://github.com/golang/go/wiki/Go-Release-Cycle#release-maintenance
[1] https://issues.guix.gnu.org/49921
[2] https://issues.guix.gnu.org/50227#10

--
Sarah

bug#50493: Update go-build-system to use Go 1.17.

[Leo Famulari]

On Thu, Sep 09, 2021 at 03:14:57PM -0700, Sarah Morgensen wrote:
> This is intended to be a tracking issue for updating go-build-system to
> use Go 1.17 by default.  The build system currently uses Go 1.14, which
> no longer receives security updates [0].

Thanks for putting this together.

> Here's what needs to be done:
> 
> 1. Address the aarch64 bug [1], either by verifying that it only happens
> on emulated systems, or by fixing it if not.

It looks like there was a patch for this issue:

https://github.com/golang/go/issues/20763#issuecomment-314627882

And it was apparently added to Go, but maybe it was not sufficient, did
not make it into a release, or we are seeing a different bug.

> 2. Make some much-needed improvements to go-build-system [2]:
> 
> * replace the *slow* phase 'remove-go-references with -trimpath option
> 
> * avoid building a directory union of Go inputs (and then copying it!)

Are these required to use 1.17? I agree they are important problems to
fix but we shouldn't let them block the upgrade.

> * add a "go-std-cache-for-build" package (hidden; name to be
> bikeshedded), which will be an implicit input in go-build-system (needed
> since we removed the pre-built standard library in go-1.17)

Okay. Feel free to choose the name. I think it's your shed.


> 3. Some/all Docker packages still require Go 1.14; make them use go-1.14
> explicitly.  Alternatively, update Docker (significant effort involved).

Okay.

> 4. Update or patch broken packages.  I've got patches for four so far:
> 
> go-github-com-urfave-cli
> go-github-com-urfave-cli-v2
> go-github-com-sevlyar-go-daemon
> go-github-com-keybase-go-ps
>
> I'll submit them as a separate patch and block this with it.  If anyone
> else would like to see if their favorite package works with Go 1.17, you
> can build it with:

Okay, great.
  
>   go build --with-input=go@1.14=go@1.17 <package>

Syncthing builds! That's great.

bug#50493: [PATCH staging 0/4] build-system/go: Use go-1.17 by default.

[Sarah Morgensen]

Hello Guix,

This patch series updates the Go build system to use Go 1.17 by default.

As a necessity, because go-1.17 no longer contains its own pre-compiled
standard library, an implicit input is added to the build system which
contains the build cache from building the standard library with the
go-build-system.  (This way, we can always guarantee that the cache is valid
because it's built with the same flags as the build system.)  The build system
is also modified to make use of this cache.

(We use the build cache rather than archives because the Go project is moving
away from archives towards an authoritative build cache.)

Three packages are modified to keep using an older Go:

* docker now explicitly uses go-1.14

* go-ipfs and go-github-com-go-ipfs-cmdkit-files now explitly use go-1.16
(go-ipfs and gx badly need to be upgraded!)

There are about 900 dependent packages according to `guix refresh -l go', but
I don't know how many are leaf packages, so I marked this for staging.  The
actual series is based on master because go-1.17 is only present in master.

I have built them all out successfully on x86_64.

Sarah Morgensen (4):
  build-system/go: Add #:substitutable? argument.
  build-system/go: Initialize build cache from input packages.
  build-system/go: Add pre-built standard library as implicit input.
  build-system/go: Use go-1.17 by default.

 gnu/packages/docker.scm        |  2 +-
 gnu/packages/golang.scm        | 32 +++++++++++++++++++++++++++++++-
 gnu/packages/ipfs.scm          |  6 ++++--
 guix/build-system/go.scm       | 24 ++++++++++++++++++++----
 guix/build/go-build-system.scm | 26 +++++++++++++++++++++++---
 5 files changed, 79 insertions(+), 11 deletions(-)


base-commit: a640ee7da3311e4fa560c7163d2dd7d24a25585a
-- 
2.33.0

bug#50493: [PATCH 1/4] build-system/go: Add #:substitutable? argument.

[Sarah Morgensen]

* guix/build-system/go.scm (go-build): Add 'substitutable?' argument.
(go-cross-build): Likewise.
---
 guix/build-system/go.scm | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index 4c1a732107..8aa065c4ff 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -169,7 +169,8 @@ (define* (go-build store name inputs
                    (imported-modules %go-build-system-modules)
                    (modules '((guix build go-build-system)
                               (guix build union)
-                              (guix build utils))))
+                              (guix build utils)))
+                   (substitutable? #t))
   (define builder
    `(begin
       (use-modules ,@modules)
@@ -211,6 +212,7 @@ (define* (go-build store name inputs
                                 #:system system
                                 #:modules imported-modules
                                 #:outputs outputs
+                                #:substitutable? substitutable?
                                 #:guile-for-build guile-for-build))
 
 (define* (go-cross-build store name
@@ -234,7 +236,8 @@ (define* (go-cross-build store name
                          (imported-modules %go-build-system-modules)
                          (modules '((guix build go-build-system)
                                     (guix build union)
-                                    (guix build utils))))
+                                    (guix build utils)))
+                         (substitutable? #t))
   "Cross-build NAME using GO, where TARGET is a GNU triplet and with INPUTS."
   (define builder
     `(begin
@@ -303,6 +306,7 @@ (define* (go-cross-build store name
                                   #:inputs (append native-drvs target-drvs)
                                   #:outputs outputs
                                   #:modules imported-modules
+                                  #:substitutable? substitutable?
                                   #:guile-for-build guile-for-build))
 
 (define go-build-system
-- 
2.33.0

bug#50493: [PATCH 2/4] build-system/go: Initialize build cache from input packages.

[Sarah Morgensen]

* guix/build/go-build-system.com (setup-go-environment): Set GOCACHE to
a location within the build directory.  Union "/var/cache/go/build"
input directories to initialize the cache.  Generate "trim.txt" within
the cache, with the current time.
---
 guix/build/go-build-system.scm | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/guix/build/go-build-system.scm b/guix/build/go-build-system.scm
index 645d2fe680..9f249b0777 100644
--- a/guix/build/go-build-system.scm
+++ b/guix/build/go-build-system.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2020 Jack Hill <jackhill@jackhill.us>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2020, 2021 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -138,9 +139,28 @@ (define* (setup-go-environment #:key inputs outputs goos goarch #:allow-other-ke
 where executables (\"commands\") are installed to.  This phase is sometimes used
 by packages that use (guix build-system gnu) but have a handful of Go
 dependencies, so it should be self-contained."
-  ;; The Go cache is required starting in Go 1.12.  We don't actually use it but
-  ;; we need it to be a writable directory.
-  (setenv "GOCACHE" "/tmp/go-cache")
+  (define (search-input-directories dir)
+    (filter directory-exists?
+            (map (match-lambda
+                   ((name . directory)
+                    (string-append directory "/" dir)))
+                 inputs)))
+
+  ;; Seed the Go build cache with the build caches from input packages.
+  (let ((cache (string-append (getcwd) "/go-build")))
+    (setenv "GOCACHE" cache)
+    (union-build cache
+                 (search-input-directories "/var/cache/go/build")
+                 ;; Creating all directories isn't that bad, because there are
+                 ;; only ever 256 of them.
+                 #:create-all-directories? #t
+                 #:log-port (%make-void-port "w"))
+
+    ;; Tell Go that the cache was recently trimmed, so it doesn't try to.
+    (call-with-output-file (string-append cache "/trim.txt")
+      (lambda (port)
+        (format port "~a" (current-time)))))
+
   ;; Using the current working directory as GOPATH makes it easier for packagers
   ;; who need to manipulate the unpacked source code.
   (setenv "GOPATH" (getcwd))
-- 
2.33.0

bug#50493: [PATCH 3/4] build-system/go: Add pre-built standard library as implicit input.

[Sarah Morgensen]

* gnu/packages/golang.scm (make-go-std): New procedure.
* guix/build-system/go.scm (make-go-std): New procedure.
(lower): Use it.  Add pre-built standard library to inputs.
---
 gnu/packages/golang.scm  | 30 ++++++++++++++++++++++++++++++
 guix/build-system/go.scm | 16 ++++++++++++++--
 2 files changed, 44 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index f02d0aa9df..19f1ef37d5 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -792,6 +792,36 @@ (define-public go-1.17
 
 (define-public go go-1.14)
 
+(define-public (make-go-std go)
+  "Return a package which builds the standard library for Go compiler GO."
+  (package
+    (name (string-append (package-name go) "-std"))
+    (version (package-version go))
+    (source #f)
+    (build-system go-build-system)
+    (arguments
+     `(#:import-path "std"
+       #:build-flags `("-pkgdir" "pkg") ; "Install" to build directory.
+       #:allow-go-reference? #t
+       #:substitutable? #f ; Faster to build than download.
+       #:tests? #f ; Already tested in the main Go build.
+       #:go ,go
+       #:phases
+       (modify-phases %standard-phases
+         (delete 'unpack)
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (out-cache (string-append out "/var/cache/go/build")))
+               (copy-recursively (getenv "GOCACHE") out-cache)
+               (delete-file (string-append out-cache "/trim.txt"))
+               (delete-file (string-append out-cache "/README")))))
+         (delete 'install-license-files))))
+    (home-page (package-home-page go))
+    (synopsis "Cached standard library build for Go")
+    (description (package-description go))
+    (license (package-license go))))
+
 (define-public go-0xacab-org-leap-shapeshifter
   (let ((commit "0aa6226582efb8e563540ec1d3c5cfcd19200474")
         (revision "12"))
diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index 8aa065c4ff..a579477500 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2017 Leo Famulari <leo@famulari.name>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2021 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -109,6 +110,9 @@ (define (default-go)
   (let ((go (resolve-interface '(gnu packages golang))))
     (module-ref go 'go)))
 
+(define (make-go-std)
+  (module-ref (resolve-interface '(gnu packages golang)) 'make-go-std))
+
 (define* (lower name
                 #:key source inputs native-inputs outputs system target
                 (go (default-go))
@@ -118,6 +122,14 @@ (define* (lower name
   (define private-keywords
     '(#:source #:target #:go #:inputs #:native-inputs))
 
+  (define inputs-with-cache
+    ;; XXX: Avoid a circular dependency.  This should be rewritten with
+    ;; 'package-mapping' or similar.
+    (let ((go-std-name (string-append (package-name go) "-std")))
+      (if (string-prefix? go-std-name name)
+          inputs
+          (cons `(,go-std-name ,((make-go-std) go)) inputs))))
+
   (bag
     (name name)
     (system system)
@@ -127,7 +139,7 @@ (define* (lower name
                         '())
                      ,@`(("go" ,go))
                      ,@native-inputs
-                     ,@(if target '() inputs)
+                     ,@(if target '() inputs-with-cache)
                      ,@(if target
                          ;; Use the standard cross inputs of
                          ;; 'gnu-build-system'.
@@ -135,7 +147,7 @@ (define* (lower name
                          '())
                      ;; Keep the standard inputs of 'gnu-build-system'.
                      ,@(standard-packages)))
-    (host-inputs (if target inputs '()))
+    (host-inputs (if target inputs-with-cache '()))
 
     ;; The cross-libc is really a target package, but for bootstrapping
     ;; reasons, we can't put it in 'host-inputs'.  Namely, 'cross-gcc' is a
-- 
2.33.0

bug#50493: [PATCH 4/4] build-system/go: Use go-1.17 by default.

[Sarah Morgensen]

Build all Go packages with go-1.17 by default.  Explicitly specify an
older Go version in packages which cannot build with go-1.17.

* gnu/packages/golang.scm (go): Update from 'go-1.14' to 'go-1.17'.
* gnu/packages/docker.scm (docker)[native-inputs]: Explicitly use
'go-1.14'.
* gnu/packages/ipfs.scm (go-ipfs)[arguments]: Set #:go to 'go-1.16'.
(go-github-com-ipfs-go-ipfs-cmdkit-files)[arguments]: Likewise.
---
 gnu/packages/docker.scm | 2 +-
 gnu/packages/golang.scm | 2 +-
 gnu/packages/ipfs.scm   | 6 ++++--
 3 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/docker.scm b/gnu/packages/docker.scm
index 8bac1b89ce..fa666cbb04 100644
--- a/gnu/packages/docker.scm
+++ b/gnu/packages/docker.scm
@@ -594,7 +594,7 @@ (define-public docker
        ("xz" ,xz)))
     (native-inputs
      `(("eudev" ,eudev)      ; TODO: Should be propagated by lvm2 (.pc -> .pc)
-       ("go" ,go)
+       ("go" ,go-1.14)
        ("gotestsum" ,gotestsum)
        ("pkg-config" ,pkg-config)))
     (synopsis "Docker container component library, and daemon")
diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index 19f1ef37d5..d1ed30455c 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -790,7 +790,7 @@ (define-public go-1.17
                   "README.md" "SECURITY.md"))))))))
     (inputs (alist-delete "gcc:lib" (package-inputs go-1.16)))))
 
-(define-public go go-1.14)
+(define-public go go-1.17)
 
 (define-public (make-go-std go)
   "Return a package which builds the standard library for Go compiler GO."
diff --git a/gnu/packages/ipfs.scm b/gnu/packages/ipfs.scm
index 57701991e3..8d5b4f1767 100644
--- a/gnu/packages/ipfs.scm
+++ b/gnu/packages/ipfs.scm
@@ -46,7 +46,8 @@ (define-public go-github-com-ipfs-go-ipfs-cmdkit-files
               "0qk6fshgdmhp8dip2ksm13j6nywi41m9mn0czkvmw6b697z85l2r"))))
       (build-system go-build-system)
       (arguments
-       '(#:unpack-path "github.com/ipfs/go-ipfs-cmdkit"
+       `(#:go ,go-1.16
+         #:unpack-path "github.com/ipfs/go-ipfs-cmdkit"
          #:import-path "github.com/ipfs/go-ipfs-cmdkit/files"))
       (home-page "https://github.com/ipfs/go-ipfs-cmdkit")
       (synopsis "Shared types, functions and values for go-ipfs")
@@ -226,7 +227,8 @@ (define-public go-ipfs
        (file-name (string-append name "-" version "-source"))))
     (build-system go-build-system)
     (arguments
-     '(#:unpack-path "github.com/ipfs/go-ipfs"
+     `(#:go ,go-1.16
+       #:unpack-path "github.com/ipfs/go-ipfs"
        #:import-path "github.com/ipfs/go-ipfs/cmd/ipfs"
        #:phases (modify-phases %standard-phases
                   (add-before 'reset-gzip-timestamps 'make-files-writable
-- 
2.33.0

bug#50493: Update go-build-system to use Go 1.17.

[Sarah Morgensen]

Hello,

Apologies it's taken so long for a proper response!

>> 2. Make some much-needed improvements to go-build-system [2]:
>> 
>> * replace the *slow* phase 'remove-go-references with -trimpath option
>> 
>> * avoid building a directory union of Go inputs (and then copying it!)
>
> Are these required to use 1.17? I agree they are important problems to
> fix but we shouldn't let them block the upgrade.

You're right.  I've gone ahead and sent a patch series to this bug which
rolls the cache, docker, and the update all together.

I've added one more blocker: #50688 [0], which is a dependency of chezmoi.

This should just leave #49921, the aarch64 issue, then this is patch is
ready to go.  With the patch from #50688, I was to build out all
dependents listed by `guix refresh -l go' (about 900).

Thanks for sticking with this! :)

[0] https://issues.guix.gnu.org/50688

--
Sarah

bug#50493: Update go-build-system to use Go 1.17.

[Leo Famulari]

Alright, it's time to land these patches. I tested with a handful of
packages on x86_64 and it seems fine. I can test every package before
pushing.

However, Go 1.17.4 doesn't build for us on aarch64. I tried building the
derivation for this scheduled CI job:

https://ci.guix.gnu.org/build/1952348/details
/gnu/store/bldlnwxjwbi1iidm6cdw20kkpgy5gk09-go-1.17.4.drv

And it fails the test suite due to missing libgcc (I think), or maybe
something related to 'hello.exe', which seems weird:

------
$ guix build /gnu/store/bldlnwxjwbi1iidm6cdw20kkpgy5gk09-go-1.17.4.drv
[...]
--- FAIL: TestTrampolineCgo (78.58s)
    link_test.go:739: executable failed to run: exit status 127
        /tmp/guix-build-go-1.17.4.drv-0/TestTrampolineCgo602252092/001/hello.exe: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
    link_test.go:742: unexpected output:
        /tmp/guix-build-go-1.17.4.drv-0/TestTrampolineCgo602252092/001/hello.exe: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
FAIL
FAIL    cmd/link        425.553s
ok      cmd/link/internal/benchmark     0.449s
ok      cmd/link/internal/ld    84.727s
ok      cmd/link/internal/loader        0.366s
ok      cmd/nm  73.342s
ok      cmd/objdump     96.072s
ok      cmd/pack        77.197s
ok      cmd/pprof       155.165s
ok      cmd/trace       2.437s
ok      cmd/vet 318.126s
FAIL
go tool dist: Failed: exit status 1
command "sh" "run.bash" "--no-rebuild" failed with status 1
builder for `/gnu/store/bldlnwxjwbi1iidm6cdw20kkpgy5gk09-go-1.17.4.drv' failed with exit code 1
------

I notice that we remove libgcc from the dependencies of go-1.17:

https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/golang.scm?id=b484301f529e246df80380a641822f09a8775755#n799

Anyways, any ideas?

bug#50493: Update go-build-system to use Go 1.17.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2488 bytes --]

On Tue, Dec 14, 2021 at 03:11:04PM -0500, Leo Famulari wrote:
> Alright, it's time to land these patches. I tested with a handful of
> packages on x86_64 and it seems fine. I can test every package before
> pushing.
> 
> However, Go 1.17.4 doesn't build for us on aarch64. I tried building the
> derivation for this scheduled CI job:
> 
> https://ci.guix.gnu.org/build/1952348/details
> /gnu/store/bldlnwxjwbi1iidm6cdw20kkpgy5gk09-go-1.17.4.drv

The link shows problems with the armhf portion of the bootstrap.

> And it fails the test suite due to missing libgcc (I think), or maybe
> something related to 'hello.exe', which seems weird:
> 
> ------
> $ guix build /gnu/store/bldlnwxjwbi1iidm6cdw20kkpgy5gk09-go-1.17.4.drv
> [...]
> --- FAIL: TestTrampolineCgo (78.58s)
>     link_test.go:739: executable failed to run: exit status 127
>         /tmp/guix-build-go-1.17.4.drv-0/TestTrampolineCgo602252092/001/hello.exe: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
>     link_test.go:742: unexpected output:
>         /tmp/guix-build-go-1.17.4.drv-0/TestTrampolineCgo602252092/001/hello.exe: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory
> FAIL
> FAIL    cmd/link        425.553s
> ok      cmd/link/internal/benchmark     0.449s
> ok      cmd/link/internal/ld    84.727s
> ok      cmd/link/internal/loader        0.366s
> ok      cmd/nm  73.342s
> ok      cmd/objdump     96.072s
> ok      cmd/pack        77.197s
> ok      cmd/pprof       155.165s
> ok      cmd/trace       2.437s
> ok      cmd/vet 318.126s
> FAIL
> go tool dist: Failed: exit status 1
> command "sh" "run.bash" "--no-rebuild" failed with status 1
> builder for `/gnu/store/bldlnwxjwbi1iidm6cdw20kkpgy5gk09-go-1.17.4.drv' failed with exit code 1
> ------
> 
> I notice that we remove libgcc from the dependencies of go-1.17:
> 
> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/golang.scm?id=b484301f529e246df80380a641822f09a8775755#n799
> 
> Anyways, any ideas?

I'll try poking it a bit with my aarch64 board. We can also switch the
aarch64 go builds to bootstrap from gccgo, that might make a difference
too.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#50493: Update go-build-system to use Go 1.17.

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 395 bytes --]

On Wed, Dec 15, 2021 at 12:40:07PM +0200, Efraim Flashner wrote:
> I'll try poking it a bit with my aarch64 board. We can also switch the
> aarch64 go builds to bootstrap from gccgo, that might make a difference
> too.

Thanks, let us know how it goes.

I tried building go-1.17.4 for aarch64 with libgcc as an input, but it
failed in another way. So any progress you can make would be helpful.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#50493: Update go-build-system to use Go 1.17.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

On Wed, Dec 15, 2021 at 12:44:05PM -0500, Leo Famulari wrote:
> On Wed, Dec 15, 2021 at 12:40:07PM +0200, Efraim Flashner wrote:
> > I'll try poking it a bit with my aarch64 board. We can also switch the
> > aarch64 go builds to bootstrap from gccgo, that might make a difference
> > too.
> 
> Thanks, let us know how it goes.
> 
> I tried building go-1.17.4 for aarch64 with libgcc as an input, but it
> failed in another way. So any progress you can make would be helpful.

I got pretty far commenting out binutils-gold and adding back in gcc:lib
and the substitute snippets. Then I started failing tests requiring the
linker, so it looks like I'll actually have to troubleshoot
binutils-gold.

I thought about trying gccgo in place of go-1.4 for the bootstrap, but
I'm pretty sure I'd still need binutils-gold.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#50493: Update go-build-system to use Go 1.17.

[Leo Famulari]

On Wed, Dec 15, 2021 at 08:03:32PM +0200, Efraim Flashner wrote:
> I got pretty far commenting out binutils-gold and adding back in gcc:lib
> and the substitute snippets. Then I started failing tests requiring the
> linker, so it looks like I'll actually have to troubleshoot
> binutils-gold.
> 
> I thought about trying gccgo in place of go-1.4 for the bootstrap, but
> I'm pretty sure I'd still need binutils-gold.

Maybe, in the meantime, we could update go-build-system to use go-1.16,
which does at least build on aarch64. This Go release is still
supported, at least for a little while longer.

It would be nice to not be using Go 1.14 by default in the upcoming
release.

What do people think about that idea?

bug#50493: Update go-build-system to use Go 1.17.

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1336 bytes --]

On Fri, Dec 17, 2021 at 05:09:01PM -0500, Leo Famulari wrote:
> On Wed, Dec 15, 2021 at 08:03:32PM +0200, Efraim Flashner wrote:
> > I got pretty far commenting out binutils-gold and adding back in gcc:lib
> > and the substitute snippets. Then I started failing tests requiring the
> > linker, so it looks like I'll actually have to troubleshoot
> > binutils-gold.
> > 
> > I thought about trying gccgo in place of go-1.4 for the bootstrap, but
> > I'm pretty sure I'd still need binutils-gold.
> 
> Maybe, in the meantime, we could update go-build-system to use go-1.16,
> which does at least build on aarch64. This Go release is still
> supported, at least for a little while longer.
> 
> It would be nice to not be using Go 1.14 by default in the upcoming
> release.
> 
> What do people think about that idea?

All of go is blocked on aarch64 with binutils-gold not compiling. I have
a patch locally to add gcc:lib for (target-arm?) to go-1.17 which I'll
push after testing with binutils-gold. I think going straight to 1.17 is
fine, there's no regression against the current support for aarch64.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#50493: Update go-build-system to use Go 1.17.

[Leo Famulari]

[-- Attachment #1.1: Type: text/plain, Size: 520 bytes --]

On Sun, Dec 19, 2021 at 10:19:02AM +0200, Efraim Flashner wrote:
> All of go is blocked on aarch64 with binutils-gold not compiling. I have
> a patch locally to add gcc:lib for (target-arm?) to go-1.17 which I'll
> push after testing with binutils-gold. I think going straight to 1.17 is
> fine, there's no regression against the current support for aarch64.

Oh, good point.

Based on the attached patches (rebased / adjusted for current master),
I'll proceed with testing the build of all Go packages on x86_64-linux.

[-- Attachment #1.2: 0001-build-system-go-Add-substitutable-argument.patch --]
[-- Type: text/plain, Size: 2060 bytes --]

From f26d2b8b30867d23a417af2c40ba8062adb87bd7 Mon Sep 17 00:00:00 2001
From: Sarah Morgensen <iskarian@mgsn.dev>
Date: Mon, 13 Dec 2021 20:11:47 -0500
Subject: [PATCH 1/4] build-system/go: Add #:substitutable? argument.

* guix/build-system/go.scm (go-build): Add 'substitutable?' argument.
(go-cross-build): Likewise.

Signed-off-by: Leo Famulari <leo@famulari.name>
---
 guix/build-system/go.scm | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index 18824c79d9..0b2bf1c11b 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -172,7 +172,8 @@ (define* (go-build name inputs
                    (imported-modules %go-build-system-modules)
                    (modules '((guix build go-build-system)
                               (guix build union)
-                              (guix build utils))))
+                              (guix build utils)))
+                   (substitutable? #t))
   (define builder
     (with-imported-modules imported-modules
       #~(begin
@@ -182,6 +183,7 @@ (define builder
                     #:system #$system
                     #:phases #$phases
                     #:outputs #$(outputs->gexp outputs)
+                    #:substitutable? #$substitutable?
                     #:goarch #$goarch
                     #:goos #$goos
                     #:search-paths '#$(sexp->gexp
@@ -222,7 +224,8 @@ (define* (go-cross-build name
                          (imported-modules %go-build-system-modules)
                          (modules '((guix build go-build-system)
                                     (guix build union)
-                                    (guix build utils))))
+                                    (guix build utils)))
+                         (substitutable? #t))
   "Cross-build NAME using GO, where TARGET is a GNU triplet and with INPUTS."
   (define builder
     #~(begin

base-commit: 99f290bf5ba59e3218b95d7505ac27f989250aad
-- 
2.34.0


[-- Attachment #1.3: 0002-build-system-go-Initialize-build-cache-from-input-pa.patch --]
[-- Type: text/plain, Size: 2766 bytes --]

From a437f30bfaa4965cb42acb37f387e2c8106c7884 Mon Sep 17 00:00:00 2001
From: Sarah Morgensen <iskarian@mgsn.dev>
Date: Sun, 19 Sep 2021 22:20:49 -0700
Subject: [PATCH 2/4] build-system/go: Initialize build cache from input
 packages.

* guix/build/go-build-system.com (setup-go-environment): Set GOCACHE to
a location within the build directory.  Union "/var/cache/go/build"
input directories to initialize the cache.  Generate "trim.txt" within
the cache, with the current time.

Signed-off-by: Leo Famulari <leo@famulari.name>
---
 guix/build/go-build-system.scm | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/guix/build/go-build-system.scm b/guix/build/go-build-system.scm
index 4768ee8562..7f25e05d0d 100644
--- a/guix/build/go-build-system.scm
+++ b/guix/build/go-build-system.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2020 Jack Hill <jackhill@jackhill.us>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2020, 2021 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -138,9 +139,28 @@ (define* (setup-go-environment #:key inputs outputs goos goarch #:allow-other-ke
 where executables (\"commands\") are installed to.  This phase is sometimes used
 by packages that use (guix build-system gnu) but have a handful of Go
 dependencies, so it should be self-contained."
-  ;; The Go cache is required starting in Go 1.12.  We don't actually use it but
-  ;; we need it to be a writable directory.
-  (setenv "GOCACHE" "/tmp/go-cache")
+  (define (search-input-directories dir)
+    (filter directory-exists?
+            (map (match-lambda
+                   ((name . directory)
+                    (string-append directory "/" dir)))
+                 inputs)))
+
+  ;; Seed the Go build cache with the build caches from input packages.
+  (let ((cache (string-append (getcwd) "/go-build")))
+    (setenv "GOCACHE" cache)
+    (union-build cache
+                 (search-input-directories "/var/cache/go/build")
+                 ;; Creating all directories isn't that bad, because there are
+                 ;; only ever 256 of them.
+                 #:create-all-directories? #t
+                 #:log-port (%make-void-port "w"))
+
+    ;; Tell Go that the cache was recently trimmed, so it doesn't try to.
+    (call-with-output-file (string-append cache "/trim.txt")
+      (lambda (port)
+        (format port "~a" (current-time)))))
+
   ;; Using the current working directory as GOPATH makes it easier for packagers
   ;; who need to manipulate the unpacked source code.
   (setenv "GOPATH" (getcwd))
-- 
2.34.0


[-- Attachment #1.4: 0003-build-system-go-Add-pre-built-standard-library-as-im.patch --]
[-- Type: text/plain, Size: 4564 bytes --]

From cddf7d95c9134075aec45e2f2e49073f92aadfcb Mon Sep 17 00:00:00 2001
From: Sarah Morgensen <iskarian@mgsn.dev>
Date: Sun, 19 Sep 2021 22:20:50 -0700
Subject: [PATCH 3/4] build-system/go: Add pre-built standard library as
 implicit input.

* gnu/packages/golang.scm (make-go-std): New procedure.
* guix/build-system/go.scm (make-go-std): New procedure.
(lower): Use it.  Add pre-built standard library to inputs.

Signed-off-by: Leo Famulari <leo@famulari.name>
---
 gnu/packages/golang.scm  | 30 ++++++++++++++++++++++++++++++
 guix/build-system/go.scm | 16 ++++++++++++++--
 2 files changed, 44 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index c863388475..7989a3a138 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -806,6 +806,36 @@ (define-public go-1.17
 
 (define-public go go-1.14)
 
+(define-public (make-go-std go)
+  "Return a package which builds the standard library for Go compiler GO."
+  (package
+    (name (string-append (package-name go) "-std"))
+    (version (package-version go))
+    (source #f)
+    (build-system go-build-system)
+    (arguments
+     `(#:import-path "std"
+       #:build-flags `("-pkgdir" "pkg") ; "Install" to build directory.
+       #:allow-go-reference? #t
+       #:substitutable? #f ; Faster to build than download.
+       #:tests? #f ; Already tested in the main Go build.
+       #:go ,go
+       #:phases
+       (modify-phases %standard-phases
+         (delete 'unpack)
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (out-cache (string-append out "/var/cache/go/build")))
+               (copy-recursively (getenv "GOCACHE") out-cache)
+               (delete-file (string-append out-cache "/trim.txt"))
+               (delete-file (string-append out-cache "/README")))))
+         (delete 'install-license-files))))
+    (home-page (package-home-page go))
+    (synopsis "Cached standard library build for Go")
+    (description (package-description go))
+    (license (package-license go))))
+
 (define-public go-0xacab-org-leap-shapeshifter
   (let ((commit "0aa6226582efb8e563540ec1d3c5cfcd19200474")
         (revision "12"))
diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index 0b2bf1c11b..09148f8730 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -4,6 +4,7 @@
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2021 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2021 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -112,6 +113,9 @@ (define (default-go)
   (let ((go (resolve-interface '(gnu packages golang))))
     (module-ref go 'go)))
 
+(define (make-go-std)
+  (module-ref (resolve-interface '(gnu packages golang)) 'make-go-std))
+
 (define* (lower name
                 #:key source inputs native-inputs outputs system target
                 (go (default-go))
@@ -121,6 +125,14 @@ (define* (lower name
   (define private-keywords
     '(#:target #:go #:inputs #:native-inputs))
 
+  (define inputs-with-cache
+    ;; XXX: Avoid a circular dependency.  This should be rewritten with
+    ;; 'package-mapping' or similar.
+    (let ((go-std-name (string-append (package-name go) "-std")))
+      (if (string-prefix? go-std-name name)
+          inputs
+          (cons `(,go-std-name ,((make-go-std) go)) inputs))))
+
   (bag
     (name name)
     (system system)
@@ -130,7 +142,7 @@ (define private-keywords
                         '())
                      ,@`(("go" ,go))
                      ,@native-inputs
-                     ,@(if target '() inputs)
+                     ,@(if target '() inputs-with-cache)
                      ,@(if target
                          ;; Use the standard cross inputs of
                          ;; 'gnu-build-system'.
@@ -138,7 +150,7 @@ (define private-keywords
                          '())
                      ;; Keep the standard inputs of 'gnu-build-system'.
                      ,@(standard-packages)))
-    (host-inputs (if target inputs '()))
+    (host-inputs (if target inputs-with-cache '()))
 
     ;; The cross-libc is really a target package, but for bootstrapping
     ;; reasons, we can't put it in 'host-inputs'.  Namely, 'cross-gcc' is a
-- 
2.34.0


[-- Attachment #1.5: 0004-build-system-go-Use-go-1.17-by-default.patch --]
[-- Type: text/plain, Size: 2655 bytes --]

From 6f9d52c1117279be567e2ee9f56f88e470867e92 Mon Sep 17 00:00:00 2001
From: Sarah Morgensen <iskarian@mgsn.dev>
Date: Sun, 19 Sep 2021 22:20:51 -0700
Subject: [PATCH 4/4] build-system/go: Use go-1.17 by default.

Build all Go packages with go-1.17 by default.  Explicitly specify an
older Go version in packages which cannot build with go-1.17.

* gnu/packages/golang.scm (go): Update from 'go-1.14' to 'go-1.17'.
* gnu/packages/docker.scm (docker)[native-inputs]: Explicitly use 'go-1.14'.
* gnu/packages/ipfs.scm (go-github-com-ipfs-go-ipfs-cmdkit-files)[arguments]:
Set #:go to 'go-1.16'.

Signed-off-by: Leo Famulari <leo@famulari.name>
---
 gnu/packages/docker.scm | 2 +-
 gnu/packages/golang.scm | 2 +-
 gnu/packages/ipfs.scm   | 3 ++-
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/docker.scm b/gnu/packages/docker.scm
index cdf62f5e9a..d012ed43c6 100644
--- a/gnu/packages/docker.scm
+++ b/gnu/packages/docker.scm
@@ -585,7 +585,7 @@ (define-public docker
        ("xz" ,xz)))
     (native-inputs
      (list eudev ; TODO: Should be propagated by lvm2 (.pc -> .pc)
-           go gotestsum pkg-config))
+           go-1.14 gotestsum pkg-config))
     (synopsis "Docker container component library, and daemon")
     (description "This package provides a framework to assemble specialized
 container systems.  It includes components for orchestration, image
diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index 7989a3a138..1ac6c81900 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -804,7 +804,7 @@ (define-public go-1.17
            (alist-replace "go" (list go-1.16) (package-native-inputs go-1.16))
            (package-native-inputs go-1.16))))))
 
-(define-public go go-1.14)
+(define-public go go-1.17)
 
 (define-public (make-go-std go)
   "Return a package which builds the standard library for Go compiler GO."
diff --git a/gnu/packages/ipfs.scm b/gnu/packages/ipfs.scm
index f566b850aa..b567b23353 100644
--- a/gnu/packages/ipfs.scm
+++ b/gnu/packages/ipfs.scm
@@ -48,7 +48,8 @@ (define-public go-github-com-ipfs-go-ipfs-cmdkit-files
               "0qk6fshgdmhp8dip2ksm13j6nywi41m9mn0czkvmw6b697z85l2r"))))
       (build-system go-build-system)
       (arguments
-       '(#:unpack-path "github.com/ipfs/go-ipfs-cmdkit"
+       `(#:go ,go-1.16
+         #:unpack-path "github.com/ipfs/go-ipfs-cmdkit"
          #:import-path "github.com/ipfs/go-ipfs-cmdkit/files"))
       (home-page "https://github.com/ipfs/go-ipfs-cmdkit")
       (synopsis "Shared types, functions and values for go-ipfs")
-- 
2.34.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#50493: Update go-build-system to use Go 1.17.

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 515 bytes --]

On Sun, Dec 19, 2021 at 08:30:18PM -0500, Leo Famulari wrote:
> Based on the attached patches (rebased / adjusted for current master),
> I'll proceed with testing the build of all Go packages on x86_64-linux.

Well, Go builds are very fast.

There are no new failures of packages using go-build-system with these
patches, assuming I accurately collected all of them.

So, I invite people to apply these patches and try *using* their
favorite Go programs, and hopefully we can push these patches in the
coming days.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#50493: Update go-build-system to use Go 1.17.

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 267 bytes --]

On Sun, Dec 19, 2021 at 09:31:46PM -0500, Leo Famulari wrote:
> So, I invite people to apply these patches and try *using* their
> favorite Go programs, and hopefully we can push these patches in the
> coming days.

Pushed as 717be95339c953f53a22b9d98378c76025263737

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]