unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / Atom feed
* bug#50493: Update go-build-system to use Go 1.17.
@ 2021-09-09 22:14 Sarah Morgensen
  2021-09-09 22:37 ` Leo Famulari
  2021-09-20  5:19 ` bug#50493: [PATCH staging 0/4] build-system/go: Use go-1.17 by default Sarah Morgensen
  0 siblings, 2 replies; 8+ messages in thread
From: Sarah Morgensen @ 2021-09-09 22:14 UTC (permalink / raw)
  To: 50493

Hello Guix,

This is intended to be a tracking issue for updating go-build-system to
use Go 1.17 by default.  The build system currently uses Go 1.14, which
no longer receives security updates [0].

Here's what needs to be done:

1. Address the aarch64 bug [1], either by verifying that it only happens
on emulated systems, or by fixing it if not.

2. Make some much-needed improvements to go-build-system [2]:

* replace the *slow* phase 'remove-go-references with -trimpath option

* avoid building a directory union of Go inputs (and then copying it!)

* add a "go-std-cache-for-build" package (hidden; name to be
bikeshedded), which will be an implicit input in go-build-system (needed
since we removed the pre-built standard library in go-1.17)

3. Some/all Docker packages still require Go 1.14; make them use go-1.14
explicitly.  Alternatively, update Docker (significant effort involved).

4. Update or patch broken packages.  I've got patches for four so far:

go-github-com-urfave-cli
go-github-com-urfave-cli-v2
go-github-com-sevlyar-go-daemon
go-github-com-keybase-go-ps

I'll submit them as a separate patch and block this with it.  If anyone
else would like to see if their favorite package works with Go 1.17, you
can build it with:

  go build --with-input=go@1.14=go@1.17 <package>

[0] https://github.com/golang/go/wiki/Go-Release-Cycle#release-maintenance
[1] https://issues.guix.gnu.org/49921
[2] https://issues.guix.gnu.org/50227#10

--
Sarah




^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#50493: Update go-build-system to use Go 1.17.
  2021-09-09 22:14 bug#50493: Update go-build-system to use Go 1.17 Sarah Morgensen
@ 2021-09-09 22:37 ` Leo Famulari
  2021-09-20  5:28   ` Sarah Morgensen
  2021-09-20  5:19 ` bug#50493: [PATCH staging 0/4] build-system/go: Use go-1.17 by default Sarah Morgensen
  1 sibling, 1 reply; 8+ messages in thread
From: Leo Famulari @ 2021-09-09 22:37 UTC (permalink / raw)
  To: Sarah Morgensen; +Cc: 50493

On Thu, Sep 09, 2021 at 03:14:57PM -0700, Sarah Morgensen wrote:
> This is intended to be a tracking issue for updating go-build-system to
> use Go 1.17 by default.  The build system currently uses Go 1.14, which
> no longer receives security updates [0].

Thanks for putting this together.

> Here's what needs to be done:
> 
> 1. Address the aarch64 bug [1], either by verifying that it only happens
> on emulated systems, or by fixing it if not.

It looks like there was a patch for this issue:

https://github.com/golang/go/issues/20763#issuecomment-314627882

And it was apparently added to Go, but maybe it was not sufficient, did
not make it into a release, or we are seeing a different bug.

> 2. Make some much-needed improvements to go-build-system [2]:
> 
> * replace the *slow* phase 'remove-go-references with -trimpath option
> 
> * avoid building a directory union of Go inputs (and then copying it!)

Are these required to use 1.17? I agree they are important problems to
fix but we shouldn't let them block the upgrade.

> * add a "go-std-cache-for-build" package (hidden; name to be
> bikeshedded), which will be an implicit input in go-build-system (needed
> since we removed the pre-built standard library in go-1.17)

Okay. Feel free to choose the name. I think it's your shed.


> 3. Some/all Docker packages still require Go 1.14; make them use go-1.14
> explicitly.  Alternatively, update Docker (significant effort involved).

Okay.

> 4. Update or patch broken packages.  I've got patches for four so far:
> 
> go-github-com-urfave-cli
> go-github-com-urfave-cli-v2
> go-github-com-sevlyar-go-daemon
> go-github-com-keybase-go-ps
>
> I'll submit them as a separate patch and block this with it.  If anyone
> else would like to see if their favorite package works with Go 1.17, you
> can build it with:

Okay, great.
  
>   go build --with-input=go@1.14=go@1.17 <package>

Syncthing builds! That's great.




^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#50493: [PATCH staging 0/4] build-system/go: Use go-1.17 by default.
  2021-09-09 22:14 bug#50493: Update go-build-system to use Go 1.17 Sarah Morgensen
  2021-09-09 22:37 ` Leo Famulari
@ 2021-09-20  5:19 ` Sarah Morgensen
  2021-09-20  5:20   ` bug#50493: [PATCH 1/4] build-system/go: Add #:substitutable? argument Sarah Morgensen
                     ` (3 more replies)
  1 sibling, 4 replies; 8+ messages in thread
From: Sarah Morgensen @ 2021-09-20  5:19 UTC (permalink / raw)
  To: 50493

Hello Guix,

This patch series updates the Go build system to use Go 1.17 by default.

As a necessity, because go-1.17 no longer contains its own pre-compiled
standard library, an implicit input is added to the build system which
contains the build cache from building the standard library with the
go-build-system.  (This way, we can always guarantee that the cache is valid
because it's built with the same flags as the build system.)  The build system
is also modified to make use of this cache.

(We use the build cache rather than archives because the Go project is moving
away from archives towards an authoritative build cache.)

Three packages are modified to keep using an older Go:

* docker now explicitly uses go-1.14

* go-ipfs and go-github-com-go-ipfs-cmdkit-files now explitly use go-1.16
(go-ipfs and gx badly need to be upgraded!)

There are about 900 dependent packages according to `guix refresh -l go', but
I don't know how many are leaf packages, so I marked this for staging.  The
actual series is based on master because go-1.17 is only present in master.

I have built them all out successfully on x86_64.

Sarah Morgensen (4):
  build-system/go: Add #:substitutable? argument.
  build-system/go: Initialize build cache from input packages.
  build-system/go: Add pre-built standard library as implicit input.
  build-system/go: Use go-1.17 by default.

 gnu/packages/docker.scm        |  2 +-
 gnu/packages/golang.scm        | 32 +++++++++++++++++++++++++++++++-
 gnu/packages/ipfs.scm          |  6 ++++--
 guix/build-system/go.scm       | 24 ++++++++++++++++++++----
 guix/build/go-build-system.scm | 26 +++++++++++++++++++++++---
 5 files changed, 79 insertions(+), 11 deletions(-)


base-commit: a640ee7da3311e4fa560c7163d2dd7d24a25585a
-- 
2.33.0





^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#50493: [PATCH 1/4] build-system/go: Add #:substitutable? argument.
  2021-09-20  5:19 ` bug#50493: [PATCH staging 0/4] build-system/go: Use go-1.17 by default Sarah Morgensen
@ 2021-09-20  5:20   ` Sarah Morgensen
  2021-09-20  5:20   ` bug#50493: [PATCH 2/4] build-system/go: Initialize build cache from input packages Sarah Morgensen
                     ` (2 subsequent siblings)
  3 siblings, 0 replies; 8+ messages in thread
From: Sarah Morgensen @ 2021-09-20  5:20 UTC (permalink / raw)
  To: 50493

* guix/build-system/go.scm (go-build): Add 'substitutable?' argument.
(go-cross-build): Likewise.
---
 guix/build-system/go.scm | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index 4c1a732107..8aa065c4ff 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -169,7 +169,8 @@ (define* (go-build store name inputs
                    (imported-modules %go-build-system-modules)
                    (modules '((guix build go-build-system)
                               (guix build union)
-                              (guix build utils))))
+                              (guix build utils)))
+                   (substitutable? #t))
   (define builder
    `(begin
       (use-modules ,@modules)
@@ -211,6 +212,7 @@ (define* (go-build store name inputs
                                 #:system system
                                 #:modules imported-modules
                                 #:outputs outputs
+                                #:substitutable? substitutable?
                                 #:guile-for-build guile-for-build))
 
 (define* (go-cross-build store name
@@ -234,7 +236,8 @@ (define* (go-cross-build store name
                          (imported-modules %go-build-system-modules)
                          (modules '((guix build go-build-system)
                                     (guix build union)
-                                    (guix build utils))))
+                                    (guix build utils)))
+                         (substitutable? #t))
   "Cross-build NAME using GO, where TARGET is a GNU triplet and with INPUTS."
   (define builder
     `(begin
@@ -303,6 +306,7 @@ (define* (go-cross-build store name
                                   #:inputs (append native-drvs target-drvs)
                                   #:outputs outputs
                                   #:modules imported-modules
+                                  #:substitutable? substitutable?
                                   #:guile-for-build guile-for-build))
 
 (define go-build-system
-- 
2.33.0





^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#50493: [PATCH 2/4] build-system/go: Initialize build cache from input packages.
  2021-09-20  5:19 ` bug#50493: [PATCH staging 0/4] build-system/go: Use go-1.17 by default Sarah Morgensen
  2021-09-20  5:20   ` bug#50493: [PATCH 1/4] build-system/go: Add #:substitutable? argument Sarah Morgensen
@ 2021-09-20  5:20   ` Sarah Morgensen
  2021-09-20  5:20   ` bug#50493: [PATCH 3/4] build-system/go: Add pre-built standard library as implicit input Sarah Morgensen
  2021-09-20  5:20   ` bug#50493: [PATCH 4/4] build-system/go: Use go-1.17 by default Sarah Morgensen
  3 siblings, 0 replies; 8+ messages in thread
From: Sarah Morgensen @ 2021-09-20  5:20 UTC (permalink / raw)
  To: 50493

* guix/build/go-build-system.com (setup-go-environment): Set GOCACHE to
a location within the build directory.  Union "/var/cache/go/build"
input directories to initialize the cache.  Generate "trim.txt" within
the cache, with the current time.
---
 guix/build/go-build-system.scm | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/guix/build/go-build-system.scm b/guix/build/go-build-system.scm
index 645d2fe680..9f249b0777 100644
--- a/guix/build/go-build-system.scm
+++ b/guix/build/go-build-system.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2020 Jack Hill <jackhill@jackhill.us>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2020, 2021 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -138,9 +139,28 @@ (define* (setup-go-environment #:key inputs outputs goos goarch #:allow-other-ke
 where executables (\"commands\") are installed to.  This phase is sometimes used
 by packages that use (guix build-system gnu) but have a handful of Go
 dependencies, so it should be self-contained."
-  ;; The Go cache is required starting in Go 1.12.  We don't actually use it but
-  ;; we need it to be a writable directory.
-  (setenv "GOCACHE" "/tmp/go-cache")
+  (define (search-input-directories dir)
+    (filter directory-exists?
+            (map (match-lambda
+                   ((name . directory)
+                    (string-append directory "/" dir)))
+                 inputs)))
+
+  ;; Seed the Go build cache with the build caches from input packages.
+  (let ((cache (string-append (getcwd) "/go-build")))
+    (setenv "GOCACHE" cache)
+    (union-build cache
+                 (search-input-directories "/var/cache/go/build")
+                 ;; Creating all directories isn't that bad, because there are
+                 ;; only ever 256 of them.
+                 #:create-all-directories? #t
+                 #:log-port (%make-void-port "w"))
+
+    ;; Tell Go that the cache was recently trimmed, so it doesn't try to.
+    (call-with-output-file (string-append cache "/trim.txt")
+      (lambda (port)
+        (format port "~a" (current-time)))))
+
   ;; Using the current working directory as GOPATH makes it easier for packagers
   ;; who need to manipulate the unpacked source code.
   (setenv "GOPATH" (getcwd))
-- 
2.33.0





^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#50493: [PATCH 3/4] build-system/go: Add pre-built standard library as implicit input.
  2021-09-20  5:19 ` bug#50493: [PATCH staging 0/4] build-system/go: Use go-1.17 by default Sarah Morgensen
  2021-09-20  5:20   ` bug#50493: [PATCH 1/4] build-system/go: Add #:substitutable? argument Sarah Morgensen
  2021-09-20  5:20   ` bug#50493: [PATCH 2/4] build-system/go: Initialize build cache from input packages Sarah Morgensen
@ 2021-09-20  5:20   ` Sarah Morgensen
  2021-09-20  5:20   ` bug#50493: [PATCH 4/4] build-system/go: Use go-1.17 by default Sarah Morgensen
  3 siblings, 0 replies; 8+ messages in thread
From: Sarah Morgensen @ 2021-09-20  5:20 UTC (permalink / raw)
  To: 50493

* gnu/packages/golang.scm (make-go-std): New procedure.
* guix/build-system/go.scm (make-go-std): New procedure.
(lower): Use it.  Add pre-built standard library to inputs.
---
 gnu/packages/golang.scm  | 30 ++++++++++++++++++++++++++++++
 guix/build-system/go.scm | 16 ++++++++++++++--
 2 files changed, 44 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index f02d0aa9df..19f1ef37d5 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -792,6 +792,36 @@ (define-public go-1.17
 
 (define-public go go-1.14)
 
+(define-public (make-go-std go)
+  "Return a package which builds the standard library for Go compiler GO."
+  (package
+    (name (string-append (package-name go) "-std"))
+    (version (package-version go))
+    (source #f)
+    (build-system go-build-system)
+    (arguments
+     `(#:import-path "std"
+       #:build-flags `("-pkgdir" "pkg") ; "Install" to build directory.
+       #:allow-go-reference? #t
+       #:substitutable? #f ; Faster to build than download.
+       #:tests? #f ; Already tested in the main Go build.
+       #:go ,go
+       #:phases
+       (modify-phases %standard-phases
+         (delete 'unpack)
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (out-cache (string-append out "/var/cache/go/build")))
+               (copy-recursively (getenv "GOCACHE") out-cache)
+               (delete-file (string-append out-cache "/trim.txt"))
+               (delete-file (string-append out-cache "/README")))))
+         (delete 'install-license-files))))
+    (home-page (package-home-page go))
+    (synopsis "Cached standard library build for Go")
+    (description (package-description go))
+    (license (package-license go))))
+
 (define-public go-0xacab-org-leap-shapeshifter
   (let ((commit "0aa6226582efb8e563540ec1d3c5cfcd19200474")
         (revision "12"))
diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index 8aa065c4ff..a579477500 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2017 Leo Famulari <leo@famulari.name>
 ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net>
 ;;; Copyright © 2021 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -109,6 +110,9 @@ (define (default-go)
   (let ((go (resolve-interface '(gnu packages golang))))
     (module-ref go 'go)))
 
+(define (make-go-std)
+  (module-ref (resolve-interface '(gnu packages golang)) 'make-go-std))
+
 (define* (lower name
                 #:key source inputs native-inputs outputs system target
                 (go (default-go))
@@ -118,6 +122,14 @@ (define* (lower name
   (define private-keywords
     '(#:source #:target #:go #:inputs #:native-inputs))
 
+  (define inputs-with-cache
+    ;; XXX: Avoid a circular dependency.  This should be rewritten with
+    ;; 'package-mapping' or similar.
+    (let ((go-std-name (string-append (package-name go) "-std")))
+      (if (string-prefix? go-std-name name)
+          inputs
+          (cons `(,go-std-name ,((make-go-std) go)) inputs))))
+
   (bag
     (name name)
     (system system)
@@ -127,7 +139,7 @@ (define* (lower name
                         '())
                      ,@`(("go" ,go))
                      ,@native-inputs
-                     ,@(if target '() inputs)
+                     ,@(if target '() inputs-with-cache)
                      ,@(if target
                          ;; Use the standard cross inputs of
                          ;; 'gnu-build-system'.
@@ -135,7 +147,7 @@ (define* (lower name
                          '())
                      ;; Keep the standard inputs of 'gnu-build-system'.
                      ,@(standard-packages)))
-    (host-inputs (if target inputs '()))
+    (host-inputs (if target inputs-with-cache '()))
 
     ;; The cross-libc is really a target package, but for bootstrapping
     ;; reasons, we can't put it in 'host-inputs'.  Namely, 'cross-gcc' is a
-- 
2.33.0





^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#50493: [PATCH 4/4] build-system/go: Use go-1.17 by default.
  2021-09-20  5:19 ` bug#50493: [PATCH staging 0/4] build-system/go: Use go-1.17 by default Sarah Morgensen
                     ` (2 preceding siblings ...)
  2021-09-20  5:20   ` bug#50493: [PATCH 3/4] build-system/go: Add pre-built standard library as implicit input Sarah Morgensen
@ 2021-09-20  5:20   ` Sarah Morgensen
  3 siblings, 0 replies; 8+ messages in thread
From: Sarah Morgensen @ 2021-09-20  5:20 UTC (permalink / raw)
  To: 50493

Build all Go packages with go-1.17 by default.  Explicitly specify an
older Go version in packages which cannot build with go-1.17.

* gnu/packages/golang.scm (go): Update from 'go-1.14' to 'go-1.17'.
* gnu/packages/docker.scm (docker)[native-inputs]: Explicitly use
'go-1.14'.
* gnu/packages/ipfs.scm (go-ipfs)[arguments]: Set #:go to 'go-1.16'.
(go-github-com-ipfs-go-ipfs-cmdkit-files)[arguments]: Likewise.
---
 gnu/packages/docker.scm | 2 +-
 gnu/packages/golang.scm | 2 +-
 gnu/packages/ipfs.scm   | 6 ++++--
 3 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/docker.scm b/gnu/packages/docker.scm
index 8bac1b89ce..fa666cbb04 100644
--- a/gnu/packages/docker.scm
+++ b/gnu/packages/docker.scm
@@ -594,7 +594,7 @@ (define-public docker
        ("xz" ,xz)))
     (native-inputs
      `(("eudev" ,eudev)      ; TODO: Should be propagated by lvm2 (.pc -> .pc)
-       ("go" ,go)
+       ("go" ,go-1.14)
        ("gotestsum" ,gotestsum)
        ("pkg-config" ,pkg-config)))
     (synopsis "Docker container component library, and daemon")
diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index 19f1ef37d5..d1ed30455c 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -790,7 +790,7 @@ (define-public go-1.17
                   "README.md" "SECURITY.md"))))))))
     (inputs (alist-delete "gcc:lib" (package-inputs go-1.16)))))
 
-(define-public go go-1.14)
+(define-public go go-1.17)
 
 (define-public (make-go-std go)
   "Return a package which builds the standard library for Go compiler GO."
diff --git a/gnu/packages/ipfs.scm b/gnu/packages/ipfs.scm
index 57701991e3..8d5b4f1767 100644
--- a/gnu/packages/ipfs.scm
+++ b/gnu/packages/ipfs.scm
@@ -46,7 +46,8 @@ (define-public go-github-com-ipfs-go-ipfs-cmdkit-files
               "0qk6fshgdmhp8dip2ksm13j6nywi41m9mn0czkvmw6b697z85l2r"))))
       (build-system go-build-system)
       (arguments
-       '(#:unpack-path "github.com/ipfs/go-ipfs-cmdkit"
+       `(#:go ,go-1.16
+         #:unpack-path "github.com/ipfs/go-ipfs-cmdkit"
          #:import-path "github.com/ipfs/go-ipfs-cmdkit/files"))
       (home-page "https://github.com/ipfs/go-ipfs-cmdkit")
       (synopsis "Shared types, functions and values for go-ipfs")
@@ -226,7 +227,8 @@ (define-public go-ipfs
        (file-name (string-append name "-" version "-source"))))
     (build-system go-build-system)
     (arguments
-     '(#:unpack-path "github.com/ipfs/go-ipfs"
+     `(#:go ,go-1.16
+       #:unpack-path "github.com/ipfs/go-ipfs"
        #:import-path "github.com/ipfs/go-ipfs/cmd/ipfs"
        #:phases (modify-phases %standard-phases
                   (add-before 'reset-gzip-timestamps 'make-files-writable
-- 
2.33.0





^ permalink raw reply	[flat|nested] 8+ messages in thread

* bug#50493: Update go-build-system to use Go 1.17.
  2021-09-09 22:37 ` Leo Famulari
@ 2021-09-20  5:28   ` Sarah Morgensen
  0 siblings, 0 replies; 8+ messages in thread
From: Sarah Morgensen @ 2021-09-20  5:28 UTC (permalink / raw)
  To: Leo Famulari; +Cc: 50493

Hello,

Apologies it's taken so long for a proper response!

>> 2. Make some much-needed improvements to go-build-system [2]:
>> 
>> * replace the *slow* phase 'remove-go-references with -trimpath option
>> 
>> * avoid building a directory union of Go inputs (and then copying it!)
>
> Are these required to use 1.17? I agree they are important problems to
> fix but we shouldn't let them block the upgrade.

You're right.  I've gone ahead and sent a patch series to this bug which
rolls the cache, docker, and the update all together.

I've added one more blocker: #50688 [0], which is a dependency of chezmoi.

This should just leave #49921, the aarch64 issue, then this is patch is
ready to go.  With the patch from #50688, I was to build out all
dependents listed by `guix refresh -l go' (about 900).

Thanks for sticking with this! :)

[0] https://issues.guix.gnu.org/50688

--
Sarah




^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2021-09-20  6:07 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-09 22:14 bug#50493: Update go-build-system to use Go 1.17 Sarah Morgensen
2021-09-09 22:37 ` Leo Famulari
2021-09-20  5:28   ` Sarah Morgensen
2021-09-20  5:19 ` bug#50493: [PATCH staging 0/4] build-system/go: Use go-1.17 by default Sarah Morgensen
2021-09-20  5:20   ` bug#50493: [PATCH 1/4] build-system/go: Add #:substitutable? argument Sarah Morgensen
2021-09-20  5:20   ` bug#50493: [PATCH 2/4] build-system/go: Initialize build cache from input packages Sarah Morgensen
2021-09-20  5:20   ` bug#50493: [PATCH 3/4] build-system/go: Add pre-built standard library as implicit input Sarah Morgensen
2021-09-20  5:20   ` bug#50493: [PATCH 4/4] build-system/go: Use go-1.17 by default Sarah Morgensen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).