unofficial mirror of guix-patches@gnu.org 
 help / color / mirror / code / Atom feed
* [bug#50348] [PATCH] gnu: Add go-1.17.
@ 2021-09-02 22:09 Sarah Morgensen
  2021-09-09 19:45 ` [bug#50348] [PATCH v2] " Sarah Morgensen
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Sarah Morgensen @ 2021-09-02 22:09 UTC (permalink / raw)
  To: 50348

* gnu/packages.golang.scm (go-1.17): New variable.
---
Hello Guix,

This patch (finally) adds Go 1.17, with some further improvements!

Highlights:

* 43% closure size compared to go-1.16 (~50% output size)

* No longer dependent on single GCC version (fixes #36823, #39400)[0][1]

* Many tests re-enabled

* 'doc' output merged (makes docs usable with `godoc')

* Installs in FHS paths rather than directly in $out/

* Builds nearly all our Go packages

* Phases refactored

For the few Guix Go packages which require an update or patch to build with Go
1.17, I will send those later.  But I don't think those should be required to
merge this, since go-1.17 won't be used to build Go packages by default.

What do you think? (Details below the break, for those interested.)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This update removes Go's dependence on canonical-gcc by replacing the patch
that hardcodes the gcc-lib runpath in Go's own linker with one that tells Go
to use the host linker (which knows how to add runpaths) when linking against
libraries (glibc, gcc-lib, etc).  This removes GCC from Go's closure (fixing
[0]), and allows users to build with any GCC library (fixing [1]).

In the future, we may be able to drop this patch and instead use
GO_EXTLINK_ENABLED=1 or similar when [2] or [3] is resolved.

Removing the GCC dependency reduces closure size by ~107MiB.  Properly
stripping the binaries shaves off ~20MiB.  No longer installing pre-compiled
standard library archives shaves off another ~180MiB.

Quoting my explanatory comment:

> Notably, we do not install archives (180M), which Go will happily recompile
> quickly (and cache) if needed, almost surely faster than they could be
> substituted.

> The main motivation for pre-compiled archives is to use libc-linked `net' or
> `os' packages without a C compiler, but on Guix a C compiler is necessary to
> properly link the final binaries anyway.  Many build flags also invalidate
> these pre-compiled archives, so in practice Go often recompiles them anyway.

> Upstream is also planning to no longer install these archives:
> <https://github.com/golang/go/issues/47257>

> When necessary, a custom pre-compiled library package can be created
> with `#:import-path "std"' and used with `-pkgdir'.

That latter step should only be necessary when updating the build-system to
use this Go.

Finally, rather than installing directly in $out, Go now installs to /bin,
/lib/go, /share/go, and /share/doc/go (albeit with symlinks to them from
/lib/go).  Also, the 'doc' output, with less than 1MB, was merged into 'out',
which makes `godoc' work correctly for the standard library.

[0] <https://issues.guix.gnu.org/36823> Go retains a reference to GCC.
[1] https://issues.guix.gnu.org/39400
[2] <https://github.com/golang/go/issues/31544> cmd/go: spurious error message
when external linking a pure Go program
[3] <https://github.com/golang/go/issues/43525> cmd/link: provide a way to
always use external linking if cgo is used

--
Sarah

 gnu/packages/golang.scm | 182 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 182 insertions(+)

diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index b6e8b84749..8da7c9bf85 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -608,6 +608,188 @@ (define-public go-1.16
              (alist-replace "go" (list gccgo-10) (package-native-inputs go-1.14))
              (package-native-inputs go-1.14))))))
 
+(define-public go-1.17
+  (package
+    (inherit go-1.16)
+    (name "go")
+    (version "1.17")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/golang/go")
+             (commit (string-append "go" version))))
+       (file-name (git-file-name name version))
+       (sha256
+        (base32
+         "1psra6j95ws38mx3scc1whky8cwk32mazqs0wzzdwbs8ppl8iwl5"))))
+    (outputs '("out" "tests")) ; 'tests' contains distribution tests.
+    (arguments
+     `(#:modules ((ice-9 match)
+                  (guix build gnu-build-system)
+                  (guix build utils))
+       #:phases
+       (modify-phases %standard-phases
+         (replace 'configure
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let ((output (assoc-ref outputs "out"))
+                   (loader (string-append (assoc-ref inputs "libc")
+                                          ,(glibc-dynamic-linker))))
+               (setenv "GOOS" "linux")
+               (setenv "GO_LDSO" loader)
+               (setenv "GOROOT" (getcwd))
+               (setenv "GOROOT_FINAL" (string-append output "/lib/go"))
+               (setenv "GOGC" "400")
+               (setenv "GOCACHE" "/tmp/go-cache"))))
+         (add-after 'unpack 'patch-source
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let* ((net-base (assoc-ref inputs "net-base"))
+                    (tzdata-path (string-append (assoc-ref inputs "tzdata")
+                                                "/share/zoneinfo")))
+               ;; XXX: Remove when #49729 is merged?
+               (for-each make-file-writable (find-files "src"))
+
+               ;; Having the patch in the 'patches' field of <origin> breaks
+               ;; the 'TestServeContent' test due to the fact that
+               ;; timestamps are reset.  Thus, apply it from here.
+               (invoke "patch" "-p1" "--force" "-i"
+                       (assoc-ref inputs "go-skip-gc-test.patch"))
+               (invoke "patch" "-p1" "--force" "-i"
+                       (assoc-ref inputs "go-fix-script-tests.patch"))
+
+               (substitute* "src/os/os_test.go"
+                 (("/usr/bin") (getcwd))
+                 (("/bin/sh") (which "sh")))
+
+               (substitute* "src/cmd/go/testdata/script/cgo_path_space.txt"
+                 (("/bin/sh") (which "sh")))
+
+               ;; fix shebang for testar script
+               ;; note the target script is generated at build time.
+               (substitute* "misc/cgo/testcarchive/carchive_test.go"
+                 (("/usr/bin/env bash") (which "bash")))
+
+               (substitute* "src/net/lookup_unix.go"
+                 (("/etc/protocols")
+                  (string-append net-base "/etc/protocols")))
+               (substitute* "src/net/port_unix.go"
+                 (("/etc/services")
+                  (string-append net-base "/etc/services")))
+               (substitute* "src/time/zoneinfo_unix.go"
+                 (("/usr/share/zoneinfo/") tzdata-path)))))
+         (add-after 'patch-source 'disable-failing-tests
+           (lambda _
+             ;; Disable failing tests: these tests attempt to access
+             ;; commands or network resources which are neither available
+             ;; nor necessary for the build to succeed.
+             (for-each
+              (match-lambda
+                ((file test)
+                 (let ((regex (string-append "^(func\\s+)(" test "\\()")))
+                   (substitute* file
+                     ((regex all before test_name)
+                      (string-append before "Disabled" test_name))))))
+              '(("src/net/cgo_unix_test.go" "TestCgoLookupPort")
+                ("src/net/cgo_unix_test.go" "TestCgoLookupPortWithCancel")
+                ;; 127.0.0.1 doesn't exist
+                ("src/net/cgo_unix_test.go" "TestCgoLookupPTR")
+                ("src/net/cgo_unix_test.go" "TestCgoLookupPTRWithCancel")
+                ;; /etc/services doesn't exist
+                ("src/net/parse_test.go" "TestReadLine")
+                ;; The user's directory doesn't exist
+                ("src/os/os_test.go" "TestUserHomeDir")))
+
+             ;; These tests fail on aarch64-linux
+             (substitute* "src/cmd/dist/test.go"
+               (("t.registerHostTest\\(\"testsanitizers/msan.*") ""))))
+         (add-after 'patch-source 'enable-external-linking
+           (lambda _
+             ;; Invoke GCC to link any archives created with GCC (that is, any
+             ;; packages built using 'cgo'), because Go doesn't know how to
+             ;; handle the runpaths but GCC does.  Use substitute* rather than
+             ;; a patch since these files are liable to change often.
+             ;;
+             ;; XXX: Replace with GO_EXTLINK_ENABLED=1 or similar when
+             ;; <https://github.com/golang/go/issues/31544> and/or
+             ;; <https://github.com/golang/go/issues/43525> are resolved.
+             (substitute* "src/cmd/link/internal/ld/config.go"
+               (("iscgo && externalobj") "iscgo"))
+             (substitute* '("src/cmd/nm/nm_cgo_test.go"
+                            "src/cmd/dist/test.go")
+               (("^func.*?nternalLink\\(\\).*" all)
+                (string-append all "\n\treturn false\n")))))
+         (replace 'build
+           (lambda* (#:key (parallel-build? #t) #:allow-other-keys)
+             (let* ((njobs (if parallel-build? (parallel-job-count) 1)))
+               (with-directory-excursion "src"
+                 (setenv "GOMAXPROCS" (number->string njobs))
+                 (invoke "sh" "make.bash" "--no-banner")))))
+         (replace 'check
+           (lambda* (#:key target (tests? (not target)) (parallel-tests? #t)
+                     #:allow-other-keys)
+             (let* ((njobs (if parallel-tests? (parallel-job-count) 1)))
+               (when tests?
+                 (with-directory-excursion "src"
+                   (setenv "GOMAXPROCS" (number->string njobs))
+                   (invoke "sh" "run.bash" "--no-rebuild"))))))
+         (add-before 'install 'unpatch-perl-shebangs
+           (lambda _
+             ;; Avoid inclusion of perl in closure by rewriting references
+             ;; to perl input in sourcecode generators and test scripts
+             (substitute* (cons "src/net/http/cgi/testdata/test.cgi"
+                                (find-files "src" "\\.pl$"))
+               (("^#!.*") "#!/usr/bin/env perl\n"))))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             ;; Notably, we do not install archives (180M), which Go will
+             ;; happily recompile quickly (and cache) if needed, almost
+             ;; surely faster than they could be substituted.
+             ;;
+             ;; The main motivation for pre-compiled archives is to use
+             ;; libc-linked `net' or `os' packages without a C compiler,
+             ;; but on Guix a C compiler is necessary to properly link the
+             ;; final binaries anyway.  Many build flags also invalidate
+             ;; these pre-compiled archives, so in practice Go often
+             ;; recompiles them anyway.
+             ;;
+             ;; Upstream is also planning to no longer install these
+             ;; archives: <https://github.com/golang/go/issues/47257>
+             ;;
+             ;; When necessary, a custom pre-compiled library package can
+             ;; be created with `#:import-path "std"' and used with
+             ;; `-pkgdir'.
+             (let* ((out (assoc-ref outputs "out"))
+                    (tests (assoc-ref outputs "tests")))
+               (for-each
+                (lambda (file)
+                  (copy-recursively file (string-append out "/lib/go/" file)))
+                '("lib" "VERSION" "pkg/include" "pkg/tool"))
+
+               (for-each
+                (match-lambda
+                  ((file dest output)
+                   ;; Copy to output/dest and symlink from output/lib/go/file.
+                   (let ((file* (string-append output "/lib/go/" file))
+                         (dest* (string-append output "/" dest)))
+                     (copy-recursively file dest*)
+                     (mkdir-p (dirname file*))
+                     (symlink (string-append "../../" dest) file*))))
+                `(("bin"          "bin"                 ,out)
+                  ("src"          "share/go/src"        ,out)
+                  ("misc"         "share/go/misc"       ,out)
+                  ("doc"          "share/doc/go/doc"    ,out)
+                  ("api"          "share/go/api"        ,tests)
+                  ("test"         "share/go/test"       ,tests))))))
+         (add-after 'install 'install-doc-files
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((out (assoc-ref outputs "out")))
+               (for-each
+                (lambda (file)
+                  (install-file file (string-append out "/share/doc/go")))
+                '("AUTHORS" "CONTRIBUTORS" "CONTRIBUTING.md" "PATENTS"
+                  "README.md" "SECURITY.md"))))))))
+    (inputs (alist-delete "gcc:lib" (package-inputs go-1.16)))))
+
 (define-public go go-1.14)
 
 (define-public go-0xacab-org-leap-shapeshifter

base-commit: 95c29d2746943733cbe8df7013854d45bb0df413
-- 
2.31.1





^ permalink raw reply related	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-09-09 21:37 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-09-02 22:09 [bug#50348] [PATCH] gnu: Add go-1.17 Sarah Morgensen
2021-09-09 19:45 ` [bug#50348] [PATCH v2] " Sarah Morgensen
2021-09-09 20:50   ` Leo Famulari
2021-09-09 20:55   ` bug#50348: " Leo Famulari
2021-09-09 20:27 ` [bug#50348] [PATCH] " Leo Famulari
2021-09-09 21:36   ` Sarah Morgensen
2021-09-09 20:53 ` Leo Famulari

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).