[bug#30579] Go 1.10

[bug#30579] Go 1.10

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 951 bytes --]

This patch adds Go 1.10 but leaves Go 1.9 as the default Go.

Go 1.10 changes the way that Go decides whether to re-use built objects
or to recompile them [0]. Reading the release notes, it seems to
implement something like a memoized cache for built objects and test
results.

Go 1.10 works with our go-build-system, but the built objects are not
re-used. Instead, either most or all required objects are rebuilt, so
the build system will need some adjustment.

I think it's useful to have this new version of Go packaged, especially
if somebody wants to decipher how the new compiler decides when to
re-use the built objects :)

Also, there is a data race found in the test suite when bootstrapping
with GCC 5, so this patch changes the entire Go stack to GCC 6. I'll
send a report upstream.

[0] The "build cache" is mentioned in the link below. The mechanism is
called "content-based staleness" in some places.
https://golang.org/doc/go1.10#build

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#30579] [PATCH] gnu: Add go@1.10.

[Leo Famulari]

Building Go 1.10 with the Go 1.4 bootstrap, Thread Sanitizer from GCC 5
finds a data race during the the test suite of Go 1.10. With GCC 6, the
race doesn't seem to be present.

.##### ../misc/cgo/testsanitizers
--- FAIL: TestShared (0.03s)
    --- FAIL: TestShared/tsan_shared (2.54s)
        cshared_test.go:71: `/tmp/guix-build-go-1.10.drv-0/TestShared367937240/tsan_shared` exited with exit status 66
                ==================
                WARNING: ThreadSanitizer: data race (pid=28778)
                  Write of size 8 at 0x7ffff5495c30 by thread T1:
                    #0 pthread_attr_getstacksize <null> (tsan_shared+0x00000041be82)
                    #1 x_cgo_init <null> (libtsan_shared.so+0x00000008aeaf)
                    #2 runtime.rt0_go /tmp/guix-build-go-1.10.drv-0/go/src/runtime/asm_amd64.s:199 (libtsan_shared.so+0x000000081586)

                  Previous write of size 8 at 0x7ffff5495c30 by main thread:
                    #0 <null> <null> (0x000000000001)

                  Location is stack of thread T1.

                  Thread T1 (tid=28786, running) created by main thread at:
                    #0 pthread_create <null> (tsan_shared+0x00000040c794)
                    #1 _cgo_try_pthread_create <null> (libtsan_shared.so+0x00000008ad70)
                    #2 x_cgo_sys_thread_create <null> (libtsan_shared.so+0x00000008ab12)
                    #3 _rt0_amd64_lib /tmp/guix-build-go-1.10.drv-0/go/src/runtime/asm_amd64.s:56 (libtsan_shared.so+0x0000000813b6)

                SUMMARY: ThreadSanitizer: data race ??:0 __interceptor_pthread_attr_getstacksize

* gnu/packages/golang.scm (go-1.10): New variable.
(go-1.4)[native-inputs]: Use gcc:lib version 6.
---
 gnu/packages/golang.scm | 115 +++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 114 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index f11cc76d9..a542eaba8 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -187,7 +187,7 @@
     (inputs
      `(("tzdata" ,tzdata)
        ("pcre" ,pcre)
-       ("gcc:lib" ,gcc "lib")))
+       ("gcc:lib" ,gcc-6 "lib")))
     (native-inputs
      `(("pkg-config" ,%pkg-config)
        ("which" ,which)
@@ -372,6 +372,119 @@ in the style of communicating sequential processes (@dfn{CSP}).")
        ,@(package-native-inputs go-1.4)))
     (supported-systems %supported-systems)))
 
+(define-public go-1.10
+  (package
+    (inherit go-1.9)
+    (name "go")
+    (version "1.10")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append "https://storage.googleapis.com/golang/"
+                           name version ".src.tar.gz"))
+       (sha256
+        (base32
+         "093z1h0gmi458kv7smpx0ph6jm7ss9mzxa432kysbz85jhl4kppk"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments go-1.9)
+       ((#:phases phases)
+        `(modify-phases ,phases
+           (replace 'prebuild
+             (lambda* (#:key inputs outputs #:allow-other-keys)
+               (let* ((gcclib (string-append (assoc-ref inputs "gcc:lib") "/lib"))
+                      (ld (string-append (assoc-ref inputs "libc") "/lib"))
+                      (loader (car (find-files ld "^ld-linux.+")))
+                      (net-base (assoc-ref inputs "net-base"))
+                      (tzdata-path
+                       (string-append (assoc-ref inputs "tzdata") "/share/zoneinfo"))
+                      (output (assoc-ref outputs "out")))
+
+                 ;; Removing net/ tests, which fail when attempting to access
+                 ;; network resources not present in the build container.
+                 (for-each delete-file
+                           '("net/listen_test.go"
+                             "net/parse_test.go"
+                             "net/cgo_unix_test.go"))
+
+                 (substitute* "os/os_test.go"
+                   (("/usr/bin") (getcwd))
+                   (("/bin/pwd") (which "pwd"))
+                   (("/bin/sh") (which "sh")))
+
+                 ;; Add libgcc to runpath
+                 (substitute* "cmd/link/internal/ld/lib.go"
+                   (("!rpath.set") "true"))
+                 (substitute* "cmd/go/internal/work/gccgo.go"
+                   (("cgoldflags := \\[\\]string\\{\\}")
+                    (string-append "cgoldflags := []string{"
+                                   "\"-rpath=" gcclib "\""
+                                   "}"))
+                   (("\"-lgcc_s\", ")
+                    (string-append
+                     "\"-Wl,-rpath=" gcclib "\", \"-lgcc_s\", ")))
+                 (substitute* "cmd/go/internal/work/gc.go"
+                   (("ldflags = setextld\\(ldflags, compiler\\)")
+                    (string-append
+                     "ldflags = setextld(ldflags, compiler)\n"
+                     "ldflags = append(ldflags, \"-r\")\n"
+                     "ldflags = append(ldflags, \"" gcclib "\")\n")))
+
+                 ;; Disable failing tests: these tests attempt to access
+                 ;; commands or network resources which are neither available
+                 ;; nor necessary for the build to succeed.
+                 (for-each
+                  (match-lambda
+                    ((file regex)
+                     (substitute* file
+                       ((regex all before test_name)
+                        (string-append before "Disabled" test_name)))))
+                  '(("net/net_test.go" "(.+)(TestShutdownUnix.+)")
+                    ("net/dial_test.go" "(.+)(TestDialTimeout.+)")
+                    ("os/os_test.go" "(.+)(TestHostname.+)")
+                    ("time/format_test.go" "(.+)(TestParseInSydney.+)")
+                    ("time/format_test.go" "(.+)(TestParseInLocation.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestEcho.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestCommandRelativeName.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestCatStdin.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestCatGoodAndBadFile.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestExitStatus.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestPipes.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestStdinClose.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestIgnorePipeErrorOnSuccess.+)")
+                    ("syscall/syscall_unix_test.go" "(.+)(TestPassFD\\(.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestExtraFiles/areturn.+)")
+                    ("cmd/go/go_test.go" "(.+)(TestCoverageWithCgo.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestOutputStderrCapture.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestExtraFiles.+)")
+                    ("os/exec/exec_test.go" "(.+)(TestExtraFilesRace.+)")
+                    ("net/lookup_test.go" "(.+)(TestLookupPort.+)")
+                    ("syscall/exec_linux_test.go"
+                     "(.+)(TestCloneNEWUSERAndRemapNoRootDisableSetgroups.+)")))
+
+                 ;; fix shebang for testar script
+                 ;; note the target script is generated at build time.
+                 (substitute* "../misc/cgo/testcarchive/carchive_test.go"
+                   (("#!/usr/bin/env") (string-append "#!" (which "env"))))
+
+                 (substitute* "net/lookup_unix.go"
+                   (("/etc/protocols") (string-append net-base "/etc/protocols")))
+                 (substitute* "net/port_unix.go"
+                   (("/etc/services") (string-append net-base "/etc/services")))
+                 (substitute* "time/zoneinfo_unix.go"
+                   (("/usr/share/zoneinfo/") tzdata-path))
+                 (substitute* (find-files "cmd" "\\.go")
+                   (("/lib(64)?/ld-linux.*\\.so\\.[0-9]") loader))
+                 #t)))
+           (replace 'set-bootstrap-variables
+             (lambda* (#:key outputs inputs #:allow-other-keys)
+               ;; Tell the build system where to find the bootstrap Go.
+               (let ((go  (assoc-ref inputs "go")))
+                 (setenv "GOROOT_BOOTSTRAP" go)
+                 (setenv "GOGC" "400")
+                 ;; Go 1.10 tries to write to $HOME in a test
+                 (setenv "HOME" "/tmp")
+                 #t)))))))))
+
 (define-public go go-1.9)
 
 (define-public go-github-com-alsm-ioprogress
-- 
2.16.2

[bug#30579] [PATCH] gnu: Add go@1.10.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 9123 bytes --]

Leo Famulari <leo@famulari.name> writes:

> Building Go 1.10 with the Go 1.4 bootstrap, Thread Sanitizer from GCC 5
> finds a data race during the the test suite of Go 1.10. With GCC 6, the
> race doesn't seem to be present.

Can you add this comment next to the GCC input?  That way it's obvious
for people wondering about it, without having to grep the commit log.

Other than that LGTM.

>
> .##### ../misc/cgo/testsanitizers
> --- FAIL: TestShared (0.03s)
>     --- FAIL: TestShared/tsan_shared (2.54s)
>         cshared_test.go:71: `/tmp/guix-build-go-1.10.drv-0/TestShared367937240/tsan_shared` exited with exit status 66
>                 ==================
>                 WARNING: ThreadSanitizer: data race (pid=28778)
>                   Write of size 8 at 0x7ffff5495c30 by thread T1:
>                     #0 pthread_attr_getstacksize <null> (tsan_shared+0x00000041be82)
>                     #1 x_cgo_init <null> (libtsan_shared.so+0x00000008aeaf)
>                     #2 runtime.rt0_go /tmp/guix-build-go-1.10.drv-0/go/src/runtime/asm_amd64.s:199 (libtsan_shared.so+0x000000081586)
>
>                   Previous write of size 8 at 0x7ffff5495c30 by main thread:
>                     #0 <null> <null> (0x000000000001)
>
>                   Location is stack of thread T1.
>
>                   Thread T1 (tid=28786, running) created by main thread at:
>                     #0 pthread_create <null> (tsan_shared+0x00000040c794)
>                     #1 _cgo_try_pthread_create <null> (libtsan_shared.so+0x00000008ad70)
>                     #2 x_cgo_sys_thread_create <null> (libtsan_shared.so+0x00000008ab12)
>                     #3 _rt0_amd64_lib /tmp/guix-build-go-1.10.drv-0/go/src/runtime/asm_amd64.s:56 (libtsan_shared.so+0x0000000813b6)
>
>                 SUMMARY: ThreadSanitizer: data race ??:0 __interceptor_pthread_attr_getstacksize
>
> * gnu/packages/golang.scm (go-1.10): New variable.
> (go-1.4)[native-inputs]: Use gcc:lib version 6.
> ---
>  gnu/packages/golang.scm | 115 +++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 114 insertions(+), 1 deletion(-)
>
> diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
> index f11cc76d9..a542eaba8 100644
> --- a/gnu/packages/golang.scm
> +++ b/gnu/packages/golang.scm
> @@ -187,7 +187,7 @@
>      (inputs
>       `(("tzdata" ,tzdata)
>         ("pcre" ,pcre)
> -       ("gcc:lib" ,gcc "lib")))
> +       ("gcc:lib" ,gcc-6 "lib")))
>      (native-inputs
>       `(("pkg-config" ,%pkg-config)
>         ("which" ,which)
> @@ -372,6 +372,119 @@ in the style of communicating sequential processes (@dfn{CSP}).")
>         ,@(package-native-inputs go-1.4)))
>      (supported-systems %supported-systems)))
>  
> +(define-public go-1.10
> +  (package
> +    (inherit go-1.9)
> +    (name "go")
> +    (version "1.10")
> +    (source
> +     (origin
> +       (method url-fetch)
> +       (uri (string-append "https://storage.googleapis.com/golang/"
> +                           name version ".src.tar.gz"))
> +       (sha256
> +        (base32
> +         "093z1h0gmi458kv7smpx0ph6jm7ss9mzxa432kysbz85jhl4kppk"))))
> +    (arguments
> +     (substitute-keyword-arguments (package-arguments go-1.9)
> +       ((#:phases phases)
> +        `(modify-phases ,phases
> +           (replace 'prebuild
> +             (lambda* (#:key inputs outputs #:allow-other-keys)
> +               (let* ((gcclib (string-append (assoc-ref inputs "gcc:lib") "/lib"))
> +                      (ld (string-append (assoc-ref inputs "libc") "/lib"))
> +                      (loader (car (find-files ld "^ld-linux.+")))
> +                      (net-base (assoc-ref inputs "net-base"))
> +                      (tzdata-path
> +                       (string-append (assoc-ref inputs "tzdata") "/share/zoneinfo"))
> +                      (output (assoc-ref outputs "out")))
> +
> +                 ;; Removing net/ tests, which fail when attempting to access
> +                 ;; network resources not present in the build container.
> +                 (for-each delete-file
> +                           '("net/listen_test.go"
> +                             "net/parse_test.go"
> +                             "net/cgo_unix_test.go"))
> +
> +                 (substitute* "os/os_test.go"
> +                   (("/usr/bin") (getcwd))
> +                   (("/bin/pwd") (which "pwd"))
> +                   (("/bin/sh") (which "sh")))
> +
> +                 ;; Add libgcc to runpath
> +                 (substitute* "cmd/link/internal/ld/lib.go"
> +                   (("!rpath.set") "true"))
> +                 (substitute* "cmd/go/internal/work/gccgo.go"
> +                   (("cgoldflags := \\[\\]string\\{\\}")
> +                    (string-append "cgoldflags := []string{"
> +                                   "\"-rpath=" gcclib "\""
> +                                   "}"))
> +                   (("\"-lgcc_s\", ")
> +                    (string-append
> +                     "\"-Wl,-rpath=" gcclib "\", \"-lgcc_s\", ")))
> +                 (substitute* "cmd/go/internal/work/gc.go"
> +                   (("ldflags = setextld\\(ldflags, compiler\\)")
> +                    (string-append
> +                     "ldflags = setextld(ldflags, compiler)\n"
> +                     "ldflags = append(ldflags, \"-r\")\n"
> +                     "ldflags = append(ldflags, \"" gcclib "\")\n")))
> +
> +                 ;; Disable failing tests: these tests attempt to access
> +                 ;; commands or network resources which are neither available
> +                 ;; nor necessary for the build to succeed.
> +                 (for-each
> +                  (match-lambda
> +                    ((file regex)
> +                     (substitute* file
> +                       ((regex all before test_name)
> +                        (string-append before "Disabled" test_name)))))
> +                  '(("net/net_test.go" "(.+)(TestShutdownUnix.+)")
> +                    ("net/dial_test.go" "(.+)(TestDialTimeout.+)")
> +                    ("os/os_test.go" "(.+)(TestHostname.+)")
> +                    ("time/format_test.go" "(.+)(TestParseInSydney.+)")
> +                    ("time/format_test.go" "(.+)(TestParseInLocation.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestEcho.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestCommandRelativeName.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestCatStdin.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestCatGoodAndBadFile.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestExitStatus.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestPipes.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestStdinClose.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestIgnorePipeErrorOnSuccess.+)")
> +                    ("syscall/syscall_unix_test.go" "(.+)(TestPassFD\\(.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestExtraFiles/areturn.+)")
> +                    ("cmd/go/go_test.go" "(.+)(TestCoverageWithCgo.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestOutputStderrCapture.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestExtraFiles.+)")
> +                    ("os/exec/exec_test.go" "(.+)(TestExtraFilesRace.+)")
> +                    ("net/lookup_test.go" "(.+)(TestLookupPort.+)")
> +                    ("syscall/exec_linux_test.go"
> +                     "(.+)(TestCloneNEWUSERAndRemapNoRootDisableSetgroups.+)")))
> +
> +                 ;; fix shebang for testar script
> +                 ;; note the target script is generated at build time.
> +                 (substitute* "../misc/cgo/testcarchive/carchive_test.go"
> +                   (("#!/usr/bin/env") (string-append "#!" (which "env"))))
> +
> +                 (substitute* "net/lookup_unix.go"
> +                   (("/etc/protocols") (string-append net-base "/etc/protocols")))
> +                 (substitute* "net/port_unix.go"
> +                   (("/etc/services") (string-append net-base "/etc/services")))
> +                 (substitute* "time/zoneinfo_unix.go"
> +                   (("/usr/share/zoneinfo/") tzdata-path))
> +                 (substitute* (find-files "cmd" "\\.go")
> +                   (("/lib(64)?/ld-linux.*\\.so\\.[0-9]") loader))
> +                 #t)))
> +           (replace 'set-bootstrap-variables
> +             (lambda* (#:key outputs inputs #:allow-other-keys)
> +               ;; Tell the build system where to find the bootstrap Go.
> +               (let ((go  (assoc-ref inputs "go")))
> +                 (setenv "GOROOT_BOOTSTRAP" go)
> +                 (setenv "GOGC" "400")
> +                 ;; Go 1.10 tries to write to $HOME in a test
> +                 (setenv "HOME" "/tmp")
> +                 #t)))))))))
> +
>  (define-public go go-1.9)
>  
>  (define-public go-github-com-alsm-ioprogress
> -- 
> 2.16.2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#30579] Go 1.10

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 296 bytes --]

On Thu, Feb 22, 2018 at 01:25:03PM -0500, Leo Famulari wrote:
> Also, there is a data race found in the test suite when bootstrapping
> with GCC 5, so this patch changes the entire Go stack to GCC 6. I'll
> send a report upstream.

Upstream bug report:

https://github.com/golang/go/issues/24046

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#30579] Go 1.10

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]

Leo Famulari <leo@famulari.name> writes:

> This patch adds Go 1.10 but leaves Go 1.9 as the default Go.
>
> Go 1.10 changes the way that Go decides whether to re-use built objects
> or to recompile them [0]. Reading the release notes, it seems to
> implement something like a memoized cache for built objects and test
> results.
>
> Go 1.10 works with our go-build-system, but the built objects are not
> re-used. Instead, either most or all required objects are rebuilt, so
> the build system will need some adjustment.
>
> I think it's useful to have this new version of Go packaged, especially
> if somebody wants to decipher how the new compiler decides when to
> re-use the built objects :)

Definitely.  Thanks for working on it!  :-)

>
> Also, there is a data race found in the test suite when bootstrapping
> with GCC 5, so this patch changes the entire Go stack to GCC 6. I'll
> send a report upstream.
>
> [0] The "build cache" is mentioned in the link below. The mechanism is
> called "content-based staleness" in some places.
> https://golang.org/doc/go1.10#build

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]