[bug#43872] Closing due to consistency issues

[Helio Machado <0x2b3bfa0@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 335 bytes --]

Hi!
I started yesterday to improve the Go build system and create a solid
recursive importer. It implies some structural change for every package,
but it's probably worth it and I'll submit a new patch once it's ready.
I'm attaching some internal documentation for my hack for historical
purposes.
Thank you very much for your advice!

[-- Attachment #1.2: Type: text/html, Size: 403 bytes --]

[-- Attachment #2: Guix-go-modules.md --]
[-- Type: text/markdown, Size: 14961 bytes --]

# Go modules within Guix

Using the new [Go modules][4] with the Guix package manager poses some interesting issues, as the package importer is not implemented yet and Go's own package manager clashes heavily with the Guix package manager.

I wanted to port a package without needing to follow the cumbersome process of manually importing dozens of dependencies, so the less-friction hack turned to be the repurposing of the build derivation as a deterministic downloader module, able to pull all the dependencies with the native Go package manager as if it were one of the Guix official download methods.

The suggested approach is rather unorthodox and may not be a good long-term solution: after all, it resorts to all the possible trickery to reduce changes outside of the package to a minimum.

## Network access inside the build container
[As per the `libstore` source code][0], fixed-output derivations have unrestricted network access as long as their contents are kept deterministic and you provide a hash of the final state. _Id est_, you can gather files from a remote repository as long as you're able to get the same commit or version every time.

> _When `hash` and `hash-algo` are given, a fixed-output derivation is created —i.e., one whose result is known in advance, such as a file download. If, in addition, `recursive?`` is true, then that fixed output may be an executable file or a directory and hash must be the hash of an archive containing this output. [(Source)][1]_

By extending [`guix/build-system/go.scm`][2], we can pass through three additional keyword arguments to `build-expression->derivation` specifying the derivation hash:

```patch
diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index f8ebaefb27..f6b6a7809d 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -88,6 +88,9 @@
                    (allow-go-reference? #f)
                    (system (%current-system))
                    (guile #f)
+                   (derivation-hash #f)
+                   (derivation-hash-algorithm #f)
+                   (derivation-hash-recursive? #f)
                    (imported-modules %go-build-system-modules)
                    (modules '((guix build go-build-system)
                               (guix build union)
@@ -114,6 +117,9 @@
                 #:build-flags ,build-flags
                 #:tests? ,tests?
                 #:allow-go-reference? ,allow-go-reference?
+                #:derivation-hash ,derivation-hash
+                ; #:derivation-hash-algorithm ,derivation-hash-algorithm ; FIXME
+                #:derivation-hash-recursive? ,derivation-hash-recursive?
                 #:inputs %build-inputs)))

   (define guile-for-build
@@ -131,6 +137,9 @@
                                 #:system system
                                 #:modules imported-modules
                                 #:outputs outputs
+                                #:hash derivation-hash
+                                #:hash-algo derivation-hash-algorithm
+                                #:recursive? derivation-hash-recursive?
                                 #:guile-for-build guile-for-build))

 (define go-build-system
```

This way, any package can create a fixed-output build derivation with network access in order to pull additional files during the build process without breaking the build determinism:

```scheme
(package
  (build-system go-build-system)
  (arguments
    (#:derivation-hash (base32 "···")
     #:derivation-hash-algorithm 'sha256
     #:derivation-hash-recursive? #t)))
```

## X509 certificates for network requests

Once we introduce networking capabilities inside the container, we need to provision it with certificates so it's able to authenticate secure connections to the outside world. In order to achieve this, we would need to include the `certs` package first:

```patch
diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index fe34d34491..03f8959b7f 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -46,6 +46,7 @@
   #:use-module (guix build-system go)
   #:use-module (gnu packages)
   #:use-module (gnu packages admin)
+  #:use-module (gnu packages certs)
   #:use-module (gnu packages gcc)
   #:use-module (gnu packages glib)
   #:use-module (gnu packages base)
```

Then, we can include the certificate bundle at the package level through the `inputs` keyword argument:

```scheme
(package
  (build-system go-build-system)
  (inputs `(("nss-certs" ,nss-certs))))
```

Additionally, we're going to need a few environment variables pointing to the actual certificate directory, because it won't be at the standard location:

```scheme
(package
  (build-system go-build-system)
  (arguments
   `(#:phases
      (modify-phases %standard-phases
        (add-after 'setup-go-environment 'add-certificates
          (lambda _
            (setenv "SSL_CERT_DIR" (string-append
              (assoc-ref %build-inputs "nss-certs")
              "/etc/ssl/certs"))
            (setenv "SSL_CERT_FILE" (string-append
              (getenv "SSL_CERT_DIR")
              "/ca-certificates.crt"))
            #t))))))
```

## Support for the new module system

Now, we need to enable the new module system through the `GO111MODULE` environment variable, as it's being disabled by default from [`guix/build/go-build-system.scm`][3] due to the previous lack of support. As this proposal is highly experimental and every package expects it set to `off`, it's probably better to toggle it on an the package level:

```scheme
(package
  (build-system go-build-system)
  (arguments
   `(#:phases
      (modify-phases %standard-phases
        (add-after 'setup-go-environment 'enable-modules
          (lambda _
            (setenv "GO111MODULE" "on")
            #t))))))
```

## Example package: `go-ethereum`

```scheme
(define-public go-github-com-ethereum-go-ethereum
  (package
    (name "go-github-com-ethereum-go-ethereum")
    (version "1.9.22")
    (source (origin
              (method git-fetch)
              (uri (git-reference
                    (url "https://github.com/ethereum/go-ethereum")
                    (commit (string-append "v" version))))
              (file-name (git-file-name name version))
              (sha256
               (base32
                "08i31xil2lygfcn2igsvn4hpg8xnf8l6g914f78hgl4wj6v1dja9"))))
    (build-system go-build-system)
    (arguments
     `(#:import-path "github.com/ethereum/go-ethereum"
       #:tests? #f ; tests are broken
       #:derivation-hash ,(base32 "1ab7gd90g85ciwsrw6zjaxs76j7y0a11kavp50xkwmm26x34sjss")
       #:derivation-hash-algorithm sha256
       #:derivation-hash-recursive? ,#t
       #:phases
       (modify-phases %standard-phases
         (add-after 'setup-go-environment 'modify-go-environment
           (lambda _
             (setenv "GO111MODULE" "on")
             (setenv "SSL_CERT_DIR" (string-append
               (assoc-ref %build-inputs "nss-certs")
               "/etc/ssl/certs"))
             (setenv "SSL_CERT_FILE" (string-append
               (getenv "SSL_CERT_DIR")
               "/ca-certificates.crt"))
             #t))
         (replace 'build
           (lambda* (#:key import-path build-flags #:allow-other-keys)
             (with-directory-excursion (string-append "src/" import-path)
               (invoke "go" "run" "build/ci.go" "install"))))
         (replace 'install
           (lambda* (#:key outputs import-path #:allow-other-keys)
             (let* ((out (assoc-ref outputs "out"))
               (source (string-append (getenv "GOPATH") "/src/" import-path "/build"))
               (dest out))
             (mkdir-p dest)
             (copy-recursively source dest #:keep-mtime? #t))))
         (replace 'check
           (lambda* (#:key tests? import-path #:allow-other-keys)
             (if tests?
               (with-directory-excursion (string-append "src/" import-path)
                 (invoke "go" "run" "build/ci.go" "test"))
               #t))))))
    (inputs `(("nss-certs" ,nss-certs)))
    (home-page "https://github.com/ethereum/go-ethereum")
    (synopsis "Official Go implementation of the Ethereum protocol")
    (description "This repository contains the official
Go implementation of the Ethereum protocol and command
line tools for interacting with the blockchain.")
    (license license:lgpl3)))
```

## Full patches, ready to apply

```patch
diff --git a/gnu/packages/golang.scm b/gnu/packages/golang.scm
index fe34d34491..85c368942b 100644
--- a/gnu/packages/golang.scm
+++ b/gnu/packages/golang.scm
@@ -46,6 +46,7 @@
   #:use-module (guix build-system go)
   #:use-module (gnu packages)
   #:use-module (gnu packages admin)
+  #:use-module (gnu packages certs)
   #:use-module (gnu packages gcc)
   #:use-module (gnu packages glib)
   #:use-module (gnu packages base)
@@ -933,6 +934,62 @@ time.")
 is similar to Go's standard library @code{json} and @code{xml} package.")
     (license license:expat)))

+(define-public go-github-com-ethereum-go-ethereum
+  (package
+    (name "go-github-com-ethereum-go-ethereum")
+    (version "1.9.22")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/ethereum/go-ethereum")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "08i31xil2lygfcn2igsvn4hpg8xnf8l6g914f78hgl4wj6v1dja9"))))
+    (build-system go-build-system)
+    (arguments
+     `(#:import-path "github.com/ethereum/go-ethereum"
+       #:tests? #f ; tests are broken
+       #:derivation-hash ,(base32 "1ab7gd90g85ciwsrw6zjaxs76j7y0a11kavp50xkwmm26x34sjss")
+       #:derivation-hash-algorithm sha256
+       #:derivation-hash-recursive? ,#t
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'setup-go-environment 'modify-go-environment
+           (lambda _
+             (setenv "GO111MODULE" "on")
+             (setenv "SSL_CERT_DIR" (string-append
+               (assoc-ref %build-inputs "nss-certs")
+               "/etc/ssl/certs"))
+             (setenv "SSL_CERT_FILE" (string-append
+               (getenv "SSL_CERT_DIR")
+               "/ca-certificates.crt"))
+             #t))
+         (replace 'build
+           (lambda* (#:key import-path build-flags #:allow-other-keys)
+             (with-directory-excursion (string-append "src/" import-path)
+               (invoke "go" "run" "build/ci.go" "install"))))
+         (replace 'install
+           (lambda* (#:key outputs import-path #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+               (source (string-append (getenv "GOPATH") "/src/" import-path "/build"))
+               (dest out))
+             (mkdir-p dest)
+             (copy-recursively source dest #:keep-mtime? #t))))
+         (replace 'check
+           (lambda* (#:key tests? import-path #:allow-other-keys)
+             (if tests?
+               (with-directory-excursion (string-append "src/" import-path)
+                 (invoke "go" "run" "build/ci.go" "test"))
+               #t))))))
+    (inputs `(("nss-certs" ,nss-certs)))
+    (home-page "https://github.com/ethereum/go-ethereum")
+    (synopsis "Official Go implementation of the Ethereum protocol")
+    (description "This repository contains the official Go implementation of
+the Ethereum protocol and command line tools for interacting with the blockchain.")
+    (license license:lgpl3)))
+
 (define-public go-github-com-getsentry-raven-go
   (let ((commit "5c24d5110e0e198d9ae16f1f3465366085001d92")
         (revision "0"))
```

```patch
diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index f8ebaefb27..e8743e3929 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -88,6 +88,9 @@
                    (allow-go-reference? #f)
                    (system (%current-system))
                    (guile #f)
+                   (derivation-hash #f)
+                   (derivation-hash-algorithm #f)
+                   (derivation-hash-recursive? #f)
                    (imported-modules %go-build-system-modules)
                    (modules '((guix build go-build-system)
                               (guix build union)
@@ -114,6 +117,9 @@
                 #:build-flags ,build-flags
                 #:tests? ,tests?
                 #:allow-go-reference? ,allow-go-reference?
+                #:derivation-hash ,derivation-hash
+                ; FIXME #:derivation-hash-algorithm ,derivation-hash-algorithm
+                #:derivation-hash-recursive? ,derivation-hash-recursive?
                 #:inputs %build-inputs)))

   (define guile-for-build
@@ -131,6 +137,9 @@
                                 #:system system
                                 #:modules imported-modules
                                 #:outputs outputs
+                                #:hash derivation-hash
+                                #:hash-algo derivation-hash-algorithm
+                                #:recursive? derivation-hash-recursive?
                                 #:guile-for-build guile-for-build))

 (define go-build-system
```

## Pending loose ends

* I've been unable to uncomment the `FIXME` line at `guix/build-system/go.scm` without getting an undecipherable [for me] stack trace. This should be fixed to keep compatibility with other hashing algorithms.

* We're only installing the built binary packages to the root of the derivation output, but there might be more interesting build artifacts to be added to use this package as a module.


# Guix testing environment

I'm documenting my recipe for creating a testing environment from a clean install.

## Upgrading the system
```bash
export PATH="/home/$USER/.config/guix/current/bin:$PATH"
```
```bash
guix pull
hash guix
guix pull
```
```bash
GUIX_PROFILE="/home/$USER/.guix-profile"
source "$GUIX_PROFILE/etc/profile"
```

## Installing the build dependencies
```bash
guix install git autoconf automake gettext texinfo graphviz help2man
```

## Getting the source code
```bash
git clone https://git.savannah.gnu.org/git/guix.git
cd guix
```

## Entering inside a container
```bash
guix environment guix --pure
```

## Setting up the project
```bash
./bootstrap
./configure --localstatedir=/var
make
```

## Sourcing `guix` profile inside the container
```bash
GUIX_PROFILE="/home/$USER/.guix-profile"
source "$GUIX_PROFILE/etc/profile"
```

## Running commands
```bash
./pre-inst-env guix --version
```

[0]: https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc#n1915
[1]: https://guix.gnu.org/manual/en/html_node/Derivations.html
[2]: https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/go.scm#n129
[3]: https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/go-build-system.scm#n149
[4]: https://blog.golang.org/using-go-modules