[bug#74370] [PATCH 0/4] Module aware go build system, downloader

[bug#74370] [PATCH 0/4] Module aware go build system, downloader

[Jørgen Kvalsvik]

Hi all,

I wrote a downloader+build system for go that is module aware.  It
builds on a few slightly different assumptions than the go build system,
but if you have worked on that most should feel familiar. I am posting
these patches to get some feedback and to check with you if this
approach is even viable, or if I should abandon it altogether. My real
goal has been to build hugo, a popular website generator, which has a
large list of dependencies, most of them in go.

Go itself has pretty strong ideas on how to compose systems. In
particular, it wants to statically link everything, pin every dependency
and transitive dependency (combining build system with packaging and
distribution), and strongly prefer full-source builds. This causes some
friction.

These patches chooses to just roll with it, and uses go mod
downloaditself to fetch sources and create a large image of all the
dependencies a project needs to build. This effectively makes it
pointless to package libraries, as the programs using the libraries
would get them from source anyway, but fixing that is future work.

There are a few things I would like to try to do (re-using built
libraries and storing each module in a separate derivation in
particular), but I don't want to tackle that problem unless there is
real interest for this approach.

Thoughts?

Jørgen Kvalsvik (4):
  guix: Add go module fetcher
  guix: add go module aware build system
  guix: Add module aware 'guix import go'
  gnu: Add go-buf.

 Makefile.am                        |   2 +
 gnu/packages/golang-xyz.scm        |  63 +++++++++---
 guix/build-system/go.scm           | 120 ++++++++++++++++++++++
 guix/build/go-mod-build-system.scm | 154 +++++++++++++++++++++++++++++
 guix/go-mod-download.scm           | 146 +++++++++++++++++++++++++++
 guix/import/go.scm                 |  88 ++++++++++-------
 guix/scripts/import/go.scm         |   6 ++
 7 files changed, 532 insertions(+), 47 deletions(-)
 create mode 100644 guix/build/go-mod-build-system.scm
 create mode 100644 guix/go-mod-download.scm


base-commit: 33665c52c4670bc3b4d337c89ac9cc6c4c69b26f
-- 
2.39.5

[bug#74372] [PATCH 1/4] guix: Add go module fetcher

[Jørgen Kvalsvik]

Add a new fetcher for go programs, based on go mod download.  This is
largely how go programs *want* to be fetched, and some pseudo-private
dependencies [1] are difficult to wrangle without letting go do the
work.  This patch only adds support for git as the root module fetch.

The approach is conceptually simple:

1. Fetch the root package to build (usually a git repo)
2. Find all the go.mod files in this tree and run go mod download
3. Store the sources of the package, its direct- and transitive
   dependencies as the source.

The source is in source/, the dependencies in go/pkg as encouraged by
the toolchain. Go tooling is generally very opinionated on how things
are supposed to be done, and fighting it usually brings pain.

This approach is not without drawbacks. Go programs tend to be very
liberal with dependencies and specific with versions, which will in
practice leads to a **large** set of sources. At a system level there
will be a lot of duplicated sources, and a lot of slightly different
versions which could be compatible.  This is not guix specific but
rather how go programs design their environments.

Another is libraries.  Go wants to statically link everything, and
generally likes to work with source, and every program pins dependencies
to different revisions.  The go-mod-fetch getting the source of all
libraries the libraries breaks the general packaing approach of
packaging libraries separately, but this was never really leveraged by
go anyway.  This switches the focus to packaging applications rather
than libraries.

Finally, the package.source needs two hashes - one for the direct origin
(e.g. git), and one for the sum of libraries fetched by go.mod.

[1] github.com/bufbuild/buf 1.46 requires
    buf.build/gen/go/bufbuild/bufplugin/protocolbuffers/go
    v1.35.1-20241031151143-70f632351282.1 which is only available as a
    .zip and has a layout that go mod understands, but is alien to the
    go-build-system.

* guix/go-mod-download.scm: New file.
* gnu/local.mk: Register it.

Change-Id: I84c00df07393a9978124667e3e2497aec7009252
---
 Makefile.am              |   1 +
 guix/go-mod-download.scm | 146 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 147 insertions(+)
 create mode 100644 guix/go-mod-download.scm

diff --git a/Makefile.am b/Makefile.am
index 3a35b7becd..fc00947f4f 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -102,6 +102,7 @@ MODULES =					\
   guix/android-repo-download.scm		\
   guix/bzr-download.scm            		\
   guix/git-download.scm				\
+  guix/go-mod-download.scm			\
   guix/hg-download.scm				\
   guix/hash.scm					\
   guix/swh.scm					\
diff --git a/guix/go-mod-download.scm b/guix/go-mod-download.scm
new file mode 100644
index 0000000000..1a7ffbb6ac
--- /dev/null
+++ b/guix/go-mod-download.scm
@@ -0,0 +1,146 @@
+;;; GNU Guix --- Functional package management for GNU
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+;;; TODOs:
+;;; 1. Support non-git root repositories
+;;; 2. Store/cache individual module downloads
+
+(define-module (guix go-mod-download)
+  #:use-module (guix build utils)
+  #:use-module (guix derivations)
+  #:use-module (guix packages)
+  #:use-module (guix gexp)
+  #:use-module (guix modules)
+  #:use-module (guix monads)
+  #:use-module (guix records)
+  #:use-module (guix store)
+  #:use-module (guix git-download)
+
+  #:export (go-mod-reference
+            go-mod-reference?
+            go-mod-source
+            go-mod-go
+
+            go-mod-fetch))
+
+(define (go-package)
+  "Return the default Go package."
+  (let ((distro (resolve-interface '(gnu packages golang))))
+    (module-ref distro 'go)))
+
+(define (nss-certs-package)
+  "Return the default nss-certs package."
+  (let ((distro (resolve-interface '(gnu packages certs))))
+    (module-ref distro 'nss-certs)))
+
+;; The source key accepts the same kinds as the package record, but mostly go
+;; use git.  The go key specifies which go version to use, which might be
+;; necessary if any module sets a newer toolchain in go.mod
+(define-record-type* <go-mod-reference>
+  go-mod-reference make-go-mod-reference
+  go-mod-reference?
+  (source go-mod-source)
+  (go go-mod-go (default (go-package)) (thunked)))
+
+;; Fetch all direct and indirect dependencies of the go modules in the source
+;; tree (usually a git repo) using go mod download.
+(define* (go-mod-fetch source hash-algo hash
+                       #:optional name
+                       #:key (system (%current-system))
+                       (guile (default-guile))
+                       (go (go-package))
+                       (nss-certs (nss-certs-package)))
+  (define guile-json
+    (module-ref (resolve-interface '(gnu packages guile)) 'guile-json-4))
+
+  (define* (build source go)
+    (with-imported-modules (source-module-closure
+                            '((guix build utils)
+                              (guix build download)
+                              (srfi srfi-34)))
+      (with-extensions (list guile-json)
+        #~(begin
+            (use-modules
+             (guix build download)
+             (guix build utils)
+             (srfi srfi-34))
+            (let* ((cert-dir (string-append #$nss-certs "/etc/ssl/certs"))
+                   (src-dir (string-append #$output "/source"))
+                   (mod-cache (string-append #$output "/go/pkg"))
+                   (xgo (string-append #$go "/bin/go")))
+              ;; go.mod files can specify a minimum required toolchain which could
+              ;; cause go mod download to fetch and install a newer compiler
+              ;; if invoked with an older one.
+              (setenv "GOTOOLCHAIN" (string-append "go" (getenv "go toolchain")))
+              (setenv "SSL_CERT_DIR" cert-dir)
+              (setenv "GOCACHE" "/homeless-shelter")
+              (setenv "GOPATH" "/homeless-shelter")
+              (setenv "GOMODCACHE" mod-cache)
+
+              (mkdir-p src-dir)
+              (mkdir-p mod-cache)
+              (copy-recursively #$source src-dir)
+              (let* ((go-mods (find-files src-dir "go.mod")))
+                ;; go mod will update the go.mod with transitive dependencies if
+                ;; they are not set, and fail with an error if the file is not
+                ;; writable.
+                (for-each make-file-writable go-mods)
+                (with-throw-handler
+                    #t
+                  (lambda _
+                    (for-each
+                     (lambda (go-mod)
+                       (with-directory-excursion (dirname go-mod)
+                         ;; go mod download must be run twice - the first
+                         ;; fetches direct dependencies and *records*
+                         ;; transitive dependencies, the second run fetches
+                         ;; the transitive dependencies.
+                         (and
+                          (invoke xgo "mod" "download")
+                          (invoke xgo "mod" "download"))))
+                     go-mods)
+                    #t)
+                  (lambda (key . args)
+                    (display (string-append "Fetching modules failed.\n"
+                                            "Here are the results of `go env`:\n"))
+                    (invoke xgo "env")))))))))
+
+  (let* ((mod-source (go-mod-source source))
+         (mod-ref  (origin-uri mod-source))
+         (mod-go (go-mod-go source))
+         (mod-hash (origin-hash mod-source))
+         (mod-hash-value (content-hash-value mod-hash))
+         (mod-hash-algo (content-hash-algorithm mod-hash)))
+
+  (mlet* %store-monad ((guile-for-build (package->derivation guile system))
+                       (git-source (git-fetch mod-ref mod-hash-algo
+                                              mod-hash-value
+                                              #:system system
+                                              #:guile guile-for-build)))
+    (gexp->derivation (or name "go-mod-fetch") (build git-source mod-go)
+                      #:script-name "go-mod-fetch"
+                      #:env-vars
+                      `(("go toolchain" . ,(package-version mod-go)))
+                      #:leaked-env-vars '("http_proxy" "https_proxy"
+                                          "LC_ALL" "LC_MESSAGES" "LANG"
+                                          "COLUMNS")
+                      #:system system
+                      #:local-build? #t ;don't offload repo cloning
+                      #:recursive? #t
+                      #:hash-algo hash-algo
+                      #:hash hash
+                      #:guile-for-build guile-for-build))))
-- 
2.39.5

[bug#74374] [PATCH 2/4] guix: add go module aware build system

[Jørgen Kvalsvik]

Add a go module aware build system, and make it available through
build-system/go.scm.  The go-mod-build and supporting functions is
largely a copy-and-paste job of the go-build and could probably be
refactored.

The build process when using go modules is slightly different from the
non-module version, and relies on sources already being fetched with
go-mod-fetch.  This revision does not do anything clever with reusing
compiled packages, but it might be possible to store precompiled modules
to the store path without adding explicit entries in golang-*.scm

* guix/build-system/go.scm (%go-mod-build-system-modules): New define.
(mod-lower): New function.
(go-mod-build): New function.
(go-mod-build-system): New function.
* guix/build-system/go-mod-build-system.scm: New file.
* gnu/local.mk: Register it.

Change-Id: I394089073b894e8cf9da5aa18759c939fca45a31
---
 Makefile.am                        |   1 +
 guix/build-system/go.scm           | 120 ++++++++++++++++++++++
 guix/build/go-mod-build-system.scm | 154 +++++++++++++++++++++++++++++
 3 files changed, 275 insertions(+)
 create mode 100644 guix/build/go-mod-build-system.scm

diff --git a/Makefile.am b/Makefile.am
index fc00947f4f..5768b721aa 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -226,6 +226,7 @@ MODULES =					\
   guix/build/minify-build-system.scm		\
   guix/build/font-build-system.scm		\
   guix/build/go-build-system.scm		\
+  guix/build/go-mod-build-system.scm		\
   guix/build/android-repo.scm			\
   guix/build/asdf-build-system.scm		\
   guix/build/bzr.scm				\
diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index 226688f2d2..1e60fd9471 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -38,6 +38,8 @@ (define-module (guix build-system go)
   #:export (%go-build-system-modules
             go-build
             go-build-system
+            go-mod-build
+            go-mod-build-system
 
             go-pseudo-version?
             go-target
@@ -117,6 +119,12 @@ (define %go-build-system-modules
     (guix build union)
     ,@%default-gnu-imported-modules))
 
+(define %go-mod-build-system-modules
+  ;; Build-side modules imported and used by default.
+  `((guix build go-mod-build-system)
+    (guix build union)
+    ,@%default-gnu-imported-modules))
+
 (define (default-go)
   ;; Lazily resolve the binding to avoid a circular dependency.
   (let ((go (resolve-interface '(gnu packages golang))))
@@ -181,6 +189,57 @@ (define inputs-with-cache
     (build (if target go-cross-build go-build))
     (arguments (strip-keyword-arguments private-keywords arguments))))
 
+(define* (mod-lower name
+                #:key source inputs native-inputs outputs system target
+                (go (if (supported-package? (default-go))
+                      (default-go)
+                      (default-gccgo)))
+                #:allow-other-keys
+                #:rest arguments)
+  "Return a bag for NAME."
+  (define private-keywords
+    '(#:target #:go #:inputs #:native-inputs))
+
+  (define inputs-with-cache
+    ;; XXX: Avoid a circular dependency.  This should be rewritten with
+    ;; 'package-mapping' or similar.
+    (let ((go-std-name (string-append (package-name go) "-std")))
+      (if (string-prefix? go-std-name name)
+          inputs
+          (cons `(,go-std-name ,((make-go-std) go)) inputs))))
+
+  (bag
+    (name name)
+    (system system)
+    (target target)
+    (build-inputs `(,@(if source
+                        `(("source" ,source))
+                        '())
+                     ,@`(("go" ,go))
+                     ,@native-inputs
+                     ,@(if target '() inputs-with-cache)
+                     ,@(if target
+                         ;; Use the standard cross inputs of
+                         ;; 'gnu-build-system'.
+                         (standard-cross-packages target 'host)
+                         '())
+                     ;; Keep the standard inputs of 'gnu-build-system'.
+                     ,@(standard-packages)))
+    (host-inputs (if target inputs-with-cache '()))
+
+    ;; The cross-libc is really a target package, but for bootstrapping
+    ;; reasons, we can't put it in 'host-inputs'.  Namely, 'cross-gcc' is a
+    ;; native package, so it would end up using a "native" variant of
+    ;; 'cross-libc' (built with 'gnu-build'), whereas all the other packages
+    ;; would use a target variant (built with 'gnu-cross-build'.)
+    (target-inputs (if target
+                     (standard-cross-packages target 'target)
+                     '()))
+
+    (outputs outputs)
+    (build go-mod-build)
+    (arguments (strip-keyword-arguments private-keywords arguments))))
+
 (define* (go-build name inputs
                    #:key
                    source
@@ -310,9 +369,70 @@ (define %outputs
                       #:substitutable? substitutable?
                       #:guile-for-build guile)))
 
+(define* (go-mod-build name inputs
+                   #:key
+                   source
+                   (phases '%standard-phases)
+                   (outputs '("out"))
+                   (search-paths '())
+                   (install-source? #t)
+                   (import-path "")
+                   (unpack-path "")
+                   (build-flags ''())
+                   (tests? #t)
+                   (parallel-build? #t)
+                   (parallel-tests? #t)
+                   (allow-go-reference? #f)
+                   (system (%current-system))
+                   (goarch #f)
+                   (goos #f)
+                   (guile #f)
+                   (imported-modules %go-mod-build-system-modules)
+                   (modules '((guix build go-mod-build-system)
+                              (guix build union)
+                              (guix build utils)))
+                   (substitutable? #t))
+  (define builder
+    (with-imported-modules imported-modules
+      #~(begin
+          (use-modules #$@modules)
+          (go-build #:name #$name
+                    #:source #+source
+                    #:system #$system
+                    #:phases #$phases
+                    #:outputs #$(outputs->gexp outputs)
+                    #:substitutable? #$substitutable?
+                    #:goarch #$goarch
+                    #:goos #$goos
+                    #:search-paths '#$(sexp->gexp
+                                       (map search-path-specification->sexp
+                                            search-paths))
+                    #:install-source? #$install-source?
+                    #:import-path #$import-path
+                    #:unpack-path #$unpack-path
+                    #:build-flags #$build-flags
+                    #:tests? #$tests?
+                    #:parallel-build? #$parallel-build?
+                    #:parallel-tests? #$parallel-tests?
+                    #:allow-go-reference? #$allow-go-reference?
+                    #:inputs #$(input-tuples->gexp inputs)))))
+
+  (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
+                                                  system #:graft? #f)))
+    (gexp->derivation name builder
+                      #:system system
+                      #:guile-for-build guile)))
+
 (define go-build-system
   (build-system
     (name 'go)
     (description
      "Build system for Go programs")
     (lower lower)))
+
+(define go-mod-build-system
+  (build-system
+    (name 'go)
+    (description
+     "Build system for Go programs, module aware")
+    (lower mod-lower)))
diff --git a/guix/build/go-mod-build-system.scm b/guix/build/go-mod-build-system.scm
new file mode 100644
index 0000000000..80a43a6a60
--- /dev/null
+++ b/guix/build/go-mod-build-system.scm
@@ -0,0 +1,154 @@
+;;; GNU Guix --- Functional package management for GNU
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build go-mod-build-system)
+  #:use-module ((guix build gnu-build-system) #:prefix gnu:)
+  #:use-module (guix build utils)
+  #:use-module (ice-9 match)
+  #:export (%standard-phases
+            go-build))
+
+;; Commentary:
+;;
+;; Build procedures for Go packages, using go modules.  This is the
+;; builder-side code.
+;;
+;; Software written in Go is either a 'package' (i.e. library) or 'command'
+;; (i.e. executable).  The module approach is currently heavily biased towards
+;; building executables.
+;;
+;; Unlike the go build system, this builder does not rely on the workspace
+;; or GOPATH, but instead assumes all modules are a part of the input source
+;; (otherwise, go build tries to download it which would fail).  Go projects
+;; rigidly specify dependencies which is handled by the sources being resolved
+;; and downloaded together.  The compiler is fast enough that building
+;; everything from source on a per-package basis is not a great speed loss,
+;; and the benefit from precompiling libraries is reduced by go statically
+;; linking everything anyway.
+;;
+;; TODO:
+;; * Re-use compiled packages
+;;
+;; Code:
+
+(define* (setup-go-environment #:key inputs outputs import-path goos goarch
+                               #:allow-other-keys)
+  "Prepare a Go build environment.  We need to tell go to use the specific
+toolchain even if a module specifies a (slightly) newer one.  We must also
+tell the go build command where to find downloaded packages (go/pkg) and
+where executables (\"commands\") are installed to."
+  (let* ((mod-cache (string-append (getcwd) "/go/pkg"))
+        (src-dir (string-append (getcwd) "/source"))
+        (go-dir (assoc-ref inputs "go"))
+        (out-dir (assoc-ref outputs "out")))
+
+    ;; TODO: Get toolchain from the program itself or package.version, not the
+    ;; store path
+    (setenv "GOTOOLCHAIN" (string-delete #\- (strip-store-file-name go-dir)))
+    (setenv "GOMODCACHE" mod-cache)
+    (setenv "GOCACHE" (string-append (getcwd) "/go/cache"))
+    (setenv "GOBIN" (string-append out-dir "/bin"))
+    (setenv "GO111MODULE" "on")
+    (setenv "GOARCH" (or goarch (getenv "GOHOSTARCH")))
+    (setenv "GOOS" (or goos (getenv "GOHOSTOS")))
+    (match goarch
+      ("arm"
+       (setenv "GOARM" "7"))
+      ((or "mips" "mipsel")
+       (setenv "GOMIPS" "hardfloat"))
+      ((or "mips64" "mips64le")
+       (setenv "GOMIPS64" "hardfloat"))
+      ((or "ppc64" "ppc64le")
+       (setenv "GOPPC64" "power8"))
+      (_ #t))))
+
+(define* (build #:key import-path build-flags (parallel-build? #t)
+                #:allow-other-keys)
+  "Build the package named by IMPORT-PATH."
+  (let* ((njobs (if parallel-build? (parallel-job-count) 1)))
+    (setenv "GOMAXPROCS" (number->string njobs)))
+
+  (with-throw-handler
+    #t
+    (lambda _
+      ;; TODO: This should maybe support list to install multiple commands
+      ;; from the same project in the same package
+      (with-directory-excursion (string-append "source/" import-path)
+        (apply invoke "go" "build"
+               "-v" ; print the name of packages as they are compiled
+               "-x" ; print each command as it is invoked
+               ;; Respectively, strip the symbol table and debug
+               ;; information, and the DWARF symbol table.
+               "-ldflags=-s -w"
+               `(,@build-flags))))
+    (lambda (key . args)
+      (display (string-append "Building '" import-path "' failed.\n"
+                              "Here are the results of `go env`:\n"))
+      (invoke "go" "env"))))
+
+(define* (check #:key tests? import-path (parallel-tests? #t)
+                #:allow-other-keys)
+  "Run the tests for the package named by IMPORT-PATH."
+  (when tests?
+    (let* ((njobs (if parallel-tests? (parallel-job-count) 1)))
+      (setenv "GOMAXPROCS" (number->string njobs)))
+    (with-directory-excursion (string-append "source/" import-path)
+      (invoke "go" "test")))
+  #t)
+
+(define* (install #:key install-source? source outputs import-path
+                  #:allow-other-keys)
+  (with-directory-excursion (string-append "source/" import-path)
+    (display "INSTALLING PROGRAM\n")
+    (invoke "go" "install"
+               "-v" ; print the name of packages as they are compiled
+               "-x" ; print each command as it is invoked
+               ;; Respectively, strip the symbol table and debug
+               ;; information, and the DWARF symbol table.
+               "-ldflags=-s -w"))
+
+  ;; TODO: This is probably less interesting when using the go-mod builder
+  (when install-source?
+    (let* ((out (assoc-ref outputs "out"))
+           (src (string-append source "/source"))
+           (dest (string-append out "/src")))
+      (mkdir-p dest)
+      (copy-recursively src dest #:keep-mtime? #t)))
+  #t)
+
+(define* (install-license-files #:rest args)
+  "Install license files matching LICENSE-FILE-REGEXP to 'share/doc'.  Adjust
+the standard install-license-files phase to first enter the correct directory."
+  (with-directory-excursion "source"
+    (apply (assoc-ref gnu:%standard-phases 'install-license-files) args)))
+
+
+(define %standard-phases
+  (modify-phases gnu:%standard-phases
+    (delete 'bootstrap)
+    (delete 'configure)
+    (delete 'patch-generated-file-shebangs)
+    (add-before 'build 'setup-go-environment setup-go-environment)
+    (replace 'build build)
+    (replace 'check check)
+    (replace 'install install)
+    (replace 'install-license-files install-license-files)))
+
+(define* (go-build #:key inputs (phases %standard-phases)
+                      #:allow-other-keys #:rest args)
+  "Build the given Go package, applying all of PHASES in order."
+  (apply gnu:gnu-build #:inputs inputs #:phases phases args))
-- 
2.39.5

[bug#74371] [PATCH 3/4] guix: Add module aware 'guix import go'

[Jørgen Kvalsvik]

Emit module aware package go packages.  It does not compute the hash of the
full module yet, and only supports git, but goes a long way towards making it
easy to package new go programs using the module-aware build system.

* guix/import/go.scm (go-module->guix-package): Add go.mod awareness
* guix/scripts/import/go.scm (show-help): Document -m, --mod
(%options): Accept them.

Change-Id: I4efd7260d69276279940e21698ecc7eb57232a67
---
 guix/import/go.scm         | 88 ++++++++++++++++++++++++--------------
 guix/scripts/import/go.scm |  6 +++
 2 files changed, 61 insertions(+), 33 deletions(-)

diff --git a/guix/import/go.scm b/guix/import/go.scm
index dd9298808d..967aa54d58 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -615,6 +615,7 @@ (define (validate-version version available-versions module-path)
 (define* (go-module->guix-package module-path #:key
                                   (goproxy "https://proxy.golang.org")
                                   version
+                                  go-mod?
                                   pin-versions?
                                   #:allow-other-keys)
   "Return the package S-expression corresponding to MODULE-PATH at VERSION, a Go package.
@@ -641,43 +642,62 @@ (define* (go-module->guix-package module-path #:key
          (meta-data (fetch-module-meta-data root-module-path))
          (vcs-type (module-meta-vcs meta-data))
          (vcs-repo-url (module-meta-data-repo-url meta-data goproxy))
+         (home-page (format #f "https://~a" root-module-path))
          (synopsis (go-package-synopsis module-path))
-         (description (go-package-description module-path))
-         (licenses (go-package-licenses module-path)))
-    (values
-     `(package
-        (name ,guix-name)
-        (version ,(strip-v-prefix version*))
-        (source
-         ,(vcs->origin vcs-type vcs-repo-url version*))
-        (build-system go-build-system)
-        (arguments
-         (list ,@(if (version>? min-go-version (package-version (go-package)))
-                     `(#:go ,(string->number min-go-version))
-                     '())
-               #:import-path ,module-path
-               ,@(if (string=? module-path-sans-suffix root-module-path)
-                     '()
-                     `(#:unpack-path ,root-module-path))))
-        ,@(maybe-propagated-inputs
-           (map (match-lambda
-                  ((name version)
-                   (go-module->guix-package-name name (strip-v-prefix version)))
-                  (name
-                   (go-module->guix-package-name name)))
-                dependencies))
-        (home-page ,(format #f "https://~a" root-module-path))
-        (synopsis ,synopsis)
-        (description ,(and=> description beautify-description))
-        (license ,(match (list->licenses licenses)
+         (description (and=> (go-package-description module-path) beautify-description))
+         (licenses (go-package-licenses module-path))
+         (license (match (list->licenses licenses)
                     (() #f)                       ;unknown license
-                    ((license)                    ;a single license
-                     license)
+                    ((license) license)           ;a single license
                     ((license ...)                ;a list of licenses
                      `(list ,@license)))))
-     (if pin-versions?
-         dependencies+versions
-         dependencies))))
+    (if go-mod?
+        (values
+         `(package
+            (name ,guix-name)
+            (version ,(strip-v-prefix version*))
+            (source
+             (origin
+               (method go-mod-fetch)
+               (uri (go-mod-reference
+                     (source ,(vcs->origin vcs-type vcs-repo-url version*))))
+               (sha256
+                (base32
+                 ;; FIXME: fetch & compute checksum
+                 "0000000000000000000000000000000000000000000000000000"))))
+             (build-system go-mod-build-system)
+             (home-page ,home-page)
+             (synopsis ,synopsis)
+             (description ,description)
+             (license ,license)))
+        (values
+         `(package
+            (name ,guix-name)
+            (version ,(strip-v-prefix version*))
+            (source ,(vcs->origin vcs-type vcs-repo-url version*))
+            (build-system go-build-system)
+         (arguments
+          (list ,@(if (version>? min-go-version (package-version (go-package)))
+                      `(#:go ,(string->number min-go-version))
+                      '())
+                #:import-path ,module-path
+                ,@(if (string=? module-path-sans-suffix root-module-path)
+                      '()
+                      `(#:unpack-path ,root-module-path))))
+         ,@(maybe-propagated-inputs
+            (map (match-lambda
+                   ((name version)
+                    (go-module->guix-package-name name (strip-v-prefix version)))
+                   (name
+                    (go-module->guix-package-name name)))
+                 dependencies))
+         (home-page ,home-page)
+         (synopsis ,synopsis)
+         (description ,description)
+         (license ,license)
+        (if pin-versions?
+            dependencies+versions
+            dependencies))))))
 
 (define go-module->guix-package*
   (lambda args
@@ -699,6 +719,7 @@ (define go-module->guix-package*
 (define* (go-module-recursive-import package-name
                                      #:key (goproxy "https://proxy.golang.org")
                                      version
+                                     go-mod?
                                      pin-versions?)
 
   (recursive-import
@@ -709,6 +730,7 @@ (define* (go-module-recursive-import package-name
       (receive (package-sexp dependencies)
           (go-module->guix-package* name #:goproxy goproxy
                                     #:version version
+                                    #:go-mod? go-mod?
                                     #:pin-versions? pin-versions?)
         (values package-sexp dependencies))))
    #:guix-name go-module->guix-package-name
diff --git a/guix/scripts/import/go.scm b/guix/scripts/import/go.scm
index b90c6ac72f..a12c3a9b2f 100644
--- a/guix/scripts/import/go.scm
+++ b/guix/scripts/import/go.scm
@@ -50,6 +50,8 @@ (define (show-help)
   (display (G_ "
   -h, --help             display this help and exit"))
   (display (G_ "
+  -m, --mod              generate go-module based packages"))
+  (display (G_ "
   -r, --recursive        generate package expressions for all Go modules
                          that are not yet in Guix"))
   (display (G_ "
@@ -65,6 +67,9 @@ (define %options
                  (lambda args
                    (show-help)
                    (exit 0)))
+         (option '(#\m "mod") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'go-mod? #t result)))
          (option '(#\r "recursive") #f #f
                  (lambda (opt name arg result)
                    (alist-cons 'recursive #t result)))
@@ -106,6 +111,7 @@ (define (parse-options)
            (let ((arguments (list name
                                   #:goproxy (assoc-ref opts 'goproxy)
                                   #:version version
+                                  #:go-mod?  (assoc-ref opts 'go-mod?)
                                   #:pin-versions?
                                   (assoc-ref opts 'pin-versions?))))
              (if (assoc-ref opts 'recursive)
-- 
2.39.5

[bug#74373] [PATCH 4/4] gnu: Add go-buf.

[Jørgen Kvalsvik]

* gnu/packages/golang-xyz.scm (go-buf): New variable.

Change-Id: I2ff7ee23bcf5413ad3b4b4b9cbfc8a007b0b9b80
---
 gnu/packages/golang-xyz.scm | 63 ++++++++++++++++++++++++++++---------
 1 file changed, 49 insertions(+), 14 deletions(-)

diff --git a/gnu/packages/golang-xyz.scm b/gnu/packages/golang-xyz.scm
index 0d6debcac4..0d8147e484 100644
--- a/gnu/packages/golang-xyz.scm
+++ b/gnu/packages/golang-xyz.scm
@@ -68,6 +68,7 @@ (define-module (gnu packages golang-xyz)
   #:use-module (guix build-system copy)
   #:use-module (guix gexp)
   #:use-module (guix git-download)
+  #:use-module (guix go-mod-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
   #:use-module (gnu packages)
@@ -7591,31 +7592,65 @@ (define-public glua
       #:import-path "github.com/yuin/gopher-lua/cmd/glua"
       #:unpack-path "github.com/yuin/gopher-lua"))))
 
+(define-public go-buf
+  (package
+    (name "go-buf")
+    (version "1.46.0")
+    (source
+     (origin
+       (method go-mod-fetch)
+       (uri (go-mod-reference
+             (go go-1.23)
+             (source
+              (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/bufbuild/buf")
+                      (commit (string-append "v" version))))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "049rp4f2i759xmws6pqrw3208x1v0zdcmi3ymjbxmaq4hijl9daw"))))))
+       (sha256
+        (base32 "1sk2jhazaa1y2y5m1nmd753s0gr52c8967xr6j9pf2d675fzi7q0"))))
+    (arguments
+     (list
+      #:go go-1.23
+      #:import-path "cmd/buf"))
+    (build-system go-mod-build-system)
+    (home-page "https://github.com/bufbuild/buf")
+    (synopsis "Buf")
+    (description
+     "The @@url{https://buf.build,(code buf)} CLI is the best tool for working with
+@@url{https://protobuf.dev,Protocol Buffers}.  It provides:.")
+    (license license:asl2.0)))
+
 (define-public go-chroma
   (package
     (name "go-chroma")
     (version "2.14.0")
     (source
      (origin
-       (method git-fetch)
-       (uri (git-reference
-             (url "https://github.com/alecthomas/chroma")
-             (commit (string-append "v" version))))
-       (file-name (git-file-name name version))
+       (method go-mod-fetch)
+       (uri (go-mod-reference
+             (source
+              (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/alecthomas/chroma")
+                      (commit (string-append "v" version))))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "1qgr4gywjks869sc85wb8nby612b8wvsa1dwpsbanjsljq7wq7mp"))))))
        (sha256
         (base32
-         "1qgr4gywjks869sc85wb8nby612b8wvsa1dwpsbanjsljq7wq7mp"))))
-    (build-system go-build-system)
+         "1xq2v3qr71mbjm328a6snqvba83nq70z6qqjfgh5mp25bwkwy6bl"))))
+    (build-system go-mod-build-system)
     (arguments
      (list
       #:install-source? #f
-      #:import-path "github.com/alecthomas/chroma/cmd/chroma"))
-    (native-inputs
-     (list go-github-com-alecthomas-assert-v2
-           go-github-com-alecthomas-chroma-v2
-           go-github-com-alecthomas-kong
-           go-github-com-mattn-go-colorable
-           go-github-com-mattn-go-isatty))
+      #:import-path "cmd/chroma"))
     (home-page "https://github.com/alecthomas/chroma")
     (synopsis "General purpose syntax highlighter in pure Golang")
     (description
-- 
2.39.5

[bug#74374] [PATCH 2/4] guix: add go module aware build system

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 117 bytes --]

Hi,

I try to work on this, which issue or problem did you solve
with this variant of go-build-system?

Thanks,
Oleg

[-- Attachment #2: Type: text/html, Size: 278 bytes --]