From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id SIlyFkK5N2eoagEAe85BDQ:P1 (envelope-from ) for ; Fri, 15 Nov 2024 21:12:34 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id SIlyFkK5N2eoagEAe85BDQ (envelope-from ) for ; Fri, 15 Nov 2024 22:12:34 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=nBk+w72P; dkim=fail ("headers rsa verify failed") header.d=kolabnow.com header.s=dkim20240523 header.b=qu3YTNjn; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1731705154; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=yPzaXCYSdaziVreq7nuo+jL0yuX+QD9obeGjigv5Dl8=; b=GaQRYr4SXtcvW1SAKMaZTffWcwn/CMmS3PoQdarB1cU2ubBis3tPU07HYG1X72MNYo/wvz aUE3XsXhbfkIKAMLoN2c7qTTQUmVI3FF1OJal1cukQ6tteSKsp3QrefuFnSo1zzb+DOH74 XQcS3esZBdEDomV79eNCNtl8dgbtrj6iGznG++Mb8bg/ahbtowq6t42zFZaPgbzpL/riEE XAxrgeJHuvdH1CYLsCr8Oib25UyjY+uDxJhLN2Jx76EZBN6lYKuApNEpgE9GaAVBRHWQQI KCKyn5e4p4hXXIFTs7Us94Ll1W1hyzccU/8eCfYbkqi5d00OFLdahpJ37Gddww== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1731705154; a=rsa-sha256; cv=none; b=ujs0wiSd2tfxxzzDLAhU+HTxb/Fiw9SyL9LuHi/MduCYgP2L784Qf6WuzTNBUvQyxJM0f+ vsw5zJM47wsXdJ3yU66ov1EOMo2stdHx5QugV+bFhxv8q89w9l9vZIjunQfKnlUAg2Kg1U 3C8hnpoYqSRfF3h1DbENZPeHaic9JZvQEsaUJzeQ+U9R4cSKBFQ/NEZQ3oyjBWusHm+cSn 2RGjyfKElNDr1tDusVbU6XOVo8xNvLV/kxCe9aI/l/B77jOMgOVTN/5FvZkFDXbJTgOWlx RLX0N14KYQ7S5occnfKCx/J01qlcJJYHpaiXtmCTYv/V9TZUuHR9rfMSidqyrg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=nBk+w72P; dkim=fail ("headers rsa verify failed") header.d=kolabnow.com header.s=dkim20240523 header.b=qu3YTNjn; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=none Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DDD3579012 for ; Fri, 15 Nov 2024 22:12:33 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tC3bp-0002E2-4a; Fri, 15 Nov 2024 16:12:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bn-0002Do-VC for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:03 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tC3bn-00069Y-NI for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:References:In-Reply-To:Date:From:To:Subject; bh=yPzaXCYSdaziVreq7nuo+jL0yuX+QD9obeGjigv5Dl8=; b=nBk+w72PtVN7TSuWhj8DqPmjrf3cc+qj/9Cif+dRMp6iwurtH+3wWagK0zKppENMhKrAhST7XmntuX9H8bNwc4ndczfJKmAQR7CTEoY4vVa5dNH9ap8Ntf3o2MMLiH1fXCtDrP1bNus8SzWRr8SuiEtffqqTtykUb0jrG9scg1jM6e2GTtLy+qpdmd3ZMHKMQf0pn4J2N4k7HpYUeTih2Bel/Pv+LTg2pf2XIz0Z8i0Geo79k1ZUlnDyP8cC5fOMculSfWKjiuCNtJLC0xxSZ3WqSUhtv4OTEQwycsAmJlYvKarqNQAyorCQ34eCgAnhkwSX1u5Vz+GfMHrOhdkzUQ==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1tC3bn-0000GM-HI for guix-patches@gnu.org; Fri, 15 Nov 2024 16:12:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#74372] [PATCH 1/4] guix: Add go module fetcher Resent-From: =?UTF-8?Q?J=C3=B8rgen?= Kvalsvik Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 15 Nov 2024 21:12:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 74372 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 74372@debbugs.gnu.org Cc: =?UTF-8?Q?J=C3=B8rgen?= Kvalsvik X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.1731705105925 (code B ref -1); Fri, 15 Nov 2024 21:12:03 +0000 Received: (at submit) by debbugs.gnu.org; 15 Nov 2024 21:11:45 +0000 Received: from localhost ([127.0.0.1]:51077 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tC3bT-0000Eb-OG for submit@debbugs.gnu.org; Fri, 15 Nov 2024 16:11:44 -0500 Received: from lists.gnu.org ([209.51.188.17]:48708) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1tC3bQ-0000Dz-6V for submit@debbugs.gnu.org; Fri, 15 Nov 2024 16:11:41 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bN-0002Bt-Ui for guix-patches@gnu.org; Fri, 15 Nov 2024 16:11:37 -0500 Received: from mx.kolabnow.com ([212.103.80.155]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tC3bE-00066P-1c for guix-patches@gnu.org; Fri, 15 Nov 2024 16:11:32 -0500 Received: from localhost (unknown [127.0.0.1]) by mx.kolabnow.com (Postfix) with ESMTP id E0B863069A9E for ; Fri, 15 Nov 2024 22:11:18 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kolabnow.com; h= content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:date:subject:subject:from:from:received :received:received; s=dkim20240523; t=1731705075; x=1733519476; bh=yPzaXCYSdaziVreq7nuo+jL0yuX+QD9obeGjigv5Dl8=; b=qu3YTNjntF4G 6zNBfOomZ50I2sPOO87FGl6ELIQCWzksH1Ed3GElueeX8/1/PtBiNicqyrQt1U0o 3jPsSse2n5j6Hf6AKiKWkTwG1kuNHfjtxXYj6BTaTM0o5tffzSb/PDS308UAXJKA XZtrzQBj/fn3M+3dQeOSKgYkOgQkYI9G41MpRTsbgAlbFUbKTp/1dPikN6L3ilZz U90nz4A8kfELN6nrIrY98DywtrBkx6ZNwn8PnrkjGehZrDreXrfc7x/CYxL3qAEB 9fwvL3MIr5RTKa5+1JATlpm/Z+V1WN7sI055GQ7PVHQ7CJdQN7F5W0vAl2XxQM6v aVEyLkYE6Q== X-Virus-Scanned: amavis at mykolab.com Received: from mx.kolabnow.com ([127.0.0.1]) by localhost (ext-mx-out013.mykolab.com [127.0.0.1]) (amavis, port 10024) with ESMTP id ZWUaZ5VU-YZZ for ; Fri, 15 Nov 2024 22:11:15 +0100 (CET) Received: from int-mx011.mykolab.com (unknown [10.9.13.11]) by mx.kolabnow.com (Postfix) with ESMTPS id 20A6B3069A98 for ; Fri, 15 Nov 2024 22:11:14 +0100 (CET) Received: from ext-subm010.mykolab.com (unknown [10.9.6.10]) by int-mx011.mykolab.com (Postfix) with ESMTPS id E5A1D30EDB4A for ; Fri, 15 Nov 2024 22:11:14 +0100 (CET) From: =?UTF-8?Q?J=C3=B8rgen?= Kvalsvik Date: Fri, 15 Nov 2024 22:11:03 +0100 Message-Id: <20241115211106.2759121-2-j@lambda.is> In-Reply-To: <20241115211106.2759121-1-j@lambda.is> References: <20241115211106.2759121-1-j@lambda.is> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=212.103.80.155; envelope-from=j@lambda.is; helo=mx.kolabnow.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -3.79 X-Spam-Score: -3.79 X-Migadu-Queue-Id: DDD3579012 X-Migadu-Scanner: mx13.migadu.com X-TUID: BnugPjFubRxG Add a new fetcher for go programs, based on go mod download. This is largely how go programs *want* to be fetched, and some pseudo-private dependencies [1] are difficult to wrangle without letting go do the work. This patch only adds support for git as the root module fetch. The approach is conceptually simple: 1. Fetch the root package to build (usually a git repo) 2. Find all the go.mod files in this tree and run go mod download 3. Store the sources of the package, its direct- and transitive dependencies as the source. The source is in source/, the dependencies in go/pkg as encouraged by the toolchain. Go tooling is generally very opinionated on how things are supposed to be done, and fighting it usually brings pain. This approach is not without drawbacks. Go programs tend to be very liberal with dependencies and specific with versions, which will in practice leads to a **large** set of sources. At a system level there will be a lot of duplicated sources, and a lot of slightly different versions which could be compatible. This is not guix specific but rather how go programs design their environments. Another is libraries. Go wants to statically link everything, and generally likes to work with source, and every program pins dependencies to different revisions. The go-mod-fetch getting the source of all libraries the libraries breaks the general packaing approach of packaging libraries separately, but this was never really leveraged by go anyway. This switches the focus to packaging applications rather than libraries. Finally, the package.source needs two hashes - one for the direct origin (e.g. git), and one for the sum of libraries fetched by go.mod. [1] github.com/bufbuild/buf 1.46 requires buf.build/gen/go/bufbuild/bufplugin/protocolbuffers/go v1.35.1-20241031151143-70f632351282.1 which is only available as a .zip and has a layout that go mod understands, but is alien to the go-build-system. * guix/go-mod-download.scm: New file. * gnu/local.mk: Register it. Change-Id: I84c00df07393a9978124667e3e2497aec7009252 --- Makefile.am | 1 + guix/go-mod-download.scm | 146 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 guix/go-mod-download.scm diff --git a/Makefile.am b/Makefile.am index 3a35b7becd..fc00947f4f 100644 --- a/Makefile.am +++ b/Makefile.am @@ -102,6 +102,7 @@ MODULES = \ guix/android-repo-download.scm \ guix/bzr-download.scm \ guix/git-download.scm \ + guix/go-mod-download.scm \ guix/hg-download.scm \ guix/hash.scm \ guix/swh.scm \ diff --git a/guix/go-mod-download.scm b/guix/go-mod-download.scm new file mode 100644 index 0000000000..1a7ffbb6ac --- /dev/null +++ b/guix/go-mod-download.scm @@ -0,0 +1,146 @@ +;;; GNU Guix --- Functional package management for GNU +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +;;; TODOs: +;;; 1. Support non-git root repositories +;;; 2. Store/cache individual module downloads + +(define-module (guix go-mod-download) + #:use-module (guix build utils) + #:use-module (guix derivations) + #:use-module (guix packages) + #:use-module (guix gexp) + #:use-module (guix modules) + #:use-module (guix monads) + #:use-module (guix records) + #:use-module (guix store) + #:use-module (guix git-download) + + #:export (go-mod-reference + go-mod-reference? + go-mod-source + go-mod-go + + go-mod-fetch)) + +(define (go-package) + "Return the default Go package." + (let ((distro (resolve-interface '(gnu packages golang)))) + (module-ref distro 'go))) + +(define (nss-certs-package) + "Return the default nss-certs package." + (let ((distro (resolve-interface '(gnu packages certs)))) + (module-ref distro 'nss-certs))) + +;; The source key accepts the same kinds as the package record, but mostly go +;; use git. The go key specifies which go version to use, which might be +;; necessary if any module sets a newer toolchain in go.mod +(define-record-type* + go-mod-reference make-go-mod-reference + go-mod-reference? + (source go-mod-source) + (go go-mod-go (default (go-package)) (thunked))) + +;; Fetch all direct and indirect dependencies of the go modules in the source +;; tree (usually a git repo) using go mod download. +(define* (go-mod-fetch source hash-algo hash + #:optional name + #:key (system (%current-system)) + (guile (default-guile)) + (go (go-package)) + (nss-certs (nss-certs-package))) + (define guile-json + (module-ref (resolve-interface '(gnu packages guile)) 'guile-json-4)) + + (define* (build source go) + (with-imported-modules (source-module-closure + '((guix build utils) + (guix build download) + (srfi srfi-34))) + (with-extensions (list guile-json) + #~(begin + (use-modules + (guix build download) + (guix build utils) + (srfi srfi-34)) + (let* ((cert-dir (string-append #$nss-certs "/etc/ssl/certs")) + (src-dir (string-append #$output "/source")) + (mod-cache (string-append #$output "/go/pkg")) + (xgo (string-append #$go "/bin/go"))) + ;; go.mod files can specify a minimum required toolchain which could + ;; cause go mod download to fetch and install a newer compiler + ;; if invoked with an older one. + (setenv "GOTOOLCHAIN" (string-append "go" (getenv "go toolchain"))) + (setenv "SSL_CERT_DIR" cert-dir) + (setenv "GOCACHE" "/homeless-shelter") + (setenv "GOPATH" "/homeless-shelter") + (setenv "GOMODCACHE" mod-cache) + + (mkdir-p src-dir) + (mkdir-p mod-cache) + (copy-recursively #$source src-dir) + (let* ((go-mods (find-files src-dir "go.mod"))) + ;; go mod will update the go.mod with transitive dependencies if + ;; they are not set, and fail with an error if the file is not + ;; writable. + (for-each make-file-writable go-mods) + (with-throw-handler + #t + (lambda _ + (for-each + (lambda (go-mod) + (with-directory-excursion (dirname go-mod) + ;; go mod download must be run twice - the first + ;; fetches direct dependencies and *records* + ;; transitive dependencies, the second run fetches + ;; the transitive dependencies. + (and + (invoke xgo "mod" "download") + (invoke xgo "mod" "download")))) + go-mods) + #t) + (lambda (key . args) + (display (string-append "Fetching modules failed.\n" + "Here are the results of `go env`:\n")) + (invoke xgo "env"))))))))) + + (let* ((mod-source (go-mod-source source)) + (mod-ref (origin-uri mod-source)) + (mod-go (go-mod-go source)) + (mod-hash (origin-hash mod-source)) + (mod-hash-value (content-hash-value mod-hash)) + (mod-hash-algo (content-hash-algorithm mod-hash))) + + (mlet* %store-monad ((guile-for-build (package->derivation guile system)) + (git-source (git-fetch mod-ref mod-hash-algo + mod-hash-value + #:system system + #:guile guile-for-build))) + (gexp->derivation (or name "go-mod-fetch") (build git-source mod-go) + #:script-name "go-mod-fetch" + #:env-vars + `(("go toolchain" . ,(package-version mod-go))) + #:leaked-env-vars '("http_proxy" "https_proxy" + "LC_ALL" "LC_MESSAGES" "LANG" + "COLUMNS") + #:system system + #:local-build? #t ;don't offload repo cloning + #:recursive? #t + #:hash-algo hash-algo + #:hash hash + #:guile-for-build guile-for-build)))) -- 2.39.5