From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 20a0CuCcKWF5bAAAgWs5BA (envelope-from ) for ; Sat, 28 Aug 2021 04:18:08 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id CJh9BeCcKWGffgAA1q6Kng (envelope-from ) for ; Sat, 28 Aug 2021 02:18:08 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 71A4822EC6 for ; Sat, 28 Aug 2021 04:18:07 +0200 (CEST) Received: from localhost ([::1]:52736 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mJnv4-000442-HX for larch@yhetil.org; Fri, 27 Aug 2021 22:18:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33400) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mJnv0-00043t-IN for guix-patches@gnu.org; Fri, 27 Aug 2021 22:18:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:41681) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mJnv0-0005Pb-AD for guix-patches@gnu.org; Fri, 27 Aug 2021 22:18:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mJnuz-0002Yi-UQ for guix-patches@gnu.org; Fri, 27 Aug 2021 22:18:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50227] [PATCH 0/3] go-build-system and GOPATH improvements References: <20210827151052.12611-1-marius@gnu.org> Resent-From: Sarah Morgensen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 28 Aug 2021 02:18:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50227 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Marius Bakke Cc: 50227@debbugs.gnu.org Received: via spool by 50227-submit@debbugs.gnu.org id=B50227.16301170269772 (code B ref 50227); Sat, 28 Aug 2021 02:18:01 +0000 Received: (at 50227) by debbugs.gnu.org; 28 Aug 2021 02:17:06 +0000 Received: from localhost ([127.0.0.1]:53227 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mJnu1-0002XR-R1 for submit@debbugs.gnu.org; Fri, 27 Aug 2021 22:17:06 -0400 Received: from out2.migadu.com ([188.165.223.204]:40815) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mJntu-0002Ww-SZ for 50227@debbugs.gnu.org; Fri, 27 Aug 2021 22:16:59 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1630117012; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=L5AvilHbeKCDQ6z9WNs9BPq76LvRyHPDTiqFawpXbBs=; b=LtXwburSBX7w7QLOentYrNNMAvZFI8swLhtT4Z7ia+e4Vrt5eno9AF8YdPHLzinhghVrB+ 6l3sGYHb38JfLXdH4tj6/k17LmUs1P2vOn69p8BugRgDKesAZm74uCrZFK7EsfFXf/6Wg0 7I35nfNgqwoKtR0JDbT9u89A6BUOt0U= From: Sarah Morgensen Date: Fri, 27 Aug 2021 19:16:49 -0700 In-Reply-To: Marius Bakke's message of "Fri, 27 Aug 2021 17:10:52 +0200 (9 hours, 41 minutes, 46 seconds ago)" Message-ID: <86mtp248ku.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: text/plain X-Migadu-Auth-User: iskarian@mgsn.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1630117087; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=L5AvilHbeKCDQ6z9WNs9BPq76LvRyHPDTiqFawpXbBs=; b=XreTKx4cjs6scPxYSGthxHNpT/GfFEk3UxCNtu8F6C9GN0lZSk8HQvXUJ9TgB49TYg0ESp QhupFB0lq7Y+VphSy5A3ymkGAYjSOHu9OpZWWwOHmfXEFVDY/W0uAW8PnHUcUn4pouxJZM RB5fYVEWokUOTPWfkPyPQyfZjxcire0wCip2OJXxHNrE4R+ONksQSISM7b2f5kYHMFyRTa pOO8YTQOoWFeGWp7DGVJlwMXRgW3e3MKnS16pZtmU/v99CFK4n8eW8utZblO1TS3gTNDId cIqzE03H894pBjlo65bw/cQYx9y9FfFV75Qg1MhG0KlXH4DGF/IC23AjcL2m2w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1630117087; a=rsa-sha256; cv=none; b=TPfuoH72O7oLsSx47SkuPD7Q1QMWHSQfFkLh4/Q368HzKV4AwHRT84RbCwpKw7ITZ19TAD 33ON830hVb9XUJ/Z5RIeywH5FxIbD/nMNR5aoEsQAYxQae/8qg7j7YuOx6eK0RVcpoyVft no4jsCmoXuEaxqz12N/vub0otSOidViSGmGtkCU6l94zaRlMBy8zh7Fh/wAD5cLzAvYKxh 24ZC4aIa6WXx8L11EBWyaTokI/95c98+AOqKeEiXhBGvoJvjSOLO41aQTLs/zVoRqS7uEh A8WlVVSFXGt/OomwRjY1TXNvForSvLbFuUqPfGq2c7APv6ugcYuxPDyQk87NrA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=mgsn.dev header.s=key1 header.b=LtXwburS; dmarc=fail reason="SPF not aligned (relaxed)" header.from=mgsn.dev (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: 0.17 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=mgsn.dev header.s=key1 header.b=LtXwburS; dmarc=fail reason="SPF not aligned (relaxed)" header.from=mgsn.dev (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 71A4822EC6 X-Spam-Score: 0.17 X-Migadu-Scanner: scn0.migadu.com X-TUID: 4SYGAftiJBss Hello Marius, (Apologies in advance for the length of this treatise! I did not have the time to be concise.) Marius Bakke writes: > These patches adjust the Go build system to use Guix's regular > native-search-paths mechanism instead of ad-hoc GOPATH trickery. I have been working on overhauling the Go build system behind the scenes; I expected to have a patch ready last week, but I fell down the "modules" rabbit hole after learning GOPATH is expected to be deprecated as soon as 1.18. Sorry for the duplicated work! (I also have a Go 1.17 ready to launch, but I've been attempting to nail down Go build system changes first so I didn't introduce anything incompatible.) In any case, I hadn't thought of using search paths, that's quite clever! I like it. Before falling down the "modules" rabbit-hole, here is what was working for my GOPATH-based Go build system: 1. Install source in "out/share/go/src" rather than "out/src", and then simply create a directory union of "/share/go/src" from inputs. This avoids accidentally including non-go packages with a "/src" directory. If you did this, then you could make the GOPATH search path be "/share/go/..". 2. Split the GOPATH as you have done (except with only two components; one for the package we're building and one for the union in 1). 3. Reuse build artifacts by copying the $GOPATH[0]/pkg directory to "out/share/go/pkg" in the install phase. They will be transparently used since they will be in GOPATH. (You can use "out/lib/go/pkg", but you must set 'strip-directories' to avoid stripping Go archives, and then include "/lib/go/.." in the GOPATH search path.) However, "go install" will eventually deprecate installing archives [0], perhaps even before GOPATH is deprecated. 4. Use -trimpath instead of remove-go-references, as you did. Also, to avoid rebuilding the standard library with '-trimpath' for every package (since the Go build cache does not persist between build environments): a) modify the Go package to build standard libraries with -trimpath, which would unfortunately mean most users of the Go package would find that ~180MB of space wasted; or b) build a '-trimpath' version of the standard library separately and use it with '-pkgdir' (which would prevent #3 from working) or by building a directory union of Go and Go-std-library-with-trimpath and setting GOROOT=/path/to/union. Personally, I'm partial to a), along with removing the pre-compiled standard library from the Go package since it ends up recompiled more often than not, is very fast to recompile, and it will eventually no longer be distributed or used by Go [0]. > > The context is that I needed to hack on a Go package, and was somewhat > surprised that my usual workflow of "guix environment PKG" did not work. > > It still does not work "out of the box", but these patches bring it a > step further. Now "all" that is needed is to: > > $ cd ~/src/go-foo > $ guix environment go-example-com-foo > $ MYGOPATH="$HOME/tmp/go" > $ NAMESPACE="$MYGOPATH/src/example.com/foo" > $ mkdir -p $(dirname $NAMESPACE) > $ ln -s $PWD $NAMESPACE # or git worktree add $NAMESPACE > $ export GOPATH="$MYGOPATH:$GOPATH" > $ go build # no 'go get' necessary! Interesting. I hadn't thought of the use-case for actually hacking on go packages like this! I'll have to think of how modules-mode can be made to work with this. (A digression: the current issue with fully implementing module-aware mode is that Go really wants a specific version for each dependency. If we just populate the module cache with the versions we have, it will inevitable complain when a package we try to build wants a version we do not have. I see a few solutions: 1. Put all dependencies in the module cache, and rewrite the main module's go.mod (that is, add replace directives) to replace all dependencies with the versions we have. 2. Rewrite the go.mod to replace all dependencies with the local filesystem path of the versions we have. 3. Put all dependencies in the vendor/ directory, and use -mod=vendor. Any pre-existing vendor directory must be handled properly. These three solutions fail to allow re-using the build cache (and therefore build artifacts), because Go computes the build cache keys differently for main and non-main modules. Building in Go is generally fast, so we probably shouldn't compromise much to enable reusing the build cache, but a few ideas for doing so: 4. Set up a dummy go.mod out of the source tree, which 'replace's all dependencies AND the module we're building like in 1) or 2). This may have to account for replace directives in the go.mod of the module we're building, though. 5. Put the module we're building in the module cache, and build it with "go install module@version". The same caveat as in 4) applies, as well as that "go install module@version" only works for main packages (that is, packages which produce an executable).) > > I don't know how feasible it is to avoid making a local directory and > symlinking the project to the expected namespace. Still a complete Go > newbie, but this approach feels more natural and idiomatic Guix-wise. My intuition is that if you're working in GOPATH-mode, you already have a ~/go/src directory or similar, and your project is probably under ~/go/src/my-project. Then, in order to hack on it Guix-like, you would $ cd ~/go/src/my-project $ guix environment go-github-com-me-my-project $ export GOPATH=~/go:$GOPATH $ go build I'm not sure what a similar idiom for Guix-like hacking in module-aware mode would be; we'd have to set GOMODCACHE or something, but it would be very easy for Go to overwrite (or fail to overwrite) things without GOPROXY=off. Alternatively, if we make a "full" go proxy directory layout, we can do GOPROXY=file://path/to/gomodcache or even a search path like GOPROXY=file:///gnu/store/p1/gomodcache|file://gnu/store/p2/gomodcache though I'm not sure how well that would scale w.r.t. number of packages. Both of these GOPROXY methods have the advantage over setting GOMODCACHE that the user could modify GOPROXY to include the default proxy, and would still be able to get packages and versions not packaged by Guix. I suppose there's no reason we couldn't set both GOPATH and e.g. GOPROXY. [0] https://github.com/golang/go/issues/47257 -- Sarah