From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id mBZcOQmoMmGMmgAAgWs5BA (envelope-from ) for ; Sat, 04 Sep 2021 00:56:09 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id eDNINAmoMmGqIwAAB5/wlQ (envelope-from ) for ; Fri, 03 Sep 2021 22:56:09 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6AAB1A6B5 for ; Sat, 4 Sep 2021 00:56:09 +0200 (CEST) Received: from localhost ([::1]:34674 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mMI6S-0007tC-Hd for larch@yhetil.org; Fri, 03 Sep 2021 18:56:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43176) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mMI6M-0007se-NK for guix-patches@gnu.org; Fri, 03 Sep 2021 18:56:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:34561) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mMI6M-0006En-Fr for guix-patches@gnu.org; Fri, 03 Sep 2021 18:56:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mMI6M-0003V8-DZ for guix-patches@gnu.org; Fri, 03 Sep 2021 18:56:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50227] [PATCH 0/3] go-build-system and GOPATH improvements References: <20210827151052.12611-1-marius@gnu.org> Resent-From: Sarah Morgensen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 03 Sep 2021 22:56:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50227 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Marius Bakke Cc: 50227@debbugs.gnu.org Received: via spool by 50227-submit@debbugs.gnu.org id=B50227.163070975513446 (code B ref 50227); Fri, 03 Sep 2021 22:56:02 +0000 Received: (at 50227) by debbugs.gnu.org; 3 Sep 2021 22:55:55 +0000 Received: from localhost ([127.0.0.1]:46107 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMI6F-0003Uo-9C for submit@debbugs.gnu.org; Fri, 03 Sep 2021 18:55:55 -0400 Received: from out0.migadu.com ([94.23.1.103]:43780) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMI6C-0003Ud-CU for 50227@debbugs.gnu.org; Fri, 03 Sep 2021 18:55:54 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1630709750; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=XWCoaPMnbcZYjNZS03cTEU1hrEilr0QN+AHhO0CqNh8=; b=M115jdTbvC26eCIs8JS7KfSdit2G8QYd+dPiXQFOvTW2AdhZAQb/icD88pursql21pNTU5 oon75Ldi7XODNRzxKlFQnSjkQxQpOlR6rbdi/KwzGF989QntEvnwzH/va71asJx+fwABbc CgM/ZNt9zjemkhAQocNIqxUyyI7cY4g= From: Sarah Morgensen Date: Fri, 03 Sep 2021 15:55:48 -0700 In-Reply-To: Marius Bakke's message of "Sat, 28 Aug 2021 16:52:31 +0200 (15 hours, 15 minutes, 44 seconds ago)" Message-ID: <86ilzhxo97.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: text/plain X-Migadu-Auth-User: iskarian@mgsn.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1630709769; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=XWCoaPMnbcZYjNZS03cTEU1hrEilr0QN+AHhO0CqNh8=; b=oZyxB3RkCfKXtSsMk2KlYAWU8YFCEB78TPjk9Vw4T8qSDR70yjnuJSka7UhKB0dbq6VcMg QYudu1b+Bto93CyjBJx+5phJThVSigW0p/kYR6z2+dP/qd5m5Gn8xeoJFZL9UNdYnxDdlr eJtlrJXyyR2+x/RqvmRca0iEU0Ykbw6Se7+6cWLtYwJbWfHMpGh3zLg43u2K7pgoiTkxjI Uv3Jqg2JNwGcFSU/BAbQ9/ztZsxAul9Mf1NGGY9VimaTl2hIM8bXyWCuJsH3E+mPwyFimM pA6n6m1kEhc2Lo/epcMkRezVxozDMvo+6GubkaoiOFxHnHkXKknU9Sh+YlPdjQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1630709769; a=rsa-sha256; cv=none; b=cfssRWFLSXHKYPbZEB59i9ldR+TP4kcs0agnaRrs5k9HuciIDm//bmlY5Zky6g6KpINTyD G8GA8+Q4bR1as6/LUkleTV/FcTWc9DYYu7VK5fAlFFxVPkDLFsMVGwfoGLHReuZyQaaEPD I9HzrB6oopq7fHln9tyUDmfTRXnnMBFrpawNQ4lLVXH1esHYKbt4n5WIKb2/Fglly9FQNM d1uA81S57IBY9q1+GS08JYsd1+mMfPLA0wP/IF0p4cfIfK3vn7eOtF/TYIHNQQmkRfs5ui qx9IK3Bt1ElZtAlts0w9OrVVOxGk9QVTPJ+Ib3eRmW9ZNJcXpiXdtucd/JqxvA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=mgsn.dev header.s=key1 header.b=M115jdTb; dmarc=fail reason="SPF not aligned (relaxed)" header.from=mgsn.dev (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.32 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=mgsn.dev header.s=key1 header.b=M115jdTb; dmarc=fail reason="SPF not aligned (relaxed)" header.from=mgsn.dev (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 6AAB1A6B5 X-Spam-Score: -1.32 X-Migadu-Scanner: scn1.migadu.com X-TUID: NSjj1OuwdXPJ Hello, Marius Bakke writes: > That is excellent news, thank you! I briefly looked into "modules" > while researching this, but realized that the rabbit hole would be too > deep for me. I did not notice that GOPATH was being deprecated however. The Go 1.16 release notes actually stated that a deprecation notice would be printed in Go 1.17 but then they didn't do that! I think they're waiting for the "workspaces" feature [0] to be implemented first. [0] https://github.com/golang/go/issues/45713 >> 4. Use -trimpath instead of remove-go-references, as you did. Also, to >> avoid rebuilding the standard library with '-trimpath' for every package >> (since the Go build cache does not persist between build environments): >> >> a) modify the Go package to build standard libraries with -trimpath, >> which would unfortunately mean most users of the Go package would >> find that ~180MB of space wasted; or >> >> b) build a '-trimpath' version of the standard library separately and >> use it with '-pkgdir' (which would prevent #3 from working) or by >> building a directory union of Go and Go-std-library-with-trimpath >> and setting GOROOT=/path/to/union. >> >> Personally, I'm partial to a), along with removing the pre-compiled >> standard library from the Go package since it ends up recompiled more >> often than not, is very fast to recompile, and it will eventually no >> longer be distributed or used by Go [0]. > > Removing the compiled libraries sounds fine to me. I suppose we'll > still need -trimpath for executables ("main.go")? Yes, we'll need '-trimpath' in all invocations of build tools, including 'test' as well, or it will actually recompile everything without '-trimpath' and then test that! >> (A digression: the current issue with fully implementing module-aware >> mode is that Go really wants a specific version for each dependency. If >> we just populate the module cache with the versions we have, it will >> inevitable complain when a package we try to build wants a version we do >> not have. I see a few solutions: >> >> 1. Put all dependencies in the module cache, and rewrite the main >> module's go.mod (that is, add replace directives) to replace all >> dependencies with the versions we have. >> >> 2. Rewrite the go.mod to replace all dependencies with the local >> filesystem path of the versions we have. >> >> 3. Put all dependencies in the vendor/ directory, and use -mod=vendor. >> Any pre-existing vendor directory must be handled properly. >> >> These three solutions fail to allow re-using the build cache (and >> therefore build artifacts), because Go computes the build cache keys >> differently for main and non-main modules. Building in Go is generally >> fast, so we probably shouldn't compromise much to enable reusing the >> build cache, but a few ideas for doing so: >> >> 4. Set up a dummy go.mod out of the source tree, which 'replace's all >> dependencies AND the module we're building like in 1) or 2). This may >> have to account for replace directives in the go.mod of the module we're >> building, though. >> >> 5. Put the module we're building in the module cache, and build it with >> "go install module@version". The same caveat as in 4) applies, as well >> as that "go install module@version" only works for main packages (that >> is, packages which produce an executable).) > > Thank you for this analysis. The vendoring option is compelling, if it > does not require patching the go.mod files, and can work also for > packages where unbundling is not feasible (or downstream channels with > less strict packaging policies). Yes, and it has the upside that '-mod=vendor' automatically disables network access for most tools. On the other hand, I think the only solution that properly moves in the direction of allowing hacking on Go packages is the GOPROXY search-path approach. This requires some more thought, for sure. > For reusing build artifacts, perhaps we can piggy-back on whatever is > implemented for Bazel as mentioned in [0]. Bazel is an entire build system replacing cmd/go, which (presumably) constructs its own build graph, etc, so probably a little heavy-handed for our use-case. We probably want to stay as close as possible to vanilla tooling. Probably the best we can do is saving and re-using the build cache, since they seem insistent on moving everything into the cache. > Given the conflicting work here, what do you think we should do? I'm > happy to scrap this PR as it was largely an exercise to learn > go-build-system, in addition to scratching a very minor itch. > > Is the reduced complexity worth it while waiting for the gomodules > rewrite, and if so, are there parts that can be merged with your work > such as using $out/share/go? > > Let me know if I can be of assistance. :-) > >> [0] https://github.com/golang/go/issues/47257 I think I would suggest breaking up the Go build system changes into: 1. Make the changes roughly included in your patch, along with making a "go-std-cache-for-build" package (hidden?) which will be an implicit input in the Go build system (perhaps non-substitutable? it will be faster to build than download on nearly all machines), and seeding the build cache with it in 'setup-go-environment'. We skip re-using build artifacts. 2. Update Go build system to use Go 1.16, leaving only docker with Go 1.14 (via "#:go ,go-1.14"). We should be ready to do this as soon as [0] is fixed [1]. 3. Enable building multiple Go packages in one Guix package, and merge all Guix packages such that one Guix package == one module path. 4. Make gomodules changes. 5. Release Go 1.18, which will bootstrap from Go 1.16 or Gccgo 11 [2]. 6. Update Go build system to use Go 1.18. [0] https://issues.guix.gnu.org/49921 [1] https://logs.guix.gnu.org/guix/2021-08-11.log#213401 [2] build: Adopt Go 11.6 as bootstrap toolchain for Go 1.18 -- Sarah