From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id UC5QHSXiVWFwoQAAgWs5BA (envelope-from ) for ; Thu, 30 Sep 2021 18:13:25 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id UFnPGCXiVWH+KgAAbx9fmQ (envelope-from ) for ; Thu, 30 Sep 2021 16:13:25 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C88D5E5AA for ; Thu, 30 Sep 2021 18:13:24 +0200 (CEST) Received: from localhost ([::1]:52296 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mVygU-0004WD-Sy for larch@yhetil.org; Thu, 30 Sep 2021 12:13:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36914) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mVy1i-0005Kx-UN for guix-devel@gnu.org; Thu, 30 Sep 2021 11:31:15 -0400 Received: from mail-il1-x133.google.com ([2607:f8b0:4864:20::133]:35563) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mVy1g-0002b8-S2 for guix-devel@gnu.org; Thu, 30 Sep 2021 11:31:14 -0400 Received: by mail-il1-x133.google.com with SMTP id x2so7338513ilm.2 for ; Thu, 30 Sep 2021 08:31:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=1Bg5bfthsiAzrf3TS5bHWxTxlMz2QD289CNrSJ+nndc=; b=TKK2CInFbCjcuGI2FqbCIAVFlpaZDVnO9lHZBVu2w6ixOqCMFb/MK98xKVqWiwxN4i ihiG1t/nkuZYBRjfDJF7V2VcEEo28Kau42B4mGsFrkd4KrgBVd9E9/3sOo2kxRMD94QT 7DOPgHPIY1ZMWmPAy28OrpxDzNda1RpihxQdKGsf3vPksOA71jHv8fVun+7n4P4kO4kI pT+fqiT7cPlhJmuJrKVwjh7S0I89xVODRiJ0DilV4zZ234vkEntYiqGEWq1sjKNN/eHP Rnuheqjx6SGOmT5i4T7s1QZ2EhVPimZKL1F5K3XZoflTyB9wY56L9RVMTmE1mzzqzvUG ELrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=1Bg5bfthsiAzrf3TS5bHWxTxlMz2QD289CNrSJ+nndc=; b=pXEyyEwh56hweUyi1Y7uiojrQqDYwN27UoGrxo760lbUCd0cF0S8lD+H4UvoizpRag BSfTlZ/8ibiQsELrwjnOHpttS2DmOjydwmhgm7T22zVd/7tAaZb5tgnrUag9CNqHysPb RYW8FMAXYMFHX+HtzSSchpbbVc/RZ5iKClCZi9NOeuSmK8cG9MLqTTm046Q+3VsaO3O1 bAesPRe6R/jtCdmXVOZbbT/LsyEnw/jpJyxwXXB0LNYaT82Ks6sDDwaSVUP59iJFUGQS U1niMZS5vn/Y7zu4GA6YMh7/WZhitGBmFFkFofa0KfxhEK13RLbIfAf1xVRe2puux0Am X0sw== X-Gm-Message-State: AOAM533nJc84iIVEACm8leugYCswWeatDn9s+nOYvKNFbcIgrm3Z9iq7 POkhOxV4LCMsVbRArb2x2N/DbpmkRwNfXA== X-Google-Smtp-Source: ABdhPJxp+rWytcg7V8LCV57n+KUKelDflJOb0lqwS9tnVi9lpup9fbbONrg2KDDDH3bt8cFzo1kQzw== X-Received: by 2002:a05:6e02:e13:: with SMTP id a19mr4906352ilk.11.1633015870455; Thu, 30 Sep 2021 08:31:10 -0700 (PDT) Received: from washu-v4 (172-221-246-205.res.spectrum.com. [172.221.246.205]) by smtp.gmail.com with ESMTPSA id a5sm2073664ilf.27.2021.09.30.08.31.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Sep 2021 08:31:09 -0700 (PDT) From: Katherine Cox-Buday To: Sarah Morgensen Subject: Could the Go importer use the Go toolchain? (was Re: Go importer and packages with version flags) References: <87a6jwr5kq.fsf@gmail.com> <86bl4au4zm.fsf@mgsn.dev> Date: Thu, 30 Sep 2021 10:31:08 -0500 In-Reply-To: <86bl4au4zm.fsf@mgsn.dev> (Sarah Morgensen's message of "Wed, 29 Sep 2021 20:17:17 -0700") Message-ID: <87fstmcc77.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2607:f8b0:4864:20::133; envelope-from=cox.katherine.e@gmail.com; helo=mail-il1-x133.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1633018405; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=1Bg5bfthsiAzrf3TS5bHWxTxlMz2QD289CNrSJ+nndc=; b=WrZCGV5m83Nw/IzUhhLDwT98SMkpfMoWqDF4sIJ+hX+tAwQD6eYqbH+xYq8Ffq62IAPO2b W2VfktiRBykUD9uiV+1yA9xmfRc9RXpin8i0wX+iNJUKk2Wja64+/ErLzWNwBWf/Mae2yW BBv20LuqGIcm5PnLxwTWMngDZkWOlZWyVbCR4Gm57sB1gWCbwwc7Du4rfiN4dGNm1MxiId yZIhRnCi8KeUfYrUfL20rq4A2Y4VV5i6bvIkACKFiOpUz/3TQR6yeoXLaqoCSmTQZ2p/2B LMTTy4vMDSMEb1NezSyLI+5zXNiW4D14h95FIVAyzpngK0sjIusfkMuvVQ8PlQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1633018405; a=rsa-sha256; cv=none; b=eD87ck16muZuNc4FDYofJ3L8xyyWSIXF3rKjV/66HNwGZh068dG99HzGn/kpxaKlCQlYuJ 6pzQAp0ooQI+YmQ6Xr6cmk5kzWGooUauzPaaMM94bpGMkMUPfKHJ7xSYvtlkhdGN0CYD6i xSHPc3B1eQEWoKOwnCTTGz1WbhUnM9IC4JJ64kEhroGc92DbsqLrWbJIYgBqz4ceB8Pmpx p8MD/MNemDCrwA123Kn03idiXHEnziSqAF5sufpW0E9l9V3OsnxybV29cwPeVTF8OtQB45 /+gTmlTPMPMg0xzRlDjB8KYMlOMS2jgwKsNvXvNp3QlJ0ZKp+zjpL1rLxA+Smg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=TKK2CInF; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -1.70 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=TKK2CInF; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: C88D5E5AA X-Spam-Score: -1.70 X-Migadu-Scanner: scn1.migadu.com X-TUID: m6ia9t/OhDhN Sarah Morgensen writes: >> As an aside, when I started writing the importer, I didn't know it was a >> possibility to just use the Go toolchain to help us generate packages. I >> thought "the Guix way" was to do it all in scheme. It's nice that it's in >> scheme, but I would love to leverage the Go toolchain instead. >> >> IMO, module resolution and graph dependencies in Go are complicated enough >> that I'd much rather take the effort we put in trying to replicate and keep >> up with the Go toolchain's behavior, and spend that effort elsewhere. >> >> Does anyone have opinions on this? > > Hmmm. Setting aside whether or not we want to use external tools in > general... > > If we use the Go tool, we shift the problem domain into "translating between > `go' and Guix." > > For example: when Go downloads a module, it only saves the module, not the > whole repository*, so we can't use that to generate the hash. (* Except it > does if you used GOPROXY=direct, but that's in the module cache, and only a > bare repository.) We could use the ~GOMODCACHE~ environment variable[0] to specify where these files are placed. > Or, the fact that we import the latest available version of packages (unless > --pin-versions is used), but Go uses exact versions selected with MVS [0]. Won't the importer import the import path you specify? The Go toolchain can also specify a ~@latest~ "version" to fetch the latest. > You might also be interested in taking a look at Gentoo dealing with this > translation problem [1]. > > I'm not saying that this would necessarily be a bad tradeoff either, but it's > definitely a tradeoff. Agreed. I think the problem space of managing the translation between the toolchain and Guix is probably both smaller and less volatile. I.e., the toolchain operations that dump dependency graph type stuff tend to support JSON output, and the download bits have options for where stuff gets stored on disk. As you know, the steps to go from an import path to a repository on-disk is complicated. If we could keep that complexity encapsulated upstream, which also happens to be the reference implementation, it could probably make the importer resilient to change and maybe even faster. > Did you have something particular in mind as far as leveraging the Go tooling? I haven't though too much about it, but ~GOPROXY=direct~ was my guess too. Thinking about this, would a shallow fetch into a bare repository be so bad? It made me start wondering why our ~git-download~ type /doesn't/ have this behavior. Are we unnecessarily pulling down the entire commit history with all tags and branches of a large repository? I don't even know what the philosophy is being doing this or not. Isn't the only commitment a Guix package makes for software at a particular version/commit that the store contains that immutable version of the source, not the complete repository? Also, Guix supports all kinds of download types, one of which is zip files. The rust importer generates packages that fetch ~.tar.gz~'s from https://create.io. When I started writing the importer, I specifically kept the packages it generated decoupled from Google's module cache, but to make the comparison: the Go importer could do the same thing as the Rust importer: download zips from https://proxy.golang.org. If this line of reasoning is wrong, it also looks like the toolchain respects standard VCS configuration. I wonder if there's a way to manipulate the fetches so that they're not shallow or bare. TLDR: A shallow fetch into a bare repository might actually be /ideal/! It's small, and still directly connected to the package's actual source. > [0] > > [1] cmd/go: allow extraction of > urls used to download dependencies > > -- > Sarah [0] - https://golang.org/ref/mod#environment-variables [1] - https://golang.org/ref/mod#private-module-repo-auth -- Katherine