From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id m9o0NVQsVWG0ogAAgWs5BA (envelope-from ) for ; Thu, 30 Sep 2021 05:17:40 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id +FlkMFQsVWGGXgAA1q6Kng (envelope-from ) for ; Thu, 30 Sep 2021 03:17:40 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id EA16911571 for ; Thu, 30 Sep 2021 05:17:39 +0200 (CEST) Received: from localhost ([::1]:43642 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mVmZn-0003dU-4Y for larch@yhetil.org; Wed, 29 Sep 2021 23:17:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58468) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mVmZc-0003dI-Pl for guix-devel@gnu.org; Wed, 29 Sep 2021 23:17:28 -0400 Received: from out0.migadu.com ([2001:41d0:2:267::]:36532) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mVmZW-0004ct-An for guix-devel@gnu.org; Wed, 29 Sep 2021 23:17:28 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1632971839; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=puLkrume9PCiuON2SODz+y3LnWAyX3d/y4cDPI5jUU4=; b=e5mQgW13zuJthReNCMa8o6J6j2v3gVw3L3kh0cMI7W9nG92KHebPAZxZaw/+DxuF3eRNJy FykhY6dgZDt8yqCilYFHtC77srH7CRrgFLxnKDbC4+a4UKQLXtZ8C3vfUpKx6E7x9zIwMX BiYPvmJxE1jmu+UoBA/cdDk9SF4nwY4= From: Sarah Morgensen To: Katherine Cox-Buday , Jack Hill Subject: Re: Go importer and packages with version flags In-Reply-To: <87a6jwr5kq.fsf@gmail.com> (message from Katherine Cox-Buday on Tue, 28 Sep 2021 12:08:21 -0500) Date: Wed, 29 Sep 2021 20:17:17 -0700 Message-ID: <86bl4au4zm.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2001:41d0:2:267::; envelope-from=iskarian@mgsn.dev; helo=out0.migadu.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1632971860; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=puLkrume9PCiuON2SODz+y3LnWAyX3d/y4cDPI5jUU4=; b=g8TZdx9vlixkZhgV6YkaXKR9XJSbfSgxUpWAFL5qmKE6CuBHLHKSvNjDBiMEaI9QjeIION bzlu4au4dplPLhd8f46PmlvMk5RFkA8FNwGhqnuIryYWUqQHbJN4Q1FY1424/+VHIOMO7g g/3lFv6hY+BPL8eVvYz+/T6xw5H9tpemMdYCjw/3q19egE6DnTK+v6jQ12aylYl0YrV5GG 4rtbYRBEnUwZZv2qiNFc6kFNgHEysEWOARuvbOYbD4OIdWz8IYuTji6oUBLIsXAjgAZz5n QuQfxOiQXDxDRAbOpTY5d3qn2fR78yVj3qEy+yPEsvk3PzleduElI818v6cm8Q== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1632971860; a=rsa-sha256; cv=none; b=L3fP989odxmIdb1Cxlb4BxMnaqvR4wpGASYarH/4ngmoZNs+9uvgcsek64WyX9uCVxFo7R 8Zp62DKWykb5pVd2lFdOHOiLIMBqYs1Qfllt+avVkLPKj33Njdwa2fc5vNW3V2lVgILKZG UVaDwqWfEWQo6MbeQH00c/eiCfMD3zEQmO+q6c9/Io56d+jHJqcdcSiK1MoVY5wmMFCoQH xfwzkU7WRBV7Rmjfqf5wYTAvWT+8OMjKh6JMbYI5iU15fphCOUt+BNHLQoBpRA3vGS0QoX Js5ikC8YklS7+kMDXGhTjcZm26N2bpU2w+p/j95srw/qbYP/13Tyso24IL7JBg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=mgsn.dev header.s=key1 header.b=e5mQgW13; dmarc=pass (policy=none) header.from=mgsn.dev; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.10 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=mgsn.dev header.s=key1 header.b=e5mQgW13; dmarc=pass (policy=none) header.from=mgsn.dev; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: EA16911571 X-Spam-Score: -3.10 X-Migadu-Scanner: scn0.migadu.com X-TUID: gGnR/Ustd5Ch Katherine Cox-Buday writes: > Sarah Morgensen writes: > >> Hi Katherine, Jack, >> >> Katherine Cox-Buday writes: >> >>> Jack Hill writes: >>> >>>> Hi Guix, >>> >>> Hey, Jack, a few thoughts. >>> >>>> While I was was working with the go importer today, it suggested I package >>>> go-github-com-russross-blackfriday-v2. Fair enough, except we already have a >>>> package for go-github-com-russross-blackfriday. >>> >>> I was poking around a rust code-base the other day and I noticed our crate >>> importer, and thus a lot of crate packages, have major-version suffixes. I >>> think one of the unique benefits of Guix is that it can simultaneously have >>> multiple versions of libraries installed, and I think we should allow this >>> for library packages. >>> >>> I know that leads to dependency graph explosion, but perhaps we only commit >>> to substitutes for the latest version, and thus any packages using old >>> versions. It should converge over time unless packages go unmaintained. >>> >>> I thought our current stance was to only allow one version at a time, but >>> the crate packages made me question this. I'd like clarity too. >> >> I think there's a bit of a difference between (our packages for) the Rust and >> Go ecosystems here. >> >> In the Go ecosystem, a module is uniquely identified by its module path, >> e.g. "github.com/russross/blackfriday/v2". According to Go's major version >> suffix rules [0], "[s]tarting with major version 2, module paths must have a >> major version suffix like /v2 that matches the major version." Therefore, >> each major version is logically a different module according to Go, and so I >> think we should treat them as separate packages as well. (Note that in many >> cases we can use 'inherit' for brevity.) > > That's a great point! I hadn't considered that we could leverage this to > consider major versioned Go modules as separate packages. That's great! > >> Additionally, the major version suffix rules dictate that "[i]f an old package >> and a new package have the same import path, the new package must be backwards >> compatible with the old package." Assuming upstream sources follow the rules, >> we should be able to update each Go package within each major version without >> breaking dependencies. >> >> (A corollary to that is that packages often break if you try to use a v2 when >> it is expecting a v1.) >> >> I think this differs from Rust, where we have, for example, package-0.1 and >> package-0.2 coexisting. This difference should prevent dependency graph >> explosion for Go. > > It's nice that our Rust packages are enjoying the same stance, but I'm still > not clear on why? Does Rust have the same guarantees? I'll leave this for someone who actually knows the Rust ecosystem to answer :) > >> There are some caveats with "major version suffixes": >> >> * Major versions 0 and 1 don't get a version suffix (so no /v1)... >> >> * ...except for module paths starting with "gopkg.in/", which always have >> a major version suffix, but theirs is styled ".v1" rather than "/v1". >> >> * Modules may either be located in the repository root, or in a "/v2" >> subdirectory (for major version 2). This makes things difficult for our >> importer, because we can't know whether the unpack path should include "/v2" >> without looking at the repository contents. >> >> This is why Jack had to manually add "/v2" to the unpack path. I recently >> made some changes to the importer to better handle, for example, >> "github.com/example/repository/subproject", but it doesn't yet discriminate >> between "/subproject" and "/v2", so it treated "/v2" like a subdirectory of >> the repository. (Until we fix this properly, the importer should probably not >> count major version suffixes when calculating the unpack path, since most >> projects don't use a "/v2" subdirectory.) > > As an aside, when I started writing the importer, I didn't know it was a > possibility to just use the Go toolchain to help us generate packages. I > thought "the Guix way" was to do it all in scheme. It's nice that it's in > scheme, but I would love to leverage the Go toolchain instead. > > IMO, module resolution and graph dependencies in Go are complicated enough > that I'd much rather take the effort we put in trying to replicate and keep > up with the Go toolchain's behavior, and spend that effort elsewhere. > > Does anyone have opinions on this? Hmmm. Setting aside whether or not we want to use external tools in general... If we use the Go tool, we shift the problem domain into "translating between `go' and Guix." For example: when Go downloads a module, it only saves the module, not the whole repository*, so we can't use that to generate the hash. (* Except it does if you used GOPROXY=direct, but that's in the module cache, and only a bare repository.) Or, the fact that we import the latest available version of packages (unless --pin-versions is used), but Go uses exact versions selected with MVS [0]. You might also be interested in taking a look at Gentoo dealing with this translation problem [1]. I'm not saying that this would necessarily be a bad tradeoff either, but it's definitely a tradeoff. Did you have something particular in mind as far as leveraging the Go tooling? > >> All that to say... I think we should definitely maintain coexisting Go v2, v3, >> etc. package definitions. We should probably go the way of Rust though, so we >> have them all in the same package, at different versions: >> >> (define-public go-github-com-russross-blackfriday-v2 >> (package >> (name "go-github-com-russross-blackfriday") >> (version "2.1.0") >> >> instead of as different packages: >> >> (define-public "go-github-com-russross-blackfriday-v2" >> (package >> (name "go-github-com-russross-blackfriday-v2") >> (version "2.1.0") >> >> And of course, it should be policy to remove dependency packages with no >> dependents. (Perhaps we could write a new linter to warn if a "go-" package >> has no inheriters and no dependents?) > > I disagree with this part, only because it's possible the packages are > dependent on non-public (or at least not Guix mainstream) Guix packages. We > get the wealth of the commons if we maintain this package in Guix > proper. However, I think this is definitely an edge case. Good point. >> Does that sound reasonable? > > Reasonable? No, incredible! :) Thank you. I think I've just spent too much time reading Go docs in the past couple months... not that I even write Go! :) [0] [1] cmd/go: allow extraction of urls used to download dependencies -- Sarah