Re: Go importer and packages with version flags

[Katherine Cox-Buday <cox.katherine.e@gmail.com>]

Sarah Morgensen <iskarian@mgsn.dev> writes:

> Hi Katherine, Jack,
>
> Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:
>
>> Jack Hill <jackhill@jackhill.us> writes:
>>
>>> Hi Guix,
>>
>> Hey, Jack, a few thoughts.
>>
>>> While I was was working with the go importer today, it suggested I package
>>> go-github-com-russross-blackfriday-v2. Fair enough, except we already have a
>>> package for go-github-com-russross-blackfriday.
>>
>> I was poking around a rust code-base the other day and I noticed our crate
>> importer, and thus a lot of crate packages, have major-version suffixes. I
>> think one of the unique benefits of Guix is that it can simultaneously have
>> multiple versions of libraries installed, and I think we should allow this
>> for library packages.
>>
>> I know that leads to dependency graph explosion, but perhaps we only commit
>> to substitutes for the latest version, and thus any packages using old
>> versions. It should converge over time unless packages go unmaintained.
>>
>> I thought our current stance was to only allow one version at a time, but
>> the crate packages made me question this. I'd like clarity too.
>
> I think there's a bit of a difference between (our packages for) the Rust and
> Go ecosystems here.
>
> In the Go ecosystem, a module is uniquely identified by its module path,
> e.g. "github.com/russross/blackfriday/v2".  According to Go's major version
> suffix rules [0], "[s]tarting with major version 2, module paths must have a
> major version suffix like /v2 that matches the major version."  Therefore,
> each major version is logically a different module according to Go, and so I
> think we should treat them as separate packages as well.  (Note that in many
> cases we can use 'inherit' for brevity.)

That's a great point! I hadn't considered that we could leverage this to consider major versioned Go modules as separate packages. That's great!

> Additionally, the major version suffix rules dictate that "[i]f an old package
> and a new package have the same import path, the new package must be backwards
> compatible with the old package."  Assuming upstream sources follow the rules,
> we should be able to update each Go package within each major version without
> breaking dependencies.
>
> (A corollary to that is that packages often break if you try to use a v2 when
> it is expecting a v1.)
>
> I think this differs from Rust, where we have, for example, package-0.1 and
> package-0.2 coexisting.  This difference should prevent dependency graph
> explosion for Go.

It's nice that our Rust packages are enjoying the same stance, but I'm still not clear on why? Does Rust have the same guarantees?

> There are some caveats with "major version suffixes":
>
> * Major versions 0 and 1 don't get a version suffix (so no /v1)...
>
>     * ...except for module paths starting with "gopkg.in/", which always have
>       a major version suffix, but theirs is styled ".v1" rather than "/v1".
>
> * Modules may either be located in the repository root, or in a "/v2"
>   subdirectory (for major version 2).  This makes things difficult for our
>   importer, because we can't know whether the unpack path should include "/v2"
>   without looking at the repository contents.
>
> This is why Jack had to manually add "/v2" to the unpack path.  I recently
> made some changes to the importer to better handle, for example,
> "github.com/example/repository/subproject", but it doesn't yet discriminate
> between "/subproject" and "/v2", so it treated "/v2" like a subdirectory of
> the repository.  (Until we fix this properly, the importer should probably not
> count major version suffixes when calculating the unpack path, since most
> projects don't use a "/v2" subdirectory.)

As an aside, when I started writing the importer, I didn't know it was a possibility to just use the Go toolchain to help us generate packages. I thought "the Guix way" was to do it all in scheme. It's nice that it's in scheme, but I would love to leverage the Go toolchain instead.

IMO, module resolution and graph dependencies in Go are complicated enough that I'd much rather take the effort we put in trying to replicate and keep up with the Go toolchain's behavior, and spend that effort elsewhere.

Does anyone have opinions on this?

> All that to say... I think we should definitely maintain coexisting Go v2, v3,
> etc. package definitions.  We should probably go the way of Rust though, so we
> have them all in the same package, at different versions:
>
> (define-public go-github-com-russross-blackfriday-v2
>   (package
>     (name "go-github-com-russross-blackfriday")
>     (version "2.1.0")
>
> instead of as different packages:
>
> (define-public "go-github-com-russross-blackfriday-v2"
>   (package
>     (name "go-github-com-russross-blackfriday-v2")
>     (version "2.1.0")
>
> And of course, it should be policy to remove dependency packages with no
> dependents.  (Perhaps we could write a new linter to warn if a "go-" package
> has no inheriters and no dependents?)

I disagree with this part, only because it's possible the packages are dependent on non-public (or at least not Guix mainstream) Guix packages. We get the wealth of the commons if we maintain this package in Guix proper. However, I think this is definitely an edge case.

> Does that sound reasonable?

Reasonable? No, incredible! :)

-- 
Katherine