Build determinism, dependency granularity, and dependency scope

Build determinism, dependency granularity, and dependency scope

[Stephen Scheck]

I have been trying to package an open source application written in Go for
Guix, and along the way as I've come to understand the mechanics better,
I've realized a few things which are a bit disconcerting. I'll refer to the
package for Yggdrasil, as it was recommended to me as a good blueprint to
follow for the project I'm trying to package.

If you take a look at the package definition for Yggdrasil 0.3.15, here are
some of the Golang dependencies:

    (propagated-inputs
       ;; ...
       ("go-golang-org-x-net" ,go-golang-org-x-net)
       ("go-golang-org-x-text" ,go-golang-org-x-text)
       ;; ... )

If you look at the project's `go.mod` file [1], you have:

    golang.org/x/net v0.0.0-20200301022130-244492dfa37a
    golang.org/x/text v0.3.3-0.20191230102452-929e72ca90de

But if you look at the commits for the packages defined in the Guix tree,
they do not correspond. And the `go-golang-org-x-text` package in the Guix
tree (version "0.3.2") does not even meet the minimum version specified in
`go.mod`.

Also, it occurs to me that someone could decide to bump the version for one
of these packages up in the global Guix tree at any time to satisfy the
version requirements of some other package which require a newer version,
but because at the single package level there is only a reference to the
package name but not the version, all dependencies in the tree will be
carried along for the ride (!).

Now, there's nothing preventing someone from defining versioned packages in
the Guix tree, such as a `go-golang-org-x-text-929e72ca90de`, and referring
to those in dependent packages, but in practice that doesn't seem to be
done and most packages appear to have only one version, except for some
things like major language/platform versions (e.g. openjdk).

Am I missing something here?

It seems like what is needed would something like a package-scoped
"dependency constructor", allowing you to declare required versions
per-package:

    (propagated-inputs
       ;; ...
       ("go-golang-org-x-net" (go-module "golang.org/x/net" "244492dfa37a"))
       ("go-golang-org-x-text" (go-module "golang.org/x/text"
"929e72ca90de"))
       ;; ... )

[1] https://github.com/yggdrasil-network/yggdrasil-go/blob/v0.3.15/go.mod

Re: Build determinism, dependency granularity, and dependency scope

[raingloom]

On Tue, 24 Nov 2020 16:20:35 -0500
Stephen Scheck <singularsyntax@gmail.com> wrote:

> I have been trying to package an open source application written in
> Go for Guix, and along the way as I've come to understand the
> mechanics better, I've realized a few things which are a bit
> disconcerting. I'll refer to the package for Yggdrasil, as it was
> recommended to me as a good blueprint to follow for the project I'm
> trying to package.
> 
> If you take a look at the package definition for Yggdrasil 0.3.15,
> here are some of the Golang dependencies:
> 
>     (propagated-inputs
>        ;; ...
>        ("go-golang-org-x-net" ,go-golang-org-x-net)
>        ("go-golang-org-x-text" ,go-golang-org-x-text)
>        ;; ... )
> 
> If you look at the project's `go.mod` file [1], you have:
> 
>     golang.org/x/net v0.0.0-20200301022130-244492dfa37a
>     golang.org/x/text v0.3.3-0.20191230102452-929e72ca90de
> 
> But if you look at the commits for the packages defined in the Guix
> tree, they do not correspond. And the `go-golang-org-x-text` package
> in the Guix tree (version "0.3.2") does not even meet the minimum
> version specified in `go.mod`.
> 
> Also, it occurs to me that someone could decide to bump the version
> for one of these packages up in the global Guix tree at any time to
> satisfy the version requirements of some other package which require
> a newer version, but because at the single package level there is
> only a reference to the package name but not the version, all
> dependencies in the tree will be carried along for the ride (!).
> 
> Now, there's nothing preventing someone from defining versioned
> packages in the Guix tree, such as a
> `go-golang-org-x-text-929e72ca90de`, and referring to those in
> dependent packages, but in practice that doesn't seem to be done and
> most packages appear to have only one version, except for some things
> like major language/platform versions (e.g. openjdk).
> 
> Am I missing something here?
> 
> It seems like what is needed would something like a package-scoped
> "dependency constructor", allowing you to declare required versions
> per-package:
> 
>     (propagated-inputs
>        ;; ...
>        ("go-golang-org-x-net" (go-module "golang.org/x/net"
> "244492dfa37a")) ("go-golang-org-x-text" (go-module
> "golang.org/x/text" "929e72ca90de"))
>        ;; ... )
> 
> [1]
> https://github.com/yggdrasil-network/yggdrasil-go/blob/v0.3.15/go.mod

Multiple versions leads to more maintenance burden and a bigger store.

But I admit I didn't really investigate which exact module versions
Yggdrasil is compatible with.

Re: Build determinism, dependency granularity, and dependency scope

[Leo Famulari]

On Tue, Nov 24, 2020 at 04:20:35PM -0500, Stephen Scheck wrote:
> But if you look at the commits for the packages defined in the Guix tree,
> they do not correspond. And the `go-golang-org-x-text` package in the Guix
> tree (version "0.3.2") does not even meet the minimum version specified in
> `go.mod`.

Right.

> Am I missing something here?

No. The way that dependencies are handled in Go-world does concord with
Guix on a conceptual level — it's definitely possible to have hundreds
of versions of each Go library — but it's impractical with the current
Guix tooling.

> It seems like what is needed would something like a package-scoped
> "dependency constructor", allowing you to declare required versions
> per-package:
> 
>     (propagated-inputs
>        ;; ...
>        ("go-golang-org-x-net" (go-module "golang.org/x/net" "244492dfa37a"))
>        ("go-golang-org-x-text" (go-module "golang.org/x/text"
> "929e72ca90de"))
>        ;; ... )

Yes, I've suggested something like this before (somewhere in the mailing
list archives). The specifications would also need to include the source
hash, but otherwise we've come to the same conclusion. Ideally, they
would be generated automatically from go.mod.

Previously, I made sure these "standard" Go libraries matched the
versions specified by Syncthing, since I maintained that package
carefully. Since Syncthing switched to Go modules, I haven't found the
time to rework Guix's go-build-system to work correctly, and I haven't
paid attention to what's been happening with Go on Guix.

A good stopgap option is to use vendored dependencies (heresy, I know),
assuming they are free software and the upstream sources include them.
This works fine and is better than not having Go software at all.

In the long run, Guix's Go support needs a complete overhaul.

Re: Build determinism, dependency granularity, and dependency scope

[Stephen Scheck]

On Wed, Nov 25, 2020 at 10:52 PM raingloom <raingloom@riseup.net> wrote:

>
> Multiple versions leads to more maintenance burden and a bigger store.
>

That is true, but IMO with Guix's focus on build determinism and
repeatability, it's a price that must be paid.


> But I admit I didn't really investigate which exact module versions
> Yggdrasil is compatible with.
>

As an end-user, if I'm installing the Yggdrasil package with a specific
version, say 0.3.15, that corresponds to a specific Git commit tag, I want
that package to maintain fidelity with the dependency versions locked down
by the upstream project maintainers. As it stands, the Guix dependency
declaration mechanism is lossy, flattening out dependencies to a single
global scope which all packages in a profile must assume.

Unfortunately, for me, that really limits its usefulness.

Re: Build determinism, dependency granularity, and dependency scope

[Stephen Scheck]

On Wed, Nov 25, 2020 at 6:15 PM Leo Famulari <leo@famulari.name> wrote:

>
> No. The way that dependencies are handled in Go-world does concord with
> Guix on a conceptual level — it's definitely possible to have hundreds
> of versions of each Go library — but it's impractical with the current
> Guix tooling.
>

Java-based Guix packages also suffer from this problem (actually, I'm far
more familiar with dependency management in the JVM landscape than for Go,
but the use of granularly versioned and scoped, distributed dependency
models by both languages appears to be similar on the surface).

For example, the `java-log4j-core` Guix package (at version 2.4.1 in the
Guix tree) has a dependency on `java-fasterxml-jackson-core` (at version
2.9.4), but the corresponding Log4j release asserts a dependency version of
2.6.2 in its `pom.xml` [1].


> A good stopgap option is to use vendored dependencies (heresy, I know),
> assuming they are free software and the upstream sources include them.
> This works fine and is better than not having Go software at all.
>

In the case of the Go application I was trying to package, it does not
include vendored dependencies. And I don't have any relationship or
check-in privileges with the project - it is simply something I wanted to
use in an environment with other Guix-sourced packages. Well, I guess it
would be straightforward to fork the GitHub source, run `go mod vendor` [2]
and check in the vendor directory with a specific tag such as
"vx.y.z-guix-vendored". Whether the project maintainers would accept such a
pull request, or if it would be considered bad form to refer to a forked
repository in a Guix package definition instead of the official repo if
not, I don't know.

In the long run, Guix's Go support needs a complete overhaul.
>

Indeed.

[1] https://github.com/apache/logging-log4j2/blob/rel/2.4.1/pom.xml#L177
[2] https://golang.org/ref/mod#vendoring

Re: Build determinism, dependency granularity, and dependency scope

[Leo Famulari]

On Fri, Nov 27, 2020 at 02:08:19PM -0500, Stephen Scheck wrote:
> Java-based Guix packages also suffer from this problem (actually, I'm far
> more familiar with dependency management in the JVM landscape than for Go,
> but the use of granularly versioned and scoped, distributed dependency
> models by both languages appears to be similar on the surface).
> 
> For example, the `java-log4j-core` Guix package (at version 2.4.1 in the
> Guix tree) has a dependency on `java-fasterxml-jackson-core` (at version
> 2.9.4), but the corresponding Log4j release asserts a dependency version of
> 2.6.2 in its `pom.xml` [1].

Yes, sounds quite similar... more room for improvement.

> In the case of the Go application I was trying to package, it does not
> include vendored dependencies. And I don't have any relationship or
> check-in privileges with the project - it is simply something I wanted to
> use in an environment with other Guix-sourced packages. Well, I guess it
> would be straightforward to fork the GitHub source, run `go mod vendor` [2]
> and check in the vendor directory with a specific tag such as
> "vx.y.z-guix-vendored". Whether the project maintainers would accept such a
> pull request, or if it would be considered bad form to refer to a forked
> repository in a Guix package definition instead of the official repo if
> not, I don't know.

Right, that should work, and would be a good way to get started for a
"private" package, but we (Guix) really need to improve our tools here.
On that note... I just remembered we have a WIP patch that adds a
recursive Go module importer:

https://bugs.gnu.org/44178

I encourage you to try it out! I see you have a patch in Guix, so maybe
you learned how to apply patches and use the 'pre-inst-env' script to
test them, as described in the manual section Contributing? Please don't
hesitate to ask for help :)

I think the ideal solution is to combine recursive module importing with
the "dependency constructor".