packaging a golang package

packaging a golang package

[Timmy Douglas]

I recently installed guix for the first time and I wanted to try to
package my first program. The one I decided to try is written in go and
uses go.mod (https://github.com/coredns/coredns/blob/master/go.mod) for
modules. Running `go build` would normally download those if they don't
exist.

I took a look at a couple of other packages, and it looks like the right
way to do it would be to package the individual modules as seen in
guix/gnu/packages/golang.scm. Has anyone tried automating this sort of
thing? There are almost 40 dependencies...

If some of the packages already exist, but are a different version,
what's the right way to resolve the conflict?

Thanks,



```coredns.scm

(define-module (coredns)
               #:use-module (guix packages)
               #:use-module (guix git-download)
               #:use-module (guix build-system go)
               #:use-module ((guix licenses) #:prefix license:))

(define-public coredns
  (package
   (name "coredns")
   (version "1.8.0")
   (source (origin
            (method git-fetch)
            (uri (git-reference
                  (url "https://github.com/coredns/coredns.git")
                  (commit (string-append "v" version))))
            (file-name (git-file-name name version))
            (sha256
             (base32
              "04hkz70s5i7ndwyg39za3k83amvmi90rkjm8qp3w3a8fbmq4q4y6"))))
   (build-system go-build-system)
   (arguments
    '(#:import-path "github.com/coredns/coredns"))
   (inputs
    `(;fixme
      ))
   (synopsis "CoreDNS is a DNS server/forwarder, written in Go, that chains plugins.")
   (description "CoreDNS is a fast and flexible DNS server. The key word here is flexible: with CoreDNS you are able to do what you want with your DNS data by utilizing plugins. If some functionality is not provided out of the box you can add it by writing a plugin.")
   (home-page "https://github.com/coredns/coredns")
   (license license:asl2.0)))

coredns

Re: packaging a golang package

[raingloom]

On Thu, 07 Jan 2021 23:01:49 -0800
Timmy Douglas <mail@timmydouglas.com> wrote:

> I recently installed guix for the first time and I wanted to try to
> package my first program. The one I decided to try is written in go
> and uses go.mod
> (https://github.com/coredns/coredns/blob/master/go.mod) for modules.
> Running `go build` would normally download those if they don't exist.
> 
> I took a look at a couple of other packages, and it looks like the
> right way to do it would be to package the individual modules as seen
> in guix/gnu/packages/golang.scm. Has anyone tried automating this
> sort of thing? There are almost 40 dependencies...

There is a WIP importer:
https://issues.guix.gnu.org/44178

Re: packaging a golang package

[Leo Famulari]

On Fri, Jan 8, 2021, at 02:01, Timmy Douglas wrote:
> 
> I recently installed guix for the first time and I wanted to try to
> package my first program. The one I decided to try is written in go and
> uses go.mod (https://github.com/coredns/coredns/blob/master/go.mod) for
> modules. Running `go build` would normally download those if they don't
> exist.
> 
> I took a look at a couple of other packages, and it looks like the right
> way to do it would be to package the individual modules as seen in
> guix/gnu/packages/golang.scm. Has anyone tried automating this sort of
> thing? There are almost 40 dependencies...

Thanks for working on this!

It's true, Go programs usually have a dependency graph that is uncomfortably large — although not impossibly large like Rust.

There is a work-in-progress implementation of a Go package importer that I believe should make it easier:

https://issues.guix.gnu.org/issue/44178

You could try it out and, if it works for you, send those patches, and give feedback on the importer as well :)

As for the versioning, the Guix package tree can include multiple versions of a package. Search for "inherit" in the crates-io.scm package module for some examples. We would like to improve how this works, again for Go and Rust, because it's such a common requirement.

Re: packaging a golang package

[Timmy Douglas]

"Leo Famulari" <leo@famulari.name> writes:

> On Fri, Jan 8, 2021, at 02:01, Timmy Douglas wrote:
>> 
>> I recently installed guix for the first time and I wanted to try to
>> package my first program. The one I decided to try is written in go and
>> uses go.mod (https://github.com/coredns/coredns/blob/master/go.mod) for
>> modules. Running `go build` would normally download those if they don't
>> exist.
>> 
>> I took a look at a couple of other packages, and it looks like the right
>> way to do it would be to package the individual modules as seen in
>> guix/gnu/packages/golang.scm. Has anyone tried automating this sort of
>> thing? There are almost 40 dependencies...
>
> Thanks for working on this!
>
> It's true, Go programs usually have a dependency graph that is
> uncomfortably large — although not impossibly large like Rust.
>
> There is a work-in-progress implementation of a Go package importer
> that I believe should make it easier:
>
> https://issues.guix.gnu.org/issue/44178
>
> You could try it out and, if it works for you, send those patches, and
> give feedback on the importer as well :)

Thanks for the pointer!

I tried both Katherine's patch (with dftxbs3e's mini fix for
recursive-import) and Helio's patch (with a git remote add/cherry-pick)
and had trouble with both of them. I spent a little more time with
Katherine's patch so I'll go into that more below.

Part of the issue is that I haven't used Scheme in like 15 years (and
when I did, it wasn't for anything non-trivial). But I'm also really
struggling to debug what's going on:

I run `make && ./pre-inst-env guix import go -r github.com/coredns/coredns`:

Starting download of /tmp/guix-file.pZqXNO
From https://proxy.golang.org/k8s.io/klog/@v/v1.0.0.mod...
 v1.0.0.mod  68B                       88KiB/s 00:00 [##################] 100.0%
metadata#f
Backtrace:
In ice-9/boot-9.scm:
  1736:10 13 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          12 (apply-smob/0 #<thunk 7f98846a3740>)
In ice-9/boot-9.scm:
    718:2 11 (call-with-prompt _ _ #<procedure default-prompt-handle?>)
In ice-9/eval.scm:
    619:8 10 (_ #(#(#<directory (guile-user) 7f98842e0f00>)))
In guix/ui.scm:
  2154:12  9 (run-guix-command _ . _)
In guix/scripts/import.scm:
   120:11  8 (guix-import . _)
In ice-9/eval.scm:
    159:9  7 (_ _)
In guix/import/utils.scm:
   468:27  6 (recursive-import _ #:repo->guix-package _ #:guix-name _ ?)
In srfi/srfi-1.scm:
   586:17  5 (map1 (("k8s.io/klog" #f) ("k8s.io/client-go" #f) (?) ?))
In guix/import/utils.scm:
   457:33  4 (lookup-node "k8s.io/klog" #f)
In guix/utils.scm:
    703:8  3 (call-with-temporary-output-file #<procedure 7f9871b8c7?>)
In ice-9/eval.scm:
   293:34  2 (_ #(#(#(#(#(#(#(#(#<directory ?> ?) ?) ?) ?) ?) ?) ?) ?))
    155:9  1 (_ #(#(#<directory (guix import go) 7f9882267f00>) #f))
In unknown file:
           0 (list-ref #f 1)

ERROR: In procedure list-ref:
In procedure list-ref: Wrong type argument in position 1: #f


I understand that (list-ref #f 1) won't work, but I can't figure out
where that code is based on the information in the backtrace. Seems like
the variables/symbols/filenames are missing everywhere where it counts.

I loaded the code up with #'display and found that I needed to add
another git scs/vcs entry for k8s.io, and that the
fetch-module-meta-data method tries to use regexes to parse html:

root-module-path= k8s.io/klog
line=
line=            <html><head>
line=                  <meta name="go-import"
line=                        content="k8s.io/klog
line=                                 git https://github.com/kubernetes/klog">
line=                  <meta name="go-source"
line=                        content="k8s.io/klog
line=                                 https://github.com/kubernetes/klog
line=                                 https://github.com/kubernetes/klog/tree/master{/dir}
line=                                 https://github.com/kubernetes/klog/blob/master{/dir}/{file}#L{line}">
line=            </head></html>
line=

(define (fetch-module-meta-data module-path)
  "Fetches module meta-data from a module's landing page. This is necessary
because goproxy servers don't currently provide all the information needed to
build a package."
  (let* ((port (http-fetch (string->uri (format #f "https://~a?go-get=1" module-path))))
         (module-metadata #f)
         (meta-tag-prefix "<meta name=\"go-import\" content=\"")
         (meta-tag-prefix-length (string-length meta-tag-prefix)))
    (do ((line (read-line port) (read-line port)))
        ((or (eof-object? line)
             module-metadata))
      (let ((meta-tag-index (string-contains line meta-tag-prefix)))
        (display "line=")
        (display line)
        (newline)
        (when meta-tag-index
          (let* ((start (+ meta-tag-index meta-tag-prefix-length))
                 (end (string-index line #\" start)))
            (set! module-metadata
              (string-split (substring/shared line start end) #\space))))))
    (close-port port)
    module-metadata))


I don't think the regex would match due to content="" being put on
another line--so #f is returned and causes a type exception later. I'm
a little more used to statically typed languages so tracing it back to
there took longer than I would have liked.

Maybe I need to find a video tutorial on writing Scheme for Guix or
similiar? I felt like I was doing the wrong thing with
printf/make/pre-inst-env guix. I tried starting geiser in emacs, but
didn't really know what I should evaluate. It also added an unbearable
lag to typing in the scheme buffer and didn't provide auto-completion or
symbol lookup, so I didn't go too far with it...

Would be interested if Katherine or Helio have any more updates.

Re: packaging a golang package

[Timmy Douglas]

I also tried Helio's branch, rebased on the latest, with the following
changes:

diff --git a/guix/import/go-modules.scm b/guix/import/go-modules.scm
index 20fc5c9642..811c9f60ae 100644
--- a/guix/import/go-modules.scm
+++ b/guix/import/go-modules.scm
@@ -290,7 +290,7 @@ VERSION, MODULE-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
                                (() "")
                                (_ home-page)))
                  (synopsis ,synopsis)
-                 (description ,(beautify-description description))
+                 (description ,(beautify-description (or description "")))
                  (license ,(match license
                              (() #f)
                              ((license) license)
@@ -327,12 +327,11 @@ specified, attempt to fetch that version. Otherwise, fetch the latest version."
 (define* (go-module-recursive-import name #:optional version)
   "Given a module NAME and, optionally its VERSION, produce package definitions
 for itself and every dependency, recursively."
-  (recursive-import (if version (string-append name "@" version) name) #f
+  (recursive-import (if version (string-append name "@" version) name)
                     #:repo->guix-package
-                    (lambda (name repo)
+                    (lambda* (name #:key repo version)
                       (let-values (((name version) (package-name->name+version name)))
-                            (go-module->guix-package name version))
-                        )
+                        (go-module->guix-package name version)))
                     #:guix-name go-module-name+version->package-name+version))



I liked how Helio's version uses the go proxy to simplify the parsing of
the package lists, but I ran this:

./pre-inst-env guix import go-modules -r github.com/coredns/coredns

and it took what felt like thirty minutes before failing with:

error: failed to retrieve package information from "https://proxy.golang.org/replace/@v/(.info": 410 ("Gone")
Backtrace:
In ice-9/boot-9.scm:
  1736:10 18 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          17 (apply-smob/0 #<thunk 7fe73061b740>)
In ice-9/boot-9.scm:
    718:2 16 (call-with-prompt _ _ #<procedure default-prompt-handle?>)
In ice-9/eval.scm:
    619:8 15 (_ #(#(#<directory (guile-user) 7fe730258f00>)))
In guix/ui.scm:
  2154:12 14 (run-guix-command _ . _)
In guix/scripts/import.scm:
   121:11 13 (guix-import . _)
In ice-9/eval.scm:
    159:9 12 (_ _)
In guix/import/utils.scm:
   464:27 11 (recursive-import _ #:repo->guix-package _ #:guix-name _ ?)
In srfi/srfi-1.scm:
   586:29 10 (map1 (("github.com/davecgh/go-spew@v1.1.1" #f) (# #) ?))
   586:29  9 (map1 (("github.com/google/go-cmp@v0.4.0" #f) ("g?" ?) ?))
   586:29  8 (map1 (("github.com/google/gofuzz@v1.1.0" #f) ("g?" ?) ?))
   586:29  7 (map1 (("github.com/kr/pretty@v0.2.0" #f) ("githu?" ?) ?))
   586:29  6 (map1 (("github.com/spf13/pflag@v1.0.5" #f) ("gol?" ?) ?))
   586:29  5 (map1 (("golang.org/x/tools@v0.0.0-20200505023115?" ?) ?))
   586:29  4 (map1 (("gopkg.in/check.v1@v1.0.0-20190902080502-?" ?) ?))
   586:29  3 (map1 (("k8s.io/klog/v2@v2.2.0" #f) ("sigs.k8s.io?" ?) ?))
   586:29  2 (map1 (("sigs.k8s.io/yaml@v1.2.0" #f) ("replace@(" #f)))
   586:17  1 (map1 (("replace@(" #f)))
In guix/import/utils.scm:
   453:33  0 (lookup-node "replace@(" #f)

guix/import/utils.scm:453:33: In procedure lookup-node:
Wrong number of values returned to continuation (expected 2)


Looks like it ran into the replace syntax and didn't parse it correctly?
https://golang.org/ref/mod#go-mod-file


I sort of wonder if the `git-fetch` portion of go packages should be
something like `git-fetch-and-restore-go-modules` Maybe something like
`go mod vendor` could be used an the hash would be calculated on that?
I'm not familiar enough with the internals of go modules, but I'm not
really seeing the value in generating a bunch of package metadata on all
the module dependencies.

Re: packaging a golang package

[Helio Machado]

Hi!

Looks like it ran into the replace syntax and didn't parse it correctly?
> https://golang.org/ref/mod#go-mod-file


So it seems. I'll debug it as soon as I can; the import process is
unbearably long.

I sort of wonder if the `git-fetch` portion of go packages should be
> something like `git-fetch-and-restore-go-modules` Maybe something like
> `go mod vendor` could be used an the hash would be calculated on that?
> I'm not familiar enough with the internals of go modules, but I'm not
> really seeing the value in generating a bunch of package metadata on all
> the module dependencies.
>

Guix seems to have a strong opinion about dependency vendoring, but it's
technically viable as long as you don't produce architecture-specific
artifacts when packaging. See https://issues.guix.info/43872 for more
information about the pitfalls I encountered while packaging go-ethereum
the fast way.

Re: packaging a golang package

[jgart]

There's also an emacs package by Ryan Prior to help with the tedium of packaging go dependencies:

https://github.com/ryanprior/emacs-guix-packaging

from the README:

Some packages are complex due to a large number of transitive dependencies. I've
found this to be true of software in the golang ecosystem where it's not
uncommon to see 50+ transitive dependencies, and occasionally many more.

guix-packaging provides some facilities to help track progress on complex packages.

*** Commands

**** ~M-x~ ~guix-packaging-go-mod-to-checklist-dwim~
Turns go module definitions into an org/markdown checklist, suitable to keep
track of packaging progress.

**** ~M-x~ ~guix-packaging-go-mod-to-checkbox~
Replace a single go module definition with a checkbox.

Re: packaging a golang package

[Timmy Douglas]

Helio Machado <0x2b3bfa0@gmail.com> writes:

> So it seems. I'll debug it as soon as I can; the import process is
> unbearably long.

let me know what you find out--i can give it another try when you're
ready.

>> I sort of wonder if the `git-fetch` portion of go packages should be
>> something like `git-fetch-and-restore-go-modules` Maybe something like
>> `go mod vendor` could be used an the hash would be calculated on that?
>> I'm not familiar enough with the internals of go modules, but I'm not
>> really seeing the value in generating a bunch of package metadata on all
>> the module dependencies.
>
> Guix seems to have a strong opinion about dependency vendoring, but it's
> technically viable as long as you don't produce architecture-specific
> artifacts when packaging. See https://issues.guix.info/43872 for more
> information about the pitfalls I encountered while packaging go-ethereum
> the fast way.

I tried the go mod vendor approach on a fork I made and it was fairly
straightforward:

1. Update module name in go.mod

2. Run `go mod vendor`

3. Fix .gitignore to make git adding vendor/ easier

the downside is that the package is tied to my personal git. I think
merging updates wouldn't be too hard and I like how it doesn't pollute
the package repo with a bunch of dependencies (cat vendor/modules.txt|wc -l
--> 697), but I'm not sure it'd pass the bar to be accepted into
guix. But it's probably good enough for me to use personally.

$ cat ~/.config/guix-packages/coredns.scm

(define-module (coredns)
               #:use-module (guix packages)
               #:use-module (guix git-download)
               #:use-module (guix build-system go)
               #:use-module ((guix licenses) #:prefix license:))

(define-public coredns
  (let ((commit "78de01a9cddf140c04ec3c4095195177d21cacff")
        (revision "0"))
    (package
      (name "coredns")
      (version (git-version "1.8.1" revision commit))
      (source (origin
                (method git-fetch)
                (uri (git-reference
                      (url "https://github.com/timmydo/coredns")
                      (commit commit)))
                (file-name (git-file-name name version))
                (sha256
                 (base32 "02gdj866mz17p1f0fgfjpbb9cah2ykziacahpkw0viq1vj231hai"))))
      (build-system go-build-system)
      (arguments
       '(#:import-path "github.com/timmydo/coredns"))
      (synopsis "DNS server/forwarder, written in Go, that chains plugins")
      (description "CoreDNS is a fast and flexible DNS server.
  The key word here is flexible: with CoreDNS you are able to do
what you want with your DNS data by utilizing plugins.  If some
functionality is not provided out of the box you can add it by
writing a plugin.")
      (home-page "https://github.com/coredns/coredns")
      (license license:asl2.0))))

coredns

Re: packaging a golang package

[Francois.JOULAUD--- via]

Hello,

On Sun, Jan 17, 2021 at 02:31:39PM +0100, Helio Machado wrote:
> Looks like it ran into the replace syntax and didn't parse it correctly?
> > https://golang.org/ref/mod#go-mod-file

New version of the patch on https://issues.guix.gnu.org/issue/44178#10
fixes some of those problems. There is still others bugs left but you
could give it a new try. Git repo with patch applied is available at [6]

> Guix seems to have a strong opinion about dependency vendoring, but it's
> technically viable as long as you don't produce architecture-specific
> artifacts when packaging. See https://issues.guix.info/43872 for more
> information about the pitfalls I encountered while packaging go-ethereum
> the fast way.

It seems to me we have three possible ways to handle Go packaging.

First is to use vendored dependencies (when upstream provides them). This
one has the merits of simplicity (as we just have to download the
source and launch `go build` or whatever is needed) and less risks of
bugs. The downsides are known: difficult to audit Licences, duplication
of code source, difficulty to follow security bugs, etc.

Second is to re-vendor package from go.mod (coming directly from code
source or synthetized from source) by creating a source derivation
including all dependencies. This is the strategy followed by Nixpkgs
as explained in [1]. (It seems to me this is the route followed in
the patches to  from Helio in [2] but I did not take the time to read
them.) With this approach we still have some of the downsides of using
vendored upstream but it is at least easier to verify the existence
of upstream.

Third is to package every dependencies. This is the most transparent way
and the one that leads to less code duplication but the downside is the
difficulty to scale with the number of packages (even if the objective of
patch at [3] is to automate it) and the need to have only one reference
version for each package in the distribution which have its own share of
problems (one of them being that we don't use those tested by upstream
as stated in [4]).

I think Guix should support all three in its tooling to be able to support
several use-cases. For inclusion of packages in Guix proper we should try
the individual packaging of dependencies and see how it works. Perhaps we
will need exceptions as the one Debian does for Kubernetes [5]. Perhaps
we will need to fallback to vendoring or re-vendoring in more cases but
we will learn as we walk.

Best regards,
François

[1]: https://github.com/NixOS/nixpkgs/issues/84826
[2]: https://github.com/0x2b3bfa0/guix-go-modules/commit/5defe897065c5d3e63740932b360474132c77877
[3]: https://issues.guix.gnu.org/issue/44178#10
[4]: https://github.com/NixOS/nixpkgs/issues/84826#issuecomment-616663815
[5]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=971515
[6]: https://github.com/kat-co/guix/tree/create-go-importer

Re: packaging a golang package

[Helio Machado]

(Quick note: my patches follow the third approach, not the second)

On Mon, 25 Jan 2021 at 21:49, JOULAUD François <
Francois.JOULAUD@radiofrance.com> wrote:

> Hello,
>
> On Sun, Jan 17, 2021 at 02:31:39PM +0100, Helio Machado wrote:
> > Looks like it ran into the replace syntax and didn't parse it correctly?
> > > https://golang.org/ref/mod#go-mod-file
>
> New version of the patch on https://issues.guix.gnu.org/issue/44178#10
> fixes some of those problems. There is still others bugs left but you
> could give it a new try. Git repo with patch applied is available at [6]
>
> > Guix seems to have a strong opinion about dependency vendoring, but it's
> > technically viable as long as you don't produce architecture-specific
> > artifacts when packaging. See https://issues.guix.info/43872 for more
> > information about the pitfalls I encountered while packaging go-ethereum
> > the fast way.
>
> It seems to me we have three possible ways to handle Go packaging.
>
> First is to use vendored dependencies (when upstream provides them). This
> one has the merits of simplicity (as we just have to download the
> source and launch `go build` or whatever is needed) and less risks of
> bugs. The downsides are known: difficult to audit Licences, duplication
> of code source, difficulty to follow security bugs, etc.
>
> Second is to re-vendor package from go.mod (coming directly from code
> source or synthetized from source) by creating a source derivation
> including all dependencies. This is the strategy followed by Nixpkgs
> as explained in [1]. (It seems to me this is the route followed in
> the patches to  from Helio in [2] but I did not take the time to read
> them.) With this approach we still have some of the downsides of using
> vendored upstream but it is at least easier to verify the existence
> of upstream.
>
> Third is to package every dependencies. This is the most transparent way
> and the one that leads to less code duplication but the downside is the
> difficulty to scale with the number of packages (even if the objective of
> patch at [3] is to automate it) and the need to have only one reference
> version for each package in the distribution which have its own share of
> problems (one of them being that we don't use those tested by upstream
> as stated in [4]).
>
> I think Guix should support all three in its tooling to be able to support
> several use-cases. For inclusion of packages in Guix proper we should try
> the individual packaging of dependencies and see how it works. Perhaps we
> will need exceptions as the one Debian does for Kubernetes [5]. Perhaps
> we will need to fallback to vendoring or re-vendoring in more cases but
> we will learn as we walk.
>
> Best regards,
> François
>
> [1]: https://github.com/NixOS/nixpkgs/issues/84826
> [2]:
> https://github.com/0x2b3bfa0/guix-go-modules/commit/5defe897065c5d3e63740932b360474132c77877
> [3]: https://issues.guix.gnu.org/issue/44178#10
> [4]: https://github.com/NixOS/nixpkgs/issues/84826#issuecomment-616663815
> [5]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=971515
> [6]: https://github.com/kat-co/guix/tree/create-go-importer

Re: packaging a golang package

[Katherine Cox-Buday]

Hello again, François! I've redirected this thread to guix-devel, and
the bug since we've begun discussing implementation details.

JOULAUD François <Francois.JOULAUD@radiofrance.com> writes:

> First is to use vendored dependencies (when upstream provides them). This
> one has the merits of simplicity (as we just have to download the
> source and launch `go build` or whatever is needed) and less risks of
> bugs. The downsides are known: difficult to audit Licences, duplication
> of code source, difficulty to follow security bugs, etc.

-1 to this approach (explanation below).

> Second is to re-vendor package from go.mod (coming directly from code
> source or synthetized from source) by creating a source derivation
> including all dependencies. This is the strategy followed by Nixpkgs
> as explained in [1]. (It seems to me this is the route followed in
> the patches to  from Helio in [2] but I did not take the time to read
> them.) With this approach we still have some of the downsides of using
> vendored upstream but it is at least easier to verify the existence
> of upstream.

I don't fully understand this approach, so I don't understand how this
is different to approach three? Does this mean we create a pseudo,
non-public package which states all the dependencies as inputs? If so,
why wouldn't we just go with option three?

> Third is to package every dependencies. This is the most transparent way
> and the one that leads to less code duplication but the downside is the
> difficulty to scale with the number of packages (even if the objective of
> patch at [3] is to automate it) and the need to have only one reference
> version for each package in the distribution which have its own share of
> problems (one of them being that we don't use those tested by upstream
> as stated in [4]).

I think this is the eternal conflict between distributions and
code-bases. As a software engineer, I am a fan of vendoring. It removes
entire classes of issues:

- Dependency on remote servers to be up to fetch dependencies and
  perform builds
- Requiring some kind of system to pin versions
- Needing to stay on top of releases even if that doesn't meet your
  schedule or needs

As a packager for a distribution, I dislike vendoring because of the
reasons you outlined above, _but_ I also dislike building upstream
software with versions of dependencies that weren't approved, tested,
and verified, upstream. It seems to me like that's a recipe for
unstable, maybe even insecure, software.

Normally, distributions are forced to do this because their packaging
and build systems are only set up to support one version at a time. Guix
can theoretically support all versions, but doesn't want to take on the
dependency graph explosion this approach would cause.

If we go on historical precedent, and how Guix handles the other
language ecosystems, we should only have one version of each dependency,
and build all software which relies on that dependency on the version we
have packaged. Guix has already begun taking on the difficult challenges
with this approach, including patching upstream to work with versions of
dependencies upstream was not built for.

I dislike it, but I also don't think we should try to solve the broader
class of issues while trying to implement an importer. It should be a
larger discussion within the Guix community across _all_ language
packages about how we might achieve upstream parity while still
maintaining our goals as a distribution, all while not crippling our
infrastructure and people :)

That's my viewpoint, but I think we should all defer to some of the
longer-term maintainers who have helped guide Guix.

Thanks for writing up these options.

--
Katherine

Re: packaging a golang package

[Timmy Douglas]

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> Hello again, François! I've redirected this thread to guix-devel, and
> the bug since we've begun discussing implementation details.
>
> JOULAUD François <Francois.JOULAUD@radiofrance.com> writes:
>
>> First is to use vendored dependencies (when upstream provides them). This
>> one has the merits of simplicity (as we just have to download the
>> source and launch `go build` or whatever is needed) and less risks of
>> bugs. The downsides are known: difficult to audit Licences, duplication
>> of code source, difficulty to follow security bugs, etc.
>
> -1 to this approach (explanation below).

Seems like there could be two sub-scenarios here: the code could use go
modules or not.

>> Second is to re-vendor package from go.mod (coming directly from code
>> source or synthetized from source) by creating a source derivation
>> including all dependencies. This is the strategy followed by Nixpkgs
>> as explained in [1]. (It seems to me this is the route followed in
>> the patches to  from Helio in [2] but I did not take the time to read
>> them.) With this approach we still have some of the downsides of using
>> vendored upstream but it is at least easier to verify the existence
>> of upstream.
>
> I don't fully understand this approach, so I don't understand how this
> is different to approach three? Does this mean we create a pseudo,
> non-public package which states all the dependencies as inputs? If so,
> why wouldn't we just go with option three?

I think this approach is like saying you git clone the upstream repo,
run go mod vendor or go mod download, then go build. Or whatever
buildGoModule does in nix (https://github.com/NixOS/nixpkgs/issues/84826)

I read through that issue and would personally vote for this approach of
using `go` to restore the code.

I think trying to reimplement go module restore process with Guix
packages is bordering on Not Invented Here. There would be a never
ending battle of trying to reimplement the go module restore process and
the amount of source packages would really clutter things up. I think
some of the issues with the distro wanting to change a package could be
solved with a feature to patch go.mod before calling `go` to restore.

>> Third is to package every dependencies. This is the most transparent way
>> and the one that leads to less code duplication but the downside is the
>> difficulty to scale with the number of packages (even if the objective of
>> patch at [3] is to automate it) and the need to have only one reference
>> version for each package in the distribution which have its own share of
>> problems (one of them being that we don't use those tested by upstream
>> as stated in [4]).
>
> I think this is the eternal conflict between distributions and
> code-bases. As a software engineer, I am a fan of vendoring. It removes
> entire classes of issues:
>
> - Dependency on remote servers to be up to fetch dependencies and
>   perform builds
> - Requiring some kind of system to pin versions
> - Needing to stay on top of releases even if that doesn't meet your
>   schedule or needs
>
> As a packager for a distribution, I dislike vendoring because of the
> reasons you outlined above, _but_ I also dislike building upstream
> software with versions of dependencies that weren't approved, tested,
> and verified, upstream. It seems to me like that's a recipe for
> unstable, maybe even insecure, software.
>
> Normally, distributions are forced to do this because their packaging
> and build systems are only set up to support one version at a time. Guix
> can theoretically support all versions, but doesn't want to take on the
> dependency graph explosion this approach would cause.
>
> If we go on historical precedent, and how Guix handles the other
> language ecosystems, we should only have one version of each dependency,
> and build all software which relies on that dependency on the version we
> have packaged. Guix has already begun taking on the difficult challenges
> with this approach, including patching upstream to work with versions of
> dependencies upstream was not built for.

That sounds like a pretty difficult challenge. Seems like it could
quickly become untenable if a commonly used library had some sort of
breaking change between versions and different versions of it were used
by different packages.

I don't think anything is stopping Guix from having multiple versions,
but hand-assigning them would feel pretty manual and error-prone. On the
other hand, like you said, the dependency graph explosion from importing
everything would be overwhelming.

> I dislike it, but I also don't think we should try to solve the broader
> class of issues while trying to implement an importer. It should be a
> larger discussion within the Guix community across _all_ language
> packages about how we might achieve upstream parity while still
> maintaining our goals as a distribution, all while not crippling our
> infrastructure and people :)
>
> That's my viewpoint, but I think we should all defer to some of the
> longer-term maintainers who have helped guide Guix.

You wrote up both sides pretty well, but I couldn't tell if you had a
strong opinion on having go restore dependencies vs Guix packaging
dependencies. You had a lot of strong points for vendoring, but you're
writing an importer...

> Thanks for writing up these options.

+1

Re: packaging a golang package

[adfeno--- via]

[-- Attachment #1.1: Type: text/plain, Size: 3708 bytes --]

If by vendoring we mean bundling and also make users fetch data from places not explicitly committed to the GNU FSDG, then allow me to jump in to add some important notes.

Em 27/01/2021 11:31, Katherine Cox-Buday escreveu:
> As a packager for a distribution, I dislike vendoring because of the
> reasons you outlined above, _but_ I also dislike building upstream
> software with versions of dependencies that weren't approved, tested,
> and verified, upstream. It seems to me like that's a recipe for
> unstable, maybe even insecure, software.

I also agree that this would be problematic, but I fear that if we surrender to vendoring, we might defeat the purpose of GNU Guix.

Besides, since GNU Guix is committed to GNU FSDG, the package maintainers (or reviewers at least) would *theoretically* be obligated to observe the GNU FSDG requirements, otherwise the package is considered buggy, and one of those requirements is to guarantee that all functional/practical data/work are free/libre and that non-functional/non-practical data/works grant at least freedom 2 in full (to share and sell unlimited number of copies of the original to anyone for any purpose), all this would require at least a pass/check on the source files, the same check that is normally used in the processes of unbundling/unvendoring. So we might as well cut the path short.

> I dislike it, but I also don't think we should try to solve the broader
> class of issues while trying to implement an importer. It should be a
> larger discussion within the Guix community across _all_ language
> packages about how we might achieve upstream parity while still
> maintaining our goals as a distribution, all while not crippling our
> infrastructure and people :)

I'm OK with the importer approach but, *in my opinion*, I don't think this tackles the true issue described on the 4th paragraph of the “License Rules” described on the GNU FSDG ([1]), this is why I opened Guix bug #45450 ([2]).

If Guix could make at least the suggestion (a) from Guix bug #45450 ([2]) work, I would be amazed, since it would remove the repositories from the individual Guix recipes that provide the single-language package managers.

Despite not being a developer focused on the plethora of single-language package managers out there, and not even being a developer per see, I'm also in favor of coordinating an effort between Guix and other package managers, but I think expressed/explicit commitment to the GNU FSDG by those single-language package managers is a sine qua non for all free/libre system distributions, including GuixSD, which GNU Guix also maintains.


# References


[1]: https://www.gnu.org/distros/free-system-distribution-guidelines.html#license-rules .

[2]: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45450#5 .


-- 
* Ativista do software livre
	* https://libreplanet.org/wiki/User:Adfeno
	* Membro dos grupos avaliadores de
		* Software (Free Software Directory)
		* Distribuições de sistemas (FreedSoftware)
		* Sites (Free JavaScript Action Team)
	* Não sou advogado e não fomento os não livres
* Sempre veja o spam/lixo eletrônico do teu e-mail
	* Ou coloque todos os recebidos na caixa de entrada
* Sempre assino e-mails com OpenPGP
	* Chave pública: vide endereço anterior
	* Qualquer outro pode ser fraude
	* Se não tens OpenPGP, ignore o anexo "signature.asc"
* Ao enviar anexos
	* Docs., planilhas e apresentações: use OpenDocument
	* Outros tipos: vide endereço anterior
* Use protocolos de comunicação federadas
	* Vide endereço anterior
* Mensagens secretas somente via
	* XMPP com OMEMO
	* E-mail criptografado e assinado com OpenPGP


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: packaging a golang package

[Ludovic Courtès]

Hi,

adfeno--- via <help-guix@gnu.org> skribis:

> If by vendoring we mean bundling and also make users fetch data from places not explicitly committed to the GNU FSDG, then allow me to jump in to add some important notes.
>
> Em 27/01/2021 11:31, Katherine Cox-Buday escreveu:
>> As a packager for a distribution, I dislike vendoring because of the
>> reasons you outlined above, _but_ I also dislike building upstream
>> software with versions of dependencies that weren't approved, tested,
>> and verified, upstream. It seems to me like that's a recipe for
>> unstable, maybe even insecure, software.
>
> I also agree that this would be problematic, but I fear that if we surrender to vendoring, we might defeat the purpose of GNU Guix.

I sympathize with that feeling.

It’s definitely a hard problem.  Even Debian, which has been a
lighthouse for many on these matters, recently gave up:

  https://lwn.net/Articles/843313/

I think both Katherine’s concerns and yours are valid.

IMO, the importer should be able to import things recursively and assume
we’re not going to bundle anything.  It’d be up to the packager, then,
to opt out and selectively use bundled copies of dependencies, if and
when that appears necessary.

> I'm OK with the importer approach but, *in my opinion*, I don't think this tackles the true issue described on the 4th paragraph of the “License Rules” described on the GNU FSDG ([1]), this is why I opened Guix bug #45450 ([2]).

IMO, ‘guix import’ does not “steer users towards obtaining any nonfree
information” any more than wget does.  It’s a tool for packagers that
returns a package definition or template thereof, and it’s up to the
packager to decide what to do with it.

Thanks,
Ludo’.

Re: packaging a golang package

[adfeno--- via]

[-- Attachment #1.1: Type: text/plain, Size: 1731 bytes --]

Em 28/01/2021 13:03, Ludovic Courtès escreveu:
> IMO, ‘guix import’ does not “steer users towards obtaining any nonfree
> information” any more than wget does.  It’s a tool for packagers that
> returns a package definition or template thereof, and it’s up to the
> packager to decide what to do with it.

I do agree with you, sorry if for some reason it sounded otherwise. My intention is not to censor the user on that matter. Let me make it clear what I meant in the previous message:

From the bug report I referenced, one can see that what I find strange is that the cargo provided by Guix (installable through `guix package -i rust:cargo') has cargo's default repository enabled. The bug report referenced has some ideas to try to solve this (although I didn't make extensive test to see if they are all possible and doable).


-- 
* Ativista do software livre
	* https://libreplanet.org/wiki/User:Adfeno
	* Membro dos grupos avaliadores de
		* Software (Free Software Directory)
		* Distribuições de sistemas (FreedSoftware)
		* Sites (Free JavaScript Action Team)
	* Não sou advogado e não fomento os não livres
* Sempre veja o spam/lixo eletrônico do teu e-mail
	* Ou coloque todos os recebidos na caixa de entrada
* Sempre assino e-mails com OpenPGP
	* Chave pública: vide endereço anterior
	* Qualquer outro pode ser fraude
	* Se não tens OpenPGP, ignore o anexo "signature.asc"
* Ao enviar anexos
	* Docs., planilhas e apresentações: use OpenDocument
	* Outros tipos: vide endereço anterior
* Use protocolos de comunicação federadas
	* Vide endereço anterior
* Mensagens secretas somente via
	* XMPP com OMEMO
	* E-mail criptografado e assinado com OpenPGP


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: packaging a golang package

[Elais Player]

[-- Attachment #1: Type: text/plain, Size: 3151 bytes --]

Hey I just want to throw out there that I have a very WIP go module importer[1] sitting on my channel. It uses the proxy.golang.org[2] api to fetch the module's dependencies and source. The single module importer works for the most part but I need to fix it so that the recursive importer doesn't fail when it tries to fetch a dependency in a project's go.mod file that isn't proxied.

There are a few problems that should be looked at for my importer.

1.) it does not include metadata or licensing information, so after importing this way a prudent packager will have to go through and add this information for everything that was pulled down. 

2.) I don't think I'm doing a good job of filtering out similar versions of packages, i tried to follow the examples in the cargo importer but I'm not quite sure what all is going on in there. 

3.) It uses a call to `guix environment --ad-hoc go -- go mod edit -json` to parse module dependencies from the go.mod file, maybe this can be done in a more idiomatic way?

I think its a good start if anyone would like to help get it over the hump.

[1] https://git.sr.ht/~elais/orange/tree/master/item/guix/import/go.scm

[2] https://proxy.golang.org

On Thu, Jan 28, 2021, at 21:10, adfeno--- via Development of GNU Guix and the GNU System distribution. wrote:
> Em 28/01/2021 13:03, Ludovic Courtès escreveu:
> > IMO, ‘guix import’ does not “steer users towards obtaining any nonfree
> > information” any more than wget does.  It’s a tool for packagers that
> > returns a package definition or template thereof, and it’s up to the
> > packager to decide what to do with it.
> 
> I do agree with you, sorry if for some reason it sounded otherwise. My intention is not to censor the user on that matter. Let me make it clear what I meant in the previous message:
> 
> From the bug report I referenced, one can see that what I find strange is that the cargo provided by Guix (installable through `guix package -i rust:cargo') has cargo's default repository enabled. The bug report referenced has some ideas to try to solve this (although I didn't make extensive test to see if they are all possible and doable).
> 
> 
> -- 
> * Ativista do software livre
> * https://libreplanet.org/wiki/User:Adfeno
> * Membro dos grupos avaliadores de
> * Software (Free Software Directory)
> * Distribuições de sistemas (FreedSoftware)
> * Sites (Free JavaScript Action Team)
> * Não sou advogado e não fomento os não livres
> * Sempre veja o spam/lixo eletrônico do teu e-mail
> * Ou coloque todos os recebidos na caixa de entrada
> * Sempre assino e-mails com OpenPGP
> * Chave pública: vide endereço anterior
> * Qualquer outro pode ser fraude
> * Se não tens OpenPGP, ignore o anexo "signature.asc"
> * Ao enviar anexos
> * Docs., planilhas e apresentações: use OpenDocument
> * Outros tipos: vide endereço anterior
> * Use protocolos de comunicação federadas
> * Vide endereço anterior
> * Mensagens secretas somente via
> * XMPP com OMEMO
> * E-mail criptografado e assinado com OpenPGP
> 
> 
> 
> *Attachments:*
>  * signature.asc

[-- Attachment #2: Type: text/html, Size: 4146 bytes --]