unofficial mirror of help-guix@gnu.org 
 help / color / mirror / Atom feed
* packaging a golang package
@ 2021-01-08  7:01 Timmy Douglas
  2021-01-08 18:33 ` raingloom
  2021-01-08 19:01 ` Leo Famulari
  0 siblings, 2 replies; 7+ messages in thread
From: Timmy Douglas @ 2021-01-08  7:01 UTC (permalink / raw)
  To: help-guix


I recently installed guix for the first time and I wanted to try to
package my first program. The one I decided to try is written in go and
uses go.mod (https://github.com/coredns/coredns/blob/master/go.mod) for
modules. Running `go build` would normally download those if they don't
exist.

I took a look at a couple of other packages, and it looks like the right
way to do it would be to package the individual modules as seen in
guix/gnu/packages/golang.scm. Has anyone tried automating this sort of
thing? There are almost 40 dependencies...

If some of the packages already exist, but are a different version,
what's the right way to resolve the conflict?

Thanks,



```coredns.scm

(define-module (coredns)
               #:use-module (guix packages)
               #:use-module (guix git-download)
               #:use-module (guix build-system go)
               #:use-module ((guix licenses) #:prefix license:))

(define-public coredns
  (package
   (name "coredns")
   (version "1.8.0")
   (source (origin
            (method git-fetch)
            (uri (git-reference
                  (url "https://github.com/coredns/coredns.git")
                  (commit (string-append "v" version))))
            (file-name (git-file-name name version))
            (sha256
             (base32
              "04hkz70s5i7ndwyg39za3k83amvmi90rkjm8qp3w3a8fbmq4q4y6"))))
   (build-system go-build-system)
   (arguments
    '(#:import-path "github.com/coredns/coredns"))
   (inputs
    `(;fixme
      ))
   (synopsis "CoreDNS is a DNS server/forwarder, written in Go, that chains plugins.")
   (description "CoreDNS is a fast and flexible DNS server. The key word here is flexible: with CoreDNS you are able to do what you want with your DNS data by utilizing plugins. If some functionality is not provided out of the box you can add it by writing a plugin.")
   (home-page "https://github.com/coredns/coredns")
   (license license:asl2.0)))

coredns


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: packaging a golang package
  2021-01-08  7:01 packaging a golang package Timmy Douglas
@ 2021-01-08 18:33 ` raingloom
  2021-01-08 19:01 ` Leo Famulari
  1 sibling, 0 replies; 7+ messages in thread
From: raingloom @ 2021-01-08 18:33 UTC (permalink / raw)
  To: Timmy Douglas; +Cc: help-guix

On Thu, 07 Jan 2021 23:01:49 -0800
Timmy Douglas <mail@timmydouglas.com> wrote:

> I recently installed guix for the first time and I wanted to try to
> package my first program. The one I decided to try is written in go
> and uses go.mod
> (https://github.com/coredns/coredns/blob/master/go.mod) for modules.
> Running `go build` would normally download those if they don't exist.
> 
> I took a look at a couple of other packages, and it looks like the
> right way to do it would be to package the individual modules as seen
> in guix/gnu/packages/golang.scm. Has anyone tried automating this
> sort of thing? There are almost 40 dependencies...

There is a WIP importer:
https://issues.guix.gnu.org/44178


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: packaging a golang package
  2021-01-08  7:01 packaging a golang package Timmy Douglas
  2021-01-08 18:33 ` raingloom
@ 2021-01-08 19:01 ` Leo Famulari
  2021-01-10  0:32   ` Timmy Douglas
  1 sibling, 1 reply; 7+ messages in thread
From: Leo Famulari @ 2021-01-08 19:01 UTC (permalink / raw)
  To: Timmy Douglas, help-guix

On Fri, Jan 8, 2021, at 02:01, Timmy Douglas wrote:
> 
> I recently installed guix for the first time and I wanted to try to
> package my first program. The one I decided to try is written in go and
> uses go.mod (https://github.com/coredns/coredns/blob/master/go.mod) for
> modules. Running `go build` would normally download those if they don't
> exist.
> 
> I took a look at a couple of other packages, and it looks like the right
> way to do it would be to package the individual modules as seen in
> guix/gnu/packages/golang.scm. Has anyone tried automating this sort of
> thing? There are almost 40 dependencies...

Thanks for working on this!

It's true, Go programs usually have a dependency graph that is uncomfortably large — although not impossibly large like Rust.

There is a work-in-progress implementation of a Go package importer that I believe should make it easier:

https://issues.guix.gnu.org/issue/44178

You could try it out and, if it works for you, send those patches, and give feedback on the importer as well :)

As for the versioning, the Guix package tree can include multiple versions of a package. Search for "inherit" in the crates-io.scm package module for some examples. We would like to improve how this works, again for Go and Rust, because it's such a common requirement.


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: packaging a golang package
  2021-01-08 19:01 ` Leo Famulari
@ 2021-01-10  0:32   ` Timmy Douglas
  2021-01-11  6:09     ` Timmy Douglas
  0 siblings, 1 reply; 7+ messages in thread
From: Timmy Douglas @ 2021-01-10  0:32 UTC (permalink / raw)
  To: Leo Famulari, help-guix, Katherine Cox-Buday, Helio Machado

"Leo Famulari" <leo@famulari.name> writes:

> On Fri, Jan 8, 2021, at 02:01, Timmy Douglas wrote:
>> 
>> I recently installed guix for the first time and I wanted to try to
>> package my first program. The one I decided to try is written in go and
>> uses go.mod (https://github.com/coredns/coredns/blob/master/go.mod) for
>> modules. Running `go build` would normally download those if they don't
>> exist.
>> 
>> I took a look at a couple of other packages, and it looks like the right
>> way to do it would be to package the individual modules as seen in
>> guix/gnu/packages/golang.scm. Has anyone tried automating this sort of
>> thing? There are almost 40 dependencies...
>
> Thanks for working on this!
>
> It's true, Go programs usually have a dependency graph that is
> uncomfortably large — although not impossibly large like Rust.
>
> There is a work-in-progress implementation of a Go package importer
> that I believe should make it easier:
>
> https://issues.guix.gnu.org/issue/44178
>
> You could try it out and, if it works for you, send those patches, and
> give feedback on the importer as well :)

Thanks for the pointer!

I tried both Katherine's patch (with dftxbs3e's mini fix for
recursive-import) and Helio's patch (with a git remote add/cherry-pick)
and had trouble with both of them. I spent a little more time with
Katherine's patch so I'll go into that more below.

Part of the issue is that I haven't used Scheme in like 15 years (and
when I did, it wasn't for anything non-trivial). But I'm also really
struggling to debug what's going on:

I run `make && ./pre-inst-env guix import go -r github.com/coredns/coredns`:

Starting download of /tmp/guix-file.pZqXNO
From https://proxy.golang.org/k8s.io/klog/@v/v1.0.0.mod...
 v1.0.0.mod  68B                       88KiB/s 00:00 [##################] 100.0%
metadata#f
Backtrace:
In ice-9/boot-9.scm:
  1736:10 13 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          12 (apply-smob/0 #<thunk 7f98846a3740>)
In ice-9/boot-9.scm:
    718:2 11 (call-with-prompt _ _ #<procedure default-prompt-handle?>)
In ice-9/eval.scm:
    619:8 10 (_ #(#(#<directory (guile-user) 7f98842e0f00>)))
In guix/ui.scm:
  2154:12  9 (run-guix-command _ . _)
In guix/scripts/import.scm:
   120:11  8 (guix-import . _)
In ice-9/eval.scm:
    159:9  7 (_ _)
In guix/import/utils.scm:
   468:27  6 (recursive-import _ #:repo->guix-package _ #:guix-name _ ?)
In srfi/srfi-1.scm:
   586:17  5 (map1 (("k8s.io/klog" #f) ("k8s.io/client-go" #f) (?) ?))
In guix/import/utils.scm:
   457:33  4 (lookup-node "k8s.io/klog" #f)
In guix/utils.scm:
    703:8  3 (call-with-temporary-output-file #<procedure 7f9871b8c7?>)
In ice-9/eval.scm:
   293:34  2 (_ #(#(#(#(#(#(#(#(#<directory ?> ?) ?) ?) ?) ?) ?) ?) ?))
    155:9  1 (_ #(#(#<directory (guix import go) 7f9882267f00>) #f))
In unknown file:
           0 (list-ref #f 1)

ERROR: In procedure list-ref:
In procedure list-ref: Wrong type argument in position 1: #f


I understand that (list-ref #f 1) won't work, but I can't figure out
where that code is based on the information in the backtrace. Seems like
the variables/symbols/filenames are missing everywhere where it counts.

I loaded the code up with #'display and found that I needed to add
another git scs/vcs entry for k8s.io, and that the
fetch-module-meta-data method tries to use regexes to parse html:

root-module-path= k8s.io/klog
line=
line=            <html><head>
line=                  <meta name="go-import"
line=                        content="k8s.io/klog
line=                                 git https://github.com/kubernetes/klog">
line=                  <meta name="go-source"
line=                        content="k8s.io/klog
line=                                 https://github.com/kubernetes/klog
line=                                 https://github.com/kubernetes/klog/tree/master{/dir}
line=                                 https://github.com/kubernetes/klog/blob/master{/dir}/{file}#L{line}">
line=            </head></html>
line=

(define (fetch-module-meta-data module-path)
  "Fetches module meta-data from a module's landing page. This is necessary
because goproxy servers don't currently provide all the information needed to
build a package."
  (let* ((port (http-fetch (string->uri (format #f "https://~a?go-get=1" module-path))))
         (module-metadata #f)
         (meta-tag-prefix "<meta name=\"go-import\" content=\"")
         (meta-tag-prefix-length (string-length meta-tag-prefix)))
    (do ((line (read-line port) (read-line port)))
        ((or (eof-object? line)
             module-metadata))
      (let ((meta-tag-index (string-contains line meta-tag-prefix)))
        (display "line=")
        (display line)
        (newline)
        (when meta-tag-index
          (let* ((start (+ meta-tag-index meta-tag-prefix-length))
                 (end (string-index line #\" start)))
            (set! module-metadata
              (string-split (substring/shared line start end) #\space))))))
    (close-port port)
    module-metadata))


I don't think the regex would match due to content="" being put on
another line--so #f is returned and causes a type exception later. I'm
a little more used to statically typed languages so tracing it back to
there took longer than I would have liked.

Maybe I need to find a video tutorial on writing Scheme for Guix or
similiar? I felt like I was doing the wrong thing with
printf/make/pre-inst-env guix. I tried starting geiser in emacs, but
didn't really know what I should evaluate. It also added an unbearable
lag to typing in the scheme buffer and didn't provide auto-completion or
symbol lookup, so I didn't go too far with it...

Would be interested if Katherine or Helio have any more updates.


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: packaging a golang package
  2021-01-10  0:32   ` Timmy Douglas
@ 2021-01-11  6:09     ` Timmy Douglas
  2021-01-17 13:31       ` Helio Machado
  0 siblings, 1 reply; 7+ messages in thread
From: Timmy Douglas @ 2021-01-11  6:09 UTC (permalink / raw)
  To: Leo Famulari, help-guix, Katherine Cox-Buday, Helio Machado


I also tried Helio's branch, rebased on the latest, with the following
changes:

diff --git a/guix/import/go-modules.scm b/guix/import/go-modules.scm
index 20fc5c9642..811c9f60ae 100644
--- a/guix/import/go-modules.scm
+++ b/guix/import/go-modules.scm
@@ -290,7 +290,7 @@ VERSION, MODULE-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
                                (() "")
                                (_ home-page)))
                  (synopsis ,synopsis)
-                 (description ,(beautify-description description))
+                 (description ,(beautify-description (or description "")))
                  (license ,(match license
                              (() #f)
                              ((license) license)
@@ -327,12 +327,11 @@ specified, attempt to fetch that version. Otherwise, fetch the latest version."
 (define* (go-module-recursive-import name #:optional version)
   "Given a module NAME and, optionally its VERSION, produce package definitions
 for itself and every dependency, recursively."
-  (recursive-import (if version (string-append name "@" version) name) #f
+  (recursive-import (if version (string-append name "@" version) name)
                     #:repo->guix-package
-                    (lambda (name repo)
+                    (lambda* (name #:key repo version)
                       (let-values (((name version) (package-name->name+version name)))
-                            (go-module->guix-package name version))
-                        )
+                        (go-module->guix-package name version)))
                     #:guix-name go-module-name+version->package-name+version))



I liked how Helio's version uses the go proxy to simplify the parsing of
the package lists, but I ran this:

./pre-inst-env guix import go-modules -r github.com/coredns/coredns

and it took what felt like thirty minutes before failing with:

error: failed to retrieve package information from "https://proxy.golang.org/replace/@v/(.info": 410 ("Gone")
Backtrace:
In ice-9/boot-9.scm:
  1736:10 18 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          17 (apply-smob/0 #<thunk 7fe73061b740>)
In ice-9/boot-9.scm:
    718:2 16 (call-with-prompt _ _ #<procedure default-prompt-handle?>)
In ice-9/eval.scm:
    619:8 15 (_ #(#(#<directory (guile-user) 7fe730258f00>)))
In guix/ui.scm:
  2154:12 14 (run-guix-command _ . _)
In guix/scripts/import.scm:
   121:11 13 (guix-import . _)
In ice-9/eval.scm:
    159:9 12 (_ _)
In guix/import/utils.scm:
   464:27 11 (recursive-import _ #:repo->guix-package _ #:guix-name _ ?)
In srfi/srfi-1.scm:
   586:29 10 (map1 (("github.com/davecgh/go-spew@v1.1.1" #f) (# #) ?))
   586:29  9 (map1 (("github.com/google/go-cmp@v0.4.0" #f) ("g?" ?) ?))
   586:29  8 (map1 (("github.com/google/gofuzz@v1.1.0" #f) ("g?" ?) ?))
   586:29  7 (map1 (("github.com/kr/pretty@v0.2.0" #f) ("githu?" ?) ?))
   586:29  6 (map1 (("github.com/spf13/pflag@v1.0.5" #f) ("gol?" ?) ?))
   586:29  5 (map1 (("golang.org/x/tools@v0.0.0-20200505023115?" ?) ?))
   586:29  4 (map1 (("gopkg.in/check.v1@v1.0.0-20190902080502-?" ?) ?))
   586:29  3 (map1 (("k8s.io/klog/v2@v2.2.0" #f) ("sigs.k8s.io?" ?) ?))
   586:29  2 (map1 (("sigs.k8s.io/yaml@v1.2.0" #f) ("replace@(" #f)))
   586:17  1 (map1 (("replace@(" #f)))
In guix/import/utils.scm:
   453:33  0 (lookup-node "replace@(" #f)

guix/import/utils.scm:453:33: In procedure lookup-node:
Wrong number of values returned to continuation (expected 2)


Looks like it ran into the replace syntax and didn't parse it correctly?
https://golang.org/ref/mod#go-mod-file


I sort of wonder if the `git-fetch` portion of go packages should be
something like `git-fetch-and-restore-go-modules` Maybe something like
`go mod vendor` could be used an the hash would be calculated on that?
I'm not familiar enough with the internals of go modules, but I'm not
really seeing the value in generating a bunch of package metadata on all
the module dependencies.


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: packaging a golang package
  2021-01-11  6:09     ` Timmy Douglas
@ 2021-01-17 13:31       ` Helio Machado
  0 siblings, 0 replies; 7+ messages in thread
From: Helio Machado @ 2021-01-17 13:31 UTC (permalink / raw)
  To: Timmy Douglas, Leo Famulari, help-guix, Katherine Cox-Buday,
	Helio Machado

Hi!

Looks like it ran into the replace syntax and didn't parse it correctly?
> https://golang.org/ref/mod#go-mod-file


So it seems. I'll debug it as soon as I can; the import process is
unbearably long.

I sort of wonder if the `git-fetch` portion of go packages should be
> something like `git-fetch-and-restore-go-modules` Maybe something like
> `go mod vendor` could be used an the hash would be calculated on that?
> I'm not familiar enough with the internals of go modules, but I'm not
> really seeing the value in generating a bunch of package metadata on all
> the module dependencies.
>

Guix seems to have a strong opinion about dependency vendoring, but it's
technically viable as long as you don't produce architecture-specific
artifacts when packaging. See https://issues.guix.info/43872 for more
information about the pitfalls I encountered while packaging go-ethereum
the fast way.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: packaging a golang package
@ 2021-01-20  3:27 jgart
  0 siblings, 0 replies; 7+ messages in thread
From: jgart @ 2021-01-20  3:27 UTC (permalink / raw)
  To: help-guix

There's also an emacs package by Ryan Prior to help with the tedium of packaging go dependencies:

https://github.com/ryanprior/emacs-guix-packaging

from the README:

Some packages are complex due to a large number of transitive dependencies. I've
found this to be true of software in the golang ecosystem where it's not
uncommon to see 50+ transitive dependencies, and occasionally many more.

guix-packaging provides some facilities to help track progress on complex packages.

*** Commands

**** ~M-x~ ~guix-packaging-go-mod-to-checklist-dwim~
Turns go module definitions into an org/markdown checklist, suitable to keep
track of packaging progress.

**** ~M-x~ ~guix-packaging-go-mod-to-checkbox~
Replace a single go module definition with a checkbox.


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2021-01-20  3:27 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-08  7:01 packaging a golang package Timmy Douglas
2021-01-08 18:33 ` raingloom
2021-01-08 19:01 ` Leo Famulari
2021-01-10  0:32   ` Timmy Douglas
2021-01-11  6:09     ` Timmy Douglas
2021-01-17 13:31       ` Helio Machado
2021-01-20  3:27 jgart

unofficial mirror of help-guix@gnu.org 

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix-user/0 guix-user/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 guix-user guix-user/ https://yhetil.org/guix-user \
		help-guix@gnu.org
	public-inbox-index guix-user

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.user
	nntp://news.gmane.io/gmane.comp.gnu.guix.user


AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git