[go-build-system] How to access properties or keys of a package on the build side?

[go-build-system] How to access properties or keys of a package on the build side?

[Frederick Muriithi]

Maybe the following will help clarify the issue:

The package being built needs the sources of the dependenc(y/ies) to
be available in its GOPATH.

The build system needs to set the GOPATH such that the Go build system
will find the the dependencies and build against them.

I have a repository[1] where I'm currently working on definitions for
IPFS before translating them for mainline guix.
If you checkout commit f7f97edc635381384c3532eb8a2bd92cf10a5ac9 and
look at the code there, you can see that I manually set up GOPATH in
the build of go-log so that it can find go-logging (the modified
version[2]).

Now, when building, you need to lay out the code such that go is able
to find everything, and in my code, I do that manually in the
"setup-go-workspace" phase.

I hope this will help clarify issues a bit.

[1] (https://github.com/fredmanglis/guix-ipfs)
[1] (https://github.com/whyrusleeping/go-logging)

-- 
Frederick M. Muriithi

Re: [go-build-system]

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 2981 bytes --]

On Tue, Aug 29, 2017 at 03:54:40PM +0300, Frederick Muriithi wrote:
> Maybe the following will help clarify the issue:
> 
> The package being built needs the sources of the dependenc(y/ies) to
> be available in its GOPATH.
> 
> The build system needs to set the GOPATH such that the Go build system
> will find the the dependencies and build against them.

Right.

> I have a repository[1] where I'm currently working on definitions for
> IPFS before translating them for mainline guix.
> If you checkout commit f7f97edc635381384c3532eb8a2bd92cf10a5ac9 and
> look at the code there, you can see that I manually set up GOPATH in
> the build of go-log so that it can find go-logging (the modified
> version[2]).

IPFS is interesting. I hope we can get it in Guix.

I have a work-in-progress of a go-build-system with packages for
Syncthing up here:

https://github.com/lfam/guix/tree/wip-syncthing

It seems to work until the Syncthing build fails like this:

------
/gnu/store/wr8r001nxr8ryfnw1kkgrmbij4yqldxi-golang-org-x-net-0.0.0-0.ffcf1be/src/golang.org/x/net/internal/netreflect/socket_19.go:15:2: errInvalidType redeclared in this block
        previous declaration at /gnu/store/wr8r001nxr8ryfnw1kkgrmbij4yqldxi-golang-org-x-net-0.0.0-0.ffcf1be/src/golang.org/x/net/internal/netreflect/socket.go:15:2
/gnu/store/wr8r001nxr8ryfnw1kkgrmbij4yqldxi-golang-org-x-net-0.0.0-0.ffcf1be/src/golang.org/x/net/internal/netreflect/socket_19.go:16:2: errOpNoSupport redeclared in this block
        previous declaration at /gnu/store/wr8r001nxr8ryfnw1kkgrmbij4yqldxi-golang-org-x-net-0.0.0-0.ffcf1be/src/golang.org/x/net/internal/netreflect/socket.go:16:2
/gnu/store/wr8r001nxr8ryfnw1kkgrmbij4yqldxi-golang-org-x-net-0.0.0-0.ffcf1be/src/golang.org/x/net/internal/netreflect/socket_19.go:20:37: SocketOf redeclared in this block
        previous declaration at /gnu/store/wr8r001nxr8ryfnw1kkgrmbij4yqldxi-golang-org-x-net-0.0.0-0.ffcf1be/src/golang.org/x/net/internal/netreflect/socket.go:20:37
/gnu/store/wr8r001nxr8ryfnw1kkgrmbij4yqldxi-golang-org-x-net-0.0.0-0.ffcf1be/src/golang.org/x/net/internal/netreflect/socket_19.go:30:49: PacketSocketOf redeclared in this block
        previous declaration at /gnu/store/wr8r001nxr8ryfnw1kkgrmbij4yqldxi-golang-org-x-net-0.0.0-0.ffcf1be/src/golang.org/x/net/internal/netreflect/socket.go:30:49
------

I have a hunch about what might be wrong [0] but do any Go programmers
have some guidance?

[0] It could be related to the use of these package unions:
https://github.com/lfam/guix/blob/wip-syncthing/gnu/packages/syncthing.scm#L1105
The affected package is part of a group of packages that uses a union
like that:
https://github.com/lfam/guix/blob/wip-syncthing/gnu/packages/syncthing.scm#L1845

I wonder if we should instead make a union of all the Go inputs so that
GOPATH has a single entry, rather than just using unions for selected
packages and letting GOPATH be a list of directories.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [go-build-system]

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

On Thu, Aug 31, 2017 at 08:17:15AM -0400, Leo Famulari wrote:
> I have a work-in-progress of a go-build-system with packages for
> Syncthing up here:
> 
> https://github.com/lfam/guix/tree/wip-syncthing
> 
> It seems to work until the Syncthing build fails like this:

I'm *very* happy to report that these issues are resolved. The problem
was caused by the lack of a unique file-name for some of the Syncthing
dependencies source code checkouts. [0]

I'll push an updated branch to my tree on GitHub shortly.

[0] I think there are some dragons there... if not bugs. Report to
follow.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[go-build-system] How to access properties or keys of a package on the build side?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 2467 bytes --]

Recently I made some progress on finishing the prototype go-build-system
from Petter [0], and I need some advice.

AFAICT, building a Go program requires the program's Go dependencies to
have a particular filesystem layout, which corresponds to the way these
dependencies are imported by the program's code.

So, a Go build system needs to somehow create a symlink union of the
dependency graph, making the dependencies available at the correct paths
relative to the root of the build environment. AFAICT, we can't figure
these paths out programatically; they must be supplied by the packager
in advance, for example:

------
(arguments
  `(#:import-path "github.com/AudriusButkevicius/go-nat-pmp"))
------

Petter's prototype creates the symlink union, but instead of using the
#:import-path key, it instead duplicates the import path while listing a
package's dependencies, like this:

------
(inputs
 `(("github.com/AudriusButkevicius/go-nat-pmp"
    ,golang-github-com-audriusbutkevicius-go-nat-pmp)
------

... putting the import path in the car of the input, and using it as shown
below:

------
(define* (symlinking #:key inputs #:allow-other-keys)
  (for-each (lambda (input)
              (let ((import-path (car input))
                    (storepath (cdr input)))
                ; Don't deal with the compiler or source inputs
                (if (and (not (string=? import-path "go"))
                         (not (string=? import-path "source")))
                    (begin
                      (mkdir-p (string-append
                                "src/"
                                (string-take import-path
                                             (string-rindex import-path #\/))))
                      (let ((from (string-append storepath "/src/" import-path))
                            (to (string-append "src/" import-path)))
                        (if (file-exists? to) (delete-file-recursively to))
                        (symlink (string-append
                                  storepath "/src/" import-path)
                                 (string-append
                                  "src/" import-path)))))))
            inputs))
------

Are there any examples in the code base of accessing the inputs' keys? That
seems like a better solution to me, but I'm not sure how to achieve it.

[0]
https://lists.gnu.org/archive/html/guix-devel/2016-12/msg00399.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [go-build-system] How to access properties or keys of a package on the build side?

[Ricardo Wurmus]

Leo Famulari <leo@famulari.name> writes:

> So, a Go build system needs to somehow create a symlink union of the
> dependency graph, making the dependencies available at the correct paths
> relative to the root of the build environment. AFAICT, we can't figure
> these paths out programatically; they must be supplied by the packager
> in advance, for example:
>
> ------
> (arguments
>   `(#:import-path "github.com/AudriusButkevicius/go-nat-pmp"))
> ------
>
> Petter's prototype creates the symlink union, but instead of using the
> #:import-path key, it instead duplicates the import path while listing a
> package's dependencies, like this:
>
> ------
> (inputs
>  `(("github.com/AudriusButkevicius/go-nat-pmp"
>     ,golang-github-com-audriusbutkevicius-go-nat-pmp)
> ------
[…]
>
> Are there any examples in the code base of accessing the inputs' keys? That
> seems like a better solution to me, but I'm not sure how to achieve it.

Instead of recording this information in the input labels (which I
consider rather inelegant) you could put the key in the properties
field.  In the build phase of another package you can access it like
this:

   (arguments
     `(#:phases …
       (lambda _
         …
         ,(assoc-ref (package-properties the-package) 'the-key)
         …
         )…))

You could also access the arguments of another package with
“package-arguments”.  Using the “properties” field isn’t pretty because
it is a free form alist.

You can use “find-tail” to jump to the keyword in “arguments” and then
pick the following value.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [go-build-system] How to access properties or keys of a package on the build side?

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> You could also access the arguments of another package with
> “package-arguments”.  Using the “properties” field isn’t pretty because
> it is a free form alist.
>
> You can use “find-tail” to jump to the keyword in “arguments” and then
> pick the following value.

Here an example:

--8<---------------cut here---------------start------------->8---
(define (assoc-args-ref pkg key)
  "Return the value associated with KEY in the arguments of package PKG,
or #F if there is no such key."
  (let ((ref (find-tail (cut eq? <> key)
                        (package-arguments pkg))))
    (and=> ref second)))

(assoc-args-ref golang-github-com-audriusbutkevicius-go-nat-pmp
                #:import-path)

=> "github.com/AudriusButkevicius/go-nat-pmp"
--8<---------------cut here---------------end--------------->8---


-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: [go-build-system] How to access properties or keys of a package on the build side?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1208 bytes --]

On Mon, Aug 28, 2017 at 10:32:41PM +0200, Ricardo Wurmus wrote:
> 
> Ricardo Wurmus <rekado@elephly.net> writes:
> 
> > You could also access the arguments of another package with
> > “package-arguments”.  Using the “properties” field isn’t pretty because
> > it is a free form alist.
> >
> > You can use “find-tail” to jump to the keyword in “arguments” and then
> > pick the following value.
> 
> Here an example:
> 
> --8<---------------cut here---------------start------------->8---
> (define (assoc-args-ref pkg key)
>   "Return the value associated with KEY in the arguments of package PKG,
> or #F if there is no such key."
>   (let ((ref (find-tail (cut eq? <> key)
>                         (package-arguments pkg))))
>     (and=> ref second)))
> 
> (assoc-args-ref golang-github-com-audriusbutkevicius-go-nat-pmp
>                 #:import-path)
> 
> => "github.com/AudriusButkevicius/go-nat-pmp"
> --8<---------------cut here---------------end--------------->8---

Thanks, this is a nice example.

I realized that the GOPATH variable can be a list of directories, like
PATH, so I'm working in that direction instead of building the symlink
union.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [go-build-system] How to access properties or keys of a package on the build side?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1564 bytes --]

On Mon, Aug 28, 2017 at 10:32:41PM +0200, Ricardo Wurmus wrote:
> 
> Ricardo Wurmus <rekado@elephly.net> writes:
> 
> > You could also access the arguments of another package with
> > “package-arguments”.  Using the “properties” field isn’t pretty because
> > it is a free form alist.
> >
> > You can use “find-tail” to jump to the keyword in “arguments” and then
> > pick the following value.
> 
> Here an example:
> 
> --8<---------------cut here---------------start------------->8---
> (define (assoc-args-ref pkg key)
>   "Return the value associated with KEY in the arguments of package PKG,
> or #F if there is no such key."
>   (let ((ref (find-tail (cut eq? <> key)
>                         (package-arguments pkg))))
>     (and=> ref second)))
> 
> (assoc-args-ref golang-github-com-audriusbutkevicius-go-nat-pmp
>                 #:import-path)
> 
> => "github.com/AudriusButkevicius/go-nat-pmp"
> --8<---------------cut here---------------end--------------->8---

I didn't get very far in this direction before deciding to try another
technique. But wouldn't this not work on the build side, since (guix
packages) is not available there?

I considered building an alist of store items and "import paths" and
passing that into the build from the host side, but it started to feel
like the wrong approach.

Instead my latest approach [0] is to "record" the packages' import paths
via their store items' filesystem layouts.

[0]
https://lists.gnu.org/archive/html/guix-devel/2017-08/msg00323.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [go-build-system] How to access properties or keys of a package on the build side?

[Ricardo Wurmus]

Leo Famulari <leo@famulari.name> writes:

> On Mon, Aug 28, 2017 at 10:32:41PM +0200, Ricardo Wurmus wrote:
>>
>> Ricardo Wurmus <rekado@elephly.net> writes:
>>
>> > You could also access the arguments of another package with
>> > “package-arguments”.  Using the “properties” field isn’t pretty because
>> > it is a free form alist.
>> >
>> > You can use “find-tail” to jump to the keyword in “arguments” and then
>> > pick the following value.
>>
>> Here an example:
>>
>> --8<---------------cut here---------------start------------->8---
>> (define (assoc-args-ref pkg key)
>>   "Return the value associated with KEY in the arguments of package PKG,
>> or #F if there is no such key."
>>   (let ((ref (find-tail (cut eq? <> key)
>>                         (package-arguments pkg))))
>>     (and=> ref second)))
>>
>> (assoc-args-ref golang-github-com-audriusbutkevicius-go-nat-pmp
>>                 #:import-path)
>>
>> => "github.com/AudriusButkevicius/go-nat-pmp"
>> --8<---------------cut here---------------end--------------->8---
>
> I didn't get very far in this direction before deciding to try another
> technique. But wouldn't this not work on the build side, since (guix
> packages) is not available there?

It would work because you’re executing this on the host side and paste
the result into the S-expression that is the value of the “arguments”
field.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net