Go build system

Go build system

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 846 bytes --]

Now that we have go-1.4 and (almost) go-1.5, we should start thinking
about a go-build-system.

I just wrote my first package using Go, the crude Syncthing package that
is attached. It still needs a lot of work, especially since it builds
Syncthing's dependencies from bundled copies instead of external
packages.

But, it does illustrate some of the assumptions that Go makes when
building. It seems that Go is very particular about directory
structures; it would be better if we could avoid these contortions by
setting some environment variables.

Should Go packages refer to the compiler? This Syncthing package does
retain a reference.

I hope to get some replies from some people who have been building Go
software for longer than 1 day ;)

This is the guide that I used to create this package:
https://docs.syncthing.net/dev/building.html

[-- Attachment #2: 0001-WIP-Add-syncthing.patch --]
[-- Type: text/x-diff, Size: 6865 bytes --]

From 24fad181e128d987cc4066265bbd6e34ff4f5461 Mon Sep 17 00:00:00 2001
From: Leo Famulari <leo@famulari.name>
Date: Mon, 11 Jul 2016 16:37:17 -0400
Subject: [PATCH] WIP: Add syncthing.

* gnu/packages/syncthing.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
---
 gnu/local.mk               |   1 +
 gnu/packages/syncthing.scm | 123 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 124 insertions(+)
 create mode 100644 gnu/packages/syncthing.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index d011844..fe3b1d2 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -322,6 +322,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/suckless.scm			\
   %D%/packages/swig.scm				\
   %D%/packages/sxiv.scm				\
+  %D%/packages/syncthing.scm			\
   %D%/packages/synergy.scm			\
   %D%/packages/task-management.scm		\
   %D%/packages/tbb.scm				\
diff --git a/gnu/packages/syncthing.scm b/gnu/packages/syncthing.scm
new file mode 100644
index 0000000..97ca8ec
--- /dev/null
+++ b/gnu/packages/syncthing.scm
@@ -0,0 +1,123 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2016 Leo Famulari <leo@famulari.name>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages syncthing)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix download)
+  #:use-module (guix git-download)
+  #:use-module (guix licenses)
+  #:use-module (guix packages)
+  #:use-module (gnu packages golang))
+
+(define-public syncthing
+  (package
+    (name "syncthing")
+    (version "v0.13.10")
+    (source (origin
+             (method git-fetch)
+             (uri (git-reference
+                   (url "https://github.com/syncthing/syncthing.git")
+                   (commit version)))
+             (file-name (string-append name "-" version))
+             (sha256
+              (base32
+               "07q3j6mnrza719rnvbkdsmvlkyr2pch5sj2l204m5iy5mxaghpx7"))))
+
+    ;; TODO Make go-build-system.
+    (build-system gnu-build-system)
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (delete 'configure) ; No ./configure script.
+
+         ;; This is the directory structure recommended by Syncthings "guide to
+         ;; building": https://docs.syncthing.net/dev/building.html
+         ;; Can we simplify this step?
+         (replace 'unpack
+           (lambda* (#:key source #:allow-other-keys)
+             (let ((tree "src/github.com/syncthing/syncthing"))
+               (copy-recursively source tree))))
+
+         (add-after 'unpack 'set-env
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let ((source (assoc-ref inputs "source")))
+               (setenv "GOPATH" (getcwd))
+
+               ;; This should control where `go run build.go install` installs
+               ;; things, but it seems to have no effect in this case.
+               (setenv "GOBIN" (assoc-ref outputs "out"))
+
+               ;; Enable use of bundled dependencies. This is a stop-gap
+               ;; until the dependencies are packaged properly.
+               (setenv "GO15VENDOREXPERIMENT" "1"))))
+
+         ;; This is related to the unpack phase. Can it be avoided?
+         (add-before 'build 'chdir
+           (lambda _ (chdir "src/github.com/syncthing/syncthing")))
+
+         (replace 'build
+           (lambda _
+             (zero? (system* "go" "run" "build.go"
+                             ;; Disable Syncthing's built-in updater.
+                             "-no-upgrade"
+                             ;; This might not be necessary if building from a
+                             ;; tarball.
+                             "-version" ,version))))
+         (replace 'check
+           (lambda _
+             (zero? (system* "go" "run" "build.go" "test"))))
+
+         ;; TODO Make this use `go run build.go install`.
+         (replace 'install
+           (lambda _
+             (copy-recursively "bin" (string-append (assoc-ref %outputs "out")
+                                                    "/bin"))))
+         ;; TODO These man pages are generated from a different Git
+         ;; repo, https://github.com/syncthing/docs.
+         (add-after 'install 'install-doc
+           (lambda* (#:key outputs source #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (man1 (string-append out "/share/man/man1"))
+                    (man5 (string-append out "/share/man/man5"))
+                    (man7 (string-append out "/share/man/man7"))
+                    (src (string-append source "/man/")))
+               (install-file (string-append src "syncthing.1") man1)
+               (install-file (string-append src "syncthing-config.5") man5)
+               (install-file (string-append src "syncthing-stignore.5") man5)
+               (install-file (string-append src "syncthing-bep.7") man7)
+               (install-file (string-append src "syncthing-device-ids.7") man7)
+               (install-file (string-append src "syncthing-event-api.7") man7)
+               (install-file (string-append src "syncthing-faq.7") man7)
+               (install-file (string-append src "syncthing-globaldisco.7") man7)
+               (install-file (string-append src "syncthing-localdisco.7") man7)
+               (install-file (string-append src "syncthing-networking.7") man7)
+               (install-file (string-append src "syncthing-relay.7") man7)
+               (install-file (string-append src "syncthing-rest-api.7") man7)
+               (install-file (string-append src "syncthing-security.7") man7)
+               (install-file (string-append src "syncthing-versioning.7") man7)
+               (install-file (string-append src "syncthing.1") man7)
+             #t))))))
+    (native-inputs
+     `(("go" ,go-1.5)))
+    (synopsis "Decentralized filesystem synchronization")
+    (description "Syncthing is a peer-to-peer file synchronization tool that
+supports a wide variety of computing platforms.  It uses the Block Exchange
+Protocol.")
+    (home-page "https://syncthing.net")
+    ;; TODO Either delete the bundled dependencies or list their licenses here.
+    (license mpl2.0)))
-- 
2.9.0

Re: Go build system

[Ludovic Courtès]

Hello!

Leo Famulari <leo@famulari.name> skribis:

> I just wrote my first package using Go, the crude Syncthing package that
> is attached. It still needs a lot of work, especially since it builds
> Syncthing's dependencies from bundled copies instead of external
> packages.
>
> But, it does illustrate some of the assumptions that Go makes when
> building. It seems that Go is very particular about directory
> structures; it would be better if we could avoid these contortions by
> setting some environment variables.

It outlines the command sequence that needs to be run.  I’d suggest
starting from that in ‘go-build-system’.  Let’s make it work for this
package, and then we can adjust if some of the assumptions happened to
be specific to Syncthing.

> Should Go packages refer to the compiler? This Syncthing package does
> retain a reference.

I suppose it keeps a reference to run-time support libraries provided by
the ‘go’ package?

> I hope to get some replies from some people who have been building Go
> software for longer than 1 day ;)

I have infinitely less experience than that ;-) but since nobody replied
I thought I’d share my 2¢.

Ludo’.

Re: Go build system

[Andy Wingo]

On Mon 25 Jul 2016 00:25, ludo@gnu.org (Ludovic Courtès) writes:

>> Should Go packages refer to the compiler? This Syncthing package does
>> retain a reference.
>
> I suppose it keeps a reference to run-time support libraries provided by
> the ‘go’ package?

I believe that unless you specifically compile shared objects, that the
runtime should be statically linked in.  There is still a dynamic link
to libc and some things like that.  But, I could be wrong :)  It could
also be that it embeds the path to the build tools somehow.

</ignorant-reply>

Andy

Re: Go build system

[Leo Famulari]

On Mon, Jul 25, 2016 at 12:25:40AM +0200, Ludovic Courtès wrote:
> It outlines the command sequence that needs to be run.  I’d suggest
> starting from that in ‘go-build-system’.  Let’s make it work for this
> package, and then we can adjust if some of the assumptions happened to
> be specific to Syncthing.

Okay, I'll try it out.

We will have to decide what to do about bundled dependencies. Bundling
the source code of dependencies appears to be standard practice in the
world of Go. The compiler began supporting this directly in the 1.5
series:
https://github.com/golang/go/wiki/PackageManagementTools#go15vendorexperiment

This is the manifest of Syncthing's dependencies, which are bundled in
the same directory:
https://github.com/syncthing/syncthing/blob/master/vendor/manifest

If that manifest is a standard thing, we could make a go-importer that
used it to create new packages.

I don't fully understand Debian's packaging, but it seems that they only
use external packages to provide 2 of 39 dependencies, bootstrap and
font-awesome:
https://anonscm.debian.org/git/pkg-go/packages/syncthing.git/tree/debian/rules#n42

Go & bundling

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> We will have to decide what to do about bundled dependencies. Bundling
> the source code of dependencies appears to be standard practice in the
> world of Go.

Bundling appears to be standard practice in the world.
:-)

> The compiler began supporting this directly in the 1.5 series:
> https://github.com/golang/go/wiki/PackageManagementTools#go15vendorexperiment
>
> This is the manifest of Syncthing's dependencies, which are bundled in
> the same directory:
> https://github.com/syncthing/syncthing/blob/master/vendor/manifest
>
> If that manifest is a standard thing, we could make a go-importer that
> used it to create new packages.

Indeed, that would be pretty cool as it effectively provides an easy way
for users to “unbundle” if they want to.  Much more transparent that
what is often practiced.

> I don't fully understand Debian's packaging, but it seems that they only
> use external packages to provide 2 of 39 dependencies, bootstrap and
> font-awesome:
> https://anonscm.debian.org/git/pkg-go/packages/syncthing.git/tree/debian/rules#n42

It might be that even the bravest have given up.  ;-)

Ludo’.

Re: Go & bundling

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

2016-07-26 0:05 GMT+02:00 Ludovic Courtès <ludo@gnu.org>:

> Leo Famulari <leo@famulari.name> skribis:
>
> > We will have to decide what to do about bundled dependencies. Bundling
> > the source code of dependencies appears to be standard practice in the
> > world of Go.
>
> Bundling appears to be standard practice in the world.
> :-)
>
> > The compiler began supporting this directly in the 1.5 series:
> >
> https://github.com/golang/go/wiki/PackageManagementTools#go15vendorexperiment
> >
> > This is the manifest of Syncthing's dependencies, which are bundled in
> > the same directory:
> > https://github.com/syncthing/syncthing/blob/master/vendor/manifest
> >
> > If that manifest is a standard thing, we could make a go-importer that
> > used it to create new packages.
>
> Indeed, that would be pretty cool as it effectively provides an easy way
> for users to “unbundle” if they want to.  Much more transparent that
> what is often practiced.
>

The "new packages" indicated in the manifest could, in turn, have bundled
dependencies. So the importer should be a recursive one.

Like the one that Jelle is using for npm.

Right ?

[-- Attachment #2: Type: text/html, Size: 1881 bytes --]

Re: Go & bundling

[Alex Griffin]

Leo Famulari <leo@famulari.name> wrote:
> If that manifest is a standard thing, we could make a go-importer that
> used it to create new packages.

The manifest is not standardized, or rather, there are too many
competing standards.

On Mon, Jul 25, 2016, at 08:56 PM, Catonano wrote:
> The "new packages" indicated in the manifest could, in turn, have bundled dependencies. So the importer should be a recursive one.
> Like the one that Jelle is using for npm. 
> 
> Right ?

If we are going to unbundle packages, I think it would be best if our
code ignored any manifest files entirely. Versioning is a mess in the Go
ecosystem, and usually the version specified in them only reflects the
most recent commit that the developers had on their machine when they
ran the vendor command. Rarely is it an actual tagged, stable release.

BTW, the following command will probably be useful. It lists the
dependencies of the current package (whose source is in the current
directory), minus the stdlib.

    go list -f '{{join .Deps "\n"}}' | xargs go list -f '{{if not
    .Standard}}{{.ImportPath}}{{end}}'

I hope that helps!
-- 
Alex Griffin

Re: Go build system

[Christopher Baines]

[-- Attachment #1.1: Type: text/plain, Size: 552 bytes --]

On 25/07/16 20:50, Leo Famulari wrote:
> I don't fully understand Debian's packaging, but it seems that they only
> use external packages to provide 2 of 39 dependencies, bootstrap and
> font-awesome:
> https://anonscm.debian.org/git/pkg-go/packages/syncthing.git/tree/debian/rules#n42

Just in case you are trying to understand the Debian package, be aware
that some files are removed from the tarball with the debian/copyright
FilesExcluded [1].

1:
https://anonscm.debian.org/git/pkg-go/packages/syncthing.git/tree/debian/copyright#n4


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: Go & bundling

[Alex Griffin]

I realized my previous message was probably confusing. Here are some
missing details that you may need to make sense of it.

* Go embeds its dependency information right into the source code.

* The command I gave gets its list of dependencies by parsing source
code, not that manifest file. Also, it lists packages, not repositories.
There may be several packages coming from a single repository.

* Those manifest files, then, are just lists of bundled libraries. This
might be a litte confusing coming from other languages, because it is
not directly analogous to the package manager manifests that other
languages use. They are not written manually, nor does the tooling
necessarily make it easy to keep up-to-date (it may take 2-3 separately
maintained, non-standard tools to keep a vendor/ directory in good
working order). The packages in them may not even be dependencies any
more if the maintainer has not been vigilant.

Ultimately I think a Go importer will be tricky to get right. In some
cases it may even make the most sense to just use what's bundled,
unfortunately...
-- 
Alex Griffin

Re: Go build system

[Leo Famulari]

On Tue, Jul 26, 2016 at 07:33:25AM +0100, Christopher Baines wrote:
> On 25/07/16 20:50, Leo Famulari wrote:
> > https://anonscm.debian.org/git/pkg-go/packages/syncthing.git/tree/debian/rules#n42
> 
> Just in case you are trying to understand the Debian package, be aware
> that some files are removed from the tarball with the debian/copyright
> FilesExcluded [1].

Ah, so they remove most of the dependencies, although not the ones found
under 'gui/', if I understand correctly.

I wonder where they get them from, in that case? Maybe they download
them while building?

I don't want to upgrade my Debian system to Sid to try the package,
since I won't be able to roll back afterwards. Is anyone interested in
trying to install the Debian package if they are running Sid?

Re: Go & bundling

[Leo Famulari]

On Mon, Jul 25, 2016 at 11:06:14PM -0500, Alex Griffin wrote:
> Leo Famulari <leo@famulari.name> wrote:
> > If that manifest is a standard thing, we could make a go-importer that
> > used it to create new packages.
> 
> The manifest is not standardized, or rather, there are too many
> competing standards.

Indeed. I looked into some other Go programs, and I found a variety of
manifest formats.

> If we are going to unbundle packages, I think it would be best if our
> code ignored any manifest files entirely. Versioning is a mess in the Go
> ecosystem, and usually the version specified in them only reflects the
> most recent commit that the developers had on their machine when they
> ran the vendor command. Rarely is it an actual tagged, stable release.

Right, it seems that dependencies are referred to by Git commits and
URLs (is it ever not Git?). So, we could end up with *a lot* of
different versions of each library in the package tree.

> BTW, the following command will probably be useful. It lists the
> dependencies of the current package (whose source is in the current
> directory), minus the stdlib.
> 
>     go list -f '{{join .Deps "\n"}}' | xargs go list -f '{{if not
>     .Standard}}{{.ImportPath}}{{end}}'

Thank you! I will play with this.