From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pjotr Prins Subject: Re: Go packaging Date: Wed, 4 Oct 2017 06:19:18 +0200 Message-ID: <20171004041918.GA21664@thebird.nl> References: <20171003151504.GA27166@jasmine.lan> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:52260) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dzbB7-0000dL-5j for guix-devel@gnu.org; Wed, 04 Oct 2017 00:21:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dzbB3-00078t-5k for guix-devel@gnu.org; Wed, 04 Oct 2017 00:21:01 -0400 Received: from mail.thebird.nl ([95.154.246.10]:54652) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dzbB2-0006sw-Se for guix-devel@gnu.org; Wed, 04 Oct 2017 00:20:57 -0400 Content-Disposition: inline In-Reply-To: <20171003151504.GA27166@jasmine.lan> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Leo Famulari Cc: guix-devel@gnu.org Thanks Leo for the explanation. Now I understand why Go programs, such as the IPFS implementation, have so many dependencies... What I understand now is that packages get built 'lazily' and there is really no way to force a build - other than running the target software. I noticed the hash values in dependencies, so these are git checkouts (i.e., they are versioned, but not in the standard sense). It is an interesting approach - because it does guarantee people are using the same (deep) software stack. I noticed that too: On Tue, Oct 03, 2017 at 11:15:04AM -0400, Leo Famulari wrote: > Based on my work creating a go-build-system and packaging a non-trivial > Go application [0], I want to start a discussion on how we can > efficiently package Go software in Guix. > > Go software is developed rather differently from most of what we > package, and I think our package abstraction does not fit Go libraries > well. > > The primary differences are 0) Go libraries are typically bundled as > source code, 1) Go software is typically unversioned, 2) static archives > (.a) are the norm, and 3) Go libraries are expected to be built as part > of the build process of the calling application. That is, there is no > standard way to build an entire Go library on its own. > > Elaboration: > > 0,1) The problem with every application bundling unversioned libraries > is that we need to package a different Git commit of the library for > every application we package, or risk our packages not working. Adding > package variants per-version is a bit messy currently. > > 3) As an example of how Go libaries are built piecemeal, take the core > networking library, 'golang.org/x/net' [1]. It includes dozens of > submodules such as bpf, icmp, ipv4, ipv6, etc. There is no way to build > all these submodules with a single command. Instead, each one is built > when it is needed during the build process of the calling application. > There are no build scripts. The library compilation process is > standardized as, for example, `go install golang.org/x/net/ipv4`. > > This means that the entire networking library would consist of several > dozen Guix packages, multiplied by the number of different Git commits > required by the calling applications. It's unreasonable, in my opinion. > > My suggestion is that we have two layers of Go library packages: a) a > template layer that includes the source URI, unpack-path, and other > package metadata, and b) a layer to create instances of the package > within the inputs field of the calling package. > > Perhaps the instantiation layer could look like this in practice: > > (define-public my-go-program > [...] > (inputs > `(("golang.org/x/net" > ,(go-package golang-org-x-net > (version "ffcf1bedda") > (import-paths '("golang.org/x/net/ipv4" > "golang.org/x/net/context" > "golang.org/x/net/ipv6" > "golang.org/x/net/internal/iana"))))))) When I read the package.json file it includes things like: "gxDependencies": [ { "hash": "QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52", "name": "go-log", "version": "1.2.0" }, { "author": "whyrusleeping", "hash": "QmZfwmhbcgSDGqGaoMMYx8jxBGauZw75zPjnZAyfwPso7M", "name": "go-libp2p-secio", "version": "1.1.8" }, { "author": "whyrusleeping", "hash": "QmaPbCnUMBohSGo3KnxEa2bHqyJVVeEEcwtqJAYxerieBo", "name": "go-libp2p-crypto", "version": "1.5.0" }, pac indeed, there are even two versions in there for 'whyrusleeping' deep dependencies ;). (would that be a reference to Ruby's Why?). On the surface, similar to Rubygems, I think it is no problem to distribute source packages from Guix and have them compile on the fly. In a way that is also an interesting model for late optimizations - something we are lacking in our current infrastructure. What I think we should do is import above json file and generate GNU binary packages that are GO source bundles. Provided GO can use a live build directory outside the store it will only compile bundles once, on demand. Ruby 'compiles' or interprets every time, so that is one up on Go ;). That target directory would be mutable, so that is a downside, a potential security risk. What you are saying is that, inside the build system, we pull in all packages as sources and do a complete compile - which flies in the face of how dependencies are built independently today. One question is, can you force that full compilation? We ought to have a look at how Nix packaged Go builds because they are already have a solution. Be interesting to see if they found a way to compile packages 'greedily', the way Python does it. Pj.