From mboxrd@z Thu Jan 1 00:00:00 1970 From: Leo Famulari Subject: Re: Go packaging Date: Wed, 4 Oct 2017 10:22:25 -0400 Message-ID: <20171004142225.GA2547@jasmine.lan> References: <20171003151504.GA27166@jasmine.lan> <20171004041918.GA21664@thebird.nl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="x+6KMIRAuhnl3hBn" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:41213) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dzkZl-0002BS-49 for guix-devel@gnu.org; Wed, 04 Oct 2017 10:23:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dzkZg-0001I3-C0 for guix-devel@gnu.org; Wed, 04 Oct 2017 10:23:05 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:48121) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dzkZf-0001Ew-UN for guix-devel@gnu.org; Wed, 04 Oct 2017 10:23:00 -0400 Content-Disposition: inline In-Reply-To: <20171004041918.GA21664@thebird.nl> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Pjotr Prins Cc: guix-devel@gnu.org --x+6KMIRAuhnl3hBn Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Oct 04, 2017 at 06:19:18AM +0200, Pjotr Prins wrote: Thanks for your comments, Pjotr! > Thanks Leo for the explanation. Now I understand why Go programs, such > as the IPFS implementation, have so many dependencies... Yes, so many. As for transitive dependencies... well, I probably won't package IPFS in my free time because of the huge number of packages required. > What I understand now is that packages get built 'lazily' and there is > really no way to force a build - other than running the target > software. Not exactly =E2=80=94 you can build a Go library module with `go install name-of-module`. My packaging of Syncthing's dependencies does this, creating '.a' static library archives, which are used later when building Syncthing itself. https://github.com/lfam/guix/blob/contrib-syncthing/gnu/packages/syncthing.= scm The difficulty for Guix is that we can't build the entire library (all its modules) with a single command. As far as I can tell, we'd have to recurse the filesystem tree of the library source code and try building each directory. It sounds unreliable to me. > I noticed the hash values in dependencies, so these are git checkouts > (i.e., they are versioned, but not in the standard sense). Right, there are useful "version" references in the form of the Git commit hashes, but the problem is that it seems each Go application uses a different commit. > When I read the package.json file it includes things like: >=20 > "gxDependencies": [ > { > "hash": "QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52", > "name": "go-log", > "version": "1.2.0" > }, > { > "author": "whyrusleeping", > "hash": "QmZfwmhbcgSDGqGaoMMYx8jxBGauZw75zPjnZAyfwPso7M", > "name": "go-libp2p-secio", > "version": "1.1.8" > }, > { > "author": "whyrusleeping", > "hash": "QmaPbCnUMBohSGo3KnxEa2bHqyJVVeEEcwtqJAYxerieBo", > "name": "go-libp2p-crypto", > "version": "1.5.0" > }, > pac >=20 > indeed, there are even two versions in there for 'whyrusleeping' > deep dependencies ;). (would that be a reference to Ruby's Why?). In this case, I don't know what whyrusleeping refers to. go-libp2p-secio and go-libp2p-crypto are the names of the programs themselves. > On the surface, similar to Rubygems, I think it is no problem to > distribute source packages from Guix and have them compile on the fly. > In a way that is also an interesting model for late optimizations - > something we are lacking in our current infrastructure. What I think > we should do is import above json file and generate GNU binary > packages that are GO source bundles. That package.json file is not a standard thing in the Go world. I've found that Go applications use a variety of dependency manifest formats, or just use Git submodules. > Provided GO can use a live build directory outside the store it will > only compile bundles once, on demand. Ruby 'compiles' or interprets > every time, so that is one up on Go ;). That target directory would be > mutable, so that is a downside, a potential security risk. I considered something like this earlier in this effort, but it's not what I'm proposing now. As you say, it's messy to compile outside of the store at run-time, and I don't think we need to do it. Rather, I think we should have a special go-package procedure, used in the inputs field of the calling application, which would build the relevant library modules of the correct Git commit. Does that make sense? > We ought to have a look at how Nix packaged Go builds because they are > already have a solution. Be interesting to see if they found a way to > compile packages 'greedily', the way Python does it. I looked at their build system itself a few weeks ago when I was still learning how Go compilation works. I agree, it would be fruitful to see how they handle the issues I've raised here. --x+6KMIRAuhnl3hBn Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlnU7p0ACgkQJkb6MLrK fwgQHg/+N0gLHWTlx7BSWpnTuwktzZSiW2H+q77w8Z18iE3w5Ec7SsNa6Hn3jQe8 3N9pP+MzJfZXKF6fXiTaFqknWvoUXMiU4FdpfYhDDfYsaREUjv+BBXw08D4RpD5e Bhxeiw9a0bdwclZKYKq/pC1pOwwFq0FsnGrDpw/SEdt5VYrTQ9Kiht/ZVelCplxD xeFkLw397E1E6V7iuz734Pq+Hc/Tvp0vzQ0kJp8osvDlJNfWocs5NxrkNw2MFCgD /WdMS0GjmLwB1o3onCmtnE6DOPob3a7d1inRxB5KkPHqn2fNcHVM3ywuDSHfl59q QmKUa3hs3FLnOOGhscUUTNXRtWtuj1E5IwD3rSUlppkozeqgXIZICz87EDNFjtjC 5LqkLe4uZJhIQgpAclb4SMHjYoVNO1Cd7ExmYeehTNTvLQfvRibiibAe8DZdSynb W8x54YzUfSqnPZyporC+NlM0jc7xVg4zJVUxyAoDtlVN7CDpamtdztcyNW9VopCd EuHA9/9zd5Jd1i+mtWQ7UNgw5lE30RniYR+QYVS3RHfwi4oOos9A71uKx5RllBvw oXP4/LSCoqEzn8pSMmcJ+uZYP9ktPnPqsEBEQK+XmGhAq4bwMnyqoM4aHgugUG3J tNLKlyPQSyQLHw71HIq+7U7J0GMUEl3Ul4SbmXyUm4RyL2idBe0= =41Ve -----END PGP SIGNATURE----- --x+6KMIRAuhnl3hBn--