From mboxrd@z Thu Jan  1 00:00:00 1970
From: Leo Famulari <leo@famulari.name>
Subject: Re: Go packaging
Date: Wed, 4 Oct 2017 10:22:25 -0400
Message-ID: <20171004142225.GA2547@jasmine.lan>
References: <20171003151504.GA27166@jasmine.lan>
	<20171004041918.GA21664@thebird.nl>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="x+6KMIRAuhnl3hBn"
Return-path: <guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org>
Received: from eggs.gnu.org ([2001:4830:134:3::10]:41213)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <leo@famulari.name>) id 1dzkZl-0002BS-49
	for guix-devel@gnu.org; Wed, 04 Oct 2017 10:23:11 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <leo@famulari.name>) id 1dzkZg-0001I3-C0
	for guix-devel@gnu.org; Wed, 04 Oct 2017 10:23:05 -0400
Received: from out1-smtp.messagingengine.com ([66.111.4.25]:48121)
	by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.71) (envelope-from <leo@famulari.name>) id 1dzkZf-0001Ew-UN
	for guix-devel@gnu.org; Wed, 04 Oct 2017 10:23:00 -0400
Content-Disposition: inline
In-Reply-To: <20171004041918.GA21664@thebird.nl>
List-Id: "Development of GNU Guix and the GNU System distribution."
	<guix-devel.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/guix-devel>,
	<mailto:guix-devel-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/guix-devel/>
List-Post: <mailto:guix-devel@gnu.org>
List-Help: <mailto:guix-devel-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/guix-devel>,
	<mailto:guix-devel-request@gnu.org?subject=subscribe>
Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org
Sender: "Guix-devel" <guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org>
To: Pjotr Prins <pjotr.public12@thebird.nl>
Cc: guix-devel@gnu.org


--x+6KMIRAuhnl3hBn
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, Oct 04, 2017 at 06:19:18AM +0200, Pjotr Prins wrote:

Thanks for your comments, Pjotr!

> Thanks Leo for the explanation. Now I understand why Go programs, such
> as the IPFS implementation, have so many dependencies...

Yes, so many. As for transitive dependencies... well, I probably won't
package IPFS in my free time because of the huge number of packages
required.

> What I understand now is that packages get built 'lazily' and there is
> really no way to force a build - other than running the target
> software.

Not exactly =E2=80=94 you can build a Go library module with `go install
name-of-module`. My packaging of Syncthing's dependencies does this,
creating '.a' static library archives, which are used later when
building Syncthing itself.

https://github.com/lfam/guix/blob/contrib-syncthing/gnu/packages/syncthing.=
scm

The difficulty for Guix is that we can't build the entire library (all
its modules) with a single command. As far as I can tell, we'd have to
recurse the filesystem tree of the library source code and try building
each directory. It sounds unreliable to me.

> I noticed the hash values in dependencies, so these are git checkouts
> (i.e., they are versioned, but not in the standard sense).

Right, there are useful "version" references in the form of the Git
commit hashes, but the problem is that it seems each Go application uses
a different commit.

> When I read the package.json file it includes things like:
>=20
>   "gxDependencies": [
>     {
>       "hash": "QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52",
>       "name": "go-log",
>       "version": "1.2.0"
>     },
>     {
>       "author": "whyrusleeping",
>       "hash": "QmZfwmhbcgSDGqGaoMMYx8jxBGauZw75zPjnZAyfwPso7M",
>       "name": "go-libp2p-secio",
>       "version": "1.1.8"
>     },
>     {
>       "author": "whyrusleeping",
>       "hash": "QmaPbCnUMBohSGo3KnxEa2bHqyJVVeEEcwtqJAYxerieBo",
>       "name": "go-libp2p-crypto",
>       "version": "1.5.0"
>     },
> pac
>=20
> indeed, there are even two versions in there for 'whyrusleeping'
> deep dependencies ;). (would that be a reference to Ruby's Why?).

In this case, I don't know what whyrusleeping refers to.
go-libp2p-secio and go-libp2p-crypto are the names of the programs
themselves.

> On the surface, similar to Rubygems, I think it is no problem to
> distribute source packages from Guix and have them compile on the fly.
> In a way that is also an interesting model for late optimizations -
> something we are lacking in our current infrastructure. What I think
> we should do is import above json file and generate GNU binary
> packages that are GO source bundles.

That package.json file is not a standard thing in the Go world.
I've found that Go applications use a variety of dependency manifest
formats, or just use Git submodules.

> Provided GO can use a live build directory outside the store it will
> only compile bundles once, on demand. Ruby 'compiles' or interprets
> every time, so that is one up on Go ;). That target directory would be
> mutable, so that is a downside, a potential security risk.

I considered something like this earlier in this effort, but it's not
what I'm proposing now. As you say, it's messy to compile outside of the
store at run-time, and I don't think we need to do it.

Rather, I think we should have a special go-package procedure, used in
the inputs field of the calling application, which would build the
relevant library modules of the correct Git commit. Does that make
sense?

> We ought to have a look at how Nix packaged Go builds because they are
> already have a solution. Be interesting to see if they found a way to
> compile packages 'greedily', the way Python does it.

I looked at their build system itself a few weeks ago when I was still
learning how Go compilation works. I agree, it would be fruitful to see
how they handle the issues I've raised here.

--x+6KMIRAuhnl3hBn
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlnU7p0ACgkQJkb6MLrK
fwgQHg/+N0gLHWTlx7BSWpnTuwktzZSiW2H+q77w8Z18iE3w5Ec7SsNa6Hn3jQe8
3N9pP+MzJfZXKF6fXiTaFqknWvoUXMiU4FdpfYhDDfYsaREUjv+BBXw08D4RpD5e
Bhxeiw9a0bdwclZKYKq/pC1pOwwFq0FsnGrDpw/SEdt5VYrTQ9Kiht/ZVelCplxD
xeFkLw397E1E6V7iuz734Pq+Hc/Tvp0vzQ0kJp8osvDlJNfWocs5NxrkNw2MFCgD
/WdMS0GjmLwB1o3onCmtnE6DOPob3a7d1inRxB5KkPHqn2fNcHVM3ywuDSHfl59q
QmKUa3hs3FLnOOGhscUUTNXRtWtuj1E5IwD3rSUlppkozeqgXIZICz87EDNFjtjC
5LqkLe4uZJhIQgpAclb4SMHjYoVNO1Cd7ExmYeehTNTvLQfvRibiibAe8DZdSynb
W8x54YzUfSqnPZyporC+NlM0jc7xVg4zJVUxyAoDtlVN7CDpamtdztcyNW9VopCd
EuHA9/9zd5Jd1i+mtWQ7UNgw5lE30RniYR+QYVS3RHfwi4oOos9A71uKx5RllBvw
oXP4/LSCoqEzn8pSMmcJ+uZYP9ktPnPqsEBEQK+XmGhAq4bwMnyqoM4aHgugUG3J
tNLKlyPQSyQLHw71HIq+7U7J0GMUEl3Ul4SbmXyUm4RyL2idBe0=
=41Ve
-----END PGP SIGNATURE-----

--x+6KMIRAuhnl3hBn--