Consolidating Go packages

Consolidating Go packages

[Leo Famulari]

We can improve the organization of some of our Go language packages in a
way that will make it easier to use and maintain them, and will also
more closely match the way they are developed and used upstream.

For example, take go-golang-org-x-crypto-pbkdf2 and
go-golang-org-x-crypto-salsa20. These are libraries that can be imported
into Go code independently of each other.

However, they are both part of the upstream Git repository at
<https://golang.org/x/crypto> and it's not idiomatic to distribute them
separately, or to allow their versions to diverge. There are other
consequences that are seriously unidiomatic, such as needing to make
Guix packages for "internal" components that really should not be
exposed at all. Overall, it complicates our Go packaging effort.

The big difference for Guix packagers will be that these library
collection meta-packages like x/crypto will not be compiled at all when
they are "built" — they will just be source code. This is normal when
developing Go software but it may obscure the dependency graph.

Originally we chose to package these things separately because if you
try to compile a Go program that imports one of them, but it can't be
found, the error message specifically complains that the library is
missing, not that the entire x/crypto Git repo is missing.

I think that kind of "exploratory" packaging process, where you try to
build a Guix package and add dependencies one at a time when the build
fails, is pretty common, but it led us astray here.

I'll push a Git branch 'go-consolidation' on Savannah soon and will push
it to master soon-ish, pending your comments.

Re: Consolidating Go packages

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2866 bytes --]

On Wed, Aug 14, 2019 at 03:06:17PM -0400, Leo Famulari wrote:
> We can improve the organization of some of our Go language packages in a
> way that will make it easier to use and maintain them, and will also
> more closely match the way they are developed and used upstream.
> 

So to summarize, while it is techincally possible to "build" them, in
actuality they are used in source code form as part of the build
process.

Or, to put another way, they are unbundled sources that we need to
"rebundle" to build our chosen package.

> For example, take go-golang-org-x-crypto-pbkdf2 and
> go-golang-org-x-crypto-salsa20. These are libraries that can be imported
> into Go code independently of each other.
> 
> However, they are both part of the upstream Git repository at
> <https://golang.org/x/crypto> and it's not idiomatic to distribute them
> separately, or to allow their versions to diverge. There are other
> consequences that are seriously unidiomatic, such as needing to make
> Guix packages for "internal" components that really should not be
> exposed at all. Overall, it complicates our Go packaging effort.
> 
> The big difference for Guix packagers will be that these library
> collection meta-packages like x/crypto will not be compiled at all when
> they are "built" — they will just be source code. This is normal when
> developing Go software but it may obscure the dependency graph.
> 

So, either it should be just a "source snippet" or it should be a hidden
package.

> Originally we chose to package these things separately because if you
> try to compile a Go program that imports one of them, but it can't be
> found, the error message specifically complains that the library is
> missing, not that the entire x/crypto Git repo is missing.
> 
> I think that kind of "exploratory" packaging process, where you try to
> build a Guix package and add dependencies one at a time when the build
> fails, is pretty common, but it led us astray here.
> 
> I'll push a Git branch 'go-consolidation' on Savannah soon and will push
> it to master soon-ish, pending your comments.
> 

With the limited time I've spent with rust packages it seems to me that
rust basically behaves similiarly. One difference to what you've written
above is that an incremental (see what it complains about and add it)
approach does work.  Something I've thought about with rust packages is
unless it provides an actual executable it should be a hidden
package.  I'm guessing no one should ever install the x/cryto package
but its source absolutely needs to be packaged to build other packages
with it.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Consolidating Go packages

[Leo Famulari]

On Thu, Aug 15, 2019 at 11:19:06AM +0300, Efraim Flashner wrote:
> So to summarize, while it is techincally possible to "build" them, in
> actuality they are used in source code form as part of the build
> process.

That's true for Guix but not if you were building Go software outside of
Guix. In that case they would be compiled and re-used through the use of
Go's built-in memoized cache (it's very similar to what Guix does). It's
basically a limitation or bug in our go-build-system that re-using
compiled Go objects does not work.

> So, either it should be just a "source snippet" or it should be a hidden
> package.

Nobody needs to install these libraries but they should be packages so
they show up in the Guix UI (`guix package --show`). I think we can
treat them like other shared library packages. And there is currently
some "building" from our point of view — a package-specific filesystem
hierarchy is created.

Re: Consolidating Go packages

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 509 bytes --]

On Wed, Aug 14, 2019 at 03:06:17PM -0400, Leo Famulari wrote:
> I'll push a Git branch 'go-consolidation' on Savannah soon and will push
> it to master soon-ish, pending your comments.

Here it is:

https://git.savannah.gnu.org/cgit/guix.git/log/?h=wip-go-consolidation

It includes a minor update to Go 1.12.10.

It doesn't address a few cases of Go codebases being split up, such as
tegola, golua, some Docker things, and maybe some more. But it does
handle the important 'x' packages and most of the rest.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]