Updates for Go

Updates for Go

[Katherine Cox-Buday]

Hey all,

Our Go ecosystem is currently in need of a lot of love.

* The Go Team

There is currently no branch for Go updates. I know Leo had tried to get 
one setup at one point[0] but ran into issues. I'm unclear if they were 
ever resolved, but the branch isn't there, and we need one.

There's also no one on Guix's Go team. I've created a patch to add 
myself[1], although my availability has historically been very 
intermittent. Please consider joining since the Go ecosystem contains 
some pretty critical packages (e.g. Docker, syncthing, etc.).

* Go Versions

Go v1.21.0 has just been released. Go's release policy[2] states that 
releases are supported until two major releases have been made. We're 
building everything in Go 1.17.3 by default right now which puts our Go 
ecosystem at risk since our packages won't be using security fixes that 
have since been released.

To help address this, I've proposed[3] a Go v1.21.0 package, but not 
made it the default.

Next, we should get started on compiling everything with a supported 
version of Go. It would be nice to switch the default to Go v1.21.0 
since it has some nice performance improvements. To do this, we'll need 
the feature branch so we can begin working on what's broken.

What do you all think? If it's just me trying to fix broken packages, 
we're going to be on Go v1.17 for a long time!

[0] https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00097.html
[1] https://issues.guix.gnu.org/65314
[2] https://go.dev/doc/devel/release#policy
[3] https://issues.guix.gnu.org/65317

--
Katherine

Re: Updates for Go

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Katherine,

On Tue, Aug 15, 2023 at 12:59 PM Katherine Cox-Buday
<cox.katherine.e@gmail.com> wrote:
>
> There's also no one on Guix's Go team. I've created a patch to add
> myself[1]

Your courage and initiative are inspiring. (Unfortunately, my skills
are lousy.) Anybody with an interest in Golang should please speak up!

Kind regards
Felix

Re: Updates for Go

[Katherine Cox-Buday]

On 8/16/23 11:25 AM, Felix Lechner via Development of GNU Guix and the 
GNU System distribution. wrote:
> Hi Katherine,
> 
> On Tue, Aug 15, 2023 at 12:59 PM Katherine Cox-Buday
> <cox.katherine.e@gmail.com> wrote:
>>
>> There's also no one on Guix's Go team. I've created a patch to add
>> myself[1]
> 
> Your courage and initiative are inspiring. (Unfortunately, my skills
> are lousy.) Anybody with an interest in Golang should please speak up!

Thank you, but I'm just another hacker trying to do her part to maintain 
software she loves. I am always grateful to the other Guix hackers doing 
amazing work!

I tried to emphasize in my message that even if you dislike Go, there 
are some critical packages like Docker that are essentially going 
unmaintained because they are written in Go. *This includes security 
fixes*. As it stands, I cannot recommend people use our packages which 
are compiled by Go v1.17.3 in production. I think every release of Go 
(43 of them!) since that one has contained a security patch.

Even if you dislike Go, but can work your way through a package, please 
consider signing up!

--
Katherine

Re: Updates for Go

[Wilko Meyer]

Hi,

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> Even if you dislike Go, but can work your way through a package,
> please consider signing up!

I started picking up Golang for work related use recently again; have
been somewhat regularly writing it between 2015 and 2018-ish, but always
favored using something like C or Rust over Golang.

That being said, I'd actually be willing to put some time and effort
into Guixes Go ecosystem; even though I haven't been on Guix for that
long and would probably have to read through prior contributions to
golang.scm to get the gist on how the go-build-system and packaging all
things go work and to contribute something useful.

Is there a list of current TODOs somewhere? Or would one start by
bumping packages to build with a more recent/non-EoL go version and see
if that works out?

Best Regards,

Wilko Meyer

Re: Updates for Go

[Nguyễn Gia Phong via Development of GNU Guix and the GNU System distribution.]

On 2023-08-17 at 23:54+02:00, Wilko Meyer wrote:
> Is there a list of current TODOs somewhere? Or would one start
> by bumping packages to build with a more recent/non-EoL go version
> and see if that works out?

Most Go packages are quite dated by a few years,
so that's probably a good idea.

One does not simply drive by a thread without personal agenda,
so I hereby request thy mentorship on adding the senpai IRC client:
https://issues.guix.gnu.org/64222

I've been asking for eyes on #guix on Libera but mostly got ghosted )-;

Re: Updates for Go

[Attila Lendvai]

at one point i tried to compile some large projects written in golang in a reproducible way, and making sure that they use the exact same versions of all their dependencies.

in short: there's a philosophical mismatch between how guix and the golang crowd looks at building go apps. guix likes to explicitly represent every dependency in the guix namespace. golang has its own way of gathering the dependencies (and the nixos crowd don't mind "vendoring" the dependencies).

i've also worked on the golang importer (https://issues.guix.gnu.org/55242 which needs a bit more care). once it works reliably enough, then it'd be possible to import the entire transitive closure of the dependencies into an isolated namespace in guix, which would be a halfway solution between the two conflicting philosophies.

for now i've abandoned that endeavour in favor of adding binary packages to my channel, but i made some notes on the way:

https://issues.guix.gnu.org/43872

a go-build-system patch
http://issues.guix.gnu.org/50227
discussion with iskarian
https://logs.guix.gnu.org/guix/2021-08-31.log#024401
https://logs.guix.gnu.org/guix/2021-08-31.log#180940
<lfam>	iskarian: If you search for my name and "go" or "golang" in the mailing list archives, you should find a lot of discussion
https://savannah.gnu.org/users/lfam
<lfam>	Here's a more recent message from me: https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00227.html
<iskarian>	Ah, I see I've unknowingly repeated you! https://lists.gnu.org/archive/html/guix-patches/2021-08/msg01222.html
<lfam>	Heh, it's gratifying that someone else came to the same conclusion. It means I wasn't totally in the weeds

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“The anxiety children feel at constantly being tested, their fear of failure, punishment, and disgrace, severely reduces their ability both to perceive and to remember, and drives them away from the material being studied into strategies for fooling teachers into thinking they know what they really don't know.”
	— John Holt (1923–1985), 'How Children Learn' (1967)

Re: Updates for Go

[Katherine Cox-Buday]

Summary: these are good things to talk about. I think we should focus on 
using the current approach to get our Go ecosystem into a supported 
state before we talk about these things.

On 8/19/23 5:31 AM, Attila Lendvai wrote:
> at one point i tried to compile some large projects written in golang in a reproducible way, and making sure that they use the exact same versions of all their dependencies.
>
> in short: there's a philosophical mismatch between how guix and the golang crowd looks at building go apps. guix likes to explicitly represent every dependency in the guix namespace. golang has its own way of gathering the dependencies (and the nixos crowd don't mind "vendoring" the dependencies).

Yeah, there's a lot of unpack in this space for sure. To repeat what I 
think you're saying to ensure I understand it, the way I would 
characterize it is that there are two concerns:

1. The version of module dependencies a module is built with

2. How the toolchain resolves dependencies


(1) is not a new or unique problem to Go. Distros and software have 
always had mis-aligned views on what specific versions a package should 
use. Distros want to maintain fewer packages, and developers want to 
reference specific versions of dependencies. Guix is in a a better 
position in that it's theoretically possible to carry many different 
versions of the same package without conflicts. My understanding is that 
the limiting factor now are the resources that go into doing that (e.g. 
build-farms, cognitive overhead, etc.).

I don't think (2) is actually a problem. A lot of projects like to 
"vendor" their dependencies (i.e. check in the version of their 
dependency in the same repository as the host module), and there's valid 
reasons to do that. Fortunately, Go's vendoring is strongly tied to 
specific versions and the code is checksummed to ensure the vendored 
copy isn't modified. So it follows that all Guix has to do is ignore 
vendored dependencies and refer to the exact same version of the Guix 
package.

That is to say, at least the way I've framed it, I think (1) is the root 
of any issue that exists.

> i've also worked on the golang importer (https://issues.guix.gnu.org/55242 which needs a bit more care). once it works reliably enough, then it'd be possible to import the entire transitive closure of the dependencies into an isolated namespace in guix, which would be a halfway solution between the two conflicting philosophies.
Very nice! Our Go importer has improved so much, and further 
improvements such as these will make managing the Go ecosystem even easier.
> for now i've abandoned that endeavour in favor of adding binary packages to my channel, but i made some notes on the way:
>
> https://issues.guix.gnu.org/43872
>
> a go-build-system patch
> http://issues.guix.gnu.org/50227
> discussion with iskarian
> https://logs.guix.gnu.org/guix/2021-08-31.log#024401

I don't think this is an issue specifically with Go. As long as I've 
been involved in software, there has been an issue with N software 
depending on M versions of the same thing.

> https://logs.guix.gnu.org/guix/2021-08-31.log#180940
> <lfam>	iskarian: If you search for my name and "go" or "golang" in the mailing list archives, you should find a lot of discussion
> https://savannah.gnu.org/users/lfam
> <lfam>	Here's a more recent message from me: https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00227.html
> <iskarian>	Ah, I see I've unknowingly repeated you! https://lists.gnu.org/archive/html/guix-patches/2021-08/msg01222.html
> <lfam>	Heh, it's gratifying that someone else came to the same conclusion. It means I wasn't totally in the weeds

The smallest divisible unit of a Go repository that is independently 
distributable is now a Go module. Modules are what are resolvable, 
versioned, and check-summed. As a rule that may have exceptions: Guix 
packages should neither encapsulate anything larger nor smaller than 
that. Some of the messages you're referencing are right around the time 
modules were being reified into the Go ecosystem.

I think there is an opportunity and need for Guix to try and reach a 
consensus on what our primitives and approaches are for the Go ecosystem 
and write them down in the manual. I think our current approach is 
workable, but there's obviously still some confusion and maybe debate to 
be had.

Having said all of that, I think we should focus on using our current 
approach to get everything compiling on a supported version of Go. I 
think our packages and substitutes are probably carrying CVEs that have 
been fixed upstream, and, in my opinion, we need to resolve that ASAP.

WDYT?

Re: Updates for Go

[Katherine Cox-Buday]

On 8/17/23 3:54 PM, Wilko Meyer wrote:
> That being said, I'd actually be willing to put some time and effort
> into Guixes Go ecosystem; even though I haven't been on Guix for that
> long and would probably have to read through prior contributions to
> golang.scm to get the gist on how the go-build-system and packaging all
> things go work and to contribute something useful.
>
> Is there a list of current TODOs somewhere? Or would one start by
> bumping packages to build with a more recent/non-EoL go version and see
> if that works out?

Thank you for volunteering!

I'm not aware of a TODO list anywhere other than the issue tracker 
(https://issues.guix.gnu.org/search?query=golang+is%3Aopen).

Personally, I think the immediate emphasis should be on making bringing 
our Go ecosystem onto a supported version of Go (ideally 1.21.0). If 
there is consensus on that, then ensuring that some of our packages with 
larger dependency graphs compile would be a good place to start.

We also really need a Go branch to host a lot of this work. I can start 
looking at what's needed for that.

It would also be useful to get https://issues.guix.gnu.org/65317 (add 
go-1.21) reviewed, even if you don't have commit access. I've been 
exercising the package since I sent the patch, and I think v3 is correct 
(at least functionally), but it could use more exercising and a review 
of the scheme code.

--

Katherine

Re: Updates for Go

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi,

On Mon, Aug 21, 2023 at 9:11 AM Katherine Cox-Buday
<cox.katherine.e@gmail.com> wrote:
>
> the immediate emphasis should be on making bringing
> our Go ecosystem onto a supported version of Go

From my experience of packaging Gocryptfs in Debian and here, perhaps
some reconsideration should be given to the widely unpopular idea of
using more package functions in Guix. Ending in asterisks, they would
allow packagers, and perhaps even automated tools, to select exactly
the particular versions or commits specified for each prerequisite in
go.mod.

Thank you for all your efforts in making Guix better for everyone!

Kind regards
Felix

Re: Updates for Go

[Katherine Cox-Buday]

On 8/21/23 11:53 AM, Felix Lechner wrote:
> Hi,
>
> On Mon, Aug 21, 2023 at 9:11 AM Katherine Cox-Buday
> <cox.katherine.e@gmail.com> wrote:
>> the immediate emphasis should be on making bringing
>> our Go ecosystem onto a supported version of Go
>  From my experience of packaging Gocryptfs in Debian and here, perhaps
> some reconsideration should be given to the widely unpopular idea of
> using more package functions in Guix. Ending in asterisks, they would
> allow packagers, and perhaps even automated tools, to select exactly
> the particular versions or commits specified for each prerequisite in
> go.mod.
I have questions about this, but for now, I'm going to try and keep 
bringing the conversation back to just getting our ecosystem compiling 
on a supported version of Go.

Re: Updates for Go

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

Hi Felix,

Felix Lechner via "Development of GNU Guix and the GNU System
distribution." <guix-devel@gnu.org> writes:

> From my experience of packaging Gocryptfs in Debian and here, perhaps
> some reconsideration should be given to the widely unpopular idea of
> using more package functions in Guix. Ending in asterisks, they would
> allow packagers, and perhaps even automated tools, to select exactly
> the particular versions or commits specified for each prerequisite in
> go.mod.

Package functions are really nice, but they don't mesh well with CLI
usage, as well as CI: you'd need to also export a "default" version of
each package for those two to work properly.  Also, while having pinned
versions of dependencies upstream seems like the consensus, I'm not sure
we'd like doing that, be it for the exponential CI work that would be
required.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Updates for Go

[Attila Lendvai]

> each package for those two to work properly. Also, while having pinned
> versions of dependencies upstream seems like the consensus, I'm not sure
> we'd like doing that, be it for the exponential CI work that would be
> required.


not arguing either way, FWIW:

- rumour has it that golang compiles very fast, and

- IIUC currently the go build system in guix does not reuse build artifacts, i.e. it recompiles everything for each leaf package.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“What you do speaks so loud I cannot hear what you say.”
	— Ralph Waldo Emerson (1803–1882)

Re: Updates for Go

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Attila,

On Tue, Aug 22, 2023 at 6:14 AM Attila Lendvai <attila@lendvai.name> wrote:
>
> currently the go build system in guix does not reuse build artifacts

Can Golang reuse build artifacts?

Kind regards
Felix

Re: Updates for Go

[Katherine Cox-Buday]

On 8/22/23 8:24 AM, Felix Lechner via Development of GNU Guix and the 
GNU System distribution. wrote:
> Hi Attila,
> 
> On Tue, Aug 22, 2023 at 6:14 AM Attila Lendvai <attila@lendvai.name> wrote:
>>
>> currently the go build system in guix does not reuse build artifacts
> 
> Can Golang reuse build artifacts?

I don't think it's recommended right now. See discussion here:

- https://github.com/golang/go/issues/47788#issuecomment-1019562695
- https://github.com/golang/go/issues/47788#issuecomment-1258773167

Summary:

It sounds like due to the way the compiler can optimize code (e.g. 
inlining), `buildmode=shared` is not recommended, but in the future they 
are looking at allowing linking against a single shared library.

--
Katherine

Re: Updates for Go

[wolf]

[-- Attachment #1: Type: text/plain, Size: 1132 bytes --]

On 2023-08-22 13:14:05 +0000, Attila Lendvai wrote:
> > each package for those two to work properly. Also, while having pinned
> > versions of dependencies upstream seems like the consensus, I'm not sure
> > we'd like doing that, be it for the exponential CI work that would be
> > required.
> 
> 
> not arguing either way, FWIW:
> 
> - rumour has it that golang compiles very fast, and

Usually people claiming this assume there is already a compiler cache present.
The compiler heavily caches the work done.

Sure, golang compiles faster than C++ for example, but anecdotal data point: at
$DAYJOB we had to start persisting the compiler cache to make CI fast enough.

> 
> - IIUC currently the go build system in guix does not reuse build artifacts, i.e. it recompiles everything for each leaf package.
> 
> -- 
> • attila lendvai
> • PGP: 963F 5D5F 45C7 DFCD 0A39
> --
> “What you do speaks so loud I cannot hear what you say.”
> 	— Ralph Waldo Emerson (1803–1882)
> 
> 

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Updates for Go

[Katherine Cox-Buday]

On 8/27/23 9:41 AM, wolf wrote:
> Sure, golang compiles faster than C++ for example, but anecdotal data point: at
> $DAYJOB we had to start persisting the compiler cache to make CI fast enough.

I've seen similar things done at companies. This is perhaps an 
interesting avenue to pursue later: if Go doesn't support 
linked-libraries by that time, maybe our "linked libraries" could be to 
store what the compiler caches. More investigation needed obviously.

But, Go does compile very fast, and that's a metric the Go team tracks 
and tries to improve.

>> - IIUC currently the go build system in guix does not reuse build artifacts, i.e. it recompiles everything for each leaf package.

That is my understanding as well. We bring in the source for any package 
that requires it.

--

Katherine

Re: Updates for Go

[Wilko Meyer]

Hi Katherine,

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> Thank you for volunteering!
>
> I'm not aware of a TODO list anywhere other than the issue tracker
> (https://issues.guix.gnu.org/search?query=golang+is%3Aopen).

I've spend some time during the last days getting familiar with the
go-build-system in guix and how it works internally, and while reading
guix/build/go-build-system.scm I actually found such a list as
commentary from 2018-01-06 (e3900a4d64e):

;; TODO:
;; * Avoid copying dependencies into the build environment and / or avoid using
;; a tmpdir when creating the inputs union.
;; * Use Go modules [4]
;; * Re-use compiled packages [5]
;; * Avoid the go-inputs hack
;; * Stop needing remove-go-references (-trimpath ? )
;; * Remove module packages, only offering the full Git repos? This is
;; more idiomatic, I think, because Go downloads Git repos, not modules.
;; What are the trade-offs?

this is probably not too relevant for now, but maybe it'd be good to see
which of these bullets still apply and move those as issues to the issue
tracker (if they aren't issues yet, haven't checked this).

> Personally, I think the immediate emphasis should be on making
> bringing our Go ecosystem onto a supported version of Go (ideally
> 1.21.0). If there is consensus on that, then ensuring that some of our
> packages with larger dependency graphs compile would be a good place
> to start.

I'd definitely agree on that, bringing guixes go ecosystem to 1.21.0
should be a good and reasonable start. 

> It would also be useful to get https://issues.guix.gnu.org/65317 (add
> go-1.21) reviewed, even if you don't have commit access. I've been
> exercising the package since I sent the patch, and I think v3 is
> correct (at least functionally), but it could use more exercising and
> a review of the scheme code.

I'll try to have a look into this later on. Have to keep this mail short
as my lunchbreak's almost over; but will definitely spend some time on
this later on this day!

Best Regards,

Wilko Meyer

Re: Updates for Go

[david larsson]

On 2023-08-17 16:25, Katherine Cox-Buday wrote:
[..]
> 
> Even if you dislike Go, but can work your way through a package,
> please consider signing up!

Hi,

Im not a fan of Go, but I've wanted to package some Go packages. Ive 
only managed to write 2 packages for my private channel so far, but they 
were simple. If there is a guide or so somewhere explaining how to do 
this, then maybe I could complete and contribute a package that I am 
working on (with its dependencies). In particular I got stuck packaging 
nwg-menu with its dependencies, such as gotk3, which will probably take 
me a long time to finish at my current skill level. I might be able to 
contribute other packages as well in the future when Ive learned more 
about it.

Thank you for giving Go in Guix some attention.

Best regards,
David

Re: Updates for Go

[Katherine Cox-Buday]

On 8/22/23 12:06 PM, david larsson wrote:

> Im not a fan of Go, but I've wanted to package some Go packages. Ive 
> only managed to write 2 packages for my private channel so far, but they 
> were simple. If there is a guide or so somewhere explaining how to do 
> this, then maybe I could complete and contribute a package that I am 
> working on (with its dependencies).

The best we have currently are:

- https://guix.gnu.org/manual/en/guix.html#index-go_002dbuild_002dsystem
- https://guix.gnu.org/manual/en/guix.html#index-go

Guix packages are always welcome! The biggest help at the moment would 
be helping to bring existing packages onto Go v1.21.0 so that we can 
make that the default, and improving the Go importer so that people like 
you can just run `guix import go <module>` most of the time and get a 
working package out of it.

> Thank you for giving Go in Guix some attention.

You're welcome! I love Guix and appreciate the work of all the other 
maintainers. Together, we have something special :)

--
Katherine

Re: Updates for Go

[John Kehayias]

Hi Katherine,

On Wed, Aug 23, 2023 at 10:12 AM, Katherine Cox-Buday wrote:

> On 8/22/23 8:24 AM, Felix Lechner via Development of GNU Guix and the
> GNU System distribution. wrote:
>> Hi Attila,
>>
>> On Tue, Aug 22, 2023 at 6:14 AM Attila Lendvai <attila@lendvai.name> wrote:
>>>
>>> currently the go build system in guix does not reuse build artifacts
>>
>> Can Golang reuse build artifacts?
>
> I don't think it's recommended right now. See discussion here:
>
> - <https://github.com/golang/go/issues/47788#issuecomment-1019562695>
> - <https://github.com/golang/go/issues/47788#issuecomment-1258773167>
>
> Summary:
>
> It sounds like due to the way the compiler can optimize code (e.g.
> inlining), `buildmode=shared` is not recommended, but in the future they
> are looking at allowing linking against a single shared library.

I've not been following in detail this discussion, but where do we currently stand? Is the proposed Go 1.21 patch basically ready? Should we create a branch and build job to start seeing how far we get in making 1.21 the default Go in Guix?

Like others, I have a few random Go packages (a bunch locally I really need to clean up too) and am not familiar with the language and our packaging much. Still, if I can help review/push some patches and get things moving, please let me know.

And thanks for all your work here, it is appreciated!

John

Re: Updates for Go

[Katherine Cox-Buday]

On 8/25/23 6:29 PM, John Kehayias wrote:
> I've not been following in detail this discussion, but where do we currently stand? Is the proposed Go 1.21 patch basically ready?

As far as I know, yes. I've been using it locally since I submitted the 
patch, and things seem to be working as expected. I think the Guile code 
still needs a review, but as far as I know, it's a pretty standard package.

> Should we create a branch and build job to start seeing how far we get in making 1.21 the default Go in Guix?

I think we should do a two-phase process:

1. Check the package in.

2. Begin working on making it default.

There is utility in having the package available for people to use prior 
to it being the default.

> Like others, I have a few random Go packages (a bunch locally I really need to clean up too) and am not familiar with the language and our packaging much. Still, if I can help review/push some patches and get things moving, please let me know.

The following would help:

1. Testing that the new package behaves correctly for them

2. Submitting individual patches for packages they work with to manually 
specify Go v1.21.0 as the version to use. We can then remove this later.

3. Investigating what work will need to be done for our larger Go 
packages with many leaves.

4. Setting up a Go branch to do the work to cut everything over (I've 
volunteered to do this, but I don't know much about it, so if someone 
does and gets to it first that will be a pleasant surprise!).

> And thanks for all your work here, it is appreciated!

You're welcome! Thanks for asking!

--

Katherine