Golang go-updates feature branch?

Golang go-updates feature branch?

[Leo Famulari]

Hello!

Now that our build farm is running smoothly, I propose we revive the
practice of feature branches, when appropriate.

The first one that I propose is a go-updates branch, where we update the
Go compilers. This will affect ~500 packages.

If I understand correctly, Go is a relatively self-contained reverse
dependency graph within Guix and thus a good candidate for this
approach. It contains the Go libaries, and then some end-user
applications, and they don't really affect other packages. So, it should
be easy to understand when the update is ready to push.

I can set up a jobset on ci.guix.gnu.org if people like the idea.

Leo

Re: Golang go-updates feature branch?

[John Kehayias]

Hi Leo! and hi Guixers,

On Sun, Jan 08, 2023 at 02:22 PM, Leo Famulari wrote:

> Hello!
>
> Now that our build farm is running smoothly, I propose we revive the
> practice of feature branches, when appropriate.
>

Heartily agree here. This has come up a few times on #guix and generally with support (don't let me speak for everyone though). I think the idea of smaller and more frequent feature branches is a great idea: less code changes coming to master to review at once (compared to big core-updates merges), substitutes and closer to master to be easier for people to pull the branch and test, and some areas despite the number of builds would work nicely as a branch than just pushed (and lingering) into core-updates for much later.

> The first one that I propose is a go-updates branch, where we update the
> Go compilers. This will affect ~500 packages.
>
> If I understand correctly, Go is a relatively self-contained reverse
> dependency graph within Guix and thus a good candidate for this
> approach. It contains the Go libaries, and then some end-user
> applications, and they don't really affect other packages. So, it should
> be easy to understand when the update is ready to push.
>

I can't comment on the Go ecosystem, but that sounds good to me.

Another one I would personally like to see is Mesa: they move quickly, newer versions are needed for recent hardware, and despite the number of rebuilds it causes Mesa is careful with removing older support. This is also an area where getting people to test by running their desktop system on a newer version would be helpful, and I think the feature branch approach makes it much easier (i.e. in core-updates there is a ton to test and look for).

Anyway, perhaps that is a separate discussion and part of a larger discussion on number of rebuilds and how we classify changes.

> I can set up a jobset on ci.guix.gnu.org if people like the idea.
>
> Leo

So yes, I'm all for this general approach and think it is one that will make Guix better for the end-user and on the development side.

Thanks for bringing this up Leo, something I've also been wanting to move towards!

John

Re: Golang go-updates feature branch?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 5558 bytes --]

On Sun, Jan 08, 2023 at 10:32:36PM +0000, John Kehayias wrote:
> Heartily agree here. This has come up a few times on #guix and generally with support (don't let me speak for everyone though). I think the idea of smaller and more frequent feature branches is a great idea: less code changes coming to master to review at once (compared to big core-updates merges), substitutes and closer to master to be easier for people to pull the branch and test, and some areas despite the number of builds would work nicely as a branch than just pushed (and lingering) into core-updates for much later.

I agree with all your points. We used to work this way back in the day
when there weren't very many Guix packages. There was always a
'core-updates' job, but if it was limited to packages that truly are
"core". Then, for several years, the build farm was underpowered
relative to the number of packages and it became impractical. I think we
are past that now.

I've attached a couple patches to get started but I need help getting
them to work.

There is 'etc/go-manifest.scm', which does work with `guix package -m
etc/go-manifest.scm --dry-run`.

And there is a diff for 'build-aux/cuirass/evaluate.scm', so that I can
test it with `make cuirass-jobs`.

But, `make cuirass-jobs` crashes with this manifest and I don't know
why:

------
$ make cuirass-jobs
Compiling Scheme modules...
Compiling Scheme modules...
Compiling Scheme modules...
Compiling Scheme modules...
Compiling Scheme modules...
rm -rf "cuirass-jobs"
  GEN      cuirass-jobs
Computing Guix derivation for 'x86_64-linux'... /
In thread:
uncaught throw to %exception: (#<&inferior-exception arguments: (wrong-type-arg "primitive-load" "Wrong type argument in position ~A (expecting ~A): ~S" (1 "string" #f) (#f)) inferior: #<inferior pipe (0 1 1) 7f28f3ffd780> stack: ((#f ("ice-9/boot-9.scm" 1779 13)) (raise-exception ("ice-9/boot-9.scm" 1682 16)) (raise-exception ("ice-9/boot-9.scm" 1684 16)) (primitive-load (#f #f #f)) (save-module-excursion ("ice-9/boot-9.scm" 2835 4)) (#f (#f #f #f)) (map1 ("srfi/srfi-1.scm" 585 17)) (append-map ("srfi/srfi-1.scm" 672 15)) (#f ("gnu/ci.scm" 449 8)) (map1 ("srfi/srfi-1.scm" 585 17)) (append-map ("srfi/srfi-1.scm" 672 15)) (cuirass-jobs ("gnu/ci.scm" 489 4)) (#f ("ice-9/eval.scm" 158 9)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10)) (call-with-prompt ("ice-9/boot-9.scm" 723 2)) (#f (#f #f #f)) (#f ("guix/repl.scm" 98 21)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10)) (with-exception-handler ("ice-9/boot-9.scm" 1746 15)) (#f ("guix/repl.scm" 125 7)))>)
In thread:
uncaught throw to %exception: (#<&inferior-exception arguments: (wrong-type-arg "primitive-load" "Wrong type argument in position ~A (expecting ~A): ~S" (1 "string" #f) (#f)) inferior: #<inferior pipe (0 1 1) 7f28f3ffd740> stack: ((#f ("ice-9/boot-9.scm" 1779 13)) (raise-exception ("ice-9/boot-9.scm" 1682 16)) (raise-exception ("ice-9/boot-9.scm" 1684 16)) (primitive-load (#f #f #f)) (save-module-excursion ("ice-9/boot-9.scm" 2835 4)) (#f (#f #f #f)) (map1 ("srfi/srfi-1.scm" 585 17)) (append-map ("srfi/srfi-1.scm" 672 15)) (#f ("gnu/ci.scm" 449 8)) (map1 ("srfi/srfi-1.scm" 585 17)) (append-map ("srfi/srfi-1.scm" 672 15)) (cuirass-jobs ("gnu/ci.scm" 489 4)) (#f ("ice-9/eval.scm" 158 9)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10)) (call-with-prompt ("ice-9/boot-9.scm" 723 2)) (#f (#f #f #f)) (#f ("guix/repl.scm" 98 21)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10)) (with-exception-handler ("ice-9/boot-9.scm" 1746 15)) (#f ("guix/repl.scm" 125 7)))>)
In thread:
uncaught throw to %exception: (#<&inferior-exception arguments: (wrong-type-arg "primitive-load" "Wrong type argument in position ~A (expecting ~A): ~S" (1 "string" #f) (#f)) inferior: #<inferior pipe (0 1 1) 7f28f3ffd7c0> stack: ((#f ("ice-9/boot-9.scm" 1779 13)) (raise-exception ("ice-9/boot-9.scm" 1682 16)) (raise-exception ("ice-9/boot-9.scm" 1684 16)) (primitive-load (#f #f #f)) (save-module-excursion ("ice-9/boot-9.scm" 2835 4)) (#f (#f #f #f)) (map1 ("srfi/srfi-1.scm" 585 17)) (append-map ("srfi/srfi-1.scm" 672 15)) (#f ("gnu/ci.scm" 449 8)) (map1 ("srfi/srfi-1.scm" 585 17)) (append-map ("srfi/srfi-1.scm" 672 15)) (cuirass-jobs ("gnu/ci.scm" 489 4)) (#f ("ice-9/eval.scm" 158 9)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10)) (call-with-prompt ("ice-9/boot-9.scm" 723 2)) (#f (#f #f #f)) (#f ("guix/repl.scm" 98 21)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10)) (with-exception-handler ("ice-9/boot-9.scm" 1746 15)) (#f ("guix/repl.scm" 125 7)))>)
In thread:
uncaught throw to %exception: (#<&inferior-exception arguments: (wrong-type-arg "primitive-load" "Wrong type argument in position ~A (expecting ~A): ~S" (1 "string" #f) (#f)) inferior: #<inferior pipe (0 1 1) 7f28f3ffd9c0> stack: ((#f ("ice-9/boot-9.scm" 1779 13)) (raise-exception ("ice-9/boot-9.scm" 1682 16)) (raise-exception ("ice-9/boot-9.scm" 1684 16)) (primitive-load (#f #f #f)) (save-module-excursion ("ice-9/boot-9.scm" 2835 4)) (#f (#f #f #f)) (map1 ("srfi/srfi-1.scm" 585 17)) (append-map ("srfi/srfi-1.scm" 672 15)) (#f ("gnu/ci.scm" 449 8)) (map1 ("srfi/srfi-1.scm" 585 17)) (append-map ("srfi/srfi-1.scm" 672 15)) (cuirass-jobs ("gnu/ci.scm" 489 4)) (#f ("ice-9/eval.scm" 158 9)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10)) (call-with-prompt ("ice-9/boot-9.scm" 723 2)) (#f (#f #f #f)) (#f ("guix/repl.scm" 98 21)) (with-exception-handler ("ice-9/boot-9.scm" 1751 10)) (with-exception-handler ("ice-9/boot-9.scm" 1746 15)) (#f ("guix/repl.scm" 125 7)))>)
------

[-- Attachment #2: evaluate.scm.diff --]
[-- Type: text/plain, Size: 807 bytes --]

diff --git a/build-aux/cuirass/evaluate.scm b/build-aux/cuirass/evaluate.scm
index 7ae5c266d1..92743508df 100644
--- a/build-aux/cuirass/evaluate.scm
+++ b/build-aux/cuirass/evaluate.scm
@@ -96,7 +96,9 @@ (define derivation
                             inferior store
                             `(lambda (store)
                                (cuirass-jobs store
-                                             '((subset . all)
+                                             '((subset
+                                                . (manifests
+                                                   "etc/go-manifest.scm"))
                                                (systems . ,(list system))
                                                (channels . ,channels))))))
                           (file

[-- Attachment #3: go-manifest.scm --]
[-- Type: text/plain, Size: 1222 bytes --]

;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2023 Leo Famulari <leo@famulari.name>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(use-modules (guix packages)
             (guix profiles)
             (gnu packages)
             (guix build-system go))

(manifest
  (map package->manifest-entry
       (fold-packages
         (lambda (package result)
           (if (or (eq? (package-build-system package) go-build-system)
                   (equal? (package-name package) "go"))
             (cons package result)
             result))
         '())))

Re: Golang go-updates feature branch?

[Katherine Cox-Buday]

Leo Famulari <leo@famulari.name> writes:

> Now that our build farm is running smoothly, I propose we revive the
> practice of feature branches, when appropriate.

It was on my TODO list to bring this up on the mailing list! Thank you, Leo!

I also think this is a fantastic idea, and like John, I think mesa is another good candidate (and not only because I recently would have benefited from this).

> The first one that I propose is a go-updates branch, where we update
> the Go compilers. This will affect ~500 packages.
>
> If I understand correctly, Go is a relatively self-contained reverse
> dependency graph within Guix and thus a good candidate for this
> approach. It contains the Go libaries, and then some end-user
> applications, and they don't really affect other packages. So, it
> should be easy to understand when the update is ready to push.
>
> I can set up a jobset on ci.guix.gnu.org if people like the idea.

This was also on my TODO list for this week, and I would love to
coordinate! I'm still pretty unfamiliar with the email based workflow
and managing our CI, so this could be a great mentoring opportunity!

The things I wanted to do:

* Dust off 55210 to support a `go-next` package.

There are two uses for compilers: (1) Building Guix packages (2)
Installing in profiles for use. For (2), in addition to feature
branches, it would be nice to be able to quickly release the latest Go
package without having to immediately worry about testing all of Guix.
The idea with 55210 was to try and do this. I think this is the first
thing we should do, and we should land Go 1.19 as the first `go-next`.

We should also try and 

* Bring the default Go in Guix up to Go 1.9

* Update/create some packages

I have these in my channel I use for staging upstream changes. You can
see them from this line down:

https://github.com/kat-co/guix-channels/blob/5c17bdd6c4ae801effdf1069df69ec4e2ce0a0dd/upstream/packages/golang.scm#L429

- go-golang-org-x-vuln
- go-mvdan-cc-unparam
- go-github-com-google-go-cmdtest
- go-github-com-client9-misspell
- go-mvdan-cc-xurls-v2
- go-mvdan-cc-gofumpt
- go-github-com-google-go-cmp-cmp
- go-github-com-jba-templatecheck
- go-github-com-google-safehtml
- go-github-com-jba-printsrc
- go-golang-org-x-exp
- go-golang-org-x-mod
- go-golang-org-x-sync

There are more packages there I need to upstream, but my goal was to get
the following bullet-points into master:

* Create a package for `gopls` (Go's Language Server Protocol daemon)

I've already done the work here:

https://github.com/kat-co/guix-channels/blob/5c17bdd6c4ae801effdf1069df69ec4e2ce0a0dd/upstream/packages/golang.scm#L817-L860

* Create a home service for running gopls

I've already done the work here:

https://github.com/kat-co/guix-channels/blob/upstream-staging/upstream/home/services/gopls.scm

Go 1.20 is supposed to be released in February, so we'd have a chance to exercise our new Go feature-branch for 1.19 and then quickly 1.20.

I'll reach out on IRC to coordinate!

-- 
Katherine

Re: Golang go-updates feature branch?

[Leo Famulari]

On Wed, Jan 11, 2023 at 10:11:09AM -0700, Katherine Cox-Buday wrote:
> I also think this is a fantastic idea, and like John, I think mesa is another good candidate (and not only because I recently would have benefited from this).

Yeah, mesa is *the* archetypal package for the 'staging' branch, since
its reverse dependency graph is too big to rebuild on the 'master'
branch but it's not a "core" package. If we do go down that road, we
will still need to be judicious about how often we do it. Mesa has ~4000
dependents and we don't want to boil the ocean.

Before we can continue, I / we have to debug the patches I sent a few
messages ago:

https://lists.gnu.org/archive/html/guix-devel/2023-01/msg00131.html

Paging guix-sysadmin for assistance...

Re: Golang go-updates feature branch?

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

Hi Leo,

Leo Famulari <leo@famulari.name> writes:

> And there is a diff for 'build-aux/cuirass/evaluate.scm', so that I can
> test it with `make cuirass-jobs`.
>
> But, `make cuirass-jobs` crashes with this manifest and I don't know
> why:

The inferior that's used in 'build-aux/cuirass/evaluate.scm' is built
from the current check-out by first adding the checkout to the store and
then building it the usual way.  However, that copy only selects files
that belong to the git tree using the git-predicate.  If you add the
manifest to the git checkout by first committing it, it should proceed
happily (at least it did over here).

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Golang go-updates feature branch?

[Leo Famulari]

On Tue, Feb 14, 2023 at 10:22:07PM +0100, Josselin Poiret wrote:
> The inferior that's used in 'build-aux/cuirass/evaluate.scm' is built
> from the current check-out by first adding the checkout to the store and
> then building it the usual way.  However, that copy only selects files
> that belong to the git tree using the git-predicate.  If you add the
> manifest to the git checkout by first committing it, it should proceed
> happily (at least it did over here).

Thank you for your help Josselin! It's much appreciated.

As part of our effort to move towards a "feature branch" development
workflow, it will be useful to collect these tips so that everyone can
create and test their own manifests and Cuirass specs.

I wonder where it should go: a blog post, the cookbook, the Guix manual,
etc?

Re: Golang go-updates feature branch?

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

Hi Leo,

Leo Famulari <leo@famulari.name> writes:

> Thank you for your help Josselin! It's much appreciated.

Happy to help!

> As part of our effort to move towards a "feature branch" development
> workflow, it will be useful to collect these tips so that everyone can
> create and test their own manifests and Cuirass specs.
>
> I wonder where it should go: a blog post, the cookbook, the Guix manual,
> etc?

What's the reason behind branch-specific manifests? I'd imagine we'd
want to test that Guix as a whole still works, even when upgrading just
specific parts.  Otherwise, I guess this shouldn't qualify for a blog
post, maybe the cookbook but I'd even just add this as a comment at the
very top of `build-aux/cuirass/evaluate.scm`, since that's where people
will go looking for it.  If it does end up becoming widespread, perhaps
a section of the manual/cookbook dedicated to how feature branches work
could be a nice home for these bits of info.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Golang go-updates feature branch?

[Leo Famulari]

On Thu, Feb 16, 2023 at 09:05:42PM +0100, Josselin Poiret wrote:
> What's the reason behind branch-specific manifests? I'd imagine we'd
> want to test that Guix as a whole still works, even when upgrading just
> specific parts.  Otherwise, I guess this shouldn't qualify for a blog
> post, maybe the cookbook but I'd even just add this as a comment at the
> very top of `build-aux/cuirass/evaluate.scm`, since that's where people
> will go looking for it.  If it does end up becoming widespread, perhaps
> a section of the manual/cookbook dedicated to how feature branches work
> could be a nice home for these bits of info.

Well, now that you ask, I guess there's not a good reason to use
manifests here.

After creating the kernel-updates CI job based on a manifest, I figured
we'd use the same pattern for this, but I agree it's a bit clunky.

On the other hand, Cuirass does tend to obscure the salient information
when testing a branch that doesn't rebuild the world (i.e.
core-updates).

For example, for kernel updates, I want to know 1) if the kernel
packages built and 2) do the system tests still pass? If I simply asked
Cuirass to build everything on the kernel-updates branch, it might end
up showing me the result of a bunch of unrelated package builds just
because they happened to be selected as part of this jobset rather than
'master', due to (un)lucky timing. Does that make sense?

Zooming out, our tooling still has a very long way to go. There are so
many crucial features of Nix's Hydra (which we used previously) that we
are still missing from Cuirass, and it's made it *much* more difficult
to efficiently and carefully perform big updates to Guix, even though
the build farm hardware is more capable than when we used to run Hydra.
This was all so much easier with Hydra.

Sorry for the rambling response that went off-topic. This has been on my
mind.

Re: Golang go-updates feature branch?

[Simon Tournier]

Hi,

On Thu, 16 Feb 2023 at 11:54, Leo Famulari <leo@famulari.name> wrote:
> On Tue, Feb 14, 2023 at 10:22:07PM +0100, Josselin Poiret wrote:
>> The inferior that's used in 'build-aux/cuirass/evaluate.scm' is built
>> from the current check-out by first adding the checkout to the store and
>> then building it the usual way.  However, that copy only selects files
>> that belong to the git tree using the git-predicate.  If you add the
>> manifest to the git checkout by first committing it, it should proceed
>> happily (at least it did over here).

[...]

> I wonder where it should go: a blog post, the cookbook, the Guix manual,
> etc?

Maybe it could start with the Cookbook, waiting there is enough coherent
materials.

Cheers,
simon