Branch management, wip-ppc64le, and POWER9

Branch management, wip-ppc64le, and POWER9

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1792 bytes --]

Hi,

Since I'm now making commits on wip-ppc64le to add support for POWER9
machines like the Talos II and Blackbird from Raptor Computing Systems,
I'd like to understand more about how I am expected to manage that
branch.  Before I touched it, it sat with just one lonely commit,
essentially unused.  So it seems to me like I'm not stepping on anyone's
toes, and that I should just make commits like I normally would on
master, staging, or core-updates, being careful to follow the usual
guidelines.

Specifically, I'm curious to know:

- Is it usually expected that wip branches can be rebased?  I don't plan
  to rebase wip-ppc64le, since I'd like to be able to coordinate with
  others using this branch, but I'm not sure what others expect.

- Should I just periodically merge from master, or perhaps core-updates,
  to keep wip-ppc64le up to date and resolve conflicts early?  I've
  already merged master into this branch once, to get it back up to
  date.  I haven't merged core-updates into it yet.  I think
  periodically merging branches into wip-ppc64le is the right thing to
  do, since eventually the end goal will be to merge wip-ppc64le into
  core-updates to get POWER9 support into a release.  However, I suppose
  if I merge from different branches, I could wind up dealing with
  unnecessary conflicts.

- For Efraim specifically, there is some overlap between the work
  required on wip-ppc64le and the work required on wip-ppc.  I have
  already cherry-picked one of your commits
  (2da8fcfdee7cfde8110a68806f3c4d497f217fe5) onto wip-ppc64le (as commit
  52f0b3db6ef3d3c5067c704c2190f72d2994a999), since it was needed.  How
  would you like to coordinate this work?

- Are there any other tips/advice/rules?

Thank you,

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Branch management, wip-ppc64le, and POWER9

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3465 bytes --]


Chris Marusich <cmmarusich@gmail.com> writes:

> Since I'm now making commits on wip-ppc64le to add support for POWER9
> machines like the Talos II and Blackbird from Raptor Computing Systems,
> I'd like to understand more about how I am expected to manage that
> branch.  Before I touched it, it sat with just one lonely commit,
> essentially unused.  So it seems to me like I'm not stepping on anyone's
> toes, and that I should just make commits like I normally would on
> master, staging, or core-updates, being careful to follow the usual
> guidelines.
>
> Specifically, I'm curious to know:
>
> - Is it usually expected that wip branches can be rebased?  I don't plan
>   to rebase wip-ppc64le, since I'd like to be able to coordinate with
>   others using this branch, but I'm not sure what others expect.

I think rebasing introduces issues with commit signatures, since you'd
then be signing others commits. If multiple people are committing to the
branch, I'd treat it like staging/core-updates, and merge rather than
rebasing.

> - Should I just periodically merge from master, or perhaps core-updates,
>   to keep wip-ppc64le up to date and resolve conflicts early?  I've
>   already merged master into this branch once, to get it back up to
>   date.  I haven't merged core-updates into it yet.  I think
>   periodically merging branches into wip-ppc64le is the right thing to
>   do, since eventually the end goal will be to merge wip-ppc64le into
>   core-updates to get POWER9 support into a release.  However, I suppose
>   if I merge from different branches, I could wind up dealing with
>   unnecessary conflicts.

core-updates is pretty broken at the moment (building the channel
derivation fails), so I wouldn't recommend that.

Merging master as soon as conflicts arise sounds good, I think it's
easier to handle the conflicts in smaller batches, so more frequent
merges are useful.

Once both branches are in a good state, maybe then will be the time to
merge core-updates in to wip-ppc64le to test that combination of
changes, and then if that goes well, merge wip-ppc64le in to
core-updates and stop using the wip-ppc64le branch.

> - For Efraim specifically, there is some overlap between the work
>   required on wip-ppc64le and the work required on wip-ppc.  I have
>   already cherry-picked one of your commits
>   (2da8fcfdee7cfde8110a68806f3c4d497f217fe5) onto wip-ppc64le (as commit
>   52f0b3db6ef3d3c5067c704c2190f72d2994a999), since it was needed.  How
>   would you like to coordinate this work?
>
> - Are there any other tips/advice/rules?

I'd really like to develop the Guix Data Service instance here [1] and
associated Guix Build Coordinator instance in to something that can be
used to work with non-master branches.

1: https://data.guix-patches.cbaines.net/

wip-ppc64le is already being processed [2], but ppc64le derivations
aren't being computed. I think adding a change like [3] to the branch
would make sense, and prompt the Guix Data Service to compute those
derivations.

2: https://data.guix-patches.cbaines.net/repository/2/branch/wip-ppc64le
3: https://git.guix-patches.cbaines.net/guix-patches/commit/?h=wip-lle-bout-le1&id=53dcca860787ae6cf3949d9c03c0108c47d47c9e

Once there are computed derivations, then the connected Guix Build
Coordinator instance can start building them. This isn't meant for
providing substitutes to users, but instead for quality assurance. Does
that all make sense?

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Branch management, wip-ppc64le, and POWER9

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

On Thu, Feb 04, 2021 at 09:38:43AM +0000, Christopher Baines wrote:
> > Specifically, I'm curious to know:
> >
> > - Is it usually expected that wip branches can be rebased?  I don't plan
> >   to rebase wip-ppc64le, since I'd like to be able to coordinate with
> >   others using this branch, but I'm not sure what others expect.
> 
> I think rebasing introduces issues with commit signatures, since you'd
> then be signing others commits. If multiple people are committing to the
> branch, I'd treat it like staging/core-updates, and merge rather than
> rebasing.

The signatures will be lost, but for "wip-" branches the expectation has
always been that history may be rewritten. For me, rebasing is a more
comfortable workflow for this kind of exploratory branch. I recommend
against using merges here.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Branch management, wip-ppc64le, and POWER9

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]

On Thu, Feb 04, 2021 at 01:53:37PM -0500, Leo Famulari wrote:
> The signatures will be lost, but for "wip-" branches the expectation has
> always been that history may be rewritten. For me, rebasing is a more
> comfortable workflow for this kind of exploratory branch. I recommend
> against using merges here.

And by the way, one can re-sign an arbitrary range of commits like this:

$ which git-sign
git-sign () {
	case $# in
		("0") range=HEAD  ;;
		("1") range=$1  ;;
		(*) echo "too many arguments" >&2
			return 1 ;;
	esac
	git rebase "$range" --exec "git commit --amend --no-edit --gpg-sign" || git rebase --abort
}

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Branch management, wip-ppc64le, and POWER9

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 5943 bytes --]

Hi,

Christopher Baines <mail@cbaines.net> writes:

> I think rebasing introduces issues with commit signatures, since you'd
> then be signing others commits. If multiple people are committing to the
> branch, I'd treat it like staging/core-updates, and merge rather than
> rebasing.

Leo Famulari <leo@famulari.name> writes:

> The signatures will be lost, but for "wip-" branches the expectation has
> always been that history may be rewritten. For me, rebasing is a more
> comfortable workflow for this kind of exploratory branch. I recommend
> against using merges here.

The original signatures will be lost, yes.  Depending on how you've
configured Git, your own signature may or may not be applied to the
newly recreated commits.  I just tested this in a throwaway repository,
and when I rebase commits, I re-sign my commits. However, that is
probably because I've set commit.gpgSign to true in my git config.
Either way, you're right that it's important to watch out for.

Regarding rebasing vs. merging, both can work.  I think it depends on
who is making the changes.  For now, I would prefer to avoid rebasing on
wip-ppc64le.  If somebody wants to commit, please feel free, but please
make sure the commits are of the same quality that you would normally
expect for changes going to master, core-updates, or staging, since
eventually we will be merging it into those branches.

If we need to rebase wip-ppc64le, that's fine, but let's talk about it
first so nobody (especially me!) is caught off-guard.

For other wip branches, I think whoever is working on them should agree
on how to coordinate.  I'm just focusing on wip-ppc64le right now, since
I want POWER9 support.

Christopher Baines <mail@cbaines.net> writes:

> core-updates is pretty broken at the moment (building the channel
> derivation fails), so I wouldn't recommend that.

This is good to know!  I'll try to find a stable point from which to
merge, when the time comes (which might be soon!).

> Merging master as soon as conflicts arise sounds good, I think it's
> easier to handle the conflicts in smaller batches, so more frequent
> merges are useful.

I agree.  Frequent merges (or rebases) are important for avoiding
conflicts.  Merges from stable points, if possible, are better than
merges from random commits in a branch, though.

> Once both branches are in a good state, maybe then will be the time to
> merge core-updates in to wip-ppc64le to test that combination of
> changes, and then if that goes well, merge wip-ppc64le in to
> core-updates and stop using the wip-ppc64le branch.

On IRC, dftxbs3e also mentioned that it would be unwise to merge from
core-updates at this time for other reasons, beyond simply "guix pull"
possibly being broken.  So we'll be holding off on merging core-updates
into wip-ppc64le just yet.  See here for details (search for "merge"):

https://logs.guix.gnu.org/guix/2021-02-06.log

When wip-ppc64le is in better shape, we can decide what to do.  In the
meantime, dftxbs3e and I agree that it would be best to cherry-pick
essential fixes that are already on core-updates (e.g., I cherry-picked
a48a3f0640d76cb5e5945557c9aae6dabce39d93 onto wip-ppc64le), rather than
merging core-updates prematurely.

>> - For Efraim specifically, there is some overlap between the work
>>   required on wip-ppc64le and the work required on wip-ppc.  I have
>>   already cherry-picked one of your commits
>>   (2da8fcfdee7cfde8110a68806f3c4d497f217fe5) onto wip-ppc64le (as commit
>>   52f0b3db6ef3d3c5067c704c2190f72d2994a999), since it was needed.  How
>>   would you like to coordinate this work?
>>
>> - Are there any other tips/advice/rules?
>
> I'd really like to develop the Guix Data Service instance here [1] and
> associated Guix Build Coordinator instance in to something that can be
> used to work with non-master branches.
>
> 1: https://data.guix-patches.cbaines.net/
>
> wip-ppc64le is already being processed [2], but ppc64le derivations
> aren't being computed. I think adding a change like [3] to the branch
> would make sense, and prompt the Guix Data Service to compute those
> derivations.
>
> 2: https://data.guix-patches.cbaines.net/repository/2/branch/wip-ppc64le
> 3: https://git.guix-patches.cbaines.net/guix-patches/commit/?h=wip-lle-bout-le1&id=53dcca860787ae6cf3949d9c03c0108c47d47c9e
>
> Once there are computed derivations, then the connected Guix Build
> Coordinator instance can start building them. This isn't meant for
> providing substitutes to users, but instead for quality assurance. Does
> that all make sense?

Yes, that makes sense, and I totally agree.  We definitely want to keep
the other architectures in mind, and hopefully this will be a great tool
to help understand any impact there.  I expect our changes to cause
full-world rebuilds for all architectures, since it requires changes in
packages deep in the dependency graph, but none of the changes on
wip-ppc64le should break anything for the other architectures.

I plan to make a change similar to
53dcca860787ae6cf3949d9c03c0108c47d47c9e, in time.  I also want to get
Cuirass working at some point soon so we can more easily gauge the
health of the nascent powerpc64le-linux port.  But for now the most
important thing is probably to make the changes that will allow us to
build guix-binary.powerpc64le-linux.tar.xz, specifically these three
steps:

1) Make changes to guix.m4 etc. so we "officially" support powerpc64le
2) Run make update-guix-package (and commit any changes)
3) Run make guix-binary.powerpc64le-linux.tar.xz

That should give us a distributable binary copy of guix to work with,
which will be helpful.  Right now, I'm just manually running
pre-inst-env from a checkout on a Debian ppc64le machine, where I've
manually cobbled together the dependencies required to build Guix.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]