Heads-up: hard reset of the 'staging' branch

Heads-up: hard reset of the 'staging' branch

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1935 bytes --]

Guix,

I have good news and bad news.  The good news is that the new commit
verification infrastructure works great.  'make authenticate' will
verify that all commits were signed by a key that was authorized by
.guix-authorizations at that point in time.

The bad news is that we need to ensure .guix-authorizations has been
updated on any branches that new committers/keys will be pushing to.
Currently the 'staging' branch has one commit
(8229ce3116c1f522c7157ab2dcd50dc2d765686a) signed by a
not-yet-authorized key (it had been authorized on 'master' by
d074f73aacc5a39aed0202d6e45721f53f34a8c0, but that was not yet merged to
'staging' at the time).

To fix it properly without leaving a gap where 'make authenticate' will
fail, we actually need to rewrite the history.  Luckily git supports
rebasing merges(!), and the merge we need was the next commit on that
branch.

I have pushed a 'staging2' branch where I did the following:

 1) git rebase -i --rebase-merges 8229ce3116c1f522c7157ab2dcd50dc2d765686a~

 2) Moved 8229ce3116c1f522c7157ab2dcd50dc2d765686a after
f00270d35a6ca814903a9392caedc29d44959088 (the first merge that includes
.guix-authorizations) -- it was "one step down" in Magits interactive
rebase menu.

 3) "solved" three merge conflicts (actually git rerere remembered the
resolutions, and I could have used git rebase --rerere-autoupdate to
make the process entirely automatic).

I intend to move the current 'staging' branch to 'staging-old', and
rename 'staging2' to 'staging' once I'm fully confident in the result
and resolution.

In the mean time, comments or replications of the experiment welcome.

In other good news, the new pre-push hook proposed by Ludovic in
<https://lists.gnu.org/archive/html/guix-devel/2020-05/msg00371.html>
will eliminate this issue as long as people remember to activate it.
For total confidence we should perform it on the server side though.

Sorry for the inconvenience!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Heads-up: hard reset of the 'staging' branch

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1255 bytes --]

Marius Bakke <marius@gnu.org> writes:

> I have pushed a 'staging2' branch where I did the following:
>
>  1) git rebase -i --rebase-merges 8229ce3116c1f522c7157ab2dcd50dc2d765686a~
>
>  2) Moved 8229ce3116c1f522c7157ab2dcd50dc2d765686a after
> f00270d35a6ca814903a9392caedc29d44959088 (the first merge that includes
> .guix-authorizations) -- it was "one step down" in Magits interactive
> rebase menu.
>
>  3) "solved" three merge conflicts (actually git rerere remembered the
> resolutions, and I could have used git rebase --rerere-autoupdate to
> make the process entirely automatic).
>
> I intend to move the current 'staging' branch to 'staging-old', and
> rename 'staging2' to 'staging' once I'm fully confident in the result
> and resolution.

Of course I sent this before actually testing the branch!  'git rerere'
had forgotten a very important merge resolution from
354880e7209a2aec0360dfe5b08b1873c084e2dd: the "tzdata-for-tests" package
needs to be kept on version 2019c during that merge to prevent a
full rebuild via "python".

I will do a new rebase tomorrow and make sure each of the merges are
correct.

Meanwhile, do not pull 'staging2' unless you are freezing and the 
computer is your main heat source.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Heads-up: hard reset of the 'staging' branch

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 3026 bytes --]

Marius Bakke <marius@gnu.org> writes:

> Marius Bakke <marius@gnu.org> writes:
>
>> I have pushed a 'staging2' branch where I did the following:
>>
>>  1) git rebase -i --rebase-merges 8229ce3116c1f522c7157ab2dcd50dc2d765686a~
>>
>>  2) Moved 8229ce3116c1f522c7157ab2dcd50dc2d765686a after
>> f00270d35a6ca814903a9392caedc29d44959088 (the first merge that includes
>> .guix-authorizations) -- it was "one step down" in Magits interactive
>> rebase menu.
>>
>>  3) "solved" three merge conflicts (actually git rerere remembered the
>> resolutions, and I could have used git rebase --rerere-autoupdate to
>> make the process entirely automatic).
>>
>> I intend to move the current 'staging' branch to 'staging-old', and
>> rename 'staging2' to 'staging' once I'm fully confident in the result
>> and resolution.
>
> Of course I sent this before actually testing the branch!  'git rerere'
> had forgotten a very important merge resolution from
> 354880e7209a2aec0360dfe5b08b1873c084e2dd: the "tzdata-for-tests" package
> needs to be kept on version 2019c during that merge to prevent a
> full rebuild via "python".
>
> I will do a new rebase tomorrow and make sure each of the merges are
> correct.

So the three merges that needed rebasing were:

  f00270d35a6ca814903a9392caedc29d44959088
  1ffd7a7e514072039a1bc92eb1a09b8119b06c91
  34d8640b1a85198fa060a6ddbdbf10ad6e2415ea

During the first, apart from the _actual_ (git style) conflict,
tzdata-for-tests needed manual merging as mentioned earlier.

I solved it by "git checkout f00270d35 -- gnu/packages/base.scm" during
the merge.

For 1ffd7a7e there was another "hidden" (not detected by git) conflict
in gnu/packages/cups.scm, though not as grave (only a leftover
variable).  Solved by "git checkout 1ffd7a7e --
gnu/packages/cups.scm".

34d8640b1 did not have any surprise conflicts.

The new rebased merges are:

  9edb3f66fd807b096b48283debdcddccfea34bad
  aa13c5657d4f8b5dd52beda88a9a8ccc59ebca86
  8a7a5dc7805f4628e60f90af6b2416f951d0c034

They should be identical to the previous merges, with one difference:
the first merge no longer has a conflict in linux.scm because it was
moved to the rebased ddcfc467968d8367e62cb24cbc9e89bb5af0fe6d.

The new 'staging' branch have these rebased merges and is identical to
the previous 'staging' (excluding .git).  The previous branch is
available as 'staging-old' for the time being in case anyone wants to
verify.

In the future I'll be annotating such "hidden" merge conflicts in the
commit message somehow (perhaps using the regular ChangeLog style).  I
hope we don't need to perform branch surgery again, but there are
legitimate cases for making similar merges to different branches.

Another key takeaway is that 'git rerere' does not remember any
"manual" merge resolutions: it only saves resolutions for things that
would introduce the regular git conflict markers.

If you read this far, I'm sorry!  Happy to say the situation is under
control and the staging branch will continue its normal operation
shortly.  :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Heads-up: hard reset of the 'staging' branch

[Ludovic Courtès]

Hi!

Marius Bakke <marius@gnu.org> skribis:

> I have good news and bad news.  The good news is that the new commit
> verification infrastructure works great.  'make authenticate' will
> verify that all commits were signed by a key that was authorized by
> .guix-authorizations at that point in time.
>
> The bad news is that we need to ensure .guix-authorizations has been
> updated on any branches that new committers/keys will be pushing to.
> Currently the 'staging' branch has one commit
> (8229ce3116c1f522c7157ab2dcd50dc2d765686a) signed by a
> not-yet-authorized key (it had been authorized on 'master' by
> d074f73aacc5a39aed0202d6e45721f53f34a8c0, but that was not yet merged to
> 'staging' at the time).
>
> To fix it properly without leaving a gap where 'make authenticate' will
> fail, we actually need to rewrite the history.  Luckily git supports
> rebasing merges(!), and the merge we need was the next commit on that
> branch.

To be clear, it wouldn’t just “leave a gap”: all future commits would
also be rejected.  The authentication code ensures that each commit is
signed by one of the keys authorized in its parent commit(s).  (See the
latest discussions at <https://issues.guix.gnu.org/22883>.)

This is a good opportunity to remind all fellow committers of the latest
changes in that area, which are summarized here:

  https://guix.gnu.org/manual/devel/en/html_node/Commit-Access.html

Please take a look.

SCARY WARNING:

  When ‘guix pull’ runs that authentication code, which I hope will be
  the case in a few weeks, any such mistakes means that users will not
  be able to pull at all, so we all have to be very cautious.  If we do
  make a mistake, we’ll have to reset the branch to a known-good state,
  like you did.

That said… thanks *a lot* for carrying out this rebase, Marius!  I’ve
never done a rebase including merge commits before (sounds scary!), and
I learned that ‘git rerere’ is not a typo.  Thumbs up!

Ludo’.

Re: Heads-up: hard reset of the 'staging' branch

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2745 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi!
>
> Marius Bakke <marius@gnu.org> skribis:
>
>> I have good news and bad news.  The good news is that the new commit
>> verification infrastructure works great.  'make authenticate' will
>> verify that all commits were signed by a key that was authorized by
>> .guix-authorizations at that point in time.
>>
>> The bad news is that we need to ensure .guix-authorizations has been
>> updated on any branches that new committers/keys will be pushing to.
>> Currently the 'staging' branch has one commit
>> (8229ce3116c1f522c7157ab2dcd50dc2d765686a) signed by a
>> not-yet-authorized key (it had been authorized on 'master' by
>> d074f73aacc5a39aed0202d6e45721f53f34a8c0, but that was not yet merged to
>> 'staging' at the time).
>>
>> To fix it properly without leaving a gap where 'make authenticate' will
>> fail, we actually need to rewrite the history.  Luckily git supports
>> rebasing merges(!), and the merge we need was the next commit on that
>> branch.
>
> To be clear, it wouldn’t just “leave a gap”: all future commits would
> also be rejected.  The authentication code ensures that each commit is
> signed by one of the keys authorized in its parent commit(s).  (See the
> latest discussions at <https://issues.guix.gnu.org/22883>.)

Indeed, sorry for being unclear.  The gap I was referring to was based
on a hypothetical situation where we worked around this issue in
git-authenticate.scm, similar to %commits-with-known-bad-signature.

> This is a good opportunity to remind all fellow committers of the latest
> changes in that area, which are summarized here:
>
>   https://guix.gnu.org/manual/devel/en/html_node/Commit-Access.html
>
> Please take a look.
>
> SCARY WARNING:
>
>   When ‘guix pull’ runs that authentication code, which I hope will be
>   the case in a few weeks, any such mistakes means that users will not
>   be able to pull at all, so we all have to be very cautious.  If we do
>   make a mistake, we’ll have to reset the branch to a known-good state,
>   like you did.

I am really looking forward to strong authentication in 'guix pull'.
Sounds like a good excuse to make a new release!  :-)

> That said… thanks *a lot* for carrying out this rebase, Marius!  I’ve
> never done a rebase including merge commits before (sounds scary!), and
> I learned that ‘git rerere’ is not a typo.  Thumbs up!

Rebasing merges was a first for me too, and hopefully a last!  I only
learned about (and enabled) 'git rerere' recently and it has saved me a
lot of time already, mainly when a merge had to be aborted and redone.

But it's a double edged sword, no wonder it's not enabled by default.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Heads-up: hard reset of the 'staging' branch

[Ludovic Courtès]

Hey,

Marius Bakke <marius@gnu.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> To be clear, it wouldn’t just “leave a gap”: all future commits would
>> also be rejected.  The authentication code ensures that each commit is
>> signed by one of the keys authorized in its parent commit(s).  (See the
>> latest discussions at <https://issues.guix.gnu.org/22883>.)
>
> Indeed, sorry for being unclear.  The gap I was referring to was based
> on a hypothetical situation where we worked around this issue in
> git-authenticate.scm, similar to %commits-with-known-bad-signature.

As it turns out, ‘%commits-with-known-bad-signature’ is unused.  :-)
I’m actually reluctant to supporting it now because I don’t see how it
could be implemented without also offering a trivial way to escape
verification.

>> This is a good opportunity to remind all fellow committers of the latest
>> changes in that area, which are summarized here:
>>
>>   https://guix.gnu.org/manual/devel/en/html_node/Commit-Access.html
>>
>> Please take a look.
>>
>> SCARY WARNING:
>>
>>   When ‘guix pull’ runs that authentication code, which I hope will be
>>   the case in a few weeks, any such mistakes means that users will not
>>   be able to pull at all, so we all have to be very cautious.  If we do
>>   make a mistake, we’ll have to reset the branch to a known-good state,
>>   like you did.
>
> I am really looking forward to strong authentication in 'guix pull'.
> Sounds like a good excuse to make a new release!  :-)

Yup!

Ludo’.