Core-updates

Core-updates

[Andreas Enge]

Hello,

one more thing that would be nice to have is gtk-doc:
   http://hydra.gnu.org:3000/build/1345571
I already tried to upgrade to 1.25, but that did not change the error.
And maybe node for those who are interested in it:
   http://hydra.gnu.org:3000/build/1326189

And there are a few failures of python packages.

Two more cases of non-determinism: The updated versions of dealii and zsh
built without problem on my x86_64 laptop, but fail on hydra (on x86_64
and i686).

Andreas

core-updates merged!

[Ludovic Courtès]

Hello!

Andreas Enge <andreas@enge.fr> skribis:

> one more thing that would be nice to have is gtk-doc:
>    http://hydra.gnu.org:3000/build/1345571
> I already tried to upgrade to 1.25, but that did not change the error.
> And maybe node for those who are interested in it:
>    http://hydra.gnu.org:3000/build/1326189
>
> And there are a few failures of python packages.

I’m afraid these problems are still there, but overall we’re doing
pretty good, so I merged ‘core-updates’ into ‘master’!

Noteworthy changes include the upgrade to libc 2.23, GNU/Hurd and
cross-compilation improvements, closure size reductions, and lots of
things we’ve long forgotten.  :-)

On GuixSD, to ease transition to libc 2.23, you’ll want to install libc
2.22 locale data globally (info "(guix) Locales"):

--8<---------------cut here---------------start------------->8---
(use-modules (gnu system locale))

(operating-system
  ;; …
  (locale-libcs (cons glibc-2.22 %default-locale-libcs)))
--8<---------------cut here---------------end--------------->8---

Enjoy!

Ludo’.

Re: core-updates merged!

[ng0]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello!
>
> Andreas Enge <andreas@enge.fr> skribis:
>
>> one more thing that would be nice to have is gtk-doc:
>>    http://hydra.gnu.org:3000/build/1345571
>> I already tried to upgrade to 1.25, but that did not change the error.
>> And maybe node for those who are interested in it:
>>    http://hydra.gnu.org:3000/build/1326189
>>
>> And there are a few failures of python packages.
>
> I’m afraid these problems are still there, but overall we’re doing
> pretty good, so I merged ‘core-updates’ into ‘master’!
>
> Noteworthy changes include the upgrade to libc 2.23, GNU/Hurd and
> cross-compilation improvements, closure size reductions, and lots of
> things we’ve long forgotten.  :-)
>
> On GuixSD, to ease transition to libc 2.23, you’ll want to install libc
> 2.22 locale data globally (info "(guix) Locales"):
>
> --8<---------------cut here---------------start------------->8---
> (use-modules (gnu system locale))
>
> (operating-system
>   ;; …
>   (locale-libcs (cons glibc-2.22 %default-locale-libcs)))
> --8<---------------cut here---------------end--------------->8---
>
> Enjoy!
>
> Ludo’.
>

Thanks for pushing this. One question about the comment you made:
Why is this necessary? I rebuilt my systems with this and I get various
"invalid locale" errors now.
-- 
♥Ⓐ  ng0
Current Keys: https://we.make.ritual.n0.is/ng0.txt
For non-prism friendly talk find me on http://www.psyced.org

Re: core-updates merged!

[Ludovic Courtès]

Hello!

ng0 <ng0@we.make.ritual.n0.is> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> On GuixSD, to ease transition to libc 2.23, you’ll want to install libc
>> 2.22 locale data globally (info "(guix) Locales"):
>>
>> --8<---------------cut here---------------start------------->8---
>> (use-modules (gnu system locale))
>>
>> (operating-system
>>   ;; …
>>   (locale-libcs (cons glibc-2.22 %default-locale-libcs)))
>> --8<---------------cut here---------------end--------------->8---
>>
>> Enjoy!
>>
>> Ludo’.
>>
>
> Thanks for pushing this. One question about the comment you made:
> Why is this necessary?

This is necessary for binaries linked against libc 2.22, so they can
access libc 2.22 locale data:

  https://www.gnu.org/software/guix/manual/html_node/Locales.html#Locale-Data-Compatibility-Considerations

> I rebuilt my systems with this and I get various "invalid locale"
> errors now.

As discussed on IRC, SNAFU!  For reasons yet to be elucidated, the
glibc@2.23 package no longer honors /run/current-system/locale.

Commit ab3a64507a792e4da0527b423fbc28f8768e736a works around it by
setting GUIX_LOCPATH=/run/currrent-system/locale on GuixSD.  This is an
acceptable workaround, having no visible drawback.

Please report any other issues!

Ludo’.

Re: core-updates merged!

[Leo Famulari]

On Tue, Aug 02, 2016 at 07:32:23PM +0200, Ludovic Courtès wrote:
> As discussed on IRC, SNAFU!  For reasons yet to be elucidated, the
> glibc@2.23 package no longer honors /run/current-system/locale.

I believe that this commit in glibc@2.23 is the culprit:

http://repo.or.cz/glibc.git/commit/90fe682d3067163aa773feecf497ef599429457a

The variable 'libc_cv_localedir', which we set as
"/run/current-system/locale/" in the glibc/linux package definition, has
been renamed to 'libc_cv_complocaledir'.

Should it be enough to make the following change? This reverts ab3a6450
(system: Define 'GUIX_LOCPATH' to work around 'glibc' package defect.)
and changes the name of the variable.

diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index a476837..bb1879a 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -535,8 +535,7 @@ store.")
             ;;
             ;; `--localedir' is not honored, so work around it.
             ;; See <http://sourceware.org/ml/libc-alpha/2013-03/msg00093.html>.
-            ;; FIXME: This hack no longer works on 2.23!
-            (string-append "libc_cv_localedir=/run/current-system/locale/"
+            (string-append "libc_cv_complocaledir=/run/current-system/locale/"
                            ,version)
 
             (string-append "--with-headers="
diff --git a/gnu/system.scm b/gnu/system.scm
index d6bf6c4..04dd7a8 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -545,12 +545,7 @@ use 'plain-file' instead~%")
 
     ;; By default, applications that use D-Bus, such as Emacs, abort at startup
     ;; when /etc/machine-id is missing.  Make sure these warnings are non-fatal.
-    ("DBUS_FATAL_WARNINGS" . "0")
-
-    ;; XXX: Normally we wouldn't need to do this, but our glibc@2.23 package
-    ;; looks things up in 'PREFIX/lib/locale' instead of
-    ;; '/run/current-system/locale' as was intended.
-    ("GUIX_LOCPATH" . "/run/current-system/locale")))
+    ("DBUS_FATAL_WARNINGS" . "0")))
 
 (define %setuid-programs
   ;; Default set of setuid-root programs.
diff --git a/gnu/tests/base.scm b/gnu/tests/base.scm
index 7170ab1..a6278b2 100644
--- a/gnu/tests/base.scm
+++ b/gnu/tests/base.scm
@@ -178,18 +178,6 @@ info --version")
              '(false-if-exception (getaddrinfo "does-not-exist"))
              marionette))
 
-          (test-equal "locale"
-            "en_US.utf8"
-            (marionette-eval '(begin
-                                ;; XXX: This 'setenv' call wouldn't be needed
-                                ;; but our glibc@2.23 currently ignores
-                                ;; /run/current-system/locale.
-                                (setenv "GUIX_LOCPATH"
-                                        "/run/current-system/locale")
-                                (let ((before (setlocale LC_ALL "en_US.utf8")))
-                                  (setlocale LC_ALL before)))
-                             marionette))
-
           (test-assert "screendump"
             (begin
               (marionette-control (string-append "screendump " #$output

Re: core-updates merged!

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Tue, Aug 02, 2016 at 07:32:23PM +0200, Ludovic Courtès wrote:
>> As discussed on IRC, SNAFU!  For reasons yet to be elucidated, the
>> glibc@2.23 package no longer honors /run/current-system/locale.
>
> I believe that this commit in glibc@2.23 is the culprit:
>
> http://repo.or.cz/glibc.git/commit/90fe682d3067163aa773feecf497ef599429457a
>
> The variable 'libc_cv_localedir', which we set as
> "/run/current-system/locale/" in the glibc/linux package definition, has
> been renamed to 'libc_cv_complocaledir'.

Good catch!

> diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
> index a476837..bb1879a 100644
> --- a/gnu/packages/base.scm
> +++ b/gnu/packages/base.scm
> @@ -535,8 +535,7 @@ store.")
>              ;;
>              ;; `--localedir' is not honored, so work around it.
>              ;; See <http://sourceware.org/ml/libc-alpha/2013-03/msg00093.html>.
> -            ;; FIXME: This hack no longer works on 2.23!
> -            (string-append "libc_cv_localedir=/run/current-system/locale/"
> +            (string-append "libc_cv_complocaledir=/run/current-system/locale/"
>                             ,version)
>  
>              (string-append "--with-headers="
> diff --git a/gnu/system.scm b/gnu/system.scm
> index d6bf6c4..04dd7a8 100644
> --- a/gnu/system.scm
> +++ b/gnu/system.scm
> @@ -545,12 +545,7 @@ use 'plain-file' instead~%")
>  
>      ;; By default, applications that use D-Bus, such as Emacs, abort at startup
>      ;; when /etc/machine-id is missing.  Make sure these warnings are non-fatal.
> -    ("DBUS_FATAL_WARNINGS" . "0")
> -
> -    ;; XXX: Normally we wouldn't need to do this, but our glibc@2.23 package
> -    ;; looks things up in 'PREFIX/lib/locale' instead of
> -    ;; '/run/current-system/locale' as was intended.
> -    ("GUIX_LOCPATH" . "/run/current-system/locale")))
> +    ("DBUS_FATAL_WARNINGS" . "0")))
>  
>  (define %setuid-programs
>    ;; Default set of setuid-root programs.
> diff --git a/gnu/tests/base.scm b/gnu/tests/base.scm
> index 7170ab1..a6278b2 100644
> --- a/gnu/tests/base.scm
> +++ b/gnu/tests/base.scm
> @@ -178,18 +178,6 @@ info --version")
>               '(false-if-exception (getaddrinfo "does-not-exist"))
>               marionette))
>  
> -          (test-equal "locale"
> -            "en_US.utf8"
> -            (marionette-eval '(begin
> -                                ;; XXX: This 'setenv' call wouldn't be needed
> -                                ;; but our glibc@2.23 currently ignores
> -                                ;; /run/current-system/locale.
> -                                (setenv "GUIX_LOCPATH"
> -                                        "/run/current-system/locale")
> -                                (let ((before (setlocale LC_ALL "en_US.utf8")))
> -                                  (setlocale LC_ALL before)))
> -                             marionette))

Here we should keep the test, but remove ‘setenv’:

  (marionette-eval '(let ((before (setlocale LC_ALL "en_US.utf8")))
                      (setlocale LC_ALL before))
                    marionette)

That will catch this regression in the future.

Otherwise LGTM; could you push it to core-updates-next?

Thank you for the fast investigation!

Ludo’.

Re: core-updates merged!

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

On Tue, Aug 02, 2016 at 11:28:59PM +0200, Ludovic Courtès wrote:
> Leo Famulari <leo@famulari.name> skribis:
> Otherwise LGTM; could you push it to core-updates-next?

I merged master into core-updates-next and made the change in a
subsequent commit.

Unfortunately, I can't push the branch to Savannah because it contains
the following unsigned commits (one of them is mine, oops!):

f21403e2b6f5a9491937a0cc9f31fc113998ce5e
9bc84dfea9560c497c91863e7b5021860bd3c254
745ad37a780b28f72ac9dfcc8092317e577e51c9
2d74d94b17b23ab95072da68553d85ac0b3bfede
aebd383d04b351465cfb14e4fd0949b67d4b282e

What should we do?

In the meantime, you can inspect the updated branch on my personal Git
repo:
https://github.com/lfam/guix/commits/core-updates-next

And, since these big merge commits make me nervous, I made notes of all
the conflicts that I had to resolve. They are attached.

[-- Attachment #2: conflicts --]
[-- Type: text/plain, Size: 3041 bytes --]

Before merge, core-updates-next HEAD was
ef21276053b980c9eca8df027408ae85bd6af3d8

(gnu packages admin): No change on core-updates next, package di added
on master.

(gnu packages base): The "previous" glibc package, used to provide
locales for the previously packaged glibc, was named for 2.21 instead of
2.22.

(gnu packages bioinformatics): Package diamond was on version 0.8.7
instead of master's 0.8.17.

Package vsearch was at version 2.0.0 instead of master's 2.0.1.

(gnu packages commencement): New private packages bison-boot0 and
flex-boot0 present on core-updates-next but not on master.

Conflict between 'kernel-headers-boot0' and 'linux-libre-headers-boot0'.
Picked the former since it was deliberately introduced relatively
recently.

(gnu packages cross-base): Conflict between xheaders and xlinux-headers.
Chose xheaders.

(gnu packages crypto): Annoying conflict due to alphabetization of
module imports.
A few conflicts regarding license prefixes.

(gnu packages databases): Version conflict for sqlite.

(gnu packages dav): Version conflict for vdirsyncer.

(gnu packages emacs): New packages not found on HEAD (why is this a
conflict?)

(gnu packages game-development): New packages not found on HEAD.

(gnu packages games): New packages not found on HEAD.

(gnu packages guile): New packages not found on HEAD.

(gnu packages haskell): Period-at-end-of-synopsis conflict.

(gnu packages imagemagick): Version conflict for imagemagick.

(gnu packages linux): Version conflicts in linux-libres.

(gnu packages music): Version conflict in beets.

(gnu packages password-utils): Conflict in authorship lines.

(gnu packages perl): Conflict in authorship lines.

(gnu packages python): Conflict in authorship lines.
Many conflicts due to iyzsong's input rearranging blitz. Took all the
new arrangements.
Conflict in wrap-python3. Took HEAD, which wraps more things.

(gnu packages scheme): Conflict related to introduction of
ghostscript-gs. Took master.

(gnu packages tex): Version conflicts in texlive packages. Took latest.

(gnu packages tls): Version conflict in gnutls.

(gnu packages version-control): Version conflict in git.

(gnu packages video): New configure flags for mpv from master.

(gnu packages xorg): Conflict in authorship lines.

gnu/local.mk: HEAD did not have patch 'mysql-fix-failing-test.patch'

(gnu tests): master exports more marionette-related variables.

guix/config.scm.in: Related to 0b0086e (config: Export the raw
installation directories.). Took master.

(guix packages): Related to 1929fdba (packages: <origin> no longer has
an 'imported-modules' field.). Took master.

(gnu tests base): Related to d2fa61bc (tests: Add Avahi and NSS-mDNS
test.). Took master.
Took master for all conflicts, since no work had been done on
core-updates-next.

(gnu tests install): Took master for all conflicts, since no work had
been done on core-updates-next.

doc/guix.texi: The mcron documentation was duplicated for some reason.
Git did not report this conflict. I discovered it when `make` failed.

Re: core-updates merged!

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> I merged master into core-updates-next and made the change in a
> subsequent commit.
>
> Unfortunately, I can't push the branch to Savannah because it contains
> the following unsigned commits (one of them is mine, oops!):
>
> f21403e2b6f5a9491937a0cc9f31fc113998ce5e
> 9bc84dfea9560c497c91863e7b5021860bd3c254
> 745ad37a780b28f72ac9dfcc8092317e577e51c9
> 2d74d94b17b23ab95072da68553d85ac0b3bfede
> aebd383d04b351465cfb14e4fd0949b67d4b282e
>
> What should we do?

I think you should start from the pre-merge ‘core-updates-next’, sign
commits that are unsigned (I thought Manolis signed them all on the last
rebase?), then merge, and finally push.

Not ideal, but hey!

Ludo’.

Re: core-updates merged!

[Leo Famulari]

On Wed, Aug 03, 2016 at 06:42:34PM +0200, Ludovic Courtès wrote:
> I think you should start from the pre-merge ‘core-updates-next’, sign
> commits that are unsigned (I thought Manolis signed them all on the last
> rebase?), then merge, and finally push.

Unfortunately, signing old commits causes subsequent history to be
rewritten, and the subsequent signatures are lost. I would have to
re-sign commits all the way back to June (for aebd383). And Git users'
local history would become invalid.

By the way, all the commits that are rejected by the hook are from the
master branch.

> Not ideal, but hey!

Very! I'll wait for replies before taking this drastic action.

I think we are hitting something like the problem I warned about here:
http://lists.gnu.org/archive/html/guix-devel/2016-07/msg01220.html

Re: core-updates merged!

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Wed, Aug 03, 2016 at 06:42:34PM +0200, Ludovic Courtès wrote:
>> I think you should start from the pre-merge ‘core-updates-next’, sign
>> commits that are unsigned (I thought Manolis signed them all on the last
>> rebase?), then merge, and finally push.
>
> Unfortunately, signing old commits causes subsequent history to be
> rewritten, and the subsequent signatures are lost. I would have to
> re-sign commits all the way back to June (for aebd383). And Git users'
> local history would become invalid.

Yeah, but despite lacking the ‘wip-’ prefix as we usually do, this
branch was “rebaseable”, so I think it’s OK.

Hopefully that will no longer happen in the future.

> I think we are hitting something like the problem I warned about here:
> http://lists.gnu.org/archive/html/guix-devel/2016-07/msg01220.html

Yes, that’s annoying, but it’s a one-time transitional cost.

Thanks!

Ludo’.

Re: core-updates merged!

[Leo Famulari]

On Wed, Aug 03, 2016 at 07:56:05PM +0200, Ludovic Courtès wrote:
> Leo Famulari <leo@famulari.name> skribis:
> 
> > On Wed, Aug 03, 2016 at 06:42:34PM +0200, Ludovic Courtès wrote:
> >> I think you should start from the pre-merge ‘core-updates-next’, sign
> >> commits that are unsigned (I thought Manolis signed them all on the last
> >> rebase?), then merge, and finally push.
> >
> > Unfortunately, signing old commits causes subsequent history to be
> > rewritten, and the subsequent signatures are lost. I would have to
> > re-sign commits all the way back to June (for aebd383). And Git users'
> > local history would become invalid.
> 
> Yeah, but despite lacking the ‘wip-’ prefix as we usually do, this
> branch was “rebaseable”, so I think it’s OK.
> 
> Hopefully that will no longer happen in the future.
> 
> > I think we are hitting something like the problem I warned about here:
> > http://lists.gnu.org/archive/html/guix-devel/2016-07/msg01220.html
> 
> Yes, that’s annoying, but it’s a one-time transitional cost.

Just to clarify, I would be re-signing (with my own key) and rebasing
all commits that were made on the master branch and merged on
core-updates-next, going back to sometime in June 2016.

Whenever we merge this core-updates-next branch back into master, the
master branch's history will be rewritten, starting with aebd383d0.
Right? Or am I misunderstanding?

I tried it, to see what would happen.

$ git rebase aebd383d04b351465cfb14e4fd0949b67d4b282e^ --exec "git commit --amend --no-edit --gpg-sign" || git rebase --abort

But for some reason, it ends up trying to work on commits from February
(starting at "build-system/gnu: Do not patch symlinks"), and then fails
to apply the commit that updates Python 2 to 2.7.11. Nothing should fail
to apply, since I'm not changing any files. Am I doing it wrong, or is
it a bug in Git? Perhaps some complication rebasing through previous
merges?

Re: core-updates merged!

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

>> > I think we are hitting something like the problem I warned about here:
>> > http://lists.gnu.org/archive/html/guix-devel/2016-07/msg01220.html
>> 
>> Yes, that’s annoying, but it’s a one-time transitional cost.
>
> Just to clarify, I would be re-signing (with my own key) and rebasing
> all commits that were made on the master branch and merged on
> core-updates-next, going back to sometime in June 2016.

I think core-updates-next is just a dozen commits or so, right?

> Whenever we merge this core-updates-next branch back into master, the
> master branch's history will be rewritten, starting with aebd383d0.
> Right? Or am I misunderstanding?

I think we should remerge core-updates-next on top of master, making it
the new core-updates.

From there on, we will not rebase core-updates and only do merges in
that branch, as usual.

> I tried it, to see what would happen.
>
> $ git rebase aebd383d04b351465cfb14e4fd0949b67d4b282e^ --exec "git commit --amend --no-edit --gpg-sign" || git rebase --abort
>
> But for some reason, it ends up trying to work on commits from February
> (starting at "build-system/gnu: Do not patch symlinks"), and then fails
> to apply the commit that updates Python 2 to 2.7.11. Nothing should fail
> to apply, since I'm not changing any files. Am I doing it wrong, or is
> it a bug in Git? Perhaps some complication rebasing through previous
> merges?

Hmm.  Perhaps the explanation is the merged commit that I screwed, which
I’ll write about just now.

Ludo’.

Re: core-updates merged!

[Leo Famulari]

On Wed, Aug 03, 2016 at 10:01:48PM +0200, Ludovic Courtès wrote:
> I think core-updates-next is just a dozen commits or so, right?

> I think we should remerge core-updates-next on top of master, making it
> the new core-updates.

Do you mean `git checkout core-updates-next && git merge master`? That's
what I've done.

But, it means that some unsigned commits come to core-updates-next from
master, and I can't push the result to Savannah because of the new
signature hook.

Re: core-updates merged!

[Andreas Enge]

On Wed, Aug 03, 2016 at 05:01:29PM -0400, Leo Famulari wrote:
> Do you mean `git checkout core-updates-next && git merge master`? That's
> what I've done.

Another option would be the following:
git checkout master
git checkout -b core-updates
git cherry-pick "commit 1 from core-updates-next"
git cherry-pick "commit 2 from core-updates-next"
...

If there are only a dozen commits in core-updates-next, this could be
feasible, with the danger of forgetting some.

Andreas

Re: core-updates merged!

[Leo Famulari]

On Wed, Aug 03, 2016 at 11:27:28PM +0200, Andreas Enge wrote:
> On Wed, Aug 03, 2016 at 05:01:29PM -0400, Leo Famulari wrote:
> > Do you mean `git checkout core-updates-next && git merge master`? That's
> > what I've done.
> 
> Another option would be the following:
> git checkout master
> git checkout -b core-updates
> git cherry-pick "commit 1 from core-updates-next"
> git cherry-pick "commit 2 from core-updates-next"
> ...
> 
> If there are only a dozen commits in core-updates-next, this could be
> feasible, with the danger of forgetting some.

I'm not sure if this is the right way to check, but

$ git log --oneline master..core-updates-next | wc -l
161

Does anyone else want to test this? I just tried to copy the master
branch and push it as TEMP-signature-test, and it was rejected by
Savannah. Although, I didn't get any detail from Savannah about why it
was rejected, which I did get when I tried to do it with
core-updates-next earlier. So perhaps there is some other issue now...

‘core-updates’ merge is a squashed commit

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

Hello!

Someone reported that the commit stats for the new release were
suspicious and found that the recent core-updates merge was fishy.

In fact, commit 455859a50f88f625d13fc2f304111f02369b366b, which is the
core-updates merge, is *not* a merge commit.  Instead it seems to be a
squashed commit of all of core-updates.  Consequently, part of the
history of the files touched by this merge is squashed into this single
pseudo-merge commit.  :-/

Apologies for this mess.  I don’t know how this happened.  I suspect a
mixture of me not paying enough attention, and Magit + Git + gpg somehow
leading me in the wrong direction.

This can be fixed locally with a graft to give the merge commit the two
parents it is supposed to have:

  git replace --graft 455859a50f88f625d13fc2f304111f02369b366b \
     742effef5629667b274087adc70b06abab86b252 a8cb87abe98d57fb763d5b14524dc32c96bd31b5 

Unfortunately, grafts are local, so anyone cloning the repo will see the
squashed merge commit.  Also, the replacement commit lacks a signature.

I don’t know how this impacts the core-updates-next rebase that Leo and
I were discussing.

If anyone has advice on grafts, or on how to avoid this in the future,
I’m all ears!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: ‘core-updates’ merge is a squashed commit

[Leo Famulari]

On Wed, Aug 03, 2016 at 10:29:21PM +0200, Ludovic Courtès wrote:
> In fact, commit 455859a50f88f625d13fc2f304111f02369b366b, which is the
> core-updates merge, is *not* a merge commit.  Instead it seems to be a
> squashed commit of all of core-updates.  Consequently, part of the
> history of the files touched by this merge is squashed into this single
> pseudo-merge commit.  :-/

Dang, that means we lost the commit history of core-updates. It won't be
in the Git log.

> This can be fixed locally with a graft to give the merge commit the two
> parents it is supposed to have:
> 
>   git replace --graft 455859a50f88f625d13fc2f304111f02369b366b \
>      742effef5629667b274087adc70b06abab86b252 a8cb87abe98d57fb763d5b14524dc32c96bd31b5 

I tried this and Git said "fatal: could not parse a8cb87abe98d57fb763d5b14524dc32c96bd31b5"

I don't have a Git object of that name in my repo.

Re: ‘core-updates’ merge is a squashed commit

[Mark H Weaver]

Leo Famulari <leo@famulari.name> writes:

> On Wed, Aug 03, 2016 at 10:29:21PM +0200, Ludovic Courtès wrote:
>> In fact, commit 455859a50f88f625d13fc2f304111f02369b366b, which is the
>> core-updates merge, is *not* a merge commit.  Instead it seems to be a
>> squashed commit of all of core-updates.  Consequently, part of the
>> history of the files touched by this merge is squashed into this single
>> pseudo-merge commit.  :-/
>
> Dang, that means we lost the commit history of core-updates. It won't be
> in the Git log.

For now, I pushed my most recent copy of the lost 'core-updates' branch
to Savannah as 'core-updates-2016-08-01', to make sure it's not lost.

>> This can be fixed locally with a graft to give the merge commit the two
>> parents it is supposed to have:
>> 
>>   git replace --graft 455859a50f88f625d13fc2f304111f02369b366b \
>>      742effef5629667b274087adc70b06abab86b252 a8cb87abe98d57fb763d5b14524dc32c96bd31b5 

How about reverting the squashed commit and then re-doing a proper merge
from 'core-updates-2016-08-01' into 'master'?

      Mark

Re: ‘core-updates’ merge is a squashed commit

[Andreas Enge]

On Thu, Aug 04, 2016 at 03:50:51AM -0400, Mark H Weaver wrote:
> How about reverting the squashed commit and then re-doing a proper merge
> from 'core-updates-2016-08-01' into 'master'?

If this is possible without too many complications, that sounds like a good
option; the earlier the better, probably, before there are too many new
conflicts. Hm, I just tried the revert part, and it already creates a conflict.

Andreas

Re: ‘core-updates’ merge is a squashed commit

[Mark H Weaver]

Andreas Enge <andreas@enge.fr> writes:

> On Thu, Aug 04, 2016 at 03:50:51AM -0400, Mark H Weaver wrote:
>> How about reverting the squashed commit and then re-doing a proper merge
>> from 'core-updates-2016-08-01' into 'master'?
>
> If this is possible without too many complications, that sounds like a good
> option; the earlier the better, probably, before there are too many new
> conflicts. Hm, I just tried the revert part, and it already creates a conflict.

On my local machine, I did this successfully by first reverting a couple
of later commits, then reverting the squashed merge, then doing a proper
merge, and finally cherry-picking the later commits.  I verified that
this sequence of operations made no changes to the tree.

Unfortunately, when I tried to push it to 'master', it was rejected
because of 56 unsigned commits on the 'core-updates' branch.  Note that
all of the new commits I made (the reverts, the merge, and the
cherry-picks) were all signed by me.  The unsigned commits that are
blocking this have been on the 'core-updates' branch for quite some
time.

If there are unsigned commits on 'core-updates-next', I guess we'll run
into the same problem trying to merge that branch later.

       Mark

Re: ‘core-updates’ merge is a squashed commit

[Andreas Enge]

On Thu, Aug 04, 2016 at 08:36:28AM -0400, Mark H Weaver wrote:
> On my local machine, I did this successfully by first reverting a couple
> of later commits, then reverting the squashed merge, then doing a proper
> merge, and finally cherry-picking the later commits.  I verified that
> this sequence of operations made no changes to the tree.

Good news, thanks for all this work!

> Unfortunately, when I tried to push it to 'master', it was rejected
> because of 56 unsigned commits on the 'core-updates' branch.  Note that
> all of the new commits I made (the reverts, the merge, and the
> cherry-picks) were all signed by me.  The unsigned commits that are
> blocking this have been on the 'core-updates' branch for quite some
> time.
> 
> If there are unsigned commits on 'core-updates-next', I guess we'll run
> into the same problem trying to merge that branch later.

A possible solution would be to disable the commit hook that forces
signatures, make these commits, and reenable the hook; hoping that noone
adds an unsigned commit in the meantime... It would require to coordinate
with a savannah administrator.

Andreas

Re: ‘core-updates’ merge is a squashed commit

[Leo Famulari]

On Thu, Aug 04, 2016 at 02:40:32PM +0200, Andreas Enge wrote:
> On Thu, Aug 04, 2016 at 08:36:28AM -0400, Mark H Weaver wrote:
> Good news, thanks for all this work!

Indeed!

> > Unfortunately, when I tried to push it to 'master', it was rejected
> > because of 56 unsigned commits on the 'core-updates' branch.  Note that
> > all of the new commits I made (the reverts, the merge, and the
> > cherry-picks) were all signed by me.  The unsigned commits that are
> > blocking this have been on the 'core-updates' branch for quite some
> > time.
> > 
> > If there are unsigned commits on 'core-updates-next', I guess we'll run
> > into the same problem trying to merge that branch later.

Yes, I had this problem yesterday.

> A possible solution would be to disable the commit hook that forces
> signatures, make these commits, and reenable the hook; hoping that noone
> adds an unsigned commit in the meantime... It would require to coordinate
> with a savannah administrator.

I think we have to do this, unfortunately.

Re: ‘core-updates’ merge is a squashed commit

[Mark H Weaver]

Leo Famulari <leo@famulari.name> writes:

> On Thu, Aug 04, 2016 at 02:40:32PM +0200, Andreas Enge wrote:
>> On Thu, Aug 04, 2016 at 08:36:28AM -0400, Mark H Weaver wrote:
>> Good news, thanks for all this work!
>
> Indeed!
>
>> > Unfortunately, when I tried to push it to 'master', it was rejected
>> > because of 56 unsigned commits on the 'core-updates' branch.  Note that
>> > all of the new commits I made (the reverts, the merge, and the
>> > cherry-picks) were all signed by me.  The unsigned commits that are
>> > blocking this have been on the 'core-updates' branch for quite some
>> > time.
>> > 
>> > If there are unsigned commits on 'core-updates-next', I guess we'll run
>> > into the same problem trying to merge that branch later.
>
> Yes, I had this problem yesterday.
>
>> A possible solution would be to disable the commit hook that forces
>> signatures, make these commits, and reenable the hook; hoping that noone
>> adds an unsigned commit in the meantime... It would require to coordinate
>> with a savannah administrator.
>
> I think we have to do this, unfortunately.

I just did this.  Huge thanks to Lisa Marie Maginnis, GNU sysadmin
extraordinaire, for promptly responding to my plea for help :)

However, we should take steps to avoid this problem in the future.  The
most recent unsigned commit on 'core-updates' was from only 3 days ago,
by Andreas.  "gnu: unison: Add input ghostscript".  I'm fairly sure this
was after we added the commit hook, so I guess the hook only applies to
'master'.

So, if I understand correctly, we're currently in a state where unsigned
commits can accidentally get pushed to other branches, making those
branches unmergeable into 'master'.

      Mark

Re: ‘core-updates’ merge is a squashed commit

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> Leo Famulari <leo@famulari.name> writes:
>
>> On Thu, Aug 04, 2016 at 02:40:32PM +0200, Andreas Enge wrote:
>>> On Thu, Aug 04, 2016 at 08:36:28AM -0400, Mark H Weaver wrote:
>>> Good news, thanks for all this work!
>>
>> Indeed!
>>
>>> > Unfortunately, when I tried to push it to 'master', it was rejected
>>> > because of 56 unsigned commits on the 'core-updates' branch.  Note that
>>> > all of the new commits I made (the reverts, the merge, and the
>>> > cherry-picks) were all signed by me.  The unsigned commits that are
>>> > blocking this have been on the 'core-updates' branch for quite some
>>> > time.
>>> > 
>>> > If there are unsigned commits on 'core-updates-next', I guess we'll run
>>> > into the same problem trying to merge that branch later.
>>
>> Yes, I had this problem yesterday.
>>
>>> A possible solution would be to disable the commit hook that forces
>>> signatures, make these commits, and reenable the hook; hoping that noone
>>> adds an unsigned commit in the meantime... It would require to coordinate
>>> with a savannah administrator.
>>
>> I think we have to do this, unfortunately.
>
> I just did this.  Huge thanks to Lisa Marie Maginnis, GNU sysadmin
> extraordinaire, for promptly responding to my plea for help :)

Awesome, thank you all!

> However, we should take steps to avoid this problem in the future.  The
> most recent unsigned commit on 'core-updates' was from only 3 days ago,
> by Andreas.  "gnu: unison: Add input ghostscript".  I'm fairly sure this
> was after we added the commit hook, so I guess the hook only applies to
> 'master'.
>
> So, if I understand correctly, we're currently in a state where unsigned
> commits can accidentally get pushed to other branches, making those
> branches unmergeable into 'master'.

The hook can be seen at (search for “Attached Files”):

  https://savannah.gnu.org/support/?109104

I think it should work equally well for all branches.  That said, the
hook is so simple that it may well be simplistic, so do not hesitate to
send improvements to the hook in this Savannah ticket.

Thanks!

Ludo’.

Re: ‘core-updates’ merge is a squashed commit

[Andreas Enge]

On Thu, Aug 04, 2016 at 09:23:50AM -0400, Mark H Weaver wrote:
> I just did this.  Huge thanks to Lisa Marie Maginnis, GNU sysadmin
> extraordinaire, for promptly responding to my plea for help :)

Thanks to both of you!

> However, we should take steps to avoid this problem in the future.  The
> most recent unsigned commit on 'core-updates' was from only 3 days ago,
> by Andreas.  "gnu: unison: Add input ghostscript".  I'm fairly sure this
> was after we added the commit hook, so I guess the hook only applies to
> 'master'.
> So, if I understand correctly, we're currently in a state where unsigned
> commits can accidentally get pushed to other branches, making those
> branches unmergeable into 'master'.

Sorry for that! I did not think it would be possible, and working locally
without signing every private commit is much more convenient.

Indeed, I just pushed an unsigned commit to a new branch, and it worked.

This is strange: The update hook should work with any branch, and not
only master. Maybe it is the way it has been set up at savannah?

Andreas

Re: ‘core-updates’ merge is a squashed commit

[Mathieu Lirzin]

Hi,

Andreas Enge <andreas@enge.fr> writes:

> On Thu, Aug 04, 2016 at 09:23:50AM -0400, Mark H Weaver wrote:
>> I just did this.  Huge thanks to Lisa Marie Maginnis, GNU sysadmin
>> extraordinaire, for promptly responding to my plea for help :)
>
> Thanks to both of you!
>
>> However, we should take steps to avoid this problem in the future.  The
>> most recent unsigned commit on 'core-updates' was from only 3 days ago,
>> by Andreas.  "gnu: unison: Add input ghostscript".  I'm fairly sure this
>> was after we added the commit hook, so I guess the hook only applies to
>> 'master'.
>> So, if I understand correctly, we're currently in a state where unsigned
>> commits can accidentally get pushed to other branches, making those
>> branches unmergeable into 'master'.
>
> Sorry for that! I did not think it would be possible, and working locally
> without signing every private commit is much more convenient.

With gpg-agent and git properly setup, signing every local commit is not
that inconvenient IME.

I recommend you to give a try.  ;)

-- 
Mathieu Lirzin

Re: ‘core-updates’ merge is a squashed commit

[Leo Famulari]

On Thu, Aug 04, 2016 at 04:45:56PM +0200, Mathieu Lirzin wrote:
> With gpg-agent and git properly setup, signing every local commit is not
> that inconvenient IME.

And if you don't want to sign every commit as you work (it can be
tedious if your gpg-agent has a short cache lifetime), you can use
git-rebase to sign a commit range before pushing, as in this shell
script:

git-sign () {
	case $# in
		("0") range=HEAD  ;;
		("1") range=$1  ;;
		(*) echo "too many arguments" >&2; return 1 ;;
	esac

	git rebase "$range" --exec "git commit --amend --no-edit --gpg-sign" || git rebase --abort
}

So, if I have 4 commits to push, I do `git-sign HEAD~4`.

Re: ‘core-updates’ merge is a squashed commit

[Andreas Enge]

On Thu, Aug 04, 2016 at 12:37:07PM -0400, Leo Famulari wrote:
> And if you don't want to sign every commit as you work (it can be
> tedious if your gpg-agent has a short cache lifetime), you can use
> git-rebase to sign a commit range before pushing, as in this shell
> script:
> 
> git-sign () {
> 	case $# in
> 		("0") range=HEAD  ;;
> 		("1") range=$1  ;;
> 		(*) echo "too many arguments" >&2; return 1 ;;
> 	esac
> 
> 	git rebase "$range" --exec "git commit --amend --no-edit --gpg-sign" || git rebase --abort
> }
> 
> So, if I have 4 commits to push, I do `git-sign HEAD~4`.

So it is not enough to just do a
   git rebase -S
? I thought this would re-sign all my commits on top of the base of the rebase.
If not, this could explain my earlier mistake...

Andreas

Re: ‘core-updates’ merge is a squashed commit

[Leo Famulari]

On Thu, Aug 04, 2016 at 08:32:42PM +0200, Andreas Enge wrote:
> On Thu, Aug 04, 2016 at 12:37:07PM -0400, Leo Famulari wrote:
> > 	git rebase "$range" --exec "git commit --amend --no-edit --gpg-sign" || git rebase --abort
> 
> So it is not enough to just do a
>    git rebase -S
> ? I thought this would re-sign all my commits on top of the base of the rebase.
> If not, this could explain my earlier mistake...

I don't remember exactly why I began using the `git rebase --exec ...`
command. If yours works, I'd like to switch to it!

Re: ‘core-updates’ merge is a squashed commit

[Andreas Enge]

On Thu, Aug 04, 2016 at 04:45:56PM +0200, Mathieu Lirzin wrote:
> With gpg-agent and git properly setup, signing every local commit is not
> that inconvenient IME.
> I recommend you to give a try.  ;)

Ever so often I try gpg@2 (I do not remember whether 2.0 or 2.1), it works
for a while, it stops working, and then I get back to @1...

Andreas

Re: ‘core-updates’ merge is a squashed commit

[Andy Wingo]

On Thu 04 Aug 2016 14:36, Mark H Weaver <mhw@netris.org> writes:

> Unfortunately, when I tried to push it to 'master', it was rejected
> because of 56 unsigned commits on the 'core-updates' branch.

What's the rationale for requiring non-HEAD commits to be signed when
pushing?  For me a signed HEAD implicitly signs all parent comments, in
my mental trust model anyway :)

Andy

Re: ‘core-updates’ merge is a squashed commit

[Leo Famulari]

On Thu, Aug 04, 2016 at 05:06:15PM +0200, Andy Wingo wrote:
> On Thu 04 Aug 2016 14:36, Mark H Weaver <mhw@netris.org> writes:
> 
> > Unfortunately, when I tried to push it to 'master', it was rejected
> > because of 56 unsigned commits on the 'core-updates' branch.
> 
> What's the rationale for requiring non-HEAD commits to be signed when
> pushing?  For me a signed HEAD implicitly signs all parent comments, in
> my mental trust model anyway :)

How would the rest of us distinguish between

1) a range of your commits with a signed HEAD
2) a range of your commits with a signed HEAD that you pushed after I
pushed a commit created with `git commit --author="Andy Wingo"

Re: ‘core-updates’ merge is a squashed commit

[Andy Wingo]

On Thu 04 Aug 2016 18:44, Leo Famulari <leo@famulari.name> writes:

> How would the rest of us distinguish between
>
> 1) a range of your commits with a signed HEAD
> 2) a range of your commits with a signed HEAD that you pushed after I
> pushed a commit created with `git commit --author="Andy Wingo"

I'm not sure what the threat model here is, and surely this is mostly
because I am ignorant :)  Would you mind elaborating a bit more?

Andy

Re: ‘core-updates’ merge is a squashed commit

[Leo Famulari]

On Thu, Aug 04, 2016 at 06:55:34PM +0200, Andy Wingo wrote:
> On Thu 04 Aug 2016 18:44, Leo Famulari <leo@famulari.name> writes:
> 
> > How would the rest of us distinguish between
> >
> > 1) a range of your commits with a signed HEAD
> > 2) a range of your commits with a signed HEAD that you pushed after I
> > pushed a commit created with `git commit --author="Andy Wingo"
> 
> I'm not sure what the threat model here is, and surely this is mostly
> because I am ignorant :)  Would you mind elaborating a bit more?

I admit, the example is really contrived.

My point is that, as far as I know, there is no way to know who exactly
is behind an unsigned Git commit.

The "Author" and "Commit" information seen in `git log --format=full` is
trivially forged, for example by altering the [user] field of your Git
configuration file.

Re: ‘core-updates’ merge is a squashed commit

[Andy Wingo]

On Thu 04 Aug 2016 22:05, Leo Famulari <leo@famulari.name> writes:

> On Thu, Aug 04, 2016 at 06:55:34PM +0200, Andy Wingo wrote:
>> On Thu 04 Aug 2016 18:44, Leo Famulari <leo@famulari.name> writes:
>> 
>> > How would the rest of us distinguish between
>> >
>> > 1) a range of your commits with a signed HEAD
>> > 2) a range of your commits with a signed HEAD that you pushed after I
>> > pushed a commit created with `git commit --author="Andy Wingo"
>> 
>> I'm not sure what the threat model here is, and surely this is mostly
>> because I am ignorant :)  Would you mind elaborating a bit more?
>
> I admit, the example is really contrived.
>
> My point is that, as far as I know, there is no way to know who exactly
> is behind an unsigned Git commit.
>
> The "Author" and "Commit" information seen in `git log --format=full` is
> trivially forged, for example by altering the [user] field of your Git
> configuration file.

Yeah.  I guess I don't see see "author misattribution on unsigned
commits" as part of the threat model.

My mental model is that if you have a signed commit A with unsigned
parents B, C, ..., that it's the person who signed commit A who signs
off on commits B, C, and so on.  That person attests to the integrity of
that range of commits, *including* the author field(s).

If you sign a HEAD which brings in an unsigned commit that you (or
someone else) forged to use me (say) as --author, it's true, I can claim
not to have made it.  But that seems a bit irrelevant to any property we
care about; dunno...

Andy

Re: ‘core-updates’ merge is a squashed commit

[Leo Famulari]

On Fri, Aug 05, 2016 at 09:35:59AM +0200, Andy Wingo wrote:
> Yeah.  I guess I don't see see "author misattribution on unsigned
> commits" as part of the threat model.
> 
> My mental model is that if you have a signed commit A with unsigned
> parents B, C, ..., that it's the person who signed commit A who signs
> off on commits B, C, and so on.  That person attests to the integrity of
> that range of commits, *including* the author field(s).

But, how does anyone know that the person who signed A attests to B and
C? I don't think Git has a feature that conveys that intention.

Re: ‘core-updates’ merge is a squashed commit

[Andy Wingo]

On Fri 05 Aug 2016 16:59, Leo Famulari <leo@famulari.name> writes:

> On Fri, Aug 05, 2016 at 09:35:59AM +0200, Andy Wingo wrote:
>> Yeah.  I guess I don't see see "author misattribution on unsigned
>> commits" as part of the threat model.
>> 
>> My mental model is that if you have a signed commit A with unsigned
>> parents B, C, ..., that it's the person who signed commit A who signs
>> off on commits B, C, and so on.  That person attests to the integrity of
>> that range of commits, *including* the author field(s).
>
> But, how does anyone know that the person who signed A attests to B and
> C? I don't think Git has a feature that conveys that intention.

Why would you sign a commit if you don't attest to intermediate unsigned
commits?

A

Re: ‘core-updates’ merge is a squashed commit

[Leo Famulari]

On Fri, Aug 05, 2016 at 06:50:30PM +0200, Andy Wingo wrote:
> Why would you sign a commit if you don't attest to intermediate unsigned
> commits?

If I push A-B-C with a signed HEAD immediately after somebody pushes a
forged D, won't it look like I vouch for D? How could a 3rd party tell
whether D was pushed by me or somebody else?

Does your suggested method address this hypothetical situation?

Re: ‘core-updates’ merge is a squashed commit

[Mark H Weaver]

Leo Famulari <leo@famulari.name> writes:

> On Fri, Aug 05, 2016 at 06:50:30PM +0200, Andy Wingo wrote:
>> Why would you sign a commit if you don't attest to intermediate unsigned
>> commits?
>
> If I push A-B-C with a signed HEAD immediately after somebody pushes a
> forged D, won't it look like I vouch for D?

This can't happen unless D is already in your local repo before you
commit locally.  If someone pushes D immediately before you try to push
A-B-C, your push will be rejected.  At that point, you need to pull,
rebase on top of D, and try again.  You can always see the ancestor
commits before signing the new comit.

I haven't thought deeply on this, but it seems to me that Andy's
suggestion has a lot of merit.  We could choose to decide, as a matter
of policy, that if you sign a commit with unsigned ancestor commit(s),
you are effectively vouching for those ancestor commits.  We could
modify the commit hook to accept a push as long as the new HEAD commit
is signed by an authorized key, disregarding the ancestors.

There's one thing that each of us would need to be careful of, though.
If we adopt this policy, then before signing a commit, we'd need to
first verify that the parent commit has been signed, lest we
accidentally vouch for an unsigned commit that we know nothing about.

In practice, this could only happen if Savannah is compromised or
there's a man-in-the-middle attack, because Savannah is supposed to
ensure that pushes with unsigned HEADs are rejected.

What do you think?

      Mark

Re: ‘core-updates’ merge is a squashed commit

[Leo Famulari]

On Fri, Aug 05, 2016 at 08:59:32PM -0400, Mark H Weaver wrote:
> Leo Famulari <leo@famulari.name> writes:
> > If I push A-B-C with a signed HEAD immediately after somebody pushes a
> > forged D, won't it look like I vouch for D?
> 
> This can't happen unless D is already in your local repo before you
> commit locally.  If someone pushes D immediately before you try to push
> A-B-C, your push will be rejected.  At that point, you need to pull,
> rebase on top of D, and try again.  You can always see the ancestor
> commits before signing the new comit.
> 
> I haven't thought deeply on this, but it seems to me that Andy's
> suggestion has a lot of merit.  We could choose to decide, as a matter
> of policy, that if you sign a commit with unsigned ancestor commit(s),
> you are effectively vouching for those ancestor commits.  We could
> modify the commit hook to accept a push as long as the new HEAD commit
> is signed by an authorized key, disregarding the ancestors.
> 
> There's one thing that each of us would need to be careful of, though.
> If we adopt this policy, then before signing a commit, we'd need to
> first verify that the parent commit has been signed, lest we
> accidentally vouch for an unsigned commit that we know nothing about.
> 
> In practice, this could only happen if Savannah is compromised or
> there's a man-in-the-middle attack, because Savannah is supposed to
> ensure that pushes with unsigned HEADs are rejected.
>
> What do you think?

Okay, I think your analysis is right. If the HEAD of a push is always
signed, and that policy is known, then it is possible to know who is
responsible for pushing each commit.

But, I also think the primary point of signing the commits is to record
the identity of the person responsible for the commit, and so I think
the policy should be to sign each commit. [0]

The identity information provided by a "sign the HEAD" system seems
relatively brittle to me.

Somebody looking at a clone of the Git repo may not know about a commit
hook, or a "be careful" policy, that may or may not have been
consistently active (it seems that hooks don't come with `git clone`).
This person won't be able to easily attribute the unsigned commits.

Isn't it better for the identity information to be inherent to the Git
commits themselves, since those are what is preserved by Git? Git does
not preserve hooks or policies.

Also, is there some problem with signing each commit? I don't know why
we'd want to stop doing this.

[0] But, I also think that the hook we put in place to enforce this is
going to have to be modified, at least until we no longer need to push
this last batch of unsigned commits. At the very least, it should be
made to actually verify the signatures.

Re: ‘core-updates’ merge is a squashed commit

[Andy Wingo]

On Sat 06 Aug 2016 04:07, Leo Famulari <leo@famulari.name> writes:

> But, I also think the primary point of signing the commits is to record
> the identity of the person responsible for the commit, and so I think
> the policy should be to sign each commit. [0]

To me this is not the value that signing brings; rather, signing
protects against an attack in which a malicious third party updates the
Guix git repository to have a vulnerable commit.

Given that most people run "guix pull" without inspecting the commits,
this is real value: it would be possible to even make "guix pull" only
accept updates whose HEAD is signed by a key in the keyring.  Having the
hook only accept signed HEADs is a good start along that path of course.

> Isn't it better for the identity information to be inherent to the Git
> commits themselves, since those are what is preserved by Git? Git does
> not preserve hooks or policies.

The convention that a signature goes along with responsibility is also a
policy -- any path we take is a convention.

> Also, is there some problem with signing each commit? I don't know why
> we'd want to stop doing this.

I think there's a risk of signing fatigue.  The more signatures you make
with your key, the more likely it is that you sign something that you
didn't mean to.  To me it makes sense to reduce the number of signatures
to the minimum necessary to preserve whatever security properties we are
interested in; but YMMV obviously :)

Andy

Re: ‘core-updates’ merge is a squashed commit

[Andreas Enge]

On Fri, Aug 05, 2016 at 08:59:32PM -0400, Mark H Weaver wrote:
> I haven't thought deeply on this, but it seems to me that Andy's
> suggestion has a lot of merit.  We could choose to decide, as a matter
> of policy, that if you sign a commit with unsigned ancestor commit(s),
> you are effectively vouching for those ancestor commits.  We could
> modify the commit hook to accept a push as long as the new HEAD commit
> is signed by an authorized key, disregarding the ancestors.
> 
> There's one thing that each of us would need to be careful of, though.
> If we adopt this policy, then before signing a commit, we'd need to
> first verify that the parent commit has been signed, lest we
> accidentally vouch for an unsigned commit that we know nothing about.

I am not very happy about such a policy; if I sign a commit, I am only
signing my commit, and not all of its history, or even only its history
up to the previous signed commit. Also, while signing each commit is
a simple git configuration option, needing to verify the history before
each commit would be a hassle that as far as I can see is not easily
automated.

> In practice, this could only happen if Savannah is compromised or
> there's a man-in-the-middle attack, because Savannah is supposed to
> ensure that pushes with unsigned HEADs are rejected.

Agreed, this mitigates the problem above. But I feel better with the
current situation.

Andreas

Re: ‘core-updates’ merge is a squashed commit

[Andy Wingo]

On Sat 06 Aug 2016 09:52, Andreas Enge <andreas@enge.fr> writes:

> I am not very happy about such a policy; if I sign a commit, I am only
> signing my commit, and not all of its history

For what it's worth, this is not quite true.  The SHA1 hashes of parent
commits are part of the commit object itself, so if you sign a commit
you are also signing its history.

Andy

Re: ‘core-updates’ merge is a squashed commit

[Mike Gerwitz]

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]

On Thu, Aug 04, 2016 at 17:06:15 +0200, Andy Wingo wrote:
> What's the rationale for requiring non-HEAD commits to be signed when
> pushing?  For me a signed HEAD implicitly signs all parent comments, in
> my mental trust model anyway :)

That could be a potentially daunting/impossible task for the person
signing a commit.

Aside from asserting one's identity, GPG-signed commits also (can) help
in the event that the system of one of the Guix hackers with commit
access is compromised.  Attacking Savannah is one way to compromise the
repo, but compromising one of the many Guix hackers' systems is another.

If a commit is signed in the hacker's local repo, it cannot be
manipulated by an attacker, nor can an attacker sign a new malicious
commit.  Unless, of course, the GPG key resides on the same box, the
attacker can get a hold of it, and can use a keylogger/etc to get the
passphrase.  Smart cards help here.

I also recommend against auto-signing commmits on rebase unless you
first verify that each commit within that range has a valid signature
beforehand.

Not fool-proof, but nothing is. :)

-- 
Mike Gerwitz
Free Software Hacker+Activist | GNU Maintainer & Volunteer
https://mikegerwitz.com       | GPG Key ID: 0x8EE30EAB

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: ‘core-updates’ merge is a squashed commit

[Leo Famulari]

On Thu, Aug 04, 2016 at 03:50:51AM -0400, Mark H Weaver wrote:
> How about reverting the squashed commit and then re-doing a proper merge
> from 'core-updates-2016-08-01' into 'master'?

I think we should do this. It's worth a little blemish in the commit
history in order to retain the history of this core-updates cycle.

Re: core-updates merged!

[Leo Famulari]

On Wed, Aug 03, 2016 at 12:04:46AM -0400, Leo Famulari wrote:
> On Tue, Aug 02, 2016 at 11:28:59PM +0200, Ludovic Courtès wrote:
> > Leo Famulari <leo@famulari.name> skribis:
> > Otherwise LGTM; could you push it to core-updates-next?
> 
> I merged master into core-updates-next and made the change in a
> subsequent commit.
> 
> Unfortunately, I can't push the branch to Savannah because it contains
> the following unsigned commits (one of them is mine, oops!):
> 
> f21403e2b6f5a9491937a0cc9f31fc113998ce5e
> 9bc84dfea9560c497c91863e7b5021860bd3c254
> 745ad37a780b28f72ac9dfcc8092317e577e51c9
> 2d74d94b17b23ab95072da68553d85ac0b3bfede
> aebd383d04b351465cfb14e4fd0949b67d4b282e
> 
> What should we do?

This is still preventing me from starting the new core-updates branch.

I guess that we should do the same thing we did when fixing the squashed
merge: disable the assert-commit-signed hook temporarily for the push.

It was also suggested to cherry-pick the 11 commits from
core-updates-next onto master — essentially rebasing them. I can try
that if everyone is okay with me re-signing those commits.

> In the meantime, you can inspect the updated branch on my personal Git
> repo:
> https://github.com/lfam/guix/commits/core-updates-next

Mark, if you want to do method that requires disabling the hook, feel
free to take my (signed) work from this link. It will save you from
doing a lot of merge conflict resolution.

Re: core-updates merged!

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 980 bytes --]

On Sat, Aug 06, 2016 at 10:42:22AM -0400, Leo Famulari wrote:
> It was also suggested to cherry-pick the 11 commits from
> core-updates-next onto master — essentially rebasing them. I can try
> that if everyone is okay with me re-signing those commits.

As discussed on #guix [0], I've pushed the result of this to
WIP-core-updates [1].

I rewrote an obsolete curl update patch to instead remove the curl graft
found on the master branch.

Since this required me to re-sign these commits, will somebody please
review the differences between WIP-core-updates and core-updates-next,
to make sure I've done the right thing?

When I get the "okay", I will rename the branch to core-updates, and
then it will be "open for business" :)

If all goes well, hopefully this is the last time we ever have to
re-sign each other's work.

[0]
https://gnunet.org/bot/log/guix/2016-08-10#T1099908

[1]
http://git.savannah.gnu.org/cgit/guix.git/log/?h=WIP-core-updates

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: core-updates merged!

[Manolis Ragkousis]

Hello Leo

On 08/10/16 22:49, Leo Famulari wrote:
> Since this required me to re-sign these commits, will somebody please
> review the differences between WIP-core-updates and core-updates-next,
> to make sure I've done the right thing?
> 
> When I get the "okay", I will rename the branch to core-updates, and
> then it will be "open for business" :)

Seems good to me. My local changes apply cleanly to the new branch and
my patches work as expected.

Thank you for taking care of this.
Manolis

Core-updates is ready for your patches!

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 658 bytes --]

On Sat, Aug 13, 2016 at 10:15:58AM +0300, Manolis Ragkousis wrote:
> On 08/10/16 22:49, Leo Famulari wrote:
> > Since this required me to re-sign these commits, will somebody please
> > review the differences between WIP-core-updates and core-updates-next,
> > to make sure I've done the right thing?
> > 
> > When I get the "okay", I will rename the branch to core-updates, and
> > then it will be "open for business" :)
> 
> Seems good to me. My local changes apply cleanly to the new branch and
> my patches work as expected.

Thanks for your review, Manolis.

I've renamed WIP-core-updates to core-updates, and deleted
core-updates-next.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: core-updates merged!

[Leo Famulari]

On Tue, Aug 02, 2016 at 11:28:59PM +0200, Ludovic Courtès wrote:
> Leo Famulari <leo@famulari.name> skribis:
> 
> > On Tue, Aug 02, 2016 at 07:32:23PM +0200, Ludovic Courtès wrote:
> >> As discussed on IRC, SNAFU!  For reasons yet to be elucidated, the
> >> glibc@2.23 package no longer honors /run/current-system/locale.

> > The variable 'libc_cv_localedir', which we set as
> > "/run/current-system/locale/" in the glibc/linux package definition, has
> > been renamed to 'libc_cv_complocaledir'.

> >              ;; See <http://sourceware.org/ml/libc-alpha/2013-03/msg00093.html>.
> > -            ;; FIXME: This hack no longer works on 2.23!
> > -            (string-append "libc_cv_localedir=/run/current-system/locale/"
> > +            (string-append "libc_cv_complocaledir=/run/current-system/locale/"
> >                             ,version)

> Otherwise LGTM; could you push it to core-updates-next?

I've rebuilt my GuixSD system with this change, and I can confirm that
it works.

Setting GUIX_LOCPATH does not work in all cases, at least for me.

For example, the remote shell program 'mosh' requires a UTF-8
environment on the host. Even with GUIX_LOCPATH set correctly in my
login profile, mosh does not find the locales.

Also, the calendar app 'ikhal' is no longer working, and setting
GUIX_LOCPATH does not work for me.

I think we should prioritize this fix.

core-updates

[Ricardo Wurmus]

Hey,

I’ve requested a new evaluation of core-updates on hydra because sbcl
and a few other packages have since been fixed.  This should show us
fewer packages that we need to worry about before merging core-updates.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: core-updates

[Ludovic Courtès]

Hi!

Ricardo Wurmus <rekado@elephly.net> skribis:

> I’ve requested a new evaluation of core-updates on hydra because sbcl
> and a few other packages have since been fixed.  This should show us
> fewer packages that we need to worry about before merging core-updates.

As of this writing hydra.gnu.org is still building 1.8K jobs (mostly
armhf):

  https://hydra.gnu.org/jobset/gnu/core-updates

berlin.guixsd.org is roughly done, building mostly for aarch64:

  https://berlin.guixsd.org/status/

The weather is OK, though it’s surprisingly lower than what I’d expect
given that everything has been built according to Hydra/Cuirass:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix weather --substitute-urls="https://berlin.guixsd.org https://mirror.hydra.gnu.org"
computing 7,047 package derivations for x86_64-linux...
looking for 7,348 store items on https://berlin.guixsd.org...
https://berlin.guixsd.org
  55.2% substitutes available (4,057 out of 7,348)
  7,756.8 MiB of nars (compressed)
  23,011.1 MiB on disk (uncompressed)
  0.000 seconds per request (2.1 seconds in total)
  3,553.4 requests per second
looking for 7,348 store items on https://mirror.hydra.gnu.org...
updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
https://mirror.hydra.gnu.org
  37.3% substitutes available (2,744 out of 7,348)
  9,299.7 MiB of nars (compressed)
  23,535.4 MiB on disk (uncompressed)
  0.266 seconds per request (1,953.6 seconds in total)
  3.8 requests per second
--8<---------------cut here---------------end--------------->8---

FWIW, the closure of my system as reported by:

  guix size $(readlink -f /run/current-system) | tail -1

has shrunk from 2.7G on master to 2.2G on ‘core-updates’, which is nice
(it includes QEMU among other things, which is why it’s so big.)

Ludo’.

Re: core-updates

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> The weather is OK, though it’s surprisingly lower than what I’d expect
> given that everything has been built according to Hydra/Cuirass:

The patch for GHC 8 is not yet part of core-updates, which will
invalidate a lot of packages.  We should get that commit from master and
apply it to core-updates, and then start another evaluation.

Have there been any big failures that could explain the low coverage?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: core-updates

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> I’ve requested a new evaluation of core-updates on hydra because sbcl
>> and a few other packages have since been fixed.  This should show us
>> fewer packages that we need to worry about before merging core-updates.
>
> As of this writing hydra.gnu.org is still building 1.8K jobs (mostly
> armhf):
>
>   https://hydra.gnu.org/jobset/gnu/core-updates
>
> berlin.guixsd.org is roughly done, building mostly for aarch64:
>
>   https://berlin.guixsd.org/status/
>
> The weather is OK, though it’s surprisingly lower than what I’d expect
> given that everything has been built according to Hydra/Cuirass:
>
> $ ./pre-inst-env guix weather --substitute-urls="https://berlin.guixsd.org https://mirror.hydra.gnu.org"
> computing 7,047 package derivations for x86_64-linux...
> looking for 7,348 store items on https://berlin.guixsd.org...
> https://berlin.guixsd.org
>   55.2% substitutes available (4,057 out of 7,348)
>   7,756.8 MiB of nars (compressed)
>   23,011.1 MiB on disk (uncompressed)
>   0.000 seconds per request (2.1 seconds in total)
>   3,553.4 requests per second
> looking for 7,348 store items on https://mirror.hydra.gnu.org...
> updating list of substitutes from 'https://mirror.hydra.gnu.org'... 100.0%
> https://mirror.hydra.gnu.org
>   37.3% substitutes available (2,744 out of 7,348)
>   9,299.7 MiB of nars (compressed)
>   23,535.4 MiB on disk (uncompressed)
>   0.266 seconds per request (1,953.6 seconds in total)
>   3.8 requests per second

Hydra has certainly built far more than 37% of jobs on core updates.
The percentage of built store items should be closer to 90% or more.

I guess the discrepancy is because most of those store items had never
been requested, so the NARs haven't yet been built.  Does that make
sense?

      Mark