GNU Guix (pull?) on i686 broke after zstd grafting

GNU Guix (pull?) on i686 broke after zstd grafting

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

Hello!

After commit: 
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=6f873731a030dd7ecbd8a5e756b38b26306f6966

This happened:
https://ci.guix.gnu.org/build/369538/details

I made the commit, and not sure what to do here.

The test suite seems to fail on i686..?

Sorry for any inconvenience caused.

Thank you

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GNU Guix (pull?) on i686 broke after zstd grafting

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 656 bytes --]

On Sat, 2021-03-06 at 06:09 +0100, Léo Le Bouter wrote:
> Hello!
> 
> After commit: 
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=6f873731a030dd7ecbd8a5e756b38b26306f6966
> 
> This happened:
> https://ci.guix.gnu.org/build/369538/details
> 
> I made the commit, and not sure what to do here.
> 
> The test suite seems to fail on i686..?
> 
> Sorry for any inconvenience caused.
> 
> Thank you

I think it is an error from ci.guix.gnu.org workers actually running
out of memory because:

guix build --system=i686-linux --no-grafts zstd@1.4.9

succeeded locally for me.

So please restart the job, scared me a bit there.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GNU Guix (pull?) on i686 broke after zstd grafting

[Ludovic Courtès]

Hi Léo,

Léo Le Bouter <lle-bout@zaclys.net> skribis:

> After commit: 
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=6f873731a030dd7ecbd8a5e756b38b26306f6966
>
> This happened:
> https://ci.guix.gnu.org/build/369538/details
>
> I made the commit, and not sure what to do here.
>
> The test suite seems to fail on i686..?

I’ve retried this i686-linux build and it seems to fail consistently:

--8<---------------cut here---------------start------------->8---
*** zstd command line interface 32-bits v1.4.9, by Yann Collet ***
(L3) Buffered :   0 MB - Consumed :   0 MB - Compressed :   0 MB => 0.00%
roundTripTest: datagen -g8M  | zstd -v19 -T0 --long | zstd -d19 -T0 --long

*** zstd command line interface 32-bits v1.4.9, by Yann Collet ***
Note: 48 physical core(s) detected
zstd: error 11 : Allocation error : not enough memory 
zstd: /*stdin*\: unexpected end of file
Files tmp1 and tmp2 differ
make[1]: *** [Makefile:329: test-zstd] Error 1
make[1]: Leaving directory '/tmp/guix-build-zstd-1.4.9.drv-0/zstd-1.4.9/tests'
make: *** [Makefile:90: shortest] Error 2

Test suite failed, dumping logs.
command "make" "check" "-j" "4" "CC=gcc" "PREFIX=/gnu/store/krx4mfj2yyg099kr1bj9pg0rs03fnymb-zstd-1.4.9" "LIBDIR=/gnu/store/la177xpxz00biapfhrrnx51pf1d9r8r3-zstd-1.4.9-lib/lib" "INCLUDEDIR=/gnu/store/la177xpxz00biapfhrrnx51pf1d9r8r3-zstd-1.4.9-lib/include" "HAVE_LZMA=0" "HAVE_LZ4=0" "HAVE_ZLIB=0" failed with status 2
builder for `/gnu/store/w0lbczib88vv83id42i2dx05v84fmpbv-zstd-1.4.9.drv' failed with exit code 1
build of /gnu/store/w0lbczib88vv83id42i2dx05v84fmpbv-zstd-1.4.9.drv failed
--8<---------------cut here---------------end--------------->8---

So I think there’s a genuine bug here.  Could you take a look?  At
worst, we should skip the offending test on i686 (and perhaps ARMv7?).

Thanks,
Ludo’.

Re: GNU Guix (pull?) on i686 broke after zstd grafting

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

On Wed, 2021-03-10 at 11:37 +0100, Ludovic Courtès wrote:
> Hi Léo,

Hi Ludo!

> So I think there’s a genuine bug here.  Could you take a look?  At
> worst, we should skip the offending test on i686 (and perhaps
> ARMv7?).

I reported upstream and I got an answer, waiting for fix but also we
could do something based on their comment:

> So now that we have plausible explanation, what can be done about it
> ?
> 
> - Ignore this test. It's a just test error. Disable it.
> - Change the test, so that it passes. For example, use a fixed nb of 
> threads, like -T4.
> - Update the rule regulating -T0, determining automatically a nb of
> threads based on present nb of cores. Make it conditional for 32-bit
> systems, select a limit for this case (slightly more complex, will
> take time).
> - Update the thread-pool policy to allocate new threads only when
> needed, instead of up front (definitely more complex, will take even
> more time).

I am very tired now however, also not sure how to disable individual
tests there.

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GNU Guix (pull?) on i686 broke after zstd grafting

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

On Wed, 2021-03-10 at 11:37 +0100, Ludovic Courtès wrote:
> So I think there’s a genuine bug here.  Could you take a look?  At
> worst, we should skip the offending test on i686 (and perhaps
> ARMv7?).

I pushed 2bcfb944bdd2f476ef8d34802fed436e4fdda0ab which disables tests
entirely in the graft.

At least this fixes the issue, if anyone wants to still enable tests,
feel free, no energy for that myself now.

This will do while we wait for an upstream patch.

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GNU Guix (pull?) on i686 broke after zstd grafting

[zimoun]

Hi,

On Thu, 11 Mar 2021 at 03:11, Léo Le Bouter <lle-bout@zaclys.net> wrote:
> On Wed, 2021-03-10 at 11:37 +0100, Ludovic Courtès wrote:
>> So I think there’s a genuine bug here.  Could you take a look?  At
>> worst, we should skip the offending test on i686 (and perhaps
>> ARMv7?).
>
> I pushed 2bcfb944bdd2f476ef8d34802fed436e4fdda0ab which disables tests
> entirely in the graft.

This disable the complete test suite for all the architecture.  I have
not look into the details but it seems better to only disable the
offending test only the architecture affected.

Cheers,
simon

Re: GNU Guix (pull?) on i686 broke after zstd grafting

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

On Thu, 2021-03-11 at 10:37 +0100, zimoun wrote:
> This disable the complete test suite for all the architecture.  I
> have
> not look into the details but it seems better to only disable the
> offending test only the architecture affected.

Yes it does that and it would be better not to but zstd 1.4.9 (without
disabled tests) is in core-updates and we are waiting for upstream to
fix the test-suite, this change is not durable as it's a graft for a
security update.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GNU Guix (pull?) on i686 broke after zstd grafting

[zimoun]

On Thu, 11 Mar 2021 at 10:40, Léo Le Bouter <lle-bout@zaclys.net> wrote:
> On Thu, 2021-03-11 at 10:37 +0100, zimoun wrote:
>> This disable the complete test suite for all the architecture.  I
>> have
>> not look into the details but it seems better to only disable the
>> offending test only the architecture affected.
>
> Yes it does that and it would be better not to but zstd 1.4.9 (without
> disabled tests) is in core-updates and we are waiting for upstream to
> fix the test-suite, this change is not durable as it's a graft for a
> security update.

Well, IMHO 1) «not durable» could mean months and 2) disabling all the
tests for all the architectures is wrong especially when only one test
is failing for only one architecture.

Cheers,
simon

Re: GNU Guix (pull?) on i686 broke after zstd grafting

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 525 bytes --]

On Thu, 2021-03-11 at 10:58 +0100, zimoun wrote:
> Well, IMHO 1) «not durable» could mean months and 2) disabling all
> the
> tests for all the architectures is wrong especially when only one
> test
> is failing for only one architecture.

I know that, I was tired yesterday and didnt want to block anyone
running i686-linux or aarch64-linux/armhf-linux (it looked to fail
there also?).

Please fix the situation if you can, or I'll try and do it at some
point but can't say when, lots of things going on now.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[zimoun]

[-- Attachment #1: Type: text/plain, Size: 1874 bytes --]

Hi,

This commit 6f873731a030dd7ecbd8a5e756b38b26306f6966:

<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=6f873731a030dd7ecbd8a5e756b38b26306f6966>

fixes CVE-2021-24032 which says: "Beginning in v1.4.1 and prior to
v1.4.9, output files were created with default permissions. [...]".

The mentioned commit replaces zstd@1.4.4 by zstd@1.4.9 which seems
more than just grafting.  Well,1.4.4 was released on Nov 2019 and
1.4.9 some days ago.

I agree that security is important but we lived more than one and half
year with 1.4.4 so the upgrade to 1.4.9 should only go to
core-updates, not as a 'replacement' graft.  IMHO.

The consequence of this change was the breakage of "guix pull" on
master for at least i686.  Which leads to the commit
2bcfb944bdd2f476ef8d34802fed436e4fdda0ab disabling the zstd test-suite
for all the architectures.

<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=2bcfb944bdd2f476ef8d34802fed436e4fdda0ab>

Noting that "guix pull" should be still failing for at least i686 on
core-updates because of the test suite of zstd@1.4.9.


The question is: should the next release 1.2.1 contain zstd@1.4.9 as
graft?  Or do we revert the commit and simply fix it on core-updates
and wait for the next core-updates cycle.  Personally, I am in favor
of the latter.  WDYT?

The issue is the test:

    roundTripTest -g8M "19 -T0 --long"

which fails for the value 19 but not other values as 18 or 20 or many
others.  After a quick reading of the doc, I am not sure to understand
the meaning of such value.  Input welcome.

BTW, on my machine the attached patch builds for both x86_64 and i686
(emulated).

   ./pre-inst-env guix build zstd@1.4.9 --system=i686-linux --no-grafts

Depending on the answer of the previous question, the patch should go
to master or core-updates.  And other architectures should be examined
with care.


Cheers,
simon

[-- Attachment #2: fix-zstd-i686.patch --]
[-- Type: text/x-patch, Size: 1059 bytes --]

diff --git a/gnu/packages/compression.scm b/gnu/packages/compression.scm
index 827ad43dc2..86ce3a697d 100644
--- a/gnu/packages/compression.scm
+++ b/gnu/packages/compression.scm
@@ -32,6 +32,7 @@
 ;;; Copyright © 2020 Léo Le Bouter <lle-bout@zaclys.net>
 ;;; Copyright © 2021 Antoine Côté <antoine.cote@posteo.net>
 ;;; Copyright © 2021 Vincent Legoll <vincent.legoll@gmail.com>
+;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -1483,7 +1484,13 @@ speed.")
         (base32 "14yj7309gsvg39rki4xqnd6w5idmqi0655v1fc0mk1m2kvhp9b19"))))
     (arguments
      (substitute-keyword-arguments (package-arguments zstd)
-       ((#:tests? _ #t) #f)))))
+       ((#:phases phases)
+        `(modify-phases ,phases
+           (add-after 'unpack 'fix-test-i686
+             (lambda _
+               (substitute* "tests/playTests.sh"
+                 (("roundTripTest -g8M \"19 -T0 --long\"")
+                  "roundTripTest -g8M \"22 -T0 --long\""))))))))))
 
 (define-public pzstd
   (package

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 233 bytes --]

I suggest we disable the test-suite or the specific test in the interim
for other architectures.

The CVE-2021-24032 is Base Score: 9.1 CRITICAL - which is exceptionally
high so fixing it is an absolute necessity in any branch.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]

On Tue, Mar 16, 2021 at 06:06:28PM +0100, Léo Le Bouter wrote:
> The CVE-2021-24032 is Base Score: 9.1 CRITICAL - which is exceptionally
> high so fixing it is an absolute necessity in any branch.

This is off-topic, but I think that CVE scoring is not really that
useful. This bug is a local TOCTOU race which is bad but hardly
critical, IMO. For something to be critical, it should enable remote
execution of arbitrary code.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 2332 bytes --]

On Tue, 2021-03-16 at 13:48 -0400, Leo Famulari wrote:
> This is off-topic, but I think that CVE scoring is not really that
> useful. This bug is a local TOCTOU race which is bad but hardly
> critical, IMO. For something to be critical, it should enable remote
> execution of arbitrary code.

Well you don't know what people use zstd for, easily escalates to more
critical issues depending on people's use case. Also I think CRITICAL
reasoning here is also because it's a trivial to understand and exploit
issue, it's not like an obscure memory safety issue with no known PoC
but probable exploitation.

I do not agree in general not patching CVEs even if low (publicized)
severity as long as it's possible for us to do it. Often the
vulnerabilities have an unobserved attack angle and severity may be
underevaluated. The zstd patch was tested on x86_64-linux it's
unfortunate the test suite fails (errornously, not an actual fault) on
32bit archs, otherwise it's no issue. I wish the zstd test suite was
more reliable in general, generating random data in their test suite
doesnt help determinism here. I think I tried on i686-linux to build as
well and it succeeded for me so I pushed, but it didnt fail on me and
when I retried later it did, so definitely some non-determinism here.

Since we know there's no actual fault in the test suite because it
passes I thought it was relatively fine to disable the test suite
temporarily until core-updates comes in (if we don't change versions in
between and revisit).

Zimoun:
> which fails for the value 19 but not other values as 18 or 20 or many
> others.  After a quick reading of the doc, I am not sure to
> understand
> the meaning of such value.  Input welcome.

https://github.com/facebook/zstd/issues/2528 - I asked upstream earlier
and see their answer

> I agree that security is important but we lived more than one and
> half
> year with 1.4.4 so the upgrade to 1.4.9 should only go to
> core-updates, not as a 'replacement' graft.  IMHO.

To add, I don't think we should reason that way, it's not because we
lived with something that we should live with it longer, I don't want
unpatched zstd (or any other) CVEs on my system. Actually I am not sure
1.4.4 had any CVE before that one though, so that must also be why.

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[zimoun]

Hi,

On Tue, 16 Mar 2021 at 18:06, Léo Le Bouter <lle-bout@zaclys.net> wrote:

> I suggest we disable the test-suite or the specific test in the interim
> for other architectures.

The patch attached in the previous email tweaks the offending test to
allow the test suite to pass on both architectures x86_64 and i686.  I
am not able to test the other architectures.

Well, this upgrading zstd from 1.4.4 to 1.4.9 is one way to fix, but
we could also graft by backporting a patch.  As Debian did for 1.4.8:

<https://salsa.debian.org/med-team/libzstd/-/blob/master/debian/patches/0018-fix-file-permissions-on-compression.patch>


> The CVE-2021-24032 is Base Score: 9.1 CRITICAL - which is exceptionally
> high so fixing it is an absolute necessity in any branch.

For Suse, the severity is moderate and they rank to 6.2.

<https://www.suse.com/security/cve/CVE-2021-24032/>

Well, even if I agree that security is often important, more haste and
less speed, is generally good. :-)


Cheers,
simon

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

On Tue, Mar 16, 2021 at 05:34:34PM +0100, zimoun wrote:
> The question is: should the next release 1.2.1 contain zstd@1.4.9 as
> graft?  Or do we revert the commit and simply fix it on core-updates
> and wait for the next core-updates cycle.  Personally, I am in favor
> of the latter.  WDYT?

The release should not contain any grafts, if we can help it.

On the wip-next-release branch, I've simply updated zstd to 1.4.9:

https://git.savannah.gnu.org/cgit/guix.git/log/?h=wip-next-release

I do agree that updating this program 5 versions in a graft was perhaps
too much.

We should always try to cherry-pick bug-fix patches when grafting.

Otherwise the risk of breakage is too high. At least, these types of
patches should be reviewed on guix-patches. Léo, can you send them to
guix-patches in the future?

Sometimes it is okay to update things in a graft, but it depends on the
situation.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

On Tue, 2021-03-16 at 13:55 -0400, Leo Famulari wrote:
> I do agree that updating this program 5 versions in a graft was
> perhaps
> too much.
> 
> We should always try to cherry-pick bug-fix patches when grafting.
> 
> Otherwise the risk of breakage is too high. At least, these types of
> patches should be reviewed on guix-patches. Léo, can you send them to
> guix-patches in the future?
> 
> Sometimes it is okay to update things in a graft, but it depends on
> the
> situation.

1.4.4 and 1.4.9 are ABI compatible? At least that's the reason I
believed it wasnt risky. I can send them to the mailing list especially
with such a core package (GNU Guix dependency). But often it stays
there and no one is looking so. E.g. the unzip vulnerability patches,
nobody looked until I actually pushed them out of waiting for reviews,
I tried to hint multiple people on IRC during several days, no answer
still, so I ended up pushing it, turns out I had several mistakes in it
and because it was pushed well some people looked at it and helped
fixing which was welcome.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[zimoun]

On Tue, 16 Mar 2021 at 19:08, Léo Le Bouter <lle-bout@zaclys.net> wrote:
On Tue, 2021-03-16 at 13:55 -0400, Leo Famulari wrote:

> > I do agree that updating this program 5 versions in a graft was
> > perhaps
> > too much.
> >
> > We should always try to cherry-pick bug-fix patches when grafting.
> >
> > Otherwise the risk of breakage is too high. At least, these types of
> > patches should be reviewed on guix-patches. Léo, can you send them to
> > guix-patches in the future?
> >
> > Sometimes it is okay to update things in a graft, but it depends on
> > the
> > situation.
>
> 1.4.4 and 1.4.9 are ABI compatible? At least that's the reason I
> believed it wasnt risky. I can send them to the mailing list especially
> with such a core package (GNU Guix dependency). But often it stays
> there and no one is looking so. E.g. the unzip vulnerability patches,
> nobody looked until I actually pushed them out of waiting for reviews,
> I tried to hint multiple people on IRC during several days, no answer
> still, so I ended up pushing it, turns out I had several mistakes in it
> and because it was pushed well some people looked at it and helped
> fixing which was welcome.

Well, it seems better to send such changes to guix-patches, waiting 15
days, and then if no comment, push.  It is what the manual describes:

        Non-trivial patches should always be posted to
        guix-patches@gnu.org (trivial patches include fixing typos,
        etc.). […]

        For patches that just add a new package, and a simple one, it’s OK to
        commit, if you’re confident […].  Likewise for package upgrades, except
        upgrades that trigger a lot of rebuilds […].

        […]

        […] If you didn’t receive any reply after two weeks, and if
        you’re confident, it’s OK to commit.

<http://guix.gnu.org/manual/devel/en/guix.html#Commit-Access>

And from my understanding, it is a non-trivial patch which triggers a
lot of rebuilds.  Double reasons. ;-)


Cheers,
simon

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 1040 bytes --]

On Tue, 2021-03-16 at 19:46 +0100, zimoun wrote:
> Well, it seems better to send such changes to guix-patches, waiting
> 15
> days, and then if no comment, push.  It is what the manual describes:
> 
>         Non-trivial patches should always be posted to
>         guix-patches@gnu.org (trivial patches include fixing typos,
>         etc.). […]
> 
>         For patches that just add a new package, and a simple one,
> it’s OK to
>         commit, if you’re confident […].  Likewise for package
> upgrades, except
>         upgrades that trigger a lot of rebuilds […].
> 
>         […]
> 
>         […] If you didn’t receive any reply after two weeks, and if
>         you’re confident, it’s OK to commit.
> 
> <http://guix.gnu.org/manual/devel/en/guix.html#Commit-Access>
> 
> And from my understanding, it is a non-trivial patch which triggers a
> lot of rebuilds.  Double reasons. ;-)

It's not just like any other patch, it's a security patch, so **15**
days..

> 
> Cheers,
> simon

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[zimoun]

On Tue, 16 Mar 2021 at 19:51, Léo Le Bouter <lle-bout@zaclys.net> wrote:
> On Tue, 2021-03-16 at 19:46 +0100, zimoun wrote:

> > Well, it seems better to send such changes to guix-patches, waiting
> > 15
> > days, and then if no comment, push.  It is what the manual describes:
> >
> >         Non-trivial patches should always be posted to
> >         guix-patches@gnu.org (trivial patches include fixing typos,
> >         etc.). […]
> >
> >         For patches that just add a new package, and a simple one,
> > it’s OK to
> >         commit, if you’re confident […].  Likewise for package
> > upgrades, except
> >         upgrades that trigger a lot of rebuilds […].
> >
> >         […]
> >
> >         […] If you didn’t receive any reply after two weeks, and if
> >         you’re confident, it’s OK to commit.
> >
> > <http://guix.gnu.org/manual/devel/en/guix.html#Commit-Access>
> >
> > And from my understanding, it is a non-trivial patch which triggers a
> > lot of rebuilds.  Double reasons. ;-)
>
> It's not just like any other patch, it's a security patch, so **15**
> days..

"To me, security is important. But it's no less important than
everything *else* that is also important!"

And nothing blocks you to have the patch in your tree in these (max)
15 days review time.

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[zimoun]

Hi,

On Tue, 16 Mar 2021 at 18:56, Leo Famulari <leo@famulari.name> wrote:
>
> On Tue, Mar 16, 2021 at 05:34:34PM +0100, zimoun wrote:
> > The question is: should the next release 1.2.1 contain zstd@1.4.9 as
> > graft?  Or do we revert the commit and simply fix it on core-updates
> > and wait for the next core-updates cycle.  Personally, I am in favor
> > of the latter.  WDYT?
>
> The release should not contain any grafts, if we can help it.
>
> On the wip-next-release branch, I've simply updated zstd to 1.4.9:

I guess that it will not build for i686.  Does it?
If not, the patch attached to the previous email tweaks the offending
test; as the original author of zstd has suggested:

<https://github.com/facebook/zstd/issues/2528#issuecomment-794704882>

(Thanks to the other Leo for opening the issue.)


Well, I am confused.  If the update of zstd from 1.4.4 to 1.4.9 does
not imply a huge rebuild, why is it a graft?  And not a simple update?


Cheers,
simon

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]

On Tue, 2021-03-16 at 19:19 +0100, zimoun wrote:
> I guess that it will not build for i686.  Does it?
> If not, the patch attached to the previous email tweaks the offending
> test; as the original author of zstd has suggested:
> 
> <https://github.com/facebook/zstd/issues/2528#issuecomment-794704882>
> 
> (Thanks to the other Leo for opening the issue.)

Indeed it would not pass tests, thanks for the patch.

> 
> Well, I am confused.  If the update of zstd from 1.4.4 to 1.4.9 does
> not imply a huge rebuild, why is it a graft?  And not a simple
> update?

Well there is some huge rebuild involved, but there is something else
happening here, the zstd package as a specification now refers to 
zstd@1.4.9 and not zstd@1.4.4 (as grafted) because the version is
newer, I should've made the zstd@1.4.9 graft package definition private
here as I do now for other grafts.

$ ./pre-inst-env guix refresh -l zstd
Building the following 2 packages would ensure 2 dependent packages are
rebuilt: ecl-zstd@1.0-1.d144582 cl-zstd@1.0-1.d144582

We see only 2 here, but it's a false result, zstd is a dependency to
way more,

Then if we do this:

$ ./pre-inst-env guix refresh -l zstd@1.4.4
Building the following 5115 packages would ensure 10443 dependent
packages are rebuilt
[...]

There we are, almost all packages need to be rebuilt.

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Leo Famulari]

On Tue, Mar 16, 2021 at 07:19:53PM +0100, zimoun wrote:
> I guess that it will not build for i686.  Does it?

I don't know. Either we will find out when building on CI, or people can
test it manually now.

We might consider building the wip-next-release earlier than you had
suggested. There is a large number of major updates on the branch, so it
will unfortunately be more like core-updates than I had planned for.

If that's not acceptable, we can narrow the scope of the existing
grafts, by cherry-picking bug fix patches rather than updating the
entire package.

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[zimoun]

Hi,

On Tue, 16 Mar 2021 at 20:18, Leo Famulari <leo@famulari.name> wrote:
> On Tue, Mar 16, 2021 at 07:19:53PM +0100, zimoun wrote:
> > I guess that it will not build for i686.  Does it?
>
> I don't know. Either we will find out when building on CI, or people can
> test it manually now.

Please try out the patch from:

<https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00295.html>

and if it works for you, please apply it.

> We might consider building the wip-next-release earlier than you had
> suggested. There is a large number of major updates on the branch, so it
> will unfortunately be more like core-updates than I had planned for.

Let's do that. :-)

> If that's not acceptable, we can narrow the scope of the existing
> grafts, by cherry-picking bug fix patches rather than updating the
> entire package.

You are right, it could be nice to have ASAP an idea about what is
acceptable and then how to narrow.


Cheers,
simon

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Leo Famulari]

On Tue, Mar 16, 2021 at 08:25:50PM +0100, zimoun wrote:
> Hi,
> 
> On Tue, 16 Mar 2021 at 20:18, Leo Famulari <leo@famulari.name> wrote:
> > On Tue, Mar 16, 2021 at 07:19:53PM +0100, zimoun wrote:
> > > I guess that it will not build for i686.  Does it?
> >
> > I don't know. Either we will find out when building on CI, or people can
> > test it manually now.
> 
> Please try out the patch from:
> 
> <https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00295.html>
> 
> and if it works for you, please apply it.

No, sorry :) Someone else (maybe an i686 user?) will have to find the
time to test it.

Security-czar needed? WAS: Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Bengt Richter]

Hi all,

On +2021-03-16 15:29:43 -0400, Leo Famulari wrote:
> On Tue, Mar 16, 2021 at 08:25:50PM +0100, zimoun wrote:
> > Hi,
> > 
> > On Tue, 16 Mar 2021 at 20:18, Leo Famulari <leo@famulari.name> wrote:
> > > On Tue, Mar 16, 2021 at 07:19:53PM +0100, zimoun wrote:
> > > > I guess that it will not build for i686.  Does it?
> > >
> > > I don't know. Either we will find out when building on CI, or people can
> > > test it manually now.
> > 
> > Please try out the patch from:
> > 
> > <https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00295.html>
> > 
> > and if it works for you, please apply it.
> 
> No, sorry :) Someone else (maybe an i686 user?) will have to find the
> time to test it.
> 

I would feel better about running guix on my laptop if I
knew all you developers had gotten together and elected
a "security czar" who is the most competent of you to monitor
security and also cares the most, and had the power to prevent
applying unreviewed patches, and making sure all CVEs are taken
care of, and kitchen doors not left open the way we did in the '50s.

Sorry if it sounds like I think guix security is lax.
Please convince me it's not so ;)

Thanks, nevertheless, for all the great technical work!

Just wish I could type
    guix --what-and-who-am-I-trusting-q --full-report
and get a complete list, with batting averages of the
developers (regressions vs fixes), packages (estimated
number of times executed without problem, dangerous bugs
in development history, etc).

</rant>

-- 
Regards,
Bengt Richter

Re: Security-czar needed? WAS: Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Leo Famulari]

On Tue, Mar 16, 2021 at 10:46:11PM +0100, Bengt Richter wrote:
> Just wish I could type
>     guix --what-and-who-am-I-trusting-q --full-report
> and get a complete list, with batting averages of the
> developers (regressions vs fixes), packages (estimated
> number of times executed without problem, dangerous bugs
> in development history, etc).

Leaving aside the rest of your suggestion, which has merit, I strongly
object to ranking Guix contributors in that way. Most of us feel bad
enough about our mistakes without some kind of public scoreboard.

In general, as the person who was the de facto security team leader for
several years, I feel that such a position should be supported in a
material way.

Re: Security-czar needed? WAS: Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]

On Tue, 2021-03-16 at 22:46 +0100, Bengt Richter wrote:
> I would feel better about running guix on my laptop if I
> knew all you developers had gotten together and elected
> a "security czar" who is the most competent of you to monitor
> security and also cares the most, and had the power to prevent
> applying unreviewed patches, and making sure all CVEs are taken
> care of, and kitchen doors not left open the way we did in the '50s.
> 
> Sorry if it sounds like I think guix security is lax.
> Please convince me it's not so ;)
> 
> Thanks, nevertheless, for all the great technical work!
> 
> Just wish I could type
>     guix --what-and-who-am-I-trusting-q --full-report
> and get a complete list, with batting averages of the
> developers (regressions vs fixes), packages (estimated
> number of times executed without problem, dangerous bugs
> in development history, etc).
> 
> </rant>
> 

I think we can handle this without granting us any special powers, I
like it that we don't have roles actually!

We can discuss, debate, agree to common goals, I don't think we are
going to enter into conflict, we hear each other, we communicate, I
think that's a really good thing in GNU Guix :-D

Lots of other communities enter into conflict fast and stop
communicating, GNU Guix is not that, there's a spirit of goodwill of
everyone and that's really pleasing to live as a contributor and user.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Security-czar needed? WAS: Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[zimoun]

Hi,

On Wed, 17 Mar 2021 at 07:24, Léo Le Bouter <lle-bout@zaclys.net> wrote:

> I think we can handle this without granting us any special powers, I
> like it that we don't have roles actually!
>
> We can discuss, debate, agree to common goals, I don't think we are
> going to enter into conflict, we hear each other, we communicate, I
> think that's a really good thing in GNU Guix :-D
>
> Lots of other communities enter into conflict fast and stop
> communicating, GNU Guix is not that, there's a spirit of goodwill of
> everyone and that's really pleasing to live as a contributor and user.

I agree and am aligned with these words. (Without saying there is de
facto hats. :-))

The downside is that sometimes things are stalling.  Examples:
core-updates unmerged since ~10 months, patches that fall in the crack,
old bugs never closed, etc. Pick any non fun stuff. :-)

Hard topic about collective work in general: is it possible to scale
with only implicit hats and no explicit ones? :-)  Hat meaning feel in
charge and do the job to make it happen.


Cheers,
simon

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

On Tue, 2021-03-16 at 15:29 -0400, Leo Famulari wrote:
> > [...]
> 
> No, sorry :) Someone else (maybe an i686 user?) will have to find the
> time to test it.

I haven't tried the patch, but note that x86-64 systems are also
i686 systems, so users of x86-64 systems can try

  ./pre-inst-env guix build --system=i686-linux zstd


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1425 bytes --]

Hi L[ée]o,

Wow, Léo.  You've done some seriously impressive CVE squashing in 
such a short timespan, and I'm very grateful to have you on board.

Leo Famulari 写道：
> I do agree that updating this program 5 versions in a graft was 
> perhaps
> too much.
>
> We should always try to cherry-pick bug-fix patches when 
> grafting.
> Otherwise the risk of breakage is too high.

I agree.  Whilst grafts are indispensible for timely deployment of 
security patches, they're also a dirty hack composed entirely of 
rough edges.

They exist for one purpose: patch out known vulnerabilities. 
Every extra change not strictly required for security is a 
liability.

We sometimes get away with grafting entire releases (OpenSSL comes 
to mind), but this is not an ideal to emulate.

> At least, these types of patches should be reviewed on 
> guix-patches.
> Léo, can you send them to guix-patches in the future?

I have the same request :-)  Please submit non-trivial patches for 
review (and, unfortunately, grafts are hardly ever trivial).  This 
isn't a comment on your work; it's our standard way of doing 
things.

I know we're not the #1 bestest project when it comes to the swift 
review of patches.  I understand the sense of urgency in fixing 
things that one feels should have been fixed long ago.  Thank you 
for helping us to improve on both points.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Vincent Legoll]

Hello,

On Tue, Mar 16, 2021 at 9:53 PM Tobias Geerinckx-Rice <me@tobias.gr> wrote:
> Wow, Léo.  You've done some seriously impressive CVE squashing in
> such a short timespan, and I'm very grateful to have you on board.

Yes, impressive, I have been following the repology page about potentially
vulnerable & upgradable packages for Guix, and the number has significantly
decreased the last weeks, kudos !

I did some package updates (chosen from the very same page) but unlike
you, I only cherry-picked the low hanging fruits from there and punted on
the more involved ones. A good part of that ended on core-updates due to
the rebuilds needed.

I think we really should be shortening our releases cycles (core-updates,
staging merges), because piling upon those branches for too long increase
the disruption in a way that is probably more exponential than linear.

My perception is the following (please correct me if I'm wrong):

A graft involves work on master for the inherited package & graft, sometimes
an update of the package on core updates, then the cleanup (which are more
or less all done in a short time frame when we want to release). So while it
may good enough for some fixes, they should be limited in number and in time,
which also comes to the release early, release often (in a reasonable way).

I was told that we can always update packages because guix easily allows
anyone to go back to a working state, the same reasoning should be applicable
to staging and core-updates merging. Why delay them for too long if
the potential
disruption is mitigated by going back to a workinig profile or system generation
(modulo the substitute availability which is almost only a compute resource
problem)

Cheers

-- 
Vincent Legoll

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Leo Famulari]

On Tue, Mar 16, 2021 at 10:18:08PM +0100, Vincent Legoll wrote:
> I think we really should be shortening our releases cycles (core-updates,
> staging merges), because piling upon those branches for too long increase
> the disruption in a way that is probably more exponential than linear.

For most grafted packages, it's always been the goal to regularly
ungraft, maybe within a few weeks. However, we have never actually had
the build farm capacity to do this for all the platforms that we
support.

Currently, I think we have the capacity for x86_64 and i686 (they use
the same build machines), but not for anything else.

Some packages that qualify for grafts can usually be updated without any
breakage, like OpenSSL.

But other packages, like glibc, cannot be updated in isolation. They
require extensive validation and updates of other packages, sometimes
even requiring patching. It's not just a matter of compute capacity.

This distinction actually highlights what is meant by "core" in Guix.
Due to lack of build farm capacity, core-updates has come to encompass
any changes that causes more than 1800 rebuilds. But, "core" is actually
defined explicitly in Guix [0], and it's literally the core of the
package graph and the GNU system. As the number of packages in Guix has
grown, more and more non-core packages have come to fit in core-updates.
Maybe there is room for improvment here, but I don't know what it is.

I do think we should strive to ungraft things more quickly, maybe after
clarifying the support status of the armhf, aarch64, and ppc64le
platforms.

[0] Check the core-package? procedure in guix/scripts/refresh.scm

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 1515 bytes --]

On Tue, 2021-03-16 at 21:53 +0100, Tobias Geerinckx-Rice wrote:
> Hi L[ée]o,
> 
> Wow, Léo.  You've done some seriously impressive CVE squashing in 
> such a short timespan, and I'm very grateful to have you on board.

I spent few days on this, it's not that much! I did not do much work, I
didnt manually backport any patch (or only some with trivial changes
that can be made by manually editing the patch file), I don't feel I
deserve such thanks, even if they are appreciated, and we are all to be
thanked for our work, it's not that impressive, I think I just went
down with serious motivation looking at it and doing the repetitive
work there.

> I have the same request :-)  Please submit non-trivial patches for 
> review (and, unfortunately, grafts are hardly ever trivial).  This 
> isn't a comment on your work; it's our standard way of doing 
> things.
> 
> I know we're not the #1 bestest project when it comes to the swift 
> review of patches.  I understand the sense of urgency in fixing 
> things that one feels should have been fixed long ago.  Thank you 
> for helping us to improve on both points.

I understand, I am sorry for unsolicited breakage anything might have
caused, I will try to put it up for review, I already do try, zstd was
one of my earlier grafts, I did realize even by myself that after
pushing it I was touching a very critical part of GNU Guix and it
would've been better maybe to ask for review before.

> Kind regards,
> 
> T G-R

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Release 1.2.1: zstd 1.4.4 -> 1.4.9: grafting or core-updates?

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

For reference, crossposting:

I pushed 00c67375b17f4a4cfad53399d1918f2e7eba2c7d to core-updates. Your
patch. Thank you for it. Let's watch for upstream zstd fix also.

I pushed 9feef62b73e284e106717a386624d6da90750a3d to master.

Ubuntu released a patch in the mean time, so while we couldnt make such
patch in a timely manner because the backport was non-trivial and
security-sensitive also didnt want to risk failing to fix the flaw
because I don't have much expertise on it, Ubuntu now has done that
work and we can just use it.

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]