OpenSSL “DROWN” vulnerability & grafts

OpenSSL “DROWN” vulnerability & grafts

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1820 bytes --]

Hello!

OpenSSL 1.0.2g was released today, fixing several serious security
vulnerabilities, several of which are referred to as “DROWN” (as has
become security-marketing tradition.)

This gave a good incentive to fix the “grafting” mechanism described at:

  https://www.gnu.org/software/guix/manual/html_node/Security-Updates.html

The problem was that until now, grafting was not recursive:
<http://bugs.gnu.org/22139>.  This is fixed in c22a132, so we “rushed”
to use it in ‘master’ for the OpenSSL upgrade, which is done in caeadfd.

So now is the time to find out how well the new implementation scales
and to address any limitations.  :-)

A potentially disturbing thing with the new code is that it starts
building/downloading things early, typically before it has written “The
following derivations will be built”; see
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22139#13>.

A limitation of the current implementation is that the replacement
package must have exactly the same name and version as the package being
replaced.  So OpenSSL 1.0.2g shows up as /gnu/store/…-openssl-1.0.2f.

The store file name of the old OpenSSL is given by:

  guix build openssl --no-grafts

… and the new one is given by:

  guix build openssl

For example, to verify which OpenSSL(s) your whole profile refers to,
you can run:

  guix gc -R $(readlink -f ~/.guix-profile) | grep openssl

and check the store file names that you get (make sure to turn off
guix-prettify-mode :-)).  Likewise for a GuixSD generation:

  guix gc -R $(guix system build config.scm) | grep openssl

And for running processes:

  lsof | grep /gnu/store/.*openssl

Seems like this tricks could go in the manual under “Security Updates”
no?

Feedback welcome!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: OpenSSL “DROWN” vulnerability & grafts

[Christopher Allan Webber]

Ludovic Courtès writes:

> Hello!
>
> OpenSSL 1.0.2g was released today, fixing several serious security
> vulnerabilities, several of which are referred to as “DROWN” (as has
> become security-marketing tradition.)
>
> This gave a good incentive to fix the “grafting” mechanism described at:
>
>   https://www.gnu.org/software/guix/manual/html_node/Security-Updates.html
>
> The problem was that until now, grafting was not recursive:
> <http://bugs.gnu.org/22139>.  This is fixed in c22a132, so we “rushed”
> to use it in ‘master’ for the OpenSSL upgrade, which is done in caeadfd.
>
> So now is the time to find out how well the new implementation scales
> and to address any limitations.  :-)
>
> A potentially disturbing thing with the new code is that it starts
> building/downloading things early, typically before it has written “The
> following derivations will be built”; see
> <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22139#13>.
>
> A limitation of the current implementation is that the replacement
> package must have exactly the same name and version as the package being
> replaced.  So OpenSSL 1.0.2g shows up as /gnu/store/…-openssl-1.0.2f.
>
> The store file name of the old OpenSSL is given by:
>
>   guix build openssl --no-grafts
>
> … and the new one is given by:
>
>   guix build openssl
>
> For example, to verify which OpenSSL(s) your whole profile refers to,
> you can run:
>
>   guix gc -R $(readlink -f ~/.guix-profile) | grep openssl
>
> and check the store file names that you get (make sure to turn off
> guix-prettify-mode :-)).  Likewise for a GuixSD generation:
>
>   guix gc -R $(guix system build config.scm) | grep openssl
>
> And for running processes:
>
>   lsof | grep /gnu/store/.*openssl
>
> Seems like this tricks could go in the manual under “Security Updates”
> no?
>
> Feedback welcome!
>
> Ludo’.

Just wanted to repeat a *HUGE* congratulations and thank you on landing
this feature!  So necessary!

Re: OpenSSL “DROWN” vulnerability & grafts

[Ludovic Courtès]

I’ve posted a longer explanation of grafts at:

  https://savannah.gnu.org/forum/forum.php?forum_id=8470

Thanks to Chris, David, and Mark for reviewing an earlier draft!

Ludo’.

Re: OpenSSL “DROWN” vulnerability & grafts

[Mathieu Lirzin]

ludo@gnu.org (Ludovic Courtès) writes:

> I’ve posted a longer explanation of grafts at:
>
>   https://savannah.gnu.org/forum/forum.php?forum_id=8470
>
> Thanks to Chris, David, and Mark for reviewing an earlier draft!

Thanks for taking the time to explain it.

The persistent data structures perspective is helpful (at least for me).

-- 
Mathieu Lirzin

Re: OpenSSL “DROWN” vulnerability & grafts

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

That was a very clear and informative read. I appreciate your style of
writing, with links to relevant docs. Thank you!

One thing is still not clear to me about grafts. While you are using a
graft, is it possible the binaries you build will be different from those
built by someone not using the graft? If so, is that a problem, or is there
a reason that it isn't a problem (perhaps because guix will reject binaries
whose hash is different than expected, or something like that)?

- Chris


>
>

[-- Attachment #2: Type: text/html, Size: 715 bytes --]

Re: OpenSSL “DROWN” vulnerability & grafts

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2490 bytes --]

On Tue, Mar 01, 2016 at 10:16:47PM +0100, Ludovic Courtès wrote:
> Hello!
> 
> OpenSSL 1.0.2g was released today, fixing several serious security
> vulnerabilities, several of which are referred to as “DROWN” (as has
> become security-marketing tradition.)
> 
> This gave a good incentive to fix the “grafting” mechanism described at:
> 
>   https://www.gnu.org/software/guix/manual/html_node/Security-Updates.html
> 
> The problem was that until now, grafting was not recursive:
> <http://bugs.gnu.org/22139>.  This is fixed in c22a132, so we “rushed”
> to use it in ‘master’ for the OpenSSL upgrade, which is done in caeadfd.
> 
> So now is the time to find out how well the new implementation scales
> and to address any limitations.  :-)
> 
> A potentially disturbing thing with the new code is that it starts
> building/downloading things early, typically before it has written “The
> following derivations will be built”; see
> <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22139#13>.
> 
> A limitation of the current implementation is that the replacement
> package must have exactly the same name and version as the package being
> replaced.  So OpenSSL 1.0.2g shows up as /gnu/store/…-openssl-1.0.2f.
> 
> The store file name of the old OpenSSL is given by:
> 
>   guix build openssl --no-grafts
> 
> … and the new one is given by:
> 
>   guix build openssl
> 
> For example, to verify which OpenSSL(s) your whole profile refers to,
> you can run:
> 
>   guix gc -R $(readlink -f ~/.guix-profile) | grep openssl
> 
> and check the store file names that you get (make sure to turn off
> guix-prettify-mode :-)).  Likewise for a GuixSD generation:
> 
>   guix gc -R $(guix system build config.scm) | grep openssl
> 
> And for running processes:
> 
>   lsof | grep /gnu/store/.*openssl
> 
> Seems like this tricks could go in the manual under “Security Updates”
> no?
> 
> Feedback welcome!
> 
> Ludo’.

BIG thanks for getting this working, its a great way to keep our systems
up and running while taking care of the security issues.

One issue that I noticed on my slow netbook is that `guix package -u`,
with no updates, now takes ~15 minutes, while before it was ~30 seconds.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: OpenSSL “DROWN” vulnerability & grafts

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> One issue that I noticed on my slow netbook is that `guix package -u`,
> with no updates, now takes ~15 minutes, while before it was ~30 seconds.

Ouch!  Including substitute info fetching?

There are a couple of optimizations that I’m willing to implement as
soon as I stop doing email.  ;-)

Ludo’.

Re: OpenSSL “DROWN” vulnerability & grafts

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1113 bytes --]

On Wed, 02 Mar 2016 22:36:51 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
>  [...]  
> 
> Ouch!  Including substitute info fetching?
> 
> There are a couple of optimizations that I’m willing to implement as
> soon as I stop doing email.  ;-)
> 
> Ludo’.

ran this this morning, about 8 hours after my last `guix package -u`:
efraim@debian-netbook:~/workspace/guix$ time ./pre-inst-env guix package -u && time ./pre-inst-env guix package -u && time ./pre-inst-env guix package -u

real	15m0.833s
user	5m53.268s
sys	1m26.056s

real	14m27.095s
user	5m50.848s
sys	1m25.156s

real	14m37.017s
user	5m55.564s
sys	1m30.536s

so I don't think the substitute info is the part that's slowing it down.
Conky shows me both guix hand guix-daemon working, so more likely something
else, like figuring out the substitutions.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: OpenSSL “DROWN” vulnerability & grafts

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> efraim@debian-netbook:~/workspace/guix$ time ./pre-inst-env guix package -u && time ./pre-inst-env guix package -u && time ./pre-inst-env guix package -u
>
> real	15m0.833s
> user	5m53.268s
> sys	1m26.056s
>
> real	14m27.095s
> user	5m50.848s
> sys	1m25.156s
>
> real	14m37.017s
> user	5m55.564s
> sys	1m30.536s
>
> so I don't think the substitute info is the part that's slowing it down.
> Conky shows me both guix hand guix-daemon working, so more likely something
> else, like figuring out the substitutions.

Terrible.  Commit d4da602 should improve the situation.  You’re welcome
to try again and report back!

ludo@gnu.org (Ludovic Courtès) skribis:

> A potentially disturbing thing with the new code is that it starts
> building/downloading things early, typically before it has written “The
> following derivations will be built”; see
> <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22139#13>.

With commit c90cb5c, the grafting code tries hard to avoid building: it
uses substitute information, when available, to determine the run-time
dependencies of packages (which in turn allows it to decide whether the
package needs to be grafted.)

Ludo’.

Re: OpenSSL “DROWN” vulnerability & grafts

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1429 bytes --]

On Sat, Mar 05, 2016 at 12:24:59AM +0100, Ludovic Courtès wrote:
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > efraim@debian-netbook:~/workspace/guix$ time ./pre-inst-env guix package -u && time ./pre-inst-env guix package -u && time ./pre-inst-env guix package -u
> >
> > real	15m0.833s
> > user	5m53.268s
> > sys	1m26.056s
> >
> > real	14m27.095s
> > user	5m50.848s
> > sys	1m25.156s
> >
> > real	14m37.017s
> > user	5m55.564s
> > sys	1m30.536s
> >
> > so I don't think the substitute info is the part that's slowing it down.
> > Conky shows me both guix hand guix-daemon working, so more likely something
> > else, like figuring out the substitutions.
> 
> Terrible.  Commit d4da602 should improve the situation.  You’re welcome
> to try again and report back!
> 

Much better! The first run I was using my computer also, so it was a bit
slower.

efraim@debian-netbook:~/workspace/guix$ time ./pre-inst-env guix \
package -u && time ./pre-inst-env guix package -u && time ./pre-inst-env \
guix package -u

real    2m41.774s
user    1m50.228s
sys 0m2.388s

real    1m58.805s
user    1m49.560s
sys 0m2.336s

real    1m57.415s
user    1m49.660s
sys 0m2.408s

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: OpenSSL “DROWN” vulnerability & grafts

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Sat, Mar 05, 2016 at 12:24:59AM +0100, Ludovic Courtès wrote:
>> Efraim Flashner <efraim@flashner.co.il> skribis:
>> 
>> > efraim@debian-netbook:~/workspace/guix$ time ./pre-inst-env guix package -u && time ./pre-inst-env guix package -u && time ./pre-inst-env guix package -u
>> >
>> > real	15m0.833s
>> > user	5m53.268s
>> > sys	1m26.056s
>> >
>> > real	14m27.095s
>> > user	5m50.848s
>> > sys	1m25.156s
>> >
>> > real	14m37.017s
>> > user	5m55.564s
>> > sys	1m30.536s
>> >
>> > so I don't think the substitute info is the part that's slowing it down.
>> > Conky shows me both guix hand guix-daemon working, so more likely something
>> > else, like figuring out the substitutions.
>> 
>> Terrible.  Commit d4da602 should improve the situation.  You’re welcome
>> to try again and report back!
>> 
>
> Much better!

Good, thanks for benchmarking!

There are other things (‘bag-grafts’, namely) that are done repeatedly
when doing an upgrade and that would benefit from a bit of
caching/memoization.  Coming next!

Ludo’.