Securing the software distribution chain

Securing the software distribution chain

[Justus Winter]

[-- Attachment #1: Type: text/plain, Size: 2919 bytes --]

Hello :)

doing some packaging lately I noticed a weak link in Guix'
authentication chain.  Artifacts downloaded by Guix are authenticated
using a hashsum included in the packaging definition, and 'guix
download' will compute this hashsum over artifacts, but the step of
authenticating the artifact is a manual one, mentioned in the
'Submitting Patches' checklist:

> Before submitting a patch that adds or modifies a package definition,
> please run through this check list:
>
> 1. If the authors of the packaged software provide a cryptographic
>    signature for the release tarball, make an effort to verify the
>    authenticity of the archive. For a detached GPG signature file this
>    would be done with the gpg --verify command.

For me, this is a tooling problem.  'guix download' should authenticate
the downloaded artifact before it computes and prints the hashsum.
There are two problems to solve: It needs to locate the signature, and
it has to know the set of cryptographic identities eligible to create
the signature.

For some transports, like git, locating the signatures is not a problem,
but for http, there is just no standard on where the detatched signature
is located, or even what data it is computed over (for example,
kernel.org signs the uncompressed tarball).

So I think two things need to happen before this step can be improved:
The package metadata should include the URL of the signature and a set
of cryptographic identities eligible for signing the artifact.


Thinking a bit more about having a hashsum as part of the packaging
definition, it seems to me that this is a bit of a modelling error.
Because there is no standardized way of authenticating a source
distribution, Guix defers this step to the packager.  And if there is no
way to authenticate an artifact (because upstream doesn't provide
signatures), we at least get TOFU, i.e. the assurance that any user
gets the same artifact as the packager.

This doesn't seem terribly problematic, but it doesn't support parts of
the artifact changing, because in this model, this cannot be
distinguished from an attack.  Is there a way an artifact may change in
a valid way that Guix (and other distributions) may want to support?

We believe there is.  We propose to solve the problem of locating the
signatures by bundling them with the source distribution.  Instead of
using a detached PGP signature, we want to distribute the source as a
signed PGP message.

Now, if you compute a hashsum over such an artifact, you are in effect
notarizing the signatures in the message and the message payload.  If
the developers add a signature to the message, the hashsum changes and
your notarization breaks.

The preferred way to support this is to not verify that the hashsum over
the artifact matches, but to verify the PGP signatures over the payload
using the set of eligible signing keys in the package metadata.


Thoughts?
Justus

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Securing the software distribution chain

[Ludovic Courtès]

Hi Justus!

Justus Winter <teythoon@avior.uberspace.de> skribis:

>> Before submitting a patch that adds or modifies a package definition,
>> please run through this check list:
>>
>> 1. If the authors of the packaged software provide a cryptographic
>>    signature for the release tarball, make an effort to verify the
>>    authenticity of the archive. For a detached GPG signature file this
>>    would be done with the gpg --verify command.
>
> For me, this is a tooling problem.  'guix download' should authenticate
> the downloaded artifact before it computes and prints the hashsum.
> There are two problems to solve: It needs to locate the signature, and
> it has to know the set of cryptographic identities eligible to create
> the signature.

Note that ‘guix refresh -u’ and ‘guix import gnu’ (and maybe other
importers too?) take care of tarball authentication already.  ‘guix
download’ could share part of the mechanism.  I agree it would be nice.

> For some transports, like git, locating the signatures is not a problem,
> but for http, there is just no standard on where the detatched signature
> is located, or even what data it is computed over (for example,
> kernel.org signs the uncompressed tarball).

Yes, (guix upstream), which is what ‘guix refresh -u’ uses, takes care
of the kernel.org “special case”.

> So I think two things need to happen before this step can be improved:
> The package metadata should include the URL of the signature and a set
> of cryptographic identities eligible for signing the artifact.

The idea of storing cryptographic metadata directly in <origin> has been
discussed a few times:

  https://lists.gnu.org/archive/html/help-guix/2016-08/msg00132.html
  https://lists.gnu.org/archive/html/guix-devel/2015-10/msg00118.html
  https://git.savannah.gnu.org/cgit/guix.git/tree/TODO?id=6b8875c838d637773813899b35a9b5ea4acfd146#n29

To me, the biggest shortcoming of this approach is if this metadata is
primarily “ornamental”: if in practice, ‘guix build -S’ doesn’t use it
(and it has no reason to use it), then that metadata is likely to become
stale without anyone noticing.

Of course we could have additional tools to make use of that info, say
‘guix build -S --authenticate’ or something.  But that would still be
optional.

> Thinking a bit more about having a hashsum as part of the packaging
> definition, it seems to me that this is a bit of a modelling error.
> Because there is no standardized way of authenticating a source
> distribution, Guix defers this step to the packager.  And if there is no
> way to authenticate an artifact (because upstream doesn't provide
> signatures), we at least get TOFU, i.e. the assurance that any user
> gets the same artifact as the packager.
>
> This doesn't seem terribly problematic, but it doesn't support parts of
> the artifact changing, because in this model, this cannot be
> distinguished from an attack.  Is there a way an artifact may change in
> a valid way that Guix (and other distributions) may want to support?

What do you mean?  If, for example, a tarball is modified in-place
upstream, it’s an error from Guix’ viewpoint.

> We believe there is.  We propose to solve the problem of locating the
> signatures by bundling them with the source distribution.  Instead of
> using a detached PGP signature, we want to distribute the source as a
> signed PGP message.
>
> Now, if you compute a hashsum over such an artifact, you are in effect
> notarizing the signatures in the message and the message payload.  If
> the developers add a signature to the message, the hashsum changes and
> your notarization breaks.
>
> The preferred way to support this is to not verify that the hashsum over
> the artifact matches, but to verify the PGP signatures over the payload
> using the set of eligible signing keys in the package metadata.

In practice, we don’t get to choose the authentication method.  You’re
proposing a different authentication method here, but we’re just
downstream: you’ll have to convince upstreams first.  :-)

On a related note, and perhaps that’s what you mean by “parts of the
artifact changing”, see the discussion on authenticating source code
archived at Software Heritage:

  https://sympa.inria.fr/sympa/arc/swh-devel/2016-07/msg00009.html
  https://forge.softwareheritage.org/T2430#46046
  https://issues.guix.gnu.org/42162#4

Content-addressing is nice, but not very useful if each tool (IPFS, SWH,
Git, Guix) has its own way to address content…

Thoughts?

Ludo’.

Re: Securing the software distribution chain

[zimoun]

Dear,

On Mon, 27 Jul 2020 at 14:54, Ludovic Courtès <ludo@gnu.org> wrote:

> Of course we could have additional tools to make use of that info, say
> ‘guix build -S --authenticate’ or something.  But that would still be
> optional.

What do you mean?

The command "guix build -S" returns the tarball (where non-free code
is removed).  Therefore, this hypothetical and optional
"--authenticate" would authenticate against who?  The user who runs
the command; well I am not sure it is an useful use-case.  The build
farm which would authenticate substitutes, but the commits are already
signed so it would not add some trust


> Note that ‘guix refresh -u’ and ‘guix import gnu’ (and maybe other
> importers too?) take care of tarball authentication already.  ‘guix
> download’ could share part of the mechanism.  I agree it would be nice.

[...]

> On a related note, and perhaps that’s what you mean by “parts of the
> artifact changing”, see the discussion on authenticating source code
> archived at Software Heritage:
>
>   https://sympa.inria.fr/sympa/arc/swh-devel/2016-07/msg00009.html
>   https://forge.softwareheritage.org/T2430#46046
>   https://issues.guix.gnu.org/42162#4
>
> Content-addressing is nice, but not very useful if each tool (IPFS, SWH,
> Git, Guix) has its own way to address content…

Well, the challenge seems here.  First transition from url-fetch
signed tarballs to authenticable content-addressed code such as signed
git-fetch and second be able to bridge the different address contents.
Or let fall in the trap [1]. :-)

1: https://xkcd.com/927/

All the best,
simon

Re: Securing the software distribution chain

[Justus Winter]

[-- Attachment #1: Type: text/plain, Size: 4694 bytes --]

Hello :)

Ludovic Courtès <ludo@gnu.org> writes:
> Justus Winter <teythoon@avior.uberspace.de> skribis:
>> So I think two things need to happen before this step can be improved:
>> The package metadata should include the URL of the signature and a set
>> of cryptographic identities eligible for signing the artifact.
>
> The idea of storing cryptographic metadata directly in <origin> has been
> discussed a few times:
>
>   https://lists.gnu.org/archive/html/help-guix/2016-08/msg00132.html
>   https://lists.gnu.org/archive/html/guix-devel/2015-10/msg00118.html
>   https://git.savannah.gnu.org/cgit/guix.git/tree/TODO?id=6b8875c838d637773813899b35a9b5ea4acfd146#n29
>
> To me, the biggest shortcoming of this approach is if this metadata is
> primarily “ornamental”: if in practice, ‘guix build -S’ doesn’t use it
> (and it has no reason to use it), then that metadata is likely to become
> stale without anyone noticing.

On the other hand, most packaging metadata is ornamental.  There is no
way to tell if synopsis or description is up to date or correct.  With
signing metadata there is a way to detect this, so I don't see how
tracking signing metadata is worse than tracking the description.

> Of course we could have additional tools to make use of that info, say
> ‘guix build -S --authenticate’ or something.  But that would still be
> optional.

Don't make it optional!  Empower and encourage us downstream users to
verify upstream signatures and verify that the packager is honest, like
you empower us to build software and verify that the substitution
builder is honest.

You know, I recently packaged dkgpg.  That software is distributed as a
signed tarball (or rather, there is a detached signature for the
tarball).  Did I check that signature before storing the hashsum in the
packages metadata?  You know I did.  Currently, you just need to trust
me on that.  Or did I?

>> Thinking a bit more about having a hashsum as part of the packaging
>> definition, it seems to me that this is a bit of a modelling error.
>> Because there is no standardized way of authenticating a source
>> distribution, Guix defers this step to the packager.  And if there is no
>> way to authenticate an artifact (because upstream doesn't provide
>> signatures), we at least get TOFU, i.e. the assurance that any user
>> gets the same artifact as the packager.
>>
>> This doesn't seem terribly problematic, but it doesn't support parts of
>> the artifact changing, because in this model, this cannot be
>> distinguished from an attack.  Is there a way an artifact may change in
>> a valid way that Guix (and other distributions) may want to support?
>
> What do you mean?  If, for example, a tarball is modified in-place
> upstream, it’s an error from Guix’ viewpoint.

I mean that the tarball inside the OpenPGP message stays the same, but
we attach more signatures to the OpenPGP message over time.

>> We believe there is.  We propose to solve the problem of locating the
>> signatures by bundling them with the source distribution.  Instead of
>> using a detached PGP signature, we want to distribute the source as a
>> signed PGP message.
>>
>> Now, if you compute a hashsum over such an artifact, you are in effect
>> notarizing the signatures in the message and the message payload.  If
>> the developers add a signature to the message, the hashsum changes and
>> your notarization breaks.
>>
>> The preferred way to support this is to not verify that the hashsum over
>> the artifact matches, but to verify the PGP signatures over the payload
>> using the set of eligible signing keys in the package metadata.
>
> In practice, we don’t get to choose the authentication method.  You’re
> proposing a different authentication method here, but we’re just
> downstream: you’ll have to convince upstreams first.  :-)

Well, I'm a upstream, and this is your heads up that you are not
prepared to deal with our source distribution scheme of choice.  And,
we're hoping to convince other projects to use our scheme as well.

We've been talking about this problem internally, and we have concluded
that it is useful to be able to pin the source tarball to an exact
version as you do now.  That is possible with our scheme by hashing only
the literal data of the signed OpenPGP message.  That requires some
OpenPGP parsing, but guix/openpgp.scm should support that.

> Content-addressing is nice, but not very useful if each tool (IPFS, SWH,
> Git, Guix) has its own way to address content…

I don't understand the relation to content-addressing, sorry.


Thanks :)
Justus

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Securing the software distribution chain

[Ludovic Courtès]

Hi!

Justus Winter <teythoon@avior.uberspace.de> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> The idea of storing cryptographic metadata directly in <origin> has been
>> discussed a few times:
>>
>>   https://lists.gnu.org/archive/html/help-guix/2016-08/msg00132.html
>>   https://lists.gnu.org/archive/html/guix-devel/2015-10/msg00118.html
>>   https://git.savannah.gnu.org/cgit/guix.git/tree/TODO?id=6b8875c838d637773813899b35a9b5ea4acfd146#n29
>>
>> To me, the biggest shortcoming of this approach is if this metadata is
>> primarily “ornamental”: if in practice, ‘guix build -S’ doesn’t use it
>> (and it has no reason to use it), then that metadata is likely to become
>> stale without anyone noticing.
>
> On the other hand, most packaging metadata is ornamental.  There is no
> way to tell if synopsis or description is up to date or correct.  With
> signing metadata there is a way to detect this, so I don't see how
> tracking signing metadata is worse than tracking the description.

[...]

>> Of course we could have additional tools to make use of that info, say
>> ‘guix build -S --authenticate’ or something.  But that would still be
>> optional.
>
> Don't make it optional!  Empower and encourage us downstream users to
> verify upstream signatures and verify that the packager is honest, like
> you empower us to build software and verify that the substitution
> builder is honest.

Fundamentally, users trust packagers: they run their code.  Yet,
making it easier for users to audit the work of packagers sounds like a
good idea.  I agree with the philosophy.

I guess I poorly explained myself.  To put it differently, there is no
obvious place where signature verification fits here: the model is that
Guix code itself is authentic, and thus all that matters is ensuring
that we get the right content (checking the hash of origins).  Signature
metadata is “silent” in that model.

We can introduce signature verification in (guix download): every time
code is downloaded and signature metadata is available, we verify its
signature.  Unfortunately, I’m afraid this is likely to lead to lots of
false positives, and in particular failure to retrieve the OpenPGP key.

WDYT?  Where would you integrate that?

>>> Thinking a bit more about having a hashsum as part of the packaging
>>> definition, it seems to me that this is a bit of a modelling error.
>>> Because there is no standardized way of authenticating a source
>>> distribution, Guix defers this step to the packager.  And if there is no
>>> way to authenticate an artifact (because upstream doesn't provide
>>> signatures), we at least get TOFU, i.e. the assurance that any user
>>> gets the same artifact as the packager.
>>>
>>> This doesn't seem terribly problematic, but it doesn't support parts of
>>> the artifact changing, because in this model, this cannot be
>>> distinguished from an attack.  Is there a way an artifact may change in
>>> a valid way that Guix (and other distributions) may want to support?
>>
>> What do you mean?  If, for example, a tarball is modified in-place
>> upstream, it’s an error from Guix’ viewpoint.
>
> I mean that the tarball inside the OpenPGP message stays the same, but
> we attach more signatures to the OpenPGP message over time.

I see.  We’ve only encountered detached signatures so far.

>>> We believe there is.  We propose to solve the problem of locating the
>>> signatures by bundling them with the source distribution.  Instead of
>>> using a detached PGP signature, we want to distribute the source as a
>>> signed PGP message.
>>>
>>> Now, if you compute a hashsum over such an artifact, you are in effect
>>> notarizing the signatures in the message and the message payload.  If
>>> the developers add a signature to the message, the hashsum changes and
>>> your notarization breaks.
>>>
>>> The preferred way to support this is to not verify that the hashsum over
>>> the artifact matches, but to verify the PGP signatures over the payload
>>> using the set of eligible signing keys in the package metadata.
>>
>> In practice, we don’t get to choose the authentication method.  You’re
>> proposing a different authentication method here, but we’re just
>> downstream: you’ll have to convince upstreams first.  :-)
>
> Well, I'm a upstream, and this is your heads up that you are not
> prepared to deal with our source distribution scheme of choice.  And,
> we're hoping to convince other projects to use our scheme as well.
>
> We've been talking about this problem internally, and we have concluded
> that it is useful to be able to pin the source tarball to an exact
> version as you do now.  That is possible with our scheme by hashing only
> the literal data of the signed OpenPGP message.  That requires some
> OpenPGP parsing, but guix/openpgp.scm should support that.

(We = Sequoia-PGP, right?)

It should be possible to use to support this scheme by implementing a
new method for origins.  We would still require the origin hash to be
that of the extracted directory or tarball, which appropriate tooling
could simplify.

Thanks,
Ludo’.

Re: Securing the software distribution chain

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 914 bytes --]

On Mon, Aug 24, 2020 at 04:36:22PM +0200, Ludovic Courtès wrote:
> Hi!
> 
> Justus Winter <teythoon@avior.uberspace.de> skribis:
> 
> > Ludovic Courtès <ludo@gnu.org> writes:
> 
> [...]
> 
> We can introduce signature verification in (guix download): every time
> code is downloaded and signature metadata is available, we verify its
> signature.  Unfortunately, I’m afraid this is likely to lead to lots of
> false positives, and in particular failure to retrieve the OpenPGP key.
> 
> WDYT?  Where would you integrate that?
> 

Debian does sometimes add a public gpg key or the tarball signature
inside their debian folder. Not exactly sure how that would map for us
though.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]