“Building a Secure Software Supply Chain with GNU Guix”

“Building a Secure Software Supply Chain with GNU Guix”

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]

Hello Guix!

I’m happy to announce the publication of a refereed paper in the
Programming journal:

  https://doi.org/10.22152/programming-journal.org/2023/7/1

It talks about the “secure update” mechanism used for channels and how
it fits together with functional deployment, reproducible builds, and
bootstrapping.  Comments from reviewers showed that explaining the whole
context was important to allow people not familiar with Guix or Nix to
understand why The Update Framework (TUF) isn’t a good match, why
Git{Hub,Lab} “verified” badges aren’t any good, and so on.

What’s presented there is not new if you’ve been following along, but
hopefully it puts things in perspective for outsiders.

I also think that one battle here is to insist on verifiability when a
lot of work about supply chain security goes into “attestation” (with
in-toto, sigstore, Google’s SLSA, and the likes.)

Enjoy!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 869 bytes --]

Re: “Building a Secure Software Supply Chain with GNU Guix”

[bokr]

On +2022-06-30 16:13:10 +0200, Ludovic Courtès wrote:
> Hello Guix!
> 
> I’m happy to announce the publication of a refereed paper in the
> Programming journal:
> 
>   https://doi.org/10.22152/programming-journal.org/2023/7/1
> 
> It talks about the “secure update” mechanism used for channels and how
> it fits together with functional deployment, reproducible builds, and
> bootstrapping.  Comments from reviewers showed that explaining the whole
> context was important to allow people not familiar with Guix or Nix to
> understand why The Update Framework (TUF) isn’t a good match, why
> Git{Hub,Lab} “verified” badges aren’t any good, and so on.
> 
> What’s presented there is not new if you’ve been following along, but
> hopefully it puts things in perspective for outsiders.
> 
> I also think that one battle here is to insist on verifiability when a
> lot of work about supply chain security goes into “attestation” (with
> in-toto, sigstore, Google’s SLSA, and the likes.)
> 
> Enjoy!
> 
> Ludo’.

Congratulations!

And thank you! I needed that assurance that guix really takes trust
seriously, and has a convincing internals story to back it up.

The "artifact" at [1] has a README.md [2] that's IMO definitely
also worth a read even if you aren't going to execute the image.

[1] <https://zenodo.org/record/6581453>
[2] <https://zenodo.org/record/6581453/files/README.md?download=1>

About this example (I like documentation that provides
things you can try):
--8<---------------cut here---------------start------------->8---
  5. Going back to our target revision, we can see that `gpg` can indeed
     verify signatures now: `git checkout
     20303c0b1c75bc4770cdfa8b8c6b33fd6e77c168 && git log --oneline
     --show-signature`.  `gpg` warns about expired keys but, as the
     paper notes, OpenPGP key expiration dates are ignored for our
     purposes (and the authentication code in Guix does *not* use
     `gpg`).
--8<---------------cut here---------------end--------------->8---

I think IWBN to have some kind of trust code come with that git output,
like gpg's 1-5 but indicating how well the committer/signer trusts
that using the code will *not* cause a problem.

I would like it if every commit had to have a code like that.

Even if it was "0," indicating that the committer judged
security to be irrelevant, I'd feel better knowing it was
part of committer workflow to be nudged into thinking
about the security aspect of the commit.

(Code alternative: an answer to the old real-opinon-extractor:
"How much money at what odds will you bet me this 
commit will not cause me problems?" ;-)

<ignorable class=rambling>
Hm, actually I think a 3-digit LTS code is required for reviewing:
with L indicating trust that the contribution is Legally ok,
and  T indicating trust in Technical competence of contributors
snd  S indicating trust in the Social aspect of the contribution crim/saint

OTTOMH encoding: digits 0-9: 0=NO Info, 1-9: subtract 5 =-> -4..+4,
with negatives meaning un-good opposites of positives. So code 191 would
be -4,+4,-4 for e.g.
  L-4: certain to have patent problems,
  T+4: contributed by a professional hacker,
  S-4: known criminal in supply chain.
</ignorable>

--
Regards,
Bengt Richter

Re: “Building a Secure Software Supply Chain with GNU Guix”

[zimoun]

Hi Bengt,

On jeu., 30 juin 2022 at 23:37, bokr@bokr.com wrote:

> I think IWBN to have some kind of trust code come with that git output,
> like gpg's 1-5 but indicating how well the committer/signer trusts
> that using the code will *not* cause a problem.

Well, from my understanding, Guix is dealing with 4 sort of code:

 1. Guix recipe of a package
 2. Guix service
 3. Guix itself
 4. Upstream 

I do not think committers are pushing code about #1, #2 or #3 that they
know beforehand it will cause a problem.

Therefore, I do not see how it could be implemented without being rooted
in committer feelings, opinion or self-confidence, i.e., highly variable
from one committer to the other.

The GPG trust level works because it is based on the web of trust.
Here, there is no web, IMHO.

Most of the security issues are from #4.  Considering how hard it is to
find and tackle the security issues, there is only two strategies, IMHO:
do not trust which implies deep audit of distributed source code and so
restrict the set of available packages (it is somehow an OpenBSD
approach); or accept more packages which means somehow trust upstream,
to some extent.


However, all in all, it asks what is expected by the reviewing process,
as discussed [1]. :-)

1: <https://yhetil.org/guix/87r13aifi3.fsf_-_@gnu.org>


Cheers,
simon

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Bengt Richter]

Hi Simon, and all,

On +2022-07-01 11:21:43 +0200, zimoun wrote:
> Hi Bengt,
> 
> On jeu., 30 juin 2022 at 23:37, bokr@bokr.com wrote:
> 
> > I think IWBN to have some kind of trust code come with that git output,
> > like gpg's 1-5 but indicating how well the committer/signer trusts
> > that using the code will *not* cause a problem.
> 
> Well, from my understanding, Guix is dealing with 4 sort of code:
> 
>  1. Guix recipe of a package
>  2. Guix service
>  3. Guix itself
>  4. Upstream 
> 
> I do not think committers are pushing code about #1, #2 or #3 that they
> know beforehand it will cause a problem.
>

Hm, -- unless <context-requirements-not-met> ... ? :)

> Therefore, I do not see how it could be implemented without being rooted
> in committer feelings, opinion or self-confidence, i.e., highly variable
> from one committer to the other.
> 
> The GPG trust level works because it is based on the web of trust.
> Here, there is no web, IMHO.
>

Well, guix developers who know each other well "in real life" have a pretty
good web, if not formal, no? :)
It's just not accessible to newcoming outsiders, who can't see the trust
codes in the insiders' heads :)

> Most of the security issues are from #4.  Considering how hard it is to
> find and tackle the security issues, there is only two strategies, IMHO:
> do not trust which implies deep audit of distributed source code and so
> restrict the set of available packages (it is somehow an OpenBSD
> approach); or accept more packages which means somehow trust upstream,
> to some extent.
>

Agreed, #4 is usually the source of security issues.

I'm just looking for some greppable coded hint of the difference between
a package that consists of e.g. a reverse polish calculator homework
assignemnt that a nerdy friend showed how to submit as a package, vs.
e.g. a package where the comments say over 10K subscribers have now been
running this hundreds of times daily for 2 months of beta testing with
no reported problems. Vs. This is alpha stuff, but seems harmless enough
if you run it in a container.

I'm not asking any guarantees, just a professional's quick judgement.
Like a chef's quick opinion on the cantaloupes at the open market. 

This is separate from the issue of whether to include a package under guix.
No blocker if the cantaloupe is not ripe, but helpful to have a sticker
saying so, for those who (for lack of time perhaps) want to order on line
and use grep instead of their nose :)

> 
> However, all in all, it asks what is expected by the reviewing process,
> as discussed [1]. :-)
> 
> 1: <https://yhetil.org/guix/87r13aifi3.fsf_-_@gnu.org>
> 
> 
> Cheers,
> simon

I am not forgetting that I should be thankful for anything I am provided
freely. So thank you all!
--
Regards,
Bengt Richter

Re: “Building a Secure Software Supply Chain with GNU Guix”

[zimoun]

Hi,

On Sun, 03 Jul 2022 at 12:38, Bengt Richter <bokr@bokr.com> wrote:
>> I do not think committers are pushing code about #1, #2 or #3 that they
>> know beforehand it will cause a problem.
>
> Hm, -- unless <context-requirements-not-met> ... ? :)
>

I do not understand what you mean?

>> The GPG trust level works because it is based on the web of trust.
>> Here, there is no web, IMHO.
>
> Well, guix developers who know each other well "in real life" have a pretty
> good web, if not formal, no? :)

Maybe I miss something.  IIUC, you are proposing to attach a level of
trust to each commit.

If this level for one commit is set by one committer, then the outcome
is poor because this level strongly depends on the committer.  Committer
A could say 0 and committer B would say 3 for the same commit, other
said the level depends on who do the job; therefore it is too dependent
on the committer mood to be useful, security-wise.  In this case, there
is no web of trust.

If this level for one commit is set by more than one committer, then it
is not affordable because it means we are doing double (or more) review
when the project is trying to just deal with merging all the
submissions.  In this case, there is a web of trust.  But it is not
doable considering the rate of commits.


> I'm just looking for some greppable coded hint of the difference between
> a package that consists of e.g. a reverse polish calculator homework
> assignemnt that a nerdy friend showed how to submit as a package, vs.
> e.g. a package where the comments say over 10K subscribers have now been
> running this hundreds of times daily for 2 months of beta testing with
> no reported problems. Vs. This is alpha stuff, but seems harmless enough
> if you run it in a container.

Run OpenBSD. ;-)


> I'm not asking any guarantees, just a professional's quick judgement.
> Like a chef's quick opinion on the cantaloupes at the open market. 

Why this professional's quick judgment should come from the package
manager (packager, reviewer, committer) and not from a community around
the specific software whatever how it is distributed?


Cheers,
simon

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Bengt Richter]

Hi zimoun,

On +2022-07-04 10:21:13 +0200, zimoun wrote:
> Hi,
> 
> On Sun, 03 Jul 2022 at 12:38, Bengt Richter <bokr@bokr.com> wrote:
> >> I do not think committers are pushing code about #1, #2 or #3 that they
> >> know beforehand it will cause a problem.
> >
> > Hm, -- unless <context-requirements-not-met> ... ? :)
> >
> 
> I do not understand what you mean?
>
I just meant I think I have seen partial fixes committed, noting that it will
only work given some required context, like run time must be x86_64 (or else
"<context-requirements-not-met>" and problem can be expected :)

Sorry for the noise. NNTR  :/
--
Regards,
Bengt Richter

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Ludovic Courtès]

Hi,

bokr@bokr.com skribis:

> I think IWBN to have some kind of trust code come with that git output,
> like gpg's 1-5 but indicating how well the committer/signer trusts
> that using the code will *not* cause a problem.
>
> I would like it if every commit had to have a code like that.

I very much agree with what zimoun wrote: it’s very hard to assess the
security implications of a Guix commit (especially a commit that adds,
say, a 100K lines-of-code package), and we shouldn’t ask too much of
packagers.

Ludo’.

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 2203 bytes --]


Good article!

There's still some questions to ask. I'm concerned about the safety of
the evaluation of channel code. IIRC, there's no sandbox for the
evaluation of package in channel. So, it's possible to inject some
side-effect code into a channel like

```
(define-module (my channel code))

(display "I'm planning to do evil things here!")

(define-public some-package ...)
(define-public another-package ...)
```

We have PGP sign and git commit chain to make sure the commits are
committed by trusted people. But it's still possible for the channel
owner to inject malicious code into the channel in a future commit. Like
what Marak Squires did in faker.js project :( or the committer of Guix
was attacked by an evil maid.

In Nix flakes, there's pure evaluation to make sure no side-effectful
code is allowed. But Guix channel is less restricted than a Nix flake.
It's a important problem to make sure the evaluation is safe for the user.


Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> [[PGP Signed Part:Undecided]]
> Hello Guix!
>
> I’m happy to announce the publication of a refereed paper in the
> Programming journal:
>
>   https://doi.org/10.22152/programming-journal.org/2023/7/1
>
> It talks about the “secure update” mechanism used for channels and how
> it fits together with functional deployment, reproducible builds, and
> bootstrapping.  Comments from reviewers showed that explaining the whole
> context was important to allow people not familiar with Guix or Nix to
> understand why The Update Framework (TUF) isn’t a good match, why
> Git{Hub,Lab} “verified” badges aren’t any good, and so on.
>
> What’s presented there is not new if you’ve been following along, but
> hopefully it puts things in perspective for outsiders.
>
> I also think that one battle here is to insist on verifiability when a
> lot of work about supply chain security goes into “attestation” (with
> in-toto, sigstore, Google’s SLSA, and the likes.)
>
> Enjoy!
>
> Ludo’.
>
> [[End of PGP Signed Part]]


-- 
Retrieve my PGP public key:

  gpg --recv-keys 481F5EEEBA425ADC13247C76A6E672D981B8E744

Zihao

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Ludovic Courtès]

Hi,

Zhu Zihao <all_but_last@163.com> skribis:

> There's still some questions to ask. I'm concerned about the safety of
> the evaluation of channel code. IIRC, there's no sandbox for the
> evaluation of package in channel. So, it's possible to inject some
> side-effect code into a channel like
>
> ```
> (define-module (my channel code))
>
> (display "I'm planning to do evil things here!")
>
> (define-public some-package ...)
> (define-public another-package ...)
> ```

Yes.

> We have PGP sign and git commit chain to make sure the commits are
> committed by trusted people. But it's still possible for the channel
> owner to inject malicious code into the channel in a future commit. Like
> what Marak Squires did in faker.js project :( or the committer of Guix
> was attacked by an evil maid.

I’m not aware of the faker.js story, do you have a link?

The model here is that users trust authorized committers.  When you
think about it, there’s no way around it, because at the end of the day,
you’re installing software that an authorized committer added to the
channel.

To put it differently, side effects in the .scm file as you show above
are just one of the many ways an authorized committer could harm users.

> In Nix flakes, there's pure evaluation to make sure no side-effectful
> code is allowed. But Guix channel is less restricted than a Nix flake.
> It's a important problem to make sure the evaluation is safe for the user.

Yes, I understand.  I don’t think that makes a practical difference
though: when you pull from a Guix channel or fetch a Nix flake, that’s
because you want to install software according to what that
channel/flake provides.  So whether evil code is in the channel/flake
(as Scheme/Nix code) or in the package(s) themselves makes little
difference.

Does that make sense?

(Besides, there’s no mechanism for authenticated updates of flakes or of
Nixpkgs, which is the core of the paper.)

Thanks for your feedback!

Ludo’.

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 1746 bytes --]


Ludovic Courtès <ludovic.courtes@inria.fr> writes:

>> We have PGP sign and git commit chain to make sure the commits are
>> committed by trusted people. But it's still possible for the channel
>> owner to inject malicious code into the channel in a future commit. Like
>> what Marak Squires did in faker.js project :( or the committer of Guix
>> was attacked by an evil maid.
>
> I’m not aware of the faker.js story, do you have a link?

https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/

Here's a detailed report about Marak and faker.js.

>> In Nix flakes, there's pure evaluation to make sure no side-effectful
>> code is allowed. But Guix channel is less restricted than a Nix flake.
>> It's a important problem to make sure the evaluation is safe for the user.
>
> Yes, I understand.  I don’t think that makes a practical difference
> though: when you pull from a Guix channel or fetch a Nix flake, that’s
> because you want to install software according to what that
> channel/flake provides.  So whether evil code is in the channel/flake
> (as Scheme/Nix code) or in the package(s) themselves makes little
> difference.
>
> Does that make sense?

My two cents: When depolying a manifest, we use `guix package -p
<path-to-profile> -m <path-to-manifest>`, This command consists two
parts. Guix will first evaluate the packages specified in the manifest,
and build the profile. And then populate the profile to given
destination. The first part can be done in a sandboxed environment, or a
non-privileged account like "nobody".

-- 
Retrieve my PGP public key:

  gpg --recv-keys 481F5EEEBA425ADC13247C76A6E672D981B8E744

Zihao

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Ludovic Courtès]

Hi,

Zhu Zihao <all_but_last@163.com> skribis:

> https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/
>
> Here's a detailed report about Marak and faker.js.

Interesting.  But yeah, a Guix committer could change Guix anytime to
print “LIBERTY” (that’s very much the spirit of the project ;-)) or they
could, simply, unwillingly introduce bugs.  No technical mechanism can
prevent that.

>>> In Nix flakes, there's pure evaluation to make sure no side-effectful
>>> code is allowed. But Guix channel is less restricted than a Nix flake.
>>> It's a important problem to make sure the evaluation is safe for the user.
>>
>> Yes, I understand.  I don’t think that makes a practical difference
>> though: when you pull from a Guix channel or fetch a Nix flake, that’s
>> because you want to install software according to what that
>> channel/flake provides.  So whether evil code is in the channel/flake
>> (as Scheme/Nix code) or in the package(s) themselves makes little
>> difference.
>>
>> Does that make sense?
>
> My two cents: When depolying a manifest, we use `guix package -p
> <path-to-profile> -m <path-to-manifest>`, This command consists two
> parts. Guix will first evaluate the packages specified in the manifest,
> and build the profile. And then populate the profile to given
> destination. The first part can be done in a sandboxed environment, or a
> non-privileged account like "nobody".

Sure, though at a technical level is trickier than this, and again, it
doesn’t change the fact that you’ll end up running code provided by the
very same developers.

Thanks,
Ludo’.

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

>> My two cents: When depolying a manifest, we use `guix package -p
>> <path-to-profile> -m <path-to-manifest>`, This command consists two
>> parts. Guix will first evaluate the packages specified in the manifest,
>> and build the profile. And then populate the profile to given
>> destination. The first part can be done in a sandboxed environment, or a
>> non-privileged account like "nobody".
>
> Sure, though at a technical level is trickier than this, and again, it
> doesn’t change the fact that you’ll end up running code provided by the
> very same developers.

It may still be worthwhile adding a few restrictions.  One scenario that
caught me by suprise: deleting files in a snippet.  I forgot quoting the
snippet and ended up running the deletion code on the host side – in my
working directory.  Nothing bad happened and we’ve never committed any
mistake like that to the repository, but it is conceivable that
something like this gets past the review and ends up deleting files.

A safety net would prevent a small mistake like a missing quote from
having potentially large side effects.

Of course, all this is orthogonal to securing the supply chain, which is
what the paper is about.

-- 
Ricardo

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Maxime Devos]

Ludovic Courtès schreef op ma 18-07-2022 om 10:45 [+0200]:
> The model here is that users trust authorized committers.  When you
> think about it, there’s no way around it, because at the end of the
> day, you’re installing software that an authorized committer added to
> the channel.

FWIW, something I haven't seen mentioned yet is that the trust problem
could be reduced by some kind of multisig system, where multiple
independent persons would need to sign the commit for it to be
accepted, though that might be technically hard to implement and
probably be too people-time-expensive currently.

Greetings,
Maxime.

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Arun Isaac]

Hi Ludo,

>   https://doi.org/10.22152/programming-journal.org/2023/7/1

This is an excellent read! Are there plans to release this git
authentication system as a separate tool so that other non-Guix projects
may use it easily?

Thanks,
Arun

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Ludovic Courtès]

Hi!

Arun Isaac <arunisaac@systemreboot.net> skribis:

> This is an excellent read! Are there plans to release this git
> authentication system as a separate tool so that other non-Guix projects
> may use it easily?

Not really.  ‘guix git authenticate’ is already usable outside and the
modules behind it are well isolated from the rest of Guix though.

  https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-git-authenticate.html

I don’t plan personally to make it a separately-maintained tool, but it
could be interesting.  More generally, getting in touch with the Git
developers and also with other projects with similar concerns would be
great.

Ludo’.

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Arun Isaac]

> I don’t plan personally to make it a separately-maintained tool, but it
> could be interesting.  More generally, getting in touch with the Git
> developers and also with other projects with similar concerns would be
> great.

That's good to know, thanks! Yup, git integration would be
great. Meanwhile, interested folks can already use `guix git
authenticate' like Maxime is doing for Scheme-GNUnet.

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Maxime Devos]

Arun Isaac schreef op di 19-07-2022 om 12:51 [+0530]:
> 
> Hi Ludo,
> 
> >    https://doi.org/10.22152/programming-journal.org/2023/7/1
> 
> This is an excellent read! Are there plans to release this git
> authentication system as a separate tool so that other non-Guix
> projects may use it easily?

FWIW I have been using "guix git authenticate" + .guix-authorizations
in the Git repo of Scheme-GNUnet for letting users verify whether they
got an 'authentic' copy.

Greetings,
Maxime.

Re: “Building a Secure Software Supply Chain with GNU Guix”

[Jeremiah]

This is why things like SELinux exist, combine with separate binaries
for the functionality that impacts things outside of the store to
quickly minimize possible damage. If the binary can only create links
the possible damage is quite limited.

But the much more dangerous modification is much more subtle and can go
months to years without being noticed. To which there is no defense. As
there is no way to know when a person you trust will go crazy, turn evil
or flip the switch they planned many years ago.

Heck, the possible exploits that could be in the bootstrap seeds could
so subtle you wouldn't notice or even hidden in the kernel itself:
https://gitlab.com/bauen1/stage0-backdoor.git

Making reviews by third parties cheap, make forking cheap and never
assuming that anyone should be completely trusted is usually a secure
place to start.

And why the bootstrap seeds README starts with:
NEVER TRUST ANYTHING IN HERE

I could be evil after all

-Jeremiah