[bug#43160] Validate the result of our linux-libre sources clean up

[Maxim Cournoyer <maxim.cournoyer@gmail.com>]

Hello Mark,

Sorry for the delayed reply.

Mark H Weaver <mhw@netris.org> writes:

> Hi Maxim,
>
> Thanks again for the patches that you've already pushed.
> They are a great improvement.

Thank you for the kind words.

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> I'm opposed to it because it would make it prohibitively difficult to
>>> push micro kernel updates (most of which contain potential security
>>> fixes) before Linux-libre has published their tarball release.
>>
>> Following recent discussions, I had understood that you agreed to wait
>> for the Linux-libre releases before bumping our own releases.
>
> Sorry, but that's incorrect.  I agreed to either wait or to look for new
> blobs myself.  I've already exercised that second option a couple of
> times since making that pledge.  I've found it to be quite easy.  The
> overwhelming majority of commits to the stable branches are bug fixes
> that obviously do not add anything resembling a blob.

Thanks for clarifying the option you chose.

> If you haven't already done so, I'd encourage you to look over the
> upstream commits between two consecutive upstream stable releases, to
> see more concretely what I'm talking about, and how straightforward it
> is to conclude that no new blobs could possibly have been added.

I just did for v5.4.8..v5.4.9 and v5.8.5..v5.8.6.  The human brain is
quite good at spotting patterns, so it wasn't too difficult indeed, but
the bug fix releases had a rather large number of commits (191 and 254,
respectively).  That's a serious commitment to make.

>> It seems the Linux-libre releases occur fast enough to not pose much
>> of a security issue; below is what I did to arrive to this conclusion.
>
> The timestamp data you collected is clearly off by several hours.  It
> appears to show that over two thirds of linux-libre releases come out
> *before* the corresponding linux release.  One third of them appear to
> have come out more than 4 hours before the corresponding linux release.
> That cannot be right.  I guess someone's clock is off by several hours,
> or somehow the timezones are not being taken into account.

The git commands I used are supposed to take the timezones into account
(converting the author dates into my local time), but as you mentioned
something seems off.

> I can tell you that within the last couple of weeks, since my pledge to
> either wait for linux-libre or to check for blobs myself, I very clearly
> remember upstream updates being tagged a few hours after midnight one
> morning, and that it was not until after dark the next evening before
> linux-libre had tagged their releases.  That instance is not reflected
> in the data you collected, which again suggests to me that there's a
> very significant systemic error in that data.
>
> Even with this apparently large bias in the data, it shows that
> linux-libre-5.7.7 was tagged almost 22 hours after the corresponding
> linux release.  Most likely that was actually at least 28 hours.
>
> Please don't misunderstand me: I do not fault the Linux-libre developers
> for not being quick enough.  On the contrary, they've been consistently
> *excellent* in that regard for as long as I've been paying attention.  I
> certainly don't fault them for occasionally spending some time away from
> their computers.
>
> What I _do_ fault them for is *insisting* on placing themselves in the
> middle of what should be a fast path for security updates from the
> upstream developers to us.

I agree that it is sub-optimal, but I don't see how that could be made
differently, with upstream Linux being at odds with our ideals.  We
can't have it both ways.

> To my mind, the practices that the Linux-libre developers are attempting
> to impose on us feel like Service as a Software Substitute (SaaSS), but
> in this case enforced by social pressure (and the suggestion that it
> might violate the GNU FSDG) instead of the usual technical restrictions.
>
> I'll also note that what you're proposing will apparently not be enough
> to satisfy Jason and Alexandre.  They've gone so far as to suggest that
> it's improper for an FSDG-compliant distribution to ever download
> non-FSDG-compliant source code to a user machine.  It seems that to
> satisfy them, Guix developers would apparently need to host our own
> distribution site for cleaned-up copies of source tarballs, and to use
> some separate tools to produce and upload new source tarballs to this
> site for every update of software whose upstream source is not
> FSDG-compliant.

I've reached the same conclusion, after engaging with them in the
#linux-libre channel.  To my understanding, we are not violating the
current GNU FSDG.  The spirit of the FSDG, as I understand it, is
basically "don't thwart users towards nonfree software".  I don't think
downloading a nonfree archive with the objective of turning it into free
software counts as such.  To the contrary, it can be seen as some form
of empowerment of how the process of freeing the software they use work,
enabling them to more directly improve such process.

I'd really like someone explain to me how the process of cleaning
nonfree software into free software can be considered a freedom issue.
So far the people I've asked either support their argument using their
personal interpretation of the FSDG or simply do not produce an answer.

[...]

>>> also make it prohibitively difficult to perform deblobbed bisections
>>> between two adjacent versions from the upstream stable git repository.
>>
>> In my opinion, we should not trade our correctness guarantee in exchange
>> for convenience,
>
> First, I think that it's a mistake to suggest that any "correctness
> guarantee" exists or could exist in Guix, with or without your proposed
> patch.  Massive amounts of new code are flowing into Guix from upstream
> projects on a daily basis, almost none of which is checked for
> FSDG-compliance ahead of time.  It is widely acknowledged, even in the
> FSDG document itself, that the most we can realistically hope to do is
> to pledge to fix FSDG-compliance problems in a timely fashion if they
> are brought to our attention.

I agree about the broader FSDG-compliance correctness guarantee.  I'm
sorry if I was unclear, but what I meant was in the more narrow sense of
correctness w.r.t. the official Linux-libre releases.

> Secondly, I think you're exaggerating the remaining risk.
>
> I acknowledge that before these discussions began, the risk of
> introducing new blobs was as high as 3% per Linux-libre update, which I
> agree was too high.  However, we've made several important changes since
> then.  Most importantly, I pledged to either wait for Linux-libre
> updates or to manually check for new blobs, and you introduced a
> 'deblob-check' pass in 'make-linux-libre-source'.  Leo also made changes
> to eliminate the risk of old deblob scripts being accidentally used.

Those are good improvements, yes.  My fear is that the amount of
complexity upgrading our Linux-libre package is starting to reach
unwieldy levels.  Consider a newcomer wanting to upgrade our Linux-libre
package.  They bump the sources.  Run 'make defconfig'.  Successfully
build a kernel.  Send a patch.  One of us reviews it, it looks OK, gets
merged.  It's easy to see what can silently go wrong.  I'd prefer
leaving the Linux-libre team handle that complexity rather than
duplicate it.

[...]

>>> In my opinion, at minimum, the 'linux-libre-upstream-source' argument to
>>> 'make-linux-libre-source' should optional.
>>
>> Perhaps, like for the change proposed by Leo, the edge case of bisecting
>> per-commit could be accommodated by reverting this patch when needed?
>
> That can be done easily for Leo's patch, but not for yours.
>
> In the case of Leo's patch, if I choose to manually check for new blobs,
> I can simply change one line in a 'deblob-script-X.Y' definition, like
> this:
>
>  (define-public linux-libre-5.8-version "5.8.7")
>  (define deblob-scripts-5.8
>    (linux-libre-deblob-scripts
> -   linux-libre-5.8-version
> +   "5.8.6"
>     (base32 "07z7sglyrfh0706icqqf3shadf638pvyid9386r661ds5lbsa2mw")
>     (base32 "0j6jba5fcddqlb42f95gjl78jisfla4nswqila074gglcrbnl9q7")))
>
> In contrast, your patch changes the 'make-linux-libre-source' procedure
> to *require* an existing 'linux-libre' tarball that precisely matches
> what it's going to produce.  In other words, it removes the ability to
> produce a new tarball, and has been reduced to merely being a verifier
> of pre-existing tarballs.

I think that's a good thing, at least for the usual use case.  But now
that you've mentioned that there *is* an occasional need to push
security updates faster than the Linux-libre official releases can
allow, I've modified the patch to make it optional, as you suggested.
I'd suggest that disabling that check be only allowed for seasoned
linux-libre maintainers such as you or Leo, and only when strictly
necessary (serious security threats affecting the Guix System and no
Linux-libre release yet).  Any other casual "bump" should fully make use
of the verification machinery.

> Reverting those changes would not only be extremely invasive for a
> simple micro kernel update, but would also, as a side effect, entail
> redeblobing and rebuilding *every* version of linux-libre in Guix, even
> if only one or two of the kernels needed updates.

I was only considering reverting this as a local hack in the context of
per-commit bisection.

> I guess from my perspective, I see a lot of disadvantages:
> disempowerment of users, occasional unnecessary delays on the order of
> tens of hours to deploy security updates, not to mention having to do
> another expensive git checkout and comparison on every kernel build, and
> for what?

This ensures we benefit from the experience of the Linux-libre
developers in screening the Linux sources for new nonfree threats.

> The main argument in favor, namely to reduce the risk of blobs being
> accidentally included in our kernel updates, seems to be adequately
> addressed by (1) my pledge to either wait or to check for blobs myself,
> and (2) the recently-added 'deblob-check' invocation in
> 'make-linux-libre-source'.  Do you think these are insufficient?

The trust chain is made simpler if we validate the Linux-libre result
(i.e. we need only trust the Linux-libre project).  As it is used by a
number of distros, we can hope for any mistake they might do to be
quickly flagged.  If we choose instead to generate our own Linux-libre
releases, doing our own "development", then people need to trust you or
Leo, or whoever bumped the linux-libre package for having done due
diligence (screening new commits for nonfree code).  With all due
respect, it seems easier to put our trust in the Linux-libre project for
this task, as this is their purpose and they have greater exposure.

> Thanks again for this discussion, and for the work you've already done
> to improve our deblobbing.

I'm equally thankful to you for your patience in handling all these
discussions, and for sticking around after all these years, keeping the
flow of kernel and icecat updates (amongst other) coming!

Please see the revised patch which I'll git send-email shortly.

Maxim