From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id oL/oDFCJVl9gQgAA0tVLHw (envelope-from ) for ; Mon, 07 Sep 2020 19:26:08 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id MC3WCFCJVl+5OQAA1q6Kng (envelope-from ) for ; Mon, 07 Sep 2020 19:26:08 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B20339402CB for ; Mon, 7 Sep 2020 19:26:07 +0000 (UTC) Received: from localhost ([::1]:39788 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kFMmE-0005dx-DU for larch@yhetil.org; Mon, 07 Sep 2020 15:26:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54488) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kFMmA-0005dc-Ee for guix-patches@gnu.org; Mon, 07 Sep 2020 15:26:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:39361) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kFMmA-0003rx-5H for guix-patches@gnu.org; Mon, 07 Sep 2020 15:26:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kFMmA-0005Z8-2S for guix-patches@gnu.org; Mon, 07 Sep 2020 15:26:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#43160] Validate the result of our linux-libre sources clean up Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 07 Sep 2020 19:26:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43160 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Mark H Weaver Cc: 43160@debbugs.gnu.org, Leo Famulari Received: via spool by 43160-submit@debbugs.gnu.org id=B43160.159950675021375 (code B ref 43160); Mon, 07 Sep 2020 19:26:02 +0000 Received: (at 43160) by debbugs.gnu.org; 7 Sep 2020 19:25:50 +0000 Received: from localhost ([127.0.0.1]:50907 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kFMly-0005Yg-95 for submit@debbugs.gnu.org; Mon, 07 Sep 2020 15:25:50 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:36482) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kFMlw-0005YR-Am for 43160@debbugs.gnu.org; Mon, 07 Sep 2020 15:25:48 -0400 Received: by mail-qt1-f194.google.com with SMTP id n10so10412656qtv.3 for <43160@debbugs.gnu.org>; Mon, 07 Sep 2020 12:25:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=zicvDX1sO3XY2L29b0qEzOxywwFSMwCYBDh9COOjsmA=; b=gZkUH/+Xd0i50C5ji75vKrfieIEU38GK0uAlA6mPgastMpcDFWQ28ZQ9NpZYwZmktJ eFeI9gpu24fgwm8ZPqnbPnLJZ0M/4QrIs9Nu6YAZ6mVMKfrZx0YbeuNQGA1pnWH2pclh oPFJp+F6zTzsQKZSDfC1NMbkHiP6EElXrIFpvKjtkLsMajg8AsyTRsI2BSiPf0/f7sOR ciJhw2DsBQahJjscuKc+5YgV88PlFMOZUlhJvWZuDMmSGJzP95Pp2wHaAhCc6ODSQpzI 8DsUO1kmKzkNqJwS0bll3kg2Ul5I83fuL98iYk+0OXvHTJ18NU4ankHIBxkmMVeoEE9j at2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=zicvDX1sO3XY2L29b0qEzOxywwFSMwCYBDh9COOjsmA=; b=lQQl5MXR33YDm2BW10UlvCRUS43YSN2af/v4zLEzbBSssrtZBAwze5B+kDgA2oX+Cx 6eom/OOLpkQwLgxalA6GscsqgM0LLmEgCQKnc05o7gxWmEpXrh5K3+QOsCtxFlWVout0 EsnSVfk26CRDSjlPmWe2PuVRjUju/MRMRW/wqafE/2OeBXB732Qt3wgIu5Nqh3lTh6Jm pehDf5GZS1Bqy3J0G9wLo2PMY1C8ioqxG+qJca0RzQIoqpwf7CubGx9zIdqBdRuRYMVN ovDvAg5Ay005hToLRLFSvGX3IOcFIfS/yckY9PGwHkNAqfyXO2bW096+lRQAQ47FEFKi FXiw== X-Gm-Message-State: AOAM533iF7VCCvNRdYnObS6AMimDNyYuEGfhLuf6eS/ummQvcG3xIe9q vqSsru2uxL9tZ9m9jdOMZiM= X-Google-Smtp-Source: ABdhPJxWD3qKjSvjbQ/irlMjfRBWsv9uCdcbDlvKYNvTexf5WeoGp8srnmUK4bTmZegQlk7VyTR5DQ== X-Received: by 2002:ac8:1c82:: with SMTP id f2mr16725959qtl.305.1599506742286; Mon, 07 Sep 2020 12:25:42 -0700 (PDT) Received: from hurd (dsl-205-233-125-88.b2b2c.ca. [205.233.125.88]) by smtp.gmail.com with ESMTPSA id e1sm12398876qtb.0.2020.09.07.12.25.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2020 12:25:41 -0700 (PDT) From: Maxim Cournoyer References: <20200902182922.GA26301@jasmine.lan> <87363z28fs.fsf@netris.org> <20200902221552.GA32317@jasmine.lan> <87zh67zqfa.fsf@netris.org> <87h7sedz0w.fsf_-_@gmail.com> <874kodsh21.fsf@netris.org> Date: Mon, 07 Sep 2020 15:25:54 -0400 In-Reply-To: <874kodsh21.fsf@netris.org> (Mark H. Weaver's message of "Fri, 04 Sep 2020 11:21:47 -0400") Message-ID: <87imcpbd8d.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.0 (-) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=fail (rsa verify failed) header.d=gmail.com header.s=20161025 header.b=gZkUH/+X; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Spam-Score: 0.09 X-TUID: 6Uq80NqvjO7/ Hi Mark! Mark H Weaver writes: > Hi Maxim, > > Maxim Cournoyer writes: >> I'd like to point you to the following patches, as they touch the >> generation of the linux-libre sources, in case they hadn't caught your >> attention: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43160. > > Thanks very much for bringing this to my attention. I do not subscribe > to the guix-patches list, so I would not have seen this otherwise. > > I'm in favor of the following patches: > > gnu: linux-libre: Use Python 3 in make-linux-libre-source. > gnu: make-linux-libre-source: Set output port buffering to line mode. > gnu: linux-libre: Validate that the cleaned up tarball is free of blobs. > > Thanks for these. Please push them whenever you feel is appropriate. Thanks for taking a look! I've now done so. > On other other hand, I'm strongly opposed to the following patch: > > gnu: linux-libre: Compare generated sources against Linux-libre releases. > > I'm opposed to it because it would make it prohibitively difficult to > push micro kernel updates (most of which contain potential security > fixes) before Linux-libre has published their tarball release. Following recent discussions, I had understood that you agreed to wait for the Linux-libre releases before bumping our own releases. It seems the Linux-libre releases occur fast enough to not pose much of a security issue; below is what I did to arrive to this conclusion. For Linux stable, the author dates of the last releases of the version 5 series, omitting release candidates: --8<---------------cut here---------------start------------->8--- $ git tag | grep -E '5\.[0-9]+\.' | grep -v -- '-rc' \ | sort -t '.' -k1,1n -k2,2n -k3,3n | tail -n20 \ | xargs -i{} git log --date='format:%c' --pretty='%ad %d' {} -1 Wed 24 Jun 2020 05:49:26 PM GMT (tag: v5.7.6) Tue 30 Jun 2020 04:21:22 PM GMT (tag: v5.7.7) Thu 09 Jul 2020 09:39:40 AM GMT (tag: v5.7.8) Thu 16 Jul 2020 08:13:36 AM GMT (tag: v5.7.9) Wed 22 Jul 2020 09:34:29 AM GMT (tag: v5.7.10) Wed 29 Jul 2020 10:20:01 AM GMT (tag: v5.7.11) Fri 31 Jul 2020 06:47:17 PM GMT (tag: v5.7.12) Wed 05 Aug 2020 09:58:51 AM GMT (tag: v5.7.13) Fri 07 Aug 2020 09:33:11 AM GMT (tag: v5.7.14) Tue 11 Aug 2020 03:35:42 PM GMT (tag: v5.7.15) Wed 19 Aug 2020 08:24:20 AM GMT (tag: v5.7.16) Fri 21 Aug 2020 01:07:46 PM GMT (tag: v5.7.17) Wed 26 Aug 2020 11:42:25 AM GMT (tag: v5.7.18) Thu 27 Aug 2020 09:30:50 AM GMT (tag: v5.7.19, origin/linux-5.7.y) Tue 11 Aug 2020 03:48:12 PM GMT (tag: v5.8.1) Wed 19 Aug 2020 08:27:10 AM GMT (tag: v5.8.2) Fri 21 Aug 2020 01:15:22 PM GMT (tag: v5.8.3) Wed 26 Aug 2020 11:49:20 AM GMT (tag: v5.8.4) Thu 27 Aug 2020 09:31:49 AM GMT (tag: v5.8.5) Thu 03 Sep 2020 11:29:52 AM GMT (tag: v5.8.6, origin/linux-5.8.y) --8<---------------cut here---------------end--------------->8--- Similarly, for Linux-libre: --8<---------------cut here---------------start------------->8--- git tag | grep -E 'sources/v5\.[0-9]+\.' | grep -v -- '-rc' \ | sort -t '.' -k1,1n -k2,2n -k3,3n | tail -n20 \ | xargs -i{} git log --date='format:%c' --pretty='%ad %d' {} -1 Wed 24 Jun 2020 02:51:34 PM GMT (tag: sources/v5.7.6-gnu) Wed 01 Jul 2020 02:01:47 PM GMT (tag: sources/v5.7.7-gnu) Thu 09 Jul 2020 08:59:49 AM GMT (tag: sources/v5.7.8-gnu) Thu 16 Jul 2020 11:51:43 AM GMT (tag: sources/v5.7.9-gnu) Wed 22 Jul 2020 06:40:22 AM GMT (tag: sources/v5.7.10-gnu) Wed 29 Jul 2020 06:33:25 AM GMT (tag: sources/v5.7.11-gnu) Fri 31 Jul 2020 02:22:04 PM GMT (tag: sources/v5.7.12-gnu) Wed 05 Aug 2020 05:44:37 AM GMT (tag: sources/v5.7.13-gnu) Fri 07 Aug 2020 04:46:28 AM GMT (tag: sources/v5.7.14-gnu) Tue 11 Aug 2020 02:48:28 PM GMT (tag: sources/v5.7.15-gnu) Wed 19 Aug 2020 02:14:46 PM GMT (tag: sources/v5.7.16-gnu) Fri 21 Aug 2020 09:37:45 AM GMT (tag: sources/v5.7.17-gnu) Wed 26 Aug 2020 07:27:54 AM GMT (tag: sources/v5.7.18-gnu) Thu 27 Aug 2020 01:14:21 PM GMT (tag: sources/v5.7.19-gnu) Tue 11 Aug 2020 02:47:58 PM GMT (tag: sources/v5.8.1-gnu) Wed 19 Aug 2020 02:15:42 PM GMT (tag: sources/v5.8.2-gnu) Fri 21 Aug 2020 09:37:45 AM GMT (tag: sources/v5.8.3-gnu) Wed 26 Aug 2020 07:27:54 AM GMT (tag: sources/v5.8.4-gnu) Thu 27 Aug 2020 01:14:21 PM GMT (tag: sources/v5.8.5-gnu) Thu 03 Sep 2020 07:14:30 AM GMT (tag: sources/v5.8.6-gnu) --8<---------------cut here---------------end--------------->8--- While the author dates of the commits don't appear to be very precise (some Linux-libre commits would have occurred before their Linux counterpart), we can at least see that each Linux release was met with a Linux-libre on the same day for all except the 5.7.7 release. Also, if we compare with our own Linux-libre update timings: --8<---------------cut here---------------start------------->8--- git log --grep 'gnu: linux-libre: Update to 5' --date='format:%c' \ --pretty='%ad %s' | head -n20 | sort -r -t '.' -k1,1n -k2,2n -k3,3n Thu 11 Jun 2020 04:15:35 PM GMT gnu: linux-libre: Update to 5.4.46. Thu 18 Jun 2020 12:39:23 AM GMT gnu: linux-libre: Update to 5.4.47 Mon 22 Jun 2020 09:02:33 PM GMT gnu: linux-libre: Update to 5.4.48. Wed 24 Jun 2020 09:08:00 PM GMT gnu: linux-libre: Update to 5.4.49. Wed 01 Jul 2020 01:31:06 PM GMT gnu: linux-libre: Update to 5.4.50. Thu 09 Jul 2020 04:40:27 PM GMT gnu: linux-libre: Update to 5.4.51. Thu 16 Jul 2020 03:37:05 PM GMT gnu: linux-libre: Update to 5.4.52. Thu 23 Jul 2020 12:28:46 AM GMT gnu: linux-libre: Update to 5.4.53. Wed 29 Jul 2020 05:14:00 PM GMT gnu: linux-libre: Update to 5.4.54. Sat 01 Aug 2020 12:07:08 AM GMT gnu: linux-libre: Update to 5.4.55. Wed 05 Aug 2020 03:21:53 PM GMT gnu: linux-libre: Update to 5.4.56. Sat 01 Aug 2020 12:39:30 PM GMT gnu: linux-libre: Update to 5.7.12. Fri 07 Aug 2020 09:37:11 PM GMT gnu: linux-libre: Update to 5.7.14. Tue 11 Aug 2020 05:34:48 PM GMT gnu: linux-libre: Update to 5.7.15. Wed 19 Aug 2020 07:35:03 PM GMT gnu: linux-libre: Update to 5.7.16. Thu 20 Aug 2020 04:03:46 PM GMT gnu: linux-libre: Update to 5.8.2. Fri 21 Aug 2020 09:01:17 PM GMT gnu: linux-libre: Update to 5.8.3. Wed 26 Aug 2020 04:01:11 PM GMT gnu: linux-libre: Update to 5.8.4. Thu 27 Aug 2020 04:13:32 PM GMT gnu: linux-libre: Update to 5.8.5. Thu 03 Sep 2020 01:56:31 PM GMT gnu: linux-libre: Update to 5.8.6. --8<---------------cut here---------------end--------------->8--- For the subset that we did package, we were always trailing the Linux-libre releases, so the argument that waiting for their releases would hamper our security doesn't seem to hold. > also make it prohibitively difficult to perform deblobbed bisections > between two adjacent versions from the upstream stable git repository. In my opinion, we should not trade our correctness guarantee in exchange for convenience, especially if the convenience is only gained in such a corner case as per-commit bisection of the Linux kernel. It'd be oversimplifying to say that the Linux-libre developers just run their scripts to produce a release; they also manually screen the new upstream changes and update their scripts accordingly. To give due credit to their efforts, we should not simply run their scripts with a newer version/commit of Linux and expect arriving at a correct result. > In my opinion, at minimum, the 'linux-libre-upstream-source' argument to > 'make-linux-libre-source' should optional. Perhaps, like for the change proposed by Leo, the edge case of bisecting per-commit could be accommodated by reverting this patch when needed? It seems more important that the common case be rigorously verified. Also note that it should be possible to: 1) Test each packaged release in Guix to "bisect" (duh) 2) Test any Linux stable release via the Linux-libre git repo, building with a command such as "guix build --with-git-url=linux-libre=git://linux-libre.fsfla.org/releases.git --with-commit=linux-libre=sources/v5.8.3-gnu linux-libre". Unfortunately this can't be done from the command line using 'guix system build ...' but it should be easy to define your own linux-libre package using the 'make-linux-libre*' procedure (which will gladly accept any linux-libre source). For when the per-commit granularity is not required. In the future, the linux-libre git repo will apply their clean ups per commit, allowing to do like 2) above for any commit. > I find it depressing that Jason's and Alexandre's attempts to browbeat > us to limit ourselves to deblob only the precise tarballs that they > produce, and to always wait for them to produce them before pushing > security fixes (although it takes less than 10 minutes to look over the > upstream commits for new blobs) have gained traction here. Despite the somewhat corrosive tone of the exchange, some valid points were made. I've scavenged these and adapted the recipe. I think the end result is a win-win situation for both Linux-libre and Guix. As shown above, there hasn't been a case where the Linux-libre effort slowed down the deployment of a new Linux kernel version in Guix. I don't foresee this changing. What do you think? Are there holes in my analysis/understanding? Thank you, Maxim