[bug#43160] [PATCH] gnu: linux-libre: Use Python 3 in make-linux-libre-source.

[bug#43160] [PATCH] gnu: linux-libre: Use Python 3 in make-linux-libre-source.

[Maxim Cournoyer]

Successfully tested with all of the linux-libre versions we carry in Guix:
4.4.234, 4.9.234, 4.14.195, 4.19.142, 5.4.61 and 5.8.5.

* gnu/packages/linux.scm (make-linux-libre-source): Replace python-2 by
python-wrapper.
---
 gnu/packages/linux.scm | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index d3365e7a4b..e9bfca25af 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -296,11 +296,7 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                          #+(canonical-package bzip2)
                          #+(canonical-package gzip)
                          #+(canonical-package tar)
-                         ;; The comments in the 'deblob-check' script
-                         ;; claim that it supports Python 2 and 3, but
-                         ;; in fact it fails when run in Python 3 as
-                         ;; of version 5.1.3.
-                         #+python-2))
+                         #+python-wrapper))
 
                   (with-directory-excursion "/tmp/bin"
 
-- 
2.27.0

[bug#43160] [PATCH 1/2] gnu: make-linux-libre-source: Set output port buffering to line mode.

[Maxim Cournoyer]

* gnu/packages/linux.scm (make-linux-libre-source): Set output port buffering
to line mode via setvbuf.  Remove the ad-hoc calls to force-output.
---
 gnu/packages/linux.scm | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index e9bfca25af..9507178fb6 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -279,6 +279,9 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                              (srfi srfi-1)
                              (ice-9 match)
                              (ice-9 ftw))
+
+                (setvbuf (current-output-port) 'line)
+
                 (let ((dir (string-append "linux-" #$version)))
 
                   (mkdir "/tmp/bin")
@@ -315,12 +318,10 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                   (if (file-is-directory? #+upstream-source)
                       (begin
                         (format #t "Copying upstream linux source...~%")
-                        (force-output)
                         (invoke "cp" "--archive" #+upstream-source dir)
                         (invoke "chmod" "--recursive" "u+w" dir))
                       (begin
                         (format #t "Unpacking upstream linux tarball...~%")
-                        (force-output)
                         (invoke "tar" "xf" #$upstream-source)
                         (match (scandir "."
                                         (lambda (name)
@@ -335,11 +336,9 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                   (with-directory-excursion dir
                     (setenv "PYTHON" (which "python"))
                     (format #t "Running deblob script...~%")
-                    (force-output)
                     (invoke "/tmp/bin/deblob"))
 
                   (format #t "~%Packing new Linux-libre tarball...~%")
-                  (force-output)
                   (invoke "tar" "cvfa" #$output
                           ;; Avoid non-determinism in the archive.
                           "--mtime=@0"
-- 
2.27.0

[bug#43160] [PATCH 2/2] gnu: linux-libre: Validate that the cleaned up tarball is free of blobs.

[Maxim Cournoyer]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=yes, Size: 1505 bytes --]

* gnu/packages/linux.scm (make-linux-libre-source): Call the deblob-check
script on the generated tarball archive with the --use-awk and --list-blobs
options.
---
 gnu/packages/linux.scm | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 9507178fb6..cd9c3a18fa 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -48,6 +48,7 @@
 ;;; Copyright © 2020 John Soo <jsoo1@asu.edu>
 ;;; Copyright © 2020 Michael Rohleder <mike@rohleder.de>
 ;;; Copyright © 2020 Anders Thuné <asse.97@gmail.com>
+;;; Copyright © 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -90,6 +91,7 @@
   #:use-module (gnu packages flex)
   #:use-module (gnu packages file)
   #:use-module (gnu packages freedesktop)
+  #:use-module (gnu packages gawk)
   #:use-module (gnu packages gcc)
   #:use-module (gnu packages gettext)
   #:use-module (gnu packages glib)
@@ -346,7 +348,11 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                           "--group=root:0"
                           "--sort=name"
                           "--hard-dereference"
-                          dir))))))))))
+                          dir)
+
+                  (format #t "~%Validating the generated tarball...~%")
+                  (invoke "/tmp/bin/deblob-check" "--use-awk" "--list-blobs"
+                          #$output))))))))))
 
 \f
 ;;;
-- 
2.27.0

[bug#43160] [PATCH v2 1/4] gnu: linux-libre: Use Python 3 in make-linux-libre-source.

[Maxim Cournoyer]

Successfully tested with all of the linux-libre versions we carry in Guix:
4.4.234, 4.9.234, 4.14.195, 4.19.142, 5.4.61 and 5.8.5.

* gnu/packages/linux.scm (make-linux-libre-source): Replace python-2 by
python-wrapper.  Do not set the PYTHON environment variable, which is not
required when using python-wrapper.
---
 gnu/packages/linux.scm | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index d3b3f4de9c..8edbe4e7e4 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -296,11 +296,7 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                          #+(canonical-package bzip2)
                          #+(canonical-package gzip)
                          #+(canonical-package tar)
-                         ;; The comments in the 'deblob-check' script
-                         ;; claim that it supports Python 2 and 3, but
-                         ;; in fact it fails when run in Python 3 as
-                         ;; of version 5.1.3.
-                         #+python-2))
+                         #+python-wrapper))
 
                   (with-directory-excursion "/tmp/bin"
 
@@ -337,7 +333,6 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                            (error "multiple directories found" dirs)))))
 
                   (with-directory-excursion dir
-                    (setenv "PYTHON" (which "python"))
                     (format #t "Running deblob script...~%")
                     (force-output)
                     (invoke "/tmp/bin/deblob"))
-- 
2.27.0

[bug#43160] [PATCH v2 2/4] gnu: make-linux-libre-source: Set output port buffering to line mode.

[Maxim Cournoyer]

* gnu/packages/linux.scm (make-linux-libre-source): Set output port buffering
to line mode via setvbuf.  Remove the ad-hoc calls to force-output.
---
 gnu/packages/linux.scm | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 8edbe4e7e4..1b923f0c0a 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -279,6 +279,9 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                              (srfi srfi-1)
                              (ice-9 match)
                              (ice-9 ftw))
+
+                (setvbuf (current-output-port) 'line)
+
                 (let ((dir (string-append "linux-" #$version)))
 
                   (mkdir "/tmp/bin")
@@ -315,12 +318,10 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                   (if (file-is-directory? #+upstream-source)
                       (begin
                         (format #t "Copying upstream linux source...~%")
-                        (force-output)
                         (invoke "cp" "--archive" #+upstream-source dir)
                         (invoke "chmod" "--recursive" "u+w" dir))
                       (begin
                         (format #t "Unpacking upstream linux tarball...~%")
-                        (force-output)
                         (invoke "tar" "xf" #$upstream-source)
                         (match (scandir "."
                                         (lambda (name)
@@ -334,11 +335,9 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
 
                   (with-directory-excursion dir
                     (format #t "Running deblob script...~%")
-                    (force-output)
                     (invoke "/tmp/bin/deblob"))
 
                   (format #t "~%Packing new Linux-libre tarball...~%")
-                  (force-output)
                   (invoke "tar" "cvfa" #$output
                           ;; Avoid non-determinism in the archive.
                           "--mtime=@0"
-- 
2.27.0

[bug#43160] [PATCH v2 3/4] gnu: linux-libre: Validate that the cleaned up tarball is free of blobs.

[Maxim Cournoyer]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=yes, Size: 1925 bytes --]

* gnu/packages/linux.scm (make-linux-libre-source): Call the deblob-check
script on the generated tarball archive with the --use-awk and --list-blobs
options.
---
 gnu/packages/linux.scm | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 1b923f0c0a..e177386312 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -48,6 +48,7 @@
 ;;; Copyright © 2020 John Soo <jsoo1@asu.edu>
 ;;; Copyright © 2020 Michael Rohleder <mike@rohleder.de>
 ;;; Copyright © 2020 Anders Thuné <asse.97@gmail.com>
+;;; Copyright © 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -90,6 +91,7 @@
   #:use-module (gnu packages flex)
   #:use-module (gnu packages file)
   #:use-module (gnu packages freedesktop)
+  #:use-module (gnu packages gawk)
   #:use-module (gnu packages gcc)
   #:use-module (gnu packages gettext)
   #:use-module (gnu packages glib)
@@ -299,6 +301,7 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                          #+(canonical-package bzip2)
                          #+(canonical-package gzip)
                          #+(canonical-package tar)
+                         #+(canonical-package gawk)
                          #+python-wrapper))
 
                   (with-directory-excursion "/tmp/bin"
@@ -345,7 +348,11 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                           "--group=root:0"
                           "--sort=name"
                           "--hard-dereference"
-                          dir))))))))))
+                          dir)
+
+                  (format #t "~%Scanning the generated tarball for blobs...~%")
+                  (invoke "/tmp/bin/deblob-check" "--use-awk" "--list-blobs"
+                          #$output))))))))))
 
 \f
 ;;;
-- 
2.27.0

[bug#43160] [PATCH v2 4/4] gnu: linux-libre: Compare generated sources against Linux-libre releases.

[Maxim Cournoyer]

* gnu/packages/linux.scm (make-linux-libre-source): Rename the UPSTREAM-SOURCE
parameter to LINUX-UPSTREAM-SOURCE.  Add a new LINUX-LIBRE-UPSTREAM-SOURCE
parameter.  Update doc.  Adjust variable names.  Capitalize "Linux" in the
user messages.  Remove empty directories from the generated sources, then
invoke diff between these sources and those of the corresponding Linux-libre
release.
(%upstream-linux-source): Convert the hash as base32 inside the definition, to
simplify its use.
(%upstream-linux-libre-source): New procedure.
(linux-libre-5.8-pristine-source): Add a LIBRE-HASH binding and use it with
%UPSTREAM-LINUX-LIBRE-SOURCE to provide the Linux-libre release origin to the
make-linux-libre-source procedure call.
(linux-libre-5.4-pristine-source): Likewise.
(linux-libre-4.19-pristine-source): Likewise.
(linux-libre-4.14-pristine-source): Likewise.
(linux-libre-4.9-pristine-source): Likewise.
(linux-libre-4.4-pristine-source): Likewise.
---
 gnu/packages/linux.scm | 63 ++++++++++++++++++++++++++++++++----------
 1 file changed, 48 insertions(+), 15 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index e177386312..020eb1670c 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -258,10 +258,14 @@ from forcing GEXP-PROMISE."
                       #:guile-for-build guile)))
 
 (define (make-linux-libre-source version
-                                 upstream-source
+                                 linux-upstream-source
+                                 linux-libre-upstream-source
                                  deblob-scripts)
   "Return a 'computed' origin that generates a Linux-libre tarball from the
-corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
+corresponding LINUX-UPSTREAM-SOURCE (an origin), using the given
+DEBLOB-SCRIPTS.  The generated Linux-libre source is compared against the
+corresponding LINUX-LIBRE-UPSTREAM-SOURCE upstream release (an origin), to
+ensure correctness."
   (match deblob-scripts
     ((deblob-version (? origin? deblob) (? origin? deblob-check))
      (unless (string=? deblob-version (version-major+minor version))
@@ -318,14 +322,14 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                       (("/bin/sed") (which "sed"))
                       (("/usr/bin/python") (which "python"))))
 
-                  (if (file-is-directory? #+upstream-source)
+                  (if (file-is-directory? #+linux-upstream-source)
                       (begin
-                        (format #t "Copying upstream linux source...~%")
-                        (invoke "cp" "--archive" #+upstream-source dir)
+                        (format #t "Copying upstream Linux source...~%")
+                        (invoke "cp" "--archive" #+linux-upstream-source dir)
                         (invoke "chmod" "--recursive" "u+w" dir))
                       (begin
-                        (format #t "Unpacking upstream linux tarball...~%")
-                        (invoke "tar" "xf" #$upstream-source)
+                        (format #t "Unpacking upstream Linux tarball...~%")
+                        (invoke "tar" "xf" #$linux-upstream-source)
                         (match (scandir "."
                                         (lambda (name)
                                           (and (not (member name '("." "..")))
@@ -352,7 +356,16 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
 
                   (format #t "~%Scanning the generated tarball for blobs...~%")
                   (invoke "/tmp/bin/deblob-check" "--use-awk" "--list-blobs"
-                          #$output))))))))))
+                          #$output)
+
+                  (format #t "~%Comparing with the upstream Linux-libre \
+release...~%")
+                  ;; Git doesn't track empty directories, so remove them from
+                  ;; our local tree for the sake of comparison.
+                  (invoke "find" dir "-type" "d" "-empty" "-delete")
+                  (invoke "diff" "-ur"
+                          dir
+                          #+linux-libre-upstream-source))))))))))
 
 \f
 ;;;
@@ -381,55 +394,75 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
     (uri (string-append "mirror://kernel.org"
                         "/linux/kernel/v" (version-major version) ".x/"
                         "linux-" version ".tar.xz"))
-    (sha256 hash)))
+    (sha256 (base32 hash))))
 
+(define (%upstream-linux-libre-source version hash)
+  (origin
+    (method git-fetch)
+    (uri (git-reference
+          (url "git://linux-libre.fsfla.org/releases.git")
+          (commit (string-append "sources/v" version "-gnu"))))
+    (file-name (git-file-name "linux-libre-source" version))
+    (sha256 (base32 hash))))
 
 (define-public linux-libre-5.8-version "5.8.5")
 (define-public linux-libre-5.8-pristine-source
   (let ((version linux-libre-5.8-version)
-        (hash (base32 "0zwl0nk3x6fxwsbnmpx1drh7v0116yhgamisb1pghd472mmw6klx")))
+        (hash "0zwl0nk3x6fxwsbnmpx1drh7v0116yhgamisb1pghd472mmw6klx")
+        (libre-hash "0blgkbfvl5p6y6fj0xkdnd0dk2qla02pc37gj7dc3ha0asxv4mp8"))
    (make-linux-libre-source version
                             (%upstream-linux-source version hash)
+                            (%upstream-linux-libre-source version libre-hash)
                             deblob-scripts-5.8)))
 
 (define-public linux-libre-5.4-version "5.4.61")
 (define-public linux-libre-5.4-pristine-source
   (let ((version linux-libre-5.4-version)
-        (hash (base32 "197y2yb60m1k8i7mig4pa9wsrklfxq81ba3zfahwb2b31w2kvwc6")))
+        (hash "197y2yb60m1k8i7mig4pa9wsrklfxq81ba3zfahwb2b31w2kvwc6")
+        (libre-hash "1ycbalnlmgbaq3yh7yc7l8gw7c8d2x4jbwildf04zgfq9g0lv78m"))
    (make-linux-libre-source version
                             (%upstream-linux-source version hash)
+                            (%upstream-linux-libre-source version libre-hash)
                             deblob-scripts-5.4)))
 
 (define-public linux-libre-4.19-version "4.19.142")
 (define-public linux-libre-4.19-pristine-source
   (let ((version linux-libre-4.19-version)
-        (hash (base32 "19372sri4962dqf5rbr211lrfpckmj11kxsginfcwwid4hfdn4k9")))
+        (hash "19372sri4962dqf5rbr211lrfpckmj11kxsginfcwwid4hfdn4k9")
+        (libre-hash "1281d0rx17yiy9723ig381jq3bww59xqggisbxhdrxvfbxv0vvp4"))
     (make-linux-libre-source version
                              (%upstream-linux-source version hash)
+                             (%upstream-linux-libre-source version libre-hash)
                              deblob-scripts-4.19)))
 
 (define-public linux-libre-4.14-version "4.14.195")
 (define-public linux-libre-4.14-pristine-source
   (let ((version linux-libre-4.14-version)
-        (hash (base32 "08d08la3h48fbdlr3h8zbvdghydx3x9cwb4yrnm0n93hhrwjhkrr")))
+        (hash "08d08la3h48fbdlr3h8zbvdghydx3x9cwb4yrnm0n93hhrwjhkrr")
+        (libre-hash "0vgfw8jv3mnn6d9pvccqvx4v143ck02inivnhmxylq0nqfxb7nj4"))
     (make-linux-libre-source version
                              (%upstream-linux-source version hash)
+                             (%upstream-linux-libre-source version libre-hash)
                              deblob-scripts-4.14)))
 
 (define-public linux-libre-4.9-version "4.9.234")
 (define-public linux-libre-4.9-pristine-source
   (let ((version linux-libre-4.9-version)
-        (hash (base32 "1qw26x2qc29yr094c7scw68m9yz4j0b2c4f92rvi3s31s928avvm")))
+        (hash "1qw26x2qc29yr094c7scw68m9yz4j0b2c4f92rvi3s31s928avvm")
+        (libre-hash "1p7dpsqad9vra22r00ha6vg2fap4jjplfkcaskz9fvih6m4m7wgp"))
     (make-linux-libre-source version
                              (%upstream-linux-source version hash)
+                             (%upstream-linux-libre-source version libre-hash)
                              deblob-scripts-4.9)))
 
 (define-public linux-libre-4.4-version "4.4.234")
 (define-public linux-libre-4.4-pristine-source
   (let ((version linux-libre-4.4-version)
-        (hash (base32 "123354h05fip161rzlxc8h0cn5lh0d1gz06gc5b7zyz9i2lxv539")))
+        (hash "123354h05fip161rzlxc8h0cn5lh0d1gz06gc5b7zyz9i2lxv539")
+        (libre-hash "07adliis6kln7531jwwl0h2v9wkzn2j3jn2zjlyashxd9p85kywm"))
     (make-linux-libre-source version
                              (%upstream-linux-source version hash)
+                             (%upstream-linux-libre-source version libre-hash)
                              deblob-scripts-4.4)))
 
 (define %boot-logo-patch
-- 
2.27.0

[bug#43173] Ensure that the correct linux-libre deblobbing scripts are used

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

In recent discussions [0], people raised the possibility that we might
accidentally leave non-free firmware blobs in our linux-libre packages.

If I understand correctly, the root of the issue is that, currently, we
manually specify the versions of the deblobbing scripts. They are not
changed with every linux-libre release, so it is usually okay to use an
older version number — the scripts themselves will be identical.
However, sometimes the scripts do change, and we might not notice, and
thus we would fail to remove every blob from the kernel sources.

These two patches should make that failure mode impossible, by 1) making
sure that the file names of the deblobbing scripts' store items include
the full version number of the kernel and 2) only defining the version
in one place. The hashes of the deblob scripts will be checked
automatically when Guix downloads them for each new kernel release.

I had to move the linux-libre-nnn-version variables to an earlier part
of the file, so that they are defined when referenced in the
deblob-scripts-nnn procedures. I regret changing the way this code is
organized... your advice is welcome!

[0] https://lists.gnu.org/archive/html/guix-devel/2020-08/msg00040.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#43173] [PATCH 1/2] gnu: Do not truncate the version of the linux-libre deblobbing script file names.

[Leo Famulari]

* gnu/packages/linux.scm (linux-libre-deblob-scripts): Use VERSION instead of
VERSION-MAJOR+MINOR.
---
 gnu/packages/linux.scm | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index b742688f79..8b66ed2b60 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -193,7 +193,7 @@ defconfig.  Return the appropriate make target if applicable, otherwise return
           (method url-fetch)
           (uri (string-append "https://linux-libre.fsfla.org"
                               "/pub/linux-libre/releases/" version "-gnu/"
-                              "deblob-" (version-major+minor version)))
+                              "deblob-" version))
           (file-name (string-append "linux-libre-deblob-"
                                     (version-major+minor version)))
           (sha256 deblob-hash))
@@ -202,8 +202,7 @@ defconfig.  Return the appropriate make target if applicable, otherwise return
           (uri (string-append "https://linux-libre.fsfla.org"
                               "/pub/linux-libre/releases/" version "-gnu/"
                               "deblob-check"))
-          (file-name (string-append "linux-libre-deblob-check-"
-                                    (version-major+minor version)))
+          (file-name (string-append "linux-libre-deblob-check-" version))
           (sha256 deblob-check-hash))))
 
 (define deblob-scripts-5.8
-- 
2.28.0

[bug#43173] [PATCH 2/2] gnu: linux-libre: Enforce the use of the correct deblobbing scripts.

[Leo Famulari]

* gnu/packages/linux.scm (deblob-scripts-5.8, deblob-scripts-5.4,
deblob-scripts-4.19, deblob-scripts-4.14, deblob-scripts-4.9,
deblob-scripts-4.4): Use the respective LINUX-LIBRE-N-VERSION variables.
---
 gnu/packages/linux.scm | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 8b66ed2b60..727f3e07cc 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -180,6 +180,21 @@ defconfig.  Return the appropriate make target if applicable, otherwise return
         ((string-prefix? "powerpc64le-" system) "ppc64_defconfig")
         (else "defconfig")))
 
+\f
+;;;
+;;; Kernel versions.
+;;;
+
+;; The current "stable" kernel
+(define-public linux-libre-5.8-version "5.8.5")
+
+;; The "longterm" kernels with long-term upstream support
+(define-public linux-libre-5.4-version "5.4.61")
+(define-public linux-libre-4.19-version "4.19.142")
+(define-public linux-libre-4.14-version "4.14.195")
+(define-public linux-libre-4.9-version "4.9.234")
+(define-public linux-libre-4.4-version "4.4.234")
+
 \f
 ;;;
 ;;; Kernel source code deblobbing.
@@ -207,37 +222,37 @@ defconfig.  Return the appropriate make target if applicable, otherwise return
 
 (define deblob-scripts-5.8
   (linux-libre-deblob-scripts
-   "5.8.4"
+   linux-libre-5.8-version
    (base32 "07z7sglyrfh0706icqqf3shadf638pvyid9386r661ds5lbsa2mw")
    (base32 "0j6jba5fcddqlb42f95gjl78jisfla4nswqila074gglcrbnl9q7")))
 
 (define deblob-scripts-5.4
   (linux-libre-deblob-scripts
-   "5.4.61"
+   linux-libre-5.4-version
    (base32 "0ckxn7k5zgcqk30dq943bnamr6a6zjbw2aqjl3x30f4kvh5f6k25")
    (base32 "1b3q88i2qfdxyvpi9f7jds0qlb8hfpw87mgia096ax6822c2cmyb")))
 
 (define deblob-scripts-4.19
   (linux-libre-deblob-scripts
-   "4.19.142"
+   linux-libre-4.19-version
    (base32 "02zs405awaxydbapka4nz8h6lmnc0dahgczqsrs5s2bmzjyyqkcy")
    (base32 "1jiaw0as1ippkrjdpd52657w5mz9qczg3y2hlra7m9k0xawwiqlf")))
 
 (define deblob-scripts-4.14
   (linux-libre-deblob-scripts
-   "4.14.195"
+   linux-libre-4.14-version
    (base32 "091jk9jkn9jf39bxpc7395bhcb7p96nkg3a8047380ki06lnfxh6")
    (base32 "1qij18inijj6c3ma8hv98yjagnzxdxyn134da9fd23ky8q6hbvky")))
 
 (define deblob-scripts-4.9
   (linux-libre-deblob-scripts
-   "4.9.234"
+   linux-libre-4.9-version
    (base32 "1wvldzlv7q2xdbadas87dh593nxr4a8p5n0f8zpm72lja6w18hmg")
    (base32 "0fxajshb75siq39lj5h8xvhdj8lcmddkslwlyj65rhlwk6g2r4b2")))
 
 (define deblob-scripts-4.4
   (linux-libre-deblob-scripts
-   "4.4.234"
+   linux-libre-4.4-version
    (base32 "0x2j1i88am54ih2mk7gyl79g25l9zz4r08xhl482l3fvjj2irwbw")
    (base32 "0hhin1jpfkd6nwrb6xqxjzl3hdxy4pn8a15hy2d3d83yw6pflbsf")))
 
@@ -382,7 +397,6 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
     (sha256 hash)))
 
 
-(define-public linux-libre-5.8-version "5.8.5")
 (define-public linux-libre-5.8-pristine-source
   (let ((version linux-libre-5.8-version)
         (hash (base32 "0zwl0nk3x6fxwsbnmpx1drh7v0116yhgamisb1pghd472mmw6klx")))
@@ -390,7 +404,6 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                             (%upstream-linux-source version hash)
                             deblob-scripts-5.8)))
 
-(define-public linux-libre-5.4-version "5.4.61")
 (define-public linux-libre-5.4-pristine-source
   (let ((version linux-libre-5.4-version)
         (hash (base32 "197y2yb60m1k8i7mig4pa9wsrklfxq81ba3zfahwb2b31w2kvwc6")))
@@ -398,7 +411,6 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                             (%upstream-linux-source version hash)
                             deblob-scripts-5.4)))
 
-(define-public linux-libre-4.19-version "4.19.142")
 (define-public linux-libre-4.19-pristine-source
   (let ((version linux-libre-4.19-version)
         (hash (base32 "19372sri4962dqf5rbr211lrfpckmj11kxsginfcwwid4hfdn4k9")))
@@ -406,7 +418,6 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                              (%upstream-linux-source version hash)
                              deblob-scripts-4.19)))
 
-(define-public linux-libre-4.14-version "4.14.195")
 (define-public linux-libre-4.14-pristine-source
   (let ((version linux-libre-4.14-version)
         (hash (base32 "08d08la3h48fbdlr3h8zbvdghydx3x9cwb4yrnm0n93hhrwjhkrr")))
@@ -414,7 +425,6 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                              (%upstream-linux-source version hash)
                              deblob-scripts-4.14)))
 
-(define-public linux-libre-4.9-version "4.9.234")
 (define-public linux-libre-4.9-pristine-source
   (let ((version linux-libre-4.9-version)
         (hash (base32 "1qw26x2qc29yr094c7scw68m9yz4j0b2c4f92rvi3s31s928avvm")))
@@ -422,7 +432,6 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                              (%upstream-linux-source version hash)
                              deblob-scripts-4.9)))
 
-(define-public linux-libre-4.4-version "4.4.234")
 (define-public linux-libre-4.4-pristine-source
   (let ((version linux-libre-4.4-version)
         (hash (base32 "123354h05fip161rzlxc8h0cn5lh0d1gz06gc5b7zyz9i2lxv539")))
-- 
2.28.0

[bug#43173] Ensure that the correct linux-libre deblobbing scripts are used

[Mark H Weaver]

Leo Famulari <leo@famulari.name> writes:

> In recent discussions [0], people raised the possibility that we might
> accidentally leave non-free firmware blobs in our linux-libre packages.
>
> If I understand correctly, the root of the issue is that, currently, we
> manually specify the versions of the deblobbing scripts. They are not
> changed with every linux-libre release, so it is usually okay to use an
> older version number — the scripts themselves will be identical.
> However, sometimes the scripts do change, and we might not notice, and
> thus we would fail to remove every blob from the kernel sources.
>
> These two patches should make that failure mode impossible, by 1) making
> sure that the file names of the deblobbing scripts' store items include
> the full version number of the kernel and 2) only defining the version
> in one place. The hashes of the deblob scripts will be checked
> automatically when Guix downloads them for each new kernel release.
[...]
> [0] https://lists.gnu.org/archive/html/guix-devel/2020-08/msg00040.html

In the aforementioned discussion, I agreed to either:

(1) Wait until the linux-libre project publishes a new release, or
(2) Check for new blobs myself in the upstream release.

Since then, I've actually chosen option (2) a couple of times.  I did so
by reviewing each of the upstream commits looking for new blobs.
I found that it took on the order of 10-15 minutes per release.

With this proposed change, we will lose an easy way to exercise option
(2), and will effectively be constrained to always wait until
linux-libre produces a new release.

I'll leave it to the maintainers to decide what to do here, but I wanted
to make it clear what's at stake.  Personally, I do not find Jason and
Alexandre's arguments compelling, and would be in favor of retaining the
option to push these security updates more quickly by checking for new
blobs manually.

     Thanks,
       Mark

[bug#43173] Ensure that the correct linux-libre deblobbing scripts are used

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On Wed, Sep 02, 2020 at 05:07:56PM -0400, Mark H Weaver wrote:
> In the aforementioned discussion, I agreed to either:
> 
> (1) Wait until the linux-libre project publishes a new release, or
> (2) Check for new blobs myself in the upstream release.
> 
> Since then, I've actually chosen option (2) a couple of times.  I did so
> by reviewing each of the upstream commits looking for new blobs.
> I found that it took on the order of 10-15 minutes per release.
> 
> With this proposed change, we will lose an easy way to exercise option
> (2), and will effectively be constrained to always wait until
> linux-libre produces a new release.

We would still be able to use that method, by effectively reverting this
patch, as desired. The intended effect of this patch is that it will not
be possible to accidentally use the incorrect deblob scripts.

I think we should try to make it harder to make mistakes, but not forget
that we can remove the guardrails when we want to.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#43173] Ensure that the correct linux-libre deblobbing scripts are used

[Mark H Weaver]

Hi Leo,

Leo Famulari <leo@famulari.name> writes:
> We would still be able to use that method, by effectively reverting this
> patch, as desired.

I suppose that's true.  Fair enough :)

> The intended effect of this patch is that it will not be possible to
> accidentally use the incorrect deblob scripts.

I agree that it would be good to prevent this.

> I think we should try to make it harder to make mistakes, but not forget
> that we can remove the guardrails when we want to.

That makes sense.  I withdraw my objection to the overall approach, but
I have a suggestion regarding the file organization:

Instead of having all 'linux-libre-*-version' definitions in one
section, all 'deblob-scripts-*' definitions in a second section, and all
'linux-libre-*-pristine-source' definitions in a third, I suggest
putting, for each kernel version, all three of these definitions
together in one place.  That way, when performing the most common kernel
updates, everything that needs to be changed is in one place, and the
corresponding patch to Guix would have just one hunk.

More concretely, this would mean moving each 'deblob-scripts-X.Y'
definition down, between the corresponding 'linux-libre-X.Y-version' and
'linux-libre-X.Y-pristine-source' definitions.

What do you think?

       Mark

[bug#43160] [PATCH v2 1/4] gnu: linux-libre: Use Python 3 in make-linux-libre-source.

[Mathieu Othacehe]

Hello Maxim,

> Successfully tested with all of the linux-libre versions we carry in Guix:
> 4.4.234, 4.9.234, 4.14.195, 4.19.142, 5.4.61 and 5.8.5.

This looks fine. Did you check if cross-compilation is also working
correctly?

Thanks,

Mathieu

[bug#43160] [PATCH v2 1/4] gnu: linux-libre: Use Python 3 in make-linux-libre-source.

[Maxim Cournoyer]

Hello Mathieu,

Mathieu Othacehe <othacehe@gnu.org> writes:

> Hello Maxim,
>
>> Successfully tested with all of the linux-libre versions we carry in Guix:
>> 4.4.234, 4.9.234, 4.14.195, 4.19.142, 5.4.61 and 5.8.5.
>
> This looks fine. Did you check if cross-compilation is also working
> correctly?

I just tested with guix build --system=linux-armhf --check --source, and
it seems to be working correctly.  I pushed that single commit to master
as f029bca1032c7e032f2920540b0aa1a3733e2cc9.

Thank you,

Maxim

[bug#43160] [PATCH v2 1/4] gnu: linux-libre: Use Python 3 in make-linux-libre-source.

[Mathieu Othacehe]

Hey Maxim,

> I just tested with guix build --system=linux-armhf --check --source, and
> it seems to be working correctly.  I pushed that single commit to master
> as f029bca1032c7e032f2920540b0aa1a3733e2cc9.

Great. I just had a look to the rest of the patchset. This seems fine to
me :).

Thanks,

Mathieu

[bug#43160] [PATCH v2 1/4] gnu: linux-libre: Use Python 3 in make-linux-libre-source.

[Mike Rosset]

Mathieu Othacehe <othacehe@gnu.org> writes:

> Great. I just had a look to the rest of the patchset. This seems fine to
> me :).
>
> Thanks,
>
> Mathieu

Hello Mathieu

Thanks for looking at this.  I have split the patch into 3 commits and
 emailed as a new series.

Mike

[bug#43160] Validate the result of our linux-libre sources clean up

[Mark H Weaver]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
> I'd like to point you to the following patches, as they touch the
> generation of the linux-libre sources, in case they hadn't caught your
> attention: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43160.

Thanks very much for bringing this to my attention.  I do not subscribe
to the guix-patches list, so I would not have seen this otherwise.

I'm in favor of the following patches:

  gnu: linux-libre: Use Python 3 in make-linux-libre-source.
  gnu: make-linux-libre-source: Set output port buffering to line mode.
  gnu: linux-libre: Validate that the cleaned up tarball is free of blobs.

Thanks for these.  Please push them whenever you feel is appropriate.

On other other hand, I'm strongly opposed to the following patch:

  gnu: linux-libre: Compare generated sources against Linux-libre releases.

I'm opposed to it because it would make it prohibitively difficult to
push micro kernel updates (most of which contain potential security
fixes) before Linux-libre has published their tarball release.  It would
also make it prohibitively difficult to perform deblobbed bisections
between two adjacent versions from the upstream stable git repository.

In my opinion, at minimum, the 'linux-libre-upstream-source' argument to
'make-linux-libre-source' should optional.

I find it depressing that Jason's and Alexandre's attempts to browbeat
us to limit ourselves to deblob only the precise tarballs that they
produce, and to always wait for them to produce them before pushing
security fixes (although it takes less than 10 minutes to look over the
upstream commits for new blobs) have gained traction here.

      Thanks,
        Mark

[bug#43160] [PATCH v2 1/4] gnu: linux-libre: Use Python 3 in make-linux-libre-source.

[Maxim Cournoyer]

Hello!

Mathieu Othacehe <othacehe@gnu.org> writes:

> Hey Maxim,
>
>> I just tested with guix build --system=linux-armhf --check --source, and
>> it seems to be working correctly.  I pushed that single commit to master
>> as f029bca1032c7e032f2920540b0aa1a3733e2cc9.
>
> Great. I just had a look to the rest of the patchset. This seems fine to
> me :).

Thank you! As you and Mark agreed that the first 3 were good to go, I've
now pushed them.  The last one is still in discussion with Mark.

Thanks,

Maxim

[bug#43173] Ensure that the correct linux-libre deblobbing scripts are used

[Leo Famulari]

[-- Attachment #1.1: Type: text/plain, Size: 864 bytes --]

On Wed, Sep 02, 2020 at 07:53:02PM -0400, Mark H Weaver wrote:
> Instead of having all 'linux-libre-*-version' definitions in one
> section, all 'deblob-scripts-*' definitions in a second section, and all
> 'linux-libre-*-pristine-source' definitions in a third, I suggest
> putting, for each kernel version, all three of these definitions
> together in one place.  That way, when performing the most common kernel
> updates, everything that needs to be changed is in one place, and the
> corresponding patch to Guix would have just one hunk.
> 
> More concretely, this would mean moving each 'deblob-scripts-X.Y'
> definition down, between the corresponding 'linux-libre-X.Y-version' and
> 'linux-libre-X.Y-pristine-source' definitions.
> 
> What do you think?

That's better than what I had in mind — thank you! I've attached a
revised patch.

[-- Attachment #1.2: 0001-gnu-linux-libre-Enforce-the-use-of-the-correct-deblo.patch --]
[-- Type: text/plain, Size: 6462 bytes --]

From 6cbdf7e70ba0d9b98171a425bd249c702f8286de Mon Sep 17 00:00:00 2001
From: Leo Famulari <leo@famulari.name>
Date: Sat, 5 Sep 2020 14:46:04 -0400
Subject: [PATCH] gnu: linux-libre: Enforce the use of the correct deblobbing
 scripts.

* gnu/packages/linux.scm (deblob-scripts-5.8, deblob-scripts-5.4,
deblob-scripts-4.19, deblob-scripts-4.14, deblob-scripts-4.9,
deblob-scripts-4.4): Use the respective LINUX-LIBRE-X.Y-VERSION variables.
---
 gnu/packages/linux.scm | 71 +++++++++++++++++++++---------------------
 1 file changed, 35 insertions(+), 36 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index ae0bf081e9..ca9fa32e12 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -207,42 +207,6 @@ defconfig.  Return the appropriate make target if applicable, otherwise return
           (file-name (string-append "linux-libre-deblob-check-" version))
           (sha256 deblob-check-hash))))
 
-(define deblob-scripts-5.8
-  (linux-libre-deblob-scripts
-   "5.8.6"
-   (base32 "07z7sglyrfh0706icqqf3shadf638pvyid9386r661ds5lbsa2mw")
-   (base32 "0j6jba5fcddqlb42f95gjl78jisfla4nswqila074gglcrbnl9q7")))
-
-(define deblob-scripts-5.4
-  (linux-libre-deblob-scripts
-   "5.4.62"
-   (base32 "0ckxn7k5zgcqk30dq943bnamr6a6zjbw2aqjl3x30f4kvh5f6k25")
-   (base32 "1b3q88i2qfdxyvpi9f7jds0qlb8hfpw87mgia096ax6822c2cmyb")))
-
-(define deblob-scripts-4.19
-  (linux-libre-deblob-scripts
-   "4.19.143"
-   (base32 "02zs405awaxydbapka4nz8h6lmnc0dahgczqsrs5s2bmzjyyqkcy")
-   (base32 "1jiaw0as1ippkrjdpd52657w5mz9qczg3y2hlra7m9k0xawwiqlf")))
-
-(define deblob-scripts-4.14
-  (linux-libre-deblob-scripts
-   "4.14.196"
-   (base32 "091jk9jkn9jf39bxpc7395bhcb7p96nkg3a8047380ki06lnfxh6")
-   (base32 "1qij18inijj6c3ma8hv98yjagnzxdxyn134da9fd23ky8q6hbvky")))
-
-(define deblob-scripts-4.9
-  (linux-libre-deblob-scripts
-   "4.9.235"
-   (base32 "1wvldzlv7q2xdbadas87dh593nxr4a8p5n0f8zpm72lja6w18hmg")
-   (base32 "0fxajshb75siq39lj5h8xvhdj8lcmddkslwlyj65rhlwk6g2r4b2")))
-
-(define deblob-scripts-4.4
-  (linux-libre-deblob-scripts
-   "4.4.235"
-   (base32 "0x2j1i88am54ih2mk7gyl79g25l9zz4r08xhl482l3fvjj2irwbw")
-   (base32 "0hhin1jpfkd6nwrb6xqxjzl3hdxy4pn8a15hy2d3d83yw6pflbsf")))
-
 (define* (computed-origin-method gexp-promise hash-algo hash
                                  #:optional (name "source")
                                  #:key (system (%current-system))
@@ -383,7 +347,14 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
     (sha256 hash)))
 
 
+;; The current "stable" kernel. That is, the most recently released major
+;; version.
 (define-public linux-libre-5.8-version "5.8.6")
+(define deblob-scripts-5.8
+  (linux-libre-deblob-scripts
+   linux-libre-5.8-version
+   (base32 "07z7sglyrfh0706icqqf3shadf638pvyid9386r661ds5lbsa2mw")
+   (base32 "0j6jba5fcddqlb42f95gjl78jisfla4nswqila074gglcrbnl9q7")))
 (define-public linux-libre-5.8-pristine-source
   (let ((version linux-libre-5.8-version)
         (hash (base32 "180bka8a0f2ykaifgb323pzgh0n909mlrsk08l08zmifggnh19cc")))
@@ -391,7 +362,15 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                             (%upstream-linux-source version hash)
                             deblob-scripts-5.8)))
 
+;; The "longterm" kernels — the older releases with long-term upstream support.
+;; Here are the support timelines:
+;; <https://www.kernel.org/category/releases.html>
 (define-public linux-libre-5.4-version "5.4.62")
+(define deblob-scripts-5.4
+  (linux-libre-deblob-scripts
+   linux-libre-5.4-version
+   (base32 "0ckxn7k5zgcqk30dq943bnamr6a6zjbw2aqjl3x30f4kvh5f6k25")
+   (base32 "1b3q88i2qfdxyvpi9f7jds0qlb8hfpw87mgia096ax6822c2cmyb")))
 (define-public linux-libre-5.4-pristine-source
   (let ((version linux-libre-5.4-version)
         (hash (base32 "0w49y8lymz23x4mr5byaxnrkhm56lwfhnqkra07hqyfr5y63v216")))
@@ -400,6 +379,11 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                             deblob-scripts-5.4)))
 
 (define-public linux-libre-4.19-version "4.19.143")
+(define deblob-scripts-4.19
+  (linux-libre-deblob-scripts
+   linux-libre-4.19-version
+   (base32 "02zs405awaxydbapka4nz8h6lmnc0dahgczqsrs5s2bmzjyyqkcy")
+   (base32 "1jiaw0as1ippkrjdpd52657w5mz9qczg3y2hlra7m9k0xawwiqlf")))
 (define-public linux-libre-4.19-pristine-source
   (let ((version linux-libre-4.19-version)
         (hash (base32 "1383yfwb962mhn25b3b3zqrwnpyp01g5xclsv14wra0fdz33ahra")))
@@ -408,6 +392,11 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                              deblob-scripts-4.19)))
 
 (define-public linux-libre-4.14-version "4.14.196")
+(define deblob-scripts-4.14
+  (linux-libre-deblob-scripts
+   linux-libre-4.14-version
+   (base32 "091jk9jkn9jf39bxpc7395bhcb7p96nkg3a8047380ki06lnfxh6")
+   (base32 "1qij18inijj6c3ma8hv98yjagnzxdxyn134da9fd23ky8q6hbvky")))
 (define-public linux-libre-4.14-pristine-source
   (let ((version linux-libre-4.14-version)
         (hash (base32 "16mhqymwkgqi8zalcij5c754smc8ysvfw6l2cwshr4scipsv4qay")))
@@ -416,6 +405,11 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                              deblob-scripts-4.14)))
 
 (define-public linux-libre-4.9-version "4.9.235")
+(define deblob-scripts-4.9
+  (linux-libre-deblob-scripts
+   linux-libre-4.9-version
+   (base32 "1wvldzlv7q2xdbadas87dh593nxr4a8p5n0f8zpm72lja6w18hmg")
+   (base32 "0fxajshb75siq39lj5h8xvhdj8lcmddkslwlyj65rhlwk6g2r4b2")))
 (define-public linux-libre-4.9-pristine-source
   (let ((version linux-libre-4.9-version)
         (hash (base32 "1hqcb3zw4546h6x5xy2mywdznha8813lx15mxbgfbvwm4qhsc9g6")))
@@ -424,6 +418,11 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                              deblob-scripts-4.9)))
 
 (define-public linux-libre-4.4-version "4.4.235")
+(define deblob-scripts-4.4
+  (linux-libre-deblob-scripts
+   linux-libre-4.4-version
+   (base32 "0x2j1i88am54ih2mk7gyl79g25l9zz4r08xhl482l3fvjj2irwbw")
+   (base32 "0hhin1jpfkd6nwrb6xqxjzl3hdxy4pn8a15hy2d3d83yw6pflbsf")))
 (define-public linux-libre-4.4-pristine-source
   (let ((version linux-libre-4.4-version)
         (hash (base32 "0w5pkv936zb0shjgnpv17gcp5n8f91djznzq54p6j1bl5q2qdyqd")))
-- 
2.28.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#43173] Ensure that the correct linux-libre deblobbing scripts are used

[Mark H Weaver]

Leo Famulari <leo@famulari.name> writes:

> That's better than what I had in mind — thank you! I've attached a
> revised patch.
> From 6cbdf7e70ba0d9b98171a425bd249c702f8286de Mon Sep 17 00:00:00 2001
> From: Leo Famulari <leo@famulari.name>
> Date: Sat, 5 Sep 2020 14:46:04 -0400
> Subject: [PATCH] gnu: linux-libre: Enforce the use of the correct deblobbing
>  scripts.
>
> * gnu/packages/linux.scm (deblob-scripts-5.8, deblob-scripts-5.4,
> deblob-scripts-4.19, deblob-scripts-4.14, deblob-scripts-4.9,
> deblob-scripts-4.4): Use the respective LINUX-LIBRE-X.Y-VERSION variables.

This new patch looks good to me.  Feel free to push.

    Thanks!
      Mark

bug#43173: Ensure that the correct linux-libre deblobbing scripts are used

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 826 bytes --]

On Sat, Sep 05, 2020 at 07:07:01PM -0400, Mark H Weaver wrote:
> Leo Famulari <leo@famulari.name> writes:
> 
> > That's better than what I had in mind — thank you! I've attached a
> > revised patch.
> > From 6cbdf7e70ba0d9b98171a425bd249c702f8286de Mon Sep 17 00:00:00 2001
> > From: Leo Famulari <leo@famulari.name>
> > Date: Sat, 5 Sep 2020 14:46:04 -0400
> > Subject: [PATCH] gnu: linux-libre: Enforce the use of the correct deblobbing
> >  scripts.
> >
> > * gnu/packages/linux.scm (deblob-scripts-5.8, deblob-scripts-5.4,
> > deblob-scripts-4.19, deblob-scripts-4.14, deblob-scripts-4.9,
> > deblob-scripts-4.4): Use the respective LINUX-LIBRE-X.Y-VERSION variables.
> 
> This new patch looks good to me.  Feel free to push.

Thanks for your review! Pushed as fe752d8c4545735edd71362805cbe78b78b8e9ab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#43160] Validate the result of our linux-libre sources clean up

[Maxim Cournoyer]

Hi Mark!

Mark H Weaver <mhw@netris.org> writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>> I'd like to point you to the following patches, as they touch the
>> generation of the linux-libre sources, in case they hadn't caught your
>> attention: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43160.
>
> Thanks very much for bringing this to my attention.  I do not subscribe
> to the guix-patches list, so I would not have seen this otherwise.
>
> I'm in favor of the following patches:
>
>   gnu: linux-libre: Use Python 3 in make-linux-libre-source.
>   gnu: make-linux-libre-source: Set output port buffering to line mode.
>   gnu: linux-libre: Validate that the cleaned up tarball is free of blobs.
>
> Thanks for these.  Please push them whenever you feel is appropriate.

Thanks for taking a look!  I've now done so.

> On other other hand, I'm strongly opposed to the following patch:
>
>   gnu: linux-libre: Compare generated sources against Linux-libre releases.
>
> I'm opposed to it because it would make it prohibitively difficult to
> push micro kernel updates (most of which contain potential security
> fixes) before Linux-libre has published their tarball release.

Following recent discussions, I had understood that you agreed to wait
for the Linux-libre releases before bumping our own releases.  It seems
the Linux-libre releases occur fast enough to not pose much of a
security issue; below is what I did to arrive to this conclusion.

For Linux stable, the author dates of the last releases of the version 5
series, omitting release candidates:

--8<---------------cut here---------------start------------->8---
$ git tag | grep -E '5\.[0-9]+\.' | grep -v -- '-rc' \
    | sort -t '.' -k1,1n -k2,2n -k3,3n | tail -n20 \
    | xargs -i{} git log --date='format:%c' --pretty='%ad %d' {} -1
Wed 24 Jun 2020 05:49:26 PM GMT  (tag: v5.7.6)
Tue 30 Jun 2020 04:21:22 PM GMT  (tag: v5.7.7)
Thu 09 Jul 2020 09:39:40 AM GMT  (tag: v5.7.8)
Thu 16 Jul 2020 08:13:36 AM GMT  (tag: v5.7.9)
Wed 22 Jul 2020 09:34:29 AM GMT  (tag: v5.7.10)
Wed 29 Jul 2020 10:20:01 AM GMT  (tag: v5.7.11)
Fri 31 Jul 2020 06:47:17 PM GMT  (tag: v5.7.12)
Wed 05 Aug 2020 09:58:51 AM GMT  (tag: v5.7.13)
Fri 07 Aug 2020 09:33:11 AM GMT  (tag: v5.7.14)
Tue 11 Aug 2020 03:35:42 PM GMT  (tag: v5.7.15)
Wed 19 Aug 2020 08:24:20 AM GMT  (tag: v5.7.16)
Fri 21 Aug 2020 01:07:46 PM GMT  (tag: v5.7.17)
Wed 26 Aug 2020 11:42:25 AM GMT  (tag: v5.7.18)
Thu 27 Aug 2020 09:30:50 AM GMT  (tag: v5.7.19, origin/linux-5.7.y)
Tue 11 Aug 2020 03:48:12 PM GMT  (tag: v5.8.1)
Wed 19 Aug 2020 08:27:10 AM GMT  (tag: v5.8.2)
Fri 21 Aug 2020 01:15:22 PM GMT  (tag: v5.8.3)
Wed 26 Aug 2020 11:49:20 AM GMT  (tag: v5.8.4)
Thu 27 Aug 2020 09:31:49 AM GMT  (tag: v5.8.5)
Thu 03 Sep 2020 11:29:52 AM GMT  (tag: v5.8.6, origin/linux-5.8.y)
--8<---------------cut here---------------end--------------->8---

Similarly, for Linux-libre:

--8<---------------cut here---------------start------------->8---
git tag | grep -E 'sources/v5\.[0-9]+\.' | grep -v -- '-rc' \
    | sort -t '.' -k1,1n -k2,2n -k3,3n | tail -n20 \
    | xargs -i{} git log --date='format:%c' --pretty='%ad %d' {} -1
Wed 24 Jun 2020 02:51:34 PM GMT  (tag: sources/v5.7.6-gnu)
Wed 01 Jul 2020 02:01:47 PM GMT  (tag: sources/v5.7.7-gnu)
Thu 09 Jul 2020 08:59:49 AM GMT  (tag: sources/v5.7.8-gnu)
Thu 16 Jul 2020 11:51:43 AM GMT  (tag: sources/v5.7.9-gnu)
Wed 22 Jul 2020 06:40:22 AM GMT  (tag: sources/v5.7.10-gnu)
Wed 29 Jul 2020 06:33:25 AM GMT  (tag: sources/v5.7.11-gnu)
Fri 31 Jul 2020 02:22:04 PM GMT  (tag: sources/v5.7.12-gnu)
Wed 05 Aug 2020 05:44:37 AM GMT  (tag: sources/v5.7.13-gnu)
Fri 07 Aug 2020 04:46:28 AM GMT  (tag: sources/v5.7.14-gnu)
Tue 11 Aug 2020 02:48:28 PM GMT  (tag: sources/v5.7.15-gnu)
Wed 19 Aug 2020 02:14:46 PM GMT  (tag: sources/v5.7.16-gnu)
Fri 21 Aug 2020 09:37:45 AM GMT  (tag: sources/v5.7.17-gnu)
Wed 26 Aug 2020 07:27:54 AM GMT  (tag: sources/v5.7.18-gnu)
Thu 27 Aug 2020 01:14:21 PM GMT  (tag: sources/v5.7.19-gnu)
Tue 11 Aug 2020 02:47:58 PM GMT  (tag: sources/v5.8.1-gnu)
Wed 19 Aug 2020 02:15:42 PM GMT  (tag: sources/v5.8.2-gnu)
Fri 21 Aug 2020 09:37:45 AM GMT  (tag: sources/v5.8.3-gnu)
Wed 26 Aug 2020 07:27:54 AM GMT  (tag: sources/v5.8.4-gnu)
Thu 27 Aug 2020 01:14:21 PM GMT  (tag: sources/v5.8.5-gnu)
Thu 03 Sep 2020 07:14:30 AM GMT  (tag: sources/v5.8.6-gnu)
--8<---------------cut here---------------end--------------->8---

While the author dates of the commits don't appear to be very precise
(some Linux-libre commits would have occurred before their Linux
counterpart), we can at least see that each Linux release was met with a
Linux-libre on the same day for all except the 5.7.7 release.

Also, if we compare with our own Linux-libre update timings:

--8<---------------cut here---------------start------------->8---
git log --grep 'gnu: linux-libre: Update to 5' --date='format:%c' \
    --pretty='%ad %s' | head -n20 | sort -r -t '.' -k1,1n -k2,2n -k3,3n
Thu 11 Jun 2020 04:15:35 PM GMT gnu: linux-libre: Update to 5.4.46.
Thu 18 Jun 2020 12:39:23 AM GMT gnu: linux-libre: Update to 5.4.47
Mon 22 Jun 2020 09:02:33 PM GMT gnu: linux-libre: Update to 5.4.48.
Wed 24 Jun 2020 09:08:00 PM GMT gnu: linux-libre: Update to 5.4.49.
Wed 01 Jul 2020 01:31:06 PM GMT gnu: linux-libre: Update to 5.4.50.
Thu 09 Jul 2020 04:40:27 PM GMT gnu: linux-libre: Update to 5.4.51.
Thu 16 Jul 2020 03:37:05 PM GMT gnu: linux-libre: Update to 5.4.52.
Thu 23 Jul 2020 12:28:46 AM GMT gnu: linux-libre: Update to 5.4.53.
Wed 29 Jul 2020 05:14:00 PM GMT gnu: linux-libre: Update to 5.4.54.
Sat 01 Aug 2020 12:07:08 AM GMT gnu: linux-libre: Update to 5.4.55.
Wed 05 Aug 2020 03:21:53 PM GMT gnu: linux-libre: Update to 5.4.56.
Sat 01 Aug 2020 12:39:30 PM GMT gnu: linux-libre: Update to 5.7.12.
Fri 07 Aug 2020 09:37:11 PM GMT gnu: linux-libre: Update to 5.7.14.
Tue 11 Aug 2020 05:34:48 PM GMT gnu: linux-libre: Update to 5.7.15.
Wed 19 Aug 2020 07:35:03 PM GMT gnu: linux-libre: Update to 5.7.16.
Thu 20 Aug 2020 04:03:46 PM GMT gnu: linux-libre: Update to 5.8.2.
Fri 21 Aug 2020 09:01:17 PM GMT gnu: linux-libre: Update to 5.8.3.
Wed 26 Aug 2020 04:01:11 PM GMT gnu: linux-libre: Update to 5.8.4.
Thu 27 Aug 2020 04:13:32 PM GMT gnu: linux-libre: Update to 5.8.5.
Thu 03 Sep 2020 01:56:31 PM GMT gnu: linux-libre: Update to 5.8.6.
--8<---------------cut here---------------end--------------->8---

For the subset that we did package, we were always trailing the
Linux-libre releases, so the argument that waiting for their releases
would hamper our security doesn't seem to hold.

> also make it prohibitively difficult to perform deblobbed bisections
> between two adjacent versions from the upstream stable git repository.

In my opinion, we should not trade our correctness guarantee in exchange
for convenience, especially if the convenience is only gained in such a
corner case as per-commit bisection of the Linux kernel.  It'd be
oversimplifying to say that the Linux-libre developers just run their
scripts to produce a release; they also manually screen the new upstream
changes and update their scripts accordingly.  To give due credit to
their efforts, we should not simply run their scripts with a newer
version/commit of Linux and expect arriving at a correct result.

> In my opinion, at minimum, the 'linux-libre-upstream-source' argument to
> 'make-linux-libre-source' should optional.

Perhaps, like for the change proposed by Leo, the edge case of bisecting
per-commit could be accommodated by reverting this patch when needed?
It seems more important that the common case be rigorously verified.

Also note that it should be possible to:

1) Test each packaged release in Guix to "bisect" (duh)
2) Test any Linux stable release via the Linux-libre git repo, building
with a command such as "guix build
--with-git-url=linux-libre=git://linux-libre.fsfla.org/releases.git
--with-commit=linux-libre=sources/v5.8.3-gnu linux-libre". Unfortunately
this can't be done from the command line using 'guix system build ...'
but it should be easy to define your own linux-libre package using the
'make-linux-libre*' procedure (which will gladly accept any linux-libre
source).

For when the per-commit granularity is not required.

In the future, the linux-libre git repo will apply their clean ups per
commit, allowing to do like 2) above for any commit.

> I find it depressing that Jason's and Alexandre's attempts to browbeat
> us to limit ourselves to deblob only the precise tarballs that they
> produce, and to always wait for them to produce them before pushing
> security fixes (although it takes less than 10 minutes to look over the
> upstream commits for new blobs) have gained traction here.

Despite the somewhat corrosive tone of the exchange, some valid points
were made.  I've scavenged these and adapted the recipe.  I think the
end result is a win-win situation for both Linux-libre and Guix.

As shown above, there hasn't been a case where the Linux-libre effort
slowed down the deployment of a new Linux kernel version in Guix.  I
don't foresee this changing.

What do you think? Are there holes in my analysis/understanding?

Thank you,

Maxim

[bug#43160] Validate the result of our linux-libre sources clean up

[Mark H Weaver]

Hi Maxim,

Thanks again for the patches that you've already pushed.
They are a great improvement.

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> I'm opposed to it because it would make it prohibitively difficult to
>> push micro kernel updates (most of which contain potential security
>> fixes) before Linux-libre has published their tarball release.
>
> Following recent discussions, I had understood that you agreed to wait
> for the Linux-libre releases before bumping our own releases.

Sorry, but that's incorrect.  I agreed to either wait or to look for new
blobs myself.  I've already exercised that second option a couple of
times since making that pledge.  I've found it to be quite easy.  The
overwhelming majority of commits to the stable branches are bug fixes
that obviously do not add anything resembling a blob.

If you haven't already done so, I'd encourage you to look over the
upstream commits between two consecutive upstream stable releases, to
see more concretely what I'm talking about, and how straightforward it
is to conclude that no new blobs could possibly have been added.

> It seems the Linux-libre releases occur fast enough to not pose much
> of a security issue; below is what I did to arrive to this conclusion.

The timestamp data you collected is clearly off by several hours.  It
appears to show that over two thirds of linux-libre releases come out
*before* the corresponding linux release.  One third of them appear to
have come out more than 4 hours before the corresponding linux release.
That cannot be right.  I guess someone's clock is off by several hours,
or somehow the timezones are not being taken into account.

I can tell you that within the last couple of weeks, since my pledge to
either wait for linux-libre or to check for blobs myself, I very clearly
remember upstream updates being tagged a few hours after midnight one
morning, and that it was not until after dark the next evening before
linux-libre had tagged their releases.  That instance is not reflected
in the data you collected, which again suggests to me that there's a
very significant systemic error in that data.

Even with this apparently large bias in the data, it shows that
linux-libre-5.7.7 was tagged almost 22 hours after the corresponding
linux release.  Most likely that was actually at least 28 hours.

Please don't misunderstand me: I do not fault the Linux-libre developers
for not being quick enough.  On the contrary, they've been consistently
*excellent* in that regard for as long as I've been paying attention.  I
certainly don't fault them for occasionally spending some time away from
their computers.

What I _do_ fault them for is *insisting* on placing themselves in the
middle of what should be a fast path for security updates from the
upstream developers to us.

To my mind, the practices that the Linux-libre developers are attempting
to impose on us feel like Service as a Software Substitute (SaaSS), but
in this case enforced by social pressure (and the suggestion that it
might violate the GNU FSDG) instead of the usual technical restrictions.

I'll also note that what you're proposing will apparently not be enough
to satisfy Jason and Alexandre.  They've gone so far as to suggest that
it's improper for an FSDG-compliant distribution to ever download
non-FSDG-compliant source code to a user machine.  It seems that to
satisfy them, Guix developers would apparently need to host our own
distribution site for cleaned-up copies of source tarballs, and to use
some separate tools to produce and upload new source tarballs to this
site for every update of software whose upstream source is not
FSDG-compliant.

I don't know about you, but I find their demands *oppressive*.

>> also make it prohibitively difficult to perform deblobbed bisections
>> between two adjacent versions from the upstream stable git repository.
>
> In my opinion, we should not trade our correctness guarantee in exchange
> for convenience,

First, I think that it's a mistake to suggest that any "correctness
guarantee" exists or could exist in Guix, with or without your proposed
patch.  Massive amounts of new code are flowing into Guix from upstream
projects on a daily basis, almost none of which is checked for
FSDG-compliance ahead of time.  It is widely acknowledged, even in the
FSDG document itself, that the most we can realistically hope to do is
to pledge to fix FSDG-compliance problems in a timely fashion if they
are brought to our attention.

Secondly, I think you're exaggerating the remaining risk.

I acknowledge that before these discussions began, the risk of
introducing new blobs was as high as 3% per Linux-libre update, which I
agree was too high.  However, we've made several important changes since
then.  Most importantly, I pledged to either wait for Linux-libre
updates or to manually check for new blobs, and you introduced a
'deblob-check' pass in 'make-linux-libre-source'.  Leo also made changes
to eliminate the risk of old deblob scripts being accidentally used.

It seems to me that these changes already reduce the risk of
accidentally introducing new blobs in our Linux-libre packages to near
zero, and probably at least an order of magnitude less than the risk of
non-FSDG-compliant code being introduced in, e.g., ungoogled-chromium.

> It'd be oversimplifying to say that the Linux-libre developers just
> run their scripts to produce a release; they also manually screen the
> new upstream changes and update their scripts accordingly.

Agreed, and we now account for that by either (1) waiting for them to
certify a new release or (2) checking manually for blobs ourselves.

> To give due credit to their efforts, we should not simply run their
> scripts with a newer version/commit of Linux and expect arriving at a
> correct result.

I've already agreed not to do that anymore.

It seems to me that some of these arguments are outdated, based on our
practices from before these discussions began.

>> In my opinion, at minimum, the 'linux-libre-upstream-source' argument to
>> 'make-linux-libre-source' should optional.
>
> Perhaps, like for the change proposed by Leo, the edge case of bisecting
> per-commit could be accommodated by reverting this patch when needed?

That can be done easily for Leo's patch, but not for yours.

In the case of Leo's patch, if I choose to manually check for new blobs,
I can simply change one line in a 'deblob-script-X.Y' definition, like
this:

--8<---------------cut here---------------start------------->8---
 (define-public linux-libre-5.8-version "5.8.7")
 (define deblob-scripts-5.8
   (linux-libre-deblob-scripts
-   linux-libre-5.8-version
+   "5.8.6"
    (base32 "07z7sglyrfh0706icqqf3shadf638pvyid9386r661ds5lbsa2mw")
    (base32 "0j6jba5fcddqlb42f95gjl78jisfla4nswqila074gglcrbnl9q7")))
--8<---------------cut here---------------end--------------->8---

In contrast, your patch changes the 'make-linux-libre-source' procedure
to *require* an existing 'linux-libre' tarball that precisely matches
what it's going to produce.  In other words, it removes the ability to
produce a new tarball, and has been reduced to merely being a verifier
of pre-existing tarballs.

Reverting those changes would not only be extremely invasive for a
simple micro kernel update, but would also, as a side effect, entail
redeblobing and rebuilding *every* version of linux-libre in Guix, even
if only one or two of the kernels needed updates.

I guess from my perspective, I see a lot of disadvantages:
disempowerment of users, occasional unnecessary delays on the order of
tens of hours to deploy security updates, not to mention having to do
another expensive git checkout and comparison on every kernel build, and
for what?

The main argument in favor, namely to reduce the risk of blobs being
accidentally included in our kernel updates, seems to be adequately
addressed by (1) my pledge to either wait or to check for blobs myself,
and (2) the recently-added 'deblob-check' invocation in
'make-linux-libre-source'.  Do you think these are insufficient?

Thanks again for this discussion, and for the work you've already done
to improve our deblobbing.

       Regards,
         Mark

[bug#43160] Validate the result of our linux-libre sources clean up

[Maxim Cournoyer]

Hello Mark,

Sorry for the delayed reply.

Mark H Weaver <mhw@netris.org> writes:

> Hi Maxim,
>
> Thanks again for the patches that you've already pushed.
> They are a great improvement.

Thank you for the kind words.

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> I'm opposed to it because it would make it prohibitively difficult to
>>> push micro kernel updates (most of which contain potential security
>>> fixes) before Linux-libre has published their tarball release.
>>
>> Following recent discussions, I had understood that you agreed to wait
>> for the Linux-libre releases before bumping our own releases.
>
> Sorry, but that's incorrect.  I agreed to either wait or to look for new
> blobs myself.  I've already exercised that second option a couple of
> times since making that pledge.  I've found it to be quite easy.  The
> overwhelming majority of commits to the stable branches are bug fixes
> that obviously do not add anything resembling a blob.

Thanks for clarifying the option you chose.

> If you haven't already done so, I'd encourage you to look over the
> upstream commits between two consecutive upstream stable releases, to
> see more concretely what I'm talking about, and how straightforward it
> is to conclude that no new blobs could possibly have been added.

I just did for v5.4.8..v5.4.9 and v5.8.5..v5.8.6.  The human brain is
quite good at spotting patterns, so it wasn't too difficult indeed, but
the bug fix releases had a rather large number of commits (191 and 254,
respectively).  That's a serious commitment to make.

>> It seems the Linux-libre releases occur fast enough to not pose much
>> of a security issue; below is what I did to arrive to this conclusion.
>
> The timestamp data you collected is clearly off by several hours.  It
> appears to show that over two thirds of linux-libre releases come out
> *before* the corresponding linux release.  One third of them appear to
> have come out more than 4 hours before the corresponding linux release.
> That cannot be right.  I guess someone's clock is off by several hours,
> or somehow the timezones are not being taken into account.

The git commands I used are supposed to take the timezones into account
(converting the author dates into my local time), but as you mentioned
something seems off.

> I can tell you that within the last couple of weeks, since my pledge to
> either wait for linux-libre or to check for blobs myself, I very clearly
> remember upstream updates being tagged a few hours after midnight one
> morning, and that it was not until after dark the next evening before
> linux-libre had tagged their releases.  That instance is not reflected
> in the data you collected, which again suggests to me that there's a
> very significant systemic error in that data.
>
> Even with this apparently large bias in the data, it shows that
> linux-libre-5.7.7 was tagged almost 22 hours after the corresponding
> linux release.  Most likely that was actually at least 28 hours.
>
> Please don't misunderstand me: I do not fault the Linux-libre developers
> for not being quick enough.  On the contrary, they've been consistently
> *excellent* in that regard for as long as I've been paying attention.  I
> certainly don't fault them for occasionally spending some time away from
> their computers.
>
> What I _do_ fault them for is *insisting* on placing themselves in the
> middle of what should be a fast path for security updates from the
> upstream developers to us.

I agree that it is sub-optimal, but I don't see how that could be made
differently, with upstream Linux being at odds with our ideals.  We
can't have it both ways.

> To my mind, the practices that the Linux-libre developers are attempting
> to impose on us feel like Service as a Software Substitute (SaaSS), but
> in this case enforced by social pressure (and the suggestion that it
> might violate the GNU FSDG) instead of the usual technical restrictions.
>
> I'll also note that what you're proposing will apparently not be enough
> to satisfy Jason and Alexandre.  They've gone so far as to suggest that
> it's improper for an FSDG-compliant distribution to ever download
> non-FSDG-compliant source code to a user machine.  It seems that to
> satisfy them, Guix developers would apparently need to host our own
> distribution site for cleaned-up copies of source tarballs, and to use
> some separate tools to produce and upload new source tarballs to this
> site for every update of software whose upstream source is not
> FSDG-compliant.

I've reached the same conclusion, after engaging with them in the
#linux-libre channel.  To my understanding, we are not violating the
current GNU FSDG.  The spirit of the FSDG, as I understand it, is
basically "don't thwart users towards nonfree software".  I don't think
downloading a nonfree archive with the objective of turning it into free
software counts as such.  To the contrary, it can be seen as some form
of empowerment of how the process of freeing the software they use work,
enabling them to more directly improve such process.

I'd really like someone explain to me how the process of cleaning
nonfree software into free software can be considered a freedom issue.
So far the people I've asked either support their argument using their
personal interpretation of the FSDG or simply do not produce an answer.

[...]

>>> also make it prohibitively difficult to perform deblobbed bisections
>>> between two adjacent versions from the upstream stable git repository.
>>
>> In my opinion, we should not trade our correctness guarantee in exchange
>> for convenience,
>
> First, I think that it's a mistake to suggest that any "correctness
> guarantee" exists or could exist in Guix, with or without your proposed
> patch.  Massive amounts of new code are flowing into Guix from upstream
> projects on a daily basis, almost none of which is checked for
> FSDG-compliance ahead of time.  It is widely acknowledged, even in the
> FSDG document itself, that the most we can realistically hope to do is
> to pledge to fix FSDG-compliance problems in a timely fashion if they
> are brought to our attention.

I agree about the broader FSDG-compliance correctness guarantee.  I'm
sorry if I was unclear, but what I meant was in the more narrow sense of
correctness w.r.t. the official Linux-libre releases.

> Secondly, I think you're exaggerating the remaining risk.
>
> I acknowledge that before these discussions began, the risk of
> introducing new blobs was as high as 3% per Linux-libre update, which I
> agree was too high.  However, we've made several important changes since
> then.  Most importantly, I pledged to either wait for Linux-libre
> updates or to manually check for new blobs, and you introduced a
> 'deblob-check' pass in 'make-linux-libre-source'.  Leo also made changes
> to eliminate the risk of old deblob scripts being accidentally used.

Those are good improvements, yes.  My fear is that the amount of
complexity upgrading our Linux-libre package is starting to reach
unwieldy levels.  Consider a newcomer wanting to upgrade our Linux-libre
package.  They bump the sources.  Run 'make defconfig'.  Successfully
build a kernel.  Send a patch.  One of us reviews it, it looks OK, gets
merged.  It's easy to see what can silently go wrong.  I'd prefer
leaving the Linux-libre team handle that complexity rather than
duplicate it.

[...]

>>> In my opinion, at minimum, the 'linux-libre-upstream-source' argument to
>>> 'make-linux-libre-source' should optional.
>>
>> Perhaps, like for the change proposed by Leo, the edge case of bisecting
>> per-commit could be accommodated by reverting this patch when needed?
>
> That can be done easily for Leo's patch, but not for yours.
>
> In the case of Leo's patch, if I choose to manually check for new blobs,
> I can simply change one line in a 'deblob-script-X.Y' definition, like
> this:
>
>  (define-public linux-libre-5.8-version "5.8.7")
>  (define deblob-scripts-5.8
>    (linux-libre-deblob-scripts
> -   linux-libre-5.8-version
> +   "5.8.6"
>     (base32 "07z7sglyrfh0706icqqf3shadf638pvyid9386r661ds5lbsa2mw")
>     (base32 "0j6jba5fcddqlb42f95gjl78jisfla4nswqila074gglcrbnl9q7")))
>
> In contrast, your patch changes the 'make-linux-libre-source' procedure
> to *require* an existing 'linux-libre' tarball that precisely matches
> what it's going to produce.  In other words, it removes the ability to
> produce a new tarball, and has been reduced to merely being a verifier
> of pre-existing tarballs.

I think that's a good thing, at least for the usual use case.  But now
that you've mentioned that there *is* an occasional need to push
security updates faster than the Linux-libre official releases can
allow, I've modified the patch to make it optional, as you suggested.
I'd suggest that disabling that check be only allowed for seasoned
linux-libre maintainers such as you or Leo, and only when strictly
necessary (serious security threats affecting the Guix System and no
Linux-libre release yet).  Any other casual "bump" should fully make use
of the verification machinery.

> Reverting those changes would not only be extremely invasive for a
> simple micro kernel update, but would also, as a side effect, entail
> redeblobing and rebuilding *every* version of linux-libre in Guix, even
> if only one or two of the kernels needed updates.

I was only considering reverting this as a local hack in the context of
per-commit bisection.

> I guess from my perspective, I see a lot of disadvantages:
> disempowerment of users, occasional unnecessary delays on the order of
> tens of hours to deploy security updates, not to mention having to do
> another expensive git checkout and comparison on every kernel build, and
> for what?

This ensures we benefit from the experience of the Linux-libre
developers in screening the Linux sources for new nonfree threats.

> The main argument in favor, namely to reduce the risk of blobs being
> accidentally included in our kernel updates, seems to be adequately
> addressed by (1) my pledge to either wait or to check for blobs myself,
> and (2) the recently-added 'deblob-check' invocation in
> 'make-linux-libre-source'.  Do you think these are insufficient?

The trust chain is made simpler if we validate the Linux-libre result
(i.e. we need only trust the Linux-libre project).  As it is used by a
number of distros, we can hope for any mistake they might do to be
quickly flagged.  If we choose instead to generate our own Linux-libre
releases, doing our own "development", then people need to trust you or
Leo, or whoever bumped the linux-libre package for having done due
diligence (screening new commits for nonfree code).  With all due
respect, it seems easier to put our trust in the Linux-libre project for
this task, as this is their purpose and they have greater exposure.

> Thanks again for this discussion, and for the work you've already done
> to improve our deblobbing.

I'm equally thankful to you for your patience in handling all these
discussions, and for sticking around after all these years, keeping the
flow of kernel and icecat updates (amongst other) coming!

Please see the revised patch which I'll git send-email shortly.

Maxim

[bug#43160] [PATCH v3 1/2] gnu: linux-libre: Compare generated sources against Linux-libre releases.

[Maxim Cournoyer]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=yes, Size: 10474 bytes --]

* gnu/packages/linux.scm (make-linux-libre-source): Rename the UPSTREAM-SOURCE
parameter to LINUX-UPSTREAM-SOURCE.  Add a new LINUX-LIBRE-UPSTREAM-SOURCE
parameter.  Update doc.  Adjust variable names.  Capitalize "Linux" in the
user messages.  Remove empty directories from the generated sources, then
invoke diff between these sources and those of the corresponding Linux-libre
release, unless LINUX-LIBRE-UPSTREAM-SOURCE is #f.
(%upstream-linux-source): Convert the hash as base32 inside the definition, to
simplify its use.
(%upstream-linux-libre-source): New procedure.
(linux-libre-5.8-pristine-source): Add a LIBRE-HASH binding and use it with
%UPSTREAM-LINUX-LIBRE-SOURCE to provide the Linux-libre release origin to the
make-linux-libre-source procedure call.
(linux-libre-5.4-pristine-source): Likewise.
(linux-libre-4.19-pristine-source): Likewise.
(linux-libre-4.14-pristine-source): Likewise.
(linux-libre-4.9-pristine-source): Likewise.
(linux-libre-4.4-pristine-source): Likewise.
---
 gnu/packages/linux.scm | 79 ++++++++++++++++++++++++++++++++----------
 1 file changed, 61 insertions(+), 18 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 72fb3ca49d..1df66330cb 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -221,10 +221,18 @@ from forcing GEXP-PROMISE."
                       #:guile-for-build guile)))
 
 (define (make-linux-libre-source version
-                                 upstream-source
+                                 linux-upstream-source
+                                 linux-libre-upstream-source
                                  deblob-scripts)
   "Return a 'computed' origin that generates a Linux-libre tarball from the
-corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
+corresponding LINUX-UPSTREAM-SOURCE (an origin), using the given
+DEBLOB-SCRIPTS.  The generated Linux-libre source is compared against the
+corresponding LINUX-LIBRE-UPSTREAM-SOURCE upstream release (an origin), to
+ensure correctness.  This comparison is skipped when
+LINUX-LIBRE-UPSTREAM-SOURCE is set to #f.  This can be used in exceptional
+cases where for security reasons an update must be pushed before the
+Linux-libre project could publish a cleaned up tree.  Manual screening of the
+new Linux changes for nonfree code is required when skipping the comparison."
   (match deblob-scripts
     ((deblob-version (? origin? deblob) (? origin? deblob-check))
      (unless (string=? deblob-version (version-major+minor version))
@@ -281,14 +289,14 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
                       (("/bin/sed") (which "sed"))
                       (("/usr/bin/python") (which "python"))))
 
-                  (if (file-is-directory? #+upstream-source)
+                  (if (file-is-directory? #+linux-upstream-source)
                       (begin
-                        (format #t "Copying upstream linux source...~%")
-                        (invoke "cp" "--archive" #+upstream-source dir)
+                        (format #t "Copying upstream Linux source...~%")
+                        (invoke "cp" "--archive" #+linux-upstream-source dir)
                         (invoke "chmod" "--recursive" "u+w" dir))
                       (begin
-                        (format #t "Unpacking upstream linux tarball...~%")
-                        (invoke "tar" "xf" #$upstream-source)
+                        (format #t "Unpacking upstream Linux tarball...~%")
+                        (invoke "tar" "xf" #$linux-upstream-source)
                         (match (scandir "."
                                         (lambda (name)
                                           (and (not (member name '("." "..")))
@@ -315,7 +323,22 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
 
                   (format #t "~%Scanning the generated tarball for blobs...~%")
                   (invoke "/tmp/bin/deblob-check" "--use-awk" "--list-blobs"
-                          #$output))))))))))
+                          #$output)
+
+                  (if #+linux-libre-upstream-source
+                      (begin
+
+                        ;; Git doesn't track empty directories, so remove them
+                        ;; from our local tree for the sake of comparison.
+                        (invoke "find" dir "-type" "d" "-empty" "-delete")
+                        (invoke "diff" "-ur"
+                                dir
+                                #+linux-libre-upstream-source))
+                      (begin
+                        (format #t "~%Skipping comparison with the upstream \
+Linux-libre release...  Ensure new sources have been manually verified \
+against nonfree software.~%")
+                        #t)))))))))))
 
 \f
 ;;;
@@ -344,8 +367,16 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
     (uri (string-append "mirror://kernel.org"
                         "/linux/kernel/v" (version-major version) ".x/"
                         "linux-" version ".tar.xz"))
-    (sha256 hash)))
+    (sha256 (base32 hash))))
 
+(define (%upstream-linux-libre-source version hash)
+  (origin
+    (method git-fetch)
+    (uri (git-reference
+          (url "git://linux-libre.fsfla.org/releases.git")
+          (commit (string-append "sources/v" version "-gnu"))))
+    (file-name (git-file-name "linux-libre-source" version))
+    (sha256 (base32 hash))))
 
 ;; The current "stable" kernel. That is, the most recently released major
 ;; version.
@@ -357,9 +388,11 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
    (base32 "0j6jba5fcddqlb42f95gjl78jisfla4nswqila074gglcrbnl9q7")))
 (define-public linux-libre-5.8-pristine-source
   (let ((version linux-libre-5.8-version)
-        (hash (base32 "0xm901zvvrwsb9k88la6pb65nybi43bygiyz1z68njwsx6ripxik")))
+        (hash "0xm901zvvrwsb9k88la6pb65nybi43bygiyz1z68njwsx6ripxik")
+        (libre-hash "0zjw82xrmlgmjb5w0ar4mhjsn9pf8halwzq6dvv71hmrmskjxbyn"))
    (make-linux-libre-source version
                             (%upstream-linux-source version hash)
+                            (%upstream-linux-libre-source version libre-hash)
                             deblob-scripts-5.8)))
 
 ;; The "longterm" kernels — the older releases with long-term upstream support.
@@ -373,10 +406,12 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
    (base32 "1b3q88i2qfdxyvpi9f7jds0qlb8hfpw87mgia096ax6822c2cmyb")))
 (define-public linux-libre-5.4-pristine-source
   (let ((version linux-libre-5.4-version)
-        (hash (base32 "1vymhl6p7i06gfgpw9iv75bvga5sj5kgv46i1ykqiwv6hj9w5lxr")))
-   (make-linux-libre-source version
-                            (%upstream-linux-source version hash)
-                            deblob-scripts-5.4)))
+        (hash "1vymhl6p7i06gfgpw9iv75bvga5sj5kgv46i1ykqiwv6hj9w5lxr")
+        (libre-hash "150cz1h9cn8klh8dhnbhb9zmxc6pf6x9rj5fa2wv9k7r42lk9kis"))
+    (make-linux-libre-source version
+                             (%upstream-linux-source version hash)
+                             (%upstream-linux-libre-source version libre-hash)
+                             deblob-scripts-5.4)))
 
 (define-public linux-libre-4.19-version "4.19.144")
 (define deblob-scripts-4.19
@@ -386,9 +421,11 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
    (base32 "1jiaw0as1ippkrjdpd52657w5mz9qczg3y2hlra7m9k0xawwiqlf")))
 (define-public linux-libre-4.19-pristine-source
   (let ((version linux-libre-4.19-version)
-        (hash (base32 "0jnj65bdy5y9lcj5zhrn4iaszpww8z41ac66j00l75sd931l1g9k")))
+        (hash "0jnj65bdy5y9lcj5zhrn4iaszpww8z41ac66j00l75sd931l1g9k")
+        (libre-hash "04lijps8qjk3kwsgvkw9plhmy5rxgrp6ld82d96jgjm27s5xd308"))
     (make-linux-libre-source version
                              (%upstream-linux-source version hash)
+                             (%upstream-linux-libre-source version libre-hash)
                              deblob-scripts-4.19)))
 
 (define-public linux-libre-4.14-version "4.14.197")
@@ -399,9 +436,11 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
    (base32 "1qij18inijj6c3ma8hv98yjagnzxdxyn134da9fd23ky8q6hbvky")))
 (define-public linux-libre-4.14-pristine-source
   (let ((version linux-libre-4.14-version)
-        (hash (base32 "029h46yki2hxdbn7afmnf3yar1pnwrpszx76irsa5mf8gnrasyp0")))
+        (hash "029h46yki2hxdbn7afmnf3yar1pnwrpszx76irsa5mf8gnrasyp0")
+        (libre-hash "1hbp1shhhifk3xy8026c466vpfpgll11xx1kawq97llx1pars4hn"))
     (make-linux-libre-source version
                              (%upstream-linux-source version hash)
+                             (%upstream-linux-libre-source version libre-hash)
                              deblob-scripts-4.14)))
 
 (define-public linux-libre-4.9-version "4.9.235")
@@ -412,9 +451,11 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
    (base32 "0fxajshb75siq39lj5h8xvhdj8lcmddkslwlyj65rhlwk6g2r4b2")))
 (define-public linux-libre-4.9-pristine-source
   (let ((version linux-libre-4.9-version)
-        (hash (base32 "1hqcb3zw4546h6x5xy2mywdznha8813lx15mxbgfbvwm4qhsc9g6")))
+        (hash "1hqcb3zw4546h6x5xy2mywdznha8813lx15mxbgfbvwm4qhsc9g6")
+        (libre-hash "0sz73pxdz4kl4fyfvbkm7xzdhzx8x2xajr93mhapc65hssyz3059"))
     (make-linux-libre-source version
                              (%upstream-linux-source version hash)
+                             (%upstream-linux-libre-source version libre-hash)
                              deblob-scripts-4.9)))
 
 (define-public linux-libre-4.4-version "4.4.235")
@@ -425,9 +466,11 @@ corresponding UPSTREAM-SOURCE (an origin), using the given DEBLOB-SCRIPTS."
    (base32 "0hhin1jpfkd6nwrb6xqxjzl3hdxy4pn8a15hy2d3d83yw6pflbsf")))
 (define-public linux-libre-4.4-pristine-source
   (let ((version linux-libre-4.4-version)
-        (hash (base32 "0w5pkv936zb0shjgnpv17gcp5n8f91djznzq54p6j1bl5q2qdyqd")))
+        (hash "0w5pkv936zb0shjgnpv17gcp5n8f91djznzq54p6j1bl5q2qdyqd")
+        (libre-hash "1pydy3cr4malqlr69ksw22nphpydfmpbrfh190ahgym741zdfncg"))
     (make-linux-libre-source version
                              (%upstream-linux-source version hash)
+                             (%upstream-linux-libre-source version libre-hash)
                              deblob-scripts-4.4)))
 
 (define %boot-logo-patch
-- 
2.28.0

[bug#43160] [PATCH v3 2/2] linux-libre: Enable multi-core xz compression during tarball generation.

[Maxim Cournoyer]

* gnu/packages/linux.scm (make-linux-libre-source): Add an NCORES binding, and
use it to configure the number of threads xz should use via the XZ_DEFAULTS
environment variable.
---
 gnu/packages/linux.scm | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 1df66330cb..d6441fa181 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -255,7 +255,8 @@ new Linux changes for nonfree code is required when skipping the comparison."
 
                 (setvbuf (current-output-port) 'line)
 
-                (let ((dir (string-append "linux-" #$version)))
+                (let ((dir (string-append "linux-" #$version))
+                      (ncores (number->string (parallel-job-count))))
 
                   (mkdir "/tmp/bin")
                   (set-path-environment-variable
@@ -289,6 +290,9 @@ new Linux changes for nonfree code is required when skipping the comparison."
                       (("/bin/sed") (which "sed"))
                       (("/usr/bin/python") (which "python"))))
 
+                  ;; This enables xz multi-core compression/decompression.
+                  (setenv "XZ_DEFAULTS" (string-append "--threads=" ncores))
+
                   (if (file-is-directory? #+linux-upstream-source)
                       (begin
                         (format #t "Copying upstream Linux source...~%")
-- 
2.28.0

[bug#43160] [PATCH v3 1/2] gnu: linux-libre: Compare generated sources against Linux-libre releases.

[Mark H Weaver]

I've grown tired of arguing about this.  You have the authority, so do
as you will, and I will take my leave from maintenance of the kernel
packages.  Thanks for asking my opinion, anyway.

      Mark

[bug#43160] [PATCH v3 1/2] gnu: linux-libre: Compare generated sources against Linux-libre releases.

[Maxim Cournoyer]

Hello Mark,

Mark H Weaver <mhw@netris.org> writes:

> I've grown tired of arguing about this.  You have the authority, so do
> as you will, and I will take my leave from maintenance of the kernel
> packages.  Thanks for asking my opinion, anyway.
>
>       Mark

I was hoping this latest modified patch would meet both our goals
(strictly verified for the usual case, with an option to switch to
manual verification of the kernel sources for the exceptional security
quick releases).

Sorry to have worn you out on this.  I'll leave 2 weeks for the issue to
settle, hoping you might reconsider.

Thanks again,

Maxim

[bug#43160] [PATCH v3 1/2] gnu: linux-libre: Compare generated sources against Linux-libre releases.

[Mark H Weaver]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> I've grown tired of arguing about this.  You have the authority, so do
>> as you will, and I will take my leave from maintenance of the kernel
>> packages.  Thanks for asking my opinion, anyway.
>>
>>       Mark
>
> I was hoping this latest modified patch would meet both our goals
> (strictly verified for the usual case, with an option to switch to
> manual verification of the kernel sources for the exceptional security
> quick releases).
>
> Sorry to have worn you out on this.  I'll leave 2 weeks for the issue to
> settle, hoping you might reconsider.

I appreciate that.  I'll attempt another followup in the next few days.

       Mark

[bug#43160] [PATCH v3 1/2] gnu: linux-libre: Compare generated sources against Linux-libre releases.

[Mark H Weaver]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> I was hoping this latest modified patch would meet both our goals
> (strictly verified for the usual case, with an option to switch to
> manual verification of the kernel sources for the exceptional security
> quick releases).
>
> Sorry to have worn you out on this.  I'll leave 2 weeks for the issue to
> settle, hoping you might reconsider.

I'm sorry for not following up on this sooner.  My opinion on this issue
has not changed, but I've run out of energy to continue arguing about
it, and anyway it's probably more important to make the Linux-libre
developers happy.  Do as you think best, and I'll make adjustments on my
private branch as needed.

     Thanks,
       Mark

bug#43160: linux-libre: compare guix-generated sources against upstream releases

[Maxim Cournoyer]

Hi Mark,

Mark H Weaver <mhw@netris.org> writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> I was hoping this latest modified patch would meet both our goals
>> (strictly verified for the usual case, with an option to switch to
>> manual verification of the kernel sources for the exceptional security
>> quick releases).
>>
>> Sorry to have worn you out on this.  I'll leave 2 weeks for the issue to
>> settle, hoping you might reconsider.
>
> I'm sorry for not following up on this sooner.  My opinion on this issue
> has not changed, but I've run out of energy to continue arguing about
> it, and anyway it's probably more important to make the Linux-libre
> developers happy.  Do as you think best, and I'll make adjustments on my
> private branch as needed.

I think there's still value in this series, but due to the already high
build requirements of running the verification script, I don't think
adding more to it is a good idea.

A better idea will be to build straight from the Git Linux-libre
repository, which will lighten the load to build these kernels while
simplifying things a bit.

Closing for now.

-- 
Thanks,
Maxim