Linux-libre 5.8 and beyond

Linux-libre 5.8 and beyond

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1640 bytes --]

Thanks for updating linux-libre to 5.7!

I saw the 5.8 was out, and gave a quick shot at updating it, but it hung
python indefinitely during the deblobbing process. I also tried
switching to python 3 instead of python 2, but it had the same
issue. Apparently this is a known issue:

  https://lists.gnu.org/archive/html/info-gnu/2020-08/msg00001.html

When I tried switching deblob to use gawk instead, it produced the
cleaned tarball, but resulted in syntax errors when building
linux-libre.

So I asked a bit in #linux-libre on freenode and they wondered why we
don't use the git repository instead of running the deblob scripts again
in guix.

One of the issues might be that linux-libre may occasionally remove
releases that accidentally contained non-free code breaking guix's
ability to build old versions. Not sure exactly where guix's balance
between functional package management and software freedom interplays
there.

That said, using their git repository could allow guix to take advantage
of the software heritage as a fallback; though I'm not quite sure how
well that would work with removed versions.

Downloading the git repository of a project as large as linux-libre
every time is probably somewhat expensive. Though the process of
deblobbing in guix is also quite expensive...


There's more debugging to do (and admittedly, wrapping my head around
the deblobbing code in linux.scm is a bit difficult) and the linux-libre
folks are somewhat interested to figure out what exactly is wrong with
the process building on guix.


Not sure how much time I can throw at it, but curious what others think!


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Linux-libre 5.8 and beyond

[Mark H Weaver]

Hi,

Vagrant Cascadian <vagrant@debian.org> wrote:
> Thanks for updating linux-libre to 5.7!

Yes, many thanks to Leo Famulari for taking care of that (large) job.

> I saw the 5.8 was out, and gave a quick shot at updating it, but it hung
> python indefinitely during the deblobbing process. I also tried
> switching to python 3 instead of python 2, but it had the same
> issue. Apparently this is a known issue:
> 
>   https://lists.gnu.org/archive/html/info-gnu/2020-08/msg00001.html

Thanks for bringing this to our attention.  Until the deblobbing issue
is resolved, in the definition of 'linux-libre-5.8-pristine-source', we
could simply replace the call to 'make-linux-libre-source' with an
ordinary 'origin' form that fetches the deblobbed source tarball from
the linux-libre project, using (linux-libre-urls linux-libre-5.8-version)
as the URI.

The bigger issue is that the default configurations will need to be
updated again before 5.7.x reaches end-of-life, which will be quite
soon.  Otherwise we'll need to revert back to 5.4.x in order to get
upstream security updates.

> So I asked a bit in #linux-libre on freenode and they wondered why we
> don't use the git repository instead of running the deblob scripts again
> in guix.
> 
> One of the issues might be that linux-libre may occasionally remove
> releases that accidentally contained non-free code breaking guix's
> ability to build old versions.

Last I checked, the linux-libre project periodically deletes most of its
older tarballs, even if there are no accidents.  This problem came to my
attention while trying to help someone determine which version of
linux-libre introduced a bug on their system.  I was about to suggest
bisecting point versions before realizing that the relevant linux-libre
tarballs had all been deleted.  Moreover, if we had succeeded in finding
the first buggy release, the next step would have been to do a 'git
bisect' to determine the precise commit that introduced the bug.

Other reasons to run the deblob scripts ourselves include:

* It may be useful for users with newer hardware devices, which are not
  yet well supported by the latest stable release, to use an arbitrary
  commit from either Linus' mainline git repository or some other
  subsystem tree.

* It allows us to update to a new point version (which usually includes
  security fixes) more quickly, before the linux-libre project reacts.

* It allows us to avoid trusting the integrity of the systems used by
  the linux-libre project to produce their deblobbed tarballs.

     Regards,
       Mark

Re: Linux-libre 5.8 and beyond

[Mark H Weaver]

Hi Vagrant,

Vagrant Cascadian <vagrant@debian.org> wrote:
> I saw the 5.8 was out, and gave a quick shot at updating it, but it hung
> python indefinitely during the deblobbing process.

I was unable to reproduce this problem.  I simply added version 5.8 in
the usual way, without changing the deblobbing code at all, and the
deblobbing process worked correctly on my Thinkpad X200 (x86_64-linux).

I pushed commit cb97d076491495aa956dbff93679a51cc5708010 to 'master',
which adds the linux-libre@5.8 source and headers packages.  You should
be able to build the deblobbed and patched source with the following
command:

  guix build -S -e '(@ (gnu packages linux) linux-libre-5.8-source)'

Does it work for you?  If so, how does it differ from what you tried
before?

Note that the default kernel configurations for 5.8 still need to be
added.  Leo, would you like to work on that?

     Regards,
       Mark

Re: Linux-libre 5.8 and beyond

[Vagrant Cascadian]

On 2020-08-08, Mark H. Weaver wrote:
> Vagrant Cascadian <vagrant@debian.org> wrote:
>> I saw the 5.8 was out, and gave a quick shot at updating it, but it hung
>> python indefinitely during the deblobbing process.
>
> I was unable to reproduce this problem.  I simply added version 5.8 in
> the usual way, without changing the deblobbing code at all, and the
> deblobbing process worked correctly on my Thinkpad X200 (x86_64-linux).

Curious. At a quick glance it looks like the same hashes for linux and
the deblob and scripts that I used. I encountered the issue both on a
pinebook pro (aarch64) and a ~6 year old i5 (x86_64) laptop, which I
figured were different enough that it was a problem in the code...


> I pushed commit cb97d076491495aa956dbff93679a51cc5708010 to 'master',
> which adds the linux-libre@5.8 source and headers packages.  You should
> be able to build the deblobbed and patched source with the following
> command:
>
>   guix build -S -e '(@ (gnu packages linux) linux-libre-5.8-source)'
>
> Does it work for you?  If so, how does it differ from what you tried
> before?

Will try to test again sometime in the coming days.


live well,
  vagrant

Re: Linux-libre 5.8 and beyond

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1766 bytes --]

On 2020-08-08, Vagrant Cascadian wrote:
> On 2020-08-08, Mark H. Weaver wrote:
>> Vagrant Cascadian <vagrant@debian.org> wrote:
>>> I saw the 5.8 was out, and gave a quick shot at updating it, but it hung
>>> python indefinitely during the deblobbing process.
>>
>> I was unable to reproduce this problem.  I simply added version 5.8 in
>> the usual way, without changing the deblobbing code at all, and the
>> deblobbing process worked correctly on my Thinkpad X200 (x86_64-linux).
>
> Curious. At a quick glance it looks like the same hashes for linux and
> the deblob and scripts that I used.

At a longer glance, it looks like I failed to update the hashes
correctly. The hashes for deblob-check 5.7 and deblob-check 5.8 both
began with "1n" and I must have somehow glazed over the differences and
not updated the local commit.

How guix actually managed to download deblob-check 5.8 from a different
URL and proceed to attempt to use the "old" store item without noticing
the hash was different still remains a mystery to me... I would have
expected it to error out before getting that far.


> I encountered the issue both on a
> pinebook pro (aarch64) and a ~6 year old i5 (x86_64) laptop, which I
> figured were different enough that it was a problem in the code...
>
>
>> I pushed commit cb97d076491495aa956dbff93679a51cc5708010 to 'master',
>> which adds the linux-libre@5.8 source and headers packages.  You should
>> be able to build the deblobbed and patched source with the following
>> command:
>>
>>   guix build -S -e '(@ (gnu packages linux) linux-libre-5.8-source)'
>>
>> Does it work for you?  If so, how does it differ from what you tried
>> before?
>
> Will try to test again sometime in the coming days.

in progress...


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Linux-libre 5.8 and beyond

[Leo Famulari]

On Sat, Aug 08, 2020 at 11:30:40PM -0400, Mark H Weaver wrote:
> Note that the default kernel configurations for 5.8 still need to be
> added.  Leo, would you like to work on that?

I'm planning to do that when the 5.8 kernel becomes the "stable" kernel.
I assume this will happen soon.

Re: Linux-libre 5.8 and beyond

[Mark H Weaver]

Hi Vagrant,

Vagrant Cascadian <vagrant@debian.org> wrote:
> At a longer glance, it looks like I failed to update the hashes
> correctly. The hashes for deblob-check 5.7 and deblob-check 5.8 both
> began with "1n" and I must have somehow glazed over the differences and
> not updated the local commit.

Ah, okay, that makes sense.  I guess you accidentally used version 5.7
of deblob-check on the 5.8 kernel.

Note that although base32 encodes 5 bits per character, the first
character of a base32-encoded sha256 hash can only be 0 or 1, since
there's only 1 bit remaining to encode after the other 255 bits have
been encoded in the last 51 characters.

> How guix actually managed to download deblob-check 5.8 from a different
> URL and proceed to attempt to use the "old" store item without noticing
> the hash was different still remains a mystery to me... I would have
> expected it to error out before getting that far.

If the file name and hash matches a previously downloaded file in your
store, the guix daemon uses that one and skips the download, regardless
of the URL.  That's why no error was reported.  There's no version
number in the file name of the 'deblob-check' file.

    Thanks!
      Mark

bug#42789: Linux-libre 5.8 and beyond

[Leo Famulari]

On Sun, Aug 09, 2020 at 06:17:48PM -0400, Mark H Weaver wrote:
> If the file name and hash matches a previously downloaded file in your
> store, the guix daemon uses that one and skips the download, regardless
> of the URL.  That's why no error was reported.  There's no version
> number in the file name of the 'deblob-check' file.

We should try to make these files include the version number to avoid
this kind of mistake in the future.

I've CC-ed bug-guix so that we don't forget.

bug#42789: Linux-libre 'deblob-check' file-names do not include a version number

[Leo Famulari]

[-- Attachment #1.1: Type: text/plain, Size: 175 bytes --]

I've attached a patch that should prevent this kind of mistake in the
future.

We should queue it up to be pushed with the next kernel updates (on the
kernel-updates branch).

[-- Attachment #1.2: 0001-gnu-Use-a-descriptive-file-name-for-linux-libre-debl.patch --]
[-- Type: text/plain, Size: 1073 bytes --]

From e9ca6405e351baf4356a7300aa252d25056a322c Mon Sep 17 00:00:00 2001
From: Leo Famulari <leo@famulari.name>
Date: Mon, 10 Aug 2020 12:40:49 -0400
Subject: [PATCH] gnu: Use a descriptive file-name for linux-libre
 'deblob-check' scripts.

Fixes <https://bugs.gnu.org/42789>.

* gnu/packages/linux.scm (linux-libre-deblob-scripts): Use file-name for
the deblob-check script.
---
 gnu/packages/linux.scm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index a2d6d384ee..9d553e7772 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -200,6 +200,7 @@ defconfig.  Return the appropriate make target if applicable, otherwise return
           (uri (string-append "https://linux-libre.fsfla.org"
                               "/pub/linux-libre/releases/" version "-gnu/"
                               "deblob-check"))
+          (file-name (string-append "linux-libre-" version "-deblob-check"))
           (sha256 deblob-check-hash))))
 
 (define deblob-scripts-5.8
-- 
2.28.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux-libre 5.8 and beyond

[Bengt Richter]

On +2020-08-09 18:17:48 -0400, Mark H Weaver wrote:
> 
> Note that although base32 encodes 5 bits per character, the first
> character of a base32-encoded sha256 hash can only be 0 or 1, since
> there's only 1 bit remaining to encode after the other 255 bits have
> been encoded in the last 51 characters.
> 
UIAM, that's only true for the nix flavor (which is default for guix hash, I think)
of base32. Again UIAM, the nix view of a 256-bit sha256sum hash is little-endian,
and shifts 5 bits out the bottom, as if with euclidean/ 32, and so winds up with
the 1 or 0 last, at the top.

I think all the others base32's shift 5 bits at a time from the big end, and
could have the full range 0-31 for the top digit, however translated to glyphs.
Which also means the last value on the right is a 1 or 0 in the top bit, valued 16 or 0.

Of course, different length digests may produce other remainder end values.

BTW, how did nix get such a weird alphabet for 0-31 ? Watermarking themselves? :)

-- 
Regards,
Bengt Richter

Re: Linux-libre 5.8 and beyond

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

Bengt,

Bengt Richter 写道：
> BTW, how did nix get such a weird alphabet for 0-31 ? 
> Watermarking themselves? :)

This question probably deserves a Nix FAQ entry by now, if there 
isn't one already :-)

  “This is to reduce the possibility that hash representations 
  contain character sequences that are potentially offensive to 
  someusers (a known possibility with alphanumeric representations 
  of numbers).”
    -- https://edolstra.github.io/pubs/phd-thesis.pdf

Excercises for the puerile reader are obvious.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#42789: Linux-libre 5.8 and beyond

[Mark H Weaver]

Hi Leo,

Leo Famulari <leo@famulari.name> wrote:
> On Sun, Aug 09, 2020 at 06:17:48PM -0400, Mark H Weaver wrote:
>> If the file name and hash matches a previously downloaded file in your
>> store, the guix daemon uses that one and skips the download, regardless
>> of the URL.  That's why no error was reported.  There's no version
>> number in the file name of the 'deblob-check' file.
>
> We should try to make these files include the version number to avoid
> this kind of mistake in the future.

That's a good idea.  In a later message you posted a proposed patch:

> From e9ca6405e351baf4356a7300aa252d25056a322c Mon Sep 17 00:00:00 2001
> From: Leo Famulari <leo@famulari.name>
> Date: Mon, 10 Aug 2020 12:40:49 -0400
> Subject: [PATCH] gnu: Use a descriptive file-name for linux-libre
>  'deblob-check' scripts.
> 
> Fixes <https://bugs.gnu.org/42789>.
> 
> * gnu/packages/linux.scm (linux-libre-deblob-scripts): Use file-name for
> the deblob-check script.
> ---
>  gnu/packages/linux.scm | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
> index a2d6d384ee..9d553e7772 100644
> --- a/gnu/packages/linux.scm
> +++ b/gnu/packages/linux.scm
> @@ -200,6 +200,7 @@ defconfig.  Return the appropriate make target if applicable, otherwise return
>            (uri (string-append "https://linux-libre.fsfla.org"
>                                "/pub/linux-libre/releases/" version "-gnu/"
>                                "deblob-check"))
> +          (file-name (string-append "linux-libre-" version "-deblob-check"))
>            (sha256 deblob-check-hash))))
>  
>  (define deblob-scripts-5.8

If we're going to prefix "linux-libre-" to the name, which I agree is a
good idea, maybe we should add the same prefix to the other 'deblob'
script, for consistency.  Also for consistency, I think the version
number should be at the end, after "deblob-check", as is the case for
the other deblob script.

There's also the question of whether the micro version number should be
included in the file name.  In practice, these deblob scripts almost
never change from one micro version to the next.  Also, I suspect
(although I've not yet confirmed it) that these deblob scripts likely
work for older kernels in the same stable series.

For those reasons, at present the micro version number appears only in
the URLs, and not in either the file names or in the version number as
recorded in the first element of the triplet returned by
'linux-libre-deblob-scripts'.  I'd personally be inclined to keep it
that way, although I don't feel strongly about it.

What do you think?

      Thanks,
        Mark

bug#42789: Linux-libre 5.8 and beyond

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1278 bytes --]

On Mon, Aug 10, 2020 at 11:46:34PM -0400, Mark H Weaver wrote:
> If we're going to prefix "linux-libre-" to the name, which I agree is a
> good idea, maybe we should add the same prefix to the other 'deblob'
> script, for consistency.  Also for consistency, I think the version
> number should be at the end, after "deblob-check", as is the case for
> the other deblob script.
>
> There's also the question of whether the micro version number should be
> included in the file name.  In practice, these deblob scripts almost
> never change from one micro version to the next.  Also, I suspect
> (although I've not yet confirmed it) that these deblob scripts likely
> work for older kernels in the same stable series.
> 
> For those reasons, at present the micro version number appears only in
> the URLs, and not in either the file names or in the version number as
> recorded in the first element of the triplet returned by
> 'linux-libre-deblob-scripts'.  I'd personally be inclined to keep it
> that way, although I don't feel strongly about it.

Good ideas. I've taken them into account on my local branch while I test
building the final tarballs. I'll push to the kernel-updates branch once
I am satisfied with the result.

Thanks for your feedback!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#42789: Linux-libre 5.8 and beyond

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

On Tue, Aug 11, 2020 at 06:46:01PM -0400, Leo Famulari wrote:
> Good ideas. I've taken them into account on my local branch while I test
> building the final tarballs. I'll push to the kernel-updates branch once
> I am satisfied with the result.

I've pushed an updated patch to the kernel-updates branch:

https://git.savannah.gnu.org/cgit/guix.git/commit/?h=kernel-updates&id=387446c93261868bd4c4188687b508aa0a2da7e8

It will be built on the build farm here, based on commit b1f5568b1:

https://ci.guix.gnu.org/jobset/kernel-updates

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#42789: Linux-libre 5.8 and beyond

[Mark H Weaver]

Leo Famulari <leo@famulari.name> wrote:

> I've pushed an updated patch to the kernel-updates branch:
> 
> https://git.savannah.gnu.org/cgit/guix.git/commit/?h=kernel-updates&id=387446c93261868bd4c4188687b508aa0a2da7e8

Looks good to me.  Thank you!

       Mark

bug#42789: Linux-libre 5.8 and beyond

[Leo Famulari]

On Wed, Aug 12, 2020 at 05:34:21PM -0400, Mark H Weaver wrote:
> Leo Famulari <leo@famulari.name> wrote:
> 
> > I've pushed an updated patch to the kernel-updates branch:
> > 
> > https://git.savannah.gnu.org/cgit/guix.git/commit/?h=kernel-updates&id=387446c93261868bd4c4188687b508aa0a2da7e8
> 
> Looks good to me.  Thank you!

Pushed as 1bc275034fe8070c720ca341db2d9588480aa044

Re: Linux-libre 5.8 and beyond

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 609 bytes --]

Hi,

I have started handling major updates of linux-libre for Guix, starting
with version 5.7 (collaborators are invited!).

I didn't read this discussion because it's quite long and I don't
perceive that anything needs to change with how we package linux-libre.
It has worked well for several years.

If there are concrete problems to report or changes to request, please
let us know by opening a bug ticket at <bug-guix@gnu.org>, or by sending
a patch to <guix-patches@gnu.org>.

Otherwise, I don't think this conversation is very productive and I
request that it either stops or moves somewhere else.

Leo

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux-libre 5.8 and beyond

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1189 bytes --]

On Tue, Aug 25, 2020 at 05:01:07PM -0400, Leo Famulari wrote:
> If there are concrete problems to report or changes to request, please
> let us know by opening a bug ticket at <bug-guix@gnu.org>, or by sending
> a patch to <guix-patches@gnu.org>.

I'd like to explain more clearly what I meant by my last message.

First, it's important to remember that we share a common goal: the
creation of freely licensed computing systems. The people involved in
the Guix and linux-libre projects have worked hard for years towards
this goal — most of us as volunteers.

For many of us, it's natural to identify with our work, and when people
suggest changes or offer criticism that we think is incorrect or misses
the mark somehow, it's also natural to have negative feelings in
response.

But, we must remember that the other party may not understand the
context of their suggestion deeply enough to know why it should not be
implemented. There are technical *and* social contexts at play, and they
are rarely understood until one has been working on the project for
quite a while.

We should be generous and charitable when interpreting each other's
messages and goals.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux-libre 5.8 and beyond

[Katherine Cox-Buday]

Leo Famulari <leo@famulari.name> writes:

> On Tue, Aug 25, 2020 at 05:01:07PM -0400, Leo Famulari wrote:
> But, we must remember that the other party may not understand the
> context of their suggestion deeply enough to know why it should not be
> implemented. There are technical *and* social contexts at play, and they
> are rarely understood until one has been working on the project for
> quite a while.
>
> We should be generous and charitable when interpreting each other's
> messages and goals.

What a great outlook from a maintainer of an open source project. I have
read nothing else in this thread, but this is wonderful.

-- 
Katherine