Re: Linux-libre git repository

[Vagrant Cascadian <vagrant@reproducible-builds.org>]

[-- Attachment #1: Type: text/plain, Size: 4903 bytes --]

On 2020-08-12, Mark H Weaver wrote:
> Mark H Weaver <mhw@netris.org> wrote:
>>> the linux-libre project periodically deletes most of its older
>>> tarballs, even if there are no accidents.
>
> Jason Self <jason@bluehome.net> responded:
>> Just FYI that git://linux-libre.fsfla.org/releases.git was created
>> mainly to solve that problem. Versions are now pretty much permanent.
>
> That's helpful, thanks.  I didn't know about this.  Out of curiosity, is
> this git repository advertised anywhere?  I wasn't able to easily find
> it on <https://www.fsfla.org/ikiwiki/selibre/linux-libre/>, but I didn't
> look carefully, perhaps I missed it.

News item for 2020-05-31 mentions it, but clearly it should be more
prominently displayed or documented.


> One question: Would it solve the problem that I mentioned in my earlier
> email, namely the problem of how to determine which precise commit
> introduced a regression between two stable kernel releases?  If not, I
> think that justifies the machinery that Guix includes to do the
> deblobbing itself.

The granularity appears to be at the level of released tags. I see tags
with one commit per release, with an independent history from previous
versions; I *think* git bisect wouldn't work without some manual
fiddling and you'd have to manually bisect based on version.


I tried a quick experiment using the linux-libre git repository to build
a package for arm64 in gnu/packages/linux.scm:

(define-public linux-libre-fsfla-git-arm64-generic
  (let* ((version "5.8.1-gnu")
         (source
          (origin
            (method git-fetch)
            (uri (git-reference
                  (url "git://linux-libre.fsfla.org/releases.git")
                  (commit (string-append "sources/v" version))))
            (file-name (git-file-name "linux-libre-fsfla-git" version))
            (sha256
             (base32
              "05v2l4r34nbkv6wpgrzydlb0fkpswpvzdya9vx30wap3n9a9wp6n"))
           (patches
            (list %boot-logo-patch
                  %linux-libre-arm-export-__sync_icache_dcache-patch)))))
    (make-linux-libre*
     version
     source
     '("aarch64-linux")
     #:defconfig "defconfig"
     #:extra-version "fsfla-git-arm64-generic")))

The source checkout was quite slow to download, and took up ~1GB in the
store once completed. I'm not sure how guix's git origin works exactly;
if it downloads the entire git history even to perform a shallow
checkout of a single commit, and then throws out the git history? It did
appear to be calling git with flags to perform a shallow checkout.

It certainly was slower than downloading a compressed tarball. The
de-duplication of /gnu/store might still be beneficial if you have
significantly more than ~10 versions in /gnu/store, as not every file
changes with every release, but overall using compressed tarballs seems
to be faster to download and extract even on a slow machine.


This partly points to challenges with guix's handling of git
repositories, exacerbated by larger git repositories. It would be more
viable if there was some way to cache git results such as running "git
clone --bare ~/.cache/guix/..." if not present, and "git fetch origin"
if present and then populating the store from cached git repository,
much like done with "guix pull" ... Surely this has been brought up
before? Maybe this breaks the purity of guix's functional paradigm, but
arguably no more than a caching http proxy really.


It is also possible to retrieve tarballs directly from linux-libre git
tags, though I know at least projects hosted on github this does
occasionally result in non-identical tarballs. Not sure what factors
might trigger this, other than changing tags, but possibly different git
versions, tar versions and flags, and compression tool versions and
optimizations could be a factor. Reproducible builds has documented some
potential causes:

  https://reproducible-builds.org/docs/archives/


There are also the released linux-libre tarballs, though that may have
the persistence issue previously mentioned. The code to do so is still
present in guix, I made a package using:

(define-public linux-libre-fsfla-arm64-generic
  (make-linux-libre "5.8.1"
                    "1v7glmvz3laj1awh5zrqclp2pzfs0cjf6y3n6v97j7z901s1vlxd"
                   '("aarch64-linux")
                   #:defconfig "defconfig"
                   #:extra-version "fsfla-arm64-generic"))

After patching the make-linux-libre call to also include a patch needed
for newer versions:

-                           (patches (list %boot-logo-patch)))
+                           (patches (list %boot-logo-patch
+                                         %linux-libre-arm-export-__sync_icache_dcache-patch)))


Not sure why that patch isn't upstream; Debian has been carrying it for
some years now... and my guix build failed to build without it on
aarch64/arm64.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]