Linux-libre source code will be taken offline

Linux-libre source code will be taken offline

[Leo Famulari]

According to the linux-libre team in the #gnu-linux-libre IRC channel on
Libera.chat, all releases of linux-libre before 4.4.282, 4.9.281,
4.14.245, 4.19.205, 5.4.143, 5.10.61, and 5.13.13 will be deleted from
their servers, and their Git repo is also going to be rewritten to
remove them.

So, if anybody wanted to build old versions of Guix System (or
linux-libre based on old revisions of guix.git), some third party should
preserve and mirror those files and the Git repo:

https://linux-libre.fsfla.org/pub/linux-libre/releases/
git://linux-libre.fsfla.org/releases.git

The reason for the removal is that some nonfree things were accidentally
not removed by the previous releases of the deblob scripts.

Re: Linux-libre source code will be taken offline

[Jason Self]

[-- Attachment #1: Type: text/plain, Size: 1315 bytes --]

For some clarity on the situation:
http://www.fsfla.org/pipermail/linux-libre/2021-August/003439.html

The scripts are not being removed and my understanding is that Guix
only uses the scripts anyway.

> and their Git repo is also going to be rewritten to remove them.

The tags for the kernel source code would be removed (but not for the
cleanup scripts) but my understanding is that this doesn't requite
rewriting history.

The release tarballs are already gone but the tags corresponding to the
libre kernel versions continue to exist in releases.git. My
understanding is that the desire is to eventually remove those
although I'm not sure about the timing but certainly long enough into
the future to allow time to adapt.

Also, the cleanup scripts will continue to exist in git as well, in both
old and new versions, even after the tags for the corresponding libre
kernel release have been deleted, so an interested person would still
be able to recreate the appropriate source code even after tag deletion
has happened. Given that they result in kernel source code with known
freedom problems it seems very undesirable to use the old versions
though.

This could be an example that it's desirable to use releases.git to
obtained needed pieces, whether scripts or the source code.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux-libre source code will be taken offline

[Leo Famulari]

On Sat, Sep 04, 2021 at 01:32:16PM -0700, Jason Self wrote:
> The scripts are not being removed and my understanding is that Guix
> only uses the scripts anyway.

Okay, that's great. We do use the scripts.

Re: Linux-libre source code will be taken offline

[Mark H Weaver]

Leo Famulari <leo@famulari.name> writes:

> On Sat, Sep 04, 2021 at 01:32:16PM -0700, Jason Self wrote:
>> The scripts are not being removed and my understanding is that Guix
>> only uses the scripts anyway.
>
> Okay, that's great. We do use the scripts.

Unfortunately, the older deblobbing scripts have now been removed from
the HTTPS URLs that our linux-libre build recipes fetched them from, so
I guess there will now be difficulties for users trying to reproduce any
Guix system more than about a month old.

If we wish to preserve Guix users' ability to reproduce older systems,
we will need an 'origin' to fetch the Linux-libre deblob scripts from
that has a policy of retaining older releases, unchanged and at a fixed
location.  Apparently the HTTPS URLs at <https://linux-libre.fsfla.org>
that we currently use are not suitable for that purpose.

The Linux-libre git repository *might* be suitable, but I haven't yet
seen a commitment from the Linux-libre project that they will retain the
tags for older flawed deblobbing scripts in their repository going
forward.  Without tags protecting them, I guess the commits could be
deleted by 'git-gc' even if we referenced them directly by their commit
hashes.

Perhaps the SWH fallback is the answer we need, if they are archiving
the Linux-libre git repository.  Does anyone know if they are?

Thoughts?

     Thanks,
       Mark

-------------------- Start of forwarded message --------------------
From: Alexandre Oliva <lxoliva@fsfla.org>
To: linux-libre@fsfla.org, info-gnu@gnu.org
Subject: GNU Linux-libre's nonfree releases removed
Date: Mon, 27 Sep 2021 14:08:01 -0300

https://linux-libre.fsfla.org/#retired-2021-09-27
2021-09-27 - Removing obsolete releases and repositories

We celebrate the 38th anniversary of the GNU Project and of the Free
Software movement by removing past releases found to contain nonfree
software.  Their cleaning-up scripts remain available from the git
repository, and from releases/old/gen6.

We're also removing long-obsolete repositories that still contained
builds of those and of even earlier releases.  Instead of freed-ebian
and planet, we recommend Freesh. Instead of rt, we recommend LibeRTy.  A
freeloong replacement might be added to Freesh if there's enough demand.

Freed-ora repositories for recent Fedora releases remain available, but
we're still looking for new maintainers for Freed-ora.

-- 
Alexandre Oliva, happy hacker                https://FSFLA.org/blogs/lxo/
   Free Software Activist                       GNU Toolchain Engineer
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about <https://stallmansupport.org>

-- 
If you have a working or partly working program that you'd like
to offer to the GNU project as a GNU package,
see https://www.gnu.org/help/evaluation.html.
-------------------- End of forwarded message --------------------

-- 
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about <https://stallmansupport.org>.

Re: Linux-libre source code will be taken offline

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1791 bytes --]

On 2021-09-27, Mark H Weaver wrote:
> Leo Famulari <leo@famulari.name> writes:
>
>> On Sat, Sep 04, 2021 at 01:32:16PM -0700, Jason Self wrote:
>>> The scripts are not being removed and my understanding is that Guix
>>> only uses the scripts anyway.
>>
>> Okay, that's great. We do use the scripts.
>
> Unfortunately, the older deblobbing scripts have now been removed from
> the HTTPS URLs that our linux-libre build recipes fetched them from, so
> I guess there will now be difficulties for users trying to reproduce any
> Guix system more than about a month old.
>
> If we wish to preserve Guix users' ability to reproduce older systems,
> we will need an 'origin' to fetch the Linux-libre deblob scripts from
> that has a policy of retaining older releases, unchanged and at a fixed
> location.  Apparently the HTTPS URLs at <https://linux-libre.fsfla.org>
> that we currently use are not suitable for that purpose.
>
> The Linux-libre git repository *might* be suitable, but I haven't yet
> seen a commitment from the Linux-libre project that they will retain the
> tags for older flawed deblobbing scripts in their repository going
> forward.  Without tags protecting them, I guess the commits could be
> deleted by 'git-gc' even if we referenced them directly by their commit
> hashes.
>
> Perhaps the SWH fallback is the answer we need, if they are archiving
> the Linux-libre git repository.  Does anyone know if they are?

The most promising two I found were:

  https://archive.softwareheritage.org/browse/origin/directory/?origin_url=git://linux-libre.fsfla.org/releases.git
  https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://jxself.org/git/linux-libre.git

Not sure exactly how Software Heritage handles rebased branches...

live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Linux-libre source code will be taken offline

[Ludovic Courtès]

Hi,

Vagrant Cascadian <vagrant@debian.org> skribis:

> Not sure exactly how Software Heritage handles rebased branches...

It keeps the history of the history, so to speak.  A “snapshot” in SWH
parlance contains the branches as they existed at one point in time.

Ludo’.

Re: Linux-libre source code will be taken offline

[Leo Famulari]

On Mon, Sep 27, 2021 at 06:30:21PM -0400, Mark H Weaver wrote:
> If we wish to preserve Guix users' ability to reproduce older systems,
> we will need an 'origin' to fetch the Linux-libre deblob scripts from
> that has a policy of retaining older releases, unchanged and at a fixed
> location.  Apparently the HTTPS URLs at <https://linux-libre.fsfla.org>
> that we currently use are not suitable for that purpose.

I didn't check that the files are bit-identical, but my understanding is
that the "old" revisions of the deblobbing scripts are available here:

https://linux-libre.fsfla.org/pub/linux-libre/releases/old/

Of course, adding this to the list of URIs in linux-libre-deblob-scripts
won't help users of old Guix revisions... Software Heritage and other
archives that support content-addressed lookups are the only solution
for that.

Re: Linux-libre source code will be taken offline

[zimoun]

Hi,

On Tue, 28 Sept 2021 at 01:34, Leo Famulari <leo@famulari.name> wrote:

> Of course, adding this to the list of URIs in linux-libre-deblob-scripts
> won't help users of old Guix revisions... Software Heritage and other
> archives that support content-addressed lookups are the only solution
> for that.

I do not know how Software Heritage (SWH) handles this case.
Especially for these deblob scripts.  For informatio, the current
status* about SWH is:

  - all the git-fetch packages are "saved" by running manually "guix
lint -c archival" or click to the Save Button on SWH webpage. :-)
  - (almost) all the url-fetch are ingested by SWH using the
guix.gnu.org/sources.json

The content-adressed lookup works fine for Git repo (from channel to
package).  However, it is not ready yet for tarballs.  In short, at
package time, we have a checksum which is content+metadata; then SWH
archives only the content and drops the metadata; and this content is
content-addressed using SWH-ID.  At fallback time, the knowlegde of
checksum does not guarantee to be able to lookup for the content.  And
even if it happens, SWH does rebuild the exact same tarball (because
of the metadata drops), i.e., there is a high probably to have a
checksum mismatch somehow.

Therefore, here it is the aime of Disarchive.  It somehow builds this
map between the checksum and the SWH content-addressed content.

Bricks are there but missing glue. :-)

*current status about SWH: if I have not missed a recent feature. ;-)


An improvement is done by the patch to add these computed origins to
sources.json; see here:

<http://issues.guix.gnu.org/issue/50515>

and this patch is blocked by missing comment on this one:
<http://issues.guix.gnu.org/issue/50620>.

However, if now upstream removed the material, bah it is another story
to save this very same material to SWH. :-)

All the best,
simon

Re: Linux-libre source code will be taken offline

[zimoun]

(Sorry, for the typos.)

On Tue, 28 Sept 2021 at 02:13, zimoun <zimon.toutoune@gmail.com> wrote:

> even if it happens, SWH does rebuild the exact same tarball (because

SWH does *NOT* rebuild the exact same tarball.

Re: Linux-libre source code will be taken offline

[Jason Self]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]

On Mon, 27 Sep 2021 19:30:29 -0400
Leo Famulari <leo@famulari.name> wrote:

> I didn't check that the files are bit-identical, but my understanding
> is that the "old" revisions of the deblobbing scripts are available
> here:
> 
> https://linux-libre.fsfla.org/pub/linux-libre/releases/old/

Yes. In gen6. They have been moved, not deleted.

The versioning and locations in terms of gnuN and genN are knowable and
predictable in advance. I wonder if there is, or could be made, a way to
leverage that so that future moving of files can be done without
causing problems, as long as the files themselves remain otherwise
identical. As an example, the current cleanup scripts might be found in
old/gen7 in the future. Although using git would probably be a better
choice as it would seem to eliminate URL hunting.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux-libre source code will be taken offline

[zimoun]

Hi,

On Mon, 27 Sep 2021 at 17:46, Jason Self <jself@gnu.org> wrote:

>> https://linux-libre.fsfla.org/pub/linux-libre/releases/old/
>
> Yes. In gen6. They have been moved, not deleted.
>
> The versioning and locations in terms of gnuN and genN are knowable and
> predictable in advance. I wonder if there is, or could be made, a way to
> leverage that so that future moving of files can be done without
> causing problems, as long as the files themselves remain otherwise
> identical. As an example, the current cleanup scripts might be found in
> old/gen7 in the future. Although using git would probably be a better
> choice as it would seem to eliminate URL hunting.

Guix has the availability to transparently build any old version using
“guix time-machine”, i.e.,

  guix time-machine --commit=0c7c84407d65f3d03ad1fe3984ae4d524992f498 \
       -- build linux-libre

should build the Linux (libre) kernel as it was on 2020, 25th May.

If the user allow substitutes, then the necessary materials is fetch
from machines hosted in Berlin and maintain by Guix folk.

However, if the user does not allow substitutes, then the source are
first fetched from upstream.  Here several cases of origin.  Upstream is
still up, everything is fine.  Upstream disappeared in the meantime, it
depends on the “type” of the origin and the core issue is the mapping
between the information at package time (e.g., 2020, 25th May) and the
servers providing a fallback at request time for this missing source.

When the upstream source is a Git repo, this map is a simple
contend-addressed lookup by a (almost) straightforward resolver.

When the upstream source is not Git repo, this map becomes harder and
requires – in addition to a fallback server – an external resolver:
something that maps from the information at package time (2020, 25th
May) to the fallback server.

If the package linux-libre defined on 2020, 25th May (written on stone)
points to an URL source which disappears, this Guix time-machine feature
becomes doomed because URL is a really bad contend-addressed system as
all the broken internet shows us.

For sure, the infrastructure needs to evolve for a better future; easier
maintainability for instance.  However, please consider the archivist
point of view and help to not break the past. :-)


Cheers,
simon

Re: Linux-libre source code will be taken offline

[Jason Self]

[-- Attachment #1: Type: text/plain, Size: 3660 bytes --]

On Tue, 28 Sep 2021 10:43:20 +0200
zimoun <zimon.toutoune@gmail.com> wrote:

> Hi,
> 
> On Mon, 27 Sep 2021 at 17:46, Jason Self <jself@gnu.org> wrote:
> 
>  [...]  
> >
> > Yes. In gen6. They have been moved, not deleted.
> >
> > The versioning and locations in terms of gnuN and genN are knowable
> > and predictable in advance. I wonder if there is, or could be made,
> > a way to leverage that so that future moving of files can be done
> > without causing problems, as long as the files themselves remain
> > otherwise identical. As an example, the current cleanup scripts
> > might be found in old/gen7 in the future. Although using git would
> > probably be a better choice as it would seem to eliminate URL
> > hunting.  
> 
> Guix has the availability to transparently build any old version using
> “guix time-machine”, i.e.,
> 
>   guix time-machine --commit=0c7c84407d65f3d03ad1fe3984ae4d524992f498
> \ -- build linux-libre
> 
> should build the Linux (libre) kernel as it was on 2020, 25th May.
> 
> If the user allow substitutes, then the necessary materials is fetch
> from machines hosted in Berlin and maintain by Guix folk.
> 
> However, if the user does not allow substitutes, then the source are
> first fetched from upstream.  Here several cases of origin.  Upstream
> is still up, everything is fine.  Upstream disappeared in the
> meantime, it depends on the “type” of the origin and the core issue
> is the mapping between the information at package time (e.g., 2020,
> 25th May) and the servers providing a fallback at request time for
> this missing source.
> 
> When the upstream source is a Git repo, this map is a simple
> contend-addressed lookup by a (almost) straightforward resolver.
> 
> When the upstream source is not Git repo, this map becomes harder and
> requires – in addition to a fallback server – an external resolver:
> something that maps from the information at package time (2020, 25th
> May) to the fallback server.
> 
> If the package linux-libre defined on 2020, 25th May (written on
> stone) points to an URL source which disappears, this Guix
> time-machine feature becomes doomed because URL is a really bad
> contend-addressed system as all the broken internet shows us.
> 
> For sure, the infrastructure needs to evolve for a better future;
> easier maintainability for instance.  However, please consider the
> archivist point of view and help to not break the past. :-)

It's not really breaking the past if this is how the past worked in
reality: That previous generations of scripts are moved to old/genN,
but more of Guix's representation of how the past worked which says
that they not move, which doesn't reflect the actual reality of the
past. The two don't seem equivalent.

It seems that Guix can handle multiple download locations already,
either from the main location or from others so why is the old/gen7
location not already in the kernel build recipe? If a new freedom
problem were found that resulted in the need to come up with an 8th
generation, the current ones will be findable in old/gen7. Is Guix
build machinery currently aware of that and ready to check old/gen7
now for whenever that future move happens? If not, then this would seem
to create future breakage when that happens. This move is 100% knowable
and predictable in advance so why not have it ready for now and put
old/gen7 into the recipe for the kernel, even if it's just an additional
hardcoded URL and not something dynamically computed? If not, using git
would seem to be a better choice. I'm not sure why it's not used
already.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux-libre source code will be taken offline

[Jason Self]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

Granted that old/gen7 is not currently a valid URL but we can know
that, 5 or 10 years from now, when Linux-libre has moved on to the 8th,
9th or even 10th generation, the 7th generation scripts will exist
there. If Guix can begin checking those additional locations now then,
in the future once the current Guix version becomes an old version,
hopefully it can transparently handle that transition without issue. Or
just use git. :)

Even this method is just a band-aid though and has a different set of
challenges for the long-term. I'd be happy to discuss that with whoever
is interested.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux-libre source code will be taken offline

[zimoun]

Hi,

On Tue, 28 Sept 2021 at 16:32, Jason Self <jself@gnu.org> wrote:
>
> Granted that old/gen7 is not currently a valid URL but we can know
> that, 5 or 10 years from now, when Linux-libre has moved on to the 8th,
> 9th or even 10th generation, the 7th generation scripts will exist
> there. If Guix can begin checking those additional locations now then,
> in the future once the current Guix version becomes an old version,
> hopefully it can transparently handle that transition without issue. Or
> just use git. :)

I am not familiar with the Linux Libre scripts.  Are these scripts
already under a Git repo?

The method you are proposing seems awkward; as you say, old/gen7 is
not currently a valid URL.  And you are proposing that you set in
stone now this URL expecting that maybe it will be valid in the
future.  Ah future cannot be predicted. ;-)  Who knows that this old/
folder will not be renamed as 'not-fully-free' or whatever.

Working with URL limits the time-travel because it is a really poor
mechanism (and not robust at all) for content-address something, IMHO.

Well, maybe I miss a point or something.

Cheers,
simon

Re: Linux-libre source code will be taken offline

[Jason Self]

[-- Attachment #1: Type: text/plain, Size: 990 bytes --]

On Tue, 28 Sep 2021 19:11:58 +0200
zimoun <zimon.toutoune@gmail.com> wrote:

> The method you are proposing seems awkward; as you say, old/gen7 is
> not currently a valid URL.  And you are proposing that you set in
> stone now this URL expecting that maybe it will be valid in the
> future.  Ah future cannot be predicted. ;-) 

On the contrary: The file versioning and locations are 100% knowable
and predictable in advance. This has been the point I've been trying to
get across this entire time.

> I am not familiar with the Linux Libre scripts.  Are these scripts
> already under a Git repo?

Yes, git://linux-libre.fsfla.org/releases.git which carries tagged
releases, scripts, and logs. As you can see it goes all the way back to
2.6.21.

The primary motivation for the creation of this git repository was
actually for Guix, as a solution to the feedback from the tarballs
being removed due to the lack of disk space, but it appears that it is
not being used.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux-libre source code will be taken offline

[zimoun]

Hi,

On Tue, 28 Sept 2021 at 19:52, Jason Self <jself@gnu.org> wrote:

> Yes, git://linux-libre.fsfla.org/releases.git which carries tagged
> releases, scripts, and logs. As you can see it goes all the way back to
> 2.6.21.

Thanks.

Does it make sense to switch from 'url-fetch' to 'git-fetch' in
linux-libre-deblob-scripts?  Modulo what the Vagrant's remark.

--8<---------------cut here---------------start------------->8---
(define (linux-libre-deblob-scripts version gnu-revision
                                    deblob-hash
                                    deblob-check-hash)
  (list (version-major+minor version)
        (origin
          (method url-fetch)
          (uri (string-append "https://linux-libre.fsfla.org"
                              "/pub/linux-libre/releases/" version "-"
gnu-revision "/"
                              "deblob-" (version-major+minor version)))
          (file-name (string-append "linux-libre-deblob-"
                                    version "-" gnu-revision))
          (sha256 deblob-hash))
        (origin
          (method url-fetch)
          (uri (string-append "https://linux-libre.fsfla.org"
                              "/pub/linux-libre/releases/" version "-"
gnu-revision "/"
                              "deblob-check"))
          (file-name (string-append "linux-libre-deblob-check-"
version "-" gnu-revision))
          (sha256 deblob-check-hash))))
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

Re: Linux-libre source code will be taken offline

[Maxim Cournoyer]

Hi Simon,

zimoun <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Tue, 28 Sept 2021 at 19:52, Jason Self <jself@gnu.org> wrote:
>
>> Yes, git://linux-libre.fsfla.org/releases.git which carries tagged
>> releases, scripts, and logs. As you can see it goes all the way back to
>> 2.6.21.
>
> Thanks.
>
> Does it make sense to switch from 'url-fetch' to 'git-fetch' in
> linux-libre-deblob-scripts?  Modulo what the Vagrant's remark.

To me, yes!

Heck, it was discussed before between maintainers to switch to the Git
repository directly to make things simpler, lighter and error proof, so
we could fetch from the linux-libre repository directly and streamline
things.

So if you are motivation to work on that, please forge ahead!

Thanks,

Maxim

Re: Linux-libre source code will be taken offline

[zimoun]

Hi Maxim,

On Fri, 8 Oct 2021 at 03:58, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> Heck, it was discussed before between maintainers to switch to the Git
> repository directly to make things simpler, lighter and error proof, so
> we could fetch from the linux-libre repository directly and streamline
> things.
>
> So if you are motivation to work on that, please forge ahead!

Do not hold your breath. :-)
I have some pieces on my plate so I will give a look later of no one
beats me. :-)

Cheers,
simon

Re: Linux-libre source code will be taken offline

[Ludovic Courtès]

Hi!

Leo Famulari <leo@famulari.name> skribis:

> According to the linux-libre team in the #gnu-linux-libre IRC channel on
> Libera.chat, all releases of linux-libre before 4.4.282, 4.9.281,
> 4.14.245, 4.19.205, 5.4.143, 5.10.61, and 5.13.13 will be deleted from
> their servers, and their Git repo is also going to be rewritten to
> remove them.
>
> So, if anybody wanted to build old versions of Guix System (or
> linux-libre based on old revisions of guix.git), some third party should
> preserve and mirror those files and the Git repo:
>
> https://linux-libre.fsfla.org/pub/linux-libre/releases/
> git://linux-libre.fsfla.org/releases.git

At the time you wrote this message (Sep. 4), I and others who were
present on IRC downloaded the tarballs to be on the safe side, so thanks
for the heads-up!

Now we have to see what’s available on berlin & co., and the extent to
which SWH can help with this situation.  So far I’m not sure about the
tarball contents, but the repo seems to be archived:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,m (guix swh)
scheme@(guix swh)> (lookup-origin "https://guix.gnu.org/sources.json")
$2 = #<<origin> visits-url: "https://archive.softwareheritage.org/api/1/origin/https://guix.gnu.org/sources.json/visits/" type: #<unspecified> url: "https://guix.gnu.org/sources.json">
scheme@(guix swh)> (car (origin-visits $2))
$3 = #<<visit> date: #<date nanosecond: 807978 second: 20 minute: 38 hour: 17 day: 8 month: 9 year: 2021 zone-offset: 0> origin: "https://guix.gnu.org/sources.json" url: "https://archive.softwareheritage.org/api/1/origin/https://guix.gnu.org/sources.json/visit/313/" snapshot-url: "https://archive.softwareheritage.org/api/1/snapshot/bfdb775f4768dd3d20231effbf29b96cd7184985/" status: partial number: 313>
scheme@(guix swh)> (define s (visit-snapshot $3))
scheme@(guix swh)> (car ((@@ (gnu packages linux) linux-libre-urls) "5.14.1" "gnu"))
$4 = "https://linux-libre.fsfla.org/pub/linux-libre/releases/5.14.1-gnu/linux-libre-5.14.1-gnu.tar.xz"
scheme@(guix swh)> (lookup-snapshot-branch s $4)
$5 = #f
scheme@(guix swh)> (lookup-snapshot-branch s (car ((@@ (gnu packages linux) linux-libre-urls) "5.13.14" "gnu1")))
$6 = #f
scheme@(guix swh)> (lookup-snapshot-branch s (car ((@@ (gnu packages linux) linux-libre-urls) "5.10.62" "gnu1")))
$7 = #f
scheme@(guix swh)> (lookup-origin "git://linux-libre.fsfla.org/releases.git")
$8 = #<<origin> visits-url: "https://archive.softwareheritage.org/api/1/origin/git://linux-libre.fsfla.org/releases.git/visits/" type: #<unspecified> url: "git://linux-libre.fsfla.org/releases.git">
scheme@(guix swh)> (car (origin-visits $8))
$9 = #<<visit> date: #<date nanosecond: 729324 second: 34 minute: 9 hour: 17 day: 4 month: 9 year: 2021 zone-offset: 0> origin: "git://linux-libre.fsfla.org/releases.git" url: "https://archive.softwareheritage.org/api/1/origin/git://linux-libre.fsfla.org/releases.git/visit/4/" snapshot-url: "https://archive.softwareheritage.org/api/1/snapshot/cbf57c5bd0c5161c8df8853c64e96040b5c9cd9c/" status: full number: 4>
--8<---------------cut here---------------end--------------->8---

For some reason, we seem to be exporting only one tarball in
sources.json currently (the file that SWH periodically reads):

--8<---------------cut here---------------start------------->8---
$ wget -qO - https://guix.gnu.org/sources.json|jq |grep linux-libre.fsfla.org
        "https://linux-libre.fsfla.org/pub/linux-libre/releases/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz",
        "https://linux-libre.fsfla.org/pub/linux-libre/releases/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz",
--8<---------------cut here---------------end--------------->8---

We should check why we’re not providing all the URLs and fix it.

> The reason for the removal is that some nonfree things were accidentally
> not removed by the previous releases of the deblob scripts.

IMO such issues should be treated as regular bugs, without compromising
on transparency and availability.

Thanks,
Ludo’.

Why linux-libre source code is not in sources.json

[zimoun]

Hi,

On Thu, 09 Sep 2021 at 00:00, Ludovic Courtès <ludo@gnu.org> wrote:

> Now we have to see what’s available on berlin & co., and the extent to
> which SWH can help with this situation.  So far I’m not sure about the
> tarball contents, but the repo seems to be archived:

As we know, the issue with tarball on SWH is that we cannot guarantee
the map between the information stored at package time and the
information SWH serves at fetch time.  Metadata, etc. so SWH cooking can
return a tarball having the checksum Guix that expects or something
different.

We (at least me :-)) need to invest energy into Disarchive.
Definitively! 

> For some reason, we seem to be exporting only one tarball in
> sources.json currently (the file that SWH periodically reads):
>
> --8<---------------cut here---------------start------------->8---
> $ wget -qO - https://guix.gnu.org/sources.json|jq |grep linux-libre.fsfla.org
>         "https://linux-libre.fsfla.org/pub/linux-libre/releases/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz",
>         "https://linux-libre.fsfla.org/pub/linux-libre/releases/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz",
> --8<---------------cut here---------------end--------------->8---
>
> We should check why we’re not providing all the URLs and fix it.

Interesting… :-)

The file ’https://guix.gnu.org/sources.json’ is basically built using
’fold-packages’ (see ’all-packages’ in
guix-artwork/website/apps/packages/data.scm) and ’package-source’ (see
’sources-json-builder’ in
guix-artwork/website/apps/packages/builder.scm).  And indeed,
’linux-libre’ is missing.

Let consider this snippet as ’/tmp/foo.scm’ mimicking the builder of
JSON files:

--8<---------------cut here---------------start------------->8---
(use-modules (guix packages)
             (gnu packages))

(define all-packages
  (sort
   (fold-packages (lambda (package lst)
                    (if (or (package-superseded package)
                            (package-replacement package))
                        lst
                        (cons package lst)))
                  '())
   (lambda (p1 p2)
     (string<? (package-name p1)
               (package-name p2)))))

(map (lambda (p)
       (format #t "~a~%" (package-source p)))
     all-packages)
--8<---------------cut here---------------end--------------->8---

Then indeed, only 5.4.20 is seen.

--8<---------------cut here---------------start------------->8---
$ guix repl /tmp/foo.scm | grep 'linux-libre.fsfla'

#<origin ("https://linux-libre.fsfla.org/pub/linux-libre/releases/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz" "ftp://alpha.gnu.org/gnu/guix/mirror/linux-libre-5.4.20-gnu.tar.xz" "mirror://gnu/linux-libre/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz") #<content-hash sha256:1qxhf6dmcwjblzx8fgn6vr10p38xw10iwh6d1y1v1mxb25y30b47> () 7faf439cfba0>
#<origin ("https://linux-libre.fsfla.org/pub/linux-libre/releases/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz" "ftp://alpha.gnu.org/gnu/guix/mirror/linux-libre-5.4.20-gnu.tar.xz" "mirror://gnu/linux-libre/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz") #<content-hash sha256:1qxhf6dmcwjblzx8fgn6vr10p38xw10iwh6d1y1v1mxb25y30b47> () 7faf439cfba0>
--8<---------------cut here---------------end--------------->8---

The difference is that this package ’linux-libre-headers-5.4.20’ is
created with ’make-linux-libre-headers’ and the others with
’make-linux-libre-headers*’.  Subtle. ;-)  In other words,

--8<---------------cut here---------------start------------->8---
$ guix repl
scheme@(guix-user)> ,use(guix packages)
scheme@(guix-user)> ,use(gnu packages linux)
scheme@(guix-user)> (package-source linux-libre-headers-5.4.20)
$1 = #<origin ("https://linux-libre.fsfla.org/pub/linux-libre/releases/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz" "ftp://alpha.gnu.org/gnu/guix/mirror/linux-libre-5.4.20-gnu.tar.xz" "mirror://gnu/linux-libre/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz") #<content-hash sha256:1qxhf6dmcwjblzx8fgn6vr10p38xw10iwh6d1y1v1mxb25y30b47> () 7f881c1ec4e0>
scheme@(guix-user)> (package-source linux-libre-headers-5.13)
$2 = #<origin #<promise #<procedure 7f881c1dd900 at gnu/packages/linux.scm:248:8 ()>> #<content-hash sha256:#f> (#<origin "http://www.fsfla.org/svn/fsfla/software/linux-libre/lemote/gnewsense/branches/3.16/100gnu+freedo.patch" #<content-hash sha256:1hk9swxxc80bmn2zd2qr5ccrjrk28xkypwhl4z0qx4hbivj7qm06> () 7f881c1ec840> #<origin "https://salsa.debian.org/kernel-team/linux/raw/34a7d9011fcfcfa38b68282fd2b1a8797e6834f0/debian/patches/bugfix/arm/arm-mm-export-__sync_icache_dcache-for-xen-privcmd.patch" #<content-hash sha256:1ifnfhpakzffn4b8n7x7w5cps9mzjxlkcfz9zqak2vaw8nzvl39f> () 7f881c1ec7e0> "/gnu/store/yrqr7syxbm4pddzlgc4pwn9wixmpy9xh-guix-module-union/share/guile/site/3.0/gnu/packages/patches/linux-libre-arm64-generic-pinebook-lcd.patch") 7f881c1ec780>
--8<---------------cut here---------------end--------------->8---

Therefore, the builder of JSON (mainly ’origin->json’) does not consider
such cases and assume that ’origin-uri’ can be applied.  Well, I will
try to improve the situation if no one beats me. :-)


Cheers,
simon

Re: Why linux-libre source code is not in sources.json

[zimoun]

Hi,

On Thu, 09 Sep 2021 at 01:46, zimoun <zimon.toutoune@gmail.com> wrote:

> --8<---------------cut here---------------start------------->8---
> scheme@(guix-user)> (package-source linux-libre-headers-5.13)
> $2 = #<origin #<promise #<procedure 7f881c1dd900 at gnu/packages/linux.scm:248:8 ()>> #<content-hash sha256:#f> (#<origin "http://www.fsfla.org/svn/fsfla/software/linux-libre/lemote/gnewsense/branches/3.16/100gnu+freedo.patch" #<content-hash sha256:1hk9swxxc80bmn2zd2qr5ccrjrk28xkypwhl4z0qx4hbivj7qm06> () 7f881c1ec840> #<origin "https://salsa.debian.org/kernel-team/linux/raw/34a7d9011fcfcfa38b68282fd2b1a8797e6834f0/debian/patches/bugfix/arm/arm-mm-export-__sync_icache_dcache-for-xen-privcmd.patch" #<content-hash sha256:1ifnfhpakzffn4b8n7x7w5cps9mzjxlkcfz9zqak2vaw8nzvl39f> () 7f881c1ec7e0> "/gnu/store/yrqr7syxbm4pddzlgc4pwn9wixmpy9xh-guix-module-union/share/guile/site/3.0/gnu/packages/patches/linux-libre-arm64-generic-pinebook-lcd.patch") 7f881c1ec780>
> --8<---------------cut here---------------end--------------->8---
>
> Therefore, the builder of JSON (mainly ’origin->json’) does not consider
> such cases and assume that ’origin-uri’ can be applied.  Well, I will
> try to improve the situation if no one beats me. :-)

Ouch, it appears to me complicated because:

 a) at the package level, what is the source of linux-libre?
 b) this source is the result of some computed-origin-method

Somehow, SWH ingests URLs and no more.  And the current implementation
looks like:

--8<---------------cut here---------------start------------->8---
(define-public linux-libre-5.14-pristine-source
  (let ((version linux-libre-5.14-version)
        (hash (base32 "1iq8s031fviccc4710biwl7gxqdimm3nhlvxd0m3fykvhhmcanq0")))
   (make-linux-libre-source version
                            (%upstream-linux-source version hash)
                            deblob-scripts-5.14)))
--8<---------------cut here---------------end--------------->8---

where ’make-linux-libre-source’ returns a ’computed-origin-method’.  And
the ’origin-uri’ of ’linux-libre-5.14-pristine-source’ is a ’gexp’.
Then inside this ’gexp’, you can read the ’%upstream-linux-source’ URL:

--8<---------------cut here---------------start------------->8---
#<gexp-input native
 #<origin
  #"mirror://kernel.org/linux/kernel/v5.x/linux-5.13.14.tar.xz"
  #<content-hash sha256:06lbjsbr86qa8yai5gfclbfxvcqsw33kxj9b4r93hh6z1wajmx82>
--8<---------------cut here---------------end--------------->8---

and I do not know if it is possible to extract such thing.

Moreover, the ’deblob-scripts-5.14’ is an origin from
’linux-libre-deblob-scripts’ which returns a list of 2 origins.  These 2
URLs do not appear in ’linux-libre-5.14-pristine-source’, for instance.

Therefore, I do not know how to extract the source URLs for the package
’linux-libre-5.14’.


Ideas?

Cheers,
simon

Re: Why linux-libre source code is not in sources.json

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

Hi,

> [...]
> 
> where ’make-linux-libre-source’ returns a ’computed-origin-method’.  And
> the ’origin-uri’ of ’linux-libre-5.14-pristine-source’ is a ’gexp’.
> Then inside this ’gexp’, you can read the ’%upstream-linux-source’ URL:
> 
> --8<---------------cut here---------------start------------->8---
> #<gexp-input native
>  #<origin
>   #"mirror://kernel.org/linux/kernel/v5.x/linux-5.13.14.tar.xz"
>   #<content-hash sha256:06lbjsbr86qa8yai5gfclbfxvcqsw33kxj9b4r93hh6z1wajmx82>
> --8<---------------cut here---------------end--------------->8---
> 
> and I do not know if it is possible to extract such thing.

To extract the <origin> from the <gexp-input>, use gexp-input-thing.
To extract the <gexp-input> from the <gexp>, use the unexported
gexp-references.

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Why linux-libre source code is not in sources.json

[zimoun]

Hi,

On Thu, 9 Sept 2021 at 11:51, Maxime Devos <maximedevos@telenet.be> wrote:

> > where ’make-linux-libre-source’ returns a ’computed-origin-method’.  And
> > the ’origin-uri’ of ’linux-libre-5.14-pristine-source’ is a ’gexp’.
> > Then inside this ’gexp’, you can read the ’%upstream-linux-source’ URL:
> >
> > --8<---------------cut here---------------start------------->8---
> > #<gexp-input native
> >  #<origin
> >   #"mirror://kernel.org/linux/kernel/v5.x/linux-5.13.14.tar.xz"
> >   #<content-hash sha256:06lbjsbr86qa8yai5gfclbfxvcqsw33kxj9b4r93hh6z1wajmx82>
> > --8<---------------cut here---------------end--------------->8---
> >
> > and I do not know if it is possible to extract such thing.
>
> To extract the <origin> from the <gexp-input>, use gexp-input-thing.
> To extract the <gexp-input> from the <gexp>, use the unexported
> gexp-references.

Thanks!  It does the job. :-)

I will fix sources.json if no one beats me.

With the (ugly) snippet below,  I get almost the linux-libre, I guess.
The question is now how does SWH ingest the script with the URL:

<https://linux-libre.fsfla.org/pub/linux-libre/releases/5.13.14-gnu1/deblob-5.13>

Another story. :-)

Cheers,
simon


--8<---------------cut here---------------start------------->8---
$ guix repl /tmp/linux.scm | grep linux-libre
(mirror://kernel.org/linux/kernel/v5.x/linux-5.13.14.tar.xz
https://linux-libre.fsfla.org/pub/linux-libre/releases/5.13.14-gnu1/deblob-check
https://linux-libre.fsfla.org/pub/linux-libre/releases/5.13.14-gnu1/deblob-5.13)
(mirror://kernel.org/linux/kernel/v5.x/linux-5.13.14.tar.xz
https://linux-libre.fsfla.org/pub/linux-libre/releases/5.13.14-gnu1/deblob-check
https://linux-libre.fsfla.org/pub/linux-libre/releases/5.13.14-gnu1/deblob-5.13)
(mirror://kernel.org/linux/kernel/v4.x/linux-4.4.283.tar.xz
https://linux-libre.fsfla.org/pub/linux-libre/releases/4.4.283-gnu1/deblob-check
https://linux-libre.fsfla.org/pub/linux-libre/releases/4.4.283-gnu1/deblob-4.4)
(mirror://kernel.org/linux/kernel/v4.x/linux-4.19.206.tar.xz
https://linux-libre.fsfla.org/pub/linux-libre/releases/4.19.206-gnu1/deblob-check
https://linux-libre.fsfla.org/pub/linux-libre/releases/4.19.206-gnu1/deblob-4.19)

[...]

(https://linux-libre.fsfla.org/pub/linux-libre/releases/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz
ftp://alpha.gnu.org/gnu/guix/mirror/linux-libre-5.4.20-gnu.tar.xz
mirror://gnu/linux-libre/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz)
(mirror://kernel.org/linux/kernel/v4.x/linux-4.14.246.tar.xz
https://linux-libre.fsfla.org/pub/linux-libre/releases/4.14.246-gnu1/deblob-check
https://linux-libre.fsfla.org/pub/linux-libre/releases/4.14.246-gnu1/deblob-4.14)
(https://linux-libre.fsfla.org/pub/linux-libre/releases/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz
ftp://alpha.gnu.org/gnu/guix/mirror/linux-libre-5.4.20-gnu.tar.xz
mirror://gnu/linux-libre/5.4.20-gnu/linux-libre-5.4.20-gnu.tar.xz)
--8<---------------cut here---------------end--------------->8---


--8<---------------cut here---------------start------------->8---
(use-modules (guix packages)
             (guix gexp)
             (gnu packages)
             (gnu packages linux)
             (srfi srfi-1)
             (ice-9 match))

(define gexp-references (@@ (guix gexp) gexp-references))

(define (thing->string thing)
  (match thing
    ((? gexp-input? g)
     (match (gexp-input-thing g)
       ((? origin? o)
        (origin-uri o))
       (_ #f)))
    (_ #f)))

(define (get-uri pkg)
  (match (package-source pkg)
    ((? origin? o)
     (match (origin-uri o)
       ((? string? s)
        (list s))
       ((? list? lst)
        lst)
       ((? promise? prom)
        (match (force prom)
          ((? gexp? g)
           (filter (lambda (x) x)
                   (delete-duplicates
                    (map thing->string
                         (gexp-references g)))))
          (_
           (list #f))))
       (_
        (list #f))))
    (_ #f)))

(define all-packages
  (sort
   (fold-packages (lambda (package lst)
                    (if (or (package-superseded package)
                            (package-replacement package))
                        lst
                        (cons package lst)))
                  '())
   (lambda (p1 p2)
     (string<? (package-name p1)
               (package-name p2)))))

(map (lambda (p)
       (format #t "~a~%" (get-uri p)))
     all-packages)
--8<---------------cut here---------------end--------------->8---

Linux-libre source code via SWH sources.json

[zimoun]

Hi,

On Thu, 09 Sep 2021 at 00:00, Ludovic Courtès <ludo@gnu.org> wrote:

> We should check why we’re not providing all the URLs and fix it.

Done in patch#50515.

<http://issues.guix.gnu.org/issue/50515>

Well, I think that all the packages with true ’origin?’ are now in
’sources.json’.  If not, please comment in #50515. :-)

However, we need to check if the deblob scripts are ingested by SWH.
Well, I am not sure because their listener of ’sources.json’ only
ingests tarballs and other zip files but not plain file as the deblob
scripts.  Who knows? :-)

Cheers,
simon