bug#14851: linux-libre-3.3.8-gnu disappeared

bug#14851: linux-libre-3.3.8-gnu disappeared

[Andreas Enge]

The build of linux-libre-headers-3.3.8 currently fails because the tarball 
cannot be downloaded any more from
   http://linux-libre.fsfla.org/pub/linux-libre/releases/3.3.8-gnu/
Instead, there is a new version in
   http://linux-libre.fsfla.org/pub/linux-libre/releases/3.3.8-gnu1/

Should we switch to this one? Or update to the newest version? In all 
cases, I am afraid a complete rebuild will be triggered, so we might as 
well advance in the version numbers.

Andreas

bug#14851: linux-libre-3.3.8-gnu disappeared

[Ludovic Courtès]

Hello,

(Cc: Alexandre Oliva of Linux-Libre.)

Andreas Enge <andreas@enge.fr> skribis:

> The build of linux-libre-headers-3.3.8 currently fails because the tarball 
> cannot be downloaded any more from
>    http://linux-libre.fsfla.org/pub/linux-libre/releases/3.3.8-gnu/
> Instead, there is a new version in
>    http://linux-libre.fsfla.org/pub/linux-libre/releases/3.3.8-gnu1/

I just checked and the latter has a different SHA256
(1860as3dwh7f3im1sq3x06awmil9vppl6igg2gnva5w9mbkw2fc8).

> Should we switch to this one? Or update to the newest version? In all 
> cases, I am afraid a complete rebuild will be triggered, so we might as 
> well advance in the version numbers.

Indeed.

Since we’re about to release a new version of Guix, I’d rather keep
using 3.3.8.

Alexandre: could you reinstate the original
http://linux-libre.fsfla.org/pub/linux-libre/releases/3.3.8-gnu/linux-libre-3.3.8-gnu.tar.xz?

It would be ideal if the tarballs were on ftp.gnu.org.  I could do it if
you don’t want to bother, provided the FTP admins allow it.  WDYT?

Thanks,
Ludo’.

bug#14851: linux-libre-3.3.8-gnu disappeared

[Jason Self]

[-- Attachment #1: Type: text/plain, Size: 351 bytes --]

My understanding is that the change from gnu to gnu1 is the result of
changes that were needed to the deblobbing. Specifically to allow the
loading of the firmware used by ath9k-htc, now that it's free [0]...
i.e. 3.3.8-gnu and 3.3.8-gnu1 aren't the same source code.

[0]
https://www.fsf.org/news/ryf-certification-thinkpenguin-usb-with-atheros-chip

bug#14851: linux-libre-3.3.8-gnu disappeared

[Ludovic Courtès]

"Jason Self" <jason@bluehome.net> skribis:

> My understanding is that the change from gnu to gnu1 is the result of
> changes that were needed to the deblobbing. Specifically to allow the
> loading of the firmware used by ath9k-htc, now that it's free [0]...
> i.e. 3.3.8-gnu and 3.3.8-gnu1 aren't the same source code.

OK, thanks for the info.

It’s hard for us to deal with disappearing software.  In this case,
we’re currently using this tarball just to install the Linux-Libre
headers to build libc, and our ‘linux-libre’ package can use a different
upstream version than ‘linux-libre-headers’ anyway.

Thanks,
Ludo’.

bug#14851: linux-libre-3.3.8-gnu disappeared

[Ludovic Courtès]

I’ve asked ftp-upload@ if they could allow me to upload linux-libre
tarballs to ftp.gnu.org.

In the meantime, I’ve uploaded this tarball to
ftp://alpha.gnu.org/gnu/guix/mirror and changed linux.scm to refer to it.

Thanks,
Ludo’.

bug#14851: linux-libre-3.3.8-gnu disappeared

[Alexandre Oliva]

On Jul 12, 2013, ludo@gnu.org (Ludovic Courtès) wrote:

> Since we’re about to release a new version of Guix, I’d rather keep
> using 3.3.8.

> Alexandre: could you reinstate the original
> http://linux-libre.fsfla.org/pub/linux-libre/releases/3.3.8-gnu/linux-libre-3.3.8-gnu.tar.xz?

I suppose you don't want to prevent users of guix from using ath9k wifi
cards, so I strongly suggest switching to 3.3.8-gnu1.  Indeed, I think
you'd be better off with some LTS version of GNU Linux-libre, rather
than the dead 3.3 branch.  But that's your call.

> It would be ideal if the tarballs were on ftp.gnu.org.  I could do it if
> you don’t want to bother, provided the FTP admins allow it.  WDYT?

I'd be glad with such an arrangement.  I keep on failing to figure out
how to fit the weekly publishing of multiple releases into a workflow
that includes collecting information and sending it to ftp.gnu.org
without keeping another local copy of stuff that was published before.
That's one of the hold-up factors for me.  As for the tarballs, they're
all signed, so figuring out a way to upload just the bits created since
the last push, and pushing them to live, is what's missing.

Now, another possibility that I think would make more sense for guix is
to have its sources consolidated in a single place, rather than
scattered all over and at risk of having them pulled from under you.  At
the very least, you ought to keep a copy of sources you use to build
binaries you publish, so that you can satisfy your obligation to offer
the corresponding source, be it a legal (copyleft) or moral (software in
gnu ought to be free) obligation.

When we get GNU Linux-libre at ftp.gnu.org, it could then be hard links,
so that if we remove some tarball it won't go away from your “copy”, but
until then, you might be better off holding your own copy rather than
assuming our primary repository has infinite space.  Unfortunately it
doesn't, and I have to clean things up quite often.  For sources, I at
least keep enough bits around that the tarballs can be reconstructed in
a bit-exact fashion, but for binaries, when they're gone, they're gone
forever.  However, considering we put out multiple GBs of builds per
week, I don't think it's realistic to keep them all forever.  Not in our
own server, not at ftp.gnu.org.

-- 
Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
You must be the change you wish to see in the world. -- Gandhi
Be Free! -- http://FSFLA.org/   FSF Latin America board member
Free Software Evangelist      Red Hat Brazil Compiler Engineer

bug#14851: linux-libre-3.3.8-gnu disappeared

[Ludovic Courtès]

Alexandre Oliva <lxoliva@fsfla.org> skribis:

> On Jul 12, 2013, ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Since we’re about to release a new version of Guix, I’d rather keep
>> using 3.3.8.
>
>> Alexandre: could you reinstate the original
>> http://linux-libre.fsfla.org/pub/linux-libre/releases/3.3.8-gnu/linux-libre-3.3.8-gnu.tar.xz?
>
> I suppose you don't want to prevent users of guix from using ath9k wifi
> cards, so I strongly suggest switching to 3.3.8-gnu1.

Of course not, but again, this one is used to get headers against which
to build glibc, so that’s not a problem.

> Indeed, I think you'd be better off with some LTS version of GNU
> Linux-libre, rather than the dead 3.3 branch.  But that's your call.

When we have a stand-alone, bootable distro, we’ll certainly want to
synchronize with you for the choice of the default kernel version.

>> It would be ideal if the tarballs were on ftp.gnu.org.  I could do it if
>> you don’t want to bother, provided the FTP admins allow it.  WDYT?
>
> I'd be glad with such an arrangement.

Great.

> Now, another possibility that I think would make more sense for guix is
> to have its sources consolidated in a single place, rather than
> scattered all over and at risk of having them pulled from under you.

Actually, our continuous integration server at http://hydra.gnu.org does
that transparently: it caches all the source tarballs, along with build
byproducts.

So when a tarball vanishes from its upstream site, it’s usually not a
blocking problem.  Yet, it’s preferable to have them elsewhere, because
they will eventually be garbage-collected from hydra.gnu.org.

[...]

> When we get GNU Linux-libre at ftp.gnu.org, it could then be hard links,
> so that if we remove some tarball it won't go away from your “copy”, but
> until then, you might be better off holding your own copy rather than
> assuming our primary repository has infinite space.  Unfortunately it
> doesn't, and I have to clean things up quite often.  For sources, I at
> least keep enough bits around that the tarballs can be reconstructed in
> a bit-exact fashion, but for binaries, when they're gone, they're gone
> forever.  However, considering we put out multiple GBs of builds per
> week, I don't think it's realistic to keep them all forever.  Not in our
> own server, not at ftp.gnu.org.

What do you mean by “multiple GBs of builds per week”?  Linux{,-Libre}
releases are not that frequent, are they?

The policy at ftp.gnu.org has always been to keep everything forever,
AFAIK.

If size turns out to be a problem, we could choose to keep only LTS
releases on ftp.gnu.org, for instance.  That’s something to discuss
with the GNU sysadmins.

Thanks for your feedback!

Ludo’.

bug#14851: linux-libre-3.3.8-gnu disappeared

[Alexandre Oliva]

On Jul 19, 2013, ludo@gnu.org (Ludovic Courtès) wrote:

> What do you mean by “multiple GBs of builds per week”?  Linux{,-Libre}
> releases are not that frequent, are they?

They are.  ATM there are 4 active stable releases that each get one new
release per week.  There are 3 other LTS stable branches that get
releases less often.  And then, there are the weekly development -rcs
for the next release, that I seldom start tracking long before the
release is out.  This is just for sources, and it adds up to almost 1GB
per week.

Adding binaries to the picture grows the entire size by a little bit
when it comes to the freesh and planet .debs, and to a larger extent
when it comes to gnewsense/yeeloong (that gets one build per source
release we put out, with a total deb+tar size about the same size of the
source release subdir), and the freed-ora builds (that involve 1-4 rpm
builds per active Fedora release per week, each one weighting some 2GB
total; for most of the time there are 2 or 3 active releases; sometimes
there are 4, depending on whether or not I've already cleaned up the -rc
series for the upcoming release that gets built for the rawhide tree).

> If size turns out to be a problem, we could choose to keep only LTS
> releases on ftp.gnu.org, for instance.

Keeping only LTS wouldn't help guix, though, since you're not using an
LTS release.

-- 
Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
You must be the change you wish to see in the world. -- Gandhi
Be Free! -- http://FSFLA.org/   FSF Latin America board member
Free Software Evangelist      Red Hat Brazil Compiler Engineer

bug#14851: linux-libre-3.3.8-gnu disappeared

[Ludovic Courtès]

Alexandre Oliva <lxoliva@fsfla.org> skribis:

> On Jul 19, 2013, ludo@gnu.org (Ludovic Courtès) wrote:
>
>> What do you mean by “multiple GBs of builds per week”?  Linux{,-Libre}
>> releases are not that frequent, are they?
>
> They are.  ATM there are 4 active stable releases that each get one new
> release per week.  There are 3 other LTS stable branches that get
> releases less often.  And then, there are the weekly development -rcs
> for the next release, that I seldom start tracking long before the
> release is out.  This is just for sources, and it adds up to almost 1GB
> per week.

Ah OK, I wasn’t aware of that.

> Adding binaries to the picture grows the entire size by a little bit

Yes, but binaries is not what we’re interested in here.  :-)

>> If size turns out to be a problem, we could choose to keep only LTS
>> releases on ftp.gnu.org, for instance.
>
> Keeping only LTS wouldn't help guix, though, since you're not using an
> LTS release.

Heh but that’s not set in stone, and we don’t use Linux-Libre for
anything beyond building libc.

That’ll change in the near future though.

Thanks,
Ludo’.

bug#14851: linux-libre-3.3.8-gnu disappeared

[Andreas Enge]

Am Freitag, 19. Juli 2013 schrieb Alexandre Oliva:
> However, considering we put out multiple GBs of builds per
> week, I don't think it's realistic to keep them all forever.  Not in our
> own server, not at ftp.gnu.org.

As far as I know, ftp.gnu.org would only be concerned with the source, that 
is, the content of 
   http://linux-libre.fsfla.org/pub/linux-libre/releases/
and not with binary packages in upper level directories, and these source 
tarballs are usually kept indefinitely.

One also need not add the patch and diff files, so that the amount of 
storage would be quite manageable.

Andreas

bug#14851: linux-libre-3.3.8-gnu disappeared

[Alexandre Oliva]

On Jul 19, 2013, Andreas Enge <andreas@enge.fr> wrote:

> Am Freitag, 19. Juli 2013 schrieb Alexandre Oliva:
>> However, considering we put out multiple GBs of builds per
>> week, I don't think it's realistic to keep them all forever.  Not in our
>> own server, not at ftp.gnu.org.

> As far as I know, ftp.gnu.org would only be concerned with the source,

I don't know about that.  I've seen binaries in ftp.gnu.org in the past.
This should indeed reduce significantly the space requirements to a bit
less than 1GB per week, considering 4 releases per week in 3 different
compression formats.

OTOH, if we're to be strict, we should publish only the deblob scripts,
for those are the ultimate sources produced by the GNU Linux-libre
project.  The tarballs and diffs and deltas are just the result of
running those scripts on tarballs produced by third parties.

The scripts would be trivial to retain indefinitely; even more so
because they seldom change for stable releases.

> and these source tarballs are usually kept indefinitely.

> One also need not add the patch and diff files, so that the amount of 
> storage would be quite manageable.

The diff files are probably no big deal (10% increase, not an order of
magnitude like the build tarballs and debuginfo rpms).  Excluding them
would probably make for more work in the upload script, though.

-- 
Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
You must be the change you wish to see in the world. -- Gandhi
Be Free! -- http://FSFLA.org/   FSF Latin America board member
Free Software Evangelist      Red Hat Brazil Compiler Engineer

bug#14851: #14851 - linux-libre-3.3.8-gnu disappeared

[ng0]

Andreas, Ludovic, and others:

  This bug report was last modified 3 years and 258 days ago.

It is my understanding that this bug could be closed. I see no more
current problems and it makes no sense to keep the bug open just in case
the problem appears again.

Would you agree?

bug#14851: #14851 - linux-libre-3.3.8-gnu disappeared

[Ludovic Courtès]

ng0 <contact.ng0@cryptolab.net> skribis:

> Andreas, Ludovic, and others:
>
>   This bug report was last modified 3 years and 258 days ago.
>
> It is my understanding that this bug could be closed. I see no more
> current problems and it makes no sense to keep the bug open just in case
> the problem appears again.
>
> Would you agree?

I do!  Especially with the content-addressed mirror we’ve been hosting
with ‘guix publish’.

Ludo’.