bug#28664: gst-plugins-base-1.12.3.tar.xz nar on berlin returns 404

bug#28664: gst-plugins-base-1.12.3.tar.xz nar on berlin returns 404

[Maxim Cournoyer]

Hello,

While attempting to update today I got that:

--8<---------------cut here---------------start------------->8---
Downloading https://berlin.guixsd.org/nar/gzip/3vfvi0qcb5fwwm2pdy3vx06v4jhmwr3d-gtk%2B-2.24.31...
 gtk%2B-2.24.31  10.0MiB                                                                                                                                                                          1.3MiB/s 00:07 [####################] 100.0%

Downloading https://berlin.guixsd.org/nar/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz...
guix substitute: error: download from 'https://berlin.guixsd.org/nar/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz' failed: 404, "Not Found"
guix package: error: build failed: some substitutes for the outputs of derivation `/gnu/store/vcjfcp3va23yqcvg9gbcqwk9fm19hh1c-gst-plugins-base-1.12.3.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
--8<---------------cut here---------------end--------------->8---

Maxim

bug#28664: gst-plugins-base-1.12.3.tar.xz nar on berlin returns 404

[Ludovic Courtès]

Hello Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Downloading https://berlin.guixsd.org/nar/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz...
> guix substitute: error: download from 'https://berlin.guixsd.org/nar/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz' failed: 404, "Not Found"

I think it’s a bug in ‘guix publish’: the narinfo is still here:

--8<---------------cut here---------------start------------->8---
$ wget -q -O - https://berlin.guixsd.org/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697.narinfo
StorePath: /gnu/store/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz
URL: nar/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz
Compression: none
NarHash: sha256:07rx2px28pv6vqvzww4i0sldjray4q4jfggz9x0vp3wqla78rwm5
NarSize: 3117480
References: 
FileSize: 3117480
System: i686-linux
Deriver: mx8rpg1qqbl4244r2xsn3g59lw671sh3-gst-plugins-base-1.12.3.tar.xz.drv
Signature: 1;berlin.guixsd.org;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjMEZFNjA0QjNBRkJGRDYzRjM3RjA3NjJCMjg2MDBGMDFFRUVEQkZCNTE5RTQyNDAwMjNGOEE4NzlBRTA4OTVFOSMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDhEMzhFNkFGM0ZGNzFGNENBMTBENTIwRjU5OTUyREZGNEMxQTREMzREODkwNEYwODVFMDY4Q0FGNTgyNjE4MCMpCiAgIChzICMwNDQwODg2RDEzQTVERkM5RDE4NURFMEJBNzkxMTdGREUxNEQ0OUEyMDE3MUJDQ0M4RjQxRUQ4NDQ5NzcyNDM1IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjOEQxNTZGMjk1RDI0QjBEOUE4NkZBNTc0MUE4NDBGRjJEMjRGNjBGN0I2QzQxMzQ4MTRBRDU1NjI1OTcxQjM5NCMpCiAgICkKICApCiApCg==
--8<---------------cut here---------------end--------------->8---

but the tarball itself is missing.

On berlin, the narinfo is indeed around:

--8<---------------cut here---------------start------------->8---
$ sudo ls -l /var/cache/guix/publish/none/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697*
-rw------- 1 guix-publish guix-publish 998 Sep 19 11:12 /var/cache/guix/publish/none/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz.narinfo
--8<---------------cut here---------------end--------------->8---

So the problem is that for uncompressed store items (‘guix publish’
serves *.tar.xz without any additional compression, so it uses the
‘none’ compression method from its viewpoint), ‘guix publish’ does not
keep the nar in /var/cache/guix/publish, so it cannot guarantee its TTL.

The fix is probably simply to store nars even for uncompressed store
items, even if that increases disk usage.

Ludo’.

bug#28664: gst-plugins-base-1.12.3.tar.xz nar on berlin returns 404

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> So the problem is that for uncompressed store items (‘guix publish’
> serves *.tar.xz without any additional compression, so it uses the
> ‘none’ compression method from its viewpoint), ‘guix publish’ does not
> keep the nar in /var/cache/guix/publish, so it cannot guarantee its TTL.
>
> The fix is probably simply to store nars even for uncompressed store
> items, even if that increases disk usage.

Done in e5788ebbe1f45a1088249b9138de17b330609712.

(We still need to deploy the fix on berlin.guixsd.org & co.)

Ludo’.

bug#28664: gst-plugins-base-1.12.3.tar.xz nar on berlin returns 404

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> So the problem is that for uncompressed store items (‘guix publish’
>> serves *.tar.xz without any additional compression, so it uses the
>> ‘none’ compression method from its viewpoint), ‘guix publish’ does not
>> keep the nar in /var/cache/guix/publish, so it cannot guarantee its TTL.
>>
>> The fix is probably simply to store nars even for uncompressed store
>> items, even if that increases disk usage.
>
> Done in e5788ebbe1f45a1088249b9138de17b330609712.
>
> (We still need to deploy the fix on berlin.guixsd.org & co.)

It’s now deployed on berlin.guixsd.org.

Ludo’.

bug#28664: Old narinfos of tarballs purged on berlin

[Ludovic Courtès]

Hello there,

As a followup to <https://bugs.gnu.org/28664>, Ricardo upgraded to the
latest Guix on berlin, and I’ve now removed old .narinfos corresponding
to tarballs for which we potentially lacked the corresponding nar:

  sudo find /var/cache/guix/publish/none -mtime +7 -exec rm -v {} \;

There’s caching in nginx and on the user’s machines, so for a while we
might still get 200 for the narinfo and 404 for the corresponding nar,
but that will become consistent over time.

Apparently hydra.gnu.org is largely unaffected, presumably because we
GC’d less aggressively or have more GC roots.

Ludo’.

bug#28664: Old narinfos of tarballs purged on berlin

[Ludovic Courtès]

Hi,

ludo@gnu.org (Ludovic Courtès) skribis:

> As a followup to <https://bugs.gnu.org/28664>, Ricardo upgraded to the
> latest Guix on berlin, and I’ve now removed old .narinfos corresponding
> to tarballs for which we potentially lacked the corresponding nar:
>
>   sudo find /var/cache/guix/publish/none -mtime +7 -exec rm -v {} \;

This missed ~100 of them, so I’ve run the following command, which is
more accurate:

  root@berlin /var/cache/guix/publish/none# rm -v $(for i in *.narinfo ; do [ -f `echo $i | sed -es/narinfo/nar/g` ] || echo $i ; done)

Ludo’.

bug#28664: Old narinfos of tarballs purged on berlin

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> As a followup to <https://bugs.gnu.org/28664>, Ricardo upgraded to the
>> latest Guix on berlin, and I’ve now removed old .narinfos corresponding
>> to tarballs for which we potentially lacked the corresponding nar:
>>
>>   sudo find /var/cache/guix/publish/none -mtime +7 -exec rm -v {} \;
>
> This missed ~100 of them, so I’ve run the following command, which is
> more accurate:
>
>   root@berlin /var/cache/guix/publish/none# rm -v $(for i in *.narinfo ; do [ -f `echo $i | sed -es/narinfo/nar/g` ] || echo $i ; done)

I just did the same on hydra.gnu.org, which was starting to have 404 for
these substitutes.

Ludo’.

bug#28664: Old narinfos of tarballs purged on berlin

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> ludo@gnu.org (Ludovic Courtès) skribis:
>>
>>> As a followup to <https://bugs.gnu.org/28664>, Ricardo upgraded to the
>>> latest Guix on berlin, and I’ve now removed old .narinfos corresponding
>>> to tarballs for which we potentially lacked the corresponding nar:
>>>
>>>   sudo find /var/cache/guix/publish/none -mtime +7 -exec rm -v {} \;
>>
>> This missed ~100 of them, so I’ve run the following command, which is
>> more accurate:
>>
>>   root@berlin /var/cache/guix/publish/none# rm -v $(for i in *.narinfo ; do [ -f `echo $i | sed -es/narinfo/nar/g` ] || echo $i ; done)
>
> I just did the same on hydra.gnu.org, which was starting to have 404 for
> these substitutes.

With narinfo caching people were starting to get 404s for
mirror.hydra.gnu.org/guix/nar/xyz.  Based on an incomplete log of the
command above, I’ve repopulated /var/cache/guix/publish/none on
hydra.gnu.org in a pretty crude way:

  for i in $(sed -es/'^removed `\([^-]\+\)-.*/\1/g' < /tmp/file-list ); do wget -q -O /dev/null http://localhost:9999/$i.narinfo || echo $i ; done

That’s 1,135 tarballs back in /var/cache/guix/publish/none.

The remaining missing items will come back eventually as narinfo
requests hit ‘guix publish’ itself (i.e., when narinfo caches on user
machines and in nginx have expired).

Ludo’.