bug#35010: Many CPAN download URLs are no longer available

bug#35010: Many CPAN download URLs are no longer available

[Mark H Weaver]

Many perl packages cannot be built on hydra.gnu.org because the source
code is no longer available at the source URLs known to Guix.  Moreover,
apparently they were gone by the time Hydra made its first build
attempt.

Here are some examples:

  mirror://cpan/authors/id/E/ET/ETHER/URI-1.76.tar.gz
  mirror://cpan/authors/id/S/SY/SYOHEX/Mouse-v2.5.6.tar.gz
  mirror://cpan/authors/id/S/SZ/SZABGAB/Class-Date-1.1.17.tar.gz
  mirror://cpan/authors/id/S/SZ/SZABGAB/SVG-2.84.tar.gz
  mirror://cpan/authors/id/A/AB/ABW/Template-Toolkit-2.28.tar.gz
  mirror://cpan/authors/id/D/DA/DAGOLDEN/Log-Any-1.707.tar.gz
  mirror://cpan/authors/id/K/KE/KENTNL/Carp-Clan-6.07.tar.gz
  mirror://cpan/authors/id/R/RJ/RJBS/Carp-1.50.tar.gz
  mirror://cpan/authors/id/D/DA/DAGOLDEN/File-Temp-0.2309.tar.gz
  mirror://cpan/authors/id/M/MA/MAKAMAKA/Text-CSV-1.99.tar.gz

and these are the corresponding download derivations:

  /gnu/store/c73chq3k1z1n2wnbh0sbi2vwip6dfzbk-URI-1.76.tar.gz.drv
  /gnu/store/550r5gdrvs5370rsa2zwp1bs0xwj5dnm-Mouse-v2.5.6.tar.gz.drv
  /gnu/store/qxkgrj9gz1kp9id2d7d3mzrfn5b56019-Class-Date-1.1.17.tar.gz.drv
  /gnu/store/7239snjzan94plk58hw2zgachq71rh64-SVG-2.84.tar.gz.drv
  /gnu/store/qn1w1wvy5x2wngq4845j60i3qvkwhwvi-Template-Toolkit-2.28.tar.gz.drv
  /gnu/store/w1cb4r6c0v37xcjav6z16fzbc2zp47ff-Log-Any-1.707.tar.gz.drv
  /gnu/store/7gdg0aqsc3i7iwnrajcv6skkjbnf6z4r-Carp-Clan-6.07.tar.gz.drv
  /gnu/store/2b088r47jay82d91w62hr35lszqbdrkk-Carp-1.50.tar.gz.drv
  /gnu/store/jsddlzd5aba8vnjz1adggcxmvkgv5qf6-File-Temp-0.2309.tar.gz.drv
  /gnu/store/y2yb19kf1pfblhqwxcll7rrh6pgpr9sh-Text-CSV-1.99.tar.gz.drv

At least some, and probably most, of these URIs were updated quite
recently.  For example, the 'perl-mouse', 'perl-carp-clan', and
'perl-file-temp' were all updated on March 23, and presumably the source
URIs worked at that point, but then all three URIs had to be updated two
days later to fix the broken download links.

The CPAN download URLs that we are using are apparently not long-lived.
Does anyone know if CPAN, or perhaps another site, offers more stable
download URIs for these packages?

      Thanks,
        Mark

bug#35010: Many CPAN download URLs are no longer available

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 2015 bytes --]

Mark,

[Quick stream-of consciousness reply on a train, whee.]

Mark H. Weaver wrote:
> At least some, and probably most, of these URIs were updated 
> quite
> recently.  For example, the 'perl-mouse', 'perl-carp-clan', and
> 'perl-file-temp' were all updated on March 23, and presumably 
> the source
> URIs worked at that point, but then all three URIs had to be 
> updated two
> days later to fix the broken download links.

- The wave of Perl updates this March was me, using ‘guix refresh 
  -u’ (and manually checking for breakages, of course).

I never use ‘guix refresh’, until after about a year I forget why, 
use ‘guix refresh’ once, and promptly remember.

The updater for CPAN packages is at best caveat-quality.  It 
helpfully downloads the updated tarball to the store, but doesn't 
update the URL.  Since the file is in the store, everything works 
fine on the updater's machine, then breaks everywhere else.

Background: the problem here is that CPAN URLs contain the 
uploader name, e.g. (Karen?) ETHER(idge) in the case of

  mirror://cpan/authors/id/E/ET/ETHER/URI-1.76.tar.gz

which doesn't change every VERSION (so the problem is somewhat 
hidden) but more frequently than the author/maintainer would.

I'm planning on finally taking care this problem after I get home 
(and after I finally get the Overdrives set up, cough), either by 
making the CPAN updater also rewrite URL fields (if possible?), or 
finding out whether there's a way to construct these URLs without 
using the uploader name, or… well, that's all I have for now.

- While fixing the remaining fallout from this, I did find a few 
  other broken CPAN links for packages that I hadn't recently 
  touched.  Updating them broke others, so I just left them alone. 
  I don't know if these are more broken refreshes from longer ago 
  that went unnoticed (so basically nobody uses these packages), 
  or if there's another unrelated problem.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#35010: Many CPAN download URLs are no longer available

[Ludovic Courtès]

Hello,

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

> The updater for CPAN packages is at best caveat-quality.  It helpfully
> downloads the updated tarball to the store, but doesn't update the
> URL.  Since the file is in the store, everything works fine on the
> updater's machine, then breaks everywhere else.
>
> Background: the problem here is that CPAN URLs contain the uploader
> name, e.g. (Karen?) ETHER(idge) in the case of
>
>  mirror://cpan/authors/id/E/ET/ETHER/URI-1.76.tar.gz
>
> which doesn't change every VERSION (so the problem is somewhat hidden)
> but more frequently than the author/maintainer would.

Ouch, I didn’t know that, this is terrible.

The good news is that the (guix upstream) framework gets to see the
correct URL:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (package-latest-release perl-uri (force %updaters))
$6 = #<<upstream-source> package: "perl-uri" version: "1.76" urls: ("mirror://cpan/authors/id/O/OA/OALDERS/URI-1.76.tar.gz") signature-urls: #f input-changes: #<procedure 7fced1c44ca0 at guix/import/cpan.scm:308:7 (x)>>
--8<---------------cut here---------------end--------------->8---

So with a bit of work we could change ‘package-update’ to automatically
update the URL.

(Time passes…)

Commit 42314ffa072f31cc1cb44df38b1f8fcca19d9d3c should fix this.
I should have let figure it out ;-), but somehow I ended up
investigating too much, bah!

Let me know what you think.

In the meantime, to fix the Perl packages, you could maybe run:

  guix lint -c source $(guix package -A ^perl-)

Then you could perhaps comment the ‘version>?’ test in (guix upstream)
to force ‘guix package -u’ to update these seemingly up-to-date
packages.

HTH,
Ludo’.

bug#35010: Many CPAN download URLs are no longer available

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 669 bytes --]

Ludo',

Ludovic Courtès wrote:
> Commit 42314ffa072f31cc1cb44df38b1f8fcca19d9d3c should fix this.
> I should have let [you?] figure it out ;-), but somehow I ended
> up investigating too much, bah!

Now I had time to rant about package names.  Win-win!  …wait

> Let me know what you think.

[…]

> In the meantime, to fix the Perl packages, you could maybe run:
>
>   guix lint -c source $(guix package -A ^perl-)
>
> Then you could perhaps comment the ‘version>?’ test in (guix 
> upstream)
> to force ‘guix package -u’ to update these seemingly up-to-date
> packages.

I'll deffo give it a try.  Thanks!

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#35010: Many CPAN download URLs are no longer available

[Mark H Weaver]

Ludovic Courtès <ludo@gnu.org> writes:

> The good news is that the (guix upstream) framework gets to see the
> correct URL:
>
> scheme@(guile-user)> (package-latest-release perl-uri (force %updaters))
> $6 = #<<upstream-source> package: "perl-uri" version: "1.76" urls: ("mirror://cpan/authors/id/O/OA/OALDERS/URI-1.76.tar.gz") signature-urls: #f input-changes: #<procedure 7fced1c44ca0 at guix/import/cpan.scm:308:7 (x)>>
>
> So with a bit of work we could change ‘package-update’ to automatically
> update the URL.
>
> (Time passes…)
>
> Commit 42314ffa072f31cc1cb44df38b1f8fcca19d9d3c should fix this.

Nice!  I haven't looked closely, but does this somehow update the code
that computes source URLs using 'string-append'?  If so, that's
impressive :)

In other news, here's another batch of bad source URLs that Hydra
recently failed to fetch:

  mirror://savannah/starfighter/1.7/project-starfighter-1.7-src.tar.gz
  mirror://cpan/authors/id/P/PI/PIJLL/DateTime-Calendar-Julian-0.100.tar.gz
  mirror://cpan/authors/id/A/AD/ADAMK/Object-Signature-1.08.tar.gz
  mirror://cpan/authors/id/J/JH/JHI/Time-HiRes-1.9760.tar.gz
  mirror://cpan/authors/id/F/FR/FREW/Class-C3-Componentised-1.001002.tar.gz
  mirror://cpan/authors/id/R/RG/RGARCIA/base-2.23.tar.gz

       Mark

bug#35010: Many CPAN download URLs are no longer available

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> The good news is that the (guix upstream) framework gets to see the
>> correct URL:
>>
>> scheme@(guile-user)> (package-latest-release perl-uri (force %updaters))
>> $6 = #<<upstream-source> package: "perl-uri" version: "1.76" urls: ("mirror://cpan/authors/id/O/OA/OALDERS/URI-1.76.tar.gz") signature-urls: #f input-changes: #<procedure 7fced1c44ca0 at guix/import/cpan.scm:308:7 (x)>>
>>
>> So with a bit of work we could change ‘package-update’ to automatically
>> update the URL.
>>
>> (Time passes…)
>>
>> Commit 42314ffa072f31cc1cb44df38b1f8fcca19d9d3c should fix this.
>
> Nice!  I haven't looked closely, but does this somehow update the code
> that computes source URLs using 'string-append'?  If so, that's
> impressive :)

Well, yes and no.  :-)

It looks like that’s what it does, but in practice it simply does a
string substitution on the ‘dirname’ of the URI, which probably works in
most cases (it definitely works for CPAN packages.)

> In other news, here's another batch of bad source URLs that Hydra
> recently failed to fetch:
>
>   mirror://savannah/starfighter/1.7/project-starfighter-1.7-src.tar.gz
>   mirror://cpan/authors/id/P/PI/PIJLL/DateTime-Calendar-Julian-0.100.tar.gz
>   mirror://cpan/authors/id/A/AD/ADAMK/Object-Signature-1.08.tar.gz
>   mirror://cpan/authors/id/J/JH/JHI/Time-HiRes-1.9760.tar.gz
>   mirror://cpan/authors/id/F/FR/FREW/Class-C3-Componentised-1.001002.tar.gz
>   mirror://cpan/authors/id/R/RG/RGARCIA/base-2.23.tar.gz

Thanks!

Ludo’.

bug#35010: Many CPAN download URLs are no longer available

[Mark H Weaver]

Hi Tobias,

There are still at least two more bad URLs from this batch of perl
updates circa March 23rd:

  mirror://cpan/authors/id/A/AB/ABW/Template-Toolkit-2.28.tar.gz
  mirror://cpan/authors/id/M/MA/MAKAMAKA/Text-CSV-1.99.tar.gz

Could you take a look?

     Thanks,
       Mark

bug#35010: Many CPAN download URLs are no longer available

[Mark H Weaver]

I believe that these broken URLs are now fixed, so I'm closing this bug.

      Mark