[bug#58017] bug#57978: [PATCH 2/2] substitute: Retry downloading when a nar is unavailable.

[bug#58017] bug#57978: [PATCH 2/2] substitute: Retry downloading when a nar is unavailable.

[zimoun]

Hi,

On ven., 23 sept. 2022 at 08:19, Ludovic Courtès <ludo@gnu.org> wrote:

> Fixes <https://issues.guix.gnu.org/57978>
> Reported by Attila Lendvai <attila@lendvai.name>.
>
> Previously, if a narinfo was available but its corresponding nar was
> missing (for instance because the narinfo was cached and the server
> became unreachable in the meantime), 'guix substitute --substitute'
> would try to download the nar from its preferred location and abort when
> that fails.  This change forces one retry with each of the URLs.
>
> * guix/scripts/substitute.scm (download-nar): Do not catch
> 'http-get-error?' exceptions.
> (system-error?, network-error?, process-substitution/fallback): New
> procedures.
> (process-substitution): Call 'process-substitution/fallback' upon
> 'network-error?'.
> * tests/substitute.scm ("substitute, first URL has narinfo but lacks nar, second URL unauthorized")
> ("substitute, first URL has narinfo but nar is 404, both URLs authorized")
> ("substitute, first URL has narinfo but nar is 404, one URL authorized")
> ("substitute, narinfo is available but nar is missing"): New tests.

LGTM.


> +(test-equal "substitute, first URL has narinfo but nar is 404, one URL authorized"
> +  "Substitutable data."
> +  (with-narinfo*
> +      (string-append %narinfo "Signature: "
> +                     (signature-field
> +                      %narinfo
> +                      #:public-key %wrong-public-key))
> +      %main-substitute-directory
> +
> +    (with-http-server `((200 ,(string-append %narinfo "Signature: "
> +                                             (signature-field
> +                                              %narinfo
> +                                              #:public-key %wrong-public-key)))
> +                        (404 "Sorry, nar is missing!"))
> +      (let ((url1 (%local-url)))
> +        (parameterize ((%http-server-port 0))
> +          (with-http-server `((200 ,(string-append %narinfo "Signature: "
> +                                                   (signature-field %narinfo)))
> +                              (404 "Sorry, nar is missing!"))
> +            (let ((url2 (%local-url)))
> +              (dynamic-wind
> +                (const #t)
> +                (lambda ()
> +                  (parameterize ((substitute-urls
> +                                  (list url1 url2
> +                                        (string-append "file://"
> +                                                       %main-substitute-directory))))
> +                    (request-substitution (string-append (%store-prefix)
> +                                                         "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-foo")
> +                                          "substitute-retrieved"))
> +                  (call-with-input-file "substitute-retrieved" get-string-all))
> +                (lambda ()
> +                  (false-if-exception (delete-file "substitute-retrieved")))))))))))

Although I do not understand this test.  Why is 404 appearing twice?


Cheers,
simon

[bug#58017] bug#57978: [PATCH 2/2] substitute: Retry downloading when a nar is unavailable.

[Ludovic Courtès]

Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

>> +  (with-narinfo*
>> +      (string-append %narinfo "Signature: "
>> +                     (signature-field
>> +                      %narinfo
>> +                      #:public-key %wrong-public-key))
>> +      %main-substitute-directory
>> +
>> +    (with-http-server `((200 ,(string-append %narinfo "Signature: "
>> +                                             (signature-field
>> +                                              %narinfo
>> +                                              #:public-key %wrong-public-key)))
>> +                        (404 "Sorry, nar is missing!"))
>> +      (let ((url1 (%local-url)))
>> +        (parameterize ((%http-server-port 0))
>> +          (with-http-server `((200 ,(string-append %narinfo "Signature: "
>> +                                                   (signature-field %narinfo)))
>> +                              (404 "Sorry, nar is missing!"))
>> +            (let ((url2 (%local-url)))
>> +              (dynamic-wind
>> +                (const #t)
>> +                (lambda ()
>> +                  (parameterize ((substitute-urls
>> +                                  (list url1 url2
>> +                                        (string-append "file://"
>> +                                                       %main-substitute-directory))))

[...]

> Although I do not understand this test.  Why is 404 appearing twice?

That’s because it’s testing with 3 substitute URLs.

Thanks for taking a look!

Ludo’.

[bug#58017] [PATCH 2/2] substitute: Retry downloading when a nar is unavailable.

[Ludovic Courtès]

Hi Maxime,

Maxime Devos <maximedevos@telenet.be> skribis:

>> +(test-equal "substitute, first URL has narinfo but nar is 404, both URLs authorized"
>> +  "Substitutable data."
>> +  (with-narinfo*
>> +      (string-append %narinfo "Signature: "
>> +                     (signature-field %narinfo))
>> +      %main-substitute-directory
>> +
>> +    (with-http-server `((200 ,(string-append %narinfo "Signature: "
>> +                                             (signature-field %narinfo)))
>> +                        (404 "Sorry, nar is missing!"))
>> +      (dynamic-wind
>> +        (const #t)
>> +        (lambda ()
>> +          (parameterize ((substitute-urls
>> +                          (list (%local-url)
>> +                                (string-append "file://"
>> +                                               %main-substitute-directory))))
>> +            (request-substitution (string-append (%store-prefix)
>> +                                                 "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-foo")
>> +                                  "substitute-retrieved"))
>> +          (call-with-input-file "substitute-retrieved" get-string-all))
>> +        (lambda ()
>> +          (false-if-exception (delete-file "substitute-retrieved")))))))
>
> Shouldn't it only ignore 'file not found' (ENOENT?) exceptions?

By “it”, do you mean ‘dynamic-wind’ should be replaced by a ‘catch’
form?

We could discuss it, but note that this patch just keeps with the style
of existing tests.

> This test, and some others, can be improved by also checking the
> URI. While currently 'with-http-server' does not support that, there
> are (5 months, with the v1 having seen some reviewing and a v2
> available) patches for that at <https://issues.guix.gnu.org/53389>.
>
> That patch also _requires_ always mentioning the URI, if the cover
> letter is correct.  It also allows simplifying the use of '%local-url'
> a bit.

Ah, thanks for the reminder!  I’ve just spent most of the day reviewing
patches, but not that one…

Ludo’.