bug#63634: nar 404 leads to hard ‘guix substitute’ crash

["Ludovic Courtès" <ludo@gnu.org>]

I experienced it a second time and have more info:

--8<---------------cut here---------------start------------->8---
substituting /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0...
Backtrace:
In ice-9/boot-9.scm:
  1752:10 19 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          18 (apply-smob/0 #<thunk 7f4f9bf172a0>)
In ice-9/boot-9.scm:
    724:2 17 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8 16 (_ #(#(#<directory (guile-user) 7f4f9bf1cc80>)))
In guix/ui.scm:
   2300:7 15 (run-guix . _)
  2263:10 14 (run-guix-command _ . _)
In ice-9/boot-9.scm:
  1752:10 13 (with-exception-handler _ _ #:unwind? _ # _)
  1752:10 12 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
   854:15 11 (_)
    646:2 10 (process-substitution _ _ _ #:cache-urls _ #:acl _ # _ # …)
In ice-9/boot-9.scm:
  1752:10  9 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
    463:7  8 (download-nar #<<narinfo> path: "/gnu/store/apw1y9nf8r…> …)
In ice-9/boot-9.scm:
  1747:15  7 (with-exception-handler #<procedure 7f4f9846af00 at ic…> …)
  1685:16  6 (raise-exception _ #:continuable? _)
  1683:16  5 (raise-exception _ #:continuable? _)
  1685:16  4 (raise-exception _ #:continuable? _)
  1780:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
  1685:16  2 (raise-exception _ #:continuable? _)
  1683:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
ERROR:
  1. &http-get-error:
      uri: #<<uri> scheme: https userinfo: #f host: "bordeaux.guix.gnu.org" port: #f path: "/nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0" query: #f fragment: #f>
      code: 404
      reason: "Not Found"
      headers: ((server . "nginx") (date . #<date nanosecond: 0 second: 31 minute: 59 hour: 8 day: 22 month: 5 year: 2023 zone-offset: 0>) (content-type text/plain (charset . "utf-8")) (content-length . 3) (connection keep-alive))
  2. &message: "https://bordeaux.guix.gnu.org/nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0: HTTP download failed: 404 (\"Not Found\")"
substitution of /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0 failed
guix build: error: corrupt input while restoring archive from #<closed: file 7f1693f9c9a0>
--8<---------------cut here---------------end--------------->8---

I have ‘--substitute-urls=https://ci.guix.gnu.org https://bordeaux.guix.gnu.org’.
Let’s see what’s available on each server:

--8<---------------cut here---------------start------------->8---
$ wget -O- https://ci.guix.gnu.org/apw1y9nf8rqgxvjnlr1isbhpd502bcs5.narinfo
--2023-05-22 11:00:08--  https://ci.guix.gnu.org/apw1y9nf8rqgxvjnlr1isbhpd502bcs5.narinfo
Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40
Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-05-22 11:00:08 ERROR 404: Not Found.
$ wget -O- https://bordeaux.guix.gnu.org/apw1y9nf8rqgxvjnlr1isbhpd502bcs5.narinfo
--2023-05-22 11:03:36--  https://bordeaux.guix.gnu.org/apw1y9nf8rqgxvjnlr1isbhpd502bcs5.narinfo
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 185.233.100.56, 2a0c:e300::58
Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1715 (1.7K) [text/plain]
Saving to: ‘STDOUT’

-                                     0%[                                                                   ]       0  --.-KB/s               StorePath: /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
NarHash: sha256:1am3v254jr9dpgd9r4dkjqxfgbp1xm0vl1hxl49cmmh97dg117z8
NarSize: 144790464
References: 8rz7yh6zdzp4b78f4n9wqj3hav2md4d4-isl-0.24 930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0 brzrf6qx5d83j1zvirc1xk14wrhyx4hf-binutils-cross-aarch64-linux-gnu-2.38 cinj0g8krh5s428jawsazawlfarp912k-gcc-cross-aarch64-linux-gnu-11.3.0-lib cs3hw1wnxgijjzsd61whc8ar3qy9wjd6-mpfr-4.2.0 gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35 ib2n2vzqpchc3bhh9i712w5sq9zapn8d-gmp-6.2.1 rbjq50q2ik8c1glkj5f0ksnwfz64g16g-mpc-1.3.1 rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16 rjgpaj54q1q9n3hcvpzyp29fgi2nx6ls-ld-wrapper-aarch64-linux-gnu-0 slzq3zqwj75lbrg4ly51hfhbv2vhryv5-zlib-1.2.13 y4ndp9n5krkaiwgvw58c61n7h8ls9b0f-glibc-cross-aarch64-linux-gnu-2.35
System: x86_64-linux
Deriver: fqb09jygm70cmq03xmjm7al6gzajp1p5-gcc-cross-aarch64-linux-gnu-11.3.0.drv
Signature: 1;bayfront;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNzA2MTI1QUFGMjcxQzM2MDM3RkY2MTIwMzI0NzI2NTVDMkU3OEY2NTA3OUY0MzgxMzY4NDhDQzA0OTVFRTIxRiMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDg1RTA2ODE3RTFENTdCQzE1NTAxOUNGNjA0QTQxMTlFRjNBREVGQ0VFOTE4NjQzMDIzQTExQzk4RDM3REVBOCMpCiAgIChzICMwMUIwMkQyNjI0MDhCMDIzMUMxMUVDODI1Q0Q2MTkzNTU5NDRCQUNBMzg4NjEwRDIxOEIzNTVCNjM1NjFCQjk3IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjN0Q2MDI5MDJEM0EyREJCODNGOEEwRkI5ODYwMkE3NTRDNTQ5M0IwQjc3OEM4RDFERDRFMEY0MURFMTRERTM0RiMpCiAgICkKICApCiApCg==
URL: nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
Compression: lzip
FileSize: 24767829
-                                   100%[==================================================================>]   1.67K  --.-KB/s    in 0s      

2023-05-22 11:03:36 (20.6 MB/s) - written to stdout [1715/1715]

$ wget -O/dev/null https://bordeaux.guix.gnu.org/nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
--2023-05-22 11:04:02--  https://bordeaux.guix.gnu.org/nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 185.233.100.56, 2a0c:e300::58
Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24767829 (24M) [text/plain]
Saving to: ‘/dev/null’

/dev/null                           100%[==================================================================>]  23.62M  44.8MB/s    in 0.5s    

2023-05-22 11:04:03 (44.8 MB/s) - ‘/dev/null’ saved [24767829/24767829]

$ wget -O/dev/null https://bordeaux.guix.gnu.org/nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
--2023-05-22 11:04:28--  https://bordeaux.guix.gnu.org/nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 185.233.100.56, 2a0c:e300::58
Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-05-22 11:04:28 ERROR 404: Not Found.
--8<---------------cut here---------------end--------------->8---

So it seems ‘guix substitute’ picked /nar/zstd, even though
bordeaux.guix is not advertising that.

Or is it?

--8<---------------cut here---------------start------------->8---
$ sudo cat /var/guix/substitute/cache/kzwjeblndsbkjzmjailrt4bnhguil7tqjmewzcyw22hgajbhfy3q/apw1y9nf8rqgxvjnlr1isbhpd502bcs5
(narinfo (version 2) (cache-uri "https://bordeaux.guix.gnu.org") (date 1683633757) (ttl 15552000) (value "StorePath: /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nNarHash: sha256:1am3v254jr9dpgd9r4dkjqxfgbp1xm0vl1hxl49cmmh97dg117z8\nNarSize: 144790464\nReferences: 8rz7yh6zdzp4b78f4n9wqj3hav2md4d4-isl-0.24 930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0 brzrf6qx5d83j1zvirc1xk14wrhyx4hf-binutils-cross-aarch64-linux-gnu-2.38 cinj0g8krh5s428jawsazawlfarp912k-gcc-cross-aarch64-linux-gnu-11.3.0-lib cs3hw1wnxgijjzsd61whc8ar3qy9wjd6-mpfr-4.2.0 gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35 ib2n2vzqpchc3bhh9i712w5sq9zapn8d-gmp-6.2.1 rbjq50q2ik8c1glkj5f0ksnwfz64g16g-mpc-1.3.1 rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16 rjgpaj54q1q9n3hcvpzyp29fgi2nx6ls-ld-wrapper-aarch64-linux-gnu-0 slzq3zqwj75lbrg4ly51hfhbv2vhryv5-zlib-1.2.13 y4ndp9n5krkaiwgvw58c61n7h8ls9b0f-glibc-cross-aarch64-linux-gnu-2.35\nSystem: x86_64-linux\nDeriver: fqb09jygm70cmq03xmjm7al6gzajp1p5-gcc-cross-aarch64-linux-gnu-11.3.0.drv\nSignature: 1;bayfront;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNzA2MTI1QUFGMjcxQzM2MDM3RkY2MTIwMzI0NzI2NTVDMkU3OEY2NTA3OUY0MzgxMzY4NDhDQzA0OTVFRTIxRiMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDg1RTA2ODE3RTFENTdCQzE1NTAxOUNGNjA0QTQxMTlFRjNBREVGQ0VFOTE4NjQzMDIzQTExQzk4RDM3REVBOCMpCiAgIChzICMwMUIwMkQyNjI0MDhCMDIzMUMxMUVDODI1Q0Q2MTkzNTU5NDRCQUNBMzg4NjEwRDIxOEIzNTVCNjM1NjFCQjk3IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjN0Q2MDI5MDJEM0EyREJCODNGOEEwRkI5ODYwMkE3NTRDNTQ5M0IwQjc3OEM4RDFERDRFMEY0MURFMTRERTM0RiMpCiAgICkKICApCiApCg==\nURL: nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nCompression: lzip\nFileSize: 24767829\nURL: nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nCompression: zstd\nFileSize: 39822104"))ludo@ribbon ~/src/guix [env]$ 
--8<---------------cut here---------------end--------------->8---

So it seems that bordeaux.guix did advertise zstd at some point (and
that narinfo is still in cache) but no longer does.

Chris, can you confirm?

I’ll see how to get ‘guix substitute’ to handle that gracefully.

Ludo’.