From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id CD0GCuQ6/l/WZAAA0tVLHw (envelope-from ) for ; Wed, 13 Jan 2021 00:12:20 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id KIq2BeQ6/l9qBgAAbx9fmQ (envelope-from ) for ; Wed, 13 Jan 2021 00:12:20 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DC0C29403D6 for ; Wed, 13 Jan 2021 00:12:18 +0000 (UTC) Received: from localhost ([::1]:50460 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kzTlo-0004JS-8w for larch@yhetil.org; Tue, 12 Jan 2021 19:12:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53234) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kzTla-0004JF-0L for bug-guix@gnu.org; Tue, 12 Jan 2021 19:12:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50432) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kzTlZ-00079g-Pi for bug-guix@gnu.org; Tue, 12 Jan 2021 19:12:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kzTlZ-00045n-KZ; Tue, 12 Jan 2021 19:12:01 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#45828: guix build: error: got unexpected path `Backtrace:' from substituter Resent-From: Julien Lepiller Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 13 Jan 2021 00:12:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45828 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Christopher Baines Received: via spool by 45828-submit@debbugs.gnu.org id=B45828.161049671115715 (code B ref 45828); Wed, 13 Jan 2021 00:12:01 +0000 Received: (at 45828) by debbugs.gnu.org; 13 Jan 2021 00:11:51 +0000 Received: from localhost ([127.0.0.1]:33745 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kzTlO-00045O-UX for submit@debbugs.gnu.org; Tue, 12 Jan 2021 19:11:51 -0500 Received: from lepiller.eu ([89.234.186.109]:60282) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kzTlL-00045F-T8 for 45828@debbugs.gnu.org; Tue, 12 Jan 2021 19:11:49 -0500 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id 459e96ce; Wed, 13 Jan 2021 00:11:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=lepiller.eu; h=date:from :to:cc:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=dkim; bh=J3mYxA4yfWTi JJ5DYgpT5jfMqdSoWOavKxy3k8FG1hA=; b=WQ1JwAkp9ynEBkGtzj32btdH6qBK LaG06hlo70xjsrAybK7PLfgCJjuLy6ql01vvTKP0CNNvEmU9C+BwTFUJQDNSsadH 1w3cDYFZaN00xio2BDe8ii+J1GybJ/W3ny8MmcByC7qkqhbTGLU1VGjEMJbLFkSE vNQyZEuPuAPtIDirVvLILAVWCEQpwQorJrpIxdcpM7MF8admvtzzZCCYpSJS0MY2 iknH3j5WENHVfvLDBIBzugADt7lwnUI4Y626i1d4g8LHj3HiWvD9zxTjg5hhiPt1 cYJgstnoDxKJYoawCVxBzSD4l2LQa2PcxaLTfOd2aM+hHK3GvGuoef3mVQ== Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id ca12f73c (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Wed, 13 Jan 2021 00:11:42 +0000 (UTC) Date: Wed, 13 Jan 2021 01:11:35 +0100 From: Julien Lepiller Message-ID: <20210113011135.760c4faa@tachikoma.lepiller.eu> In-Reply-To: <87turlai45.fsf@cbaines.net> References: <87turlai45.fsf@cbaines.net> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 45828@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.26 Authentication-Results: aspmx1.migadu.com; dkim=fail (headers rsa verify failed) header.d=lepiller.eu header.s=dkim header.b=WQ1JwAkp; dmarc=fail reason="SPF not aligned (relaxed)" header.from=lepiller.eu (policy=none); spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: DC0C29403D6 X-Spam-Score: -1.26 X-Migadu-Scanner: scn1.migadu.com X-TUID: c8pkEnL2Mhbz Le Tue, 12 Jan 2021 22:46:34 +0000, Christopher Baines a =C3=A9crit : > Leo Famulari writes: >=20 > > Recently, many people on the #guix IRC channel reported frequent > > non-deterministic failures of any operation involving substitution, > > like this: > > > > ------ > > $ ./pre-inst-env guix build --no-grafts poezio mpdris2 sonata > > beets-bandcamp beets substitute:=20 > > guix build: error: got unexpected path `Backtrace:' from substituter > > ------ > > > > `guix describe` reports commit > > b4384e61165623b16b77b8cab16c81423c6853ed for both my user's Guix > > and the guix-dameon. =20 >=20 > I might have managed to reproduce the error happening on the daemon > side: >=20 > =E2=86=92 > /gnu/store/4j8vn0gbqz5adj1y02nnwcfwmqsjgj8s-guix-1.2.0-6.799f066/bin/guix > substitute --query info > /gnu/store/3c01q1f16kljfry70qjg6cs6k8winfzg-guix-package-cache > /gnu/store/6lk8anal4s62gk3d30vgxppykbd5jcfj-guix-85e97c969 > /gnu/store/9zl2zbh3q2jnbfvxgnhw8j3f637ni7z4-guix-cli > /gnu/store/ihricijvy16zwkd2n671xlyrn02sqhf9-guix-manual > /gnu/store/m3j427qnlp81vsdj3x9ds7s4i051r1vz-guix-system-tests > /gnu/store/mbv9j7wwqvwnr5awzbi126jdsj3h64h5-guix-packages > /gnu/store/n2m1ay7kpa5f4fls4vvcy46ar1fdl0wk-guix-system > /gnu/store/p4q9ajlb3l7x8xglqs6fflch2iwjqwaj-guix-module-union > /gnu/store/snhx33fgjj2xnc5vy96sr3c8jqw9c7s0-guix-85e97c969-modules > /gnu/store/vnrlvz9pxl5qrpy5x8y51v6awz7yzn8q-guix-packages-base > /gnu/store/z4wj18vyzaas2yqb0577cc3japy4fi7z-guix-config > /gnu/store/zdjfbsj1a94vdbbg9r0cx4jcqnwxazxs-guix-translated-texinfo > Backtrace: In ice-9/boot-9.scm: 1736:10 5 (with-exception-handler _ > _ #:unwind? _ # _) In unknown file: 4 (apply-smob/0 # 7f23d4f2e380>) In ice-9/boot-9.scm: 718:2 3 (call-with-prompt _ _ > 7f23d4f2e380>#) In ice-9/eval.s= cm: > 7f23d4f2e380>619:8 2 (_ #(#(# 7f23d4f2e380>7f23d4b70f00>))) In guix/ui.scm: 2127:12 1 > 7f23d4f2e380>7f23d4b70f00>(run-guix-command _ . _) In > 7f23d4f2e380>7f23d4b70f00>guix/scripts/substitute.scm: 1256:4 0 > 7f23d4f2e380>7f23d4b70f00>(guix-substitute . _) >=20 > guix/scripts/substitute.scm:1256:4: In procedure guix-substitute: > Throw to key `bad-response' with args `("Bad Response-Line: ~s" > (""))'. >=20 >=20 > It's hard to tell if that's actually consistent with the error > though. Repeating the same test after the restart of guix-publish on > ci.guix.gnu.org works without printing a backtrace. During the issue, I tried to manually check what berlin had to say: $ curl https://ci.guix.gnu.org/3c01q1f16kljfry70qjg6cs6k8winfzg.narinfo -D- HTTP/1.1 500 Internal Server Error Server: nginx Date: Tue, 12 Jan 2021 22:34:01 GMT Transfer-Encoding: chunked Connection: keep-alive and after the restart: $ curl https://ci.guix.gnu.org/3c01q1f16kljfry70qjg6cs6k8winfzg.narinfo -D- HTTP/1.1 404 Not Found Server: nginx Date: Tue, 12 Jan 2021 22:34:44 GMT Content-Type: text/plain;charset=3Dutf-8 Content-Length: 61 Connection: keep-alive Resource not found: /3c01q1f16kljfry70qjg6cs6k8winfzg.narinfo So I tried to learn more about what happens, so I put the first response in a file, `response`. I used netcat to provide the file over the network: nc -l -p 8080 < response then I tried the following: $ guix build /gnu/store/3c01q1f16kljfry70qjg6cs6k8winfzg-random --substitute-urls=3Dhttp://localhost:8080 substitute:=20 guix build: error: got unexpected path `Backtrace:' from substituter Then, I restarted the daemon to pass it this substitute url: # guix-daemon --build-users-group=3Dguixbuild --substitute-urls=3Dhttp://localhost:8080 and from another terminal: $ /gnu/store/wr0shvj2dy8h8w2m1pil9r9798ai6nyy-guix-command substitute --query info /gnu/store/3c01q1f16kljfry70qjg6cs6k8winfzg-guix-package-cache Backtrace: 2 (primitive-load "/gnu/store/wr0shvj2dy8h8w2m1pil9r9798a=E2=80= =A6") In guix/ui.scm: 2127:12 1 (run-guix-command _ . _) In guix/scripts/substitute.scm: 1256:4 0 (guix-substitute . _) guix/scripts/substitute.scm:1256:4: In procedure guix-substitute: In procedure =3D: Wrong type argument in position 1: #f So my backtrace is different. For some reason, it seems that guile cannot read anything from the body of that response, but I don't know what's happening: In the REPL, the following: ,m (guix scripts substitute) (call-with-input-file "response" (lambda (port) (let* ((r (read-response port)) (body (response-body-port r))) (pk 'body body 'r r) (get-u8 body)))) Gives a backtrace: ;;; (body # r #< version: (1 . 1) code: 500 reason-phrase: "Internal Server Error Server: nginx" headers: ((date . #) (transfer-encoding (chunked)) (connection keep-alive)) port: #>) ice-9/boot-9.scm:1669:16: In procedure raise-exception: In procedure =3D: Wrong type argument in position 1: #f In ice-9/ports.scm: 445:17 3 (call-with-input-file _ _ #:binary _ #:encoding _ # _) In unknown file: 2 (get-u8 #) In web/http.scm: 1978:17 1 (read! #vu8(115 99 104 101 109 101 64 40 103 117 105 =E2=80=A6= ) =E2=80=A6) In ice-9/boot-9.scm: 1669:16 0 (raise-exception _ #:continuable? _) HTH!