From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id 2KnkOY1byGOhKAEAbAwnHQ (envelope-from ) for ; Wed, 18 Jan 2023 21:50:22 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id GCEGOY1byGPc2AAAG6o9tA (envelope-from ) for ; Wed, 18 Jan 2023 21:50:21 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9DB75859 for ; Wed, 18 Jan 2023 21:50:21 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pIFNm-0008Pp-3t; Wed, 18 Jan 2023 15:50:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pIFNj-0008Pg-54 for bug-guix@gnu.org; Wed, 18 Jan 2023 15:50:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pIFNi-0002IK-Gq for bug-guix@gnu.org; Wed, 18 Jan 2023 15:50:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pIFNi-00047T-1F for bug-guix@gnu.org; Wed, 18 Jan 2023 15:50:02 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#56709: Channel opening failure with guix deploy References: <87fsituexa.fsf@posteo.org> In-Reply-To: <87fsituexa.fsf@posteo.org> Resent-From: "Thompson, David" Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 18 Jan 2023 20:50:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 56709 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 56709@debbugs.gnu.org Received: via spool by 56709-submit@debbugs.gnu.org id=B56709.167407498015801 (code B ref 56709); Wed, 18 Jan 2023 20:50:01 +0000 Received: (at 56709) by debbugs.gnu.org; 18 Jan 2023 20:49:40 +0000 Received: from localhost ([127.0.0.1]:41909 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pIFNM-00046n-CI for submit@debbugs.gnu.org; Wed, 18 Jan 2023 15:49:40 -0500 Received: from mail-qt1-f169.google.com ([209.85.160.169]:38721) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pIFNJ-00046Z-SI for 56709@debbugs.gnu.org; Wed, 18 Jan 2023 15:49:38 -0500 Received: by mail-qt1-f169.google.com with SMTP id z9so8227241qtv.5 for <56709@debbugs.gnu.org>; Wed, 18 Jan 2023 12:49:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=worcester-edu.20210112.gappssmtp.com; s=20210112; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=Ayyh+qTk8w/Xvo2+GOFGdwN+68hezI4cgSTIhLwqXec=; b=g/P+nj2ymRdzcOQfMbUf2/nWT+WoZxoJIunHgG8uU7R9jzfJsZMza9MPZaiHaVyE1C vde1aZnHXqygHCiBrkBkCVAyLr4vGRYgzIErvzs3YBxscRvz05kUCLtAaVTi4t/XvRtQ lw3h7oKteOJai23pGWyAhyEds43cMHWsULDPvQi7qZy+j+KZ7k2GPSjZrBPsVcsNLUGd o3LSUj46TPz7256ZiOLiIt6G/pZ3E3ggKKCJUyUp59NvWPEa5Z6wFzHEOwke3NUzWyx3 9H/4G1VcsmWA0AaE0gThVx3bpkDo/Rw5OQcZDECNAmSplCce49uUXHNULGVPiIFuaZUB 50sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Ayyh+qTk8w/Xvo2+GOFGdwN+68hezI4cgSTIhLwqXec=; b=EWYrqtVq6ehWBoF/2reAGNP+YI1kJ40HylRodDXg8Y++hVHU8Xo5PHC2bXdhpm7asy ZC4Go/sV/hEBz+m+GrbLwx9rnOJSVpwSHCQtK8HjhwcP7jpET6n+Bhfz6CdaKNmbZEqN I5Uaybjano+7Jw+ky56JPXN1CTAzosvLug0hQgV2hKoga8I1byFwqKFhas575DwLUkHH NSjQgc0YaATUHtPH6SdpyEGibpSPDnIlR2axfdw70e/efKUCwLvYckDD02eZwqSGcFfA 2nCFdj1S8CtHP6/PME2LLFutDrhAueNXBg+SIkYtO84g6ogwvjQhVZyjymajvOIWw0c7 qIPw== X-Gm-Message-State: AFqh2kpGdl2MFuWtE3Iwb/f0DhIojkmpj/yJhZVG+H0QMPYWPo6Rp6Ik sCNIYDhSRK8S0uEw1RwRw+d8EK8KxkxjlisqumoLyuFkxvyBJMpn X-Google-Smtp-Source: AMrXdXvw/DfGV2AHD0AD4M5A98X+GjNydVvDJaEkOIkqjbxQIsHXzkkdQzCgYc/0vXouv6WREvkDbNvDlkK+VxHqpEg= X-Received: by 2002:ac8:5ecd:0:b0:3b1:221c:c72 with SMTP id s13-20020ac85ecd000000b003b1221c0c72mr359404qtx.569.1674074971512; Wed, 18 Jan 2023 12:49:31 -0800 (PST) MIME-Version: 1.0 From: "Thompson, David" Date: Wed, 18 Jan 2023 15:49:20 -0500 Message-ID: Content-Type: text/plain; charset="UTF-8" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1674075021; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=Ayyh+qTk8w/Xvo2+GOFGdwN+68hezI4cgSTIhLwqXec=; b=eiSvA3myJZd3EmqQtmIfBs4+/WByFEQsJx7f4Px53QLzCvEwoPlofpb/4bx+lREjRawC5E tD8cva149BK3iiEXj6fHE6p1ZWG4i2rgYgqaNGZZXP5k/CoAmrSQQ6Lrw9zZRplNO5/81A ANEPEdhbC7hFD0XXGqpTLlpK76tQQRUzOJnC0DLdBiO1f+L0V79xTPN/97J3/rPY9WGPl5 jXo1UHD03J2l5/VNLHJwHLrbM/rn4702vZZAx+ZvB0xClLoIbQoiZcSAE6Dh8dvc981F1o 85qSPlkPpDd4BLiHweUkCe6jqvqEeR+LtQD/xKhityYiTBeOderTuexIdf9d3A== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=worcester-edu.20210112.gappssmtp.com header.s=20210112 header.b="g/P+nj2y"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1674075021; a=rsa-sha256; cv=none; b=roV6jcZDsV4+sLnxv5ZnyvfEa0McToyOg0OCLjs6DCeEKARnSxmrkdS1UUx88kzkV/xl0P quL29pku5rkNLbmD7hQPKZrQtMSGptDoJApvQyPoLXS3sa2vkQtTS6l3IZVW7aN/nM3s46 hutM1UlNoclOewbe5bL0YeZoRGXmvrdUfnYfl51v98SLrMMhVQ/G7xXKDOfwuLRjSoDD1n bT6F0iGR/ONIoRRmgCfYqgukY6lhuFj6Vqphk6OFPeQSxfe0uWJS4QUUQVXHjzKm5cfru6 CcNilO/2fmYJp6Ntgrhhombc7lteDwhXaPGQfFXIEz3XTkE7aWyvT+I3r7cd9g== X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: 2.37 X-Spam-Score: 2.37 X-Migadu-Queue-Id: 9DB75859 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=worcester-edu.20210112.gappssmtp.com header.s=20210112 header.b="g/P+nj2y"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-TUID: nDW8EMp2/gZg Hello, This problem is strangely transient. I've seen it happen to others when it wasn't happening to me with the same remote machine. Now I am having this problem again on 2 different servers that I manage. I dug around a bit and found that calls to 'open-remote-pipe*' from guile-ssh have some chance of failure even though the SSH session is fine. This procedure is called many times during a deploy, so the odds are high that one of them will fail. I got lucky once today and had a deploy finish but that was after many failures. I was able to unblock myself by hacking call sites to repeatedly call 'open-remote-pipe*' in a loop, like this: (let loop () (or (false-if-exception (apply open-remote-pipe* session OPEN_BOTH repl-command)) (loop))) I also added some 'pk' logging and found that 'open-remote-pipe*' would typically succeed on the first or second try. I think there could be a bit more investigation done to better understand *why* this happens in the first place, but as a resiliency tactic I think it would be appropriate to write a wrapper procedure that retries a few times before giving up. Thoughts? - Dave