From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id SGS1OUu4916zEQAA0tVLHw (envelope-from ) for ; Sat, 27 Jun 2020 21:21:15 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id oL6ENUu4914gCwAA1q6Kng (envelope-from ) for ; Sat, 27 Jun 2020 21:21:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1C0729404D4 for ; Sat, 27 Jun 2020 21:21:15 +0000 (UTC) Received: from localhost ([::1]:50868 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jpIGA-00057R-39 for larch@yhetil.org; Sat, 27 Jun 2020 17:21:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49736) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jpIFy-000569-PU for bug-guix@gnu.org; Sat, 27 Jun 2020 17:21:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:33162) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jpIFy-0008Iu-FZ for bug-guix@gnu.org; Sat, 27 Jun 2020 17:21:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jpIFy-0005dx-Ar for bug-guix@gnu.org; Sat, 27 Jun 2020 17:21:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#41702: `guix environment` performance issues Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 27 Jun 2020 21:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41702 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Lars-Dominik Braun Received: via spool by 41702-submit@debbugs.gnu.org id=B41702.159329283821641 (code B ref 41702); Sat, 27 Jun 2020 21:21:02 +0000 Received: (at 41702) by debbugs.gnu.org; 27 Jun 2020 21:20:38 +0000 Received: from localhost ([127.0.0.1]:44708 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpIFZ-0005cz-TX for submit@debbugs.gnu.org; Sat, 27 Jun 2020 17:20:38 -0400 Received: from eggs.gnu.org ([209.51.188.92]:35264) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpIFX-0005cl-88 for 41702@debbugs.gnu.org; Sat, 27 Jun 2020 17:20:36 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:33545) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jpIFR-0008B6-Ss; Sat, 27 Jun 2020 17:20:29 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=38494 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jpIFR-0003Ps-Az; Sat, 27 Jun 2020 17:20:29 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20200604082316.GA3146@zpidnp36> <87mu5gtbwg.fsf@gnu.org> <20200608090453.GC3166@zpidnp36> <87k10hp6ba.fsf@gnu.org> <20200609091522.GB8597@zpidnp36> Date: Sat, 27 Jun 2020 23:20:27 +0200 In-Reply-To: <20200609091522.GB8597@zpidnp36> (Lars-Dominik Braun's message of "Tue, 9 Jun 2020 11:15:22 +0200") Message-ID: <87wo3sjjdw.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -3.3 (---) X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 41702@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Spam-Score: -1.01 X-TUID: m1X+P3BxiiS4 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Lars-Dominik Braun skribis: > That would have been my best guess too, but it does not seem to be the bi= ggest > problem right now. Looking at the numbers again (both patches applied) wi= th the > attached manifest[1], I see that: > > ---snip--- > Local UNIX socket with and without --no-grafts: > N Min Max Median Avg Stdd= ev > x 10 6.07 6.35 6.145 6.16 0.082327= 26 > + 10 17.47 17.89 17.545 17.602 0.143511= 52 > Difference at 99.0% confidence > 11.442 +/- 0.150576 > 185.747% +/- 4.07133% > > Local UNIX socket vs. guix://localhost transport: > N Min Max Median Avg Stdd= ev > x 10 17.47 17.89 17.545 17.602 0.143511= 52 > + 10 17.43 18.1 17.61 17.642 0.201317= 88 > No difference proven at 99.0% confidence > > Local UNIX socket vs ssh://localhost transport: > N Min Max Median Avg Stdd= ev > x 10 17.47 17.89 17.545 17.602 0.143511= 52 > + 10 33.46 35.27 34.315 34.359 0.538732= 05 > Difference at 99.0% confidence > 16.757 +/- 0.5074 > 95.1994% +/- 3.13957% > ---snap--- > > So I would conclude: > > 1) Grafting still takes a lot of time and needs more work Yes, I noticed that there=E2=80=99s still redundant work being done, so we = may need global caching in addition to what 58bb833365db4e8934a386497d5b00a063cfd27d did. > 2) Linux optimizes localhost networking pretty well > 3) Our SSH transport is terribly slow Yes. The patch below is a noticeable improvement for me. On my laptop, GUIX_DAEMON_SOCKET=3Dssh://localhost ./pre-inst-env guix build libreoffic= e -n=20 goes from 5.8s to 3.3s. It just does the same thing as we do for guix://. Could you check what results it gives you? > So, I guess it would make sense for me to look at the SSH transport itself > again and see if there are any other low-hanging fruit. Not sure how much= I can > help with profiling guile/guix itself. A different/better RPC protocol is > probably GSoC/v2.0-worthy? There=E2=80=99s a project to rewrite the daemon in Scheme, started by Caleb Ristvedt (reepa) as part of GSoC a few years ago. This could be an opportunity to add a new version of the protocol that would support pipelining. Thanks, Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable diff --git a/guix/ssh.scm b/guix/ssh.scm index 2d7ca7d01d..a9312b7c8c 100644 --- a/guix/ssh.scm +++ b/guix/ssh.scm @@ -98,7 +98,7 @@ actual key does not match." key type)))))))) =20 (define* (open-ssh-session host #:key user port identity - host-key + host-key open-connection (compression %compression) (timeout 3600)) "Open an SSH session for HOST and return it. IDENTITY specifies the file @@ -110,6 +110,10 @@ When HOST-KEY is true, it must be a string like \"ssh-= ed25519 AAAAC3Nz=E2=80=A6 root@example.org\"; the server is authenticated and an error is raised if = its host key is different from HOST-KEY. =20 +When OPEN-CONNECTION is true, it must be a two-argument procedure; it is +passed HOST and PORT and must return a socket (a file port). When +OPEN-CONNECTION is false, Guile-SSH takes care of opening the connection. + Install TIMEOUT as the maximum time in seconds after which a read or write operation on a channel of the returned session is considered as failing. =20 @@ -134,6 +138,13 @@ Throw an error on failure." ;; Honor ~/.ssh/config. (session-parse-config! session) =20 + (when open-connection + (let* ((sock (open-connection host port))) + ;; Since 'session-set!' doesn't increase the revealed count of SOCK + ;; (as of Guile-SSH 0.12.0), do it ourselves. + (set-port-revealed! sock 1) + (session-set! session 'fd sock))) + (match (connect! session) ('ok (if host-key diff --git a/guix/store/ssh.scm b/guix/store/ssh.scm index 09c0832505..d07f0d7467 100644 --- a/guix/store/ssh.scm +++ b/guix/store/ssh.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright =C2=A9 2017 Ludovic Court=C3=A8s +;;; Copyright =C2=A9 2017, 2020 Ludovic Court=C3=A8s ;;; ;;; This file is part of GNU Guix. ;;; @@ -19,6 +19,7 @@ (define-module (guix store ssh) #:use-module (guix ssh) #:use-module (web uri) + #:use-module (ssh session) #:export (connect-to-daemon)) =20 ;;; Commentary: @@ -29,11 +30,27 @@ ;;; ;;; End: =20 +(define (open-connection host port) + "Open a connection to HOST and PORT. Use the standard SSH port if PORT = is +false." + (let* ((lst (getaddrinfo host + (if port (number->string port) "ssh") + (if port AI_NUMERICSERV 0))) + (addr (addrinfo:addr (car lst))) + (sock (socket (sockaddr:fam addr) SOCK_STREAM 0))) + ;; Setting this option makes a dramatic difference because it avoids t= he + ;; "ACK delay" on our RPC messages. + (setsockopt sock IPPROTO_TCP TCP_NODELAY 1) + + (connect sock addr) + sock)) + (define (connect-to-daemon uri) "Connect to the SSH daemon at URI, a URI object with the 'ssh' scheme." (remote-daemon-channel (open-ssh-session (uri-host uri) #:port (or (uri-port uri) 22) - #:user (uri-userinfo uri)))) + #:user (uri-userinfo uri) + #:open-connection open-connection))) =20 ;;; ssh.scm ends here --=-=-=--