unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / code / Atom feed
From: Lars-Dominik Braun <ldb@leibniz-psychology.org>
To: "Ludovic Courtès" <ludo@gnu.org>
Cc: 41702@debbugs.gnu.org
Subject: bug#41702: `guix environment` performance issues
Date: Tue, 9 Jun 2020 11:15:22 +0200	[thread overview]
Message-ID: <20200609091522.GB8597@zpidnp36> (raw)
In-Reply-To: <87k10hp6ba.fsf@gnu.org>


[-- Attachment #1.1: Type: text/plain, Size: 3682 bytes --]

Hey,

> That’s over SSH, right?
correct, the worst possible case: Inside two VM’s on a Laptop, SSH transport
between them and /gnu+/var/guix on an NFS share (nfsd is in the same VM as
guix-daemon).

> Probably what’s killing us is the round-trip time for all these small
> RPCs.  We would need pipelining but the RPC protocol is not designed to
> make that easy.
That would have been my best guess too, but it does not seem to be the biggest
problem right now. Looking at the numbers again (both patches applied) with the
attached manifest[1], I see that:

---snip---
Local UNIX socket with and without --no-grafts:
    N           Min           Max        Median           Avg        Stddev
x  10          6.07          6.35         6.145          6.16    0.08232726
+  10         17.47         17.89        17.545        17.602    0.14351152
Difference at 99.0% confidence
        11.442 +/- 0.150576
        185.747% +/- 4.07133%

Local UNIX socket vs. guix://localhost transport:
    N           Min           Max        Median           Avg        Stddev
x  10         17.47         17.89        17.545        17.602    0.14351152
+  10         17.43          18.1         17.61        17.642    0.20131788
No difference proven at 99.0% confidence

Local UNIX socket vs ssh://localhost transport:
    N           Min           Max        Median           Avg        Stddev
x  10         17.47         17.89        17.545        17.602    0.14351152
+  10         33.46         35.27        34.315        34.359    0.53873205
Difference at 99.0% confidence
        16.757 +/- 0.5074
        95.1994% +/- 3.13957%
---snap---

So I would conclude:

1) Grafting still takes a lot of time and needs more work
2) Linux optimizes localhost networking pretty well
3) Our SSH transport is terribly slow

Moving to non-localhost communication between two VM’s:

---snip---
guix://localhost vs. guix://remote-host transport:
    N           Min           Max        Median           Avg        Stddev
x  10         17.43          18.1         17.61        17.642    0.20131788
+  10         20.88         22.58        21.095        21.222    0.49689704
Difference at 99.0% confidence
        3.58 +/- 0.487934
        20.2925% +/- 2.85159%

guix://remote-host vs. ssh://remote-host:
    N           Min           Max        Median           Avg        Stddev
x  10         20.88         22.58        21.095        21.222    0.49689704
+  10          30.1         32.56        31.005        31.093    0.70740606
Difference at 99.0% confidence
        9.871 +/- 0.786769
        46.5131% +/- 4.35326%
---snap---

Conclusion here is the same: Not alot of impact of networking/NFS and SSH
transport is still terribly slow. (Confusingly faster than localhost though.)

> Perhaps you could “strace -Tt” the thing to check whether this
> hypothesis is correct by looking at the time we spend waiting for
> replies?
I’m not sure this will yield meaningful data for SSH, so I analyzet it for
guix://localhost vs. guix://remote-host. Takeaway is, yes, of course there is a
statistically significant difference and it’s about 40%±50%, which means this
method is pretty useless, because we can’t bin RPC’s by type.

So, I guess it would make sense for me to look at the SSH transport itself
again and see if there are any other low-hanging fruit. Not sure how much I can
help with profiling guile/guix itself. A different/better RPC protocol is
probably GSoC/v2.0-worthy?

Sorry for all the lengthy emails,
Lars

[1] You’ll need this channel: https://github.com/leibniz-psychology/guix-zpid


[-- Attachment #1.2: manifest.scm --]
[-- Type: text/plain, Size: 881 bytes --]

(specifications->manifest
 '(
	 "coreutils"
	 "findutils"
	 "procps"
	 "strace"
	 "openssh"
	 "mit-krb5"
	 "bash"
	 "which"
	 "zip"
	 "geeqie"
	 "util-linux"
	 "grep"
	 "glibc"
	 ;; for locales
	 "glibc-locales"
	 ;; front-end software
	 "jupyter-zpid"
	 "python-jupyterlab"
	 ;; available notebook kernel
	 ; provided by jupyter-zpid
 ;"python-ipykernel"
 "r-irkernel"
 "rstudio-server-zpid"
 "r"
 ;; for RMarkdown
 "r-knitr"
 "r-yaml"
 "r-markdown"
 "r-rmarkdown"
 "texlive"
 ;; commonly used r packages
 "r-psych"
 "r-ggplot2"
 "r-lattice"
 "r-foreign"
 "r-readr"
 "r-haven"
 "r-dplyr"
 "r-tidyr"
 "r-stringr"
 "r-forecast"
 "r-lme4"
 "r-nlme"
 "r-nnet"
 "r-glmnet"
 "r-caret"
 "r-xmisc"
 "r-splitstackshape"
 "r-tm"
 "r-quanteda"
 "r-topicmodels"
 "r-stm"
 ;;"r-parallel"
 "r-dt"
 "r-nlp"
 "r-data-table"
 "r-hmisc"
 "r-learnr"
 "r-metafor"
 ;; for rmarkdown
 "ghc-pandoc"
 ))

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

  reply	other threads:[~2020-06-09  9:16 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-04  8:23 bug#41702: `guix environment` performance issues Lars-Dominik Braun
2020-06-06 16:08 ` Ludovic Courtès
2020-06-06 21:40   ` Ludovic Courtès
2020-06-08  9:04   ` Lars-Dominik Braun
2020-06-08 21:59     ` Ludovic Courtès
2020-06-09  9:15       ` Lars-Dominik Braun [this message]
2020-06-27 21:20         ` Ludovic Courtès
2020-06-30  9:59           ` Lars-Dominik Braun
2020-07-01 10:49           ` Ludovic Courtès
2020-07-01 11:24             ` Lars-Dominik Braun
2020-07-01 21:53               ` Ludovic Courtès
2020-07-02  7:00                 ` Lars-Dominik Braun
2020-07-02 12:03                   ` Lars-Dominik Braun
2020-07-06  8:49                     ` Ludovic Courtès
2020-07-06 12:58                       ` Lars-Dominik Braun
2020-07-20  9:50                         ` Lars-Dominik Braun
2020-07-20 21:51                           ` Ludovic Courtès
2020-10-22 21:03                             ` Maxim Cournoyer
2020-10-23  7:26                               ` Lars-Dominik Braun
2020-10-23 13:07                                 ` bug#44175: [optimization] Grafting is too slow Maxim Cournoyer
2020-10-23 21:17                                   ` Ludovic Courtès
2020-10-25  2:33                                     ` Maxim Cournoyer
2020-10-25 16:43                                       ` Ludovic Courtès
2020-10-26  7:56                                         ` Lars-Dominik Braun
2020-10-23 14:51                                 ` bug#41702: `guix environment` performance issues Ludovic Courtès

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://guix.gnu.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200609091522.GB8597@zpidnp36 \
    --to=ldb@leibniz-psychology.org \
    --cc=41702@debbugs.gnu.org \
    --cc=ludo@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).