From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Suhail Singh Newsgroups: gmane.emacs.bugs Subject: bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP Date: Fri, 06 Sep 2024 20:19:34 -0400 Message-ID: <87h6asl3hl.fsf@gmail.com> References: <87o75241qh.fsf@gmail.com> <86plpi2ixa.fsf@gnu.org> <87seudke20.fsf@gmail.com> <8634md2vc2.fsf@gnu.org> <87h6asrk42.fsf@gmx.de> <86a5gk28ht.fsf@gnu.org> <87le0450x8.fsf@gmx.de> <865xr8228d.fsf@gnu.org> <87jzfofzcl.fsf@gmail.com> <8634mc1ty0.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="11187"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: michael.albinus@gmx.de, Suhail Singh , 73046@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sat Sep 07 02:21:43 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1smjCw-0002m6-Lg for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 07 Sep 2024 02:21:42 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1smjCY-0002YW-DH; Fri, 06 Sep 2024 20:21:20 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1smjCK-0002Y5-CG for bug-gnu-emacs@gnu.org; Fri, 06 Sep 2024 20:21:05 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1smjCI-0004Nn-4g for bug-gnu-emacs@gnu.org; Fri, 06 Sep 2024 20:21:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:Date:References:In-Reply-To:From:To:Subject; bh=Yb95tMa3N/DtKzjYtzukYMsK39eI6vNVOM+li69v7bs=; b=foz3H7peSefkHmjm+Hh6y0oJzVcE/EGNKwgp1lXhdp1/FjYW3gNOVThg+jI0qoRH/4uYstoRJuZmGF0e+HibHFHX6APfCHOJabOvLDcQWR00kEsuoHAH4rBeg8tpNt/sJx81V0Vsg6tAgpiI/6FMb9jC7GdU5bgnVvZnjMUSo5BvriuQFuyGgQZinE6d63u18t0Y7QrTkrpk+zc2EpypHqyX8E6TpwcFazJXVr7AdgRzAqReiDy8EqZ/DtZM0qvuwkRST1eEDa8/idt/WtjqMSTm2Nvm8c5bNpemhp7JcgMKzGNL0LeMlmTjKDqJpmGwcwpXYAZHXto42yOprn/dzw==; Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1smjCI-0002Hh-5N for bug-gnu-emacs@gnu.org; Fri, 06 Sep 2024 20:21:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Suhail Singh Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 07 Sep 2024 00:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 73046 X-GNU-PR-Package: emacs Original-Received: via spool by 73046-submit@debbugs.gnu.org id=B73046.17256684548760 (code B ref 73046); Sat, 07 Sep 2024 00:21:02 +0000 Original-Received: (at 73046) by debbugs.gnu.org; 7 Sep 2024 00:20:54 +0000 Original-Received: from localhost ([127.0.0.1]:54411 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1smjCA-0002HE-9T for submit@debbugs.gnu.org; Fri, 06 Sep 2024 20:20:54 -0400 Original-Received: from mail-qt1-f193.google.com ([209.85.160.193]:56443) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1smjC8-0002H0-0o for 73046@debbugs.gnu.org; Fri, 06 Sep 2024 20:20:53 -0400 Original-Received: by mail-qt1-f193.google.com with SMTP id d75a77b69052e-457e2537611so15544041cf.0 for <73046@debbugs.gnu.org>; Fri, 06 Sep 2024 17:20:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725668385; x=1726273185; darn=debbugs.gnu.org; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=Yb95tMa3N/DtKzjYtzukYMsK39eI6vNVOM+li69v7bs=; b=d9ES5Mdrj/YTXjze1LSxDgqenn6PGAoADRtU7ZvadbhIfIjvevZM8LGEy4JJPzRH4w U846cyODiOfyCLqg5Bnq7Kl2TvqxiHuSEKedH/ATv4qwIzQhmO5/LF38q4PSBZAqb4tY 9SkqaF+3nPq7uKyhfIqCBMiioMebogsBnmqxWZLx8twERfm2PEqztSW2TnKuZUAgKXnK S4hEwIl0kPs3XaZfxAd8PzntPJi7+fiW20k8BRxd1RcVKgf36Yo+1aDXHoL3HPaQ+NQo 4Q93gJWNiJY34zcZlzuXMF9TxBrUpAPN62We+fQNNFelxZpoV+qq3jKbSlIsC9iEtknb 0NjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725668385; x=1726273185; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Yb95tMa3N/DtKzjYtzukYMsK39eI6vNVOM+li69v7bs=; b=Vdan8NQ6sC9Ft3JmXQk873w8/kCXRNhpOIyddL2jLjLvEiuotlRfzDMQ1yA/k2n4qw 5oOwOyGali+P6iptEO9JeeGiUOh1dF9/7DGH8Tj4rHHVjM5C3NG4ktsg7TJs6xpGsuI0 g0WF1LoTBtq1JnjJvgSqTB1IJcGGfcspIVHTUrE9uJ9i6XRblnJTDOsaCkVG+zUtyCmF T5idd7UyIrkWUe/eS68LObKIDOX8ypxRPUnrHmgdJ2fMuukT9fOHU5R1ndDB2yV3EuUz UCF/7QoOooB/22FM2wMetpf0E5x+1BsY/ZoyBV5K/3gOUi5ypNxhFcma8X6oc/y8iFwF oqzw== X-Forwarded-Encrypted: i=1; AJvYcCVdq9yqx526yDMpymhRECfN11FI75vu2v1Fy/tJZx4zQ4UAaOU8K+x07/zIGoY/fA3EU2pdtw==@debbugs.gnu.org X-Gm-Message-State: AOJu0YypQwkcgCvCwp0Q+oztv0j/epSa76pXUeLgdaKnjgD3KOeto+s5 twdrVJHgQuHbT0PcptrYjUZ57f8oIY6WvzuEf82DI3K6K050QZvrXpOdPuuc X-Google-Smtp-Source: AGHT+IFTP7+UkC0CbQWaGSvf85wLpRBOsLely8yse6/jT1UF+d29o+ZbdMrz97dAc3/c4mVweHNf9w== X-Received: by 2002:a05:622a:1649:b0:44f:f11a:63bf with SMTP id d75a77b69052e-4580c744ec6mr36607051cf.41.1725668385453; Fri, 06 Sep 2024 17:19:45 -0700 (PDT) Original-Received: from gnus ([65.94.70.53]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45822f61ac7sm149311cf.71.2024.09.06.17.19.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 17:19:45 -0700 (PDT) In-Reply-To: <8634mc1ty0.fsf@gnu.org> (Eli Zaretskii's message of "Fri, 06 Sep 2024 22:08:23 +0300") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:291337 Archived-At: Eli Zaretskii writes: > So if it takes 4 to 5 sec to move a 20KB file, then how much stuff > needs to be moved for the Dired listing? What does the below show, if > you run it on that remote machine? > > $ ls -al | wc Given that the contents of the remote directory can be exactly replicated (apart from metadata such as mtime etc) by the below code (that I shared in another response), you should be able to answer the question trivially: #+name: perf-fix/tramp/font-lock-on-dired/reproducer #+begin_src sh cd /tmp rm -rf /tmp/test mkdir /tmp/test cd /tmp/test for i in `seq -w 0 21`; do mkdir -p src/"$i" done mkdir -p links cd links for i in `seq -w 0 21`; do ln -sf /tmp/test/src/"$i" done #+end_src For the record, the output is as follows: #+begin_src sh :dir /ssh:${affected-host}:/tmp cd /tmp/test/links ls -al | wc #+end_src #+RESULTS: : 25 262 1599 > It needs to show around 40KB to explain 10 sec of delay. I don't understand your reasoning. In fact if the output of ls -al was indeed around 40kb I would have been very surprised. The time taken for transferring the 20KB file included initial connection costs whereas TRAMP would presumably be reusing the same connection, but would be sending multiple small requests. I don't see how one can be compared to the other, other than to say (generally) that when connection is slow both workflows would take greater time (which is what we observe). Note that disabling font-lock improved response delay considerably. That means the delay is not due to transferring information contained in `ls --dired', which is considerably fast (relatively speaking), but in doing the additional checks that Michael mentioned: >>> It seems to be related to font-locking, indeed. See variable >>> `dired-font-lock-keywords'. It specifies face recognition running basic >>> file oprtations. For example, ";; Broken Symbolic link" calls >>> `file-truename' and `file-exists-p', while "Symbolic link to a directory" >>> and ";; Symbolic link to a non-directory" invoke `file-truename' and >>> `file-directory-p'. I did some further investigation; summarizing findings below: A. On Host-A, the network connection is fairly slow s.t. transferring a 20KB file takes ~5s. On Host-B, the network connection is fairly fast. B. On Host-A, the time taken to refresh dired buffer containing 22 Subdirectories (/tmp/test/src as in above code snippet) is 0.70-0.75s with font-lock enabled, and about the same with font-lock disabled. These times exclude the time taken to establish the intiial connection over TRAMP. C. On Host-A, the time taken to refresh dired buffer containing 22 symlinks (each symlink pointing to a directory, i.e., /tmp/test/links in the above code snippet) is 0.70-0.75s with font-lock disabled. With font-lock enabled the time taken is ~13-14s and the CPU is at 100%. These times exclude the time taken to establish the intiial connection over TRAMP. D. On Host-B, the time taken to display dired buffer for /tmp/test/links with font-lock enabled is ~2s greater than when font-lock is disabled. When /tmp/test/links contains 100 symlinks to directories (instead of 22), the time taken when font-lock is enabled is ~6s greater than when font-lock is disabled. Given above, I conclude: 1. The issue is present when there are symlinks to directories. 2. The issue is worse when there are greater number of symlinks to directories. 2. The issue is worse when the connection is slower. However, it is still observable when the connection is faster - if you're having difficulty reproducing, increase the number of symlinks to directories in /tmp/test/links above. 3. Given that when connection is slower, not only is the time taken for font-locking greater, but the CPU is at 100%, I suspect that the relevant code is doing some kind of busy-waiting. The above observations seem consistent with Michael's comments above regd. font-lock checks for "Broken Symbolink link" and "Symbolic link to a directory". As such, if Michael's proposal below is implemented I believe it would be an adequate fix to the issue: >>> I believe it would be helpful to suppress these checks via a user >>> option. And no, the checks shouldn't be suppressed for remote >>> directories in general, on a fast connection they are valuable. I hope that clarifies things, and gives you sufficient information to be able to reproduce -- Suhail