bug#59151: Major performance issues in dired over ssh (Tramp) when remote directory contains symlinks

bug#59151: Major performance issues in dired over ssh (Tramp) when remote directory contains symlinks

[Mikhail Pomaznoy]

Greetings emacs devs, or to whom it may concern!

I am having troubles with dired listings of remote directories 
containing many files and symlinks. When I scroll through such a dired 
buffer Emacs lags (over a second) on every page-down scroll.

The problem is easily reproducible with -Q if the directory contains a 
hundred files and a hundred symbolic links to some other hundred files 
somewhere on the filesystem. If I kill the links in the dired buffer 
(without deleting any actual files or links) then everything scrolls 
smoothly as usual.

Could you please check if thats a real issue in your setting? ssh 
connection I am using is not so slow...

Kind regards,

-Mikhail


In GNU Emacs 28.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.34, 
cairo version 1.17.6)

  of 2022-09-12 built on frederik
Windowing system distributor 'The X.Org Foundation', version 11.0.12101004
System Description: Arch Linux

Configured using:
  'configure --with-x-toolkit=gtk3 --with-native-compilation
  --sysconfdir=/etc --prefix=/usr --libexecdir=/usr/lib
  --localstatedir=/var --with-cairo --with-harfbuzz --with-libsystemd
  --with-modules 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt
  -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
  -fstack-clash-protection -fcf-protection -g
  -ffile-prefix-map=/build/emacs/src=/usr/src/debug -flto=auto'
  'LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 M17N_FLT MODULES NATIVE_COMP NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
   value of $LANG: en_US.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Dired by name

Minor modes in effect:
   shell-dirtrack-mode: t
   tooltip-mode: t
   global-eldoc-mode: t
   show-paren-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   buffer-read-only: t
   line-number-mode: t
   indent-tabs-mode: t
   transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny rfc822 mml mml-sec epa
derived epg rfc6068 epg-config gnus-util rmail rmail-loaddefs
text-property-search mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils dired-aux dired dired-loaddefs tramp-sh
tramp-cache tramp tramp-loaddefs trampver tramp-integration files-x
tramp-compat shell pcomplete comint ansi-color ring parse-time iso8601
time-date ls-lisp format-spec auth-source cl-seq eieio eieio-core
cl-macs eieio-loaddefs cl-loaddefs cl-lib password-cache json subr-x map
seq byte-opt gv bytecomp byte-compile cconv iso-transl tooltip eldoc
paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
button loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 103629 9492)
  (symbols 48 8400 0)
  (strings 32 27457 1311)
  (string-bytes 1 928685)
  (vectors 16 17653)
  (vector-slots 8 390979 10629)
  (floats 8 27 45)
  (intervals 56 2626 0)
  (buffers 992 14))

bug#59151: Major performance issues in dired over ssh (Tramp) when remote directory contains symlinks

[Michael Albinus]

Mikhail Pomaznoy <mikpom@mikpom.ru> writes:

> Greetings emacs devs, or to whom it may concern!

Hi Mikhail,

> I am having troubles with dired listings of remote directories
> containing many files and symlinks. When I scroll through such a dired
> buffer Emacs lags (over a second) on every page-down scroll.
>
> The problem is easily reproducible with -Q if the directory contains a
> hundred files and a hundred symbolic links to some other hundred files
> somewhere on the filesystem. If I kill the links in the dired buffer
> (without deleting any actual files or links) then everything scrolls
> smoothly as usual.
>
> Could you please check if thats a real issue in your setting? ssh
> connection I am using is not so slow...

Dired does a lot of work for applying proper faces for the file
status. For symlinked files, for example, it calls file-truename,
file-exists-p, file-directory-p, for every single file in the
listing. Some of them even several times.

This takes time for remote files, especially if you don't use Tramp
caches, or caches with a short timeout.

I recommend you to disable the buffer-local font-lock-mode in a remote
dired buffer with so many symlinked files, via `M-x font-lock-mode'.

> Kind regards,
>
> -Mikhail

Best regards, Michael.

bug#59151: Major performance issues in dired over ssh (Tramp) when remote directory contains symlinks

[Mikhail Pomaznoy]

> I recommend you to disable the buffer-local font-lock-mode in a remote
> dired buffer with so many symlinked files, via `M-x font-lock-mode'.
Indeed, disabling font-lock works like a charm and restores the performance

> This takes time for remote files, especially if you don't use Tramp
> caches, or caches with a short timeout.
What an average Emacs Tramp user can do about those caches? searching 
for keywords `tramp cache` in customize menu of emacs gives nothing. If 
I search for `cache` in Tramp manual there is just a single section 
which doesn't look relevant for this case. Is the caches you talking 
about something "sophisticated" or simply documentation is lacking?

Of course certain lag is fine and expected when the directory is listed 
for the first time. But on every scroll ?

Thanks for your reply,

-Mikhail

On 11/10/22 16:16, Michael Albinus wrote:
> Mikhail Pomaznoy <mikpom@mikpom.ru> writes:
>
>> Greetings emacs devs, or to whom it may concern!
> Hi Mikhail,
>
>> I am having troubles with dired listings of remote directories
>> containing many files and symlinks. When I scroll through such a dired
>> buffer Emacs lags (over a second) on every page-down scroll.
>>
>> The problem is easily reproducible with -Q if the directory contains a
>> hundred files and a hundred symbolic links to some other hundred files
>> somewhere on the filesystem. If I kill the links in the dired buffer
>> (without deleting any actual files or links) then everything scrolls
>> smoothly as usual.
>>
>> Could you please check if thats a real issue in your setting? ssh
>> connection I am using is not so slow...
> Dired does a lot of work for applying proper faces for the file
> status. For symlinked files, for example, it calls file-truename,
> file-exists-p, file-directory-p, for every single file in the
> listing. Some of them even several times.
>
> This takes time for remote files, especially if you don't use Tramp
> caches, or caches with a short timeout.
>
> I recommend you to disable the buffer-local font-lock-mode in a remote
> dired buffer with so many symlinked files, via `M-x font-lock-mode'.
>
>> Kind regards,
>>
>> -Mikhail
> Best regards, Michael.

bug#59151: Major performance issues in dired over ssh (Tramp) when remote directory contains symlinks

[Michael Albinus]

Mikhail Pomaznoy <mikpom@mikpom.ru> writes:

Hi Mikhail,

>> I recommend you to disable the buffer-local font-lock-mode in a remote
>> dired buffer with so many symlinked files, via `M-x font-lock-mode'.
> Indeed, disabling font-lock works like a charm and restores the performance

Thanks for the feedback, I'm closing the bug.

>> This takes time for remote files, especially if you don't use Tramp
>> caches, or caches with a short timeout.
> What an average Emacs Tramp user can do about those caches? searching
> for keywords `tramp cache` in customize menu of emacs gives
> nothing. If I search for `cache` in Tramp manual there is just a
> single section which doesn't look relevant for this case. Is the
> caches you talking about something "sophisticated" or simply
> documentation is lacking?

See remote-file-name-inhibit-cache. It's default value is 10, meaning
caching for 10 seconds. You can increase this value, or set it to nil.

> Of course certain lag is fine and expected when the directory is
> listed for the first time. But on every scroll ?
>
> Thanks for your reply,
>
> -Mikhail

Best regards, Michael.