bug#49264: 28.0.50; project.el+tramp performance issue

bug#49264: 28.0.50; project.el+tramp performance issue

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi:

Using tramp I tried to use project.el with a command like
project-switch-to-buffer and it took like 10 minutes to complete.

I ran a profiler and I found that most of the time was taken by an
external function: global-tags-try-project-root

project-current is called in a loop for all the opened buffers it calls
project--find-in-directory that calls project-find-functions and there
is going all the time.

After some optimization in an external package; now the time is half
than before but still very slow to use the command (around 3-5 minutes
to complete) and running again the profiler I get this:

         5637  89% - command-execute
         5549  88%  - byte-code
         5549  88%   - project--read-project-buffer
         5549  88%    - let*
         5336  85%     - read-buffer
         5323  84%      - ivy-completing-read
         5323  84%       - ivy-read
         4941  78%        - ivy--reset-state
         4941  78%         - ivy--buffer-list
         4941  78%          - internal-complete-buffer
         4941  78%           - #<lambda -0x1a357caf01243d61>
         4941  78%            - and
         4941  78%             - equal
         4941  78%              - save-current-buffer
         4941  78%               - project-current
         4941  78%                - project--find-in-directory
         4548  72%                 - project-try-vc
         4537  72%                  - vc-responsible-backend
         4478  71%                   - #<compiled 0xd3f2e32af0966f7>
         4478  71%                    - vc-call-backend
         4478  71%                     - apply
         1470  23%                      + vc-svn-responsible-p
         1142  18%                      + vc-bzr-responsible-p
          970  15%                      + vc-hg-responsible-p
          390   6%                      + vc-git-responsible-p
          156   2%                      + vc-cvs-responsible-p
          126   2%                      + vc-rcs-responsible-p
          108   1%                      + vc-sccs-responsible-p
           98   1%                      + vc-src-responsible-p
           57   0%                   + tramp-file-name-handler
           11   0%                  + vc-file-getprop
          393   6%                 + global-tags-try-project-root
          375   5%        + read-from-minibuffer
           13   0%      + if
          213   3%     + project-current
           88   1%  + funcall-interactively
          572   9% + ...
           51   0% + timer-event-handler
            8   0% + redisplay_internal (C function)


As you can see most of the time is still taken by project-current and I
can't really understand why:

1) Are so many samples 4548 seems a very high number for only 25 opened
buffers.

2) why project-try-vc still takes so much...? Specially for unfrequent
vc systems in our days like svn or bzr that I am not using.

As a workaround I removed all the uninteresting handlers from
vc-handled-backends and I get better times now, but IMHO it is still
very inefficient (almost a minute for project-switch-to-buffer is
excessive). And make it practically unusable.

In any case:

Maybe (I think I mentioned this before) `project.el` needs a sort of
cache to speedup some functions like `project-current` that are called
very frequently inside the project.el code.

Related with https://debbugs.gnu.org/cgi/bugreport.cgi?bug=42966

VCS changing is not something that happens very often to require a check
of all the backends everytime, several times for every buffer in many
project.el functions right? Specially when using tramp.

vc has vc-file-prop-obarray; maybe vc-responsible-backend should cache
it's result there to avoid repeating time consuming computations?

Either in the local system the performance penalty seems to be
significant to me.

Best,
Ergus



In GNU Emacs 28.0.50 (build 50, x86_64-pc-linux-gnu, GTK+ Version 3.24.29, cairo version 1.17.4)
 of 2021-06-26 built on Ergus
Repository revision: b8f9e58ef72402e69a1f0960816184d52e5d2d29
Repository branch: master
System Description: Arch Linux

Configured using:
 'configure --prefix=/home/ergo/.local/ --with-mailutils --with-json
 --with-x-toolkit=gtk3 --with-xft --with-wide-int --with-modules
 --with-cairo --with-harfbuzz --with-native-compilation'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER
PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM
GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: C++//law

Minor modes in effect:
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  diff-hl-margin-mode: t
  windmove-mode: t
  subword-mode: t
  hide-ifdef-mode: t
  preproc-font-lock-mode: t
  shell-dirtrack-mode: t
  show-paren-mode: t
  global-auto-revert-mode: t
  xclip-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  electric-pair-mode: t
  flyspell-mode: t
  company-mode: t
  flycheck-mode: t
  counsel-mode: t
  ivy-mode: t
  composable-mark-mode: t
  composable-mode: t
  repeat-mode: t
  xterm-mouse-mode: t
  winner-mode: t
  save-place-mode: t
  which-key-mode: t
  override-global-mode: t
  delete-selection-mode: t
  savehist-mode: t
  global-display-fill-column-indicator-mode: t
  display-fill-column-indicator-mode: t
  global-display-line-numbers-mode: t
  display-line-numbers-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Load-path shadows:
/usr/share/emacs/site-lisp/cmake-mode hides /home/ergo/.emacs.d/elpa/cmake-mode-20210104.1831/cmake-mode
/usr/share/emacs/site-lisp/notmuch-crypto hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-crypto
/usr/share/emacs/site-lisp/notmuch-compat hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-compat
/usr/share/emacs/site-lisp/notmuch-hello hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-hello
/usr/share/emacs/site-lisp/notmuch hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch
/usr/share/emacs/site-lisp/notmuch-show hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-show
/usr/share/emacs/site-lisp/notmuch-maildir-fcc hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-maildir-fcc
/usr/share/emacs/site-lisp/coolj hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/coolj
/usr/share/emacs/site-lisp/notmuch-draft hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-draft
/usr/share/emacs/site-lisp/notmuch-tree hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-tree
/usr/share/emacs/site-lisp/notmuch-parser hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-parser
/usr/share/emacs/site-lisp/notmuch-lib hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-lib
/usr/share/emacs/site-lisp/notmuch-mua hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-mua
/usr/share/emacs/site-lisp/notmuch-message hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-message
/usr/share/emacs/site-lisp/notmuch-address hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-address
/usr/share/emacs/site-lisp/notmuch-wash hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-wash
/usr/share/emacs/site-lisp/notmuch-tag hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-tag
/usr/share/emacs/site-lisp/notmuch-print hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-print
/usr/share/emacs/site-lisp/notmuch-query hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-query
/usr/share/emacs/site-lisp/notmuch-jump hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-jump
/usr/share/emacs/site-lisp/notmuch-company hides /home/ergo/.emacs.d/elpa/notmuch-20210627.1741/notmuch-company
/home/ergo/.emacs.d/elpa/transient-20210619.1100/transient hides /home/ergo/.local/share/emacs/28.0.50/lisp/transient

Features:
(shadow sort notmuch-company notmuch-lib notmuch-version notmuch-compat
mm-view mml-smime smime dig mail-extr emacsbug sendmail magit-extras
hi-lock magit-bookmark magit-submodule magit-obsolete magit-blame
magit-stash magit-reflog magit-bisect magit-push magit-pull magit-fetch
magit-clone magit-remote magit-commit magit-sequence magit-notes
magit-worktree magit-tag magit-merge magit-branch magit-reset
magit-files magit-refs magit-status magit package browse-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap url-handlers magit-repos magit-apply
magit-wip magit-log which-func imenu magit-diff smerge-mode diff
git-commit log-edit message rmc puny rfc822 mml mml-sec epa derived epg
epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr
mailabbrev mail-utils gmm-utils mailheader add-log magit-core
magit-autorevert magit-margin magit-transient magit-process with-editor
server magit-mode transient magit-git magit-section magit-utils crm
tramp-cmds global-tags ht generator async counsel-gtags pulse
mc-separate-operations mc-edit-lines mc-hide-unmatched-lines-mode
mc-mark-more mc-cycle-cursors multiple-cursors-core rect move-dup
diff-hl-margin eieio-opt speedbar ezimage dframe shortdoc help-fns
radix-tree vc-annotate amx s windmove misearch multi-isearch ffap
url-parse url-vars face-remap vc-hg macrostep-c cmacexp macrostep
cap-words superword subword hideif preproc-font-lock cc-mode cc-fonts
cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
dired-aux diff-hl-dired diff-hl log-view pcvs-util vc-dir ewoc vc
tramp-cache tramp-sh tramp tramp-loaddefs trampver tramp-integration
files-x tramp-compat shell pcomplete parse-time iso8601 time-date
ls-lisp format-spec auth-source password-cache thingatpt vc-git
diff-mode vc-dispatcher bookmark pp paren autorevert filenotify xclip
yasnippet-snippets yasnippet elec-pair flyspell-correct-ivy
flyspell-correct flyspell ispell company-keywords company-gtags
company-dabbrev-code company-dabbrev company-files company-semantic
company-template company-capf company flycheck json map find-func dash
counsel xdg xref project dired-x dired dired-loaddefs compile
text-property-search comint ansi-color swiper ivy-avy avy ivy flx
ivy-faces ivy-overlay colir pcase term/tmux term/xterm xterm jka-compr
init composable composable-mark powerline comp comp-cstr warnings subr-x
powerline-separators color powerline-themes repeat xt-mouse
simple-16-theme winner ring saveplace diminish edmacro kmacro which-key
advice configmail cl-extra help-mode use-package use-package-ensure
use-package-delight use-package-diminish use-package-bind-key bind-key
use-package-core disp-table delsel savehist easy-mmode
display-fill-column-indicator display-line-numbers info ede/auto
eieio-base cl-seq seq eieio byte-opt bytecomp byte-compile cconv
eieio-core cl-macs gv eieio-loaddefs cl-loaddefs cl-lib tex-site rx
slime-autoloads early-init iso-transl tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 811583 75249)
 (symbols 48 30651 0)
 (strings 32 107218 13868)
 (string-bytes 1 4408538)
 (vectors 16 57480)
 (vector-slots 8 1324399 45245)
 (floats 8 342 1198)
 (intervals 56 53473 1428)
 (buffers 992 37))

bug#49264: 28.0.50; project.el+tramp performance issue

[Eli Zaretskii]

> Date: Tue, 29 Jun 2021 00:11:00 +0200
> From:  Ergus via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Using tramp I tried to use project.el with a command like
> project-switch-to-buffer and it took like 10 minutes to complete.
> 
> I ran a profiler and I found that most of the time was taken by an
> external function: global-tags-try-project-root

That doesn't follow from the profile you show.  According to the
profile, global-tags-try-project-root takes just 6% of the CPU time.

> project-current is called in a loop for all the opened buffers it calls
> project--find-in-directory that calls project-find-functions and there
> is going all the time.

I don't see project-find-functions in the profile.  Where is it and
how does it come into this picture?

> After some optimization in an external package; now the time is half
> than before but still very slow to use the command (around 3-5 minutes
> to complete) and running again the profiler I get this:
> 
>          5637  89% - command-execute
>          5549  88%  - byte-code
>          5549  88%   - project--read-project-buffer
>          5549  88%    - let*
>          5336  85%     - read-buffer
>          5323  84%      - ivy-completing-read
>          5323  84%       - ivy-read
>          4941  78%        - ivy--reset-state
>          4941  78%         - ivy--buffer-list
>          4941  78%          - internal-complete-buffer
>          4941  78%           - #<lambda -0x1a357caf01243d61>
>          4941  78%            - and
>          4941  78%             - equal
>          4941  78%              - save-current-buffer
>          4941  78%               - project-current
>          4941  78%                - project--find-in-directory
>          4548  72%                 - project-try-vc
>          4537  72%                  - vc-responsible-backend
>          4478  71%                   - #<compiled 0xd3f2e32af0966f7>
>          4478  71%                    - vc-call-backend
>          4478  71%                     - apply
>          1470  23%                      + vc-svn-responsible-p
>          1142  18%                      + vc-bzr-responsible-p
>           970  15%                      + vc-hg-responsible-p
>           390   6%                      + vc-git-responsible-p
>           156   2%                      + vc-cvs-responsible-p
>           126   2%                      + vc-rcs-responsible-p
>           108   1%                      + vc-sccs-responsible-p
>            98   1%                      + vc-src-responsible-p
>            57   0%                   + tramp-file-name-handler
>            11   0%                  + vc-file-getprop
>           393   6%                 + global-tags-try-project-root
>           375   5%        + read-from-minibuffer
>            13   0%      + if
>           213   3%     + project-current
>            88   1%  + funcall-interactively
>           572   9% + ...
>            51   0% + timer-event-handler
>             8   0% + redisplay_internal (C function)
> 
> 
> As you can see most of the time is still taken by project-current and I
> can't really understand why:

AFAICT, most of the time is taken by 'apply', but the profile doesn't
show which function is called by 'apply'.  Can you tell which function
is that?

> 1) Are so many samples 4548 seems a very high number for only 25 opened
> buffers.

These two numbers are unrelated.  4548 is the number of time the
profiler found the program counter inside project-try-vc and the
functions it calls.  This number has no relation to the number of
buffers you have, it just means that code runs slowly.

> 2) why project-try-vc still takes so much...? Specially for unfrequent
> vc systems in our days like svn or bzr that I am not using.

That was explained on emacs-devel.  However, ...

> As a workaround I removed all the uninteresting handlers from
> vc-handled-backends and I get better times now, but IMHO it is still
> very inefficient (almost a minute for project-switch-to-buffer is
> excessive). And make it practically unusable.

... after removing the "unused" VC back-ends, you say that the code
still runs very slowly.  So is the issue with VC back-ends still
relevant, and if so, how?

More importantly, what is the profile after you remove the extra VC
calls?

> VCS changing is not something that happens very often to require a check
> of all the backends everytime, several times for every buffer in many
> project.el functions right? Specially when using tramp.

Once again, given what you say above, this doesn't sound important,
does it?  The slow processing is elsewhere, and without seeing a
profile with VC calls removed, it's hard to make progress in this
matter, or give you some advice regarding potential reason(s).

> vc has vc-file-prop-obarray; maybe vc-responsible-backend should cache
> it's result there to avoid repeating time consuming computations?

Again: is this issue relevant, given that without the VC calls the
code is still very slow?

bug#49264: 28.0.50; project.el+tramp performance issue

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Eli:

On Tue, Jun 29, 2021 at 03:05:35PM +0300, Eli Zaretskii wrote:
>> Date: Tue, 29 Jun 2021 00:11:00 +0200
>> From:  Ergus via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>>
>> Using tramp I tried to use project.el with a command like
>> project-switch-to-buffer and it took like 10 minutes to complete.
>>
>> I ran a profiler and I found that most of the time was taken by an
>> external function: global-tags-try-project-root
>
>That doesn't follow from the profile you show.  According to the
>profile, global-tags-try-project-root takes just 6% of the CPU time.
>
The profile is `After some optimization in an external package`. As
explained 2 paragraphs next. Before the fix it was taking more or less
the same percent than vc-responsible-backend.

>> project-current is called in a loop for all the opened buffers it calls
>> project--find-in-directory that calls project-find-functions and there
>> is going all the time.
>
>I don't see project-find-functions in the profile.  Where is it and
>how does it come into this picture?

project-find-functions is a hook called in project--find-in-directory

(defun project--find-in-directory (dir)
   (run-hook-with-args-until-success 'project-find-functions dir))

>
>> After some optimization in an external package; now the time is half
>> than before but still very slow to use the command (around 3-5 minutes
>> to complete) and running again the profiler I get this:
>>
>>          5637  89% - command-execute
>>          5549  88%  - byte-code
>>          5549  88%   - project--read-project-buffer
>>          5549  88%    - let*
>>          5336  85%     - read-buffer
>>          5323  84%      - ivy-completing-read
>>          5323  84%       - ivy-read
>>          4941  78%        - ivy--reset-state
>>          4941  78%         - ivy--buffer-list
>>          4941  78%          - internal-complete-buffer
>>          4941  78%           - #<lambda -0x1a357caf01243d61>
>>          4941  78%            - and
>>          4941  78%             - equal
>>          4941  78%              - save-current-buffer
>>          4941  78%               - project-current
>>          4941  78%                - project--find-in-directory
>>          4548  72%                 - project-try-vc
>>          4537  72%                  - vc-responsible-backend
>>          4478  71%                   - #<compiled 0xd3f2e32af0966f7>
>>          4478  71%                    - vc-call-backend
>>          4478  71%                     - apply
>>          1470  23%                      + vc-svn-responsible-p
>>          1142  18%                      + vc-bzr-responsible-p
>>           970  15%                      + vc-hg-responsible-p
>>           390   6%                      + vc-git-responsible-p
>>           156   2%                      + vc-cvs-responsible-p
>>           126   2%                      + vc-rcs-responsible-p
>>           108   1%                      + vc-sccs-responsible-p
>>            98   1%                      + vc-src-responsible-p
>>            57   0%                   + tramp-file-name-handler
>>            11   0%                  + vc-file-getprop
>>           393   6%                 + global-tags-try-project-root
>>           375   5%        + read-from-minibuffer
>>            13   0%      + if
>>           213   3%     + project-current
>>            88   1%  + funcall-interactively
>>           572   9% + ...
>>            51   0% + timer-event-handler
>>             8   0% + redisplay_internal (C function)
>>
>>
>> As you can see most of the time is still taken by project-current and I
>> can't really understand why:
>
>AFAICT, most of the time is taken by 'apply', but the profile doesn't
>show which function is called by 'apply'.  Can you tell which function
>is that?
>
apply is called in vc-responsible-backend and looking at the percentages
it is obvious that the times are going in

   23%                      + vc-svn-responsible-p
   18%                      + vc-bzr-responsible-p
   15%                      + vc-hg-responsible-p
    6%                      + vc-git-responsible-p
    2%                      + vc-cvs-responsible-p
    2%                      + vc-rcs-responsible-p
    1%                      + vc-sccs-responsible-p
    1%                      + vc-src-responsible-p
------
   68%

These are the backends' functions that are passed to vc-call-backend in
the mapcar in vc-responsible-backend and the output of
vc-find-backend-function.

>> 1) Are so many samples 4548 seems a very high number for only 25 opened
>> buffers.
>
>These two numbers are unrelated.  4548 is the number of time the
>profiler found the program counter inside project-try-vc and the
>functions it calls.  This number has no relation to the number of
>buffers you have, it just means that code runs slowly.
>
Ok

>> 2) why project-try-vc still takes so much...? Specially for unfrequent
>> vc systems in our days like svn or bzr that I am not using.
>
>That was explained on emacs-devel.  However, ...
>
>> As a workaround I removed all the uninteresting handlers from
>> vc-handled-backends and I get better times now, but IMHO it is still
>> very inefficient (almost a minute for project-switch-to-buffer is
>> excessive). And make it practically unusable.
>
>... after removing the "unused" VC back-ends, you say that the code
>still runs very slowly.  So is the issue with VC back-ends still
>relevant, and if so, how?
>
Yes, it is. It is still slow, lets say around 20-40 seconds to complete
the command. But that's that's much faster than before (3-5 minutes);
but still too slow to make the command usable.

>More importantly, what is the profile after you remove the extra VC
>calls?
>
          790  83% - command-execute
          631  66%  - byte-code
          631  66%   - project--read-project-buffer
          436  46%    + ivy-completing-read
          188  19%    - project-current
          188  19%     - project--find-in-directory
          188  19%      - project-try-vc
          135  14%       - vc-responsible-backend
          135  14%        - #<compiled -0x11223b7c225fdde9>
          135  14%         - vc-call-backend
          135  14%          - apply
          105  11%           - vc-hg-responsible-p
          105  11%            - vc-find-root
          102  10%             - locate-dominating-file
           76   8%              + tramp-file-name-handler
           26   2%              + abbreviate-file-name
           30   3%           - vc-git-responsible-p
           30   3%            - vc-find-root
           30   3%             - locate-dominating-file
           30   3%              + tramp-file-name-handler
           41   4%       - project--vc-merge-submodules-p
           41   4%        - project--value-in-dir
           41   4%         - hack-dir-local-variables-non-file-buffer
           41   4%          - hack-dir-local-variables
           37   3%           - dir-locals-find-file
           27   2%            - locate-dominating-file
           21   2%             + abbreviate-file-name
            6   0%             + dir-locals--all-files
           10   1%            + dir-locals--all-files
           12   1%       + vc-call-backend
            7   0%    + #<compiled 0x34145703cd88c72>


Only with git and mercurial backends and with 10 remote buffers open (in
the first profile there were like 25). Remember that the times grow
linearly with the number of buffers.

As a note here, when N files are in the same directory the normal thing
is that all of them share the VCS. So calling a check function for all
of them is redundant and slow.

>> VCS changing is not something that happens very often to require a check
>> of all the backends everytime, several times for every buffer in many
>> project.el functions right? Specially when using tramp.
>
>Once again, given what you say above, this doesn't sound important,
>does it?  The slow processing is elsewhere, and without seeing a
>profile with VC calls removed, it's hard to make progress in this
>matter, or give you some advice regarding potential reason(s).
>

Reducing 3-5 minutes to 20-40 seconds when there is only hg and git
sounds relevant for me. Still slow for a command, but important
considering that it is still asking the filesystem for every
file/buffer.

>> vc has vc-file-prop-obarray; maybe vc-responsible-backend should cache
>> it's result there to avoid repeating time consuming computations?
>
>Again: is this issue relevant, given that without the VC calls the
>code is still very slow?

bug#49264: 28.0.50; project.el+tramp performance issue

[Eli Zaretskii]

> Date: Wed, 30 Jun 2021 00:21:42 +0200
> From: Ergus <spacibba@aol.com>
> Cc: 49264@debbugs.gnu.org
> 
> >AFAICT, most of the time is taken by 'apply', but the profile doesn't
> >show which function is called by 'apply'.  Can you tell which function
> >is that?
> >
> apply is called in vc-responsible-backend and looking at the percentages
> it is obvious that the times are going in
> 
>    23%                      + vc-svn-responsible-p
>    18%                      + vc-bzr-responsible-p
>    15%                      + vc-hg-responsible-p
>     6%                      + vc-git-responsible-p
>     2%                      + vc-cvs-responsible-p
>     2%                      + vc-rcs-responsible-p
>     1%                      + vc-sccs-responsible-p
>     1%                      + vc-src-responsible-p
> ------
>    68%
> 
> These are the backends' functions that are passed to vc-call-backend in
> the mapcar in vc-responsible-backend and the output of
> vc-find-backend-function.

But if you still need to wait for dozens of seconds with just 2
backends, which take only 20% of the time according to the above, then
how do you want to speed this up in your case?  How about not using
ivy (which AFAICT is the root cause of this slowness)?

> >... after removing the "unused" VC back-ends, you say that the code
> >still runs very slowly.  So is the issue with VC back-ends still
> >relevant, and if so, how?
> >
> Yes, it is. It is still slow, lets say around 20-40 seconds to complete
> the command. But that's that's much faster than before (3-5 minutes);
> but still too slow to make the command usable.

It's still unacceptably slow, so that couldn't be a solution,
certainly not a general one, IMO.

> As a note here, when N files are in the same directory the normal thing
> is that all of them share the VCS. So calling a check function for all
> of them is redundant and slow.

AFAIR, that's not really true, and ISTR project.el aims to support the
use cases with several different VC backends.

And again, waiting for 30 seconds when you have just 10 buffers is
unacceptable.  E.g., in the session where I'm writing this, I have 80
buffers that visit files in a single branch of the Emacs Git
repository, almost an order of magnitude more than your 10 buffers.

So we must find some better way of getting reasonable performance in
this use case.  If the round-trip of the VC backend to a remote
filesystem is the bottleneck, let's try to speed that up in some way.

bug#49264: 28.0.50; project.el+tramp performance issue

[Phil Sainty]

On 2021-07-01 00:46, Eli Zaretskii wrote:
>> As a note here, when N files are in the same directory the normal 
>> thing
>> is that all of them share the VCS. So calling a check function for all
>> of them is redundant and slow.
> 
> AFAIR, that's not really true, and ISTR project.el aims to support the
> use cases with several different VC backends.

It's probably worth considering that while one *can* have multiple VC
backends active in a single directory, it's *extremely* common not to.

If there was a user option which effectively opted out of the multiple-
backend support in favour of performance-oriented assumptions and
caching, along with some mechanism for flushing the cache on demand
(advertised to the user as part of the user option documentation),
then users could then enable that option as a performance measure
provided that they were confident that the default functionality was
redundant for their use-cases (as I suspect it would be for many
people).


-Phil

bug#49264: 28.0.50; project.el+tramp performance issue

[Eli Zaretskii]

> Date: Thu, 01 Jul 2021 01:25:07 +1200
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: Ergus <spacibba@aol.com>, 49264@debbugs.gnu.org
> 
> > AFAIR, that's not really true, and ISTR project.el aims to support the
> > use cases with several different VC backends.
> 
> It's probably worth considering that while one *can* have multiple VC
> backends active in a single directory, it's *extremely* common not to.

I think Dmitry once told me this isn't so, but maybe I'm dreaming.

> If there was a user option which effectively opted out of the multiple-
> backend support in favour of performance-oriented assumptions

You mean, ask the user to specify the backend for a project?  That
could work, I think.

But again, Jimmy's use case was unbearably slow even with a single VC
backend.

bug#49264: 28.0.50; project.el+tramp performance issue

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Wed, Jun 30, 2021 at 03:46:26PM +0300, Eli Zaretskii wrote:
>> Date: Wed, 30 Jun 2021 00:21:42 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: 49264@debbugs.gnu.org
>>
>> >AFAICT, most of the time is taken by 'apply', but the profile doesn't
>> >show which function is called by 'apply'.  Can you tell which function
>> >is that?
>> >
>> apply is called in vc-responsible-backend and looking at the percentages
>> it is obvious that the times are going in
>>
>>    23%                      + vc-svn-responsible-p
>>    18%                      + vc-bzr-responsible-p
>>    15%                      + vc-hg-responsible-p
>>     6%                      + vc-git-responsible-p
>>     2%                      + vc-cvs-responsible-p
>>     2%                      + vc-rcs-responsible-p
>>     1%                      + vc-sccs-responsible-p
>>     1%                      + vc-src-responsible-p
>> ------
>>    68%
>>
>> These are the backends' functions that are passed to vc-call-backend in
>> the mapcar in vc-responsible-backend and the output of
>> vc-find-backend-function.
>
>But if you still need to wait for dozens of seconds with just 2
>backends, which take only 20% of the time according to the above, then
>how do you want to speed this up in your case?  How about not using
>ivy (which AFAICT is the root cause of this slowness)?
>
In the test Dimitry requested I sent the numbers of eval:

(benchmark 1 '(project-current))

Only with git and mercurial it takes:

Elapsed time: 3.018998s (0.109912s in 1 GCs)

With the entire list in vc-handled-backends

Elapsed time: 8.197923s (0.507396s in 6 GCs)

So, ivy is not the problem.

>> >... after removing the "unused" VC back-ends, you say that the code
>> >still runs very slowly.  So is the issue with VC back-ends still
>> >relevant, and if so, how?
>> >
>> Yes, it is. It is still slow, lets say around 20-40 seconds to complete
>> the command. But that's that's much faster than before (3-5 minutes);
>> but still too slow to make the command usable.
>
>It's still unacceptably slow, so that couldn't be a solution,
>certainly not a general one, IMO.
>
Agree, but it is another symptom about where the problem is.

>> As a note here, when N files are in the same directory the normal thing
>> is that all of them share the VCS. So calling a check function for all
>> of them is redundant and slow.
>
>AFAIR, that's not really true, and ISTR project.el aims to support the
>use cases with several different VC backends.
>
I can't speak for all the possible configurations; but IMO the most
common configurations around are projects with single vcs OR projects
with subprojects (git submodules) in different subdirs I am not aware of
any project where different vcs are in the same root directory... For
sure I have never seen that...

For example something like:

project0
|- file0-0
|- dir0
|   |- file0-1
|   |- file0-1
|
|- subproject1
|  |- file1-1
|  |- file1-2
|  |- dir1
|
|- subproject2
    |- file2-1
    |- file2-2
    |- dir2
    |  |- file2-3
    |
    |- subproject3
       |- file3-1
       |- dir2
          |- file3-2


Again: IMHO if we support this configuration we support 99% of the
projects around. And I already proposed an option (using the default-dir
as cache index) that already cover this with no functionality penalty.

>And again, waiting for 30 seconds when you have just 10 buffers is
>unacceptable.  E.g., in the session where I'm writing this, I have 80
>buffers that visit files in a single branch of the Emacs Git
>repository, almost an order of magnitude more than your 10 buffers.
>
With my solution we only impact performance:
1) The first time we call the command
AND
2) All the 80 files are all in different directories.

Otherwise there will be some benefit either for files located in the
same directory and/or the next time you call the command (or open a file
in one of the directories emacs already knows about.).

>So we must find some better way of getting reasonable performance in
>this use case.  If the round-trip of the VC backend to a remote
>filesystem is the bottleneck, let's try to speed that up in some way.

In case you are interested about the issue with the external package:

https://github.com/FelipeLema/emacs-counsel-gtags/issues/29

bug#49264: 28.0.50; project.el+tramp performance issue

[Dmitry Gutov]

Hi!

Thanks for the report, I'll follow up on this discussion some more 
later, but some initial observations:

On 29.06.2021 01:11, Ergus via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> As a workaround I removed all the uninteresting handlers from
> vc-handled-backends and I get better times now, but IMHO it is still
> very inefficient (almost a minute for project-switch-to-buffer is
> excessive). And make it practically unusable.

Could you evaluate (benchmark 1 '(project-current)) in one of your 
buffers? That should give us an estimate how long it takes to find the 
"current project" on your remote system.

If I'm right, project-switch-to-buffer should take 25 x (that time).

If you indeed only have 25 buffers (including hidden ones).

> In any case:
> 
> Maybe (I think I mentioned this before) `project.el` needs a sort of
> cache to speedup some functions like `project-current` that are called
> very frequently inside the project.el code.

The difficulty here is probably with the large latency to the remote 
system. And our current approach calls the "find current project" logic 
for each open buffer.

Even if we add the "current project" cache, it will only take effect at 
second and further invocations. Your first project-switch-to-buffer call 
will still take 3-5 minutes, which is unacceptable.

Please get back to us with the requested measurements, and perhaps other 
observations (any research initiative is welcome), but if finding the 
current vc root for each buffer is unavoidably slow, we'll finally need 
to switch to a "project-contains-buffer-p generic" approach, previously 
discussed in e.g. "Speed up project-kill-buffers" thread on emacs-devel.

bug#49264: 28.0.50; project.el+tramp performance issue

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Tue, Jun 29, 2021 at 04:00:02PM +0300, Dmitry Gutov wrote:
>Hi!
>
>Thanks for the report, I'll follow up on this discussion some more 
>later, but some initial observations:
>
>On 29.06.2021 01:11, Ergus via Bug reports for GNU Emacs, the Swiss 
>army knife of text editors wrote:
>>As a workaround I removed all the uninteresting handlers from
>>vc-handled-backends and I get better times now, but IMHO it is still
>>very inefficient (almost a minute for project-switch-to-buffer is
>>excessive). And make it practically unusable.
>
>Could you evaluate (benchmark 1 '(project-current)) in one of your 
>buffers? That should give us an estimate how long it takes to find the 
>"current project" on your remote system.
>
Only with git and mercurial it takes:

Elapsed time: 3.018998s (0.109912s in 1 GCs)

With the entire list in vc-handled-backends

Elapsed time: 8.197923s (0.507396s in 6 GCs)

>If I'm right, project-switch-to-buffer should take 25 x (that time).
>
>If you indeed only have 25 buffers (including hidden ones).
>
>>In any case:
>>
>>Maybe (I think I mentioned this before) `project.el` needs a sort of
>>cache to speedup some functions like `project-current` that are called
>>very frequently inside the project.el code.
>
>The difficulty here is probably with the large latency to the remote 
>system. And our current approach calls the "find current project" 
>logic for each open buffer.
>
>Even if we add the "current project" cache, it will only take effect 
>at second and further invocations. Your first project-switch-to-buffer 
>call will still take 3-5 minutes, which is unacceptable.
>
>Please get back to us with the requested measurements, and perhaps 
>other observations (any research initiative is welcome), but if 
>finding the current vc root for each buffer is unavoidably slow, we'll 
>finally need to switch to a "project-contains-buffer-p generic" 
>approach, previously discussed in e.g. "Speed up project-kill-buffers" 
>thread on emacs-devel.

Another (and maybe even simpler) optimization, may be to consider that
all the buffers with the same default-directory should have the same vcs
and vc root (and probably belong to same project). (I think that all vcs
backends only do the search based on default-directory at the end.)

So if the association in the cache is for `default-directory` instead of
individual file names; then, less files will need to evaluate the search
of vc root, and vc-backend only 1/subdir will search the first time. It
is a very good trade of IMO.

This will reduce the time even the first time we iterate project-current
over all opened buffers if multiple files are in the same directory
(example: sources in src, includes in include and so on). And I think it
will cover 99% of normal use cases.

This won't affect nested projects, git-submodules or similar, because
those will be in different subdirs.

bug#49264: 28.0.50; project.el+tramp performance issue

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi:

Any progress/suggestion on this? 

bug#49264: 28.0.50; project.el+tramp performance issue

[Dmitry Gutov]

On 26.07.2021 19:56, Ergus via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> Any progress/suggestion on this?

Only internally.

Let me get back to you on this in a couple of days.

bug#49264: 28.0.50; project.el+tramp performance issue

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

Hi again,

On 27.07.2021 02:04, Dmitry Gutov wrote:
> On 26.07.2021 19:56, Ergus via Bug reports for GNU Emacs, the Swiss army 
> knife of text editors wrote:
>> Any progress/suggestion on this?
> 
> Only internally.
> 
> Let me get back to you on this in a couple of days.

Sorry for the long wait, this feature ties into another old discussion, 
and I wasn't sure how to proceed best.

Anyway, here's a patch to try (attached). It should recover performance 
lost back in 4ca13d98c9e while retaining flexibility (and even adding more).

There are still less than optimal places there (e.g. file-in-directory-p 
is not very fast), and the modules list is read from the disk every 
time, so some further optimization should be possible.

But first please try this version.

[-- Attachment #2: project-buffers.diff --]
[-- Type: text/x-patch, Size: 3124 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 4620ea8f47..f164c19ace 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -51,6 +51,11 @@
 ;; files and its relations to external directories.  `project-files'
 ;; should be consistent with `project-ignores'.
 ;;
+;; `project-buffers' can be overridden if the project has some unusual
+;; shape (e.g. it contains files residing outside of its root, or some
+;; files inside the root must not be considered a part of it). It
+;; should be consistent with `project-files'.
+;;
 ;; This list can change in future versions.
 ;;
 ;; VC project:
@@ -334,6 +339,16 @@ project--remote-file-names
                 (concat remote-id file))
               local-files))))
 
+(cl-defgeneric project-buffers (project)
+  "Return the list of all live buffers that belong to PROJECT."
+  (let ((root (project-root project))
+        bufs)
+    (dolist (buf (buffer-list))
+      (when (file-in-directory-p (buffer-local-value 'default-directory buf)
+                                 root)
+        (push buf bufs)))
+    (nreverse bufs)))
+
 (defgroup project-vc nil
   "Project implementation based on the VC package."
   :version "25.1"
@@ -628,6 +643,23 @@ project--value-in-dir
       (hack-dir-local-variables-non-file-buffer))
     (symbol-value var)))
 
+(cl-defmethod project-buffers ((project (head vc)))
+  (let* ((root (project-root project))
+         (modules (unless (or (project--vc-merge-submodules-p root)
+                              (project--submodule-p root))
+                    (mapcar
+                     (lambda (m) (concat root m))
+                     (project--git-submodules))))
+         dd
+         bufs)
+    (dolist (buf (buffer-list))
+      (setq dd (buffer-local-value 'default-directory buf))
+      (when (and (file-in-directory-p dd root)
+                 (not (cl-find-if (lambda (module) (file-in-directory-p dd module))
+                                  modules)))
+        (push buf bufs)))
+    (nreverse bufs)))
+
 \f
 ;;; Project commands
 
@@ -1014,13 +1046,11 @@ project--read-project-buffer
          (current-buffer (current-buffer))
          (other-buffer (other-buffer current-buffer))
          (other-name (buffer-name other-buffer))
+         (buffers (project-buffers pr))
          (predicate
           (lambda (buffer)
             ;; BUFFER is an entry (BUF-NAME . BUF-OBJ) of Vbuffer_alist.
-            (and (cdr buffer)
-                 (equal pr
-                        (with-current-buffer (cdr buffer)
-                          (project-current)))))))
+            (memq (cdr buffer) buffers))))
     (read-buffer
      "Switch to buffer: "
      (when (funcall predicate (cons other-name other-buffer))
@@ -1160,7 +1190,7 @@ project--buffers-to-kill
 What buffers should or should not be killed is described
 in `project-kill-buffer-conditions'."
   (let (bufs)
-    (dolist (buf (project--buffer-list pr))
+    (dolist (buf (project-buffers pr))
       (when (project--kill-buffer-check buf project-kill-buffer-conditions)
         (push buf bufs)))
     bufs))

bug#49264: 28.0.50; project.el+tramp performance issue

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Mon, Aug 09, 2021 at 03:59:55AM +0300, Dmitry Gutov wrote:
>Hi again,
>
>On 27.07.2021 02:04, Dmitry Gutov wrote:
>
>Sorry for the long wait, this feature ties into another old 
>discussion, and I wasn't sure how to proceed best.
>
>Anyway, here's a patch to try (attached). It should recover 
>performance lost back in 4ca13d98c9e while retaining flexibility (and 
>even adding more).
>
>There are still less than optimal places there (e.g. 
>file-in-directory-p is not very fast), and the modules list is read 
>from the disk every time, so some further optimization should be 
>possible.
>
>But first please try this version.

Thanks Dmitry:

This patch reduced the C-x p b time to just a few (~5) seconds when I
have like 20 remote buffers.

At the moment I haven't have time to stress it a bit more, but it
improves the situation significantly compared to the previous situation.

In my opinion this is a good improvement and may be installed on master,
but probably it is not enough.

I made a manual fast profiling and I see that most of the time in
project-buffers actually goes to tramp-sh-file-name-handler.

          207  42%         - project-buffers
          207  42%          - apply
          207  42%           - #<compiled -0x1f5d919efefc0a09>
          200  40%            - file-in-directory-p
          200  40%             - tramp-file-name-handler
          197  40%              - apply
          197  40%               - tramp-sh-file-name-handler
          197  40%                - tramp-handle-file-in-directory-p
          183  37%                 - tramp-run-real-handler
          183  37%                  - file-in-directory-p
          124  25%                   - file-equal-p
          124  25%                    - tramp-file-name-handler
          121  24%                     - apply
          121  24%                      - tramp-sh-file-name-handler
          121  24%                       - tramp-handle-file-equal-p
           85  17%                        - tramp-run-real-handler
           85  17%                         - file-equal-p
           52  10%                          - file-truename
           52  10%                           - tramp-file-name-handler
           41   8%                            - apply
           41   8%                             - tramp-sh-file-name-handler
           41   8%                              - tramp-sh-handle-file-truename
           28   5%                               + file-remote-p
           10   2%                               + file-local-name
            3   0%                               + file-name-as-directory

It goes specifically to file-in-directory-p as you said. So maybe the
improvement there may be also desirable if the difference after the
optimization can reduce the time for file-in-directory-p (or the caller)
at least to the half.

So, very thanks again.

Ergus.

bug#49264: 28.0.50; project.el+tramp performance issue

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 2839 bytes --]

Hi Ergus,

On 17.08.2021 03:45, Ergus wrote:

> I made a manual fast profiling and I see that most of the time in
> project-buffers actually goes to tramp-sh-file-name-handler.
> 
>           207  42%         - project-buffers
>           207  42%          - apply
>           207  42%           - #<compiled -0x1f5d919efefc0a09>
>           200  40%            - file-in-directory-p
>           200  40%             - tramp-file-name-handler
>           197  40%              - apply
>           197  40%               - tramp-sh-file-name-handler
>           197  40%                - tramp-handle-file-in-directory-p
>           183  37%                 - tramp-run-real-handler
>           183  37%                  - file-in-directory-p
>           124  25%                   - file-equal-p
>           124  25%                    - tramp-file-name-handler
>           121  24%                     - apply
>           121  24%                      - tramp-sh-file-name-handler
>           121  24%                       - tramp-handle-file-equal-p
>            85  17%                        - tramp-run-real-handler
>            85  17%                         - file-equal-p
>            52  10%                          - file-truename
>            52  10%                           - tramp-file-name-handler
>            41   8%                            - apply
>            41   8%                             - tramp-sh-file-name-handler
>            41   8%                              - 
> tramp-sh-handle-file-truename
>            28   5%                               + file-remote-p
>            10   2%                               + file-local-name
>             3   0%                               + file-name-as-directory
> 
> It goes specifically to file-in-directory-p as you said. So maybe the
> improvement there may be also desirable if the difference after the
> optimization can reduce the time for file-in-directory-p (or the caller)
> at least to the half.

Thanks for testing.

Try the attached new version please. It should eliminate that particular 
bottleneck.

[-- Attachment #2: project-buffers.diff --]
[-- Type: text/x-patch, Size: 3248 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 4620ea8f47..f9b302bb2b 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -51,6 +51,11 @@
 ;; files and its relations to external directories.  `project-files'
 ;; should be consistent with `project-ignores'.
 ;;
+;; `project-buffers' can be overridden if the project has some unusual
+;; shape (e.g. it contains files residing outside of its root, or some
+;; files inside the root must not be considered a part of it). It
+;; should be consistent with `project-files'.
+;;
 ;; This list can change in future versions.
 ;;
 ;; VC project:
@@ -334,6 +339,16 @@ project--remote-file-names
                 (concat remote-id file))
               local-files))))
 
+(cl-defgeneric project-buffers (project)
+  "Return the list of all live buffers that belong to PROJECT."
+  (let ((root (expand-file-name (file-name-as-directory (project-root project))))
+        bufs)
+    (dolist (buf (buffer-list))
+      (when (string-prefix-p root (expand-file-name
+                                   (buffer-local-value 'default-directory buf)))
+        (push buf bufs)))
+    (nreverse bufs)))
+
 (defgroup project-vc nil
   "Project implementation based on the VC package."
   :version "25.1"
@@ -628,6 +643,23 @@ project--value-in-dir
       (hack-dir-local-variables-non-file-buffer))
     (symbol-value var)))
 
+(cl-defmethod project-buffers ((project (head vc)))
+  (let* ((root (expand-file-name (file-name-as-directory (project-root project))))
+         (modules (unless (or (project--vc-merge-submodules-p root)
+                              (project--submodule-p root))
+                    (mapcar
+                     (lambda (m) (format "%s%s/" root m))
+                     (project--git-submodules))))
+         dd
+         bufs)
+    (dolist (buf (buffer-list))
+      (setq dd (expand-file-name (buffer-local-value 'default-directory buf)))
+      (when (and (string-prefix-p root dd)
+                 (not (cl-find-if (lambda (module) (string-prefix-p module dd))
+                                  modules)))
+        (push buf bufs)))
+    (nreverse bufs)))
+
 \f
 ;;; Project commands
 
@@ -1014,13 +1046,11 @@ project--read-project-buffer
          (current-buffer (current-buffer))
          (other-buffer (other-buffer current-buffer))
          (other-name (buffer-name other-buffer))
+         (buffers (project-buffers pr))
          (predicate
           (lambda (buffer)
             ;; BUFFER is an entry (BUF-NAME . BUF-OBJ) of Vbuffer_alist.
-            (and (cdr buffer)
-                 (equal pr
-                        (with-current-buffer (cdr buffer)
-                          (project-current)))))))
+            (memq (cdr buffer) buffers))))
     (read-buffer
      "Switch to buffer: "
      (when (funcall predicate (cons other-name other-buffer))
@@ -1160,7 +1190,7 @@ project--buffers-to-kill
 What buffers should or should not be killed is described
 in `project-kill-buffer-conditions'."
   (let (bufs)
-    (dolist (buf (project--buffer-list pr))
+    (dolist (buf (project-buffers pr))
       (when (project--kill-buffer-check buf project-kill-buffer-conditions)
         (push buf bufs)))
     bufs))

bug#49264: 28.0.50; project.el+tramp performance issue

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Thu, Aug 19, 2021 at 04:19:20AM +0300, Dmitry Gutov wrote:
>Hi Ergus,
>
>On 17.08.2021 03:45, Ergus wrote:
>
>>I made a manual fast profiling and I see that most of the time in
>>project-buffers actually goes to tramp-sh-file-name-handler.
>>
>> �������� 207� 42%�������� - project-buffers
>> �������� 207� 42%��������� - apply
>> �������� 207� 42%���������� - #<compiled -0x1f5d919efefc0a09>
>> �������� 200� 40%����������� - file-in-directory-p
>> �������� 200� 40%������������ - tramp-file-name-handler
>> �������� 197� 40%������������� - apply
>> �������� 197� 40%�������������� - tramp-sh-file-name-handler
>> �������� 197� 40%��������������� - tramp-handle-file-in-directory-p
>> �������� 183� 37%���������������� - tramp-run-real-handler
>> �������� 183� 37%����������������� - file-in-directory-p
>> �������� 124� 25%������������������ - file-equal-p
>> �������� 124� 25%������������������� - tramp-file-name-handler
>> �������� 121� 24%�������������������� - apply
>> �������� 121� 24%��������������������� - tramp-sh-file-name-handler
>> �������� 121� 24%���������������������� - tramp-handle-file-equal-p
>> ��������� 85� 17%����������������������� - tramp-run-real-handler
>> ��������� 85� 17%������������������������ - file-equal-p
>> ��������� 52� 10%������������������������� - file-truename
>> ��������� 52� 10%�������������������������� - tramp-file-name-handler
>> ��������� 41�� 8%��������������������������� - apply
>> ��������� 41�� 8%���������������������������� - tramp-sh-file-name-handler
>> ��������� 41�� 8%����������������������������� - 
>>tramp-sh-handle-file-truename
>> ��������� 28�� 5%������������������������������ + file-remote-p
>> ��������� 10�� 2%������������������������������ + file-local-name
>> ���������� 3�� 0%������������������������������ + file-name-as-directory
>>
>>It goes specifically to file-in-directory-p as you said. So maybe the
>>improvement there may be also desirable if the difference after the
>>optimization can reduce the time for file-in-directory-p (or the caller)
>>at least to the half.
>
>Thanks for testing.
>
>Try the attached new version please. It should eliminate that 
>particular bottleneck.

Hi:

It feels better now. There is still a small delay, but that is normal
working with tramp. I will try it a bit more tomorrow; but the first
impression is that this solves the issue for me.

Best,
Ergus

bug#49264: 28.0.50; project.el+tramp performance issue

[Dmitry Gutov]

On 19.08.2021 06:08, Ergus via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> It feels better now. There is still a small delay, but that is normal
> working with tramp. I will try it a bit more tomorrow; but the first
> impression is that this solves the issue for me.

It does spend time in reading files up the directory (in 
project--vc-merge-submodules-p), though I hope those are cached by Tramp.

And then, if project-vc-merge-submodules is set, also reads the contents 
of .gitmodules. But it's probably not the case.

I'll push the patch now since it seems like a solid improvement either 
way. Let me know if there's anything to be done here, or we can close 
the bug.

bug#49264: 28.0.50; project.el+tramp performance issue

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Sat, Aug 21, 2021 at 05:23:09AM +0300, Dmitry Gutov wrote:
>On 19.08.2021 06:08, Ergus via Bug reports for GNU Emacs, the Swiss 
>army knife of text editors wrote:
>>It feels better now. There is still a small delay, but that is normal
>>working with tramp. I will try it a bit more tomorrow; but the first
>>impression is that this solves the issue for me.
>
>It does spend time in reading files up the directory (in 
>project--vc-merge-submodules-p), though I hope those are cached by 
>Tramp.
>
>And then, if project-vc-merge-submodules is set, also reads the 
>contents of .gitmodules. But it's probably not the case.
>
>I'll push the patch now since it seems like a solid improvement either 
>way. Let me know if there's anything to be done here, or we can close 
>the bug.

IMO we can close the issue. If there are still some problems then I will
reopen it or create a new one.

Very thanks.
Ergus

bug#49264: 28.0.50; project.el+tramp performance issue

[Dmitry Gutov]

On 21.08.2021 08:43, Ergus wrote:
> IMO we can close the issue. If there are still some problems then I will
> reopen it or create a new one.

Thanks, closing.