unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* treesitter local parser: huge slowdown and memory usage in a long file
@ 2024-02-11 21:53 Vincenzo Pupillo
  2024-02-12  4:16 ` Yuan Fu
  0 siblings, 1 reply; 31+ messages in thread
From: Vincenzo Pupillo @ 2024-02-11 21:53 UTC (permalink / raw)
  To: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

Hi, 
as a benchmark for my php-ts-mode (in 2 variants: one with tree-sitter-phpdoc 
for php comment block, and another using regular expression for comment block) 
I use tcpdf.php (from the tcpdf library). This php file has 24730 lines and 
generates 669 parser ranges, 665 of which are for phpdoc. 

As you can see from the profile (I try to edit the comment on line 2350) that I 
attached, the problem is in treesit--pre-redisplay.
I tried playing around with the code a bit but to no avail (for example, I 
limited treesit-update-ranges to window-start and window-end). 
comments say:
    ;; Force repase on _all_ the parsers might not be necessary, but
    ;; this is probably the most robust way.
  
Any ideas?
My php-ts-mode (It's a working progress) is available here: 
https://github.com/vpxyz/php-ts-mode

Thanks
V.

p.s without phpdoc emacs is as fast as with short php files.
p.p.s. nvim with treesitter is as slow as my major mode with this file.

GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.41, cairo 
version 1.18.0) of 2024-02-11

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: cpu_profiler_report --]
[-- Type: text/plain; charset="x-UTF_8J"; name="cpu_profiler_report", Size: 12025 bytes --]

[profiler-profile "28.1" cpu #s(hash-table test equal data ([redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil] 124 [nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil] 945 [line-move-visual line-move next-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil] 6 [line-move-visual line-move previous-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil] 4 [treesit--pre-redisplay run-hook-with-args redisplay--pre-redisplay-functions redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil nil nil] 2744 [jit-lock--antiblink-post-command nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil] 1 [syntax-class show-paren--categorize-paren show-paren--locate-near-paren show-paren--default show-paren-function apply timer-event-handler nil nil nil nil nil nil nil nil nil] 1 [delete-dups xselect-convert-to-targets pgtk-own-selection-internal "#<compiled -0x1728ef4c95247357>" apply gui-backend-set-selection gui-set-selection gui-select-text kill-new kill-region kill-line funcall-interactively command-execute nil nil nil] 3 [treesit-query-range treesit--update-ranges-local treesit-update-ranges treesit--pre-redisplay run-hook-with-args redisplay--pre-redisplay-functions redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil] 1759 [treesit-query-range treesit-update-ranges treesit--pre-redisplay run-hook-with-args redisplay--pre-redisplay-functions redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil] 1636 ["#<compiled 0x1c852541b14fae74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil nil] 1 [cl-delete cl-remove cl-remove-if-not treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c852541b14fae74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil] 3 [treesit-node-parent let* php-ts-mode--language-at-point treesit-language-at treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil] 1 [treesit-buffer-root-node treesit-node-at treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil nil nil] 102 [treesit-query-range treesit-update-ranges treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c852725f4eeae74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil] 2 [treesit--update-ranges-local treesit-update-ranges treesit--pre-redisplay run-hook-with-args redisplay--pre-redisplay-functions redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil] 14 [cl-remove cl-remove-if-not treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c852724ac226e74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil] 3 [treesit-parser-root-node treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c85273d2ea0ee74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil nil] 52 [treesit--font-lock-fontify-region-1 treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c85273eefe1ae74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil nil] 13 [self-insert-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil nil nil] 3 [parse-partial-sexp syntax-ppss jit-lock--antiblink-post-command nil nil nil nil nil nil nil nil nil nil nil nil nil] 2 [treesit-update-ranges treesit--pre-redisplay run-hook-with-args redisplay--pre-redisplay-functions redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil nil] 23 [treesit--update-ranges-local treesit-update-ranges treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c85273724beee74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil] 2 [comment-indent-new-line default-indent-new-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil nil] 1 [jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil nil nil] 3 [treesit-query-range treesit--update-ranges-local treesit-update-ranges treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c8527451184aa74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil] 1 [treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c852743d740ea74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil] 2 [make-closure syntax-ppss show-paren--default show-paren-function apply timer-event-handler nil nil nil nil nil nil nil nil nil nil] 1 ["#<compiled 0x16385ee6d140fc75>" newline comment-indent-new-line default-indent-new-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil] 1 ["#<compiled 0x2736abab5e0d1>" font-lock-unfontify-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c853ead11f7aa74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil] 1 ["#<compiled 0x1daadc6ebc11a146>" cl-remove cl-remove-if-not treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c853ead11f7aa74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil] 1 [internal-echo-keystrokes-prefix nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil] 1 [facep treesit--font-lock-fontify-region-1 treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c8538e0d90bb674>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil] 1 [line-move next-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil nil] 1 [frame-parameter if eval redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil nil nil] 1 [mode-line-frame-control eval redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil nil nil nil] 1 [line-move previous-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil nil] 2 [redisplay_internal\ \(C\ function\) sit-for icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil] 24 [sit-for icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil] 30 ["#<compiled 0x17c4b22424e34a24>" all-completions complete-with-action "#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_56>" completion-pcm--all-completions completion-substring--all-completions completion-flex-all-completions "#<compiled -0x1bd7cce58ccf55c9>" "#<compiled -0x4953631e8c8c907>" mapc seq-do seq-some completion--nth-completion completion-all-completions completion-all-sorted-completions icomplete--sorted-completions] 7 [interactive-form commandp "#<compiled -0x8f410b7c4bcca45>" "#<compiled 0x17c4b22424e34a24>" all-completions complete-with-action "#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_56>" completion-pcm--all-completions completion-substring--all-completions completion-flex-all-completions "#<compiled -0x1bd7cce58ccf55c9>" "#<compiled -0x4953631e8c8c907>" mapc seq-do seq-some completion--nth-completion] 1 [all-completions complete-with-action "#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_56>" completion-pcm--all-completions completion-substring--all-completions completion-flex-all-completions "#<compiled -0x1bd7cce58ccf55c9>" "#<compiled -0x4953631e8c8c907>" mapc seq-do seq-some completion--nth-completion completion-all-completions completion-all-sorted-completions icomplete--sorted-completions icomplete-completions] 68 [completion-pcm--all-completions completion-substring--all-completions completion-flex-all-completions "#<compiled -0x1bd7cce58ccf55c9>" "#<compiled -0x4953631e8c8c907>" mapc seq-do seq-some completion--nth-completion completion-all-completions completion-all-sorted-completions icomplete--sorted-completions icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default] 2 [delete-dups completion-all-sorted-completions icomplete--sorted-completions icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil] 1 ["#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_45>" minibuffer--sort-by-length-alpha completion-all-sorted-completions icomplete--sorted-completions icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil] 2 ["#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_61>" read-extended-command--affixation icomplete--augment icomplete--render-vertical icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil] 1 [window--min-size-1 window--min-size-1 window-min-size window-sizable window--resize-root-window-vertically redisplay_internal\ \(C\ function\) completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil] 1 [redisplay_internal\ \(C\ function\) completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil nil nil] 22 [completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil nil nil nil] 67 [completion--flex-score "#<compiled -0x1dd7ba1b6c096a85>" mapcar "#<compiled -0x820789829508d37>" completion-all-sorted-completions icomplete--sorted-completions icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil] 1 [try-completion complete-with-action "#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_56>" completion--complete-and-exit minibuffer-force-complete-and-exit icomplete-force-complete-and-exit icomplete-fido-ret funcall-interactively command-execute completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil] 10 [execute-extended-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil nil nil] 1 [Automatic\ GC nil] 165)) (26057 13278 384506 497000) nil]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: mem_profiler_report --]
[-- Type: text/plain; charset="x-UTF_8J"; name="mem_profiler_report", Size: 21749 bytes --]

[profiler-profile "28.1" memory #s(hash-table test equal data ([profiler-start funcall-interactively command-execute execute-extended-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil] 631 [timer--time-setter timer-set-time run-at-time execute-extended-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil] 24 [timer--time-less-p timer--activate timer-activate run-at-time execute-extended-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil] 24 [timer--time-setter timer-set-idle-time run-with-idle-timer jit-lock--antiblink-post-command nil nil nil nil nil nil nil nil nil nil nil nil] 24 [nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil] 32744 [menu-bar-update-buffers-1 menu-bar-update-buffers redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil nil nil nil] 2016 [redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil] 316718 [string-match kill-this-buffer-enabled-p redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil nil nil nil] 1024 [timer--time-setter timer-set-time run-at-time run-with-timer blink-cursor--start-timer blink-cursor-start apply timer-event-handler nil nil nil nil nil nil nil nil] 216 [timer--time-less-p timer--activate timer-activate run-at-time run-with-timer blink-cursor--start-timer blink-cursor-start apply timer-event-handler nil nil nil nil nil nil nil] 192 [timer-relative-time timer-inc-time timer-event-handler nil nil nil nil nil nil nil nil nil nil nil nil nil] 672 [timer--time-setter timer-inc-time timer-event-handler nil nil nil nil nil nil nil nil nil nil nil nil nil] 288 [time-less-p timer-event-handler nil nil nil nil nil nil nil nil nil nil nil nil nil nil] 288 [timer--time-less-p timer--activate timer-activate timer-event-handler nil nil nil nil nil nil nil nil nil nil nil nil] 264 [line-move-visual line-move next-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil] 4748 [default-font-height default-line-height line-move-partial line-move next-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil] 8184 [delete-and-extract-region "#<compiled 0x1bdcb8d5ab3d6985>" apply "#<compiled -0x1e3e289fe5154769>" buffer-substring--filter filter-buffer-substring kill-region kill-line funcall-interactively command-execute nil nil nil nil nil nil] 24840 [menu-bar-update-yank-menu kill-new kill-region kill-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil] 1152 [pgtk-own-selection-internal "#<compiled -0x1728ef4c95247357>" apply gui-backend-set-selection gui-set-selection gui-select-text kill-new kill-region kill-line funcall-interactively command-execute nil nil nil nil nil] 368 [treesit--update-ranges-local treesit-update-ranges treesit--pre-redisplay run-hook-with-args redisplay--pre-redisplay-functions redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil] 2464048 [treesit-query-range treesit--update-ranges-local treesit-update-ranges treesit--pre-redisplay run-hook-with-args redisplay--pre-redisplay-functions redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil] 6513544 [treesit-query-range treesit-update-ranges treesit--pre-redisplay run-hook-with-args redisplay--pre-redisplay-functions redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil] 23008 [treesit--pre-redisplay run-hook-with-args redisplay--pre-redisplay-functions redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil nil nil] 571762456 [jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil nil nil nil nil] 8288 [treesit--update-ranges-local treesit-update-ranges treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c852541b14fae74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil] 5200 [treesit-local-parsers-on treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c852541b14fae74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil nil] 2320 [treesit-parser-root-node treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c852541b14fae74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil nil] 1679304 [treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c852541b14fae74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil] 41440 [treesit--font-lock-fontify-region-1 treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c852541b14fae74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil nil nil] 51984 [treesit--update-ranges-local treesit-update-ranges treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil] 160 [treesit-local-parsers-at treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil] 80 [treesit-local-parsers-at treesit-node-at let* php-ts-mode--language-at-point treesit-language-at treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil] 160 [treesit-node-at let* php-ts-mode--language-at-point treesit-language-at treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil] 3072 [treesit-node-parent let* php-ts-mode--language-at-point treesit-language-at treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil] 3584 [format let* php-ts-mode--language-at-point treesit-language-at treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil] 338 [looking-at looking-at-p and cond save-excursion let* php-ts-mode--language-at-point treesit-language-at treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil] 1024 [treesit-local-parsers-at treesit-node-at treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil nil nil] 80 [treesit-buffer-root-node treesit-node-at treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil nil nil] 23857603 [treesit-node-at treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil] 3584 [treesit-parent-while treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil] 256 [treesit-local-parsers-at treesit-node-at let* php-ts-mode--language-at-point treesit-language-at treesit-node-on treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil] 80 [treesit-node-at let* php-ts-mode--language-at-point treesit-language-at treesit-node-on treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil] 1536 [treesit-node-parent let* php-ts-mode--language-at-point treesit-language-at treesit-node-on treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil] 1792 [format let* php-ts-mode--language-at-point treesit-language-at treesit-node-on treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil] 169 [treesit-local-parsers-on treesit-node-on treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil nil nil] 80 [make-closure "#<compiled 0xfaa45bf0cbe519b>" treesit--simple-indent-eval treesit--simple-indent-eval treesit-simple-indent treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil] 4144 [string-match "#<compiled -0x12329b36eb2983c1>" "#<compiled -0x185b5bdede7ccb0a>" treesit--simple-indent-eval treesit-simple-indent treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil] 1024 [treesit-simple-indent treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil] 1336 [looking-at "#<compiled 0x113256ad37d3f5de>" treesit--simple-indent-eval treesit-simple-indent treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil] 2112 [string-match "#<compiled 0x113256ad37d3f5de>" treesit--simple-indent-eval treesit-simple-indent treesit--indent-1 treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil] 1024 [indent-line-to treesit-indent indent--funcall-widened indent-for-tab-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil] 25168 [self-insert-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil nil nil] 732592 [make-closure syntax-ppss jit-lock--antiblink-post-command nil nil nil nil nil nil nil nil nil nil nil nil nil] 4144 [generate-new-buffer comment-normalize-vars comment-indent-new-line default-indent-new-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil] 21 [comment-normalize-vars comment-indent-new-line default-indent-new-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil] 2176 [timer--time-setter timer-set-time run-at-time undo-auto--boundary-ensure-timer undo-auto--undoable-change newline comment-indent-new-line default-indent-new-line funcall-interactively command-execute nil nil nil nil nil nil] 24 [newline comment-indent-new-line default-indent-new-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil] 50656 [looking-at "#<compiled 0x16385ee5b7c4fc75>" newline comment-indent-new-line default-indent-new-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil] 1152 [comment-string-strip comment-indent-new-line default-indent-new-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil] 2304 [comment-indent-new-line default-indent-new-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil nil] 101648 [comment-enter-backward comment-indent-new-line default-indent-new-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil] 1024 [comment-normalize-vars comment-indent comment-indent-new-line default-indent-new-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil] 1024 [comment-indent comment-indent-new-line default-indent-new-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil] 101824 [make-closure syntax-ppss show-paren--default show-paren-function apply timer-event-handler nil nil nil nil nil nil nil nil nil nil] 8288 [string-prefix-p treesit-query-range treesit--update-ranges-local treesit-update-ranges treesit--pre-redisplay run-hook-with-args redisplay--pre-redisplay-functions redisplay_internal\ \(C\ function\) nil nil nil nil nil nil nil nil] 7187 [delete-backward-char funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil nil nil] 25144 [treesit-query-range treesit--update-ranges-local treesit-update-ranges treesit-font-lock-fontify-region font-lock-fontify-syntactically-region font-lock-default-fontify-region font-lock-fontify-region "#<compiled 0x1c85277a0b8dae74>" jit-lock--run-functions jit-lock-fontify-now jit-lock-function redisplay_internal\ \(C\ function\) nil nil nil nil] 4144 [line-move-visual line-move previous-line funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil] 4748 [completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil nil nil nil] 37647 [timer--time-setter timer-set-time run-at-time undo-auto--boundary-ensure-timer undo-auto--undoable-change completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil] 24 [minibuffer--regexp-setup completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil nil nil] 1024 [string-match pgtk-device-class device-class minibuffer-setup-on-screen-keyboard completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil] 1024 [sit-for icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil] 816 [menu-bar-update-buffers-1 menu-bar-update-buffers redisplay_internal\ \(C\ function\) sit-for icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil] 1008 [redisplay_internal\ \(C\ function\) sit-for icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil] 168822 [mode-line-default-help-echo redisplay_internal\ \(C\ function\) sit-for icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil] 8184 [completion-pcm--pattern->regex completion-pcm--all-completions completion-substring--all-completions completion-flex-all-completions "#<compiled -0x1bd7cce58ccf55c9>" "#<compiled -0x4953631e8c8c907>" mapc seq-do seq-some completion--nth-completion completion-all-completions completion-all-sorted-completions icomplete--sorted-completions icomplete-completions icomplete-exhibit icomplete-post-command-hook] 1152 [all-completions complete-with-action "#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_56>" completion-pcm--all-completions completion-substring--all-completions completion-flex-all-completions "#<compiled -0x1bd7cce58ccf55c9>" "#<compiled -0x4953631e8c8c907>" mapc seq-do seq-some completion--nth-completion completion-all-completions completion-all-sorted-completions icomplete--sorted-completions icomplete-completions] 4352 [version-to-list "#<compiled 0x17c4b22424e34a24>" all-completions complete-with-action "#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_56>" completion-pcm--all-completions completion-substring--all-completions completion-flex-all-completions "#<compiled -0x1bd7cce58ccf55c9>" "#<compiled -0x4953631e8c8c907>" mapc seq-do seq-some completion--nth-completion completion-all-completions completion-all-sorted-completions] 3328 [delete-dups completion-all-sorted-completions icomplete--sorted-completions icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil] 113424 [minibuffer--sort-by-length-alpha completion-all-sorted-completions icomplete--sorted-completions icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil] 53296 [minibuffer--sort-by-position completion-all-sorted-completions icomplete--sorted-completions icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil] 64 [minibuffer--sort-by-key minibuffer--sort-by-position completion-all-sorted-completions icomplete--sorted-completions icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil] 85344 ["#<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_61>" read-extended-command--affixation icomplete--augment icomplete--render-vertical icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil] 162492 [move-overlay icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil] 24 [format icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil] 6378 [redisplay_internal\ \(C\ function\) completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil nil nil] 399513 [funcall-interactively command-execute completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil nil] 80 [self-insert-command funcall-interactively command-execute completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil] 288 [completion--flex-score "#<compiled -0x1dd7ba1b6c096a85>" mapcar "#<compiled -0x820789829508d37>" completion-all-sorted-completions icomplete--sorted-completions icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil] 2304 [icomplete--render-vertical icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil] 117856 [completion--hilit-from-re "#<compiled -0x2358f1b2a8a21e7>" completion-lazy-hilit icomplete--render-vertical icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil] 8128 [frame-height max-mini-window-lines icomplete--render-vertical icomplete-completions icomplete-exhibit icomplete-post-command-hook completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil] 8184 [timer--time-setter timer-set-time run-at-time run-with-timer blink-cursor--start-timer blink-cursor-start apply timer-event-handler completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil] 24 [timer--time-less-p timer--activate timer-activate run-at-time run-with-timer blink-cursor--start-timer blink-cursor-start apply timer-event-handler completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil] 24 [timer-relative-time timer-inc-time timer-event-handler completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil] 56 [timer--time-setter timer-inc-time timer-event-handler completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil] 24 [time-less-p timer-event-handler completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil nil nil] 24 [timer--time-less-p timer--activate timer-activate timer-event-handler completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil nil nil nil] 24 [completion--replace minibuffer-force-complete minibuffer-force-complete-and-exit icomplete-force-complete-and-exit icomplete-fido-ret funcall-interactively command-execute completing-read-default read-extended-command-1 read-extended-command byte-code command-execute nil nil nil nil] 72 [execute-extended-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil nil nil] 59696 [command-execute execute-extended-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil nil] 49783 [funcall-interactively command-execute execute-extended-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil nil] 16 [profiler-stop funcall-interactively command-execute execute-extended-command funcall-interactively command-execute nil nil nil nil nil nil nil nil nil nil] 1563900 [Automatic\ GC nil] 17704)) (26057 13278 386045 775000) nil]

^ permalink raw reply	[flat|nested] 31+ messages in thread
* Re: treesitter local parser: huge slowdown and memory usage in a long file
@ 2024-04-20  2:18 Yuan Fu
  2024-04-20 19:14 ` Vincenzo Pupillo
  2024-05-06  2:04 ` Dmitry Gutov
  0 siblings, 2 replies; 31+ messages in thread
From: Yuan Fu @ 2024-04-20  2:18 UTC (permalink / raw)
  To: 付禹安, Dmitry Gutov
  Cc: Ergus via Emacs development discussions.

> 
> 
> > On Feb 18, 2024, at 9:53 PM, Yuan Fu <casouri@gmail.com> wrote:
> > 
> > 
> > 
> >> On Feb 17, 2024, at 7:37 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
> >> 
> >> On 13/02/2024 10:08, Yuan Fu wrote:
> >> 
> >>>> On 12/02/2024 06:16, Yuan Fu wrote:
> >>>>> Thanks, the culprit is the call to treesit-update-ranges in
> >>>>> treesit--pre-redisplay, where we don’t pass it any specific range, so it
> >>>>> updates the range for the whole buffer. Eli, is there any way to get a
> >>>>> rough estimate the range that redisplay is refreshing? Do you think
> >>>>> something like this would work?
> >>>> 
> >>>> If we don't update the ranges outside of some interval surrounding the 
> >>>> window, what does that mean for correctness?
> >>> If the place of update and the embedded code currently in view belong to 
> >>> the same node in the host language, then when we update ranges for the 
> >>> current window-visible range, the whole node’s range is updated. So at 
> >>> least for this node, the range is correct.
> >>> If the place of update and the embedded code currently in view belong to 
> >>> different nodes in the host language, then when we update ranges for the 
> >>> current window-visible range, only the visible node’s range is updated.
> >> 
> >> Okay. What about positions after the visible part of the buffer? Can their 
> >> ranges be outdated? It's probably okay when the ranges are only used for 
> >> font-lock and syntax-ppss, but I wonder about possible other applications 
> >> (reindenting the whole buffer, for example).
> > 
> > It’s the same as positions before the visible part. For reindenting the whole 
> > buffer, treesit-indent-region will update the range for the whole buffer at 
> > the very beginning.
> > 
> >> 
> >>>> 
> >>>> Perhaps the mode has a syntax-propertize-function which behaves 
> >>>> differently (as it should) depending on the language at point. Or 
> >>>> different ranges have different syntax tables, something like that.
> >>>> 
> >>>> If the ranges, after some edit (perhaps a programmatic one, performed far 
> >>>> from the visible area), are kept not update somewhere around the beginning 
> >>>> of the buffer, do we not risk confusing the syntax-ppss parser, for 
> >>>> example?
> >>> That can happen, yes.
> >>>> 
> >>>> Come to think of it, take treesit-indent: it only updates the ranges for 
> >>>> the current line. But the line's indentation usually depends on the 
> >>>> previous buffer positions, doesn't it?
> >>> The range passed to treesit-update-ranges act as an intercepting range—we 
> >>> capture nodes that intercepts with the range and use them to update ranges. 
> >>> If the line to be indented is in an embedded language block, the whole 
> >>> block will be captured and it’s range will be given to the embedded 
> >>> language parser.
> >>> We haven’t have any problem so far mainly because most embedded code blocks 
> >>> are local, and it’s rare for some edit to take place far from the visible 
> >>> portion which affects ranges and user expects that edit to affect the 
> >>> current visible range.
> >>> I don’t have any great idea for a better way to update ranges right now. 
> >>> Let me think about that. In the meantime, I’ll push a temporary fix so V’s 
> >>> original problem can be solved.
> >> 
> >> I was thinking (since considering the same problem in mmm-mode, actually) 
> >> that it would make sense to either plug into syntax-propertize-function, or 
> >> have a parallel data structure similarly tracking the outdated buffer 
> >> regions, which would only update the part of the buffer which had been 
> >> modified since last time.
> >> 
> >> Dealing with the "remainder" of the buffer might be trickier, but maybe some 
> >> heuristic which would help detect the "no changes" case could be implemented.
> > 
> > Yeah, something similar to syntax-ppss or jit-lock. Or maybe it can be 
> > avoided, since the current on-demand range update has been working fine, 
> > until we added treesit--pre-redisplay for syntax-ppss.
> 
> This is actually a bit involved, because there could be multiple layer’s of 
> parsers: the host language sets range for a local parser, and the local parser 
> can set ranges for a nested-nested parser. Eg, we might have a markdown parser 
> for parsing doc-comments, and inside the markdown there could be code blocks 
> which require another level of nested parser.
> 
> This use-case is a bit advanced but we definitely need to support it in our 
> design. And my brain is twisted by all the dependency and range. If you guys 
> has some ideas they’ll be most welcome :-)
> 

I believe I’ve found a good way to solve this problem. I pushed the changes to master. 

Basically I added a function treesit-parser-changed-ranges that can directly return the change ranges from last reparse. This means we don’t need to use notifiers to get those change ranges anymore. Then in treesit-pre-redisplay, we reparse the primary parser and get the changed ranges from it.

Once we have the changed ranges, we update other non-primary parser’s ranges, but only within the changed ranges. Originally we were updating those parser’s ranges on the whole buffer, which led to the slowdown. Then we had to use some workaround to solve this. Now the workaround isn’t needed anymore.

I also remove some notifier functions and moved their work into treesit-pre-redisplay.

Yuan


^ permalink raw reply	[flat|nested] 31+ messages in thread
end of thread, other threads:[~2024-06-11 16:36 UTC | newest]

Thread overview: 31+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-02-11 21:53 treesitter local parser: huge slowdown and memory usage in a long file Vincenzo Pupillo
2024-02-12  4:16 ` Yuan Fu
2024-02-12 14:09   ` Eli Zaretskii
2024-02-13  8:15     ` Yuan Fu
2024-02-13  9:39       ` Vincenzo Pupillo
2024-02-13 12:59       ` Eli Zaretskii
2024-02-13  0:50   ` Dmitry Gutov
2024-02-13  8:08     ` Yuan Fu
2024-02-18  3:37       ` Dmitry Gutov
2024-02-19  5:53         ` Yuan Fu
2024-03-21  6:39           ` Yuan Fu
  -- strict thread matches above, loose matches on Subject: below --
2024-04-20  2:18 Yuan Fu
2024-04-20 19:14 ` Vincenzo Pupillo
2024-04-23  5:09   ` Yuan Fu
2024-05-06  2:04 ` Dmitry Gutov
2024-05-09  0:16   ` Yuan Fu
2024-05-12 23:44     ` Dmitry Gutov
2024-05-22  5:51       ` Yuan Fu
2024-05-22 23:42         ` Dmitry Gutov
2024-05-27 22:03           ` Yuan Fu
2024-05-27 22:24             ` Dmitry Gutov
2024-06-04  4:53               ` Yuan Fu
2024-06-04  8:19                 ` Vincenzo Pupillo
2024-06-04 15:01                   ` Eli Zaretskii
2024-06-04 20:44                     ` Vincenzo Pupillo
2024-06-06 23:59                 ` Dmitry Gutov
2024-06-07  0:13                   ` Dmitry Gutov
2024-06-11 16:36                     ` Yuan Fu
2024-05-26  4:23     ` Stefan Monnier via Emacs development discussions.
2024-05-27 22:05       ` Yuan Fu
2024-05-27 22:34         ` Stefan Monnier

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).