bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Wilhelm Kirschbaum]

With the following text in a buffer with elixir-ts-mode:

def foo() do
end

When I delete the `)` and re-insert it, the tree-sitter syntax 
tree does
not fully recover and the `do end` part is absent when observing 
using
treesit-explore-mode.

Before removing the `)`:

(source
 (call target: (identifier)
  (arguments
   (call target: (identifier)
    (arguments ( ))))
  (do_block do end)))

after re-inserting `)`:

(source
 (call target: (identifier)
  (arguments
   (call target: (identifier)
    (arguments ( ))))))

I can see the same behaviour using ruby-ts-mode when I remove the
following expression:

(setq-local syntax-propertize-function 
#'ruby-ts--syntax-propertize)

The ruby syntax I tested with:

def foo()
end

When I add a local `syntax-propertize-function' function to 
elixir-ts-mode
it also solves the problem of not fully recovering the tree-sitter
syntax tree.  It does not seem to matter what the function 
achieves
thought, so am a bit confused.


In GNU Emacs 30.0.50 (build 10, x86_64-pc-linux-gnu, GTK+ Version
 3.24.37, cairo version 1.17.8) of 2023-03-20 built on 
 melissa.local
Repository revision: 42fba8f36b19536964d6deb6a34f3fd1c02b43dd
Repository branch: master
System Description: Arch Linux

Configured using:
 'configure --with-native-compilation=aot --with-tree-sitter 
 --with-json
 --with-pgtk'
Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ 
JPEG
JSON LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY 
PDUMPER
PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF 
TOOLKIT_SCROLL_BARS
TREE_SITTER WEBP XIM GTK3 ZLIB
Important settings:
  value of $LANG: en_ZA.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Elixir

Minor modes in effect:
  electric-pair-mode: t
  erc-notifications-mode: t
  erc-list-mode: t
  erc-menu-mode: t
  erc-autojoin-mode: t
  erc-ring-mode: t
  erc-pcomplete-mode: t
  erc-track-mode: t
  erc-track-minor-mode: t
  erc-netsplit-mode: t
  erc-log-mode: t
  erc-services-mode: t
  erc-spelling-mode: t
  erc-match-mode: t
  erc-hl-nicks-mode: t
  erc-button-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  erc-networks-mode: t
  display-line-numbers-mode: t
  flyspell-mode: t
  whitespace-mode: t
  global-floatpays-mode: t
  floatpays-mode: t
  marginalia-mode: t
  savehist-mode: t
  vertico-mode: t
  global-corfu-mode: t
  corfu-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  global-flycheck-mode: t
  flycheck-mode: t
  flymake-mode: t
  global-diff-hl-mode: t
  diff-hl-mode: t
  global-git-commit-mode: t
  shell-dirtrack-mode: t
  server-mode: t
  which-key-mode: t
  override-global-mode: t
  global-treesit-auto-mode: t
  treesit-explore-mode: t
  pixel-scroll-precision-mode: t
  repeat-mode: t
  winner-mode: t
  save-place-mode: t
  which-function-mode: t
  global-auto-revert-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tab-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/wilhelm/.config/emacs/elpa/transient-20230315.1520/transient 
hides /usr/local/share/emacs/30.0.50/lisp/transient

Features:
(shadow view emacsbug shortdoc cus-start elec-pair descr-text 
sgml-mode
facemenu ruby-ts-mode ruby-mode em-unix em-term term ehelp 
em-script
em-prompt em-pred em-ls em-hist em-glob em-extpipe em-cmpl em-dirs
em-basic em-banner em-alias esh-mode esh-var eshell esh-cmd 
esh-ext
esh-opt esh-proc esh-io esh-arg esh-module esh-groups esh-util
tramp-cmds tramp-archive tramp-gvfs tramp tramp-loaddefs trampver
tramp-integration tramp-compat ls-lisp cl-print consult-imenu 
c-ts-mode
pulse jka-compr help-fns radix-tree consult-xref char-fold 
heex-ts-mode
elixir-ts-mode-tests ert-x dired-aux vc-hg vc-bzr vc-src vc-sccs 
vc-cvs
vc-rcs gnus-dired diff-hl-dired erts-mode misearch multi-isearch
goto-addr files-x git-rebase magit-extras face-remap cape 
network-stream
erc-desktop-notifications erc-imenu epa-file erc-image image-dired
image-dired-tags image-dired-external image-dired-util image-mode 
exif
erc-list erc-menu erc-join erc-ring erc-pcomplete erc-track 
erc-netsplit
erc-log erc-services erc-spelling erc-match erc-hl-nicks 
erc-button
erc-fill erc-stamp erc-goodies erc erc-backend erc-networks 
erc-common
erc-compat erc-loaddefs cursor-sensor display-line-numbers 
flyspell
ispell disp-table whitespace init.el mu4e-alert time ht alert 
log4e
notifications gntp mu4e mu4e-org mu4e-main mu4e-view gnus-art 
mm-uu
mml2015 mm-view mml-smime smime dig gnus-sum gnus-group gnus-undo
gnus-start gnus-dbus dbus gnus-cloud nnimap nnmail mail-source 
utf7 nnoo
gnus-spec gnus-int gnus-range gnus-win gnus nnheader range 
mu4e-headers
mu4e-compose mu4e-draft mu4e-actions smtpmail mu4e-search 
mu4e-lists
mu4e-bookmarks mu4e-mark mu4e-message flow-fill mule-util hl-line
mu4e-contacts mu4e-update mu4e-folders mu4e-server mu4e-context
mu4e-obsolete mu4e-vars mu4e-helpers mu4e-config ido srht plz verb 
js
c-ts-common denote mastodon mastodon-toot mastodon-iso persist
mastodon-http request flycheck-package package-build
package-build-badges package-recipe package-lint finder lisp-mnt 
eat
elfeed-show elfeed-search shr pixel-fill kinsoku elfeed-csv elfeed
elfeed-curl elfeed-log elfeed-db elfeed-lib url-queue url-file 
xml-query
fontaine modus-vivendi-theme modus-themes embark-consult 
consult-vertico
consult magit-bookmark bookmark embark-org org-element org-persist 
xdg
org-id org-refile avl-tree embark ffap marginalia savehist 
orderless
vertico kind-icon svg-lib svg dom xml corfu-terminal popon corfu 
eglot
external-completion array ert debug backtrace jsonrpc posframe
yasnippet-snippets yasnippet flycheck avy org-toc org-contrib 
org-clock
ob-shell ob-elixir async ob-async org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-src ob-comint org-pcomplete org-list
org-footnote org-faces org-entities ob-emacs-lisp ob-core ob-eval
org-cycle org-table ol org-fold org-fold-core org-keys oc 
org-loaddefs
cal-menu calendar cal-loaddefs org-version org-compat org-macs 
yaml-mode
terraform-mode hcl-mode php-mode mode-local find-func speedbar 
ezimage
dframe cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles 
cc-align
php-face php php-project flymake-proc flymake cc-engine cc-vars 
cc-defs
erlang-start erlang etags fileloop generator xref project tempo 
align
mix use-package-bind-key elixir-ts-mode dockerfile-mode sh-script 
smie
executable pcmpl-args pcmpl-linux pcmpl-unix git-modes 
gitignore-mode
gitconfig-mode conf-mode gitattributes-mode diff-hl log-view 
vc-dir ewoc
vc forge-list forge-commands forge-semi forge-bitbucket buck 
forge-gogs
gogs forge-gitea gtea forge-gitlab glab forge-github ghub-graphql 
treepy
gsexp ghub url-http url-gw nsm url-auth let-alist gnutls 
forge-notify
forge-revnote forge-pullreq forge-issue forge-topic yaml 
parse-time
iso8601 bug-reference forge-post markdown-mode color forge-repo 
forge
forge-core forge-db closql emacsql-sqlite emacsql-sqlite-common 
emacsql
emacsql-compiler git-timemachine vc-git vc-dispatcher 
magit-submodule
magit-blame magit-stash magit-reflog magit-bisect magit-push 
magit-pull
magit-fetch magit-clone magit-remote magit-commit magit-sequence
magit-notes magit-worktree magit-tag magit-merge magit-branch
magit-reset magit-files magit-refs magit-status magit magit-repos
magit-apply magit-wip magit-log magit-diff smerge-mode diff 
diff-mode
git-commit log-edit message sendmail yank-media puny rfc822 mml 
mml-sec
epa epg rfc6068 epg-config gnus-util time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums
mail-prsvr mailabbrev mail-utils gmm-utils mailheader pcvs-util 
add-log
magit-core magit-autorevert magit-margin magit-transient 
magit-process
with-editor comp comp-cstr warnings rx shell pcomplete server 
magit-mode
transient magit-git magit-base magit-section format-spec crm 
compat
functions restclient advice noutline outline which-key ripgrep 
thingatpt
grep ag vc-svn compile text-property-search comint ansi-osc 
ansi-color
find-dired s dired-rainbow dired-subtree dired-hacks-utils dash 
dired
dired-loaddefs all-the-icons all-the-icons-faces data-material
data-weathericons data-octicons data-fileicons data-faicons
data-alltheicons exec-path-from-shell edmacro kmacro bind-key 
easy-mmode
use-package-ensure use-package-core finder-inf 
erc-hl-nicks-autoloads
forge-autoloads php-mode-autoloads kind-icon-autoloads 
erlang-autoloads
eat-autoloads flycheck-package-autoloads package-lint-autoloads
mastodon-autoloads persist-autoloads request-autoloads 
ripgrep-autoloads
terraform-mode-autoloads dired-subtree-autoloads 
yaml-mode-autoloads
avy-autoloads closql-autoloads srht-autoloads plz-autoloads
markdown-mode-autoloads verb-autoloads restclient-autoloads
embark-consult-autoloads embark-autoloads erc-image-autoloads
mu4e-alert-autoloads elfeed-autoloads cape-autoloads
marginalia-autoloads hcl-mode-autoloads corfu-terminal-autoloads
popon-autoloads package-build-autoloads derived yaml-autoloads
dockerfile-mode-autoloads treesit-auto-autoloads treesit-auto 
treesit
magit-autoloads pcase git-timemachine-autoloads 
emmet-mode-autoloads
ht-autoloads org-roam-autoloads magit-section-autoloads
emacsql-autoloads ag-autoloads posframe-autoloads
yasnippet-snippets-autoloads yasnippet-autoloads 
which-key-autoloads
ox-hugo-autoloads fontaine-autoloads diff-hl-autoloads
ob-async-autoloads async-autoloads all-the-icons-autoloads 
ts-autoloads
exunit-autoloads denote-autoloads f-autoloads s-autoloads
git-modes-autoloads tomelr-autoloads vertico-autoloads
flycheck-autoloads pkg-info-autoloads epl-autoloads deft-autoloads
git-commit-autoloads with-editor-autoloads ob-elixir-autoloads
pcmpl-args-autoloads consult-dir-autoloads consult-autoloads
mix-autoloads dired-rainbow-autoloads dired-hacks-utils-autoloads
orderless-autoloads dash-autoloads alert-autoloads log4e-autoloads
gntp-autoloads ghub-autoloads treepy-autoloads corfu-autoloads
org-contrib-autoloads web-mode-autoloads transient-autoloads info
compat-autoloads exec-path-from-shell-autoloads svg-lib-autoloads
package browse-url url url-proxy url-privacy url-expand 
url-methods
url-history url-cookie generate-lisp-file url-domsuf url-util 
mailcap
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
password-cache json map byte-opt gv bytecomp byte-compile url-vars
native cl-extra help-mode pixel-scroll cua-base subr-x repeat 
winner
ring saveplace which-func imenu autorevert filenotify cus-edit pp
cus-load icons wid-edit cl-loaddefs cl-lib early-init rmc 
iso-transl
tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win 
term/common-win
pgtk-dnd tool-bar dnd fontset image regexp-opt fringe 
tabulated-list
replace newcomment text-mode lisp-mode prog-mode register page 
tab-bar
menu-bar rfn-eshadow isearch easymenu timer select scroll-bar 
mouse
jit-lock font-lock syntax font-core term/tty-colors frame 
minibuffer
nadvice seq simple cl-generic indonesian philippine cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech 
european
ethiopic indian cyrillic chinese composite emoji-zwj charscript 
charprop
case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face 
macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable 
backquote
threads dbusbind inotify dynamic-setting system-font-setting
font-render-setting cairo gtk pgtk lcms2 multi-tty 
make-network-process
native-compile emacs)

Memory information:
((conses 16 1519596 4641956)
 (symbols 48 97383 393)
 (strings 32 400012 126802)
 (string-bytes 1 12971136)
 (vectors 16 149074)
 (vector-slots 8 4019930 2141693)
 (floats 8 1373 14870)
 (intervals 56 61927 21199)
 (buffers 984 80))

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 21/03/2023 16:00, Wilhelm Kirschbaum wrote:
> 
> With the following text in a buffer with elixir-ts-mode:
> 
> def foo() do
> end
> 
> When I delete the `)` and re-insert it, the tree-sitter syntax tree does
> not fully recover and the `do end` part is absent when observing using
> treesit-explore-mode.
> 
> Before removing the `)`:
> 
> (source
> (call target: (identifier)
>   (arguments
>    (call target: (identifier)
>     (arguments ( ))))
>   (do_block do end)))
> 
> after re-inserting `)`:
> 
> (source
> (call target: (identifier)
>   (arguments
>    (call target: (identifier)
>     (arguments ( ))))))
> 
> I can see the same behaviour using ruby-ts-mode when I remove the
> following expression:
> 
> (setq-local syntax-propertize-function #'ruby-ts--syntax-propertize)
> 
> The ruby syntax I tested with:
> 
> def foo()
> end
> 
> When I add a local `syntax-propertize-function' function to elixir-ts-mode
> it also solves the problem of not fully recovering the tree-sitter
> syntax tree.  It does not seem to matter what the function achieves
> thought, so am a bit confused.

I can repro both scenarios. Except with ruby-ts-mode, the difference in 
printed AST is the added wrapping in (program ...). But the visible 
result is similar: "end" not being highlighted.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Yuan Fu]

[-- Attachment #1: Type: text/plain, Size: 2242 bytes --]



> On Mar 22, 2023, at 6:03 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> On 21/03/2023 16:00, Wilhelm Kirschbaum wrote:
>> With the following text in a buffer with elixir-ts-mode:
>> def foo() do
>> end
>> When I delete the `)` and re-insert it, the tree-sitter syntax tree does
>> not fully recover and the `do end` part is absent when observing using
>> treesit-explore-mode.
>> Before removing the `)`:
>> (source
>> (call target: (identifier)
>>  (arguments
>>   (call target: (identifier)
>>    (arguments ( ))))
>>  (do_block do end)))
>> after re-inserting `)`:
>> (source
>> (call target: (identifier)
>>  (arguments
>>   (call target: (identifier)
>>    (arguments ( ))))))
>> I can see the same behaviour using ruby-ts-mode when I remove the
>> following expression:
>> (setq-local syntax-propertize-function #'ruby-ts--syntax-propertize)
>> The ruby syntax I tested with:
>> def foo()
>> end
>> When I add a local `syntax-propertize-function' function to elixir-ts-mode
>> it also solves the problem of not fully recovering the tree-sitter
>> syntax tree.  It does not seem to matter what the function achieves
>> thought, so am a bit confused.
> 
> I can repro both scenarios. Except with ruby-ts-mode, the difference in printed AST is the added wrapping in (program ...). But the visible result is similar: "end" not being highlighted.

The problem seems to be caused by narrowing during fontification: when parsing, tree-sitter sees that BUF_ZV_BYTE is smaller than it’s visible_end, and removed the part after the parenthesis from the parse tree. I suspect this has to do with the narrowed fontification we added for dealing with long lines. 

CC’ed Eli because I don’t really know anything about the locked narrow thing besides that it exists. In particular, how does it determine when and where to narrow?


To see it in action, apply this patch, and repeat Wilhelm’s experiment: remove the “)” and reinsert it. You should see

start: 9, oend: 10, nend: 9
start: 9, oend: 10, nend: 9
start: 9, oend: 9, nend: 10
start: 9, oend: 9, nend: 10
2.5 start: 10, oend: 18, nend: 10 —> culprit

(Make sure to turn off electric-pair-mode to reduce noise.)

Yuan


[-- Attachment #2: print.diff --]
[-- Type: application/octet-stream, Size: 2389 bytes --]

diff --git a/src/treesit.c b/src/treesit.c
index 5a4fe3e8803..9b46dcef2d1 100644
--- a/src/treesit.c
+++ b/src/treesit.c
@@ -806,6 +806,7 @@ treesit_record_change (ptrdiff_t start_byte, ptrdiff_t old_end_byte,
 	  eassert (start_offset <= old_end_offset);
 	  eassert (start_offset <= new_end_offset);
 
+	  printf("start: %ld, oend: %ld, nend: %ld\n", start_offset, old_end_offset, new_end_offset);
 	  treesit_tree_edit_1 (tree, start_offset, old_end_offset,
 			       new_end_offset);
 	  XTS_PARSER (lisp_parser)->need_reparse = true;
@@ -925,6 +926,7 @@ treesit_sync_visible_region (Lisp_Object parser)
   if (visible_beg > BUF_BEGV_BYTE (buffer))
     {
       /* Tree-sitter sees: insert at the beginning.  */
+      printf("1 start: %d, oend: %d, nend: %ld\n", 0, 0, visible_beg - BUF_BEGV_BYTE (buffer));
       treesit_tree_edit_1 (tree, 0, 0, visible_beg - BUF_BEGV_BYTE (buffer));
       visible_beg = BUF_BEGV_BYTE (buffer);
       eassert (visible_beg <= visible_end);
@@ -933,6 +935,9 @@ treesit_sync_visible_region (Lisp_Object parser)
   if (visible_end < BUF_ZV_BYTE (buffer))
     {
       /* Tree-sitter sees: insert at the end.  */
+      printf("2 start: %ld, oend: %ld, nend: %ld\n", visible_end - visible_beg,
+	     visible_end - visible_beg,
+	     BUF_ZV_BYTE (buffer) - visible_beg);
       treesit_tree_edit_1 (tree, visible_end - visible_beg,
 			   visible_end - visible_beg,
 			   BUF_ZV_BYTE (buffer) - visible_beg);
@@ -942,6 +947,9 @@ treesit_sync_visible_region (Lisp_Object parser)
   else if (visible_end > BUF_ZV_BYTE (buffer))
     {
       /* Tree-sitter sees: delete at the end.  */
+      printf("2.5 start: %ld, oend: %ld, nend: %ld\n", BUF_ZV_BYTE (buffer) - visible_beg,
+			   visible_end - visible_beg,
+			   BUF_ZV_BYTE (buffer) - visible_beg);
       treesit_tree_edit_1 (tree, BUF_ZV_BYTE (buffer) - visible_beg,
 			   visible_end - visible_beg,
 			   BUF_ZV_BYTE (buffer) - visible_beg);
@@ -952,6 +960,7 @@ treesit_sync_visible_region (Lisp_Object parser)
   if (visible_beg < BUF_BEGV_BYTE (buffer))
     {
       /* Tree-sitter sees: delete at the beginning.  */
+      printf("3 start: %d, oend: %ld, nend: %d\n", 0, BUF_BEGV_BYTE (buffer) - visible_beg, 0);
       treesit_tree_edit_1 (tree, 0, BUF_BEGV_BYTE (buffer) - visible_beg, 0);
       visible_beg = BUF_BEGV_BYTE (buffer);
       eassert (visible_beg <= visible_end);

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 23/03/2023 05:03, Yuan Fu wrote:
> I suspect this has to do with the narrowed fontification we added for dealing with long lines.

I'm pretty sure that one should only get used when the buffer contains a 
line longer than 10000 characters.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Thu, 23 Mar 2023 06:20:19 +0200
> Cc: Wilhelm Kirschbaum <wkirschbaum@gmail.com>, 62333@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 23/03/2023 05:03, Yuan Fu wrote:
> > I suspect this has to do with the narrowed fontification we added for dealing with long lines.
> 
> I'm pretty sure that one should only get used when the buffer contains a 
> line longer than 10000 characters.

Yes (but I believe the threshold is 50K characters).

The narrowing must come from somewhere else.  Yuan, how did you see
that ZV was changed?  Can you run with a breakpoint in
Fnarrow_to_region, and see which Lisp code does that?

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Yuan Fu]

> On Mar 23, 2023, at 12:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Date: Thu, 23 Mar 2023 06:20:19 +0200
>> Cc: Wilhelm Kirschbaum <wkirschbaum@gmail.com>, 62333@debbugs.gnu.org,
>> Eli Zaretskii <eliz@gnu.org>
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> 
>> On 23/03/2023 05:03, Yuan Fu wrote:
>>> I suspect this has to do with the narrowed fontification we added for dealing with long lines.
>> 
>> I'm pretty sure that one should only get used when the buffer contains a 
>> line longer than 10000 characters.
> 
> Yes (but I believe the threshold is 50K characters).
> 
> The narrowing must come from somewhere else.  Yuan, how did you see
> that ZV was changed?  Can you run with a breakpoint in
> Fnarrow_to_region, and see which Lisp code does that?

I did that, some observations:

1. This doesn’t reproduce on Linux (it does on Mac).
2. The function that narrowed the region is blink-matching-open
3. If I (advice-add 'blink-matching-open :around #’ignore) on my Mac, it “solves” the problem

I guess the question now is why redisplay is triggered in blink-matching-open on Mac?

Yuan

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 23/03/2023 23:18, Yuan Fu wrote:
> 1. This doesn’t reproduce on Linux (it does on Mac).

It does reproduce on my Linux machine, though.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 23/03/2023 23:18, Yuan Fu wrote:
> I guess the question now is why redisplay is triggered in blink-matching-open

blink-matching-open calls sit-for after adjusting overlays.

sit-for starts with a redisplay.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Yuan Fu]

> On Mar 23, 2023, at 3:06 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> On 23/03/2023 23:18, Yuan Fu wrote:
>> I guess the question now is why redisplay is triggered in blink-matching-open
> 
> blink-matching-open calls sit-for after adjusting overlays.
> 
> sit-for starts with a redisplay.

But it’s not called when narrow-to-region is in effect.

Yuan

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Thu, 23 Mar 2023 16:59:28 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  Wilhelm Kirschbaum <wkirschbaum@gmail.com>,
>  62333@debbugs.gnu.org
> 
> > On Mar 23, 2023, at 3:06 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> > 
> > On 23/03/2023 23:18, Yuan Fu wrote:
> >> I guess the question now is why redisplay is triggered in blink-matching-open
> > 
> > blink-matching-open calls sit-for after adjusting overlays.
> > 
> > sit-for starts with a redisplay.
> 
> But it’s not called when narrow-to-region is in effect.

Exactly.  So the question is now: why does treesit.c see ZV changed,
if by the time it is supposed to be called from redisplay the
restriction is restored?

Is the problematic code in treesit.c really called from redisplay
triggered by sit-for in blink-matching-open?  Can you show the C and
Lisp backtrace from the call that "sees that BUF_ZV_BYTE is smaller
than its visible_end"?

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Yuan Fu]

> On Mar 23, 2023, at 11:05 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Thu, 23 Mar 2023 16:59:28 -0700
>> Cc: Eli Zaretskii <eliz@gnu.org>,
>> Wilhelm Kirschbaum <wkirschbaum@gmail.com>,
>> 62333@debbugs.gnu.org
>> 
>>> On Mar 23, 2023, at 3:06 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
>>> 
>>> On 23/03/2023 23:18, Yuan Fu wrote:
>>>> I guess the question now is why redisplay is triggered in blink-matching-open
>>> 
>>> blink-matching-open calls sit-for after adjusting overlays.
>>> 
>>> sit-for starts with a redisplay.
>> 
>> But it’s not called when narrow-to-region is in effect.
> 
> Exactly.  So the question is now: why does treesit.c see ZV changed,
> if by the time it is supposed to be called from redisplay the
> restriction is restored?
> 
> Is the problematic code in treesit.c really called from redisplay
> triggered by sit-for in blink-matching-open?  Can you show the C and
> Lisp backtrace from the call that "sees that BUF_ZV_BYTE is smaller
> than its visible_end"?

Ah, I made a mistake, it isn’t that it doesn’t reproduce on Linux, but rather it doesn’t reproduce on emacs-29. Since I’m able to reproduce it on Linux, I set a breakpoint and the backtrace explains it: blink-match-open calls forward-sexp, which calls elixir-ts—forward-sexp, which accesses the parse tree, which causes tree-sitter to reparse in the narrowed context.

But normally tree-sitter will reparse again when redisplay fontifies the buffer, because now the visible portion is back to the whole buffer. Tree-sitter didn’t do that because I made a mistake when modifying treesit_ensure_parsed. I moved the code that checks for the need_reparse flag before treesit_sync_visible_region, which sets the need_reparse flag if it detects that narrow situation has changed.

Well, this is a bit embarrassing, there is even a comment warning this mistake, but anyway, I pushed a fix, once it gets merged into master, the problem should go away. 

As things stands right now, every time blink-match-open blinks the matching parenthesis, tree-sitter would reparse the buffer twice, that’s not exactly ideal. Blink-match-open’s use of narrowing is legit, and we can’t just automatically widen in tree-sitter functions.

Yuan

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 24/03/2023 09:34, Yuan Fu wrote:
 > Well, this is a bit embarrassing, there is even a comment warning 
this mistake, but anyway, I pushed a fix, once it gets merged into 
master, the problem should go away.

Excellent, thanks.

> As things stands right now, every time blink-match-open blinks the matching parenthesis, tree-sitter would reparse the buffer twice, that’s not exactly ideal. Blink-match-open’s use of narrowing is legit, and we can’t just automatically widen in tree-sitter functions.

Not ideal indeed.

Aside from the performance impact, we could also see cases where the 
"narrowed" parse tree contains errors (due to incomplete code) which 
make the tree itself less useful. Especially when the narrowing's end is 
closer to the current position than in blink-matching-open's usage.

Not sure how to solve that, but suppose we had a way to convey to 
treesit.c that it shouldn't change the visibility bounds for the parser 
in this particular instance? Though that would raise the issue of code 
trying to use node positions beyond the current accessible range.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Sat, 25 Mar 2023 03:51:48 +0200
> Cc: Wilhelm Kirschbaum <wkirschbaum@gmail.com>, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 24/03/2023 09:34, Yuan Fu wrote:
>  > Well, this is a bit embarrassing, there is even a comment warning 
> this mistake, but anyway, I pushed a fix, once it gets merged into 
> master, the problem should go away.
> 
> Excellent, thanks.
> 
> > As things stands right now, every time blink-match-open blinks the matching parenthesis, tree-sitter would reparse the buffer twice, that’s not exactly ideal. Blink-match-open’s use of narrowing is legit, and we can’t just automatically widen in tree-sitter functions.
> 
> Not ideal indeed.
> 
> Aside from the performance impact, we could also see cases where the 
> "narrowed" parse tree contains errors (due to incomplete code) which 
> make the tree itself less useful. Especially when the narrowing's end is 
> closer to the current position than in blink-matching-open's usage.
> 
> Not sure how to solve that, but suppose we had a way to convey to 
> treesit.c that it shouldn't change the visibility bounds for the parser 
> in this particular instance? Though that would raise the issue of code 
> trying to use node positions beyond the current accessible range.

Exactly.  Which is why I don't think this is the right way.

Is there any real reason blink-matching-open narrows the buffer?  If
we could remove that narrowing, the problem with the parser's taking
notice of it would be gone.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 25/03/2023 14:34, Eli Zaretskii wrote:
>> Not ideal indeed.
>>
>> Aside from the performance impact, we could also see cases where the
>> "narrowed" parse tree contains errors (due to incomplete code) which
>> make the tree itself less useful. Especially when the narrowing's end is
>> closer to the current position than in blink-matching-open's usage.
>>
>> Not sure how to solve that, but suppose we had a way to convey to
>> treesit.c that it shouldn't change the visibility bounds for the parser
>> in this particular instance? Though that would raise the issue of code
>> trying to use node positions beyond the current accessible range.
> Exactly.  Which is why I don't think this is the right way.
> 
> Is there any real reason blink-matching-open narrows the buffer?  If
> we could remove that narrowing, the problem with the parser's taking
> notice of it would be gone.

Performance: to avoid scanning for the matching paren too far in the buffer.

Which might seem like not that big a deal, or could even be handled in a 
special way here using the parse tree, but narrowing has been used for 
this purpose for a long time (to limit various kinds of searches or 
movements), so fundamentally the problem will still be with us.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Sat, 25 Mar 2023 15:00:24 +0200
> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 25/03/2023 14:34, Eli Zaretskii wrote:
> > 
> > Is there any real reason blink-matching-open narrows the buffer?  If
> > we could remove that narrowing, the problem with the parser's taking
> > notice of it would be gone.
> 
> Performance: to avoid scanning for the matching paren too far in the buffer.

If that's the only reason, then tree-sitter based modes could widen
back in their sexp-moving functions, since they use the parse data for
this, right?

> Which might seem like not that big a deal, or could even be handled in a 
> special way here using the parse tree, but narrowing has been used for 
> this purpose for a long time (to limit various kinds of searches or 
> movements), so fundamentally the problem will still be with us.

There are many fundamental problems we cannot solve, but I'd settle
with solving some of their particular instances, one at a time.  In
this case, the problem is caused by a mode-specific function that uses
parser data, and is called inside a restriction.  So let's try solving
this particular combination of factors, if we can.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 25/03/2023 15:14, Eli Zaretskii wrote:
>> Date: Sat, 25 Mar 2023 15:00:24 +0200
>> Cc:wkirschbaum@gmail.com,casouri@gmail.com,62333@debbugs.gnu.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> On 25/03/2023 14:34, Eli Zaretskii wrote:
>>> Is there any real reason blink-matching-open narrows the buffer?  If
>>> we could remove that narrowing, the problem with the parser's taking
>>> notice of it would be gone.
>> Performance: to avoid scanning for the matching paren too far in the buffer.
> If that's the only reason, then tree-sitter based modes could widen
> back in their sexp-moving functions, since they use the parse data for
> this, right?

Not necessarily: it doesn't know the purpose for which the narrowing was 
applied. Could be for a mixed-major-mode thing, or some other purpose. 
Long lines?

Do we recall the exact design idea why tree-sitter visibility is limited 
by the narrowing bounds? Because if we wanted to widen in all similar 
situations, we might as well make it not obey the narrowing at all.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Sat, 25 Mar 2023 15:44:18 +0200
> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > If that's the only reason, then tree-sitter based modes could widen
> > back in their sexp-moving functions, since they use the parse data for
> > this, right?
> 
> Not necessarily: it doesn't know the purpose for which the narrowing was 
> applied. Could be for a mixed-major-mode thing, or some other purpose. 

mixed-major-mode shouldn't be a problem.

> Long lines?

Easy to test, and the call to widen will do nothing anyway in that
case.

> Do we recall the exact design idea why tree-sitter visibility is limited 
> by the narrowing bounds?

For the same reason: because the buffer is inaccessible outside the
restriction, and tree-sitter wants to access the buffer text.

> Because if we wanted to widen in all similar situations, we might as
> well make it not obey the narrowing at all.

It is impossible to not obey narrowing, not in Emacs.  I told that and
explained that many times already, including simple examples of what
trouble this could cause to even the most innocent Lisp code.  I hoped
that by now this should no longer be brought forward.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 25/03/2023 16:09, Eli Zaretskii wrote:
>> Date: Sat, 25 Mar 2023 15:44:18 +0200
>> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> If that's the only reason, then tree-sitter based modes could widen
>>> back in their sexp-moving functions, since they use the parse data for
>>> this, right?
>>
>> Not necessarily: it doesn't know the purpose for which the narrowing was
>> applied. Could be for a mixed-major-mode thing, or some other purpose.
> 
> mixed-major-mode shouldn't be a problem.

Why wouldn't it?

>> Long lines?
> 
> Easy to test, and the call to widen will do nothing anyway in that
> case.

Okay. Because of locked narrowing, I guess.

>> Do we recall the exact design idea why tree-sitter visibility is limited
>> by the narrowing bounds?
> 
> For the same reason: because the buffer is inaccessible outside the
> restriction, and tree-sitter wants to access the buffer text.
> 
>> Because if we wanted to widen in all similar situations, we might as
>> well make it not obey the narrowing at all.
> 
> It is impossible to not obey narrowing, not in Emacs.  I told that and
> explained that many times already, including simple examples of what
> trouble this could cause to even the most innocent Lisp code.  I hoped
> that by now this should no longer be brought forward.

Okay. But do you advocate all uses of tree-sitter to (widen) first?

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Sat, 25 Mar 2023 16:18:12 +0200
> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > mixed-major-mode shouldn't be a problem.
> 
> Why wouldn't it?

Because the sexp ends inside the current mode's block.

> >> Long lines?
> > 
> > Easy to test, and the call to widen will do nothing anyway in that
> > case.
> 
> Okay. Because of locked narrowing, I guess.

Yes.

> >> Because if we wanted to widen in all similar situations, we might as
> >> well make it not obey the narrowing at all.
> > 
> > It is impossible to not obey narrowing, not in Emacs.  I told that and
> > explained that many times already, including simple examples of what
> > trouble this could cause to even the most innocent Lisp code.  I hoped
> > that by now this should no longer be brought forward.
> 
> Okay. But do you advocate all uses of tree-sitter to (widen) first?

No, just in sexp-movement functions, at least for now.  If we discover
this is need in many more situations, we might consider more drastic
measures.  But I don't think we are there yet.  We've discovered just
one such case, in just one such mode, and it's on master.  We still
have ample time to see how widespread this is.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 25/03/2023 16:41, Eli Zaretskii wrote:
>> Date: Sat, 25 Mar 2023 16:18:12 +0200
>> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> mixed-major-mode shouldn't be a problem.
>>
>> Why wouldn't it?
> 
> Because the sexp ends inside the current mode's block.

But if the mmm framework narrowed the region to the current mode's 
block, widening will force tree-sitter to parse the whole buffer. Which 
might even result in parse errors (instead of meaningful parse nodes) 
near point.

Or consider the case when the code in the current snippet is incomplete 
(e.g. a missing paren). Then it might find a "match" somewhere far outside.

>>>> Because if we wanted to widen in all similar situations, we might as
>>>> well make it not obey the narrowing at all.
>>>
>>> It is impossible to not obey narrowing, not in Emacs.  I told that and
>>> explained that many times already, including simple examples of what
>>> trouble this could cause to even the most innocent Lisp code.  I hoped
>>> that by now this should no longer be brought forward.
>>
>> Okay. But do you advocate all uses of tree-sitter to (widen) first?
> 
> No, just in sexp-movement functions, at least for now.  If we discover
> this is need in many more situations, we might consider more drastic
> measures.  But I don't think we are there yet.  We've discovered just
> one such case, in just one such mode, and it's on master.  We still
> have ample time to see how widespread this is.

I'm not sure whether tree-sitter is going to be used with mmm-mode (or 
similar) a lot, since it has its own provisions for mixing languages.

But I'd also like to consider the other cases where we *don't* want to 
widen first. Any of them come to mind?

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Sat, 25 Mar 2023 17:25:17 +0200
> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 25/03/2023 16:41, Eli Zaretskii wrote:
> >> Date: Sat, 25 Mar 2023 16:18:12 +0200
> >> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >>
> >>> mixed-major-mode shouldn't be a problem.
> >>
> >> Why wouldn't it?
> > 
> > Because the sexp ends inside the current mode's block.
> 
> But if the mmm framework narrowed the region to the current mode's 
> block, widening will force tree-sitter to parse the whole buffer.

No, because such a mode mode should already make sure this doesn't
happen.

> >> Okay. But do you advocate all uses of tree-sitter to (widen) first?
> > 
> > No, just in sexp-movement functions, at least for now.  If we discover
> > this is need in many more situations, we might consider more drastic
> > measures.  But I don't think we are there yet.  We've discovered just
> > one such case, in just one such mode, and it's on master.  We still
> > have ample time to see how widespread this is.
> 
> I'm not sure whether tree-sitter is going to be used with mmm-mode (or 
> similar) a lot, since it has its own provisions for mixing languages.
> 
> But I'd also like to consider the other cases where we *don't* want to 
> widen first. Any of them come to mind?

No, not off the top of my head.

I think we should try this in this one case, and see if other cases
come up.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

[-- Attachment #1: Type: text/html, Size: 2521 bytes --]

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: "wkirschbaum@gmail.com" <wkirschbaum@gmail.com>,
> 	"casouri@gmail.com" <casouri@gmail.com>,
> 	"62333@debbugs.gnu.org" <62333@debbugs.gnu.org>
> Date: Sat, 25 Mar 2023 19:03:45 +0300
> 
>   But if the mmm framework narrowed the region to the current mode's 
>   block, widening will force tree-sitter to parse the whole buffer.
> 
>  No, because such a mode mode should already make sure this doesn't
>  happen.
> 
> How?

The same way it makes sure a given parser is used only on the portion
of the buffer where the corresponding language is used.

>   But I'd also like to consider the other cases where we *don't* want to 
>   widen first. Any of them come to mind?
> 
>  No, not off the top of my head.
> 
>  I think we should try this in this one case, and see if other cases
>  come up.
> 
> I think we should try to answer the relevant questions first.

If we can, sure.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 25/03/2023 18:24, Eli Zaretskii wrote:
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Cc: "wkirschbaum@gmail.com" <wkirschbaum@gmail.com>,
>> 	"casouri@gmail.com" <casouri@gmail.com>,
>> 	"62333@debbugs.gnu.org" <62333@debbugs.gnu.org>
>> Date: Sat, 25 Mar 2023 19:03:45 +0300
>>
>>    But if the mmm framework narrowed the region to the current mode's
>>    block, widening will force tree-sitter to parse the whole buffer.
>>
>>   No, because such a mode mode should already make sure this doesn't
>>   happen.
>>
>> How?
> 
> The same way it makes sure a given parser is used only on the portion
> of the buffer where the corresponding language is used.

It uses narrowing. Major modes are not supposed to 'widen' willy-nilly.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Sat, 25 Mar 2023 19:05:13 +0200
> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 25/03/2023 18:24, Eli Zaretskii wrote:
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >> Cc: "wkirschbaum@gmail.com" <wkirschbaum@gmail.com>,
> >> 	"casouri@gmail.com" <casouri@gmail.com>,
> >> 	"62333@debbugs.gnu.org" <62333@debbugs.gnu.org>
> >> Date: Sat, 25 Mar 2023 19:03:45 +0300
> >>
> >>    But if the mmm framework narrowed the region to the current mode's
> >>    block, widening will force tree-sitter to parse the whole buffer.
> >>
> >>   No, because such a mode mode should already make sure this doesn't
> >>   happen.
> >>
> >> How?
> > 
> > The same way it makes sure a given parser is used only on the portion
> > of the buffer where the corresponding language is used.
> 
> It uses narrowing.

How does that work with features such as font-lock, which do widen?

Anyway, isn't this discussion a bit premature, as no TS mode has been
used with the mmm framework yet?  Because when they do, I imagine they
will need to invent some mechanism that is more reliable than
narrowing.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Yuan Fu]

I don’t think we should blindly widen in tree-sitter sexp functions, but not because of mmm-mode—tree-sitter ranges should have that mostly covered. My primary concern is that what if narrowing is intended by the caller?

But I don’t have any good idea for blink-matching right now.

Yuan

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 25/03/2023 19:40, Eli Zaretskii wrote:
>> Date: Sat, 25 Mar 2023 19:05:13 +0200
>> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 25/03/2023 18:24, Eli Zaretskii wrote:
>>>> From: Dmitry Gutov <dgutov@yandex.ru>
>>>> Cc: "wkirschbaum@gmail.com" <wkirschbaum@gmail.com>,
>>>> 	"casouri@gmail.com" <casouri@gmail.com>,
>>>> 	"62333@debbugs.gnu.org" <62333@debbugs.gnu.org>
>>>> Date: Sat, 25 Mar 2023 19:03:45 +0300
>>>>
>>>>     But if the mmm framework narrowed the region to the current mode's
>>>>     block, widening will force tree-sitter to parse the whole buffer.
>>>>
>>>>    No, because such a mode mode should already make sure this doesn't
>>>>    happen.
>>>>
>>>> How?
>>>
>>> The same way it makes sure a given parser is used only on the portion
>>> of the buffer where the corresponding language is used.
>>
>> It uses narrowing.
> 
> How does that work with features such as font-lock, which do widen?

Using font-lock-dont-widen.

We've had this discussion several times over by now. Should it be 
documented somewhere?

> Anyway, isn't this discussion a bit premature, as no TS mode has been
> used with the mmm framework yet?

There is no reason to assume that: the combinations of modes are just a 
matter of user configuration. And so far it should be working okay.

> Because when they do, I imagine they
> will need to invent some mechanism that is more reliable than
> narrowing.

"They" is me. I'd rather the general mechanism keeps working, or we come 
up with another general mechanism that is a better substitute, before 
breaking this one.

And anyway, I like I mentioned, this will break this common pattern as well:

   (save-restriction
     (narrow-to-region ... some-limit-position)
     (forward-sexp))

I've used it in ruby-syntax-propertize-percent-literal, for example. 
Except with 'forward-list' rather than 'forward-sexp', but others can 
use the latter.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 25/03/2023 21:31, Yuan Fu wrote:
> 
> I don’t think we should blindly widen in tree-sitter sexp functions, but not because of mmm-mode—tree-sitter ranges should have that mostly covered. My primary concern is that what if narrowing is intended by the caller?

Yes.

> But I don’t have any good idea for blink-matching right now.

show-paren-mode has show-paren-data-function (which treesit should also 
plug into, I think).

Since we're looking for ad-hoc solutions, we could add a similar 
extension point for blink-matching-paren.

Or even try to reuse show-paren-data-function: just specialize in a 
limited number of cases - check that (nth 1 data) matches point, and 
then take the elements 3,4 and perform the existing checks on them. 
There might be some differences in behavior, but probably nothing some 
checks using 'syntax-after' couldn't handle.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 25 Mar 2023 12:31:58 -0700
> Cc: Dmitry Gutov <dgutov@yandex.ru>,
>  Wilhelm Kirschbaum <wkirschbaum@gmail.com>,
>  62333@debbugs.gnu.org
> 
> 
> I don’t think we should blindly widen in tree-sitter sexp functions, but not because of mmm-mode—tree-sitter ranges should have that mostly covered. My primary concern is that what if narrowing is intended by the caller?

What could be the reason for the caller to narrow when calling a
sexp-movement function?  Keep in mind that the only narrowing that
matters is one that prevents the sexp-movement function to find the
target of the movement.

> But I don’t have any good idea for blink-matching right now.

We could leave this alone for now, but in that case let's at least add
some FIXME comment in the relevant place(s), so that we could later
revisit this.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Sun, 26 Mar 2023 00:57:22 +0200
> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > How does that work with features such as font-lock, which do widen?
> 
> Using font-lock-dont-widen.

That's only for font-lock.  Parsing was not on the table when that was
introduced, so it doesn't have a similar mechanism.

> We've had this discussion several times over by now. Should it be 
> documented somewhere?

Probably, but that's a tangent.

> > Anyway, isn't this discussion a bit premature, as no TS mode has been
> > used with the mmm framework yet?
> 
> There is no reason to assume that: the combinations of modes are just a 
> matter of user configuration. And so far it should be working okay.

Again, I'm talking about using a parser library.  We may need to
introduce a way of limiting the parser to a certain range of buffer
text positions, independently of narrowing.  As we all know, narrowing
is a problematic feature to use in Lisp programs, so maybe we should
do this better in the case of parsers.  Then problems like this one
could be solved more cleanly and simply.

> And anyway, I like I mentioned, this will break this common pattern as well:
> 
>    (save-restriction
>      (narrow-to-region ... some-limit-position)
>      (forward-sexp))
> 
> I've used it in ruby-syntax-propertize-percent-literal, for example. 
> Except with 'forward-list' rather than 'forward-sexp', but others can 
> use the latter.

You want to repeat all the arguments we already brought up?

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 26/03/2023 08:04, Eli Zaretskii wrote:
>> Date: Sun, 26 Mar 2023 00:57:22 +0200
>> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> How does that work with features such as font-lock, which do widen?
>>
>> Using font-lock-dont-widen.
> 
> That's only for font-lock.  Parsing was not on the table when that was
> introduced, so it doesn't have a similar mechanism.

Parsing is on-demand, by font-lock and other features.

>> We've had this discussion several times over by now. Should it be
>> documented somewhere?
> 
> Probably, but that's a tangent.
> 
>>> Anyway, isn't this discussion a bit premature, as no TS mode has been
>>> used with the mmm framework yet?
>>
>> There is no reason to assume that: the combinations of modes are just a
>> matter of user configuration. And so far it should be working okay.
> 
> Again, I'm talking about using a parser library.  We may need to
> introduce a way of limiting the parser to a certain range of buffer
> text positions, independently of narrowing.

Except it's already limited by narrowing.

> As we all know, narrowing
> is a problematic feature to use in Lisp programs, so maybe we should
> do this better in the case of parsers.  Then problems like this one
> could be solved more cleanly and simply.

Narrowing problematic to use in Lisp?

>> And anyway, I like I mentioned, this will break this common pattern as well:
>>
>>     (save-restriction
>>       (narrow-to-region ... some-limit-position)
>>       (forward-sexp))
>>
>> I've used it in ruby-syntax-propertize-percent-literal, for example.
>> Except with 'forward-list' rather than 'forward-sexp', but others can
>> use the latter.
> 
> You want to repeat all the arguments we already brought up?

You might choose to ignore a third-party mode, but breaking a common 
pattern seems more dangerous.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Sun, 26 Mar 2023 12:25:30 +0300
> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 26/03/2023 08:04, Eli Zaretskii wrote:
> >> Date: Sun, 26 Mar 2023 00:57:22 +0200
> >> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >>
> >>> How does that work with features such as font-lock, which do widen?
> >>
> >> Using font-lock-dont-widen.
> > 
> > That's only for font-lock.  Parsing was not on the table when that was
> > introduced, so it doesn't have a similar mechanism.
> 
> Parsing is on-demand, by font-lock and other features.

So you are suggesting to introduce kludges like font-lock-dont-widen
in all of those places?  I don't even see how we will find them all in
advance, let alone fix them or make sure they do what we want.

> > Again, I'm talking about using a parser library.  We may need to
> > introduce a way of limiting the parser to a certain range of buffer
> > text positions, independently of narrowing.
> 
> Except it's already limited by narrowing.

Which is a fragile, semi-broken means, as we all know.

> > As we all know, narrowing
> > is a problematic feature to use in Lisp programs, so maybe we should
> > do this better in the case of parsers.  Then problems like this one
> > could be solved more cleanly and simply.
> 
> Narrowing problematic to use in Lisp?

Yes, because users can easily change narrowing.  We've had problems
with that many times, and even some attempts at solving it, so why are
you pretending you don't know about those deficiencies?

> >> And anyway, I like I mentioned, this will break this common pattern as well:
> >>
> >>     (save-restriction
> >>       (narrow-to-region ... some-limit-position)
> >>       (forward-sexp))
> >>
> >> I've used it in ruby-syntax-propertize-percent-literal, for example.
> >> Except with 'forward-list' rather than 'forward-sexp', but others can
> >> use the latter.
> > 
> > You want to repeat all the arguments we already brought up?
> 
> You might choose to ignore a third-party mode, but breaking a common 
> pattern seems more dangerous.

??? How does that follow from what I said?

Look, I'm trying to see how we could come up with an infrastructure
that will support multiple modes and other similar features in the
same buffer without relying on narrowing, thus bypassing the
disadvantages and difficulties that come with narrowing.  I think we
have a good chance here to come up with such a solution, specifically
for features that us a parsing library.  If you aren't interested in
discussing that, and think we should stick to narrowing, then this
goes nowhere, and I'd rather bow out of it.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 26/03/2023 13:01, Eli Zaretskii wrote:
>> Date: Sun, 26 Mar 2023 12:25:30 +0300
>> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 26/03/2023 08:04, Eli Zaretskii wrote:
>>>> Date: Sun, 26 Mar 2023 00:57:22 +0200
>>>> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
>>>> From: Dmitry Gutov <dgutov@yandex.ru>
>>>>
>>>>> How does that work with features such as font-lock, which do widen?
>>>>
>>>> Using font-lock-dont-widen.
>>>
>>> That's only for font-lock.  Parsing was not on the table when that was
>>> introduced, so it doesn't have a similar mechanism.
>>
>> Parsing is on-demand, by font-lock and other features.
> 
> So you are suggesting to introduce kludges like font-lock-dont-widen
> in all of those places?

font-lock-dont-widen is a kludge, but that's largely determined by the 
way it's defined and used.

If we take indent-for-tab-command, for example, it doesn't have such a 
variable, and doesn't really need to: the top-level command calls 
'widen', and then indent-line-function (set by major mode and altered by 
e.g. mmm-mode) is free to impose its specific bounds.

I think, as far the immediate problem is concerned, blink-matching-paren 
could use the same route. And could even try to reuse 
show-paren-data-function instead of creating its own customization point.

> I don't even see how we will find them all in
> advance, let alone fix them or make sure they do what we want.

Just do it step-by-step.

The "grand unified theory of mixed major modes" has been attempted a few 
times in the past, and never reached anything practical. That's not to 
say nobody should try again, but they should keep that in mind.

>>> Again, I'm talking about using a parser library.  We may need to
>>> introduce a way of limiting the parser to a certain range of buffer
>>> text positions, independently of narrowing.
>>
>> Except it's already limited by narrowing.
> 
> Which is a fragile, semi-broken means, as we all know.

What is a broken mess, is user-level narrowing. And how the downstream 
code can never guess the purpose the narrowing was applied for.

>>> As we all know, narrowing
>>> is a problematic feature to use in Lisp programs, so maybe we should
>>> do this better in the case of parsers.  Then problems like this one
>>> could be solved more cleanly and simply.
>>
>> Narrowing problematic to use in Lisp?
> 
> Yes, because users can easily change narrowing.  We've had problems
> with that many times, and even some attempts at solving it, so why are
> you pretending you don't know about those deficiencies?

Because I suggested deprecating certain uses of it several times, and 
always got a "no" for an answer.

>>>> And anyway, I like I mentioned, this will break this common pattern as well:
>>>>
>>>>      (save-restriction
>>>>        (narrow-to-region ... some-limit-position)
>>>>        (forward-sexp))
>>>>
>>>> I've used it in ruby-syntax-propertize-percent-literal, for example.
>>>> Except with 'forward-list' rather than 'forward-sexp', but others can
>>>> use the latter.
>>>
>>> You want to repeat all the arguments we already brought up?
>>
>> You might choose to ignore a third-party mode, but breaking a common
>> pattern seems more dangerous.
> 
> ??? How does that follow from what I said?
> 
> Look, I'm trying to see how we could come up with an infrastructure
> that will support multiple modes and other similar features in the
> same buffer without relying on narrowing, thus bypassing the
> disadvantages and difficulties that come with narrowing.  I think we
> have a good chance here to come up with such a solution, specifically
> for features that us a parsing library.  If you aren't interested in
> discussing that, and think we should stick to narrowing, then this
> goes nowhere, and I'd rather bow out of it.

What I've seen here so far is you suggesting we go ahead and break the 
existing convention and then let "them" (third-party authors including 
myself) come up with a new working one.

My stance here is we shouldn't break it before we create a new one. And 
I'm happy to discuss it, of course. Just please don't expect me to take 
the initiative at this point.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Yuan Fu]

> On Mar 25, 2023, at 4:29 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> On 25/03/2023 21:31, Yuan Fu wrote:
>> I don’t think we should blindly widen in tree-sitter sexp functions, but not because of mmm-mode—tree-sitter ranges should have that mostly covered. My primary concern is that what if narrowing is intended by the caller?
> 
> Yes.
> 
>> But I don’t have any good idea for blink-matching right now.
> 
> show-paren-mode has show-paren-data-function (which treesit should also plug into, I think).
> 
> Since we're looking for ad-hoc solutions, we could add a similar extension point for blink-matching-paren.
> 
> Or even try to reuse show-paren-data-function: just specialize in a limited number of cases - check that (nth 1 data) matches point, and then take the elements 3,4 and perform the existing checks on them. There might be some differences in behavior, but probably nothing some checks using 'syntax-after' couldn't handle.

Sounds good, I can work on this. This links to something I’ve been working on, namely some system that allows one to map abstract general constructs like pair, list, defun, assignment, etc, to grammar-specific definitions like a tree-sitter node type, and automatically construct functions accessing those information. We already have a primitive version of it: right now if you set treesit-defun-type-regexp, treesit-beginning/end-of-defun and treesit-defun-at-point would just work. I want to extend it further to allow you to get defun name, argument list, etc, and everything else I mentioned earlier.

If we can pull it off, a user only need to specify what is a pair and how to find the matching pair according to the language grammar, and the rest is automatic (a tree-sitter function for show-paren-data-function and blink-matching-paren).

Yuan

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Yuan Fu]

> On Mar 25, 2023, at 9:28 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sat, 25 Mar 2023 12:31:58 -0700
>> Cc: Dmitry Gutov <dgutov@yandex.ru>,
>> Wilhelm Kirschbaum <wkirschbaum@gmail.com>,
>> 62333@debbugs.gnu.org
>> 
>> 
>> I don’t think we should blindly widen in tree-sitter sexp functions, but not because of mmm-mode—tree-sitter ranges should have that mostly covered. My primary concern is that what if narrowing is intended by the caller?
> 
> What could be the reason for the caller to narrow when calling a
> sexp-movement function?  Keep in mind that the only narrowing that
> matters is one that prevents the sexp-movement function to find the
> target of the movement.

Maybe the user narrowed to a defun when writing code. There is a bug report on narrow-to-defun not working right in c-ts-mode, so there are definitely people who work like that.

>> But I don’t have any good idea for blink-matching right now.
> 
> We could leave this alone for now, but in that case let's at least add
> some FIXME comment in the relevant place(s), so that we could later
> revisit this.

Dmitry’s idea sounds good, we can add a tree-sitter backend for show-paren-data-function, and make the backend for blink-matching-paren configurable, then add a backend for it that uses show-paren-data-function, which in turn uses tree-sitter (or we add a backend that uses tree-sitter directly). In either case, the tree-sitter backend don’t need to use narrowing and we are good.

Yuan

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 27/03/2023 01:52, Yuan Fu wrote:
> If we can pull it off, a user only need to specify what is a pair and how to find the matching pair according to the language grammar, and the rest is automatic (a tree-sitter function for show-paren-data-function and blink-matching-paren).

Sounds good. Indeed, for this feature a registry of opener-closers 
should suffice.

Another command that could use it, is an implementation of 
backward-up-list. Somehow the default up-list manages to use SMIE's 
forward-sexp-function (and backward...) definitions automatically, but 
not tree-sitter's. I've been told it's expected for now, so I think 
opener-closers are the missing piece (either to enhance 
treesit-forward-sexp/treesit-backward-sexp to resemble the originals 
more closely, or to implement treesit-backward-up).

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Gregory Heytings]

>>>> Again, I'm talking about using a parser library.  We may need to 
>>>> introduce a way of limiting the parser to a certain range of buffer 
>>>> text positions, independently of narrowing.
>>> 
>>> Except it's already limited by narrowing.
>> 
>> Which is a fragile, semi-broken means, as we all know.
>
> What is a broken mess, is user-level narrowing. And how the downstream 
> code can never guess the purpose the narrowing was applied for.
>

Note that this is what labeled narrowings attempts to solve.  It's perhaps 
not the best way to deal with code that has existed for a long time 
(because such code is not prepared to handle that kind of narrowing 
gracefully), but I don't see why it couldn't be used in new code/modes 
such as tree-sitter ones.  It is independent of user-level narrowing, and 
downstream code can act differently depending on the reason for which the 
narrowing was applied.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Mon, 27 Mar 2023 01:00:20 +0300
> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > So you are suggesting to introduce kludges like font-lock-dont-widen
> > in all of those places?
> 
> font-lock-dont-widen is a kludge, but that's largely determined by the 
> way it's defined and used.

With parser-based features, we have an opportunity to do this in a
cleaner manner.

> If we take indent-for-tab-command, for example, it doesn't have such a 
> variable, and doesn't really need to: the top-level command calls 
> 'widen', and then indent-line-function (set by major mode and altered by 
> e.g. mmm-mode) is free to impose its specific bounds.

I thought about a lower-level, infrastructure-level, mechanism that
could be used to restrict a parser to a certain region of the buffer.
Then this could be used by every feature based on parsers, instead of
us having to invent a separate solution for each one.

> The "grand unified theory of mixed major modes" has been attempted a few 
> times in the past, and never reached anything practical.

But here we have a unique opportunity to maybe find a solution, at
least for stuff based on tree-sitter and similar libraries.  That
maybe not "grand", but certainly "respectable".

> >> Except it's already limited by narrowing.
> > 
> > Which is a fragile, semi-broken means, as we all know.
> 
> What is a broken mess, is user-level narrowing.

Which is why restricting parsers should not be a user-level feature.

> > Look, I'm trying to see how we could come up with an infrastructure
> > that will support multiple modes and other similar features in the
> > same buffer without relying on narrowing, thus bypassing the
> > disadvantages and difficulties that come with narrowing.  I think we
> > have a good chance here to come up with such a solution, specifically
> > for features that us a parsing library.  If you aren't interested in
> > discussing that, and think we should stick to narrowing, then this
> > goes nowhere, and I'd rather bow out of it.
> 
> What I've seen here so far is you suggesting we go ahead and break the 
> existing convention and then let "them" (third-party authors including 
> myself) come up with a new working one.

You are mixing two semi-separate issues: how to fix the immediate
problem with blink-matching-paren (if we consider it a serious
problem), and how to try solving similar issues as a whole.  The
latter is a long-term goal, and we can discuss it freely and calmly;
whereby ideas can be brought up even if they might not be the most
clever ones or the ones with which you personally agree.

> My stance here is we shouldn't break it before we create a new one.

No one broke anything.  We are just discussing ideas.  Please don't
exaggerate.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 26 Mar 2023 15:57:55 -0700
> Cc: Dmitry Gutov <dgutov@yandex.ru>,
>  Wilhelm Kirschbaum <wkirschbaum@gmail.com>,
>  62333@debbugs.gnu.org
> 
> > We could leave this alone for now, but in that case let's at least add
> > some FIXME comment in the relevant place(s), so that we could later
> > revisit this.
> 
> Dmitry’s idea sounds good, we can add a tree-sitter backend for show-paren-data-function, and make the backend for blink-matching-paren configurable, then add a backend for it that uses show-paren-data-function, which in turn uses tree-sitter (or we add a backend that uses tree-sitter directly). In either case, the tree-sitter backend don’t need to use narrowing and we are good.

Is this suitable for the release branch?  It sounds non-trivial, and
the problem is not very severe, to say the least.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Mon, 27 Mar 2023 08:24:42 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, wkirschbaum@gmail.com, casouri@gmail.com, 
>     62333@debbugs.gnu.org
> 
> >> Which is a fragile, semi-broken means, as we all know.
> >
> > What is a broken mess, is user-level narrowing. And how the downstream 
> > code can never guess the purpose the narrowing was applied for.
> 
> Note that this is what labeled narrowings attempts to solve.

Labeled narrowing cannot solve this because it does nothing to
alleviate the problems with user-defined narrowing.  So if the user
narrows the buffer, we cannot do anything to safely widen in the
general case, and labeled narrowing cannot help us.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Yuan Fu]

> On Mar 27, 2023, at 6:32 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sun, 26 Mar 2023 15:57:55 -0700
>> Cc: Dmitry Gutov <dgutov@yandex.ru>,
>> Wilhelm Kirschbaum <wkirschbaum@gmail.com>,
>> 62333@debbugs.gnu.org
>> 
>>> We could leave this alone for now, but in that case let's at least add
>>> some FIXME comment in the relevant place(s), so that we could later
>>> revisit this.
>> 
>> Dmitry’s idea sounds good, we can add a tree-sitter backend for show-paren-data-function, and make the backend for blink-matching-paren configurable, then add a backend for it that uses show-paren-data-function, which in turn uses tree-sitter (or we add a backend that uses tree-sitter directly). In either case, the tree-sitter backend don’t need to use narrowing and we are good.
> 
> Is this suitable for the release branch?  It sounds non-trivial, and
> the problem is not very severe, to say the least.

No, it’s intended for the master branch. Emacs-29 doesn’t have the blink-matching-paren problem anyway, since it doesn’t have treesit-forward-sexp. 

Yuan

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Gregory Heytings]

>>>>>> Again, I'm talking about using a parser library.  We may need to 
>>>>>> introduce a way of limiting the parser to a certain range of buffer 
>>>>>> text positions, independently of narrowing.
>>>>>
>>>>> Except it's already limited by narrowing.
>>>>
>>>> Which is a fragile, semi-broken means, as we all know.
>>>
>>> What is a broken mess, is user-level narrowing. And how the downstream 
>>> code can never guess the purpose the narrowing was applied for.
>>
>> Note that this is what labeled narrowings attempts to solve.
>
> Labeled narrowing cannot solve this because it does nothing to alleviate 
> the problems with user-defined narrowing.  So if the user narrows the 
> buffer, we cannot do anything to safely widen in the general case, and 
> labeled narrowing cannot help us.
>

I must be missing something, but I don't understand what you mean.  If a 
user has narrowed a buffer to a certain region A-B, and if we want to 
limit a parser to a certain region C-D, we can do that safely with

(with-narrowing C D :label 'parser-library BODY)

which will work, independently of the A-B narrowing set by the user, and 
will reset the narrowing to A-B upon return.  And functions inside BODY 
will have to use (without-narrowing :label 'parser-library ...) to get 
access to the whole buffer.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 27/03/2023 16:39, Eli Zaretskii wrote:
>> Date: Mon, 27 Mar 2023 08:24:42 +0000
>> From: Gregory Heytings<gregory@heytings.org>
>> cc: Eli Zaretskii<eliz@gnu.org>,wkirschbaum@gmail.com,casouri@gmail.com,
>>      62333@debbugs.gnu.org
>>
>>>> Which is a fragile, semi-broken means, as we all know.
>>> What is a broken mess, is user-level narrowing. And how the downstream
>>> code can never guess the purpose the narrowing was applied for.
>> Note that this is what labeled narrowings attempts to solve.
> Labeled narrowing cannot solve this because it does nothing to
> alleviate the problems with user-defined narrowing.  So if the user
> narrows the buffer, we cannot do anything to safely widen in the
> general case, and labeled narrowing cannot help us.

Is that because we don't think the user level narrowing is done purely 
for visual effect? There is, again, a fair amount of confusion there, 
but judging by regular user requests for make this or that command 
ignore user-level narrowing, it seems like "purely visual" should be the 
default interpretation. And for harder narrowing, users could have a 
separate command, or a prefit arg, or etc.

When I'm talking user-level, I mean interactive 'C-x n', of course.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 27/03/2023 11:24, Gregory Heytings wrote:
> 
>>>>> Again, I'm talking about using a parser library.  We may need to 
>>>>> introduce a way of limiting the parser to a certain range of buffer 
>>>>> text positions, independently of narrowing.
>>>>
>>>> Except it's already limited by narrowing.
>>>
>>> Which is a fragile, semi-broken means, as we all know.
>>
>> What is a broken mess, is user-level narrowing. And how the downstream 
>> code can never guess the purpose the narrowing was applied for.
>>
> 
> Note that this is what labeled narrowings attempts to solve.  It's 
> perhaps not the best way to deal with code that has existed for a long 
> time (because such code is not prepared to handle that kind of narrowing 
> gracefully), but I don't see why it couldn't be used in new code/modes 
> such as tree-sitter ones.  It is independent of user-level narrowing, 
> and downstream code can act differently depending on the reason for 
> which the narrowing was applied.

It indeed sounds like something we had considered before: different 
types of narrowing (at the time we only wanted to distinguish between 
two types: "soft" and "hard", the latter for use by mixed-mode features, 
or other stuff which wanted to preclude full widening, such as Info). 
But that fizzled out on the prototyping stage.

(Aside: tree-sitter itself has its own support for "ranges", which will 
work without any narrowing related stuff as long as grammars for all 
langs are available -- or perhaps the combination should be available as 
a grammar too, I'm not sure -- anyway, it doesn't need much help, but it 
would still be nice to support embedding tree-sitter managed code inside 
"regular" modes, or have better support for mixing said regular modes, 
something we already do.)

The new narrowing feature with labels kind of resembles that, except if 
I'm thinking about "intents", I could probably enumerate "visual" (for 
interactive narrowing), "hard" (for mixed-mode stuff or Info), or... 
"movement" (simply performed to constrain movement, but without goal to 
alter the parsing context -- such as narrowing before calling 
forward-sexp, in blink-matching-paren's example). These could be 
documented and assigned relative depth, sorting them like "hard > 
movement > visual", where code willing to undo "movement" would 
naturally undo "visual" as well, and code will to undo "hard" narrowing 
would undo them all.

How treesit-forward-sexp would handle "movement" narrowing is something 
up for discussion, though: maybe just ignore it (given that there is no 
performance tax), or maybe it would still perform a position check at 
the end (after finding the matching paren), to see if it ends up beyond 
the accessible region.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 27/03/2023 16:29, Eli Zaretskii wrote:
>> Date: Mon, 27 Mar 2023 01:00:20 +0300
>> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> So you are suggesting to introduce kludges like font-lock-dont-widen
>>> in all of those places?
>>
>> font-lock-dont-widen is a kludge, but that's largely determined by the
>> way it's defined and used.
> 
> With parser-based features, we have an opportunity to do this in a
> cleaner manner.

parser-based features don't need this at all, if considered in 
isolation. But if we try to combine them with existing mode, or existing 
packages, they need to play by the common rules. Which they currently do.

>> If we take indent-for-tab-command, for example, it doesn't have such a
>> variable, and doesn't really need to: the top-level command calls
>> 'widen', and then indent-line-function (set by major mode and altered by
>> e.g. mmm-mode) is free to impose its specific bounds.
> 
> I thought about a lower-level, infrastructure-level, mechanism that
> could be used to restrict a parser to a certain region of the buffer.
> Then this could be used by every feature based on parsers, instead of
> us having to invent a separate solution for each one.

Like narrowing, but just for parsers? But parsers obey narrowing 
already. Sounds a bit like conceptual duplication. How does this solve 
blink-matching-paren issue anyway?

>> The "grand unified theory of mixed major modes" has been attempted a few
>> times in the past, and never reached anything practical.
> 
> But here we have a unique opportunity to maybe find a solution, at
> least for stuff based on tree-sitter and similar libraries.  That
> maybe not "grand", but certainly "respectable".

tree-sitter has its own support for mixed languages.

>>>> Except it's already limited by narrowing.
>>>
>>> Which is a fragile, semi-broken means, as we all know.
>>
>> What is a broken mess, is user-level narrowing.
> 
> Which is why restricting parsers should not be a user-level feature.

We're talking elisp-level, though. The problem with interactive 
narrowing, is its impossible to tell apart from one effected by Lisp. 
And they usually have different goals.

>>> Look, I'm trying to see how we could come up with an infrastructure
>>> that will support multiple modes and other similar features in the
>>> same buffer without relying on narrowing, thus bypassing the
>>> disadvantages and difficulties that come with narrowing.  I think we
>>> have a good chance here to come up with such a solution, specifically
>>> for features that us a parsing library.  If you aren't interested in
>>> discussing that, and think we should stick to narrowing, then this
>>> goes nowhere, and I'd rather bow out of it.
>>
>> What I've seen here so far is you suggesting we go ahead and break the
>> existing convention and then let "them" (third-party authors including
>> myself) come up with a new working one.
> 
> You are mixing two semi-separate issues: how to fix the immediate
> problem with blink-matching-paren (if we consider it a serious
> problem), and how to try solving similar issues as a whole.

Not mixing anything. We have one proposal for improving 
blink-matching-paren integration anyway. It should be compatible with 
whatever we choose regarding narrowing and mixed modes.

> The
> latter is a long-term goal, and we can discuss it freely and calmly;
> whereby ideas can be brought up even if they might not be the most
> clever ones or the ones with which you personally agree.

I'm not looking for "clever".

>> My stance here is we shouldn't break it before we create a new one.
> 
> No one broke anything.  We are just discussing ideas.  Please don't
> exaggerate.

I never said anybody has broken anything already.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Mon, 27 Mar 2023 20:05:07 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: dgutov@yandex.ru, wkirschbaum@gmail.com, casouri@gmail.com, 
>     62333@debbugs.gnu.org
> 
> > Labeled narrowing cannot solve this because it does nothing to alleviate 
> > the problems with user-defined narrowing.  So if the user narrows the 
> > buffer, we cannot do anything to safely widen in the general case, and 
> > labeled narrowing cannot help us.
> 
> I must be missing something, but I don't understand what you mean.  If a 
> user has narrowed a buffer to a certain region A-B, and if we want to 
> limit a parser to a certain region C-D, we can do that safely with
> 
> (with-narrowing C D :label 'parser-library BODY)

(You mean with-restriction.)

> which will work, independently of the A-B narrowing set by the user, and 
> will reset the narrowing to A-B upon return.

It will only work if [C..D] is inside [A..B].  It will not work if a
Lisp program wants to widen [A..B], i.e. if [C..D] is larger than
[A..B].

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Tue, 28 Mar 2023 02:06:17 +0300
> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 27/03/2023 16:39, Eli Zaretskii wrote:
> >> Date: Mon, 27 Mar 2023 08:24:42 +0000
> >> From: Gregory Heytings<gregory@heytings.org>
> >> cc: Eli Zaretskii<eliz@gnu.org>,wkirschbaum@gmail.com,casouri@gmail.com,
> >>      62333@debbugs.gnu.org
> >>
> >>>> Which is a fragile, semi-broken means, as we all know.
> >>> What is a broken mess, is user-level narrowing. And how the downstream
> >>> code can never guess the purpose the narrowing was applied for.
> >> Note that this is what labeled narrowings attempts to solve.
> > Labeled narrowing cannot solve this because it does nothing to
> > alleviate the problems with user-defined narrowing.  So if the user
> > narrows the buffer, we cannot do anything to safely widen in the
> > general case, and labeled narrowing cannot help us.
> 
> Is that because we don't think the user level narrowing is done purely 
> for visual effect?

Indeed, it isn't always for visual effect.

> judging by regular user requests for make this or that command 
> ignore user-level narrowing, it seems like "purely visual" should be the 
> default interpretation.

I think you base your judgment on feedback from users who are not used
to take advantage of narrowing in editing.  I think most young people
aren't, since this feature is more-or-less unique to Emacs.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Tue, 28 Mar 2023 02:33:38 +0300
> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > With parser-based features, we have an opportunity to do this in a
> > cleaner manner.
> 
> parser-based features don't need this at all, if considered in 
> isolation. But if we try to combine them with existing mode, or existing 
> packages, they need to play by the common rules. Which they currently do.

Sorry, I don't see the relevance of that to what I wrote above.

> >> If we take indent-for-tab-command, for example, it doesn't have such a
> >> variable, and doesn't really need to: the top-level command calls
> >> 'widen', and then indent-line-function (set by major mode and altered by
> >> e.g. mmm-mode) is free to impose its specific bounds.
> > 
> > I thought about a lower-level, infrastructure-level, mechanism that
> > could be used to restrict a parser to a certain region of the buffer.
> > Then this could be used by every feature based on parsers, instead of
> > us having to invent a separate solution for each one.
> 
> Like narrowing, but just for parsers? But parsers obey narrowing 
> already. Sounds a bit like conceptual duplication. How does this solve 
> blink-matching-paren issue anyway?

We could widen without fearing that a parser will "invade" regions of
buffer text that we don't want it to wander into.

> >> The "grand unified theory of mixed major modes" has been attempted a few
> >> times in the past, and never reached anything practical.
> > 
> > But here we have a unique opportunity to maybe find a solution, at
> > least for stuff based on tree-sitter and similar libraries.  That
> > maybe not "grand", but certainly "respectable".
> 
> tree-sitter has its own support for mixed languages.

So your argument about mmm framework was a red herring, cause that
problem doesn't exist wrt tree-sitter parsers?

> >>>> Except it's already limited by narrowing.
> >>>
> >>> Which is a fragile, semi-broken means, as we all know.
> >>
> >> What is a broken mess, is user-level narrowing.
> > 
> > Which is why restricting parsers should not be a user-level feature.
> 
> We're talking elisp-level, though. The problem with interactive 
> narrowing, is its impossible to tell apart from one effected by Lisp. 
> And they usually have different goals.

Yes.  So if this new feature will not be exposed to users, those
dangers will be avoided.

> >> What I've seen here so far is you suggesting we go ahead and break the
> >> existing convention and then let "them" (third-party authors including
> >> myself) come up with a new working one.
> > 
> > You are mixing two semi-separate issues: how to fix the immediate
> > problem with blink-matching-paren (if we consider it a serious
> > problem), and how to try solving similar issues as a whole.
> 
> Not mixing anything. We have one proposal for improving 
> blink-matching-paren integration anyway. It should be compatible with 
> whatever we choose regarding narrowing and mixed modes.
> 
> > The
> > latter is a long-term goal, and we can discuss it freely and calmly;
> > whereby ideas can be brought up even if they might not be the most
> > clever ones or the ones with which you personally agree.
> 
> I'm not looking for "clever".
> 
> >> My stance here is we shouldn't break it before we create a new one.
> > 
> > No one broke anything.  We are just discussing ideas.  Please don't
> > exaggerate.
> 
> I never said anybody has broken anything already.

You did say that my ideas break something, see above.  Ideas cannot
break any code, so this argument shouldn't be brought up if you want a
calm and rational discussion.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Gregory Heytings]

>> which will work, independently of the A-B narrowing set by the user, 
>> and will reset the narrowing to A-B upon return.
>
> It will only work if [C..D] is inside [A..B].  It will not work if a 
> Lisp program wants to widen [A..B], i.e. if [C..D] is larger than 
> [A..B].
>

Hmmm... no, it will work for any values of [C..D]: (widen) is equivalent 
to (narrow-to-region 1 (1+ (buffer-size))), and in general the bounds 
given to 'narrow-to-region' are not limited by the current narrowing.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Tue, 28 Mar 2023 11:39:13 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: dgutov@yandex.ru, wkirschbaum@gmail.com, casouri@gmail.com, 
>     62333@debbugs.gnu.org
> 
> 
> >> which will work, independently of the A-B narrowing set by the user, 
> >> and will reset the narrowing to A-B upon return.
> >
> > It will only work if [C..D] is inside [A..B].  It will not work if a 
> > Lisp program wants to widen [A..B], i.e. if [C..D] is larger than 
> > [A..B].
> >
> 
> Hmmm... no, it will work for any values of [C..D]: (widen) is equivalent 
> to (narrow-to-region 1 (1+ (buffer-size))), and in general the bounds 
> given to 'narrow-to-region' are not limited by the current narrowing.

with-restriction signals an error if you call it wit a region that is
outside of the current restriction.  At least it does here.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Gregory Heytings]

>> Hmmm... no, it will work for any values of [C..D]: (widen) is 
>> equivalent to (narrow-to-region 1 (1+ (buffer-size))), and in general 
>> the bounds given to 'narrow-to-region' are not limited by the current 
>> narrowing.
>
> with-restriction signals an error if you call it wit a region that is 
> outside of the current restriction.  At least it does here.
>

Do you have a recipe?  It should not signal an error, and I can't 
reproduce that.  With this for instance:

M-: (global-set-key (kbd "C-t") (lambda () (interactive) (with-restriction 20 30 (message "{%s}" (buffer-string))))) RET

C-t (in *scratch*) displays {or text th}, even if the text is narrowed say 
to the second line.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Tue, 28 Mar 2023 12:25:03 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org, 
>     dgutov@yandex.ru
> 
> 
> >> Hmmm... no, it will work for any values of [C..D]: (widen) is 
> >> equivalent to (narrow-to-region 1 (1+ (buffer-size))), and in general 
> >> the bounds given to 'narrow-to-region' are not limited by the current 
> >> narrowing.
> >
> > with-restriction signals an error if you call it wit a region that is 
> > outside of the current restriction.  At least it does here.
> >
> 
> Do you have a recipe?  It should not signal an error, and I can't 
> reproduce that.  With this for instance:
> 
> M-: (global-set-key (kbd "C-t") (lambda () (interactive) (with-restriction 20 30 (message "{%s}" (buffer-string))))) RET

Sorry, my bad.  Bad testing.

But regardless, lifting user-defined restriction is a no-no, that's
what the problem is all about.  with-restriction cannot do anything
against this.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 28/03/2023 14:32, Eli Zaretskii wrote:
>> Date: Tue, 28 Mar 2023 02:06:17 +0300
>> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 27/03/2023 16:39, Eli Zaretskii wrote:
>>>> Date: Mon, 27 Mar 2023 08:24:42 +0000
>>>> From: Gregory Heytings<gregory@heytings.org>
>>>> cc: Eli Zaretskii<eliz@gnu.org>,wkirschbaum@gmail.com,casouri@gmail.com,
>>>>       62333@debbugs.gnu.org
>>>>
>>>>>> Which is a fragile, semi-broken means, as we all know.
>>>>> What is a broken mess, is user-level narrowing. And how the downstream
>>>>> code can never guess the purpose the narrowing was applied for.
>>>> Note that this is what labeled narrowings attempts to solve.
>>> Labeled narrowing cannot solve this because it does nothing to
>>> alleviate the problems with user-defined narrowing.  So if the user
>>> narrows the buffer, we cannot do anything to safely widen in the
>>> general case, and labeled narrowing cannot help us.
>>
>> Is that because we don't think the user level narrowing is done purely
>> for visual effect?
> 
> Indeed, it isn't always for visual effect.

When isn't it? Is there a way to determine that from code?

>> judging by regular user requests for make this or that command
>> ignore user-level narrowing, it seems like "purely visual" should be the
>> default interpretation.
> 
> I think you base your judgment on feedback from users who are not used
> to take advantage of narrowing in editing.  I think most young people
> aren't, since this feature is more-or-less unique to Emacs.

Either narrowing should be used to change lexical/grammatical/etc 
context, or it should not. Do we have any documentation that says one or 
the other way? That should affect how Lisp code deals with narrowing -- 
which interactive functions should widen, and so on.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 28/03/2023 14:38, Eli Zaretskii wrote:
>> Date: Tue, 28 Mar 2023 02:33:38 +0300
>> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> With parser-based features, we have an opportunity to do this in a
>>> cleaner manner.
>>
>> parser-based features don't need this at all, if considered in
>> isolation. But if we try to combine them with existing mode, or existing
>> packages, they need to play by the common rules. Which they currently do.
> 
> Sorry, I don't see the relevance of that to what I wrote above.
> 
>>>> If we take indent-for-tab-command, for example, it doesn't have such a
>>>> variable, and doesn't really need to: the top-level command calls
>>>> 'widen', and then indent-line-function (set by major mode and altered by
>>>> e.g. mmm-mode) is free to impose its specific bounds.
>>>
>>> I thought about a lower-level, infrastructure-level, mechanism that
>>> could be used to restrict a parser to a certain region of the buffer.
>>> Then this could be used by every feature based on parsers, instead of
>>> us having to invent a separate solution for each one.
>>
>> Like narrowing, but just for parsers? But parsers obey narrowing
>> already. Sounds a bit like conceptual duplication. How does this solve
>> blink-matching-paren issue anyway?
> 
> We could widen without fearing that a parser will "invade" regions of
> buffer text that we don't want it to wander into.

So any code that wants to restrict a "parser" based buffer, would need 
to use a different primitive to narrow? And vice versa, any code that 
uses a parser, will need to (widen) first, to ensure that the parser is 
not affected by any restriction set up by any code previously?

Anyway, I don't see why we should institute a special category for these 
buffers. Most language modes which define syntax-propertize-function are 
basically "parser-based", in most of the relevant respects. Except the 
accuracy is less, and we write the parsing code ourselves.

>>>> The "grand unified theory of mixed major modes" has been attempted a few
>>>> times in the past, and never reached anything practical.
>>>
>>> But here we have a unique opportunity to maybe find a solution, at
>>> least for stuff based on tree-sitter and similar libraries.  That
>>> maybe not "grand", but certainly "respectable".
>>
>> tree-sitter has its own support for mixed languages.
> 
> So your argument about mmm framework was a red herring, cause that
> problem doesn't exist wrt tree-sitter parsers?

Nope, see the first paragraph of my previous reply (the "no relevance" one).

>>>> My stance here is we shouldn't break it before we create a new one.
>>>
>>> No one broke anything.  We are just discussing ideas.  Please don't
>>> exaggerate.
>>
>> I never said anybody has broken anything already.
> 
> You did say that my ideas break something, see above.  Ideas cannot
> break any code, so this argument shouldn't be brought up if you want a
> calm and rational discussion.

Ideas cannot, but implementing them can. "This or that change will break 
an existing convention" is a rational argument.

Shall we stop quibbling over words?

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Wed, 29 Mar 2023 00:08:53 +0300
> Cc: wkirschbaum@gmail.com, gregory@heytings.org, casouri@gmail.com,
>  62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> Is that because we don't think the user level narrowing is done purely
> >> for visual effect?
> > 
> > Indeed, it isn't always for visual effect.
> 
> When isn't it? Is there a way to determine that from code?

I'm not sure I understand the question, but if I do, then narrowing to
prevent search functions and commands from finding irrelevant hits is
one example that comes to mind.

> >> judging by regular user requests for make this or that command
> >> ignore user-level narrowing, it seems like "purely visual" should be the
> >> default interpretation.
> > 
> > I think you base your judgment on feedback from users who are not used
> > to take advantage of narrowing in editing.  I think most young people
> > aren't, since this feature is more-or-less unique to Emacs.
> 
> Either narrowing should be used to change lexical/grammatical/etc 
> context, or it should not. Do we have any documentation that says one or 
> the other way? That should affect how Lisp code deals with narrowing -- 
> which interactive functions should widen, and so on.

I was talking about user commands that narrow, so I'm not sure I
understand how documentation could help.  When the user types "C-x n n",
there's nothing Emacs can do except obey.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Wed, 29 Mar 2023 00:19:41 +0300
> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> Like narrowing, but just for parsers? But parsers obey narrowing
> >> already. Sounds a bit like conceptual duplication. How does this solve
> >> blink-matching-paren issue anyway?
> > 
> > We could widen without fearing that a parser will "invade" regions of
> > buffer text that we don't want it to wander into.
> 
> So any code that wants to restrict a "parser" based buffer, would need 
> to use a different primitive to narrow?

No, the idea is to create the parser with these restrictions to begin
with.

> And vice versa, any code that uses a parser, will need to (widen)
> first, to ensure that the parser is not affected by any restriction
> set up by any code previously?

No, if the parser is restricted, there should be no need to "widen"
it, at least not in most cases I could think about where a parser
should be restricted to a part of the buffer.

> Anyway, I don't see why we should institute a special category for these 
> buffers.

Because IMO it's cleaner and simpler than using narrowing, and doesn't
suffer from the problems we see in narrowing.

> >>>> The "grand unified theory of mixed major modes" has been attempted a few
> >>>> times in the past, and never reached anything practical.
> >>>
> >>> But here we have a unique opportunity to maybe find a solution, at
> >>> least for stuff based on tree-sitter and similar libraries.  That
> >>> maybe not "grand", but certainly "respectable".
> >>
> >> tree-sitter has its own support for mixed languages.
> > 
> > So your argument about mmm framework was a red herring, cause that
> > problem doesn't exist wrt tree-sitter parsers?
> 
> Nope, see the first paragraph of my previous reply (the "no relevance" one).

Which first paragraph is that?  In what "previous reply" should I look
and where?  (And why couldn't you simply copy that here, to make
following the discussion easier?)

> >>>> My stance here is we shouldn't break it before we create a new one.
> >>>
> >>> No one broke anything.  We are just discussing ideas.  Please don't
> >>> exaggerate.
> >>
> >> I never said anybody has broken anything already.
> > 
> > You did say that my ideas break something, see above.  Ideas cannot
> > break any code, so this argument shouldn't be brought up if you want a
> > calm and rational discussion.
> 
> Ideas cannot, but implementing them can. "This or that change will break 
> an existing convention" is a rational argument.
> 
> Shall we stop quibbling over words?

I'm "quibbling" over words because your particular selection of words
makes serious discussion nigh impossible.  Not the first time, either.
As soon as there's some disagreement, sooner or later those words
(like "quibbling") come out.  Whether it's because of some attitude or
not, I don't know, but you may wish to reflect on that and perhaps try
to express your disagreements using different words.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Gregory Heytings]

>
> No, the idea is to create the parser with these restrictions to begin 
> with.
>

Could you perhaps explain, with some fictitious code, what kind of 
solution you imagine?  I'm not sure I understand (euphemism for: I'm sure 
I don't understand) the problem statement.

In what way would the restrictions you have in mind be different from 
those of a regular narrowing?

Also, would such a parser always/never/sometimes obey the user narrowing?

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Thu, 30 Mar 2023 15:50:31 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Dmitry Gutov <dgutov@yandex.ru>, wkirschbaum@gmail.com, casouri@gmail.com, 
>     62333@debbugs.gnu.org
> 
> > No, the idea is to create the parser with these restrictions to begin 
> > with.
> 
> Could you perhaps explain, with some fictitious code, what kind of 
> solution you imagine?  I'm not sure I understand (euphemism for: I'm sure 
> I don't understand) the problem statement.

Something like

  (treesit-make-parser LANGUAGE BUFFER nil START END)

and

  (treesit-parser-set-included-ranges RANGES...)

(the latter already exists).

> In what way would the restrictions you have in mind be different from 
> those of a regular narrowing?

User-defined narrowing will never contradict parser restrictions.

> Also, would such a parser always/never/sometimes obey the user narrowing?

It will always obey narrowing (it must), but we can then widen the
buffer temporarily inside some functions without caring about the
semantics of the narrowing and its source/purpose.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Gregory Heytings]

>> Could you perhaps explain, with some fictitious code, what kind of 
>> solution you imagine?  I'm not sure I understand (euphemism for: I'm 
>> sure I don't understand) the problem statement.
>
> Something like
>
>  (treesit-make-parser LANGUAGE BUFFER nil START END)
>
> and
>
>  (treesit-parser-set-included-ranges RANGES...)
>
> (the latter already exists).
>

Where "RANGES..." are included in [START..END], right?

>> In what way would the restrictions you have in mind be different from 
>> those of a regular narrowing?
>
> User-defined narrowing will never contradict parser restrictions.
>

You mean, they will be independent, right?  In other words, if the user 
sets the narrowing to 1000-1200 in a buffer in which treesit-make-parser 
has been called, say, once with 'php 400 1100' and once with 'js 1100 
1500', the two parsers will continue to have access to these ranges?

>> Also, would such a parser always/never/sometimes obey the user 
>> narrowing?
>
> It will always obey narrowing (it must), but we can then widen the 
> buffer temporarily inside some functions without caring about the 
> semantics of the narrowing and its source/purpose.
>

Here I'm confused, that sentence seems to contradict the previous one. 
"It" in "it will always obey narrowing" is the parser, right, and 
"narrowing" is "the narrowing bounds set by 'treesit-make-parser'", right?

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Thu, 30 Mar 2023 16:28:58 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org, 
>     dgutov@yandex.ru
> 
> > Something like
> >
> >  (treesit-make-parser LANGUAGE BUFFER nil START END)
> >
> > and
> >
> >  (treesit-parser-set-included-ranges RANGES...)
> >
> > (the latter already exists).
> >
> 
> Where "RANGES..." are included in [START..END], right?

Probably.  Or maybe instead of START and END we should have RANGES...

> > User-defined narrowing will never contradict parser restrictions.
> 
> You mean, they will be independent, right?  In other words, if the user 
> sets the narrowing to 1000-1200 in a buffer in which treesit-make-parser 
> has been called, say, once with 'php 400 1100' and once with 'js 1100 
> 1500', the two parsers will continue to have access to these ranges?

The parsers _can_ have access to those ranges, if they need it for
some reason.  In general, everything in Emacs should honor the current
restriction, unless there's a good reason to ignore it.  The problem
with ignoring it is that we can never know which code/user defined the
restriction and for what purpose.  I hope that keeping the parser's
restrictions as part of the parser itself will allow us to break free
of that issue when we have to widen.

> >> Also, would such a parser always/never/sometimes obey the user 
> >> narrowing?
> >
> > It will always obey narrowing (it must), but we can then widen the 
> > buffer temporarily inside some functions without caring about the 
> > semantics of the narrowing and its source/purpose.
> >
> 
> Here I'm confused, that sentence seems to contradict the previous one. 
> "It" in "it will always obey narrowing" is the parser, right, and 
> "narrowing" is "the narrowing bounds set by 'treesit-make-parser'", right?

See above.  I hope that explains what I meant.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Gregory Heytings]

>>> User-defined narrowing will never contradict parser restrictions.
>>
>> You mean, they will be independent, right?  In other words, if the user 
>> sets the narrowing to 1000-1200 in a buffer in which 
>> treesit-make-parser has been called, say, once with 'php 400 1100' and 
>> once with 'js 1100 1500', the two parsers will continue to have access 
>> to these ranges?
>
> The parsers _can_ have access to those ranges, if they need it for some 
> reason.  In general, everything in Emacs should honor the current 
> restriction, unless there's a good reason to ignore it.
>

Okay, so in the above example by default the parsers will only have access 
to 1000-1100 for the first one, and 1100-1200 for the second one until the 
user removes the restrictions.  Unless they need to widen the buffer for 
some (good) reason.

If they do widen, will the parsers get access to [400..1100] and 
[1100..1500], or to the whole buffer?  Or will they have two ways to 
widen, one to get access to the whole region on which they have been 
defined, and another one to get access to the whole buffer?

>
> The problem with ignoring it is that we can never know which code/user 
> defined the restriction and for what purpose.  I hope that keeping the 
> parser's restrictions as part of the parser itself will allow us to 
> break free of that issue when we have to widen.
>

At least it's a possibility that seems worth investigating.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Thu, 30 Mar 2023 17:27:02 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org, 
>     dgutov@yandex.ru
> 
> > The parsers _can_ have access to those ranges, if they need it for some 
> > reason.  In general, everything in Emacs should honor the current 
> > restriction, unless there's a good reason to ignore it.
> 
> Okay, so in the above example by default the parsers will only have access 
> to 1000-1100 for the first one, and 1100-1200 for the second one until the 
> user removes the restrictions.  Unless they need to widen the buffer for 
> some (good) reason.
> 
> If they do widen, will the parsers get access to [400..1100] and 
> [1100..1500], or to the whole buffer?

The former.

> > The problem with ignoring it is that we can never know which code/user 
> > defined the restriction and for what purpose.  I hope that keeping the 
> > parser's restrictions as part of the parser itself will allow us to 
> > break free of that issue when we have to widen.
> 
> At least it's a possibility that seems worth investigating.

I think so, yes.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Gregory Heytings]

>>> The parsers _can_ have access to those ranges, if they need it for 
>>> some reason.  In general, everything in Emacs should honor the current 
>>> restriction, unless there's a good reason to ignore it.
>>
>> Okay, so in the above example by default the parsers will only have 
>> access to 1000-1100 for the first one, and 1100-1200 for the second one 
>> until the user removes the restrictions.  Unless they need to widen the 
>> buffer for some (good) reason.
>>
>> If they do widen, will the parsers get access to [400..1100] and 
>> [1100..1500], or to the whole buffer?
>
> The former.
>

Okay, thanks.  Now I think I understand what you want.  I'll think about 
it.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Gregory Heytings]

>>>> The parsers _can_ have access to those ranges, if they need it for 
>>>> some reason.  In general, everything in Emacs should honor the 
>>>> current restriction, unless there's a good reason to ignore it.
>>> 
>>> Okay, so in the above example by default the parsers will only have 
>>> access to 1000-1100 for the first one, and 1100-1200 for the second 
>>> one until the user removes the restrictions.  Unless they need to 
>>> widen the buffer for some (good) reason.
>>> 
>>> If they do widen, will the parsers get access to [400..1100] and 
>>> [1100..1500], or to the whole buffer?
>> 
>> The former.
>
> Okay, thanks.  Now I think I understand what you want.  I'll think about 
> it.
>

Two other questions:

Should there exist a mechanism to give the parsers access to the whole 
buffer (IOW, to let them ignore both the bounds passed to 
'treesit-make-parser' and the restriction set by the user)?

And of course, the bounds given to 'treesit-make-parser' should become 
markers, right, that is, their position should change whenever the buffer 
contents change, right?

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 29/03/2023 14:08, Eli Zaretskii wrote:
>> Date: Wed, 29 Mar 2023 00:08:53 +0300
>> Cc: wkirschbaum@gmail.com, gregory@heytings.org, casouri@gmail.com,
>>   62333@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>>> Is that because we don't think the user level narrowing is done purely
>>>> for visual effect?
>>>
>>> Indeed, it isn't always for visual effect.
>>
>> When isn't it? Is there a way to determine that from code?
> 
> I'm not sure I understand the question, but if I do, then narrowing to
> prevent search functions

If we're talking about isearch, then that seems like a natural 
consequence of visual effect (hiding the remainder of the buffer): even 
if isearch highlighted those other hits, they would not be visible.

> and commands from finding irrelevant hits is
> one example that comes to mind.

More or less the same, except we have the user option 
widen-automatically, which apparently (?) allows any command to "widen 
when they want to"?

Used by popular demand in e.g. xref--goto-char. And IIUC 'find-tag' just 
always calls 'widen' irrespective of this variable's value.

>>>> judging by regular user requests for make this or that command
>>>> ignore user-level narrowing, it seems like "purely visual" should be the
>>>> default interpretation.
>>>
>>> I think you base your judgment on feedback from users who are not used
>>> to take advantage of narrowing in editing.  I think most young people
>>> aren't, since this feature is more-or-less unique to Emacs.
>>
>> Either narrowing should be used to change lexical/grammatical/etc
>> context, or it should not. Do we have any documentation that says one or
>> the other way? That should affect how Lisp code deals with narrowing --
>> which interactive functions should widen, and so on.
> 
> I was talking about user commands that narrow, so I'm not sure I
> understand how documentation could help.  When the user types "C-x n n",
> there's nothing Emacs can do except obey.

There is really only one main user command that narrows, and that's 
narrow-to-region, bound to 'C-x n n'.

But none of that describes how other Emacs features should react to it.

Simple example: if the beginning of the narrowed region falls inside a 
(let's say multine) string, should the visible remainder of that string 
continue to be highlighted as a string? Or should the buffer contents 
after the string's closer now be highlighted as being inside a string?

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 29/03/2023 14:17, Eli Zaretskii wrote:
>> Date: Wed, 29 Mar 2023 00:19:41 +0300
>> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>>> Like narrowing, but just for parsers? But parsers obey narrowing
>>>> already. Sounds a bit like conceptual duplication. How does this solve
>>>> blink-matching-paren issue anyway?
>>>
>>> We could widen without fearing that a parser will "invade" regions of
>>> buffer text that we don't want it to wander into.
>>
>> So any code that wants to restrict a "parser" based buffer, would need
>> to use a different primitive to narrow?
> 
> No, the idea is to create the parser with these restrictions to begin
> with.
> 
>> And vice versa, any code that uses a parser, will need to (widen)
>> first, to ensure that the parser is not affected by any restriction
>> set up by any code previously?
> 
> No, if the parser is restricted, there should be no need to "widen"
> it, at least not in most cases I could think about where a parser
> should be restricted to a part of the buffer.
> 
>> Anyway, I don't see why we should institute a special category for these
>> buffers.
> 
> Because IMO it's cleaner and simpler than using narrowing, and doesn't
> suffer from the problems we see in narrowing.

You are not describing anything that one couldn't do with a major mode 
before tree-sitter parsers (choose a region, apply font-lock keywords, 
etc), so that should give you an idea that the suggestion is probably 
not novel. And doesn't solve the existing problems in new ways.

Why would we apply that approach to "proper" parsers only?

>>>>>> The "grand unified theory of mixed major modes" has been attempted a few
>>>>>> times in the past, and never reached anything practical.
>>>>>
>>>>> But here we have a unique opportunity to maybe find a solution, at
>>>>> least for stuff based on tree-sitter and similar libraries.  That
>>>>> maybe not "grand", but certainly "respectable".
>>>>
>>>> tree-sitter has its own support for mixed languages.
>>>
>>> So your argument about mmm framework was a red herring, cause that
>>> problem doesn't exist wrt tree-sitter parsers?
>>
>> Nope, see the first paragraph of my previous reply (the "no relevance" one).
> 
> Which first paragraph is that?  In what "previous reply" should I look
> and where?  (And why couldn't you simply copy that here, to make
> following the discussion easier?)

The reply you answered with "I don't see the relevance". There is really 
no good way to continue the explanation after that. Copying it into 
multiple successive emails doesn't sound logical either.

>>>>>> My stance here is we shouldn't break it before we create a new one.
>>>>>
>>>>> No one broke anything.  We are just discussing ideas.  Please don't
>>>>> exaggerate.
>>>>
>>>> I never said anybody has broken anything already.
>>>
>>> You did say that my ideas break something, see above.  Ideas cannot
>>> break any code, so this argument shouldn't be brought up if you want a
>>> calm and rational discussion.
>>
>> Ideas cannot, but implementing them can. "This or that change will break
>> an existing convention" is a rational argument.
>>
>> Shall we stop quibbling over words?
> 
> I'm "quibbling" over words because your particular selection of words
> makes serious discussion nigh impossible.  Not the first time, either.
> As soon as there's some disagreement, sooner or later those words
> (like "quibbling") come out.  Whether it's because of some attitude or
> not, I don't know, but you may wish to reflect on that and perhaps try
> to express your disagreements using different words.

"Breaking a convention" is an industry term.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 30/03/2023 19:04, Eli Zaretskii wrote:
>> Date: Thu, 30 Mar 2023 15:50:31 +0000
>> From: Gregory Heytings <gregory@heytings.org>
>> cc: Dmitry Gutov <dgutov@yandex.ru>, wkirschbaum@gmail.com, casouri@gmail.com,
>>      62333@debbugs.gnu.org
>>
>>> No, the idea is to create the parser with these restrictions to begin
>>> with.
>>
>> Could you perhaps explain, with some fictitious code, what kind of
>> solution you imagine?  I'm not sure I understand (euphemism for: I'm sure
>> I don't understand) the problem statement.
> 
> Something like
> 
>    (treesit-make-parser LANGUAGE BUFFER nil START END)

What kind of code would be calling this?

What happens when the user types a character before START? What happens 
when they delete a character at START or END? Or a whole line?

> and
> 
>    (treesit-parser-set-included-ranges RANGES...)
> 
> (the latter already exists).

Indeed, a way to do this using tree-sitter indeed already exists, 
offloading the parsing of the regions to tree-sitter. OTOH, this way we 
only get tree-sitter features sorted into regions (parse tree, 
indentation, highlighting).

Whatever other Lisp-level features we wanted to behave differently 
between regions, we'd have do implement them the old way.

>> In what way would the restrictions you have in mind be different from
>> those of a regular narrowing?
> 
> User-defined narrowing will never contradict parser restrictions.

IOW, user-defined narrowing should be only for visual purposes, as 
described in another email. And perhaps to restrict movement (which is 
related: you can't move to where you can't see).

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

Sorry

On 31/03/2023 04:10, Dmitry Gutov wrote:
> Simple example: if the beginning of the narrowed region falls inside a 
> (let's say multine)
              ^
               multiline

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Thu, 30 Mar 2023 22:08:10 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org, 
>     dgutov@yandex.ru
> 
> Should there exist a mechanism to give the parsers access to the whole 
> buffer (IOW, to let them ignore both the bounds passed to 
> 'treesit-make-parser' and the restriction set by the user)?

We already have that, I think: treesit-parser-set-included-ranges.
Although I wonder when and why would this be a good idea, given that a
parser should have full access to the entire text that is written in
the corresponding language.

> And of course, the bounds given to 'treesit-make-parser' should become 
> markers

Of course.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Fri, 31 Mar 2023 04:10:54 +0300
> Cc: wkirschbaum@gmail.com, gregory@heytings.org, casouri@gmail.com,
>  62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 29/03/2023 14:08, Eli Zaretskii wrote:
> >> Date: Wed, 29 Mar 2023 00:08:53 +0300
> >> Cc: wkirschbaum@gmail.com, gregory@heytings.org, casouri@gmail.com,
> >>   62333@debbugs.gnu.org
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >>
> >>>> Is that because we don't think the user level narrowing is done purely
> >>>> for visual effect?
> >>>
> >>> Indeed, it isn't always for visual effect.
> >>
> >> When isn't it? Is there a way to determine that from code?
> > 
> > I'm not sure I understand the question, but if I do, then narrowing to
> > prevent search functions
> 
> If we're talking about isearch, then that seems like a natural 
> consequence of visual effect (hiding the remainder of the buffer): even 
> if isearch highlighted those other hits, they would not be visible.

If you consider narrowing in this example to be "for visual effect",
then everything in Emacs is "for visual effect".  After all, Emacs is
a visual editor, showing the results of editing to the user at all
times.  But this POV makes this part of the discussion useless.

> > I was talking about user commands that narrow, so I'm not sure I
> > understand how documentation could help.  When the user types "C-x n n",
> > there's nothing Emacs can do except obey.
> 
> There is really only one main user command that narrows, and that's 
> narrow-to-region, bound to 'C-x n n'.

Any user command can narrow as part of its job.

> Simple example: if the beginning of the narrowed region falls inside a 
> (let's say multine) string, should the visible remainder of that string 
> continue to be highlighted as a string?

No.

> Or should the buffer contents after the string's closer now be
> highlighted as being inside a string?

Yes.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Fri, 31 Mar 2023 04:25:41 +0300
> Cc: wkirschbaum@gmail.com, casouri@gmail.com, 62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 30/03/2023 19:04, Eli Zaretskii wrote:
> >> Date: Thu, 30 Mar 2023 15:50:31 +0000
> >> From: Gregory Heytings <gregory@heytings.org>
> >> cc: Dmitry Gutov <dgutov@yandex.ru>, wkirschbaum@gmail.com, casouri@gmail.com,
> >>      62333@debbugs.gnu.org
> >>
> >>> No, the idea is to create the parser with these restrictions to begin
> >>> with.
> >>
> >> Could you perhaps explain, with some fictitious code, what kind of
> >> solution you imagine?  I'm not sure I understand (euphemism for: I'm sure
> >> I don't understand) the problem statement.
> > 
> > Something like
> > 
> >    (treesit-make-parser LANGUAGE BUFFER nil START END)
> 
> What kind of code would be calling this?
> 
> What happens when the user types a character before START? What happens 
> when they delete a character at START or END? Or a whole line?
> 
> > and
> > 
> >    (treesit-parser-set-included-ranges RANGES...)
> > 
> > (the latter already exists).
> 
> Indeed, a way to do this using tree-sitter indeed already exists, 
> offloading the parsing of the regions to tree-sitter. OTOH, this way we 
> only get tree-sitter features sorted into regions (parse tree, 
> indentation, highlighting).
> 
> Whatever other Lisp-level features we wanted to behave differently 
> between regions, we'd have do implement them the old way.
> 
> >> In what way would the restrictions you have in mind be different from
> >> those of a regular narrowing?
> > 
> > User-defined narrowing will never contradict parser restrictions.
> 
> IOW, user-defined narrowing should be only for visual purposes, as 
> described in another email. And perhaps to restrict movement (which is 
> related: you can't move to where you can't see).

I'm sorry, I cannot afford a discussion where you decided up front you
disagree, and just keep looking for arguments in favor of your
disagreement.  Such discussions are a waste of our time, and I have
very little of it to waste.  So I won't be responding to any further
messages here, unless they ask informative questions where I think I
can contribute something useful.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Cc: wkirschbaum@gmail.com, gregory@heytings.org, casouri@gmail.com,
>  62333@debbugs.gnu.org
> Date: Fri, 31 Mar 2023 09:19:35 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Simple example: if the beginning of the narrowed region falls inside a 
> > (let's say multine) string, should the visible remainder of that string 
> > continue to be highlighted as a string?
> 
> No.
> 
> > Or should the buffer contents after the string's closer now be
> > highlighted as being inside a string?
> 
> Yes.

To clarify: these my answers are in the context of the current
handling of narrowing, not in the context of restricting the parser
(which you seem to reject as a useful idea anyway).

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 31/03/2023 10:46, Eli Zaretskii wrote:
>> Cc:wkirschbaum@gmail.com,gregory@heytings.org,casouri@gmail.com,
>>   62333@debbugs.gnu.org
>> Date: Fri, 31 Mar 2023 09:19:35 +0300
>> From: Eli Zaretskii<eliz@gnu.org>
>>
>>> Simple example: if the beginning of the narrowed region falls inside a
>>> (let's say multine) string, should the visible remainder of that string
>>> continue to be highlighted as a string?
>> No.
>>
>>> Or should the buffer contents after the string's closer now be
>>> highlighted as being inside a string?
>> Yes.
> To clarify: these my answers are in the context of the current
> handling of narrowing, not in the context of restricting the parser
> (which you seem to reject as a useful idea anyway).

Interesting.

You do realize that it doesn't work this way currently, right?

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 31/03/2023 09:19, Eli Zaretskii wrote:
>> If we're talking about isearch, then that seems like a natural
>> consequence of visual effect (hiding the remainder of the buffer): even
>> if isearch highlighted those other hits, they would not be visible.
> If you consider narrowing in this example to be "for visual effect",
> then everything in Emacs is "for visual effect".  After all, Emacs is
> a visual editor, showing the results of editing to the user at all
> times.  But this POV makes this part of the discussion useless.

Okay, let's rephrase that: instead of "visual effect", we can say it's 
for "visually hiding" parts of buffers. But not for changing their 
behaviors otherwise (e.g. changing syntax highlighting, etc).

In your other answer regarding the inside of a string, you seemed to 
have a different idea, though. Like, user narrowing would be able to 
affect all that stuff too.

>>> I was talking about user commands that narrow, so I'm not sure I
>>> understand how documentation could help.  When the user types "C-x n n",
>>> there's nothing Emacs can do except obey.
>> There is really only one main user command that narrows, and that's
>> narrow-to-region, bound to 'C-x n n'.
> Any user command can narrow as part of its job.

This subthread goes back to my complaint that commands don't know how to 
*interpret* the current narrowing, thus which effects it should have.

To repeat:

   Either narrowing should be used to change lexical/grammatical/etc
   context, or it should not. Do we have any documentation that says one
   or the other way? That should affect how Lisp code deals with
   narrowing -- which interactive functions should widen, and so on.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Fri, 31 Mar 2023 15:38:33 +0300
> Cc: wkirschbaum@gmail.com, gregory@heytings.org, casouri@gmail.com,
>  62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 31/03/2023 10:46, Eli Zaretskii wrote:
> >> Cc:wkirschbaum@gmail.com,gregory@heytings.org,casouri@gmail.com,
> >>   62333@debbugs.gnu.org
> >> Date: Fri, 31 Mar 2023 09:19:35 +0300
> >> From: Eli Zaretskii<eliz@gnu.org>
> >>
> >>> Simple example: if the beginning of the narrowed region falls inside a
> >>> (let's say multine) string, should the visible remainder of that string
> >>> continue to be highlighted as a string?
> >> No.
> >>
> >>> Or should the buffer contents after the string's closer now be
> >>> highlighted as being inside a string?
> >> Yes.
> > To clarify: these my answers are in the context of the current
> > handling of narrowing, not in the context of restricting the parser
> > (which you seem to reject as a useful idea anyway).
> 
> Interesting.
> 
> You do realize that it doesn't work this way currently, right?

Yes.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> Date: Fri, 31 Mar 2023 15:46:58 +0300
> Cc: wkirschbaum@gmail.com, gregory@heytings.org, casouri@gmail.com,
>  62333@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 31/03/2023 09:19, Eli Zaretskii wrote:
> >> If we're talking about isearch, then that seems like a natural
> >> consequence of visual effect (hiding the remainder of the buffer): even
> >> if isearch highlighted those other hits, they would not be visible.
> > If you consider narrowing in this example to be "for visual effect",
> > then everything in Emacs is "for visual effect".  After all, Emacs is
> > a visual editor, showing the results of editing to the user at all
> > times.  But this POV makes this part of the discussion useless.
> 
> Okay, let's rephrase that: instead of "visual effect", we can say it's 
> for "visually hiding" parts of buffers. But not for changing their 
> behaviors otherwise (e.g. changing syntax highlighting, etc).

I disagree that narrowing for, say, searching should be interpreted
like that.

> >>> I was talking about user commands that narrow, so I'm not sure I
> >>> understand how documentation could help.  When the user types "C-x n n",
> >>> there's nothing Emacs can do except obey.
> >> There is really only one main user command that narrows, and that's
> >> narrow-to-region, bound to 'C-x n n'.
> > Any user command can narrow as part of its job.
> 
> This subthread goes back to my complaint that commands don't know how to 
> *interpret* the current narrowing, thus which effects it should have.

We agree about that, at least.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 31/03/2023 16:03, Eli Zaretskii wrote:
>> Date: Fri, 31 Mar 2023 15:38:33 +0300
>> Cc: wkirschbaum@gmail.com, gregory@heytings.org, casouri@gmail.com,
>>   62333@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 31/03/2023 10:46, Eli Zaretskii wrote:
>>>> Cc:wkirschbaum@gmail.com,gregory@heytings.org,casouri@gmail.com,
>>>>    62333@debbugs.gnu.org
>>>> Date: Fri, 31 Mar 2023 09:19:35 +0300
>>>> From: Eli Zaretskii<eliz@gnu.org>
>>>>
>>>>> Simple example: if the beginning of the narrowed region falls inside a
>>>>> (let's say multine) string, should the visible remainder of that string
>>>>> continue to be highlighted as a string?
>>>> No.
>>>>
>>>>> Or should the buffer contents after the string's closer now be
>>>>> highlighted as being inside a string?
>>>> Yes.
>>> To clarify: these my answers are in the context of the current
>>> handling of narrowing, not in the context of restricting the parser
>>> (which you seem to reject as a useful idea anyway).
>>
>> Interesting.
>>
>> You do realize that it doesn't work this way currently, right?
> 
> Yes.

So... do you think we should flip font-lock-dont-widen's default to t? 
Then the behavior can be like you answered, and we could alter other 
features (e.g. indentation) too.

Note that that variable goes back to 2002, though.

And regarding indentation: with that kind of the change, narrowing 
region to a fragment of a function definition (cutting out its 
beginning) will usually make indentation shallower, when code outside of 
narrowing is considered absent.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Yuan Fu]

> On Mar 31, 2023, at 5:46 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> On 31/03/2023 09:19, Eli Zaretskii wrote:
>>> If we're talking about isearch, then that seems like a natural
>>> consequence of visual effect (hiding the remainder of the buffer): even
>>> if isearch highlighted those other hits, they would not be visible.
>> If you consider narrowing in this example to be "for visual effect",
>> then everything in Emacs is "for visual effect".  After all, Emacs is
>> a visual editor, showing the results of editing to the user at all
>> times.  But this POV makes this part of the discussion useless.
> 
> Okay, let's rephrase that: instead of "visual effect", we can say it's for "visually hiding" parts of buffers. But not for changing their behaviors otherwise (e.g. changing syntax highlighting, etc).
> 
> In your other answer regarding the inside of a string, you seemed to have a different idea, though. Like, user narrowing would be able to affect all that stuff too.
> 
>>>> I was talking about user commands that narrow, so I'm not sure I
>>>> understand how documentation could help.  When the user types "C-x n n",
>>>> there's nothing Emacs can do except obey.
>>> There is really only one main user command that narrows, and that's
>>> narrow-to-region, bound to 'C-x n n'.
>> Any user command can narrow as part of its job.
> 
> This subthread goes back to my complaint that commands don't know how to *interpret* the current narrowing, thus which effects it should have.
> 
> To repeat:
> 
>  Either narrowing should be used to change lexical/grammatical/etc
>  context, or it should not. Do we have any documentation that says one
>  or the other way? That should affect how Lisp code deals with
>  narrowing -- which interactive functions should widen, and so on.

To maybe rephrase this in another way:

I think the distinction lies between “I want to narrow to this defun and work on it without distraction” vs “treat this region as an isolated buffer”. The former used by users, the latter used by lisp programs like Info and mmm-mode. The former still considers the visible region part of the whole buffer, just temporarily narrowed for convenience, the latter wants to make everything thinks the visible region _is_ the whole buffer.

 It might be good for tree-sitter or other parsers to be exempt from (but still acknowledges) the first kind of narrowing. This way the parser can avoid unnecessary re-parse, and always provide the optimal information. We just need to modify tree-sitter functions to check for this narrowing and don’t return anything beyond the boundaries. It’s probably going to be a lot of hair, but should be doable, I think?

This way, most lisp programs still obeys the narrowing, but specific things like tree-sitter can choose to secretly look around while still appear to obey the narrowing, when peeking around wouldn’t hurt. And when the narrowing is really indented, tree-sitter (or other parser) knows not to look around.


Yuan

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 31/03/2023 21:43, Yuan Fu wrote:
> I think the distinction lies between “I want to narrow to this defun and work on it without distraction” vs “treat this region as an isolated buffer”. The former used by users, the latter used by lisp programs like Info and mmm-mode. The former still considers the visible region part of the whole buffer, just temporarily narrowed for convenience, the latter wants to make everything thinks the visible region_is_  the whole buffer.

This distinction lies in how user-level features are implemented, 
though. E.g. for font-lock we have the variable font-lock-dont-widen 
which determines whether it will widen before getting the syntax 
information (calling syntax-ppss and so on) and acting on it.

>   It might be good for tree-sitter or other parsers to be exempt from (but still acknowledges) the first kind of narrowing. This way the parser can avoid unnecessary re-parse, and always provide the optimal information. We just need to modify tree-sitter functions to check for this narrowing and don’t return anything beyond the boundaries. It’s probably going to be a lot of hair, but should be doable, I think?
> 
> This way, most lisp programs still obeys the narrowing, but specific things like tree-sitter can choose to secretly look around while still appear to obey the narrowing, when peeking around wouldn’t hurt. And when the narrowing is really indented, tree-sitter (or other parser) knows not to look around.

I'm not sure this is different from the "regular" major modes which use 
syntax-ppss as their parser. They also call 'widen' in a lot of cases to 
"peek around".

So that was my point: the external requirements for tree-sitter modes' 
behavior don't seem that different from the "classic" modes.

Narrowing in blink-matching-paren, BTW, also should have a similar 
cache-busting effect on syntax-ppss, with the main difference that it 
keeps two caches around, so it might end up reparsing only once (the 
narrowed parse state), hence the performance impact, over the smaller 
range of text, would be limited.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Fri, 31 Mar 2023 11:43:49 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  Wilhelm Kirschbaum <wkirschbaum@gmail.com>,
>  Gregory Heytings <gregory@heytings.org>,
>  62333@debbugs.gnu.org
> 
> I think the distinction lies between “I want to narrow to this defun and work on it without distraction” vs “treat this region as an isolated buffer”. The former used by users, the latter used by lisp programs like Info and mmm-mode. The former still considers the visible region part of the whole buffer, just temporarily narrowed for convenience, the latter wants to make everything thinks the visible region _is_ the whole buffer.

Users can do both, for whatever reasons.

>  It might be good for tree-sitter or other parsers to be exempt from (but still acknowledges) the first kind of narrowing. This way the parser can avoid unnecessary re-parse, and always provide the optimal information. We just need to modify tree-sitter functions to check for this narrowing and don’t return anything beyond the boundaries. It’s probably going to be a lot of hair, but should be doable, I think?

I don't see why it would be a lot of hair.  If a parser always has the
regions on which it is supposed to work, then a Lisp program using a
parser can simply widen the buffer when it needs to be sure a
narrowing doesn't get in the way.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Dmitry Gutov]

On 01/04/2023 08:47, Eli Zaretskii wrote:
>> From: Yuan Fu<casouri@gmail.com>
>> Date: Fri, 31 Mar 2023 11:43:49 -0700
>> Cc: Eli Zaretskii<eliz@gnu.org>,
>>   Wilhelm Kirschbaum<wkirschbaum@gmail.com>,
>>   Gregory Heytings<gregory@heytings.org>,
>>   62333@debbugs.gnu.org
>>
>> I think the distinction lies between “I want to narrow to this defun and work on it without distraction” vs “treat this region as an isolated buffer”. The former used by users, the latter used by lisp programs like Info and mmm-mode. The former still considers the visible region part of the whole buffer, just temporarily narrowed for convenience, the latter wants to make everything thinks the visible region_is_  the whole buffer.
> Users can do both, for whatever reasons.

Users can try to do both, but we interpret interactive narrowing (C-x n 
n) in a specific way, and currently, it's the former.

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Yuan Fu]

> On Mar 31, 2023, at 10:47 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Fri, 31 Mar 2023 11:43:49 -0700
>> Cc: Eli Zaretskii <eliz@gnu.org>,
>> Wilhelm Kirschbaum <wkirschbaum@gmail.com>,
>> Gregory Heytings <gregory@heytings.org>,
>> 62333@debbugs.gnu.org
>> 
>> I think the distinction lies between “I want to narrow to this defun and work on it without distraction” vs “treat this region as an isolated buffer”. The former used by users, the latter used by lisp programs like Info and mmm-mode. The former still considers the visible region part of the whole buffer, just temporarily narrowed for convenience, the latter wants to make everything thinks the visible region _is_ the whole buffer.
> 
> Users can do both, for whatever reasons.

Yeah, but it might be beneficial for lisp programs to be able to distinguish between the two types of narrowing, and act differently.

>> It might be good for tree-sitter or other parsers to be exempt from (but still acknowledges) the first kind of narrowing. This way the parser can avoid unnecessary re-parse, and always provide the optimal information. We just need to modify tree-sitter functions to check for this narrowing and don’t return anything beyond the boundaries. It’s probably going to be a lot of hair, but should be doable, I think?
> 
> I don't see why it would be a lot of hair.  If a parser always has the
> regions on which it is supposed to work, then a Lisp program using a
> parser can simply widen the buffer when it needs to be sure a
> narrowing doesn't get in the way.

Yes, the current situation is straightforward. I’m just saying that supporting what I described would require some work (let tree-sitter be able to peek outside the visible region but still pretend to respect narrowing).

Yuan

bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 2 Apr 2023 15:08:01 -0700
> Cc: Dmitry Gutov <dgutov@yandex.ru>,
>  Wilhelm Kirschbaum <wkirschbaum@gmail.com>,
>  Gregory Heytings <gregory@heytings.org>,
>  62333@debbugs.gnu.org
> 
> >> I think the distinction lies between “I want to narrow to this defun and work on it without distraction” vs “treat this region as an isolated buffer”. The former used by users, the latter used by lisp programs like Info and mmm-mode. The former still considers the visible region part of the whole buffer, just temporarily narrowed for convenience, the latter wants to make everything thinks the visible region _is_ the whole buffer.
> > 
> > Users can do both, for whatever reasons.
> 
> Yeah, but it might be beneficial for lisp programs to be able to distinguish between the two types of narrowing, and act differently.

I think we've found that to be impossible in practice.

> >> It might be good for tree-sitter or other parsers to be exempt from (but still acknowledges) the first kind of narrowing. This way the parser can avoid unnecessary re-parse, and always provide the optimal information. We just need to modify tree-sitter functions to check for this narrowing and don’t return anything beyond the boundaries. It’s probably going to be a lot of hair, but should be doable, I think?
> > 
> > I don't see why it would be a lot of hair.  If a parser always has the
> > regions on which it is supposed to work, then a Lisp program using a
> > parser can simply widen the buffer when it needs to be sure a
> > narrowing doesn't get in the way.
> 
> Yes, the current situation is straightforward. I’m just saying that supporting what I described would require some work (let tree-sitter be able to peek outside the visible region but still pretend to respect narrowing).

I don't understand where you see difficulties with that.  We have
already several features that widen the buffer momentarily, for
whatever reasons.  As long as the narrowing is restored after doing
whatever it needed to widen for, there's no reason to "pretend"
anything.