From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Yuan Fu Newsgroups: gmane.emacs.bugs Subject: bug#62333: 30.0.50; Issue with tree-sitter syntax tree during certain changes Date: Wed, 22 Mar 2023 20:03:20 -0700 Message-ID: <09539C5E-23DA-4B00-A3F6-873A41D6A2CE@gmail.com> References: <87fs9yur7r.fsf@gmail.com> <2fd8f2b8-d9c4-c825-a789-f2d42324859f@yandex.ru> Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Content-Type: multipart/mixed; boundary="Apple-Mail=_2866BD3A-595E-469D-A094-41C9C5DC1924" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="13669"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Wilhelm Kirschbaum , 62333@debbugs.gnu.org, Eli Zaretskii To: Dmitry Gutov Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Mar 23 04:04:19 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pfBFS-0003Lj-Ok for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 23 Mar 2023 04:04:18 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pfBFE-0008Tq-AV; Wed, 22 Mar 2023 23:04:04 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pfBFC-0008Tf-Iq for bug-gnu-emacs@gnu.org; Wed, 22 Mar 2023 23:04:02 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pfBFC-0005M5-9d for bug-gnu-emacs@gnu.org; Wed, 22 Mar 2023 23:04:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pfBFB-0005dq-Rj for bug-gnu-emacs@gnu.org; Wed, 22 Mar 2023 23:04:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Yuan Fu Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 23 Mar 2023 03:04:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 62333 X-GNU-PR-Package: emacs Original-Received: via spool by 62333-submit@debbugs.gnu.org id=B62333.167954062021651 (code B ref 62333); Thu, 23 Mar 2023 03:04:01 +0000 Original-Received: (at 62333) by debbugs.gnu.org; 23 Mar 2023 03:03:40 +0000 Original-Received: from localhost ([127.0.0.1]:37007 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pfBEq-0005d9-FV for submit@debbugs.gnu.org; Wed, 22 Mar 2023 23:03:40 -0400 Original-Received: from mail-pl1-f175.google.com ([209.85.214.175]:37613) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pfBEo-0005cu-Co for 62333@debbugs.gnu.org; Wed, 22 Mar 2023 23:03:38 -0400 Original-Received: by mail-pl1-f175.google.com with SMTP id o2so13674061plg.4 for <62333@debbugs.gnu.org>; Wed, 22 Mar 2023 20:03:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679540612; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=JemUBy5tdmu/V0eid1nZqffVPwNiwZavVNdpr9ueyRk=; b=iDuciPbhCWtoiirHa9544Lrb2IBht9C1IQ72Ooj54/nBZ9j4ZsizKRj+HeakK6Drch YZvq01Sb3aQm0R3i9ofzLV41SdZ9tteGe808HK2jZKeWMrzUwrp+vCwYSYFr/i9ccg1I t2i/ZAQ95nvNXoOCJpeyRe/nC3f3xFCGnREuwm8es58JDsdccbS+R+1M7/di6C+cnUwr YQVssQiodBh4Kz6XGNVbK9frqekwEBlM9uAihWgF0RUed9vVpIN6LxJYKACilx70XVGK lCiWBlIOPsctLIbkMLzlixIQOL+n+DkxfbFG+Waz0xnMeMPbZFsC9Q5TAD2yoecOEdm5 KL7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679540612; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JemUBy5tdmu/V0eid1nZqffVPwNiwZavVNdpr9ueyRk=; b=RvN5gCgQAwr4+9aU3Vg9WdObsLjAnroum1FEFoq3MZDEE1vFoPZY/6dL489baLcuMG mE29UAv03Ri+RWO5RERn9w64UdQ5f8UT1SrMP7PFg2roDuFNPpMyp4fcPzdZdyJcPIgP 0M1sf23d7W6Gh7xPZrjQ+ZFFRnyn855kcW7nDLUiFExEGhwRMHF5vh0U47euorizGCLY Q5rZumOlM4MDxD25ouCOyjaD0sUnRfTNgefXWBHTMSZVSK+yOwAUaOc373ifL0Vi/F4T diLI0av48G60qhObRS79CZiOpz8RtDvjQQvyfh9kt0l8YIVrahAMS8DlM6t57iloFRYm KRpQ== X-Gm-Message-State: AO0yUKX5dESEtYDWAVR3t+Lc6oSqHr8IYBQYHebqnS+/WGVJXLC63TGn 8mK2KhPUmG+yBXepL5KBEuk= X-Google-Smtp-Source: AK7set+/uiw3rObRZAXJJZfsUVoO07ApTfZM2Ecf4rIc6cX80ycypF1gOTQpBWMJS4ioY0GrvzyZcg== X-Received: by 2002:a05:6a20:b23:b0:d9:e45d:95cd with SMTP id x35-20020a056a200b2300b000d9e45d95cdmr1219730pzf.17.1679540612383; Wed, 22 Mar 2023 20:03:32 -0700 (PDT) Original-Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id a17-20020aa780d1000000b0062814f38d33sm3866986pfn.170.2023.03.22.20.03.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Mar 2023 20:03:32 -0700 (PDT) In-Reply-To: <2fd8f2b8-d9c4-c825-a789-f2d42324859f@yandex.ru> X-Mailer: Apple Mail (2.3731.400.51.1.1) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:258425 Archived-At: --Apple-Mail=_2866BD3A-595E-469D-A094-41C9C5DC1924 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Mar 22, 2023, at 6:03 PM, Dmitry Gutov wrote: >=20 > On 21/03/2023 16:00, Wilhelm Kirschbaum wrote: >> With the following text in a buffer with elixir-ts-mode: >> def foo() do >> end >> When I delete the `)` and re-insert it, the tree-sitter syntax tree = does >> not fully recover and the `do end` part is absent when observing = using >> treesit-explore-mode. >> Before removing the `)`: >> (source >> (call target: (identifier) >> (arguments >> (call target: (identifier) >> (arguments ( )))) >> (do_block do end))) >> after re-inserting `)`: >> (source >> (call target: (identifier) >> (arguments >> (call target: (identifier) >> (arguments ( )))))) >> I can see the same behaviour using ruby-ts-mode when I remove the >> following expression: >> (setq-local syntax-propertize-function #'ruby-ts--syntax-propertize) >> The ruby syntax I tested with: >> def foo() >> end >> When I add a local `syntax-propertize-function' function to = elixir-ts-mode >> it also solves the problem of not fully recovering the tree-sitter >> syntax tree. It does not seem to matter what the function achieves >> thought, so am a bit confused. >=20 > I can repro both scenarios. Except with ruby-ts-mode, the difference = in printed AST is the added wrapping in (program ...). But the visible = result is similar: "end" not being highlighted. The problem seems to be caused by narrowing during fontification: when = parsing, tree-sitter sees that BUF_ZV_BYTE is smaller than it=E2=80=99s = visible_end, and removed the part after the parenthesis from the parse = tree. I suspect this has to do with the narrowed fontification we added = for dealing with long lines.=20 CC=E2=80=99ed Eli because I don=E2=80=99t really know anything about the = locked narrow thing besides that it exists. In particular, how does it = determine when and where to narrow? To see it in action, apply this patch, and repeat Wilhelm=E2=80=99s = experiment: remove the =E2=80=9C)=E2=80=9D and reinsert it. You should = see start: 9, oend: 10, nend: 9 start: 9, oend: 10, nend: 9 start: 9, oend: 9, nend: 10 start: 9, oend: 9, nend: 10 2.5 start: 10, oend: 18, nend: 10 =E2=80=94> culprit (Make sure to turn off electric-pair-mode to reduce noise.) Yuan --Apple-Mail=_2866BD3A-595E-469D-A094-41C9C5DC1924 Content-Disposition: attachment; filename=print.diff Content-Type: application/octet-stream; x-unix-mode=0644; name="print.diff" Content-Transfer-Encoding: 7bit diff --git a/src/treesit.c b/src/treesit.c index 5a4fe3e8803..9b46dcef2d1 100644 --- a/src/treesit.c +++ b/src/treesit.c @@ -806,6 +806,7 @@ treesit_record_change (ptrdiff_t start_byte, ptrdiff_t old_end_byte, eassert (start_offset <= old_end_offset); eassert (start_offset <= new_end_offset); + printf("start: %ld, oend: %ld, nend: %ld\n", start_offset, old_end_offset, new_end_offset); treesit_tree_edit_1 (tree, start_offset, old_end_offset, new_end_offset); XTS_PARSER (lisp_parser)->need_reparse = true; @@ -925,6 +926,7 @@ treesit_sync_visible_region (Lisp_Object parser) if (visible_beg > BUF_BEGV_BYTE (buffer)) { /* Tree-sitter sees: insert at the beginning. */ + printf("1 start: %d, oend: %d, nend: %ld\n", 0, 0, visible_beg - BUF_BEGV_BYTE (buffer)); treesit_tree_edit_1 (tree, 0, 0, visible_beg - BUF_BEGV_BYTE (buffer)); visible_beg = BUF_BEGV_BYTE (buffer); eassert (visible_beg <= visible_end); @@ -933,6 +935,9 @@ treesit_sync_visible_region (Lisp_Object parser) if (visible_end < BUF_ZV_BYTE (buffer)) { /* Tree-sitter sees: insert at the end. */ + printf("2 start: %ld, oend: %ld, nend: %ld\n", visible_end - visible_beg, + visible_end - visible_beg, + BUF_ZV_BYTE (buffer) - visible_beg); treesit_tree_edit_1 (tree, visible_end - visible_beg, visible_end - visible_beg, BUF_ZV_BYTE (buffer) - visible_beg); @@ -942,6 +947,9 @@ treesit_sync_visible_region (Lisp_Object parser) else if (visible_end > BUF_ZV_BYTE (buffer)) { /* Tree-sitter sees: delete at the end. */ + printf("2.5 start: %ld, oend: %ld, nend: %ld\n", BUF_ZV_BYTE (buffer) - visible_beg, + visible_end - visible_beg, + BUF_ZV_BYTE (buffer) - visible_beg); treesit_tree_edit_1 (tree, BUF_ZV_BYTE (buffer) - visible_beg, visible_end - visible_beg, BUF_ZV_BYTE (buffer) - visible_beg); @@ -952,6 +960,7 @@ treesit_sync_visible_region (Lisp_Object parser) if (visible_beg < BUF_BEGV_BYTE (buffer)) { /* Tree-sitter sees: delete at the beginning. */ + printf("3 start: %d, oend: %ld, nend: %d\n", 0, BUF_BEGV_BYTE (buffer) - visible_beg, 0); treesit_tree_edit_1 (tree, 0, BUF_BEGV_BYTE (buffer) - visible_beg, 0); visible_beg = BUF_BEGV_BYTE (buffer); eassert (visible_beg <= visible_end); --Apple-Mail=_2866BD3A-595E-469D-A094-41C9C5DC1924--