bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Alan Mackenzie]

Hello, Emacs.

In the emacs-29 branch, with the latest commit being:

commit 558b59d81b938fc434e62523106360b9704c88e2 (HEAD -> emacs-29, origin/emacs-29, origin/HEAD)
Author: Yuan Fu <casouri@gmail.com>
Date:   Thu Dec 29 11:52:06 2022 -0800

    Add color fontification in css-ts-mode (bug#60405)

:
(i) emacs -Q
(ii) (setq backtrace-on-redisplay-error t)
(iii) C-x b string.cc <RET>
(iv) M-x c++-ts-mode <RET>
(v) Type "char" (without the quote marks).

This throws an exception in the redisplay code.  The backtrace in the
*Redisplay-trace* buffer is:

Error: treesit-query-error ("Node type error at" 195 "Debug the query with `treesit-query-validate'")
  debug-early-backtrace()
  debug-early(error (treesit-query-error "Node type error at" 195 "Debug the query with `treesit-query-validate'"))
  treesit-font-lock-fontify-region(1 5 nil)
  font-lock-default-fontify-region(1 5 nil)
  font-lock-fontify-region(1 5)
  #f(compiled-function (fun) #<bytecode -0x156ee3e72a1d5e83>)(font-lock-fontify-region)
  jit-lock--run-functions(1 5)
  jit-lock-fontify-now(1 5)
  jit-lock-function(1)
  redisplay_internal\ \(C\ function\)()

..  That this insertion causes an exception is clearly a bug.

But the contents of the backtrace seem to me to be less helpful than they
might be.  What does the "the" in "the query" refer to?  There is no
referent yet established.  What does the 195 refer to?  It clearly isn't
a buffer position, which is what it would mean in lots of other Emacs
error messages.

Just as a matter of interest, in c-ts-mode, the above actions run without
problem.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Yuan Fu]

> On Dec 30, 2022, at 4:24 AM, Alan Mackenzie <acm@muc.de> wrote:
> 
> Hello, Emacs.
> 
> In the emacs-29 branch, with the latest commit being:
> 
> commit 558b59d81b938fc434e62523106360b9704c88e2 (HEAD -> emacs-29, origin/emacs-29, origin/HEAD)
> Author: Yuan Fu <casouri@gmail.com>
> Date:   Thu Dec 29 11:52:06 2022 -0800
> 
>    Add color fontification in css-ts-mode (bug#60405)
> 
> :
> (i) emacs -Q
> (ii) (setq backtrace-on-redisplay-error t)
> (iii) C-x b string.cc <RET>
> (iv) M-x c++-ts-mode <RET>
> (v) Type "char" (without the quote marks).
> 
> This throws an exception in the redisplay code.  The backtrace in the
> *Redisplay-trace* buffer is:
> 
> Error: treesit-query-error ("Node type error at" 195 "Debug the query with `treesit-query-validate'")
>  debug-early-backtrace()
>  debug-early(error (treesit-query-error "Node type error at" 195 "Debug the query with `treesit-query-validate'"))
>  treesit-font-lock-fontify-region(1 5 nil)
>  font-lock-default-fontify-region(1 5 nil)
>  font-lock-fontify-region(1 5)
>  #f(compiled-function (fun) #<bytecode -0x156ee3e72a1d5e83>)(font-lock-fontify-region)
>  jit-lock--run-functions(1 5)
>  jit-lock-fontify-now(1 5)
>  jit-lock-function(1)
>  redisplay_internal\ \(C\ function\)()
> 
> ..  That this insertion causes an exception is clearly a bug.
> 
> But the contents of the backtrace seem to me to be less helpful than they
> might be.  What does the "the" in "the query" refer to?  There is no
> referent yet established.  What does the 195 refer to?  It clearly isn't
> a buffer position, which is what it would mean in lots of other Emacs
> error messages.
> 
> Just as a matter of interest, in c-ts-mode, the above actions run without
> problem.

Thank you! I couldn’t reproduce it on the latest trunk, even though I checked that there isn’t any relevant change between that commit and the latest one. I made a small modification to treesit.c so now the signal will contain the source of the query that caused the error. Could you try again with trunk and show me the backtrace? This time it should contain the problematic query.

Yuan

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Eli Zaretskii]

> Cc: 60427@debbugs.gnu.org
> From: Yuan Fu <casouri@gmail.com>
> Date: Fri, 30 Dec 2022 17:02:36 -0800
> 
> Thank you! I couldn’t reproduce it on the latest trunk

AFAIU, the report was about the emacs-29 branch, not about the master
branch.  So what did you mean by "latest trunk" above?

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Yuan Fu]

> On Dec 30, 2022, at 10:42 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Cc: 60427@debbugs.gnu.org
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Fri, 30 Dec 2022 17:02:36 -0800
>> 
>> Thank you! I couldn’t reproduce it on the latest trunk
> 
> AFAIU, the report was about the emacs-29 branch, not about the master
> branch.  So what did you mean by "latest trunk" above?

Sorry, imprecise wording, I mean latest emacs-29 :-)

Yuan

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Alan Mackenzie]

Hello, Yuan.

On Fri, Dec 30, 2022 at 22:55:41 -0800, Yuan Fu wrote:


> > On Dec 30, 2022, at 10:42 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> >> Cc: 60427@debbugs.gnu.org
> >> From: Yuan Fu <casouri@gmail.com>
> >> Date: Fri, 30 Dec 2022 17:02:36 -0800

> >> Thank you! I couldn’t reproduce it on the latest trunk

> > AFAIU, the report was about the emacs-29 branch, not about the master
> > branch.  So what did you mean by "latest trunk" above?

> Sorry, imprecise wording, I mean latest emacs-29 :-)

I've just downloaded the latest commits in the emacs-29 branch,
bootstrapped, and I still see the problem, both in X GUI and on a Linux
console.  Just as a matter of interest, I get the error after typing the
h in "char", so the problem would appear not to be related to the
specific character string "char".

As you mentioned, the backtrace now looks a bit different, namely:

Error: treesit-query-error ("Node type error at" 195 "[\"break\" \"case\" \"const\" \"continue\" \"default\" \"do\" \"else\" \"enum\" \"extern\" \"for\" \"goto\" \"if\" \"inline\" \"register\" \"return\" \"sizeof\" \"static\" \"struct\" \"\switch\" \"typedef\" \"union\" \"volatile\" \"while\" \"and\" \"and_eq\" \"bitand\" \"bitor\" \"catch\" \"class\" \"co_await\" \"co_return\" \"co_yield\" \"compl\" \"concept\" \"consteval\" \"constexpr\" \"constinit\" \"decltype\" \"delete\\" \"explicit\" \"final\" \"friend\" \"mutable\" \"namespace\" \"new\" \"noexcept\" \"not\" \"not_eq\" \"operator\" \"or\" \"or_eq\" \"override\" \"private\" \"protected\" \"public\" \"requires\" \"template\" \"throw\" \"try\" \"typename\"\ \"using\" \"virtual\" \"xor\" \"xor_eq\"] @font-lock-keyword-face (auto) @font-
 lock-keyword-face (this) @font-lock-keyword-face" "Debug the query with `treesit-query-validate'")
  debug-early-backtrace()
  debug-early(error (treesit-query-error "Node type error at" 195 "[\"break\" \"case\" \"const\" \"continue\" \"default\" \"do\" \"else\" \"enum\" \"extern\" \"for\" \"goto\" \"if\" \"inline\" \"register\" \"return\" \"sizeof\" \"static\" \\"struct\" \"switch\" \"typedef\" \"union\" \"volatile\" \"while\" \"and\" \"and_eq\" \"bitand\" \"bitor\" \"catch\" \"class\" \"co_await\" \"co_return\" \"co_yield\" \"compl\" \"concept\" \"consteval\" \"constexpr\" \"constinit\" \"declty\pe\" \"delete\" \"explicit\" \"final\" \"friend\" \"mutable\" \"namespace\" \"new\" \"noexcept\" \"not\" \"not_eq\" \"operator\" \"or\" \"or_eq\" \"override\" \"private\" \"protected\" \"public\" \"requires\" \"template\" \"throw\" \"try\"\ \"typename\" \"using\" \"virtual\" \"xor\" \"xor_eq\"] @font-lock-keyword-face 
 (auto) @font-lock-keyword-face (this) @font-lock-keyword-face" "Debug the query with `treesit-query-validate'"))
  treesit-font-lock-fontify-region(1 5 nil)
  font-lock-default-fontify-region(1 5 nil)
  font-lock-fontify-region(1 5)
  #f(compiled-function (fun) #<bytecode -0x156e29a3e4e81ac3>)(font-lock-fontify-region)
  jit-lock--run-functions(1 5)
  jit-lock-fontify-now(1 5)
  jit-lock-function(1)
  redisplay_internal\ \(C\ function\)()


I forgot to give my configuration arguments, maybe that has something to
do with it.  They're:

  ./configure --with-gif=no --with-tiff=no --with-gpm --with-native-compilation

..

> Yuan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Eli Zaretskii]

> Date: Sat, 31 Dec 2022 10:29:21 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, 60427@debbugs.gnu.org, acm@muc.de
> From: Alan Mackenzie <acm@muc.de>
> 
> I've just downloaded the latest commits in the emacs-29 branch,
> bootstrapped, and I still see the problem, both in X GUI and on a Linux
> console.  Just as a matter of interest, I get the error after typing the
> h in "char", so the problem would appear not to be related to the
> specific character string "char".

I cannot reproduce this with the current emacs-29 branch, neither with
native-compilation nor without it.  I also tried in a -nw session, and
couldn't reproduce there as well.

How did you get and build the C++ grammar file for tree-sitter?  If
you cloned their repository, what is the last commit from which you
produced the shared library file?

And which version of the tree-sitter library are you using?

Finally, is the recipe you posted complete? are you sure you did
nothing else since starting Emacs?

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Eli Zaretskii]

> Date: Sat, 31 Dec 2022 10:29:21 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, 60427@debbugs.gnu.org, acm@muc.de
> From: Alan Mackenzie <acm@muc.de>
> 
> As you mentioned, the backtrace now looks a bit different, namely:
> 
> Error: treesit-query-error ("Node type error at" 195 "[\"break\" \"case\" \"co

Yuan, what is "195" in this error message?  According to the source
code, its is "error_offset+1", but what is that?  If it's a buffer
position, then how come we see 195 when the buffer has only "char" in
it?

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Yuan Fu]

> On Dec 31, 2022, at 3:05 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Date: Sat, 31 Dec 2022 10:29:21 +0000
>> Cc: Eli Zaretskii <eliz@gnu.org>, 60427@debbugs.gnu.org, acm@muc.de
>> From: Alan Mackenzie <acm@muc.de>
>> 
>> As you mentioned, the backtrace now looks a bit different, namely:
>> 
>> Error: treesit-query-error ("Node type error at" 195 "[\"break\" \"case\" \"co
> 
> Yuan, what is "195" in this error message?  According to the source
> code, its is "error_offset+1", but what is that?  If it's a buffer
> position, then how come we see 195 when the buffer has only "char" in
> it?

It is the position of the error in the query string. Any suggestion to make that clearer in the message? Maybe “Node type error in the query string at”?

Yuan

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Yuan Fu]

> On Dec 31, 2022, at 2:29 AM, Alan Mackenzie <acm@muc.de> wrote:
> 
> Hello, Yuan.
> 
> On Fri, Dec 30, 2022 at 22:55:41 -0800, Yuan Fu wrote:
> 
> 
>>> On Dec 30, 2022, at 10:42 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>>>> Cc: 60427@debbugs.gnu.org
>>>> From: Yuan Fu <casouri@gmail.com>
>>>> Date: Fri, 30 Dec 2022 17:02:36 -0800
> 
>>>> Thank you! I couldn’t reproduce it on the latest trunk
> 
>>> AFAIU, the report was about the emacs-29 branch, not about the master
>>> branch.  So what did you mean by "latest trunk" above?
> 
>> Sorry, imprecise wording, I mean latest emacs-29 :-)
> 
> I've just downloaded the latest commits in the emacs-29 branch,
> bootstrapped, and I still see the problem, both in X GUI and on a Linux
> console.  Just as a matter of interest, I get the error after typing the
> h in "char", so the problem would appear not to be related to the
> specific character string "char".
> 
> As you mentioned, the backtrace now looks a bit different, namely:
> 
> Error: treesit-query-error ("Node type error at" 195 "[\"break\" \"case\" \"const\" \"continue\" \"default\" \"do\" \"else\" \"enum\" \"extern\" \"for\" \"goto\" \"if\" \"inline\" \"register\" \"return\" \"sizeof\" \"static\" \"struct\" \"\switch\" \"typedef\" \"union\" \"volatile\" \"while\" \"and\" \"and_eq\" \"bitand\" \"bitor\" \"catch\" \"class\" \"co_await\" \"co_return\" \"co_yield\" \"compl\" \"concept\" \"consteval\" \"constexpr\" \"constinit\" \"decltype\" \"delete\\" \"explicit\" \"final\" \"friend\" \"mutable\" \"namespace\" \"new\" \"noexcept\" \"not\" \"not_eq\" \"operator\" \"or\" \"or_eq\" \"override\" \"private\" \"protected\" \"public\" \"requires\" \"template\" \"throw\" \"try\" \"typename\"\ \"using\" \"virtual\" \"xor\" \"xor_eq\"] @font-lock-keyword-face (auto) @font-lock-keyword-face (this) @font-lock-keyword-face" "Debug the query with `treesit-query-validate'")
>  debug-early-backtrace()
>  debug-early(error (treesit-query-error "Node type error at" 195 "[\"break\" \"case\" \"const\" \"continue\" \"default\" \"do\" \"else\" \"enum\" \"extern\" \"for\" \"goto\" \"if\" \"inline\" \"register\" \"return\" \"sizeof\" \"static\" \\"struct\" \"switch\" \"typedef\" \"union\" \"volatile\" \"while\" \"and\" \"and_eq\" \"bitand\" \"bitor\" \"catch\" \"class\" \"co_await\" \"co_return\" \"co_yield\" \"compl\" \"concept\" \"consteval\" \"constexpr\" \"constinit\" \"declty\pe\" \"delete\" \"explicit\" \"final\" \"friend\" \"mutable\" \"namespace\" \"new\" \"noexcept\" \"not\" \"not_eq\" \"operator\" \"or\" \"or_eq\" \"override\" \"private\" \"protected\" \"public\" \"requires\" \"template\" \"throw\" \"try\"\ \"typename\" \"using\" \"virtual\" \"xor\" \"xor_eq\"] @font-lock-keyword-face (auto) @font-lock-keyword-face (this) @font-lock-keyword-face" "Debug the query with `treesit-query-validate'"))
>  treesit-font-lock-fontify-region(1 5 nil)
>  font-lock-default-fontify-region(1 5 nil)
>  font-lock-fontify-region(1 5)
>  #f(compiled-function (fun) #<bytecode -0x156e29a3e4e81ac3>)(font-lock-fontify-region)
>  jit-lock--run-functions(1 5)
>  jit-lock-fontify-now(1 5)
>  jit-lock-function(1)
>  redisplay_internal\ \(C\ function\)()
> 
> 
> I forgot to give my configuration arguments, maybe that has something to
> do with it.  They're:
> 
>  ./configure --with-gif=no --with-tiff=no --with-gpm --with-native-compilation
> 
> ..

As Eli observed, this seems like a mismatch between the C++ language grammar you used and the queries used by c++-ts-mode. We need to pin down or declare the version of the language grammar a major mode is written on. Maybe in NEWS file?

Yuan

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 31 Dec 2022 14:11:03 -0800
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  60427@debbugs.gnu.org
> 
> As Eli observed, this seems like a mismatch between the C++ language grammar you used and the queries used by c++-ts-mode. We need to pin down or declare the version of the language grammar a major mode is written on. Maybe in NEWS file?

I just asked what version of the grammar and library Alan uses.

If we know which version is required. can we devise a test for it?
Then we could run that test when the language grammar is first loaded,
and reject versions that are not suitable.

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 31 Dec 2022 14:06:20 -0800
> Cc: Alan Mackenzie <acm@muc.de>,
>  60427@debbugs.gnu.org
> 
> > On Dec 31, 2022, at 3:05 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >> Date: Sat, 31 Dec 2022 10:29:21 +0000
> >> Cc: Eli Zaretskii <eliz@gnu.org>, 60427@debbugs.gnu.org, acm@muc.de
> >> From: Alan Mackenzie <acm@muc.de>
> >> 
> >> As you mentioned, the backtrace now looks a bit different, namely:
> >> 
> >> Error: treesit-query-error ("Node type error at" 195 "[\"break\" \"case\" \"co
> > 
> > Yuan, what is "195" in this error message?  According to the source
> > code, its is "error_offset+1", but what is that?  If it's a buffer
> > position, then how come we see 195 when the buffer has only "char" in
> > it?
> 
> It is the position of the error in the query string. Any suggestion to make that clearer in the message? Maybe “Node type error in the query string at”?

And the "query string" is that long string enclosed in [] brackets?

"Node type error at query string position" is the best I can come up
with, but the problem is that figuring out where is that is not easy,
given all the backslashes.  Not sure what we can do about that,
though.

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Alan Mackenzie]

Hello, Eli.

On Sat, Dec 31, 2022 at 13:02:26 +0200, Eli Zaretskii wrote:
> > Date: Sat, 31 Dec 2022 10:29:21 +0000
> > Cc: Eli Zaretskii <eliz@gnu.org>, 60427@debbugs.gnu.org, acm@muc.de
> > From: Alan Mackenzie <acm@muc.de>

> > I've just downloaded the latest commits in the emacs-29 branch,
> > bootstrapped, and I still see the problem, both in X GUI and on a Linux
> > console.  Just as a matter of interest, I get the error after typing the
> > h in "char", so the problem would appear not to be related to the
> > specific character string "char".

> I cannot reproduce this with the current emacs-29 branch, neither with
> native-compilation nor without it.  I also tried in a -nw session, and
> couldn't reproduce there as well.

> How did you get and build the C++ grammar file for tree-sitter?  If
> you cloned their repository, what is the last commit from which you
> produced the shared library file?

I installed it using my GNU/Linux distribution's package manager.
That's Gentoo.

> And which version of the tree-sitter library are you using?

I've got tree-sitter-cpp 0.20.0.  In /usr/lib64, I get:

    /usr/lib64/libtree-sitter-cpp.so -> libtree-sitter-cpp.so.13

> Finally, is the recipe you posted complete? are you sure you did
> nothing else since starting Emacs?

Yes, it's complete.  I just tried following my own recipe exactly, and
it produced the unwanted results described.

The feeling at the moment is that that libtree-sitter-cpp.so.13 isn't
compatible with Emacs 29.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#60427: Emacs-29, c++-ts-mode: typing "char" into an empty buffer causes an exception in redisplay.

[Eli Zaretskii]

> Date: Sun, 1 Jan 2023 19:31:26 +0000
> Cc: casouri@gmail.com, 60427@debbugs.gnu.org, acm@muc.de
> From: Alan Mackenzie <acm@muc.de>
> 
> > How did you get and build the C++ grammar file for tree-sitter?  If
> > you cloned their repository, what is the last commit from which you
> > produced the shared library file?
> 
> I installed it using my GNU/Linux distribution's package manager.
> That's Gentoo.
> 
> > And which version of the tree-sitter library are you using?
> 
> I've got tree-sitter-cpp 0.20.0.  In /usr/lib64, I get:
> 
>     /usr/lib64/libtree-sitter-cpp.so -> libtree-sitter-cpp.so.13

That's more than a year ago (Oct 2021), if my reading of their Git
logs is correct.  Please try a newer version, perhaps from their Git
repository, if you can.

> The feeling at the moment is that that libtree-sitter-cpp.so.13 isn't
> compatible with Emacs 29.

Could be.  But we won't know for sure until you try a newer version of
the grammar library.