Tree Sitter and highlights.scm file

Tree Sitter and highlights.scm file

[Tad Lispy]

Hi! It's my first post ever to a mailing list, so I hope I do everything
right.  I'm also not very experienced with Emacs Lisp.

Here is my context. I'm working on a major mode for a new programming
language.  I'd like to leverage the built-in Tree Sitter support for
syntax highlighting.  My colleague works on the Tree Sitter grammar. In
their repository there is a file called `queries/highlights.scm`. It's a
scheme file that contains queries for different syntactic elements. Most
other grammars I see also have such file, so I think it's standard in
Tree Sitter grammars. I think it's generated by the `tree-sitter`
command line program and later customized by a grammar author. It looks
like all the necessary information for syntax highlighting is there.
Since it's also a lisp, it should be possible to use this file to
automatically derive font lock rules. But all examples of tree sitter
modes I see re-implement all the queries in Emacs Lisp with
`treesit-font-lock-rules` function. Why is that?

Before Emacs 29 there was a suite of 3rd party packages maintained by
Tuấn-Anh Nguyễn. One of the packages is a library of Tree Sitter
languages, that for each language contains the `highlights.scm` file.
See for example
https://github.com/emacs-tree-sitter/tree-sitter-langs/blob/master/queries/elm/highlights.scm.
So it looks like previously it was possible to utilize this file. Can we
still do it with Emacs 29?

Kind regards,
Tad Lispy

Re: Tree Sitter and highlights.scm file

[Eli Zaretskii]

> Date: Thu, 30 Nov 2023 11:35:42 +0000
> From: Tad Lispy <hello@tad-lispy.com>
> 
> Hi! It's my first post ever to a mailing list, so I hope I do everything
> right.  I'm also not very experienced with Emacs Lisp.
> 
> Here is my context. I'm working on a major mode for a new programming
> language.  I'd like to leverage the built-in Tree Sitter support for
> syntax highlighting.  My colleague works on the Tree Sitter grammar. In
> their repository there is a file called `queries/highlights.scm`. It's a
> scheme file that contains queries for different syntactic elements. Most
> other grammars I see also have such file, so I think it's standard in
> Tree Sitter grammars. I think it's generated by the `tree-sitter`
> command line program and later customized by a grammar author. It looks
> like all the necessary information for syntax highlighting is there.
> Since it's also a lisp, it should be possible to use this file to
> automatically derive font lock rules. But all examples of tree sitter
> modes I see re-implement all the queries in Emacs Lisp with
> `treesit-font-lock-rules` function. Why is that?
> 
> Before Emacs 29 there was a suite of 3rd party packages maintained by
> Tuấn-Anh Nguyễn. One of the packages is a library of Tree Sitter
> languages, that for each language contains the `highlights.scm` file.
> See for example
> https://github.com/emacs-tree-sitter/tree-sitter-langs/blob/master/queries/elm/highlights.scm.
> So it looks like previously it was possible to utilize this file. Can we
> still do it with Emacs 29?

Yuan, any comments?

Re: Tree Sitter and highlights.scm file

[Yuan Fu]

On 12/1/23 12:25 AM, Eli Zaretskii wrote:
>> Date: Thu, 30 Nov 2023 11:35:42 +0000
>> From: Tad Lispy <hello@tad-lispy.com>
>>
>> Hi! It's my first post ever to a mailing list, so I hope I do everything
>> right.  I'm also not very experienced with Emacs Lisp.
>>
>> Here is my context. I'm working on a major mode for a new programming
>> language.  I'd like to leverage the built-in Tree Sitter support for
>> syntax highlighting.  My colleague works on the Tree Sitter grammar. In
>> their repository there is a file called `queries/highlights.scm`. It's a
>> scheme file that contains queries for different syntactic elements. Most
>> other grammars I see also have such file, so I think it's standard in
>> Tree Sitter grammars. I think it's generated by the `tree-sitter`
>> command line program and later customized by a grammar author. It looks
>> like all the necessary information for syntax highlighting is there.
>> Since it's also a lisp, it should be possible to use this file to
>> automatically derive font lock rules. But all examples of tree sitter
>> modes I see re-implement all the queries in Emacs Lisp with
>> `treesit-font-lock-rules` function. Why is that?
That gives the major mode author more control. For complex languages, 
highlights.scm along often isn't enough.
>> Before Emacs 29 there was a suite of 3rd party packages maintained by
>> Tuấn-Anh Nguyễn. One of the packages is a library of Tree Sitter
>> languages, that for each language contains the `highlights.scm` file.
>> See for example
>> https://github.com/emacs-tree-sitter/tree-sitter-langs/blob/master/queries/elm/highlights.scm.
>> So it looks like previously it was possible to utilize this file. Can we
>> still do it with Emacs 29?
> Yuan, any comments?
Yes, it wouldn't be hard to write something to take highlights.scm. 
Couple things to note: 1) highlights.scm provided by a grammar is often 
rather basic, and editors usually extend them; 2) the highlights.scm 
file wouldn't be portable if they make use of predicates, which is 
editor-specific. Eg, neovim's highlight.scm files uses a couple 
predicates that we don't support; 3) highlight.scm files mark patterns 
with textmate semantic names like @comment, @string, @function.call, 
tree-sitter font lock expects face names like 
@font-lock-function-call-face, so we need a table that maps @comment to 
@font-lock-comment-face, etc; 4) we have the concept of "features", for 
highlights.scm, we either put everything into one feature, or 
automatically group patterns into features by the face they use.

Mostly it just needs someone to implement (and write documentation) :-)

Yuan