Treesitter query question, matching only substring of a node

Treesitter query question, matching only substring of a node

[Danny Freeman]

Hello,

I am currently seeing what it will look like to add treesitter support
for clojure-mode. I am using this grammer:
https://github.com/sogaiu/tree-sitter-clojure
and it is working great so far.

It has definitions for nodes that match full namespaced keywords and
symbols, such that

:plain-keyword      matches (kwd_lit) node
:namespace/keyword  matches (kwd_lit) node

and similarly for symbols

'plain-symbol       matches (sym_lit) node
'namespaced/symbol  matches (sym_lit) node

In clojure, the part before the / character is the NAMESPACE, and the
part after the / character is the NAME.

The current clojure-mode highlights the namespaced part with a different
face than the name part.

I am trying to use this existing grammer to apply different faces to the
namespaced and named part. I can write a query that matches the whole
keyword easily, using `treesit-font-lock-rules`

```
  (treesit-font-lock-rules
   ...
   :feature 'keyword
   :language 'clojure
   '((kwd_lit) @clojure-keyword-face)
   ...
   )
```
This works fine.

What I'm trying to do now is capture a substring of that node, the
namespaced part.

I can use the `:match` predicate to identify the namespace part, but the
matched group doesn't get captured by anything

```
(defvar kw-query
  `((((kwd_lit) @kw)
     (:match "^:.*/" @kw) ;; Can I capture this??
     )))

(treesit-query-string ":namespaced/keyword" kw-query 'clojure)
```

Is something like this possible with the treesitter query engine in
Emacs? Or do I need to handle this at the grammer level? Perhaps with a
field?

There is an issue that was closed in the grammer's repository asking
about this, but their advise was to do some substring matching in the
editor, so here I am
https://github.com/sogaiu/tree-sitter-clojure/issues/28 

-- 
Danny Freeman

Re: Treesitter query question, matching only substring of a node

[Yuan Fu]

> On Dec 4, 2022, at 10:27 AM, Danny Freeman <danny@dfreeman.email> wrote:
> 
> Hello,
> 
> I am currently seeing what it will look like to add treesitter support
> for clojure-mode. I am using this grammer:
> https://github.com/sogaiu/tree-sitter-clojure
> and it is working great so far.
> 
> It has definitions for nodes that match full namespaced keywords and
> symbols, such that
> 
> :plain-keyword      matches (kwd_lit) node
> :namespace/keyword  matches (kwd_lit) node
> 
> and similarly for symbols
> 
> 'plain-symbol       matches (sym_lit) node
> 'namespaced/symbol  matches (sym_lit) node
> 
> In clojure, the part before the / character is the NAMESPACE, and the
> part after the / character is the NAME.
> 
> The current clojure-mode highlights the namespaced part with a different
> face than the name part.
> 
> I am trying to use this existing grammer to apply different faces to the
> namespaced and named part. I can write a query that matches the whole
> keyword easily, using `treesit-font-lock-rules`
> 
> ```
>  (treesit-font-lock-rules
>   ...
>   :feature 'keyword
>   :language 'clojure
>   '((kwd_lit) @clojure-keyword-face)
>   ...
>   )
> ```
> This works fine.
> 
> What I'm trying to do now is capture a substring of that node, the
> namespaced part.
> 
> I can use the `:match` predicate to identify the namespace part, but the
> matched group doesn't get captured by anything
> 
> ```
> (defvar kw-query
>  `((((kwd_lit) @kw)
>     (:match "^:.*/" @kw) ;; Can I capture this??
>     )))
> 
> (treesit-query-string ":namespaced/keyword" kw-query 'clojure)
> ```
> 
> Is something like this possible with the treesitter query engine in
> Emacs? Or do I need to handle this at the grammer level? Perhaps with a
> field?
> 
> There is an issue that was closed in the grammer's repository asking
> about this, but their advise was to do some substring matching in the
> editor, so here I am
> https://github.com/sogaiu/tree-sitter-clojure/issues/28 

I suggest capturing the whole node and using a function to fontify the symbol. With a function you can do anything you want, including applying different faces to substrings. To do that, simply define a function and use the function name as the capture name.

You can have a look at c-ts-mode--fontify-variable for a simple example.

Yuan

Re: Treesitter query question, matching only substring of a node

[Danny Freeman]

Yuan Fu <casouri@gmail.com> writes:

> I suggest capturing the whole node and using a function to fontify the symbol. With a function you
> can do anything you want, including applying different faces to substrings. To do that, simply
> define a function and use the function name as the capture name.
>
> You can have a look at c-ts-mode--fontify-variable for a simple example.

This is exactly what I was looking for. I can't believe I didn't think
of this when I was reading the documentation.

Thank you for your help!

-- 
Danny Freeman

Re: Treesitter query question, matching only substring of a node

[Yuan Fu]

> On Dec 5, 2022, at 7:25 AM, Danny Freeman <danny@dfreeman.email> wrote:
> 
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>> I suggest capturing the whole node and using a function to fontify the symbol. With a function you
>> can do anything you want, including applying different faces to substrings. To do that, simply
>> define a function and use the function name as the capture name.
>> 
>> You can have a look at c-ts-mode--fontify-variable for a simple example.
> 
> This is exactly what I was looking for. I can't believe I didn't think
> of this when I was reading the documentation.
> 
> Thank you for your help!

Let me know if you have ideas on how could the documentation be improved to make it clearer. I have the curse of knowledge on me and everything seems so easy and clear ;-) 

Yuan

Re: Treesitter query question, matching only substring of a node

[Danny Freeman]

Yuan Fu <casouri@gmail.com> writes:

>> On Dec 5, 2022, at 7:25 AM, Danny Freeman <danny@dfreeman.email> wrote:
>> 
>> 
>> Yuan Fu <casouri@gmail.com> writes:
>> 
>>> I suggest capturing the whole node and using a function to fontify the symbol. With a function you
>>> can do anything you want, including applying different faces to substrings. To do that, simply
>>> define a function and use the function name as the capture name.
>>> 
>>> You can have a look at c-ts-mode--fontify-variable for a simple example.
>> 
>> This is exactly what I was looking for. I can't believe I didn't think
>> of this when I was reading the documentation.
>> 
>> Thank you for your help!
>
> Let me know if you have ideas on how could the documentation be improved to make it clearer. I have the curse of knowledge on me and everything seems so easy and clear ;-) 
>
> Yuan

So far I am only working on font locking, and have NO experience with
font locking in Emacs or with tree-sitter. With that in mind, using the
starter-guide file, the html-manual, an existing grammar, and examples
from python mode, I was able to get a working prototype in clojure-mode
in 1 day. I think that speaks to the quality of the documentation!

The treesit-explore-mode is a really great tool for implementing the
syntax highlighting as well. It is a real treat to work with.

Here is how I ended up using a function to fontify the symbols the way I
needed to, in case you are curious:

https://github.com/dannyfreeman/clojure-mode/blob/513403c5e97330b65d024e2e904c165021540480/clojure-ts-mode.el#L162-L177

Right now that entire file is a rough proof of concept. There is a lot
more work that needs to be done before it's ready for the general
public.

-- 
Danny Freeman

Re: Treesitter query question, matching only substring of a node

[Yuan Fu]

> On Dec 6, 2022, at 7:13 AM, Danny Freeman <danny@dfreeman.email> wrote:
> 
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>>> On Dec 5, 2022, at 7:25 AM, Danny Freeman <danny@dfreeman.email> wrote:
>>> 
>>> 
>>> Yuan Fu <casouri@gmail.com> writes:
>>> 
>>>> I suggest capturing the whole node and using a function to fontify the symbol. With a function you
>>>> can do anything you want, including applying different faces to substrings. To do that, simply
>>>> define a function and use the function name as the capture name.
>>>> 
>>>> You can have a look at c-ts-mode--fontify-variable for a simple example.
>>> 
>>> This is exactly what I was looking for. I can't believe I didn't think
>>> of this when I was reading the documentation.
>>> 
>>> Thank you for your help!
>> 
>> Let me know if you have ideas on how could the documentation be improved to make it clearer. I have the curse of knowledge on me and everything seems so easy and clear ;-) 
>> 
>> Yuan
> 
> So far I am only working on font locking, and have NO experience with
> font locking in Emacs or with tree-sitter. With that in mind, using the
> starter-guide file, the html-manual, an existing grammar, and examples
> from python mode, I was able to get a working prototype in clojure-mode
> in 1 day. I think that speaks to the quality of the documentation!

Great!

> 
> The treesit-explore-mode is a really great tool for implementing the
> syntax highlighting as well. It is a real treat to work with.
> 
> Here is how I ended up using a function to fontify the symbols the way I
> needed to, in case you are curious:
> 
> https://github.com/dannyfreeman/clojure-mode/blob/513403c5e97330b65d024e2e904c165021540480/clojure-ts-mode.el#L162-L177
> 
> Right now that entire file is a rough proof of concept. There is a lot
> more work that needs to be done before it's ready for the general
> public.

It looks great!

Yuan