treesit: how to get it to parse multiple languages

treesit: how to get it to parse multiple languages

[Andrew De Angelis]

[-- Attachment #1: Type: text/plain, Size: 2131 bytes --]

Hello everyone and thanks for all your work!

I'm trying to get a better understanding of treesit.el, and I've stumbled
on a couple of things that make me think the manual is either
outdated/faulty, or just not entirely clear and I'm missing something.

The latter is most likely, but I'd appreciate any help in figuring out what
exactly is wrong in my approach/setup. I would be happy to contribute to
the manual, if needed, to ensure it is clearer.

This is the relevant section of the manual:
https://www.gnu.org/software/emacs/manual/html_node/elisp/Multiple-Languages.html
I've started out with simply trying to recreate the setup described in the
manual, but I've run into some issues.
Here's what I've done so far:
 - I've defined a very simple `html-ts-mode`, using the elisp functions
from the manual:
https://github.com/andrewdea/poc-html-ts-mode/blob/main/html-ts-mode.el
 - I activate this mode when visiting the example.html file (which is also
copied from the manual):
https://github.com/andrewdea/poc-html-ts-mode/blob/main/example.html
 - the queries seem to be working as expected: when I'm in a buffer
visiting example.html, evaluating `(treesit-query-capture 'html css-query)`
and `(treesit-query-capture 'html js-query)` return the expected nodes
 - ISSUE: `treesit-update-ranges` doesn't seem to be working as expected:
even if I call it multiple times, the parser for the whole buffer seems to
still be 'html. `(treesit-language-at (point))` always returns 'html, even
when I'm inside the nodes captured by the css-query or js-query.

Some additional context: the reason I'm looking into tree-sitter (and its
functionalities to support multiple languages) is to potentially use it to
fontify markdown code blocks and to improve emacs support for python
notebooks. For markdown, I was trying a similar approach to the HTML one
described in the manual, but ran into other similar issues:
https://www.reddit.com/r/emacs/comments/1gcrv8k/syntaxhighlighting_codeblocks_in_markdown/
.
I'm just including this as context.

Let me know if any of this is not clear.

Thanks in advance for all your help!

Best,
Andrew

[-- Attachment #2: Type: text/html, Size: 2902 bytes --]

Re: treesit: how to get it to parse multiple languages

[Eli Zaretskii]

> From: Andrew De Angelis <bobodeangelis@gmail.com>
> Date: Sun, 3 Nov 2024 13:28:57 -0500
> 
> I'm trying to get a better understanding of treesit.el, and I've stumbled on a couple of things that make me
> think the manual is either outdated/faulty, or just not entirely clear and I'm missing something.
> 
> The latter is most likely, but I'd appreciate any help in figuring out what exactly is wrong in my
> approach/setup. I would be happy to contribute to the manual, if needed, to ensure it is clearer.
> 
> This is the relevant section of the manual:
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Multiple-Languages.html
> I've started out with simply trying to recreate the setup described in the manual, but I've run into some
> issues. 
> Here's what I've done so far:
>  - I've defined a very simple `html-ts-mode`, using the elisp functions from the manual:
> https://github.com/andrewdea/poc-html-ts-mode/blob/main/html-ts-mode.el
>  - I activate this mode when visiting the example.html file (which is also copied from the manual):
> https://github.com/andrewdea/poc-html-ts-mode/blob/main/example.html
>  - the queries seem to be working as expected: when I'm in a buffer visiting example.html, evaluating
> `(treesit-query-capture 'html css-query)` and `(treesit-query-capture 'html js-query)` return the expected
> nodes
>  - ISSUE: `treesit-update-ranges` doesn't seem to be working as expected: even if I call it multiple times, the
> parser for the whole buffer seems to still be 'html. `(treesit-language-at (point))` always returns 'html, even
> when I'm inside the nodes captured by the css-query or js-query.
> 
> Some additional context: the reason I'm looking into tree-sitter (and its functionalities to support multiple
> languages) is to potentially use it to fontify markdown code blocks and to improve emacs support for python
> notebooks. For markdown, I was trying a similar approach to the HTML one described in the manual, but ran
> into other similar issues:
> https://www.reddit.com/r/emacs/comments/1gcrv8k/syntaxhighlighting_codeblocks_in_markdown/.
> I'm just including this as context.
> 
> Let me know if any of this is not clear.
> 
> Thanks in advance for all your help!

Yuan, can you help Andrew?

Re: treesit: how to get it to parse multiple languages

[Yuan Fu]

> On Nov 4, 2024, at 4:02 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Andrew De Angelis <bobodeangelis@gmail.com>
>> Date: Sun, 3 Nov 2024 13:28:57 -0500
>> 
>> I'm trying to get a better understanding of treesit.el, and I've stumbled on a couple of things that make me
>> think the manual is either outdated/faulty, or just not entirely clear and I'm missing something.
>> 
>> The latter is most likely, but I'd appreciate any help in figuring out what exactly is wrong in my
>> approach/setup. I would be happy to contribute to the manual, if needed, to ensure it is clearer.
>> 
>> This is the relevant section of the manual:
>> https://www.gnu.org/software/emacs/manual/html_node/elisp/Multiple-Languages.html
>> I've started out with simply trying to recreate the setup described in the manual, but I've run into some
>> issues. 
>> Here's what I've done so far:
>> - I've defined a very simple `html-ts-mode`, using the elisp functions from the manual:
>> https://github.com/andrewdea/poc-html-ts-mode/blob/main/html-ts-mode.el
>> - I activate this mode when visiting the example.html file (which is also copied from the manual):
>> https://github.com/andrewdea/poc-html-ts-mode/blob/main/example.html
>> - the queries seem to be working as expected: when I'm in a buffer visiting example.html, evaluating
>> `(treesit-query-capture 'html css-query)` and `(treesit-query-capture 'html js-query)` return the expected
>> nodes
>> - ISSUE: `treesit-update-ranges` doesn't seem to be working as expected: even if I call it multiple times, the
>> parser for the whole buffer seems to still be 'html. `(treesit-language-at (point))` always returns 'html, even
>> when I'm inside the nodes captured by the css-query or js-query.
>> 
>> Some additional context: the reason I'm looking into tree-sitter (and its functionalities to support multiple
>> languages) is to potentially use it to fontify markdown code blocks and to improve emacs support for python
>> notebooks. For markdown, I was trying a similar approach to the HTML one described in the manual, but ran
>> into other similar issues:
>> https://www.reddit.com/r/emacs/comments/1gcrv8k/syntaxhighlighting_codeblocks_in_markdown/.
>> I'm just including this as context.
>> 
>> Let me know if any of this is not clear.
>> 
>> Thanks in advance for all your help!
> 
> Yuan, can you help Andrew?

Ah yes, thanks for the ping. Andrew, I take that your problem is with treesit-language-at, right? Specifically, it doesn’t return expected results. That’s because for treesit-language-at to work, major mode needs to define treesit-language-at-function.

This confusion has came up a couple times now, evidently treesit-language-at is not very intuitive. Hopefully it’ll be fixed by our updated manual for Emacs 30. In Emacs 30, we define treesit-language-at-function in the example code:

   Emacs automates this process in ‘treesit-update-ranges’.  A
multi-language major mode should set ‘treesit-range-settings’ so that
‘treesit-update-ranges’ knows how to perform this process automatically.
Major modes should use the helper function ‘treesit-range-rules’ to
generate a value that can be assigned to ‘treesit-range-settings’.  The
settings in the following example directly translate into operations
shown above.

     (setq treesit-range-settings
           (treesit-range-rules
            :embed 'javascript
            :host 'html
            '((script_element (raw_text) @capture))
            :embed 'css
            :host 'html
            '((style_element (raw_text) @capture))))

     ;; Major modes with multiple languages should always set
     ;; `treesit-language-at-point-function' (which see).
     (setq treesit-language-at-point-function
           (lambda (pos)
             (let* ((node (treesit-node-at pos 'html))
                    (parent (treesit-node-parent node)))
               (cond
                ((and node parent
                      (equal (treesit-node-type node) "raw_text")
                      (equal (treesit-node-type parent) "script_element"))
                 'javascript)
                ((and node parent
                      (equal (treesit-node-type node) "raw_text")
                      (equal (treesit-node-type parent) "style_element"))
                 'css)
                (t 'html)))))

And FYI, in Emacs 30 we added local parsers, that might make implementing code/markdown blocks in a notebook easier.

Yuan

Re: treesit: how to get it to parse multiple languages

[Andrew De Angelis]

[-- Attachment #1: Type: text/plain, Size: 5574 bytes --]

Thanks so much!

I took a look at the Emacs 30 manual and it's a lot clearer, it's perfect!
I think one thing that would truly be ideal is if there is a major mode out
there that already implements multiple-language functionalities using
treesitter. Seeing all the components in action would be quite helpful: the
simple HTML examples are very clarifying but they can only do so much.

Do you all know if such a mode exists? `(ripgrep-regexp "local-parser"
source-directory)` on the master branch only shows me matches in
`treesit.el` itself (and associated ChangeLog / manual).
If it doesn't exist yet I'm happy to give it a knack when implementing the
notebook mode. I might have run into some more questions then :)

On Tue, Nov 5, 2024 at 1:47 AM Yuan Fu <casouri@gmail.com> wrote:

>
>
> > On Nov 4, 2024, at 4:02 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >
> >> From: Andrew De Angelis <bobodeangelis@gmail.com>
> >> Date: Sun, 3 Nov 2024 13:28:57 -0500
> >>
> >> I'm trying to get a better understanding of treesit.el, and I've
> stumbled on a couple of things that make me
> >> think the manual is either outdated/faulty, or just not entirely clear
> and I'm missing something.
> >>
> >> The latter is most likely, but I'd appreciate any help in figuring out
> what exactly is wrong in my
> >> approach/setup. I would be happy to contribute to the manual, if
> needed, to ensure it is clearer.
> >>
> >> This is the relevant section of the manual:
> >>
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Multiple-Languages.html
> >> I've started out with simply trying to recreate the setup described in
> the manual, but I've run into some
> >> issues.
> >> Here's what I've done so far:
> >> - I've defined a very simple `html-ts-mode`, using the elisp functions
> from the manual:
> >> https://github.com/andrewdea/poc-html-ts-mode/blob/main/html-ts-mode.el
> >> - I activate this mode when visiting the example.html file (which is
> also copied from the manual):
> >> https://github.com/andrewdea/poc-html-ts-mode/blob/main/example.html
> >> - the queries seem to be working as expected: when I'm in a buffer
> visiting example.html, evaluating
> >> `(treesit-query-capture 'html css-query)` and `(treesit-query-capture
> 'html js-query)` return the expected
> >> nodes
> >> - ISSUE: `treesit-update-ranges` doesn't seem to be working as
> expected: even if I call it multiple times, the
> >> parser for the whole buffer seems to still be 'html.
> `(treesit-language-at (point))` always returns 'html, even
> >> when I'm inside the nodes captured by the css-query or js-query.
> >>
> >> Some additional context: the reason I'm looking into tree-sitter (and
> its functionalities to support multiple
> >> languages) is to potentially use it to fontify markdown code blocks and
> to improve emacs support for python
> >> notebooks. For markdown, I was trying a similar approach to the HTML
> one described in the manual, but ran
> >> into other similar issues:
> >>
> https://www.reddit.com/r/emacs/comments/1gcrv8k/syntaxhighlighting_codeblocks_in_markdown/
> .
> >> I'm just including this as context.
> >>
> >> Let me know if any of this is not clear.
> >>
> >> Thanks in advance for all your help!
> >
> > Yuan, can you help Andrew?
>
> Ah yes, thanks for the ping. Andrew, I take that your problem is with
> treesit-language-at, right? Specifically, it doesn’t return expected
> results. That’s because for treesit-language-at to work, major mode needs
> to define treesit-language-at-function.
>
> This confusion has came up a couple times now, evidently
> treesit-language-at is not very intuitive. Hopefully it’ll be fixed by our
> updated manual for Emacs 30. In Emacs 30, we define
> treesit-language-at-function in the example code:
>
>    Emacs automates this process in ‘treesit-update-ranges’.  A
> multi-language major mode should set ‘treesit-range-settings’ so that
> ‘treesit-update-ranges’ knows how to perform this process automatically.
> Major modes should use the helper function ‘treesit-range-rules’ to
> generate a value that can be assigned to ‘treesit-range-settings’.  The
> settings in the following example directly translate into operations
> shown above.
>
>      (setq treesit-range-settings
>            (treesit-range-rules
>             :embed 'javascript
>             :host 'html
>             '((script_element (raw_text) @capture))
>             :embed 'css
>             :host 'html
>             '((style_element (raw_text) @capture))))
>
>      ;; Major modes with multiple languages should always set
>      ;; `treesit-language-at-point-function' (which see).
>      (setq treesit-language-at-point-function
>            (lambda (pos)
>              (let* ((node (treesit-node-at pos 'html))
>                     (parent (treesit-node-parent node)))
>                (cond
>                 ((and node parent
>                       (equal (treesit-node-type node) "raw_text")
>                       (equal (treesit-node-type parent) "script_element"))
>                  'javascript)
>                 ((and node parent
>                       (equal (treesit-node-type node) "raw_text")
>                       (equal (treesit-node-type parent) "style_element"))
>                  'css)
>                 (t 'html)))))
>
> And FYI, in Emacs 30 we added local parsers, that might make implementing
> code/markdown blocks in a notebook easier.
>
> Yuan

[-- Attachment #2: Type: text/html, Size: 7221 bytes --]

Re: treesit: how to get it to parse multiple languages

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]

On Sun, 10 Nov 2024, Andrew De Angelis wrote:

> a major mode out
> there that already implements multiple-language functionalities using
> treesitter.
>
> …
>
> Do you all know if such a mode exists?

In Emacs 30, php-ts-mode?

https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/php-ts-mode.el

-- 
Peter Oliver

Re: treesit: how to get it to parse multiple languages

[Juri Linkov]

>> a major mode out
>> there that already implements multiple-language functionalities using
>> treesitter.
>>
>> …
>>
>> Do you all know if such a mode exists?
>
> In Emacs 30, php-ts-mode?
>
> https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/php-ts-mode.el

It would be nice if html-ts-mode supported embedded css-ts-mode and js-ts-mode
the same way.

Re: treesit: how to get it to parse multiple languages

[Yuan Fu]

> On Nov 11, 2024, at 11:28 AM, Juri Linkov <juri@linkov.net> wrote:
> 
>>> a major mode out
>>> there that already implements multiple-language functionalities using
>>> treesitter.
>>> 
>>> …
>>> 
>>> Do you all know if such a mode exists?
>> 
>> In Emacs 30, php-ts-mode?
>> 
>> https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/php-ts-mode.el
> 
> It would be nice if html-ts-mode supported embedded css-ts-mode and js-ts-mode
> the same way.

Yeah, it can be a good poster child for writing multi-lang modes. My time is very limited now, so anybody sees this, feel free to give it a try. And maybe document the code extensively too ;-)

Yuan

Re: treesit: how to get it to parse multiple languages

[Vincenzo Pupillo]

In data martedì 19 novembre 2024 08:01:28 Ora standard dell’Europa centrale, 
Yuan Fu ha scritto:
> > On Nov 11, 2024, at 11:28 AM, Juri Linkov <juri@linkov.net> wrote:
> >>> a major mode out
> >>> there that already implements multiple-language functionalities using
> >>> treesitter.
> >>> 
> >>> …
> >>> 
> >>> Do you all know if such a mode exists?
> >> 
> >> In Emacs 30, php-ts-mode?
> >> 
> >> https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/php-ts-mo
> >> de.el> 
> > It would be nice if html-ts-mode supported embedded css-ts-mode and
> > js-ts-mode the same way.
> 
> Yeah, it can be a good poster child for writing multi-lang modes. My time is
> very limited now, so anybody sees this, feel free to give it a try. And
> maybe document the code extensively too ;-)

It's almost done (it's called mhtml-ts-mode), I hope to submit a patch within 
the next week.

Vincenzo

Re: treesit: how to get it to parse multiple languages

[Yuan Fu]

> On Nov 19, 2024, at 12:13 PM, Vincenzo Pupillo <v.pupillo@gmail.com> wrote:
> 
> In data martedì 19 novembre 2024 08:01:28 Ora standard dell’Europa centrale, 
> Yuan Fu ha scritto:
>>> On Nov 11, 2024, at 11:28 AM, Juri Linkov <juri@linkov.net> wrote:
>>>>> a major mode out
>>>>> there that already implements multiple-language functionalities using
>>>>> treesitter.
>>>>> 
>>>>> …
>>>>> 
>>>>> Do you all know if such a mode exists?
>>>> 
>>>> In Emacs 30, php-ts-mode?
>>>> 
>>>> https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/php-ts-mo
>>>> de.el> 
>>> It would be nice if html-ts-mode supported embedded css-ts-mode and
>>> js-ts-mode the same way.
>> 
>> Yeah, it can be a good poster child for writing multi-lang modes. My time is
>> very limited now, so anybody sees this, feel free to give it a try. And
>> maybe document the code extensively too ;-)
> 
> It's almost done (it's called mhtml-ts-mode), I hope to submit a patch within 
> the next week.
> 
> Vincenzo

That’s awesome!

Yuan 

Re: treesit: how to get it to parse multiple languages

[Vincenzo Pupillo]

In data domenica 24 novembre 2024 06:09:39 Ora standard dell’Europa centrale, 
Yuan Fu ha scritto:
> 
> That’s awesome!
> 
> Yuan

It's pretty much ready, the only thing I'm missing is the ability to create an 
Imenu with symbols from both HTML and the other embedded languages. I have 
tried a few approaches but it seems that it can only be done for the main 
language. Is there any way to do something like that?

Thank you.

Vincenzo

Re: treesit: how to get it to parse multiple languages

[Juri Linkov]

>> That’s awesome!
>
> It's pretty much ready, the only thing I'm missing is the ability to create an 
> Imenu with symbols from both HTML and the other embedded languages. I have 
> tried a few approaches but it seems that it can only be done for the main 
> language. Is there any way to do something like that?

Imenu is a small detail that could be figured out after pushing your main work.

Re: treesit: how to get it to parse multiple languages

[Yuan Fu]

> On Nov 28, 2024, at 11:22 PM, Juri Linkov <juri@linkov.net> wrote:
> 
>>> That’s awesome!
>> 
>> It's pretty much ready, the only thing I'm missing is the ability to create an 
>> Imenu with symbols from both HTML and the other embedded languages. I have 
>> tried a few approaches but it seems that it can only be done for the main 
>> language. Is there any way to do something like that?
> 
> Imenu is a small detail that could be figured out after pushing your main work.

Oops, sorry for the late reply. Lets’ do this: I’ll extend treesit-simple-imenu to support multi-language. Once we have that you can add imenu support for your new mode.

Yuan

Re: treesit: how to get it to parse multiple languages

[Vincenzo Pupillo]

In data venerdì 29 novembre 2024 08:37:28 Ora standard dell’Europa centrale, 
Yuan Fu ha scritto:
> > On Nov 28, 2024, at 11:22 PM, Juri Linkov <juri@linkov.net> wrote:
> >>> That’s awesome!
> >> 
> >> It's pretty much ready, the only thing I'm missing is the ability to
> >> create an Imenu with symbols from both HTML and the other embedded
> >> languages. I have tried a few approaches but it seems that it can only
> >> be done for the main language. Is there any way to do something like
> >> that?
> > 
> > Imenu is a small detail that could be figured out after pushing your main
> > work.
> Oops, sorry for the late reply. Lets’ do this: I’ll extend
> treesit-simple-imenu to support multi-language. Once we have that you can
> add imenu support for your new mode.
Thank you Yuan. 
I just submitted mhtml-ts-mode for inclusion.
Please take a look at bug #74610.
@Andrew: I tried to explain as best as I can how to write a major-mode that 
handles multiple languages, check it out.

Vincenzo

Re: treesit: how to get it to parse multiple languages

[Andrew De Angelis]

[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]

Thanks a lot Vincenzo! Excited to take a deeper look at it.
By the way, I did start working on a possible python-notebook mode, with
planned multi-language (markdown and python) support:
https://github.com/andrewdea/enb

Progress has been a bit slow due to other commitments: so far I only have a
rudimentary parser that 'renders' the JSON into a markdown/python buffer.
But I'm hoping to add more functionalities over the coming months.

Thank you all again for all the help on this.

On Fri, Nov 29, 2024 at 5:13 PM Vincenzo Pupillo <v.pupillo@gmail.com>
wrote:

> In data venerdì 29 novembre 2024 08:37:28 Ora standard dell’Europa
> centrale,
> Yuan Fu ha scritto:
> > > On Nov 28, 2024, at 11:22 PM, Juri Linkov <juri@linkov.net> wrote:
> > >>> That’s awesome!
> > >>
> > >> It's pretty much ready, the only thing I'm missing is the ability to
> > >> create an Imenu with symbols from both HTML and the other embedded
> > >> languages. I have tried a few approaches but it seems that it can only
> > >> be done for the main language. Is there any way to do something like
> > >> that?
> > >
> > > Imenu is a small detail that could be figured out after pushing your
> main
> > > work.
> > Oops, sorry for the late reply. Lets’ do this: I’ll extend
> > treesit-simple-imenu to support multi-language. Once we have that you can
> > add imenu support for your new mode.
> Thank you Yuan.
> I just submitted mhtml-ts-mode for inclusion.
> Please take a look at bug #74610.
> @Andrew: I tried to explain as best as I can how to write a major-mode
> that
> handles multiple languages, check it out.
>
> Vincenzo
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 2306 bytes --]

Re: treesit: how to get it to parse multiple languages

[Yuan Fu]

> On Nov 10, 2024, at 6:35 AM, Andrew De Angelis <bobodeangelis@gmail.com> wrote:
> 
> Thanks so much!
> 
> I took a look at the Emacs 30 manual and it's a lot clearer, it's perfect!
> I think one thing that would truly be ideal is if there is a major mode out there that already implements multiple-language functionalities using treesitter. Seeing all the components in action would be quite helpful: the simple HTML examples are very clarifying but they can only do so much.
> 
> Do you all know if such a mode exists? `(ripgrep-regexp "local-parser" source-directory)` on the master branch only shows me matches in `treesit.el` itself (and associated ChangeLog / manual).
> If it doesn't exist yet I'm happy to give it a knack when implementing the notebook mode. I might have run into some more questions then :)

As Peter has pointed out, php-ts-mode is a multi-language mode. And elixir-ts-mode too. You can grep for “:local”, or "treesit-range-rules”.

Yuan