Validating tree-sitter major modes and grammar

Validating tree-sitter major modes and grammar

[Yuan Fu]

Continuing from the tree-sitter maternity thread, I cooked up some script to go over each builtin tree-sitter mode, clone the grammars it uses, and check whether the font-lock queries are compatible with the latest version of the grammar. If everything works fine, the script adds some comment in the source file listing the version that was checked.

For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:

;;; Tree-sitter language versions
;;
;; c-ts-mode is known to work with the following languages and version:
;; - tree-sitter-c: v0.20.8-61-g3efee11
;;
;; c++-ts-mode is known to work with the following languages and version:
;; - tree-sitter-cpp: v0.20.5-49-gf41b4f6
;;
;; We try our best to make builtin modes work with latest grammar
;; versions, so a more recent grammar version has a good chance to work.
;; Send us a bug report if it doesn't.

The version is taken from “git describe”. If the grammar isn’t versioned, it’ll just be the hash of HEAD. In order for “git describe” to work, I had to do full clones, and some tree-sitter grammar’s repo is very slow to clone. I wonder if there’s anything we can do to speed it up? 

Any thoughts/comments? I pushed the code to master so others can play with it.

BTW, the script revealed that cmake-ts-mode isn’t compatible with the latest grammar. I’ll try fix it later.

Yuan

Re: Validating tree-sitter major modes and grammar

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Fri, 20 Dec 2024 01:09:10 -0800
> Cc: Peter Oliver <p.d.oliver@mavit.org.uk>,
>  Björn Bidar <bjorn.bidar@thaodan.de>,
>  Stefan Kangas <stefankangas@gmail.com>
> 
> Continuing from the tree-sitter maternity thread, I cooked up some script to go over each builtin tree-sitter mode, clone the grammars it uses, and check whether the font-lock queries are compatible with the latest version of the grammar. If everything works fine, the script adds some comment in the source file listing the version that was checked.
> 
> For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:

I'm not sure what exactly you are suggesting, but commands and Lisp
programs that are for the Emacs maintainers (as opposed to users)
should be in the admin/ directory, not in the lisp/ directory.

If you think this should be used by Emacs users, please describe how
would a user use such a command, because I don't think I understand
that.

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 20, 2024, at 4:24 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Fri, 20 Dec 2024 01:09:10 -0800
>> Cc: Peter Oliver <p.d.oliver@mavit.org.uk>,
>> Björn Bidar <bjorn.bidar@thaodan.de>,
>> Stefan Kangas <stefankangas@gmail.com>
>> 
>> Continuing from the tree-sitter maternity thread, I cooked up some script to go over each builtin tree-sitter mode, clone the grammars it uses, and check whether the font-lock queries are compatible with the latest version of the grammar. If everything works fine, the script adds some comment in the source file listing the version that was checked.
>> 
>> For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:
> 
> I'm not sure what exactly you are suggesting, but commands and Lisp
> programs that are for the Emacs maintainers (as opposed to users)
> should be in the admin/ directory, not in the lisp/ directory.
> 
> If you think this should be used by Emacs users, please describe how
> would a user use such a command, because I don't think I understand
> that.

It’s for maintainers, I moved the script to admin/treesit-admin.el.

Yuan

Re: Validating tree-sitter major modes and grammar

[Björn Bidar]

Yuan Fu <casouri@gmail.com> writes:

> Continuing from the tree-sitter maternity thread, I cooked up some
> script to go over each builtin tree-sitter mode, clone the grammars it
> uses, and check whether the font-lock queries are compatible with the
> latest version of the grammar. If everything works fine, the script
> adds some comment in the source file listing the version that was
> checked.

Oh that sounds very good. This could be a good workaround for the sort
of fast and lose nature of tree-sitter.

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 23, 2024, at 3:41 AM, Arsen Arsenović <arsen@aarsen.me> wrote:
> 
> Hi Yuan,
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>> Continuing from the tree-sitter maternity thread, I cooked up some script to go
>> over each builtin tree-sitter mode, clone the grammars it uses, and check
>> whether the font-lock queries are compatible with the latest version of the
>> grammar. If everything works fine, the script adds some comment in the source
>> file listing the version that was checked.
>> 
>> For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:
>> 
>> ;;; Tree-sitter language versions
>> ;;
>> ;; c-ts-mode is known to work with the following languages and version:
>> ;; - tree-sitter-c: v0.20.8-61-g3efee11
>> ;;
>> ;; c++-ts-mode is known to work with the following languages and version:
>> ;; - tree-sitter-cpp: v0.20.5-49-gf41b4f6
>> ;;
>> ;; We try our best to make builtin modes work with latest grammar
>> ;; versions, so a more recent grammar version has a good chance to work.
>> ;; Send us a bug report if it doesn't.
>> 
>> The version is taken from “git describe”. If the grammar isn’t versioned, it’ll
>> just be the hash of HEAD. In order for “git describe” to work, I had to do full
>> clones, and some tree-sitter grammar’s repo is very slow to clone. I wonder if
>> there’s anything we can do to speed it up?
> 
> Maybe this:
> 
>  ~$ time git clone --filter=blob:none https://github.com/tree-sitter/tree-sitter-c/
>  [...]
> 
>  real 0m1.380s
>  user 0m0.104s
>  sys 0m0.069s
>  ~$ cd tree-sitter-c
>  ~/tree-sitter-c$ git describe HEAD
>  v0.20.8-61-g3efee11
> 
> A normal clone was 7.2 seconds, for reference.  Note that all future
> checkouts will be slowed down (as they will need to fetch fresh blobs),
> but if you're only cloning for HEAD anyway, that's fine.
> -- 
> Arsen Arsenović

Thanks! This is exactly what I need.

Yuan

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 20, 2024, at 1:09 AM, Yuan Fu <casouri@gmail.com> wrote:
> 
> Continuing from the tree-sitter maternity thread, I cooked up some script to go over each builtin tree-sitter mode, clone the grammars it uses, and check whether the font-lock queries are compatible with the latest version of the grammar. If everything works fine, the script adds some comment in the source file listing the version that was checked.
> 
> For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:
> 
> ;;; Tree-sitter language versions
> ;;
> ;; c-ts-mode is known to work with the following languages and version:
> ;; - tree-sitter-c: v0.23.4
> ;;
> ;; c++-ts-mode is known to work with the following languages and version:
> ;; - tree-sitter-cpp: v0.23.4-1-gf41b4f6
> ;;
> ;; We try our best to make builtin modes work with latest grammar
> ;; versions, so a more recent grammar version has a good chance to work.
> ;; Send us a bug report if it doesn't.
> 
> The version is taken from “git describe”. If the grammar isn’t versioned, it’ll just be the hash of HEAD. In order for “git describe” to work, I had to do full clones, and some tree-sitter grammar’s repo is very slow to clone. I wonder if there’s anything we can do to speed it up? 
> 
> Any thoughts/comments? I pushed the code to master so others can play with it.
> 
> BTW, the script revealed that cmake-ts-mode isn’t compatible with the latest grammar. I’ll try fix it later.

(This is now fixed.)



CC’ing maintainers for builtin modes for comments. Also, Eli, what’s your thoughts on this after the clarification? 

Yuan

Re: Validating tree-sitter major modes and grammar

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 23 Dec 2024 19:12:18 -0800
> Cc: Peter Oliver <p.d.oliver@mavit.org.uk>,
>  Björn Bidar <bjorn.bidar@thaodan.de>,
>  Stefan Kangas <stefankangas@gmail.com>,
>  Perry Smith <pedz@easesoftware.com>, Perry Smith <pedz@easesoftware.com>,
>  Jostein Kjønigsen <jostein@secure.kjonigsen.net>,
>  Theodor Thornhill <theo@thornhill.no>,
>  Vincenzo Pupillo <v.pupillo@gmail.com>, Randy Taylor <dev@rjt.dev>,
>  Wilhelm Kirschbaum <wkirschbaum@gmail.com>, john muhl <jm@pub.pink>,
>  Dmitry Gutov <dgutov@yandex.ru>, john muhl <jm@pub.pink>,
>  Juri Linkov <juri@linkov.net>
> 
> 
> 
> > On Dec 20, 2024, at 1:09 AM, Yuan Fu <casouri@gmail.com> wrote:
> > 
> > Continuing from the tree-sitter maternity thread, I cooked up some script to go over each builtin tree-sitter mode, clone the grammars it uses, and check whether the font-lock queries are compatible with the latest version of the grammar. If everything works fine, the script adds some comment in the source file listing the version that was checked.
> > 
> > For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:
> > 
> > ;;; Tree-sitter language versions
> > ;;
> > ;; c-ts-mode is known to work with the following languages and version:
> > ;; - tree-sitter-c: v0.23.4
> > ;;
> > ;; c++-ts-mode is known to work with the following languages and version:
> > ;; - tree-sitter-cpp: v0.23.4-1-gf41b4f6
> > ;;
> > ;; We try our best to make builtin modes work with latest grammar
> > ;; versions, so a more recent grammar version has a good chance to work.
> > ;; Send us a bug report if it doesn't.
> > 
> > The version is taken from “git describe”. If the grammar isn’t versioned, it’ll just be the hash of HEAD. In order for “git describe” to work, I had to do full clones, and some tree-sitter grammar’s repo is very slow to clone. I wonder if there’s anything we can do to speed it up? 
> > 
> > Any thoughts/comments? I pushed the code to master so others can play with it.
> > 
> > BTW, the script revealed that cmake-ts-mode isn’t compatible with the latest grammar. I’ll try fix it later.
> 
> (This is now fixed.)
> 
> 
> 
> CC’ing maintainers for builtin modes for comments. Also, Eli, what’s your thoughts on this after the clarification? 

Sorry, I thought we had this discussion concluded and closed.  What
are the issues that still need decisions?

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 24, 2024, at 4:19 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Mon, 23 Dec 2024 19:12:18 -0800
>> Cc: Peter Oliver <p.d.oliver@mavit.org.uk>,
>> Björn Bidar <bjorn.bidar@thaodan.de>,
>> Stefan Kangas <stefankangas@gmail.com>,
>> Perry Smith <pedz@easesoftware.com>, Perry Smith <pedz@easesoftware.com>,
>> Jostein Kjønigsen <jostein@secure.kjonigsen.net>,
>> Theodor Thornhill <theo@thornhill.no>,
>> Vincenzo Pupillo <v.pupillo@gmail.com>, Randy Taylor <dev@rjt.dev>,
>> Wilhelm Kirschbaum <wkirschbaum@gmail.com>, john muhl <jm@pub.pink>,
>> Dmitry Gutov <dgutov@yandex.ru>, john muhl <jm@pub.pink>,
>> Juri Linkov <juri@linkov.net>
>> 
>> 
>> 
>>> On Dec 20, 2024, at 1:09 AM, Yuan Fu <casouri@gmail.com> wrote:
>>> 
>>> Continuing from the tree-sitter maternity thread, I cooked up some script to go over each builtin tree-sitter mode, clone the grammars it uses, and check whether the font-lock queries are compatible with the latest version of the grammar. If everything works fine, the script adds some comment in the source file listing the version that was checked.
>>> 
>>> For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:
>>> 
>>> ;;; Tree-sitter language versions
>>> ;;
>>> ;; c-ts-mode is known to work with the following languages and version:
>>> ;; - tree-sitter-c: v0.23.4
>>> ;;
>>> ;; c++-ts-mode is known to work with the following languages and version:
>>> ;; - tree-sitter-cpp: v0.23.4-1-gf41b4f6
>>> ;;
>>> ;; We try our best to make builtin modes work with latest grammar
>>> ;; versions, so a more recent grammar version has a good chance to work.
>>> ;; Send us a bug report if it doesn't.
>>> 
>>> The version is taken from “git describe”. If the grammar isn’t versioned, it’ll just be the hash of HEAD. In order for “git describe” to work, I had to do full clones, and some tree-sitter grammar’s repo is very slow to clone. I wonder if there’s anything we can do to speed it up? 
>>> 
>>> Any thoughts/comments? I pushed the code to master so others can play with it.
>>> 
>>> BTW, the script revealed that cmake-ts-mode isn’t compatible with the latest grammar. I’ll try fix it later.
>> 
>> (This is now fixed.)
>> 
>> 
>> 
>> CC’ing maintainers for builtin modes for comments. Also, Eli, what’s your thoughts on this after the clarification? 
> 
> Sorry, I thought we had this discussion concluded and closed.  What
> are the issues that still need decisions?

About running the script semi-regularly and adding recommended version comments to builtin modes. Are there any concerns?

If most people think it’s a good idea, I’ll run the script for all the builtin modes, which will add the comments. And I plan to run it myself from time to time to update the comments. Maintainers for a particular mode can also run it to update the comment themselves. And I’ll add more detailed documentation about how to use the script.

Yuan

Re: Validating tree-sitter major modes and grammar

[Dmitry Gutov]

On 24/12/2024 23:30, Yuan Fu wrote:
> If most people think it’s a good idea, I’ll run the script for all the builtin modes, which will add the comments. And I plan to run it myself from time to time to update the comments. Maintainers for a particular mode can also run it to update the comment themselves. And I’ll add more detailed documentation about how to use the script.

Sounds great to me.

Hopefully we manage to convey well that the "last tested version" is not 
necessarily the latest that will work, just the one that we tested.

(I dropped most people from Cc to avoid spamming.)

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 24, 2024, at 3:09 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> On 24/12/2024 23:30, Yuan Fu wrote:
>> If most people think it’s a good idea, I’ll run the script for all the builtin modes, which will add the comments. And I plan to run it myself from time to time to update the comments. Maintainers for a particular mode can also run it to update the comment themselves. And I’ll add more detailed documentation about how to use the script.
> 
> Sounds great to me.
> 
> Hopefully we manage to convey well that the "last tested version" is not necessarily the latest that will work, just the one that we tested.

Right now the comment says this at the end:

;; We try our best to make builtin modes work with latest grammar
;; versions, so a more recent grammar version has a good chance to work.
;; Send us a bug report if it doesn't.

But let me know if it can be improved in some way.

Yuan

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 22, 2024, at 4:44 PM, Björn Bidar <bjorn.bidar@thaodan.de> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>> Continuing from the tree-sitter maternity thread, I cooked up some
>> script to go over each builtin tree-sitter mode, clone the grammars it
>> uses, and check whether the font-lock queries are compatible with the
>> latest version of the grammar. If everything works fine, the script
>> adds some comment in the source file listing the version that was
>> checked.
> 
> Oh that sounds very good. This could be a good workaround for the sort
> of fast and lose nature of tree-sitter.
> 

Great. Do you think the “verified version” comment will be helpful for packagers?

Yuan

Re: Validating tree-sitter major modes and grammar

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 1664 bytes --]

On Mon, 23 Dec 2024, Yuan Fu wrote:

>> On Dec 22, 2024, at 4:44 PM, Björn Bidar <bjorn.bidar@thaodan.de> wrote:
>>
>> Yuan Fu <casouri@gmail.com> writes:
>>
>>> Continuing from the tree-sitter maternity thread, I cooked up some
>>> script to go over each builtin tree-sitter mode, clone the grammars it
>>> uses, and check whether the font-lock queries are compatible with the
>>> latest version of the grammar. If everything works fine, the script
>>> adds some comment in the source file listing the version that was
>>> checked.
>>
>> Oh that sounds very good. This could be a good workaround for the sort
>> of fast and lose nature of tree-sitter.
>
> Great. Do you think the “verified version” comment will be helpful for packagers?

It helps answer the question, “I am building a new Emacs release.  Do I need to update these Tree-sitter parsers?”.

However, as a parser packager, I also need to answer the question, “This parser has released a new version.  Can I package it now, or do I need to wait for Emacs 30?  Emacs 31?”.  A static comment in the source code of a released version of Emacs will likely be too out-of-date to answer that.

Other ideas:

- Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.

- Include a function in Emacs that packagers can call themselves to check the compatibility of installed parsers.  They could run this check every time they build an updated parser (either manually, as a part of the package build, or as a part of their distribution’s automated CI testing).

-- 
Peter Oliver

Re: Validating tree-sitter major modes and grammar

[Michael Albinus]

Peter Oliver <p.d.oliver@mavit.org.uk> writes:

Hi Peter,

> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.

This is performed already, see the GitLab jobs build-image-tree-sitter
and test-tree-sitter.

Recent results are for example
<https://emba.gnu.org/emacs/emacs/-/jobs/95076> and
<https://emba.gnu.org/emacs/emacs/-/jobs/95071>.

Best regards, Michael.

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 24, 2024, at 6:20 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> Peter Oliver <p.d.oliver@mavit.org.uk> writes:
> 
> Hi Peter,
> 
>> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.
> 
> This is performed already, see the GitLab jobs build-image-tree-sitter
> and test-tree-sitter.
> 
> Recent results are for example
> <https://emba.gnu.org/emacs/emacs/-/jobs/95076> and
> <https://emba.gnu.org/emacs/emacs/-/jobs/95071>.
> 
> Best regards, Michael.

Ah yes, though this is more for us (Emacs devs) than for packagers.

Yuan

Re: Validating tree-sitter major modes and grammar

[Michael Albinus]

Yuan Fu <casouri@gmail.com> writes:

Hi Yuan & Peter,

>>> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.
>>
>> This is performed already, see the GitLab jobs build-image-tree-sitter
>> and test-tree-sitter.
>
> Ah yes, though this is more for us (Emacs devs) than for packagers.

Sure. But if this can be improved for packagers, let us know.

> Yuan

Best regards, Michael.

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 25, 2024, at 12:20 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
> Hi Yuan & Peter,
> 
>>>> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.
>>> 
>>> This is performed already, see the GitLab jobs build-image-tree-sitter
>>> and test-tree-sitter.
>> 
>> Ah yes, though this is more for us (Emacs devs) than for packagers.
> 
> Sure. But if this can be improved for packagers, let us know.

Is it possible for EMBA to run a CI which produces a HTML webpage, and host that webpage somewhere? I’m referring to this idea of Peter’s. 

> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.

I’m thinking of a table where rows are languages, columns are Emacs releases (29, 30, master), and cells are the latest revision of a language grammar that’s compatible with an Emacs release.

Yuan

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 24, 2024, at 4:36 AM, Peter Oliver <p.d.oliver@mavit.org.uk> wrote:
> 
> On Mon, 23 Dec 2024, Yuan Fu wrote:
> 
>>> On Dec 22, 2024, at 4:44 PM, Björn Bidar <bjorn.bidar@thaodan.de> wrote:
>>> 
>>> Yuan Fu <casouri@gmail.com> writes:
>>> 
>>>> Continuing from the tree-sitter maternity thread, I cooked up some
>>>> script to go over each builtin tree-sitter mode, clone the grammars it
>>>> uses, and check whether the font-lock queries are compatible with the
>>>> latest version of the grammar. If everything works fine, the script
>>>> adds some comment in the source file listing the version that was
>>>> checked.
>>> 
>>> Oh that sounds very good. This could be a good workaround for the sort
>>> of fast and lose nature of tree-sitter.
>> 
>> Great. Do you think the “verified version” comment will be helpful for packagers?
> 
> It helps answer the question, “I am building a new Emacs release.  Do I need to update these Tree-sitter parsers?”.
> 
> However, as a parser packager, I also need to answer the question, “This parser has released a new version.  Can I package it now, or do I need to wait for Emacs 30?  Emacs 31?”.  A static comment in the source code of a released version of Emacs will likely be too out-of-date to answer that.
> 
> Other ideas:
> 
> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.
> 
> - Include a function in Emacs that packagers can call themselves to check the compatibility of installed parsers.  They could run this check every time they build an updated parser (either manually, as a part of the package build, or as a part of their distribution’s automated CI testing).

I can add another script that finds the latest “version” that’s compatible with a builtin mode. The comments will be helpful for packaging a new Emacs release. And the new script can help with packaging new grammar versions with Emacs.

Yuan

Re: Validating tree-sitter major modes and grammar

[Björn Bidar]

Yuan Fu <casouri@gmail.com> writes:

>> On Dec 22, 2024, at 4:44 PM, Björn Bidar <bjorn.bidar@thaodan.de> wrote:
>> 
>> Yuan Fu <casouri@gmail.com> writes:
>> 
>>> Continuing from the tree-sitter maternity thread, I cooked up some
>>> script to go over each builtin tree-sitter mode, clone the grammars it
>>> uses, and check whether the font-lock queries are compatible with the
>>> latest version of the grammar. If everything works fine, the script
>>> adds some comment in the source file listing the version that was
>>> checked.
>> 
>> Oh that sounds very good. This could be a good workaround for the sort
>> of fast and lose nature of tree-sitter.
>> 
>
> Great. Do you think the “verified version” comment will be helpful for packagers?
>

It would be helpful in case something goes wrong as a kind of sanity
check but it won't help much for an indicator when to update since I
don't think that would be blocker for updating.

It is helpful but none the less the goal should be that such errors
should be preventable between grammar versions.