Validating tree-sitter major modes and grammar

Validating tree-sitter major modes and grammar

[Yuan Fu]

Continuing from the tree-sitter maternity thread, I cooked up some script to go over each builtin tree-sitter mode, clone the grammars it uses, and check whether the font-lock queries are compatible with the latest version of the grammar. If everything works fine, the script adds some comment in the source file listing the version that was checked.

For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:

;;; Tree-sitter language versions
;;
;; c-ts-mode is known to work with the following languages and version:
;; - tree-sitter-c: v0.20.8-61-g3efee11
;;
;; c++-ts-mode is known to work with the following languages and version:
;; - tree-sitter-cpp: v0.20.5-49-gf41b4f6
;;
;; We try our best to make builtin modes work with latest grammar
;; versions, so a more recent grammar version has a good chance to work.
;; Send us a bug report if it doesn't.

The version is taken from “git describe”. If the grammar isn’t versioned, it’ll just be the hash of HEAD. In order for “git describe” to work, I had to do full clones, and some tree-sitter grammar’s repo is very slow to clone. I wonder if there’s anything we can do to speed it up? 

Any thoughts/comments? I pushed the code to master so others can play with it.

BTW, the script revealed that cmake-ts-mode isn’t compatible with the latest grammar. I’ll try fix it later.

Yuan

Re: Validating tree-sitter major modes and grammar

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Fri, 20 Dec 2024 01:09:10 -0800
> Cc: Peter Oliver <p.d.oliver@mavit.org.uk>,
>  Björn Bidar <bjorn.bidar@thaodan.de>,
>  Stefan Kangas <stefankangas@gmail.com>
> 
> Continuing from the tree-sitter maternity thread, I cooked up some script to go over each builtin tree-sitter mode, clone the grammars it uses, and check whether the font-lock queries are compatible with the latest version of the grammar. If everything works fine, the script adds some comment in the source file listing the version that was checked.
> 
> For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:

I'm not sure what exactly you are suggesting, but commands and Lisp
programs that are for the Emacs maintainers (as opposed to users)
should be in the admin/ directory, not in the lisp/ directory.

If you think this should be used by Emacs users, please describe how
would a user use such a command, because I don't think I understand
that.

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 20, 2024, at 4:24 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Fri, 20 Dec 2024 01:09:10 -0800
>> Cc: Peter Oliver <p.d.oliver@mavit.org.uk>,
>> Björn Bidar <bjorn.bidar@thaodan.de>,
>> Stefan Kangas <stefankangas@gmail.com>
>> 
>> Continuing from the tree-sitter maternity thread, I cooked up some script to go over each builtin tree-sitter mode, clone the grammars it uses, and check whether the font-lock queries are compatible with the latest version of the grammar. If everything works fine, the script adds some comment in the source file listing the version that was checked.
>> 
>> For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:
> 
> I'm not sure what exactly you are suggesting, but commands and Lisp
> programs that are for the Emacs maintainers (as opposed to users)
> should be in the admin/ directory, not in the lisp/ directory.
> 
> If you think this should be used by Emacs users, please describe how
> would a user use such a command, because I don't think I understand
> that.

It’s for maintainers, I moved the script to admin/treesit-admin.el.

Yuan

Re: Validating tree-sitter major modes and grammar

[Björn Bidar]

Yuan Fu <casouri@gmail.com> writes:

> Continuing from the tree-sitter maternity thread, I cooked up some
> script to go over each builtin tree-sitter mode, clone the grammars it
> uses, and check whether the font-lock queries are compatible with the
> latest version of the grammar. If everything works fine, the script
> adds some comment in the source file listing the version that was
> checked.

Oh that sounds very good. This could be a good workaround for the sort
of fast and lose nature of tree-sitter.

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 23, 2024, at 3:41 AM, Arsen Arsenović <arsen@aarsen.me> wrote:
> 
> Hi Yuan,
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>> Continuing from the tree-sitter maternity thread, I cooked up some script to go
>> over each builtin tree-sitter mode, clone the grammars it uses, and check
>> whether the font-lock queries are compatible with the latest version of the
>> grammar. If everything works fine, the script adds some comment in the source
>> file listing the version that was checked.
>> 
>> For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:
>> 
>> ;;; Tree-sitter language versions
>> ;;
>> ;; c-ts-mode is known to work with the following languages and version:
>> ;; - tree-sitter-c: v0.20.8-61-g3efee11
>> ;;
>> ;; c++-ts-mode is known to work with the following languages and version:
>> ;; - tree-sitter-cpp: v0.20.5-49-gf41b4f6
>> ;;
>> ;; We try our best to make builtin modes work with latest grammar
>> ;; versions, so a more recent grammar version has a good chance to work.
>> ;; Send us a bug report if it doesn't.
>> 
>> The version is taken from “git describe”. If the grammar isn’t versioned, it’ll
>> just be the hash of HEAD. In order for “git describe” to work, I had to do full
>> clones, and some tree-sitter grammar’s repo is very slow to clone. I wonder if
>> there’s anything we can do to speed it up?
> 
> Maybe this:
> 
>  ~$ time git clone --filter=blob:none https://github.com/tree-sitter/tree-sitter-c/
>  [...]
> 
>  real 0m1.380s
>  user 0m0.104s
>  sys 0m0.069s
>  ~$ cd tree-sitter-c
>  ~/tree-sitter-c$ git describe HEAD
>  v0.20.8-61-g3efee11
> 
> A normal clone was 7.2 seconds, for reference.  Note that all future
> checkouts will be slowed down (as they will need to fetch fresh blobs),
> but if you're only cloning for HEAD anyway, that's fine.
> -- 
> Arsen Arsenović

Thanks! This is exactly what I need.

Yuan

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 20, 2024, at 1:09 AM, Yuan Fu <casouri@gmail.com> wrote:
> 
> Continuing from the tree-sitter maternity thread, I cooked up some script to go over each builtin tree-sitter mode, clone the grammars it uses, and check whether the font-lock queries are compatible with the latest version of the grammar. If everything works fine, the script adds some comment in the source file listing the version that was checked.
> 
> For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:
> 
> ;;; Tree-sitter language versions
> ;;
> ;; c-ts-mode is known to work with the following languages and version:
> ;; - tree-sitter-c: v0.23.4
> ;;
> ;; c++-ts-mode is known to work with the following languages and version:
> ;; - tree-sitter-cpp: v0.23.4-1-gf41b4f6
> ;;
> ;; We try our best to make builtin modes work with latest grammar
> ;; versions, so a more recent grammar version has a good chance to work.
> ;; Send us a bug report if it doesn't.
> 
> The version is taken from “git describe”. If the grammar isn’t versioned, it’ll just be the hash of HEAD. In order for “git describe” to work, I had to do full clones, and some tree-sitter grammar’s repo is very slow to clone. I wonder if there’s anything we can do to speed it up? 
> 
> Any thoughts/comments? I pushed the code to master so others can play with it.
> 
> BTW, the script revealed that cmake-ts-mode isn’t compatible with the latest grammar. I’ll try fix it later.

(This is now fixed.)



CC’ing maintainers for builtin modes for comments. Also, Eli, what’s your thoughts on this after the clarification? 

Yuan

Re: Validating tree-sitter major modes and grammar

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 23 Dec 2024 19:12:18 -0800
> Cc: Peter Oliver <p.d.oliver@mavit.org.uk>,
>  Björn Bidar <bjorn.bidar@thaodan.de>,
>  Stefan Kangas <stefankangas@gmail.com>,
>  Perry Smith <pedz@easesoftware.com>, Perry Smith <pedz@easesoftware.com>,
>  Jostein Kjønigsen <jostein@secure.kjonigsen.net>,
>  Theodor Thornhill <theo@thornhill.no>,
>  Vincenzo Pupillo <v.pupillo@gmail.com>, Randy Taylor <dev@rjt.dev>,
>  Wilhelm Kirschbaum <wkirschbaum@gmail.com>, john muhl <jm@pub.pink>,
>  Dmitry Gutov <dgutov@yandex.ru>, john muhl <jm@pub.pink>,
>  Juri Linkov <juri@linkov.net>
> 
> 
> 
> > On Dec 20, 2024, at 1:09 AM, Yuan Fu <casouri@gmail.com> wrote:
> > 
> > Continuing from the tree-sitter maternity thread, I cooked up some script to go over each builtin tree-sitter mode, clone the grammars it uses, and check whether the font-lock queries are compatible with the latest version of the grammar. If everything works fine, the script adds some comment in the source file listing the version that was checked.
> > 
> > For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:
> > 
> > ;;; Tree-sitter language versions
> > ;;
> > ;; c-ts-mode is known to work with the following languages and version:
> > ;; - tree-sitter-c: v0.23.4
> > ;;
> > ;; c++-ts-mode is known to work with the following languages and version:
> > ;; - tree-sitter-cpp: v0.23.4-1-gf41b4f6
> > ;;
> > ;; We try our best to make builtin modes work with latest grammar
> > ;; versions, so a more recent grammar version has a good chance to work.
> > ;; Send us a bug report if it doesn't.
> > 
> > The version is taken from “git describe”. If the grammar isn’t versioned, it’ll just be the hash of HEAD. In order for “git describe” to work, I had to do full clones, and some tree-sitter grammar’s repo is very slow to clone. I wonder if there’s anything we can do to speed it up? 
> > 
> > Any thoughts/comments? I pushed the code to master so others can play with it.
> > 
> > BTW, the script revealed that cmake-ts-mode isn’t compatible with the latest grammar. I’ll try fix it later.
> 
> (This is now fixed.)
> 
> 
> 
> CC’ing maintainers for builtin modes for comments. Also, Eli, what’s your thoughts on this after the clarification? 

Sorry, I thought we had this discussion concluded and closed.  What
are the issues that still need decisions?

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 24, 2024, at 4:19 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Mon, 23 Dec 2024 19:12:18 -0800
>> Cc: Peter Oliver <p.d.oliver@mavit.org.uk>,
>> Björn Bidar <bjorn.bidar@thaodan.de>,
>> Stefan Kangas <stefankangas@gmail.com>,
>> Perry Smith <pedz@easesoftware.com>, Perry Smith <pedz@easesoftware.com>,
>> Jostein Kjønigsen <jostein@secure.kjonigsen.net>,
>> Theodor Thornhill <theo@thornhill.no>,
>> Vincenzo Pupillo <v.pupillo@gmail.com>, Randy Taylor <dev@rjt.dev>,
>> Wilhelm Kirschbaum <wkirschbaum@gmail.com>, john muhl <jm@pub.pink>,
>> Dmitry Gutov <dgutov@yandex.ru>, john muhl <jm@pub.pink>,
>> Juri Linkov <juri@linkov.net>
>> 
>> 
>> 
>>> On Dec 20, 2024, at 1:09 AM, Yuan Fu <casouri@gmail.com> wrote:
>>> 
>>> Continuing from the tree-sitter maternity thread, I cooked up some script to go over each builtin tree-sitter mode, clone the grammars it uses, and check whether the font-lock queries are compatible with the latest version of the grammar. If everything works fine, the script adds some comment in the source file listing the version that was checked.
>>> 
>>> For example, I ran the script for c-ts-mode and c++-ts-mode, and this is the comment inserted to c-ts-mode.el:
>>> 
>>> ;;; Tree-sitter language versions
>>> ;;
>>> ;; c-ts-mode is known to work with the following languages and version:
>>> ;; - tree-sitter-c: v0.23.4
>>> ;;
>>> ;; c++-ts-mode is known to work with the following languages and version:
>>> ;; - tree-sitter-cpp: v0.23.4-1-gf41b4f6
>>> ;;
>>> ;; We try our best to make builtin modes work with latest grammar
>>> ;; versions, so a more recent grammar version has a good chance to work.
>>> ;; Send us a bug report if it doesn't.
>>> 
>>> The version is taken from “git describe”. If the grammar isn’t versioned, it’ll just be the hash of HEAD. In order for “git describe” to work, I had to do full clones, and some tree-sitter grammar’s repo is very slow to clone. I wonder if there’s anything we can do to speed it up? 
>>> 
>>> Any thoughts/comments? I pushed the code to master so others can play with it.
>>> 
>>> BTW, the script revealed that cmake-ts-mode isn’t compatible with the latest grammar. I’ll try fix it later.
>> 
>> (This is now fixed.)
>> 
>> 
>> 
>> CC’ing maintainers for builtin modes for comments. Also, Eli, what’s your thoughts on this after the clarification? 
> 
> Sorry, I thought we had this discussion concluded and closed.  What
> are the issues that still need decisions?

About running the script semi-regularly and adding recommended version comments to builtin modes. Are there any concerns?

If most people think it’s a good idea, I’ll run the script for all the builtin modes, which will add the comments. And I plan to run it myself from time to time to update the comments. Maintainers for a particular mode can also run it to update the comment themselves. And I’ll add more detailed documentation about how to use the script.

Yuan

Re: Validating tree-sitter major modes and grammar

[Dmitry Gutov]

On 24/12/2024 23:30, Yuan Fu wrote:
> If most people think it’s a good idea, I’ll run the script for all the builtin modes, which will add the comments. And I plan to run it myself from time to time to update the comments. Maintainers for a particular mode can also run it to update the comment themselves. And I’ll add more detailed documentation about how to use the script.

Sounds great to me.

Hopefully we manage to convey well that the "last tested version" is not 
necessarily the latest that will work, just the one that we tested.

(I dropped most people from Cc to avoid spamming.)

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 24, 2024, at 3:09 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> On 24/12/2024 23:30, Yuan Fu wrote:
>> If most people think it’s a good idea, I’ll run the script for all the builtin modes, which will add the comments. And I plan to run it myself from time to time to update the comments. Maintainers for a particular mode can also run it to update the comment themselves. And I’ll add more detailed documentation about how to use the script.
> 
> Sounds great to me.
> 
> Hopefully we manage to convey well that the "last tested version" is not necessarily the latest that will work, just the one that we tested.

Right now the comment says this at the end:

;; We try our best to make builtin modes work with latest grammar
;; versions, so a more recent grammar version has a good chance to work.
;; Send us a bug report if it doesn't.

But let me know if it can be improved in some way.

Yuan

Re: Validating tree-sitter major modes and grammar

[Dmitry Gutov]

Hi Yuan,

On 25/12/2024 10:39, Yuan Fu wrote:
>> Hopefully we manage to convey well that the "last tested version" is not necessarily the latest that will work, just the one that we tested.
> Right now the comment says this at the end:
> 
> ;; We try our best to make builtin modes work with latest grammar
> ;; versions, so a more recent grammar version has a good chance to work.
> ;; Send us a bug report if it doesn't.
> 
> But let me know if it can be improved in some way.

Just a nit, but we could add "too" at the end:

   so a more recent grammar has a good chance to work too.

(I removed the word "version" for easier refilling, but that's 
definitely not neccesary.)

About the paragraph before, just a thought but do we want to use a 
standard phrase like "has been tested":

   ;; c-ts-mode has been tested with the following languages and version:

? Seems a bit more regular for a casual reader, which conveying the same 
meaning. I'm not a native speaker, though.

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 25, 2024, at 6:23 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> Hi Yuan,
> 
> On 25/12/2024 10:39, Yuan Fu wrote:
>>> Hopefully we manage to convey well that the "last tested version" is not necessarily the latest that will work, just the one that we tested.
>> Right now the comment says this at the end:
>> ;; We try our best to make builtin modes work with latest grammar
>> ;; versions, so a more recent grammar version has a good chance to work.
>> ;; Send us a bug report if it doesn't.
>> But let me know if it can be improved in some way.
> 
> Just a nit, but we could add "too" at the end:
> 
>  so a more recent grammar has a good chance to work too.
> 
> (I removed the word "version" for easier refilling, but that's definitely not neccesary.)
> 
> About the paragraph before, just a thought but do we want to use a standard phrase like "has been tested":
> 
>  ;; c-ts-mode has been tested with the following languages and version:
> 
> ? Seems a bit more regular for a casual reader, which conveying the same meaning. I'm not a native speaker, though.

Thanks, I applied these changes. Next time we run the scripts they’ll take effect.

Yuan

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 22, 2024, at 4:44 PM, Björn Bidar <bjorn.bidar@thaodan.de> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>> Continuing from the tree-sitter maternity thread, I cooked up some
>> script to go over each builtin tree-sitter mode, clone the grammars it
>> uses, and check whether the font-lock queries are compatible with the
>> latest version of the grammar. If everything works fine, the script
>> adds some comment in the source file listing the version that was
>> checked.
> 
> Oh that sounds very good. This could be a good workaround for the sort
> of fast and lose nature of tree-sitter.
> 

Great. Do you think the “verified version” comment will be helpful for packagers?

Yuan

Re: Validating tree-sitter major modes and grammar

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 1664 bytes --]

On Mon, 23 Dec 2024, Yuan Fu wrote:

>> On Dec 22, 2024, at 4:44 PM, Björn Bidar <bjorn.bidar@thaodan.de> wrote:
>>
>> Yuan Fu <casouri@gmail.com> writes:
>>
>>> Continuing from the tree-sitter maternity thread, I cooked up some
>>> script to go over each builtin tree-sitter mode, clone the grammars it
>>> uses, and check whether the font-lock queries are compatible with the
>>> latest version of the grammar. If everything works fine, the script
>>> adds some comment in the source file listing the version that was
>>> checked.
>>
>> Oh that sounds very good. This could be a good workaround for the sort
>> of fast and lose nature of tree-sitter.
>
> Great. Do you think the “verified version” comment will be helpful for packagers?

It helps answer the question, “I am building a new Emacs release.  Do I need to update these Tree-sitter parsers?”.

However, as a parser packager, I also need to answer the question, “This parser has released a new version.  Can I package it now, or do I need to wait for Emacs 30?  Emacs 31?”.  A static comment in the source code of a released version of Emacs will likely be too out-of-date to answer that.

Other ideas:

- Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.

- Include a function in Emacs that packagers can call themselves to check the compatibility of installed parsers.  They could run this check every time they build an updated parser (either manually, as a part of the package build, or as a part of their distribution’s automated CI testing).

-- 
Peter Oliver

Re: Validating tree-sitter major modes and grammar

[Michael Albinus]

Peter Oliver <p.d.oliver@mavit.org.uk> writes:

Hi Peter,

> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.

This is performed already, see the GitLab jobs build-image-tree-sitter
and test-tree-sitter.

Recent results are for example
<https://emba.gnu.org/emacs/emacs/-/jobs/95076> and
<https://emba.gnu.org/emacs/emacs/-/jobs/95071>.

Best regards, Michael.

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 24, 2024, at 6:20 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> Peter Oliver <p.d.oliver@mavit.org.uk> writes:
> 
> Hi Peter,
> 
>> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.
> 
> This is performed already, see the GitLab jobs build-image-tree-sitter
> and test-tree-sitter.
> 
> Recent results are for example
> <https://emba.gnu.org/emacs/emacs/-/jobs/95076> and
> <https://emba.gnu.org/emacs/emacs/-/jobs/95071>.
> 
> Best regards, Michael.

Ah yes, though this is more for us (Emacs devs) than for packagers.

Yuan

Re: Validating tree-sitter major modes and grammar

[Michael Albinus]

Yuan Fu <casouri@gmail.com> writes:

Hi Yuan & Peter,

>>> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.
>>
>> This is performed already, see the GitLab jobs build-image-tree-sitter
>> and test-tree-sitter.
>
> Ah yes, though this is more for us (Emacs devs) than for packagers.

Sure. But if this can be improved for packagers, let us know.

> Yuan

Best regards, Michael.

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 25, 2024, at 12:20 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
> Hi Yuan & Peter,
> 
>>>> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.
>>> 
>>> This is performed already, see the GitLab jobs build-image-tree-sitter
>>> and test-tree-sitter.
>> 
>> Ah yes, though this is more for us (Emacs devs) than for packagers.
> 
> Sure. But if this can be improved for packagers, let us know.

Is it possible for EMBA to run a CI which produces a HTML webpage, and host that webpage somewhere? I’m referring to this idea of Peter’s. 

> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.

I’m thinking of a table where rows are languages, columns are Emacs releases (29, 30, master), and cells are the latest revision of a language grammar that’s compatible with an Emacs release.

Yuan

Re: Validating tree-sitter major modes and grammar

[Michael Albinus]

Yuan Fu <casouri@gmail.com> writes:

Hi Yuan,

>>>>> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.
>>>> 
>>>> This is performed already, see the GitLab jobs build-image-tree-sitter
>>>> and test-tree-sitter.
>>> 
>>> Ah yes, though this is more for us (Emacs devs) than for packagers.
>> 
>> Sure. But if this can be improved for packagers, let us know.
>
> Is it possible for EMBA to run a CI which produces a HTML webpage, and host that webpage somewhere? I’m referring to this idea of Peter’s. 
>
>> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.
>
> I’m thinking of a table where rows are languages, columns are Emacs releases (29, 30, master), and cells are the latest revision of a language grammar that’s compatible with an Emacs release.

Unfortunately, I don't know GitLab sufficiently for answering this question.

> Yuan

Best regatds, Michael.

Re: Validating tree-sitter major modes and grammar

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]

On Wed, 25 Dec 2024, Yuan Fu wrote:

> Is it possible for EMBA to run a CI which produces a HTML webpage, and host that webpage somewhere? I’m referring to this idea of Peter’s.

In general it is possible, using a GitLab feature called Pages (https://emba.gnu.org/help/user/project/pages/index.md).

Some configuration is required (https://emba.gnu.org/help/administration/pages/index.md).  I don’t know if this has been done for EMBA (but I suspect not, because https://emba.gnu.org/emacs/emacs/pages returns 404; alternatively, it’s possible the feature simply needs switching on at https://emba.gnu.org/emacs/emacs/edit#js-general-project-settings).

Once this is done, you don’t have to do much more than, from your CI job, write HTML files to a directory called pages.

> I’m thinking of a table where rows are languages, columns are Emacs releases (29, 30, master), and cells are the latest revision of a language grammar that’s compatible with an Emacs release.

Sounds perfect.

-- 
Peter Oliver

Re: Validating tree-sitter major modes and grammar

[Michael Albinus]

Peter Oliver <p.d.oliver@mavit.org.uk> writes:

Hi Peter,

>> Is it possible for EMBA to run a CI which produces a HTML webpage, and host that webpage somewhere? I’m referring to this idea of Peter’s.
>
> In general it is possible, using a GitLab feature called Pages (https://emba.gnu.org/help/user/project/pages/index.md).
>
> Some configuration is required
> (https://emba.gnu.org/help/administration/pages/index.md).  I don’t
> know if this has been done for EMBA (but I suspect not, because
> https://emba.gnu.org/emacs/emacs/pages returns 404; alternatively,
> it’s possible the feature simply needs switching on at
> https://emba.gnu.org/emacs/emacs/edit#js-general-project-settings).
>
> Once this is done, you don’t have to do much more than, from your CI job, write HTML files to a directory called pages.
>
>> I’m thinking of a table where rows are languages, columns are Emacs releases (29, 30, master), and cells are the latest revision of a language grammar that’s compatible with an Emacs release.
>
> Sounds perfect.

Nice. I'll see whether I can do something useful with this.

Best regards, Michael.

Re: Validating tree-sitter major modes and grammar

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Peter & Yuan,

>>> Is it possible for EMBA to run a CI which produces a HTML webpage, and host that webpage somewhere? I’m referring to this idea of Peter’s.
>>
>> In general it is possible, using a GitLab feature called Pages (https://emba.gnu.org/help/user/project/pages/index.md).
>>
>> Some configuration is required
>> (https://emba.gnu.org/help/administration/pages/index.md).  I don’t
>> know if this has been done for EMBA (but I suspect not, because
>> https://emba.gnu.org/emacs/emacs/pages returns 404; alternatively,
>> it’s possible the feature simply needs switching on at
>> https://emba.gnu.org/emacs/emacs/edit#js-general-project-settings).
>>
>> Once this is done, you don’t have to do much more than, from your CI job, write HTML files to a directory called pages.
>>
>>> I’m thinking of a table where rows are languages, columns are Emacs releases (29, 30, master), and cells are the latest revision of a language grammar that’s compatible with an Emacs release.
>>
>> Sounds perfect.
>
> Nice. I'll see whether I can do something useful with this.

As a starting point, we could use the JUnit reports, which are already
generated. They are not formatted to what we want, but they contain all
test results in XML files for better access.

See for example
<https://emba.gnu.org/emacs/emacs/-/pipelines/31215/test_report> and
click on "Tests". Choose job test-tree-sitter.

There are no data when you inspect "View details", but this can be changed.

Best regards, Michael.

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

[-- Attachment #1: Type: text/plain, Size: 2105 bytes --]



> On Dec 28, 2024, at 4:02 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> Michael Albinus <michael.albinus@gmx.de> writes:
> 
> Hi Peter & Yuan,
> 
>>>> Is it possible for EMBA to run a CI which produces a HTML webpage, and host that webpage somewhere? I’m referring to this idea of Peter’s.
>>> 
>>> In general it is possible, using a GitLab feature called Pages (https://emba.gnu.org/help/user/project/pages/index.md).
>>> 
>>> Some configuration is required
>>> (https://emba.gnu.org/help/administration/pages/index.md).  I don’t
>>> know if this has been done for EMBA (but I suspect not, because
>>> https://emba.gnu.org/emacs/emacs/pages returns 404; alternatively,
>>> it’s possible the feature simply needs switching on at
>>> https://emba.gnu.org/emacs/emacs/edit#js-general-project-settings).
>>> 
>>> Once this is done, you don’t have to do much more than, from your CI job, write HTML files to a directory called pages.
>>> 
>>>> I’m thinking of a table where rows are languages, columns are Emacs releases (29, 30, master), and cells are the latest revision of a language grammar that’s compatible with an Emacs release.
>>> 
>>> Sounds perfect.
>> 
>> Nice. I'll see whether I can do something useful with this.
> 
> As a starting point, we could use the JUnit reports, which are already
> generated. They are not formatted to what we want, but they contain all
> test results in XML files for better access.
> 
> See for example
> <https://emba.gnu.org/emacs/emacs/-/pipelines/31215/test_report> and
> click on "Tests". Choose job test-tree-sitter.
> 
> There are no data when you inspect "View details", but this can be changed.
> 
> Best regards, Michael.

Hmm, ok, it’s a bit unclear to me how do we use this. Anyway, as a first step, I added a command that generates a html file that looks like the below. I’ll try setting up a cron job on my machine, then we can figure out how do we run it on EMBA and how to host the generated webpage.

Peter, do you have any suggestions after seeing this example webpage?


[-- Attachment #2: Screenshot 2024-12-30 at 12.03.17 AM.png --]
[-- Type: image/png, Size: 286801 bytes --]

[-- Attachment #3: Type: text/plain, Size: 9 bytes --]




Yuan

Re: Validating tree-sitter major modes and grammar

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

On Mon, 30 Dec 2024, Yuan Fu wrote:

> Anyway, as a first step, I added a command that generates a html file that looks like the below. I’ll try setting up a cron job on my machine, then we can figure out how do we run it on EMBA and how to host the generated webpage.
>
> Peter, do you have any suggestions after seeing this example webpage?

Looks good.

Rather than the latest commit, I’d be more interested in the latest release (i.e., commit tagged with a tag starting “v”).  I’d only consider packaging a Git snapshot as a last resort to solve an important bug, so tagged commits are more relevant to me.

-- 
Peter Oliver

Re: Validating tree-sitter major modes and grammar

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

On Thu, 2 Jan 2025, Peter Oliver wrote:

> On Mon, 30 Dec 2024, Yuan Fu wrote:
>
>>  Anyway, as a first step, I added a command that generates a html file that
>>  looks like the below. I’ll try setting up a cron job on my machine, then
>>  we can figure out how do we run it on EMBA and how to host the generated
>>  webpage.
>>
>>  Peter, do you have any suggestions after seeing this example webpage?
>
> Looks good.
>
> Rather than the latest commit, I’d be more interested in the latest release 
> (i.e., commit tagged with a tag starting “v”).  I’d only consider packaging a 
> Git snapshot as a last resort to solve an important bug, so tagged commits 
> are more relevant to me.

So, then, something along the lines of the attached (although, actually, maybe we want both?).

-- 
Peter Oliver

[-- Attachment #2: Type: text/plain, Size: 1278 bytes --]

From c1405db6560273ce4eaf0d5c1b13932b876b72a0 Mon Sep 17 00:00:00 2001
From: Peter Oliver <git@mavit.org.uk>
Date: Fri, 3 Jan 2025 12:02:50 +0000
Subject: [PATCH 1/2] New function treesit--language-git-version-tags

* lisp/treesit.el (treesit--language-git-version-tags): New function to
  list version tags for a Tree-sitter parser Git repository.
---
 lisp/treesit.el | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/lisp/treesit.el b/lisp/treesit.el
index f3c553fdb8f..a89fa2ae13e 100644
--- a/lisp/treesit.el
+++ b/lisp/treesit.el
@@ -4262,6 +4262,18 @@ treesit--language-git-revision
       (string-trim (buffer-string)))
      (t nil))))
 
+(defun treesit--language-git-version-tags (repo-dir)
+  "Return a list of Git version tags in REPO-DIR, sorted latest first.
+
+Return the output of \"git tag --list --sort=-version:refname 'v*'\". If
+anything goes wrong, return nil."
+  (with-temp-buffer
+    (cond
+     ((eq 0 (call-process "git" nil t nil "-C" repo-dir "tag"
+                          "--list" "--sort=-version:refname" "v*"))
+      (split-string (buffer-string)))
+     (t nil))))
+
 (defun treesit--language-git-timestamp (repo-dir)
   "Return the commit date in REPO-DIR in UNIX epoch.
 
-- 
2.47.1


[-- Attachment #3: Type: text/plain, Size: 6207 bytes --]

From 0a0348f6410772919007e922f1cc338159d741e4 Mon Sep 17 00:00:00 2001
From: Peter Oliver <git@mavit.org.uk>
Date: Fri, 3 Jan 2025 12:09:32 +0000
Subject: [PATCH 2/2] Tweak Tree-sitter compatibility report to show only
 tagged versions

* admin/tree-sitter/compat-template.html: Rename class head to latest.
* admin/tree-sitter/treesit-admin.el
  (treesit-admin--find-latest-compatible-revision): Inspect only tagged
  commits, as given by treesit--language-git-revision.
---
 admin/tree-sitter/compat-template.html |  4 +--
 admin/tree-sitter/treesit-admin.el     | 39 +++++++++++++-------------
 2 files changed, 22 insertions(+), 21 deletions(-)

diff --git a/admin/tree-sitter/compat-template.html b/admin/tree-sitter/compat-template.html
index 23b5b55d7b3..680301d14a4 100644
--- a/admin/tree-sitter/compat-template.html
+++ b/admin/tree-sitter/compat-template.html
@@ -23,14 +23,14 @@
           width: 10rem;
           word-break: break-all;
       }
-      .head {
+      .latest {
           background: lightgreen;
       }
     </style>
   </head>
   <body>
     <h1>Emacs tree-sitter grammar compatibility</h1>
-    <p>This is an auto-generated report of the last compatible version for each grammar in each Emacs version.  A <span class="head">green background</span> on the version indicates that the Emacs version is compatible with the latest commit in the upstream grammar repo.</p>
+    <p>This is an auto-generated report of the last compatible version for each grammar in each Emacs version.  A <span class="head">green background</span> on the version indicates that the Emacs version is compatible with the latest tag in the upstream grammar repo.</p>
     <p>This report is generated on ___REPLACE_TIME___.</p>
     <table>
 ___REPLACE_TABLE___
diff --git a/admin/tree-sitter/treesit-admin.el b/admin/tree-sitter/treesit-admin.el
index 3711eb0a301..a69131c1c9d 100644
--- a/admin/tree-sitter/treesit-admin.el
+++ b/admin/tree-sitter/treesit-admin.el
@@ -296,9 +296,9 @@ treesit-admin--find-latest-compatible-revision
 
 Return a plist of the form
 
-    (:version VERSION :head-version HEAD-VERSION :timstamp TIMESTAMP).
+    (:version VERSION :latest-version LATEST-VERSION :timstamp TIMESTAMP).
 
-HEAD-VERSION is the version of the HEAD, VERSION is the latest
+LATEST-VERSION is the latest version, VERSION is the latest
 compatible version.  TIMESTAMP is the commit date of VERSION in UNIX
 epoch format."
   (let ((treesit-extra-load-path (list grammar-dir))
@@ -309,7 +309,8 @@ treesit-admin--find-latest-compatible-revision
         (emacs-executable
          (or emacs-executable
              (expand-file-name invocation-name invocation-directory)))
-        head-version version exit-code timestamp)
+        (latest-version nil)
+        latest-version version exit-code timestamp)
     (when (not recipe)
       (signal 'treesit-error `("Cannot find recipe" ,language)))
     (pcase-let ((`(,url ,revision ,source-dir ,cc ,c++ ,commit)
@@ -318,17 +319,14 @@ treesit-admin--find-latest-compatible-revision
         (treesit--git-clone-repo url revision workdir)
         (when commit
           (treesit--git-checkout-branch workdir commit))
-        (setq head-version (treesit--language-git-revision workdir))
-        (treesit--build-grammar
-         workdir grammar-dir language source-dir cc c++)
-        (while (not (eq exit-code 0))
-          (unless (null exit-code)
-            (treesit--git-checkout-branch workdir "HEAD~")
-            (treesit--build-grammar
-             workdir grammar-dir language source-dir cc c++))
-          (setq version (treesit--language-git-revision workdir))
+        (dolist (tag (treesit--language-git-version-tags workdir))
+          (unless latest-version
+            (setq latest-version tag))
+          (treesit--git-checkout-branch workdir tag)
+          (treesit--build-grammar
+           workdir grammar-dir language source-dir cc c++)
           (setq timestamp (treesit--language-git-timestamp workdir))
-          (message "Validateing version %s" version)
+          (message "Validating version %s" tag)
           (setq exit-code
                 (call-process
                  emacs-executable nil t nil
@@ -340,8 +338,11 @@ treesit-admin--find-latest-compatible-revision
                               (if (treesit-admin--validate-mode-lang
                                    ',mode ',language)
                                   (kill-emacs 0)
-                                (kill-emacs -1)))))))))
-    (list :version version :head-version head-version :timestamp timestamp)))
+                                (kill-emacs -1))))))
+          (when (eq exit-code 0)
+            (setq version tag)
+            (cl-return)))))
+    (list :version version :latest-version latest-version :timestamp timestamp)))
 
 (defun treesit-admin--last-compatible-grammar-for-modes
     (modes source-alist grammar-dir &optional emacs-executable)
@@ -353,9 +354,9 @@ treesit-admin--last-compatible-grammar-for-modes
 
 Return an alist of an alist of a plist:
 
-    ((MODE . ((LANG . (:version VERSION :head-VERSION HEAD-VERSION)) ...)) ...)
+    ((MODE . ((LANG . (:version VERSION :latest-version LATEST-VERSION)) ...)) ...)
 
-VERSION and HEAD-VERSION in the plist are the same as in
+VERSION and LATEST-VERSION in the plist are the same as in
 `treesit-admin--find-latest-compatible-revision'."
   (mapcar
    (lambda (mode)
@@ -427,9 +428,9 @@ treesit-admin--generate-compatibility-report
           (let* ((key (cons lang emacs-version))
                  (plist (gethash key database))
                  (version (plist-get plist :version))
-                 (head-version (plist-get plist :head-version))
+                 (latest-version (plist-get plist :latest-version))
                  (classname
-                  (if (equal version head-version) "head" "")))
+                  (if (equal version latest-version) "latest" "")))
             (if (not plist)
                 (insert "<td></td>")
               (insert (format "<td class=\"%s\">%s</td>"
-- 
2.47.1

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Jan 3, 2025, at 4:26 AM, Peter Oliver <p.d.oliver@mavit.org.uk> wrote:
> 
> On Thu, 2 Jan 2025, Peter Oliver wrote:
> 
>> On Mon, 30 Dec 2024, Yuan Fu wrote:
>> 
>>> Anyway, as a first step, I added a command that generates a html file that
>>> looks like the below. I’ll try setting up a cron job on my machine, then
>>> we can figure out how do we run it on EMBA and how to host the generated
>>> webpage.
>>> 
>>> Peter, do you have any suggestions after seeing this example webpage?
>> 
>> Looks good.
>> 
>> Rather than the latest commit, I’d be more interested in the latest release (i.e., commit tagged with a tag starting “v”).  I’d only consider packaging a Git snapshot as a last resort to solve an important bug, so tagged commits are more relevant to me.
> 
> So, then, something along the lines of the attached (although, actually, maybe we want both?).
> 
> -- 
> Peter Oliver<0001-New-function-treesit-language-git-version-tags.patch><0002-Tweak-Tree-sitter-compatibility-report-to-show-only-.patch>

Thanks! What you showed looks good. And yeah, we should have both “last tag” and “last commit”. “Last commit” would be useful for people that install tree-sitter grammars with download scripts.

Let me see how can I construct the table to show both.

Yuan

Re: Validating tree-sitter major modes and grammar

[Yuan Fu]

> On Dec 24, 2024, at 4:36 AM, Peter Oliver <p.d.oliver@mavit.org.uk> wrote:
> 
> On Mon, 23 Dec 2024, Yuan Fu wrote:
> 
>>> On Dec 22, 2024, at 4:44 PM, Björn Bidar <bjorn.bidar@thaodan.de> wrote:
>>> 
>>> Yuan Fu <casouri@gmail.com> writes:
>>> 
>>>> Continuing from the tree-sitter maternity thread, I cooked up some
>>>> script to go over each builtin tree-sitter mode, clone the grammars it
>>>> uses, and check whether the font-lock queries are compatible with the
>>>> latest version of the grammar. If everything works fine, the script
>>>> adds some comment in the source file listing the version that was
>>>> checked.
>>> 
>>> Oh that sounds very good. This could be a good workaround for the sort
>>> of fast and lose nature of tree-sitter.
>> 
>> Great. Do you think the “verified version” comment will be helpful for packagers?
> 
> It helps answer the question, “I am building a new Emacs release.  Do I need to update these Tree-sitter parsers?”.
> 
> However, as a parser packager, I also need to answer the question, “This parser has released a new version.  Can I package it now, or do I need to wait for Emacs 30?  Emacs 31?”.  A static comment in the source code of a released version of Emacs will likely be too out-of-date to answer that.
> 
> Other ideas:
> 
> - Run these checks regularly (from https://emba.gnu.org/, perhaps?) for the most-recent Emacs release, outputting a report or webpage that can be referred to by packagers.
> 
> - Include a function in Emacs that packagers can call themselves to check the compatibility of installed parsers.  They could run this check every time they build an updated parser (either manually, as a part of the package build, or as a part of their distribution’s automated CI testing).

I can add another script that finds the latest “version” that’s compatible with a builtin mode. The comments will be helpful for packaging a new Emacs release. And the new script can help with packaging new grammar versions with Emacs.

Yuan

Re: Validating tree-sitter major modes and grammar

[Björn Bidar]

Yuan Fu <casouri@gmail.com> writes:

>> On Dec 22, 2024, at 4:44 PM, Björn Bidar <bjorn.bidar@thaodan.de> wrote:
>> 
>> Yuan Fu <casouri@gmail.com> writes:
>> 
>>> Continuing from the tree-sitter maternity thread, I cooked up some
>>> script to go over each builtin tree-sitter mode, clone the grammars it
>>> uses, and check whether the font-lock queries are compatible with the
>>> latest version of the grammar. If everything works fine, the script
>>> adds some comment in the source file listing the version that was
>>> checked.
>> 
>> Oh that sounds very good. This could be a good workaround for the sort
>> of fast and lose nature of tree-sitter.
>> 
>
> Great. Do you think the “verified version” comment will be helpful for packagers?
>

It would be helpful in case something goes wrong as a kind of sanity
check but it won't help much for an indicator when to update since I
don't think that would be blocker for updating.

It is helpful but none the less the goal should be that such errors
should be preventable between grammar versions.