Re: Call for volunteers: add tree-sitter support to major modes

Re: Call for volunteers: add tree-sitter support to major modes

[João Paulo Labegalini de Carvalho]

[-- Attachment #1: Type: text/plain, Size: 381 bytes --]

I would like to work on the c-mode if no one else is already working on
those.

I can also volunteer for c++-mode or shell-script-mode if needed.

-- 
João Paulo L. de Carvalho
Ph.D Computer Science |  IC-UNICAMP | Campinas , SP - Brazil
Postdoctoral Research Fellow | University of Alberta | Edmonton, AB - Canada
joao.carvalho@ic.unicamp.br
joao.carvalho@ualberta.ca

[-- Attachment #2: Type: text/html, Size: 750 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com>
> Date: Tue, 11 Oct 2022 17:14:36 -0600
> 
> I would like to work on the c-mode if no one else is already working on those.
> 
> I can also volunteer for c++-mode or shell-script-mode if needed.

What and how we should handle the C and derived modes is currently
under discussion.  So if you could start with shell-script-mode,
that'd be ideal, I think.

Thanks.

Re: Call for volunteers: add tree-sitter support to major modes

[João Paulo Labegalini de Carvalho]

[-- Attachment #1: Type: text/plain, Size: 531 bytes --]

On Tue, Oct 11, 2022 at 11:43 PM Eli Zaretskii <eliz@gnu.org> wrote:

> What and how we should handle the C and derived modes is currently
> under discussion.  So if you could start with shell-script-mode,
> that'd be ideal, I think.
>

For sure. I will start working on shell-script-mode.


-- 
João Paulo L. de Carvalho
Ph.D Computer Science |  IC-UNICAMP | Campinas , SP - Brazil
Postdoctoral Research Fellow | University of Alberta | Edmonton, AB - Canada
joao.carvalho@ic.unicamp.br
joao.carvalho@ualberta.ca

[-- Attachment #2: Type: text/html, Size: 1155 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com>
> Date: Wed, 12 Oct 2022 09:09:26 -0600
> Cc: emacs-devel@gnu.org
> 
> On Tue, Oct 11, 2022 at 11:43 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  What and how we should handle the C and derived modes is currently
>  under discussion.  So if you could start with shell-script-mode,
>  that'd be ideal, I think.
> 
> For sure. I will start working on shell-script-mode. 

Thanks!

Re: Call for volunteers: add tree-sitter support to major modes

[João Paulo Labegalini de Carvalho]

[-- Attachment #1: Type: text/plain, Size: 1863 bytes --]

I have finally got some time to work on this.

From my initial understanding of `sh-script-mode' is that it supports many
shell scripting languages via an elegant "inheritance" structured code.

As part of the tree-sitter project, I only found a repo to generate the
parser for bash. So it is my impression that other shell languages might
not be correctly parsed by tree-sitter-bash from here:
https://github.com/tree-sitter/tree-sitter-bash.

My idea to incrementally add support for shell languages is to start with
bash and setup three `tree-sitter-font-lock-rules' for it, like so:

(defvar sh-script--treesit-settings
    (treesit-font-lock-rules
        :language 'bash
        :feature 'basic
        ;; queries for 'basic feature here
        :language 'bash
        :feature 'moderate
        ;; queries for 'moderate feature here
        :language 'bash
        :feature 'full
        ;; queries for 'full feature here))

Would that be acceptable? Or should I use a function in the `language:'
field that returns a shell language symbol?

On Wed, Oct 12, 2022 at 9:36 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com>
> > Date: Wed, 12 Oct 2022 09:09:26 -0600
> > Cc: emacs-devel@gnu.org
> >
> > On Tue, Oct 11, 2022 at 11:43 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> >  What and how we should handle the C and derived modes is currently
> >  under discussion.  So if you could start with shell-script-mode,
> >  that'd be ideal, I think.
> >
> > For sure. I will start working on shell-script-mode.
>
> Thanks!
>


-- 
João Paulo L. de Carvalho
Ph.D Computer Science |  IC-UNICAMP | Campinas , SP - Brazil
Postdoctoral Research Fellow | University of Alberta | Edmonton, AB - Canada
joao.carvalho@ic.unicamp.br
joao.carvalho@ualberta.ca

[-- Attachment #2: Type: text/html, Size: 3454 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[João Paulo Labegalini de Carvalho]

[-- Attachment #1: Type: text/plain, Size: 2333 bytes --]

With the work a did today I was able to answer my own questions.

I managed to use `sh-feature' function to obtain the lists of keywords &
builtins for fontification.

I am planning to soon submit a patch for review but mainly for the style of
bits I have implemented.

On Fri., Oct. 21, 2022, 10:47 a.m. João Paulo Labegalini de Carvalho, <
jaopaulolc@gmail.com> wrote:

> I have finally got some time to work on this.
>
> From my initial understanding of `sh-script-mode' is that it supports many
> shell scripting languages via an elegant "inheritance" structured code.
>
> As part of the tree-sitter project, I only found a repo to generate the
> parser for bash. So it is my impression that other shell languages might
> not be correctly parsed by tree-sitter-bash from here:
> https://github.com/tree-sitter/tree-sitter-bash.
>
> My idea to incrementally add support for shell languages is to start with
> bash and setup three `tree-sitter-font-lock-rules' for it, like so:
>
> (defvar sh-script--treesit-settings
>     (treesit-font-lock-rules
>         :language 'bash
>         :feature 'basic
>         ;; queries for 'basic feature here
>         :language 'bash
>         :feature 'moderate
>         ;; queries for 'moderate feature here
>         :language 'bash
>         :feature 'full
>         ;; queries for 'full feature here))
>
> Would that be acceptable? Or should I use a function in the `language:'
> field that returns a shell language symbol?
>
> On Wed, Oct 12, 2022 at 9:36 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > From: João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com>
>> > Date: Wed, 12 Oct 2022 09:09:26 -0600
>> > Cc: emacs-devel@gnu.org
>> >
>> > On Tue, Oct 11, 2022 at 11:43 PM Eli Zaretskii <eliz@gnu.org> wrote:
>> >
>> >  What and how we should handle the C and derived modes is currently
>> >  under discussion.  So if you could start with shell-script-mode,
>> >  that'd be ideal, I think.
>> >
>> > For sure. I will start working on shell-script-mode.
>>
>> Thanks!
>>
>
>
> --
> João Paulo L. de Carvalho
> Ph.D Computer Science |  IC-UNICAMP | Campinas , SP - Brazil
> Postdoctoral Research Fellow | University of Alberta | Edmonton, AB -
> Canada
> joao.carvalho@ic.unicamp.br
> joao.carvalho@ualberta.ca
>

[-- Attachment #2: Type: text/html, Size: 4315 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 21, 2022, at 4:45 PM, João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com> wrote:
> 
> With the work a did today I was able to answer my own questions.
> 
> I managed to use `sh-feature' function to obtain the lists of keywords & builtins for fontification.
> 
> I am planning to soon submit a patch for review but mainly for the style of bits I have implemented.

Great go hear!

Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 21, 2022, at 6:52 PM, Yuan Fu <casouri@gmail.com> wrote:
> 
> 
> 
>> On Oct 21, 2022, at 4:45 PM, João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com> wrote:
>> 
>> With the work a did today I was able to answer my own questions.
>> 
>> I managed to use `sh-feature' function to obtain the lists of keywords & builtins for fontification.
>> 
>> I am planning to soon submit a patch for review but mainly for the style of bits I have implemented.
> 
> Great go hear!

I meant great to hear, of course :-)

Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com>
> Date: Fri, 21 Oct 2022 17:45:41 -0600
> Cc: emacs-devel@gnu.org
> 
> With the work a did today I was able to answer my own questions.
> 
> I managed to use `sh-feature' function to obtain the lists of keywords & builtins for fontification.
> 
> I am planning to soon submit a patch for review but mainly for the style of bits I have implemented.

Thanks for working on this.

Re: Call for volunteers: add tree-sitter support to major modes

[João Paulo Labegalini de Carvalho]

[-- Attachment #1: Type: text/plain, Size: 1846 bytes --]

I am getting a query error but I don't understand why.

The following query is fine:

(defvar sh-script--treesit-bash-keywords
  '("case" "do" "done" "elif" "else" "esac" "export" "fi" "for"
    "function" "if" "in" "unset" "while" "then"))

(treesit-validate-query 'bash `([ ,@sh-script--treesit-bash-keywords ]
@font-lock-keyword-face))

However the following query is said INVALID by `treesit-validate-query':
(treesit-validate-query 'bash `([ ,@(sh-feature sh-leading-keywords) ]
@font-lock-keyword-face))
Node type error at: 3
["*time"* "!" "do" "done" ...] @font-lock-keyword-face

*time"* is highlighted in the *tree-sitter check query* buffer.

Even though the forms below evaluate to equivalent forms:
`([ ,@sh-script--treesit-bash-keywords] @font-lock-keyword-face)
*evaluates to:*
([ "case" "do" "done" "elif" ... ] @font-lock-keyword-face)

`([ ,@(sh-feature sh-leading-keywords) ] @font-lock-keyword-face)
*evaluates to:*
(["time" "!" "do" "done" ...] @font-lock-keyword-face)


Any clues to what I am doing wrong?

On Sat, Oct 22, 2022 at 12:42 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com>
> > Date: Fri, 21 Oct 2022 17:45:41 -0600
> > Cc: emacs-devel@gnu.org
> >
> > With the work a did today I was able to answer my own questions.
> >
> > I managed to use `sh-feature' function to obtain the lists of keywords &
> builtins for fontification.
> >
> > I am planning to soon submit a patch for review but mainly for the style
> of bits I have implemented.
>
> Thanks for working on this.
>


-- 
João Paulo L. de Carvalho
Ph.D Computer Science |  IC-UNICAMP | Campinas , SP - Brazil
Postdoctoral Research Fellow | University of Alberta | Edmonton, AB - Canada
joao.carvalho@ic.unicamp.br
joao.carvalho@ualberta.ca

[-- Attachment #2: Type: text/html, Size: 3434 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 22, 2022, at 8:51 AM, João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com> wrote:
> 
> I am getting a query error but I don't understand why.
> 
> The following query is fine:
> 
> (defvar sh-script--treesit-bash-keywords
>   '("case" "do" "done" "elif" "else" "esac" "export" "fi" "for"
>     "function" "if" "in" "unset" "while" "then"))
> 
> (treesit-validate-query 'bash `([ ,@sh-script--treesit-bash-keywords ] @font-lock-keyword-face))
> 
> However the following query is said INVALID by `treesit-validate-query':
> (treesit-validate-query 'bash `([ ,@(sh-feature sh-leading-keywords) ] @font-lock-keyword-face))
> Node type error at: 3
> ["time" "!" "do" "done" ...] @font-lock-keyword-face
> 
> time" is highlighted in the *tree-sitter check query* buffer. 
> 
> Even though the forms below evaluate to equivalent forms:
> `([ ,@sh-script--treesit-bash-keywords] @font-lock-keyword-face)
> evaluates to:
> ([ "case" "do" "done" "elif" ... ] @font-lock-keyword-face)
> 
> `([ ,@(sh-feature sh-leading-keywords) ] @font-lock-keyword-face)
> evaluates to:
> (["time" "!" "do" "done" ...] @font-lock-keyword-face)
> 
> 
> Any clues to what I am doing wrong?

It is saying that there is no “time” node in bash grammar. You probably need to consult the grammar file of tree-sitter-bash to see what are the keywords it recognizes.

For example, running the following snippet 

(let (collection)
  (goto-char (point-min))
  (while (re-search-forward "'[^ ][^ ]+?'" nil t)
    (push (match-string 0) collection))
  (pop-to-buffer "*result*")
  (dolist (keyword (cl-remove-duplicates collection :test #'equal))
    (insert keyword "\n")))

in the grammar.js gives me

'\\\\'
'>('
'<('
'$('
':-'
':?'
'${'
')*'
'([^'
'[^'
'--'
'++'
'alternative'
'consequence'
'right'
'>='
'<='
'-='
'!='
'operator'
'left'
'<<<'
'destination'
'>|'
'>&'
'<&'
'&>>'
'&>'
'>>'
'descriptor'
'index'
'=='
'=~'
'argument'
'unsetenv'
'unset'
'local'
'readonly'
'export'
'typeset'
'declare'
']]'
'[['
'||'
'&&'
'|&'
'name'
'function'
';;&'
';&'
'fallthrough'
';;'
'termination'
'esac'
'case'
'else'
'elif'
'fi'
'then'
'if'
'done'
'do'
'until'
'while'
'))'
'update'
'condition'
'initializer'
'(('
'value'
'in'
'variable'
'select'
'for'
'redirect'
'body'
'\n'
'<<-'
'<<'
'+='
'bash'
'\\s'
'\\'
'\\]'
'\\['


Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[João Paulo Labegalini de Carvalho]

[-- Attachment #1: Type: text/plain, Size: 3753 bytes --]

Thanks for pointing that out. I am still getting familiar with tree-sitter
thus the message sounded very generic.

According to the GNU Bash Reference Manual, "time" is a reserved
<https://www.gnu.org/software/bash/manual/bash.html#Major-Differences-From-The-Bourne-Shell>
word. But tree-sitter does not include it in the grammar.

Should I submit a PR for tree-sitter-bash to add "time", and possibly other
missing reserved word/keywords, or should I just fontify those recognized
by tree-sitter-bash in shell-script-mode?

I believe the best course of action is to make tree-sitter-bash as complete
as possible. That way the regex and tree-sitter fontification will fontify
exactly the same things.

What do you guys think?

On Sun, Oct 23, 2022 at 10:20 PM Yuan Fu <casouri@gmail.com> wrote:

>
>
> > On Oct 22, 2022, at 8:51 AM, João Paulo Labegalini de Carvalho <
> jaopaulolc@gmail.com> wrote:
> >
> > I am getting a query error but I don't understand why.
> >
> > The following query is fine:
> >
> > (defvar sh-script--treesit-bash-keywords
> >   '("case" "do" "done" "elif" "else" "esac" "export" "fi" "for"
> >     "function" "if" "in" "unset" "while" "then"))
> >
> > (treesit-validate-query 'bash `([ ,@sh-script--treesit-bash-keywords ]
> @font-lock-keyword-face))
> >
> > However the following query is said INVALID by `treesit-validate-query':
> > (treesit-validate-query 'bash `([ ,@(sh-feature sh-leading-keywords) ]
> @font-lock-keyword-face))
> > Node type error at: 3
> > ["time" "!" "do" "done" ...] @font-lock-keyword-face
> >
> > time" is highlighted in the *tree-sitter check query* buffer.
> >
> > Even though the forms below evaluate to equivalent forms:
> > `([ ,@sh-script--treesit-bash-keywords] @font-lock-keyword-face)
> > evaluates to:
> > ([ "case" "do" "done" "elif" ... ] @font-lock-keyword-face)
> >
> > `([ ,@(sh-feature sh-leading-keywords) ] @font-lock-keyword-face)
> > evaluates to:
> > (["time" "!" "do" "done" ...] @font-lock-keyword-face)
> >
> >
> > Any clues to what I am doing wrong?
>
> It is saying that there is no “time” node in bash grammar. You probably
> need to consult the grammar file of tree-sitter-bash to see what are the
> keywords it recognizes.
>
> For example, running the following snippet
>
> (let (collection)
>   (goto-char (point-min))
>   (while (re-search-forward "'[^ ][^ ]+?'" nil t)
>     (push (match-string 0) collection))
>   (pop-to-buffer "*result*")
>   (dolist (keyword (cl-remove-duplicates collection :test #'equal))
>     (insert keyword "\n")))
>
> in the grammar.js gives me
>
> '\\\\'
> '>('
> '<('
> '$('
> ':-'
> ':?'
> '${'
> ')*'
> '([^'
> '[^'
> '--'
> '++'
> 'alternative'
> 'consequence'
> 'right'
> '>='
> '<='
> '-='
> '!='
> 'operator'
> 'left'
> '<<<'
> 'destination'
> '>|'
> '>&'
> '<&'
> '&>>'
> '&>'
> '>>'
> 'descriptor'
> 'index'
> '=='
> '=~'
> 'argument'
> 'unsetenv'
> 'unset'
> 'local'
> 'readonly'
> 'export'
> 'typeset'
> 'declare'
> ']]'
> '[['
> '||'
> '&&'
> '|&'
> 'name'
> 'function'
> ';;&'
> ';&'
> 'fallthrough'
> ';;'
> 'termination'
> 'esac'
> 'case'
> 'else'
> 'elif'
> 'fi'
> 'then'
> 'if'
> 'done'
> 'do'
> 'until'
> 'while'
> '))'
> 'update'
> 'condition'
> 'initializer'
> '(('
> 'value'
> 'in'
> 'variable'
> 'select'
> 'for'
> 'redirect'
> 'body'
> '\n'
> '<<-'
> '<<'
> '+='
> 'bash'
> '\\s'
> '\\'
> '\\]'
> '\\['
>
>
> Yuan



-- 
João Paulo L. de Carvalho
Ph.D Computer Science |  IC-UNICAMP | Campinas , SP - Brazil
Postdoctoral Research Fellow | University of Alberta | Edmonton, AB - Canada
joao.carvalho@ic.unicamp.br
joao.carvalho@ualberta.ca

[-- Attachment #2: Type: text/html, Size: 5934 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[João Paulo Labegalini de Carvalho]

[-- Attachment #1: Type: text/plain, Size: 4575 bytes --]

This is a better link to show all Bash's reserved words/keywords:
 https://www.gnu.org/software/bash/manual/bash.html#Reserved-Words
<https://www.gnu.org/software/bash/manual/bash.html#Reserved-Words>

I will reach out to tree-sitter-bash folks to add the missing things to the
grammar.

On Mon, Oct 24, 2022 at 9:41 AM João Paulo Labegalini de Carvalho <
jaopaulolc@gmail.com> wrote:

> Thanks for pointing that out. I am still getting familiar with tree-sitter
> thus the message sounded very generic.
>
> According to the GNU Bash Reference Manual, "time" is a reserved
> <https://www.gnu.org/software/bash/manual/bash.html#Major-Differences-From-The-Bourne-Shell>
> word. But tree-sitter does not include it in the grammar.
>
> Should I submit a PR for tree-sitter-bash to add "time", and possibly
> other missing reserved word/keywords, or should I just fontify those
> recognized by tree-sitter-bash in shell-script-mode?
>
> I believe the best course of action is to make tree-sitter-bash as
> complete as possible. That way the regex and tree-sitter fontification will
> fontify exactly the same things.
>
> What do you guys think?
>
> On Sun, Oct 23, 2022 at 10:20 PM Yuan Fu <casouri@gmail.com> wrote:
>
>>
>>
>> > On Oct 22, 2022, at 8:51 AM, João Paulo Labegalini de Carvalho <
>> jaopaulolc@gmail.com> wrote:
>> >
>> > I am getting a query error but I don't understand why.
>> >
>> > The following query is fine:
>> >
>> > (defvar sh-script--treesit-bash-keywords
>> >   '("case" "do" "done" "elif" "else" "esac" "export" "fi" "for"
>> >     "function" "if" "in" "unset" "while" "then"))
>> >
>> > (treesit-validate-query 'bash `([ ,@sh-script--treesit-bash-keywords ]
>> @font-lock-keyword-face))
>> >
>> > However the following query is said INVALID by `treesit-validate-query':
>> > (treesit-validate-query 'bash `([ ,@(sh-feature sh-leading-keywords) ]
>> @font-lock-keyword-face))
>> > Node type error at: 3
>> > ["time" "!" "do" "done" ...] @font-lock-keyword-face
>> >
>> > time" is highlighted in the *tree-sitter check query* buffer.
>> >
>> > Even though the forms below evaluate to equivalent forms:
>> > `([ ,@sh-script--treesit-bash-keywords] @font-lock-keyword-face)
>> > evaluates to:
>> > ([ "case" "do" "done" "elif" ... ] @font-lock-keyword-face)
>> >
>> > `([ ,@(sh-feature sh-leading-keywords) ] @font-lock-keyword-face)
>> > evaluates to:
>> > (["time" "!" "do" "done" ...] @font-lock-keyword-face)
>> >
>> >
>> > Any clues to what I am doing wrong?
>>
>> It is saying that there is no “time” node in bash grammar. You probably
>> need to consult the grammar file of tree-sitter-bash to see what are the
>> keywords it recognizes.
>>
>> For example, running the following snippet
>>
>> (let (collection)
>>   (goto-char (point-min))
>>   (while (re-search-forward "'[^ ][^ ]+?'" nil t)
>>     (push (match-string 0) collection))
>>   (pop-to-buffer "*result*")
>>   (dolist (keyword (cl-remove-duplicates collection :test #'equal))
>>     (insert keyword "\n")))
>>
>> in the grammar.js gives me
>>
>> '\\\\'
>> '>('
>> '<('
>> '$('
>> ':-'
>> ':?'
>> '${'
>> ')*'
>> '([^'
>> '[^'
>> '--'
>> '++'
>> 'alternative'
>> 'consequence'
>> 'right'
>> '>='
>> '<='
>> '-='
>> '!='
>> 'operator'
>> 'left'
>> '<<<'
>> 'destination'
>> '>|'
>> '>&'
>> '<&'
>> '&>>'
>> '&>'
>> '>>'
>> 'descriptor'
>> 'index'
>> '=='
>> '=~'
>> 'argument'
>> 'unsetenv'
>> 'unset'
>> 'local'
>> 'readonly'
>> 'export'
>> 'typeset'
>> 'declare'
>> ']]'
>> '[['
>> '||'
>> '&&'
>> '|&'
>> 'name'
>> 'function'
>> ';;&'
>> ';&'
>> 'fallthrough'
>> ';;'
>> 'termination'
>> 'esac'
>> 'case'
>> 'else'
>> 'elif'
>> 'fi'
>> 'then'
>> 'if'
>> 'done'
>> 'do'
>> 'until'
>> 'while'
>> '))'
>> 'update'
>> 'condition'
>> 'initializer'
>> '(('
>> 'value'
>> 'in'
>> 'variable'
>> 'select'
>> 'for'
>> 'redirect'
>> 'body'
>> '\n'
>> '<<-'
>> '<<'
>> '+='
>> 'bash'
>> '\\s'
>> '\\'
>> '\\]'
>> '\\['
>>
>>
>> Yuan
>
>
>
> --
> João Paulo L. de Carvalho
> Ph.D Computer Science |  IC-UNICAMP | Campinas , SP - Brazil
> Postdoctoral Research Fellow | University of Alberta | Edmonton, AB -
> Canada
> joao.carvalho@ic.unicamp.br
> joao.carvalho@ualberta.ca
>


-- 
João Paulo L. de Carvalho
Ph.D Computer Science |  IC-UNICAMP | Campinas , SP - Brazil
Postdoctoral Research Fellow | University of Alberta | Edmonton, AB - Canada
joao.carvalho@ic.unicamp.br
joao.carvalho@ualberta.ca

[-- Attachment #2: Type: text/html, Size: 7191 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com> writes:

> I am getting a query error but I don't understand why.
>
> The following query is fine:
>
> (defvar sh-script--treesit-bash-keywords
>   '("case" "do" "done" "elif" "else" "esac" "export" "fi" "for"
>     "function" "if" "in" "unset" "while" "then"))
>
> (treesit-validate-query 'bash `([ ,@sh-script--treesit-bash-keywords ]
> @font-lock-keyword-face))
>
> However the following query is said INVALID by `treesit-validate-query':
> (treesit-validate-query 'bash `([ ,@(sh-feature sh-leading-keywords) ]
> @font-lock-keyword-face))
> Node type error at: 3
> ["*time"* "!" "do" "done" ...] @font-lock-keyword-face
>
> *time"* is highlighted in the *tree-sitter check query* buffer.
>
> Even though the forms below evaluate to equivalent forms:
> `([ ,@sh-script--treesit-bash-keywords] @font-lock-keyword-face)
> *evaluates to:*
> ([ "case" "do" "done" "elif" ... ] @font-lock-keyword-face)
>
> `([ ,@(sh-feature sh-leading-keywords) ] @font-lock-keyword-face)
> *evaluates to:*
> (["time" "!" "do" "done" ...] @font-lock-keyword-face)
>
>
> Any clues to what I am doing wrong?

Can you post the whole mode in its current state?  Then I can help you
debug it.  I've seen errors like this myself, but most of the time it
has been some sort of error, like a typo in one of the strings.

I feel there's too little to know exactly what is wrong from your mail,
so the whole source would be nice!

Thanks,
Theo

Re: Call for volunteers: add tree-sitter support to major modes

[João Paulo Labegalini de Carvalho]

[-- Attachment #1: Type: text/plain, Size: 2308 bytes --]

Sorry, Theodor. I should have added the code I am actually trying to make
it work.

I just come from a different culture where a minimal example is
preferred over a complete/longer code to show a bug/unexpected behavior. I
will make sure to add the code next time.

But I believe Yuan spotted the issue. Now I need to figure out a way to
proceed. (Please check my response to Yuan).

On Mon, Oct 24, 2022 at 12:23 AM Theodor Thornhill <theo@thornhill.no>
wrote:

> João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com> writes:
>
> > I am getting a query error but I don't understand why.
> >
> > The following query is fine:
> >
> > (defvar sh-script--treesit-bash-keywords
> >   '("case" "do" "done" "elif" "else" "esac" "export" "fi" "for"
> >     "function" "if" "in" "unset" "while" "then"))
> >
> > (treesit-validate-query 'bash `([ ,@sh-script--treesit-bash-keywords ]
> > @font-lock-keyword-face))
> >
> > However the following query is said INVALID by `treesit-validate-query':
> > (treesit-validate-query 'bash `([ ,@(sh-feature sh-leading-keywords) ]
> > @font-lock-keyword-face))
> > Node type error at: 3
> > ["*time"* "!" "do" "done" ...] @font-lock-keyword-face
> >
> > *time"* is highlighted in the *tree-sitter check query* buffer.
> >
> > Even though the forms below evaluate to equivalent forms:
> > `([ ,@sh-script--treesit-bash-keywords] @font-lock-keyword-face)
> > *evaluates to:*
> > ([ "case" "do" "done" "elif" ... ] @font-lock-keyword-face)
> >
> > `([ ,@(sh-feature sh-leading-keywords) ] @font-lock-keyword-face)
> > *evaluates to:*
> > (["time" "!" "do" "done" ...] @font-lock-keyword-face)
> >
> >
> > Any clues to what I am doing wrong?
>
> Can you post the whole mode in its current state?  Then I can help you
> debug it.  I've seen errors like this myself, but most of the time it
> has been some sort of error, like a typo in one of the strings.
>
> I feel there's too little to know exactly what is wrong from your mail,
> so the whole source would be nice!
>
> Thanks,
> Theo
>


-- 
João Paulo L. de Carvalho
Ph.D Computer Science |  IC-UNICAMP | Campinas , SP - Brazil
Postdoctoral Research Fellow | University of Alberta | Edmonton, AB - Canada
joao.carvalho@ic.unicamp.br
joao.carvalho@ualberta.ca

[-- Attachment #2: Type: text/html, Size: 3447 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

On 24 October 2022 17:44:43 CEST, "João Paulo Labegalini de Carvalho" <jaopaulolc@gmail.com> wrote:
>Sorry, Theodor. I should have added the code I am actually trying to make
>it work.

Yeah, I do that too :) 

>
>I just come from a different culture where a minimal example is
>preferred over a complete/longer code to show a bug/unexpected behavior. I
>will make sure to add the code next time.
>

No need :) 

>But I believe Yuan spotted the issue. Now I need to figure out a way to
>proceed. (Please check my response to Yuan).
>

I'm glad! 

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Payas Relekar]

Po Lu <luangruo@yahoo.com> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Yes, more of an Emacs 30 thing, so 2024-ish.
>>
>> That is, during the next year, I'm guessing that we'll be gaining
>> tree-sitter support for all the major programming languages, and when
>> that is better than what we already have, we should consider making
>> tree-sitter a prerequisite for Emacs (and these modes) and rapidly phase
>> out the old code.
>
> I and many others will _very_ strongly object to removing the build
> without tree-sitter.  Emacs has never needed a non-system library to
> edit text, and that should not start now.
>
> On X, we even support the build without XCB, as even though it has been
> ubitous for decades, it is not specified in any document and cannot be
> implemented by third parties.  Unlike Xlib, which is part of the
> specifications for release 6 of the X Window System, version 11,
> published by the X Consortium.
>
>> It's also a question of how much user breakage we'll tolerate/aim for.
>> That is, there's a gazillion tuning knobs all over the modes to tweak
>> stuff that will be rendered obsolete by tree-sitter, and which people
>> may be fond of.
>
> It will certainly break a lot.  Last I heard tree-sitter itself has
> problems with complex macro constructs in C and C++ code.
>
> CC Mode has less, because it is too dumb to try to parse them.

Currently, the 'dumb' implementation is default and tree-sitter based
one needs to be setup and enabled by the user (however easy it may be).
If the tree-sitter is in core, and the old 'dumb' way is no longer in
core, can the current way be done the other way around? i.e. disable
tree-sitter from core for cc-mode and add current cc-mode as extra
package and use that the way currently external tree-sitter mode is
used? This way even if tree-sitter is build-time dependency, it won't
affect runtime.

C++ is notoriously difficult to parse for something like tree-sitter due
to context sensitiveness of operators, and I understand your pain. But
for languages with better syntax and semantics, it has been immense
improvement in performance and usefulness over the 'dumb' regex parsing.

--

Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

The Emacs 29 release branch is planned to be cut in about 1.5 months,
and we want tree-sitter support to be in Emacs 29.  The Emacs Git
repository has a feature branch that integrates tree-sitter with
Emacs.  That branch is being prepared for landing on master.  However,
currently there's only one major-mode which can use tree-sitter
support on the branch: Python mode.  We need to add tree-sitter
integration to more major modes, so that Emacs 29 could have
reasonably useful tree-sitter support options in at least the popular
major modes.

This is a call for volunteers to work on adding tree-sitter support to
major modes beyond Python, and on improving the existing support in
Python mode.

Some of the popular major modes for which we would like to see
tree-sitter support in Emacs 29 are:

 c-mode
 c++-mode
 emacs-lisp-mode
 js-mode
 js-json-mode
 ruby-mode
 shell-script-mode

We probably won't have all of them by Emacs 29, but we hope to have at
least some.

So if you are interested in enhancing your favorite major-mode with
tree-sitter support, and by doing that making Emacs 29 much better,
please checkout the feature/tree-sitter branch, and start working on
some mode(s).  Tree-sitter support for a major-mode should include
enhancement of one or more of the following Emacs features:

 - font-lock
 - indentation
 - navigation (beginning-of-defun-function etc.)
 - imenu
 - which-func
 - thing-at-point

We think that font-lock is the absolute minimum, followed by
indentation.

To help with this effort, Yuan Fu, who did most of the development on
this feature branch, wrote a document with guidance notes which aim to
help you implement tree-sitter support for the above features.  You
will find that document and some other helpful documentation in the
admin/notes/tree-sitter directory and its subdirectories on the
branch.  That directory also includes scripts for easy compilation of
tree-sitter language modules (which you will need to download from the
tree-sitter site).

We hope that the existing support for Python can serve as working
example of how to add tree-sitter support for other major modes, and
together with the guidance document it will allow you to add support
for more modes with minimum effort.  Don't hesitate to ask questions
if something is unclear or you need more advice.

We hope volunteers will step forward in time to make tree-sitter
integration in Emacs 29 functional and useful.  To avoid waste of
effort, please announce here on which major mode(s) you decided to
work.

Thanks in advance.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

On 9 October 2022 11:03:48 CEST, Eli Zaretskii <eliz@gnu.org> wrote:
>The Emacs 29 release branch is planned to be cut in about 1.5 months,
>and we want tree-sitter support to be in Emacs 29.  The Emacs Git
>repository has a feature branch that integrates tree-sitter with
>Emacs.  That branch is being prepared for landing on master.  However,
>currently there's only one major-mode which can use tree-sitter
>support on the branch: Python mode.  We need to add tree-sitter
>integration to more major modes, so that Emacs 29 could have
>reasonably useful tree-sitter support options in at least the popular
>major modes.
>
>This is a call for volunteers to work on adding tree-sitter support to
>major modes beyond Python, and on improving the existing support in
>Python mode.
>
>Some of the popular major modes for which we would like to see
>tree-sitter support in Emacs 29 are:
>
> c-mode
> c++-mode
> emacs-lisp-mode
> js-mode
> js-json-mode
> ruby-mode
> shell-script-mode
>

I can do js-mode. I already have made a patch before the api stabilised, and am pretty familiar with the whole thing. I'll look at python and go from that. 


>We probably won't have all of them by Emacs 29, but we hope to have at
>least some.
>
>So if you are interested in enhancing your favorite major-mode with
>tree-sitter support, and by doing that making Emacs 29 much better,
>please checkout the feature/tree-sitter branch, and start working on
>some mode(s).  Tree-sitter support for a major-mode should include
>enhancement of one or more of the following Emacs features:
>
> - font-lock
> - indentation
> - navigation (beginning-of-defun-function etc.)
> - imenu
> - which-func
> - thing-at-point
>
>We think that font-lock is the absolute minimum, followed by
>indentation.
>
>To help with this effort, Yuan Fu, who did most of the development on
>this feature branch, wrote a document with guidance notes which aim to
>help you implement tree-sitter support for the above features.  You
>will find that document and some other helpful documentation in the
>admin/notes/tree-sitter directory and its subdirectories on the
>branch.  That directory also includes scripts for easy compilation of
>tree-sitter language modules (which you will need to download from the
>tree-sitter site).
>
>We hope that the existing support for Python can serve as working
>example of how to add tree-sitter support for other major modes, and
>together with the guidance document it will allow you to add support
>for more modes with minimum effort.  Don't hesitate to ask questions
>if something is unclear or you need more advice.
>
>We hope volunteers will step forward in time to make tree-sitter
>integration in Emacs 29 functional and useful.  To avoid waste of
>effort, please announce here on which major mode(s) you decided to
>work.
>
>Thanks in advance.
>

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> Date: Sun, 09 Oct 2022 11:47:31 +0200
> From: Theodor Thornhill <theo@thornhill.no>
> 
> > c-mode
> > c++-mode
> > emacs-lisp-mode
> > js-mode
> > js-json-mode
> > ruby-mode
> > shell-script-mode
> >
> 
> I can do js-mode. I already have made a patch before the api stabilised, and am pretty familiar with the whole thing. I'll look at python and go from that. 

Thank you very much, we'll appreciate it.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:
>> > shell-script-mode
>> >
>> 
>> I can do js-mode. I already have made a patch before the api
>> stabilised, and am pretty familiar with the whole thing. I'll look at
>> python and go from that.
>
> Thank you very much, we'll appreciate it.

Here you go - how does this look?  There's support for font-locking,
navigation to class, methods and other stuff.  It also indents just
fine, and has an implementation for which-function.

I don't think there's a need for imenu here, though that could be
implemented as well.  Eglot handles imenu just fine, and my guess is
that most people will move over to that when available.

Please see attached patch, I think it is simple enough.

Theodor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-tree-sitter-functionality-to-js-mode.patch --]
[-- Type: text/x-diff, Size: 17455 bytes --]

From dc63d9d57ee3c6f03804ba356caefef84892867f Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Sun, 9 Oct 2022 14:35:47 +0200
Subject: [PATCH] Add tree-sitter functionality to js-mode

* lisp/progmodes/js.el (treesit): Add tree-sitter dependency

(js-use-treesitter): Defcustom to enable tree-sitter as the engine for
font-lock, indentation and more.

(js--treesit-backward-up-list): Helper to enable indenting of empty bodies.

(js--treesit-indent-rules): Rules to be used when indenting JavaScript code.

(js--treesit-keywords): Keywords as seen with tree-sitter.

(js--treesit-settings): Font-Lock settings for JavaScript.

(js-treesit-current-defun): Helper for which-function-mode.

(js--treesit-move-to-node): Helper for movement to specific nodes in
the tree-sitter tree.

(js--treesit-beginning-of-defun, js--treesit-end-of-defun): Movement
functions for js-mode.

(js--treesit-enable): Helper function to enable tree-sitter.  Skips
over most initalization done by the CC-Mode variant.
---
 lisp/progmodes/js.el | 384 ++++++++++++++++++++++++++++++++++---------
 1 file changed, 304 insertions(+), 80 deletions(-)

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index b920ef6c2c..4ec13ca57a 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -53,6 +53,7 @@
 (require 'imenu)
 (require 'json)
 (require 'prog-mode)
+(require 'treesit)
 
 (eval-when-compile
   (require 'cl-lib)
@@ -3400,6 +3401,227 @@ js-jsx--detect-after-change
 (c-lang-defconst c-paragraph-start
   js-mode "\\(@[[:alpha:]]+\\>\\|$\\)")
 
+;;; Tree sitter integration
+(defcustom js-use-treesitter nil
+  "If non-nil, `js-mode' tries to use tree-sitter.
+Currently `js-mode' uses tree-sitter for font-locking,
+indentation, which-function and movement functions."
+  :version "29.1"
+  :type 'boolean
+  :safe 'booleanp)
+
+(defun js--treesit-backward-up-list ()
+  (lambda (_node _parent _bol &rest _)
+    (save-excursion
+      (backward-up-list 1 nil t)
+      (goto-char
+       (treesit-node-start
+        (treesit-node-at (point))))
+      (back-to-indentation)
+      (point))))
+
+(defvar js--treesit-indent-rules
+  `((javascript
+     (no-node (js--treesit-backward-up-list) ,js-indent-level)
+     ((node-is "}") parent-bol 0)
+     ((node-is ")") parent-bol 0)
+     ((node-is "]") parent-bol 0)
+     ((node-is ">") parent-bol 0)
+     ((node-is ".") parent-bol ,js-indent-level)
+     ((parent-is "named_imports") parent-bol ,js-indent-level)
+     ((parent-is "statement_block") parent-bol ,js-indent-level)
+     ((parent-is "variable_declarator") parent-bol ,js-indent-level)
+     ((parent-is "arguments") parent-bol ,js-indent-level)
+     ((parent-is "array") parent-bol ,js-indent-level)
+     ((parent-is "formal_parameters") parent-bol ,js-indent-level)
+     ((parent-is "template_substitution") parent-bol ,js-indent-level)
+     ((parent-is "object_pattern") parent-bol ,js-indent-level)
+     ((parent-is "object") parent-bol ,js-indent-level)
+     ((parent-is "arrow_function") parent-bol ,js-indent-level)
+     ((parent-is "parenthesized_expression") parent-bol ,js-indent-level)
+
+     ;; JSX
+     ((parent-is "jsx_opening_element") parent ,js-indent-level)
+     ((node-is "jsx_closing_element") parent 0)
+     ((node-is "jsx_text") parent ,js-indent-level)
+     ((parent-is "jsx_element") parent ,js-indent-level)
+     ((node-is "/") parent 0)
+     ((parent-is "jsx_self_closing_element") parent ,js-indent-level))))
+
+(defvar js--treesit-keywords
+  '("as" "async" "await" "break" "case" "catch" "class" "const" "continue"
+    "debugger" "default" "delete" "do" "else" "export" "extends" "finally"
+    "for" "from" "function" "get" "if" "import" "in" "instanceof" "let" "new"
+    "of" "return" "set" "static" "switch" "switch" "target" "throw" "try"
+    "typeof" "var" "void" "while" "with" "yield"))
+
+(defvar js--treesit-settings
+  (treesit-font-lock-rules
+   :language 'javascript
+   :override t
+   `(
+     ((identifier) @font-lock-constant-face
+      (:match "^[A-Z_][A-Z_\\d]*$" @font-lock-constant-face))
+
+     (new_expression
+      constructor: (identifier) @font-lock-type-face)
+
+     (function
+      name: (identifier) @font-lock-function-name-face)
+
+     (class_declaration
+      name: (identifier) @font-lock-type-face)
+
+     (function_declaration
+      name: (identifier) @font-lock-function-name-face)
+
+     (method_definition
+      name: (property_identifier) @font-lock-function-name-face)
+
+     (variable_declarator
+      name: (identifier) @font-lock-function-name-face
+      value: [(function) (arrow_function)])
+
+     (variable_declarator
+      name: (array_pattern (identifier) (identifier) @font-lock-function-name-face)
+      value: (array (number) (function)))
+
+     (assignment_expression
+      left: [(identifier) @font-lock-function-name-face
+             (member_expression property: (property_identifier) @font-lock-function-name-face)]
+      right: [(function) (arrow_function)])
+
+     (call_expression
+      function: [(identifier) @font-lock-function-name-face
+                 (member_expression
+                  property: (property_identifier) @font-lock-function-name-face)])
+
+     (variable_declarator
+      name: (identifier) @font-lock-variable-name-face)
+
+     (assignment_expression
+      left: [(identifier) @font-lock-variable-name-face
+             (member_expression property: (property_identifier) @font-lock-variable-name-face)])
+
+     (for_in_statement
+      left: (identifier) @font-lock-variable-name-face)
+
+     (arrow_function
+      parameter: (identifier) @font-lock-variable-name-face)
+
+     (pair key: (property_identifier) @font-lock-variable-name-face)
+
+     (pair value: (identifier) @font-lock-variable-name-face)
+
+     (pair
+      key: (property_identifier) @font-lock-function-name-face
+      value: [(function) (arrow_function)])
+
+     ((shorthand_property_identifier) @font-lock-variable-name-face)
+
+     (pair_pattern key: (property_identifier) @font-lock-variable-name-face)
+
+     ((shorthand_property_identifier_pattern) @font-lock-variable-name-face)
+
+     (array_pattern (identifier) @font-lock-variable-name-face)
+
+     (jsx_opening_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_closing_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_self_closing_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_attribute
+      (property_identifier)
+      @font-lock-constant-face)
+
+     [(this) (super)] @font-lock-keyword-face
+
+     [(true) (false) (null)] @font-lock-constant-face
+     (regex pattern: (regex_pattern)) @font-lock-string-face
+     (number) @font-lock-constant-face
+
+     (string) @font-lock-string-face
+     (comment) @font-lock-comment-face
+     [,@js--treesit-keywords] @font-lock-keyword-face
+
+     ;; FIXME: We need to be able to set the priority for font-locking
+     ;; somehow.  We cannot just override all of the template string,
+     ;; as that would mess up interpolated expressions
+     ;;
+     ;; (template_string) @font-lock-string-face
+     (template_substitution ["${" "}"] @font-lock-constant-face)
+     )))
+
+
+(defun js-treesit-current-defun ()
+  "Return name of surrounding function.
+This function can be used as a value in `which-func-functions'"
+  (let ((node (treesit-node-at (point)))
+        (name-list ()))
+    (cl-loop while node
+             if (pcase (treesit-node-type node)
+                  ("function_declaration" t)
+                  ("method_definition" t)
+                  ("class_declaration" t)
+                  ("variable_declarator" t)
+                  (_ nil))
+             do (push (treesit-node-text
+                       (treesit-node-child-by-field-name node "name")
+                       t)
+                      name-list)
+             do (setq node (treesit-node-parent node))
+             finally return  (string-join name-list "."))))
+
+
+(defun js--treesit-move-to-node (fn)
+  (when-let ((found-node
+              (treesit-parent-until
+               (treesit-node-at (point))
+               (lambda (parent)
+                 (let ((parent-type (treesit-node-type parent)))
+                   (or (equal "class_declaration" parent-type)
+                       (equal "method_definition" parent-type)
+                       (equal "function_declaration" parent-type)
+                       (equal "variable_declarator" parent-type)
+                       ;; TODO: More declarations here
+                       ))))))
+    (goto-char (funcall fn found-node))))
+
+(defun js--treesit-beginning-of-defun (&optional _arg)
+  (js--treesit-move-to-node #'treesit-node-start))
+
+(defun js--treesit-end-of-defun (&optional _arg)
+  (js--treesit-move-to-node #'treesit-node-end))
+
+(defun js--treesit-enable ()
+  (unless (and (treesit-can-enable-p)
+              (treesit-language-available-p 'javascript))
+    (error "Tree sitter isn't available"))
+
+  ;; Comments
+  (setq-local comment-start "// ")
+  (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
+  (setq-local comment-end "")
+
+  (setq-local treesit-simple-indent-rules js--treesit-indent-rules)
+  (setq-local indent-line-function #'treesit-indent)
+  (setq-local beginning-of-defun-function #'js--treesit-beginning-of-defun)
+  (setq-local end-of-defun-function #'js--treesit-end-of-defun)
+
+  (setq-local font-lock-defaults '(nil t))
+  (setq-local treesit-font-lock-settings js--treesit-settings)
+
+  (add-hook 'which-func-functions #'js-treesit-current-defun nil t)
+
+  (treesit-font-lock-enable))
+
 ;;; Main Function
 
 ;;;###autoload
@@ -3407,86 +3629,88 @@ js-mode
   "Major mode for editing JavaScript."
   :group 'js
   ;; Ensure all CC Mode "lang variables" are set to valid values.
-  (c-init-language-vars js-mode)
-  (setq-local indent-line-function #'js-indent-line)
-  (setq-local beginning-of-defun-function #'js-beginning-of-defun)
-  (setq-local end-of-defun-function #'js-end-of-defun)
-  (setq-local open-paren-in-column-0-is-defun-start nil)
-  (setq-local font-lock-defaults
-              (list js--font-lock-keywords nil nil nil nil
-                    '(font-lock-syntactic-face-function
-                      . js-font-lock-syntactic-face-function)))
-  (setq-local syntax-propertize-function #'js-syntax-propertize)
-  (add-hook 'syntax-propertize-extend-region-functions
-            #'syntax-propertize-multiline 'append 'local)
-  (add-hook 'syntax-propertize-extend-region-functions
-            #'js--syntax-propertize-extend-region 'append 'local)
-  (setq-local prettify-symbols-alist js--prettify-symbols-alist)
-
-  (setq-local parse-sexp-ignore-comments t)
-  (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
-
-  ;; Comments
-  (setq-local comment-start "// ")
-  (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
-  (setq-local comment-end "")
-  (setq-local fill-paragraph-function #'js-fill-paragraph)
-  (setq-local normal-auto-fill-function #'js-do-auto-fill)
-
-  ;; Parse cache
-  (add-hook 'before-change-functions #'js--flush-caches t t)
-
-  ;; Frameworks
-  (js--update-quick-match-re)
-
-  ;; Syntax extensions
-  (unless (js-jsx--detect-and-enable)
-    (add-hook 'after-change-functions #'js-jsx--detect-after-change nil t))
-  (js-use-syntactic-mode-name)
-
-  ;; Imenu
-  (setq imenu-case-fold-search nil)
-  (setq imenu-create-index-function #'js--imenu-create-index)
-
-  ;; for filling, pretend we're cc-mode
-  (c-foreign-init-lit-pos-cache)
-  (add-hook 'before-change-functions #'c-foreign-truncate-lit-pos-cache nil t)
-  (setq-local comment-line-break-function #'c-indent-new-comment-line)
-  (setq-local comment-multi-line t)
-  (setq-local electric-indent-chars
-	      (append "{}():;," electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
-  (setq-local electric-layout-rules
-	      '((?\; . after) (?\{ . after) (?\} . before)))
-
-  (let ((c-buffer-is-cc-mode t))
-    ;; FIXME: These are normally set by `c-basic-common-init'.  Should
-    ;; we call it instead?  (Bug#6071)
-    (make-local-variable 'paragraph-start)
-    (make-local-variable 'paragraph-separate)
-    (make-local-variable 'paragraph-ignore-fill-prefix)
-    (make-local-variable 'adaptive-fill-mode)
-    (make-local-variable 'adaptive-fill-regexp)
-    ;; While the full CC Mode style system is not yet in use, set the
-    ;; pertinent style variables manually.
-    (c-initialize-builtin-style)
-    (let ((style (cc-choose-style-for-mode 'js-mode c-default-style)))
-      (c-set-style style))
-    (setq c-block-comment-prefix "* "
-          c-comment-prefix-regexp "//+\\|\\**")
-    (c-setup-paragraph-variables))
-
-  ;; Important to fontify the whole buffer syntactically! If we don't,
-  ;; then we might have regular expression literals that aren't marked
-  ;; as strings, which will screw up parse-partial-sexp, scan-lists,
-  ;; etc. and produce maddening "unbalanced parenthesis" errors.
-  ;; When we attempt to find the error and scroll to the portion of
-  ;; the buffer containing the problem, JIT-lock will apply the
-  ;; correct syntax to the regular expression literal and the problem
-  ;; will mysteriously disappear.
-  ;; FIXME: We should instead do this fontification lazily by adding
-  ;; calls to syntax-propertize wherever it's really needed.
-  ;;(syntax-propertize (point-max))
-  )
+  (if js-use-treesitter
+      (js--treesit-enable)
+    (c-init-language-vars js-mode)
+    (setq-local indent-line-function #'js-indent-line)
+    (setq-local beginning-of-defun-function #'js-beginning-of-defun)
+    (setq-local end-of-defun-function #'js-end-of-defun)
+    (setq-local open-paren-in-column-0-is-defun-start nil)
+    (setq-local font-lock-defaults
+                (list js--font-lock-keywords nil nil nil nil
+                      '(font-lock-syntactic-face-function
+                        . js-font-lock-syntactic-face-function)))
+    (setq-local syntax-propertize-function #'js-syntax-propertize)
+    (add-hook 'syntax-propertize-extend-region-functions
+              #'syntax-propertize-multiline 'append 'local)
+    (add-hook 'syntax-propertize-extend-region-functions
+              #'js--syntax-propertize-extend-region 'append 'local)
+    (setq-local prettify-symbols-alist js--prettify-symbols-alist)
+
+    (setq-local parse-sexp-ignore-comments t)
+    (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
+
+    ;; Comments
+    (setq-local comment-start "// ")
+    (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
+    (setq-local comment-end "")
+    (setq-local fill-paragraph-function #'js-fill-paragraph)
+    (setq-local normal-auto-fill-function #'js-do-auto-fill)
+
+    ;; Parse cache
+    (add-hook 'before-change-functions #'js--flush-caches t t)
+
+    ;; Frameworks
+    (js--update-quick-match-re)
+
+    ;; Syntax extensions
+    (unless (js-jsx--detect-and-enable)
+      (add-hook 'after-change-functions #'js-jsx--detect-after-change nil t))
+    (js-use-syntactic-mode-name)
+
+    ;; Imenu
+    (setq imenu-case-fold-search nil)
+    (setq imenu-create-index-function #'js--imenu-create-index)
+
+    ;; for filling, pretend we're cc-mode
+    (c-foreign-init-lit-pos-cache)
+    (add-hook 'before-change-functions #'c-foreign-truncate-lit-pos-cache nil t)
+    (setq-local comment-line-break-function #'c-indent-new-comment-line)
+    (setq-local comment-multi-line t)
+    (setq-local electric-indent-chars
+	        (append "{}():;," electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
+    (setq-local electric-layout-rules
+	        '((?\; . after) (?\{ . after) (?\} . before)))
+
+    (let ((c-buffer-is-cc-mode t))
+      ;; FIXME: These are normally set by `c-basic-common-init'.  Should
+      ;; we call it instead?  (Bug#6071)
+      (make-local-variable 'paragraph-start)
+      (make-local-variable 'paragraph-separate)
+      (make-local-variable 'paragraph-ignore-fill-prefix)
+      (make-local-variable 'adaptive-fill-mode)
+      (make-local-variable 'adaptive-fill-regexp)
+      ;; While the full CC Mode style system is not yet in use, set the
+      ;; pertinent style variables manually.
+      (c-initialize-builtin-style)
+      (let ((style (cc-choose-style-for-mode 'js-mode c-default-style)))
+        (c-set-style style))
+      (setq c-block-comment-prefix "* "
+            c-comment-prefix-regexp "//+\\|\\**")
+      (c-setup-paragraph-variables))
+
+    ;; Important to fontify the whole buffer syntactically! If we don't,
+    ;; then we might have regular expression literals that aren't marked
+    ;; as strings, which will screw up parse-partial-sexp, scan-lists,
+    ;; etc. and produce maddening "unbalanced parenthesis" errors.
+    ;; When we attempt to find the error and scroll to the portion of
+    ;; the buffer containing the problem, JIT-lock will apply the
+    ;; correct syntax to the regular expression literal and the problem
+    ;; will mysteriously disappear.
+    ;; FIXME: We should instead do this fontification lazily by adding
+    ;; calls to syntax-propertize wherever it's really needed.
+    ;;(syntax-propertize (point-max))
+    ))
 
 ;;;###autoload
 (define-derived-mode js-json-mode js-mode "JSON"
-- 
2.34.1

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: emacs-devel@gnu.org
> Date: Sun, 09 Oct 2022 14:41:28 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> > shell-script-mode
> >> >
> >> 
> >> I can do js-mode. I already have made a patch before the api
> >> stabilised, and am pretty familiar with the whole thing. I'll look at
> >> python and go from that.
> >
> > Thank you very much, we'll appreciate it.
> 
> Here you go - how does this look?  There's support for font-locking,
> navigation to class, methods and other stuff.  It also indents just
> fine, and has an implementation for which-function.

Thanks.  Yuan, any comments?  Feel free to install when you are happy
with this.

> I don't think there's a need for imenu here, though that could be
> implemented as well.  Eglot handles imenu just fine, and my guess is
> that most people will move over to that when available.

Fair enough.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

On 9 October 2022 16:04:12 CEST, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: emacs-devel@gnu.org
>> Date: Sun, 09 Oct 2022 14:41:28 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> >> > shell-script-mode
>> >> >
>> >> 
>> >> I can do js-mode. I already have made a patch before the api
>> >> stabilised, and am pretty familiar with the whole thing. I'll look at
>> >> python and go from that.
>> >
>> > Thank you very much, we'll appreciate it.
>> 
>> Here you go - how does this look?  There's support for font-locking,
>> navigation to class, methods and other stuff.  It also indents just
>> fine, and has an implementation for which-function.
>
>Thanks.  Yuan, any comments?  Feel free to install when you are happy
>with this.
>
>> I don't think there's a need for imenu here, though that could be
>> implemented as well.  Eglot handles imenu just fine, and my guess is
>> that most people will move over to that when available.
>
>Fair enough.

BTW, I can do json as well, should be quick enough :) I'll add it in a different patch

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> Date: Sun, 09 Oct 2022 17:18:52 +0200
> From: Theodor Thornhill <theo@thornhill.no>
> CC: emacs-devel@gnu.org
> 
> BTW, I can do json as well, should be quick enough :) I'll add it in a different patch

TIA

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sun, 09 Oct 2022 17:18:52 +0200
>> From: Theodor Thornhill <theo@thornhill.no>
>> CC: emacs-devel@gnu.org
>> 
>> BTW, I can do json as well, should be quick enough :) I'll add it in a different patch
>
> TIA

So I added tree sitter support for js-json-mode, js-jsx-mode as well.
The changes were simple enough not to warrant a separate patch, IMO, so
I just updated the old one.

What do you think, Yuan and Eli?

Theo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-tree-sitter-functionality-to-js-mode.patch --]
[-- Type: text/x-diff, Size: 20705 bytes --]

From 324b53ac9e4fe90bfa466a237189c5c1182e9e17 Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Sun, 9 Oct 2022 21:22:55 +0200
Subject: [PATCH] Add tree-sitter functionality to js-mode

* lisp/progmodes/js.el (treesit): Add tree-sitter dependency

(js-use-treesitter): Defcustom to enable tree-sitter as the engine for
font-lock, indentation and more.

(js--treesit-backward-up-list): Helper to enable indenting of empty bodies.

(js--treesit-indent-rules): Rules to be used when indenting JavaScript code.

(js--treesit-keywords): Keywords as seen with tree-sitter.

(js--treesit-settings): Font-Lock settings for JavaScript.

(js-treesit-current-defun): Helper for which-function-mode.

(js--treesit-move-to-node): Helper for movement to specific nodes in
the tree-sitter tree.

(js--treesit-beginning-of-defun, js--treesit-end-of-defun): Movement
functions for js-mode.

(js--treesit-enable): Helper function to enable tree-sitter.  Skips
over most initalization done by the CC-Mode variant.

(js-json-use-treesitter): Defcustom to enable tree-sitter as the
engine for font-lock and indentation for JSON files.

(js--json-treesit-settings): Font-Lock settings for JSON.

(js--json-treesit-indent-rules): Rules to be used when indenting JSON code

(js--json-treesit-enable): Helper function to enable tree-sitter.
Skips over most initialization done by the CC-Mode variant, in
addition to ignoring some JavaScript specific settings.

(js-mode, js-json-mode, js-jsx-mode): Add support for tree-sitter
functionalities.
---
 lisp/progmodes/js.el | 450 +++++++++++++++++++++++++++++++++++--------
 1 file changed, 368 insertions(+), 82 deletions(-)

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index b920ef6c2c..ca6cadec10 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -53,6 +53,7 @@
 (require 'imenu)
 (require 'json)
 (require 'prog-mode)
+(require 'treesit)
 
 (eval-when-compile
   (require 'cl-lib)
@@ -3400,6 +3401,227 @@ js-jsx--detect-after-change
 (c-lang-defconst c-paragraph-start
   js-mode "\\(@[[:alpha:]]+\\>\\|$\\)")
 
+;;; Tree sitter integration
+(defcustom js-use-treesitter nil
+  "If non-nil, `js-mode' tries to use tree-sitter.
+Currently `js-mode' uses tree-sitter for font-locking,
+indentation, which-function and movement functions."
+  :version "29.1"
+  :type 'boolean
+  :safe 'booleanp)
+
+(defun js--treesit-backward-up-list ()
+  (lambda (_node _parent _bol &rest _)
+    (save-excursion
+      (backward-up-list 1 nil t)
+      (goto-char
+       (treesit-node-start
+        (treesit-node-at (point))))
+      (back-to-indentation)
+      (point))))
+
+(defvar js--treesit-indent-rules
+  `((javascript
+     (no-node (js--treesit-backward-up-list) ,js-indent-level)
+     ((node-is "}") parent-bol 0)
+     ((node-is ")") parent-bol 0)
+     ((node-is "]") parent-bol 0)
+     ((node-is ">") parent-bol 0)
+     ((node-is ".") parent-bol ,js-indent-level)
+     ((parent-is "named_imports") parent-bol ,js-indent-level)
+     ((parent-is "statement_block") parent-bol ,js-indent-level)
+     ((parent-is "variable_declarator") parent-bol ,js-indent-level)
+     ((parent-is "arguments") parent-bol ,js-indent-level)
+     ((parent-is "array") parent-bol ,js-indent-level)
+     ((parent-is "formal_parameters") parent-bol ,js-indent-level)
+     ((parent-is "template_substitution") parent-bol ,js-indent-level)
+     ((parent-is "object_pattern") parent-bol ,js-indent-level)
+     ((parent-is "object") parent-bol ,js-indent-level)
+     ((parent-is "arrow_function") parent-bol ,js-indent-level)
+     ((parent-is "parenthesized_expression") parent-bol ,js-indent-level)
+
+     ;; JSX
+     ((parent-is "jsx_opening_element") parent ,js-indent-level)
+     ((node-is "jsx_closing_element") parent 0)
+     ((node-is "jsx_text") parent ,js-indent-level)
+     ((parent-is "jsx_element") parent ,js-indent-level)
+     ((node-is "/") parent 0)
+     ((parent-is "jsx_self_closing_element") parent ,js-indent-level))))
+
+(defvar js--treesit-keywords
+  '("as" "async" "await" "break" "case" "catch" "class" "const" "continue"
+    "debugger" "default" "delete" "do" "else" "export" "extends" "finally"
+    "for" "from" "function" "get" "if" "import" "in" "instanceof" "let" "new"
+    "of" "return" "set" "static" "switch" "switch" "target" "throw" "try"
+    "typeof" "var" "void" "while" "with" "yield"))
+
+(defvar js--treesit-settings
+  (treesit-font-lock-rules
+   :language 'javascript
+   :override t
+   `(
+     ((identifier) @font-lock-constant-face
+      (:match "^[A-Z_][A-Z_\\d]*$" @font-lock-constant-face))
+
+     (new_expression
+      constructor: (identifier) @font-lock-type-face)
+
+     (function
+      name: (identifier) @font-lock-function-name-face)
+
+     (class_declaration
+      name: (identifier) @font-lock-type-face)
+
+     (function_declaration
+      name: (identifier) @font-lock-function-name-face)
+
+     (method_definition
+      name: (property_identifier) @font-lock-function-name-face)
+
+     (variable_declarator
+      name: (identifier) @font-lock-function-name-face
+      value: [(function) (arrow_function)])
+
+     (variable_declarator
+      name: (array_pattern (identifier) (identifier) @font-lock-function-name-face)
+      value: (array (number) (function)))
+
+     (assignment_expression
+      left: [(identifier) @font-lock-function-name-face
+             (member_expression property: (property_identifier) @font-lock-function-name-face)]
+      right: [(function) (arrow_function)])
+
+     (call_expression
+      function: [(identifier) @font-lock-function-name-face
+                 (member_expression
+                  property: (property_identifier) @font-lock-function-name-face)])
+
+     (variable_declarator
+      name: (identifier) @font-lock-variable-name-face)
+
+     (assignment_expression
+      left: [(identifier) @font-lock-variable-name-face
+             (member_expression property: (property_identifier) @font-lock-variable-name-face)])
+
+     (for_in_statement
+      left: (identifier) @font-lock-variable-name-face)
+
+     (arrow_function
+      parameter: (identifier) @font-lock-variable-name-face)
+
+     (pair key: (property_identifier) @font-lock-variable-name-face)
+
+     (pair value: (identifier) @font-lock-variable-name-face)
+
+     (pair
+      key: (property_identifier) @font-lock-function-name-face
+      value: [(function) (arrow_function)])
+
+     ((shorthand_property_identifier) @font-lock-variable-name-face)
+
+     (pair_pattern key: (property_identifier) @font-lock-variable-name-face)
+
+     ((shorthand_property_identifier_pattern) @font-lock-variable-name-face)
+
+     (array_pattern (identifier) @font-lock-variable-name-face)
+
+     (jsx_opening_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_closing_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_self_closing_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_attribute
+      (property_identifier)
+      @font-lock-constant-face)
+
+     [(this) (super)] @font-lock-keyword-face
+
+     [(true) (false) (null)] @font-lock-constant-face
+     (regex pattern: (regex_pattern)) @font-lock-string-face
+     (number) @font-lock-constant-face
+
+     (string) @font-lock-string-face
+     (comment) @font-lock-comment-face
+     [,@js--treesit-keywords] @font-lock-keyword-face
+
+     ;; FIXME: We need to be able to set the priority for font-locking
+     ;; somehow.  We cannot just override all of the template string,
+     ;; as that would mess up interpolated expressions
+     ;;
+     ;; (template_string) @font-lock-string-face
+     (template_substitution ["${" "}"] @font-lock-constant-face)
+     )))
+
+
+(defun js-treesit-current-defun ()
+  "Return name of surrounding function.
+This function can be used as a value in `which-func-functions'"
+  (let ((node (treesit-node-at (point)))
+        (name-list ()))
+    (cl-loop while node
+             if (pcase (treesit-node-type node)
+                  ("function_declaration" t)
+                  ("method_definition" t)
+                  ("class_declaration" t)
+                  ("variable_declarator" t)
+                  (_ nil))
+             do (push (treesit-node-text
+                       (treesit-node-child-by-field-name node "name")
+                       t)
+                      name-list)
+             do (setq node (treesit-node-parent node))
+             finally return  (string-join name-list "."))))
+
+
+(defun js--treesit-move-to-node (fn)
+  (when-let ((found-node
+              (treesit-parent-until
+               (treesit-node-at (point))
+               (lambda (parent)
+                 (let ((parent-type (treesit-node-type parent)))
+                   (or (equal "class_declaration" parent-type)
+                       (equal "method_definition" parent-type)
+                       (equal "function_declaration" parent-type)
+                       (equal "variable_declarator" parent-type)
+                       ;; TODO: More declarations here
+                       ))))))
+    (goto-char (funcall fn found-node))))
+
+(defun js--treesit-beginning-of-defun (&optional _arg)
+  (js--treesit-move-to-node #'treesit-node-start))
+
+(defun js--treesit-end-of-defun (&optional _arg)
+  (js--treesit-move-to-node #'treesit-node-end))
+
+(defun js--treesit-enable ()
+  (unless (and (treesit-can-enable-p)
+               (treesit-language-available-p 'javascript))
+    (error "Tree sitter isn't available"))
+
+  ;; Comments
+  (setq-local comment-start "// ")
+  (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
+  (setq-local comment-end "")
+
+  (setq-local treesit-simple-indent-rules js--treesit-indent-rules)
+  (setq-local indent-line-function #'treesit-indent)
+  (setq-local beginning-of-defun-function #'js--treesit-beginning-of-defun)
+  (setq-local end-of-defun-function #'js--treesit-end-of-defun)
+
+  (setq-local font-lock-defaults '(nil t))
+  (setq-local treesit-font-lock-settings js--treesit-settings)
+
+  (add-hook 'which-func-functions #'js-treesit-current-defun nil t)
+
+  (treesit-font-lock-enable))
+
 ;;; Main Function
 
 ;;;###autoload
@@ -3407,93 +3629,155 @@ js-mode
   "Major mode for editing JavaScript."
   :group 'js
   ;; Ensure all CC Mode "lang variables" are set to valid values.
-  (c-init-language-vars js-mode)
-  (setq-local indent-line-function #'js-indent-line)
-  (setq-local beginning-of-defun-function #'js-beginning-of-defun)
-  (setq-local end-of-defun-function #'js-end-of-defun)
-  (setq-local open-paren-in-column-0-is-defun-start nil)
-  (setq-local font-lock-defaults
-              (list js--font-lock-keywords nil nil nil nil
-                    '(font-lock-syntactic-face-function
-                      . js-font-lock-syntactic-face-function)))
-  (setq-local syntax-propertize-function #'js-syntax-propertize)
-  (add-hook 'syntax-propertize-extend-region-functions
-            #'syntax-propertize-multiline 'append 'local)
-  (add-hook 'syntax-propertize-extend-region-functions
-            #'js--syntax-propertize-extend-region 'append 'local)
-  (setq-local prettify-symbols-alist js--prettify-symbols-alist)
-
-  (setq-local parse-sexp-ignore-comments t)
-  (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
+  (if js-use-treesitter
+      (js--treesit-enable)
+    (c-init-language-vars js-mode)
+    (setq-local indent-line-function #'js-indent-line)
+    (setq-local beginning-of-defun-function #'js-beginning-of-defun)
+    (setq-local end-of-defun-function #'js-end-of-defun)
+    (setq-local open-paren-in-column-0-is-defun-start nil)
+    (setq-local font-lock-defaults
+                (list js--font-lock-keywords nil nil nil nil
+                      '(font-lock-syntactic-face-function
+                        . js-font-lock-syntactic-face-function)))
+    (setq-local syntax-propertize-function #'js-syntax-propertize)
+    (add-hook 'syntax-propertize-extend-region-functions
+              #'syntax-propertize-multiline 'append 'local)
+    (add-hook 'syntax-propertize-extend-region-functions
+              #'js--syntax-propertize-extend-region 'append 'local)
+    (setq-local prettify-symbols-alist js--prettify-symbols-alist)
+
+    (setq-local parse-sexp-ignore-comments t)
+    (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
+
+    ;; Comments
+    (setq-local comment-start "// ")
+    (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
+    (setq-local comment-end "")
+    (setq-local fill-paragraph-function #'js-fill-paragraph)
+    (setq-local normal-auto-fill-function #'js-do-auto-fill)
+
+    ;; Parse cache
+    (add-hook 'before-change-functions #'js--flush-caches t t)
+
+    ;; Frameworks
+    (js--update-quick-match-re)
+
+    ;; Syntax extensions
+    (unless (js-jsx--detect-and-enable)
+      (add-hook 'after-change-functions #'js-jsx--detect-after-change nil t))
+    (js-use-syntactic-mode-name)
+
+    ;; Imenu
+    (setq imenu-case-fold-search nil)
+    (setq imenu-create-index-function #'js--imenu-create-index)
+
+    ;; for filling, pretend we're cc-mode
+    (c-foreign-init-lit-pos-cache)
+    (add-hook 'before-change-functions #'c-foreign-truncate-lit-pos-cache nil t)
+    (setq-local comment-line-break-function #'c-indent-new-comment-line)
+    (setq-local comment-multi-line t)
+    (setq-local electric-indent-chars
+	        (append "{}():;," electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
+    (setq-local electric-layout-rules
+	        '((?\; . after) (?\{ . after) (?\} . before)))
+
+    (let ((c-buffer-is-cc-mode t))
+      ;; FIXME: These are normally set by `c-basic-common-init'.  Should
+      ;; we call it instead?  (Bug#6071)
+      (make-local-variable 'paragraph-start)
+      (make-local-variable 'paragraph-separate)
+      (make-local-variable 'paragraph-ignore-fill-prefix)
+      (make-local-variable 'adaptive-fill-mode)
+      (make-local-variable 'adaptive-fill-regexp)
+      ;; While the full CC Mode style system is not yet in use, set the
+      ;; pertinent style variables manually.
+      (c-initialize-builtin-style)
+      (let ((style (cc-choose-style-for-mode 'js-mode c-default-style)))
+        (c-set-style style))
+      (setq c-block-comment-prefix "* "
+            c-comment-prefix-regexp "//+\\|\\**")
+      (c-setup-paragraph-variables))
+
+    ;; Important to fontify the whole buffer syntactically! If we don't,
+    ;; then we might have regular expression literals that aren't marked
+    ;; as strings, which will screw up parse-partial-sexp, scan-lists,
+    ;; etc. and produce maddening "unbalanced parenthesis" errors.
+    ;; When we attempt to find the error and scroll to the portion of
+    ;; the buffer containing the problem, JIT-lock will apply the
+    ;; correct syntax to the regular expression literal and the problem
+    ;; will mysteriously disappear.
+    ;; FIXME: We should instead do this fontification lazily by adding
+    ;; calls to syntax-propertize wherever it's really needed.
+    ;;(syntax-propertize (point-max))
+    ))
+
+(defcustom js-json-use-treesitter nil
+  "If non-nil, `js-json-mode' tries to use tree-sitter.
+Currently `js-json-mode' uses tree-sitter for font-locking and
+indentation."
+  :version "29.1"
+  :type 'boolean
+  :safe 'booleanp)
+
+(defvar js--json-treesit-settings
+  (treesit-font-lock-rules
+   :language 'json
+   :override t
+   `(
+     (pair
+      key: (_) @font-lock-string-face)
+
+     (string) @font-lock-string-face
+
+     (number) @font-lock-constant-face
+
+     [(null) (true) (false)] @font-lock-constant-face
+
+     (escape_sequence) @font-lock-constant-face
+
+     (comment) @font-lock-comment-face
+     )))
+
+
+(defvar js--json-treesit-indent-rules
+  `((json
+     (no-node (js--treesit-backward-up-list) ,js-indent-level)
+     ((node-is "}") parent-bol 0)
+     ((node-is ")") parent-bol 0)
+     ((node-is "]") parent-bol 0)
+     ((parent-is "object") parent-bol ,js-indent-level)
+     )))
+
+(defun js--json-treesit-enable ()
+  (unless (and (treesit-can-enable-p)
+               (treesit-language-available-p 'json))
+    (error "Tree sitter isn't available"))
 
   ;; Comments
   (setq-local comment-start "// ")
   (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
   (setq-local comment-end "")
-  (setq-local fill-paragraph-function #'js-fill-paragraph)
-  (setq-local normal-auto-fill-function #'js-do-auto-fill)
-
-  ;; Parse cache
-  (add-hook 'before-change-functions #'js--flush-caches t t)
-
-  ;; Frameworks
-  (js--update-quick-match-re)
-
-  ;; Syntax extensions
-  (unless (js-jsx--detect-and-enable)
-    (add-hook 'after-change-functions #'js-jsx--detect-after-change nil t))
-  (js-use-syntactic-mode-name)
-
-  ;; Imenu
-  (setq imenu-case-fold-search nil)
-  (setq imenu-create-index-function #'js--imenu-create-index)
-
-  ;; for filling, pretend we're cc-mode
-  (c-foreign-init-lit-pos-cache)
-  (add-hook 'before-change-functions #'c-foreign-truncate-lit-pos-cache nil t)
-  (setq-local comment-line-break-function #'c-indent-new-comment-line)
-  (setq-local comment-multi-line t)
-  (setq-local electric-indent-chars
-	      (append "{}():;," electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
-  (setq-local electric-layout-rules
-	      '((?\; . after) (?\{ . after) (?\} . before)))
-
-  (let ((c-buffer-is-cc-mode t))
-    ;; FIXME: These are normally set by `c-basic-common-init'.  Should
-    ;; we call it instead?  (Bug#6071)
-    (make-local-variable 'paragraph-start)
-    (make-local-variable 'paragraph-separate)
-    (make-local-variable 'paragraph-ignore-fill-prefix)
-    (make-local-variable 'adaptive-fill-mode)
-    (make-local-variable 'adaptive-fill-regexp)
-    ;; While the full CC Mode style system is not yet in use, set the
-    ;; pertinent style variables manually.
-    (c-initialize-builtin-style)
-    (let ((style (cc-choose-style-for-mode 'js-mode c-default-style)))
-      (c-set-style style))
-    (setq c-block-comment-prefix "* "
-          c-comment-prefix-regexp "//+\\|\\**")
-    (c-setup-paragraph-variables))
-
-  ;; Important to fontify the whole buffer syntactically! If we don't,
-  ;; then we might have regular expression literals that aren't marked
-  ;; as strings, which will screw up parse-partial-sexp, scan-lists,
-  ;; etc. and produce maddening "unbalanced parenthesis" errors.
-  ;; When we attempt to find the error and scroll to the portion of
-  ;; the buffer containing the problem, JIT-lock will apply the
-  ;; correct syntax to the regular expression literal and the problem
-  ;; will mysteriously disappear.
-  ;; FIXME: We should instead do this fontification lazily by adding
-  ;; calls to syntax-propertize wherever it's really needed.
-  ;;(syntax-propertize (point-max))
-  )
+
+  (setq-local treesit-simple-indent-rules js--json-treesit-indent-rules)
+  (setq-local indent-line-function #'treesit-indent)
+  (setq-local beginning-of-defun-function #'ignore)
+  (setq-local end-of-defun-function #'ignore)
+
+  (setq-local font-lock-defaults '(nil t))
+  (setq-local treesit-font-lock-settings js--json-treesit-settings)
+
+  (treesit-font-lock-enable))
+
 
 ;;;###autoload
 (define-derived-mode js-json-mode js-mode "JSON"
-  (setq-local js-enabled-frameworks nil)
-  ;; Speed up `syntax-ppss': JSON files can be big but can't hold
-  ;; regexp matchers nor #! thingies (and `js-enabled-frameworks' is nil).
-  (setq-local syntax-propertize-function #'ignore))
+  (if js-json-use-treesitter
+      (js--json-treesit-enable)
+    (setq-local js-enabled-frameworks nil)
+    ;; Speed up `syntax-ppss': JSON files can be big but can't hold
+    ;; regexp matchers nor #! thingies (and `js-enabled-frameworks' is nil).
+    (setq-local syntax-propertize-function #'ignore)))
 
 ;; Since we made JSX support available and automatically-enabled in
 ;; the base `js-mode' (for ease of use), now `js-jsx-mode' simply
@@ -3520,9 +3804,11 @@ js-jsx-mode
 `js-jsx-enable' in `js-mode-hook'.  You may be better served by
 one of the aforementioned options instead of using this mode."
   :group 'js
-  (js-jsx-enable)
-  (setq-local comment-region-function #'js-jsx--comment-region)
-  (js-use-syntactic-mode-name))
+  (if js-use-treesitter
+      (js--treesit-enable)
+    (js-jsx-enable)
+    (setq-local comment-region-function #'js-jsx--comment-region)
+    (js-use-syntactic-mode-name)))
 
 (defun js-jsx--comment-region (beg end &optional arg)
   (if (or (js-jsx--context)
-- 
2.34.1

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

Theodor Thornhill <theo@thornhill.no> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Date: Sun, 09 Oct 2022 17:18:52 +0200
>>> From: Theodor Thornhill <theo@thornhill.no>
>>> CC: emacs-devel@gnu.org
>>> 
>>> BTW, I can do json as well, should be quick enough :) I'll add it in a different patch
>>
>> TIA
>
> So I added tree sitter support for js-json-mode, js-jsx-mode as well.
> The changes were simple enough not to warrant a separate patch, IMO, so
> I just updated the old one.
>
> What do you think, Yuan and Eli?

A small followup patch - this should be the last to consider for now -
sorry the many patches :)

Theodor

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-tree-sitter-functionality-to-js-mode.patch --]
[-- Type: text/x-diff, Size: 20555 bytes --]

From cf56a8b91f91b8714a7ae7a88458edbe000878ce Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Sun, 9 Oct 2022 21:22:55 +0200
Subject: [PATCH] Add tree-sitter functionality to js-mode

* lisp/progmodes/js.el (treesit): Add tree-sitter dependency

(js-use-treesitter): Defcustom to enable tree-sitter as the engine for
font-lock, indentation and more.

(js--treesit-backward-up-list): Helper to enable indenting of empty bodies.

(js--treesit-indent-rules): Rules to be used when indenting JavaScript code.

(js--treesit-keywords): Keywords as seen with tree-sitter.

(js--treesit-settings): Font-Lock settings for JavaScript.

(js-treesit-current-defun): Helper for which-function-mode.

(js--treesit-move-to-node): Helper for movement to specific nodes in
the tree-sitter tree.

(js--treesit-beginning-of-defun, js--treesit-end-of-defun): Movement
functions for js-mode.

(js--treesit-enable): Helper function to enable tree-sitter.  Skips
over most initalization done by the CC-Mode variant.

(js-json-use-treesitter): Defcustom to enable tree-sitter as the
engine for font-lock and indentation for JSON files.

(js--json-treesit-settings): Font-Lock settings for JSON.

(js--json-treesit-indent-rules): Rules to be used when indenting JSON code

(js--json-treesit-enable): Helper function to enable tree-sitter.
Skips over most initialization done by the CC-Mode variant, in
addition to ignoring some JavaScript specific settings.

(js-mode, js-json-mode, js-jsx-mode): Add support for tree-sitter
functionalities.
---
 lisp/progmodes/js.el | 451 +++++++++++++++++++++++++++++++++++--------
 1 file changed, 369 insertions(+), 82 deletions(-)

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index b920ef6c2c..2dda729b7b 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -53,6 +53,7 @@
 (require 'imenu)
 (require 'json)
 (require 'prog-mode)
+(require 'treesit)
 
 (eval-when-compile
   (require 'cl-lib)
@@ -3400,6 +3401,228 @@ js-jsx--detect-after-change
 (c-lang-defconst c-paragraph-start
   js-mode "\\(@[[:alpha:]]+\\>\\|$\\)")
 
+;;; Tree sitter integration
+(defcustom js-use-treesitter nil
+  "If non-nil, `js-mode' tries to use tree-sitter.
+Currently `js-mode' uses tree-sitter for font-locking,
+indentation, which-function and movement functions."
+  :version "29.1"
+  :type 'boolean
+  :safe 'booleanp)
+
+(defun js--treesit-backward-up-list ()
+  (lambda (_node _parent _bol &rest _)
+    (save-excursion
+      (backward-up-list 1 nil t)
+      (goto-char
+       (treesit-node-start
+        (treesit-node-at (point))))
+      (back-to-indentation)
+      (point))))
+
+(defvar js--treesit-indent-rules
+  `((javascript
+     (no-node (js--treesit-backward-up-list) ,js-indent-level)
+     ((node-is "}") parent-bol 0)
+     ((node-is ")") parent-bol 0)
+     ((node-is "]") parent-bol 0)
+     ((node-is ">") parent-bol 0)
+     ((node-is ".") parent-bol ,js-indent-level)
+     ((parent-is "named_imports") parent-bol ,js-indent-level)
+     ((parent-is "statement_block") parent-bol ,js-indent-level)
+     ((parent-is "variable_declarator") parent-bol ,js-indent-level)
+     ((parent-is "arguments") parent-bol ,js-indent-level)
+     ((parent-is "array") parent-bol ,js-indent-level)
+     ((parent-is "formal_parameters") parent-bol ,js-indent-level)
+     ((parent-is "template_substitution") parent-bol ,js-indent-level)
+     ((parent-is "object_pattern") parent-bol ,js-indent-level)
+     ((parent-is "object") parent-bol ,js-indent-level)
+     ((parent-is "arrow_function") parent-bol ,js-indent-level)
+     ((parent-is "parenthesized_expression") parent-bol ,js-indent-level)
+
+     ;; JSX
+     ((parent-is "jsx_opening_element") parent ,js-indent-level)
+     ((node-is "jsx_closing_element") parent 0)
+     ((node-is "jsx_text") parent ,js-indent-level)
+     ((parent-is "jsx_element") parent ,js-indent-level)
+     ((node-is "/") parent 0)
+     ((parent-is "jsx_self_closing_element") parent ,js-indent-level))))
+
+(defvar js--treesit-keywords
+  '("as" "async" "await" "break" "case" "catch" "class" "const" "continue"
+    "debugger" "default" "delete" "do" "else" "export" "extends" "finally"
+    "for" "from" "function" "get" "if" "import" "in" "instanceof" "let" "new"
+    "of" "return" "set" "static" "switch" "switch" "target" "throw" "try"
+    "typeof" "var" "void" "while" "with" "yield"))
+
+(defvar js--treesit-settings
+  (treesit-font-lock-rules
+   :language 'javascript
+   :override t
+   `(
+     ((identifier) @font-lock-constant-face
+      (:match "^[A-Z_][A-Z_\\d]*$" @font-lock-constant-face))
+
+     (new_expression
+      constructor: (identifier) @font-lock-type-face)
+
+     (function
+      name: (identifier) @font-lock-function-name-face)
+
+     (class_declaration
+      name: (identifier) @font-lock-type-face)
+
+     (function_declaration
+      name: (identifier) @font-lock-function-name-face)
+
+     (method_definition
+      name: (property_identifier) @font-lock-function-name-face)
+
+     (variable_declarator
+      name: (identifier) @font-lock-function-name-face
+      value: [(function) (arrow_function)])
+
+     (variable_declarator
+      name: (array_pattern (identifier) (identifier) @font-lock-function-name-face)
+      value: (array (number) (function)))
+
+     (assignment_expression
+      left: [(identifier) @font-lock-function-name-face
+             (member_expression property: (property_identifier) @font-lock-function-name-face)]
+      right: [(function) (arrow_function)])
+
+     (call_expression
+      function: [(identifier) @font-lock-function-name-face
+                 (member_expression
+                  property: (property_identifier) @font-lock-function-name-face)])
+
+     (variable_declarator
+      name: (identifier) @font-lock-variable-name-face)
+
+     (assignment_expression
+      left: [(identifier) @font-lock-variable-name-face
+             (member_expression property: (property_identifier) @font-lock-variable-name-face)])
+
+     (for_in_statement
+      left: (identifier) @font-lock-variable-name-face)
+
+     (arrow_function
+      parameter: (identifier) @font-lock-variable-name-face)
+
+     (pair key: (property_identifier) @font-lock-variable-name-face)
+
+     (pair value: (identifier) @font-lock-variable-name-face)
+
+     (pair
+      key: (property_identifier) @font-lock-function-name-face
+      value: [(function) (arrow_function)])
+
+     ((shorthand_property_identifier) @font-lock-variable-name-face)
+
+     (pair_pattern key: (property_identifier) @font-lock-variable-name-face)
+
+     ((shorthand_property_identifier_pattern) @font-lock-variable-name-face)
+
+     (array_pattern (identifier) @font-lock-variable-name-face)
+
+     (jsx_opening_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_closing_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_self_closing_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_attribute
+      (property_identifier)
+      @font-lock-constant-face)
+
+     [(this) (super)] @font-lock-keyword-face
+
+     [(true) (false) (null)] @font-lock-constant-face
+     (regex pattern: (regex_pattern)) @font-lock-string-face
+     (number) @font-lock-constant-face
+
+     (string) @font-lock-string-face
+     (comment) @font-lock-comment-face
+     [,@js--treesit-keywords] @font-lock-keyword-face
+
+     ;; FIXME: We need to be able to set the priority for font-locking
+     ;; somehow.  We cannot just override all of the template string,
+     ;; as that would mess up interpolated expressions
+     ;;
+     ;; (template_string) @font-lock-string-face
+     (template_substitution ["${" "}"] @font-lock-constant-face)
+     )))
+
+
+(defun js-treesit-current-defun ()
+  "Return name of surrounding function.
+This function can be used as a value in `which-func-functions'"
+  (let ((node (treesit-node-at (point)))
+        (name-list ()))
+    (cl-loop while node
+             if (pcase (treesit-node-type node)
+                  ("function_declaration" t)
+                  ("method_definition" t)
+                  ("class_declaration" t)
+                  ("variable_declarator" t)
+                  (_ nil))
+             do (push (treesit-node-text
+                       (treesit-node-child-by-field-name node "name")
+                       t)
+                      name-list)
+             do (setq node (treesit-node-parent node))
+             finally return  (string-join name-list "."))))
+
+(defun js--treesit-move-to-node (fn)
+  (when-let ((found-node
+              (treesit-parent-until
+               (treesit-node-at (point))
+               (lambda (parent)
+                 (treesit-query-capture
+                  parent
+                  js-treesit--defun-query)))))
+    (goto-char (funcall fn found-node))))
+
+(defun js--treesit-beginning-of-defun (&optional _arg)
+  (js--treesit-move-to-node #'treesit-node-start))
+
+(defun js--treesit--end-of-defun (&optional _arg)
+  (js--tressit-move-to-node #'treesit-node-end))
+
+(defvar js-treesit--defun-query
+  "[(class_declaration)
+    (method_definition)
+    (function_declaration)
+    (variable_declarator)] @defun")
+
+(defun js--treesit-enable ()
+  (unless (and (treesit-can-enable-p)
+               (treesit-language-available-p 'javascript))
+    (error "Tree sitter isn't available"))
+
+  ;; Comments
+  (setq-local comment-start "// ")
+  (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
+  (setq-local comment-end "")
+
+  (setq-local treesit-simple-indent-rules js--treesit-indent-rules)
+  (setq-local indent-line-function #'treesit-indent)
+  (setq-local beginning-of-defun-function #'js--treesit-beginning-of-defun)
+  (setq-local end-of-defun-function #'js--treesit-end-of-defun)
+
+  (setq-local font-lock-defaults '(nil t))
+  (setq-local treesit-font-lock-settings js--treesit-settings)
+
+  (add-hook 'which-func-functions #'js-treesit-current-defun nil t)
+
+  (treesit-font-lock-enable))
+
 ;;; Main Function
 
 ;;;###autoload
@@ -3407,93 +3630,155 @@ js-mode
   "Major mode for editing JavaScript."
   :group 'js
   ;; Ensure all CC Mode "lang variables" are set to valid values.
-  (c-init-language-vars js-mode)
-  (setq-local indent-line-function #'js-indent-line)
-  (setq-local beginning-of-defun-function #'js-beginning-of-defun)
-  (setq-local end-of-defun-function #'js-end-of-defun)
-  (setq-local open-paren-in-column-0-is-defun-start nil)
-  (setq-local font-lock-defaults
-              (list js--font-lock-keywords nil nil nil nil
-                    '(font-lock-syntactic-face-function
-                      . js-font-lock-syntactic-face-function)))
-  (setq-local syntax-propertize-function #'js-syntax-propertize)
-  (add-hook 'syntax-propertize-extend-region-functions
-            #'syntax-propertize-multiline 'append 'local)
-  (add-hook 'syntax-propertize-extend-region-functions
-            #'js--syntax-propertize-extend-region 'append 'local)
-  (setq-local prettify-symbols-alist js--prettify-symbols-alist)
-
-  (setq-local parse-sexp-ignore-comments t)
-  (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
+  (if js-use-treesitter
+      (js--treesit-enable)
+    (c-init-language-vars js-mode)
+    (setq-local indent-line-function #'js-indent-line)
+    (setq-local beginning-of-defun-function #'js-beginning-of-defun)
+    (setq-local end-of-defun-function #'js-end-of-defun)
+    (setq-local open-paren-in-column-0-is-defun-start nil)
+    (setq-local font-lock-defaults
+                (list js--font-lock-keywords nil nil nil nil
+                      '(font-lock-syntactic-face-function
+                        . js-font-lock-syntactic-face-function)))
+    (setq-local syntax-propertize-function #'js-syntax-propertize)
+    (add-hook 'syntax-propertize-extend-region-functions
+              #'syntax-propertize-multiline 'append 'local)
+    (add-hook 'syntax-propertize-extend-region-functions
+              #'js--syntax-propertize-extend-region 'append 'local)
+    (setq-local prettify-symbols-alist js--prettify-symbols-alist)
+
+    (setq-local parse-sexp-ignore-comments t)
+    (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
+
+    ;; Comments
+    (setq-local comment-start "// ")
+    (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
+    (setq-local comment-end "")
+    (setq-local fill-paragraph-function #'js-fill-paragraph)
+    (setq-local normal-auto-fill-function #'js-do-auto-fill)
+
+    ;; Parse cache
+    (add-hook 'before-change-functions #'js--flush-caches t t)
+
+    ;; Frameworks
+    (js--update-quick-match-re)
+
+    ;; Syntax extensions
+    (unless (js-jsx--detect-and-enable)
+      (add-hook 'after-change-functions #'js-jsx--detect-after-change nil t))
+    (js-use-syntactic-mode-name)
+
+    ;; Imenu
+    (setq imenu-case-fold-search nil)
+    (setq imenu-create-index-function #'js--imenu-create-index)
+
+    ;; for filling, pretend we're cc-mode
+    (c-foreign-init-lit-pos-cache)
+    (add-hook 'before-change-functions #'c-foreign-truncate-lit-pos-cache nil t)
+    (setq-local comment-line-break-function #'c-indent-new-comment-line)
+    (setq-local comment-multi-line t)
+    (setq-local electric-indent-chars
+	        (append "{}():;," electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
+    (setq-local electric-layout-rules
+	        '((?\; . after) (?\{ . after) (?\} . before)))
+
+    (let ((c-buffer-is-cc-mode t))
+      ;; FIXME: These are normally set by `c-basic-common-init'.  Should
+      ;; we call it instead?  (Bug#6071)
+      (make-local-variable 'paragraph-start)
+      (make-local-variable 'paragraph-separate)
+      (make-local-variable 'paragraph-ignore-fill-prefix)
+      (make-local-variable 'adaptive-fill-mode)
+      (make-local-variable 'adaptive-fill-regexp)
+      ;; While the full CC Mode style system is not yet in use, set the
+      ;; pertinent style variables manually.
+      (c-initialize-builtin-style)
+      (let ((style (cc-choose-style-for-mode 'js-mode c-default-style)))
+        (c-set-style style))
+      (setq c-block-comment-prefix "* "
+            c-comment-prefix-regexp "//+\\|\\**")
+      (c-setup-paragraph-variables))
+
+    ;; Important to fontify the whole buffer syntactically! If we don't,
+    ;; then we might have regular expression literals that aren't marked
+    ;; as strings, which will screw up parse-partial-sexp, scan-lists,
+    ;; etc. and produce maddening "unbalanced parenthesis" errors.
+    ;; When we attempt to find the error and scroll to the portion of
+    ;; the buffer containing the problem, JIT-lock will apply the
+    ;; correct syntax to the regular expression literal and the problem
+    ;; will mysteriously disappear.
+    ;; FIXME: We should instead do this fontification lazily by adding
+    ;; calls to syntax-propertize wherever it's really needed.
+    ;;(syntax-propertize (point-max))
+    ))
+
+(defcustom js-json-use-treesitter nil
+  "If non-nil, `js-json-mode' tries to use tree-sitter.
+Currently `js-json-mode' uses tree-sitter for font-locking and
+indentation."
+  :version "29.1"
+  :type 'boolean
+  :safe 'booleanp)
+
+(defvar js--json-treesit-settings
+  (treesit-font-lock-rules
+   :language 'json
+   :override t
+   `(
+     (pair
+      key: (_) @font-lock-string-face)
+
+     (string) @font-lock-string-face
+
+     (number) @font-lock-constant-face
+
+     [(null) (true) (false)] @font-lock-constant-face
+
+     (escape_sequence) @font-lock-constant-face
+
+     (comment) @font-lock-comment-face
+     )))
+
+
+(defvar js--json-treesit-indent-rules
+  `((json
+     (no-node (js--treesit-backward-up-list) ,js-indent-level)
+     ((node-is "}") parent-bol 0)
+     ((node-is ")") parent-bol 0)
+     ((node-is "]") parent-bol 0)
+     ((parent-is "object") parent-bol ,js-indent-level)
+     )))
+
+(defun js--json-treesit-enable ()
+  (unless (and (treesit-can-enable-p)
+               (treesit-language-available-p 'json))
+    (error "Tree sitter isn't available"))
 
   ;; Comments
   (setq-local comment-start "// ")
   (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
   (setq-local comment-end "")
-  (setq-local fill-paragraph-function #'js-fill-paragraph)
-  (setq-local normal-auto-fill-function #'js-do-auto-fill)
-
-  ;; Parse cache
-  (add-hook 'before-change-functions #'js--flush-caches t t)
-
-  ;; Frameworks
-  (js--update-quick-match-re)
-
-  ;; Syntax extensions
-  (unless (js-jsx--detect-and-enable)
-    (add-hook 'after-change-functions #'js-jsx--detect-after-change nil t))
-  (js-use-syntactic-mode-name)
-
-  ;; Imenu
-  (setq imenu-case-fold-search nil)
-  (setq imenu-create-index-function #'js--imenu-create-index)
-
-  ;; for filling, pretend we're cc-mode
-  (c-foreign-init-lit-pos-cache)
-  (add-hook 'before-change-functions #'c-foreign-truncate-lit-pos-cache nil t)
-  (setq-local comment-line-break-function #'c-indent-new-comment-line)
-  (setq-local comment-multi-line t)
-  (setq-local electric-indent-chars
-	      (append "{}():;," electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
-  (setq-local electric-layout-rules
-	      '((?\; . after) (?\{ . after) (?\} . before)))
-
-  (let ((c-buffer-is-cc-mode t))
-    ;; FIXME: These are normally set by `c-basic-common-init'.  Should
-    ;; we call it instead?  (Bug#6071)
-    (make-local-variable 'paragraph-start)
-    (make-local-variable 'paragraph-separate)
-    (make-local-variable 'paragraph-ignore-fill-prefix)
-    (make-local-variable 'adaptive-fill-mode)
-    (make-local-variable 'adaptive-fill-regexp)
-    ;; While the full CC Mode style system is not yet in use, set the
-    ;; pertinent style variables manually.
-    (c-initialize-builtin-style)
-    (let ((style (cc-choose-style-for-mode 'js-mode c-default-style)))
-      (c-set-style style))
-    (setq c-block-comment-prefix "* "
-          c-comment-prefix-regexp "//+\\|\\**")
-    (c-setup-paragraph-variables))
-
-  ;; Important to fontify the whole buffer syntactically! If we don't,
-  ;; then we might have regular expression literals that aren't marked
-  ;; as strings, which will screw up parse-partial-sexp, scan-lists,
-  ;; etc. and produce maddening "unbalanced parenthesis" errors.
-  ;; When we attempt to find the error and scroll to the portion of
-  ;; the buffer containing the problem, JIT-lock will apply the
-  ;; correct syntax to the regular expression literal and the problem
-  ;; will mysteriously disappear.
-  ;; FIXME: We should instead do this fontification lazily by adding
-  ;; calls to syntax-propertize wherever it's really needed.
-  ;;(syntax-propertize (point-max))
-  )
+
+  (setq-local treesit-simple-indent-rules js--json-treesit-indent-rules)
+  (setq-local indent-line-function #'treesit-indent)
+  (setq-local beginning-of-defun-function #'ignore)
+  (setq-local end-of-defun-function #'ignore)
+
+  (setq-local font-lock-defaults '(nil t))
+  (setq-local treesit-font-lock-settings js--json-treesit-settings)
+
+  (treesit-font-lock-enable))
+
 
 ;;;###autoload
 (define-derived-mode js-json-mode js-mode "JSON"
-  (setq-local js-enabled-frameworks nil)
-  ;; Speed up `syntax-ppss': JSON files can be big but can't hold
-  ;; regexp matchers nor #! thingies (and `js-enabled-frameworks' is nil).
-  (setq-local syntax-propertize-function #'ignore))
+  (if js-json-use-treesitter
+      (js--json-treesit-enable)
+    (setq-local js-enabled-frameworks nil)
+    ;; Speed up `syntax-ppss': JSON files can be big but can't hold
+    ;; regexp matchers nor #! thingies (and `js-enabled-frameworks' is nil).
+    (setq-local syntax-propertize-function #'ignore)))
 
 ;; Since we made JSX support available and automatically-enabled in
 ;; the base `js-mode' (for ease of use), now `js-jsx-mode' simply
@@ -3520,9 +3805,11 @@ js-jsx-mode
 `js-jsx-enable' in `js-mode-hook'.  You may be better served by
 one of the aforementioned options instead of using this mode."
   :group 'js
-  (js-jsx-enable)
-  (setq-local comment-region-function #'js-jsx--comment-region)
-  (js-use-syntactic-mode-name))
+  (if js-use-treesitter
+      (js--treesit-enable)
+    (js-jsx-enable)
+    (setq-local comment-region-function #'js-jsx--comment-region)
+    (js-use-syntactic-mode-name)))
 
 (defun js-jsx--comment-region (beg end &optional arg)
   (if (or (js-jsx--context)
-- 
2.34.1

Re: Call for volunteers: add tree-sitter support to major modes

[Emanuel Berg]

Theodor Thornhill wrote:

> A small followup patch - this should be the last to consider
> for now - sorry the many patches :)

On the contrary ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Emanuel Berg <incal@dataswamp.org> writes:

> Theodor Thornhill wrote:
>
>> A small followup patch - this should be the last to consider
>> for now - sorry the many patches :)
>
> On the contrary ...
>

I don't follow - what do you mean?

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 9, 2022, at 2:21 PM, Theodor Thornhill <theo@thornhill.no> wrote:
> 
> Theodor Thornhill <theo@thornhill.no> writes:
> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>>>> Date: Sun, 09 Oct 2022 17:18:52 +0200
>>>> From: Theodor Thornhill <theo@thornhill.no>
>>>> CC: emacs-devel@gnu.org
>>>> 
>>>> BTW, I can do json as well, should be quick enough :) I'll add it in a different patch
>>> 
>>> TIA
>> 
>> So I added tree sitter support for js-json-mode, js-jsx-mode as well.
>> The changes were simple enough not to warrant a separate patch, IMO, so
>> I just updated the old one.
>> 
>> What do you think, Yuan and Eli?
> 
> A small followup patch - this should be the last to consider for now -
> sorry the many patches :)
> 
> Theodor
> <0001-Add-tree-sitter-functionality-to-js-mode.patch>

Looks good! Here are some comments.

+
+     ;; FIXME: We need to be able to set the priority for font-locking
+     ;; somehow.  We cannot just override all of the template string,
+     ;; as that would mess up interpolated expressions
+     ;;
+     ;; (template_string) @font-lock-string-face
+     (template_substitution ["${" "}"] @font-lock-constant-face)
+     )))

What exactly do you mean by priority here? Why doesn't :override t
work?

+
+(defvar js-treesit--defun-query
+  "[(class_declaration)
+    (method_definition)
+    (function_declaration)
+    (variable_declarator)] @defun")

This should be compiled.

+
+(defun js--treesit-enable ()
+  (unless (and (treesit-can-enable-p)
+               (treesit-language-available-p 'javascript))
+    (error "Tree sitter isn't available"))

I don't think we should error here, I'd displaying a message instead.

+
+  ;; Comments
+  (setq-local comment-start "// ")
+  (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
+  (setq-local comment-end "")

I think it's best to not repeat code, could you move this outside the
(if tree-sitter) form and have it run regardless?

+(defun js--json-treesit-enable ()
+  (unless (and (treesit-can-enable-p)
+               (treesit-language-available-p 'json))
+    (error "Tree sitter isn't available"))

Same as above, IMO message is better.

Thanks,
Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> 
> +
> +(defun js--treesit-enable ()
> +  (unless (and (treesit-can-enable-p)
> +               (treesit-language-available-p 'javascript))
> +    (error "Tree sitter isn't available"))
> 
> I don't think we should error here, I'd displaying a message instead.

Actually we should also fall back to non-tree-sitter if the conditions aren’t met.

Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 9 Oct 2022 21:05:09 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  emacs-devel@gnu.org
> 
> > +
> > +(defun js--treesit-enable ()
> > +  (unless (and (treesit-can-enable-p)
> > +               (treesit-language-available-p 'javascript))
> > +    (error "Tree sitter isn't available"))
> > 
> > I don't think we should error here, I'd displaying a message instead.
> 
> Actually we should also fall back to non-tree-sitter if the conditions aren’t met.

Automatically and silently?  I don't think so.  The user should know
the knob they pressed didn't work.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sun, 9 Oct 2022 21:05:09 -0700
>> Cc: Eli Zaretskii <eliz@gnu.org>,
>>  emacs-devel@gnu.org
>> 
>> > +
>> > +(defun js--treesit-enable ()
>> > +  (unless (and (treesit-can-enable-p)
>> > +               (treesit-language-available-p 'javascript))
>> > +    (error "Tree sitter isn't available"))
>> > 
>> > I don't think we should error here, I'd displaying a message instead.
>> 
>> Actually we should also fall back to non-tree-sitter if the conditions aren’t met.
>
> Automatically and silently?  I don't think so.  The user should know
> the knob they pressed didn't work.

How about we error and add a suggestion to turn off `*-use-treesitter'?

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: emacs-devel@gnu.org
> Date: Mon, 10 Oct 2022 08:35:21 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Actually we should also fall back to non-tree-sitter if the conditions aren’t met.
> >
> > Automatically and silently?  I don't think so.  The user should know
> > the knob they pressed didn't work.
> 
> How about we error and add a suggestion to turn off `*-use-treesitter'?

Yes, something like that.

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 9, 2022, at 11:28 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sun, 9 Oct 2022 21:05:09 -0700
>> Cc: Eli Zaretskii <eliz@gnu.org>,
>> emacs-devel@gnu.org
>> 
>>> +
>>> +(defun js--treesit-enable ()
>>> +  (unless (and (treesit-can-enable-p)
>>> +               (treesit-language-available-p 'javascript))
>>> +    (error "Tree sitter isn't available"))
>>> 
>>> I don't think we should error here, I'd displaying a message instead.
>> 
>> Actually we should also fall back to non-tree-sitter if the conditions aren’t met.
> 
> Automatically and silently?  I don't think so.  The user should know
> the knob they pressed didn't work.

Not silently, print a message, raise a user-error at the end, but don’t fail completely and leave the user in a broken mode. I suggest:

(define-derived-mode mode
  (if (and use-tree-sitter tree-sitter-works)
      init-tree-sitter
    init-non-tree-sitter)

  (if (and use-tree-sitter (not tree-sitter-works))
      ;; Or ‘message’.
      (user-error "Tree-sitter can’t be enabled")))

If we think this is good, I’ll add it to the starter guide.

Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 9 Oct 2022 23:46:27 -0700
> Cc: Theodor Thornhill <theo@thornhill.no>,
>  emacs-devel@gnu.org
> 
> > Automatically and silently?  I don't think so.  The user should know
> > the knob they pressed didn't work.
> 
> Not silently, print a message, raise a user-error at the end, but don’t fail completely and leave the user in a broken mode. I suggest:
> 
> (define-derived-mode mode
>   (if (and use-tree-sitter tree-sitter-works)
>       init-tree-sitter
>     init-non-tree-sitter)
> 
>   (if (and use-tree-sitter (not tree-sitter-works))
>       ;; Or ‘message’.
>       (user-error "Tree-sitter can’t be enabled")))

Yes, this should leave the user with the original mode without
tree-sitter.

> If we think this is good, I’ll add it to the starter guide.

Please do, and thanks.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

[-- Attachment #1: Type: text/plain, Size: 1939 bytes --]

Hi!

> Looks good! Here are some comments.
>
> +
> +     ;; FIXME: We need to be able to set the priority for font-locking
> +     ;; somehow.  We cannot just override all of the template string,
> +     ;; as that would mess up interpolated expressions
> +     ;;
> +     ;; (template_string) @font-lock-string-face
> +     (template_substitution ["${" "}"] @font-lock-constant-face)
> +     )))
>
> What exactly do you mean by priority here? Why doesn't :override t
> work?
>

In template strings in JavaScript we have issues with precedence in that
we default to string font locking for everything that does _not_ have
specific rules.  An image speaks a thousand words, so I'll add
screenshots.  What I need is some sort of way to ensure that inside of
template strings font-locking should _not_ happen

> +
> +(defvar js-treesit--defun-query
> +  "[(class_declaration)
> +    (method_definition)
> +    (function_declaration)
> +    (variable_declarator)] @defun")
>
> This should be compiled.
>
> +
> +(defun js--treesit-enable ()
> +  (unless (and (treesit-can-enable-p)
> +               (treesit-language-available-p 'javascript))
> +    (error "Tree sitter isn't available"))
>
> I don't think we should error here, I'd displaying a message instead.
>
> +
> +  ;; Comments
> +  (setq-local comment-start "// ")
> +  (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
> +  (setq-local comment-end "")
>
> I think it's best to not repeat code, could you move this outside the
> (if tree-sitter) form and have it run regardless?
>
> +(defun js--json-treesit-enable ()
> +  (unless (and (treesit-can-enable-p)
> +               (treesit-language-available-p 'json))
> +    (error "Tree sitter isn't available"))
>
> Same as above, IMO message is better.
>

I added some variartion of this.  I think also message is better,
because user-error makes the logic a little harder.  What do you think?


Thanks, see attached patch:


[-- Attachment #2: font-lock-on-string.png --]
[-- Type: image/png, Size: 13881 bytes --]

[-- Attachment #3: no-font-lock.png --]
[-- Type: image/png, Size: 13313 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0001-Add-tree-sitter-functionality-to-js-mode.patch --]
[-- Type: text/x-diff, Size: 20617 bytes --]

From 42628d295ffa2a83a113a0d77a70d360c119060d Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Sun, 9 Oct 2022 21:22:55 +0200
Subject: [PATCH] Add tree-sitter functionality to js-mode

* lisp/progmodes/js.el (treesit): Add tree-sitter dependency

(js-use-treesitter): Defcustom to enable tree-sitter as the engine for
font-lock, indentation and more.

(js--treesit-backward-up-list): Helper to enable indenting of empty bodies.

(js--treesit-indent-rules): Rules to be used when indenting JavaScript code.

(js--treesit-keywords): Keywords as seen with tree-sitter.

(js--treesit-settings): Font-Lock settings for JavaScript.

(js-treesit-current-defun): Helper for which-function-mode.

(js--treesit-move-to-node): Helper for movement to specific nodes in
the tree-sitter tree.

(js--treesit-beginning-of-defun, js--treesit-end-of-defun): Movement
functions for js-mode.

(js--treesit-enable): Helper function to enable tree-sitter.  Skips
over most initalization done by the CC-Mode variant.

(js-json-use-treesitter): Defcustom to enable tree-sitter as the
engine for font-lock and indentation for JSON files.

(js--json-treesit-settings): Font-Lock settings for JSON.

(js--json-treesit-indent-rules): Rules to be used when indenting JSON code

(js--json-treesit-enable): Helper function to enable tree-sitter.
Skips over most initialization done by the CC-Mode variant, in
addition to ignoring some JavaScript specific settings.

(js-mode, js-json-mode, js-jsx-mode): Add support for tree-sitter
functionalities.
---
 lisp/progmodes/js.el | 460 +++++++++++++++++++++++++++++++++++--------
 1 file changed, 376 insertions(+), 84 deletions(-)

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index b920ef6c2c..d1f655cdc4 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -53,6 +53,7 @@
 (require 'imenu)
 (require 'json)
 (require 'prog-mode)
+(require 'treesit)
 
 (eval-when-compile
   (require 'cl-lib)
@@ -3400,100 +3401,389 @@ js-jsx--detect-after-change
 (c-lang-defconst c-paragraph-start
   js-mode "\\(@[[:alpha:]]+\\>\\|$\\)")
 
+;;; Tree sitter integration
+(defcustom js-use-treesitter nil
+  "If non-nil, `js-mode' tries to use tree-sitter.
+Currently `js-mode' uses tree-sitter for font-locking,
+indentation, which-function and movement functions."
+  :version "29.1"
+  :type 'boolean
+  :safe 'booleanp)
+
+(defun js--treesit-backward-up-list ()
+  (lambda (_node _parent _bol &rest _)
+    (save-excursion
+      (backward-up-list 1 nil t)
+      (goto-char
+       (treesit-node-start
+        (treesit-node-at (point))))
+      (back-to-indentation)
+      (point))))
+
+(defvar js--treesit-indent-rules
+  `((javascript
+     (no-node (js--treesit-backward-up-list) ,js-indent-level)
+     ((node-is "}") parent-bol 0)
+     ((node-is ")") parent-bol 0)
+     ((node-is "]") parent-bol 0)
+     ((node-is ">") parent-bol 0)
+     ((node-is ".") parent-bol ,js-indent-level)
+     ((parent-is "named_imports") parent-bol ,js-indent-level)
+     ((parent-is "statement_block") parent-bol ,js-indent-level)
+     ((parent-is "variable_declarator") parent-bol ,js-indent-level)
+     ((parent-is "arguments") parent-bol ,js-indent-level)
+     ((parent-is "array") parent-bol ,js-indent-level)
+     ((parent-is "formal_parameters") parent-bol ,js-indent-level)
+     ((parent-is "template_substitution") parent-bol ,js-indent-level)
+     ((parent-is "object_pattern") parent-bol ,js-indent-level)
+     ((parent-is "object") parent-bol ,js-indent-level)
+     ((parent-is "arrow_function") parent-bol ,js-indent-level)
+     ((parent-is "parenthesized_expression") parent-bol ,js-indent-level)
+
+     ;; JSX
+     ((parent-is "jsx_opening_element") parent ,js-indent-level)
+     ((node-is "jsx_closing_element") parent 0)
+     ((node-is "jsx_text") parent ,js-indent-level)
+     ((parent-is "jsx_element") parent ,js-indent-level)
+     ((node-is "/") parent 0)
+     ((parent-is "jsx_self_closing_element") parent ,js-indent-level))))
+
+(defvar js--treesit-keywords
+  '("as" "async" "await" "break" "case" "catch" "class" "const" "continue"
+    "debugger" "default" "delete" "do" "else" "export" "extends" "finally"
+    "for" "from" "function" "get" "if" "import" "in" "instanceof" "let" "new"
+    "of" "return" "set" "static" "switch" "switch" "target" "throw" "try"
+    "typeof" "var" "void" "while" "with" "yield"))
+
+(defvar js--treesit-settings
+  (treesit-font-lock-rules
+   :language 'javascript
+   :override t
+   `(
+     ((identifier) @font-lock-constant-face
+      (:match "^[A-Z_][A-Z_\\d]*$" @font-lock-constant-face))
+
+     (new_expression
+      constructor: (identifier) @font-lock-type-face)
+
+     (function
+      name: (identifier) @font-lock-function-name-face)
+
+     (class_declaration
+      name: (identifier) @font-lock-type-face)
+
+     (function_declaration
+      name: (identifier) @font-lock-function-name-face)
+
+     (method_definition
+      name: (property_identifier) @font-lock-function-name-face)
+
+     (variable_declarator
+      name: (identifier) @font-lock-function-name-face
+      value: [(function) (arrow_function)])
+
+     (variable_declarator
+      name: (array_pattern (identifier) (identifier) @font-lock-function-name-face)
+      value: (array (number) (function)))
+
+     (assignment_expression
+      left: [(identifier) @font-lock-function-name-face
+             (member_expression property: (property_identifier) @font-lock-function-name-face)]
+      right: [(function) (arrow_function)])
+
+     (call_expression
+      function: [(identifier) @font-lock-function-name-face
+                 (member_expression
+                  property: (property_identifier) @font-lock-function-name-face)])
+
+     (variable_declarator
+      name: (identifier) @font-lock-variable-name-face)
+
+     (assignment_expression
+      left: [(identifier) @font-lock-variable-name-face
+             (member_expression property: (property_identifier) @font-lock-variable-name-face)])
+
+     (for_in_statement
+      left: (identifier) @font-lock-variable-name-face)
+
+     (arrow_function
+      parameter: (identifier) @font-lock-variable-name-face)
+
+     (pair key: (property_identifier) @font-lock-variable-name-face)
+
+     (pair value: (identifier) @font-lock-variable-name-face)
+
+     (pair
+      key: (property_identifier) @font-lock-function-name-face
+      value: [(function) (arrow_function)])
+
+     ((shorthand_property_identifier) @font-lock-variable-name-face)
+
+     (pair_pattern key: (property_identifier) @font-lock-variable-name-face)
+
+     ((shorthand_property_identifier_pattern) @font-lock-variable-name-face)
+
+     (array_pattern (identifier) @font-lock-variable-name-face)
+
+     (jsx_opening_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_closing_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_self_closing_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_attribute
+      (property_identifier)
+      @font-lock-constant-face)
+
+     [(this) (super)] @font-lock-keyword-face
+
+     [(true) (false) (null)] @font-lock-constant-face
+     (regex pattern: (regex_pattern)) @font-lock-string-face
+     (number) @font-lock-constant-face
+
+     (string) @font-lock-string-face
+     (comment) @font-lock-comment-face
+     [,@js--treesit-keywords] @font-lock-keyword-face
+
+     ;; FIXME: We need to be able to set the priority for font-locking
+     ;; somehow.  We cannot just override all of the template string,
+     ;; as that would mess up interpolated expressions
+     ;;
+     ;; (template_string) @font-lock-string-face
+     (template_substitution ["${" "}"] @font-lock-constant-face)
+     )))
+
+
+(defun js-treesit-current-defun ()
+  "Return name of surrounding function.
+This function can be used as a value in `which-func-functions'"
+  (let ((node (treesit-node-at (point)))
+        (name-list ()))
+    (cl-loop while node
+             if (pcase (treesit-node-type node)
+                  ("function_declaration" t)
+                  ("method_definition" t)
+                  ("class_declaration" t)
+                  ("variable_declarator" t)
+                  (_ nil))
+             do (push (treesit-node-text
+                       (treesit-node-child-by-field-name node "name")
+                       t)
+                      name-list)
+             do (setq node (treesit-node-parent node))
+             finally return  (string-join name-list "."))))
+
+(defun js--treesit-move-to-node (fn)
+  (when-let ((found-node
+              (treesit-parent-until
+               (treesit-node-at (point))
+               (lambda (parent)
+                 (treesit-query-capture
+                  parent
+                  js-treesit--defun-query)))))
+    (goto-char (funcall fn found-node))))
+
+(defun js--treesit-beginning-of-defun (&optional _arg)
+  (js--treesit-move-to-node #'treesit-node-start))
+
+(defun js--treesit--end-of-defun (&optional _arg)
+  (js--tressit-move-to-node #'treesit-node-end))
+
+(defvar js-treesit--defun-query
+  (treesit-query-compile
+   'javascript
+   "[(class_declaration)
+    (method_definition)
+    (function_declaration)
+    (variable_declarator)] @defun"))
+
+(defun js--treesit-can-enable-p ()
+  (if (and js-use-treesitter
+           (treesit-can-enable-p)
+           (treesit-language-available-p 'javascript))
+      t
+    (message "Cannot enable Tree Sitter for JavaScript.")
+    nil))
+
+(defun js--treesit-enable ()
+  (setq-local treesit-simple-indent-rules js--treesit-indent-rules)
+  (setq-local indent-line-function #'treesit-indent)
+
+  (setq-local beginning-of-defun-function #'js--treesit-beginning-of-defun)
+  (setq-local end-of-defun-function #'js--treesit-end-of-defun)
+
+  (setq-local font-lock-defaults '(nil t))
+  (setq-local treesit-font-lock-settings js--treesit-settings)
+
+  (add-hook 'which-func-functions #'js-treesit-current-defun nil t)
+
+  (treesit-font-lock-enable))
+
 ;;; Main Function
 
 ;;;###autoload
 (define-derived-mode js-mode prog-mode "JavaScript"
   "Major mode for editing JavaScript."
   :group 'js
-  ;; Ensure all CC Mode "lang variables" are set to valid values.
-  (c-init-language-vars js-mode)
-  (setq-local indent-line-function #'js-indent-line)
-  (setq-local beginning-of-defun-function #'js-beginning-of-defun)
-  (setq-local end-of-defun-function #'js-end-of-defun)
-  (setq-local open-paren-in-column-0-is-defun-start nil)
-  (setq-local font-lock-defaults
-              (list js--font-lock-keywords nil nil nil nil
-                    '(font-lock-syntactic-face-function
-                      . js-font-lock-syntactic-face-function)))
-  (setq-local syntax-propertize-function #'js-syntax-propertize)
-  (add-hook 'syntax-propertize-extend-region-functions
-            #'syntax-propertize-multiline 'append 'local)
-  (add-hook 'syntax-propertize-extend-region-functions
-            #'js--syntax-propertize-extend-region 'append 'local)
-  (setq-local prettify-symbols-alist js--prettify-symbols-alist)
-
-  (setq-local parse-sexp-ignore-comments t)
-  (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
-
   ;; Comments
   (setq-local comment-start "// ")
   (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
   (setq-local comment-end "")
-  (setq-local fill-paragraph-function #'js-fill-paragraph)
-  (setq-local normal-auto-fill-function #'js-do-auto-fill)
-
-  ;; Parse cache
-  (add-hook 'before-change-functions #'js--flush-caches t t)
-
-  ;; Frameworks
-  (js--update-quick-match-re)
-
-  ;; Syntax extensions
-  (unless (js-jsx--detect-and-enable)
-    (add-hook 'after-change-functions #'js-jsx--detect-after-change nil t))
-  (js-use-syntactic-mode-name)
-
-  ;; Imenu
-  (setq imenu-case-fold-search nil)
-  (setq imenu-create-index-function #'js--imenu-create-index)
-
-  ;; for filling, pretend we're cc-mode
-  (c-foreign-init-lit-pos-cache)
-  (add-hook 'before-change-functions #'c-foreign-truncate-lit-pos-cache nil t)
-  (setq-local comment-line-break-function #'c-indent-new-comment-line)
-  (setq-local comment-multi-line t)
-  (setq-local electric-indent-chars
-	      (append "{}():;," electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
-  (setq-local electric-layout-rules
-	      '((?\; . after) (?\{ . after) (?\} . before)))
-
-  (let ((c-buffer-is-cc-mode t))
-    ;; FIXME: These are normally set by `c-basic-common-init'.  Should
-    ;; we call it instead?  (Bug#6071)
-    (make-local-variable 'paragraph-start)
-    (make-local-variable 'paragraph-separate)
-    (make-local-variable 'paragraph-ignore-fill-prefix)
-    (make-local-variable 'adaptive-fill-mode)
-    (make-local-variable 'adaptive-fill-regexp)
-    ;; While the full CC Mode style system is not yet in use, set the
-    ;; pertinent style variables manually.
-    (c-initialize-builtin-style)
-    (let ((style (cc-choose-style-for-mode 'js-mode c-default-style)))
-      (c-set-style style))
-    (setq c-block-comment-prefix "* "
-          c-comment-prefix-regexp "//+\\|\\**")
-    (c-setup-paragraph-variables))
-
-  ;; Important to fontify the whole buffer syntactically! If we don't,
-  ;; then we might have regular expression literals that aren't marked
-  ;; as strings, which will screw up parse-partial-sexp, scan-lists,
-  ;; etc. and produce maddening "unbalanced parenthesis" errors.
-  ;; When we attempt to find the error and scroll to the portion of
-  ;; the buffer containing the problem, JIT-lock will apply the
-  ;; correct syntax to the regular expression literal and the problem
-  ;; will mysteriously disappear.
-  ;; FIXME: We should instead do this fontification lazily by adding
-  ;; calls to syntax-propertize wherever it's really needed.
-  ;;(syntax-propertize (point-max))
-  )
+
+  (if (js--treesit-can-enable-p)
+      (js--treesit-enable)
+    ;; Ensure all CC Mode "lang variables" are set to valid values.
+    (c-init-language-vars js-mode)
+    (setq-local indent-line-function #'js-indent-line)
+    (setq-local beginning-of-defun-function #'js-beginning-of-defun)
+    (setq-local end-of-defun-function #'js-end-of-defun)
+    (setq-local open-paren-in-column-0-is-defun-start nil)
+    (setq-local font-lock-defaults
+                (list js--font-lock-keywords nil nil nil nil
+                      '(font-lock-syntactic-face-function
+                        . js-font-lock-syntactic-face-function)))
+    (setq-local syntax-propertize-function #'js-syntax-propertize)
+    (add-hook 'syntax-propertize-extend-region-functions
+              #'syntax-propertize-multiline 'append 'local)
+    (add-hook 'syntax-propertize-extend-region-functions
+              #'js--syntax-propertize-extend-region 'append 'local)
+    (setq-local prettify-symbols-alist js--prettify-symbols-alist)
+
+    (setq-local parse-sexp-ignore-comments t)
+    (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
+
+    (setq-local fill-paragraph-function #'js-fill-paragraph)
+    (setq-local normal-auto-fill-function #'js-do-auto-fill)
+
+    ;; Parse cache
+    (add-hook 'before-change-functions #'js--flush-caches t t)
+
+    ;; Frameworks
+    (js--update-quick-match-re)
+
+    ;; Syntax extensions
+    (unless (js-jsx--detect-and-enable)
+      (add-hook 'after-change-functions #'js-jsx--detect-after-change nil t))
+    (js-use-syntactic-mode-name)
+
+    ;; Imenu
+    (setq imenu-case-fold-search nil)
+    (setq imenu-create-index-function #'js--imenu-create-index)
+
+    ;; for filling, pretend we're cc-mode
+    (c-foreign-init-lit-pos-cache)
+    (add-hook 'before-change-functions #'c-foreign-truncate-lit-pos-cache nil t)
+    (setq-local comment-line-break-function #'c-indent-new-comment-line)
+    (setq-local comment-multi-line t)
+    (setq-local electric-indent-chars
+	        (append "{}():;," electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
+    (setq-local electric-layout-rules
+	        '((?\; . after) (?\{ . after) (?\} . before)))
+
+    (let ((c-buffer-is-cc-mode t))
+      ;; FIXME: These are normally set by `c-basic-common-init'.  Should
+      ;; we call it instead?  (Bug#6071)
+      (make-local-variable 'paragraph-start)
+      (make-local-variable 'paragraph-separate)
+      (make-local-variable 'paragraph-ignore-fill-prefix)
+      (make-local-variable 'adaptive-fill-mode)
+      (make-local-variable 'adaptive-fill-regexp)
+      ;; While the full CC Mode style system is not yet in use, set the
+      ;; pertinent style variables manually.
+      (c-initialize-builtin-style)
+      (let ((style (cc-choose-style-for-mode 'js-mode c-default-style)))
+        (c-set-style style))
+      (setq c-block-comment-prefix "* "
+            c-comment-prefix-regexp "//+\\|\\**")
+      (c-setup-paragraph-variables))
+
+    ;; Important to fontify the whole buffer syntactically! If we don't,
+    ;; then we might have regular expression literals that aren't marked
+    ;; as strings, which will screw up parse-partial-sexp, scan-lists,
+    ;; etc. and produce maddening "unbalanced parenthesis" errors.
+    ;; When we attempt to find the error and scroll to the portion of
+    ;; the buffer containing the problem, JIT-lock will apply the
+    ;; correct syntax to the regular expression literal and the problem
+    ;; will mysteriously disappear.
+    ;; FIXME: We should instead do this fontification lazily by adding
+    ;; calls to syntax-propertize wherever it's really needed.
+    ;;(syntax-propertize (point-max))
+    ))
+
+(defcustom js-json-use-treesitter nil
+  "If non-nil, `js-json-mode' tries to use tree-sitter.
+Currently `js-json-mode' uses tree-sitter for font-locking and
+indentation."
+  :version "29.1"
+  :type 'boolean
+  :safe 'booleanp)
+
+(defvar js--json-treesit-settings
+  (treesit-font-lock-rules
+   :language 'json
+   :override t
+   `(
+     (pair
+      key: (_) @font-lock-string-face)
+
+     (string) @font-lock-string-face
+
+     (number) @font-lock-constant-face
+
+     [(null) (true) (false)] @font-lock-constant-face
+
+     (escape_sequence) @font-lock-constant-face
+
+     (comment) @font-lock-comment-face
+     )))
+
+
+(defvar js--json-treesit-indent-rules
+  `((json
+     (no-node (js--treesit-backward-up-list) ,js-indent-level)
+     ((node-is "}") parent-bol 0)
+     ((node-is ")") parent-bol 0)
+     ((node-is "]") parent-bol 0)
+     ((parent-is "object") parent-bol ,js-indent-level)
+     )))
+
+
+(defun js--json-treesit-can-enable-p ()
+  (if (and js-json-use-treesitter
+           (treesit-can-enable-p)
+           (treesit-language-available-p 'json))
+      t
+    (error "Cannot enable Tree Sitter for JSON.")
+    nil))
+
+
+(defun js--json-treesit-enable ()
+  (setq-local treesit-simple-indent-rules js--json-treesit-indent-rules)
+  (setq-local indent-line-function #'treesit-indent)
+
+  (setq-local beginning-of-defun-function #'ignore)
+  (setq-local end-of-defun-function #'ignore)
+
+  (setq-local font-lock-defaults '(nil t))
+  (setq-local treesit-font-lock-settings js--json-treesit-settings)
+
+  (treesit-font-lock-enable))
+
 
 ;;;###autoload
 (define-derived-mode js-json-mode js-mode "JSON"
-  (setq-local js-enabled-frameworks nil)
-  ;; Speed up `syntax-ppss': JSON files can be big but can't hold
-  ;; regexp matchers nor #! thingies (and `js-enabled-frameworks' is nil).
-  (setq-local syntax-propertize-function #'ignore))
+  (if (js--json-treesit-can-enable-p)
+      (js--json-treesit-enable)
+    (setq-local js-enabled-frameworks nil)
+    ;; Speed up `syntax-ppss': JSON files can be big but can't hold
+    ;; regexp matchers nor #! thingies (and `js-enabled-frameworks' is nil).
+    (setq-local syntax-propertize-function #'ignore)))
 
 ;; Since we made JSX support available and automatically-enabled in
 ;; the base `js-mode' (for ease of use), now `js-jsx-mode' simply
@@ -3520,9 +3810,11 @@ js-jsx-mode
 `js-jsx-enable' in `js-mode-hook'.  You may be better served by
 one of the aforementioned options instead of using this mode."
   :group 'js
-  (js-jsx-enable)
-  (setq-local comment-region-function #'js-jsx--comment-region)
-  (js-use-syntactic-mode-name))
+  (if (js--treesit-can-enable-p)
+      (js--treesit-enable)
+    (js-jsx-enable)
+    (setq-local comment-region-function #'js-jsx--comment-region)
+    (js-use-syntactic-mode-name)))
 
 (defun js-jsx--comment-region (beg end &optional arg)
   (if (or (js-jsx--context)
-- 
2.34.1

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

[-- Attachment #1: Type: text/plain, Size: 2736 bytes --]



> On Oct 10, 2022, at 12:08 AM, Theodor Thornhill <theo@thornhill.no> wrote:
> 
> Hi!
> 
>> Looks good! Here are some comments.
>> 
>> +
>> +     ;; FIXME: We need to be able to set the priority for font-locking
>> +     ;; somehow.  We cannot just override all of the template string,
>> +     ;; as that would mess up interpolated expressions
>> +     ;;
>> +     ;; (template_string) @font-lock-string-face
>> +     (template_substitution ["${" "}"] @font-lock-constant-face)
>> +     )))
>> 
>> What exactly do you mean by priority here? Why doesn't :override t
>> work?
>> 
> 
> In template strings in JavaScript we have issues with precedence in that
> we default to string font locking for everything that does _not_ have
> specific rules.  An image speaks a thousand words, so I'll add
> screenshots.  What I need is some sort of way to ensure that inside of
> template strings font-locking should _not_ happen
> 
>> +
>> +(defvar js-treesit--defun-query
>> +  "[(class_declaration)
>> +    (method_definition)
>> +    (function_declaration)
>> +    (variable_declarator)] @defun")
>> 
>> This should be compiled.
>> 
>> +
>> +(defun js--treesit-enable ()
>> +  (unless (and (treesit-can-enable-p)
>> +               (treesit-language-available-p 'javascript))
>> +    (error "Tree sitter isn't available"))
>> 
>> I don't think we should error here, I'd displaying a message instead.
>> 
>> +
>> +  ;; Comments
>> +  (setq-local comment-start "// ")
>> +  (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
>> +  (setq-local comment-end "")
>> 
>> I think it's best to not repeat code, could you move this outside the
>> (if tree-sitter) form and have it run regardless?
>> 
>> +(defun js--json-treesit-enable ()
>> +  (unless (and (treesit-can-enable-p)
>> +               (treesit-language-available-p 'json))
>> +    (error "Tree sitter isn't available"))
>> 
>> Same as above, IMO message is better.
>> 
> 
> I added some variartion of this.  I think also message is better,
> because user-error makes the logic a little harder.  What do you think?

Yeah user-error is mostly the same as a message, since nobody (I think) turns on beeping anymore. Though I hope there is a way to warn users that’s not missable. Messages (and user-error) could be covered by later messages. Hmm, maybe we can use warnings.

For template string please see that patch that applies on top of your patch. It seems to work for me. (Also see screenshot attached, brackets have different colors because I forgot to turn off rainbow-delimiter-mode when taking the shot). Also some of the lines are longer than 70 columns. Could you wrap those lines?

Thanks!
Yuan


[-- Attachment #2: template-string.patch --]
[-- Type: application/octet-stream, Size: 2241 bytes --]

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index d1f655cdc48..387acb4ebb1 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -3402,7 +3402,7 @@ c-paragraph-start
   js-mode "\\(@[[:alpha:]]+\\>\\|$\\)")
 
 ;;; Tree sitter integration
-(defcustom js-use-treesitter nil
+(defcustom js-use-tree-sitter nil
   "If non-nil, `js-mode' tries to use tree-sitter.
 Currently `js-mode' uses tree-sitter for font-locking,
 indentation, which-function and movement functions."
@@ -3459,7 +3459,9 @@ js--treesit-settings
   (treesit-font-lock-rules
    :language 'javascript
    :override t
-   `(
+   `(;; Everything overrides template string.
+     (template_string) @font-lock-string-face
+
      ((identifier) @font-lock-constant-face
       (:match "^[A-Z_][A-Z_\\d]*$" @font-lock-constant-face))
 
@@ -3551,13 +3553,7 @@ js--treesit-settings
      (comment) @font-lock-comment-face
      [,@js--treesit-keywords] @font-lock-keyword-face
 
-     ;; FIXME: We need to be able to set the priority for font-locking
-     ;; somehow.  We cannot just override all of the template string,
-     ;; as that would mess up interpolated expressions
-     ;;
-     ;; (template_string) @font-lock-string-face
-     (template_substitution ["${" "}"] @font-lock-constant-face)
-     )))
+     (template_substitution ["${" "}"] @font-lock-constant-face))))
 
 
 (defun js-treesit-current-defun ()
@@ -3604,7 +3600,7 @@ js-treesit--defun-query
     (variable_declarator)] @defun"))
 
 (defun js--treesit-can-enable-p ()
-  (if (and js-use-treesitter
+  (if (and js-use-tree-sitter
            (treesit-can-enable-p)
            (treesit-language-available-p 'javascript))
       t
@@ -3716,7 +3712,7 @@ js-mode
     ;;(syntax-propertize (point-max))
     ))
 
-(defcustom js-json-use-treesitter nil
+(defcustom js-json-use-tree-sitter nil
   "If non-nil, `js-json-mode' tries to use tree-sitter.
 Currently `js-json-mode' uses tree-sitter for font-locking and
 indentation."
@@ -3755,7 +3751,7 @@ js--json-treesit-indent-rules
 
 
 (defun js--json-treesit-can-enable-p ()
-  (if (and js-json-use-treesitter
+  (if (and js-json-use-tree-sitter
            (treesit-can-enable-p)
            (treesit-language-available-p 'json))
       t

[-- Attachment #3: Screen Shot 2022-10-10 at 8.37.34 AM.png --]
[-- Type: image/png, Size: 47813 bytes --]

[-- Attachment #4: Type: text/plain, Size: 2 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

[-- Attachment #1: Type: text/plain, Size: 948 bytes --]

>> I added some variartion of this.  I think also message is better,
>> because user-error makes the logic a little harder.  What do you
>> think?
>
> Yeah user-error is mostly the same as a message, since nobody (I
> think) turns on beeping anymore. Though I hope there is a way to warn
> users that’s not missable. Messages (and user-error) could be covered
> by later messages. Hmm, maybe we can use warnings.
>
> For template string please see that patch that applies on top of your
> patch. It seems to work for me. (Also see screenshot attached,
> brackets have different colors because I forgot to turn off
> rainbow-delimiter-mode when taking the shot). Also some of the lines
> are longer than 70 columns. Could you wrap those lines?
>

Yeah, it is better, though we still get some bleeding into the parens
etc.  Anyway, I think we can tweak this later.

Applied your changes, see latest revision.

Thanks,
Theo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-tree-sitter-functionality-to-js-mode.patch --]
[-- Type: text/x-diff, Size: 20554 bytes --]

From 10a224e293b0e880fcf77886730f1b89cd95c235 Mon Sep 17 00:00:00 2001
From: Theodor Thornhill <theo@thornhill.no>
Date: Sun, 9 Oct 2022 21:22:55 +0200
Subject: [PATCH] Add tree-sitter functionality to js-mode

* lisp/progmodes/js.el (treesit): Add tree-sitter dependency

(js-use-treesitter): Defcustom to enable tree-sitter as the engine for
font-lock, indentation and more.

(js--treesit-backward-up-list): Helper to enable indenting of empty bodies.

(js--treesit-indent-rules): Rules to be used when indenting JavaScript code.

(js--treesit-keywords): Keywords as seen with tree-sitter.

(js--treesit-settings): Font-Lock settings for JavaScript.

(js-treesit-current-defun): Helper for which-function-mode.

(js--treesit-move-to-node): Helper for movement to specific nodes in
the tree-sitter tree.

(js--treesit-beginning-of-defun, js--treesit-end-of-defun): Movement
functions for js-mode.

(js--treesit-enable): Helper function to enable tree-sitter.  Skips
over most initalization done by the CC-Mode variant.

(js-json-use-treesitter): Defcustom to enable tree-sitter as the
engine for font-lock and indentation for JSON files.

(js--json-treesit-settings): Font-Lock settings for JSON.

(js--json-treesit-indent-rules): Rules to be used when indenting JSON code

(js--json-treesit-enable): Helper function to enable tree-sitter.
Skips over most initialization done by the CC-Mode variant, in
addition to ignoring some JavaScript specific settings.

(js-mode, js-json-mode, js-jsx-mode): Add support for tree-sitter
functionalities.
---
 lisp/progmodes/js.el | 462 +++++++++++++++++++++++++++++++++++--------
 1 file changed, 378 insertions(+), 84 deletions(-)

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index b920ef6c2c..61ab0bb401 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -53,6 +53,7 @@
 (require 'imenu)
 (require 'json)
 (require 'prog-mode)
+(require 'treesit)
 
 (eval-when-compile
   (require 'cl-lib)
@@ -3400,100 +3401,391 @@ js-jsx--detect-after-change
 (c-lang-defconst c-paragraph-start
   js-mode "\\(@[[:alpha:]]+\\>\\|$\\)")
 
+;;; Tree sitter integration
+(defcustom js-use-tree-sitter nil
+  "If non-nil, `js-mode' tries to use tree-sitter.
+Currently `js-mode' uses tree-sitter for font-locking,
+indentation, which-function and movement functions."
+  :version "29.1"
+  :type 'boolean
+  :safe 'booleanp)
+
+(defun js--treesit-backward-up-list ()
+  (lambda (_node _parent _bol &rest _)
+    (save-excursion
+      (backward-up-list 1 nil t)
+      (goto-char
+       (treesit-node-start
+        (treesit-node-at (point))))
+      (back-to-indentation)
+      (point))))
+
+(defvar js--treesit-indent-rules
+  `((javascript
+     (no-node (js--treesit-backward-up-list) ,js-indent-level)
+     ((node-is "}") parent-bol 0)
+     ((node-is ")") parent-bol 0)
+     ((node-is "]") parent-bol 0)
+     ((node-is ">") parent-bol 0)
+     ((node-is ".") parent-bol ,js-indent-level)
+     ((parent-is "named_imports") parent-bol ,js-indent-level)
+     ((parent-is "statement_block") parent-bol ,js-indent-level)
+     ((parent-is "variable_declarator") parent-bol ,js-indent-level)
+     ((parent-is "arguments") parent-bol ,js-indent-level)
+     ((parent-is "array") parent-bol ,js-indent-level)
+     ((parent-is "formal_parameters") parent-bol ,js-indent-level)
+     ((parent-is "template_substitution") parent-bol ,js-indent-level)
+     ((parent-is "object_pattern") parent-bol ,js-indent-level)
+     ((parent-is "object") parent-bol ,js-indent-level)
+     ((parent-is "arrow_function") parent-bol ,js-indent-level)
+     ((parent-is "parenthesized_expression") parent-bol ,js-indent-level)
+
+     ;; JSX
+     ((parent-is "jsx_opening_element") parent ,js-indent-level)
+     ((node-is "jsx_closing_element") parent 0)
+     ((node-is "jsx_text") parent ,js-indent-level)
+     ((parent-is "jsx_element") parent ,js-indent-level)
+     ((node-is "/") parent 0)
+     ((parent-is "jsx_self_closing_element") parent ,js-indent-level))))
+
+(defvar js--treesit-keywords
+  '("as" "async" "await" "break" "case" "catch" "class" "const" "continue"
+    "debugger" "default" "delete" "do" "else" "export" "extends" "finally"
+    "for" "from" "function" "get" "if" "import" "in" "instanceof" "let" "new"
+    "of" "return" "set" "static" "switch" "switch" "target" "throw" "try"
+    "typeof" "var" "void" "while" "with" "yield"))
+
+(defvar js--treesit-settings
+  (treesit-font-lock-rules
+   :language 'javascript
+   :override t
+   `(;; Everything overrides template string.
+     (template_string) @font-lock-string-face
+
+     ((identifier) @font-lock-constant-face
+      (:match "^[A-Z_][A-Z_\\d]*$" @font-lock-constant-face))
+
+     (new_expression
+      constructor: (identifier) @font-lock-type-face)
+
+     (function
+      name: (identifier) @font-lock-function-name-face)
+
+     (class_declaration
+      name: (identifier) @font-lock-type-face)
+
+     (function_declaration
+      name: (identifier) @font-lock-function-name-face)
+
+     (method_definition
+      name: (property_identifier) @font-lock-function-name-face)
+
+     (variable_declarator
+      name: (identifier) @font-lock-function-name-face
+      value: [(function) (arrow_function)])
+
+     (variable_declarator
+      name: (array_pattern
+             (identifier)
+             (identifier)
+             @font-lock-function-name-face)
+      value: (array (number) (function)))
+
+     (assignment_expression
+      left: [(identifier) @font-lock-function-name-face
+             (member_expression property: (property_identifier)
+                                @font-lock-function-name-face)]
+      right: [(function) (arrow_function)])
+
+     (call_expression
+      function: [(identifier) @font-lock-function-name-face
+                 (member_expression
+                  property:
+                  (property_identifier) @font-lock-function-name-face)])
+
+     (variable_declarator
+      name: (identifier) @font-lock-variable-name-face)
+
+     (assignment_expression
+      left: [(identifier) @font-lock-variable-name-face
+             (member_expression
+              property: (property_identifier) @font-lock-variable-name-face)])
+
+     (for_in_statement
+      left: (identifier) @font-lock-variable-name-face)
+
+     (arrow_function
+      parameter: (identifier) @font-lock-variable-name-face)
+
+     (pair key: (property_identifier) @font-lock-variable-name-face)
+
+     (pair value: (identifier) @font-lock-variable-name-face)
+
+     (pair
+      key: (property_identifier) @font-lock-function-name-face
+      value: [(function) (arrow_function)])
+
+     ((shorthand_property_identifier) @font-lock-variable-name-face)
+
+     (pair_pattern key: (property_identifier) @font-lock-variable-name-face)
+
+     ((shorthand_property_identifier_pattern) @font-lock-variable-name-face)
+
+     (array_pattern (identifier) @font-lock-variable-name-face)
+
+     (jsx_opening_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_closing_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_self_closing_element
+      [(nested_identifier (identifier)) (identifier)]
+      @font-lock-function-name-face)
+
+     (jsx_attribute
+      (property_identifier)
+      @font-lock-constant-face)
+
+     [(this) (super)] @font-lock-keyword-face
+
+     [(true) (false) (null)] @font-lock-constant-face
+     (regex pattern: (regex_pattern)) @font-lock-string-face
+     (number) @font-lock-constant-face
+
+     (string) @font-lock-string-face
+     (comment) @font-lock-comment-face
+     [,@js--treesit-keywords] @font-lock-keyword-face
+
+     (template_substitution ["${" "}"] @font-lock-constant-face))))
+
+
+(defun js-treesit-current-defun ()
+  "Return name of surrounding function.
+This function can be used as a value in `which-func-functions'"
+  (let ((node (treesit-node-at (point)))
+        (name-list ()))
+    (cl-loop while node
+             if (pcase (treesit-node-type node)
+                  ("function_declaration" t)
+                  ("method_definition" t)
+                  ("class_declaration" t)
+                  ("variable_declarator" t)
+                  (_ nil))
+             do (push (treesit-node-text
+                       (treesit-node-child-by-field-name node "name")
+                       t)
+                      name-list)
+             do (setq node (treesit-node-parent node))
+             finally return  (string-join name-list "."))))
+
+(defun js--treesit-move-to-node (fn)
+  (when-let ((found-node
+              (treesit-parent-until
+               (treesit-node-at (point))
+               (lambda (parent)
+                 (treesit-query-capture
+                  parent
+                  js-treesit--defun-query)))))
+    (goto-char (funcall fn found-node))))
+
+(defun js--treesit-beginning-of-defun (&optional _arg)
+  (js--treesit-move-to-node #'treesit-node-start))
+
+(defun js--treesit--end-of-defun (&optional _arg)
+  (js--tressit-move-to-node #'treesit-node-end))
+
+(defvar js-treesit--defun-query
+  (treesit-query-compile
+   'javascript
+   "[(class_declaration)
+    (method_definition)
+    (function_declaration)
+    (variable_declarator)] @defun"))
+
+(defun js--treesit-can-enable-p ()
+  (if (and js-use-tree-sitter
+           (treesit-can-enable-p)
+           (treesit-language-available-p 'javascript))
+      t
+    (message "Cannot enable Tree Sitter for JavaScript.")
+    nil))
+
+(defun js--treesit-enable ()
+  (setq-local treesit-simple-indent-rules js--treesit-indent-rules)
+  (setq-local indent-line-function #'treesit-indent)
+
+  (setq-local beginning-of-defun-function #'js--treesit-beginning-of-defun)
+  (setq-local end-of-defun-function #'js--treesit-end-of-defun)
+
+  (setq-local font-lock-defaults '(nil t))
+  (setq-local treesit-font-lock-settings js--treesit-settings)
+
+  (add-hook 'which-func-functions #'js-treesit-current-defun nil t)
+
+  (treesit-font-lock-enable))
+
 ;;; Main Function
 
 ;;;###autoload
 (define-derived-mode js-mode prog-mode "JavaScript"
   "Major mode for editing JavaScript."
   :group 'js
-  ;; Ensure all CC Mode "lang variables" are set to valid values.
-  (c-init-language-vars js-mode)
-  (setq-local indent-line-function #'js-indent-line)
-  (setq-local beginning-of-defun-function #'js-beginning-of-defun)
-  (setq-local end-of-defun-function #'js-end-of-defun)
-  (setq-local open-paren-in-column-0-is-defun-start nil)
-  (setq-local font-lock-defaults
-              (list js--font-lock-keywords nil nil nil nil
-                    '(font-lock-syntactic-face-function
-                      . js-font-lock-syntactic-face-function)))
-  (setq-local syntax-propertize-function #'js-syntax-propertize)
-  (add-hook 'syntax-propertize-extend-region-functions
-            #'syntax-propertize-multiline 'append 'local)
-  (add-hook 'syntax-propertize-extend-region-functions
-            #'js--syntax-propertize-extend-region 'append 'local)
-  (setq-local prettify-symbols-alist js--prettify-symbols-alist)
-
-  (setq-local parse-sexp-ignore-comments t)
-  (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
-
   ;; Comments
   (setq-local comment-start "// ")
   (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
   (setq-local comment-end "")
-  (setq-local fill-paragraph-function #'js-fill-paragraph)
-  (setq-local normal-auto-fill-function #'js-do-auto-fill)
-
-  ;; Parse cache
-  (add-hook 'before-change-functions #'js--flush-caches t t)
-
-  ;; Frameworks
-  (js--update-quick-match-re)
-
-  ;; Syntax extensions
-  (unless (js-jsx--detect-and-enable)
-    (add-hook 'after-change-functions #'js-jsx--detect-after-change nil t))
-  (js-use-syntactic-mode-name)
-
-  ;; Imenu
-  (setq imenu-case-fold-search nil)
-  (setq imenu-create-index-function #'js--imenu-create-index)
-
-  ;; for filling, pretend we're cc-mode
-  (c-foreign-init-lit-pos-cache)
-  (add-hook 'before-change-functions #'c-foreign-truncate-lit-pos-cache nil t)
-  (setq-local comment-line-break-function #'c-indent-new-comment-line)
-  (setq-local comment-multi-line t)
-  (setq-local electric-indent-chars
-	      (append "{}():;," electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
-  (setq-local electric-layout-rules
-	      '((?\; . after) (?\{ . after) (?\} . before)))
-
-  (let ((c-buffer-is-cc-mode t))
-    ;; FIXME: These are normally set by `c-basic-common-init'.  Should
-    ;; we call it instead?  (Bug#6071)
-    (make-local-variable 'paragraph-start)
-    (make-local-variable 'paragraph-separate)
-    (make-local-variable 'paragraph-ignore-fill-prefix)
-    (make-local-variable 'adaptive-fill-mode)
-    (make-local-variable 'adaptive-fill-regexp)
-    ;; While the full CC Mode style system is not yet in use, set the
-    ;; pertinent style variables manually.
-    (c-initialize-builtin-style)
-    (let ((style (cc-choose-style-for-mode 'js-mode c-default-style)))
-      (c-set-style style))
-    (setq c-block-comment-prefix "* "
-          c-comment-prefix-regexp "//+\\|\\**")
-    (c-setup-paragraph-variables))
-
-  ;; Important to fontify the whole buffer syntactically! If we don't,
-  ;; then we might have regular expression literals that aren't marked
-  ;; as strings, which will screw up parse-partial-sexp, scan-lists,
-  ;; etc. and produce maddening "unbalanced parenthesis" errors.
-  ;; When we attempt to find the error and scroll to the portion of
-  ;; the buffer containing the problem, JIT-lock will apply the
-  ;; correct syntax to the regular expression literal and the problem
-  ;; will mysteriously disappear.
-  ;; FIXME: We should instead do this fontification lazily by adding
-  ;; calls to syntax-propertize wherever it's really needed.
-  ;;(syntax-propertize (point-max))
-  )
+
+  (if (js--treesit-can-enable-p)
+      (js--treesit-enable)
+    ;; Ensure all CC Mode "lang variables" are set to valid values.
+    (c-init-language-vars js-mode)
+    (setq-local indent-line-function #'js-indent-line)
+    (setq-local beginning-of-defun-function #'js-beginning-of-defun)
+    (setq-local end-of-defun-function #'js-end-of-defun)
+    (setq-local open-paren-in-column-0-is-defun-start nil)
+    (setq-local font-lock-defaults
+                (list js--font-lock-keywords nil nil nil nil
+                      '(font-lock-syntactic-face-function
+                        . js-font-lock-syntactic-face-function)))
+    (setq-local syntax-propertize-function #'js-syntax-propertize)
+    (add-hook 'syntax-propertize-extend-region-functions
+              #'syntax-propertize-multiline 'append 'local)
+    (add-hook 'syntax-propertize-extend-region-functions
+              #'js--syntax-propertize-extend-region 'append 'local)
+    (setq-local prettify-symbols-alist js--prettify-symbols-alist)
+
+    (setq-local parse-sexp-ignore-comments t)
+    (setq-local which-func-imenu-joiner-function #'js--which-func-joiner)
+
+    (setq-local fill-paragraph-function #'js-fill-paragraph)
+    (setq-local normal-auto-fill-function #'js-do-auto-fill)
+
+    ;; Parse cache
+    (add-hook 'before-change-functions #'js--flush-caches t t)
+
+    ;; Frameworks
+    (js--update-quick-match-re)
+
+    ;; Syntax extensions
+    (unless (js-jsx--detect-and-enable)
+      (add-hook 'after-change-functions #'js-jsx--detect-after-change nil t))
+    (js-use-syntactic-mode-name)
+
+    ;; Imenu
+    (setq imenu-case-fold-search nil)
+    (setq imenu-create-index-function #'js--imenu-create-index)
+
+    ;; for filling, pretend we're cc-mode
+    (c-foreign-init-lit-pos-cache)
+    (add-hook 'before-change-functions #'c-foreign-truncate-lit-pos-cache nil t)
+    (setq-local comment-line-break-function #'c-indent-new-comment-line)
+    (setq-local comment-multi-line t)
+    (setq-local electric-indent-chars
+	        (append "{}():;," electric-indent-chars)) ;FIXME: js2-mode adds "[]*".
+    (setq-local electric-layout-rules
+	        '((?\; . after) (?\{ . after) (?\} . before)))
+
+    (let ((c-buffer-is-cc-mode t))
+      ;; FIXME: These are normally set by `c-basic-common-init'.  Should
+      ;; we call it instead?  (Bug#6071)
+      (make-local-variable 'paragraph-start)
+      (make-local-variable 'paragraph-separate)
+      (make-local-variable 'paragraph-ignore-fill-prefix)
+      (make-local-variable 'adaptive-fill-mode)
+      (make-local-variable 'adaptive-fill-regexp)
+      ;; While the full CC Mode style system is not yet in use, set the
+      ;; pertinent style variables manually.
+      (c-initialize-builtin-style)
+      (let ((style (cc-choose-style-for-mode 'js-mode c-default-style)))
+        (c-set-style style))
+      (setq c-block-comment-prefix "* "
+            c-comment-prefix-regexp "//+\\|\\**")
+      (c-setup-paragraph-variables))
+
+    ;; Important to fontify the whole buffer syntactically! If we don't,
+    ;; then we might have regular expression literals that aren't marked
+    ;; as strings, which will screw up parse-partial-sexp, scan-lists,
+    ;; etc. and produce maddening "unbalanced parenthesis" errors.
+    ;; When we attempt to find the error and scroll to the portion of
+    ;; the buffer containing the problem, JIT-lock will apply the
+    ;; correct syntax to the regular expression literal and the problem
+    ;; will mysteriously disappear.
+    ;; FIXME: We should instead do this fontification lazily by adding
+    ;; calls to syntax-propertize wherever it's really needed.
+    ;;(syntax-propertize (point-max))
+    ))
+
+(defcustom js-json-use-tree-sitter nil
+  "If non-nil, `js-json-mode' tries to use tree-sitter.
+Currently `js-json-mode' uses tree-sitter for font-locking and
+indentation."
+  :version "29.1"
+  :type 'boolean
+  :safe 'booleanp)
+
+(defvar js--json-treesit-settings
+  (treesit-font-lock-rules
+   :language 'json
+   :override t
+   `(
+     (pair
+      key: (_) @font-lock-string-face)
+
+     (string) @font-lock-string-face
+
+     (number) @font-lock-constant-face
+
+     [(null) (true) (false)] @font-lock-constant-face
+
+     (escape_sequence) @font-lock-constant-face
+
+     (comment) @font-lock-comment-face
+     )))
+
+
+(defvar js--json-treesit-indent-rules
+  `((json
+     (no-node (js--treesit-backward-up-list) ,js-indent-level)
+     ((node-is "}") parent-bol 0)
+     ((node-is ")") parent-bol 0)
+     ((node-is "]") parent-bol 0)
+     ((parent-is "object") parent-bol ,js-indent-level)
+     )))
+
+
+(defun js--json-treesit-can-enable-p ()
+  (if (and js-json-use-tree-sitter
+           (treesit-can-enable-p)
+           (treesit-language-available-p 'json))
+      t
+    (error "Cannot enable Tree Sitter for JSON.")
+    nil))
+
+
+(defun js--json-treesit-enable ()
+  (setq-local treesit-simple-indent-rules js--json-treesit-indent-rules)
+  (setq-local indent-line-function #'treesit-indent)
+
+  (setq-local beginning-of-defun-function #'ignore)
+  (setq-local end-of-defun-function #'ignore)
+
+  (setq-local font-lock-defaults '(nil t))
+  (setq-local treesit-font-lock-settings js--json-treesit-settings)
+
+  (treesit-font-lock-enable))
+
 
 ;;;###autoload
 (define-derived-mode js-json-mode js-mode "JSON"
-  (setq-local js-enabled-frameworks nil)
-  ;; Speed up `syntax-ppss': JSON files can be big but can't hold
-  ;; regexp matchers nor #! thingies (and `js-enabled-frameworks' is nil).
-  (setq-local syntax-propertize-function #'ignore))
+  (if (js--json-treesit-can-enable-p)
+      (js--json-treesit-enable)
+    (setq-local js-enabled-frameworks nil)
+    ;; Speed up `syntax-ppss': JSON files can be big but can't hold
+    ;; regexp matchers nor #! thingies (and `js-enabled-frameworks' is nil).
+    (setq-local syntax-propertize-function #'ignore)))
 
 ;; Since we made JSX support available and automatically-enabled in
 ;; the base `js-mode' (for ease of use), now `js-jsx-mode' simply
@@ -3520,9 +3812,11 @@ js-jsx-mode
 `js-jsx-enable' in `js-mode-hook'.  You may be better served by
 one of the aforementioned options instead of using this mode."
   :group 'js
-  (js-jsx-enable)
-  (setq-local comment-region-function #'js-jsx--comment-region)
-  (js-use-syntactic-mode-name))
+  (if (js--treesit-can-enable-p)
+      (js--treesit-enable)
+    (js-jsx-enable)
+    (setq-local comment-region-function #'js-jsx--comment-region)
+    (js-use-syntactic-mode-name)))
 
 (defun js-jsx--comment-region (beg end &optional arg)
   (if (or (js-jsx--context)
-- 
2.34.1

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 10, 2022, at 9:29 AM, Theodor Thornhill <theo@thornhill.no> wrote:
> 
>>> I added some variartion of this.  I think also message is better,
>>> because user-error makes the logic a little harder.  What do you
>>> think?
>> 
>> Yeah user-error is mostly the same as a message, since nobody (I
>> think) turns on beeping anymore. Though I hope there is a way to warn
>> users that’s not missable. Messages (and user-error) could be covered
>> by later messages. Hmm, maybe we can use warnings.
>> 
>> For template string please see that patch that applies on top of your
>> patch. It seems to work for me. (Also see screenshot attached,
>> brackets have different colors because I forgot to turn off
>> rainbow-delimiter-mode when taking the shot). Also some of the lines
>> are longer than 70 columns. Could you wrap those lines?
>> 
> 
> Yeah, it is better, though we still get some bleeding into the parens
> etc.  Anyway, I think we can tweak this later.
> 
> Applied your changes, see latest revision.
> 
> Thanks,
> Theo
> 
> <0001-Add-tree-sitter-functionality-to-js-mode.patch>

Thanks! Looks good, I’ll apply it. BTW, you don’t need to explain what does a function do in the commit message, just what changed. For new functions, you can just say “New function”. And you can merge several entries together, like

(js--treesit-backward-up-list)
(js--treesit-indent-rules)
(js--treesit-keywords)
(js--treesit-settings): New functions.


Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 10, 2022, at 10:16 AM, Yuan Fu <casouri@gmail.com> wrote:
> 
> 
> 
>> On Oct 10, 2022, at 9:29 AM, Theodor Thornhill <theo@thornhill.no> wrote:
>> 
>>>> I added some variartion of this.  I think also message is better,
>>>> because user-error makes the logic a little harder.  What do you
>>>> think?
>>> 
>>> Yeah user-error is mostly the same as a message, since nobody (I
>>> think) turns on beeping anymore. Though I hope there is a way to warn
>>> users that’s not missable. Messages (and user-error) could be covered
>>> by later messages. Hmm, maybe we can use warnings.
>>> 
>>> For template string please see that patch that applies on top of your
>>> patch. It seems to work for me. (Also see screenshot attached,
>>> brackets have different colors because I forgot to turn off
>>> rainbow-delimiter-mode when taking the shot). Also some of the lines
>>> are longer than 70 columns. Could you wrap those lines?
>>> 
>> 
>> Yeah, it is better, though we still get some bleeding into the parens
>> etc.  Anyway, I think we can tweak this later.
>> 
>> Applied your changes, see latest revision.
>> 
>> Thanks,
>> Theo
>> 
>> <0001-Add-tree-sitter-functionality-to-js-mode.patch>
> 
> Thanks! Looks good, I’ll apply it. BTW, you don’t need to explain what does a function do in the commit message, just what changed. For new functions, you can just say “New function”. And you can merge several entries together, like
> 
> (js--treesit-backward-up-list)
> (js--treesit-indent-rules)
> (js--treesit-keywords)
> (js--treesit-settings): New functions.
> 
> 
> Yuan

Pushed. Please also see the change I made after your patch, for beginning/end-of-defun functions I added a new function treesit-search-forward-goto which is very handy, it should be faster, simpler to use, and wouldn’t get stuck like the old approach. And I changed “variable_declarator” to “lexical_declaration”.

Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

>
> Pushed. Please also see the change I made after your patch, for
> beginning/end-of-defun functions I added a new function
> treesit-search-forward-goto which is very handy, it should be faster,
> simpler to use, and wouldn’t get stuck like the old approach. And I
> changed “variable_declarator” to “lexical_declaration”.
>

Thank you!  I've been looking for that, hehe.  Well, I stole that in the
upcoming patch to typescript-mode.el.

Thanks,
Theodor

Re: Call for volunteers: add tree-sitter support to major modes

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 2575 bytes --]

Testing js-mode with tree-sitter enabled, it seems more responsive than 
the old elisp-implementation, and it also picks up more syntax elements, 
like function-definitions/calls which it didn't use to.

This puts it on par with js2-mode, but probably more correct and with 
better performance.

All in all I would consider this a upgrade all users should want.

--
Jostein

On 10.10.2022 09:08, Theodor Thornhill wrote:
> Hi!
>
>> Looks good! Here are some comments.
>>
>> +
>> +     ;; FIXME: We need to be able to set the priority for font-locking
>> +     ;; somehow.  We cannot just override all of the template string,
>> +     ;; as that would mess up interpolated expressions
>> +     ;;
>> +     ;; (template_string) @font-lock-string-face
>> +     (template_substitution ["${" "}"] @font-lock-constant-face)
>> +     )))
>>
>> What exactly do you mean by priority here? Why doesn't :override t
>> work?
>>
> In template strings in JavaScript we have issues with precedence in that
> we default to string font locking for everything that does _not_ have
> specific rules.  An image speaks a thousand words, so I'll add
> screenshots.  What I need is some sort of way to ensure that inside of
> template strings font-locking should _not_ happen
>
>> +
>> +(defvar js-treesit--defun-query
>> +  "[(class_declaration)
>> +    (method_definition)
>> +    (function_declaration)
>> +    (variable_declarator)] @defun")
>>
>> This should be compiled.
>>
>> +
>> +(defun js--treesit-enable ()
>> +  (unless (and (treesit-can-enable-p)
>> +               (treesit-language-available-p 'javascript))
>> +    (error "Tree sitter isn't available"))
>>
>> I don't think we should error here, I'd displaying a message instead.
>>
>> +
>> +  ;; Comments
>> +  (setq-local comment-start "// ")
>> +  (setq-local comment-start-skip "\\(?://+\\|/\\*+\\)\\s *")
>> +  (setq-local comment-end "")
>>
>> I think it's best to not repeat code, could you move this outside the
>> (if tree-sitter) form and have it run regardless?
>>
>> +(defun js--json-treesit-enable ()
>> +  (unless (and (treesit-can-enable-p)
>> +               (treesit-language-available-p 'json))
>> +    (error "Tree sitter isn't available"))
>>
>> Same as above, IMO message is better.
>>
> I added some variartion of this.  I think also message is better,
> because user-error makes the logic a little harder.  What do you think?
>
>
> Thanks, see attached patch:
>
-- 
Vennlig hilsen
*Jostein Kjønigsen*

jostein@kjonigsen.net 🍵 jostein@gmail.com
https://jostein.kjønigsen.no <https://jostein.kjønigsen.no>

[-- Attachment #2: Type: text/html, Size: 3428 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:

> Testing js-mode with tree-sitter enabled, it seems more responsive than 
> the old elisp-implementation, and it also picks up more syntax elements, 
> like function-definitions/calls which it didn't use to.
>
> This puts it on par with js2-mode, but probably more correct and with 
> better performance.
>
> All in all I would consider this a upgrade all users should want.
>


Good news!

Re: Call for volunteers: add tree-sitter support to major modes

[Dmitry Gutov]

On 09.10.2022 15:41, Theodor Thornhill wrote:
> I don't think there's a need for imenu here, though that could be
> implemented as well.  Eglot handles imenu just fine, and my guess is
> that most people will move over to that when available.

IME Imenu is handy to have available when the project is small, or does 
not yet compile, or the language server is not installed, or not working 
properly. Etc.

Re: Call for volunteers: add tree-sitter support to major modes

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

> Some of the popular major modes for which we would like to see
> tree-sitter support in Emacs 29 are:
>

[...]

>  emacs-lisp-mode

How much of a priority is using tree sitter for Emacs Lisp?  It appears
that this[0] is the grammar implementation, and it currently doesn't
support the following:

    Autoload cookies
    Definitions (e.g. defun, defvar, defmacro)

As far as I understand, this means that from your list of enhanced
features, the following will either be restricted or not work at all:

 - font-lock
 - indentation
 - imenu
 - which-func

Also, when taking a look at the current implementation for Python[1][2],
I cannot but wonder if a few high-level functions/macros should be added
first to make it easier.

One more thing, it looks like the usage of tree-sitter is enabled for
python by toggling `python-use-tree-sitter'.  Will every mode now have
this kind of an option?  I would hope not...

[0] https://github.com/Wilfred/tree-sitter-elisp
[1] https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/python.el?h=feature/tree-sitter#n953
[2] https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/python.el?h=feature/tree-sitter#n5324

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Philip Kaludercic <philipk@posteo.net> writes:

> How much of a priority is using tree sitter for Emacs Lisp?  It appears
> that this[0] is the grammar implementation, and it currently doesn't
> support the following:
>
>     Autoload cookies
>     Definitions (e.g. defun, defvar, defmacro)
>
> As far as I understand, this means that from your list of enhanced
> features, the following will either be restricted or not work at all:
>
>  - font-lock
>  - indentation
>  - imenu
>  - which-func

BTW, what exactly is the problem with the existing emacs-lisp-mode? How
do we expect tree-sitter will make it better?

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,  Yuan Fu
>  <casouri@gmail.com>
> Date: Sun, 09 Oct 2022 20:27:55 +0800
> 
> BTW, what exactly is the problem with the existing emacs-lisp-mode? How
> do we expect tree-sitter will make it better?

I'd like Emacs to stop using ad-hoc-ery for fontification,
indentation, and whatnot.  It doesn't have to be better, just based on
the language grammar.

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> I'd like Emacs to stop using ad-hoc-ery for fontification,

I can't say as for fontification.

> indentation

But isn't SMIE the solution for that?

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: philipk@posteo.net,  emacs-devel@gnu.org,  casouri@gmail.com
> Date: Sun, 09 Oct 2022 22:01:20 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > indentation
> 
> But isn't SMIE the solution for that?

I don't think we should develop parsing technology in-house, so
ideally no.

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: emacs-devel@gnu.org, Yuan Fu <casouri@gmail.com>
> Date: Sun, 09 Oct 2022 12:26:09 +0000
> 
> >  emacs-lisp-mode
> 
> How much of a priority is using tree sitter for Emacs Lisp?  It appears
> that this[0] is the grammar implementation, and it currently doesn't
> support the following:
> 
>     Autoload cookies
>     Definitions (e.g. defun, defvar, defmacro)
> 
> As far as I understand, this means that from your list of enhanced
> features, the following will either be restricted or not work at all:
> 
>  - font-lock
>  - indentation
>  - imenu
>  - which-func

If emacs-lisp-mode cannot be reasonably supported, we can omit it for
now.

> Also, when taking a look at the current implementation for Python[1][2],
> I cannot but wonder if a few high-level functions/macros should be added
> first to make it easier.
> 
> One more thing, it looks like the usage of tree-sitter is enabled for
> python by toggling `python-use-tree-sitter'.  Will every mode now have
> this kind of an option?  I would hope not...
> 
> [0] https://github.com/Wilfred/tree-sitter-elisp
> [1] https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/python.el?h=feature/tree-sitter#n953
> [2] https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/python.el?h=feature/tree-sitter#n5324

That's a separate discussion, so please start a new thread.

Re: Call for volunteers: add tree-sitter support to major modes

[Brian]

Hello,

I looked through some major modes and IIUC they already have an
author assigned to them.  I haven't seen this practice on other
FLOSS projects I encountered GitHub, any pointers on how this
process works on Emacs?

TIA

On Sun, Oct 9, 2022 at 9:03 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> The Emacs 29 release branch is planned to be cut in about 1.5 months,
> and we want tree-sitter support to be in Emacs 29.  The Emacs Git
> repository has a feature branch that integrates tree-sitter with
> Emacs.  That branch is being prepared for landing on master.  However,
> currently there's only one major-mode which can use tree-sitter
> support on the branch: Python mode.  We need to add tree-sitter
> integration to more major modes, so that Emacs 29 could have
> reasonably useful tree-sitter support options in at least the popular
> major modes.
>
> This is a call for volunteers to work on adding tree-sitter support to
> major modes beyond Python, and on improving the existing support in
> Python mode.
>
> Some of the popular major modes for which we would like to see
> tree-sitter support in Emacs 29 are:
>
>  c-mode
>  c++-mode
>  emacs-lisp-mode
>  js-mode
>  js-json-mode
>  ruby-mode
>  shell-script-mode
>
> We probably won't have all of them by Emacs 29, but we hope to have at
> least some.
>
> So if you are interested in enhancing your favorite major-mode with
> tree-sitter support, and by doing that making Emacs 29 much better,
> please checkout the feature/tree-sitter branch, and start working on
> some mode(s).  Tree-sitter support for a major-mode should include
> enhancement of one or more of the following Emacs features:
>
>  - font-lock
>  - indentation
>  - navigation (beginning-of-defun-function etc.)
>  - imenu
>  - which-func
>  - thing-at-point
>
> We think that font-lock is the absolute minimum, followed by
> indentation.
>
> To help with this effort, Yuan Fu, who did most of the development on
> this feature branch, wrote a document with guidance notes which aim to
> help you implement tree-sitter support for the above features.  You
> will find that document and some other helpful documentation in the
> admin/notes/tree-sitter directory and its subdirectories on the
> branch.  That directory also includes scripts for easy compilation of
> tree-sitter language modules (which you will need to download from the
> tree-sitter site).
>
> We hope that the existing support for Python can serve as working
> example of how to add tree-sitter support for other major modes, and
> together with the guidance document it will allow you to add support
> for more modes with minimum effort.  Don't hesitate to ask questions
> if something is unclear or you need more advice.
>
> We hope volunteers will step forward in time to make tree-sitter
> integration in Emacs 29 functional and useful.  To avoid waste of
> effort, please announce here on which major mode(s) you decided to
> work.
>
> Thanks in advance.
>

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Brian <brian.miller.0235@gmail.com>
> Date: Sun, 9 Oct 2022 14:36:22 +0000
> Cc: emacs-devel@gnu.org
> 
> I looked through some major modes and IIUC they already have an
> author assigned to them.  I haven't seen this practice on other
> FLOSS projects I encountered GitHub, any pointers on how this
> process works on Emacs?

What do you mean by "author assigned to them"?  Major modes have
authors, but they don't necessarily work on adding tree-sitter support
to the (and some no longer actively develop the mode at all).

For which modes are you interested to work on adding tree-sitter
support?

Re: Call for volunteers: add tree-sitter support to major modes

[Brian]

I do not understand why only one person is credited if it is a
collective effort.  Why does Emacs files have author fields, what
Is their significance, history?  I have only seen Emacs do this
and couldn't find anything on it in CONTRIBUTING file, so the
process feels somewhat opaque.

I could work with c, c++, js (seems taken) or elisp but haven't
completely thought it out or if I'd have enough time yet.

On Sun, Oct 9, 2022 at 2:53 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Brian <brian.miller.0235@gmail.com>
> > Date: Sun, 9 Oct 2022 14:36:22 +0000
> > Cc: emacs-devel@gnu.org
> >
> > I looked through some major modes and IIUC they already have an
> > author assigned to them.  I haven't seen this practice on other
> > FLOSS projects I encountered GitHub, any pointers on how this
> > process works on Emacs?
>
> What do you mean by "author assigned to them"?  Major modes have
> authors, but they don't necessarily work on adding tree-sitter support
> to the (and some no longer actively develop the mode at all).
>
> For which modes are you interested to work on adding tree-sitter
> support?

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Brian <brian.miller.0235@gmail.com>
> Date: Sun, 9 Oct 2022 15:20:41 +0000
> Cc: emacs-devel@gnu.org
> 
> I do not understand why only one person is credited if it is a
> collective effort.  Why does Emacs files have author fields, what
> Is their significance, history?  I have only seen Emacs do this
> and couldn't find anything on it in CONTRIBUTING file, so the
> process feels somewhat opaque.

Not only the original author is credited.  There's also "maintainer",
and then there's the etc/AUTHORS file and the "Acknowledgements" node
of the Emacs manual.

In a large project that runs for so many years, it is not practical to
have all the contributors spelled in the files to which they
contributed, so we do it differently.

Re: Call for volunteers: add tree-sitter support to major modes

[Brian]

Then what is the purpose of an author field, if attributions are in
those files?  It creates the impression that writing a single file then
forgetting about it is the most valuable contribution in this project.
I don't see why having a long history should require this, wouldn't it
work if everyone was credited in the same place?

On Sun, Oct 9, 2022 at 3:39 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> Not only the original author is credited.  There's also "maintainer",
> and then there's the etc/AUTHORS file and the "Acknowledgements" node
> of the Emacs manual.
>
> In a large project that runs for so many years, it is not practical to
> have all the contributors spelled in the files to which they
> contributed, so we do it differently.

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Brian <brian.miller.0235@gmail.com>
> Date: Sun, 9 Oct 2022 16:03:20 +0000
> Cc: emacs-devel@gnu.org
> 
> Then what is the purpose of an author field, if attributions are in
> those files?

It's the original author, the one who wrote the first version.

> It creates the impression that writing a single file then
> forgetting about it is the most valuable contribution in this project.
> I don't see why having a long history should require this, wouldn't it
> work if everyone was credited in the same place?

It's just how it is in Emacs.  It maybe takes some getting used to,
that's all.  Once you get used to it, you know that people's
contributions is more than just writing a file.

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 9, 2022, at 9:03 AM, Brian <brian.miller.0235@gmail.com> wrote:
> 
> Then what is the purpose of an author field, if attributions are in
> those files?  It creates the impression that writing a single file then
> forgetting about it is the most valuable contribution in this project.
> I don't see why having a long history should require this, wouldn't it
> work if everyone was credited in the same place?

I don’t think that’s the impression most people get from it. And I don’t think most people care.

Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Brian]

On Sun, Oct 9, 2022 at 11:45 PM Yuan Fu <casouri@gmail.com> wrote:
>
>
> > On Oct 9, 2022, at 9:03 AM, Brian <brian.miller.0235@gmail.com> wrote:
> >
> > Then what is the purpose of an author field, if attributions are in
> > those files?  It creates the impression that writing a single file then
> > forgetting about it is the most valuable contribution in this project.
> > I don't see why having a long history should require this, wouldn't it
> > work if everyone was credited in the same place?
>
> I don’t think that’s the impression most people get from it. And I don’t think most people care.
>
> Yuan

That is ideal of course, and I believe it is true for all Emacs regular
contributors, I meant to ask for everyone else who isn't one yet.
I also take it that you mostly work on existing things and not on
adding brand new features?

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Brian <brian.miller.0235@gmail.com> writes:

> That is ideal of course, and I believe it is true for all Emacs regular
> contributors, I meant to ask for everyone else who isn't one yet.
> I also take it that you mostly work on existing things and not on
> adding brand new features?

FWIW, I wouldn't care about how (or whether or not) I am attributed on
either new work or existing features.  It's not important.

Re: Call for volunteers: add tree-sitter support to major modes

[Brian]

On Mon, Oct 10, 2022 at 10:10 AM Po Lu <luangruo@yahoo.com> wrote:
>
> FWIW, I wouldn't care about how (or whether or not) I am attributed on
> either new work or existing features.  It's not important.

Leaving all this to Git is the best solution I think.

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 10, 2022, at 2:34 AM, Brian <brian.miller.0235@gmail.com> wrote:
> 
> On Sun, Oct 9, 2022 at 11:45 PM Yuan Fu <casouri@gmail.com> wrote:
>> 
>> 
>>> On Oct 9, 2022, at 9:03 AM, Brian <brian.miller.0235@gmail.com> wrote:
>>> 
>>> Then what is the purpose of an author field, if attributions are in
>>> those files?  It creates the impression that writing a single file then
>>> forgetting about it is the most valuable contribution in this project.
>>> I don't see why having a long history should require this, wouldn't it
>>> work if everyone was credited in the same place?
>> 
>> I don’t think that’s the impression most people get from it. And I don’t think most people care.
>> 
>> Yuan
> 
> That is ideal of course, and I believe it is true for all Emacs regular
> contributors, I meant to ask for everyone else who isn't one yet.
> I also take it that you mostly work on existing things and not on
> adding brand new features?

Yeah, except for tree-sitter ;-)

Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Monnier]

> I do not understand why only one person is credited if it is a
> collective effort.  Why does Emacs files have author fields, what
> Is their significance, history?

I don't have an official answer to give, but for all the packages to
which I contributed, I've usually seen something like the following
happen:

- Someone write a package.  When they distribute it they naturally put
  "Author:" there with their name and email address (not necessarily
  for fame but rather in order to provide a point of contact for
  feedback).
- Time passes, other people start contributing to it.
- Nobody dares to say that "I'm the author" since that would be a lie,
  so they don't touch the "Author:" line and just either leave their
  name out because they don't care about recognition and feedback, or
  put themselves as "Maintainer:".

So in the end `Author:` all too often means the one who got the original
impetus.  Which fraction of the overall work that turns out to be
depends a lot on the package.  At the same time, in most cases, the
original impetus imposes a general direction/design/approach/vision
which tends to persist long after that author stepped down.


        Stefan

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sun, 09 Oct 2022 23:04:33 -0400
> 
> I don't have an official answer to give, but for all the packages to
> which I contributed, I've usually seen something like the following
> happen:
> 
> - Someone write a package.  When they distribute it they naturally put
>   "Author:" there with their name and email address (not necessarily
>   for fame but rather in order to provide a point of contact for
>   feedback).
> - Time passes, other people start contributing to it.
> - Nobody dares to say that "I'm the author" since that would be a lie,
>   so they don't touch the "Author:" line and just either leave their
>   name out because they don't care about recognition and feedback, or
>   put themselves as "Maintainer:".
> 
> So in the end `Author:` all too often means the one who got the original
> impetus.  Which fraction of the overall work that turns out to be
> depends a lot on the package.  At the same time, in most cases, the
> original impetus imposes a general direction/design/approach/vision
> which tends to persist long after that author stepped down.

That's my impression as well.  And I do agree that the original
author's contribution is significant enough to justify the Author
header.

Re: Call for volunteers: add tree-sitter support to major modes

[Brian]

On Mon, Oct 10, 2022 at 6:25 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> > Date: Sun, 09 Oct 2022 23:04:33 -0400
> >
> > I don't have an official answer to give, but for all the packages to
> > which I contributed, I've usually seen something like the following
> > happen:
> >
> > - Someone write a package.  When they distribute it they naturally put
> >   "Author:" there with their name and email address (not necessarily
> >   for fame but rather in order to provide a point of contact for
> >   feedback).
> > - Time passes, other people start contributing to it.
> > - Nobody dares to say that "I'm the author" since that would be a lie,
> >   so they don't touch the "Author:" line and just either leave their
> >   name out because they don't care about recognition and feedback, or
> >   put themselves as "Maintainer:".
> >
> > So in the end `Author:` all too often means the one who got the original
> > impetus.  Which fraction of the overall work that turns out to be
> > depends a lot on the package.  At the same time, in most cases, the
> > original impetus imposes a general direction/design/approach/vision
> > which tends to persist long after that author stepped down.

I see, thank you both for detailed clarification.

> That's my impression as well.  And I do agree that the original
> author's contribution is significant enough to justify the Author
> header.

I think it also means a second wind will be somehow less visible than
a brand-new impetus, but that's just me.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Eli Zaretskii <eliz@gnu.org> writes:

>
> Some of the popular major modes for which we would like to see
> tree-sitter support in Emacs 29 are:
>
>  c-mode
>  c++-mode
>  emacs-lisp-mode
>  js-mode
>  js-json-mode
>  ruby-mode
>  shell-script-mode
>

I'd like to add in Java support here as well.  But since java is part of
cc-mode.el, maybe it'd be best if it got its own mode, say
java-treesitter-mode.el, so that we don't add a dependency into cc-mode?
If you agree, I can create such a major-mode as well.

Also, how should we mandate what modes get the new tree-sitter support?
It is pretty fast to add new modes, and adding support for languages not
already supported in core, such as say, golang, should be very feasible.
But maybe that is something that should live in elpa?

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Kangas]

Theodor Thornhill <theo@thornhill.no> writes:

> Also, how should we mandate what modes get the new tree-sitter support?

Why not all of them (within reason, e.g. let's skip joke languages)?

> It is pretty fast to add new modes, and adding support for languages not
> already supported in core, such as say, golang, should be very feasible.
> But maybe that is something that should live in elpa?

What would be the reasons not to have it in core?  I think such support
should already be there, myself.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Stefan Kangas <stefankangas@gmail.com> writes:

> Theodor Thornhill <theo@thornhill.no> writes:
>
>> Also, how should we mandate what modes get the new tree-sitter support?
>
> Why not all of them (within reason, e.g. let's skip joke languages)?
>
>> It is pretty fast to add new modes, and adding support for languages not
>> already supported in core, such as say, golang, should be very feasible.
>> But maybe that is something that should live in elpa?
>
> What would be the reasons not to have it in core?  I think such support
> should already be there, myself.

I have no idea, but people seem to get really angry when someone
suggests moving stuff to core...

Well, anyways - I've already started making support for the languages I
use over at [0], and they should be easy additions when they too are
mature.  If you want, I can continue this work.  Or just submit that
repo to ELPA sometime in the future?

Theo


[0]: https://git.sr.ht/~theo/tree-sitter-modes/tree

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 9, 2022, at 1:01 PM, Theodor Thornhill <theo@thornhill.no> wrote:
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> 
>> Some of the popular major modes for which we would like to see
>> tree-sitter support in Emacs 29 are:
>> 
>> c-mode
>> c++-mode
>> emacs-lisp-mode
>> js-mode
>> js-json-mode
>> ruby-mode
>> shell-script-mode
>> 
> 
> I'd like to add in Java support here as well.  But since java is part of
> cc-mode.el, maybe it'd be best if it got its own mode, say
> java-treesitter-mode.el, so that we don't add a dependency into cc-mode?
> If you agree, I can create such a major-mode as well.

I think we can use the same major mode, just skip the cc-mode initialization and only enable tree-sitter stuff if java-use-tree-sitter is true. But I want to know Alan’s thought on this. I included him in CC.

> Also, how should we mandate what modes get the new tree-sitter support?
> It is pretty fast to add new modes, and adding support for languages not
> already supported in core, such as say, golang, should be very feasible.
> But maybe that is something that should live in elpa?

If you want to add tree-sitter support for some built-in mode, go for it. But I would focus on the modes we already have in core. For modes like golang where there is a widely used mode out there (in this case on ELPA), we should either ask to move it into core first, or ask the author to maybe add tree-sitter support, or submit a patch to him/her, rather than creating a duplicate go-mode in core just for the sake of tree-sitter.

Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Alan Mackenzie]

Hello, Yuan.

On Sun, Oct 09, 2022 at 17:01:45 -0700, Yuan Fu wrote:


> > On Oct 9, 2022, at 1:01 PM, Theodor Thornhill <theo@thornhill.no> wrote:

> > Eli Zaretskii <eliz@gnu.org> writes:


> >> Some of the popular major modes for which we would like to see
> >> tree-sitter support in Emacs 29 are:

> >> c-mode
> >> c++-mode
> >> emacs-lisp-mode
> >> js-mode
> >> js-json-mode
> >> ruby-mode
> >> shell-script-mode


> > I'd like to add in Java support here as well.  But since java is part
> > of cc-mode.el, maybe it'd be best if it got its own mode, say
> > java-treesitter-mode.el, so that we don't add a dependency into
> > cc-mode?  If you agree, I can create such a major-mode as well.

> I think we can use the same major mode, just skip the cc-mode
> initialization and only enable tree-sitter stuff if
> java-use-tree-sitter is true. But I want to know Alan’s thought on
> this. I included him in CC.

Thanks!  At the moment, I'm still trying to get a handle on what
tree-sitter is and can do.  The idea of including lots of tree-sitter
modes in Emacs 29 came up very suddenly over the weekend.

> > Also, how should we mandate what modes get the new tree-sitter
> > support?  It is pretty fast to add new modes, and adding support for
> > languages not already supported in core, such as say, golang, should
> > be very feasible.  But maybe that is something that should live in
> > elpa?

> If you want to add tree-sitter support for some built-in mode, go for
> it. But I would focus on the modes we already have in core. For modes
> like golang where there is a widely used mode out there (in this case
> on ELPA), we should either ask to move it into core first, or ask the
> author to maybe add tree-sitter support, or submit a patch to him/her,
> rather than creating a duplicate go-mode in core just for the sake of
> tree-sitter.

> Yuan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> 
>> I think we can use the same major mode, just skip the cc-mode
>> initialization and only enable tree-sitter stuff if
>> java-use-tree-sitter is true. But I want to know Alan’s thought on
>> this. I included him in CC.
> 
> Thanks!  At the moment, I'm still trying to get a handle on what
> tree-sitter is and can do.  The idea of including lots of tree-sitter
> modes in Emacs 29 came up very suddenly over the weekend.

I can fill you in :-)

Basically tree-sitter gives you a parser so fast that it can produce a correct syntax tree of the buffer as you edit the buffer. For us developers, having a syntax tree available all the time simplifies many things. 

For example, font-lock: instead of writing regex patterns to find entities to highlight, you now asks the syntax tree: “give me all the function_definition nodes between BEG and END”. And you get a bunch of function_definition nodes. We know function_definition nodes has the following structure:

(function_definition name: (identifier) body: (block_statement …))

So we can just get the “name” child node, which is an identifier, and apply font-lock-function-name-face to the text corresponding to that node.

Basically there is no need to parse or pattern-match ourselves anymore, the answer is readily available.

Another example, for beginning/end-of-defun, there is no need to look for the beginning/end pattern for a function anymore. We just ask the syntax tree to give us the next/previous (say) “function_definition” node after/before POS, and if there is one, jump to the end/beginning of that node.

For indent it’s the same story: you can get the node at the beginning of current line, then by its type (identifier, statement, expression, bracket, etc) and its relative position in the tree, you can decide how to indent it.

Hope that helps,
Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> Date: Mon, 10 Oct 2022 19:44:40 +0000
> Cc: Theodor Thornhill <theo@thornhill.no>, Eli Zaretskii <eliz@gnu.org>,
>  emacs-devel <emacs-devel@gnu.org>
> From: Alan Mackenzie <acm@muc.de>
> 
> > I think we can use the same major mode, just skip the cc-mode
> > initialization and only enable tree-sitter stuff if
> > java-use-tree-sitter is true. But I want to know Alan’s thought on
> > this. I included him in CC.
> 
> Thanks!  At the moment, I'm still trying to get a handle on what
> tree-sitter is and can do.  The idea of including lots of tree-sitter
> modes in Emacs 29 came up very suddenly over the weekend.

Alan, any progress in that direction?  Do you have any thoughts to
share?

TIA

Re: Call for volunteers: add tree-sitter support to major modes

[Alan Mackenzie]

Hello, Eli.

On Mon, Oct 17, 2022 at 20:59:05 +0300, Eli Zaretskii wrote:
> > Date: Mon, 10 Oct 2022 19:44:40 +0000
> > Cc: Theodor Thornhill <theo@thornhill.no>, Eli Zaretskii <eliz@gnu.org>,
> >  emacs-devel <emacs-devel@gnu.org>
> > From: Alan Mackenzie <acm@muc.de>

> > > I think we can use the same major mode, just skip the cc-mode
> > > initialization and only enable tree-sitter stuff if
> > > java-use-tree-sitter is true. But I want to know Alan’s thought on
> > > this. I included him in CC.

> > Thanks!  At the moment, I'm still trying to get a handle on what
> > tree-sitter is and can do.  The idea of including lots of tree-sitter
> > modes in Emacs 29 came up very suddenly over the weekend.

> Alan, any progress in that direction?  Do you have any thoughts to
> share?

Generally, that the replacement of CC Mode's engine with tree-sitter is
a good idea; CC Mode as it currently is is too much work to maintain,
and in the long run is unsustainable.

At the same time, I have no idea how good tree-sitter's grammars for CC
Mode's modes are.  If I understand rightly, a call to TS requires start
and end buffer positions to be calculated by Emacs rather than TS
itself.  This might reduce some of the expected speed gain.

I would like to keep all of CC Mode's user features in the new mode, if
at all possible.  I don't know how easy it will be, for example, to keep
CC Mode's indentation styles working in the new mode.

I can't honestly see myself as able to integrate tree-sitter into CC
Mode in the short time before the planned pre-test of Emacs 29.  I think
we should take up the offer made by João Paulo Labegalini de Carvalho
<jaopaulolc@gmail.com> on 2020-10-11 to work on C Mode and possibly C++
Mode.

I'm afraid I've only read a small part of this thread on emacs-devel, so
there's likely quite a bit of relevant stuff I've missed.

> TIA

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Monnier]

> I would like to keep all of CC Mode's user features in the new mode, if
> at all possible.  I don't know how easy it will be, for example, to keep
> CC Mode's indentation styles working in the new mode.

I suspect this will take a while.  But we're not aiming that high: we
want to offer a "basic" tree-sitter functionality in Emacs-29.1 so users
can start trying it out and using it if they like it (but they can also
just keep using the current CC-mode engine if the tree-sitter code is
not flexible for them yet) and then we can try and integrate the
indentation styles at a later time.


        Stefan

Re: Call for volunteers: add tree-sitter support to major modes

[Ketevan Lomidze]

On Mon, 17 Oct 2022 18:04:23 -0400
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> > I would like to keep all of CC Mode's user features in the new mode, if
> > at all possible.  I don't know how easy it will be, for example, to keep
> > CC Mode's indentation styles working in the new mode.
>
> I suspect this will take a while.  But we're not aiming that high: we
> want to offer a "basic" tree-sitter functionality in Emacs-29.1 so users
> can start trying it out and using it if they like it (but they can also
> just keep using the current CC-mode engine if the tree-sitter code is
> not flexible for them yet) and then we can try and integrate the
> indentation styles at a later time.
>
>
>         Stefan

CC Mode handles too many languages by itself, this is impressive yet it seems
unnecessarily hard.  Wouldn't it be better if each language has its
own mode?

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Alan Mackenzie <acm@muc.de> writes:

> Generally, that the replacement of CC Mode's engine with tree-sitter is
> a good idea; CC Mode as it currently is is too much work to maintain,
> and in the long run is unsustainable.
>
> At the same time, I have no idea how good tree-sitter's grammars for CC
> Mode's modes are.  If I understand rightly, a call to TS requires start
> and end buffer positions to be calculated by Emacs rather than TS
> itself.  This might reduce some of the expected speed gain.
>
> I would like to keep all of CC Mode's user features in the new mode, if
> at all possible.  I don't know how easy it will be, for example, to keep
> CC Mode's indentation styles working in the new mode.
>
> I can't honestly see myself as able to integrate tree-sitter into CC
> Mode in the short time before the planned pre-test of Emacs 29.  I think
> we should take up the offer made by João Paulo Labegalini de Carvalho
> <jaopaulolc@gmail.com> on 2020-10-11 to work on C Mode and possibly C++
> Mode.
>
> I'm afraid I've only read a small part of this thread on emacs-devel, so
> there's likely quite a bit of relevant stuff I've missed.

Could someone please tell me how well tree-sitter supports pre-standard
C with liberal (and sometimes non-standard) use of the C language?  I'm
talking about code that looks like this:

MACRO_USED_TO_DEFINE_SPECIAL_FUNCTIONS (function_name, cells, transform)
     some_kind_of_ptr cells;
     another_kind_of_ptr *transform;
{
  extern maybe_tls (int) errno;
  extern caddr_t bar (_P (another_kind_of_ptr, ...));
  int rc;

  BEGIN_A_KIND_OF_SECTION ({
      ENTRY (dx, dy, shx, shy)
	float dx, dy, shx, shy;

      if (!bar (other_function (dx, dy, shx, shy),
		etc, etc, etc))
	die ("bar", sys_errlist[errno]);
    }, register float, section_name);

  rc = more_code_here (&section_name_desc, etc);
  return rc;
}

I don't doubt that tree-sitter is good at parsing newer languages like
Typescript, but does it support C all that well?

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

[-- Attachment #1: Type: text/plain, Size: 2321 bytes --]



> On Oct 17, 2022, at 8:24 PM, Po Lu <luangruo@yahoo.com> wrote:
> 
> Alan Mackenzie <acm@muc.de> writes:
> 
>> Generally, that the replacement of CC Mode's engine with tree-sitter is
>> a good idea; CC Mode as it currently is is too much work to maintain,
>> and in the long run is unsustainable.
>> 
>> At the same time, I have no idea how good tree-sitter's grammars for CC
>> Mode's modes are.  If I understand rightly, a call to TS requires start
>> and end buffer positions to be calculated by Emacs rather than TS
>> itself.  This might reduce some of the expected speed gain.
>> 
>> I would like to keep all of CC Mode's user features in the new mode, if
>> at all possible.  I don't know how easy it will be, for example, to keep
>> CC Mode's indentation styles working in the new mode.
>> 
>> I can't honestly see myself as able to integrate tree-sitter into CC
>> Mode in the short time before the planned pre-test of Emacs 29.  I think
>> we should take up the offer made by João Paulo Labegalini de Carvalho
>> <jaopaulolc@gmail.com> on 2020-10-11 to work on C Mode and possibly C++
>> Mode.
>> 
>> I'm afraid I've only read a small part of this thread on emacs-devel, so
>> there's likely quite a bit of relevant stuff I've missed.
> 
> Could someone please tell me how well tree-sitter supports pre-standard
> C with liberal (and sometimes non-standard) use of the C language?  I'm
> talking about code that looks like this:
> 
> MACRO_USED_TO_DEFINE_SPECIAL_FUNCTIONS (function_name, cells, transform)
>     some_kind_of_ptr cells;
>     another_kind_of_ptr *transform;
> {
>  extern maybe_tls (int) errno;
>  extern caddr_t bar (_P (another_kind_of_ptr, ...));
>  int rc;
> 
>  BEGIN_A_KIND_OF_SECTION ({
>      ENTRY (dx, dy, shx, shy)
> 	float dx, dy, shx, shy;
> 
>      if (!bar (other_function (dx, dy, shx, shy),
> 		etc, etc, etc))
> 	die ("bar", sys_errlist[errno]);
>    }, register float, section_name);
> 
>  rc = more_code_here (&section_name_desc, etc);
>  return rc;
> }
> 
> I don't doubt that tree-sitter is good at parsing newer languages like
> Typescript, but does it support C all that well?

A quick side-by-side comparison of a primitive tree-sitter font-lock I wrote vs c-mode:

Tree-sitter:



C-mode:


Yuan


[-- Attachment #2.1: Type: text/html, Size: 4084 bytes --]

[-- Attachment #2.2: Screen Shot 2022-10-17 at 9.37.47 PM.png --]
[-- Type: image/png, Size: 124685 bytes --]

[-- Attachment #2.3: Screen Shot 2022-10-17 at 9.38.39 PM.png --]
[-- Type: image/png, Size: 125257 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Yuan Fu <casouri@gmail.com> writes:

> A quick side-by-side comparison of a primitive tree-sitter font-lock I
> wrote vs c-mode:

Thanks.  Sadly, it looks like the tree-sitter fontification fares much
worse than the current CC Mode engine.

What about indentation?

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

On October 18, 2022 9:35:18 AM GMT+03:00, Po Lu <luangruo@yahoo.com> wrote:
> Yuan Fu <casouri@gmail.com> writes:
> 
> > A quick side-by-side comparison of a primitive tree-sitter font-lock I
> > wrote vs c-mode:
> 
> Thanks.  Sadly, it looks like the tree-sitter fontification fares much
> worse than the current CC Mode engine.


It does?  Which part(s) are "much worse" and why?

The code you presented is invalid C, so I cannot see how we could pass any objective judgment about which fontification is "correct".

This kind of "argument" is not useful.

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> It does?  Which part(s) are "much worse" and why?
>
> The code you presented is invalid C, so I cannot see how we could pass
> any objective judgment about which fontification is "correct".

It is not invalid C, at least not for this (proprietary) C compiler.

Anyway, I'll try to explain what the problem is with the fontification.
BEGIN_A_KIND_OF_SECTION starts a closure, ENTRY (dx, dy, shx, shy)
expands to the important part of which is a nested function declaration
(that is then put in a struct, etc, but that is not really relevant
here):

  [use: fcsr, f20, f21, f22 capture: etc]
  __section_name_entry (dx, dy, shx, shy, AP, ...)
      float dx, dy, shx, shy;
      frame_t *AP;
  {
    ...

Notice how tree_sitter fontifies everything between ENTRY and shy as a
type, while dx, dy, shx, shy are actually arguments in the "K&R area".
They are fontified correctly by CC Mode, which only fontifies "float" as
a type.

The fontification of "_P" is also wrong.  It is a macro that expands to
nothing if prototype checking is not enabled in the compiler, and
otherwise literally expands to each one of its arguments.  Yes, the way
the macro is defined is not standard C, but CC Mode has so far fared
quite well:

#   define _P(args...) ##args

register float is the type of argument that is expected to be taken by
the closure.  There can only be several: register float, register
doublefloat, register int, doubleword, long.

section_name is an argument to the macro, and is used to name both
section_name_desc and __section_name_entry, and other things:

  struct [align: 8] proc_jump_struct section_name_desc;
  [use: fcsr, f20, f21, f22 capture: etc]
  __section_name_entry (dx, dy...
  ...
  section_name_desc.entry = (char *) &__section_name_entry;
  ... and more code here ...

so it should not be fontified as a function either.  In tree-sitter,
"cells" (an argument in the K&R area) is also fontified as a function.

> This kind of "argument" is not useful.

Why?

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

[-- Attachment #1: Type: text/plain, Size: 3075 bytes --]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org,  Yuan Fu <casouri@gmail.com>,  Alan Mackenzie
>  <acm@muc.de>,  theo@thornhill.no
> Date: Tue, 18 Oct 2022 18:36:08 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It does?  Which part(s) are "much worse" and why?
> >
> > The code you presented is invalid C, so I cannot see how we could pass
> > any objective judgment about which fontification is "correct".
> 
> It is not invalid C, at least not for this (proprietary) C compiler.

This part definitely isn't valid:

  extern maybe_tls (int) errno;

 $ gcc -c buggy_c.c
 buggy_c.c: In function 'maybe_tls':
 buggy_c.c:1:26: error: expected declaration specifiers before 'errno'
     1 |   extern maybe_tls (int) errno;
       |                          ^~~~~
 buggy_c.c:1:21: error: parameter name omitted
     1 |   extern maybe_tls (int) errno;
       |                     ^~~
 buggy_c.c:2: error: expected '{' at end of input

And this:

  BEGIN_A_KIND_OF_SECTION ({
      ENTRY (dx, dy, shx, shy)
	float dx, dy, shx, shy;

      if (!bar (other_function (dx, dy, shx, shy),
		etc, etc, etc))
	die ("bar", sys_errlist[errno]);
    }, register float, section_name);

can only be valid C if BEGIN_A_KIND_OF_SECTION does something very
creative, like expands to nothing at all.  So, for a fair discussion,
please show the macros used here, and let's see what does tree-sitter
do when it's shown the macros.

> Notice how tree_sitter fontifies everything between ENTRY and shy as a
> type, while dx, dy, shx, shy are actually arguments in the "K&R area".
> They are fontified correctly by CC Mode, which only fontifies "float" as
> a type.

Why is that "much worse" than what CC Mode does?  Just yesterday I had
a perfectly valid C code where CC Mode was unable to fontify several
function calls correctly.  I attach it below.  You can see it
yourself: go to line 2000 in treesit.c of the feature/tree-stter
branch.  Unlike many CC Mode bugs, this one isn't transient: killing
the buffer and revisiting the file reproduces the same problem.  (This
problem seems to be solved in Emacs 29, btw.  But I see its like
almost every day in other situations.)

I'm quite sure CC Mode "succeeds" with your example of invalid C code
simply by sheer luck.  And what's more important to me is that it
fails in much simpler, straightforward cases, where I wouldn't expect
any trouble at all.

> The fontification of "_P" is also wrong.

The code you posted didn't include the definition of P_, so how do you
expect a parser to guess what it is?  Include it in the code, and
let's see then what happens.

>   struct [align: 8] proc_jump_struct section_name_desc;
>   [use: fcsr, f20, f21, f22 capture: etc]
>   __section_name_entry (dx, dy...
>   ...

This is valid C?

And after all that said, I'll trade CC Mode's fontifications for much
more performant C mode any day, even if the faster mode sometimes
errs.  The CC Mode's performance is abysmally slow.

> > This kind of "argument" is not useful.
> 
> Why?

Because it's biased.

Here's the picture I promised:


[-- Attachment #2: bad_fontification.PNG --]
[-- Type: image/png, Size: 22910 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> This part definitely isn't valid:
>
>   extern maybe_tls (int) errno;
>
>  $ gcc -c buggy_c.c
>  buggy_c.c: In function 'maybe_tls':
>  buggy_c.c:1:26: error: expected declaration specifiers before 'errno'
>      1 |   extern maybe_tls (int) errno;
>        |                          ^~~~~
>  buggy_c.c:1:21: error: parameter name omitted
>      1 |   extern maybe_tls (int) errno;
>        |                     ^~~
>  buggy_c.c:2: error: expected '{' at end of input

Well, what if tree-sitter is given the following macro definition?

#    if !defined (LinkTargetKO) && !defined (__cc_lint__)
#    define maybe_tls(type) _task_local_/**/type
#    else
#    define maybe_tls(type) type
#    endif

> And this:
>
>   BEGIN_A_KIND_OF_SECTION ({
>       ENTRY (dx, dy, shx, shy)
> 	float dx, dy, shx, shy;
>
>       if (!bar (other_function (dx, dy, shx, shy),
> 		etc, etc, etc))
> 	die ("bar", sys_errlist[errno]);
>     }, register float, section_name);
>
> can only be valid C if BEGIN_A_KIND_OF_SECTION does something very
> creative, like expands to nothing at all.  So, for a fair discussion,
> please show the macros used here, and let's see what does tree-sitter
> do when it's shown the macros.

This particular macro is a little too long for me to trim down into a
form I'm actually allowed to send here.  But what happens when
tree-sitter is shown the other macro, maybe_tls?

> Why is that "much worse" than what CC Mode does?  Just yesterday I had
> a perfectly valid C code where CC Mode was unable to fontify several
> function calls correctly.  I attach it below.  You can see it
> yourself: go to line 2000 in treesit.c of the feature/tree-stter
> branch.  Unlike many CC Mode bugs, this one isn't transient: killing
> the buffer and revisiting the file reproduces the same problem.  (This
> problem seems to be solved in Emacs 29, btw.  But I see its like
> almost every day in other situations.)
>
> I'm quite sure CC Mode "succeeds" with your example of invalid C code
> simply by sheer luck.  And what's more important to me is that it
> fails in much simpler, straightforward cases, where I wouldn't expect
> any trouble at all.

Trust me, I have my own bad experiences with CC Mode fontification.  See
the slew of bugs I found and reported with Emacs 29's CC Mode just this
week: bug#58537, bug#58534, bug#58539.  My problem is the blanket
dismissal of what CC Mode actually gets right that seems to be
alarmingly prevalent around here ("... in Emacs 30, let's get rid of CC
Mode!")

> The code you posted didn't include the definition of P_, so how do you
> expect a parser to guess what it is?  Include it in the code, and
> let's see then what happens.

#    ifdef (__cc_check_protos__)
#    define _P(protos...) /**/protos
#    else
#    define _P(protos...)
#    endif

> This is valid C?

It is in this compiler (and C preprocessor), which has not seen major
updates since 1994, and is yet still used by suits.

> And after all that said, I'll trade CC Mode's fontifications for much
> more performant C mode any day, even if the faster mode sometimes
> errs.  The CC Mode's performance is abysmally slow.

> Here's the picture I promised:

I agree that's bad, but so is the tree-sitter fontification of K&R
areas.

According to this ticket, tree-sitter currently doesn't support
them at all: https://github.com/tree-sitter/tree-sitter-c/issues/75.  At
least there is work in that area, but it has not seen activity since
Sept 2021.

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Monnier]

> Trust me, I have my own bad experiences with CC Mode fontification.  See
> the slew of bugs I found and reported with Emacs 29's CC Mode just this
> week: bug#58537, bug#58534, bug#58539.  My problem is the blanket
> dismissal of what CC Mode actually gets right that seems to be
> alarmingly prevalent around here ("... in Emacs 30, let's get rid of CC
> Mode!")

Some people seem very optimistic about what tree-sitter will bring, yes.
I think our track record should be enough to reassure users that we
won't just throw away CC-mode if tree-sitter doesn't prove to be
a worthy replacement *and* that it won't be thrown away any time soon
even if tree-sitter is much better.

Also, the fact that Alan still maintains CC-mode for Emacs-21 suggests
that CC-mode will still be maintained well into the 2030s, regardless if
it's still included in Emacs by then.

So, let's all take a deep breath and just wait to see how things play
out before we make any assumption or decision.


        Stefan

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,  casouri@gmail.com,
>   acm@muc.de,  theo@thornhill.no
> Date: Wed, 19 Oct 2022 21:15:00 -0400
> 
> > Trust me, I have my own bad experiences with CC Mode fontification.  See
> > the slew of bugs I found and reported with Emacs 29's CC Mode just this
> > week: bug#58537, bug#58534, bug#58539.  My problem is the blanket
> > dismissal of what CC Mode actually gets right that seems to be
> > alarmingly prevalent around here ("... in Emacs 30, let's get rid of CC
> > Mode!")
> 
> Some people seem very optimistic about what tree-sitter will bring, yes.
> I think our track record should be enough to reassure users that we
> won't just throw away CC-mode if tree-sitter doesn't prove to be
> a worthy replacement *and* that it won't be thrown away any time soon
> even if tree-sitter is much better.

No one seriously proposed getting rid of what CC Mode does now in the
areas of fontification and indentation.  What's currently on the table
is to provide optional behavior based on tree-sitter, and let users
try that and us collect feedback and decide how to develop Emacs in
these areas in the future.  I hope we will be able to start that in
Emacs 29.

> So, let's all take a deep breath and just wait to see how things play
> out before we make any assumption or decision.

No such assumptions or decisions are expected any time soon, so
everyone can stop holding their breath and start breathing again.

Re: Call for volunteers: add tree-sitter support to major modes

[Jostein Kjønigsen]

On 20.10.2022 03:15, Stefan Monnier wrote:
> Some people seem very optimistic about what tree-sitter will bring, yes.

I may be one of those people.

So for the record, I would like to make it absolutely clear that I think 
tree-sitter will bring improvements. Not unconditionally, but certainly 
it will help with some things.

I know from experience that it will improve greatly upon the classic 
(cc-mode based) csharp-mode. typescript-mode is also suffering when 
encountering complex grammar, which tree-sitter seems to solve adequately.

These are languages without the ambiguities of macros, and syntax can be 
parsed reliably based on single documents alone. These are perfect for 
tree-sitter.

> I think our track record should be enough to reassure users that we
> won't just throw away CC-mode if tree-sitter doesn't prove to be
> a worthy replacement *and* that it won't be thrown away any time soon
> even if tree-sitter is much better.

Tree-sitter is just code, like all other code out there. It's not going 
to magically improve everything everywhere. But that should be OK, as 
long as the use of it is constrained to areas where its parsers are 
objectively superior.

Even as one of the "tree-sitter advocates" in this thread, I don't see 
any value in shoehorning it into areas where it objectively doesn't 
provide us with anything. If cryptic, macro-ridden C is something it is 
bad at, then perhaps we shouldn't use it for that.

But let's not dissuade people who want to try it out from seeing if we 
can get better results than Emacs currently provides :)

> So, let's all take a deep breath and just wait to see how things play
> out before we make any assumption or decision.

Definitely this. We're trying out something new, to improve upon or 
support new languages we haven't been able to do perfectly so far.

In my books that's exclusively a good thing.

--
Jostein

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org,  casouri@gmail.com,  acm@muc.de,  theo@thornhill.no
> Date: Thu, 20 Oct 2022 08:19:28 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > This part definitely isn't valid:
> >
> >   extern maybe_tls (int) errno;
> >
> >  $ gcc -c buggy_c.c
> >  buggy_c.c: In function 'maybe_tls':
> >  buggy_c.c:1:26: error: expected declaration specifiers before 'errno'
> >      1 |   extern maybe_tls (int) errno;
> >        |                          ^~~~~
> >  buggy_c.c:1:21: error: parameter name omitted
> >      1 |   extern maybe_tls (int) errno;
> >        |                     ^~~
> >  buggy_c.c:2: error: expected '{' at end of input
> 
> Well, what if tree-sitter is given the following macro definition?
> 
> #    if !defined (LinkTargetKO) && !defined (__cc_lint__)
> #    define maybe_tls(type) _task_local_/**/type
> #    else
> #    define maybe_tls(type) type
> #    endif

If you use macros to selectively remove parts of the source at
critical places, you can completely change how the source code is seen
by the parser.

I don't know how tree-sitter processes macro definitions, but maybe
someone else will.

> Trust me, I have my own bad experiences with CC Mode fontification.  See
> the slew of bugs I found and reported with Emacs 29's CC Mode just this
> week: bug#58537, bug#58534, bug#58539.  My problem is the blanket
> dismissal of what CC Mode actually gets right that seems to be
> alarmingly prevalent around here ("... in Emacs 30, let's get rid of CC
> Mode!")

No one seriously intends to throw away what CC Mode does, let alone
dismiss it in a blanket manner.  This is a strawman.  This discussion
has a lot of noise, but you should be experienced enough to know whose
voices to hear and whose to dismiss as merely a personal opinion that
has no bearing on what will actually happen in Emacs.

> I agree that's bad, but so is the tree-sitter fontification of K&R
> areas.

We need to get our priorities right.

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Monnier]

> I don't doubt that tree-sitter is good at parsing newer languages like
> Typescript, but does it support C all that well?

I don't think we should care very much at this point: it will only
become relevant when or if we want to consider:
- making tree-sitter the default parser for C files.
- removing the CC-mode code.

The second is definitely not planned for the foreseeable future, AFAIK.
As for the first, I can see reasons why we might want to consider it in
the medium term, but definitely not for Emacs-29.


        Stefan

Re: Call for volunteers: add tree-sitter support to major modes

[Jostein Kjønigsen]

>
> Could someone please tell me how well tree-sitter supports pre-standard
> C with liberal (and sometimes non-standard) use of the C language?  I'm
> talking about code that looks like this:
>
> MACRO_USED_TO_DEFINE_SPECIAL_FUNCTIONS (function_name, cells, transform)
>       some_kind_of_ptr cells;
>       another_kind_of_ptr *transform;
> {
>    extern maybe_tls (int) errno;
>    extern caddr_t bar (_P (another_kind_of_ptr, ...));
>    int rc;
>
>    BEGIN_A_KIND_OF_SECTION ({
>        ENTRY (dx, dy, shx, shy)
> 	float dx, dy, shx, shy;
>
>        if (!bar (other_function (dx, dy, shx, shy),
> 		etc, etc, etc))
> 	die ("bar", sys_errlist[errno]);
>      }, register float, section_name);
>
>    rc = more_code_here (&section_name_desc, etc);
>    return rc;
> }
>
> I don't doubt that tree-sitter is good at parsing newer languages like
> Typescript, but does it support C all that well?
>
If you want to get an idea of the syntaxes which are supported by 
tree-sitter, all language grammar-specifications have test-cases 
associated with them.

Some of the test-cases for "vanilla" C can be found here:

https://github.com/tree-sitter/tree-sitter-c/tree/master/test/corpus

You may find the amibiguities file in particular interesting: 
https://github.com/tree-sitter/tree-sitter-c/blob/master/test/corpus/ambiguities.txt

For C++ there is a separate grammar all together:

https://github.com/tree-sitter/tree-sitter-cpp/tree/master/test/corpus

It similarly has defined test-cases for how ambiguous statements should 
be parsed:

https://github.com/tree-sitter/tree-sitter-cpp/blob/master/test/corpus/ambiguities.txt

--

Jostein

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Date: Sun, 09 Oct 2022 22:01:07 +0200
> 
> I'd like to add in Java support here as well.

Sure, that'd be very good, TIA.

> But since java is part of cc-mode.el, maybe it'd be best if it got
> its own mode, say java-treesitter-mode.el, so that we don't add a
> dependency into cc-mode?  If you agree, I can create such a
> major-mode as well.

It depends on practical considerations.  If cc-mode's Java part has
significant features beyond those we intend to have supported by
tree-sitter, then a minor variation of cc-mode's java-mode is probably
the best; it could be via some minor mode or simply an optional
feature turned on by a defcustom.

OTOH, if Java support in cc-mode is more or less the same features as
we intend to provide with tree-sitter, a separate mode is probably the
best.

Alan, any comments and/or suggestions?

> Also, how should we mandate what modes get the new tree-sitter support?

Ideally, all of them.  The list I presented was just the minimal
program, considering the short time until Emacs 29 starts its release
cycle.  But there's no upper bound on the modes which should get
tree-sitter support.

> It is pretty fast to add new modes, and adding support for languages not
> already supported in core, such as say, golang, should be very feasible.
> But maybe that is something that should live in elpa?

I don't think programming-language modes should be outside of core, as
a matter of principle.  So, unless there are some specific reasons not
to include a mode (like lack of copyright assignments or the
author/maintainer objections), such modes should be in core.

There's also an issue of having popular modes for those languages as
3rd-party packages out there: it is best to try to bring them into
core and then add tree-sitter to them (unless they already support
it), than to come up with a completely separate implementation -- this
will allow their today's users to keep using the same more, instead of
having to decide whether to switch.

But otherwise, if it's easy to add support for more languages using
tree-sitter, let's do that, yes.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Hi Eli,

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Date: Sun, 09 Oct 2022 22:01:07 +0200
>> 
>> I'd like to add in Java support here as well.
>
> Sure, that'd be very good, TIA.
>

My pleasure.

>> But since java is part of cc-mode.el, maybe it'd be best if it got
>> its own mode, say java-treesitter-mode.el, so that we don't add a
>> dependency into cc-mode?  If you agree, I can create such a
>> major-mode as well.
>
> It depends on practical considerations.  If cc-mode's Java part has
> significant features beyond those we intend to have supported by
> tree-sitter, then a minor variation of cc-mode's java-mode is probably
> the best; it could be via some minor mode or simply an optional
> feature turned on by a defcustom.
>
> OTOH, if Java support in cc-mode is more or less the same features as
> we intend to provide with tree-sitter, a separate mode is probably the
> best.
>
> Alan, any comments and/or suggestions?
>

Yeah, that makes sense - I'll let Alan chime in.

>> Also, how should we mandate what modes get the new tree-sitter support?
>
> Ideally, all of them.  The list I presented was just the minimal
> program, considering the short time until Emacs 29 starts its release
> cycle.  But there's no upper bound on the modes which should get
> tree-sitter support.
>
>> It is pretty fast to add new modes, and adding support for languages not
>> already supported in core, such as say, golang, should be very feasible.
>> But maybe that is something that should live in elpa?
>
> I don't think programming-language modes should be outside of core, as
> a matter of principle.  So, unless there are some specific reasons not
> to include a mode (like lack of copyright assignments or the
> author/maintainer objections), such modes should be in core.
>
> There's also an issue of having popular modes for those languages as
> 3rd-party packages out there: it is best to try to bring them into
> core and then add tree-sitter to them (unless they already support
> it), than to come up with a completely separate implementation -- this
> will allow their today's users to keep using the same more, instead of
> having to decide whether to switch.
>
> But otherwise, if it's easy to add support for more languages using
> tree-sitter, let's do that, yes.

Agreed.  Well, I'm a maintainer and contributor to two fairly popular
modes, 'csharp-mode' and 'typescript-mode'.  The former is in ELPA, the
latter in nongnu ELPA.  For the former, I'm not sure if bringing the
cc-mode variant into core is a good plan, for several reasons.

- It has some perf issues with multiline strings
- There's some problems with indentation and other very c# related stuff
- It _will_ cause more maintainer burden onto Alan, because of cc-mode
  lineage.


In this case it makes sense for me to add only the tree-sitter variant,
found here [0].


For 'typescript-mode', it isn't feasible to enter the most use variant
into core because of copyright issues, but more importantly, it has some
very long-standing unfixed issues that are impossible to solve.

- tsx cannot be supported reliably by js-mode derivation because it uses
  same syntax for types and tsx ( <, >).  This is handled just fine by
  tree-sitter.  Same for template string literals.

- The tree-sitter variant is made by me and Jostein (CC'd) [1], so there are
  no issues there with assignment.


My suggestion is to add the tree-sitter variant in these cases, and let
the other modes die a slow, deprecated death down the line.

What do you think?

Theo

[0]: https://git.sr.ht/~theo/tree-sitter-modes/tree/master/item/csharp-mode.el
[1]: https://git.sr.ht/~theo/tree-sitter-modes/tree/master/item/typescript-mode.el 

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: emacs-devel@gnu.org, jostein@kjonigsen.net
> Date: Mon, 10 Oct 2022 08:44:56 +0200
> 
> Well, I'm a maintainer and contributor to two fairly popular
> modes, 'csharp-mode' and 'typescript-mode'.  The former is in ELPA, the
> latter in nongnu ELPA.  For the former, I'm not sure if bringing the
> cc-mode variant into core is a good plan, for several reasons.
> 
> - It has some perf issues with multiline strings
> - There's some problems with indentation and other very c# related stuff
> - It _will_ cause more maintainer burden onto Alan, because of cc-mode
>   lineage.
> 
> 
> In this case it makes sense for me to add only the tree-sitter variant,
> found here [0].
> 
> 
> For 'typescript-mode', it isn't feasible to enter the most use variant
> into core because of copyright issues, but more importantly, it has some
> very long-standing unfixed issues that are impossible to solve.
> 
> - tsx cannot be supported reliably by js-mode derivation because it uses
>   same syntax for types and tsx ( <, >).  This is handled just fine by
>   tree-sitter.  Same for template string literals.
> 
> - The tree-sitter variant is made by me and Jostein (CC'd) [1], so there are
>   no issues there with assignment.
> 
> 
> My suggestion is to add the tree-sitter variant in these cases, and let
> the other modes die a slow, deprecated death down the line.
> 
> What do you think?

SGTM, but I'd like to hear from Lars as well.

Another possible concern is: what will users of the tree-sitter
variants we add miss wrt the existing variants?  IOW, what useful
features will be missing from the support of those languages by these
new modes?

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

[...]

>> For 'typescript-mode', it isn't feasible to enter the most use variant
>> into core because of copyright issues, but more importantly, it has some
>> very long-standing unfixed issues that are impossible to solve.
>> 
>> - tsx cannot be supported reliably by js-mode derivation because it uses
>>   same syntax for types and tsx ( <, >).  This is handled just fine by
>>   tree-sitter.  Same for template string literals.
>> 
>> - The tree-sitter variant is made by me and Jostein (CC'd) [1], so there are
>>   no issues there with assignment.
>> 
>> 
>> My suggestion is to add the tree-sitter variant in these cases, and let
>> the other modes die a slow, deprecated death down the line.
>> 
>> What do you think?
>
> SGTM, but I'd like to hear from Lars as well.
>
> Another possible concern is: what will users of the tree-sitter
> variants we add miss wrt the existing variants?  IOW, what useful
> features will be missing from the support of those languages by these
> new modes?

In some cases, such as the two I mentioned, nothing.  With others, such
as go-mode, something.  This could be some integration with package
managers, documentation lookup functions, etc.  In most cases, but not
all this is solved by LSP.  In the cases where there are benefits in the
original mode that needs to be handled case by case.

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: acm@muc.de, emacs-devel@gnu.org, jostein@kjonigsen.net
> Date: Mon, 10 Oct 2022 11:04:21 +0200
> 
> > Another possible concern is: what will users of the tree-sitter
> > variants we add miss wrt the existing variants?  IOW, what useful
> > features will be missing from the support of those languages by these
> > new modes?
> 
> In some cases, such as the two I mentioned, nothing.  With others, such
> as go-mode, something.  This could be some integration with package
> managers, documentation lookup functions, etc.  In most cases, but not
> all this is solved by LSP.  In the cases where there are benefits in the
> original mode that needs to be handled case by case.

If the missing features are minor or non-existent, I see no problem at
all.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: acm@muc.de, emacs-devel@gnu.org, jostein@kjonigsen.net
>> Date: Mon, 10 Oct 2022 11:04:21 +0200
>> 
>> > Another possible concern is: what will users of the tree-sitter
>> > variants we add miss wrt the existing variants?  IOW, what useful
>> > features will be missing from the support of those languages by these
>> > new modes?
>> 
>> In some cases, such as the two I mentioned, nothing.  With others, such
>> as go-mode, something.  This could be some integration with package
>> managers, documentation lookup functions, etc.  In most cases, but not
>> all this is solved by LSP.  In the cases where there are benefits in the
>> original mode that needs to be handled case by case.
>
> If the missing features are minor or non-existent, I see no problem at
> all.

I'll start adding the others too then.  In this thread or as separate
bugs?

Or I could commit them myself if allowed - you decide.

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: acm@muc.de, emacs-devel@gnu.org, jostein@kjonigsen.net
> Date: Mon, 10 Oct 2022 11:20:36 +0200
> 
> I'll start adding the others too then.  In this thread or as separate
> bugs?

Separate bugs, please.

> Or I could commit them myself if allowed - you decide.

I'd prefer to give Yuan Fu an opportunity to review the patches before
you install them on the branch.

Thanks.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: acm@muc.de, emacs-devel@gnu.org, jostein@kjonigsen.net
>> Date: Mon, 10 Oct 2022 11:20:36 +0200
>> 
>> I'll start adding the others too then.  In this thread or as separate
>> bugs?
>
> Separate bugs, please.
>
>> Or I could commit them myself if allowed - you decide.
>
> I'd prefer to give Yuan Fu an opportunity to review the patches before
> you install them on the branch.
>
> Thanks.

Sure - that would be my suggestion anyways.  Just wanted to offload the
pressure a little bit.

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Monnier]

> In some cases, such as the two I mentioned, nothing.  With others, such
> as go-mode, something.  This could be some integration with package
> managers, documentation lookup functions, etc.  In most cases, but not
> all this is solved by LSP.  In the cases where there are benefits in the
> original mode that needs to be handled case by case.

"Solve by LSP" will only solve it for some users.  To best server the
users, we still need to actively make efforts to integrate the old and
the new modes, such as by making the old mode into a derived mode of the
new mode, and then move missing features from the old mode to the
new one.

Experience tells me that if there's a builtin mode and a third-party
mode, their natural course is to diverging.  That results in more work,
worse modes, and a poor user experience (silly choice between two
similar contenders, confusing info on the web, ...).

It's very hard to integrate two different modes after the fact, so we
should make sure the new mode is integrated with the old one from the
start (usually by making the old mode a derived mode of the new one),
and work together with the old mode's maintainer to keep them in-sync
and reduce the differences over time.


        Stefan

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

On 11 October 2022 23:38:18 CEST, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> In some cases, such as the two I mentioned, nothing.  With others, such
>> as go-mode, something.  This could be some integration with package
>> managers, documentation lookup functions, etc.  In most cases, but not
>> all this is solved by LSP.  In the cases where there are benefits in the
>> original mode that needs to be handled case by case.
>
>"Solve by LSP" will only solve it for some users.  To best server the
>users, we still need to actively make efforts to integrate the old and
>the new modes, such as by making the old mode into a derived mode of the
>new mode, and then move missing features from the old mode to the
>new one.
>
>Experience tells me that if there's a builtin mode and a third-party
>mode, their natural course is to diverging.  That results in more work,
>worse modes, and a poor user experience (silly choice between two
>similar contenders, confusing info on the web, ...).
>
>It's very hard to integrate two different modes after the fact, so we
>should make sure the new mode is integrated with the old one from the
>start (usually by making the old mode a derived mode of the new one),
>and work together with the old mode's maintainer to keep them in-sync
>and reduce the differences over time.
>

I agree. The point is, though, for c# and ts specifically, they have huge outstanding issues already rendering them obsolete. Other modes are in a way better state, and where it matters we should absolutely do as you suggest. 

I'm holding off c# for now because that one is cc mode based, whereas ts can leverage js-mode. 

Theo 

Re: Call for volunteers: add tree-sitter support to major modes

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> My suggestion is to add the tree-sitter variant in these cases, and let
>> the other modes die a slow, deprecated death down the line.
>> 
>> What do you think?
>
> SGTM, but I'd like to hear from Lars as well.

It's somewhat confusing that we have some modes that are tree-sitter-only
and some what can switch between using tree-sitter and not, but I guess
that's inevitable.

But I think the in-tree tree-sitter typescript-mode will have to be
called something else than the out-of-tree non-tree-sitter one, at
least.

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Theodor Thornhill <theo@thornhill.no>,  acm@muc.de,
>   emacs-devel@gnu.org,  jostein@kjonigsen.net
> Date: Tue, 11 Oct 2022 02:34:04 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> My suggestion is to add the tree-sitter variant in these cases, and let
> >> the other modes die a slow, deprecated death down the line.
> >> 
> >> What do you think?
> >
> > SGTM, but I'd like to hear from Lars as well.
> 
> It's somewhat confusing that we have some modes that are tree-sitter-only
> and some what can switch between using tree-sitter and not, but I guess
> that's inevitable.

We could arrange for a very minimal font-lock without tree-sitter
(like, for example, only strings and comments?), and use the defaults
for the indentation commands.  Theodor, can that be done with a
relatively small effort?

> But I think the in-tree tree-sitter typescript-mode will have to be
> called something else than the out-of-tree non-tree-sitter one, at
> least.

That's desirable, yes.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

On 11 October 2022 08:30:29 CEST, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: Theodor Thornhill <theo@thornhill.no>,  acm@muc.de,
>>   emacs-devel@gnu.org,  jostein@kjonigsen.net
>> Date: Tue, 11 Oct 2022 02:34:04 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> My suggestion is to add the tree-sitter variant in these cases, and let
>> >> the other modes die a slow, deprecated death down the line.
>> >> 
>> >> What do you think?
>> >
>> > SGTM, but I'd like to hear from Lars as well.
>> 
>> It's somewhat confusing that we have some modes that are tree-sitter-only
>> and some what can switch between using tree-sitter and not, but I guess
>> that's inevitable.
>
>We could arrange for a very minimal font-lock without tree-sitter
>(like, for example, only strings and comments?), and use the defaults
>for the indentation commands.  Theodor, can that be done with a
>relatively small effort?

Yes, we could actually just delegate that work to vanilla js-mode, as Typescript is just a superset of Javascript. That would mean we would get the benefits of that lineage, but missing some more advanced highlights etc. 

What do you think? 

>
>> But I think the in-tree tree-sitter typescript-mode will have to be
>> called something else than the out-of-tree non-tree-sitter one, at
>> least.
>
>That's desirable, yes.

Sure, I can rename it to tsx-mode, because that's the parser being used. Why is that desirable, though? 

Theodor

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> Date: Tue, 11 Oct 2022 08:41:58 +0200
> From: Theodor Thornhill <theo@thornhill.no>
> CC: acm@muc.de, emacs-devel@gnu.org, jostein@kjonigsen.net
> 
> >> It's somewhat confusing that we have some modes that are tree-sitter-only
> >> and some what can switch between using tree-sitter and not, but I guess
> >> that's inevitable.
> >
> >We could arrange for a very minimal font-lock without tree-sitter
> >(like, for example, only strings and comments?), and use the defaults
> >for the indentation commands.  Theodor, can that be done with a
> >relatively small effort?
> 
> Yes, we could actually just delegate that work to vanilla js-mode, as Typescript is just a superset of Javascript. That would mean we would get the benefits of that lineage, but missing some more advanced highlights etc. 
> 
> What do you think? 

Sounds good, but I don't use these modes, so take my opinions with a
grain of salt.

> >> But I think the in-tree tree-sitter typescript-mode will have to be
> >> called something else than the out-of-tree non-tree-sitter one, at
> >> least.
> >
> >That's desirable, yes.
> 
> Sure, I can rename it to tsx-mode, because that's the parser being used.

SGTM.

> Why is that desirable, though? 

It prevents user confusion and conflicts, I guess?

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

>
> Sounds good, but I don't use these modes, so take my opinions with a
> grain of salt.
>
>> >> But I think the in-tree tree-sitter typescript-mode will have to be
>> >> called something else than the out-of-tree non-tree-sitter one, at
>> >> least.
>> >
>> >That's desirable, yes.
>> 
>> Sure, I can rename it to tsx-mode, because that's the parser being used.
>
> SGTM.
>

Good, I'll integrate js.el into typescript-mode.el

>> Why is that desirable, though? 
>
> It prevents user confusion and conflicts, I guess?

In many cases, yes, but in the Typescript case most users will be
advised to move on from that mode.  I don't really think there should be
an issue - that repo can be archived when we have reasonable support.
Jostein and/or others should of course chime in on this.

How about we let it live as typescript-mode.el for now, and I can
absolutely make sure the renaming happens should we decide to do so.

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Theodor Thornhill <theo@thornhill.no>
> Cc: larsi@gnus.org, acm@muc.de, emacs-devel@gnu.org, jostein@kjonigsen.net
> Date: Tue, 11 Oct 2022 09:23:41 +0200
> 
> >> Sure, I can rename it to tsx-mode, because that's the parser being used.
> >
> > SGTM.
> >
> 
> Good, I'll integrate js.el into typescript-mode.el
> 
> >> Why is that desirable, though? 
> >
> > It prevents user confusion and conflicts, I guess?
> 
> In many cases, yes, but in the Typescript case most users will be
> advised to move on from that mode.  I don't really think there should be
> an issue - that repo can be archived when we have reasonable support.
> Jostein and/or others should of course chime in on this.
> 
> How about we let it live as typescript-mode.el for now, and I can
> absolutely make sure the renaming happens should we decide to do so.

My personal preference is to avoid conflicts even if we consider them
unlikely and/or desirable.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: larsi@gnus.org, acm@muc.de, emacs-devel@gnu.org, jostein@kjonigsen.net
>> Date: Tue, 11 Oct 2022 09:23:41 +0200
>> 
>> >> Sure, I can rename it to tsx-mode, because that's the parser being used.
>> >
>> > SGTM.
>> >
>> 
>> Good, I'll integrate js.el into typescript-mode.el
>> 
>> >> Why is that desirable, though? 
>> >
>> > It prevents user confusion and conflicts, I guess?
>> 
>> In many cases, yes, but in the Typescript case most users will be
>> advised to move on from that mode.  I don't really think there should be
>> an issue - that repo can be archived when we have reasonable support.
>> Jostein and/or others should of course chime in on this.
>> 
>> How about we let it live as typescript-mode.el for now, and I can
>> absolutely make sure the renaming happens should we decide to do so.
>
> My personal preference is to avoid conflicts even if we consider them
> unlikely and/or desirable.


OK, I'll think of some name.  Defaulting to js works just fine, btw.  So
now we have vanilla support also in the new typescript mode.  In this
case we avoided the tree-sitter requirement, which is nice!

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Jostein Kjønigsen]

On 11 Oct 2022, at 09:46, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> 
>> 
>> From: Theodor Thornhill <theo@thornhill.no>
>> Cc: larsi@gnus.org, acm@muc.de, emacs-devel@gnu.org, jostein@kjonigsen.net
>> Date: Tue, 11 Oct 2022 09:23:41 +0200
>> 
>>>> Sure, I can rename it to tsx-mode, because that's the parser being used.
>>> 
>>> SGTM.
>>> 
>> 
>> Good, I'll integrate js.el into typescript-mode.el
>> 
>>>> Why is that desirable, though? 
>>> 
>>> It prevents user confusion and conflicts, I guess?
>> 
>> In many cases, yes, but in the Typescript case most users will be
>> advised to move on from that mode.  I don't really think there should be
>> an issue - that repo can be archived when we have reasonable support.
>> Jostein and/or others should of course chime in on this.
>> 
>> How about we let it live as typescript-mode.el for now, and I can
>> absolutely make sure the renaming happens should we decide to do so.
> 
> My personal preference is to avoid conflicts even if we consider them
> unlikely and/or desirable.

Just to be clear: I’m one of the few maintainers of the utterly inadequate typescript-mode found on MELPA (and so is Theodor).

Development of the Elisp-based parser has already stalled and there’s really not much development going on there at all. Nor has there been for quite a time. 

I’m fairly certain iI speak on behalf of most of us when I say that once the tree-sitter based typescript-mode is available via Emacs, we will halt development entirely, and just add the required code to avoid colliding with an existing Emacs-implementation. 

There will be -no collusion- by introducing the typescript-mode.el patch as typescript-mode, because this will effectively become the new living implementation. 

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Kangas]

Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:

> I’m fairly certain iI speak on behalf of most of us when I say that
> once the tree-sitter based typescript-mode is available via Emacs, we
> will halt development entirely, and just add the required code to
> avoid colliding with an existing Emacs-implementation.

If that's the plan (sounds good to me, FWIW), it's probably a good idea
to add the necessary workarounds to the external mode ASAP, so that
users will have time to test it and report any issues well ahead of the
release of Emacs 29.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Stefan Kangas <stefankangas@gmail.com> writes:

> Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:
>
>> I’m fairly certain iI speak on behalf of most of us when I say that
>> once the tree-sitter based typescript-mode is available via Emacs, we
>> will halt development entirely, and just add the required code to
>> avoid colliding with an existing Emacs-implementation.
>
> If that's the plan (sounds good to me, FWIW), it's probably a good idea
> to add the necessary workarounds to the external mode ASAP, so that
> users will have time to test it and report any issues well ahead of the
> release of Emacs 29.


I agree with you - issue created [0]

[0]: https://github.com/emacs-typescript/typescript.el/issues/178

Re: Call for volunteers: add tree-sitter support to major modes

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 2577 bytes --]

If someone has typescript-mode installed from MELPA and upgrade to 
Emacs-29... Which typescript-mode are they going to be running?

Emacs does not support namespaces, so there would be no way to separate 
those two. And if the MELPA version runs, it would shadow the Emacs29 
implementation.

Since I believe in keeping things simple for users, I think another 
option we could consider would be to simply make typescript-mode (the 
plain elisp version hosted on MELPA) simply not register itself if it 
finds that the tree-sitter version is already available from Emacs. And 
then we could stop updating it, and people would eventually stop 
installing it.

Speaking as one of the MELPA typescript-mode co-maintainers, I have 
absolutely zero plans for maintaining the old elisp-based version once a 
better, tree-sitter version has been mainlined into Emacs itself.

--

Jostein

On 11.10.2022 08:41, Theodor Thornhill wrote:
>
> On 11 October 2022 08:30:29 CEST, Eli Zaretskii<eliz@gnu.org>  wrote:
>>> From: Lars Ingebrigtsen<larsi@gnus.org>
>>> Cc: Theodor Thornhill<theo@thornhill.no>,acm@muc.de,
>>>    emacs-devel@gnu.org,jostein@kjonigsen.net
>>> Date: Tue, 11 Oct 2022 02:34:04 +0200
>>>
>>> Eli Zaretskii<eliz@gnu.org>  writes:
>>>
>>>>> My suggestion is to add the tree-sitter variant in these cases, and let
>>>>> the other modes die a slow, deprecated death down the line.
>>>>>
>>>>> What do you think?
>>>> SGTM, but I'd like to hear from Lars as well.
>>> It's somewhat confusing that we have some modes that are tree-sitter-only
>>> and some what can switch between using tree-sitter and not, but I guess
>>> that's inevitable.
>> We could arrange for a very minimal font-lock without tree-sitter
>> (like, for example, only strings and comments?), and use the defaults
>> for the indentation commands.  Theodor, can that be done with a
>> relatively small effort?
> Yes, we could actually just delegate that work to vanilla js-mode, as Typescript is just a superset of Javascript. That would mean we would get the benefits of that lineage, but missing some more advanced highlights etc.
>
> What do you think?
>
>>> But I think the in-tree tree-sitter typescript-mode will have to be
>>> called something else than the out-of-tree non-tree-sitter one, at
>>> least.
>> That's desirable, yes.
> Sure, I can rename it to tsx-mode, because that's the parser being used. Why is that desirable, though?
>
> Theodor
-- 
Vennlig hilsen
*Jostein Kjønigsen*

jostein@kjonigsen.net 🍵 jostein@gmail.com
https://jostein.kjønigsen.no <https://jostein.kjønigsen.no>

[-- Attachment #2: Type: text/html, Size: 4596 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:

> If someone has typescript-mode installed from MELPA and upgrade to 
> Emacs-29... Which typescript-mode are they going to be running?
>
> Emacs does not support namespaces, so there would be no way to separate 
> those two. And if the MELPA version runs, it would shadow the Emacs29 
> implementation.
>
> Since I believe in keeping things simple for users, I think another 
> option we could consider would be to simply make typescript-mode (the 
> plain elisp version hosted on MELPA) simply not register itself if it 
> finds that the tree-sitter version is already available from Emacs. And 
> then we could stop updating it, and people would eventually stop 
> installing it.
>
> Speaking as one of the MELPA typescript-mode co-maintainers, I have 
> absolutely zero plans for maintaining the old elisp-based version once a 
> better, tree-sitter version has been mainlined into Emacs itself.
>
> --
>

Yeah, agreed.  Though the new in-tree one could be renamed to tsx-mode.
That will support both typescript and tsx, anyways.  Or it could be made
to not work on 29.1, hehe.  I don't really have a strong opinion, but
seeing the state of the current typescript-mode, I do think there's a
case here for actually superseding it.  But that's not really my call to
be made. 

_I_ think the best name here is typescript-mode, but it could be
tsx-mode, typescript-tree-sitter-mode or anything else.

How about deprecating typescript.el the moment the 29.1 branch is cut?
After we can but some sort of structure in place for the two to coexist?

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Lars Ingebrigtsen]

Theodor Thornhill <theo@thornhill.no> writes:

> How about deprecating typescript.el the moment the 29.1 branch is cut?
> After we can but some sort of structure in place for the two to coexist?

It doesn't work that way -- people will have the old typescript-mode in
their ~/.emacs.d/elpa/ files for years still to come, and things will
just be messy.

So I think the way forward here would be to just plonk the new code into
the package, but choose totally different code paths depending on
whether we're in Emacs 29 (and have tree-sitter) or not.  Then you can
start removing the non-tree-sitter code from the file afterwards.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

On 11 October 2022 20:31:06 CEST, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>Theodor Thornhill <theo@thornhill.no> writes:
>
>> How about deprecating typescript.el the moment the 29.1 branch is cut?
>> After we can but some sort of structure in place for the two to coexist?
>
>It doesn't work that way -- people will have the old typescript-mode in
>their ~/.emacs.d/elpa/ files for years still to come, and things will
>just be messy.
>
>So I think the way forward here would be to just plonk the new code into
>the package, but choose totally different code paths depending on
>whether we're in Emacs 29 (and have tree-sitter) or not.  Then you can
>start removing the non-tree-sitter code from the file afterwards.

I see what you mean, but seeing how that mode is pretty broken, and we have a real chance to fix this now it'll be a missed opportunity...

But I don't see why we should expect emacs not to change, but that may just be me. It would be in everyone's interest having proper Typescript support in core, as that is almost the de facto language of the frontend world.

Well, I guess we'll have to skip Typescript, then.

Re: Call for volunteers: add tree-sitter support to major modes

[Lars Ingebrigtsen]

Theodor Thornhill <theo@thornhill.no> writes:

> I see what you mean, but seeing how that mode is pretty broken, and we
> have a real chance to fix this now it'll be a missed opportunity...
>
> But I don't see why we should expect emacs not to change, but that may
> just be me. It would be in everyone's interest having proper
> Typescript support in core, as that is almost the de facto language of
> the frontend world.
>
> Well, I guess we'll have to skip Typescript, then. 

I don't understand what you mean at all here -- I have not suggested
skipping Typescript or not having support for it in-core Emacs 29.  I've
just proposed a practical plan to arrive at that without hosing users.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

On 11 October 2022 20:54:13 CEST, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>Theodor Thornhill <theo@thornhill.no> writes:
>
>> I see what you mean, but seeing how that mode is pretty broken, and we
>> have a real chance to fix this now it'll be a missed opportunity...
>>
>> But I don't see why we should expect emacs not to change, but that may
>> just be me. It would be in everyone's interest having proper
>> Typescript support in core, as that is almost the de facto language of
>> the frontend world.
>>
>> Well, I guess we'll have to skip Typescript, then. 
>
>I don't understand what you mean at all here -- I have not suggested
>skipping Typescript or not having support for it in-core Emacs 29.  I've
>just proposed a practical plan to arrive at that without hosing users.
>

Then I must have missed your plan - let me reread :) 

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

On 11 October 2022 20:54:13 CEST, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>Theodor Thornhill <theo@thornhill.no> writes:
>
>> I see what you mean, but seeing how that mode is pretty broken, and we
>> have a real chance to fix this now it'll be a missed opportunity...
>>
>> But I don't see why we should expect emacs not to change, but that may
>> just be me. It would be in everyone's interest having proper
>> Typescript support in core, as that is almost the de facto language of
>> the frontend world.
>>
>> Well, I guess we'll have to skip Typescript, then. 
>
>I don't understand what you mean at all here -- I have not suggested
>skipping Typescript or not having support for it in-core Emacs 29.  I've
>just proposed a practical plan to arrive at that without hosing users.
>

You mean duplicating the in-tree code into the package? 

Re: Call for volunteers: add tree-sitter support to major modes

[Lars Ingebrigtsen]

Theodor Thornhill <theo@thornhill.no> writes:

> You mean duplicating the in-tree code into the package? 

Yup.  Or rather, just plonk it all into the package, and put the package
into Emacs as an in-tree package.

Hm...  Oh!  I missed that this was a NONGNU package -- then that isn't
possible.

OK, back to the drawing board.  Anybody have any ideas?

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Theodor Thornhill <theo@thornhill.no> writes:
>
>> You mean duplicating the in-tree code into the package? 
>
> Yup.  Or rather, just plonk it all into the package, and put the package
> into Emacs as an in-tree package.
>
> Hm...  Oh!  I missed that this was a NONGNU package -- then that isn't
> possible.
>
> OK, back to the drawing board.  Anybody have any ideas?

I guess just going with ts-mode will be fine :-)

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Lars Ingebrigtsen]

Theodor Thornhill <theo@thornhill.no> writes:

> I guess just going with ts-mode will be fine :-)

Yeah, I guess that's fine.  (It's slightly confusing to add both
"tree-sitter" and "ts" at the same time, but eh.)

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Theodor Thornhill <theo@thornhill.no> writes:
>
>> I guess just going with ts-mode will be fine :-)
>
> Yeah, I guess that's fine.  (It's slightly confusing to add both
> "tree-sitter" and "ts" at the same time, but eh.)

Hehe, yeah.  Names...

In any case - TS is widely used for Typescript, so it may not be _as_
confusing.

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Monnier]

> OK, back to the drawing board.  Anybody have any ideas?

(add-to-list 'major-mode-remap-alist '(typescript-mode . ts-mode)) ?


        Stefan

Re: Call for volunteers: add tree-sitter support to major modes

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> OK, back to the drawing board.  Anybody have any ideas?
>
> (add-to-list 'major-mode-remap-alist '(typescript-mode . ts-mode)) ?

Yeah, that'd work.

Re: Call for volunteers: add tree-sitter support to major modes

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Theodor Thornhill <theo@thornhill.no> writes:
>
>> How about deprecating typescript.el the moment the 29.1 branch is cut?
>> After we can but some sort of structure in place for the two to coexist?
>
> It doesn't work that way -- people will have the old typescript-mode in
> their ~/.emacs.d/elpa/ files for years still to come, and things will
> just be messy.
>
> So I think the way forward here would be to just plonk the new code into
> the package, but choose totally different code paths depending on
> whether we're in Emacs 29 (and have tree-sitter) or not.  Then you can
> start removing the non-tree-sitter code from the file afterwards.

Perhaps I missed something, but why not just name the new mode 'ts-mode'
(just like we jave 'js-mode' not 'javascript-mode')?

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

On 11 October 2022 21:20:18 CEST, Philip Kaludercic <philipk@posteo.net> wrote:
>Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Theodor Thornhill <theo@thornhill.no> writes:
>>
>>> How about deprecating typescript.el the moment the 29.1 branch is cut?
>>> After we can but some sort of structure in place for the two to coexist?
>>
>> It doesn't work that way -- people will have the old typescript-mode in
>> their ~/.emacs.d/elpa/ files for years still to come, and things will
>> just be messy.
>>
>> So I think the way forward here would be to just plonk the new code into
>> the package, but choose totally different code paths depending on
>> whether we're in Emacs 29 (and have tree-sitter) or not.  Then you can
>> start removing the non-tree-sitter code from the file afterwards.
>
>Perhaps I missed something, but why not just name the new mode 'ts-mode'
>(just like we jave 'js-mode' not 'javascript-mode')?

Lol very nice. Consider it done :)

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Theodor Thornhill <theo@thornhill.no> writes:

> My suggestion is to add the tree-sitter variant in these cases, and let
> the other modes die a slow, deprecated death down the line.

In that case, will any of you object if someone writes a parser
generator in C and an implementation of the tree-sitter runtime library
that can be included with Emacs itself (namely, with the copyright
assigned to the FSF?)

They will be able to use any existing tree-sitter grammar, of course.

My opinion is that Emacs should not degrade its text editing
capabilities so drastically if a non-system library (i.e. not ncurses or
Xlib) is not present.

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 10, 2022, at 9:43 PM, Po Lu <luangruo@yahoo.com> wrote:
> 
> Theodor Thornhill <theo@thornhill.no> writes:
> 
>> My suggestion is to add the tree-sitter variant in these cases, and let
>> the other modes die a slow, deprecated death down the line.
> 
> In that case, will any of you object if someone writes a parser
> generator in C and an implementation of the tree-sitter runtime library
> that can be included with Emacs itself (namely, with the copyright
> assigned to the FSF?)
> 
> They will be able to use any existing tree-sitter grammar, of course.

Tree-sitter library and the “official” language definitions are all MIT licensed. So nothing prevents us from including it copyright-wise.

> 
> My opinion is that Emacs should not degrade its text editing
> capabilities so drastically if a non-system library (i.e. not ncurses or
> Xlib) is not present.

Considering that Emacs currently doesn’t have C# and typescript support in core at all, I wouldn’t call it a downgrade. But I agree the idea that a major mode requiring external dependency to function is not very comfortable.

Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

On 11 October 2022 07:14:36 CEST, Yuan Fu <casouri@gmail.com> wrote:
>
>
>> On Oct 10, 2022, at 9:43 PM, Po Lu <luangruo@yahoo.com> wrote:
>> 
>> Theodor Thornhill <theo@thornhill.no> writes:
>> 
>>> My suggestion is to add the tree-sitter variant in these cases, and let
>>> the other modes die a slow, deprecated death down the line.
>> 
>> In that case, will any of you object if someone writes a parser
>> generator in C and an implementation of the tree-sitter runtime library
>> that can be included with Emacs itself (namely, with the copyright
>> assigned to the FSF?)
>> 
>> They will be able to use any existing tree-sitter grammar, of course.
>
>Tree-sitter library and the “official” language definitions are all MIT licensed. So nothing prevents us from including it copyright-wise.
>
>> 
>> My opinion is that Emacs should not degrade its text editing
>> capabilities so drastically if a non-system library (i.e. not ncurses or
>> Xlib) is not present.
>
>Considering that Emacs currently doesn’t have C# and typescript support in core at all, I wouldn’t call it a downgrade. But I agree the idea that a major mode requiring external dependency to function is not very comfortable.
>

There's no issue with adding cc mode c# support in core, as the copyright is sorted for it. However, that will create maintenance burden for Alan, as he has the most knowledge on its inner workings. 

If we want I can prepare a patch with both, in the style of js-mode. 

What do you think? 
Theodor

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> Date: Tue, 11 Oct 2022 07:33:42 +0200
> From: Theodor Thornhill <theo@thornhill.no>
> CC: Eli Zaretskii <eliz@gnu.org>, Alan Mackenzie <acm@muc.de>,
>  emacs-devel <emacs-devel@gnu.org>, jostein@kjonigsen.net
> 
> There's no issue with adding cc mode c# support in core, as the copyright is sorted for it. However, that will create maintenance burden for Alan, as he has the most knowledge on its inner workings. 
> 
> If we want I can prepare a patch with both, in the style of js-mode. 
> 
> What do you think? 

We are waiting for Alan to chime in, after he catches up with the
background.  Let's be patient.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Tue, 11 Oct 2022 07:33:42 +0200
>> From: Theodor Thornhill <theo@thornhill.no>
>> CC: Eli Zaretskii <eliz@gnu.org>, Alan Mackenzie <acm@muc.de>,
>>  emacs-devel <emacs-devel@gnu.org>, jostein@kjonigsen.net
>> 
>> There's no issue with adding cc mode c# support in core, as the copyright is sorted for it. However, that will create maintenance burden for Alan, as he has the most knowledge on its inner workings. 
>> 
>> If we want I can prepare a patch with both, in the style of js-mode. 
>> 
>> What do you think? 
>
> We are waiting for Alan to chime in, after he catches up with the
> background.  Let's be patient.

Yes, absolutely.  I just tried to make the point that for C# mode
there's a quick way to get support for both non-tree-sitter and
tree-sitter.  A little less so in the case for Typescript.

:)

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Yuan Fu <casouri@gmail.com> writes:

> MIT license

Which one?

> So nothing prevents us from including it copyright-wise.

The tree-sitter library does not have copyright assigned to the FSF.

IMNHSO a text editor should be able to edit languages it supports
out-of-the-box, with only system libraries and what is in
emacs-28.2.tar.gz.

There is no precedent for requiring a third party library that users
have to build themselves to edit text comfortably.  Thus, tree-sitter
(or a 100% reproduction, which is my idea) should come with Emacs
itself.

Optional dependencies such as librsvg, libxml and gnutls are not
precedent here - all three are widely available, and relate to features
that are orthogonal to text editing.

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Theodor Thornhill <theo@thornhill.no>,  Eli Zaretskii <eliz@gnu.org>,
>   Alan Mackenzie <acm@muc.de>,  emacs-devel <emacs-devel@gnu.org>,
>   jostein@kjonigsen.net
> Date: Tue, 11 Oct 2022 13:47:27 +0800
> 
> IMNHSO a text editor should be able to edit languages it supports
> out-of-the-box, with only system libraries and what is in
> emacs-28.2.tar.gz.

That is about to happen, please read the other messages in this
thread.  But "able to edit languages" and "able to provide a
reasonably modern support for those languages" aren't necessarily the
same.  Like rendering complex scripts, for example: Emacs supports
Arabic, but without HarfBuzz display of Arabic is illegible to Arabic
and Farsi speakers.  Does that strike you as a problem?  If not, why
not?

> There is no precedent for requiring a third party library that users
> have to build themselves to edit text comfortably.

There is, actually, see above.

> Optional dependencies such as librsvg, libxml and gnutls are not
> precedent here - all three are widely available, and relate to features
> that are orthogonal to text editing.

GnuTLS, at least, is no longer optional: most servers will reject
clear-text connections.  TLS is nowadays a basic requirement, AFAIU.

So we already depend on external libraries for reasonable
functioning.  There's nothing new here.

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> That is about to happen, please read the other messages in this
> thread.  But "able to edit languages" and "able to provide a
> reasonably modern support for those languages" aren't necessarily the
> same.

[...]

> Like rendering complex scripts, for example: Emacs supports
> Arabic, but without HarfBuzz display of Arabic is illegible to Arabic
> and Farsi speakers.  Does that strike you as a problem?  If not, why
> not?

English is the lingua franca of computer programmers.  Thus, if Emacs
supports English, then it will immediately work for the vast majority of
people who have reason to use it.  Arabic and Farsi are significantly
more niche languages that are generally not used in computer code at
all, so it is okay to ask people who want to program in those languages
to install some other library.

Besides, HarfBuzz is arguably a system library.  Modern GNU/Linux
systems cannot provide text display without it at all.

> GnuTLS, at least, is no longer optional: most servers will reject
> clear-text connections.  TLS is nowadays a basic requirement, AFAIU.
>
> So we already depend on external libraries for reasonable
> functioning.  There's nothing new here.

Network access is not required for "reasonable functioning" - we even
have builds that do not support it at all.

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: casouri@gmail.com,  theo@thornhill.no,  acm@muc.de,
>   emacs-devel@gnu.org,  jostein@kjonigsen.net
> Date: Tue, 11 Oct 2022 15:50:41 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Like rendering complex scripts, for example: Emacs supports
> > Arabic, but without HarfBuzz display of Arabic is illegible to Arabic
> > and Farsi speakers.  Does that strike you as a problem?  If not, why
> > not?
> 
> English is the lingua franca of computer programmers.

Yes, but Emacs is not only for editing computer programs.  It is also
for editing human-readable text, email exchange, writing program
documentation, and other areas where support for various scripts and
languages is important.  Otherwise, why would we invest such a
significant effort into providing and developing those features?  Why
did Emacs so painstakingly go from unibyte text representation to
multibyte in version 20?

> Arabic and Farsi are significantly more niche languages that are
> generally not used in computer code at all

Actually, they are: in comments and strings.

Anyway, this line of reasoning is a non-starter.  You are, in effect,
denying the whole chunk of Emacs history and development since the
late 90s.

> Besides, HarfBuzz is arguably a system library.  Modern GNU/Linux
> systems cannot provide text display without it at all.

The same will arguably happen with tree-sitter, once Emacs starts
using it seriously.

> > GnuTLS, at least, is no longer optional: most servers will reject
> > clear-text connections.  TLS is nowadays a basic requirement, AFAIU.
> >
> > So we already depend on external libraries for reasonable
> > functioning.  There's nothing new here.
> 
> Network access is not required for "reasonable functioning"

Of course, it is.

> we even have builds that do not support it at all.

We will also support a build without tree-sitter.  It will lack
important features, exactly like builds without the other optional
libraries lack important features.  But it will allow one to edit
program source files.

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Yes, but Emacs is not only for editing computer programs.  It is also
> for editing human-readable text, email exchange, writing program
> documentation, and other areas where support for various scripts and
> languages is important.  Otherwise, why would we invest such a
> significant effort into providing and developing those features?  Why
> did Emacs so painstakingly go from unibyte text representation to
> multibyte in version 20?

Program documentation is mostly written in English.  But anyway, see
below.

> Actually, they are: in comments and strings.
>
> Anyway, this line of reasoning is a non-starter.  You are, in effect,
> denying the whole chunk of Emacs history and development since the
> late 90s.

No.  Support for many different scripts is obviously nice-to-have, but
is not at all crucial for a text editor that primarily edits code.

> The same will arguably happen with tree-sitter, once Emacs starts
> using it seriously.

I'm sorry, but I really cannot agree there.  Emacs aside, only niche
editors utilize tree-sitter to provide core parsing functionality.

No operating system will start including it by default just because
it has become an important dependency of Emacs.

> Of course, it is.

What feature in Emacs requires connecting to the internet to work?
package.el aside: packages can easily be retrieved via ftp, installed
manually, and are not required for Emacs to work well.

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: casouri@gmail.com,  theo@thornhill.no,  acm@muc.de,
>   emacs-devel@gnu.org,  jostein@kjonigsen.net
> Date: Tue, 11 Oct 2022 16:23:43 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Yes, but Emacs is not only for editing computer programs.  It is also
> > for editing human-readable text, email exchange, writing program
> > documentation, and other areas where support for various scripts and
> > languages is important.  Otherwise, why would we invest such a
> > significant effort into providing and developing those features?  Why
> > did Emacs so painstakingly go from unibyte text representation to
> > multibyte in version 20?
> 
> Program documentation is mostly written in English.

No, it isn't.  It was, like 30 years ago.  But not today.

> > Anyway, this line of reasoning is a non-starter.  You are, in effect,
> > denying the whole chunk of Emacs history and development since the
> > late 90s.
> 
> No.  Support for many different scripts is obviously nice-to-have, but
> is not at all crucial for a text editor that primarily edits code.

Emacs is not "primarily for editing code", it is a general-purpose
text-editing environment, suitable to edit _any_ text for _any_
purpose.

> What feature in Emacs requires connecting to the internet to work?
> package.el aside: packages can easily be retrieved via ftp, installed
> manually, and are not required for Emacs to work well.

When did you last time try to use FTP?  It is disabled on many
servers, and is considered unsafe for many years.  It is no longer
relevant for modern connectivity.

Look, this argument goes nowhere, so let's just stop wasting our time.

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Look, this argument goes nowhere, so let's just stop wasting our time.

Fine by me.

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Kangas]

Po Lu <luangruo@yahoo.com> writes:

> No operating system will start including it by default just because
> it has become an important dependency of Emacs.

I expect that any reasonable (read: good) GNU/Linux distribution will
just install treesitter as a dependency of Emacs.

If there are any distributions out there that don't include Emacs by
default, then that should obviously be fixed as a high priority bug.
But that's a separate issue.  ;-)

Re: Call for volunteers: add tree-sitter support to major modes

[Visuwesh]

[செவ்வாய் அக்டோபர் 11, 2022] Eli Zaretskii wrote:

>> [...]
>> Besides, HarfBuzz is arguably a system library.  Modern GNU/Linux
>> systems cannot provide text display without it at all.
>
> The same will arguably happen with tree-sitter, once Emacs starts
> using it seriously.
>
> [...]
>
>> we even have builds that do not support it at all.
>
> We will also support a build without tree-sitter.  It will lack
> important features, exactly like builds without the other optional
> libraries lack important features.  But it will allow one to edit
> program source files.

I've read the rest of this thread and understand the argument but how
memory hungry does Emacs get when enabling tree-sitter?  As far as I
vaguely remember from the original tree-sitter inclusion discussion,
tree-sitter used about the same memory as the buffer did for smallish
files.  Even more vaguely, I remember opening xdisp.c with tree-sitter
enabled shot up the memory usage to an unacceptable limit by my
computer's standards.
As someone with a mere 4G of RAM and 4G of swap space, I'm wondering if
enabling tree-sitter will make my laptop even more memory hungry than it
already can be when stressed [1].


[1] This happens when I have a lot of tabs open in Chromium+a handful of
20~100M PDF files opened in Emacs via pdf-tools.

Re: Call for volunteers: add tree-sitter support to major modes

[Daniel Martín]

Visuwesh <visuweshm@gmail.com> writes:

>
> I've read the rest of this thread and understand the argument but how
> memory hungry does Emacs get when enabling tree-sitter?  As far as I
> vaguely remember from the original tree-sitter inclusion discussion,
> tree-sitter used about the same memory as the buffer did for smallish
> files.  Even more vaguely, I remember opening xdisp.c with tree-sitter
> enabled shot up the memory usage to an unacceptable limit by my
> computer's standards.
> As someone with a mere 4G of RAM and 4G of swap space, I'm wondering if
> enabling tree-sitter will make my laptop even more memory hungry than it
> already can be when stressed [1].

In my preliminary tests with Emacs 29, Emacs -Q (macOS GUI app) starts
at around 48 MB of RAM.  When I (require 'treesit), memory usage climbs
to about 57 MB.  If I open a big Python file [1] and enable Tree-sitter
for the buffer, memory stays around 60-65 MB after a few scroll
operations and basic editing.  I can't test with xdisp.c as we don't
have C Tree-sitter support, AFAIK.

I think memory usage will vary depending on how we people use the
library, but I haven't seen any noticeable impact on memory usage
compared to a standard Emacs session.

[1]:
https://github.com/tensorflow/tensorflow/blob/8056627d6a794f9acaff1a8165cc247bd252db31/tensorflow/python/ops/sparse_ops.py

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 11, 2022, at 9:56 AM, Daniel Martín <mardani29@yahoo.es> wrote:
> 
> Visuwesh <visuweshm@gmail.com> writes:
> 
>> 
>> I've read the rest of this thread and understand the argument but how
>> memory hungry does Emacs get when enabling tree-sitter?  As far as I
>> vaguely remember from the original tree-sitter inclusion discussion,
>> tree-sitter used about the same memory as the buffer did for smallish
>> files.  Even more vaguely, I remember opening xdisp.c with tree-sitter
>> enabled shot up the memory usage to an unacceptable limit by my
>> computer's standards.
>> As someone with a mere 4G of RAM and 4G of swap space, I'm wondering if
>> enabling tree-sitter will make my laptop even more memory hungry than it
>> already can be when stressed [1].
> 
> In my preliminary tests with Emacs 29, Emacs -Q (macOS GUI app) starts
> at around 48 MB of RAM.  When I (require 'treesit), memory usage climbs
> to about 57 MB.  If I open a big Python file [1] and enable Tree-sitter
> for the buffer, memory stays around 60-65 MB after a few scroll
> operations and basic editing.  I can't test with xdisp.c as we don't
> have C Tree-sitter support, AFAIK.
> 
> I think memory usage will vary depending on how we people use the
> library, but I haven't seen any noticeable impact on memory usage
> compared to a standard Emacs session.

Tree-sitter uses about 10x the buffer size to store the parse tree. You can set treesit-max-buffer-size to inhibit usage of tree-sitter on large buffers.

Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 10 Oct 2022 22:14:36 -0700
> Cc: Theodor Thornhill <theo@thornhill.no>,
>  Eli Zaretskii <eliz@gnu.org>,
>  Alan Mackenzie <acm@muc.de>,
>  emacs-devel <emacs-devel@gnu.org>,
>  jostein@kjonigsen.net
> 
> Tree-sitter library and the “official” language definitions are all MIT licensed. So nothing prevents us from including it copyright-wise.

I see no reason whatsoever to consider bundling tree-sitter.  Building
it is very easy, and w=once Emacs starts using it seriously, the Emacs
distros will take care of offering suitable tree-sitter distributions.

> > My opinion is that Emacs should not degrade its text editing
> > capabilities so drastically if a non-system library (i.e. not ncurses or
> > Xlib) is not present.
> 
> Considering that Emacs currently doesn’t have C# and typescript support in core at all, I wouldn’t call it a downgrade. But I agree the idea that a major mode requiring external dependency to function is not very comfortable.

I don't agree, and don't feel myself uncomfortable at all.  For the
same reason I don't feel uncomfortable because we use HarfBuzz for
rendering and GnuTLS for networking.

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't agree, and don't feel myself uncomfortable at all.  For the
> same reason I don't feel uncomfortable because we use HarfBuzz for
> rendering and GnuTLS for networking.

How relevant is fontification for editing code, and how relevant are
HarfBuzz and GnuTLS for the same?

Emacs is very useful for editing code without any of HarfBuzz, Xft,
Cairo, and TLS support.  If CC Mode dies a "slow, deprecated death",
then Emacs will not be useful at all without tree-sitter installed.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Po Lu <luangruo@yahoo.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> I don't agree, and don't feel myself uncomfortable at all.  For the
>> same reason I don't feel uncomfortable because we use HarfBuzz for
>> rendering and GnuTLS for networking.
>
> How relevant is fontification for editing code, and how relevant are
> HarfBuzz and GnuTLS for the same?
>

Well, we are not talking just about fontification, but we will get all
sorts of ast-powered benefits.  Navigation, speed, movement, context
aware deletion/inserting and more. There's a whole body of ingenious
stuff that's been made in other editors with this framework.  They will
immediately be relevant for emacs as well. This is much too simplistic a
view, I'm afraid.

> Emacs is very useful for editing code without any of HarfBuzz, Xft,
> Cairo, and TLS support.  If CC Mode dies a "slow, deprecated death",
> then Emacs will not be useful at all without tree-sitter installed.

I did not suggest CC Mode dies a slow, deprecated death.  I suggested
the modes out-of-tree that have obvious issues that has been lingering
for years will be deprecated.  That's different, and not at all in line
with what you are saying.

My point was that I'd consider it unfair to Alan (though he should
answer that himself) moving the C# variant to core, because that would
inevitably but even more stress on him.  _Every_ time I've had some
issues maintaining some problems in C# mode it has caused weeks of work
for Alan.

With tree-sitter, the same bugs were solved with 5 minutes coding.  I'm
not kidding.

If interested, you can dig around lists and issues discussing multiline
strings in CC Mode.

Thanks,

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Theodor Thornhill <theo@thornhill.no> writes:

> Well, we are not talking just about fontification, but we will get all
> sorts of ast-powered benefits.  Navigation, speed, movement, context
> aware deletion/inserting and more. There's a whole body of ingenious
> stuff that's been made in other editors with this framework.  They will
> immediately be relevant for emacs as well. This is much too simplistic a
> view, I'm afraid.

Interesting.  I understand that tree-sitter is faster, but could you
please explain what you mean by "context aware deletion/insertion"?

> My point was that I'd consider it unfair to Alan (though he should
> answer that himself) moving the C# variant to core, because that would
> inevitably but even more stress on him.  _Every_ time I've had some
> issues maintaining some problems in C# mode it has caused weeks of work
> for Alan.

Then I guess I misunderstood what you said.

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Po Lu <luangruo@yahoo.com> writes:

> Theodor Thornhill <theo@thornhill.no> writes:
>
>> Well, we are not talking just about fontification, but we will get all
>> sorts of ast-powered benefits.  Navigation, speed, movement, context
>> aware deletion/inserting and more. There's a whole body of ingenious
>> stuff that's been made in other editors with this framework.  They will
>> immediately be relevant for emacs as well. This is much too simplistic a
>> view, I'm afraid.
>
> Interesting.  I understand that tree-sitter is faster, but could you
> please explain what you mean by "context aware deletion/insertion"?
>

Sure!

(The following example is from the other tree-sitter implementation, but
it would be as simple, if not simpler in the new one)

Consider this code:

```
(defun csharp-delete-method-at-point ()
  "Deletes the method at point."
  (interactive)
  (when-let ((method (tree-sitter-node-at-point 'method_declaration)))
    (delete-region (tsc-node-start-position method)
                   (tsc-node-end-position method))))
```

If point is _anywhere_ inside a method, this function will delete the
whole thing.  It knows the beginning and end, and doesn't have to do
anything else than returning the relevant points.  The same could be
done with i.e. string, where if:

```
"some string here where this | denotes where point is"
```

You could create paredit-like functionality to delete _until_ the last
quote, etc.  Compare them to Vims 'cip', 'dap' etc.  These are only
trivial examples, but you could also imagine creating refactoring
functions where you extract variables and insert them in the scope of
other functions and so forth.  The huge thing here is that you have the
_actual_ parse tree.  Indentation is also now simpler, more correct.
There are all sorts of things we could evolve from here.  Font-lock is
just because that is the clearest, easiest and fastest benefit.

>> My point was that I'd consider it unfair to Alan (though he should
>> answer that himself) moving the C# variant to core, because that would
>> inevitably but even more stress on him.  _Every_ time I've had some
>> issues maintaining some problems in C# mode it has caused weeks of work
>> for Alan.
>
> Then I guess I misunderstood what you said.

No worries!

Thanks,
Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Theodor Thornhill <theo@thornhill.no> writes:

> Sure!
>
> (The following example is from the other tree-sitter implementation, but
> it would be as simple, if not simpler in the new one)
>
> Consider this code:
>
> ```
> (defun csharp-delete-method-at-point ()
>   "Deletes the method at point."
>   (interactive)
>   (when-let ((method (tree-sitter-node-at-point 'method_declaration)))
>     (delete-region (tsc-node-start-position method)
>                    (tsc-node-end-position method))))
> ```
>
> If point is _anywhere_ inside a method, this function will delete the
> whole thing.  It knows the beginning and end, and doesn't have to do
> anything else than returning the relevant points.  The same could be
> done with i.e. string, where if:
>
> ```
> "some string here where this | denotes where point is"
> ```
>
> You could create paredit-like functionality to delete _until_ the last
> quote, etc.  Compare them to Vims 'cip', 'dap' etc.  These are only
> trivial examples, but you could also imagine creating refactoring
> functions where you extract variables and insert them in the scope of
> other functions and so forth.  The huge thing here is that you have the
> _actual_ parse tree.  Indentation is also now simpler, more correct.
> There are all sorts of things we could evolve from here.  Font-lock is
> just because that is the clearest, easiest and fastest benefit.

Thank you for the detailed explanation.

Re: Call for volunteers: add tree-sitter support to major modes

[Yuan Fu]

> On Oct 11, 2022, at 12:13 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Mon, 10 Oct 2022 22:14:36 -0700
>> Cc: Theodor Thornhill <theo@thornhill.no>,
>> Eli Zaretskii <eliz@gnu.org>,
>> Alan Mackenzie <acm@muc.de>,
>> emacs-devel <emacs-devel@gnu.org>,
>> jostein@kjonigsen.net
>> 
>> Tree-sitter library and the “official” language definitions are all MIT licensed. So nothing prevents us from including it copyright-wise.
> 
> I see no reason whatsoever to consider bundling tree-sitter.  Building
> it is very easy, and w=once Emacs starts using it seriously, the Emacs
> distros will take care of offering suitable tree-sitter distributions.

I agree, I was replying to Po’s explaining there is no copyright issue.

> 
>>> My opinion is that Emacs should not degrade its text editing
>>> capabilities so drastically if a non-system library (i.e. not ncurses or
>>> Xlib) is not present.
>> 
>> Considering that Emacs currently doesn’t have C# and typescript support in core at all, I wouldn’t call it a downgrade. But I agree the idea that a major mode requiring external dependency to function is not very comfortable.
> 
> I don't agree, and don't feel myself uncomfortable at all.  For the
> same reason I don't feel uncomfortable because we use HarfBuzz for
> rendering and GnuTLS for networking.

Oh, I’m perfectly comfortable with having build dependency on tree-sitter. I was not comfortable with having a built-in major mode that simply errors out when tree-sitter is not available. But if we are going to include some minimal support into typescript-mode in the case where tree-sitter is unavailable, as you suggested in a message, then I’m very comfortable ;-)

Yuan

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Alan Mackenzie <acm@muc.de>,
>   emacs-devel@gnu.org,  jostein@kjonigsen.net
> Date: Tue, 11 Oct 2022 12:43:36 +0800
> 
> Theodor Thornhill <theo@thornhill.no> writes:
> 
> > My suggestion is to add the tree-sitter variant in these cases, and let
> > the other modes die a slow, deprecated death down the line.
> 
> In that case, will any of you object if someone writes a parser
> generator in C and an implementation of the tree-sitter runtime library
> that can be included with Emacs itself (namely, with the copyright
> assigned to the FSF?)

Yes, I will.

It isn't our business to develop language parsers, and we have no hope
of keeping up with the technology developments in that area and with
new languages being invented, well enough to offer reasonable and
modern support in Emacs for editing programs in those languages.

It's the same reason why we depend on text shaping engines like
HarfBuzz for rendering complex scripts and other typography-related
features, on image libraries to display images, on GnuTLS to support
TLS connections, etc. etc.

We cannot possibly be experts in all of those fields, we don't have
the manpower for that.  In the case of support for editing programs we
actually tried for many years, and the result is before our eyes --
and it's unsatisfactory at best.  We should advance Emacs towards
leveraging existing technologies, not try developing those
technologies in-house, or replacing them by cheap hacks and kludges.

> My opinion is that Emacs should not degrade its text editing
> capabilities so drastically if a non-system library (i.e. not ncurses or
> Xlib) is not present.

It is completely impractical to expect us to be able to develop this
in-house.  If tree-sitter becomes defunct (something that doesn't seem
like it will happen any time soon), we will have to find a suitable
replacement -- like we did with text-shaping engine, when the FLT
library stopped being actively developed.  Dealing with these changes
in the outside world is an important part of the job of the Emacs
maintainers (as is identifying new emerging technologies that can be
leveraged for new and enhanced Emacs features).  We are doing this
part for several years already, and doing it successfully.  I see no
reason to change that, and all the reasons, including past experience,
to keep up.

The present font-lock and indentation features based on our own code
can remain in Emacs forever, if someone wants a safe fallback for when
worse comes to worst.  But these implementations are from my POV a
dead end: they cannot be developed any further, and for some
languages, like C++, are already all but failing to reasonably and
efficiently support their complex grammars.  Using expertise of others
is simply a logical step based on past experience; the decision to
develop this in-house would be a step back and a disaster in the long
run, IMNSHO.

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Yes, I will.
>
> It isn't our business to develop language parsers, and we have no hope
> of keeping up with the technology developments in that area and with
> new languages being invented, well enough to offer reasonable and
> modern support in Emacs for editing programs in those languages.

It isn't quite our business to develop mini-gmp either, yet it's also
kept in-tree.

> It's the same reason why we depend on text shaping engines like
> HarfBuzz for rendering complex scripts and other typography-related
> features, on image libraries to display images, on GnuTLS to support
> TLS connections, etc. etc.

But editing text mostly does not involve editing complex shaped text,
and likewise, TLS connections and image display is mostly orthogonal to
the business of editing text.

> We cannot possibly be experts in all of those fields, we don't have
> the manpower for that.  In the case of support for editing programs we
> actually tried for many years, and the result is before our eyes --
> and it's unsatisfactory at best.  We should advance Emacs towards
> leveraging existing technologies, not try developing those
> technologies in-house, or replacing them by cheap hacks and kludges.

Then why is mini-gmp kept in-tree? Portable multiple precision
arithmetic is at least as complicated (if not more) than tree-sitter,
yet mini-gmp is a completely satisfactory solution for people who do not
have libgmp installed.

mini-tree-sitter would fill the same role: it would expose the same API
as tree-sitter, but be small enough for Emacs to include and use when
tree-sitter is not available.

> It is completely impractical to expect us to be able to develop this
> in-house.

Why?  The tree-sitter runtime library is only about 16,000 lines of C
source code and headers, which is less than xdisp.c alone, and only
slightly more than keyboard.c.  In addition, the path to the current
implementation of tree-sitter is well-trodden, so it can't be that hard
to walk down that path again.

Such an implementation would be able to make use of existing and new
tree-sitter grammars without changes.

> If tree-sitter becomes defunct (something that doesn't seem like it
> will happen any time soon)

Famous last words.

> , we will have to find a suitable replacement -- like we did with
> text-shaping engine, when the FLT library stopped being actively
> developed.  Dealing with these changes in the outside world is an
> important part of the job of the Emacs maintainers

Text shaping is, at least IME, immensely more complicated than
interactive language parsing.

> The present font-lock and indentation features based on our own code
> can remain in Emacs forever, if someone wants a safe fallback for when
> worse comes to worst.

That logic could also have been applied to bignums, yet there was no
objection to including mini-gmp.

> But these implementations are from my POV a dead end: they cannot be
> developed any further, and for some languages, like C++, are already
> all but failing to reasonably and efficiently support their complex
> grammars.  Using expertise of others is simply a logical step based on
> past experience; the decision to develop this in-house would be a step
> back and a disaster in the long run, IMNSHO.

I'm not disputing that fact, which is why I proposed to write an
*implementation* of tree-sitter for inclusion with Emacs, and not to
develop an equivalent in-house.

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: theo@thornhill.no,  acm@muc.de,  emacs-devel@gnu.org,
>   jostein@kjonigsen.net
> Date: Tue, 11 Oct 2022 15:31:09 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Yes, I will.
> >
> > It isn't our business to develop language parsers, and we have no hope
> > of keeping up with the technology developments in that area and with
> > new languages being invented, well enough to offer reasonable and
> > modern support in Emacs for editing programs in those languages.
> 
> It isn't quite our business to develop mini-gmp either, yet it's also
> kept in-tree.

I'm okay with removing mini-gmp, if the GMP library is supported on
all the platforms we care about.  But mini-gmp is from Gnulib, so we
don't need to develop or maintain it.

> > It's the same reason why we depend on text shaping engines like
> > HarfBuzz for rendering complex scripts and other typography-related
> > features, on image libraries to display images, on GnuTLS to support
> > TLS connections, etc. etc.
> 
> But editing text mostly does not involve editing complex shaped text,

Only as long as you use Latin scripts.  And even there, we want to
support typographic ligatures and other features of modern fonts.

> and likewise, TLS connections and image display is mostly orthogonal to
> the business of editing text.

Emacs is not just a text editor.  Networking is nowadays an integral
part of any development environment.  E.g., package.el would be
impossible without TLS connections.

> > It is completely impractical to expect us to be able to develop this
> > in-house.
> 
> Why?

Because we have past experience to indicate that.  Look at Semantic,
look at CC Mode, etc.  My conclusion from that is clear, and it's
non-negotiable, not as long as I'm doing this job.

> The tree-sitter runtime library is only about 16,000 lines of C
> source code and headers, which is less than xdisp.c alone, and only
> slightly more than keyboard.c.

Yes, and how many people do we have on board who know all of xdisp.c
and all of keyboard.c?  Exactly zero.

Really, how many more examples of this do we need to understand what
is and what isn't future-proof for Emacs??  How blind can we be if,
based on long and rich past experience such as what we have, we don't
realize that developing key technologies in-house is a dead end for
us?  Are we really _that_ blind??

> > , we will have to find a suitable replacement -- like we did with
> > text-shaping engine, when the FLT library stopped being actively
> > developed.  Dealing with these changes in the outside world is an
> > important part of the job of the Emacs maintainers
> 
> Text shaping is, at least IME, immensely more complicated than
> interactive language parsing.

I don't see how that matters.  Both are outside of our expertise.

> > The present font-lock and indentation features based on our own code
> > can remain in Emacs forever, if someone wants a safe fallback for when
> > worse comes to worst.
> 
> That logic could also have been applied to bignums, yet there was no
> objection to including mini-gmp.

mini-gmp is from Gnulib, so it isn't our maintenance burden.

> > But these implementations are from my POV a dead end: they cannot be
> > developed any further, and for some languages, like C++, are already
> > all but failing to reasonably and efficiently support their complex
> > grammars.  Using expertise of others is simply a logical step based on
> > past experience; the decision to develop this in-house would be a step
> > back and a disaster in the long run, IMNSHO.
> 
> I'm not disputing that fact, which is why I proposed to write an
> *implementation* of tree-sitter for inclusion with Emacs, and not to
> develop an equivalent in-house.

Sorry, I cannot agree with such a terrible waste of our scarce
resources.  I cannot prevent anyone from working on whatever they
like, but don't expect me to agree to adding such an implementation to
the Emacs sources and taking responsibility for its maintenance.  It's
completely against every goal towards which I worked since I became
the Emacs maintainer, and is very wrong for the reasons I already
abundantly explained.

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Only as long as you use Latin scripts.

Or CJK ones, which work perfectly fine without HarfBuzz text shaping.
I have no experience with others.

> And even there, we want to support typographic ligatures and other
> features of modern fonts.

Typographic ligatures and modern fonts are also orthogonal to editing
code, which can be done reasonably on 8x13 or 9x15.

> Emacs is not just a text editor.  Networking is nowadays an integral
> part of any development environment.  E.g., package.el would be
> impossible without TLS connections.

But why would package.el and similar features be an integral part of any
development environment?

Remove networking support and TLS from Emacs, and people will still use
it.  Then, delete (or slowly waste away) CC Mode, and almost all of our
users will disappear.

> Because we have past experience to indicate that.  Look at Semantic,
> look at CC Mode, etc.  My conclusion from that is clear, and it's
> non-negotiable, not as long as I'm doing this job.

The difference between Semantic and tree-sitter is that Semantic is
comparatively slow, and written in Emacs Lisp, while the latter is
written in C, a much faster language with multiple higher quality
implementations.  CC Mode barely qualifies as "language parsing
technology."

Not to mention that the scope of CC Mode is really much wider than that
of tree-sitter.  i.e. tree-sitter does not provide style guessing
functionality.

> Really, how many more examples of this do we need to understand what
> is and what isn't future-proof for Emacs??  How blind can we be if,
> based on long and rich past experience such as what we have, we don't
> realize that developing key technologies in-house is a dead end for
> us?  Are we really _that_ blind??

What is actually demonstrated is that Emacs Lisp and the existing C
primitives are unsuitable for implementing language parsing technology.

> Sorry, I cannot agree with such a terrible waste of our scarce
> resources.

Too bad.

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: theo@thornhill.no,  acm@muc.de,  emacs-devel@gnu.org,
>   jostein@kjonigsen.net
> Date: Tue, 11 Oct 2022 16:15:00 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Only as long as you use Latin scripts.
> 
> Or CJK ones, which work perfectly fine without HarfBuzz text shaping.
> I have no experience with others.

Well, Emacs supports, and will continue to support, many others.

> > And even there, we want to support typographic ligatures and other
> > features of modern fonts.
> 
> Typographic ligatures and modern fonts are also orthogonal to editing
> code

No, they aren't: witness the requests to support ligatures in program
sources, which are driven by modern fonts like Iosevka and FiraCode.

> > Emacs is not just a text editor.  Networking is nowadays an integral
> > part of any development environment.  E.g., package.el would be
> > impossible without TLS connections.
> 
> But why would package.el and similar features be an integral part of any
> development environment?

Can you imagine a development environment nowadays that is unable to
update or upgrade optional packages or install new optional packages?
I can't.

> Remove networking support and TLS from Emacs, and people will still use
> it.  Then, delete (or slowly waste away) CC Mode, and almost all of our
> users will disappear.

I think both will cause users to disappear.  But we can argue about
this till Kingdom Come, because such an experiment will never happen.

> > Because we have past experience to indicate that.  Look at Semantic,
> > look at CC Mode, etc.  My conclusion from that is clear, and it's
> > non-negotiable, not as long as I'm doing this job.
> 
> The difference between Semantic and tree-sitter is that Semantic is
> comparatively slow, and written in Emacs Lisp, while the latter is
> written in C, a much faster language with multiple higher quality
> implementations.

You see only the aspects that support your POV, and dismiss the rest.
One aspect you ignore is the fact that Semantic is where it is today
because people who developed it stopped its development.  This is
directly relevant to your proposal to develop parsing capabilities as
part of Emacs -- the result in the long run will be the same, because
no one stays in Emacs development forever.

By contrast, using external implementations of technologies which are
important to IDEs has a much higher chance of finding Free Software
implementations that we can use, even if some of them become defunct.

> CC Mode barely qualifies as "language parsing technology."

Yes, and look at the problems with which it needs to struggle, and the
resulting performance issues.

> Not to mention that the scope of CC Mode is really much wider than that
> of tree-sitter.  i.e. tree-sitter does not provide style guessing
> functionality.

We will keep the parts that are independent of language parsing.

> > Really, how many more examples of this do we need to understand what
> > is and what isn't future-proof for Emacs??  How blind can we be if,
> > based on long and rich past experience such as what we have, we don't
> > realize that developing key technologies in-house is a dead end for
> > us?  Are we really _that_ blind??
> 
> What is actually demonstrated is that Emacs Lisp and the existing C
> primitives are unsuitable for implementing language parsing technology.

That, too.  But it's the more general conclusions that matter in the
long run.

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> No, they aren't: witness the requests to support ligatures in program
> sources, which are driven by modern fonts like Iosevka and FiraCode.

People request it, yes, but that does not make it directly relevant to
programming.

> Can you imagine a development environment nowadays that is unable to
> update or upgrade optional packages or install new optional packages?
> I can't.

That is the business of the package manager, not Emacs.

> You see only the aspects that support your POV, and dismiss the rest.
> One aspect you ignore is the fact that Semantic is where it is today
> because people who developed it stopped its development.  This is
> directly relevant to your proposal to develop parsing capabilities as
> part of Emacs -- the result in the long run will be the same, because
> no one stays in Emacs development forever.

> By contrast, using external implementations of technologies which are
> important to IDEs has a much higher chance of finding Free Software
> implementations that we can use, even if some of them become defunct.

My POV was shaped by something similar, except outside Emacs.  How many
people today still look to the PHIGS (and its Extension to X) for
three-dimensional graphics?  Or OpenGL 1.4?

Or what about Display PostScript, XIE, trapezoid rastering in the X
rendering extension, and quite soon Cairo?

Or what about DRI1 and XFree86-DRI for direct rendering?

Those were important implementations of technologies that were and are
directly or indirectly relevant for every GUI program in existence, and
seemed firmly entrenched when they were first created.  Yet now most of
them only exist in the memory of a select few people.

That is why it is normally desirable to implement something yourself (if
possible, unfortunately not in the case of PEX or GL 1.4), so even if
support fades and and development stalls, it can be left for 10 or 20
years and still work as satisfactorily as it used to.

Re: Call for volunteers: add tree-sitter support to major modes

[Daniel Martín]

Po Lu <luangruo@yahoo.com> writes:

>
> That is why it is normally desirable to implement something yourself (if
> possible, unfortunately not in the case of PEX or GL 1.4), so even if
> support fades and and development stalls, it can be left for 10 or 20
> years and still work as satisfactorily as it used to.

I understand your concern, but I don't think that potential problem will
affect us significantly if we are able to keep the Tree-sitter
integration separate enough from the Emacs core functionality.  If
development of Tree-sitter ever stalls, Emacs can still fallback to the
parse-partial-sexp-based approach, which will be functional and I expect
it will continue to receive updates from people that prefer it to
Tree-sitter.

I think we're too early to know about the implications of Tree-sitter
integration in detail, so my only suggestion is to keep the integration
easy to remove/replace (we can expect Emacs will outlive Tree-sitter).

Re: Call for volunteers: add tree-sitter support to major modes

[Jostein Kjønigsen]

> On 11 Oct 2022, at 09:46, Po Lu <luangruo@yahoo.com> wrote:
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> Yes, I will.
>> 
>> It isn't our business to develop language parsers, and we have no hope
>> of keeping up with the technology developments in that area and with
>> new languages being invented, well enough to offer reasonable and
>> modern support in Emacs for editing programs in those languages.
> 
> It isn't quite our business to develop mini-gmp either, yet it's also
> kept in-tree.
> 
>> It's the same reason why we depend on text shaping engines like
>> HarfBuzz for rendering complex scripts and other typography-related
>> features, on image libraries to display images, on GnuTLS to support
>> TLS connections, etc. etc.
> 
> But editing text mostly does not involve editing complex shaped text,
> and likewise, TLS connections and image display is mostly orthogonal to
> the business of editing text.
> 
>> We cannot possibly be experts in all of those fields, we don't have
>> the manpower for that.  In the case of support for editing programs we
>> actually tried for many years, and the result is before our eyes --
>> and it's unsatisfactory at best.  We should advance Emacs towards
>> leveraging existing technologies, not try developing those
>> technologies in-house, or replacing them by cheap hacks and kludges.
> 
> Then why is mini-gmp kept in-tree? Portable multiple precision
> arithmetic is at least as complicated (if not more) than tree-sitter,
> yet mini-gmp is a completely satisfactory solution for people who do not
> have libgmp installed.
> 
> mini-tree-sitter would fill the same role: it would expose the same API
> as tree-sitter, but be small enough for Emacs to include and use when
> tree-sitter is not available.
> 
>> It is completely impractical to expect us to be able to develop this
>> in-house.
> 
> Why?  The tree-sitter runtime library is only about 16,000 lines of C
> source code and headers, which is less than xdisp.c alone, and only
> slightly more than keyboard.c.  In addition, the path to the current
> implementation of tree-sitter is well-trodden, so it can't be that hard
> to walk down that path again.
> 
> Such an implementation would be able to make use of existing and new
> tree-sitter grammars without changes.
> 
>> If tree-sitter becomes defunct (something that doesn't seem like it
>> will happen any time soon)
> 
> Famous last words.

Absolutely true. But I don’t consider that a big enough problem to be worried. 

tree-sitter will just be one way to implement a major-mode. There’s still others. There can be even more. 

tree-sitter has momentum outside Emacs and high quality parsers for quite a lot of complex languages which are troublesome to implement parsers for in Elisp, as evident by our current best-effort parsers. 

While there’s no guarantee it will be maintained forever, like Emacs, it doesn’t have to either. 

It just needs to be maintained for as long as the languages whose parsers are implemented using it are maintained. 

Once a language stops developing, it shouldn’t be a problem for us that tree-sitter based parsers for it stops developing as well. 

If a new, better thing comes along ten years down the line, there’s nothing preventing us from adapting that for the languages and major-modes which at that time will benefit from it. 

Just like we’re doing with tree-sitter now, for the major-modes and languages with complex grammar which will benefits from it. 

Disclaimer: I may be biased here, because I’ve spent the last ten+ years trying to maintain parsers for C# and typescript in Elisp (and failing at that). And TSX has been a thorough no-go. 

For some of these languages, tree-sitter really is the only realistic option we have now if we want to support these languages at the level our users(and programmers in general) expect. 

I see little to be gained by being philosophical at this point when the reality is as clear and harsh as it is. 

> 
>> , we will have to find a suitable replacement -- like we did with
>> text-shaping engine, when the FLT library stopped being actively
>> developed.  Dealing with these changes in the outside world is an
>> important part of the job of the Emacs maintainers
> 
> Text shaping is, at least IME, immensely more complicated than
> interactive language parsing.
> 
>> The present font-lock and indentation features based on our own code
>> can remain in Emacs forever, if someone wants a safe fallback for when
>> worse comes to worst.
> 
> That logic could also have been applied to bignums, yet there was no
> objection to including mini-gmp.
> 
>> But these implementations are from my POV a dead end: they cannot be
>> developed any further, and for some languages, like C++, are already
>> all but failing to reasonably and efficiently support their complex
>> grammars.  Using expertise of others is simply a logical step based on
>> past experience; the decision to develop this in-house would be a step
>> back and a disaster in the long run, IMNSHO.
> 
> I'm not disputing that fact, which is why I proposed to write an
> *implementation* of tree-sitter for inclusion with Emacs, and not to
> develop an equivalent in-house.

Re: Call for volunteers: add tree-sitter support to major modes

[Dmitry Gutov]

On 11.10.2022 10:31, Po Lu wrote:
> I'm not disputing that fact, which is why I proposed to write an
> *implementation*  of tree-sitter for inclusion with Emacs, and not to
> develop an equivalent in-house.

If you're confident in your ability to do so, why no wait until it seems 
likely that tree-sitter is about to become defunct/bitrot/abandoned, 
before doing that?

At that point is will either become clear that reimplementing or 
maintaining it ourselves is a good idea (perhaps with Emacs specifics), 
or some reasonably similar competing technology arises, which we would 
be able to migrate to.

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Monnier]

Po Lu [2022-10-11 12:43:36] wrote:
> Theodor Thornhill <theo@thornhill.no> writes:
>> My suggestion is to add the tree-sitter variant in these cases, and let
>> the other modes die a slow, deprecated death down the line.
> In that case, will any of you object if someone writes a parser
> generator in C and an implementation of the tree-sitter runtime library
> that can be included with Emacs itself (namely, with the copyright
> assigned to the FSF?)

Beside Eli's objection, with which I completely agree, I can't see
any benefit at all.

> My opinion is that Emacs should not degrade its text editing
> capabilities so drastically if a non-system library (i.e. not ncurses
> or Xlib) is not present.

That's easy to fix: make tree-sitter a required dependency.
No need to maintain our own version of it.

If/when tree-sitter becomes unmaintained, we may decide to revisit this
choice, but I see no reason to cross this bridge in the foreseeable future.

We already reinvent many more wheels than we should.


        Stefan

Re: Call for volunteers: add tree-sitter support to major modes

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> That's easy to fix: make tree-sitter a required dependency.
> No need to maintain our own version of it.

I'd be for that, I think.

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Monnier]

Lars Ingebrigtsen [2022-10-11 23:49:25] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> That's easy to fix: make tree-sitter a required dependency.
>> No need to maintain our own version of it.
> I'd be for that, I think.

I don't see a need to do that for Emacs-29.  But I expect it will happen
sooner or later, yes.


        Stefan

Re: Call for volunteers: add tree-sitter support to major modes

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I don't see a need to do that for Emacs-29.  But I expect it will happen
> sooner or later, yes.

Yes, more of an Emacs 30 thing, so 2024-ish.

That is, during the next year, I'm guessing that we'll be gaining
tree-sitter support for all the major programming languages, and when
that is better than what we already have, we should consider making
tree-sitter a prerequisite for Emacs (and these modes) and rapidly phase
out the old code.

But that all depends on how things go and how rapidly the modes gain
tree-sitter support.  Just looking at what's been achieved over the last
week makes me hopeful, though.

It's also a question of how much user breakage we'll tolerate/aim for.
That is, there's a gazillion tuning knobs all over the modes to tweak
stuff that will be rendered obsolete by tree-sitter, and which people
may be fond of.

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Yes, more of an Emacs 30 thing, so 2024-ish.
>
> That is, during the next year, I'm guessing that we'll be gaining
> tree-sitter support for all the major programming languages, and when
> that is better than what we already have, we should consider making
> tree-sitter a prerequisite for Emacs (and these modes) and rapidly phase
> out the old code.

I and many others will _very_ strongly object to removing the build
without tree-sitter.  Emacs has never needed a non-system library to
edit text, and that should not start now.

On X, we even support the build without XCB, as even though it has been
ubitous for decades, it is not specified in any document and cannot be
implemented by third parties.  Unlike Xlib, which is part of the
specifications for release 6 of the X Window System, version 11,
published by the X Consortium.

> It's also a question of how much user breakage we'll tolerate/aim for.
> That is, there's a gazillion tuning knobs all over the modes to tweak
> stuff that will be rendered obsolete by tree-sitter, and which people
> may be fond of.

It will certainly break a lot.  Last I heard tree-sitter itself has
problems with complex macro constructs in C and C++ code.

CC Mode has less, because it is too dumb to try to parse them.

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Kangas]

Po Lu <luangruo@yahoo.com> writes:

> On X, we even support the build without XCB, as even though it has been
> ubitous for decades, it is not specified in any document and cannot be
> implemented by third parties.

Maybe we should not do that, then?

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Stefan Kangas <stefankangas@gmail.com> writes:

> Maybe we should not do that, then?

No.

The build without XCB provides the canonical implementation of the X
terminal interface, and has been helpful many times in the past in that
role.

Being standardized, Xlib is also guaranteed to exist, no matter how much
X changes in the future.  XCB has not proven itself nearly as well.

Re: Call for volunteers: add tree-sitter support to major modes

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Po Lu <luangruo@yahoo.com>,  Theodor Thornhill <theo@thornhill.no>,  Eli
>  Zaretskii <eliz@gnu.org>,  Alan Mackenzie <acm@muc.de>,
>  emacs-devel@gnu.org,  jostein@kjonigsen.net
> Date: Tue, 11 Oct 2022 18:00:21 -0400
> 
> Lars Ingebrigtsen [2022-10-11 23:49:25] wrote:
> > Stefan Monnier <monnier@iro.umontreal.ca> writes:
> >> That's easy to fix: make tree-sitter a required dependency.
> >> No need to maintain our own version of it.
> > I'd be for that, I think.
> 
> I don't see a need to do that for Emacs-29.  But I expect it will happen
> sooner or later, yes.

We will definitely NOT do that for Emacs 29.  It's too early for that
kind of commitment.

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> That's easy to fix: make tree-sitter a required dependency.
> No need to maintain our own version of it.
>
> If/when tree-sitter becomes unmaintained, we may decide to revisit this
> choice, but I see no reason to cross this bridge in the foreseeable future.

Once again, famous last words.

Emacs currently has no dependencies other than system libraries:
terminfo/termcap/curses.  As a result, it builds out of the box on AIX
with the XL C compiler.

It does not depend on gnutls, HarfBuzz, Cairo, or even libraries that
really are available everywhere such as XCB.  Why require tree-sitter?

Re: Call for volunteers: add tree-sitter support to major modes

[João Paulo Labegalini de Carvalho]

[-- Attachment #1: Type: text/plain, Size: 749 bytes --]

On Tue., Oct. 11, 2022, 6:26 p.m. Po Lu, <luangruo@yahoo.com> wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> > That's easy to fix: make tree-sitter a required dependency.
> > No need to maintain our own version of it.
> >
> > If/when tree-sitter becomes unmaintained, we may decide to revisit this
> > choice, but I see no reason to cross this bridge in the foreseeable
> future.
>
> Once again, famous last words.
>
> Emacs currently has no dependencies other than system libraries:
> terminfo/termcap/curses.  As a result, it builds out of the box on AIX
> with the XL C compiler.


Just curious. Is it important more important to keep emacs easy to build on
a closed-source system or potentially improve/modernize its internals?

[-- Attachment #2: Type: text/html, Size: 1224 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com> writes:

> Just curious. Is it important more important to keep emacs easy to
> build on a closed-source system or potentially improve/modernize its
> internals?

Making optional dependencies required does not "potentially improve" or
"modernize" its internals.

On the contrary, it often deletes code that may be of value for
reference in the future.

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Kangas]

Po Lu <luangruo@yahoo.com> writes:

> On the contrary, it often deletes code that may be of value for
> reference in the future.

Isn't that why we use a VCS?

Re: Call for volunteers: add tree-sitter support to major modes

[Po Lu]

Stefan Kangas <stefankangas@gmail.com> writes:

> Isn't that why we use a VCS?

A VCS doesn't magically update past code that has been removed.

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Monnier]

> My suggestion is to add the tree-sitter variant in these cases, and let
> the other modes die a slow, deprecated death down the line.

I generally agree with this plan, but note that packages tend not to die
as conveniently as one might like, so I'd urge you to make them die an
orderly death, by actively redirecting users (and potential (confused?)
new maintainers) to the new code, and generally helping
transition/integration.


        Stefan

Re: Call for volunteers: add tree-sitter support to major modes

[Roman Rudakov]

Theodor Thornhill <theo@thornhill.no> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>
>> Some of the popular major modes for which we would like to see
>> tree-sitter support in Emacs 29 are:
>>
>>  c-mode
>>  c++-mode
>>  emacs-lisp-mode
>>  js-mode
>>  js-json-mode
>>  ruby-mode
>>  shell-script-mode
>>
>
> I'd like to add in Java support here as well.  But since java is part 
> of
> cc-mode.el, maybe it'd be best if it got its own mode, say
> java-treesitter-mode.el, so that we don't add a dependency into 
> cc-mode?
> If you agree, I can create such a major-mode as well.
>
> Also, how should we mandate what modes get the new tree-sitter 
> support?
> It is pretty fast to add new modes, and adding support for languages 
> not
> already supported in core, such as say, golang, should be very 
> feasible.
> But maybe that is something that should live in elpa?
>
> Theo

Hi! I had an idea to work on tree-sitter support for Java, but I don't
have a lot of experience with Emacs-lisp and I haven't signed a
copyright assignment. Though I used emacs for java development for a
while with tree-sitter and I have some queries in my config which might
be useful.

https://git.sr.ht/~rrudakov/dotfiles/tree/master/item/emacs.d/emacs.org#L2453-2623

Feel free to use it.
--
Best regards, Roman

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Roman Rudakov <rrudakov@fastmail.com> writes:

> Theodor Thornhill <theo@thornhill.no> writes:
>

[...]

> Hi! I had an idea to work on tree-sitter support for Java, but I don't
> have a lot of experience with Emacs-lisp and I haven't signed a
> copyright assignment. Though I used emacs for java development for a
> while with tree-sitter and I have some queries in my config which might
> be useful.
>
> https://git.sr.ht/~rrudakov/dotfiles/tree/master/item/emacs.d/emacs.org#L2453-2623
>


Thanks! Will do, have you done the copyright assignments, though?  Just
so I know whether I can steal you stuff ;)

Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Roman Rudakov]

Theodor Thornhill <theo@thornhill.no> writes:

> Roman Rudakov <rrudakov@fastmail.com> writes:
>
>> Theodor Thornhill <theo@thornhill.no> writes:
>>
>
> [...]
>
>> Hi! I had an idea to work on tree-sitter support for Java, but I 
>> don't
>> have a lot of experience with Emacs-lisp and I haven't signed a
>> copyright assignment. Though I used emacs for java development for a
>> while with tree-sitter and I have some queries in my config which 
>> might
>> be useful.
>>
>> https://git.sr.ht/~rrudakov/dotfiles/tree/master/item/emacs.d/emacs.org#L2453-2623
>>
>
>
> Thanks! Will do, have you done the copyright assignments, though? 
> Just
> so I know whether I can steal you stuff ;)
>
> Theo

I haven't. I'll start the process shortly.
--
Best regards, Roman

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

>>
>> Theo
>
> I haven't. I'll start the process shortly.
> --

Nice :)

Re: Call for volunteers: add tree-sitter support to major modes

[Roman Rudakov]

[-- Attachment #1: Type: text/plain, Size: 782 bytes --]


Theodor Thornhill <theo@thornhill.no> writes:

> Roman Rudakov <rrudakov@fastmail.com> writes:
>
>> Theodor Thornhill <theo@thornhill.no> writes:
>>
>
> [...]
>
>> Hi! I had an idea to work on tree-sitter support for Java, but I 
>> don't
>> have a lot of experience with Emacs-lisp and I haven't signed a
>> copyright assignment. Though I used emacs for java development for 
>> a
>> while with tree-sitter and I have some queries in my config which 
>> might
>> be useful.
>>
>> https://git.sr.ht/~rrudakov/dotfiles/tree/master/item/emacs.d/emacs.org#L2453-2623
>>
>
>
> Thanks! Will do, have you done the copyright assignments, though? 
> Just
> so I know whether I can steal you stuff ;)
>
> Theo

Hello Theo!

Just FYI, my copyright assignment is done.
--
Best regards, Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: Call for volunteers: add tree-sitter support to major modes

[Theodor Thornhill]

Roman Rudakov <rrudakov@fastmail.com> writes:

> Theodor Thornhill <theo@thornhill.no> writes:
>
>> Roman Rudakov <rrudakov@fastmail.com> writes:
>>
>>> Theodor Thornhill <theo@thornhill.no> writes:
>>>

[...]

>
> Hello Theo!
>
> Just FYI, my copyright assignment is done.

Nice - I'll look into Java and c# after Alan has voiced his opinion on
the CC Mode/tree-sitter situation!

Thanks,
Theo

Re: Call for volunteers: add tree-sitter support to major modes

[Stefan Monnier]

> Just FYI, my copyright assignment is done.

Wow!  That was fast!


        Stefan