Maybe we're taking a wrong approach towards tree-sitter

Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

I could not follow the conversation <<cc-mode fontification feels
random>> particularly well, as it seemed somehow disjoint in a manner I
cannot explain, but it seemed as if consensus has been reached that
Emacs will provide optional functionality integrating yet another
external package, this time tree-sitter.

Unlike features like native JSON, however, I believe tree-sitter is the
first optional package providing notable functionality that would
require a toolchain that depends on LLVM (that of Rust, which
tree-sitter is implemented in), and is therefore inaccessible to people
not running popular systems; I.E., how would one make tree-sitter work
in MS-DOS (Emacs on FreeDOS is a must-have for me, and it would be a
great annoyance if cc-mode, or similar external packages depend on
tree-sitter in the future), or on an Itanium system running GNU/Linux?

I think we should focus on portably reimplementing the relevant
functionality within Emacs, preferably in Lisp.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Stephen Leake]

Andrei Kuznetsov <r12451428287@163.com> writes:

> it seemed as if consensus has been reached that
> Emacs will provide optional functionality integrating yet another
> external package, this time tree-sitter.
>
> Unlike features like native JSON, however, I believe tree-sitter is the
> first optional package providing notable functionality that would
> require a toolchain that depends on LLVM (that of Rust, which
> tree-sitter is implemented in), and is therefore inaccessible to people
> not running popular systems; 

The tree-sitter runtime, that Emacs would link with, it implemented in
C, partly for this reason. It would be compiled with whatever Emacs is
compiled with, or the system compiler.

Some of the tree-sitter development tools are implemented in Rust; you
only need Rust if you are developing/fixing a grammar for a language.

-- 
-- Stephe

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Manuel Giraud]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

[...]

> The tree-sitter runtime, that Emacs would link with, it implemented in
> C, partly for this reason. It would be compiled with whatever Emacs is
> compiled with, or the system compiler.
>
> Some of the tree-sitter development tools are implemented in Rust; you
> only need Rust if you are developing/fixing a grammar for a language.

Hi,

I too did not follow the tree-sitter discussion closely. But AFAIU,
tree-sitter provides tools to generate a parser (in C) from a grammar.

So, is it the generated parsers (for any language Emacs supports) that
will be versionned into the emacs tree?
-- 
Manuel Giraud

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

Manuel Giraud <manuel@ledu-giraud.fr> writes:

> I too did not follow the tree-sitter discussion closely. But AFAIU,
> tree-sitter provides tools to generate a parser (in C) from a grammar.

If that is the case, it certainly seems grave! I don't think an Emacs
that requires source modifications for extending vital editing
functionality is a good idea.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> From: Andrei Kuznetsov <r12451428287@163.com>
> Date: Wed, 28 Jul 2021 19:48:18 +0800
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, emacs-devel@gnu.org
> 
> Manuel Giraud <manuel@ledu-giraud.fr> writes:
> 
> > I too did not follow the tree-sitter discussion closely. But AFAIU,
> > tree-sitter provides tools to generate a parser (in C) from a grammar.
> 
> If that is the case, it certainly seems grave! I don't think an Emacs
> that requires source modifications for extending vital editing
> functionality is a good idea.

TS's code is written in plain C, and doesn't require any regeneration
or source modifications.  Anything else is misunderstanding.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

Eli Zaretskii <eliz@gnu.org> writes:

> TS's code is written in plain C, and doesn't require any regeneration
> or source modifications.  Anything else is misunderstanding.

I am confused by TS's documentation, but if my understanding is correct,
shouldn't it be a parser generator that generates C code?  In that case,
how would users load new parsers or modify existing ones?  Perhaps
through something similar to the existing native module support?

I might be making a grave misunderstanding (or several)

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> From: Andrei Kuznetsov <r12451428287@163.com>
> Cc: manuel@ledu-giraud.fr,  stephen_leake@stephe-leake.org,
>   emacs-devel@gnu.org
> Date: Wed, 28 Jul 2021 21:14:48 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > TS's code is written in plain C, and doesn't require any regeneration
> > or source modifications.  Anything else is misunderstanding.
> 
> I am confused by TS's documentation, but if my understanding is correct,
> shouldn't it be a parser generator that generates C code?

TS is not a parser generator, it's a parser that accepts the language
grammar from external files.

> In that case, how would users load new parsers or modify existing
> ones?

If you want to modify a TS grammar file, you can (not in C).  But why
would you want to?  The whole point of using TS is NOT to require that
the Emacs development team or Emacs users should know enough about
parsing of the many languages Emacs supports to modify the grammar.
We want another, independent development team to take care of that,
and we want to use the results of their development with minimum
fuss.  Exactly like we do with other libraries developed by other
projects: the image libraries, GnuTLS, HarfBuzz, etc.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

Eli Zaretskii <eliz@gnu.org> writes:

> If you want to modify a TS grammar file, you can (not in C).  But why
> would you want to?  The whole point of using TS is NOT to require that
> the Emacs development team or Emacs users should know enough about
> parsing of the many languages Emacs supports to modify the grammar.
> We want another, independent development team to take care of that,
> and we want to use the results of their development with minimum
> fuss.  Exactly like we do with other libraries developed by other
> projects: the image libraries, GnuTLS, HarfBuzz, etc.

Interesting perspective.  Thanks for the clarification

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Dmitry Gutov]

On 28.07.2021 16:27, Eli Zaretskii wrote:
> The whole point of using TS is NOT to require that
> the Emacs development team or Emacs users should know enough about
> parsing of the many languages Emacs supports to modify the grammar.
> We want another, independent development team to take care of that,

I think we know both, though? There are a number of niche languages that 
only Emacs supports.

Or at least that aren't likely to get good support in TreeSitter.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Dmitry Gutov]

Sorry,

On 28.07.2021 17:24, Dmitry Gutov wrote:
> I think we know both, though? There are a number of niche languages that 
              ^ want

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Daniele Nicolodi]

On 28/07/2021 16:24, Dmitry Gutov wrote:
> On 28.07.2021 16:27, Eli Zaretskii wrote:
>> The whole point of using TS is NOT to require that
>> the Emacs development team or Emacs users should know enough about
>> parsing of the many languages Emacs supports to modify the grammar.
>> We want another, independent development team to take care of that,
> 
> I think we know both, though? There are a number of niche languages that 
> only Emacs supports.
> 
> Or at least that aren't likely to get good support in TreeSitter.

I don't see how adding support for TreeSitter can cause any problem to
those. Would you like to elaborate? No one is proposing to disable other
mechanism for fontification and syntax analysis in Emacs.

Cheers,
Dan

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> Cc: stephen_leake@stephe-leake.org, manuel@ledu-giraud.fr, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 28 Jul 2021 17:24:35 +0300
> 
> On 28.07.2021 16:27, Eli Zaretskii wrote:
> > The whole point of using TS is NOT to require that
> > the Emacs development team or Emacs users should know enough about
> > parsing of the many languages Emacs supports to modify the grammar.
> > We want another, independent development team to take care of that,
> 
> I think we know both, though? There are a number of niche languages that 
> only Emacs supports.
> 
> Or at least that aren't likely to get good support in TreeSitter.

Either someone motivated will write a TS grammar for them, or they
will continue be supported by "other means".

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Perry E. Metzger]

On 7/28/21 12:10, Eli Zaretskii wrote:
>> There are a number of niche languages that only Emacs supports.
>> Or at least that aren't likely to get good support in TreeSitter.
> Either someone motivated will write a TS grammar for them, or they
> will continue be supported by "other means".
>
It would be nice, of course, if people would contribute new grammars to 
Tree Sitter, as that will benefit everyone using a tool that works with 
Tree Sitter.

Perry

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> Date: Wed, 28 Jul 2021 12:24:54 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> 
> On 7/28/21 12:10, Eli Zaretskii wrote:
> >> There are a number of niche languages that only Emacs supports.
> >> Or at least that aren't likely to get good support in TreeSitter.
> > Either someone motivated will write a TS grammar for them, or they
> > will continue be supported by "other means".
> >
> It would be nice, of course, if people would contribute new grammars to 
> Tree Sitter, as that will benefit everyone using a tool that works with 
> Tree Sitter.

Yes, it would be nice.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrei Kuznetsov <r12451428287@163.com>
>> Date: Wed, 28 Jul 2021 19:48:18 +0800
>> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, emacs-devel@gnu.org
>> 
>> Manuel Giraud <manuel@ledu-giraud.fr> writes:
>> 
>> > I too did not follow the tree-sitter discussion closely. But AFAIU,
>> > tree-sitter provides tools to generate a parser (in C) from a grammar.
>> 
>> If that is the case, it certainly seems grave! I don't think an Emacs
>> that requires source modifications for extending vital editing
>> functionality is a good idea.
>
> TS's code is written in plain C, and doesn't require any regeneration
> or source modifications.  Anything else is misunderstanding.

That's true for the common TS runtime, which implements the parser and
error recovery, but the code for each language, that builds the LR parse
table and some other data structures, is generated in C from a grammar
file written in javascript, and must be linked into Emacs somehow. In
addition, some languages require an "external scanner", which is more
code in C that is specific to the language.

Ideally, there would be some sort of plugin, so new languages could be
added at run-time; maybe we could add a protocol on top of emacs
modules. I don't know how Yuan is handling this now.

-- 
-- Stephe

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Yuan Fu]

> On Jul 29, 2021, at 7:12 PM, Stephen Leake <stephen_leake@stephe-leake.org> wrote:
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> From: Andrei Kuznetsov <r12451428287@163.com>
>>> Date: Wed, 28 Jul 2021 19:48:18 +0800
>>> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, emacs-devel@gnu.org
>>> 
>>> Manuel Giraud <manuel@ledu-giraud.fr> writes:
>>> 
>>>> I too did not follow the tree-sitter discussion closely. But AFAIU,
>>>> tree-sitter provides tools to generate a parser (in C) from a grammar.
>>> 
>>> If that is the case, it certainly seems grave! I don't think an Emacs
>>> that requires source modifications for extending vital editing
>>> functionality is a good idea.
>> 
>> TS's code is written in plain C, and doesn't require any regeneration
>> or source modifications.  Anything else is misunderstanding.
> 
> That's true for the common TS runtime, which implements the parser and
> error recovery, but the code for each language, that builds the LR parse
> table and some other data structures, is generated in C from a grammar
> file written in javascript, and must be linked into Emacs somehow.

Languages don’t need to be linked into Emacs. They can be in dynamic modules.

Yuan

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Stephen Leake]

Yuan Fu <casouri@gmail.com> writes:

>> On Jul 29, 2021, at 7:12 PM, Stephen Leake <stephen_leake@stephe-leake.org> wrote:
>> That's true for the common TS runtime, which implements the parser and
>> error recovery, but the code for each language, that builds the LR parse
>> table and some other data structures, is generated in C from a grammar
>> file written in javascript, and must be linked into Emacs somehow.
>
> Languages don’t need to be linked into Emacs. They can be in dynamic
> modules.

Dynamic modules are linked, at run-time. That's how the code that calls
them knows what addresses to call.

So I think you are saying the tree-sitter runtime will be 
linked into Emacs at emacs compile time, while the languages can be
linked in at run-time. That's good.

-- 
-- Stephe

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> That's true for the common TS runtime, which implements the parser and
> error recovery, but the code for each language, that builds the LR parse
> table and some other data structures, is generated in C from a grammar
> file written in javascript, and must be linked into Emacs somehow. In
> addition, some languages require an "external scanner", which is more
> code in C that is specific to the language.

Interesting.  I assume it would be possible to reuse the source grammar
files?

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Arthur Miller]

Andrei Kuznetsov <r12451428287@163.com> writes:

 Leake <stephen_leake@stephe-leake.org> writes:
>
>> That's true for the common TS runtime, which implements the parser and
>> error recovery, but the code for each language, that builds the LR parse
>> table and some other data structures, is generated in C from a grammar
>> file written in javascript, and must be linked into Emacs somehow. In
>> addition, some languages require an "external scanner", which is more
>> code in C that is specific to the language.
>
> Interesting.  I assume it would be possible to reuse the source grammar
> files?

It probably is, and looking at neowim's gh repo, there are some
instructions on how to create a grammar for new language:

https://github.com/nvim-treesitter/nvim-treesitter

The process could probably be somehow automated from lisp.

I have though a sincere question about this entire tree-sitter
venture. Is it really worth trouble in Emacs case? As I understand TS it
is a specialized regex matcher, and looking at some language specs leave
me with that feeling (for example the grammar for bash):

https://github.com/tree-sitter/tree-sitter-bash/blob/master/src/grammar.json

I undestand that having specialized regex matcher is more efficient than
some generalized regular matcher current font-locking in Emacs relies
upon, but is it *that* more efficient to be worth the extra troubles?
TS seem to keep state (a node) for each character typed, that will be a
lot of memory consumed in some big files. If this syntax tree it keeps
to implement what it does can be re-used for something else than it
could be very useful, but just for syntax-highlight and indentation?
Some years ago, when opening some 10k lines as found in Emacs src dir, I
noticed some slowdown on font lock. But nowadays I don't experience any
hickups with syntax hightlighting or indentation.

Anyway, it is very educating to see TS get merged into Emacs and to read
Eli's tips and guidance about Emacs internals.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Óscar Fuentes]

Arthur Miller <arthur.miller@live.com> writes:

> I undestand that having specialized regex matcher is more efficient than
> some generalized regular matcher current font-locking in Emacs relies
> upon, but is it *that* more efficient to be worth the extra troubles?

AFAIU this is not about efficience, but mainly about correctness (modern
languages are increasingly more difficult to analyze) and also about
decreasing the maintenance load. In the process, Emacs gets support for
some new languages too.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Arthur Miller]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> I undestand that having specialized regex matcher is more efficient than
>> some generalized regular matcher current font-locking in Emacs relies
>> upon, but is it *that* more efficient to be worth the extra troubles?
>
> AFAIU this is not about efficience, but mainly about correctness (modern
> languages are increasingly more difficult to analyze)

Ok, I understand, and I can buy that one. Question is if it is still
worth just for the syntax hightlight and indentation? If I get some
spurious color here or there sometimes not colored, do I care?

Can that syntax tree of TS be exposed to lisp and used for some other
purposes, or is it just internal to TS and only output we see is some
colors on the screen?

>                                                       and also about
> decreasing the maintenance load.
Sure, but it is also a limitation. If Emacs will rely on TS maintainers
to create new grammars and update existing ones when language changes,
it means Emacs users will have to wait for changes until they are
fixed upstream, similar as how gnu/linux distros work regarding
packaging. Of course, a user who wish to modify or introduce new
language can always rely on old font-lock or go through pain of TS
toolilng based on JS and custom tools. Lisp frontend to that toolchain
can probably be developed but that is even more work.

>                                  In the process, Emacs gets support for
> some new languages too.

Yes, it is always nice I guess :). Is there really demand for some
language currently provided in TS and not in Emacs?

I don't know, I am maybe overly sceptical to TS; I don't mean it is a
bad package, and I am sure it has it's place in other editors, I am just
not sure how it fits in Emacs where everything is easily configurable
and extensible.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Ergus]

On Fri, Jul 30, 2021 at 03:30:42PM +0200, Arthur Miller wrote:
>�scar Fuentes <ofv@wanadoo.es> writes:
>
>> Arthur Miller <arthur.miller@live.com> writes:
>>
>>> I undestand that having specialized regex matcher is more efficient than
>>> some generalized regular matcher current font-locking in Emacs relies
>>> upon, but is it *that* more efficient to be worth the extra troubles?
>>
>> AFAIU this is not about efficience, but mainly about correctness (modern
>> languages are increasingly more difficult to analyze)
>
>Ok, I understand, and I can buy that one. Question is if it is still
>worth just for the syntax hightlight and indentation? If I get some
>spurious color here or there sometimes not colored, do I care?
>

Yes, we care. Syntax highlight for an editor is a basic feature in 2021.

>Can that syntax tree of TS be exposed to lisp and used for some other
>purposes,

This is the idea. use the tree for navigations like up-list or
goto-defun for example. Maybe not the tree directly, but the information
it provides (maybe calling TS function wrappers or setting the TS
information as text properties).

>or is it just internal to TS and only output we see is some
>colors on the screen?
>
How we use it is more a design choice. We can access the tree
information with the TS api or we can just put the tree's information as
text properties... imagination is the limit ;)

>>                                                       and also about
>> decreasing the maintenance load.
>Sure, but it is also a limitation. If Emacs will rely on TS maintainers
>to create new grammars and update existing ones when language changes,
>it means Emacs users will have to wait for changes until they are
>fixed upstream, similar as how gnu/linux distros work regarding
>packaging. Of course, a user who wish to modify or introduce new
>language can always rely on old font-lock or go through pain of TS
>toolilng based on JS and custom tools. Lisp frontend to that toolchain
>can probably be developed but that is even more work.
>
Sincerely; create a grammar for TS is much simpler than create a mode
with font-lock, navigation commands, indentation rules and some
flymake. All the modes with TS will be a bit more consistent in colors
and keybindings (now we have modes where all commands use different
prefixes, or lacking navigation or with different indentation
customs. So using them is like learning different editors for every
language)

>>                                  In the process, Emacs gets support for
>> some new languages too.
>
>Yes, it is always nice I guess :). Is there really demand for some
>language currently provided in TS and not in Emacs?
>

Indeed. As I mentioned before web developers are using VScode or neovim
because Angular, React, Nodejs and Python are painfully supported
(compared to VScode or Sublime). Rust is very limited supported in
emacs, so users rely on external packages like rust-mode, elpy or
anaconda that introduce different bindings, collisions and require some
complex setups for the basics.

>I don't know, I am maybe overly sceptical to TS; I don't mean it is a
>bad package, and I am sure it has it's place in other editors, I am just
>not sure how it fits in Emacs where everything is easily configurable
>and extensible.
>
It is just a good trade-off configurable enough for 99% of the use
cases. Unless we expect all the users to be advanced lisp hackers to
customize their fontlocking, indentation and navigation functions for
every single prog-mode.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Arthur Miller]

Ergus <spacibba@aol.com> writes:

> On Fri, Jul 30, 2021 at 03:30:42PM +0200, Arthur Miller wrote:
>>�scar Fuentes <ofv@wanadoo.es> writes:
>>
>>> Arthur Miller <arthur.miller@live.com> writes:
>>>
>>>> I undestand that having specialized regex matcher is more efficient than
>>>> some generalized regular matcher current font-locking in Emacs relies
>>>> upon, but is it *that* more efficient to be worth the extra troubles?
>>>
>>> AFAIU this is not about efficience, but mainly about correctness (modern
>>> languages are increasingly more difficult to analyze)
>>
>>Ok, I understand, and I can buy that one. Question is if it is still
>>worth just for the syntax hightlight and indentation? If I get some
>>spurious color here or there sometimes not colored, do I care?
>>
>
> Yes, we care. Syntax highlight for an editor is a basic feature in 2021.
Of course, but I didn't meant Emacs should be without one, wtf, it's not
all or nothing :). What I said is do I really care if a file of 10k
source lines has a word here or there not highlighted, which I haven't
noticed with current implementation either.

>>Can that syntax tree of TS be exposed to lisp and used for some other
>>purposes,
>
> This is the idea. use the tree for navigations like up-list or
> goto-defun for example. Maybe not the tree directly, but the information
> it provides (maybe calling TS function wrappers or setting the TS
> information as text properties).
Ok, that might be useful.

> Indeed. As I mentioned before web developers are using VScode or neovim
> because Angular, React, Nodejs and Python are painfully supported
> (compared to VScode or Sublime). Rust is very limited supported in
> emacs, so users rely on external packages like rust-mode, elpy or
> anaconda that introduce different bindings, collisions and require some
> complex setups for the basics.
Don't we rely on external packages for lots of things. Almost all of
external packages you mentioned provide more than just syntax highlight,
and indentation, so we will probably continue to use those for other
reasons  even wen TS enters Emacs.

>        Unless we expect all the users to be advanced lisp hackers to
> customize their fontlocking, indentation and navigation functions for
> every single prog-mode.

Is it considered advanced lisp hackery to add extra keywords to
font-lock in their init file? I always think of myself as an elisp
noob. Thanks for boosting my ego :-).

Don't take me wrong, I mean nothing bad, I just find answers a tad bit
too extreme for my taste, but thanks for the input, it is interesting read.

I guess I'll be less sceptical and see what TS brings, anyway, thanks
for the all the work to all of you who work on it.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Date: Fri, 30 Jul 2021 15:30:42 +0200
> Cc: emacs-devel@gnu.org
> 
> >                                                       and also about
> > decreasing the maintenance load.
> Sure, but it is also a limitation. If Emacs will rely on TS maintainers
> to create new grammars and update existing ones when language changes,
> it means Emacs users will have to wait for changes until they are
> fixed upstream, similar as how gnu/linux distros work regarding
> packaging.

We have the same "problem" with every other library we use: the image
libraries, GnuTLS, HarfBuzz, etc.

Besides, TS is used by quite a few projects, so how long do you think
it will take for serious problems in language support to be fixed?

OTOH, take a look at some places in Emacs that don't have active
maintainers: problems there sometimes take forever to fix.  This is
what happens when a project wants to control everything in its domain,
but lacks manpower for doing so.

It is not reasonable to expect Emacs to have experts on board for
parsing every language on the face of Earth.  It won't work.

> I don't know, I am maybe overly sceptical to TS; I don't mean it is a
> bad package, and I am sure it has it's place in other editors, I am just
> not sure how it fits in Emacs where everything is easily configurable
> and extensible.

Not everything.  Again, take the other optional libraries we use as
examples: they cannot be extended inside Emacs.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Date: Fri, 30 Jul 2021 15:30:42 +0200
>> Cc: emacs-devel@gnu.org
>> 
>> >                                                       and also about
>> > decreasing the maintenance load.
>> Sure, but it is also a limitation. If Emacs will rely on TS maintainers
>> to create new grammars and update existing ones when language changes,
>> it means Emacs users will have to wait for changes until they are
>> fixed upstream, similar as how gnu/linux distros work regarding
>> packaging.
>
> We have the same "problem" with every other library we use: the image
> libraries, GnuTLS, HarfBuzz, etc.
>
> Besides, TS is used by quite a few projects, so how long do you think
> it will take for serious problems in language support to be fixed?

Yes, I understand that, I didn't meant so much problems as general
configurability after pesonal preferences and extendability. That is
what people seem to praise on Reddit when it comes to Emacs.

> OTOH, take a look at some places in Emacs that don't have active
> maintainers: problems there sometimes take forever to fix.  This is
> what happens when a project wants to control everything in its domain,
> but lacks manpower for doing so.
>
> It is not reasonable to expect Emacs to have experts on board for
> parsing every language on the face of Earth.  It won't work.
>
>> I don't know, I am maybe overly sceptical to TS; I don't mean it is a
>> bad package, and I am sure it has it's place in other editors, I am just
>> not sure how it fits in Emacs where everything is easily configurable
>> and extensible.
>
> Not everything.  Again, take the other optional libraries we use as
> examples: they cannot be extended inside Emacs.


TS is a bit special library since it hooks into a part of Emacs that
people do extend often but I do understand it is just a library that
adds some extra value like the others. I guess you are correct about its
popularity among other projects, that might do work for Emacs indeed. As
Ergus pointed out it will bring lots out of the box to many people, so I
guess it is a win at least in that section.

Thanks for the answer.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Ergus]

On Fri, Jul 30, 2021 at 02:06:00PM +0200, Arthur Miller wrote:
>Andrei Kuznetsov <r12451428287@163.com> writes:
>
> Leake <stephen_leake@stephe-leake.org> writes:
>>
>>> That's true for the common TS runtime, which implements the parser and
>>> error recovery, but the code for each language, that builds the LR parse
>>> table and some other data structures, is generated in C from a grammar
>>> file written in javascript, and must be linked into Emacs somehow. In
>>> addition, some languages require an "external scanner", which is more
>>> code in C that is specific to the language.
>>
>> Interesting.  I assume it would be possible to reuse the source grammar
>> files?
>
>It probably is, and looking at neowim's gh repo, there are some
>instructions on how to create a grammar for new language:
>
>https://github.com/nvim-treesitter/nvim-treesitter
>
>The process could probably be somehow automated from lisp.
>
>I have though a sincere question about this entire tree-sitter
>venture. Is it really worth trouble in Emacs case? As I understand TS it
>is a specialized regex matcher, and looking at some language specs leave
>me with that feeling (for example the grammar for bash):
>
>https://github.com/tree-sitter/tree-sitter-bash/blob/master/src/grammar.json
>
>I undestand that having specialized regex matcher is more efficient than
>some generalized regular matcher current font-locking in Emacs relies
>upon, but is it *that* more efficient to be worth the extra troubles?
>TS seem to keep state (a node) for each character typed, that will be a
>lot of memory consumed in some big files. If this syntax tree it keeps
>to implement what it does can be re-used for something else than it
>could be very useful, but just for syntax-highlight and indentation?
>Some years ago, when opening some 10k lines as found in Emacs src dir, I
>noticed some slowdown on font lock. But nowadays I don't experience any
>hickups with syntax hightlighting or indentation.
>
>Anyway, it is very educating to see TS get merged into Emacs and to read
>Eli's tips and guidance about Emacs internals.
>
The TS thing came out due to some issues in the c-mode highlighting
reported in that thread: correctness and speed (slowing down things like
scrolling). c-mode does its best, but C++ is evolving and more complex
analysis comes with a penalty and more and more code complexity in the
parser. Same happens with new languages very extended.

It will be very difficult to implement a complete/competitive mode like
c-mode for all the new languages that are very popular today (rust,
typescript; even python). So we end having some "weak" modes with
inconsistencies and different bindings and color themes. Those become
unmaintained after a time because the developers migrate to more
complete editors/ide and new developers just don't come to emacs because
it does not satisfy their needs to start with.

  Probably I am wrong but 99% of the web developers (React, Nodejs,
Angular) are using VSCode, the rest are with neovim; so we don't even
have people with enough knowledge and motivation to implement one of
those in Emacs one by one.

Because these languages are more complex to analyze and because we don't
have people to maintain a mode for all of them. Trying to do so will
spend too much developer time reinventing what TS already does (and does
it right, efficiently and with a support community).

So; maintaining a mode for every language we currently don't support is
not scalable over time. And reimplementing a replacement for TS in Elisp
won't worth it and will end up being very slow and repeating all the
errors that TS developers have already solved. TS may be useful not only
for syntax highlight and indentation but also for code navigation and
some basic syntax checking.

Basically TS is: One "infrastructure" to rule them all.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Arthur Miller]

Ergus <spacibba@aol.com> writes:
>  Probably I am wrong but 99% of the web developers (React, Nodejs,
> Angular) are using VSCode, the rest are with neovim; so we don't even
> have people with enough knowledge and motivation to implement one of
> those in Emacs one by one.
That might be for other reasons as well, like interaction modell,
wording and other idiosyncrasies of Emacs as discussed in numerous
threads about making Emacs popular, because of certain company is
backing VSCode etc. There are other editors like Adobe's Brackets which
came before VSCode and is by far less popular than VSCode. Looking at
recent MS business moves (AI, Github, copilot ...), it is now
understandable why they pour resources into a free code editor. I
wondered how come when they first released it, now the picture clears. I
don't think Emacs or barely some other editor can compete with MS,
simply nobody has so much resource. That is of course not an argument
for or against TS, just a thought about people prefereing a tool. Yes, I
agree with you that syntax highlight out of the box for a certain
library like Node oor Vue might help Emacs. I have nothing against that
argument.

>                                                TS may be useful not only
> for syntax highlight and indentation but also for code navigation and
> some basic syntax checking.
Yes, that would be a nice thing if it could be used for more than just
syntax and indentation.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Perry E. Metzger]

On 7/30/21 08:06, Arthur Miller wrote:
> I undestand that having specialized regex matcher is more efficient than
> some generalized regular matcher current font-locking in Emacs relies
> upon, but is it *that* more efficient to be worth the extra troubles?

It is not a question of efficiency. You cannot parse a context free 
grammar using regular expressions. The reason that almost all our 
highlight modes produce random garbage throughout is that you cannot 
parse a context free grammar using regular expressions. (For many 
languages, correctness isn't just occasionally violated, it's generally 
violated.)

Reliable highlighting regardless of code formatting, reliable 
indentation assistance, reliable code folding, and other such features 
require that the editor be able to both parse the program being edited 
_and_ that the editor be able to incrementally re-parse it as it changes 
in minimal time.

Other editors now have such features and make good use of them. Highly 
reliable code folding alone is worth the price of admission IMHO. 
(Currently, the best we can do for code folding is assume that the 
indentation is correct.)

LSP has been revolutionary in improving the programmer's experience in 
Emacs. Tree Sitter will provide significant additional improvement.

> TS seem to keep state (a node) for each character typed, that will be a
> lot of memory consumed in some big files.

No one will be forced to turn it on. I, however, almost certainly will. 
My productivity is more important to me than my RAM budget. Those that 
don't like it, though, won't have to pay the RAM tax.

Perry

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Stephen Leake]

Andrei Kuznetsov <r12451428287@163.com> writes:

> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> That's true for the common TS runtime, which implements the parser and
>> error recovery, but the code for each language, that builds the LR parse
>> table and some other data structures, is generated in C from a grammar
>> file written in javascript, and must be linked into Emacs somehow. In
>> addition, some languages require an "external scanner", which is more
>> code in C that is specific to the language.
>
> Interesting.  I assume it would be possible to reuse the source grammar
> files?

If they are licensed as free software, yes, of course.

What sort of reuse do you have in mind?

-- 
-- Stephe

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: Andrei Kuznetsov <r12451428287@163.com>,  manuel@ledu-giraud.fr,
>   emacs-devel@gnu.org
> Date: Thu, 29 Jul 2021 16:12:56 -0700
> 
> > TS's code is written in plain C, and doesn't require any regeneration
> > or source modifications.  Anything else is misunderstanding.
> 
> That's true for the common TS runtime, which implements the parser and
> error recovery, but the code for each language, that builds the LR parse
> table and some other data structures, is generated in C from a grammar
> file written in javascript, and must be linked into Emacs somehow.

That "linking" happens when Emacs is linked against the TS library,
right?

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Cc: Andrei Kuznetsov <r12451428287@163.com>,  manuel@ledu-giraud.fr,
>>   emacs-devel@gnu.org
>> Date: Thu, 29 Jul 2021 16:12:56 -0700
>> 
>> > TS's code is written in plain C, and doesn't require any regeneration
>> > or source modifications.  Anything else is misunderstanding.
>> 
>> That's true for the common TS runtime, which implements the parser and
>> error recovery, but the code for each language, that builds the LR parse
>> table and some other data structures, is generated in C from a grammar
>> file written in javascript, and must be linked into Emacs somehow.
>
> That "linking" happens when Emacs is linked against the TS library,
> right?

I don't know what you mean by "the TS library".

I'm guessing you mean the tree-sitter runtime, in which case no, that
does not include any languages.

-- 
-- Stephe

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: r12451428287@163.com,  manuel@ledu-giraud.fr,  emacs-devel@gnu.org
> Date: Sat, 31 Jul 2021 05:12:54 -0700
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Stephen Leake <stephen_leake@stephe-leake.org>
> >> Cc: Andrei Kuznetsov <r12451428287@163.com>,  manuel@ledu-giraud.fr,
> >>   emacs-devel@gnu.org
> >> Date: Thu, 29 Jul 2021 16:12:56 -0700
> >> 
> >> > TS's code is written in plain C, and doesn't require any regeneration
> >> > or source modifications.  Anything else is misunderstanding.
> >> 
> >> That's true for the common TS runtime, which implements the parser and
> >> error recovery, but the code for each language, that builds the LR parse
> >> table and some other data structures, is generated in C from a grammar
> >> file written in javascript, and must be linked into Emacs somehow.
> >
> > That "linking" happens when Emacs is linked against the TS library,
> > right?
> 
> I don't know what you mean by "the TS library".

I mean libtree-sitter.a produced by building the library.

> I'm guessing you mean the tree-sitter runtime, in which case no, that
> does not include any languages.

"Include" in what sense?

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Cc: r12451428287@163.com,  manuel@ledu-giraud.fr,  emacs-devel@gnu.org
>> Date: Sat, 31 Jul 2021 05:12:54 -0700
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> >> Cc: Andrei Kuznetsov <r12451428287@163.com>,  manuel@ledu-giraud.fr,
>> >>   emacs-devel@gnu.org
>> >> Date: Thu, 29 Jul 2021 16:12:56 -0700
>> >> 
>> >> > TS's code is written in plain C, and doesn't require any regeneration
>> >> > or source modifications.  Anything else is misunderstanding.
>> >> 
>> >> That's true for the common TS runtime, which implements the parser and
>> >> error recovery, but the code for each language, that builds the LR parse
>> >> table and some other data structures, is generated in C from a grammar
>> >> file written in javascript, and must be linked into Emacs somehow.
>> >
>> > That "linking" happens when Emacs is linked against the TS library,
>> > right?
>> 
>> I don't know what you mean by "the TS library".
>
> I mean libtree-sitter.a produced by building the library.
>
>> I'm guessing you mean the tree-sitter runtime, in which case no, that
>> does not include any languages.
>
> "Include" in what sense?

There is no code in libtree-sitter.a that provides a language; all
languages are built separately, by the language developers.

https://github.com/tree-sitter/tree-sitter builds libtree-sitter.a, and
the command line tools to build a language.

https://github.com/tree-sitter/tree-sitter-python builds the object file
providing the python language.

There are many other languages, each with its own repository.

-- 
-- Stephe

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: r12451428287@163.com,  manuel@ledu-giraud.fr,  emacs-devel@gnu.org
> Date: Sat, 31 Jul 2021 09:55:46 -0700
> 
> >> I don't know what you mean by "the TS library".
> >
> > I mean libtree-sitter.a produced by building the library.
> >
> >> I'm guessing you mean the tree-sitter runtime, in which case no, that
> >> does not include any languages.
> >
> > "Include" in what sense?
> 
> There is no code in libtree-sitter.a that provides a language; all
> languages are built separately, by the language developers.
> 
> https://github.com/tree-sitter/tree-sitter builds libtree-sitter.a, and
> the command line tools to build a language.
> 
> https://github.com/tree-sitter/tree-sitter-python builds the object file
> providing the python language.
> 
> There are many other languages, each with its own repository.

We are talking past each other.  But I don't think we should keep
arguing about this, because there's no real disagreement here to argue
about.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> The tree-sitter runtime, that Emacs would link with, it implemented in
> C, partly for this reason. It would be compiled with whatever Emacs is
> compiled with, or the system compiler.

Interesting.  I was not aware of that.

> Some of the tree-sitter development tools are implemented in Rust; you
> only need Rust if you are developing/fixing a grammar for a language.

If I understand this correctly, it means one would require the Rust
toolchain to support new languages in tree-sitter, or to improve
existing support.  Would that really fit Emacs?  I think many people
might not be comfortable learning such a large language and toolchain to
develop editing tools for Emacs.

Furthermore, is there any concrete reason this could not be done in
Lisp?

Note: Somehow I sent a reply earlier, and not a follow-up.  I apologize
for the duplicate.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> From: Andrei Kuznetsov <r12451428287@163.com>
> Date: Wed, 28 Jul 2021 19:43:03 +0800
> Cc: emacs-devel@gnu.org
> 
> > Some of the tree-sitter development tools are implemented in Rust; you
> > only need Rust if you are developing/fixing a grammar for a language.
> 
> If I understand this correctly, it means one would require the Rust
> toolchain to support new languages in tree-sitter, or to improve
> existing support.  Would that really fit Emacs?  I think many people
> might not be comfortable learning such a large language and toolchain to
> develop editing tools for Emacs.
> 
> Furthermore, is there any concrete reason this could not be done in
> Lisp?

This has been discussed.  Patches to convert the TS grammar files to
Emacs Lisp and/or to maintain and develop them in Emacs Lisp will be
most welcome.

As usual with Free Software, it isn't an issue of what's desirable,
it's an issue with someone stepping forward to do the job of
developing this stuff.

TIA

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

Eli Zaretskii <eliz@gnu.org> writes:

> This has been discussed.  Patches to convert the TS grammar files to
> Emacs Lisp and/or to maintain and develop them in Emacs Lisp will be
> most welcome.

Does "to maintain and develop them in Emacs Lisp" include facilities
providing functionality similar to TS but not compatible with TS grammar
files? If so I think I may have something up my sleeve, though in an
early state.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> From: Andrei Kuznetsov <r12451428287@163.com>
> Cc: stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Wed, 28 Jul 2021 20:06:23 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > This has been discussed.  Patches to convert the TS grammar files to
> > Emacs Lisp and/or to maintain and develop them in Emacs Lisp will be
> > most welcome.
> 
> Does "to maintain and develop them in Emacs Lisp" include facilities
> providing functionality similar to TS but not compatible with TS grammar
> files?

We are talking about the grammar files to be used by TS, so they
should be compatible, of course.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

Eli Zaretskii <eliz@gnu.org> writes:

> We are talking about the grammar files to be used by TS, so they
> should be compatible, of course.

I see.  Thanks for the clarification.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Ergus]

On Wed, Jul 28, 2021 at 07:43:03PM +0800, Andrei Kuznetsov wrote:
>Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> The tree-sitter runtime, that Emacs would link with, it implemented in
>> C, partly for this reason. It would be compiled with whatever Emacs is
>> compiled with, or the system compiler.
>
>Interesting.  I was not aware of that.
>
>> Some of the tree-sitter development tools are implemented in Rust; you
>> only need Rust if you are developing/fixing a grammar for a language.
>
>If I understand this correctly, it means one would require the Rust
>toolchain to support new languages in tree-sitter, or to improve
>existing support.  Would that really fit Emacs?  I think many people
>might not be comfortable learning such a large language and toolchain to
>develop editing tools for Emacs.
>
>Furthermore, is there any concrete reason this could not be done in
>Lisp?
>
I will say:
1) Performance (discussed in the previous thread):
2) Not reinvent the wheel.

Tree-sitter is very well maintained, optimized and with very specialized
algorithms; and we lack manpower to duplicate all that effort; and
implementing it in lisp won't really worth the efforts and may be
unmaintainable and slow.

Tree-sitter hopefully won't get abandoned in the future because many
editors use it right now (including neovim) and the community is very
dynamic.

Another advantage is that with tree-sitter as a back-end we could
officially (almost for free) support many languages that are currently
unsupported officially and may require a lot of effort to support them
in a minimal way (or currently supported in some inconsistent way, with
incoherent bindings/colors/indentations. Ex: Typescripts, Rust, Julia)

>Note: Somehow I sent a reply earlier, and not a follow-up.  I apologize
>for the duplicate.
>
>

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

Ergus <spacibba@aol.com> writes:

> 1) Performance (discussed in the previous thread):

FWIW I have been experimenting with an increcemental GLR parser
generator in Emacs Lisp.  While I have not put in the effort to couple
it with font-lock and such, from anecdotal examination it does not
perform badly with a naive C grammar.  The initial parse does take
several seconds on large files, but afterwards I did not notice a
significant drop in editor responsiveness.

> 2) Not reinvent the wheel.

While tree-sitter may be nice and all, it doesn't seem to offer the
usual extensibility expected from Emacs.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> From: Andrei Kuznetsov <r12451428287@163.com>
> Date: Wed, 28 Jul 2021 21:07:40 +0800
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>, emacs-devel@gnu.org
> 
> While tree-sitter may be nice and all, it doesn't seem to offer the
> usual extensibility expected from Emacs.

Which extensibility did you have in mind that TS doesn't support?

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

Eli Zaretskii <eliz@gnu.org> writes:

>> While tree-sitter may be nice and all, it doesn't seem to offer the
>> usual extensibility expected from Emacs.

> Which extensibility did you have in mind that TS doesn't support?

Let us assume that a generated TS grammar contains a (C) function akin
to `semantic-lex-unterminated-syntax-detected', and I wish to achieve
similar results to binding
`semantic-lex-unterminated-syntax-end-function' to a function of my
choice.  Would that be possible?

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> From: Andrei Kuznetsov <r12451428287@163.com>
> Cc: spacibba@aol.com,  stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Wed, 28 Jul 2021 21:27:43 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> While tree-sitter may be nice and all, it doesn't seem to offer the
> >> usual extensibility expected from Emacs.
> 
> > Which extensibility did you have in mind that TS doesn't support?
> 
> Let us assume that a generated TS grammar contains a (C) function akin
> to `semantic-lex-unterminated-syntax-detected', and I wish to achieve
> similar results to binding
> `semantic-lex-unterminated-syntax-end-function' to a function of my
> choice.  Would that be possible?

(TS doesn't generate a grammar, it comes with grammar files prepared
externally.)

If you are talking about affecting how TS does lexical analysis for
some language, then I see no reason why we in the Emacs project would
want to do that.  We don't _want_ to develop parsers if we can use
parsers available out there.  Lexical analysis of a parser is
determined by the language it parses, so you need only to change the
parser when the language changes, or to fix a bug.  Both are part of
the job of the TS developers, so there should be no need for us to get
busy with that.  Exactly like we do with other libraries we use that
aren't developed as part of the Emacs project.

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

Eli Zaretskii <eliz@gnu.org> writes:

> If you are talking about affecting how TS does lexical analysis for
> some language, then I see no reason why we in the Emacs project would
> want to do that.  We don't _want_ to develop parsers if we can use
> parsers available out there.  Lexical analysis of a parser is
> determined by the language it parses, so you need only to change the
> parser when the language changes, or to fix a bug.  Both are part of
> the job of the TS developers, so there should be no need for us to get
> busy with that.  Exactly like we do with other libraries we use that
> aren't developed as part of the Emacs project.

Okay, thanks for the clarification

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Manuel Giraud]

Andrei Kuznetsov <r12451428287@163.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> If you are talking about affecting how TS does lexical analysis for
>> some language, then I see no reason why we in the Emacs project would
>> want to do that.  We don't _want_ to develop parsers if we can use
>> parsers available out there.  Lexical analysis of a parser is
>> determined by the language it parses, so you need only to change the
>> parser when the language changes, or to fix a bug.  Both are part of
>> the job of the TS developers, so there should be no need for us to get
>> busy with that.  Exactly like we do with other libraries we use that
>> aren't developed as part of the Emacs project.
>
> Okay, thanks for the clarification

Yes, thanks for these clarifications and sorry for my
misunderstanding. I still have one question left though: will the
parsers C code (and TS C code) land into the emacs repo or will TS be
accessible as an external library?
-- 
Manuel Giraud

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Perry E. Metzger]

On 7/28/21 10:41, Manuel Giraud wrote:
> Yes, thanks for these clarifications and sorry for my
> misunderstanding. I still have one question left though: will the
> parsers C code (and TS C code) land into the emacs repo or will TS be
> accessible as an external library?

I don't think that has been fully decided. It may be necessary to have a 
patched version of Tree Sitter available to integrate properly with the 
rest of the Emacs runtime.

Perry

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Eli Zaretskii]

> From: Manuel Giraud <manuel@ledu-giraud.fr>
> Cc: Eli Zaretskii <eliz@gnu.org>,  spacibba@aol.com,
>   stephen_leake@stephe-leake.org,  emacs-devel@gnu.org
> Date: Wed, 28 Jul 2021 16:41:01 +0200
> 
> I still have one question left though: will the parsers C code (and
> TS C code) land into the emacs repo or will TS be accessible as an
> external library?

The latter.  Unless something very unexpected will be discovered about
TS that could not be fixed by the TS developers.

Re: [SPAM UNSURE] Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Stephen Leake]

Andrei Kuznetsov <r12451428287@163.com> writes:

> Ergus <spacibba@aol.com> writes:
>
>> 1) Performance (discussed in the previous thread):
>
> FWIW I have been experimenting with an increcemental GLR parser
> generator in Emacs Lisp.  

The "generator" and the "runtime" are two separate programs, with
separate functions, used at different times.

The generator takes the javascript language grammar file and translates
it (thru lots of hairy computations) into code that builds a parse table
and other data structures. The tree-sitter generator outputs that code
in C; it might be possible to adapt it to output in elisp (the wisitoken
generator used to output elisp, but i gave that up when I implemented
error recover in Ada; elisp is way to slow for that).

The "runtime" uses the parse table to parse text at runtime, in response
to user actions on the buffer. To be useful in an interactive editing
context, it must have robust error recovery. What is your error recovery
algorithm?

> While I have not put in the effort to couple it with font-lock and
> such, from anecdotal examination it does not perform badly with a
> naive C grammar. 

Are you talking about the generator or runtime here?

> The initial parse does take several seconds on large files, 

That's the runtime. Actual time for xdisp.c, preferably compared with a
tree-sitter parse run on the same machine, would be helpful.

How long does the generator take?

> but afterwards I did not notice a significant drop in editor
> responsiveness.

This seems to imply that the runtime supports incremental parse, so it
does not reparse the whole buffer each time; is that true?

>> 2) Not reinvent the wheel.
>
> While tree-sitter may be nice and all, it doesn't seem to offer the
> usual extensibility expected from Emacs.

It's all open-source, but it is very complicated and may be beyond many
people's ability to change correctly.

It requires running a C compiler to change it, but so do other parts of
Emacs (for example, the json parser).

So what is it missing?

-- 
-- Stephe

Re: [SPAM UNSURE] Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> The "generator" and the "runtime" are two separate programs, with
> separate functions, used at different times.
>
> The generator takes the javascript language grammar file and translates
> it (thru lots of hairy computations) into code that builds a parse table
> and other data structures. The tree-sitter generator outputs that code
> in C; it might be possible to adapt it to output in elisp (the wisitoken
> generator used to output elisp, but i gave that up when I implemented
> error recover in Ada; elisp is way to slow for that).
>
> The "runtime" uses the parse table to parse text at runtime, in response
> to user actions on the buffer. To be useful in an interactive editing
> context, it must have robust error recovery. What is your error recovery
> algorithm?

Currently extremely naive.  After an error occurs, it skips productions
until it can parses without errors, and just continues from there.  I
plan to improve it somewhat in the near future.

> Are you talking about the generator or runtime here?

The runtime.  The parser generator does not seem to be astonishingly
fast, but I don't think most people will have any cause to run it very
often.

> That's the runtime. Actual time for xdisp.c, preferably compared with a
> tree-sitter parse run on the same machine, would be helpful.

I'm currently pre-occupied and unable to work on this, but I will return
with these measurements as soon as reasonably possible.

> How long does the generator take?

I did not measure that, but as most people would be loading compiled
parsers, and not running the generator, I don't think it would matter
too much.  FWIW macroexpansion of the macro `defgrammar' blocks Emacs
for a second or 2.

> This seems to imply that the runtime supports incremental parse, so it
> does not reparse the whole buffer each time; is that true?

Indeed.  I've not yet figured out a particularly good way of recording
changes though -- as of present it relies on its own versions of
self-insert-command, kill-region, et cetera.

Re: [SPAM UNSURE] Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Andrei Kuznetsov]

Andrei Kuznetsov <r12451428287@163.com> writes:


> until it can parses without errors, and just continues from there.
           ^^^^^^^^^^

I meant to say "parse" instead.

Further, as for "without errors", it skips until it finds the next
synchronizing token, attempts to parse starting from that token, and if
that fails repeats the process until either EOF is reached or it is
successful.

Re: [SPAM UNSURE] Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Stephen Leake]

Andrei Kuznetsov <r12451428287@163.com> writes:

> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> How long does the generator take?
>
> I did not measure that, but as most people would be loading compiled
> parsers, and not running the generator, I don't think it would matter
> too much.  FWIW macroexpansion of the macro `defgrammar' blocks Emacs
> for a second or 2.

It can matter a lot for large grammars. wisitoken used to take hours to
generate the LR1 parse table for Ada; now it takes a couple minutes.
tree-sitter never finishes that grammar.

A naive LR grammar generator can easily be O (n**3) or worse in the
grammar size; I spent a lot of time optimizing wisitoken so it can
handle Ada reasonably.

-- 
-- Stephe

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Perry E. Metzger]

On 7/28/21 07:43, Andrei Kuznetsov wrote:
> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> Some of the tree-sitter development tools are implemented in Rust; you
>> only need Rust if you are developing/fixing a grammar for a language.
> If I understand this correctly, it means one would require the Rust
> toolchain to support new languages in tree-sitter, or to improve
> existing support.

That's not true. Tree Sitter is not written even partially in Rust. It 
does have Rust bindings for people who use Rust.

Perry

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Stephen Leake]

"Perry E. Metzger" <perry@piermont.com> writes:

> On 7/28/21 07:43, Andrei Kuznetsov wrote:
>> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>>
>>> Some of the tree-sitter development tools are implemented in Rust; you
>>> only need Rust if you are developing/fixing a grammar for a language.
>> If I understand this correctly, it means one would require the Rust
>> toolchain to support new languages in tree-sitter, or to improve
>> existing support.
>
> That's not true. Tree Sitter is not written even partially in Rust. It
> does have Rust bindings for people who use Rust.

https://github.com/tree-sitter/tree-sitter/tree/master/cli/src

-- 
-- Stephe

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Perry E. Metzger]

On 7/29/21 19:28, Stephen Leake wrote:
> "Perry E. Metzger" <perry@piermont.com> writes:
>
>> That's not true. Tree Sitter is not written even partially in Rust. It
>> does have Rust bindings for people who use Rust.
> https://github.com/tree-sitter/tree-sitter/tree/master/cli/src
>
That's an optional CLI and is not part of the library runtime.

Perry

Re: [SPAM UNSURE] Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Stephen Leake]

"Perry E. Metzger" <perry@piermont.com> writes:

> On 7/29/21 19:28, Stephen Leake wrote:
>> "Perry E. Metzger" <perry@piermont.com> writes:
>>
>>> That's not true. Tree Sitter is not written even partially in Rust. It
>>> does have Rust bindings for people who use Rust.
>> https://github.com/tree-sitter/tree-sitter/tree/master/cli/src
>>
> That's an optional CLI and is not part of the library runtime.

Yes, and the optional CLI is part of the tree-sitter project, so the
statement "Tree Sitter is not written even partially in Rust" is simply
wrong.

Please be more careful when you say "tree-sitter", but mean
"tree-sitter runtime"; it is not always clear from context.

-- 
-- Stephe

Re: [SPAM UNSURE] Maybe we're taking a wrong approach towards tree-sitter

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I don't think we should reject Rust code for the GNU system.  There is
no need for such a drastic step.  We already use software that is
built in Rust.

Tree sitter is not going to be a part of Emacs; its use is not limited
to Emacs.  Other programs will work with it too.  So I don't see any
special reason to replace parts of it with Emacs Lisp code.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Maybe we're taking a wrong approach towards tree-sitter

[Perry E. Metzger]

On 7/27/21 21:57, Andrei Kuznetsov wrote:
> Unlike features like native JSON, however, I believe tree-sitter is the
> first optional package providing notable functionality that would
> require a toolchain that depends on LLVM (that of Rust, which
> tree-sitter is implemented in)

Tree sitter is written in C. It has an available set of Rust bindings. 
It compiles perfectly well with any C compiler.

Perry

Re: Maybe we're taking a wrong approach towards tree-sitter

[Stephen Leake]

"Perry E. Metzger" <perry@piermont.com> writes:

> On 7/27/21 21:57, Andrei Kuznetsov wrote:
>> Unlike features like native JSON, however, I believe tree-sitter is the
>> first optional package providing notable functionality that would
>> require a toolchain that depends on LLVM (that of Rust, which
>> tree-sitter is implemented in)
>
> Tree sitter is written in C. 

There are many parts to tree-sitter. The runtime, which uses
language-specific parse tables to parse use files, is written in C.

The command line tools (cli), one of which converts the language grammar
file written in javascript into C code that builds the parse table, are
written in Rust;

https://github.com/tree-sitter/tree-sitter/tree/master/cli/src

> It has an available set of Rust bindings. It compiles perfectly well
> with any C compiler.

Here you are describing the runtime, which is what must be linked with
Emacs for a major-mode to use the tree-sitter parser.

-- 
-- Stephe