Re: Tree-sitter api

[Eli Zaretskii <eliz@gnu.org>]

> From: Yuan Fu <casouri@gmail.com>
> Date: Thu, 16 Sep 2021 23:56:20 -0700
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>  Tuấn-Anh Nguyễn <ubolonton@gmail.com>,
>  Theodor Thornhill <theo@thornhill.no>,
>  Clément Pit-Claudel <cpitclaudel@gmail.com>,
>  Emacs developers <emacs-devel@gnu.org>,
>  stephen_leake@stephe-leake.org
> 
> We have documentation for all the tree-sitter features provided by Emacs and a bit more, but I don’t think it is possible to document the language definitions. We can think of language definitions as BNF grammars for each language, how do you document that?

Why do we need to document the language definitions?  When a Lisp
programmer defines font-lock and indentation for a programming
language in the current Emacs, do they necessarily need to consult the
language grammar?

> Say, for the language definition for Scheme below, how do we document it?
> 
> <token> --> <identifier> | <boolean> | <number>
>      | <character> | <string>
>      | ( | ) | #( | 
> ' | ` | , | ,@ | .
> <delimiter> --> <whitespace> | ( | ) | " | ;
> <whitespace> --> <space or newline>
> <comment> --> ;  <all subsequent characters up to a
>                  line break>
> ...
> <number> --> <num 2>| <num 8>
>      | <num 10>| <num 16>
> …

This stuff should be known to TS; the Lisp programmer only needs to be
aware of the results of lexical and syntactical analysis, in terms of
their Lisp expressions (Lisp data structures with appropriate symbols
and fields).

> And I want to also point out that as Emacs core developers, we can’t possibly provide a good translation from convention language names to their tree-sitter name (C# -> c-sharp). Maybe we can do a half-decent job, but 1) that won’t cover all available languages, and 2) if there is a new language, we need to wait for the next release to update our translation. It is better for the major mode writers to provide the information on how to translate names.

The database used by the conversion should definitely be extensible.
But that doesn't mean it should be empty.

Anyway, we've spent enough time on this issue.  If you are still
unconvinced, feel free to do it your way, and let the chips fall as
they may.