Re: Using incremental parsing in Emacs (via: emacs rendering comparisson between emacs23 and emacs26.3)

[Stefan Monnier <monnier@iro.umontreal.ca>]

>> tree-sitter, like LSP, is something Emacs should embrace.
>   https://lists.gnu.org/archive/html/emacs-devel/2020-01/msg00059.html

Ah, thanks Eli: I guess I skipped over that while catching up.

> Would someone like to try to figure out how we could use the
> incremental parsing technology in Emacs for making our
> programming-language support more accurate and efficient?  One package
> that implements this technology is tree-sitter:
>
>   https://tree-sitter.github.io/tree-sitter/

Yes, adding support for this would be great.  

> AFAIU, these capabilities could be used as an alternative to
> regexp- and syntax-pps-based font-lock, better code folding,
> completion, refactoring, and other similar features; in general, any
> feature which would benefit from having a parse tree for the source
> code in a buffer.

Some of those features could be provided by LSP as well, but IIUC the
way LSP is designed and usually used makes it somewhat inadequate for
synchronous use, when you want an immediate answer.

tree-sitter is designed exactly for that: it can parse "immediately",
in the same sense as `syntax-ppss`, so LSP seems inapplicable (in the
near future at least) for things like font-lock and navigation, and
indentation, whereas tree-sitter should work great for that.

[ W.r.t disucssions around LSP's use of JSON: AFAICT, parsing and
  emitting json can be done as efficiently as any other format, AFAICT,
  so I don't see the use of JSON as a problem in the protocol.  ]

> To be able to use such libraries, we need to figure out how to
> integrate them into the core, what kind of interfaces would be needed
> for that, and what kind of infrastructure we would need for basing
> Lisp features on those libraries.

The existing third party packages should be good starting points to come
up with a design.  But I think an important issue is to figure out how
to make tree-sitter usable for the end users: AFAICT the main issue
being how to let end users download and install new grammars.
IIUC grammars are written in Javascript (or some subset thereof?) and
then somehow compiled to C code.  Having them as C code implies either
the end-user need to have a C compiler or distributing pre-compiled
binaries with all the trouble this entails (with all the variations of
OSes, and architectures, and ABIs, ..., plus issues related to
licensing, security, ...).

Maybe those grammars could be compiled to some other representation (I
don't know if it is made mostly of data-tables or actual code or what)?


        Stefan