[NonGNU ELPA] New packages proposal: tree-sitter

[NonGNU ELPA] New packages proposal: tree-sitter

[Tuấn-Anh Nguyễn]

Hi,

I want to add 3 Tree-sitter packages to NonGNU ELPA:
- tsc: Low-level bindings of core tree-sitter functionalities, exposed to ELisp
  through a dynamic module, as well as facilities to compile that dynamic module
  from source.
- tree-sitter: High-level integration with Emacs's editing facilities, including
  per-buffer syntax trees and code highlighting mechanism.
- tree-sitter-langs: Collection of language-specific highlighting patterns, as
  well as facilities to compile/download language grammars.

After checking the guidance, I see that the packages satisfy most of the rules,
except for the following:

1. Copyright notices: I use the header SPDX-License-Identifier in each file,
   instead of putting in the full text. Is this acceptable?

2. No code downloading over the Internet:

Currently, on supported platforms (Linux/macOS/Windows on x86_64):
- The tsc package would download the pre-compiled dynamic module from GitHub.
- The tree-sitter-langs package would download an archive of pre-compiled
  grammars from GitHub (in the form of shared dynamic libraries).

Note that the packages do provide facilities for users to compile the binaries
on their own. (Documentation on how to do so is to be written.) The dynamic
module's source would be included in the tar archive. The sources of language
grammars would need to be obtained by other means, e.g. by downloading from
various git repositories over the Internet.

What's the best way to resolve this? I can think of several ways:
a. Disable downloading of pre-compiled binaries entirely, and ask users to
   compile on their own.
b. Include the pre-compiled binaries (for several platforms) in the tar
   archives.
c. Host the pre-compiled binaries separately, on NonGNU ELPA server
instead of GitHub.

Are there other questions/comments/objections?

Links:
- Doc: https://emacs-tree-sitter.github.io/
- Source for tree-sitter and tsc:
https://github.com/emacs-tree-sitter/elisp-tree-sitter
- Source for tree-sitter-langs:
https://github.com/emacs-tree-sitter/tree-sitter-langs

--
Tuấn-Anh Nguyễn
Software Engineer

Re: [NonGNU ELPA] New packages proposal: tree-sitter

[Eli Zaretskii]

> From: Tuấn-Anh Nguyễn <ubolonton@gmail.com>
> Date: Fri, 27 Aug 2021 17:48:11 +0700
> 
> I want to add 3 Tree-sitter packages to NonGNU ELPA:
> - tsc: Low-level bindings of core tree-sitter functionalities, exposed to ELisp
>   through a dynamic module, as well as facilities to compile that dynamic module
>   from source.
> - tree-sitter: High-level integration with Emacs's editing facilities, including
>   per-buffer syntax trees and code highlighting mechanism.
> - tree-sitter-langs: Collection of language-specific highlighting patterns, as
>   well as facilities to compile/download language grammars.

I'd be much happier if you could coordinate with Yuan Fu
<casouri@gmail.com>, who works on integrating TS into the Emacs core.
This stuff should IMO be in core, not in 3rd-party packages, let alone
on nonGNU ELPA.

Can you tell which functionality is missing from Yuan Fu's work that
is supported by those 3 packages?  Can that functionality be added to
Yuan Fu's code?

Thanks.

Re: [NonGNU ELPA] New packages proposal: tree-sitter

[Yuan Fu]

> I'd be much happier if you could coordinate with Yuan Fu
> <casouri@gmail.com>, who works on integrating TS into the Emacs core.
> This stuff should IMO be in core, not in 3rd-party packages, let alone
> on nonGNU ELPA.

I guess I should have reached out sooner, communicating that I’m about to do it, but on the other hand I don’t want to just come in and talk about what I’m going to do and such and such, before I actually did anything. And my general understanding is that there aren’t much interest in integrating tree-sitter into the C core. So...

> Can you tell which functionality is missing from Yuan Fu's work that
> is supported by those 3 packages?  Can that functionality be added to
> Yuan Fu's code?

The code is at https://github.com/casouri/emacs, in the “ts” branch. The main things to look at are src/tree-sitter.c, lisp/tree-sitter.el, and doc/lispref/parsing.texi.

There is an on-going discussion about tree-sitter API in Emacs, starting at https://lists.gnu.org/archive/html/emacs-devel/2021-08/msg00246.html

Yuan

Re: [NonGNU ELPA] New packages proposal: tree-sitter

[Tuấn-Anh Nguyễn]

> I'd be much happier if you could coordinate with Yuan Fu
> <casouri@gmail.com>, who works on integrating TS into the Emacs core.

That's a good idea. I'll start by reviewing that work when I have time.

> This stuff should IMO be in core, not in 3rd-party packages, let alone
> on nonGNU ELPA.

As I've already said before, I don't see this as an "either or" situation. I
want tree-sitter to be integrated into Emacs core too, eventually. Seems like we
are still a long way from there, though. These packages already work, and are
used by many people.

> I guess I should have reached out sooner, communicating that I’m about to do it, but on the other hand I don’t want to just come in and talk about what I’m going to do and such and such, before I actually did anything. And my general understanding is that there aren’t much interest in integrating tree-sitter into the C core. So...

Great initiative! To clarify, again: I wanted tree-sitter to be integrated into
Emacs core, and proposed the dynamic-module-based approach as an interim
solution that would better facilitate experimentation while we work towards
that end goal. That has worked out reasonably well. When Emacs core has
tree-sitter integrated, these packages can hopefully be deprecated.

> > Can you tell which functionality is missing from Yuan Fu's work that
> > is supported by those 3 packages?  Can that functionality be added to
> > Yuan Fu's code?
>
> The code is at https://github.com/casouri/emacs, in the “ts” branch. The main things to look at are src/tree-sitter.c, lisp/tree-sitter.el, and doc/lispref/parsing.texi.
>
> There is an on-going discussion about tree-sitter API in Emacs, starting at https://lists.gnu.org/archive/html/emacs-devel/2021-08/msg00246.html

Thanks, I'll check it out. In the meantime, please feel free to study the
packages, borrow the code/ideas/design decisions, and ask questions about them,
or about tree-sitter in general.

Thanks,
--
Tuấn-Anh Nguyễn
Software Engineer