Re: Tree-sitter integration in python.el

[Yuan Fu <casouri@gmail.com>]

> On Sep 26, 2022, at 12:10 PM, Jostein Kjønigsen <jostein@secure.kjonigsen.net> wrote:
> 
> On 22.09.2022 20:42, Yuan Fu wrote:
>> Hi, 
>> 
>> I’ve added tree-sitter version for font-lock and which-func in python.el. And I’d love to hear some feedback from python.el maintainers. Specifically, does it look right and which other part of python.el could actually benefit from a parse tree? I wrote a tree-sitter imenu indexer for python and it performed worse than the current one, presumably because it traverses the whole parse tree whereas the current one only scans the buffer once or so and do some regex matching.
>> 
>> Here is the commit: 
>> https://git.savannah.gnu.org/cgit/emacs.git/commit/?h=feature/tree-sitter&id=1cdb24fe35a9ff2e4f92c5acc93a5a5b0e70d93f
>> 
>> 
>> Yuan
>> 
>> 
>> 
> Hey Yuan.
> 
> Thanks for putting in all this work into tree-sitter in Emacs!
> 
> Trying the latest Emacs-version in feature/tree-sitter I got an error I tend to get "a lot" with tree-sitter based modes, which I hoped bundling things with Emacs would solve, namely obtaining the original shared-object containing the compiled grammer.
> 
> Like for your python-mode, I get this:
> 
> File mode specification error: (treesit-load-language-error python (/home/jostein/.emacs.d/tree-sitter/libtree-sitter-python: cannot open shared object file: No such file or directory /home/jostein/.emacs.d/tree-sitter/libtree-sitter-python.so: cannot open shared object file: No such file or directory libtree-sitter-python: cannot open shared object file: No such file or directory libtree-sitter-python.so: cannot open shared object file: No such file or directory))
> 
> I realize third-party modes are on their own, but when tree-sitter is compiled with Emacs, I would at least expect the Emacs-build to also produce these .so-files.
> 
> What are your thoughts on how we can best, across the Emacs-verse, provide these libraries? Or at least for the modes which are bundled with Emacs itself?
> 
> Being a tree-sitter based-developer myself, I know where I can go to get this compiled to make the mode runnable, but surely that's not how we can deploy this en-masse. Most people will be stuck at this point, and we will need to come up with a better answer.
> 
> -- 
>  Kind regards
> Jostein Kjønigsen
> 
> jostein@kjonigsen.net 🍵 jostein@gmail.com
> https://jostein.kjønigsen.no

This has been discussed before and our conclusion is to treat language definitions like other dynamic libraries, expecting package manager to install them. There are a couple of reasons. Tree-sitter library and language definitions must be on the same “protocol” version to work. I expect this version to change slowly, but in principle we don’t want to bundle a language definition that could conflict with the tree-sitter library provided by the system. Also, as a dynamic object file, it is machine-dependent. I don’t know if we bundle anything machine-dependent in Emacs distribution, but at any rate it is unusual. 

I do have some ideas to make it easier for users, like hosting them on ELPA or NONGNU ELPA, and user can install them by package-install. Eg, a package tree-sitter-installer-linux, and M-x tree-sitter-installer-linux RET c RET will install C language definition for you. Stefan, WDYT?

Yuan