Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Yuan Fu <casouri@gmail.com>]

> On Nov 18, 2022, at 1:54 PM, Jostein Kjønigsen <jostein@secure.kjonigsen.net> wrote:
> 
> Hey everyone.
> 
> I know this has been said before, by people which by far has been contributing much more than me... But I still don't think it's a good idea to replace the implementation in existing major-modes with tree-sitter implementations, nor selectively activate tree-sitter in major-modes prone to inhetitence and derivation.
> 
> Me and Theodor faced these same issues with "our" C# and TypeScript major-modes, and the only "clean" way we agreed we could make this work was to create wholly new implementations. I can come up with many good, objective reasons for this, but I think Theodor has already represented this view fairly well.
> 
> While for the sake of brevity, I'll not diving deeply into this particular thing, I will say this: A new tree-sitter based major-mode free of compatibility concerns allowed us to create entirely new major-modes fixing most of our existing bugs, faster than we before would be able to fix a single bug. My personal view is that mixing existing major-modes with tree-sitter represents absolutely the worst of all worlds. It maintains all existing complexities, provides us with very few benefits, and at the same time adds complexities we didn't use to have. To me, that's a net negative.
> 
> Somewhat surprising to me, I see this is a somewhat controversial point of view and not as clear cut a matter as I would have expected it to be. I realize and respect that final decisions in these matter might take some time to mature. Time which given our current approach detracts from the momentum tree-sitter has been having.

Hey Jostein,

Thank you very much for your thoughtful input! I originally thought of tree-sitter as merely a mean/tool that major modes could use to enhance their features. But in reality it seems that tree-sitter replaces a large enough chunk of a major-mode’s responsibility, which has caused all these compatibility problems we’ve encountered. Examples include undoing cc-mode’s setup when turning on tree-sitter, invalidating many existing major-mode variables, and the major mode inheritance problem I presented in the previous message. 

These difficulties has nudged us to decouple the tree-sitter and non-tree-sitter version further and further, from trying to make tree-sitter a feature that enables/disabled in a major mode by a command, to a major mode configuration that causes the major mode to setup differently, to my suggestion of using separate major mode but share a single virtual parent mode.

I think using separate major mode but share a single virtual parent mode gets the benefit of both worlds:
1. Tree-sitter and non-tree-sitter are separated in different modes, meaning it gets all the benefit Jostein and Theo described.
2. Backward-compatible. Existing configuration to the non-tree-sitter version works as they do before. (Here I’m mainly thinking about hooks: Hooks added to c-mode-hook still runs in c-native-mode.)
3. Solves the inheritance problem I described.
4. Minimal changes to the existing modes.

> At this point poor Yuan Fu here has spent quite a bit of time and effort getting a core tree-sitter interface into Emacs. I would really hate to see all that effort be for nothing, because we end up conflating a creating a core tree-sitter feature with how this feature should best be employed in subsequent major-modes.
> 
> So in the name of pragmatism, I propose a compromise of sorts.
> 
> Instead of waiting for "every" major-mode to be re-implemented into a tree-sitter derivative in the feature/tree-sitter branch before we merge... How about we just accept the current "core" tree-sitter implementation as good enough, and consider merging that to git master as is.
> 
> This will allow us to land one important mile-stone, while giving us the head-room to further discuss how we should best implement/reimplement/"upgrade" existing major-modes to take advantage of tree-sitter.
> 
> It will also allow third-party packages to make use of tree-sitter in Emacs core instead coming up with its own implementation, beginning a defacto standardization of this new API (which I may note already has a competing implementation in MELPA).
> 
> That is to say: We should land the core library, with holding it hostage to all its possible consumers being implemented.
> 
> How about it? Are there any good arguments for NOT merging feature/tree-sitter at this point? :)

The good news is, feature/tree-sitter is merging in a few days! And we’ll make further improvements on the master. So rest assured! :-) I think we can at least get C, Python, Javascript, Typescript, Bash, JONS and CSS to go with the up coming release, largely thanks to Theo’s (and tree-sitter’s) productivity. I personally don’t know enough of C++ and Java to polish them, but they have a good chance too.

Taking a step back, I think developing major mode support with the tree-sitter API is a good idea. We found countless shortcomings of the API when developing major modes with it.

Yuan