Re: Tree-sitter introduction documentation

[Dmitry Gutov <dgutov@yandex.ru>]

On 27/12/2022 15:38, Eli Zaretskii wrote:
>> Date: Tue, 27 Dec 2022 14:43:06 +0200
>> Cc: monnier@iro.umontreal.ca, theophilusx@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> WDYT about what we have in NEWS about this?
>>
>> Those instructions seem to be written foremost with distro maintainers
>> in mind.
> 
> Or users who build and install Emacs by themselves.
> 
> Frankly, I don't see why we, the upstream project, need to worry about
> anyone else.  It isn't our job.  That's what distros are there for.

Previously all one needed for a language support mode is to download and 
load an .el file. As we drift to the idea of using externally-maintained 
grammars, and the "native" modes become less useful (possibly deprecated 
in 5-10), it seems like it will become more of our responsibility to 
streamline.

>> Definitely better to have them than not, but I'd hate to present
>> them to the average user.
> 
> "Hate"?  That's a strong word.

Also a figure of speech.

> The questions that the NEWS entries
> answer were asked here and elsewhere several times, so presumably that
> information has some non-trivial value.

Of course.

>> Do we expect all (most?) distros to compile all the popular grammars?
> 
> I honestly don't know.  On the one hand, there aren't many Emacs modes
> which use tree-sitter, but OTOH they could start growing like
> mushrooms once Emacs 29 hits the streets.  I do expect them to offer
> the ones they consider useful/needed, for some value of that.  I
> really don't see any significant difference in this regard between
> grammar libraries and, say, librsvg.  Both are used in Emacs, and the
> lack of either disables useful Emacs features.  So it's a no-brainer
> for me.  But then I'm not a distro maintainer, and never have been.

On the flip side, the third-party modes can provide their own 
download-compile-install instructions, which will make it easier to end 
users.

The barrier to creating such a mode, though, is now higher.

>> That would still leave out the users of the less popular languages whose
>> grammars were not included. Or grammars which saw updates since the
>> distro-distributed version (so it's useful to install the newer version).
> 
> What's the solution?  All the "solutions" I saw until now require a
> working and well-configured C/C++ compiler (sometimes both C and C++),
> linker, and C/C++ runtimes.  A user who has them already installed can
> easily build a grammar library with two simple commands.  A user who
> doesn't have a C/C++ development environment will not find those
> "solutions" useful at all.  And asking us to distribute binaries for
> half a dozen popular systems is IMNSHO unreasonable.

I think it's common enough for a user to have build tools installed, but 
not know well enough how to set up a C project. Think junior-middle 
developers in a number of languages which are not C.

Or just first grade students.

>> I wouldn't worry too much about the maintenance burden (keeping the list
>> of urls up-to-date?), especially since we could refer to such lists by
>> other projects.
> 
> I cannot disagree more.  Look at this from my POV: once the list
> becomes even semi-official, people will expect it to be of the same
> high quality as all the rest of Emacs, and they _will_ complain and
> report inaccuracies.  It's a nuisance, especially for such a "hot"
> feature set.

They will report inaccuracies, which will be helpful to fixing them. 
That is certainly a workload, but still small compared to the current 
flow of bug reports, I think. Or the many hours one would spend fixing a 
font- or redisplay-related problem.

Anyway, my point was not to put this burden on you specifically. If you 
might recall, I've always advocated toward "smaller core with many 
plugins" as a model of Emacs development.

> And which "other projects"? who can track those and know which ones
> have the most accurate, up-to-date, and comprehensive list?  I'm a bit
> interested in this (and have several dozens of grammar libraries built
> locally), and I discover another project with a useful list of
> grammars almost every day.  These things are highly dynamic: I see
> some of the grammars get updates every couple of days.  Some languages
> have more than one grammar library maintained by different people --
> who will figure out which one is better for us and keep that
> information up-to-date?

The Neovim repo will likely be a good resource for this in the near 
future. This file in particular: 
https://github.com/nvim-treesitter/nvim-treesitter/blob/f2b1d727e6ad46238baa84c4d1f968a297e415ab/lua/nvim-treesitter/parsers.lua

But it brings me to another concern, showcased by this commit: 
https://github.com/nvim-treesitter/nvim-treesitter/commit/0cb637ca9f4389172933e5aba36387ab8430b6fb

The AST for one version of a grammar might be incompatible enough with a 
newer one, making the TS queries, font-lock and indentation rules 
obsolete or at least slightly broken. nvim-treesitter works around this 
by locking the repository version of a grammar corresponding to the 
current language support code.

How much this will be a problem in practice for us? I'm not sure. 
Perhaps most popular grammars have had enough time to mature by now.

>> I think ELPA is a better place for this feature, though. Because we
>> always want the user to get the latest version of the recipes.
> 
> That solves only part of the problem.  (And not an important part: our
> Git repository is public, so people can track it and download updates
> for files as easily as they track ELPA.)

One is not exactly like the other from an end user's POV.

> The hard part -- keeping the
> information accurate and up-to-date -- still needs a motivated
> volunteer.  And we hardly have resources to work on our code and docs,
> let alone help people install external software.
> 
> (Of course, if such a motivated volunteer steps forward, he or she
> will be most welcome.)

My guess is we have a few people here already who might be interested.