Re: Tree-sitter integration on feature/tree-sitter

[Eli Zaretskii <eliz@gnu.org>]

> From: Yoav Marco <yoavm448@gmail.com>
> Cc: casouri@gmail.com, emacs-devel@gnu.org
> Date: Thu, 12 May 2022 20:22:30 +0300
> 
> > But maybe we should make this discussion more concrete.  Can you show
> > the queries and explain how they are produced from the font-lock rules
> > (or whatever else they are produced from)?  How many different queries
> > do we expect to have in a garden-variety major mode for a PL, and what
> > do they depend on?
> 
> So first of all, query is kind of an aggregate term, since one query
> string/sexp can contain many "query patterns". I expect most major modes
> to have one big query string/sexp, and maybe a handful more that are
> optional to users. treesit allows you to set as many query strings/sexps
> as you want for syntax highlighting. Outside of that, queries are also
> how packages like evil-textobj-tree-sitter work, with the backend of the
> elisp-tree-sitter which uses a dynamic module.
> 
> Queries are specific to the parse tree and therefore to the parser. Most
> parsers have a queries/highlights.scm file in their repo, and
> tree-sitter-langs contains a bunch of these:
> 
> > https://github.com/emacs-tree-sitter/tree-sitter-langs/#readme
> >
> > Highlighting query patterns for a language are in the file
> > queries/<lang>/highlights.scm. Most of them are intentionally
> > different from those from upstream repositories, which are more geared
> > towards GitHub’s use cases. We try to be more consistent with Emacs’s
> > existing conventions. (For some languages, this is WIP, so their
> > patterns may look similar to upstream’s.)
> 
> The query I used in the benchmarks is tree-sitter-langs's
> queries/c/highlights.scm, which is a rather big file. One thing to check
> that I only thought of now is how long it takes with treesit having to
> compile and run multiple queries.

Is it true that there's just one query for each PL mode, and it is
fixed (doesn't change) and doesn't depend on the buffer contents in
any way?  If that is true, the major mode could compile the query
whenever it is initialized, and then reuse it in every buffer that is
under that major mode.

If the above conclusion is not correct, then please tell what are the
differences between the query/queries of different buffers, and how do
they depend on the buffer contents.

> >   . the time it takes to visit xdisp.c and display the first window-full
> >   . visit xdisp.c, then immediately go to its end
> >   . C-v in xdisp.c (repeat many times to see how much a single C-v
> >     takes)
> 
> Okay, we can try that. What's the proper way to trigger a "natural
> fontification" as would occur in the GUI without opening an interactive
> session?

There isn't any (IIUC what you are asking).  Fontification is a
feature of interactive sessions, and is basically meaningless without
normal redisplay.

> I'd rather use the groundwork that's actually used by users,
> and not get stuff like the JIT chunck size wrong. In general I'm not too
> familiar with that part of Emacs; the benchmarks up to now used
> with-temp-buffer, would that suffice for these new benchmarks?

Using with-temp-buffer could cause problems, because not everything is
set up as it would when actually visiting the file.  Why is
with-temp-buffer necessary for the benchmarks?

But if it turns out that a query doesn't depend on the buffer
contents, I think this is a moot point, and the major mode could
compile the query just once when its first loaded.

Thanks.