Re: Tree-sitter integration on feature/tree-sitter

[Yoav Marco <yoavm448@gmail.com>]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Yoav Marco <yoavm448@gmail.com>
>> Cc: casouri@gmail.com, emacs-devel@gnu.org
>> Date: Thu, 12 May 2022 19:26:50 +0300
>>
>> How I understand it, if it takes 23.474s to fontify 2332 times without
>> query caching and 0.037s with, then 99.7% of the time is spent in
>> recompiling the same query, or (23.474 - 0.037)/2332 = 10ms per
>> fontification.
>
> Yes, and 10 ms is negligibly short.  So, while the relative speedup is
> very significant, I still don't see any reason for caching the
> queries.
>
> But maybe we should make this discussion more concrete.  Can you show
> the queries and explain how they are produced from the font-lock rules
> (or whatever else they are produced from)?  How many different queries
> do we expect to have in a garden-variety major mode for a PL, and what
> do they depend on?

So first of all, query is kind of an aggregate term, since one query
string/sexp can contain many "query patterns". I expect most major modes
to have one big query string/sexp, and maybe a handful more that are
optional to users. treesit allows you to set as many query strings/sexps
as you want for syntax highlighting. Outside of that, queries are also
how packages like evil-textobj-tree-sitter work, with the backend of the
elisp-tree-sitter which uses a dynamic module.

Queries are specific to the parse tree and therefore to the parser. Most
parsers have a queries/highlights.scm file in their repo, and
tree-sitter-langs contains a bunch of these:

> https://github.com/emacs-tree-sitter/tree-sitter-langs/#readme
>
> Highlighting query patterns for a language are in the file
> queries/<lang>/highlights.scm. Most of them are intentionally
> different from those from upstream repositories, which are more geared
> towards GitHub’s use cases. We try to be more consistent with Emacs’s
> existing conventions. (For some languages, this is WIP, so their
> patterns may look similar to upstream’s.)

The query I used in the benchmarks is tree-sitter-langs's
queries/c/highlights.scm, which is a rather big file. One thing to check
that I only thought of now is how long it takes with treesit having to
compile and run multiple queries.

>> Explaination for the whole table:
>>
>> |   |                     | font-lock | TS sexp |     TS | TS query reuse |
>> | 1 | xdisp.c all at once |    12.886 |   0.031 |  0.016 |          0.017 |
>> | 2 | 20 × 512c           |     0.273 |   0.214 |  0.209 |          0.000 |
>> | 3 | 512c to end         |       4m+ |  24.177 | 23.474 |          0.037 |
>>
>> Rows:
>> - Benchmark 1 xdisp.c all at once: run font-lock-font-lock-fontify-region
>>   on the entire buffer once
>> - Benchmark 2 20 × 512c: fontify the next 512 characters 20 times
>> - Benchmark 2 20 × 512c: fontify the next 512 characters until the
>>   buffer ends
>
> Thanks.  I think these benchmarks are not very useful.  Representative
> benchmarks I can think of are:
>
>   . the time it takes to visit xdisp.c and display the first window-full
>   . visit xdisp.c, then immediately go to its end
>   . C-v in xdisp.c (repeat many times to see how much a single C-v
>     takes)

Okay, we can try that. What's the proper way to trigger a "natural
fontification" as would occur in the GUI without opening an interactive
session? I'd rather use the groundwork that's actually used by users,
and not get stuff like the JIT chunck size wrong. In general I'm not too
familiar with that part of Emacs; the benchmarks up to now used
with-temp-buffer, would that suffice for these new benchmarks?

>> I thought garbage collection could take care of that. Is that
>> problematic?
>
> GC can take care of queries that the Lisp program no longer needs, but
> the Lisp program should first decide that it no longer needs them.
> Like stop referencing them in any data structure.

Is that a problem? If anyone's generating queries and putting them in
lists, that would be a problem whether they're strings or compiled
objects.

 - Yoav