Re: Tree-sitter integration on feature/tree-sitter

[Eli Zaretskii <eliz@gnu.org>]

> From: Yoav Marco <yoavm448@gmail.com>
> Cc: casouri@gmail.com, emacs-devel@gnu.org
> Date: Thu, 12 May 2022 19:26:50 +0300
> 
> How I understand it, if it takes 23.474s to fontify 2332 times without
> query caching and 0.037s with, then 99.7% of the time is spent in
> recompiling the same query, or (23.474 - 0.037)/2332 = 10ms per
> fontification.

Yes, and 10 ms is negligibly short.  So, while the relative speedup is
very significant, I still don't see any reason for caching the
queries.

But maybe we should make this discussion more concrete.  Can you show
the queries and explain how they are produced from the font-lock rules
(or whatever else they are produced from)?  How many different queries
do we expect to have in a garden-variety major mode for a PL, and what
do they depend on?

> Explaination for the whole table:
> 
> |   |                     | font-lock | TS sexp |     TS | TS query reuse |
> | 1 | xdisp.c all at once |    12.886 |   0.031 |  0.016 |          0.017 |
> | 2 | 20 × 512c           |     0.273 |   0.214 |  0.209 |          0.000 |
> | 3 | 512c to end         |       4m+ |  24.177 | 23.474 |          0.037 |
> 
> Rows:
> - Benchmark 1 xdisp.c all at once: run font-lock-font-lock-fontify-region
>   on the entire buffer once
> - Benchmark 2 20 × 512c: fontify the next 512 characters 20 times
> - Benchmark 2 20 × 512c: fontify the next 512 characters until the
>   buffer ends

Thanks.  I think these benchmarks are not very useful.  Representative
benchmarks I can think of are:

  . the time it takes to visit xdisp.c and display the first window-full
  . visit xdisp.c, then immediately go to its end
  . C-v in xdisp.c (repeat many times to see how much a single C-v
    takes)

> >> >> If we expose "compiled query” we don’t need to cache them either.
> >> >
> >> > Then the Lisp program will have to do that, which is even worse,
> >> > because the problems I described will now have to be solved by Lisp
> >> > application programmers, each time anew.
> >>
> >> Will they? They'd just need to compile their queries once, when defining
> >> them or when setting treesit-font-lock-defaults.
> >
> > And decide when to discard them.
> 
> I thought garbage collection could take care of that. Is that
> problematic?

GC can take care of queries that the Lisp program no longer needs, but
the Lisp program should first decide that it no longer needs them.
Like stop referencing them in any data structure.