Re: Using incremental parsing in Emacs

[arthur miller <arthur.miller@live.com>]

Eli Zaretskii <eliz@gnu.org> writes:

> On January 6, 2020 6:39:02 AM GMT+02:00, HaiJun Zhang <netjune@outlook.com> wrote:
>> Could someone explain how the jit-lock in Emacs works in the following
>> case?
>> 
>> 1. emacs -Q
>> 2. open the attachment file and goto end of buffer
>> 3. M-x desktop-save and quit emacs
>> 4. emacs -Q
>> 5. M-x desktop-read
>> 
>> I see the buffer is fontified correctly. Does it parse the whole
>> buffer?
>> 
>> 
>> 在 2020年1月6日 +0800 AM11:42，Eli Zaretskii <eliz@gnu.org>，写道：
>> > > From: arthur miller <arthur.miller@live.com>
>> > > CC: "monnier@iro.umontreal.ca" <monnier@iro.umontreal.ca>,
>> "alan@idiocy.org"
>> > > <alan@idiocy.org>, "emacs-devel@gnu.org" <emacs-devel@gnu.org>
>> > > Date: Sun, 5 Jan 2020 20:28:43 +0000
>> > >
>> > > When I said batch-processing I ment processing a file or chunk of
>> buffer (region) just before it is to be
>> > > displayed to a user. Same for other "insertions" from macro
>> expansions or similar.
>> >
>> > Then we always do "batch processing", because the display engine has
>> > no good idea what exactly changed in the buffer. So it always
>> > processes some minimal chunk of text that it can prove to itself
>> that
>> > the changes were all inside that chunk.
>> >
>> > > I also don't think teee-sitter is needed for syntax coloring.
>> Tree-sitter seems to be very expensive regex engine
>> > > in that case.
>> >
>> > They claim to be less expensive than regexp-based coloring,
>> especially
>> > with very long lines.
>> >
>
> We never parse the whole buffer, only its chunk that is slightly larger than what would be actually displayed in a window.
>
> This works by the display engine calling the fontification-functions for the
> buffer text it is about to display, whenever it finds a chunk of text whose
> 'fontified' text property is nil.

Thanks for the explanation.

Can I ask another related think: if I would to break the buffer
in chunks to send it to different threads, how can I find an
'edge' (in absence of better term) of an expression?

Say I happened to make a split in the middle of a comment, or some
expression, is there already something I can use to figure out
how to adjust split so I break on whole expressions, and not in
the middle?

About tree-sitter, I ment more in terms of RAM, it must cost to keep
all those AST nodes in RAM. In general it will keep entire file as an
AST copy in RAM.

But they are certainly more effective in terms of CPU then regular
expressions since tree-sitter seems to do only minimal work needed
when updating the AST, while regular expressions are more of a brute
force approach.

If you can use tree-sitter for other purposes then just syntax colouring
then tree-sitter might be definitely be a winner.