unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
From: Ergus <spacibba@aol.com>
To: Eli Zaretskii <eliz@gnu.org>
Cc: ofv@wanadoo.es, emacs-devel@gnu.org
Subject: Re: cc-mode fontification feels random
Date: Sat, 12 Jun 2021 13:01:03 +0200	[thread overview]
Message-ID: <20210612110103.u6kuh3d5vahxmxlt@Ergus> (raw)
In-Reply-To: <83sg1n8t71.fsf@gnu.org>

On Sat, Jun 12, 2021 at 09:58:58AM +0300, Eli Zaretskii wrote:
>> Date: Sat, 12 Jun 2021 03:08:44 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: emacs-devel@gnu.org
>>
>> BTW: Eli was concerned about the extra copy of the buffer text to send
>> it to tree-sitter. In this case the time to memcopy an array with all
>> xdisp text is ~0.00085 seconds.
>
>If the intent is to use buffer-(sub)string, then you forget the price
>of consing.  That would trigger frequent GC cycles, which will all but
>kill the otherwise fast performance.
>
>> Any way if we don't want the copy we can use
>> ts_parser_set_included_ranges to exclude the gap and pass the text
>> pointer directly without any copy.
>
>I hope someone will try that and report the results.
>
>The other design issue with TS integration is that I'd like it to plug
>into the JIT font-lock interface of the display engine, so that we
>don't unnecessarily fontify parts of the buffer that won't be
>displayed, and always do fontify the parts that will be. 

If I understand something about our cc-mode functionalities (and many of
those functionalities we don't want to loose like indentation and code
navigation). Probably the "right" way to use tree-sitter (maybe Alan
wants give a more precise technical description) is not only fontify but
use the tree information to add contextual information to the text
(something that I think cc-mode does.) And then let font-lock do the
magic.

The tree-sitter tree is basically contextual information, and (for
example) if we have processed the whole buffer and we already have the
tree, then scrolling won't need to parse anything, adding or removing
text is a localized modification, so with the previous tree we can
re-parse only the modified region. The choice may be then if we
propertize the text of the whole buffer or just the visible region OR if
we want to "propertize on demand".

This will save us from the hard parsing in cc-mode to fontify "on the
fly".


> I don't
>really care if TS actually processes a much larger chunk of text, if
>it does that quickly enough, but processing the resulting faces will
>take time on the Emacs side, and that is better avoided. 

But then we won't get all the contextual information we need for
indentation, code navigation or fold the code right?

so we'll be still "sub-utilizing" the tree sitter features that may give
useful functionalities we already have in cc-mode, and we may also like
to have in other more "limited" modes.

> More
>importantly, integration into JIT font-lock machinery means we don't
>need to use other hooks, which is a step back, since using such hooks
>for fontification was already shown to have serious problems in pre-21
>Emacs: they don't always catch all the changes which require
>re-fontification.
>
I see two approaches here:

1) add the tree-sitter properties/faces to the buffer text (fully or
partially on the visible regions)

2) use the tree-sitter information directly from the tree and add the
visible properties from there.

This second one will require a more complete api of tree-sitter
functions exposed to elisp, but in my opinion it worth it in accuracy,
speed and simplicity (a single API to rule them all). And to support
many languages we don't actually have like rust or the fancy C++ > 11. 

+

Remember that TS has the partial parsing options (specifying the regions
to parse), the re-parsing option (using a previous tree for the same
buffer as a hint which reduces the times abruptly), or even a tree
comparison function that produces a new tree with the differences with
the "hint" tree to know what needs to be updated.

Plus all the navigation function like find parent or child nodes,
parsing error handling, iterate over nodes and so on.




  reply	other threads:[~2021-06-12 11:01 UTC|newest]

Thread overview: 274+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-04  3:16 cc-mode fontification feels random Daniel Colascione
2021-06-04  6:10 ` Eli Zaretskii
2021-06-04  7:10   ` Theodor Thornhill
2021-06-04 10:08     ` João Távora
2021-06-04 10:39       ` Eli Zaretskii
2021-06-04 10:59         ` Philipp
2021-06-04 11:05           ` João Távora
2021-06-04 11:22             ` Eli Zaretskii
2021-06-04 12:44               ` Dmitry Gutov
2021-06-04 13:46               ` João Távora
2021-06-04 14:11                 ` Eli Zaretskii
2021-06-04 11:18           ` Eli Zaretskii
2021-06-04 16:43       ` Jim Porter
     [not found]         ` <83k0n9l9pv.fsf@gnu.org>
2021-06-04 19:41           ` Jim Porter
2021-06-04 19:53             ` Eli Zaretskii
2021-06-04 20:05               ` Jim Porter
2021-06-04 20:11                 ` Joost Kremers
2021-06-05  6:51                   ` Eli Zaretskii
2021-06-05 10:14                     ` Joost Kremers
2021-06-05 11:31                       ` Eli Zaretskii
2021-06-05 12:12                         ` Joost Kremers
2021-06-05 13:23                     ` Stefan Monnier
2021-06-05 17:08                       ` Óscar Fuentes
2021-06-05 17:31                         ` Stefan Monnier
2021-06-05 17:32                         ` Eli Zaretskii
2021-06-05 18:46                     ` João Távora
2021-06-05  6:41                 ` Eli Zaretskii
2021-06-05  9:32                   ` João Távora
2021-06-05  9:59                     ` Ergus
2021-06-05 11:29                       ` Eli Zaretskii
2021-06-05 11:55                         ` Daniel Colascione
2021-06-05 12:27                           ` Eli Zaretskii
2021-06-05 17:59                             ` Jim Porter
2021-06-05 18:56                               ` Daniel Martín
2021-06-05 12:43                         ` Ergus
2021-06-05 13:59                       ` Remote GUI Emacs really works (was: cc-mode fontification feels random) Óscar Fuentes
2021-06-05 11:25                     ` cc-mode fontification feels random Eli Zaretskii
2021-06-05  9:46                   ` Ergus
2021-06-05 11:27                     ` Eli Zaretskii
2021-06-04 20:14               ` Yuri Khan
2021-06-04 10:25     ` Eli Zaretskii
2021-06-04 10:05   ` Daniel Colascione
2021-06-04 10:22     ` Eli Zaretskii
2021-06-04 10:34       ` João Távora
2021-06-04 10:43         ` Eli Zaretskii
2021-06-04 18:25         ` Stefan Monnier
2021-06-04 18:36           ` Daniel Colascione
2021-06-04 19:11             ` Eli Zaretskii
2021-06-04 19:16               ` Daniel Colascione
2021-06-04 19:26                 ` Eli Zaretskii
2021-06-04 19:33                   ` Daniel Colascione
2021-06-04 19:51                     ` Eli Zaretskii
2021-06-05  0:29             ` Stefan Monnier
2021-06-05  6:32               ` Eli Zaretskii
2021-06-04 19:07           ` Eli Zaretskii
2021-06-04 19:26             ` Daniel Colascione
2021-06-04 19:32               ` Eli Zaretskii
2021-06-04 10:41       ` Eli Zaretskii
2021-06-04 10:42 ` Ergus
2021-06-04 15:54 ` Alan Mackenzie
2021-06-04 18:30   ` Daniel Colascione
2021-06-06 11:37     ` Alan Mackenzie
2021-06-06 11:57       ` Eli Zaretskii
2021-06-06 12:27         ` Alan Mackenzie
2021-06-06 12:44           ` Eli Zaretskii
2021-06-06 14:19             ` Alan Mackenzie
2021-06-06 17:06               ` Eli Zaretskii
2021-06-06 17:44       ` Stefan Monnier
2021-06-06 18:00         ` Eli Zaretskii
2021-06-06 18:18           ` Stefan Monnier
2021-06-06 18:33             ` Daniel Colascione
2021-06-06 20:24               ` Stefan Monnier
2021-06-06 20:27                 ` Daniel Colascione
2021-06-06 20:38                   ` Stefan Monnier
2021-06-06 19:03             ` Eli Zaretskii
2021-06-06 20:28               ` Stefan Monnier
2021-06-07  7:35                 ` martin rudalics
2021-06-07 13:20                   ` Stefan Monnier
2021-06-07 13:37                     ` Eli Zaretskii
2021-06-08  0:06                       ` Daniel Colascione
2021-06-08 15:16                       ` Stefan Monnier
2021-06-07 15:58                     ` martin rudalics
2021-06-08  4:01                     ` Richard Stallman
2021-06-08 15:29                       ` Stefan Monnier
2021-06-08 15:52                         ` Eli Zaretskii
2021-06-08 16:36                           ` Stefan Monnier
2021-06-08 18:11                             ` Daniel Colascione
2021-06-08 18:25                               ` Eli Zaretskii
2021-06-08 18:28                                 ` Daniel Colascione
2021-06-08 18:54                                   ` Eli Zaretskii
2021-06-09 18:22                                 ` Alan Mackenzie
2021-06-09 18:36                                   ` Eli Zaretskii
2021-06-09 18:51                                     ` Daniel Colascione
2021-06-09 19:04                                       ` Eli Zaretskii
2021-06-09 20:07                                       ` chad
2021-06-10  6:43                                         ` Eli Zaretskii
2021-06-09 20:17                                       ` Dmitry Gutov
2021-06-09 21:03                                     ` Alan Mackenzie
2021-06-10  2:21                                       ` Daniel Colascione
2021-06-10  6:55                                         ` Eli Zaretskii
2021-06-10  6:58                                           ` Daniel Colascione
2021-06-10  7:19                                             ` Eli Zaretskii
2021-06-10  6:39                                       ` Eli Zaretskii
2021-06-10 16:46                                         ` Alan Mackenzie
2021-06-10 17:01                                           ` Eli Zaretskii
2021-06-10 17:07                                             ` Daniel Colascione
2021-06-10 17:22                                               ` Eli Zaretskii
2021-06-10 17:33                                                 ` Daniel Colascione
2021-06-10 17:39                                                   ` Eli Zaretskii
2021-06-10 17:40                                                 ` Óscar Fuentes
2021-06-10 17:44                                                   ` Eli Zaretskii
2021-06-11 16:11                                                 ` Alan Mackenzie
2021-06-11 17:53                                                   ` Eli Zaretskii
2021-06-11 18:02                                                     ` Daniel Colascione
2021-06-11 18:22                                                       ` Eli Zaretskii
2021-06-11 18:28                                                         ` Daniel Colascione
2021-06-11 19:12                                                           ` Alan Mackenzie
2021-06-11 19:23                                                           ` Eli Zaretskii
2021-06-11 18:47                                                         ` Alan Mackenzie
2021-06-11 19:32                                                           ` Eli Zaretskii
2021-06-11 19:46                                                             ` Alan Mackenzie
2021-06-11 19:50                                                               ` Eli Zaretskii
2021-06-11 18:42                                                       ` Stefan Monnier
2021-06-11 19:31                                                         ` Eli Zaretskii
2021-06-11 19:57                                                           ` Stefan Monnier
2021-06-11 23:25                                                             ` Ergus
2021-06-11 23:52                                                               ` Óscar Fuentes
2021-06-12  1:08                                                                 ` Ergus
2021-06-12  3:20                                                                   ` Stefan Monnier
2021-06-12 11:07                                                                     ` Ergus
2021-06-12  6:58                                                                   ` Eli Zaretskii
2021-06-12 11:01                                                                     ` Ergus [this message]
2021-06-12 11:25                                                                       ` Eli Zaretskii
2021-06-12 15:04                                                                         ` Ergus
2021-06-12 15:16                                                                           ` Eli Zaretskii
2021-06-12 15:23                                                                             ` Ergus
2021-06-12 15:35                                                                               ` Eli Zaretskii
2021-06-12 14:00                                                                     ` Stefan Monnier
2021-06-12 14:20                                                                       ` Eli Zaretskii
2021-06-12 14:33                                                                         ` Stefan Monnier
2021-06-12 15:06                                                                           ` Eli Zaretskii
2021-06-12 15:46                                                                             ` Stefan Monnier
2021-06-12  6:50                                                                 ` Eli Zaretskii
2021-06-12  5:20                                                               ` Theodor Thornhill
2021-06-12 13:40                                                                 ` Stefan Monnier
2021-06-12 15:56                                                                   ` Theodor Thornhill
2021-06-12 16:59                                                                     ` Ergus
2021-06-12 17:51                                                                       ` Theodor Thornhill
2021-06-12 17:25                                                                     ` Stefan Monnier
2021-06-12 17:53                                                                       ` Theodor Thornhill
2021-06-12 17:54                                                                       ` Ergus
2021-06-12 18:02                                                                       ` Daniel Colascione
2021-06-12 18:39                                                                         ` Ergus
2021-06-12  6:38                                                             ` Eli Zaretskii
2021-06-12 13:44                                                               ` Stefan Monnier
2021-06-12 14:14                                                                 ` Eli Zaretskii
2021-06-11 20:06                                                           ` Alan Mackenzie
2021-06-12  6:44                                                             ` Eli Zaretskii
2021-06-12  8:00                                                               ` Daniel Colascione
2021-06-12  8:08                                                                 ` Eli Zaretskii
2021-06-12  9:31                                                                   ` Alan Mackenzie
2021-06-11 19:48                                                         ` Eli Zaretskii
2021-06-11 18:34                                                     ` Alan Mackenzie
2021-06-10 17:26                                               ` Óscar Fuentes
2021-06-10 17:39                                               ` andrés ramírez
2021-06-10 21:06                                           ` Stefan Monnier
2021-06-11  6:14                                             ` Eli Zaretskii
2021-06-10 15:16                                       ` Ergus
2021-06-10 15:34                                         ` Óscar Fuentes
2021-06-10 19:06                                           ` Ergus
2021-06-10 19:28                                             ` Eli Zaretskii
2021-06-10 21:56                                               ` Ergus
2021-06-10 15:59                                         ` Jim Porter
2021-06-10 21:02                                         ` Stefan Monnier
2021-06-11 20:21                                           ` Ergus
2021-06-11 20:27                                             ` Stefan Monnier
2021-06-11 20:37                                               ` Daniel Colascione
2021-06-11 20:52                                                 ` Stefan Monnier
2021-06-12  6:46                                                   ` Eli Zaretskii
2021-06-12  8:03                                                     ` Daniel Colascione
2021-06-12  8:13                                                       ` Eli Zaretskii
2021-06-12 13:51                                                       ` Stefan Monnier
2021-06-12  8:47                                                   ` Daniele Nicolodi
2021-06-12  8:57                                                     ` tomas
2021-06-12 14:04                                                     ` Stefan Monnier
2021-06-09 19:05                                   ` Daniel Colascione
2021-06-09 19:11                                     ` Eli Zaretskii
2021-06-09 20:20                                     ` Alan Mackenzie
2021-06-09 20:36                                       ` Stefan Monnier
2021-06-10  7:01                                         ` Daniel Colascione
2021-06-10  7:21                                           ` Eli Zaretskii
2021-06-10  2:21                                       ` Daniel Colascione
2021-06-19  9:25                                         ` Alan Mackenzie
2021-06-19 15:24                                           ` Alan Mackenzie
2021-07-09 14:06                                             ` Daniel Colascione
2021-07-11 18:12                                               ` Stephen Leake
2021-07-15 18:13                                                 ` Perry E. Metzger
2021-07-15 22:43                                                   ` Tree Sitter (was Re: cc-mode fontification feels random) Perry E. Metzger
2021-07-19 23:49                                                     ` Stephen Leake
2021-07-20 14:53                                                       ` Perry E. Metzger
2021-07-21  0:04                                                         ` Stephen Leake
2021-07-21  1:28                                                           ` Stefan Monnier
2021-07-21 14:43                                                             ` Perry E. Metzger
2021-07-21 16:21                                                               ` Daniel Colascione
2021-07-21 19:15                                                                 ` Perry E. Metzger
2021-07-22  1:16                                                                   ` Daniel Colascione
2021-07-22 13:18                                                                     ` Perry E. Metzger
2021-07-22 13:49                                                                     ` Yuan Fu
2021-07-24 20:05                                                                     ` [SPAM UNSURE] " Stephen Leake
2021-07-25  0:41                                                                       ` Daniel Colascione
2021-07-26  4:24                                                                         ` [SPAM UNSURE] " Stephen Leake
2021-07-25 18:01                                                                       ` Perry E. Metzger
2021-07-22 14:00                                                           ` Perry E. Metzger
2021-07-24  1:17                                                             ` Richard Stallman
2021-07-25 16:13                                                               ` Stephen Leake
2021-07-25 19:52                                                                 ` Ada (was Re: Tree Sitter) Perry E. Metzger
2021-07-26  5:05                                                                   ` Stephen Leake
2021-07-26  9:42                                                                     ` Stephen Leake
2021-07-26 14:01                                                                       ` Perry E. Metzger
2021-07-26 13:45                                                                     ` Perry E. Metzger
2021-07-27  0:26                                                                   ` Richard Stallman
2021-07-27 12:38                                                                     ` Perry E. Metzger
2021-07-26  2:23                                                                 ` Tree Sitter (was Re: cc-mode fontification feels random) John Yates
2021-07-24 19:59                                                             ` Stephen Leake
2021-07-24 21:21                                                               ` OFF-TOPIC: Ada availability (was: Tree Sitter) Óscar Fuentes
2021-07-25  7:31                                                                 ` tomas
2021-06-08 18:11                             ` cc-mode fontification feels random Eli Zaretskii
2021-06-08 21:25                               ` Stefan Monnier
2021-06-09  3:39                         ` Richard Stallman
2021-06-09  8:34                         ` martin rudalics
2021-06-09 13:14                           ` `open-paren-in-column-0-is-defun-start` (was: cc-mode fontification feels random) Stefan Monnier
2021-06-09 15:15                             ` Yuri Khan
2021-06-09 15:16                               ` Yuri Khan
2021-06-12 17:29                           ` cc-mode fontification feels random João Távora
2021-06-13  8:50                             ` martin rudalics
2021-06-13  9:05                               ` João Távora
2021-06-13  9:39                                 ` martin rudalics
2021-06-13 10:06                                   ` João Távora
2021-06-13 14:52                                     ` martin rudalics
2021-06-13 15:25                                       ` João Távora
2021-06-14  8:29                                         ` martin rudalics
2021-06-14  8:40                                           ` João Távora
2021-06-14  9:00                                             ` martin rudalics
2021-06-14  9:14                                               ` João Távora
2021-06-14 11:28                                           ` Eli Zaretskii
2021-06-14 14:39                                           ` Stefan Monnier
2021-06-15 22:38                                             ` Ergus
2021-06-07 12:08                 ` Eli Zaretskii
2021-06-08 15:22                   ` Stefan Monnier
2021-06-08 15:46                     ` Eli Zaretskii
2021-06-05 20:25   ` Dmitry Gutov
2021-06-06 11:53     ` Alan Mackenzie
2021-06-06 17:08       ` Dmitry Gutov
2021-08-30 18:50 ` [PATCH] " Alan Mackenzie
2021-08-30 19:03   ` Perry E. Metzger
2021-08-30 19:18     ` Alan Mackenzie
2021-08-30 19:25   ` Eli Zaretskii
2021-08-30 19:28     ` Daniel Colascione
2021-08-30 19:37       ` Eli Zaretskii
2021-08-30 20:11       ` Stefan Monnier
2021-08-31 10:54         ` Alan Mackenzie
2021-08-31 13:23           ` Eli Zaretskii
2021-08-31 16:02             ` Alan Mackenzie
2021-08-31 16:21               ` Eli Zaretskii
2021-08-31 16:46                 ` Alan Mackenzie
2021-08-31 17:02                   ` Eli Zaretskii
2021-08-31 18:56           ` Stefan Monnier
2021-08-31 21:17             ` Alan Mackenzie
2021-08-31 21:47               ` Stefan Monnier
2021-10-22 20:13               ` [Committed PATCH] " Alan Mackenzie
2021-10-24 20:18                 ` Alan Mackenzie
2021-08-31 13:18         ` [PATCH] " Eli Zaretskii
2021-08-30 20:03     ` Alan Mackenzie
2021-08-31 11:53       ` Eli Zaretskii

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.gnu.org/software/emacs/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210612110103.u6kuh3d5vahxmxlt@Ergus \
    --to=spacibba@aol.com \
    --cc=eliz@gnu.org \
    --cc=emacs-devel@gnu.org \
    --cc=ofv@wanadoo.es \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).