More font-lock faces for tree-sitter

More font-lock faces for tree-sitter

[Randy Taylor]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

Hi.

Since tree-sitter will allow for more fine-grained syntax highlighting, I think it would be useful to define more faces to give the user more customization for syntax highlighting.
I already make use of these for my own queries.

Here are some that I had in mind:
- font-lock-escape-face
- For escape characters in strings.
- font-lock-property-face
- Usually an object property, like a member in a class or struct.
- font-lock-punctuation-face
- In addition, we could get even more specific: font-lock-punctuation-{delimiter, bracket, special}
- font-lock-number-face
- font-lock-operator-face

We can certainly get even more fine-grained, but I think this is a good start and wanted to kick off the discussion. Feel free to offer more suggestions.
Happy to send a patch for the above if interested.

[-- Attachment #2: Type: text/html, Size: 1620 bytes --]

Re: More font-lock faces for tree-sitter

[Eli Zaretskii]

> Date: Fri, 28 Oct 2022 02:26:45 +0000
> From: Randy Taylor <dev@rjt.dev>
> 
> Since tree-sitter will allow for more fine-grained syntax highlighting, I think it would be useful to define more
> faces to give the user more customization for syntax highlighting.
> I already make use of these for my own queries.
> 
> Here are some that I had in mind:
> - font-lock-escape-face
>   - For escape characters in strings.
> - font-lock-property-face
>   - Usually an object property, like a member in a class or struct.
> - font-lock-punctuation-face
>   - In addition, we could get even more specific: font-lock-punctuation-{delimiter, bracket, special}
> - font-lock-number-face
> - font-lock-operator-face
> 
> We can certainly get even more fine-grained, but I think this is a good start and wanted to kick off the
> discussion. Feel free to offer more suggestions.
> 
> Happy to send a patch for the above if interested.

Feel free to send a patch, and thanks.

Re: More font-lock faces for tree-sitter

[Stefan Monnier]

> Here are some that I had in mind:
> - font-lock-escape-face
> - For escape characters in strings.
> - font-lock-property-face
> - Usually an object property, like a member in a class or struct.
> - font-lock-punctuation-face
> - In addition, we could get even more specific: font-lock-punctuation-{delimiter, bracket, special}
> - font-lock-number-face
> - font-lock-operator-face

Note that most of the above faces relate to the lexical category of the
element, i.e. things that are usually very easy to handle with regexps
and which thus aren't made significantly easier by tree-sitter (whose
strength is to recognize actual syntactic structures).

Where tree-sitter shines is for example when you need to distinguish the
two different kinds of commas in a code like:

    x = foo (y++, (z++, y+z));


-- Stefan

Re: More font-lock faces for tree-sitter

[Stefan Kangas]

Randy Taylor <dev@rjt.dev> writes:

> Here are some that I had in mind:
> - font-lock-escape-face
> - For escape characters in strings.
> - font-lock-property-face
> - Usually an object property, like a member in a class or struct.
> - font-lock-punctuation-face
> - In addition, we could get even more specific: font-lock-punctuation-{delimiter, bracket, special}
> - font-lock-number-face
> - font-lock-operator-face

I'm not sure we should rush to add a ton of new faces, just because we
can.  Language modes can still add new faces themselves, so the
discussion here is about which set of default faces we should have.

For example, is there any benefit to adding `font-lock-number-face'?
I don't think I've seen many language modes doing that so far.  Is there
a high demand for it?  Is there a big benefit?  Personally, I don't feel
confident answering that.

Adding a face, especially a default one, is an implicit invitation to
theme developers to use it.  If we add too many faces, we risk, in the
worst case, seeing a proliferation of angry fruit salad.

This feature is not even merged to master yet, so perhaps we should take
a more careful approach:

Time and experience will let us see how tree-sitter is used in practice.
If many language modes end up adding a face for something, we can say:
okay, now we have proof that this highlighting is generally useful.
Let's add a corresponding built-in face that everyone can inherit from.

To my mind, doing it the other way around risks putting the cart before
the horse.  Just my two cents.

Re: More font-lock faces for tree-sitter

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Fri, 28 Oct 2022 08:10:31 -0700
> 
> For example, is there any benefit to adding `font-lock-number-face'?
> I don't think I've seen many language modes doing that so far.  Is there
> a high demand for it?  Is there a big benefit?  Personally, I don't feel
> confident answering that.

Other IDEs do provide this, AFAIR.

> Adding a face, especially a default one, is an implicit invitation to
> theme developers to use it.  If we add too many faces, we risk, in the
> worst case, seeing a proliferation of angry fruit salad.

That's where fontification levels should come in to give users control
on how angry their salad is.

Re: More font-lock faces for tree-sitter

[Randy Taylor]

Stefan Kangas <stefankangas@gmail.com> writes:

> I'm not sure we should rush to add a ton of new faces, just because we
> can.  Language modes can still add new faces themselves, so the
> discussion here is about which set of default faces we should have.

> For example, is there any benefit to adding `font-lock-number-face'?
> I don't think I've seen many language modes doing that so far.  Is there
> a high demand for it?  Is there a big benefit?  Personally, I don't feel
> confident answering that.

Any benefit? That depends on who you ask. I already make use of all of
these, have for quite some time, and would be very sad without them. I
was planning to contribute C and C++ syntax highlighting support for
tree-sitter (and maybe more languages if time permits), and was hoping
to use these faces.

Without these faces I will need to define them myself in my own config
and use my own queries, which is really annoying IMO.

The big benefit is that we are empowering users to customize syntax
highlighting at a more fine-grained level to their own liking, and I
think that's a good thing.

> Adding a face, especially a default one, is an implicit invitation to
> theme developers to use it.  If we add too many faces, we risk, in the
> worst case, seeing a proliferation of angry fruit salad.

I don't see the problem with that. Theme developers will make the
themes that they want (and I for one am a fan of angry fruit
salad). And users can customize those faces themselves anyway, so
what's the problem?

Besides, theme developers can already do that can't they?

> This feature is not even merged to master yet, so perhaps we should take
> a more careful approach:
> Time and experience will let us see how tree-sitter is used in practice.
> If many language modes end up adding a face for something, we can say:
> okay, now we have proof that this highlighting is generally useful.
> Let's add a corresponding built-in face that everyone can inherit from.

We already know how tree-sitter is used in practice.

neovim has had tree-sitter support for quite some time now and
supports many more than these faces. They even go down to the level of
letting you customize loops and conditionals differently (which I'm
not proposing - like you say, we can wait and see for that stuff).

The same applies to the Emacs tree-sitter dynamic module
implementation (which has been around since 2020), which I and many
others have been using since it came out.

I agree that we should be careful about which faces we add, but I
think the faces that I listed are pretty general and apply to most
languages, so it makes sense to include them.

Re: More font-lock faces for tree-sitter

[Yuan Fu]

> On Oct 28, 2022, at 8:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Stefan Kangas <stefankangas@gmail.com>
>> Date: Fri, 28 Oct 2022 08:10:31 -0700
>> 
>> For example, is there any benefit to adding `font-lock-number-face'?
>> I don't think I've seen many language modes doing that so far.  Is there
>> a high demand for it?  Is there a big benefit?  Personally, I don't feel
>> confident answering that.
> 
> Other IDEs do provide this, AFAIR.
> 
>> Adding a face, especially a default one, is an implicit invitation to
>> theme developers to use it.  If we add too many faces, we risk, in the
>> worst case, seeing a proliferation of angry fruit salad.
> 
> That's where fontification levels should come in to give users control
> on how angry their salad is.
> 

More font-lock faces would be nice, but hopefully they are not too random (and we don’t bikeshed on them), and new ones should inherit from existing ones, so theme authors don’t need to update their theme.

Yuan