bug#69625: 30.0.50; [PATCH] rust-ts-mode doesn't fontify some enum

bug#69625: 30.0.50; [PATCH] rust-ts-mode doesn't fontify some enum

[Yuan Fu]

X-Debug-CC: dev@rjt.dev <mailto:dev@rjt.dev>

(I lied a little bit about on the [PATCH] part: I have a solution but didn’t turn it into a patch yet.)

The problem is follows: given the rust code below, some enum are not fontified with type face under font lock level 3, and those enum are fontified as function or variable under font lock level 4.

fn main() {
    func(MyEnum::VariantA(0));
    func(MyEnum::VariantB);
    func(VariantC);
    func(VariantD(0));
}

VariantA and VariantB are fontified correctly, but VariantC and VariantD are not.

I think a simple rule that fontifies every capitalized identifier would fix this. But I don’t know if that’ll create other problem. AFAIK capitalized identifier is always some type in rust, right?

This is first reported on rust-mode’s GitHub repo: https://github.com/rust-lang/rust-mode/issues/518

Yuan

bug#69625: 30.0.50; [PATCH] rust-ts-mode doesn't fontify some enum

[Dmitry Gutov]

On 08/03/2024 06:43, Yuan Fu wrote:
> The problem is follows: given the rust code below, some enum are not
> fontified with type face under font lock level 3, and those enum are
> fontified as function or variable under font lock level 4.
> 
> fn main() {
>      func(MyEnum::VariantA(0));
>      func(MyEnum::VariantB);
>      func(VariantC);
>      func(VariantD(0));
> }
> 
> VariantA and VariantB are fontified correctly, but VariantC and VariantD
>   are not.
> 
> I think a simple rule that fontifies every capitalized identifier would
> fix this. But I don’t know if that’ll create other problem. AFAIK
> capitalized identifier is always some type in rust, right?

This might be more of an issue with highlighters order. As you can see, 
level 3 fontifies these as 'type' already, and does that because the 
identifiers are capitalized.

Some level 4 rules probably either come earlier than where they should 
be (thus applying before the ones with the "capitalized" matcher), or 
they should use a "not capitalized" matcher if moving them down would 
cause other problems.

bug#69625: 30.0.50; [PATCH] rust-ts-mode doesn't fontify some enum

[Randy Taylor]

On Thursday, March 7th, 2024 at 23:43, Yuan Fu <casouri@gmail.com> wrote:
> 
> X-Debug-CC: dev@rjt.dev mailto:dev@rjt.dev

Should be X-Debbugs-CC ;).

> 
> 
> (I lied a little bit about on the [PATCH] part: I have a solution but didn’t turn it into a patch yet.)
> 
> The problem is follows: given the rust code below, some enum are not fontified with type face under font lock level 3, and those enum are fontified as function or variable under font lock level 4.
> 
> fn main() {
> func(MyEnum::VariantA(0));
> func(MyEnum::VariantB);
> func(VariantC);
> func(VariantD(0));
> }
> 
> VariantA and VariantB are fontified correctly, but VariantC and VariantD are not.
> 
> I think a simple rule that fontifies every capitalized identifier would fix this. But I don’t know if that’ll create other problem. AFAIK capitalized identifier is always some type in rust, right?
> 

For VariantA and VariantD, these are constructors being used as functions,
so I think they are technically being highlighted correctly at level 4.
Whether or not that is desired is another question - I think that is simply
a matter of opinion/preference.
VariantA gets highlighted as a type and not a function at level 3 because that
level doesn't support functions, but does support types. Maybe that could be
considered a bug in that it shouldn't be highlighted at all for level 3?
I'm not sure how worth it would be to do something about that though, or how
easy.

For VariantC, our (and tree-sitter's) best guess is that it's a variable.
We can't really know it's a type without guessing - like assuming a capitalized
identifier is a type, and I don't think that's something we can assume. People
can have capitalized functions and variables even if that goes against Rust's
usual style. Perhaps as a compromise we could introduce a variable (or something)
that lets the user specify that all capitalized identifiers should be treated as
types? Maybe it even makes sense to default it to that behaviour since I believe
most Rust code follows that style.

I don't really think this is a tree-sitter problem to solve but more so an LSP one:
tree-sitter can't know that all of these are enums based on how they are used.
As was mentioned in that GitHub thread, LSP with semantic tokens is the way to go
for the most accuracy. But even with semantic tokens, how to highlight enum variants
being used as functions comes down to opinion methinks.

> This is first reported on rust-mode’s GitHub repo: https://github.com/rust-lang/rust-mode/issues/518
> 
> Yuan

bug#69625: 30.0.50; [PATCH] rust-ts-mode doesn't fontify some enum

[Dmitry Gutov]

On 09/03/2024 05:50, Randy Taylor wrote:
> VariantA gets highlighted as a type and not a function at level 3 because that
> level doesn't support functions, but does support types. Maybe that could be
> considered a bug in that it shouldn't be highlighted at all for level 3?

Probably.

> I'm not sure how worth it would be to do something about that though, or how
> easy.

Same. I haven't looked into it.

> For VariantC, our (and tree-sitter's) best guess is that it's a variable.
> We can't really know it's a type without guessing - like assuming a capitalized
> identifier is a type, and I don't think that's something we can assume. People
> can have capitalized functions and variables even if that goes against Rust's
> usual style. Perhaps as a compromise we could introduce a variable (or something)
> that lets the user specify that all capitalized identifiers should be treated as
> types? Maybe it even makes sense to default it to that behaviour since I believe
> most Rust code follows that style.

We do have some rules already that are based off whether an identifier 
is capitalized. I.e. some for use_as_clause, and another for 
highlighting an identifier with font-lock-constant-face if it's 
ALL_CAPS. So it might be logical to carry on with that approach.

If the style is consistent enough across the ecosystem, of course.

We could add a variable too, but that'd make the rules more complex so 
it would be helpful to understand first whether there are users who 
would benefit.

bug#69625: 30.0.50; [PATCH] rust-ts-mode doesn't fontify some enum

[Randy Taylor]

On Thursday, March 14th, 2024 at 21:52, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> 
> On 09/03/2024 05:50, Randy Taylor wrote:
> 
> > VariantA gets highlighted as a type and not a function at level 3 because that
> > level doesn't support functions, but does support types. Maybe that could be
> > considered a bug in that it shouldn't be highlighted at all for level 3?
> 
> 
> Probably.
> 
> > I'm not sure how worth it would be to do something about that though, or how
> > easy.
> 
> 
> Same. I haven't looked into it.
> 
> > For VariantC, our (and tree-sitter's) best guess is that it's a variable.
> > We can't really know it's a type without guessing - like assuming a capitalized
> > identifier is a type, and I don't think that's something we can assume. People
> > can have capitalized functions and variables even if that goes against Rust's
> > usual style. Perhaps as a compromise we could introduce a variable (or something)
> > that lets the user specify that all capitalized identifiers should be treated as
> > types? Maybe it even makes sense to default it to that behaviour since I believe
> > most Rust code follows that style.
> 
> 
> We do have some rules already that are based off whether an identifier
> is capitalized. I.e. some for use_as_clause, and another for
> highlighting an identifier with font-lock-constant-face if it's
> ALL_CAPS. So it might be logical to carry on with that approach.
> 
> If the style is consistent enough across the ecosystem, of course.
> 

Indeed, but those rules (minus the ALL_CAPS one) don't apply to all
identifiers but rather specific kinds (most if not all applying to
use declarations).

> We could add a variable too, but that'd make the rules more complex so
> it would be helpful to understand first whether there are users who
> would benefit.

Agreed. This is the first I've heard it mentioned.

bug#69625: 30.0.50; [PATCH] rust-ts-mode doesn't fontify some enum

[Yuan Fu]

> On Mar 14, 2024, at 6:52 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> On 09/03/2024 05:50, Randy Taylor wrote:
>> VariantA gets highlighted as a type and not a function at level 3 because that
>> level doesn't support functions, but does support types. Maybe that could be
>> considered a bug in that it shouldn't be highlighted at all for level 3?
> 
> Probably.
> 
>> I'm not sure how worth it would be to do something about that though, or how
>> easy.
> 
> Same. I haven't looked into it.
> 
>> For VariantC, our (and tree-sitter's) best guess is that it's a variable.
>> We can't really know it's a type without guessing - like assuming a capitalized
>> identifier is a type, and I don't think that's something we can assume. People
>> can have capitalized functions and variables even if that goes against Rust's
>> usual style. Perhaps as a compromise we could introduce a variable (or something)
>> that lets the user specify that all capitalized identifiers should be treated as
>> types? Maybe it even makes sense to default it to that behaviour since I believe
>> most Rust code follows that style.
> 
> We do have some rules already that are based off whether an identifier is capitalized. I.e. some for use_as_clause, and another for highlighting an identifier with font-lock-constant-face if it's ALL_CAPS. So it might be logical to carry on with that approach.
> 
> If the style is consistent enough across the ecosystem, of course.
> 
> We could add a variable too, but that'd make the rules more complex so it would be helpful to understand first whether there are users who would benefit.

For some reason I couldn’t see Randy’s messages. So sorry if I missed anything. I suggest we go ahead with guessing and add the variable if enough people complain. Personally speaking I believe the vast majority of Rust community wouldn’t write Rust code with capitalized variable and non-capitalized types. The Rust community is very much inclined to the standard style, AFAICT.

Yuan