Re: Standardizing tree-sitter fontification features

[Yuan Fu <casouri@gmail.com>]

> On Nov 24, 2022, at 6:56 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> Basic tokens:
>> 
>> delimiter       ,.;
>> operator        = != ||
> 
> From the grammar point of view, both of these are simply infix thingies.
> Defining precisely the difference between them can be a bit more
> delicate, tho.  I guess intuitively the idea is that "operator"
> corresponds to some kind of run-time operation whereas "delimiter"
> serves only to figure out where things start and end but doesn't do
> anything in itself.

Maybe think in semantics? If it produces some value, it’s an operator, otherwise it’s a delimiter/punctuation. And I should have used == instead of = in the example. I was thinking of ==.

> 
> Not clear where the `=` used for definitions (as in "let x = foo in
> bar") should fall.  Same for the "," used to construct pairs
> or the conjunction/disjunction in Prolog written `,` and `;`
> respectively :-)

I think stuff like `:` in Haskell (I only know Haskell) can be considered operator, but `::` would be punctuation. With that said, I didn’t put much thought into delimiters, and it seems to be a slippery slope (as shown in your `=` example). Maybe we should limit it to the narrowest scope for now, just things like , ; that truly are delimiting things.

> 
>> string-interpolation    f"text {variable}"
>> escape-sequence         "\n\t\\"
>> function                every function identifier
> 
> I think we definitely need to distinguish a reference/use of a function
> from a definition of a function.  I do want my function identifiers
> highlighted in my function definitions but *not* in my function calls.

That’s when assignment/definition come in. See my reply to Randy.

> 
>> variable                every variable identifier
> 
> Same here.
> [ Note that in many languages (e.g. Scheme and C), functions and
>  "variables" are the same (i.e. Lisp-1 in the world of Lisp).  ]

We can say that if the symbol is used as a function, highlight in function face, and if it is used as a value, highlight in variables face.

> 
>> type                    every type identifier
> 
> And same here as well.
> [ I will spare you the discussion of what should happen for dependently
>  typed languages where types are "normal" values.  ]

“Anything capitalized in type face”, I guess ;-)

> 
>> Also, some of the features are very busy, it would be good if we can disable
>> they by default. The default value of font-lock-maximum-decoration is t,
>> meaning use everything, which is not very helpful...
> 
> If you compare the "style" of font-lock rules used until now to those
> provide in the new tree-sitter modes, I think it makes sense for
> tree-sitter modes to default to "medium" decorations.

Like setting font-lock-maximum-decoration to a number in major mode body, it its default value is t? That changes the meaning of t in font-lock-maximum-decoration. Can we change its default value to 3, and put the busy features at level 4? I need to survey existing major modes and see how many levels do they have right now. 

Yuan