bug#67061: [PATCH] Improve syntax highlighting for python-ts-mode

[Yuan Fu <casouri@gmail.com>]

> On Dec 16, 2023, at 4:26 PM, Denis Zubarev <dvzubarev@yandex.ru> wrote:
> 
> Sorry for the delayed response.
>  > For many Emacs users, they want a quieter or even minimal fontification.
>  I'm not against it. I just think that highlighting of an interpolation
> as a string is wrong. Is it possible to set quiet fontification in
> emacs-lisp mode, in such a way that `keywords' in doc-strings were
> fontified as a doc-string itself? I think it is similar to
> interpolation, it serves the purpose of separating different semantic
> elements from each other. IMHO, users who like quiet levels will benefit
> from interpolation highlighted differently.

I definitely can see your point, and it makes a lot of sense. I don’t really know people who want quieter fontification wants (probably both approach has their supporters) so can’t speak for them. But in general, I think it wouldn’t hurt to have the option.

>  > Instead of the third level, the check should use the value
> > treesit-font-lock-level. And it should check for each level smaller than
> > or equal to treesit-font-lock-level.
>  Done.

Thank you for your hard work! I’m just here talking and you went ahead and did all the work :-)

>  > Non-essential fontification like "var" shouldn't be
> > activated at that level. So I suggest to put it in the variable feature,
> > along with many other non-essential fontifications. (Variable feature is
> > placed at level 4.)
>  I added a new feature variable-definition for variables defined for local scopes and put it on the 3rd level.
> I also added rules to variable-definition feature for variables in list
> comprehension ( [var+1 for var in []] ) and as_pattern (with T as var:,
> except E as var:, case str() as var:).
>  I've noticed that vars in `for var1, (var2, var3) in []:` are highlighted by the rule from the assignment feature (specifically `pattern_list`, `tuple_pattern`).
> It seems easy to fix `pattern_list`, but not so easy for
> `tuple_pattern`, since this node may occur recursively.
> I didn't touch these rules for now.

Ok, makes sense.

>    Summary for all changes in the patch.
>  New feature variable-definition:
> `for var in range(3)`
> `[var+1 for var in []]`
> `with T as var:`
> `except E as var:`
> `case str() as var:`
> highlight var as font-lock-variable-name-face
>  assignment feature:
> var := 3 (named_expression)
> var *= 3 (augmented_assignment)
> Highlight var as font-lock-variable-name-face.
>  Make list_splat_pattern query more precise.
> list_splat_pattern may appear not only in assignments: var, *rest = call(),
> but in the parameter list too: def f(*args).
> Highlight args only for the first case in assignment feature.
>   type feature:
> Fontify built-ins (dict,list,etc.) as types when they are used in type hints.
> support nested union types, for example `Lvl1 | Lvl2[Lvl3[Lvl3], Lvl2]`.
> This structure is represented via nesting binary_operator and subscript nodes in the grammar.
> Function python--treesit-fontify-union-types iterates over all children and highlight identifier nodes.
>  Fontify base class names in the class definition: class Temp(Base1, pack0.Base2):
> Fontify class patterns in case statement: case [TempC() | bytes(b)]:
> Highlight the second argument as a type in isinstance/issubclass call:
> isinstance(var2, (str, dict, Type1)); issubclass(var1, int|str)
>  For all dotted names of a type highlight only the last part of the name,
> e.g. collections.abc.Iterator.
>   decorator feature:
> Highlight dotted names: @pytest.mark.skip
> Function python--treesit-fontify-dotted-decorator iterates over all nested attribute nodes and highlight identifier nodes.
>  When font-lock-level is set 4, `skip` had function-call face in: @pytest.mark.skip(reason='t')
> Add `:override t` to decorator feature to override function-call face.
>   string feature:
> Enable interpolation highlighting only if string-interpolation is
> presented on the enabled levels of treesit-font-lock-feature-list.
> Fix fontification of strings inside of f-strings interpolation,
> e.g. for f"beg {'nested'}" - 'nested' was not fontified as string.
>   function feature:
> Do not override the face of builtin functions (all, bytes etc.) with
> the function call face
>   keyword feature:
> Add "is not"  to the `python--treesit-keywords` list.

Thanks. I think the only thing that’s still up to discussion is the variable-definition rules. Others can be merged to emacs-29.

Yuan