bug#62951: 29.0.90; c-ts-mode: Incorrect fontification due to FOR_EACH_TAIL_SAFE

[Eli Zaretskii <eliz@gnu.org>]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 22 Apr 2023 17:28:25 -0700
> Cc: 62951@debbugs.gnu.org
> 
> >> I’m aware of this issue, but the truth is there isn’t a good solution to
> >> it. We need to recognize FOR_EACH_TAIL_SAFE (not hard) and fix arbitrary
> >> code after it (hard). In this case it’s a if statement, with macro calls
> >> and AND operation in it’s condition, it’s already three things we need
> >> to recognize and somehow handle. It can also be a for loop, a switch
> >> case, a function call, a while loop. If we want to fix FOR_EACH_TAIL we
> >> would need to handle every possible thing, at that point we might as
> >> well have wrote a parser :-)
> > 
> > Sorry, I don't understand the difficulties.  The body of FOR_EACH_TAIL
> > and a few similar macros we use could be on of the following:
> > 
> >  . a single simple statement
> >  . an 'if' clause
> >  . a 'while' loop
> >  . a 'do' loop
> >  . a 'for' loop
> >  . a brace-delimited block (this one already works, AFAICS, so we
> >    perhaps need not anything about it)
> > 
> > (In practice, only the first 2 and the last one are used, AFAICS.)
> > 
> > Doesn't tree-sitter tell us enough to figure out which of the above is
> > in the body?  If so, why would we need to write a full parser?
> 
> Ok, the full parser part is a bit exaggeration :-) But my point is it’s a lot of dirty code for a subpar result. Take
> 
> if (NILP (XCDR (tail)) && STRINGP (XCAR (tail)))
> 
> for example, it’s parsed as a function definition and all the tokens in the condition are parsed as weird things like macro declarator, error, function declarator, type, etc. And if the condition changes to something else, say it has another layer of function call, it’ll parse differently.

But the top-level construct is 'if', no?  Isn't that enough?

Also, can we detect the FOR_EACH_TAIL etc. macros themselves, and then
treat their body specially?

> > Please understand: good support for editing Emacs C sources is from my
> > POV imperative for c-ts-mode to gain traction.  One of my gripes about
> > CC Mode was insufficient support for our macro system and for various
> > GCC attributes; that improved recently to some extend, but not enough,
> > and at a price of introducing ugly lists of strings that cause trouble
> > when used in file-local variables.  We must do better in c-ts-mode!
> > 
> > So please make an effort of providing reasonable built-in solutions
> > for these idiosyncrasies of the Emacs C sources, conditioned on the
> > new variable c-ts-mode-emacs-sources-support, at least for those of
> > them that are used widely enough.  It is very important.
> 
> What do you think of extending the parser to support these macros instead? (So we fork tree-sitter-c.)

This goes against the purpose of using tree-sitter and its grammars.
I don't think we should maintain and develop language grammars,
especially since the production of the C sources from them needs
non-trivial system resources and additional tools.

Maybe coming up with a way of extending the C grammar in some more
general way, and then submitting the changes to the tree-sitter-c
developers for inclusion would be OK.

But I very much hope that we could support these macros at a lower
cost, by some tailored Lisp.  Please give it a try.  Even if the
result works only for the cases we actually use in our sources, it
might be "good enough" for us.