SMIE implementation for the C-like languages

[Arthur Evstifeev <mail@ap4y.me>]

I'm currently maintaining smie implementation for the Swift
language. Language itself is close to the family of C-like languages
with some differences to the language constructions. I'm looking for
some advice about applying smie to the languages that use braces as a
terminators for the code blocks.

1. As stated in documentation tokens that are defined in syntax table
don't have to be tokenised in lexer. I tried to go this way, but
encountered situations where defined grammars are not respected. It
seems that smie only tries to indent closer token with respect to the
opener, rather than parent token defined by grammar. As far as I
understand (after checking source code for
smie-indent-forward/backward-token) this is expected behavior. So I
decided to tokenise braces in lexer and it works for most of the cases,
but I encountered issues with paren blinking: in some situations
blinking fails with "Mismatched parenthesis". During some tests I
decided to change lexer rules for braces to return begin/end tokens
instead of braces. I noticed that smie still tries to indent "}" token
in some situations, specifically `:close-all . "}"`. This seem to be
related to blinking  problem during blinking we have. So my question is
what will be the semantically correct way of handling braces for the C-like
languages? And secondary question is it expected that smie tries to
indent tokens that are not returned by lexer?

2. As a sort of continuation of the previous problem, we are having
problem understanding what will be semantically correct way of defining
`sexp` for the smie based mode. At the moment we see a different
behavior between non-smie c++ mode (which is close to the Swift)
and something like ruby-mode. One of the contributers summarised
differences in this post
https://github.com/chrisbarrett/swift-mode/pull/117#issuecomment-154753070.
I personally think grammar based sexp provided by smie are extremely
useful, but they yield confusing results when it comes to blinking
parens. For example grammar for "if" from here:
https://github.com/chrisbarrett/swift-mode/blob/simplify_smie/swift-mode.el#L74-L129
works well for indentation and movements, but blinking on the close
("}") returns "if" token. So the question is it a right approach to use
braces as a tokens for the grammars for C-like languages or there is a
better way to support blinking and grammars at the same time?

Thank you,
Arthur