SMIE implementation for the C-like languages

SMIE implementation for the C-like languages

[Arthur Evstifeev]

I'm currently maintaining smie implementation for the Swift
language. Language itself is close to the family of C-like languages
with some differences to the language constructions. I'm looking for
some advice about applying smie to the languages that use braces as a
terminators for the code blocks.

1. As stated in documentation tokens that are defined in syntax table
don't have to be tokenised in lexer. I tried to go this way, but
encountered situations where defined grammars are not respected. It
seems that smie only tries to indent closer token with respect to the
opener, rather than parent token defined by grammar. As far as I
understand (after checking source code for
smie-indent-forward/backward-token) this is expected behavior. So I
decided to tokenise braces in lexer and it works for most of the cases,
but I encountered issues with paren blinking: in some situations
blinking fails with "Mismatched parenthesis". During some tests I
decided to change lexer rules for braces to return begin/end tokens
instead of braces. I noticed that smie still tries to indent "}" token
in some situations, specifically `:close-all . "}"`. This seem to be
related to blinking  problem during blinking we have. So my question is
what will be the semantically correct way of handling braces for the C-like
languages? And secondary question is it expected that smie tries to
indent tokens that are not returned by lexer?

2. As a sort of continuation of the previous problem, we are having
problem understanding what will be semantically correct way of defining
`sexp` for the smie based mode. At the moment we see a different
behavior between non-smie c++ mode (which is close to the Swift)
and something like ruby-mode. One of the contributers summarised
differences in this post
https://github.com/chrisbarrett/swift-mode/pull/117#issuecomment-154753070.
I personally think grammar based sexp provided by smie are extremely
useful, but they yield confusing results when it comes to blinking
parens. For example grammar for "if" from here:
https://github.com/chrisbarrett/swift-mode/blob/simplify_smie/swift-mode.el#L74-L129
works well for indentation and movements, but blinking on the close
("}") returns "if" token. So the question is it a right approach to use
braces as a tokens for the grammars for C-like languages or there is a
better way to support blinking and grammars at the same time?

Thank you,
Arthur

Re: SMIE implementation for the C-like languages

[Stefan Monnier]

> Language itself is close to the family of C-like languages
> with some differences to the language constructions.  I'm looking for
> some advice about applying smie to the languages that use braces as a
> terminators for the code blocks.

Indeed, SMIE is not great for that currently.

I have an "smc-mode" (i.e. SMIE-based c-mode) here which I wrote as an
exercise to try and see what it takes to get SMIE working acceptably for
the C language syntax.  It's not usable (it was really meant as an
experimental prototype/proof-of-concept), but if you're interested to
look at it, I could make it available somewhere.

> 1. As stated in documentation tokens that are defined in syntax table
> don't have to be tokenised in lexer. I tried to go this way, but
> encountered situations where defined grammars are not respected.

Not sure which situations you're referring to.

> It seems that smie only tries to indent closer token with respect to
> the opener, rather than parent token defined by grammar.

By default, yes.  Of course, the smie-rule-function is there to tweak
that as/when needed.

> cases, but I encountered issues with paren blinking: in some
> situations blinking fails with "Mismatched parenthesis".

Same as before: if you don't give an example, it's hard to know what
might be the cause.

> During some tests I decided to change lexer rules for braces to return
> begin/end tokens instead of braces. I noticed that smie still tries to
> indent "}" token in some situations, specifically `:close-all . "}"`.

At this point, I'm very confused, because I don't know what your code
does when, nor when you see which behavior.

> So my question is what will be the semantically correct way of
> handling braces for the C-like languages?

Where?  In the lexer, the grammar, or the indentation rules?
Note that even if you answer this question, there's no single right
answer: you largely get to decide and pick between different consequences.

> And secondary question is it expected that smie tries to indent tokens
> that are not returned by lexer?

If your tokenizer returns nil (or "") and you're in front of a paren,
then SMIE will take this paren to be the next token, yes.

> 2. As a sort of continuation of the previous problem, we are having
> problem understanding what will be semantically correct way of defining
> `sexp` for the smie based mode. At the moment we see a different
> behavior between non-smie c++ mode (which is close to the Swift)
> and something like ruby-mode. One of the contributers summarised
> differences in this post
> https://github.com/chrisbarrett/swift-mode/pull/117#issuecomment-154753070.
> I personally think grammar based sexp provided by smie are extremely
> useful, but they yield confusing results when it comes to blinking
> parens. For example grammar for "if" from here:
> https://github.com/chrisbarrett/swift-mode/blob/simplify_smie/swift-mode.el#L74-L129

Does Swift allow a "{ ... }" block to appear on its own (rather than
as part of a while/if/...), like in C?

If it does (and you want swift-mode to support those blocks), then
I think your approach to use rules like

   ("while" exp "{" insts "}")

will be very problematic because SMIE's parser when parsing backward
(which is the more usual direction) won't know whether to stop after
skipping a {...} or whether to keep going on the off-chance that this
{...} is really part of a for/while/if...
[ The specific complaint you should get is that "{" will appear both as
  an "opener" and as a "neither" (aka "inner") token.  ]

> works well for indentation and movements, but blinking on the close
> ("}") returns "if" token.

Indeed, if these two notions of "sexp" need to be different, then you should
probably disable SMIE's builtin paren-blinking support.


        Stefan

Re: SMIE implementation for the C-like languages

[Arthur Evstifeev]

Stefan Monnier writes:

Thanks for the answers and sorry for the lack of examples in my first post.

>> Language itself is close to the family of C-like languages
>> with some differences to the language constructions.  I'm looking for
>> some advice about applying smie to the languages that use braces as a
>> terminators for the code blocks.
>
> Indeed, SMIE is not great for that currently.
>
> I have an "smc-mode" (i.e. SMIE-based c-mode) here which I wrote as an
> exercise to try and see what it takes to get SMIE working acceptably for
> the C language syntax.  It's not usable (it was really meant as an
> experimental prototype/proof-of-concept), but if you're interested to
> look at it, I could make it available somewhere.
>

If it's not hard to do, I'll appreciate that.

>> 1. As stated in documentation tokens that are defined in syntax table
>> don't have to be tokenised in lexer. I tried to go this way, but
>> encountered situations where defined grammars are not respected.
>
> Not sure which situations you're referring to.
>

For example for such grammar:

(id)
(inst ("if" exp "{" insts "}")
      (exp))
(insts (insts ";" insts) (inst))
(exp (exp "." id)
     (id ":" exp)
     (exp "=" exp))
(exps (exps "," exps) (exp))

When in trying to indent such code:

if true {
    |bar
}

Token "bar" is positioned incorrectly and I see such requests from smie
and lexer and indentation rules:

forward: 15 -> 18 = bar
backward: 15 -> 10 =
forward: 9 -> 9 =
backward: 9 -> 4 = true
backward: 4 -> 1 = if
forward: 9 -> 9 =
:after '{'; sibling-p:nil parent:(nil 4 if) hanging:t == nil
forward: 9 -> 9 =  [2 times]
backward: 9 -> 4 = true
backward: 4 -> 1 = if
forward: 9 -> 9 =
:before '{'; sibling-p:nil parent:(nil 4 if) hanging:t == nil
forward: 9 -> 9 =
backward: 9 -> 4 = true [2 times]
backward: 4 -> 1 = if [3 times]
:list-intro 'if'; sibling-p:nil parent:nil hanging:nil == nil
forward: 4 -> 8 = true
:elem 'args'; sibling-p:nil parent:nil hanging:nil == nil
forward: 4 -> 8 = true
:elem 'basic'; sibling-p:nil parent:nil hanging:nil == 4
forward: 9 -> 9 =
:elem 'basic'; sibling-p:nil parent:nil hanging:t == 4

This logging output and indentation requests don't seem to be respecting
defined grammar.

>> It seems that smie only tries to indent closer token with respect to
>> the opener, rather than parent token defined by grammar.
>
> By default, yes.  Of course, the smie-rule-function is there to tweak
> that as/when needed.
>
>> cases, but I encountered issues with paren blinking: in some
>> situations blinking fails with "Mismatched parenthesis".
>
> Same as before: if you don't give an example, it's hard to know what
> might be the cause.

For the previous case, if we change lexer to tokenize braces and try to
indent the same construction, indentation will be correct and smie
output will be more inline with defined grammar:

forward: 22 -> 25 = bar
backward: 22 -> 9 = {
forward: 9 -> 10 = {
backward: 10 -> 9 = {
backward: 9 -> 4 = true
backward: 4 -> 1 = if
forward: 9 -> 10 = {
:after '{'; sibling-p:nil parent:(nil 1 if) hanging:t == nil
forward: 9 -> 10 = {
backward: 9 -> 4 = true
backward: 4 -> 1 = if
forward: 9 -> 10 = {
:before '{'; sibling-p:nil parent:(nil 1 if) hanging:t == nil
forward: 9 -> 10 = {
:elem 'basic'; sibling-p:nil parent:nil hanging:t == 4

But blink-matching-open calls for the simple code block:

{
}|

will return "Mismatched parenthesis" error.

>
>> During some tests I decided to change lexer rules for braces to return
>> begin/end tokens instead of braces. I noticed that smie still tries to
>> indent "}" token in some situations, specifically `:close-all . "}"`.
>
> At this point, I'm very confused, because I don't know what your code
> does when, nor when you see which behavior.
>

If we modify lexer from the last case, so instead of braces we will
return begin/end tokens, something like this:

((looking-at "{") (forward-char 1) "begin")
((looking-at "}") (forward-char 1) "begin")

And change grammar accordingly then try to indent this:

if true {
    bar
|}

smie output will still contain request for indentation of "}" even if
lexer didn't return such token:

forward: 19 -> 20 = end
backward: 20 -> 19 = end
backward: 19 -> 18 = ;
backward: 18 -> 15 = bar
backward: 15 -> 9 = begin
backward: 9 -> 4 = true
backward: 4 -> 1 = if
:close-all '}'; sibling-p:t parent:(nil 1 if) hanging:nil == nil
backward: 20 -> 19 = end
backward: 19 -> 18 = ;
backward: 18 -> 15 = bar
backward: 15 -> 9 = begin
backward: 9 -> 4 = true
backward: 4 -> 1 = if

>> So my question is what will be the semantically correct way of
>> handling braces for the C-like languages?
>
> Where?  In the lexer, the grammar, or the indentation rules?
> Note that even if you answer this question, there's no single right
> answer: you largely get to decide and pick between different consequences.
>

Ok, with other answers this makes sense to me.

>> And secondary question is it expected that smie tries to indent tokens
>> that are not returned by lexer?
>
> If your tokenizer returns nil (or "") and you're in front of a paren,
> then SMIE will take this paren to be the next token, yes.
>

This question was about situation when lexer tokenizes such tokens
instead of relying on smie like in the last example.

>> 2. As a sort of continuation of the previous problem, we are having
>> problem understanding what will be semantically correct way of defining
>> `sexp` for the smie based mode. At the moment we see a different
>> behavior between non-smie c++ mode (which is close to the Swift)
>> and something like ruby-mode. One of the contributers summarised
>> differences in this post
>> https://github.com/chrisbarrett/swift-mode/pull/117#issuecomment-154753070.
>> I personally think grammar based sexp provided by smie are extremely
>> useful, but they yield confusing results when it comes to blinking
>> parens. For example grammar for "if" from here:
>> https://github.com/chrisbarrett/swift-mode/blob/simplify_smie/swift-mode.el#L74-L129
>
> Does Swift allow a "{ ... }" block to appear on its own (rather than
> as part of a while/if/...), like in C?
>
> If it does (and you want swift-mode to support those blocks), then
> I think your approach to use rules like
>
>    ("while" exp "{" insts "}")
>
> will be very problematic because SMIE's parser when parsing backward
> (which is the more usual direction) won't know whether to stop after
> skipping a {...} or whether to keep going on the off-chance that this
> {...} is really part of a for/while/if...
> [ The specific complaint you should get is that "{" will appear both as
>   an "opener" and as a "neither" (aka "inner") token.  ]
>

We came to the same conclusion recently and will try to alter grammar to
remove this ambiguity.

>> works well for indentation and movements, but blinking on the close
>> ("}") returns "if" token.
>
> Indeed, if these two notions of "sexp" need to be different, then you should
> probably disable SMIE's builtin paren-blinking support.

I tried to disable post-self-insert-hook post smie-stup same way as
octave-mode does:

(remove-hook 'post-self-insert-hook #'smie-blink-matching-open 'local)

But it doesn't change behavior of the blinking: for the same if
construction blinking happens for the "if" token. Is there a different
way of altering this behavior?

>
>
>         Stefan

Thank you,
Arthur

--
Sent with my mu4e

Re: SMIE implementation for the C-like languages

[Stefan Monnier]

>> I have an "smc-mode" (i.e. SMIE-based c-mode) here which I wrote as an
>> exercise to try and see what it takes to get SMIE working acceptably for
>> the C language syntax.  It's not usable (it was really meant as an
>> experimental prototype/proof-of-concept), but if you're interested to
>> look at it, I could make it available somewhere.
> If it's not hard to do, I'll appreciate that.

I'll see how best to do that.  Don't hold your breath, tho: to me this
experiment just convinced me that SMIE is not up for it (yet?): the code
ends up doing a lot of parsing "by hand" instead of relying on SMIE.

> When in trying to indent such code:

> if true {
>     |bar
> }

> Token "bar" is positioned incorrectly and I see such requests from smie
> and lexer and indentation rules:

> forward: 15 -> 18 = bar
> backward: 15 -> 10 =
> forward: 9 -> 9 =
> backward: 9 -> 4 = true
> backward: 4 -> 1 = if
> forward: 9 -> 9 =
> :after '{'; sibling-p:nil parent:(nil 4 if) hanging:t == nil
> forward: 9 -> 9 =  [2 times]
> backward: 9 -> 4 = true
> backward: 4 -> 1 = if
> forward: 9 -> 9 =
> :before '{'; sibling-p:nil parent:(nil 4 if) hanging:t == nil
> forward: 9 -> 9 =
> backward: 9 -> 4 = true [2 times]
> backward: 4 -> 1 = if [3 times]
> :list-intro 'if'; sibling-p:nil parent:nil hanging:nil == nil
> forward: 4 -> 8 = true
> :elem 'args'; sibling-p:nil parent:nil hanging:nil == nil
> forward: 4 -> 8 = true
> :elem 'basic'; sibling-p:nil parent:nil hanging:nil == 4
> forward: 9 -> 9 =
> :elem 'basic'; sibling-p:nil parent:nil hanging:t == 4

> This logging output and indentation requests don't seem to be respecting
> defined grammar.

I don't see anything out of the ordinary here.  Which part seems odd to you?

> But blink-matching-open calls for the simple code block:

> {
> }|

> will return "Mismatched parenthesis" error.

That's because it tries to match "}" with an opening "if" (since your
grammar states ("if" exp "{" insts "}") which implies that "{" is an
infix terminal).

> smie output will still contain request for indentation of "}" even if
> lexer didn't return such token:

> forward: 19 -> 20 = end
> backward: 20 -> 19 = end
> backward: 19 -> 18 = ;
> backward: 18 -> 15 = bar
> backward: 15 -> 9 = begin
> backward: 9 -> 4 = true
> backward: 4 -> 1 = if
> :close-all '}'; sibling-p:t parent:(nil 1 if) hanging:nil == nil

Indeed.  Please send this via M-x report-emacs-bug so we get a number to
track this problem.

> But it doesn't change behavior of the blinking: for the same if
> construction blinking happens for the "if" token.  Is there a different
> way of altering this behavior?

My guess would be that the default blinking code uses forward-sexp which
goes through forward-sexp-function which SMIE sets up as well.  Try set
this var back to nil in swift-mode buffers, see if that helps.


        Stefan

Re: SMIE implementation for the C-like languages

[Arthur Evstifeev]

Stefan Monnier writes:

>> When in trying to indent such code:
>
>> if true {
>>     |bar
>> }
>
>> Token "bar" is positioned incorrectly and I see such requests from smie
>> and lexer and indentation rules:
>
>> forward: 15 -> 18 = bar
>> backward: 15 -> 10 =
>> forward: 9 -> 9 =
>> backward: 9 -> 4 = true
>> backward: 4 -> 1 = if
>> forward: 9 -> 9 =
>> :after '{'; sibling-p:nil parent:(nil 4 if) hanging:t == nil
>> forward: 9 -> 9 =  [2 times]
>> backward: 9 -> 4 = true
>> backward: 4 -> 1 = if
>> forward: 9 -> 9 =
>> :before '{'; sibling-p:nil parent:(nil 4 if) hanging:t == nil
>> forward: 9 -> 9 =
>> backward: 9 -> 4 = true [2 times]
>> backward: 4 -> 1 = if [3 times]
>> :list-intro 'if'; sibling-p:nil parent:nil hanging:nil == nil
>> forward: 4 -> 8 = true
>> :elem 'args'; sibling-p:nil parent:nil hanging:nil == nil
>> forward: 4 -> 8 = true
>> :elem 'basic'; sibling-p:nil parent:nil hanging:nil == 4
>> forward: 9 -> 9 =
>> :elem 'basic'; sibling-p:nil parent:nil hanging:t == 4
>
>> This logging output and indentation requests don't seem to be respecting
>> defined grammar.
>
> I don't see anything out of the ordinary here.  Which part seems odd
> to you?

I'm a bit suspicious about :list-intro 'if', :elem 'args' and that :elem
'basic' was requested 2 times, especially if you compare against second
example I sent yesterday (which has a proper indentation). What is
specifically looking confusing to me is the whole situation with
braces. I don't understand why smie produces different indentation
results for cases when we tokenize braces and when we don't. I
think both cases shouldn't be treated differently in grammars or
indentation rules. From implementation perspective I understand why it
happens at the moment.

>
>> But blink-matching-open calls for the simple code block:
>
>> {
>> }|
>
>> will return "Mismatched parenthesis" error.
>
> That's because it tries to match "}" with an opening "if" (since your
> grammar states ("if" exp "{" insts "}") which implies that "{" is an
> infix terminal).
>

Yes that's true, but braces are also a part of the syntax table. Since smie
allow any sequence of sexp anywhere, I think this code block should be
handled. I'm not sure what is the best way to do that but maybe
something like a fallback logic to the syntax tables for such cases?

>> smie output will still contain request for indentation of "}" even if
>> lexer didn't return such token:
>
>> forward: 19 -> 20 = end
>> backward: 20 -> 19 = end
>> backward: 19 -> 18 = ;
>> backward: 18 -> 15 = bar
>> backward: 15 -> 9 = begin
>> backward: 9 -> 4 = true
>> backward: 4 -> 1 = if
>> :close-all '}'; sibling-p:t parent:(nil 1 if) hanging:nil == nil
>
> Indeed.  Please send this via M-x report-emacs-bug so we get a number to
> track this problem.
>

I submitted a new bug report.

>> But it doesn't change behavior of the blinking: for the same if
>> construction blinking happens for the "if" token.  Is there a different
>> way of altering this behavior?
>
> My guess would be that the default blinking code uses forward-sexp which
> goes through forward-sexp-function which SMIE sets up as well.  Try set
> this var back to nil in swift-mode buffers, see if that helps.
>

I tried to reset forward-sexp-function and it works when I'm not
tokenizing braces. When braces handled in lexer, I'm getting "Mismatched
parenthesis" error similar to one of the previous examples.

Thank you,
Arthur

Re: SMIE implementation for the C-like languages

[Stefan Monnier]

>>> if true {
>>>    |bar
>>> }
[...]
> I'm a bit suspicious about :list-intro 'if', :elem 'args' and that :elem
> 'basic' was requested 2 times, especially if you compare against second
> example I sent yesterday (which has a proper indentation).

Without stepping through the rules-function to see where is what and
comparing to the precise grammar you used, I can't tell you in detail if
that's really necessary or not, but normally the above case will need to
compute the virtual indentation of the "{" and this "{...}" probably
looks like a sexp to SMIE, so SMIE sees the above as "KEYWORD(if) SEXP1
SEXP2", so the "SEXP1 SEXP1" may be a function call (SEXP1 being the
function and SEXP2 the argument).

>> That's because it tries to match "}" with an opening "if" (since your
>> grammar states ("if" exp "{" insts "}") which implies that "{" is an
>> infix terminal).
> Yes that's true, but braces are also a part of the syntax table.

That's right, so there are two conflicting "definitions" of what { and } do.
I'm not surprised that SMIE doesn't deal well with such situations,
because I'm myself not really sure how it should be handled.

> Since smie allow any sequence of sexp anywhere, I think this code
> block should be handled.

But the smie-grammar refines the definition of "sexp" provided by the
syntax-table, so it's not clear any more whether "{...}" is really
a sexp or is only the second half of "if ... { ... }".

> I'm not sure what is the best way to do that but maybe something like
> a fallback logic to the syntax tables for such cases?

I understand what you mean, but I don't know how to translate that into
code, because once you look at it from the code's point of view, the
requirements are often conflicting.

>>> But it doesn't change behavior of the blinking: for the same if
>>> construction blinking happens for the "if" token.  Is there a different
>>> way of altering this behavior?
>> My guess would be that the default blinking code uses forward-sexp which
>> goes through forward-sexp-function which SMIE sets up as well.  Try set
>> this var back to nil in swift-mode buffers, see if that helps.
> I tried to reset forward-sexp-function and it works when I'm not
> tokenizing braces. When braces handled in lexer, I'm getting "Mismatched
> parenthesis" error similar to one of the previous examples.

I suggest you report this as a bug as well, providing enough details to
reproduce it.  I don't know if it's indeed a bug or not, but we need to
investigate in more details.


        Stefan

Re: SMIE implementation for the C-like languages

[Stefan Monnier]

>> I have an "smc-mode" (i.e. SMIE-based c-mode) here which I wrote as an
>> exercise to try and see what it takes to get SMIE working acceptably for
>> the C language syntax.  It's not usable (it was really meant as an
>> experimental prototype/proof-of-concept), but if you're interested to
>> look at it, I could make it available somewhere.
> If it's not hard to do, I'll appreciate that.

I just added it to elpa.git..  See for example
http://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/packages/sm-c-mode


        Stefan

Re: SMIE implementation for the C-like languages

[Arthur Evstifeev]

Stefan Monnier writes:

[...]
>>>> But it doesn't change behavior of the blinking: for the same if
>>>> construction blinking happens for the "if" token.  Is there a different
>>>> way of altering this behavior?
>>> My guess would be that the default blinking code uses forward-sexp which
>>> goes through forward-sexp-function which SMIE sets up as well.  Try set
>>> this var back to nil in swift-mode buffers, see if that helps.
>> I tried to reset forward-sexp-function and it works when I'm not
>> tokenizing braces. When braces handled in lexer, I'm getting "Mismatched
>> parenthesis" error similar to one of the previous examples.
>
> I suggest you report this as a bug as well, providing enough details to
> reproduce it.  I don't know if it's indeed a bug or not, but we need to
> investigate in more details.
>

Sorry for the delay. I will open a new ticket about inconsistencies in
braces behavior and how it results in issue with paren blinking. I
understand that there is no clear solution for this problem at the
moment and will appreciate any improvements. We will try to restructure
our grammar to not threat "{" as neither token in grammars and will see
how it goes. Thanks a lot for the answers!

Arthur

Re: SMIE implementation for the C-like languages

[Markus Triska]

Hi Arthur,

Arthur Evstifeev <mail@ap4y.me> writes:

> I will open a new ticket about inconsistencies in
> braces behavior and how it results in issue with paren blinking. I
> understand that there is no clear solution for this problem at the
> moment and will appreciate any improvements. We will try to restructure
> our grammar to not threat "{" as neither token in grammars and will see
> how it goes. Thanks a lot for the answers!

Many thanks for looking into this! Some related information that I hope
helps or at least motivates: The current implementation of prolog-mode
also suffers from such inconsistencies, where something quite unintended
is often erroneously highlighted. If you find out how to solve this for
your case, then I hope the SMIE definitions for prolog-mode can be fixed
analogously. For more information, please see case (1) in the issue:

   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21526#35

All the best!
Markus

Re: SMIE implementation for the C-like languages

[Stefan Monnier]

> analogously. For more information, please see case (1) in the issue:
>
>    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21526#35

Case (1) doesn't look like a comparable issue.  In your case, you should
just remove (:smie-closer-alist (t . ".")) from the grammar so that SMIE
doesn't try to highlight it as a "closing" element.

I like that feature, but obviously you don't, so all it takes is
a config var.


        Stefan