SMIE help

SMIE help

[Ivan Andrus]

I'm trying to write a SMIE grammar for the GAP language [1], since it will be much better than what I have at the moment.  GAP is a fairly simple language, similar to ruby (and maybe Pascal).  Because of that I foolishly thought I would be able to figure it out.  I started with the example grammar and stole some from ruby-mode.

What I have works in many ways (though it could no doubt be simplified).  My indentation is wildly off however in at least one case, and I have no idea how to fix it.  The case of interest is the first line inside of a function (in the example the line starting with local).  Interestingly other lines in a function are fine.

Feel free to point me at other documentation or discussions that may be of use.  Any other tips would be appreciated as well.

Thanks,
Ivan

[1] http://www.gap-system.org/Manuals/doc/ref/chap4.html


;; One variation of the lisp code that I have used
(defvar gap-indent-step 4)

;; This sort of works
(defconst gap-smie-grammar
  (smie-prec2->grammar
   (smie-bnf->prec2
    '((id)
      (inst (exp) ;not technically, but probably makes sense
            ;; (exp ":=" exp)
            (id ":=" exp)
            ("function" insts "end")
            ("repeat" insts "until" exp)
            ("while" exp "do" insts "od")
            ("for" id "in" exp "do" insts "od")
            ("if" if-body "fi")
            ("return" exp)
            ("local" exps)
            )
      (insts (insts ";" insts) (inst))
      (exp ("(" exps ")")
           ("[" exps "]")
           ("{" exps "}")

           ("not" exp)

           (exp ".." exp)
           (exp "and" exp)
           (exp "or" exp)
           (exp "<" exp)
           (exp "<=" exp)
           (exp "=" exp)
           (exp ">=" exp)
           (exp ">" exp)
           (exp "<>" exp)
           ;; (exp "in" exp)
           (exp "+" exp)
           (exp "-" exp)
           (exp "*" exp)
           (exp "/" exp)
           (exp "mod" exp)
           (exp "^" exp)
           )
      (exps (exps "," exps) (exp))
      (itheni (insts) (exp "then" insts))
      (ielsei (itheni) (itheni "else" insts))
      (if-body (ielsei) (if-body "elif" if-body)))

    '((assoc ";"))
    '((assoc ","))
    '((assoc "elif"))
    '((assoc "not")
      (assoc "..")
      (assoc "and" "or")
      (assoc "<" "<=" "=" ">=" ">" "<>" ;; "in"
             )
      (assoc "+" "-")
      (assoc "*" "/" "mod")
      (assoc "^")))))


(defun gap-smie-rules (kind token)
  (pcase (cons kind token)
    (`(:list-intro . "function")
     gap-indent-step)
    (`(:after . "function")
     gap-indent-step)

    (`(:after . ,(or `"if" `"else" `"then" `"elif" `"do" `"repeat" `"while"))
     gap-indent-step)

    (`(:before . ";")
     (cond
      ((smie-rule-parent-p "function" "repeat" "while" "for"
                           "if" "then" "elif" "else" "when")
       (smie-rule-parent gap-indent-step))
      ))

    (`(:after . ,(or "=" ":=" "+" "-" "*" "/" "^"
                     ">" "<" ">=" "<=" "<>" "and" "or"))
     (if (smie-rule-parent-p ";" nil) gap-indent-step))
    ))

(smie-setup gap-smie-grammar #'gap-smie-rules)




# example of well indented GAP code
FittingHeight := function (G)
    local  fittingLength, F;

    fittingLength := 0;
    F := FittingSubgroup( G );
    while Order(F) > 1 do
        G := G / F;

        F := FittingSubgroup( G );
        fittingLength := fittingLength+1;
    od;

    if Order(G) > 1 then
        return fail;
    fi;
    return fittingLength;
end;

Re: SMIE help

[Stefan Monnier]

> What I have works in many ways (though it could no doubt be
> simplified).  My indentation is wildly off however in at least one
> case, and I have no idea how to fix it.  The case of interest is the
> first line inside of a function (in the example the line starting with
> local).  Interestingly other lines in a function are fine.

> # example of well indented GAP code
> FittingHeight := function (G)
>     local  fittingLength, F;
> 
>     fittingLength := 0;

Here's the problem: with the grammar you have, the above is parsed as
something like the following structure:

  (:= FittingHeight
      (function
       ((G) (local (, fittingLength F)))
       (:= fittingLength 0)))

So the "local ..." is aligned with "(G)".

You can see that it's a parsing problem by hitting C-M-b from right
after the first semi-colon of your example: it should jump to just
before "local" but instead it jumps to just before "(G)", as if "(G)"
was part of the first instruction.

SMIE is not very good with "positional rules", as is the case here where
there is no clear keyword between "(G)" and the function's body.
By "positional" I mean that "the *first* sexp-like thingy after
`function' is special".

You can solve this problem at 2 levels:
- the grammar level: you can't really fix it in gap-smie-grammar itself
  because SMIE's class of grammars is too limited to understand this,
  but you can change the lexer such that it treats the close-paren after
  "(G" specially and returns another token.
- the indent-rules level: you give up parsing the code correctly and
  instead patch things up in gap-smie-rules by adding a rule for
  (:after . ")") which checks if this is the closing parent of
  a "function (...)".

Doing it in indent-rules will lead to more efficient code, usually, but
may also force you to add more and more special cases (tho in this
particular case, it might work OK).
In this particular case, you might like to try and patch things up in
gap-smie-rules, along the lines of:

    (`(:after . ")")
     (save-excursion
       (up-list -1)
       (when (equal "function" (car (smie-indent-backward-token)))
	 `(column . ,(+ gap-indent-step (smie-indent-virtual))))))
    
After that, you'll also want to change the smie-rules for (:before
. "function"), probably, so as to treat "function (..)\n" as "hanging":

    (`(:before . "function")
     (when (save-excursion
	     (forward-word 1)
	     (forward-sexp 1)
	     (smie-rule-hanging-p))
       (smie-rule-parent)))
    
so that the body of "function" is not indented relative to "function"
but relative to "FittingHeight :=".


        Stefan

Re: SMIE help

[Ivan Andrus]

On Nov 29, 2013, at 2:28 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> What I have works in many ways (though it could no doubt be
>> simplified).  My indentation is wildly off however in at least one
>> case, and I have no idea how to fix it.  The case of interest is the
>> first line inside of a function (in the example the line starting with
>> local).  Interestingly other lines in a function are fine.
> 
>> # example of well indented GAP code
>> FittingHeight := function (G)
>>    local  fittingLength, F;
>> 
>>    fittingLength := 0;
> 
> Here's the problem: with the grammar you have, the above is parsed as
> something like the following structure:
> 
>  (:= FittingHeight
>      (function
>       ((G) (local (, fittingLength F)))
>       (:= fittingLength 0)))
> 
> So the "local ..." is aligned with "(G)".
> 
> You can see that it's a parsing problem by hitting C-M-b from right
> after the first semi-colon of your example: it should jump to just
> before "local" but instead it jumps to just before "(G)", as if "(G)"
> was part of the first instruction.

I see.  That makes sense.

> SMIE is not very good with "positional rules", as is the case here where
> there is no clear keyword between "(G)" and the function's body.
> By "positional" I mean that "the *first* sexp-like thingy after
> `function' is special".
> 
> You can solve this problem at 2 levels:
> - the grammar level: you can't really fix it in gap-smie-grammar itself
>  because SMIE's class of grammars is too limited to understand this,
>  but you can change the lexer such that it treats the close-paren after
>  "(G" specially and returns another token.
> - the indent-rules level: you give up parsing the code correctly and
>  instead patch things up in gap-smie-rules by adding a rule for
>  (:after . ")") which checks if this is the closing parent of
>  a "function (...)".
> 
> Doing it in indent-rules will lead to more efficient code, usually, but
> may also force you to add more and more special cases (tho in this
> particular case, it might work OK).

I presume solving it in the lexer will also cause the C-M-b to work as expected in your example above.  I use sexp-based motion all the time, so I should probably take that route (at least eventually).  For the moment I think I'll try the indent-rules route since it does seem easier.

> In this particular case, you might like to try and patch things up in
> gap-smie-rules, along the lines of:
> 
>    (`(:after . ")")
>     (save-excursion
>       (up-list -1)
>       (when (equal "function" (car (smie-indent-backward-token)))
> 	 `(column . ,(+ gap-indent-step (smie-indent-virtual))))))

That works great!

> After that, you'll also want to change the smie-rules for (:before
> . "function"), probably, so as to treat "function (..)\n" as "hanging":
> 
>    (`(:before . "function")
>     (when (save-excursion
> 	     (forward-word 1)
> 	     (forward-sexp 1)
> 	     (smie-rule-hanging-p))
>       (smie-rule-parent)))
> 
> so that the body of "function" is not indented relative to "function"
> but relative to "FittingHeight :=".


Thank you very much for SMIE and for answering my questions.

-Ivan

Re: SMIE help

[Ivan Andrus]

On Nov 29, 2013, at 10:05 PM, Ivan Andrus <darthandrus@gmail.com> wrote:
> 
> Thank you very much for SMIE and for answering my questions.


I've been able to make some great progress.  I'm nearly to the point where I think I could be happy with it.  Except that I found what seems to be a bug, namely that

if decomp then
    # This doesn't work for some reason over non-prime fields
    return [c, soc, decomps];
fi;

indents differently than

if decomp then # This doesn't work for some reason over non-prime fields
              return [c, soc, decomps];
fi;

Of course, # is the comment character.  Is this expected behavior?  Do I need to take comments into account somehow?

-Ivan

Re: SMIE help

[Stefan Monnier]

> if decomp then
>     # This doesn't work for some reason over non-prime fields
>     return [c, soc, decomps];
> fi;
>
> indents differently than
>
> if decomp then # This doesn't work for some reason over non-prime fields
>               return [c, soc, decomps];
> fi;
>
> Of course, # is the comment character.  Is this expected behavior?

No.

> Do I need to take comments into account somehow?

Depends.  Make it a bug report, including the code needed to reproduce
the problem.  Maybe it's a bug in your code, but it might also be in
smie.el.


        Stefan

Re: SMIE help

[Ivan Andrus]

On Dec 3, 2013, at 6:47 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> if decomp then
>>    # This doesn't work for some reason over non-prime fields
>>    return [c, soc, decomps];
>> fi;
>> 
>> indents differently than
>> 
>> if decomp then # This doesn't work for some reason over non-prime fields
>>              return [c, soc, decomps];
>> fi;
>> 
>> Of course, # is the comment character.  Is this expected behavior?
> 
> No.
> 
>> Do I need to take comments into account somehow?
> 
> Depends.  Make it a bug report, including the code needed to reproduce
> the problem.  Maybe it's a bug in your code, but it might also be in
> smie.el.


Bug is 16041.

-Ivan