SMIE examples and questions

SMIE examples and questions

[Aurélien Aptel]

Hi,

Are there *simple* complete examples of SMIE out there? I find the
manual not that helpful to get started and existing modes look pretty
complex :(

Is the forward lexer required or is the backward one enough?
What are the assumptions of SMIE regarding the lexer?
- What should it return when there is nothing to read (eg backward and
begining of buffer)?
- I assume if the point is in the middle of a token, it should return
the complete current token (including text after point) and place the
point at token start?
- Does point position matter regarding spaces (assuming they have no
meaning)? Is it ok if the lexer places the point sometime after and
sometime before them?
- Can I safely call it manually without involving SMIE?
- I assume backward and forward lexer should always have the same
behaviour, even for spaces?

I understand there is no way to dump the AST after parsing, and that
grammar errors not being reported is a feature. How am I supposed to
debug parsing/indenting? If you have any tips or workflow, like the
more practical order to implement things, or how to debug that would
be greatly appreciated.

Thanks!

Re: SMIE examples and questions

[Stefan Monnier]

> Are there *simple* complete examples of SMIE out there? I find the
> manual not that helpful to get started and existing modes look pretty
> complex :(

The only simple and complete ones are for language that are probably
too simple.

Maybe the rnc-mode and dts-mode GNU ELPA packages?  You might also want
to look at my SMIE paper (https://programming-journal.org/2021/5/1/)
which walks through a small part of the rnc-mode SMIE code.

> Is the forward lexer required or is the backward one enough?

Both are needed.

> What are the assumptions of SMIE regarding the lexer?
> - What should it return when there is nothing to read (eg backward and
>   begining of buffer)?

nil, usually.

> - I assume if point is in the middle of a token, it should return
>   the complete current token (including text after point) and place the
>   point at token start?

The caller of the lexer should normally make sure it's never in the
middle of a token.  This doesn't work if you have tokens that can span
multiple lines (e.g. when LF itself is significant, such as the
implicit ; in Javascript).

> - Does point position matter regarding spaces (assuming they have no
>   meaning)?  Is it ok if the lexer places the point sometime after and
>   sometime before them?

It should place point right "after" (in the direction of movement) the
token it has read.

> - Can I safely call it manually without involving SMIE?

Yes.

> - I assume backward and forward lexer should always have the same
>   behaviour, even for spaces?

Yes and no, since one should stop to the left of the spaces and the
other to the right.

> I understand there is no way to dump the AST after parsing, and that
> grammar errors not being reported is a feature.

There's no AST, indeed, and there are no possible errors (IOW errors
can't be detected).

> How am I supposed to debug parsing/indenting?

I mostly use M-C-f and M-C-b for parsing.  For the indentation rules you
can use `M-x smie-edebug`.

> If you have any tips or workflow, like the more practical order to
> implement things, or how to debug that would be greatly appreciated.

Start with the simple cases and then work your way up.

If you want more concrete help, don't hesitate to send me your current
code along with some sample source showing how it should be indented and
the problems you're facing.


        Stefan

Re: SMIE examples and questions

[Aurélien Aptel]

The language is Poke. It is C-like.
The main thing are (possibly anonymous) structure like so:

    struct
    {
      int32_t foo;
      int8_t bar;
    };

Some expressions can be embedded in the type, and there is a binding
system similar to typedef where you can do:

    deftype foo = struct {...}; devar bar = 42;

Sample source: https://git.savannah.gnu.org/cgit/poke.git/tree/pickles/elf.pk
There's an ANTLRv4 grammar here:
https://git.savannah.gnu.org/cgit/poke.git/tree/etc/poke.g4

So far I only have the font-locking...
https://git.savannah.gnu.org/cgit/poke.git/tree/etc/poke-mode.el
And this anemic grammar:

(defvar poke-smie-grammar
  (smie-prec2->grammar
   (smie-bnf->prec2
    '((id)
      (def ("defvar" id "=" inst)
       ("deftype" id "=" inst))
      ;; (exp (exp "+" exp)
      ;;        (exp "-" exp)
      ;;        (exp "*" exp)
      ;;        (exp "==" exp)
      ;;        (exp "!=" exp)
      ;;        (exp "<" exp)
      ;;        (exp "<=" exp)
      ;;        (exp ">" exp)
      ;;        (exp ">=" exp)
      ;;        ("(" exp ")"))
      (inst ("struct" insts)
        ("union" insts)
        (id id))
      (insts (insts ";" insts) (inst))))))