Re: SMIE

[Matt DeBoard <matt.deboard@gmail.com>]

As regarding inclusion in GNU ELPA, I'm just a caretaker for the
project on behalf of the Elixir-lang people, but as it's already in
MELPA I'm sure it's fine.

On Wed, Jul 9, 2014 at 11:53 PM, Matt DeBoard <matt.deboard@gmail.com> wrote:
> In general I’m having a hard time connecting the dots between the BNF
> grammar table creation, the smie-rules (i.e. :before, :after, etc.),
> tokenization, indentation, and so forth, and how it all comes together
> to make this indentation machine work.
>
> It’s not that the manual is poorly written. In fact between the manual
> and the code comments it’s pretty comprehensive. It’s just a high
> volume of new information. I think it’ll take me awhile to ingest
> everything, but when I do I’d be glad to contribute back.
>
> Regarding the specific issue I mentioned, I closed that out tonight.
> There was some overly “greedy” definitions in the syntax table, so
> there’s that. I’m slowly starting to pare things away. The bit you
> wrote about :list-intro is interesting. When you say that it sees two
> or more concatenated expressions, how does that tie in to the BNF
> grammar definitions?
>
> On Wed, Jul 9, 2014 at 11:02 PM, Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
>>> Hi there. As the subject line says I’m writing for help with SMIE.
>>
>> Cool!
>>
>>> I am currently working on elixir-mode
>>> <https://github.com/elixir-lang/emacs-elixir>, having (apparently) taking
>>> over the mode as the latest in a line of contributors.
>>
>> I'd love to include this in GNU ELPA.  Interested?
>>
>>> Specifically I’m having trouble understanding the mental model for how
>>> tokenisation & indentation works.  For example, in this
>>> <https://github.com/elixir-lang/emacs-elixir/issues/18> issue, indentation
>>> errors seem to crop up only after separating lines of code with blank
>>> lines.
>>> I have spent, seriously, hundreds of hours trying to sort out what’s
>>> happening here and I am at my wits’ end.
>>
>> IIUC, Elixir syntax does not treat all whitespace as "irrelevant",
>> contrary to the default tokenizer of SMIE.
>>
>>> Does this issue ring any bells with issues you’ve dealt with in
>>> the past?
>>
>> Yes, indeed.  Octave and sh are two other languages that use SMIE and
>> where some whitespace is syntactically significant.
>>
>> What you need to do is to change the tokenizer so that instead of
>> skipping all whitespace, it turns the syntactically-significant
>> whitespace into a token (you can name it any way you like; in the above
>> languages, it turns out to be syntactically equivalent to a semi-colon,
>> so we call it ";").
>>
>> I know absolutely nothing about Elixir or its syntax, so I can't give
>> you specific details, but you can look at octave.el and sh-script.el
>> for examples.  Feel free to email me back with more details if you need
>> further help.
>>
>>> Final question, how is it determined if a token is a :list-intro token?
>>
>> Not sure I understand the question.  The issue is for the indentation
>> rules, when it sees two (or more) concatenated expressions (e.g. "exp1
>> exp2"), should it assume that exp2 is something like an argument to the
>> exp1 function (and hence exp2 (and exp3, ...) should be indented like
>> a function argument) or are all those "expressions" just a list, where
>> the first is not more special than the second?
>> This usually depends on the context.  E.g. in a situation like
>>
>>    fun x1 x2 x3 =>
>>
>> x2 is not an argument passed to the function x1;  Instead x1, x2, and x3
>> are "siblings" and should be indented to the same level.  So to decide
>> how to indent x2 and x3 w.r.t x1, SMIE calls the smie-rule-function with
>> (:list-intro . "fun") so smie-rule-function can tell it that "fun"
>> introduces a *list* of "things" rather than being followed by a "normal
>> expression".
>>
>> Does that make more sense?
>>
>>> I have read the SMIE manual ten times, at least, but I’m really
>>> struggling.  I would truly appreciate your help.
>>
>> I'm not very good at writing manuals, sorry.  But I promise to do my
>> best to help you get SMIE working well.  In return, I would appreciate
>> if you could help me improve the doc by giving, if not actual patches,
>> at least suggestions of how to rewrite the doc, or what to add to it
>> (usually, you can only make such suggestions after you finally
>> understand what's going on, and at the same time it's
>> important/necessary/useful to try and remember what it was that you
>> didn't understand).
>>
>>
>>         Stefan