Re: SMIE

Re: SMIE

[Stefan Monnier]

> Hi there. As the subject line says I’m writing for help with SMIE.

Cool!

> I am currently working on elixir-mode
> <https://github.com/elixir-lang/emacs-elixir>, having (apparently) taking
> over the mode as the latest in a line of contributors.

I'd love to include this in GNU ELPA.  Interested?

> Specifically I’m having trouble understanding the mental model for how
> tokenisation & indentation works.  For example, in this
> <https://github.com/elixir-lang/emacs-elixir/issues/18> issue, indentation
> errors seem to crop up only after separating lines of code with blank
> lines.
> I have spent, seriously, hundreds of hours trying to sort out what’s
> happening here and I am at my wits’ end.

IIUC, Elixir syntax does not treat all whitespace as "irrelevant",
contrary to the default tokenizer of SMIE.

> Does this issue ring any bells with issues you’ve dealt with in
> the past?

Yes, indeed.  Octave and sh are two other languages that use SMIE and
where some whitespace is syntactically significant.

What you need to do is to change the tokenizer so that instead of
skipping all whitespace, it turns the syntactically-significant
whitespace into a token (you can name it any way you like; in the above
languages, it turns out to be syntactically equivalent to a semi-colon,
so we call it ";").

I know absolutely nothing about Elixir or its syntax, so I can't give
you specific details, but you can look at octave.el and sh-script.el
for examples.  Feel free to email me back with more details if you need
further help.

> Final question, how is it determined if a token is a :list-intro token?

Not sure I understand the question.  The issue is for the indentation
rules, when it sees two (or more) concatenated expressions (e.g. "exp1
exp2"), should it assume that exp2 is something like an argument to the
exp1 function (and hence exp2 (and exp3, ...) should be indented like
a function argument) or are all those "expressions" just a list, where
the first is not more special than the second?
This usually depends on the context.  E.g. in a situation like

   fun x1 x2 x3 =>

x2 is not an argument passed to the function x1;  Instead x1, x2, and x3
are "siblings" and should be indented to the same level.  So to decide
how to indent x2 and x3 w.r.t x1, SMIE calls the smie-rule-function with
(:list-intro . "fun") so smie-rule-function can tell it that "fun"
introduces a *list* of "things" rather than being followed by a "normal
expression".

Does that make more sense?

> I have read the SMIE manual ten times, at least, but I’m really
> struggling.  I would truly appreciate your help.

I'm not very good at writing manuals, sorry.  But I promise to do my
best to help you get SMIE working well.  In return, I would appreciate
if you could help me improve the doc by giving, if not actual patches,
at least suggestions of how to rewrite the doc, or what to add to it
(usually, you can only make such suggestions after you finally
understand what's going on, and at the same time it's
important/necessary/useful to try and remember what it was that you
didn't understand).


        Stefan

Re: SMIE

[Matt DeBoard]

In general I’m having a hard time connecting the dots between the BNF
grammar table creation, the smie-rules (i.e. :before, :after, etc.),
tokenization, indentation, and so forth, and how it all comes together
to make this indentation machine work.

It’s not that the manual is poorly written. In fact between the manual
and the code comments it’s pretty comprehensive. It’s just a high
volume of new information. I think it’ll take me awhile to ingest
everything, but when I do I’d be glad to contribute back.

Regarding the specific issue I mentioned, I closed that out tonight.
There was some overly “greedy” definitions in the syntax table, so
there’s that. I’m slowly starting to pare things away. The bit you
wrote about :list-intro is interesting. When you say that it sees two
or more concatenated expressions, how does that tie in to the BNF
grammar definitions?

On Wed, Jul 9, 2014 at 11:02 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> Hi there. As the subject line says I’m writing for help with SMIE.
>
> Cool!
>
>> I am currently working on elixir-mode
>> <https://github.com/elixir-lang/emacs-elixir>, having (apparently) taking
>> over the mode as the latest in a line of contributors.
>
> I'd love to include this in GNU ELPA.  Interested?
>
>> Specifically I’m having trouble understanding the mental model for how
>> tokenisation & indentation works.  For example, in this
>> <https://github.com/elixir-lang/emacs-elixir/issues/18> issue, indentation
>> errors seem to crop up only after separating lines of code with blank
>> lines.
>> I have spent, seriously, hundreds of hours trying to sort out what’s
>> happening here and I am at my wits’ end.
>
> IIUC, Elixir syntax does not treat all whitespace as "irrelevant",
> contrary to the default tokenizer of SMIE.
>
>> Does this issue ring any bells with issues you’ve dealt with in
>> the past?
>
> Yes, indeed.  Octave and sh are two other languages that use SMIE and
> where some whitespace is syntactically significant.
>
> What you need to do is to change the tokenizer so that instead of
> skipping all whitespace, it turns the syntactically-significant
> whitespace into a token (you can name it any way you like; in the above
> languages, it turns out to be syntactically equivalent to a semi-colon,
> so we call it ";").
>
> I know absolutely nothing about Elixir or its syntax, so I can't give
> you specific details, but you can look at octave.el and sh-script.el
> for examples.  Feel free to email me back with more details if you need
> further help.
>
>> Final question, how is it determined if a token is a :list-intro token?
>
> Not sure I understand the question.  The issue is for the indentation
> rules, when it sees two (or more) concatenated expressions (e.g. "exp1
> exp2"), should it assume that exp2 is something like an argument to the
> exp1 function (and hence exp2 (and exp3, ...) should be indented like
> a function argument) or are all those "expressions" just a list, where
> the first is not more special than the second?
> This usually depends on the context.  E.g. in a situation like
>
>    fun x1 x2 x3 =>
>
> x2 is not an argument passed to the function x1;  Instead x1, x2, and x3
> are "siblings" and should be indented to the same level.  So to decide
> how to indent x2 and x3 w.r.t x1, SMIE calls the smie-rule-function with
> (:list-intro . "fun") so smie-rule-function can tell it that "fun"
> introduces a *list* of "things" rather than being followed by a "normal
> expression".
>
> Does that make more sense?
>
>> I have read the SMIE manual ten times, at least, but I’m really
>> struggling.  I would truly appreciate your help.
>
> I'm not very good at writing manuals, sorry.  But I promise to do my
> best to help you get SMIE working well.  In return, I would appreciate
> if you could help me improve the doc by giving, if not actual patches,
> at least suggestions of how to rewrite the doc, or what to add to it
> (usually, you can only make such suggestions after you finally
> understand what's going on, and at the same time it's
> important/necessary/useful to try and remember what it was that you
> didn't understand).
>
>
>         Stefan

Re: SMIE

[Matt DeBoard]

As regarding inclusion in GNU ELPA, I'm just a caretaker for the
project on behalf of the Elixir-lang people, but as it's already in
MELPA I'm sure it's fine.

On Wed, Jul 9, 2014 at 11:53 PM, Matt DeBoard <matt.deboard@gmail.com> wrote:
> In general I’m having a hard time connecting the dots between the BNF
> grammar table creation, the smie-rules (i.e. :before, :after, etc.),
> tokenization, indentation, and so forth, and how it all comes together
> to make this indentation machine work.
>
> It’s not that the manual is poorly written. In fact between the manual
> and the code comments it’s pretty comprehensive. It’s just a high
> volume of new information. I think it’ll take me awhile to ingest
> everything, but when I do I’d be glad to contribute back.
>
> Regarding the specific issue I mentioned, I closed that out tonight.
> There was some overly “greedy” definitions in the syntax table, so
> there’s that. I’m slowly starting to pare things away. The bit you
> wrote about :list-intro is interesting. When you say that it sees two
> or more concatenated expressions, how does that tie in to the BNF
> grammar definitions?
>
> On Wed, Jul 9, 2014 at 11:02 PM, Stefan Monnier
> <monnier@iro.umontreal.ca> wrote:
>>> Hi there. As the subject line says I’m writing for help with SMIE.
>>
>> Cool!
>>
>>> I am currently working on elixir-mode
>>> <https://github.com/elixir-lang/emacs-elixir>, having (apparently) taking
>>> over the mode as the latest in a line of contributors.
>>
>> I'd love to include this in GNU ELPA.  Interested?
>>
>>> Specifically I’m having trouble understanding the mental model for how
>>> tokenisation & indentation works.  For example, in this
>>> <https://github.com/elixir-lang/emacs-elixir/issues/18> issue, indentation
>>> errors seem to crop up only after separating lines of code with blank
>>> lines.
>>> I have spent, seriously, hundreds of hours trying to sort out what’s
>>> happening here and I am at my wits’ end.
>>
>> IIUC, Elixir syntax does not treat all whitespace as "irrelevant",
>> contrary to the default tokenizer of SMIE.
>>
>>> Does this issue ring any bells with issues you’ve dealt with in
>>> the past?
>>
>> Yes, indeed.  Octave and sh are two other languages that use SMIE and
>> where some whitespace is syntactically significant.
>>
>> What you need to do is to change the tokenizer so that instead of
>> skipping all whitespace, it turns the syntactically-significant
>> whitespace into a token (you can name it any way you like; in the above
>> languages, it turns out to be syntactically equivalent to a semi-colon,
>> so we call it ";").
>>
>> I know absolutely nothing about Elixir or its syntax, so I can't give
>> you specific details, but you can look at octave.el and sh-script.el
>> for examples.  Feel free to email me back with more details if you need
>> further help.
>>
>>> Final question, how is it determined if a token is a :list-intro token?
>>
>> Not sure I understand the question.  The issue is for the indentation
>> rules, when it sees two (or more) concatenated expressions (e.g. "exp1
>> exp2"), should it assume that exp2 is something like an argument to the
>> exp1 function (and hence exp2 (and exp3, ...) should be indented like
>> a function argument) or are all those "expressions" just a list, where
>> the first is not more special than the second?
>> This usually depends on the context.  E.g. in a situation like
>>
>>    fun x1 x2 x3 =>
>>
>> x2 is not an argument passed to the function x1;  Instead x1, x2, and x3
>> are "siblings" and should be indented to the same level.  So to decide
>> how to indent x2 and x3 w.r.t x1, SMIE calls the smie-rule-function with
>> (:list-intro . "fun") so smie-rule-function can tell it that "fun"
>> introduces a *list* of "things" rather than being followed by a "normal
>> expression".
>>
>> Does that make more sense?
>>
>>> I have read the SMIE manual ten times, at least, but I’m really
>>> struggling.  I would truly appreciate your help.
>>
>> I'm not very good at writing manuals, sorry.  But I promise to do my
>> best to help you get SMIE working well.  In return, I would appreciate
>> if you could help me improve the doc by giving, if not actual patches,
>> at least suggestions of how to rewrite the doc, or what to add to it
>> (usually, you can only make such suggestions after you finally
>> understand what's going on, and at the same time it's
>> important/necessary/useful to try and remember what it was that you
>> didn't understand).
>>
>>
>>         Stefan

Re: SMIE

[chad]

On 09 Jul 2014, at 20:59, Matt DeBoard <matt.deboard@gmail.com> wrote:

> As regarding inclusion in GNU ELPA, I'm just a caretaker for the
> project on behalf of the Elixir-lang people, but as it's already in
> MELPA I'm sure it's fine.

I would go a step further than Stefan and say that MELPA is basically the archetype of not-fine in this domain: finding a package in MELPA means, basically "someone, somewhere, wrote some code that might do something, in some of the versions I've had at some point". Finding a package in GNU ELPA (or a handful of other elpa repositories) adds things like "this version should actually do what's written on the tin" to potentially-interesting properties like "copyrights already assigned" and "can be fixed by interested emacs maintainers".

I hope that helps,
~Chad

Re: SMIE

[Matt DeBoard]

[-- Attachment #1: Type: text/plain, Size: 990 bytes --]

Sorry for the thread necromancy. What do I need to do for inclusion in GNU
ELPA?


On Thu, Jul 10, 2014 at 12:00 PM, chad <yandros@gmail.com> wrote:

> On 09 Jul 2014, at 20:59, Matt DeBoard <matt.deboard@gmail.com> wrote:
>
> > As regarding inclusion in GNU ELPA, I'm just a caretaker for the
> > project on behalf of the Elixir-lang people, but as it's already in
> > MELPA I'm sure it's fine.
>
> I would go a step further than Stefan and say that MELPA is basically the
> archetype of not-fine in this domain: finding a package in MELPA means,
> basically "someone, somewhere, wrote some code that might do something, in
> some of the versions I've had at some point". Finding a package in GNU ELPA
> (or a handful of other elpa repositories) adds things like "this version
> should actually do what's written on the tin" to potentially-interesting
> properties like "copyrights already assigned" and "can be fixed by
> interested emacs maintainers".
>
> I hope that helps,
> ~Chad
>
>

[-- Attachment #2: Type: text/html, Size: 1461 bytes --]

Re: SMIE

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 1516 bytes --]

On August 16, 2014 at 5:35:29 AM, Matt DeBoard (matt.deboard@gmail.com) wrote:
Sorry for the thread necromancy. What do I need to do for inclusion in GNU ELPA?
You need to get everyone with non-trivial contributions to the project to sign the FSF copyright assignment

document. You’ll also have to stop accepting non-trivial contributions from people who haven’t signed the document.

The latter is my primary problem with having a package go to ELPA as you’re losing a lot of useful contributions from GitHub,

as most folks are unlikely to want to deal with a paper document, which is unfortunate. 




On Thu, Jul 10, 2014 at 12:00 PM, chad <yandros@gmail.com> wrote:
On 09 Jul 2014, at 20:59, Matt DeBoard <matt.deboard@gmail.com> wrote:

> As regarding inclusion in GNU ELPA, I'm just a caretaker for the
> project on behalf of the Elixir-lang people, but as it's already in
> MELPA I'm sure it's fine.

I would go a step further than Stefan and say that MELPA is basically the archetype of not-fine in this domain: finding a package in MELPA means, basically "someone, somewhere, wrote some code that might do something, in some of the versions I've had at some point". Finding a package in GNU ELPA (or a handful of other elpa repositories) adds things like "this version should actually do what's written on the tin" to potentially-interesting properties like "copyrights already assigned" and "can be fixed by interested emacs maintainers".

I hope that helps,
~Chad




[-- Attachment #2: Type: text/html, Size: 3519 bytes --]

Re: SMIE

[Stefan Monnier]

> You’ll also have to stop accepting non-trivial contributions from
> people who haven’t signed the document.

Not quite, no.  You simply have to ask them to sign the
copyright paperwork.  Most people are willing to do that in my experience.


        Stefan

Re: SMIE

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On August 17, 2014 at 2:55:10 PM, Stefan Monnier (monnier@iro.umontreal.ca) wrote:
> You’ll also have to stop accepting non-trivial contributions from 
> people who haven’t signed the document. 

Not quite, no. You simply have to ask them to sign the 
copyright paperwork. Most people are willing to do that in my experience. 


Stefan 
My point isn’t that people aren’t willing sign the copyright agreement in principle, but that many people will not be thrilled to

deal with a *paper* copyright agreement that they have to send via standard mail and wait for a couple of weeks to get

back a response, just to get a tiny fix/change accepted. I think that GNU projects would benefit a lot from a contributor agreement that can be

signed digitally. (Clojure’s development recently switched to digital contributor agreements and the number of people who signed the agreement skyrocketed).



People hate dealing with paperwork and that’s a fact of life...

[-- Attachment #2: Type: text/html, Size: 2149 bytes --]

Re: SMIE

[Tassilo Horn]

Bozhidar Batsov <bozhidar.batsov@gmail.com> writes:

> My point isn’t that people aren’t willing sign the copyright agreement
> in principle, but that many people will not be thrilled to deal with a
> *paper* copyright agreement that they have to send via standard mail
> and wait for a couple of weeks to get back a response, just to get a
> tiny fix/change accepted. I think that GNU projects would benefit a
> lot from a contributor agreement that can be signed
> digitally.

GNU does accept digitally signed CAs, but currently only for some
countries.  See

  https://www.fsf.org/blogs/licensing/fsf-to-begin-accepting-scanned-assignments-from-germany

Not sure if that has been extended to more countries in the meantime.

Bye,
Tassilo

BTW: Can you teach your mail client to quote text you are replying to?
The quoted text is indented in the HTML part but not in the plain text
part.

Bye,
Tassilo

Re: SMIE

[Stefan Monnier]

> My point isn’t that people aren’t willing sign the copyright agreement
> in principle,

I was responding to your saying "You’ll also have to stop accepting".
Maybe you meant something else, but that's what you had written.

> but that many people will not be thrilled to deal with a *paper*
> copyright agreement that they have to send via standard mail and wait
> for a couple of weeks to get back a response, just to get a tiny
> fix/change accepted.

Additionally to Tassilo's answer: if it's tiny it doesn't need
such paperwork.

Yes, copyright paperwork annoys a few people.  Some of those are
sufficiently annoyed to refuse doing it, but most of them (like
yourself) will still accept doing it.  Especially if you remind them
that it covers all Emacs contributions rather than just the
current patch.


        Stefan

Re: SMIE

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We accept digital signatures from countries for which our lawyer
says that is ok.  That's how it has to be.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: SMIE

[Bastien]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> We accept digital signatures from countries for which our lawyer
> says that is ok.  That's how it has to be.

Those countries are the United States and Germany.

What does it take to accept digital signatures from other countries?
E.g. from France?  Can we help here?

-- 
 Bastien

Re: SMIE

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    What does it take to accept digital signatures from other countries?
    E.g. from France?  Can we help here?

I don't know, but a priori it seems unlikely that you or I could
do anything that would change this legal issue.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: SMIE

[Matt DeBoard]

[-- Attachment #1: Type: text/plain, Size: 2123 bytes --]

To bring the thread back around full-circle to the topic of SMIE...

Stefan I wound up rewriting the `forward-token' and `backward-token'
functions, discarding the previous code (which I inherited) in favor of
much simpler, more concise code I found in `ruby-mode' and `swift-mode'. I
took up this project (https://github.com/elixir-lang/emacs-elixir) just a
couple months ago, so I thought the convolution of the code I replaced was
a result of SMIE. I was skeptical about the degree to which I would be able
to simplify the logic.

However, after cutting out over 150 lines of code by leaning heavily on
SMIE, I just wanted to say thanks. SMIE -- once you get it -- is really
valuable. There's a learning curve for sure, but it's a really pragmatic
tool.

To any future readers who are looking for guidance on how to implement
SMIE-driven indentation in your language mode, below are a few links of
simple, concise, valid SMIE implementations that can serve as reference. In
order of simplicity (IMO):

elixir-mode:

https://github.com/elixir-lang/emacs-elixir/blob/v1.4.8/elixir-smie.el

swift-mode:

https://github.com/chrisbarrett/swift-mode/blob/6abb78812bd5795950c1872787575c8eb72e8574/swift-mode.el

ruby-mode:

http://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/ruby-mode.el?h=emacs-24&id=2605b19cfa2ff47066c91dfd2650b40645d51b32


On Tue, Aug 19, 2014 at 9:27 PM, Richard Stallman <rms@gnu.org> wrote:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>     What does it take to accept digital signatures from other countries?
>     E.g. from France?  Can we help here?
>
> I don't know, but a priori it seems unlikely that you or I could
> do anything that would change this legal issue.
>
> --
> Dr Richard Stallman
> President, Free Software Foundation
> 51 Franklin St
> Boston MA 02110
> USA
> www.fsf.org  www.gnu.org
> Skype: No way! That's nonfree (freedom-denying) software.
>   Use Ekiga or an ordinary phone call.
>
>

[-- Attachment #2: Type: text/html, Size: 3395 bytes --]

Re: SMIE

[Ivan Andrus]

[-- Attachment #1: Type: text/plain, Size: 2597 bytes --]

I enjoyed using it for gap-mode [1] once I got the hang of it, though I would hesitate to suggest that people should look at my implementation. :-)  And I love, love, love the sexp-based movement it gives me for free.

-Ivan

[1] https://bitbucket.org/gvol/gap-mode/src/tip/gap-smie.el?at=default

On Aug 19, 2014, at 9:19 PM, Matt DeBoard <matt.deboard@gmail.com> wrote:

> To bring the thread back around full-circle to the topic of SMIE...
> 
> Stefan I wound up rewriting the `forward-token' and `backward-token' functions, discarding the previous code (which I inherited) in favor of much simpler, more concise code I found in `ruby-mode' and `swift-mode'. I took up this project (https://github.com/elixir-lang/emacs-elixir) just a couple months ago, so I thought the convolution of the code I replaced was a result of SMIE. I was skeptical about the degree to which I would be able to simplify the logic.
> 
> However, after cutting out over 150 lines of code by leaning heavily on SMIE, I just wanted to say thanks. SMIE -- once you get it -- is really valuable. There's a learning curve for sure, but it's a really pragmatic tool.
> 
> To any future readers who are looking for guidance on how to implement SMIE-driven indentation in your language mode, below are a few links of simple, concise, valid SMIE implementations that can serve as reference. In order of simplicity (IMO):
> 
> elixir-mode:
> 
> https://github.com/elixir-lang/emacs-elixir/blob/v1.4.8/elixir-smie.el
> 
> swift-mode:
> 
> https://github.com/chrisbarrett/swift-mode/blob/6abb78812bd5795950c1872787575c8eb72e8574/swift-mode.el
> 
> ruby-mode:
> 
> http://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/ruby-mode.el?h=emacs-24&id=2605b19cfa2ff47066c91dfd2650b40645d51b32
> 
> 
> On Tue, Aug 19, 2014 at 9:27 PM, Richard Stallman <rms@gnu.org> wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>     What does it take to accept digital signatures from other countries?
>     E.g. from France?  Can we help here?
> 
> I don't know, but a priori it seems unlikely that you or I could
> do anything that would change this legal issue.
> 
> --
> Dr Richard Stallman
> President, Free Software Foundation
> 51 Franklin St
> Boston MA 02110
> USA
> www.fsf.org  www.gnu.org
> Skype: No way! That's nonfree (freedom-denying) software.
>   Use Ekiga or an ordinary phone call.
> 
> 


[-- Attachment #2: Type: text/html, Size: 4282 bytes --]

Re: SMIE

[Stefan Monnier]

> I enjoyed using it for gap-mode [1] once I got the hang of it, though
> I would hesitate to suggest that people should look at my
> implementation. :-)  And I love, love, love the sexp-based movement it gives
> me for free.

Thanks guys.  I'm really glad you like it,


        Stefan

Re: SMIE

[Matt DeBoard]

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]

I know this is a long shot, but I am reading through ruby-mode.el's grammar
table, and it uses the token "iuwu-mod" but I have no idea what that means.
It also doesn't appear to be documented and a google search doesn't yield
any meaningful clues.


On Wed, Aug 20, 2014 at 9:25 AM, Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > I enjoyed using it for gap-mode [1] once I got the hang of it, though
> > I would hesitate to suggest that people should look at my
> > implementation. :-)  And I love, love, love the sexp-based movement it
> gives
> > me for free.
>
> Thanks guys.  I'm really glad you like it,
>
>
>         Stefan
>

[-- Attachment #2: Type: text/html, Size: 1039 bytes --]

Re: SMIE

[Stefan Monnier]

> I know this is a long shot, but I am reading through ruby-mode.el's grammar
> table, and it uses the token "iuwu-mod" but I have no idea what that means.

The SMIE grammar does not use "keywords names" but "token names".
Most tokens represent a keyword and use the same name as that keyword,
but sometimes the same keyword is mapped to different tokens depending
on context.  That's what the *-forward-token and *-backward-token
functions are for.

If you look at ruby-smie--forward-token, for example, you'll see that it
can return "iuwu-mod" in the following case:

           ((member tok '("unless" "if" "while" "until"))
            (if (save-excursion (forward-word -1) (ruby-smie--bosp))
                tok "iuwu-mod"))

So I guess you can now guess what "iuwu" stands for.  As for the "mod",
it stands for "statement MODifier", IIRC.


        Stefan

Re: SMIE

[David Kastrup]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> I know this is a long shot, but I am reading through ruby-mode.el's grammar
>> table, and it uses the token "iuwu-mod" but I have no idea what that means.
>
> The SMIE grammar does not use "keywords names" but "token names".
> Most tokens represent a keyword and use the same name as that keyword,
> but sometimes the same keyword is mapped to different tokens depending
> on context.  That's what the *-forward-token and *-backward-token
> functions are for.
>
> If you look at ruby-smie--forward-token, for example, you'll see that it
> can return "iuwu-mod" in the following case:
>
>            ((member tok '("unless" "if" "while" "until"))
>             (if (save-excursion (forward-word -1) (ruby-smie--bosp))
>                 tok "iuwu-mod"))
>
> So I guess you can now guess what "iuwu" stands for.

"conditional".  Would that have been so hard?

-- 
David Kastrup

Re: SMIE

[Dmitry]

David Kastrup <dak@gnu.org> writes:

> "conditional".  Would that have been so hard?

Too generic. And I wouldn't really call "while" and "until" conditional
keywords.

It's a statement modifier because it's specifically modifies the
how the statement that goes before it, is interpreted.

Re: SMIE

[Andreas Röhler]

On 10.07.2014 19:00, chad wrote:
> On 09 Jul 2014, at 20:59, Matt DeBoard <matt.deboard@gmail.com> wrote:
>
>> As regarding inclusion in GNU ELPA, I'm just a caretaker for the
>> project on behalf of the Elixir-lang people, but as it's already in
>> MELPA I'm sure it's fine.
>
> I would go a step further than Stefan and say that MELPA is basically the archetype of not-fine in this domain: finding a package in MELPA means, basically "someone, somewhere, wrote some code that might do something, in some of the versions I've had at some point". Finding a package in GNU ELPA (or a handful of other elpa repositories) adds things like "this version should actually do what's written on the tin" to potentially-interesting properties like "copyrights already assigned" and "can be fixed by interested emacs maintainers".
>
> I hope that helps,
> ~Chad

Hi Chad,

being able to use and distribute code written by "someone, somewhere,..." IMO is perfectly fine, it's the spirit of freedom.

OTOH that argument fixing by "interested Emacs maintainers" seems not to work that great as expected.
Fixing a language mode requires some knowledge of the language. Otherwise there is a risk of nagging the maintainer.

WRT to the pure amount of existing languages, lets have a friendly attitude towards Emacs' community contributions.

Andreas

Re: SMIE

[Stefan Monnier]

> In general I’m having a hard time connecting the dots between the BNF
> grammar table creation, the smie-rules (i.e. :before, :after, etc.),
> tokenization, indentation, and so forth, and how it all comes together
> to make this indentation machine work.

The tokenizer turns the text (sequence of chars) into a sequence of
"tokens", which should be thought of as "words".  This step gets rid of
things like comments and (syntactically insignificant) whitespace.

The BNF then describes which of those tokens are special (people usually
call them "keywords" if they look like human words and "operators" if
they look like math entities) and how they relate to each other to
provide structure.

These two together are sufficient to get C-M-f and C-M-b to do something
useful such as skip from "begin" to the matching "end" and vice-versa.
C-M-f/C-M-b also do something useful when called right before/after an
"infix" keyword: they skip over the corresponding right/left element.
E.g. C-M-b starting from right after "then" should jump to the matching
"if" (it may jump to either just before it or just after it, depending
on the particular way the BNF is specified), and C-M-f from right before
"+" should jump over the whole expression that's being added.

It's generally very useful to debug the BNF and tokenizer using C-M-f
and C-M-b: if these don't jump to the right place, then the indentation
rules won't work well anyway.  Note that these C-M-f and C-M-b may
sometimes stop before and sometimes after the "destination keyword",
depending on details of the BNF grammar, but usually either choice is
fine in the sense that it's "close enough" that the indentation rules
can then be made to work.

The indentation algorithm then works by looking at the immediately
surrounding tokens (the one before and the one after), asks the
rules-function what is the indentation rule for those tokens, and when
needed jumps to the "parent" with (more or less) C-M-b.

> there’s that. I’m slowly starting to pare things away. The bit you
> wrote about :list-intro is interesting. When you say that it sees two
> or more concatenated expressions, how does that tie in to the BNF
> grammar definitions?

When you have "exp1 exp2 exp3", there is no keyword/operator involved
(tho keyword/operators may be involved inside exp[123], of course), so
the BNF definition doesn't say anything about it.

When the BNF says ("BEGIN" exp "END"), SMIE interprets it as "we can
have any number of `exp' or any other non-special token between BEGIN
and END".  And that is true of all BNF rules in SMIE (SMIE always
accepts any number of non-special tokens anywhere between the special
tokens, and any repetition).

IOW it doesn't tie in to the BNF grammar definition at all, because this
is imposed implicitly for all grammars by SMIE's underlying algorithm.

E.g. the grammar

   (defvar my-smie-bnf '((id)
                         (exp ("BEGIN" exp "END")
                              (id ":=" exp))))

will happily consider "a b c := BEGIN d e END" as a valid `exp'.
Actually, it will also accept "BEGIN a b c END := BEGIN d e END",
because as soon as you have such a "parenthesis-like" element, SMIE will
also accept it anywhere (it is a really dumb parsing algorithm which
pays very little attention to the context).

> One final question. In the case of e.g.
> > Can't resolve the precedence cycle: .do < else:. < .do
> What does the placement of the dots (left of "do", right of "else:") mean?

SMIE reduces the BNF grammar into a simple table of precedences.
Each token gets assigned 2 precedences: a left-precedence and
a right-precedence.  So the "." indicates which precedence is affected:
the above says that there's a cycle between the left precedence of "do"
and the right precedence of "else:".

A good way to think about those < and > is as "parenthese":

The rule "else:. < .do" means that SMIE thinks (because of some part of
the BNF grammar) that if the code looks like:

   ... else: .. .. do ...

it should be parsed as

   ... else: ..(.. do ...

rather than as

   ... else: ..).. do ...

And the rule ".do < else:." (or better said "else:. > .do") says just
the reverse:

   ... else: .. .. do ...

it should be parsed as

   ... else: ..).. do ...

rather than as

   ... else: ..(.. do ...

So, SMIE doesn't know which is true.  Sadly, the code I wrote does not
keep track of where those inequalities come from, so it's unable to
point you to the particular part of the BNF rules which made it think
that "else:. < .do" or that ".do < else:.".

> As regarding inclusion in GNU ELPA, I'm just a caretaker for the
> project on behalf of the Elixir-lang people, but as it's already in
> MELPA I'm sure it's fine.

Inclusion into GNU ELPA is slightly different in that it's a half-way
inclusion in Emacs: on the technical side, it means that there's a Git
branch of the code kept along side Emacs's repository (and to which
Emacs maintainers can write) from which the GNU ELPA package get built,
and on the legal side it means that the code has to have the same
copyright status as Emacs, which concretely means that all non-trivial
contributors need to have signed some copyright paperwork.


        Stefan

Re: SMIE

[Stephen Leake]

Matt DeBoard <matt.deboard@gmail.com> writes:

> In general I’m having a hard time connecting the dots between the BNF
> grammar table creation, the smie-rules (i.e. :before, :after, etc.),
> tokenization, indentation, and so forth, and how it all comes together
> to make this indentation machine work.

You might consider using wisi
(http://stephe-leake.org/emacs/ada-mode/emacs-ada-mode.html, and in
Gnu ELPA) instead of SMIE.

The main difference is that wisi is a full LALR parser, while SMIE is an
incremental operator grammar parser. In practice, that means the wisi
grammar is _much_ closer to the language BNF, and thus possibly easier
to understand.

On the downside, wisi must parse the entire compilation unit (top level
grammar item), so it can be slow on large files. I'm working on
implementing a parser state cache to speed up re-parsing while editing.

Currently, wisi is only used in Ada mode and gpr mode (GNAT
project files), so there are fewer examples to work from. 

I'd be interested in your feedback.

-- 
-- Stephe

Re: SMIE

[Matt DeBoard]

One final question. In the case of e.g.

> Can't resolve the precedence cycle: .do < else:. < .do

What does the placement of the dots (left of "do", right of "else:") mean?

On Wed, Jul 9, 2014 at 11:02 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> Hi there. As the subject line says I’m writing for help with SMIE.
>
> Cool!
>
>> I am currently working on elixir-mode
>> <https://github.com/elixir-lang/emacs-elixir>, having (apparently) taking
>> over the mode as the latest in a line of contributors.
>
> I'd love to include this in GNU ELPA.  Interested?
>
>> Specifically I’m having trouble understanding the mental model for how
>> tokenisation & indentation works.  For example, in this
>> <https://github.com/elixir-lang/emacs-elixir/issues/18> issue, indentation
>> errors seem to crop up only after separating lines of code with blank
>> lines.
>> I have spent, seriously, hundreds of hours trying to sort out what’s
>> happening here and I am at my wits’ end.
>
> IIUC, Elixir syntax does not treat all whitespace as "irrelevant",
> contrary to the default tokenizer of SMIE.
>
>> Does this issue ring any bells with issues you’ve dealt with in
>> the past?
>
> Yes, indeed.  Octave and sh are two other languages that use SMIE and
> where some whitespace is syntactically significant.
>
> What you need to do is to change the tokenizer so that instead of
> skipping all whitespace, it turns the syntactically-significant
> whitespace into a token (you can name it any way you like; in the above
> languages, it turns out to be syntactically equivalent to a semi-colon,
> so we call it ";").
>
> I know absolutely nothing about Elixir or its syntax, so I can't give
> you specific details, but you can look at octave.el and sh-script.el
> for examples.  Feel free to email me back with more details if you need
> further help.
>
>> Final question, how is it determined if a token is a :list-intro token?
>
> Not sure I understand the question.  The issue is for the indentation
> rules, when it sees two (or more) concatenated expressions (e.g. "exp1
> exp2"), should it assume that exp2 is something like an argument to the
> exp1 function (and hence exp2 (and exp3, ...) should be indented like
> a function argument) or are all those "expressions" just a list, where
> the first is not more special than the second?
> This usually depends on the context.  E.g. in a situation like
>
>    fun x1 x2 x3 =>
>
> x2 is not an argument passed to the function x1;  Instead x1, x2, and x3
> are "siblings" and should be indented to the same level.  So to decide
> how to indent x2 and x3 w.r.t x1, SMIE calls the smie-rule-function with
> (:list-intro . "fun") so smie-rule-function can tell it that "fun"
> introduces a *list* of "things" rather than being followed by a "normal
> expression".
>
> Does that make more sense?
>
>> I have read the SMIE manual ten times, at least, but I’m really
>> struggling.  I would truly appreciate your help.
>
> I'm not very good at writing manuals, sorry.  But I promise to do my
> best to help you get SMIE working well.  In return, I would appreciate
> if you could help me improve the doc by giving, if not actual patches,
> at least suggestions of how to rewrite the doc, or what to add to it
> (usually, you can only make such suggestions after you finally
> understand what's going on, and at the same time it's
> important/necessary/useful to try and remember what it was that you
> didn't understand).
>
>
>         Stefan