ruby-mide, SMIE and token priority

ruby-mide, SMIE and token priority

[Dmitry Gutov]

Stefan,

There's one example in indent/ruby.rb I don't know what to do about:

bar.foo(tee) do
   bar
end

The problem seems to be that "." is considered the parent token of "do". 
Probably because the left priority of "do" is not a number.

It would be more natural if the parent of "do" was ";" on the preceding 
line, instead. Is it at all possible to change the grammar this way?

Or should we just handle this as a special case in the 
`ruby-smie-rules', and in case of "do", instead of delegating to the 
parent, skip the parents until we find ";"?

P.S. And here's a similar example with a curly block:

bar.foo(tee) {
   bar
}

Re: ruby-mide, SMIE and token priority

[Stefan Monnier]

> bar.foo(tee) do
>   bar
> end

What is the AST corresponding to this code?
I mean, is it a method call on "bar" with method "foo" and 2 arguments
(tee and do...end)?  I.e. (. bar foo tee (do-end bar))?

> The problem seems to be that "." is considered the parent token of
> "do".  Probably because the left priority of "do" is not a number.

SMIE parses it as (. bar (call foo tee (do-end bar))).

> It would be more natural if the parent of "do" was ";" on the preceding
> line, instead.

You mean parse it as (call (. bar foo tee) (do-end bar))?

> Is it at all possible to change the grammar this way?

You'd probably have to use a trick similar to the " @ " used on the
space between the method name and the multiple-args.

> Or should we just handle this as a special case in the `ruby-smie-rules',
> and in case of "do", instead of delegating to the parent, skip the parents
> until we find ";"?

Depends: the rule of thumb is that if you have to choose between "make
the grammar match the AST" or "make the grammar match the indentation",
then better choose "match the AST" and then adjust the indentation via
the smie-rules.

So if the indentation you want is indeed not faithful to the AST, then
indeed it's better to fix it in smie-rules.  But otherwise, if you can
fix it in the grammar, then it's preferable.


        Stefan

Re: ruby-mide, SMIE and token priority

[Dmitry Gutov]

On 07.11.2013 06:30, Stefan Monnier wrote:
>> bar.foo(tee) do
>>    bar
>> end
>
> What is the AST corresponding to this code?
> I mean, is it a method call on "bar" with method "foo" and 2 arguments
> (tee and do...end)?  I.e. (. bar foo tee (do-end bar))?

More or less, yes. "bar" could also be a method, but the structure would 
be the same.

>> The problem seems to be that "." is considered the parent token of
>> "do".  Probably because the left priority of "do" is not a number.
>
> SMIE parses it as (. bar (call foo tee (do-end bar))).
 >
>> It would be more natural if the parent of "do" was ";" on the preceding
>> line, instead.
>
> You mean parse it as (call (. bar foo tee) (do-end bar))?

Guess so. The argument in favor of doing it this way is the grammar is 
already a bit skewed toward indentation (for example, " @ " has higher 
priority than "."), but that's likely something to be fixed, too.

>> Is it at all possible to change the grammar this way?
>
> You'd probably have to use a trick similar to the " @ " used on the
> space between the method name and the multiple-args.

Ah, okay. Sounds not very efficient, performance-wise.

>> Or should we just handle this as a special case in the `ruby-smie-rules',
>> and in case of "do", instead of delegating to the parent, skip the parents
>> until we find ";"?
>
> Depends: the rule of thumb is that if you have to choose between "make
> the grammar match the AST" or "make the grammar match the indentation",
> then better choose "match the AST" and then adjust the indentation via
> the smie-rules.
>
> So if the indentation you want is indeed not faithful to the AST, then
> indeed it's better to fix it in smie-rules.

Let's try this way, then.

Re: ruby-mide, SMIE and token priority

[Stefan Monnier]

>>> Is it at all possible to change the grammar this way?
>> You'd probably have to use a trick similar to the " @ " used on the
>> space between the method name and the multiple-args.
> Ah, okay. Sounds not very efficient, performance-wise.

Could be.  Every trick we add to the tokenizer is a potential
performance problem, indeed.  On the contrary, code in the
rules-function is generally not performance sensitive.


        Stefan

Re: ruby-mide, SMIE and token priority

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

I've just noticed that issue myself. Is there any progress on it?


On 7 November 2013 18:02, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> >>> Is it at all possible to change the grammar this way?
> >> You'd probably have to use a trick similar to the " @ " used on the
> >> space between the method name and the multiple-args.
> > Ah, okay. Sounds not very efficient, performance-wise.
>
> Could be.  Every trick we add to the tokenizer is a potential
> performance problem, indeed.  On the contrary, code in the
> rules-function is generally not performance sensitive.
>
>
>         Stefan
>
>

[-- Attachment #2: Type: text/html, Size: 1074 bytes --]

Re: ruby-mide, SMIE and token priority

[Dmitry Gutov]

Bozhidar Batsov <bozhidar@batsov.com> writes:

> I've just noticed that issue myself. Is there any progress on it?

Yes. Try the latest trunk, and check out the examples in
test/indent/ruby.rb. All of them currently work (or else there would be
a comment saying that some don't).

If you have any new broken examples or disagree with some of the choices
in ruby.rb, please tell.

>
> On 7 November 2013 18:02, Stefan Monnier <monnier@iro.umontreal.ca>
> wrote:
>
>     >>> Is it at all possible to change the grammar this way?
>     >> You'd probably have to use a trick similar to the " @ " used on
>     the
>     >> space between the method name and the multiple-args.
>     > Ah, okay. Sounds not very efficient, performance-wise.
>     
>     
>     Could be. Every trick we add to the tokenizer is a potential
>     performance problem, indeed. On the contrary, code in the
>     rules-function is generally not performance sensitive.
>     
>     
>     Stefan
>     
>     

Re: ruby-mide, SMIE and token priority

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

At first glance everything looks good to me. I was extremely happy to see
you've fixed the indentation for:

if foo &&
   bar
end

The old (incorrect) behaviour was quite an annoyance for many Ruby
programmers.



On 11 November 2013 14:56, Dmitry Gutov <dgutov@yandex.ru> wrote:

> Bozhidar Batsov <bozhidar@batsov.com> writes:
>
> > I've just noticed that issue myself. Is there any progress on it?
>
> Yes. Try the latest trunk, and check out the examples in
> test/indent/ruby.rb. All of them currently work (or else there would be
> a comment saying that some don't).
>
> If you have any new broken examples or disagree with some of the choices
> in ruby.rb, please tell.
>
> >
> > On 7 November 2013 18:02, Stefan Monnier <monnier@iro.umontreal.ca>
> > wrote:
> >
> >     >>> Is it at all possible to change the grammar this way?
> >     >> You'd probably have to use a trick similar to the " @ " used on
> >     the
> >     >> space between the method name and the multiple-args.
> >     > Ah, okay. Sounds not very efficient, performance-wise.
> >
> >
> >     Could be. Every trick we add to the tokenizer is a potential
> >     performance problem, indeed. On the contrary, code in the
> >     rules-function is generally not performance sensitive.
> >
> >
> >     Stefan
> >
> >
>

[-- Attachment #2: Type: text/html, Size: 2081 bytes --]

Re: ruby-mide, SMIE and token priority

[Dmitry Gutov]

On 11.11.2013 17:13, Bozhidar Batsov wrote:
> At first glance everything looks good to me. I was extremely happy to
> see you've fixed the indentation for:
>
> if foo &&
>    bar
> end
>
> The old (incorrect) behaviour was quite an annoyance for many Ruby
> programmers.

I'm glad you like it.

Lining up foo and bar makes sense from the syntax standpoint, so SMIE 
more or less does that by default, but I was afraid it would hurt 
readability: this way, "bar" is indented only 1 column more than the 
block body, so it could be mistaken for the first statement more easily.