Re: trunk r114534: Get Ruby's SMIE code to pass the test suite.

[Dmitry Gutov <dgutov@yandex.ru>]

On 21.10.2013 20:46, Stefan Monnier wrote:
>>> Maybe change the tokenizer so that a ".qux" *at smie-bolp* is tokenized
>>> as "." and "qux"?
>> That will probably break the following modified example:
>> class C
>>    def foo
>>      self
>>        .end
>>      D.new
>>       .class
>>    end
>> end
>
> Yes, the problem is that the set of desired indentation does not match
> the structure of a fixed parsing.  So, some of the differences need to
> be handled in ad-hoc ways in the ruby-smie-rules function.
>
> E.g. for (:before . ".") we'd have to look at the previous token and if
> it's of the form "foo.bar", then manually align with the "." of
> "foo.bar".

Indentation is not the only problem there. Tokenizing ".end" as [".", 
"end"] also breaks sexp movement and (for example) 
`smie--matching-block-data'. The dot concatenation logic is the result 
of you fixing that in revision 114545.

>> But I guess we can make the tokenizer both return "." as a separate
>> token and include it in the token that goes after it.
>
> I'm not sure I understand exactly what you're suggesting, but I'd
> venture to say that you can't do that.

I don't see why not. I've installed a patch along these lines in 114738, 
and it fixes the existing examples. Unfortunately, it broke the more 
complex situations:

foo do
   bar
     .tee
end

def bar
   foo
     .baz
end

...apparently because (goto-char (cadr (smie-indent--parent))), when 
called before "." in these examples, brings point to the end of the 
first line, and so (smie-rule-parent ruby-indent-level) returns a 
surprising result.

Any suggestions what to do about that?