bug#22960: 25.0.92; smie-indent-keyword triggers the "Bumped into unknown token" error

bug#22960: 25.0.92; smie-indent-keyword triggers the "Bumped into unknown token" error

[Dmitry Gutov]

When trying to indent the beginning of a multiline string literal.

Here's an example Ruby code (almost unmodified, from a real project):

foo_bar(
   "# coding:koi8-r
         \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 = 42
         puts \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1")

Try to reindent the second line.  During its execution,
smie-indent-keyword calls smie-indent-forward-token from the third line
(in the middle of the string literal).  ruby-smie--forward-token returns
"" there (because why not?), and smie-indent-forward-token freaks out.

IMHO, smie-forward-token-function doesn't have to know how to handle
that context, and the fault lies in smie-indent-keyword.

Stefan, could you please take a look?


In GNU Emacs 25.0.92.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.7)
 of 2016-03-05 built on axl
Repository revision: dbb02bfadc38297c06bbd3462e53192d0ced76dc
Windowing system distributor 'The X.Org Foundation', version 11.0.11702000
System Description:	Ubuntu 15.10

bug#22960: 25.0.92; smie-indent-keyword triggers the "Bumped into unknown token" error

[Stefan Monnier]

> foo_bar(
>    "# coding:koi8-r
>          \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 = 42
>          puts \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1")
>
> Try to reindent the second line.  During its execution,

I installed the patch below into emacs-25, which should fix it.


        Stefan


diff --git a/lisp/emacs-lisp/smie.el b/lisp/emacs-lisp/smie.el
index 495ba7c..1d8f0cb 100644
--- a/lisp/emacs-lisp/smie.el
+++ b/lisp/emacs-lisp/smie.el
@@ -1493,7 +1493,10 @@ should not be computed on the basis of the following token."
                            (let ((endpos (point)))
                              (goto-char pos)
                              (forward-line 1)
-                             (and (equal res (smie-indent-forward-token))
+                             ;; As seen in bug#22960, pos may be inside
+                             ;; a string, and forward-token may then stumble.
+                             (and (ignore-errors
+                                    (equal res (smie-indent-forward-token)))
                                   (eq (point) endpos)))))
                     nil
                   (goto-char pos)

bug#22960: 25.0.92; smie-indent-keyword triggers the "Bumped into unknown token" error

[Dmitry Gutov]

On 03/17/2016 04:56 AM, Stefan Monnier wrote:
>> foo_bar(
>>    "# coding:koi8-r
>>          \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1 = 42
>>          puts \xd0\xd2\xcf\xd7\xc5\xd2\xcb\xc1")
>>
>> Try to reindent the second line.  During its execution,
>
> I installed the patch below into emacs-25, which should fix it.

Thanks, that did it.

By the way, while testing it, I've noticed that f54e5ccf added new 
behavior to indent-for-tab-command. Do you think it should be mentioned 
in NEWS?