bug#21875: 24.5; Invalid tokens in smie indentation rules

bug#21875: 24.5; Invalid tokens in smie indentation rules

[mail]

For the simple grammar like this:

(id)
(inst ("if" exp "begin" insts "end")
      (exp))
(insts (insts ";" insts) (inst))
(exp (exp "." id)
     (id ":" exp)
     (exp "=" exp))
(exps (exps "," exps) (exp))

And lexer rule that overrides braces with begin/end tokens, for example
like this:

((looking-at "{") (forward-char 1) "begin")
((looking-at "}") (forward-char 1) "end")

When trying to indent such construction:

if true {
    bar
|}

I'm recieveing indentation request for :close-all '}', but no such
token was returned by lexer, full log:

forward: 19 -> 20 = end
backward: 20 -> 19 = end
backward: 19 -> 18 = ;
backward: 18 -> 15 = bar
backward: 15 -> 9 = begin
backward: 9 -> 4 = true
backward: 4 -> 1 = if
:close-all '}'; sibling-p:t parent:(nil 1 if) hanging:nil == nil
backward: 20 -> 19 = end
backward: 19 -> 18 = ;
backward: 18 -> 15 = bar
backward: 15 -> 9 = begin
backward: 9 -> 4 = true
backward: 4 -> 1 = if



In GNU Emacs 24.5.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.6)
 of 2015-09-10 on foutrelis
Windowing system distributor `The X.Org Foundation', version 11.0.11702000
System Description:	Arch Linux

Configured using:
 `configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong
 --param=ssp-buffer-size=4' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'

Important settings:
  value of $EMACSLOADPATH: /home/ap4y/github/swift-mode/.cask/24.5.1/elpa/flycheck-cask-20150920.453:/home/ap4y/github/swift-mode/.cask/24.5.1/elpa/flycheck-20151027.755:/home/ap4y/github/swift-mode/.cask/24.5.1/elpa/dash-20151021.113:/home/ap4y/github/swift-mode/.cask/24.5.1/elpa/pkg-info-20150517.443:/home/ap4y/github/swift-mode/.cask/24.5.1/elpa/epl-20150517.433:/home/ap4y/github/swift-mode/.cask/24.5.1/elpa/let-alist-1.0.4:/home/ap4y/github/swift-mode/.cask/24.5.1/elpa/s-20150924.406:/usr/share/emacs/24.5/lisp:/usr/share/emacs/24.5/lisp/vc:/usr/share/emacs/24.5/lisp/url:/usr/share/emacs/24.5/lisp/textmodes:/usr/share/emacs/24.5/lisp/progmodes:/usr/share/emacs/24.5/lisp/play:/usr/share/emacs/24.5/lisp/org:/usr/share/emacs/24.5/lisp/nxml:/usr/share/emacs/24.5/lisp/net:/usr/share/emacs/24.5/lisp/mh-e:/usr/share/emacs/24.5/lisp/mail:/usr/share/emacs/24.5/lisp/leim:/usr/share/emacs/24.5/lisp/language:/usr/share/emacs/24.5/lisp/international:/usr/share/emacs/24.5/lisp/gnus:/usr/share/emacs/24.5/lisp/eshell:/usr/share/emacs/24.5/lisp/erc:/usr/share/emacs/24.5/lisp/emulation:/usr/share/emacs/24.5/lisp/emacs-parallel:/usr/share/emacs/24.5/lisp/emacs-lisp:/usr/share/emacs/24.5/lisp/cedet:/usr/share/emacs/24.5/lisp/calendar:/usr/share/emacs/24.5/lisp/calc:/usr/share/emacs/24.5/lisp/obsolete
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Swift

Minor modes in effect:
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
File mode specification error: (wrong-type-argument stringp func-args)

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util
help-fns mail-prsvr mail-utils vc-git swift-mode edmacro kmacro easymenu
derived pcase smie cl-loaddefs cl-lib comint ansi-color ring rx
time-date tooltip electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel x-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment lisp-mode prog-mode register page menu-bar
rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax
facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer nadvice loaddefs button
faces cus-face macroexp files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind gfilenotify dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty emacs)

Memory information:
((conses 16 81358 6097)
 (symbols 48 18856 0)
 (miscs 40 38 135)
 (strings 32 13410 4787)
 (string-bytes 1 381215)
 (vectors 16 10445)
 (vector-slots 8 396568 6410)
 (floats 8 66 65)
 (intervals 56 205 0)
 (buffers 960 12)
 (heap 1024 36497 1152))

bug#21875: 24.5; Invalid tokens in smie indentation rules

[Glenn Morris]

Stefan, FYI:


mail@ap4y.me wrote:

> For the simple grammar like this:
>
> (id)
> (inst ("if" exp "begin" insts "end")
>       (exp))
> (insts (insts ";" insts) (inst))
> (exp (exp "." id)
>      (id ":" exp)
>      (exp "=" exp))
> (exps (exps "," exps) (exp))
>
> And lexer rule that overrides braces with begin/end tokens, for example
> like this:
>
> ((looking-at "{") (forward-char 1) "begin")
> ((looking-at "}") (forward-char 1) "end")
>
> When trying to indent such construction:
>
> if true {
>     bar
> |}
>
> I'm recieveing indentation request for :close-all '}', but no such
> token was returned by lexer, full log:
>
> forward: 19 -> 20 = end
> backward: 20 -> 19 = end
> backward: 19 -> 18 = ;
> backward: 18 -> 15 = bar
> backward: 15 -> 9 = begin
> backward: 9 -> 4 = true
> backward: 4 -> 1 = if
> :close-all '}'; sibling-p:t parent:(nil 1 if) hanging:nil == nil
> backward: 20 -> 19 = end
> backward: 19 -> 18 = ;
> backward: 18 -> 15 = bar
> backward: 15 -> 9 = begin
> backward: 9 -> 4 = true
> backward: 4 -> 1 = if
>
>
>
> In GNU Emacs 24.5.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.6)
>  of 2015-09-10 on foutrelis
> Windowing system distributor `The X.Org Foundation', version 11.0.11702000
> System Description:	Arch Linux
>
> Configured using:
>  `configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
>  --localstatedir=/var --with-x-toolkit=gtk3 --with-xft
>  'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong
>  --param=ssp-buffer-size=4' CPPFLAGS=-D_FORTIFY_SOURCE=2
>  LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'
>
> Important settings:
>   value of $EMACSLOADPATH: /home/ap4y/github/swift-mode/.cask/24.5.1/elpa/flycheck-cask-20150920.453:/home/ap4y/github/swift-mode/.cask/24.5.1/elpa/flycheck-20151027.755:/home/ap4y/github/swift-mode/.cask/24.5.1/elpa/dash-20151021.113:/home/ap4y/github/swift-mode/.cask/24.5.1/elpa/pkg-info-20150517.443:/home/ap4y/github/swift-mode/.cask/24.5.1/elpa/epl-20150517.433:/home/ap4y/github/swift-mode/.cask/24.5.1/elpa/let-alist-1.0.4:/home/ap4y/github/swift-mode/.cask/24.5.1/elpa/s-20150924.406:/usr/share/emacs/24.5/lisp:/usr/share/emacs/24.5/lisp/vc:/usr/share/emacs/24.5/lisp/url:/usr/share/emacs/24.5/lisp/textmodes:/usr/share/emacs/24.5/lisp/progmodes:/usr/share/emacs/24.5/lisp/play:/usr/share/emacs/24.5/lisp/org:/usr/share/emacs/24.5/lisp/nxml:/usr/share/emacs/24.5/lisp/net:/usr/share/emacs/24.5/lisp/mh-e:/usr/share/emacs/24.5/lisp/mail:/usr/share/emacs/24.5/lisp/leim:/usr/share/emacs/24.5/lisp/language:/usr/share/emacs/24.5/lisp/international:/usr/share/emacs/24.5/lisp/gnus:/usr/share/emacs/24.5/lisp/eshell:/usr/share/emacs/24.5/lisp/erc:/usr/share/emacs/24.5/lisp/emulation:/usr/share/emacs/24.5/lisp/emacs-parallel:/usr/share/emacs/24.5/lisp/emacs-lisp:/usr/share/emacs/24.5/lisp/cedet:/usr/share/emacs/24.5/lisp/calendar:/usr/share/emacs/24.5/lisp/calc:/usr/share/emacs/24.5/lisp/obsolete
>   value of $LANG: en_US.UTF-8
>   locale-coding-system: utf-8-unix
>
> Major mode: Swift
>
> Minor modes in effect:
>   tooltip-mode: t
>   electric-indent-mode: t
>   mouse-wheel-mode: t
>   tool-bar-mode: t
>   menu-bar-mode: t
>   file-name-shadow-mode: t
>   global-font-lock-mode: t
>   font-lock-mode: t
>   blink-cursor-mode: t
>   auto-composition-mode: t
>   auto-encryption-mode: t
>   auto-compression-mode: t
>   line-number-mode: t
>   transient-mark-mode: t
>
> Recent messages:
> For information about GNU Emacs and the GNU system, type C-h C-a.
> File mode specification error: (wrong-type-argument stringp func-args)
>
> Load-path shadows:
> None found.

bug#21875: 24.5; Invalid tokens in smie indentation rules

[Stefan Monnier]

>> And lexer rule that overrides braces with begin/end tokens, for example
>> like this:
>>
>> ((looking-at "{") (forward-char 1) "begin")
>> ((looking-at "}") (forward-char 1) "end")

This is not enough info: we need the complete code so we can run it on
our side and single-step through it to see what's going on.


        Stefan

bug#21875: 24.5; Invalid tokens in smie indentation rules

[Glenn Morris]

[resending including the OP in the list of recipients]

Stefan Monnier wrote:

>>> And lexer rule that overrides braces with begin/end tokens, for example
>>> like this:
>>>
>>> ((looking-at "{") (forward-char 1) "begin")
>>> ((looking-at "}") (forward-char 1) "end")
>
> This is not enough info: we need the complete code so we can run it on
> our side and single-step through it to see what's going on.

bug#21875: 24.5; Invalid tokens in smie indentation rules

[Arthur Evstifeev]

[-- Attachment #1: Type: text/plain, Size: 232 bytes --]


Sorry for the delay. I have attached full code of the swift-mode with
minimal smie implementation. I didn't clean other pieces of the mode
like syntax table and highlighting code, hope this won't be too
inconvenient to work with.


[-- Attachment #2: swift-mode.el --]
[-- Type: application/emacs-lisp, Size: 22676 bytes --]

[-- Attachment #3: Type: text/plain, Size: 464 bytes --]


Glenn Morris writes:

> [resending including the OP in the list of recipients]
>
> Stefan Monnier wrote:
>
>>>> And lexer rule that overrides braces with begin/end tokens, for example
>>>> like this:
>>>>
>>>> ((looking-at "{") (forward-char 1) "begin")
>>>> ((looking-at "}") (forward-char 1) "end")
>>
>> This is not enough info: we need the complete code so we can run it on
>> our side and single-step through it to see what's going on.

--
Sent with my mu4e