bug#5757: 23.1; String literal parse problem in ruby-mode

bug#5757: 23.1; String literal parse problem in ruby-mode

[Rhett Sutphin]

ruby-mode will misunderstand a ruby double-quoted string literal which
contains a single quote and ends with a question mark.  It thinks that
the string literal is unterminated, which contaminates the syntax
highlighting for the remainder of the buffer.

Example ruby code which will demonstrate the problem:

["Is 'this' a string?"], [:something, :else]

If there's anything between the question mark and the terminating
double-quote, the string will be correctly interpreted.

Thanks,
Rhett Sutphin
rhett@detailedbalance.net


In GNU Emacs 23.1.1 (i386-apple-darwin9.8.0, NS apple-appkit-949.54)
  of 2009-08-16 on black.local
Windowing system distributor `Apple', version 10.3.949
configured using `configure  '--with-ns''

Important settings:
   value of $LC_ALL: nil
   value of $LC_COLLATE: nil
   value of $LC_CTYPE: nil
   value of $LC_MESSAGES: nil
   value of $LC_MONETARY: nil
   value of $LC_NUMERIC: nil
   value of $LC_TIME: nil
   value of $LANG: nil
   value of $XMODIFIERS: nil
   locale-coding-system: nil
   default-enable-multibyte-characters: t

Major mode: Ruby

Minor modes in effect:
   yas/global-mode: t
   yas/minor-mode: t
   auto-fill-function: do-auto-fill
   global-auto-revert-mode: t
   hi-lock-mode: t
   shell-dirtrack-mode: t
   hl-line-mode: t
   global-linum-mode: t
   linum-mode: t
   show-paren-mode: t
   nxhtml-global-minor-mode: t
   recentf-mode: t
   mouse-wheel-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   global-auto-composition-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   column-number-mode: t
   line-number-mode: t
   transient-mark-mode: t

Recent input:
<right> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <left> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <right> <right> <right> <right> <right>
<right> <right> <right> <right> <right> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up>
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <left>
<left> <right> <right> ' <right> <left> " <left> <left>
<left> <left> <right> <left> <right> <right> <left>
<left> <right> <right> <right> <left> <left> <left>
<left> <right> <right> <right> <right> <right> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <down>
<down> <down> <down> <down> <down> <down> <down> <right>
<right> <right> <right> <right> <right> <left> <right>
<right> <right> <right> <right> <right> <right> <right>
<right> <right> <right> <right> <left> <left> <left>
<left> <left> <left> <left> <left> <left> <left> <left>
<left> <backspace> <left> <left> <backspace> ' <right>
<right> <right> <right> <right> <right> <right> <right>
<right> <right> <right> <right> <right> <right> , SPC
[ : a ] <left> <left> <left> <left> <left> <left> <left>
<left> <left> <left> <left> <left> <left> <left> <left>
<left> <left> <left> <left> <left> <backspace> <right>
<right> <right> <right> <right> ' <right> <right> <right>
<right> <right> <right> ' <backspace> <left> <left>
<left> <left> <left> <left> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> ' '
' ' ' ' ' ' ' ' ' ' ' <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> " <right> <right> <right> <right> <right>
<right> ? <backspace> <left> <left> <left> ? <down-mouse-1>
<mouse-1> M-x r e p o r t - e m <tab> <return>

Recent messages:
Auto-saving...done
Quit
Mark set
Undo! [6 times]
Quit
Auto-saving...done
Making completion list...
Quit
ruby-indent-to: invalid nest [2 times]
call-interactively: End of buffer

bug#5757: The line which causes the bug is 1185

[Pål de Vibe]

[-- Attachment #1: Type: text/plain, Size: 275 bytes --]

Line 1185:
          ("\\(^\\|[^\\\\]\\)\\(\\\\\\\\\\)*[?$]\\([#\"'`]\\)" 3 (1 . nil))

A workaround (with, to me, uknown consequences) is to remove the question mark 
from the line, like this:

          ("\\(^\\|[^\\\\]\\)\\(\\\\\\\\\\)*[$]\\([#\"'`]\\)" 3 (1 . nil))

[-- Attachment #2: Type: text/html, Size: 725 bytes --]

bug#5757: String literal parse problem in ruby-mode

[Chong Yidong]

Hi Nakada-san,

Could you help review Pål de Vibe's proposed fix to the following
problem in Emacs ruby-mode?  Thanks.


Pål de Vibe <pauldevibe@yahoo.no> writes:

>> ruby-mode will misunderstand a ruby double-quoted string literal which
>> contains a single quote and ends with a question mark.  It thinks that
>> the string literal is unterminated, which contaminates the syntax
>> highlighting for the remainder of the buffer.
>>
>> Example ruby code which will demonstrate the problem:
>>
>> ["Is 'this' a string?"], [:something, :else]
>>
>> If there's anything between the question mark and the terminating
>> double-quote, the string will be correctly interpreted.
>
> Line 1185:
> ("\\(^\\|[^\\\\]\\)\\(\\\\\\\\\\)*[?$]\\([#\"'`]\\)" 3 (1 . nil))
>
> A workaround (with, to me, uknown consequences) is to remove the
> question mark from the line, like this:
>
> ("\\(^\\|[^\\\\]\\)\\(\\\\\\\\\\)*[$]\\([#\"'`]\\)" 3 (1 . nil))

bug#5757: String literal parse problem in ruby-mode

[Stefan Monnier]

> Could you help review Pål de Vibe's proposed fix to the following
> problem in Emacs ruby-mode?  Thanks.

His proposed fix is not correct: in ruby (like in Elisp) ?<char> is used
for chars (including ?' and ?") and $' and $" are also special vars, so
his fix just disables the special treatment of ?.
For Emacs-24, we use a different chunk of code which doesn't suffer from
this problem (mostly calling syntax-ppss to determine if we're inside
a string).


        Stefan


> Pål de Vibe <pauldevibe@yahoo.no> writes:

>>> ruby-mode will misunderstand a ruby double-quoted string literal which
>>> contains a single quote and ends with a question mark.  It thinks that
>>> the string literal is unterminated, which contaminates the syntax
>>> highlighting for the remainder of the buffer.
>>> 
>>> Example ruby code which will demonstrate the problem:
>>> 
>>> ["Is 'this' a string?"], [:something, :else]
>>> 
>>> If there's anything between the question mark and the terminating
>>> double-quote, the string will be correctly interpreted.
>> 
>> Line 1185:
>> ("\\(^\\|[^\\\\]\\)\\(\\\\\\\\\\)*[?$]\\([#\"'`]\\)" 3 (1 . nil))
>> 
>> A workaround (with, to me, uknown consequences) is to remove the
>> question mark from the line, like this:
>> 
>> ("\\(^\\|[^\\\\]\\)\\(\\\\\\\\\\)*[$]\\([#\"'`]\\)" 3 (1 . nil))

bug#5757: String literal parse problem in ruby-mode

[Pål de Vibe]

[-- Attachment #1: Type: text/plain, Size: 1651 bytes --]

Hi. This wasn't really a proposed fix but more a hint to help debugging. I
haven't had time to really study the code. Thanks for the feedback.

On Mon, Apr 4, 2011 at 9:23 AM, Stefan Monnier <monnier@iro.umontreal.ca>wrote:

> > Could you help review Pål de Vibe's proposed fix to the following
> > problem in Emacs ruby-mode?  Thanks.
>
> His proposed fix is not correct: in ruby (like in Elisp) ?<char> is used
> for chars (including ?' and ?") and $' and $" are also special vars, so
> his fix just disables the special treatment of ?.
> For Emacs-24, we use a different chunk of code which doesn't suffer from
> this problem (mostly calling syntax-ppss to determine if we're inside
> a string).
>
>
>        Stefan
>
>
> > Pål de Vibe <pauldevibe@yahoo.no> writes:
>
> >>> ruby-mode will misunderstand a ruby double-quoted string literal which
> >>> contains a single quote and ends with a question mark.  It thinks that
> >>> the string literal is unterminated, which contaminates the syntax
> >>> highlighting for the remainder of the buffer.
> >>>
> >>> Example ruby code which will demonstrate the problem:
> >>>
> >>> ["Is 'this' a string?"], [:something, :else]
> >>>
> >>> If there's anything between the question mark and the terminating
> >>> double-quote, the string will be correctly interpreted.
> >>
> >> Line 1185:
> >> ("\\(^\\|[^\\\\]\\)\\(\\\\\\\\\\)*[?$]\\([#\"'`]\\)" 3 (1 . nil))
> >>
> >> A workaround (with, to me, uknown consequences) is to remove the
> >> question mark from the line, like this:
> >>
> >> ("\\(^\\|[^\\\\]\\)\\(\\\\\\\\\\)*[$]\\([#\"'`]\\)" 3 (1 . nil))
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 2370 bytes --]

bug#5757: String literal parse problem in ruby-mode

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> His proposed fix is not correct: in ruby (like in Elisp) ?<char> is used
> for chars (including ?' and ?") and $' and $" are also special vars, so
> his fix just disables the special treatment of ?.
> For Emacs-24, we use a different chunk of code which doesn't suffer from
> this problem (mostly calling syntax-ppss to determine if we're inside
> a string).

Is this worth backporting to Emacs 23?  If not, let's close this bug.

bug#5757: String literal parse problem in ruby-mode

[Stefan Monnier]

>> His proposed fix is not correct: in ruby (like in Elisp) ?<char> is used
>> for chars (including ?' and ?") and $' and $" are also special vars, so
>> his fix just disables the special treatment of ?.
>> For Emacs-24, we use a different chunk of code which doesn't suffer from
>> this problem (mostly calling syntax-ppss to determine if we're inside
>> a string).
> Is this worth backporting to Emacs 23?  If not, let's close this bug.

I think it's more important to keep in touch with the upstream
maintainers.  E.g. I still haven't heard anything from them regarding
the changes I've made for syntax-propertize (which happen to fix the
OP's problem).  I'm worried we might not stay in sync.


        Stefan