bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly

bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly

[Bozhidar Batsov]

In most editors/IDEs code like this

regexp = /
  start         # some text
  \s            # white space char
  (group)       # first group
  (?:alt1|alt2) # some alternation
  end
/x

will have the comments font-locked as comments, because comments are
allowed in the extended regexp literal syntax (/x). It would be nice
if this was taken into account in ruby-mode as well.



In GNU Emacs 24.3.50.1 (x86_64-apple-darwin13.0.0, NS apple-appkit-1265.00)
 of 2014-01-27 on Bozhidars-MacBook-Pro.local
Windowing system distributor `Apple', version 10.3.1265
Configured using:
 `configure --prefix=/usr/local/Cellar/emacs/HEAD --without-dbus
 --enable-locallisppath=/usr/local/share/emacs/site-lisp
 --infodir=/usr/local/Cellar/emacs/HEAD/share/info/emacs
 --without-gnutls --with-ns --disable-ns-self-contained'

Important settings:
  locale-coding-system: utf-8-unix

Major mode: Ruby

Minor modes in effect:
  ruby-tools-mode: t
  inf-ruby-minor-mode: t
  diff-auto-refine-mode: t
  subword-mode: t
  guru-mode: t
  erc-truncate-mode: t
  erc-spelling-mode: t
  erc-list-mode: t
  erc-menu-mode: t
  erc-autojoin-mode: t
  erc-ring-mode: t
  erc-networks-mode: t
  erc-pcomplete-mode: t
  erc-track-mode: t
  erc-match-mode: t
  erc-button-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-netsplit-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  global-flycheck-mode: t
  flycheck-mode: t
  which-function-mode: t
  flx-ido-mode: t
  ido-ubiquitous-mode: t
  winner-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  whitespace-mode: t
  global-anzu-mode: t
  anzu-mode: t
  projectile-global-mode: t
  projectile-mode: t
  flyspell-mode: t
  volatile-highlights-mode: t
  global-hl-line-mode: t
  shell-dirtrack-mode: t
  recentf-mode: t
  savehist-mode: t
  show-smartparens-global-mode: t
  show-smartparens-mode: t
  smartparens-mode: t
  global-auto-revert-mode: t
  delete-selection-mode: t
  prelude-global-mode: t
  prelude-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
' d SPC s u g g e s t SPC t h e SPC u s e SPC o f SPC 
t h e SPC ` <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> <backspace> f SPC a SPC d i 
f f e r e n t SPC f a c e SPC t h a n SPC t h e SPC 
s t r i n g SPC f a c e SPC u s e d SPC o f <backspace> 
<backspace> f o r SPC t h e SPC l i t e r a i l <backspace> 
<backspace> l SPC i t s e l f , SPC b u t SPC e v e 
n SPC i t SPC w o u l d SPC b e SPC a n SPC i m p r 
o v e m e n t . SPC I SPC a s o <backspace> <backspace> 
l s o SPC t h i n k <return> t h a t SPC o n l y SPC 
v a l i d SPC m o d i f i e r s SPC s h o u l d SPC 
b e SPC c <backspace> f o n t SPC l o c k e d . SPC 
M-b M-b M-f C-d - C-x b <return> <return> % r / t e 
s t / i <help-echo> C-x b <return> C-p C-p C-p C-p 
C-p C-p C-p C-n C-p C-p C-a C-f C-d n C-n C-n C-e <return> 
$ <backspace> % r { t e s t } i C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-p C-p C-p C-p C-p C-p C-p C-n 
C-n C-n C-n C-e C-c C-c y M-x r e p o r t <return>

Recent messages:
Wrote /Users/bozhidar/*message*-20140301-152412
Saving file /Users/bozhidar/projects/test.rb...
Wrote /Users/bozhidar/projects/test.rb
Auto-saving...done
Mark set
Send this bug report to the Emacs maintainers? (y or n)  y
Sending...
Mark set [2 times]
Sending via mail...
Sending...done

Load-path shadows:
/Users/bozhidar/.emacs.d/elpa/tabulated-list-20120406.2251/tabulated-list hides /usr/local/Cellar/emacs/HEAD/share/emacs/24.3.50/lisp/emacs-lisp/tabulated-list

Features:
(shadow sort mail-extr emacsbug vc-annotate vc vc-dispatcher nxml-uchnm
rng-xsd xsd-regexp rng-cmpct edebug crm ielm hippie-exp markdown-mode
noutline outline gitignore-mode conf-mode yaml-mode ace-jump-mode ffap
url-parse url-vars ruby-tools inf-ruby ruby-mode-expansions
smartparens-ruby ruby-mode smie misearch multi-isearch jka-compr
eieio-opt speedbar sb-image ezimage dframe mule-util find-dired
magit-key-mode magit view epa derived epg epg-config diff-mode
git-rebase-mode git-commit-mode server log-edit message sendmail rfc822
mml mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047
rfc2045 ietf-drums mailabbrev mail-utils gmm-utils mailheader pcvs-util
add-log executable vc-git lisp-mnt network-stream starttls tls cider
cider-mode cider-repl cider-eldoc clojure-test-mode cider-interaction
arc-mode archive-mode cider-client nrepl-client cider-util ewoc
superword subword clojure-mode-expansions clojure-mode inf-lisp
rainbow-mode color rainbow-delimiters elisp-slime-nav guru-mode
prelude-key-chord key-chord prelude-xml nxml-mode-expansions
html-mode-expansions sgml-mode smartparens-html rng-nxml rng-valid
rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt rng-util rng-pttrn
nxml-ns nxml-mode nxml-outln nxml-rap nxml-util nxml-glyph nxml-enc
xmltok prelude-web prelude-scss prelude-scheme prelude-ruby prelude-perl
prelude-org prelude-js prelude-erc erc-truncate erc-autoaway
erc-spelling erc-notify erc-log erc-list erc-menu erc-join erc-ring
erc-networks erc-pcomplete erc-track erc-match erc-button erc-fill
erc-stamp erc-netsplit erc-goodies erc erc-backend erc-compat
prelude-emacs-lisp prelude-css prelude-common-lisp slime-autoloads
prelude-clojure prelude-lisp prelude-c prelude-programming flycheck
help-mode rx f which-func imenu prelude-ido smex flx-ido flx
ido-ubiquitous warnings ido prelude-osx exec-path-from-shell
prelude-global-keybindings prelude-editor winner undo-tree diff esh-var
esh-io esh-cmd esh-opt esh-ext esh-proc esh-arg eldoc esh-groups eshell
esh-module esh-mode esh-util re-builder whitespace browse-kill-ring
midnight ediff-merg ediff-wind ediff-diff ediff-mult ediff-help
ediff-init ediff-util ediff dired-x dired anzu projectile pkg-info
find-func grep compile s bookmark pp expand-region text-mode-expansions
er-basic-expansions expand-region-core expand-region-custom flyspell
ispell etags volatile-highlights hl-line windmove tramp-cache tramp-sh
tramp tramp-compat auth-source gnus-util mm-util mail-prsvr
password-cache tramp-loaddefs trampver shell pcomplete comint ansi-color
ring format-spec recentf tree-widget wid-edit savehist saveplace
diminish smartparens-config smartparens autorevert filenotify delsel
prelude-mode easy-mmode edmacro kmacro prelude-core epl advice help-fns
dash thingatpt prelude-ui zenburn-theme prelude-packages finder-inf
ack-and-a-half-autoloads diminish-autoloads erlang-autoloads
flx-ido-autoloads eieio byte-opt bytecomp byte-compile cconv eieio-core
grizzl-autoloads key-chord-autoloads logito-autoloads info easymenu
cl-macs gv move-text-autoloads pkg-info-autoloads puppet-mode-autoloads
rainbow-delimiters-autoloads request-autoloads rubocop-autoloads
volatile-highlights-autoloads yaml-mode-autoloads package cl cl-loaddefs
cl-lib time-date tooltip electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel ns-win tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process cocoa ns
multi-tty emacs)

bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly

[Dmitry Gutov]

Bozhidar Batsov <bozhidar@batsov.com> writes:

> In most editors/IDEs code like this
>
> regexp = /
>   start         # some text
>   \s            # white space char
>   (group)       # first group
>   (?:alt1|alt2) # some alternation
>   end
> /x
>
> will have the comments font-locked as comments, because comments are
> allowed in the extended regexp literal syntax (/x). It would be nice
> if this was taken into account in ruby-mode as well.

Not sure how to implement it best.

Ideally, we'd have a new kind of syntax instead of strings (native
regexp support?), which would make font-lock fontify comments inside.

Or maybe a modifier on the string syntax?

Of course, we could just scan the contents of every regexp, look for any
hash character that doesn't look like it starts interpolation, and
forcibly fontify the text from it till the end of line.

bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 1308 bytes --]

On Sunday, March 2, 2014 at 12:19 AM, Dmitry Gutov wrote:
> Bozhidar Batsov <bozhidar@batsov.com (mailto:bozhidar@batsov.com)> writes:
>  
> > In most editors/IDEs code like this
> >  
> > regexp = /
> > start # some text
> > \s # white space char
> > (group) # first group
> > (?:alt1|alt2) # some alternation
> > end
> > /x
> >  
> > will have the comments font-locked as comments, because comments are
> > allowed in the extended regexp literal syntax (/x). It would be nice
> > if this was taken into account in ruby-mode as well.
> >  
>  
>  
> Not sure how to implement it best.
>  
> Ideally, we'd have a new kind of syntax instead of strings (native
> regexp support?), which would make font-lock fontify comments inside.
>  
> Or maybe a modifier on the string syntax?
Native regexp support is preferable IMO. After all - regexps are not strings. If they were treated differently we’d also be able to have extra highlighting for things like named groups, quantifiers, regexp classes, etc. I guess, however, that this would require a lot of work.
>  
> Of course, we could just scan the contents of every regexp, look for any
> hash character that doesn't look like it starts interpolation, and
> forcibly fontify the text from it till the end of line.
>  
>  



[-- Attachment #2: Type: text/html, Size: 2196 bytes --]

bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly

[Dmitry Gutov]

On 02.03.2014 13:03, Bozhidar Batsov wrote:
> Native regexp support is preferable IMO. After all - regexps are not
> strings. If they were treated differently we’d also be able to have
> extra highlighting for things like named groups, quantifiers, regexp
> classes, etc. I guess, however, that this would require a lot of work.

Since the regexp syntax can be different between languages, we probably 
won't get all of that automatically. The direct benefits from the native 
support I can see is highlighting comments (for regexps with appropriate 
modifiers, in languages that support that), new faces for regexp itself, 
and maybe for groups, quantifiers, etc.

The highlighting of the elements inside regexp would probably have to be 
done the old-fashioned way, in font-lock-keywords (although that code 
could be shared between many languages). We could implement something 
like it right now, the main difference is just the lack of standard faces.

bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 1484 bytes --]

On Sunday, March 2, 2014 at 5:03 PM, Dmitry Gutov wrote:
> On 02.03.2014 13:03, Bozhidar Batsov wrote:
> > Native regexp support is preferable IMO. After all - regexps are not
> > strings. If they were treated differently we’d also be able to have
> > extra highlighting for things like named groups, quantifiers, regexp
> > classes, etc. I guess, however, that this would require a lot of work.
> >  
>  
>  
> Since the regexp syntax can be different between languages, we probably  
> won't get all of that automatically. The direct benefits from the native  
> support I can see is highlighting comments (for regexps with appropriate  
> modifiers, in languages that support that), new faces for regexp itself,  
> and maybe for groups, quantifiers, etc.
>  
>  

That would be totally sufficient, but I have no idea how much work it would require.  
>  
> The highlighting of the elements inside regexp would probably have to be  
> done the old-fashioned way, in font-lock-keywords (although that code  
> could be shared between many languages). We could implement something  
> like it right now, the main difference is just the lack of standard faces.
>  
>  

Many modes introduce "non-standard” faces, so the lack of non-standard faces is not a significant problem I think (though I’d love to see faces for regexps, command execution (%x, `` as in sh-mode) and symbols/keywords (as they are present in many languages outside Ruby).  
  


[-- Attachment #2: Type: text/html, Size: 2202 bytes --]

bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly

[Stefan Monnier]

> Ideally, we'd have a new kind of syntax instead of strings (native
> regexp support?), which would make font-lock fontify comments inside.

This is not completely new.  We have a similar problem for example with
TeX's math $...$ and with PostScript's strings (...(...)..).

> Or maybe a modifier on the string syntax?

My preference would be to think about it as a "multi-mode" case, and
hence make it possible to specify a different syntax-table to use within
the regexp.

I think of it along the lines of a new syntax-class, applied to the "/"
char, which would change the syntax-table for the subsequent text.


        Stefan

bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly

[Dmitry Gutov]

On 07.03.2014 23:04, Stefan Monnier wrote:
> My preference would be to think about it as a "multi-mode" case, and
> hence make it possible to specify a different syntax-table to use within
> the regexp.

I remember this idea, but have a hard time viewing it in the context of 
our latest discussion on the subject of multi-modes.

First, why only syntax-table? For this specific case, a syntax table 
change is not required, we only need to be able to view the text between 
/'s as a separate context (but - and this is a change from certain other 
multi-mode uses - still fontify uncommented text inside them with the 
regexp face). But in the general case, we would at least want to be able 
to change font-lock-keywords, too.

> I think of it along the lines of a new syntax-class, applied to the "/"
> char, which would change the syntax-table for the subsequent text.

How would this interact with a new hook that would `syntax-ppss' would 
run on the cached entries?

Would its default value look for the chars bearing the new syntax class?

bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly

[Stefan Monnier]

>> My preference would be to think about it as a "multi-mode" case, and
>> hence make it possible to specify a different syntax-table to use within
>> the regexp.
> I remember this idea, but have a hard time viewing it in the context of our
> latest discussion on the subject of multi-modes.

> First, why only syntax-table?

It was not meant as excluding other data.

> For this specific case, a syntax table change is not required, we only
> need to be able to view the text between /'s as a separate context

Not necessarily required in all cases, but in order to handle the
various existing cases in various languages, we need some way to
indicate, for instance:
- Do double quotes introduce strings when used within this new context?
- Do comment chars introduce comments when used within this new context?
- Does \ still escape chars in this new context?
We usually use syntax-tables for that.  They do provide more flexibility
than needed (so far), such as making it possible to allow different
commenting conventions within the new context.  But it seems like
a "simple" way to handle the problem, so the extra generality is a bonus.

> (but - and this is a change from certain other multi-mode uses - still
> fontify uncommented text inside them with the regexp face).  But in the
> general case, we would at least want to be able to change
> font-lock-keywords, too.

`font-lock-keywords' can use `syntax-ppss' to decide which rules to use
(since syntax-ppss would have to include the "context state" somewhere
in its output).  It's not terribly convenient to do currently, so we may
want to provide a new replacement for font-lock-keywords which makes it
easier (and avoids relying on "eval" while we're at it).

>> I think of it along the lines of a new syntax-class, applied to the "/"
>> char, which would change the syntax-table for the subsequent text.
> How would this interact with a new hook that would `syntax-ppss' would run
> on the cached entries?

The way I imagine it, it would work at the parse-partial-sexp level
(extending the returned PPSS with a new field indicating the current
syntax-table or something similar).  So syntax.el would stay
basically unchanged.

How that would interact with a new hook?  Haven't thought about it yet.

> Would its default value look for the chars bearing the new syntax class?

I don't really understand the question.  But I'd guess that I don't know
the answer.


        Stefan

bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly

[Dmitry Gutov]

On 10.03.2014 16:40, Stefan Monnier wrote:
> We usually use syntax-tables for that.  They do provide more flexibility
> than needed (so far), such as making it possible to allow different
> commenting conventions within the new context.  But it seems like
> a "simple" way to handle the problem, so the extra generality is a bonus.

To handle this part of the problem (literals within literals), yes, 
certainly.

>> (but - and this is a change from certain other multi-mode uses - still
>> fontify uncommented text inside them with the regexp face).  But in the
>> general case, we would at least want to be able to change
>> font-lock-keywords, too.
>
> `font-lock-keywords' can use `syntax-ppss' to decide which rules to use
> (since syntax-ppss would have to include the "context state" somewhere
> in its output).  It's not terribly convenient to do currently, so we may
> want to provide a new replacement for font-lock-keywords which makes it
> easier (and avoids relying on "eval" while we're at it).

If we're speaking of multiple modes in general, syntax-ppss might not be 
enough to determine which keywords to highlight. Even if it includes 
information about new syntax classes.

Suppose we have an HTML file with <script> and <style> tags inside. 
Would points inside of these tags have different states? What might be 
the difference between them?

Would some font-lock code compare syntax table reference attached to a 
syntax class, with a syntax table it wants?

bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly

[Stefan Monnier]

> If we're speaking of multiple modes in general, syntax-ppss might not be
> enough to determine which keywords to highlight.  Even if it includes
> information about new syntax classes.

It wouldn't really include info *about* new syntax classes, but the new
syntax class I suggest (which specifies a syntax-table to use) would
require changing the "parse-partial-sexp state" by keeping track of the
"current syntax-table".

"Syntax classes" are "events" which cause changes in the
parse-partial-sexp state machine, and syntax-ppss is a way to get the
state of that machine at a given position.

> Suppose we have an HTML file with <script> and <style> tags inside. Would
> points inside of these tags have different states?

Yes.  E.g. (nth 10 (syntax-ppss)) would return a new "context" info
(which could just be "the syntax table to use in that area").

> Would some font-lock code compare syntax table reference attached
> to a syntax class, with a syntax table it wants?

Yes, quite possibly.


        Stefan