bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly

[Stefan Monnier <monnier@iro.umontreal.ca>]

>> My preference would be to think about it as a "multi-mode" case, and
>> hence make it possible to specify a different syntax-table to use within
>> the regexp.
> I remember this idea, but have a hard time viewing it in the context of our
> latest discussion on the subject of multi-modes.

> First, why only syntax-table?

It was not meant as excluding other data.

> For this specific case, a syntax table change is not required, we only
> need to be able to view the text between /'s as a separate context

Not necessarily required in all cases, but in order to handle the
various existing cases in various languages, we need some way to
indicate, for instance:
- Do double quotes introduce strings when used within this new context?
- Do comment chars introduce comments when used within this new context?
- Does \ still escape chars in this new context?
We usually use syntax-tables for that.  They do provide more flexibility
than needed (so far), such as making it possible to allow different
commenting conventions within the new context.  But it seems like
a "simple" way to handle the problem, so the extra generality is a bonus.

> (but - and this is a change from certain other multi-mode uses - still
> fontify uncommented text inside them with the regexp face).  But in the
> general case, we would at least want to be able to change
> font-lock-keywords, too.

`font-lock-keywords' can use `syntax-ppss' to decide which rules to use
(since syntax-ppss would have to include the "context state" somewhere
in its output).  It's not terribly convenient to do currently, so we may
want to provide a new replacement for font-lock-keywords which makes it
easier (and avoids relying on "eval" while we're at it).

>> I think of it along the lines of a new syntax-class, applied to the "/"
>> char, which would change the syntax-table for the subsequent text.
> How would this interact with a new hook that would `syntax-ppss' would run
> on the cached entries?

The way I imagine it, it would work at the parse-partial-sexp level
(extending the returned PPSS with a new field indicating the current
syntax-table or something similar).  So syntax.el would stay
basically unchanged.

How that would interact with a new hook?  Haven't thought about it yet.

> Would its default value look for the chars bearing the new syntax class?

I don't really understand the question.  But I'd guess that I don't know
the answer.


        Stefan