Regarding Emacs, js.el, template-strings and syntax-tables

Regarding Emacs, js.el, template-strings and syntax-tables

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 2863 bytes --]

Hey everyone.

First off, while the subject of the examples employed in this email are
all related to ECMAScript, the kind of syntax-statements discussed is in
no way new or unique to ECMAScript. Off hand I know at least TypeScript
and C# supports this too, not to mention many web-based templating
languages, like Jekyll.
But everyone seems to love JavaScript, so lets use that as an example.
Newer version of ECMAScript supports in-string templating. That is, you
can escape the string-sequence and insert computational expressions
inside the string-literal. This allows a cleaner syntax than composing
the string using concatenation.
As an example, instead of:

> function getGreeting(name) {
>     let greeting = "Hello, " + name + "!";
>     return greeting;
> }

you can write:

> function getGreeting(name) {
>     let greeting = `Hello, ${name}!`;
>     return greeting;
> }

For more details on this feature, MDN has a good write-up[1]. 

Now for the problem:

From what I can see js.el does not support this new syntax yet, so that
everything inside the string is treated (and highlighted) semantically
as a string. Another area where similar problems exist, but with
comments (as opposed to string) is JSDoc[2].
In combination with other minor-modes like flyspell-prog-mode which spell-
checks strings and comments, this can quickly get annoying. It also
means syntax-based analysis for things like auto-completion will simply
not trigger.
This is a problem many major-modes needs to handle (as such I'm here
speaking as a package maintainer for MELPA package typescript.el). It
would be nice if Emacs at its core had mechanisms which made supporting
this kind of syntax easier.
Fundamentally, we would need a way to escape an on-going string or
comment, and end that escape-sequence until either we have a new escape-
sequence or the original string/comment delimiter to finally terminate
the string/comment.
To the best of my knowledge, and please correct me if I'm wrong, this
all involves fundamental syntax, and it would be best to handle this
with syntax-tables if possible. But last time I read the
documentation on syntax-tables, there was no way for to escape
ongoing strings or comments using syntax-tables (and again, please
correct me if I'm wrong!)
I'm sure if Emacs supported this at its core, it would be much easier in
major-mode implementations to support these kind of syntaxes.
Is any work being done to extend syntax-tables to support this?

Is any work being done to support template-strings in js.el?

--
Cheers
Jostein Kjønigsen

jostein@kjonigsen.net 🍵 jostein@gmail.com
https://jostein.kjonigsen.net



Links:

  1. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals
  2. http://usejsdoc.org/about-getting-started.html

[-- Attachment #2: Type: text/html, Size: 4264 bytes --]

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Stefan Monnier]

>> function getGreeting(name) {
>> let greeting = `Hello, ${name}!`;
>> return greeting;
>> }

Indeed this problem has been with us in sh-script.el for many years.
There are several different aspects at play:
- how should font-lock display `name`.  Should it have a normal face as
  if it weren't inside a string?  Or should have a kind of combination
  of string-face and something else?
- how should we handle nestings of such constructs.
- several parts of Emacs like to know if we're inside
  code/string/comment (usually using (nth [3|4|8] (syntax-ppss))).
  Should these all consider "name" as being inside code?
  Or inside string?  Or should we revisit all those cases one-by-one?

In sh-script, we use syntax-propertize to make sure the whole string and
its contents is all considered as a string and nothing else.  This was
the simplest solution and probably corresponds to the current behavior
of js.el (and typescript.el) as well.


        Stefan

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 3358 bytes --]

On Thu, Aug 24, 2017, at 12:53 PM, Stefan Monnier wrote:
> Indeed this problem has been with us in sh-script.el for many years.
> There are several different aspects at play:
> - how should font-lock display `name`.  Should it have a normal
>   face as>   if it weren't inside a string?  Or should have a kind of combination>   of string-face and something else?

I'd vote for "plain" face, and name being styled as it would have been
outside the string-context. If a mode-author wants to fancy it up, we
should definitely have a way to easily accommodate that though.
Having a separate expression-in-string-or-comment face is probably the
best approach in that regard.
> - how should we handle nestings of such constructs.

That's a good question, but I think a even better question would be to
ask what that would look like, and *if* we should strive to support it
in the first place.
The most contrived (again JS) example I can think of would be inline code-
references in a comment in a expression in a string. Something like:
    let greeting = "Hello, ${ /** @name string here refers to  the
    employer, not the employee! */ employee_name }";
If we were to support these kind of endless constructs (which I'm sure
someone will find a way to make), we would need to consider this a level
of nesting, just like we do with parens and curly braces and what not.
> - several parts of Emacs like to know if we're inside
>   code/string/comment (usually using (nth [3|4|8] (syntax-ppss))).
>   Should these all consider "name" as being inside code?
>   Or inside string?  Or should we revisit all those cases one-by-one?
It's hard to come up with a blanket solution for this, just like that. I
definitely recognize the problem though.
In increasing the expressiveness of the syntax-construction, it's
generally hard to say "all new occurrences of this new construct map
perfectly to this old understanding of the same property" (as in the
Liskov Substitution Principle).
If a change means revisiting cases one by one, the Emacs ecosystem
will be in a state of mess for years to come, so that's clearly not
an option.
How about instead extending (syntax-ppss) to keep all existing
behaviour for all existing properties, but include some new ones
with more information? Then mode-authors which wants to exploit
these new properties can probe for those in a reasonable backwards-
compatible manner?
Ideally these extra properties should contain at least the following:

* is in escape string or comment
* start of this-level string/comment escape-sequence (nth 8 contains the
  start of the string/comment)* level of string/comment-expression nesting

As far as I can see, that should about cover it.

> In sh-script, we use syntax-propertize to make sure the whole
> string and> its contents is all considered as a string and nothing else.  This was> the simplest solution and probably corresponds to the current behavior> of js.el (and typescript.el) as well.
> 

While obviously the simplest, it's syntactically *inaccurate *and this
inaccuracy is causing issues in other packages who rely on accurate syntax-
information.
It would be really nice if we could have our cake and eat it too.

--
Jostein Kjønigsen

jostein@kjonigsen.net 🍵 jostein@gmail.com
https://jostein.kjonigsen.net




[-- Attachment #2: Type: text/html, Size: 4989 bytes --]

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]


> In sh-script, we use syntax-propertize to make sure the whole
> string and> its contents is all considered as a string and nothing else.  This was> the simplest solution and probably corresponds to the current behavior> of js.el (and typescript.el) as well.

I'd also like to add that I've been told syntax-propertize is a property
of font-lock and relying on it for correct operation on syntax-tables is
... not entirely guaranteed.
Basically, if a user disables font-lock for whatever reason (maybe
scripted automation?), you can no longer rely on the syntax-properties
to be correct.
It might be a bigger problem in theory than in practice, but again
it represents yet another *inaccuracy* in the syntax-model and
analysis of it.
It would be nice to get this stuff cleaned up.

--
Jostein Kjønigsen




[-- Attachment #2: Type: text/html, Size: 1408 bytes --]

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Stefan Monnier]

> I'd also like to add that I've been told syntax-propertize is a property
> of font-lock

No, it's not.  font-lock-syntactic-keywords was.

> and relying on it for correct operation on syntax-tables is
> ... not entirely guaranteed.

Indeed, that's why I introduced syntax-propertize (and recently extended
it so that it's applied on-the-fly during operations such as
forward-sexp).


        Stefan

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Stefan Monnier]

> I'd vote for "plain" face, and name being styled as it would have been
> outside the string-context. If a mode-author wants to fancy it up, we
> should definitely have a way to easily accommodate that though.
> Having a separate expression-in-string-or-comment face is probably the
> best approach in that regard.

If we want to handle nesting, the easiest solution is either "color
everything like a string" or "color `name` as if it weren't in a string".

Anything in between will tend ask "how do I distinguish
a string-within-a-string from a string-not-within-a-string, and what
about a string-within-a-string-within-a-string?".

So I tend to agree with you here.  But Anders Lindgren seems to disagree.

>> - how should we handle nestings of such constructs.
> That's a good question, but I think a even better question would be to
> ask what that would look like, and *if* we should strive to support it
> in the first place.

For sh-script we definitely need to support it, because it's often used
for things equivalent to nesting function calls as in (f1 (f2 z (f3 x) y)).
E.g.:

   x = "blabla $(echo "$foo" | bar $(find . -print))"

> If we were to support these kind of endless constructs (which I'm sure
> someone will find a way to make), we would need to consider this a level
> of nesting, just like we do with parens and curly braces and what not.

Yes, at the level of syntax-tables (and parse-partial-sexp) we'll
definitely want to handle arbitrary nestings.  W.r.t how to display it,
we will probably prefer to disregard the nesting level and only pay
attention to the top-level or the bottom-level (or both).

> How about instead extending (syntax-ppss) to keep all existing
> behaviour for all existing properties, but include some new ones
> with more information?

Yes, of course, that's what we'd do.  I think we'd add yet-another
element which would work a bit like the (nth 9 ppss) but would keep
a stack of states (POS . PPSS) rather than a stack of buffer positions.

> It would be really nice if we could have our cake and eat it too.

I love cake,


        Stefan

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 3718 bytes --]

Hi!

I've been playing around with constructs like that for quite some time,
here are some thoughts on the matter:

--------

If you use a string face with, say, both a background and a foreground
color, it will matter if you *replace* the existing face as opposed to
*prepend* a new face on top of it.

For example, the default Ruby mode replace the face, which doesn't look
good when the string face has a background color:

"Hello #{name}"

With *prepend* it would have looked like:

"Hello #{name}"

In other words, the font-lock rule should use "prepend" and not "t" as the
override flag. (I've got it on my todo-list to change a lot of "t":s to
"prepend" but haven't had time to do it yet.)

--------

One question is if the delimiters around the expression should be
highlighted as well? In Ruby they are highlighted. In my cmake-font-lock[1]
package I have opted not to highlight them, as it makes it easier to read
the variable name:

"Hello ${name}"

Another reason for this is that CMake supports nested such constructs,
which look better this way:

"Hello ${name${tail}}"

[1] https://github.com/Lindydancer/cmake-font-lock

--------

> how should font-lock display `name`.  Should it have a normal face as if
it weren't inside a string?  Or should have a kind of combination of
string-face and something else?

In all cases I've seen, the content is displayed using
`font-lock-variable-name-face', even for the cases where the content is
more complex than a plain variable. I would say that this is OK, as they
stand out and, most of the time, it is a plain variable anyway.

As I said above, using *prepend*, the end result will be a mix of
font-lock-variable-name-face and font-lock-string-face.

--------

> several parts of Emacs like to know if we're inside
>  code/string/comment (usually using (nth [3|4|8] (syntax-ppss))).
>  Should these all consider "name" as being inside code?
>  Or inside string?  Or should we revisit all those cases one-by-one?

I see this as a pure syntax highlighting thing, so I would say that
syntax-ppss should treat them as strings.

--------

A side note: I've started to make an inventory of syntax highlighting in
various modes in Emacs, as part of a font-lock regression test suite I've
accumulated over the years. The suite and the inventory is far from
complete, but it might be worth looking into:


https://github.com/Lindydancer/font-lock-regression-suite/blob/master/doc/CommentsOnMajorModes.org

One mode that stands out in this respect is "bat" mode, it handles "%alpha"
but not "%alpha_beta", and doesn't support the construct in strings at all.

    -- Anders

On Thu, Aug 24, 2017 at 12:53 PM, Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> >> function getGreeting(name) {
> >> let greeting = `Hello, ${name}!`;
> >> return greeting;
> >> }
>
> Indeed this problem has been with us in sh-script.el for many years.
> There are several different aspects at play:
> - how should font-lock display `name`.  Should it have a normal face as
>   if it weren't inside a string?  Or should have a kind of combination
>   of string-face and something else?
> - how should we handle nestings of such constructs.
> - several parts of Emacs like to know if we're inside
>   code/string/comment (usually using (nth [3|4|8] (syntax-ppss))).
>   Should these all consider "name" as being inside code?
>   Or inside string?  Or should we revisit all those cases one-by-one?
>
> In sh-script, we use syntax-propertize to make sure the whole string and
> its contents is all considered as a string and nothing else.  This was
> the simplest solution and probably corresponds to the current behavior
> of js.el (and typescript.el) as well.
>
>
>         Stefan
>
>
>

[-- Attachment #2: Type: text/html, Size: 6287 bytes --]

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Stefan Monnier]

> In all cases I've seen, the content is displayed using
> `font-lock-variable-name-face', even for the cases where the content is
> more complex than a plain variable. I would say that this is OK, as they
> stand out and, most of the time, it is a plain variable anyway.

Interesting.  So the way this should be handled might vary between
languages: in sh such nesting is common, with fairly complex code nested
inside string, whereas from what you say in Ruby this is sufficiently
rare that we may prefer to use another approach (that doesn't work as
well in the general case but is nicer in the simple(&common) case).


        Stefan

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Dmitry Gutov]

On 8/24/17 5:20 PM, Stefan Monnier wrote:
> whereas from what you say in Ruby this is sufficiently
> rare that we may prefer to use another approach (that doesn't work as
> well in the general case but is nicer in the simple(&common) case).

I disagree that Ruby is sufficiently different. Maybe arbitrary nesting 
is not as frequent, but it happens, and some other editors support it 
better than Emacs.

As soon as we fix that, I'd switch to resetting to the default face for 
the nested expression.

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Stefan Monnier]

> A side note: I've started to make an inventory of syntax highlighting in
> various modes in Emacs, as part of a font-lock regression test suite I've
> accumulated over the years. The suite and the inventory is far from
> complete, but it might be worth looking into:


> https://github.com/Lindydancer/font-lock-regression-suite/blob/master/doc/CommentsOnMajorModes.org

> One mode that stands out in this respect is "bat" mode, it handles "%alpha"
> but not "%alpha_beta", and doesn't support the construct in strings at all.

Which version of bat-mode did you test?
Same question for some of the comments in the above web-page.  E.g. you
say "Strings containing “<<” are treated as heredoc comments", but AFAIK
this should only happen in fairly old versions of Emacs.
I also see a reference to "bell_206.ps" but I can't find this file.


        Stefan

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

>
> Which version of bat-mode did you test?
>

The one that comes with Emacs 25.1. For example (with a custom theme to
make strings stand out):

echo this should be a string, right?
foo %alpha
foo %alpha_beta
foo "%alpha"set alpha=gammaset alpha_beta=gamma
foo foo.in



> Same question for some of the comments in the above web-page.  E.g. you
> say "Strings containing “<<” are treated as heredoc comments", but AFAIK
> this should only happen in fairly old versions of Emacs.
>

I see this in Emacs 25.1, for example:

echo "<<test"
test

I reported this over a year ago, for Emacs 25.0.93, see bug#23526.


I also see a reference to "bell_206.ps" but I can't find this file.
>

It's from http://people.sc.fsu.edu/~jburkardt/data/ps/ps.html -- I'll see
if I can add it (or update the doc).

    -- Anders

[-- Attachment #2: Type: text/html, Size: 2295 bytes --]

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Stefan Monnier]

>> Which version of bat-mode did you test?
> The one that comes with Emacs 25.1. For example (with a custom theme to
> make strings stand out):

> echo this should be a string, right?
> foo %alpha
> foo %alpha_beta
> foo "%alpha"set alpha=gammaset alpha_beta=gamma
> foo foo.in

Have you reported these as bugs?  They should be easy to fix.
I know nothing about the "bat" language, so I don't even know how the
above *should* be parsed.

>> Same question for some of the comments in the above web-page.  E.g. you
>> say "Strings containing “<<” are treated as heredoc comments", but AFAIK
>> this should only happen in fairly old versions of Emacs.
> I see this in Emacs 25.1, for example:
>
> echo "<<test"
> test
>
> I reported this over a year ago, for Emacs 25.0.93, see bug#23526.

Sorry, didn't see this one (I'm not subscribed to the bugs list any more,
so I only see the bugs that are Cc'd to me).

I believe the patch below should fix it (should appear in trunk
real-soon-now).


        Stefan


diff --git a/lisp/progmodes/sh-script.el b/lisp/progmodes/sh-script.el
index 54c47b719f..9cfbb39d53 100644
--- a/lisp/progmodes/sh-script.el
+++ b/lisp/progmodes/sh-script.el
@@ -975,7 +975,7 @@ sh-font-lock-open-heredoc
 be indented (i.e. a <<- was used rather than just <<).
 Point is at the beginning of the next line."
   (unless (or (memq (char-before start) '(?< ?>))
-	      (sh-in-comment-or-string start)
+	      (sh-in-comment-or-string (1+ start))
               (sh--inside-noncommand-expression start))
     ;; We're looking at <<STRING, so we add "^STRING$" to the syntactic
     ;; font-lock keywords to detect the end of this here document.

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

Hi!

On Tue, Aug 29, 2017 at 3:36 PM, Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> Have you reported these as bugs?  They should be easy to fix.
> I know nothing about the "bat" language, so I don't even know how the
> above *should* be parsed.
>

No, I haven't reported them. I was planning to fix them myself, but this is
quite far down on my Emacs todo list...


> I reported this over a year ago, for Emacs 25.0.93, see bug#23526.
>
> Sorry, didn't see this one (I'm not subscribed to the bugs list any more,
> so I only see the bugs that are Cc'd to me).
>

Oh, I'll remember to CC you, should I report font-lock bugs in the future.


I believe the patch below should fix it (should appear in trunk
> real-soon-now).
>

I tried it here, and it appears to be working. Thanks!

    -- Anders

[-- Attachment #2: Type: text/html, Size: 1580 bytes --]

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > No, I haven't reported them. I was planning to fix them myself, but this is
  > quite far down on my Emacs todo list...

It would be useful for you to fix them; but if you're too busy with
other useful things to fix these soon, please at least report them now.
Then others might fix them before you do.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 274 bytes --]

>
> It would be useful for you to fix them; but if you're too busy with
> other useful things to fix these soon, please at least report them now.
> Then others might fix them before you do.
>

OK, I just reported the bat-mode font-lock issues, see bug#28311.

    -- Anders

[-- Attachment #2: Type: text/html, Size: 569 bytes --]

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Dmitry Gutov]

On 8/24/17 3:31 PM, Anders Lindgren wrote:

> If you use a string face with, say, both a background and a foreground 
> color, it will matter if you *replace* the existing face as opposed to 
> *prepend* a new face on top of it.
> 
> For example, the default Ruby mode replace the face,

That is a legacy behavior, as far as I'm concerned. In some other 
editors that I've looked at (e.g. Vim and either Sublime Text or Atom), 
the contents of the interpolation construct look like normal text. You 
can have nested strings and interpolations inside them, so it's a good 
semantic cue.

> --------
> 
> One question is if the delimiters around the expression should be 
> highlighted as well? In Ruby they are highlighted. In my 
> cmake-font-lock[1] package I have opted not to highlight them, as it 
> makes it easier to read the variable name:

If the contents are highlighted differently, the variable name will 
stand out just the same, even with the delimiters highlighted.


> In all cases I've seen, the content is displayed using 
> `font-lock-variable-name-face', even for the cases where the content is 
> more complex than a plain variable. I would say that this is OK, as they 
> stand out and, most of the time, it is a plain variable anyway.

We don't use font-lock-variable-name-face for "plain variables" outside 
of strings, though. Not in ruby-mode, at least.

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 2670 bytes --]

>
> If you use a string face with, say, both a background and a foreground
>> color, it will matter if you *replace* the existing face as opposed to
>> *prepend* a new face on top of it.
>>
>> For example, the default Ruby mode replace the face,
>>
>
> That is a legacy behavior, as far as I'm concerned. In some other editors
> that I've looked at (e.g. Vim and either Sublime Text or Atom), the
> contents of the interpolation construct look like normal text. You can have
> nested strings and interpolations inside them, so it's a good semantic cue.


I guess it all depends om the level of complexity that the language
supports. In the shell and Ruby cases, nested constructs can be complex, so
I guess it makes sense to go back to the normal face (contrary to what I
said before). In other languages where the constructs only can be simple
variables, I would prefer to retain the string face (for the sake of things
like the background color) and color the variable name by prepending the
variable name face.


One question is if the delimiters around the expression should be
>> highlighted as well? In Ruby they are highlighted. In my cmake-font-lock[1]
>> package I have opted not to highlight them, as it makes it easier to read
>> the variable name:
>>
>
> If the contents are highlighted differently, the variable name will stand
> out just the same, even with the delimiters highlighted.


In my experience, I can read code easier if the delimiters aren't
highlighted the same way the content is. (In other words, the current Ruby
implementation isn't ideal for me.)


In all cases I've seen, the content is displayed using
>> `font-lock-variable-name-face', even for the cases where the content is
>> more complex than a plain variable. I would say that this is OK, as they
>> stand out and, most of the time, it is a plain variable anyway.
>>
>
> We don't use font-lock-variable-name-face for "plain variables" outside of
> strings, though. Not in ruby-mode, at least.
>

While we're on the subject. I've been thinking about highlighting
parameters in functions and blocks in Ruby. It wouldn't be too difficult,
and it would make code easier to read. Unfortunately, it doesn't include
all variables as local variables can be created on the fly using a plain
assignment in Ruby.

As a parallel, my lisp-extra-font-lock package (
https://github.com/Lindydancer/lisp-extra-font-lock) do this in Lisp modes
for function and lambda parameters and local variables introduces using
`let`, `dolist` et.c. Now that I've been using it for a couple of years, I
would not dream of going back to the near black-and-white world of the
default lisp modes.

    -- Anders

[-- Attachment #2: Type: text/html, Size: 3950 bytes --]

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Dmitry Gutov]

On 9/5/17 10:00 AM, Anders Lindgren wrote:
> In other languages where the constructs only can be 
> simple variables, I would prefer to retain the string face (for the sake 
> of things like the background color) and color the variable name by 
> prepending the variable name face.

Sure, why not.

> In my experience, I can read code easier if the delimiters aren't 
> highlighted the same way the content is. (In other words, the current 
> Ruby implementation isn't ideal for me.)

Yes, so in the case of ruby-mode, the delimiters will have some special 
face (keyword? IDK), and the content will have the default face.

> While we're on the subject. I've been thinking about highlighting 
> parameters in functions and blocks in Ruby.

Let's maybe make it an optional mode, so ruby-mode doesn't stand out too 
much (or discuss whether all modes should do that if possible, on 
emacs-devel). But I've been looking at this too, albeit from a different 
standpoint (completion of local variable names).

> It wouldn't be too 
> difficult, and it would make code easier to read. Unfortunately, it 
> doesn't include all variables as local variables can be created on the 
> fly using a plain assignment in Ruby.

Finding assignments is easy (just a regexp). Tracking variable scopes 
(defined by their belonging to methods, or blocks, or even class/module 
bodies) looks harder to me. We can do that by parsing expressions with 
SMIE, but that's not fast if we have to parse the whole class body 
(there are some big classes out there).

Limiting ourselves to only methods might be fine, though.

> As a parallel, my lisp-extra-font-lock package 
> (https://github.com/Lindydancer/lisp-extra-font-lock) do this in Lisp 
> modes for function and lambda parameters and local variables introduces 
> using `let`, `dolist` et.c. Now that I've been using it for a couple of 
> years, I would not dream of going back to the near black-and-white world 
> of the default lisp modes.

I like the idea, but seeing the red on the screenshot is a bit 
off-putting. red is for errors.

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 2621 bytes --]

>
> In my experience, I can read code easier if the delimiters aren't
>> highlighted the same way the content is. (In other words, the current Ruby
>> implementation isn't ideal for me.)
>>
>
> Yes, so in the case of ruby-mode, the delimiters will have some special
> face (keyword? IDK), and the content will have the default face.


Keyword-face or builtin-face on top of string-face would work well.


While we're on the subject. I've been thinking about highlighting
>> parameters in functions and blocks in Ruby.
>>
>
> Let's maybe make it an optional mode, so ruby-mode doesn't stand out too
> much (or discuss whether all modes should do that if possible, on
> emacs-devel). But I've been looking at this too, albeit from a different
> standpoint (completion of local variable names).


An optional mode would do nicely.


It wouldn't be too difficult, and it would make code easier to read.
>> Unfortunately, it doesn't include all variables as local variables can be
>> created on the fly using a plain assignment in Ruby.
>>
>
> Finding assignments is easy (just a regexp). Tracking variable scopes
> (defined by their belonging to methods, or blocks, or even class/module
> bodies) looks harder to me. We can do that by parsing expressions with
> SMIE, but that's not fast if we have to parse the whole class body (there
> are some big classes out there).
>
> Limiting ourselves to only methods might be fine, though.


Methods and do-blocks would be straight forward, and make Ruby code a lot
easier to read.


As a parallel, my lisp-extra-font-lock package (
>> https://github.com/Lindydancer/lisp-extra-font-lock) do this in Lisp
>> modes for function and lambda parameters and local variables introduces
>> using `let`, `dolist` et.c. Now that I've been using it for a couple of
>> years, I would not dream of going back to the near black-and-white world of
>> the default lisp modes.
>>
>
> I like the idea, but seeing the red on the screenshot is a bit
> off-putting. red is for errors.
>

The screenshot demonstrates all features at once, so it might look a bit
overwhelming -- in normal code it doesn't highlight that much.

I've opted to define new faces and make them inherit from the predefined.
This allows the faces to be customized (e.g. if you don't like red), and
they work when a custom theme is used. I settled on the warning face for
two constructs, as it was the least bad one available. So far, I've never
misinterpreted a highlighted variable name for an error, or vise versa.

Anyway, I understand your concern. I'll revisit this and see if there is
another way to do it.

    -- Anders

[-- Attachment #2: Type: text/html, Size: 4275 bytes --]

Re: Regarding Emacs, js.el, template-strings and syntax-tables

[Dmitry Gutov]

On 9/5/17 12:30 PM, Anders Lindgren wrote:

> Methods and do-blocks would be straight forward, and make Ruby code a 
> lot easier to read.

I'd like to see the code. Detecting when point is outside of any method 
might be trickier (or more expensive). Similarly for bodies of methods 
defined with define_method.