bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Ken Mankoff]

I'm having issues with the new prettify-symbols-mode. I'm not sure if
this is a bug, or just a fact that the implementation is limited, in
which case this is a feature request for a more complete implementation.

Some symbols are sometimes not being treated correctly depending on what
characters follow. For example, I have the following setup for coding
Python:

    (prettify-symbols-mode t)
    (global-prettify-symbols-mode t)
    (add-hook 'python-mode-hook
              (lambda ()
                (push '("**2" . ?²) prettify-symbols-alist)
                (push '("_x" . ?ᵪ) prettify-symbols-alist)
                (push '("delta" . ?δ) prettify-symbols-alist)))


The issue may be somewhat subjective. For example, should foo_xx appear
with a subscript x and then a normal x? Or should it appear as I assume
you are reading it with no prettification? I would argue for the latter.
What about foo_x+2? Regardless, I've created a matrix of
prettifications, what I'd expect, and what happens.

| Characters | Expected               | Actual                        | Good? |
|------------+------------------------+-------------------------------+-------|
| foo_x      | subscript x            | subscript x                   | Y     |
| foo**2     | superscript 2          | superscript 2                 | Y     |
| delta      | delta symbol           | delta symbol                  | Y     |
| foo_x+     | subscript x            | No subscript                  | N     |
| foo_xi     | no subscript           | subscript                     | N     |
| foo_x[42]  | subscript              | subscript                     | Y     |
| foo_x**2   | subscript, superscript | no subscript, yes superscript | N     |
| foo**200   | no superscript         | superscript 2                 | N     |
| delta(42)  | delta symbol(42)       | symbol                        | Y     |
| delta+42   | symbol                 | symbol                        | Y     |
| delta**2   | symbol, superscript    | symbol, superscritp           | Y     |
| delta_x    | symbol, subscript      | no symbol                     | N     |

There are some inconsistencies, like why _x+ loses prettification, but
delta+ retains it, or why foo_x_x_x works, but delta_x does not.

Thanks,

  -k.

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Stefan Monnier]

> Some symbols are sometimes not being treated correctly depending on what
> characters follow. For example, I have the following setup for coding
> Python:

>     (prettify-symbols-mode t)
>     (global-prettify-symbols-mode t)
>     (add-hook 'python-mode-hook
>               (lambda ()
>                 (push '("**2" . ?²) prettify-symbols-alist)
>                 (push '("_x" . ?ᵪ) prettify-symbols-alist)
>                 (push '("delta" . ?δ) prettify-symbols-alist)))

Can you try the patch below and see if it does what you want?


        Stefan


diff --git a/lisp/progmodes/prog-mode.el b/lisp/progmodes/prog-mode.el
index 5037020..475dd32 100644
--- a/lisp/progmodes/prog-mode.el
+++ b/lisp/progmodes/prog-mode.el
@@ -73,11 +73,13 @@ Regexp match data 0 points to the chars."
   ;; Check that the chars should really be composed into a symbol.
   (let* ((start (match-beginning 0))
 	 (end (match-end 0))
-	 (syntaxes (if (eq (char-syntax (char-after start)) ?w)
-		       '(?w ?_) '(?. ?\\)))
+	 (syntax-beg (if (eq (char-syntax (char-after start)) ?w)
+                         '(?w ?_) '(?. ?\\)))
+	 (syntax-end (if (eq (char-syntax (char-before end)) ?w)
+                         '(?w ?_) '(?. ?\\)))
 	 match)
-    (if (or (memq (char-syntax (or (char-before start) ?\s)) syntaxes)
-	    (memq (char-syntax (or (char-after end) ?\s)) syntaxes)
+    (if (or (memq (char-syntax (or (char-before start) ?\s)) syntax-beg)
+	    (memq (char-syntax (or (char-after end) ?\s)) syntax-end)
             ;; syntax-ppss could modify the match data (bug#14595)
             (progn (setq match (match-string 0)) (nth 8 (syntax-ppss))))
 	;; No composition for you.  Let's actually remove any composition

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Ken Mankoff]

* On 2014-11-21 at 13:15, Stefan Monnier wrote:
>> Some symbols are sometimes not being treated correctly depending on
>> what characters follow. For example, I have the following setup for
>> coding Python:
>
>>     (prettify-symbols-mode t)
>>     (global-prettify-symbols-mode t)
>>     (add-hook 'python-mode-hook
>>               (lambda ()
>>                 (push '("**2" . ?²) prettify-symbols-alist)
>>                 (push '("_x" . ?ᵪ) prettify-symbols-alist)
>>                 (push '("delta" . ?δ) prettify-symbols-alist)))
>
> Can you try the patch below and see if it does what you want?
>
>
> diff --git a/lisp/progmodes/prog-mode.el b/lisp/progmodes/prog-mode.el
> index 5037020..475dd32 100644
> --- a/lisp/progmodes/prog-mode.el
> +++ b/lisp/progmodes/prog-mode.el
> @@ -73,11 +73,13 @@ Regexp match data 0 points to the chars."
>    ;; Check that the chars should really be composed into a symbol.
>    (let* ((start (match-beginning 0))
>  	 (end (match-end 0))
> -	 (syntaxes (if (eq (char-syntax (char-after start)) ?w)
> -		       '(?w ?_) '(?. ?\\)))
> +	 (syntax-beg (if (eq (char-syntax (char-after start)) ?w)
> +                         '(?w ?_) '(?. ?\\)))
> +	 (syntax-end (if (eq (char-syntax (char-before end)) ?w)
> +                         '(?w ?_) '(?. ?\\)))
>  	 match)
> -    (if (or (memq (char-syntax (or (char-before start) ?\s)) syntaxes)
> -	    (memq (char-syntax (or (char-after end) ?\s)) syntaxes)
> +    (if (or (memq (char-syntax (or (char-before start) ?\s)) syntax-beg)
> +	    (memq (char-syntax (or (char-after end) ?\s)) syntax-end)
>              ;; syntax-ppss could modify the match data (bug#14595)
>              (progn (setq match (match-string 0)) (nth 8 (syntax-ppss))))
>  	;; No composition for you.  Let's actually remove any composition


Much improved! Of my examples, only one case no longer works which is
delta_x or foo_x_x_x_x, for example.

  -k.
  

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Stefan Monnier]

> Much improved! Of my examples, only one case no longer works which is
> delta_x or foo_x_x_x_x, for example.

This would seem to indicate that the syntax-table of the major-mode in
effect has marked the underscore character with "symbol syntax"
(denoted confusingly enough by an underscore character,
in the `C-u C-x =' help).

Such a setting is appropriate is "delta_x" is one identifier, but not if
it's supposed to be 3 elements (identifier "delta", infix "_", and
identifier "x").

So, is this setting correct (i.e. does your language treat "delta_x" as
a single identifier, and you're trying to prettify subparts of
identifiers)?


        Stefan

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Ken Mankoff]

* On 2014-11-22 at 11:23, Stefan Monnier wrote:
>> Much improved! Of my examples, only one case no longer works which is
>> delta_x or foo_x_x_x_x, for example.
>
> This would seem to indicate that the syntax-table of the major-mode in
> effect has marked the underscore character with "symbol syntax"
> (denoted confusingly enough by an underscore character, in the `C-u
> C-x =' help).
>
> Such a setting is appropriate is "delta_x" is one identifier, but not
> if it's supposed to be 3 elements (identifier "delta", infix "_", and
> identifier "x").
>
> So, is this setting correct (i.e. does your language treat "delta_x"
> as a single identifier, and you're trying to prettify subparts of
> identifiers)?

I'm not sure. I'm just using python and elpy (and therefore
python-mode). I think underscores are normal characters, and a_b is a
generic variable name, three characters long. There is nothing special
about _ or _b (compared to LaTeX and LaTeX-mode, where there is
meaning).

Does this mean I should file an issue with python-mode, or more likely
that everything is working as it should, and I'm just running into an
edge case that can only be covered by a more complex implementation that
uses regexes?

  -k.
  

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Stefan Monnier]

>> So, is this setting correct (i.e. does your language treat "delta_x"
>> as a single identifier, and you're trying to prettify subparts of
>> identifiers)?
> I'm not sure. I'm just using python and elpy (and therefore
> python-mode).

Duh!  Sorry, I somehow failed to see "python-mode" in your original
bug report.  So yes, you're trying to prettify subparts of identifiers,
and prettify-symbols-mode currently provides no support at all for that.

> that everything is working as it should, and I'm just running into an
> edge case that can only be covered by a more complex implementation that
> uses regexes?

Exactly.

for _x and **2, I think prettify-symbols-mode is probably not a good
solution anyway (because it won't extend to _xy or to **24).  You'd be
better off with a font-lock rule which just shifts the text up/down (you
might like to look at the way we do just that in text-mode.el), which
can work with any sequence of character rather than being limited to
those few characters which have a "superscript" form in Unicode.

But w.r.t "delta" in "delta_x" it would make a lot of sense for
prettify-symbols-mode to provide support for that.


        Stefan

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Ted Zlatanov]

On Mon, 24 Nov 2014 09:53:43 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

SM> Duh!  Sorry, I somehow failed to see "python-mode" in your original
SM> bug report.  So yes, you're trying to prettify subparts of identifiers,
SM> and prettify-symbols-mode currently provides no support at all for that.
...
SM> But w.r.t "delta" in "delta_x" it would make a lot of sense for
SM> prettify-symbols-mode to provide support for that.

I don't think it would--I would keep `prettify-symbols-mode' strict. I
think Ken needs a different mode that's yet to be written:
`prettify-regex-mode'?

"delta_x" is an entirely different thing from "delta" in all the
programming languages I can think of, so I would not expect to see δ_x
all over the code just because I want to prettify deltas themselves. If
anything, it would make finding the real standalone deltas harder.

Ted

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Stefan Monnier]

SM> But w.r.t "delta" in "delta_x" it would make a lot of sense for
SM> prettify-symbols-mode to provide support for that.
> I don't think it would--I would keep `prettify-symbols-mode' strict. I
> think Ken needs a different mode that's yet to be written:
> `prettify-regex-mode'?

By "provide support for that", I meant to provide the option to specify
symbols with regexps.


        Stefan

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Ken Mankoff]

* On 2014-11-25 at 04:49, Ted Zlatanov wrote:
> On Mon, 24 Nov 2014 09:53:43 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
>
> SM> Duh!  Sorry, I somehow failed to see "python-mode" in your original
> SM> bug report.  So yes, you're trying to prettify subparts of identifiers,
> SM> and prettify-symbols-mode currently provides no support at all for that.
> ...
> SM> But w.r.t "delta" in "delta_x" it would make a lot of sense for
> SM> prettify-symbols-mode to provide support for that.
>
> I don't think it would--I would keep `prettify-symbols-mode' strict. I
> think Ken needs a different mode that's yet to be written:
> `prettify-regex-mode'?

This mode exists - pretty-symbols-mode
https://github.com/drothlis/pretty-symbols

I just try to use official packages when possible, hence my recent
switch to prettify-symbols-mode.

  -k.
  

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Ted Zlatanov]

On Tue, 25 Nov 2014 09:51:56 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

SM> But w.r.t "delta" in "delta_x" it would make a lot of sense for
SM> prettify-symbols-mode to provide support for that.
>> I don't think it would--I would keep `prettify-symbols-mode' strict. I
>> think Ken needs a different mode that's yet to be written:
>> `prettify-regex-mode'?

SM> By "provide support for that", I meant to provide the option to specify
SM> symbols with regexps.

But that would require some special adaptation because it wouldn't work
in reverse. IOW, how do you propose to preserve the current behavior of
the prettified symbol "delta" while also allowing for the regex "delta"?

Ted

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Stefan Monnier]

> But that would require some special adaptation because it wouldn't work
> in reverse. IOW, how do you propose to preserve the current behavior of
> the prettified symbol "delta" while also allowing for the regex "delta"?

We could allow elements of the form (REGEXP CHARACTER PREDICATE)
instead of only (STRING . CHARACTER) in p-s-alist.


        Stefan

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Ted Zlatanov]

On Tue, 25 Nov 2014 12:36:28 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> But that would require some special adaptation because it wouldn't work
>> in reverse. IOW, how do you propose to preserve the current behavior of
>> the prettified symbol "delta" while also allowing for the regex "delta"?

SM> We could allow elements of the form (REGEXP CHARACTER PREDICATE)
SM> instead of only (STRING . CHARACTER) in p-s-alist.

OK, that would work and be backwards compatible.  Do you want me to
attempt it?

Ted

bug#19145: 24.4; prettify-symbols-mode inconsistent behavior

[Stefan Monnier]

> OK, that would work and be backwards compatible.  Do you want me to
> attempt it?

Be my guest, yes,


        Stefan