using glyphs by default in perl-mode

using glyphs by default in perl-mode

[Dan Nicolaescu]

It looks like perl-mode now likes to replace -> with a glyph by
default.  It is not a good idea.

And example where this does not work:

emacs -Q --daemon

In a terminal do:
emacsclient -t

C-x C-f perlfile.pl RET

The above will replace -> with a glyph for an arrow, 
so  the "perlfile.pl" buffer is unreadable, the glyph cannot be
displayed in a terminal and it shows up as a question mark.

Re: using glyphs by default in perl-mode

[Stefan Monnier]

> The above will replace -> with a glyph for an arrow, 
> so the "perlfile.pl" buffer is unreadable, the glyph cannot be
> displayed in a terminal and it shows up as a question mark.

Hmm... works for me (i.e. the arrow is displayed just fine in the terminal).
Please make it a bug report with the usual details to reproduce it.


        Stefan

Re: using glyphs by default in perl-mode

[James Cloos]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The above will replace -> with a glyph for an arrow, 
>> so the "perlfile.pl" buffer is unreadable, the glyph cannot be
>> displayed in a terminal and it shows up as a question mark.

SM> Hmm... works for me (i.e. the arrow is displayed just fine in the terminal).
SM> Please make it a bug report with the usual details to reproduce it.

I bet the difference is the set of fonts each terminal is configured to use.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: using glyphs by default in perl-mode

[Stefan Monnier]

>>> The above will replace -> with a glyph for an arrow, 
>>> so the "perlfile.pl" buffer is unreadable, the glyph cannot be
>>> displayed in a terminal and it shows up as a question mark.
SM> Hmm... works for me (i.e. the arrow is displayed just fine in the terminal).
SM> Please make it a bug report with the usual details to reproduce it.
> I bet the difference is the set of fonts each terminal is configured to use.

I don't think so: the terminals I know don't replace a char with
a question mark if the font has no glyph for it (they use some other
representation, like an empty box).  It sounds instead like your
terminal is using an encoding that does not include those chars we're
using, so Emacs replaces them with a question mark.

What is your terminal's encoding?  What about your locale?


        Stefan

Re: using glyphs by default in perl-mode

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> The above will replace -> with a glyph for an arrow, 
>>>> so the "perlfile.pl" buffer is unreadable, the glyph cannot be
>>>> displayed in a terminal and it shows up as a question mark.
> SM> Hmm... works for me (i.e. the arrow is displayed just fine in the terminal).
> SM> Please make it a bug report with the usual details to reproduce it.
>> I bet the difference is the set of fonts each terminal is configured to use.
>
> I don't think so: the terminals I know don't replace a char with
> a question mark if the font has no glyph for it (they use some other
> representation, like an empty box).  It sounds instead like your
> terminal is using an encoding that does not include those chars we're
> using, so Emacs replaces them with a question mark.
>
> What is your terminal's encoding?  What about your locale?

LANG is C
XTERM_LOCALE is C

Re: using glyphs by default in perl-mode

[Stefan Monnier]

>>>>> The above will replace -> with a glyph for an arrow, 
>>>>> so the "perlfile.pl" buffer is unreadable, the glyph cannot be
>>>>> displayed in a terminal and it shows up as a question mark.
SM> Hmm... works for me (i.e. the arrow is displayed just fine in the
SM> terminal).  Please make it a bug report with the usual details to
SM> reproduce it.
>>> I bet the difference is the set of fonts each terminal is configured
>>> to use.
>> I don't think so: the terminals I know don't replace a char with
>> a question mark if the font has no glyph for it (they use some other
>> representation, like an empty box).  It sounds instead like your
>> terminal is using an encoding that does not include those chars we're
>> using, so Emacs replaces them with a question mark.
>> What is your terminal's encoding?  What about your locale?
> LANG is C
> XTERM_LOCALE is C

I guess we could try and only enable those compositions for locales that
rely on something like utf-8?

Another solution for affected users is to switch to a more modern
locale, like en_US.utf-8, or to customize perl-prettify-symbols.
Non-ASCII chars are handy at various places and are becoming more and
more frequent (in Emacs and elsewhere), so it's a good idea to "upgrade"
your locale.


        Stefan

Re: using glyphs by default in perl-mode

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>>>> The above will replace -> with a glyph for an arrow, 
>>>>>> so the "perlfile.pl" buffer is unreadable, the glyph cannot be
>>>>>> displayed in a terminal and it shows up as a question mark.
> SM> Hmm... works for me (i.e. the arrow is displayed just fine in the
> SM> terminal).  Please make it a bug report with the usual details to
> SM> reproduce it.
>>>> I bet the difference is the set of fonts each terminal is configured
>>>> to use.
>>> I don't think so: the terminals I know don't replace a char with
>>> a question mark if the font has no glyph for it (they use some other
>>> representation, like an empty box).  It sounds instead like your
>>> terminal is using an encoding that does not include those chars we're
>>> using, so Emacs replaces them with a question mark.
>>> What is your terminal's encoding?  What about your locale?
>> LANG is C
>> XTERM_LOCALE is C
>
> I guess we could try and only enable those compositions for locales that
> rely on something like utf-8?

I does not work on the Linux console of a Fedora 18 machine using
en_us.UTF-8.

What is worse is the only way to get rid of it is to turn off the option
and then restart emacs.

It looks like having this option on by default is not a good idea.

Re: using glyphs by default in perl-mode

[Stefan Monnier]

> I does not work on the Linux console of a Fedora 18 machine using
> en_us.UTF-8.

Do you also see question marks, or do you see something else?

> What is worse is the only way to get rid of it is to turn off the option
> and then restart Emacs.

That should be easy to fix if deemed important.


        Stefan

Re: using glyphs by default in perl-mode

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I does not work on the Linux console of a Fedora 18 machine using
>> en_us.UTF-8.
>
> Do you also see question marks, or do you see something else?

It looks like a small white square. 

>> What is worse is the only way to get rid of it is to turn off the option
>> and then restart Emacs.
>
> That should be easy to fix if deemed important.
>
>
>         Stefan

Re: using glyphs by default in perl-mode

[Stefan Monnier]

>>> I does not work on the Linux console of a Fedora 18 machine using
>>> en_us.UTF-8.
>> Do you also see question marks, or do you see something else?
> It looks like a small white square. 

Thanks.  This one is a problem of font, which might be solvable by
fiddling with your console's font, but Emacs has no way to tell whether
those chars can be displayed (contrary to the "LANG=C" case where Emacs
knows full well that those symbols won't get displayed since it doesn't
even pass them to the terminal).


        Stefan

Re: using glyphs by default in perl-mode

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> I does not work on the Linux console of a Fedora 18 machine using
>>>> en_us.UTF-8.
>>> Do you also see question marks, or do you see something else?
>> It looks like a small white square. 
>
> Thanks.  This one is a problem of font, which might be solvable by
> fiddling with your console's font, but Emacs has no way to tell whether

This is the default font for the console.  Normal users cannot change
the console font.

This means that a Linux console is unusable to edit perl code by
default.

What problem does showing a glyph instead of -> solve?  This looks like
a personal preference, and one that causes problems.

Please revert this change, it is fundamentally broken as a default.

Re: using glyphs by default in perl-mode

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@gnu.org>
> Date: Thu, 09 May 2013 12:31:32 -0400
> Cc: James Cloos <cloos@jhcloos.com>, emacs-devel@gnu.org
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >>>> I does not work on the Linux console of a Fedora 18 machine using
> >>>> en_us.UTF-8.
> >>> Do you also see question marks, or do you see something else?
> >> It looks like a small white square. 
> >
> > Thanks.  This one is a problem of font, which might be solvable by
> > fiddling with your console's font, but Emacs has no way to tell whether
> 
> This is the default font for the console.  Normal users cannot change
> the console font.
> 
> This means that a Linux console is unusable to edit perl code by
> default.
> 
> What problem does showing a glyph instead of -> solve?  This looks like
> a personal preference, and one that causes problems.

Does char-displayable-p detect that this glyph cannot be displayed in
your environment?  If so, perhaps we could have the cake and eat it,
too.

Re: using glyphs by default in perl-mode

[Stefan Monnier]

> Does char-displayable-p detect that this glyph cannot be displayed in
> your environment?

It might/should when he uses LANG=C but not when he uses a utf-8 environment.

> If so, perhaps we could have the cake and eat it, too.

I don't think we can (even less so if you consider emacsclient use).

But most of the people I showed it to liked the feature, which is why
I'd like to keep it ON by default, even though it's obviously not
a terribly important feature.


        Stefan

Re: using glyphs by default in perl-mode

[Dan Nicolaescu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Dan Nicolaescu <dann@gnu.org>
>> Date: Thu, 09 May 2013 12:31:32 -0400
>> Cc: James Cloos <cloos@jhcloos.com>, emacs-devel@gnu.org
>> 
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> 
>> >>>> I does not work on the Linux console of a Fedora 18 machine using
>> >>>> en_us.UTF-8.
>> >>> Do you also see question marks, or do you see something else?
>> >> It looks like a small white square. 
>> >
>> > Thanks.  This one is a problem of font, which might be solvable by
>> > fiddling with your console's font, but Emacs has no way to tell whether
>> 
>> This is the default font for the console.  Normal users cannot change
>> the console font.
>> 
>> This means that a Linux console is unusable to edit perl code by
>> default.
>> 
>> What problem does showing a glyph instead of -> solve?  This looks like
>> a personal preference, and one that causes problems.
>
> Does char-displayable-p detect that this glyph cannot be displayed in
> your environment?  If so, perhaps we could have the cake and eat it,
> too.

Using char-displayable-p to change the contents of a buffer does not
work in the multi-tty world.  What is displayable when the buffer is
shown in one terminal, might not be in another.

Re: using glyphs by default in perl-mode

[Dan Nicolaescu]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> Does char-displayable-p detect that this glyph cannot be displayed in
>> your environment?
>
> It might/should when he uses LANG=C but not when he uses a utf-8 environment.
>
>> If so, perhaps we could have the cake and eat it, too.
>
> I don't think we can (even less so if you consider emacsclient use).
>
> But most of the people I showed it to liked the feature, which is why
> I'd like to keep it ON by default, even though it's obviously not
> a terribly important feature.

Given that the feature breaks on the Linux console, which is a small,
but still a significant target (it probably has more users that a lot of
other systems that emacs still supports) it does not seem like it's a
good idea to have it on.

Re: using glyphs by default in perl-mode

[Andreas Röhler]

Am 09.05.2013 19:22, schrieb Stefan Monnier:
>> Does char-displayable-p detect that this glyph cannot be displayed in
>> your environment?
>
> It might/should when he uses LANG=C but not when he uses a utf-8 environment.
>
>> If so, perhaps we could have the cake and eat it, too.
>
> I don't think we can (even less so if you consider emacsclient use).
>
> But most of the people I showed it to liked the feature, which is why
> I'd like to keep it ON by default, even though it's obviously not
> a terribly important feature.
>
>
>          Stefan
>
>

Hi Stefan,

in cases, something might do some good mostly, but fails sometimes, isn't it wise to keep it off by default?
IMO reliability weights more than feature/fanciness.

Cheers,

Andreas

Re: using glyphs by default in perl-mode

[James Cloos]

>>>>> "SM" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

SM> But most of the people I showed it to liked the feature, which is why
SM> I'd like to keep it ON by default, even though it's obviously not
SM> a terribly important feature.

As a side note, and from a usability POV, I find that corrupting source
code like that makes it harder to read and grok.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: using glyphs by default in perl-mode

[Richard Stallman]

    This is the default font for the console.  Normal users cannot change
    the console font.

    This means that a Linux console is unusable to edit perl code by
    default.

It is very bad to use, by default, characters that terminals can't
display.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Thu, 09 May 2013 13:22:45 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> Does char-displayable-p detect that this glyph cannot be displayed in
>> your environment?

SM> It might/should when he uses LANG=C but not when he uses a utf-8 environment.

>> If so, perhaps we could have the cake and eat it, too.

SM> I don't think we can (even less so if you consider emacsclient use).

SM> But most of the people I showed it to liked the feature, which is why
SM> I'd like to keep it ON by default, even though it's obviously not
SM> a terribly important feature.

I heard from one friend who had trouble because of it already.  He
blamed me ("the Emacs guy") of course :)

As much as I like the feature and will keep it on for myself, I vote to
turn it off.  It should be a top-level preference "glyphify" in
`prog-mode' and easy to turn on, though.  There's many other languages and
contexts where it's handy, and a unified front for it would be nice.

For instance, I use the pretty-lambdas thingy for all Lisp work, and it
makes sense for Python, but not for Perl.  `prog-mode' could support that[1].

Ted

[1] it probably already does, of course

Re: using glyphs by default in perl-mode

[Stefan Monnier]

> As much as I like the feature and will keep it on for myself, I vote to
> turn it off.  It should be a top-level preference "glyphify" in
> `prog-mode' and easy to turn on, though.  There's many other languages and
> contexts where it's handy, and a unified front for it would be nice.

I like that solution.  Could you take care of it?


        Stefan

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Fri, 17 May 2013 11:48:23 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> As much as I like the feature and will keep it on for myself, I vote to
>> turn it off.  It should be a top-level preference "glyphify" in
>> `prog-mode' and easy to turn on, though.  There's many other languages and
>> contexts where it's handy, and a unified front for it would be nice.

SM> I like that solution.  Could you take care of it?

It's now on my TODO list.

Ted

Re: using glyphs by default in perl-mode

[Dan Nicolaescu]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Fri, 17 May 2013 11:48:23 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
>
>>> As much as I like the feature and will keep it on for myself, I vote to
>>> turn it off.  It should be a top-level preference "glyphify" in
>>> `prog-mode' and easy to turn on, though.  There's many other languages and
>>> contexts where it's handy, and a unified front for it would be nice.
>
> SM> I like that solution.  Could you take care of it?
>
> It's now on my TODO list.

Can you please disable this by default until it's done?  It has negative
and annoying and very hard to remove effects...

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Wed, 29 May 2013 20:01:12 -0400 Dan Nicolaescu <dann@gnu.org> wrote: 

DN> Ted Zlatanov <tzz@lifelogs.com> writes:
>> On Fri, 17 May 2013 11:48:23 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
>> 
>>>> As much as I like the feature and will keep it on for myself, I vote to
>>>> turn it off.  It should be a top-level preference "glyphify" in
>>>> `prog-mode' and easy to turn on, though.  There's many other languages and
>>>> contexts where it's handy, and a unified front for it would be nice.
>> 
SM> I like that solution.  Could you take care of it?
>> 
>> It's now on my TODO list.

DN> Can you please disable this by default until it's done?  It has negative
DN> and annoying and very hard to remove effects...

Maintainers: OK to do this?  I'll take care of the details.

Ted

Re: using glyphs by default in perl-mode

[Stefan Monnier]

DN> Can you please disable this by default until it's done?  It has negative
DN> and annoying and very hard to remove effects...
> Maintainers: OK to do this?  I'll take care of the details.

I'd rather we do it all in one go.  It can't be that hard to add
a config var to prog-mode and then use it in perl-mode.


        Stefan

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Thu, 30 May 2013 13:18:34 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

DN> Can you please disable this by default until it's done?  It has negative
DN> and annoying and very hard to remove effects...
>> Maintainers: OK to do this?  I'll take care of the details.

SM> I'd rather we do it all in one go.  It can't be that hard to add
SM> a config var to prog-mode and then use it in perl-mode.

OK, I'll "accelerate development synergy" :)

Ted

Re: using glyphs by default in perl-mode

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> DN> Can you please disable this by default until it's done?  It has negative
> DN> and annoying and very hard to remove effects...
>> Maintainers: OK to do this?  I'll take care of the details.
>
> I'd rather we do it all in one go.  It can't be that hard to add
> a config var to prog-mode and then use it in perl-mode.

The final result still is that the feature is off by default, right?
emacs cannot be used to edit perl in some configurations, so the faster
this ends the better.

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 2731 bytes --]

On Sat, 01 Jun 2013 02:29:05 -0400 Dan Nicolaescu <dann@gnu.org> wrote: 

DN> Stefan Monnier <monnier@iro.umontreal.ca> writes:
DN> Can you please disable this by default until it's done?  It has negative
DN> and annoying and very hard to remove effects...
>>> Maintainers: OK to do this?  I'll take care of the details.
>> 
>> I'd rather we do it all in one go.  It can't be that hard to add
>> a config var to prog-mode and then use it in perl-mode.

DN> The final result still is that the feature is off by default, right?
DN> emacs cannot be used to edit perl in some configurations, so the faster
DN> this ends the better.

OK, so this only happens if font-lock decoration level more than 2 is
requested: in perl-mode.el, we do

#+begin_src lisp
  (setq font-lock-defaults '((perl-font-lock-keywords
			      perl-font-lock-keywords-1
			      perl-font-lock-keywords-2)
...
#+end_src

The keywords-2 list is the one with the Unicode glyphs.

So perhaps there should be a way to specify max font-lock decoration in
terminal vs. graphical.  Anyhow, just something I noticed... no big deal.

The attached patch does the following:

1) move `perl--prettify-symbols-alist' and the functions for it,
`perl--font-lock-compose-symbol' and `perl--font-lock-symbols-keywords'
to simple.el (with the "prog-" prefix).

2) declare `perl-prettify-symbols' obsolete in favor of the alist
`prog-prettify-symbols'.

I gave `prog-prettify-symbols' two default choices:
`prog--prettify-symbols-alist-basic' (just -> => ::) and
`prog--prettify-symbols-alist-extended' (basic plus much more).  It can
also be a free-form alist with key=string, value=character.  I expect
programming modes will need to augment this list somehow, but I'd like
the user to simply say "I want basics or extended" instead of having to
select individual glyphs for every language.  Maybe the choice should be
between :basic and :extended as symbols?  Ideas welcome.

`prog-prettify-symbols' defaults to nil, so this feature is turned off
by default.

3) tell `perl-mode' to use `prog--font-lock-symbols-keywords' instead of
`perl--font-lock-symbols-keywords'.

The patch grows simple.el.  Perhaps it should live in prog.el or
prettify.el.  prog.el seems more sensible.

I can't get `perl-mode' to show the symbols, so the patch is broken
right now (I checked the data structures and they're OK; this is
something in the font-lock plumbing I think).  I also wonder if it's
possible to add the prettify support at the `prog-mode' level, so each
mode doesn't have to explicitly inline those keywords.

Unfortunately my font-lock-fu is not good, so I'd really appreciate help
with the last two issues.  I spent a few hours on them but couldn't work
them out.

Thanks
Ted


[-- Attachment #2: prog-prettify.patch --]
[-- Type: text/x-diff, Size: 4811 bytes --]

=== modified file 'lisp/progmodes/perl-mode.el'
--- lisp/progmodes/perl-mode.el	2013-05-07 18:06:13 +0000
+++ lisp/progmodes/perl-mode.el	2013-06-01 13:28:52 +0000
@@ -163,39 +163,7 @@
   :version "24.4"
   :type 'boolean)
 
-(defconst perl--prettify-symbols-alist
-  '(;;("andalso" . ?∧) ("orelse"  . ?∨) ("as" . ?≡)("not" . ?¬)
-    ;;("div" . ?÷) ("*"   . ?×) ("o"   . ?○)
-    ("->"  . ?→)
-    ("=>"  . ?⇒)
-    ;;("<-"  . ?←) ("<>"  . ?≠) (">="  . ?≥) ("<="  . ?≤) ("..." . ?⋯)
-    ("::" . ?∷)
-    ))
-
-(defun perl--font-lock-compose-symbol ()
-  "Compose a sequence of ascii chars into a symbol.
-Regexp match data 0 points to the chars."
-  ;; Check that the chars should really be composed into a symbol.
-  (let* ((start (match-beginning 0))
-	 (end (match-end 0))
-	 (syntaxes (if (eq (char-syntax (char-after start)) ?w)
-		       '(?w) '(?. ?\\))))
-    (if (or (memq (char-syntax (or (char-before start) ?\ )) syntaxes)
-	    (memq (char-syntax (or (char-after end) ?\ )) syntaxes)
-            (nth 8 (syntax-ppss)))
-	;; No composition for you.  Let's actually remove any composition
-	;; we may have added earlier and which is now incorrect.
-	(remove-text-properties start end '(composition))
-      ;; That's a symbol alright, so add the composition.
-      (compose-region start end (cdr (assoc (match-string 0)
-                                            perl--prettify-symbols-alist)))))
-  ;; Return nil because we're not adding any face property.
-  nil)
-
-(defun perl--font-lock-symbols-keywords ()
-  (when perl-prettify-symbols
-    `((,(regexp-opt (mapcar 'car perl--prettify-symbols-alist) t)
-       (0 (perl--font-lock-compose-symbol))))))
+(make-obsolete-variable 'perl-prettify-symbols 'prog-prettify-symbols "24.4")
 
 (defconst perl-font-lock-keywords-1
   '(;; What is this for?
@@ -244,7 +212,7 @@
      ("\\<\\(continue\\|goto\\|last\\|next\\|redo\\)\\>[ \t]*\\(\\sw+\\)?"
       (1 font-lock-keyword-face) (2 font-lock-constant-face nil t))
      ("^[ \t]*\\(\\sw+\\)[ \t]*:[^:]" 1 font-lock-constant-face)
-     ,@(perl--font-lock-symbols-keywords)))
+     ,@(prog--font-lock-symbols-keywords)))
   "Gaudy level highlighting for Perl mode.")
 
 (defvar perl-font-lock-keywords perl-font-lock-keywords-1

=== modified file 'lisp/simple.el'
--- lisp/simple.el	2013-05-25 02:21:49 +0000
+++ lisp/simple.el	2013-06-01 13:23:00 +0000
@@ -400,6 +400,55 @@
   ;; Any programming language is always written left to right.
   (setq bidi-paragraph-direction 'left-to-right))
 
+(defconst prog--prettify-symbols-alist-basic
+  '(("->"  . ?→)
+    ("=>"  . ?⇒)
+    ("::" . ?∷)
+    )
+  "Basic list of symbols to prettify, useful in most languages.")
+
+(defconst prog--prettify-symbols-alist-extended
+  (append prog--prettify-symbols-alist-basic
+          '(("andalso" . ?∧) ("orelse"  . ?∨) ("as" . ?≡)("not" . ?¬)
+            ("div" . ?÷) ("*"   . ?×) ("o"   . ?○)
+            ("<-"  . ?←) ("<>"  . ?≠) (">="  . ?≥) ("<="  . ?≤) ("..." . ?⋯)
+            ))
+  "Extended list of symbols to prettify.")
+
+(defcustom prog-prettify-symbols nil
+  "Which symbols should be prettified.
+Set to an alist in the form `((\"->\" . ?→) ...)'."
+  :type `(choice
+          (const :tag "Basic list: -> => ::" ,prog--prettify-symbols-alist-basic)
+          (const :tag "Extended list: basic plus much more" ,prog--prettify-symbols-alist-extended)
+          (alist :tag "Define your own list" :key-type string :value-type character))
+  :group 'languages)
+
+(defun prog--font-lock-compose-symbol ()
+  "Compose a sequence of ascii chars into a symbol.
+Regexp match data 0 points to the chars."
+  ;; Check that the chars should really be composed into a symbol.
+  (let* ((start (match-beginning 0))
+	 (end (match-end 0))
+	 (syntaxes (if (eq (char-syntax (char-after start)) ?w)
+		       '(?w) '(?. ?\\))))
+    (if (or (memq (char-syntax (or (char-before start) ?\ )) syntaxes)
+	    (memq (char-syntax (or (char-after end) ?\ )) syntaxes)
+            (nth 8 (syntax-ppss)))
+	;; No composition for you.  Let's actually remove any composition
+	;; we may have added earlier and which is now incorrect.
+	(remove-text-properties start end '(composition))
+      ;; That's a symbol alright, so add the composition.
+      (compose-region start end (cdr (assoc (match-string 0)
+                                            prog--prettify-symbols-alist)))))
+  ;; Return nil because we're not adding any face property.
+  nil)
+
+(defun prog--font-lock-symbols-keywords ()
+  (when prog-prettify-symbols
+    `((,(regexp-opt (mapcar 'car prog-prettify-symbols) t)
+       (0 (prog--font-lock-compose-symbol))))))
+
 ;; Making and deleting lines.
 
 (defvar hard-newline (propertize "\n" 'hard t 'rear-nonsticky '(hard))

Re: using glyphs by default in perl-mode

[Stefan Monnier]

> 1) move `perl--prettify-symbols-alist' and the functions for it,
> `perl--font-lock-compose-symbol' and `perl--font-lock-symbols-keywords'
> to simple.el (with the "prog-" prefix).

You might like to consider the pretty-lambda for Lisp as well as the
sml-font-lock-symbols in elpa/packages/sml-mode/sml-mode.el.
[ There are others out there for, e.g. for Hasekll-mode.  ]

> 2) declare `perl-prettify-symbols' obsolete in favor of the alist
> `prog-prettify-symbols'.

No need for such obsolescence declaration, since it's never been in any
released version.

> I gave `prog-prettify-symbols' two default choices:
> `prog--prettify-symbols-alist-basic' (just -> => ::) and
> `prog--prettify-symbols-alist-extended' (basic plus much more).  It can
> also be a free-form alist with key=string, value=character.  I expect
> programming modes will need to augment this list somehow, but I'd like
> the user to simply say "I want basics or extended" instead of having to
> select individual glyphs for every language.  Maybe the choice should be
> between :basic and :extended as symbols?  Ideas welcome.

In my experience, every major mode might want to provide its own list.

> The patch grows simple.el.  Perhaps it should live in prog.el or
> prettify.el.  prog.el seems more sensible.

I think prog-mode.el is in order, yes.

> I also wonder if it's possible to add the prettify support at the
> `prog-mode' level, so each mode doesn't have to explicitly inline
> those keywords.

That would be nice, indeed.  But I fear it's going to be tricky to make
it work right, so let's keep it for later.

> Unfortunately my font-lock-fu is not good, so I'd really appreciate help
> with the last two issues.  I spent a few hours on them but couldn't work
> them out.

I don't have much time to devote right now, but the current problem is
probably a minor oversight.


        Stefan

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 2970 bytes --]

On Sat, 01 Jun 2013 10:47:04 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> 1) move `perl--prettify-symbols-alist' and the functions for it,
>> `perl--font-lock-compose-symbol' and `perl--font-lock-symbols-keywords'
>> to simple.el (with the "prog-" prefix).

SM> You might like to consider the pretty-lambda for Lisp as well as the
SM> sml-font-lock-symbols in elpa/packages/sml-mode/sml-mode.el.
SM> [ There are others out there for, e.g. for Hasekll-mode.  ]

OK, I'll add those once `perl-mode' works.

>> 2) declare `perl-prettify-symbols' obsolete in favor of the alist
>> `prog-prettify-symbols'.

SM> No need for such obsolescence declaration, since it's never been in any
SM> released version.

OK.

>> I gave `prog-prettify-symbols' two default choices:
>> `prog--prettify-symbols-alist-basic' (just -> => ::) and
>> `prog--prettify-symbols-alist-extended' (basic plus much more).  It can
>> also be a free-form alist with key=string, value=character.  I expect
>> programming modes will need to augment this list somehow, but I'd like
>> the user to simply say "I want basics or extended" instead of having to
>> select individual glyphs for every language.  Maybe the choice should be
>> between :basic and :extended as symbols?  Ideas welcome.

SM> In my experience, every major mode might want to provide its own list.

OK, see patch-2 below.

>> The patch grows simple.el.  Perhaps it should live in prog.el or
>> prettify.el.  prog.el seems more sensible.

SM> I think prog-mode.el is in order, yes.

I'll do that in patch-3, for now it's all in simple.el.

>> I also wonder if it's possible to add the prettify support at the
>> `prog-mode' level, so each mode doesn't have to explicitly inline
>> those keywords.

SM> That would be nice, indeed.  But I fear it's going to be tricky to make
SM> it work right, so let's keep it for later.

OK.  We now build the list of keywords dynamically when `perl-mode' is
called, based on `prog-prettify-symbols'.  That variable can be 'basic,
'extended, 'all (a symbol does a lookup in the buffer-local
`prog-prettify-symbols-alist') or a directly defined alist.

On the `perl-mode' side, you just do

#+begin_src lisp
  (setq-local prog-prettify-symbols-alist perl--prettify-symbols-alist)
#+end_src

and then `(prog-font-lock-symbols-keywords)' automatically will give you
the font-lock keywords.

>> Unfortunately my font-lock-fu is not good, so I'd really appreciate help
>> with the last two issues.  I spent a few hours on them but couldn't work
>> them out.

SM> I don't have much time to devote right now, but the current problem is
SM> probably a minor oversight.

I have the data working and AFAICT the correct data is passed to
font-lock but something is going wrong inside the fontification call.
That's hard to debug.  I'll work on it more (I think you posted a trick
for debugging font-lock a while ago) but if anyone can spot the issue, I
would appreciate any suggestions.

Thanks
Ted


[-- Attachment #2: prog-prettify2.patch --]
[-- Type: text/x-diff, Size: 7458 bytes --]

=== modified file 'lisp/progmodes/perl-mode.el'
--- lisp/progmodes/perl-mode.el	2013-05-07 18:06:13 +0000
+++ lisp/progmodes/perl-mode.el	2013-06-02 04:30:35 +0000
@@ -158,44 +158,21 @@
 ;; Regexps updated with help from Tom Tromey <tromey@cambric.colorado.edu> and
 ;; Jim Campbell <jec@murzim.ca.boeing.com>.
 
-(defcustom perl-prettify-symbols t
-  "If non-nil, some symbols will be displayed using Unicode chars."
-  :version "24.4"
-  :type 'boolean)
+(defconst perl--prettify-symbols-alist-basic
+  '(("->"  . ?→)
+    ("=>"  . ?⇒)
+    ("::" . ?∷)))
+
+(defconst perl--prettify-symbols-alist-extended
+  `(,@perl--prettify-symbols-alist-basic
+    ("andalso" . ?∧) ("orelse"  . ?∨) ("as" . ?≡)("not" . ?¬)
+    ("div" . ?÷) ("*"   . ?×) ("o"   . ?○)
+    ("<-"  . ?←) ("<>"  . ?≠) (">="  . ?≥) ("<="  . ?≤) ("..." . ?⋯)))
 
 (defconst perl--prettify-symbols-alist
-  '(;;("andalso" . ?∧) ("orelse"  . ?∨) ("as" . ?≡)("not" . ?¬)
-    ;;("div" . ?÷) ("*"   . ?×) ("o"   . ?○)
-    ("->"  . ?→)
-    ("=>"  . ?⇒)
-    ;;("<-"  . ?←) ("<>"  . ?≠) (">="  . ?≥) ("<="  . ?≤) ("..." . ?⋯)
-    ("::" . ?∷)
-    ))
-
-(defun perl--font-lock-compose-symbol ()
-  "Compose a sequence of ascii chars into a symbol.
-Regexp match data 0 points to the chars."
-  ;; Check that the chars should really be composed into a symbol.
-  (let* ((start (match-beginning 0))
-	 (end (match-end 0))
-	 (syntaxes (if (eq (char-syntax (char-after start)) ?w)
-		       '(?w) '(?. ?\\))))
-    (if (or (memq (char-syntax (or (char-before start) ?\ )) syntaxes)
-	    (memq (char-syntax (or (char-after end) ?\ )) syntaxes)
-            (nth 8 (syntax-ppss)))
-	;; No composition for you.  Let's actually remove any composition
-	;; we may have added earlier and which is now incorrect.
-	(remove-text-properties start end '(composition))
-      ;; That's a symbol alright, so add the composition.
-      (compose-region start end (cdr (assoc (match-string 0)
-                                            perl--prettify-symbols-alist)))))
-  ;; Return nil because we're not adding any face property.
-  nil)
-
-(defun perl--font-lock-symbols-keywords ()
-  (when perl-prettify-symbols
-    `((,(regexp-opt (mapcar 'car perl--prettify-symbols-alist) t)
-       (0 (perl--font-lock-compose-symbol))))))
+  `((basic ,@perl--prettify-symbols-alist-basic)
+    (extended ,@perl--prettify-symbols-alist-extended)
+    (all ,@perl--prettify-symbols-alist-extended)))
 
 (defconst perl-font-lock-keywords-1
   '(;; What is this for?
@@ -243,13 +220,17 @@
      ;; Fontify keywords with/and labels as we do in `c++-font-lock-keywords'.
      ("\\<\\(continue\\|goto\\|last\\|next\\|redo\\)\\>[ \t]*\\(\\sw+\\)?"
       (1 font-lock-keyword-face) (2 font-lock-constant-face nil t))
-     ("^[ \t]*\\(\\sw+\\)[ \t]*:[^:]" 1 font-lock-constant-face)
-     ,@(perl--font-lock-symbols-keywords)))
+     ("^[ \t]*\\(\\sw+\\)[ \t]*:[^:]" 1 font-lock-constant-face)))
   "Gaudy level highlighting for Perl mode.")
 
 (defvar perl-font-lock-keywords perl-font-lock-keywords-1
   "Default expressions to highlight in Perl mode.")
 
+;; used to add font-lock keywords dynamically
+(defvar perl-augmented-font-lock-keywords)
+(defvar perl-augmented-font-lock-keywords-1)
+(defvar perl-augmented-font-lock-keywords-2)
+
 (defvar perl-quote-like-pairs
   '((?\( . ?\)) (?\[ . ?\]) (?\{ . ?\}) (?\< . ?\>)))
 
@@ -685,11 +666,22 @@
   (setq-local comment-start-skip "\\(^\\|\\s-\\);?#+ *")
   (setq-local comment-indent-function #'perl-comment-indent)
   (setq-local parse-sexp-ignore-comments t)
+(setq prog-prettify-symbols 'all)
+  ;; Define the symbols to be prettified
+  (setq-local prog-prettify-symbols-alist perl--prettify-symbols-alist)
+
   ;; Tell font-lock.el how to handle Perl.
-  (setq font-lock-defaults '((perl-font-lock-keywords
-			      perl-font-lock-keywords-1
-			      perl-font-lock-keywords-2)
-			     nil nil ((?\_ . "w")) nil
+  (setq perl-augmented-font-lock-keywords (append perl-font-lock-keywords
+                                         (prog-font-lock-symbols-keywords)))
+  (setq perl-augmented-font-lock-keywords-1 (append perl-font-lock-keywords-1
+                                         (prog-font-lock-symbols-keywords)))
+  (setq perl-augmented-font-lock-keywords-2 (append perl-font-lock-keywords-2
+                                         (prog-font-lock-symbols-keywords)))
+
+  (setq font-lock-defaults '((perl-augmented-font-lock-keywords
+                              perl-augmented-font-lock-keywords-1
+                              perl-augmented-font-lock-keywords-2)
+                             nil nil ((?\_ . "w")) nil
                              (font-lock-syntactic-face-function
                               . perl-font-lock-syntactic-face-function)))
   (setq-local syntax-propertize-function #'perl-syntax-propertize-function)

=== modified file 'lisp/simple.el'
--- lisp/simple.el	2013-05-25 02:21:49 +0000
+++ lisp/simple.el	2013-06-02 04:03:20 +0000
@@ -397,9 +397,51 @@
   "Major mode for editing programming language source code."
   (set (make-local-variable 'require-final-newline) mode-require-final-newline)
   (set (make-local-variable 'parse-sexp-ignore-comments) t)
+  (set (make-local-variable 'prog-prettify-symbols-alist) nil)
   ;; Any programming language is always written left to right.
   (setq bidi-paragraph-direction 'left-to-right))
 
+(defcustom prog-prettify-symbols nil
+  "Which symbols should be prettified.
+When set to `basic' or `extended' or `all', the actual choices
+are made by the mode that derives from `prog-mode'."
+  :type '(choice
+          (const :tag "No thanks" nil)
+          (const :tag "Basic list" basic)
+          (const :tag "Extended list: basic plus much more" extended)
+          (const :tag "Everything: because you can!" all)
+          (alist :tag "Define your own list" :key-type string :value-type character))
+  :group 'languages)
+
+(defun prog--font-lock-compose-symbol (alist)
+  "Compose a sequence of ascii chars into a symbol.
+Regexp match data 0 points to the chars."
+  ;; Check that the chars should really be composed into a symbol.
+  (let* ((start (match-beginning 0))
+	 (end (match-end 0))
+	 (syntaxes (if (eq (char-syntax (char-after start)) ?w)
+		       '(?w) '(?. ?\\))))
+    (if (or (memq (char-syntax (or (char-before start) ?\ )) syntaxes)
+	    (memq (char-syntax (or (char-after end) ?\ )) syntaxes)
+            (nth 8 (syntax-ppss)))
+	;; No composition for you.  Let's actually remove any composition
+	;; we may have added earlier and which is now incorrect.
+	(remove-text-properties start end '(composition))
+      ;; That's a symbol alright, so add the composition.
+      (compose-region start end (cdr (assoc (match-string 0) alist)))))
+  ;; Return nil because we're not adding any face property.
+  nil)
+
+(defun prog-font-lock-symbols-keywords ()
+  (when prog-prettify-symbols
+    (let ((alist (if (symbolp prog-prettify-symbols)
+                     ;; the cdr of the cons cell has all we need
+                     (cdr (assoc prog-prettify-symbols
+                                 prog-prettify-symbols-alist))
+                   prog-prettify-symbols)))
+    `((,(regexp-opt (mapcar 'car alist) t)
+       (0 (prog--font-lock-compose-symbol ,alist)))))))
+
 ;; Making and deleting lines.
 
 (defvar hard-newline (propertize "\n" 'hard t 'rear-nonsticky '(hard))

Re: using glyphs by default in perl-mode

[Stefan Monnier]

> I have the data working and AFAICT the correct data is passed to
> font-lock but something is going wrong inside the fontification call.
> That's hard to debug.  I'll work on it more (I think you posted a trick
> for debugging font-lock a while ago) but if anyone can spot the issue, I
> would appreciate any suggestions.

(setq font-lock-support-mode nil)

and then try again (e.g. turn font-lock off and then on).


        Stefan

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Sun, 02 Jun 2013 11:52:51 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I have the data working and AFAICT the correct data is passed to
>> font-lock but something is going wrong inside the fontification call.
>> That's hard to debug.  I'll work on it more (I think you posted a trick
>> for debugging font-lock a while ago) but if anyone can spot the issue, I
>> would appreciate any suggestions.

SM> (setq font-lock-support-mode nil)

SM> and then try again (e.g. turn font-lock off and then on).

OK, please see attached.  It works!  I was missing a quote in the macro expansion.

Done:
- `prog-mode' now has the functions to do font-lock symbol composition

- the user's defcustom is `prog-prettify-symbols' which can be 'basic,
'extended, 'all, or an alist.  If it's a symbol it's looked up in the
buffer-local `prog-prettify-symbols-alist'.

- every mode simply binds `prog-prettify-symbols-alist' to an alist with
key = 'basic,'extended, and 'all.  The value is an alist of `(string . character)'.

Then the mode can use the output of
`prog-prettify-font-lock-symbols-keywords' in its font-lock keywords.
The attached patch shows it for `perl-mode' but other modes should be
simpler.

Still TODO:

- move all `prog-mode' code to prog.el

- support Haskell, Lisp pretty-lambdas, sml-mode, and cfengine-mode

- maybe support images as well, as a path or inlined?  Not sure.

- can the alist mechanism above be made simpler for every mode?

Please review the patch, especially the defcustoms.  Once you think it's
OK, I'll move to prog.el and commit (which will turn off the prettified
symbols by default in `perl-mode', the most urgent item).  Then I can go
down the TODO list.

Thanks!
Ted

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 8 bytes --]

+patch


[-- Attachment #2: prog-prettify3.patch --]
[-- Type: text/x-diff, Size: 7654 bytes --]

=== modified file 'lisp/progmodes/perl-mode.el'
--- lisp/progmodes/perl-mode.el	2013-05-07 18:06:13 +0000
+++ lisp/progmodes/perl-mode.el	2013-06-03 01:48:28 +0000
@@ -158,44 +158,21 @@
 ;; Regexps updated with help from Tom Tromey <tromey@cambric.colorado.edu> and
 ;; Jim Campbell <jec@murzim.ca.boeing.com>.
 
-(defcustom perl-prettify-symbols t
-  "If non-nil, some symbols will be displayed using Unicode chars."
-  :version "24.4"
-  :type 'boolean)
+(defconst perl--prettify-symbols-alist-basic
+  '(("->"  . ?→)
+    ("=>"  . ?⇒)
+    ("::" . ?∷)))
+
+(defconst perl--prettify-symbols-alist-extended
+  `(,@perl--prettify-symbols-alist-basic
+    ("andalso" . ?∧) ("orelse"  . ?∨) ("as" . ?≡)("not" . ?¬)
+    ("div" . ?÷) ("*"   . ?×) ("o"   . ?○)
+    ("<-"  . ?←) ("<>"  . ?≠) (">="  . ?≥) ("<="  . ?≤) ("..." . ?⋯)))
 
 (defconst perl--prettify-symbols-alist
-  '(;;("andalso" . ?∧) ("orelse"  . ?∨) ("as" . ?≡)("not" . ?¬)
-    ;;("div" . ?÷) ("*"   . ?×) ("o"   . ?○)
-    ("->"  . ?→)
-    ("=>"  . ?⇒)
-    ;;("<-"  . ?←) ("<>"  . ?≠) (">="  . ?≥) ("<="  . ?≤) ("..." . ?⋯)
-    ("::" . ?∷)
-    ))
-
-(defun perl--font-lock-compose-symbol ()
-  "Compose a sequence of ascii chars into a symbol.
-Regexp match data 0 points to the chars."
-  ;; Check that the chars should really be composed into a symbol.
-  (let* ((start (match-beginning 0))
-	 (end (match-end 0))
-	 (syntaxes (if (eq (char-syntax (char-after start)) ?w)
-		       '(?w) '(?. ?\\))))
-    (if (or (memq (char-syntax (or (char-before start) ?\ )) syntaxes)
-	    (memq (char-syntax (or (char-after end) ?\ )) syntaxes)
-            (nth 8 (syntax-ppss)))
-	;; No composition for you.  Let's actually remove any composition
-	;; we may have added earlier and which is now incorrect.
-	(remove-text-properties start end '(composition))
-      ;; That's a symbol alright, so add the composition.
-      (compose-region start end (cdr (assoc (match-string 0)
-                                            perl--prettify-symbols-alist)))))
-  ;; Return nil because we're not adding any face property.
-  nil)
-
-(defun perl--font-lock-symbols-keywords ()
-  (when perl-prettify-symbols
-    `((,(regexp-opt (mapcar 'car perl--prettify-symbols-alist) t)
-       (0 (perl--font-lock-compose-symbol))))))
+  `((basic ,@perl--prettify-symbols-alist-basic)
+    (extended ,@perl--prettify-symbols-alist-extended)
+    (all ,@perl--prettify-symbols-alist-extended)))
 
 (defconst perl-font-lock-keywords-1
   '(;; What is this for?
@@ -243,13 +220,17 @@
      ;; Fontify keywords with/and labels as we do in `c++-font-lock-keywords'.
      ("\\<\\(continue\\|goto\\|last\\|next\\|redo\\)\\>[ \t]*\\(\\sw+\\)?"
       (1 font-lock-keyword-face) (2 font-lock-constant-face nil t))
-     ("^[ \t]*\\(\\sw+\\)[ \t]*:[^:]" 1 font-lock-constant-face)
-     ,@(perl--font-lock-symbols-keywords)))
+     ("^[ \t]*\\(\\sw+\\)[ \t]*:[^:]" 1 font-lock-constant-face)))
   "Gaudy level highlighting for Perl mode.")
 
 (defvar perl-font-lock-keywords perl-font-lock-keywords-1
   "Default expressions to highlight in Perl mode.")
 
+;; used to add font-lock keywords dynamically
+(defvar perl-augmented-font-lock-keywords)
+(defvar perl-augmented-font-lock-keywords-1)
+(defvar perl-augmented-font-lock-keywords-2)
+
 (defvar perl-quote-like-pairs
   '((?\( . ?\)) (?\[ . ?\]) (?\{ . ?\}) (?\< . ?\>)))
 
@@ -685,11 +666,22 @@
   (setq-local comment-start-skip "\\(^\\|\\s-\\);?#+ *")
   (setq-local comment-indent-function #'perl-comment-indent)
   (setq-local parse-sexp-ignore-comments t)
+
+  ;; Define the symbols to be prettified
+  (setq-local prog-prettify-symbols-alist perl--prettify-symbols-alist)
+
   ;; Tell font-lock.el how to handle Perl.
-  (setq font-lock-defaults '((perl-font-lock-keywords
-			      perl-font-lock-keywords-1
-			      perl-font-lock-keywords-2)
-			     nil nil ((?\_ . "w")) nil
+  (setq perl-augmented-font-lock-keywords (append perl-font-lock-keywords
+                                         (prog-prettify-font-lock-symbols-keywords)))
+  (setq perl-augmented-font-lock-keywords-1 (append perl-font-lock-keywords-1
+                                         (prog-prettify-font-lock-symbols-keywords)))
+  (setq perl-augmented-font-lock-keywords-2 (append perl-font-lock-keywords-2
+                                         (prog-prettify-font-lock-symbols-keywords)))
+
+  (setq font-lock-defaults '((perl-augmented-font-lock-keywords
+                              perl-augmented-font-lock-keywords-1
+                              perl-augmented-font-lock-keywords-2)
+                             nil nil ((?\_ . "w")) nil
                              (font-lock-syntactic-face-function
                               . perl-font-lock-syntactic-face-function)))
   (setq-local syntax-propertize-function #'perl-syntax-propertize-function)

=== modified file 'lisp/simple.el'
--- lisp/simple.el	2013-05-25 02:21:49 +0000
+++ lisp/simple.el	2013-06-03 01:11:54 +0000
@@ -393,13 +393,56 @@
 	  (end (progn (forward-sexp 1) (point))))
       (indent-region start end nil))))
 
+;; TODO: move to prog-mode.el
 (define-derived-mode prog-mode fundamental-mode "Prog"
   "Major mode for editing programming language source code."
   (set (make-local-variable 'require-final-newline) mode-require-final-newline)
   (set (make-local-variable 'parse-sexp-ignore-comments) t)
+  (set (make-local-variable 'prog-prettify-symbols-alist) nil)
   ;; Any programming language is always written left to right.
   (setq bidi-paragraph-direction 'left-to-right))
 
+(defcustom prog-prettify-symbols nil
+  "Which symbols should be prettified.
+When set to `basic' or `extended' or `all', the actual choices
+are made by the mode that derives from `prog-mode'."
+  :type '(choice
+          (const :tag "No thanks" nil)
+          (const :tag "Basic list" basic)
+          (const :tag "Extended list: basic plus much more" extended)
+          (const :tag "Everything: because you can!" all)
+          (alist :tag "Define your own list" :key-type string :value-type character))
+  :group 'languages)
+
+(defun prog--prettify-font-lock-compose-symbol (alist)
+  "Compose a sequence of ascii chars into a symbol.
+Regexp match data 0 points to the chars."
+  ;; Check that the chars should really be composed into a symbol.
+  (let* ((start (match-beginning 0))
+	 (end (match-end 0))
+	 (syntaxes (if (eq (char-syntax (char-after start)) ?w)
+		       '(?w) '(?. ?\\))))
+    (if (or (memq (char-syntax (or (char-before start) ?\ )) syntaxes)
+	    (memq (char-syntax (or (char-after end) ?\ )) syntaxes)
+            (nth 8 (syntax-ppss)))
+	;; No composition for you.  Let's actually remove any composition
+	;; we may have added earlier and which is now incorrect.
+	(remove-text-properties start end '(composition))
+      ;; That's a symbol alright, so add the composition.
+      (compose-region start end (cdr (assoc (match-string 0) alist)))))
+  ;; Return nil because we're not adding any face property.
+  nil)
+
+(defun prog-prettify-font-lock-symbols-keywords ()
+  (when prog-prettify-symbols
+    (let ((alist (if (symbolp prog-prettify-symbols)
+                     ;; the cdr of the cons cell has all we need
+                     (cdr (assoc prog-prettify-symbols
+                                 prog-prettify-symbols-alist))
+                   prog-prettify-symbols)))
+    `((,(regexp-opt (mapcar 'car alist) t)
+       (0 (prog--prettify-font-lock-compose-symbol ',alist)))))))
+
 ;; Making and deleting lines.
 
 (defvar hard-newline (propertize "\n" 'hard t 'rear-nonsticky '(hard))

Re: using glyphs by default in perl-mode

[Stefan Monnier]

Please use prog-mode.el rather than prog.el.

> +(defconst perl--prettify-symbols-alist-basic
> +  '(("->"  . ?→)
> +    ("=>"  . ?⇒)
> +    ("::" . ?∷)))

I don't think "basic" and "extended" make much sense currently, so the
basic user setting should be a boolean.  Major modes should simply
provide *the* alist that makes sense for them.

> +(defconst perl--prettify-symbols-alist-extended
> +  `(,@perl--prettify-symbols-alist-basic
> +    ("andalso" . ?∧) ("orelse"  . ?∨) ("as" . ?≡)("not" . ?¬)
> +    ("div" . ?÷) ("*"   . ?×) ("o"   . ?○)
> +    ("<-"  . ?←) ("<>"  . ?≠) (">="  . ?≥) ("<="  . ?≤) ("..." . ?⋯)))

E.g. this makes very little sense for Perl, since these mapping all come
from sml-mode.el ;-)

> +;; used to add font-lock keywords dynamically
> +(defvar perl-augmented-font-lock-keywords)
> +(defvar perl-augmented-font-lock-keywords-1)
> +(defvar perl-augmented-font-lock-keywords-2)

This smells internal, so please use "--" in their name.

> +  (set (make-local-variable 'prog-prettify-symbols-alist) nil)

Just make the default value be nil instead.

> +(defcustom prog-prettify-symbols nil
> +  "Which symbols should be prettified.
> +When set to `basic' or `extended' or `all', the actual choices
> +are made by the mode that derives from `prog-mode'."
> +  :type '(choice
> +          (const :tag "No thanks" nil)
> +          (const :tag "Basic list" basic)
> +          (const :tag "Extended list: basic plus much more" extended)
> +          (const :tag "Everything: because you can!" all)
> +          (alist :tag "Define your own list" :key-type string :value-type character))
> +  :group 'languages)

Make it into a boolean.

Other than that, looks good, please install.


        Stefan

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 2415 bytes --]

On Mon, 03 Jun 2013 03:04:41 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> Please use prog-mode.el rather than prog.el.
>> +(defconst perl--prettify-symbols-alist-basic
>> +  '(("->"  . ?→)
>> +    ("=>"  . ?⇒)
>> +    ("::" . ?∷)))

SM> I don't think "basic" and "extended" make much sense currently, so the
SM> basic user setting should be a boolean.  Major modes should simply
SM> provide *the* alist that makes sense for them.

OK.

>> +(defconst perl--prettify-symbols-alist-extended
>> +  `(,@perl--prettify-symbols-alist-basic
>> +    ("andalso" . ?∧) ("orelse"  . ?∨) ("as" . ?≡)("not" . ?¬)
>> +    ("div" . ?÷) ("*"   . ?×) ("o"   . ?○)
>> +    ("<-"  . ?←) ("<>"  . ?≠) (">="  . ?≥) ("<="  . ?≤) ("..." . ?⋯)))

SM> E.g. this makes very little sense for Perl, since these mapping all come
SM> from sml-mode.el ;-)

Yeah, I didn't pay attention.  OK.

>> +;; used to add font-lock keywords dynamically
>> +(defvar perl-augmented-font-lock-keywords)
>> +(defvar perl-augmented-font-lock-keywords-1)
>> +(defvar perl-augmented-font-lock-keywords-2)

SM> This smells internal, so please use "--" in their name.

OK.

>> +  (set (make-local-variable 'prog-prettify-symbols-alist) nil)

SM> Just make the default value be nil instead.

OK.

>> +(defcustom prog-prettify-symbols nil
>> +  "Which symbols should be prettified.
>> +When set to `basic' or `extended' or `all', the actual choices
>> +are made by the mode that derives from `prog-mode'."
>> +  :type '(choice
>> +          (const :tag "No thanks" nil)
>> +          (const :tag "Basic list" basic)
>> +          (const :tag "Extended list: basic plus much more" extended)
>> +          (const :tag "Everything: because you can!" all)
>> +          (alist :tag "Define your own list" :key-type string :value-type character))
>> +  :group 'languages)

SM> Make it into a boolean.

SM> Other than that, looks good, please install.

Please take a final look.  I added this to emacs-lisp-mode (just
lambda), perl-mode, and cfengine3-mode.  It's easy to do this once you
start doing it but it still feels like an annoying amount of work.

It also feels like it should support arbitrary strings (to compose
Unicode) and even images.  WDYT?  Too much?

I couldn't move the code to prog-mode.el.  The emacs-lisp-mode include
needs the prog-mode map and barfed all over.  Sorry.  But it should be
OK otherwise.

Ted


[-- Attachment #2: prog-prettify4.patch --]
[-- Type: text/x-diff, Size: 10394 bytes --]

=== modified file 'lisp/emacs-lisp/lisp-mode.el'
--- lisp/emacs-lisp/lisp-mode.el	2013-05-29 14:14:16 +0000
+++ lisp/emacs-lisp/lisp-mode.el	2013-06-04 01:11:36 +0000
@@ -187,6 +187,11 @@
               font-lock-string-face))))
     font-lock-comment-face))
 
+;; used to add font-lock keywords dynamically
+(defvar lisp--augmented-font-lock-keywords)
+(defvar lisp--augmented-font-lock-keywords-1)
+(defvar lisp--augmented-font-lock-keywords-2)
+
 (defun lisp-mode-variables (&optional lisp-syntax keywords-case-insensitive)
   "Common initialization routine for lisp modes.
 The LISP-SYNTAX argument is used by code in inf-lisp.el and is
@@ -223,9 +228,20 @@
   (setq-local imenu-generic-expression lisp-imenu-generic-expression)
   (setq-local multibyte-syntax-as-symbol t)
   (setq-local syntax-begin-function 'beginning-of-defun)
+  (setq-local prog-prettify-symbols-alist lisp--prettify-symbols-alist)
+  (setq lisp--augmented-font-lock-keywords
+        (append lisp-font-lock-keywords
+                (prog-prettify-font-lock-symbols-keywords)))
+  (setq lisp--augmented-font-lock-keywords-1
+        (append lisp-font-lock-keywords-1
+                (prog-prettify-font-lock-symbols-keywords)))
+  (setq lisp--augmented-font-lock-keywords-2
+        (append lisp-font-lock-keywords-2
+                (prog-prettify-font-lock-symbols-keywords)))
   (setq font-lock-defaults
-	`((lisp-font-lock-keywords
-	   lisp-font-lock-keywords-1 lisp-font-lock-keywords-2)
+	`((lisp--augmented-font-lock-keywords
+	   lisp--augmented-font-lock-keywords-1
+           lisp--augmented-font-lock-keywords-2)
 	  nil ,keywords-case-insensitive nil nil
 	  (font-lock-mark-block-function . mark-defun)
 	  (font-lock-syntactic-face-function
@@ -448,6 +464,9 @@
   :type 'hook
   :group 'lisp)
 
+(defconst lisp--prettify-symbols-alist
+  `(("lambda"  . ,(make-char 'greek-iso8859-7 107))))
+
 (define-derived-mode emacs-lisp-mode prog-mode "Emacs-Lisp"
   "Major mode for editing Lisp code to run in Emacs.
 Commands:

=== modified file 'lisp/progmodes/cfengine.el'
--- lisp/progmodes/cfengine.el	2013-03-22 19:06:53 +0000
+++ lisp/progmodes/cfengine.el	2013-06-04 01:43:47 +0000
@@ -527,6 +527,17 @@
   ;; Doze path separators.
   (modify-syntax-entry ?\\ "." table))
 
+(defconst cfengine3--prettify-symbols-alist
+  '(("->"  . ?→)
+    ("=>"  . ?⇒)
+    ("::" . ?∷)
+    ("bundle agent" . ?⊕)
+    ("bundle server" . ?∑)
+    ("body" . ?⊙)
+    ("bundle" . ?𝛽)))
+
+(defvar cfengine3--augmented-font-lock-keywords)
+
 ;;;###autoload
 (define-derived-mode cfengine3-mode prog-mode "CFE3"
   "Major mode for editing CFEngine3 input.
@@ -538,8 +549,18 @@
   (cfengine-common-syntax cfengine3-mode-syntax-table)
 
   (set (make-local-variable 'indent-line-function) #'cfengine3-indent-line)
+
+  ;; Define the symbols to be prettified
+  (setq-local prog-prettify-symbols-alist cfengine3--prettify-symbols-alist)
+
+  ;; Tell font-lock.el how to handle Perl.
+  (setq cfengine3--augmented-font-lock-keywords
+        (append cfengine3-font-lock-keywords
+                (prog-prettify-font-lock-symbols-keywords)))
+
   (setq font-lock-defaults
-        '(cfengine3-font-lock-keywords nil nil nil beginning-of-defun))
+        '(cfengine3--augmented-font-lock-keywords
+          nil nil nil beginning-of-defun))
 
   ;; Use defuns as the essential syntax block.
   (set (make-local-variable 'beginning-of-defun-function)

=== modified file 'lisp/progmodes/perl-mode.el'
--- lisp/progmodes/perl-mode.el	2013-05-07 18:06:13 +0000
+++ lisp/progmodes/perl-mode.el	2013-06-04 00:44:35 +0000
@@ -158,44 +158,10 @@
 ;; Regexps updated with help from Tom Tromey <tromey@cambric.colorado.edu> and
 ;; Jim Campbell <jec@murzim.ca.boeing.com>.
 
-(defcustom perl-prettify-symbols t
-  "If non-nil, some symbols will be displayed using Unicode chars."
-  :version "24.4"
-  :type 'boolean)
-
 (defconst perl--prettify-symbols-alist
-  '(;;("andalso" . ?∧) ("orelse"  . ?∨) ("as" . ?≡)("not" . ?¬)
-    ;;("div" . ?÷) ("*"   . ?×) ("o"   . ?○)
-    ("->"  . ?→)
+  '(("->"  . ?→)
     ("=>"  . ?⇒)
-    ;;("<-"  . ?←) ("<>"  . ?≠) (">="  . ?≥) ("<="  . ?≤) ("..." . ?⋯)
-    ("::" . ?∷)
-    ))
-
-(defun perl--font-lock-compose-symbol ()
-  "Compose a sequence of ascii chars into a symbol.
-Regexp match data 0 points to the chars."
-  ;; Check that the chars should really be composed into a symbol.
-  (let* ((start (match-beginning 0))
-	 (end (match-end 0))
-	 (syntaxes (if (eq (char-syntax (char-after start)) ?w)
-		       '(?w) '(?. ?\\))))
-    (if (or (memq (char-syntax (or (char-before start) ?\ )) syntaxes)
-	    (memq (char-syntax (or (char-after end) ?\ )) syntaxes)
-            (nth 8 (syntax-ppss)))
-	;; No composition for you.  Let's actually remove any composition
-	;; we may have added earlier and which is now incorrect.
-	(remove-text-properties start end '(composition))
-      ;; That's a symbol alright, so add the composition.
-      (compose-region start end (cdr (assoc (match-string 0)
-                                            perl--prettify-symbols-alist)))))
-  ;; Return nil because we're not adding any face property.
-  nil)
-
-(defun perl--font-lock-symbols-keywords ()
-  (when perl-prettify-symbols
-    `((,(regexp-opt (mapcar 'car perl--prettify-symbols-alist) t)
-       (0 (perl--font-lock-compose-symbol))))))
+    ("::" . ?∷)))
 
 (defconst perl-font-lock-keywords-1
   '(;; What is this for?
@@ -243,13 +209,17 @@
      ;; Fontify keywords with/and labels as we do in `c++-font-lock-keywords'.
      ("\\<\\(continue\\|goto\\|last\\|next\\|redo\\)\\>[ \t]*\\(\\sw+\\)?"
       (1 font-lock-keyword-face) (2 font-lock-constant-face nil t))
-     ("^[ \t]*\\(\\sw+\\)[ \t]*:[^:]" 1 font-lock-constant-face)
-     ,@(perl--font-lock-symbols-keywords)))
+     ("^[ \t]*\\(\\sw+\\)[ \t]*:[^:]" 1 font-lock-constant-face)))
   "Gaudy level highlighting for Perl mode.")
 
 (defvar perl-font-lock-keywords perl-font-lock-keywords-1
   "Default expressions to highlight in Perl mode.")
 
+;; used to add font-lock keywords dynamically
+(defvar perl--augmented-font-lock-keywords)
+(defvar perl--augmented-font-lock-keywords-1)
+(defvar perl--augmented-font-lock-keywords-2)
+
 (defvar perl-quote-like-pairs
   '((?\( . ?\)) (?\[ . ?\]) (?\{ . ?\}) (?\< . ?\>)))
 
@@ -685,11 +655,25 @@
   (setq-local comment-start-skip "\\(^\\|\\s-\\);?#+ *")
   (setq-local comment-indent-function #'perl-comment-indent)
   (setq-local parse-sexp-ignore-comments t)
+
+  ;; Define the symbols to be prettified
+  (setq-local prog-prettify-symbols-alist perl--prettify-symbols-alist)
+
   ;; Tell font-lock.el how to handle Perl.
-  (setq font-lock-defaults '((perl-font-lock-keywords
-			      perl-font-lock-keywords-1
-			      perl-font-lock-keywords-2)
-			     nil nil ((?\_ . "w")) nil
+  (setq perl--augmented-font-lock-keywords
+        (append perl-font-lock-keywords
+                (prog-prettify-font-lock-symbols-keywords)))
+  (setq perl--augmented-font-lock-keywords-1
+        (append perl-font-lock-keywords-1
+                (prog-prettify-font-lock-symbols-keywords)))
+  (setq perl--augmented-font-lock-keywords-2
+        (append perl-font-lock-keywords-2
+                (prog-prettify-font-lock-symbols-keywords)))
+
+  (setq font-lock-defaults '((perl--augmented-font-lock-keywords
+                              perl--augmented-font-lock-keywords-1
+                              perl--augmented-font-lock-keywords-2)
+                             nil nil ((?\_ . "w")) nil
                              (font-lock-syntactic-face-function
                               . perl-font-lock-syntactic-face-function)))
   (setq-local syntax-propertize-function #'perl-syntax-propertize-function)

=== modified file 'lisp/simple.el'
--- lisp/simple.el	2013-05-25 02:21:49 +0000
+++ lisp/simple.el	2013-06-04 00:59:14 +0000
@@ -393,13 +393,56 @@
 	  (end (progn (forward-sexp 1) (point))))
       (indent-region start end nil))))
 
+;; TODO: move to prog-mode.el
 (define-derived-mode prog-mode fundamental-mode "Prog"
   "Major mode for editing programming language source code."
   (set (make-local-variable 'require-final-newline) mode-require-final-newline)
   (set (make-local-variable 'parse-sexp-ignore-comments) t)
+  (make-local-variable 'prog-prettify-symbols-alist)
   ;; Any programming language is always written left to right.
   (setq bidi-paragraph-direction 'left-to-right))
 
+(defvar prog-prettify-symbols-alist nil)
+
+(defcustom prog-prettify-symbols nil
+  "Which symbols should be prettified.
+When set to `basic' or `extended' or `all', the actual choices
+are made by the mode that derives from `prog-mode'."
+  :type '(choice
+          (const :tag "No thanks" nil)
+          (const :tag "Mode defaults" t)
+          (alist :tag "Mode defaults augmented with your own list"
+                 :key-type string :value-type character))
+  :group 'languages)
+
+(defun prog--prettify-font-lock-compose-symbol (alist)
+  "Compose a sequence of ascii chars into a symbol.
+Regexp match data 0 points to the chars."
+  ;; Check that the chars should really be composed into a symbol.
+  (let* ((start (match-beginning 0))
+	 (end (match-end 0))
+	 (syntaxes (if (eq (char-syntax (char-after start)) ?w)
+		       '(?w) '(?. ?\\))))
+    (if (or (memq (char-syntax (or (char-before start) ?\ )) syntaxes)
+	    (memq (char-syntax (or (char-after end) ?\ )) syntaxes)
+            (nth 8 (syntax-ppss)))
+	;; No composition for you.  Let's actually remove any composition
+	;; we may have added earlier and which is now incorrect.
+	(remove-text-properties start end '(composition))
+      ;; That's a symbol alright, so add the composition.
+      (compose-region start end (cdr (assoc (match-string 0) alist)))))
+  ;; Return nil because we're not adding any face property.
+  nil)
+
+(defun prog-prettify-font-lock-symbols-keywords ()
+  (when prog-prettify-symbols
+    (let ((alist (append prog-prettify-symbols-alist
+                         (if (listp prog-prettify-symbols)
+                             prog-prettify-symbols
+                           nil))))
+      `((,(regexp-opt (mapcar 'car alist) t)
+         (0 (prog--prettify-font-lock-compose-symbol ',alist)))))))
+
 ;; Making and deleting lines.
 
 (defvar hard-newline (propertize "\n" 'hard t 'rear-nonsticky '(hard))

Re: using glyphs by default in perl-mode

[Davis Herring]

> +When set to `basic' or `extended' or `all', the actual choices
> +are made by the mode that derives from `prog-mode'."

This was missed in the conversion to boolean.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: using glyphs by default in perl-mode

[Stefan Monnier]

> It's easy to do this once you start doing it but it still feels like
> an annoying amount of work.

We definitely need to add some helper functions to make it easier.
The major modes should really only need to do

  (setq-local prog-prettify-symbols-alist <blabla>)

and then

  (prog-prettify-setup-font-lock)

but we'll figure that out later.
  
> I couldn't move the code to prog-mode.el.

Fine.

> The emacs-lisp-mode include needs the prog-mode map and barfed all
> over.  Sorry.  But it should be OK otherwise.

prog-mode.el needs to be preloaded (i.e. added to loadup.el), indeed.

> +;; used to add font-lock keywords dynamically

Please capitalize and punctuate your comments.

> +(defconst lisp--prettify-symbols-alist
> +  `(("lambda"  . ,(make-char 'greek-iso8859-7 107))))

We'd probably want to refine it so it's only prettified when it comes
right after a parenthesis, but it's OK for now.  OTOH using make-char
here is a bad idea: better just put the Unicode char in there (this
make-char seems to be copied from old pre-Unicode code) either as ?λ
or as ?\u03BB.

> +    ("bundle agent" . ?⊕)
> +    ("bundle server" . ?∑)
> +    ("body" . ?⊙)
> +    ("bundle" . ?𝛽)))

Interesting!


        Stefan

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 2158 bytes --]

On Tue, 04 Jun 2013 13:20:28 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> It's easy to do this once you start doing it but it still feels like
>> an annoying amount of work.

SM> We definitely need to add some helper functions to make it easier.
SM> The major modes should really only need to do

SM>   (setq-local prog-prettify-symbols-alist <blabla>)

SM> and then

SM>   (prog-prettify-setup-font-lock)

SM> but we'll figure that out later.

OK :)
  
>> I couldn't move the code to prog-mode.el.

SM> Fine.

>> The emacs-lisp-mode include needs the prog-mode map and barfed all
>> over.  Sorry.  But it should be OK otherwise.

SM> prog-mode.el needs to be preloaded (i.e. added to loadup.el), indeed.

See attached patch.  It only fixes comments and attempts to move to
prog-mode.el.  Maybe you'll know the way to fix it.  I get:

Attempt to autoload define-derived-mode while preparing to dump

If it's trivial to fix, please let me know how.  If not, I'll commit as
simple.el and update the ChangeLog.

>> +;; used to add font-lock keywords dynamically

SM> Please capitalize and punctuate your comments.

Done.

>> +(defconst lisp--prettify-symbols-alist
>> +  `(("lambda"  . ,(make-char 'greek-iso8859-7 107))))

SM> We'd probably want to refine it so it's only prettified when it comes
SM> right after a parenthesis, but it's OK for now.  OTOH using make-char
SM> here is a bad idea: better just put the Unicode char in there (this
SM> make-char seems to be copied from old pre-Unicode code) either as ?λ
SM> or as ?\u03BB.

Done.

>> +    ("bundle agent" . ?⊕)
>> +    ("bundle server" . ?∑)
>> +    ("body" . ?⊙)
>> +    ("bundle" . ?𝛽)))

SM> Interesting!

That's exactly what the CFEngine guys said :)  Reverted to just arrows
and :: prettification, although I may add more to reflect the Promise
Theory notation Mark Burgess uses.

On Tue, 04 Jun 2013 08:28:42 -0600 Davis Herring <herring@lanl.gov> wrote: 

>> +When set to `basic' or `extended' or `all', the actual choices
>> +are made by the mode that derives from `prog-mode'."

DH> This was missed in the conversion to boolean.

Thanks, nice catch.  Fixed.

Ted


[-- Attachment #2: prog-prettify5.patch --]
[-- Type: text/x-diff, Size: 13297 bytes --]

=== modified file 'lisp/emacs-lisp/lisp-mode.el'
--- lisp/emacs-lisp/lisp-mode.el	2013-05-29 14:14:16 +0000
+++ lisp/emacs-lisp/lisp-mode.el	2013-06-04 19:53:24 +0000
@@ -187,6 +187,11 @@
               font-lock-string-face))))
     font-lock-comment-face))
 
+;; Temporary variables used to add font-lock keywords dynamically.
+(defvar lisp--augmented-font-lock-keywords)
+(defvar lisp--augmented-font-lock-keywords-1)
+(defvar lisp--augmented-font-lock-keywords-2)
+
 (defun lisp-mode-variables (&optional lisp-syntax keywords-case-insensitive)
   "Common initialization routine for lisp modes.
 The LISP-SYNTAX argument is used by code in inf-lisp.el and is
@@ -223,9 +228,20 @@
   (setq-local imenu-generic-expression lisp-imenu-generic-expression)
   (setq-local multibyte-syntax-as-symbol t)
   (setq-local syntax-begin-function 'beginning-of-defun)
+  (setq-local prog-prettify-symbols-alist lisp--prettify-symbols-alist)
+  (setq lisp--augmented-font-lock-keywords
+        (append lisp-font-lock-keywords
+                (prog-prettify-font-lock-symbols-keywords)))
+  (setq lisp--augmented-font-lock-keywords-1
+        (append lisp-font-lock-keywords-1
+                (prog-prettify-font-lock-symbols-keywords)))
+  (setq lisp--augmented-font-lock-keywords-2
+        (append lisp-font-lock-keywords-2
+                (prog-prettify-font-lock-symbols-keywords)))
   (setq font-lock-defaults
-	`((lisp-font-lock-keywords
-	   lisp-font-lock-keywords-1 lisp-font-lock-keywords-2)
+	`((lisp--augmented-font-lock-keywords
+	   lisp--augmented-font-lock-keywords-1
+           lisp--augmented-font-lock-keywords-2)
 	  nil ,keywords-case-insensitive nil nil
 	  (font-lock-mark-block-function . mark-defun)
 	  (font-lock-syntactic-face-function
@@ -448,6 +464,9 @@
   :type 'hook
   :group 'lisp)
 
+(defconst lisp--prettify-symbols-alist
+  '(("lambda"  . ?λ)))
+
 (define-derived-mode emacs-lisp-mode prog-mode "Emacs-Lisp"
   "Major mode for editing Lisp code to run in Emacs.
 Commands:

=== modified file 'lisp/loadup.el'
--- lisp/loadup.el	2013-05-16 09:58:56 +0000
+++ lisp/loadup.el	2013-06-04 20:01:08 +0000
@@ -206,6 +206,7 @@
 (load "rfn-eshadow")
 
 (load "menu-bar")
+(load "prog-mode")
 (load "emacs-lisp/lisp")
 (load "textmodes/page")
 (load "register")

=== added file 'lisp/prog-mode.el'
--- lisp/prog-mode.el	1970-01-01 00:00:00 +0000
+++ lisp/prog-mode.el	2013-06-04 19:57:53 +0000
@@ -0,0 +1,99 @@
+;;; prog-mode.el --- basic programming support for Emacs  -*- lexical-binding: t -*-
+
+;; Copyright (C) 2013 Free Software Foundation, Inc.
+
+;; Maintainer: FSF
+;; Keywords: internal,languages
+;; Package: emacs
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Major mode meant to be the parent of programming modes.
+
+;;; Code:
+
+(defvar prog-mode-map
+  (let ((map (make-sparse-keymap)))
+    (define-key map [?\C-\M-q] 'prog-indent-sexp)
+    map)
+  "Keymap used for programming modes.")
+
+(defun prog-indent-sexp (&optional defun)
+  "Indent the expression after point.
+When interactively called with prefix, indent the enclosing defun
+instead."
+  (interactive "P")
+  (save-excursion
+    (when defun
+      (end-of-line)
+      (beginning-of-defun))
+    (let ((start (point))
+	  (end (progn (forward-sexp 1) (point))))
+      (indent-region start end nil))))
+
+(define-derived-mode prog-mode fundamental-mode "Prog"
+  "Major mode for editing programming language source code."
+  (set (make-local-variable 'require-final-newline) mode-require-final-newline)
+  (set (make-local-variable 'parse-sexp-ignore-comments) t)
+  (make-local-variable 'prog-prettify-symbols-alist)
+  ;; Any programming language is always written left to right.
+  (setq bidi-paragraph-direction 'left-to-right))
+
+(defvar prog-prettify-symbols-alist nil)
+
+(defcustom prog-prettify-symbols nil
+  "Whether symbols should be prettified.
+When set to an alist in the form `(STRING . CHARACTER)' it will
+augment the mode's native prettify alist."
+  :type '(choice
+          (const :tag "No thanks" nil)
+          (const :tag "Mode defaults" t)
+          (alist :tag "Mode defaults augmented with your own list"
+                 :key-type string :value-type character))
+  :group 'languages)
+
+(defun prog--prettify-font-lock-compose-symbol (alist)
+  "Compose a sequence of ascii chars into a symbol.
+Regexp match data 0 points to the chars."
+  ;; Check that the chars should really be composed into a symbol.
+  (let* ((start (match-beginning 0))
+	 (end (match-end 0))
+	 (syntaxes (if (eq (char-syntax (char-after start)) ?w)
+		       '(?w) '(?. ?\\))))
+    (if (or (memq (char-syntax (or (char-before start) ?\ )) syntaxes)
+	    (memq (char-syntax (or (char-after end) ?\ )) syntaxes)
+            (nth 8 (syntax-ppss)))
+	;; No composition for you.  Let's actually remove any composition
+	;; we may have added earlier and which is now incorrect.
+	(remove-text-properties start end '(composition))
+      ;; That's a symbol alright, so add the composition.
+      (compose-region start end (cdr (assoc (match-string 0) alist)))))
+  ;; Return nil because we're not adding any face property.
+  nil)
+
+(defun prog-prettify-font-lock-symbols-keywords ()
+  (when prog-prettify-symbols
+    (let ((alist (append prog-prettify-symbols-alist
+                         (if (listp prog-prettify-symbols)
+                             prog-prettify-symbols
+                           nil))))
+      `((,(regexp-opt (mapcar 'car alist) t)
+         (0 (prog--prettify-font-lock-compose-symbol ',alist)))))))
+
+(provide 'prog-mode)
+;;; prog-mode.el ends here

=== modified file 'lisp/progmodes/cfengine.el'
--- lisp/progmodes/cfengine.el	2013-03-22 19:06:53 +0000
+++ lisp/progmodes/cfengine.el	2013-06-04 19:54:07 +0000
@@ -527,6 +527,13 @@
   ;; Doze path separators.
   (modify-syntax-entry ?\\ "." table))
 
+(defconst cfengine3--prettify-symbols-alist
+  '(("->"  . ?→)
+    ("=>"  . ?⇒)
+    ("::" . ?∷)))
+
+(defvar cfengine3--augmented-font-lock-keywords)
+
 ;;;###autoload
 (define-derived-mode cfengine3-mode prog-mode "CFE3"
   "Major mode for editing CFEngine3 input.
@@ -538,8 +545,18 @@
   (cfengine-common-syntax cfengine3-mode-syntax-table)
 
   (set (make-local-variable 'indent-line-function) #'cfengine3-indent-line)
+
+  ;; Define the symbols to be prettified
+  (setq-local prog-prettify-symbols-alist cfengine3--prettify-symbols-alist)
+
+  ;; Tell font-lock.el how to handle cfengine3 keywords..
+  (setq cfengine3--augmented-font-lock-keywords
+        (append cfengine3-font-lock-keywords
+                (prog-prettify-font-lock-symbols-keywords)))
+
   (setq font-lock-defaults
-        '(cfengine3-font-lock-keywords nil nil nil beginning-of-defun))
+        '(cfengine3--augmented-font-lock-keywords
+          nil nil nil beginning-of-defun))
 
   ;; Use defuns as the essential syntax block.
   (set (make-local-variable 'beginning-of-defun-function)

=== modified file 'lisp/progmodes/perl-mode.el'
--- lisp/progmodes/perl-mode.el	2013-05-07 18:06:13 +0000
+++ lisp/progmodes/perl-mode.el	2013-06-04 19:54:45 +0000
@@ -158,44 +158,10 @@
 ;; Regexps updated with help from Tom Tromey <tromey@cambric.colorado.edu> and
 ;; Jim Campbell <jec@murzim.ca.boeing.com>.
 
-(defcustom perl-prettify-symbols t
-  "If non-nil, some symbols will be displayed using Unicode chars."
-  :version "24.4"
-  :type 'boolean)
-
 (defconst perl--prettify-symbols-alist
-  '(;;("andalso" . ?∧) ("orelse"  . ?∨) ("as" . ?≡)("not" . ?¬)
-    ;;("div" . ?÷) ("*"   . ?×) ("o"   . ?○)
-    ("->"  . ?→)
+  '(("->"  . ?→)
     ("=>"  . ?⇒)
-    ;;("<-"  . ?←) ("<>"  . ?≠) (">="  . ?≥) ("<="  . ?≤) ("..." . ?⋯)
-    ("::" . ?∷)
-    ))
-
-(defun perl--font-lock-compose-symbol ()
-  "Compose a sequence of ascii chars into a symbol.
-Regexp match data 0 points to the chars."
-  ;; Check that the chars should really be composed into a symbol.
-  (let* ((start (match-beginning 0))
-	 (end (match-end 0))
-	 (syntaxes (if (eq (char-syntax (char-after start)) ?w)
-		       '(?w) '(?. ?\\))))
-    (if (or (memq (char-syntax (or (char-before start) ?\ )) syntaxes)
-	    (memq (char-syntax (or (char-after end) ?\ )) syntaxes)
-            (nth 8 (syntax-ppss)))
-	;; No composition for you.  Let's actually remove any composition
-	;; we may have added earlier and which is now incorrect.
-	(remove-text-properties start end '(composition))
-      ;; That's a symbol alright, so add the composition.
-      (compose-region start end (cdr (assoc (match-string 0)
-                                            perl--prettify-symbols-alist)))))
-  ;; Return nil because we're not adding any face property.
-  nil)
-
-(defun perl--font-lock-symbols-keywords ()
-  (when perl-prettify-symbols
-    `((,(regexp-opt (mapcar 'car perl--prettify-symbols-alist) t)
-       (0 (perl--font-lock-compose-symbol))))))
+    ("::" . ?∷)))
 
 (defconst perl-font-lock-keywords-1
   '(;; What is this for?
@@ -243,13 +209,17 @@
      ;; Fontify keywords with/and labels as we do in `c++-font-lock-keywords'.
      ("\\<\\(continue\\|goto\\|last\\|next\\|redo\\)\\>[ \t]*\\(\\sw+\\)?"
       (1 font-lock-keyword-face) (2 font-lock-constant-face nil t))
-     ("^[ \t]*\\(\\sw+\\)[ \t]*:[^:]" 1 font-lock-constant-face)
-     ,@(perl--font-lock-symbols-keywords)))
+     ("^[ \t]*\\(\\sw+\\)[ \t]*:[^:]" 1 font-lock-constant-face)))
   "Gaudy level highlighting for Perl mode.")
 
 (defvar perl-font-lock-keywords perl-font-lock-keywords-1
   "Default expressions to highlight in Perl mode.")
 
+;; Temporary variables used to add font-lock keywords dynamically.
+(defvar perl--augmented-font-lock-keywords)
+(defvar perl--augmented-font-lock-keywords-1)
+(defvar perl--augmented-font-lock-keywords-2)
+
 (defvar perl-quote-like-pairs
   '((?\( . ?\)) (?\[ . ?\]) (?\{ . ?\}) (?\< . ?\>)))
 
@@ -685,11 +655,25 @@
   (setq-local comment-start-skip "\\(^\\|\\s-\\);?#+ *")
   (setq-local comment-indent-function #'perl-comment-indent)
   (setq-local parse-sexp-ignore-comments t)
+
+  ;; Define the symbols to be prettified.
+  (setq-local prog-prettify-symbols-alist perl--prettify-symbols-alist)
+
   ;; Tell font-lock.el how to handle Perl.
-  (setq font-lock-defaults '((perl-font-lock-keywords
-			      perl-font-lock-keywords-1
-			      perl-font-lock-keywords-2)
-			     nil nil ((?\_ . "w")) nil
+  (setq perl--augmented-font-lock-keywords
+        (append perl-font-lock-keywords
+                (prog-prettify-font-lock-symbols-keywords)))
+  (setq perl--augmented-font-lock-keywords-1
+        (append perl-font-lock-keywords-1
+                (prog-prettify-font-lock-symbols-keywords)))
+  (setq perl--augmented-font-lock-keywords-2
+        (append perl-font-lock-keywords-2
+                (prog-prettify-font-lock-symbols-keywords)))
+
+  (setq font-lock-defaults '((perl--augmented-font-lock-keywords
+                              perl--augmented-font-lock-keywords-1
+                              perl--augmented-font-lock-keywords-2)
+                             nil nil ((?\_ . "w")) nil
                              (font-lock-syntactic-face-function
                               . perl-font-lock-syntactic-face-function)))
   (setq-local syntax-propertize-function #'perl-syntax-propertize-function)

=== modified file 'lisp/simple.el'
--- lisp/simple.el	2013-05-25 02:21:49 +0000
+++ lisp/simple.el	2013-06-04 19:58:10 +0000
@@ -372,34 +372,6 @@
   "Parent major mode from which special major modes should inherit."
   (setq buffer-read-only t))
 
-;; Major mode meant to be the parent of programming modes.
-
-(defvar prog-mode-map
-  (let ((map (make-sparse-keymap)))
-    (define-key map [?\C-\M-q] 'prog-indent-sexp)
-    map)
-  "Keymap used for programming modes.")
-
-(defun prog-indent-sexp (&optional defun)
-  "Indent the expression after point.
-When interactively called with prefix, indent the enclosing defun
-instead."
-  (interactive "P")
-  (save-excursion
-    (when defun
-      (end-of-line)
-      (beginning-of-defun))
-    (let ((start (point))
-	  (end (progn (forward-sexp 1) (point))))
-      (indent-region start end nil))))
-
-(define-derived-mode prog-mode fundamental-mode "Prog"
-  "Major mode for editing programming language source code."
-  (set (make-local-variable 'require-final-newline) mode-require-final-newline)
-  (set (make-local-variable 'parse-sexp-ignore-comments) t)
-  ;; Any programming language is always written left to right.
-  (setq bidi-paragraph-direction 'left-to-right))
-
 ;; Making and deleting lines.
 
 (defvar hard-newline (propertize "\n" 'hard t 'rear-nonsticky '(hard))

Re: using glyphs by default in perl-mode

[Stefan Monnier]

> See attached patch.  It only fixes comments and attempts to move to
> prog-mode.el.  Maybe you'll know the way to fix it.  I get:
> Attempt to autoload define-derived-mode while preparing to dump
> If it's trivial to fix, please let me know how.  If not, I'll commit as
> simple.el and update the ChangeLog.

Just commit it in simple.el for now.  I'll move it later.


        Stefan

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Tue, 04 Jun 2013 21:27:19 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> See attached patch.  It only fixes comments and attempts to move to
>> prog-mode.el.  Maybe you'll know the way to fix it.  I get:
>> Attempt to autoload define-derived-mode while preparing to dump
>> If it's trivial to fix, please let me know how.  If not, I'll commit as
>> simple.el and update the ChangeLog.

SM> Just commit it in simple.el for now.  I'll move it later.

Done and in trunk.  Thanks for all the help and reviews from you and
Davis Herring.

If anyone wants to update http://www.emacswiki.org/emacs/PrettyLambda
and http://www.emacswiki.org/emacs/PrettySymbol feel free!

(Aside: I attempted to use `compose-region' to build more complicated
compositions besides a single character.  The docstring says the
COMPONENTS format can be several things, including:

"If it is a vector or list, it is a sequence of alternate characters and
composition rules, where (2N)th elements are characters and (2N+1)th
elements are composition rules to specify how to compose (2N+2)th
elements with previously composed N glyphs."

I stared at this for a while then I gave up.  It really needs one or two
examples and an alternate wording, because I have no idea what it says.)

I also thought about `put-text-property' of an image but that's pretty
ugly and special-cases all the code, plus it has update issues in some
cases (deleting characters and reinserting them doesn't trigger the
image again).  It would be really nice to be able to use
`compose-region' and use an image to create a glyph.  For now it's a
TODO but I think it would be a hit with users.  If anyone is interested
or wants to tell me how it could work, let me know.

The other TODO item is to simplify this support so it doesn't require
special effort when setting `font-lock-defaults'.  I don't know how that
would work internally so I'll wait for Stefan or someone else to give me
hints or implement it.

Thanks
Ted

Re: using glyphs by default in perl-mode

[Stefan Monnier]

> Done and in trunk.  Thanks for all the help and reviews from you and
> Davis Herring.

Thank you, Ted.

> I also thought about `put-text-property' of an image but that's pretty
> ugly and special-cases all the code, plus it has update issues in some
> cases (deleting characters and reinserting them doesn't trigger the
> image again).  It would be really nice to be able to use
> `compose-region' and use an image to create a glyph.  For now it's a
> TODO but I think it would be a hit with users.  If anyone is interested
> or wants to tell me how it could work, let me know.

I think this is pushing it "too far": I'm not opposed to such a feature
in general, but I'm not sure it would benefit much from sharing some
code with the current prog-prettify code.
BTW, one problematic part with the use of images is when you need to
scale them to fit the text's size.

> The other TODO item is to simplify this support so it doesn't require
> special effort when setting `font-lock-defaults'.  I don't know how that
> would work internally so I'll wait for Stefan or someone else to give me
> hints or implement it.

I think the best way to do that is going to be to provide a function
that uses font-lock-add-keywords internally.  That function might even
take the symbols-alist as argument and do the (setq-local
prog-prettify-alist..) itself, so major modes can just call
(prog-prettify '(("=>" . ?⇒))).


        Stefan

Re: using glyphs by default in perl-mode

[Glenn Morris]

(with-broken-record-mode
  Please add a NEWS entry, and give new defcustoms :version tags.
  Thanks.)

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Wed, 05 Jun 2013 12:13:00 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> I also thought about `put-text-property' of an image but that's pretty
>> ugly and special-cases all the code, plus it has update issues in some
>> cases (deleting characters and reinserting them doesn't trigger the
>> image again).  It would be really nice to be able to use
>> `compose-region' and use an image to create a glyph.  For now it's a
>> TODO but I think it would be a hit with users.  If anyone is interested
>> or wants to tell me how it could work, let me know.

SM> I think this is pushing it "too far": I'm not opposed to such a feature
SM> in general, but I'm not sure it would benefit much from sharing some
SM> code with the current prog-prettify code.

OK.  I'll shelve/stash it for now.

SM> BTW, one problematic part with the use of images is when you need to
SM> scale them to fit the text's size.

Of course.  I was thinking of a dynamically produced glyph that scales
down to the text size.  I think SVG can do something like that, scaling
gracefully.  If you want a bigger image, you increase the font size!

Currently images don't respect the text size which makes them mostly
unusable in serious editing.

>> The other TODO item is to simplify this support so it doesn't require
>> special effort when setting `font-lock-defaults'.  I don't know how that
>> would work internally so I'll wait for Stefan or someone else to give me
>> hints or implement it.

SM> I think the best way to do that is going to be to provide a function
SM> that uses font-lock-add-keywords internally.  That function might even
SM> take the symbols-alist as argument and do the (setq-local
SM> prog-prettify-alist..) itself, so major modes can just call
SM> (prog-prettify '(("=>" . ?⇒))).

That would be best.  I'll look into it.

Ted

Re: using glyphs by default in perl-mode

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 05 Jun 2013 12:13:00 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
>
>>> I also thought about `put-text-property' of an image but that's pretty
>>> ugly and special-cases all the code, plus it has update issues in some
>>> cases (deleting characters and reinserting them doesn't trigger the
>>> image again).  It would be really nice to be able to use
>>> `compose-region' and use an image to create a glyph.  For now it's a
>>> TODO but I think it would be a hit with users.  If anyone is interested
>>> or wants to tell me how it could work, let me know.
>
> SM> I think this is pushing it "too far": I'm not opposed to such a feature
> SM> in general, but I'm not sure it would benefit much from sharing some
> SM> code with the current prog-prettify code.
>
> OK.  I'll shelve/stash it for now.
>
> SM> BTW, one problematic part with the use of images is when you need to
> SM> scale them to fit the text's size.
>
> Of course.  I was thinking of a dynamically produced glyph that scales
> down to the text size.  I think SVG can do something like that, scaling
> gracefully.  If you want a bigger image, you increase the font size!
>
> Currently images don't respect the text size which makes them mostly
> unusable in serious editing.

Do you have an idea what the interface would look like?

Since I worked on the ImageMagick support I'm interested in this.


>
>>> The other TODO item is to simplify this support so it doesn't require
>>> special effort when setting `font-lock-defaults'.  I don't know how that
>>> would work internally so I'll wait for Stefan or someone else to give me
>>> hints or implement it.
>
> SM> I think the best way to do that is going to be to provide a function
> SM> that uses font-lock-add-keywords internally.  That function might even
> SM> take the symbols-alist as argument and do the (setq-local
> SM> prog-prettify-alist..) itself, so major modes can just call
> SM> (prog-prettify '(("=>" . ?⇒))).
>
> That would be best.  I'll look into it.
>
> Ted
>
>

-- 
Joakim Verona

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Wed, 05 Jun 2013 13:28:12 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Wed, 05 Jun 2013 12:13:00 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

SM> I think the best way to do that is going to be to provide a function
SM> that uses font-lock-add-keywords internally.  That function might even
SM> take the symbols-alist as argument and do the (setq-local
SM> prog-prettify-alist..) itself, so major modes can just call
SM> (prog-prettify '(("=>" . ?⇒))).

TZ> That would be best.  I'll look into it.

This is done.

Ted

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Wed, 05 Jun 2013 12:43:36 -0400 Glenn Morris <rgm@gnu.org> wrote: 

GM> (with-broken-record-mode
GM>   Please add a NEWS entry, and give new defcustoms :version tags.
GM>   Thanks.)

I wasn't sure this merited a NEWS entry, but can add one if you think
so.  Should I wait until the interface has solidified, since this will
probably be added to several other modes?  Stefan wanted sml-mode IIRC
and I'm taking requests for other modes to prettify :)

I added the :version, sorry about that.

Ted

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Wed, 05 Jun 2013 19:42:39 +0200 joakim@verona.se wrote: 

j> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Of course.  I was thinking of a dynamically produced glyph that scales
>> down to the text size.  I think SVG can do something like that, scaling
>> gracefully.  If you want a bigger image, you increase the font size!
>> 
>> Currently images don't respect the text size which makes them mostly
>> unusable in serious editing.

j> Do you have an idea what the interface would look like?

j> Since I worked on the ImageMagick support I'm interested in this.

My idea doesn't make the current image support useless.  It's a
convenience for when you want images to behave like text, not like a
text property.

From the developer's side it should simply be a call like (make-char)
that produces a character outside the Unicode range.  Maybe later
ImageMagick effects can be stacked on top, but for now just the image
would be enough.

When Emacs paints the glyph for that character, it should draw a scaled
image (stored in a lookup table) to the font size, instead of trying to
find the character in the current font.  So it's like a on-the-fly font,
I guess.

Ted

Re: using glyphs by default in perl-mode

[Glenn Morris]

Ted Zlatanov wrote:

> I wasn't sure this merited a NEWS entry, but can add one if you think
> so.

How else will people know the feature exists, since it defaults to off?
Better to add an entry than not. It can always be removed at release
time, if it turns out not to be useful.

>  Should I wait until the interface has solidified, since this will
> probably be added to several other modes?

It is always better to add (at least a placeholder) entry, sooner rather
than later. Otherwise it might be forgotten.

   New option, `prog-prettify-symbols'.

Done.

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Wed, 05 Jun 2013 14:07:43 -0400 Glenn Morris <rgm@gnu.org> wrote: 

GM> Ted Zlatanov wrote:
>> I wasn't sure this merited a NEWS entry, but can add one if you think
>> so.

GM> How else will people know the feature exists, since it defaults to off?
GM> Better to add an entry than not. It can always be removed at release
GM> time, if it turns out not to be useful.

OK.

>> Should I wait until the interface has solidified, since this will
>> probably be added to several other modes?

GM> It is always better to add (at least a placeholder) entry, sooner rather
GM> than later. Otherwise it might be forgotten.

GM>    New option, `prog-prettify-symbols'.

GM> Done.

OK, added.  I also did some doc fixes.

Ted

Re: using glyphs by default in perl-mode

[Stefan Monnier]

> Of course.  I was thinking of a dynamically produced glyph that scales
> down to the text size.  I think SVG can do something like that, scaling
> gracefully.  If you want a bigger image, you increase the font size!

Scaling is good, but it's difficult to know which size to use.
Worse, the same image might need to be displayed at different sizes in
different windows, since the same text may be rendered with different
font size in different windows.

IOW, I think this requires support on the C side, e.g. to specify the
image height as a multiple of the line height or as a multiple of the
"current font"'s height.


        Stefan

Re: using glyphs by default in perl-mode

[Stefan Monnier]

> probably be added to several other modes?  Stefan wanted sml-mode IIRC
> and I'm taking requests for other modes to prettify :)

BTW, I don't really want sml-mode to use it (yet), neither haskell-mode.
But I'd like it to be good enough that those modes can use it (in a few
years, when Emacs-24.4 is old).


        Stefan

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Wed, 05 Jun 2013 14:55:13 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> probably be added to several other modes?  Stefan wanted sml-mode IIRC
>> and I'm taking requests for other modes to prettify :)

SM> BTW, I don't really want sml-mode to use it (yet), neither haskell-mode.
SM> But I'd like it to be good enough that those modes can use it (in a few
SM> years, when Emacs-24.4 is old).

I don't see a problem either way, it's your call.

Ted

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Wed, 05 Jun 2013 14:52:59 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> Of course.  I was thinking of a dynamically produced glyph that scales
>> down to the text size.  I think SVG can do something like that, scaling
>> gracefully.  If you want a bigger image, you increase the font size!

SM> Scaling is good, but it's difficult to know which size to use.
SM> Worse, the same image might need to be displayed at different sizes in
SM> different windows, since the same text may be rendered with different
SM> font size in different windows.

SM> IOW, I think this requires support on the C side, e.g. to specify the
SM> image height as a multiple of the line height or as a multiple of the
SM> "current font"'s height.

My description was entirely from the developer's side of how it should
behave to make working with images easier.  Unfortunately I have no idea
where to even start to implement this...

Ted

Re: using glyphs by default in perl-mode

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 05 Jun 2013 14:52:59 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
>
>>> Of course.  I was thinking of a dynamically produced glyph that scales
>>> down to the text size.  I think SVG can do something like that, scaling
>>> gracefully.  If you want a bigger image, you increase the font size!
>
> SM> Scaling is good, but it's difficult to know which size to use.
> SM> Worse, the same image might need to be displayed at different sizes in
> SM> different windows, since the same text may be rendered with different
> SM> font size in different windows.
>
> SM> IOW, I think this requires support on the C side, e.g. to specify the
> SM> image height as a multiple of the line height or as a multiple of the
> SM> "current font"'s height.
>
> My description was entirely from the developer's side of how it should
> behave to make working with images easier.  Unfortunately I have no idea
> where to even start to implement this...

There is some support for zoom factor etc in the imagemagick interface
code. it could perhaps be modified.

>
> Ted
>
>

-- 
Joakim Verona

Re: using glyphs by default in perl-mode

[Stefan Monnier]

>>> probably be added to several other modes?  Stefan wanted sml-mode IIRC
>>> and I'm taking requests for other modes to prettify :)

SM> BTW, I don't really want sml-mode to use it (yet), neither haskell-mode.
SM> But I'd like it to be good enough that those modes can use it (in a few
SM> years, when Emacs-24.4 is old).

> I don't see a problem either way, it's your call.

IIRC the current code is not good enough for Haskell's needs, where for
example "." can mean function composition (and hence should be rendered
as something like ∘) but can also mean other things in some contexts, so
a simple regexp doesn't cut it.


        Stefan

Re: using glyphs by default in perl-mode

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Wed, 05 Jun 2013 10:45:26 -0400
> 
> (Aside: I attempted to use `compose-region' to build more complicated
> compositions besides a single character.  The docstring says the
> COMPONENTS format can be several things, including:
> 
> "If it is a vector or list, it is a sequence of alternate characters and
> composition rules, where (2N)th elements are characters and (2N+1)th
> elements are composition rules to specify how to compose (2N+2)th
> elements with previously composed N glyphs."
> 
> I stared at this for a while then I gave up.  It really needs one or two
> examples and an alternate wording, because I have no idea what it says.)

There's a reference to reference-point-alist, which I think has those
details.

As for examples, you can see them in tv-util.el, for example.

> It would be really nice to be able to [...] use an image to create a
> glyph.

I don't understand this: Emacs _can_ display an image, so what can you
possibly mean by "use an image to create a glyph"?  What is a "glyph"
in this context?

bug#13189: using glyphs by default in perl-mode

[Jambunathan K]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Wed, 05 Jun 2013 10:45:26 -0400
>> 
>> (Aside: I attempted to use `compose-region' to build more complicated
>> compositions besides a single character.  The docstring says the
>> COMPONENTS format can be several things, including:
>> 
>> "If it is a vector or list, it is a sequence of alternate characters and
>> composition rules, where (2N)th elements are characters and (2N+1)th
>> elements are composition rules to specify how to compose (2N+2)th
>> elements with previously composed N glyphs."
>> 
>> I stared at this for a while then I gave up.  It really needs one or two
>> examples and an alternate wording, because I have no idea what it says.)
>
> There's a reference to reference-point-alist, which I think has those
> details.
>
> As for examples, you can see them in tv-util.el, for example.

FWIW, created a cross link to 13189@debbugs.gnu.org.

Re: using glyphs by default in perl-mode

[Jambunathan K]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Wed, 05 Jun 2013 10:45:26 -0400
>> 
>> (Aside: I attempted to use `compose-region' to build more complicated
>> compositions besides a single character.  The docstring says the
>> COMPONENTS format can be several things, including:
>> 
>> "If it is a vector or list, it is a sequence of alternate characters and
>> composition rules, where (2N)th elements are characters and (2N+1)th
>> elements are composition rules to specify how to compose (2N+2)th
>> elements with previously composed N glyphs."
>> 
>> I stared at this for a while then I gave up.  It really needs one or two
>> examples and an alternate wording, because I have no idea what it says.)
>
> There's a reference to reference-point-alist, which I think has those
> details.
>
> As for examples, you can see them in tv-util.el, for example.

FWIW, created a cross link to 13189@debbugs.gnu.org.

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Fri, 07 Jun 2013 10:36:24 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> "If it is a vector or list, it is a sequence of alternate characters and
>> composition rules, where (2N)th elements are characters and (2N+1)th
>> elements are composition rules to specify how to compose (2N+2)th
>> elements with previously composed N glyphs."
>> 
>> I stared at this for a while then I gave up.  It really needs one or two
>> examples and an alternate wording, because I have no idea what it says.)

EZ> There's a reference to reference-point-alist, which I think has those
EZ> details.

EZ> As for examples, you can see them in tv-util.el, for example.

OK, but read that sentence again.  Can you really make sense of it?

It needs rewording and an inline example right after this paragraph.  It
doesn't have to be a complicated example.

>> It would be really nice to be able to [...] use an image to create a
>> glyph.

EZ> I don't understand this: Emacs _can_ display an image, so what can you
EZ> possibly mean by "use an image to create a glyph"?  What is a "glyph"
EZ> in this context?

Currently, AFAIK Emacs treats images as a text property.  This is
convenient but there are many cases where I'd rather have images behave
like typed characters: one image == one character == one glyph.  I
proposed using a virtual, dynamically generated font outside Unicode or
perhaps in a reserved Unicode space.

The result should be that asking (just an example, not an implementation
suggestion):

(make-char 'image "/tmp/gnus.png")

will produce something that respects font size, can be scaled, and looks
like a character to all Emacs functions but like an image visually.  In
text mode or without image support it would be treated like a character
that can't be rendered.

I hope that explains things better.

Ted

Re: using glyphs by default in perl-mode

[Kenichi Handa]

In article <83d2ryv0cn.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org>
writes:
> > From: Ted Zlatanov <tzz@lifelogs.com>
> > Date: Wed, 05 Jun 2013 10:45:26 -0400
> > 
> > (Aside: I attempted to use `compose-region' to build more
> > complicated
> > compositions besides a single character.  The docstring says the
> > COMPONENTS format can be several things, including:
[...]
> > 
> > I stared at this for a while then I gave up.  It really needs one or
> > two
> > examples and an alternate wording, because I have no idea what it
> > says.)

> There's a reference to reference-point-alist, which I think has those
> details.

Yes.  And, here are simple examples:

(insert (compose-string "." 0 1 '(?+ (tc . bc) ?o)))
(insert (compose-string "." 0 1 '(?+ (cc . cc) ?o)))

---
Kenichi Handa
handa@gnu.org

bug#13189: using glyphs by default in perl-mode

[Ted Zlatanov]

On Fri, 07 Jun 2013 13:15:13 +0530 Jambunathan K <kjambunathan@gmail.com> wrote: 

JK> Eli Zaretskii <eliz@gnu.org> writes:
>>> From: Ted Zlatanov <tzz@lifelogs.com>
>>> Date: Wed, 05 Jun 2013 10:45:26 -0400
>>> 
>>> (Aside: I attempted to use `compose-region' to build more complicated
>>> compositions besides a single character.  The docstring says the
>>> COMPONENTS format can be several things, including:
>>> 
>>> "If it is a vector or list, it is a sequence of alternate characters and
>>> composition rules, where (2N)th elements are characters and (2N+1)th
>>> elements are composition rules to specify how to compose (2N+2)th
>>> elements with previously composed N glyphs."
>>> 
>>> I stared at this for a while then I gave up.  It really needs one or two
>>> examples and an alternate wording, because I have no idea what it says.)
>> 
>> There's a reference to reference-point-alist, which I think has those
>> details.
>> 
>> As for examples, you can see them in tv-util.el, for example.

JK> FWIW, created a cross link to 13189@debbugs.gnu.org.

Has that bug been resolved with the latest changes to perl-mode and
prog-mode-prettify?

Ted

compose-region docstring update (was: using glyphs by default in perl-mode)

[Ted Zlatanov]

On Fri, 07 Jun 2013 08:53:36 -0400 Kenichi Handa <handa@gnu.org> wrote: 

KH> In article <83d2ryv0cn.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org>
KH> writes:
>> > From: Ted Zlatanov <tzz@lifelogs.com>
>> > Date: Wed, 05 Jun 2013 10:45:26 -0400
>> > 
>> > (Aside: I attempted to use `compose-region' to build more
>> > complicated
>> > compositions besides a single character.  The docstring says the
>> > COMPONENTS format can be several things, including:
KH> [...]
>> > 
>> > I stared at this for a while then I gave up.  It really needs one
>> > or two examples and an alternate wording, because I have no idea
>> > what it says.)

>> There's a reference to reference-point-alist, which I think has those
>> details.

KH> Yes.  And, here are simple examples:

KH> (insert (compose-string "." 0 1 '(?+ (tc . bc) ?o)))
KH> (insert (compose-string "." 0 1 '(?+ (cc . cc) ?o)))

Thanks for the examples.  Can you explain them (I think I do, but your
words would be welcome help)?  I will update the docstring once I
understand this well enough.

Ted

Re: using glyphs by default in perl-mode

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Fri, 07 Jun 2013 08:48:48 -0400
> 
> OK, but read that sentence again.  Can you really make sense of it?

Yes, definitely.  I wonder what makes it illegible for you.  (No, I
didn't write that doc string.)

> >> It would be really nice to be able to [...] use an image to create a
> >> glyph.
> 
> EZ> I don't understand this: Emacs _can_ display an image, so what can you
> EZ> possibly mean by "use an image to create a glyph"?  What is a "glyph"
> EZ> in this context?
> 
> Currently, AFAIK Emacs treats images as a text property.

More accurately, you display images by creating text properties with
images as their values.

> This is
> convenient but there are many cases where I'd rather have images behave
> like typed characters: one image == one character == one glyph.

Then create a font.  That's what you want.  Emacs cannot display text
as something else except via text properties or overlays.

> (make-char 'image "/tmp/gnus.png")
> 
> will produce something that respects font size, can be scaled, and looks
> like a character to all Emacs functions but like an image visually.  In
> text mode or without image support it would be treated like a character
> that can't be rendered.

This doesn't make sense to me: a character has many properties and
attributes that the above doesn't provide.  Displaying an image as a
character means that you will need to implement a font library, or
something close.

Why can't you generate a set of strings with display properties, and
then insert them into a buffer?  After all, a single-character string
should do what you want, no?

Re: using glyphs by default in perl-mode

[Ted Zlatanov]

On Fri, 07 Jun 2013 17:52:10 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Fri, 07 Jun 2013 08:48:48 -0400
>> 
>> OK, but read that sentence again.  Can you really make sense of it?

EZ> Yes, definitely.  I wonder what makes it illegible for you.  (No, I
EZ> didn't write that doc string.)

The way it's written makes it hard to read, not illegible.

You and others who know how text composition works are a terrible use
case for this docstring.  I'll propose a revised version.

>> Currently, AFAIK Emacs treats images as a text property.

EZ> More accurately, you display images by creating text properties with
EZ> images as their values.

Yes, exactly.

>> This is convenient but there are many cases where I'd rather have
>> images behave like typed characters: one image == one character ==
>> one glyph.

EZ> Then create a font.  That's what you want.  Emacs cannot display text
EZ> as something else except via text properties or overlays.

Yes!  But I don't want to create the font, I want Emacs to do it for me.

>> (make-char 'image "/tmp/gnus.png")
>> 
>> will produce something that respects font size, can be scaled, and looks
>> like a character to all Emacs functions but like an image visually.  In
>> text mode or without image support it would be treated like a character
>> that can't be rendered.

EZ> This doesn't make sense to me: a character has many properties and
EZ> attributes that the above doesn't provide.

They will have to be faked, simulated, or ignored.

EZ> Displaying an image as a character means that you will need to
EZ> implement a font library, or something close.

It's a feature request based on my experience with images in Emacs
today.  If it goes on my queue, it may never get done because I have no
knowledge of the font and image internals.  But obviously that's what
will happen if no one else thinks it's worth implementing... so this is
my proposal, to gather interest.

EZ> Why can't you generate a set of strings with display properties, and
EZ> then insert them into a buffer?  After all, a single-character string
EZ> should do what you want, no?

It doesn't.  It doesn't respect font size, most importantly.  It needs
to scale so I can insert 20 images at size 15 and have them look good.
And when I change to size 25, I want the images to flow like text.  I
don't want to treat images in a special way in some cases.

The origin of this was that I'd like to extend `prog-prettify-symbols'
to support images as well.  It would be so much easier to say

("eli" . ,(make-char 'image "eli.png"))

and pass the character to be inserted, not a special extension to this
simple format to support images.  BTW, SVG would be great here too.

Fundamentally, I'm talking about treating images as characters because
Emacs is good at dealing with characters.  Text properties are useful
and I think the image support there is great, but text properties are
not first-class citizens of a buffer.

For instance, to come back to the first part of this post, it would be
very nice to compose characters with image glyphs or SVG glyphs
(gradients and vector graphics).  It would make some pretty amazing
visuals possible.  I've done this with Java Swing, and you can see some
examples at http://filthyrichclients.org/ (I highly recommend the
book).

HTML is a very good analogy.  It has images as a text property through
CSS and it has images as an <img> element, which ends up as an actual
DOM node (IIUC the terminology, I hope you'll know what I mean).  They
both have their place.  Emacs only has the former.

Ted

bug#13189: using glyphs by default in perl-mode

[Stefan Monnier]

> Has that bug been resolved with the latest changes to perl-mode and
> prog-mode-prettify?

I believe so, yes,


        Stefan

Re: bug#13189: using glyphs by default in perl-mode

[Stefan Monnier]

> Has that bug been resolved with the latest changes to perl-mode and
> prog-mode-prettify?

I believe so, yes,


        Stefan

bug#13189: using glyphs by default in perl-mode

[Glenn Morris]

Stefan Monnier wrote:

JK> FWIW, created a cross link to 13189 <at> debbugs.gnu.org.

>> Has that bug been resolved with the latest changes to perl-mode and
>> prog-mode-prettify?
>
> I believe so, yes,

What? Bug 13189 is "please document compose-region (more)".

Simply cc-ing it in the middle of an unrelated discussion where someone
happened to say "I can't understand compose-region" isn't helpful.

bug#13189: using glyphs by default in perl-mode

[Ted Zlatanov]

On Fri, 07 Jun 2013 12:44:34 -0400 Glenn Morris <rgm@gnu.org> wrote: 

GM> Stefan Monnier wrote:
JK> FWIW, created a cross link to 13189 <at> debbugs.gnu.org.

>>> Has that bug been resolved with the latest changes to perl-mode and
>>> prog-mode-prettify?
>> 
>> I believe so, yes,

GM> What? Bug 13189 is "please document compose-region (more)".

GM> Simply cc-ing it in the middle of an unrelated discussion where someone
GM> happened to say "I can't understand compose-region" isn't helpful.

I got fooled by the subject.  Sorry.

Ted

bug#13189: using glyphs by default in perl-mode

[Jambunathan K]

Glenn Morris <rgm@gnu.org> writes:

> What? Bug 13189 is "please document compose-region (more)".
>
> Simply cc-ing

I had CCed machines not people (I took care to remove names.)

> it in the middle of an unrelated discussion 

What cropped in the middle of the discussion was relevant to the bug.

> where someone happened to say "I can't understand compose-region"
> isn't helpful.

It was Eli's reference to tv-util.el that I CCed (not Ted's
question). If someone were to look at closing that bug, he can readily
borrow Kenichi's followup examples.

HTH.

bug#13189: using glyphs by default in perl-mode

[Stefan Monnier]

> What? Bug 13189 is "please document compose-region (more)".

I'm glad one of us is still sober.


        Stefan

bug#13189: using glyphs by default in perl-mode

[Glenn Morris]

So you send a mail to 13189@debbugs only that says "on emacs-devel,
someone said that the examples in tv-util.el may be helpful". Instead of
cc'ing an ongoing emacs-devel discussion with the subject "using glyphs
by default in perl-mode".

Re: compose-region docstring update (was: using glyphs by default in perl-mode)

[Kenichi Handa]

In article <877gi6qa57.fsf_-_@lifelogs.com>, Ted Zlatanov <tzz@lifelogs.com> writes:

KH> Yes.  And, here are simple examples:

KH> (insert (compose-string "." 0 1 '(?+ (tc . bc) ?o)))
KH> (insert (compose-string "." 0 1 '(?+ (cc . cc) ?o)))

> Thanks for the examples.  Can you explain them (I think I do, but your
> words would be welcome help)?  I will update the docstring once I
> understand this well enough.

I'll try to explain the first example.  In it,
compose-string has the optional 4th argument COMPONENTS.
The explanation of COMPONENTS (of type "list") is explained in
compose-region as below:

If it is a vector or list, it is a sequence of alternate characters and
composition rules, where (2N)th elements are characters and (2N+1)th
elements are composition rules to specify how to compose (2N+2)th
elements with previously composed N glyphs.

So, as for (?+ (tc . bc) ?o), ?+ is the first alternate
character, (tc . bc) is the first composition rule, ?o is
the second alternate character.  This means ?o is composed
with ?+ by the rule (tc . bc).

And the docstring of reference-point-alist explains the meaing of (tc
. bc).  According to the docstring, it means that ?o is compose with ?+
by aligning the top-center point of the glyphs of ?+ and the
bottom-center point of the glyph of ?o.  That yeilds a glyph whose
lower part is ?+ and upper part is ?o.

---
Kenichi Handa
handa@gnu.org