bug#3640: I can't see all characters in my files anymore

bug#3640: I can't see all characters in my files anymore

[jidanni]

Help, I'm now shipping files to people with all kinds of embedded junk
in them like
|        character: ‎ (8206, #o20016, #x200e)
|preferred charset: unicode (Unicode (ISO10646))
|       code point: 0x200E
|           syntax: . 	which means: punctuation
|      buffer code: #xE2 #x80 #x8E
|        file code: #xE2 #x80 #x8E (encoded by coding system utf-8-unix)
|          display: by this font (glyph code)
|    xft:-unknown-FreeSans-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x8F1)
|
|Character code properties: customize what to show
|  name: LEFT-TO-RIGHT MARK
|  general-category: Cf (Other, Format)
all because I can't see them anymore now that I use emacs23.
I'm like a kid using MicroSoft. I can't see exactly what is in my files
anymore. I can't find the switch to turn it back off.

bug#3640: I can't see all characters in my files anymore

[Chong Yidong]

> Help, I'm now shipping files to people with all kinds of embedded junk
> in them like
> |        character: ‎ (8206, #o20016, #x200e)
> |preferred charset: unicode (Unicode (ISO10646))
> |       code point: 0x200E
> |           syntax: . 	which means: punctuation
> |      buffer code: #xE2 #x80 #x8E
> |        file code: #xE2 #x80 #x8E (encoded by coding system utf-8-unix)
> |          display: by this font (glyph code)
> |    xft:-unknown-FreeSans-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x8F1)
> |
> |Character code properties: customize what to show
> |  name: LEFT-TO-RIGHT MARK
> |  general-category: Cf (Other, Format)
> all because I can't see them anymore now that I use emacs23.

Your font claims to be able to display character code 8206.  So, Emacs
does not get in the way.  This is not a bug.

bug#3640: I can't see all characters in my files anymore

[jidanni]

>>>>> "CY" == Chong Yidong <cyd@stupidchicken.com> writes:
>> Help, I'm now shipping files to people with all kinds of embedded junk
>> in them like
>> |        character: ‎ (8206, #o20016, #x200e)
>> |preferred charset: unicode (Unicode (ISO10646))
>> |       code point: 0x200E
>> |           syntax: . 	which means: punctuation
>> |      buffer code: #xE2 #x80 #x8E
>> |        file code: #xE2 #x80 #x8E (encoded by coding system utf-8-unix)
>> |          display: by this font (glyph code)
>> |    xft:-unknown-FreeSans-normal-normal-normal-*-16-*-*-*-*-0-iso10646-1 (#x8F1)
>> |
>> |Character code properties: customize what to show
>> |  name: LEFT-TO-RIGHT MARK
>> |  general-category: Cf (Other, Format)
>> all because I can't see them anymore now that I use emacs23.

CY> Your font claims to be able to display character code 8206.  So, Emacs
CY> does not get in the way.  This is not a bug.

Yes and I'm sure it does display it properly: the proper way is to make
it disappear, as it is a special direction changing character and not a
normal character.

This is where emacs needs to step in and give the user a way to be aware
his file contains such things.

Just like whitespace mode makes one aware one has whitespace.

I'll see if whitespace.el will fill my needs...

bug#3640: I can't see all characters in my files anymore

[jidanni]

OK, I cooked up this workaround in .emacs:

;;Just to see if there are any zero width >‎<, U+200E LEFT-TO-RIGHT MARK
;;hiding in our files, we do:
(global-whitespace-mode 1)
(setq whitespace-style '(spaces))
;;Set neighbor characters on fire in order to see where the
;;"invisible man" is:
(setq whitespace-space-regexp "\\(.?\x200E+.?\\)")
(put 'whitespace-space-before-tab 'face-alias 'whitespace-space);fire color
;;Works except if it is the only char on a line. (In emacs -nw it is
;;width 1, not zero, for me.)

What a pain. But then to turn it on in emacs-w3m,
(add-hook 'w3m-mode-hook
	  (lambda ()
	    (whitespace-mode 1)))
But that kills all of emacs-w3m's font-locks.

Why can't there be a way to not allow any "invisible man" characters
from hiding, like it was in emacs22. Just make them show up as this
dotted box like in xterm, so we know something is funny there.
Is there some variable that can stop 0 width characters being smuggled
around under the users' noses unaware?

I mean isn't that what separates us from the MicroSoft users, they don't
know what is going on inside their files, but we do -- until now.