Finding and mapping all UTF-8 characters

Finding and mapping all UTF-8 characters

[deech]

Hi all,
I recently cut-and-pasted large chunks of text into an HTML document.
When I tried to save the document I was warned that it was ISO-Latin
but there were UTF-8 characters in the text.

Is there a way to (1) search for the UTF-8 encoded characters in a
document and (2) map them to a sensible ASCII character?

Thanks ...
-deech

Re: Finding and mapping all UTF-8 characters

[Pascal J. Bourguignon]

deech <aditya.siram@gmail.com> writes:

> Hi all,
> I recently cut-and-pasted large chunks of text into an HTML document.
> When I tried to save the document I was warned that it was ISO-Latin
> but there were UTF-8 characters in the text.

I doubt it warned that.

ISO-Latin is not a character encoding, it is a familly of character
encodings.  A HTML document is not encoded by a familly of encodings,
but by one single encoding.

UTF-8 is a character encoding.  A character is not a character
encoding.

So  a sentence saying that "a document is  ISO-Latin but there are
UTF-8 characters in the text." is totally meaningless.


> Is there a way to (1) search for the UTF-8 encoded characters in a
> document and

No it is not possible, because characters in a document are not
encoded, they are characters, that's all.


> (2) map them to a sensible ASCII character?

How do you map sensibly ∈, ㎲, 纺 or ⇣ to the characters in the ASCII
character set?

But even if you choosed a mapping (you could for example map the
character to their names: ELEMENT_OF, SQUARE_MU_S, U7EBA, and
DOWNWARDS_DASHED_ARROW), why would you want to do such a thing?

HTML is perfectly able to use encodings that can encode unicode
characters, and all the current browsers are able to deal with HTML
documents encoding unicode characters, so why would you want to
massacre your document?

(There's a valid reason to be wanting to do that, but if you don't
know it, then you don't have it).


-- 
__Pascal Bourguignon__

Re: Finding and mapping all UTF-8 characters

[Peter Dyballa]

Am 05.12.2009 um 17:03 schrieb deech:

> Is there a way to (1) search for the UTF-8 encoded characters in a
> document

Yes. In GNU Emacs 23 I've seen in the *Warnings* buffer hyper-links to  
the characters not fitting into the specified encoding.

You could also search for the usual prefixes of UTF-{7,8,16} encoded  
characters.

> and (2) map them to a sensible ASCII character?


How can you map 100,000 or 200,000 characters to a very limited set of  
100? This mapping would be candidate for the most successful  
compression algorithm...


Besides, it's not sane to save a file in an encoding a when the file's  
header tells its contents is in encoding b.

--
Greetings

   Pete

If you don't find it in the index, look very carefully through the  
entire catalogue.
		–  Sears, Roebuck, and Co., Consumer's Guide, 1897

Re: Finding and mapping all UTF-8 characters

[harven]

deech <aditya.siram@gmail.com> writes:

> Hi all,
> I recently cut-and-pasted large chunks of text into an HTML document.
> When I tried to save the document I was warned that it was ISO-Latin
> but there were UTF-8 characters in the text.

The warning actually contains a list of these characters, and you can click
on them to see where they are located in the buffer.

> Is there a way to (1) search for the UTF-8 encoded characters in a
> document and (2) map them to a sensible ASCII character?
>
> Thanks ...
> -deech

Instead of converting to latin-1, it is probably better to save the file
in another coding system. Just do
M-x set-buffer-file-coding-system RET utf-8 RET

On the other hand, if you were surprised by the unicode characters,
then this probably means that there are few of them. Have a look at
the iso-cvt.el package for setting a conversion table.
The command iso-sgml2iso is pretty close to what you want.

Now, if you want to search a buffer for all characters belonging to 
some category, you can use a regexp. 

\ca matches any ascii characters (newlines excluded). Same as [[:ascii:]].
\Ca matches any non-ascii characters (newlines included).
\cl matches any latin characters (newlines excluded).
\Cl matches any non-latin characters (newlines included).

So the following command copies all non-latin characters to the scratch buffer.
M-x replace-regexp RET \Cl RET \,(princ \& (get-buffer "*scratch*"))