Re: replacing characters and whacky trans-buffer conversion

[Oliver Scholz <alkibiades@gmx.de>]

ken <gebser@speakeasy.net> writes:

> An email comes in with this (emdash) character in it: –
>
> It looks like an em-dash until the text containing it is pasted into an
> emacs buffer; then it appears as a series of "garbage characters".
> (Copy and paste the emdash into an emacs buffer yourself, and perhaps
> you'll see what I mean.)

I am afraid, your're making assumptions here that don't hold true. I
am using Emacs to read mail and news. Your message was encoded in
UTF-8 and what I see is an en-dash U+2013 (are you sure this was
supposed to be an e_m_-dash?). This does not change when I kill&yank
it into another buffer.

I assume you are using another (non-Emacs) E-Mail client. And you are
copying&pasting it via X-selection, Gnome-Clipboard, MS Windows, the
MacOS GUI or something like this? You might be interested in fixing
this instead of the retroactive fix you're trying to undertake. (I
wouldn't know how though, X-selection is a mystery to me.)

(Have you tried saving the message from your Email application instead
of c&p?)

> To me and, possibly to you, this emdash appears in emacs as nine (9)
> "garbage" characters.
>
> Because I want to programmatically replace these 9 garbage characters
> into something latin1-friendly, I copy-and-paste these nine characters
> into an *.el file containing a line like this:
>
>   (replace-string "–" "--" nil (point-min) (point-max))
                    ^^^

I assume this consisted of nine garbage characters when you wrote your
message? Because here it showed up as a valid U+2013 (EN DASH) encoded
in UTF-8. (I'd have no explanation for this though, because EN DASH in
unrecognized UTF-8 would show up as 3 garbage characters, not 9.)

> The sought string (i.e., the first argument above) isn't found, however
> because, for some whacky reason, the emdash pasted into the *.el file is
> different-- by one character-- from exactly the same emdash pasted into
> the other emacs buffer (the one I'm saving the email in).

The reason is: characters aren't bytes. Because of some bug or
missing feature you are exposed to the internal byte/octet
representation of character in one way or the other (I'd guess the
internal representation used by the clipboard.) Emacs tries to fit
this into its own internal representation, unfortunately it has two of
them: unibyte and multibyte.

> In the emacs buffer containing the email, the fourth garbage character
> (as shown by C-u C-x=) is:

[...]
> buffer code: 0x86 0xE2
>   file code: not encodable by coding system undecided-unix

This is a multibyte buffer.

> In the *.el buffer, the fourth garbage character (which should be
> exactly the same character) is:

[...]
> buffer code: 0xE2
>   file code: 0xE2 (encoded by coding system raw-text-unix)

This is a unibyte buffer. Normally el-files shouldn't be encoded in
raw-text-unix. Were you visiting this file with find-file-literally?
If not what setting has caused this?

It is not a proper solution, but if you are really keen on searching
and replacing the octets rather than fixing the character encoding,
then the proper way would be to make sure that *both* buffers are
unibyte, because that's for dealing with binaries.


    Oliver
-- 
16 Ventôse an 215 de la Révolution
Liberté, Egalité, Fraternité!