Encoding/decoding problems

Encoding/decoding problems

[Deniz Dogan]

I'm fetching an XML document that's uses iso-8859-1 coding with 
`url-retrieve' and then I parse it using `xml-parse-region'.

After that, I get the parts of the document that I want and insert them 
into a buffer.  However, the Swedish characters å, ä and ö are displayed 
as \345, \344 and \326 respectively.

I've tried messing around with `encode-coding-region' and 
`decode-coding-region' but I'm really not sure what to do here.

Any help appreciated!

Deniz

Re: Encoding/decoding problems

[Eli Zaretskii]

> Date: Thu, 28 Jul 2011 10:18:04 +0200
> From: Deniz Dogan <deniz@dogan.se>
> 
> I'm fetching an XML document that's uses iso-8859-1 coding with 
> `url-retrieve' and then I parse it using `xml-parse-region'.
> 
> After that, I get the parts of the document that I want and insert them 
> into a buffer.  However, the Swedish characters å, ä and ö are displayed 
> as \345, \344 and \326 respectively.
> 
> I've tried messing around with `encode-coding-region' and 
> `decode-coding-region' but I'm really not sure what to do here.

I suggest to start with describing a reproducible recipe for this
problem.  Not sure if this forum is appropriate, perhaps emacs-devel
is a better place (as it sounds like you are describing a bug).

Re: Encoding/decoding problems

[Deniz Dogan]

On 2011-07-28 11:01, Eli Zaretskii wrote:
>> Date: Thu, 28 Jul 2011 10:18:04 +0200
>> From: Deniz Dogan<deniz@dogan.se>
>>
>> I'm fetching an XML document that's uses iso-8859-1 coding with
>> `url-retrieve' and then I parse it using `xml-parse-region'.
>>
>> After that, I get the parts of the document that I want and insert them
>> into a buffer.  However, the Swedish characters å, ä and ö are displayed
>> as \345, \344 and \326 respectively.
>>
>> I've tried messing around with `encode-coding-region' and
>> `decode-coding-region' but I'm really not sure what to do here.
>
> I suggest to start with describing a reproducible recipe for this
> problem.  Not sure if this forum is appropriate, perhaps emacs-devel
> is a better place (as it sounds like you are describing a bug).
>

Here is the code to reproduce it:

(defun fetch-and-show ()
   (interactive)
   (let* ((old-buffer (current-buffer))
          (url "http://dogan.se/sites/default/files/example.xml")
          (buffer (url-retrieve-synchronously url)))
     (with-current-buffer buffer
       (let ((doc (car (xml-parse-region (point-min) (point-max)))))
         (with-current-buffer old-buffer
           (insert
            (nth 2 (nth 2 (nth 3 doc)))))))))

The XML file is encoded in iso-8859-1 with a bunch of Swedish characters 
here and there.  The buffer I'm testing this with is *scratch* with 
utf-8-unix.  It should insert "hallå" but inserts "hall\345".

I have no idea whether I should use `encode-region-string' or 
`decode-region-string' or what.  I'd doubt it's a bug to be honest, it's 
probably my lack of understanding that's causing this.

Deniz

Re: Encoding/decoding problems

[Jambunathan K]

Deniz Dogan <deniz@dogan.se> writes:

> On 2011-07-28 11:01, Eli Zaretskii wrote:
>>> Date: Thu, 28 Jul 2011 10:18:04 +0200
>>> From: Deniz Dogan<deniz@dogan.se>
>>>
>>> I'm fetching an XML document that's uses iso-8859-1 coding with
>>> `url-retrieve' and then I parse it using `xml-parse-region'.
>>>
>>> After that, I get the parts of the document that I want and insert them
>>> into a buffer.  However, the Swedish characters å, ä and ö are displayed
>>> as \345, \344 and \326 respectively.
>>>
>>> I've tried messing around with `encode-coding-region' and
>>> `decode-coding-region' but I'm really not sure what to do here.
>>
>> I suggest to start with describing a reproducible recipe for this
>> problem.  Not sure if this forum is appropriate, perhaps emacs-devel
>> is a better place (as it sounds like you are describing a bug).
>>
>
> Here is the code to reproduce it:
>
> (defun fetch-and-show ()
>   (interactive)
>   (let* ((old-buffer (current-buffer))
>          (url "http://dogan.se/sites/default/files/example.xml")
>          (buffer (url-retrieve-synchronously url)))
>     (with-current-buffer buffer
>       (let ((doc (car (xml-parse-region (point-min) (point-max)))))
>         (with-current-buffer old-buffer
>           (insert
>            (nth 2 (nth 2 (nth 3 doc)))))))))
>
> The XML file is encoded in iso-8859-1 with a bunch of Swedish
> characters here and there.  The buffer I'm testing this with is
> *scratch* with utf-8-unix.  It should insert "hallå" but inserts
> "hall\345".


FWIW, this works as expected. Note the use of decode coding string.

(defun fetch-and-show ()
  (interactive)
  (let* ((old-buffer (current-buffer))
         (url "http://dogan.se/sites/default/files/example.xml")
         (buffer (url-retrieve-synchronously url)))
    (with-current-buffer buffer
      (let ((doc (car (xml-parse-region (point-min) (point-max)))))
        (with-current-buffer old-buffer
	  (insert
	   (decode-coding-string (nth 2 (nth 2 (nth 3 doc))) 'iso-8859-1)))))))

>
> I have no idea whether I should use `encode-region-string' or
> decode-region-string' or what.  I'd doubt it's a bug to be honest,
> it's probably my lack of understanding that's causing this.
>
> Deniz
>
>

-- 

Re: Encoding/decoding problems

[Eli Zaretskii]

> Date: Thu, 28 Jul 2011 11:26:53 +0200
> From: Deniz Dogan <deniz@dogan.se>
> 
> (defun fetch-and-show ()
>    (interactive)
>    (let* ((old-buffer (current-buffer))
>           (url "http://dogan.se/sites/default/files/example.xml")
>           (buffer (url-retrieve-synchronously url)))
>      (with-current-buffer buffer
>        (let ((doc (car (xml-parse-region (point-min) (point-max)))))
>          (with-current-buffer old-buffer
>            (insert
>             (nth 2 (nth 2 (nth 3 doc)))))))))
> 
> The XML file is encoded in iso-8859-1 with a bunch of Swedish characters 
> here and there.  The buffer I'm testing this with is *scratch* with 
> utf-8-unix.  It should insert "hallå" but inserts "hall\345".
> 
> I have no idea whether I should use `encode-region-string' or 
> `decode-region-string' or what.

"Encoding" means converting Emacs's internal representation into an
external representation you want to send to a disk file or another
program.  "Decoding" is the opposite conversion: from an external
representation that you found in a disk file or received from a
network socket to the internal representation Emacs uses in its buffer
and string objects.

So you want "decode-" functions, in this case decode-coding-string,
since you've got the external representation in a string.

Re: Encoding/decoding problems

[Deniz Dogan]

On 2011-07-28 14:18, Eli Zaretskii wrote:
>> Date: Thu, 28 Jul 2011 11:26:53 +0200
>> From: Deniz Dogan<deniz@dogan.se>
>>
>> (defun fetch-and-show ()
>>     (interactive)
>>     (let* ((old-buffer (current-buffer))
>>            (url "http://dogan.se/sites/default/files/example.xml")
>>            (buffer (url-retrieve-synchronously url)))
>>       (with-current-buffer buffer
>>         (let ((doc (car (xml-parse-region (point-min) (point-max)))))
>>           (with-current-buffer old-buffer
>>             (insert
>>              (nth 2 (nth 2 (nth 3 doc)))))))))
>>
>> The XML file is encoded in iso-8859-1 with a bunch of Swedish characters
>> here and there.  The buffer I'm testing this with is *scratch* with
>> utf-8-unix.  It should insert "hallå" but inserts "hall\345".
>>
>> I have no idea whether I should use `encode-region-string' or
>> `decode-region-string' or what.
>
> "Encoding" means converting Emacs's internal representation into an
> external representation you want to send to a disk file or another
> program.  "Decoding" is the opposite conversion: from an external
> representation that you found in a disk file or received from a
> network socket to the internal representation Emacs uses in its buffer
> and string objects.
>
> So you want "decode-" functions, in this case decode-coding-string,
> since you've got the external representation in a string.
>

Thanks for the explanation and thank you to Jambunathan for the help.

It appeared that I had simply done the decoding at the wrong stage of 
the processing somehow (the real example is a lot larger).

Again, thanks

Deniz