decode-coding-string question

decode-coding-string question

[Ted Zlatanov]

This should decode to нуль but doesn't (I get the same string instead):

(decode-coding-string "íîëü" 'cp1251)

Am I missing something obvious?  Do I need to encode the string to
something else?

Ted

Re: decode-coding-string question

[Dmitry Dzhus]

Ted Zlatanov wrote:

> This should decode to нуль but doesn't (I get the same string instead):
>
> (decode-coding-string "íîëü" 'cp1251)
>
> Am I missing something obvious?  Do I need to encode the string to
> something else?

0. «íóëü», not «íîëü»

1. (decode-coding-string (string-make-unibyte "íóëü") 'cp1251)
-- 
Happy Hacking.

http://sphinx.net.ru
む

Re: decode-coding-string question

[Eli Zaretskii]

> From: Dmitry Dzhus <dima@sphinx.net.ru>
> Date: Fri, 15 Aug 2008 02:19:11 +0400
> 
> Ted Zlatanov wrote:
> 
> > This should decode to нуль but doesn't (I get the same string instead):
> >
> > (decode-coding-string "íîëü" 'cp1251)
> >
> > Am I missing something obvious?  Do I need to encode the string to
> > something else?
> 
> 0. «íóëü», not «íîëü»

That's not important, the original problem remains, even with a
different spelling of the word.

> 1. (decode-coding-string (string-make-unibyte "íóëü") 'cp1251)

Yes, that's it: decode-coding-string works reliably only on unibyte
strings.  In multibyte strings, random byte values will be interpreted
according to rules that are appropriate for Emacs internal
representation of text in buffers and strings, not according to what
we humans expect.

Re: decode-coding-string question

[Ted Zlatanov]

On Fri, 15 Aug 2008 02:19:11 +0400 Dmitry Dzhus <dima@sphinx.net.ru> wrote: 

DD> (decode-coding-string (string-make-unibyte "íóëü") 'cp1251)

Thanks.

There should probably be a specific function for this:

(decode-coding-string-as-unibyte "íóëü" 'cp1251)

ditto for decode-coding-region.  Should I add it or is that not
generally useful?  A flag is not as good because both functions have
several flags already.

Ted

Re: decode-coding-string question

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Fri, 15 Aug 2008 10:54:20 -0500
> 
> There should probably be a specific function for this:
> 
> (decode-coding-string-as-unibyte "íóëü" 'cp1251)
> 
> ditto for decode-coding-region.  Should I add it or is that not
> generally useful?

Personally, I think it's not useful, since decode-coding-region and
decode-coding-string are used only on unibyte text.  But feel free to
raise this on emacs-devel.

Re: decode-coding-string question

[Ted Zlatanov]

On Fri, 15 Aug 2008 20:04:42 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Fri, 15 Aug 2008 10:54:20 -0500
>> 
>> There should probably be a specific function for this:
>> 
>> (decode-coding-string-as-unibyte "íóëü" 'cp1251)
>> 
>> ditto for decode-coding-region.  Should I add it or is that not
>> generally useful?

EZ> Personally, I think it's not useful, since decode-coding-region and
EZ> decode-coding-string are used only on unibyte text.  But feel free to
EZ> raise this on emacs-devel.

How would you recommend decoding text from particular encodings?  Given
text like the one shown above in a buffer, only decode-coding-region
seems to DTRT, and it's not interactive.

Context: I have a file full of CP1251 data and don't want to use Perl's
Encode module because I'm stubborn and think Emacs should handle it :)

On Fri, 15 Aug 2008 20:06:59 +0400 Dmitry Dzhus <dima@sphinx.net.ru> wrote: 

DD> That was nitpicking somewhat irrelevant to unibyte-multibyte problem:
DD> Ted expected to get «нуль», and it's «íóëü», though the string he
DD> originally provided — «íîëü» — decodes to «ноль»; however, both «нуль»
DD> and «ноль» mean «zero».

Yes, I was translating from Russian and knew the text said "zero" but
didn't remember the correct spelling.  Thanks for checking.

Ted

Re: decode-coding-string question

[Nikolaj Schumacher]

Ted Zlatanov <tzz@lifelogs.com> wrote:

> Context: I have a file full of CP1251 data and don't want to use Perl's
> Encode module because I'm stubborn and think Emacs should handle it :)

Maybe `file-coding-system-alist' or `coding-system-for-read' are of help?


regards,
Nikolaj Schumacher

Re: decode-coding-string question

[David Golden]

Ted Zlatanov wrote:


> Context: I have a file full of CP1251 data and don't want to use
> Perl's Encode module because I'm stubborn and think Emacs should
> handle it :)

Just in case: If you have a file full of cp1251, and you know it's
cp1251, it' s usually  best to just open it as cp1251 in the first
place!

C-x RET c cp1251 C-x C-f myfile.txt

It's typically only if you've got a file full of fragments in 
different encodings (horrible mail spool formats and the like) that
you want to decode and reencode particular subregions of whole files.

If you've already opened a file and its encoding is misdetected,
you can also hit 
C-x RET r cp1251 
to "revert" the buffer to the file reopened in the specified encoding.

Re: decode-coding-string question

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Mon, 18 Aug 2008 08:58:55 -0500
> 
> How would you recommend decoding text from particular encodings?  Given
> text like the one shown above in a buffer, only decode-coding-region
> seems to DTRT, and it's not interactive.

If you mean interactively, i.e. you visited a buffer and then
discovered that it was decoded incorrectly, and the actual encoding is
different, then "C-x RET c cp1251 RET M-x revert-buffer RET" should do
what you want, I think.

> Context: I have a file full of CP1251 data and don't want to use Perl's
> Encode module because I'm stubborn and think Emacs should handle it :)

What about the rest of the file? is it encoded in some other encoding?
If not, then the above recipe should do.  If it doesn't, please tell
more details.

> On Fri, 15 Aug 2008 20:06:59 +0400 Dmitry Dzhus <dima@sphinx.net.ru> wrote: 
> 
> DD> That was nitpicking somewhat irrelevant to unibyte-multibyte problem:
> DD> Ted expected to get «нуль», and it's «íóëü», though the string he
> DD> originally provided — «íîëü» — decodes to «ноль»; however, both «нуль»
> DD> and «ноль» mean «zero».
> 
> Yes, I was translating from Russian and knew the text said "zero" but
> didn't remember the correct spelling.

AFAIK, both spellings are right.

Re: decode-coding-string question

[Kevin Rodgers]

Eli Zaretskii wrote:
>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Mon, 18 Aug 2008 08:58:55 -0500
>>
>> How would you recommend decoding text from particular encodings?  Given
>> text like the one shown above in a buffer, only decode-coding-region
>> seems to DTRT, and it's not interactive.
> 
> If you mean interactively, i.e. you visited a buffer and then
> discovered that it was decoded incorrectly, and the actual encoding is
> different, then "C-x RET c cp1251 RET M-x revert-buffer RET" should do
> what you want, I think.

aka `C-x RET r cp1251 RET', right?

-- 
Kevin Rodgers
Denver, Colorado, USA

Re: decode-coding-string question

[Ted Zlatanov]

On Mon, 18 Aug 2008 22:11:37 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Mon, 18 Aug 2008 08:58:55 -0500
>> 
>> How would you recommend decoding text from particular encodings?  Given
>> text like the one shown above in a buffer, only decode-coding-region
>> seems to DTRT, and it's not interactive.

EZ> If you mean interactively, i.e. you visited a buffer and then
EZ> discovered that it was decoded incorrectly, and the actual encoding is
EZ> different, then "C-x RET c cp1251 RET M-x revert-buffer RET" should do
EZ> what you want, I think.

>> Context: I have a file full of CP1251 data and don't want to use Perl's
>> Encode module because I'm stubborn and think Emacs should handle it :)

EZ> What about the rest of the file? is it encoded in some other encoding?
EZ> If not, then the above recipe should do.  If it doesn't, please tell
EZ> more details.

I often have to open mangled files with mixed-up encodings; it's
convenient to set the coding-system after I look at the text, and only
apply it to a region.

On Mon, 18 Aug 2008 18:45:29 +0100 David Golden <david.golden@oceanfree.net> wrote: 
[similar advice]

I see it's an uncommon operation, so I'll use the make-unibyte-string
recipe Eli and others recommended.

Thanks
Ted

Re: decode-coding-string question

[David Golden]

Ted Zlatanov wrote:

 
> I often have to open mangled files with mixed-up encodings; it's
> convenient to set the coding-system after I look at the text, and only
> apply it to a region.
> 

Uhm.

M-x recode-region 

also exists (at least in CVS emacs, no idea when it was introduced).

Re: decode-coding-string question

[Ted Zlatanov]

On Tue, 19 Aug 2008 00:01:07 +0100 David Golden <david.golden@oceanfree.net> wrote: 

DG> Ted Zlatanov wrote:
>> I often have to open mangled files with mixed-up encodings; it's
>> convenient to set the coding-system after I look at the text, and only
>> apply it to a region.
>> 

DG> M-x recode-region 

DG> also exists (at least in CVS emacs, no idea when it was introduced).

That's exactly what I needed (and it's nicely interactive too).  I did
apropos for 'decod' and didn't see 'recode-*'.

Thanks
Ted

Re: decode-coding-string question

[Dmitry Dzhus]

Eli Zaretskii wrote:

>> From: Dmitry Dzhus <dima@sphinx.net.ru>
>> Date: Fri, 15 Aug 2008 02:19:11 +0400
>> 
>> Ted Zlatanov wrote:
>> 
>> > This should decode to нуль but doesn't (I get the same string instead):
>> >
>> > (decode-coding-string "íîëü" 'cp1251)
>> >
>> > Am I missing something obvious?  Do I need to encode the string to
>> > something else?
>> 
>> 0. «íóëü», not «íîëü»
>
> That's not important, the original problem remains, even with a
> different spelling of the word.

That was nitpicking somewhat irrelevant to unibyte-multibyte problem:
Ted expected to get «нуль», and it's «íóëü», though the string he
originally provided — «íîëü» — decodes to «ноль»; however, both «нуль»
and «ноль» mean «zero».
-- 
Happy Hacking.

http://sphinx.net.ru
む

Re: decode-coding-string question

[David Golden]

Ted Zlatanov wrote:

> This should decode to нуль but doesn't (I get the same string
> instead):
> 
> (decode-coding-string "íîëü" 'cp1251)
> 
> Am I missing something obvious?  Do I need to encode the string to
> something else?
>
Guessing you're using a new multibyte/unicode emacs, and noting that I
do not currently fully understand  emacs encoding handling, but...
probably - try:

(decode-coding-string (encode-coding-string "íîëü" 'iso-8859-1) 'cp1251)