Coding system prefer

Coding system prefer

[Maze]

Hello!
I need to work with files in cp1251 and cp866 and i want emacs
autodetect coding system. For correct file opening i try to use coding-
system-priority-list.

I do: prefer-coding-system cp866-dos
Looking C-h С :
Priority order for recognizing coding systems when reading files:
  1. cp866
  2. utf-8 (alias: mule-utf-8)
  3. iso-2022-7bit
  4. iso-2022-7bit-lock (alias: iso-2022-int-1)
etc...
Opening files in cp866 is working correct.

Next, do prefer-coding-system cp1251-dos

Result:
Priority order for recognizing coding systems when reading files:
  1. windows-1251 (alias: cp1251 windows-1251)
  2. utf-8 (alias: mule-utf-8)
  3. iso-2022-7bit
  4. iso-2022-7bit-lock (alias: iso-2022-int-1)
etc...

Opening files in cp1251 is working correct, but in cp866 is not. :(

Then i used (set-coding-system-priority 'cp866 'cp1251) with the same
effect.

How can i use both coding together?

Re: Coding system prefer

[Jason Rumney]

On Mar 3, 7:46 pm, Maze <maze...@gmail.com> wrote:

> I need to work with files in cp1251 and cp866
...
> How can i use both coding together?

You can't. Both these encodings are indistinguishable from binary
data, so all files will be auto-detected as whichever is first in the
priority list.

Re: Coding system prefer

[Teemu Likonen]

On 2009-03-03 06:58 (-0800), Jason Rumney wrote:

> On Mar 3, 7:46 pm, Maze <maze...@gmail.com> wrote:
>> I need to work with files in cp1251 and cp866
> ...
>> How can i use both coding together?
>
> You can't. Both these encodings are indistinguishable from binary
> data, so all files will be auto-detected as whichever is first in the
> priority list.

If the files can be detected by filename then variable auto-coding-alist
may be useful. As the name says it's an association list consisting of
items "(REGEXP . CODING-SYSTEM)". There is also the variable
auto-coding-functions which is a list of functions for determining
conding systems. For more info, see:

    C-h v auto-coding-alist RET
    C-h v auto-coding-functions RET

Re: Coding system prefer

[Maze]

On 3 мар, 19:18, Teemu Likonen <tliko...@iki.fi> wrote:

> If the files can be detected by filename then variable auto-coding-alist
> may be useful.

No. All files are *.txt :(
Can't  believe, that Emacs don't auto-detect these encodings...
May be I can use any external package for Emacs to resolve this
problem?

Re: Coding system prefer

[Peter Dyballa]

Am 03.03.2009 um 12:46 schrieb Maze:

> How can i use both coding together?

By using file local variables à la

	<comment> -*- mode: LaTeX; coding: utf-8; -*-

or, when you're lucky, associating particular encodings with file  
name extensions.

--
Greetings

   Pete

Every instructor assumes that you have nothing else to do except  
study for that instructor's course.
				– Fourth Law of Applied Terror

Re: Coding system prefer

[Peter Dyballa]

Am 03.03.2009 um 21:55 schrieb Maze:

> Can't  believe, that Emacs don't auto-detect these encodings...

How do you detect the encodings?

Can you write the algorithm? I mean, not in Emacs Lisp, just in English?

--
Greetings

   Pete

One cannot live by television, video games, top ten CDs, and dumb  
movies alone.
				– Amiri Baraka, 1999

Re: Coding system prefer

[Fedor Khod'kov]

Tue, Mar 03, 2009 at 12:55:32PM -0800, Maze wrote:
> On 3 мар, 19:18, Teemu Likonen <tliko...@iki.fi> wrote:
> 
> > If the files can be detected by filename then variable auto-coding-alist
> > may be useful.
> 
> No. All files are *.txt :(
> Can't  believe, that Emacs don't auto-detect these encodings...
> May be I can use any external package for Emacs to resolve this
> problem?

Probably this can help:
auto-enca.el --- automatically detect coding system with Enca
http://www.archivum.info/gnu.emacs.sources/2007-06/msg00035.html

It uses external utility called "enca" to detect charset.  This
utility can be found here:
http://packages.debian.org/source/lenny/enca
--

Re: Coding system prefer

[Sergio]

On 4 мар, 15:11, Peter Dyballa <Peter_Dyba...@Web.DE> wrote:
> Am 03.03.2009 um 21:55 schrieb Maze:

>> Can't  believe, that Emacs don't auto-detect these encodings...

> How do you detect the encodings?
> Can you write the algorithm? I mean, not in Emacs Lisp, just in English?

The FAR file manager, http://en.wikipedia.org/wiki/FAR_Manager does it
quite
reliably using statistics about the character frequency distribution.
The tables themselves are quite small (about 1 Kbyte); the 8-bit
encodings are language-dependent, the Unicode encodings are
autodetected in a more general way.

Here are the tables I have for Cyrillic:

,----
| c:/Program Files/FAR/Addons/Tables/Cyrillic:
| total used in directory 12 available 29790096
| drwxrwxrwx  1 spokrovs Domain Users    0 09-22 19:24 .
| drwxrwxrwx  1 spokrovs Domain Users    0 09-22 19:24 ..
| drwxrwxrwx  1 spokrovs Domain Users    0 09-22 19:24 E-Mail Double
Conversion
| -rw-rw-rw-  1 spokrovs Domain Users 1079 2005-07-04  DKOI8
(Mainframe).reg
| -rw-rw-rw-  1 spokrovs Domain Users 1063 2005-07-01  DM (Amiga).reg
| -rw-rw-rw-  1 spokrovs Domain Users  723 2005-07-04  Descript.ion
| -rw-rw-rw-  1 spokrovs Domain Users 1111 2006-02-13  Dist.Rus.reg
| -rw-rw-rw-  1 spokrovs Domain Users 1153 2006-02-13  Dist.Ukr.reg
| -rw-rw-rw-  1 spokrovs Domain Users 1108 2006-03-23  ISO-8859-5.reg
| -rw-rw-rw-  1 spokrovs Domain Users 1084 2006-03-23  KOI8-R.reg
| -rw-rw-rw-  1 spokrovs Domain Users 1016 2006-03-23  KOI8-U.reg
| -rw-rw-rw-  1 spokrovs Domain Users 1186 2006-03-23  Macintosh
Standard.reg
| -rw-rw-rw-  1 spokrovs Domain Users 1104 2005-07-01  RUSCII (GOST
Ukrainian).reg
| -rw-rw-rw-  1 spokrovs Domain Users 1073 2006-03-23
Windows-1251.reg
`----

I think there is a similar package in emacs, although its emphasis is
on language recognition rather then on the encoding.

--
Sergei

Re: Coding system prefer

[Fedor Khod'kov]

Tue, Mar 03, 2009 at 12:55:32PM -0800, Maze wrote:
> On 3 мар, 19:18, Teemu Likonen <tliko...@iki.fi> wrote:
> 
> > If the files can be detected by filename then variable auto-coding-alist
> > may be useful.
> 
> No. All files are *.txt :(
> Can't  believe, that Emacs don't auto-detect these encodings...
> May be I can use any external package for Emacs to resolve this
> problem?

Probably this can help:
auto-enca.el --- automatically detect coding system with Enca
http://www.archivum.info/gnu.emacs.sources/2007-06/msg00035.html

It uses external utility called "enca" to detect charset.  This
utility can be found here:
http://packages.debian.org/source/lenny/enca
--

Re: Coding system prefer

[Eli Zaretskii]

> From: Maze <mazebox@gmail.com>
> Date: Tue, 3 Mar 2009 12:55:32 -0800 (PST)
> 
> Can't  believe, that Emacs don't auto-detect these encodings...

Apparently, Emacs developers don't see this as a serious problem.

How about if you donate a solution to it?

Re: Coding system prefer

[Miles Bader]

Sergio <sergio.pokrovskij@gmail.com> writes:
> The FAR file manager, http://en.wikipedia.org/wiki/FAR_Manager does it
> quite reliably using statistics about the character frequency
> distribution.

Does that work for anything except text files containing prose?

-Miles

-- 
`Suppose Korea goes to the World Cup final against Japan and wins,' Moon said.
`All the past could be forgiven.'   [NYT]

Re: Coding system prefer

[Sergio]

On Mar 5, 10:19 am, Miles Bader <mi...@gnu.org> wrote:
> Sergio <sergio.pokrovs...@gmail.com> writes:
>> The FAR file manager,http://en.wikipedia.org/wiki/FAR_Managerdoes it
>> quite reliably using statistics about the character frequency
>> distribution.

> Does that work for anything except text files containing prose?

Yes, it does.

Of course it does not work for a binary file; but it works fine for a
text file in formal language, like C program with Russian strings or a
text with HTML markup.

I never explored the internals, but I guess that normally one can
ignore the ASCII part; only codes greater than 127 really matter.  Of
these, one can easily detect utf-8 or other unicode encoding (at least
for the alphabetic planes; I never need the CJK part).  And there are
8-bit codes, in which the higher part is characteristic.

And usually the noise part (like markup or formal language statements)
is in ASCII.

I never needed EBCDIC or any other encoding which is not a superset of
ASCII.

--
Sergei

Re: Coding system prefer

[Maze]

Thanks to All!