detect-coding-string doesn't return all possibilities

detect-coding-string doesn't return all possibilities

[Jesper Harder]

`detect-coding-string' doesn't return all possible coding systems for a
string.  For example:

(detect-coding-string (encode-coding-string "안녕" 'korean-iso-8bit))
=> (iso-latin-1 iso-latin-1 raw-text chinese-big5 no-conversion)

Clearly korean-iso-8bit is a possible coding system, but this is not
detected.

The same bug happens in CVS Emacs.


In GNU Emacs 21.2.1 (i386-redhat-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2002-04-09 on porky.devel.redhat.com
configured using `configure  i386-redhat-linux --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --with-gcc --with-pop --with-sound'
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.iso88591
  locale-coding-system: iso-latin-1
  default-enable-multibyte-characters: t

Re: detect-coding-string doesn't return all possibilities

[Kenichi Handa]

[-- Attachment #1: Type: text/plain, Size: 1454 bytes --]

Sorry for the late reply.

In article <m3smtw31uq.fsf@defun.localdomain>, Jesper Harder <harder@ifa.au.dk> writes:
> `detect-coding-string' doesn't return all possible coding systems for a
> string.  For example:

> (detect-coding-string (encode-coding-string "¾È³ç" 'korean-iso-8bit))
> => (iso-latin-1 iso-latin-1 raw-text chinese-big5 no-conversion)

> Clearly korean-iso-8bit is a possible coding system, but this is not
> detected.

detect-coding-string doesn't return all possible coding
systems, but returns a possible coding systems Emacs may
automatically detect in the current language environment.

Result of (encode-coding-string "¾È³ç" 'korean-iso-8bit) is
a sequence of bytes of the range 0xA0..0xFF.

For instance, if you are in "Latin-1" language environment,
such a byte sequence is detected as iso-latin-1.  If you are
in "Korean" language environment, it is detected as
korean-iso-8bit, and in that case, detect-coding-string
returns this: (korean-iso-8bit iso-latin-1 raw-text
chinese-big5 no-conversion)

But the docstring of detect-coding-system is surely not
good.  I've just changed the first paragraph as this.  How
is it?

Return a list of possible coding systems used on decoding a byte
sequence containing the bytes in STRING when the coding system
`undecided' is specified.  The list is ordered by priority decided in
the current language environment.

---
Ken'ichi HANDA
handa@m17n.org

[-- Attachment #2: Type: text/plain, Size: 148 bytes --]

_______________________________________________
Bug-gnu-emacs mailing list
Bug-gnu-emacs@gnu.org
http://mail.gnu.org/mailman/listinfo/bug-gnu-emacs

Re: detect-coding-string doesn't return all possibilities

[Jesper Harder]

[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]

Kenichi Handa <handa@m17n.org> writes:

> detect-coding-string doesn't return all possible coding systems, but
> returns a possible coding systems Emacs may automatically detect in
> the current language environment.

Ah, I see.  

Do you know of any other way to decide if using a given coding system
for decoding a string would give a valid result?

A function similar to this would be really useful:

(defun possible-coding-system-for-string-p (str coding-system)
  "Return t if CODING-SYSTEM is a possible coding system for decoding
   STR."
  ...)

The issue comes from a discussion on the Gnus development list (I've
included one of the messages from that thread below).

Gnus does not work very well when using CVS Emacs in an UTF-8 locale,
because a lot of non-MIME capable clients don't include proper charset
information.  This causes Gnus to decode many Latin-1 strings as UTF-8.

It would help a lot if we could detect that a string cannot possibly be
encoded in UTF-8.  I know that it's not always possible to distinguish,
but just detecting strings that are invalid as UTF-8 would be very
helpful.  This doesn't just apply to UTF-8 but to any coding system, of
course.

> But the docstring of detect-coding-system is surely not
> good.  I've just changed the first paragraph as this.  How
> is it?

It's good.



[-- Attachment #2: Type: message/rfc822, Size: 2704 bytes --]

From: Jesper Harder <harder@myrealbox.com>
Subject: Re: charset=macintosh
Date: Sun, 09 Mar 2003 04:56:44 +0100
Message-ID: <m3adg5w5tv.fsf@defun.localdomain>


Simon Josefsson <jas@extundo.com> writes:

> But what if you are saying about UTF-8 clients being MIME capable is
> true, and since UTF-8 is typically never preferred by current emacsen,
> doesn't emacs' current guessing works the best we can hope for?
> Doesn't it detect among ISO-8859-X, ISO-2022 and Big5 properly?

No.  I was hoping we could do something like this (for headers):

(let ((coding-systems (detect-coding-string string)))
  (if (memq default coding-systems)
      (decode-coding-string string default)
    (decode-coding-string string (car coding-systems))))

i.e. if the default coding system is valid for the string, then use
that; otherwise use whatever Emacs thinks is the most likely coding
system.  I think this would be ideal.

But unfortunately `detect-coding-string' _doesn't_ return a complete
list of possible coding systems.  Consider this scenario: 

  I'm using Emacs in a Latin-1 locale.  dk.* newsgroups work fine
  because latin-1 is the default.  But I also subscribe to, say, a few
  Korean newsgroups.  The entry in `gnus-groups-charset-alist':

         ("\\(^\\|:\\)han\\>" euc-kr)

  should take care of selecting the proper default charset.  But *oops*,
  `detect-coding-string' doesn't think that euc-kr is a possible charset
  for a Korean string encoded in euc-kr:

       (detect-coding-string (encode-coding-string "안녕" 'euc-kr))
       => (iso-latin-1 iso-latin-1 raw-text japanese-shift-jis 
           chinese-big5 no-conversion)

So the above approach would fail.

>   2) Users with emacs in UTF-8 prefers UTF-8 too often, even when the
>      data is invalid UTF-8 and another encoding should be selected.
>
> The second situation is a bug, and I hope we can fix this.

Yep, 2) is the most serious problem.  Especially because more and more
people are (often unknowingly) using an UTF-8 locale because Redhat 8
switched to UTF-8 by default.  Those people would experience Gnus as
broken when reading hierarchies like dk.* or de.*.



[-- Attachment #3: Type: text/plain, Size: 148 bytes --]

_______________________________________________
Bug-gnu-emacs mailing list
Bug-gnu-emacs@gnu.org
http://mail.gnu.org/mailman/listinfo/bug-gnu-emacs

Re: detect-coding-string doesn't return all possibilities

[Kenichi Handa]

In article <m3adfyefc7.fsf@defun.localdomain>, Jesper Harder <harder@ifa.au.dk> writes:
> Do you know of any other way to decide if using a given coding system
> for decoding a string would give a valid result?

> A function similar to this would be really useful:

> (defun possible-coding-system-for-string-p (str coding-system)
>   "Return t if CODING-SYSTEM is a possible coding system for decoding
>    STR."
>   ...)

How about this?

(defun decode-coding-string-safely (string coding-system)
  "Decode STRING by CODING-SYSYTEM and return the result.
If STRING contains invalid byte sequence for CODING-SYSTEM, return nil."
  (let* ((decoded (decode-coding-string str coding-system))
	 (charsets (find-charset-string decoded)))
    (and (not (memq 'eight-bit-control charsets))
	 (not (memq 'eight-bit-graphic charsets))
	 decoded)))

---
Ken'ichi HANDA
handa@m17n.org

Re: detect-coding-string doesn't return all possibilities

[Jesper Harder]

Kenichi Handa <handa@m17n.org> writes:

> Jesper Harder <harder@ifa.au.dk> writes:
>
>> Do you know of any other way to decide if using a given coding system
>> for decoding a string would give a valid result?

> How about this?
>
> (defun decode-coding-string-safely (string coding-system)
>   "Decode STRING by CODING-SYSYTEM and return the result.
> If STRING contains invalid byte sequence for CODING-SYSTEM, return nil."

Execellent.  Thank you!