Re: undecided vs utf-8

[Kenichi Handa <handa@m17n.org>]

In article <m362wco5zx.fsf@quimbies.gnus.org>, Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Kenichi Handa <handa@m17n.org> writes:
> > It's perhaps because you are in some of iso-8859-1 locale.

> I don't think I am, but I might be wrong.  There are so many locale
> variables, but I always try to put my machines into "C" locale.

??? When the locale is "C", emacs prefers utf-8 the most.

% LANG=C emacs -Q -batch --eval '(message "%s" (car (coding-system-priority-list)))'

should prints utf-8.

> > I don't want to add such a heuristic in
> > decode-coding-string/region (the lowest functions available
> > from Lisp).  Please note that above sequence is also valid
> > as Big5.  If people are in Big5 locale, it's hard to answer
> > which of utf-8 or big5 is preferred unless we implement NLP
> > system.

> I don't know how the big5 encoding looks like, but when it comes to
> iso-8859-1 vs utf-8, then there are many utf-8 strings that are valid
> iso-8859-1 strings, but there are few iso-8859-1 strings that are valid
> utf-8 strings.  Therefore it seems to make sense to prefer utf-8 over
> iso-8859-1.  Perhaps.

Please consider the reason why one is in iso-8859-1 locale
nowadays.  Isn't it because he prefers iso-8859-1 orver
utf-8?

> > Perhaps making an upper layer function that will accept a
> > list of preferred coding systems will be good; something
> > like this.
> >
> > (defun detect-and-decode-coding-string (str preferred)
> >   (let ((detected (detect-coding-string str))
> > 	decided)
> >     (while (and preferred (not decided)) 
> >       (if (memq (car preferred) detected)
> > 	  (setq decided (car preferred))
> > 	(setq preferred (cdr preferred))))
> >     (decode-coding-string str (or decided (car detected)))))

> Well, this is about `undecided', and the C layer does DWIM-ish
> processing when you ask it to decode `undecided', doesn't it?

I don't know which Emacs' behaviour you describe as DWIM-ish.

> The use case that made me look into this -- erc -- is somewhat special.
> The irc protocol does no charset tagging, and some clients send some
> charsets, and some send others, which is why erc uses `undecided' as the
> default coding system.  Typically on a channel you'll see somebody using
> a local (iso-8859-* is popular) charset, and others using utf-8.

> Perhaps the fix here isn't to do anything with `undecided' per se, but
> just fix erc.  It's trivial enough -- just have the default be, say,
> `undecided-or-utf-8', and then handle that by running
> `detect-coding-string' over it, see whether it's utf-8, and then either
> use that or pass `undecided' down into the decoding functions.

> I don't know.  What do you think?

I think the best way is to provide users an easy way to
specify a correct coding-system when they see a decoding
error as well as the method to customize the default
coding-system for erc.

---
Kenichi Handa
handa@m17n.org