bug#22436: read-coding-system uses wrong default when called from write-file

bug#22436: read-coding-system uses wrong default when called from write-file

[Richard Copley]

From "emacs -Q", enter a ‘ in the scratch buffer (e.g., by typing "C-x
8 ["). Save or write the buffer. The minibuffer prompts "Select coding
system (default chinese-iso-8bit): ".

In GNU Emacs 25.0.50.1 (x86_64-w64-mingw32)
 of 2016-01-22 built on MACHINE
Repository revision: f7dc6d8b5bb318e02a4016d93f8b34de0716f4dc
Windowing system distributor 'Microsoft Corp.', version 10.0.10586
Configured using:
 'configure --prefix /c/emacs/emacs-20160122-175552
 --without-imagemagick --disable-dependency-tracking
 --enable-locallisppath=%emacs_dir%/../site-lisp 'CFLAGS=-Og -g -ggdb''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS

Important settings:
  value of $LANG: ENG
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Saving file c:/temp/xyzzy...
Quit

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec epg epg-config gnus-util mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util help-fns mail-prsvr mail-utils help-mode
easymenu cl-loaddefs pcase cl-lib iso-transl time-date mule-util tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
dos-w32 ls-lisp disp-table w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame
cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote w32notify dbusbind w32
multi-tty make-network-process emacs)

Memory information:
((conses 16 91651 6242)
 (symbols 56 19640 0)
 (miscs 48 37 107)
 (strings 32 16064 5326)
 (string-bytes 1 434641)
 (vectors 16 15778)
 (vector-slots 8 819512 106167)
 (floats 8 162 20)
 (intervals 56 349 42)
 (buffers 976 12))

bug#22436: read-coding-system uses wrong default when called from write-file

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Fri, 22 Jan 2016 19:31:07 +0000
> 
> >From "emacs -Q", enter a ‘ in the scratch buffer (e.g., by typing "C-x
> 8 ["). Save or write the buffer. The minibuffer prompts "Select coding
> system (default chinese-iso-8bit): ".

Why do you think that's wrong?

bug#22436: read-coding-system uses wrong default when called from write-file

[Richard Copley]

On 22 January 2016 at 21:23, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Richard Copley <rcopley@gmail.com>
>> Date: Fri, 22 Jan 2016 19:31:07 +0000
>>
>> >From "emacs -Q", enter a ‘ in the scratch buffer (e.g., by typing "C-x
>> 8 ["). Save or write the buffer. The minibuffer prompts "Select coding
>> system (default chinese-iso-8bit): ".
>
> Why do you think that's wrong?

Of course it's wrong. What do you mean?

bug#22436: read-coding-system uses wrong default when called from write-file

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Fri, 22 Jan 2016 23:23:40 +0000
> Cc: 22436@debbugs.gnu.org
> 
> On 22 January 2016 at 21:23, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Richard Copley <rcopley@gmail.com>
> >> Date: Fri, 22 Jan 2016 19:31:07 +0000
> >>
> >> >From "emacs -Q", enter a ‘ in the scratch buffer (e.g., by typing "C-x
> >> 8 ["). Save or write the buffer. The minibuffer prompts "Select coding
> >> system (default chinese-iso-8bit): ".
> >
> > Why do you think that's wrong?
> 
> Of course it's wrong. What do you mean?

Wrong how? in what way? what did you expect to happen instead?

Your locale's default encoding, cp1252, cannot encode this character,
so Emacs asked you to provide an encoding that can, and offered
chinese-iso-8bit as the default.

An encoding can be considered "wrong" if it fails to encode a
character.  AFAICS, chinese-iso-8bit does succeed in encoding it, so
the above definition of "wrong" doesn't fit.

In addition, there should be a *Warning* buffer displayed where you
have a long list of encodings Emacs knows about that can encode this
character; chinese-iso-8bit is just one of them, but the text there
tells you to select any one of them.  Do you see that buffer
displayed?

Given all this information, please do tell why you think what Emacs
did was wrong, and what did you think Emacs should have done instead.

Thanks.

bug#22436: read-coding-system uses wrong default when called from write-file

[Richard Copley]

On 23 January 2016 at 05:56, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Richard Copley <rcopley@gmail.com>
>> Date: Fri, 22 Jan 2016 23:23:40 +0000
>> Cc: 22436@debbugs.gnu.org
>>
>> On 22 January 2016 at 21:23, Eli Zaretskii <eliz@gnu.org> wrote:
>> >> From: Richard Copley <rcopley@gmail.com>
>> >> Date: Fri, 22 Jan 2016 19:31:07 +0000
>> >>
>> >> >From "emacs -Q", enter a ‘ in the scratch buffer (e.g., by typing "C-x
>> >> 8 ["). Save or write the buffer. The minibuffer prompts "Select coding
>> >> system (default chinese-iso-8bit): ".
>> >
>> > Why do you think that's wrong?
>>
>> Of course it's wrong. What do you mean?
>
> Wrong how? in what way? what did you expect to happen instead?

Wrong in the ordinary sense of the word, in that it is not right.
It is however exactly what I expected.

> Your locale's default encoding, cp1252, cannot encode this character,
> so Emacs asked you to provide an encoding that can, and offered
> chinese-iso-8bit as the default.

Yes.

> An encoding can be considered "wrong" if it fails to encode a
> character.  AFAICS, chinese-iso-8bit does succeed in encoding it, so
> the above definition of "wrong" doesn't fit.

It encodes the character to #xa1 #xae . When I open the file that is
decoded as "¡®", without asking any questions. Isn't that what you
saw? Are you going to tell me that's right too?

> In addition, there should be a *Warning* buffer displayed where you
> have a long list of encodings Emacs knows about that can encode this
> character; chinese-iso-8bit is just one of them, but the text there
> tells you to select any one of them.  Do you see that buffer
> displayed?

Yes, but chinese-iso-8bit is not on the list.

> Given all this information, please do tell why you think what Emacs
> did was wrong,

Do you seriously think it's the right thing to do? That's the last thing
I expected. I don't know how to respond.

> and what did you think Emacs should have done instead.

I think Emacs should pick something less surprising and unhelpful,
like utf-8.

bug#22436: read-coding-system uses wrong default when called from write-file

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Sat, 23 Jan 2016 09:45:39 +0000
> Cc: 22436@debbugs.gnu.org
> 
> > An encoding can be considered "wrong" if it fails to encode a
> > character.  AFAICS, chinese-iso-8bit does succeed in encoding it, so
> > the above definition of "wrong" doesn't fit.
> 
> It encodes the character to #xa1 #xae . When I open the file that is
> decoded as "¡®", without asking any questions. Isn't that what you
> saw? Are you going to tell me that's right too?

Emacs never asks questions when decoding text, only when encoding it.
When it visits a file, it guesses the encoding as best it can, and
that's what you see.

If you type "C-x RET c chinese-iso-8bit RET C-x C-f FILE RET", do you
see that character correctly decoded and correctly displayed?

> > In addition, there should be a *Warning* buffer displayed where you
> > have a long list of encodings Emacs knows about that can encode this
> > character; chinese-iso-8bit is just one of them, but the text there
> > tells you to select any one of them.  Do you see that buffer
> > displayed?
> 
> Yes, but chinese-iso-8bit is not on the list.

It is: that's the first one in the list, gb2312.  It's an alias for
chinese-iso-8bit, which is the formal name.

> > Given all this information, please do tell why you think what Emacs
> > did was wrong,
> 
> Do you seriously think it's the right thing to do? That's the last thing
> I expected. I don't know how to respond.
> 
> > and what did you think Emacs should have done instead.
> 
> I think Emacs should pick something less surprising and unhelpful,
> like utf-8.

Unless it gets some hint from the locale or from customizations, Emacs
does not prefer any encoding to any other encoding.  GB2312 is a very
reasonable choice in some locales, and might be what you want.  How is
Emacs to know, if your locale is US English?  All it knows is that
your default encoding cannot be used.

What Emacs did here is produce a list of all the encodings that can
encode that character, sorted them according what it knows about your
preferences (which is nothing in this case), and then picked up the
first encoding in the resulting list.  And that's what you see.  UTF-8
is also in the list, but it comes later.

If you want Emacs to prefer UTF-8, you can have that.  Just type

  M-x prefer-coding-system RET utf-8 RET

That's it, Emacs will from now on use UTF-8 whenever it can without
asking.  If you repeat your recipe after setting that preference, it
will silently encode the buffer/region with UTF-8, no questions asked.

You can have this customization in your ~/.emacs file, of course, if
you want it permanent.

bug#22436: read-coding-system uses wrong default when called from write-file

[Richard Copley]

Thanks for explaining that. I imagined Emacs would use a
heuristic based on the contents of the file.
If there were a fallback ordering that preferred utf-8 or some
other universally applicable coding system it would be easier
for most users and less confusing for beginners.

bug#22436: read-coding-system uses wrong default when called from write-file

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Sat, 23 Jan 2016 11:53:13 +0000
> Cc: 22436@debbugs.gnu.org
> 
> Thanks for explaining that. I imagined Emacs would use a
> heuristic based on the contents of the file.

There is indeed a heuristic, but it's not very smart, and only works
well when the encoding is blatantly clear.  This is an area where
improvements are welcome.  In any case, when presented with a buffer
that includes a single non-ASCII character that can be encoded by a
dozen of possible encodings, it's hard to be too smart based on the
contents alone.

> If there were a fallback ordering that preferred utf-8 or some
> other universally applicable coding system it would be easier
> for most users and less confusing for beginners.

Such a fallback does exist, see coding-system-priority-list.  What
"M-x prefer-coding-system RET" does is manipulate the priorities so
that the encoding you typed is preferred.

Do we close this report, or is there something else to do about it?

Thanks.

bug#22436: read-coding-system uses wrong default when called from write-file

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Your locale's default encoding, cp1252, cannot encode this character,
> so Emacs asked you to provide an encoding that can, and offered
> chinese-iso-8bit as the default.

While correct, it is a slightly less than helpful default.  Most people
deal (at most) with two charsets: One local, and if not, then Unicode.
(Which would be utf8 on most systems, and possibly ucs16 on Windows, I
dunno.)

So `chinese-iso-8bit' is a surprising default.  Could the default be
improved upon?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22436: read-coding-system uses wrong default when called from write-file

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: Richard Copley <rcopley@gmail.com>,  22436@debbugs.gnu.org
> Date: Sat, 23 Jan 2016 14:12:20 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Your locale's default encoding, cp1252, cannot encode this character,
> > so Emacs asked you to provide an encoding that can, and offered
> > chinese-iso-8bit as the default.
> 
> While correct, it is a slightly less than helpful default.

It's not a default, it's just the first member of a list sorted
according to some comparison function.

> Most people deal (at most) with two charsets: One local, and if not,
> then Unicode.

That's only correct if the encoded text is going to be used on the
same platform.  This email message is an excellent example where this
logic is simply false, because you and I live in different locales,
and "Unicode" means different things to us.

(Anyway, when you say "most people", did you consider how many people
in the world consider chinese-iso-8bit a very good first choice? ;-)

> Which would be utf8 on most systems, and possibly ucs16 on Windows,
> I dunno.

Using UTF-16 on Windows would be disastrous: almost no program,
certainly not those which are ports of GNU software, can do anything
useful with such an encoding.  Emacs is the only exception I know of.

> So `chinese-iso-8bit' is a surprising default.

It's not a default.

> Could the default be improved upon?

We can sort the list differently.  But if all we want is to always get
UTF-8 at the head, there's a much easier way, see my other message
where I mentioned prefer-coding-system.

bug#22436: read-coding-system uses wrong default when called from write-file

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> (Anyway, when you say "most people", did you consider how many people
> in the world consider chinese-iso-8bit a very good first choice? ;-)

:-)

>> Which would be utf8 on most systems, and possibly ucs16 on Windows,
>> I dunno.
>
> Using UTF-16 on Windows would be disastrous: almost no program,
> certainly not those which are ports of GNU software, can do anything
> useful with such an encoding.  Emacs is the only exception I know of.

Right.  I thought I had read somewhere that Windows was UTF-16-ey, but I
know nothing about Windows...

>> So `chinese-iso-8bit' is a surprising default.
>
> It's not a default.
>
>> Could the default be improved upon?
>
> We can sort the list differently.  But if all we want is to always get
> UTF-8 at the head, there's a much easier way, see my other message
> where I mentioned prefer-coding-system.

I guess I'm just saying that I think the least surprising default coding
system here would be utf-8.  So Emacs should offer that as the first
option unless the user has said otherwise.  That is, Emacs should say
utf-8 unless the user has called `prefer-coding-system' with something
else.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22436: read-coding-system uses wrong default when called from write-file

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: rcopley@gmail.com,  22436@debbugs.gnu.org
> Date: Sat, 23 Jan 2016 14:55:50 +0100
> 
> >> Could the default be improved upon?
> >
> > We can sort the list differently.  But if all we want is to always get
> > UTF-8 at the head, there's a much easier way, see my other message
> > where I mentioned prefer-coding-system.
> 
> I guess I'm just saying that I think the least surprising default coding
> system here would be utf-8.

For you, maybe.  What about all those Chinese out there?  What if I'm
writing an email to someone in China?  Etc., etc.

> That is, Emacs should say utf-8 unless the user has called
> `prefer-coding-system' with something else.

Most people who care about that already have their locales use UTF-8.
So the problem largely doesn't exist.  It's not an accident that this
was reported by a Windows user.

Anyway, this stuff worked for years, why should we suddenly change?

bug#22436: read-coding-system uses wrong default when called from write-file

[Richard Copley]

>> > We can sort the list differently.  But if all we want is to always get
>> > UTF-8 at the head, there's a much easier way, see my other message
>> > where I mentioned prefer-coding-system.
>>
>> I guess I'm just saying that I think the least surprising default coding
>> system here would be utf-8.
>
> For you, maybe.  What about all those Chinese out there?  What if I'm
> writing an email to someone in China?  Etc., etc.

What about them, and everybody else? UTF-8 is good enough
for all of us.

Eli, you asked earlier if this report should be closed. I guess so!
As you've explained, it's not a bug. I don't like the default
behaviour, but it's pretty far down the list of things that I don't
like about "emacs -Q".

bug#22436: read-coding-system uses wrong default when called from write-file

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Mon, 25 Jan 2016 22:38:15 +0000
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>, 22436@debbugs.gnu.org
> 
> >> > We can sort the list differently.  But if all we want is to always get
> >> > UTF-8 at the head, there's a much easier way, see my other message
> >> > where I mentioned prefer-coding-system.
> >>
> >> I guess I'm just saying that I think the least surprising default coding
> >> system here would be utf-8.
> >
> > For you, maybe.  What about all those Chinese out there?  What if I'm
> > writing an email to someone in China?  Etc., etc.
> 
> What about them, and everybody else? UTF-8 is good enough
> for all of us.

Not in the Far East, I'm afraid.

> Eli, you asked earlier if this report should be closed. I guess so!
> As you've explained, it's not a bug. I don't like the default
> behaviour, but it's pretty far down the list of things that I don't
> like about "emacs -Q".

OK, closing.  Thanks for pointing that out.