bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Eli Zaretskii]

The new rmail-mime feature causes Rmail to set
buffer-file-coding-system to an incorrect value.  Expected result:
buffer-file-coding-system is according to the charset= header, but in
fact buffer-file-coding-system is almost always set to undecided.

This is a regression from Emacs 23.2.90.

Also, once the headers were decoded by rfc2047-decode-region, it would
be good to set buffer-file-coding-system to last-coding-system-used,
if the buffer's encoding is still undecided.


In GNU Emacs 23.2.91.1 (i386-mingw-nt5.1.2600)
 of 2010-12-11 on HOME-C4E4A596F7
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4)'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ENU
  value of $XMODIFIERS: nil
  locale-coding-system: cp1255
  default enable-multibyte-characters: t

Major mode: RMAIL

Minor modes in effect:
  diff-auto-refine-mode: t
  desktop-save-mode: t
  show-paren-mode: t
  display-time-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  line-number-mode: t

Recent input:
C-x b I N B <tab> <return> <C-next> <C-next> <C-next> 
<C-next> <C-next> <C-next> <C-next> <C-next> <C-next> 
<M-end> <help-echo> M-x r e p o r t <tab> <return>

Recent messages:
setting up indent stuff
Indentation variables are now local.
Indentation setup for shell type sh
Setting up indent for shell type sh
setting up indent stuff
Indentation variables are now local.
Indentation setup for shell type sh
Note: file is write protected [3 times]
Wrote d:/usr/eli/.emacs.desktop.lock
Desktop: 234 buffers restored.

Load-path shadows:
None found.

Features:
(shadow mailalias sendmail emacsbug ld-script sh-script executable
dired-x dired-aux dired tcl comint ring make-mode nxml-uchnm rng-xsd
xsd-regexp rng-cmpct rng-nxml rng-valid rng-loc rng-uri rng-parse
nxml-parse rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode
nxml-outln nxml-rap nxml-util nxml-glyph nxml-enc xmltok sgml-mode
arc-mode archive-mode parse-time conf-mode newcomment jka-compr
tar-mode add-log diff-mode texinfo generic vc-cvs cc-mode cc-fonts
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
regexp-opt flyspell ispell org-wl org-w3m org-vm org-rmail org-mhe
org-mew org-irc org-jsinfo org-infojs org-html org-exp org-exp-blocks
org-agenda org-info org-gnus org-bibtex org-bbdb org byte-opt warnings
bytecomp byte-compile advice help-fns advice-preload org-footnote
org-src org-list org-faces org-compat org-macs noutline outline
easy-mmode vc-bzr info rmailsum rmailmm message ecomplete rfc822 mml
easymenu mml-sec password-cache mm-decode mm-bodies mm-encode mailcap
mailabbrev nnheader gnus-util netrc gmm-utils wid-edit mailheader
canlock sha1 hex-util hashcash mail-parse rfc2231 rmail rfc2047
rfc2045 ietf-drums time-date qp mm-util mail-prsvr mail-utils desktop
server filecache saveplace generic-x paren battery time tooltip
ediff-hook vc-hooks lisp-float-type mwheel dos-w32 disp-table ls-lisp
w32-win w32-vars tool-bar dnd fontset image fringe lisp-mode register
page menu-bar rfn-eshadow timer select scroll-bar mldrag mouse
jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
loaddefs button minibuffer faces cus-face files text-properties
overlay md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process multi-tty
emacs)

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Kenichi Handa]

In article <83ei9miqef.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> The new rmail-mime feature causes Rmail to set
> buffer-file-coding-system to an incorrect value.  Expected result:
> buffer-file-coding-system is according to the charset= header, but in
> fact buffer-file-coding-system is almost always set to undecided.

> This is a regression from Emacs 23.2.90.

> Also, once the headers were decoded by rfc2047-decode-region, it would
> be good to set buffer-file-coding-system to last-coding-system-used,
> if the buffer's encoding is still undecided.

I'm now working on various bug-fixing and improvement of
rmail's mime handling, and fixing the above is included in
the work.  The problem is that which coding system to set if
the header and each text part of multipart have different
"charset".  At the moment, I'm going to commit a code that
set the FIRST coding system used for decoding some part
(header, parts of multipart, forwarded message, etc).

---
Kenichi Handa
handa@m17n.org

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: 7626@debbugs.gnu.org
> Date: Mon, 13 Dec 2010 09:29:01 +0900
> 
> I'm now working on various bug-fixing and improvement of
> rmail's mime handling, and fixing the above is included in
> the work.

Thank you.

> The problem is that which coding system to set if the header and
> each text part of multipart have different "charset".  At the
> moment, I'm going to commit a code that set the FIRST coding system
> used for decoding some part (header, parts of multipart, forwarded
> message, etc).

That might be a good logic.  The important thing here is to DTRT as
much as possible wrt replying to the message: rmail-reply sets the
buffer for composing the response to be encoded in the same encoding
as the original message.  So whatever you do in the Rmail buffer when
you display the message should yield an encoding most probably
suitable for the reply.

TIA

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Eli Zaretskii]

The new Rmail/Rmail-MIME code (revision 100323 on the emacs-23 branch)
solved most of the problems originally reported here, but it still has
some issues.  Here are the problems I see with the new code, based on
less than an hour of testing:

 . If the message has an explicit specification of its charset, the
   EOL type of the RMAIL buffer is left unspecified (displayed as `:'
   in the mode line).  By contrast, when there's no charset= spec, the
   EOL type is correctly set to -unix.

 . Sometimes, the RMAIL buffer's encoding is set to windows-1252,
   which is not present anywhere in the message headers.  The only
   source of non-ASCII characters in the 2 messages where I saw this
   problem was in the RFC 2047 encoded addresses, but those used
   8859-1, as in "Jan =?iso-8859-1?Q?Dj=E4rv?=", not windows-1252.
   `describe-text-properties' indeed shows the charset of the decoded
   address as windows-1252, so the reason is probably somewhere in
   `rfc2047-decode-region'.

Thanks.

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Kenichi Handa]

In article <83tyhsq395.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

>  . If the message has an explicit specification of its charset, the
>    EOL type of the RMAIL buffer is left unspecified (displayed as `:'
>    in the mode line).  By contrast, when there's no charset= spec, the
>    EOL type is correctly set to -unix.

I think the buffer-file-coding-system of rmail buffer should
leave the EOL type unspecified in any cases.  So I added
this code:

	  (set-buffer-file-coding-system
	   (coding-system-base rmail-mime-coding-system) t t))

---
Kenichi Handa
handa@m17n.org

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: 7626@debbugs.gnu.org
> Date: Fri, 14 Jan 2011 13:28:10 +0900
> 
> In article <83tyhsq395.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> >  . If the message has an explicit specification of its charset, the
> >    EOL type of the RMAIL buffer is left unspecified (displayed as `:'
> >    in the mode line).  By contrast, when there's no charset= spec, the
> >    EOL type is correctly set to -unix.
> 
> I think the buffer-file-coding-system of rmail buffer should
> leave the EOL type unspecified in any cases.

Why would this be useful?  In any case, this is a change in behavior
from previous Rmail versions.

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Kenichi Handa]

In article <E1PdnM6-00077h-27@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> > I think the buffer-file-coding-system of rmail buffer should
> > leave the EOL type unspecified in any cases.

> Why would this be useful?  In any case, this is a change in behavior
> from previous Rmail versions.

Rmail-mime does not perform file I/O, but decodes a text
that is already stored in a mbox buffer.  So, we don't have
any information about by what kind of EOL format the decoded
text should be encoded out to a file.  If we leave the EOL
type unspecified, it will be encoded by whatever EOL format
defined for the current system.

---
Kenichi Handa
handa@m17n.org

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: 7626@debbugs.gnu.org
> Date: Mon, 17 Jan 2011 10:22:35 +0900
> 
> Rmail-mime does not perform file I/O, but decodes a text
> that is already stored in a mbox buffer.

But the mbox format uses Unix EOLs.  In fact, Rmail will barf if you
say "C-u g FILE RET", and FILE has DOS EOLs.  So the EOL format is
known in advance in this case.

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Kenichi Handa]

In article <E1Pemui-00053A-Iv@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > Rmail-mime does not perform file I/O, but decodes a text
> > that is already stored in a mbox buffer.

> But the mbox format uses Unix EOLs.

On Windows too?  I didn't know that.

> In fact, Rmail will barf if you
> say "C-u g FILE RET", and FILE has DOS EOLs.  So the EOL format is
> known in advance in this case.

Are you arguing that typing 'o' (rmail-output) will write
DOS EOL file when EOL type of buffer-file-coding-system of
rmail buffer is undecided and the system_eol_type is DOS?
Then, isn't it a bug of rmail-output?

The reason I decided to leave EOL type undecided is for the
case of M-x write-region on rmail buffer.  In that case, I
think, following system_eol_type is the right thing.

---
Kenichi Handa
handa@m17n.org

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: 7626@debbugs.gnu.org
> Date: Mon, 17 Jan 2011 20:35:38 +0900
> 
> In article <E1Pemui-00053A-Iv@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > But the mbox format uses Unix EOLs.
> 
> On Windows too?  I didn't know that.

It must, because Rmail reads the mbox file or stream with
no-conversion.

> > In fact, Rmail will barf if you
> > say "C-u g FILE RET", and FILE has DOS EOLs.  So the EOL format is
> > known in advance in this case.
> 
> Are you arguing that typing 'o' (rmail-output) will write
> DOS EOL file when EOL type of buffer-file-coding-system of
> rmail buffer is undecided and the system_eol_type is DOS?
> Then, isn't it a bug of rmail-output?

That's a separate issue.  I just don't like seeing a buffer whose file
was read with no-conversion show ":" as the EOL mnemonic.  It's not a
catastrophe, I just don't see why we should change this aspect of
Rmail which is how it behaved for several Emacs releases.

> The reason I decided to leave EOL type undecided is for the
> case of M-x write-region on rmail buffer.  In that case, I
> think, following system_eol_type is the right thing.

But we don't behave like that with buffers that visit files, do we?
Why is this use-case different?

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Kenichi Handa]

In article <E1Peo84-0006Nu-O4@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> That's a separate issue.  I just don't like seeing a buffer whose file
> was read with no-conversion show ":" as the EOL mnemonic.

After the change of rmail to mbox-base, the buffer whose
file was read with no-conversion is " *message-viewer
RMAIL*", and the buffer "RMAIL" is what showing the decoding
result of a part of " *message-viewer RMAIL*".  Even if you
see "-1:" or "-1(unix)" in the modeline, it doesn't mean the
mbox file is read with iso-8859-1.

So, I don't understand why you so much care EOL part even if
you don't care that the text-encoding part is shown as "1"
(meaning iso-8859-1) instead of "=" (meaning no-conversion).

> It's not a
> catastrophe, I just don't see why we should change this aspect of
> Rmail which is how it behaved for several Emacs releases.

I haven't realized that the change results in the actual
difference on screen on Windows.  On GNU/Linux, both *-unix
and undecided are indicated by ":".

Anyway, if Windows users are already familiar with seeing
"(unix)" in the modeline of RMAIL buffer, don't want to
change it, and want to make M-x write-region to write out a
region of RMAIL buffer with Unix-style EOL as before, I'll
change the current behavior.

> > The reason I decided to leave EOL type undecided is for the
> > case of M-x write-region on rmail buffer.  In that case, I
> > think, following system_eol_type is the right thing.

> But we don't behave like that with buffers that visit files, do we?

Of course no, because in that case, it's the right thing to
follow the file's EOL format.  But, when you open a new
buffer, write a few lines of text, and save some region into
a file, the EOL type of the saved file is CRLF on Windows.

> Why is this use-case different?

As I wrote above, RMAIL buffer's buffer-file-coding-system
doesn't reflect how the corresponding mbox file is read.

---
Kenichi Handa
handa@m17n.org

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: 7626@debbugs.gnu.org
> Date: Tue, 18 Jan 2011 20:44:21 +0900
> 
> I haven't realized that the change results in the actual
> difference on screen on Windows.  On GNU/Linux, both *-unix
> and undecided are indicated by ":".
> 
> Anyway, if Windows users are already familiar with seeing
> "(unix)" in the modeline of RMAIL buffer, don't want to
> change it, and want to make M-x write-region to write out a
> region of RMAIL buffer with Unix-style EOL as before, I'll
> change the current behavior.

I don't know about others.  I'm familiar (and I customized the Unix
EOL mnemonic to just "/").

It's up to you.  You can close the bug if you want.

Thanks for working on this.

bug#7626: 23.2.91; Rmail shows incorrect message encoding in the mode line

[Kenichi Handa]

In article <E1PfDP9-0003ub-E7@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> > Anyway, if Windows users are already familiar with seeing
> > "(unix)" in the modeline of RMAIL buffer, don't want to
> > change it, and want to make M-x write-region to write out a
> > region of RMAIL buffer with Unix-style EOL as before, I'll
> > change the current behavior.

> I don't know about others.  I'm familiar (and I customized the Unix
> EOL mnemonic to just "/").

> It's up to you.  You can close the bug if you want.

Then please just advice me which is better for general
Windows users; write-region on RMAIL buffer writes out a
file of DOS-style EOL or Unix-style EOL.  If it's the
former, I'll keep the current behavior.  Otherwise I'll
change it back.

---
Kenichi Handa
handa@m17n.org