bug#1051: 23.0.60; rmail decoding bug

bug#1051: 23.0.60; rmail decoding bug

[Richard M. Stallman]

Rmail decodes this message (and many others like it)
incorrectly.  Each pair of quoted-printable characters is supposed
to convert to one character in the Emacs buffer, but instead
it shows up as two.

    From dorascilipoti@alice.it  Sun Sep 28 14:33:13 2008
    Return-path: <dorascilipoti@alice.it>
    Envelope-to: rms@gnu.org
    Delivery-date: Sun, 28 Sep 2008 14:33:13 -0400
    Received: from mx10.gnu.org ([199.232.76.166]:39989)
	    by fencepost.gnu.org with esmtp (Exim 4.67)
	    (envelope-from <dorascilipoti@alice.it>)
	    id 1Kk14z-0002qJ-3Q
	    for rms@gnu.org; Sun, 28 Sep 2008 14:33:13 -0400
    Received: from Debian-exim by monty-python.gnu.org with spam-scanned (Exim 4.60)
	    (envelope-from <dorascilipoti@alice.it>)
	    id 1Kk171-0005Bt-Ma
	    for rms@gnu.org; Sun, 28 Sep 2008 14:35:22 -0400
    X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on monty-python
    X-Spam-Level: 
    X-Spam-Status: No, score=-0.6 required=5.0 tests=AWL,BAYES_00,
	    DNS_FROM_RFC_ABUSE,DNS_FROM_RFC_POST autolearn=no version=3.1.0
    Received: from smtp-out28.alice.it ([85.33.2.28]:4844)
	    by monty-python.gnu.org with esmtp (Exim 4.60)
	    (envelope-from <dorascilipoti@alice.it>)
	    id 1Kk171-0005B3-6H
	    for rms@gnu.org; Sun, 28 Sep 2008 14:35:19 -0400
    Received: from FBCMMO02.fbc.local ([192.168.68.196]) by smtp-out28.alice.it with Microsoft SMTPSVC(6.0.3790.1830);
	     Sun, 28 Sep 2008 20:35:17 +0200
    Received: from FBCMCL01B03.fbc.local ([192.168.69.84]) by FBCMMO02.fbc.local with Microsoft SMTPSVC(6.0.3790.1830);
	     Sun, 28 Sep 2008 20:35:17 +0200
    Received: from [192.168.1.101] ([79.16.197.32]) by FBCMCL01B03.fbc.local with Microsoft SMTPSVC(6.0.3790.1830);
	     Sun, 28 Sep 2008 20:35:14 +0200
    Subject: Re: Hamaca
    From: Dora Scilipoti <dorascilipoti@alice.it>
    Reply-To: dorascilipoti@alice.it
    To: rms@gnu.org
    Cc: dorascilipoti@gmail.com
    In-Reply-To: <E1KjzUp-0002Ad-Un@fencepost.gnu.org>
    References: <A071D5B468700B439E1980A52A399B5D01ED0F83@FBCMST05V06.fbc.local>
	     <E1KjzUp-0002Ad-Un@fencepost.gnu.org>
    Content-Type: text/plain; charset=utf-8
    Date: Sun, 28 Sep 2008 20:37:59 +0200
    Message-Id: <1222627079.4471.16.camel@Osiris>
    Mime-Version: 1.0
    X-Mailer: Evolution 2.10.3 
    Content-Transfer-Encoding: quoted-printable
    X-OriginalArrivalTime: 28 Sep 2008 18:35:15.0239 (UTC) FILETIME=[F3002B70:01C92198]
    X-detected-operating-system: by monty-python.gnu.org: Windows 2000 SP4, XP SP1+


    > Una vez entrada como root, puedes hacer `su dora' para cambiar a la
    > cuenta `dora'.

    Claro, pero el problema es que no puedo lanzar ning=C3=BAn programa desde m=
    i
    cuenta de usuario:=20

    Xlib: connection to ":0.0" refused by server
    Xlib: No protocol specified

    He encontrado una soluci=C3=B3n parcial y provisoria copiando el
    directorio /home/dora/.evolution en /root/.evolution, de esta manera
    puedo ver mis correos anteriores.



In GNU Emacs 23.0.60.3 (mipsel-unknown-linux-gnu, GTK+ Version 2.12.11)
 of 2008-09-25 on lemote-menglan
configured using `configure  'CFLAGS=-O0 -g -Wno-pointer-sign' 'mipsel-unknown-linux-gnu' 'build_alias=mipsel-unknown-linux-gnu' 'host_alias=mipsel-unknown-linux-gnu' 'target_alias=mipsel-unknown-linux-gnu''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Mail

Minor modes in effect:
  shell-dirtrack-mode: t
  gpm-mouse-mode: t
  tooltip-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Recent input:
1 . RET C-c C-c C-d x r C-x o C-u C-n C-u C-n C-n C-@ 
C-u C-n C-n C-n C-n C-n ESC , RET P o k DEL DEL DEL 
P e DEL l e C-a C-k P l DEL O k , SPC p l e a s e SPC 
d SPC s o . C-a C-d ESC f ESC f ESC f o C-c C-c C-u 
C-p C-u C-p C-@ C-u C-n ESC w C-x C-f r e s e / DEL 
DEL u DEL DEL u s e / i t - i s TAB RET C-v C-v C-u 
C-u C-n C-u C-n C-p C-u C-y RET C-d C-d C-d C-d C-d 
C-d C-n C-u C-d C-d C-n C-u C-d C-d C-d C-p C-d C-n 
C-n C-u C-d C-d C-d ESC q C-x C-s C-p C-p C-p C-p C-@ 
C-u C-n C-n ESC w C-x C-f n e w - m a TAB RET ESC > 
C-u C-u C-p C-u C-p C-p C-u C-p C-p C-y C-x C-s C-x 
b R TAB RET C-d x C-d x SPC ESC v SPC o f l a s h TAB 
RET C-d x C-d C-d C-d C-d C-d x C-d C-d x C-x C-s p 
C-d C-d x C-d C-d x o d o r a TAB RET C-x C-f n o ESC 
b i C-e o u t DEL DEL DEL u / DEL t / r m s 0 6 3 DEL 
TAB 2 RET C-s R e : SPC h a DEL DEL H a m a C-s C-a 
C-u C-u C-p C-u C-p C-u C-p C-u C-p C-u C-p C-@ C-v 
C-u C-u C-n C-u C-n C-n ESC w C-x 4 m ESC x r e p o 
r t SPC e TAB RET

Recent messages:
Expunging deleted messages...done
Saving file /home/rms/RMAIL...
Wrote /home/rms/RMAIL
Expunging deleted messages...done
Expunging deleted messages...done
Added to /home/rms/xmail/dora.xmail
Making completion list...
Mark saved where search started
Mark set
Saved text from "From dorascilipoti@alice.it  Sun Sep 28 "

bug#1051: 23.0.60; rmail decoding bug

[Eli Zaretskii]

> From: "Richard M. Stallman" <rms@gnu.org>
> Date: Mon, 29 Sep 2008 13:13:16 -0400
> Cc: 
> 
> Rmail decodes this message (and many others like it)
> incorrectly.  Each pair of quoted-printable characters is supposed
> to convert to one character in the Emacs buffer, but instead
> it shows up as two.

I cannot reproduce this bug, neither in Emacs 23.0.60 built from
today's morning CVS trunk, nor in Emacs 22.3.  I see a single
character for each of these pairs.

Do you see the same problem in "emacs -Q"?

bug#1051: 23.0.60; rmail decoding bug

[Richard M. Stallman]

    I cannot reproduce this bug, neither in Emacs 23.0.60 built from
    today's morning CVS trunk, nor in Emacs 22.3.  I see a single
    character for each of these pairs.

    Do you see the same problem in "emacs -Q"?

I've discovered that the problem does not happen when I visit that
message directly with C-u M-x rmail RET FILENAME RET.

It does happen when I use C-u g FILENAME to get that message
as new mail into my RMAIL file.

Both cases are the same with and without -Q.

This unfortunately leaves me with no test case I can send.

bug#1051: 23.0.60; rmail decoding bug

[Eli Zaretskii]

> From: "Richard M. Stallman" <rms@gnu.org>
> CC: 1051@emacsbugs.donarmstrong.com, emacs-pretest-bug@gnu.org,
> 	bug-gnu-emacs@gnu.org
> Date: Tue, 30 Sep 2008 00:55:08 -0400
> 
> I've discovered that the problem does not happen when I visit that
> message directly with C-u M-x rmail RET FILENAME RET.
> 
> It does happen when I use C-u g FILENAME to get that message
> as new mail into my RMAIL file.
> 
> Both cases are the same with and without -Q.
> 
> This unfortunately leaves me with no test case I can send.

I can reproduce this by reading the test case you sent twice: once
with "C-u M-x rmail", then with "C-u g".  The second time I get the
message displayed incorrectly.

I will debug this and see what I find.

bug#1051: 23.0.60; rmail decoding bug

[Eli Zaretskii]

> Date: Tue, 30 Sep 2008 10:32:08 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-pretest-bug@gnu.org, bug-gnu-emacs@gnu.org,
> 	1051@emacsbugs.donarmstrong.com
> 
> I can reproduce this by reading the test case you sent twice: once
> with "C-u M-x rmail", then with "C-u g".  The second time I get the
> message displayed incorrectly.
> 
> I will debug this and see what I find.

The problem is within mail-unquote-printable-region: it relies on
insert-char to insert a unibyte character, even if the target buffer
is a multibyte buffer.  In Emacs 22.x this works, but not in Emacs 23.

Perhaps Handa-san can suggest what is the best way of inserting
unibyte characters into a multibyte buffer in Emacs 23.  Obviously,
insert-file-contents does that when coding-system-for-read is bound to
no-conversion.

bug#1051: 23.0.60; rmail decoding bug

[Kenichi Handa]

In article <uljxajac7.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> The problem is within mail-unquote-printable-region: it relies on
> insert-char to insert a unibyte character, even if the target buffer
> is a multibyte buffer.  In Emacs 22.x this works, but not in Emacs 23.

> Perhaps Handa-san can suggest what is the best way of inserting
> unibyte characters into a multibyte buffer in Emacs 23.  Obviously,
> insert-file-contents does that when coding-system-for-read is bound to
> no-conversion.

The Lisp API for that is insert-byte.

By the way, we still don't have a proper API for reading an
eight-bit character as byte.  What we can do now for that is
something like these:
  (multibyte-char-to-unibyte (char-after POS))
or
  (encode-char (char-after POS) 'eight-bit)

It may be good to provide byte-after, following-byte, and
preceding-byte (all signal an error if the character is not
an ASCII nor eight-bit character).  What do you think?

---
Kenichi Handa
handa@ni.aist.go.jp

bug#1051: 23.0.60; rmail decoding bug

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: 1051@emacsbugs.donarmstrong.com, rms@gnu.org, emacs-pretest-bug@gnu.org,
>         bug-gnu-emacs@gnu.org
> Date: Tue, 30 Sep 2008 19:59:08 +0900
> 
> In article <uljxajac7.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The problem is within mail-unquote-printable-region: it relies on
> > insert-char to insert a unibyte character, even if the target buffer
> > is a multibyte buffer.  In Emacs 22.x this works, but not in Emacs 23.
> 
> > Perhaps Handa-san can suggest what is the best way of inserting
> > unibyte characters into a multibyte buffer in Emacs 23.  Obviously,
> > insert-file-contents does that when coding-system-for-read is bound to
> > no-conversion.
> 
> The Lisp API for that is insert-byte.

Thanks, this indeed fixes the problem.

Richard, please see if the patch below fixes the problem for you as
well.

> It may be good to provide byte-after, following-byte, and
> preceding-byte (all signal an error if the character is not
> an ASCII nor eight-bit character).  What do you think?

I agree that it would be nice to have such a feature, but perhaps a
single API

  (get-byte POS)

would be enough?  This could default to point if POS is nil or
omitted, and could even read from a string if POS is a string.


2008-09-30  Eli Zaretskii  <eliz@gnu.org>

	* mail/mail-utils.el (mail-unquote-printable-region): Use
	insert-byte instead of insert-char, when the UNIBYTE arg is
	non-nil.

Index: lisp/mail/mail-utils.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/mail/mail-utils.el,v
retrieving revision 1.65
retrieving revision 1.66
diff -c -r1.65 -r1.66
*** lisp/mail/mail-utils.el	6 May 2008 07:22:25 -0000	1.65
--- lisp/mail/mail-utils.el	30 Sep 2008 11:53:21 -0000	1.66
***************
*** 141,148 ****
  		     (if unibyte
  			 (progn
  			   (replace-match "")
! 			   ;; insert-char will insert this as unibyte,
! 			   (insert-char char 1))
  		       (replace-match (make-string 1 char) t t))))
  		  (noerror
  		   (setq failed t))
--- 141,149 ----
  		     (if unibyte
  			 (progn
  			   (replace-match "")
! 			   ;; insert-byte will insert this as a
! 			   ;; corresponding eight-bit character.
! 			   (insert-byte char 1))
  		       (replace-match (make-string 1 char) t t))))
  		  (noerror
  		   (setq failed t))

bug#1051: 23.0.60; rmail decoding bug

[Richard M. Stallman]

    Richard, please see if the patch below fixes the problem for you as
    well.

Yes, it's fixed now.

bug#1051: 23.0.60; rmail decoding bug

[Kenichi Handa]

In article <uiqsdkfmo.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > It may be good to provide byte-after, following-byte, and
> > preceding-byte (all signal an error if the character is not
> > an ASCII nor eight-bit character).  What do you think?

> I agree that it would be nice to have such a feature, but perhaps a
> single API

>   (get-byte POS)

> would be enough?  This could default to point if POS is nil or
> omitted, and could even read from a string if POS is a string.

Ah!  How about something like this?

(defun get-byte (pos &optional string)
  "Return a byte at position POS of the current buffer..
If POS is nil, it defaults to point.
If the second optional arg STRING is non-nil, return a byte in
STRING at index POS.
An error is signaled if the character at POS is not ASCII
nor eight-bit character."
  ...)

---
Kenichi Handa
handa@ni.aist.go.jp