improve rmail's MIME handling

improve rmail's MIME handling

[Kenichi Handa]

A while ago, I reported that I need various settings to make
rmail handle MIME better (i.e. subject decoding/encoding),
and showed my settings including some additional functions,
but it seems that no one is working on this field.  So, I
started to work on it.  The attached is the first (and easy)
patch for summary lines.  Nowadays, not handling MIME
properly is regarded as a bug, and thus I want to install it
for Emacs 23.  What do you think?

---
Kenichi Handa
handa@m17n.org


2010-11-11  Kenichi Handa  <handa@m17n.org>

	* mail/rmailsum.el (rmail-header-summary): Handle multiline
	Subject: field.
	(rmail-summary-line-decoder): Set the default to
	rfc2047-decode-string.

=== modified file 'lisp/mail/rmailsum.el'
--- lisp/mail/rmailsum.el	2010-08-29 16:17:13 +0000
+++ lisp/mail/rmailsum.el	2010-11-11 02:33:42 +0000
@@ -364,12 +364,15 @@
       (aset rmail-summary-vector (1- msgnum) line))
     line))
 
-(defcustom rmail-summary-line-decoder (function identity)
+(require 'rfc2047)
+
+(defcustom rmail-summary-line-decoder (function rfc2047-decode-string)
   "Function to decode a Rmail summary line.
 It receives the summary line for one message as a string
 and should return the decoded string.
 
-By default, it is `identity', which returns the string unaltered."
+By default, it is `rfc2047-decode-string', which decodes MIME-encoded
+subject."
   :type 'function
   :group 'rmail-summary)
 
@@ -589,10 +592,17 @@
 						     (t (- mch 14))))
 				      (min len (+ lo 25)))))))))
    (concat (if (re-search-forward "^Subject:" nil t)
-	       (progn (skip-chars-forward " \t")
-		      (buffer-substring (point)
-					(progn (end-of-line)
-					       (point))))
+	       (let (pos str)
+		 (skip-chars-forward " \t")
+		 (setq pos (point))
+		 (forward-line 1)
+		 (setq str (buffer-substring pos (1- (point))))
+		 (while (looking-at "\\s ")
+		   (setq str (concat str " " 
+				     (buffer-substring (match-end 0)
+						       (line-end-position))))
+		   (forward-line 1))
+		 str)
 	     (re-search-forward "[\n][\n]+" nil t)
 	     (buffer-substring (point) (progn (end-of-line) (point))))
 	   "\n")))

Re: improve rmail's MIME handling

[Andreas Schwab]

Kenichi Handa <handa@m17n.org> writes:

> +(defcustom rmail-summary-line-decoder (function rfc2047-decode-string)
>    "Function to decode a Rmail summary line.
>  It receives the summary line for one message as a string
>  and should return the decoded string.
>  
> -By default, it is `identity', which returns the string unaltered."
> +By default, it is `rfc2047-decode-string', which decodes MIME-encoded
> +subject."
>    :type 'function
>    :group 'rmail-summary)

Please don't forget to add a :version tag.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: improve rmail's MIME handling

[Stefan Monnier]

> patch for summary lines.  Nowadays, not handling MIME
> properly is regarded as a bug, and thus I want to install it
> for Emacs 23.  What do you think?

I'd agree.


        Stefan "Who even doubts the usefulness of having a custom var to
                choose some other behavior"

Re: improve rmail's MIME handling

[Chong Yidong]

Kenichi Handa <handa@m17n.org> writes:

> A while ago, I reported that I need various settings to make
> rmail handle MIME better (i.e. subject decoding/encoding),
> and showed my settings including some additional functions,
> but it seems that no one is working on this field.  So, I
> started to work on it.  The attached is the first (and easy)
> patch for summary lines.  Nowadays, not handling MIME
> properly is regarded as a bug, and thus I want to install it
> for Emacs 23.  What do you think?

Please go ahead.  Thanks.

Re: improve rmail's MIME handling

[Kenichi Handa]

In article <jwvoc9umwyz.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > patch for summary lines.  Nowadays, not handling MIME
> > properly is regarded as a bug, and thus I want to install it
> > for Emacs 23.  What do you think?

> I'd agree.

In article <87y68xmbec.fsf@stupidchicken.com>, Chong Yidong <cyd@stupidchicken.com> writes:

> Please go ahead.  Thanks.

Ok.  I've just installed changes which are bigger than what
I expected at first.  I first thought that we could use the
code in rmailmm.el almost directly, but found some problems.
Among them, these were fairly big problems:

(1) It doesn't handle a nested multipart which typically
    happen in a message that contains forwarded multipart
    message.

(2) Handling of a message that has big attachments is very
    slow because it builds decoded data string before
    just showing the button
      "Attached XXXX file: FILENAME.pdf (XXXkB)"
    We should build a data to be saved on the fly
    (i.e. after user's clicking).

Now the default value of rmail-enable-mime is t, and thus a
MIME message is properly decoded in RMAIL buffer
automatically.  One can still customize rmail-enable-mime to
nil, and type the command 'v' to view a decoded message in
the separete "*RMAIL*" buffer as before.

By the way, in the long run, I think it is better to utilize
gnus/mm*.el for rmail instead of further improving
rmailmm.el.

---
Kenichi Handa
handa@m17n.org

Re: improve rmail's MIME handling

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Date: Fri, 26 Nov 2010 13:24:09 +0900
> Cc: emacs-devel@gnu.org
> 
> > Please go ahead.  Thanks.
> 
> Ok.  I've just installed changes which are bigger than what
> I expected at first.

Thanks.

> Now the default value of rmail-enable-mime is t, and thus a MIME
> message is properly decoded in RMAIL buffer automatically.

Why is this a good idea, especially on the Emacs 23 branch, in the
middle of a pretest?  Previously, Rmail would never do anything with
attachments without my say-so, and thus was one of the safest MUAs on
Earth.  Why change that now?

Of course, if other Rmail users think we should make it t by default,
I can always customize it back, so if I'm the only one who cares,
let's leave it at t.

Re: improve rmail's MIME handling

[Kenichi Handa]

In article <83r5e874ab.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > Now the default value of rmail-enable-mime is t, and thus a MIME
> > message is properly decoded in RMAIL buffer automatically.

> Why is this a good idea, especially on the Emacs 23 branch, in the
> middle of a pretest?

I interpreted Stefan and Yidong's "go ahead" as so.

> Previously, Rmail would never do anything with
> attachments without my say-so, and thus was one of the safest MUAs on
> Earth.  Why change that now?

Rmail still does nothing non-safe thing with attachments.  It
just shows you something like this line (with proper buttons):

Attached image/jpeg file: temp.jpg (281kB) Display

instead of showing something like this raw base64 data:

Content-Type: image/jpeg
Content-Disposition: attachment; filename=temp.jpg
Content-Transfer-Encoding: base64

/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsK
CwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/wAALCAO3AqgBAREA/8QAHwAAAQUBAQEB
AQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1Fh
....

And, a text content is of course decoded properly, but it
had been decoded anyway.  The difference is that, with the
new code, text contents in multipart (and encoded-words in
headers) are also decoded correctly.

> Of course, if other Rmail users think we should make it t by default,
> I can always customize it back, so if I'm the only one who cares,
> let's leave it at t.

I vote for `t', but of course I follow the opinion of the
majority.

---
Kenichi Handa
handa@m17n.org

Re: improve rmail's MIME handling

[Stefan Monnier]

> Why is this a good idea, especially on the Emacs 23 branch, in the
> middle of a pretest?

Nowadays, a mail reader which doesn't handle MIME out of the box counts
as broken for me.

> Previously, Rmail would never do anything with attachments without my
> say-so, and thus was one of the safest MUAs on Earth.  Why change
> that now?

Because it was felt to be long overdue.

> Of course, if other Rmail users think we should make it t by default,
> I can always customize it back, so if I'm the only one who cares,
> let's leave it at t.

While I think handling MIME out of the box is indispensable, I also
think it should be easy to view the raw message easily, so I hope
there's a command that lets you see the raw message, turning off all
MIME processing.  FWIW, I couldn't find such a command in Gnus :-(.


        Stefan

Re: improve rmail's MIME handling

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> FWIW, I couldn't find such a command in Gnus :-(.

`C-u C-u g'.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: improve rmail's MIME handling

[Stefan Monnier]

>> FWIW, I couldn't find such a command in Gnus :-(.
> `C-u C-u g'.

Aha!
That seems to work perfectly, thanks,


        Stefan

Re: improve rmail's MIME handling

[Kenichi Handa]

In article <jwvsjyoywuv.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> While I think handling MIME out of the box is indispensable, I also
> think it should be easy to view the raw message easily, so I hope
> there's a command that lets you see the raw message, turning off all
> MIME processing.

It's easy to implement such a feature.

In article <m3zkswcfcl.fsf@quimbies.gnus.org>, Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> > FWIW, I couldn't find such a command in Gnus :-(.

> `C-u C-u g'.

In RMAIL, as `g' is for getting new mails, it can't be used
for toggling the MIME handling.  In Gnus it seems C-u C-u g
also toggles headers.  So one idea is to use `t' in RMAIL to
toggle MIME handling too.  But, in RMAIL, `t' still decodes
subect, etc. according to RFC2047.  Should we change it?

Another idea is to use the most useless key 'v' (rmail-mime)
when rmail-enable-mime is t.

---
Kenichi Handa
handa@m17n.org

Re: improve rmail's MIME handling

[Bob Rogers]

   From: Kenichi Handa <handa@m17n.org>
   Date: Mon, 29 Nov 2010 09:59:16 +0900

   In RMAIL, as `g' is for getting new mails, it can't be used
   for toggling the MIME handling.  In Gnus it seems C-u C-u g
   also toggles headers.  So one idea is to use `t' in RMAIL to
   toggle MIME handling too.  But, in RMAIL, `t' still decodes
   subect, etc. according to RFC2047 . . .

FWIW, the VM mail reader (which also uses "g" to get mail and "t" to
toggle headers) uses "D", for "decode MIME".

					-- Bob Rogers
					   http://www.rgrjr.com/

Re: improve rmail's MIME handling

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> Date: Mon, 29 Nov 2010 09:59:16 +0900
> 
> Another idea is to use the most useless key 'v' (rmail-mime)
> when rmail-enable-mime is t.

My vote is for this one.

Re: improve rmail's MIME handling

[Kenichi Handa]

In article <19699.7405.907052.380667@rgr.rgrjr.com>, Bob Rogers <rogers-emacs@rgrjr.dyndns.org> writes:
[...]
>    In RMAIL, as `g' is for getting new mails, it can't be used
>    for toggling the MIME handling.  In Gnus it seems C-u C-u g
>    also toggles headers.  So one idea is to use `t' in RMAIL to
>    toggle MIME handling too.  But, in RMAIL, `t' still decodes
>    subect, etc. according to RFC2047 . . .

> FWIW, the VM mail reader (which also uses "g" to get mail and "t" to
> toggle headers) uses "D", for "decode MIME".

Does "D" just decode MIME instead of togging the decoding
and undecoding of MIME?  Does VM decode encoded-words of
headers only when "D" is typed?

---
Kenichi Handa
handa@m17n.org

Re: improve rmail's MIME handling

[Richard Stallman]

It's ok to enable mime handling in Rmail by default if it is done in a
convenient way.  A convenient way is one that helps without forcing
itself on you, without switching buffers unless you request it.
It has to let you read textual attachments in the message
buffer itself.

It should not try to decode the non-textual attachments until you ask
for specific ones.

I would suggest making the non-textual attachments invisible, showing
only their header lines.  If you want to see the contents of one, you
type v on it.


-- 
Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org

Re: improve rmail's MIME handling

[Stephen J. Turnbull]

Richard Stallman writes:

 > It's ok to enable mime handling in Rmail by default if it is done in a
 > convenient way.  A convenient way is one that helps without forcing
 > itself on you, without switching buffers unless you request it.

I hope it's OK to create a presentation buffer for the current
message.

 > It has to let you read textual attachments in the message
 > buffer itself.

For text/plain and text/richtext, VM does this by default without
problem (modulo doing it in a presentation buffer rather than in the
original message buffer).

This may be hard for text/html, at least if you get mail from people
using non-Emacs MUAs.  There's good reason why people push HTML mail
out to specialized browsers.

 > It should not try to decode the non-textual attachments until you ask
 > for specific ones.

By popular demand, VM switched from that approach to decoding images
by default.  Of course this can be switched off, and the list of MIME
media types to automatically display is configurable.  The point: if
RMail is going to continue to be the preferred MUA of a tiny minority,
switched off by default is probably OK.  But if the intent is to make
Rmail attractive to average users, you'd better poll users on this.

 > I would suggest making the non-textual attachments invisible, showing
 > only their header lines.

It's better to have a single summary line.  The header lines of body
parts are often complex and uninformative (eg, the media subtypes and
content-type parameters rarely matter except to the decoding
programs).  Users want to know media type, file name, and content
description, which should easily be abstracted into 10 + 20 + 40 = 70
columns in most cases.

More important is checks that file name extensions match media type,
and that magic numbers match media type (usually the underlying
library gets that right, but sometimes not, and Emacs does implement
some types in Lisp).  Those are not easy for users to do.

 > If you want to see the contents of one, you type v on it.

Yep, that's the VM model, except that the default action (often but by
no means always = view) is activated by RET, and $ is a prefix key
that introduces bindings for alternative actions (including view, save
(which may have multiple variants, eg, for message/rfc822 parts or for
application/* parts with multiple decoder implementations), and
sometimes others).

Re: improve rmail's MIME handling

[Bob Rogers]

   From: Kenichi Handa <handa@m17n.org>
   Date: Mon, 29 Nov 2010 15:38:27 +0900

   In article <19699.7405.907052.380667@rgr.rgrjr.com>, Bob Rogers <rogers-emacs@rgrjr.dyndns.org> writes:
   [...]
   >    In RMAIL, as `g' is for getting new mails, it can't be used
   >    for toggling the MIME handling.  In Gnus it seems C-u C-u g
   >    also toggles headers.  So one idea is to use `t' in RMAIL to
   >    toggle MIME handling too.  But, in RMAIL, `t' still decodes
   >    subect, etc. according to RFC2047 . . .

   > FWIW, the VM mail reader (which also uses "g" to get mail and "t" to
   > toggle headers) uses "D", for "decode MIME".

   Does "D" just decode MIME instead of togging the decoding
   and undecoding of MIME?  Does VM decode encoded-words of
   headers only when "D" is typed?

   ---
   Kenichi Handa
   handa@m17n.org

It's a toggle; here is the start of the doc string for
vm-decode-mime-message:

	Decode the MIME objects in the current message.
	The first time this command is run on a message, decoding is
	done.  The second time, buttons for all the objects are
	displayed instead.  The third time, the raw, undecoded data is
	displayed.

	If decoding, the decoded objects might be displayed immediately,
	or buttons might be displayed that you need to activate to view
	the object.  See the documentation for the variables

	    vm-auto-displayed-mime-content-types
	    vm-auto-displayed-mime-content-type-exceptions
	    vm-mime-internal-content-types
	    vm-mime-internal-content-type-exceptions
	    vm-mime-external-content-types-alist

	to see how to control whether you see buttons or objects.

Headers in the message display seem to be decoded at the same time
(headers in the summary buffer are apparently decoded by a separate
mechanism).  And messages are initially decoded by default.

					-- Bob

Re: improve rmail's MIME handling

[Richard Stallman]

     > It's ok to enable mime handling in Rmail by default if it is done in a
     > convenient way.  A convenient way is one that helps without forcing
     > itself on you, without switching buffers unless you request it.

    I hope it's OK to create a presentation buffer for the current
    message.

I am not sure what the question means.

The present Rmail always displays the current message in a separate
buffer.  However, it swaps text between that buffer and RMAIL
so that the current message does appear in RMAIL.

What's your question?

-- 
Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org

Re: improve rmail's MIME handling

[Richard Stallman]

     > I would suggest making the non-textual attachments invisible, showing
     > only their header lines.

    It's better to have a single summary line.

Could you be less terse?  I don't understand what you have in mind.

-- 
Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org

Re: improve rmail's MIME handling

[Stephen J. Turnbull]

Richard Stallman writes:

 > What's your question?

What did you mean by "without switching buffers"?

Re: improve rmail's MIME handling

[Stephen J. Turnbull]

Richard Stallman writes:
 >      > I would suggest making the non-textual attachments invisible, showing
 >      > only their header lines.
 > 
 >     It's better to have a single summary line.
 > 
 > Could you be less terse?  I don't understand what you have in mind.

I explained exactly what I meant in the following two or three
sentences, a line constructed as described there containing a summary
of the MIME part's header lines.

Re: improve rmail's MIME handling

[PJ Weisberg]

-PJ



On Wed, Dec 1, 2010 at 5:10 AM, Richard Stallman <rms@gnu.org> wrote:
>     > I would suggest making the non-textual attachments invisible, showing
>     > only their header lines.
>
>    It's better to have a single summary line.
>
> Could you be less terse?  I don't understand what you have in mind.

I believe he's suggesting that something like this:

Content-Type: image/jpeg; name="puppy.jpg"
Content-Disposition: attachment; filename="puppy.jpg"
Content-Transfer-Encoding: base64

is meaningless to almost everyone, and it would be better to show
something like this:

Image: puppy.jpg (137K)

Re: improve rmail's MIME handling

[Glenn Morris]

PJ Weisberg wrote:

> is meaningless to almost everyone, and it would be better to show
> something like this:
>
> Image: puppy.jpg (137K)

This is basically what the existing trunk rmail-mime already does.

Re: improve rmail's MIME handling

[Stephen J. Turnbull]

Glenn Morris writes:
 > PJ Weisberg wrote:
 > 
 > > is meaningless to almost everyone, and it would be better to show
 > > something like this:
 > >
 > > Image: puppy.jpg (137K)
 > 
 > This is basically what the existing trunk rmail-mime already does.

Good.  Keep it that way.

(I never claimed it was an original idea.)

Re: improve rmail's MIME handling

[Richard Stallman]

     > What's your question?

    What did you mean by "without switching buffers"?

That's a good question.  I'll try to say this at greater length.

I mean that it should not spontaneously show you a decoded attachment
instead instead of the RMAIL buffer that shows the whole current
message.

When the current Rmail shows the current message, it does so by
swapping text between the RMAIL buffer and another buffer.  But that's
an internal implementation detail.  The user doesn't see this as a
buffer switch.

-- 
Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org

Re: improve rmail's MIME handling

[Richard Stallman]

    I believe he's suggesting that something like this:

    Content-Type: image/jpeg; name="puppy.jpg"
    Content-Disposition: attachment; filename="puppy.jpg"
    Content-Transfer-Encoding: base64

    is meaningless to almost everyone, and it would be better to show
    something like this:

    Image: puppy.jpg (137K)

I agree with that.

-- 
Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org, www.gnu.org