Change in rmail-insert-mime-forwarded-message

Change in rmail-insert-mime-forwarded-message

[Richard Stallman]

Your change in rmail-insert-mime-forwarded-message caused a horrible
bug: the entire RMAIL file was inserted in the message.

I see two ways to fix this: either to revert your change, and insert
the message as viewed, or to insert only the current (undecoded)
message.  I don't know which way is better.  Why do you think it is
better to insert the undecoded message rather than the visible
message?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Change in rmail-insert-mime-forwarded-message

[Eli Zaretskii]

> Date: Sun, 06 Jan 2013 21:18:56 -0500
> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Your change in rmail-insert-mime-forwarded-message caused a horrible
> bug: the entire RMAIL file was inserted in the message.

I use that change all the time, and I don't see this.  For me, it only
inserts the single message being forwarded.

Can you show a reproducible recipe for this?

Re: Change in rmail-insert-mime-forwarded-message

[Mark Lillibridge]

Richard Stallman <rms@gnu.org> writes:

>  Your change in rmail-insert-mime-forwarded-message caused a horrible
>  bug: the entire RMAIL file was inserted in the message.
>  
>  I see two ways to fix this: either to revert your change, and insert
>  the message as viewed, or to insert only the current (undecoded)
>  message.  I don't know which way is better.  Why do you think it is
>  better to insert the undecoded message rather than the visible
>  message?

    You want to insert the undecoded but unmbox-escaped message.  When I
tried the patch, I got the undecoded but mbox-escaped message but maybe
I missed a corner case.  I believe I submitted a bug about the need for
adding the mbox-unescaping part as well.  The visible message is very
likely not a valid RFC message because it is missing headers and
boundary parts.  It also likely is missing some of the attachments.

- Mark

Re: Change in rmail-insert-mime-forwarded-message

[Richard Stallman]

    I use that change all the time, and I don't see this.  For me, it only
    inserts the single message being forwarded.

I did more experimenting, and it seems that the problem
occurs after rmail-expunge.

It is not hard to fix.

The question is, is it right to send the undecoded message
or is it better to send what is visible?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Change in rmail-insert-mime-forwarded-message

[Richard Stallman]

	You want to insert the undecoded but unmbox-escaped message.

I am not sure what that means.  What is "unmbox-escaped"?  How
does that relate to decoding?

And why do you think this behavior is the right behavior, rather than
forwarding the message as displayed?

      The visible message is very
    likely not a valid RFC message because it is missing headers and
    boundary parts.  It also likely is missing some of the attachments.

Yes, but why do you think that is wrong?  If f forwards whatever
is displayed, the user has the choice to type v and forward
the undecoded message, or not type v, and forward the message as
decoded with selected parts visible.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Change in rmail-insert-mime-forwarded-message

[Mark Lillibridge]

Richard Stallman <rms@gnu.org> writes:

>  	You want to insert the undecoded but unmbox-escaped message.
>  
>  I am not sure what that means.  What is "unmbox-escaped"?  How
>  does that relate to decoding?

    Messages stored in mbox format have lines starting with >*From\
escaped by adding an extra > at the front.  This escaping needs to be
reversed to get the original message back.  See bug #13329 for more
details.


>  And why do you think this behavior is the right behavior, rather than
>  forwarding the message as displayed?
>  
>        The visible message is very
>      likely not a valid RFC message because it is missing headers and
>      boundary parts.  It also likely is missing some of the attachments.
>  
>  Yes, but why do you think that is wrong?  If f forwards whatever
>  is displayed, the user has the choice to type v and forward
>  the undecoded message, or not type v, and forward the message as
>  decoded with selected parts visible.

    Ignoring defaults for the moment, it seems like ideally we should be
able to do any of three things:

    (1) forward the exact original message with all formatting and
    attachments included.  That pretty much has to be done via an
    RFC822(sp?) attachment of the original message as received.  As noted by
    bug #9766, some email clients are unable to display such attachments.
    
    (2) include the text seen by the Rmail user at the time forward is
    called.  I do not think we should use RFC822(?) message attachments for
    this: I haven't read all the relevant RFC's, but I assume the
    destination email client will attempt to display an RFC822(sp?) message
    attachment as if it was a separately received message.  E.g., deal with
    its in-line multipart alternative parts appropriately, etc.  If you
    attach an invalid message (e.g., bad MIME headers), I would expect the
    recipient to see garbage or an error message for the message attachment.
    
        The text here can be placed directly in the message body, possibly
    prefixed with "> "'s.  Pretty much any email client should be able to
    see the included text but there's no way to include attachments or
    render HTML as HTML in the destination client.  E.g., this method cannot
    be used with HTML-only messages.  It also often loses formatting (e.g.,
    colors) that is present only in HTML parts.
    
    (3) A smarter version of (2) that both makes the original message
    body visible to the destination client even for clients that don't
    handle RFC message attachments (this means handling both the text
    and HTML in-line parts correctly if formatting is not to be lost)
    and attaches all the original non-in-line attachments as well.  This
    would be fairly tricky to do right; see my email about this in the
    bug discussion for bug #9766.  Something like this is what Outlook
    does, for example.


    At the moment, in the absence of availability of (3), I use (1) for
forward and use resend message to deal with people with limited mailers.
I don't think (2) is a reasonable default because it drops attachments,
formatting, and doesn't work for HTML-only messages.

- Mark

Re: Change in rmail-insert-mime-forwarded-message

[Stephen J. Turnbull]

Mark Lillibridge writes:

 >     Messages stored in mbox format have lines starting with >*From\
 > escaped by adding an extra > at the front.  This escaping needs to be
 > reversed to get the original message back.  See bug #13329 for more
 > details.

It's not particularly relevant to the problem Richard is reporting,
but that's incorrect.  mbox formats vary, but the most common ones
stuff a ">" if and only if the string "From " occurs at the beginning
of a line.  If ">From" occurs at the beginning of a line, you can't
know whether it was stuffed by the MDA or by the message author or
what.

The only portable way I know to handle this involves hiding the
message text from MDAs by encoding "From" or ">From" as quoted-
printable.

Steve

Re: Change in rmail-insert-mime-forwarded-message

[Mark Lillibridge]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

>  Mark Lillibridge writes:
>  
>   >     Messages stored in mbox format have lines starting with >*From\
>   > escaped by adding an extra > at the front.  This escaping needs to be
>   > reversed to get the original message back.  See bug #13329 for more
>   > details.
>  
>  It's not particularly relevant to the problem Richard is reporting,
>  but that's incorrect.  mbox formats vary, but the most common ones
>  stuff a ">" if and only if the string "From " occurs at the beginning
>  of a line.  If ">From" occurs at the beginning of a line, you can't
>  know whether it was stuffed by the MDA or by the message author or
>  what.

    Yes, mbox formats vary; I was describing mboxrd above, which I
believe is the current Rmail default and does not suffer from this
problem.  See bug #6574 for why mboxo, which you describe above,
corrupts messages with ">From " lines.  Either way, unescaping is
required for forwarding or resending messages (destination system may
not use mbox at all).


>  The only portable way I know to handle this involves hiding the
>  message text from MDAs by encoding "From" or ">From" as quoted-
>  printable.

    Agreed.  Base64 also works.  I don't think either of the message
sending libraries in Emacs does this, however.

- Mark

Re: Change in rmail-insert-mime-forwarded-message

[Stephen J. Turnbull]

Mark Lillibridge writes:
 > 
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > 
 > >  Mark Lillibridge writes:
 > >  
 > >   >     Messages stored in mbox format have lines starting with >*From\
 > >   > escaped by adding an extra > at the front.  This escaping needs to be
 > >   > reversed to get the original message back.  See bug #13329 for more
 > >   > details.
 > >  
 > >  It's not particularly relevant to the problem Richard is reporting,
 > >  but that's incorrect.  mbox formats vary, but the most common ones
 > >  stuff a ">" if and only if the string "From " occurs at the beginning
 > >  of a line.  If ">From" occurs at the beginning of a line, you can't
 > >  know whether it was stuffed by the MDA or by the message author or
 > >  what.
 > 
 >     Yes, mbox formats vary; I was describing mboxrd above, which I
 > believe is the current Rmail default and does not suffer from this
 > problem.

Of course it does, if the MDA being used From-stuffs.  Rmail is not a
substitute for the system MDA or MTA.  So by the time Rmail sees the
message, it's already ambiguous.

 > See bug #6574 for why mboxo, which you describe above, corrupts
 > messages with ">From " lines.  Either way, unescaping is required
 > for forwarding or resending messages (destination system may not
 > use mbox at all).

Of course it's not "required"; this is a cosmetic issue.  A stuffed
message is inherently ambiguous, and any attempt to unstuff is going
to be heuristic.  If you unstuff this:

>From some of my correspondents, I occasionally receive mail
>containing quotes in this style.

before forwarding, some recipients will undoubtedly be unhappy with
the results because their MDA doesn't stuff.  Others won't notice
because their MDA does stuff.  Is it worth trying to accurately
identify these odd cases?

Re: Change in rmail-insert-mime-forwarded-message

[Richard Stallman]

	(1) forward the exact original message with all formatting and
	attachments included.  That pretty much has to be done via an
	RFC822(sp?) attachment of the original message as received.  As noted by
	bug #9766, some email clients are unable to display such attachments.

	(2) include the text seen by the Rmail user at the time forward is
	called.

Both of these were possible before your change.
The first was done with v f
and the second with plain f.

        I do not think we should use RFC822(?) message attachments for
	this: I haven't read all the relevant RFC's, but I assume the
	destination email client will attempt to display an RFC822(sp?) message
	attachment as if it was a separately received message. 

f always puts the forwarded message into mime attachment, and there is
no problem in this case.  The message _as displayed in rmail_ does not
contain mime part headers.  It has things like this

    [1:text/plain Hide]

instead.

								E.g., deal with
	its in-line multipart alternative parts appropriately, etc.  If you
	attach an invalid message (e.g., bad MIME headers), I would expect the
	recipient to see garbage or an error message for the message attachment.

The problem would never happen when sending the message as
mime-processed by Rmail.

	(3) A smarter version of (2) that both makes the original message
	body visible to the destination client even for clients that don't
	handle RFC message attachments

I can't make concrete sense of that, sorry.

	At the moment, in the absence of availability of (3), I use (1) for
    forward and use resend message to deal with people with limited mailers.
    I don't think (2) is a reasonable default because it drops attachments,
    formatting, and doesn't work for HTML-only messages.

With your change, only (1) is possible.  I think (2) is the right
default, but in any case, it is better to have both options.

So I think I will revert your change.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Change in rmail-insert-mime-forwarded-message

[Eli Zaretskii]

> Date: Mon, 07 Jan 2013 21:11:24 -0500
> From: Richard Stallman <rms@gnu.org>
> CC: mark.lillibridge@hp.com, emacs-devel@gnu.org
> 
> The question is, is it right to send the undecoded message
> or is it better to send what is visible?

You can only send what is visible if the attachment was human-readable
text.  You cannot do that with binary attachments.

Also, if the mail you forward has both text and HTML format, the
recipient will be unable to display HTML or text depending on her mail
agent, but will see both of them.

So I think it is better to send them encoded, so they appear as
attachments on the receiving side.

Re: Change in rmail-insert-mime-forwarded-message

[Richard Stallman]

    You can only send what is visible if the attachment was human-readable
    text.  You cannot do that with binary attachments.

Yes you can.  Type v, and the attachment will be visible just as it
was sent.  My change (reverting Mark's change) will add functionality
and lose none.

So I think I should do this -- the arguments against it seem
not to be based on a mistaken idea of the effects.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Change in rmail-insert-mime-forwarded-message

[Mark Lillibridge]

Richard Stallman <rms@gnu.org> writes:

>  	(1) forward the exact original message with all formatting and
>  	attachments included.  That pretty much has to be done via an
>  	RFC822(sp?) attachment of the original message as received.  As noted by
>  	bug #9766, some email clients are unable to display such attachments.
>  
>  	(2) include the text seen by the Rmail user at the time forward is
>  	called.
>  
>  Both of these were possible before your change.
>  The first was done with v f
>  and the second with plain f.

    My wording for (2) wasn't very good and you didn't quote the
relevant bits, but it was intend to require that the text be put in the
message body not a message attachment, which cannot be read by users of
lame message clients.  Your 'f' does not actually correctly implement
(2) as I meant it.  I agree that your 'vf' (assuming suitable bug fixes
have been applied) implements (1).


>          I do not think we should use RFC822(?) message attachments for
>  	this: I haven't read all the relevant RFC's, but I assume the
>  	destination email client will attempt to display an RFC822(sp?) message
>  	attachment as if it was a separately received message. 
>  
>  f always puts the forwarded message into mime attachment, and there is
>  no problem in this case.  The message _as displayed in rmail_ does not
>  contain mime part headers.  It has things like this
>  
>      [1:text/plain Hide]
>  
>  instead.

    Depends on how the user is viewing the message.  Hitting "t" will
definitely get you the MIME headers but not the correct corresponding
body.  (A rare case, I grant you.)  Also note that the charset encoding
header is likely also gone, probably causing characters to be displayed
incorrectly.



>  	(3) A smarter version of (2) that both makes the original message
>  	body visible to the destination client even for clients that don't
>  	handle RFC message attachments
>  
>  I can't make concrete sense of that, sorry.

    Let me try a concrete example.  Suppose you have a message with a
text part message body "this is in red", message body HTML part with the
same except actually in red, and an attached (non-in-line) PDF.

With (3), the forwarded message would have:

  * a text message body part with something like "===== forwarded message
    =====", the Rmail filtered headers, then "this is in red"

  * the equivalent as an HTML message body part (including the headers
    protected via <PRE> or the like), with the "this is in red" part
    actually in red.

  * the attached PDF as an individual attachment (same suggested file name)

Optionally, you could also attach the original message as an RFC822(?)
message attachment part.

    For even more fun, there should be an easy way for the user to add
text before the "===== forwarded message =====" that shows up in both
parts.  See what I mean about being nontrivial?


>  	At the moment, in the absence of availability of (3), I use (1) for
>      forward and use resend message to deal with people with limited mailers.
>      I don't think (2) is a reasonable default because it drops attachments,
>      formatting, and doesn't work for HTML-only messages.
>  
>  With your change, only (1) is possible.  I think (2) is the right
>  default, but in any case, it is better to have both options.
>  
>  So I think I will revert your change.

    I agree that we should have more than one option, which is what I
recommended in the relevant bug report.  Correct code for (2) or ideally
(3) needs to be written and linked to a different invocation path.   If
only one is supported, then (1) is the better choice in my opinion.  I
think users can set rmail-enable-mime-composing to nil if they want (2) for
'f' but currently this is a startup choice rather than on a forward by
forward basis.

    Hmmm.  Richard, does the behavior of "f" when
rmail-enable-mime-composing is set to nil do what you want?  If so, we
just need to figure out a good way to make that behavior and my patch
behavior for "f" available at the same time.

    I'm not really keen on using the "vf" trick; it has the side effect
of changing the display of the original message being forwarded.  It
also seems not unlikely that users might want to send the raw message
with (1) to users of lame email clients but your use of "vf" for (2)
preempts that possibility.  I can also imagine trying to forward the
same message twice and having muscle memory type "vf" twice, screwing up
the second forward.

- Mark

Re: Change in rmail-insert-mime-forwarded-message

[Eli Zaretskii]

> Date: Thu, 10 Jan 2013 01:19:31 -0500
> From: Richard Stallman <rms@gnu.org>
> CC: mark.lillibridge@hp.com, emacs-devel@gnu.org
> 
>     You can only send what is visible if the attachment was human-readable
>     text.  You cannot do that with binary attachments.
> 
> Yes you can.  Type v, and the attachment will be visible just as it
> was sent.

Do you mean I must manually type v every time I need to forward a
message, just in case it has binary attachments?  That would be a
nuisance.  I surely hope you meant that typing f will do that (and
whatever else is necessary) for me.

> My change (reverting Mark's change) will add functionality
> and lose none.

Before the change, f _never_ worked when the original message included
attachments.  No mail agent I use, including Rmail, was able to
display the forwarded message correctly, including its attachments.
After the change, f does work.  So reverting the change is going
backwards.  If the change introduces bugs in some specific situations,
those bugs should be fixed, without reverting the change itself.

Re: Change in rmail-insert-mime-forwarded-message

[Richard Stallman]

	Hmmm.  Richard, does the behavior of "f" when
    rmail-enable-mime-composing is set to nil do what you want?

I just tried it, and it seems ok.

								 If so, we
    just need to figure out a good way to make that behavior and my patch
    behavior for "f" available at the same time.

That's ok with me.  Any suggestions?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Change in rmail-insert-mime-forwarded-message

[Eli Zaretskii]

> Date: Thu, 10 Jan 2013 23:43:33 -0500
> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> 	Hmmm.  Richard, does the behavior of "f" when
>     rmail-enable-mime-composing is set to nil do what you want?
> 
> I just tried it, and it seems ok.
> 
> 								 If so, we
>     just need to figure out a good way to make that behavior and my patch
>     behavior for "f" available at the same time.
> 
> That's ok with me.  Any suggestions?

Test rmail-enable-mime-composing's value and do one or the other
depending on that value?

Re: Change in rmail-insert-mime-forwarded-message

[Mark Lillibridge]

Eli Zaretskii <eliz@gnu.org> writes:

>  > Date: Thu, 10 Jan 2013 23:43:33 -0500
>  > From: Richard Stallman <rms@gnu.org>
>  > Cc: emacs-devel@gnu.org
>  > 
>  > 	Hmmm.  Richard, does the behavior of "f" when
>  >     rmail-enable-mime-composing is set to nil do what you want?
>  > 
>  > I just tried it, and it seems ok.
>  > 
>  > 								 If so, we
>  >     just need to figure out a good way to make that behavior and my patch
>  >     behavior for "f" available at the same time.
>  > 
>  > That's ok with me.  Any suggestions?
>  
>  Test rmail-enable-mime-composing's value and do one or the other
>  depending on that value?

    The obvious first step would seem to be to modify rmail-forward
(below) to have an optional argument that overrides the setting of
rmail-enable-mime-composing.  

    The question is how best to select this argument interactively?
Unfortunately, the prefix argument is already being used to toggle
between regular forward and resend message.  We could make C-u f resend
and C-u C-u f forward the non-default way, but that feels kludgy to me.
Accordingly, I'm thinking adding a new command key would be better.  We
could add F for the alternative forward method or move resend to its own
key, R, and make C-u f the alternative forward (not muscle memory
backwards compatible).  If you don't like shifting (I don't think Rmail
currently uses any shifted letters), y and z look available but are not
very memorable.  Either way, this probably means adding a short helper
function that calls rmail-forward with the appropriate arguments.  An
advantage of "F" is that it would allow using C-u F in the future for
alternative methods of forwarding like (3).

    Note that there are a few other ways to forward messages (menubar,
from summary buffer) that would also need to be changed similarly to
whatever we do with the main one.  We should probably rename (with
obsolete alias) rmail-enable-mime-composing to something like
rmail-default-forwarding-method if we do this.

    I don't have a strong argument offhand for which way the default
should be set, but if it's (2), it might be useful to have a mini-buffer
message if the message has attachments suggesting that the user might
want to use (1) instead.

- Mark



(defun rmail-forward (resend)
  "Forward the current message to another user.
With prefix argument, \"resend\" the message instead of forwarding it;
see the documentation of `rmail-resend'."
  (interactive "P")
  (if (zerop rmail-current-message)
      (error "No message to forward"))
  (if resend
      (call-interactively 'rmail-resend)
    (let ((forward-buffer rmail-buffer)
	  (msgnum rmail-current-message)
	  (subject (concat "["
			   (let ((from (or (mail-fetch-field "From")
					   (mail-fetch-field ">From"))))
			     (if from
				 (concat (mail-strip-quoted-names from) ": ")
			       ""))
			   (or (mail-fetch-field "Subject") "")
			   "]")))
      (if (rmail-start-mail
	   nil nil subject nil nil rmail-buffer
	   (list (list 'rmail-mark-message
		       forward-buffer
		       (with-current-buffer rmail-buffer
			 (aref rmail-msgref-vector msgnum))
		       rmail-forwarded-attr-index))
	   ;; If only one window, use it for the mail buffer.
	   ;; Otherwise, use another window for the mail buffer
	   ;; so that the Rmail buffer remains visible
	   ;; and sending the mail will get back to it.
	   (and (not rmail-mail-new-frame) (one-window-p t)))
	  ;; The mail buffer is now current.
	  (save-excursion
	    ;; Insert after header separator--before signature if any.
	    (rfc822-goto-eoh)
	    (forward-line 1)
	    (if (and rmail-enable-mime rmail-enable-mime-composing
		     rmail-insert-mime-forwarded-message-function)
		(prog1
		    (funcall rmail-insert-mime-forwarded-message-function
			     forward-buffer)
		  ;; rmail-insert-mime-forwarded-message-function
		  ;; works by inserting MML tags into forward-buffer.
		  ;; The MUA will need to convert it to MIME before
		  ;; sending.  mail-encode-mml tells them to do that.
		  ;; message.el does that automagically.
		  (or (eq mail-user-agent 'message-user-agent)
		      (setq mail-encode-mml t)))
	      (insert "------- Start of forwarded message -------\n")
	      ;; Quote lines with `- ' if they start with `-'.
	      (let ((beg (point)) end)
		(setq end (point-marker))
		(set-marker-insertion-type end t)
		(insert-buffer-substring forward-buffer)
		(goto-char beg)
		(while (re-search-forward "^-" end t)
		  (beginning-of-line)
		  (insert "- ")
		  (forward-line 1))
		(goto-char end)
		(skip-chars-backward "\n")
		(if (< (point) end)
		    (forward-char 1))
		(delete-region (point) end)
		(set-marker end nil))
	      (insert "------- End of forwarded message -------\n"))
	    (push-mark))))))

Re: Change in rmail-insert-mime-forwarded-message

[Richard Stallman]

	The obvious first step would seem to be to modify rmail-forward
    (below) to have an optional argument that overrides the setting of
    rmail-enable-mime-composing.  

Yes, that would do it.

    Unfortunately, the prefix argument is already being used to toggle
    between regular forward and resend message.  We could make C-u f resend
    and C-u C-u f forward the non-default way, but that feels kludgy to me.

It's not ideal, but it is better than nothing.

      We
    could add F for the alternative forward method or move resend to its own
    key, R, and make C-u f the alternative forward (not muscle memory
    backwards compatible).

These are ok too,

      We should probably rename (with
    obsolete alias) rmail-enable-mime-composing to something like
    rmail-default-forwarding-method if we do this.

With any of these changes in interface, we don't need a variable like
rmail-enable-mime-composing or rmail-default-forwarding-method, under
any name.  The interface will be enough.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Change in rmail-insert-mime-forwarded-message

[Richard Stallman]

I checked in a fix for the bug of inserting more than the current message.
I didn't change the command interface.  Are you going to do that?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Change in rmail-insert-mime-forwarded-message

[Glenn Morris]

So, what, if anything, should be done about this in the emacs-24 branch
(ie for the 24.3 release)?

I haven't seen a recipe for the "accidentally forward your entire
mail file" problem, but if it happens and you don't notice, it's
obviously awful, and much worse than the original issue (#9521).

My inclination is not to backport the subsequent trunk rmail changes to
emacs-24 at this stage, but to revert the 2012-12-29 rmailmm change in
emacs-24 (only). Then at least 24.3 will be no worse than 24.1,2, and
the new stuff can get more testing in trunk.

But I'll defer to you who actually use Rmail.

Re: Change in rmail-insert-mime-forwarded-message

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: mdl@alum.mit.edu,  eliz@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 14 Jan 2013 21:13:29 -0500
> 
> My inclination is not to backport the subsequent trunk rmail changes to
> emacs-24 at this stage, but to revert the 2012-12-29 rmailmm change in
> emacs-24 (only).

Please don't.  The original problem was awful, and the bug in
allegedly introduced happens only under circumstances I have never
seen on my system.

Re: Change in rmail-insert-mime-forwarded-message

[Richard Stallman]

    So, what, if anything, should be done about this in the emacs-24 branch
    (ie for the 24.3 release)?

Is the change that caused the bug included in the 24.3 release?

    My inclination is not to backport the subsequent trunk rmail changes to
    emacs-24 at this stage, but to revert the 2012-12-29 rmailmm change in
    emacs-24 (only). Then at least 24.3 will be no worse than 24.1,2, and
    the new stuff can get more testing in trunk.

I am surprised this change was put into the release at such a late
date.  It did not fix a regression.  I think reverting it in the
release is right.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Change in rmail-insert-mime-forwarded-message

[Eli Zaretskii]

> Date: Tue, 15 Jan 2013 00:27:12 -0500
> From: Richard Stallman <rms@gnu.org>
> CC: mdl@alum.mit.edu, eliz@gnu.org, emacs-devel@gnu.org
> 
>     So, what, if anything, should be done about this in the emacs-24 branch
>     (ie for the 24.3 release)?
> 
> Is the change that caused the bug included in the 24.3 release?

Yes.

>     My inclination is not to backport the subsequent trunk rmail changes to
>     emacs-24 at this stage, but to revert the 2012-12-29 rmailmm change in
>     emacs-24 (only). Then at least 24.3 will be no worse than 24.1,2, and
>     the new stuff can get more testing in trunk.
> 
> I am surprised this change was put into the release at such a late
> date.  It did not fix a regression.

It was a very simple fix, and my testing have clearly shown that it
does TRT.

> I think reverting it in the release is right.

'f' in Rmail was dysfunctional before this fix; it will be
dysfunctional again, if the change is reverted without fixing it in
some other way.

Re: Change in rmail-insert-mime-forwarded-message

[Glenn Morris]

Richard Stallman wrote:

> Is the change that caused the bug included in the 24.3 release?

Yes.

> I am surprised this change was put into the release at such a late
> date.  It did not fix a regression.  I think reverting it in the
> release is right.

This decision would much easier to make if you could give a reproducible
emacs -Q etc recipe showing how to get the bug you saw.

Re: Change in rmail-insert-mime-forwarded-message

[Richard Stallman]

    > I think reverting it in the release is right.

    'f' in Rmail was dysfunctional before this fix;

On the contrary, it was more functional.  It could forward either the
mime-decoded message or the raw message, and did both jobs correctly.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Change in rmail-insert-mime-forwarded-message

[Richard Stallman]

    This decision would much easier to make if you could give a reproducible
    emacs -Q etc recipe showing how to get the bug you saw.

Visit a mail file with 3 messages, type d x, then forward one.
It should insert both.

I already installed a fix for that bug.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Change in rmail-insert-mime-forwarded-message

[Glenn Morris]

Richard Stallman wrote:

> Visit a mail file with 3 messages, type d x, then forward one.
> It should insert both.

Thanks, I can reproduce that in current emacs-24 branch.
Reverting the 2012-12-29 to rmailmm makes it go away, so I will do that
(in emacs-24 only, since you fixed it in trunk).

Re: Change in rmail-insert-mime-forwarded-message

[Mark Lillibridge]

Richard Stallman <rms@gnu.org> writes:

>  I checked in a fix for the bug of inserting more than the current message.
>  I didn't change the command interface.  Are you going to do that?

    Sorry, I'm swamped at work at the moment and am having trouble just
keeping up with the email thread.  :-(  I should have more time in
February, but feel free to proceed without me.

- Mark

Re: Change in rmail-insert-mime-forwarded-message

[Richard Stallman]

Would you like to fix it in February?
-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: Change in rmail-insert-mime-forwarded-message

[Mark Lillibridge]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

>  Mark Lillibridge writes:
>   > 
>   > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>   > 
>   > >  Mark Lillibridge writes:
>   > >  
>   > >   >     Messages stored in mbox format have lines starting with >*From\
>   > >   > escaped by adding an extra > at the front.  This escaping needs to be
>   > >   > reversed to get the original message back.  See bug #13329 for more
>   > >   > details.
>   > >  
>   > >  It's not particularly relevant to the problem Richard is reporting,
>   > >  but that's incorrect.  mbox formats vary, but the most common ones
>   > >  stuff a ">" if and only if the string "From " occurs at the beginning
>   > >  of a line.  If ">From" occurs at the beginning of a line, you can't
>   > >  know whether it was stuffed by the MDA or by the message author or
>   > >  what.
>   > 
>   >     Yes, mbox formats vary; I was describing mboxrd above, which I
>   > believe is the current Rmail default and does not suffer from this
>   > problem.
>  
>  Of course it does, if the MDA being used From-stuffs.  Rmail is not a
>  substitute for the system MDA or MTA.  So by the time Rmail sees the
>  message, it's already ambiguous.

    Yes, Rmail cannot prevent/fix damage that has already been incurred
due to broken mail distribution agents.  Hopefully, in the future we can
get those broken mail distribution agents fixed so they use mboxrd and
do not cause damage.


>   > See bug #6574 for why mboxo, which you describe above, corrupts
>   > messages with ">From " lines.  Either way, unescaping is required
>   > for forwarding or resending messages (destination system may not
>   > use mbox at all).
>  
>  Of course it's not "required"; this is a cosmetic issue.  A stuffed
>  message is inherently ambiguous, and any attempt to unstuff is going
>  to be heuristic.  If you unstuff this:
>  
>  >From some of my correspondents, I occasionally receive mail
>  >containing quotes in this style.
>  
>  before forwarding, some recipients will undoubtedly be unhappy with
>  the results because their MDA doesn't stuff.  Others won't notice
>  because their MDA does stuff.  Is it worth trying to accurately
>  identify these odd cases?

    You would like Rmail to additionally damage the message again in the
hopes that the two damages cancel out?  You asked Rmail to send the
message:

!  From some of my correspondents, I occasionally receive mail
!  >containing quotes in this style.

("!  " added to prevent damage by broken mail distribution agents)

I would expect to see that on the destination modulo broken mail
distribution agents in the middle.  Note that the ">" before the From is
not part of the message per the mbox format rules; sending it would be
corrupting the message.  

    Maybe it's just me, but purposely corrupting the message in the
hopes that broken software on the other end will uncorrupt it seems
wrong.  (If you were sure the software the other end would predictably
react so as to fix things, that would be another matter.)

- Mark

Re: Change in rmail-insert-mime-forwarded-message

[Mark Lillibridge]

Richard Stallman <rms@gnu.org> writes:

>      > I think reverting it in the release is right.
>  
>      'f' in Rmail was dysfunctional before this fix;
>  
>  On the contrary, it was more functional.  It could forward either the
>  mime-decoded message or the raw message, and did both jobs correctly.

    I believe it did both incorrectly.  The raw message wasn't
mbox-unescaped properly and the mime-decoded message was incorrectly
placed in a RFC822 attachment (see previous email why this causes
problems).  I think Eli's real point, however, is that the need to use v
then f to forward attachments is completely nonobvious to users so
effectively Rmail doesn't support forwarding attachments pre fix in
their eyes.  Arguably, this could be addressed by a comment in the news
file but if we are going to change the user interface anyways, let's do
it only once.

- Mark

Re: Change in rmail-insert-mime-forwarded-message

[Stephen J. Turnbull]

Mark Lillibridge writes:

 >     Yes, Rmail cannot prevent/fix damage that has already been incurred
 > due to broken mail distribution agents.  Hopefully, in the future we can
 > get those broken mail distribution agents fixed so they use mboxrd and
 > do not cause damage.

That's not good enough.  You also have to fix all of the ancient mbox
files that people have lying around.  Not going to happen.

 >     You would like Rmail to additionally damage the message again in the
 > hopes that the two damages cancel out?  You asked Rmail to send the
 > message:

No, I would like an MUA to ignore From-stuffing entirely (ie, treat
">" as just a character when it appears in the message body, no matter
what the context) when copying to and from mbox files.

It might be nice if the MUA would beautify the *display* of mail being
read (but not composed!) by suppressing one ">" when followed by
"From" in a context where it seems that there is an extra ">".  I've
got better things to do that try to write such code, though.

In composition, the user can "fix" it if they think those are the
appropriate semantics.  Why not let them?

 > Note that the ">" before the From is not part of the message per
 > the mbox format rules; sending it would be corrupting the message.

There is no standard for "mbox formatting rules", just a wide variety
of practices that are very similar but not identical.[1]  If the MDA does
From-stuffing, the ">" might be part of the format, or it might be
part of the message -- it depends on what came in over the wire, and
you don't have normally access to that.  If the MDA doesn't do From-
stuffing, it's part of the message.  Both are correct, according to
their own designs.  You don't have to like those designs, but the MDAs
are conforming to the specs they were written to.

"When faced with ambiguity, refuse to guess."

Why-yes-I'm-unambiguously-a-Tim-Peters-fan-ly y'rs,

Footnotes: 
[1]  http://www.jwz.org/doc/content-length.html.  Sadly enough,
nothing has much changed since then, except the proliferation of sites
that don't use the mbox format.  Nevertheless, there are still plenty
of sites that do, especially those based on free OSes where Emacs is
likely to be the platform of choice for at least some users.  And they
have some *very* old mbox files....

Re: Change in rmail-insert-mime-forwarded-message

[Mark Lillibridge]

Richard Stallman <rms@gnu.org> writes:

>  Would you like to fix it in February?

    Well, that certainly didn't happen.  :-(  Sorry about failing to
reply in a timely manner.  Work crisis level went through the roof and
has stayed high since.  I was really hoping to help you guys out with
some actual patches rather than just reporting bugs this year, but it
looks like that is not going to happen at least before fall.

- Mark