bug#28266: 25.2; RMAIL FCC coding issue

bug#28266: 25.2; RMAIL FCC coding issue

[Charles A. Roelli]

From emacs -q (25.2 or 26.0.50):

M-: (setq mail-user-agent 'sendmail-user-agent) RET
C-x m
someone@example.com C-n
test RET
FCC: /tmp/emacs-bug-mbox
C-u 2 C-n
This is a test mail, containing ‘’ quotes.
C-c C-c
transport RET ; On my system, this doesn't actually send the mail, so
	      ; this is the mail-sending system I choose to reproduce
	      ; the issue.  In any case, the recipient email doesn't
	      ; really exist so it should be okay to choose this.
C-u M-x rmail /tmp/emacs-bug-mbox ; Observe correct message contents.
C-x o ; Switch back to the *mail* buffer.
C-c C-c y ; Resend it.
n ; Open the resent copy.

This is RMAIL's copy of the resent mail:

----------
Date: Mon, 28 Aug 2017 20:41:35 +0200
From: charles@localhost (Charles A. Roelli)
To: someone@example.com
Subject: test
Content-type: text/plain; charset=utf-8

This is a test mail, containing ^X^Y quotes.
----------

Notice that the quotes have been converted to ^X/^Y
characters (characters have been corrupted).  I tested with
another non-ASCII character (é), and it got converted to \351.

The same issue does not occur with the default value of
mail-user-agent (`message-user-agent').

bug#28266: 25.2; RMAIL FCC coding issue

[Eli Zaretskii]

> Date: Mon, 28 Aug 2017 20:59:58 +0200
> From: charles@aurox.ch (Charles A. Roelli)
> 
> >From emacs -q (25.2 or 26.0.50):
> 
> M-: (setq mail-user-agent 'sendmail-user-agent) RET
> C-x m
> someone@example.com C-n
> test RET
> FCC: /tmp/emacs-bug-mbox
> C-u 2 C-n
> This is a test mail, containing ‘’ quotes.
> C-c C-c
> transport RET ; On my system, this doesn't actually send the mail, so
> 	      ; this is the mail-sending system I choose to reproduce
> 	      ; the issue.  In any case, the recipient email doesn't
> 	      ; really exist so it should be okay to choose this.
> C-u M-x rmail /tmp/emacs-bug-mbox ; Observe correct message contents.
> C-x o ; Switch back to the *mail* buffer.
> C-c C-c y ; Resend it.
> n ; Open the resent copy.
> 
> This is RMAIL's copy of the resent mail:
> 
> ----------
> Date: Mon, 28 Aug 2017 20:41:35 +0200
> From: charles@localhost (Charles A. Roelli)
> To: someone@example.com
> Subject: test
> Content-type: text/plain; charset=utf-8
> 
> This is a test mail, containing ^X^Y quotes.
> ----------
> 
> Notice that the quotes have been converted to ^X/^Y
> characters (characters have been corrupted).  I tested with
> another non-ASCII character (é), and it got converted to \351.

I cannot reproduce this with the current emacs-26 branch tip.
However, in my case, Emacs asked for a suitable coding-system each
time I wanted to send a message.  Answering UTF-8 produced a valid
encoding in both cases.

So perhaps the difference, and the reason for a problem you see, is
the default encoding in your case, which you didn't show.  (In my
case, the default is Latin-1.)

bug#28266: 25.2; RMAIL FCC coding issue

[Charles A. Roelli]

> Date: Fri, 22 Sep 2017 16:13:58 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> I cannot reproduce this with the current emacs-26 branch tip.
> However, in my case, Emacs asked for a suitable coding-system each
> time I wanted to send a message.  Answering UTF-8 produced a valid
> encoding in both cases.
> 
> So perhaps the difference, and the reason for a problem you see, is
> the default encoding in your case, which you didn't show.  (In my
> case, the default is Latin-1.)

Hmm, I was never queried for a coding system.  I've repeated the issue
on GNU/Linux with 25.3 and macOS with 25.3 and emacs-26.  And I don't
set any special locales/language environments:

> Important settings:
>   value of $LANG: en_GB.UTF-8
>   locale-coding-system: utf-8-unix

bug#28266: 25.2; RMAIL FCC coding issue

[Eli Zaretskii]

> Date: Fri, 22 Sep 2017 20:39:45 +0200
> From: charles@aurox.ch (Charles A. Roelli)
> CC: 28266@debbugs.gnu.org
> 
> > Date: Fri, 22 Sep 2017 16:13:58 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > 
> > I cannot reproduce this with the current emacs-26 branch tip.
> > However, in my case, Emacs asked for a suitable coding-system each
> > time I wanted to send a message.  Answering UTF-8 produced a valid
> > encoding in both cases.
> > 
> > So perhaps the difference, and the reason for a problem you see, is
> > the default encoding in your case, which you didn't show.  (In my
> > case, the default is Latin-1.)
> 
> Hmm, I was never queried for a coding system.

Because your default is UTF-8.  This is normal.

> I've repeated the issue on GNU/Linux with 25.3 and macOS with 25.3
> and emacs-26.  And I don't set any special locales/language
> environments:
> 
> > Important settings:
> >   value of $LANG: en_GB.UTF-8
> >   locale-coding-system: utf-8-unix

With en_US.UTF-8 on GNU/Linux, using the current emacs-26 branch, I
see no problem: both messages are sent with a correct encoding.

What happens if you don't visit the FCC file after the first message
is sent, only after the second?  Do you still see the second message
encoded incorrectly?

bug#28266: 25.2; RMAIL FCC coding issue

[Charles A. Roelli]

> Date: Fri, 22 Sep 2017 23:08:13 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Date: Fri, 22 Sep 2017 20:39:45 +0200
> > From: charles@aurox.ch (Charles A. Roelli)
> > CC: 28266@debbugs.gnu.org
> > 
> > > Date: Fri, 22 Sep 2017 16:13:58 +0300
> > > From: Eli Zaretskii <eliz@gnu.org>
> > > 
> > > I cannot reproduce this with the current emacs-26 branch tip.
> > > However, in my case, Emacs asked for a suitable coding-system each
> > > time I wanted to send a message.  Answering UTF-8 produced a valid
> > > encoding in both cases.
> > > 
> > > So perhaps the difference, and the reason for a problem you see, is
> > > the default encoding in your case, which you didn't show.  (In my
> > > case, the default is Latin-1.)
> > 
> > Hmm, I was never queried for a coding system.
> 
> Because your default is UTF-8.  This is normal.
> 
> > I've repeated the issue on GNU/Linux with 25.3 and macOS with 25.3
> > and emacs-26.  And I don't set any special locales/language
> > environments:
> > 
> > > Important settings:
> > >   value of $LANG: en_GB.UTF-8
> > >   locale-coding-system: utf-8-unix
> 
> With en_US.UTF-8 on GNU/Linux, using the current emacs-26 branch, I
> see no problem: both messages are sent with a correct encoding.

Yes, I believe that in my case too, the messages are (or would be)
sent with the correct encoding.  The encoding issue I mentioned seems
to only apply to the saving of the mail buffer into the file specified
in the FCC header.

> What happens if you don't visit the FCC file after the first message
> is sent, only after the second?  Do you still see the second message
> encoded incorrectly?

No, in that case, both messages are encoded correctly.  The encoding
issue seems to only occur when the buffer is already open.

The text of the outgoing mail buffer is inserted into the file
specified by the FCC header using "insert-buffer-substring" in
function "rmail-output-to-rmail-buffer" (use C-u C-M-x on this
function before hitting C-c C-c y to resend the mail).  If you switch
to "tembuf" (argument to "rmail-output-to-rmail-buffer") in another
frame while debugging this function, you can see in the modeline that
its coding system is utf-8-unix.  Then if you switch to the buffer
containing the FCC file, the modeline indicates that it's unibyte.

Note also that variable "enable-multibyte-characters" is bound to nil
around this call.

bug#28266: 25.2; RMAIL FCC coding issue

[Alexis]

Charles A. Roelli <charles@aurox.ch> writes:

> Yes, I believe that in my case too, the messages are (or would 
> be)
> sent with the correct encoding.  The encoding issue I mentioned 
> seems
> to only apply to the saving of the mail buffer into the file 
> specified
> in the FCC header.

To me this sounds similar to #25645:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25645

Or is that entirely unrelated?


Alexis.

bug#28266: 25.2; RMAIL FCC coding issue

[Lars Ingebrigtsen]

Alexis <flexibeast@gmail.com> writes:

> Charles A. Roelli <charles@aurox.ch> writes:
>
>> Yes, I believe that in my case too, the messages are (or would be)
>> sent with the correct encoding.  The encoding issue I mentioned
>> seems
>> to only apply to the saving of the mail buffer into the file
>> specified
>> in the FCC header.
>
> To me this sounds similar to #25645:
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25645

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

Looks quite similar.

Charles, are you still seeing these problems in recent Emacs versions?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#28266: 25.2; RMAIL FCC coding issue

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> To me this sounds similar to #25645:
>>
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25645
>
> (I'm going through old bug reports that unfortunately weren't resolved
> at the time.)
>
> Looks quite similar.
>
> Charles, are you still seeing these problems in recent Emacs versions?

There was no response in a month, and since 25645 was fixed, I'm going
to assume that that fix also fixed this bug report, and I'm closing it.
If this is still an issue in recent Emacs versions, please respond to
the debbugs address and we'll reopen.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no