changes to message-send-and-exit in Emacs 26

changes to message-send-and-exit in Emacs 26

[Alex Bennée]

Hi,

I'm trying to track down what is happening in Emacs when I call
message-send-and-exit to send an email from mu4e. I mailed emacs-user:

  Subject: Changes to message-mode and encoding in Emacs26
  Date: Tue, 26 Sep 2017 16:41:03 +0100
  Message-ID: <87tvzpcvsg.fsf@linaro.org>

but I suspect the details of encoding need some input from developers.
So in brief why does the act of sending an email result in:

            character: é (displayed as é) (codepoint 233, #o351, #xe9)
    preferred charset: unicode-bmp (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
code point in charset: 0xE9
               script: latin
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), c:Chinese, j:Japanese, l:Latin, v:Viet
             to input: type "C-x 8 RET e9" or "C-x 8 RET LATIN SMALL LETTER E WITH ACUTE"
          buffer code: #xC3 #xA9
            file code: #xC3 #xA9 (encoded by coding system utf-8-unix)
              display: terminal code #xC3 #xA9

ending up as:

            character:  (displayed as ) (codepoint 4194243, #o17777703, #x3fffc3)
    preferred charset: eight-bit (Raw bytes 128-255)
code point in charset: 0xC3
               syntax: w 	which means: word
             category: L:Left-to-right (strong)
             to input: type "C-x 8 RET 3fffc3"
          buffer code: #xC3
            file code: not encodable by coding system utf-8-unix
              display: not encodable for terminal

Any pointers?

--
Alex Bennée

Re: changes to message-send-and-exit in Emacs 26

[Eli Zaretskii]

> From: Alex Bennée <alex.bennee@linaro.org>
> Date: Thu, 28 Sep 2017 19:42:07 +0100
> 
> I'm trying to track down what is happening in Emacs when I call
> message-send-and-exit to send an email from mu4e. I mailed emacs-user:
> 
>   Subject: Changes to message-mode and encoding in Emacs26
>   Date: Tue, 26 Sep 2017 16:41:03 +0100
>   Message-ID: <87tvzpcvsg.fsf@linaro.org>
> 
> but I suspect the details of encoding need some input from developers.
> So in brief why does the act of sending an email result in:
> 
>             character: é (displayed as é) (codepoint 233, #o351, #xe9)
>     preferred charset: unicode-bmp (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
> code point in charset: 0xE9
>                script: latin
>                syntax: w 	which means: word
>              category: .:Base, L:Left-to-right (strong), c:Chinese, j:Japanese, l:Latin, v:Viet
>              to input: type "C-x 8 RET e9" or "C-x 8 RET LATIN SMALL LETTER E WITH ACUTE"
>           buffer code: #xC3 #xA9
>             file code: #xC3 #xA9 (encoded by coding system utf-8-unix)
>               display: terminal code #xC3 #xA9
> 
> ending up as:
> 
>             character:  (displayed as ) (codepoint 4194243, #o17777703, #x3fffc3)
>     preferred charset: eight-bit (Raw bytes 128-255)
> code point in charset: 0xC3
>                syntax: w 	which means: word
>              category: L:Left-to-right (strong)
>              to input: type "C-x 8 RET 3fffc3"
>           buffer code: #xC3
>             file code: not encodable by coding system utf-8-unix
>               display: not encodable for terminal
> 
> Any pointers?

I'm guessing that message-send-and-exit calls message-send in this
case.  Then my suggestion would be to step through that function in
Edebug and see how the message text is communicated to mu4e and how it
is encoded in the process.

In general, I'd suggest to bring some mu4e developer on board of this
discussion, and ask them to describe how mu4e plugs into
message-send-and-exit.

Re: changes to message-send-and-exit in Emacs 26

[Alex Bennée]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Alex Bennée <alex.bennee@linaro.org>
>> Date: Thu, 28 Sep 2017 19:42:07 +0100
>>
>> I'm trying to track down what is happening in Emacs when I call
>> message-send-and-exit to send an email from mu4e. I mailed emacs-user:
>>
>>   Subject: Changes to message-mode and encoding in Emacs26
>>   Date: Tue, 26 Sep 2017 16:41:03 +0100
>>   Message-ID: <87tvzpcvsg.fsf@linaro.org>
>>
>> but I suspect the details of encoding need some input from developers.
>> So in brief why does the act of sending an email result in:
>>
>>             character: é (displayed as é) (codepoint 233, #o351, #xe9)
>>     preferred charset: unicode-bmp (Unicode Basic Multilingual Plane (U+0000..U+FFFF))
>> code point in charset: 0xE9
>>                script: latin
>>                syntax: w 	which means: word
>>              category: .:Base, L:Left-to-right (strong), c:Chinese, j:Japanese, l:Latin, v:Viet
>>              to input: type "C-x 8 RET e9" or "C-x 8 RET LATIN SMALL LETTER E WITH ACUTE"
>>           buffer code: #xC3 #xA9
>>             file code: #xC3 #xA9 (encoded by coding system utf-8-unix)
>>               display: terminal code #xC3 #xA9
>>
>> ending up as:
>>
>>             character:  (displayed as ) (codepoint 4194243, #o17777703, #x3fffc3)
>>     preferred charset: eight-bit (Raw bytes 128-255)
>> code point in charset: 0xC3
>>                syntax: w 	which means: word
>>              category: L:Left-to-right (strong)
>>              to input: type "C-x 8 RET 3fffc3"
>>           buffer code: #xC3
>>             file code: not encodable by coding system utf-8-unix
>>               display: not encodable for terminal
>>
>> Any pointers?
>
> I'm guessing that message-send-and-exit calls message-send in this
> case.  Then my suggestion would be to step through that function in
> Edebug and see how the message text is communicated to mu4e and how it
> is encoded in the process.

Looking at Alexis' follow-up link I suspect the bug described in:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25645

Looking at the backtrace there it looks like mu4e is involved in the Fcc
handling to ensure the sent email ends up in the right Maildir after it
is sent.

Looking at the mu4e code it looks like it hooks into message-send-hook
but weirdly as I compose this reply this doesn't seem to be set.

> In general, I'd suggest to bring some mu4e developer on board of this
> discussion, and ask them to describe how mu4e plugs into
> message-send-and-exit.

I've CC'd Dirk who may be able to shed more like on the hooks into
message-send.

Looking at the Fcc code it doesn't look like it does anything funky to
the encoding although it does trigger a (write-file) after adding the
Fcc header.

--
Alex Bennée