bug#75397: 29.4; GNUS message mode lacks Cyrillic support (GNU/Linux)

bug#75397: 29.4; GNUS message mode lacks Cyrillic support (GNU/Linux)

[Nickolai Dobrynin]

1. Type M-gnus to start GNUS.
2. Start composing a message by pressing 'm'.
3. Enter the first, say, 5 letters of the Cyrillic alphabet, i.e. абвгд.
4. Press C-c C-d to save the draft.
5. Press S D e to resume editing.
6. Bizarrely enough, instead of "абвгд", you will see a sequence of digits "01234".

Is this even real?? By the way, this does not occur with ordinary text
files. And everything works fine in the view mode inside the draft
group. Pressing S D e, on the other hand, makes things go haywire.

Incidentally, a similar problem, though not as bad, occurs with German
characters (e.g. ä) which get replaced with \number after pressing S D e.

Many thanks.

bug#75397: 29.4; GNUS message mode lacks Cyrillic support (GNU/Linux)

[Andreas Schwab]

On Jan 06 2025, Nickolai Dobrynin wrote:

> 1. Type M-gnus to start GNUS.
> 2. Start composing a message by pressing 'm'.
> 3. Enter the first, say, 5 letters of the Cyrillic alphabet, i.e. абвгд.
> 4. Press C-c C-d to save the draft.
> 5. Press S D e to resume editing.
> 6. Bizarrely enough, instead of "абвгд", you will see a sequence of digits "01234".
>
> Is this even real?? By the way, this does not occur with ordinary text
> files. And everything works fine in the view mode inside the draft
> group. Pressing S D e, on the other hand, makes things go haywire.

The problem is that message-dont-send doesn't encode the message when it
saves it.  gnus-summary-resend-message-edit assumes that the stored
message is encoded and goes through an unibyte buffer while trying to
decode it, which then strips off all non-ascii bits from the characters.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#75397: 29.4; GNUS message mode lacks Cyrillic support (GNU/Linux)

[Eli Zaretskii]

> Cc: 75397@debbugs.gnu.org
> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Mon, 06 Jan 2025 10:36:34 +0100
> 
> On Jan 06 2025, Nickolai Dobrynin wrote:
> 
> > 1. Type M-gnus to start GNUS.
> > 2. Start composing a message by pressing 'm'.
> > 3. Enter the first, say, 5 letters of the Cyrillic alphabet, i.e. абвгд.
> > 4. Press C-c C-d to save the draft.
> > 5. Press S D e to resume editing.
> > 6. Bizarrely enough, instead of "абвгд", you will see a sequence of digits "01234".
> >
> > Is this even real?? By the way, this does not occur with ordinary text
> > files. And everything works fine in the view mode inside the draft
> > group. Pressing S D e, on the other hand, makes things go haywire.
> 
> The problem is that message-dont-send doesn't encode the message when it
> saves it.

I must be missing something here, because I don't see what you
describe above:

  (defun message-dont-send ()
    "Don't send the message you have been editing.
  Instead, just auto-save the buffer and then bury it."
    (interactive nil message-mode)
    (set-buffer-modified-p t)
    (save-buffer)  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
    (let ((actions message-postpone-actions))
      (message-bury (current-buffer))
      (message-do-actions actions)))

So message-dont-send saves the buffer.  And that buffer has a
buffer-file-coding-system set in message-set-auto-save-file-name,
which is called when message-mode is turned on (which is supposed to
happen when you press 'm' in Gnus):

    (setq buffer-file-coding-system message-draft-coding-system)))

And message-draft-coding-system is:


  (defvar message-draft-coding-system mm-auto-save-coding-system
    "Coding system to compose mail.")

which in my testing, starting from "emacs -Q", produces utf-8-emacs,
as I'd expect.

In my testing, the draft is saved using utf-8-emacs, and if I them
visit it, I get the Cyrillic text back.  As expected.

So where does the above breaks?  Is the value of
mm-auto-save-coding-system different?  Otherwise, save-buffer should
have encoded the draft's text.  What did I miss?

bug#75397: 29.4; GNUS message mode lacks Cyrillic support (GNU/Linux)

[Andreas Schwab]

On Jan 08 2025, Eli Zaretskii wrote:

>> Cc: 75397@debbugs.gnu.org
>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Date: Mon, 06 Jan 2025 10:36:34 +0100
>> 
>> On Jan 06 2025, Nickolai Dobrynin wrote:
>> 
>> > 1. Type M-gnus to start GNUS.
>> > 2. Start composing a message by pressing 'm'.
>> > 3. Enter the first, say, 5 letters of the Cyrillic alphabet, i.e. абвгд.
>> > 4. Press C-c C-d to save the draft.
>> > 5. Press S D e to resume editing.
>> > 6. Bizarrely enough, instead of "абвгд", you will see a sequence of digits "01234".
>> >
>> > Is this even real?? By the way, this does not occur with ordinary text
>> > files. And everything works fine in the view mode inside the draft
>> > group. Pressing S D e, on the other hand, makes things go haywire.
>> 
>> The problem is that message-dont-send doesn't encode the message when it
>> saves it.
>
> I must be missing something here, because I don't see what you
> describe above:

It is about MIME encoding.

>   (defun message-dont-send ()
>     "Don't send the message you have been editing.
>   Instead, just auto-save the buffer and then bury it."
>     (interactive nil message-mode)
>     (set-buffer-modified-p t)
>     (save-buffer)  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>     (let ((actions message-postpone-actions))
>       (message-bury (current-buffer))
>       (message-do-actions actions)))

This just saves the raw buffer contents, but
gnus-summary-resend-message-edit expects it to be a proper RFC2822 mail,
with suitable MIME headers.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#75397: 29.4; GNUS message mode lacks Cyrillic support (GNU/Linux)

[Robert Pluim]

>>>>> On Wed, 08 Jan 2025 17:53:16 +0100, Andreas Schwab <schwab@linux-m68k.org> said:

    Andreas> On Jan 08 2025, Eli Zaretskii wrote:
    >>> Cc: 75397@debbugs.gnu.org
    >>> From: Andreas Schwab <schwab@linux-m68k.org>
    >>> Date: Mon, 06 Jan 2025 10:36:34 +0100
    >>> 
    >>> On Jan 06 2025, Nickolai Dobrynin wrote:
    >>> 
    >>> > 1. Type M-gnus to start GNUS.
    >>> > 2. Start composing a message by pressing 'm'.
    >>> > 3. Enter the first, say, 5 letters of the Cyrillic alphabet, i.e. абвгд.
    >>> > 4. Press C-c C-d to save the draft.
    >>> > 5. Press S D e to resume editing.
    >>> > 6. Bizarrely enough, instead of "абвгд", you will see a sequence of digits "01234".
    >>> >
    >>> > Is this even real?? By the way, this does not occur with ordinary text
    >>> > files. And everything works fine in the view mode inside the draft
    >>> > group. Pressing S D e, on the other hand, makes things go haywire.
    >>> 
    >>> The problem is that message-dont-send doesn't encode the message when it
    >>> saves it.
    >> 
    >> I must be missing something here, because I don't see what you
    >> describe above:

    Andreas> It is about MIME encoding.

    >> (defun message-dont-send ()
    >> "Don't send the message you have been editing.
    >> Instead, just auto-save the buffer and then bury it."
    >> (interactive nil message-mode)
    >> (set-buffer-modified-p t)
    >> (save-buffer)  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
    >> (let ((actions message-postpone-actions))
    >> (message-bury (current-buffer))
    >> (message-do-actions actions)))

    Andreas> This just saves the raw buffer contents, but
    Andreas> gnus-summary-resend-message-edit expects it to be a proper RFC2822 mail,
    Andreas> with suitable MIME headers.

Yes. Which is why the command to edit an unsent draft is 'e' (or 'D
e').

Robert
-- 

bug#75397: 29.4; GNUS message mode lacks Cyrillic support (GNU/Linux)

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: ndobrynin@gmail.com,  75397@debbugs.gnu.org
> Date: Wed, 08 Jan 2025 17:53:16 +0100
> 
> On Jan 08 2025, Eli Zaretskii wrote:
> 
> >> Cc: 75397@debbugs.gnu.org
> >> From: Andreas Schwab <schwab@linux-m68k.org>
> >> Date: Mon, 06 Jan 2025 10:36:34 +0100
> >> 
> >> On Jan 06 2025, Nickolai Dobrynin wrote:
> >> 
> >> > 1. Type M-gnus to start GNUS.
> >> > 2. Start composing a message by pressing 'm'.
> >> > 3. Enter the first, say, 5 letters of the Cyrillic alphabet, i.e. абвгд.
> >> > 4. Press C-c C-d to save the draft.
> >> > 5. Press S D e to resume editing.
> >> > 6. Bizarrely enough, instead of "абвгд", you will see a sequence of digits "01234".
> >> >
> >> > Is this even real?? By the way, this does not occur with ordinary text
> >> > files. And everything works fine in the view mode inside the draft
> >> > group. Pressing S D e, on the other hand, makes things go haywire.
> >> 
> >> The problem is that message-dont-send doesn't encode the message when it
> >> saves it.
> >
> > I must be missing something here, because I don't see what you
> > describe above:
> 
> It is about MIME encoding.
> 
> >   (defun message-dont-send ()
> >     "Don't send the message you have been editing.
> >   Instead, just auto-save the buffer and then bury it."
> >     (interactive nil message-mode)
> >     (set-buffer-modified-p t)
> >     (save-buffer)  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
> >     (let ((actions message-postpone-actions))
> >       (message-bury (current-buffer))
> >       (message-do-actions actions)))
> 
> This just saves the raw buffer contents, but
> gnus-summary-resend-message-edit expects it to be a proper RFC2822 mail,
> with suitable MIME headers.

OK, but then where is it documented that "S D e" in Gnus Summary is
supposed to read the draft saved by "C-c C-d" in the mail buffer?
Could it be that these two are not supposed to work together?

The node "Drafts" in the Gnus manual describesa different procedure.
If I follow it, then after typing "D e" in the nndraft:drafts group, I
get placed in the *unsent mail* buffer with the correct Cyrillic text.

Nickolai, did you try to use these different commands in your
scenario?

bug#75397: Any workarounds for this?

[Nickolai Dobrynin]

Hi Andreas,

1. Can you suggest a workaround for this?

2. For clarity's sake, would it be better to change the title of the
bug report to, e.g., "messages get corrupted after being saved to "drafts" b/c
message-dont-send doesn't encode them before saving"? This would
probably summarize the problem much better to the rest of the Emacs team.

bug#75397: 29.4; GNUS message mode lacks Cyrillic support (GNU/Linux)

[Eli Zaretskii]

> From: "Nickolai Dobrynin" <ndobrynin@gmail.com>
> Date: Tue, 07 Jan 2025 19:13:49 +0100
> 
> 1. Can you suggest a workaround for this?
> 
> 2. For clarity's sake, would it be better to change the title of the
> bug report to, e.g., "messages get corrupted after being saved to "drafts" b/c
> message-dont-send doesn't encode them before saving"? This would
> probably summarize the problem much better to the rest of the Emacs team.

No need.  The Emacs team, such as it is, took good note of your
report, and will investigate the ways to fix this soon.  At that time,
we may have suggestions for workarounds or patches for fixing the
problem.  It's just that we didn't have yet time to do so.

For now, the only workaround to try I can suggest is to explicitly
bind the coding-system while saving the draft, like this:

  C-x RET c utf-8-emacs RET C-c C-d

If after that "S D e" doesn't restore the original Cyrillic text, try
this instead:

  C-x RET c utf-8-emacs RET S D e

P.S. Please don't change the Subject line when you respond to messages
in this thread: it makes it a bit harder to find the previous/next
messages when reading the discussions.  I've restored the original
Subject on my response.