Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

[suzume]

> On Feb 26, 2024, at 2:00, emacs-devel-request@gnu.org wrote:
> 
> Message: 12
> Date: Sun, 25 Feb 2024 08:45:33 +0000
> From: Jean-Christophe Helary
> 	<jean.christophe.helary@traductaire-libre.org>


> Subject:
> 	=?utf-8?Q?Traduction_fran=C3=A7aise_des_manuels_d=E2=80=99Emacs,
> 	_=C3=A9tat_des_lieux.?=

Is there anything I can do on my side to make sure that the subject is 
properly encoded/decoded on the server side? Or is it a setting that 
needs to be modified on the server side?

> Message-ID:
> 	<41159D22-D383-445D-9990-46F79A6D743B@traductaire-libre.org>
> Content-Type: text/plain; charset=utf-8

Re: Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

[Eli Zaretskii]

> Date: Wed, 28 Feb 2024 00:18:59 +0000
> From: "suzume@protonmail.com" <suzume@protonmail.com>
> 
> 
> > On Feb 26, 2024, at 2:00, emacs-devel-request@gnu.org wrote:
> > 
> > Message: 12
> > Date: Sun, 25 Feb 2024 08:45:33 +0000
> > From: Jean-Christophe Helary
> > 	<jean.christophe.helary@traductaire-libre.org>
> 
> 
> > Subject:
> > 	=?utf-8?Q?Traduction_fran=C3=A7aise_des_manuels_d=E2=80=99Emacs,
> > 	_=C3=A9tat_des_lieux.?=
> 
> Is there anything I can do on my side to make sure that the subject is 
> properly encoded/decoded on the server side? Or is it a setting that 
> needs to be modified on the server side?

Sorry, I don't understand what you are asking.  What is "improper"
with the above subject, in your opinion?  And in what MUA are you
looking at it?

Re: Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

[Eli Zaretskii]

> Date: Wed, 28 Feb 2024 12:14:29 +0000
> From: "suzume@protonmail.com" <suzume@protonmail.com>
> Cc: emacs-devel@gnu.org
> 
> > On Feb 28, 2024, at 21:09, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >> Date: Wed, 28 Feb 2024 00:18:59 +0000
> >> From: "suzume@protonmail.com" <suzume@protonmail.com>
> >> 
> >> 
> >>> On Feb 26, 2024, at 2:00, emacs-devel-request@gnu.org wrote:
> >>> 
> >>> Message: 12
> >>> Date: Sun, 25 Feb 2024 08:45:33 +0000
> >>> From: Jean-Christophe Helary
> >>> 	<jean.christophe.helary@traductaire-libre.org>
> >> 
> >> 
> >>> Subject:
> >>> 	=?utf-8?Q?Traduction_fran=C3=A7aise_des_manuels_d=E2=80=99Emacs,
> >>> 	_=C3=A9tat_des_lieux.?=
> >> 
> >> Is there anything I can do on my side to make sure that the subject is
> >> properly encoded/decoded on the server side? Or is it a setting that
> >> needs to be modified on the server side?
> > 
> > Sorry, I don't understand what you are asking.  What is "improper"
> > with the above subject, in your opinion?  And in what MUA are you
> > looking at it?
> 
> This is how it looks in the digest listing here:

Which is okay, I think.  That's how the author's MUA sent it; the
Subject is encoded using quoted-printable method, I think.

> I'm using Apple's Mail.app. I don't usually have issues displaying 
> contents in French/Japanese, etc. even mixed with other languages.

Maybe something is wrong here, then.  I'm not an expert on this stuff;
does anyone know whether this Subject encoding is OK or not?

Re: Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

[Jean-Christophe Helary]

> On Feb 28, 2024, at 21:20, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Date: Wed, 28 Feb 2024 12:14:29 +0000
>> From: "suzume@protonmail.com" <suzume@protonmail.com>
>> Cc: emacs-devel@gnu.org
>> 
>>> On Feb 28, 2024, at 21:09, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>>> Date: Wed, 28 Feb 2024 00:18:59 +0000
>>>> From: "suzume@protonmail.com" <suzume@protonmail.com>
>>>> 
>>>> 
>>>>> On Feb 26, 2024, at 2:00, emacs-devel-request@gnu.org wrote:
>>>>> 
>>>>> Message: 12
>>>>> Date: Sun, 25 Feb 2024 08:45:33 +0000
>>>>> From: Jean-Christophe Helary
>>>>> 	<jean.christophe.helary@traductaire-libre.org>
>>>> 
>>>> 
>>>>> Subject:
>>>>> 	=?utf-8?Q?Traduction_fran=C3=A7aise_des_manuels_d=E2=80=99Emacs,
>>>>> 	_=C3=A9tat_des_lieux.?=
>>>> 
>>>> Is there anything I can do on my side to make sure that the subject is
>>>> properly encoded/decoded on the server side? Or is it a setting that
>>>> needs to be modified on the server side?
>>> 
>>> Sorry, I don't understand what you are asking.  What is "improper"
>>> with the above subject, in your opinion?  And in what MUA are you
>>> looking at it?
>> 
>> This is how it looks in the digest listing here:
> 
> Which is okay, I think.  That's how the author's MUA sent it; the
> Subject is encoded using quoted-printable method, I think.

Sorry, the author is me, Jean-Christophe, I started this thread too, 
because if there is an issue with the way I send mails, maybe I should 
fix that.

>> I'm using Apple's Mail.app. I don't usually have issues displaying
>> contents in French/Japanese, etc. even mixed with other languages.
> 
> Maybe something is wrong here, then.  I'm not an expert on this stuff;
> does anyone know whether this Subject encoding is OK or not?

Re: Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2100 bytes --]

On Wed, Feb 28, 2024 at 02:20:17PM +0200, Eli Zaretskii wrote:
> > Date: Wed, 28 Feb 2024 12:14:29 +0000
> > From: "suzume@protonmail.com" <suzume@protonmail.com>
> > Cc: emacs-devel@gnu.org
> > 
> > > On Feb 28, 2024, at 21:09, Eli Zaretskii <eliz@gnu.org> wrote:
> > > 
> > >> Date: Wed, 28 Feb 2024 00:18:59 +0000
> > >> From: "suzume@protonmail.com" <suzume@protonmail.com>
> > >> 
> > >> 
> > >>> On Feb 26, 2024, at 2:00, emacs-devel-request@gnu.org wrote:
> > >>> 
> > >>> Message: 12
> > >>> Date: Sun, 25 Feb 2024 08:45:33 +0000
> > >>> From: Jean-Christophe Helary
> > >>> 	<jean.christophe.helary@traductaire-libre.org>
> > >> 
> > >> 
> > >>> Subject:
> > >>> 	=?utf-8?Q?Traduction_fran=C3=A7aise_des_manuels_d=E2=80=99Emacs,
> > >>> 	_=C3=A9tat_des_lieux.?=
> > >> 
> > >> Is there anything I can do on my side to make sure that the subject is
> > >> properly encoded/decoded on the server side? Or is it a setting that
> > >> needs to be modified on the server side?
> > > 
> > > Sorry, I don't understand what you are asking.  What is "improper"
> > > with the above subject, in your opinion?  And in what MUA are you
> > > looking at it?
> > 
> > This is how it looks in the digest listing here:
> 
> Which is okay, I think.  That's how the author's MUA sent it; the
> Subject is encoded using quoted-printable method, I think.
> 
> > I'm using Apple's Mail.app. I don't usually have issues displaying 
> > contents in French/Japanese, etc. even mixed with other languages.
> 
> Maybe something is wrong here, then.  I'm not an expert on this stuff;
> does anyone know whether this Subject encoding is OK or not?

I didn't dive into all details ($DAYJOB), but prima vista it looks
OK. Mime [1] prescribes "Encoded Word" for Unicode encoding in mail
headers, and this very much looks like Q encoding.

Now one might rightfully argue that the client is supposed to decode
that and show Unicode in its full glory (other MUAs do it). Perhaps
this is the OP's point?

Cheers

[1] https://en.wikipedia.org/wiki/MIME#Encoded-Word
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 391 bytes --]

On Wed, Feb 28, 2024 at 12:23:29PM +0000, Jean-Christophe Helary wrote:

[...]

> Sorry, the author is me, Jean-Christophe, I started this thread too, 
> because if there is an issue with the way I send mails, maybe I should 
> fix that.

The only you can do is avoid non-ASCII in header fields. But then
you don't help advancing MIME support in mail clients :-)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

[Yuri Khan]

On Wed, 28 Feb 2024 at 19:21, Eli Zaretskii <eliz@gnu.org> wrote:

> > >>> Subject:
> > >>>   =?utf-8?Q?Traduction_fran=C3=A7aise_des_manuels_d=E2=80=99Emacs,
> > >>>   _=C3=A9tat_des_lieux.?=

> > This is how it looks in the digest listing here:
>
> Which is okay, I think.  That's how the author's MUA sent it; the
> Subject is encoded using quoted-printable method, I think.
>
> Maybe something is wrong here, then.  I'm not an expert on this stuff;
> does anyone know whether this Subject encoding is OK or not?

I think the operative word here is “digest”. The above encoded string
is quoted not from the Subject header field on an individual message
but from the body of a message that combines multiple mailing list
messages.

In the ideal world, the server software producing digests will take
each individual message, decode its subject according to the encoding
declared, decode the body according to the Content-Type: text/plain;
charset= header, build the digest in Unicode land, then encode to
UTF-8 and mail that to digest subscribers.

I’m suspecting the digest formatter might not be MIME-aware with
respect to subject encoding.

Re: Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

[Jean-Christophe Helary]

> On Feb 28, 2024, at 21:54, Yuri Khan <yuri.v.khan@gmail.com> wrote:
> 
> On Wed, 28 Feb 2024 at 19:21, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>>>>>> Subject:
>>>>>>  =?utf-8?Q?Traduction_fran=C3=A7aise_des_manuels_d=E2=80=99Emacs,
>>>>>>  _=C3=A9tat_des_lieux.?=
> 
>>> This is how it looks in the digest listing here:
>> 
>> Which is okay, I think.  That's how the author's MUA sent it; the
>> Subject is encoded using quoted-printable method, I think.
>> 
>> Maybe something is wrong here, then.  I'm not an expert on this stuff;
>> does anyone know whether this Subject encoding is OK or not?
> 
> I think the operative word here is “digest”.

Indeed.

> The above encoded string
> is quoted not from the Subject header field on an individual message
> but from the body of a message that combines multiple mailing list
> messages.

That's correct.

> In the ideal world, the server software producing digests will take
> each individual message, decode its subject according to the encoding
> declared, decode the body according to the Content-Type: text/plain;
> charset= header, build the digest in Unicode land, then encode to
> UTF-8 and mail that to digest subscribers.
> 
> I’m suspecting the digest formatter might not be MIME-aware with
> respect to subject encoding.

Is that something that we need to report to the GNU project? Other 
mailman lists that I use don't seem to have that issue.

Re: Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1285 bytes --]

On Wed, Feb 28, 2024 at 07:54:34PM +0700, Yuri Khan wrote:
> On Wed, 28 Feb 2024 at 19:21, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > >>> Subject:
> > > >>>   =?utf-8?Q?Traduction_fran=C3=A7aise_des_manuels_d=E2=80=99Emacs,
> > > >>>   _=C3=A9tat_des_lieux.?=

[...]

> > Maybe something is wrong here, then.  I'm not an expert on this stuff;
> > does anyone know whether this Subject encoding is OK or not?
> 
> I think the operative word here is “digest”. The above encoded string
> is quoted not from the Subject header field on an individual message
> but from the body of a message that combines multiple mailing list
> messages.

Gotcha. I just missed that (now that you mention it, it's up there,
in the Subject line -- guilty as charged).

> In the ideal world, the server software producing digests will take
> each individual message, decode its subject according to the encoding
> declared, decode the body according to the Content-Type: text/plain;
> charset= header, build the digest in Unicode land, then encode to
> UTF-8 and mail that to digest subscribers.
> 
> I’m suspecting the digest formatter might not be MIME-aware with
> respect to subject encoding.

You nailed it, I think. The digestor is being naive.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

[Kazuhiro Ito]

> >> I'm using Apple's Mail.app. I don't usually have issues displaying
> >> contents in French/Japanese, etc. even mixed with other languages.
> > 
> > Maybe something is wrong here, then.  I'm not an expert on this stuff;
> > does anyone know whether this Subject encoding is OK or not?

Obviously not.
I read your original message from newsgroup via Wanderlust.  Your
message's subject is below.

--------
Subject: =?utf-8?Q?Traduction_fran=C3=A7aise_des_manuels_d=E2=80=99Emacs,
 _=C3=A9tat_des_lieux.?=
--------

SPACE and LF within encoded-word are not allowed.
RFC 2047 (in section 2) says

> IMPORTANT: 'encoded-word's are designed to be recognized as 'atom's
> by an RFC 822 parser.  As a consequence, unencoded white space
> characters (such as SPACE and HTAB) are FORBIDDEN within an
> 'encoded-word'.  For example, the character sequence
>
>    =?iso-8859-1?q?this is some text?=
>
> would be parsed as four 'atom's, rather than as a single 'atom' (by
> an RFC 822 parser) or 'encoded-word' (by a parser which understands
> 'encoded-words').  The correct way to encode the string "this is some
> text" is to encode the SPACE characters as well, e.g.
>
>    =?iso-8859-1?q?this=20is=20some=20text?=

-- 
Kazuhiro Ito

Re: Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

[Andreas Schwab]

On Feb 28 2024, Yuri Khan wrote:

> I think the operative word here is “digest”. The above encoded string
> is quoted not from the Subject header field on an individual message
> but from the body of a message that combines multiple mailing list
> messages.

The non-digest message that was received by the mailing list has exactly
the same encoded subject header, so the digest context does not change
anything here.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Encoding issue in the digest (Re: Emacs-devel Digest, Vol 240, Issue 25)

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  "suzume@protonmail.com"
>  <suzume@protonmail.com>,  emacs-devel@gnu.org
> Date: Wed, 28 Feb 2024 14:28:44 +0100
> 
> On Feb 28 2024, Yuri Khan wrote:
> 
> > I think the operative word here is “digest”. The above encoded string
> > is quoted not from the Subject header field on an individual message
> > but from the body of a message that combines multiple mailing list
> > messages.
> 
> The non-digest message that was received by the mailing list has exactly
> the same encoded subject header, so the digest context does not change
> anything here.

Right, I see that as well.

So I guess if the subject is not correctly encoded, the culprit is the
MUA used by Jean-Christophe, or maybe some of the MTA servers on the
way between his MUA and us the recipients.