RMAIL doesn't always recognize rmail-mime-charset-pattern

RMAIL doesn't always recognize rmail-mime-charset-pattern

[David Kuehling]

[-- Attachment #1: Type: text/plain, Size: 1077 bytes --]

RMAIL didn't decode a mail, which contained a 

Content-Type: text/plain; charset=euc-jp

header.  Instead it decoded it as iso-latin-1-unix.  I wanted to
reproduce the bug, so I saved the mail into INBOX format.  But when I
opened the inbox format mail, the coding system was detected correctly.

I don't understand how this can happen.  When receiving and opening
mails, the decoding is done by the same routine,
rmail-convert-to-babyl-format.

This is difficult to debug and rmail.el is extremly ugly.  If just
someone with more knowledge of rmail could have a look at it...

I attach the INBOX format mail as buggy.mail.gz, the wronlgy converted
BABYL format mail as buggy.xmail.gz.  If you want to try to reproduce
the mail retrieval bug, you can subscribe to the mailinglist, that's
sending those mails:

https://ieeecs.ece.utexas.edu/mailman/listinfo/gakushuu_kotd

(I'm retrieving the mails using a pop3 po:<...> mailbox)

David Kühling
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40


[-- Attachment #2: INBOX-format mail --]
[-- Type: application/octet-stream, Size: 6219 bytes --]

[-- Attachment #3: BABYL-format mail --]
[-- Type: application/octet-stream, Size: 6600 bytes --]

[-- Attachment #4: Type: text/plain, Size: 35 bytes --]


PS: I'm not subscribed, please CC

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[Eli Zaretskii]

> From: David Kuehling <dvdkhlng@gmx.de>
> Date: 18 May 2002 15:18:47 +0200
> 
> RMAIL didn't decode a mail, which contained a 
> 
> Content-Type: text/plain; charset=euc-jp
> 
> header.  Instead it decoded it as iso-latin-1-unix.  I wanted to
> reproduce the bug, so I saved the mail into INBOX format.  But when I
> opened the inbox format mail, the coding system was detected correctly.
> 
> I don't understand how this can happen.  When receiving and opening
> mails, the decoding is done by the same routine,
> rmail-convert-to-babyl-format.

I cannot reproduce this, either.  I took your buggy.mail, removed the
"X-Coding-system:" line, then typed "C-u g buggy.mail RET" inside an
RMAIL buffer.  The message was correctly decoded as EUC-JP.

Did you try to remove the "X-Coding-system:" header, and see if that
allows you to reproduce the problem?  If not, can you please try that?

> (I'm retrieving the mails using a pop3 po:<...> mailbox)

Do you still have the original mailbox retrieved by pop3?  If so, can
you spot some differences between the original file and buggy.mail
you sent with your report?

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[David Kuehling]

>>>>> "Eli" == Eli Zaretskii <eliz@is.elta.co.il> writes:

> I cannot reproduce this, either.  I took your buggy.mail, removed the
> "X-Coding-system:" line, then typed "C-u g buggy.mail RET" inside an
> RMAIL buffer.  The message was correctly decoded as EUC-JP.

I just tried the same.  My Emacs also decoded the message correctly.
That's strange.

> Do you still have the original mailbox retrieved by pop3?  If so, can
> you spot some differences between the original file and buggy.mail you
> sent with your report?

I don't have the original mailbox.  But I'll receive the next mail of
that kind tomorrow.  Hopefully I'll then remember to make a copy of my
mailserver's inbox before using Emacs' mail retrieval...

David Kühling
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[David Kuehling]

[-- Attachment #1: Type: text/plain, Size: 1077 bytes --]

>>>>> "Eli" == Eli Zaretskii <eliz@is.elta.co.il> writes:

> Do you still have the original mailbox retrieved by pop3?

Today I made a copy of my inbox file, before using Emacs' pop3 mail
retrieval.  The bug is now reproducable, when receiving that mail box
via pop3 (2nd mail in the mailbox file).  But it won't occur, when I use
C-u g savedmailbox <Ret>.  That's cool!

And I just discovered another mysterious bug.  The last mailinglist mail
in the mailbox (from debian-japanese) contains a 

Content-Type: TEXT/PLAIN; charset=iso-8859-1

header, although its contents are obviously iso-2022-jp (7bit) encoded.
When receiving the mailbox via pop3, the japanese characters in the mail
show up correctly, and an

X-Coding-System: iso-2022-jp-unix

header is generated.  How can that happen?  The same mail, however, read
via C-u g ... will be decoded as iso-8859-1 (as it should be).

I attach the full mailbox: savedmailbox.gz.

David Kühling
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40


[-- Attachment #2: savedmailbox.gz --]
[-- Type: application/octet-stream, Size: 16121 bytes --]

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[Eli Zaretskii]

On 19 May 2002, David Kuehling wrote:

> Today I made a copy of my inbox file, before using Emacs' pop3 mail
> retrieval.  The bug is now reproducable, when receiving that mail box
> via pop3 (2nd mail in the mailbox file).  But it won't occur, when I use
> C-u g savedmailbox <Ret>.  That's cool!

This probably means that the culprit is something related to process I/O, 
not file I/O.  Do you read your pop3-fetched mail by receiving output of 
some process, or does that process create a file which you then direct 
RMAIL to read?

> And I just discovered another mysterious bug.  The last mailinglist mail
> in the mailbox (from debian-japanese) contains a 
> 
> Content-Type: TEXT/PLAIN; charset=iso-8859-1
> 
> header, although its contents are obviously iso-2022-jp (7bit) encoded.
> When receiving the mailbox via pop3, the japanese characters in the mail
> show up correctly, and an
> 
> X-Coding-System: iso-2022-jp-unix
> 
> header is generated.  How can that happen?  The same mail, however, read
> via C-u g ... will be decoded as iso-8859-1 (as it should be).

Probably due to the same factor: somehow, when RMAIL reads your mail ``as 
usual'', it applies some other default.

I will dwell on this and see what can I come up with.

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[David Kuehling]

[-- Attachment #1: Type: text/plain, Size: 1349 bytes --]

>>>>> "Eli" == Eli Zaretskii <eliz@is.elta.co.il> writes:

> This probably means that the culprit is something related to process
> I/O, not file I/O.  Do you read your pop3-fetched mail by receiving
> output of some process, or does that process create a file which you
> then direct RMAIL to read?

I use the standard pop3 mail facility of emacs; using a "po:username"
mailbox in `rmail-primary-inbox-list' (and the environment variable
MAILHOST specifying the mail server's name).

I had a closer look on how emacs pop3 mail retrieval works.  It uses the
movemail program for copying the remote mailbox into a local temporary
file and then gets mail from that file as it would do with a local
inbox.  So I used the movemail program for manually moving the mail.
The resulting file, mailbox-popretrieved (attached), shows a few small
differences from the orignal mailbox.  There are some additional lines
and flags added to the mails.

That mail, read using C-u g mailbox-popretrieved <Ret>, now triggers the
bug, ie EUC-JP is not correctly decoded.  Is this due to the minor
differences?  Hopefully you can reproduce the bug (else it might already
be fixed with your version... I'm running 21.1.1).

David Kühling
-- 
GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40


[-- Attachment #2: mailbox-popretrieved.gz --]
[-- Type: application/octet-stream, Size: 15852 bytes --]

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[Eli Zaretskii]

> From: Eli Zaretskii <eliz@is.elta.co.il>
> Date: Sun, 19 May 2002 16:27:11 +0300 (IDT)
> 
> Probably due to the same factor: somehow, when RMAIL reads your mail ``as 
> usual'', it applies some other default.

Umm, it is probably important to know what exactly do you type to read
your mail through pop3 (not "C-u g", the other, normal, procedure).
Could you please describe that?