Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[Eli Zaretskii]

[I'm taking this to emacs-devel, since I think we need to discuss the issue.]

> From: David Kuehling <dvdkhlng@gmx.de>
> Date: 19 May 2002 17:24:25 +0200
> 
> I had a closer look on how emacs pop3 mail retrieval works.  It uses the
> movemail program for copying the remote mailbox into a local temporary
> file and then gets mail from that file as it would do with a local
> inbox.  So I used the movemail program for manually moving the mail.
> The resulting file, mailbox-popretrieved (attached), shows a few small
> differences from the orignal mailbox.  There are some additional lines
> and flags added to the mails.
> 
> That mail, read using C-u g mailbox-popretrieved <Ret>, now triggers the
> bug, ie EUC-JP is not correctly decoded.  Is this due to the minor
> differences?  Hopefully you can reproduce the bug (else it might already
> be fixed with your version... I'm running 21.1.1).

Yes, I can reproduce this now, and the problem is indeed the format
used by movemail to write the mailbox file: it's the Babyl format.
(That format is not unique to pop3 protocol, though.)

It is actually very simple: when rmail-convert-to-babyl-format sees a
mailbox in Babyl format, it decodes the messages using
decode-coding-region.  Needless to say, decode-coding-region does not
know anything about the special importance of the charset= header, so
it simply summons its usual guesswork trying to detect the encoding of
each message.  The result is that, with some messages, you will get
your default coding system as the message encoding; I'm guessing that
Latin-1 is your default (my default is different, so I got a different
result).

By contrast, when rmail-convert-to-babyl-format sees an mbox format,
it honors the charset= headers, so the result is correct.

Handa-san, can you tell why does the Babyl branch of
rmail-convert-to-babyl-format use decode-coding-region without ever
looking at the charset= header?  It sounds like a bug, since movemail
uses that format.

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[Richard Stallman]

    Handa-san, can you tell why does the Babyl branch of
    rmail-convert-to-babyl-format use decode-coding-region without ever
    looking at the charset= header?  It sounds like a bug, since movemail
    uses that format.

Just the opposite--it is right for Rmail to read a Babyl file this way,
since they are supposed to be saved as a whole using a single coding system.
The bug is in movemail.  If it is to produce a Babyl file, it has to
handle the decoding.  That, however, is not practical.
It seems clear that movemail should not convert to Babyl format.

We are changing Rmail to get rid of Babyl format entirely; when that
change is installed, we will need movemail to output inbox format.  It
sounds like we should make this change now, so as to fix the current
bug.

Does anyone want to do it?

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[Paul Michael Reilly]

 > Just the opposite--it is right for Rmail to read a Babyl file this way,
 > since they are supposed to be saved as a whole using a single coding system.
 > The bug is in movemail.  If it is to produce a Babyl file, it has to
 > handle the decoding.  That, however, is not practical.
 > It seems clear that movemail should not convert to Babyl format.
 > 
 > We are changing Rmail to get rid of Babyl format entirely; when that
 > change is installed, we will need movemail to output inbox format.  It
 > sounds like we should make this change now, so as to fix the current
 > bug.

I don't think this is correct.  I'm using Rmail/mbox and I needed no
changes to movemail.

-pmr

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[Eli Zaretskii]

On Mon, 20 May 2002, Paul Michael Reilly wrote:

> I'm using Rmail/mbox and I needed no changes to movemail.

You mean, you cannot reproduce the problem reported by David with the 
file he sent?

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[Paul Michael Reilly]

 > You mean, you cannot reproduce the problem reported by David with the 
 > file he sent?

No, I never tried to reproduce that problem.

David, please resend your original message to me and I'll look
into it.

Thanks,

-pmr

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[Eli Zaretskii]

On Mon, 20 May 2002, Richard Stallman wrote:

> Just the opposite--it is right for Rmail to read a Babyl file this way,
> since they are supposed to be saved as a whole using a single coding system.

IIRC, Babyl files saved by Emacs are read by a different branch of code 
in rmail-convert-to-babyl.  They don't need to be decoded by 
decode-coding-region because they take the encoding from the 
X-Coding-System: header inserted by the initial conversion.

But I agree that changing movemail to save the mailbox in mbox format 
will solve the problem in a nicer manner.

Re: RMAIL doesn't always recognize rmail-mime-charset-pattern

[Kenichi Handa]

"Eli Zaretskii" <eliz@is.elta.co.il> writes:
> By contrast, when rmail-convert-to-babyl-format sees an mbox format,
> it honors the charset= headers, so the result is correct.

> Handa-san, can you tell why does the Babyl branch of
> rmail-convert-to-babyl-format use decode-coding-region without ever
> looking at the charset= header?  It sounds like a bug, since movemail
> uses that format.

I don't remember.  :-(

Perhaps, I thought that Babyl format is produced only by
rmail, thus, that part should have been already encoded by
rmail-file-coding-sysstem or something.

---
Ken'ichi HANDA
handa@etl.go.jp